Paper deep dive

SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

Arther Tian, Alex Ding, Simon Wu, Aaron Chan

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 37

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%

Last extracted: 7/22/2026, 3:19:02 AM

Summary

The paper introduces SFGA, a statistics-first gating architecture for supervised fine-tuning (SFT) data procurement. SFGA evaluates candidate corpora on diversity, utility, and redundancy using cheap blind measurements. A gate decides whether to accept a verdict based on statistical confidence; if uncertain, it escalates to an adjudicative debate between buy and reject advocates. The system achieves 0.90 accuracy at $0.017 per unit, outperforming baselines while exposing biases in LLM judges.

Entities (9)

SFGA · system · 100%Diversity · metric · 95%Redundancy · metric · 95%Utility · metric · 95%Buy-Advocate · agent · 90%Dolly-15k · dataset · 90%Reject-Advocate · agent · 90%Supervised Fine-Tuning · task · 90%LLM Judge · model · 85%

Relation Signals (8)

SFGA → usesmetric → Utility

confidence 95% · SFGA measures three intrinsic quality axes of a candidate corpus—diversity, utility, and redundancy

SFGA → usesmetric → Redundancy

confidence 95% · SFGA measures three intrinsic quality axes of a candidate corpus—diversity, utility, and redundancy

SFGA → usesmetric → Diversity

confidence 95% · SFGA measures three intrinsic quality axes of a candidate corpus—diversity, utility, and redundancy

SFGA → escalatesto → Adjudicative Debate

confidence 90% · otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge

SFGA → evaluateson → Dolly-15k

confidence 90% · We construct a 2×3×2 grid of injected quality knobs on top of Dolly-15k

Adjudicative Debate → involvesagent → Buy-Advocate

confidence 90% · adjudicative debate between a buy-advocate and a reject-advocate judge

Adjudicative Debate → involvesagent → Reject-Advocate

confidence 90% · adjudicative debate between a buy-advocate and a reject-advocate judge

LLM Judge → exhibitsbias → Positional Bias

confidence 85% · expose negativity and positional biases that a naive LLM-judge would hide

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring. We present \sys{}, a statistics-first gating architecture that treats procurement as a cost-aware routing problem over three intrinsic quality axes -- diversity, utility, and redundancy. Cheap blind measurements are summarised into per-axis estimates with confidence intervals; a gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree, otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge, resolved by a presiding verdict. On a controlled benchmark of 12 datasets ($2{\times}3{\times}2$ grid over the three axes) with 5 seeds, the gate reaches 0.90 accuracy and 0.83 $F_1$ at \$0.017 per unit, sitting between an always-verify baseline (0.75) and an oracle upper bound (0.98) while spending less than always-escalate (\$0.020). We further report honest negative diagnostics of the debate path: a con-side win rate of 0.80 ($p\approx3{\times}10^{-6}$) and a 52\% position-flip rate under advocate swapping expose negativity and positional biases that a naive LLM-judge would hide. We frame the injected-knob evaluation explicitly as a controlled synthetic benchmark for measurement fidelity and routing calibration, and delimit external validity as future work.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

35,735 characters extracted from source content.

Expand or collapse full text

SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement Arther Tiana, Alex Dinga,*, Frank Chena Simon Wua, Aaron Chana aDGrid AI *Corresponding author: alex.ding@dgrid.ai Abstract Procuring supervised fine-tuning (SFT) data forces a buyer to decide, before any downstream training, whether a candidate corpus is worth acquiring. We present SFGA, a statistics-first gating architecture that treats procurement as a cost-aware routing problem over three intrinsic quality axes—diversity, utility, and redundancy. Cheap blind measurements are summarised into per-axis estimates with confidence intervals; a gate accepts a decision only when intervals are tight, sample sizes are adequate, and the axes agree, otherwise it escalates the case to an adjudicative debate between a buy-advocate and a reject-advocate judge, resolved by a presiding verdict. On a controlled benchmark of 12 datasets (2×3×22×3×2 grid over the three axes) with 5 seeds, the gate reaches 0.90 accuracy and 0.83 F1F_1 at $0.017 per unit, sitting between an always-verify baseline (0.75) and an oracle upper bound (0.98) while spending less than always-escalate ($0.020). We further report honest negative diagnostics of the debate path: a con-side win rate of 0.80 (p≈3×10−6p≈ 3×10^-6) and a 52% position-flip rate under advocate swapping expose negativity and positional biases that a naive LLM-judge would hide. We frame the injected-knob evaluation explicitly as a controlled synthetic benchmark for measurement fidelity and routing calibration, and delimit external validity as future work. 1 Introduction Supervised fine-tuning (SFT) [20, 26] increasingly runs on data a team did not create itself: corpora purchased from vendors, scraped and repackaged by third parties, or pooled across an organization. Whoever pays for such data must decide, before committing to a fine-tuning run, whether a candidate corpus is worth acquiring at all [10, 13]—a judgment made under a budget, on a corpus that is usually too large to read and too opaque to trust. The decision is two-sided: acquire a corpus that quietly degrades the model and the buyer loses both money and a training cycle; pass over a genuinely useful corpus and the opportunity is gone. Procurement, not post-hoc filtering, is where the money is committed, and it is the layer we target. Three families of methods bear on this judgment, each with a characteristic failure. Human audit is the most trusted—careful human curation of even a small corpus can suffice for alignment [29]—but it scales poorly and cannot be afforded at corpus volume. A single-shot LLM judge [28, 18] scales but is costly per unit at scale and, as we quantify in Section˜5, is biased and poorly calibrated: its verdict swings with the order in which alternatives are presented [24] and skews systematically toward rejection. Statistics over cheap intrinsic measurements is the least expensive option and, on clear-cut cases, is genuinely decisive; but at the boundaries—when an estimate straddles a threshold, when the sample is too small to support a conclusion, or when quality axes disagree—it commits confidently to answers it has no basis for. We take these methods to be complementary rather than competing, and argue that the scarce resource—expensive adjudication—should be spent only where cheap statistics are demonstrably insufficient. SFGA measures three intrinsic quality axes of a candidate corpus—diversity, utility, and redundancy—summarizes each as an interval-valued estimate, and consults a gate. The gate commits to a statistics-only verdict only when the intervals are tight, the samples adequate, and the axes unanimous; every ambiguous, borderline, or conflicting case is escalated to an adjudicative debate [8, 12] between a buy-advocate and a reject-advocate, resolved by a presiding verdict. The same escalation path that rescues hard cases is also where judge pathologies live, so we audit it directly—via advocate swapping—rather than trusting its verdicts at face value. While this cheap-first, escalate-when-uncertain, cost-aware philosophy is shared with a broader line of the authors’ work on trustworthy evaluation [3], both the problem addressed here—data procurement—and the mechanism—a statistical gate over corpus-level quality axes—are new. On a controlled benchmark of 12 datasets spanning a 2×3×22×3×2 grid of injected quality knobs, run across 5 seeds, SFGA reaches 0.90 routing accuracy at $0.017 per unit—close to an oracle that routes with ground truth (0.98) and, notably, cheaper than escalating every case ($0.020) while far more accurate than never escalating (0.75). It is also the best-calibrated strategy we test. Against a random router matched to its exact budget (0.82), the remaining gap isolates the value of the routing decision from the value of the money spent. Figure˜1 situates SFGA on the cost–reliability frontier of existing practice. We frame the injected-knob evaluation explicitly as a controlled study of measurement fidelity and routing calibration, and treat external validity as future work (Section˜6). Cost per unitReliabilityexisting cost–reliabilityfrontierPure statisticscheap, brittle at boundariesSingle LLM judgecostly, biasedHuman audittrusted, unscalableSFGA (ours)escalate only the hard cases Figure 1: Positioning. Existing procurement practice trades cost against reliability; SFGA spends cheaply by default and pays for adjudication only on contested cases. SFGA makes three contributions: • A statistics-first procurement gate. A decision rule over interval-valued estimates of diversity, utility, and redundancy that commits to a cheap verdict only when the statistics are decisive and unanimous, and otherwise routes to adjudication (Section˜3). • Adjudicative escalation with honest diagnostics. A buy/reject advocate debate with a presiding verdict, audited for positional and negativity bias via advocate swapping—turning judge pathologies into measured quantities rather than hidden failure modes (Section˜5). • A controlled benchmark and cost-matched evaluation. A 2×3×22×3×2 synthetic grid with a knob-derived gold label used strictly for measurement-fidelity and routing-calibration analysis, plus a cost-matched random baseline that isolates the gate’s contribution from its budget (Sections˜4 and 5). 2 Related Work Data quality and procurement. Two recent surveys map the fast-growing space of data selection for language models [2, 21]. A separate line quantifies the worth of data through its marginal contribution to a trained model, most influentially via the Shapley value [10, 13]. These valuation methods answer a different question from ours and at a different price: they require training or retraining to estimate a datum’s contribution, whereas we decide whether to acquire a corpus before any training, from cheap intrinsic measurements plus selective adjudication. SFT data selection and filtering. A large body of work shows that small, carefully chosen instruction sets can match or beat much larger ones [29, 6], and proposes automatic criteria for choosing them: complexity, quality, and diversity in DEITA [17], tag-based diversity in InsTag [19], instruction-following difficulty in IFD and Superfiltering [16, 15], and learnability-style scores in Instruction Mining [5]. Redundancy and diversity have their own tools—exact and semantic deduplication [14, 1] and sentence embeddings for coverage [22]—and the whole enterprise rests on instruction-tuning foundations [20, 26, 25]. The crucial distinction is the unit of decision: these methods score or filter individual samples to assemble a training subset, while we operate at the corpus layer and emit a procurement verdict, governed by a statistical stopping rule and an escalation path rather than a per-sample score. LLM-as-judge and multi-agent debate. Using strong models to evaluate outputs is now standard [28, 18], but such judges carry well-documented biases—their verdicts depend on option order and other superficial factors [24], which directly motivates our position-swap diagnostic. Debate has been proposed both to improve model reasoning [8] and as an alignment mechanism in which adversarial advocates surface information for a judge [12]. We borrow the adversarial structure but put it to a narrower use—debate is not our default but the escalation path for contested procurement cases only—and we treat the judge’s biases as quantities to be measured rather than assumed away. Statistical testing and sequential decision. The gate is built from classical tools: the Wilson interval for proportions [27], sequential hypothesis testing [23], the bootstrap for interval estimation [9], and the Brier score [4] and modern calibration analysis [11] for scoring probabilistic verdicts. Our contribution is not a new estimator but their composition into a decision rule that recognizes when statistics are sufficient and, only otherwise, pays for adjudication. 3 System Design 3.1 Problem setup and notation A candidate data package D is inspected by scanning a fixed prefix. For each quality axis a∈div,util,reda∈\div,util,red\ a blind measurement yields a point estimate s^a s_a with a confidence interval CIa=[ℓa,ha]CI_a=[ _a,h_a] (width wCI,a=ha−ℓaw_CI,a=h_a- _a) over n observations. A gate emits a path decision δ∈verify,escalateδ∈\ verify, escalate\; the chosen path returns a procurement recommendation ρ∈buy,caution,reject,ask_moreρ∈\ buy, caution, reject, ask\_more\ and a calibrated probability p p. The controlled gold label g⋆g (Section˜4) is used for offline evaluation only and never enters the online decision. 3.2 Main architecture Figure˜2 shows the end-to-end pipeline. Candidate SFT data package Dscan fixed prefix (∼ 200 rows)Blind measurementsdiversity (domain-bucket entropy) ⋅· utility (instr.–resp. usability)redundancy (near-duplicate) ⋅· authenticity (aux.)Statistics engineper-axis proportion / entropy CI→ StatResult(s^a,CIa,n)( s_a,\,CI_a,\,n)Gate (decision hub, Equation˜3)decisive? wCI,a≤0.25∧n≥20w_CI,a\!≤\!0.25 n\!≥\!20 for ≥2/3≥ 2/3 axes∧ non-borderline ∧ unanimousverify (cheap)stats → recommendationbuy, caution, reject, ask_moreescalate (debate)PRO-BUY judge ∥\;\|\; CON-BUY judge→ one round → verdictposition-swap ablation (both orderings)Verdict & procurement recommendationcalibrated p p ⋅· decision ρ trackerlatency → $Gold g⋆g controlled trutheval-onlydecisive & unanimousweak / borderline / conflictoffline scoring only Figure 2: End-to-end architecture. Cheap blind measurements feed a statistics engine whose interval-valued estimates drive a gate (red, the decision hub). Decisive and unanimous cases take the cheap verify path; weak, borderline, or conflicting cases escalate to an adjudicative debate between buy- and reject-advocates with a presiding verdict. The controlled gold label g⋆g (left) is used for offline scoring only and never enters the online decision. 3.3 The gate rule An axis estimate is decisive when it has adequate sample size and a tight interval: decisive(a)=[na≥nmin]∧[wCI,a≤wmax].decisive(a)\;=\; [\,n_a≥ n_ \, ]\; \; [\,w_CI,a≤ w_ \, ]. (1) A decisive axis is classified relative to its threshold τa _a with a borderline band of half-width m: side(a)=borderlineℓa<τa<ha or |s^a−τa|≤m,passclears τa in the good direction,failotherwise.side(a)= cases borderline& _a< _a<h_a or | s_a- _a|≤ m,\\ pass&clears _a in the good direction,\\ fail&otherwise. cases (2) The gate takes the cheap verify path iff a fraction f of axes are decisive, none are borderline, and the decisive axes are unanimous; otherwise it escalates: δ=verify|a:decisive|≥f|A|,none borderline, all agree,escalateweak CI, borderline, or pass/fail conflict.δ= cases verify&|\a:decisive\|≥ f\,|A|,\ none borderline, all agree,\\[2.0pt] escalate&weak CI, borderline, or pass/fail conflict. cases (3) We use nmin=20n_ =20, wmax=0.25w_ =0.25, f=2/3f=2/3, m=0.05m=0.05, thresholds τdiv=0.15 _div=0.15, τutil=0.55 _util=0.55, τred=0.25 _red=0.25, and z=1.96z=1.96 intervals (all in Table˜5). 3.4 Adjudicative escalation When the gate returns escalate—because intervals are wide, an estimate is borderline, or the axes disagree—the case is handed to an adjudicative debate rather than to a single judge. Two advocates receive the same statistical evidence, including which metrics are flagged as weak, but are assigned opposing and fixed stances: a buy-advocate argues that the corpus is acceptable to purchase, and a reject-advocate argues against acquisition and stresses risk. Each advocate first states an opening argument confined to intrinsic quality—diversity and coverage, instruction–response utility, and within-pack redundancy—and then, in a single rebuttal round, answers the opponent’s opening. A presiding chair reads both openings and both rebuttals alongside the statistics and returns a procurement recommendation ρ∈buy,caution,reject,ask_moreρ∈\ buy, caution, reject, ask\_more\ together with a calibrated buy-probability p p. Figure˜3 traces the full branch. The escalation path is at once the most expensive component and the one most exposed to the documented biases of LLM judges, so we instrument it rather than trust it. For every escalated case we additionally run the debate with the two advocate roles swapped in presentation order; a verdict that changes under this swap reflects dependence on position rather than on evidence. We report the resulting positional and negativity diagnostics in Section˜5 and treat them as first-class results rather than caveats: the purpose of a statistics-first design is precisely to keep this fallible path off the critical route for the cases that cheap statistics already settle. Per-axis StatResult(s^a,CIa,na)( s_a,\ CI_a,\ n_a)≥2/3≥\!2/3 axesdecisive?any axisborderline?decisive axesunanimous?verify: map stats → recommendationrecommendation ρ + calibrated p p escalate: PRO/CON debate → chair verdict yesnounanimousnot decisiveborderlineconflict Figure 3: Gate decision flow. The single most load-bearing figure: the gate is a branch, not a waterfall. 4 Experimental Setup Controlled benchmark. We construct a 2×3×22×3×2 grid of injected quality knobs on top of Dolly-15k [7]: a diversity mode (low/high), a utility degradation rate (0/25/50%0/25/50\%), and a redundancy duplication rate (0/30%0/30\%), giving 12 datasets of a few hundred instruction–response pairs each. Every knob has a known ground-truth value. Diversity is set by sampling across nine domain buckets, so the high mode approaches uniform coverage; the utility rate is the fraction of pairs deliberately corrupted, so a 25%25\% rate fixes utiltrue=0.75util_true=0.75; and the redundancy rate is the fraction of near-duplicates injected. Each dataset is run over 5 seeds, yielding 60 routing units. Gold label (offline only). The procurement gold is a deterministic function of the injected knobs: a corpus should be bought iff its true diversity, utility, and (low) redundancy all clear fixed thresholds, divtrue≥0.15∧utiltrue≥0.55∧redtrue≤0.25div_true≥ 0.15 _true≥ 0.55 _true≤ 0.25. This label defines correctness for accuracy, F1F_1, and calibration, but it never enters the online decision—the gate and both paths see only the blind measurements. Because the same knobs that define the gold are also what the measurements recover, we treat the benchmark as a controlled test of measurement fidelity and routing calibration rather than of downstream external validity, a distinction we return to in Section˜6. Blind measurements. Each axis is estimated from a scanned prefix with no access to the knobs: diversity as the entropy of the empirical distribution over the nine domain buckets, utility as the pass rate of an instruction–response usability check, and redundancy as the near-duplicate rate; an auxiliary authenticity rate is also computed. Every estimate carries an interval and a sample count, as formalized in Section˜3. Judges and cost. Escalated cases invoke three LLM roles—the buy- and reject-advocates and the presiding chair (Section˜3)—drawn from a single model family. All calls are real, and the per-unit cost is the token-derived dollar cost accumulated along a unit’s path. Restricting to one model family and one base corpus is a deliberate scope choice whose limits we state in Section˜6. Metrics and baselines. We score each strategy by routing accuracy and F1F_1 against the gold label, by mean per-unit cost in dollars, and by the calibration (expected calibration error and Brier score) of its self-reported buy-probability. Table˜1 lists the strategies we compare, from a cheap statistics-only floor to an oracle upper bound, including a random router held to the gate’s exact budget so that the routing decision can be separated from the money spent. Table 1: Baselines and evaluators (roles). Full config in Table˜5. Method Mechanism Role SFGA stats-first route → verify/escalate proposed always_verify statistics only, never escalate cheap floor always_escalate debate on every unit expensive ceiling oracle_route routes with gold upper bound random_cost_matched random path at matched budget budget control stats_only statistics with no recommendation mapping ablation direct_llm single-shot judge, no debate weak baseline 5 Results 5.1 Measurement fidelity (controlled) Everything downstream depends on the blind measurements being faithful proxies for the injected knobs, so we check this first. Across all three axes the measured estimates recover the injected values almost perfectly (Spearman ρ≈1.00ρ≈ 1.00; Figure˜4). We read this conservatively: because the measurements and the gold label are both functions of the same injected knobs, near-perfect recovery is close to an identity by construction. It establishes that the measurement layer is not the bottleneck on this bench—a necessary sanity check—but it is not, and we do not present it as, evidence of downstream external validity (Section˜6). Figure 4: Measurement fidelity on the controlled bench: blind estimates vs. injected ground truth for each axis, with the identity line. Recovery is near-perfect by construction and is a fidelity check, not external validity. 5.2 Routing Figure 5: Routing accuracy vs. mean per-unit cost. The gate (star) sits near the oracle in accuracy while spending less than always-escalate, and beats a random router held to its exact budget (dotted, equal-budget guide). Table˜2 and Figure˜5 are the headline comparison. Routing the blind measurements through the gate reaches 0.90 accuracy and 0.83 F1F_1 at $0.017 per unit. Two comparisons carry the result. First, against the cheap always-verify floor (0.75), escalation buys a fifteen-point accuracy gain: the cases the gate forwards are exactly the ones statistics alone gets wrong. Second, and less obviously, escalating every case is worse on both axes—always-escalate scores 0.83 accuracy and only 0.69 F1F_1 at a higher $0.020—because indiscriminate debate drags the biased adjudication path (quantified in Section˜5.5) onto cases the statistics had already settled correctly. The gate thus captures most of the oracle’s accuracy (0.98) without ever seeing the gold, and its margin over a random router held to its exact budget (0.82 at the same $0.017) shows the gain comes from where it spends, not merely how much. Table 2: Routing results. The gate approaches the oracle at a fraction of the always-escalate detour, and beats the cost-matched random baseline. Strategy Acc. F1F_1 Cost ($) n SFGA (gate) 0.9000.900 0.8330.833 0.01710.0171 60 always_verify 0.7500.750 0.7270.727 0.00430.0043 60 always_escalate 0.8330.833 0.6870.687 0.01960.0196 60 oracle_route 0.9830.983 0.9760.976 0.01050.0105 60 random_cost_matched 0.8180.818 – 0.01710.0171 60 stats_only 0.7500.750 0.7270.727 0.00000.0000 36 direct_llm 0.5000.500 0.3080.308 0.00000.0000 36 On the routing units the gate escalates 50 of the 60 and keeps 10 on the cheap path, confirming that it reserves spending for statistically ambiguous cases. The stripped ablations bound the two ends: statistics with no recommendation mapping (stats_only) matches the verify floor at 0.75, while a single-shot judge with no debate (direct_llm) collapses to 0.50, near chance. Neither statistics alone nor a lone judge suffices; the structure that routes between them is what does the work. 5.3 Stratified behaviour Decomposing the aggregate by diversity stratum shows where the difficulty lies. The low-diversity stratum is easy: every low-diversity corpus fails the gold’s diversity threshold and should be rejected, so there are no positive cases, the gate rejects all 30 correctly (1.00 accuracy), and F1F_1 is undefined by construction rather than by failure. The real test is the high-diversity stratum, where buy and reject cases coexist; there the gate reaches 0.80 accuracy and 0.83 F1F_1. We report the split so the headline 0.90 is read correctly—as an average over an easy half and a hard half, not a uniform number. Table 3: Stratified gate accuracy. The dlow stratum contains no positive (buy) cases, so its F1F_1 is undefined-by-construction rather than a failure. Stratum Acc. F1F_1 n dlow 1.0001.000 0.0000.000 30 dhigh 0.8000.800 0.8330.833 30 5.4 Calibration Figure 6: Calibration error (ECE and Brier; lower is better) per strategy. The gate (highlighted) is the best calibrated, ahead of the always-verify floor and even of oracle-route. Correct routing is not enough for procurement, where a buyer acts on a probability rather than a bare label. The gate’s self-reported buy-probability is the best calibrated of all strategies (ECE 0.241, Brier 0.144; Figure˜6, with exact values in Table˜4). It is better calibrated than oracle-route (ECE 0.413), which routes perfectly using the gold yet reports overconfident probabilities, and than the always-verify floor (ECE 0.400). Blending tight statistics on easy cases with an adjudicated probability on hard ones yields confidence that tracks correctness better than either path does alone. Table 4: Calibration of self-reported buy-probability (exact values for Figure˜6). Strategy ECE Brier n SFGA (gate) 0.2410.241 0.1440.144 60 always_escalate 0.2550.255 0.1520.152 60 random_cost_matched 0.2450.245 0.1690.169 60 always_verify 0.4000.400 0.3020.302 60 oracle_route 0.4130.413 0.2210.221 60 stats_only 0.4180.418 0.3060.306 36 direct_llm 0.4150.415 0.3980.398 36 5.5 Debate diagnostics (honest negatives) Figure 7: Diagnostics of the escalation path: (a) the reject advocate wins most debates; (b) verdicts flip on about half of position-swapped pairs; (c) escalate scores still track true quality, but noisily; (d) they are far less stable across seeds than verify scores. Auditing the escalation path is where we report against our own method (Figure˜7). The reject advocate wins 0.80 of debates (p≈3×10−6p≈ 3× 10^-6), a clear negativity skew; and swapping the advocates’ presentation order flips the verdict on 52% of the 60 paired cases, a strong positional bias—precisely the failure modes documented for LLM judges [24]. These are not fatal to the design. The escalate scores still track true quality (Spearman 0.94, with four adjacent-rank violations), and they are far noisier across seeds than the verify scores (mean standard deviation 0.098 vs. 0.011), which is the empirical signature of a path that should be used sparingly. This is the core argument for a statistics-first architecture: the adjudication path is useful but unreliable, so the gate is built to keep it off the critical route for every case the cheap statistics can already settle (Section˜3). 6 Discussion Implications. The central finding is that spending should be routed, not scaled. On this bench a statistics-first gate captures nearly all of the oracle’s routing value (0.90 vs. 0.98 accuracy) while consulting the expensive adjudication path on only the cases cheap statistics cannot settle—and it does so at lower cost than escalating every case, which is not merely wasteful but actively worse because it exposes clear-cut cases to a biased judge. The lesson generalizes beyond this system: when an accurate-but-expensive evaluator has known pathologies, a cheap decisive front end that gates access to it can be better, on both cost and accuracy, than using the expensive evaluator everywhere. Practitioner guidelines. The gate exposes a small number of interpretable knobs that trade budget for caution. Widening the maximum interval width wmaxw_ or lowering the decisive fraction f sends more cases to the cheap path, cutting cost at the risk of committing on thin evidence; tightening them, or widening the borderline band m, escalates more and buys accuracy with money. Because the adjudication path carries a measurable negativity skew, a buyer who cares more about missing good corpora than about admitting weak ones should treat a reject verdict from debate as weaker evidence than a buy verdict, and the calibrated probability—not the label—should drive the acquisition decision. Limitations. Three limits bound our claims. First and most important, the evaluation is closed-loop: the gold label is a deterministic function of the injected knobs, and the blind measurements recover those same knobs, so our accuracy figures demonstrate internal measurement fidelity and routing calibration, not that a gate-approved corpus actually trains a better model. We have been careful throughout to frame the results this way. Second, the study uses a single base corpus (Dolly-15k) and a single judge model family, so transfer across corpora and model families is untested. Third, the low-diversity stratum contains no positive cases, so part of the aggregate accuracy comes from an easy all-reject regime, as the stratified analysis makes explicit. 7 Conclusion and Future Work We presented SFGA, a statistics-first gate that treats SFT data procurement as cost-aware routing: cheap interval-valued measurements of diversity, utility, and redundancy decide most cases directly, and only ambiguous, borderline, or conflicting ones are escalated to an adjudicative debate. On a controlled benchmark the gate approaches oracle accuracy at a fraction of the always-escalate detour, is the best-calibrated strategy tested, and—rather than hide the debate path’s behaviour—quantifies its negativity and positional biases, turning the case for keeping such a path off the critical route into a measured argument. The clearest next step follows directly from the main limitation: replace the knob-derived gold with a real short SFT run and a held-out evaluation, so that correctness is defined by whether a gate-approved corpus actually improves a model, breaking the closed loop. Beyond that, adding a second base corpus and a second judge family would test whether the routing behaviour and the measured biases transfer, and richer escalation protocols—more debate rounds, or an aggregation over swapped orderings that cancels positional bias—may narrow the remaining gap to the oracle. References [1] A. Abbas, K. Tirumala, D. Simig, S. Ganguli, and A. S. Morcos (2023) SemDeDup: data-efficient learning at web-scale through semantic deduplication. arXiv preprint arXiv:2303.09540. Cited by: §2. [2] A. Albalak, Y. Elazar, S. M. Xie, S. Longpre, N. Lambert, X. Wang, N. Muennighoff, B. Hou, L. Pan, H. Jeong, C. Raffel, S. Chang, T. Hashimoto, and W. Y. Wang (2024) A survey on data selection for language models. arXiv preprint arXiv:2402.16827. Cited by: §2. [3] Anonymous (2024) Prior work by the authors on trustworthy, cost-aware evaluation (anonymized for review). Note: Full citation withheld for double-blind review; to be restored in the camera-ready version. Cited by: §1. [4] G. W. Brier (1950) Verification of forecasts expressed in terms of probability. Monthly Weather Review 78 (1), p. 1–3. Cited by: §2. [5] Y. Cao, Y. Kang, and L. Sun (2023) Instruction mining: high-quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290. Cited by: §2. [6] L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V. Yadav, Z. Tang, V. Srinivasan, T. Zhou, H. Huang, and H. Jin (2024) AlpaGasus: training a better alpaca with fewer data. In International Conference on Learning Representations (ICLR), Note: arXiv:2307.08701 Cited by: §2. [7] M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin (2023) Free dolly: introducing the world’s first truly open instruction-tuned LLM. Note: Databricks Blogdatabricks-dolly-15k dataset Cited by: §4. [8] Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch (2024) Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning (ICML), Note: arXiv:2305.14325 Cited by: §1, §2. [9] B. Efron (1979) Bootstrap methods: another look at the jackknife. The Annals of Statistics 7 (1), p. 1–26. Cited by: §2. [10] A. Ghorbani and J. Zou (2019) Data shapley: equitable valuation of data for machine learning. In Proceedings of the 36th International Conference on Machine Learning (ICML), p. 2242–2251. Note: arXiv:1904.02868 Cited by: §1, §2. [11] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger (2017) On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning (ICML), p. 1321–1330. Note: arXiv:1706.04599 Cited by: §2. [12] G. Irving, P. Christiano, and D. Amodei (2018) AI safety via debate. arXiv preprint arXiv:1805.00899. Cited by: §1, §2. [13] R. Jia, D. Dao, B. Wang, F. A. Hubis, N. Hynes, N. M. Gürel, B. Li, C. Zhang, D. Song, and C. J. Spanos (2019) Towards efficient data valuation based on the shapley value. In Proceedings of the 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), p. 1167–1176. Cited by: §1, §2. [14] K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini (2022) Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), Note: arXiv:2107.06499 Cited by: §2. [15] M. Li, Y. Zhang, S. He, Z. Li, H. Zhao, J. Wang, N. Cheng, and T. Zhou (2024) Superfiltering: weak-to-strong data filtering for fast instruction-tuning. arXiv preprint arXiv:2402.00530. Cited by: §2. [16] M. Li, Y. Zhang, Z. Li, J. Chen, L. Chen, N. Cheng, J. Wang, T. Zhou, and J. Xiao (2023) From quantity to quality: boosting LLM performance with self-guided data selection for instruction tuning. arXiv preprint arXiv:2308.12032. Cited by: §2. [17] W. Liu, W. Zeng, K. He, Y. Jiang, and J. He (2024) What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In International Conference on Learning Representations (ICLR), Note: arXiv:2312.15685 Cited by: §2. [18] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu (2023) G-Eval: NLG evaluation using GPT-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 2511–2522. Note: arXiv:2303.16634 Cited by: §1, §2. [19] K. Lu, H. Yuan, Z. Yuan, R. Lin, J. Lin, C. Tan, C. Zhou, and J. Zhou (2024) #InsTag: instruction tagging for analyzing supervised fine-tuning of large language models. In International Conference on Learning Representations (ICLR), Note: arXiv:2308.07074 Cited by: §2. [20] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe (2022) Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 35, p. 27730–27744. Note: arXiv:2203.02155 Cited by: §1, §2. [21] Y. Qin, Y. Yang, P. Guo, G. Li, H. Shao, Y. Shi, Z. Xu, Y. Gu, K. Li, and X. Sun (2024) Unleashing the power of data tsunami: a comprehensive survey on data assessment and selection for instruction tuning of language models. arXiv preprint arXiv:2408.02085. Cited by: §2. [22] N. Reimers and I. Gurevych (2019) Sentence-BERT: sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP-IJCNLP), p. 3982–3992. Note: arXiv:1908.10084 Cited by: §2. [23] A. Wald (1945) Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics 16 (2), p. 117–186. Cited by: §2. [24] P. Wang, L. Li, L. Chen, Z. Cai, D. Zhu, B. Lin, Y. Cao, Q. Liu, T. Liu, and Z. Sui (2024) Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), p. 9440–9450. Note: arXiv:2305.17926 Cited by: §1, §2, §5.5. [25] Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023) Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), p. 13484–13508. Note: arXiv:2212.10560 Cited by: §2. [26] J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le (2022) Finetuned language models are zero-shot learners. In International Conference on Learning Representations (ICLR), Note: arXiv:2109.01652 Cited by: §1, §2. [27] E. B. Wilson (1927) Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22 (158), p. 209–212. Cited by: §2. [28] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica (2023) Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 36. Note: arXiv:2306.05685 Cited by: §1, §2. [29] C. Zhou, P. Liu, P. Xu, S. Iyer, J. Sun, Y. Mao, X. Ma, A. Efrat, P. Yu, L. Yu, S. Zhang, G. Ghosh, M. Lewis, L. Zettlemoyer, and O. Levy (2023) LIMA: less is more for alignment. In Advances in Neural Information Processing Systems (NeurIPS), Note: arXiv:2305.11206 Cited by: §1, §2. Appendix A Hyperparameters and gate configuration Table 5: Gate and debate configuration. Parameter Value min sample size nminn_ 20 max CI width wmaxw_ 0.25 decisive fraction f 0.67 borderline margin m 0.05 large-n threshold 50 CI z 1.96 diversity threshold τdiv _div 0.15 utility threshold τutil _util 0.55 redundancy threshold τred _red 0.25 authenticity threshold 0.55 seeds 5 grid 2×3×22× 3× 2 (12 datasets)

Instant research discovery

SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

Intelligence

Summary

Entities (9)

Relation Signals (8)

Abstract

Tags

Links

PDF

Full Text

Paper	Published	Area	Tags	Intel	Citations