Paper deep dive
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
Yitao Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/20/2026, 5:59:18 AM
Summary
The paper introduces a method to disambiguate between 'system drift' (changes in the product being evaluated) and 'judge drift' (changes in the LLM used as a monitor) in continuous LLM evaluation pipelines. It proposes a dual-process monitoring system: a main process using a cheap proxy and a strong judge to detect system drift, and a second 'anchor' process using a fixed, human-labeled set to detect judge drift. By using anytime-valid e-processes and a guard-window rule, the system can attribute drift to either the system or the judge with high precision. Experimental results show the method successfully detects silent version bumps and prompt changes while significantly outperforming traditional rolling z-tests in false-alarm rates and cost-efficiency.
Entities (7)
Relation Signals (4)
Anchor Set → detects → Judge Drift
confidence 100% · The anchor process is therefore a one-way mirror: it sees judge drift, and nothing else.
Gemini-3.5-Flash → isexampleof → Judge Drift
confidence 100% · a silent version bump (gemini-3.1-pro → 3.5-flash) detected as judge drift
LLM Judge → issubjectto → Judge Drift
confidence 100% · a silent version bump or a silent prompt-and-policy update can change how it scores
E-process → monitors → System Drift
confidence 100% · a per-stratum prediction-powered e-process... that accrues anytime-valid evidence of system drift
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.
Tags
Links
- Source: https://arxiv.org/abs/2606.15474v1
- Canonical: https://arxiv.org/abs/2606.15474v1
Trouble viewing inline? Open PDF directly →
Full Text
95,900 characters extracted from source content.
Expand or collapse full text
Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines Yitao Li yitaoli416@gmail.com Abstract Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores — so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in none,system,judge\ none, system, judge\. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/6060/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110110 of 120120 runs at guard width 300300 — while the industry-default rolling z-test false-alarms on 75%75\% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240240/240). The monitor runs at ≈0.64≈ 0.64 of the cost of strong-judging every item, or 0.210.21 in a cheaper-but-deafer regime. 1 Introduction Large language model products are now evaluated continuously by other large language models. An LLM-as-judge (Zheng et al., 2023) scores each interaction, a dashboard tracks the rolling mean, and an on-call team is paged when the score drifts down. The implicit contract is that the judge is a fixed yardstick, so a falling score means a falling product. But the strong judge is itself a model behind an API: a version bump or a silent prompt-and-policy update can change how it scores, with no announcement and no change to the product at all. When the monitor fires, the team faces a question the dashboard cannot answer — which thing drifted, the system or the judge? Acting on the wrong answer is expensive in both directions: rolling back a healthy product, or shipping a real regression because the alarm was dismissed as “just the judge.” This is not a hypothetical edge case; it is a property of the tools already in use. Consider the de-facto industry practice for watching a judge: re-test the judge-versus-human gap on a rolling window with a z-test at α=0.05α=0.05, every time a new observation arrives. Run on the same drift-free stream our anchor process consumes, that procedure false-alarms on 75%75\% of streams where nothing ever changes, and 1616 of 6060 of its “detections” fire before the change point even exists (Section˜5). Its apparent vigilance is the absence of error control, not sensitivity: a fixed-α test re-run at every step has no validity guarantee at a data-dependent stopping time. Continuous monitoring demands anytime-valid tooling — e-values and test supermartingales that may be inspected at every step without inflating the false-alarm rate (Ville, 1939; Waudby-Smith and Ramdas, 2022; Howard et al., 2022). Once those are in place, the detector reliably tells you that something drifted. Attribution — whether it was the system or the judge — is the remaining hole, and it is the one we fill. The construction. We start from the prediction-powered monitor a cost-conscious team would deploy anyway: a cheap judge μ scored on every item, the expensive strong judge y sampled under a budget, and a per-stratum prediction-powered e-process (Csillag et al., 2025b; Angelopoulos et al., 2023) that accrues anytime-valid evidence of system drift (Section˜3). To this we add one component: a fixed, human-labeled anchor set A, held out before monitoring begins and re-scored by the current judge at a steady interleave, driving a second e-process that watches the judge’s gap to its frozen human labels (Section˜4). A guard-window rule combines the two stopping times into a verdict in none,system,judge\ none, system, judge\. The construction turns on a deliberate asymmetry. The anchors are frozen, so the product’s drift can never reach them — improve the system or wreck it, the gap to human labels on those fixed items does not move. Only a change in the judge can move it. The anchor process is therefore a one-way mirror: it sees judge drift, and nothing else. Contributions. 1. The anchor construction and guard-window attribution rule (Section˜4) — a human-anchored, anytime-valid e-process that disentangles judge drift from system drift, emitting an explicit none,system,judge\ none, system, judge\ verdict rather than an undifferentiated alarm. 2. Four propositions (Section˜4): anytime-validity of the anchor family, one-way identification (only the judge can move the anchors), the attribution race, and process orthogonality. The race proposition exposes a design law — the anchor process must be provisioned to out-run the main process it guards, since a contaminated reference can otherwise fire a spurious system alarm before the anchors catch up. 3. Two real judge changes, not only synthetic drift (Section˜5): a silent version bump (gemini-3.1-pro → 3.5-flash) detected as judge drift in 60/6060/60 runs with 0 judge-to-system misattribution at the featured configuration, and a strict-prompt policy change that genuinely contaminates the main monitor yet is correctly attributed on 110110 of 120120 contaminated runs at guard width W=300W=300 — both replicated exhibit-for-exhibit on a second domain (TL;DR summarization) with nothing re-tuned. The replication is a test of the theory, not only of robustness: the invariances transfer (the same version bump leaves the same +0.03+0.03 to +0.07+0.07 signature in both domains), and where the domains differ, the differences are the ones the propositions predict — the strict-prompt shift is larger on summaries, so the anchors fire faster, attribution becomes perfect (240/240240/240), and the co-provisioning band moves toward slower anchor rates, exactly the attribution race. 4. Operational evidence that classical alternatives fail (Section˜5): on the identical anchor stream, naive repeated z-testing false-alarms on 75%75\% of drift-free streams, and a carefully calibrated Page–Hinkley detector holds its false-alarm budget but detects the real lenient bump only 8%8\% of the time and requires held-out calibration streams the e-process does not. 5. The cost-aware foundation (Section˜3): a stratified prediction-powered monitor that detects localized blind-spot regressions at roughly one third the cost of strong-evaluating every item, with an e-wealth escalation trigger that is far cheaper than covariate-triggered prior rules (which oversample to near full-evaluation cost or read inert under a biased cheap judge), while merely tying a matched fixed budget on the faintest drift. Section˜2 situates these contributions; Section˜3 builds the budgeted main monitor; Section˜4 adds the anchor construction and its guarantees; and Section˜5 validates all of them on two datasets and two real judge changes before Section˜6 discusses limitations. 2 Related Work Prediction-powered inference and anytime-valid testing. Prediction-powered inference (Angelopoulos et al., 2023, 2024) uses a cheap predictor to debias a small set of expensive labels, and has been extended in many directions: cross-fitting (Zrnic and Candès, 2024), frequentist–assisted–Bayes constructions (Kilian et al., 2025), federated-and-bagged variants (Cortinovis and Caron, 2025), conformal procedures (Csillag et al., 2025a), and power analysis (Chen et al., 2026). Our main monitor builds on the prediction-powered e-value of Csillag et al. (2025b), whose Thm. 2.1 our per-stratum process inherits, and sits in the broader anytime-valid tradition: test (super)martingales and Ville’s inequality (Ville, 1939), time-uniform concentration (Howard et al., 2025, 2022), betting confidence sequences (Waudby-Smith and Ramdas, 2022), e-detectors for sequential change (Shin et al., 2023), sequential comparison of forecasters (Choe and Ramdas, 2023), and prediction-powered risk monitoring of deployed models (Zhang et al., 2026). We claim no new validity theory here: the combination of a cheap-proxy PPI estimator, an anytime-valid e-process, and drift detection already exists in this cluster. Our Section˜3 contribution is the specific assembly — a per-(rubric×stratum) Bonferroni e-process for localized blind-spot drift, plus an e-wealth escalation trigger — benchmarked against this cluster on a single cost-versus-latency frontier. The genuinely new material is the anchor construction and attribution analysis of Section˜4. Active and adaptive acquisition. Several lines couple a sampling rule to a downstream inference or detection target: active inference that labels where uncertainty is highest (Zrnic and Candès, 2026), the active-acquisition rule in Csillag et al.’s Appendix B.2 (which we include as a baseline), round-robin active change detection (Chaudhuri et al., 2024), adaptive-sampling change detection (Yi and Yang, 2025), and drift-to-action controllers that pair a detector with an acquisition policy (Lamaakal et al., 2026). A cautionary result frames our escalation finding: Sfyraki and Wang (2026) prove that for prediction-powered mean estimation, uncertainty-driven adaptive sampling is asymptotically no better than a fixed budget. Our result is consistent with theirs — our e-wealth trigger merely ties a matched fixed budget on the faintest drift — and our claim is narrowly about cost-efficiency relative to other adaptive triggers (covariate-driven rules waste budget or read inert under a biased cheap judge), not asymptotic dominance over fixed-τ. LLM-as-judge reliability. A growing literature studies the reliability of LLM judges (Zheng et al., 2023): correctly reporting results obtained with an imperfect judge (Lee et al., 2026), valid downstream inference under noisy judge labels (Feng et al., 2026), and quantifying judge bias and uncertainty (Fiedler, 2026). That work corrects or qualifies a static judge — it treats the judge’s errors as a fixed, if unknown, distortion to be debiased. We address a different failure mode: a judge whose behavior changes over time, and the problem of separating those changes from changes in the system under evaluation. To our knowledge, no prior work runs a human-anchored e-process specifically to detect and attribute drift in the judge’s identity. 3 Cost-Aware Monitoring with a Budgeted Strong Judge 3.1 Stream model A deployment emits items i=1,2,…i=1,2,… in a stream. Each item is scored on R rubrics, with every score normalized to [0,1][0,1], and carries an observable stratum label t∈1,…,Kt∈\1,…,K\ (here a topic). Two judges are available. A cheap judge μ is run on every item and returns a vector μi∈[0,1]R _i∈[0,1]^R; a strong judge y returns yi∈[0,1]Ry_i∈[0,1]^R but is expensive and is therefore queried only on a sampled subset of items, under a budget. In this section y is the trusted reference: a drop in y is, by assumption, a real drop in quality. 3.2 A per-stratum prediction-powered e-process The monitor maintains one e-process per (rubric × stratum) cell, testing the one-sided null that the cell mean has not fallen below a healthy bar, H0sys:[yi[r]∣i∈cell (r,t)]≥q0[r,t]−0.05,H_0^sys: [y_i[r] i (r,t)]\;≥\;q_0[r,t]-0.05, (1) where the per-cell bar q0[r,t]q_0[r,t] is the mean strong-judge score over in-control items of stratum t, estimated once from held-out calibration data, and the 0.050.05 slack absorbs calibration noise. Each cell accumulates wealth through prediction-powered e-values (Csillag et al., 2025b): the cheap score μi[r] _i[r] supplies a debiasing prediction, and on sampled items the strong label corrects it, so a small number of strong queries yields a valid e-value with the cheap judge as free side information (Angelopoulos et al., 2023). The bets are predictable — chosen from the past — and capped at an effective λmax(πi) _ ( _i) that depends only on the (predictable) sampling probability πi _i, which keeps each cell’s wealth a nonnegative supermartingale under Equation˜1. An alarm fires when any cell’s wealth crosses the Bonferroni threshold RK/αsysRK/ _sys. Validity is then inherited from Csillag et al. (2025b, Thm. 2.1) together with the predictability of π and Ville’s inequality (Ville, 1939): the family’s probability of ever falsely alarming, at any data-dependent stopping time, is at most αsys _sys. The per-stratum design is not a refinement but a necessity. A localized blind-spot drift — a quality regression confined to one stratum, on a rubric where the cheap judge is blind — is diluted to invisibility when pooled. Empirically, under the mild drift a single global per-rubric PPI e-process never detects at any budget (censoring rate 1.001.00 at strong-sampling rates of 0.10.1, 0.20.2, and 0.30.3), whereas the stratified monitor detects at the same budgets; the localization is what makes the signal visible. 3.3 Acquisition modes The budget can be spent two ways. Under a fixed budget the strong judge is queried with probability πi=τ _i=τ (allocated uniformly, by stratum, or by a bandit). Under evidence-triggered escalation the probability rises from a floor π0 _0 toward 11 as a stratum’s accumulated e-wealth approaches its alarm threshold, πi=π0+(1−π0)⋅clip(i−1threshold, 0, 1). _i\;=\; _0+(1- _0)·clip\! ( W_i-1threshold,\,0,\,1 ). (2) Crucially, πi _i is a function of past wealth only, so it is predictable and anytime-validity is preserved by construction. The trigger keys on accumulated e-evidence rather than on the cheap covariate, because the cheap judge is blind to exactly the drift we are trying to catch. 3.4 Cost model Each judge call costs (input tokens)⋅(input price)+(output tokens)⋅(output price)(input tokens)·(input price)+(output tokens)·(output price), with representative counts of ≈600≈ 600 input and ≈30≈ 30 output tokens. At the measured prices this is $0.000195 0.000195 per cheap call and $0.00156 0.00156 per strong call (the cheap judge runs on all L processed items; the strong judge on S sampled items). Relative to strong-evaluating every item, the cost fraction is cost-fraction=Lcμ+ScyLcy= 0.125+SL,cost-fraction\;=\; L\,c_μ+S\,c_yL\,c_y\;=\;0.125+ SL, (3) where full evaluation (π≡1π≡ 1) is 1.01.0 by definition. The 0.1250.125 is the irreducible cheap floor and S/LS/L is the realized strong-sampling rate. 3.5 What the foundation buys On a semi-synthetic localized drift, every variant of the detector catches the regression at cost-fraction 0.210.21–0.430.43 — about one third of full evaluation — with measured false-alarm 0.0000.000 across all 1616 policy points. The two modes trace a cost–reliability frontier: escalation occupies the cheap corner (0.210.21–0.280.28, sitting near the 0.125+π00.125+ _0 floor when nothing is wrong), fixed budgets buy reliability (fixed@0.30.3 reaches censoring 0.280.28 at cost 0.430.43), and full evaluation sets the latency floor (median 327327 steps to detection) at full price. On the cost–detection plane (Figure˜1) this ordering is visible on both datasets: evidence-triggered escalation lies up-and-left of the fixed-budget family — more detection per strong-judge dollar once π0 _0 clears the hard-drift floor — while paying in detection latency, its weak axis, which the plot omits and Table˜6 records. We defer the frontier table and the full ten-method comparison to Appendix˜B. One contrast from that comparison matters here: keying acquisition on the cheap covariate is wasteful or inert — a boundary-proximity trigger detects only by oversampling to cost ≈0.87≈ 0.87–0.890.89. The published predicted-e-growth rule, meanwhile, reads identically zero under an overrating cheap judge, whereas e-wealth escalation detects at roughly one third of that cost. On faint drift, escalation merely ties a matched fixed budget, consistent with the result of Sfyraki and Wang (2026) that uncertainty-driven sampling is asymptotically no better than fixed-τ for prediction-powered mean estimation. Figure 1: The C1 cost–detection frontier on HelpSteer2 (left) and TL;DR (right): detection power (1−censor1-censor) against cost-fraction (log axis; full strong-evaluation =1.0=1.0). Evidence-triggered escalation (ours, blue) lies up-and-left of the fixed-budget family — whose three allocation policies (uniform, stratified, bandit) coincide, so they are drawn as one centroid line with a hairline min–max band — detecting more per strong-judge dollar across the π0 _0 ladder; full evaluation (⋆ ) buys the final increment of power at full price. Detection latency, escalation’s weak axis, is omitted here and tabulated per policy in Table˜6. The TL;DR panel is the nothing-re-tuned replication detailed in Section˜5. Everything above rests on trusting y. The next section drops that assumption. 4 Disentangling Judge Drift: the Anchor Construction 4.1 The problem with a drifting reference The monitor of Section˜3 treats the strong judge y as ground truth. But y is itself an LLM service: a version bump or a silent prompt/policy update can change its scoring without notice. When that happens, a main-process alarm is ambiguous — did the system get worse, or did the judge change? — and the failure is compounding: the contaminated y feeds the very process meant to certify system drift, so the detector can confidently report a system regression that is really an artifact of its own reference moving. We disentangle the two with a second, independent e-process anchored to fixed human labels. 4.2 The anchor set Definition 1. An anchor set =a1,…,akA=\a_1,…,a_k\ is a fixed, held-out collection of items with frozen human labels h(a)∈[0,1]Rh(a)∈[0,1]^R, drawn before monitoring starts and excluded from all main-process calibration. At every position i≡0(modρ)i≡ 0 ρ (an interleave rate 1/ρ1/ρ), one anchor a is drawn uniformly from A and re-scored by the current strong judge, yielding yi(a)y_i(a). Because the anchor items and their human labels never change, the only thing that can move yi(a)y_i(a) is the judge. We summarize each re-judgement by a rescaled gap statistic g~i=clip(yi(a)−h(a)+12, 0, 1)∈[0,1]R, g_i\;=\;clip\! ( y_i(a)-h(a)+12,\,0,\,1 )\;∈\;[0,1]^R, (4) with baseline q~0∈(0,1)R q_0∈(0,1)^R (a per-rubric anchor baseline, not to be confused with the per-cell quality bars q0[r,t]q_0[r,t] of Section˜3), the mean rescaled gap of the time-0 judge over the full anchor set. The baseline is computed once before monitoring; because the time-0 judge scores are cached, q~0 q_0 is the exact mean over A and carries no finite-sample error. This is the asymmetry the construction exploits: system drift changes the incoming items but leaves the fixed anchors untouched, whereas a judge change moves both the main scores and the anchors — the anchors see judge drift and only judge drift. 4.3 The anchor e-process For each rubric r and direction d∈below,aboved∈\below,above\ we run a betting e-process (Waudby-Smith and Ramdas, 2022; Csillag et al., 2025b) with e-values ei= 1+λisd(q~0[r]−g~i[r]),sbelow=+1,sabove=−1,e_i\;=\;1+ _i\,s_d\, ( q_0[r]- g_i[r] ), s_below=+1,\;s_above=-1, (5) where the bet λi∈[0,λmax(q~0[r],d)] _i∈[0, _ ( q_0[r],d)] is predictable (a function of g~1,…,g~i−1 g_1,…, g_i-1 only; in code the plug-in λ≈mean(d)/mean(d2)λ (d)/mean(d^2)). The wealth is n(r,d)=∏i≤neiW_n^(r,d)= _i≤ ne_i, and the family alarms when maxr,dn(r,d)≥2R/αjudge _r,dW_n^(r,d)≥ 2R/ _judge (Bonferroni over the 2R2R rubric–direction pairs). The two directions test, respectively, that the judge has become more lenient or more harsh on the anchors relative to the human labels. 4.4 The attribution rule Let τsys _sys and τanc _anc be the first alarm positions of the main and anchor families (∞ if neither ever fires), and fix a guard window W≥0W≥ 0. The verdict is verdict=noneif neither family fires,judgeif τanc<∞ and (τsys=∞ or τanc≤τsys+W),systemif τsys<∞ and (τanc=∞ or τanc>τsys+W).verdict= cases none&if neither family fires,\\[2.0pt] judge&if _anc<∞ and ( _sys=∞ or _anc≤ _sys+W),\\[2.0pt] system&if _sys<∞ and ( _anc=∞ or _anc> _sys+W). cases (6) The rule tests two nulls. The anchor null H0ancH_0^anc (no judge change) states that, conditional on the past, each observed rescaled gap has mean q~0[r] q_0[r] for every rubric — the current judge’s gap-to-human distribution on the fixed anchor set matches the time-0 baseline. The system null H0sysH_0^sys is the per-stratum main-process null of Equation˜1. When both families fire within the guard window the verdict is judge by design: a contaminated y invalidates the main alarm, so we do not attempt simultaneous attribution. 4.5 Guarantees The validity claims below are inherited from standard test-supermartingale results; the contribution is the construction these results apply to, not new probability theory. We state each proposition and a one-line proof idea, deferring full proofs to Appendix˜A. Proposition 1 (Anytime-validity of the anchor process). Under H0ancH_0^anc, for every rubric–direction pair (n(r,d))n(W_n^(r,d))_n is a nonnegative supermartingale with 0=1W_0=1, and consequently ℙ(∃n:maxr,dn(r,d)≥2R/αjudge)≤αjudge,P (∃\,n:\ _r,dW_n^(r,d)≥ 2R/ _judge )\;≤\; _judge, uniformly over all stopping times: at any data-dependent monitoring horizon, the probability that the anchor family ever falsely fires is at most αjudge _judge. Proof idea. Boundedness of g~i g_i and the directional bound on λi _i make each ei≥0e_i≥ 0; predictability of λi _i together with [g~i[r]∣past]=q~0[r]E[ g_i[r] ]= q_0[r] under H0ancH_0^anc gives [ei∣past]=1E[e_i ]=1, so each wealth is a nonnegative supermartingale. Ville’s inequality (Ville, 1939) bounds each pair’s crossing probability by αjudge/2R _judge/2R, and a union bound over the 2R2R pairs finishes (full proof in Appendix˜A). ∎ Remark 1. The bound needs no distributional assumptions beyond boundedness — the gaps need not be Gaussian, i.i.d. across rubrics, or stationary in variance — and no held-out calibration streams or fixed horizon, in contrast to classical change detectors that must set a threshold against H0H_0 streams and are calibrated only at that horizon. Proposition 2 (Identification). The anchor observations depend only on the fixed items A, their frozen labels h, and the current judge. System drift — any change in the distribution of new items or their true quality — leaves the anchor gap distribution unchanged, so H0ancH_0^anc can be violated only by a change in the judge. Consequently: (a) under pure system drift, ℙ(verdict=judge)≤αjudgeP(verdict= judge)≤ _judge, uniformly in W; and (b) under no drift at all, ℙ(verdict≠none)≤αsys+αjudgeP(verdict≠ none)≤ _sys+ _judge. Proof idea. A false judge verdict requires the anchor family to fire, whose probability Proposition˜1 bounds by αjudge _judge irrespective of W; (b) is a union bound over the two families’ separate false-fire events. See Appendix˜A. ∎ Remark 2. This is the disentanglement asymmetry: the anchor process is a one-way test for judge identity. Its converse direction — catching judge drift quickly enough to win the race against a contaminated main process — is power, not validity, and is the subject of Proposition˜3. Proposition 3 (The attribution race). Under judge drift (with or without simultaneous system drift) the misattribution event verdict=system\verdict= system\ requires the contaminated main process to fire more than W positions before the anchors: ℙ(verdict=system∣judge drift)=ℙ(τsys+W<τanc),P(verdict= system drift)\;=\;P( _sys+W< _anc), which is monotonically non-increasing in the guard width W, in the anchor interleave rate 1/ρ1/ρ, and in the anchor set size k (more anchor evidence per position yields stochastically smaller τanc _anc), and monotonically non-decreasing in the main process’s sampling power (more strong calls on a contaminated y yield stochastically smaller τsys _sys). Proof idea. The event verdict=system\verdict= system\ under judge drift is exactly τsys+W<τanc\ _sys+W< _anc\ by Equation˜6; the monotonicities follow from stochastic ordering of the stopping times in each knob. See Appendix˜A. ∎ Remark 3. The guard window costs attribution latency but never validity: raising W delays system attribution yet cannot manufacture false judge verdicts beyond the αjudge _judge of Proposition˜2(a), precisely because the anchor bound holds at any horizon. The race also leaves detection untouched: the total event drift flagged at all=τsys∧τanc<∞\drift flagged at all\=\ _sys _anc<∞\ does not depend on W; the rule only decides the label. Proposition 4 (Orthogonality). τanc _anc is independent of the main-process configuration (policy, budget, π), and the main process’s distribution is independent of the anchor interleave: anchor re-judgements are out-of-band calls on held-out items that never enter the main e-processes or their calibration. The design therefore splits cleanly — the anchor budget (k,ρ)(k,ρ) and guard W govern judge-drift latency and the race, the main configuration governs system power, and αjudge,αsys _judge, _sys are separate Bonferroni budgets. Proof idea. The anchor observations are a deterministic function of (,h,current judge)(A,h,current judge) and the interleave schedule, none of which depend on the main acquisition rule; symmetrically, held-out anchor calls never enter the main statistics. See Appendix˜A. ∎ Remark 4. Orthogonality holds for the two processes; the verdict still couples them through the race of Proposition˜3. We verify these guarantees empirically in Section˜5: Proposition˜1 and Proposition˜2 as false-judge rates at or below αjudge _judge under pure system drift and no drift; Proposition˜3 through the guard-window sweep and the cost frontier; and Proposition˜4 through the constant judge-latency rows and constant system-power columns of that frontier. Two boundaries are worth stating plainly. The construction claims no optimality — the plug-in λ approximates the log-optimal bet, and the attribution rule is not claimed to minimize misattribution at a fixed budget — and it assumes the human labels h remain a valid reference: if the anchor items’ true quality standard itself drifts conceptually, the anchor process correctly reports judge-vs-anchor disagreement, but its reading as judge drift weakens until a periodic anchor refresh re-establishes the baseline. 5 Experiments We evaluate four claims empirically: the attribution rule disentangles judge drift from system drift (Proposition˜2); the misattribution rate is a tunable race in the guard width, anchor rate, and anchor size (Proposition˜3); the two processes are orthogonal and must be co-provisioned (Proposition˜4); and the anytime-valid anchor process dominates the classical alternatives operationally. Every experiment is run on two datasets — HelpSteer2 assistant-response evals and TL;DR summarization evals — with nothing re-tuned between them: the same judge roster, featured configuration, thresholds, and α budgets. The second dataset thereby functions as an out-of-sample test of the propositions, not only a robustness check: invariances must transfer, and cross-domain differences must land where the theory says (Section˜5.6). We close by showing every property also generalizes across all topic–rubric cells of both datasets. 5.1 Setup We use two datasets. The primary is HelpSteer2 (Wang et al., 2024): 14981498 items, each scored on R=5R=5 rubrics (helpfulness, correctness, coherence, complexity, verbosity) and carrying a topic label over K=6K=6 strata. The second is the TL;DR summarization axis evals (Stiennon et al., 2022): 14981498 summarization items scored on R=4R=4 axes (overall, accuracy, coverage, coherence) with subreddit strata (K=5K=5). On both, the cheap judge μ is gemini-3.1-flash-lite and the strong judge y is gemini-3.1-pro-preview (Google DeepMind, 2025); the frozen anchor labels h are each dataset’s human ratings (HelpSteer2 per-attribute 0–44, TL;DR 77-point Likert, both rescaled to [0,1][0,1] — the anchor baseline gap absorbs the constant scale offset). We monitor streams of length N=1200N=1200 with the change point at position 300300, averaging over 6060 seeds, and set αsys=αjudge=0.1 _sys= _judge=0.1. We inject three judge drifts spanning the realism axis. Real-lenient: every item is re-judged by gemini-3.5-flash, an actual silent version bump, which runs uniformly more lenient than the baseline judge (per-rubric mean gap shift +0.022+0.022 to +0.072+0.072 on HelpSteer2, +0.031+0.031 to +0.071+0.071 on TL;DR — the same bump, the same signature, in both domains). Real-harsh: the same model is re-prompted with a stricter scoring prompt (“v2-strict”), a real judge-behavior change that shifts scores down by −0.043-0.043 to −0.152-0.152 on HelpSteer2 (verbosity the lone exception at +0.025+0.025) and by −0.097-0.097 to −0.230-0.230 on TL;DR (summaries offer more flaws to penalize). Synthetic: a tunable shift −0.25-0.25 plus noise. System drift is injected by drawing the target cell (creative×helpfulness on HelpSteer2, relationships×overall on TL;DR) post-change from a regressed-blind pool (y low, μ high), the exact blind spot that makes the cheap judge alone insufficient. Crossing the two drifts gives four ground-truth conditions none,system,judge,both\ none, system, judge, both\. Unless a knob is swept, we report a single featured configuration: a fixed-budget main process at sampling rate 0.30.3 (the high-power point on the cost frontier of Section˜3) with k=200k=200 anchors interleaved at rate 1/51/5 and guard W=300W=300 (justified in Section˜5.4). This configuration runs at cost-fraction ≈0.64≈ 0.64 (HelpSteer2) / 0.630.63–0.650.65 (TL;DR) of strong-evaluating every item (the 0.1250.125 cheap floor, 0.300.30 main sampling, and 1/51/5 anchor share), versus 1.01.0 for full evaluation; a cheap-but-deaf alternative (escalation π0=0.02 _0=0.02 with rate-2020 anchors) costs 0.210.21 but misses most system regressions. 5.2 Disentanglement and detection Table˜1 reports the verdict confusion matrices under the synthetic harsh shift on both datasets. With matched anchors, system detection is 41/6041/60 (HelpSteer2) and 56/6056/60 (TL;DR) — exactly each fixed@0.30.3 main’s power from Section˜3 — while judge and both are detected 60/6060/60 and 5555–59/6059/60. On HelpSteer2 the pure judge and both rows never land in system (judge→ 0/1200/120); on TL;DR a small spill remains at the default guard (5+1/1205+1/120) — the guard race of Section˜5.3, where widening W closes it. False judge verdicts stay at or below αjudge _judge on both datasets: 2/602/60 on none and 2/602/60 on system (rate ≈0.03<0.1≈ 0.03<0.1, as expected — the rate-55 interleave gives the anchor family four times the observations, so a few α-budget false fires appear by design). The right half of Table˜1 is the co-provisioning lesson. With under-provisioned anchors (k=50k=50, rate 1/201/20), the high-power main reads the contaminated y so fast that it fires roughly 300300 items before the anchors catch up: 4848–50/6050/60 (HelpSteer2) and 5454–58/6058/60 (TL;DR) of the genuine judge-drift runs land in system and the disentanglement collapses. The anchor process must be provisioned to out-run the main process it guards — anchor detection latency (120120/140140 items for the matched anchors, versus 420420/440440 for the slow ones) has to come in under the main’s contaminated-fire latency plus W. Table 1: Verdict confusion (60 seeds, synthetic judge shift −0.25-0.25, fixed@0.3 main, guard W=50W=50), both datasets. Left: matched anchors (k=200k=200, rate 1/51/5). Right: under-provisioned anchors (k=50k=50, rate 1/201/20) — the contaminated main out-runs the anchors and the disentanglement collapses on both datasets. k=200k=200, 1/51/5 (matched) k=50k=50, 1/201/20 true \ pred none system judge none system judge HelpSteer2 (median judge latency 120120 / 420420) none 58 0 2 60 0 0 system 17 41 2 20 40 0 judge 0 0 60 0 48 12 both 0 0 60 0 50 10 TL;DR (median judge latency 140140 / 440440) none 58 0 2 60 0 0 system 2 56 2 2 58 0 judge 0 5 55 0 58 2 both 0 1 59 0 54 6 Table˜2 turns to a real silent version bump and the anchor-budget frontier. The lenient gemini-3.5-flash bump is moderate (mean shift ≤+0.072≤+0.072, magnitude comparable to the synthetic −0.05-0.05/−0.10-0.10 sweep points), so detection depends on anchor budget — on both datasets: 7%7\% at k=50k=50/rate-2020 on each, rising to 100%100\% (60/6060/60) on HelpSteer2 and 83%83\% (50/6050/60) on TL;DR at the featured k=200k=200/rate-55 (the bump is subtler per-axis on TL;DR’s featured cell, so the same budget buys less of the frontier; misses fail safe to none). The pure-judge row never lands in system at any budget on either dataset, and false-judge rates stay under αjudge _judge. The featured fixed@0.30.3 replications hold: judge detection 60/6060/60 at latency 498498 (HelpSteer2) and 50/6050/60 at latency 555555 (TL;DR), judge→ 0. Table 2: A real version bump (gemini-3.1-pro → 3.5-flash) vs anchor budget, both datasets. 60 seeds; escalation main (featured fixed@0.3 replications: 60/60 at latency 498 on HelpSteer2, 50/60 at latency 555 on TL;DR, judge→ 0 on both). anchors rate detection (judge / both) judge→ none→ latency HelpSteer2 (mean shift ≤+0.072≤+0.072) k=50k=50 1/201/20 7% (4/60) / 7% 0 0/60 660 k=150k=150 1/81/8 62% (37/60) / 60% 0 1/60 652 k=200k=200 1/51/5 100% (60/60) / 98% 0 2/60 498 TL;DR (mean shift ≤+0.071≤+0.071) k=50k=50 1/201/20 7% (4/60) / 7% 0 0/60 810 k=150k=150 1/81/8 38% (23/60) / 38% 0 2/60 516 k=200k=200 1/51/5 83% (50/60) / 83% 0 2/60 555 The one-sided caveat. The lenient bump scores y′≥y ≥ y on every rubric, and the main monitor is one-sided “below” — it fires only on quality drops. A lenient judge therefore cannot push the main toward a system alarm at all, so the judge→ =0=0 of Table˜2 is partly structural for this drift direction, not solely the anchors’ doing. Table˜2 thus demonstrates detection of a subtle real bump; the genuine test of disentanglement-under-contamination is a harsher judge change that does push the main downward, which is the subject of Section˜5.3. Both demonstrations are needed; neither alone supports the full claim. 5.3 A real harsh judge change: contamination and the guard race The v2-strict re-judge is a real LLM judge-behavior change: the same model is instructed to “reserve 44 for flawless work … when torn, choose the lower.” Table˜3 characterizes both real drifts per rubric on both datasets — the lenient bump is a uniform leniency shift (≤+0.072≤+0.072) with the same +0.03+0.03 to +0.07+0.07 signature in both domains (a property of the version bump, not the dataset), while the harsh re-judge shifts strictly downward: into the −0.10-0.10 to −0.15-0.15 band on HelpSteer2 (verbosity the lone, near-noise exception at +0.025+0.025) and deeper, −0.10-0.10 to −0.23-0.23, on TL;DR. This is the synthetic sweep’s hardest regime: slow anchors against a fast contaminated main. Note that item-level agreement degrades far more than the means suggest (exact agreement 2626–84%84\%), and the rubrics where the two strong judges agree least (verbosity, complexity, corr 0.460.46–0.600.60) mirror the cheap judge’s own blind spots — the anchor process fires on the systematic component, not the per-item noise, which is why it needs k≈200k≈ 200 anchors for the subtle bump. Table 3: Strong-judge gap y′−y -y per rubric, both real drifts, both datasets (≈1496≈ 1496–14981498 items judged by both judges). shift is the mean gap shift the anchor process monitors; corr and agree% are item-level. The lenient bump moves scores up uniformly with the same signature in both domains; the harsh re-judge moves them down — harder on TL;DR, where summaries offer more flaws to penalize. lenient (3.5-flash) harsh (v2-strict) rubric shift corr agree% shift corr agree% HelpSteer2 helpfulness +0.064+0.064 0.88 63.6% −0.152-0.152 0.90 45.0% correctness −0.008-0.008 0.86 70.0% −0.120-0.120 0.88 55.1% coherence +0.022+0.022 0.76 82.5% −0.092-0.092 0.79 62.9% complexity +0.050+0.050 0.59 72.5% −0.043-0.043 0.57 77.5% verbosity +0.072+0.072 0.46 62.6% +0.025+0.025 0.60 71.0% TL;DR overall +0.048+0.048 0.89 69.7% −0.230-0.230 0.84 26.2% accuracy +0.031+0.031 0.91 84.4% −0.097-0.097 0.89 65.7% coverage +0.071+0.071 0.87 65.6% −0.193-0.193 0.83 31.9% coherence +0.037+0.037 0.81 78.6% −0.134-0.134 0.78 51.0% Table˜4 reports the featured config under this real contamination, sweeping the guard window W on both datasets. The harsh shift genuinely contaminates the one-sided main, but the anchors flip the large majority of contaminated fires to the correct judge verdict, and the residual spill is a tunable race, not a defect: judge→ shrinks monotonically in W — 29→18→1029→ 18→ 10 of 120120 on HelpSteer2 and 5→1→05→ 1→ 0 on TL;DR — exactly the monotonicity of Proposition˜3, because spills happen only when the contaminated main fires more than W items before the anchors. TL;DR reaches a perfect 240/240240/240 over its two drift rows at W=300W=300: its harsh shift is larger (overall −0.23-0.23), so the anchors are faster (median latency 205205 versus 370370) and win the race outright — a bigger judge change is easier to attribute. Critically, the none and system rows are identical at every W on both datasets (58/0/258/0/2 and 17/41/217/41/2; 58/0/258/0/2 and 2/56/22/56/2): widening the guard costs nothing in system detection or false-judge rate — only attribution latency and a sliver of anchor cost. The guard window is the third knob, after anchor size and interleave rate, and the cheapest: it buys correctness with latency rather than dollars. Table 4: Real harsh judge change (v2-strict re-judge), featured config (fixed@0.3 main, k=200k=200/rate-55 anchors), 60 seeds, sweeping guard W, both datasets. Spill = judge→ total over the two drift rows (of 120). The none and system rows are identical at every W on both datasets: guard width costs only attribution latency. TL;DR’s larger shift makes the anchors faster, closing the spill entirely at W=300W=300. guard W judge row (none/sys/judge) both row (none/sys/judge) spill /120 latency cost HelpSteer2 (shifts −0.04-0.04 to −0.15-0.15) 50 0 / 10 / 50 0 / 19 / 41 29 370 0.64 150 0 / 8 / 52 0 / 10 / 50 18 370 0.66–0.67 300 0 / 3 / 57 0 / 7 / 53 10 370 0.68–0.71 TL;DR (shifts −0.10-0.10 to −0.23-0.23) 50 0 / 3 / 57 0 / 2 / 58 5 205 0.63–0.65 150 0 / 1 / 59 0 / 0 / 60 1 205 0.63–0.68 300 0 / 0 / 60 0 / 0 / 60 0 205 0.63–0.73 Figure˜2 shows one seed of the featured config under this real harsh drift on each dataset. The change point is at 300300. On HelpSteer2 the anchor process crosses its threshold (≈4.61≈ 4.61) at position 735735, well before the contaminated main crosses its threshold (≈5.70≈ 5.70) at 10981098; on TL;DR the larger shift accelerates both processes — anchors at 370370, contaminated main at 567567 — but the anchors keep winning. In both panels the verdict is judge: the contaminated main fire lands inside the guard window (shaded) opened by the anchor alarm. Figure 2: The attribution race under real harsh judge drift (one seed, featured config), on HelpSteer2 (left) and TL;DR (right). Both e-process wealth trajectories are shown against their Bonferroni alarm thresholds; the change point is at 300300. The anchor process alarms first on both datasets (735735 vs 10981098 on HelpSteer2; 370370 vs 567567 on TL;DR), so the main fire falls inside the guard window (shaded) and the verdict is correctly judge. 5.4 Co-provisioning: main power × anchor rate Figure˜3 sweeps five main configurations against three anchor interleave rates (real harsh drift, k=200k=200, W=50W=50, 3030 seeds) on both datasets, establishing four structural facts. First, the two processes are orthogonal as designed (Proposition˜4): judge latency depends only on the anchor rate (830/695/430830/695/430 items at rate 20/10/520/10/5 on HelpSteer2; 640/330/202640/330/202 on TL;DR — across every main config), system power depends only on the main config (constant down each main’s rows), and the false-judge rate is flat at ≤0.07≤αjudge≤ 0.07≤ _judge. Second, at under-anchored points the sum of judge-detection and spill is ≈1≈ 1: those points do not miss the drift, they mislabel it — detection and attribution are separate. Third, the spill grows monotonically with main power at fixed rate — on HelpSteer2’s rate-55 column, 0.03→0.07→0.18→0.330.03→ 0.07→ 0.18→ 0.33 (escalation π0=0.02 _0=0.02, fixed@0.10.1, fixed@0.20.2, fixed@0.30.3) — so the feasible region is a diagonal band and the anchor budget must be co-provisioned with main power, not bolted on. The TL;DR panel shows the same geometry entered from the fast-anchor side: its larger harsh shift lets rate-55 anchors out-run even the fixed@0.30.3 main (spill ≤0.05≤ 0.05), and the band sits at rates 1010–2020 instead (0.420.42 at fixed@0.30.3/rate-1010, 0.900.90 at rate-2020) — the race of Proposition˜3, with the drift magnitude setting where the band falls. Fourth, the guard W is the orthogonal escape hatch: the sweep of Section˜5.3 pulled the HelpSteer2 corner’s spill from 2929 to 1010 of 120120 at W=300W=300 (and TL;DR’s from 55 to 0), so points outside the band are recoverable with attribution latency instead of anchor dollars. Together these justify the featured configuration: a high-power main matched with rate-55 anchors and a wide guard. Figure 3: Co-provisioning frontier (real harsh drift, k=200k=200, W=50W=50, 30 seeds), on HelpSteer2 (left) and TL;DR (right). Judge latency depends only on the anchor rate and system power only on the main config (Proposition˜4); the misattribution spill grows along the main-power axis at fixed anchor rate, so the feasible region is a diagonal band — main power and anchor rate must be raised together. TL;DR’s larger harsh shift shifts the band toward slower anchor rates: the same geometry, set by the drift magnitude. 5.5 Baselines: why anytime-validity matters operationally We compare the anchor e-process against two classical change detectors consuming the identical anchor observation stream (k=200k=200, rate 1/51/5, 6060 seeds, both datasets); Table˜5 reports false-alarm rate, detection rate, and latency on each drift. The default practice fails outright. naive-z — a rolling-window z-test against baseline at α=0.05α=0.05, re-tested every observation with no sequential correction, the industry standard — false-alarms on 75%75\% (HelpSteer2) and 67%67\% (TL;DR) of no-drift streams, and 16/6016/60 and 10/6010/60 of its “detections” fire before the change even happens: its apparent speed is the absence of error control, not aggression. ph-calib is Page–Hinkley (Page, 1954) per rubric×direction, with the family threshold calibrated on 6060 held-out H0H_0 anchor streams to FWER 0.10.1 — the strongest classical recipe. It holds its false-alarm budget (0.080.08/0.030.03) but is blind to the subtle real lenient bump (8%8\% on HelpSteer2, 40%40\% on TL;DR), and where it does detect it is slower than the e-process (295295 versus 120120 on the synthetic shift; 248248 versus 205205 and 165165 versus 140140 on TL;DR). The anchor e-process dominates on both datasets: false-alarm 0.03≤αjudge0.03≤ _judge with no calibration data and a horizon-free guarantee, 97%97\% detection on all three HelpSteer2 drifts and 8383–100%100\% on TL;DR (the 83%83\% on the lenient bump is the anchor-budget frontier of Table˜2, not a method limitation), and the best latency at every drift among false-alarm-controlled methods. Anytime-validity is not a theoretical nicety: the default practice is unusable for continuous monitoring, and the properly-calibrated classical alternative pays with both blindness to subtle real drift and a calibration-data tax. Figure 4: The anchor e-process versus classical change detectors on the identical anchor stream, on HelpSteer2 (left) and TL;DR (right). naive-z (rolling z-test, the industry default) has no sequential error control and false-alarms on 75%75\%/67%67\% of no-drift streams; calibrated Page–Hinkley holds α but misses the subtle real lenient bump; the anytime-valid e-process controls false alarms at αjudge _judge with no calibration data and detects all drifts up to its anchor budget. Table 5: Anchor-process baselines on the identical anchor stream (60 seeds), both datasets. FA = false-alarm rate on no-drift; detection rate / median latency on each of the three judge drifts. naive-z additionally fires 16/6016/60 (HelpSteer2) and 10/6010/60 (TL;DR) alarms before the change point. detection rate median latency method FA lenient harsh synth lenient harsh synth HelpSteer2 eprocess 0.03 0.97 0.97 0.97 498 370 120 naive-z 0.75 0.73 0.73 0.73 130 138 40 ph-calib 0.08 0.08 0.65 1.00 555 530 295 TL;DR eprocess 0.03 0.83 1.00 1.00 555 205 140 naive-z 0.67 0.83 0.83 0.83 168 72 50 ph-calib 0.03 0.40 1.00 1.00 550 248 165 5.6 Generalization: every cell of both datasets Every topic–rubric cell. Running the featured config across all 2828 runnable HelpSteer2 topic×rubric cells (two near-ceiling coherence cells are skipped — their regressed-blind pools are empty), judge detection is 1.001.00 in 27/2827/28 cells (0.970.97 in factual_qa:coherence) at latency 9898–128128 items, judge→ is 0.000.00 in 27/2827/28 (0.030.03 in that same cell), and the false-judge rate is at most 0.100.10. On TL;DR’s 19/2019/20 runnable subreddit×axis cells the picture is the same: judge detection 0.850.85–0.970.97 at latency 118118–145145 in every cell, judge→ ≤0.15≤ 0.15 at W=50W=50, and false-judge ≤0.10=αjudge≤ 0.10= _judge. The C2 properties are cell-independent by construction — the anchor process never looks at the main-process cell. Only C1 system power varies (range 0.000.00–1.001.00 across HelpSteer2 cells, 0.400.40–1.001.00 on TL;DR, weakest where μ is most generous; missed system regressions fail safe to none, never a false judge). The full per-cell table is in Appendix˜C. Cross-dataset reads. The exhibit-by-exhibit replication tests the theory in two distinct ways: the invariances must transfer, and the differences must be the ones the propositions predict. Both happen. On the invariance side, the same real version bump shows the same +0.03+0.03 to +0.07+0.07 per-rubric signature in both domains (Table˜3) — a property of the bump, not the dataset — turning the lenient case from a case study into a property of the version bump itself; and each dataset has one low-resolution rubric (verbosity on HelpSteer2, coherence on TL;DR) with identical phenomenology — weak μ correlation, near-ceiling humans, weakest judge–human agreement — a property of rubric design, not of the monitor. On the prediction side, the domains differ in exactly one relevant input — the strict-prompt shift is larger on summaries — and Proposition˜3 says a larger gap shift makes the anchors stochastically faster, so attribution should improve everywhere that race appears. It does, three times over, with nothing re-tuned: anchor latency drops (205205 versus 370370), the guard sweep closes to zero spill at W=300W=300 (Table˜4), and the co-provisioning band moves toward slower anchor rates (Figure˜3). These are out-of-sample structural predictions coming true, not robustness checks. The one quantity that is not predicted to transfer, C1 system power, indeed varies for its own reason: it is higher on TL;DR (56/6056/60 versus 41/6041/60, Table˜1) because all four axes are informative for μ, unlike HelpSteer2’s complexity and verbosity. 6 Discussion and Limitations The construction buys a sharp guarantee — judge drift is never silently charged to the system — at the price of several scoped assumptions, which we state plainly. Anchor staleness. Identification (Proposition˜2) assumes the frozen human labels h remain a valid reference. If the anchor items’ true quality standard drifts conceptually — the meaning of a rubric shifts, say — the anchor process still correctly reports judge-versus-anchor disagreement, but its interpretation as judge drift weakens, since the disagreement could now be the world moving under a fixed yardstick. A periodic anchor refresh, re-labeling against the current standard, re-establishes the baseline. Pooled anchor family. The anchor e-process aggregates over rubric×direction only; it carries no topic dimension. This is a deliberate power/budget choice: judge drift is plausibly global, and a per-topic anchor family would split the k anchors across cells and multiply the Bonferroni factor by K. The cost is reduced power against topic-localized judge drift — a judge that changed only on, say, code-heavy items. Topic stratification lives entirely on the system-process side. both is reported as judge, by design. When both families fire within the guard window the verdict is judge (Equation˜6): a contaminated y invalidates the main alarm, so we do not attempt simultaneous attribution of a co-occurring system regression. The honest reading is that a confirmed judge change suspends trust in the main verdict until the judge is re-anchored. No optimality. We claim validity and identification, not efficiency. The plug-in bet λ approximates the log-optimal bet rather than realizing it, and the attribution rule is not claimed to minimize misattribution at a fixed anchor budget. The race monotonicities of Proposition˜3 say which direction each knob moves the spill, not that the knobs are set optimally. System power is cell-dependent, and the horizon is single. The disentanglement properties are cell-independent by construction, but the main monitor’s system power is not: it is near zero on the cheap-judge-blind rubrics (complexity, verbosity) and on the math cells, where the regressed-blind pool is empty or the cheap judge is uninformative. A missed system regression fails safe to none — never to a false judge. The experiments use a single monitoring horizon (N=1200N=1200, change point at 300300) and a representative token-cost model (input/output token counts at measured prices), which ignores topic-assignment overhead; absolute cost fractions should be read as representative, not universal. The honest cost menu. There is no single price. The system traces a menu against strong-evaluating every item (1.01.0): a cheap-but-deaf configuration (escalation π0 _0 with sparse rate-2020 anchors) monitors at ≈0.21≈ 0.21 but misses most system regressions at the hard cell; the featured powered-and-guarded configuration (fixed@0.30.3 main with rate-55 anchors) costs ≈0.64≈ 0.64 and catches system drift with zero misattribution. Notably, the anchor budget is the smaller part of the increment from cheap to powered: of the 0.430.43 cost-fraction gap, the anchor share is only 0.200.20 — most of the added cost buys main-process power, not anchor coverage. A practical recipe. These limitations compose into a deployment order, which is the payoff for a practitioner. First, pick the main configuration on the cost/power frontier of Section˜3 — this fixes the system power you can afford and, with it, the contaminated-fire latency the anchors must beat. Second, size the anchor budget (k(k, interleave rate)) so that anchor detection latency, at the drift magnitudes you actually care about, comes in under the main’s contaminated-fire latency — the co-provisioning band of Section˜5.4, not a bolted-on afterthought. Third, widen the guard W to absorb the residual spill: it is the cheapest knob, buying attribution correctness with latency rather than dollars, and (Proposition˜3) it can never manufacture a false judge verdict. 7 Conclusion Every team that monitors an LLM product with an LLM judge eventually hits the same question: when the dashboard turns red, is it the product or the judge? We have shown that a small, fixed set of human-labeled anchors, re-scored by the current judge and watched by one extra anytime-valid e-process, answers it — separating system drift from judge drift into an explicit three-state verdict with three transparent knobs (anchor size k, interleave rate 1/ρ1/ρ, and guard width W) and a guarantee that judge drift is never charged to the system beyond αjudge _judge. The construction is validated not only on synthetic drift but on two real judge changes — a silent version bump and a strict-prompt policy update — across two domains, with nothing re-tuned between them; where the domains differ, the differences are themselves predicted by the attribution race (a larger judge shift yields faster anchors, perfect attribution, and a co-provisioning band shifted toward slower anchor rates). Three directions remain open: principled anchor-refresh policies that keep h a valid reference as standards evolve; per-stratum anchor families that recover power against topic-localized judge drift; and optimal betting and attribution rules that minimize misattribution at a fixed anchor budget. References Angelopoulos et al. [2023] Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference, 2023. URL https://arxiv.org/abs/2301.09633. Angelopoulos et al. [2024] Anastasios N. Angelopoulos, John C. Duchi, and Tijana Zrnic. PPI++: Efficient prediction-powered inference, 2024. URL https://arxiv.org/abs/2311.01453. Chaudhuri et al. [2024] Anamitra Chaudhuri, Georgios Fellouris, and Ali Tajer. Round robin active sequential change detection for dependent multi-channel data, 2024. URL https://arxiv.org/abs/2403.16297. Chen et al. [2026] Yiqun T. Chen, Moran Guo, and Shengy Li. Power analysis for prediction-powered inference, 2026. URL https://arxiv.org/abs/2603.16041. Choe and Ramdas [2023] Yo Joong Choe and Aaditya Ramdas. Comparing sequential forecasters, 2023. URL https://arxiv.org/abs/2110.00115. Cortinovis and Caron [2025] Stefano Cortinovis and François Caron. FAB-PPI: Frequentist, assisted by Bayes, prediction-powered inference, 2025. URL https://arxiv.org/abs/2502.02363. Csillag et al. [2025a] Daniel Csillag, Pedro Dall’Antonia, Claudio José Struchiner, and Guilherme Tegoni Goedert. Extending prediction-powered inference through conformal prediction, 2025a. URL https://arxiv.org/abs/2510.16166. Csillag et al. [2025b] Daniel Csillag, Claudio José Struchiner, and Guilherme Tegoni Goedert. Prediction-powered E-Values, 2025b. URL https://arxiv.org/abs/2502.04294. Feng et al. [2026] Chen Feng, Minghe Shen, Ananth Balashankar, Carsten Gerner-Beuerle, and Miguel R. D. Rodrigues. Noisy but valid: Robust statistical evaluation of LLMs with imperfect judges, 2026. URL https://arxiv.org/abs/2601.20913. Fiedler [2026] James Fiedler. Bias and uncertainty in LLM-as-a-Judge estimation, 2026. URL https://arxiv.org/abs/2605.06939. Google DeepMind [2025] Google DeepMind. Gemini model family. https://deepmind.google/technologies/gemini/, 2025. Howard et al. [2022] Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences, 2022. URL https://arxiv.org/abs/1810.08240. Howard et al. [2025] Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform Chernoff bounds via nonnegative supermartingales, 2025. URL https://arxiv.org/abs/1808.03204. Kilian et al. [2025] Valentin Kilian, Stefano Cortinovis, and François Caron. Anytime-valid, Bayes-assisted, prediction-powered inference, 2025. URL https://arxiv.org/abs/2505.18000. Lamaakal et al. [2026] Ismail Lamaakal, Chaymae Yahyati, Khalid El Makkaoui, Ibrahim Ouahbi, and Yassine Maleh. Drift-to-Action controllers: Budgeted interventions with online risk certificates, 2026. URL https://arxiv.org/abs/2603.08578. Lee et al. [2026] Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, and Kangwook Lee. How to correctly report LLM-as-a-Judge evaluations, 2026. URL https://arxiv.org/abs/2511.21140. Page [1954] E. S. Page. Continuous inspection schemes. Biometrika, 41(1/2):100–115, 1954. Sfyraki and Wang [2026] Maria-Eleni Sfyraki and Jun-Kun Wang. Revisiting active sequential prediction-powered mean estimation, 2026. URL https://arxiv.org/abs/2604.18569. Shin et al. [2023] Jaehyeok Shin, Aaditya Ramdas, and Alessandro Rinaldo. E-detectors: a nonparametric framework for sequential change detection, 2023. URL https://arxiv.org/abs/2203.03532. Shiryaev [1963] Albert N. Shiryaev. On optimum methods in quickest detection problems. Theory of Probability & Its Applications, 8(1):22–46, 1963. Stiennon et al. [2022] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325. Ville [1939] Jean Ville. Étude critique de la notion de collectif. Gauthier-Villars, Paris, 1939. Wang et al. [2024] Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models, 2024. URL https://arxiv.org/abs/2406.08673. Waudby-Smith and Ramdas [2022] Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting, 2022. URL https://arxiv.org/abs/2010.09686. Yi and Yang [2025] Yanqing Yi and Su-Fen Yang. Change detection with adaptive sampling for binary responses, 2025. URL https://arxiv.org/abs/2512.15507. Zhang et al. [2026] Guangyi Zhang, Yunlong Cai, Guanding Yu, and Osvaldo Simeone. Prediction-powered risk monitoring of deployed models for detecting harmful distribution shifts, 2026. URL https://arxiv.org/abs/2602.02229. Zheng et al. [2023] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, 2023. URL https://arxiv.org/abs/2306.05685. Zrnic and Candès [2024] Tijana Zrnic and Emmanuel J. Candès. Cross-prediction-powered inference, 2024. URL https://arxiv.org/abs/2309.16598. Zrnic and Candès [2026] Tijana Zrnic and Emmanuel J. Candès. Active statistical inference, 2026. URL https://arxiv.org/abs/2403.03208. Appendix A Proofs We prove Propositions 1–4 as stated in Section˜4, in the same notation. Throughout, “past” at position i denotes the σ-algebra ℱi−1F_i-1 generated by the anchor observations g~1,…,g~i−1 g_1,…, g_i-1 (and, for the main process, the stream and acquisition history up to i−1i-1). All validity claims are inherited from standard test-(super)martingale results; the contribution is the construction they apply to, not new probability theory. Proof of Proposition˜1 (anytime-validity). Fix a rubric–direction pair (r,d)(r,d) and write ei=1+λisd(q~0[r]−g~i[r])e_i=1+ _i\,s_d\,( q_0[r]- g_i[r]) with sbelow=+1s_below=+1, sabove=−1s_above=-1 (Equation˜5). Nonnegativity. By Equation˜4, g~i[r]∈[0,1] g_i[r]∈[0,1] and q~0[r]∈(0,1) q_0[r]∈(0,1), so the signed gap sd(q~0[r]−g~i[r])s_d\,( q_0[r]- g_i[r]) lies in [−1,1][-1,1]. The bet is constrained to λi∈[0,λmax(q~0[r],d)] _i∈[0, _ ( q_0[r],d)], where the directional cap λmax(q~0[r],d) _ ( q_0[r],d) is exactly the largest λ≥0λ≥ 0 for which 1+λ⋅sd(q~0[r]−g)1+λ· s_d\,( q_0[r]-g) stays nonnegative for every attainable g∈[0,1]g∈[0,1] (for d=belowd=below the binding case is g=1g=1, giving λmax=1/(1−q~0[r]) _ =1/(1- q_0[r]); for d=aboved=above, g=0g=0, giving λmax=1/q~0[r] _ =1/ q_0[r]). Hence ei≥0e_i≥ 0 for all i. Conditional mean. The bet λi _i is predictable — a function of g~1,…,g~i−1 g_1,…, g_i-1 only (in code the plug-in λ≈mean(d)/mean(d2)λ (d)/mean(d^2) computed from past gaps). Under H0ancH_0^anc we have [g~i[r]∣ℱi−1]=q~0[r]E[ g_i[r] _i-1]= q_0[r], so [ei∣ℱi−1]=1+λisd(q~0[r]−[g~i[r]∣ℱi−1])=1+λisd⋅0=1,E[e_i _i-1]=1+ _i\,s_d\, ( q_0[r]-E[ g_i[r] _i-1] )=1+ _i\,s_d· 0=1, using predictability of λi _i to pull it outside the conditional expectation. Supermartingale. Therefore [n(r,d)∣ℱn−1]=n−1(r,d)⋅[en∣ℱn−1]=n−1(r,d)E[W_n^(r,d) _n-1]=W_n-1^(r,d)·E[e_n _n-1]=W_n-1^(r,d), with 0(r,d)=1W_0^(r,d)=1 (empty product). Each (n(r,d))n(W_n^(r,d))_n is a nonnegative martingale, hence a nonnegative supermartingale. Crossing bound and union. Ville’s inequality [Ville, 1939] for a nonnegative supermartingale with unit initial value gives, for any level β>0β>0, ℙ(∃n:n(r,d)≥β)≤1/βP(∃ n:W_n^(r,d)≥β)≤ 1/β. Taking β=2R/αjudgeβ=2R/ _judge bounds each pair’s ever-crossing probability by αjudge/2R _judge/2R. A union bound over the 2R2R rubric–direction pairs gives ℙ(∃n:maxr,dn(r,d)≥2R/αjudge)≤ 2R⋅αjudge2R=αjudge.P (∃ n: _r,dW_n^(r,d)≥ 2R/ _judge )\;≤\;2R· _judge2R= _judge. Ville’s inequality is time-uniform, so the bound holds at any data-dependent stopping time. ∎ Remark 5. The bound needs no distributional assumptions beyond boundedness: the gaps need not be Gaussian, i.i.d. across rubrics, or stationary in variance. It also needs no held-out calibration streams and no fixed horizon, in contrast to the classical change detectors of Section˜5.5, which set a threshold against H0H_0 streams and are calibrated only at that horizon. Remark 6 (Exactness of the baseline). Anchors are sampled with replacement from the fixed set A, so under H0ancH_0^anc the conditional-mean condition [g~i[r]∣ℱi−1]=q~0[r]E[ g_i[r] _i-1]= q_0[r] holds by construction of q~0 q_0 up to the finite-sample error of q~0 q_0 itself. Because q~0[r] q_0[r] is defined as the exact mean rescaled gap of the time-0 judge over all of A, and the time-0 judge scores are cached, this error is zero by construction: the unchanged judge re-scoring a uniformly drawn anchor has gap mean exactly q~0[r] q_0[r]. Proof of Proposition˜2 (identification). Each anchor observation at any position is a deterministic function of (a) the fixed items A, (b) their frozen labels h, and (c) the current judge, through Equation˜4. System drift — any change in the distribution of new stream items or in their true quality — alters none of (a)–(c), so it leaves the anchor gap distribution unchanged. Hence H0ancH_0^anc can be violated only by a change in the judge. (a) Under pure system drift the judge is unchanged, so H0ancH_0^anc holds. A judge verdict (Equation˜6) requires τanc<∞ _anc<∞, i.e. the anchor family fires; Proposition˜1 bounds the probability of that event by αjudge _judge at any horizon, and this bound does not involve W. Therefore ℙ(verdict=judge∣pure system drift)≤αjudgeP(verdict= judge system drift)≤ _judge, uniformly in W. (b) Under no drift at all, both nulls hold. The event verdict≠none\verdict≠ none\ requires at least one family to fire, i.e. τsys<∞∪τanc<∞\ _sys<∞\∪\ _anc<∞\. By the main-process guarantee (Section˜3, inherited from Csillag et al. [2025b, Thm. 2.1] with predictable π and Ville’s inequality) ℙ(τsys<∞)≤αsysP( _sys<∞)≤ _sys, and by Proposition˜1 ℙ(τanc<∞)≤αjudgeP( _anc<∞)≤ _judge. A union bound gives ℙ(verdict≠none)≤αsys+αjudgeP(verdict≠ none)≤ _sys+ _judge; the two budgets are separate by design. ∎ Proof of Proposition˜3 (the attribution race). Suppose judge drift has occurred (with or without simultaneous system drift). By the verdict rule Equation˜6, verdict=systemverdict= system holds exactly when τsys<∞ _sys<∞ and (τanc=∞ _anc=∞ or τanc>τsys+W _anc> _sys+W), which is precisely the event τsys+W<τanc\ _sys+W< _anc\ (adopting the convention τsys+W<∞=τanc _sys+W<∞= _anc when the anchors never fire). Hence ℙ(verdict=system∣judge drift)=ℙ(τsys+W<τanc).P(verdict= system drift)=P( _sys+W< _anc). Monotonicity in W. For fixed sample paths of the two stopping times, the indicator τsys+W<τanc1\ _sys+W< _anc\ is non-increasing in W pointwise; taking expectations preserves the order, so the misattribution probability is monotonically non-increasing in W. Monotonicity in the anchor budget. Increasing the interleave rate 1/ρ1/ρ or the anchor set size k supplies more anchor evidence per position, which stochastically decreases τanc _anc (first-order stochastic dominance: more accumulated wealth per step reaches the threshold 2R/αjudge2R/ _judge no later). A stochastically smaller τanc _anc makes the event τsys+W<τanc\ _sys+W< _anc\ less likely, so the misattribution probability is non-increasing in 1/ρ1/ρ and in k. Monotonicity in main power. Increasing the main process’s sampling power (more strong calls on the contaminated y) stochastically decreases τsys _sys, by the same first-order argument applied to the main e-process: more strong labels per step grow the contaminated wealth no slower toward its threshold. A stochastically smaller τsys _sys makes the misattribution event more likely, so the probability is non-decreasing in main power. These are stochastic-dominance arguments at the level of rigor of the formal-statements source; we do not claim a sharper coupling. ∎ Remark 7. The race involves no validity trade-off. Raising W delays a system attribution but cannot create false judge verdicts beyond the αjudge _judge of Proposition˜2(a), because the anchor bound of Proposition˜1 holds at any horizon — this time-uniformity is exactly what makes the guard window free. Detection is also untouched: the total event drift flagged at all=τsys∧τanc<∞\drift flagged at all\=\ _sys _anc<∞\ does not depend on W; the rule only decides the label. Empirically (Section˜5.3) the spill shrinks monotonically in W (29→18→1029→ 18→ 10 of 120120 at W=50/150/300W=50/150/300) and in the anchor rate, and grows with main power (0.03→0.330.03→ 0.33 at rate-55 across the main ladder). Proof of Proposition˜4 (orthogonality). The anchor observations g~i g_i are a deterministic function of (,h,current judge)(A,h,current judge) and the fixed interleave schedule (Equation˜4), none of which depend on the main acquisition rule (policy, budget, or π). Therefore the law of the anchor process — and in particular τanc _anc — does not depend on the main-process configuration. Symmetrically, the anchor re-judgements are out-of-band calls on held-out items that, by the anchor-set definition (Section˜4), are excluded from all main-process calibration and never enter the main e-values; the anchor process never reads stream items. Hence the main process’s distribution does not depend on the anchor interleave. Given the judge state, the two stopping times are functions of disjoint randomness, and the design splits cleanly: the anchor budget (k,ρ)(k,ρ) and guard W govern judge-drift latency and the race; the main configuration governs system power; and αjudge,αsys _judge, _sys are separate Bonferroni budgets. ∎ Remark 8. Orthogonality holds for the two processes; the verdict still couples them through the race of Proposition˜3 (the comparison τanc _anc vs τsys+W _sys+W reads both stopping times). Result 5 of Section˜5.4 verifies the process-level orthogonality empirically: judge latency (830/695/430830/695/430) is constant across all five main configs, and each main’s system power is constant across anchor rates. What is not claimed. Two boundaries, transcribed from the formal-statements source. No optimality: the plug-in λ approximates the log-optimal bet, and the attribution rule is not claimed to minimize misattribution at a fixed budget. Anchor staleness: Proposition˜2 assumes the human labels h remain a valid reference; if the anchor items’ true quality standard drifts conceptually (e.g. the rubric meaning changes), the anchor process correctly reports judge-vs-anchor disagreement, but its reading as judge drift weakens until a periodic anchor refresh re-establishes the baseline. Finally, the anchor family is pooled across strata (rubric×direction only), so a judge change confined to one stratum is detected at reduced power; topic stratification lives entirely on the system-process side. Appendix B The Cost-Aware Foundation in Detail This appendix gives the full cost-frontier table, the ten-method prior-work comparison, and the trigger ablations summarized in Section˜3. All numbers are for the semi-synthetic localized blind-spot drift in the helpfulness×creative cell of HelpSteer2 (stream N=1200N=1200, change at 300300), averaged over 6060 Monte-Carlo seeds, at αsys=0.1 _sys=0.1. Cost model recap. Each judge call costs (input tokens)⋅(input price)+(output tokens)⋅(output price)(input tokens)·(input price)+(output tokens)·(output price) at representative counts of ≈600≈ 600 input and ≈30≈ 30 output tokens, giving $0.000195 0.000195 per cheap call and $0.00156 0.00156 per strong call. The cheap judge runs on all L processed items and the strong judge on S sampled items, so relative to strong-evaluating every item the cost-fraction=(Lcμ+Scy)/(Lcy)=0.125+S/Lcost-fraction=(Lc_μ+Sc_y)/(Lc_y)=0.125+S/L, with full evaluation (π≡1π≡ 1) equal to 1.01.0 by definition. The 0.1250.125 is the irreducible cheap floor and S/LS/L is the realized strong-sampling rate. B.1 The cost–reliability frontier Table˜6 reports the per-stratum monitor across both acquisition modes. Latency is the median steps from the change point over uncensored reps; censor is the fraction of reps that never alarmed within the horizon. Table 6: Cost–reliability frontier for the per-stratum monitor (helpfulness×creative, ylow=0.25y_low=0.25, 60 seeds). Latency = median steps to detection over uncensored reps; censor = fraction never alarming within the horizon (N=1200N=1200, change at 300300). approach latency censor cost-fraction false-alarm in-control cost escalate π0=0.01 _0=0.01 — 1.00 — 0.000 0.142 escalate π0=0.02 _0=0.02 750 0.95 0.21 0.000 0.154 escalate π0=0.05 _0=0.05 710 0.82 0.23 0.000 0.184 escalate π0=0.1 _0=0.1 706 0.53 0.28 0.000 0.236 fixed uniform @0.1 789 0.98 0.24 0.000 0.226 fixed @0.2 (uni/strat/bandit) ∼660 660 0.50 0.33 0.000 0.327 fixed @0.3 (uni/strat/bandit) ∼585 585 0.28 0.43 0.000 0.428 full strong-eval (π=1π=1) 327 0.02 1.00 0.000 1.125 Every detector catches the drift at cost-fraction 0.210.21–0.430.43, about one third of full strong evaluation, with false-alarm 0.0000.000 across all 1616 policy points (empirically anytime-valid). The two modes trace a frontier: escalation is cheapest (0.210.21–0.280.28), sitting at the 0.125+π00.125+ _0 floor in-control — near-zero strong calls when nothing is wrong; fixed budgets buy reliability (@0.2 → censor 0.500.50, @0.3 → censor 0.280.28); full evaluation is fastest (latency 327327) at full price. Escalation π0=0.1 _0=0.1 (cost 0.280.28, censor 0.530.53) is about as reliable as fixed @0.2 (cost 0.330.33, censor 0.500.50) but cheaper, missing the strict censor-≤0.5≤ 0.5 cut by one rep. This drift is intrinsically hard (even full-eval censors 2%2\%), which compresses escalation’s margin; a stronger or cleaner drift, or a higher π0 _0, lets escalation dominate the frontier outright. B.2 The ten-method prior-work comparison We place the four prior-work families in one harness on one frontier, calibrated to the same false-alarm level. Table˜7 lists what each method sees and acquires; “sees μ” means it runs the cheap judge on every item, and the cost-fraction charges each method only for what it calls. Table 7: The ten-method roster. “isolates” names the design axis each method isolates. group # method sees μ acquires y cost-frac isolates A. cheap-only A1 same e-process on μ ✓ never 0.125 cheap carries no drift info A2 Page–Hinkley on μ ✓ never 0.125 concept-drift family A3 Shiryaev–Roberts on μ ✓ never 0.125 LR/e-process detector B. label-only B1 e-process on y, no PPI — fixed τ τ value of the cheap proxy C. global PPI C1 per-rubric PPI, no strata ✓ fixed τ 0.125+τ0.125+τ value of stratification D. per-stratum PPI D1 fixed-budget τ ✓ fixed τ 0.125+τ0.125+τ — (fixed-budget foil) D2 covariate: π↑π as μ→q0μ→q_0 ✓ covariate μ 0.125+S/L0.125+S/L covariate vs e-wealth D3 Csillag App B.2: π∝π pred. e-growth ✓ covariate μ 0.125+S/L0.125+S/L the published active-PPI rule D4 escalation (ours): π↑π with e-wealth ✓ past e-evidence 0.125+S/L0.125+S/L — (our trigger) E. reference E1 full strong-eval π=1π=1 — every item 1.0 speed ceiling The prior-work mapping: A2/A3 are the concept-drift family (Page–Hinkley [Page, 1954]/Shiryaev–Roberts [Shiryaev, 1963]); B1 is label-only SAVI; C1 is the global (un-stratified) PPI e-process of Csillag et al. [2025b]; D2 is Zrnic–Candès uncertainty-triggered active inference [Zrnic and Candès, 2026]; D3 is the Csillag App B.2 active-PPI rule; D4 is ours; E1 is the naive ground-truth-everything monitor. Every e-process method is false-alarm-valid by construction (Bonferroni), and all four per-stratum PPI rules (D1–D4) share the same per-item bet cap λmax(πi) _ ( _i) on their realized (predictable) acquisition probability, so no method gets a betting head-start. The acquisition-rule headline. Table˜8 ranks the four per-stratum PPI rules, which differ only in the acquisition rule, against drift strength. “Detects” means censor ≤0.5≤ 0.5 and false-alarm ≤α≤α. Because the strong label y lives on the HelpSteer2 5-level grid 0,0.25,0.5,0.75,1.0\0,0.25,0.5,0.75,1.0\, the four-point drift sweep (ylow∈0.15,0.25,0.35,0.45y_low∈\0.15,0.25,0.35,0.45\) collapses to two effective magnitudes in this cell: ylow=0.15y_low=0.15 draws a pool of one extreme item (severe), while ylow∈0.25,0.35,0.45y_low∈\0.25,0.35,0.45\ draw the identical 21-item pool (mild, byte-identical in the raw JSON). We report the comparison as severe vs mild. Table 8: Acquisition-rule ranking vs drift strength (per-stratum PPI; D1–D4 differ only in the trigger, sharing the per-item bet cap). Severe = ylow=0.15y_low=0.15; mild = ylow≥0.25y_low≥ 0.25. severe (ylow=0.15y_low=0.15) mild (ylow≥0.25y_low≥ 0.25) D1 fixed-budget best cheap detector: @0.2 censor 0.12 (cost 0.33), @0.3 censor 0.05 (cost 0.43) @0.2 censor 0.50 (cost 0.33), @0.3 censor 0.27 (cost 0.43) D2 covariate-boundary detects fast but by oversampling: censor 0.00 at every π0 _0, latency ∼185 185, cost ∼0.87 0.87 detects, still by oversampling: censor 0.00 at every π0 _0, latency ∼430 430, cost ∼0.88 0.88 D3 covariate-egrowth weak: π0=0.1 _0=0.1 censor 0.80 (cost 0.24), else censor 1.00 fails (censor ≥0.95≥ 0.95 at every π0 _0) D4 escalation (ours) strong and cheap: π0=0.1 _0=0.1 censor 0.22 (cost 0.28) ties fixed-budget: π0=0.1 _0=0.1 censor 0.53 (cost 0.28) ≈ D1@0.2 censor 0.50 (cost 0.33) The covariate-triggered prior-art rules do not fail to detect — they fail to detect efficiently, each for a distinct reason. D2 (boundary) detects on both drifts but only by oversampling to cost ≈0.87≈ 0.87–0.890.89 (near full-eval): its “label near the decision boundary” heuristic fires near every rubric’s bar across the 3030 cells, so π is pushed high regardless of where the drift is. D3 (the published Csillag App B.2 rule) is inert: its predicted-e-growth trigger reads identically zero because the cheap judge overrates every healthy bar (q0−μ<0q_0-μ<0 on all five rubrics: helpfulness −0.18-0.18, complexity −0.23-0.23, verbosity −0.27-0.27), so it degenerates to fixed-budget at rate π0 _0 and fails on the mild drift. D4 (ours), keying on accumulated e-evidence rather than the blind cheap covariate, detects at cost 0.280.28 — roughly one third of D2’s — by spending only once evidence accrues; on the faint mild drift it merely ties a matched fixed budget (D4@0.1 censor 0.53 cost 0.28 ≈ D1@0.2 censor 0.50 cost 0.33), exactly the regime of Sfyraki and Wang [2026] where an adaptive trigger buys essentially nothing over fixed-τ for prediction-powered mean estimation. We report the tie as-is. The central result, restated: keying acquisition on the blind cheap covariate is either wasteful (D2 oversamples to ≈0.88≈ 0.88) or inert (D3 reads zero under the overrating bias); keying on accumulated e-evidence (D4) detects at one third of that cost. Two further contrasts. Stratification vs global: per-(rubric×topic) stratification is decisive. At every matched budget, global PPI (C1) censors strictly worse than per-stratum fixed-budget (D1) — severe @0.2 global 0.680.68 vs stratified 0.120.12, @0.3 0.580.58 vs 0.050.05; on the mild drift global PPI censors 1.001.00 at all three budgets while stratified detects at @0.2 (0.500.50) and @0.3 (0.270.27). A localized blind-spot drift is diluted to invisibility when pooled. Classic detectors do not transfer: the two concept-drift detectors are threshold-calibrated to family-wise false-alarm ≈α≈α on a held-out, disjoint H0H_0 seed block, yet off that sample held-out Page–Hinkley records false-alarm 0.1170.117 (just over α) and held-out Shiryaev–Roberts 0.2670.267 (well over α) — the realized maximum false-alarm across all methods and settings, 0.2670.267, is driven entirely by the classic SR detector, while the e-process spine holds at false-alarm ≤0.017≤ 0.017 (the lone non-zero e-process value is covariate-boundary π0=0.1 _0=0.1 at 0.0170.017, one H0H_0 alarm in 6060, within α). The cheap proxy alone never sees this drift: all three cheap-only methods censor near 1.01.0 everywhere (A1 ablation 1.001.00, A3 SR 1.001.00, A2 Page–Hinkley 0.980.98 severe / 0.920.92 mild), by construction of the blind-spot drift. Appendix C All-Cells Results Table˜9 gives the full per-(topic×rubric) cell table summarized in Section˜5.6, for the featured configuration (fixed@0.3 main, k=200k=200 anchors at rate 1/51/5, guard W=50W=50, synthetic shift −0.25-0.25, 2020 seeds) on HelpSteer2. The two near-ceiling coherence cells coding:coherence and math:coherence are skipped: their regressed-blind pool (y≤0.5∧μ≥0.6y≤ 0.5 μ≥ 0.6) is empty because coherence sits at ∼0.92 0.92. Columns: judge detection rate, median judge-drift latency (items), judge→ spill rate, false-judge rate, and system detection rate. The C2 properties are cell-independent by construction — the anchor process never looks at the main-process cell. Judge detection is 1.001.00 in 27/2827/28 cells (and 0.970.97 in factual_qa:coherence), at latency 9898–128128 items everywhere; judge→ is 0.000.00 in 27/2827/28 (0.030.03 in that same cell); and the false-judge rate is at most 0.100.10, under αjudge _judge. Only the system detection rate varies, a pure C1 sensitivity property: correctness and coherence cells detect at 0.800.80–1.001.00, helpfulness in the middle (0.450.45–0.950.95), and complexity/verbosity ≈0≈ 0 everywhere (the rubrics where μ is blind and the regressed pools are thin), with all math cells ≈0≈ 0. Every missed system regression fails safe to none, never a false judge. Table 9: Every topic×rubric cell at the featured config (HelpSteer2, fixed@0.3 main, k=200k=200/rate-55 anchors, W=50W=50, synthetic −0.25-0.25, 20 seeds). 28 runnable cells in two column blocks; coding:coherence and math:coherence are skipped (empty regressed-blind pool). jdet = judge detection, jlat = median judge latency (items), j→ = judge→ spill, fj = false-judge, sdet = system detection. cell jdet jlat j→ fj sdet cell jdet jlat j→ fj sdet coding:complexity 1.00 122 0.00 0.00 0.00 math:complexity 1.00 120 0.00 0.05 0.00 coding:correctness 1.00 122 0.00 0.00 0.80 math:correctness 1.00 122 0.00 0.10 0.00 coding:helpfulness 1.00 125 0.00 0.00 0.60 math:helpfulness 1.00 120 0.00 0.05 0.00 coding:verbosity 1.00 125 0.00 0.00 0.05 math:verbosity 1.00 120 0.00 0.05 0.00 creative:coherence 1.00 110 0.00 0.00 1.00 other:coherence 1.00 112 0.00 0.00 1.00 creative:complexity 1.00 118 0.00 0.05 0.00 other:complexity 1.00 125 0.00 0.05 0.00 creative:correctness 1.00 115 0.00 0.00 1.00 other:correctness 1.00 118 0.00 0.00 1.00 creative:helpfulness 1.00 115 0.00 0.00 0.65 other:helpfulness 1.00 115 0.00 0.00 0.60 creative:verbosity 1.00 110 0.00 0.00 0.00 other:verbosity 1.00 128 0.00 0.00 0.05 factual_qa:coherence 0.97 98 0.03 0.05 0.95 reasoning:coherence 1.00 118 0.00 0.03 1.00 factual_qa:complexity 1.00 122 0.00 0.00 0.00 reasoning:complexity 1.00 115 0.00 0.00 0.00 factual_qa:correctness 1.00 115 0.00 0.00 0.95 reasoning:correctness 1.00 120 0.00 0.05 0.85 factual_qa:helpfulness 1.00 108 0.00 0.05 0.95 reasoning:helpfulness 1.00 120 0.00 0.05 0.45 factual_qa:verbosity 1.00 118 0.00 0.05 0.25 reasoning:verbosity 1.00 112 0.00 0.05 0.10 The TL;DR replication of this sweep (Section˜5.6) runs 19/2019/20 cells (AskReddit:coherence skipped, near-ceiling): judge detection 0.850.85–0.970.97 at latency 118118–145145 in every cell, judge→ ≤0.15≤ 0.15 at W=50W=50, false-judge ≤0.10=αjudge≤ 0.10= _judge, and system detection 0.400.40–1.001.00 (weakest on coverage cells, the rubric where μ is most generous). Appendix D Cheap-Judge Calibration The cheap judge μ (gemini-3.1-flash-lite) is run on every item as free side information for the prediction-powered main process. Table˜10 reports how much signal it carries per rubric, from the fitted-calibration study: a per-rubric μ→yμ→ y correction (shift / affine / isotonic) is fit on the first 600600 HelpSteer2 items and measured on the held-out next 600600. Table 10: Cheap-judge calibration against the strong judge, held-out (n=600n=600 test). Bias =μ−y=μ-y; corr is Pearson; affine slope a is the fitted μ→yμ→ y gain; RMSE↓ is the held-out RMSE reduction of the affine correction vs raw μ. rubric μ−yμ-y bias corr(μ,y)(μ,y) affine slope a RMSE↓ vs raw μ read helpfulness +0.13+0.13 0.81 0.87 −19%-19\% informative, just generous correctness +0.01+0.01 0.81 0.80 −6%-6\% already accurate — don’t correct coherence −0.02-0.02 0.63 0.60 −16%-16\% tracks complexity +0.16+0.16 0.54 0.40 −54%-54\% biased but correctable verbosity +0.22+0.22 0.23 0.28 −49%-49\% blind: μ≈μ≈ uninformative The corrections transfer cleanly: held-out residual bias ≈0≈ 0 on every rubric, and calibration RMSE ≈ test RMSE (no overfit). The affine fit ≈ isotonic fit (since μ lives on a 5-level grid), and a 1-parameter shift already removes most of the error; the blind-spot structure is stable across the split (|Δbias|≤0.01| |≤ 0.01, |Δblind%|≤5| \%|≤ 5 points). The takeaway: calibration removes the cheap judge’s systematic generosity but cannot create resolution it never had. The affine slope a measures how much μ is worth: a≈1a≈ 1 (helpfulness) means keep its signal and just shift it; a≈0a≈ 0 (verbosity) means μ is essentially the base rate, so the strong judge is required there. Helpfulness and complexity become good calibrated proxies; verbosity does not — which is exactly the rubric where the localized blind-spot drift of Appendix˜B hides, and where the large judge-version gap of Table˜3 appears. Appendix E Judge Prompts The strong and cheap judges share a scoring prompt; only the model snapshot differs. Drift is injected on the strong judge in two ways: a real version bump (re-judging with a smaller model) needs no prompt change, while the real harsh change re-judges with the same model under a deliberately stricter scoring prompt. We quote both prompt versions verbatim from the registry (driftjudge/data/judges.py); the rubric list and prompt/response are interpolated at the marked fields. Baseline scoring prompt (v1). You are a strict evaluator. Score the assistant RESPONSE to the PROMPT on each rubric using an integer 0-4 (HelpSteer2 scale: 0 worst, 4 best). Reply with ONLY a JSON object mapping each rubric name to its integer score. Rubrics: rubrics PROMPT: prompt RESPONSE: response Harsh scoring prompt (v2-strict). A deliberately harsher scoring policy on the same scale and model, emulating a real silent scoring-policy update (C2’s contamination direction). The money clause is the strict tie-break: You are an exacting, skeptical evaluator. Score the assistant RESPONSE to the PROMPT on each rubric using an integer 0-4 (HelpSteer2 scale: 0 worst, 4 best). Be strict: reserve 4 for flawless work, penalize every error, omission, or unsupported claim you notice, and when torn between two adjacent scores always choose the lower. Reply with ONLY a JSON object mapping each rubric name to its integer score. Rubrics: rubrics PROMPT: prompt RESPONSE: response The TL;DR summarization domain uses parallel variants v1-summ and v2-strict-summ (same instructions, with POST/SUMMARY framing and the four summarization axes substituted for the rubric meanings). An unknown prompt_version raises. The harsh re-judge has 33 HelpSteer2 items (and 11 TL;DR item) that the strong endpoint cannot score (content filter); these fall back to the undrifted y, so the contaminated stream is 1495/14981495/1498 (HelpSteer2) and 1497/14981497/1498 (TL;DR). Appendix F Reproducibility The pipeline has two phases. The data layer is a one-time paid Gemini cache (the only network/API phase): every item is scored once by the cheap judge, once by the strong judge, and once per drift slice, and the human labels are joined in. All experiments afterward are compute-only: they read the cached scores from artifacts/ and re-run the monitor in simulation, so reproducing every number below requires no API access once the cache exists. The cached artifacts are not committed to the source tree (artifacts/ is gitignored); re-collecting scores from a later judge version would yield different numbers — which is precisely the paper’s point. For exact public reproduction, the intended release vehicle is a separate versioned artifact archive, not committed source files. Figures regenerate from the checked-in computed outputs via make figures (scripts/paper_figures.py). Datasets and licensing. HelpSteer2 [Wang et al., 2024] is released under C-BY-4.0 (© NVIDIA); the OpenAI summarize-from-feedback feedback data is released under OpenAI’s modified MIT license; and the underlying Webis TLDR corpus is listed under C-BY-4.0. The reproducibility artifact redistributes only derived judge scores and human labels/metadata, with attribution, not raw source text. Gemini scores are Gemini-API outputs. Data layer (one-time, paid). HF_HUB_OFFLINE=1 uv run --extra pipeline python scripts/add_human_labels.py GEMINI_API_KEY=... uv run --extra pipeline python scripts/judge_drift_data.py \ --drift-snapshot gemini-3.5-flash --limit 1498 GEMINI_API_KEY=... HF_HUB_OFFLINE=1 uv run --extra pipeline python \ scripts/judge_drift_data.py \ --drift-snapshot gemini-3.1-pro-preview --prompt-version v2-strict --limit 1498 HelpSteer2 experiments (compute-only). # Result 1b -- high-power main (fixed @0.3), matched vs old anchor budget: uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 200 --anchor-rate 5 \ --main-mode fixed --main-budget 0.3 uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 50 --anchor-rate 20 \ --main-mode fixed --main-budget 0.3 # Result 3 -- real lenient bump, featured fixed@0.3 main: uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 200 --anchor-rate 5 \ --main-mode fixed --main-budget 0.3 --real-drift-snapshot gemini-3.5-flash # Result 4 -- real harsh drift, guard sweep (data study first): uv run python scripts/study_judge_version_gap.py \ --snapshot gemini-3.1-pro-preview --prompt-version v2-strict for W in 50 150 300; do \ uv run python scripts/c2_anchor_experiment.py --seeds 60 --k 200 --anchor-rate 5 \ --guard $W --main-mode fixed --main-budget 0.3 \ --real-drift-snapshot gemini-3.1-pro-preview --drift-prompt-version v2-strict; done # Result 5 -- co-provisioning frontier (real harsh drift): uv run python scripts/c2_anchor_experiment.py --seeds 30 --k 200 --guard 50 --frontier \ --real-drift-snapshot gemini-3.1-pro-preview --drift-prompt-version v2-strict # Result 6 -- every cell at the featured config -> artifacts/c2_cells.json: uv run python scripts/c2_anchor_experiment.py --seeds 20 --k 200 --anchor-rate 5 \ --main-mode fixed --main-budget 0.3 --all-cells # Result 7 -- baselines on the identical anchor stream: uv run python scripts/c2_baseline_comparison.py --seeds 60 TL;DR second dataset. The TL;DR data layer is a separate one-time paid run (raw files fetched per driftjudge/data/tldr.py); every experiment reuses the same drivers with --dataset tldr (results land in *_tldr.json artifacts, so the HelpSteer2 result JSONs are untouched). The TL;DR halves of every table and figure mirror the HelpSteer2 commands above: # data layer (paid, one-time, ~6k calls): GEMINI_API_KEY=... uv run --extra pipeline python scripts/tldr_data.py --max-workers 256 GEMINI_API_KEY=... uv run --extra pipeline python scripts/tldr_data.py --max-workers 256 \ --drift-snapshot gemini-3.5-flash --prompt-version v1-summ GEMINI_API_KEY=... uv run --extra pipeline python scripts/tldr_data.py --max-workers 256 \ --drift-snapshot gemini-3.1-pro-preview --prompt-version v2-strict-summ # experiments (compute-only): E="uv run python scripts/c2_anchor_experiment.py --dataset tldr" # confusion (matched + under-provisioned): $E --seeds 60 --k 200 --anchor-rate 5 --main-mode fixed --main-budget 0.3 $E --seeds 60 --k 50 --anchor-rate 20 --main-mode fixed --main-budget 0.3 # lenient bump: anchor-budget sweep (escalation main) + featured fixed@0.3: $E --seeds 60 --k 50 --anchor-rate 20 --real-drift-snapshot gemini-3.5-flash $E --seeds 60 --k 150 --anchor-rate 8 --real-drift-snapshot gemini-3.5-flash $E --seeds 60 --k 200 --anchor-rate 5 --real-drift-snapshot gemini-3.5-flash $E --seeds 60 --k 200 --anchor-rate 5 --main-mode fixed --main-budget 0.3 \ --real-drift-snapshot gemini-3.5-flash # harsh drift: per-axis study + guard sweep: uv run python scripts/study_judge_version_gap.py --dataset tldr uv run python scripts/study_judge_version_gap.py --dataset tldr \ --snapshot gemini-3.1-pro-preview --prompt-version v2-strict for W in 50 150 300; do \ $E --seeds 60 --k 200 --anchor-rate 5 --guard $W --main-mode fixed \ --main-budget 0.3 --real-drift-snapshot gemini-3.1-pro-preview \ --drift-prompt-version v2-strict; done # co-provisioning frontier, every cell, and baselines: $E --seeds 30 --k 200 --guard 50 --frontier \ --real-drift-snapshot gemini-3.1-pro-preview --drift-prompt-version v2-strict $E --seeds 20 --k 200 --anchor-rate 5 --main-mode fixed --main-budget 0.3 --all-cells uv run python scripts/c2_baseline_comparison.py --dataset tldr --seeds 60 All experiment outputs land in gitignored JSON under artifacts/ (e.g. c2_confusion.json, c2_frontier.json, c2_cells.json, c2_baselines.json); the C1 cost frontier is reproduced by uv run python scripts/c1_stratified_experiment.py --seeds 60 and the ten-method comparison by uv run python scripts/c1_method_comparison.py --seeds 60.