Paper deep dive
Design-Based Supervised Learning with Noisy Human Labels
Robert Chew, Matthew R. Williams
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 2:37:09 AM
Summary
The paper introduces Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method to correct biases in automated labels when human audit labels are noisy and only partially adjudicated by experts. PA-DSL uses a nested correction approach: first, it creates pseudo-labels for the audit set using adjudicated cases to correct human coder noise; second, it uses these corrected pseudo-labels to debias the full population of automated labels. The method maintains nominal coverage and reduces RMSE compared to using only adjudicated labels.
Entities (6)
Relation Signals (5)
PA-DSL → corrects → Noisy Human Labels
confidence 95% · It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels.
PA-DSL → extends → DSL
confidence 90% · This paper introduces Partially Adjudicated Design-Based Supervised Learning (PA-DSL)... To our knowledge, this is the first work to extend rectification methods beyond the usual gold-audit setting
PA-DSL → improves → RMSE
confidence 90% · PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels
Automated Classifiers → produce → Noisy Labels
confidence 90% · Automated classifiers now make it possible to code data at scale... But classifier-generated labels are noisy measurements
Wikipedia Detox → usedin → PA-DSL
confidence 90% · In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels when noisy human labels contain recoverable signal.
Tags
Links
- Source: https://arxiv.org/abs/2607.15455v1
- Canonical: https://arxiv.org/abs/2607.15455v1
Trouble viewing inline? Open PDF directly →
Full Text
87,461 characters extracted from source content.
Expand or collapse full text
Design-Based Supervised Learning with Noisy Human Labels Robert Chew RTI International Matthew R. Williams U.S. Bureau of Labor Statistics Abstract Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10–17% relative to using only adjudicated labels when noisy human labels contain recoverable signal. 1 Introduction Empirical research in the social sciences, business, law, and related fields often depends on coding raw texts, images, or records into variables that measure substantive constructs (Krippendorff, 2018; Grimmer and Stewart, 2013). Traditionally, this coding has relied on human or expert judgment, making it costly, time-consuming, and prone to inconsistency. Automated classifiers now make it possible to code data at scale, ranging from traditional supervised text classifiers to newer LLM-based systems (Gilardi et al., 2023; Chew et al., 2023; Törnberg, 2025). But classifier-generated labels are noisy measurements, not observed outcomes (Wang et al., 2020). Systematic prediction errors can bias downstream estimates and produce undercovered confidence intervals, even when classifiers achieve high overall accuracy (Neuhaus, 1999). This has motivated a growing set of rectification methods that use higher-quality validation or audit data to correct estimates based on full-population surrogate labels (Hopkins and King, 2010; Wang et al., 2020; Angelopoulos et al., 2023; Egami et al., 2023; Fong and Tyler, 2021). The common structure is simple: inexpensive automated labels provide scale, while a smaller set of trusted labels anchors the correction. Design-Based Supervised Learning (DSL), for example, combines a full-population surrogate score with a probability-sampled audit to obtain valid inference for downstream estimands such as regression coefficients (Egami et al., 2023). In many real labeling pipelines, however, the audit labels used for correction are not themselves gold standard. Human annotators disagree, and disagreement often reflects ambiguity, subjectivity, or guideline interpretation rather than only random coder error (Aroyo and Welty, 2015; Uma et al., 2021; Chew et al., 2026). Because expert reconciliation is costly (Snow et al., 2008; Chau et al., 2020), a common workflow is to have one or more human annotators label all items from an “audited” sample, with only selected items from it receiving expert adjudication. This multi-reviewer or adjudication design is widespread, reappearing across domains diverse as content analysis, systematic reviews, clinical endpoint assessment, and NLP annotation (Lombard et al., 2002; Page et al., 2021; Held, 2019; Finlayson and Erjavec, 2017). Existing rectification strategies do not fully address this setting because the audit used to correct the surrogate may itself be measured with error. This paper introduces Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a design-based estimator for downstream inference under hierarchical label noise. Conceptually, PA-DSL is not a new estimating principle from scratch, but a nested application of known-probability AIPW/DSL corrections to a partially adjudicated annotation design. It uses an inner correction to turn the adjudicated subset into audit-level pseudo-labels, and an outer correction to use those pseudo-labels to debias inference based on the full-population surrogate. To our knowledge, this is the first work to extend rectification methods beyond the usual gold-audit setting, allowing the human audit labels used for correction to be noisy and only partially adjudicated. Contributions. The paper makes three contributions. First, it formalizes a three-tier nested measurement design with noisy audit labels and partial adjudication. Second, it proves that the resulting pseudo-outcome is design-valid for any downstream estimating equation affine in the latent outcome, provided the audit and adjudication probabilities are known by design. Third, it shows in synthetic and semi-synthetic experiments that PA-DSL retains the coverage of adjudication-only DSL while improving efficiency when the noisy audit labels contain recoverable information about the adjudicated construct. 2 Setup: Hierarchical Label Noise and Target Estimand Observed data. We observe an i.i.d. sample of size N, where each unit i contributes Wi=(i,Qi,Ri,Gi,1,Gi,2,Vi,Ai),W_i\;=\; ( X_i,\,Q_i,\,R_i,\,G_i,1,\,G_i,2,\,V_i,\,A_i ), (1) generated jointly with a latent binary truth Yi⋆∈0,1Y _i∈\0,1\ that the analyst does not observe. Here i∈ℝp+1 X_i ^p+1 is a covariate vector with leading intercept; Qi∈[0,1]Q_i∈[0,1] is a real-valued surrogate score; Ri∈0,1R_i∈\0,1\ is the audit indicator with known sampling probability πi∈(0,1) _i∈(0,1); Gi,1,Gi,2∈0,1G_i,1,G_i,2∈\0,1\ are two human coder labels observed iff Ri=1R_i=1; Vi∈0,1V_i∈\0,1\ is the adjudication indicator with known probability ρi≡ℙ(Vi=1∣Ri=1,Zi)∈(0,1) _i (V_i=1 R_i=1,Z_i)∈(0,1) a known function of the inner features defined just below; and Ai∈0,1A_i∈\0,1\ is the adjudicated label, observed iff RiVi=1R_iV_i=1. The score QiQ_i need not be calibrated; it is treated as an observed surrogate feature whose systematic errors are corrected by the audit design. We write Bi=(Qi,i⊤)⊤B_i=(Q_i, X_i ) for the outer feature vector and Zi=(Qi,i⊤,Gi,1,Gi,2)⊤Z_i=(Q_i, X_i ,G_i,1,G_i,2) for the inner feature vector. We use two coder labels as the running case because this is a common annotation workflow and makes disagreement-based adjudication easy to describe. However, the construction does not require exactly two labels: more generally, ZiZ_i may contain any audit-tier information observed when Ri=1R_i=1, including KiK_i coder labels, coder identities, agreement indicators, vote shares, uncertainty scores, item metadata, or other features used in the adjudication design. The probabilities πi _i and ρi _i are design parameters chosen by the researcher, so they are known, not estimated. Figure˜1 summarizes the nested three-tier sampling design. Throughout, the tiers refer to nested measurement availability, not mutually exclusive samples. Tier 1 contains all N units and includes (Xi,Qi)(X_i,Q_i). Tier 2 is the audit subset i:Ri=1\i:R_i=1\, which also includes noisy coder labels (Gi1,Gi2)(G_i1,G_i2). Tier 3 is the adjudicated subset i:RiVi=1\i:R_iV_i=1\, which also includes the adjudicated label AiA_i. Hence tier 3 ⊂ tier 2 ⊂ tier 1. Step 1Three-tier SampleQQXXG1G_1G2G_2AAStep 2Inner Pass (Eq. 3)QQXXG1G_1G2G_2AAM MStep 3Outer Pass: DSL on M M (Eq. 4)QQXXM MY~ Yβ β Figure 1: PA-DSL pipeline. Step 1 shows the three nested measurement tiers. Every unit has covariates and an automated score (X,Q)(X,Q) (Tier 1, gray); a randomized audit sample with known probability πi _i adds two noisy human labels (G1,G2)(G_1,G_2) (Tier 2, blue); and a randomized subset of audited units with known probability ρi _i receives an adjudicated label A (Tier 3, amber). Bar heights encode measurement tier, and dashed outlines mark labels that are unobserved. Step 2 is the new inner correction, which uses the adjudicated cases to turn the noisy audit information into an audit-tier pseudo-label M M (purple). Step 3 applies standard DSL to M M, producing the final pseudo-outcome Y~ Y (coral) used to estimate β β. Target estimand. In the applications motivating this paper, the scientific target is often not whether each individual text or record is classified correctly, but how a target construct varies across groups, treatments, or covariates. For subjective annotation tasks, we interpret Yi⋆Y _i as the label that would be assigned to unit i under the target adjudication protocol, rather than as an externally verified objective truth. We therefore target a population-level regression summary for this adjudication-defined construct: the logistic projection of Yi⋆Y _i onto the covariates. Formally, we focus on the parameter β∈ℝp+1β ^p+1 defined as the unique solution of the population score equation [i(Yi⋆−expit(i⊤β))]= 0,E [ X_i\, (Y _i-expit( X_i β) ) ]\;=\;0, (2) reporting the slope β1 _1 on the first non-intercept covariate. This is the logistic-regression M-estimand for Y⋆Y given X. Validity for an objective construct beyond the protocol-defined target requires the additional measurement-validity assumption that the adjudication protocol itself captures that construct. The working model serves only as a specification choice for the score equation; it has a literal conditional-probability interpretation if and only if it holds exactly, which we do not assume. Equation˜2 is outcome-affine: the score is linear in Yi⋆Y _i for any fixed β. The construction in Section 3 produces an unbiased pseudo-outcome Y~i Y_i for Yi⋆Y _i, so any outcome-affine estimating equation (the population mean, generalized linear regression, inverse-propensity weighted treatment effects, etc.) can be solved by substituting Y~i Y_i for Yi⋆Y _i. We treat the population prevalence [Y⋆]E[Y ] as the special case in which the score is Yi⋆−ψY _i-ψ; see Appendix˜A for that variant and its empirical results. Importantly, we do not assume that QiQ_i is calibrated, unbiased, or a consistent estimator of Yi⋆Y_i . The surrogate score is treated as an observed feature available on the full frame. Its systematic errors are corrected by the audit/adjudication design, as in standard DSL. Identification comes from the known audit and adjudication probabilities together with the ignorability/correctness assumptions below, not from any correctness property of QiQ_i. The quality of QiQ_i affects efficiency through the nuisance functions, but not design validity. Identifying assumptions. The audit/adjudication design supplies four assumptions; together they identify the population means we will need. Assumption 1 (Audit ignorability). Ri⟂Yi⋆∣(i,Qi)R_i \!\!\! Y _i ( X_i,Q_i), with πi _i bounded away from 0 and 11. Holds by construction whenever audit assignment is a randomized function of observed design variables. Assumption 2 (Adjudication ignorability). Vi⟂Yi⋆∣(Ri=1,Zi)V_i \!\!\! Y _i (R_i=1,Z_i), with ρi _i bounded away from 0 and 11. Holds whenever adjudication is randomized with known probability as a known function of ZiZ_i — for example, high-probability adjudication on coder disagreement and lower but nonzero adjudication on agreement cases. Strictly deterministic prioritization rules require an additional positive sampling floor in every design stratum used for inference (Appendix˜H sweeps this floor). Assumption 3 (Protocol-level Adjudication correctness). On adjudicated units, Ai=Yi⋆A_i=Y_i , where Yi⋆Y_i denotes the label that would be assigned under the target adjudication protocol. Assumption 4 (Coder labels are observed noisy features). The coder labels (Gi,1,Gi,2)(G_i,1,G_i,2) and any other audit-tier metadata included in ZiZ_i are observed on the audit set and may be arbitrary noisy functions of the latent truth, item difficulty, coder identity, task context, or other factors. We do not require coder independence, unbiasedness, a fixed number of coders, or a parametric coder-error model. Assumption 4 is a measurement condition rather than an identifying model: All audit-tier information, including (Gi,1,Gi,2)(G_i,1,G_i,2), enter only as observed features in ZiZ_i. Their bias, dependence, and heterogeneity are absorbed into μ0(Zi) _0(Z_i); design validity comes from the known adjudication probability ρi _i and adjudication correctness. 3 Estimator: Partially Adjudicated DSL Standard DSL (Egami et al., 2023) replaces the unobserved Yi⋆Y _i with a pseudo-outcome of the form g^(Bi)+(Ri/πi)(Yigold−g^(Bi)) g(B_i)+(R_i/ _i) (Y_i^gold- g(B_i) ), where g g is a cross-fit nuisance regression of the gold label on BiB_i and the Horvitz–Thompson correction debiases that regression at the known audit weight. The construction assumes Yigold=Yi⋆Y_i^gold=Y _i on the audit set; when the audit labels are themselves noisy, substituting any one of Gi,1G_i,1, Gi,2G_i,2, or their majority vote inherits the human bias. We address this by nesting a second design-based correction inside, exploiting the fact that the adjudicated label AiA_i is unbiased for Yi⋆Y _i on RiVi=1\R_iV_i=1\ with known sampling weight ρi _i. Step 1 — Inner correction. Given any cross-fit estimator μ^(⋅) μ(·) of the inner nuisance μ0(z)=[Ai∣Zi=z,Ri=1,Vi=1] _0(z)=E[A_i Z_i=z,R_i=1,V_i=1] trained on the adjudicated subset RiVi=1\R_iV_i=1\, define the audit-level pseudo-label M^i=μ^(Zi)+Viρi(Ai−μ^(Zi)),i∈Ri=1. M_i\;=\; μ(Z_i)\;+\; V_i _i\, (A_i- μ(Z_i) ), i∈\R_i=1\. (3) Here μ^(Zi) μ(Z_i) uses the surrogate, the covariates, and the two coder labels to predict adjudicated truth, and the Vi/ρiV_i/ _i term debiases that prediction on the adjudicated subset. This definition is unchanged if ZiZ_i contains more than two coder labels or additional audit-tier metadata; the inner correction only requires that the adjudication probability ρi=P(Vi=1∣Ri=1,Zi) _i=P(V_i=1 R_i=1,Z_i) be known by design and bounded away from zero. Step 2 — Outer correction. Given a cross-fit estimator g^(⋅) g(·) of the outer nuisance g0(b)=μ0(Zi)∣Bi=b,Ri=1g_0(b)=E\ _0(Z_i) B_i=b,R_i=1\ trained on the audit set Ri=1\R_i=1\ using M^i M_i as response, define the final pseudo-outcome Y~i=g^(Bi)+Riπi(M^i−g^(Bi)),i=1,…,N. Y_i\;=\; g(B_i)\;+\; R_i _i\, ( M_i- g(B_i) ), i=1,…,N. (4) Under Assumptions 1–3, this target satisfies g0(Bi)=(Yi⋆∣Bi)g_0(B_i)=E(Y_i B_i); the realized pseudo-label M^i M_i is used only to estimate this population nuisance. Substituting Eq.˜3 into Eq.˜4 gives the closed form below, which makes the three roles transparent Y~i=g^(Bi)⏟full-frame regression+Riπi[μ^(Zi)−g^(Bi)]⏟audit correction+RiViπiρi[Ai−μ^(Zi)]⏟adjudication correction \; Y_i\;=\; g(B_i)_full-frame regression\;+\; R_i _i\, [ μ(Z_i)- g(B_i) ]_audit correction\;+\; R_iV_i _i _i\, [A_i- μ(Z_i) ]_adjudication correction\; (5) Thus, PA-DSL uses adjudication only to correct the part of the audit information that cannot be trusted, while still allowing the unadjudicated human labels to improve efficiency through μ^(Zi) μ(Z_i). Step 3 — Downstream estimating equation. Substitute Y~i Y_i for Yi⋆Y _i in the population score Eq.˜2 and solve the empirical analog: 1N∑i=1Ni(Y~i−expit(i⊤β^))= 0, 1N _i=1^N X_i\, ( Y_i-expit( X_i β) )\;=\;0, (6) returning the cross-fit estimator β β. Standard errors come from the sandwich formula evaluated at (i,Y~i)( X_i, Y_i). Why Y~i Y_i is design-valid. Both corrections in Eqs.˜3 and 4 are AIPW-style augmentations (Robins et al., 1994) with known design weights. Under Assumptions 1 to 3, iterated expectations give [M^i∣Ri=1,Zi]=[Yi⋆∣Ri=1,Zi]E[ M_i R_i=1,Z_i]=E[Y _i R_i=1,Z_i] and [Y~i∣Bi]=[Yi⋆∣Bi]E[ Y_i B_i]=E[Y _i B_i], regardless of nuisance specification. Thus the nuisances affect efficiency, not design validity; full algebra and influence functions are in Appendix˜B. Algorithm 1 PA-DSL cross-fit pseudo-outcome. Input: data (i,Qi,Ri,Gi,1,Gi,2,Vi,Ai)i=1N\( X_i,Q_i,R_i,G_i,1,G_i,2,V_i,A_i)\_i=1^N; design weights πi,ρii=1N\ _i, _i\_i=1^N; folds K; nuisance learners for μ^,g μ, g. Output: pseudo-outcome Y~ii=1N\ Y_i\_i=1^N. 1. Partition 1,…,N\1,…,N\ into folds ℱ1,…,ℱKF_1,…,F_K. Let fold(i)fold(i) denote the fold containing unit i. 2. Inner pass. For k=1,…,Kk=1,…,K: (a) Fit μ^(−k) μ^(-k) on (Zi,Ai):RiVi=1,i∉ℱk\(Z_i,A_i):R_iV_i=1,\;i _k\. (b) For audited units i∈ℱki _k (Ri=1R_i=1), set M^i=μ^(−k)(Zi)+(Vi/ρi)(Ai−μ^(−k)(Zi)) M_i= μ^(-k)(Z_i)+(V_i/ _i) (A_i- μ^(-k)(Z_i) ). 3. Outer pass. For k=1,…,Kk=1,…,K: (a) Fit g^(−k) g^(-k) on (Bi,M^i):Ri=1,i∉ℱk\(B_i, M_i):R_i=1,\;i _k\, using the out-of-fold M^i M_i from step 2(b). (b) For all units i∈ℱki _k, set Y~i=g^(−k)(Bi)+(Ri/πi)(M^i−g^(−k)(Bi)) Y_i= g^(-k)(B_i)+(R_i/ _i) ( M_i- g^(-k)(B_i) ). 4. Return Y~ii=1N\ Y_i\_i=1^N. Cross-fitting. Algorithm 1 uses a shared fold partition for the inner and outer passes, which is the implementation used in the experiments. For the cleanest theoretical argument, Appendix B first analyzes a leakage-free nested variant in which the outer nuisance for a held-out fold is trained using audit pseudo-labels constructed without any information from that fold. The shared-partition version differs only through an indirect dependence of the outer nuisance on the held-out fold through the inner pseudo-labels used in outer training. Appendix B isolates this difference as a stability remainder and states the high-level condition under which the shared-partition implementation has the same first-order expansion. Write m(Wi;β,η)=i(Y~i(η)−expit(i⊤β))m(W_i;β,η)= X_i\,( Y_i(η)-expit( X_i β)) for the estimating function, with η=(μ,g)η=(μ,g) now an explicit argument, and let β0 _0 denote the population value of β defined by Eq.˜2. Proposition 1 (Design validity and asymptotic normality). Suppose Assumptions 1 to 4 hold, the design weights πi,ρi _i, _i are known and bounded away from zero and one, ‖i‖2+δ<∞E\| X_i\|^2+δ<∞ for some δ>0δ>0, and =[ii⊤expit′(i⊤β0)]H=E\! [ X_i X_i expit \! ( X_i _0 ) ] is nonsingular. For the leakage-free nested cross-fit estimator, under cross-fit L2L^2-consistent nuisance estimation, β β defined by Eq.˜6 satisfies N(β^−β0)=−11N∑i=1Nm(Wi;β0,η0)+op(1), N( β- _0)=H^-1 1 N _i=1^Nm(W_i; _0, _0)+o_p(1), and hence N(β^−β0)↝(,−1Ω−1⊤), N( β- _0) \! (0,\,H^-1 ^-1 ), where Ω=[m(Wi;β0,η0)m(Wi;β0,η0)⊤]=Varm(Wi;β0,η0),η0=(μ0,g0). =E\! [m(W_i; _0, _0)m(W_i; _0, _0) ]=Var\m(W_i; _0, _0)\, _0=( _0,g_0). The equality to the variance holds because m(Wi;β0,η0)=E\m(W_i; _0, _0)\=0. The covariance matrix −1Ω−1⊤H^-1 ^-1 is consistently estimated by the empirical cross-fit sandwich estimator. Moreover, for any fixed nuisance pair η=(μ,g)η=(μ,g), [m(Wi;β0,η)]=,E\! [m(W_i; _0,η) ]=0, so the score is design-unbiased at β0 _0 regardless of nuisance specification. The same expansion also holds for the shared-partition implementation in Algorithm˜1 under the high-level stability condition stated in Appendix˜B. Full assumptions and the influence-function derivation for the leakage-free nested estimator are given in Appendix˜B. Unlike doubly robust settings with estimated sampling weights, no product-rate condition is needed because the known design weights make the linear nuisance remainder conditionally mean-zero; Appendix B gives the full argument and the shared-partition stability condition. We instantiate Algorithm˜1 with K=5K=5 cross-fitting folds and use L2L^2-regularized logistic regression for both μ μ and g g. In the experiments, we balance folds on the twenty Q-quintile × BdiffB_diff-quartile cells as a finite-sample stabilization heuristic. This fold stratification is not part of the identifying design and is not required for design validity; ordinary random folds or other balanced fold constructions could also be used. Prediction clipping, the full estimator catalogue, and an ablation over the inner feature set appear in Appendices˜D and F. 4 Experimental Design Because Y⋆Y is latent in real annotation problems, empirical benchmarks must either assume a proxy truth or move to simulation. We therefore combine a synthetic Monte Carlo study, where Y⋆Y is known by construction and the measurement design can be varied, with a semi-synthetic study on the Wikipedia Detox corpus (Wulczyn et al., 2017), where real text and crowdworker labels replace the parametric annotation model but the target is necessarily a reference-panel proxy. The two studies respectively test design validity against a known target and robustness under real annotator disagreement. Synthetic data-generating process. Each replication generates N=50,000N=50,000 units. The design is intentionally richer than independent label-flip noise. Motivated by work on human label variation and learning from disagreement, we allow some items to be intrinsically ambiguous, some coders to be systematically more or less reliable, and machines and humans to have partially distinct error modes (Plank, 2022; Uma et al., 2021; Jiang and de Marneffe, 2022). Latent labels Yi⋆Y_i are drawn from a logistic model in p=5p=5 correlated covariates. Three latent difficulty channels then modulate human and surrogate accuracy: DiD_i, a shared item-difficulty channel affecting both humans and machines; HiH_i, a human-only nuisance channel; and MiM_i, a machine-only nuisance channel. The surrogate QiQ_i depends on Yi⋆Y_i , covariates, DiD_i, and MiM_i; the coder labels Gi1,Gi2G_i1,G_i2 depend on Yi⋆Y_i , coder-specific sensitivity/specificity, DiD_i, and HiH_i. Thus coder errors are heterogeneous and correlated through hard items, and surrogate errors are allowed to differ from human errors. This structure mirrors empirical annotation settings in which disagreement reflects a mixture of ambiguity, annotator effects, and task artifacts, while downstream regression remains vulnerable to response misclassification (Neuhaus, 1999; Wang et al., 2020). The audit indicator is uniform Bernoulli(π=0.10π=0.10), reflecting a modest manual-review budget. Adjudication is either uniform at mean rate ρ¯ ρ or disagreement-weighted to mean ρ¯ ρ. The latter targets cases where the two coders disagree, motivated by evidence that disagreement often reflects item ambiguity, task subjectivity, or guideline instability rather than only idiosyncratic annotator noise (Plank, 2022; Uma et al., 2021; Jiang and de Marneffe, 2022). We summarize the simulation design with three primary scenarios: benign (high-quality surrogate, low coder noise, uniform adjudication, ρ¯=0.25 ρ=0.25); realistic (medium-quality surrogate, moderate coder noise, disagreement-driven adjudication, ρ¯=0.25 ρ=0.25); and hard (weak surrogate, high coder heterogeneity, sparse adjudication, ρ¯=0.10 ρ=0.10). Full equations, default parameter values, and the scenario parameter table appear in Appendix˜E. Semi-synthetic study (Wikipedia Detox). The Wikipedia Detox corpus contains talk-page comments labeled by crowdworkers for whether the comment contains a personal attack (i.e., hostile or insulting language directed at another contributor). In our notation, Yi⋆Y _i denotes whether comment i would be judged to contain a personal attack under the target labeling standard. Because this status is unobserved in the Wikipedia data, the study is not a field validation against external ground truth. We instead evaluate against a researcher-constructed proxy truth estimated from a held-out reference panel of crowdworkers. Thus, the Detox study evaluates recovery of a reference-panel operationalization of the construct under real annotator disagreement, not validity against an external oracle for personal attack. In each replication, we re-draw the audit and adjudication samples under known design probabilities. The corpus comprises N≈1.16×105N≈ 1.16× 10^5 talk-page comments with multiple crowdworker attack ratings per comment. Real-world covariates retained for the analysis are the editor’s logged-in status, namespace indicator, comment year (standardized), and standardized log comment length; the surrogate QiQ_i is a TF–IDF logistic classifier fit out-of-fold on the comment text. We partition the crowdworkers into a reference panel (used for the proxy truth) and two audit panels (used as Gi,1,Gi,2G_i,1,G_i,2). The main-text proxy truth is a binary Dawid–Skene estimate from a held-out reference panel (Dawid and Skene, 1979); results under hard-majority and soft-mean-share proxies appear in Appendix˜G. Audit and adjudication indicators are re-drawn each replication from Ri∼Bernoulli(0.10)R_i (0.10) and disagreement-weighted ρi _i with ρ¯=0.25 ρ=0.25; we use R=200R=200 replications. Estimators. Table˜1 summarizes the main estimators. PA-DSL uses Eq.˜5 with inner features Zi=(Qi,i,Gi,1,Gi,2)Z_i=(Q_i, X_i,G_i,1,G_i,2) and L2-regularized logistic nuisances. Appendix Appendix˜F reports a variant that augments the inner feature set with coder-pair indicators. Table 1: Estimators compared in the main experiments. Group Estimator Description Plug-in Surrogate-only Plug in QiQ_i Plug-in Human-naive HT estimator using Gi,1G_i,1 with weight 1/π1/π Noisy-gold DSL DSL-MajVote Standard DSL using two-coder majority label Valid feasible DSL-AdjOnly DSL on adjudicated cases with weight 1/(πρi)1/(π _i) Proposed PA-DSL Nested audit/adjudication pseudo-outcome Infeasible DSL-Oracle DSL using Yi⋆Y _i on the audit set 5 Results We organize the empirical results around four claims, focusing on the logistic-regression slope β1 _1 throughout. The synthetic simulation analysis uses R=1,000R=1,000 replications per scenario with true β1=1 _1=1; the Detox study uses R=200R=200 replications with the Dawid–Skene proxy truth as the target. Prevalence and the inner-stage ablation are tabulated in Appendices˜A and F. Claim 1: noisy plug-ins fail. Figure˜2 shows that Surrogate-only, Human-naive, and DSL-MajVote are all substantially biased, with bias ranging from −0.41-0.41 in the benign scenario to −0.85-0.85 in the hard scenario and empirical 95%95\% coverage equal to 0.0000.000 in every cell. Standard DSL does not repair this failure when its “gold” audit label is itself noisy: the correction removes surrogate-side bias relative to the audit label, and therefore inherits bias in that label. Majority vote also fails because coder errors are correlated through shared item difficulty, so the bias common to both coders survives aggregation. Figure 2: Synthetic study, β1 _1 across three scenarios (R=1,000R=1,000 Monte Carlo replications, true β1=1 _1=1). Left: bias. Right: empirical 95%95\% coverage. The four plug-in baselines on the left of each panel are biased −0.41-0.41 to −0.85-0.85 with zero coverage; the three audit-corrected estimators on the right (DSL-AdjOnly, PA-DSL, DSL-oracle) recover near-zero bias and nominal coverage in every scenario. Claim 2: audit-corrected estimators recover validity. Figure˜2 also shows that DSL-AdjOnly, PA-DSL, and the infeasible DSL-oracle all sit within Monte Carlo error of zero bias and report empirical coverage in [0.94,0.96][0.94,0.96] across all three scenarios. Adjudication-only DSL is design-valid because it uses only the adjudicated subset RiVi=1\R_iV_i=1\ with the joint sampling weight 1/(πρi).1/(π _i). The price it pays is that the unadjudicated coder labels Gi,1,Gi,2G_i,1,G_i,2 on the rest of the audit set are simply discarded. Claim 3: PA-DSL improves efficiency where the inner layer has signal. Figure˜3 reports the efficiency ratios of PA-DSL versus DSL-AdjOnly (the strongest feasible baseline) and DSL-oracle (the infeasible benchmark), with paired-bootstrap 95%95\% CIs over the 1,0001,000 Monte Carlo replications. Relative to DSL-AdjOnly, the RMSE ratio is 1.211.21 (95%95\% CI [1.17,1.25][1.17,1.25]) in benign, 1.111.11 ([1.08,1.14][1.08,1.14]) in realistic, and 1.011.01 ([1.00,1.02][1.00,1.02]) in hard, with corresponding variance ratios 1.461.46 ([1.36,1.57][1.36,1.57]), 1.211.21 ([1.16,1.27][1.16,1.27]), and 1.021.02 ([1.00,1.04][1.00,1.04]). The benign and realistic gains are clear; the hard-regime ratio’s lower bound sits within 5×10−45× 10^-4 of 11, so PA-DSL reverts to the adjudication-only benchmark when the inner layer has no recoverable signal. In this scenario, the surrogate is weak, the coders are noisy, and the adjudication budget is sparse, so predicting AiA_i from (Qi,Xi,Gi,1,Gi,2)(Q_i,X_i,G_i,1,G_i,2) buys little leverage beyond the πiρi _i _i-weighted adjudicated subset. The reversion is the right behavior: PA-DSL does not manufacture signal absent from the inner features. Classification-oriented diagnostics in Appendix L show the same pattern at the prediction level: μ^(Zi) μ(Z_i) improves on g^(Bi) g(B_i) in the benign and realistic settings, but adds little signal in the hard setting. Figure 3: Relative efficiency of PA-DSL on β1 _1. Bars above one favor PA-DSL; whiskers are paired-bootstrap 95%95\% CIs over the 1,0001,000 Monte Carlo replications. Against DSL-AdjOnly (blue), PA-DSL achieves materially smaller variance and CI length in benign and realistic; in hard the inner stage extracts essentially no additional signal and the lower CI bound sits within 5×10−45× 10^-4 of 11. Against DSL-oracle (orange) the gap to the infeasible benchmark is roughly 1.6×1.6× in benign and widens to 6.3×6.3× in hard, the price paid for not observing Y⋆Y on the audit set. Claim 4: Detox replicates the qualitative pattern. Table˜2 shows the same comparison on the Wikipedia Detox corpus with the Dawid–Skene proxy truth (β1=−1.42 _1=-1.42 on logged_in). The plug-in baselines and the standard-DSL variants fed with noisy gold are biased by 0.330.33–0.390.39 with zero empirical coverage; DSL-AdjOnly and PA-DSL both achieve near-zero bias and coverage at or slightly above 95%95\%. PA-DSL’s RMSE ratio against DSL-AdjOnly is 1.181.18 (95%95\% paired-bootstrap CI [1.10,1.27][1.10,1.27]) and its CI-length ratio is 1.131.13 ([1.12,1.13][1.12,1.13]); both are clearly above 11, so the efficiency gain is statistically distinguishable from the no-gain point. The same qualitative ranking is reproduced under the hard-majority and soft-mean proxies in Appendix˜G. As above, this is evidence for the sampling-and-adjudication machinery under real text and real crowdworker disagreement, with respect to the constructed proxy truth rather than an observed latent construct. Table 2: Detox semi-synthetic study, β1 _1 on logged_in under the Dawid–Skene proxy truth (R=200R=200 replications, target β1=−1.42 _1=-1.42). Bold marks the column-best for MC SD, RMSE, and CI length among design-valid estimators (DSL-AdjOnly, PA-DSL); coverage is a calibration diagnostic and bias is reported but not bolded. The light rule separates the infeasible DSL-oracle benchmark. Method Bias MC SD RMSE Coverage CI len. Surrogate-only 0.387 -0.387 0.0000.000 0.3870.387 0.0000.000 0.0430.043 Human-naive 0.332 -0.332 0.0450.045 0.3350.335 0.0000.000 0.2080.208 DSL-MajVote 0.334 -0.334 0.0370.037 0.3360.336 0.0000.000 0.1790.179 DSL-AdjOnly 0.005 -0.005 0.0980.098 0.0980.098 0.9350.935 0.3680.368 PA-DSL 0.009 -0.009 0.082 0.082 0.083 0.083 0.9750.975 0.327 0.327 DSL-oracle 0.005 -0.005 0.0350.035 0.0360.036 0.9650.965 0.1440.144 6 Discussion PA-DSL extends design-based supervised learning to hierarchical annotation pipelines in which adjudication is treated as the gold-standard label but is available only for a subset of audited cases. The results suggest several practical lessons for designing and analyzing such pipelines. When the proposal helps and when it does not. PA-DSL delivers efficiency gains over adjudication-only DSL when the audit-tier features Zi=(Qi,Xi,Gi,1,Gi,2)Z_i=(Q_i,X_i,G_i,1,G_i,2) predict the adjudicated label AiA_i beyond the outer features Bi=(Qi,Xi)B_i=(Q_i,X_i). When this inner layer contains little additional signal, PA-DSL behaves like the adjudication-only benchmark, as in the hard synthetic regime. This pattern is formalized by the variance comparison in Appendix˜B, where the gain over adjudication-only DSL is proportional to the Z-explainable component μ0(Zi)−g0(Bi) _0(Z_i)-g_0(B_i). The inner ablations in Appendix˜I also show that the augmented-IPW form is not merely cosmetic: regression-only correction produced severe undercoverage in the realistic and hard regimes, underscoring the need to combine the outcome model with the known adjudication design. Design implications. The design sensitivity analysis in Appendix˜H suggests a practical operating range in the settings we studied: audit rates of roughly π∈[0.10,0.20]π∈[0.10,0.20] combined with disagreement-driven adjudication at ρ¯∈[0.10,0.25] ρ∈[0.10,0.25]. Disagreement-driven adjudication should include a positive adjudication floor even among agreement cases; otherwise, the design cannot identify error patterns in apparently easy items. More generally, the two-coder setup used in our experiments is not required by the framework: ZiZ_i may include any audit-tier information observed when Ri=1R_i=1, including KiK_i coder labels, coder identities, agreement indicators, vote shares, uncertainty scores, or other metadata. The same inner correction applies as long as the adjudication probability ρi=P(Vi=1∣Ri=1,Zi) _i=P(V_i=1 R_i=1,Z_i) is known and bounded away from zero. The realized audit and adjudication probabilities should also be recorded with the labels, since PA-DSL relies on known πi _i and ρi _i rather than estimating them after the fact. A pilot audit can help assess whether coder labels contain enough signal about the adjudicated construct to justify collecting additional human labels before increasing the adjudication budget. Scope and limitations. Although LLM-based annotation is one motivating use case, PA-DSL is a classifier-agnostic design-based correction: the surrogate QiQ_i may come from an LLM, a supervised classifier, a rules-based score, or any other automated labeling system observed on the full frame. However, PA-DSL is intended for prospective, known-probability audit/adjudication designs, or for settings in which the audit and adjudication probabilities can be reconstructed from the design record. It is not a generic post hoc correction for arbitrary deterministic review pipelines: if audited or adjudicated cases were selected using undocumented rules, or if some design strata have zero adjudication probability, the required positivity and ignorability conditions may fail. Validity also relies on Assumption˜3. We interpret this assumption as requiring adjudication to recover the target label under the adjudication protocol. Appendix˜K shows that PA-DSL inherits adjudicator bias relative to Yi⋆Y _i when Assumption˜3 fails. Thus, PA-DSL corrects the sampling and measurement structure only relative to the adjudicated construct; it cannot by itself guarantee that the adjudication protocol captures the substantive construct researchers intended. Estimators that remain valid under imperfect adjudication, treating AiA_i as a noisy proxy with a known design weight, are a natural next step. The Detox study should be interpreted in this same spirit. It is a semi-synthetic evaluation using real text and real crowdworker disagreement, but the audit/adjudication indicators are re-drawn with known probabilities and the target is a held-out reference-panel operationalization rather than an external field-validated oracle. Non-binary outcomes, scores nonlinear in Yi⋆Y _i, and dependent-class missingness lie outside the current scope. References A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic (2023) Prediction-powered inference. Science 382 (6671), p. 669–674. Cited by: Appendix C, §1. L. Aroyo and C. Welty (2015) Truth is a lie: crowd truth and the seven myths of human annotation. AI magazine 36 (1), p. 15–24. Cited by: §1. H. Chau, S. Balaneshin, K. Liu, and O. Linda (2020) Understanding the tradeoff between cost and quality of expert annotations for keyphrase extraction. In Proceedings of the 14th Linguistic Annotation Workshop, p. 74–86. Cited by: §1. R. Chew, J. Bollenbacher, M. Wenger, J. Speer, and A. Kim (2023) LLM-assisted content analysis: using large language models to support deductive coding. arXiv preprint arXiv:2306.14924. Cited by: §1. R. Chew, S. Eckman, C. Kern, and F. Kreuter (2026) From ground truth to measurement: a statistical framework for human labeling. arXiv preprint arXiv:2604.07591. Cited by: §1. A. P. Dawid and A. M. Skene (1979) Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) 28 (1), p. 20–28. Cited by: §4. N. A. de Pieuchon, A. Daoud, C. T. Jerzak, M. Johansson, and R. Johansson (2025) Benchmarking debiasing methods for llm-based parameter estimates. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 19768–19783. Cited by: Appendix C. N. Egami, M. Hinck, B. Stewart, and H. Wei (2023) Using imperfect surrogates for downstream inference: design-based supervised learning for social science applications of large language models. Advances in Neural Information Processing Systems 36, p. 68589–68601. Cited by: §1, §3. M. A. Finlayson and T. Erjavec (2017) Overview of annotation creation: processes and tools. In Handbook of linguistic annotation, p. 167–191. Cited by: §1. C. Fong and M. Tyler (2021) Machine learning predictions as regression covariates. Political Analysis 29 (4), p. 467–484. Cited by: §1. F. Gilardi, M. Alizadeh, and M. Kubli (2023) ChatGPT outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences 120 (30), p. e2305016120. Cited by: §1. J. Grimmer and B. M. Stewart (2013) Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Political analysis 21 (3), p. 267–297. Cited by: §1. C. Held (2019) When do we need clinical endpoint adjudication in clinical trials?. Upsala Journal of Medical Sciences 124 (1), p. 42–45. Cited by: §1. D. J. Hopkins and G. King (2010) A method of automated nonparametric content analysis for social science. American Journal of Political Science 54 (1), p. 229–247. Cited by: §1. N. Jiang and M. de Marneffe (2022) Investigating reasons for disagreement in natural language inference. Transactions of the Association for Computational Linguistics 10, p. 1357–1374. Cited by: §4. K. Krippendorff (2018) Content analysis: an introduction to its methodology. Sage publications. Cited by: §1. M. Lombard, J. Snyder-Duch, and C. C. Bracken (2002) Content analysis in mass communication: assessment and reporting of intercoder reliability. Human communication research 28 (4), p. 587–604. Cited by: §1. J. M. Neuhaus (1999) Bias and efficiency loss due to misclassified responses in binary regression. Biometrika 86 (4), p. 843–855. Cited by: §1, §4. M. J. Page, D. Moher, P. M. Bossuyt, I. Boutron, T. C. Hoffmann, C. D. Mulrow, L. Shamseer, J. M. Tetzlaff, E. A. Akl, S. E. Brennan, et al. (2021) PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. bmj 372. Cited by: §1. B. Plank (2022) The “problem” of human label variation: on ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Cited by: §4. J. M. Robins, A. Rotnitzky, and L. P. Zhao (1994) Estimation of regression coefficients when some regressors are not always observed. Journal of the American Statistical Association 89 (427), p. 846–866. Cited by: §3. R. Snow, B. O’connor, D. Jurafsky, and A. Y. Ng (2008) Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, p. 254–263. Cited by: §1. P. Törnberg (2025) Large language models outperform expert coders and supervised classifiers at annotating political social media messages. Social Science Computer Review 43 (6), p. 1181–1195. Cited by: §1. A. N. Uma, T. Fornaciari, D. Hovy, S. Paun, B. Plank, and M. Poesio (2021) Learning from disagreement: a survey. Journal of Artificial Intelligence Research 72, p. 1385–1470. Cited by: §1, §4. S. Wang, T. H. McCormick, and J. T. Leek (2020) Methods for correcting inference based on outcomes predicted by machine learning. Proceedings of the National Academy of Sciences 117 (48), p. 30266–30275. Cited by: §1, §1, §4. E. Wulczyn, N. Thain, and L. Dixon (2017) Ex machina: personal attacks seen at scale. In Proceedings of the 26th International Conference on World Wide Web, p. 1391–1399. Cited by: §4. Appendix A Prevalence as a Special Case The population prevalence ψ=[Y⋆]ψ=E[Y ] is the special case of the outcome-affine framework in which the score is Yi⋆−ψY _i-ψ. The PA-DSL estimator instantiates as ψ^=N−1∑iY~i ψ=N^-1 _i Y_i with Y~i Y_i from Eq.˜5, and the empirical-variance standard error SE^(ψ^)=(N(N−1))−1/2(∑i(Y~i−ψ^)2)1/2 SE( ψ)=(N(N-1))^-1/2 ( _i( Y_i- ψ)^2 )^1/2 is consistent for N−1/2N^-1/2 times the standard deviation of the influence function in Eq.˜7. Table˜3 reports the synthetic results for this target across the three scenarios; the qualitative ranking matches the β1 _1 results in the main text. Table 3: Synthetic study, prevalence ψ (R=1,000R=1,000 replications, realized truth Y¯⋆≈0.426 Y ≈ 0.426). Bold marks the column-best for MC SD, RMSE, and CI length per scenario among design-valid estimators (DSL-AdjOnly, PA-DSL); coverage is a calibration diagnostic and bias is reported but not bolded. The light rule separates the infeasible DSL-oracle benchmark. Scenario Method Bias MC SD RMSE Coverage CI len. benign Surrogate-only 0.0905 -0.0905 0.00120.0012 0.09050.0905 0.0000.000 0.00470.0047 Human-naive 0.0010 -0.0010 0.01100.0110 0.01110.0111 0.8980.898 0.03540.0354 DSL-MajVote 0.0008 -0.0008 0.00850.0085 0.00860.0086 0.8170.817 0.02320.0232 DSL-AdjOnly 0.0003 -0.0003 0.00780.0078 0.00780.0078 0.9450.945 0.03050.0305 PA-DSL 0.0003 -0.0003 0.0063 0.0063 0.0063 0.0063 0.9490.949 0.0247 0.0247 DSL-oracle −0.0000-0.0000 0.00420.0042 0.00420.0042 0.9480.948 0.01630.0163 realistic Surrogate-only −0.0605-0.0605 0.00120.0012 0.06050.0605 0.0000.000 0.00490.0049 Human-naive 0.0014 -0.0014 0.01730.0173 0.01740.0174 0.6920.692 0.03540.0354 DSL-MajVote 0.0011 -0.0011 0.01620.0162 0.01620.0162 0.5700.570 0.02660.0266 DSL-AdjOnly −0.0002-0.0002 0.01930.0193 0.01930.0193 0.9410.941 0.07500.0750 PA-DSL −0.0003-0.0003 0.0170 0.0170 0.0170 0.0170 0.9510.951 0.0669 0.0669 DSL-oracle −0.0000-0.0000 0.00560.0056 0.00560.0056 0.9490.949 0.02200.0220 hard Surrogate-only −0.1434-0.1434 0.00130.0013 0.14350.1435 0.0000.000 0.00510.0051 Human-naive 0.0074 -0.0074 0.02230.0223 0.02350.0235 0.5430.543 0.03570.0357 DSL-MajVote 0.0074 -0.0074 0.02190.0219 0.02310.0231 0.4340.434 0.02720.0272 DSL-AdjOnly −0.0004-0.0004 0.03410.0341 0.03410.0341 0.9380.938 0.12940.1294 PA-DSL −0.0003-0.0003 0.0339 0.0339 0.0338 0.0338 0.9380.938 0.1293 0.1293 DSL-oracle 0.0000 -0.0000 0.00600.0060 0.00600.0060 0.9480.948 0.02370.0237 Appendix B Identification, Influence Function, and Variance Notation and population nuisances. For notational simplicity, this appendix writes the audit and adjudication probabilities as π(Bi)π(B_i) and ρ(Zi)ρ(Z_i). If the actual audit probability depends on additional observed design variables, those variables are either included in BiB_i or added to the conditioning set below. Likewise, any observed variables used to set adjudication probabilities are included in ZiZ_i. The arguments require only that the relevant conditional inclusion probabilities are known by design and bounded away from zero. We write the population nuisance functions as μ0(z)=(Ai∣Zi=z,Ri=1,Vi=1),g0(b)=μ0(Zi)∣Bi=b,Ri=1. _0(z)=E(A_i Z_i=z,R_i=1,V_i=1), g_0(b)=E\ _0(Z_i) B_i=b,R_i=1\. Under Assumptions 1–3, μ0(Zi)=(Yi⋆∣Zi,Ri=1) _0(Z_i)=E(Y_i Z_i,R_i=1) and g0(Bi)=(Yi⋆∣Bi)g_0(B_i)=E(Y_i B_i). Thus g0g_0 is a population target; it does not depend on the realized nuisance estimate or on a particular cross-fit pseudo-label. Pointwise identification. Under Assumption˜1, [Yi⋆∣i,Qi]=[Yi⋆∣i,Qi,Ri=1]E[Y _i X_i,Q_i]=E[Y _i X_i,Q_i,R_i=1]. Under Assumptions˜2 and 3, [Yi⋆∣Zi,Ri=1]=[Ai∣Zi,RiVi=1]=μ0(Zi)E[Y _i Z_i,R_i=1]=E[A_i Z_i,R_iV_i=1]= _0(Z_i). Composing, [μ0(Zi)∣Bi,Ri=1]=[Yi⋆∣Bi,Ri=1]=[Yi⋆∣Bi]=g0(Bi)E[ _0(Z_i) B_i,R_i=1]=E[Y _i B_i,R_i=1]=E[Y _i B_i]=g_0(B_i). Design-validity of Y~i Y_i. Fix arbitrary nuisance functions η=(μ,g)η=(μ,g). For audited units define Mi(μ)=μ(Zi)+Viρ(Zi)Ai−μ(Zi).M_i(μ)=μ(Z_i)+ V_iρ(Z_i)\A_i-μ(Z_i)\. When the nuisances are estimated by cross-fitting, the following argument is understood conditionally on the training folds, so that μ and g are fixed functions for the evaluation fold. First consider the inner correction. Conditional on Ri=1R_i=1 and ZiZ_i, EMi(μ)∣Ri=1,Zi E\M_i(μ) R_i=1,Z_i\ =μ(Zi)+E[Viρ(Zi)Ai−μ(Zi)∣Ri=1,Zi] =μ(Z_i)+E [ V_iρ(Z_i)\A_i-μ(Z_i)\ R_i=1,Z_i ] =μ(Zi)+EAi−μ(Zi)∣Ri=1,Zi,Vi=1 =μ(Z_i)+E\A_i-μ(Z_i) R_i=1,Z_i,V_i=1\ =E(Ai∣Ri=1,Zi,Vi=1). =E(A_i R_i=1,Z_i,V_i=1). By adjudication correctness, Ai=Yi⋆A_i=Y_i on RiVi=1\R_iV_i=1\, and by adjudication ignorability, E(Ai∣Ri=1,Zi,Vi=1)=E(Yi⋆∣Ri=1,Zi).E(A_i R_i=1,Z_i,V_i=1)=E(Y_i R_i=1,Z_i). Hence EMi(μ)∣Ri=1,Zi=E(Yi⋆∣Ri=1,Zi).E\M_i(μ) R_i=1,Z_i\=E(Y_i R_i=1,Z_i). No restriction on the joint distribution of (Gi,1,Gi,2)(G_i,1,G_i,2) is used here: once we condition on ZiZ_i, the coder labels are simply observed covariates, and all coder dependence or bias is part of the conditional mean μ0(Zi) _0(Z_i). Now consider the outer correction. Conditional on BiB_i, EY~i(η)∣Bi E\ Y_i(η) B_i\ =g(Bi)+E[Riπ(Bi)Mi(μ)−g(Bi)∣Bi] =g(B_i)+E [ R_iπ(B_i)\M_i(μ)-g(B_i)\ B_i ] =g(Bi)+EMi(μ)−g(Bi)∣Ri=1,Bi =g(B_i)+E\M_i(μ)-g(B_i) R_i=1,B_i\ =EMi(μ)∣Ri=1,Bi. =E\M_i(μ) R_i=1,B_i\. Using iterated expectations and the inner result, EMi(μ)∣Ri=1,Bi E\M_i(μ) R_i=1,B_i\ =E[EMi(μ)∣Ri=1,Zi∣Ri=1,Bi] =E [E\M_i(μ) R_i=1,Z_i\ R_i=1,B_i ] =E[E(Yi⋆∣Ri=1,Zi)∣Ri=1,Bi] =E [E(Y_i R_i=1,Z_i) R_i=1,B_i ] =E(Yi⋆∣Ri=1,Bi). =E(Y_i R_i=1,B_i). Finally, by audit ignorability, E(Yi⋆∣Ri=1,Bi)=E(Yi⋆∣Bi).E(Y_i R_i=1,B_i)=E(Y_i B_i). Therefore EY~i(η)∣Bi=E(Yi⋆∣Bi).E\ Y_i(η) B_i\=E(Y_i B_i). Since XiX_i is measurable with respect to BiB_i, E[XiY~i(η)−expit(Xi⊤β)]=E[XiYi⋆−expit(Xi⊤β)].E [X_i\ Y_i(η)-expit(X_i β)\ ]=E [X_i\Y_i -expit(X_i β)\ ]. At β=β0β= _0, the right-hand side is zero by definition of β0 _0. Thus Em(Wi;β0,η)=0E\m(W_i; _0,η)\=0 for every fixed nuisance pair η. This fixed-nuisance statement establishes design-unbiasedness of the score, but it does not imply that arbitrary inconsistent nuisance estimators have the same first-order expansion. The influence-function representation below is obtained by expanding around η0=(μ0,g0) _0=( _0,g_0) and therefore requires the stated cross-fit nuisance convergence conditions. Influence function. The cross-fit estimator ψ ψ admits the asymptotic linear expansion N(ψ^−ψ)=N−1/2∑iφ(Wi;μ0,g0)+op(1) N( ψ-ψ)=N^-1/2 _i (W_i; _0,g_0)+o_p(1) with influence function φ(Wi;μ0,g0)=g0(Bi)−ψ+Riπi[μ0(Zi)−g0(Bi)]+RiViπiρi[Ai−μ0(Zi)]. (W_i; _0,g_0)=g_0(B_i)-ψ+ R_i _i [ _0(Z_i)-g_0(B_i) ]+ R_iV_i _i _i [A_i- _0(Z_i) ]. (7) The estimator β β from Eq.˜6 has influence function −1i(Y~i(η0)−expit(i⊤β0)),=[ii⊤expit′(i⊤β0)],H^-1\, X_i\, ( Y_i( _0)-expit( X_i _0) ), =E[ X_i X_i \,expit ( X_i _0)], (8) obtained directly from the M-estimator score m(Wi;β0,η0)m(W_i; _0, _0); the prevalence-target φ is not the right object for β β because the logistic score is not the prevalence influence function. Why no product-rate condition is required. Let δg(B)=g^(B)−g0(B) _g(B)= g(B)-g_0(B) and δμ(Z)=μ^(Z)−μ0(Z) _μ(Z)= μ(Z)- _0(Z). Direct expansion of Y~i Y_i gives Y~i(η^)−Y~i(η0)=(1−Riπi)δg(Bi)+Riπi(1−Viρi)δμ(Zi), Y_i( η)- Y_i( _0)= (1- R_i _i ) _g(B_i)+ R_i _i (1- V_i _i ) _μ(Z_i), (9) so that m(Wi;β0,η^)−m(Wi;β0,η0)=i⋅[r.h.s. of Eq.˜9]m(W_i; _0, η)-m(W_i; _0, _0)= X_i·[r.h.s.\ of @crefcreftype~refnumeq:nuis-perturb]. The two nuisance errors enter linearly, but each is multiplied by a known-design, conditionally mean-zero sampling residual. This is why the argument requires only L2L^2 consistency of the nuisance estimators, rather than a product-rate condition. Leakage-free nested cross-fitting. We first analyze a leakage-free nested version of the estimator. For a held-out evaluation fold ℱkF_k, the inner nuisance used to construct audit pseudo-labels for the outer training sample is fit without using observations from ℱkF_k. The outer nuisance used to evaluate ℱkF_k is therefore trained on pseudo-labels that contain no information from the held-out fold. Conditional on the training data for that fold, the nuisance functions are fixed with respect to the evaluation observations. For such an evaluation observation, the conditional means of both bracketed factors in Eq.˜9 are zero: [1−Riπi|Bi]=0E\! [1- R_i _i\, |\,B_i ]=0 by Assumption˜1, and [Riπi(1−Viρi)|Zi,Bi]=0E\! [ R_i _i (1- V_i _i )\, |\,Z_i,B_i ]=0 by Assumption˜2; the inner factor has conditional mean zero given Ri=1R_i=1 and ZiZ_i. Hence [m(Wi;β0,η^)−m(Wi;β0,η0)|training data]=.E\! [m(W_i; _0, η)-m(W_i; _0, _0)\, |\,training data ]=0. (10) Thus the nuisance error contributes only a centered empirical-process remainder. Because πi _i and ρi _i are bounded away from zero, and ‖i‖2+δ<∞E\| X_i\|^2+δ<∞, there is a finite constant C such that [‖m(Wi;β0,η^)−m(Wi;β0,η0)‖2|training data]≤C(‖g^−g0‖2,X,B2+‖μ^−μ0‖2,X,Z2),E\! [ \|m(W_i; _0, η)-m(W_i; _0, _0) \|^2\, |\,training data ]≤ C (\| g-g_0\|_2,X,B^2+\| μ- _0\|_2,X,Z^2 ), (11) where ‖h‖2,X,B2=[‖i‖2h(Bi)2],‖r‖2,X,Z2=[‖i‖2r(Zi)2].\|h\|_2,X,B^2=E\! [\| X_i\|^2h(B_i)^2 ], \|r\|_2,X,Z^2=E\! [\| X_i\|^2r(Z_i)^2 ]. Therefore, N(PN−P)m(W;β0,η^)−m(W;β0,η0)=Op(‖g^−g0‖2,X,B+‖μ^−μ0‖2,X,Z)=op(1), N(P_N-P)\m(W; _0, η)-m(W; _0, _0)\=O_p\! (\| g-g_0\|_2,X,B+\| μ- _0\|_2,X,Z )=o_p(1), under cross-fit X-weighted L2L^2-consistent nuisance estimation. Combining this centered empirical-process bound with the fixed-nuisance design-unbiasedness result above and a standard mean-value expansion in β around β0 _0 gives the asymptotic linear expansion in Proposition˜1 for the leakage-free nested estimator. No product-rate condition appears because the known design weights make the linear nuisance remainder conditionally mean-zero. Shared-partition cross-fitting. Algorithm 1 uses a computationally simpler shared-partition implementation, in which the same folds are used in the inner and outer passes. For fold ℱkF_k, write g^sp(−k) g_sp^(-k) for the outer nuisance actually used by the algorithm. Let g^lf(−k) g_lf^(-k) denote the leakage-free counterpart: the outer nuisance that would be obtained by constructing each training pseudo-label M^j M_j, j∉ℱkj _k, using an inner fit that excludes both ℱkF_k and ℱfold(j)F_fold(j). The shared-partition estimator has the same first-order expansion as the leakage-free estimator provided the leakage remainder satisfies 1N∑k=1K∑i∈ℱkXi(1−Riπi)g^sp(−k)(Bi)−g^lf(−k)(Bi)=op(1). 1 N _k=1^K _i _kX_i (1- R_i _i ) \ g_sp^(-k)(B_i)- g_lf^(-k)(B_i) \=o_p(1). (SP) To see this, add and subtract the leakage-free score: 1N∑k=1K∑i∈ℱk[m(Wi;β0,μ^(−k),g^sp(−k))−m(Wi;β0,η0)] 1 N _k=1^K _i _k [m (W_i; _0, μ^(-k), g_sp^(-k) )-m(W_i; _0, _0) ] =1N∑k=1K∑i∈ℱk[m(Wi;β0,μ^(−k),g^lf(−k))−m(Wi;β0,η0)] = 1 N _k=1^K _i _k [m (W_i; _0, μ^(-k), g_lf^(-k) )-m(W_i; _0, _0) ] +1N∑k=1K∑i∈ℱkXi(1−Riπi)g^sp(−k)(Bi)−g^lf(−k)(Bi). + 1 N _k=1^K _i _kX_i (1- R_i _i ) \ g_sp^(-k)(B_i)- g_lf^(-k)(B_i) \. (12) The first term is op(1)o_p(1) by the leakage-free argument. The second term is op(1)o_p(1) by condition (SP). Hence the shared-partition estimator has the same asymptotic linear representation as the leakage-free estimator whenever condition (SP) holds. Discussion of the shared-partition stability condition. Condition (SP) is a stability condition on the indirect leakage created by reusing the same fold partition in the inner and outer stages. It is expected to hold when the inner learner is stable under adding or removing one fold, the outer learner is Lipschitz-stable in its training responses, and the resulting difference ‖g^sp(−k)−g^lf(−k)‖2,X=op(1)\| g_sp^(-k)- g_lf^(-k)\|_2,X=o_p(1) is sufficiently small that the weighted empirical average in (SP) is op(1)o_p(1). This is plausible for stable nuisance learners in fixed-dimensional smooth classes, such as the fixed-hyperparameter L2L^2-regularized logistic regressions used in the experiments, under standard nonsingularity and bounded-moment conditions. A formal verification requires learner-specific deletion stability controlling the effect of excluding ℱkF_k from the inner fits on the resulting outer nuisance. The condition may fail for highly adaptive nuisance procedures, including tuning, stacking, or model selection performed inside the cross-fitting loop, because such procedures can transmit evaluation-fold information into g^sp(−k) g_sp^(-k). Variance decomposition. Let H(Z,B)=μ0(Z)−g0(B),ε=A−μ0(Z),H(Z,B)\;=\; _0(Z)-g_0(B), \;=\;A- _0(Z), (13) so that φ(W;μ0,g0)=(g0(B)−ψ)+Rπ(B)H(Z,B)+RVπ(B)ρ(Z)ε (W; _0,g_0)= (g_0(B)-ψ )+ Rπ(B)H(Z,B)+ RVπ(B)\,ρ(Z) . Because [H(Z,B)∣B,R=1]=0E[H(Z,B) B,R=1]=0 (by definition of g0g_0) and [ε∣Z,R=1,V=1]=0E[ Z,R=1,V=1]=0 (by definition of μ0 _0), the three summands are mutually orthogonal martingale increments. Iterated expectations give [(R/π(B))2H(Z,B)2]=[H(Z,B)2/π(B)]E[(R/π(B))^2H(Z,B)^2]=E[H(Z,B)^2/π(B)] and [(RV/(π(B)ρ(Z)))2ε2]=[σA2(Z)/(π(B)ρ(Z))]E[(RV/(π(B)ρ(Z)))^2 ^2]=E[ _A^2(Z)/(π(B)\,ρ(Z))] where σA2(Z)=Var(A∣Z,R=1,V=1) _A^2(Z)=Var(A Z,R=1,V=1), so Var[φ]=Var[g0(B)]+[H(Z,B)2π(B)]+[σA2(Z)π(B)ρ(Z)].Var[ ]\;=\;Var[g_0(B)]\;+\;E\! [ H(Z,B)^2π(B) ]\;+\;E\! [ _A^2(Z)π(B)\,ρ(Z) ]. (14) For constant π(B)≡π(B)≡π and ρ(Z)≡ρ(Z)≡ρ, Var[φ]=Var[g0(B)]+π−1[(μ0(Z)−g0(B))2]+(πρ)−1[σA2(Z)]Var[ ]=Var[g_0(B)]+π^-1E[( _0(Z)-g_0(B))^2]+(πρ)^-1E[ _A^2(Z)]. At π=ρ=1π=ρ=1 this collapses to Var(A)Var(A) via the law of total variance, as it must. Efficiency gain over DSL-AdjOnly. The adjudication-only DSL pseudo-outcome treats RiViR_iV_i as the sampling indicator and uses the joint design weight 1/(π(Bi)ρ(Zi))1/(π(B_i)\,ρ(Z_i)) to debias the outer regression g g, giving influence function φadj=(g0(B)−ψ)+(RV/(π(B)ρ(Z)))(A−g0(B)) _adj=(g_0(B)-ψ)+(RV/(π(B)\,ρ(Z)))(A-g_0(B)) — structurally a single-stage AIPW with the joint sampling indicator. Decomposing A−g0(B)=H(Z,B)+εA-g_0(B)=H(Z,B)+ and applying the same orthogonality argument, Var[φadj]=Var[g0(B)]+[H(Z,B)2π(B)ρ(Z)]+[σA2(Z)π(B)ρ(Z)].Var[ _adj]\;=\;Var[g_0(B)]\;+\;E\! [ H(Z,B)^2π(B)\,ρ(Z) ]\;+\;E\! [ _A^2(Z)π(B)\,ρ(Z) ]. (15) Subtracting Eq.˜14 from Eq.˜15 gives Var[φadj]−Var[φ]=[1−ρ(Z)π(B)ρ(Z)(μ0(Z)−g0(B))2]≥ 0,Var[ _adj]-Var[ ]\;=\;E\! [ 1-ρ(Z)π(B)\,ρ(Z)\, ( _0(Z)-g_0(B) )^2 ]\;≥\;0, (16) which simplifies to ((1−ρ)/(πρ))[(μ0(Z)−g0(B))2]((1-ρ)/(πρ))\,E[( _0(Z)-g_0(B))^2] for constant π,ρπ,ρ. For the logistic-regression target in the main text, the same comparison applies to the covariance of the score influence function XiY~i(η0)−expit(Xi⊤β0)X_i\ Y_i( _0)-expit(X_i _0)\. In particular, the corresponding difference in score covariance matrices is Ωadj−Ω=[XiXi⊤1−ρ(Zi)π(Bi)ρ(Zi)(μ0(Zi)−g0(Bi))2], _adj- \;=\;E\! [X_iX_i 1-ρ(Z_i)π(B_i)\,ρ(Z_i)\, ( _0(Z_i)-g_0(B_i) )^2 ], (17) which is positive semidefinite. Applying the usual M-estimation transformation, the corresponding asymptotic covariance difference for β β is H−1(Ωadj−Ω)H−1⊤,H^-1( _adj- )H^-1 , also positive semidefinite. Thus the scalar comparison in Eq.˜16 has the same interpretation for the main logistic-regression estimand: PA-DSL removes the extra ρ(Zi)−1ρ(Z_i)^-1 inflation from the component of adjudicated truth explained by audit-tier features beyond BiB_i. The PA-DSL estimator does not reduce the residual adjudication noise term σA2(Z) _A^2(Z), which can only be learned from adjudicated units and remains weighted by 1/(π(B)ρ(Z))1/(π(B)ρ(Z)) in both estimators. What PA-DSL does is move the Z-explainable component μ0(Z)−g0(B) _0(Z)-g_0(B) from the sparse adjudication weight 1/(π(B)ρ(Z))1/(π(B)ρ(Z)) to the larger audit weight 1/π(B)1/π(B). The gain in Eq.˜16 is therefore large precisely when (i) adjudication is sparse, so ρ(Z)ρ(Z) is small, and (i) the audit-level features Z explain substantial variation in the adjudicated label, and collapses to zero when Z has no incremental predictive power over B. Appendix C Why build on DSL rather than PPI? Prediction Powered Inference (PPI) is a closely related prediction-assisted inference framework [Angelopoulos et al., 2023]. We build on DSL because our measurement process is explicitly design-based: the audit and adjudication samples are drawn with known probabilities chosen by the analyst. The DSL pseudo-outcome is therefore the natural object to extend, and partial adjudication is handled by nesting a second known-weight correction inside the usual audit correction. This is not a claim that PPI is inappropriate; rather, deriving a partially adjudicated PPI analogue is a separate extension. Recent benchmarking finds that DSL can be competitive with, and often more efficient than, PPI for LLM-based parameter estimation, though performance varies by setting [de Pieuchon et al., 2025]. Appendix D Implementation Details All nuisances are L2-regularized logistic regression unless otherwise noted (C=1C=1, max-iter 20002000); predictions are clipped to [10−3,1−10−3][10^-3,1-10^-3] to bound the importance weights in Eq.˜5. Cross-fitting uses K=5K=5 folds, stratified on twenty Q-quintile × BdiffB_diff-quartile strata where Bdiff=1−2|p⋆−0.5|B_diff=1-2|p -0.5| is the boundary statistic. β is fit by damped iteratively reweighted least squares with a 10−610^-6 ridge for numerical stability; standard errors use the empirical sandwich on the cross-fit pseudo-outcome. The estimators evaluated in the main text and appendices are: Surrogate-only, Human-naive, DSL-MajVote (two coders, ties randomly broken), DSL-AdjOnly, DSL-oracle, PA-DSL (logistic, baseline inner features), PA-DSL-coder (logistic, coder-pair features), PA-DSL-IPW-only (inner μ^≡0 μ≡ 0; pure Horvitz–Thompson at the inner stage), and PA-DSL-Reg-only (no V/ρV/ρ debias at the inner stage; pure regression). Appendix E Synthetic Data-Generating Process Each replication generates N=50,000N=50,000 units. Covariates i X_i comprise an intercept and p=5p=5 continuous features drawn from a multivariate normal with unit variances and equicorrelation ρcov=0.2 _cov=0.2. Latent labels are Yi⋆∣i∼Bernoulli(expit(i⊤))Y _i X_i (expit( X_i β)), with =(−0.4, 1.0,−0.8, 0.6, 0.0, 0.0)⊤ β=(-0.4,\,1.0,\,-0.8,\,0.6,\,0.0,\,0.0) . The boundary statistic is Bi,diff=1−2|expit(i⊤)−0.5|B_i,diff=1-2|expit( X_i β)-0.5|. Three latent difficulty channels are Di D_i =δ0+δ1Bi,diff+δ2|Xi,1|+δ3|Xi,2|+Ui,Ui∼N(0,σD2), = _0+ _1B_i,diff+ _2|X_i,1|+ _3|X_i,2|+U_i, U_i N(0, _D^2), Hi H_i =η1Xi,3+η2Xi,4+ξi,ξi∼N(0,σH2), = _1X_i,3+ _2X_i,4+ _i, _i N(0, _H^2), Mi M_i =ζ1Xi,4+ζ2Xi,5+νi,νi∼N(0,σM2). = _1X_i,4+ _2X_i,5+ _i, _i N(0, _M^2). The surrogate is Qi=expit(τ0+τ1Yi⋆+τ2Xi,1+τ3Xi,2−λDDi−λMMi+εi)Q_i=expit( _0+ _1Y _i+ _2X_i,1+ _3X_i,2- _DD_i- _MM_i+ _i) with εi∼N(0,σQ2) _i N(0, _Q^2). Coder j has sensitivity intercept αj(1)∼N(asens,σcoder2)α^(1)_j N(a_sens, _coder^2) and specificity intercept αj(0)∼N(aspec,σcoder2)α^(0)_j N(a_spec, _coder^2); conditional on Yi⋆Y _i and the difficulty channels, ℙ(Gij=1∣Yi⋆=1)=expit(αj(1)−γDDi−γHHi)P(G_ij=1 Y _i=1)=expit(α^(1)_j- _DD_i- _HH_i) and the symmetric expression on Yi⋆=0Y _i=0. The audit indicator is uniform Bernoulli(π); audited units draw two distinct coders without replacement. Under uniform adjudication ρi≡ρ¯ _i≡ ρ; under disagreement-based adjudication ρi∝d0+d1Gi,1≠Gi,2 _i d_0+d_11\G_i,1≠ G_i,2\ rescaled to mean ρ¯ ρ. Table 4: Scenario parameters varied across the synthetic sweep. Default values not listed: δ0=0 _0=0, δ1=1.5 _1=1.5, δ2=δ3=0.4 _2= _3=0.4, σD=0.75 _D=0.75, η1=0.8 _1=0.8, η2=−0.5 _2=-0.5, σH=0.5 _H=0.5, ζ1=0.6 _1=0.6, ζ2=0.5 _2=0.5, σM=0.5 _M=0.5, τ0=0 _0=0, τ2=0.3 _2=0.3, τ3=−0.2 _3=-0.2, asens=2.4a_sens=2.4, aspec=2.6a_spec=2.6, J=8J=8, d0=0.1d_0=0.1, d1=1.0d_1=1.0. Parameter Benign Realistic Hard τ1 _1 2.0 1.5 1.0 λD _D 0.5 1.0 1.4 λM _M 0.4 0.8 1.2 σQ _Q 0.7 1.0 1.4 σcoder _coder 0.20 0.35 0.50 γD _D 0.5 1.0 1.4 γH _H 0.35 0.7 1.0 Adjudication policy uniform disagreement disagreement ρ¯ ρ 0.25 0.25 0.10 Appendix F Inner-Stage Feature Ablation Table˜5 ablates the inner nuisance feature set (R=200R=200 replications). Coder-pair features add no detectable improvement over the baseline feature set in any scenario. We adopt the baseline-feature logistic variant as the default PA-DSL throughout the main text. Table 5: Inner-stage feature ablation, β1 _1 (R=200R=200 replications, true β1=1 _1=1). Bold marks the column-best for MC SD, RMSE, and CI length among the design-valid estimators (DSL-AdjOnly, PA-DSL variants); coverage is a calibration diagnostic and bias is reported but not bolded. Scenario Inner stage Bias MC SD RMSE Coverage CI len. benign DSL-AdjOnly (no inner) 0.0060.006 0.0570.057 0.0570.057 0.9650.965 0.2130.213 PA-DSL (logistic, baseline) 0.0050.005 0.047 0.047 0.047 0.047 0.9500.950 0.173 0.173 PA-DSL (logistic, +coder) 0.0060.006 0.0480.048 0.0480.048 0.9350.935 0.1740.174 realistic DSL-AdjOnly (no inner) 0.0270.027 0.1300.130 0.1330.133 0.9600.960 0.5240.524 PA-DSL (logistic, baseline) 0.0150.015 0.116 0.116 0.117 0.117 0.9450.945 0.466 0.466 PA-DSL (logistic, +coder) 0.0160.016 0.1180.118 0.1180.118 0.9400.940 0.4690.469 hard DSL-AdjOnly (no inner) 0.0520.052 0.248 0.248 0.253 0.253 0.9500.950 0.932 0.932 PA-DSL (logistic, baseline) 0.0510.051 0.2500.250 0.2550.255 0.9450.945 0.932 0.932 PA-DSL (logistic, +coder) 0.0520.052 0.2510.251 0.2550.255 0.9500.950 0.9480.948 Appendix G Detox: Additional Proxy Truths Table˜6 reports the Detox β1 _1 comparison under the two proxy-truth constructions not shown in the main text: hard majority vote on the reference panel (R=50R=50 replications, target β1=−1.50 _1=-1.50) and the soft mean attack share (R=50R=50, target β1=−1.10 _1=-1.10). The qualitative ranking is unchanged: the plug-in baselines and DSL-MajVote fed with noisy gold miss the proxy with zero coverage on the hard-majority proxy; DSL-AdjOnly and PA-DSL both achieve nominal coverage; PA-DSL reduces RMSE relative to DSL-AdjOnly by 1313–16%16\%. Because Y⋆Y is unobservable in the field, the displayed truth is the proxy refit on the full reference panel; DSL-oracle is therefore an upper bound on achievable accuracy under each proxy, not an oracle for the latent attack construct. Table 6: Detox semi-synthetic study, β1 _1 on logged_in under the two proxy-truth constructions deferred from the main text. Bold marks the column-best for MC SD, RMSE, and CI length per proxy among design-valid estimators (DSL-AdjOnly, PA-DSL); coverage is a calibration diagnostic and bias is reported but not bolded. The light rule separates the infeasible DSL-oracle benchmark. Proxy Method Bias MC SD RMSE Coverage CI len. hard maj. Surrogate-only 0.393 -0.393 0.0000.000 0.3930.393 0.000.00 0.0500.050 Human-naive 0.409 -0.409 0.0460.046 0.4120.412 0.000.00 0.2080.208 DSL-MajVote 0.416 -0.416 0.0390.039 0.4180.418 0.000.00 0.1800.180 DSL-AdjOnly −0.005-0.005 0.1020.102 0.1010.101 0.980.98 0.4430.443 PA-DSL −0.006-0.006 0.088 0.088 0.087 0.087 0.980.98 0.398 0.398 DSL-oracle −0.000-0.000 0.0470.047 0.0460.046 0.960.96 0.1790.179 soft mean Surrogate-only 0.285 -0.285 0.0000.000 0.2850.285 0.000.00 0.0350.035 Human-naive 0.009 -0.009 0.0460.046 0.0460.046 0.980.98 0.2080.208 DSL-MajVote 0.017 -0.017 0.0380.038 0.0410.041 1.001.00 0.1790.179 DSL-AdjOnly 0.013 -0.013 0.0670.067 0.0680.068 0.900.90 0.2590.259 PA-DSL 0.012 -0.012 0.056 0.056 0.057 0.057 0.980.98 0.225 0.225 DSL-oracle 0.003 -0.003 0.0190.019 0.0190.019 1.001.00 0.0970.097 Appendix H Design Curves: Audit and Adjudication Sweep The three benign / realistic / hard scenarios in the main text are static design points. To answer the practical question “how much adjudication should an analyst buy, and how should adjudicated items be selected?", we sweep the audit rate π∈0.05,0.10,0.20π∈\0.05,0.10,0.20\, the mean adjudication rate ρ¯∈0.10,0.25,0.50 ρ∈\0.10,0.25,0.50\, the adjudication policy ∈\uniform, disagreement\, and an optional positivity floor ρmin,agree∈0,0.05 _ ,agree∈\0,0.05\ on agreement cases (under disagreement-driven adjudication, units with Gi,1=Gi,2G_i,1=G_i,2 would otherwise receive ρ≈d0/(d0+d1)ρ¯ρ≈ d_0/(d_0+d_1)\, ρ, which can be very small). Other parameters are fixed at the realistic-scenario values (Table˜4); each cell uses R=200R=200 replications. Figure˜4 reports the β1 _1 RMSE ratio RMSE(DSL-AdjOnly)/RMSE(PA-DSL)RMSE( DSL-AdjOnly)/RMSE( PA-DSL) across the sweep, with paired-bootstrap 95%95\% CIs. Three patterns emerge. First, disagreement-driven adjudication is consistently higher in point estimate than uniform adjudication across the grid: the disagreement panels (bottom row) sit roughly 55–1010 percentage points above the uniform panels at matched mean budget, with overlapping but offset CIs at R=200R=200 replications per cell. The mechanism is that adjudication concentrates on the cases where coder labels disagree — exactly where the inner μ μ has the largest residual to debias and the most leverage to add over DSL-AdjOnly. Second, the gain is largest at the moderate audit rate π=0.10π=0.10 under disagreement (orange line, peak ratio 1.151.15 at ρ¯=0.10 ρ=0.10); π=0.05π=0.05 leaves too little inner training data, while π=0.20π=0.20 already buys enough adjudicated truth that the inner correction has less to add and the ratio drops toward 1.061.06 at ρ¯=0.50 ρ=0.50. Third, the positivity floor on agreement cases has only small, mostly within-CI effects on the realistic-scenario sweep. The ρmin,agree=0.05 _ ,agree=0.05 panels are nearly indistinguishable from the no-floor panels. The floor would matter more at scenarios with very small d0/(d0+d1)d_0/(d_0+d_1) ratios (where the unfloored agreement-case ρi _i is close to zero); we leave that exploration to future work. For practical pipelines, the sweep suggests targeting π∈[0.10,0.20]π∈[0.10,0.20] with disagreement-driven adjudication at ρ¯∈[0.10,0.25] ρ∈[0.10,0.25] as a sweet spot of audit cost vs. inferential efficiency. Figure 4: Design-curve sweep on the realistic scenario. RMSE ratio RMSE(DSL-AdjOnly)/RMSE(PA-DSL)RMSE( DSL-AdjOnly)/RMSE( PA-DSL) for β1 _1, with 95%95\% paired-bootstrap CIs (R=200R=200 reps per cell, 10001000 bootstrap draws). Higher values favor PA-DSL. The dashed line at 1.01.0 marks the no-gain point. Appendix I Inner-Stage AIPW Ablation PA-DSL’s inner stage combines an outcome model μ^(Z) μ(Z) with a Horvitz–Thompson correction V/ρV/ρ around it. We isolate the contribution of each piece by re-running the synthetic analysis with inner-stage variants: • IPW-only. Set μ^≡0 μ≡ 0, so M^i=(Vi/ρi)Ai M_i=(V_i/ _i)\,A_i. Pure Horvitz–Thompson at the inner stage. Design-valid (unbiased for any nuisance choice) but does not exploit the surrogate or coder labels at the inner stage, and is sensitive to small ρi _i values. • IPW-only (trunc). As IPW-only but with the truncated weight ρi↦max(ρi,0.05) _i ( _i,0.05), capping 1/ρi≤201/ _i≤ 20. This is a numerical sensitivity check that disentangles weight-stability failures from fundamental design issues; truncation breaks design-validity under the original sampling weights. • Reg-only. Skip the V/ρV/ρ debias; set M^i=μ^(Zi) M_i= μ(Z_i). Pure regression. Lower variance than IPW-only when μ μ fits well, but the sandwich SE no longer accounts for the nuisance estimation noise, so confidence intervals can severely undercover. Table˜7 reports the comparison on the three synthetic scenarios at R=200R=200. The IPW-only variant pays a steep variance price for not exploiting the inner features. In benign and realistic, RMSE roughly triples; in hard the estimator becomes numerically unstable. The disagreement-weighted adjudication policy at ρ¯=0.10 ρ=0.10 produces audited units with ρi _i as low as ∼0.02 \!0.02 (1/ρi≈451/ _i≈ 45), and the Y~i Y_i pseudo-outcomes inherit those weights. Two of R=200R=200 hard-scenario replications produce pseudo-outcome distributions extreme enough to push the logistic IRLS solver outside its convergence regime, yielding |β^1|| β_1| on the order of 10510^5–10610^6 and dominating the column means.111The truncated variant PA-DSL-IPW-only-trunc caps 1/ρi≤201/ _i≤ 20 and restores numerical stability on the hard scenario at the cost of a small bias relative to the original design. We report it as a sensitivity check that disentangles a small-ρ stability artefact from a fundamental design defect; under the original (un-truncated) sampling weights, IPW-only is the design-valid pure-Horvitz–Thompson baseline. The Reg-only variant achieves lower MC variance than full PA-DSL in benign and realistic, and looks nearly unbiased, but the empirical coverage drops to 0.6450.645 in benign, 0.2600.260 in realistic, and 0.0700.070 in hard. The sandwich SE on (i,Y~i)( X_i, Y_i) is computed as if M^i=μ^(Zi) M_i= μ(Z_i) were a fixed quantity, but μ μ is itself a noisy estimator; without the V/ρV/ρ correction term to absorb that noise into the estimating equation, the SE underestimates the true variability and CIs miss the truth far more often than nominal. The full PA-DSL estimator avoids both failure modes: the regression component μ^(Z) μ(Z) stabilizes the inner pseudo-label relative to IPW-only (cutting variance by an order of magnitude), while the V/ρV/ρ correction restores design validity and makes the sandwich variance reflect the adjudication-stage uncertainty. Table 7: Inner-stage AIPW ablation, β1 _1 across the three synthetic scenarios (R=200R=200 replications, true β1=1 _1=1). IPW-only: inner μ^≡0 μ≡ 0. IPW-only (trunc): IPW-only with truncated weights 1/ρ≤201/ρ≤ 20 — a numerical sensitivity check that breaks design-validity under the original sampling weights, included only to disentangle weight-stability failures from fundamental design issues. † The truncation engages only when some ρi<0.05 _i<0.05, which happens only on the hard scenario; on benign and realistic the IPW-only-trunc row reproduces IPW-only verbatim. Reg-only: no V/ρV/ρ debias at the inner stage. Bold marks the column-best for MC SD, RMSE, and CI length per scenario among design-valid estimators (excluding Reg-only and IPW-only-trunc, which are design-invalid by construction); coverage is a calibration diagnostic and bias is reported but not bolded. The light rule separates the infeasible DSL-oracle benchmark. Scenario Inner stage Bias MC SD RMSE Coverage CI len. benign DSL-AdjOnly 0.006 -0.006 0.0570.057 0.0570.057 0.9650.965 0.2130.213 PA-DSL (full) 0.005 -0.005 0.047 0.047 0.047 0.047 0.9500.950 0.173 0.173 PA-DSL (IPW-only) 0.020 -0.020 0.1390.139 0.1400.140 0.9700.970 0.5590.559 PA-DSL (IPW-only, trunc)† 0.020 -0.020 0.1390.139 0.1400.140 0.9700.970 0.5590.559 PA-DSL (Reg-only) 0.001 -0.001 0.0460.046 0.0460.046 0.6450.645 0.0790.079 DSL-oracle 0.006 -0.006 0.0290.029 0.0300.030 0.9350.935 0.1100.110 realistic DSL-AdjOnly 0.027 -0.027 0.1300.130 0.1330.133 0.9600.960 0.5240.524 PA-DSL (full) 0.015 -0.015 0.116 0.116 0.117 0.117 0.9450.945 0.466 0.466 PA-DSL (IPW-only) 0.085 -0.085 0.2850.285 0.2960.296 0.9700.970 1.1151.115 PA-DSL (IPW-only, trunc)† 0.085 -0.085 0.2850.285 0.2960.296 0.9700.970 1.1151.115 PA-DSL (Reg-only) −0.007-0.007 0.0660.066 0.0660.066 0.2600.260 0.0530.053 DSL-oracle 0.009 -0.009 0.0390.039 0.0400.040 0.9250.925 0.1480.148 hard DSL-AdjOnly 0.052 -0.052 0.248 0.248 0.253 0.253 0.9500.950 0.932 0.932 PA-DSL (full) 0.051 -0.051 0.2500.250 0.2550.255 0.9450.945 0.932 0.932 PA-DSL (IPW-only) numerically unstable; see footnote in prose PA-DSL (IPW-only, trunc) −0.258-0.258 0.1760.176 0.3120.312 0.6550.655 0.6950.695 PA-DSL (Reg-only) −0.014-0.014 0.1230.123 0.1230.123 0.0700.070 0.0230.023 DSL-oracle 0.010 -0.010 0.0420.042 0.0430.043 0.9450.945 0.1580.158 Appendix J Shared-Partition vs. K2K^2-fold Cross-fit Proposition˜1(b) covers two implementations: leakage-free nested cross-fitting and the shared-partition Algorithm˜1 under stability condition Eq.˜SP. Table˜8 reports a paired sanity check on the realistic scenario (N=50,000N=50,000, ρ¯=0.25 ρ=0.25, disagreement-weighted adjudication, R=200R=200 replications, paired per-rep seeds): the two implementations give the same point estimate to within Monte Carlo error, consistent with Eq.˜SP holding for the fixed-hyperparameter L2-regularized logistic nuisances used here. Table 8: Shared-partition vs. K2K^2-fold double cross-fit, β1 _1 on the realistic scenario (R=200R=200 paired replications). The “mean abs. diff.” column is the per-replication |β^1sp−β^1K2|| β_1^sp- β_1^K^2| averaged across replications. Cross-fit scheme Bias MC SD RMSE Coverage CI len. Mean abs. diff. Shared partition (Algorithm˜1) 0.01480.0148 0.11600.1160 0.11660.1166 0.9450.945 0.46640.4664 — K2K^2-fold double cross-fit 0.01480.0148 0.11590.1159 0.11650.1165 0.9450.945 0.46660.4666 0.00090.0009 The mean absolute paired difference 0.00090.0009 is two orders of magnitude smaller than either implementation’s MC SD (≈0.116≈ 0.116), and the sample correlation between the two paired estimates is 0.999950.99995. The shared-partition shortcut reproduces the leakage-free estimator essentially exactly at the cost of K−1K-1 fewer inner fits per outer fold. Appendix K Imperfect-Adjudication Sensitivity Assumption˜3 (Ai=Yi⋆A_i=Y _i on adjudicated units) is load-bearing for design-validity with respect to Yi⋆Y _i. To illustrate what happens when adjudication is systematically wrong, Table˜9 compares the realistic scenario (N=50,000N=50,000, R=500R=500 replications) under correct adjudication against a biased-adjudication mechanism that flips AiA_i with probability 0.0750.075 when Xi,1>0X_i,1>0 and 0.0250.025 otherwise — a 5% average error correlated with the regression’s primary covariate. Under correct adjudication, both DSL-AdjOnly and PA-DSL are unbiased and have nominal coverage. Under biased adjudication, both inherit the adjudicator bias relative to Yi⋆Y _i and miss the truth at higher rates. PA-DSL retains its design-validity for the adjudicated construct (the function [Ai∣i]E[A_i X_i]), but that construct is no longer [Yi⋆∣i]E[Y _i X_i] when adjudicators systematically disagree with the latent label. The audit/adjudication design probabilities cannot recover information adjudicators do not provide; designs that further relax Assumption˜3 require an additional measurement tier (e.g. treating AiA_i as a noisy proxy of Yi⋆Y _i with a known design weight). Table 9: Imperfect-adjudication sensitivity, β1 _1 on the realistic scenario (R=500R=500 replications, target β1=1 _1=1). Correct A=Y*: Assumption˜3 holds. Biased A: AiA_i flipped with probability 0.0750.075 when Xi,1>0X_i,1>0 else 0.0250.025. Bold marks the column-best for MC SD, RMSE, and CI length within each adjudication-target block among design-valid estimators (DSL-AdjOnly, PA-DSL). The light rule separates the infeasible DSL-oracle benchmark. Adjudication target Method Bias MC SD RMSE Coverage CI len. Correct A=Y⋆A=Y DSL-AdjOnly 0.020 -0.020 0.1350.135 0.1360.136 0.9520.952 0.5220.522 PA-DSL 0.013 -0.013 0.123 0.123 0.123 0.123 0.9400.940 0.467 0.467 DSL-oracle 0.007 -0.007 0.0370.037 0.0380.038 0.9460.946 0.1480.148 Biased A DSL-AdjOnly −0.124-0.124 0.1310.131 0.1800.180 0.7900.790 0.5080.508 PA-DSL −0.129-0.129 0.122 0.122 0.178 0.178 0.7840.784 0.472 0.472 DSL-oracle 0.007 -0.007 0.0370.037 0.0380.038 0.9460.946 0.1480.148 Appendix L Classification-oriented predictor diagnostics. Although PA-DSL targets downstream estimating equations rather than individual classification, we report supplemental prediction diagnostics to help interpret the efficiency mechanism. Table 10 compares the full-frame surrogate QiQ_i, the outer nuisance prediction g^(Bi) g(B_i), and the inner nuisance prediction μ^(Zi) μ(Z_i), evaluated against the adjudicated/proxy target. The final AIPW pseudo-outcomes are not included as classifiers because they are estimating-equation devices and need not be calibrated probabilities or lie in [0,1][0,1]. The diagnostics support the mechanism described in Appendix B. In the benign and realistic synthetic settings, μ^(Zi) μ(Z_i) improves prediction relative to g^(Bi) g(B_i), indicating that the audit-tier features add adjudication-relevant information beyond Bi=(Qi,Xi)B_i=(Q_i,X_i). In the hard setting, this incremental signal is small and the inner prediction does not improve on g^(Bi) g(B_i), matching the near-zero efficiency gain reported in the main results. The Detox study shows a modest but consistent improvement from μ^(Zi) μ(Z_i), consistent with the observed PA-DSL efficiency gain. Table 10: Classification-oriented diagnostics for predictive components. AUC is higher-is-better; Brier score is lower-is-better. The RMS difference is n−1∑i(μ^(Zi)−g^(Bi))21/2\n^-1 _i( μ(Z_i)- g(B_i))^2\^1/2, a descriptive measure of incremental audit-tier signal. Diagnostics are evaluated against the adjudicated/proxy target and are not used as inferential validity criteria. QiQ_i g^(Bi) g(B_i) μ^(Zi) μ(Z_i) Study / scenario AUC Brier AUC Brier AUC Brier RMS(μ^−g^)( μ- g) Synthetic: benign 0.949 0.118 0.968 0.070 0.990 0.037 0.300 Synthetic: realistic 0.776 0.206 0.864 0.150 0.869 0.146 0.156 Synthetic: hard 0.639 0.292 0.810 0.176 0.794 0.182 0.064 Detox 0.903 0.096 0.902 0.095 0.908 0.088 0.096