Paper deep dive

Optimal Design for Active Preference Learning with Biased LLM Judges

Zhongman Du, Huiming Zhang, Haodong Zhu, Baochang Zhang

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 149

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/4/2026, 3:19:51 AM

Summary

The paper introduces Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy for active preference learning that accounts for biased LLM judges. Unlike methods that rely on separate calibration, NAOD incorporates judge deviations (nuisance parameters) directly into the acquisition design to minimize policy risk. Theoretical analysis establishes a sharp conditional local asymptotic minimax lower bound on policy risk, showing that representation error can reverse oracle design advantages. Empirical evaluation on Chatbot Arena data demonstrates that NAOD reduces mean proxy policy regret by 29.1% compared to matched target-information designs and improves human-preference prediction.

Entities (7)

NAOD · method · 98%Chatbot Arena · dataset · 97%LLM Judges · entity · 95%Policy Risk · metric · 93%Frank-Wolfe Algorithm · algorithm · 92%Bradley-Terry Model · model · 90%Target-Info Design · baseline-method · 88%

Relation Signals (5)

NAOD → evaluatedon → Chatbot Arena

confidence 96% · evaluate NAOD on Chatbot Arena data

NAOD → uses → Frank-Wolfe Algorithm

confidence 95% · uses the Frank-Wolfe algorithm for optimization.

NAOD → addresses → judge bias

confidence 94% · To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD)... prioritizes policy-relevant target information after nuisance adjustment

NAOD → outperforms → Target-Info Design

confidence 92% · NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design

LLM Judges → exhibit → Bias

confidence 90% · the preferences of the judges may deviate from those of the target human population... expose residual judge bias.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank-Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

146,090 characters extracted from source content.

Expand or collapse full text

Optimal Design for Active Preference Learning with Biased LLM Judges Zhongman Du Huiming Zhang Haodong Zhu Affiliation: Beihang University Zhongguancun Academy Baochang Zhang Affiliation: Hangzhou Innovation Institute of Beihang University Abstract Learning from human preferences is central to large language model (LLM) alignment, but human preference annotation is costly. Active preference learning reduces this cost by selecting informative comparisons, and LLM judges can provide additional scalable feedback. However, the preferences of the judges may deviate from those of the target human population. Even after calibration on trusted reference data, active acquisition can shift the comparison distribution and expose residual judge bias. We therefore incorporate judge deviations into the acquisition design rather than relying on a separate calibration stage. Under joint estimation, comparisons that appear highly informative about the reward may also reflect judge bias and therefore provide less information about human preferences. To address this issue, we propose Nuisance-Adjusted Optimal Design (NAOD), a comparison-selection strategy that prioritizes policy-relevant target information after nuisance adjustment and uses the Frank–Wolfe algorithm for optimization. Theoretically, we establish a sharp conditional local asymptotic minimax lower bound on policy risk and construct an estimator that attains it. We further characterize the finite-sample cost of learning the nuisance representation and show that representation error can reverse an oracle design advantage. Finally, we validate these predictions experimentally and evaluate NAOD on Chatbot Arena data across 17 judges, 15 budget configurations, and 15 random cluster-level splits. NAOD reduces the mean regret of proxy policy by 29.1% relative to a matched target-information design, outperforms existing methods, and improves human-preference prediction on held-out data. 1 Introduction Preference feedback (Christiano et al., 2017) provides a standard way to align large language models (LLMs) with human preferences when desired behavior is difficult to specify directly (Ouyang et al., 2022; Rafailov et al., 2023). Reward models are trained on pairwise response comparisons and then used to guide downstream policy optimization (Stiennon et al., 2020). The resulting policy therefore depends not only on how the reward model is trained, but also on which comparisons are selected for feedback. When human annotation is costly or limited, active preference learning and experimental design make more efficient use of a fixed annotation budget by selecting informative comparisons (Muldrew et al., 2024; Mukherjee et al., 2024). Recent methods further incorporate reward-model information (Lin et al., 2026) or downstream policy objectives (Feng et al., 2025) into their acquisition criteria. A complementary approach to reducing annotation cost is to obtain preference feedback from LLM judges (Lee et al., 2024). However, their judgments can differ systematically from the preferences of the target human population (Zheng et al., 2023). A natural response is to calibrate the judge against trusted human data before relying on its feedback (Polo et al., 2025). However, calibration can deteriorate under distribution shift (Ovadia et al., 2019; Chan et al., 2020), while active acquisition itself changes the comparison distribution by selecting which pairs receive feedback (Farquhar et al., 2021). Residual judge biases that cancel under the reference calibration distribution can therefore become unbalanced on the acquired distribution. Thus, pair selection determines both how much information is collected and which judge errors can influence the learned reward. It is additionally difficult to model these judge deviations during estimation. The human reward and judge-specific deviation must now be estimated jointly and distinguished from one another. On some selected pairs, changes in the human reward and changes in the judge deviation can have nearly the same effect on label probabilities. A design that treats all apparent reward information as target information can therefore overstate the precision with which the human target can be estimated. Figure 1: Overview of the NAOD acquisition and estimation pipeline. Historical human–judge feedback is used to construct the frozen target and judge-deviation representations. Before current outcomes are observed, NAOD selects comparisons using nuisance-adjusted target information weighted by policy relevance. After selection, trusted human and judge feedback are combined in a joint estimator to update the target reward and induced policy. For acquisition, what matters is not the raw information about reward parameters, but the information about the human target that remains after judge-specific deviations have been accounted for. Even this remaining information is not equally important across reward directions, because some reward errors substantially alter downstream policy decisions whereas others have little effect on the induced policy. Reward-model accuracy alone therefore need not reflect downstream policy performance (Gao et al., 2023; Wen et al., 2025; Frick et al., 2025). We therefore propose Nuisance-Adjusted Optimal Design (NAOD), a design strategy that selects comparisons using nuisance-adjusted target information weighted by downstream policy relevance. Figure 1 provides an overview of the NAOD pipeline from pre-acquisition design to post-selection estimation. A sharp conditional local asymptotic minimax lower bound for policy risk is established, together with an estimator that attains it. The resulting risk coincides with the nuisance-adjusted, policy-weighted objective optimized by NAOD. The same target–nuisance cross-information controls both residual exposure and the loss of target information. Further, when the judge-deviation representation is learned, a design-dependent risk term appears that can reverse oracle design rankings. Empirically, controlled experiments validate the predicted policy-risk behavior and the design-ranking reversal. On Chatbot Arena, across 17 judges, 15 budget configurations, and 15 random cluster-level splits, NAOD reduces mean proxy policy regret by 29.1%29.1\% relative to the matched Target-info design and improves held-out human-preference prediction. Its gains tend to be larger when target–nuisance coupling is stronger, consistent with the mechanism predicted by our theory. Our contributions can be summarized as follows: • We formulate active preference learning with biased LLM judges as a joint target–nuisance design problem and characterize how acquisition exposes judge residuals. • We introduce NAOD, establish its sharp local minimax interpretation, and characterize the additional risk from learning the judge-deviation representation. • We develop a practical comparison-selection algorithm for finite candidate pools, validate our theoretical risk predictions in synthetic experiments, and evaluate NAOD on Chatbot Arena with held-out human evaluation. 2 Related Work Active preference acquisition and nuisance-aware design. Active preference learning reduces feedback cost by selecting informative comparisons. Prior work studies adaptive pairwise comparisons (Maystre & Grossglauser, 2017), preference-based reinforcement learning (Lee et al., 2021), query-efficient learning from pairwise preferences (Wu & Sun, 2024), and subset selection for reliable LLM ranking (Liusie et al., 2024). Information-based methods use Fisher information for reward modeling (Shen et al., 2025) or D-optimal design with randomized Frank–Wolfe optimization (Thekumparampil et al., 2025), while other work incorporates downstream policy relevance or reward uncertainty into acquisition (Hu et al., 2024; Sun et al., 2025b). When observations jointly inform target and nuisance parameters, acquisition must also account for nuisance uncertainty (Sloman et al., 2024). Our work combines policy-aware preference acquisition with nuisance-aware experimental design under LLM-judge feedback. LLM judges and calibration. LLM judges provide scalable evaluation but exhibit systematic biases, including position effects (Wang et al., 2024), stylistic preferences (Feuer et al., 2025), and broader recurring biases across human and model evaluators (Chen et al., 2024; Ye et al., 2025). Related work also studies pairwise feedback from systematically biased evaluators (Tang et al., 2025). Existing mitigation approaches characterize limits of debiasing with limited high-quality labels (Dorner et al., 2025), fine-tune evaluators on debiased data (Park et al., 2024), optimize prompts (Zhou et al., 2024), calibrate pairwise prediction distributions (Li et al., 2025), or use human calibration data to correct judge estimates and allocate calibration samples (Lee et al., 2026). Our focus is instead on how judge discrepancies should change which comparisons are acquired for human reward estimation. 3 Problem Setting The Bradley-Terry model of human preferences. Let z=(s,yA,yB)z=(s,y_A,y_B) denote an ordered comparison, where s is a prompt and yA,yBy_A,y_B are two candidate responses. Let ϕ⁡(s,y)∈ℝdφ(s,y) ^d be a fixed d-dimensional feature map. We model the reward as linear in this feature representation, rθ(s,y)=ϕ(s,y)⊤θr_θ(s,y)=φ(s,y) θ, where θ∈ℝdθ ^d is the reward parameter. We define the comparison feature X⁡(z)=ϕ⁡(s,yA)−ϕ⁡(s,yB)X(z)=φ(s,y_A)-φ(s,y_B). Let Y∈0,1Y∈\0,1\ denote the human preference label, with Y=1Y=1 indicating that yAy_A is preferred to yBy_B. Under the Bradley-Terry model (Bradley & Terry, 1952), the preference probability is determined by the reward difference between the two responses, pθ(z)=ℙθ(Y=1∣z)=σ⁡(rθ(s,yA)−rθ(s,yB))=σ⁡(X(z)⊤θ),p_θ(z)=P_θ(Y=1 z)=σ\! (r_θ(s,y_A)-r_θ(s,y_B) )=σ\! (X(z) θ ), (1) where σ⁡(u)=(1+e−u)−1σ(u)=(1+e^-u)^-1 is the sigmoid function. We write θ⋆θ for the reward parameter of the target human population and p⋆=pθ⋆p =p_θ for its preference probability. Performance metric. We evaluate the learned reward through the policy it induces. Let μe _e denote the evaluation distribution over prompts. For each prompt s, let ⁡(s)A(s) be a finite set of candidate responses and let πref(⋅∣s) _ref(· s) be a reference policy on ⁡(s)A(s) with full support. Given a regularization parameter τ>0τ>0, let πθ(⋅∣s) _θ(· s) denote the optimizer of the KL-regularized reward objective (Azar et al., 2012; Liu et al., 2024). Its closed form is πθ(y∣s)=πref(y∣s)exp⁡(rθ(s,y)/τ)∑y′∈⁡(s)πref(y′∣s)exp⁡(rθ(s,y′)/τ). _θ(y s)= _ref(y s) \! (r_θ(s,y)/τ ) _y (s) _ref(y s) \! (r_θ(s,y )/τ ). (2) The corresponding optimal regularized value is F⁡(θ)=τs∼μelog∑y∈⁡(s)πref(y∣s)exp⁡(rθ(s,y)/τ).F(θ)=τ\,E_s _e _y (s) _ref(y s) \! (r_θ(s,y)/τ ). (3) When the target reward parameter is θ⋆θ , deploying the policy induced by θ incurs the loss DF(θ⋆,θ)=F(θ⋆)−F(θ)−∇F(θ)⊤(θ⋆−θ)=τs∼μeKL(πθ(⋅∣s)∥πθ⋆(⋅∣s)).D_F(θ ,θ)=F(θ )-F(θ)-∇ F(θ) (θ -θ)=τ\,E_s _eKL\! ( _θ(· s)\,\|\, _θ (· s) ). (4) See Appendix A.1 for derivations. Thus, our performance metric measures error through its effect on the induced policy. LLM judge and active acquisition. For a fixed LLM judge and annotation protocol, let q⁡(z)∈[0,1]q(z)∈[0,1] denote its preference probability for comparison z. Its preferences need not match those of the target human population. We define the judge discrepancy on the probability scale as δ⁡(z)=q⁡(z)−p⋆(z)δ(z)=q(z)-p (z). Let ν denote a reference distribution over comparisons and let Z∼νZ ν. Conventional probability calibration with respect to the target human population requires ν[Y∣q⁡(Z)]=q⁡(Z)E_ν[Y q(Z)]=q(Z) (Guo et al., 2017; Futami & Fujisawa, 2024; Chidambaram & Ge, 2025). Under the target preference model in equation 1, this is equivalent to ν[δ⁡(Z)∣q⁡(Z)]=0.E_ν[δ(Z) q(Z)]=0. Thus, under ν, calibration constrains the conditional mean of the judge discrepancy given q⁡(Z)q(Z). However, active acquisition changes the comparison distribution. Let ξ denote the acquired comparison distribution and assume ξ≪νξ ν, so that the density ratio ω⁡(z)=dξ/dν(z)ω(z)=dξ/dν(z) is well defined. Acquisition-dependent residual exposure. Probability calibration is defined under the reference distribution ν and is not guaranteed to hold under the acquired distribution ξ (Park et al., 2020). For reward estimation, the relevant issue is how judge discrepancies align with the reward-feature directions. Under the acquired distribution ξ, define the residual contribution to the target score as bξ=ξ[δ⁡(Z)X(Z)]b_ξ=E_ξ[δ(Z)X(Z)], and define bν=ν[δ⁡(Z)X(Z)]b_ν=E_ν[δ(Z)X(Z)] analogously under the reference distribution. This residual-feature moment can be viewed as a multiaccuracy-style residual moment over linear reward-feature tests (Kim et al., 2019; Kern et al., 2024). A change of measure gives bξ=bν+ν[ω⁡(Z)−1δ(Z)X(Z)].b_ξ=b_ν+E_ν [\ω(Z)-1\δ(Z)X(Z) ]. (5) The second term captures how acquisition reweights judge discrepancies across reward-feature directions. Calibration does not imply bν=0b_ν=0, and even when bν=0b_ν=0, acquisition can yield bξ≠0b_ξ≠ 0. In that case, treating judge feedback as target feedback shifts the population score away from zero at θ⋆θ , potentially moving the fitted reward away from the human target. Appendix B.1 gives the corresponding parameter and policy-regret consequences. Thus, active acquisition can expose judge discrepancies that are balanced under the reference distribution. This motivates accounting for judge deviations directly in the acquisition design rather than relying on reference calibration alone. 4 Nuisance-Adjusted Optimal Design 4.1 Joint Target-Nuisance Model We model human and judge preferences jointly. Let W⁡(z)∈ℝrW(z) ^r be a fixed representation of judge deviations, where r is its dimension, and let a∈ℝra ^r be the associated nuisance parameter. We model the fixed judge preference probability q⁡(z)q(z) introduced in Section 3 using a Bradley–Terry model augmented with a structured judge-specific component (Movva et al., 2026), qθ,a(z)=σ⁡(X(z)⊤θ+W(z)⊤a).q_θ,a(z)=σ\! (X(z) θ+W(z) a ). (6) Human labels follow the preference model pθp_θ in equation 1, while W(z)⊤aW(z) a represents the difference between judge and human log odds. For the theory in this section, we assume that q(z)=qθ⋆,a⋆(z)q(z)=q_θ ,a (z) for some unknown nuisance parameter a⋆a . Thus, W(z)⊤a⋆W(z) a parameterizes the judge-human discrepancy on the log-odds scale, while δ⁡(z)δ(z) measures the corresponding discrepancy on the probability scale. Throughout this section, W is treated as fixed. Section 5.1 studies the additional error when W is learned from historical human and judge feedback. For the approximate-design formulation, we first consider a finite support of L comparison types z1,…,zLz_1,…,z_L. Write Xi=X⁡(zi)X_i=X(z_i) and Wi=W⁡(zi)W_i=W(z_i). At type i, we collect nin_i judge labels, each indicating a preference for the first response with probability qθ,a(zi)q_θ,a(z_i). An independent trusted sample contains ncn_c human labels, whose preference probabilities are given by pθp_θ. Conditional on the comparison features, all current labels are independent Bernoulli observations. Let n=∑i=1Lnin= _i=1^Ln_i be the total number of judge labels. We represent the asymptotic allocation by ξ=(ξ1,…,ξL)ξ=( _1,…, _L), where ni/n→ξin_i/n→ _i, ξi≥0 _i≥ 0, and ∑i=1Lξi=1 _i=1^L _i=1. We also assume that nc/n→κn_c/n→κ for some κ∈[0,∞)κ∈[0,∞). We condition on the historical data used to construct W and select the allocation. These data are independent of the current labels. 4.2 Effective Target Information Joint estimation must separate target-score variation from nuisance-score variation (Sloman et al., 2024; Barker & Kavalieris, 2001). Let k=d+rk=d+r be the joint parameter dimension. Define γ=(θ⊤,a⊤)⊤∈ℝkγ=(θ ,a ) ^k and the corresponding feature vi=(Xi⊤,Wi⊤)⊤v_i=(X_i ,W_i ) . Fix a local analysis center γ0=(θ0⊤,a0⊤)⊤ _0=( _0 ,a_0 ) , where θ0 _0 and a0a_0 are the target and nuisance components, respectively. The Bernoulli variance of the judge label at this center is ti=qθ0,a0(zi)1−qθ0,a0(zi)t_i=q_ _0,a_0(z_i)\1-q_ _0,a_0(z_i)\, and one judge label contributes Fisher information Ji=tivivi⊤J_i=t_iv_iv_i . Let HcH_c denote the limiting Fisher information per trusted label at θ0 _0. Let XcX_c denote a random comparison feature in the trusted sample. Then Hc=⁡[σ′(Xc⊤θ0)XcXc⊤]H_c=E[σ (X_c _0)X_cX_c ]. Let Aκ(ξ)A_κ(ξ), DξD_ξ, and CξC_ξ denote the target, nuisance, and cross-information blocks, respectively. The joint information normalized by the judge sample size is Mκ(ξ)=κdiag⁡(Hc,0r×r)+∑i=1LξiJi=(Aκ(ξ)CξCξ⊤Dξ).M_κ(ξ)= (H_c,0_r× r)+ _i=1^L _iJ_i= pmatrixA_κ(ξ)&C_ξ\\ C_ξ &D_ξ pmatrix. (7) Here, Aκ(ξ)A_κ(ξ) is the target-information block, DξD_ξ is the nuisance-information block, and CξC_ξ is the target–nuisance cross-information block. Specifically, Aκ(ξ)=κHc+∑i=1LξitiXiXi⊤A_κ(ξ)=κ H_c+ _i=1^L _it_iX_iX_i , Cξ=∑i=1LξitiXiWi⊤C_ξ= _i=1^L _it_iX_iW_i , and Dξ=∑i=1LξitiWiWi⊤D_ξ= _i=1^L _it_iW_iW_i . When Mκ(ξ)M_κ(ξ) is positive definite, the effective information for the human reward parameter is Ieff,κ(ξ)=Aκ(ξ)−CξDξ−1Cξ⊤.I_eff,κ(ξ)=A_κ(ξ)-C_ξD_ξ^-1C_ξ . (8) Estimating the nuisance parameter reduces the information available for θ from Aκ(ξ)A_κ(ξ) to the Schur complement Ieff,κ(ξ)I_eff,κ(ξ), with information loss CξDξ−1Cξ⊤C_ξD_ξ^-1C_ξ (Fewster & Jupp, 2013). The inverse of Ieff,κ(ξ)I_eff,κ(ξ) is the target block of Mκ(ξ)−1M_κ(ξ)^-1. The cross-information also connects this adjustment to the residual exposure in equation 5. Holding the human parameter at θ0 _0, define bξ(a)=∑i=1Lξiqθ0,a(zi)−pθ0(zi)Xib_ξ(a)= _i=1^L _i\q_ _0,a(z_i)-p_ _0(z_i)\X_i. Let Cξ(a)C_ξ(a) denote the cross-information evaluated at (θ0,a)( _0,a), so that Cξ(a0)=CξC_ξ(a_0)=C_ξ. Differentiation gives the exact Jacobian identity ∂bξ(a)∂a⊤=Cξ(a). ∂ b_ξ(a)∂ a =C_ξ(a). (9) The same cross-information governs the sensitivity of the exposed target score and the information loss from nuisance estimation. See Appendices A.2 and A.3 for the statistical foundations of the joint and effective information, and Appendices B.1 and B.3 for the residual-exposure and target–nuisance coupling results. 4.3 Policy-Weighted Design and Minimax Risk Effective information describes how precisely the human reward can be estimated after nuisance adjustment. The policy regret in equation 4 determines which estimation errors matter. Define the policy curvature at θ0 _0 by G0=∇2F(θ0)=1τs∼μeCovy∼πθ0(⋅∣s)[ϕ(s,y)].G_0=∇^2F( _0)= 1τE_s _eCov_y _ _0(· s)[φ(s,y)]. For h∈ℝdh ^d, the local regret satisfies DF(θ0,θ0+h)=12h⊤G0h+o⁡(‖h‖2)D_F( _0, _0+h)= 12h G_0h+o(\|h\|^2) as h→0h→ 0. Thus, G0G_0 weights reward-parameter directions by their effect on the induced policy. For a reward estimator θ^n θ_n, we measure policy risk by ⁡[DF(θ⋆,θ^n)]E[D_F(θ , θ_n)]. Let Ge=diag⁡(G0,0r×r)G_e=diag(G_0,0_r× r), and let trtr denote the matrix trace. Combining policy curvature with effective target information gives Φκ(ξ)=12tr⁡(G0Ieff,κ(ξ)−1)=12tr⁡(GeMκ(ξ)−1). _κ(ξ)= 12tr\! (G_0I_eff,κ(ξ)^-1 )= 12tr\! (G_eM_κ(ξ)^-1 ). (10) This is a weighted optimal design criterion for the target parameter (Pukelsheim, 2006). Let Ξ be a nonempty compact convex set of allocations on the L comparison types, and assume that Mκ(ξ)M_κ(ξ) is uniformly positive definite over Ξ . We define nuisance-adjusted optimal design (NAOD) by ξ⋆∈argminξ∈Ξ⁡Φκ(ξ).ξ ∈ *arg\,min_ξ∈ _κ(ξ). Since Mκ(ξ)M_κ(ξ) is affine in ξ, the objective is convex. To state its local minimax characterization, let η=(h⊤,υ⊤)⊤η=(h , ) , where υ∈ℝr ^r, and define γn,η=γ0+η/n _n,η= _0+η/ n, with target component θn,h=θ0+h/n _n,h= _0+h/ n. Write ηE_η for expectation under this local model, conditional on the historical data. Theorem 4.1 (Conditional local policy risk). Fix ξ and condition on the historical data. Under the regularity conditions in Appendix A.1, suppose Mκ(ξ)≻0M_κ(ξ) 0. If G0≻0G_0 0, then limR→∞lim infn→∞infTnsup‖η‖≤RnηDF(θn,h,Tn)≥Φκ(ξ), _R→∞ _n→∞ _T_n _\|η\|≤ Rn\,E_ηD_F( _n,h,T_n)≥ _κ(ξ), (11) where TnT_n ranges over reward estimators based on the current observations. Under the additional trusted-sample and initialization conditions in Appendix B.2, with κ>0κ>0, the projected one-step estimator defined there attains this bound uniformly on every fixed local ball. Thus, for the projected one-step estimator, ηDF(θn,h,θ^n)=Φκ(ξ)/n+o⁡(n−1)E_ηD_F( _n,h, θ_n)= _κ(ξ)/n+o(n^-1) uniformly over every fixed local ball. The case G0⪰0G_0 0 is handled by removing policy-null directions. A matched Target-info design replaces Ieff,κ(ξ)I_eff,κ(ξ) by Aκ(ξ)A_κ(ξ) in equation 10 while retaining the same final joint estimator. The two criteria can select different designs, and Proposition B.1 shows that Target-info can incur strictly larger leading policy risk. 4.4 Finite-Pool Comparison Selection We now translate the approximate design into the selection of B distinct comparisons from a finite pool of N candidates. Let xi∈[0,1]x_i∈[0,1] be the relaxed inclusion weight of candidate i, and write x=(x1,…,xN)x=(x_1,…,x_N). Before querying the pool, we evaluate the information atoms and policy weight at a center fitted from historical feedback, yielding J^i J_i and G^e G_e. Let I^c I_c denote the corresponding total trusted information, embedded in the joint parameter space. The fitted information under x is ℐ^(x)=I^c+∑i=1NxiJ^i I(x)= I_c+ _i=1^Nx_i J_i. Let C partition the candidate indices into groups from which at most one comparison may be selected. Without group restrictions, each candidate forms its own group. Let S0S_0 be a fixed feasible seed, possibly empty, used when needed to ensure positive definite fitted information. Its comparisons count toward the budget B. The relaxed feasible set is B=x∈[0,1]N|∑i=1Nxi=B,xi=1for i∈S0,∑i∈cxi≤1for c∈.X_B= \x∈[0,1]^N\; |\; _i=1^Nx_i=B, x_i=1\ for i∈ S_0, _i∈ cx_i≤ 1\ for c \. We solve minx∈B⁡fB(x),fB(x)=12tr⁡[G^eℐ^(x)−1]. _x _Bf_B(x), f_B(x)= 12tr\! [ G_e I(x)^-1 ]. (12) For ξi=xi/B _i=x_i/B and κ=nc/Bκ=n_c/B, this is the fitted finite-pool counterpart of Φκ(ξ)/B _κ(ξ)/B, preserving the total-information scaling of policy risk. We solve the convex relaxation using the Frank–Wolfe algorithm (Jaggi, 2013), followed by feasible largest-weight rounding and improving exchanges. Algorithm 1 in Appendix C.1 gives the complete acquisition procedure, and the same appendix provides a computable optimization certificate for the returned subset. Because rounding need not preserve the relaxed allocation proportions, the statistical guarantee is stated in terms of the information of the selected comparisons themselves. For an asymptotic sequence of selected sets SBS_B containing B distinct comparisons selected before their outcomes are observed, let JBiJ_Bi denote the true information of candidate i at γ0 _0, and let Ic,BexpI^exp_c,B denote the total expected trusted information. Define the normalized information of the selected set by MB=B−1(Ic,Bexp+∑i∈SBJBi)M_B=B^-1\! (I^exp_c,B+ _i∈ S_BJ_Bi ). Let θ^B θ_B be the target component of the projected one-step estimator based on the trusted sample and the selected judge labels. Theorem 4.2 (Policy risk for distinct comparisons). Condition on the historical data, candidate features, and selected set. Under the distinct-comparison regularity conditions in Appendix C.2, suppose MBM_B is uniformly positive definite. Under the local model γ0+η/B _0+η/ B, BηDF(θ0+hB,θ^B)−12tr⁡(GeMB−1)⟶0B\,E_ηD_F\! ( _0+ h B, θ_B )- 12tr(G_eM_B^-1) 0 (13) uniformly over bounded local parameters. If MBM_B converges to a positive definite limit, the corresponding conditional local minimax lower bound also holds. The theorem evaluates the information of the realized subset and therefore retains the effect of rounding. Appendix C.2 gives the proof, while Appendix C.3 gives the plug-in analysis and computational details. 5 Learning the Judge-Deviation Representation In this section, we study the additional error introduced when the judge-deviation representation is learned from historical feedback. This error contributes a design-dependent term to policy risk and can reverse the ordering of acquisition rules. 5.1 Finite-Sample Representation Cost We use the same finite support z1,…,zLz_1,…,z_L and evaluate policy risk at the human target θ0=θ⋆ _0=θ . Let m denote the number of independent historical units used to learn the judge-deviation representation. Write W^m∈ℝL×r W_m ^L× r for the learned representation, with row i given by W^m,i⊤ W_m,i . Let a0m∈ℝra_0m ^r be the nuisance component of the local center γ0m=(θ0⊤,a0m⊤)⊤ _0m=( _0 ,a_0m ) . The nuisance parameter a is re-estimated from the current data. Write logit⁡(u)=log⁡u/(1−u)logit(u)= \u/(1-u)\ for u∈(0,1)u∈(0,1), and define the remaining logit error em=(em,1,…,em,L)⊤e_m=(e_m,1,…,e_m,L) by em,i=logitq(zi)−Xi⊤θ0−W^m,i⊤a0m,i=1,…,L.e_m,i=logitq(z_i)-X_i _0- W_m,i a_0m, i=1,…,L. (14) Let ξm,i=ni/n _m,i=n_i/n denote the judge allocation proportion for type i. Define vm,i=(Xi⊤,W^m,i⊤)⊤v_m,i=(X_i , W_m,i ) and tm,i=σ′(vm,i⊤γ0m)t_m,i=σ (v_m,i _0m). Let MmM_m denote the normalized joint information at γ0m _0m, formed as in equation 7 from the learned representation, the actual judge allocation, and the expected trusted information. Set Φm=12tr⁡(GeMm−1) _m= 12tr(G_eM_m^-1), and let Pθ=[Id 0d×r]P_θ=[I_d\;0_d× r] select the target coordinates. For e=(e1,…,eL)⊤∈ℝLe=(e_1,…,e_L) ^L, define Bme=PθMm−1∑i=1Lξm,itm,ivm,iei,Rm=Bm⊤G0Bm.B_me=P_θM_m^-1 _i=1^L _m,it_m,iv_m,ie_i, R_m=B_m G_0B_m. (15) The first-order effect of eme_m on the target parameter is BmemB_me_m. Let θ^n,m θ_n,m be the target component of the projected one-step estimator based on the learned representation and the current observations. Expectations below are over both the historical and current data. Theorem 5.1 (Policy risk with a learned representation). Under the regularity conditions in Appendix D.2, suppose nc/n→κ>0n_c/n→κ>0 and the smallest eigenvalue of MmM_m is bounded away from zero. Let m,n→∞m,n→∞ with n/m→λP∈[0,∞)n/m→ _P∈[0,∞). Suppose (Φm,Rm)→(Φ0,R0)( _m,R_m)→( _0,R_0) in probability for deterministic limits, and mem m\,e_m converges in distribution to a random vector ZPZ_P with mean bPb_P and covariance ΣP _P. Then ⁡[DF(θ0,θ^n,m)]=Φ0n+CPm+o⁡(n−1+m−1),CP=12tr⁡(R0ΣP)+bP⊤R0bP.E\! [D_F( _0, θ_n,m) ]= _0n+ C_Pm+o(n^-1+m^-1), C_P= 12 \tr(R_0 _P)+b_P R_0b_P \. (16) The first term is the oracle sampling risk and the second is the representation-learning cost, which depends on the direction of the representation error through R0R_0 and includes both covariance and bias contributions. Proofs and additional representation-learning results are given in Appendix D. 5.2 Design Ranking Reversals The representation-cost term in equation 16 can reverse the ordering induced by the oracle sampling risk. We compare NAOD with the matched Target-info design under the same learned representation, label budgets, and final estimator. Let ℒN(n,m)L_N(n,m) and ℒT(n,m)L_T(n,m) denote their expected policy risks. Write ΦN,ΦT _N, _T for their limiting oracle risk constants and CP,N,CP,TC_P,N,C_P,T for their representation-cost coefficients. Applying Theorem 5.1 to the two designs gives n⁡ℒN(n,m)−ℒT(n,m)⟶ΦN−ΦT+λP(CP,N−CP,T).n\L_N(n,m)-L_T(n,m)\ _N- _T+ _P(C_P,N-C_P,T). (17) If ΦN<ΦT _N< _T, the leading-order ordering reverses whenever λP(CP,N−CP,T)>ΦT−ΦN _P(C_P,N-C_P,T)> _T- _N. Thus, NAOD can have lower oracle sampling risk while being more sensitive to representation error. When n/m→0n/m→ 0, the representation-cost term vanishes and the oracle ordering is recovered. When n/m→λP>0n/m→ _P>0, it can contribute at leading order. Thus, NAOD optimizes the fitted design criterion rather than the full two-stage risk. 6 Experiments 6.1 Synthetic Experiments Effective information predicts policy risk. We first consider a scalar three-type Bernoulli construction with a constant nuisance component and equal trusted and judge sample sizes, nc=n_c=n. NAOD, Target-info, and Random use the same correctly specified joint model and projected one-step estimator and differ only in the acquisition design. Each setting is evaluated over 6,000 independent replications. Across n∈240,960,3840,15360n∈\240,960,3840,15360\, the empirical scaled risks in Figure 2(a) closely track their corresponding theoretical information constants, supporting the effective-information characterization of leading policy risk. Figure 2: Controlled tests of the theoretical predictions. Dashed references in panels (a) and (b) denote the corresponding theoretical predictions. Panel (c) reports the NAOD-minus-Target-info scaled-risk difference. Learning the representation adds the predicted risk term. We next use a scalar two-type Bernoulli construction with nc=n_c=n, in which the first-order cost of learning the judge-deviation representation is explicit. For this construction, Φ0=50/49 _0=50/49 and CP=25/98C_P=25/98. Figure 2(b) tests the risk expansion in equation 16 across n∈960,3840,15360n∈\960,3840,15360\ and m/n∈1/4,1,4m/n∈\1/4,1,4\. The empirical scaled risks closely follow the theoretical prediction Φ0+(n/m)CP _0+(n/m)C_P, with the largest finite-sample deviation occurring at the smallest representation-learning sample size. Representation error can reverse design rankings. We then learn the representation in the three-type experiment and compare NAOD with Target-info under otherwise identical conditions, with n=nc=960n=n_c=960. Figure 2(c) shows the paired NAOD-minus-Target-info scaled-risk difference as m/nm/n varies. With limited historical data, the difference is positive and Target-info has lower risk. As the historical sample grows, the gap shrinks, becomes indistinguishable from zero around m/n=1m/n=1, and turns negative by m/n=2m/n=2, where NAOD has lower risk. This confirms the design-ranking reversal mechanism in Section 5.2. Full constructions, finite-sample diagnostics, and additional analyses are reported in Appendices E.2 and E.3. 6.2 Chatbot Arena Evaluation Experimental setup. We evaluate finite-pool acquisition on a frozen Chatbot Arena LLM-judge archive (Chiang et al., 2024) with 49,63549,635 usable comparisons in 42,87542,875 connected text clusters and 17 judges. Current budgets are H∈32,64,128,256,512H∈\32,64,128,256,512\ human labels and B∈256,512,1024B∈\256,512,1024\ judge records, yielding 15 budget combinations. Each combination is evaluated over 15 cluster-level splits. Within each split, representation learning, acquisition, and evaluation use disjoint clusters. At most one comparison is selected from each candidate cluster. All acquisition methods differ only in the acquisition rule. Frozen Qwen3 embeddings (Zhang et al., 2025) are used to construct judge-specific two-dimensional target and nuisance representations, with the full procedure given in Appendix F.1. Representation learning uses 8,5758,575 shared upstream human–judge comparisons, and initialization uses 128 historical comparisons. These shared resources are fixed across acquisition rules and are not counted in the current (H,B)(H,B) budgets. We compare NAOD against five acquisition baselines, Random, Entropy, D-opt, PA D-opt (Shen et al., 2025), and Target-info. Initial and Human-only serve as reference baselines from the common initialization. Judge feedback uses the archived A/B/tie probabilities (Sun et al., 2025a) through the soft target pA+12pTp_A+ 12p_T, where pAp_A and pTp_T are the archived probabilities of an A win and a tie, respectively. For these bounded soft targets, Appendix A.4 gives the corresponding sandwich-risk characterization. Under mean correctness, Theorem A.4 shows that the population curvature criterion underlying NAOD equals the worst-case leading sandwich policy-risk coefficient over bounded soft-feedback laws with the same conditional means. Evaluation. The primary endpoint is proxy policy regret relative to a human-preference reference fitted only from upstream data. We additionally report held-out human cross-entropy and tie-aware human choice accuracy. Test losses are first averaged within connected clusters and then equally across clusters. Within each split, judges and budget combinations are macro-averaged. Paired uncertainty is reported using two-sided 95% t-intervals across the 15 split-level differences. Results. NAOD achieves the lowest mean proxy policy regret and human cross-entropy in Table 1, together with the highest mean human choice accuracy. Relative to the matched Target-info design, NAOD reduces mean proxy policy regret by 29.11%29.11\%, with 14 of 15 split averages favoring NAOD. Relative to D-opt, the reduction is 18.87%18.87\%, with 12 of 15 split averages favoring NAOD. Table 1: Overall Chatbot Arena results. Means over 17 judges, 15 budget regimes, and 15 random cluster-level splits. Regret is reported in 10−310^-3 units, CE in nats per pair, and accuracy in percent. The last two columns report NAOD’s relative regret reduction and paired split wins against each row. Bold denotes the best mean. Method Proxy regret ↓ Human CE ↓ Acc. ↑ Drop (%) Wins Initial 8.86 0.6752 57.82 77.53 13/15 Human-only 18.25 0.6829 56.84 89.09 15/15 Random 4.88 0.6727 58.22 59.18 15/15 Entropy 13.42 0.6784 57.10 85.17 15/15 D-opt 2.45 0.6703 58.19 18.87 12/15 PA D-opt 2.46 0.6703 58.19 18.93 12/15 Target-info 2.81 0.6705 58.21 29.11 14/15 NAOD 1.99 0.6699 58.27 — — Budget dependence. NAOD reduces mean proxy policy regret relative to Target-info in all 15 budget settings. At a fixed human budget H, the relative advantage consistently narrows as the judge budget B increases. Moreover, settings with the same ratio H/BH/B can exhibit substantially different gains, showing that absolute budget size matters in addition to the human-to-judge budget ratio. The complete 5×35× 3 budget grid is reported in Figure 4 in Appendix F.3. Target–nuisance coupling. The acquisition gain varies across judges and tends to be larger when Target-info induces stronger target–nuisance coupling. This pattern is consistent with the theoretical prediction that nuisance-aware acquisition is most valuable when target and nuisance information are strongly coupled. The corresponding judge-level analysis is reported in Figure 5 in Appendix F.3. 7 Conclusion We studied active preference learning with LLM-judge feedback that may systematically deviate from target human preferences. NAOD prioritizes policy-relevant target information after nuisance adjustment. The design criterion has a sharp local minimax characterization, while controlled experiments validate the predicted risk behavior and Arena results show that gains tend to be larger under stronger target–nuisance coupling. Representation error can nevertheless change the preferred acquisition design, motivating future work on jointly learning judge-deviation representations and selecting comparisons. AI use statement Generative AI tools were used to assist with language polishing and to check the algebraic consistency of author-derived mathematical derivations. All AI-assisted suggestions and checks were reviewed and independently verified by the authors. The authors take full responsibility for the final content of the paper. Ethics statement This work uses archived Chatbot Arena preference data and collects no new human-subject data. The available preference feedback and LLM-judge outputs may reflect population- and model-specific biases and may not be universally representative. Preference models and acquisition strategies based on such feedback should therefore be evaluated with respect to the intended target population and deployment setting. Reproducibility statement We provide the code required to reproduce the reported experimental results in the supplementary material. The main text and appendices provide the statistical model, acquisition and estimation procedures, optimization details, and experimental protocols required for reproduction, together with the assumptions and proofs underlying the theoretical results. References Azar et al. (2012) Mohammad Gheshlaghi Azar, Vicenç Gómez, and Hilbert J. Kappen. Dynamic policy programming. Journal of Machine Learning Research, 13(103):3207–3245, 2012. Barker & Kavalieris (2001) Richard J. Barker and Laimonis Kavalieris. Efficiency gain from auxiliary data requiring additional nuisance parameters. Biometrics, 57(2):563–566, 2001. Bradley & Terry (1952) Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3-4):324–345, 1952. Chan et al. (2020) Alex Chan, Ahmed Alaa, Zhaozhi Qian, and Mihaela Van Der Schaar. Unlabelled data improves Bayesian uncertainty calibration under covariate shift. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, p. 1392–1402. PMLR, 2020. Chen et al. (2024) Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or LLMs as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 8301–8327. Association for Computational Linguistics, 2024. Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 8359–8388. PMLR, 2024. Chidambaram & Ge (2025) Muthu Chidambaram and Rong Ge. Reassessing how to compare and improve the calibration of machine learning models. In International Conference on Learning Representations, 2025. Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, p. 4299–4307, 2017. Dorner et al. (2025) Florian E. Dorner, Vivian Y. Nastl, and Moritz Hardt. Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data. In International Conference on Learning Representations, 2025. Farquhar et al. (2021) Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. In International Conference on Learning Representations, 2021. Feng et al. (2025) Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, and Yaqi Duan. PILAF: Optimal human preference sampling for reward modeling. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 16744–16776. PMLR, 2025. Feuer et al. (2025) Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, and John P. Dickerson. Style outweighs substance: Failure modes of LLM judges in alignment benchmarking. In International Conference on Learning Representations, 2025. Fewster & Jupp (2013) R. M. Fewster and P. E. Jupp. Information on parameters of interest decreases under transformations. Journal of Multivariate Analysis, 120:34–39, 2013. Frick et al. (2025) Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. How to evaluate reward models for RLHF. In International Conference on Learning Representations, 2025. Futami & Fujisawa (2024) Futoshi Futami and Masahiro Fujisawa. Information-theoretic generalization analysis for expected calibration error. In Advances in Neural Information Processing Systems, volume 37, p. 84246–84297, 2024. Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, p. 10835–10866. PMLR, 2023. Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, p. 1321–1330. PMLR, 2017. Hu et al. (2024) Xiao Hu, Jianxiong Li, Xianyuan Zhan, Qing-Shan Jia, and Ya-Qin Zhang. Query-policy misalignment in preference-based reinforcement learning. In International Conference on Learning Representations, 2024. Jaggi (2013) Martin Jaggi. Revisiting Frank-Wolfe: Projection-free sparse convex optimization. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, p. 427–435. PMLR, 2013. Kern et al. (2024) Christoph Kern, Michael P. Kim, and Angela Zhou. Multi-accurate CATE is robust to unknown covariate shifts. Transactions on Machine Learning Research, 2024. Kim et al. (2019) Michael P. Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, p. 247–254. ACM, 2019. Lee et al. (2026) Chungpa Lee, Thomas Zeng, Jongwon Jeong, Jy-yong Sohn, and Kangwook Lee. How to correctly report LLM-as-a-judge evaluations. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026. Lee et al. (2024) Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with AI feedback. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 26874–26901. PMLR, 2024. Lee et al. (2021) Kimin Lee, Laura M. Smith, and Pieter Abbeel. PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, p. 6152–6163. PMLR, 2021. Li et al. (2025) Haitao Li, Junjie Chen, Qingyao Ai, Zhumin Chu, Yujia Zhou, Qian Dong, and Yiqun Liu. CalibraEval: Calibrating prediction distribution to mitigate selection bias in LLMs-as-judges. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 16537–16552. Association for Computational Linguistics, 2025. Lin et al. (2026) Xiaoqiang Lin, Arun Verma, Zhongxiang Dai, Daniela Rus, See-Kiong Ng, and Bryan Kian Hsiang Low. ActiveDPO: Active direct preference optimization for sample-efficient alignment. In International Conference on Learning Representations, 2026. Liu et al. (2024) Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. In Advances in Neural Information Processing Systems, volume 37, p. 138663–138697, 2024. Liusie et al. (2024) Adian Liusie, Vatsal Raina, Yassir Fathullah, and Mark Gales. Efficient LLM comparative assessment: A product of experts framework for pairwise comparisons. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 6835–6855. Association for Computational Linguistics, 2024. Maystre & Grossglauser (2017) Lucas Maystre and Matthias Grossglauser. Just sort it! A simple and effective approach to active preference learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, p. 2344–2353. PMLR, 2017. Movva et al. (2026) Rajiv Movva, Smitha Milli, Sewon Min, and Emma Pierson. What’s in my human feedback? learning interpretable descriptions of preference data. In International Conference on Learning Representations, 2026. Mukherjee et al. (2024) Subhojyoti Mukherjee, Anusha Lalitha, Kousha Kalantari, Aniket Deshmukh, Ge Liu, Yifei Ma, and Branislav Kveton. Optimal design for human preference elicitation. In Advances in Neural Information Processing Systems, volume 37, p. 90132–90159, 2024. Muldrew et al. (2024) William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 36577–36590. PMLR, 2024. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, p. 27730–27744, 2022. Ovadia et al. (2019) Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, volume 32, p. 13991–14002, 2019. Park et al. (2024) Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. OffsetBias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1043–1067. Association for Computational Linguistics, 2024. Park et al. (2020) Sangdon Park, Osbert Bastani, James Weimer, and Insup Lee. Calibrated prediction with covariate shift via unsupervised domain adaptation. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, p. 3219–3229. PMLR, 2020. Polo et al. (2025) Felipe Maia Polo, Xinhe Wang, Mikhail Yurochkin, Gongjun Xu, Moulinath Banerjee, and Yuekai Sun. Bridging human and LLM judgments: Understanding and narrowing the gap. In Advances in Neural Information Processing Systems, volume 38, p. 16857–16908, 2025. Pukelsheim (2006) Friedrich Pukelsheim. Optimal Design of Experiments. Society for Industrial and Applied Mathematics, 2006. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, p. 53728–53741, 2023. Shen et al. (2025) Yunyi Shen, Hao Sun, and Jean-Francois Ton. Active reward modeling: Adaptive preference labeling for large language model alignment. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 54410–54430. PMLR, 2025. Sloman et al. (2024) Sabina J. Sloman, Ayush Bharti, Julien Martinelli, and Samuel Kaski. Bayesian active learning in the presence of nuisance parameters. In Proceedings of the Fortieth Conference on Uncertainty in Artificial Intelligence, volume 244 of Proceedings of Machine Learning Research, p. 3245–3263. PMLR, 2024. Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, volume 33, p. 3008–3021, 2020. Sun et al. (2025a) Guangzhi Sun, Anmol Kagrecha, Potsawee Manakul, Phil Woodland, and Mark Gales. SkillAggregation: Reference-free LLM-dependent aggregation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15532–15548. Association for Computational Linguistics, 2025a. Sun et al. (2025b) Zexu Sun, Yiju Guo, Yankai Lin, Xu Chen, Qi Qi, Xing Tang, Xiuqiang He, and Ji-Rong Wen. Uncertainty and influence aware reward model refinement for reinforcement learning from human feedback. In International Conference on Learning Representations, 2025b. Tang et al. (2025) Ming Tang, Yuxuan Zhou, and Chao Huang. Tackling biased evaluators in dueling bandits. In Advances in Neural Information Processing Systems, volume 38, p. 83569–83602, 2025. Thekumparampil et al. (2025) Kiran Koshy Thekumparampil, Gaurush Hiranandani, Kousha Kalantari, Shoham Sabach, and Branislav Kveton. Comparing few to rank many: Active human preference learning using randomized Frank-Wolfe method. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 59355–59376. PMLR, 2025. Wang et al. (2024) Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9440–9450. Association for Computational Linguistics, 2024. Wen et al. (2025) Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In International Conference on Learning Representations, 2025. Wu & Sun (2024) Runzhe Wu and Wen Sun. Making RL with preference-based feedback efficient via randomization. In International Conference on Learning Representations, 2024. Ye et al. (2025) Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM-as-a-judge. In International Conference on Learning Representations, 2025. Zhang et al. (2025) Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36, p. 46595–46623, 2023. Zhou et al. (2024) Han Zhou, Xingchen Wan, Yinhong Liu, Nigel Collier, Ivan Vulić, and Anna Korhonen. Fairer preferences elicit improved human-aligned large language model judgments. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 1241–1252. Association for Computational Linguistics, 2024. Appendix Appendix A Technical Foundations This appendix collects the common statistical and policy-loss facts used throughout the analysis. We specify the conditional experiment and policy loss, establish the local likelihood expansion, characterize efficient target information and policy-null directions, and finally separate likelihood curvature from score covariance for soft feedback. A.1 Notation, Regularity, and Policy Loss Conditioning and notation. Let ℱPF_P denote the historical information available before the current outcomes are observed, including the frozen nuisance representation, comparison features, allocation, and numerical settings. Unless explicitly stated otherwise, probabilities and expectations in the fixed-representation theory are conditional on ℱPF_P. Write γ=(θ⊤,a⊤)⊤∈ℝkγ=(θ ,a ) ^k, with k=d+rk=d+r, and let Pθ=[Id 0]P_θ=[I_d\;\;0] select the target coordinates. A judge observation of comparison type i has augmented feature vi=(Xi⊤,Wi⊤)⊤v_i=(X_i ,W_i ) , while a trusted observation with feature XcjX_cj has padded feature vcj=(Xcj⊤,0⊤)⊤v_cj=(X_cj ,0 ) . There are nin_i judge observations of type i, n=∑i=1Lnin= _i=1^Ln_i judge observations in total, and ncn_c trusted observations. For bounded η=(h⊤,υ⊤)⊤η=(h , ) , the local parameter is γn,η=γ0+η/n _n,η= _0+η/ n, with target component θn,h=θ0+h/n _n,h= _0+h/ n. Regularity. For the fixed-support theory, L,d,rL,d,r are fixed and all feature vectors are uniformly bounded. Conditional on their features, current judge and trusted outcomes are mutually independent Bernoulli variables under the joint model in equation 6. A compact convex product set =θ×aK=K_θ×K_a contains γ0 _0 in its interior, with a bounded open neighborhood available for derivative expansions. The proportions satisfy ni/n→ξin_i/n→ _i and nc/n→κ<∞n_c/n→κ<∞. Uniformly over bounded root-n neighborhoods, the normalized expected information converges to the positive-definite matrix Mκ(ξ)M_κ(ξ) in equation 7. Trusted covariates may be bounded deterministic arrays with the stated information limit or independent bounded random covariates independent of the judge block. The policy value F is twice continuously differentiable near the target region, with bounded uniformly continuous Hessian. For the Gibbs policy considered here, boundedness follows directly from the covariance representation below, while uniform continuity follows on the compact target neighborhood. Bounded features and the compact parameter region also imply a common interior probability bound q⁡(1−q)≥tmin>0q(1-q)≥ t_ >0. Additional conditions required by the concrete preliminary estimator are stated in Appendix B.2. Gibbs policy and directed policy loss. For the Gibbs policy in equation 2 and value function in equation 3, substituting the log probability ratio into the regularized policy objective gives Vθ(π)=F(θ)−τμeKL(π(⋅∣s)∥πθ(⋅∣s)).V_θ(π)=F(θ)-τ\,E_ _eKL\! (π(· s)\,\|\, _θ(· s) ). Thus πθ _θ is the unique optimizer. Differentiating the log partition function yields ∇F(θ)=s∼μey∼πθ(⋅∣s)[ϕ(s,y)],∇2F(θ)=1τs∼μeCovy∼πθ(⋅∣s)[ϕ(s,y)].∇ F(θ)=E_s _eE_y _θ(· s)[φ(s,y)], ∇^2F(θ)= 1τE_s _eCov_y _θ(· s)[φ(s,y)]. In particular, F is convex. If the policy optimized at parameter y is evaluated under reward parameter x, linearity of the reward gives Vx(πy)=F(y)+∇F(y)⊤(x−y)V_x( _y)=F(y)+∇ F(y) (x-y), and therefore F(x)−Vx(πy)=DF(x,y)=τμeKL(πy(⋅∣s)∥πx(⋅∣s)).F(x)-V_x( _y)=D_F(x,y)=τ\,E_ _eKL\! ( _y(· s)\,\|\, _x(· s) ). Hence the first Bregman argument is the reward under which performance is evaluated, whereas the second determines the deployed policy. At the local center, DF(θ0,θ0+h)=12h⊤G0h+o⁡(‖h‖2),G0=∇2F(θ0).D_F( _0, _0+h)= 12h G_0h+o(\|h\|^2), G_0=∇^2F( _0). (18) This gives the policy weighting used by NAOD. A.2 Scores, Information, and Local Asymptotic Normality Let sp⁡(u)=log⁡(1+eu)sp(u)= (1+e^u). For judge labels YiℓY_i and trusted labels HjH_j, the total negative log likelihood is Qn(γ)=∑i=1L∑ℓ=1nisp⁡(vi⊤γ)−Yiℓvi⊤γ+∑j=1ncsp⁡(Xcj⊤θ)−HjXcj⊤θ.Q_n(γ)= _i=1^L _ =1^n_i \sp(v_i γ)-Y_i v_i γ \+ _j=1^n_c \sp(X_cj θ)-H_jX_cj θ \. Define the total score and observed information by Un(γ)=−∇Qn(γ)U_n(γ)=-∇ Q_n(γ) and In(γ)=∇2Qn(γ)I_n(γ)=∇^2Q_n(γ). A judge observation with probability qi(γ)=σ⁡(vi⊤γ)q_i(γ)=σ(v_i γ) contributes (Yiℓ−qi(γ))vi(Y_i -q_i(γ))v_i to the score and qi(γ)1−qi(γ)vivi⊤q_i(γ)\1-q_i(γ)\v_iv_i to the information. Trusted observations have the same form with padded features. Hence QnQ_n is globally convex. At the correctly specified Bernoulli center, Mn:=1nIn(γ0)=1nCov⁡Un(γ0)⟶Mκ(ξ).M_n:= 1nEI_n( _0)= 1nCov\U_n( _0)\ M_κ(ξ). The equality between score covariance and curvature is specific to the correctly specified Bernoulli experiment. Appendix A.4 treats soft feedback separately. Let Δn=Un(γ0)/n _n=U_n( _0)/ n. Bounded independent score summands in fixed dimension give supn‖Δn‖4<∞ _nE\| _n\|^4<∞, while the normalized Hessian differs from its expectation by OL4(n−1/2)O_L^4(n^-1/2) when current trusted covariates are random. For fixed covariate arrays, it is deterministic conditional on the features. Proposition A.1 (Conditional LAN). Under the regularity conditions of Appendix A.1, Δn⇒N⁡(0,Mκ(ξ)), _n N\! (0,M_κ(ξ) ), and, writing ℓn=−Qn _n=-Q_n, for every fixed R<∞R<∞, ℓn(γ0+ηn)−ℓn(γ0)=η⊤Δn−12η⊤Mκ(ξ)η+oP(1) _n\! ( _0+ η n )- _n( _0)=η _n- 12η M_κ(ξ)η+o_P(1) uniformly over ‖η‖≤R\|η\|≤ R. The same derivative and moment bounds hold uniformly along bounded local truth sequences. Proof. Every scalar projection of Δn _n is a triangular array of independent centered bounded terms whose largest summand is O(n−1/2)O(n^-1/2) and whose total variance converges to the corresponding quadratic form of Mκ(ξ)M_κ(ξ). The triangular-array central limit theorem and the Cramér–Wold device give the Gaussian limit. A third-order Taylor expansion of the log likelihood around γ0 _0 gives the displayed quadratic likelihood ratio. Bounded third derivatives make the remainder O(n−1/2)O(n^-1/2) uniformly on each fixed local ball, while normalized information converges uniformly to Mκ(ξ)M_κ(ξ). ∎ A.3 Efficient Target Information and Policy-Null Directions Suppress (κ,ξ)(κ,ξ) and partition the correctly specified Bernoulli information as M=(ACC⊤D)≻0.M= pmatrixA&C\\ C &D pmatrix 0. Let (Sθ,Sa)(S_θ,S_a) denote the target and nuisance components of the limiting LAN score, whose covariance is M. Since D≻0D 0, define Seff=Sθ−CD−1SaS_ eff=S_θ-CD^-1S_a. Then Cov⁡(Seff,Sa)=0,Cov⁡(Seff)=A−CD−1C⊤=Ieff.Cov(S_ eff,S_a)=0, (S_ eff)=A-CD^-1C =I_ eff. Thus the Schur complement removes exactly the target-score component linearly reproducible by nuisance-score variation. Equivalently, for every target direction h, h⊤Ieffh=minb∈ℝr⁡h⊤Ah−2h⊤Cb+b⊤Db,h I_ effh= _b ^r \h Ah-2h Cb+b Db \, with minimizer b=D−1C⊤hb=D^-1C h. Block elimination gives PθM−1Pθ⊤=Ieff−1.P_θM^-1P_θ =I_ eff^-1. (19) Finally, with K=A−1/2CD−1/2K=A^-1/2CD^-1/2, A−1/2IeffA−1/2=I−K⊤.A^-1/2I_ effA^-1/2=I-K . (20) The latter is the whitening identity used by the target–nuisance coupling analysis. At the saturation extreme, if the judge target-score directions lie entirely in the nuisance-score span, judge feedback contributes no additional effective information for the human target. In particular, if Wi=XiW_i=X_i with unrestricted nuisance coefficients, the judge likelihood identifies only θ+aθ+a, and Ieff=κHcI_ eff=κ H_c. Policy-null directions. Theorem 4.1 states the lower bound directly for G0≻0G_0 0. For the Gibbs policy, the reduction required when G0⪰0G_0 0 is exact. Proposition A.2 (Exact policy-null quotient). Let =ker⁡∇2F(θ0)N= ∇^2F( _0). For the finite-response Gibbs policy with full-support reference policy, N is the same at every finite parameter. For every h∈h and finite t, πθ+th(⋅∣s)=πθ(⋅∣s)for μe-almost every s. _θ+th(· s)= _θ(· s) $ _e$-almost every s. In orthogonal coordinates θ=(β,ζ)∈⟂⊕θ=(β,ζ) , the value decomposes as F⁡(β,ζ)=f⁡(β)+c⊤ζF(β,ζ)=f(β)+c ζ, so DF((β,ζ),(β′,ζ′))=Df(β,β′).D_F ((β,ζ),(β ,ζ ) )=D_f(β,β ). Unless the policy loss is identically zero, ∇2f(β0)≻0∇^2f( _0) 0. Proof. For h∈h , the covariance representation of the policy Hessian gives 0=h⊤∇2F(θ0)h=1τμeVarπθ0(⋅∣s)h⊤ϕ(s,y).0=h ∇^2F( _0)h= 1τE_ _eVar_ _ _0(· s)\h φ(s,y)\. Because the Gibbs policy has full support, h⊤ϕ(s,y)h φ(s,y) must be constant over responses for almost every prompt. Adding thth therefore multiplies every unnormalized Gibbs weight at that prompt by the same factor, leaving the normalized policy unchanged and making F affine along h. The same response-constant characterization holds at every finite parameter, so the null space is parameter-independent. The affine component cancels from the Bregman divergence, giving the quotient representation. ∎ Policy-null target coordinates may therefore be treated as additional nuisance coordinates. Applying the positive-curvature argument on ⟂N yields the same policy-weighted trace criterion in the original coordinates. A.4 Soft Feedback and Sandwich Policy Risk For binary evaluation comparisons with equal reference weights and τ=1τ=1, let Fe(θ)=esp⁡(X⊤θ)F_e(θ)=E_esp(X θ) up to an affine term. Under the correctly specified human model, CE⁡(θ)−CE⁡(θ⋆)=DFe(θ,θ⋆),Regret⁡(πθ;θ⋆)=DFe(θ⋆,θ).CE(θ)-CE(θ )=D_F_e(θ,θ ), ( _θ;θ )=D_F_e(θ ,θ). The two losses therefore have opposite Bregman orientations, although both have the same local quadratic expansion 12(θ−θ⋆)⊤∇2Fe(θ⋆)(θ−θ⋆)+o⁡(‖θ−θ⋆‖2). 12(θ-θ ) ∇^2F_e(θ )(θ-θ )+o(\|θ-θ \|^2). This distinction is used in Appendix F.2, where proxy policy regret and held-out human cross-entropy are separate endpoints. Now allow conditionally independent responses Ynj∈[0,1]Y_nj∈[0,1]. The logistic loss remains sp⁡(vnj⊤γ)−Ynjvnj⊤γsp(v_nj γ)-Y_njv_nj γ, with Un(γ)=∑jvnjYnj−qnj(γ),In(γ)=∑jqnj(γ)1−qnj(γ)vnjvnj⊤,U_n(γ)= _jv_nj\Y_nj-q_nj(γ)\, I_n(γ)= _jq_nj(γ)\1-q_nj(γ)\v_njv_nj , where qnj(γ)=σ⁡(vnj⊤γ)q_nj(γ)=σ(v_nj γ). Hence soft and Bernoulli responses share the same logistic curvature, while their score covariances need not agree. Proposition A.3 (Soft-label sandwich policy risk). Suppose interior population roots γn†=(θn†⁣⊤,an†⁣⊤)⊤→γ† _n =( _n ,a_n ) →γ satisfy Un(γn†)=0EU_n( _n )=0, and Mn:=1nIn(γn†)→M≻0,Ωn:=1nCov⁡Un(γn†)→Ω⪰0.M_n:= 1nEI_n( _n )→ M 0, _n:= 1nCov\U_n( _n )\→ 0. Assume a regular estimator satisfies n(γ^n−γn†)=M−1Un(γn†)n+oL2(1). n( γ_n- _n )=M^-1 U_n( _n ) n+o_L^2(1). Partition M=(ACC⊤D)M= ( smallmatrixA&C\\ C &D smallmatrix ), set Ieff=A−CD−1C⊤I_ eff=A-CD^-1C , L=[Id−CD−1]L=[I_d\;\;-CD^-1], and Ωeff=LΩL⊤ _ eff=L L . Then, with G†=∇2F(θ†)G =∇^2F(θ ), nDF(θn†,θ^n)⟶Ψsoft:=12tr⁡(G†Ieff−1ΩeffIeff−1).n\,ED_F( _n , θ_n) _ soft:= 12tr\! (G I_ eff^-1 _ effI_ eff^-1 ). If every row is additionally mean-correct, ⁡[Ynj∣vnj]=qnj(γn†)E[Y_nj v_nj]=q_nj( _n ), then Ω⪯M,Ωeff⪯Ieff,Ψsoft≤12tr⁡(G†Ieff−1). M, _ eff I_ eff, _ soft≤ 12tr\! (G I_ eff^-1 ). For correctly specified Bernoulli feedback, all three inequalities become equalities. Proof. The assumed asymptotic linear expansion gives target covariance PθM−1ΩM−1Pθ⊤P_θM^-1 M^-1P_θ . Since LM=[Ieff 0]LM=[I_ eff\;\;0], block elimination gives PθM−1=Ieff−1LP_θM^-1=I_ eff^-1L, which yields the stated sandwich covariance and, after the local Bregman expansion, the policy-risk limit. Under mean correctness and 0≤Ynj≤10≤ Y_nj≤ 1, Var⁡(Ynj∣vnj)=qnj(1−qnj)−⁡[Ynj(1−Ynj)∣vnj]≤qnj(1−qnj).Var(Y_nj v_nj)=q_nj(1-q_nj)-E[Y_nj(1-Y_nj) v_nj]≤ q_nj(1-q_nj). Conditional independence and summation of the score-covariance contributions give Ω⪯M M. Congruence by L gives Ωeff⪯Ieff _ eff I_ eff and hence the risk bound. For Bernoulli responses, Ynj(1−Ynj)=0Y_nj(1-Y_nj)=0 almost surely, so equality is recovered. ∎ Theorem A.4 (Least-favorable bounded soft feedback). Fix an admissible design ξ∈Ξξ∈ and suppose the conditions of Proposition A.3 hold under mean correctness with limiting root γ†=γ0γ = _0. Let Ψsoft(ξ,P) _ soft(ξ;P) denote the limiting sandwich policy-risk coefficient under a conditionally independent response law P with the same conditional means and responses in [0,1][0,1]. The supremum over all such response laws for which the asymptotic expansion in Proposition A.3 holds is supPΨsoft(ξ,P)=Φκ(ξ). _P _ soft(ξ;P)= _κ(ξ). The supremum is attained by conditionally Bernoulli feedback with the same conditional means. Consequently, arg⁡min⁡supPξ∈Ξ⁡Ψsoft(ξ,P)=arg⁡minξ∈ΞΦκ(ξ). _ξ∈ _P _ soft(ξ;P)= _ξ∈ _κ(ξ). Thus the NAOD criterion minimizes the worst-case leading sandwich policy-risk coefficient over the bounded mean-correct soft-feedback class. Proof. Fix ξ. Under mean correctness, the logistic curvature depends on the conditional means but not on the conditional variances of the responses. Hence all response laws considered in the theorem have the same limiting curvature Mκ(ξ)M_κ(ξ) and the same effective information Ieff,κ(ξ)I_ eff,κ(ξ). For any such response law P, Proposition A.3 gives ΩP⪯Mκ(ξ). _P M_κ(ξ). With L=[Id−CξDξ−1]L=[I_d\;\;-C_ξD_ξ^-1], congruence preserves the Loewner order and therefore Ωeff,P=LΩPL⊤⪯LMκ(ξ)L⊤=Ieff,κ(ξ). _ eff,P=L _PL LM_κ(ξ)L =I_ eff,κ(ξ). Since G0⪰0G_0 0 and Ieff,κ(ξ)≻0I_ eff,κ(ξ) 0, Ψsoft(ξ,P)≤12tr⁡(G0Ieff,κ(ξ)−1)=Φκ(ξ). _ soft(ξ;P)≤ 12tr\! (G_0I_ eff,κ(ξ)^-1 )= _κ(ξ). Now take conditionally Bernoulli responses with the same conditional means. Then Var⁡(Ynj∣vnj)=qnj(γ0)1−qnj(γ0),Var(Y_nj v_nj)=q_nj( _0)\1-q_nj( _0)\, so the score covariance equals the logistic curvature. Hence ΩP=Mκ(ξ) _P=M_κ(ξ) and Ωeff,P=Ieff,κ(ξ) _ eff,P=I_ eff,κ(ξ), which gives Ψsoft(ξ,P)=Φκ(ξ) _ soft(ξ;P)= _κ(ξ). The upper bound is therefore attainable and the first claim follows. Because the equality supPΨsoft(ξ,P)=Φκ(ξ) _P _ soft(ξ;P)= _κ(ξ) holds pointwise for every admissible design, minimizing both sides over Ξ gives the stated equality of minimizer sets. ∎ The Arena experiment in Section 6.2 uses the archived soft target YJ=pA+12pT∈[0,1]Y_J=p_A+ 12p_T∈[0,1]. NAOD therefore uses the same logistic curvature geometry as in the Bernoulli theory without identifying curvature with score covariance. Under mean correctness, Proposition A.3 gives the corresponding sandwich risk, while Theorem A.4 shows that the population curvature criterion underlying NAOD is the worst-case leading sandwich policy-risk coefficient over bounded mean-correct soft feedback. Under mean misspecification, the sandwich result describes fluctuations around the joint population root rather than eliminating displacement of that root from the human target. Appendix B Theory for Nuisance-Adjusted Optimal Design This appendix develops the theory used in Section 3 and Sections 4.1–4.3. We first connect acquisition-dependent residual exposure to target displacement and policy loss, then prove the conditional local minimax characterization and attainment of the NAOD criterion. We finally give the explicit separation from Target-info and formalize the target–nuisance coupling diagnostic used in the empirical analysis. B.1 Acquisition-Dependent Residual Exposure Section 3 defines the probability-scale judge discrepancy δ⁡(z)=q⁡(z)−p⋆(z)δ(z)=q(z)-p (z) and the acquired residual score bξ=ξ[δ⁡(Z)X(Z)]b_ξ=E_ξ[δ(Z)X(Z)]. The change-of-measure identity in equation 5 does not require a parametric model for the judge discrepancy. We now give its consequences for a nuisance-ignorant reward fit. Suppose an interior population root θξ _ξ satisfies ξ[(q−pθξ)X]=0E_ξ[(q-p_ _ξ)X]=0, and write Δ=θξ−θ⋆ = _ξ-θ . Define the integrated statistical and policy curvatures Hξ=∫01ξ[σ′(X⊤(θ⋆+tΔ))XX⊤]t,Gξ=2∫01(1−t)∇2F(θξ−tΔ)t.H_ξ= _0^1E_ξ\! [σ \! (X (θ +t ) )X ]dt, G_ξ=2 _0^1(1-t)∇^2F( _ξ-t )\,dt. Subtracting the population score equations and integrating the logistic derivative along the segment from θ⋆θ to θξ _ξ gives HξΔ=bξH_ξ =b_ξ. Hence, whenever Hξ≻0H_ξ 0, θξ−θ⋆=Hξ−1bξ,DF(θ⋆,θξ)=12bξ⊤Hξ−1GξHξ−1bξ. _ξ-θ =H_ξ^-1b_ξ, D_F(θ , _ξ)= 12\,b_ξ H_ξ^-1G_ξH_ξ^-1b_ξ. Thus policy impact depends on how the residual aligns with reward-score directions under the acquired law rather than on residual magnitude alone. The joint target–nuisance model gives a complementary local interpretation. Holding the human parameter at θ0 _0, define bξ(a)=∑i=1Lξiqθ0,a(zi)−pθ0(zi)Xib_ξ(a)= _i=1^L _i\q_ _0,a(z_i)-p_ _0(z_i)\X_i. Differentiation gives ∂bξ(a)/∂a⊤=Cξ(a)∂ b_ξ(a)/∂ a =C_ξ(a), as in equation 9. Since bξ(0)=0b_ξ(0)=0, the fundamental theorem of calculus yields bξ(a)=∫01Cξ(ta)at,bξ(a0+υn)−bξ(a0)=Cξ(a0)υn+O⁡(n−1).b_ξ(a)= _0^1C_ξ(ta)a\,dt, b_ξ\! (a_0+ n )-b_ξ(a_0)= C_ξ(a_0) n+O(n^-1). Hence the same cross-information that produces the Schur-complement adjustment in equation 8 also controls the local sensitivity of the exposed target score. B.2 Conditional Local Minimax Risk and Attainment We condition throughout on the historical information ℱPF_P, so the representation W, allocation ξ, and all design quantities are fixed before the current outcomes are observed. Write M=Mκ(ξ)M=M_κ(ξ) as in equation 7 and Ge=diag⁡(G0,0)G_e=diag(G_0,0). Preliminary estimation and projected one-step estimator. For attainment, assume κ>0κ>0, a common positive lower bound on normalized trusted likelihood curvature over θK_θ, a common positive lower bound on the weighted nuisance Gram matrix ∑i(ni/n)WiWi⊤ _i(n_i/n)W_iW_i , and a fixed positive distance of the local truths from the boundary of =θ×aK=K_θ×K_a. Let cs>0c_s>0 be smaller than one half of a valid asymptotic lower bound on λmin(M) _ (M). Let θ~n θ_n minimize the trusted negative log likelihood on θK_θ. Holding its target contribution fixed as an offset, let a~n∈arg⁡mina∈a∑i=1L∑ℓ=1nisp⁡(Xi⊤θ~n+Wi⊤a)−Yiℓ(Xi⊤θ~n+Wi⊤a). a_n∈ _a _a _i=1^L _ =1^n_i \sp(X_i θ_n+W_i a)-Y_i (X_i θ_n+W_i a) \. Set γ~n=(θ~n⊤,a~n⊤)⊤ γ_n=( θ_n , a_n ) . If the declared nuisance-curvature check fails, a fixed interior anchor may be used. Under the stated conditions, this event is asymptotically negligible. The guarded one-step update is γn∘=γ~n+In(γ~n)−1Un(γ~n),λminIn(γ~n)/n≥cs,γ~n,otherwise,γ^n=Proj⁡(γn∘), _n = cases γ_n+I_n( γ_n)^-1U_n( γ_n),& _ \I_n( γ_n)/n\≥ c_s,\\ γ_n,&otherwise, cases γ_n=Proj_K( _n ), with target output θ^n=Pθγ^n θ_n=P_θ γ_n. The preliminary and one-step update may use the same current observations. No independence between them is assumed. Strong convexity of the trusted likelihood and the bounded-score moment bound from Appendix A.2 give ∥θ~n−θn,h∥L4=O(n−1/2)\| θ_n- _n,h\|_L^4=O(n^-1/2). At the true nuisance coefficient, the normalized offset score is a centered judge-score average plus a term Lipschitz in θ~n−θn,h θ_n- _n,h. The nuisance Gram lower bound and the common interior probability bound therefore give ∥a~n−an,υ∥L4=O(n−1/2)\| a_n-a_n, \|_L^4=O(n^-1/2), and hence ∥γ~n−γn,η∥L4=O(n−1/2)\| γ_n- _n,η\|_L^4=O(n^-1/2) uniformly on every fixed local parameter ball. To obtain the one-step expansion, put en=γ~n−γn,ηe_n= γ_n- _n,η, H^n=In(γ~n)/n H_n=I_n( γ_n)/n, and H¯n=∫01In(γn,η+ten)/nt H_n= _0^1I_n( _n,η+te_n)/n\,dt. Since the derivative of the score is minus the information, on the guard event the Newton update satisfies the exact identity n(γn∘−γn,η)=H^n−1Un(γn,η)n+H^n−1(H^n−H¯n)nen. n( _n - _n,η)= H_n^-1 U_n( _n,η) n+ H_n^-1( H_n- H_n) n\,e_n. Normalized Hessians are uniformly Lipschitz on the compact neighborhood, so ‖H^n−H¯n‖op≤C‖en‖\| H_n- H_n\|_op≤ C\|e_n\|. The second term is therefore OL2(n−1/2)O_L^2(n^-1/2). Hessian concentration from Appendix A.2, local information convergence, and the fixed spectral guard permit replacing H^n−1 H_n^-1 by M−1M^-1. The guard complement has vanishing squared contribution by the fourth-moment bound, and the fixed boundary margin makes the final projection asymptotically inactive in mean square. Consequently, n(γ^n−γn,η)=M−1Un(γn,η)n+oL2(1) n( γ_n- _n,η)=M^-1 U_n( _n,η) n+o_L^2(1) uniformly on bounded local parameter sets. The same argument will be used in Appendix D.2 around a nearby population root rather than the correctly specified center. More generally, if an interior population root γn† _n has normalized expected Hessian and score covariance converging to the same M≻0M 0, and a preliminary satisfies rn=∥γ~n−γn†∥L4=o(n−1/4)r_n=\| γ_n- _n \|_L^4=o(n^-1/4), then the nonlinear Newton remainder is OL2(nrn2)=o⁡(1)O_L^2( n\,r_n^2)=o(1) and n(γ^n−γn†)=M−1Un(γn†)n+oL2(1). n( γ_n- _n )=M^-1 U_n( _n ) n+o_L^2(1). Conditional local minimax lower bound. We now prove the lower-bound part of Theorem 4.1. Assume first G0≻0G_0 0. Fix ε∈(0,1) ∈(0,1) and a bounded local parameter set. Continuity of the policy Hessian gives a neighborhood of θ0 _0 on which ∇2F⪰(1−ε)G0∇^2F (1- )G_0. Project the rescaled output of an arbitrary reward estimator onto a fixed G0G_0-metric ball containing the target local parameters. Metric projection cannot increase its quadratic distance to any point in this ball. For decisions outside the local neighborhood, convexity together with positive local curvature gives a fixed positive policy loss, whereas the projected quadratic loss remains bounded. Hence, for all sufficiently large n, nDF(θ0+hn,Tn)≥1−ε2‖T¯n−h‖G02,nD_F\! ( _0+ h n,T_n )≥ 1- 2\| T_n-h\|_G_0^2, where T¯n T_n denotes the corresponding localized rescaled decision. Place a smooth product prior on η supported on (−t,t)k(-t,t)^k, with prior Fisher information π2t−2Ikπ^2t^-2I_k. Let En=T¯n−PθηE_n= T_n-P_θη and let SnS_n be the joint score of the conditional likelihood and this local prior. Integration by parts in the local parameter gives ⁡[EnSn⊤]=PθE[E_nS_n ]=P_θ. Writing Jn=⁡[SnSn⊤]J_n=E[S_nS_n ], positive semidefiniteness of ⁡[(En−PθJn−1Sn)(En−PθJn−1Sn)⊤]E\! [(E_n-P_θJ_n^-1S_n)(E_n-P_θJ_n^-1S_n) ] therefore gives ⁡[EnEn⊤]⪰PθJn−1Pθ⊤,Jn=ηℐn(η)+π2t−2Ik,E[E_nE_n ] P_θJ_n^-1P_θ , J_n=E_ηI_n(η)+π^2t^-2I_k, where ℐn(η)I_n(η) is Fisher information for the local coordinate η. The regularity conditions in Appendix A.1 imply that ℐn(η)→MI_n(η)→ M uniformly on every fixed prior support. Taking the trace against G0G_0, using worst-case risk to dominate prior-averaged risk, and then letting n→∞n→∞, t→∞t→∞, and ε↓0 0 yields equation 11, because equation 19 gives PθM−1Pθ⊤=Ieff,κ−1P_θM^-1P_θ =I_eff,κ^-1. When G0⪰0G_0 0 is singular, Proposition A.2 removes the exact policy-null coordinates and the same argument applies on the policy-relevant quotient. Attainment. Under correct specification, Un(γn,η)/nU_n( _n,η)/ n is centered, its covariance converges to M, and its fourth moments are uniformly bounded. Combining the one-step expansion with the local Bregman expansion in equation 18 and the resulting uniform integrability gives nηDF(θn,h,θ^n)⟶12tr⁡(GeM−1)=Φκ(ξ)n\,E_ηD_F( _n,h, θ_n) 12tr(G_eM^-1)= _κ(ξ) uniformly on every fixed local parameter ball. Thus the projected one-step estimator attains the lower-bound constant in Theorem 4.1. B.3 Separation and Target–Nuisance Coupling Proposition B.1 (Separation from Target-info). Let λ>1λ>1, and consider the scalar two-type model with (X1,X2)=(2λ,2)(X_1,X_2)=(2λ,2), (W1,W2)=(2,−2λ)(W_1,W_2)=(2,-2λ), θ0=a0=0 _0=a_0=0, and Hc=(λ2+1)/2H_c=(λ^2+1)/2. For unrestricted allocation with κ>0κ>0 and G0>0G_0>0, Target-info selects ξT=(1,0) _T=(1,0), whereas NAOD selects ξ⋆=(λ,1)/(λ+1)ξ =(λ,1)/(λ+1). Their leading risk ratio is Φκ(ξT)Φκ(ξ⋆)=1+2(λ2+1)κ(λ+1)2>1. _κ( _T) _κ(ξ )=1+ 2(λ^2+1)κ(λ+1)^2>1. Proof. At the zero center, every judge variance is 1/41/4. If x∈[0,1]x∈[0,1] denotes the fraction of judge labels assigned to the first type, the two judge information atoms are J1=(λ2λ1),J2=(1−λ−λ2).J_1= pmatrixλ^2&λ\\ λ&1 pmatrix, J_2= pmatrix1&-λ\\ -λ&λ^2 pmatrix. The raw target information is κHc+xλ2+(1−x)κ H_c+xλ^2+(1-x), which is strictly increasing in x because λ>1λ>1. Hence Target-info selects x=1x=1. For the judge block J⁡(x)=xJ1+(1−x)J2J(x)=xJ_1+(1-x)J_2, the nuisance entry is D⁡(x)=x+λ2(1−x)D(x)=x+λ^2(1-x) and detJ⁡(x)=x⁡(1−x)(λ2+1)2 J(x)=x(1-x)(λ^2+1)^2. Because trusted information enters only the target block, the effective target information is Ieff(x)=κHc+x⁡(1−x)(λ2+1)2x+λ2(1−x).I_eff(x)=κ H_c+ x(1-x)(λ^2+1)^2x+λ^2(1-x). The derivative of the second term has roots λ/(λ+1)λ/(λ+1) and λ/(λ−1)λ/(λ-1). Only the first lies in [0,1][0,1], and the derivative changes from positive to negative there. Thus NAOD selects x⋆=λ/(λ+1)x =λ/(λ+1). At this allocation, Ieff(x⋆)=κHc+(λ2+1)2/(λ+1)2I_eff(x )=κ H_c+(λ^2+1)^2/(λ+1)^2, whereas Ieff(1)=κHcI_eff(1)=κ H_c. Since the scalar leading policy risk is G0/2Ieff(x)G_0/\2I_eff(x)\ by equation 10, substituting Hc=(λ2+1)/2H_c=(λ^2+1)/2 gives the stated ratio. ∎ This construction isolates the acquisition criterion because both rules use the same correctly specified joint model and final estimator, while Target-info favors the type with the largest raw target block even when that type cannot separate target from nuisance variation. Target–nuisance coupling. For any positive-definite design, define Kξ=Aκ(ξ)−1/2CξDξ−1/2,ρ2(ξ)=∥Kξ∥op2.K_ξ=A_κ(ξ)^-1/2C_ξD_ξ^-1/2, ρ^2(ξ)=\|K_ξ\|_op^2. Positive definiteness of the Schur complement implies 0≤ρ2(ξ)<10≤ρ^2(ξ)<1. By the whitening identity in equation 20, if ρj2 _j^2 are the eigenvalues of KξKξ⊤K_ξK_ξ , then 1−ρj21- _j^2 are the corresponding fractions of whitened target information remaining after nuisance adjustment. Thus ρ2(ξ)ρ^2(ξ) records the strongest target–nuisance coupling. Values near one indicate a target-score direction that is nearly reproducible by nuisance variation and for which raw Target-info can substantially overstate usable target information. This coupling measure is geometric rather than policy-weighted, so its consequence for policy risk also depends on whether the coupled target directions receive substantial weight under G0G_0. The Arena analysis in Figure 5 and Table 12 evaluates ρ2ρ^2 for the Target-info-selected design and uses it as a diagnostic of target–nuisance confounding rather than as a finite-sample prediction of the realized NAOD gain. Appendix C Finite-Pool Selection and Computation This appendix gives the optimization, statistical, and computational details for the finite-pool procedure in Section 4.4. We first justify the convex relaxation and its executable optimality certificate, then prove Theorem 4.2 directly for the realized set of distinct comparisons. We finally separate plug-in error from numerical optimization error and record the computational cost of the implementation. C.1 Finite-Pool Optimization and Certificates For the fitted finite-pool criterion in equation 12, write ℐ^x=ℐ^(x) I_x= I(x). The seed S0S_0 is chosen only when needed to keep fitted information positive definite. Because all information atoms are positive semidefinite, positive definiteness of the trusted-plus-seed information then implies ℐ^x≻0 I_x 0 for every x∈Bx _B. The map M↦12tr⁡(GM−1)M 12tr(GM^-1) is convex for G⪰0G 0 on the positive-definite cone. Indeed, for a symmetric perturbation H, d2dt212tr⁡G(M+tH)−1|t=0=tr⁡(GM−1HM−1HM−1)≥0. d^2dt^2 . 12tr\! \G(M+tH)^-1 \ |_t=0=tr\! (GM^-1HM^-1HM^-1 )≥ 0. Since ℐ^(x) I(x) is affine in x, fBf_B is convex on BX_B. Differentiation gives the gradient used by the linear oracle. For a feasible relaxed point x, define the full Frank–Wolfe gap gFW(x)=maxu∈B⁡∇fB(x)⊤(x−u).g_FW(x)= _u _B∇ f_B(x) (x-u). Convexity gives 0≤fB(x)−minu∈B⁡fB(u)≤gFW(x)0≤ f_B(x)- _u _Bf_B(u)≤ g_FW(x). The word full is important because the linear oracle must optimize over the complete declared feasible set. Under the group constraints in Section 4.4, this oracle is exact. After fixing the seed coordinates, it selects the smallest gradient coefficient in each available group and then the required number of groups with the smallest such coefficients. Let x x be the final relaxed iterate and let S be the returned feasible subset after rounding and exchanges. Write fB(S)=fB(S)f_B(S)=f_B(1_S), and let fB,int⋆f _B,int denote the minimum over feasible integer selections. Since every integer design is feasible for the relaxation, 0≤fB(S)−fB,int⋆≤fB(S)−fB(x^)+gFW(x^).0≤ f_B(S)-f _B,int≤ f_B(S)-f_B( x)+g_FW( x). (21) Thus the final Frank–Wolfe gap and the actual rounding-and-exchange change give a computable certificate for the returned subset under the frozen fitted criterion. This is an optimization certificate, not a statistical confidence interval and not a certificate that the fitted information equals population information. The complete optimization and integerization procedure is summarized in Algorithm 1. Algorithm 1 NAOD finite-pool acquisition 1: Input Fitted information atoms J^ii=1N\ J_i\_i=1^N, policy weight G^e G_e, trusted information I^c I_c, budget B, feasible set BX_B, seed S0S_0, tolerance ε>0 >0, and iteration limit TmaxT_ . 2: Choose x∈Bx _B with ℐ^(x)≻0 I(x) 0. 3: for t=1,…,Tmaxt=1,…,T_ do 4: Compute st∈argminu∈B∇fB(x)⊤us_t∈ *arg\,min_u _B∇ f_B(x) u. 5: if ∇fB(x)⊤(x−st)≤ε∇ f_B(x) (x-s_t)≤ then 6: break 7: end if 8: Choose αt∈argminα∈[0,1]⁡fB(x+α⁡(st−x)) _t∈ *arg\,min_α∈[0,1]f_B(x+α(s_t-x)). 9: Update x←x+αt(st−x)x← x+ _t(s_t-x). 10: end for 11: Set x^←x x← x and recompute gFW(x^)g_FW( x). 12: Round x x by largest weights to a feasible set S of size B, including S0S_0 and respecting all group capacities. 13: Apply feasible exchanges that decrease fB(S)f_B(S) and preserve positive-definite fitted information. 14: Freeze S before observing its judge labels. 15: return S and the certificate in equation 21. The final gap is recomputed regardless of whether optimization stops by tolerance or by the iteration limit. Reaching the limit is not treated as convergence. Improving exchanges can only reduce the certificate because equation 21 is evaluated at the final returned subset. The selected IDs and their fitted information are frozen before any selected judge outcome is revealed. C.2 Statistical Guarantees for Distinct Comparisons We now prove Theorem 4.2. Unlike the fixed-support analysis in Appendix B.2, the selected comparisons need not repeat a finite set of feature types. The proof therefore retains each selected row and works with the realized information matrix MBM_B appearing in the theorem. Distinct-comparison regularity. Condition on the historical data, candidate features, and the selected set SBS_B, all fixed before the selected outcomes are observed. The dimensions d and r are fixed. Selected and trusted feature rows are uniformly bounded. Current labels are conditionally independent and correctly specified under the joint Bernoulli model. The parameter centers lie a fixed positive distance from the boundary of a common compact product set. Assume nc,B/Bn_c,B/B is bounded above and away from zero, trusted likelihood curvature has a common positive lower bound, and λmin(MB)≥μ>0 _ (M_B)≥μ>0. The guards, projection region, and derivative bounds are the same as in Appendix B.2. For γB,η=γ0+η/B _B,η= _0+η/ B, let MB(η)M_B(η) be the expected joint information divided by B at the local truth. Bounded rows and bounded logistic derivatives imply, for every fixed R<∞R<∞, sup‖η‖≤R∥MB(η)−MB∥op=OR(B−1/2). _\|η\|≤ R\|M_B(η)-M_B\|_op=O_R(B^-1/2). No convergence of MBM_B is needed for this comparison. Let UBU_B denote the total joint score from the trusted block and the selected judge records, and define ΔB,η=UB(γB,η)/B _B,η=U_B( _B,η)/ B. There are O⁡(B)O(B) independent bounded score summands, so ΔB,η _B,η has uniformly bounded fourth moments on bounded local parameter sets, and the normalized observed information has L4L^4 fluctuation O(B−1/2)O(B^-1/2) when trusted covariates are random. Trusted curvature gives the target preliminary an OL4(B−1/2)O_L^4(B^-1/2) error. Moreover, the nuisance principal block of MBM_B is at least μIrμ I_r. Since logistic variances are at most 1/41/4, B−1∑i∈SBWBiWBi⊤⪰4μIrB^-1 _i∈ S_BW_BiW_Bi 4μ I_r, which supplies the strong convexity required by the offset nuisance fit. The full preliminary is therefore root-B in L4L^4. Applying the exact Newton expansion from Appendix B.2 with the finite-array information retained gives B(γ^B−γB,η)=MB(η)−1ΔB,η+oL2(1) B( γ_B- _B,η)=M_B(η)^-1 _B,η+o_L^2(1) uniformly over bounded η and eligible frozen selected arrays. The guard complement and final projection are negligible by the common information floor, fourth-moment bounds, and boundary margin. Under correct specification, Cov⁡(ΔB,η)=MB(η)Cov( _B,η)=M_B(η). Hence the leading expected policy quadratic form is 12tr⁡GeMB(η)−1 12tr\G_eM_B(η)^-1\. The preceding OR(B−1/2)O_R(B^-1/2) information comparison, the inverse-information bound, and the local Bregman expansion in equation 18 then yield exactly equation 13, uniformly over bounded local parameters. This proves the risk statement in Theorem 4.2 without requiring the selected empirical proportions to approach a fixed-support allocation. If MB→M∞≻0M_B→ M_∞ 0, the bounded triangular-array central limit theorem and the same third-order likelihood expansion as in Proposition A.1 give a LAN experiment with information M∞M_∞. The localized lower-bound argument of Appendix B.2 therefore gives the corresponding conditional local minimax bound. If G0G_0 is singular, Proposition A.2 applies on the policy-relevant quotient. The theorem is conditional on the realized selected set and uses its true information. It therefore preserves the statistical effect of rounding, but it does not assert that every rounded set satisfies the information floor, nor does fitted positive definiteness imply true positive definiteness. Those are separate plug-in questions addressed next. C.3 Plug-In Error, Numerical Accuracy, and Computation Plug-in error. Define the population counterpart by fB0(x)=12tr⁡Geℐ0(x)−1f_B^0(x)= 12tr\G_eI_0(x)^-1\, where ℐ0(x)=Ic+∑i=1NxiJiI_0(x)=I_c+ _i=1^Nx_iJ_i, and recall that fBf_B in equation 12 uses the fitted quantities G^e G_e and ℐ^(x) I(x). Suppose both information matrices have eigenvalues at least μ¯B>0 μ_B>0 over the feasible family. The inverse identity and the trace inequality give |fB(x)−fB0(x)|≤‖G^e−Ge‖∗2μ¯B+tr⁡(Ge)2μ¯B2‖ℐ^(x)−ℐ0(x)‖op.|f_B(x)-f_B^0(x)|≤ \| G_e-G_e\|_*2 μ_B+ tr(G_e)2 μ_B^2\| I(x)-I_0(x)\|_op. For example, bounds ‖I^c−Ic‖op≤δc\| I_c-I_c\|_op≤ _c and maxi⁡‖J^i−Ji‖op≤δJ _i\| J_i-J_i\|_op≤ _J imply supx∈B‖ℐ^(x)−ℐ0(x)‖op≤δc+BδJ _x _B\| I(x)-I_0(x)\|_op≤ _c+B _J because every feasible design has total mass B. Let ΔB=supx∈B|fB(x)−fB0(x)| _B= _x _B|f_B(x)-f_B^0(x)|. If SB0,⋆S_B^0, minimizes the population criterion over feasible integer selections, then the fitted optimization certificate and two applications of the uniform plug-in bound give 0≤fB0(S)−fB0(SB0,⋆)≤2ΔB+fB(S)−fB(x^)+gFW(x^).0≤ f_B^0(S)-f_B^0(S_B^0, )≤ 2 _B+f_B(S)-f_B( x)+g_FW( x). Thus plug-in error and numerical/rounding error enter separately. A small Frank–Wolfe certificate establishes accurate optimization of the fitted objective. It does not by itself establish that the fitted objective accurately represents the population criterion or that the joint model is correctly specified. Numerical accuracy of the estimator. The statistical proofs in Appendix B.2 are written for exact trusted and offset preliminary minimizers, but approximate convex solutions preserve the required rates under a quantitative stopping rule. For a normalized convex objective Q with Hessian bounded below by cIcI, let t⋆t be its constrained minimizer and define the global first-order gap gQ(t)=maxu∈⁡∇Q(t)⊤(t−u)g_Q(t)= _u ∇ Q(t) (t-u). Strong convexity and convexity give c2‖t−t⋆‖2≤Q⁡(t)−Q⁡(t⋆)≤gQ(t). c2\|t-t \|^2≤ Q(t)-Q(t )≤ g_Q(t). Hence gaps of order O⁡(nc−1)O(n_c^-1) for the normalized trusted objective and O⁡(B−1)O(B^-1) for the normalized nuisance objective yield numerical errors of order O(nc−1/2)O(n_c^-1/2) and O(B−1/2)O(B^-1/2), respectively, or smaller, leaving the one-step expansions unchanged. Selection line searches need not themselves prove convergence because the final Frank–Wolfe gap is recomputed at the returned relaxed point and is the quantity entering equation 21. Computation. For k=d+rk=d+r, each fitted information atom has rank-one form J^i=t^iv^iv^i⊤ J_i= t_i v_i v_i . Storing (t^i,v^i)( t_i, v_i) rather than a dense k×k× k matrix for every candidate requires O⁡(Nk+k2)O(Nk+k^2) memory. After factorizing ℐ^x I_x, the gradient coordinate can be evaluated as ∂fB(x)∂xi=−12t^iv^i⊤ℐ^x−1G^eℐ^x−1v^i. ∂ f_B(x)∂ x_i=- 12 t_i\, v_i I_x^-1 G_e I_x^-1 v_i. A dense full information-and-gradient evaluation costs O⁡(Nk2+k3)O(Nk^2+k^3), with the k3k^3 term coming from the factorization. The implementation uses positive-definite factorizations and linear solves rather than forming matrix inverses explicitly. Total selection time additionally depends on the number of Frank–Wolfe iterations, line-search evaluations, rounding, and improving exchanges. Appendix D Learning the Judge-Deviation Representation This appendix proves the representation-learning results in Sections 5.1 and 5.2. We first characterize the target effect of representation error and the exact cancellation of modeled nuisance directions, then prove the finite-sample representation cost in Theorem 5.1 and its design-ranking consequence. We finally give the higher-order population-root expansion that clarifies the limit of first-order cancellation. D.1 Representation Error and Oracle Recovery Recall the learned-representation residual eme_m in equation 14 and the target influence operator BmB_m and policy kernel RmR_m in equation 15. We first justify the first-order interpretation of these quantities. Proposition D.1 (Local representation-error expansion). Fix the learned representation, nuisance center, and allocation, and suppose the regularity conditions of Appendices A.1 and B.2 hold with positive joint information MmM_m. If the true judge logits equal the nominal logits at γ0m _0m plus a sufficiently small deterministic vector e∈ℝLe ^L, while the trusted model remains correctly specified at θ0 _0, then the fitted population score has a unique local root γm†(e) _m (e) satisfying γm†(e)−γ0m=Mm−1∑i=1Lξm,itm,ivm,iei+O⁡(‖e‖2). _m (e)- _0m=M_m^-1 _i=1^L _m,it_m,iv_m,ie_i+O(\|e\|^2). Consequently, Pθγm†(e)−γ0m=Bme+O⁡(‖e‖2).P_θ\ _m (e)- _0m\=B_me+O(\|e\|^2). Proof. Let Ψm(γ,e) _m(γ,e) denote the normalized expected fitted score when the true judge logit at type i is vm,i⊤γ0m+eiv_m,i _0m+e_i, while the trusted probabilities remain those of θ0 _0. At (γ0m,0)( _0m,0), the score is zero, its derivative with respect to γ is −Mm-M_m, and its derivative with respect to e applied to a direction e is ∑iξm,itm,ivm,iei _i _m,it_m,iv_m,ie_i. The inverse-information bound and bounded logistic derivatives give a local implicit-function expansion with the stated linear term and an O⁡(‖e‖2)O(\|e\|^2) remainder. Taking target coordinates and using the definition of BmB_m in equation 15 gives the second display. ∎ The operator BmB_m annihilates representation errors that lie in the learned nuisance span. In particular, for any c∈ℝrc ^r, BmW^mc=PθMm−1∑i=1Lξm,itm,ivm,iW^m,i⊤c=PθMm−1Mm(0c)=0,B_m W_mc=P_θM_m^-1 _i=1^L _m,it_m,iv_m,i W_m,i c=P_θM_m^-1M_m 0c=0, so BmW^m=0B_m W_m=0. Thus an error component in the column span of W^m W_m has no first-order effect on the target parameter. More strongly, if e=W^mce= W_mc and a0m+ca_0m+c remains in the admissible interior neighborhood, then γ0m+(0⊤,c⊤)⊤ _0m+(0 ,c ) reproduces every perturbed judge logit exactly while leaving the trusted model unchanged. Such a represented error is therefore absorbed entirely by the nuisance coefficient, although estimating that coefficient still reduces effective target information through equation 8. Proposition D.1 also gives a generic oracle-recovery condition. Along a deterministic sequence of learned representations and designs, if ∥em∥=o(n−1/4)\|e_m\|=o(n^-1/4) and n‖Bmem‖→0 n\,\|B_me_m\|→ 0, then the target population-root displacement is o(n−1/2)o(n^-1/2). The nearby-root one-step expansion in Appendix B.2 then recovers the oracle leading policy risk. For random representation errors, expected-risk recovery additionally requires the moment control used below. D.2 Finite-Sample Representation Cost and Design Reversals Representation-learning regularity. Let ℱmF_m denote the historical information used to learn the judge-deviation representation. Conditional on ℱmF_m, the aligned representation W^m W_m, nuisance center a0ma_0m, and realized allocation proportions ξm,i _m,i are fixed before the current outcomes are observed. We assume fixed support and parameter dimensions, common bounded feature and logit ranges, a common compact product parameter set whose nominal centers γ0m=(θ0⊤,a0m⊤)⊤ _0m=( _0 ,a_0m ) remain a fixed positive distance from the boundary, and the trusted-curvature, nuisance-curvature, guard, and projection conditions of Appendix B.2 with common constants. Conditional on ℱmF_m, current judge outcomes are independent Bernoulli variables with their actual probabilities and are independent of a correctly specified trusted block. The trusted-to-judge sample ratio is bounded above and away from zero, and Mm⪰μIM_m μ I for some fixed μ>0μ>0. In addition to the convergence assumptions in Theorem 5.1, assume supm‖mem‖4<∞ _mE\| m\,e_m\|^4<∞. Proof of Theorem 5.1. Condition on ℱmF_m. On the event that ‖em‖\|e_m\| lies in the common local neighborhood, Proposition D.1 gives a population root γm† _m satisfying γm†−γ0m=Mm−1∑i=1Lξm,itm,ivm,iem,i+O⁡(‖em‖2). _m - _0m=M_m^-1 _i=1^L _m,it_m,iv_m,ie_m,i+O(\|e_m\|^2). The fourth-moment assumption gives ℙ⁡(‖em‖>ε)=O⁡(m−2)P(\|e_m\|> )=O(m^-2) for every fixed sufficiently small ε>0 >0. Since n/mn/m is bounded and policy loss is bounded on the common compact parameter set, these exceptional historical samples contribute o⁡(1)o(1) to the expected policy loss after multiplication by n. On the local event, let Δm=Un(γm†)n,Am†=1n[In(γm†)∣ℱm]. _m= U_n( _m ) n, A_m = 1nE[I_n( _m ) _m]. Because γm† _m is the conditional population root, ⁡[Δm∣ℱm]=0E[ _m _m]=0. Bounded logistic derivatives and Proposition D.1 imply Am†=Mm+O⁡(‖em‖)A_m =M_m+O(\|e_m\|) and Cov⁡(Δm∣ℱm)=Mm+O⁡(‖em‖)Cov( _m _m)=M_m+O(\|e_m\|). The trusted-plus-offset preliminary has L4L^4 distance O(n−1/2+m−1/2)O(n^-1/2+m^-1/2) from γm† _m . Since n/mn/m is bounded, the Newton argument of Appendix B.2, retaining the conditional matrix Am†A_m , gives n(γ^n,m−γm†)=(Am†)−1Δm+oL2(1). n( γ_n,m- _m )=(A_m )^-1 _m+o_L^2(1). Combining this expansion with the population-root expansion and taking target coordinates yields n(θ^n,m−θ0)=Pθ(Am†)−1Δm+nBmem+oL2(1). n( θ_n,m- _0)=P_θ(A_m )^-1 _m+ n\,B_me_m+o_L^2(1). Indeed, the omitted quadratic root remainder is negligible because n‖em‖L42=O⁡(n/m)=o⁡(1) n\,\|e_m\|_L^4^2=O( n/m)=o(1). The first leading term has conditional mean zero given ℱmF_m, whereas BmemB_me_m is ℱmF_m-measurable, so their cross second moment vanishes. Using the local policy expansion in equation 18, the common Hessian bound, and the fourth-moment controls therefore gives the finite-sequence identity nDF(θ0,θ^n,m)=Φm+n2[em⊤Rmem]+o⁡(1).n\,ED_F( _0, θ_n,m)=E _m+ n2\,E[e_m R_me_m]+o(1). This identity separates current-sample variation from representation-induced displacement before taking the historical-sample limit. Write Zm=memZ_m= m\,e_m. The information floor uniformly bounds Φm _m and ‖Rm‖op\|R_m\|_op, so Φm→Φ0 _m→ _0 in probability implies Φm→Φ0E _m→ _0. Moreover, Zm⇒ZPZ_m Z_P and Rm→R0R_m→ R_0 in probability imply Zm⊤RmZm⇒ZP⊤R0ZPZ_m R_mZ_m Z_P R_0Z_P. The uniform fourth-moment bound makes these quadratic forms uniformly integrable, and hence ⁡[Zm⊤RmZm]⟶⁡[ZP⊤R0ZP]=tr⁡(R0ΣP)+bP⊤R0bP.E[Z_m R_mZ_m] [Z_P R_0Z_P]=tr(R_0 _P)+b_P R_0b_P. Since n/m→λPn/m→ _P, substitution into the preceding finite-sequence identity and division by n give exactly the risk expansion in equation 16. ∎ The same expansion gives the ranking result in Section 5.2. Applying equation 16 to NAOD and Target-info under the same representation-learning protocol, label budgets, and final estimator, and subtracting the two risks, gives equation 17. Thus the oracle ordering is recovered when n/m→0n/m→ 0, whereas for n/m→λP>0n/m→ _P>0 the design-dependent representation cost can contribute at the same leading order as the oracle sampling risk. D.3 Higher-Order Representation Effects First-order influence does not by itself characterize finite representation error. We therefore record the second-order population-root expansion that determines when cancellation of BmemB_me_m is sufficient for oracle recovery. For a fixed regular design, let V=[v1,…,vL]V=[v_1,…,v_L], Λξ=diag⁡(ξ1,…,ξL) _ξ=diag( _1,…, _L), T=diag⁡(t1,…,tL)T=diag(t_1,…,t_L), and define ℒξ=M−1VΛξTL_ξ=M^-1V _ξT. For vc=(Xc⊤,0⊤)⊤v_c=(X_c ,0 ) , define QJ(e,f) Q_J(e,f) =∑i=1Lξiσ′(vi⊤γ0)vieifi, = _i=1^L _iσ (v_i _0)v_ie_if_i, QM(h,g) Q_M(h,g) =∑i=1Lξiσ′(vi⊤γ0)vi(vi⊤h)(vi⊤g)+κ[σ′(Xc⊤θ0)vc(vc⊤h)(vc⊤g)]. = _i=1^L _iσ (v_i _0)v_i(v_i h)(v_i g)+κ\,E\! [σ (X_c _0)v_c(v_c h)(v_c g) ]. Proposition D.2 (Second-order population-root expansion). Suppose the true judge logits are vi⊤γ0+eiv_i _0+e_i, while the trusted model remains correctly specified at θ0 _0. For sufficiently small e, the nearby fitted population root satisfies γ†(e)−γ0=s1(e)+s2(e)+O⁡(‖e‖3),γ (e)- _0=s_1(e)+s_2(e)+O(\|e\|^3), s2(e)=12M−1QJ(e,e)−QM(s1(e),s1(e)),s_2(e)= 12M^-1\! \Q_J(e,e)-Q_M(s_1(e),s_1(e)) \, where s1(e)=ℒξes_1(e)=L_ξe. If e=Wbe=Wb and a0+ba_0+b remains in the admissible interior neighborhood, then γ†(e)=γ0+(0⊤,b⊤)⊤γ (e)= _0+(0 ,b ) exactly, so the target displacement vanishes to every order and in particular s2(Wb)=0s_2(Wb)=0. Proof. Let ℓi=vi⊤γ0 _i=v_i _0. The normalized population score at γ0+h _0+h under logit perturbation e is Ψ⁡(h,e)=∑i=1Lξiσ⁡(ℓi+ei)−σ⁡(ℓi+vi⊤h)vi+κ[σ⁡(Xc⊤θ0)−σ⁡(Xc⊤θ0+vc⊤h)vc]. (h,e)= _i=1^L _i\σ( _i+e_i)-σ( _i+v_i h)\v_i+κ\,E\! [\σ(X_c _0)-σ(X_c _0+v_c h)\v_c ]. At (h,e)=(0,0)(h,e)=(0,0), the derivatives with respect to h and e are −M-M and VΛξTV _ξT, respectively, so the local root satisfies h=O⁡(‖e‖)h=O(\|e\|). A second-order Taylor expansion of the true-mean and fitted-mean terms gives 0=VΛξTe+12QJ(e,e)−Mh−12QM(h,h)+O⁡(‖e‖3+‖h‖3).0=V _ξTe+ 12Q_J(e,e)-Mh- 12Q_M(h,h)+O(\|e\|^3+\|h\|^3). The first-order relation gives h−s1(e)=O⁡(‖e‖2)h-s_1(e)=O(\|e\|^2). Bilinearity of QMQ_M then allows QM(h,h)Q_M(h,h) to be replaced by QM(s1,s1)Q_M(s_1,s_1) at third-order error, yielding the stated expansion. If e=Wbe=Wb, shifting only the nuisance coefficient reproduces the perturbed judge logits exactly and leaves the trusted probabilities unchanged, establishing exact cancellation. ∎ Proposition D.2 explains why first-order cancellation alone is insufficient at the n−1/4n^-1/4 scale. Along a sequence of learned representations and designs, a sufficient generic condition for representation error to be negligible at the oracle root-n scale is n∥Bmem∥→0,∥em∥=o(n−1/4), n\,\|B_me_m\|→ 0, \|e_m\|=o(n^-1/4), together with the corresponding moment control when eme_m is random. Errors lying exactly in the learned nuisance span are different because they are absorbed by recentering the nuisance coefficient and are not subject to this generic first-order remainder condition. Appendix E Synthetic Experiments This appendix provides the constructions and diagnostics underlying the synthetic results in Section 6.1. Section E.1 specifies the common protocol, Section E.2 gives the complete numerical evidence behind Figure 2, and Section E.3 tests mechanism boundaries not needed in the main text. E.1 Experimental Protocol Common experimental setup. Unless an intervention explicitly changes one component, all controlled studies use independent Bernoulli observations from the canonical joint-logit model in equation 6, with the same target and nuisance representations, parameter region, and final estimator across acquisition rules. Oracle studies condition on a fixed nuisance representation. Representation-learning studies first generate independent historical human–judge data, freeze the learned representation and acquisition rule, and then generate independent current trusted and judge observations. Thus acquisition comparisons change the information collected rather than the outcome-generating law or final estimation procedure. Estimator and numerical implementation. The main studies use the projected one-step estimator of Appendix B.2. Trusted logistic regression supplies the target preliminary, an offset nuisance fit supplies the nuisance preliminary, and one guarded joint update is followed by projection. Numerical search and safeguard settings are summarized in Table 2. Numerical damping is used only to stabilize optimization and is never counted as statistical information. Fixed-support theoretical criteria are evaluated using the realized integer counts after rounding. The distinct-comparison experiment instead uses the finite-pool procedure of Algorithm 1 and the certificate in equation 21. Table 2: Common controlled-study protocol. Experiment-specific constructions and sample sizes are given in Sections E.2–E.3. Component Convention Current outcomes Independent Bernoulli trusted and judge labels Final estimator Trusted preliminary + offset nuisance preliminary + one joint update Target / nuisance radii 22 and 33, respectively, unless stated otherwise Preliminary gap tolerance 10−610^-6 divided by fit sample size Nuisance-Gram threshold 10−1010^-10 Joint-information guard 10−510^-5 Numerical damping 10−12I10^-12I, search only Final likelihood Unpenalized Fixed-support design solver SLSQP, analytic gradient, tolerance 10−1210^-12, cap 600600 Finite-pool procedure Algorithm 1 framework, iteration cap 180180 Primary endpoint Scaled policy loss nDFnD_F, unless stated otherwise Replication units and uncertainty. For fixed-support studies, one independently generated Bernoulli dataset is one Monte Carlo replication, and MC SE is the sample standard deviation across replications divided by the square root of their number. In nested representation-learning studies, current losses are first averaged within each independent historical realization and uncertainty is computed across those historical realizations. In the distinct-pool study, current losses are first averaged within each independently generated pool and uncertainty is computed across pools. Error bars in the appendix figures are 1.961.96 MC SE. Inner current draws are therefore not treated as additional independent outer replications. E.2 Controlled Mechanism Studies Effective information and policy risk. The experiment behind Figure 2(a) uses target features X=(3,2,−1)X=(3,2,-1), constant nuisance feature W=1W=1, target center θ0=0 _0=0, nuisance center a0=log⁡(3/2)a_0= (3/2), and nc=n_c=n trusted labels with feature one. The judge probability is 0.60.6 at every comparison type. The evaluation policy has feature gap two and temperature one, so F⁡(θ)=log⁡(1+e2θ)F(θ)= (1+e^2θ) up to an additive constant and G0=1G_0=1. Every allocation has coordinate floor 0.050.05. NAOD, Target-info, and Random share the same correctly specified joint model and projected one-step estimator and differ only in acquisition. Each method–budget pair uses 6,0006,000 independent datasets, and Φκ _κ is evaluated at the realized integer allocation. Table 3: Effective-information experiment underlying Figure 2(a). Each row uses 6,0006,000 independent repetitions. Φ is evaluated using the realized integer counts. n Design Φ Mean nDFnD_F MC SE 240 NAOD 0.426 0.430 0.008 240 Target-info 1.139 1.164 0.021 240 Random 0.530 0.527 0.010 960 NAOD 0.426 0.441 0.008 960 Target-info 1.139 1.161 0.022 960 Random 0.530 0.518 0.010 3840 NAOD 0.426 0.426 0.008 3840 Target-info 1.139 1.105 0.020 3840 Random 0.530 0.526 0.009 15360 NAOD 0.426 0.431 0.008 15360 Target-info 1.139 1.144 0.021 15360 Random 0.530 0.541 0.010 Table 3 shows that empirical risks track the information constants across all four budgets. Because the likelihood and final estimator are identical across methods, the Target-info gap isolates acquisition geometry. Raw target-score variation is valuable only to the extent that it remains distinguishable from nuisance variation. This variance mechanism is separate from the exposure effect in equation 5. Under reference weights (0.1,0.2,0.7)(0.1,0.2,0.7), the probability-residual score balances to zero, whereas uniform exposure gives bξ=2/15b_ξ=2/15. Finite-sample representation-learning cost. The experiment behind Figure 2(b) uses two equally weighted comparison types with X=(1,−1)⊤X=(1,-1) , true nuisance direction W0=(1,1)⊤W_0=(1,1) , θ0=0 _0=0, and a0=c=log⁡(3/2)a_0=c= (3/2). Human probabilities are 0.50.5 and judge probabilities are 0.60.6. The historical sample contains m independent units, each contributing one human and one judge label, with the units split equally across the two comparison types. Let h^i h_i and j^i j_i denote the corresponding sample means, let s=1/4s=1/4, t=0.24t=0.24, and set g=1/2g=1/2. The historical learner first estimates the rotation δmraw=g⁡(j^1−h^1)−(j^2−h^2)2ct. _m raw= g\( j_1- h_1)-( j_2- h_2)\2ct. We then set δ^m=clip⁡(δmraw,−0.5,0.5) δ_m=clip( _m raw,-0.5,0.5) and W^m=(1+δ^m,1−δ^m)⊤ W_m=(1+ δ_m,1- δ_m) . The current design is equally weighted with nc=n_c=n. Propagating the historical rotation through the target-influence operator gives Φ0=50/49 _0=50/49 and CP=25/98C_P=25/98, so equation 16 predicts scaled risk Φ0+(n/m)CP _0+(n/m)C_P. Each setting below uses 6,0006,000 independent historical–current replications. Table 4: Representation-learning cost underlying Figure 2(b). Prediction is Φ0+(n/m)CP _0+(n/m)C_P. “Clip” is the percentage of historical fits for which the bounded rotation is active. n m/nm/n Prediction Mean nDFnD_F MC SE Clip (%) 960 1/41/4 2.041 1.891 0.034 3.1 960 11 1.276 1.252 0.023 0.0 960 44 1.084 1.085 0.020 0.0 3840 1/41/4 2.041 1.993 0.037 0.0 3840 11 1.276 1.281 0.024 0.0 3840 44 1.084 1.116 0.020 0.0 15360 1/41/4 2.041 2.046 0.037 0.0 15360 11 1.276 1.299 0.024 0.0 15360 44 1.084 1.091 0.020 0.0 At fixed m/nm/n, the predicted scaled risk is asymptotically constant rather than vanishing with the absolute current sample size. The empirical values in Table 4 approach these common levels as both samples grow, confirming that representation learning contributes at the same leading order as current-sample variance when n/mn/m does not vanish. The largest discrepancy occurs in the only setting with nonzero clipping, identifying a finite-sample departure from the local regime rather than a different asymptotic mechanism. Design-ranking reversal. The experiment behind Figure 2(c) returns to the three-type persistent-residual construction with n=nc=960n=n_c=960. For each m/n∈1/4,1/2,1,2m/n∈\1/4,1/2,1,2\, we generate 300300 independent balanced historical datasets, each containing m independent units with one human and one judge label per unit. At each comparison type, clipped human and judge sample means are transformed to logits and differenced. The resulting discrepancy vector is normalized to form the learned one-dimensional nuisance representation. A historical trusted logistic fit supplies the target center and an offset logistic fit supplies the nuisance coefficient. NAOD and Target-info use this same learned representation, current budgets, and final estimator. Each historical realization generates eight current datasets per design. Losses are first averaged within the historical realization, and paired uncertainty is computed across the 300300 outer averages. Table 5 reports the NAOD-minus-Target-info scaled-risk difference. Positive values favor Target-info. Table 5: Design-ranking reversal underlying Figure 2(c). Differences are paired over 300300 independent historical realizations, each averaging eight current datasets. Intervals are pointwise two-sided t299t_299 intervals. m/nm/n NAOD–Target-info Outer SE 95% interval 1/41/4 +1.588+1.588 0.237 [1.122, 2.054][1.122,\,2.054] 1/21/2 +0.772+0.772 0.200 [0.378, 1.165][0.378,\,1.165] 11 −0.039-0.039 0.100 [−0.235, 0.157][-0.235,\,0.157] 22 −0.446-0.446 0.049 [−0.541,−0.350][-0.541,\,-0.350] The sign change is the empirical counterpart of equation 17. Representation sensitivity dominates when historical information is scarce, the two leading contributions nearly balance around m/n=1m/n=1, and the oracle acquisition advantage dominates once representation error is sufficiently reduced. Because both designs use the same learned nuisance model and final estimator, the reversal is induced by how acquisition responds to representation error rather than by a difference in fitted model class. E.3 Additional Ablations and Diagnostics The direction of omitted error matters. The representation analysis in Appendix D.1 predicts that policy cost depends on how omitted logit error passes through the target-influence operator, not on its norm alone. At the zero-center NAOD allocation, let u1u_1 be the unit direction aligned with the target-influence row. A near-null direction u2u_2 is obtained by projecting the all-ones vector onto the orthogonal complement of u1u_1 and rotating it by π/12π/12 toward u1u_1. At n=3840n=3840, both directions receive the same magnitudes ρ∈0,1,2,3,4,6,8ρ∈\0,1,2,3,4,6,8\, with true judge logits perturbed by ρuj/nρ u_j/ n. Each point uses 6,0006,000 independent datasets. Figure 3: Controlled diagnostics. (a) Directional omitted-error diagnostic. Dashed curves are local predictions from the frozen logit-influence kernel. Solid curves are empirical mean scaled policy risks. The two perturbation directions have identical Euclidean magnitudes. (b) Policy-anisotropy diagnostic. Dashed curves are information-based predictions. Solid curves are empirical mean scaled policy risks. The Unweighted rule changes only the acquisition weight and is evaluated under the same policy loss. Error bars show 1.961.96 MC SE over 6,0006,000 independent datasets. Figure 3(a) shows that equal error magnitude can produce sharply different policy costs. The aligned perturbation follows the predicted quadratic increase, whereas the near-null perturbation remains much less consequential. Residual magnitude alone therefore cannot rank acquisition risk. The relevant quantity is its policy-weighted target projection through the target-influence operator and G0G_0. Policy weighting changes which information is useful. To isolate G0G_0, we use target rows (3,0),(2,0),(−1,0),(0,3),(0,2),(0,−1)(3,0),(2,0),(-1,0),(0,3),(0,2),(0,-1), constant nuisance feature W=1W=1, θ0=0 _0=0, and a0=log⁡(3/2)a_0= (3/2). Half of the trusted observations use each coordinate direction, giving Hc=I2/8H_c=I_2/8. The evaluation distribution weights two binary-action contexts by 0.80.8 and 0.20.2, each with feature gap two, so F⁡(θ)=0.8log⁡(1+e2θ1)+0.2log⁡(1+e2θ2),G0=diag⁡(0.8,0.2).F(θ)=0.8 (1+e^2 _1)+0.2 (1+e^2 _2), G_0=diag(0.8,0.2). The Unweighted ablation replaces this policy curvature by I2/2I_2/2 only during acquisition and is evaluated under the same true policy loss. The allocation floor is 0.030.03, nc=n_c=n, and each setting uses 6,0006,000 repetitions. Figure 3(b) shows that NAOD remains below the Unweighted design across all budgets, even though both account for nuisance estimation. The gap therefore isolates policy weighting. IeffI_eff determines which target directions remain identifiable after nuisance adjustment, while G0G_0 determines which of those directions matter for downstream policy loss. Target-info incurs the additional cost of ignoring target–nuisance coupling. Nuisance rank separates misspecification from information loss. Using the same two-dimensional construction at n=3840n=3840, we vary only the fitted nuisance rank while keeping the actual judge probability at 0.60.6. Underfit uses no nuisance feature, Correct uses the constant feature, Overfit adds an unnecessary axis-group contrast, and Saturated uses W=I6W=I_6. In the saturated model, every judge target-score direction lies in the nuisance span. Table 6: Nuisance-rank ablation at n=3840n=3840. The Underfit nominal criterion is not a true-risk prediction because its fitted model omits the actual judge discrepancy. Fitted model r Nominal Φ Mean nDFnD_F MC SE Underfit 0 0.391 27.562 0.064 Correct 1 0.686 0.699 0.010 Overfit 2 0.772 0.764 0.011 Saturated 6 4.000 4.015 0.059 Table 6 separates three distinct effects. Underfitting creates omitted-discrepancy bias, so its favorable nominal criterion is misleading. Overfitting remains correctly specified but incurs a variance cost from an unnecessary nuisance direction. Saturation is also correctly specified, but the Schur complement removes all judge-derived target information because nuisance variation can reproduce every judge target-score direction. Nuisance dimension is therefore neither uniformly beneficial nor uniformly harmful. Its effect depends on representation adequacy and target–nuisance geometry. Distinct finite-pool selection. We generate 2424 independent pools, each containing N=600N=600 target-feature rows sampled uniformly from [−2,2]2[-2,2]^2. We set Wi=1W_i=1, θ0=0 _0=0, a0=log⁡(3/2)a_0= (3/2), and use 300300 trusted labels with balanced coordinate features and the same 0.8/0.20.8/0.2 policy weighting as above. Judge budgets are B∈60,180,360B∈\60,180,360\. NAOD and Target-info use the finite-pool relaxation, rounding, and exchange procedure of Algorithm 1 with their respective frozen objectives and a common charged seed, while Random samples the remaining IDs without replacement. For each pool and budget, 300300 independent outcome datasets are generated. Only selected outcomes enter the fit, and uncertainty is computed across the 2424 pool-level mean losses. Table 7: Distinct finite-pool experiment. Risks are reported in 10−310^-3 units. Nominal risk uses the actual selected-subset information. SE is computed across 2424 independent pool-level means, each averaging 300300 current datasets. B Design Nominal risk Mean DFD_F Pool SE (×10−3)(× 10^-3) (×10−3)(× 10^-3) (×10−3)(× 10^-3) 60 NAOD 6.108 6.146 0.081 60 Target-info 6.145 6.181 0.081 60 Random 8.945 8.851 0.117 180 NAOD 3.420 3.361 0.053 180 Target-info 3.430 3.373 0.051 180 Random 5.287 5.191 0.082 360 NAOD 2.459 2.376 0.029 360 Target-info 2.461 2.385 0.030 360 Random 3.293 3.224 0.046 Table 7 shows that the realized-subset criterion tracks empirical policy loss without requiring the rounded subset to approximate a fixed-support fractional allocation. Across the 2424 pools, the largest NAOD Frank–Wolfe gap is 4.0×10−84.0× 10^-8 and the largest integer certificate is 1.1×10−61.1× 10^-6. The corresponding Target-info maxima are 5.7×10−95.7× 10^-9 and 1.2×10−71.2× 10^-7. These values certify numerical optimization of each frozen criterion, not statistical uncertainty. As a numerical check, we also compared the projected one-step estimator with a fully converged constrained joint likelihood on paired datasets. Their scaled-risk differences decreased with sample size and were negligible relative to the acquisition gaps. Appendix F Chatbot Arena Evaluation F.1 Data, Representations, and Experimental Protocol Frozen archive and cluster construction. We use the frozen revision of the Chatbot Arena LLM-judge archive. The raw archive contains 49,93849,938 rows. After removing non-string or empty text and identical-response comparisons, 49,63549,635 usable rows remain in 42,87542,875 connected text clusters. Text is Unicode NFKC normalized and whitespace is collapsed. Connected components are formed before filtering by transitive exact-text matching on the prompt or either response. Cluster identity is the ownership unit for splitting and the capacity unit for acquisition. Table 8: Arena cluster partition within each repeated split. Roles are cluster-disjoint. Candidate clusters may contain multiple rows, but at most one row from a connected cluster can be acquired. Role Clusters Use Upstream 8,5758,575 Target representation, nuisance representation, and human reference Historical initialization 128128 Initial target and nuisance estimates Policy support 1,0241,024 Policy weighting covariates with no labels Current human reservoir 512512 Nested trusted budgets H≤512H≤ 512 Candidate 24,06124,061 Judge acquisition with cluster capacity one Test 8,5758,575 Held-out human evaluation Table 8 summarizes these role assignments. Upstream, historical initialization, policy support, and the human reservoir use one representative row from each assigned cluster. Candidate retains all eligible rows subject to cluster capacity one. Test performance is first averaged within connected clusters. We use 1515 prespecified repeated cluster-level splits. Each split relearns the downstream representations and human reference, while all 1515 budget combinations within a split share these upstream objects. Target and nuisance representations. Response features are extracted with Qwen3-Embedding-0.6B. The pair representation is the difference between the A and B response embeddings. An upstream-fitted whitened PCA reduces this representation to 128128 coordinates. For each judge, we fit four ridge heads separately to human preferences and to that judge’s hard preferences. The leading two-dimensional singular-vector span of the resulting coefficient vectors defines the target representation XJX_J. Policy-weighted residual learning is performed separately. The resulting standardized swap-antisymmetric residual score rJ(z)r_J(z) defines the nuisance representation WJ(z)=(1,rJ(z))⊤W_J(z)=(1,r_J(z)) . Thus the target and nuisance representations are learned from upstream data and frozen before current acquisition. Table 9 records the frozen numerical configuration used across methods. Table 9: Frozen Arena configuration. All acquisition methods use the same representation, initialization, current budgets, and final estimator unless required otherwise by the method definition. Component Configuration Embedding Qwen3-Embedding-0.6B, last-token pooling, L2L_2 normalization Maximum input 2,0482,048 tokens with prompt cap 512512 Base reduction Upstream-fitted whitened PCA, 128128 dimensions Target representation Judge-specific SVD span, dimension d=2d=2 Target-head penalties 32,128,512,204832,128,512,2048 Nuisance representation Intercept and standardized learned residual, dimension r=2r=2 Residual learner ExtraTrees, 128128 trees, depth 1010, min leaf 2020, feature fraction 0.50.5, weight clip [0.1,10][0.1,10] Inner residual folds 44 Human budgets H∈32,64,128,256,512H∈\32,64,128,256,512\ Judge budgets B∈256,512,1024B∈\256,512,1024\ Parameter boxes ‖θ‖∞≤20\|θ\|_∞≤ 20, ‖a‖∞≤10\|a\|_∞≤ 10 Final step size α=1α=1 Finite-pool solver Frank–Wolfe cap 180180, tolerance 10−510^-5 Cluster capacity At most one selected comparison per connected cluster Feedback and common estimator. Human outcomes are encoded as 11 for A, 00 for B, and 1/21/2 for ties. For judge feedback, the archived A/B/tie probabilities are converted to the soft target yJ=pA+12pTy_J=p_A+ 12p_T. The logistic quasi-likelihood therefore uses the archived graded preference signal directly. A common historical set of 128128 comparisons supplies the initial target and nuisance estimates before current acquisition. After current trusted feedback and the selected judge feedback are revealed, every acquisition method uses the same unpenalized joint observed-information update and the same fixed parameter boxes. Historical initialization rows are not reused in the final score. NAOD and Target-info use the same charged feasibility anchors. Their comparison therefore changes the acquisition criterion while holding the representation, budgets, and final estimator fixed. Resource accounting. The varying current resource is (H,B)(H,B). The 8,5758,575 upstream human–judge comparisons used for representation learning and the 128128 historical comparisons used for initialization are shared across acquisition rules and are not counted in H or B. Policy support contributes covariates without labels, while test human outcomes are used only for evaluation. F.2 Metrics, Baselines, and Statistical Analysis Acquisition baselines. Random samples feasible candidate comparisons uniformly. Entropy prioritizes candidates with high predictive uncertainty. D-opt and PA D-opt use Fisher-information acquisition on the common judge-specific target representation, with PA D-opt additionally incorporating the historical initialization as past information. Target-info minimizes the policy-weighted target-information criterion without nuisance adjustment. NAOD instead uses effective target information after nuisance adjustment. Initial performs no current update. Human-only updates from the common initialization using current trusted feedback without acquired judge outcomes. Evaluation endpoints. The primary endpoint is proxy policy regret DF(θref,θ^)D_F( _ref, θ), where θref _ref is fitted only from upstream human feedback. We additionally report held-out human cross-entropy and tie-aware human choice accuracy. Cross-entropy uses the full predicted probability. For hard choice accuracy, we predict A when the fitted preference probability is at least 0.50.5 and B otherwise. Human ties receive half credit. Aggregation and paired uncertainty. Test losses are first averaged within connected clusters and then equally across clusters. Judges are macro-averaged within each split. For overall results, the 1515 budget combinations are also equally weighted. The repeated split is the inference unit. For loss endpoints, paired gain is baseline minus NAOD. For accuracy, paired gain is NAOD minus baseline. Positive values therefore favor NAOD throughout. Two-sided 95%95\% Student t intervals are computed from the 1515 paired split-level differences using t14t_14. Budget and judge intervals are pointwise intervals. Relative proxy-regret reduction is 100(1−R¯NAOD/R¯b)100(1- R_NAOD/ R_b), which is a ratio of aggregated mean regrets rather than an average of split-specific percentage reductions. F.3 Complete Results and Diagnostics Paired comparisons across methods. The main text reports absolute endpoint means. Table 10 reports the corresponding paired split-level effects and their uncertainty. Table 10: Overall paired Arena comparisons. Positive values favor NAOD. Proxy-regret and human-CE gains are in 10−310^-3 units. Accuracy gains are percentage points. Brackets give paired 95%95\% t14t_14 intervals across the 1515 repeated splits. Baseline Proxy-regret gain Human-CE gain Accuracy gain Initial +6.870[3.020,10.721]+6.870\,[3.020,10.721] +5.308[1.952,8.664]+5.308\,[1.952,8.664] +0.448[0.117,0.780]+0.448\,[0.117,0.780] Human-only +16.259[8.849,23.668]+16.259\,[8.849,23.668] +12.981[7.214,18.748]+12.981\,[7.214,18.748] +1.426[0.331,2.520]+1.426\,[0.331,2.520] Random +2.886[2.455,3.318]+2.886\,[2.455,3.318] +2.814[2.388,3.241]+2.814\,[2.388,3.241] +0.050[−0.069,0.169]+0.050\,[-0.069,0.169] Entropy +11.433[4.783,18.083]+11.433\,[4.783,18.083] +8.527[3.669,13.385]+8.527\,[3.669,13.385] +1.166[0.079,2.252]+1.166\,[0.079,2.252] D-opt +0.463[0.093,0.833]+0.463\,[0.093,0.833] +0.385[0.011,0.759]+0.385\,[0.011,0.759] +0.075[0.028,0.121]+0.075\,[0.028,0.121] PA D-opt +0.465[0.095,0.834]+0.465\,[0.095,0.834] +0.386[0.013,0.760]+0.386\,[0.013,0.760] +0.075[0.029,0.121]+0.075\,[0.029,0.121] Target-info +0.818[0.348,1.287]+0.818\,[0.348,1.287] +0.531[0.211,0.850]+0.531\,[0.211,0.850] +0.058[0.015,0.101]+0.058\,[0.015,0.101] Relative to Target-info, the proxy-regret gain is accompanied by lower held-out human cross-entropy and higher held-out choice accuracy. The D-opt and PA D-opt differences are smaller but remain positive in the overall paired analysis. The smaller accuracy gain is expected because hard choice accuracy changes only when the predicted preference crosses the 0.50.5 threshold, whereas cross-entropy remains sensitive to probability improvements that preserve the predicted winner. The resulting accuracy gain is positive in 1212 of 1515 repeated splits. Budget dependence. Figure 4 reports relative proxy-regret reductions over the complete 5×35× 3 budget grid. Table 11 reports the paired absolute gains and uncertainty for Target-info and D-opt. Figure 4: Budget dependence on Chatbot Arena. NAOD relative reduction in mean proxy policy regret across the complete 5×35× 3 Arena budget grid. Positive values favor NAOD. Panel (a) compares against Target-info and panel (b) against D-opt. Cell values are ratios of macro-averaged mean regrets rather than averages of split-specific percentage reductions. Table 11: Paired proxy-regret gains across the Arena budget grid. Entries are baseline-minus-NAOD gains in 10−310^-3 units with pointwise paired 95%95\% t14t_14 intervals. H B Target-info–NAOD D-opt–NAOD 32 256 +2.103[0.263,3.944]+2.103\,[0.263,3.944] +1.543[0.612,2.474]+1.543\,[0.612,2.474] 32 512 +1.141[0.447,1.835]+1.141\,[0.447,1.835] +0.715[0.138,1.291]+0.715\,[0.138,1.291] 32 1024 +0.408[0.143,0.673]+0.408\,[0.143,0.673] +0.160[−0.192,0.512]+0.160\,[-0.192,0.512] 64 256 +1.731[0.507,2.955]+1.731\,[0.507,2.955] +1.234[0.359,2.109]+1.234\,[0.359,2.109] 64 512 +1.123[0.476,1.770]+1.123\,[0.476,1.770] +0.703[0.125,1.281]+0.703\,[0.125,1.281] 64 1024 +0.471[0.184,0.758]+0.471\,[0.184,0.758] +0.234[−0.119,0.587]+0.234\,[-0.119,0.587] 128 256 +1.172[0.205,2.138]+1.172\,[0.205,2.138] +0.814[0.062,1.565]+0.814\,[0.062,1.565] 128 512 +0.843[0.071,1.615]+0.843\,[0.071,1.615] +0.434[−0.117,0.986]+0.434\,[-0.117,0.986] 128 1024 +0.367[−0.044,0.778]+0.367\,[-0.044,0.778] +0.092[−0.303,0.487]+0.092\,[-0.303,0.487] 256 256 +0.844[0.211,1.477]+0.844\,[0.211,1.477] +0.466[0.008,0.924]+0.466\,[0.008,0.924] 256 512 +0.553[0.221,0.885]+0.553\,[0.221,0.885] +0.186[−0.222,0.594]+0.186\,[-0.222,0.594] 256 1024 +0.176[−0.044,0.396]+0.176\,[-0.044,0.396] −0.084[−0.405,0.238]-0.084\,[-0.405,0.238] 512 256 +0.664[0.203,1.124]+0.664\,[0.203,1.124] +0.395[0.001,0.790]+0.395\,[0.001,0.790] 512 512 +0.515[0.199,0.831]+0.515\,[0.199,0.831] +0.157[−0.186,0.500]+0.157\,[-0.186,0.500] 512 1024 +0.155[−0.044,0.354]+0.155\,[-0.044,0.354] −0.103[−0.369,0.162]-0.103\,[-0.369,0.162] The Target-info mean comparison favors NAOD in all 1515 budget cells, with 1212 pointwise paired intervals entirely above zero. At every fixed human budget, the relative gain narrows as the judge budget increases. The pattern is not determined by H/BH/B alone. Along H/B=1/8H/B=1/8, the relative reductions are 50.1%50.1\%, 36.7%36.7\%, and 15.3%15.3\% for (H,B)=(32,256),(64,512),(128,1024)(H,B)=(32,256),(64,512),(128,1024). Selection overlap increases from 0.4460.446 to 0.5050.505 to 0.5680.568 over the same sequence. Absolute information scale and finite-pool geometry therefore matter in addition to the human-to-judge budget ratio. The D-opt gap narrows more sharply at large judge budgets. At (256,1024)(256,1024) and (512,1024)(512,1024), the mean difference changes sign, while both paired intervals include zero. The separation from D-opt is therefore budget dependent rather than uniform across the grid. Judge-level target–nuisance coupling. Figure 5 summarizes the judge-level association between Target-info coupling and realized policy gain, while Table 12 reports the corresponding coupling values and paired policy gains with pointwise 95%95\% intervals. Figure 5: Judge-level coupling and realized acquisition gain. The horizontal axis is Target-info target–nuisance coupling ρ2ρ^2 and the vertical axis is the Target-info-minus-NAOD proxy-regret gain in 10−310^-3 units. Each point represents one judge after averaging over the 1515 budget combinations and 1515 repeated cluster-level splits. The Spearman rank correlation is rs=0.576r_s=0.576. Positive vertical values favor NAOD. Table 12: Judge-level coupling and policy gain. ρT2 _T^2 is the squared target–nuisance coupling under Target-info. Policy gain is Target-info minus NAOD in 10−310^-3 units with pointwise paired 95%95\% t14t_14 intervals. Judge ρT2 _T^2 Policy gain Athene-70B 0.570 +0.428[−0.290,1.145]+0.428\,[-0.290,1.145] dolphin-2.1-mistral-7b 0.956 +2.063[1.076,3.049]+2.063\,[1.076,3.049] dolphin-2.5-mixtral-8x7b 0.960 +2.819[1.248,4.390]+2.819\,[1.248,4.390] Hermes-3-Llama-3.1-70B 0.610 +0.366[−0.305,1.038]+0.366\,[-0.305,1.038] Meta-Llama-3-70B-Instruct 0.459 −0.037[−0.358,0.285]-0.037\,[-0.358,0.285] Meta-Llama-3-8B-Instruct 0.716 +0.733[0.069,1.397]+0.733\,[0.069,1.397] Meta-Llama-3-8B 0.958 +4.024[1.184,6.865]+4.024\,[1.184,6.865] Mistral-7B-Instruct-v0.1 0.713 +0.096[−0.035,0.227]+0.096\,[-0.035,0.227] Mistral-7B-Instruct-v0.2 0.666 +0.491[0.129,0.854]+0.491\,[0.129,0.854] Mistral-7B-OpenOrca 0.646 +0.067[−0.151,0.285]+0.067\,[-0.151,0.285] Mixtral-8x7B-Instruct-v0.1 0.604 +0.749[0.332,1.166]+0.749\,[0.332,1.166] OpenHermes-2-Mistral-7B 0.676 +0.194[−0.141,0.529]+0.194\,[-0.141,0.529] OpenHermes-2.5-Mistral-7B 0.611 +0.531[0.153,0.909]+0.531\,[0.153,0.909] Qwen2-72B-Instruct 0.551 +0.211[−0.252,0.673]+0.211\,[-0.252,0.673] StableBeluga-7B 0.915 +0.873[0.234,1.511]+0.873\,[0.234,1.511] Starling-LM-7B-alpha 0.646 +0.391[0.045,0.736]+0.391\,[0.045,0.736] zephyr-7b-beta 0.647 −0.099[−0.283,0.084]-0.099\,[-0.283,0.084] Mean policy gain is positive for 1515 of the 1717 judges. The largest gains occur for Meta-Llama-3-8B and the two dolphin judges, whose Target-info coupling values all exceed 0.950.95. By contrast, Meta-Llama-3-70B-Instruct has strong residual predictability but substantially weaker coupling and essentially no policy gain. This contrast separates residual learnability from target–nuisance interference. The Spearman association in Figure 5 summarizes this judge-level relationship. Across all budget and split designs, mean squared coupling decreases from 0.7000.700 under Target-info to 0.2900.290 under NAOD. The mean selected-set overlap is 0.5210.521. NAOD therefore changes which comparisons are acquired in a direction that reduces target–nuisance coupling rather than applying a different estimator to essentially the same selected set. Representation and numerical diagnostics. All 255=15×17255=15× 17 split-by-judge nuisance fits achieve positive out-of-fold soft-label cross-entropy gain, with mean gain 0.03700.0370. The learned nuisance representation therefore captures systematic judge residual structure across the full evaluation. The audit finds no budget, duplicate-ID, duplicate-cluster, out-of-pool, projection, or design failures in 22,95022,950 acquisition arrays. Across 7,6507,650 NAOD and Target-info designs, the maximum Frank–Wolfe gap is 8.62×10−88.62× 10^-8 and the maximum integer certificate is 6.01×10−76.01× 10^-7, indicating that the observed acquisition differences are not explained by unresolved selection optimization.

Instant research discovery