Paper deep dive
HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data
Xinrui Ruan, Zhenyu Zhao, Waverly Wei, Yueshan Zhang, Zeyu Zheng, Sui Huang, Jingshen Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 7/5/2026, 2:46:12 AM
Summary
The paper introduces HERO (History Enhanced RObust model evaluation), a two-step framework designed to improve the reliability (bias reduction) and sensitivity (variance reduction) of generative AI model evaluation. It addresses the problem of expensive 'gold' labels versus noisy 'silver' labels by leveraging historical evaluation data. Step 1 uses an EM algorithm to calibrate silver labeler performance (sensitivity and specificity) from historical gold annotations to reduce bias. Step 2 employs score-based control variates using shared historical covariates (like prompt features) to reduce the variance of the estimator. The framework is applicable to various tasks including single model evaluation, comparative evaluation, and safety moderation.
Entities (7)
Relation Signals (5)
HERO → implements → EM Algorithm
confidence 100% · HERO estimates (αj,βj) for each labeler jj from the historical dataset using an EM algorithm
Silver Labels → iscalibratedby → Gold Labels
confidence 100% · HERO calibrates silver labelers' performance learned from historical gold annotations
HERO → uses → Historical Data
confidence 100% · HERO (History Enhanced RObst model evaluation), a novel framework that uses historical data to suppress bias...
HERO → utilizes → Control Variates
confidence 100% · combines bias reduction from labeler reliability estimation with variance reduction from control variates
Prompt Features → servesas → Control Variate
confidence 90% · Prompt features are a natural shared covariate... The historical mean can therefore serve as a stable reference in a control-variate adjustment
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these "gold" labels are expensive to collect and limited in quantity. Organizations thus often turn to collecting vast but noisy "silver" labels from crowdsourced workers or vendor annotators as proxies for gold labels. Because gold remains the evaluation target, naively aggregating noisy silver labels may introduce bias, and estimators built on sparsely observed gold labels may have high variance to resolve the model performance gaps that guide practical decisions. Model evaluation has become an ongoing operational practice rather than a one-time exercise, with evaluation rounds repeating across model versions, releases, and content domains. A natural question is whether the previous historical evaluation data can be used to improve each new round of evaluation. We introduce HERO (History Enhanced RObust model evaluation), a novel framework that uses historical data to suppress bias (improve reliability) and reduce variance (improve sensitivity) in model performance evaluation. HERO calibrates silver labelers' performance learned from historical gold annotations, and stabilizes the resulting estimator by anchoring it to covariate information measured with high precision in the historical data. HERO can be broadly applied across multiple common evaluation tasks, and remains valid when only a subset of historical labelers appears in the current round. We establish conditions under which the bias and variance reductions hold, showcase HERO's performance in simulation studies, and demonstrate its effectiveness on real-world model evaluation benchmarking datasets.
Tags
Links
- Source: https://arxiv.org/abs/2606.29784v1
- Canonical: https://arxiv.org/abs/2606.29784v1
Trouble viewing inline? Open PDF directly →
Full Text
79,000 characters extracted from source content.
Expand or collapse full text
HERO: Improving the Reliability and Sensitivity of Generative Model Evaluation Using Historical Data Xinrui Ruan1,2,* Zhenyu Zhao2,* Waverly Wei3 Yueshan Zhang4 Zeyu Zheng5 Sui Huang2 Jingshen Wang1,† ( 1Division of Biostatistics, University of California, Berkeley 2Roblox Corporation 3Department of Data Sciences and Operations, University of Southern California 4School of Mathematical Sciences, Nankai University 5Department of Industrial Engineering and Operations Research, University of California, Berkeley ) Abstract Reliable generative AI models critically rely on expert human annotations to evaluate output quality, yet these “gold” labels are expensive to collect and limited in quantity. Organizations thus often turn to collecting vast but noisy “silver” labels from crowdsourced workers or vendor annotators as proxies for gold labels. Because gold remains the evaluation target, naively aggregating noisy silver labels may introduce bias, and estimators built on sparsely observed gold labels may have high variance to resolve the model performance gaps that guide practical decisions. Model evaluation has become an ongoing operational practice rather than a one-time exercise, with evaluation rounds repeating across model versions, releases, and content domains. A natural question is whether the previous historical evaluation data can be used to improve each new round of evaluation. We introduce HERO (History Enhanced RObust model evaluation), a novel framework that uses historical data to suppress bias (improve reliability) and reduce variance (improve sensitivity) in model performance evaluation. HERO calibrates silver labelers’ performance learned from historical gold annotations, and stabilizes the resulting estimator by anchoring it to covariate information measured with high precision in the historical data. HERO can be broadly applied across multiple common evaluation tasks, and remains valid when only a subset of historical labelers appears in the current round. We establish conditions under which the bias and variance reductions hold, showcase HERO’s performance in simulation studies, and demonstrate its effectiveness on real-world model evaluation benchmarking datasets. Keywords: Historical data; Noisy labels; Bias reduction, Control variates. †footnotetext: * Xinrui Ruan and Zhenyu Zhao contributed equally. This work was done while Xinrui Ruan was an intern at Roblox Corporation. † Correspondence: jingshenwang@berkeley.edu. 1 Introduction 1.1 Motivation and Contribution Human evaluation is at the heart of developing and deploying reliable generative AI models [3]. Many organizations rely heavily on human annotations to evaluate model output quality, assess safety, calibrate moderation systems, and compare candidate model versions before release [20, 15, 9]. Small differences in estimated model quality, even on the order of a few percentage points in win rate or defect rate, determine which model is shipped to hundreds of millions of users [25, 22]. As widely documented in natural language processing and related fields, a key challenge within an annotation task is that expert-annotated “gold” labels (a.k.a. ground truth labels) are expensive and scarce [20], so organizations often instead rely on judgments from a pool of less specialized “silver” annotators whose labeling reliability varies. The so-called gold labels may be produced by a single expert annotator or by consensus among multiple experts within the organization, and they serve as the best available reference standard. In contrast, large-scale evaluations typically rely on silver labels collected from broader, less specialized sources, including crowd or vendor annotators and automated judges such as LLMs [12] or VLMs [2]. Throughout this paper, we view silver labels as the primary operational labels used in evaluation, and gold labels as a more accurate but limited supervision source used to assess and calibrate them. Because silver annotators differ in reliability, simply aggregating their labels without adjustment can reduce the reliability of evaluation and introduce bias in the estimated model performance. Many methods have been developed to account for annotator variability [5, 20, 28, 17], but most operate within a single evaluation round and do not fully leverage the unique structure of modern model evaluation settings. We note that two novel features distinguish modern AI model evaluations from classical annotation tasks. First, organizations that evaluate generative models accumulate large-scale historical annotation data across prior evaluation rounds. These datasets often include repeated observations of annotator behavior and subsets of instances with gold labels, which together reveal patterns in labeler reliability. Second, the historical and current model evaluation tasks may share content that transfers across rounds, such as prompt features. Generation prompts often persist across rounds because organizations evaluate successive model versions on a stable prompt distribution drawn from real user traffic or curated benchmarks [3], so the same prompt features that appeared in earlier rounds also characterize the current evaluation set. With these two features, in principle, labeler reliability learned from historical gold labels can calibrate current annotations, and the shared generation prompts available in a large historical sample can stabilize current model evaluation estimates. We defer a more complete comparison with existing literature to Appendix F. To fully exploit historical data for reliable model evaluation, we propose HERO (History Enhanced RObust model evaluation), a framework that adjusts noisy annotation-based estimators with historical evaluation data to reduce both bias and variance in model performance estimates. The key contributions of our work include: • We elucidate two unique features that distinguish modern AI evaluation from classical annotation literature (Figure 2), namely the accumulation of historical annotation data with gold-labeled subsets, and prompt feature transfer from earlier rounds to the current evaluation round. • We identify bias sources that arise in this evaluation setting through real-world benchmarking data, isolating heterogeneous silver labeler reliability and labeler pool shift between rounds (Figure 1). • We propose a theoretically grounded yet simple method, HERO, that combines bias reduction from labeler reliability estimation with variance reduction from control variates (Section 3). Our control variate construction is customized at the score level, departing from the existing literature that operates on raw outcomes. • We show that the proposed method applies across multiple common model evaluation use cases (Table 1) and validate our method with comprehensive empirical evidence (Sections 4 and 5). 1.2 Model Evaluation Target Parameters and Tasks We organize common model evaluation scenarios into a unified observational structure. Consider n instances indexed by i=1,…,ni=1,…,n. For each instance, Yi∗Y_i^* denotes the operational ground-truth gold label, the best available adjudicated assessment of the generated output, and ZiZ_i denotes the noisy silver labels collected from less specialized annotators or automated AI systems as proxies for Yi∗Y_i^*. The target parameter is a function of Yi∗Y_i^*, and the task-specific outcome YiY_i is constructed from Yi∗Y_i^* according to the scenario. Table 1 summarizes four evaluation scenarios, listing the outcome variable YiY_i, the corresponding target parameter, and the available covariates for each. We summarize each case in Table 1 below and defer its detailed explanation to the appendix. Use Case Outcome YiY_i Target Parameter 1. Single Model Evaluation Gold label for Model A: Yi=Yi,A∗Y_i=Y^*_i, A Average performance: μA=[Yi] _ A=E[Y_i] 2. Model Comparative Evaluation Difference in gold labels: Yi=Yi,A∗−Yi,B∗Y_i=Y^*_i, A-Y^*_i, B Average performance difference: α=[Yi]α=E[Y_i] 3. Side-by-Side Model Comparison Binary preference indicator: Yi=(Yi,A∗≻Yi,B∗)Y_i= 1(Y^*_i, A Y^*_i, B) Win rate for Model A: πA≻B=[Yi] _ A B=E[Y_i] 4. Safety Moderation Calibration True positive flag: Yi=Yi∗YiAIY_i=Y_i^*Y_i AI Precision =[Yi]/[YiAI]=E[Y_i]/E[Y_i AI]; Recall =[Yi]/[Yi∗]=E[Y_i]/E[Y_i^*] Table 1: Summary of model evaluation use cases. Figure 1: Panels (A)–(C) show heterogeneous silver labeler agreement rates, sensitivity αj _j, and specificity βj _j, with 95% confidence intervals in our benchmarking data (Appendix F, Asset detect detection data). Panel (D) shows the labeler pool shift between the two model evaluation rounds. 2 Bias and Variance in Model Evaluation Estimating the target parameter [Yi]E[Y_i] from gold labels alone is rarely feasible in practice. This is because many evaluation tasks provide no gold labels at all. Even when a subset of instances is annotated by a gold labeler, the labeled fraction is typically small. The sample mean over this subset is unbiased, but its variance remains too large to resolve the differences in model performance that guide practical decisions. For these reasons, practitioners often construct a proxy for YiY_i by aggregating the available silver labels ZiZ_i. Let μ μ denote an estimator of [Yi]E[Y_i] constructed from a silver-labeler-based proxy. To assess estimator quality, we decompose the mean-squared error (MSE) into bias and variance: [(μ^−[Yi])2]⏟MSE relative to target=([μ^]−[Yi])2⏟Bias2+Var(μ^)⏟Variance. E\! [ ( μ-E[Y_i] )^2 ]_MSE relative to target\;=\; (E[ μ]-E[Y_i] )^2_Bias^2\;+\; Var( μ)_Variance. (1) Both components introduce distinct challenges. We start by examining the bias component through two sources that arise in generative model evaluation. The first source of bias is annotator heterogeneity. The inter-rater reliability literature consistently finds variation in annotator accuracy across domains from medical imaging [24, 18] to natural language processing [21], even under shared guidelines [4, 8, 5, 27]. Differences in expertise, interpretation standards, and decision thresholds cause some labelers to favor precision over recall, while others show the opposite tendency. Aggregation without calibration, such as the majority vote-based estimators, allows these systematic tendencies, particularly from lower-quality labelers, to shift the estimate of [Yi]E[Y_i] away from the true target. As shown in Figure 1 Panels A-C, the seven silver labelers in our benchmarking data span agreement rates from below 0.4 to above 0.8, and the sensitivity αj _j and specificity βj _j panels reveal asymmetric error profiles behind that spread, with some labelers favoring positives, others favoring negatives, and a third group noisy on both axes. A majority-vote estimator weights these labelers equally and therefore inherits the mixture rather than the gold-labeler target. A second source of bias arises when the composition of the labeler pool changes between evaluation rounds. For example, in comparative evaluation tasks (Use Case 2 in Table 1), organizations may evaluate a newer model against an earlier version at different times. The labeler pools that annotate each model may only partially overlap. A reliable annotator who contributes a large fraction of Model A labels may complete only a small fraction of Model B labels, and vice versa. When the proportion of reliable annotators differs between the two rounds, the observed label difference reflects both true differences in model quality and shifts in the composition of the labeler pool. An estimator that treats the two label sets as exchangeable confounds these two sources of variation. Figure 1 Panel D illustrates this shift in our benchmarking data. Two labelers active in the Model A round do not appear in the Model B round, and the labelers active in both rounds contribute different proportions of labels across the two models. The variance component of the MSE in Equation (1) remains a challenge even after addressing the bias issues mentioned above. Even with a well-calibrated silver label based proxy, a small labeled set or high labeling outcome heterogeneity across instances can produce confidence intervals too wide to resolve the performance gaps that guide practical decisions. A variance reduction technique that halves the estimator variance is equivalent to doubling the effective sample size, so that an evaluation previously requiring 1,000 labeled instances can reach the same precision with 500, cutting annotation cost and turnaround time proportionally. 3 Method: History Enhanced RObust (HERO) Model Evaluation Figure 2: Overview of the History-Enhanced Robust (HERO) evaluation framework. HERO combines a large historical dataset (Y~i,Z~i,X~i)i=1N( Y_i, Z_i, X_i)_i=1^N with a small current sample (Zi,Xi)i=1n(Z_i,X_i)_i=1^n (n≪Nn N) to estimate the target parameter μ=[Yi]μ=E[Y_i] with YiY_i unobserved on current evaluation data. In this paper, we propose a unified framework to reduce both bias and variance for an accurate (improved reliability) and efficient (improved sensitivity) estimation of the target parameter μ. Achieving both goals requires an external source of information beyond the only current silver labels. As organizations that evaluate generative models often maintain historical evaluation datasets from prior rounds, our framework leverages these historical data to construct an estimator of μ with reduced MSE. We term the proposed framework History Enhanced RObust (HERO) model evaluation and organize it as a two-step procedure that first reduces the bias and then controls the variance. At a very high level, HERO works as follows. When we analyze current evaluation data with only silver labels, HERO first constructs calibrated gold proxies that weight each silver labeler by the reliability learned from historical data, yielding a bias-reduced estimator μ^HERO-1 μ_HERO-1. HERO then incorporates historical covariates (e.g., prompt features) as control variates to further reduce the variance of μ^HERO-1 μ_HERO-1, yielding the final estimator μ^HERO-2 μ_HERO-2. This estimator satisfies: • μ^HERO-1 μ_HERO-1 and μ^HERO-2 μ_HERO-2 both have reduced bias compared to the majority-vote-based estimator. • The variance of μ^HERO-2 μ_HERO-2 is reduced compared to μ^HERO-1 μ_HERO-1 when shared covariates between the current and historical data are available. 3.1 Historical and Current Evaluation Data To distinguish historical data from the current evaluation task, we introduce a consistent notation. We use a tilde to denote quantities associated with historical data. The historical dataset consists of N instances collected from earlier evaluation rounds and is defined as follows: • Noisy silver labels Z~i Z_i from a pool of m silver labelers. Not all labelers annotate every instance. The availability indicator A~ij∈0,1 A_ij∈\0,1\ records whether labeler j provided a label for instance i. • A set of instances together with their associated covariates X~ii=1N\ X_i\_i=1^N, such as prompt features, metadata, or contextual information. These prompts may reappear in future evaluation tasks. • Gold labels Y~i Y_i are collected on a subset of historical instances to audit silver labeler performance and identify which labelers are consistently accurate. These gold labels are obtained under a randomized routing mechanism with known selection probability given X~i X_i. The routing indicator R~i∈0,1 R_i∈\0,1\ equals 11 when instance i is sent to a gold labeler and 0 otherwise. Gold labels are observed only when R~i=1 R_i=1. Meanwhile, the current evaluation dataset contains n instances, where n≪Nn N: • A set of instances together with their associated covariates Xii=1n\X_i\_i=1^n, including prompt features, metadata, and contextual information. • No gold labels are observed in the current task. • Noisy silver labels ZiZ_i with availability indicators Aij∈0,1A_ij∈\0,1\. Throughout this section, all outcomes and labels are binary, so that Yi∈0,1Y_i∈\0,1\ and Y~i∈0,1 Y_i∈\0,1\. The target parameter remains μ=[Yi]μ=E[Y_i] as defined in Section 1.2, but gold label YiY_i is unobserved. Scope of historical data. Our framework allows two kinds of mismatch between current and historical data. First, the current annotator pool can differ, but it is a subset of the labelers who contributed to the historical data. Second, the historical task itself may differ from the current task, since HERO calibrates labeler reliability rather than item-level labels. 3.2 HERO Step 1: Bias Reduction via Quantifying Labeler Reliability To account for model evaluation bias introduced by silver labeler heterogeneity, historical data provide a natural source for revealing persistent patterns in labeler behavior. HERO Step 1 extracts these reliability patterns from the historical data and transfers them to the current evaluation task, where no gold labels are available. More concretely, we characterize each silver labeler j by two parameters that describe intrinsic properties of that labeler’s decision process, αj _j =P(Zij=1∣Yi=1),βj=P(Zij=0∣Yi=0). =P(Z_ij=1 Y_i=1), _j=P(Z_ij=0 Y_i=0). Here αj _j is the sensitivity and βj _j is the specificity. Because these parameters describe the labeler rather than the task, they are expected to transfer from the historical dataset to the current dataset even when the underlying evaluation content/task differs. HERO estimates (αj,βj)( _j, _j) for each labeler j from the historical dataset (X~i,R~iY~i,Z~i,R~i,A~ij)i=1N\( X_i, R_i Y_i, Z_i, R_i, A_ij)\_i=1^N using an EM algorithm that treats unobserved gold labels as latent variables, and use observed gold labelers as anchors. Given a candidate value y∈0,1y∈\0,1\ for the gold label, the likelihood of the observed silver labels for historical instance i is ℓ~i(y)=∏j:A~ij=1P(Z~ij∣Y~i=y), _i(y)= _j\,:\, A_ij=1P( Z_ij Y_i=y), (2) where P(Z~ij∣Y~i=1)=αjZ~ij(1−αj)1−Z~ijP( Z_ij Y_i=1)= _j^\, Z_ij\,(1- _j)^1- Z_ij and P(Z~ij∣Y~i=0)=(1−βj)Z~ijβj 1−Z~ijP( Z_ij Y_i=0)=(1- _j) Z_ij\, _j^\,1- Z_ij. An instance without a gold label (R~i=0 R_i=0) contributes the marginal μ~ℓ~i(1)+(1−μ~)ℓ~i(0) μ\, _i(1)+(1- μ)\, _i(0), where μ~=[Y~]≠[Y] μ=E[ Y] [Y]. The full historical log-likelihood is ℒ~(μ~,αj,βjj=1m)=∑i=1N[R~ilogℓ~i(Y~i)+(1−R~i)log(μ~ℓ~i(1)+(1−μ~)ℓ~i(0))]. L ( μ,\,\ _j, _j\_j=1^m )= _i=1^N [ R_i\, _i( Y_i)+(1- R_i)\, ( μ\, _i(1)+(1- μ)\, _i(0) ) ]. (3) To stabilize the EM updates, HERO initializes the parameters using only the gold-labeled subset =i:R~i=1G=\i: R_i=1\. Specifically, we set μ~(0)=||−1∑i∈Y~i,αj(0)=∑i∈A~ijY~iZ~ij∑i∈A~ijY~i,βj(0)=∑i∈A~ij(1−Y~i)(1−Z~ij)∑i∈A~ij(1−Y~i). μ^(0)=|G|^-1 _i Y_i, _j^(0)= _i A_ij Y_i Z_ij _i A_ij Y_i, _j^(0)= _i A_ij(1- Y_i)(1- Z_ij) _i A_ij(1- Y_i). HERO then maximizes ℒ~ L via the EM algorithm (see Appendix Algorithm 1, Stage A), producing estimates (α^j,β^j)( α_j, β_j) that are treated as fixed when estimating μ on the current data. The current-data likelihood li(y)l_i(y) follows the same structure as defined in (2), with (Zij,Aij)(Z_ij,A_ij) replacing (Z~ij,A~ij)( Z_ij, A_ij) and the estimates (α^j,β^j)( α_j, β_j) replacing the unknown parameters. Because no gold labels are observed, every instance contributes through the marginal form. The HERO Step 1 estimator maximizes the log-likelihood over μ∈(0,1)μ∈(0,1), μ^hero-1=argmaxμ∈(0,1)∑i=1nlog[μℓi(1)+(1−μ)ℓi(0)]. μ_ hero-1= _μ∈(0,1)\; _i=1^n [μ\, _i(1)+(1-μ)\, _i(0) ]. (4) The first-order condition is ∑i=1nψi(μ):=∑i=1nℓi(1)−ℓi(0)μℓi(1)+(1−μ)ℓi(0)=0, _i=1^n _i(μ):= _i=1^n _i(1)- _i(0)μ\, _i(1)+(1-μ)\, _i(0)=0, (5) which can be solved by any standard one-dimensional optimizer because μ is scalar. Here, ψi(μ) _i(μ) denotes the individual score contribution from instance i. Remark 1 (Comparison with majority vote) The instance-level ratio ℓi(1)/ℓi(0) _i(1)/ _i(0) controls how each instance influences μ^hero-1 μ_ hero-1. When most high-sensitivity labelers agree on a positive label, this ratio is large, pulling the estimate toward the value implied by accurate annotators. Conversely, agreement among labelers with low specificity on a negative label carries less weight. When all labelers share identical accuracy, ℓi(1)/ℓi(0) _i(1)/ _i(0) reduces to a monotone function of the majority-vote count, and the estimator recovers a majority-vote-type solution as a special case. 3.3 HERO Step 2: Variance Reduction via Score-Based Control Variates On top of bias reduction, HERO can further reduce estimator variance by leveraging auxiliary covariates that are shared between the historical and current datasets. Prompt features are a natural shared covariate, since organizations typically evaluate successive model versions on a stable prompt distribution drawn from real user traffic or curated benchmarks, and the same prompt features that appeared in earlier rounds also characterize the current evaluation set. The key observation is that the historical dataset (N≫nN n), being much larger than the current evaluation sample, provides a substantially more precise estimate of the covariate mean. This historical mean can therefore serve as a stable reference in a control-variate adjustment without changing the target parameter. Yet, the classical control variate adjustment does not transfer directly in our task, because the Step 1 estimator is a calibrated functional of EM posteriors rather than a sample mean. We thus propose to apply the control variate adjustment to the estimating equation that defines the Step 1 HERO estimator, with a shared covariate function whose population mean is supplied by the historical sample. Let XiX_i denote a covariate observed in the current evaluation task, and let m^X=1N∑r=1NX~r m_X= 1N _r=1^N X_r denote the sample mean of the same covariate in the historical data. We construct a control-variate estimator based on the score equation in (5). After obtaining the pilot HERO estimator μ^hero-1 μ_ hero-1 from Step 1, define the estimated score contribution for instance i as ψ^i:=ψi(μ^hero-1)=ℓi(1)−ℓi(0)μ^hero-1ℓi(1)+(1−μ^hero-1)ℓi(0). ψ_i:= _i( μ_ hero-1)= _i(1)- _i(0) μ_ hero-1 _i(1)+ (1- μ_ hero-1 ) _i(0). (6) We then estimate the control-variate coefficient by regressing the pilot score ψ^i ψ_i on XiX_i: γ^=Var^(X)−1Cov^(X,ψ^), γ= Var(X)^-1 Cov(X, ψ), (7) where Var^(X)=1n∑i=1n(Xi−X¯)(Xi−X¯)⊤ Var(X)= 1n _i=1^n(X_i- X)(X_i- X) and Cov^(X,ψ^)=1n∑i=1n(Xi−X¯)ψ^i Cov(X, ψ)= 1n _i=1^n(X_i- X) ψ_i. Whenever the covariate mean is shared between the historical and current datasets, the difference X¯−m^X X- m_X is asymptotically mean zero. Therefore, the adjustment term γ^⊤(X¯−m^X) γ ( X- m_X) can be used to reduce variance without changing the asymptotic expectation of the estimator. We define the control-variate adjusted estimator μ^hero-2 μ_ hero-2 as the solution to 1n∑i=1nψi(μ)−γ^⊤(X¯−m^X)=0. 1n _i=1^n _i(μ)- γ ( X- m_X)=0. (8) As in Step 1, this equation can also be solved using any standard one-dimensional optimizer. This adjustment reduces variance whenever the covariate XiX_i is correlated with the score contribution ψi _i, which quantifies how each observation influences the estimator. If XiX_i carries little information about the outcome, then γ γ is close to zero, and the estimator effectively reduces to μ^hero-1 μ_ hero-1. Consequently, HERO Step 2 improves efficiency when informative auxiliary covariates are available, while leaving the estimator essentially unchanged when they are not. 3.4 Theoretical Guarantees On top of being practical and easy to implement, HERO enjoys theoretical guarantees along three dimensions: reliability (Theorem 1), sensitivity (Theorem 2), and valid statistical inference (Theorem 3). Reliability means that HERO accurately targets the gold-label estimand, rather than a proxy estimand distorted by heterogeneous silver labelers. Sensitivity means that HERO reduces variance, which may improve the detection of small but practically meaningful model-performance differences. Valid statistical inference means that the estimator admits an asymptotic distribution, providing the basis for credible confidence intervals. Due to space limit, we defer the formal theorem statements and proofs to Appendix C. Figure 3: Experiment 1: (A) absolute bias, (B) Monte Carlo standard deviation, and (C) RMSE. 4 Simulation Study In this section, we evaluate the performance of HERO method against several benchmark methods. We summarize our takeaway as follows: First, HERO-1 remains accurate even when the historical gold-label rate is small, whereas the sample-proportion calibration baseline is unstable when only a few gold-routed examples are available per labeler. This shows HERO-1 can extract useful reliability information from the full historical label matrix, not only from the gold-labeled subset. Second, HERO-2 further improves sensitivity by reducing variance and RMSE relative to HERO-1 when the shared covariate is informative. Third, under increasing labeler heterogeneity, HERO-1 and HERO-2 remain substantially more robust than baseline methods, confirming that historical reliability calibration and score-level control variates reduce estimation errors. Due to the page limit, we defer the data generating process details to Appendix. In what follows, we introduce the methods in comparison, evaluation metrics, and simualtion study results. Methods in comparison. We compare five estimators: (1) Majority vote. (2) Dawid–Skene method with current data only: The current-only Dawid–Skene baseline fits the standard latent-class model using the current silver labels alone, jointly estimating μ and the labeler-specific parameters (αj,βj)j=1J\( _j, _j)\_j=1^J. (3) Historical sample-proportion calibration. We estimate each labeler’s sensitivity and specificity using only the gold-labeled subset of the historical data. With Laplace smoothing. We then maximize the current-data likelihood over μ with these calibration estimates fixed. (4) HERO-1 only use historical data for bias reduction. (5) HERO-2 applies the score-anchored control-variate adjustment to HERO-1. In this simulation, the population mean [X]=0E[X]=0 is known, but we use X¯hist X_hist to mirror the practical setting in which HERO relies on a high-precision historical estimate rather than oracle knowledge. Appendix Table 4 summarizes the simulation settings. The target μ0 _0 is computed from the data-generating model, μ0=Xσ(β0+β1X) _0=E_X\σ( _0+ _1X)\, and is fixed within each experimental setting. Experiment 1: Historical gold-label rate. The first experiment varies the amount of historical gold supervision. In Setting (i), we use a larger labeler pool and a stronger covariate signal: n=300n=300, N=800N=800, J=10J=10, and β1=3 _1=3. We vary ρ∈0.05,0.06,…,0.10ρ∈\0.05,0.06,…,0.10\. Experiment 2: Labeler heterogeneity. The second experiment studies robustness to heterogeneity in silver-labeler accuracy. We use the same baseline configuration as Experiment 1 Setting (i), with n=300n=300, N=800N=800, J=10J=10, β0=−0.2 _0=-0.2, and β1=3 _1=3. We hold the mean reliability fixed at μα=0.85 _α=0.85 and μβ=0.55 _β=0.55, and vary only the Beta concentration parameter k. Under the parametrization αj∼Betakμα,k(1−μα),βj∼Betakμβ,k(1−μβ), _j \k _α,k(1- _α)\,\ _j \k _β,k(1- _β)\, the means satisfy [αj]=μαE[ _j]= _α and [βj]=μβE[ _j]= _β for every k, while Var(αj)=μα(1−μα)k+1,Var(βj)=μβ(1−μβ)k+1.Var( _j)= _α(1- _α)k+1,\ Var( _j)= _β(1- _β)k+1. Thus, decreasing k increases labeler heterogeneity without changing the average sensitivity or specificity. We consider the five heterogeneity levels in Appendix Table 4. For each level, we evaluate historical gold-label rates ρ∈0.05,0.10ρ∈\0.05,0.10\. Figure 4: Experiment 2: Comparison of (A) absolute bias, (B) Monte Carlo standard deviation, and (C) RMSE as the Beta concentration k decreases, increasing the dispersion of labeler-specific sensitivities and specificities Evaluation metrics. For each configuration, we run B=200B=200 Monte Carlo replications. We report absolute bias, Monte Carlo standard deviation, and root mean squared error (RMSE). All EM algorithms use convergence tolerance 10−610^-6 and a maximum of 200 iterations. Simulation study results. Figure 3 shows that Majority vote has nearly constant bias as ρ varies, since it does not use historical gold labels. The current-only Dawid–Skene baseline also does not improve with ρ and remains limited by the identifiability and finite-sample challenges of estimating all labeler parameters from the small current round alone. The historical sample-proportion baseline improves as ρ increases, but it is unstable when the number of gold-routed examples per labeler is small. In contrast, HERO-1 is substantially less sensitive to low gold-label rates because its anchored EM step uses both the gold-routed and ungolded historical instances. HERO-2 further reduces SD and RMSE by exploiting the correlation between XiX_i and the score contribution, with larger variance reduction in the stronger-signal setting. We provide additional results in Appendix Figure 6. Figure 4 shows that majority vote becomes increasingly unreliable as labeler heterogeneity grows, because it assigns equal weight to labelers with different error profiles. The current-only Dawid–Skene baseline partially adapts to heterogeneous reliability but remains unstable because the current sample is small and unanchored by gold labels. Historical sample-proportion calibration improves when enough gold labels are available, but it can still have high variance when each labeler receives few gold-routed examples. HERO-1 maintains low bias across heterogeneity levels by pooling information from the full historical label matrix, and HERO-2 achieves the lowest RMSE by combining this calibration with score-level variance reduction. 5 Results in Real-World Benchmarking Data In this section, we evaluate the proposed HERO method on two real-world datasets provided by a technology company that routinely conducts large-scale generative model evaluations. The datasets were collected to assess the performance of 3D asset generation models and contain both historical and current evaluation data. The case studies cover two representative tasks: generation defect evaluation and safety evaluation. In each task, every prompt–asset pair is evaluated by a small number of randomly selected silver labelers according to the task-specific evaluation objective, and a random subset of pairs is additionally evaluated by a gold labeler, who is a well-trained expert. To assess statistical performance and robustness, we use a resampling-based approach to report absolute bias, SD, and RMSE for the five estimators described in Section 4. Additional details on the real-world datasets are provided in Appendix E. Estimator Mean estimate |Bias||Bias| SD RMSE Majority Vote 0.1100.110 0.1300.130 0.0190.019 0.1310.131 Dawid–Skene 0.1200.120 0.1200.120 0.0270.027 0.1230.123 MLE (hist. sample mean) 0.0690.069 0.1710.171 0.1210.121 0.2090.209 HERO-1 0.3020.302 0.0620.062 0.0700.070 0.0930.093 HERO-2 0.2880.288 0.0480.048 0.0640.064 0.0800.080 Table 2: Case study I (image defect detection), gold-label rate 0.100.10. Figure 5: Case study I: Results in model safety evaluation dataset. Table 2 and Figure 5 show that HERO method reduces absolute bias and RMSE relative to the baseline methods. This improvement is driven by the presence of labeler heterogeneity and labeler-pool shift between the historical and current datasets, as illustrated in Figure 1. HERO is designed to exploit precisely these sources of information by estimating historical labeler reliabilities and transferring them to the current evaluation task. Furthermore, HERO-2 achieves additional variance reduction through control variates. In Case Study I, shown in Table 2, HERO-2 uses GPT-5-predicted labels as the control variate, as all generated contents were evaluated by the same model. In Case Study I, shown in Figure 5, HERO-2 uses TF-IDF features as the control variate. In both cases, HERO-2 reduces the SD of HERO-1 by approximately 5%–10%, demonstrating that the proposed variance-reduction framework is effective across different choices of control variates. In contrast, the classical Dawid–Skene baseline does not effectively learn labeler heterogeneity in either case. As a result, it performs worse than HERO and is often closer to Majority Vote in terms of bias and RMSE. Limitation. Despite these promising real-world results, HERO Step 1 requires labeler reliability to transfer from historical data to the current evaluation task and Step 2 requires the shared covariates to have comparable distributions across the two periods. We view these requirements as mild, but they may not hold uniformly across model evaluation settings. References [1] D. Borkan, L. Dixon, J. Sorensen, N. Thain, and L. Vasserman (2019) Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, p. 491–500. Cited by: Appendix G. [2] D. Chen, R. Chen, S. Zhang, Y. Wang, Y. Liu, H. Zhou, Q. Zhang, Y. Wan, P. Zhou, and L. Sun (2024) Mllm-as-a-judge: assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning, Cited by: §1.1. [3] W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, et al. (2024) Chatbot arena: an open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132. Cited by: Appendix G, §1.1, §1.1. [4] J. Cohen (1960) A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20 (1), p. 37–46. Cited by: §2. [5] A. P. Dawid and A. M. Skene (1979) Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) 28 (1), p. 20–28. Cited by: Appendix G, §1.1, §2. [6] A. P. Dempster, N. M. Laird, and D. B. Rubin (1977) Maximum likelihood from incomplete data via the em algorithm. Journal of the royal statistical society: series B (methodological) 39 (1), p. 1–22. Cited by: Appendix D, Appendix D. [7] A. Deng, Y. Xu, R. Kohavi, and T. Walker (2013) Improving the sensitivity of online controlled experiments by utilizing pre-experiment data. In Proceedings of the sixth ACM international conference on Web search and data mining, p. 123–132. Cited by: Appendix G. [8] J. L. Fleiss (1971) Measuring nominal scale agreement among many raters. Psychological Bulletin 76 (5), p. 378–382. Cited by: §2. [9] B. Frénay and M. Verleysen (2013) Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems 25 (5), p. 845–869. Cited by: §1.1. [10] S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith (2020) Realtoxicityprompts: evaluating neural toxic degeneration in language models. In Findings of the association for computational linguistics: EMNLP 2020, p. 3356–3369. Cited by: Appendix G. [11] P. Glasserman (2004) Monte carlo methods in financial engineering. Vol. 53, Springer. Cited by: Appendix G. [12] J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, et al. (2024) A survey on llm-as-a-judge. The Innovation. Cited by: §1.1. [13] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al. (2022) Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. Cited by: Appendix G. [14] W. Lin (2013) Agnostic notes on regression adjustments to experimental data: reexamining freedman’s critique. The Annals of Applied Statistics, p. 295–318. Cited by: Appendix G. [15] C. Northcutt, L. Jiang, and I. Chuang (2021) Confident learning: estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research 70, p. 1373–1411. Cited by: §1.1. [16] A. B. Owen (2013-11) Monte carlo theory, methods and examples. Stanford University. External Links: Link Cited by: Appendix G. [17] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy (2010) Learning from crowds.. Journal of machine learning research 11 (4). Cited by: Appendix G, §1.1. [18] V. C. Raykar, S. Yu, L. H. Zhao, G. H. Valadez, C. Florin, L. Bogoni, and L. Moy (2010) Learning from crowds. Journal of Machine Learning Research 11, p. 1297–1322. Cited by: §2. [19] V. S. Sheng, F. Provost, and P. G. Ipeirotis (2008) Get another label? improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, p. 614–622. Cited by: Appendix G. [20] R. Snow, B. O’connor, D. Jurafsky, and A. Y. Ng (2008) Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, p. 254–263. Cited by: §1.1, §1.1, §1.1. [21] R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng (2008) Cheap and fast—but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 254–263. Cited by: §2. [22] G. Stein, J. Cresswell, R. Hosseinzadeh, Y. Sui, B. Ross, V. Villecroze, Z. Liu, A. L. Caterini, E. Taylor, and G. Loaiza-Ganem (2023) Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. Advances in Neural Information Processing Systems 36, p. 3732–3784. Cited by: §1.1. [23] A. W. Van der Vaart (2000) Asymptotic statistics. Vol. 3, Cambridge university press. Cited by: Appendix D, Appendix D. [24] S. K. Warfield, K. H. Zou, and W. M. Wells (2004) An algorithm for the validation of image segmentation. IEEE Transactions on Medical Imaging 23 (7), p. 903–921. Cited by: §2. [25] L. Weidinger, I. D. Raji, H. Wallach, M. Mitchell, A. Wang, O. Salaudeen, R. Bommasani, D. Ganguli, S. Koyejo, and W. Isaac (2025) Toward an evaluation science for generative ai systems. arXiv preprint arXiv:2503.05336. Cited by: §1.1. [26] P. Welinder, S. Branson, P. Perona, and S. Belongie (2010) The multidimensional wisdom of crowds. Advances in neural information processing systems 23. Cited by: Appendix G. [27] J. Whitehill, P. Ruvolo, T. Wu, J. Bergsma, and J. R. Movellan (2009) Whose vote should count more: optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems, Vol. 22, p. 2035–2043. Cited by: §2. [28] J. Whitehill, T. Wu, J. Bergsma, J. Movellan, and P. Ruvolo (2009) Whose vote should count more: optimal integration of labels from labelers of unknown expertise. Advances in neural information processing systems 22. Cited by: Appendix G, §1.1. [29] C. J. Wu (1983) On the convergence properties of the em algorithm. The Annals of statistics, p. 95–103. Cited by: Appendix D, Appendix D. [30] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. (2023) Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: Appendix G. Appendix A Appendix Appendix B Use cases Following the setup in Section 2 in the main manuscript, we provide more details on the four use cases for model evaluations: Use Case 1. Single Model Evaluation. This scenario focuses on evaluating the performance of a single model (such as defect rate), Model A. We define the outcome as the gold label for the model, Yi=Yi,A∗.Y_i=Y^*_i, A. The target parameter, which represents the average performance or expected quality, is defined as μA=[Yi] _ A=E[Y_i]. Use Case 2. Model Comparative Evaluation. This scenario independently evaluates two models, Model A and Model B, on the same prompts to calculate the absolute difference in their performance. We define the outcome as the difference in gold labels between the two models, Yi=Yi,A∗−Yi,B∗.Y_i=Y^*_i, A-Y^*_i, B. The target parameter for the average difference in performance is expressed as α=[Yi]α=E[Y_i]. Use Case 3. Side-by-Side Model Comparison. This scenario involves a direct, simultaneous comparison between outputs from multiple models to establish a relative preference. We define the outcome YiY_i as a binary indicator of preference rather than an absolute score, such that it equals 11 if the gold labeler prefers Model A over Model B, and 0 otherwise. This can be formulated as Yi=(Yi,A∗≻Yi,B∗).Y_i= 1(Y^*_i, A Y^*_i, B). The target parameter, representing the win rate or expected preference for Model A, is πA≻B=[Yi] _ A B=E[Y_i]. Use Case 4. Safety Moderation Calibration. This scenario treats the online safety moderation system as an AI automated rater and the offline human review as the gold truth. We let YiAI∈0,1Y_i AI∈\0,1\ represent the online model’s evaluation, where YiAI=1Y_i AI=1 indicates the content is flagged as unsafe. We let Yi∗∈0,1Y_i^*∈\0,1\ represent the offline human-labeled ground truth, where Yi∗=1Y_i^*=1 indicates an actual safety violation. We define the outcome as the intersection of the gold and AI labels, representing a true positive flag, Yi=Yi∗YiAI.Y_i=Y_i^*Y_i AI. The target parameters include the actual violation percentage or prevalence base rate, p=[Yi∗]p=E[Y_i^*]. Additionally, we estimate the online model precision, which is the probability of a true violation given the model flagged it, calculated as Precision=[Yi][YiAI]Precision= E[Y_i]E[Y_i AI]. The online model recall, which is the probability the model flagged the content given it is a true violation, is defined as Recall=[Yi][Yi∗]Recall= E[Y_i]E[Y_i^*]. Appendix C HERO Algorithmic Presentation Please see Algorithm 1. Input: Historical data (Z~i,A~i,R~i,Y~i,X~i)i=1N\( Z_i, A_i, R_i, Y_i, X_i)\_i=1^N; Current data (Zi,Ai,Xi)i=1n\(Z_i,A_i,X_i)\_i=1^n Output: μ^HERO μ_HERO 1 2 /* Stage A. Initialize from the gold-routed historical subset and maximize ℒ~(μ~,αj,βj) L( μ,\,\ _j, _j\) via EM */ 3 Let =i:R~i=1G=\i: R_i=1\. Initialize, for j=1,…,mj=1,…,m, μ~←||−1∑i∈Y~i,αj←∑i∈A~ijY~iZ~ij∑i∈A~ijY~i,βj←∑i∈A~ij(1−Y~i)(1−Z~ij)∑i∈A~ij(1−Y~i). μ←|G|^-1 _i Y_i, _j← _i A_ij Y_i Z_ij _i A_ij Y_i, _j← _i A_ij(1- Y_i)(1- Z_ij) _i A_ij(1- Y_i). Clip all initial values to [δ,1−δ][δ,1-δ] for a small δ>0δ>0 4 repeat 5 E-step: 6 foreach i=1,…,Ni=1,…,N do 7 if R~i=1 R_i=1 then γ~i←Y~i γ_i← Y_i 8 else γ~i←μ~ℓ~i(1)/[μ~ℓ~i(1)+(1−μ~)ℓ~i(0)] γ_i← μ\, _i(1) / [ μ\, _i(1)+(1- μ)\, _i(0) ] 9 10 M-step: 11 μ~←N−1∑i=1Nγ~i μ← N^-1 _i=1^N γ_i 12 foreach j=1,…,mj=1,…,m do 13 αj←∑iA~ijγ~iZ~ij∑iA~ijγ~i,βj←∑iA~ij(1−γ~i)(1−Z~ij)∑iA~ij(1−γ~i). _j← _i A_ij γ_i Z_ij _i A_ij γ_i, _j← _i A_ij(1- γ_i)(1- Z_ij) _i A_ij(1- γ_i). Clip αj,βj _j, _j to [δ,1−δ][δ,1-δ] 14 15 Clip μ~ μ to [δ,1−δ][δ,1-δ] 16 17until relative change in ℒ~ L <ϵ=o(N−1)<ε=o(N^-1) 18Store (α^j,β^j)j=1m( α_j, β_j)_j=1^m 19 /* Stage B. Maximize L(μ)L(μ) with (α^j,β^j)( α_j, β_j) */ 20 foreach i=1,…,ni=1,…,n do 21 ℓi(1)←∏j:Aij=1α^jZij(1−α^j)1−Zij _i(1)← _j:A_ij=1 α_j^Z_ij(1- α_j)^1-Z_ij, ℓi(0)←∏j:Aij=1(1−β^j)Zijβ^j1−Zij _i(0)← _j:A_ij=1(1- β_j)^Z_ij β_j^1-Z_ij 22 23μ^HERO-1←argmaxμ∈(0,1)∑i=1nlog[μℓi(1)+(1−μ)ℓi(0)] μ_HERO-1← _μ∈(0,1) _i=1^n [μ\, _i(1)+(1-μ)\, _i(0) ] 24 /* Stage C. Variance reduction via control variates */ 25 if no shared covariates available then return μ^HERO-1 μ_HERO-1 26 m^X←N−1∑r=1NX~r m_X← N^-1 _r=1^N X_r, X¯←n−1∑i=1nXi X← n^-1 _i=1^nX_i 27 foreach i=1,…,ni=1,…,n do 28 ψ^i←ℓi(1)−ℓi(0)μ^HERO-1ℓi(1)+(1−μ^HERO-1)ℓi(0) ψ_i← _i(1)- _i(0) μ_HERO-1\, _i(1)+(1- μ_HERO-1)\, _i(0) 29 30γ^←[1n∑i=1n(Xi−X¯)(Xi−X¯)⊤]−1[1n∑i=1n(Xi−X¯)ψ^i] γ← [ 1n _i=1^n(X_i- X)(X_i- X) ]^-1 [ 1n _i=1^n(X_i- X) ψ_i ] μ^HERO-2←Find in (0,1) such that μ_HERO-2 in (0,1) such that 1n∑i=1nℓi(1)−ℓi(0)μℓi(1)+(1−μ)ℓi(0)−γ^⊤(X¯−m^X)=0 1n _i=1^n _i(1)- _i(0)μ\, _i(1)+(1-μ)\, _i(0)- γ ( X- m_X)=0 return μ^HERO-2 μ_HERO-2 Algorithm 1 HERO Appendix D Theoretical Guarantees of HERO This section establishes the theoretical guarantees of HERO. The theoretical results rely on three assumptions on the labeling mechanism, the transportability of labeler reliability between historical and current data, and the sampling structure. Lemma 1 shows that the reliability estimates α^j α_j and β^j β_j converge at rate Op(N−1/2)O_p(N^-1/2) when historical gold labels anchor the EM, and Lemma 2 shows that the same EM algorithm without a gold anchor may fail to recover the true labeler parameters. Three theorems then state the main results. Theorem 1 shows that HERO-1 attains the parametric rate Op(n−1/2)O_p(n^-1/2) for the gold-label target, while the majority-vote and unanchored Dawid–Skene baselines are not guaranteed to converge to that target. Theorem 2 shows that HERO-2 reduces the asymptotic variance of HERO-1 whenever the shared covariate correlates with the score function, and preserves the bias of HERO-1 up to higher-order terms. Theorem 3 establishes asymptotic normality for both estimators, providing the basis for standard error estimation and Wald-type confidence intervals. Assumption 1 (Randomized labeling) The silver-label availability indicators AijA_ij and A~ij A_ij are completely randomized, satisfying P(Aij=1)≥πmin>0,P(A~ij=1)≥πmin>0,P(A_ij=1)≥ _ >0, P( A_ij=1)≥ _ >0, and the historical gold-routing indicators R~i R_i are completely randomized, satisfying P(R~i=1)=ρ≥ρmin>0,P( R_i=1)=ρ≥ _ >0, where πmin _ and ρmin _ are positive constants. Assumption 2 (Correct specification and transportable reliability) For each labeler j, there exist constants (αj,βj)( _j, _j) satisfying c<αj,βj<1−c< _j, _j<1-c for some c>0c>0 such that the conditional reliabilities are identical in historical and current data: Pr(Z~ij=1∣Y~i=1)=Pr(Zij=1∣Yi=1)=αj, ( Z_ij=1 Y_i=1)= (Z_ij=1 Y_i=1)= _j, Pr(Z~ij=0∣Y~i=0)=Pr(Zij=0∣Yi=0)=βj, ( Z_ij=0 Y_i=0)= (Z_ij=0 Y_i=0)= _j, for all instances i and all labelers j. The silver labels are independent across labelers conditional on the gold label: Pr(Z~i1=z1,…,Z~iJ=zJ|Y~i=y) ( Z_i1=z_1,…, Z_iJ=z_J\; |\; Y_i=y ) =∏j=1mPr(Z~ij=zj∣Y~i=y), = _j=1^m ( Z_ij=z_j Y_i=y), (9) Pr(Zi1=z1,…,ZiJ=zJ|Yi=y) (Z_i1=z_1,…,Z_iJ=z_J\; |\;Y_i=y ) =∏j=1mPr(Zij=zj∣Yi=y), = _j=1^m (Z_ij=z_j Y_i=y), (10) for all instances i and labelers j=1,…,mj=1,…,m. The parameters μ=[Y]μ=E[Y] and μ~=[Y~] μ=E[ Y] may differ, but are bounded away from 0 and 11: c<μ,μ~<1−c.c<μ, μ<1-c. Lemma 1 (Convergence of HERO historical reliability estimates) Under Assumptions 1 and 2, the estimators for αj _j and βj _j in HERO algorithm satisfy α^j−αj=Op(N−1/2),β^j−βj=Op(N−1/2),for each labeler j. α_j- _j=O_p(N^-1/2), β_j- _j=O_p(N^-1/2), each labeler j. Proof. Let =i:R~i=1G=\i: R_i=1\ denote the set of gold-routed historical instances. By the algorithm, the EM is initialized using the gold-only sample-mean estimators μ~(0)=1||∑i∈Y~i,α^j(0)=∑i∈A~ijY~iZ~ij∑i∈A~ijY~i,β^j(0)=∑i∈A~ij(1−Y~i)(1−Z~ij)∑i∈A~ij(1−Y~i),j=1,…,m. μ^(0)= 1|G| _i Y_i, α_j^(0)= _i A_ij Y_i Z_ij _i A_ij Y_i, β_j^(0)= _i A_ij(1- Y_i)(1- Z_ij) _i A_ij(1- Y_i), j=1,…,m. Under Assumption 1, the gold-routing indicators R~i R_i are completely randomized with P(R~i=1)=ρ≥ρmin>0P( R_i=1)=ρ≥ _ >0 and the silver-label availability indicators A~ij A_ij are completely randomized with P(A~ij=1)≥πmin>0P( A_ij=1)≥ _ >0, so by the law of large numbers and the delta method, for each j, α^j(0)−αj=Op((Nρπj)−1/2),β^j(0)−βj=Op((Nρπj)−1/2),μ~(0)−μ~=Op((Nρ)−1/2), α_j^(0)- _j=O_p ((Nρ _j)^-1/2 ), β_j^(0)- _j=O_p ((Nρ _j)^-1/2 ), μ^(0)- μ=O_p ((Nρ)^-1/2 ), which implies that the initialization lies within a neighborhood of the gold-oriented population parameter η0=(μ~,α1,β1,…,αm,βm) _0=( μ, _1, _1,…, _m, _m). Under Assumption 2, the historical data follow the Dawid–Skene model with true gold-oriented parameters η0=(μ~,α1,β1,…,αm,βm) _0=( μ, _1, _1,…, _m, _m), so that for each i and j the conditional distributions of Z~ij Z_ij given Y~i Y_i are correctly specified and independent across labelers. The observed-data log-likelihood for the historical dataset is ℒ~N(μ~,αj,βj)=∑i=1N[R~ilogℓ~i(Y~i)+(1−R~i)log(μ~ℓ~i(1)+(1−μ~)ℓ~i(0))]. L_N( μ,\ _j, _j\)= _i=1^N [ R_i _i( Y_i)+(1- R_i) ( μ _i(1)+(1- μ) _i(0) ) ]. By construction, this likelihood is twice continuously differentiable in a neighborhood of η0 _0, and the Fisher information matrix ℐη=−E[∂2∂η∂η⊤logP(Z~i∣Y~i)|η=η0]I_η=-E [ ∂^2∂η∂η P( Z_i Y_i) |_η= _0 ] is nonsingular because each labeler contributes a positive amount of information and αj,βj∈(c,1−c) _j, _j∈(c,1-c) for some c>0c>0, ensuring strong identifiability. By standard results on EM for latent-variable models [6, 29], the EM map is monotone in the log-likelihood, i.e., ℒ~N(η(t+1))≥ℒ~N(η(t)), L_N(η^(t+1))≥ L_N(η^(t)), and converges to a stationary point of ℒ~N L_N. Since the initialization η(0)η^(0) is obtained from the gold-only sample mean and satisfies ‖η(0)−η0‖=Op((Nρπmin)−1/2)\|η^(0)- _0\|=O_p((Nρ _ )^-1/2), it lies within the basin of attraction of the gold-oriented local maximizer η^N η_N, and therefore the EM iterates converge to η^N η_N. By the stopping criterion ϵ=o(N−1)ε=o(N^-1), the EM optimization error satisfies ‖η(T)−η^N‖=op(N−1/2),\|η^(T)- η_N\|=o_p(N^-1/2), so that it is negligible relative to the statistical variability of the maximizer. Finally, applying standard M-estimation theory for local maximizers of smooth likelihoods [23], we have η^N−η0=Op(N−1/2), η_N- _0=O_p(N^-1/2), and combining with the EM optimization error yields α^j−αj=Op(N−1/2),β^j−βj=Op(N−1/2),μ^−μ~=Op(N−1/2),j=1,…,m. α_j- _j=O_p(N^-1/2), β_j- _j=O_p(N^-1/2), μ- μ=O_p(N^-1/2), j=1,…,m. ∎ Lemma 2 (EM without gold anchor may fail) Consider the Dawid–Skene EM applied to historical data or current data without using any gold labels. Then the EM does not necessarily recover the true parameters α and β. Proof. Let the observed-data likelihood for the unsupervised DS problem be ℒ~N(μ,αj,βj)=∑i=1Nlog(μℓ~i(1)+(1−μ)ℓ~i(0)), L_N(μ,\ _j, _j\)= _i=1^N (μ _i(1)+(1-μ) _i(0) ), with ℓ~i(1)=∏j:A~ij=1αjZ~ij(1−αj)1−Z~ij,ℓ~i(0)=∏j:A~ij=1(1−βj)Z~ijβj1−Z~ij. _i(1)= _j: A_ij=1 _j Z_ij(1- _j)^1- Z_ij, _i(0)= _j: A_ij=1(1- _j) Z_ij _j^1- Z_ij. Consider the second derivative of the likelihood with respect to any αj _j: ∂2∂αj2log(μℓ~i(1)+(1−μ)ℓ~i(0))=−μ2(∂αjℓ~i(1))2+μ(μ∂αj2ℓ~i(1)+(1−μ)∂αj2ℓ~i(0))ℓ~i(1)(μℓ~i(1)+(1−μ)ℓ~i(0))2, ∂^2∂ _j^2 (μ _i(1)+(1-μ) _i(0) )= -μ^2 ( _ _j _i(1) )^2+μ (μ _ _j^2 _i(1)+(1-μ) _ _j^2 _i(0) ) _i(1)(μ _i(1)+(1-μ) _i(0))^2, which depends on both ℓ~i(1) _i(1) and ℓ~i(0) _i(0). Since ℓ~i(0) _i(0) depends nonlinearly on βj _j and ℓ~i(1) _i(1) depends on αj _j, the Hessian is not negative semidefinite in general. Therefore, ℒ~N L_N is not concave in (μ,αj,βj)(μ, _j, _j). Because EM is a local algorithm, it is only guaranteed to converge to a stationary point of ℒ~N L_N [6, 29]. Hence, EM without a gold anchor does not guarantee consistent recovery of the true maxima. ∎ Theorem 1 (Estimation rate and comparison with baselines) Under Assumptions 1 and 2, the HERO-1 estimator satisfies μ^HERO-1−μ=Op(n−1/2)+Op(N−1/2). μ_HERO-1-μ=O_p(n^-1/2)+O_p(N^-1/2). In contrast, the Majority Vote estimator and the classical Dawid–Skene EM estimator without gold anchor do not necessarily converge to the gold-oriented truth. Proof. Decompose the HERO-1 estimation error as μ^HERO-1−μ=(μ^HERO-1−μ∗)+(μ∗−μ), μ_HERO-1-μ=( μ_HERO-1-μ^*)+(μ^*-μ), where μ∗=argmaxμ∈(0,1)∑i=1nlog[μℓi(1;αj,βj)+(1−μ)ℓi(0;αj,βj)]μ^*= _μ∈(0,1) _i=1^n [μ _i(1; _j, _j)+(1-μ) _i(0; _j, _j) ] is the oracle MLE computed using the true labeler parameters (αj,βj)( _j, _j). By standard MLE theory [23], the oracle MLE satisfies μ∗−μ=Op(n−1/2),μ^*-μ=O_p(n^-1/2), because the log-likelihood for the current data is smooth and concave in μ, and the n instances are independent. From Lemma 1, the historical HERO EM estimates satisfy α^j−αj=Op(N−1/2),β^j−βj=Op(N−1/2),j=1,…,m. α_j- _j=O_p(N^-1/2), β_j- _j=O_p(N^-1/2), j=1,…,m. Since the current-data likelihood is smooth in (αj,βj)( _j, _j), a first-order Taylor expansion gives ℓi(1;α^j)−ℓi(1;αj)=Op(N−1/2),ℓi(0;β^j)−ℓi(0;βj)=Op(N−1/2), _i(1; α_j)- _i(1; _j)=O_p(N^-1/2), _i(0; β_j)- _i(0; _j)=O_p(N^-1/2), uniformly in i. Applying the delta method to the MLE equation for μ^HERO-1 μ_HERO-1, we obtain μ^HERO-1−μ∗=Op(N−1/2). μ_HERO-1-μ^*=O_p(N^-1/2). Combining the two contributions via the triangle inequality yields μ^HERO-1−μ=Op(n−1/2)+Op(N−1/2). μ_HERO-1-μ=O_p(n^-1/2)+O_p(N^-1/2). ∎ Assumption 3 (Instance sampling and moments) The current instances i=1,…,n\i=1,…,n\ and historical instances i=1,…,N\i=1,…,N\ are independently and identically distributed from the same population, with sample size N≫nN n. The HERO-1 score contributions ψi(μ0) _i( _0) and covariates XiX_i have finite second moments: [ψi(μ0)2]<∞,[Xi2]<∞.E[ _i( _0)^2]<∞, [X_i^2]<∞. Theorem 2 (Variance reduction of HERO-2 over HERO-1) Under Assumption 3, the asymptotic variances of the HERO-1 and HERO-2 estimators satisfy (μ^hero-2)=(μ^hero-1)−ℐ0−2ℂovψi(μ0),Xi⊤ΣX−1ℂovXi,ψi(μ0),V( μ_ hero-2)=V( μ_ hero-1)-I_0^-2Cov\ _i( _0),X_i\ _X^-1Cov\X_i, _i( _0)\, where ℐ0=−∂μ[ψi(μ)]|μ=μ0>0I_0=- _μE[ _i(μ)] |_μ= _0>0. Consequently, (μ^hero-2)≤(μ^hero-1),V( μ_ hero-2) ( μ_ hero-1), with strict reduction whenever ℂovXi,ψi(μ0)≠0Cov\X_i, _i( _0)\≠ 0. Moreover, HERO-2 preserves the bias reduction of HERO-1: Bias(μ^hero-2)=Bias(μ^hero-1)+O(N−1/2)+o(n−1/2).Bias( μ_ hero-2)=Bias( μ_ hero-1)+O(N^-1/2)+o(n^-1/2). Proof. We first consider the oracle HERO-2 estimator using the true control-variate coefficient γ∗γ^* and the population mean mXm_X. Linearizing the score equation around the true parameter μ0 _0, we have μ^hero-2−μ0=I0−11n∑i=1n(ψi(μ0)−γ∗⊤(Xi−mX))+Rn, μ_ hero-2- _0=I_0^-1 1n _i=1^n ( _i( _0)-γ^* (X_i-m_X) )+R_n, where I0=−∂μ[ψi(μ)]|μ=μ0>0I_0=- _μE[ _i(μ)] |_μ= _0>0 and Rn=op(n−1/2)R_n=o_p(n^-1/2) is the remainder from the Taylor expansion of the score function. By the standard projection property of linear regression, the variance decomposes as Var[ψi(μ0)−γ∗⊤(Xi−mX)]=Var[ψi(μ0)]−Cov(ψi(μ0),Xi)⊤ΣX−1Cov(Xi,ψi(μ0))≤Var[ψi(μ0)],Var [ _i( _0)-γ^* (X_i-m_X) ]=Var[ _i( _0)]-Cov( _i( _0),X_i) _X^-1Cov(X_i, _i( _0)) [ _i( _0)], with strict inequality whenever Cov(ψi(μ0),Xi)≠0Cov( _i( _0),X_i)≠ 0. Using Assumption 3, the historical mean m^X m_X satisfies m^X−mX=Op(N−1/2) m_X-m_X=O_p(N^-1/2) and the estimated coefficient γ γ satisfies γ^−γ∗=op(1) γ-γ^*=o_p(1). These contribute only a higher-order term of Op(N−1/2)+op(n−1/2)O_p(N^-1/2)+o_p(n^-1/2) to the asymptotic expansion of μ^hero-2 μ_ hero-2 since N≫nN n, which does not affect the first-order variance reduction. Therefore, replacing mXm_X by m^X m_X and γ∗γ^* by γ γ preserves the variance reduction, and the bias satisfies Bias(μ^hero-2)=Bias(μ^hero-1)+O(N−1/2)+o(n−1/2).Bias( μ_ hero-2)=Bias( μ_ hero-1)+O(N^-1/2)+o(n^-1/2). This proves that HERO-2 strictly reduces variance relative to HERO-1 while preserving its bias properties. ∎ Theorem 3 (Asymptotic normality of HERO-1 and HERO-2) Suppose Assumptions 1–3 hold, then HERO-1 satisfies n(μ^hero-1−μ0)→(0,ℐ0−2arψi(μ0)), n( μ_ hero-1- _0) dN (0,\,I_0^-2Var\ _i( _0)\ ), and HERO-2 satisfies n(μ^hero-2−μ0)↝N(0,ℐ0−2arψi(μ0)−γ∗⊤(Xi−mX)). n( μ_ hero-2- _0) N (0,\,I_0^-2Var\ _i( _0)-γ^* (X_i-m_X)\ ). Proof. For HERO-1, consider first the idealized MLE using the true labeler reliabilities αj _j and βj _j. The score equation for μ is smooth and strictly concave, so by the classical M-estimator theory, we have n(μ^hero-1(α,β)−μ0)→(0,ℐ0−2Varψi(μ0)), n ( μ_ hero-1(α,β)- _0 ) dN (0,I_0^-2Var\ _i( _0)\ ), where ψi(μ0) _i( _0) is the per-instance score and ℐ0=−∂μ[ψi(μ)]|μ=μ0>0I_0=- _μE[ _i(μ)]|_μ= _0>0. By Lemma 1, the estimated α^j,β^j α_j, β_j from the historical data satisfy α^j−αj=Op(N−1/2) α_j- _j=O_p(N^-1/2), β^j−βj=Op(N−1/2) β_j- _j=O_p(N^-1/2). Using Assumption 3 that N≫nN n, the contribution of the estimation error to the μ-MLE is I0−1∂ψi∂(α,β)(α^−α,β^−β)=op(n−1/2),I_0^-1 ∂ _i∂(α,β)( α-α, β-β)=o_p(n^-1/2), so n(μ^hero-1−μ0) n( μ_ hero-1- _0) has the same asymptotic distribution as the idealized MLE. For HERO-2, consider first the oracle estimator using the true control-variate coefficient γ∗γ^* and population mean mXm_X. Linearizing the adjusted score equation, we have n(μ^hero-2(γ∗,mX)−μ0)=ℐ0−11n∑i=1n(ψi(μ0)−γ∗⊤(Xi−mX))+op(1), n ( μ_ hero-2(γ^*,m_X)- _0 )=I_0^-1 1 n _i=1^n ( _i( _0)-γ^* (X_i-m_X) )+o_p(1), which immediately yields the asymptotic normality with variance ℐ0−2Var[ψi(μ0)−γ∗⊤(Xi−mX)]I_0^-2Var[ _i( _0)-γ^* (X_i-m_X)]. Finally, by the argument in the proof of Theorem 2, replacing γ∗γ^* and mXm_X by their estimators γ γ and m^X m_X contributes only op(1)o_p(1) to the n n-scale expansion. Hence the asymptotic distribution of HERO-2 is unchanged. This completes the proof. ∎ Appendix E Additional simulation details and results In this section, we shall provide additional simulation setup details in Section E.1 and additional simulation results in Section E.2. E.1 Simulation setup additional details. In Table 3, we shall provide the details of how we define heterogeneity level in Experiment 2. In Table 4, we summarize the simulation parameters used in the main manuscript. In what follows, we shall describe the data generating process. In each Monte Carlo replication, we generate two datasets: a large historical evaluation sample of size N and a smaller current evaluation sample of size n. The two samples share the same prompt-feature distribution and the same pool of silver labelers, but gold labels are observed only for a randomized subset of historical instances. This construction matches the intended deployment setting of HERO, where historical gold labels are used to learn persistent labeler reliability and historical covariates provide a high-precision control-variate anchor. For each instance i, we draw a scalar covariate Xi∼(0,1),X_i (0,1), and generate the latent gold label from the logistic model Yi∣Xi∼Bernoullip(Xi),Y_i X_i \p(X_i)\, p(x)=σ(β0+β1x)p(x)=σ( _0+ _1x), σ(t)=11+exp(−t).σ(t)= 11+ (-t). The estimand is therefore μ0:=Xσ(β0+β1X). _0:=E_X\σ( _0+ _1X)\. We set β0=−0.2 _0=-0.2. The slope β1 _1 controls the correlation between the auxiliary covariate XiX_i and the latent label YiY_i: larger β1 _1 makes XiX_i more informative and therefore increases the potential gain from the HERO-2 control-variate adjustment. Each silver labeler j∈1,…,Jj∈\1,…,J\ has a sensitivity αj _j and specificity βj _j, αj=Pr(Zij=1∣Yi=1),βj=Pr(Zij=0∣Yi=0). _j= (Z_ij=1 Y_i=1), _j= (Z_ij=0 Y_i=0). To induce persistent labeler heterogeneity, we draw αj∼iidBetakμα,k(1−μα),βj∼iidBetakμβ,k(1−μβ). _j iid Beta\k _α,k(1- _α)\, _j iid Beta\k _β,k(1- _β)\. Throughout Experiment 1 we use μα=0.85 _α=0.85, μβ=0.55 _β=0.55, and k=8k=8. Thus, the average silver labeler has high sensitivity but only moderate specificity, creating an asymmetric error pattern under which uncalibrated majority vote is expected to be biased. The concentration parameter k controls across-labeler heterogeneity: larger k yields a more homogeneous labeler pool around the same mean reliability, whereas smaller k yields more dispersed labeler-specific error rates. Each labeler j annotates instance i with probability Aij∼Bernoulli(πj),πj∼Uniform(0.2,0.6),A_ij ( _j),\ _j (0.2,0.6), independently across instances conditional on πj _j. If Aij=1A_ij=1, the observed silver label is generated as Zij∣Yi=Bernoulli(αj),Z_ij Y_i=Bernoulli( _j), if Yi=1Y_i=1, and Zij∣Yi=Bernoulli(1−βj)Z_ij Y_i=Bernoulli(1- _j), if Yi=0Y_i=0. Instances with no observed silver labels are excluded from all estimators; this event is rare in the reported settings and affects all methods equally. The historical and current datasets are generated from the same mechanism, with independent instances but a common labeler pool. For the historical sample, a randomized routing indicator Ri∼Bernoulli(ρ)R_i (ρ) determines whether the latent gold label is revealed. The parameter ρ is the historical gold-label rate. For the current sample, Ri=0R_i=0 for all instances, so no current gold labels are observed. Table 3: Heterogeneity levels used in Experiment 2. Smaller k corresponds to greater across-labeler dispersion while preserving the same mean sensitivity and specificity. Category k [αj]E[ _j] SD(αj)SD( _j) [βj]E[ _j] SD(βj)SD( _j) Very low 150150 0.850.85 0.0290.029 0.550.55 0.0400.040 Low 3030 0.850.85 0.0640.064 0.550.55 0.0890.089 Moderate 1212 0.850.85 0.0990.099 0.550.55 0.1380.138 High 66 0.850.85 0.1350.135 0.550.55 0.1880.188 Extreme 33 0.850.85 0.1790.179 0.550.55 0.2490.249 Table 4: Simulation parameters. Simulation parameters used in the Monte Carlo experiments. Parameter Value Description n 150150 Current instances in Experiment 1 Setting (i) n 300300 Current instances in Experiment 1 Setting (i) and Experiment 2 N 400400 Historical instances in Experiment 1 Setting (i) N 800800 Historical instances in Experiment 1 Setting (i) and Experiment 2 J 55 Number of labelers in Experiment 1 Setting (i) J 1010 Number of labelers in Experiment 1 Setting (i) and Experiment 2 β0 _0 −0.2-0.2 Logistic intercept β1 _1 1.01.0 Logistic slope in Experiment 1 Setting (i) β1 _1 3.03.0 Logistic slope in Experiment 1 Setting (i) and Experiment 2 μα _α 0.850.85 Mean silver-labeler sensitivity μβ _β 0.550.55 Mean silver-labeler specificity k 88 Baseline Beta concentration in Experiment 1 πj _j Uniform(0.2,0.6)Uniform(0.2,0.6) Labeler-specific annotation probability ρ 0.01,0.02,…,0.10\0.01,0.02,…,0.10\ Historical gold-label rate in Experiment 1 Setting (i) ρ 0.05,0.06,…,0.10\0.05,0.06,…,0.10\ Historical gold-label rate in Experiment 1 Setting (i) k 150,30,12,6,3\150,30,12,6,3\ Beta concentration levels in Experiment 2 ρ 0.05,0.10\0.05,0.10\ Historical gold-label rates in Experiment 2 B 200200 Monte Carlo replications E.2 Additional simulation results In what follows, we show another setting of Experiment 1. In this setting, we use a small current sample and a moderate covariate signal: n=150n=150, N=400N=400, J=5J=5, and β1=1 _1=1. We vary the historical gold-label rate over ρ∈0.01,0.02,…,0.10ρ∈\0.01,0.02,…,0.10\. Figure 6: Experiment 1 Setting (i): Comparison of (A) absolute bias, (B) variance, and (C) MSE with respect to various gold label rate. β1=1 _1=1, J=5J=5, n=150n=150, N=400N=400 Effect of historical gold-label rate in Setting (i). Panels report (A) absolute bias, (B) Monte Carlo standard deviation, and (C) RMSE for estimating μ0=[Yi] _0=E[Y_i]. The current sample is small and the covariate signal is moderate: β1=1 _1=1, J=5J=5, n=150n=150, and N=400N=400. Appendix F Additional details on real-world case studies 1. Asset defect detection dataset. This dataset is designed to evaluate generation defects in 3D generation models. It contains data from two versions of the model: the historical dataset consists of outputs generated by the older model version, while the current dataset consists of outputs generated by the newer model version. Both model versions generate 3D assets from the same set of 532 prompts. For each prompt–asset pair, silver labelers evaluate whether the generated asset contains a specific type of defect. The silver labels are binary, with Yes/No responses encoded as 1/0. There are 10 silver labelers in total, and each prompt–asset pair is evaluated by three randomly selected, distinct silver labelers. In addition, a completely random 10% subset of prompt–asset pairs is evaluated by a golden labeler, who is a well-trained expert. Most prompt–asset pairs are also evaluated by an AI labeler, GPT-5, which is prompted to provide the same binary defect label. Shared control variate. For the HERO-2 method, we use the AI labels generated by GPT-5 as control variates. We repeat this procedure 100 times and report the absolute bias, standard deviation, and root mean squared error (RMSE) based on the resampled results, following the similar evaluation protocol as in the Simulation Study. 2. Safety evaluation dataset. This dataset contains both a historical dataset and a current dataset for evaluating the safety of a 3D generation model with respect to policy violation categories. The historical dataset corresponds to evaluations along one safety dimension, whereas the current dataset corresponds to evaluations along another safety dimension. The model generates 3D assets from 2,046 prompts. For each prompt–asset pair, silver labelers provide binary safety labels along the corresponding safety dimension. There are 11 silver labelers in total, and each prompt–asset pair is evaluated by a randomly selected set of 4–7 distinct silver labelers. In addition, a completely random 80% subset of prompt–asset pairs is evaluated by a golden labeler, who is a well-trained expert, to benchmark the performance of the silver labelers. Shared control variate. For HERO-2, we use TF-IDF features extracted from the prompts as control variates. Since TF-IDF (Term Frequency-Inverse Document Frequency) features are typically high-dimensional relative to the sample size, we apply a ridge penalty when estimating the coefficient γ in the score-based control-variate adjustment. As in the previous case study, we report the absolute bias, standard deviation, and RMSE for HERO and the baseline methods. Appendix G Comparison with existing literature In this section, we compare HERO with three related strands of work: learning from noisy annotators, variance reduction techniques, and generative AI evaluation. Learning from crowds and noisy annotators has a long history in statistics and machine learning. The classical Dawid–Skene (DS) model [5] introduced an EM-based approach to estimate individual annotator error rates from noisy labels. Subsequent work extended this idea to more complex scenarios, including latent variable models for multi-dimensional annotation [26], probabilistic truth inference [28], and crowdsourced aggregation with worker reliability [17, 19]. Most of these studies assume that all labelers are noisy, and they do not exploit the existence of a small set of high-quality gold labels for calibration, leaving a gap for methods that systematically combine historical gold-labeled subset with current noisy annotations. Variance reduction has been studied extensively in statistics and experimental design. Classical tools include control variates and regression adjustment, where auxiliary information is used to reduce estimation variance without bias [16, 14]. In applied fields, [7] uses pre‑experiment measurements to improve sensitivity in controlled experiments, and Monte Carlo methods with control variates have a rich theoretical literature [11]. These techniques reduce variance by exploiting correlated auxiliary variables, but, to our knowledge, they have not been systematically adapted to the noisy‑label setting. Evaluation of generative AI raises additional challenges. Human preference judgments and pairwise comparison metrics have been used in practice to assess LLM quality [30, 3], and broad benchmarking frameworks such as HELM [13] aggregate diverse evaluation metrics. Safety and robustness evaluations, such as toxicity benchmarks, further illustrate the complexity of human annotations [10, 1]. However, there is not yet a fully statistical framework that jointly accounts for annotator heterogeneity, leverages historical gold supervision, and reduces variance in large‑scale AI model evaluation.