Paper deep dive

BayesAME: Bayesian Active Model Evaluation

Paula Cordero Encinar, Taylan Cemgil, Arnaud Doucet, Virginia Aglietti, Silvia Chiappa

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 109

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 8/4/2026, 11:23:38 AM

Summary

The paper introduces BayesAME, a sequential Bayesian framework for automatically determining the coreset size in large-scale model evaluation. It models item performance via latent abilities grouped by historical reference model scores, using a joint prior to capture similarities. BayesAME iteratively selects items to maximize information gain, updating posterior distributions to estimate performance and uncertainty. A multi-target extension captures correlations across models. Experiments show BayesAME outperforms existing methods, validating non-random selection and the use of continuous log-likelihoods over binary scores.

Entities (8)

BayesAME · method · 98%Coreset · concept · 95%Reference Models · entity · 93%Target Models · entity · 93%Latent Ability · concept · 92%Information-Gain Criterion · method · 90%Linear Model of Coregionalization · method · 88%Log-Likelihoods · metric · 85%

Relation Signals (6)

BayesAME → uses → Latent Ability

confidence 95% · BayesAME models performance as a random variable by defining a latent ability for each group of items

BayesAME → uses → Information-Gain Criterion

confidence 92% · select items to add to the coreset via an information-gain criterion

Non-Random Selection → advantageousover → Random Selection

confidence 90% · establishing that non-random coreset selection is advantageous over random selection

BayesAME → outperforms → Sequential Adaptations of Existing Methods

confidence 90% · BayesAME consistently outperforms sequential adaptations of existing methods

BayesAME → extendsto → Linear Model of Coregionalization

confidence 88% · We capture cross-model correlations... using a linear model of coregionalization

Log-Likelihoods → enhances → Estimation Accuracy

confidence 85% · leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. We propose a multi-target extension that captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection. Finally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

107,569 characters extracted from source content.

Expand or collapse full text

BayesAME: Bayesian Active Model Evaluation Paula Cordero Encinar 1 , Taylan Cemgil 2 , Arnaud Doucet 2 , Virginia Aglietti *2 and Silvia Chiappa *2 1 Imperial College London (work conducted while at Google DeepMind), 2 Google DeepMind Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. Moreover, we propose a multi-target extension that actively captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection, and demonstrating that one can reliably improve upon the random sample mean baseline. Additionally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy. 1. Introduction The massive scale and frequency of generating responses render large generative model evaluation across benchmarks a time-consuming and computationally expensive process (Liang et al., 2023). To mitigate this, a growing number of methods estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Such methods mostly require the practitioner to input a coreset size (Berrada et al., 2025; Bowyer et al., 2026; Fisch et al., 2026; Hsu and Shekhar, 2026; Huang et al., 2026; Kipnis et al., 2025; Kossen et al., 2021; Liao et al., 2025; Liu et al., 2026; Perlitz et al., 2024; Saranathan et al., 2025; Vivek et al., 2024; Zhang et al., 2025). While this is most suitable when facing strict computational limits, in scenarios where reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. In addition, the predominant approach of leveraging evaluations from historical models (reference models) has largely operated within the regime in which the models being evaluated (target models) fall within the performance range of these reference models (interpolation regime). Zhang et al. (2025) underscored the importance of also testing efficient model evaluation methods in regimes where the target models significantly deviate from this range (extrapolation regimes) to obtain a more accurate assessment of a method’s robustness. In particular, through a study that considered both regimes, the authors questioned the utility of non-random coreset selection and highlighted the difficulty of outperforming the mean of a random coreset as a performance estimate in extrapolation. In this work, we focus on efficient model evaluation leveraging reference models, specifically targeting auto- matic determination of the coreset size. We propose BayesAME, a sequential Bayesian method that models *Senior co-authors. © 2026 Google DeepMind. All rights reserved arXiv:2607.27023v1 [cs.LG] 29 Jul 2026 BayesAME: Bayesian Active Model Evaluation performance as a random variable via underlying abilities whose joint prior distribution encodes the belief that the target model behaves similarly to the reference models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. In the setting where multiple target models are being evaluated, we capture correlation among them using a linear model of coregionalization (Álvarez et al., 2012; Journel and Huijbregts, 1978) with the goal of further reducing the coreset size. To validate our approach, we comprehensively evaluate BayesAME against sequential adaptations of existing methods. Our main contributions are summarized as follows: •We formulate efficient model evaluation with automatic coreset-size determination as a Bayesian sequential process that augments the coreset until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. •We develop a computationally efficient method, equipped with robust performance estimators possessing well-defined properties. • We design active selection strategies that identify benchmark items maximizing information gain. • We extend the method to capture correlations across multiple target models to further reduce the coreset size. • We conduct an extensive empirical analysis across varying numbers of reference models, interpola- tion/extrapolation regimes, and different degrees of target model correlation, showing that BayesAME outperforms sequential adaptations of existing methods designed to automatically determine the coreset size. We further show that BayesAME remains advantageous even when the coreset size is predetermined. •Crucially, this analysis overturns recent claims in the literature by demonstrating that non-random coreset selection does improve performance and that several methods can reliably outperform the random sample mean baseline. •We demonstrate that leveraging available response log-likelihoods, rather than using binary scores, greatly improves performance estimation. 2. Problem Setup Consider a benchmark퐸=(푥 푖 , 푦 푖 ) 푁 푖=1 , where푥 푖 is the푖-th input instance and푦 푖 is the corresponding ground- truth response. LetT=푡 1 , . . . , 푡 퐾 푡 be a set of퐾 푡 target models that we wish to evaluate, and let푠 푡 푖 be the score of target model푡 ∈ Ton item(푥 푖 , 푦 푖 ). We consider the problem of estimating the full benchmark performance 푅 ★ 푡 = 1 푁 Í 푁 푖=1 푠 푡 푖 by evaluating the target model푡only on a subset of itemsC 푡 ⊆ 퐸, which we refer to as a coreset. We focus on the setting where the full benchmark scores for a set of퐾 푟 reference models푟 1 , . . . , 푟 퐾 푟 are available. Although our framework is applicable to scores푠 푡 푖 ∈ ℝ , in our experiments we consider either binary scores (푠 푡 푖 ∈ 0,1), indicating whether the model’s response matches푦 푖 , or bounded continuous scores, where, e.g., 푠 푡 푖 ∈ [0, 1] represents the probability assigned by the model to the ground-truth response 푦 푖 . Our goal is to address the setting in which the user prioritizes reliable performance estimation over minimizing cost, without a priori knowing the coreset size required to achieve this. Therefore, we focus on methods that not only operate with a user-specified coreset size, but can also automatically determine a coreset size that reflects this priority. In the remainder of the paper, with a slight abuse of notation, we use퐸andC 푡 to indicate the indices of the benchmark items rather than the items themselves. 3. BayesAME BayesAME is a sequential Bayesian method that assigns a latent ability to each group of benchmark items sharing the same reference model performances. It then treats the item scores as noisy versions of these 2 BayesAME: Bayesian Active Model Evaluation abilities and models the full benchmark performance as a random variable defined as the average of the scores, 푅 푡 = 1 푁 Í 푁 푖=1 푠 푡 푖 (where, with a slight abuse of notation, we use푠 푡 푖 to denote both the score random variable and its realization). The abilities are assigned a joint Gaussian prior distribution to encode the belief that, if the reference models achieved similar scores on two groups of items, the target model likely exhibits similar behavior. At each iteration, the posterior distribution of the latent abilities is used to derive performance estimators, quantify performance uncertainty, and select an item to add to the coreset via an information gain criterion. The sequential process terminates when the performance fluctuation and uncertainty fall below their respective user-defined thresholds. 3.1. Single-Target Setting We first consider the setting of a single target model. For the remainder of this section, we omit the target model index. Let(푠 푟 1 푖 , . . . , 푠 푟 퐾 푟 푖 )be the vector of scores achieved by the퐾 푟 reference models on item(푥 푖 , 푦 푖 ). Suppose that the reference models yield퐵unique score vectors across the푁benchmark items, which we denote as the set휙 1 , . . . , 휙 퐵 . This partitions the items into퐵disjoint buckets, such that all items assigned to bucket 푏 share the same reference score vector 휙 푏 . We consider a random vector휃=(휃 1 , . . . , 휃 퐵 ), where휃 푏 represents the target model’s underlying ability on items belonging to bucket 푏, with a Gaussian prior distribution 푝(휃)=N(휇 휃 ,Σ 휃 ), defined as: 휇 휃 푏 = 1 퐾 푟 휙 ⊤ 푏 1 퐾 푟 ,Σ 휃 푏,푏 ′ = 훼exp − 훽 퐾 푟 ∥휙 푏 − 휙 푏 ′ ∥ 2 ,(1) where 1 퐾 푟 denotes a퐾 푟 -dimensional column vector of ones,훼and훽are parameters to be learned, and∥휙 푏 −휙 푏 ′ ∥ 2 is the squared Euclidean distance. This form of the covariance encodes the belief that buckets with similar reference scores yield highly correlated latent abilities for the target model 1 . Applying this latent variable approach at the bucket level, rather than the item level, reduces overreliance on prior information when reference models behave similarly and thus a large number of items map to the same휙 푏 , which most commonly manifests when 퐾 푟 is small or when the scores are binary. For any item(푥 푖 , 푦 푖 )belonging to bucket푏, we model the target model’s score푠 푖 as a random variable with conditional distribution푝(푠 푖 | 휃 푏 )=N(휃 푏 , 푁 푏 휎 2 ), where푁 푏 is the number of items in bucket푏. Let퐻denote an푁 × 퐵binary indicator matrix, where퐻 푖,푏 =1 if the푖-th item belongs to bucket푏, and 0 otherwise. The target model’s score vector푆=(푠 1 , . . . , 푠 푁 )is then modeled as a random variable with conditional distribution 푝(푆| 휃)=N(퐻휃, 퐷), where퐷is an푁× 푁diagonal noise matrix such that퐷 푖,푖 = 휎 2 Í 퐵 푏=1 퐻 푖,푏 푁 푏 . Note that, when all items map to unique reference vectors (퐵= 푁),퐷 푖,푖 = 휎 2 . This bucket-size-dependent noise accounts for the inherent heterogeneity within larger buckets. Even though items in a large bucket share identical reference scores, the target model’s performance on each individual item might vary. By scaling the individual item variance by푁 푏 , we ensure that the variance of the average bucket score remains constant at휎 2 , regardless of how many items the bucket contains:Var 1 푁 푏 Í 푖∈bucket 푏 푠 푖 = 1 푁 2 푏 Í 푖∈bucket 푏 Var(푠 푖 )= 1 푁 2 푏 (푁 푏 · 푁 푏 휎 2 )= 휎 2 . While a more principled approach to modeling binary and bounded continuous scores would explicitly model their support, exploring this direction revealed that the approximations required not only introduced substantial computational overhead but also resulted in inferior performance (see Appendix B.3). In contrast, the Gaussian model admits closed-form posterior distributions, delivering superior scalability and performance. We therefore adopt a Gaussian model as the default choice. Let푠 C denote the subvector of푠corresponding to the items inC, and let휇 휃 |C andΣ 휃 |C be the posterior mean and covariance of휃conditioned on푠 C , such that푝(휃| 푠 C )=N(휇 휃 |C ,Σ 휃 |C ) . The posterior mean and variance of the full benchmark performance 푅= 1 푁 Í 푁 푖=1 푠 푖 are given by: 피[푅| 푠 C ]= 1 푁 © « ∑︁ 푖∈C 푠 푖 + ∑︁ 푗∈퐸 피[푠 푗 | 푠 C ] ª ® ¬ = 1 푁 ∑︁ 푖∈C 푠 푖 + 퐵 ∑︁ 푏=1 (푁 푏 − 푛 푏 )휇 휃 푏|C ! , 1 When given metadata for the reference and target models, we could employ a weighted mean and covariance, where the weights capture the similarity between the target and reference models, see Appendix B.2. 3 BayesAME: Bayesian Active Model Evaluation Var(푅| 푠 C )= 1 푁 2 ∑︁ 푗∈퐸 ∑︁ 푗 ′ ∈퐸 Cov(푠 푗 , 푠 푗 ′ | 푠 C )= 1 푁 2 퐵 ∑︁ 푏=1 (푁 푏 − 푛 푏 ) 휎 2 푁 푏 + 퐵 ∑︁ 푏 ′ =1 (푁 푏 ′ − 푛 푏 ′ )Σ 휃 푏,푏 ′ |C ! ,(2) where푛 푏 is the number of evaluated items in bucket푏. As more items are evaluated, the posterior mean increasingly relies on the actual scores, converging to the true performance푅 ★ when the entire benchmark has been evaluated (푛 푏 = 푁 푏 for all 푏), at which point the posterior variance vanishes. Performance Estimator We construct an estimator ˆ 푅of푅 ★ by adapting our approach to the number of unique reference score vectors in the benchmark, introducing a small margin푁 0 >0 to allow for slight deviations from uniqueness. In the nearly-unique reference regime퐵≥ 푁− 푁 0 , the natural and optimal performance estimator is the Bayesian posterior expectation: ˆ 푅 Bayes = 피[푅| 푠 C ]= 1 푁 ∑︁ 푖∈C 푠 푖 + 퐵 ∑︁ 푏=1 (푁 푏 − 푛 푏 )휇 휃 푏|C ! .(3) However, using Eq. (3) in the non-unique reference regime퐵 < 푁− 푁 0 presents a potential issue: if a heavily populated bucket푏has a large number of unevaluated items (푛 푏 ≪ 푁 푏 ), its posterior mean휇 휃 푏|C exerts a high weight on the overall estimate, amplifying any estimation errors. In this regime, we instead use the following estimator: ˆ 푅 CV = 1 푛 ∑︁ 푖∈C 푠 푖 + 퐵 ∑︁ 푏=1 푁 푏 푁 − 푛 푏 푛 피[휃 푏 | 푠 C ]= 1 푛 ∑︁ 푖∈C 푠 푖 + 퐵 ∑︁ 푏=1 푁 푏 푁 − 푛 푏 푛 휇 휃 푏|C ,(4) where푛indicates the size of the coresetC. This estimator serves as an in-sample proxy for the unbiased control variate estimator given in Appendix A.2. We can unify both regimes into a single estimator: ˆ 푅= 푤 ∑︁ 푖∈C 푠 푖 + 퐵 ∑︁ 푏=1 푁 푏 푁 − 푛 푏 푤 휇 휃 푏|C ,(5) where 푤= 1 푛 if 퐵 < 푁− 푁 0 and 푤= 1 푁 if 퐵≥ 푁− 푁 0 . Selection Strategy At each iteration of the sequential process, we use an information-gain criterion to select the next item on which to evaluate the target model. Specifically, we choose an item that maximally reduces the expected uncertainty of the model’s performance푅, measured via differential entropy:훼 IG (푖)= H(푅| 푠 C )− 피 푠 푖 [H(푅| 푠 C , 푠 푖 )] . Assuming candidate item(푥 푖 , 푦 푖 )belongs to bucket푏, due to the Gaussian nature of the posterior distribution, this expected information gain simplifies to: 훼 IG (푖)= 1 2 log Var(푅| 푠 C ) Var(푅| 푠 C , 푠 푖 ) =− 1 2 log © « 1− Í 퐵 푏 ′ =1 (푁 푏 ′ − 푛 푏 ′ )Σ 휃 푏,푏 ′ |C + 휎 2 푁 푏 2 푁 2 Var(푅| 푠 C )(Σ 휃 푏,푏|C + 휎 2 푁 푏 ) ª ® ¬ .(6) Note that all items in the same bucket have the same expected information-gain value. In the non-unique reference regime with imbalanced bucket sizes, purely active selection can degrade performance. This occurs because the target model may exhibit fine-grained variations in behavior that are not captured by the coarse grouping of the reference models. Consequently, when퐵 < 푁 − 푁 0 , we default to random selection, which ensures that the expected number of sampled items from each bucket remains proportional to its size. The margin푁 0 >0 allows the method to use active sampling whenever the reference score vectors are nearly unique. Furthermore, when dealing with continuous scores, unique reference score vectors may become non-unique when rounded to a specified tolerance. Under such conditions, we similarly default to random selection. We also explored alternative selection strategies that operate on batches of items rather than individual items, including non-myopic approaches. However, these provided similar performance (see Appendix B.1). 4 BayesAME: Bayesian Active Model Evaluation Stopping Criterion One of our key objectives is to determine the coreset size, rather than requiring it as input from the user. Intuitively, we would like to stop evaluating additional items once they are unlikely to meaningfully change our estimate of the target model’s performance. To formalize this, we monitor the uncertainty of푅as items are evaluated. We quantify this uncertainty using the width of the 95% credible interval of푅:푞 97.5 (푅| 푠 C )−푞 2.5 (푅| 푠 C )=2푧 0.975 √︁ Var(푅| 푠 C ), where푧 0.975 denotes the standard normal푧-score andVar(푅| 푠 C )is given in Eq. (2). We dynamically augment the coreset until a two-condition stopping criterion is satisfied:(푖)the performance estimate ˆ 푅stabilizes, defined as its range over a window of푊iterations falling below a threshold휖 1 ; and(푖)2푧 0.975 √︁ Var(푅| 푠 C ) ≤ 휖 2 . The threshold 휖 2 represents the maximum acceptable width of this uncertainty interval, translating directly into a user’s tolerated margin of error. By treating휖 1 and휖 2 as user-defined parameters, BayesAME enables the user to decide the precision required. For additional discussion, see Appendix A.3. Posterior Distribution To optimize the cost required to compute휇 휃 |C andΣ 휃 |C , throughout the sequential process we adapt our formulation as detailed below. Let퐻 C and퐷 C denote the submatrices of퐻and퐷formed by the rows corresponding toC. LetC 푏 denote the subset of푛 푏 items in the coresetCthat are in bucket푏. Finally, let ̄푠 C 1:퐵 =( ̄푠 C 1 , . . . , ̄푠 C 퐵 ), where ̄푠 C 푏 = 1 푛 푏 Í 푖∈C 푏 푠 푖 (for notational simplicity we assume푛 푏 >0 for all푏). As detailed in Appendix A.1, the posterior distribution of휃satisfies푝(휃| 푠 C )= 푝(휃| ̄푠 C 1:퐵 ). Although mathematically equivalent, computing푝(휃| 푠 C )requires inverting an푛× 푛matrix, whereas computing푝(휃| ̄푠 C 1:퐵 )requires inverting at most a 퐵× 퐵 matrix. Indeed, 푝(휃| 푠 C ) is Gaussian with mean 휇 휃 |C and covarianceΣ 휃 |C given by: 휇 휃 |C = 휇 휃 +Σ 휃 퐻 ⊤ C (퐻 C Σ 휃 퐻 ⊤ C + 퐷 C ) −1 (푠 C − 퐻 C 휇 휃 ), Σ 휃 |C =Σ 휃 −Σ 휃 퐻 ⊤ C (퐻 C Σ 휃 퐻 ⊤ C + 퐷 C ) −1 퐻 C Σ 휃 ,(7) while, starting from 푝(휃| ̄푠 C 1:퐵 ), the mean 휇 휃 |C and covarianceΣ 휃 |C can be expressed as: 휇 휃 |C = 휇 휃 +Σ 휃 (Σ 휃 +Δ) −1 ( ̄푠 C 1:퐵 − 휇 휃 ), Σ 휃 |C =Σ 휃 −Σ 휃 (Σ 휃 +Δ) −1 Σ 휃 ,(8) whereΔis a diagonal noise matrix with entriesΔ 푏,푏 = 휎 2 푁 푏 /푛 푏 . To achieve the best computational cost, we use Eq. (7) when푛 < 퐵and Eq. (8) when푛≥ 퐵. We employ Cholesky updates across both regimes, leveraging the fact that the prior covarianceΣ 휃 remains fixed between hyperparameter updates, which occur every퐹 iterations. This yields a computational complexity ofO(min(푛, 퐵) 2 ) per iteration. Algorithm We summarize the method in Algorithm 1. Starting with an empty coresetC, BayesAME iteratively selects the benchmark item maximizing the expected information gain훼 IG (푖)when퐵≥ 푁− 푁 0 (or at random otherwise). The target model is evaluated on this item, its index is added toC, and the posterior mean휇 휃 |C and covarianceΣ 휃 |C are updated, along with the performance estimate ˆ 푅. Every퐹iterations, the covariance hyperparameters훼and훽are optimized by minimizing the negative log-marginal likelihoodL(훼, 훽) of the coreset scores (see Appendix A.4). This sequential process terminates once the two-condition stopping criterion is satisfied:(푖)the range of the performance estimate ˆ 푅 over a window of푊iterations falls below threshold 휖 1 ; and (푖) the width of the 95% credible interval falls below threshold 휖 2 . 3.2. Multi-Target Setting In the case of multiple target modelsT= 푡 1 , . . . , 푡 퐾 푡 , a standard approach would be to simply apply the single-target method of Section 3.1 to each model independently. However, jointly modeling target models that exhibit correlated performance behaviors can improve estimation accuracy and reduce the required coreset size. We model the targets jointly using a linear model of coregionalization (Álvarez et al., 2012; Journel and Huijbregts, 1978). 5 BayesAME: Bayesian Active Model Evaluation Algorithm 1 BayesAME Single-target Require:Benchmark퐸; noise variance휎 2 ; margin푁 0 ; stopping criterion thresholds휖 1 , 휖 2 and window-length 푊; Optimizer, hyperparameter initial values 훼 0 , 훽 0 and update frequency 퐹 Initialize coresetC ←∅ Initialize hyperparameters 훼← 훼 0 , 훽 ← 훽 0 for 푗∈ 1, . . . , 푁 do if 퐵≥ 푁− 푁 0 then Select item via expected information gain: 푖 ∗ = argmax 푖∈퐸 훼 IG (푖)⊲ Eq. (6) else Select item via random sampling: 푖 ∗ ∼ Uniform(퐸 ) end if Obtain score 푠 푖 ∗ and add 푖 ∗ to the coreset: C ←C∪푖 ∗ Compute 휇 휃 |C andΣ 휃 |C ⊲ Eq. (7) or Eq. (8) Compute performance estimate ˆ 푅⊲ Eq. (5) Set ˆ 푅 푗 ← ˆ 푅 if 푗%퐹= 0 then Update hyperparameters: 훼, 훽 ← Optimizer(L(훼, 훽), 훼, 훽)⊲ Eq. (11) end if if 푗≥ 푊 and max 푙∈[ 푗−푊+1,푗] ( ˆ 푅 푙 )− min 푙∈[ 푗−푊+1,푗] ( ˆ 푅 푙 ) ≤ 휖 1 and 2푧 0.975 √︁ Var(푅| 푠 C ) ≤ 휖 2 then break end if end for Return ˆ 푅 We capture cross-model correlations by assuming that their abilities are driven by퐿shared, independent latent random vectors푢 푙 ∈ ℝ 퐵 . Specifically, let휃 푡 푏 represent the latent ability of target model푡on bucket푏, and let 휃 푡 =(휃 푡 1 , . . . , 휃 푡 퐵 ). We model 휃 푡 as: 휃 푡 = 휇 휃 푡 + 퐿 ∑︁ 푙=1 푤 푡 푙 푢 푙 , where휇 휃 푡 is defined as in Eq. (1),푤 푡 1 , . . . , 푤 푡 퐿 are model-specific weights to be learned, and푢 푙 ∼ N(0,Σ 휃,푙 ), whereΣ 휃,푙 is a covariance matrix specific to latent vector푙. This induces a Gaussian prior distribution over 휃=(휃 푡 1 , . . . , 휃 푡 퐾 푡 ) given by: 푝(휃)=N 휇 휃 , 퐿 ∑︁ 푙=1 푤 푙 푤 ⊤ 푙 ⊗Σ 휃,푙 ! , where⊗denotes the Kronecker product,푤 푙 = (푤 푡 1 푙 , . . . , 푤 푡 퐾 푡 푙 ) ⊤ and휇 휃 = (휇 휃 푡 1 , . . . , 휇 휃 푡 퐾 푡 ). This formula- tion provides a flexible yet computationally efficient way to capture correlation across both buckets and models by combining bucket-specific (Σ 휃,푙 푏,푏 ′ ) and model-specific (푤 푡 푙 ) terms. To provide greater modeling flexibility, we use two covariance families with complementary inductive biases: the squared exponential covariance defined as in Eq. (1), which imposes strict smoothness, and a Matérn-3/2 kernel of the form Σ 휃,푙 푏,푏 ′ = 훼 푙 1+ √ 3훽 푙 ∥휙 푏 − 휙 푏 ′ ∥ exp − √ 3훽 푙 ∥휙 푏 − 휙 푏 ′ ∥ with훼 푙 , 훽 푙 >0, which imposes weaker smoothness. Be- cause the degree of correlation between the target models is typically unknown a priori, we set퐿= 퐾 푡 . This enables the framework to dynamically adapt: it captures strong correlations by sharing latent components across models, yet it retains the ability to recover the fully independent case. The scores are then modeled similarly to the single-target setting described in Section 3.1. Selection Strategy In contrast to the single-target setting, the multi-target selection step requires simultane- ously selecting an item and a target model to evaluate on that item, substantially expanding the search space. Furthermore, relying on a purely greedy information-gain criterion across this joint item-model space can bias performance estimates, particularly during early stages when correlations across target models are still being learned. To mitigate this bias and establish a robust initial estimate of the correlation structure, we depart 6 BayesAME: Bayesian Active Model Evaluation from the empty coreset initialization used in the single-target setting. Instead, we randomly select 10% of the benchmark items and evaluate all target models on them. After this initialization step, we deploy the following selection strategy, which maintains some degree of exploration. We iterate deterministically through the target models. For a given model푡, we select the item that maximizes the expected information gain:훼 푡 IG (푖)= H(푅 푡 | 푠 C ) − 피 푠 푡 푖 [H(푅 푡 | 푠 C , 푠 푡 푖 )], where푠 C = 푠 푡 1 C 푡 1 , . . . , 푠 푡 퐾 푡 C 푡 퐾 푡 . This approach substantially reduces the search space, maintaining a selection complexity identical to that of the single-target setting. We iteratively augment the coresetsC 푡 1 , . . . ,C 푡 퐾 푡 until all target models have met the two-condition stopping criterion detailed above. We also investigated an alternative average marginalized information-gain strategy. At each iteration, we first select the item that maximizes the expected information gain averaged across all퐾 푡 target models: 훼 IG (푖)= 1 퐾 푡 Í 퐾 푡 푡=1 H(푅 푡 | 푠 C )− 피 푠 푡 푖 [H(푅 푡 | 푠 C , 푠 푡 푖 )] . After selecting an item, we choose the target model uniformly at random. We present these results in Figure 15 of Appendix C.4. 4. Related work The issue of computational inefficiency in model evaluation is primarily tackled from two, sometimes intersecting, directions: constructing smaller, static versions of existing benchmarks (or developing methods for reducing benchmarks) (Bean et al., 2025; Kipnis et al., 2025; Zhao et al., 2025), and estimating performance without fully evaluating the target model (Berrada et al., 2025; Bowyer et al., 2026; Fisch et al., 2026; Hsu and Shekhar, 2026; Huang et al., 2026; Kipnis et al., 2025; Kossen et al., 2021; Liao et al., 2025; Liu et al., 2026; Polo et al., 2024; Saranathan et al., 2025; Vivek et al., 2024; Wang et al., 2025; Zhang et al., 2025). Automatic Determination of Coreset Size With the exception of Cer-Eval (Wang et al., 2025), all methods in the second category assume that the evaluation budget is specified by the user. Cer-Eval differs substantially from our method as it does not utilize reference models, relying instead exclusively on the responses from the target model. It repeats the following steps until a strict confidence condition is met:(푖)partitioning the benchmark based on previously evaluated items to isolate regions of low variance in the target model’s scores, (푖)computing summary statistics for each partition,(푖)identifying the partition that yields the greatest expected reduction in evaluation uncertainty; and (푖푣) sampling an item from that optimal partition. Extending methods designed for user-specified coreset sizes to automatically determine the coreset size poses fundamental challenges. This task requires striving for accurate estimation across all coreset sizes, which, as we show in our experiments, is not always achievable. Furthermore, establishing a reliable stopping criterion requires confidence intervals that accurately reflect the true remaining uncertainty, a property that existing formulations often lack. Performance Modeling and Estimation From a modeling perspective, many works that leverage reference model information rely on Item Response Theory (IRT) to infer the target model’s profile (Kipnis et al., 2025; Liao et al., 2025; Polo et al., 2024). However, obtaining reliable parameter estimates with IRT requires on the order of hundreds (Jiang et al., 2026) or thousands (Kipnis et al., 2025) of reference models. Another line of work (Bowyer et al., 2026; Zhang et al., 2025) frames the problem as a regression task. ProEval (Huang et al., 2026) introduces a Gaussian process approach that shares similarities with our Bayesian modeling. However, we depart critically from their formulation to address automatic coreset size determination and more robustly handle the lack of prior information about the target model. Multi-Target Setting Finally, to the best of our knowledge, the only other work that leverages performance correlations across multiple target models rather than treating them independently is Fisch et al. (2026). This approach frames model evaluation as a matrix completion problem. While the method could be run 7 BayesAME: Bayesian Active Model Evaluation sequentially by randomly sampling the next item and model to evaluate, its confidence intervals capture only the sampling uncertainty around the mean performance, assuming the benchmark is a finite sample drawn from an infinite population of prompts. Consequently, even when all items in the finite benchmark have been evaluated, the confidence interval does not shrink to zero. In contrast, our Bayesian credible intervals provide a direct probability statement about the true, finite-sample benchmark performance given the observed data, yielding an intuitively appealing and actionable stopping criterion. 5. Experiments We evaluate our approach across seven standard benchmarks collected from three primary sources: the Open LLM Leaderboard (Fourrier et al., 2024), evaluations compiled by Zhang et al. (2025), and the HELM Lite benchmark (Liang et al., 2023). From the Open LLM Leaderboard, we extracted scores for GPQA (Rein et al., 2024), MMLU-Pro (Hendrycks et al., 2021; Wang et al., 2024), BBH (Suzgun et al., 2023), ARC-Challenge (Clark et al., 2018), and MuSR (Sprague et al., 2024). Beyond standard binary scores, we leveraged the leaderboard’s raw log-likelihoods to obtain continuous scores via a softmax transformation. These continuous scores allow us to assess the impact of exploiting richer information and mitigate the issue of identical reference model scores. To ensure a representative distribution of model capabilities, we partitioned the available models into five quantiles based on accuracy. We then sampled evenly by selecting the most popular models from each quantile, resulting in an initial set of 300 models. After extracting the data for these selections, we filtered out any records containing NaN values caused by incomplete leaderboard entries, formatting discrepancies, or evaluation timeouts. This curation process yielded the following final model counts per benchmark: GPQA: 102, MMLU-Pro: 165, BBH: 279, ARC-Challenge: 125, and MuSR: 288. From Zhang et al. (2025), we extracted binary scores for IFEval (Zhou et al., 2023) across 448 models. Finally, from HELM Lite (up to version 1.13.0), we extracted the F1 scores for the Natural QA Openbook long-answer scenario (Kwiatkowski et al., 2019) across 44 models. 5.1. Single-Target Setting Baselines Since all existing approaches that leverage reference models assume a predefined coreset size, we adapt the most effective methods to operate in the same sequential manner as Algorithm 1. Specifically,Cis iteratively augmented untilmax 푙∈[ 푗−푊,푗] ( ˆ 푅 푙 )− min 푙∈[ 푗−푊,푗] ( ˆ 푅 푙 ) ≤ 휖 1 and 2푧 0.975 √ Var 푅 ≤ 휖 2 where ˆ 푅andVar 푅 are defined below. 1. BayesAME-RS: A simpler version of BayesAME that selects items at random. 2. Seq-APW: An extension of the anchor points weighted method from Vivek et al. (2024), in whichCis augmented by choosing items that maximize space covering, as measured by correlations between the reference models’ performances. Benchmark items are partitioned into clusters by assigning each item to its nearest anchor inC. ˆ 푅= Í 푖∈C 푤 푖 푠 푖 , where푤 푖 is a weight proportional to the cluster size, whileVar 푅 is given by the following estimator of Var(푅): Var 푅 = 푠 2 ∑︁ 푖∈C 푤 2 푖 ! 푁− 푛 eff 푁 , 푛 eff = Í 푖∈C 푤 푖 2 Í 푖∈C 푤 2 푖 ,(9) where푠denotes the sample standard deviation. By construction, ˆ 푅recovers the true performance푅 ★ once the entire benchmark has been evaluated. Moreover,Var 푅 captures the sampling variance arising from the randomness in the coreset selection. Due to the finite-population correction factor, this variance decreases as the coreset grows and converges to zero when the full benchmark is evaluated. 3.RS-Mean: The simplest baseline that augmentsCby selecting items at random and estimates the full benchmark performance as the empirical average performance onC.Var 푅 is given by Eq. (9) using 푤 푖 = 1/푛. The same properties regarding ˆ 푅 and Var 푅 discussed for Seq-APW apply. 8 BayesAME: Bayesian Active Model Evaluation 4.Bayes-AIPW: A Bayesian ridge regression extension of AIPW (Zhang et al., 2025), obtained by placing a Gaussian prior distribution on the coefficients훽of the model푠 푖 = 휙 ⊤ 푖 훽+ 휀 푖 , where휙 푖 =(푠 푟 1 푖 , . . . , 푠 푟 퐾 푟 푖 ), and휀 푖 ∼ N(0, 훼 −1 )with훼 ∼ Gamma(훼 1 , 훼 2 ). By conditioning on(휙 푖 , 푠 푖 ) 푖∈C , we obtain a posterior distribution over 훽 with mean 휇 훽 |C and covarianceΣ 훽 |C . As ˆ 푅, we use the AIPW estimator: ˆ 푅= 1 푛 ∑︁ 푖∈C 푠 푖 + 1 1+ 푛 푁−푛 1 푁− 푛 ∑︁ 푖∈퐸 휇 푖|C − 1 푛 ∑︁ 푖∈C 휇 푖|C ! , where휇 푖|C = 휙 ⊤ 푖 휇 훽 |C . Note that ˆ 푅 recovers the true performance푅 ★ when all items have been evaluated. Posterior uncertainty is obtained by propagating the posterior distribution of훽through the AIPW correction, yielding: Var 푅 = ( ̄ 휙 퐸 − ̄ 휙 C ) ⊤ Σ 훽 |C ( ̄ 휙 퐸 − ̄ 휙 C )+ ˆ훼 −1 1 푛 + 1 푁−푛 1+ 푛 푁−푛 2 , where ̄ 휙 C = 1 푛 Í 푖∈C 휙 푖 , ̄ 휙 퐸 = 1 푁−푛 Í 푖∈퐸 휙 푖 , andˆ훼is the posterior mean of훼.Cis augmented by selecting items at random. Figure 6 in Appendix C.3 shows that this Bayesian extension matches or exceeds the performance of the original AIPW method. 5.Bayes-RS-Learn: A Bayesian ridge regression extension of Random-Sampling-Learn (Zhang et al., 2025), obtained by placing a Gaussian prior on the coefficients훽of the model푅= 푠 ⊤ C 훽 + 휀, where 휀∼N(0, 훼 −1 ),훼∼ Gamma(훼 1 , 훼 2 ). We perform inference on훽by conditioning on(푠 푟 푘 C , 푅 ★ 푟 푘 ) 퐾 푟 푘=1 . This yields a posterior distribution over 훽 with mean 휇 훽 |C and covarianceΣ 훽 |C . We then obtain ˆ 푅 and Var 푅 as: ˆ 푅= 푠 ⊤ C 휇 훽 |C , Var 푅 = 푠 ⊤ C Σ 훽 |C 푠 C + ˆ훼 −1 , where ˆ훼 −1 is the posterior mean of 훼 −1 . 6.ProEval: The method in Huang et al. (2026) for the setting with no prior performance data for the target model (called New Model setting with Score Features in the original paper), without abstention rule. The method defines the target model’s performance as푅= 1 푁 Í 푁 푖=1 휃 푖 , where휃= (휃 1 , . . . , 휃 푁 )is a Gaussian random vector whose prior covariance matrix is given by the empirical covariance of the reference model scores. ˆ 푅= 피[푅| 푠 C ]andVar 푅 = Var(푅| 푠 C ). Because푅is defined as the average of the latent abilities, the estimator ˆ 푅can deviate significantly from the true performance푅 ★ even for large coreset sizes, as discussed in Section 4. The confidence interval induced byVar 푅 can be interpreted as a Bayesian credible interval. 7.IRT: Two variants of IRT whereCis selected randomly, depending on the score type: one for binary scores based on the 2-PL model (Polo et al., 2024) (p-IRT) and one for continuous scores based on the Beta distribution, specifically the extension of Noel and Dauvier (2007) provided in Chen et al. (2019). The item parameters are learned using the reference models’ scores, and the scores푠 C are used to fit the target model-specific parameters. As the performance estimator, we deploy Eq. (5) with푤=1/푁. Because we fit the model parameters separately for each benchmark, treating them as scalars rather than vector-valued as in Polo et al. (2024), applying the generalized variant (gp-IRT) does not improve performance. We fit the parameters using MCMC. This provides samples from the posterior distribution of the parameters, allowing us to directly obtain the posterior distribution of푅and its variance, which we use as ˆ 푅andVar 푅 . Due to the computational cost of MCMC, we update the parameters only every 50 evaluations (resulting in piecewise-constant curves in the plots). Metrics We evaluate our framework using four metrics: RMSE Log, RMSE Gain, Evaluation-Weighted Integrated RMSE, and Cost-Accuracy Trade-Off. By computing the Root Mean Square Error (RMSE) over 200 initialization seeds, all metrics intrinsically account for the empirical variance of the performance estimates across different samples, thereby providing a rigorous evaluation of each method’s precision. The first three metrics evaluate the methods when stopping at fixed coreset sizes푛=1, . . . , 푁(thereby not using the 휖 1 , 휖 2 stopping criterion). Evaluating across the entire range of coreset sizes as input stopping points enables us to determine whether any method dominates the others regardless of the stopping point. In contrast, the Cost-Accuracy Trade-Off measures estimation error at the specific coreset sizes reached when the휖 1 , 휖 2 9 BayesAME: Bayesian Active Model Evaluation stopping criterion is met, demonstrating which method achieves the lowest estimation error at the smallest automatically determined coreset size. 1. RMSE Log: The RMSE on a logarithmic scale, multiplied by 100. 2. RMSE Gain: The relative improvement of a given method over the RS-Mean baseline, calculated as: RMSE Gain(Method)= 20 log 10 RMSE(RS-Mean) RMSE(Method) . A positive gain indicates a relative reduction in estimation error compared to the baseline. 3. Evaluation-Weighted Integrated RMSE: The weighted integral 100 ∫ 푁 0 푛 ′ RMSE(푛 ′ ) 푑푛 ′ , which ensures that late-stage errors are penalized more heavily than early-stage ones. 4.Cost-Accuracy Trade-Off: The RMSE (multiplied by 100) computed at the coreset sizes dynamically determined by the휖 1 , 휖 2 stopping criterion, plotted against the average coreset size ̄푛across seeds. We vary 휖 2 ∈ 0.005,0.0075,0.01,0.015,0.02,0.025,0.03, and fix휖 1 =0.005 since empirical results indicate that 휖 2 is the primary factor governing the stopping decision. We also evaluate our method using Spearman’s rank correlation to assess how well the estimated performance preserves the true relative ordering of the models. The results, presented in Appendix C.3, demonstrate that our method works well for tasks such as model comparison and ranking. Interpolation and Extrapolation Reference Setup To investigate the interpolation and extrapolation regimes with granularity, we partition the available models into a pool of potential reference models and a fixed set of 10 target models, following an approach similar to Zhang et al. (2025). In the interpolation regime, the target models are selected uniformly at random, with all remaining models assigned to the reference pool. In the extrapolation regime, we induce a distribution shift by designating the 10 highest-performing models as the target set, discarding the next 20% of models to create a performance gap, and using the remaining lower-performing models as potential reference models. To assess the impact of reference model quantity, we vary the size of the set of reference models by sampling 10%, 50%, or 90% of the reference pool uniformly at random. 5.1.1. Results Figure 1 shows the RMSE Log (top three rows) and RMSE Gain (bottom three rows) for GPQA with푠 푖 ∈ 0,1 (left two columns) and MMLU-Pro with푠 푖 ∈ [0,1](right two columns) across different percentages of reference models (10%, 50%, and 90%) for the interpolation and extrapolation regimes. Overall, when considering the entire range of coreset sizes, BayesAME consistently achieves the lowest RMSE Log, demonstrating superior estimation accuracy and precision. The relative gains of BayesAME and the two other most competitive methods, BayesAME-RS and Bayes-AIPW, over the RS-Mean baseline can be fully appreciated in the RMSE Gain plots (bottom three rows), which exclude the less competitive methods (Seq-APW, Bayes-RS-Learn, ProEval, and IRT) for clarity. The plots also illustrate that the gain scales positively with the number of reference models, and that this trend is more pronounced for BayesAME. This confirms that a richer reference pool establishes a stronger prior, which BayesAME successfully exploits to isolate and select the most critical benchmark items. In the two cases where퐵 < 푁− 푁 0 , which occur for GPQA with 10% of reference models, BayesAME and BayesAME-RS coincide by construction and perform comparably to Bayes-AIPW. Figure 2 displays the Cost-Accuracy Trade-Off for GPQA with푠 푖 ∈ 0,1(left two columns) and MMLU-Pro with푠 푖 ∈ [0,1](right two columns). By providing the most accurate performance estimates for any threshold value at the lowest cost across both the interpolation and extrapolation regimes, BayesAME establishes a strictly superior Pareto frontier. Plots corresponding to Figures 1 and 2 for the remaining benchmarks, which show similar conclusions, are provided in Appendix C.3. We summarize these results for 50% of reference models via the Evaluation- Weighted Integrated RMSE in Table 1. As the table shows, BayesAME outperforms the other methods in 10 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 2 1 0 1 2 3 4 5 6 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 Figure 1|Single-target Setting. GPQA with binary scores (left two columns) and MMLU-Pro with continuous scores (right two columns). RMSE Log (top three rows) and RMSE Gain (bottom three rows) across varying proportions of reference models (10%, 50%, and 90%) for the interpolation and extrapolation regimes. The 푥-axis denotes the coreset size 푛 as a percentage of the benchmark size 푁. nearly all cases. For binary scores, we have퐵 < 푁− 푁 0 for MMLU-Pro, ARC-Challenge, and MuSR in both the interpolation and extrapolation regimes, and for IFEVAL in the extrapolation regime. For continuous scores, we have that퐵 < 푁− 푁 0 up to some tolerance for ARC-Challenge in the extrapolation regime. In these cases, BayesAME-RS coincides with BayesAME by construction. The table clearly quantifies the overall benefit of using richer continuous scores over binary scores. In Appendix C.3, we demonstrate this advantage via the other three metrics. Our results demonstrate that active coreset selection becomes significantly advantageous over random coreset selection whenever reference models provide sufficiently unique item representations (e.g., via larger reference pools or continuous scores), and show that it is possible to strongly outperform RS-Mean. While non-unique representations cause BayesAME to default to random selection (matching BayesAME-RS and Bayes-AIPW), richer item signals enable active selection to effectively target informative items and strongly outperform RS-Mean, addressing the concerns raised by Zhang et al. (2025). 11 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 BayesAME BayesAME-RS BayesAIPW RS-Mean 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 BayesAME BayesAME-RS BayesAIPW RS-Mean 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Figure 2|Single-target Setting. GPQA with binary scores (left two columns) and MMLU-Pro with continuous scores (right two columns). Cost-Accuracy Trade-Off across varying proportions of reference models (10%, 50%, and 90%) for the interpolation and extrapolation regimes. The푥-axis denotes the average coreset size ̄푛 as a percentage of the benchmark size 푁. 5.2. Multi-Target Setting We compare the multi-target version of BayesAME (Section 3.2) against its single-target counterpart (Sec- tion 3.1) applied independently to each target model. To evaluate the multi-target formulation under varying degrees of correlation, we construct sets of 10 target models exhibiting either low or high correlation. Specif- ically, we form the low-correlation set using a random selection of 10 models, and the high-correlation set using the 10 most correlated models overall. We construct the set of reference models by sampling 50% of the reference model pool uniformly at random. Figure 3 shows RMSE Log, RMSE Gain, and Cost-Accuracy Trade-Off for IFEval with푠 푖 ∈ 0,1(left two columns) and GPQA with푠 푖 ∈ [0,1](right two columns). Additional results for GPQA with푠 푖 ∈ 0,1are given in Fig. 14 of Appendix C.4. The primary conclusion drawn from these plots is that jointly evaluating multiple target models provides a significant efficiency advantage when the models exhibit correlated behaviors. Specifically, in the high-correlation regime, BayesAME Multi-target demonstrates a steeper decline in RMSE Log and reach a higher RMSE Gain much earlier compared to BayesAME Single-target. This is further reflected in the Cost-Accuracy Trade-Off plots, where BayesAME Multi-target establishes a strictly superior Pareto frontier, achieving the same level of estimation accuracy at a notably lower coreset size across all threshold values. However, the low-correlation regime plots reveal a practical nuance of the multi-target framework. When target models do not share meaningful behavioral similarities, attempting to jointly model their latent abilities introduces a degree of negative transfer. As observed in the Cost-Accuracy Trade-Off plots, the multi-target approach slightly underperforms BayesAME Single-target, exhibiting higher estimation errors for a given evaluation budget. Crucially, however, this degradation is not catastrophic, the performance of the multi-target method degrades gracefully to match that of the competitive baseline Bayes-AIPW. This suggests that while the active selection process expends some resources attempting to learn nonexistent cross-model correlations, the underlying framework remains highly robust. Nonetheless, to maximize evaluation efficiency, practitioners should apply the multi-target extension primarily when there is a reasonable prior expectation of correlated performance. 12 BayesAME: Bayesian Active Model Evaluation BenchmarksBayesAME BayesAME-RS RS-Mean Seq-APW Bayes-AIPW Bayes-RS-Learn ProEvalIRT Binary GPQAInterpolation61.5466.3876.48866.6167.2199.0696.2989.61 Extrapolation68.9172.6581.73751.0873.41403.29418.1191.40 MMLU-ProInterpolation30.1830.1840.201141.0630.6890.5995.5646.38 Extrapolation37.2037.2044.89821.4037.69670.20701.84 156.33 BBHInterpolation31.7036.3550.42495.0135.5450.6357.8250.95 Extrapolation38.3542.0252.59448.0441.34448.53600.98 158.41 ARC-ChallengeInterpolation36.5336.5356.091391.2236.6870.8474.0644.19 Extrapolation36.5636.5653.21768.8338.03327.45364.2188.58 MuSRInterpolation47.8647.8669.27949.9049.1869.1268.5484.51 Extrapolation54.1954.1971.28719.8656.72289.76353.44 104.41 IFEvalInterpolation49.5356.1472.451464.9156.7176.2278.5379.06 Extrapolation58.0858.0864.54488.5460.52466.451017.33 212.27 Continuous GPQAInterpolation30.5834.4245.2550.2235.2149.0854.0381.44 Extrapolation40.4040.7556.6462.5246.77262.65313.65 142.56 MMLU-ProInterpolation13.1117.1027.8039.9917.1972.6682.6266.44 Extrapolation26.8231.1040.8347.0731.33482.43570.76 311.12 BBHInterpolation13.0417.5435.2149.3818.0126.1340.4561.84 Extrapolation29.8035.5747.1048.5033.64303.09612.49 388.23 ARC-ChallengeInterpolation26.0426.4747.6793.8226.1946.8453.5746.10 Extrapolation33.1633.1649.0743.5233.05292.45354.75 144.55 MuSRInterpolation27.8333.4250.6776.2134.3450.0262.8857.39 Extrapolation36.9642.5660.1970.6443.69246.90372.08 211.37 Natural QA Openbook Interpolation28.2629.6845.911185.8729.12275.90286.37 129.11 Extrapolation20.8328.3641.761044.0227.70226.18262.55 127.84 Table 1|Single-target Setting. Evaluation-Weighted Integrated RMSE for 50% of reference models. 6. Discussion and Conclusions A primary objective of this work was to provide an efficient model evaluation method that is capable of automatically determining a coreset size when reliable performance estimation takes priority over efficiency, while remaining flexible to accept a predefined coreset size. To this end, we introduced a simple, high- performing method, BayesAME. Furthermore, we proposed an extension to the multi-target setting that leverages performance correlations among target models to further reduce the coreset size. Another objective was to address skepticism in recent literature, which suggests that random coreset selection and simply using a random sample mean baseline gives competitive performance. Our study overturns this belief by showing that, when leveraging richer information (such as continuous scores and sufficiently varied sets of reference models), looking across the entire spectrum of coreset sizes, and accounting for the empirical variance of the performance estimates across different samples, non-random coreset selection and more robust performance estimation remain highly advantageous. Interestingly, while our results challenge the literature’s reliance on random selection, they do validate the underlying intuition that simple methods are difficult to outperform. We found that introducing highly sophisticated, principled modeling of the scores did not translate to better performance. For example, when dealing with binary or bounded continuous scores, a more principled model of the latent abilities using logit transformations (as described in Appendix B.3) failed to outperform the coarser joint Gaussian approach. Though relying on an improper prior for bounded data, the Gaussian approach proved far more effective because it allowed for highly efficient, closed-form evaluations of information gain. Similarly, our attempts to find richer, more complex representations of items (for example, including prompt embeddings), rather than simply relying on reference scores, did not yield tangible improvements. There are several limitations of BayesAME that provide promising avenues for future research. Currently, our framework assumes scalar scores. A natural step is to adapt it to handle more complex, multi-dimensional, or unstructured evaluation formats. In addition, future iterations could investigate ways to integrate architecture details, training hyperparameters, or training data distributions to further refine the prior covariance. 13 BayesAME: Bayesian Active Model Evaluation Low correlationHigh correlationLow correlationHigh correlation 20406080100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 20406080100 10 2 10 1 10 0 10 1 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090 2 1 0 1 2 3 4 5 6 102030405060708090 2 1 0 1 2 3 4 5 6 102030405060708090 2 0 2 4 6 8 102030405060708090 2 0 2 4 6 8 707580859095100 0.2 0.4 0.6 0.8 1.0 707580859095100 0.2 0.4 0.6 0.8 1.0 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Figure 3|Multi-target Setting. IFEval with binary scores (left two columns) and GPQA with continuous scores (right two columns). The rows represent RMSE Log, RMSE Gain, and Cost-Accuracy Trade-Off, respectively, for 50% of reference models for the low and high correlation regimes. Acknowledgments The authors would like to thank Michalis Titsias for valuable discussions and providing feedback on the manuscript. References M. A. Álvarez, L. Rosasco, and N. D. Lawrence. Kernels for vector-valued functions: A review. Foundations and Trends® in Machine Learning, 4:195–266, 2012. A. M. Bean, N. Seedat, S. Chen, and J. R. Schwarz. Scales++: Compute efficient evaluation subset selection with cognitive scales embeddings. arXiv preprint arXiv:2510.26384, 2025. G. Berrada, J. Kossen, M. Razzak, F. B. Smith, Y. Gal, and T. Rainforth. Scaling up active testing to large language models. In Advances in Neural Information Processing Systems, 2025. S. Bowyer, A. Locatelli, and K. Cao. Efficient benchmarking is just feature selection and multiple regression. arXiv preprint arXiv:2605.25773, 2026. R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16:1190–1208, 1995. Y. Chen, T. S. Filho, R. B. Prudencio, T. Diethe, and P. Flach.훽 3 -IRT: A new item response model and its applications. In International Conference on Artificial Intelligence and Statistics, pages 1013–1021, 2019. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. A. Fisch, D. Deutsch, J. Maynez, A. Agarwal, J. Berant, W. Cohen, A. Globerson, and J. Eisenstein. CollabEval: Statistically efficient collaborative model evaluation via matrix completion. arXiv preprint arXiv:2607.05046, 2026. C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, and T. Wolf. Open LLM leaderboard v2, 2024. 14 BayesAME: Bayesian Active Model Evaluation D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. C.-Y. Hsu and S. Shekhar. Efficient sequential evaluation of large language models. arXiv preprint arXiv:2607.17409, 2026. Y. Huang, W. Zeng, A. Kumaresan, and Z. Wang. ProEval: Proactive failure discovery and efficient performance estimation for generative AI evaluation. In International Conference on Machine Learning, 2026. H. Jiang, S. Kwon, J. Luo, Z. Xiao, and S. Zhang. Can we trust item response theory for AI evaluation? arXiv preprint arXiv:2607.15190, 2026. A. G. Journel and C. J. Huijbregts. Mining Geostatistics. Academic Press, 1978. A. Kipnis, K. Voudouris, L. M. Schulze Buschoff, and E. Schulz. metabench - A sparse benchmark of reasoning and knowledge in large language models. In International Conference on Learning Representations, 2025. J. Kossen, S. Farquhar, Y. Gal, and T. Rainforth. Active testing: Sample-efficient model evaluation. In International Conference on Machine Learning, 2021. A. Kulesza and B. Taskar. k-DPPs: Fixed-size determinantal point processes. In International Conference on Machine Learning, pages 1193–1200, 2011. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. A. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. L. Liao, Q. Zhang, R. Wu, and G. Fang. Toward a unified framework for data-efficient evaluation of large language models. arXiv preprint arXiv:2510.04051, 2025. Z. Liu, J. Zhang, C. Liu, and Y. Zhu. Active testing of large language models via approximate Neyman allocation. arXiv preprint arXiv:2605.10075, 2026. I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. D. J. C. MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5):720–736, 1992. Y. Noel and B. Dauvier. A Beta item response model for continuous bounded responses. Applied Psychological Measurement, 31(1):47–73, 2007. M. Osborne, R. Garnett, S. Roberts, C. Hart, S. Aigrain, and N. Gibson. Bayesian quadrature for ratios. In International Conference on Artificial Intelligence and Statistics, pages 832–840, 2012. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830, 2011. Y. Perlitz, E. Bandel, A. Gera, O. Arviv, L. Ein-Dor, E. Shnarch, N. Slonim, M. Shmueli-Scheuer, and L. Choshen. Efficient benchmarking (of language models). In Conference of the North American Chapter of the Association for Computational Linguistics, pages 2519–2536, 2024. 15 BayesAME: Bayesian Active Model Evaluation F. M. Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In International Conference on Machine Learning, 2024. C. E. Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. The MIT Press, 2006. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. In First Conference on Language Modeling, 2024. G. Saranathan, C. Xu, M. P. Alam, T. Kumar, M. Foltin, S. Y. Wong, and S. Bhattacharya. SubLIME: Subset selection via rank correlation prediction for data-efficient LLM evaluation. In Annual Meeting of the Association for Computational Linguistics, pages 30572–30593, 2025. Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett. MuSR: Testing the limits of chain-of-thought with multistep soft reasoning. In International Conference on Learning Representations, 2024. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. Chung, A. Chowdhery, Q. Le, E. Chi, D. Zhou, and J. Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics, pages 13003–13051, 2023. R. Vivek, K. Ethayarajh, D. Yang, and D. Kiela. Anchor points: Benchmarking models with much fewer examples. In Conference of the European Chapter of the Association for Computational Linguistics, pages 1576–1601, 2024. G. Wang, Z. Chen, B. Li, and H. Xu. Cer-Eval: Certifiable and cost-efficient evaluation framework for LLMs. arXiv preprint arXiv:2505.03814, 2025. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems (Track on Datasets and Benchmarks), 2024. G. Zhang, F. E. Dorner, and M. Hardt. How benchmark prediction from fewer data misses the mark. In Advances in Neural Information Processing Systems, 2025. H. Zhao, M. Li, L. Sun, and T. Zhou. BenTo: Benchmark reduction with in-context transferability. In International Conference on Learning Representations, 2025. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. 16 BayesAME: Bayesian Active Model Evaluation A. BayesAME Details A.1. Posterior Distribution Computation In this section, we show the equivalence between푝(휃| 푠 C )and푝(휃| ̄푠 C 1:퐵 )stated in Section 3.1. Based on this, we derive the two equivalent expressions for 휇 휃 |C andΣ 휃 |C given in Eq. (7) and Eq. (8). Recall that ̄푠 C 1:퐵 =( ̄푠 C 1 , . . . , ̄푠 C 퐵 ), where ̄푠 C 푏 = 1 푛 푏 Í 푖∈C 푏 푠 푖 andC 푏 denotes the subset of푛 푏 items inCthat are in bucket 푏. As 푝( ̄푠 C 푏 | 휃 푏 )=N 휃 푏 , 휎 2 푁 푏 푛 푏 , we have: 푝(푠 C 푏 | 휃 푏 ) ∝ exp − 1 2휎 2 푁 푏 ∑︁ 푖∈C 푏 (푠 푖 − 휃 푏 ) 2 ! = exp − 1 2휎 2 푁 푏 ∑︁ 푖∈C 푏 ((푠 푖 − ̄푠 C 푏 )+( ̄푠 C 푏 − 휃 푏 )) 2 ! = exp − 1 2휎 2 푁 푏 ∑︁ 푖∈C 푏 (푠 푖 − ̄푠 C 푏 ) 2 + 푛 푏 ( ̄푠 C 푏 − 휃 푏 ) 2 !! = exp − 1 2휎 2 푁 푏 ∑︁ 푖∈C 푏 (푠 푖 − ̄푠 C 푏 ) 2 ! exp − 푛 푏 2휎 2 푁 푏 ( ̄푠 C 푏 − 휃 푏 ) 2 ∝ exp − 푛 푏 2휎 2 푁 푏 ( ̄푠 C 푏 − 휃 푏 ) 2 ∝ 푝( ̄푠 C 푏 | 휃 푏 ), where the proportionality is with respect to 휃 푏 . This results into: 푝(휃| 푠 C ) ∝ 푝(휃)푝(푠 C | 휃)= 푝(휃) 퐵 Ö 푏=1; 푛 푏 >0 푝(푠 C 푏 | 휃 푏 ) ∝ 푝(휃) 퐵 Ö 푏=1; 푛 푏 >0 푝( ̄푠 C 푏 | 휃 푏 )= 푝(휃)푝( ̄푠 C 1:퐵 | 휃) ∝ 푝(휃| ̄푠 C 1:퐵 ), which establishes that 푝(휃| 푠 C )= 푝(휃| ̄푠 C 1:퐵 ). Having shown this equivalence, we now derive the explicit form for푝(휃| 푠 C )= 푝(휃| ̄푠 C 1:퐵 )through the Gaussian conditioning rule, which states that, for a joint distribution푝(푥 1 , 푥 2 ), the conditional distribution푝(푥 1 | 푥 2 )has mean 휇 1 +Σ 12 Σ −1 22 (푥 2 − 휇 2 ) and covarianceΣ 11 −Σ 12 Σ −1 22 Σ 21 . Recall that퐻 C and퐷 C denote the submatrices of퐻and퐷formed by the rows corresponding toC. The joint distribution of (휃, 푠 C ) can be written as: 휃 푠 C ∼N 휇 휃 퐻 C 휇 휃 , Σ 휃 Σ 휃 퐻 ⊤ C 퐻 C Σ 휃 퐻 C Σ 휃 퐻 ⊤ C + 퐷 C . Applying the Gaussian conditioning rule, we obtain: 휇 휃 |C = 휇 휃 +Σ 휃 퐻 ⊤ C (퐻 C Σ 휃 퐻 ⊤ C + 퐷 C ) −1 (푠 C − 퐻 C 휇 휃 ), Σ 휃 |C =Σ 휃 −Σ 휃 퐻 ⊤ C (퐻 C Σ 휃 퐻 ⊤ C + 퐷 C ) −1 퐻 C Σ 휃 , which corresponds to Eq. (7). LetΔbe a diagonal noise matrix with entriesΔ 푏,푏 = 휎 2 푁 푏 /푛 푏 . The joint distribution of(휃, ̄푠 C 1:퐵 )can be written as: 휃 ̄푠 C 1:퐵 ∼N 휇 휃 휇 휃 , Σ 휃 Σ 휃 Σ 휃 Σ 휃 +Δ . 17 BayesAME: Bayesian Active Model Evaluation Applying the Gaussian conditioning rule, we obtain: 휇 휃 |C = 휇 휃 +Σ 휃 (Σ 휃 +Δ) −1 ( ̄푠 C 1:퐵 − 휇 휃 ), Σ 휃 |C =Σ 휃 −Σ 휃 (Σ 휃 +Δ) −1 Σ 휃 , which corresponds to Eq. (8). A.2. Performance Estimator In this section, we discuss the estimator ˆ 푅 퐶푉 (Eq. (4)) that we use in the non-unique reference regime퐵 < 푁−푁 0 . To simplify the exposition, we adopt an item-level, rather than bucket-level, description. Specifically, for an item(푥 푖 , 푦 푖 )belonging to bucket푏, we use휇 푖|C to denote피[휃 푏 | 푠 C ]. We can rewrite Eq. (4) at the item-level as: ˆ 푅 CV = 1 푛 ∑︁ 푖∈C 푠 푖 − 휇 푖|C + 1 푁 푁 ∑︁ 푗=1 휇 푗|C . This estimator can be seen as an in-sample proxy of the following control variate estimator that uses out-of- sample posteriors: ˆ 푅 CVO = 1 푛 ∑︁ 푖∈C © « 푠 푖 − 휇 푖|C\푖 + 1 푁 푁 ∑︁ 푗=1 휇 푗|C\푖 ª ® ¬ . Note that the in-sample proxy estimator avoids the computational bottleneck of recomputing the posterior for every held-out item. ˆ 푅 CVO is unbiased with respect to the random selection of the coreset as shown below. LetC=(푖 1 , . . . , 푖 푛 )be constructed by sampling items uniformly at random with replacement, such that푖 푘 ∼ 푈(1, . . . , 푁). ˆ 푅 CVO can be expressed as: ˆ 푅 CVO = 1 푛 푛 ∑︁ 푘=1 © « 푠 푖 푘 − 휇 푖 푘 |C\푖 푘 + 1 푁 푁 ∑︁ 푗=1 휇 푗|C\푖 푘 ª ® ¬ . Taking the unconditional expectation with respect to the random selection of the coresetC yields: 피[ ˆ 푅 CVO ]= 1 푛 푛 ∑︁ 푘=1 피[푠 푖 푘 ]− 1 푛 푛 ∑︁ 푘=1 © « 피 휇 푖 푘 |C\푖 푘 − 1 푁 푁 ∑︁ 푗=1 피 휇 푗|C\푖 푘 ª ® ¬ .(10) We first focus on the term 피 휇 푖 푘 |C\푖 푘 . Using the Law of Total Expectation, we have: 피 휇 푖 푘 |C\푖 푘 = 피 C\푖 푘 피 휇 푖 푘 |C\푖 푘 |C\푖 푘 . Because푖 푘 is drawn independently from the remaining coreset itemsC\푖 푘 , the inner expectation over the uniform draw of 푖 푘 is exactly the population mean: 피 휇 푖 푘 |C\푖 푘 = 피 C\푖 푘       1 푁 푁 ∑︁ 푗=1 휇 푗|C\푖 푘       = 1 푁 푁 ∑︁ 푗=1 피 휇 푗|C\푖 푘 . Plugging this into Eq. (10), the second term cancels out, resulting in: 피[ ˆ 푅 CVO ]= 1 푛 푛 ∑︁ 푘=1 피[푠 푖 푘 ]= 1 푛 푛 ∑︁ 푘=1 피 C\푖 푘 [피[푠 푖 푘 |C\푖 푘 ]]= 1 푛 푛 ∑︁ 푘=1 피 C\푖 푘       1 푁 푁 ∑︁ 푗=1 푠 푗       = 1 푁 푁 ∑︁ 푗=1 푠 푗 = 푅. Remark: In practice, the coreset is sampled without replacement. This breaks strict independence, introducing a weak dependence between푖 푘 andC\푖 푘 and yielding a residual bias. However, this value is negligible when 푁 ≫ 1. 18 BayesAME: Bayesian Active Model Evaluation A.3. Stopping Criterion Recall that, to monitor posterior uncertainty about the random variable푅, we consider the width of the 95% credible interval for푅, which is governed byVar(푅| 푠 C ). Crucially, this quantity is not tied to a particular choice of point estimator. This distinction is important in the non-unique reference regime,퐵 < 푁− 푁 0 , where we use the control-variate estimator ˆ 푅 CV in Eq. (4). Our stopping criterion deliberately separates these two roles: the stability condition(푖)is applied to the reported estimator ˆ 푅 CV , whereas the credible-interval condition(푖)is applied to the target quantity 푅. A natural alternative would be to define a random variable푅 CV = 1 푛 Í 푖∈C 푠 푖 + Í 퐵 푏=1 푁 푏 푁 − 푛 푏 푛 휃 푏 and use Var(푅 CV | 푠 C )in the stopping criterion. However, this variance is not reliable as a measure of uncertainty about 푅since it vanishes whenever푁 푏 /푁= 푛 푏 /푛, which occurs, e.g., when there is only one bucket. Another possible estimator-specific diagnostic is the posterior MSE,피[(푅− ˆ 푅 CV ) 2 | 푠 퐶 ]= Var(푅| 푠 C )+( ˆ 푅 Bayes − ˆ 푅 CV ) 2 . This is a valid posterior risk criterion under squared-error loss: the first term captures residual uncertainty about푅, while the second captures the conditional bias of푅 CV relative to the posterior mean. We useVar(푅| 푠 C )as the default uncertainty measure because it preserves a direct credible-interval interpretation for the benchmark performance and avoids conflating posterior uncertainty with the deliberate correction introduced by the control-variate estimator. The estimator-specific discrepancy is instead monitored through the stability condition (푖). A.4. Hyperparameter Optimization Single-target Setting We tune the hyperparameters훼and훽, which govern the prior covarianceΣ 휃 , by minimizing the negative log-marginal likelihoodL(훼, 훽)of the coreset scores. As established in Appendix A.1, the conditional likelihoods푝(푠 C | 휃)and푝( ̄푠 C 1:퐵 | 휃)are proportional up to a constant independent of휃. As a result, their negative log-marginal likelihoods differ only by a constant independent of훼and훽. Therefore, to evaluate the objective efficiently, we defineL(훼, 훽) as: L(훼, 훽)= ( 1 2 (푠 C − 퐻 C 휇 휃 ) ⊤ (퐻 C Σ 휃 퐻 ⊤ C + 퐷 C ) −1 (푠 C − 퐻 C 휇 휃 )+ 1 2 log|퐻 C Σ 휃 퐻 ⊤ C + 퐷 C | if 푛 < 퐵 1 2 ( ̄푠 C 1:퐵 − 휇 휃 ) ⊤ (Σ 휃 +Δ) −1 ( ̄푠 C 1:퐵 − 휇 휃 )+ 1 2 log|Σ 휃 +Δ|if 푛≥ 퐵 (11) For additional efficiency and to maintain numerical stability, we employ a Cholesky decomposition of the respective covariance matrix. We perform this minimization via an L-BFGS-B optimizer (Byrd et al., 1995), constraining the parameters with a small lower bound of 10 −5 to ensure the covariance matrix remains positive definite. Multi-target Setting We tune the hyperparameters훼 푙 and훽 푙 which govern the prior covarianceΣ 휃,푙 , as well as the model-specific weights푤 푡 푙 퐾 푡 푡=1 for each푙=1, . . . , 퐿. Additionally, we learn the noise variance휎 2 instead of keeping it fixed as for the single-target setting. This turns minimizing the negative log-marginal likelihood into a highly non-convex, joint optimization problem. Consequently, instead of L-BFGS-B, we employ AdamW (Loshchilov and Hutter, 2019). AdamW leverages momentum to successfully navigate complex loss surfaces and provides adaptive, per-parameter learning rates, which are essential for handling parameters that operate on fundamentally different scales. Because the hyperparameters naturally stabilize as the coreset grows, we deploy an adaptive computational schedule that executes 500 gradient steps during the initial active learning iterations and progressively decreases this count in later iterations to significantly reduce computational overhead. To ensure numerical stability throughout the training process, we clip the values of훼 푙 ,훽 푙 , and휎 2 after each update, enforcing a lower bound of 10 −4 . 19 BayesAME: Bayesian Active Model Evaluation B. BayesAME Single-target Variants B.1. Batch Selection Strategies We extend the approach from the main text by introducing selection strategies that identify an optimal batch of items퐴, rather than a single item. We focus on the nearly unique reference regime퐵≥ 푁− 푁 0 . To simplify the presentation, we denoteΣ |C = 퐻Σ 휃 |C 퐻 ⊤ + 퐷, which in the case 퐵= 푁 simplifies toΣ |C =Σ 휃 |C + 휎 2 퐼 푁 . The extension of the expected information gain Eq. (6) from one item to a batch of items 퐴 is given by: 훼 IG (퐴)=H(푅| 푠 C )− 피 푠 퐴 [H(푅| 푠 C , 푠 퐴 )] =− 1 2 log 1− 1 푁 2 Var(푅| 푠 C ) (Σ 퐴,푈|C 1 |푈| ) ⊤ Σ −1 퐴,퐴|C (Σ 퐴,푈|C 1 |푈| ) , whereΣ 퐴,푈|C denotes the submatrix ofΣ |C of columns and rows corresponding to퐴and the unevaluated set 푈= 퐸 , respectively, and 1 |푈| is a|푈|-dimensional vector of ones. A fundamental property of Gaussian conditioning where the noise is not a function of the score value implies thatΣ |C depends only on the score indexes rather than their actual values. This means that batch selection strategies simplify to a purely combinatorial problem of identifying an optimal batch of indices. Once this optimal batch is identified, the specific sequence in which these items are evaluated is arbitrary. Because the expected information gain is strictly monotonically increasing with respect to the variance reduction, maximizing 훼 IG (퐴) is equivalent to maximizing the variance reduction term: 훽(퐴)= 1 푁 2 (Σ 퐴,푈|C 1 |푈| ) ⊤ Σ −1 퐴,퐴|C (Σ 퐴,푈|C 1 |푈| ).(12) Therefore, for a batch 퐴 of specified size |퐴|= 푘, we obtain the following optimization problem: 퐴 ★ = argmax 퐴⊂푈,|퐴|=푘 훽(퐴).(13) Batch Size|퐴|=2 When|퐴|=2, we can derive a closed-form expression of훽(퐴=푖, 푗)that can be efficiently vectorized: 훽(푖, 푗)= 1 푁 2 푇 푖 푇 푗 Σ 푖,푖|C Σ 푖,푗|C Σ 푖,푗|C Σ 푗,푗|C −1 푇 푖 푇 푗 = 1 푁 2 푇 2 푖 Σ 푗,푗|C + 푇 2 푗 Σ 푖,푖|C − 2푇 푖 푇 푗 Σ 푖,푗|C Σ 푖,푖|C Σ 푗,푗|C −Σ 2 푖,푗|C , where 푇 푖 = Í 푘∈푈 Σ 푖,푘|C and similarly for 푇 푗 . Batch Size|퐴| >2 As the batch size|퐴|grows, an exhaustive search over all |푈| |퐴| subsets requiresO(|푈| |퐴| |퐴| 3 ) operations, which quickly becomes computationally infeasible even for moderate|푈|. We address this by replacing the exhaustive search with an approximation algorithm that scales linearly with|푈|. Specifically, we use heuristic warm-start strategies to identify initial candidate batches, followed by a combination of local search and Sequential Monte Carlo (SMC). Coupling local search with SMC prevents the algorithm from getting stuck in local optima: we maintain a population of candidate batches, apply local search to each independently, and resample them according to their fitness 훽(퐴). We use the following warm-start strategies: •LARS: Solve anℓ 1 -penalized continuous relaxation of Eq. (13). In particular, for푤 ∈ ℝ 푁 define the following function: 푔(푤)= 1 2 푤 ⊤ Σ |C 푤− 푤 ⊤ 푣, where 푣=Σ 퐸,푈|C 1 |푈| . Fix a subset 퐴 and let 푤 퐴 푐 = 0. The gradient of 푔 with respect to 푤 퐴 is ∇ 푤 퐴 푔=Σ 퐴,퐴|C 푤 퐴 − 푣 퐴 , 20 BayesAME: Bayesian Active Model Evaluation where푣 퐴 =Σ 퐴,푈|C 1 |푈| . Setting the gradient to zero yields the optimizer푤 ★ 퐴 =Σ −1 퐴,퐴|C 푣 퐴 . Substituting푤 ★ 퐴 into 푔 gives 푔(푤 ★ 퐴 )=− 1 2 푣 ⊤ 퐴 Σ −1 퐴,퐴|C 푣 퐴 =− 푁 2 2 훽(퐴). Therefore, maximizing훽(퐴)over all subsets퐴of size푘is equivalent to solving theℓ 0 -constrained quadratic program min 푤∈ℝ 푁 1 2 푤 ⊤ Σ |C 푤− 푤 ⊤ 푣 subject to ∥푤∥ 0 ≤ 푘. Since this problem is NP-hard, we instead consider the standard ℓ 1 relaxation min 푤∈ℝ 푁 1 2 푤 ⊤ Σ |C 푤− 푤 ⊤ 푣+ 휆∥푤∥ 1 , 휆 > 0. To solve this, we apply the least angle regression (LARS) algorithm to efficiently extract an initial active set of |퐴| items. •DPP: Sample multiple diverse initial batches using a푘-determinantal point process (푘-DPP) (Kulesza and Taskar, 2011) restricted to size|퐴|, using the posterior covariance matrix of the candidates as the likelihood kernel. •Clust: Perform퐾-means clustering on the reference model score vectors휙 푖 to partition the candidate pool into|퐴|distinct clusters. Generate multiple initial sets by uniformly sampling one item from each cluster. Crucially, during the local search phase, evaluating a neighbor where only a single item in the candidate batch is swapped can be computed highly efficiently using rank-2 updates. Suppose we wish to recompute the inverse in Eq. (12) after a swap. Let푀 퐴 =Σ 퐴,퐴|C be the original|퐴|×|퐴|matrix and푀 ′ 퐴 =Σ ̃ 퐴, ̃ 퐴|C be the updated matrix. If the푚-th item is swapped from퐴to ̃ 퐴, then푀 퐴 and푀 ′ 퐴 differ only by their푚-th row and column. Let 푒 푚 ∈ ℝ |퐴| denote the푚-th standard basis vector (a vector of zeros with a 1 at the푚-th coordinate). Let푐 푚 be the푚-th column of푀 퐴 and ̃푐 푚 be the corresponding column in푀 ′ 퐴 . Defining the difference vector푑= ̃푐 푚 − 푐 푚 , we can express the update as: 푀 ′ 퐴 = 푀 퐴 + 푑푒 ⊤ 푚 + 푒 푚 푑 ⊤ − 푑 푚 푒 푚 푒 ⊤ 푚 , where 푑 푚 denotes the 푚-th coordinate of 푑. This can be factorized into 푀 ′ 퐴 = 푀 퐴 + 푈푉 ⊤ , where 푈= 푒 푚 푑− 1 2 푑 푚 푒 푚 , 푉= 푑− 1 2 푑 푚 푒 푚 푒 푚 . This rank-2 factorization allows us to apply the Woodbury matrix identity: (푀 퐴 + 푈푉 ⊤ ) −1 = 푀 −1 퐴 − 푀 −1 퐴 푈(퐼 2 + 푉 ⊤ 푀 −1 퐴 푈) −1 푉 ⊤ 푀 −1 퐴 . Because푉 ⊤ 푀 −1 퐴 푈is only a 2×2 matrix, evaluating this update is extremely fast. We can plug this inverted matrix directly back into Eq. (12), reducing the computational complexity of evaluating a neighbor to just O(|퐴| 2 ) instead ofO(|퐴| 3 ). Once the optimal batch퐴 ★ is found, we can proceed in two different ways: (i) Myo: We evaluate the target model on all items in퐴 ★ simultaneously. Batch evaluation allows us to maximize throughput when multiple evaluations can be processed in parallel; or (i) Non-Myo: We select a single item from it at random. Because this item is selected based on an optimal future horizon rather than just the immediate next step, this renders the active selection non-myopic. Figure 4 presents the RMSE Log, RMSE Gain, and Cost-Accuracy Trade-Off for GPQA with binary (left two columns) and continuous (right two columns) scores. We evaluate the selection strategies outlined above using batch sizes of|퐴|=2 and|퐴|=5. We observe that the overall performance of these batch approaches is similar to that of the single-item selection strategy. Table 2 compares the Evaluation-Weighted Integrated RMSE achieved by BayesAME with single- candidate selection, random selection, bath selection with|퐴|=2 using both myopic (Myo) and non-myopic (Non-Myo) criteria, and Bayes-AIPW across the different benchmarks. Consistent with the results of Figure 4, BayesAME with batch size|퐴|=2 exhibits performance similar to that of myopic single-candidate selection under both the myopic and non-myopic variants. 21 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 50% 90% 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 InterpolationExtrapolationInterpolationExtrapolation 50% 90% 707580859095100 0.2 0.4 0.6 0.8 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 707580859095100 0.2 0.4 0.6 0.8 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 707580859095100 0.2 0.4 0.6 0.8 707580859095100 0.2 0.4 0.6 0.8 2030405060708090100 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 2030405060708090100 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo BayesAME |A|=5 Myo LARS BayesAME |A|=5 Myo DPP BayesAME |A|=5 Myo Clust BayesAME-RS RS-Mean 2030405060708090100 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.2 0.4 0.6 0.8 1.0 Figure 4|Single-target Setting. GPQA with binary scores (left two columns) and continuous scores (right two columns). RMSE Log (top two rows) and RMSE Gain (central two rows), and Cost-Accuracy Trade-Off (bottom two rows) across 50%, and 90% of reference models (percentages with unique reference scores퐵= 푁). B.2. Metadata-Weighted Prior Distribution When rich metadata (such as, model size, training data, or hyperparameter configurations) is available for both the reference and target models, we can formulate a weighted prior distribution. Specifically, we define a weight vector푤=(푤 1 , . . . , 푤 퐾 푟 ), where푤 푘 ≥0 captures the prior positive correlation between reference model 푘 and the target model, with Í 퐾 푟 푘=1 푤 푘 = 1. We then consider the prior distribution 푝(휃)=N(휇 휃 ,Σ 휃 ) with 휇 휃 푏 = 휙 ⊤ 푏 푤,Σ 휃 푏,푏 ′ = 훼exp − 훽 퐾 푟 (휙 푏 − 휙 푏 ′ ) ⊤ 푊(휙 푏 − 휙 푏 ′ ) , where푊= diag(푤). In practice, public leaderboards rarely provide sufficiently detailed metadata to calibrate these weights reliably, so we assign uniform weights across the reference models 푤 푘 = 1/퐾 푟 . 22 BayesAME: Bayesian Active Model Evaluation BenchmarksBayesAME BayesAME-RS BayesAME |A|=2 Myo BayesAME |A|=2 Non-Myo Bayes-AIPW Binary GPQAInterpolation61.5466.3862.0861.5167.21 Extrapolation68.9172.6569.4369.5073.41 MMLU-ProInterpolation30.1830.18--30.68 Extrapolation37.2037.20--37.69 BBHInterpolation31.7036.3530.5130.9835.54 Extrapolation38.3542.0238.4738.6641.34 ARC-ChallengeInterpolation36.5336.53--36.68 Extrapolation36.5636.56--38.03 MuSRInterpolation47.8647.86--49.18 Extrapolation54.1954.19--56.72 IFEvalInterpolation49.5356.1449.9950.0056.71 Extrapolation58.0858.08--60.52 Continuous GPQAInterpolation30.5834.4230.8429.8835.21 Extrapolation40.4040.7541.1740.7646.77 MMLU-ProInterpolation13.1117.1013.14 13.3217.19 Extrapolation26.8231.1026.2426.6031.33 BBHInterpolation13.04 17.5412.8113.1318.01 Extrapolation29.8035.5730.6230.4933.64 ARC-ChallengeInterpolation26.0426.47--26.19 Extrapolation33.1633.16--33.05 MuSRInterpolation27.8333.4227.9427.4334.34 Extrapolation36.9642.5635.7036.5743.69 Natural QA Openbook Interpolation28.2629.6828.4128.8929.12 Extrapolation20.8328.3620.4320.7927.70 Table 2|Evaluation-Weighted Integrated RMSE for 50% reference models. A dash - denotes that we are in a non-unique reference regime. Therefore, selection defaults to random and the entry is omitted. B.3. Logit-Normal Prior Distribution While achieving strong empirical performance, a Gaussian prior distribution on휃is not the best choice for binary and bounded continuous scores from a modeling viewpoint, as it assigns positive probability mass outside the true support. Below, we describe more appropriate alternative formulations that we considered. Case푠 푖 ∈ [0,1]We introduce a logit-normal model that respects the support of scores. Specifically, we map both the latent abilities and the observed scores to the unconstrained real line using the logit transformation. We define the latent variable vector푓= ( 푓 1 , . . . , 푓 퐵 ), where푓 푏 = logit(휃 푏 )= ln 휃 푏 1−휃 푏 , and assign a Gaussian prior distribution on it. The scores are similarly transformed, and we assume a Gaussian conditional distribution 푝(logit(푠 푖 )| 푓 푏 )=N( 푓 푏 , 푁 푏 휎 2 )for an item푖belonging to bucket푏. The posterior distribution of푓=( 푓 1 , . . . , 푓 퐵 ) conditioned on 푠 C is given by: 푝( 푓 | 푠 C ) ∝ 푝( 푓) Ö 푖∈C 푝(logit(푠 푖 )| 푓). Under this specification, 푝( 푓 | 푠 C ) is itself Gaussian and inference over 푓 remains fully tractable. Case푠 푖 ∈ 0,1Similarly to above, consider the latent variable vector푓=( 푓 1 , . . . , 푓 푏 ), where푓 푏 = logit(휃 푏 ), and assign a Gaussian prior distribution on it. In the case of binary responses, a more appropriate model would use a Bernoulli likelihood of the form 푝(푠 푖 | 푓)= 휎( 푓 푏 ) 푠 푖 (1− 휎( 푓 푏 )) 1−푠 푖 , for an item푖belonging to bucket푏. Then, the posterior distribution of푓= ( 푓 1 , . . . , 푓 퐵 )conditioned on푠 C is given by: 푝( 푓 | 푠 C ) ∝ 푝( 푓) Ö 푖∈C 푝(푠 푖 | 푓). 23 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAME-Logit BayesAME-Logit-RS 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAME-Logit BayesAME-Logit-RS 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAME-Logit BayesAME-Logit-RS 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAME-Logit BayesAME-Logit-RS 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 Figure 5|Single-target Setting. GPQA with binary scores (left two columns) and continuous scores (right two columns). RMSE Log across 50% and 90% of reference models. In this case, we can resort to variational inference or a Laplace approximation (Rasmussen and Williams, 2006) to compute the posterior. However, we are interested in the posterior of휃= 휎( 푓)=(1+ 푒 − 푓 ) −1 , conditioned on푠 C . The posterior mean can be computed as: 피[휃 푏 | 푠 C ]= ∫ 휎( 푓 푏 )푝( 푓 | 푠 C ) d푓, which does not admit a closed-form solution because of the nonlinearity of휎. We consider three strategies for approximating this integral. •Monte Carlo estimation. The main drawback is that obtaining accurate estimates for all푁items requires a large number of samples, which is prohibitive inside the sequential active learning loop. •Other numerical integration methods. We can approximate휎( 푓 푏 )in the integral with a rescaled probit function that has the same slope at the origin (MacKay, 1992), obtaining: 피[휃 푏 | 푠 C ] ≈ 휎 피[ 푓 푏 | 푠 C ] √︁ 1+ 휋Var( 푓 푏 |푠 C )/8 . •Linear approximation. Following Osborne et al. (2012), we linearize the logistic term around a reference point 푓 푏,0 , that is, 휎( 푓 푏 )= 휎( 푓 푏,0 )+ 휎( 푓 푏,0 )(1− 휎( 푓 푏,0 ))( 푓 푏 − 푓 푏,0 ) Under this approximation, the posterior mean of휃admits a closed-form expression. To select the reference point, we introduce an auxiliary multivariate Gaussian model in the original space and set휎( 푓 푏,0 )to be its posterior mean given푠 C , i.e.,휎( 푓 푏,0 )= 피[휃 푏 | 푠 C ], clipping the value to[0,1]if necessary to remain within the valid range. Although this auxiliary Gaussian model is not appropriate as it does not have the correct range, it is used solely to inform the choice of 푓 푏,0 . While these alternative formulations offer a more principled treatment of the score support, they require approximate inference at every step and introduce substantial computational overhead. In practice, we found that these approximations resulted in worse performance. For example, Figure 5 compares BayesAME and BayesAME-RS with their corresponding versions using logit-normal models (BayesAME-Logit and BayesAME-Logit-RS) on GPQA for both binary and continuous scores, using the linear approximation described above to derive the posterior for휃. For both settings and regardless of the selection strategy, BayesAME and BayesAME-RS outperform the logit-normal formulation. 24 BayesAME: Bayesian Active Model Evaluation C. Experimental Details and Additional Results C.1. Implementations Details BayesAME For the single-target setting, we use Algorithm 1 with휎 2 =10 −4 when퐵≥ 푁− 푁 0 and휎 2 =10 −1 otherwise,푁 0 =10,푊=30. As described in Appendix A.4, optimization is performed using L-BFGS-B with 훼 0 = 훽 0 = 1, and 퐹= 50. For the multi-target setting, we set푁 0 =10, and푊=30. Of the퐿covariances, half are squared exponential (Eq. (1)) and the rest are Matérn-3/2. As described in Appendix A.4, optimization is performed using AdamW, with훼 푙 and훽 푙 initialized to 1, each푤 푡 푖 to a sample fromN(0,0.01), and휎 2 to 0.1 for binary scores and to 0.01 for continuous scores. We use퐹=50, and learning rate휂=0.01 for binary scores and휂=0.001 for continuous scores. We set the margin푁 0 =10 to safely handle regimes where reference scores are nearly unique across items. Empirically, we identified a specific failure mode for active selection: when almost all items have unique combinations of reference scores (effectively forming singleton buckets), but one single bucket contains a significantly larger number of items, the active selection biases the posterior estimation and degrades performance. Enforcing푁 0 =10 prevents the algorithm from relying on active selection in this skewed regime, yielding stable results across all scenarios considered in this work. While fixing푁 0 =10 serves as a safe value, because the total number of buckets and the exact distribution of items across them are fully known to the algorithm, a more sophisticated future approach could explicitly incorporate this distributional information to set more suitable, benchmark-specific values. We conjecture that larger threshold values may be admissible when bucket sizes are strictly balanced, as the observed estimation bias appears to be driven by the presence of a single, severely unbalanced bucket. When selecting items in batches, we scale the window size푊proportionally. Specifically, for a batch size|퐵|, we set 푊= 30/|퐵|. Baselines For Bayes-AIPW, Bayes-RS-Learn, and Seq-APW, we adapt the implementations of AIPW, Random-Sampling-Learn, and APW, respectively, provided by Zhang et al. (2025) inhttps://github.com /socialfoundations/benchmark-prediction . In Bayes-AIPW and Bayes-RS-Learn, we replace the linear ridge regression with its Bayesian counterpart as implemented inscikit-learnlibrary using the default setting (Pedregosa et al., 2011). For ProEval, we use the implementation provided by Huang et al. (2026) inhttps://github.com/google-deepmind/proeval. For IRT, we implement the binary and continuous approaches based on the methodology outlined in Polo et al. (2024) and Chen et al. (2019), respectively. C.2. Additional Benchmark Details For BBH, we only consider a subset of scenarios: temporal sequences, salient translation error detection, tracking shuffled objects (seven objects), geometric shapes, and reasoning about colored objects. For MMLU-Pro, we also restrict our evaluation to a subset of subjects, namely business and law, resulting in a total of 1890 items. C.3. Additional Results Single Target Setting Figure 6 shows that our Bayesian extension of AIPW matches or outperforms the original method of Zhang et al. (2025). We present extended results for all benchmarks considered. To facilitate navigation of the plots, Table 3 summarizes the figures corresponding to each benchmark and evaluation metric. 25 BayesAME: Bayesian Active Model Evaluation Plot TypeGPQA MMLU-Pro BBH ARC-Challenge MuSR IFEval Natural QA Openbook RMSE Log, RMSE Gain & Cost-Accuracy Tradeoff78910111212 Spearman Correlation13–13– Table 3 | Figure index for the extended experimental results across all benchmarks. InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAIPW AIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 Figure 6|Single-target Setting. GPQA (top three rows) and MMLU-Pro (bottom three rows) with binary scores (left two columns) and continuous scores (right two columns). RMSE Log for the AIPW ablation study across varying proportions of reference models (10%, 50%, and 90%) ablation for AIPW. C.4. Multi-target Setting Figure 14 show the results comparing BayesAME Multi-target with BayesAME Single-target on GPQA using both binary and continuous scores. Figure 15 reports results for the alternative selection strategy introduced in Section 3.2, which selects items based on the average marginalized information gain across target models. 26 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 2 1 0 1 2 3 4 5 6 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 20406080 2 1 0 1 2 3 4 5 6 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 BayesAME BayesAME-RS BayesAIPW RS-Mean 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 BayesAME BayesAME-RS BayesAIPW RS-Mean 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 80859095100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 5060708090100 0.2 0.4 0.6 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.2 0.4 0.6 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.2 0.4 0.6 0.8 5060708090100 0.2 0.4 0.6 0.8 5060708090100 0.2 0.4 0.6 0.8 5060708090100 0.2 0.4 0.6 0.8 Figure 7|Single-target Setting. GPQA with binary scores (left two columns) and continuous scores (right two columns). RMSE Log (top three rows), RMSE Gain (middle three rows), and Cost-Accuracy Trade-Off (bottom three rows) across varying proportions of reference models. 27 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 5060708090100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.0 0.2 0.4 0.6 0.8 1.0 5060708090100 0.0 0.2 0.4 0.6 0.8 1.0 5060708090100 0.0 0.2 0.4 0.6 0.8 1.0 5060708090100 0.0 0.2 0.4 0.6 0.8 1.0 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 20406080100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Figure 8|Single-target Setting. MMLU-Pro with binary scores (left two columns) and continuous scores (right two columns). RMSE Log (top three rows), RMSE Gain (middle three rows), and Cost-Accuracy Trade-Off (bottom three rows) across varying proportions of reference models. 28 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 5060708090100 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.2 0.4 0.6 0.8 1.0 5060708090100 0.2 0.4 0.6 0.8 1.0 5060708090100 0.2 0.4 0.6 0.8 1.0 5060708090100 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 2030405060708090100 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 2030405060708090100 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.2 0.4 0.6 0.8 1.0 Figure 9|Single-target Setting. BBH with binary scores (left two columns) and continuous scores (right two columns). RMSE Log (top three rows), RMSE Gain (middle three rows), and Cost-Accuracy Trade-Off (bottom three rows) across varying proportions of reference models. 29 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 60708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 60708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 60708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 60708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 60708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 60708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 30405060708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 30405060708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 30405060708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 30405060708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 30405060708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 30405060708090100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Figure 10|Single-target Setting. ARC-Challenge with binary scores (left two columns) and continuous scores (right two columns). RMSE Log (top three rows), RMSE Gain (middle three rows), and Cost-Accuracy Trade-Off (bottom three rows) across varying proportions of reference models. 30 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 20406080 0 2 4 6 8 10 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 707580859095100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 707580859095100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 707580859095100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 707580859095100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 707580859095100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 707580859095100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 BayesAME BayesAME-RS BayesAIPW RS-Mean 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Figure 11|Single-target Setting. MuSR with binary scores (left two columns) and continuous scores (right two columns). RMSE Log (top three rows), RMSE Gain (middle three rows), and Cost-Accuracy Trade-Off (bottom three rows) across varying proportions of reference models. 31 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 BayesAME BayesAME-RS BayesAIPW ProEval Seq-APW Bayes-RS-Learn IRT RS-Mean 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 020406080100 10 2 10 1 10 0 10 1 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 20406080 2 0 2 4 6 8 10 12 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 0 2 4 6 8 10 12 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 0 2 4 6 8 10 12 BayesAME BayesAME-RS BayesAIPW RS-Mean 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 20406080 2 0 2 4 6 8 10 12 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 707580859095100 0.2 0.4 0.6 0.8 1.0 1.2 BayesAME BayesAME-RS BayesAIPW RS-Mean 707580859095100 0.2 0.4 0.6 0.8 1.0 1.2 BayesAME BayesAME-RS BayesAIPW RS-Mean 707580859095100 0.2 0.4 0.6 0.8 1.0 1.2 707580859095100 0.2 0.4 0.6 0.8 1.0 1.2 707580859095100 0.2 0.4 0.6 0.8 1.0 1.2 707580859095100 0.2 0.4 0.6 0.8 1.0 1.2 2030405060708090100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 2030405060708090100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 2030405060708090100 0.0 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.0 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.0 0.2 0.4 0.6 0.8 1.0 2030405060708090100 0.0 0.2 0.4 0.6 0.8 1.0 Figure 12|Single-target Setting. IFEval with binary scores (left two columns) and Natural QA Openbook with continuous scores (right two columns). RMSE Log (top three rows), RMSE Gain (middle three rows), and Cost-Accuracy Trade-Off (bottom three rows) across varying proportions of reference models. 32 BayesAME: Bayesian Active Model Evaluation InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 InterpolationExtrapolationInterpolationExtrapolation 10% 50% 90% 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 BayesAME BayesAME-RS BayesAIPW RS-Mean 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 020406080100 0.0 0.2 0.4 0.6 0.8 1.0 Figure 13|Single-target Setting. GPQA (top three rows) and BBH (bottom three rows) with binary scores (left two columns) and continuous scores (right two columns). Spearman’s rank correlation across varying proportions of reference models. 33 BayesAME: Bayesian Active Model Evaluation Low correlationHigh correlationLow correlationHigh correlation 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090 2 0 2 4 6 8 102030405060708090 2 0 2 4 6 8 102030405060708090 2 0 2 4 6 8 102030405060708090 2 0 2 4 6 8 80859095100 0.2 0.4 0.6 0.8 80859095100 0.2 0.4 0.6 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Figure 14|Multiple-target Setting. GPQA with binary scores (left two columns) and with continuous scores (right two columns). The rows represent RMSE Log, RMSE Gain, and Cost-Accuracy Trade-Off, respectively. We compare the multitarget method against methods that do not model correlations across targets. Low correlationHigh correlationLow correlationHigh correlation 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME Multi-target Avg. Marg. IG BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME Multi-target Avg. Marg. IG BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME Multi-target Avg. Marg. IG BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090100 10 2 10 1 10 0 10 1 BayesAME Single-target BayesAME Multi-target BayesAME Multi-target Avg. Marg. IG BayesAME-RS Single-target BayesAIPW RS-Mean 102030405060708090 0 2 4 6 8 10 102030405060708090 0 2 4 6 8 10 102030405060708090 0 2 4 6 8 10 102030405060708090 0 2 4 6 8 10 80859095100 0.0 0.2 0.4 0.6 0.8 80859095100 0.0 0.2 0.4 0.6 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 5060708090100 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Figure 15|Multiple-target Setting. GPQA with binary scores (left two columns) and with continuous scores (right two columns). The rows represent RMSE Log, RMSE Gain, and Cost-Accuracy Trade-Off, respectively. We compare the different selection strategies in the multitarget setting. 34

Instant research discovery