Paper deep dive
Coverage Aware Active Evaluation for Failure Discovery with Paired Systems
Anjali Parashar, Rachel Luo, Apoorva Sharma, Sushant Veer, Edward Schmerling, Carson Sobolewski, Mingxin Yu, Chuchu Fan, Marco Pavone
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/17/2026, 4:10:05 AM
Summary
The paper proposes an adaptive failure discovery method for autonomous systems that leverages cheap proxy evaluations (e.g., simulators) combined with limited, expensive target system evaluations to predict and discover severe, diverse failures. The method uses a control-variate-inspired residual model to correct proxy failure signals for accurate target risk estimation and a support-aware mutual-information objective to ensure scenario diversity. Experiments across autonomous driving, manipulation, and quadruped tasks show the method discovers up to 2x more failures than baselines.
Entities (8)
Relation Signals (8)
Proposed Method → evaluatedon → KITTI
confidence 95% · We validate these hypotheses on four autonomous system tasks: nuPlan, SIMPLER, Quadruped, and KITTI
Proposed Method → evaluatedon → nuPlan
confidence 95% · We validate these hypotheses on four autonomous system tasks: nuPlan, SIMPLER, Quadruped, and KITTI
Proposed Method → evaluatedon → SIMPLER
confidence 95% · We validate these hypotheses on four autonomous system tasks: nuPlan, SIMPLER, Quadruped, and KITTI
Proposed Method → evaluatedon → Quadruped
confidence 95% · We validate these hypotheses on four autonomous system tasks: nuPlan, SIMPLER, Quadruped, and KITTI
Support-Aware Mutual Information → optimizes → Scenario Diversity
confidence 90% · we combine this predictor with a support-aware mutual-information objective that favors realistic, well-supported regions while expanding coverage across failure modes.
Control Variate Predictor → uses → Proxy Evaluations
confidence 90% · Our method learns a local predictor of target risk by correcting proxy failure signals using control-variate-inspired residual modeling.
nuPlan → usesmodel → Bayesian Neural Network
confidence 85% · We use BNN for modeling qθ for nuPlan, SIMPLER, KITTI
Quadruped → usesmodel → Multi-Layer Perceptron
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autonomous systems can fail in rare and heterogeneous ways, making real-world failure discovery difficult under limited testing budgets. Although cheaper proxies such as simulators, lower-fidelity systems, or related policies can be sampled extensively to find failures, proxy failures often do not transfer to the real world due to sim-to-real and system-to-system gaps. The key challenge is therefore to effectively leverage proxy system information for accurate prediction of severe target system failures. We propose an adaptive failure discovery method that combines proxy evaluations with limited target system results to guide scenario selection for target system testing. Our method learns a local predictor of target risk by correcting proxy failure signals using control-variate-inspired residual modeling. To find failures that are both likely and diverse, we combine this predictor with a support-aware mutual-information objective that favors realistic, well-supported regions while expanding coverage across failure modes. Across autonomous driving, manipulation, and quadruped velocity-tracking tasks, our method discovers up to 2$\times$ as many failures as random sampling and active-learning baselines, including severe and diverse failures missed by competing methods.
Tags
Links
- Source: https://arxiv.org/abs/2608.13719v1
- Canonical: https://arxiv.org/abs/2608.13719v1
Trouble viewing inline? Open PDF directly →
Full Text
67,747 characters extracted from source content.
Expand or collapse full text
Coverage Aware Active Evaluation for Failure Discovery with Paired Systems Anjali Parashar Thanks: Corresponding author Affiliation: Laboratory of Information & Decision Systems (LIDS), MIT, USA Rachel Luo Affiliation: NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia.com Apoorva Sharma Affiliation: NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia.com Sushant Veer Affiliation: NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia.com Edward Schmerling Affiliation: NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia.com Carson Sobolewski Affiliation: Laboratory of Information & Decision Systems (LIDS), MIT, USA Mingxin Yu Affiliation: Laboratory of Information & Decision Systems (LIDS), MIT, USA Chuchu Fan Affiliation: Laboratory of Information & Decision Systems (LIDS), MIT, USA Marco Pavone Affiliation: NVIDIA Research, USAanjalip,csobo,yumx35,chuchu@mit.eduraluo, apoorvas, sveer, eschmerling, mpavone@nvidia.com Abstract Autonomous systems can fail in rare and heterogeneous ways, making real-world failure discovery difficult under limited testing budgets. Although cheaper proxies such as simulators, lower-fidelity systems, or related policies can be sampled extensively to find failures, proxy failures often do not transfer to the real world due to sim-to-real and system-to-system gaps. The key challenge is therefore to effectively leverage proxy system information for accurate prediction of severe target system failures. We propose an adaptive failure discovery method that combines proxy evaluations with limited target system results to guide scenario selection for target system testing. Our method learns a local predictor of target risk by correcting proxy failure signals using control-variate-inspired residual modeling. To find failures that are both likely and diverse, we combine this predictor with a support-aware mutual-information objective that favors realistic, well-supported regions while expanding coverage across failure modes. Across autonomous driving, manipulation, and quadruped velocity-tracking tasks, our method discovers up to 2x as many failures as random sampling and active-learning baselines, including severe and diverse failures missed by competing methods. Keywords: Testing & evaluation, Adaptive Experimental Design 1 Introduction Autonomous systems must be evaluated not only for average-case performance, but also for rare, severe, and heterogeneous failures that arise under specific scenario conditions. Such failures are especially important in safety-critical domains, where a small number of unanticipated edge cases can lead to unsafe behavior after deployment [11, 12, 8, 7, 25]. However, discovering these failures through direct evaluation of the target system is difficult, since real-world or high-fidelity closed-loop tests can be expensive, time-consuming, and resource-constrained, and modern autonomous systems are often too complex to model analytically [18, 24]. Cheap proxy systems, such as simulators, lower-fidelity platforms, related policies, or alternative evaluation processes, provide a natural way to scale failure search, but also introduce a central challenge over accuracy of failure estimation. Failures discovered in a proxy may reflect proxy-specific artifacts rather than true weaknesses of the target system. A scenario may appear critical because of simplified dynamics, fidelity mismatch, or distributional gaps, yet be benign on the target system; conversely, scenarios that appear safe in the proxy may still induce severe target-system failures. Thus, the goal is not merely to find failures cheaply, but to leverage proxy system carefully to accurately support the prediction of predictive of target-system failures under a limited target-evaluation budget. Existing approaches address only parts of this problem. Simulation-based testing can efficiently search low-cost models but may not transfer to the target system, while target-system active testing directly searches for critical scenarios but often does not exploit abundant proxy evaluations [10, 9, 20, 23, 2, 24, 18]. Methods that combine proxy and target evaluations can improve estimation of aggregate target-system metrics [17, 2], but these methods are often designed for Monte Carlo sampled data that can be ineffective for failure search. Failure discovery requires a different objective: actively selecting scenarios that reveal many distinct target-system failures. This also requires estimation of scenario specific metrics as opposed to aggregate metrics proposed by existing works. Figure 1: Overview of approach. Our method uses support-aware Mutual Information (Equation 3) to subsample KMIK_MI scenarios x, which are evaluated on the cheap proxy system to get metrics ysy_s, and a surrogate model qθq_θ trained using data evaluated on target system. We subsequently optimize μCV(x) _CV(x) (Equation 2) over these sub sampled candidates to select the next batch of evaluation points as likely failure scenarios. The process is repeated until evaluation budget B is exhausted. In this work, we study adaptive failure discovery using cheap proxy evaluations and limited target-system tests. Our goal is to discover a large and diverse set of target-system failures under a fixed target-evaluation budget, where diversity corresponds to covering distinct regions of the scenario space, operating conditions, or failure modes. We propose an adaptive acquisition strategy that treats proxy evaluations as biased but informative signals. Rather than treating proxy failures as ground truth, our method learns a local failure predictor that estimates target-system risk by correcting proxy failure signals using limited target evaluations. This predictor is inspired by control variate estimation [17], where proxy observations provide a low-cost signal correlated with target behavior, while target evaluations correct for local proxy-target mismatch. The resulting estimate prioritizes scenarios that are likely to fail on the target system, rather than merely fail in the proxy. Our contributions are as follows: 1. We formulate proxy-guided failure discovery as an adaptive scenario selection problem, where the goal is to use limited target-system evaluations to discover many distinct target failure modes. 2. We propose a local, proxy-corrected failure predictor that uses limited paired proxy-target evaluations to identify scenarios likely to fail on the target system, rather than merely in the proxy evaluator. 3. We introduce a support-aware mutual-information acquisition strategy that balances target-failure likelihood with coverage of realistic and underexplored regions of the scenario space, encouraging acquisition of novel failure scenarios. 4. We evaluate our method across autonomous driving, multi-fidelity manipulation, and quadruped velocity-tracking tasks, showing that it discovers more target-system failures and reveals a broader range of critical failure modes than random sampling and active-learning baselines. Across these domains, our method discovers upto twice as many target-system failures than the strongest baseline under the same target-evaluation budget, discovering several low and high severity failures, where the baselines completely miss discovery of high severity failures. Our method also achieves highest diversity across all tasks and baselines, under a pairwise-distance-based coverage metric. These results show that cheap proxy evaluations can substantially improve failure discovery when used to guide target-system testing in an adaptive manner, rather than as direct substitutes for target-system evaluation. 2 Related Work Simulation-based failure discovery. Simulation enables scalable, low-cost testing of autonomous systems via adaptive sampling, stress testing, and failure discovery [10, 9]. However, these methods are sensitive to sim-to-real gaps and model misspecification, so scenarios appearing safe in simulation may still fail on the real system [20, 23, 18]. We address this by treating proxy evaluations as biased but informative signals, and using limited target evaluations to adaptively correct the proxy-to-target relationship during failure discovery. Sample-efficient target-system testing. A complementary line of work focuses on failure discovery directly on the target system using expert-designed scenario sets, or surrogate based Bayesian optimization, or active learning under limited budgets [18, 21, 1]. Surrogate based approaches can struggle to scale when scenarios are high-dimensional or failure landscapes are complex [24, 21]. Our method differs by not depending on restrictive model choices while modeling target-system risk. Combining proxy and target evaluations. Recent works use Control variates and related variance-reduction methods to use correlated proxy signals for improved estimation of aggregate real-world metrics under limited target evaluations [17, 2]. These methods are typically designed for Monte Carlo estimation of global quantities, which is inefficient for discovering rare failures. Our goal instead is to actively select scenarios that reveal many target-system failures, motivating an adaptive acquisition strategy that uses proxy-target correction for both failure estimation and scenario selection. Bayesian experimental design and information-based acquisition. Bayesian experimental design approaches select informative evaluations under limited budgets [22, 5], often via Mutual Information (MI) [4, 19]. However, for failure discovery, uncertainty alone may target non-critical scenarios, while exploitation alone may produce duplicate failures. We therefore combine a support-aware mutual-information objective for exploring under-covered regions with a proxy-corrected risk estimate to prioritize severe failures. Positioning of this work. Our work connects these lines by combining cheap proxy evaluations, limited target-system tests, and information-based acquisition. Unlike simulation-only testing, we do not assume proxy failures transfer directly to the target. Unlike target-only active testing, we leverage proxy evaluations to guide search. Unlike variance-reduced estimation, our goal is not to estimate an aggregate metric but to select evaluations that reveal diverse failure modes. Together, these components enable sample-efficient discovery of distinct target-system failures. 3 Problem Statement We consider a black-box dynamical system that maps scenario parameters x∈x to trajectory rollouts. Our goal is to discover failures of this target system, which can be expensive to evaluate. Failure is measured by a safety metric yr∈y_r , where, without loss of generality, larger values indicate less safe behavior. We assume access to a cheaper proxy system that provides low-cost observations ysy_s for the same scenarios. Rollout stochasticity or environmental variations induce distributions p(ys∣x)p(y_s x) and p(yr∣x)p(y_r x) for these systems. Because proxy evaluations are inexpensive, p(ys∣x)p(y_s x) can be estimated by sampling, while estimating p(yr∣x)p(y_r x) remains difficult due to high cost of evaluations (Figure 1). Our objective is to use paired proxy and target information to discover a large number of target system failures, spanning diverse scenarios or operating conditions, under a target evaluation budget B. For scenario x, let μr(x)≔p(yr∣x)[yr] _r(x) _p(y_r x)[y_r]. We define failure as μr(x)≥γ _r(x)≥γ, where γ is a user-specified severity threshold. The goal is to identify a diverse set of failures XB=xi∈∣μr(xi)≥γi=1K.X_B=\x^i _r(x^i)≥γ\_i=1^K. We promote diversity using mutual information (MI), which is commonly used to quantify epistemic uncertainty from limited evaluations [19, 4, 22, 5]. To maintain the evaluation budget, we formulate failure discovery as adaptive scenario sampling using Bayesian Experimental Design (BED) [22]. At iteration k≤Bk≤ B, given observed target evaluations k=(xi,yri)i=1kD_k=(x^i,y_r^i)_i=1^k, we select the next scenario by maximizing an acquisition function α:→ℝα:X as xk+1=argmaxx∈α(x).x_k+1= _x α(x). Our acquisition must balance two goals: selecting scenarios likely to satisfy μr(x)≥γ _r(x)≥γ, and covering diverse operating conditions through high MI. We next describe the design of this acquisition strategy. 4 Data acquisition strategy for meeting dual evaluation objectives Because the target system is black-box, directly estimating μr _r is difficult. We can instead learn a surrogate, qθ(x)q_θ(x) from observed target evaluations k=(xi,yri)i=1kD_k=\(x^i,y_r^i)\_i=1^k to approximate p(yr|x)p(y_r|x), and use it to guide future scenario selection. However, target-only surrogates ignore the readily available proxy data. To combine predictions from qθ(x)q_θ(x) learnt using limited target evaluations with biased, noisy proxy signals for accurate failure prediction, we introduce a local control-variate predictor μCV _CV in Section 4.1. To promote scenario diversity, we introduce a support-aware MI term I(Zx;Rx∣x,t)I(Z_x;R_x x,D_t) in Section 4.2, where ZxZ_x captures the evaluated data support and RxR_x denotes the potential support improvement from candidate x. Combining this diversity objective with the control-variate failure predictor gives the high level objective for data acquisition: α(x∣t)=μCV(x)⏟failure estimation+I(Zx;Rx∣x,t)⏟scenario diversity.α(x _t)= _CV(x)_failure estimation+ I(Z_x;R_x x,D_t)_scenario diversity. (1) The two objectives cumulatively represent candidates that are both likely target failures and informative for expanding support. We define μCV _CV and the support-aware MI term next. 4.1 Combining target and proxy system information for failure estimation We first describe the construction of μCV _CV for failure estimation. Consider a candidate scenario x∈x for target-system testing, where the goal is to identify scenarios with μr(x)≥γ _r(x)≥γ. As discussed above, relying only on the surrogate qθq_θ can be inefficient because it is learned from limited target data and ignores proxy evaluations p(ys)p(y_s). We therefore propose a local, scenario-specific mean estimator μCV(x) _CV(x) based on the Control Variate (CV) method [17]. By incorporating μCV _CV in α, we enable a more accurate failure estimation at a given x, than can be achieved using just YrY_r with limited samples, or, using a large number of proxy samples YsY_s. This estimator uses proxy evaluations p(ys)p(y_s) as additional information to improve the target mean estimate obtained from qθq_θ. For a candidate scenario x, we construct a neighborhood ℬxB_x and sample N scenarios xi∼ℬx^i _x for surrogate prediction. Here, ℬxB_x can denote a categorical scenario type or a Euclidean ball in continuous scenario space, as detailed in Appendix C. Within this region, we uniformly sample at most N=n+kN=n+k datapoints: paired evaluations (xi,y^ri,ysi)i=1n(x^i, y_r^i,y_s^i)_i=1^n, where y^r∼qθ(x) y_r q_θ(x), and additional proxy evaluations (xj,ysj)j=1k(x^j,y_s^j)_j=1^k. Using this data, we approximate μr(x) _r(x) using μCV(x) _CV(x), a local mean estimate of the metric at x, as: μCV(x)=1n∑i=1n(y^ri−ysi)+1k∑j=1kysi,where=(k+n)Var(Ys)−1Cov(Yr,Ys). _CV(x)= 1n _i=1^n( y_r^i- βy_s^i)+ 1k _j=1^k βy_s^i,~~where~~ β= ( kk+n )Var(Y_s)^-1Cov(Y_r,Y_s). (2) Here Ys=(ysj)j=1n+k,Yr=(y^rj)j=1nY_s=(y_s^j)_j=1^n+k,Y_r=( y_r^j)_j=1^n, and β adjusts the mean estimated using proxy metrics based on the correlation between target and proxy distributions qθ(x)q_θ(x) and p(ys|x)p(y_s|x) for x∈ℬx _x. This results in estimator μCV _CV that corresponds to minimum variance estimator of the failure statistics in the local region (see Appendix A for details). Figure 7 shows the application of this approach for failure prediction on a 1D toy task. 4.2 Formalizing scenario diversity for exploration To encourage scenario diversity in adaptive data acquisition, we aim to prioritize scenarios away from observed scenarios. To this end, we construct a support aware MI, such that maximizing MI identifies novel scenarios. As a first step, we explicitly identify scenario space covered using tD_t. This can be done by clustering observed data in K clusters in the scenario space. For a an unseen potential candidate x, let Zx∈[1,…,K,new]Z_x∈[1,…,K,new] denote cluster assignment at x, where Zx=newZ_x=new implies formation of new cluster by evaluating at x, implying that the point is not within the covered scenario space. Tied to x, we also define a support discovery variable Rx∈0,1R_x∈\0,1\, where Rx=1R_x=1 denotes that x reveals new support. Thus, our goal of encouraging diversity in scenario can be defined as discovery of novel candidates far from support and those within support with uncertainty on cluster assignment. Mathematically, this can be naturally captured by MI as I(Zx;Rx|t,x)I(Z_x;R_x|D_t,x) in a principled manner, defined using entropy H as: I(Zx;Rx∣Dt,x)=H(Zx|t,x)−H(Zx|Rx,t,x).I(Z_x;R_x D_t,x)=H(Z_x|D_t,x)-H(Z_x|R_x,D_t,x). (3) To enable a direct and tractable connection between RxR_x and ZxZ_x, we model the distribution over RxR_x as p(Rx|x,t)=∑k=1Kp(Rx|Zx=k)p(Zx=k|x,t)p(R_x|x,D_t)= _k=1^Kp(R_x|Z_x=k)p(Z_x=k|x,D_t). Here, prior p(Zx=k|t,x)p(Z_x=k|D_t,x) controls probability of assignment of a point x to cluster k, and we define the likelihood p(Rx=1|Zx=k)p(R_x=1|Z_x=k) to be very low for scenarios well within one of the k clusters, and set it to be high for Zx=newZ_x=new (see Appendix D for details). Under these modeling choices, the MI term in Equation 3 prioritizes scenarios for which cluster assignment is ambiguous, due to lying between cluster boundaries, and scenarios far from existing support. Thus, evaluating on these points enables meaningful support expansion, meeting our goal of scenario diversity. Unlike GP-based approaches such as [24], our method scales to high-dimensional scenarios because it is agnostic to the surrogate architecture qθq_θ and encodes exploration directly through support. This enables scaling beyond evaluation budgets of only tens of samples. In Figure 2, we illustrate this behavior on a large-scale, high-dimensional dataset, where the method prioritizes unexplored regions under different supports induced by evaluated samples tD_t. 4.3 Batched data acquisition for diverse failure discovery Using the two main components in Equation 1, we can optimize α for scenario discovery in a continuous domain. However, in our experimental validation, the scenario space X is a collection of discrete samples, and our objective is to get μCV≥γ _CV≥γ. Hence, we transform the optimization procedure into a severity adjusted data sampling rule with batched data acquisition of b scenarios at each iteration, for a budget of B=TbB=Tb samples. The batch size b is assumed to be user defined, and selected to be large enough to provide sufficient dataset for updating qθq_θ based on scenario dimension, and low enough that we do not surpass the budget B. At each step t, we select top-KMIK_MI candidates that maximize I(Zx;Rx|x,t)I(Z_x;R_x|x,D_t), followed by construction of the set Ωt=x|μCV(x)≥γ _t=\x| _CV(x)≥γ\. We select b candidates from Ωt _t by clustering the samples into b clusters and selecting candidate closest to cluster center for each, to further reinforce diversity within the batch. We note that the final algorithm is not a complete equivalent of Equation 1, but represents a constrained batched optimization equivalent instead, with the said constraints and batching procedures enabled for efficiency. Algorithm 1 summarizes the data acquisition procedure. Algorithm 1 Adaptive scenario discovery using (1). 1: Input: Scenario space X, initial paired 0=xi,yiri=1N0D_0=\x^i,y^r_i\_i=1^N_0 2: Initialize surrogate models q(x)q(x) trained using 0D_0. 3: for t=0t=0 to T−1T-1 do 4: Select scenario u=xi|argmaxxi∈I(Z;Rx|xi,t)i=1KMIX_u=\x^i| _x^i I(Z;R_x|x^i,D_t)\_i=1^K_MI for I in Equation 3, 5: Construct Ωt=x∣μCV(x)≥γ _t=\x _CV(x)≥γ\ for μCV _CV in Equation 2. 6: Select (xi)i=1b(x^i)_i=1^b for xi∈Ωtx^i∈ _t, closest to cluster centers for b cluster on Ωt _t. 7: Collect (yri)i=1b(y_r^i)_i=1^b for (xi)i=1b(x^i)_i=1^b using target system evaluation 8: Update dataset: t+1=t∪(xi,yri)i=1bD_t+1=D_t∪\(x^i,y^i_r)\_i=1^b 9: Update surrogate models qθq_θ, using t+1D_t+1. 10: end for 5 Experimental validation The central hypotheses of this paper are: (1) our approach can identify a concise set of scenarios that reveal failures across a range of severity levels and diverse operating conditions, and (2) it scales to larger evaluation budgets for high-dimensional and complex scenarios. We validate these hypotheses on four autonomous system tasks: nuPlan, SIMPLER, Quadruped, and KITTI, spanning autonomous driving, manipulation, and quadruped velocity tracking. These tasks represent a broad range of scenario specifications, proxy-target correlation structures, and evaluation costs. Tasks. In nuPlan, we evaluate motion-planning behavior using driving scenarios from the nuPlan dataset [14], with open-loop playback as the proxy and closed-loop rollouts as the target. Scenarios are encoded in a 384-dimensional space, and we use Time-to-Collision (TTC) as the failure metric. In SIMPLER, we evaluate the language-conditioned manipulation policy RT-1x [6] on a grasp and lift task [15], where proxy and target correspond to two photorealistic simulator environments differing in visual specifications; scenarios are parameterized by a 19-dimensional space encoding object positions, distractors, and camera pose. In Quadruped, we evaluate a Unitree Go2 quadruped [26] on a 3 dimensional command velocity tracking task, using a MuJoCo-based simulation with an RL policy [27] as the proxy, demonstrating real-world applicability of our approach. In KITTI, we analyze perception failures in autonomous driving, using a continuous failure score from GT vehicle coverage on a real world driving dataset KITTI [13], paired with its synthetic clone, Virtual KITTI [3]. These results demonstrate that our method utilizes proxy information effectively, and improves failure diversity. We also provide a supplementary video showing our approach for failure discovery on Quadruped task. We use BNN for modeling qθq_θ for nuPlan, SIMPLER, KITTI and MLP for Quadruped to show that unlike baselines such as [24, 21], our method is not sensitive to the choice of surrogate model. Appendix I provides additional implementation details for all tasks, and Appendix H shows visualization of exemplar failure scenarios discovered by our approach, in addition to dominant failure modes observed. Proxy system budget. We control proxy budget using a fixed proxy-compute budget Bp=NpcpB_p=N_pc_p across tasks, adjusting NpN_p inversely with per-scenario proxy runtime cpc_p. Total measurable computational cost is C=Ntct+Npcp+Ctraint+Ctrainp,C=N_tc_t+N_pc_p+C_train^t+C_train^p, where CtrainpC_train^p (surrogate training for proxy) is optionally used when proxy data is scarce, and Ctrainp,CtraintC_train^p,C_train^t are much smaller than evaluation costs, therefore C≈Ntct+NpcpC≈ N_tc_t+N_pc_p. We consider tasks with a variety of proxy runtimes, for example, SIMPLER has high cpc_p, ≈ 83s/eval vs. 2s/eval for Quadruped. This limits proxy data in SIMPLER to a total 110 samples (50 initial and 60 sequential) whereas in Quadruped, we are able to use 2,000 samples. In practice, target testing also incurs resource, and potential-damage costs that cannot be reduced to a scalar cost. These task-specific constraints motivate conservative target testing budget Nt=N0+BN_t=N_0+B, reflecting our focus on fixed target-evaluation budgets. Baselines. We compare our approach against the following baselines: Random (naïve MC sampling); BAMS [24], which uses a GP surrogate with multi-fidelity variance for acquisition; BNN-C/GP-C [18], which acquires worst-case scenarios via clustering with a BNN or GP surrogate; and BNN-CV, an ablation of our method without MI-driven exploration. We use GP for SIMPLER, and BNN for nuPlan for BNN-C/GP-C. Metrics. To validate our first hypothesis, we evaluate all methods using three metrics. Cumulative average metric measures the average severity of discovered scenarios up to budget k, computed as 1k∑k=1Byrk 1k _k=1^By_r^k. Cumulative coverage measures diversity among high-failure scenarios ℱtrue=x∈XB:yr≥γF_true=\x∈ X_B:y_r≥γ\ via their average pairwise distance, Ck(ℱtrue)=1k∑i,j∈[1,…,k]‖xi−xj‖22C_k(F_true)= 1k _i,j∈[1,…,k] \|x^i-x^j \|^2_2 for 1≤k≤B,1≤ k≤ B, where larger values indicate failures spread across the scenario space rather than collapsing onto a single region. Positive samples P=|ℱtrue|P= _true counts discovered failures at varying severity thresholds γ, reported for nuPlan in Table 1. Ablations We perform following ablations on our approach using the nuPlan task, (1) varying size of initial dataset (Appendix E) and form of sampling to demonstrate the utility of support aware MI (Figure 2), (2) ablations on formulation of μCV _CV in Equation 2 (Appendix E). On KITTI, we additionally demonstrate component wise ablation of our approach, isolating surrogate, proxy contribution, MI and clustering (Table 3). MI-only (MI) yields high coverage but low failure discovery; no clustering (Greedy) reduces coverage due to lesser intra-batch diversity and suffers from mode collapse; no MI with clustering (−-MI-C) under-performs on high-severity failure discovery and coverage; and no MI and no clustering (−-MI-G) further reduces coverage. Thus, ablations show that MI promotes support coverage, CV improves failure targeting, and clustering preserves within-batch diversity. Table 1: Positive Samples for different severity levels for nuPlan (mean ± std across four seeds). Task γ Random BNN-C BAMS BNN-CV Our nuPlan 0.3 0.23 ± 0.01 0.25 ± 0.01 0.28 ± 0.01 0.27 ± 0.01 0.33 ± 0.01 0.2 0.22 ± 0.00 0.24 ± 0.02 0.27 ± 0.01 0.26 ± 0.01 0.32 ± 0.01 0.05 0.21 ± 0.01 0.21 ± 0.02 0.25 ± 0.01 0.22 ± 0.02 0.29 ± 0.01 5.1 nuPlan Figure 2: MI driven exploration for different forms of data acquisition on nuPlan task. 2D T-SNE embeddings of scenario dataset showing high and low MI regions for 200 datapoints sampled from (1) small concentrated region of low failures, (2) MC sampling, (3) data recovered from a specific type of scenarios. The plots show that our support aware MI strategy successfully identifies unobserved regions agnostic to the choice of sampling strategy. Figure 3 shows that our approach achieves lower cumulative average metric than all baselines except BNN-CV, which repeatedly samples the same failures, hurting both diversity and failure count. Our approach outperforms all baselines on both cumulative coverage and number of failures discovered, where ℱF is defined by closed-loop TTC <0.3<0.3. In practice, we often lack control over how the initial scenario set is collected. Figure 2 visualizes MI values across the full nuPlan scenario embedding space, showing that our support-aware MI consistently prioritizes unexplored regions regardless of how prior samples were collected, providing a principled measure of scenario diversity. Figure 3: Baseline comparison for nuPlan task, (avg ± std across four seeds) for B=170B=170 samples, with 2020 initial random samples for training surrogate, and 150 samples acquired in batches of b=5b=5. 5.2 Quadruped Figure 4 compares our approach against BAMS and Random on the Quadruped task, where failures are defined as scenarios x∈ℝ3x ^3 with average velocity tracking error exceeding 0.70.7 (e.g., early stopping, large trajectory deviation; see supplementary video). Our method consistently discovers more diverse failures than both baselines (Figure 8), and identifies more failures across severity levels for γ=0.7γ=0.7 (Table 2). In contrast, BAMS concentrates near the threshold and often misses high-severity failures. We observe that diverse severity coverage also improves surrogate prediction. Our method achieves mean accuracy of 0.800.80 over validation failure sets at γ∈0.7,0.8,0.9,1.0γ∈\0.7,0.8,0.9,1.0\, compared to 0.700.70 for BAMS, with 0.680.68 accuracy on high-severity scenarios (yr≥1.0y_r≥ 1.0). Figure 9 shows that our surrogate captures two symmetric failure modes over (vy,wz)(v_y,w_z), consistent with the task structure, confirming that our coverage-aware acquisition actively identifies distinct failures. Figure 4: Baseline comparison for Quadruped task, (avg ± std, two seeds) for B=30B=30 samples, with 1010 initial random samples for training surrogate, and 20 samples acquired in batches of b=5b=5. Table 2: Positive samples for different γ values in Quadruped task. γ 0.7 (Our) 0.8 (Our) 1.0 (Our) 0.7 (BAMS) 0.8 (BAMS) 1.0 (BAMS) P.S. 0.6 ± 0.1 0.3 ± 0.1 0.2 ± 0.1 0.5 ± 0.1 0.3 ± 0.2 0.0 ± 0.0 Table 3: KITTI component ablation Full Greedy β=0(n+k)β=0(n+k) β=0β=0 MI −-MI-C −-MI-G Utility 0.38 0.38 0.36 0.36 0.32 0.4 0.41 P.S.-0.5 0.24 0.22 0.14 0.17 0.11 0.30 0.32 P.S.-0.7 0.03 0.01 0.02 0.01 0.02 0.02 0.01 Cov. 0.78 0.58 0.79 0.72 0.91 0.6 0.48 5.3 SIMPLER Figure 5: Examples of grasping failures for SIMPLER task. Top- scenarios for which both proxy and target systems succeed (left) and fail (right) to grasp the can. Bottom- two examples for which the proxy system succeeds and target system fails to grasp the can, discovered by our approach. Figure 10 shows baseline comparison, with BNN-CV performing comparable to Random sampling. This is due to the fact that the failure metric of this task is sparsely defined as success (yr=0.0y_r=0.0) or failure (ys=1.0y_s=1.0), so the lack of exploration leads to heavy dependence on the failures present in initial dataset for accurately learning the surrogate model. From the collected failure scenarios, we observe that perceptually challenging conditions with visual clutter are the dominant driver of failure all seeds. We discover two dominant modes for camera positioning to the right and left of reference positions, each corresponding to object placement to the left and far from the robot. The average proxy success rate is 20%20\% showing reasonable sim to real gap. Figure 5 (bottom) shows failure scenarios corresponding to this gap with success in proxy system, and failure in target system. 5.4 KITTI We use this task to highlight the merit of our approach with combined sampling against target only sampling, and quantify the contribution of proxy assisted evaluation. Specifically, we test whether proxy information improves acquisition over surrogate-only sampling, using β=0,(n+k)β=0,(n+k) (Table 3), where the surrogate receives all n+kn+k samples, and proxy and qθq_θ sampling have the same cost; therefore any gains reflect proxy information, not reduced MC error. Importantly, the proxy provides additional system information, not a cheaper substitute for MC samples from qθq_θ. At equal sample count, proxy-assisted acquisition improves failure discovery and utility, preserving coverage, achieving nearly 70%70\% higher coverage of meaningful failure modes, and increases surrogate failure recall rate from 0.180.18 to 0.490.49. We also compare with β=0β=0 (n surrogate samples). On nuPlan, weak proxy–surrogate correlation in failure regions (avg. ρ≈0.15ρ≈ 0.15) drives β→0β→ 0, yielding small gains over no proxy ablation. On KITTI, ρ≈0.5ρ≈ 0.5, therefore gains in P.S. and utility are clearer in Table 3. It can be shown that proxy information can reduce target prediction risk by capturing target variation hidden by X, whereas additional qθq_θ samples only reduce variance. This benefit increases with proxy-target correlation. Thus, β-weighted proxy guidance exposes failure regions missed by surrogate-only sampling and improves the learned target surrogate. 6 Discussion & Conclusion Our approach scales gracefully with evaluation budget and scenario complexity, validating our central hypothesis. Baseline comparisons underscore the importance of each component. BAMS degrades at 170170 evaluations in nuPlan as GP surrogates struggle to scale; a known limitation of GPs noted in [18, 24, 21]. BNN-CV’s myopic acquisition collapses onto a narrow failure region in SIMPLER, where the sparse failure metric demands exploration, and GP-C’s assumption that all proxy failures transfer to the target causes it to miss unpaired failures, despite focusing on diversity of scenarios. Our work shows that proxy corrected risk estimation and scenario diversity are intertwined. Lack of diversity leads to redundant failures, and without accurate risk estimates, evaluations are wasted. Coupling both components is essential for sample efficient discovery of distinct, severe target-system failures. 7 Limitations Our approach assumes a scenario representation in which locality, support, and diversity are meaningful. While such representations are available in our experiments, performance may depend on how well they capture behaviorally relevant differences between scenarios. Like other black-box failure discovery methods, our approach also requires target-system evaluations; larger or more complex scenario spaces may require larger budgets to learn accurate target-risk surrogates. Our method is therefore best viewed as a way to use limited target evaluations more effectively by combining proxy guidance, local correction, and diversity-aware acquisition, with future work exploring learned representations and stronger transfer across related systems. References [1] A. Anwar, R. Gupta, Z. Merchant, S. Ghosh, W. Neiswanger, and J. Thomason (2025) Efficient evaluation of multi-task robot policies with active experiment selection. arXiv preprint arXiv:2502.09829. Cited by: §2. [2] A. Badithela, D. Snyder, L. Zha, J. Mikhail, M. O’Kelly, A. Dixit, and A. Majumdar (2025) Reliable and scalable robot policy evaluation with imperfect simulators. arXiv preprint arXiv:2510.04354. Cited by: §1, §2. [3] Y. Cabon, N. Murray, and M. Humenberger (2020) Virtual KITTI 2. arXiv preprint arXiv:2001.10773. Cited by: §5. [4] X. Cao and I. W. Tsang (2021) Bayesian active learning by disagreements: a geometric perspective. arXiv preprint arXiv:2105.02543. Cited by: §2, §3. [5] K. Chaloner and I. Verdinelli (1995) Bayesian experimental design: A review. Statistical science, p. 273–304. Cited by: §2, §3. [6] O. X. Collaboration, A. O’Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang, A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna, A. Wahid, B. Burgess-Limerick, B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, E. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Frujeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang, G. Wang, H. Su, H. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj, H. Walke, H. Fang, H. Ha, I. Mordatch, I. Radosavovic, I. Leal, J. Liang, J. Abou-Chakra, J. Kim, J. Drake, J. Peters, J. Schneider, J. Hsu, J. Vakil, J. Bohg, J. Bingham, J. Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, J. Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang, J. Salvador, J. J. Lim, J. Han, K. Wang, K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan, K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K. Zeng, K. Hatch, K. Hsu, L. Itti, L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. ". Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. C. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, M. Z. Irshad, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. ". Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart’in-Mart’in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, S. Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, S. Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, S. Tian, S. Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, V. Guizilini, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y. Wu, Y. Tang, Y. Zhu, Y. Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Y. Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin (2023) Open X-Embodiment: robotic learning datasets and RT-X models. Note: https://arxiv.org/abs/2310.08864 Cited by: §5. [7] A. Corso, P. Du, K. Driggs-Campbell, and M. J. Kochenderfer (2019) Adaptive stress testing with reward augmentation for autonomous vehicle validatio. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), p. 163–168. Cited by: §1. [8] A. Corso, R. Lee, and M. J. Kochenderfer (2020) Scalable autonomous vehicle safety validation through dynamic programming and scene decomposition. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), p. 1–6. Cited by: §1. [9] C. Dawson and C. Fan (2022) Robust counterexample-guided optimization for planning from differentiable temporal logic. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), p. 7205–7212. Cited by: §1, §2. [10] C. Dawson and C. Fan (2023) A Bayesian approach to breaking things: efficiently predicting and repairing failure modes via sampling. In 7th Annual Conference on Robot Learning, External Links: Link Cited by: §1, §2. [11] T. Dreossi, T. Dang, A. Donzé, J. Kapinski, X. Jin, and J. V. Deshmukh (2015) Efficient guiding strategies for testing of temporal properties of hybrid systems. In NASA Formal Methods: 7th International Symposium, NFM 2015, Pasadena, CA, USA, April 27-29, 2015, Proceedings 7, p. 127–142. Cited by: §1. [12] J. M. Esposito, J. Kim, and V. Kumar (2005) Adaptive RRTs for validating hybrid robotic control systems. In Algorithmic foundations of robotics vi, p. 107–121. Cited by: §1. [13] A. Geiger, P. Lenz, and R. Urtasun (2012) Are we ready for autonomous driving? The KITTI vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 3354–3361. Cited by: §5. [14] N. Karnchanachari, D. Geromichalos, K. S. Tan, N. Li, C. Eriksen, S. Yaghoubi, N. Mehdipour, G. Bernasconi, W. K. Fong, Y. Guo, et al. (2024) Towards learning-based planning: the nuplan benchmark for real-world autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), p. 629–636. Cited by: §5. [15] X. Li, K. Hsu, J. Gu, K. Pertsch, O. Mees, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao (2024) Evaluating real-world robot manipulation policies in simulation. arXiv preprint arXiv:2405.05941. Cited by: §5. [16] Y. Li, A. Parashar, E. Zhou, and C. Fan (2026) Pragmatic curiosity: a hybrid learning-optimization paradigm via active inference. arXiv preprint arXiv:2602.06104. Cited by: Appendix B. [17] R. Luo, H. Yang, M. Watson, A. Sharma, S. Veer, E. Schmerling, and M. Pavone Leveraging correlation across test platforms for variance-reduced metric estimation. In 9th Annual Conference on Robot Learning, Cited by: Appendix A, Appendix C, §1, §1, §2, §4.1. [18] A. Parashar, K. Garg, J. Zhang, and C. Fan Failure prediction from few expert demonstrations. In NeurIPS 2024 Workshop on Bayesian Decision-making and Uncertainty, Cited by: §1, §1, §2, §2, §5, §6. [19] A. Parashar, Y. Li, E. Y. Yu, F. Chen, J. Neidhoefer, D. Upadhyay, and C. Fan (2026) SEED-set: scalable evolving experimental design for system-level ethical testing. arXiv preprint arXiv:2603.01630. Cited by: §2, §3. [20] A. Parashar, J. Yin, C. Dawson, P. Tsiotras, and C. Fan (2024) Learning-based bayesian inference for testing of autonomous systems. IEEE Robotics and Automation Letters (), p. 1–8. External Links: Document Cited by: §1, §2. [21] A. Parashar, J. Zhang, Y. Li, and C. Fan (2025) Cost-aware discovery of contextual failures using bayesian active learning. In 9th Annual Conference on Robot Learning, Cited by: §2, §5, §6. [22] T. Rainforth, A. Foster, D. R. Ivanova, and F. Bickford Smith (2024) Modern bayesian experimental design. Statistical Science 39 (1), p. 100–114. Cited by: §2, §3, §3. [23] A. Z. Ren, H. Dai, B. Burchfiel, and A. Majumdar (2023) AdaptSim: task-driven simulation adaptation for sim-to-real transfer. In Conference on Robot Learning, p. 3434–3452. Cited by: §1, §2. [24] A. Sinha, P. Nikdel, S. Paul, and S. Whiteson (2025) Rate-informed discovery via bayesian adaptive multifidelity sampling. In Conference on Robot Learning, p. 2579–2598. Cited by: §1, §1, §2, §4.2, §5, §5, §6. [25] A. Sinha, M. O’Kelly, R. Tedrake, and J. C. Duchi (2020) Neural bridge sampling for evaluating safety-critical autonomous systems. Advances in Neural Information Processing Systems 33, p. 6402–6416. Cited by: §1. [26] Unitree Robotics (2026) Unitree Go2 Pro Quadruped Robot. Note: https://w.unitree.com/go2Quadruped robot platform. Accessed: 2026-05-28 Cited by: §5. [27] Unitree Robotics (2026) Unitree RL Mjlab. Note: https://github.com/unitreerobotics/unitree_rl_mjlabReinforcement learning implementation for Unitree robots based on MuJoCo. Accessed: 2026-05-28 Cited by: §5. Contents Figure 6: Overview of approach. Our method uses support-aware Mutual Information (Equation 3) to subsample KMIK_MI scenarios x, which are evaluated on the cheap proxy system to get metrics ysy_s, and a surrogate model qθq_θ trained using data evaluated on target system. We subsequently optimize μCV(x) _CV(x) (Equation 2)over these sub sampled candidates to select the next batch of evaluation points as likely failure scenarios. The process is repeated until evaluation budget B is exhausted. Figure on the left shows the surrogate model learnt after 40 samples of evaluation for the target system shown. Appendix A Local minimum-variance property of the control-variate estimator We show that the estimator in Equation 2 is the minimum-variance estimator within the class of local affine control-variate estimators. Specifically, our method does not reduce surrogate bias, but uses proxy system information in a meaningful way to accelerate efficiency of failure discovery. Our proof follows proof of Theorem 1 from [17]. Fix a candidate scenario x∈x , and let Bx⊆B_x denote the local neighborhood used to estimate the target failure statistic around x. Let Yr∼p(yr∣x′)Y_r p(y_r x ), Ys∼p(ys∣x′)Y_s p(y_s x ), and x′∼Bxx B_x. In practice, the target samples YrY_r are replaced by surrogate predictions Y^r∼qθ(x′) Y_r q_θ(x ), so the following result is interpreted conditional on the learned surrogate qθq_θ. Suppose we have n paired local samples (y^ri,ysi)i=1n,\( y_r^i,y_s^i)\_i=1^n, (4) and k additional proxy-only samples ysjj=n+1n+k.\y_s^j\_j=n+1^n+k. (5) Consider the class of local affine control-variate estimators μCV(β)=μCV(x)=1n∑i=1n(y^ri−ysi)+1k∑j=1kβysi _CV(β)= _CV(x)= 1n _i=1^n( y_r^i- βy_s^i)+ 1k _j=1^kβ y_s^i (6) Assuming that the paired and proxy-only samples are drawn independently from the same local neighborhood distribution over BxB_x, this estimator is unbiased for the local target statistic: Bx[μ^CV(β)] _B_x [ μ_CV(β) ] =Bx[Yr−β⊤Ys]+Bx[β⊤Ys] =E_B_x [Y_r-β Y_s ]+E_B_x [β Y_s ] (7) =Bx[Yr]. =E_B_x[Y_r]. (8) Thus, the choice of β affects the variance but not the expectation. For a fixed β, independence of the paired and proxy-only sample sets gives Var(μ^CV(β))=1nVar(Yr−β⊤Ys)+1kVar(β⊤Ys).Var ( μ_CV(β) )= 1nVar (Y_r-β Y_s )+ 1kVar (β Y_s ). (9) Expanding the two variance terms, we obtain Var(μCV(β)) ( _CV(β) ) =1nVar(Yr)−2nβ⊤Cov(Ys,Yr)+1nβ⊤Var(Ys)β+1kβ⊤Var(Ys)β = 1nVar(Y_r)- 2nβ Cov(Y_s,Y_r)+ 1nβ Var(Y_s)β+ 1kβ Var(Y_s)β (10) =1nVar(Yr)−2nβ⊤Cov(Ys,Yr)+n+knkβ⊤Var(Ys)β. = 1nVar(Y_r)- 2nβ Cov(Y_s,Y_r)+ n+knkβ Var(Y_s)β. (11) This is a convex quadratic function of β whenever Var(Ys)Var(Y_s) is positive semidefinite, and is strictly convex when Var(Ys)Var(Y_s) is positive definite. Differentiating Equation 11 with respect to β and setting the derivative equal to zero gives −2nCov(Ys,Yr)+2(n+k)nkVar(Ys)β=0.- 2nCov(Y_s,Y_r)+ 2(n+k)nkVar(Y_s)β=0. (12) Therefore, the variance-minimizing coefficient is β⋆=kn+kVar(Ys)−1Cov(Ys,Yr).β = kn+kVar(Y_s)^-1Cov(Y_s,Y_r). (13) Substituting β⋆β into Equation 6 gives the local control-variate estimator used in . Hence, μCV _CV is the minimum-variance estimator among all estimators of the affine local control-variate form in Equation 6. The corresponding optimal variance is Var(μCV(β⋆))=1n(Var(Yr)−kn+kCov(Yr,Ys)Var(Ys)−1Cov(Yr,Ys)).Var ( _CV(β ) )= 1n (Var(Y_r)- kn+kCov(Y_r,Y_s)Var(Y_s)^-1Cov(Y_r,Y_s) ). (14) In the scalar case, this can be written as Var(μCV(β⋆))=Var(F)n(1−kn+kρ2(Yr,Ys)),Var ( _CV(β ) )= Var(F)n (1- kn+kρ^2(Y_r,Y_s) ), (15) where ρ(Yr,Ys)ρ(Y_r,Y_s) is the local Pearson correlation between the target and proxy statistics. Since ρ2(Yr,Ys)∈[0,1]ρ^2(Y_r,Y_s)∈[0,1], we have Var(μ^CV(β⋆))≤Var(Yr)n,Var ( μ_CV(β ) )≤ Var(Y_r)n, (16) which is the variance of the direct local Monte Carlo estimator using only the n surrogate induced-statistic samples over YrY_r. Therefore, whenever the proxy statistic is locally correlated with the surrogate induced statistic, the control-variate estimator strictly reduces variance. When the local correlation is zero, it matches the target-only estimator in variance. Applying this result to our setting, YrY_r corresponds to the local target failure statistic induced by the surrogate qθq_θ, and YsY_s corresponds to the local proxy statistic induced by p(ys∣x)p(y_s x). Thus, under the local sampling assumptions above and conditional on the learned surrogate, the coefficient in Equation 13 yields the minimum-variance local affine estimator of the target failure statistic. This justifies using μCV _CV as a proxy-corrected failure predictor for selecting scenarios likely to satisfy μr(x)≥γ _r(x)≥γ. Note: For MC sampled data x′∼Bxx B_x, the resulting y^r∼qθ(x) y_r q_θ(x) may give biased results in practice as the data used to train qθ(x)q_θ(x) is not unbiased, and acquired through active learning, hence this justification is only valid for the case when surrogate induced statistic is an unbiased representation of the target statistic. To bridge this practical gap, we focus on selecting right hyperparameter for the training and architecture of qθq_θ. Newer training paradigms that calibrate the loss of training qθq_θ, such that the resulting estimator is equivalent to an estimator trained on unbiased data will be investigated as a part of our future work. Appendix B Motivation for MI-adjusted data acquisition We combine MI with failure discovery as two distinct objectives in data acquisition. This setup emerges naturally under active inference based data acquisition proposed by [16]. In Appendix D, we discuss the construction of various terms that appear in Section 4.2 in detail. Appendix C Defining locality in the feature space This section discusses the details of construction of local neighborhood region ℬxB_x in detail for the case where X is a continuous space. The neighborhood ℬxB_x can be defined in multiple ways, and controls the granularity of μCV(x) _CV(x). For example, consider a continuous region, with the neighborhood ℬxB_x defined using a Euclidean ball with radius R as ℬx=x′|∥x−x′∥22≤RB_x=\x | x-x ^2_2≤ R\. Consider, R∈[Rmin,Rmax]R∈[R_min,R_max], such that for R=RmaxR=R_max, ℬx≡B_x . The corresponding μCV(x) _CV(x) is equivalent to a global average CV estimator, represented in [17]. On the other hand, prediction for R→0R→ 0 collapses to a point-wise local mean prediction. For cases where we cannot manually specify the region ℬxB_x, we propose a sensitivity aware heuristic for adaptively adjusting R∈[Rmin,Rmax]R∈[R_min,R_max], such that if the local sensitivity of prediction is high, R should be smaller and vice versa. We define a sensitivity term s(x)s(x) that measures sensitivity of prediction using local gradient information and variance of the surrogate model at x: s(x)=‖∇xμq(x)‖(σq(x)+ϵ),s~(x)=s(x)−minx′∈s(x′)maxx′∈s(x′)−minx′∈s(x′)+ϵ.s(x)= \| _x _q(x) \| ( _q(x)+ε ), s(x)= s(x)- _x s(x ) _x s(x )- _x s(x )+ε. (17) Here s~(x)∈[0,1] s(x)∈[0,1] is a normalized version of s(x)s(x) for practical application, and σq2(x)=Varqθ(x)[yr] _q^2(x)=Var_q_θ(x)[y_r]. We construct our heuristic as: Ra(x)=Rmin+(Rmax−Rmin)1+exp(s~(x)).R_a(x)=R_ + (R_ -R_ )1+ ( s(x) ). (18) Figure 7 shows μCV _CV as a function of x for a 1D toy example, with local region ℬxB_x chosen using fixed radius values and adaptive Ra(x)R_a(x) chosen using Equation 18. We observe that the prediction with RaR_a most accurately predicts the ground-truth target mean as opposed to fixed region alternatives. Figure 7: 1D synthetic task. Left-right: (1) Randomly sampled 10 scenarios and observed values of synthetic target, shown against ground truth target and proxy system. (2) Scenarios identified as high MI due to being outside support/novel candidates (Section 4.2). (3) μ^CV(x) μ_CV(x) estimated using different fixed local regions and adaptive local region selection plotted against ground truth target. Adaptively selecting local region shows minimum deviation from ground truth. Appendix D Construction of terms for MI estimation To estimate MI, we need to estimate entropy H, for which, we model the distribution over RxR_x as p(Rx|x,t)=∑k=1Kp(Rx|Zx=k)p(Zx=k|x,t)p(R_x|x,D_t)= _k=1^Kp(R_x|Z_x=k)p(Z_x=k|x,D_t). Here, prior p(Zx=k|t,x)p(Z_x=k|D_t,x) controls probability of assignment of a point x to a cluster, and can be either observed directly for probabilistic clustering approaches such as GMMs, or can be manually designed, as shown below for K-means clustering used in this work. We define the likelihood p(Rx|Zx=k)p(R_x|Z_x=k) to be high for scenarios far from existing clusters as: p(Rx=1∣Zx=k)=qk(x),p(Rx=1∣Zx=new,x,Dt)=qnew(x),p(R_x=1 Z_x=k)=q_k(x), p(R_x=1 Z_x=new,x,D_t)=q_new(x), (19) Here 0≤qk(x)≤εexist0≤ q_k(x)≤ _exist is a distance weighted distribution that is 00 close to an existing cluster, and is capped at a small value εexist _exist far away. On the other hand, qnew(x)q_new(x) is defined as: qnew(x)=σ(dmin(x)−ργ),σ(z)=11+e−z,q_new(x)=σ ( d_ (x)-ργ ), σ(z)= 11+e^-z, (20) where dmin(x)d_ (x) is the distance from x to the nearest observed cluster boundary, γ is a softness parameter and ρ is a novelty radius which controls the minimum distance threshold for a scenario x to be considered novel. Equation 19 and Equation 20 specify a model for geometric distance based support discovery such that p(Rx=1)p(R_x=1) has high probability mass around novel scenarios, leading to the discovery of new clusters. We use data adaptive approaches in practice to estimate γ,ργ,ρ and K to mitigate hyperparameter dependency. K is estimated from a range of [Kmin,Kmax][K_min,K_max] using the elbow heuristic. We derive γ,ργ,ρ directly from the geometry of the current training support tD_t. Specifically, we compute the nearest-neighbour distance for each support point xi∈tx_i _t as dnn,i=minj≠i‖xi−xj‖,d_n,i= _j≠ i\|x_i-x_j\|, (21) and set ρ=median(dnn,ii=1M),γ=std(dnn,ii=1M),ρ=median\! (\d_n,i\_i=1^M ), γ=std\! (\d_n,i\_i=1^M ), (22) Defining probability p(Zx∣x,t)p(Z_x x,D_t) Given cluster centers ckk=1K\c_k\_k=1^K with per-cluster RMS spread lk=1|k|∑i∈k‖xi−ck‖2l_k= 1|k| _i∈ k\|x_i-c_k\|^2, the prior over support state ZxZ_x at x is defined using a distance based kernel wkw_k: wk(x)=exp(−‖x−ck‖22lk2),W(x)=∑k=1Kwk(x)w_k(x)= \! (- \|x-c_k\|^22\,l_k^2 ), W(x)= _k=1^Kw_k(x) (23) p(Z=zk∣x)=wk(x)W(x)+1,p(Z=znew∣x)=1W(x)+1p(Z=z_k x)= w_k(x)W(x)+1, p(Z=z_new x)= 1W(x)+1 (24) Figure 6 shows the reconstruction of failure regions using a GP surrogate model by performing data acquisition using our approach on a tor 2D synthetic task. The task consists of two low prob failures shown in yellow, which we aim to be discover in B=40B=40 samples. Figure 6 shows that our overall approach succesfully applies to uncover both regions, showing that our MI strategy helps in searching for diverse failures, and μCV _CV helps to bridge sim and real gap. Appendix E nuPlan ablations E.1 Ablation results for size of initial dataset We conduct ablation for initial size of sampling with N0=5,10,20,50N_0=5,10,20,50 initial data points for surrogate model training for our method, results for B=100B=100 samples reported in Table 4. Clearly, for N0=50N_0=50 we observe lowest cumulative TTC and highest diversity as well as failure count, however, we observe that the difference between N0=5,10,20N_0=5,10,20 is marginal, which tells us that in the lack of sufficient initial data, our method’s performance is not substantially affected. For another 20 samples, with B=140B=140, the run initialized with N0=20N_0=20 reports 1.05,0.81.05,0.8 and 0.330.33 for TTC average, coverage and positive samples, indicating that the lack of sufficient initial samples in high dimensional scenarios can be compensated for by increasing the testing budget and collecting more sequential data. Table 4: Ablation over different initialization sizes on the nuPlan task, 100 samples, four seeds, avg ± std reported. Setting Avg. Cumulative TTC Cumulative Coverage Positive Samples ninit=5n_init=5 1.09±0.061.09± 0.06 0.58±0.040.58± 0.04 0.27±0.020.27± 0.02 ninit=10n_init=10 1.16±0.131.16± 0.13 0.48±0.090.48± 0.09 0.25±0.010.25± 0.01 ninit=20n_init=20 1.06±0.041.06± 0.04 0.75±0.020.75± 0.02 0.26±0.030.26± 0.03 ninit=50n_init=50 0.99±0.060.99± 0.06 1.10±0.131.10± 0.13 0.30±0.050.30± 0.05 E.2 Ablation results for μCV _CV Table 5 shows results for ablation over Equation 2 with β=0β=0, to remove the dependency on sim samples. We observe that with β=0β=0, there is marginal improvement in positive samples and cumulative TTC, however, this comes at the cost of coverage, which is especially compromised for samples with TTC =0.0=0.0, i.e., severe failure scenarios. This is due to the fact that while proxy scenarios, in this case constructed from open loop simulation, are a noisy estimate of the target system, having larger sampling capacity on proxy helps identify more regions where the target system might be failing, which may be missed in limited budget target only estimation. Table 5: Ablation comparing our approach with and without sim samples by setting β=0β=0 on the nuPlan task. Results reported for 20 initial samples, B=150B=150, avg and std for four seeds. Setting Avg. Cumulative TTC Cumulative Coverage Positive Samples γ=0.3γ=0.3 nominal 1.04±0.03321.04± 0.0332 0.98±0.060.98± 0.06 0.32±0.010.32± 0.01 β=0β=0 1.03±0.051.03± 0.05 0.95±0.050.95± 0.05 0.35±0.010.35± 0.01 γ=0.0γ=0.0 nominal 1.04±0.031.04± 0.03 0.87±0.050.87± 0.05 0.29±0.010.29± 0.01 β=0β=0 1.03±0.051.03± 0.05 0.81±0.080.81± 0.08 0.31±0.010.31± 0.01 Appendix F Quadruped Additional results for Quadruped, showing scatterplot of our method and BAMS in Figure 8, and Figure 9 showing heatmap of learnt failure versus success region, providing qualitative evidence that our method discovers more diverse failures. We also provide video demos of two failure and one non-fail command velocity tracking tasks in the supplementary video attached, where the scenarios are selected as those predicted fail and non fail by our method. Figure 8: Scatterplot of failure scenarios x=(vx,vy,wz)x=(v_x,v_y,w_z) (y≥0.7y≥ 0.7) for Quadruped task, generated by our method (red), and BAMS (blue). Our method shows higher diversity of scenarios unlike BAMS that tends to concentrate around failures discovered in early iterations. Figure 9: Heatmaps showing learnt surrogate model predictions projected over scenario space for Quadruped task. Projection value corresponds to median of third variable reported in parentheses. Scattered points correspond to initial 10 and subsequent 20 scenarios collected by our approach. Appendix G SIMPLER Baseline comparison for SIMPLER shown in Figure 10. We also show videos of fail and non fail tasks, where the successful task is provided as a reference, and fail tasks are sampled by our approach consisting of one task where both proxy and target system fail, and one where only target system fails, showing that our approach can discover failures unseen in simulation as well. Figure 10: Baseline comparison for SIMPLER task, showing avg ± std across three seeds for B=70B=70 samples, with 1010 initial random samples for training all surrogate models, and 60 additional samples acquired in batches of b=3b=3. Appendix H Visualization In Figure 12 we show additional examples for SIMPLER task. We also show videos of failures for Quadruped task, which correspond to high velocity in vxv_x and vyv_y, as tracking larger requires crossing the obstacles used to construct a boundary, leading to obstacle avoidance getting activated and velocity becoming zero. Figure 11: Failures for Quadruped task. Quadrupeds shown at the end of tracking duration, failure scenarios correspond to Quadruped reaching near one of the boxed and stopping due to obstacle avoidance. Figure 12: More explains of failure scenarios uncovered by our approach in the SIMPLER task. Appendix I Experiment details I.1 Model specifications We choose randomly sampled data to initialize the various surrogate models for each setup. For nuPlan and SIMPLER, we use BNN surrogate models for our method, which consist of a MLP with MC dropouts at each layer (p(dropout)=0.1p(dropout)=0.1), and for Quadruped, we use MLP, which is suitable choice due to low observed stochasticity in the velocity tracking policy. For each setup and across all baselines that use a N based architecture, we train using Adam optimizer with lr=1e-3, with 1000, 800 and 3000 epochs for nuPlan, SIMPLER and Quadruped tasks respectively. These hyperparameter are selected based on training accuracy over the initial random sampling dataset. Table 6 summarizes the model architectures for all tasks. nuPlan has inherent over-parameterization in scenario due to encoding so we benefit from a compressive architecture whereas SIMPLER benefits from over-parameterization. Table 6: Summary of neural network architectures across tasks. Task Architecture Input Hidden layers Activation nuPlan surrogate MLP 384 96→24→696→ 24→ 6 ELU + LN nuPlan BNN-C head + sigmoid — — σ SIMPLER BNN MLP 19 96→24→696→ 24→ 6 Tanh SIMPLER classifier for GP-C MLP + sigmoid 19 96→24→696→ 24→ 6 σ Quadruped MLP MLP 3 8→88→ 8 ReLU I.2 Data specifications For nuPlan, proxy dataset is directly available and used as it is, whereas SIMPLER and Quadruped are setups where querying proxy data can be costly as well. Hence, we use a BNN and MLP for constructing a surrogate model for proxy systems for SIMPLER and Quadruped, using larger initial datasets of 50 and 2000 datapoints collected offline. For SIMPLER, the initial dataset for proxy surrogate model is quite less, hence, at each step of acquisition for target system, we also update proxy system with 1 datapoint. BAMS expects a cost of collecting proxy and target data as an input, which is calibrated to match the frequency of data collection used by our approach. For BNN-C in nuPlan, we use the same architecture as ours. Note that the baseline is originally supposed to work with GPs, which can be incompatible for large size of nuPlan, hence we use a BNN there. I.3 Scenario design and failure metrics for each task For nuPlan, scenarios are driving logs from nuPlan database, which are encoded into 384-dimensional vectors, we measure TTC as failure metric generated by nuPlan, and proxy and target system correspond to open loop and closed loop simulations respectively. For SIMPLER task, we perturb visual specifications and object placements, generating a 19 dimensional scenario specification. The target object (a Coke can) is placed at a 2D tabletop position xobj∈[−0.5,−0.1]x_obj∈[-0.5,-0.1] m and yobj∈[0.0,0.4]y_obj∈[0.0,0.4] m, spanning the robot’s reachable workspace. Camera viewpoint variation is introduced via lateral translations Δxcam,Δycam∈[−0.025,0.025] x_cam,\, y_cam∈[-0.025,0.025] m relative to the nominal camera pose. Scene appearance is controlled by brightness b∈[0.70,1.10]b∈[0.70,1.10] and contrast c∈[0.90,1.25]c∈[0.90,1.25], modelling illumination variation between the proxy and target domains. Finally, distractor objects are placed at positions xdist∈[−0.5,−0.1]x_dist∈[-0.5,-0.1] m, ydist∈[0.0,0.4]y_dist∈[0.0,0.4] m with uniformly sampled yaw θdist∈[−π,π] _dist∈[-π,π] rad, introducing clutter that occludes the target object and disrupts grasping. Across three task variants, pick_horizontal, pick_vertical, and pick_standing, and choosing distractor objects from a selection of 8 objects makes a 19 dimensional scenario representation. Failure here is recorded as failure to grasp or lift the coke can, and is generated by the environment. Target and proxy systems correspond to slight difference in visual specifications, and target system has extra visual noise acting as occlusion, leading to worse performance on several scenarios. For Quadruped, the scenario is 3-d and corresponds to command velocity vx,vy,wzv_x,v_y,w_z in the range [−0.8,1.0]×[−0.8,0.8]×[−0.8,0.8][-0.8,1.0]×[-0.8,0.8]×[-0.8,0.8] and proxy system consists of a simulation with RL policy for command velocity tracking, whereas target system consists of the actual hardware with Unitree sports mode used for command velocity tracking. We initialize the quadruped at the same location within a square boundary constructed using cardboard boxes. However, we observe that failure scenarios are agnostic to initialization provided they are initialized at one of the four corners.