Paper deep dive
RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
Daniel Yang, Samuel Stante, Florian Redhardt, Lena Libon, Parnian Kassraie, Ido Hakimi, Barna Pásztor, Andreas Krause
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 6:58:17 AM
Summary
The paper introduces RewardUQ, a unified framework for evaluating uncertainty quantification (UQ) in reward models used for aligning large language models (LLMs) with human preferences. It highlights that most existing methods overlook epistemic uncertainty, leading to issues like reward hacking. The authors propose a systematic evaluation comparing common UQ methods (ensembles, Bayesian inference, mean-variance estimation) using new metrics for accuracy and calibration, including a novel ranking score. They find that model size and initialization significantly impact performance and release an open-source Python package to facilitate future research.
Entities (10)
Relation Signals (9)
RewardUQ → evaluates → Uncertainty Quantification
confidence 95% · This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models.
Epistemic Uncertainty → arisesfrom → Limited Human Feedback
confidence 92% · most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback.
Initialization → impacts → Performance
confidence 91% · Our experimental results suggest that model size and initialization have the most meaningful impact on performance
Model Size → impacts → Performance
confidence 91% · Our experimental results suggest that model size and initialization have the most meaningful impact on performance
RewardUQ → compares → Ensembles
confidence 90% · We compare common methods along standard metrics measuring accuracy and calibration... Our work focuses on the most common approaches and covers a selection of ensemble and Bayesian inference methods
RewardUQ → compares → Bayesian Inference
confidence 90% · Our work focuses on the most common approaches and covers a selection of ensemble and Bayesian inference methods
Epistemic Uncertainty → mitigates → Reward Hacking
confidence 88% · Recent work suggests that quantifying this uncertainty can... mitigate reward overoptimization in LLM post-training.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2602.24040v1
- Canonical: https://arxiv.org/abs/2602.24040v1
Trouble viewing inline? Open PDF directly →
Full Text
75,829 characters extracted from source content.
Expand or collapse full text
RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models Daniel Yang1 Samuel Stante111footnotemark: 1 Florian Redhardt111footnotemark: 1 Lena Libon111footnotemark: 1 Parnian Kassraie1 Ido Hakimi1,2 Barna Pásztor1,2 Andreas Krause1,2 1 ETH Zurich 2 ETH AI Center Equal contributions. Correspondence to dayang,sstante,fredhardt,llibon@ethz.ch. Abstract Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq. 1 Introduction Reinforcement learning from human feedback (RLHF) is a key component for aligning large language models (LLMs) with human preferences to ensure they are safe and helpful (ziegler2019fine; ouyang2022training; bai2022training). The standard RLHF process first trains a reward model on a dataset of pairwise comparisons to learn the underlying preferences, and then uses this model to align the LLM policy with reinforcement learning (RL) algorithms (christiano2017deep). However, the success of RLHF heavily relies on the quality of the reward model. This poses several challenges, as collecting high-quality human preference data is expensive and reward models trained on limited and noisy datasets are imperfect (casper2023open). Additionally, aligning an LLM with such an imperfect reward model can lead to reward hacking, where the LLM overoptimizes flawed rewards rather than intended human preferences (eisenstein2024helping; coste2023reward; gao2023scaling; amodei2016concrete). Uncertainty quantification (UQ) for reward models emerged as a promising way to address these issues by explicitly modeling epistemic uncertainty arising from limited preference data. Recent work leverages uncertainty-aware reward models to mitigate reward hacking by penalizing (lou2025uncertaintyaware; zhai2024uncertaintypenalized; yan2024rewardrobust; houliston2024uncertaintypenalized; banerjee2024reliable; coste2023reward) or filtering (sun2024uncertainty; lou2025uncertaintyaware) uncertain samples. Reward uncertainty estimates are also leveraged to improve sample efficiency and reduce data collection costs through active learning in the reward modeling (melo2024deep; dwaracherla2024efficient; das2024active) or alignment step (mehta2025sample; muldrew2024active; liu2024sampleefficient; gleave2022uncertainty; liang2021reward; christiano2017deep). However, most studies adopt a single UQ method without systematic evaluation, leaving the impact of specific design choices largely unexplored. In this work, we present RewardUQ, a unified framework for the design and evaluation of uncertainty-aware reward models. This framework represents a first step towards principled UQ in preference modeling with reward functions, with the goal to build a foundation for more reliable and sample-efficient RLHF. Our main contributions are as follows: • We introduce a unified framework which formalizes the UQ problem, standardizes existing methods in a consistent notation, and defines a common evaluation procedure. Our evaluations utilize a new ranking strategy that incorporates the accuracy and calibration of predictions under uncertainty. • We conduct a systematic evaluation of existing uncertainty-aware reward models, analyzing how architectural choices and training parameters affect the quality of uncertainty estimates. • We release an open-source Python package as an accessible and extensible platform for developing, evaluating, and deploying new UQ methods. 2 Related work 2.1 Methods for reward model uncertainty quantification Ensembles The predominant approach for uncertainty quantification for reward models in RLHF are ensembles, with the uncertainty represented by the variance across ensemble members. In its simplest form, an ensemble combines multiple reward models trained with different random seeds (coste2023reward; eisenstein2024helping; liang2021reward; christiano2017deep) and, optionally, with bootstrapped datasets (lou2025uncertaintyaware; gleave2022uncertainty). To reduce computational cost, others utilize a pretrained model and train only lightweight ensemble members such as linear heads (banerjee2024reliable; yan2024rewardrobust), multi-layer perceptron (MLP) heads (melo2024deep; liu2024sampleefficient; dwaracherla2024efficient), low-rank adaptation (LoRA) adapters (zhai2024uncertaintypenalized; sun2024uncertainty; houliston2024uncertaintypenalized; yang2024bayesian; zhang2024improving), or apply Monte Carlo (MC) dropout (mehta2025sample; zhang2025dorm). Bayesian inference with Laplace approximation An alternative approach assumes a Gaussian prior on the parameters of a single reward model and derives the uncertainty from the predictive posterior based on the Laplace approximation (cercola2025efficient). As the Hessian is often intractable over all model parameters, the Laplace approximation is typically applied to a subset of the parameters of a pretrained LLM, such as the linear head (das2024active; cercola2025efficient) or a LoRA adapter (yang2024bayesian). Mean-variance estimation Some studies utilize reward models which predict the mean and variance of a Gaussian reward distribution, capturing the aleatoric uncertainty under heteroscedastic noise (yan2024rewardrobust; lou2025uncertaintyaware; siththaranjan2023distributional; sun2025probabilistic). Reward-margin-based preference uncertainty Others leverage the margin between pointwise rewards as a measure of uncertainty about the true preference, without modeling the uncertainty about the true reward (muldrew2024active; lou2025uncertaintyaware). Our work focuses on the most common approaches and covers a selection of ensemble and Bayesian inference methods, identifying commonalities and differences, and evaluating them side by side. 2.2 Applications for reward UQ Uncertainty-aware alignment Uncertainty estimates can make the alignment step in RLHF more resilient to reward overoptimization by encouraging the LLM to avoid uncertain rewards. Common schemes involve penalizing (lou2025uncertaintyaware; zhai2024uncertaintypenalized; houliston2024uncertaintypenalized; banerjee2024reliable; coste2023reward; sun2025probabilistic) or filtering (sun2024uncertainty; lou2025uncertaintyaware) uncertain rewards. Other approaches adopt pessimistic objectives to optimize for worst-case performance under uncertainty (zhang2024mitigating; yan2024rewardrobust) or apply pessimistic best-of-N sampling (liu2025uncertainty). Active learning for reward modeling To reduce the cost of collecting high-quality preference data, uncertainty estimates can guide the label acquisition towards more informative samples, improving the sample efficiency in the reward modeling step in RLHF (melo2024deep; dwaracherla2024efficient; das2024active). Others utilize uncertainty to estimate the quality of and adaptively assign weights to preference samples (zhang2025dorm). Active learning for alignment Similarly, uncertainty in the predicted rewards can improve the sample efficiency in the alignment step in RLHF, be it through uncertainty-based selection criteria of alignment samples (mehta2025sample; muldrew2024active; christiano2017deep; cercola2025efficient) or exploration bonuses (liu2024sampleefficient; liang2021reward). These directions highlight the promise of UQ methods for reward models. Yet, most studies adopt a single method and focus on downstream applications. Even work that compare multiple methods, such as the ensemble architecture study of zhang2024improving, limit their analysis to downstream performance rather than a systematic analysis of the uncertainty quantification itself. In contrast, our work follows a complementary direction by focusing on the design and evaluation of different UQ methods, aiming to provide a clear comparison and offer insights on how to choose and use methods. With most prior work initializing their reward models from generic pre-trained models, our results suggest that most works could have benefited from better design choices, especially by choosing model initializations that are tuned for reward modeling. 3 Uncertainty quantification for reward models We introduce a unified framework for designing and evaluating uncertainty-aware reward models, which integrates a range of existing approaches and extends them with novel contributions of our own. We begin by formalizing the UQ problem for reward models in Section˜3.1, and then introduce our evaluation metrics in Sections˜3.2 and 3.3. 3.1 Problem statement We consider the reinforcement learning from human feedback (RLHF) problem, which aims to align a language model π with human preferences, such that π is more likely to generate a human-preferred completion y∼π(⋅∣x)y π(· x) for a given prompt x (ouyang2022training; stiennon2020learning). We assume preferences to be expressed as pairwise comparisons in terms of (x,y+,y−)(x,y^+,y ) with y+y^+ being preferred over y−y , denoted y+≻y−y^+ y . As standard in the literature, we assume the Bradley-Terry preference model (bradley1952rank) that models the comparison between two candidate completions y and y′y as a Bernoulli distribution with probability p(y≻y′∣x,y,y′)=σ(r(x,y)−r(x,y′)) llasection eq:bt_modeleq:bt_modelp(y y x,y,y )=σ r(x,y)-r(x,y ) (1) where σ(x)=11+exp(−x)σ(x)= 11+ (-x) is the sigmoid function and r is a reward function assigning a scalar score to any prompt-completion pair. Given a dataset train=(xi,yi+,yi−)i=1nD_train=\(x_i,y^+_i,y _i)\_i=1^n, a reward model rθr_θ is trained by maximizing the likelihood of the observed preferences or equivalently by minimizing the binary cross-entropy loss ℒbase(θ;train)=1n∑(x,y+,y−)∈train−logσ(rθ(x,y+)−rθ(x,y−)). llasection eq:loss-baseeq:loss-baseL_base(θ;D_train)= 1n _(x,y^+,y ) _train- σ r_θ(x,y^+)-r_θ(x,y ) . (2) Once trained, rθr_θ can be used to align π via RL algorithms such as PPO (schulman2017proximal) or GRPO (shao2024deepseekmath), or at inference time with best-of-N sampling (stiennon2020learning; yang2024bayesian). However, the standard RLHF framework relies on reward models which only make pointwise predictions, thereby neglecting the epistemic uncertainty arising from training on a finite dataset sampled from the large domain of natural language. An uncertainty-aware reward model additionally predicts upper and lower confidence bounds rθ¯(x,y) r_θ(x,y) and rθ¯(x,y) r_θ(x,y), quantifying its epistemic uncertainty about the true underlying reward in terms of a confidence interval ℐrθ(x,y)=[rθ¯(x,y),rθ¯(x,y)]I_r_θ(x,y)= [ r_θ(x,y), r_θ(x,y) ]. We introduce the most common methods in detail in Section˜4. Under the Bradley-Terry model assumption, the corresponding upper and lower bounds on the preference probability are given by pθ¯(y≻y′∣x,y,y′) p_θ(y y x,y,y ) =σ(rθ¯(x,y)−rθ¯(x,y′)) =σ r_θ(x,y)- r_θ(x,y ) (3) pθ¯(y≻y′∣x,y,y′) p_θ(y y x,y,y ) =σ(rθ¯(x,y)−rθ¯(x,y′)), =σ r_θ(x,y)- r_θ(x,y ) , which are based on the largest and smallest plausible reward margin between both candidate completions, respectively (mehta2025sample). The goal of an uncertainty-aware reward model is to predict preference probabilities and confidence bounds, which not only accurately reflect the true binary preferences but are also statistically well-calibrated with respect to the true preference probabilities. We introduce our precise notion of accuracy and calibration, two complementary evaluation dimensions, along with their corresponding metrics in Sections˜3.2 and 3.3. We elaborate on the epistemic and aleatoric uncertainty decomposition for preference classification in Section˜A.1, and discuss the differences between standard and preference classification by focusing on the symmetry of Equation˜1 in Section˜A.2. 3.2 Accuracy metrics While accuracy is a standard performance measure for pointwise predictions, we further extend the notion of accuracy to confidence bounds. Accuracy of predictions Given an evaluation dataset evalD_eval and a reward model rθr_θ, the predicted rewards are correct if they assign higher rewards to preferred completions.111This is equivalent to pθ(y+≻y−∣x,y+,y−)>0.5p_θ(y^+ y x,y^+,y )>0.5 under the Bradley-Terry model in Equation 1. Let (rue) (rue) =(x,y+,y−)∈eval∣rθ(x,y+)>rθ(x,y−) =\(x,y^+,y ) _eval r_θ(x,y^+)>r_θ(x,y )\ (alse) (alse) =(x,y+,y−)∈eval∣rθ(x,y+)≤rθ(x,y−) =\(x,y^+,y ) _eval r_θ(x,y^+)≤ r_θ(x,y )\ be the set of true (i.e., correct) and false (i.e., incorrect) preference predictions. The accuracy, commonly known as win rate in the context of RLHF, is defined as winrate \ rate =||n. = n. ↑ (4) Accuracy of bounds While the win rate only captures the accuracy of pointwise predictions, we extend this notion of accuracy to confidence intervals. To quantify the accuracy of the predicted reward confidence intervals ℐrθ(x,y)I_r_θ(x,y), we further categorize the true and false predictions into (onfident) (onfident) =(x,y+,y−)∈eval∣ℐrθ(x,y+)∩ℐrθ(x,y−)=∅ =\(x,y^+,y ) _eval _r_θ(x,y^+) _r_θ(x,y )= \ (nconfident) (nconfident) =(x,y+,y−)∈eval∣ℐrθ(x,y+)∩ℐrθ(x,y−)≠∅. =\(x,y^+,y ) _eval _r_θ(x,y^+) _r_θ(x,y )≠ \. Intuitively, a prediction is confident when the predicted reward confidence intervals of the preferred and non-preferred completion do not overlap, indicating no ambiguity in the predicted preference even under uncertainty. By combining the correctness of the pointwise predictions with the confidence of the predicted bounds, we define the following metrics (confident) (unconfident) (5) (true) CTrate \ rate =|∩|n = n ↑ UTrate \ rate =|∩|n = n ↘ (false) CFrate \ rate =|∩|n = n ↓ UFrate \ rate =|∩|n. = n. ↘ We refer to Section˜A.3 for a generalization of these metrics to the standard binary classification setting. Ranking score In order to compare models, we propose a ranking score that combines the accuracy metrics above into a single score. Motivated by the UQ reward model applications, this score encourages a high confident true rate to efficiently guide active learning algorithms and identify reliable training data samples. Simultaneously, it penalizes the confident false rate that could provide misleading signals. The ranking score is defined as RSα=CTratewinrate+α⋅(1−winrate)−CFrate(1−winrate)+α⋅winrate=|∩|||+α⋅||−|∩|||+α⋅||∈[−1,1]↑ llasection eq:metrics-accuracy-rankingeq:metrics-accuracy-ranking aligned RS_α&= CT\ ratewin\ rate+α·(1-win\ rate)- CF\ rate(1-win\ rate)+α·win\ rate\\ &= +α· - +α· ∈[-1,1] aligned (6) using a trade-off parameter α∈[0,1]α∈[0,1], which balances the focus on the confidence and the focus on the accuracy. For α=0α=0, RS0RS_0 considers the relative rate of confidence among true and false predictions by normalizing the CTCT and CFCF rates and represents performance in the range [−1,1][-1,1]. In other words, high RS0RS_0 scores are achievable without achieving a high win rate. For α=1.0α=1.0, RS1RS_1 looks at the absolute difference between the confident true and false predictions and represents performance in the range [winrate−1,winrate][win\ rate-1,win\ rate]. This approach assigns equal weight to CTCT and CFCF and favors a high win rate over confidence. For our evaluations in Section˜5, we choose RS0.2RS_0.2 as a balance between accuracy and confidence. A more detailed explanation of the rationale and analysis of the inherent trade-offs are provided in Section˜A.4. 3.3 Calibration metrics Calibration refers to the gap between a predicted probability and the true (or empirical) probability. We give a formal description of calibration in Section˜A.5 while providing an overview here. Calibration of predictions The expected calibration error (ECE) is commonly used to measure the calibration of predicted preference probabilities (zhai2024uncertaintypenalized; gleave2022uncertainty). It is approximated based on grouping the predicted probabilities into M bins Bmm=1M\B_m\_m=1^M and computing ECE ≈∑m=1M|Bm|n|ℙ(Bm)−pθ(Bm)| ≈ _m=1^M B_m n (B_m)-p_θ(B_m) ↓ (7) with empirical probability ℙ(Bm)P(B_m) and average predicted probability pθ(Bm)p_θ(B_m) in each bin BmB_m (guo2017calibration; pavlovic2025understanding). Calibration of bounds We extend the notion of calibration to the predicted preference probability bounds constructed in Equation˜3 by introducing the expected lower calibration error (ELCE) and expected upper calibration error (EUCE). Analog to ECE, we group the lower and upper bounds separately into M bins Bmm=1M\B_m\_m=1^M and compute ELCE ≈∑m=1M|Bm|nmax(pθ¯(Bm)−ℙ(Bm),0) ≈ _m=1^M B_m n p_θ(B_m)-P(B_m),0 ↓ EUCE ≈∑m=1M|Bm|nmax(ℙ(Bm)−pθ¯(Bm),0) ≈ _m=1^M B_m n (B_m)- p_θ(B_m),0 ↓ with pθ¯(Bm) p_θ(B_m) and pθ¯(Bm) p_θ(B_m) denoting the average predicted lower and upper bounds in the corresponding bin BmB_m. ELCE penalizes lower bounds that overestimate the true preference probability, and EUCE penalizes upper bounds that underestimate the true preference probability. Note that preference probabilities are antisymmetric in their completions argument. Accordingly, all calibration metrics are computed on a symmetrized preference evaluation set that includes flipped comparisons with opposite labels, ensuring that both directions of each preference pair contribute to the binning-based approximation of the calibration errors, as further described in Section˜A.2. Therefore, the lower bound on the probability of y≻y′y y corresponds at the same time to an upper bound for the probability of y′≻y y. Hence, ELCE and EUCE are identical in the context of preference probabilities, and we subsequently only report the expected bound calibration error (EBCE) EBCE =ELCE=EUCE. =ELCE=EUCE. ↓ (8) 4 Uncertainty-aware reward models xxyyAgLM zzlastAgMLP head AgMLP head r1r_1rKr_K ⋮ ⋮ rruu (a) MLP head ensemble xxyy AgLM AgLoRA adapter AgLoRA adapter Aglinear head Aglinear head zzlastzzlastr1r_1rKr_K ⋮ ⋮ ⋮ rruu (b) LoRA adapter ensemble xxyyAgLM zzlastAglinear head w0w_0∼(μ0,Σ0) ( _0, _0)rruu (c) Bayesian linear head xxyyAgLM zzallAgdropoutAgdropoutAgLM head AgrDPOr_DPOr1r_1rKr_K ⋮ ⋮ rruu (d) DPO-based MC dropout Figure 1: Uncertainty-aware reward model architectures compared in this work. For a given prompt x and completion y, each model extracts an embedding z from a pretrained language model (LM) and predicts a reward r and uncertainty estimate u. Blue components indicate the parts responsible for estimating the uncertainty, while and denote trainable and frozen components, respectively. In this work, we focus on the most common uncertainty-aware reward model architectures from existing work. While these models differ in how they represent epistemic uncertainty, they share several core principles as illustrated on Figure˜1. Following prior work (li2022nearoptimal; ji2024reinforcement; mehta2025sample), for any prompt-completion pair, (x,y)(x,y), we separate pointwise prediction from uncertainty quantification, and assume that each model predicts a reward rθ(x,y)r_θ(x,y) and an uncertainty estimate uθ(x,y)u_θ(x,y), which are used to construct the symmetric confidence bounds rθ¯(x,y) r_θ(x,y) =rθ(x,y)+β⋅uθ(x,y) =r_θ(x,y)+β· u_θ(x,y) (9) rθ¯(x,y) r_θ(x,y) =rθ(x,y)−β⋅uθ(x,y) =r_θ(x,y)-β· u_θ(x,y) with scaling factor β>0β>0. The reward models are trained using the standard binary cross-entropy loss defined in Equation˜2, with modifications depending on the specific architecture as described below. 4.1 MLP head ensemble (ENS-MLP) A common approach to estimate epistemic uncertainty is to train an ensemble of K independent Multi-Layer Perceptron (MLP) heads using the embedding z provided by a pretrained LLM (melo2024deep; liu2024sampleefficient; dwaracherla2024efficient), as illustrated in Figure˜1(a). Each MLP head is parametrized by θ(k)∈ℝdθ^(k) ^d and predicts a pointwise reward rθ(k)(x,y)r_θ^(k)(x,y). The pointwise reward and uncertainty estimates in Equation˜9 are computed as the mean and standard deviation over the individual rewards rθ(x,y)=1K∑k=1Krθ(k)(x,y)anduθ(x,y)=1K−1∑k=1K(rθ(k)(x,y)−rθ(x,y))2. llasection eq:models-ens_mlp-estimateseq:models-ens_mlp-estimatesr_θ(x,y)= 1K _k=1^Kr_θ^(k)(x,y) u_θ(x,y)= 1K-1 _k=1^K r_θ^(k)(x,y)-r_θ(x,y) ^2. (10) The model is trained by minimizing the loss ℒ(θ;train)=1K∑k=1K(ℒbase(θ(k);train)+λd∥θ(k)−θinit(k)∥22+γn∑(x,y+,y−)∈train(rθ(k)(x,y+)+rθ(k)(x,y−))2), llasection eq:models-ens_mlp-losseq:models-ens_mlp-loss splitL(θ;D_train)&= 1K _k=1^K _base(θ^(k);D_train)+ λd θ^(k)-θ^(k)_init _2^2\\ &+ γn _(x,y^+,y ) _train(r_θ^(k)(x,y^+)+r_θ^(k)(x,y ))^2 , split (11) which consists of the standard cross-entropy loss from Equation˜2 and two regularization terms applied on each head. The first regularization term controlled by λ encourages the parameters of each head θ(k)θ^(k) to stay close to their random initialization θinit(k)θ^(k)_init, which preserves diversity across the heads in the ensemble. The second regularization term controlled by γ centers predicted rewards around zero (eisenstein2024helping). This is a crucial practical step often overseen in practice, as the cross-entropy loss is invariant to additive constants in the reward function, which could otherwise lead to poorly calibrated uncertainty estimates due to unintended large standard deviations. 4.2 LoRA adapter ensemble (ENS-LoRA) ENS-LoRA (muhlematter2025loraensemble) extends the framework of the MLP head ensemble model defined in Section˜4.1 by training all layers of the model instead of additional MLP heads. To overcome the computational constraints of training K models, Low-Rank Adaptation (LoRA) method is used to reduce the number of trainable parameters (wang2023lora). We denote each LoRA adapter by the parameter vector θ(k)θ^(k) and initialize a linear head for each adapter to obtain a pointwise reward rθ(k)(x,y)r_θ^(k)(x,y) from embeddings z. The adapters are trained by minimizing the loss defined in Equation˜11 and the reward rθ(x,y)r_θ(x,y) and uncertainty estimates uθ(x,y)u_θ(x,y) are computed following Equation˜10. 4.3 DPO-based MC dropout (MCD-DPO) Instead of training several heads of LoRA adapters, one can also leverage Monte-Carlo (MC) dropouts before the final layer of a fine-tuned model and estimate rewards implicitly. Formally, let πθ _θ be a fine-tuned LLM initialized from a reference policy πref _ref and trained to minimize the KL-regularized loss (christiano2017deep; ouyang2022training; stiennon2020learning). This policy defines an implicit reward model as rθ(x,y)=λlogπθ(y∣x)πref(y∣x)+λlogZ(x), llasection eq:dpo_mcd-implicit_reward_modeleq:dpo_mcd-implicit_reward_modelr_θ(x,y)=λ _θ(y x) _ref(y x)+λ Z(x), (12) where λ controls the KL-regularization term and Z(x)Z(x) is the partition function (rafailov2023direct). MCD-DPO (mehta2023sample) quantifies the uncertainty of this implicit reward function by introducing a dropout layer right before the language modeling head to enable MC dropout (gal2016dropout), as shown in Figure˜1(d). During inference, K dropout masks m(k)m^(k) are sampled and applied to the embedding z of a pretrained LLM, providing an ensemble of completion probabilities πθ(y∣x;m(k)) _θ(y x;m^(k)). We obtain the implicit rewards rθ(x,y;m(k))r_θ(x,y;m^(k)) using Equation˜12, denoted by the rDPOr_DPO layer in Figure˜1(d), and utilize the mean and standard deviation over these individual rewards rθ(x,y)=1K∑k=1Krθ(x,y;m(k))anduθ(x,y)=1K−1∑k=1K(rθ(x,y;m(k))−rθ(x,y))2r_θ(x,y)= 1K _k=1^Kr_θ(x,y;m^(k)) u_θ(x,y)= 1K-1 _k=1^K r_θ(x,y;m^(k))-r_θ(x,y) ^2 as our reward and uncertainty estimate for Equation˜9. In our implementation πθ _θ is trained with the DPO loss, derived by substituting Equation˜12 into the standard cross-entropy loss in Equation˜2, ℒ(θ;train)=1n∑(x,y+,y−)∈trainm∼ℙdropout−logσ(λlogπθ(y+∣x;m)πref(y+∣x)−λlogπθ(y−∣x;m)πref(y−∣x)), llasection eq:models-dpo_mcd-losseq:models-dpo_mcd-lossL(θ;D_train)= 1n _ subarrayc(x,y^+,y ) _train\\ m _dropout subarray- σ λ _θ(y^+ x;m) _ref(y^+ x)-λ _θ(y x;m) _ref(y x) , (13) with a randomly sampled dropout mask m per sample. 4.4 Bayesian linear head (BAY-LIN) Another common approach in the literature is to consider reward estimation as a Bayesian linear regression problem (das2024active; cercola2025efficient). This method also computes the embedding z for each prompt-completion pair (x,y)(x,y) but applies a single linear reward head rθ(x,y)=θ⊤zr_θ(x,y)=θ z with a Gaussian prior on the trainable parameters θ∼(0,λ−1I)θ 0,λ^-1I . The posterior on θ is then approximated using a Laplace approximation, resulting in the following Gaussian distribution θ∣train∼approx(θMAP,H−1|θ=θMAP)θ _train N _MAP,H^-1 |_θ= _MAP with mean centered at the posterior mode θMAP=argminθ−logp(θ|train) _MAP= *argmin_θ- p(θ|D_train) and the inverse covariance given by the Hessian of the negative log-posterior H=∇θ2−logp(θ|train)H=∇^2_θ- p(θ|D_train) evaluated at θMAP _MAP. Intuitively, the Gaussian distribution is centered at and fitted to the local curvature around the posterior mode. The posterior mode is obtained by equivalently minimizing θMAP=argminθℒbase(θ;train)+λ2∥θ∥22 llasection eq:models-bay_lin-losseq:models-bay_lin-loss _MAP= *argmin_θL_base(θ;D_train)+ λ2 θ _2^2 (14) corresponding to the cross-entropy loss in Equation˜2 with ℓ2 _2-regularization. The Hessian H=∑(x,y+,y−)∈trainw(x,y+,y−)⋅(z+−z−)(z+−z−)⊤+λI, llasection eq:hessian-weightedeq:hessian-weightedH= _(x,y^+,y ) _trainw(x,y^+,y )·(z^+-z )(z^+-z ) +λ I, (15) with weights w(x,y+,y−)=σ′(rθ(x,y+)−rθ(x,y−))w(x,y^+,y )=σ r_θ(x,y^+)-r_θ(x,y ) corresponds to the empirical covariance of the feature differences with larger weights for ambiguous predictions, i.e., rθ(x,y+)≈rθ(x,y−)r_θ(x,y^+)≈ r_θ(x,y ). However, these weights depend on the current parameter estimate θ. This dependence requires the entire sum in the Hessian to be recomputed in the active learning setting, where θ is updated iteratively. To avoid these high computational costs, das2024active omit these weights, allowing the Hessian H to be updated incrementally. We follow this unweighted approach to keep our evaluation practical. The final reward and uncertainty estimate in Equation˜9 are given by the predictive posterior mean and standard deviation rθ(x,y)=θ⊤zanduθ(x,y)=z⊤H−1z.r_θ(x,y)=θ z u_θ(x,y)= z H^-1z. 5 Experiments Figure 2: Ranking scores on RewardBench across different UQ methods, training datasets, pretrained and finetuned models, and model sizes. The ranking score is defined in Equation˜6. For a systematic comparison, we train and evaluate the model architectures outlined in Section˜4 across different datasets and base models with a unified evaluation procedure as described in Section˜5.1. Our main results are detailed in Section˜5.2. 5.1 Experimental setup Evaluation procedure For each uncertainty quantification method, we first perform a parameter optimization over common parameters (e.g., learning rate, base model family, and model size) and model-specific ones (e.g., regularization parameters, dropout rate). This search is done on the training and validation split of the UltraFeedback preference dataset333https://huggingface.co/datasets/trl-lib/ultrafeedback_binarized (cui2024ultrafeedback) consisting of around 62K and 1K samples, respectively. We select the best parameters by first applying an upper threshold on ECEECE and EBCEEBCE as introduced in Equations˜7 and 8 with 0.050.05 and 0.010.01, respectively, to ensure reasonable calibration, and then ranking according to RS0.2RS_0.2444We describe our choice of α=0.2α=0.2 in Section A.4. from Equation˜6. We report the final performance on the popular RewardBench dataset555We use the filtered split from https://huggingface.co/datasets/allenai/reward-bench (lambert2024rewardbench). Finally, we train each model on two additional datasets to evaluate our results robustness to the dataset’s source and size: the Skywork preference dataset666https://huggingface.co/datasets/Skywork/Skywork-Reward-Preference-80K-v0.2 (liu2024skyworkreward) with around 77K samples and the preference dataset for the Tulu 3 8B model777https://huggingface.co/datasets/allenai/llama-3.1-tulu-3-8b-preference-mixture (lambert2025tulu) with around 273K samples. Models We initialize our models from either the Qwen 3 family888https://huggingface.co/collections/Qwen/qwen3-67d247413f0e2e4f653967f (qwenteam2025qwen3) with sizes from 0.6B to 32B, and the Skywork-Reward-V2 Qwen 3 series999https://huggingface.co/collections/Skywork/skywork-reward-v2-685c86ce5d9c9e4be500c84 (liu2025skyworkrewardv2), which are further finetuned for the reward modeling task on a large-scale preference dataset of around 26M preference pairs and range from 0.6B to 8B. This provides us with a broad coverage over model sizes and pre-training purposes. Due to the larger computational requirements of ENS-LoRA and MCD-DPO, we consider models only up to 4B. Further experimental details, including hyperparameters, are provided in Appendix˜B. 5.2 Results Insights into accuracy As illustrated in Figure˜2, no single uncertainty quantification algorithm consistently dominates according to the ranking score RS0.2RS_0.2; rather, performance is highly contingent on model size, dataset, and pre-training. A critical determinant of performance is the base model initialization. Methods that rely on a fixed LLM backbone to provide embeddings, such as BAY-LIN and ENS-MLP, benefit significantly from initialization with a task-aligned reward model (e.g., the Skywork family). Conversely, when initialized from a generic base like Qwen 3, these methods underperform compared to ENS-LoRA and MCD-DPO, which fine-tune the full model parameters and are thus less sensitive to the quality of the initial embeddings. Additionally, we observe diminishing returns in ranking scores as model size increases, a phenomenon we attribute to the higher overconfidence of larger models, which is penalized by our metric. While BAY-LIN achieves the highest performance across most settings, it lags behind ENS-MLP on the UltraFeedback dataset, preventing a definitive recommendation. However, given that prior works typically utilize generic initializations, our findings strongly suggest that adopting task-aligned base models offers a potential for performance improvement. (a) Calibration of predictions (b) Calibration of (upper) bounds Figure 3: Calibration diagrams for Qwen3-0.6B (top) and Qwen3-4B (bottom) trained on UltraFeedback and evaluated on RewardBench. The predictions are well-calibrated when they agree with the actual probability per bin (i.e., on the diagonal), while the predicted upper bounds are well-calibrated when they consistently exceed the actual probability per bin (i.e., below the diagonal). The calibration metrics are defined in Equations˜7 and 8. The color intensity of each bar is proportional to the bin size. As described in Section˜3.3, the calibration diagrams for the upper and lower bounds are equivalent. Insights into calibration We further analyze the calibration of our UQ methods, as defined in Section˜3.3, for two representative initialization models, namely, Qwen3-0.6B and Qwen3-4B. As shown in Figure˜3, the different UQ methods provide similarly calibrated preference predictions and confidence bounds with ECE rates below 0.10.1 and EBCE rates below 0.010.01. However, we observe that the distribution of predicted preference probabilities concentrates around 0.50.5 for smaller models, as seen from the color intensities in Figure˜3(a), indicating lower certainty on average. Similarly, we observe that smaller models tend to be slightly overconfident if certain, since the predicted preference probabilities for >0.5>0.5 lie below the diagonal and for <0.5<0.5 above the diagonal. We discuss the unusual appearance of calibration diagrams in the context of preference classification in Section˜A.2. We provide detailed results and further discussion on both accuracy and calibration in Appendix˜C. 6 Discussion Conclusions Uncertainty quantification for reward models offers a promising direction for robust and sample-efficient RLHF, ultimately improving the safety and helpfulness of language models. In this work, we introduced RewardUQ, a unified framework to design and evaluate uncertainty-aware reward models, which complements prior work focusing on downstream applications exclusively. The unified formalization of existing approaches, a novel metric balancing accuracy and uncertainty, and a common evaluation procedure enable a more systematic comparison of different methods. Our results show that utilizing finetuned base models results in higher accuracy, but we find that the best-performing method remains instance dependent. By releasing our framework as an open-source library, we aim to lower the barrier to uncertainty-aware alignment research and foster applications in active learning, safe alignment, and reward robustness. Limitations Our study focuses on the intrinsic evaluation of UQ methods rather than their performance in downstream reinforcement learning loops. This scope was chosen intentionally to disentangle the quality of uncertainty estimates from the confounding variables inherent in PPO or DPO fine-tuning. We posit that identifying robust UQ signals in isolation provides a more reliable foundation for researchers than costly end-to-end ablations. Furthermore, as detailed in Section˜A.4, we acknowledge that our proposed ranking score (Equation˜6) entails specific trade-offs between calibration and discrimination that may not perfectly capture every nuance of downstream utility. Future Work To bolster the generalizability of our findings, future comparisons should expand to a broader range of algorithms, model families, datasets, and metrics. Additionally, while this work provides extensive empirical benchmarks, the theoretical mechanisms governing epistemic uncertainty in reward models for preference optimization remain under-explored. We hope this work serves as a launchpad for rigorous theoretical analysis and the development of next-generation applications in active preference learning and safety-constrained alignment. impact The rigorous study of UQ for reward models offers promising improvements to various downstream applications, such as reducing the costs of data collection via active learning, enhancing the safety of LLMs via uncertainty-aware alignment, and making reward models more interpretable. However, we acknowledge that highly accurate UQ methods could eventually serve as an additional attack vector targeting the uncertainties of LLMs. While current methods do not yet pose a significant risk, we emphasize the importance of monitoring dual-use risks in the future. Acknowledgments and Disclosure of Funding This work was supported as part of the Swiss AI initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project IDs a10, a145, and infra01 on Alps. Barna Pásztor was primarily supported by the ETH AI Center through an ETH AI Center doctoral fellowship, and Ido Hakimi primarily supported by the ETH AI Center through an ETH AI Center postdoctoral fellowship. References Appendix 1 Introduction 2 Related work 2.1 Methods for reward model uncertainty quantification 2.2 Applications for reward UQ 3 Uncertainty quantification for reward models 3.1 Problem statement 3.2 Accuracy metrics 3.3 Calibration metrics 4 Uncertainty-aware reward models 4.1 MLP head ensemble (ENS-MLP) 4.2 LoRA adapter ensemble (ENS-LoRA) 4.3 DPO-based MC dropout (MCD-DPO) 4.4 Bayesian linear head (BAY-LIN) 5 Experiments 5.1 Experimental setup 5.2 Results 6 Discussion References A Theoretical details A.1 Background on the uncertainty decomposition in preference classification A.2 Background on the symmetry in preference classification A.3 Background on the accuracy metrics A.4 Background on the ranking score A.4.1 Invariances of RS0RS_0 A.4.2 Invariances of RS1RS_1 A.5 Background on the calibration metrics B Experimental details B.1 Technical setup B.2 Hyperparameters B.3 Dataset preprocessing B.4 Evaluation on RewardBench C Supplementary results Appendix A Theoretical details A.1 Background on the uncertainty decomposition in preference classification Traditionally, the total uncertainty is decomposed into epistemic uncertainty, which describes the lack of knowledge in the model, and aleatoric uncertainty, which captures the irreducible randomness in the data (hullermeier2021aleatoric; kendall2017what). The two commonly used decompositions of the total uncertainty in terms of the variance and in terms of the entropy have been explicitly derived by depeweg2018decomposition and correspond to Var[Y∣X] [Y X ] =Varθ[Y[Y∣X;θ]] =Var_θ [ E_Y [Y X;θ ] ] +θ[VarY[Y∣X;θ]] + E_θ [Var_Y [Y X;θ ] ] H(Y∣X) H Y X =I(Y;θ) =I(Y;θ) +θ[H(Y∣X,θ)], + E_θ [H Y X,θ ], with X denoting the input, Y the output and θ the parameters of the assumed underlying statistical model. The first term describes the epistemic and the second term the aleatoric uncertainty. In the context of preference classification described in Section˜3.1, X denotes the sample (x,y,y′)(x,y,y ) and Y the label y≻y′1_ \y y \. The common statistical model, which we adopt in this work, is given by y≻y′∣x,y,y′∼Ber(σ(rθ(x,y)−rθ(x,y′))) llasection eq:statistical_model_assumptioneq:statistical_model_assumptiony y x,y,y σ r_θ(x,y)-r_θ(x,y ) (16) based on the Bradley-Terry model (bradley1952rank). Importantly, this statistical model makes the following assumptions: First, it assumes that the preference label contains Bernoulli noise, which leads to aleatoric uncertainty in the preference predictions. Second, it assumes that the preference signal comes from a deterministic reward function through the Bradley-Terry model and, hence, is free of randomness. Hence, under this statistical model assumption, reward models trained on preference data are free of aleatoric uncertainty and only contain epistemic uncertainty, as the aleatoric uncertainty is fully captured by the Bernoulli noise model. Some work (lou2025uncertaintyaware; yan2024rewardrobust) use the alternative statistical model assumption y≻y′∣R,R′ y y R,R ∼Ber(σ(R−R′)) σ R-R withR∣x,y R x,y ∼(rθ(x,y),sθ2(x,y)) r_θ(x,y),s_θ^2(x,y) R′∣x,y′ R x,y ∼(rθ(x,y′),sθ2(x,y′)) r_θ(x,y ),s_θ^2(x,y ) which assumes heteroscedastic Gaussian noise in the reward, leading to aleatoric uncertainty in the underlying reward models. This assumption is equivalent to y≻y′∣x,y,y′,Δε y y x,y,y , ∼Ber(σ(rθ(x,y)−rθ(x,y′)+Δε)) σ r_θ(x,y)-r_θ(x,y )+ withΔε∣x,y,y′ x,y,y ∼(0,sθ2(x,y)+sθ2(x,y′)). 0,s_θ^2(x,y)+s_θ^2(x,y ) . Intuitively, the Gaussian noise assumption in the reward smooths the sigmoid function with a Gaussian kernel with bandwidth sθ2(x,y)+sθ2(x,y′)s_θ^2(x,y)+s_θ^2(x,y ) as we marginalize over Δε . Hence, the smoothed sigmoid function converges towards a constant function at 0.50.5 with increasing noise level, while the original sigmoid function is recovered with zero noise. In summary, we adopt the more common statistical model assumption in Equation˜16 and assume the aleatoric uncertainty to be fully captured by the Bernoulli model, while the underlying reward model is free of aleatoric uncertainty. A.2 Background on the symmetry in preference classification Pairwise preference classification is a special form of binary classification, where the goal is predict the label y≻y′∈0,11_\y y \∈\0,1\ for a preference sample (x,y,y′)(x,y,y ). Unlike standard binary classification, the label is defined through the antisymmetric relation ≻ , which implies y≻y′=1−y′≻y.1_\y y \=1-1_\y y\. Hence, there is no distinction between positive and negative classes in pairwise preference classification, since each sample (x,y,y′)(x,y,y ) is equivalent to its flipped counterpart (x,y′,y)(x,y ,y) with the class label inverted. As a result, the predictive accuracy is fully characterized by the win rate defined in Equation˜4, which jointly describes the true positive (TP), true negative (TN), false positive (FP) and false negative (FN) rate as winrate=TPrate=TNrateand1−winrate=FPrate=FNrate.win\ rate=TP\ rate=TN\ rate 1-win\ rate=FP\ rate=FN\ rate. The same antisymmetry extends to preference probabilities, yielding p(y≻y′∣x,y,y′)=1−p(y′≻y∣x,y′,y).p(y y x,y,y )=1-p(y y x,y ,y). Similarly, an upper bound on a preference probability induces a corresponding lower bound for the flipped comparison and vice versa, i.e., p¯(y≻y′∣x,y,y′) p(y y x,y,y ) =1−p¯(y′≻y∣x,y′,y) =1- p(y y x,y ,y) p¯(y≻y′∣x,y,y′) p(y y x,y,y ) =1−p¯(y′≻y∣x,y′,y). =1- p(y y x,y ,y). In theory, this is irrelevant for computing the expected calibration errors defined in Equations˜7 and 8. However, in practice, when approximating these errors via binning, it is essential to consider both (x,y,y′)(x,y,y ) and (x,y′,y)(x,y ,y), effectively doubling the evaluation set (shen2024datacentric). This ensures that predictions for both y≻y′y y and y′≻y y contribute to the empirical frequencies in the corresponding bins. As a result, the calibration diagram for predictions is point-symmetric at (0.5,0.5)(0.5,0.5),101010This is why in a calibration diagram overconfidence appears as a flat line (i.e., above the diagonal on [0.0,0.5][0.0,0.5] and below the diagonal on [0.5,1.0][0.5,1.0]), while underconfidence forms a sigmoid-shaped curve. and the calibration diagrams for upper and lower bounds are equivalent, resulting in identical calibration errors for both. A.3 Background on the accuracy metrics Our extension of accuracy metrics to predictions under uncertainty in Section˜3.2 can be generalized to standard binary classification metrics. Specifically, categorizing predictions into confident and unconfident introduces an orthogonal dimension, resulting in a three-dimensional confusion tensor along the axes (onfident),(nconfident)×(rue),(alse)×(ositive),(egative),\C(onfident),U(nconfident)\×\T(rue),F(alse)\×\P(ositive),N(egative)\, where P and N denote the set of real positives and negatives, respectively. For example, the confident true positive (CTP) rate is then defined as CTPrate=|∩|||CTP\ rate= . In preference classification, there is no distinction between positive and negative classes as described in Section˜A.2 and the confusion tensor collapses into the 2×22× 2 matrix in Equation˜5, which should not be confused with the classical binary confusion matrix. Accordingly, we normalize by the total number of samples instead of by the per-class counts. A.4 Background on the ranking score Observe that all accuracy metrics introduced in Equations˜4 and 5 can be expressed in terms of the four base counts (confident) (unconfident) (17) (true) CT CT =|∩| = ↑ UT UT =|∩| = ↘ (false) CF CF =|∩| = ↓ UF UF =|∩| = ↘ with T=CT+UT=CT+UT and F=UT+UF=UT+UF. Since the total number of samples is fixed to the size of the evaluation dataset n=CT+UT+CF+UFn=CT+UT+CF+UF, there are only three degrees of freedom, capturing the overall accuracy of predictions under uncertainty. A ranking strategy reduces these three degrees of freedom to a single score, effectively compressing two dimensions along which differently performing models are ranked equally, reflecting the inherent trade-offs made by the ranking strategy. Recall our proposed ranking score in Equation˜6, which can be expressed in terms of these counts as RSα _α =CTratewinrate+α⋅(1−winrate)−CFrate(1−winrate)+α⋅winrate = CT\ ratewin\ rate+α·(1-win\ rate)- CF\ rate(1-win\ rate)+α·win\ rate =CT+α⋅F−CF+α⋅T, = CTT+α· F- CFF+α· T, where α∈[0,1]α∈[0,1] balances the inherent trade-offs between the three degrees of freedom. The general idea is to encourage confident true predictions and penalize confident false predictions. Depending on the choice of α, the score normalizes the number of confident predictions differently and puts a different focus on confidence and accuracy. In the following, we first discuss the two edge cases α=0α=0 and α=1α=1 and then how our ranking score formulation unifies both cases. (a) Ranking score ranges (b) Ranking score weights Figure 4: Background on our ranking score for different α. While the range is invariant of the win rate for α=0α=0, it has a linear dependence for α=1α=1 as shown in Figure˜4(a). The inherent trade-off underlying the choice of α is shown in Figure˜4(b), which visualizes the weights in our ranking score in Equation˜20. For example, with α=0.2α=0.2, when the win rate increases from 0.60.6 to 0.80.8, the confidence among true predictions is upweighted from 0.880.88 to 0.950.95 by a factor of ≈1.08≈ 1.08, while the confidence among false predictions is downweighted from 0.770.77 to 0.560.56 by a factor of ≈0.73≈ 0.73. Focus on confidence (RS0RS_0) When α=0α=0, the ranking score corresponds to RS0=CTratewinrate−CFrate1−winrate=CT−CF∈[−1,1]. llasection eq:ranking-0eq:ranking-0RS_0= CT\ ratewin\ rate- CF\ rate1-win\ rate= CTT- CFF∈[-1,1]. (18) This ranking score considers the relative rate of confidence among true and false predictions. In other words, it looks at the proportions of confident predictions conditioned on the correctness of the predictions. Intuitively, it encourages confidence among true predictions and penalizes confidence among false predictions, while it does not take into account the overall ratio between true and false predictions. In particular, observe that the range of this ranking score is invariant of the win rate as shown in Figure˜4(a). Overall, this ranking score focuses on the confidence of predictions. Remark: This ranking score is conceptually related to Youden’s index (youden1950index), a summary statistic in binary classification, which is defined as J=TPTP+FN−FPTN+FP=|∩|||−|∩|||J= TPTP+FN- FPTN+FP= - . Focus on accuracy (RS1RS_1) When α=1α=1, the ranking score corresponds to RS1=CTrate−CFrate=CTn−CFn∈[winrate−1,winrate]. llasection eq:ranking-1eq:ranking-1RS_1=CT\ rate-CF\ rate= CTn- CFn∈[win\ rate-1,win\ rate]. (19) This ranking score considers the absolute rate of confidence among all samples. In other words, it looks at the joint proportions of confidence and correctness. Intuitively, it encourages confident correctness and penalizes confident incorrectness among all samples, while it does not take into account the number of uncertain true and false predictions. Observe that the range of this ranking score is determined based on this win rate as shown in Figure˜4(a). Overall, this ranking score focuses on the confidence and accuracy of predictions. Unified formulation (RSαRS_α) The issue of RS0RS_0 is that it focuses too much on the confidence and cannot distinguish between models with completely different win rates, as long as the relative proportions of confidence are the same. The issue of RS1RS_1 is that it focuses too much on the accuracy and, when setting the uncertainty globally to zero, it simplifies to RS1=2⋅winrate−1RS_1=2·win\ rate-1. Hence, we introduce α∈[0,1]α∈[0,1] to balance the trade-off between both ends. To better understand the effect of α, we can rewrite our ranking score formulation into RSα=fα(winrate)⋅CTratewinrate−fα(1−winrate)⋅CFrate1−winrate llasection eq:ranking-unifiedeq:ranking-unifiedRS_α=f_α(win\ rate)· CT\ ratewin\ rate-f_α(1-win\ rate)· CF\ rate1-win\ rate (20) which considers the relative rate of confidence, but each weighted by some factor fα(x)=x+α⋅(1−x)f_α(x)= xx+α·(1-x) depending on the win rate. Intuitively, a higher win rate puts more weight on the bonus caused by confidence among true predictions, while a lower win rate puts more weight on the penalty caused by confidence among false predictions. We visualize fα(x)f_α(x) for different choices of α in Figure˜4(b). In this work, we choose α=0.2α=0.2 as it balances well the trade-off between confidence and accuracy. A.4.1 Invariances of RS0RS_0 Recall Equation˜18, which can be expressed in terms of the base counts from Equation˜17 as RS0=CTCT+UT−CFCF+UF.RS_0= CTCT+UT- CFCF+UF. This “difference of two ratios” introduces two invariances, as analyzed in the following. Invariance 1 (by normalization) One degree of freedom is lost due to normalization in the ratios. Hence, RS0RS_0 is invariant to changes in the numerator and denominator, as long as each ratio is preserved, e.g., CTCT UTUT CFCF UFUF RS0RS_0 4040 6060 22 88 0.2 gray=0.4−0.20.2 gray=0.4-0.2 4242 6363 11 44 0.2 gray=0.4−0.20.2 gray=0.4-0.2 Intuitively, two models with different numbers of true and false predictions (i.e., different win rates) are ranked equally if they share the same proportion of confident predictions among true and false predictions. Formally, we can scale the numerators and denominators in each ratio equally (confident) (unconfident) (true) CT CT →(1+δT)CT → 1+ δT CT UT UT →(1+δT)UT → 1+ δT UT (false) CF CF →(1−δF)CF → 1- δF CF UF UF →(1−δF)UF → 1- δF UF based on some δ.111111For simplicity, we omit that δ must ensure non-negative integer counts. The resulting ranking score RS0(δ) _0(δ) =(1+δT)CT(1+δT)CT+(1+δT)UT−(1−δF)CF(1−δF)CF+(1−δF)UF = 1+ δT CT 1+ δT CT+ 1+ δT UT- 1- δF CF 1- δF CF+ 1- δF UF =CTCT+UT−CFCF+UF = CTCT+UT- CFCF+UF =const. =const. is independent of δ and thus constant. Invariance 2 (by taking the difference) The other degree of freedom is eliminated by taking the difference between the two terms. Accordingly, RS0RS_0 is invariant to changes in both terms, provided their difference stays constant, e.g., CTCT UTUT CFCF UFUF RS0RS_0 4040 6060 22 88 0.2 gray=0.4−0.20.2 gray=0.4-0.2 7070 3030 55 55 0.2 gray=0.7−0.50.2 gray=0.7-0.5 Intuitively, two models with different degrees of confidence are ranked equally if the true and false predictions are affected by the confidence level similarly. Formally, we can increase or decrease the number of confident predictions for both true and false predictions (confident) (unconfident) (true) CT CT →CT+δ⋅T → CT+δ· T UT UT →UT−δ⋅T → UT-δ· T (false) CF CF →CF+δ⋅F → CF+δ· F UF UF →UF−δ⋅F → UF-δ· F based on some δ. The resulting ranking score RS0(δ) _0(δ) =CT+δ⋅T(CT+δ⋅T)+(UT−δ⋅T)−CF+δ⋅F(CF+δ⋅F)+(UF−δ⋅F) = CT+δ· T(CT+δ· T)+(UT-δ· T)- CF+δ· F(CF+δ· F)+(UF-δ· F) =CTCT+UT−CFCF+UF = CTCT+UT- CFCF+UF =const. =const. remains independent of δ, confirming the invariance. A.4.2 Invariances of RS1RS_1 Recall Equation˜19, which can be expressed in terms of the base counts from Equation˜17 as RS1=CTCT+UT+UF+CF−CFCT+UT+UF+CF.RS_1= CTCT+UT+UF+CF- CFCT+UT+UF+CF. Note that both denominators are constant, as they correspond to the number of evaluation samples. Invariance 1 (by indistinction between unconfident predictions) One degree of freedom is lost by not considering the base counts UTUT and UFUF separately. Hence, RS1RS_1 is invariant to changes in both counts as long as their sum is preserved, e.g., CTCT UTUT CFCF UFUF RS1RS_1 4040 6060 22 88 0.35 gray≈40/110−2/1100.35 gray≈ 40/110-2/110 4040 88 22 6060 0.35 gray≈40/110−2/1100.35 gray≈ 40/110-2/110 Intuitively, the ranking score does not distinguish between unconfident true and false predictions. Formally, we can change the base counts of unconfident predictions (confident) (unconfident) (true) CT CT →CT → CT UT UT →UT+δ → UT+δ (false) CF CF →CF → CF UF UF →UF−δ → UF-δ based on some δ. The resulting ranking score RS1(δ) _1(δ) =CTCT+(UT+δ)+(UF−δ)+CF−CFCT+(UT+δ)+(UF−δ)+CF = CTCT+ UT+δ + UF-δ +CF- CFCT+ UT+δ + UF-δ +CF =CTCT+UT+UF+CF−CFCT+UT+UF+CF = CTCT+UT+UF+CF- CFCT+UT+UF+CF =const. =const. is independent of δ and thus constant. Invariance 2 (by taking the difference) The other degree of freedom is eliminated by taking the difference between the two terms. Accordingly, RS1RS_1 is invariant to changes in both terms, provided their difference stays constant, e.g., CTCT UTUT CFCF UFUF RS1RS_1 4040 6060 22 88 0.35 gray≈40/110−2/1100.35 gray≈ 40/110-2/110 4848 5252 1010 0 0.35 gray≈48/110−10/1100.35 gray≈ 48/110-10/110 Intuitively, two models with different degrees of confidence are ranked equally if the true and false predictions are affected by the confidence level similarly. Formally, we can increase or decrease the number of confident predictions for both true and false predictions (confident) (unconfident) (true) CT CT →CT+δ → CT+δ UT UT →UT−δ → UT-δ (false) CF CF →CF+δ → CF+δ UF UF →UF−δ → UF-δ based on some δ. The resulting ranking score RS1(δ) _1(δ) =(CT+δ)(CT+δ)+(UT−δ)+(UF−δ)+(CF+δ)−(CF+δ)(CT+δ)+(UT−δ)+(UF−δ)+(CF+δ) = multlined CT+δ CT+δ + UT-δ + UF-δ + CF+δ \\ - CF+δ CT+δ + UT-δ + UF-δ + CF+δ multlined CT+δ CT+δ + UT-δ + UF-δ + CF+δ \\ - CF+δ CT+δ + UT-δ + UF-δ + CF+δ =CTCT+UT+UF+CF−CFCT+UT+UF+CF = CTCT+UT+UF+CF- CFCT+UT+UF+CF =const. =const. remains independent of δ, confirming the invariance. A.5 Background on the calibration metrics For readability, we use Y≻∈0,1Y ∈\0,1\ to denote the event y≻y′y y for given x,yx,y and y′y . Calibration of predictions The predicted preference probabilities are well-calibrated if they match the true preference probabilities, i.e., ℙ(Y≻∣pθ(Y≻)=p)=pP(Y p_θ(Y )=p)=p for all p∈[0,1]p∈[0,1], following guo2017calibration. The expected calibration error (ECE) is defined as ECE=p[|ℙ(Y≻∣pθ(Y≻)=p)−p|],ECE= E_p [ (Y p_θ(Y )=p)-p ], which penalizes over- and underestimations of the true preference probabilities. Since the true probabilities are unknown in practice, we measure the deviation from the empirical probabilities. Specifically, the predicted probabilities are grouped into M bins Bmm=1M\B_m\_m=1^M and we compute ECE ≈∑m=1M|Bm|n|ℙ(Bm)−pθ(Bm)| ≈ _m=1^M B_m n (B_m)-p_θ(B_m) ↓ with empirical probability ℙ(Bm)P(B_m) and average predicted probability pθ(Bm)p_θ(B_m) in each bin BmB_m (guo2017calibration; pavlovic2025understanding), as stated in Equation˜7. Calibration of bounds The predicted preference probability bounds are well-calibrated if they are not violated by the true preference probabilities, i.e., ℙ(Y≻∣pθ¯(Y≻)=p¯) (Y p_θ(Y )= p) ≥p¯ ≥ p ℙ(Y≻∣pθ¯(Y≻)=p¯) (Y p_θ(Y )= p) ≤p¯ ≤ p for all p¯,p¯∈[0,1] p, p∈[0,1]. We introduce the expected lower calibration error (ELCE) and expected upper calibration error (EUCE) as follows ELCE =p¯[max(p¯−ℙ(Y≻∣pθ¯(Y≻)=p¯),0)] = E_ p [ p-P Y p_θ(Y = p),0 ] EUCE =p¯[max(ℙ(Y≻∣pθ¯(Y≻)=p¯)−p¯,0)] = E_ p [ Y p_θ(Y )= p - p,0 ] with ELCE penalizing lower bounds that overestimate the true preference probability and EUCE penalizing upper bounds that underestimate the true preference probability. In practice, analogous to ECE, we group the lower and upper bounds separately into M bins Bmm=1M\B_m\_m=1^M and compute ELCE ≈∑m=1M|Bm|nmax(pθ¯(Bm)−ℙ(Bm),0) ≈ _m=1^M B_m n p_θ(B_m)-P(B_m),0 ↓ EUCE ≈∑m=1M|Bm|nmax(ℙ(Bm)−pθ¯(Bm),0) ≈ _m=1^M B_m n (B_m)- p_θ(B_m),0 ↓ with pθ¯(Bm) p_θ(B_m) and pθ¯(Bm) p_θ(B_m) denoting the average predicted lower and upper bounds in the corresponding bin BmB_m, as stated in Equation˜8. Appendix B Experimental details B.1 Technical setup All models were trained on a single node equipped with four NVIDIA GH200 GPUs, providing a total of 378GB of VRAM. Our implementation is built on top of Transformers (wolf2020transformers) and TRL (vonwerra2025trl) by HuggingFace, with multi-GPU management handled by Accelerate (gugger2022accelerate). We use data parallelism for models that fit on a single GPU, and model and tensor parallelism for larger models. B.2 Hyperparameters We used the AdamW optimizer (loshchilov2018decoupled) (with a weight decay of 0, β1=0.9 _1=0.9, β2=0.999 _2=0.999, and ϵ=10−8ε=10^-8) and an effective batch size of 64 via gradient accumulation for all experiments, except for the LoRA adapter ensemble, which used a batch size of 16. We trained our models using a single epoch with a cosine learning rate scheduler and a warm-up phase of 5% of the total number of steps. The reward uncertainty bounds in Equation˜9 are constructed using β=2β=2 for the MLP head ensemble and DPO-based MC dropout and β=0.5β=0.5 for the Bayesian linear head. The best hyperparameters for each base model are given in Table˜1. MLP head ensemble We use an ensemble of K=20K=20 MLP heads. Each head is a two-layer network with 128 nodes per layer and ReLU activation functions. They are initialized using a Xavier uniform distribution with a gain of one. We conducted a grid search over the learning rate η∈10−5,10−4,10−3η∈\10^-5,10^-4,10^-3\ and regularization parameters λ∈0.0,0.1,1.0λ∈\0.0,0.1,1.0\ and γ∈0.0,0.01,0.1γ∈\0.0,0.01,0.1\ from Equation˜11. LoRA adapter ensemble We use an ensemble of K=8K=8 LoRA adapter with rank rLoRA=16r_LoRA=16 and scaling factor αLoRA=32 _LoRA=32. We conducted a grid search over the learning rate η∈10−5,10−4,10−3η∈\10^-5,10^-4,10^-3\ and regularization parameters λ∈0.001,0.01,0.1λ∈\0.001,0.01,0.1\ and choose γ=0.01γ=0.01. DPO-based MC dropout The ensemble is formed by sampling K=20K=20 dropout masks at inference time. We conducted a grid search over the learning rate η∈10−7,10−6,10−5η∈\10^-7,10^-6,10^-5\, DPO regularization parameter λ∈0.01,0.05,0.1λ∈\0.01,0.05,0.1\ and dropout rate pdropout∈0.01,0.05,0.1p_dropout∈\0.01,0.05,0.1\ from Equation˜13. Bayesian linear head We conducted a grid search over the learning rate η∈10−3,10−2,10−1η∈\10^-3,10^-2,10^-1\ and ℓ2 _2-regularization parameter λ∈10−3,10−2,10−1λ∈\10^-3,10^-2,10^-1\ from Equation˜14. Table 1: Best hyperparameters found according to our evaluation procedure described in Section˜5.1. base model ENS-MLP ENS-LoRA MCD-DPO BAY-LIN η λ γ η λ γ η λ pdropoutp_dropout η λ Qwen3-0.6B 0.001 0 0.01 0.0001 0.01 0.01 0.00001 0.01 0.05 0.01 0.01 Qwen3-1.7B 0.001 0 0.01 0.0001 0.01 0.01 0.00001 0.01 0.1 0.001 0.01 Qwen3-4B 0.001 0.1 0 0.0001 0.001 0.01 0.00001 0.01 0.1 0.001 0.001 Qwen3-8B 0.001 0.1 0 0.001 0.01 Qwen3-14B 0.001 1 0 0.001 0.01 Qwen3-32B 0.001 1 0.01 0.001 0.1 Skywork-Qwen3-0.6B 0.001 0.1 0.01 0.0001 0.01 0.01 0.00001 0.01 0.1 0.1 0.001 Skywork-Qwen3-1.7B 0.001 1 0.01 0.0001 0.01 0.01 0.00001 0.01 0.1 0.1 0.01 Skywork-Qwen3-4B 0.001 0.1 0.01 0.0001 0.001 0.01 0.00001 0.01 0.05 0.1 0.1 Skywork-Qwen3-8B 0.001 1 0.01 0.1 0.1 B.3 Dataset preprocessing For all datasets, we remove preference samples if the total sequence length of the prompt and one of the completions exceeds 2048 tokens to avoid additional evaluation noise due to the truncation of the prompt and completions. B.4 Evaluation on RewardBench RewardBench (lambert2024rewardbench) consists of the four main categories “Chat”, “Chat Hard”, “Safety”, and “Reasoning”, each with weighted subcategories. Following the standard procedure, we first compute our metrics from Sections˜3.2 and 3.3 (excluding the ranking score) within each category using the subcategory weights, then average them across categories, and finally derive the ranking score from these averages. Appendix C Supplementary results Figure˜5 provides detailed results supplementing the discussion in Section˜5.2. Figure˜5(a) provides a granular decomposition of the ranking score trends, explicitly isolating the contributions of win rate, Confident True rate, and Confident False rate. Validating the importance of initialization, methods utilizing the task-aligned Skywork base model consistently outperform the generic Qwen 3 base models across both win rate and CT rate. This advantage is particularly pronounced for fixed-head methods like BAY-LIN, which rely heavily on high-quality embeddings to produce confident, correct predictions. Furthermore, the diminishing returns in ranking scores observed in Section˜5.2 are elucidated by the behavior of the CF rate relative to other metrics. While win rates generally improve or plateau with model size, the CF rate decreases at a lower rate than the CT rate increases. Regarding calibration, Figure˜5(b) supports the general observation that most UQ methods maintain reasonable calibration, with Expected Calibration Error (ECE) and Expected Bound Calibration Error (EBCE) typically remaining below 0.10 and 0.04, respectively. However, the breakdown reveals specific instabilities that impact the aggregate performance; for example, ENS-MLP trained on the generic Qwen 3 base exhibits a sharp spike in EBCE at the 32B scale. This degradation in bound calibration aligns with the underperformance of ENS-MLP in the main ranking results for that configuration, suggesting that larger model sizes can lead to overfitting instabilities. Overall, these supplementary metrics confirm that the superior ranking of task-aligned models is driven not just by higher accuracy, but by a more favorable balance of maximizing confident true predictions while suppressing confident errors. (a) Accuracy metrics (b) Calibration metrics Figure 5: Our base metrics on RewardBench across different UQ methods, training datasets, pretrained and finetuned models, and model sizes. The accuracy metrics are defined in Equations˜4 and 5 and the calibration metrics in Equations˜7 and 8.