Paper deep dive
Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons
Kaustubh Shivshankar Shejole, Tanish Agarwal, Arpit Agarwal, Avishek Ghosh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/13/2026, 4:20:41 AM
Summary
This paper introduces BoRaEM, an algorithm for jointly learning item rewards and worker reliability (competency) from pairwise comparison data using the Boltzmann-rational model. By employing Pólya-Gamma augmentation, the authors transform the non-convex logistic likelihood into a conditionally Gaussian form, enabling an efficient Expectation-Maximization (EM) algorithm. The method reduces the M-step to a matrix sensing problem, providing theoretical convergence guarantees. Experiments demonstrate robustness against spammers and adversarial workers in crowdsourcing settings.
Entities (12)
Relation Signals (9)
BoRaEM → uses → Boltzmann-rational model
confidence 98% · We adopt the Boltzmann-rational model for pairwise comparisons... We derive an EM-based algorithm for learning under this model
BoRaEM → employs → Pólya-Gamma augmentation
confidence 96% · we adopt the Pólya-Gamma (PG) augmentation framework... to transform the logistic likelihood into a conditionally Gaussian form.
BoRaEM → extends → Bradley-Terry-Luce model
confidence 95% · which extends the Bradley-Terry-Luce model by incorporating worker competencies.
BoRaEM → isbasedon → Expectation-Maximization
confidence 95% · We derive an expectation-maximization (EM) algorithm for inference under this model.
BoRaEM → handles → Adversarial workers
confidence 94% · confirm its strong robustness to both spammers and adversarial workers
BoRaEM → handles → Spammers
confidence 94% · confirm its strong robustness to both spammers and adversarial workers
BoRaEM → reducesto → Matrix sensing
confidence 93% · This technique allows us to reduce our formulation to a matrix sensing problem
Crowdworkers → use → Amazon Mechanical Turk
confidence 92% · comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley-Terry-Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified $Q$ function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings. The code and data is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.10045v1
- Canonical: https://arxiv.org/abs/2608.10045v1
Trouble viewing inline? Open PDF directly →
Full Text
87,285 characters extracted from source content.
Expand or collapse full text
Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisons Kaustubh Shivshankar Shejole IIT Bombay Mumbai, India – 400076 Equal contribution Tanish Agarwal IIT Bombay Mumbai, India – 400076 Equal contribution Arpit Agarwal IIT Bombay Mumbai, India – 400076 Avishek Ghosh IIT Bombay Mumbai, India – 400076 Abstract The problem of learning from pairwise comparisons has been widely studied across many domains such as recommendation systems, social choice, and more recently, fine-tuning large language models. In this problem, the goal is to learn item rewards based on pairwise comparisons between them. In many scenarios, these comparisons are elicited from crowdworkers using platforms such as Amazon Mechanical Turk, Scale AI, etc. However, crowdworkers are often unreliable due to limited domain knowledge or revenue-maximizing (spamming) behavior. In this work, our goal is to understand whether worker reliability (competency) can be learned jointly with item rewards. To this end, we adopt the Boltzmann-rational model for pairwise comparisons, which extends the Bradley–Terry–Luce model by incorporating worker competencies. We derive an EM-based algorithm for learning under this model by introducing Polya-Gamma latent variables to transform the logistic likelihood into a conditionally Gaussian form, enabling tractable optimization and leading to a simplified Q function in the E-step of the algorithm. This technique allows us to reduce our formulation to a matrix sensing problem, using which we establish theoretical convergence guarantees for our algorithm. We conduct extensive experiments on real-world and synthetic datasets. These experiments demonstrate the advantages of using our algorithm over several baselines and confirm its strong robustness to both spammers and adversarial workers, highlighting its practical effectiveness in realistic crowdsourcing and reward learning settings.111The code and data is publicly available at https://github.com/KaustubhShejole/BoRa_EM. 1 Introduction Reward learning from pairwise comparisons is a widely studied problem in machine learning, with applications in recommendation systems, information retrieval, social choice, and, more recently, fine-tuning of large language models. In this problem, given pairwise comparisons between K items, the goal is to learn a reward for each item that is “consistent” with the observed comparisons. For example, in recommendation systems, one observes the choices of users over recommended items and the goal is to learn a score for each item [kalloori2018eliciting]; in social choice, one observes preferences of individuals over various policy alternatives and the goal is to aggregate the preferences into a single ranked list [list2013social]; and in fine-tuning large language models, one observes pairwise comparisons between different model responses and the goal is to learn a reward for each response [bakker2022fine, 10.5555/3495724.3495977]. Given the large-scale requirement of preference data in these applications, comparisons between items are often elicited using human crowdworkers [velayati2025crowdsourcing, chen2013pairwise]. Due to this, many crowdsourcing marketplaces such as Amazon Mechanical Turk, Scale AI, Figure Eight, and Yandex Toloka have gained prominence in recent years. However, it has been observed that crowdworkers often exhibit large variability in the quality of labels [gadiraju2015understanding, ding2022effectiveness]. There may be several factors leading to this variability such as insufficient knowledge of the domain, lack of time/effort devoted to the tasks, and revenue maximizing/spamming behavior. The default approach adopted by many platforms is to identify a gold-standard dataset and evaluate worker competency on it [chen2013pairwise]. However, in many scenarios such datasets are not available, and requiring workers to answer gold-standard questions can be inefficient and costly. In this work, we study the problem of jointly estimating item rewards and worker competencies from pairwise comparison data, without requiring a gold-standard dataset. We adopt the Boltzmann-rational model, in which each item j∈[K]j∈[K] has a latent reward rjr_j and each worker s∈[M]s∈[M] has a competency parameter βs∈[−1,1] _s∈[-1,1]. The probability that worker s prefers item w over item l is given by P[w≻l∣s]=σ(βs(rw−rl)),P[w l s]=σ\! ( _s(r_w-r_l) ), where σ denotes the logistic function. This model captures a spectrum of worker behaviors: βs=1 _s=1 corresponds to an ideal annotator, βs=0 _s=0 to a random spammer, and βs=−1 _s=-1 to an adversarial worker. Given a dataset =(wi,li,si)i=1ND=\(w_i,l_i,s_i)\_i=1^N of N pairwise comparisons, the goal is to jointly estimate the reward vector ∈ℝKr ^K and the competency vector ∈[−1,1]M β∈[-1,1]^M. We derive an expectation-maximization (EM) algorithm for inference under this model. To derive the EM algorithm, we use a Pólya-Gamma augmentation [polson2013bayesian], which introduces auxiliary variables to transform the logistic likelihood into a conditionally Gaussian form. This leads to a quadratic objective for the M-step which enables efficient alternating maximization over β and r in closed form (Section 4). While this alternating minimization can be solved efficiently, it is not guaranteed to converge to a global optimum. To address this, we show that the M-step optimization can be cast as a rank-1 matrix sensing problem. Specifically, defining X=⊤∈ℝM×KX= βr ^M× K and a linear operator :ℝM×K→ℝNA:R^M× K ^N, the M-step reduces to minimizing ‖(⊤)−‖22\|A( βr )-u\|_2^2. We establish that A satisfies the Restricted Isometry Property (RIP) given a sufficient number of samples. Leveraging results from non-convex low-rank matrix recovery [park2016nonsquarematrixsensingspurious], we prove that under the RIP condition, at each iteration of the EM algorithm, every local minimum of the M-step objective is near-global minimum. Using standard results for EM, we are able to show that the likelihood increases monotonically and converges to a stationary point of the likelihood function. We conduct extensive experiments on both real-world and synthetic datasets. The experimental results demonstrate that our algorithm outperforms baseline methods, and more importantly, it remains robust in scenarios with large numbers of spammers of various types and achieves significant performance gains over other existing approaches in this case. Thus, we provide a unified framework capable of handling diverse spammer behaviors—from random clickers to malicious adversaries—offering a practical and theoretically grounded alternative for real-world crowdsourcing platforms where worker quality is heterogeneous and unknown a priori. 2 Related Work Learning rewards from pairwise comparisons has been widely used across many domains such as economics, operations research and machine learning. The classical Bradley-Terry-Luce (BTL) model [bradley1952rank] assumes that each item is associated with an underlying reward, and pairwise comparisons between items are drawn by taking into account the reward difference between them. Specifically, the probability that item w is preferred over item l is given as P(w≻l)=σ(rw−rl),P(w l)=σ(r_w-r_l), where σ(⋅)σ(·) denotes the sigmoid function, and =rjj=1Kr=\r_j\_j=1^K represents the latent reward or score associated with each of the K items. The BTL model, however, does not capture the heterogeneity in the rewards observed by different crowdworkers. negahban2012iterative proposed RankCentrality (RC), which interprets pairwise comparisons as a directed graph, where nodes represent items and directed edges represent comparison outcomes. This formulation enables the use of random-walk–based methods for ranking items. The closest to our work is Crowd-BT model [chen2013pairwise] which extends the BTL model by incorporating worker reliability. For a worker s, the probability of preferring item i over j is modeled as Pr(i≻j∣s)=βsσ(ri−rj)+(1−βs)σ(rj−ri) (i j s)= _s\,σ(r_i-r_j)+(1- _s)\,σ(r_j-r_i), where ri,rjr_i,r_j are item rewards and βs _s captures the worker’s competency. This model is effectively a mixture of two BTL models – one with reward r and the other with negative reward i.e., −-r. The mixture weight represents the competency of the crowdworker, with βs=1/2 _s=1/2 representing a worker that labels uniformly at random. Crowd-BT is an alternating minimization approach to jointly learn the rewards and competencies. However, chen2013pairwise do not provide any theoretical guarantees, and they require a gold standard dataset to initialize worker competencies, which is not always available. On the other hand, our work adopts the Boltzmann rational choice model, which provides a more natural and theoretically grounded formulation for preference-based reward learning and alignment, and allows us to prove theoretical convergence results. Another related work is bugakova2019aggregation, which introduced the Factor-BT model to explicitly capture systematic biases present in crowdsourced pairwise comparisons. Unlike prior models, which treat task-related biases as noise or ignore them altogether, FactorBT explicitly models these biases by incorporating known but irrelevant task features (e.g., item position on screen, background design) that may influence worker decisions. For a comparison between items did_i and djd_j by worker s, described by a feature vector sij∈ℝMx_sij ^M, where M is the feature dimension. The probability that did_i is preferred over djd_j is given as follows Pr(di≻sdj)=σ(γs)⋅σ(ri−rj)+(1−σ(γs))⋅σ(⟨s,sij⟩), (d_i _sd_j)=σ( _s)·σ(r_i-r_j)+(1-σ( _s))·σ( b_s,x_sij ), where ri,rjr_i,r_j are the latent scores of items did_i and djd_j, γs _s controls the susceptibility of worker s to bias and σ(γs)σ( _s) can be considered as the the skill of the worker same as that of βs _s in our formulation, s b_s encodes the worker-specific sensitivity to task features, σ(⋅)σ(·) is the sigmoid function. The model estimates parameters ri\r_i\, γs\ _s\, and s\ b_s\ by maximizing the regularized log-likelihood of the observed pairwise comparisons. Unlike FactorBT [bugakova2019aggregation], we do not explicitly model the influence of task-irrelevant factors such as item position, and we instead focus on learning worker competency jointly with rewards. jin2020rank proposed two models that explicitly account for annotator competence: the Heterogeneous Bradley-Terry-Luce (HBTL) model and the Heterogeneous Thurstone Case V (HTCV) model. The HBTL model is based on the Boltzmann-Rational formulation given in Equation (2). In contrast, the HTCV model assumes that the probability of item w being preferred over item l by annotator s is given by P(w≻l;s)=Φ(βs(rw−rl)2),P(w l;s)= ( _s(r_w-r_l) 2 ), (1) where Φ(⋅) (·) denotes the cumulative distribution function (CDF) of the standard normal distribution, and βs _s captures the competence of annotator s. They perform alternating gradient descent for both these methods and provide theoretical convergence guarantees. Bias-Aware Ranking from Pairwise comparisons (BARP) [ferrara2024bias] extends the classic BTL model to account for evaluator biases in pairwise comparisons. Each evaluator is assigned a bias parameter that distorts the latent quality scores of items based on the group membership of items. By explicitly modeling the log-likelihood over items’ latent scores and evaluators’ biases, BARP uses an alternating optimization approach to jointly estimate both. In BARP, the probability that evaluator s prefers item i over j is modeled as P(i≻sj)=σ((ri−rj)+θs(δig−δjg)),P(i _sj)\;=\;σ\! ((r_i-r_j)+ _s\,( _ig- _jg) ), where ri,rjr_i,r_j are the latent item scores and θsδig _s _ig represents the bias of evaluator s for the group g of item i. Unlike the standard BTL model, this formulation accounts for biased perception of item scores rather than true latent scores. Organization Section 3 introduces the problem of jointly learning rewards and worker competence from pairwise comparisons. Section 4 presents the derivation of our proposed method (BoRaEM) for the Boltzmann Rational Model using Expectation–Maximization (EM), while Section 5 establishes theoretical guarantees for its convergence. Section 6 reports experimental results and analysis. Finally, Section 7 concludes with a discussion of future research directions. 3 PROBLEM FORMULATION We consider a setting where there is a set of K items, and one observes outcomes of pairwise comparisons between these items, elicited from a set of M crowdworkers. Specifically, we are given a dataset =(wi,li,si)i=1ND=\(w_i,l_i,s_i)\_i=1^N where there are N pairwise comparisons; the i-th comparison is performed by worker si∈[M]s_i∈[M] where wi∈[K]w_i∈[K] is the winning item and li∈[K]l_i∈[K] is the losing item such that lil_i is not equal to wiw_i. We assume that the outcomes of these comparisons are generated according to the Boltzmann-rational model which is an extension of the BTL model and has received a lot of attention recently [daniels2022expertise, jeon2020reward, ziebart2010maximum]. Similar to the BTL model, each item j∈[K]j∈[K] is associated with a reward rjr_j, and the probability of winning a comparison is governed by pairwise reward differences. This model also considers the heterogeneity of human decision-making. Specifically, humans are considered rational decision-makers, where the quality of their decisions is modulated by a “rationality” parameter β∈[−1,1]β∈[-1,1] [daniels2022expertise, jeon2020reward, ziebart2010maximum]. Formally, the probability that item w beats item l given worker s is given by P[w≻l;s]=exp(βsrw)exp(βsrw)+exp(βsrl)=11+exp(−βs(rw−rl))=σ(βs(rw−rl)).P[w l;s]= ( _sr_w) ( _sr_w)+ ( _sr_l)= 11+ (- _s(r_w-r_l))=σ( _s(r_w-r_l)). (2) The case βs=1 _s=1 corresponds to a perfect annotator who makes decisions according to the underlying BTL model, βs=−1 _s=-1 for an “adversarial” worker, and βs=0 _s=0 for a “spammer” who labels uniformly at random. Given a dataset of pairwise comparisons =(wi,li,si)i=1ND=\(w_i,l_i,s_i)\_i=1^N that is drawn according to the Boltzmann-rational model with reward vector =rjj=1Kr=\r_j\_j=1^K and worker competency vector =βss=1M β=\ _s\_s=1^M, the goal is to jointly estimate r and β. Since the reward model is invariant to both additive shifts and scaling, as the likelihood depends only on the product βs(rw−rl) _s(r_w-r_l), the transformations r→r+cr→ r+c and r→cr→ cr, βs→βs/c _s→ _s/c leave the likelihood unchanged, resulting in non-identifiability. To address this, we enforce identifiability by centering the rewards to remove translation invariance and normalizing them to unit root-mean-square after each EM iteration, with a compensatory rescaling of worker competencies. This fixes the latent scale and selects a unique representative solution. Additionally, mild ℓ2 _2 regularization on rewards and a Gaussian prior on competencies improve numerical stability and prevent degenerate scaling. More details about implementation are provided in Appendix E. 4 BoRaEM Algorithm In order to solve this problem, one can formulate a log-likelihood objective based on the underlying model. Specifically, following Equation (2) for the probability that a worker sis_i favors wiw_i over lil_i: p(wi≻li∣si)=σ(βsi(rwi−rli))p(w_i l_i s_i)=σ( _s_i(r_w_i-r_l_i)) (3) Let =, θ=r, β. The associated complete-data likelihood is ℒcomplete(;)=∏(wi,li,si)∈σ(βsi(rwi−rli))L_ complete( θ;D)= _(w_i,l_i,s_i) σ ( _s_i(r_w_i-r_l_i) ) (4) where D refers to the comparisons data. However, direct maximum likelihood estimation (MLE) of the parameter vectors r and β is challenging as the objective is non-concave. This complicates both optimization and theoretical analysis. To circumvent this issue, we adopt the Pólya-Gamma (PG) augmentation framework [polson2013bayesian]222We encourage readers to also refer to a useful blog on Pólya-Gamma (PG) augmentation at https://gregorygundersen.com/blog/2019/09/20/polya-gamma/.. For a latent variable ω∼PG(b,0)ω (b,0) where b>0b>0, and any ψ∈ℝψ , the logistic likelihood admits the integral representation: (eψ)a(1+eψ)b=eκψ2b∫0∞exp(−ωψ22)p(ω∣b,0)ω, (e^ψ)^a(1+e^ψ)^b= e^κψ2^b _0^∞ (- ωψ^22 )p(ω b,0)\,dω, (5) with κ=a−b/2κ=a-b/2, where p(⋅)p(·) is the density of ω. We apply this identity to our model by setting a=1a=1, b=1b=1, and ψ=ηiψ= _i where ηi:=βsi(rwi−rli) _i:= _s_i(r_w_i-r_l_i). Thus, the probability of the ithi^th pairwise comparison can be written as: p(wi,li,si|) p(w_i,l_i,s_i| θ) =σ(ηi) =σ( _i) =12eηi/2∫0∞exp(−ωiηi22)p(ωi)ωi, = 12e _i/2 _0^∞\! (- _i _i^22 )p( _i)d _i, (6) where ωi∼PG(1,0) _i (1,0) is a latent Pólya-Gamma variable introduced for each ithi^th comparison. So, the joint density p(wi,li,si,ωi|)p(w_i,l_i,s_i, _i| θ) is given by p(wi,li,si,ωi|)∝exp(−ωiηi22+ηi2)p(ωi)p(w_i,l_i,s_i, _i| θ) (- _i _i^22+ _i2 )p( _i) (7) which takes an exponential form. Expectation Maximization. Using Equation (7), and given all ωi _i, the log-likelihood objective over =(rjj=1K,βll=1M), θ\;=\; (\\,r_j\_j=1^K,\;\\, _l\_l=1^M ), simplifies to the following form upto additive constant that do not depend on θ (more details in Appendix A.2). ℓ() ( θ) =∑(wi,li,si)∈[12βsi(rwi−rli) = _(w_i,l_i,s_i) [ 12\, _s_i\,(r_w_i-r_l_i) . −12ωiβsi2(rwi−rli)2] .- 12\, _i\, _s_i^2\,(r_w_i-r_l_i)^2 ] (8) Hence, this gives an EM algorithm that iteratively estimates the latent variables ωi _i followed by log-likelihood maximization steps. Algorithm 1 describes this EM Algorithm. The Expectation step (E-step) and Maximization step (M-step) are as follows (details in Appendix A.3): 4.1 E-step At iteration t, given the current estimates (t) θ^(t) we have ηi(t)=βsi(t)(rwi(t)−rli(t)),∀i∈[N] _i^(t)\;=\; _\,s_i^(t)\, (r_\,w_i^(t)-r_\,l_i^(t) ), ∀ i∈[N] (9) Given ηi(t) _i^(t), the Pólya–Gamma random variable ωi _i follows the distribution PG(1,ηi(t))PG(1,\; _i^(t)). Hence, we can evaluate the expectation of ωi _i given the current parameters. Let κi(t)=[ωi|ηi(t)] _i^(t)=E[ _i 10000\ | 10000\ _i^(t)]. We show that (Appendix A.3.1) κi(t)=12ηi(t)tanh(ηi(t)2),ηi(t)≠014,ηi(t)=0 _i^(t)= cases 12\, _i^(t)\, ( _i^(t)2 ),& _i^(t)≠ 0\\[8.0pt] 14,& _i^(t)=0 cases (10) This gives the formulation for the Q function: Q(∣(t)) Q ( θ θ^(t) ) =∑(wi,li,si)∈[12βsi(rwi−rli) = _(w_i,l_i,s_i) [ 12\, _s_i\,(r_w_i-r_l_i) . −12κi(t)βsi2(rwi−rli)2] .- 12\, _i^(t)\, _s_i^2\,(r_w_i-r_l_i)^2 ] (11) Algorithm 1 BoRaEM Algorithm 1:Dataset of pairwise comparisons D 2:Parameter estimates =(,) θ=( β, r) 3:Initialize (0) θ^(0) 4:t←0t← 0 5:repeat 6: E-step: 7: Q(∣(t))←ω∣,(t)[ℓ(;,ω)]Q( θ θ^(t)) _ω , θ^(t)[ ( θ;D,ω)] ⊳ See (11) 8: M-step (using AM): 9: Initialize ((0),(0))←θ(t)( β^(0), r^(0))←θ^(t), k←0k← 0 10: repeat 11: (k+1)←argmaxQ(,(k)∣(t)) β^(k+1)← _ βQ( β, r^(k) θ^(t)) 12: (k+1)←argmaxQ((k+1),∣(t)) r^(k+1)← _ rQ( β^(k+1), r θ^(t)) 13: (k+1)←(k+1)−T(k+1)n r^(k+1)← r^(k+1)- 1^T r^(k+1)n 14: k←k+1k← k+1 15: until convergence 16: (t+1)←((k−1),(k−1)) θ^(t+1)←( β^(k-1), r^(k-1)) 17: t←t+1t← t+1 18:until convergence 19:return (t+1) θ^(t+1) 4.2 M-step In order to maximize the Q(∣(t))Q ( θ θ^(t) ) function in terms of =(,) θ=( β, r), we use the Alternating Maximization (AM) algorithm. We first initialize θ to (t) θ^(t). We update β corresponding to Step 11 of Algorithm 1 as follows (Appendix A.3.2) (Let di=rwi−rlid_i=r_w_i-r_l_i denote the reward difference between the preferred item wiw_i and the losing item lil_i, evaluated at the current item reward estimates within the inner iterations of the M-step.) βs=12∑i∈Isdi∑i∈Isκi(t)di2=∑i∈Isdi2∑i∈Isκi(t)di2 _s=\; 12 _\,i∈ I_sd_i _\,i∈ I_s _i^(t)\,d_i^2\;=\; _\,i∈ I_sd_i2\, _\,i∈ I_s _i^(t)\,d_i^2 (12) where Is=i∈[N]:si=sI_s\;=\; \\,i∈[N]:s_i=s \ as the index set of comparisons for worker s. The reward vector r can be obtained uniquely by solving the linear system =,Hr=b, and imposing the constraint ∑j=1Krj=0 _j=1^Kr_j=0, where the matrix H is defined as: Hjj H_j =∑i∈Wjβsi2κi(t)+∑i∈Ljβsi2κi(t), = _i∈ W_j _s_i^2 _i^(t)+ _i∈ L_j _s_i^2 _i^(t), Hjl H_jl =∑i∈jl−βsi2κi(t),j≠l, = _i _jl- _s_i^2 _i^(t), j≠ l, and, the vector b is defined as: bj=∑i∈Wj12βsi+∑i∈Lj(−12βsi),b_j= _i∈ W_j 12 _s_i+ _i∈ L_j (- 12 _s_i ), where WjW_j and LjL_j denote the sets of comparisons in which item j won and lost, respectively, and jlD_jl is the set of comparisons between items j and l. This corresponds to Steps 12 and 13 of Algorithm 1. The linear system has a unique solution (Appendix A.3.2) and can be solved efficiently using conjugate gradient descent. The implementation details are provided in Appendix E. 5 Convergence Analysis In this section we show that the M-step objective can be written as a matrix sensing problem with a rank-1 parameter matrix X=⊤X= β r . This allows us to leverage standard results from low-rank matrix recovery to analyze the geometry of the Q-function and the behavior of EM. Formally, Equation (11) can be stated in the following form. min,‖(T)−‖22, _ β, r\|A( β r^T)-u\|_2^2\,, where ∈ℝNu ^N and the linear operator :ℝM×K→ℝNA ^M× K ^N as (X)=(⟨A1,X⟩,…,⟨AN,X⟩)⊤A(X)=( A_1,X ,…, A_N,X ) with matrices Ai=2ciα(−)⊤A_i=2c_iα\ e_s_i( e_w_i- e_l_i) where α=MK/2Nα= MK/2N, ci2=κi(t)c_i^2= _i^(t) for iteration t, exe_x denotes a unit vector having value as 11 at index x. More details are in Appendix B. We assume that the worker index and the two item indices (denoted by random variables B,W,B,W, and L, respectively) are independent and uniformly distributed, B∼Unif1,2,…,MB \1,2,…,M\ and W,L∼Unif1,2,…,KW,L \1,2,…,K\. The comparison outcome is then generated according to 2. Further, we assume that β∈[−1,1]β∈[-1,1] and r∈[−R,R]r∈[-R,R] for some constant R>0R>0. Under mild assumptions detailed in Appendix B, the linear operator A satisfies the Restricted Isometry Property, according to the following definition [candes2010tight]. Definition (Restricted Isometry Property (RIP)). A linear operator :ℝM×K→ℝNA:R^M× K ^N with ((X))i=⟨Ai,X⟩(A(X))_i= A_i,X satisfies the restricted isometry property on rank-r matrices, with parameter δr _r, if the following set of inequalities hold for all rank-r matrices X: (1−δr)‖X‖F2≤‖(X)‖22≤(1+δr)‖X‖F2.(1- _r)\|X\|_F^2≤\|A(X)\|_2^2≤(1+ _r)\|X\|_F^2. This gives us the following lemma. Lemma 1. Define matrix operators Ai∈ℝM×KA_i ^M× K as Ai=2ciα(−)⊤A_i=2c_iα\ e_s_i( e_w_i- e_l_i) , where α=MK/2Nα= MK/2N. Let X∈ℝM×KX ^M× K be any matrix of arbitrary rank r with X=X 1= 0, where 1 and 0 denote the all-ones and all-zeros vectors, respectively. Define the linear operator :ℝM×K→ℝNA ^M× K ^N as (X)=(⟨A1,X⟩,…,⟨AN,X⟩)⊤A(X)=( A_1,X ,…, A_N,X ) . Then, for any η>0η>0, there exists μ>0μ>0 and δ>0δ>0 such that with probability at least 1−η1-η: (1−μ)‖X‖F2≤‖(X)‖22≤(1+μ)‖X‖F2,(1-μ)\|X\|_F^2≤\|A(X)\|_2^2≤(1+μ)\|X\|_F^2\,, provided that N≥CMKδ2ln(2η),N≥ C\; MKδ^2 ( 2η ), where C>0C>0 is some constant. Then, invoking Remark 3 from park2016nonsquarematrixsensingspurious, we can state the following theorem. Theorem 1. Let (t),(t)t≥0\ β^(t), r^(t)\_t≥ 0 be the sequence of iterates generated by the EM algorithm. Suppose the M-step at each iteration t solves the optimization problem: ((t+1),(t+1))=argmin,‖(⊤)−(t)‖22( β^(t+1), r^(t+1))= _ β, r\|A( β r )-u^(t)\|_2^2 having global minimum (∗)(t+1)( θ^*)^(t+1), where ui(t)=αciu_i^(t)= αc_i. If R=0.1R=0.1, the operator A satisfies Restricted Isometry Property with μ2=μ4=0.005 _2= _4=0.005, and every local minimum (t+1)=((t+1);(t+1)) θ^(t+1)= ( β^(t+1); r^(t+1) ) satisfies: DIST((t+1),(∗)(t+1))≤12503σ1(Xt∗)⋅‖(Xt∗−X1,t∗)‖2DIST( θ^(t+1),( θ^*)^(t+1))≤ 12503 _1(X^*_t)·\|A(X^*_t-X_1,t^*)\|_2 where Xt∗=X^*_t 1= 0 and X1,t∗X_1,t^* is the best rank-1 approximation of the true high-rank matrix Xt∗X^*_t. Therefore, each EM update computes a solution whose distance to the global optimum of the corresponding M-step objective is bounded by the intrinsic rank-one approximation error of the corresponding M-step objective. Thus, the estimation error at each iteration is controlled by ‖(Xt∗−X1,t∗)‖2\|A(X^*_t-X_1,t^*)\|_2. At iteration t, if Xt∗X_t^* is exactly rank-1, the algorithm recovers the global minimum of the function Q(∣t)Q( θ θ^t). Following Proposition 2.7.1 from Bertsekas01031997, we can say the alternating minimization (AM) procedure used to solve the M-step converges to a stationary point, since our Q(∣t)Q( θ θ^t) function is differentiable and marginally convex. In our case, in each iteration of AM, we solve for β and r exactly. Therefore, Theorem 4.3 from park2016nonsquarematrixsensingspurious implies that alternating minimization escapes saddle points and converges to a local minimum with high probability, which, by Theorem 1, is close to the global minimum. Finally, since the M-step at each iteration is solved using AM, which is a monotonic procedure [prateekjain], each EM iteration monotonically increases the likelihood. Standard convergence results for EM therefore imply that the sequence of iterates converges to a stationary point of the likelihood. 6 EXPERIMENTAL ANALYSIS 6.1 Evaluation Metrics To quantitatively evaluate the performance of the proposed methods, we employ Kendall’s Tau (τ), Accuracy (accacc), and Weighted Accuracy (waccwacc). These metrics focus on the relative ordering of items rather than absolute value errors. Kendall’s Tau is a standardized rank correlation that is robust to outliers, but it treats all pairwise swaps equally. Accuracy measures the probability of correctly ordering item pairs and is easy to interpret, though it ignores the magnitude of score differences. Weighted Accuracy penalizes larger ranking errors more strongly, but its value depends on the distribution of the scores. The definitions, including their respective advantages and limitations, are summarized in Table 3 in Appendix C. we evaluate all the representative crowdsourcing methods discussed in Section 2. More details are given in D. (a) Varying N. K=100,M=500K=100,\,M=500. (b) Varying M. K=200,N=25000K=200,\,N=25000. (c) Varying K. M=20000,N=1000M=20000,\,N=1000. Figure 1: Varying N, K, M individually when β∈[−1,1]β∈[-1,1]. 6.2 Synthetic Datasets We will sample data (having N measurements) from the distribution given in Eq. (3), with K (number of items) and M (number of workers) to study their effect and we sample β uniformly from [−1,1][-1,1]. We use 10 random seeds for data sampling. Let k denote 1000 for specifying number of measurements, workers, or items. To study the effect of each variable, we vary N, K, M individually keeping the other two variables constant as follows: 1. Varying N, Keeping K and M constant: We conduct an experiment with fixed values of the number of items K=100K=100 and the number of workers M=500M=500, while varying the number of measurements N∈2k,2.5k,3k,3.5k,4k,4.5k,5k,10k,25k,50k,100kN∈\2k, 2.5k, 3k, 3.5k, 4k, 4.5k, 5k, 10k, 25k, 50k, 100k\ to examine the effect of measurement volume on various techniques. As seen in Figure 1(a), irrespective of number of measurements the performance of methods not modeling the annotator competence remains low, whereas the methods that model annotator competence while predicting rewards i.e., BoRaEM, HBTL and CrowdBT remains high. BoRaEM seems to be robust than CrowdBT in sparse measurements setting (i.e, for measurements less than 4000). 2. Varying M, Keeping N and K constant: We vary the number of workers M∈500,1k,2k,3k,4k,5k,5.5k,6k,8kM∈\500, 1k, 2k, 3k, 4k, 5k, 5.5k, 6k, 8k\ while fixing the number of items at K=200K=200 and the number of measurements at N=25kN=25k, to analyze the impact of worker scale. As seen in Figure 1(b), all methods suffers as the number of workers are increased (or the comparisons per worker get decreased). The performance decay of CrowdBT is seen to be much higher than BoRaEM and BoRaEM is quite stable. 3. Varying K, Keeping N and M constant: We vary the number of items K∈50,100,250,500,1000,1250,1500,1750,2250,2500K∈\50, 100, 250, 500, 1000, 1250, 1500, 1750, 2250, 2500\ while fixing the number of workers at M=20kM=20k and the number of measurements at N=1kN=1k, in order to study the effect of item scale. As seen in Figure 1(c), all methods suffers as the number of items are increased. The performance of BoRaEM, HBTL and CrowdBT decays after K=1000K=1000, HBTL and BoRaEM seem to degrade slower than CrowdBT under larger N regimes. Methods such as BT, BARP, and RC, which do not explicitly model annotator competence or susceptibility to bias, exhibit consistently poor performance in the presence of adversarial workers. Appendix F presents results for the regime β∈[0,1]β∈[0,1], which contains no adversarial annotators. In this setting, these methods perform comparatively better due to absence adversaries. In contrast, BoRaEM, HBTL, and CrowdBT, which explicitly model annotator competence, achieve superior performance in denser regimes where sufficient observations are available to reliably estimate the additional parameters. However, their performance exhibits a modest decline in sparse regimes, where limited data constrains accurate parameter estimation and increases susceptibility to estimation noise. Hardware details are in Appendix J. 6.3 Real Datasets Table 1: Summary statistics of the evaluation datasets. Attribute FaceAge Dataset Reading Difficulty or Passage Dataset Items (K) 9,150 472 Workers (M) 4,091 624 Comparisons (N) 250,249 11,763 Description Comparison of two face images to identify the older individual. Pairwise difficulty assessment of reading passages. Ground Truth Chronological age (years) Standardized reading levels Spammer Handling Removal based on performance on “gold" instances (large age gaps). Not specified in original study. Table 2: Benchmark results on FaceAge and Passage datasets. Accuracy and Weighted Accuracy are reported as percentages, while Kendall’s Tau is reported as a decimal. Best overall results are shown in bold. Method Accuracy (%) Weighted Accuracy (%) Kendall’s Tau Runtime (s) FaceAge Dataset Simple BT 79.01 87.20 0.5754 0.89±0.050.89± 0.05 RC 78.04 86.44 0.5582 0.07±0.030.07± 0.03 FactorBT 79.16 87.30 0.5785 169.41±1.16169.41± 1.16 BARP 79.08 87.26 0.5769 51.08±2.8551.08± 2.85 CrowdBT 79.17 87.31 0.5787 138.16±3.86138.16± 3.86 HTCV 79.17 87.33 0.5786 41.31±1.4141.31± 1.41 HBTL 79.20 87.35 0.5793 83.10±3.3283.10± 3.32 BoRaEM (Ours) 79.21 87.36 0.5795 6.55±0.436.55± 0.43 Passage Dataset Simple BT 68.04 74.33 0.3407 1.39±0.131.39± 0.13 RC 65.22 71.30 0.2892 0.02±0.040.02± 0.04 FactorBT 69.47 75.58 0.3676 100.85±2.82100.85± 2.82 BARP 68.19 74.51 0.3435 32.05±2.3832.05± 2.38 CrowdBT 70.02 75.96 0.3779 84.25±1.8484.25± 1.84 HTCV 69.30 75.30 0.3643 46.19±1.8346.19± 1.83 HBTL 69.53 75.46 0.3688 77.72±2.8577.72± 2.85 BoRaEM (Ours) 69.59 75.56 0.3698 8.28±0.718.28± 0.71 Figure 2: Kendall’s Tau (τ) of different ranking methods under varying proportions of spammers for FaceAge dataset. For evaluating crowdsourcing methods on real-world data, we use two datasets: the FaceAge dataset (IMDB-WIKI-SbS)333http://w-personal.umich.edu/~kevynct/datasets/wsdm_rankagg_2013_readability_crowdflower_data.csv [pavlichenko2021imdb] and the Reading Difficulty (Passage) dataset444https://github.com/Toloka/IMDB-WIKI-SbS/tree/main [chen2013pairwise]. The dataset statistics are summarized in Table 1. The FaceAge dataset [pavlichenko2021imdb] is specifically tailored for large scale evaluation of various crowdsourcing methods and is a widely adopted benchmark for crowdsourced pairwise comparison experiments [ferrara2024bias]. On the FaceAge dataset, BoRaEM achieves the best performance, followed by HBTL and CrowdBT. The Passage dataset is comparatively much smaller. On the Passage dataset, we observe that all methods achieve relatively low performance, with Kendall’s tau values ranging from 0.29 to 0.38. This can be attributed to the small size of the dataset and the inherent difficulty of modeling preferences in reading difficulty, which is a cognitively complex and subjective task. On this dataset, CrowdBT achieves the highest performance, followed by HBTL and BoRaEM. Table 2 presents the results on the FaceAge and Passage datasets. BoRaEM, HBTL, HTCV, FactorBT and CrowdBT remain competitive across both datasets. These results highlight the importance of jointly modeling worker competence and item rewards. 6.4 Spammer Addition Analysis We evaluate robustness under adversarial conditions by injecting synthetic spammers commonly studied in the crowdsourcing literature. Specifically, we consider four archetypes: (i) random choosers, who respond uniformly at random without regard to item quality [vuurens2011much]; (i) position-biased spammers, who systematically favor one side of the comparison (e.g., always selecting the left or right option), also referred to as uniform spammers in prior work [day1969position, xu2016false, vuurens2011much]; (i) malicious adversaries, who intentionally provide inverted or misleading responses to degrade aggregation quality [kleindessner2018crowdsourcing, steinhardt2016avoiding]; and (iv) a combined setting, where all aforementioned spammer types are present in equal proportion, reflecting a more realistic and heterogeneous adversarial environment. For fairness, each injected spammer is assigned the same expected number of comparisons as an original worker, ensuring performance differences arise from behavioral characteristics rather than annotation volume. 6.4.1 Results We mainly focus on the FaceAge dataset, as it is a widely adopted benchmark for crowdsourcing [pavlichenko2021imdb, ferrara2024bias]. Results on the Passage dataset are provided in Appendix H. Figure 2 presents the performance of different crowdsourcing methods in terms of Kendall’s tau (τ) as the proportion of various types of spammers in the FaceAge dataset is varied. The results are averaged over 10 random seeds for generating spammers and 10 random seeds for random initializations. Similar trends are observed in terms of accuracy (accacc) and weighted accuracy (waccwacc) (see Appendix G). We add spammers following the procedure of bugakova2019aggregation. Specifically, if there are originally M annotators, we add 0.1M0.1M, 0.2M0.2M, 0.4M0.4M, 0.6M0.6M, and 0.8M0.8M spammers according to the specified proportions. As the spammer proportion increases from 0% to 44.4%, BoRaEM, HBTL, HTCV consistently perform the best, followed by CrowdBT and FactorBT, while BARP, BT, and RC exhibit substantial performance drops. These results highlight the importance of explicitly modeling annotator competencies in crowdsourcing. Further, we have performed ablations over intialization and hyperparameters, and the results are detailed in Appendix I. 7 CONCLUSION In this work, we addressed the problem of rank aggregation from pairwise comparisons in the presence of workers with varying competencies. We adapt the Boltzmann-rational model for modeling each pairwise comparison. We then use the Polya-Gamma random variable augmentation to simplify the logistic function to an exponential form enabling a simple and efficient EM formulation. We achieve tractable inference with theoretical convergence guarantees by treating this problem as a matrix sensing problem. Extensive experiments on both synthetic and real-world datasets demonstrate the advantages of using our algorithm over several baselines and the algorithm remains robust even in the presence of a large fraction of spammers, highlighting its effectiveness for reliable rank aggregation in crowdsourced settings. Future work includes designing models for robust RLHF that can jointly predict item feature importance vectors while effectively identifying and mitigating the influence of spammers. contributions Arpit Agarwal conceived the idea of modeling annotator competence using the Boltzmann Rational Model, identified its connection to the matrix sensing problem, and led the manuscript preparation. Kaustubh Shivshankar Shejole proposed the use of Pólya-Gamma augmentation, developed the method, conducted empirical studies, and generated the experimental visualizations. Tanish Agarwal developed the theoretical foundations of the proposed approach under the guidance of Arpit Agarwal and Avishek Ghosh. Acknowledgements.We would like to thank the Organizing Committee, Area Chairs, Senior Area Chairs, Meta-Reviewer, and the anonymous reviewers of UAI 2026 for their valuable feedback, constructive suggestions, and insightful comments, which helped improve the quality and presentation of this work. The first author gratefully acknowledges Kush Mangukiya for contributions during the early stages of this research and Pratham Shah, Manish Kumar for their continuous support and encouragement. References Finding the Signal in the Spam (Supplementary Material) Appendix A More Details on BoRaEM Derivation A.1 Pólya-Gamma Random Variables Following polson2013bayesian, if ω is a Pólya-Gamma distributed random variable with parameters b>0b>0 and c∈ℝc , denoted ω∼PG(b,c)ω (b,c), then it is equal in distribution to an infinite weighted sum of gamma random variables: ω=d12π2∑k=1∞gk(k−12)2+c24π2,gk∼Gamma(b,1).ω d= 12π^2\, _k=1^∞ g_k (k- 12 )^2+ c^24π^2\,, g_k (b,1). Here, “=d d=” denotes equality in distribution, and gk∼Gamma(b,1)g_k (b,1) are independent Gamma random variables. Note that this is not the density function. Instead, equality in distribution means that the PG random variable on the left-hand side has the same cumulative distribution function as the random variable on the right-hand side. polson2013bayesian showed that all finite moments of ω admit closed form expressions; in particular, [ω]=b2ctanh(c2).E[ω]= b2c\, ( c2 ). (13) They further proved the following two identities for ω∼PG(b,0)ω (b,0), which are central to Pólya-Gamma data augmentation for logistic models: 1. For any real ψ, one has (eψ)a(1+eψ)b=2−beκψ∫0∞e−ωψ2/2p(ω∣b,0)ω, (e^ψ)^a(1+e^ψ)^b=2^-be^κψ _0^∞e^-ωψ^2/2\,p(ω b,0)\,dω, (14) where κ=a−b2κ=a- b2, and p(ω∣b,0)p(ω b,0) is the density of a PG(b,0)PG(b,0) random variable. 2. The conditional density of ω given ψ is again Pólya–Gamma with parameters b and ψ, that is, (ω∣ψ)∼PG(b,ψ).(ω ψ) (b,ψ). (15) A.2 Log Likelihood Derivation Let us consider the problem formulation given in Section 3, where there is a set of K items, and one observes outcomes of pairwise comparisons between these items, elicited from a set of M crowdworkers. Specifically, we are given a dataset =(wi,li,si)i=1ND=\(w_i,l_i,s_i)\_i=1^N where there are N pairwise comparisons; the i-th comparison is performed by worker si∈[M]s_i∈[M] where wi∈[K]w_i∈[K] is the winning item and li∈[K]l_i∈[K] is the losing item such that lil_i is not equal to wiw_i. Consider Equation (14). Let ηi=βsi(rwi−rli) _i= _s_i(r_w_i-r_l_i). For a=1a=1, b=1b=1, and ψ=ηiψ= _i, we have κ=12κ= 12 and p(ωi)=PG(ωi∣1,0)p( _i)=PG( _i 1,0), which allows us to write 11+e−ηi=eηi1+eηi=12eηi/2∫0∞exp(−ωiηi22)p(ωi)ωi 11+e^- _i= e _i1+e _i= 12e _i/2 _0^∞ (- _i\, _i^22 )\,p( _i)\,d _i (16) Let =r, θ=\r, β\. Therefore, referring to Equation (9) and (16), we can write each term of the likelihood as p(wi,li,si|) p(w_i,l_i,s_i| θ) =11+exp−βsi(rwi−rli) = 11+ \- _s_i(r_w_i-r_l_i) \ (17) =12eηi/2∫0∞exp(−ωiηi22)p(ωi)ωi = 12e _i/2 _0^∞ (- _i _i^22 )\,p( _i)\,d _i =12∫0∞exp(ηi2−ωiηi22)p(ωi)ωi = 12 _0^∞ ( _i2- _i _i^22 )p( _i)\,d _i which induces the augmented joint density p(wi,li,si,ωi|) p(w_i,l_i,s_i, _i| θ) ∝exp(−ωiηi22+ηi2)p(ωi), (- _i _i^22+ _i2 )p( _i), Taking the logarithm and ignoring additive constants that do not depend on θ, we get the log-likelihood for the ithi^th comparison as log(p(wi,li,si,ωi|)) (p(w_i,l_i,s_i, _i| θ)) ∝log[exp(12ηi−ωi2ηi2)p(ωi)]+const [ ( 12 _i- _i2\, _i^2 )\,p( _i) ]+const (19) =−ωiηi22+ηi2, =- _i _i^22+ _i2, =12(βsi(rwi−rli))−ωi2(βsi(rwi−rli))2. = 12\, ( _s_i(r_w_i-r_l_i) )- _i2\, ( _s_i(r_w_i-r_l_i) )^2. (20) Summing over all N comparisons in D gives the complete data log-likelihood, given all ωi _i, ℓ(,ωi;)=12∑i∈[N][βsi(rwi−rli)−ωi(βsi(rwi−rli))2], ( θ, _i;D)= 12 _i∈[N] [\, _s_i(r_w_i-r_l_i)- _i\, ( _s_i(r_w_i-r_l_i) )^2 ], (21) up to additive constants independent of θ. A.3 Expectation-Maximization A.3.1 E-Step Given the current parameter estimate θ(t)θ^(t) and let ω denote the set of all wiw_i for i=1i=1 to N comparisons, we compute Q(θ∣θ(t))=ω∣,θ(t)[ℓ(,;)]=12∑(wi,li,si)∈ωi∣θ(t)[βsi(rwi−rli)−ωiβsi2(rwi−rli)2].Q (θ θ^(t) )=E_\,ω ,\,θ^(t) [\, ( θ, ω;D) ]= 12 _(w_i,l_i,s_i) E_\, _i θ^(t) [\, _s_i\,(r_w_i-r_l_i)- _i\, _s_i^2\,(r_w_i-r_l_i)^2 ]. Noting that the only term depending on ωi _i is ωiβsi2(rwi−rli)2 _i\, _s_i^2\,(r_w_i-r_l_i)^2, we define κi(t)=[ωi∣θ(t)]. _i^(t)=E [ _i θ^(t) ]. Then, Q(θ∣θ(t))=12∑(wi,li,si)∈[βsi(rwi−rli)−κi(t)βsi2(rwi−rli)2].Q (θ θ^(t) )= 12 _(w_i,l_i,s_i) [ _s_i\, (r_w_i-r_l_i )- _i^(t)\, _s_i^2\, (r_w_i-r_l_i )^2 ]. (22) Computing κi(t) _i^(t). At iteration t, we compute the conditional expectation κi(t)=[ωi∣ηi(t)] _i^(t)=E[ _i _i^(t)] for the latent variables ωi∼PG(1,ηi(t)) _i (1, _i^(t)), where ηi(t)=βsi(t)(rwi(t)−rli(t)) _i^(t)= _s_i^(t)(r_w_i^(t)-r_l_i^(t)). Invoking Eq. (13) yields: κi(t)=(ηi(t)≠0)tanh(ηi(t)/2)2ηi(t)+(ηi(t)=0)14. _i^(t)=I ( _i^(t)≠ 0 ) ( _i^(t)/2 )2 _i^(t)+I ( _i^(t)=0 ) 14. (23) A.3.2 M-Step Note that while updating β, r is kept fixed and vice-versa. Updating β Q(θ∣θ(t))=∑(wi,li,si)∈[12βsi(rwi−rli)−12κi(t)βsi2(rwi−rli)2],Q (θ θ^(t) )= _(w_i,l_i,s_i) [\, 12\, _s_i\,(r_w_i-r_l_i)- 12\, _i^(t)\, _s_i^2\,(r_w_i-r_l_i)^2 ], where κi(t)=[ωi∣θ(t)] _i^(t)=E [ _i θ^(t) ]. We first maximize Q with respect to each βs _s. Let Is=i:si=sI_s= \\,i:s_i=s \ denote the index set of comparisons done by worker s. Let di=rwi−rlid_i=r_w_i-r_l_i denote the reward difference between the preferred item wiw_i and the losing item lil_i, evaluated at the current item reward estimates within the inner iterations of the M-step for all i∈Isi∈ I_s. Then, the fraction of Q depending on βs _s is Qk(βs)=∑i∈Is[12βsdi−12κi(t)βs2di2].Q_k ( _s )= _\,i∈ I_s [ 12\, _s\,d_i- 12\, _i^(t)\, _s^2\,d_i^2 ]. (24) For maximization, we require ∂Qk∂βs=0 ∂ Q_k∂ _s=0, that is, ∂Qk∂βs ∂ Q_k∂ _s =∑i∈Is[12di−κi(t)βsdi2]=0, = _\,i∈ I_s [ 12\,d_i- _i^(t)\, _s\,d_i^2 ]=0, (25) which on rearranging gives 12∑i∈Isdi−βs∑i∈Isκi(t)di2=0⟹βs=12∑i∈Isdi∑i∈Isκi(t)di2=∑i∈Isdi2∑i∈Isκi(t)di2. 12 _\,i∈ I_sd_i- _s _\,i∈ I_s _i^(t)\,d_i^2=0 _s= 12 _\,i∈ I_sd_i _\,i∈ I_s _i^(t)\,d_i^2= _\,i∈ I_sd_i2\, _\,i∈ I_s _i^(t)\,d_i^2. Updating r We now maximize Q with respect to each rkr_k. Define the index sets Wj=i:wi=j and Lj=i:li=jW_j=\\,i:w_i=j\ and L_j=\\,i:l_i=j\ where WjW_j denotes the set of comparisons where item j won, and LjL_j denotes the set of comparisons where item j lost. Then, ∂Q∂rk=[∑i∈Wj[12βsi−βsi2κi(t)(rk−rli)]+∑i∈Lj[−12βsi+βsi2κi(t)(rwi−rk)]]=0. ∂ Q∂ r_k= [ _\,i∈ W_j [\, 12\, _s_i- _s_i^2\, _i^(t)\, (r_k-r_l_i ) ]+ _\,i∈ L_j [\,- 12\, _s_i+ _s_i^2\, _i^(t)\, (r_w_i-r_k ) ] ]=0. (26) which on rearranging gives rk=AkBkr_k= A_kB_k where Ak A_k =∑i∈Wj[12βsi+βsi2κi(t)rli]+∑i∈Lj[−12βsi+βsi2κi(t)rwi], and, = _\,i∈ W_j [ 12\, _s_i+ _s_i^2\, _i^(t)\,r_l_i ]+ _\,i∈ L_j [- 12\, _s_i+ _s_i^2\, _i^(t)\,r_w_i ], and, Bk B_k =∑i∈Wjβsi2κi(t)+∑i∈Ljβsi2κi(t). = _\,i∈ W_j _s_i^2\, _i^(t)+ _\,i∈ L_j _s_i^2\, _i^(t). Observe that for fixed β, these update equations can be written as a linear system H=H r=b as follows. Bkrk−∑i∈Wjβsi2κirli−∑i∈Ljβsi2κirwi=∑i∈Wj(12βsi)+∑i∈Lj(−12βsi)B_kr_k- _i∈ W_j _s_i^2 _ir_l_i- _i∈ L_j _s_i^2 _ir_w_i= _i∈ W_j ( 12 _s_i )+ _i∈ L_j (- 12 _s_i ) The coefficient matrix H and vector b are given by Hjl=∑i∈jl−βsi2κi(t),j≠lBj,j=lH_jl= cases _i _jl- _s_i^2 _i^(t), j≠ l\\ B_j, j=l\\ cases and bj=∑i∈Wj12βsi+∑i∈Lj(−12βsi)b_j= _i∈ W_j 12 _s_i+ _i∈ L_j (- 12 _s_i ) By construction, H is symmetric, Hjl≤0H_jl≤ 0 for j≠lj≠ l, and Hjj=∑l≠j|Hjl|H_j= _l≠ j|H_jl|. Thus, H is diagonally dominant with non-negative diagonal entries, and it is positive semidefinite. We can further observe that H=H 1= 0. Now, suppose x satisfies H=Hx= 0 and let j be such that xj=maxixix_j= _ix_i. Using the zero row sum property, we get that 0=(H)j=∑l≠jHjl(xl−xj)0=(Hx)_j= _l≠ jH_jl(x_l-x_j) Since Hjl≤0H_jl≤ 0 and xl−xj≤0x_l-x_j≤ 0, each term is nonnegative, hence each term must be zero. Therefore, xj=xlx_j=x_l whenever Hjl≠0H_jl≠ 0. If the comparison graph induced by the nonzero off-diagonal entries of H is connected, all entries of x are equal, hence =cx=c 1. The centering constraint ⊤=0 1 x=0 forces c=0c=0, so =x= 0. Therefore, H is positive definite on the mean-zero subspace, and the constrained linear system admits a unique solution. Appendix B Convergence Analysis Let D be the dataset of pairwise comparisons, N=||N=|D| denotes the number of datapoints, K=||K=| r| the number of items, and M=||M=| β| the number of workers. For each worker j∈[M]j∈[M], we constrain βj∈[−1,1] _j∈[-1,1]. We assume that the rewards are bounded as ri∈[−R,R]r_i∈[-R,R] for all items l∈[K]l∈[K] for some fixed constant R>0R>0. In each iteration t of the EM algorithm, we independently set ci2=κi(t)=12ηi(t)tanh(ηi(t)2),where ηi(t)=βsi(t)(rwi(t)−rli(t)).c_i^2= _i^(t)= 12 _i^(t) ( _i^(t)2 ), _i^(t)= _s_i^(t) (r_w_i^(t)-r_l_i^(t) ). Since βsi∈[−1,1] _s_i∈[-1,1] and rwi,rli∈[−R,R]r_w_i,r_l_i∈[-R,R], it follows that |ηi|≤2R | _i |≤ 2R. The function f(x)=tanh(x/2)2xf(x)= (x/2)2x is positive, even, and monotonically decreasing for x>0x>0. Therefore, there exists a constant ζ=14Rtanh(R)>0ζ= 14R (R)>0 such that ci2∈[ζ,1/4]c_i^2∈[ζ,1/4] for all i. For each worker s, let GsG_s denote the comparison graph induced by the worker’s pairwise comparisons sD_s. We assume that GsG_s is acyclic. Finally, we assume that the worker index and the two item indices (denoted by random variables B,W,B,W, and L, respectively) are independent and uniformly distributed as B∼Uniform([M]),W,L∼Uniform([K])B ([M]),W,L ([K]). The comparison outcome is then generated according to 2. Under this model, we can show the following lemma. Lemma 1. Define matrix operators Ai∈ℝM×KA_i ^M× K for each i∈[N]i∈[N] as follows: Ai⊤=2ciαsi[00⋯0⋯0⋮00⋯+1⋯0⋮00⋯−1⋯0⋮00⋯0⋯0]⋮wi⋮li⋮A_i =2c_iα array[]c@c& array[]c c c c c&&s_i&& array\\ & bmatrix0&0&·s&0&·s&0\\ & && && \\ 0&0&·s&+1&·s&0\\ & && && \\ 0&0&·s&-1&·s&0\\ & && && \\ 0&0&·s&0&·s&0 bmatrix array[]c\\[-8.61108pt] \\ w_i\\ \\ l_i\\ array array where +1+1 appears at position (wi,si)(w_i,s_i), −1-1 at position (li,si)(l_i,s_i), with all other entries equal to 0, and α=MK/2Nα= MK/2N. Equivalently, Ai=2ciα(−)⊤A_i=2c_iα\ e_s_i( e_w_i- e_l_i) . Let X∈ℝM×KX ^M× K be any matrix of arbitrary rank r with X=X 1= 0, where 1 and 0 denote the all-ones and all-zeros vectors, respectively. Define the linear operator :ℝM×K→ℝNA ^M× K ^N as (X)=(⟨A1,X⟩,…,⟨AN,X⟩)⊤A(X)=( A_1,X ,…, A_N,X ) . Then, for any η>0η>0, there exists μ>0μ>0 and δ>0δ>0 such that with probability at least 1−η1-η: (1−μ)‖X‖F2≤‖(X)‖22≤(1+μ)‖X‖F2,(1-μ)\|X\|_F^2≤\|A(X)\|_2^2≤(1+μ)\|X\|_F^2\,, provided that N≥CMKδ2ln(2η),N≥ C\; MKδ^2 ( 2η ), where C>0C>0 is some constant. Proof. We first show that A satisfies RIP in expectation, and we subsequently bound the deviation of ‖(X)‖22\|A(X)\|_2^2 from its expectation to achieve the desired result. Consider ⟨Ai,X⟩2=4ci2α2(Xsi,wi−Xsi,li)2 A_i,X ^2=4c_i^2α^2(X_s_i,w_i-X_s_i,l_i)^2. Under uniform sampling of B,W,LB,W,L and the assumption X=X 1= 0, we have that B,W,L⟨Ai,X⟩2=B[W,L[⟨Ai,X⟩2|B=si]]E_B,W,L A_i,X ^2=E_B [E_W,L [ A_i,X ^2|B=s_i ] ] Conditioned on B=siB=s_i, we get the following expression. W,L[⟨Ai,X⟩2] _W,L [ A_i,X ^2 ] =4ci2α2W,L[(Xsi,wi−Xsi,li)2] =4c_i^2α^2E_W,L [(X_s_i,w_i-X_s_i,l_i)^2 ] =4ci2α2(W,LXsi,wi2+W,LXsi,li2−2W,LXsi,wiXsi,li) =4c_i^2α^2 (E_W,LX_s_i,w_i^2+E_W,LX_s_i,l_i^2-2E_W,LX_s_i,w_iX_s_i,l_i ) =8ci2α2(W,LXsi,wi2−(W,LXsi,wi)2) =8c_i^2α^2 (E_W,LX_s_i,w_i^2- (E_W,LX_s_i,w_i )^2 ) Now, note that W,LXsi,wi2=1K∑j=1KXsi,j2andW,LXsi,wi=1K∑j=1KXsi,j,E_W,LX_s_i,w_i^2= 1K _j=1^KX_s_i,j^2 _W,LX_s_i,w_i= 1K _j=1^KX_s_i,j, and define Sb=∑j=1KXb,jS_b= _j=1^KX_b,j and Tb=∑j=1KXb,j2T_b= _j=1^KX_b,j^2. Observe that Sb=0S_b=0 for every b∈[M]b∈[M] since X=X 1= 0, which implies that W,LXsi,wi=0,and W,L[⟨Ai,X⟩2]=4ci2α2(2KTsi)E_W,LX_s_i,w_i=0,and E_W,L[ A_i,X ^2]=4c_i^2α^2( 2KT_s_i). Finally, B,W,L⟨Ai,X⟩2=1M(4ci2α2⋅2K∑j=1MTsj)=1M(4ci2α2⋅2K∑j=1M∑i=1KXji2)=1M4ci2α22K‖X‖F2=4ci2N‖X‖F2,E_B,W,L A_i,X ^2= 1M (4c_i^2α^2· 2K _j=1^MT_s_j )= 1M (4c_i^2α^2· 2K _j=1^M _i=1^KX_ji^2 )= 1M4c_i^2α^2 2K\|X\|_F^2= 4c_i^2N\|X\|_F^2, (27) where the last equality follows from the definition of α. Summing over all N datapoints, we obtain: B,W,L‖(X)‖22=∑i=1NB,W,L⟨Ai,X⟩2=‖X‖F2∑i=1N4ci2NE_B,W,L\|A(X)\|_2^2= _i=1^NE_B,W,L A_i,X ^2=\|X\|_F^2 _i=1^N 4c_i^2N Note that 4ζ≤4ci2≤14ζ≤ 4c_i^2≤ 1. Take ϵ=1−4ζ>0ε=1-4ζ>0, such that (1−ϵ)‖X‖F2≤4ζ‖X‖F2≤B,W,L‖(X)‖22≤‖X‖F2≤(1+ϵ)‖X‖F2.(1-ε)\|X\|_F^2≤ 4ζ\|X\|_F^2 _B,W,L\|A(X)\|_2^2≤\|X\|_F^2≤(1+ε)\|X\|_F^2. (28) Now, we will bound the deviation of ‖(X)‖22\|A(X)\|_2^2 from its expectation. Define random variables Yi=⟨Ai,X⟩2Y_i= A_i,X ^2; then ∑i=1NYi=‖(X)‖22 _i=1^NY_i=\|A(X)\|_2^2. We will first bound YiY_i. Note that Yi≥0Y_i≥ 0. Further, Yi=⟨Ai,X⟩2=4ci2α2(Xsi,wi−Xsi,li)2≤4ci2α2(|Xsi,wi|+|Xsi,li|)2.Y_i= A_i,X ^2=4c_i^2α^2(X_s_i,w_i-X_s_i,l_i)^2≤4c_i^2α^2(|X_s_i,w_i|+|X_s_i,l_i|)^2. Since each Xij2≤‖X‖F2X_ij^2≤\|X\|_F^2, we have that |Xij|≤‖X‖F|X_ij|≤\|X\|_F, and consequently, 0≤Yi≤16ci2α2‖X‖F20≤ Y_i≤16c_i^2α^2\|X\|_F^2 (29) Similarly, Yi2≤maxYi⋅YiEY_i^2≤ Y_i\,·\,EY_i. We know from Equation (27) that Yi=⟨Ai,X⟩2=4ci2N‖X‖F2EY_i=E A_i,X ^2= 4c_i^2N\|X\|_F^2, which gives Yi2≤16ci2α2‖X‖F2⋅4ci2N‖X‖F2=64ci4α2N‖X‖F4≤64ci2α2N‖X‖F4,EY_i^2≤16c_i^2α^2\|X\|_F^2\ ·\ 4c_i^2N\|X\|_F^2= 64c_i^4α^2N\|X\|_F^4≤ 64c_i^2α^2N\|X\|_F^4\;, (30) where the final inequality follows from the fact that ci<1c_i<1. Define random variables Zi=Yi−EYiZ_i=Y_i-\ EY_i. Then, ZiZ_i are independent and zero-mean, |Zi|≤|Yi|+|Yi|≤2|Yi|≤32ci2α2‖X‖F2|Z_i|≤|Y_i|+|EY_i|≤ 2|Y_i|≤32c_i^2α^2\|X\|_F^2, and, Zi2≤Yi2≤64ci2α2N‖X‖F4EZ_i^2 _i^2≤ 64c_i^2α^2N\|X\|_F^4. Then, using Bernstein’s inequality on ZiZ_i, we get, Pr(|∑i=1NZi|>δ‖X‖F2)≤2exp(−δ2‖X‖F4/2N⋅64ci2α2N‖X‖F4+13⋅16δci2α2‖X‖F4) ( | _i=1^NZ_i |>δ\|X\|_F^2 )≤ 2 (- δ^2\|X\|_F^4/2N· 64c_i^2α^2N\|X\|_F^4+ 13· 16δ c_i^2α^2\|X\|_F^4 ) ≤2exp(−δ2‖X‖F480ci2α2‖X‖F4) ≤ 2 (- δ^2\|X\|_F^480c_i^2α^2\|X\|_F^4 ) =2exp(−δ280ci2α2). =2 (- δ^280c_i^2α^2 ). For this to be bounded above by η, we need 2exp(−δ280ci2α2)=2exp(−δ2N40ci2MK)≤η⟹N≥CMKδ2log(2η),2 (- δ^280c_i^2α^2 )=2 (- δ^2N40c_i^2MK )≤η N≥ C MKδ^2 ( 2η ), for some constant C>0C>0. Therefore, with probability at least 1−η1-η, we get that |∑i=1NZi|≤δ‖X‖F2 | _i=1^NZ_i |≤δ\|X\|_F^2 ⟹|∑i=1N⟨Ai,X⟩2−∑i=1N⟨Ai,X⟩2|≤δ‖X‖F2 | _i=1^N A_i,X ^2- _i=1^NE A_i,X ^2 |≤δ\|X\|_F^2 ⟹|‖(X)‖22−‖(X)‖22|≤δ‖X‖F2 |\|A(X)\|_2^2-E\|A(X)\|_2^2 |≤δ\|X\|_F^2 ⟹(1−μ)‖X‖F2≤‖(X)‖22≤(1+μ)‖X‖F2 (1-μ)\|X\|_F^2≤\|A(X)\|_2^2≤(1+μ)\|X\|_F^2 with μ=ϵ+δμ=ε+δ. ∎ We now show that the M-step optimization problem can be written as a matrix sensing problem. Recall that the M-step at iteration t maximizes Q(∣t) Q( θ θ^t) =12∑i=1N(βsi(rwi−rli)−ci2βsi2(rwi−rli)2) = 12 _i=1^N ( _s_i (r_w_i-r_l_i )-c_i^2 _s_i^2 (r_w_i-r_l_i )^2 ) =−12∑i=1N(ci2βsi2(rwi−rli)2−2(12ci)ciβsi(rwi−rli)+14ci2−14ci2) =- 12 _i=1^N (c_i^2 _s_i^2 (r_w_i-r_l_i )^2-2 ( 12c_i )c_i _s_i (r_w_i-r_l_i )+ 14c_i^2- 14c_i^2 ) =−12∑i=1N(ciβsi(rwi−rli)−12ci)2+∑i=1N18ci2 =- 12 _i=1^N (c_i _s_i (r_w_i-r_l_i )- 12c_i )^2+ _i=1^N 18c_i^2 Therefore, argmaxQ(∣t) _ θQ( θ θ^t) =argmax(−12∑i=1N(ciβsi(rwi−rli)−12ci)2+∑i=1N18ci2) = _ θ (- 12 _i=1^N (c_i _s_i (r_w_i-r_l_i )- 12c_i )^2+ _i=1^N 18c_i^2 ) =argmin∑i=1N(ciβsi(rwi−rli)−12ci)2 = _ θ _i=1^N (c_i _s_i (r_w_i-r_l_i )- 12c_i )^2 =argmin∑i=1N(2ciαβsi(rwi−rli)−αci)2 = _ θ _i=1^N (2c_iα\, _s_i (r_w_i-r_l_i )- αc_i )^2 =argmin∑i=1N(⟨Ai,⊤⟩−αci)2 = _ θ _i=1^N ( A_i, β r - αc_i )^2 where the second equality follows from the fact that cic_i are constants, and the fourth equality follows from the definition of matrices AiA_i. Define the vector (t)∈ℝNu^(t) ^N with ui(t)=αciu_i^(t)= αc_i. Then, argmaxQ(∣t)=argmin‖(⊤)−(t)‖22, _ θQ( θ θ^t)= _ θ \|A( β r )-u^(t) \|_2^2, We now invoke existing matrix sensing guarantees. Let Xt∗∈ℝM×KX^*_t ^M× K denote the underlying true matrix and X1,t∗X_1,t^* denote its best rank-1 approximation. Define the distance between a factorization (,)( β, r) and X1,t∗X_1,t^* as DIST(,;X1,t∗)=min(∗,∗)∈1∗‖[]−[∗]‖FDIST( β, r;X_1,t^*)= _( β^*, r^*) _1^* \| bmatrix β\\ r bmatrix- bmatrix β^*\\ r^* bmatrix \|_F where 1∗X_1^* is the set of balanced rank-one factorizations for X1,t∗X_1,t^*, defined as 1∗=(∗,∗):∗∈ℝM,∗∈ℝK,∗⊤=X1,t∗,σ1(∗)=σ1(∗)=σ1(X1,t∗)1/2,.X_1^*= \( β^*, r^*): β^* ^M, r^* ^K, β^* r^* =X_1,t^*, _1( β^*)= _1( r^*)= _1(X_1,t^*)^1/2, \. Then, restating Remark 3 from park2016nonsquarematrixsensingspurious, Remark 1. (High-rank matrix sensing) Suppose that Xt∗X^*_t is of arbitrary rank, and let X1,t∗X_1,t^* denote its best rank-1 approximation. Let =(X∗)b=A(X^*) and let (∗,∗)( β^*, r^*) be a balanced factorization of X1,t∗X_1,t^*. If 0≤δ2≤δ4<0.0050≤ _2≤ _4<0.005, then, for any local minimum (,)( β, r) the distance to (∗,∗)( β^*, r^*) is bounded by DIST(,;Xt∗)≤12503σ1(Xt∗)⋅‖(Xt∗−X1,t∗)‖2. DIST( β, r;X^*_t)≤ 12503 _1(X^*_t)·\|A(X^*_t-X_1,t^*)\|_2. Using this, we can give the following theorem. See 1 Proof. Define Xt∗∈ℝM×KX^*_t ^M× K as any matrix consistent with the equations [(Xt∗)]i=ui(t)[A(X^*_t)]_i=u_i^(t). From the definition of the sensing matrices AiA_i, this implies ui(t)=αci=2ciα(Xt,si,wi∗−Xt,si,li∗)⟹Xt,si,wi∗−Xt,si,li∗=12ci2u_i^(t)= αc_i=2c_iα(X^*_t,\s_i,w_i\-X^*_t,\s_i,l_i\) X^*_t,\s_i,w_i\-X^*_t,\s_i,l_i\= 12c_i^2 Under the assumption that for each worker s, the comparison graph is acyclic, and with the additional constraint that ∑l∈[K]Xt,si,l∗=0 _l∈[K]X^*_t,\s_i,l\=0, this linear system is always consistent. Thus, a solution Xt∗X^*_t exists. Now, our operator A satisfies RIP on matrices X of arbitrary rank r with X=X 1= 0. Because the coefficients ci2∈[ζ,1/4]c_i^2∈[ζ,1/4], we can define R=0.1R=0.1, such that ζ>0.249ζ>0.249 and ε<0.004 <0.004 and the RIP constants μr<0.005 _r<0.005, independent of r. Note that Xt∗X^*_t may not be rank-1. So, we denote by X1,t∗X_1,t^* its best rank-1 approximation. Then, by Remark 1, any local minimum of the objective min,‖(⊤)−(t)‖22 _ β, r \|A( β r )-u^(t) \|_2^2 satisfies DIST(,;Xt∗)≤12503σ1(Xt∗)‖(Xt∗−X1,t∗)‖2.DIST( β, r;X^*_t)≤ 12503 _1(X^*_t)\|A(X^*_t-X_1,t^*)\|_2. ∎ Appendix C Evaluation Metrics: More details Table 3: Summary of Quantitative Metrics. Here, gig_i and sis_i denote the ground truth and predicted quality scores for item i∈Dgi∈ D_g, respectively. Metric Formulation Pros Cons Kendall’s Tau C−D12k(k−1) C-D 12k(k-1) Standardized rank correlation range; robust to outliers. Treats all pairwise swaps with equal importance. Accuracy ∑(gi>gj∧si>sj)∑(gi>gj) (g_i>g_j s_i>s_j) (g_i>g_j) Highly interpretable; measures correct ordering probability. Does not account for the magnitude of score gaps. Weighted Acc. ∑wij(gi>gj∧si>sj)∑wij(gi>gj) Σ w_ij\,I(g_i>g_j s_i>s_j)Σ w_ij\,I(g_i>g_j) Penalizes significant ranking errors more heavily. Metric value depends on the score distribution. Note: C and D are concordant and discordant pairs; k=|Dg|k=|D_g| is the number of items; and wij=|gi−gj|w_ij=|g_i-g_j| represents the weight of the difference between ground truth scores. Table 3 summarizes the quantitative metrics used to evaluate the predicted item rewards against the ground truth. Kendall’s Tau measures the rank correlation between item pairs, providing a standardized indication of agreement while treating all swaps equally. Accuracy computes the probability of correctly ordering item pairs, offering intuitive interpretability but ignoring the magnitude of differences. Weighted Accuracy extends this by penalizing larger ranking errors more heavily, making it sensitive to the score gaps between items. Together, these metrics provide complementary perspectives on ranking performance. Appendix D Crowdsourcing Methods Considered In this section, we evaluate all the representative crowdsourcing methods discussed in Section 2. These include classical approaches such as the Bradley–Terry–Luce (BT) model bradley1952rank, RankCentrality (RC) negahban2012iterative, CrowdBT chen2013pairwise, FactorBT bugakova2019aggregation, and BARP ferrara2024bias. The implementations of RC, CrowdBT, and BARP are obtained from the publicly available repository released by ferrara2024bias555https://github.com/Ambress92/Bias-Aware-Ranker-from-Pairwise-comparisons, while BT666https://github.com/lucasmaystre/choix and FactorBT777https://github.com/Toloka/crowd-kit are sourced from their respective open-source repositories. These methods differ in how they model pairwise comparisons and annotator reliability. All methods were adapted to PyTorch and optimized for time efficiency. Appendix E Implementation Details of BoRaEM We implement BoRaEM in PyTorch with full GPU support and deterministic seeding for reproducibility. Item rewards are initialized to zero, ensuring symmetry across items at the start of optimization. Like CrowdBT [chen2013pairwise], we initialize worker competencies to a constant positive value (0.8), corresponding to a neutral reliability assumption. This avoids early instability while allowing competency differences to emerge purely from observed comparisons. Worker competencies are updated via closed-form aggregated statistics using vectorized accumulation. A Gaussian prior with fixed variance is incorporated as ridge regularization by adding prior precision to the denominator of each worker update. This shrinks poorly observed workers toward zero and prevents degeneracy. Within each EM iteration, reward and competency updates are alternated multiple times to accelerate convergence. After each outer iteration, identifiability constraints are enforced: rewards are centered to zero mean and normalized to unit root-mean-square magnitude, with worker competencies rescaled accordingly to preserve the likelihood. The competency prior variance σβ2 _β^2 imposes a Gaussian prior on worker competencies βw _w, which appears as an additional ridge term in the update: βs←∑(w,l)∈ℐs12(rw−rl)∑(w,l)∈ℐsκwl(rw−rl)2+σβ−2, _s← _(w,l) _s 12(r_w-r_l) _(w,l) _s _wl(r_w-r_l)^2+ _β^-2, where ℐsI_s is the set of comparisons performed by worker s, and κwl _wl are the Polya-Gamma expectations defined in Equation (10). A smaller σβ _β increases the prior precision σβ−2 _β^-2, shrinking βw _w toward 0 (strong regularization), whereas a larger σβ _β reduces the prior influence, allowing βw _w to vary more freely. We set the default value of σβ _β as 1.0. The reward updates solve a linear system (H+λrI)r=b(H+ _rI)r=b, where H depends on the worker competencies and Polya-Gamma expectations. The regularization coefficient λr _r is scaled with the mean of κwl _wl to ensure numerical stability and prevent overfitting to sparse comparisons: λr=λr,base⋅max(κ¯,10−12), _r= _r,base· ( κ,10^-12), where κ¯=1||∑(w,l)∈κwl κ= 1|D| _(w,l) _wl is the mean Polya-Gamma expectation over all comparisons, and λr,base _r,base is a small constant (default 10−210^-2). This acts as a data-dependent ridge, shrinking rewards slightly toward zero, improving convergence stability. Convergence is monitored via the mean log-likelihood, and optimization terminates when its improvement falls below a fixed tolerance (10−610^-6). Similar to BoRaEM, for HBTL, we set worker competencies to 0.8. Ablation studies are provided in Appendix I. (a) Varying N. K=100,M=500K=100,\,M=500. (b) Varying M. K=200,N=25000K=200,\,N=25000. (c) Varying K. M=20000,N=1000M=20000,\,N=1000. Figure 3: Varying N, K, M individually when β∈[0,1]β∈[0,1]. Appendix F Synthetic Dataset Experiments: β∈[0,1]β∈[0,1] In this regime β∈[0,1]β∈[0,1], we do not consider malicious workers (β∈[−1,0]β∈[-1,0]). Figure 3 shows the results when we individually vary each parameter. The details are as follows: 1. Varying N, Keeping K and M constant: We conduct an experiment with fixed values of the number of items K=100K=100 and the number of workers M=500M=500, while varying the number of measurements N∈2k,2.5k,3k,3.5k,4k,4.5k,5k,10k,25k,50k,100kN∈\2k,2.5k,3k,3.5k,4k,4.5k,5k,10k,25k,50k,100k\ to examine the effect of measurement volume on various techniques. As seen in Figure 3(a), with less measurements the accuracy of methods such as BoRaEM, HBTL and CrowdBT is lower than other methods because more parameters need to be estimated from less number of measurements. Amongst, BoRaEM, HBTL and CrowdBT, we see that BoRaEM and HBTL consistently perform better than CrowdBT in sparser settimngs (e.g. till N = 10000). 2. Varying M, Keeping N and K constant: We vary the number of workers M∈500,1k,2k,3k,4k,5k,5.5k,6k,8kM∈\500,1k,2k,3k,4k,5k,5.5k,6k,8k\ while fixing the number of items at K=200K=200 and the number of measurements at N=25kN=25k, to analyze the impact of worker scale. As seen in Figure 3(b), HTCV seems to be the most stable alogrithm as compared to BoRaEM, HBTL and CrowdBT. 3. Varying K, Keeping N and M constant: We vary the number of items K∈50,100,250,500,1000,1250,1500,1750,2250,2500K∈\50,100,250,500,1000,1250,1500,1750,2250,2500\ while fixing the number of workers at M=20kM=20k and the number of measurements at N=1kN=1k, in order to study the effect of item scale. Similar to varying N and M, by varying K (Figure 3(c)), we observe the same trend where the performance of BoRaEM, HBTL, HTCV and CrowdBT get affected when number of item increases whereas BoRaEM and HTCV seem to be more robust than CrowdBT. Although Figure 3 suggests that methods such as BTL and Rank Centrality (RC) are better than BoRaEM, HBTL, HTCV, FactorBT and CrowdBT in sparser settings for β∈[0,1]β∈[0,1], Section 6 (Figure 1) demonstrates how methods that do not model annotator competence can suffer significantly in the presence of adversaries. Section 6.4 further shows how these methods degrade with the addition of spammers and adversaries in real-world datasets. Appendix G Spammer Addition in FaceAge Dataset: More details (a) Accuracy (accacc) (b) Weighted Accuracy (waccwacc) Figure 4: Performance of different ranking methods under varying proportions of spammers introduced into the FaceAge dataset. In Section 6.4, we showed that results based on Kendall’s tau demonstrate how methods such as FactorBT, BT, BARP, and RC degrade under spammer addition, whereas BoRaEM, HBTL, and CrowdBT remain robust due to their explicit modeling of annotator competence. Figures 4(a) and 4(b) present the results in terms of accuracy and weighted accuracy when various types of spammers are added to the FaceAge dataset. A similar performance pattern to that observed with Kendall’s tau is evident for these metrics as well. (a) Kendall’s tau (τ) Figure 5: Performance of different ranking methods under varying proportions of spammers introduced into the Passage dataset, measured using Kendall’s tau (τ). (a) Accuracy (accacc) (b) Weighted Accuracy (waccwacc) Figure 6: Performance of different ranking methods under varying proportions of spammers introduced into the Passage dataset, measured using Accuracy (accacc) and Weighted Accuracy (waccwacc). Appendix H Spammer Addition in Passage Dataset In this section, we present the experimental results of spammer addition on the Passage dataset. Figures 5 and 6 shows the results using Kendall’s tau as the evaluation metric. CrowdBT, HBTL, HTCV, FactorBT and BoRaEM exhibit robustness to spammers, whereas the remaining methods suffer substantially in their presence. Similar trends are observed in Figures 6(a) and 6(b), which report results in terms of accuracy and weighted accuracy, respectively. Models such as BT, FactorBT, Rank Centrality, BARP suffer heavily in presence of spammers and malicious adversaries. These findings further emphasize the importance of modeling annotator competence or susceptibility to bias in crowdsourced settings. Appendix I Ablation Study I.1 Initialization of Worker Competencies As discussed in Appendix E, we initialize all worker competency values to a fixed positive value (0.8) in case of BoRaEM and HBTL, corresponding to a neutral reliability assumption to avoid early instability and allowing competency differences to emerge purely from observed comparisons. Similarly, for CrowdBT, we follow ferrara2024bias and initialize worker competencies to 0.7. In Section 2, we discussed about the objective function for FactorBT. For a comparison between items did_i and djd_j by worker s, described by a feature vector sij∈ℝMx_sij ^M, where M is the feature dimension. The probability that did_i is preferred over djd_j is given as follows Pr(di≻sdj)=σ(γs)⋅σ(ri−rj)+(1−σ(γs))⋅σ(⟨s,sij⟩) (d_i _sd_j)=σ( _s)·σ(r_i-r_j)+(1-σ( _s))·σ( b_s,x_sij ) (31) where ri,rjr_i,r_j are the latent scores of items did_i and djd_j, γs _s controls the susceptibility of worker s to bias and σ(γs)σ( _s) can be considered as the the skill of the worker same as that of βs _s in our formulation, s b_s encodes the worker-specific sensitivity to task features, σ(⋅)σ(·) is the sigmoid function. The model estimates parameters ri\r_i\, γs\ _s\, and s\ b_s\ by maximizing the regularized log-likelihood of the observed pairwise comparisons. For FactorBT, we uniformly initialize the worker logits (γs _s) to 0.70.7, which corresponds to initializing worker competencies as σ(0.7)≈0.668σ(0.7)≈ 0.668. Effect of initialization. In this ablation study, we analyze the effect of initialization on the performance of BoRaEM, HBTL, and CrowdBT on the FaceAge dataset [pavlichenko2021imdb]. The initialization parameter represents annotator competency, denoted by β. For BoRaEM and HBTL, we vary β from −1-1 to 11 in increments of 0.10.1. For CrowdBT, we vary β from 0 to 11 in increments of 0.10.1, consistent with its probabilistic interpretation. Figure 7: Effect of initialization on BoRaEM and HBTL. Both methods exhibit symmetry around β=0β=0, indicating that initializing with competency β or −β-β leads to identical performance. This symmetry arises because their likelihood depends on the term σ(βs(rw−rl))σ( _s(r_w-r_l)), which is invariant under simultaneous sign reversal of competency and score differences. Figure 8: Effect of initialization on CrowdBT. The method exhibits symmetry around β=0.5β=0.5, meaning that initialization with β or 1−β1-β produces identical results. This follows from the model structure, where β and 1−β1-β represent complementary competency assumptions in the likelihood formulation. Figure 9: Effect of initialization for FactorBT by varying the initial worker competency, βs=σ(γs) _s=σ( _s). Figures 7 and 8 present ablation studies on the effect of initialization for BoRaEM and HBTL, CrowdBT respectively. Figure 9 presents an ablation study on the effect of initialization for FactorBT by varying the initial worker competency, βs=σ(γs) _s=σ( _s). When βs _s is initialized to 0, the component of βsσ(ri−rj) _sσ(r_i-r_j) in Equation (31) is completely disabled, reducing the model to a pure worker-bias model that cannot learn meaningful item scores. Conversely, when βs _s is initialized to 11, the worker-bias component, σ(⟨s,sij⟩)σ( _s,x_sij ), is effectively disabled at initialization, causing the model to behave as the standard Bradley–Terry model. Since γs=logit(βs) _s=logit( _s), initializing βs _s to 11 yields γs→∞ _s→∞, for which the skill gradients vanish because σ′(γs)=σ(γs)(1−σ(γs))=0σ ( _s)=σ( _s)(1-σ( _s))=0. For intermediate initializations, the model’s performance remains stable, indicating that its performance is largely insensitive to the choice of initialization. This study demonstrates the robustness of all methods to uniform initialization, where all workers are assigned the same initial competency value, with performance remaining largely stable across the initialization range. Figure 10: Impact of the mean log-likelihood convergence tolerance (ϵε) on the performance of BoRaEM, HBTL, HTCV, CrowdBT, and FactorBT. The x-axis is shown in decreasing order, from loose to strict convergence. Both methods remain largely stable across the tested range of ϵε, with minor improvements in Accuracy and Kendall’s Tau observed for smaller values (stricter convergence). These results demonstrate that the EM updates converge reliably and are not overly sensitive to the choice of stopping criterion, confirming that the default setting of ϵ=10−6ε=10^-6 is within the optimal regime. I.2 Sensitivity to Mean Log-Likelihood Convergence Tolerance As described in Appendix E, we set the mean log-likelihood convergence tolerance parameter (ϵε) to 10−610^-6. Figure 10 illustrates the sensitivity of BoRaEM, HBTL, HTCV, CrowdBT and FactorBT to different values of the convergence tolerance ϵε. The results show that both methods maintain stable performance across the range, with only minor gains in Accuracy and Kendall’s Tau for stricter convergence, supporting the choice of ϵ=10−6ε=10^-6 as a robust default. I.3 BoRaEM-Specific Ablation Studies I.3.1 Sensitivity to Competency Prior Variance (σβ _β) As described in Appendix E, we set the Competency Prior Variance parameter (σβ _β) to 1.01.0. Figure 11: Effect of the competency prior variance (σβ _β) on BoRaEM. This hyperparameter controls the strength of the Gaussian prior over worker competencies (β), where smaller values shrink competencies closer to zero and larger values allow more variability. The results show that both methods are largely stable across a wide range of σβ _β, with minor improvements in Accuracy and Kendall’s Tau when the prior is moderately loose. This indicates that the EM updates are robust to the choice of σβ _β, and the default setting (σβ=1.0 _β=1.0) provides a reasonable balance between regularization and flexibility. The competency prior variance (σβ2 _β^2) controls the strength of the Gaussian prior on worker competencies βw _w in the EM updates. A smaller σβ _β increases the prior precision σβ−2 _β^-2, which shrinks βw _w toward 0 (strong regularization), while a larger σβ _β reduces the prior influence, allowing βw _w to vary more freely. In the extreme case of σβ→∞ _β→∞, the prior is effectively removed, which can lead to overfitting and slightly lower stability in the estimates. The results above reflect this trade-off: moderate σβ _β (e.g., 1.01.0) balances regularization and flexibility, producing optimal Accuracy and Kendall’s Tau. I.3.2 Sensitivity to Reward Regularization Coefficient (λr,base _r,base) As described in Appendix E, we set the Reward Regularization Coefficient Parameter (λr,base _r,base) to 10−210^-2. BoRaEM demonstrates remarkable robustness to the choice of λr,base _r,base except at extreme values. This relative insensitivity suggests that while the regularization is essential for numerical conditioning in the linear system, the model’s primary predictive power is driven by the underlying Pólya-Gamma latent variable framework rather than sensitive hyperparameter tuning. Figure 12: Effect of the reward regularization coefficient λr,base _r,base on BoRaEM performance. Appendix J Hardware Details The experiments were conducted on a server with dual-socket Intel Xeon Gold 6348 CPUs (28 cores / 56 threads per socket, 2.6 GHz base, 3.5 GHz turbo, AVX-512), 503 GiB RAM, and 4 NVIDIA A100 GPUs. The L1, L2, and L3 caches are 2.6 MiB, 70 MiB, and 84 MiB, respectively, across 2 NUMA nodes. The system runs Ubuntu 22.04 LTS.