Paper deep dive
Strategic Candidacy in Generative AI Arenas
Chris Hays, Rachel Li, Bailey Flanigan, Manish Raghavan
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/31/2026, 1:58:12 AM
Summary
The paper analyzes the vulnerability of generative AI ranking arenas (like Chatbot Arena) to strategic manipulation via 'cloning'—submitting multiple identical model variants to exploit statistical noise in pairwise comparisons. The authors prove that the status quo Bradley-Terry-based ranking mechanism is not clone-robust. They propose a new mechanism, 'You-Rank-We-Rank' (YRWR), which requires producers to submit internal rankings of their own models to correct statistical estimates, proving it is approximately clone-robust and improves overall ranking accuracy.
Entities (5)
Relation Signals (3)
You-Rank-We-Rank → improves → Ranking Accuracy
confidence 95% · YRWR improves overall ranking accuracy.
You-Rank-We-Rank → is → clone-robust
confidence 95% · We prove that this mechanism is approximately clone-robust
Status Quo mechanism → isvulnerableto → cloning
confidence 95% · the status-quo mechanism ... indeed rewards producers for submitting model clones.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI arenas, which rank generative models from pairwise preferences of users, are a popular method for measuring the relative performance of models in the course of their organic use. Because rankings are computed from noisy preferences, there is a concern that model producers can exploit this randomness by submitting many models (e.g., multiple variants of essentially the same model) and thereby artificially improve the rank of their top models. This can lead to degradations in the quality, and therefore the usefulness, of the ranking. In this paper, we begin by establishing, both theoretically and in simulations calibrated to data from the platform Arena (formerly LMArena, Chatbot Arena), conditions under which producers can benefit from submitting clones when their goal is to be ranked highly. We then propose a new mechanism for ranking models from pairwise comparisons, called You-Rank-We-Rank (YRWR). It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality. We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once. Moreover, to the extent that model producers are able to correctly rank their own models, YRWR improves overall ranking accuracy. In further simulations, we show that indeed the mechanism is approximately clone-robust and quantify improvements to ranking accuracy, even under producer misranking.
Tags
Links
- Source: https://arxiv.org/abs/2603.26891v1
- Canonical: https://arxiv.org/abs/2603.26891v1
Trouble viewing inline? Open PDF directly →
Full Text
130,713 characters extracted from source content.
Expand or collapse full text
inkscapelatex=false, Strategic Candidacy in Generative AI Arenas Chris Hays Rachel Li Bailey Flanigan Manish Raghavan Abstract AI arenas, which rank generative models from pairwise preferences of users, are a popular method for measuring the relative performance of models in the course of their organic use. Because rankings are computed from noisy preferences, there is a concern that model producers can exploit this randomness by submitting many models (e.g., multiple variants of essentially the same model) and thereby artificially improve the rank of their top models. This can lead to degradations in the quality, and therefore the usefulness, of the ranking. In this paper, we begin by establishing, both theoretically and in simulations calibrated to data from the platform Arena (formerly LMArena, Chatbot Arena), conditions under which producers can benefit from submitting clones when their goal is to be ranked highly. We then propose a new mechanism for ranking models from pairwise comparisons, called You-Rank-We-Rank (YRWR). It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality. We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once. Moreover, to the extent that model producers are able to correctly rank their own models, YRWR improves overall ranking accuracy. In further simulations, we show that indeed the mechanism is approximately clone-robust and quantify improvements to ranking accuracy, even under producer misranking. 1 Introduction As generative AI models proliferate, there is a need to systematically compare their performance. Such evaluations allow AI users to make informed choices between models, for organizations to make informed procurement decisions, for investors to allocate investments to more promising AI labs, for AI researchers to identify promising model development techniques towards making further progress, and for the research community to focus its efforts on common tasks (Donoho, 2024, 2017). This need has motivated the emergence of generative AI arenas, in which people vote on the outputs of pairs of different models. These comparisons are then aggregated into an overall ranking over models in the arena (where higher-ranked models are those that win pairwise comparisons more often). This evaluation approach offers the key advantage (over, e.g., static benchmarks) that evaluation occurs in the course of organic use, and so quality is measured on arguably user-relevant dimensions that might otherwise be difficult to capture. Accordingly, since 2024, many such arenas have emerged (Chiang et al., 2024; Zhao et al., 2025; Chi et al., 2025; Miroyan et al., 2025; Jiang et al., 2024): foremost among them currently is Arena (formerly LMArena, Chatbot Arena) (Chiang et al., 2024), which we will use as our prototypical example. Although these platforms are new, there is already evidence that their rankings are substantively important: consumers (Morrison, 2024), investors (Glover, 2025), and tech companies themselves (Alibaba Cloud Community, 2025) consider generative AI arenas to be a credible and economically relevant signal about the quality of models. As these arenas’ rankings become more important, so grow incentives for companies to try to improve the standing of their models. In recent work, Singh et al. (2025) highlight the risks of strategic manipulations: They submit several identical copies (which we’l call clones) of the same models to Arena, finding that one is ranked several places above the other.111Arena publishes confidence intervals around estimates of model qualities, and the confidence intervals for the estimated scores for the models in the Singh et al. (2025) experiment were overlapping. Such uncertainty quantification strategies would be useful for assigning ties in the ranks between models. However, in this work, we consider the (perhaps more difficult) problem of model ranking mechanisms where ties are not allowed, and indeed, at the time of writing, Arena does not allow ties in the ranks it assigns to models. Intuitively, conditions on these platforms may be favorable to strategic candidacy: rankings are formed through a relatively small number of comparisons (typically tens to hundreds per pair of models on Arena, or thousands to tens of thousands of comparisons per model) and pairwise win margins between models are small (for example, at the time of writing, the 1st-ranked model has a 58% win rate against the 20th-ranked model). In this low-data, tightly competitive regime, small amounts of noise can substantially affect outcomes. Thus, model producers have both the incentive and the ability to improve their arena position by simply submitting identical or near-identical222For example, near-identical submissions might occur if a producer submits multiple, qualitatively similar checkpoints from the same training run. models. With this as a starting point, we study two central questions: 1. When can producers benefit from clones in existing ranking systems, and if they can, by how much? 2. Can we design clone-robust ranking systems for this setting? Our approach and contributions. We begin with a formal model of the generative AI arenas. We assume that voter preferences are generated via the statistical model used for inference in these arenas (Chiang et al., 2024), a Bradley-Terry (BT) model. Thus, voter behavior is in some sense “ideal” and the status quo statistical model used for ranking is not misspecified. However, because the number of voter preferences are limited, there is noise in the rankings generated from these preferences. We then analyze the choices of model producers, who we assume have an exogenous set of distinct models (where distinctness of models is not observed by the platform) and may submit multiple copies of the same model to the ranking mechanism. Key to our formulation is the idea that producers may hope to increase the ranking of their top models by exploiting noise in voter preferences.333This is related to but different from the concerns around “hand picked” scores highlighted in Singh et al. (2025). In their work, the focus is on using private testing (where preference data is collected before a model is included in a ranking) to try to exploit selection effects by releasing only the model ranked best during private testing. There is no such notion of private testing in our work: all models submitted to the mechanism are assumed to be released publicly. Instead, in our model, the producers’ incentive to submit multiple models comes from the fact that their utility is determined by their best-performing models, rather than the average performance of their models. In our formalization of model producer incentives, producers derive utility from being ranked first within various subsets of models ("leaderboards"), like all models, all open-weight models, models below a given cost or latency threshold, or other intrinsic features of models on which a model producer wishes to compete. We make two primary contributions, corresponding to the research questions above: Establishing clone-nonrobustness of status quo mechanisms. In Section˜3, we formally prove conditions under which the intuition above is correct: the status-quo mechanism — which fits BT via maximum likelihood estimation and directly reports the implied ranking — indeed rewards producers for submitting model clones. Each clone effectively gives the producer an extra chance of getting “lucky” with their ranking position. We confirm that this problem is significant in simulations calibrated to Arena data: we show that some models can substantially improve their rank by submitting one or a few clones to the mechanism. To our knowledge, ours is the first proof of clone-nonrobustness of a Bradley-Terry models under a correctly specified, homogeneous human preference model. Existing non-robustness results in the literature hold in circumstances when voters have heterogeneous preferences, so the Bradley-Terry model is misspecified (Procaccia et al., 2025). Proposing a new accuracy-improving mechanism, You-Rank-We-Rank (YRWR), with clone-robustness guarantees. In Section˜4, we propose a new mechanism, which we prove is O(1/s)O(1/ s)-strategyproof, where s is the total number of samples per pairwise comparison. That is, a producer cannot gain more than O(1/s)O(1/ s) utility from doing anything other then submitting exactly one copy of every model in their set of distinct models. Our mechanism asks each producer to submit a ranking over their own models. We show that this precisely negates the potential benefits from adding a clone. We also establish that, if producers truthfully report rankings, the mechanism only makes the ranking more accurate, and provide conditions under which the mechanism is truthful. Our semisynthetic experiments confirm that YRWR is both clone-robust and increases overall ranking accuracy, even when producers do not necessarily report the correct ranking over their own models. We give an overview of our mechanism versus the status quo in Figure˜1. Related work. There is a growing literature on clone-robustness of Bradley-Terry models (Procaccia et al., 2025; Gölz et al., 2025; Siththaranjan et al., 2024). These works focus on a setting where voters may have heterogeneous preferences: different types of voters might have different preference orderings over candidates. Our work is complementary to these in that we explore clone nonrobustness even when voter preferences are homogeneous, so that in infinite data, there would be one “correct” ranking over models. In other words, their work studies clone nonrobustness stemming from misspecification of the BT model, and our work studies clone nonrobustness stemming from finite-sample effects. Our proposed mechanism is similar to that of Su (2022); Su et al. (2025) implemented at ICML, which consider incentives in the academic peer review process: both involve asking participants to rank their own submissions to the mechanism. However, we incorporate the ranking differently: Whenever the ranking induced by fitted Bradley-Terry scores disagrees with the producer’s own ranking, our mechanism resolves disagreement by assigning all of the models in question the minimum of their scores as opposed to fitting an isotonic regression (which effectively takes the mean of scores when reviewer scores disagree with author rankings). In their setting, the self-ranking helps denoise scores for a fixed, exogenous set of candidates; in ours, it also enables clone-robustness when the set of candidates is endogenous — participants in the mechanism can choose which (and how many) candidates to submit. Our setting is also different because we have to account for dependencies among fitted scores (because votes are over pairs of models), whereas there is no such dependency in their model. One reason for the differences between their settings and ours is that their motivating applications are reinforcement learning from human feedback (RLHF) (i.e., evaluating model responses for a given prompt against each other) and ours are model ranking (i.e., evaluating models against each other). Our work contributes to a growing body of work in strategic behavior in AI evaluations. In particular, Chen et al. (2026) analyze a Stackleberg game between a benchmark producer and multiple model producers, where model producers may try to game the benchmark by making benchmark-specific improvements to their model. Another line of work explores strategic or adversarial voting on generative AI arenas (Huang et al., 2026, 2025; Min et al., 2025). More broadly, our work sits in a growing literature on strategy-robust statistics (Spiess, 2025; Bates et al., 2024; Shi et al., 2025), which treat statistical protocols as mechanism design problems. 2 Setup There are n model producers. Each producer i∈[n]i∈[n] has a set iK_i of distinct models where ki=|i|k_i= |K_i | is a constant. The full set of distinct models will be denoted =⋃i∈[n]iK= _i∈[n]K_i where k=||k= |K |. Producer actions. Each producer i decides how many copies of each distinct model j∈ij _i to submit to the mechanism. In particular, they may submit clones of the same model to try to benefit from evaluation noise. Formally, a producer i’s action space is zi∈ℤ≥0kiz_i _≥ 0^k_i, where zijz_ij denotes the number of copies of model j submitted by producer i. We use z−iz_-i to represent the actions of all players besides i, and z−(i,j)z_-(i,j) to represent the actions corresponding to all models except producer i’s distinct model j. We write z=(z1,…,zn)z=(z_1,…,z_n) to denote a full assignment of actions to producers. The full set of models submitted by producer i to the mechanism is ℳi(zi)=⋃j∈ij(1),…,j(zij),M_i(z_i)= _j _i\j^(1),…,j^(z_ij)\, and we write mi(zi)=|ℳi(zi)|m_i(z_i)= |M_i(z_i) |. The full set of models submitted to the mechanism by all producers is ℳ(z)=⋃i∈[n]ℳi(zi),M(z)= _i∈[n]M_i(z_i), and we write m(z)=∑i∈[n]mi(zi)m(z)= _i∈[n]m_i(z_i). When it is clear from context or when we are speaking about a generic set of models, we will drop the argument z. Figure 1: The Status Quo (sq) mechanism (top half) and the You-Rank-We-Rank (yrwr) mechanism (bottom half). Pairwise voting. After the producers submit their models for an action z, the users vote on pairwise comparisons. After seeing outputs from two anonymous models j,j′∈ℳ(z)j,j (z), the user answers either j≻j′j j (j is preferred) or j′≻j j. Following the classic Bradley-Terry (BT) model (Bradley and Terry, 1952), we assume each model j∈ℳj has some latent quality Rj∈ℝ≥0R_j _≥ 0, and each vote is drawn as an independent Bernoulli random variable with probability Pr(j≻j′)=expRjexpRj+expRj′. (j j )= R_j R_j+ R_j . (Of course, if two models j,j′j,j are clones, then Rj=Rj′R_j=R_j .) We use pj≻j′=Pr(j≻j′)p_j j = (j j ) as shorthand. We use R to denote the vector of qualities, and R(ℓ)R_( ) to denote the ℓ -th largest value in R (and we will use analogous notation for estimated qualities as well). This voter preference model is exactly that for which Arena’s ranking procedure is specified correctly. Our analysis is thus deliberately favorable to the status quo. Let there be s∈ℕs votes per pairwise model comparison. We use vj≻j′v_j j to denote the (random) number of votes in which j≻j′j j (and thus, vj′≻j+vj≻j′=sv_j j+v_j j =s.) We’l write v=vj′≻jj′≠jv=\v_j j\_j ≠ j to denote the full set of vote counts. The parameters describing model producers are the number of producers n, distinct models K, model qualities R and pairwise vote counts s. Together, we will refer to fixed values of these parameters as a problem instance. Throughout this paper, we will impose the following regularity condition on R. restatableassumptionregcond There exists a universal constant C such that, for all problem instances and j,j′∈j,j , it holds |Rj−Rj′|≤C|R_j-R_j |≤ C. The assumption says that the qualities of any two producers may not be arbitrarily different, and that this maximum difference does not grow even when we analyze problem instances with many distinct models (e.g., large K). We’l assume that estimated R R obeys the same constraint. This kind of condition is standard in analyses of Bradley-Terry models (cf. Simons and Yao (1999)) and is useful in our analysis because it ensures that, when the number of models is large, no one model can have too much influence over the rankings of other models. In practice, pairwise win rates are typically bounded away from 0 and 1, even when the pair of models are ranked far apart. Mechanism. A mechanism G is a mapping from models ℳM and votes v over pairs of models j,j′∈ℳj,j to a ranking σ over all models in ℳM. We denote mechanism outputs as G(ℳ(z);v) G(M(z);v), where we drop the v when it is clear from context. We let Σ(G(ℳ(z)) ( G(M(z)) be the distribution of ranks (over the randomness in votes) induced from running G on an instance where players play z. We will write σ(ℓ)σ( ) to indicate the model ranked ℓ -th in the ranking and σ−1(j)σ^-1(j) to indicate the ranking of model j. We use the preference relation ≻σ _σ to indicate a comparison according to the ranking σ. Finally, we will often talk about rankings over subsets of models, ordered according to a reward vector. For a generic reward vector r∈ℝmr ^m and a subset of models S⊆ℳS , we define the ranking over S according to r as rank(S,r)=σ:σj≻σj′⇔rj≥rj′.rank(S,r)=σ:σ_j σ_j r_j≥ r_j . Utilities and leaderboards. Finally, we must formalize producer utilities. A natural goal for producers, intuitively, is to have one of their models ranked first among all models in the arena. However, treating winning overall as the sole source of utility fails to explain behavior in practice, where producers routinely submit models that will not rank highly, even against their own existing models.444For example, in August 2025, OpenAI released (and eventually submitted to Arena) several open-weight models (the oss series) despite the fact that their performance was clearly limited compared to their existing flagship (closed) models; see https://openai.com/index/introducing-gpt-oss/. Instead, we propose a utility model that rewards producers for being best in a given class of relevant models, even if they do not rank first in the overall arena. Formally, we consider a collection of leaderboards ℒ⊆0,1ℳL \0,1\^M, where each leaderboard L∈ℒL is a set of competing models. For example, producers might be interested in ranking first among open-weight models, in which case the relevant leaderboard L would consist of j∈ℳ:j is open-weight\j \;:\;j is open-weight\. Similarly, we might consider leaderboards for non-reasoning models, models under a certain size/cost/latency threshold, or models supported by a particular IDE. Leaderboards need not be disjoint. Intuitively, they capture the idea that different consumers have different requirements, and indeed we could microfound this with a consumer choice model where each leaderboard is a consumer’s consideration set. Under this utility model, producers may would be incentivised to submit “weak” models that have no chance of ranking first in the overall leaderboard if these models have a chance of winning on some other relevant leaderboard. With this, we can formally define producer utility. Each producer i assigns some importance νi(L)∈[0,1] _i(L)∈[0,1] to each leaderboard L, which corresponds to the utility i receives for having the top-ranked model in L — that is, for each leaderboard L∈ℒL , producer i gets utility νi(L) _i(L) for winning L and 0 otherwise. We normalize these utilities so that ∑L∈ℒνi(L)=1 _L _i(L)=1. Formally, we let σL _L be the sub-ranking of σ over the models in leaderboard L; i.e., for all j,j′∈L,j≻σj′⇔j≻σLj′j,j ∈ L,j _σj j _ _Lj . Then, if the overall ranking from the mechanism is σ, producer i’s utility is ui(σ)=∑j∈ℳi∑L∈ℒνi(L)⋅(σL(1)=j).u_i(σ)= _j _i _L _i(L)·1( _L(1)=j). Naturally, clones must belong to the same leaderboards, as leaderboard membership is based on a model’s fixed characteristics (e.g., size, cost). Throughout, we use nonasymptotic big-O notation, so that, e.g., for sequences aii=1∞,bii=1∞\a_i\_i=1^∞,\b_i\_i=1^∞ we write bi=O(ai)b_i=O(a_i) if there exists a universal constant C such that bi≤Caib_i≤ Ca_i for all i. 3 The Status Quo Mechanism The Status Quo mechanism (sq) used by Arena (Chiang et al., 2024) is depicted in the top half of Figure˜1 and is formally defined in Algorithm˜1. sq takes in a set of models present in the arena and the pairwise vote count over them. Using the votes, it fits estimated rewards R^=(R^1,…,R^m) R=( R_1,…, R_m) via maximum likelihood estimation.555The Bradley-Terry model parameters are invariant to addition by a constant vector (i.e., data generated by parameters R and R+cR+c1 are equal in distribution), so to fit MLE it is necessary to enforce an identifiability constraint like ∑j∈ℳR^j=0 _j R_j=0. Then, it outputs the ranking implied by R R. Ties can be broken arbitrarily. In the text, we will refer to sq on inputs ℳ,vM,v as sq(ℳ) sq(M), dropping the v argument when it is clear from context. Input: Models ℳM; pairwise vote counts v Compute R^←argmaxR∈ℝm∑j≠j′vj≻j′log(exp(Rj)exp(Rj)+exp(Rj′)) R← _R ^m _j≠ j v_j j \! ( (R_j) (R_j)+ (R_j ) ) return σ=rank(ℳ,R^)σ=rank(M, R) (breaking ties arbitrarily) Algorithm 1 Status Quo (sq) The effects of clones. Before we formalize our main result for this section, we identify the key intuition for how cloning can affect the ranking distribution induced by sq. 1. The “Lottery Ticket” Effect. The cloned model provides an extra chance for the producer to win, since it receives its own random draw of pairwise votes, which we call the lottery-ticket effect. The producer benefits from taking the best outcome among these random draws, thereby increasing the probability of winning. This is a direct consequence of the fact that there are a finite number of votes per pair of models, which leads to randomness in the final ranking — if there were infinite comparisons, the ranking would be deterministic and this effect would disappear. 2. The “New Competitor” Effect. Introducing a clone causes a new-competitor effect: new pairwise votes must be collected between the cloned model and the existing models. This changes the competitive environment faced by each model. Intuitively, if the clone is very strong, most existing models will lose more of their matchups relative to the counterfactual without the clone; if it is very weak, they win more. These changes propagate to the fitted Bradley–Terry scores, potentially increasing or decreasing any individual model’s win probability, adding complexity to our analysis. However, our workhorse lemma, Lemma˜C.1, establishes that the new competitor effect is small: the change to any individual model’s win probability is no more than O(1/s)O(1/ s). Intuitively, a clone is valuable as long as the lottery ticket effect (which is always positive) outweighs the new competitor effect (which can either be positive or negative). Our main result characterizes when the lottery ticket effect outweights the potential drawbacks of the new competitor effect. Intuitively, this is when a producers’ models have a reasonable (but uncertain) chance of winning a (set of) leaderboard(s) of non-negligible importance. We next formalize these conditions via a definition. Informally, the definition says that, across leaderboards with positive total weight to a producer i, two quantities are bounded away from 0: the model j’s chance of winning, and producer i’s chance that none of their models win. These conditions are important for a model to be worth cloning because, if a model has no chance of winning, a cloned version of that model also won’t have a chance of winning. Similarly, if a producer is guaranteed to win on a set of leaderboards, there is no need to clone models to improve the producer’s chances on those leaderboards. Definition 3.1 ((ε,δ)( ,δ)-competitive model). For a fixed strategy profile z, a model j∈ℳij _i for producer i is said to be (ε,δ)( ,δ)-competitive if there exists a set of leaderboards S⊂ℒS such that ∑L∈Sνi(L)≥ε _L∈ S _i(L)≥ and, for all L∈SL∈ S, Prσ∼Σ(sq(ℳ(z)))(σL(1)=j)≥δ, andPrσ∼Σ(sq(ℳ(z)))(σL(1)∉ℳi)≥δ. _σ ( sq(M(z)))( _L(1)=j)≥δ, and _σ ( sq(M(z)))( _L(1) _i)≥δ. The parameter ε represents how important the set of leaderboards must be, and δ represents the minimum probability that model j wins on these leaderboards. The condition that there must exist a set of leaderboards with total utility more than ε is important because of dependencies across leaderboards: if a clone helps the producer win on an inconsequential leaderboard (νi(L)≈0 _i(L)≈ 0) but harms the producer on another leaderboard with non-negligible potential utility, then the costs of cloning might outweigh the benefits. We are now ready to state our main theorem in this section. It establishes that if a producer has a (ε,δ)( ,δ)-competitive model, then for a sufficiently large set of models and votes per pair of models, producers are incentivised to submit another copy of that model. restatable [Clone-nonrobustness of the status quo mechanism]theoremstatusquo For all constants ε,δ>0 ,δ>0, there exists s0,m0s_0,m_0 such that for all s≥s0,m≥m0s≥ s_0,m≥ m_0, the following holds. For any producer i, any strategy profiles z and any (ε,δ)( ,δ)-competitive model j, producer i would benefit from submitting an additional copy of j. Formally, let z′=(zi,j+1,z−i,j)z =(z_i,j+1,z_-i,j). Then σ∼Σ(sq(ℳ(z′))[ui(σ)]>σ∼Σ(sq(ℳ(z))[ui(σ)]. _σ ( sq(M(z ))[u_i(σ)]>E_σ ( sq(M(z))[u_i(σ)]. Intuitively, the smaller ε and δ are, the weaker the benefits to cloning may be — smaller ϵε means that the cloned model sits on less important (to the model producer) leaderboards and smaller δ means that the lottery ticket effect of an additional copy of the (ϵε, δ)-competitive model are smaller. Thus, smaller ε,δ ,δ imply larger s0,m0s_0,m_0, since new competitor effects decrease in s and m. The proof of Definition˜3.1 is given in Appendix˜D. We have thus established conditions under which a producer can benefit from clones. It is trivial to demonstrate that the gains from clones can be potentially very large: Example 3.2 (Constant possible gain). Consider n producers with one distinct model, each of identical quality. Each producer wins with probability 1/n1/n. Now, suppose producer 1 submits k clones (including their original model); their new probability of winning is k/(k+n−1)k/(k+n-1), which is constant (in n and mm) for k∈Ω(n)k∈ (n) and approaches 11 as k→∞.k→∞. The fact that the above example sets all rewards identically is for simplicity of intuition; what it illustrates more broadly is that a producer can flood the system with models and drive their win probability upwards arbitrarily. 3.1 Simulation study with Arena data To demonstrate the implications of Definition˜3.1, we conduct a set of simulations calibrated to Arena data to explore how much producers can benefit from clones. Figure 2: Ranks gained via cloning under the Status Quo versus You-Rank-We-Rank mechanisms, across several of Arena’s model arenas. To do this, we snapshot Arena on January 1, 2026 and focus on the largest arenas (those with the most models). For each such arena, we treat all listed models as distinct and use the platform’s published BT scores as “ground-truth” qualities, with model producers given by the associated organization metadata. Thus, our empirical approach is designed to show what would happen in the idealized case (favorable to the status quo) in which (1) the Bradley-Terry model was correctly specified and (2) Arena had perfectly estimated model qualities. Holding these qualities fixed, we then vary the number of clones of each model j over ℓ∈1,2,3,4,5 ∈\1,2,3,4,5\, each time, producing some new synthetic set of models ℳM with the original models and the clones. For each of these model sets ℳM, we generate synthetic pairwise outcomes: for each unordered pair (j,j′)(j,j ), we draw s iid Bernoulli comparisons with BT win probability pj≻j′p_j j , where s is the arena’s average number of votes per model-pair. This produces our vote counts v(j,ℓ)v^(j, ) (where j is cloned ℓ times). We also repeat this procedure in the raw instance with no clones, the vote counts for which we call v. Code to reproduce the simulations and plots is available at https://github.com/johnchrishays/strategic-candidacy-in-genai-arenas. We then run sq with vote counts v(j,ℓ)v^(j, ) for all j∈L,ℓ∈1,2,3,4,5j∈ L, ∈\1,2,3,4,5\, and also on the raw instance v (using the corresponding multiset of models each time). For each model j, we report the highest rank among all of its clones, capturing the idea that producers are rewarded for their highest-ranked model in a given leaderboard. We show the results in Figure˜2, where we plot the average, 5th, and 95th percentile of the number of ranks gained across models j∈Lj∈ L between 0 clones (raw instance) and ℓ clones, across arenas. The results for sq are on the left per arena; the right shows analogous results for our mechanism yrwr, which we will unpack in Section˜4.6. Focusing on the lines pertaining to sq, we see that across arenas, cloning a model leads it to gain in rank position, with some models moving up ≥7≥ 7 positions with just one clone. With additional clones, rank position can be reliably increased further with mild diminishing marginal returns. We remark that there are larger benefits from clones in arenas (like Coding, Expert, and Multiturn) with fewer pairwise votes per pair of models (s≤30s≤ 30). Second, we conduct additional data analysis in Appendix˜A to show that the models that benefit from clones have scores that are clustered together with several other models, which means that small increases to fitted scores yield large increases in position on the arena. 4 The You-Rank-We-Rank Mechanism The YRWR mechanism is depicted in the bottom half of Figure˜1 and defined formally in Algorithm˜2. It augments the status-quo mechanism first by taking an additional input: a set of producer-defined rankings π=πii∈[n]π=\ _i\_i∈[n], where πi _i is a ranking over ℳiM_i. Our main result does not require us to assume that this ranking is “correct” in any sense; this can be thought of as a producer’s prioritization over their models with respect to the mechanism. We will discuss this in more formality later. We will refer to YRWR on inputs ℳ,π,vM,π,v as yrwr(ℳ,π;v)(M,π;v), again dropping the v from the notation when clear from context. As in sq, yrwr first estimates quality scores R R via MLE. Then, instead of directly outputting the implied ranking, it performs a monotone score correction within each producer: for every model j∈ℳij _i, its corrected score Rˇj R_j is the minimum estimated score among all models j′∈ℳij _i ranked ahead of j according to πi _i. Finally, yrwr outputs a global ranking implied by Rˇ R. Input: Models ℳM; rankings πii∈[n]\ _i\_i∈[n]; vote counts v Compute R^←argmaxR∈ℝ|ℳ|∑j≠j′vj≻j′log(exp(Rj)exp(Rj)+exp(Rj′)) R← _R ^|M|\ _j≠ j v_j j \! ( (R_j) (R_j)+ (R_j ) ) foreach producer i∈[n]i∈[n] do foreach j∈ℳij _i do Rˇj←minR^πi(k):k≤πi−1(j) R_j← \ R_ _i(k):\ k≤ _i^-1(j)\ Output: σ=rank(ℳ,Rˇ)σ=rank(M, R) (breaking ties within each producer i according to πi _i and ties between producers arbitrarily). Algorithm 2 You-Rank-We-Rank (yrwr) Notably, our mechanism receives no external information about which models are clones and which are distinct. This is motivated by key practical challenges: the platform may have no access to proprietary models’ weights or logits and tests of whether models produce similar outputs might be computationally or statistically intractable. Moreover, even if the mechanism could require white-box model inspection, producers might minimally change the parameters of similar models or otherwise manipulate their submissions to evade clone detection. Before we state our main result of this section, we provide intuition about why the self-ranking mechanism provides clone-robustness. Intuition: Why do producers’ rankings help? Enforcing that scores obey producer ranks help produce more accurate rankings and disincentivise clones. To see why this is true, consider the distribution of fitted scores of 2 copies of model j by producer i, denoted j(1),j(2)j^(1),j^(2), where without loss of generality, we assume j(1)≻πij(2)j^(1) _ _ij^(2). Because of the minimum operation to compute Rˇj(1),Rˇj(2) R_j^(1), R_j^(2), the distribution of their maximum is exactly that of R^j(1) R_j^(1): maxRˇj(1),Rˇj(2)=R^j(1). \ R_j^(1), R_j^(2)\ d= R_j^(1). Put another way, with yrwr, the producer’s fitted score is the maximum of two draws half the time (if R^j(1)>R^j(2) R_j^(1)> R_j^(2), agreeing with the producer ranking πi _i) and the minimum otherwise. And for two identically distributed random variables, the distribution of a random variable which is the maximum of the two variables with probability half and the minimum with probability half is exactly equal to the distribution of each random variable. This means, from the perspective of the top producer-ranked model in the clones of j, it is no better to submit two models than it is to submit a single model. The key idea we are leveraging is that when submitting clones, the producer cannot know which will do better — the producer will have had to “pick a winner” between the clones in advance. By contrast, under sq, the distribution of maxR^j(1),R^j(2) \ R_j^(1), R_j^(2)\ stochastically dominates that of R^j(1) R_j^(1). This creates the selection-on-winners effect which producers could benefit from. Finally, we note that this mechanism removes the incentives for clones despite not having any special information about which models are clones. 4.1 Approximate clone-robustness of YRWR. We now show our main result in this section: for all producers i, it is an approximate dominant strategy to submit one copy of each distinct model, regardless of producers’ submitted rankings π. restatable[Approximate cloneproofness]theoremyrwrthm For all ε>0 >0, there exists s0,m0s_0,m_0 such that for all s≥s0s≥ s_0 and m≥m0m≥ m_0, the following holds. Fix any π,zπ,z, and let z′=(,z−i)z =(1,z_-i) be the profile where i instead plays one copy of each model. Then σ∼Σ(yrwr(ℳ(z′),π)[ui(σ)])≥σ∼Σ(yrwr(ℳ(z),π))[ui(σ)]−ε. _σ ( yrwr(M(z ),π)[u_i(σ)]) _σ ( yrwr(M(z),π))[u_i(σ)]- . Our proof of this result is in Appendix˜E. It relies on the intuition provided above along with an application our workhorse distributional stability result, Lemma˜C.1: We observe that the distribution of the first mechanism-ranked clone is equal to that of the first producer-ranked clone, by isotonicity. Then, using Lemma˜C.1, we establish that the win probability of the first producer-ranked clone is approximately (up to additive ε ) equal to the distribution of a single model under the strategy with one copy of each model. 4.2 Accuracy of YRWR. Thus far, we have established approximate clone-robustness of the YRWR mechanism. It is natural to ask whether this clone-robustness property comes at a cost to ranking accuracy. After all, our mechanism may modify estimated BT scores, even when there are no clones and when the pairwise preference data is generated by a BT model. Naturally, the impact of producer rankings on the accuracy of the mechanism depends to some degree on the quality of producers’ submitted rankings — but we now show that if producers’ submitted rankings are consistent with the ground-truth ranking, our mechanism only makes the scores more accurate, at least with respect to ℓ∞ _∞ distance. We will henceforth refer to the correct ranking as πi∗:=rank(ℳi,R)π^*_i:=rank(M_i,R) with π∗:=πi∗i∈[n]π^*:=\π^*_i\_i∈[n]. Proposition 4.1 (yrwr is accuracy-improving). Fix R,ℳR,M and let R^=sq(ℳ) R= sq(M) and Rˇ=yrwr(ℳ,π∗) R= yrwr(M,π^*). Then, ‖Rˇ−R‖∞≤‖R^−R‖∞.\| R-R\|_∞≤\| R-R\|_∞. The proof of Proposition˜4.1 is in Appendix˜F and follows from the fact that yrwr’s score-correction replaces each score by a minimum over a fixed subset of scores — a mapping that is 11-Lipschitz in ∥⋅∥∞\|·\|_∞, so the correction cannot increase ℓ∞ _∞ error. Proposition˜4.1 directly implies that that yrwr maintains the asymptotic correctness properties of sq. We formalize this next. Corollary 4.2 (Efficiency and correctness of yrwr). Fix R,ℳR,M, and let Rˇ=yrwr(ℳ,π∗) R= yrwr(M,π^*). Then, • Rˇ R is a s s-consistent estimator of R • If ∃γ>0:minj≠j′∈ℳ|Rj−Rj′|>γ>0∃γ>0: _j≠ j |R_j-R_j |>γ>0, then P[rank(ℳ,Rˇ)=σ∗]→1 as s→∞.P[rank(M, R)=σ^*]→ 1 as s→∞. Informally, the corollary states that, under truthful producer rankings, the modified scores produced by yrwr inherit the statistical efficiency and asymptotic almost sure ranking correctness of the sq mechanism. 4.3 Truthfulness in producer rankings π. The accuracy results above rely on producers ranking their models according to R. However, it is not immediately obvious that they will always have an incentive or knowledge to do so. In general, misreported producer rankings could lead to reductions in the accuracy of the ranking: Even with infinite data, if a producer misreports their ranking, the mechanism could drastically change the ranking to enforce isotonicity, leading to rankings which are far from correct. We explore producer’s ranking strategies and their implications next. Incentives for producer misreports. Even if a producer knows the ground-truth ranking over their own models, they may not be incentivised to truthfully report the ranking. The problem is producers’ differential utilities across different leaderboards; we illustrate this with the following example. Example 4.3. Suppose producer i has two models a,ba,b with similar true rewards but where Ra>RbR_a>R_b. Suppose a is closed-weight and b is open-weight. Assume the producer knows that they have a negligible chance of winning the leaderboard LallL_all consisting of all models. I.e., P(σLall(1)∈a,b)≈0P( _L_all(1)∈\a,b\)≈ 0. Moreover, suppose the producer assigns non-negligible weight to the leaderboard for open-weight models LopenL_open, on which b is eligible but a is not, i.e., b∈Lopenb∈ L_open, a∉Lopena∉ L_open, and νi(L)≫0 _i(L) 0. Also, suppose model b has a non-negligible chance of winning LopenL_open, i.e., P(σLopen(1)=b)≫0P( _L_open(1)=b) 0. Under the yrwr score correction, if the producer reports a≻ba b, then b’s corrected score is Rˇb=minR^a,R^b, R_b\;=\; \ R_a, R_b\, so a low realized R^a R_a—which can occur purely due to estimation noise, despite the fact that Ra>RbR_a>R_b—will cap b’s corrected score and reduce b’s chance of winning leaderboard LopenL_open. Intuitively, what is happening is that producer i misranks their models to “protect” model b from being penalized due to noisy estimates of lower priority models. This problem is resolved if the qualities of models that could win or could drag down winners are sufficiently separated relative to s such that the noise poses no risk. While one can formulate sufficient conditions for truthfulness of this flavor, we next state the simpler claim that once s is large enough, yrwr is truthful in π: Proposition 4.4 (Asymptotic truthfulness). For all z and ε>0 >0, there exists sufficiently large s0∈ℕs_0 such that for all s≥s0s≥ s_0, σ∼Σ(yrwr(ℳ(z),(πi∗,π−i)))[ui(σ)]≥σ∼Σ(yrwr(ℳ(z),(πi,π−i)))[ui(σ)]−ε. _σ ( yrwr(M(z),( _i^*, _-i)))[u_i(σ)] _σ ( yrwr(M(z),( _i, _-i)))[u_i(σ)]- . In other words, truthfulness is an approximately dominant strategy for sufficiently large s. Of course, the large s regime is exactly where the yrwr mechanism is least necessary (since in large samples, there are smaller incentives for strategic candidacy). To address the concern of producer misranking, in the next section, we consider a variant of YRWR which overrides producer rankings when enough votes have been collected to confidently order pairs of models. 4.4 An uncertainty-aware YRWR variant Even if a producer wishes to truthfully report their ranking over models, they may have uncertainty over the relative performance of their models. This could again lead to degradations in the quality of the ranking: If a producer has no information about which models are better than others, their ranking can be no better than random and, even with infinite data, the mechanism would likely produce an incorrect ranking. There are two ways to mitigate incorrect producer rankings due to producer uncertainty. First, the platform could implement private testing, which allows for the collection of preference data before a model is submitted to public leaderboards. Private testing can serve as a useful tool in circumstances where model producers are uncertain about the relative performances of their models: the producer can collect preference data about the relative performance of their models, and use it to inform the ranking they submit to the mechanism. Indeed, if each producer has only a small number of models, private testing can be very statistically efficient: There are only a small number of pairwise comparisons to make, so high-quality producer rankings can be computed with much less data than necessary for a high-quality overall ranking. Platforms like Arena already provide private testing, and as long as fresh data is collected (i.e., private testing data is not used to form the ranking) when the model is released publicly, model producers cannot benefit from selection effects due to noise in preference data during private testing. Second, the platform could implement a variant of the YRWR mechanism that only enforces producer rankings among fitted model scores that have overlapping confidence intervals. That is, at confidence level α/(m2)α/ m2, let CI^α/(m2)(j) CI_α/ m2(j) be a s s-consistent confidence interval for model j containing the MLE estimate R R (perhaps as computed in Chiang et al. (2024)). Then this uncertainty-aware (UA) YRWR variant, called ua-yrwr, would be defined by fitting the BT-MLE scores as in Algorithm˜2 but correcting scores as RˇjUA←minR^πi(k):k≤πi−1(j),CI^α(j)∩CI^α(k)≠∅, R^UA_j← \ R_ _i(k)\;:\;k≤ _i^-1(j), CI_α(j)∩ CI_α(k)≠ \, and using these scores to produce a ranking. Like the vanilla yrwr, the mechanism will take ℳ,πM,π as arguments, but it will also take a simultaneous confidence level α, which will be assumed to construct a set of confidence intervals that hold simultaneously with probability at least 1−α1-α. This variant of the mechanism is appealing because it ignores producer rankings in regimes where there is enough data to confidently rank models. Thus, in infinite data, the ranking produced by this variant would be correct, regardless of the rankings submitted by model producers. To achieve approximate clone-robustness with high probability, the confidence level α would have to be chosen to ensure simultaneous validity across all confidence intervals. We next prove analogous results for the uncertainty-aware variant as we did for the vanilla variant in Sections˜4.1 and 4.2. Proofs are deferred to Appendix˜F. Section˜4.4 (to Section˜4.1) establishes approximate clone-robustness. Section˜4.4 is identical to Section˜4.1, except that it is looser by the simultaneous confidence level α. The argument for Section˜4.4 is almost directly implied by Section˜4.1: If the confidence intervals cover R, the argument for Section˜4.1 goes through directly. If not, the change in utility can be at most the confidence level α. restatable [Approximate cloneproofness of ua-yrwr]corollaryuayrwrthm For all ε>0 >0, there exists s0,m0s_0,m_0 such that for all s≥s0s≥ s_0 and m≥m0m≥ m_0, the following holds. Fix any π,zπ,z, and let z′=(,z−i)z =(1,z_-i) be the profile where i instead plays one copy of each model. For any simultaneous confidence level for ua-yrwr α>0α>0, it holds σ∼Σ(ua-yrwr(ℳ(z′),π,α)[ui(σ)])≥σ∼Σ(ua-yrwr(ℳ(z),π,α))[ui(σ)]−ε−α. _σ ( ua-yrwr(M(z ),π,α)[u_i(σ)]) _σ ( ua-yrwr(M(z),π,α))[u_i(σ)]- -α. Proposition˜4.5 establishes s s-consistency of the fitted scores and asymptotic correctness of the estimated ranking. Proposition˜4.5 is considerably stronger than Corollary˜4.2: it holds for any producer ranking, rather than just truthful ones. The argument for the proposition is also different: Since we do not assume the producer ranking is truthful, we cannot appeal to Proposition˜4.1, so in the proof we make a direct argument for efficiency and correctness. Proposition 4.5 (Efficiency and correctness of ua-yrwr). Fix R,ℳR,M, and any set of producer rankings π. Let RˇUA=ua-yrwr(ℳ,π,α) R^UA= ua-yrwr(M,π,α). Then, • RˇUA R^UA is a s s-consistent estimator of R • If ∃γ>0:minj≠j′∈ℳ|Rj−Rj′|>γ>0∃γ>0: _j≠ j |R_j-R_j |>γ>0, then P[rank(ℳ,RˇUA)=σ∗]→1 as s→∞.P[rank(M, R^UA)=σ^*]→ 1 as s→∞. 4.5 A within-leaderboard YRWR variant In the results we have presented so far, we have established that yrwr incentivizes truthful reports in the asymptotic regime (Proposition˜4.4) but that in general, producers may misreport their true rankings in finite samples (Example˜4.3). However, if we modify the mechanism to perform within-leaderboard instead of global score correction, model producers are incentivised to submit truthful rankings in finite samples as well, as we show in our next result. Formally, this modified mechanism, called local-yrwr, is the same as yrwr except that its score correction is performed within each leaderboard: Rˇj;π,Llocal:=minR^πi(k):k≤πi−1(j)andπi(k)∈L. R^local_j;π,L:= \ R_ _i(k):\ k≤ _i^-1(j)\ and\ _i(k)∈ L \. To implement this approach, leaderboards must be explicitly defined on the platform, rather than implicitly defined by, e.g., a user who will choose the first among a subset of models. That is, the platform must be able to provide separate rankings for different leaderboards. To accomplish this, platforms could provide filters on the rankings within each arena, using metadata about each model, like model size, cost, latency, or other factors. Thus, users who wanted, e.g., to see the ranking among models below a given cost threshold, could see a ranking generated to be both truthful and clone-robust.666However, even if such a filter system could be feasibly implemented, a possible downside of local-yrwr is that it can lead to inconsistent rankings between pairs of models across leaderboards, which users might find confusing or difficult to interpret. Our next result establishes truthfulness of the local-yrwr mechanism. Proposition 4.6 (Truthfulness of local-yrwr). Fix any producer i, any strategy profile z in which producer i submits exactly one copy of each model (ℳi=Ki)(M_i=K_i), and fix any other-producer rankings π−i _-i. Then for every finite s and every alternative report πi _i, σ∼Σ(local-yrwr(ℳ(z),(πi⋆,π−i)))[ui(σ)]≥ _σ ( local-yrwr(M(z),( _i , _-i)))[u_i(σ)]\ ≥ σ∼Σ(local-yrwr(ℳ(z),(πi,π−i)))[ui(σ)]. _σ ( local-yrwr(M(z),( _i, _-i)))[u_i(σ)]. An analogous approximate strategy-proofness result holds for local-yrwr as for yrwr, using the same argument as in the proof of Section˜4.1. Intuitively, it is always approximately utility improving to remove clones from any rank correction operating over a set of clone, regardless of which other models by the same producer might or might not be in the same leaderboard. 4.6 Simulation study with Arena data yrwr vs sq on rank positions gained (Figure˜2). We now unpack the results on yrwr presented in Figure˜2, which are shown on the righthand side for each arena. Our methods are exactly the same as in Section˜3.1 except that when testing yrwr, we had to additionally generate each producer i’s ranking over their own models πi _i, including any potential clones. For a given set of submitted models produced by producer i, in the simulations for Figure˜2, we assume the producer ranked them accurately, i.e., as πi= _i= rank(ℳi,R)(M_i,R). We see a striking difference between the two mechanisms: even on arenas where s is small, yrwr admits essentially zero gains in rank for any model via cloning — even as the number of clones increases. Nonetheless, there are models that see small rank improvements from submitting clones to the mechanism, as a result of the new competitor effect. In Appendix˜A, Figure˜5, we visualize the benefits of cloning for each model relative to model quality. We show that models near the middle of the ranking and for which there are many models are similar quality are the main beneficiaries of cloning under yrwr, while models with top or bottom true qualities see nearly zero benefits to cloning. We leave further analysis of how the new competitor effect varies with fitted score and competitiveness of the model for future work. yrwr versus sq on Accuracy (Figures˜3(a) and 3(b)) Our theory tells us that under correct producer rankings πi _i, yrwr should produce more accurate reward estimates than sq (in infinity norm). Unsurprisingly, we find that these more accurate reward estimates translate to more accurate rankings: in Figure˜3, we compare the bubble-sort distances (also called Kendall-Tau distance (Kendall, 1938)) from the respective rankings produced by yrwr and sq to rank(ℳ,R)(M,R), the ground-truth ranking. We see that across all six arenas and all numbers of clones, yrwr has lower distance to the true ranking than sq, often by many tens of swaps. This is increasingly true as the number of clones increases, illustrating the importance of clone-robustness in recovering accurate rankings. (a) Correct producer rankings. (b) Noisy producer rankings. Figure 3: Difference in Kendall-Tau distance to the ground truth under the Status Quo versus You-Rank-We-Rank mechanisms, across Arena’s various arenas. Difference greater than 0 implies that the YRWR mechanism is closer to the true ranking. Our theory leaves open whether these gains in accuracy are robust to inaccuracies in producers’ rankings, i.e., scenarios in which πi≠rank(ℳi,R) _i (M_i,R). We test this by repeating the analysis in with producers’ rankings perturbed via a random utility model. To compute perturbed producer rankings π~i π_i, we add i.i.d Gaussian noise ϵε to the model qualities: R~j=Rj+ε R_j=R_j+ We sweep over the variance of ε such that the Kendall-Tau distance between the perturbed and true rankings is between 10% the length of the list and more than 100% of the length of the list, so that for a ranking of about 300 models, the Kendall Tau distance between R~ R and R is between 30 and 400. We show the results of this analysis in Figure˜3(b). We see that the ranking by yrwr remains substantially more accurate than sq for most settings of the noise variance across arenas, with the accuracy advantage diminishing as the noise increases. Eventually, when producer rankings are sufficiently noisy, the accuracy benefits of yrwr drop below zero, although this only happens when the noise in rankings is on the same order as the number of models. 5 Discussion Generative AI arenas serve as important and useful mechanisms to compare AI models under realistic use conditions. Our work studies a simple vulnerability in status quo mechanisms: producers can leverage the noise inherent to these rankings by submitting identical or near-identical models. Such cloning-based manipulations can in turn further deplete samples, leading noise to drown out signal. The alternative mechanism we propose, yrwr, reduces incentives for this type of strategic behavior. Future work could build on the framework we introduce to continue the study of strategic behavior in the face of noisy evaluation. Our model could be extended to study online evaluation, where votes and model submissions arrive sequentially, as they do in real-world live rankings. Online extensions of our problem may present new risks for status quo mechanisms, since producer strategies could incorporate time-varying and data-dependent model release decisions. It would also present challenges for the naive extension of the YRWR mechanism to the online setting, since allowing model producers to delete and add models to their rankings could still allow sequential clone attacks. For example, a model provider could sequentially submit clones to the mechanism, observe the performance of each clone, and then inserting a new clone above all preceding clones, until one of the clones achieved a sufficiently high rank. Our model could also be extended in other ways to more closely match current practice on leading platforms like Arena. For example, one could consider non-uniform allocations of votes over pairs of models (perhaps using this to improve statistical efficiency or encourage desirable model producer behavior). Also, one could consider analyzing the impacts of style control, where the platform “controls for” various aspects of model behavior that influence preference data but are misaligned with model quality, like the length of responses. Our model could also be extended to analyze the kind of utility functions of producers that lead to clone non-robustness. Key to our results is the idea that producers are optimizing for the maximum performance of their models. We expect that similar results should hold for general classes of convex or super-linearly increasing (in model rank) producer utility functions, where maximum performance matters more than average performance. More generally, one could consider more flexible utility functions over ranking outcomes than the ones studied here in order to better reflect producer incentives; for example, one could assign (potentially declining) scores to the top d rank positions, reflecting that there is benefit to being among the top models. Methods to detect clones with only black-box access could also feature into mechanisms that could dissuade clone submission in practice. And finally, platforms need not explicitly rank models; grouping them into equivalence classes or producing some entirely different kind of assessment would change the incentives for strategic behavior. As these arenas increasingly guide the development and adoption of AI models, developing mechanisms that are resilient to strategic manipulation is essential to ensuring that rankings remain a trustworthy and accurate signal for the entire community. Acknowledgments The authors thank Nathan Jo, Juanky Perdomo, Jann Spiess, Tijana Zrnic and the participants of the Stanford Causal Inference seminar for helpful discussions and feedback on this work. References Alibaba Cloud Community (2025) Note: Alibaba Cloud Community External Links: Link Cited by: §1. S. Bates, M. I. Jordan, M. Sklar, and J. A. Soloff (2024) Incentive-Theoretic Bayesian Inference for Collaborative Science. arXiv. Note: arXiv:2307.03748 [stat] External Links: Link, Document Cited by: §1. V. Bentkus (2005) A Lyapunov-type Bound in Rd. Theory of Probability & Its Applications 49 (2), p. 311–323. Note: _eprint: https://doi.org/10.1137/S0040585X97981123 External Links: Link, Document Cited by: Theorem C.4. R. A. Bradley and M. E. Terry (1952) Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39 (3/4), p. 324–345. External Links: ISSN 0006-3444, Link, Document Cited by: §2. Y. Chen, G. Zhang, and M. Hardt (2026) Leaderboard Incentives: Model Rankings under Strategic Post-Training. arXiv. Note: arXiv:2603.08371 [cs] External Links: Link, Document Cited by: §1. W. Chi, V. Chen, A. N. Angelopoulos, W. Chiang, A. Mittal, N. Jain, T. Zhang, I. Stoica, C. Donahue, and A. Talwalkar (2025) Copilot Arena: A Platform for Code LLM Evaluation in the Wild. arXiv. Note: arXiv:2502.09328 [cs] External Links: Link, Document Cited by: §1. W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, and I. Stoica (2024) Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv. Note: arXiv:2403.04132 [cs] External Links: Link, Document Cited by: §1, §1, §3, §4.4. D. Donoho (2017) 50 Years of Data Science. Journal of Computational and Graphical Statistics 26 (4), p. 745–766. Note: _eprint: https://doi.org/10.1080/10618600.2017.1384734 External Links: ISSN 1061-8600, Link, Document Cited by: §1. D. Donoho (2024) Data Science at the Singularity. Harvard Data Science Review 6 (1) (en). External Links: Link, Document Cited by: §1. G. Glover (2025) Note: Barron’s (updated Jan 27, 2025) External Links: Link Cited by: §1. P. Gölz, N. Haghtalab, and K. Yang (2025) Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?. arXiv. Note: arXiv:2505.23749 [cs] External Links: Link, Document Cited by: §1. J. Y. Huang, Y. Shen, D. Wei, and T. Broderick (2026) Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings. arXiv. Note: arXiv:2508.11847 [stat] External Links: Link, Document Cited by: §1. Y. Huang, M. Nasr, A. Angelopoulos, N. Carlini, W. Chiang, C. A. Choquette-Choo, D. Ippolito, M. Jagielski, K. Lee, K. Z. Liu, I. Stoica, F. Tramer, and C. Zhang (2025) Exploring and Mitigating Adversarial Manipulation of Voting-Based Leaderboards. arXiv. Note: arXiv:2501.07493 [cs] External Links: Link, Document Cited by: §1. D. Jiang, M. Ku, T. Li, Y. Ni, S. Sun, R. Fan, and W. Chen (2024) GenAI Arena: An Open Evaluation Platform for Generative Models. Advances in Neural Information Processing Systems 37, p. 79889–79908 (en). External Links: Link Cited by: §1. M. G. Kendall (1938) A new measure of rank correlation. Biometrika 30 (1–2), p. 81–93. External Links: Document Cited by: §4.6. R. Min, T. Pang, C. Du, Q. Liu, M. Cheng, and M. Lin (2025) Improving Your Model Ranking on Chatbot Arena by Vote Rigging. arXiv. Note: arXiv:2501.17858 [cs] External Links: Link, Document Cited by: §1. M. Miroyan, T. Wu, L. King, T. Li, J. Pan, X. Hu, W. Chiang, A. N. Angelopoulos, T. Darrell, N. Norouzi, and J. E. Gonzalez (2025) Search Arena: Analyzing Search-Augmented LLMs. arXiv. Note: arXiv:2506.05334 [cs] External Links: Link, Document Cited by: §1. R. Morrison (2024) Note: Tom’s Guide External Links: Link Cited by: §1. F. Nazarov (2003) On the Maximal Perimeter of a Convex Set in $R^n$with Respect to a Gaussian Measure. In Geometric Aspects of Functional Analysis: Israel Seminar 2001-2002, V. D. Milman and G. Schechtman (Eds.), p. 169–187 (en). External Links: ISBN 978-3-540-36428-3, Link, Document Cited by: Theorem C.5. A. D. Procaccia, B. Schiffer, and S. Zhang (2025) Clone-Robust AI Alignment. arXiv. Note: arXiv:2501.09254 [cs] External Links: Link, Document Cited by: §1, §1. F. C. Shi, M. J. Wainwright, and S. Bates (2025) Instance-Adaptive Hypothesis Tests with Heterogeneous Agents. arXiv. Note: arXiv:2510.21178 [cs] External Links: Link, Document Cited by: §1. G. Simons and Y. Yao (1999) Asymptotics When the Number of Parameters Tends to Infinity in the Bradley-Terry Model for Paired Comparisons. The Annals of Statistics 27 (3), p. 1041–1060. External Links: ISSN 0090-5364, Link Cited by: §2. S. Singh, Y. Nan, A. Wang, D. D’Souza, S. Kapoor, A. Üstün, S. Koyejo, Y. Deng, S. Longpre, N. A. Smith, B. Ermis, M. Fadaee, and S. Hooker (2025) The Leaderboard Illusion. arXiv. Note: arXiv:2504.20879 [cs] External Links: Link, Document Cited by: §1, footnote 1, footnote 3. A. Siththaranjan, C. Laidlaw, and D. Hadfield-Menell (2024) Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. arXiv. Note: arXiv:2312.08358 [cs] External Links: Link, Document Cited by: Lemma B.1, Appendix B, §1. J. Spiess (2025) Optimal Estimation When Researcher and Social Preferences Are Misaligned. Econometrica 93 (5), p. 1779–1810 (en). External Links: ISSN 0012-9682, Link, Document Cited by: §1. B. Su, J. Zhang, N. Collina, Y. Yan, D. Li, K. Cho, J. Fan, A. Roth, and W. Su (2025) The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review. arXiv. Note: arXiv:2408.13430 [stat] External Links: Link, Document Cited by: §1. W. J. Su (2022) You Are the Best Reviewer of Your Own Papers: An Owner-Assisted Scoring Mechanism. arXiv. Note: arXiv:2110.14802 [cs] External Links: Link, Document Cited by: §1. Y. Zhao, K. Zhang, T. Hu, S. Wu, R. Le Bras, Y. Liu, X. Tang, J. C. Chang, J. Dodge, J. Bragg, et al. (2025) SciArena: an open evaluation platform for non-verifiable scientific literature-grounded tasks. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: §1. Appendix A Additional empirical results In this section, we provide additional empirical results corresponding to our semisynthetic experiments on Arena data. In Figure˜4, we plot the rank improvement attained on average by adding an additional clone. The horizontal axis is the ground truth score in our simulations (i.e., the score assigned by Arena), and the vertical axis is the number of positions the average (across simulations) of the maximum of the ranks of the clones minus the average rank of the model without a clone. The mean rank difference varies widely across models and between leaderboards. There are some models and leaderboards, like Expert, Multiturn and Coding, where cloning a single model can produce a ranking increase of around 8 positions on average. The fact that there may be large benefits to clones on these leaderboards in particular may be related to two factors. First, there are many fewer votes per pair of models on these more specialized leaderboards: each of expert, multiturn and coding have fewer than 30 votes per pair of models on average. Second, the models that benefit from clones in these leaderboards have scores that are clustered together with several other models, which means that small increases to fitted scores yield large increases in position on the leaderboard. By contrast, several leaderboards, like Text, exhibit much smaller benefits to clones. This is because there are relatively more voters per pair of models. We also note that the benefits of clones mostly disappear for the very best models (furthest to the right on each plot) and the very worst models (furthest to the left on each plot). This may be related to the fact that model qualities are less concentrated at the tails. Also, the very best models can only benefit from clones insofar as they are not ranked first in the simultations without clones, which creates a ceiling for the benefits that can be attained via clones. For example, a model that is never ranked below 3rd place without clones can only improve by up to two positions. We plot the analogous results for the YRWR mechanism in Figure˜5. In each of the panels, the benefits of clones under YRWR average around zero, although there are some models that can see rank improvement of up to around two positions due to the vote reweighting effect. The leaderboards for which some models still see benefits of clones are those for which there are the most incentives for clones in the status quo mechanism: this is a result of the fact that the vote reweighting effect is larger when the number of votes per model is small. Figure 4: Rank difference between submitting one clone and no clones under the sq mechanism. Figure 5: Rank difference between submitting one clone and no clones under the yrwr mechanism. Appendix B Preliminary Lemmata The following theorem is rephrased from Siththaranjan et al. (2024) in our notation. We provide a proof for completeness. Lemma B.1 (Siththaranjan et al. (2024), Theorem 3.1). The rankings produced by BT-MLE and Borda counts are equivalent under equal matchup counts. Formally, let s be the number of votes between each pair of candidates i,ji,j. Let σBT−MLEσ^BT-MLE be the ranking induced by fitting a Bradley-Terry model via MLE on v as in Algorithm˜1. Let σBCσ^BC be the ranking induced by the Borda count on v. I.e., for two candidates j,j′j,j j≻BCj′⇔∑ℓ∈ℳ,ℓ≠jvjℓ<∑ℓ∈ℳ,ℓ≠j′vj′ℓ. j _BCj _ , ≠ jv_j < _ , ≠ j v_j . Then σBC(u)=σBT−MLE(u)σ^BC(u)=σ^BT-MLE(u) for all u=1,2,…u=1,2,…. As we will see in the proof, the fact that matchups are evenly distributed over pairs of models is important to the proof: if matchups are not evenly distributed, the result may not hold. Proof. The claim is equivalent to showing that for any two models j,j′j,j , ∑ℓ∈ℳ,ℓ≠jvjℓ<∑ℓ∈ℳ,ℓ≠j′vj′ℓ⇔R^j<R^j′ _ , ≠ jv_j < _ , ≠ j v_j R_j< R_j Now, recall that R^=argmaxR∈ℝm∑j,j′∈ℳ;j≠j′vj≻j′log(exp(Rj)exp(Rj)+exp(Rj′)). R= _R ^m _j,j ;j≠ j v_j j \! ( (R_j) (R_j)+ (R_j ) ). By concavity of the objective, it holds ∇R∑j,j′∈ℳ;j≠j′vj≻j′log(exp(Rj)exp(Rj)+exp(Rj′))|R^=0. _R _j,j ;j≠ j v_j j \! ( (R_j) (R_j)+ (R_j ) ) |_ R=0. Also, we can rewrite each partial derivative as ∂Rj∑j,j′∈ℳ;j≠j′vj≻j′log(exp(Rj)exp(Rj)+exp(Rj′)) ∂ R_j _j,j ;j≠ j v_j j \! ( (R_j) (R_j)+ (R_j ) ) = = ∑j′∈ℳ,j′≠j∂Rjvj≻j′log(exp(Rj)exp(Rj)+exp(Rj′))+(s−vj≻j′)log(exp(Rj′)exp(Rj′)+exp(Rj)) _j ,j ≠ j ∂ R_jv_j j \! ( (R_j) (R_j)+ (R_j ) )+(s-v_j j ) \! ( (R_j ) (R_j )+ (R_j) ) = = ∑j′∈ℳ,j′≠jvj≻j′−s1+exp(Rj′−Rj). _j ,j ≠ jv_j j - s1+ (R_j -R_j). Thus, we have 1s∑j′∈ℳ,j′≠jvj≻j′=∑j′∈ℳ,j′≠j11+exp(R^j′−R^j) 1s _j ,j ≠ jv_j j = _j ,j ≠ j 11+ ( R_j - R_j) by applying the fact that the partial derivative is zero at R=R^R= R. Now, the LHS is the (normalized) Borda count. Thus, for any two models j,j′j,j ∑ℓ∈ℳ,ℓ≠jvjℓ<∑ℓ∈ℳ,ℓ≠j′vj′ℓ _ , ≠ jv_j < _ , ≠ j v_j ⇔ ∑ℓ∈ℳ,ℓ≠j11+exp(R^ℓ−R^j)<∑ℓ∈ℳ,ℓ≠j′11+exp(R^ℓ−R^j′) _ , ≠ j 11+ ( R_ - R_j)< _ , ≠ j 11+ ( R_ - R_j ) ⇔ 21+exp(Rj′−Rj)−1+∑ℓ∈ℳ,ℓ≠j,ℓ≠j′11+exp(R^ℓ−R^j)−11+exp(R^ℓ−R^j′)<0. 21+ (R_j -R_j)-1+ _ , ≠ j, ≠ j 11+ ( R_ - R_j)- 11+ ( R_ - R_j )<0. Finally, note that (21+exp(Rj′−Rj)−1)+∑ℓ∈ℳ,ℓ≠j,ℓ≠j′(11+exp(R^ℓ−R^j)−11+exp(R^ℓ−R^j′))<0 ( 21+ (R_j -R_j)-1 )+ _ , ≠ j, ≠ j ( 11+ ( R_ - R_j)- 11+ ( R_ - R_j ) )<0 ⇔ Rj−Rj′<0 R_j-R_j <0 since each term inside the large parentheses is positive if Rj>Rj′R_j>R_j and negative if Rj<Rj′R_j<R_j . ∎ Appendix C New Competitor Effect Analysis In this section, we provide our workhorse stability lemma, which bounds the new competitor effect. Intuitively, it says that if the total number of votes is sufficiently large, the distributions of model scores before and after the introduction of an model cannot be too large. Since this result may be of independent interest, we state the result for the general Bradley-Terry model and (re)introduce notation: In this section, we will consider two Bradley-Terry estimation problems: 1. an MLE R^m R^m fit on a set of m≥2m≥ 2 candidates, and 2. an MLE R^m+1 R^m+1 fit on a set of m+1m+1 candidates, where the first m are the same as in (1). We’l index candidates j=1,2,…,m+1j=1,2,…,m+1, and when fitting the MLE, we will enforce the identifiability constraint that ∑j∈[m]R^jm=∑j∈[m]R^jm+1=0, _j∈[m] R_j^m= _j∈[m] R_j^m+1=0, i.e., the first m entries of each vector must sum to zero. We’l call these identifiable subspaces m⟂1 _m, and it will be clear from context whether we are talking about the subspace in ℝmR^m or ℝm+1R^m+1, depending on whether we are working with the first or second BT estimation problems. Projecting onto m⟂1 _m ensures that R^m−R^1:m+1→0 R^m- R^m+1_1:m→ 0 as s→∞s→∞ (whereas some other identifiability constraint might lead to convergence to a non-zero constant). We will let R∈ℝmR ^m denote the ground-truth qualities of the original m models (where, without loss of generality, ∑j∈[m]Rj=0 _j∈[m]R_j=0), and we’l write R^1:m+1 R^m+1_1:m for the entries of R^m+1 R^m+1 corresponding to the first m candidates. For a matrix A, we will similarly use index slice notation so that A1:i,1:jA_1:i,1:j is the first i rows and j columns of A. We’l make use of Section˜2 in this section, which we restate here: * We’l write S=s(m2)S=s m2 to denote the total number of pairwise comparisons, where s is the number of comparisons per pair. Define PmP_m to be the probability measure of R^m R^m on m⟂⊂ℝm1_m ^m, and define Pm′P _m to be the probability measure of R^1:m+1 R^m+1_1:m for m⟂⊂ℝm+11_m ^m+1. Lemma C.1 (New Competitor Effect). Let C be the set of convex events measurable with respect to PmP_m and Pm′P _m. Then, under Section˜2, there exist constants C>0C>0 and Cmm≥2\C_m\_m≥ 2 such that for all m≥2m≥ 2 and s≥1s≥ 1, supA∈|Pm(A)−Pm′(A)|≤Cm+Cms. _A \; |P_m(A)-P _m(A) |\;≤\; C m+ C_m s. Proof. At a high level, our goal will be to upper bound the convex set distance between PmP_m and Pm′P_m by the sum of three convex set distances: the distance between PmP_m and its Gaussian approximation, the distance between Pm′P_m and its Gaussian approximation, and the distance between the two Gaussian approximations. We then prove the Gaussian approximation error bounds in Lemma˜C.2 and the distance between Guassians in Lemma˜C.3 which immediately yield the inequality in the lemma. Formally, let P¯m P_m denote the law of the centered and scaled estimator S(R^m−R) S( R^m-R) on m⟂1 _m, and let P¯m′ P _m denote the law of S(R^1:m+1−R) S( R^m+1_1:m-R) on the same subspace. By S S-consistency and asymptotic normality of the MLE, there exist Normal laws Gm,Gm′G_m,G_m on m⟂1 _m such that P¯m→Gm P_m d→G_m and P¯m′→Gm′ P_m d→G_m in S. Gm,Gm′G_m,G_m are given by (0,Im−1),N(0,(Im−′1)1:m,1:m)N(0,I_m^-1),N(0,(I_m^ -1)_1:m,1:m) where Im,Im′I_m,I_m are the respective Fisher information matrices projected onto m⟂1_m . Denote the convex-set distance as d(P,P′)=def.supA∈|P(A)−P′(A)|.d_C(P,P ) def.= _A |P(A)-P (A)|. Observe that, by the triangle inequality, d(P¯m,P¯m′)≤d(P¯m,Gm)+d(Gm,Gm′)+d(Gm′,P¯m′).d_C( P_m, P _m)≤ d_C( P_m,G_m)+d_C(G_m,G _m)+d_C(G _m, P _m). Lemma C.3 gives d(Gm,Gm′)≤Cmd_C(G_m,G _m)≤ C m for a universal constant C>0C>0. Lemma C.2 yields d(P¯m,Gm)≤Cms,d(P¯m′,Gm′)≤Cm′s,d_C( P_m,G_m)≤ C_m s, d_C( P _m,G _m)≤ C _m s, for constants Cm,Cm′C_m,C_m depending only on m. Combining these bounds gives d(P¯m,P¯m′)≤Cm+Cm+Cm′s.d_C( P_m, P _m)≤ C m+ C_m+C _m s. Finally, since R^m↦S(R^m−R) R^m S( R^m-R) is an invertible affine map on the identifiable subspace and C is the class of convex sets on that subspace, the convex-set distance is invariant under applying the same invertible affine map to both distributions. Therefore the same bound holds for the unscaled laws PmP_m and Pm′P _m, which proves the claim. ∎ Lemma C.2 (Gaussian approximation error for the BT–MLE). Under Section˜2, there exists a universal constant C>0C>0 such that maxd(P¯m,Gm),d(P¯m′,Gm′)≤Cm3/4s+o(1s). \; \d_C( P_m,\,G_m),\,d_C( P_m ,\,G_m ) \≤ C\, m^3/4 s+o\! ( 1 s ). Lemma C.3 (Gaussian stability under adding one model). Under Section˜2, there exists a universal constant C>0C>0 such that d(Gm,Gm′)≤Cm.d_C(G_m,G _m)≤ C m. We now proceed with the proofs for the above two lemmas. We first (re)establish general notation and basic facts we will use throughout this section. Let the win probability between candidate i and j be pi≻j=expRiexpRj+expRip_i j= R_i R_j+ R_i and p^i≻j p_i j the same quantity substituting R R for R. Let vi≻j∼Bin(s,pi≻j)v_i j (s,p_i j) count the wins of model i over model j. Let vi≻j(k)∼Ber(pi≻j)v_i j^(k) (p_i j) denote the indicator outcome of the k-th comparison. Let Lm(R)L_m(R) be the log-likelihood for m candidates: Lm(R) L_m(R) =∑j,j′∈[m];j≠j′vj≻j′log(11+exp(Rj′−Rj)) = _j,j ∈[m];j≠ j v_j j \! ( 11+ (R_j -R_j) ) =∑j,j′∈[m];j≠j′vj≻j′logpj≻j′ = _j,j ∈[m];j≠ j v_j j p_j j Let ℓm(R)=∇Lm(R) _m(R)=∇ L_m(R) be the score function (first derivative with respect to R), i.e., for j∈[m]j∈[m], ℓm(R)j= _m(R)_j= ∑j′∈[m];j′≠jvj≻j′−spj≻j′ _j ∈[m];j ≠ jv_j j -sp_j j (C.1) and let Hm(R)=∇2Lm(R)H_m(R)=∇^2L_m(R) be the Hessian (second derivative), i.e., for j,j′∈[m]j,j ∈[m], Hm(R)j,j′=−s∑ℓ∈[m];ℓ≠jpj≻ℓ(1−pj≻ℓ)if j=j′spj≻j′(1−pj≻j′)otherwise=−s∑1≤j<j′≤mpj≻j′(1−pj≻j′)(ej−ej′)(ej−ej′)⊤. aligned H_m(R)_j,j &= cases-s _ ∈[m]; ≠ jp_j (1-p_j )&if j=j \\ sp_j j (1-p_j j )&otherwise cases\\ &=-s _1≤ j<j ≤ mp_j j (1-p_j j )\,(e_j-e_j )(e_j-e_j ) . aligned (C.2) where eje_j denotes the jjth standard basis vector. Let S=s(m2)=O(sm2)S=s m2=O(sm^2) be the total number of pairwise comparisons. Let Im=−[Hm(R)]|m⟂I_m=-E[H_m(R)] |_1_m be the Fisher information matrix for the m-model Bradley-Terry, projected onto the rank-(m−1)(m-1) subspace. Let Im+1I_m+1 be the Fisher information for the (m+1)(m+1)-model Bradley-Terry projected onto the rank-m subspace. C.1 Proof of Lemma˜C.2 We will just show the bound holds for P¯m,Gm P_m,G_m; the argument for P¯m′,Gm′ P_m ,G_m is identical. Our proof proceeds as follows: 1. We’l first write the normalized deviation of R^m R^m from R using Taylor’s theorem, so that it is (up to higher-order terms) equal to the sum of independent score contributions. 2. We’l then use this linear approximation to break the convex set distance into three terms, which we can analyze separately. 3-5. Analyses of each of the terms. Step 1. Applying Taylor’s theorem to the score function ℓ(⋅) (·) around the MLE R^m R^m, we have ℓm(R^m)=ℓm(R)+Hm(R~)(R^m−R) _m( R^m)= _m(R)+H_m( R)( R^m-R) for R~ R on the line segment between R and R^m R^m. Also, 0=ℓm(R^m)0= _m( R^m) by the fact that the MLE is a maximum. Applying this fact and rearranging, we have R^m−R=−Hm(R~)−1ℓm(R). R^m-R=-H_m( R)^-1 _m(R). Scaling by S S and adding/subtracting the Fisher information S(R^m−R)=S⋅Im−1ℓm(R)⏟Linear Approximation Term Wm+S(−Hm(R~)−1−Im−1)ℓm(R)⏟Remainder rm S( R^m-R)= S· I_m^-1 _m(R)_Linear Approximation Term W_m+ S (-H_m( R)^-1-I_m^-1 ) _m(R)_Remainder r_m Step 2. We now break the convex set distance into three terms. First, we prove an upper bound. For A∈A , define At=x:infa∈A‖x−a‖2≤t A^t=\x\;:\; _a∈ A\|x-a\|_2≤ t\ to be the t-neighborhood of A. Observe for all t>0t>0, P(S(R^m−R)∈A) P( S( R^m-R)∈ A) =P(Wm+rm∈A) =P(W_m+r_m∈ A) (Definition of Wm,rmW_m,r_m) ≤P(Wm∈At∪‖rm‖2≥t) ≤ P(W_m∈ A^t∪\|r_m\|_2≥ t) (Wm+rm∈A⊆Wm∈At∪‖rm‖2≥tW_m+r_m∈ A W_m∈ A^t∪\|r_m\|_2≥ t) ≤P(Wm∈At)+P(‖rm‖2≥t). ≤ P(W_m∈ A^t)+P(\|r_m\|_2≥ t). (Union bound) Subtracting P(Z~∈At)P( Z∈ A^t) from both sides, we have P(S(R^m−R)∈A)−P(Z~∈At) P( S( R^m-R)∈ A)-P( Z∈ A^t) ≤P(Wm∈At)−P(Z~∈At)+P(‖rm‖2≥t) ≤ P(W_m∈ A^t)-P( Z∈ A^t)+P(\|r_m\|_2≥ t) ⟹ P(S(R^m−R)∈A)−P(Z~∈A) P( S( R^m-R)∈ A)-P( Z∈ A) ≤|P(Wm∈At)−P(Z~∈At)|+P(Z~∈At∖A)+P(‖rm‖2≥t) ≤|P(W_m∈ A^t)-P( Z∈ A^t)|+P( Z∈ A^t A)+P(\|r_m\|_2≥ t) (C.3) where the implication follows from the fact that P(Z~∈At)=P(Z~∈A)+P(Z~∈At∖A)P( Z∈ A^t)=P( Z∈ A)+P( Z∈ A^t A) and taking the absolute value. Note that At∈CA^t∈ C: the Minkowski sum of convex events is a convex event. We can write the lower bound analogously. Let us overload notation and write A−t=a∈A:infx∉A‖x−a‖2≥t. A^-t=\a∈ A\;:\; _x ∈ A\|x-a\|_2≥ t\. Observe for all t>0t>0 that P(S(R^m−R)∈A) P( S( R^m-R)∈ A) =P(Wm+rm∈A) =P(W_m+r_m∈ A) (Definition of Wm,rmW_m,r_m) ≥P(Wm∈A−t)−P(‖rm‖2≥t). ≥ P(W_m∈ A^-t)-P(\|r_m\|_2≥ t). (Union bound and rearranging) Then P(S(R^m−R)∈A)−P(Z~∈A−t) P( S( R^m-R)∈ A)-P( Z∈ A^-t) ≥P(Wm∈A−t)−P(Z~∈A−t)−P(‖rm‖2≥t) ≥ P(W_m∈ A^-t)-P( Z∈ A^-t)-P(\|r_m\|_2≥ t) ⟹ P(S(R^m−R)∈A)−P(Z~∈A) P( S( R^m-R)∈ A)-P( Z∈ A) ≥−|P(Wm∈A−t)−P(Z~∈A−t)|−P(Z~∈A∖A−t)−P(‖rm‖2≥t). ≥- |P(W_m∈ A^-t)-P( Z∈ A^-t) |-P( Z∈ A A^-t)-P(\|r_m\|_2≥ t). (C.4) Similarly, note that A−t∈A^-t . In steps 3-5, we bound each of the terms in the right-hand side of Equation˜C.3. In particular, plugging in the RHS expressions in Equations˜C.5, C.6 and C.7 yields, for t>0t>0 P(S(R^m−R)∈A)−P(Z~∈A)≤Cm3/4s+Cts−1/2m−1/4+mexp(−Cst2m2). P( S( R^m-R)∈ A)-P( Z∈ A)≤ C m^3/4 s+Cts^-1/2m^-1/4+m ( -Cst^2m^2 ). Choosing t=O(m)t=O(m) yields P(S(R^m−R)∈A)−P(Z~∈A)≤Cm3/4s. P( S( R^m-R)∈ A)-P( Z∈ A)≤ C m^3/4 s. (where as usual the constant C across inequalities may change). The corresponding terms in Equation˜C.4 are bounded using the same argument, and yield the same bound. Step 3. We will show for a generic convex event A, there exists a universal constant C such that |P(Wm∈A)−P(Z~∈A)|≤Cm3/4s. |P(W_m∈ A)-P( Z∈ A)|≤ C m^3/4 s. (C.5) Plugging in AtA^t or A−tA^-t yields our upper bound on the first terms in Equations˜C.3 and C.4, respectively. To do this, we will first prove an approximation bound on the score function ℓm(R) _m(R) and then translate it into a bound on SIm−1ℓm(R) SI^-1_m _m(R). For the bound on ℓm(R) _m(R), observe that the score function at the true R is a sum of S=s(m2)S=s m2 independent score contributions ℓm(R)=∑i<j∑k=1sψij(k)=∑i<j∑k=1s(vi≻j(k)−pi≻j)(ei−ej),[ψij(k)]=0, _m(R)= _i<j _k=1^s _ij^(k)= _i<j _k=1^s(v_i j^(k)-p_i j)(e_i-e_j), [ _ij^(k)]=0, where eje_j denotes the j-th standard basis vector. Since each term is a mean-zero independent random variable, we can apply the following theorem to bound the Gaussian approximation error on the linear term. Theorem C.4 (Theorem 1.1 Bentkus (2005)). Suppose X1,…,Xn∈ℝdX_1,…,X_n ^d are independent and Xi=0EX_i=0 for all i. Let S=∑iXiS= _iX_i and define Σ=Var(S) =Var(S). Let Z∼(0,Σ)Z (0, ). There exists a universal constant c such that supA∈|P(S∈A)−P(Z∈A)|≤cd1/4β _A |P(S∈ A)-P(Z∈ A) |≤ cd^1/4β where β=∑i=1n‖Σ−1/2Xi‖23. β= _i=1^n\| ^-1/2X_i\|_2^3. Since Cov(ℓm)=ImCov( _m)=I_m, the target Gaussian is Z∼(0,Im)Z (0,I_m). Plugging this into the bound from Theorem˜C.4 yields supA∈|P(ℓm∈A)−P(Z∈A)|≤c(m−1)1/4β _A |P( _m∈ A)-P(Z∈ A)|≤ c(m-1)^1/4β where β=∑i<j∑k=1s‖Im−1/2ψij(k)‖23. β= _i<j _k=1^sE\|I_m^-1/2 _ij^(k)\|_2^3. Now it suffices to bound β. Observe that (vi≻j(k)−pi≻j))∈[−1,1](v_i j^(k)-p_i j))∈[-1,1] so ‖ψij(k)‖23 \| _ij^(k)\|_2^3 =∥(vi≻j(k)−pi≻j))(ei−ej)∥23 =E\|(v_i j^(k)-p_i j))(e_i-e_j)\|_2^3 ≤‖ei−ej‖23=23 ≤\|e_i-e_j\|_2^3= 2^3 By Lemma˜C.6, we also have ‖Im−1/2‖op≤O(1sm)\|I_m^-1/2\|_op≤ O ( 1 sm ) which implies β≤∑i<j∑k=1sO((sm)−3/2)=O(sm2)O((sm)−3/2)=O(m1/2)O(s−1/2).β≤ _i<j _k=1^sO((sm)^-3/2)=O(sm^2)O((sm)^-3/2)=O(m^1/2)O(s^-1/2). Putting the norm upper bounds together, we obtain supA∈|P(ℓm(R)∈A)−P(Z∈A)|≤O(m3/4s). _A |P( _m(R)∈ A)-P(Z∈ A)|≤ O ( m^3/4 s ). Finally, we must translate these bounds into bounds on events for SIm−1ℓm(R) SI_m^-1 _m(R) (rather than ℓm(R) _m(R)). Since convex sets are closed under linear maps, we can define the target Guassian for the scaled linear term to be Z~∼(0,SIm−1) Z (0,SI_m^-1) since Cov(SIm−1ℓm(R))=SIm−1Cov(ℓm(R))Im−1=SIm−1Cov( SI_m^-1 _m(R))=SI_m^-1Cov( _m(R))I_m^-1=SI_m^-1 and thus have supA∈|P(Wm∈A)−P(Z~∈A)|=supA∈|P(ℓm(R)∈A)−P(Z∈A)|≤O(m3/4s). _A |P(W_m∈ A)-P( Z∈ A)|= _A |P( _m(R)∈ A)-P(Z∈ A)|≤ O ( m^3/4 s ). Step 4. We will show P(Z~∈At∖A)≤Cts−1/2m−1/4. P( Z∈ A^t A)≤ Cts^-1/2m^-1/4. (C.6) To do so, we apply the following theorem: Theorem C.5 (Nazarov (2003)). There exist universal constants 0<C1<C2∈ℝ0<C_1<C_2 such that, for any mean-zero multivariate Gaussian measure F on ℝdR^d with variance matrix W, it holds C1‖W‖F≤supA∈,t>0F(A∖At)t≤C2‖W‖F. C_1 \|W\|_F≤ _A ,t>0 F(A A^t)t≤ C_2 \|W\|_F. In particular, we have ‖Σ‖F≤m‖Σ‖op≤mO(1/(sm))\| \|_F≤ m\| \|_op≤ mO(1/(sm)) where the last inequality follows from Lemma˜C.6. Thus, there is a universal constant C such that supQ∈,h>0P(Z~∈Qh∖Q)h≤Cs−1/2m−1/4 _Q ,h>0 P( Z∈ Q^h Q)h≤ Cs^-1/2m^-1/4 Thus, for fixed t, we have P(Z~∈At∖A) P( Z∈ A^t A) =t⋅P(Z~∈At∖A)t =t· P( Z∈ A^t A)t ≤t⋅supQ∈,h>0P(Z~∈Qh∖Q)h ≤ t· _Q ,h>0 P( Z∈ Q^h Q)h ≤Cts−1/2m−1/4. ≤ Cts^-1/2m^-1/4. Step 5. Finally, we establish P(‖rm‖2≥t)≤mexp(−Cst2m2) P(\|r_m\|_2≥ t)≤ m ( -Cst^2m^2 ) (C.7) Notice ‖rm‖2 \|r_m\|_2 =‖S(−Hm(R~)−1−Im−1)ℓm(R)‖2 = \| S (-H_m( R)^-1-I_m^-1 ) _m(R) \|_2 ≤S‖−Hm(R~)−1−Im−1‖op‖ℓm(R)‖2 ≤ S \|-H_m( R)^-1-I_m^-1 \|_op \| _m(R) \|_2 ≤S(‖Hm(R~)−1‖op+‖Im−1‖op)‖ℓm(R)‖2 ≤ S ( \|H_m( R)^-1 \|_op+ \|I_m^-1 \|_op ) \| _m(R) \|_2 Now, we bound each of these terms. From Lemma˜C.6, we have ‖Im−1‖op=O((sm)−1)\|I_m^-1\|_op=O((sm)^-1). Moreover, using the same proof as for Lemma˜C.6, under the assumption that maxj,j′|R^j−R^j′| _j,j | R_j- R_j | is bounded and the fact that R~ R is a convex combination of R,R^R, R, it holds ‖Hm(R~)−1‖op=O((sm)−1)\|H_m( R)^-1\|_op=O((sm)^-1). Thus, ‖rm‖2≤O(s−1)‖ℓm(R)‖2. \|r_m\|_2≤ O(s^-1)\| _m(R)\|_2. Moreover, from Lemma˜C.7, for all t, we have P(‖ℓm(R)‖≥t)≤mexp(−t23sm2) P(\| _m(R)\|≥ t)≤ m (- t^23sm^2 ) Plugging in O(s⋅t)O(s· t) for t, we have P(‖rm‖≥t) P(\|r_m\|≥ t) =mexp(−C⋅s⋅t2m2). =m ( -C· s· t^2m^2 ). ∎ C.2 Proof of Lemma˜C.3 Let Σ=Im−1 =I_m^-1 and Σ′=(Im+1−1)1:m,1:m =(I_m+1^-1)_1:m,1:m. By asymptotic normality (as s→∞s→∞) of the MLE, we can write out Gm,Gm′G_m,G_m explicitly S(R^m−R)→(0,SΣ) S( R^m-R) (0,S ) S(R^1:m+1−R)→(0,SΣ′) S( R^m+1_1:m-R) (0,S ) Since convex-set distance is invariant under scaling by a constant, we define G~m=(0,Σ) G_m=N(0, ) G~m′=(0,Σ′) G_m =N(0, ) Then d(Gm,Gm′) d_C(G_m,G_m ) =d(G~m,G~m′) =d_C( G_m, G _m) ≤dTV(G~m,G~m′) ≤ d_TV( G_m, G_m ) (C.8) ≤12KL(G~m∥G~m′) ≤ 12KL( G_m\| G_m ) =12(tr(Σ′−1Σ)−(m−1)+logdetΣ′detΣ). = 12 (tr( -1 )-(m-1)+ ). (C.9) where the last line is the formula for the KL-divergence between two centered multivariate normal distributions. Thus, showing that the convex-set distance between Gm,Gm′G_m,G_m is small can be done in terms of Σ,Σ′ , by showing Equation˜C.9 is small. To bound Equation˜C.9, we will proceed with the following steps: 1. We will decompose Σ′=(Σ+K)−1 =( +K)^-1 for some matrix K determined by the change in log-likelihood due to the additional model. 2. We will establish ‖K‖op≤s/4\|K\|_op≤s/4 and use this to upper bound the expression for the KL-divergence between multivariate normals. Step 1. With an additional model added, we can write the log-likelihood as Lm+1(R1:m,Rm+1)=def.Lm(R1:m)⏟original comparisons+Lnew(R1:m,Rm+1)⏟comparisons involving the new model,L_m+1(R_1:m,R_m+1) def.= L_m(R_1:m)_original comparisons+ L_new(R_1:m,R_m+1)_comparisons involving the new model, where Lm(R1:m)L_m(R_1:m) is the log-likelihood for comparisons between the original m models, and Lnew(R1:m,Rm+1) L_new(R_1:m,R_m+1) =def.∑j∈[m]vj≻m+1logpj≻m+1+(s−vj≻m+1)log(1−pj≻m+1). def.= _j∈[m]v_j m+1 p_j m+1+(s-v_j m+1) (1-p_j m+1). By linearity of expectations and gradients, we then have, Im+1=−[∇R2Lm(R1:m)]−[∇R2Lnew(R1:m,Rm+1)].I_m+1=-E[∇^2_RL_m(R_1:m)]-E[∇^2_RL_new(R_1:m,R_m+1)]. We will further simplify these expressions by writing a block decomposition for each term. For the first term, LmL_m depends only on R1:mR_1:m, so −[∇2Lm(R)]=(Im000).-E[∇^2L_m(R)]= pmatrixI_m&0\\ 0&0 pmatrix. The second term can be written as −[∇R2Lnew(R1:m,Rm+1)]=(I1:m,1:mnewI1:m,(m+1)newI(m+1),1:mnewI(m+1),(m+1)new)-E[∇^2_RL_new(R_1:m,R_m+1)]= pmatrixI^new_1:m,1:m&I^new_1:m,(m+1)\\ I^new_(m+1),1:m&I^new_(m+1),(m+1) pmatrix where I1:m,1:mnew=def.−[∇R1:m2Lnew(R)], I^new_1:m,1:m def.=-E\! [∇^2_R_1:mL_new(R) ], I1:m,(m+1)new=def.−[∇R1:m∂Rm+1Lnew(R)], and I^new_1:m,(m+1) def.=-E\! [ _R_1:m ∂R_m+1L_new(R) ], and I(m+1),(m+1)new=def.−[∂2∂2Rm+1Lnew(R)]. I^new_(m+1),(m+1) def.=-E\! [ ∂^2∂^2R_m+1L_new(R) ]. Thus the Fisher information for the log-likelihood with the additional model is Im+1=(Im+I1:m,1:mnewI1:m,(m+1)newI(m+1),1:mnewI(m+1),(m+1)new)I_m+1= pmatrixI_m+I^new_1:m,1:m&I^new_1:m,(m+1)\\ I^new_(m+1),1:m&I^new_(m+1),(m+1) pmatrix Applying the block inversion formula yields Σ′=(Im+1−1)1:m,1:m=(Im+I1:m,1:mnew−I1:m,(m+1)new(I(m+1),(m+1)new)−1I(m+1),1:mnew)−1 =(I_m+1^-1)_1:m,1:m= (I_m+I^new_1:m,1:m-I^new_1:m,(m+1)(I^new_(m+1),(m+1))^-1I^new_(m+1),1:m )^-1 Finally, define K=def.I1:m,1:mnew−I1:m,(m+1)new(I(m+1),(m+1)new)−1I(m+1),1:mnewK def.=I^new_1:m,1:m-I^new_1:m,(m+1)(I^new_(m+1),(m+1))^-1I^new_(m+1),1:m so Σ′=(Im+K)−1. =(I_m+K)^-1. Step 2. Since K is the Schur complement of I(m+1),(m+1)new⪰0I_(m+1),(m+1)^new 0, it holds K⪰0K 0. We write out the partial derivatives under Bradley-Terry. Let di=s⋅pi,m+1(1−pi,m+1)≤s/4d_i=s· p_i,m+1(1-p_i,m+1)≤ s/4 and D=(d)D= diag(d). Then, −∇R1:m2Lnew(R)=s⋅D -∇^2_R_1:mL_new(R)=s· D −∇R1:m∂Rm+1Lnew(R)=−s⋅d, and - _R_1:m ∂R_m+1L_new(R)=-s· d, and −∂2∂2Rm+1Lnew(R)=s⊤d. - ∂^2∂^2R_m+1L_new(R)=s1 d. Thus, taking expectations (all terms are deterministic), we have I1:m,1:mnew=s⋅D I^new_1:m,1:m=s· D I1:m,(m+1)new=I(m+1),1:mnew⊤=−s⋅d I^new_1:m,(m+1)=I^new _(m+1),1:m=-s· d I(m+1),(m+1)new=s⋅⊤d I^new_(m+1),(m+1)=s·1 d Then we can rewrite K as K K =I1:m,1:mnew−I1:m,(m+1)new(I(m+1),(m+1)new)−1I(m+1),1:mnew =I^new_1:m,1:m-I^new_1:m,(m+1)(I^new_(m+1),(m+1))^-1I^new_(m+1),1:m =D−dd⊤d. =D- d 1 d. Moreover, D−K⪰0D-K 0, since for any vector x, it holds (x⊤d)(d⊤x)=(x⊤d)2≥0(x d)(d x)=(x d)^2≥ 0 and ⊤d>01 d>0. Also, let I be the identity matrix and u=D1/2/⊤du=D^1/21/ 1 d, D−dd⊤d D- d 1 d =D1/2(I−uu⊤)D1/2 =D^1/2 (I-u )D^1/2 So ‖K‖op≤‖D1/2‖op2‖I−uu⊤‖op.\|K\|_op≤\|D^1/2\|_op^2 \|I-u \|_op. Finally, ‖I−uu⊤‖op≤1 \|I-u \|_op≤ 1 since ‖u‖2=1\|u\|_2=1 so the non-zero eigenvalue of uu⊤u is 1, which means that the eigenvalues of I−uu⊤I-u are all 1 except one which is zero. Thus, ‖K‖op≤‖D‖op≤s4. \|K\|_op≤\|D\|_op≤ s4. (C.10) With these facts in hand, we now proceed to upper bound Equation˜C.9. Denote the relative perturbation matrix A=Σ1/2KΣ1/2=Im−1/2KIm−1/2.A= ^1/2K ^1/2=I_m^-1/2KI_m^-1/2. Observe that A⪰0A 0 since for any vector x, xTΣ1/2KΣ1/2x x^T ^1/2K ^1/2x =(xTΣ1/2)K(Σ1/2x)≥0 =(x^T ^1/2)K( ^1/2x)≥ 0 by positive semidefiniteness of K. Moreover, ‖A‖op≤‖Σ1/2‖op2‖K‖op≤O(1sm)s4≤O(1m)\|A\|_op≤\| ^1/2\|_op^2\|K\|_op≤ O ( 1sm ) s4≤ O ( 1m ) under Section˜2 by applying Lemma˜C.6 and Equation˜C.10. We first compute the trace term in Equation˜C.9. Observe, tr(Σ′−1Σ) ( -1 ) =tr((Im+K)Im−1) =tr ((I_m+K)I_m^-1 ) =tr()+tr(KIm−1) =tr(I)+tr(KI_m^-1) =(m−1)+tr(Im−1/2KIm−1/2) =(m-1)+tr(I_m^-1/2KI_m^-1/2) =(m−1)+tr(A) =(m-1)+tr(A) where the last equality applies the cyclic property of trace. Next, for the determinant term in Equation˜C.9, observe that Σ′=(Im+K)−1=Σ1/2(+A)−1Σ1/2 =(I_m+K)^-1= ^1/2(I+A)^-1 ^1/2 which implies det(Σ′)=det(Σ)det(()+A)−1). ( )= ( ) ((I)+A)^-1 ). Therefore, logdetΣ′detΣ=logdet((+A)−1)=−logdet(+A) = ((I+A)^-1 )=- (I+A) Substituting into the KL divergence formula yields KL(G~m∥G~m′)=12(tr(A)−logdet(+A))KL( G_m\| G_m )= 12 (tr(A)- (I+A) ) Let λ1,…,λm−1 _1,…, _m-1 denote the eigenvalues of A, then tr(A)=∑iλi,det(+A)=∏i(1+λi),tr(A)= _i _i, (I+A)= _i(1+ _i), and KL(G~m∥G~m′)=12∑i=1m−1(λi−log(1+λi)).KL( G_m\| G_m )= 12 _i=1^m-1 ( _i- (1+ _i) ). For all λ≥0λ≥ 0, it is true that log(1+λ)≥λ−λ22 (1+λ)≥λ- λ^22 and it follows that λi−log(1+λi)≤λi−(λi−λi22)=λi22 _i- (1+ _i)≤ _i- ( _i- _i^22 )= _i^22 Summing over all eigenvalues gives: KL(G~m∥G~m′)≤14∑i=1m−1λi2≤14‖A‖F2≤(m−1)4‖A‖op2≤O(1/m)KL( G_m\| G_m )≤ 14 _i=1^m-1 _i^2≤ 14\|A\|^2_F≤ (m-1)4\|A\|^2_op≤ O(1/m) Thus, d(Gm,Gm′)≤O(m−1/2)d_C(G_m,G_m )≤ O(m^-1/2) as desired. ∎ Lemma C.6. Under Section˜2, there exists a universal constant η∈(0,1/2)η∈(0,1/2) such that ‖Im‖op≥η(1−η)sm \|I_m\|_op≥η(1-η)sm (C.11) and hence ‖Im−1‖op≤(η(1−η)sm)−1\|I_m^-1\|_op≤(η(1-η)sm)^-1. Proof of Lemma˜C.6. Note that under Assumption 2, there exists a universal constant η=1/(1+exp(C))∈(0,1/2)η=1/(1+ (C))∈(0,1/2) such that pi≻j∈[η,1−η]p_i j∈[η,1-η] for all i≠ji≠ j, and hence pi≻j(1−pi≻j)≥η(1−η)>0p_i j(1-p_i j)≥η(1-η)>0. Therefore, for any x∈⟂x 1 such that ‖x‖2=1\|x\|_2=1, x⊤Imx=s∑i<jpi≻j(1−pi≻j)(xi−xj)2≥sη(1−η)∑i<j(xi−xj)2.x I_mx=s _i<jp_i j (1-p_i j )(x_i-x_j)^2\;≥\;sη(1-η) _i<j(x_i-x_j)^2. Since ∑i<j(xi−xj)2 _i<j(x_i-x_j)^2 =12∑i,jxi2+xj2−2xixj = 12 _i,jx_i^2+x_j^2-2x_ix_j =m∑ixi2−(∑ixi)2=m =m _ix_i^2- ( _ix_i )^2=m on ⟂1 , we obtain x⊤Imx≥sη(1−η)mx I_mx\;≥\;sη(1-η)m, which implies ‖Im‖op≥sη(1−η)m\|I_m\|_op≥ sη(1-η)m and ‖Im−1‖op≤1sη(1−η)m.\|I_m^-1\|_op≤ 1sη(1-η)m. ∎ Lemma C.7. Under Section˜2, there exists a universal constant such that, for all ε>0 >0 and s≥3log(2m/ε)/ηs≥ 3 (2m/ )/η, it holds with probability at least 1−ε1- that, ‖ℓm(R)‖2≤Csm2log(m/ε) \| _m(R)\|_2≤ C sm^2 (m/ ) Proof. Recall, ‖ℓm(R)‖22 \| _m(R)\|_2^2 =∑j∈[m](∑j′∈[m];j′≠jvj≻j′−spj≻j′)2. = _j∈[m] ( _j ∈[m];j ≠ jv_j j -sp_j j )^2. Now, applying a Chernoff bound, we have with probability 1−ε/m1- /m |∑j′∈[m];j′≠jvj≻j′−spj≻j′| | _j ∈[m];j ≠ jv_j j -sp_j j | ≤3slog(2m/ε)∑j′∈[m];j≠j′pj≻j′ ≤ 3s (2m/ ) _j ∈[m];j≠ j p_j j ≤3smlog(2m/ε). ≤ 3sm (2m/ ). Thus, with probability at least 1−ε1- , ∑j∈[m](∑j′∈[m];j′≠jvj≻j′−spj≻j′)2 _j∈[m] ( _j ∈[m];j ≠ jv_j j -sp_j j )^2 ≤∑j∈[m]3smlog(2m/ε) ≤ _j∈[m]3sm (2m/ ) =3sm2log(2m/ε). =3sm^2 (2m/ ). ∎ Appendix D Proof of Definition˜3.1 We first restate the result for reference. * In our proof, we’l call the set of candidates induced by z the “original candidates” and ȷ(zιȷ+1) ^(z_ +1) the “additional clone”. Similarly, we’l call the vote distributions induced by z, z′z respectively as the “original distribution” and the “additional clone distribution”. At a high level, our proof will proceed as follows: 1. We’l observe that the win probability of a producer with an additional clone is equal to the probability that some model by the producer is ranked above all of the original candidates. 2. Next, we’l argue that the event that the additional clone ranks above all original candidates and the event that any of the original candidates by the same producer rank above the original candidates are anticorrelated. This implies the probability (with respect to the additional clone distribution) that a model by the producer with the additional clone is ranked first is no less than the probability the additional clone is ranked first plus the probability one of the original candidates by the producer is ranked first minus the product of these two probabilities. 3. We then translate these two probabilities into events that are measurable with respect to the original distribution, and apply Lemma˜C.1 to establish that the probabilities of these two events may differ from their probabilities in the original distribution by at most O(1/s)O(1/ s). 4. These facts together imply that if Definition˜3.1 is satisfied, the producer’s win probability with a clone is greater than without it, which completes the proof. Without loss of generality, suppose producer i=1i=1’s model j=1j=1 satisfies Definition˜3.1. Let w=z1,1+1w=z_1,1+1. Thus, the clone is indexed 1(w)1^(w). Define ℳ−1=ℳ(z)∖ℳ1(z1)M_-1=M(z) _1(z_1) to be all candidates but those submitted by producer 11. For a model j by producer 11, a leaderboard L, and a random ranking σ, define the event Aj(L)=σ−1(j)<σ−1(ℓ)∀ℓ∈ℳ−1∩L. A_j(L)= \σ^-1(j)<σ^-1( )\;\;∀ _-1∩ L \. That is, Aj(L)A_j(L) is the event that a model j ranks above all those by other producers ℳ−1M_-1 in the leaderboard L. We will overload notation and write, for a set of candidates S, AS(L)=⋃j∈SAj(L). A_S(L)= _j∈ SA_j(L). For S⊆ℳ1(z1)S _1(z_1) and σ∼Σ(sq,z)σ ( sq,z), note that AS=σL(1)∈S. A_S= \ _L(1)∈ S \. That is, if any model by producer 11 ranks above all candidates by other producers in L under actions z, a model by producer 11 must be ranked first in L. Moreover, if model 1(w)∈S1^(w)∈ S, S⊆ℳiS _i, and σ∼Σ(sq,z′)σ ( sq,z ), then AS=σL(1)∈SA_S=\ _L(1)∈ S\: if S contains the new clone and is a subset of producer 11s candidates, one model in S must be ranked first in L for producers’ actions z′z . Now, for the expectation and probabilities taken with respect to σ∼Σ(sq,z′)σ ( sq,z ), observe [u1(σ)] [u_1(σ)] =∑L∈ℒν1(L)Pr(σL(1)∈ℳ1(z1′)) = _L _1(L) ( _L(1) _1(z _1)) (Definition of utility.) Now, notice Pr(Aℳ1(z1′)(L)) (A_M_1(z_1 )(L)) =Pr(Aℳ1(z1)(L)∪A1(w)(L)) = (A_M_1(z_1)(L)∪ A_1^(w)(L)) (ℳ1(z1′)=ℳ1(z1)∪1(w)M_1(z_1 )=M_1(z_1)∪\1^(w)\) =Pr(Aℳ1(z1)(L))+Pr(A1(w)(L))−Pr(Aℳ1(z1′)(L)∩A1(w)(L)) = (A_M_1(z_1)(L))+ (A_1^(w)(L))- (A_M_1(z_1 )(L)∩ A_1^(w)(L)) (Inclusion-exclusion formula.) ≥Pr(Aℳ1(z1)(L))+Pr(A1(w)(L))−Pr(Aℳ1(z1′)(L))Pr(A1(w)(L)) ≥ (A_M_1(z_1)(L))+ (A_1^(w)(L))- (A_M_1(z_1 )(L)) (A_1^(w)(L)) (Lemma D.1) =Pr(Aℳ1(z1)(L))+Pr(A1(1)(L))−Pr(Aℳ1(z1′)(L))Pr(A1(1)(L)). = (A_M_1(z_1)(L))+ (A_1^(1)(L))- (A_M_1(z_1 )(L)) (A_1^(1)(L)). (σL−1(1(1))=σL−1(1(w)) _L^-1(1^(1)) d= _L^-1(1^(w))) Plugging the last expression into the sum above, we have [u1(σ)] [u_1(σ)] =∑L∈ℒν1(L)(Pr(Aℳ1(z1)(L))+Pr(A1(1)(L))−Pr(Aℳ1(z1′)(L))Pr(A1(1)(L))) = _L _1(L) ( (A_M_1(z_1)(L))+ (A_1^(1)(L))- (A_M_1(z_1 )(L)) (A_1^(1)(L)) ) Now, observe that the events Aℳ1(z)(L)A_M_1(z)(L) and A1(1)(L)A_1^(1)(L) are measurable with respect to σ∼Σ(sq,z)σ ( sq,z) (the original distribution). Thus, applying Lemma˜C.1, for all ν>0ν>0, there exists s0,m0s_0,m_0 such that for s≥s0,m≥m0s≥ s_0,m≥ m_0, Prσ∼Σ(sq,z′)(Aℳ1(z)(L)) _σ ( sq,z )(A_M_1(z)(L)) ≥Prσ∼Σ(sq,z)(Aℳ1(z)(L))−ν, and ≥ _σ ( sq,z)(A_M_1(z)(L))-ν, and Prσ∼Σ(sq,z′)(A1(1)(L)) _σ ( sq,z )(A_1^(1)(L)) ≥Prσ∼Σ(sq,z)(A1(1)(L))−ν. ≥ _σ ( sq,z)(A_1^(1)(L))-ν. Combining these with the expression above, we have σ∼Σ(sq,z′)[u1(σ)] _σ ( sq,z )[u_1(σ)] ≥∑L∈ℒν1(L)(Prσ∼Σ(sq,z)(Aℳ1(z)(L))+Prσ∼Σ(sq,z)(A1(1)(L)) ≥ _L _1(L) ( _σ ( sq,z)(A_M_1(z)(L))+ _σ ( sq,z)(A_1^(1)(L)) −Prσ∼Σ(sq,z)(Aℳ1(z)(L))⋅Prσ∼Σ(sq,z)(A1(1)(L)))−ν - _σ ( sq,z)(A_M_1(z)(L))· _σ ( sq,z)(A_1^(1)(L)) )-ν ≥∑L∈ℒν1(L)Prσ∼Σ(sq,z)(Aℳ1(z)(L)) ≥ _L _1(L) _σ ( sq,z)(A_M_1(z)(L)) +∑L∈ℒν1(L)δ(1−Prσ∼Σ(sq,z)(Aℳ1(z)(L)))−ν + _L _1(L)δ (1- _σ ( sq,z)(A_M_1(z)(L)) )-ν (Definition 3.1, first inequality) ≥∑L∈ℒν1(L)Prσ∼Σ(sq,z)(Aℳ1(z)(L))+∑L∈ℒν1(L)δ2−ν ≥ _L _1(L) _σ ( sq,z)(A_M_1(z)(L))+ _L _1(L)δ^2-ν (Definition 3.1, second inequality) ≥∑L∈ℒν1(L)Prσ∼Σ(sq,z)(Aℳ1(z)(L))+εδ2−ν ≥ _L _1(L) _σ ( sq,z)(A_M_1(z)(L))+ δ^2-ν (Definition 3.1, ε condition) Finally, as long as we set ν<εδ2ν< δ^2, we have ∑L∈ℒν1(L)Prσ∼Σ(sq,z)(Aℳ1(z)(L))+εδ2−ν _L _1(L) _σ ( sq,z)(A_M_1(z)(L))+ δ^2-ν ≥∑L∈ℒν1(L)Prσ∼Σ(sq,z)(Aℳ1(z)(L)) ≥ _L _1(L) _σ ( sq,z)(A_M_1(z)(L)) =σ∼Σ(sq,z)[u1(σ)] =E_σ ( sq,z)[u_1(σ)] where the last line is by definition. ∎ Lemma D.1. It holds Pr(AM1(z1)(L)∩A1(w))(L)≤Pr(AM1(z1)(L))Pr(A1(w)(L)). (A_M_1(z_1)(L)∩ A_1^(w))(L)≤ (A_M_1(z_1)(L)) (A_1^(w)(L)). Proof of Lemma˜D.1. From Lemma˜B.1, since matchup counts are allocated evenly across pairs, the rankings induced by Bradley-Terry fit with MLE and Borda counts are equivalent. Thus, if the set of candidates submitted to the mechanism is ℳ(z′)M(z ), we have that Aj(L)=∑ℓ∈ℳ(z′)vj≻ℓ>∑ℓ∈ℳ(z′)vj′≻ℓ∀j′∈ℳ−1∩L, A_j(L)= \ _ (z )v_j > _ (z )v_j \;\;∀ j _-1∩ L \, and similarly for AS(L)A_S(L). Now, let F=v1(w)≻jj∈ℳ1F=\v_1^(w) j\_j _1 be all vote counts between the additional clone and producer 11’s original candidates, ℳ1M_1. Let FC=v∖F^C=v F be all other vote counts, keeping one copy of each independent vote count and excluding all vj≻1(w)j∈ℳ1\v_j 1^(w)\_j _1. (I.e., if vj≻j′∈FCv_j j ∈ F^C then vj′≻j∉FCv_j j ∈ F^C, since vj≻j′+vj′≻j=sv_j j +v_j j=s.) We will argue that Pr(AM1(z1)(L)∩A1(w)(L)|FC)≤Pr(AM1(z1)(L)|F(C))Pr(A1(w)(L)|F(C)). (A_M_1(z_1)(L)∩ A_1^(w)(L)\;|\;F^C)≤ (A_M_1(z_1)(L)\;|\;F^(C)) (A_1^(w)(L)\;|\;F^(C)). (D.1) This implies the result since the conditional probabilities for all F(C)F^(C) imply the unconditional ones (by taking expectations with respect to F(C)F^(C) for the left- and right-hand sides of the equation). To show Equation˜D.1, we will apply the Harris inequality. Note that the measure induced by F (conditional on FCF^C) is a product measure, since vote counts between different pairs of candidates are independent. Thus, it is sufficient to show that AM1(z1)A_M_1(z_1) is a decreasing event and A1(w)A_1^(w) is an increasing event. Now, writing each event in terms of the Borda count, we can rewrite A1(w)(L)A_1^(w)(L) as ∑ℓ∈ℳi(zi′),ℓ≠1(w)v1(w)≻ℓ>∑ℓ∈ℳ(z′),ℓ≠jvj≻ℓ−∑ℓ∈ℳ−1v1(w)≻ℓ,∀j∈ℳ−1∩L. _ _i(z_i ), ≠ 1^(w)v_1^(w) > _ (z ), ≠ jv_j - _ _-1v_1^(w) ,\;\;∀ j _-1∩ L. Note that the left-hand side sum is over elements in F and the right-hand side sums are over elements determined by FCF^C. Now, by inspecting the left-hand side, note that the event A1(w)|FCA_1^(w)\;|\;F^C is increasing: if the inequality is satisfied and we increase an entry of v1(w)≻ℓv_1^(w) , the inequality is still satisfied. Similarly, we can write AM1(z1)(L)A_M_1(z_1)(L) as ∃j∈ℳ1(z)∩Ls.t.vj≻1(w)>∑ℓ∈ℳ(z′),ℓ≠j′vj′≻ℓ−∑ℓ∈ℳ(z),ℓ≠jvj≻ℓ,∀j′∈ℳ−1∩L. ∃ j _1(z)∩ L\;s.t.\;v_j 1^(w)> _ (z ), ≠ j v_j - _ (z), ≠ jv_j ,\;\;∀ j _-1∩ L. Now, since vj≻1(w)+v1(w)≻j=sv_j 1^(w)+v_1^(w) j=s, we can equivalently write AM1(z1)(L)A_M_1(z_1)(L) as ∃j∈ℳ1(z)∩Ls.t.v1(w)≻j<s−∑ℓ∈ℳ(z′),ℓ≠j′vj′≻ℓ+∑ℓ∈ℳ(z),ℓ≠jvj≻ℓ,∀j′∈ℳ−1∩L. ∃ j _1(z)∩ L\;s.t.\;v_1^(w) j<s- _ (z ), ≠ j v_j + _ (z), ≠ jv_j ,\;\;∀ j _-1∩ L. Again, the LHS contains terms in F and the RHS contains terms determined by FCF^C. Thus, the event AM1(z1)A_M_1(z_1) is decreasing: If the inequality is satisfied for some j∈M1(z)∩Lj∈ M_1(z)∩ L and we decrease v1(w)≻j′v_1^(w) j for some j′∈M1(z)∩Lj ∈ M_1(z)∩ L, the inequality is still satisfied. Thus, by the Harris inequality, Equation˜D.1 is satisfied and the result holds. ∎ Appendix E Proof of Section˜4.1 We first restate the result: * In our proof, we’l apply the following two lemmas. The first says that all producers approximately prefer to submit at least 1 copy of each model. Lemma E.1. For all ε>0 >0, there exists s0,m0s_0,m_0 such that for all s≥s0s≥ s_0 and m≥m0m≥ m_0, the following holds. Consider an action vector z where zi,j=0z_i,j=0 for some producer i and model j. Let z′=(1,z−(i,j))z =(1,z_-(i,j)) be the action where i submits one copy of j. Let π′π be the same as π on all pairs ranked by π and let the newly submitted model be producer-ranked last. Then σ∼Σ(yrwr,z′,π′)[ui(σ)]≥σ∼Σ(yrwr,z,π)[ui(σ)]−ε. _σ ( yrwr,z ,π )[u_i(σ)] _σ ( yrwr,z,π)[u_i(σ)]- . The second says that, for any producer i action ziz_i where there is some zij>1z_ij>1, it holds the producer approximately prefers to submit one copy of j. Lemma E.2. For all ε>0 >0, there exists s0,m0s_0,m_0 such that for all s≥s0s≥ s_0 and m≥m0m≥ m_0, the following holds. Consider an action vector z where zi,j>1z_i,j>1 for some producer i and model j. Let z′=(1,z−(i,j))z =(1,z_-(i,j)) be the action where i submits one copy of j. Let π′π be equal to the π when dropping j(2),…,j(zij)j^(2),…,j^(z_ij), keeping the ordering of remaining candidates the same. Then σ∼Σ(yrwr,z′,π′)[ui(σ)]≥σ∼Σ(yrwr,z,π)[ui(σ)]−ε. _σ ( yrwr,z^ ,π )[u_i(σ)] _σ ( yrwr,z,π)[u_i(σ)]- . We will show that together, these lemmas imply the theorem: Intuitively, for any producer i action ziz_i, we can make a series of ε -approximately utility-improving changes to the action such that we end up with action 1. And since each producer only has at most a constant number of distinct candidates by assumption, there are only a constant number of such changes. Setting ε appropriately then yields the theorem (i.e., if W is the maximum number of distinct models, setting ε in the lemma equal to ε/W /W for ε in the theorem). Formally, let z(0)=z^(0)=z and π(0)=π^(0)=π. For u∈kiu∈ k_i, let z(u)=(1,z−(i,u)(u−1))z^(u)=(1,z^(u-1)_-(i,u)) be the action that sets the first u entries of z to 1. Note that zi(mi)=z^(m_i)_i=1 so z(mi)=z′z^(m_i)=z . Similarly, define πi(u) _i^(u) to be the ranking achieved by altering πi(u−1) _i^(u-1) by 1. appending entry u to the end of the ranking if zi,u=0z_i,u=0, 2. dropping j(2),…,j(zi,u)j^(2),…,j^(z_i,u) from the ranking if zi,u>1z_i,u>1, and 3. keeping the ranking as is if zi,u=1z_i,u=1. Note that πi(mi)=π^(m_i)_i=1, so π(mi)=π′π^(m_i)=π . By telescoping, note that σ∼Σ(yrwr,z′,π′)[ui(σ)]−σ∼Σ(yrwr,z,π)[ui(σ)]=∑u=1miσ∼Σ(yrwr,z(u),π(u))[ui(σ)]−σ∼Σ(yrwr,z(u−1),π(u−1))[ui(σ)]. aligned &E_σ ( yrwr,z ,π )[u_i(σ)]-E_σ ( yrwr,z,π)[u_i(σ)]\\ =& _u=1^m_iE_σ ( yrwr,z^(u),π^(u))[u_i(σ)]-E_σ ( yrwr,z^(u-1),π^(u-1))[u_i(σ)]. aligned (E.1) At each step we apply either Lemma˜E.1 if zi,j=0z_i,j=0 or Lemma˜E.2 if zi,j>1z_i,j>1 and note that z(u−1)=z(u)z^(u-1)=z^(u) and π(u−1)=π(u)π^(u-1)=π^(u) otherwise, so the expectations are equal in that case. Thus, each term in the sum on the second line is no less than −ε/W- /W and so the sum is no less than −ε- . Rearranging the left-hand side of Equation˜E.1 yields the inequality in the theorem. Without loss of generality, in the next proofs of the next two lemmas, let the focal producer in each lemma be indexed 11 so that we may use i for generic producers. Proof of Lemma˜E.1. Let σ∼Σ(yrwr,z,π)σ ( yrwr,z,π) and σ′∼Σ(yrwr,z′,π′)σ ( yrwr,z ,π ). Then σ′[u1(σ′)]−σ[u1(σ)] _σ [u_1(σ )]-E_σ[u_1(σ)] =∑L∈ℒν1(L)(Prσ′[σ′(1)∈ℳ1(z1′)]−Prσ[σ(1)∈ℳ1(z1)]) = _L _1(L) ( _σ [σ (1) _1(z_1 )]- _σ[σ(1) _1(z_1)] ) (Definition of u1u_1) =∑L∈ℒν1(L)(∑ℓ∈ℳ1(z1′)Prσ′[σ′(1)=ℓ]−∑ℓ∈ℳ1(z1)Prσ[σ(1)=ℓ]) = _L _1(L) ( _ _1(z_1 ) _σ [σ (1)= ]- _ _1(z_1) _σ[σ(1)= ] ) (Probabilities that a model ranks first are disjoint.) =∑L∈ℒν1(L)Prσ′[σ′(1)=j]+∑L∈ℒν1(L)∑ℓ∈ℳ1(z1)Prσ′[σ′(1)=ℓ]−Prσ[σ(1)=ℓ] = _L _1(L) _σ [σ (1)=j]+ _L _1(L) _ _1(z_1) _σ [σ (1)= ]- _σ[σ(1)= ] (ℳ1(z1′)=ℳ1(z1)∪jM_1(z_1 )=M_1(z_1)∪\j\.) Let m¯ m be an upper bound on maxi|ℳi(zi)| _i |M_i(z_i) | (since we have assumed that no producer has more than a constant number of models). Finally, the first term is trivially nonnegative and each term in the second sum is no less than −ε/m¯- / m by Lemma˜C.1 (choosing ε in Lemma˜C.1 to be ε/m¯ / m). Thus, each inner sum is no less than −ε- and by assumption on ν summing to no more than 11, the outer sum must be no less than −ε- . ∎ Proof of Lemma˜E.2. Let σ∼Σ(yrwr,z,π)σ ( yrwr,z,π) and σ′∼Σ(yrwr,z′,π′)σ ( yrwr,z ,π ). Then, σ′[u1(σ′)]−σ[u1(σ)] _σ [u_1(σ )]-E_σ[u_1(σ)] =∑L∈ℒν1(L)(∑u=2zijPrσ′(σL′(1)=j(u))+∑ℓ∈ℳ1(z1)Prσ′(σL′(1)=ℓ)−Prσ(σL(1)=ℓ)), = _L _1(L) ( _u=2^z_ij _σ ( _L (1)=j^(u))+ _ _1(z_1) _σ ( _L (1)= )- _σ( _L(1)= ) ), similar to Lemma˜E.2. Now, notice that ∑u=2zijPrσ′(σL′(1)=j(u))=0 _u=2^z_ij _σ ( _L (1)=j^(u))=0 since WLOG we assumed that π(j(1))<π(j(u))π(j^(1))<π(j^(u)) for u>1u>1, so the mechanism ensures σ(j(1))<σ(j(u))σ(j^(1))<σ(j^(u)) for u>1u>1. Moreover, each term in the second RHS inner sum is no less than −ε/W- /W, since we removed at most m¯−1 m-1 candidates and so can apply Lemma˜C.1 (using parameter ε/(m¯W) /( mW)) m¯ m times for each term. Finally, there are W terms in the second RHS sum by assumption, so the whole sum is no less than −ε- . ∎ Appendix F Additional proofs Proof of Proposition˜4.1. To show the key inequality above, we will specifically prove the following chain of inequalities: ‖Rˇs−R‖∞=‖Tπ(R^s)−Tπ(R)‖∞≤‖R^s−R‖∞\| R_s-R\|_∞=\|T_π( R_s)-T_π(R)\|_∞≤\| R_s-R\|_∞ where the first step is by definition and the fact that when π is truthful, Tπ(R)=RT_π(R)=R (the rewards are unchanged by the correction). The second inequality is by the general lipschitzness of the correction, which we will prove now: Fix R′,R′∈ℝmR ,R ^m, fix a producer i and j∈Mij∈ M_i. Let Si,j:=πi(k):k≤πi−1(j)S_i,j:=\ _i(k):k≤ _i^-1(j)\ be the set of all candidates i ranked in πi _i as better than j. Then |(Tπ(R′))j−(Tπ(R′))j|=|minj′∈Si,jRj′−minj′∈Si,jRj′|≤maxj′∈Si,j|Rj′−Rj′|≤‖R′−R′‖∞.|(T_π(R ))_j-(T_π(R ))_j|= | _j ∈ S_i,jR _j \;-\; _j ∈ S_i,jR _j |≤ _j ∈ S_i,j|R _j -R _j |≤\|R -R \|_∞. The first inequality is by the fact that if the mins are very far apart, then either those mins correspond to the same j′j (and it holds with equality), or they correspond to different j′,j′j ,j and if so, the distance between the mins is a lower bound on the distance between the respective rewards for at least one of these candidates. The second inequality is just by the definition of the ℓ∞ _∞ norm (lefthand side is just max over a subset of candidates). Then, it follows that ‖Tπ(R′)−Tπ(R′)‖∞=maxj|(Tπ(R′))j−(Tπ(R′))j|≤‖R′−R′‖∞.∎\|T_π(R )-T_π(R )\|_∞= _j|(T_π(R ))_j-(T_π(R ))_j|≤\|R -R \|_∞. Proof Corollary˜4.2. By Proposition˜4.1, ‖Rˇs−R‖∞≤‖R^s−R‖∞.\| R_s-R\|_∞≤\| R_s-R\|_∞. This implies the claim because if for all random realizations of R^s R_s this is true, then for any M and any s, Pr(‖Rˇs−R‖∞>M/s)≤Pr(‖R^s−R‖∞>M/s), (\| R_s-R\|_∞>M/ s)≤ (\| R_s-R\|_∞>M/ s), And s s consistency is shown. Correctness under γ-separated true rewards is a direct implication of the first part, since the probability any pair of models is misranked goes to zero as s→∞s→∞. ∎ Proof of Proposition˜4.4. We begin by assuming that all candidates have distinct qualities. Under this assumption, there exists some γ>0γ>0 such that |Rj−Rj′|>γ|R_j-R_j |>γ for all j,j′j,j . Because MLE estimates for Bradley-Terry concentrate around their true values, R^j−R^j′ R_j- R_j concentrates around Rj−Rj′R_j-R_j . Therefore, if Rj>Rj′R_j>R_j , then Pr[R^j−R^j′<0]→s↑∞0. [ R_j- R_j <0] [s ∞]0. Consider any j,j′∈ij,j _i such that Rj>Rj′R_j>R_j . Suppose they are adjacently ranked in πi _i, but in the incorrect order (i.e., ⋯≻j′≻j≻… j j …). Swapping their ranks (call this πi′ _i ) can only reduce producer i’s utility in the case where R^j′>R^j R_j > R_j. As shown above, the probability that this occurs goes to 0 as s↑∞s ∞. Therefore, for any ε , there exists sufficiently large s such that Pr[R^j−R^j′<0]<ε [ R_j- R_j <0]< . Swapping from πi _i to πi′ _i can only weakly increase Rˇj R_j and weakly decrease Rˇj′ R_j . We ignore the effect of increasing Rˇj R_j, since it can only increase utility. Decreasing Rˇj′ R_j can reduce utility because j′j may lose a leaderboard it would have otherwise won, and occurs if and only if R^j′>R^j R_j > R_j. Formally, σ∼Σ(yrwr(ℳ(z),πi′))[ui(σ)]−σ∼Σ(yrwr(ℳ(z),πi′))[ui(σ)] _σ ( yrwr(M(z), _i ))[u_i(σ)]-E_σ ( yrwr(M(z), _i ))[u_i(σ)] ≤σ∼Σ(yrwr(ℳ(z),πi′))[ui(σ)⋅(R^j′>R^j)]−σ∼Σ(yrwr(ℳ(z),πi′))[ui(σ)⋅(R^j′>R^j)] _σ ( yrwr(M(z), _i ))[u_i(σ)·1( R_j > R_j)]-E_σ ( yrwr(M(z), _i ))[u_i(σ)·1( R_j > R_j)] ≤Pr[R^j′>R^j] ≤ [ R_j > R_j] ≤ε ≤ for sufficiently large s. Applying this argument inductively (we need at most (ki2) k_i2 swaps) shows that the true ranking πi∗ _i^* is an ε -approximate dominant strategy for sufficiently large s. Finally, we can lift the assumption that all qualities are distinct by allowing that either ranking of two candidates with identical qualities is considered truthful. Our argument applies to all pairs of candidates with distinct scores, which yields the result. ∎ Proof of Section˜4.4. Let A be the event that the confidence intervals cover R. Now, on A, the analysis in the proof of Section˜4.1 holds: Since the confidence intervals cover R, clones must have overlapping confidence intervals. Thus, the isotonic score correction will be applied to all clones and the arguments for the approximate utility improvement induced by removing clones apply as is (conditioning on A). On C A^C, since utilities are bounded in [0,1][0,1], the change in utility can be at most one. And since the confidence intervals are simultaneously valid, C A^C holds with probability at most ε . Therefore: σ∼Σ(ua-yrwr(ℳ(z′),π,α)[ui(σ)])−σ∼Σ(ua-yrwr(ℳ(z),π,α))[ui(σ)] _σ ( ua-yrwr(M(z ),π,α)[u_i(σ)])-E_σ ( ua-yrwr(M(z),π,α))[u_i(σ)] =P()(σ∼Σ(ua-yrwr(ℳ(z′),π,α)[ui(σ)|])−σ∼Σ(ua-yrwr(ℳ(z),π,α))[ui(σ)|]) =P( A)(E_σ ( ua-yrwr(M(z ),π,α)[u_i(σ)\;|\; A])-E_σ ( ua-yrwr(M(z),π,α))[u_i(σ)\;|\; A]) +(1−P())(σ∼Σ(ua-yrwr(ℳ(z′),π,α)[ui(σ)|C])−σ∼Σ(ua-yrwr(ℳ(z),π,α))[ui(σ)|C]) +(1-P( A))(E_σ ( ua-yrwr(M(z ),π,α)[u_i(σ)\;|\; A^C])-E_σ ( ua-yrwr(M(z),π,α))[u_i(σ)\;|\; A^C]) ≥1⋅(−ε)+α⋅(−1) ≥ 1·(- )+α·(-1) ∎ Proof of Proposition˜4.5. For s s-consistency, note that since the confidence intervals are s s consistent and include R R, for all j∈ℳj RˇjUA−R^j=OP(s−1/2). R^UA_j- R_j=O_P(s^-1/2). Moreover, by the MLE theorem, R^j−Rj=OP(s−1/2) R_j-R_j=O_P(s^-1/2). Thus, RˇjUA−Rj=(RˇjUA−R^j)−(Rj−R^j)=OP(s−1/2). R^UA_j-R_j=( R^UA_j- R_j)-(R_j- R_j)=O_P(s^-1/2). Correctness is a direct implication of the first part, since the probability any pair of models is misranked goes to zero as s→∞s→∞. ∎