Paper deep dive
Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging
Hamed Khosravi, Xiaoming Huo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 7:33:05 AM
Summary
The paper introduces 'Certified Corruption Budgets' (corruption tolerance) to provide anytime-valid statistical guarantees for AI model leaderboards under adaptive attacks. Unlike existing methods that assume genuine data or fixed corruption bounds, this method computes a tolerance budget that certifies a model's lead unless more than a specified number of records were corrupted. Using a betting-based approach (Deflated Capped Betting), the authors demonstrate that their method remains valid against adaptive attackers who observe published certificates, even with burst corruption. Experiments on Chatbot Arena data show the method withstands approximately 2,000 forged votes for well-separated models, whereas standard confidence intervals fail with only a few hundred rigged votes.
Entities (7)
Relation Signals (5)
Chatbot Arena → usedinevaluationof → Certified Corruption Budget
confidence 98% · In replays on 1.8 million Chatbot Arena votes... ours stays valid.
Deflated Capped Betting → implements → Certified Corruption Budget
confidence 95% · Method: Deflated Capped Betting... The tolerance is built from the wealth of a gambler... We report B^t...
Certified Corruption Budget → providesguaranteefor → Leaderboard Claims
confidence 95% · We introduce the certified corruption budget... With probability at least 1−α... the claim is correct or more than B^t records were corrupted.
Adaptive Rigging → threatens → Leaderboard Integrity
confidence 90% · Public leaderboards... are read continuously, and attackers can see every published standing. Vote rigging... can each move a ranking.
GPT-4o → comparedwith → Claude 3.5 Sonnet
confidence 85% · GPT-4o (2024-05-13) is not certified above Claude-3.5-Sonnet when all pairs are claimed at once.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Public leaderboards for AI models are read continuously, and attackers can see every published standing. Vote rigging, selective disclosure of private variants, and benchmark contamination can each move a ranking. Existing guarantees assume genuine records or bound the corruption per step, which an attacker who corrupts in bursts evades. We introduce the certified corruption budget, a tolerance $\widehat{B}_t$ computed after $t$ records and published with each pairwise claim. With probability at least $1-\alpha$, simultaneously at all times, the claim is correct or more than $\widehat{B}_t$ records were corrupted. It holds against attackers who watch every certificate, with no bound on their budget. Forged records and records altered once seen require different certificates: the certificate for forgeries fails, with probability approaching one, against an attacker who flips votes it has seen, while one that charges roughly twice as much per record remains valid, with constant bets even against attackers who see the future, and no smaller charge is valid at every level. The certified budget grows nearly as fast as any valid method allows: with a win fraction $\frac{1}{2}+\delta$, each new record adds close to $2\delta$ to the number of forged records the claim can withstand ($\delta$ flipped). Publishing the best of $V$ private variants costs only an amount growing like $\log V$. In replays on 1.8 million Chatbot Arena votes, a few hundred rigged votes make standard confidence intervals certify false orderings, while ours stays valid. On real votes, our certificate shows that clearly separated models withstand about 2,000 forged votes.
Tags
Links
- Source: https://arxiv.org/abs/2610.10597v1
- Canonical: https://arxiv.org/abs/2610.10597v1
Trouble viewing inline? Open PDF directly →
Full Text
174,377 characters extracted from source content.
Expand or collapse full text
Certified Corruption Budgets: Anytime-Valid Leaderboard Claims under Adaptive Rigging Hamed Khosravi Xiaoming Huo hkhosravi7@gatech.edu huo@gatech.edu H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA, USA Abstract Public leaderboards for AI models are read continuously, and attackers can see every published standing. Vote rigging, selective disclosure of private variants, and benchmark contamination can each move a ranking. Existing guarantees assume genuine records or bound the corruption per step, which an attacker who corrupts in bursts evades. We introduce the certified corruption budget, a tolerance B^t B_t computed after t records and published with each pairwise claim. With probability at least 1−α1-α, simultaneously at all times, the claim is correct or more than B^t B_t records were corrupted. It holds against attackers who watch every certificate, with no bound on their budget. Forged records and records altered once seen require different certificates: the certificate for forgeries fails, with probability approaching one, against an attacker who flips votes it has seen, while one that charges roughly twice as much per record remains valid, with constant bets even against attackers who see the future, and no smaller charge is valid at every level. The certified budget grows nearly as fast as any valid method allows: with a win fraction 12+δ 12+δ, each new record adds close to 2δ2δ to the number of forged records the claim can withstand (δ flipped). Publishing the best of V private variants costs only an amount growing like logV V. In replays on 1.81.8 million Chatbot Arena votes, a few hundred rigged votes make standard confidence intervals certify false orderings, while ours stays valid. On real votes, our certificate shows that clearly separated models withstand about 2,0002,000 forged votes. 1 Introduction Public leaderboards decide which artificial intelligence (AI) models are trusted, bought and built on. They are read while being computed. A leaderboard ranks models from performance records. On Chatbot Arena, each record is a battle: a user prompts two anonymous models and votes for the better answer or a tie. Its public file has nearly two million votes (Zheng et al.,, 2023; Chiang et al.,, 2024; LMSYS Org / LMArena,, 2024). On static benchmarks such as HELM, the Holistic Evaluation of Language Models, a record compares two models on one test item (Liang et al.,, 2023). Leaderboards are attacked, and the attackers see what is published. In simulations of Chatbot Arena, 381381 rigged votes (votes cast to favor a chosen model) moved a low-ranked model up four places, and anonymous models can be identified to aim such votes (Huang et al.,, 2025). Some developers privately test many variants of a model and publish only the best. One developer tested 27 (Singh et al.,, 2025). On static benchmarks, training on the test items (contamination) inflates a model’s score (Sainz et al.,, 2023; Xu et al.,, 2024). In each case the attacker can wait, see which comparisons are close and strike there. The usual statistical guarantees do not address such an attacker. Confidence intervals for ratings and ranks assume genuine records analyzed once, at a sample size fixed in advance (Chiang et al.,, 2024; Mogstad et al.,, 2024). Methods valid under analysis after every record also assume genuine records (Ramdas et al.,, 2023), and their robust versions allow a fixed fraction of corruption at every step (Saha and Ramdas,, 2026; Wang and Ramdas,, 2023), which a burst (many corrupted records close together) violates. A leaderboard’s reader needs a different statement: not “model i leads model j” but “model i leads model j unless more than a stated number of records were corrupted”. We attach to each leaderboard claim a certified corruption budget, which we call its corruption tolerance: a number of corrupted records that the claim is guaranteed to withstand. With each claim about two models i and j, the platform publishes this number, B^t B_t, computed from the first t records on the pair. Except with a small probability α chosen by the platform (say 5%5\%), and no matter when the claim is read, either the claim holds or more than B^t B_t of those records were forged or altered. We certify two claims. The lead claim, that one model leads another on average so far, is the usual interpretation of a leaderboard standing. The edge claim is that one model has had an advantage over another at some point. The two coincide when win chances are stable. The guarantee holds even if the attacker watches every published tolerance, and assumes no limit on the number of corrupted records. We compute the tolerance by betting on the claim. It is the evidence won beyond what a test on genuine records needs, divided by a cost per record. This cost is the most evidence one corrupted record can fake. The construction has four properties. First, the tolerance counts corrupted records, the unit in which attacks are measured, so it compares directly with known attacks. Second, it separates forging a vote from changing a seen one, as a colluding annotator or a contaminated test item does. A certificate for forgeries fails against altered votes. Roughly doubling the cost per record restores the guarantee. With constant bets it holds even against an attacker knowing the future. Third, for independent and identically distributed (i.i.d.) records without ties, no valid method certifies substantially more corrupted records from the same data. Fourth, it prices publishing the best of many private variants and updates cheaply per vote. Contributions. (i) We introduce corruption tolerance, the number of corrupted records a claim withstands, and three attacker classes. (i) We show that no existing method covers this setting and give a procedure that needs no bound on the number of corrupted records. (i) We prove these guarantees for both claims and extend them to whole leaderboards. (iv) We confirm the main results by simulation and on real data. On 1.81.8 million Chatbot Arena votes, well-separated models are certified against about 2,0002,000 forged votes, but GPT-4o (2024-05-13) is not certified above Claude-3.5-Sonnet when all pairs are claimed at once. On five frontier benchmarks, no top-two ordering is certified when all pairs are claimed at once. 2 Problem Formulation 2.1 Records, claims and the null hypothesis The data for a claim about models i and j are a stream of records. We extend the formulation to whole leaderboards in Section 5.4. A record is one comparison of i with j together with its outcome: 11 if i wins, 00 if j wins and 12 12 for a tie. Records are indexed by t=1,2,…t=1,2,… in the order the platform receives them, and it may choose which pair to compare next using everything it has seen. After each record it publishes a certificate for the claim (Section 2.3). We model the stream as random variables on one probability space. The randomness comes from the platform’s normal process (users, prompts, model answers and votes) and from an attacker, who may randomize. The attacker corrupts records in two ways. It may insert a forged record that the normal process never produced, such as a vote from a fake account. It may also replace a record by overwriting the value that the normal process produced, as a colluding annotator or a contaminated test item does. A replacement of a loss by a win is a flip. A record is corrupted if it is inserted or replaced, and uncorrupted otherwise. Four random variables describe the ttth record, and the platform observes only the first. The observed outcome Wt∈[0,1]W_t∈[0,1] is the outcome that the platform receives. The indicator ItI_t equals 11 if the record is inserted, and CtC_t equals 11 if it is corrupted. The uncorrupted value Xt∈[0,1]X_t∈[0,1] is the value that the normal process produced. It exists when It=0I_t=0, and we set Xt=12X_t= 12 when It=1I_t=1 (formal model in Section B.1). Thus Wt=XtW_t=X_t for an uncorrupted record, while the attacker chooses WtW_t for a corrupted record, and a replaced record hides its uncorrupted value. The number of corrupted records among the first t is the random variable Nt=∑s=1tCsN_t= _s=1^tC_s. The normal process and the attacker’s strategy together determine the joint distribution of these variables. We write ℙP for probability and E for expectation under this distribution. The time index t and the error level α are fixed numbers, not random variables. The edge claim, that i has ever had an edge over j, is false when, at every record with an uncorrupted value, i is expected to do no better than j given the past. Let the σ-field t−1G_t-1 collect everything that happened before the ttth record’s value is produced. It includes the variables of all earlier records on any pair, all published certificates, the attacker’s private randomization and ItI_t. The null hypothesis is H0ij:for every t≥1, H_0^ij: every t≥ 1, (1) [Xt∣t−1]≤12 on the event It=0. [X_t _t-1]≤ 12\ on the event \I_t=0\. The conditional expectation is random, and the inequality must hold with probability one at each non-inserted record. The hypothesis H0ijH_0^ij is a property of the joint distribution, not a random event. It is one statement about the whole stream and has no index t. With ties coded as 12 12, it says that at every such record, given the past, i is no more likely to win than to lose. The null hypothesis assumes neither independence nor a constant win probability, so the models, users and prompts may change over time. The edge claim is correct when H0ijH_0^ij fails, that is, when the event It=0,[Xt∣t−1]>12\I_t=0,\ E[X_t _t-1]> 12\ has positive probability for at least one t. A correct claim thus says that i was favored over j at one record or more (discussed in Section D.1). If [Xt∣t−1]E[X_t _t-1] equals the same constant μ at every non-inserted record (a stable mean outcome), the claim is correct exactly when μ>12μ> 12. This holds for i.i.d. uncorrupted values and under the Bradley–Terry model (Bradley and Terry,, 1952; Chiang et al.,, 2024) that Arena fits (win probabilities are logistic in score differences). This stable case is the usual reading of a leaderboard claim. The lead claim concerns the records so far. Write μs=[Xs∣s−1] _s=E[X_s _s-1] for the conditional mean of an uncorrupted value and let Δt=∑s≤t(1−Is)(μs−12) _t= _s≤ t(1-I_s) ( _s- 12 ) (2) be the uncorrupted margin accumulated over the first t records. The lead claim “i leads j” is correct at time t when Δt>0 _t>0, that is, when the normal process favored i on average over these records. It needs no stable mean, is what a standing usually asserts, and coincides with the edge claim in the stable case. We certify both. 2.2 Three classes of attackers The attacker classes differ in what the attacker knows when it acts. Every attacker is adaptive: it observes every published certificate and may choose its next action from what it has seen. No limit is placed on how many records it corrupts. P Predictable attackers decide whether to corrupt the ttth record, and what its value is, before its uncorrupted value is produced. Every insertion is of this kind, and so is a replacement that discards the uncorrupted value unseen. V Value-dependent attackers may in addition see the uncorrupted value XtX_t and then decide whether to replace it, before the next record arrives. O Offline attackers see the entire sequence of uncorrupted values, including future ones, before acting. Since their actions then depend on the future, the null for O is stated for the uncorrupted values alone. These form a sequence, indexed by u=1,2,…u=1,2,… in order of arrival, whose law does not depend on the attacker and satisfies [Xu∣X1,…,Xu−1]≤12E[X_u X_1,…,X_u-1]≤ 12. The classes are nested in what the attacker knows, and Section 5 gives the price of guarding against each. 2.3 Corruption tolerance Definition 2.1 (Corruption tolerance). Let (B^t)t≥1( B_t)_t≥ 1 take values in −1,0,1,2,…\-1,0,1,2,…\, with each B^t B_t a function of the first t observed outcomes W1,…,WtW_1,…,W_t. For α∈(0,1)α∈(0,1), the sequence is an α-valid corruption tolerance for the edge claim against an attacker class if, for every normal process and attacker in the class under which H0ijH_0^ij holds, ℙ(∃t≥1:B^t≥Nt)≤α.P (∃\,t≥ 1:\ B_t≥ N_t )≤α. (3) It is α-valid for the lead claim against the class if, for every normal process and attacker in it, ℙ(∃t≥1:Δt≤0andB^t≥Nt)≤α.P (∃\,t≥ 1:\ _t≤ 0\ and\ B_t≥ N_t )≤α. (4) For O the condition that H0ijH_0^ij holds is replaced by the condition that the uncorrupted values satisfy [Xu∣X1,…,Xu−1]≤12E[X_u X_1,…,X_u-1]≤ 12, the null for offline attackers of Section 2.2. Under H0ijH_0^ij the uncorrupted margin Δt _t is at most 00 at every t. Hence Eq. (4) implies Eq. (3), and Eq. (4) makes no hypothesis on the law. Definition 2.1 controls the type I error of a family of sequential tests. At time t the platform publishes the certificate “i has ever had an edge over j, unless more than B^t B_t of the first t records were corrupted”. The number B^t B_t is the claim’s corruption tolerance at that time. Nothing is claimed when B^t=−1 B_t=-1. For each integer b≥0b≥ 0 the event B^t≥b B_t≥ b rejects the hypothesis “H0ijH_0^ij holds and Nt≤bN_t≤ b”. When H0ijH_0^ij holds, a true hypothesis of this family is rejected exactly when B^t≥Nt B_t≥ N_t. So Eq. (3) bounds by α the probability of a false statement. A false statement is a type I error at any time and for any b. When H0ijH_0^ij fails, no type I error is possible (details in Section D.1). The platform does not observe NtN_t. It therefore publishes the conditional statement and not a rejection of H0ijH_0^ij. A tolerance, or the method that computes it, is valid against an attacker class if it is α-valid for the edge claim against that class. Validity for the lead claim is always named. The aim is the largest valid tolerance. The claim is certified against b corrupted records at time t if B^t≥b B_t≥ b, and not certified if B^t=−1 B_t=-1. The attacker’s budget is the number of records it may corrupt. The claim’s tolerance is B^t B_t. Raising B^t B_t only makes Eq. (3) harder to satisfy, and the trivial tolerance −1-1 is valid and claims nothing. For i.i.d. records without ties, our tolerance grows at nearly the fastest rate that any valid tolerance can reach. Section 5.3 proves this. The guarantee is anytime-valid. It holds at all times simultaneously, so anyone may act on the statement at a time of their choosing, even one chosen by looking at the data. With b=0b=0 it is an ordinary anytime-valid test of H0ijH_0^ij on uncorrupted records. 3 Related Work Anytime-valid methods test ranking claims under genuine data, and robust methods under a prespecified corruption level. Neither infers from the data how much adaptive corruption a claim withstands, which corruption tolerance supplies. Anytime-valid inference assumes genuine data. A nonnegative process whose expectation stays at most one under the null, at every stopping time, is an e-process. By Ville’s inequality it ever exceeds 1/α1/α with probability at most α (Ville,, 1939; Ramdas et al.,, 2023). Testing by betting builds e-processes as the wealth of a gambler who starts with one unit and bets against the null at fair odds (Shafer,, 2021; Vovk and Wang,, 2021; Grünwald et al.,, 2024; Ramdas et al.,, 2023). Confidence sequences (Darling and Robbins,, 1967; Howard et al.,, 2021, intervals valid uniformly over time;) can be built this way for bounded means (Waudby-Smith and Ramdas,, 2024), and so can two-sample tests (Shekhar and Ramdas,, 2024). They suit leaderboards that are read continuously (Khosravi and Huo,, 2026; Gao et al.,, 2026; Gu et al.,, 2026), but one forged record can multiply the wealth by a constant, and a burst can push it past any threshold. Robust sequential tests bound corruption per step. Huber, (1965) made the sequential probability ratio test (Wald,, 1945) robust by clipping the likelihood ratio for data with independent outliers. Saha and Ramdas, (2026) give anytime-valid tests in which, at every step, an adaptive adversary picks any conditional law within total-variation distance ε of the null, with ε fixed in advance. Wang and Ramdas, (2023) build confidence sequences under the same bound, stated for i.i.d. data for simplicity. They regard replacement by an adversary that has seen all observations as inherently fixed-time. Without a known contamination bound, nontrivial anytime-valid confidence intervals do not exist (Agrawal et al.,, 2024). Related work covers online change points (Tang et al.,, 2026) and, at a fixed sample size, unknown outlier fractions (Luo and Gao,, 2026). No such sequential test counts corrupted records, and none can describe a burst. Proposition 3.1 (Per-step bounds cannot describe bursts). Suppose at every step the conditional law of a binary outcome may be any law within total-variation distance ε of some Bernoulli law Bern(p)Bern(p) (outcome 11 with probability p) with p≤12p≤ 12. A forged win (the point mass at 11) is allowed only if ε≥12 ≥ 12. Then every Bernoulli law is allowed under the null, so every valid test has power at most α against every alternative. Leaderboard inference assumes genuine votes, and attack studies and certified defenses give no sequential certificate (Section D.3). 4 Method: Deflated Capped Betting 4.1 Capped betting wealth The tolerance is built from the wealth of a gambler who bets on i at every record, with a cap on the bet. Fix a cap λ¯∈(0,2) λ∈(0,2) and, for each record, a bet λt∈[0,λ¯] _t∈[0, λ] chosen from the first t−1t-1 observed records. A constant bet is the same at every record, λt=λ¯ _t= λ for all t, and we then write λ for λ¯ λ. The wealth after t records is Kt=∏s=1t(1+λs(Ws−12)),K0=1.K_t= _s=1^t (1+ _s(W_s- 12) ), K_0=1. (5) Each factor lies between m−=1−λ¯/2m_-=1- λ/2, the smallest value one record can contribute, and M+=1+λ¯/2M_+=1+ λ/2, the largest. A tie gives the factor 11. If all records are uncorrupted and H0ijH_0^ij holds, KtK_t is a nonnegative supermartingale, so ℙ(∃t:Kt≥1/α)≤αP(∃\,t:\ K_t≥ 1/α)≤α by Ville’s inequality. If i wins more often than it loses and the bet is small enough, KtK_t grows exponentially. The cap is what makes robustness possible. Whatever an attacker does to one record, it changes that record’s factor by a bounded ratio. A forged record contributes a factor of at most M+M_+. A replaced record changes its factor from at least m−m_- to at most M+M_+, a ratio of at most M+/m−M_+/m_-. We call cins(λ¯) c_ins( λ) =logM+=log(1+λ¯2), = M_+= \! (1+ λ2 ), (6) crep(λ¯) c_rep( λ) =logM+m−=log2+λ¯2−λ¯ = M_+m_-= 2+ λ2- λ the insertion cost and the replacement cost of a record, measured in nats (units of natural logarithm of wealth). For small caps, crep≈2cins≈λ¯c_rep≈ 2c_ins≈ λ. The corruption tolerance is the number of corrupted records that the evidence could have absorbed. With cost c∈cins(λ¯),crep(λ¯)c∈\c_ins( λ),c_rep( λ)\ we report B^t=max−1,⌊log(αKt)c⌋. B_t= \-1,\ (α K_t)c \. (7) In words, the evidence logKt K_t exceeds the threshold log(1/α) (1/α) of a test on uncorrupted data by log(αKt) (α K_t) nats, and each corrupted record supplies at most c of them. The insertion cost is valid against predictable attackers ( P), and the replacement cost against all classes ( O with constant bets; Theorem 5.1). 4.2 A grid of bets No single bet works well everywhere, so we run several bets and charge each its own cost before combining them. The best bet depends on the unknown margin, the amount by which i’s mean uncorrupted outcome exceeds 12 12. Smaller bets reach larger tolerances in the long run but need more records to start (Section 5.3). We therefore run m gamblers in parallel, indexed by k=1,…,mk=1,…,m. Gambler k has its own cap λ¯k∈(0,2) λ_k∈(0,2), decreasing from λ¯1 λ_1 to λ¯m λ_m, and makes the constant bet λ¯k λ_k at every record (bet k). These caps form a grid, a finite set of bet sizes spread so that one of them is close to the best bet for a wide range of margins. The gamblers have weights wk>0w_k>0 that satisfy ∑kwk≤1 _kw_k≤ 1. In all experiments λ¯k=2−(k−1)/2 λ_k=2^-(k-1)/2 for k=1,…,14k=1,…,14, from 11 down to 0.0110.011, and wk=1/14w_k=1/14. Let Kt(k)K^(k)_t be the wealth of Eq. (5) of gambler k, and write ck=c(λ¯k)c_k=c( λ_k) for the chosen cost of Eq. (6) at the cap λ¯k λ_k. A wealth is deflated by b records when it is divided by ebcke^b\,c_k, the most that b corrupted records could have multiplied it by (e is the base of the natural logarithm). The deflated average is Φt(b)=∑k=1mwkKt(k)e−bck,b≥0. _t(b)= _k=1^mw_k\,K^(k)_t\,e^-b\,c_k, b≥ 0. (8) We report B^t=maxb∈0,1,2,…:Φt(b)≥1/α, B_t= \b∈\0,1,2,…\:\ _t(b)≥ 1/α \, (9) and B^t=−1 B_t=-1 if Φt(0)<1/α _t(0)<1/α. Here b is a candidate value of NtN_t. With one bet (m=1m=1, w1=1w_1=1) this is Eq. (7). Our tolerance is never smaller than that of the best single bet run at level αwkα w_k. As Φt _t decreases in b, bisection finds B^t B_t. The procedure (Algorithm 1, Appendix A) stores m numbers per pair and needs O(mlogt)O(m t) operations per record. 4.3 An exponential wealth for the lead claim The lead claim needs a wealth with a logarithm linear in the outcome: for a bet size γ∈(0,4)γ∈(0,4) let Zt(γ)=exp(γ∑s≤t(Ws−12)−γ2t8),Z^(γ)_t= (γ _s≤ t (W_s- 12 )- γ^2t8 ), (10) with the costs cinsZ(γ)=γ/2−γ2/8c_ins^Z(γ)=γ/2-γ^2/8 and crepZ(γ)=γc_rep^Z(γ)=γ, both positive. A forged win raises logZt(γ) Z^(γ)_t by exactly γ/2−γ2/8γ/2-γ^2/8 and a replaced record by at most γ, hence the costs, and γ2t/8γ^2t/8 is the price of betting γ on outcomes in [0,1][0,1] (Hoeffding,, 1963). The lead tolerance B^tZ B^Z_t is Eq. (9) with Zt(γk)Z^( _k)_t for Kt(k)K^(k)_t, γk _k for the caps λ¯k λ_k, and cZ∈cinsZ,crepZc^Z∈\c_ins^Z,c_rep^Z\ for the cost c. 5 Theoretical Guarantees 5.1 Validity against adaptive attackers Theorem 5.1 (Validity). Let B^t B_t be the tolerance of Eq. (9) with constant bets λ¯1,…,λ¯m∈(0,2) λ_1,…, λ_m∈(0,2) and weights wk>0w_k>0, ∑kwk≤1 _kw_k≤ 1. (a) With the insertion cost c=cinsc=c_ins, B^t B_t is valid at level α against P. (b) With the replacement cost c=crepc=c_rep, it is valid against V. (c) With c=crepc=c_rep, it is valid against O under the null of O (Section 2.2). Parts (a) and (b) remain true if gambler k replaces its constant bet λ¯k λ_k by any bet λk,t∈[0,λ¯k] _k,t∈[0, λ_k] computed from the first t−1t-1 observed records. In Theorem 5.1 the attacker may corrupt any number of records, watch every published tolerance and spend its whole effort in one burst. The cheaper insertion cost suffices against forged records, and the replacement cost is required when records may be altered after being seen. Part (c) covers contamination by a developer who knows every test item in advance. 5.2 Neither cost can be lowered Theorem 5.2 (Sharpness). Consider a single constant bet λ∈(0,2)λ∈(0,2), write cins=cins(λ)c_ins=c_ins(λ) and crep=crep(λ)c_rep=c_rep(λ), let B be the attacker’s budget, and take uncorrupted values that are independent Bern(12)Bern( 12) and the tolerance of Eq. (7) with an arbitrary cost c>0c>0. (a) If c<cinsc<c_ins, an attacker in P that inserts B≥log(1/α)/(cins−c)B≥ (1/α)/(c_ins-c) forged wins before the first uncorrupted record causes a false statement with probability one. (b) If c<2cinsc<2c_ins, the attacker in V that replaces each of the first B uncorrupted losses by a win causes a false statement with probability tending to one as B→∞B→∞. In particular the insertion cost is invalid against V. (c) If 2cins≤c<crep2c_ins≤ c<c_rep, the attacker in V that replaces every uncorrupted loss by a win causes a false statement with probability at least αθe−θcinsα^θe^-θ c_ins, where θ∈[0,1)θ∈[0,1) is the largest root of eθcins+eθ(cins−c)=2e^θ c_ins+e^θ(c_ins-c)=2 (positive if and only if c>2cinsc>2c_ins). This exceeds α for all small enough α, so no cost below crepc_rep is valid at every level. (d) Among attackers in O that make no insertions and at most B replacements, the attacker in (b) maximizes KtK_t simultaneously for every t, and maximizes the probability of a false statement for every c≤crepc≤ c_rep. The same holds for the grid certificate of Eq. (9) with any costs 0<ck≤crep(λ¯k)0<c_k≤ c_rep( λ_k). By Theorem 5.2, what makes an attacker dangerous is acting on the values it sees, not seeing the future. Charging only the insertion cost, valid for predictable corruption, assumes the attacker cannot pick which uncorrupted records to corrupt. Flipping seen votes breaks this assumption. Both costs are sharp for a constant bet: none smaller is valid at every level. 5.3 How fast the tolerance grows For i.i.d. uncorrupted values without ties, the tolerance grows nearly as fast as any valid tolerance. Suppose no attack and i.i.d. uncorrupted values, so that Wt=XtW_t=X_t, with ℙ(W=1)=aP(W=1)=a, ℙ(W=0)=ℓ<aP(W=0)= <a and ties otherwise. The margin, the amount by which i’s mean outcome exceeds 12 12 (Section 4), is δ=(a−ℓ)/2>0δ=(a- )/2>0, and π=a+ℓπ=a+ is the probability of no tie. The expected log-wealth growth per record under the constant bet λ is g(λ)=log(1+λ(W−12))=δcrep(λ)−π2κ(λ),g(λ)=E \! (1+λ(W- 12) )=δ\,c_rep(λ)- π2κ(λ), (11) where κ(λ)=−log(1−λ2/4)≥0κ(λ)=- (1-λ^2/4)≥ 0. Let τb=inft:B^t≥b _b= \t: B_t≥ b\ be the number of records needed to reach tolerance b, and write x+=max(x,0)x^+= (x,0). Theorem 5.3 (Growth rate and time to certify). For such i.i.d. uncorrupted values and no attack, the tolerance of Eq. (9) with either cost c∈cins,crepc∈\c_ins,c_rep\ satisfies limt→∞B^t=ρ:=maxkg(λ¯k)+c(λ¯k), _t→∞ B_tt=ρ:= _k g( λ_k)^+c( λ_k), (12) almost surely and, for every integer b≥0b≥ 0, with min∅=+∞ =+∞, τb≤mink:g(λ¯k)>0log1αwk+bc(λ¯k)+cins(λ¯k)g(λ¯k).E\, _b≤ _k:\,g( λ_k)>0 1α w_k+b\,c( λ_k)+c_ins( λ_k)g( λ_k). (13) Moreover g(λ)/cins(λ)≥2δ−πλ/2g(λ)/c_ins(λ)≥ 2δ-πλ/2 and g(λ)/crep(λ)≥δ−πλ/6g(λ)/c_rep(λ)≥δ-πλ/6 for λ≤1λ≤ 1. Theorem 5.4 (Lower bound). Let δ∈(0,12)δ∈(0, 12) and let B^t B_t be any α-valid corruption tolerance against P (respectively V). Let q=2δq=2δ (respectively q=δq=δ). If the uncorrupted values are independent Bern(12+δ)Bern( 12+δ) and there is no attack, then for all n≥1n≥ 1 and real b≥0b≥ 0, ℙ(maxt≤nB^t≥b) ( _t≤ n B_t≥ b ) ≤α+ℙ(Bin(n,q)>b), ≤α+P (Bin(n,q)>b ), (14) ℙ(maxt≤nB^t≥0) ( _t≤ n B_t≥ 0 ) ≤nKLδ+log2log(1/α), ≤ n\,KL_δ+ 2 (1/α), (15) where Bin(n,q)Bin(n,q) is a binomial variable and KLδ=KL(Bern(12+δ)∥Bern(12))≤4δ2KL_δ=KL(Bern( 12+δ)\,\|\,Bern( 12))≤ 4δ^2 is a Kullback–Leibler divergence. Consequently, reaching an integer tolerance b by time n with probability at least 1−β1-β, for some β∈(0,1−α)β∈(0,1-α), requires n≥(1−α−β)(b+1)/qn≥(1-α-β)(b+1)/q and n≥[(1−β)log(1/α)−log2]/(4δ2)n≥[(1-β) (1/α)- 2]/(4δ^2). Moreover ℙ(B^n≥(q+η)n)≤α+e−2nη2P( B_n≥(q+η)n)≤α+e^-2nη^2 for every η>0η>0. Theorems 5.3 and 5.4 show that the growth rate approaches the optimum. By the last claim of Theorem 5.3 and by Theorem 5.4, ρ lies between q−λ¯m/2q- λ_m/2 (insertion, q=2δq=2δ) or q−λ¯m/6q- λ_m/6 (replacement, q=δq=δ) and q when λ¯m≤1 λ_m≤ 1, so ρ→qρ→ q as λ¯m→0 λ_m→ 0. The time to reach tolerance b is of order log(1/α)/δ2+b/δ (1/α)/δ^2+b/δ, the price of a sequential test plus a robustness price linear in b, with upper and lower bounds matching up to constants (Corollaries B.3 and B.4 in Appendix B). Faking a margin δ takes forging a fraction 2δ2δ of the records or flipping a fraction δ, and the tolerance reaches nearly that fraction. Figure 1: False-statement probability in n=5,000n=5,000 null records (win chance 12 12) versus the budget B (10,00010,000 runs; dotted: α). (a) B forged wins in one burst, timed to break undeflated betting. (b) The first B uncorrupted losses flipped, the best offline replacement attack (Theorem 5.2(d)). (c) B forged wins, each inserted before a record with probability B/nB/n. Markers hide the 95%95\% Monte Carlo intervals (half-width ≤0.01≤ 0.01). Baselines: Section 6. Figure 2: (a) Median and 1010–90%90\% band of B^t/t B_t/t at margin δ=0.1δ=0.1 (2,0002,000 repetitions), with the optimal rates 2δ2δ and δ. (b) Mean records to reach tolerance b at δ=0.05δ=0.05, with the upper bound of Theorem 5.3 and the lower bound of Corollary B.4. (c) Tolerance of four edge claims on real, time-ordered Chatbot Arena votes. 5.4 Selection and whole leaderboards Proposition 5.5 (Price of selection). Suppose a developer evaluates V variants of model i against model j, and the variant that each record concerns is fixed before its value is produced. Call a variant null if its records against j satisfy the null hypothesis of Eq. (1), and suppose that the null hypothesis of each null variant holds given the records of all variants. Let the published variant v^t v_t be chosen by any rule, possibly using all records. The grid certificate of Eq. (9) computed for v^t v_t with 1/α1/α replaced by V/αV/α makes a false statement about a null variant, at any time, with probability at most α, against the attacker classes and costs of Theorem 5.1(a,b). Conversely, suppose that α≤12α≤ 12, the V variants have independent Bern(12)Bern( 12) records and one constant bet λ is used. Then replacing V/αV/α by V′/αV /α with any 0<V′≤V/(2M+)0<V ≤ V/(2M_+), where M+=1+λ/2M_+=1+λ/2, gives a false statement with probability greater than α for some selection rule, even without an attack. For the best of at most V private variants, with V fixed in advance and variants possibly created as results come in, 1/α1/α must therefore scale with V, and V/αV/α suffices. Selection costs between logV−log(2M+) V- (2M_+) and logV V nats, about (logV)/c( V)/c records: 3535 forged records for the V=27V=27 variants of Singh et al., (2025) and λ=0.2λ=0.2, a quantitative argument for registering variants in advance. A leaderboard makes many claims at once, and the attacker spends one budget across all of them. Let NtN_t now count the corrupted records on all pairs up to (global) time t, and let Φtij(b) ^ij_t(b) be the deflated average of Eq. (8) for the ordered pair (i,j)(i,j). A joint claim is a set D of ordered pairs (i,j)(i,j) for which the edge claim is asserted. It is false if H0ijH_0^ij holds for some (i,j)∈D(i,j)∈ D. Proposition 5.6 (Simultaneous certificates). Assume that the pair of each record is fixed before its value is produced and that each true null H0ijH_0^ij holds given the records on all pairs. Let S be a family of sets of ordered pairs that contains the set S∗S of pairs whose null hypotheses are true. For a joint claim D that intersects some S∈S , let B^t(D) B_t(D) be the largest integer b≥0b≥ 0 such that |S|−1∑(i,j)∈SΦtij(b)≥1/α|S|^-1 _(i,j)∈ S ^ij_t(b)≥ 1/α for every S∈S with S∩D≠∅S∩ D≠ (−1-1 if there is none). Against attackers in P (with cinsc_ins) or V (with crepc_rep), ℙ(∃t,∃ false joint claim D:B^t(D)≥Nt)≤αP (∃\,t,\ ∃ false joint claim D:\ B_t(D)≥ N_t )≤α. Every published comparison, rank confidence set or set of the r best models can thus carry one tolerance, valid over all claims and times against one attacker. This is closed testing (Marcus et al.,, 1976) on deflated wealths, averaged as in anytime-valid rank sets (Khosravi and Huo,, 2026). With S all sets of pairs it needs no assumption on which claims are correct. 5.5 The lead claim Theorem 5.7 (Validity for the lead claim). Let B^tZ B^Z_t be the lead tolerance of Section 4.3 with constants 4>γ1>⋯>γm>04> _1>…> _m>0 and weights wk>0w_k>0, ∑kwk≤1 _kw_k≤ 1. (a) With the cost γ/2−γ2/8γ/2-γ^2/8, B^tZ B^Z_t is α-valid for the lead claim against the attackers in P that only insert, and (b) with the cost γ against V. (c) Against O the same holds with the cost γ and the margin of the uncorrupted sequence. (d) In (a) to (c), B^tZ B^Z_t is also α-valid in the sense of Eq. (3). For i.i.d. uncorrupted values with mean 12+δ 12+δ, ties allowed, and no attack, almost surely B^tZ/t→ρZ=maxk(γkδ−γk2/8)+/cZ(γk) B^Z_t/t→ρ^Z= _k( _kδ- _k^2/8)^+/c^Z( _k), with ρZ≥2δ−γm/4ρ^Z≥ 2δ- _m/4 (insertion cost) and ρZ≥δ−γm/8ρ^Z≥δ- _m/8 (replacement cost). Theorem 5.7 gives the lead claim the same guarantee as the edge claim, at nearly the same costs and the same optimal rates. The difference is in part (a). A record overwritten unseen still enters Δt _t through its hidden value, so part (a) covers insertions only, unlike Theorem 5.1(a) for the edge claim. A tie costs the exponential wealth a little and the betting wealth nothing, so the lead tolerance falls slightly below the edge tolerance under frequent ties. Simultaneous lead certificates follow by a Bonferroni correction (Corollary B.7). Proofs in Appendix B. 6 Experiments We check the results by simulation and by replaying rigging attacks on real Arena votes and benchmark items. The setup is in Section C.1. Methods. Ours is the grid tolerance of Eq. (9), run as the insertion certificate for forged records and as the replacement certificate for flipped or contaminated ones. The baselines are undeflated betting, a per-step robust test of the kind in Proposition 3.1, and a Wald interval recomputed as records arrive. 6.1 Validity against three attacks This experiment tests whether each method keeps its error at α under the attacks of Theorems 5.1 and 5.2. Figure 1 shows that only the replacement certificate stays valid against every attacker and that only flips defeat the insertion certificate, as the theorems predict. Under a burst of forged wins (panel a), undeflated betting and the per-step robust test err with probability 0.330.33 and 0.270.27 at B=20B=20 and always for B≥40B≥ 40, while our insertion certificate stays at or below 0.0480.048. With losses flipped (panel b), the insertion certificate fails from B=5B=5 (at B=20B=20, 0.370.37 with a single bet and 0.940.94 with the grid) and the replacement certificate stays at or below 0.0500.050. With forged wins spread at random (panel c) the per-step robust test is valid. Standard errors are ≤0.005≤ 0.005. 6.2 Growth of the tolerance This experiment measures how fast the tolerance grows against the rates of Section 5.3. Figure 2(a,b) shows that the tolerance per record rises toward the optimal rates 2δ2δ and δ and that the upper bound on the time to certify is nearly attained. At δ=0.1δ=0.1 and t=8,000t=8,000 the median B^t/t B_t/t is 0.170.17 (insertion) and 0.0830.083 (replacement), against 0.20.2 and 0.10.1. For b≥50b≥ 50 the mean τb _b is at most 19%19\% below Eq. (13); at b=400b=400 it is at most 1.91.9 times the lower bound on the mean time to certify (Corollary B.4, Appendix B) for δ=0.05δ=0.05 and replacement needs 1.61.6–1.91.9 times the records of insertion. 6.3 Chatbot Arena and static benchmarks We test the certificates on real data because that is where they would be used, with uneven pair counts, frequent ties and attacks on published standings. Arena votes. We certify pairwise claims on the time-ordered votes of the public Arena file (1.81.8 million votes, 129129 models, April 2023 to August 2024, 36%36\% ties; Chiang et al.,, 2024; LMSYS Org / LMArena,, 2024). The certificates need a battle’s pair fixed before its vote, which every sampling rule satisfies. Figure 2(c) shows four pairs over time. Well-separated models are certified against about 1,3001,300 to 2,0002,000 forged votes and half as many flipped ones, the closest pair about 200200. For separated pairs the completed vote set withstands only slightly more, and the lead certificate slightly less (Table 5 in Appendix C). When all pairs are claimed at once, no certificate separates GPT-4o (2024-05-13) and Claude-3.5-Sonnet (Figure 3 in Appendix C). Table 1: Arena rigging replay. Probability of certifying a false ordering at B=0B=0 / 100100 / 400400 attacker votes (mean of three pairs, 1,0001,000 replays each; valid if at most 0.050.05). Burst: forged votes at one random time. Flip: the weaker model’s losses flipped. Method Burst Flip Wald interval .07 / .40 / 1.0 .07 / 1.0 / 1.0 Undeflated betting .00 / .22 / .99 .00 / 1.0 / 1.0 Per-step robust .01 / .13 / .28 .01 / 1.0 / 1.0 Ours, insertion .00 / .01 / .01 .00 / 1.0 / 1.0 Ours, replacement .00 / .01 / .01 .00 / .01 / .01 Rigging replay. Table 1 asks whether a few hundred rigged votes make a method certify a false ordering. For three pairs whose promoted model is slightly worse, an attacker adds or flips votes in resampled replays. Every baseline breaks at a few hundred votes, while the matching certificate stays valid. Static benchmarks. Benchmarks are the setting of the offline attacker. A developer who knows every test item can contaminate the ones it chooses, so the replacement certificate is required. On Massive Multitask Language Understanding (MMLU; 14,04214,042 questions, four open-weight 7–9B models; Liang et al.,, 2023; Stanford CRFM,, 2025; Hendrycks et al.,, 2021) well-separated pairs withstand about nine tenths of the replaced items the released benchmark could withstand at most (Proposition B.8). On five frontier benchmarks (MMLU-Pro, GPQA, IFEval, Omni-MATH and SWE-bench Verified; Wang et al.,, 2024; Rein et al.,, 2024; Zhou et al.,, 2023; Gao et al.,, 2025; Jimenez et al.,, 2024; OpenAI,, 2024; SWE-bench team,, 2026) no top-two ordering is certified when all pairs are claimed at once, and no top-five claim withstands over 77 contaminated items (Figure 6 in Appendix C). Additional experiments. Appendix C also reports the price of selecting among variants (E4), a regime shift (E5), attacks on correct claims (E6), the rigging replay on benchmark items (E7), the lead claim under ties (E9) and fixed-time breakdown counts (E10). 7 Conclusion and Discussion Leaderboards face adaptive attackers, and existing guarantees assume genuine records or a per-step corruption level. We introduce corruption tolerance, the number of corrupted records a ranking claim withstands, and certify it at all times against attackers who watch the published tolerances and corrupt in bursts. It charges about twice as much for an altered as for a forged record and grows nearly as fast as possible on i.i.d. records without ties. A second certificate covers the claim that a model leads on average so far, and selection among V variants costs about logV V nats. On Arena it certifies some orderings against about 2,0002,000 forged votes but not GPT-4o over Claude-3.5-Sonnet claimed at once (limitations: Section D.4). Reproducibility. Details are in Section C.2. Code, data and saved results are at https://github.com/HamedKhosravi99/corruption-tolerance. References Agarwal et al., (2020) Agarwal, A., Agarwal, S., Khanna, S., and Patil, P. (2020). Rank aggregation from pairwise comparisons in the presence of adversarial corruptions. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 85–95. Agrawal et al., (2024) Agrawal, S., Mathieu, T., Basu, D., and Maillard, O.-A. (2024). CRIMED: Lower and upper bounds on regret for bandits with unbounded stochastic corruption. In Proceedings of the 35th International Conference on Algorithmic Learning Theory (ALT), volume 237 of Proceedings of Machine Learning Research, pages 74–124. Altschuler et al., (2019) Altschuler, J., Brunel, V.-E., and Malek, A. (2019). Best arm identification for contaminated bandits. Journal of Machine Learning Research, 20(91):1–39. Andrews et al., (2024) Andrews, I., Kitagawa, T., and McCloskey, A. (2024). Inference on winners. The Quarterly Journal of Economics, 139(1):305–358. Blanchard et al., (2017) Blanchard, P., El Mhamdi, E. M., Guerraoui, R., and Stainer, J. (2017). Machine learning with adversaries: Byzantine tolerant gradient descent. In Advances in Neural Information Processing Systems 30 (NIPS 2017). Bose et al., (2025) Bose, A., Lessard, L., Fazel, M., and Dvijotham, K. D. (2025). Keeping up with dynamic attackers: Certifying robustness to adaptive online data poisoning. In Proceedings of the 28th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 258 of Proceedings of Machine Learning Research, pages 4438–4446. Bradley and Terry, (1952) Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrika, 39(3/4):324–345. Chiang et al., (2024) Chiang, W.-L., Zheng, L., Sheng, Y., et al. (2024). Chatbot Arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 8359–8388. Darling and Robbins, (1967) Darling, D. A. and Robbins, H. (1967). Confidence sequences for mean, variance, and median. Proceedings of the National Academy of Sciences, 58(1):66–68. Diakonikolas and Kane, (2023) Diakonikolas, I. and Kane, D. M. (2023). Algorithmic High-Dimensional Robust Statistics. Cambridge University Press. Fischer and Ramdas, (2024) Fischer, L. and Ramdas, A. (2024). Admissible online closed testing must employ e-values. arXiv:2407.15733. Gao et al., (2025) Gao, B., Song, F., Yang, Z., et al. (2025). Omni-MATH: A universal olympiad level mathematic benchmark for large language models. In International Conference on Learning Representations (ICLR). Gao et al., (2026) Gao, H., Zhang, S., Xie, Z., Jing, B., Wang, Z., and Liu, Y. (2026). Anytime-valid LLM leaderboards via benchmark-weighted and block-factorized e-processes. arXiv:2609.32248. Grünwald et al., (2024) Grünwald, P., de Heide, R., and Koolen, W. (2024). Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(5):1091–1128. Gu et al., (2026) Gu, R., Sun, W., Gang, B., and Xia, X. (2026). Anytime-valid inference for online ranking of large language models. In Proceedings of the 43rd International Conference on Machine Learning (ICML), volume 306 of Proceedings of Machine Learning Research, pages 37229–37260. Gupta et al., (2019) Gupta, A., Koren, T., and Talwar, K. (2019). Better algorithms for stochastic bandits with adversarial corruptions. In Proceedings of the Thirty-Second Conference on Learning Theory (COLT), volume 99 of Proceedings of Machine Learning Research, pages 1562–1578. Hendrycks et al., (2021) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2021). Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR). Hoeffding, (1963) Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Journal of the American Statistical Association, 58(301):13–30. Howard et al., (2021) Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. (2021). Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055–1080. Huang et al., (2026) Huang, J. Y., Shen, Y., Wei, D., and Broderick, T. (2026). Dropping just a handful of preferences can change top large language model rankings. In International Conference on Learning Representations (ICLR). Huang et al., (2025) Huang, Y., Nasr, M., Angelopoulos, A. N., et al. (2025). Exploring and mitigating adversarial manipulation of voting-based leaderboards. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 25654–25671. Huber, (1964) Huber, P. J. (1964). Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101. Huber, (1965) Huber, P. J. (1965). A robust version of the probability ratio test. The Annals of Mathematical Statistics, 36(6):1753–1758. Jia et al., (2021) Jia, J., Cao, X., and Gong, N. Z. (2021). Intrinsic certified robustness of bagging against data poisoning attacks. Proceedings of the AAAI Conference on Artificial Intelligence, 35(9):7961–7969. Jimenez et al., (2024) Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. (2024). SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR). Kaufmann et al., (2016) Kaufmann, E., Cappé, O., and Garivier, A. (2016). On the complexity of best-arm identification in multi-armed bandit models. Journal of Machine Learning Research, 17(1):1–42. Khosravi and Huo, (2026) Khosravi, H. and Huo, X. (2026). Rank confidence sequences: Anytime-valid leaderboards. arXiv:2609.32211. Lamport et al., (1982) Lamport, L., Shostak, R., and Pease, M. (1982). The Byzantine generals problem. ACM Transactions on Programming Languages and Systems, 4(3):382–401. Levine and Feizi, (2021) Levine, A. and Feizi, S. (2021). Deep partition aggregation: Provable defenses against general poisoning attacks. In International Conference on Learning Representations (ICLR). Liang et al., (2023) Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic evaluation of language models. Transactions on Machine Learning Research. arXiv:2211.09110. LMSYS Org / LMArena, (2024) LMSYS Org / LMArena (2024). Chatbot Arena public battle data, file clean_battle_20240814_public.json. https://storage.googleapis.com/arena_external_data/public/. Public battle file, accessed 29 September 2026; no license stated. Luo and Gao, (2026) Luo, Y. and Gao, C. (2026). Adaptive robust confidence intervals. The Annals of Statistics, 54(3):1128–1152. Lykouris et al., (2018) Lykouris, T., Mirrokni, V., and Paes Leme, R. (2018). Stochastic bandits robust to adversarial corruptions. In Proceedings of the 50th Annual ACM SIGACT Symposium on Theory of Computing (STOC), pages 114–122. Ma et al., (2024) Ma, K., Xu, Q., Zeng, J., Liu, W., Cao, X., Sun, Y., and Huang, Q. (2024). Sequential manipulation against rank aggregation: Theory and algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):9353–9370. Marcus et al., (1976) Marcus, R., Peritz, E., and Gabriel, K. R. (1976). On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3):655–660. Min et al., (2025) Min, R., Pang, T., Du, C., Liu, Q., Cheng, M., and Lin, M. (2025). Improving your model ranking on Chatbot Arena by vote rigging. In Proceedings of the 42nd International Conference on Machine Learning (ICML), volume 267 of Proceedings of Machine Learning Research, pages 44252–44271. Mogstad et al., (2024) Mogstad, M., Romano, J. P., Shaikh, A. M., and Wilhelm, D. (2024). Inference for ranks with applications to mobility across neighbourhoods and academic achievement across countries. The Review of Economic Studies, 91(1):476–518. OpenAI, (2024) OpenAI (2024). Introducing SWE-bench Verified. https://openai.com/index/introducing-swe-bench-verified/. Blog post, 13 August 2024. Oyarhoseini et al., (2026) Oyarhoseini, H., Lin, J., and Karimi, A.-H. (2026). A unified perturbation framework for analyzing leaderboard stability and manipulation. arXiv:2605.15761. Ramdas et al., (2023) Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G. (2023). Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601. Rein et al., (2024) Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., and Bowman, S. R. (2024). GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling (COLM). Saha and Ramdas, (2026) Saha, A. and Ramdas, A. (2026). Huber-robust likelihood ratio tests for composite nulls and alternatives. IEEE Transactions on Information Theory, 72(1):501–520. Sainz et al., (2023) Sainz, O., Campos, J. A., García-Ferrero, I., Etxaniz, J., Lopez de Lacalle, O., and Agirre, E. (2023). NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore. Association for Computational Linguistics. Shafer, (2021) Shafer, G. (2021). Testing by betting: A strategy for statistical and scientific communication. Journal of the Royal Statistical Society Series A: Statistics in Society, 184(2):407–431. Shekhar and Ramdas, (2024) Shekhar, S. and Ramdas, A. (2024). Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory, 70(2):1178–1203. Singh et al., (2025) Singh, S., Nan, Y., Wang, A., et al. (2025). The leaderboard illusion. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025), Datasets and Benchmarks Track, pages 86910–86964. Stanford CRFM, (2025) Stanford CRFM (2025). HELM leaderboards: Per-instance outputs (MMLU release v1.13.0; Capabilities release v1.15.0). https://crfm.stanford.edu/helm/. Public bucket crfm-helm-public, accessed 3 October 2026. Steinhardt et al., (2017) Steinhardt, J., Koh, P. W., and Liang, P. (2017). Certified defenses for data poisoning attacks. In Advances in Neural Information Processing Systems 30 (NIPS 2017). SWE-bench team, (2026) SWE-bench team (2026). SWE-bench experiments: Evaluation runs on SWE-bench Verified (evaluation/verified). https://github.com/SWE-bench/experiments. Bash-only runs of mini-SWE-agent, accessed 3 October 2026; no license stated. Tang et al., (2026) Tang, E. Y. N., Chen, Y., Li, M., and Yu, Y. (2026). Online change point detection under heavy-tailedness and contamination. arXiv:2606.09737. Ville, (1939) Ville, J. (1939). Étude critique de la notion de collectif. Number 3 in Monographies des Probabilités. Gauthier-Villars, Paris. Vovk and Wang, (2021) Vovk, V. and Wang, R. (2021). E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736–1754. Wald, (1945) Wald, A. (1945). Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2):117–186. Wang and Ramdas, (2023) Wang, H. and Ramdas, A. (2023). Huber-robust confidence sequences. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 206 of Proceedings of Machine Learning Research, pages 9662–9679. Wang et al., (2024) Wang, Y., Ma, X., Zhang, G., et al. (2024). MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, pages 95266–95290. Waudby-Smith and Ramdas, (2024) Waudby-Smith, I. and Ramdas, A. (2024). Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1–27. Xu et al., (2024) Xu, C., Guan, S., Greene, D., and Kechadi, M.-T. (2024). Benchmark data contamination of large language models: A survey. arXiv:2406.04244. Yang et al., (2024) Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., and Press, O. (2024). SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), pages 50528–50652. Zhao et al., (2025) Zhao, W., Rush, A. M., and Goyal, T. (2025). Challenges in trustworthy human evaluation of chatbots. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3359–3365, Albuquerque, New Mexico. Association for Computational Linguistics. Zheng et al., (2023) Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track, pages 46595–46623. Zhong et al., (2021) Zhong, Z., Cheung, W. C., and Tan, V. Y. F. (2021). Probabilistic sequential shrinking: A best arm identification algorithm for stochastic bandits with corruptions. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 12772–12781. Zhou et al., (2023) Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. (2023). Instruction-following evaluation for large language models. arXiv:2311.07911. Contents of the Appendix Appendix A The Algorithm Algorithm 1 Corruption tolerance for the edge claim 0: level α; constant bets λ¯1,…,λ¯m∈(0,2) λ_1,…, λ_m∈(0,2); weights wk>0w_k>0 with ∑kwk≤1 _kw_k≤ 1; cost c=cinsc=c_ins (attackers in P) or c=crepc=c_rep (attackers in P, V or O) 1: Gk←0G_k← 0 for k=1,…,mk=1,…,m ⊳ log-wealths 2: for each new record t=1,2,…t=1,2,… with outcome WtW_t do 3: Gk←Gk+log(1+λ¯k(Wt−12))G_k← G_k+ (1+ λ_k(W_t- 12) ) for all k 4: Φt(b):=∑kwkexpGk−bc(λ¯k) _t(b):= _kw_k \G_k-b\,c( λ_k)\ 5: publish B^t=maxb∈0,1,…:Φt(b)≥1/α B_t= \b∈\0,1,…\: _t(b)≥ 1/α\ (bisection), or −1-1 if Φt(0)<1/α _t(0)<1/α 6: end for Why each wealth is deflated by its own cost. Each wealth must be deflated by its own cost before averaging. Deflating the average of the wealths instead would have to use the largest cost in the grid, which would waste the small bets, the ones that reach large tolerances. Appendix B Proofs B.1 Formal model and two facts used throughout Filtrations. All random objects live on one probability space. The attacker’s private randomization is a random element U. For attackers in P and V there is a filtration (t)t≥0(G_t)_t≥ 0, with U being 0G_0-measurable, that has four properties. The observed outcome WtW_t, the corruption indicator CtC_t and, if record t is not inserted, its uncorrupted value XtX_t are tG_t-measurable. The indicator ItI_t that record t is inserted is t−1G_t-1-measurable. The observed σ-field ℱtF_t, generated by W1,…,WtW_1,…,W_t, satisfies ℱt⊆tF_t _t. Finally, Wt=XtW_t=X_t whenever Ct=0C_t=0. On It=1\I_t=1\ we set Xt=12X_t= 12 for definiteness; every expression below that involves XtX_t on an inserted record is multiplied by 1−It1-I_t or 1−Ct1-C_t, so the choice has no effect. Writing ⋅1\·\ for the indicator of an event, the null hypothesis of Eq. (1) reads [(Xt−12)It=0∣t−1]≤0E[(X_t- 12)1\I_t=0\ _t-1]≤ 0. Every attacker that satisfies these conditions is in V. Its CtC_t and WtW_t may depend on XtX_t. It is in P if, in addition, CtC_t is t−1G_t-1-measurable. This class is slightly larger than the one described in Section 2.2, which also fixes the corrupted value in advance. The proof of Theorem 5.1(a) does not need that. For attackers in O the uncorrupted values form a sequence (Xu)u≥1(X_u)_u≥ 1 whose law does not depend on the attacker, with [Xu∣X1,…,Xu−1]≤12E[X_u X_1,…,X_u-1]≤ 12. The observed record t is either inserted or holds the uncorrupted value Xh(t)X_h(t), possibly replaced, where h(t)h(t) is the number of non-inserted records among the first t. The sequence (Wt,Ct,It)t≥1(W_t,C_t,I_t)_t≥ 1 may be any measurable function of (Xu)u≥1(X_u)_u≥ 1 and U such that Wt=Xh(t)W_t=X_h(t) whenever Ct=0C_t=0. In every class Nt=∑s≤tCsN_t= _s≤ tC_s. Bets. Bet k uses an ℱt−1F_t-1-measurable λk,t∈[0,λ¯k] _k,t∈[0, λ_k] (constant bets λk,t=λ¯k _k,t= λ_k are a special case). Then Kt(k)=∏s≤t(1+λk,s(Ws−12))K^(k)_t= _s≤ t (1+ _k,s(W_s- 12) ), and each factor 1+λk,t(x−12)1+ _k,t(x- 12) lies in [1−λ¯k/2, 1+λ¯k/2][1- λ_k/2,\,1+ λ_k/2] for x∈[0,1]x∈[0,1]. Let Φt(b)=∑k=1mwkKt(k)e−bc(λ¯k),B^t=maxb∈0,1,…:Φt(b)≥1/α, _t(b)= _k=1^mw_kK^(k)_te^-b\,c( λ_k), B_t= \b∈\0,1,…\: _t(b)≥ 1/α\, with B^t=−1 B_t=-1 if Φt(0)<1/α _t(0)<1/α. This is Eq. (9). With m=1m=1 and w1=1w_1=1 it is Eq. (7). Because Φt _t is decreasing in b and the set in the definition is finite when Φt(0)≥1/α _t(0)≥ 1/α (as Φt(b)→0 _t(b)→ 0), B^t B_t is well defined. The proofs compare the observed wealth with an uncorrupted wealth, a wealth computed from uncorrupted values alone. Its form depends on the attacker class. Lemma B.1 (Reduction to an uncorrupted wealth). Suppose there are nonnegative processes L(k)L^(k) with Kt(k)≤ec(λ¯k)NtLt(k)K^(k)_t≤ e^c( λ_k)N_tL^(k)_t for all k and t, pathwise. Then B^t≥Nt⊆∑kwkLt(k)≥1/α\ B_t≥ N_t\ \ _kw_kL^(k)_t≥ 1/α\ for every t. Proof. On B^t≥Nt\ B_t≥ N_t\ we have B^t≥0 B_t≥ 0, so Φt(B^t)≥1/α _t( B_t)≥ 1/α, and since Φt _t is decreasing, Φt(Nt)≥Φt(B^t)≥1/α _t(N_t)≥ _t( B_t)≥ 1/α. Finally Φt(Nt)=∑kwkKt(k)e−c(λ¯k)Nt≤∑kwkLt(k) _t(N_t)= _kw_kK^(k)_te^-c( λ_k)N_t≤ _kw_kL^(k)_t. ∎ Lemma B.2 (Ville’s inequality). If (Lt)t≥0(L_t)_t≥ 0 is a nonnegative supermartingale with respect to some filtration and L0≤1L_0≤ 1, then ℙ(∃t:Lt≥1/α)≤αP(∃ t:L_t≥ 1/α)≤α. This is classical. Ville, (1939) proved it for nonnegative martingales that start at one. Howard et al., (2021) use it in the form above, for nonnegative supermartingales. A weighted sum ∑kwkLt(k) _kw_kL^(k)_t of nonnegative supermartingales with respect to the same filtration, with ∑kwk≤1 _kw_k≤ 1 and L0(k)=1L^(k)_0=1, is again a nonnegative supermartingale, with initial value at most 11. B.2 Proof of Proposition 3.1 Under the null every Bern(p)Bern(p) with p≤12p≤ 12 is an allowed center. The distance from the point mass at 11 to Bern(p)Bern(p) is 1−p1-p, which is smallest, 12 12, at p=12p= 12, so a forged win is allowed only if ε≥12 ≥ 12. Conversely, the distance from Bern(r)Bern(r) to the allowed center Bern(12)Bern( 12) is |r−12|≤12|r- 12|≤ 12 for every r∈[0,1]r∈[0,1]. Hence, if ε≥12 ≥ 12, every conditional law of a binary outcome is allowed at every step, so every law of a binary sequence satisfies the null. A test of level α then rejects with probability at most α under each of them. □ B.3 Proof of Theorem 5.1 (a) Class P, cost cinsc_ins. Define Lt(k)=∏s≤t(1+(1−Cs)λk,s(Xs−12))L^(k)_t= _s≤ t (1+(1-C_s) _k,s(X_s- 12) ), where the factor is 11 when Cs=1C_s=1. This L(k)L^(k) is the uncorrupted wealth for P. It applies the same bets λk,s _k,s to the uncorrupted records and skips every corrupted one. Since CtC_t and λk,t _k,t are t−1G_t-1-measurable and Ct=0C_t=0 implies It=0I_t=0, [Lt(k)∣t−1]=Lt−1(k)(1+(1−Ct)λk,t[(Xt−12)It=0∣t−1])≤Lt−1(k),E[L^(k)_t _t-1]=L^(k)_t-1 (1+(1-C_t) _k,t\,E [(X_t- 12)1\I_t=0\ _t-1 ] )≤ L^(k)_t-1, using (1−Ct)It=0=1−Ct(1-C_t)1\I_t=0\=1-C_t. So L(k)L^(k) is a nonnegative supermartingale for (t)(G_t) with L0(k)=1L^(k)_0=1. Pathwise, compare the factors of record s. If Cs=0C_s=0 then Ws=XsW_s=X_s and they agree. If Cs=1C_s=1, the factor of K(k)K^(k) is at most 1+λ¯k/2=ecins(λ¯k)1+ λ_k/2=e^c_ins( λ_k) and that of L(k)L^(k) is 11. Hence Kt(k)≤ecins(λ¯k)NtLt(k)K^(k)_t≤ e^c_ins( λ_k)N_tL^(k)_t. By Lemmas B.1 and B.2, ℙ(∃t:B^t≥Nt)≤ℙ(∃t:∑kwkLt(k)≥1/α)≤αP(∃ t: B_t≥ N_t)≤P(∃ t: _kw_kL^(k)_t≥ 1/α)≤α. (b) Class V, cost crepc_rep. Define Lt(k)=∏s≤t(1+(1−Is)λk,s(Xs−12))L^(k)_t= _s≤ t (1+(1-I_s) _k,s(X_s- 12) ), which keeps the uncorrupted value of every replaced record. Since ItI_t and λk,t _k,t are t−1G_t-1-measurable, the same computation shows that L(k)L^(k) is a nonnegative supermartingale for (t)(G_t). Pathwise, compare the factors of record s. If Cs=0C_s=0 they agree. If s is inserted, the ratio of the factor of K(k)K^(k) to that of L(k)L^(k) is at most 1+λ¯k/2≤ecrep(λ¯k)1+ λ_k/2≤ e^c_rep( λ_k). If s is replaced, the ratio is at most (1+λk,s/2)/(1−λk,s/2)≤(1+λ¯k/2)/(1−λ¯k/2)=ecrep(λ¯k)(1+ _k,s/2)/(1- _k,s/2)≤(1+ λ_k/2)/(1- λ_k/2)=e^c_rep( λ_k), because x↦(1+x/2)/(1−x/2)x (1+x/2)/(1-x/2) is increasing on [0,2)[0,2). Hence Kt(k)≤ecrep(λ¯k)NtLt(k)K^(k)_t≤ e^c_rep( λ_k)N_tL^(k)_t, and we conclude as in (a). (c) Class O, cost crepc_rep, constant bets. Let L~u(k)=∏u′≤u(1+λ¯k(Xu′−12)) L^(k)_u= _u ≤ u (1+ λ_k(X_u - 12) ), a nonnegative supermartingale for the natural filtration of (Xu)(X_u), and set Lt(k)=L~h(t)(k)L^(k)_t= L^(k)_h(t). Record t of the observed sequence may be inserted. It then contributes a factor at most 1+λ¯k/2≤ecrep(λ¯k)1+ λ_k/2≤ e^c_rep( λ_k) to Kt(k)K^(k)_t and none to Lt(k)L^(k)_t. Otherwise it holds the uncorrupted value Xh(t)X_h(t), and its observed factor is at most ecrep(λ¯k)e^c_rep( λ_k) times its uncorrupted factor if the value was replaced and equal to it otherwise. Because the bet is constant, the uncorrupted factor does not depend on where the record appears in the observed sequence. Hence Kt(k)≤ecrep(λ¯k)NtL~h(t)(k)K^(k)_t≤ e^c_rep( λ_k)N_t L^(k)_h(t) pathwise. By Lemma B.1, ∃t:B^t≥Nt⊆∃u:∑kwkL~u(k)≥1/α\∃ t: B_t≥ N_t\ \∃ u: _kw_k L^(k)_u≥ 1/α\, an event that depends on the uncorrupted sequence alone and has probability at most α by Lemma B.2. The argument does not use how the attacker chooses its actions. In particular U may depend on (Xu)(X_u), and only the law of (Xu)(X_u) must not depend on the attacker. Adaptive bets in (c). With a bet computed from the observed records, the factor of the uncorrupted value XuX_u depends on observed records that an offline attacker may have chosen after seeing XuX_u, so L~(k) L^(k) need not be a supermartingale. We do not claim validity in that case. □ B.4 Proof of Theorem 5.2 Throughout, the bet is the constant λ, and for an integer N≥0N≥ 0, B^t≥N B_t≥ N if and only if log(αKt)≥cN (α K_t)≥ cN. (a) The attacker inserts B wins as records 1,…,B1,…,B. Then KB=(1+λ/2)B=eBcinsK_B=(1+λ/2)^B=e^Bc_ins and NB=BN_B=B, so B^B≥NB B_B≥ N_B if and only if Bcins−log(1/α)≥BcBc_ins- (1/α)≥ Bc, that is, B≥log(1/α)/(cins−c)B≥ (1/α)/(c_ins-c). This event does not depend on the uncorrupted values. (b) Let TBT_B be the position of the BBth uncorrupted loss and RB=TB−BR_B=T_B-B the number of uncorrupted wins before it. The attacker replaces the first B losses, so records 1,…,TB1,…,T_B are all observed as wins, KTB=eTBcinsK_T_B=e^T_Bc_ins and NTB=BN_T_B=B. A false statement is made at TBT_B if TBcins−log(1/α)≥BcT_Bc_ins- (1/α)≥ Bc, that is, if RB≥Bc−cinscins+log(1/α)cins.R_B\ ≥\ B\, c-c_insc_ins+ (1/α)c_ins. Since RBR_B is a sum of B independent geometric variables with mean 11, RB/B→1R_B/B→ 1 almost surely. If c<2cinsc<2c_ins, then (c−cins)/cins<1(c-c_ins)/c_ins<1, so the probability of the display tends to one as B→∞B→∞. Since cins<2cinsc_ins<2c_ins, this applies in particular to c=cinsc=c_ins. (c) The attacker replaces every uncorrupted loss by a win. Then every observed record is a win and logKt=tcins K_t=tc_ins. Also Nt=ΛtN_t= _t, the number of uncorrupted losses among the first t records. A false statement occurs at t if and only if St:=tcins−cΛt≥log(1/α)S_t:=tc_ins-c _t≥ (1/α). The walk S has independent increments equal to cinsc_ins or cins−c_ins-c with probability 12 12 each. If c=2cinsc=2c_ins, the walk has mean-zero bounded increments and is recurrent, so it reaches log(1/α) (1/α) with probability one. If 2cins<c<crep2c_ins<c<c_rep, let φ(θ)=12(eθcins+eθ(cins−c)) (θ)= 12(e^θ c_ins+e^θ(c_ins-c)). It is convex with φ(0)=1 (0)=1 and φ′(0)=cins−c/2<0 (0)=c_ins-c/2<0, and φ(1)=12M+(1+e−c)>12M+(1+m−/M+)=1 (1)= 12M_+(1+e^-c)> 12M_+(1+m_-/M_+)=1 because e−c>e−crep=m−/M+e^-c>e^-c_rep=m_-/M_+. So there is a unique θ∈(0,1)θ∈(0,1) with φ(θ)=1 (θ)=1, and eθSte^θ S_t is a martingale. The drift of S is negative, so St→−∞S_t→-∞ almost surely. Let σ=inft:St≥log(1/α)σ= \t:S_t≥ (1/α)\ and write t∧σ=min(t,σ)t σ= (t,σ). Since St∧σ<log(1/α)+cinsS_t σ< (1/α)+c_ins, bounded convergence gives 1=[eθSσ<∞]≤eθ(log(1/α)+cins)ℙ(σ<∞)1=E[e^θ S_σ1\σ<∞\]≤ e^θ( (1/α)+c_ins)P(σ<∞), so ℙ(σ<∞)≥αθe−θcinsP(σ<∞)≥α^θe^-θ c_ins. This exceeds α if and only if α<e−θcins/(1−θ)α<e^-θ c_ins/(1-θ). (d) Consider any attacker in O that makes no insertions and at most B replacements, and fix the uncorrupted sequence. Because the bet is constant and nothing is inserted, replacing record s multiplies KtK_t, for every t≥st≥ s, by the ratio of the new factor to the uncorrupted one. Whatever value in [0,1][0,1] the attacker writes, this ratio is at most ecrepe^c_rep if the uncorrupted value was a loss and at most 11 if it was a win. Let Λt _t be the number of uncorrupted losses among the first t records, and AtA_t the number of these whose value was replaced. Then logKt≤logL~t+crepAt K_t≤ L_t+c_repA_t, At≤min(B,Λt)A_t≤ (B, _t) and Nt≥AtN_t≥ A_t, where L~t=∏s≤t(1+λ(Xs−12)) L_t= _s≤ t(1+λ(X_s- 12)) is the wealth of the uncorrupted sequence, as in the proof of Theorem 5.1(c). The attacker of (b) achieves At=Nt=min(B,Λt)A_t=N_t= (B, _t) and logKt=logL~t+crepmin(B,Λt) K_t= L_t+c_rep (B, _t) for all t, which is the maximum of KtK_t. Moreover, for c≤crepc≤ c_rep, log(αKt)−cNt≤log(αL~t)+(crep−c)At≤log(αL~t)+(crep−c)min(B,Λt), (α K_t)-cN_t≤ (α L_t)+(c_rep-c)A_t≤ (α L_t)+(c_rep-c) (B, _t), and the right-hand side is the value of log(αKt)−cNt (α K_t)-cN_t for the attacker of (b). So whenever any such attacker makes a false statement at time t, the attacker of (b) does too, on the same uncorrupted sequence. For the grid certificate, the same computation applies to each bet. The attacker of (b) maximizes every Kt(k)K^(k)_t at once. For each k the computation gives wkKt(k)e−ckNt≤wkL~t(k)e(crep(λ¯k)−ck)min(B,Λt)w_kK^(k)_te^-c_kN_t≤ w_k L^(k)_te^(c_rep( λ_k)-c_k) (B, _t) because ck≤crep(λ¯k)c_k≤ c_rep( λ_k), with equality for the attacker of (b). Summing over k shows that Φt(Nt) _t(N_t) is largest under the attacker of (b). Since NtN_t is a nonnegative integer, Φt(Nt)≥1/α _t(N_t)≥ 1/α exactly when B^t≥Nt B_t≥ N_t. So the attacker of (b) again makes a false statement whenever any such attacker does. □ B.5 Proof of Theorem 5.3 The growth rate. With ℙ(W=1)=aP(W=1)=a, ℙ(W=0)=ℓP(W=0)= and ℙ(W=12)=1−a−ℓP(W= 12)=1-a- , g(λ)=alog(1+λ2)+ℓlog(1−λ2)=a+ℓ2log(1−λ24)+a−ℓ2log1+λ/21−λ/2=δcrep(λ)−π2κ(λ),g(λ)=a (1+ λ2)+ (1- λ2)= a+ 2 (1- λ^24)+ a- 2 1+λ/21-λ/2=δ\,c_rep(λ)- π2κ(λ), which is Eq. (11). The limit of Eq. (12). By the strong law of large numbers, t−1logKt(k)→g(λ¯k)t^-1 K^(k)_t→ g( λ_k) almost surely for each k. Two bounds sandwich B^t B_t. First, Φt(b)≥wkKt(k)e−bc(λ¯k) _t(b)≥ w_kK^(k)_te^-bc( λ_k), so B^t≥maxk⌊(logKt(k)+log(αwk))/c(λ¯k)⌋ B_t≥ _k ( K^(k)_t+ (α w_k))/c( λ_k) . Second, Φt(b)≤maxkKt(k)e−bc(λ¯k) _t(b)≤ _kK^(k)_te^-bc( λ_k) because ∑kwk≤1 _kw_k≤ 1, so B^t≤max−1,maxk⌊(logKt(k)+logα)/c(λ¯k)⌋ B_t≤ \-1, _k ( K^(k)_t+ α)/c( λ_k) \. Dividing by t, the lower bound converges to maxkg(λ¯k)/c(λ¯k) _kg( λ_k)/c( λ_k) and the upper bound to max0,maxkg(λ¯k)/c(λ¯k)=ρ \0, _kg( λ_k)/c( λ_k)\=ρ. Together with B^t≥−1 B_t≥-1, this gives B^t/t→ρ B_t/t→ρ. The bound of Eq. (13). Fix k with g(λ¯k)>0g( λ_k)>0 and let zk=log1αwk+bc(λ¯k)z_k= 1α w_k+b\,c( λ_k) and σk=inft:logKt(k)≥zk _k= \t: K^(k)_t≥ z_k\. At σk _k, Φσk(b)≥wkKσk(k)e−bc(λ¯k)≥1/α _ _k(b)≥ w_kK^(k)_ _ke^-bc( λ_k)≥ 1/α, so τb≤σk _b≤ _k. (For an integer b, Φt(b)≥1/α _t(b)≥ 1/α is equivalent to B^t≥b B_t≥ b.) The log-wealth logKt(k) K^(k)_t is a random walk with independent, bounded increments of mean g(λ¯k)>0g( λ_k)>0, bounded above by cins(λ¯k)c_ins( λ_k). By Wald’s identity at the bounded stopping time σk∧n _k n, g(λ¯k)(σk∧n)=logKσk∧n(k)≤zk+cins(λ¯k)g( λ_k)E( _k n)=E K^(k)_ _k n≤ z_k+c_ins( λ_k), since before σk _k the walk is below zkz_k and it crosses zkz_k by at most one increment. Letting n→∞n→∞ (monotone convergence) gives σk≤(zk+cins(λ¯k))/g(λ¯k)E\, _k≤(z_k+c_ins( λ_k))/g( λ_k). Minimizing over k gives Eq. (13). Corollary B.3 (Explicit constants). Under the conditions of Theorem 5.3, if δ≤13δ≤ 13 and the grid contains a bet λ¯k∈[3δ/2,3δ] λ_k∈[3δ/ 2,3δ], then for every integer b≥0b≥ 0, τb≤δ−2log1αwk+1.5(b+1)/δE\, _b≤δ^-2 1α w_k+1.5(b+1)/δ with the insertion cost and τb≤δ−2log1αwk+(4b+1.5)/δE\, _b≤δ^-2 1α w_k+(4b+1.5)/δ with the replacement cost. Explicit bounds, and proof of Corollary B.3. For λ∈(0,1]λ∈(0,1], using crep(λ)=2artanh(λ/2)c_rep(λ)=2\,artanh(λ/2), x≤artanh(x)≤x/(1−x2)x (x)≤ x/(1-x^2) and x/(1+x)≤log(1+x)≤x/(1+x)≤ (1+x)≤ x, λ≤crep(λ)≤λ1−λ2/4≤43λ,λ3≤λ/21+λ/2≤cins(λ)≤λ2,κ(λ)≤λ2/41−λ2/4≤λ23.λ≤ c_rep(λ)≤ λ1-λ^2/4≤ 43λ, λ3≤ λ/21+λ/2≤ c_ins(λ)≤ λ2, κ(λ)≤ λ^2/41-λ^2/4≤ λ^23. Using crep=2cins+κ≥2cinsc_rep=2c_ins+κ≥ 2c_ins, gcins≥2δ−π2⋅κcins≥2δ−πλ2,gcrep=δ−π2⋅κcrep≥δ−πλ6. gc_ins≥ 2δ- π2· κc_ins≥ 2δ- πλ2, gc_rep=δ- π2· κc_rep≥δ- πλ6. Also g(λ)≥δλ−π2⋅λ23≥λ(δ−λ/6)g(λ)≥δλ- π2· λ^23≥λ(δ-λ/6). If λ¯k∈[3δ/2,3δ] λ_k∈[3δ/ 2,3δ] and 3δ≤13δ≤ 1, then δ−λ¯k/6≥δ/2δ- λ_k/6≥δ/2, so g(λ¯k)≥λ¯kδ/2≥322δ2≥1.06δ2g( λ_k)≥ λ_kδ/2≥ 32 2δ^2≥ 1.06\,δ^2, while cins(λ¯k)≤1.5δc_ins( λ_k)≤ 1.5δ and crep(λ¯k)≤4δc_rep( λ_k)≤ 4δ. Substituting into Eq. (13) gives τb≤(log1αwk+1.5δb+1.5δ)/(1.06δ2)E\, _b≤( 1α w_k+1.5δ b+1.5δ)/(1.06δ^2) with the insertion cost and τb≤(log1αwk+4δb+1.5δ)/(1.06δ2)E\, _b≤( 1α w_k+4δ b+1.5δ)/(1.06δ^2) with the replacement cost, which are at most the stated bounds. □ B.6 Proof of Theorem 5.4 and of the optimal rate Let E=maxt≤nB^t≥bE=\ _t≤ n B_t≥ b\. By Definition 2.1 it is an event determined by the first n observed records, and independent auxiliary randomization in B^t B_t would change nothing below. The coupling bound of Eq. (14), insertion. Let ζ1,ζ2,… _1, _2,… be independent Bern(2δ)Bern(2δ) and X1′,X2′,…X _1,X _2,… independent Bern(12)Bern( 12), all independent. Define Yt=ζt+(1−ζt)Xt′Y_t= _t+(1- _t)X _t. Then Y1,Y2,…Y_1,Y_2,… are independent Bern(12+δ)Bern( 12+δ), since ℙ(Yt=1)=2δ+(1−2δ)/2P(Y_t=1)=2δ+(1-2δ)/2. Write ℙδP_δ and δE_δ for probability and expectation when the records are independent Bern(12+δ)Bern( 12+δ) and there is no attack. The case δ=0δ=0 is ℙ0P_0. Now define an attacker in P in a null world, that is, one in which H0ijH_0^ij holds. At position t, if ζt=1 _t=1 and fewer than ⌊b⌋ b insertions have been made, it inserts a win. Otherwise the next uncorrupted record appears, and we let its value be Xt′X _t. The uncorrupted records are then independent Bern(12)Bern( 12), because the positions at which they appear are determined by ζ, which is independent of X′X . So H0ijH_0^ij holds. The attacker decides before each record using only its own coins, so it is in P, and it never corrupts more than ⌊b⌋ b records. On the event ∑t≤nζt≤⌊b⌋\ _t≤ n _t≤ b \, the first n observed records coincide with Y1,…,YnY_1,…,Y_n. Therefore ℙδ(E)=ℙ((Y1,…,Yn)∈E)≤ℙ((W1,…,Wn)∈E)+ℙ(∑t≤nζt>⌊b⌋)≤α+ℙ(Bin(n,2δ)>b),P_δ(E)=P ((Y_1,…,Y_n)∈ E ) ((W_1,…,W_n)∈ E )+P ( _t≤ n _t> b )≤α+P (Bin(n,2δ)>b ), where the second inequality uses that on E we have B^t≥b≥Nt B_t≥ b≥ N_t for some t≤nt≤ n, which has probability at most α by validity, and that ∑t≤nζt∼Bin(n,2δ) _t≤ n _t (n,2δ) is an integer, so it exceeds ⌊b⌋ b exactly when it exceeds b. Replacement. Take ζt∼Bern(2δ) _t (2δ) and Xt′∼Bern(12)X _t ( 12) as before and Yt=Xt′+(1−Xt′)ζtY_t=X _t+(1-X _t) _t, again independent Bern(12+δ)Bern( 12+δ). The attacker observes each uncorrupted value Xt′X _t and, if Xt′=0X _t=0, ζt=1 _t=1 and fewer than ⌊b⌋ b replacements have been made, replaces it by a win. This attacker is in V. Without the limit of ⌊b⌋ b replacements, the attacker would make ∑t≤n(1−Xt′)ζt∼Bin(n,δ) _t≤ n(1-X _t) _t (n,δ) replacements among the first n records, and the same argument gives Eq. (14) with q=δq=δ. The information bound of Eq. (15). Let E0=maxt≤nB^t≥0E_0=\ _t≤ n B_t≥ 0\, an event determined by the first n records. With no attack, Nt=0N_t=0, so validity gives x0:=ℙ0(E0)≤αx_0:=P_0(E_0)≤α. Let x=ℙδ(E0)x=P_δ(E_0). By the data-processing inequality for the laws of the first n records under ℙδP_δ and ℙ0P_0, kl(x,x0)≤nKLδkl(x,x_0)≤ nKL_δ, where kl(p,r)=plogpr+(1−p)log1−p1−rkl(p,r)=p pr+(1-p) 1-p1-r. If x≥αx≥α, then kl(x,x0)≥kl(x,α)≥xlog1α−log2kl(x,x_0) (x,α)≥ x 1α- 2, which gives x≤(nKLδ+log2)/log(1/α)x≤(nKL_δ+ 2)/ (1/α). If x<αx<α, the bound holds because log2/log(1/α)≥α 2/ (1/α)≥α for all α∈(0,1)α∈(0,1) (as αlog(1/α)≤e−1<log2α (1/α)≤ e^-1< 2). Finally, KLδKL_δ is at most the χ2χ^2-divergence, χ2=δ21/2+δ21/2=4δ2χ^2= δ^21/2+ δ^21/2=4δ^2. Consequences. For integer b, Markov’s inequality gives ℙ(Bin(n,q)>b)=ℙ(Bin(n,q)≥b+1)≤nq/(b+1)P(Bin(n,q)>b)=P(Bin(n,q)≥ b+1)≤ nq/(b+1), so ℙδ(E)≥1−βP_δ(E)≥ 1-β forces nq/(b+1)≥1−α−βnq/(b+1)≥ 1-α-β. Since E⊆E0E E_0 for b≥0b≥ 0, Eq. (15) forces 4nδ2+log2≥(1−β)log(1/α)4nδ^2+ 2≥(1-β) (1/α). By Hoeffding’s inequality, ℙ(Bin(n,q)>(q+η)n)≤e−2nη2P(Bin(n,q)>(q+η)n)≤ e^-2nη^2, and B^n≥(q+η)n⊆maxt≤nB^t≥(q+η)n\ B_n≥(q+η)n\ \ _t≤ n B_t≥(q+η)n\. Hence Eq. (14) with b=(q+η)nb=(q+η)n gives ℙ(B^n≥(q+η)n)≤α+e−2nη2P( B_n≥(q+η)n)≤α+e^-2nη^2. Corollary B.4 (Lower bound on the mean time to certify). Under the conditions of Theorem 5.4, for every integer b≥0b≥ 0, τb≥maxlog(1/α)KLδ,supn≥1n(1−α−ℙ(Bin(n,q)>b)).E\, _b\ ≥\ \ (1/α)KL_δ,\ _n≥ 1\,n (1-α-P(Bin(n,q)>b) ) \. Proof. The second term follows from δτb≥nℙδ(τb>n)E_δ\, _b≥ nP_δ( _b>n) and Eq. (14). For the first, τb≥τ0 _b≥ _0 because b≥0b≥ 0, and we may assume δτ0<∞E_δ\, _0<∞. With no attack, τ0 _0 is the rejection time of a test of Bern(12)Bern( 12) with ℙ0(τ0<∞)≤αP_0( _0<∞)≤α. For each n, τ0∧n _0 n is a bounded stopping time, so Wald’s identity for the log-likelihood ratio and the data-processing inequality applied to the event τ0≤n∈ℱτ0∧n\ _0≤ n\ _ _0 n give KLδδ(τ0∧n)≥kl(ℙδ(τ0≤n),ℙ0(τ0≤n))KL_δ\,E_δ( _0 n) (P_δ( _0≤ n),P_0( _0≤ n) ) (Kaufmann et al.,, 2016, Lemma 1). As n→∞n→∞, ℙδ(τ0≤n)→1P_δ( _0≤ n)→ 1 and ℙ0(τ0≤n)≤αP_0( _0≤ n)≤α, so the limit inferior of the right-hand side is at least kl(1,α)=log(1/α)kl(1,α)= (1/α) (using that kl(p,r)kl(p,r) is decreasing in r≤pr≤ p), and the left-hand side increases to KLδδτ0KL_δ\,E_δ\, _0. ∎ Optimal rate. With π=1π=1 and the smallest bet λ¯m≤1 λ_m≤ 1, Theorem 5.3 gives ρ≥g(λ¯m)/cins(λ¯m)≥2δ−λ¯m/2ρ≥ g( λ_m)/c_ins( λ_m)≥ 2δ- λ_m/2 for the insertion cost and ρ≥δ−λ¯m/6ρ≥δ- λ_m/6 for the replacement cost. The upper limits ρ≤qρ≤ q follow from Theorem 5.4, because the tolerance is valid (Theorem 5.1). Indeed, if ρ>qρ>q, take η∈(0,ρ−q)η∈(0,ρ-q) and b=⌈(q+η)n⌉b= (q+η)n . Since B^n/n→ρ B_n/n→ρ almost surely, the left-hand side of Eq. (14) is at least ℙ(B^n≥b)→1P( B_n≥ b)→ 1. By Hoeffding’s inequality its right-hand side is at most α+e−2nη2→α<1α+e^-2nη^2→α<1, a contradiction. The coupling bound of Eq. (14) also holds with ties, and with it the limits ρ≤qρ≤ q, when W is 11, 00 or 12 12 with probabilities π/2+δπ/2+δ, π/2−δπ/2-δ and 1−π1-π. For insertion, let Xt′X _t have the null law that is 11 or 00 with probability (π/2−δ)/(1−2δ)(π/2-δ)/(1-2δ) each and 12 12 otherwise, and keep ζt∼Bern(2δ) _t (2δ). Then ζt+(1−ζt)Xt′ _t+(1- _t)X _t has the law of W. For replacement, let Xt′X _t be 11 or 00 with probability π/2π/2 each and 12 12 otherwise, and let the attacker replace a loss by a win when ζt=1 _t=1, now with ζt∼Bern(2δ/π) _t (2δ/π). Without the cap of ⌊b⌋ b replacements their number is Bin(n,δ)Bin(n,δ), and the observed law is that of W. For the insertion cost this can also be checked directly, since g/cins≤2δg/c_ins≤ 2δ is equivalent to δκ≤πκ/2δκ≤πκ/2, which holds because ℓ≥0 ≥ 0. □ B.7 Proof of Proposition 5.5 Validity. Let N be the set of null variants. For v∈v , let K(v,k)K^(v,k) be the wealth of variant v with bet k and Nt(v)N^(v)_t the number of corrupted records of variant v. Let L(v,k)L^(v,k) be the uncorrupted wealth from the proof of Theorem 5.1(a) or (b), with factor 11 on records of other variants, so that Kt(v,k)≤ec(λ¯k)Nt(v)Lt(v,k)K^(v,k)_t≤ e^c( λ_k)N^(v)_tL^(v,k)_t. The variant of each record is fixed before its value is produced, and the null of v holds given all records. So each L(v,k)L^(v,k) is a nonnegative supermartingale for the global filtration, that of all records of all variants. A statement about the published variant is false at time t only if v^t∈ v_t and ∑kwkKt(v^t,k)e−c(λ¯k)Nt(v^t)≥V/α _kw_kK^( v_t,k)_te^-c( λ_k)N^( v_t)_t≥ V/α (as in Lemma B.1), which implies L¯t:=1V∑v∈∑kwkLt(v,k)≥1V∑kwkLt(v^t,k)≥1/α L_t:= 1V _v _kw_kL^(v,k)_t≥ 1V _kw_kL^( v_t,k)_t≥ 1/α. The process L¯ L is a nonnegative supermartingale with L¯0≤||/V≤1 L_0≤|N|/V≤ 1, so Lemma B.2 applies. The choice of v^t v_t plays no role. Adaptive creation. The variants need not exist from the start. Fix V in advance and let the developer create and retire variants as results come in, with at most V of them ever receiving a record, indexed by order of creation. Let ArA_r be the event that the rrth created variant is one of the null variants in N. Its identity is known at creation, before its first record, so ArA_r and, at each of its records, the variant of the record are fixed before the value is produced. Define L(r,k)L^(r,k) as the uncorrupted wealth of the rrth created variant on ArA_r and as the constant 11 on the complement; it equals 11 until the variant receives its first record. On ArA_r the null of that variant holds given the records of all variants, so each factor of L(r,k)L^(r,k) has conditional expectation at most 11 given the past. Each L(r,k)L^(r,k) is then a nonnegative supermartingale for the global filtration, L¯t=1V∑r≤V∑kwkLt(r,k) L_t= 1V _r≤ V _kw_kL^(r,k)_t has L¯0≤1 L_0≤ 1, and the rest of the argument is unchanged. The bound V must be fixed in advance. Necessity. Let K be the wealth of one variant with constant bet λ and independent Bern(12)Bern( 12) records, and y>0y>0. Then K is a martingale, and logKt K_t is a random walk with mean increment 12log(1−λ2/4)<0 12 (1-λ^2/4)<0, so Kt→0K_t→ 0 almost surely. Let σ=inft≥1:Kt≥yσ= \t≥ 1:K_t≥ y\ and y¯=max(y,1) y= (y,1). Since Kt∧σ≤y¯M+K_t σ≤ yM_+, bounded convergence gives 1=[Kσ<∞]≤y¯M+ℙ(σ<∞)1=E[K_σ1\σ<∞\]≤ yM_+\,P(σ<∞), so ℙ(supt≥1Kt≥y)≥1/(y¯M+)P( _t≥ 1K_t≥ y)≥ 1/( yM_+). Take y=V′/αy=V /α. If V′<αV <α, then y¯=1 y=1 and a single variant already reaches the threshold with probability at least 1/M+>12≥α1/M_+> 12≥α, so assume V′≥αV ≥α. Publishing the variant whose wealth first reaches V′/αV /α makes a false statement if some variant reaches it. With V independent variants this has probability at least 1−(1−α/(V′M+))V≥1−e−Vα/(V′M+)1-(1-α/(V M_+))^V≥ 1-e^-Vα/(V M_+). If V′≤V/(2M+)V ≤ V/(2M_+), this is at least 1−e−2α>α1-e^-2α>α for α≤12α≤ 12. □ B.8 Proof of Proposition 5.6 Let the time t run over all records on all pairs, and let (t)(G_t) be the global filtration. For each ordered pair (i,j)(i,j) and bet k, let K(ij,k)K^(ij,k) be the observed wealth of bet k on the records of (i,j)(i,j). Define the uncorrupted wealth L(ij,k)L^(ij,k) as in the proof of Theorem 5.1(a) or (b), with factor 11 on records that do not concern (i,j)(i,j). By the two assumptions of the proposition, each is a nonnegative supermartingale for (t)(G_t) when H0ijH_0^ij holds. As in the proof of Theorem 5.1, Kt(ij,k)≤ec(λ¯k)NtijLt(ij,k)≤ec(λ¯k)NtLt(ij,k)K^(ij,k)_t≤ e^c( λ_k)N^ij_tL^(ij,k)_t≤ e^c( λ_k)N_tL^(ij,k)_t pathwise, where Ntij≤NtN^ij_t≤ N_t counts corrupted records on (i,j)(i,j). The set S∗S is a property of the law of the data, not random. If S∗=∅S = , no joint claim is false. Otherwise, suppose a joint claim D is false at time t and B^t(D)≥Nt B_t(D)≥ N_t. Then S∗∩D≠∅S ∩ D≠ and S∗∈S , so |S∗|−1∑(i,j)∈S∗Φtij(B^t(D))≥1/α|S |^-1 _(i,j)∈ S ^ij_t( B_t(D))≥ 1/α. Each Φtij ^ij_t is decreasing, so the same holds with NtN_t in place of B^t(D) B_t(D), and therefore L¯t:=|S∗|−1∑(i,j)∈S∗∑kwkLt(ij,k)≥1/α L_t:=|S |^-1 _(i,j)∈ S _kw_kL^(ij,k)_t≥ 1/α. The process L¯ L is a nonnegative supermartingale for (t)(G_t) with L¯0≤1 L_0≤ 1, and it does not depend on D. Lemma B.2 bounds the probability that it ever reaches 1/α1/α by α. □ On a static benchmark the records do not arrive one pair at a time. Each item scores every pair at once, and the records of one item on two pairs that share a model are dependent. The following corollary covers this case. It conditions only on earlier items, so it does not order the pairs within an item, and it counts a corrupted item once however many pairs it changes. Let ℋt−1H_t-1 be the σ-field generated by the attacker’s randomization, by all variables of the first t−1t-1 items (observed outcomes, indicators and uncorrupted values, as for t−1G_t-1) and by ItI_t. Corollary B.5 (Simultaneous certificates with item-level records). Suppose that at each time t one item arrives and produces a record WtijW^ij_t, with uncorrupted value XtijX^ij_t, for every ordered pair (i,j)(i,j) of a set of models fixed in advance. Let ItI_t indicate that the item is inserted, let Ct=1C_t=1 if it is inserted or any of its records is replaced, and let Nt=∑s≤tCsN_t= _s≤ tC_s count the corrupted items. Suppose that [Xtij∣ℋt−1]≤12E[X^ij_t _t-1]≤ 12 on It=0\I_t=0\ for every pair (i,j)∈S∗(i,j)∈ S . Then the conclusion of Proposition 5.6 holds with this NtN_t, with the cost cinsc_ins if every CtC_t is ℋt−1H_t-1-measurable and with the cost crepc_rep in general. Proof. Fix a bet k and a pair (i,j)∈S∗(i,j)∈ S . Define the uncorrupted wealth L(ij,k)L^(ij,k) from the records of (i,j)(i,j) as in the proof of Theorem 5.1(a) for the cost cinsc_ins and (b) for the cost crepc_rep, with the item-level variables CtC_t and ItI_t. The variables ItI_t and λk,t _k,t are ℋt−1H_t-1-measurable, and so is CtC_t in the first case, and the null of (i,j)(i,j) holds given ℋt−1H_t-1. The computation in that proof therefore shows that L(ij,k)L^(ij,k) is a nonnegative supermartingale for (ℋt)(H_t), and this holds for every pair in S∗S with respect to the same filtration. Pathwise, at an item with Ct=0C_t=0 the factors of K(ij,k)K^(ij,k) and L(ij,k)L^(ij,k) agree for every pair. At an item with Ct=1C_t=1, the factor of K(ij,k)K^(ij,k) is at most ec(λ¯k)e^c( λ_k) times that of L(ij,k)L^(ij,k). With the cost cinsc_ins this holds because L(ij,k)L^(ij,k) skips the whole item, so its factor is 11, while every factor of K(ij,k)K^(ij,k) is at most 1+λ¯k/21+ λ_k/2. With the cost crepc_rep it is the factor comparison in the proof of Theorem 5.1(b), and the two factors agree when the record of (i,j)(i,j) itself is uncorrupted. Hence Kt(ij,k)≤ec(λ¯k)NtLt(ij,k)K^(ij,k)_t≤ e^c( λ_k)N_tL^(ij,k)_t for every pair in S∗S , with the same NtN_t. The rest of the proof of Proposition 5.6 applies without change, with (ℋt)(H_t) in place of (t)(G_t). ∎ Offline attackers. Let the uncorrupted items form a sequence of vectors u=(Xuij)(i,j)X_u=(X^ij_u)_(i,j), u≥1u≥ 1, whose law does not depend on the attacker, and suppose that [Xuij∣1,…,u−1]≤12E[X^ij_u _1,…,X_u-1]≤ 12 for every u and every (i,j)∈S∗(i,j)∈ S . With constant bets and the cost crepc_rep, the conclusion of Proposition 5.6 then also holds against attackers in O, who may inspect the whole uncorrupted sequence before inserting items or replacing records, with NtN_t counting the corrupted items. For the proof, let L~u(ij,k)=∏u′≤u(1+λ¯k(Xu′ij−12)) L^(ij,k)_u= _u ≤ u (1+ λ_k(X^ij_u - 12) ), a nonnegative supermartingale for the filtration of the uncorrupted vectors, for every pair in S∗S at once, and let h(t)h(t) be the number of non-inserted items among the first t observed items. As in the proof of Theorem 5.1(c), an inserted item contributes a factor at most 1+λ¯k/2≤ecrep(λ¯k)1+ λ_k/2≤ e^c_rep( λ_k) to Kt(ij,k)K^(ij,k)_t and none to L~h(t)(ij,k) L^(ij,k)_h(t). A non-inserted item contributes at most ecrep(λ¯k)e^c_rep( λ_k) times its uncorrupted factor if the record of (i,j)(i,j) was replaced, and that factor otherwise. Hence Kt(ij,k)≤ecrep(λ¯k)NtL~h(t)(ij,k)K^(ij,k)_t≤ e^c_rep( λ_k)N_t L^(ij,k)_h(t) pathwise for every pair in S∗S . The argument of Proposition 5.6 then shows that a false joint claim D with B^t(D)≥Nt B_t(D)≥ N_t forces supu|S∗|−1∑(i,j)∈S∗∑kwkL~u(ij,k)≥1/α _u|S |^-1 _(i,j)∈ S _kw_k L^(ij,k)_u≥ 1/α, an event of the uncorrupted sequence alone, whose probability Lemma B.2 bounds by α. The same holds for Proposition 5.6 itself against O with constant bets. It needs the uncorrupted records, each with its pair, to form a sequence whose law does not depend on the attacker, with each true null holding given the earlier uncorrupted records and the pair of the current one. Computation. Let S be the family of all sets of pairs, as in E3c and E8. For a pair and a budget b, the smallest average over the sets containing the pair is then attained by adding the other pairs in increasing order of their deflated averages. One check therefore sorts the pairs, and the tolerance follows by bisection over b. B.9 Proof of Theorem 5.7 and Corollary B.7 Lemma B.6 (Hoeffding’s lemma). Let Y take values in [0,1][0,1], let ℋH be a σ-field and let γ>0γ>0. Then [eγ(Y−[Y∣ℋ])|ℋ]≤eγ2/8E [e^γ(Y-E[Y ])\, |\,H ]≤ e^γ^2/8. This is classical (Hoeffding,, 1963), applied under the conditional law given ℋH. The betting factor 1+λ(Ws−12)1+λ(W_s- 12) is not linear in the outcome, so on Δt≤0\ _t≤ 0\ the betting wealth is not dominated pathwise by a supermartingale of the uncorrupted values. The exponential wealth is linear in the exponent, which the proofs below use. Throughout, μs=[Xs∣s−1] _s=E[X_s _s-1], which equals 12 12 on Is=1\I_s=1\ by the convention of Section B.1. In this subsection Zt(k)=Zt(γk)Z^(k)_t=Z^( _k)_t and ck=cZ(γk)c_k=c^Z( _k). Reduction on the null event. Lemma B.1 uses domination at every time. Here it is needed only at times t with Δt≤0 _t≤ 0, when the lead claim is false. If Zt(k)≤eckNtLt(k)Z^(k)_t≤ e^c_kN_tL^(k)_t for all k on an event tA_t, then t∩B^tZ≥Nt⊆∑kwkLt(k)≥1/αA_t∩\ B^Z_t≥ N_t\ \ _kw_kL^(k)_t≥ 1/α\. The proof is that of Lemma B.1, since on B^tZ≥Nt\ B^Z_t≥ N_t\ the number NtN_t is a nonnegative integer. (a) Insertion-only attackers in P, cost γ/2−γ2/8γ/2-γ^2/8. Fix γ=γkγ= _k and write ZtZ_t for Zt(γ)Z^(γ)_t. Every corrupted record is inserted, so Cs=IsC_s=I_s, and IsI_s is s−1G_s-1-measurable. Define Lt=exp(γ∑s≤t(1−Is)(Xs−μs)−γ28∑s≤t(1−Is)).L_t= (γ _s≤ t(1-I_s)(X_s- _s)- γ^28 _s≤ t(1-I_s) ). On It=1\I_t=1\, Lt=Lt−1L_t=L_t-1. On It=0\I_t=0\, Lemma B.6 with ℋ=t−1H=G_t-1 gives [Lt∣t−1]≤Lt−1E[L_t _t-1]≤ L_t-1. Hence L is a nonnegative supermartingale with L0=1L_0=1. Pathwise, an uncorrupted record contributes γ(Xs−12)−γ2/8=γ(Xs−μs)−γ2/8+γ(μs−12)γ(X_s- 12)-γ^2/8=γ(X_s- _s)-γ^2/8+γ( _s- 12) to logZt Z_t, and an inserted record at most γ/2−γ2/8γ/2-γ^2/8. Summing, logZt≤logLt+γΔt+(γ2−γ28)Nt. Z_t≤ L_t+γ _t+ ( γ2- γ^28 )N_t. On Δt≤0\ _t≤ 0\ therefore Zt(k)≤e(γk/2−γk2/8)NtLt(k)Z^(k)_t≤ e^( _k/2- _k^2/8)N_tL^(k)_t for every k. By the reduction above, Δt≤0,B^tZ≥Nt⊆∑kwkLt(k)≥1/α\ _t≤ 0,\ B^Z_t≥ N_t\ \ _kw_kL^(k)_t≥ 1/α\ for every t, and Lemma B.2 bounds the probability of the union over t by α. For every attacker in P, CsC_s is s−1G_s-1-measurable. The same computation with CsC_s in place of IsI_s shows that the cost γ/2−γ2/8γ/2-γ^2/8 is then valid against all of P if Δt _t is replaced by the margin over the uncorrupted records, ∑s≤t(1−Cs)(μs−12) _s≤ t(1-C_s)( _s- 12). (b) Class V, cost γ. Define LtL_t as in (a). Since ItI_t is t−1G_t-1-measurable in every class, L is again a nonnegative supermartingale. Pathwise, a non-inserted record contributes γ(Ws−12)−γ2/8γ(W_s- 12)-γ^2/8 to logZt Z_t and γ(Xs−μs)−γ2/8γ(X_s- _s)-γ^2/8 to logLt L_t. The difference is γ(Ws−Xs)+γ(μs−12)≤γCs+γ(μs−12)γ(W_s-X_s)+γ( _s- 12)≤γ C_s+γ( _s- 12), since Ws=XsW_s=X_s when Cs=0C_s=0 and |Ws−Xs|≤1|W_s-X_s|≤ 1 otherwise. An inserted record contributes at most γ/2−γ2/8≤γ/2-γ^2/8≤γ to logZt Z_t and nothing to logLt L_t. Summing, logZt≤logLt+γΔt+γNt Z_t≤ L_t+γ _t+γ N_t, and we conclude as in (a). Statement of (c). Let μ~u=[Xu∣X1,…,Xu−1] μ_u=E[X_u X_1,…,X_u-1], Δ~u=∑u′≤u(μ~u′−12) _u= _u ≤ u( μ_u - 12) and h(t)h(t) be the number of non-inserted records among the first t. Part (c) states that, against every attacker in O and with the cost γ, ℙ(∃t:Δ~h(t)≤0andB^tZ≥Nt)≤αP (∃\,t:\ _h(t)≤ 0\ and\ B^Z_t≥ N_t )≤α. (c) Class O, cost γ. Let L~u=exp(γ∑u′≤u(Xu′−μ~u′)−γ2u/8) L_u= (γ _u ≤ u(X_u - μ_u )-γ^2u/8 ), a nonnegative supermartingale for the natural filtration of (Xu)(X_u) by Lemma B.6, whose law does not depend on the attacker. The term of an uncorrupted value in logZt Z_t depends on that value alone, so its position in the observed sequence does not matter. As in (b), logZt≤logL~h(t)+γΔ~h(t)+γNt Z_t≤ L_h(t)+γ _h(t)+γ N_t pathwise. Hence ∃t:Δ~h(t)≤0,B^tZ≥Nt⊆∃u:∑kwkL~u(k)≥1/α\∃ t: _h(t)≤ 0,\ B^Z_t≥ N_t\ \∃ u: _kw_k L^(k)_u≥ 1/α\, an event of the uncorrupted sequence alone, of probability at most α by Lemma B.2. As in Theorem 5.1(c), the constants γk _k must not depend on the observed records. (d) Validity in the sense of Eq. (3). Under H0ijH_0^ij, μs≤12 _s≤ 12 on Is=0\I_s=0\ for every s, so Δt≤0 _t≤ 0 for every t, and the event of Eq. (3) is that of Eq. (4). Under the null of O, Δ~u≤0 _u≤ 0 for every u likewise. The rate. By the strong law of large numbers, t−1logZt(k)→γkδ−γk2/8t^-1 Z^(k)_t→ _kδ- _k^2/8 almost surely, since [W−12]=δE[W- 12]=δ whether or not there are ties. The sandwich in the proof of Theorem 5.3 applies with g(λ¯k)g( λ_k) replaced by gZ(γk)=γkδ−γk2/8g^Z( _k)= _kδ- _k^2/8 and c(λ¯k)c( λ_k) by cZ(γk)c^Z( _k), which gives the limit ρZρ^Z. The two bounds follow at k=mk=m from gZ(γ)/(γ/2−γ2/8)≥gZ(γ)/(γ/2)=2δ−γ/4g^Z(γ)/(γ/2-γ^2/8)≥ g^Z(γ)/(γ/2)=2δ-γ/4 when gZ(γ)≥0g^Z(γ)≥ 0, from gZ(γ)/γ=δ−γ/8g^Z(γ)/γ=δ-γ/8, and from the fact that both right-hand sides are negative when gZ(γm)<0g^Z( _m)<0. The limits ρZ≤2δρ^Z≤ 2δ and ρZ≤δρ^Z≤δ follow from the coupling bound of Eq. (14), which holds with ties (Section B.6). That bound is proved with an insertion-only attacker in the case of P and with an attacker in V in the case of V, and B^tZ B^Z_t is valid against both by (d). Sharpness of the insertion cost and a lower bound for the replacement cost. B forged wins inserted before the first uncorrupted record give logZB=B(γ/2−γ2/8) Z_B=B(γ/2-γ^2/8) and NB=BN_B=B. So with any cost c<γ/2−γ2/8c<γ/2-γ^2/8 the lead tolerance makes a false statement with probability one once B≥log(1/α)/(γ/2−γ2/8−c)B≥ (1/α)/(γ/2-γ^2/8-c), as in Theorem 5.2(a). Replacing the first B uncorrupted losses by wins raises logZ Z by exactly γ per flip, and the argument of Theorem 5.2(b) shows that every cost below γ−γ2/4=2cinsZ(γ)γ-γ^2/4=2c_ins^Z(γ) then fails with probability tending to one. The cost γ is the largest increase of logZt Z_t that one flip can cause, and it is valid by (b). This argument does not show that every cost between γ−γ2/4γ-γ^2/4 and γ fails. Why (a) excludes predictable replacement. Under H0ijH_0^ij a record overwritten unseen is a null record anyway, which is why Theorem 5.1(a) covers it. For the lead claim its hidden uncorrupted value counts in Δt _t, and an attacker that overwrites the records it expects to be weak can hold Δt _t at zero while the wealth grows. The lead certificate therefore charges γ/2−γ2/8γ/2-γ^2/8 for a forged record and γ for an altered one. Fix 0<ξ<(4−γ1)/80<ξ<(4- _1)/8 and let the uncorrupted values be independent with μs=1−ξ _s=1-ξ at odd s and μs=ξ _s=ξ at even s, so that Δt=0 _t=0 at every even t. A predictable attacker overwrites every even record by a win without looking at it. Then Nt=t/2N_t=t/2 at even t, and by the strong law t−1∑s≤t(Ws−12)→12−ξ2t^-1 _s≤ t(W_s- 12)→ 12- ξ2 almost surely, so t−1(logZt(γ)−(γ/2−γ2/8)Nt)→γ16(4−8ξ−γ)t^-1 ( Z^(γ)_t-(γ/2-γ^2/8)N_t )→ γ16(4-8ξ-γ), which is positive for every γ≤γ1γ≤ _1. Every bet of the grid thus has Zt(k)e−(γk/2−γk2/8)Nt→∞Z^(k)_te^-( _k/2- _k^2/8)N_t→∞, and B^tZ≥Nt B^Z_t≥ N_t at all large even t almost surely while Δt=0 _t=0. The margin over the uncorrupted records is (12−ξ)⌈t/2⌉>0( 12-ξ) t/2 >0, so the variant at the end of (a) is not contradicted. Corollary B.7 (Simultaneous lead certificates). Let the pair of each record be fixed before its value is produced and let ℳM be a set of M ordered pairs fixed in advance. Let Δtij ^ij_t be the uncorrupted margin of Eq. (2) of pair (i,j)(i,j), let B^tZ,ij B^Z,ij_t be its lead tolerance computed from its records with M/αM/α in place of 1/α1/α, and let NtN_t count the corrupted records on all pairs. Against the attackers and costs of Theorem 5.7(a,b), ℙ(∃t,∃(i,j)∈ℳ:Δtij≤0andB^tZ,ij≥Nt)≤αP (∃\,t,\ ∃(i,j) :\ ^ij_t≤ 0\ and\ B^Z,ij_t≥ N_t )≤α. The same holds for V private variants of one model with V/αV/α. The set S∗S of true nulls in Proposition 5.6 is fixed, while the pairs with Δtij≤0 ^ij_t≤ 0 change with t, so closed testing does not transfer. A Bonferroni correction does, at a price of logM M nats per pair, and in E3c closed testing gained at most one record over it. Proof of Corollary B.7. Let t run over the records of all pairs with the global filtration (t)(G_t), as in the proof of Proposition 5.6. For a pair (i,j)(i,j) define L(ij,k)L^(ij,k) as in (a) or (b), with factor 11 on the records of other pairs. Since the pair of record t is t−1G_t-1-measurable, the computations of (a) and (b) show that L(ij,k)L^(ij,k) is a nonnegative supermartingale and that, on Δtij≤0\ ^ij_t≤ 0\, Zt(ij,k)≤eckNtijLt(ij,k)≤eckNtLt(ij,k)Z^(ij,k)_t≤ e^c_kN^ij_tL^(ij,k)_t≤ e^c_kN_tL^(ij,k)_t, where Ntij≤NtN^ij_t≤ N_t counts the corrupted records of the pair. By the reduction above with M/αM/α in place of 1/α1/α, Δtij≤0,B^tZ,ij≥Nt⊆∑kwkLt(ij,k)≥M/α\ ^ij_t≤ 0,\ B^Z,ij_t≥ N_t\ \ _kw_kL^(ij,k)_t≥ M/α\ for every t, and Lemma B.2 bounds the probability of the union over t by α/Mα/M. A union bound over the M pairs gives α. For V private variants of one model against j, the variants play the role of the pairs. □ B.10 A deterministic tolerance for a completed set of records The certificates above are statements about the normal process that produced the records. A completed set of records, such as a released benchmark or the votes of a closed period, also supports a statement about itself, with no probability and no sampling assumption. Proposition B.8 (Deterministic tolerance of a completed set). Let n records be observed with values W1,…,Wn∈[0,1]W_1,…,W_n∈[0,1] and let Δ=∑s≤n(2Ws−1)>0 = _s≤ n(2W_s-1)>0, not the margin of Eq. (2). If at most B≤⌈Δ⌉−1B≤ -1 of the records were inserted and none replaced, the mean of the uncorrupted values among them exceeds 12 12. If at most B≤⌈Δ/2⌉−1B≤ /2 -1 were replaced and none inserted, the mean of the uncorrupted values of all n records exceeds 12 12. For records in 0,12,1\0, 12,1\, Δ is the number of wins minus losses and neither bound can be raised. If Δ of the observed wins were inserted, the uncorrupted records have mean 12 12 when any remain, and none remain when Δ=n =n. If ⌈Δ/2⌉ /2 of them replaced losses, the mean of the uncorrupted values is at most 12 12. Proof. An inserted record contributes at most 11 to Δ , so the sum of 2Ws−12W_s-1 over the uncorrupted records is at least Δ−B -B, which is positive when B<ΔB< , that is, when B≤⌈Δ⌉−1B≤ -1. A replacement changes 2Ws−12W_s-1 by at most 22, so the same sum taken over the uncorrupted values of all n records is at least Δ−2B -2B, which is positive when B<Δ/2B< /2, that is, when B≤⌈Δ/2⌉−1B≤ /2 -1. For the last claim, the observed set has Δ more wins than losses, so at least Δ wins. If Δ of them were inserted, the uncorrupted records have as many wins as losses, so their mean is 12 12 when any remain (none remain when Δ=n =n). If ⌈Δ/2⌉ /2 of them replaced uncorrupted losses, the uncorrupted values have Δ−2⌈Δ/2⌉≤0 -2 /2 ≤ 0 more wins than losses, and their mean is at most 12 12. ∎ For records in 0,12,1\0, 12,1\ a completed set thus withstands exactly Δ−1 -1 forged records and ⌈Δ/2⌉−1 /2 -1 replaced ones. If a contaminated item can change a record by at most 12 12, as when only one model’s answer is altered, the guaranteed count for replaced items rises to ⌈Δ⌉−1 -1. For general values in [0,1][0,1] a given observed set need not attain the bounds. The certificates of Eq. (9) and Eq. (10) speak about the normal process. For records in 0,12,1\0, 12,1\ they are at most Δ−1 -1 and ⌈Δ/2⌉−1 /2 -1 as well. Indeed logKn(k)≤Δcins(λ¯k) K^(k)_n≤ \,c_ins( λ_k) because log(1−λ¯k/2)≤−log(1+λ¯k/2) (1- λ_k/2)≤- (1+ λ_k/2), and logZn(γ)≤ΔcinsZ(γ) Z^(γ)_n≤ \,c_ins^Z(γ) because n≥Δn≥ . Each replacement cost is at least twice the insertion cost. So the deflated average is at most 1<1/α1<1/α at b=Δb= with the insertion cost and at b=⌈Δ/2⌉b= /2 with the replacement cost. E3 and E7 report the gap. For general values in [0,1][0,1] a certificate and the deterministic bound concern different statements and need not be ordered. B.11 Attacks on correct claims An attacker may also work against a correct claim, by inserting losses for the leader or by replacing its wins with losses, as in E6. Let dins(λ¯)=−log(1−λ¯/2)=crep(λ¯)−cins(λ¯)d_ ins( λ)=- (1- λ/2)=c_rep( λ)-c_ins( λ), the most that one inserted record can lower the log-wealth of a constant bet λ¯ λ. Proposition B.9 (Price of attacking a correct claim). Consider the tolerance of Eq. (9) with constant bets and an attacker in V that corrupts B of the first t records. Let Lt(k)L^(k)_t be the uncorrupted wealth of bet k, as in the proof of Theorem 5.1, and let B^t(k)(α′)=⌊log(α′Lt(k))/c(λ¯k)⌋ B^(k)_t(α )= (α L^(k)_t)/c( λ_k) be its single-bet tolerance at level α′α on the uncorrupted values. (a) Pathwise, logKt(k)≥logLt(k)−Bdk K^(k)_t≥ L^(k)_t-Bd_k, where dk=dins(λ¯k)d_k=d_ ins( λ_k) if every corrupted record is inserted and dk=crep(λ¯k)d_k=c_rep( λ_k) otherwise. Hence B^t≥B^t(k)(αwk)−⌈Bdk/c(λ¯k)⌉ B_t≥ B^(k)_t(α w_k)- Bd_k/c( λ_k) for every k for which the right-hand side is nonnegative. (b) With the cost crepc_rep, replacing B wins by losses lowers the tolerance by exactly B: B^t=B^tunc−B B_t= B unc_t-B if B^tunc≥B B unc_t≥ B and B^t=−1 B_t=-1 otherwise, where B^tunc B unc_t is the tolerance of the uncorrupted values. (c) Suppose that the corrupted records lie among the first t0t_0 records, that the later uncorrupted values are independent of the first t0t_0 records and i.i.d. as in Theorem 5.3 with g(λ¯k)>0g( λ_k)>0, and that logLt0(k)≥log1αwk+bc(λ¯k) L^(k)_t_0≥ 1α w_k+b\,c( λ_k) for an integer b≥0b≥ 0. Then the number of records after t0t_0 until B^t≥b B_t≥ b has expectation at most (Bdk+cins(λ¯k))/g(λ¯k) (Bd_k+c_ins( λ_k) )/g( λ_k). (a) The factor of a record lies in [m−,M+][m_-,M_+] with m−=1−λ¯k/2m_-=1- λ_k/2 and M+=1+λ¯k/2M_+=1+ λ_k/2. An inserted record contributes a factor of at least m−=e−dins(λ¯k)m_-=e^-d_ ins( λ_k) to Kt(k)K^(k)_t and none to Lt(k)L^(k)_t. A replaced record changes its factor by a ratio of at least m−/M+=e−crep(λ¯k)m_-/M_+=e^-c_rep( λ_k), and dins≤crepd_ ins≤ c_rep. Multiplying over the B corrupted records gives the first claim. For the second, Φt(b)≥wkKt(k)e−bc(λ¯k)≥wkLt(k)e−Bdk−bc(λ¯k) _t(b)≥ w_kK^(k)_te^-bc( λ_k)≥ w_kL^(k)_te^-Bd_k-bc( λ_k), which is at least 1/α1/α at b=⌊(log(αwkLt(k))−Bdk)/c(λ¯k)⌋b= ( (α w_kL^(k)_t)-Bd_k)/c( λ_k) , and ⌊x−y⌋≥⌊x⌋−⌈y⌉ x-y ≥ x - y . (b) A win replaced by a loss changes the factor of every bet k from M+M_+ to m−m_-, a ratio of e−crep(λ¯k)=e−cke^-c_rep( λ_k)=e^-c_k. So Kt(k)=Lt(k)e−BckK^(k)_t=L^(k)_te^-Bc_k for every k, hence Φt(b)=Φtunc(b+B) _t(b)= unc_t(b+B) with Φunc unc the deflated average of the uncorrupted values, and the claim follows from Eq. (9). (c) Write z=log1αwk+bc(λ¯k)z= 1α w_k+b\,c( λ_k). If logKt0(k)≥z K^(k)_t_0≥ z, then Φt0(b)≥wkKt0(k)e−bc(λ¯k)≥1/α _t_0(b)≥ w_kK^(k)_t_0e^-bc( λ_k)≥ 1/α, so B^t0≥b B_t_0≥ b and there is nothing to prove. Otherwise let σ=inft>t0:logKt(k)≥zσ= \t>t_0: K^(k)_t≥ z\. At σ, Φσ(b)≥wkKσ(k)e−bc(λ¯k)≥1/α _σ(b)≥ w_kK^(k)_σe^-bc( λ_k)≥ 1/α, so B^σ≥b B_σ≥ b. After t0t_0, logKt(k) K^(k)_t grows by independent increments with mean g(λ¯k)>0g( λ_k)>0 that are bounded above by cins(λ¯k)c_ins( λ_k). By (a) and the assumption, z−logKt0(k)≤Bdkz- K^(k)_t_0≤ Bd_k. Wald’s identity at the bounded stopping time σ∧nσ n gives g(λ¯k)[(σ∧n)−t0]≤Bdk+cins(λ¯k)g( λ_k)\,E[(σ n)-t_0]≤ Bd_k+c_ins( λ_k), as in the proof of Eq. (13), and letting n→∞n→∞ completes the proof. □ The ratios in (a) explain E6. Per record, an inserted loss removes between 11 and M+/m−M_+/m_- units of bet k’s tolerance with the insertion cost, since cins≤dins≤cinsM+/m−c_ins≤ d_ ins≤ c_insM_+/m_-, and between 12 12 and 11 unit with the replacement cost, since dins=crep−cins≥crep/2d_ ins=c_rep-c_ins≥ c_rep/2. A replaced win removes crep/cins≥2c_rep/c_ins≥ 2 units with the insertion cost and exactly one unit with the replacement cost, by (b). The small bets that carry the tolerance of a well-separated pair are near the lower ends of these ranges. Appendix C Experimental Details and Additional Results C.1 Overview of the experiments Table 2 lists the experiments with the question each one answers, the results it checks, its main takeaway, and the tables and figures that report it. Table 2: Map of the experiments. Checks: the theorems and propositions that each experiment tests or illustrates. Results: where its numbers are reported. Exp. Question Checks Main takeaway Results E1 Are the certificates valid against bursts, spread forgeries and flips? Is each cost needed? Theorems 5.1 and 5.2, Proposition 3.1 The replacement certificate is valid under all four attacks; the insertion certificate fails only under flips; the baselines fail under bursts. Figure 1, Table 3 E2 Does the tolerance grow at the optimal rates? Theorems 5.3 and 5.4, Corollaries B.3 and B.4 Tolerance per record close to 2δ2δ and δ; time to certify close to the upper bound. Figure 2(a,b), Table 4 E3a How large are the tolerances on real votes? Theorems 5.1 and 5.7, Proposition B.8 Separated pairs withstand about 2,0002,000 forged votes; close pairs are not certified. Figure 2(c), Table 5 E3b Do rigged real votes fool the baselines? Theorems 5.1 and 5.2 400400 forged votes in a burst: the Wald interval and undeflated betting err in 9898 to 100%100\% of replays, our certificates in at most 2.2%2.2\% at any budget. Tables 1 and 6 E3c What holds when the whole leaderboard is claimed at once? Proposition 5.6 613613 of 16,51216,512 ordered pairs certified; GPT-4o is not certified above Claude-3.5-Sonnet. Figure 3 E4 Is the price of selecting among V variants needed and enough? Proposition 5.5 Best of 2727 null variants: false statement 0.520.52 without correction, 0.0250.025 with V/αV/α. Table 7 E5 What happens when the better model changes? Theorem 5.7, identity of Eq. (16) Both certificates return to −1-1 about 420420 records after the switch. Figure 4 E6 What can an attacker do to a correct claim? Proposition B.9 A flipped win removes one flipped-vote unit; de-certifying costs about the tolerance. Table 8 E7 Do the results carry over to a static benchmark? Theorems 5.1, 5.2 and 5.7, Proposition B.8 Tolerances are 9191 to 94%94\% of the benchmark’s own count; only the replacement certificate survives flips. Table 9, Figure 5(a,b) E8 Are frontier-model rankings on benchmarks certified? Proposition 5.6, Corollary B.5 On all five benchmarks no top-two ordering is certified simultaneously. Table 10, Figure 5(c), Figure 6 E9 What does the lead claim cost? Theorems 5.3 and 5.7 Equal to the edge tolerance on binary records; 44 to 11%11\% lower with 30%30\% ties. Table 11 E10 How far is a fixed-time sensitivity count? Theorem 5.1, Proposition B.8 The edge tolerance is 9090 to 97%97\% of the Wald count, which attacks can time. Tables 5 and 9 Grid. All experiments use the grid of Section 4, λ¯k=2−(k−1)/2 λ_k=2^-(k-1)/2 for k=1,…,14k=1,…,14 with equal weights. Its 2 2 spacing puts a bet in [3δ/2,3δ][3δ/ 2,3δ] for every margin δ from about 0.0040.004 to 13 13, as Corollary B.3 requires. All experiments use α=0.05α=0.05 and fixed seeds, and Figure 1 also shows our single-bet certificate of Eq. (7) with λ=0.2λ=0.2. Baselines. Undeflated betting compares the wealth, not deflated, with 1/α1/α (Waudby-Smith and Ramdas,, 2024). It uses the single bet λ=0.2λ=0.2 in simulations and the average over the grid in replays. Per-step robust, a simplified form of the test of Saha and Ramdas, (2026), tests the null mean 12+ε 12+ (Wang and Ramdas,, 2023, Thm. 6). This is valid when each record is corrupted with conditional probability at most ε . It is favorably calibrated with ε=B/n =B/n from the attacker’s true budget B and the number n of uncorrupted values. Wald interval is the 95%95\% normal interval for the mean outcome, recomputed every 5050 records. C.2 Reproducibility All code, the benchmark data we use, the saved results of every experiment and the scripts that rebuild every figure and table are at https://github.com/HamedKhosravi99/corruption-tolerance. Each experiment runs from one or two scripts with fixed seeds. The repository’s README maps every figure, table and number of the paper to the script and the saved files that produce it. Rebuilding all figures and tables from the saved results takes about a minute. The per-item results of HELM and SWE-bench Verified are public and included. The Chatbot Arena battle file is public but states no license, so the repository gives the commands that download and extract it instead of a copy. The code needs Python with NumPy, SciPy, pandas and Matplotlib, with versions pinned in the repository. The experiments ran on a 12-core workstation, a laptop and four-core jobs on a Linux cluster, and the longest took about an hour. C.3 E1: validity Uncorrupted values are independent Bern(12)Bern( 12), with n=5,000n=5,000 values per repetition and 10,00010,000 repetitions per cell. There are four attackers, and Figure 1 shows three of them. Burst at start inserts B forged wins before the first uncorrupted record. Burst at trigger inserts B forged wins as soon as the undeflated log-wealth with λ=0.2λ=0.2 first reaches log(1/α)−Bcins(0.2) (1/α)-Bc_ins(0.2). This is the earliest moment at which the burst carries the undeflated wealth to 1/α1/α. For the single-bet insertion certificate the timing of forged wins does not matter. Spread inserts a forged win before each uncorrupted record with probability B/nB/n until the budget is spent. Flip losses replaces the first B uncorrupted losses by wins. For constant bets and binary records, the last attacker is the optimal offline replacement attacker (Theorem 5.2(d)). Undeflated betting and the per-step robust test use the single bet λ=0.2λ=0.2. Table 3 lists the rates for B∈0,10,20,40,100B∈\0,10,20,40,100\, and the results file in the repository also has B=5B=5, 3030 and 6060. Figure 1 plots burst at trigger, flip losses and spread at all eight budgets, and burst at start appears only in Table 3. Table 3: E1: probability that a method makes a false statement at some time on null data, for each attack and attacker budget B (5,0005,000 uncorrupted Bern(12)Bern( 12) records; 10,00010,000 repetitions; standard error at most 0.0050.005). A method is valid if its entry stays at or below α=0.05α=0.05. Undefl. and Per-step: undeflated betting and the per-step robust test, with the single bet λ=0.2λ=0.2. Ins. and Rep.: our insertion and replacement certificates, with this bet (1 bet) or with the grid of Section 4 (grid). Rows: the four attacks described above. attack B Undefl. Per-step Ins. (1 bet) Ins. (grid) Rep. (1 bet) Rep. (grid) burst at start 0 0.049 0.049 0.049 0.028 0.049 0.028 burst at start 10 0.127 0.110 0.052 0.029 0.019 0.008 burst at start 20 0.317 0.263 0.048 0.029 0.006 0.003 burst at start 40 1.000 1.000 0.046 0.028 0.001 0.001 burst at start 100 1.000 1.000 0.051 0.032 0.000 0.000 burst at trigger 0 0.048 0.048 0.048 0.028 0.048 0.028 burst at trigger 10 0.124 0.104 0.047 0.027 0.015 0.014 burst at trigger 20 0.326 0.266 0.046 0.027 0.005 0.004 burst at trigger 40 1.000 1.000 0.046 0.028 0.001 0.000 burst at trigger 100 1.000 1.000 0.044 0.029 0.000 0.000 spread 0 0.052 0.052 0.052 0.033 0.052 0.033 spread 10 0.057 0.044 0.048 0.028 0.043 0.024 spread 20 0.062 0.039 0.049 0.030 0.040 0.023 spread 40 0.073 0.028 0.045 0.027 0.029 0.017 spread 100 0.156 0.014 0.049 0.029 0.015 0.009 flip losses 0 0.049 0.049 0.049 0.027 0.049 0.027 flip losses 10 0.354 0.328 0.137 0.449 0.046 0.028 flip losses 20 0.982 0.979 0.369 0.935 0.047 0.028 flip losses 40 1.000 1.000 0.947 1.000 0.050 0.030 flip losses 100 1.000 1.000 1.000 1.000 0.048 0.028 C.4 E2: growth of the tolerance and time to certify Uncorrupted records are independent Bern(12+δ)Bern( 12+δ). The horizon is 60,00060,000 records for δ=0.02δ=0.02, 20,00020,000 for δ=0.05δ=0.05 and 8,0008,000 for δ=0.1δ=0.1. Each δ has 2,0002,000 repetitions. Every repetition reached every tolerance b in Table 4 within the horizon. The column “union” is the mean of τb _b for the tolerance max−1,maxk⌊log(αwkKt(k))/ck⌋ \-1, _k (α w_kK^(k)_t)/c_k \, which takes the best bet at level αwkα w_k instead of averaging the deflated wealths. Averaging the deflated wealths needs 2.52.5–22%22\% fewer records on average than the union variant. The upper bound of Eq. (13) also applies to the union variant, since its τb _b is the minimum over k of the σk _k in the proof of Theorem 5.3. Five of its 4848 computed sample means exceed the bound, each by less than 0.3%0.3\% and by less than one Monte Carlo standard error. Three of them are among the 3030 shown in Table 4, and one of these three equals the bound after rounding. Table 4: E2: number of records τb _b until the tolerance first reaches b, on uncorrupted i.i.d. records with mean outcome 12+δ 12+δ and no attack (2,0002,000 repetitions). Mean τb _b and median: our tolerance of Eq. (9). Union: mean τb _b of the union-over-bets variant defined above. Upper and lower: the bound of Eq. (13) of Theorem 5.3 and the bound of Corollary B.4. Cost: insertion (ins.) or replacement (rep.). Means close to the upper bound show that it is nearly attained. δ cost b mean τb _b union upper Eq. (13) lower (Cor. B.4) median 0.02 ins. 0 4,866 6,268 7,175 3,744 4,146 0.02 ins. 10 5,603 6,905 7,721 3,744 4,968 0.02 ins. 50 7,692 9,011 9,456 3,744 7,156 0.02 ins. 100 9,908 11,296 11,475 3,744 9,373 0.02 ins. 400 20,936 22,550 22,505 8,411 20,429 0.02 rep. 0 4,866 6,268 7,175 3,744 4,146 0.02 rep. 10 6,175 7,512 8,257 3,744 5,502 0.02 rep. 50 9,990 11,372 11,539 3,744 9,462 0.02 rep. 100 13,930 15,458 15,642 3,835 13,318 0.02 rep. 400 34,212 36,080 36,089 16,803 33,888 0.05 ins. 0 807 1,021 1,158 598 705 0.05 ins. 10 1,041 1,241 1,329 598 958 0.05 ins. 50 1,715 1,940 2,015 598 1,637 0.05 ins. 100 2,462 2,707 2,734 773 2,397 0.05 ins. 400 6,327 6,651 6,670 3,376 6,266 0.05 rep. 0 807 1,021 1,158 598 705 0.05 rep. 10 1,249 1,458 1,516 598 1,168 0.05 rep. 50 2,509 2,755 2,779 724 2,444 0.05 rep. 100 3,879 4,164 4,216 1,538 3,809 0.05 rep. 400 11,239 11,640 11,630 6,732 11,148 0.10 ins. 0 208 258 292 149 177 0.10 ins. 10 303 357 374 149 279 0.10 ins. 50 608 669 675 184 591 0.10 ins. 100 944 1,011 1,016 390 927 0.10 ins. 400 2,739 2,836 2,836 1,698 2,729 0.10 rep. 0 208 258 292 149 177 0.10 rep. 10 400 455 472 149 382 0.10 rep. 50 978 1,048 1,056 364 956 0.10 rep. 100 1,609 1,693 1,699 773 1,596 0.10 rep. 400 5,086 5,216 5,219 3,376 5,082 C.5 E3: Chatbot Arena The three parts of E3 are the tolerances per pair (E3a), a rigging replay (E3b) and a simultaneous matrix (E3c). Arena data. The file clean_battle_20240814_public.json (2.1 GB) is public on the maintainers’ storage bucket (LMSYS Org / LMArena,, 2024) and is used by the official Arena leaderboard notebook. For each vote it contains the two model names, the winner, a Unix timestamp, a hashed voter identifier and metadata. The winner is coded as model_a, model_b, tie or tie (bothbad). The file has no prompts or responses. We keep the model names, winner, timestamp and anonymity flag (all 1,799,9911,799,991 votes are anonymous battles). No license is stated. Chiang et al., (2024) describe an active rule that chooses the next pair from past votes; the rule in force during these votes is not published. The certificates do not depend on it. They need only that the pair of a battle is fixed before its vote is cast, which holds under any sampling rule, since the user votes after reading both answers. The official notebook also down-samples the votes on the 0.1%0.1\% most repeated prompts, which leaves 1,670,2501,670,250 votes. We keep all 1,799,9911,799,991, of which 35.7%35.7\% are ties. Both tie labels are coded W=12W= 12. Table 5 (E3a) gives the tolerances at the end of the data for the twenty most frequently compared pairs. Well-separated pairs are certified against up to about 2,0002,000 forged votes, while the closest pairs are not certified. The last column is Δ=n(2W¯−1) =n(2 W-1), wins minus losses as in Proposition B.8, where n is the number of votes on the pair and W¯ W their mean outcome. By Proposition B.8 the completed vote set itself withstands Δ−1 -1 forged or ⌈Δ/2⌉−1 /2 -1 flipped votes; the certificates are below these counts by 10%10\% and 12%12\% for the first pair, the price of a claim about the normal process. The lead tolerances are 33 to 9%9\% below the edge tolerances on the pairs certified above 500500 votes, the price of the 36%36\% ties under the exponential wealth. On marginal pairs the gap is larger, and the pair that the edge certificate certifies at 1111 forged votes is not certified by the lead certificate. Table 6 (E3b) gives the rigging replay of Table 1 for each of its three pairs. These are the three pairs that Table 5 does not certify, and the trailing model is promoted. The promoted models have mean outcomes 0.4880.488–0.4950.495 over 6,0896,089–6,4706,470 votes. Each replay resamples the pair’s own votes with replacement, so the claim “the promoted model is better” is false. Rigging that makes the baselines certify this false ordering leaves the matching certificate valid (the insertion certificate for forged votes and the replacement certificate for flipped ones). In the replay, a burst inserts B forged wins for the promoted model at a uniformly random position. Spread inserts a forged win before each record with probability B/nB/n until B are inserted. Flip replaces the promoted model’s first B losses by wins. Undeflated betting there compares the grid average, not deflated, with 1/α1/α, and the per-step test uses the single bet λ=0.2λ=0.2. The run covers B up to 800800. Tables 1 and 6 show B=0B=0, 100100 and 400400. Figure 3 (E3c) shows the tolerances of the edge claim “row has ever had an edge over column”, computed from the votes of the last 120120 days and made simultaneous by closed testing over all sets of pairs (Proposition 5.6). The family is all 16,51216,512 ordered pairs of the 129129 models in the data, which does not depend on the votes. Of these pairs, 3,9263,926 have battles in the window and 613613 are certified. The ten displayed models are those with the highest net win rate (wins minus losses, per battle) among the 2525 most played models in the window. Because the guarantee covers every pair, the choice of which pairs to display may depend on the data. Closed testing certified at most one more record than a Bonferroni split of α over the 16,51216,512 pairs. Simultaneity has a price. For GPT-4o over Claude-3.5-Sonnet (5,8685,868 battles), the single-pair tolerance at the end of the data is 5353 forged or 2626 flipped votes, while the simultaneous tolerance is −1-1. For GPT-4o over GPT-4-Turbo the single-pair tolerance is 345345 forged votes and the simultaneous one 225225. Fixing the family in advance also has a price. With a family of only the 9090 ordered pairs among the displayed ten, the tolerance for GPT-4o over GPT-4-Turbo would be 280280 and three more cells would be certified. That family is chosen from the same votes, so Proposition 5.6 does not cover it. The difference is about log(16,512/90)≈5.2 (16,512/90)≈ 5.2 nats of evidence, which the closed test spends on the additional pairs instead of on tolerance. Table 5: E3a: tolerances on the real, time-ordered Chatbot Arena votes, for the twenty most frequently compared pairs. Each row concerns the edge and lead claims for the leader against the other model. An entry is the number of corrupted votes that a claim withstands after all n votes of the pair, and −1-1 means not certified. Edge and lead: the edge tolerance of Eq. (9) and the lead tolerance of Theorem 5.7, in forged and in flipped votes. Wald: the votes that the fixed-time 95%95\% Wald interval absorbs (E10). W¯ W: mean outcome, with ties counted as 12 12. Ties: share of tied votes. Δ : wins minus losses. Models are named by their Arena identifiers. edge lead Wald leader other votes W¯ W ties forged flipped forged flipped forged flipped Δ llama-3-70b-instruct llama-3-8b-instruct 14,858 0.577 0.48 2,056 1,009 1,940 957 2,139 1,056 2,287 claude-3-opus-20240229 llama-3-70b-instruct 11,971 0.554 0.35 1,053 517 984 485 1,143 565 1,304 claude-3-opus-20240229 claude-3-sonnet-20240229 9,513 0.579 0.38 1,288 630 1,217 598 1,363 672 1,495 gpt-4-turbo-2024-04-09 llama-3-70b-instruct 8,940 0.571 0.35 1,065 521 1,003 493 1,141 562 1,275 llama-3-70b-instruct reka-core-20240501 8,446 0.518 0.39 80 39 18 9 158 78 298 gemini-1.5-pro-api-0409-preview llama-3-70b-instruct 7,633 0.556 0.35 658 321 602 296 730 359 858 claude-3-opus-20240229 claude-3-haiku-20240307 7,510 0.599 0.37 1,301 635 1,238 608 1,366 672 1,480 gemini-1.5-pro-api-0409-preview llama-3-8b-instruct 7,292 0.623 0.34 1,621 791 1,563 767 1,683 828 1,794 claude-3-opus-20240229 gpt-4-0125-preview 7,190 0.515 0.33 11 5 −1-1 −1-1 87 43 222 claude-3-opus-20240229 command-r-plus 6,753 0.581 0.35 919 448 864 423 983 484 1,098 claude-3-sonnet-20240229 claude-3-haiku-20240307 6,751 0.530 0.42 214 104 152 74 281 138 400 gpt-4-0314 gpt-4-0613 6,607 0.536 0.42 299 145 237 116 364 179 480 gpt-4-turbo-2024-04-09 claude-3-opus-20240229 6,470 0.505 0.35 −1-1 −1-1 −1-1 −1-1 −1-1 −1-1 69 gpt-4o-2024-05-13 llama-3-70b-instruct 6,470 0.615 0.32 1,314 640 1,263 619 1,375 676 1,483 llama-3-70b-instruct snowflake-arctic-instruct 6,418 0.650 0.34 1,771 862 1,714 839 1,826 897 1,923 gpt-4-0125-preview gpt-4-0613 6,395 0.637 0.35 1,595 776 1,536 752 1,651 811 1,749 gemini-1.5-pro-api-0409-preview claude-3-opus-20240229 6,324 0.512 0.34 −1-1 −1-1 −1-1 −1-1 19 9 146 llama-3-8b-instruct snowflake-arctic-instruct 6,206 0.569 0.39 685 334 627 307 747 367 855 claude-3-opus-20240229 llama-3-8b-instruct 6,175 0.615 0.32 1,259 613 1,209 592 1,319 648 1,424 llama-3-70b-instruct claude-3-sonnet-20240229 6,089 0.509 0.37 −1-1 −1-1 −1-1 −1-1 −1-1 −1-1 106 Table 6: E3b: the rigging replay of Table 1 for each of the three pairs, as the probability of a false statement at B=0B=0 / 100100 / 400400 votes (1,0001,000 replays per cell). Promoted: the slightly worse model that the attacker favors. Burst: forged votes at one random time. Spread: forged votes at random times. Flip: the promoted model’s losses turned into wins. Wald: the Wald interval recomputed every 5050 votes. Undefl.: undeflated betting with the grid average. Per-step: the per-step robust test. Ins. and Rep.: our insertion and replacement certificates. An entry at or below 0.050.05 means valid. promoted vs opponent attack Wald Undefl. Per-step Ins. Rep. Claude-3-Opus vs GPT-4-Turbo burst .09 / .51 / 1.0 .00 / .28 / 1.0 .02 / .17 / .30 .00 / .02 / .01 .00 / .02 / .01 spread .08 / .27 / 1.0 .00 / .04 / .99 .01 / .01 / .00 .00 / .01 / .01 .00 / .00 / .00 flip .09 / 1.0 / 1.0 .01 / 1.0 / 1.0 .01 / 1.0 / 1.0 .01 / 1.0 / 1.0 .01 / .01 / .01 Claude-3-Opus vs Gemini-1.5-Pro-0409 burst .05 / .30 / 1.0 .00 / .16 / .98 .01 / .10 / .24 .00 / .01 / .01 .00 / .01 / .01 spread .05 / .11 / .97 .01 / .01 / .84 .00 / .00 / .00 .01 / .00 / .01 .01 / .00 / .00 flip .04 / 1.0 / 1.0 .00 / 1.0 / 1.0 .00 / 1.0 / 1.0 .00 / 1.0 / 1.0 .00 / .01 / .00 Claude-3-Sonnet vs Llama-3-70B burst .06 / .40 / 1.0 .00 / .22 / 1.0 .01 / .12 / .30 .00 / .01 / .01 .00 / .01 / .01 spread .05 / .17 / 1.0 .01 / .03 / .95 .01 / .00 / .00 .01 / .01 / .01 .01 / .00 / .00 flip .07 / 1.0 / 1.0 .01 / 1.0 / 1.0 .01 / 1.0 / 1.0 .01 / 1.0 / 1.0 .01 / .00 / .01 Figure 3: E3c: corruption tolerance of the edge claim “row model has ever had an edge over column model” in the last 120120 days of the data, simultaneous over all 16,51216,512 ordered pairs of the 129129 models (α=0.05α=0.05). Shown are the ten models with the highest net win rate among the 2525 most played in the window. Empty cells: not certified. Dots: no battles. Gray diagonal: same model. C.6 E4: selection Each of V variants has its own independent Bern(12)Bern( 12) records, with n=5,000n=5,000 per variant and 4,0004,000 repetitions. The grid certificate is evaluated at b=0b=0, and the corrected and uncorrected certificates use the same simulated records. The developer publishes the variant whose tolerance first becomes nonnegative. Publishing the best of V variants that are no better than the opponent gives a false statement with probability 0.090.09, 0.230.23 and 0.520.52 for V=3V=3, 99 and 2727 without correction. With the correction of Proposition 5.5 these probabilities are 0.0280.028, 0.0270.027 and 0.0250.025. Table 7: E4: probability of a false statement when the developer publishes the best of V variants, none of them better than the opponent (4,0004,000 repetitions; standard errors in parentheses). Uncorrected: threshold 1/α1/α. Corrected: threshold V/αV/α, as in Proposition 5.5. An entry at or below α=0.05α=0.05 means that the published certificate is valid. V uncorrected corrected 1 0.029 (0.003) 0.029 (0.003) 3 0.086 (0.004) 0.028 (0.003) 9 0.233 (0.007) 0.027 (0.003) 27 0.519 (0.008) 0.025 (0.002) C.7 E5: certificate persistence under a regime shift Uncorrupted records are binary. Model i wins each of the first 1,0001,000 records with probability 0.60.6 and each later record with probability 0.40.4. There is no attack, and 2,0002,000 repetitions of 5,0005,000 records are run with the grid of Section 4. The null hypothesis is false, so the claim is correct in the sense of Section 2. The uncorrupted margin of Eq. (2) is positive before t=2,000t=2,000 and at most zero from then on, so the lead claim is correct exactly before t=2,000t=2,000. Figure 4 shows the insertion and replacement tolerances and the empirical win rates. At the switch 99.85%99.85\% of the runs are certified, with median tolerances of 109109 forged and 5252 flipped records. After the switch the tolerance returns to −1-1 at a median of 420420 records (10–90% range 240240 to 620620). One third of the runs are still certified 500500 records after the switch, and none 2,0002,000 records after it. The cumulative win rate crosses one half only at t=2,000t=2,000. In 61%61\% of the runs the tolerance becomes nonnegative again for a few records after first reaching −1-1, when the average wealth Φt(0) _t(0) crosses 1/α1/α once more. The renewed tolerance is then at most 1414 forged records. A second certificate starts at the prespecified time t=1,001t=1,001 and sees only the later records. Its null hypothesis is true, and it certified nothing in any of the 2,0002,000 runs. The lead tolerance of Theorem 5.7, computed on the same records with the same grid, is also shown. Its certified status agrees with that of the betting tolerance on all but 0.016%0.016\% of the (run, record) cells. Its median tolerances at the switch are 109109 forged and 5151 flipped records, and its return times are the same. In 0.05%0.05\% of the runs it is nonnegative at some t≥2,000t≥ 2,000, where the lead claim is false, within the 5%5\% that Theorem 5.7 allows. The betting tolerance does so in the same runs. The experiment shows the meaning of a correct claim (Section D.1). It is not a validity failure. Figure 4: E5: regime shift with no attack (2,0002,000 repetitions, α=0.05α=0.05; median and 1010–90%90\% band). (a) Betting tolerances of Eq. (9) with bands, and lead tolerances of Theorem 5.7 (dotted, medians only). (b) Cumulative and trailing empirical win rates. The vertical dotted line marks the switch. C.8 E6: attacks on correct claims For the three most certified pairs of Table 5 an attacker works against the leader on the real votes. It inserts B losses for the leader in one burst, or it replaces B of the leader’s wins by losses. Each claim is certified against more than 1,6001,600 forged votes, so we treat it as correct. Then no statement can be false, and the question is how much tolerance the attacker destroys. By the identity of Eq. (16) of Section D.1, the final tolerance does not depend on where the corrupted records are placed. Table 8 reports the results. Each inserted loss removes about 1.051.05 forged-vote units of tolerance and 0.510.51 flipped-vote units. Each win replaced by a loss removes about 2.052.05 forged-vote units and exactly 1.001.00 flipped-vote unit. The attacker’s budget that de-certifies a claim is therefore about the tolerance itself. For Llama-3-70B over Llama-3-8B it is 1,9861,986 inserted losses or 1,0101,010 replaced wins, against tolerances of 2,0562,056 and 1,0091,009. After a burst of 800800 inserted losses at the midpoint of the data, the tolerance of Llama-3-70B over Llama-3-8B never falls below zero. It is back at half of its pre-burst value after about 2,2002,200 uncorrupted votes. Llama-3-70B over Snowflake-Arctic is below zero for 191191 of the uncorrupted votes after the burst, and Gemini-1.5-Pro-0409 over Llama-3-8B for 108108. Proposition B.9 in Appendix B accounts for these numbers. Per record, an inserted loss removes between 11 and (2+λ¯k)/(2−λ¯k)(2+ λ_k)/(2- λ_k) units of bet k’s forged-vote tolerance and between 12 12 and 11 flipped-vote unit. A replaced win removes at least 22 forged-vote units and exactly one flipped-vote unit. Recovery takes about Bd/gBd/g uncorrupted records, with d the loss per record and g the growth rate of Eq. (11). Table 8: E6: attacks on correct claims, on the real Arena votes of the three most certified pairs. n: votes on the pair. B^n B_n: tolerance after all n votes, in forged and in flipped votes. De-certify by: the number of inserted losses, or of wins replaced by losses, that brings the forged-vote tolerance to −1-1. Loss of tolerance: forged-vote tolerance removed per inserted loss or per replaced win, at B=100B=100. Proposition B.9 bounds these losses. pair n B^n B_n forged B^n B_n flipped de-certify by de-certify by loss of tolerance loss of tolerance inserted losses replaced wins per inserted loss per replaced win Llama-3-70B >> Llama-3-8B 14,858 2,056 1,009 1,986 1,010 1.04 2.04 Llama-3-70B >> Snowflake-Arctic 6,418 1,771 862 1,689 863 1.06 2.06 Gemini-1.5-Pro-0409 >> Llama-3-8B 7,292 1,621 791 1,551 792 1.05 2.05 C.9 E7: a static benchmark, MMLU Data. Stanford HELM publishes per-instance outputs of its MMLU leaderboard (release v1.13.0) (Liang et al.,, 2023; Stanford CRFM,, 2025; Hendrycks et al.,, 2021). We use the per-item exact-match results of four open-weight models on the 14,04214,042 questions of the released MMLU test set, a multiple-choice exam in 57 subjects. The models are Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Gemma-2-9B and Mistral-7B-Instruct-v0.3. The MMLU paper reports 14,07914,079 test questions. Each prompt contains five solved examples (5-shot). HELM ran Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct through the Turbo endpoints of Together AI, a cloud inference provider. These endpoints use 8-bit floating-point (FP8) quantization. Gemma-2-9B is the base model, since HELM’s MMLU leaderboard does not include the instruction-tuned one. The four models’ accuracies over the 14,04214,042 questions are 0.5420.542, 0.7140.714, 0.7040.704 and 0.5890.589. HELM’s leaderboard averages over the 5757 subjects instead and reports 0.5610.561, 0.7290.729, 0.7210.721 and 0.5990.599. A record is one question. For the ordered pair (i,j)(i,j), W=1W=1 if i is correct and j wrong, W=0W=0 for the reverse, and W=12W= 12 if both are correct or both wrong. A contaminated item is one whose record contamination altered, whatever the cause. The record order is a random permutation drawn before any outcome is looked at (seed 00). The permutation only defines an online order and does not by itself make the records independent. For the population reading of a claim we adopt a modeling assumption, that the questions are an i.i.d. sample from a larger population of questions. A claim then compares the two models on that population. Without this assumption, Proposition B.8 still gives the tolerance of the released benchmark itself, reported below. By Eq. (16) of Section D.1, the final tolerance does not depend on the order, and we confirmed that it was identical over 100100 random orders for every pair. The time to certify does depend on the order, and Table 9 reports it for the seed-00 order and its median over the 100100 orders. Tolerances. Table 9 gives the tolerances. The four well-separated pairs (Δ above 1,6001,600) are certified against 1,4691,469 to 2,2642,264 forged items and 714714 to 1,1021,102 contaminated items. The forged-item tolerances are 9191 to 94%94\% of Δ−1 -1, the number of forged items that the released benchmark itself withstands (Proposition B.8), and the contaminated-item tolerances are 8989 to 91%91\% of ⌈Δ/2⌉−1 /2 -1, its count for replaced items. Contaminating one model moves a record by at most 12 12, so for contamination alone the deterministic count is Δ−1 -1 and these tolerances are conservative. The lead certificate gives 1,2691,269 to 2,0812,081 forged and 626626 to 1,0261,026 contaminated items on the four well-separated pairs, 77 to 14%14\% below the edge certificate. Mistral-7B over Llama-3.1-8B is certified against 470470 forged items, 71%71\% of Δ . Qwen2.5-7B against Gemma-2-9B is not certified in either direction. Their net difference is 140140 wins on 2,9602,960 untied items. Rigging replay. The replay follows E3b. For each of the six pairs the promoted model is the one with the lower accuracy, so the claim “the promoted model is better” is false. Null streams resample the pair’s own items with replacement. The attacker inserts B forged items (W=1W=1) in one burst at a random position, spreads them at rate B/nB/n, or flips the first B items that the promoted model lost (W=0→1W=0→ 1, the value-dependent attacker). The methods are those of Section 6, with 1,0001,000 replays per cell. Figure 5(a,b) shows the mean over the six pairs. Forged bursts of up to 400400 items leave our certificates at or below 0.0060.006 on every pair. The Wald interval and undeflated betting reach 0.360.36 and 0.330.33 on average over the pairs, and 0.9990.999 and 0.9950.995 on the worst pair. Flipping the lost items breaks the Wald interval from 55 items (0.350.35, and 0.920.92 at 1010) and the per-step robust test at 2525 (0.860.86). The insertion certificate fails from 1010 items (0.080.08) and reaches 0.440.44 at 2525 and 0.920.92 at 5050, as Theorem 5.2(b) predicts. The replacement certificate stays at or below 0.0080.008 on every pair and budget. Table 9: E7: tolerances on HELM MMLU (14,04214,042 questions) for the six pairs, oriented by accuracy. Each row concerns the edge and lead claims for the first model against the second. Edge and lead: the edge tolerance of Eq. (9) and the lead tolerance of Theorem 5.7 after all questions, in forged and in contaminated items. Wald: the items that the fixed-time 95%95\% Wald interval absorbs (E10). Δ : wins minus losses. τ0 _0: questions until the claim is first certified, in the seed-00 order (seed 0) and as a median over 100100 random orders (median). –: not certified at the end, or never certified (τ0 _0, in the seed-00 order or in most of the 100100 orders). edge lead Wald τ0 _0 pair wins losses ties Δ forged contam. forged contam. forged contam. seed 0 median Qwen2.5-7B >> Gemma-2-9B 1,550 1,410 11,082 140 – – – – 33 16 – – Qwen2.5-7B >> Mistral-7B 2,841 1,087 10,114 1,754 1,607 781 1,411 696 1,660 815 91 141 Qwen2.5-7B >> Llama-3.1-8B 3,571 1,156 9,315 2,415 2,264 1,102 2,081 1,026 2,318 1,140 61 95.5 Gemma-2-9B >> Mistral-7B 2,672 1,058 10,312 1,614 1,469 714 1,269 626 1,521 747 204 160.5 Gemma-2-9B >> Llama-3.1-8B 3,532 1,257 9,253 2,275 2,117 1,032 1,939 956 2,174 1,069 134 101.5 Mistral-7B >> Llama-3.1-8B 2,531 1,870 9,641 661 470 229 303 149 539 265 1,281 1,195 Figure 5: E7 and E8: probability of a false statement on resampled benchmark items (1,0001,000 replays per cell, α=0.05α=0.05 dotted). (a,b) MMLU, mean over the six pairs of four open-weight models with 7 to 9 billion parameters. Forged items are inserted in one burst at a random position (a). Flipped items are the first B items the promoted model lost, turned into wins (b). (c) Flipped items on the five frontier benchmarks of E8. Each line averages the five benchmarks, each benchmark a mean over its pairs. Each band is the range over the five benchmarks. C.10 E8: frontier models on HELM and SWE-bench Verified HELM data. HELM’s capabilities leaderboard (release v1.15.0) evaluates 6868 models (Stanford CRFM,, 2025). We use fifteen frontier models of 2025 from seven developers, chosen by developer and release from HELM’s model list. They are Claude Opus 4, Claude Sonnet 4, Claude Sonnet 4.5, o3, GPT-4.1, GPT-5, GPT-5.1, gpt-oss-120b, Gemini 2.5 Pro (03-25 preview), Gemini 3 Pro (preview), Grok 4, DeepSeek-R1-0528, Kimi K2, Qwen3-235B and Qwen3-235B-2507. The exact HELM identifiers are in the repository. HELM ran the Claude models without extended thinking, their optional reasoning mode, and both Qwen3 models with FP8 quantization. HELM ran GPT-5.1 with the provider’s default setting, which turns reasoning off. Its accuracy on MMLU-Pro and GPQA (0.5790.579 and 0.4420.442) is therefore far below that of GPT-5 (0.8630.863 and 0.7910.791). The benchmarks are a sample of 1,0001,000 questions of MMLU-Pro, a harder version of MMLU (Wang et al.,, 2024), and 446446 of the 448448 graduate-level science questions of GPQA main (Graduate-Level Google-Proof Q&A) (Rein et al.,, 2024). They also include the 541541 prompts of IFEval (Instruction-Following Evaluation), with instructions whose compliance is checked automatically (Zhou et al.,, 2023), and a sample of 1,0001,000 olympiad mathematics problems of Omni-MATH (Gao et al.,, 2025). On MMLU-Pro and GPQA the prompt has no solved examples and asks for step-by-step reasoning (zero-shot chain of thought). HELM publishes per-instance outputs. On MMLU-Pro and GPQA it records whether each answer is correct. On IFEval it records the fraction of a prompt’s instructions that were followed, and we count a prompt as right only when all were followed (prompt-level strict accuracy). On Omni-MATH it records the share of language-model judges that accept the answer (00, 13 13, 12 12, 23 23 or 11), and 4.2%4.2\% of these scores are fractional. The record is then W=12(1+si−sj)W= 12(1+s_i-s_j) for the scores sis_i and sjs_j of the two models. This equals the record of E7 when both scores are 00 or 11, and Theorem 5.1 allows any W∈[0,1]W∈[0,1]. Figure 6: E8: the six highest-scoring models of each frontier benchmark, with their scores in percent. A cell gives the simultaneous tolerance of the edge claim “row model has ever had an edge over column model” in contaminated items, by closed testing over all ordered pairs of the benchmark’s models. An empty cell means not certified, and 00 means certified against no contaminated item. The last column is the certified rank bound, one plus the number of models certified above the row model. SWE-bench data. A SWE-bench task asks a model to resolve a real issue of a GitHub repository, and it counts as resolved when the repository’s tests pass (Jimenez et al.,, 2024). SWE-bench Verified is a subset of 500500 tasks screened by software developers (OpenAI,, 2024). Its maintainers publish runs of one simple agent with many models and the outcome of every task (SWE-bench team,, 2026). The agent is mini-SWE-agent, a minimal agent of the SWE-agent project (Yang et al.,, 2024), in which the model acts only through shell commands. We use the runs of its version 2 with eleven models of 2025 and 2026, all run with version 2.0.0. They are Claude Opus 4.5 and 4.6, Claude Sonnet 4.5, Claude Haiku 4.5, GPT-5.2, GPT-5 mini, Gemini 3 Flash, GLM-5, Kimi K2.5, DeepSeek V3.2 and MiniMax M2.5. A run is kept only if its per-task file has all 500500 tasks and agrees with its reported score. Three runs fail this check. The file of Gemini 3 Pro marks every task unresolved although 69.6%69.6\% is reported. GPT-5.2-Codex has no per-task file, and the file of Gemini 3.5 Flash covers 441441 tasks. On all five benchmarks the records are those of E7, with the graded record above on Omni-MATH, and the order and the population assumption are those of E7. As in E7, the final tolerance was identical over 100100 random orders for every ordered pair with Δ>0 >0. The model sets are fixed by name and release, not by the records, so the family of all ordered pairs of a benchmark’s models does not depend on the data. Each item is one time step at which every pair receives its record, and a contaminated item counts once however many pairs it changes. Corollary B.5 in Appendix B extends Proposition 5.6 to this setting, with the replacement cost also against a developer who knows every item in advance, and Theorem 5.1 applies to each pair directly. The five benchmarks were released in 2023 and 2024. The HELM results come from release v1.15.0 of November 2025 and the SWE-bench Verified runs from February 2026. Tolerances. Table 10 summarizes the tolerances. On the HELM benchmarks, 5555 to 8888 of the 210210 ordered pairs are certified pair by pair, and 3636 to 7373 simultaneously by closed testing. On SWE-bench Verified the counts are 2525 and 1414 of 110110, and all 1414 involve one of the two weakest models. On every benchmark neither of the two highest-scoring models is certified above the other simultaneously. Within the top five, the largest simultaneous tolerance is 1717 forged or 77 contaminated items (o3 over GPT-5 on Omni-MATH). On MMLU-Pro and GPQA the largest simultaneous tolerances are against GPT-5.1 without reasoning, up to 279279 forged items. No model among the six best on SWE-bench Verified (72.872.8 to 76.8%76.8\%) is certified above another even pair by pair. Figure 6 shows the simultaneous tolerances among the six highest-scoring models of each benchmark. Arena reports an approximate rank for each model, one plus the number of models whose confidence interval lies above its own (Chiang et al.,, 2024). Figure 6 also gives the certified analogue, one plus the number of models certified above a model. These rank bounds hold for all models at once with probability at least 1−α1-α when no item is corrupted (Corollary B.5). On GPQA and SWE-bench Verified the bound is 11 for each of the six models. Rigging replay. The replay of E7 runs on every pair with unequal scores, 5353 to 105105 per benchmark. Five flipped items break the Wald interval with probability 0.350.35 to 0.730.73 on average over the pairs, and ten with probability 0.870.87 to 0.990.99. On average over the pairs, the insertion certificate fails from 1010 to 2525 flipped items, as Theorem 5.2(b) predicts. Under flips the replacement certificate stays at or below 0.0150.015 on every pair and budget (means in Figure 5(c)). Twenty-five forged items in one burst push the Wald interval to 0.150.15 to 0.480.48 on average. Under bursts of up to 400400 items, our certificates stay at or below 0.0210.021 on every pair. Table 10: E8: summary per frontier benchmark. Items and models: the benchmark’s size. Certified pairs: ordered pairs certified at α=0.05α=0.05, pair by pair or simultaneously by closed testing over all ordered pairs of the benchmark’s models. Top two: simultaneous tolerance of the leader over the runner-up. Largest tolerance: the largest simultaneous tolerance among the pairs within the top five, and among all pairs. Tolerances are forged / contaminated items, and – means not certified. The top two and top five are the first rows of Figure 6. On MMLU-Pro, Claude Opus 4 and o3 tie for fifth at 85.9%85.9\%, and with o3 as fifth the entry is 2 / 0. benchmark items models certified pairs certified pairs top two largest tolerance largest tolerance (pair by pair) (simultaneously) in the top five overall MMLU-Pro 1,000 15 60 of 210 42 – 1 / 0 279 / 117 GPQA 446 15 55 of 210 36 – – 118 / 47 IFEval 541 15 67 of 210 38 – 15 / 5 88 / 33 Omni-MATH 1,000 15 88 of 210 73 – 17 / 7 272 / 111 SWE-bench Verified 500 11 25 of 110 14 – – 69 / 26 C.11 E9: the lead tolerance on i.i.d. records Uncorrupted records are i.i.d. with mean outcome 12+δ 12+δ, and there is no attack. A record is a win with probability π/2+δπ/2+δ, a loss with probability π/2−δπ/2-δ and a tie otherwise, with π=1π=1 (binary records) or π=0.7π=0.7 (30%30\% ties, close to the 36%36\% on Chatbot Arena). For δ∈0.05,0.1δ∈\0.05,0.1\, 2,0002,000 repetitions of 5,0005,000 records are drawn once. The edge tolerance of Eq. (9) and the lead tolerance of Theorem 5.7 are computed on the same records with both costs and the grid of Section 4. Table 11 reports B^t/t B_t/t at t=5,000t=5,000 and the limit rates of Theorem 5.3 and Theorem 5.7(d). For a binary outcome, log(1+γ(Ws−12))=γ(Ws−12)−γ2/8+O(γ3) (1+γ(W_s- 12))=γ(W_s- 12)-γ^2/8+O(γ^3), so on binary records the two wealths differ by O(γ3t)O(γ^3t). Accordingly the two tolerances differ in certified status on 0.01%0.01\% of the (repetition, record) cells, and their medians at t=5,000t=5,000 agree to four decimals. A tie leaves Kt(k)K^(k)_t unchanged and multiplies Zt(γ)Z^(γ)_t by e−γ2/8e^-γ^2/8. With 30%30\% ties the lead tolerance is therefore 44 to 11%11\% lower at t=5,000t=5,000, while the limit rates differ by less than 1%1\%. At δ=0.1δ=0.1 the limit rates are 0.1980.198 to 0.1990.199 against the optimum 0.20.2 for insertion and 0.0990.099 against 0.10.1 for replacement. Table 11: E9: tolerance per record B^t/t B_t/t at t=5,000t=5,000 on i.i.d. records with mean outcome 12+δ 12+δ and no attack (2,0002,000 repetitions; median and 1010–90%90\% range). Ties: share of tied records. Edge: the tolerance of Eq. (9). Lead: the lead tolerance of Theorem 5.7. Limit rates: the almost-sure limits of B^t/t B_t/t given by Theorem 5.3 and Theorem 5.7(d). The optimal rates are 2δ2δ for insertion and δ for replacement. δ ties cost edge Eq. (9) lead Eq. (10) limit rates (edge, lead) 0.05 30% ins. 0.066 (0.051–0.081) 0.059 (0.043–0.073) 0.098, 0.098 0.05 30% rep. 0.032 (0.025–0.039) 0.029 (0.021–0.036) 0.049, 0.049 0.05 0% ins. 0.058 (0.039–0.077) 0.058 (0.039–0.077) 0.098, 0.098 0.05 0% rep. 0.028 (0.019–0.037) 0.028 (0.019–0.037) 0.049, 0.049 0.10 30% ins. 0.169 (0.153–0.184) 0.160 (0.145–0.176) 0.199, 0.198 0.10 30% rep. 0.082 (0.074–0.089) 0.078 (0.071–0.086) 0.099, 0.099 0.10 0% ins. 0.160 (0.141–0.178) 0.160 (0.141–0.178) 0.198, 0.198 0.10 0% rep. 0.078 (0.069–0.087) 0.078 (0.069–0.087) 0.099, 0.099 C.12 E10: fixed-time breakdown counts For each pair of E3a and E7, the columns “Wald” of Tables 5 and 9 give a breakdown count. It is the largest number of forged or flipped records that the two-sided 95%95\% Wald interval for the mean outcome, computed once on all records, absorbs while still excluding one half. This is what a sensitivity study of the final standings computes (Huang et al.,, 2025; Oyarhoseini et al.,, 2026). It is a fixed-time quantity, and Table 1 and Figure 5(a,b) show the same interval certifying false orderings under adaptive timing. On the Arena pairs certified above 500500 votes, the edge tolerance in forged votes is 9090 to 97%97\% of the Wald count (median 95%95\%) and the lead tolerance 8282 to 94%94\%. On the four well-separated MMLU pairs the edge tolerance is 9797 to 98%98\% and the lead tolerance 8383 to 90%90\%. The Hoeffding interval, with half-width log(2/α)/(2n) (2/α)/(2n), breaks down below the edge tolerance on every certified pair. Two pairs with a positive Wald count are not certified: Gemini-1.5-Pro-0409 over Claude-3-Opus (1919 forged votes) and Qwen2.5-7B over Gemma-2-9B (3333 items). The anytime guarantee thus costs a few percent of the breakdown count of a fixed-time interval that attacks can time. Appendix D Additional Discussion D.1 The guarantee in the language of hypothesis testing Definition 2.1 controls the type I error of a family of sequential tests, one for each possible number of corrupted records. For each integer b≥0b≥ 0 the null hypothesis at time t is that H0ijH_0^ij holds and at most b of the first t records were corrupted, and the platform rejects it when B^t≥b B_t≥ b. Thus B^t B_t is the largest number of corrupted records for which the null hypothesis is rejected at time t. The second part of each null hypothesis, Nt≤bN_t≤ b, is a random event and not a property of the joint distribution, so these are hypotheses in an extended sense. The formal guarantee is Eq. (3). A type I error is the rejection of a null hypothesis that is true. If H0ijH_0^ij fails, no null hypothesis in the family is true, and no type I error can occur. If H0ijH_0^ij holds, the null hypotheses that are true at time t are those with b≥Ntb≥ N_t, and one of them is rejected exactly when B^t≥Nt B_t≥ N_t. Hence Eq. (3) states that, when H0ijH_0^ij holds, the probability of a type I error at any time t and for any b is at most α. Equivalently, the probability that any published statement is ever false is at most α. The platform does not observe NtN_t. It therefore publishes the conditional statement “i has ever had an edge over j, unless more than B^t B_t of the first t records were corrupted” and not a rejection of H0ijH_0^ij itself. For the tolerance of Eq. (9), the quantity 1/Φt(b)1/ _t(b) plays the role of a p-value for the null hypothesis that H0ijH_0^ij holds and at most b records were corrupted: B^t≥b B_t≥ b exactly when 1/Φt(b)≤α1/ _t(b)≤α. More precisely, Φt(b) _t(b) does not depend on α and Theorem 5.1 holds at every level. So for every α′∈(0,1)α ∈(0,1) and every attacker in a class that Theorem 5.1 covers with the chosen cost, when H0ijH_0^ij holds, ℙ(∃t,∃b≥Nt: 1/Φt(b)≤α′)≤α′P (∃\,t,\ ∃\,b≥ N_t:\ 1/ _t(b)≤α )≤α . Indeed, if b≥Ntb≥ N_t and Φt(b)≥1/α′ _t(b)≥ 1/α , then Φt(Nt)≥Φt(b)≥1/α′ _t(N_t)≥ _t(b)≥ 1/α because Φt _t is decreasing. In that case the tolerance computed at level α′α is at least NtN_t, an event of probability at most α′α over all t. What a correct edge claim says. Suppose that there are no ties and that i wins each of the first 1,0001,000 records with probability 0.60.6 and each later record with probability 0.40.4. Then H0ijH_0^ij is false and the edge claim is correct in the sense of Section 2. The usual reading of a leaderboard claim is the stable case of Section 2, in which [Xt∣t−1]E[X_t _t-1] is the same constant μ at every non-inserted record and the claim is correct exactly when μ>12μ> 12. Theorems 5.3 and 5.4 assume i.i.d. records, a special case of the stable case. The simulations of Section 6 also use i.i.d. uncorrupted values. Theorem 5.1 needs neither. In the example the uncorrupted margin of Eq. (2) is Δt=0.1min(t,1,000)−0.1(t−1,000)+ _t=0.1 (t,1,000)-0.1(t-1,000)^+, positive before t=2,000t=2,000 and at most zero from then on. So the lead claim of Section 2 is correct exactly before t=2,000t=2,000, and Theorem 5.7 bounds by α the probability that the lead tolerance is nonnegative at any later time (E5 in Appendix C). Unlike H0ijH_0^ij, the event Δt≤0\ _t≤ 0\ is random and may hold at some times and not at others. The lead certificate reads “i leads j, unless more than B^t B_t of the first t records were corrupted”. The tolerance itself does respond to later records. Let ntwinn win_t and ntlossn loss_t be the numbers of wins and losses of i among the first t records. For outcomes in 0,12,1\0, 12,1\, logKt(k)=12crep(λ¯k)(ntwin−ntloss)−12κ(λ¯k)(ntwin+ntloss), K^(k)_t= 12\,c_rep( λ_k)\, (n win_t-n loss_t )- 12\,κ( λ_k)\, (n win_t+n loss_t ), (16) with κ as in Eq. (11), because log(1+λ¯/2)=(crep(λ¯)−κ(λ¯))/2 (1+ λ/2)=(c_rep( λ)-κ( λ))/2 and log(1−λ¯/2)=−(crep(λ¯)+κ(λ¯))/2 (1- λ/2)=-(c_rep( λ)+κ( λ))/2 and a tie contributes log1=0 1=0. Hence Φt(b) _t(b) and B^t B_t depend on the records only through the numbers of wins and losses so far. In the example above every Kt(k)K^(k)_t tends to zero and B^t B_t returns to −1-1. In simulation it does so at a median of 420420 records after the switch (E5 in Appendix C). The platform may also start the count at a time fixed in advance. Definition 2.1 and Theorem 5.1(a,b) then apply to the records from that time on and the claim concerns those records only. E3c in Appendix C does this for the last 120120 days of the Arena data. The lead claim is certified by Theorem 5.7 with the exponential wealth of Section 4.3. D.2 Why this is the right formulation Attacks and defenses are both counted in records, from vote-rigging and sensitivity studies of Arena (Huang et al.,, 2025; Min et al.,, 2025; Huang et al.,, 2026; Oyarhoseini et al.,, 2026) to certified defenses in machine learning, sequential decision making, distributed systems and robust statistics (Section D.3). Anytime validity and an adaptive attacker are required by how leaderboards are used. A statement valid only at one planned sample size loses its guarantee under repeated checking (Howard et al.,, 2021; Khosravi and Huo,, 2026). An attacker who sees the standings can wait and strike in one burst, which a per-step bound on corruption cannot describe (Proposition 3.1). D.3 How adjacent fields count corruption Adjacent fields measure attacks and defenses by the amount of corruption, usually a count of corrupted records or participants. Vote-rigging and sensitivity studies of Arena report the number of votes that move a model or change the top model (Huang et al.,, 2025; Min et al.,, 2025; Huang et al.,, 2026; Oyarhoseini et al.,, 2026). Certified defenses against data poisoning (altering the training data to change a trained model) approximately bound the damage that a given number of poisoned points can do (Steinhardt et al.,, 2017). Others certify predictions against a number of altered points (Levine and Feizi,, 2021; Jia et al.,, 2021). Bose et al., (2025) certify an online learner when each record is poisoned with a fixed probability and an adaptive attacker picks its value, a per-step model as in Proposition 3.1. Corruption-robust bandits, sequential decision rules whose rewards an adversary may alter, pay a price linear in the total corruption (Lykouris et al.,, 2018; Gupta et al.,, 2019). Byzantine-robust systems, which tolerate participants who behave arbitrarily, count the corrupted participants (Lamport et al.,, 1982; Blanchard et al.,, 2017). In robust statistics, strong contamination lets an adversary inspect the sample and replace a fraction of it, whereas Huber contamination mixes in outliers before sampling (Huber,, 1964; Diakonikolas and Kane,, 2023). Our classes V and P are their online counterparts, with the count NtN_t in place of the fraction. An attacker in O sees all uncorrupted values before acting, as the adversary of strong contamination does. Leaderboard inference assumes genuine votes. Chiang et al., (2024) flag anomalous Arena users with a heuristic sequential test and find that versions built on nonnegative supermartingales and e-values lacked power. Arena’s confidence intervals for Bradley–Terry ratings (Bradley and Terry,, 1952; Chiang et al.,, 2024) are valid only at a fixed sample size. So are rank confidence sets (Mogstad et al.,, 2024, sets of ranks containing a model’s true rank with a stated probability;) and inference on selected winners (Andrews et al.,, 2024). Robust rank aggregation from corrupted pairwise comparisons (Agarwal et al.,, 2020) bounds the estimation error, not the error of published claims. Anytime-valid rank sets (Khosravi and Huo,, 2026) assume uncorrupted data. They combine pairwise e-processes by closed testing (Marcus et al.,, 1976; Fischer and Ramdas,, 2024), which tests intersections of hypotheses to control the probability of any false rejection. Attack studies and certified defenses give no sequential certificate. The attack and sensitivity studies of Sections 1 and D.2 (Zhao et al.,, 2025, see also) show that attacks are real and cheap but give no guarantee. Ma et al., (2024) fabricate pairwise comparisons online, without knowledge of future data, to manipulate rank aggregation; our certificates bound how many such fabrications a published claim withstands. Certified defenses against data poisoning certify one classifier or bound a long-run average loss, not a sequence of published statements (Steinhardt et al.,, 2017; Levine and Feizi,, 2021; Bose et al.,, 2025). Corruption-robust bandits bound regret, the reward lost to the best fixed choice (Lykouris et al.,, 2018; Gupta et al.,, 2019), or the error of one final choice (Zhong et al.,, 2021; Altschuler et al.,, 2019), not errors at every time. D.4 Limitations and extensions The tolerance counts records, not attackers. One attacker can control many cheap accounts, so the tolerance should be read alongside the cost of an account. The null hypothesis treats each pair separately and ignores the transitivity (if i is better than j and j than j′j , then i is better than j′j ) that Bradley–Terry models exploit. Our replays simulate attacks, since real attack logs are not public. Validity against offline attackers when the bets change with the data remains open. So do withholding attacks, which suppress uncorrupted records. For attacks on correct claims, Proposition B.9 bounds the tolerance lost per corrupted record and the time to recover it. Empirically, each inserted loss removes about one forged-vote unit of tolerance. Each win replaced by a loss removes about two, which is one flipped-vote unit. So de-certifying a claim takes about as many inserted losses as the forged-vote tolerance, or as many replaced wins as the flipped-vote one (E6 in Appendix C).