Paper deep dive
Rank Confidence Sequences:Anytime-valid Leaderboards
Hamed Khosravi, Xiaoming Huo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 9/29/2026, 12:58:43 PM
Summary
This paper introduces 'Rank Confidence Sequences,' a statistical method for generating anytime-valid confidence intervals for model rankings on leaderboards. Unlike traditional fixed-sample methods, this approach maintains valid error guarantees (at level alpha) regardless of how many times the leaderboard is inspected or when the evaluation stops. The method uses betting e-processes to measure evidence for pairwise model comparisons and closed testing over possible model orderings to derive rank sets, top-k sets, and tiers. It handles dependent scores across models and allows for early stopping of model evaluation to save computational resources.
Entities (8)
Relation Signals (7)
Hamed Khosravi → affiliatedwith → Georgia Institute of Technology
confidence 99% · Hamed Khosravi hkhosravi7@gatech.edu ... Georgia Institute of Technology
Xiaoming Huo → affiliatedwith → Georgia Institute of Technology
confidence 99% · Xiaoming Huo huo@gatech.edu ... Georgia Institute of Technology
Rank Confidence Sequences → provides → Anytime-valid guarantees
confidence 95% · A leaderboard can be read at any time without inflating its error. With probability at least 1−α, all rank statements are correct simultaneously, at every look and under any stopping rule
Rank Confidence Sequences → uses → Closed testing
confidence 95% · The construction combines betting e-processes... with closed testing over the possible orderings of the models.
Rank Confidence Sequences → uses → Betting e-processes
confidence 95% · The construction combines betting e-processes, one for each ordered pair of models, with closed testing
Rank Confidence Sequences → enables → Early Stopping
confidence 90% · The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute.
Rank Confidence Sequences → handles → Dependent scores
confidence 90% · It allows any dependence between the models’ scores on an item.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model's rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level $\alpha$, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models' scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data.
Tags
Links
- Source: https://arxiv.org/abs/2609.32211v1
- Canonical: https://arxiv.org/abs/2609.32211v1
Trouble viewing inline? Open PDF directly →
Full Text
177,218 characters extracted from source content.
Expand or collapse full text
Rank Confidence Sequences: Anytime-Valid Leaderboards Hamed Khosravi Xiaoming Huo Abstract Leaderboards rank models by their average scores on benchmark items, and they are consulted repeatedly while the evaluation is still running. Existing confidence intervals for a model’s rank control their error rate only if they are computed once, after a number of items chosen in advance. If they are recomputed as results arrive, and the evaluation stops once they look decisive, their error rate exceeds its nominal level. Anytime-valid methods keep their guarantees at all sample sizes simultaneously and hence under any stopping rule. They exist for the accuracy of one model, for one pair of models and for the set of models that may be best. For pairwise battles they also give ranks. None gives ranks when all models are scored on the same items, which makes their scores dependent. We construct rank confidence sequences: for every model, a set of ranks that contains its true rank, simultaneously for all models and at all times, at a chosen error level α, in finite samples. The construction combines betting e-processes, one for each ordered pair of models, with closed testing over the possible orderings of the models. It allows any dependence between the models’ scores on an item. The method has two advantages. A leaderboard can be inspected after every item without inflating its error rate. The evaluation of each model can stop as soon as the question asked about it is answered, which saves compute. When results are examined only once, halfway through or later, little power is lost relative to fixed-sample methods. The paper quantifies these advantages in simulations and on public leaderboard data. Hamed Khosravi hkhosravi7@gatech.edu Xiaoming Huo huo@gatech.edu H. Milton Stewart School of Industrial and Systems Engineering Georgia Institute of Technology, Atlanta, GA 30332, USA 1 Introduction The problem. Confidence intervals for leaderboard ranks are designed to be computed once, but leaderboards are consulted many times while the evaluation is still running. A benchmark leaderboard scores each of M models on a common set of test items, such as exam questions, and ranks the models by their average scores. The averages come from finitely many items, so the ranking is uncertain. Several methods therefore attach a confidence interval to each model’s rank (Mogstad et al., 2024; Al Mohamad et al., 2022; Neuhof & Benjamini, 2024). At confidence level 95%95\%, these intervals contain the true ranks of all models simultaneously with probability at least 95%95\%, some only asymptotically. The guarantee assumes a specific protocol. The number of items is chosen before any results are seen. The intervals are then computed once, after all of those items have been scored. We call such intervals fixed-sample. Repeated looks. Inspecting a leaderboard repeatedly as results arrive breaks the fixed-sample guarantee. In practice, models are scored item by item or batch by batch, and the standings are examined along the way. We call each examination a look. Because evaluation is expensive (Ghosh et al., 2026), it is cut short when the ranking seems clear and extended only when two models seem close. Each look, however, is another chance for a fixed-sample interval to be wrong. Stopping when the intervals seem decisive also tends to stop exactly when one of them is wrong. The probability that at least one look produces a false statement therefore exceeds the nominal level. This effect of repeated significance tests has long been known, and Armitage et al. (1969) computed its size. In our simulation with six equally accurate models examined 200200 times, fixed-sample rank sets at nominal level 5%5\% make at least one false statement in 30%30\% to 51%51\% of runs (Section 5, experiment E1). The fixed-sample guarantee is not wrong: it covers one look, not the way leaderboards are used. What is needed. A leaderboard that is monitored needs rank intervals that are anytime-valid. Fix a level α∈(0,1)α∈(0,1) in advance. A procedure is anytime-valid if the probability that it is wrong at any look, over the whole evaluation, is at most α. Equivalently, its error guarantee holds at all sample sizes simultaneously. It then holds at whatever sample size the evaluation stops, even when the decision to stop depends on the data seen so far. For a single mean, intervals with this property are called confidence sequences (Howard et al., 2021; Waudby-Smith & Ramdas, 2024). Two further features of leaderboards matter. First, the statement must cover the ranks of all models at once, so the method must correct for making many comparisons. Second, all models are scored on the same items, so their scores are dependent. An item that is easy for one model tends to be easy for the others, and the strength of this dependence is unknown. What exists. To our knowledge, no existing method gives anytime-valid confidence sets for the ranks of all models when the models are scored on the same items. Existing rank intervals are fixed-sample. They certify pairwise comparisons of the form “model j is better than model l” with a multiple-testing procedure and convert them into rank intervals (Mogstad et al., 2024; Al Mohamad et al., 2022; Neuhof & Benjamini, 2026a; Neuhof & Benjamini, 2026b). The anytime-valid methods that exist answer other questions. Some track the accuracy of one model (Zhou et al., 2026; Hsu & Shekhar, 2026), and one compares a single pair of models (Kotawala, 2026). Arnold et al. (2026) certify which models may be the best, a statement about rank one only. Gu et al. (2026) give anytime-valid rank sets for a different kind of data, pairwise battles. In a battle, two models answer the same prompt and a judge picks the better answer, with the pair chosen adaptively. Gu et al. (2026) correct for multiplicity by Bonferroni, testing each of the M(M−1)M(M-1) ordered pairs at level α/M(M−1)α/\M(M-1)\. The logical relations among the pairs are used only afterwards, to extend certified comparisons by transitivity. Group-sequential designs with Pocock boundaries (Pocock, 1977; Arviv et al., 2026) stop pairwise comparisons early, but only at looks scheduled in advance and without a correction across pairs. A side-by-side comparison of these methods is in Appendix F. What we do. We construct rank confidence sequences. For every model, a rank confidence sequence gives a set of ranks that contains the model’s true rank. This holds for all models at once and at every look, with probability at least 1−α1-α. A model’s ability is its mean score. Its true rank is one plus the number of models with strictly higher ability (Section 2). The ability can be the model’s average over a fixed benchmark. It can also be the model’s expected score on items drawn from a population. The guarantee holds in finite samples, with no asymptotic approximation. It requires per-item scores in [0,1][0,1]. On a fixed benchmark it also requires that the items be evaluated in a random order, which the evaluator controls. It makes no assumption about how the scores of different models are correlated. The method keeps a list of candidate orderings of the models, rankings in which ties are allowed, and reports only what all surviving orderings agree on. For each pair of models, it measures the evidence that one scores higher than the other as the wealth of an imaginary gambler betting on it. It discards an ordering once the average of these wealths over the comparisons the ordering denies has grown large. Section 3 gives the details. Contributions. 1. A leaderboard can be read at any time without inflating its error. With probability at least 1−α1-α, all rank statements are correct simultaneously, at every look and under any stopping rule (Theorem 4.2; Section 5, E1 and E2). 2. A leaderboard can state how much of its ranking the data support. Besides certified comparisons and rank sets, the report gives certified top-k sets and tiers. A top-k statement certifies that a model is among the k best, or that it is not. Tiers s=1,2,…s=1,2,… group the models by certified comparisons. A model in tier s has at least s−1s-1 models certified above it, so its true rank is s or worse (Theorems 4.3 and 4.4; Section 5, E2). 3. Evaluation compute is saved. The evaluation of a model can stop as soon as the question asked about it is answered, for example whether it is in the top three. The guarantee survives even though the decision to stop depends on the observed scores (Proposition 4.5; Section 5, E3). 4. The price of these guarantees is small. From the same pairwise evidence, the method certifies every comparison that a Bonferroni correction over the pairs certifies, and its exact version can certify more (Section 4.1). At a pre-planned look at halfway, it certifies about as many comparisons as the strongest fixed-sample procedure (Section 5, E4). 2 Problem setting Models, abilities, ranks. Each of M≥2M≥ 2 models has one scalar ability, its mean benchmark score, and its rank is one plus the number of models with strictly larger ability. Write [M]=1,…,M[M]=\1,…,M\ for the set of models and θ=(θ1,…,θM)∈ℝMθ=( _1,…, _M) ^M for the vector of abilities, where ℝR is the set of real numbers. Write #A\#A, or |A||A|, for the number of elements of a finite set A. The true rank of model j is Rj(θ)=1+#l∈[M]:θl>θj,R_j(θ)=1+\#\l∈[M]: _l> _j\, (1) abbreviated RjR_j when θ is clear from context. Rank 11 is best, and tied models share a rank. Two models tied for first both have rank 11. Let =(j,l)∈[M]2:j≠lP=\(j,l)∈[M]^2:j≠ l\ be the ordered pairs. The null hypothesis for (j,l)∈(j,l) is Hjl:θj≤θlH_jl: _j≤ _l, that j is not better than l. Rejecting HjlH_jl certifies θj>θl _j> _l, written j≻lj l. Data. All models are scored on the same items, and we assume nothing about how their scores are correlated. Item i yields the score vector xi=(xi1,…,xiM)∈[0,1]Mx_i=(x_i1,…,x_iM)∈[0,1]^M, one score per model. The most common case is binary correctness, where xij=1x_ij=1 if model j answers item i correctly and xij=0x_ij=0 otherwise. For binary scores, θj _j is the model’s accuracy. More generally, scores may take any value in [0,1][0,1]. Two sampling models. Our main setting is a fixed benchmark evaluated in random order, and the guarantee holds equally when the items are a sample from a population. Time t=0,1,2,…t=0,1,2,… counts the items revealed so far, and Xt=(Xt1,…,XtM)X_t=(X_t1,…,X_tM) is the score vector of the ttth item revealed. (F) Finite benchmark. The benchmark consists of N items and is a fixed matrix x∈[0,1]N×Mx∈[0,1]^N× M, where xijx_ij is the score of model j on item i. The ability θj=N−1∑i=1Nxij _j=N^-1 _i=1^Nx_ij is the mean score of model j on the whole benchmark. One permutation Π of [N]=1,…,N[N]=\1,…,N\ is drawn uniformly at random and shared by all models. At time t≤Nt≤ N, item Π(t) (t) is revealed. Its score vector is Xt=xΠ(t)X_t=x_ (t), the row Π(t) (t) of x. The only randomness in the data is the order of evaluation. (S) Superpopulation. X1,X2,…X_1,X_2,… are independent and identically distributed (i.i.d.) draws from a distribution on [0,1]M[0,1]^M, and θj=[X1j] _j=E[X_1j], where E denotes expectation. The joint law of X1X_1 across models is unrestricted. Under both sampling models, ℱtF_t denotes the information available after t items. Formally, write σ(⋅)σ(·) for the σ-algebra generated by the listed random variables. Then ℱt=σ(Π(1),…,Π(t),ξ)F_t=σ( (1),…, (t),ξ) under (F) and ℱt=σ(X1,…,Xt,ξ)F_t=σ(X_1,…,X_t,ξ) under (S). Here ξ is an auxiliary random variable that lets the analyst randomize decisions. It is independent of Π under (F) and of (X1,X2,…)(X_1,X_2,…) under (S). The increasing family (ℱt)t≥0(F_t)_t≥ 0 is called the filtration. A quantity is adapted if its value at time t is determined by ℱtF_t. Equivalently, it can be computed from what has been observed by time t. Goal. We want rank sets that cover the true ranks of all models at once, at every time, not at one pre-chosen sample size. Fix a level α∈(0,1)α∈(0,1) and write ℙθP_θ for probability under a data-generating law whose ability vector is θ. A rank confidence sequence at level α is a family of adapted random sets ℛj,t⊆[M]R_j,t [M], one per model j and time t, with ℙθ(∀t≥0,∀j∈[M]:Rj(θ)∈ℛj,t)≥1−αP_θ(∀ t≥ 0,\ ∀ j∈[M]:\ R_j(θ) _j,t)≥ 1-α for every data-generating law. 3 Rank confidence sequences Overview and guarantee. After every item, the method keeps the orderings of the models, ties allowed, that the data have not yet contradicted. It reports only statements on which all surviving orderings agree. It repeats three steps after each item t=1,2,…t=1,2,…. Step 1 updates a nonnegative measure of evidence for each ordered pair of models. Step 2 eliminates every ordering against which enough evidence has accumulated. Step 3 reports the certified dominances j≻lj l, rank sets, top-k sets and tiers on which the remaining orderings agree. An optional Step 4 retires models whose evaluation can stop. With probability at least 1−α1-α, the true ordering, the one given by the true abilities, is never eliminated. Every report is agreed on by all surviving orderings, the true one included, so with the same probability every statement in every report is true at all times simultaneously. The user may therefore look after every item and stop by any rule. Inputs. The method needs the scores Xtj∈[0,1]X_tj∈[0,1] of every model j∈[M]j∈[M] on a common sequence of items t=1,2,…t=1,2,…, and two choices made before any data are seen: the level α∈(0,1)α∈(0,1) and a finite grid Λ⊂(0,1) ⊂(0,1) of betting fractions (Step 1), by default Λ=0.03,0.06,0.12,0.25,0.5 =\0.03,0.06,0.12,0.25,0.5\. The grid affects how fast evidence accumulates and never the guarantee. Under (F), a benchmark of N items, the user also draws one uniformly random order of the items from a recorded seed before the evaluation, and evaluates every model in that order. Then Xt=(Xt1,…,XtM)X_t=(X_t1,…,X_tM) is the score vector of the ttth item in that order. Under (S), the item stream is open-ended and i.i.d., so we set N=∞N=∞. Step 1: update the pairwise wealths. For each ordered pair (j,l)∈(j,l) the method maintains a wealth Etjl≥0E^jl_t≥ 0. After t items it measures the evidence that model j is better than model l, or equivalently the evidence against Hjl:θj≤θlH_jl: _j≤ _l. It is the average capital of |Λ|| | gamblers, each starting with one unit and, on every item, staking a fixed fraction λ∈Λλ∈ of their capital on model j outscoring model l. Formally, for each λ∈Λλ∈ put Eλ,0jl=1E^jl_λ,0=1 and, on item t with paired difference Ztjl=Xtj−XtlZ^jl_t=X_tj-X_tl and offset btjlb^jl_t as given below, Eλ,tjl=Eλ,t−1jl(1+λZtjl−btjl1+btjl),Etjl=1|Λ|∑λ∈ΛEλ,tjl,E^jl_λ,t=E^jl_λ,t-1 (1+λ\, Z^jl_t-b^jl_t1+b^jl_t ), E^jl_t= 1| | _λ∈ E^jl_λ,t, (2) where the offset btjlb^jl_t, computed from items 1,…,t−11,…,t-1 only, bounds the conditional mean of ZtjlZ^jl_t under HjlH_jl: btjl=0under (S),max−0.99,min1,−St−1jlN−t+1under (F),St−1jl=∑s<tZsjl.b^jl_t= cases0&under (S),\\[2.0pt] \-0.99,\ \1,\ -S^jl_t-1N-t+1 \ \&under (F), cases S^jl_t-1= _s<tZ^jl_s. (3) Here St−1jlS^jl_t-1 is the cumulative paired difference over the first t−1t-1 items. Under HjlH_jl, [Ztjl∣ℱt−1]≤btjlE[Z^jl_t _t-1]≤ b^jl_t. For i.i.d. items this follows from [Ztjl]=θj−θl≤0E[Z^jl_t]= _j- _l≤ 0. On a finite benchmark the next item is drawn uniformly from the N−t+1N-t+1 items not yet revealed, whose differences sum to N(θj−θl)−St−1jl≤−St−1jlN( _j- _l)-S^jl_t-1≤-S^jl_t-1. The truncation to [−0.99,1][-0.99,1] keeps each factor in (2) nonnegative and well defined. Thus, under HjlH_jl, each factor has conditional expectation at most one, so the bets are fair or unfavorable and the wealth cannot grow in expectation. By Ville’s inequality (Ville, 1939), its probability of ever reaching 1/α1/α is at most α. Such a wealth is an e-process (Ramdas et al., 2023), a nonnegative process whose expected value stays at most one under any stopping rule. Its reciprocal acts as a p-value that remains valid when it is checked after every item. When θj>θl _j> _l, the wealth grows exponentially if, for some λ∈Λλ∈ , the factor in (2) has a positive expected logarithm. The best bet size depends on the unknown gap θj−θl _j- _l, and averaging over a grid avoids relying on a single one. The pairs (j,l)(j,l) and (l,j)(l,j) have separate wealths. Step 2: eliminate orderings and certify dominances. An ordering of the models is eliminated once the wealths that should stay small if that ordering were true average to 1/α1/α or more. The statement “j is better than l” is certified once every ordering that says otherwise has been eliminated. This is the principle of closed testing (Marcus et al., 1976), with one test per ordering. A weak order W on [M][M] is a ranking of the models in which ties are allowed. Write j≺Wlj _Wl if W ranks l strictly above j, and j∼Wlj _Wl if it ties them. Write j⪯Wlj _Wl when W ranks l at least as high as j, including a tie. Formally ⪯W _W is a reflexive, transitive and total relation on [M][M]. We represent W by a vector of levels (v1,…,vM)(v_1,…,v_M), with j⪯Wlj _Wl if and only if vj≤vlv_j≤ v_l. Let W be the set of all weak orders on [M][M]. It has 1313 elements for M=3M=3 and 545,835545,835 for M=8M=8. The abilities determine one of them, the true ordering W(θ)W(θ). It satisfies j⪯W(θ)lj _W(θ)l if and only if θj≤θl _j≤ _l. The rank (1) depends on θ only through the true ordering. Under W the rank of model j is Rj(W)=1+#l∈[M]:j≺WlR_j(W)=1+\#\l∈[M]:j _Wl\. If W were the true ordering, the hypotheses HjlH_jl that hold would be exactly those of the pairs in T(W)=(j,l)∈:j⪯WlT(W)=\(j,l) :\ j _Wl\. By Step 1, none of the wealths of these pairs would be likely to become large. The evidence against W is therefore the average of these wealths. Let EtWE^W_t denote that average. We eliminate W permanently when EtWE^W_t reaches 1/α1/α, and let tC_t be the set of surviving orderings: EtW=1|T(W)|∑(j,l)∈T(W)Etjl,t=W∈:maxs≤tEsW<1α.E^W_t= 1|T(W)| _(j,l)∈ T(W)E^jl_t, _t= \W :\ _s≤ tE^W_s< 1α \. (4) An average of e-processes on the same items is again an e-process, whatever their dependence, as for e-values (Vovk & Wang, 2021). The set tC_t of surviving orderings is therefore a confidence set for the true ordering that is valid at all times at once, ℙθ(∀t≥0:W(θ)∈t)≥1−αP_θ(∀ t≥ 0:\ W(θ) _t)≥ 1-α (Theorem 4.2). The certified dominances and the rank sets are what all surviving orderings agree on, t=(j,l)∈:l≺Wjfor all W∈t,ℛj,t=Rj(W):W∈t.D_t= \(j,l) :\ l _Wj\ for all W _t \, _j,t= \R_j(W):\ W _t \. (5) Because elimination is permanent, tC_t can only shrink and tD_t can only grow as t increases. Computing Step 2. Step 2 can be computed in three ways. Algorithm 1 enumerates the surviving weak orders. It alone gives the exact rank sets and is practical up to about M=8M=8. Algorithm 2 is a polynomial-time shortcut for any M that may certify fewer comparisons. Algorithm 3 computes the same tD_t by integer programming at larger M. It solved every instance in our experiments. The shortcut certifies (j,l)(j,l) once the pooled statistic Btjl=Etjl+∑m∈[M]∖j,lminEtjm,EtmlB^jl_t=E^jl_t+ _m∈[M] \j,l\ \E^jm_t,E^ml_t\ reaches M(M−1)/αM(M-1)/α. Each term minEtjm,Etml \E^jm_t,E^ml_t\ is indirect evidence that θj>θl _j> _l through a third model m. It is large only if θj>θm _j> _m and θm>θl _m> _l are both well supported. The shortcut certifies a subset of tD_t and at least what a Bonferroni correction over the M(M−1)M(M-1) pairs certifies, namely every pair whose wealth has reached M(M−1)/αM(M-1)/α, a rule we call e-Bonferroni. The exact test, computed by Algorithm 1 or 3, can certify more, because testing whole orderings uses the logical links among the pairwise hypotheses. For example, HjlH_jl and HlmH_lm together imply HjmH_jm, so combinations of true and false hypotheses that no ordering allows need not be guarded against. Step 3: report. Except for the exact rank sets obtained by enumeration, every reported quantity is a function of the certified set tD_t, so all of them are correct whenever tD_t contains no false dominance. The subscript t is dropped below when the time is fixed. Dominances. Each (j,l)∈t(j,l) _t is reported as “j is better than l”, written j≻lj l. Ranks. The rank interval of model j is [Lj,t,Uj,t][L_j,t,U_j,t] with Lj,t=1+#l:(l,j)∈t,Uj,t=M−#l:(j,l)∈t.L_j,t=1+\#\l:(l,j) _t\, U_j,t=M-\#\l:(j,l) _t\. (6) Enumeration can instead report the exact rank set ℛj,t⊆[Lj,t,Uj,t]R_j,t [L_j,t,U_j,t], which may be smaller. Top-k sets. For k∈[M]k∈[M], model j is certified inside the top k if Uj≤kU_j≤ k and outside it if Lj>kL_j>k. Ties can put more than k models inside. Tiers. A chain above j of length h is a sequence of models l1,…,lhl_1,…,l_h such that (l1,l2)(l_1,l_2), …, (lh−1,lh)(l_h-1,l_h) and (lh,j)(l_h,j) all belong to D. Let h(j)h(j) be the greatest such length, with h(j)=0h(j)=0 if no model is certified better than j. The tier of j is tier(j)=1+h(j)tier(j)=1+h(j), and tier s is the set j:tier(j)=s\j:tier(j)=s\. Tiers are computed by the recursion tier(j)=1+maxtier(l):(l,j)∈tier(j)=1+ \tier(l):(l,j) \, with tier(j)=1tier(j)=1 if the set is empty. A model in tier s has true rank s or worse. If D ever contains a cycle, such as j≻lj l and l≻jl j, or no ordering survives, an error has occurred, which happens with probability at most α. Report it as such. Step 4 (optional): retire models. A model may be dropped from further evaluation at any time, by any rule that uses only the scores observed so far, and the guarantee still holds. Let At⊆[M]A_t [M] be the set of models scored on item t, with At+1⊆AtA_t+1 A_t, chosen from the scores observed on items 1,…,t−11,…,t-1. The wealth of (j,l)(j,l) is updated by (2) while both j and l belong to AtA_t. Afterwards it is frozen, Etjl=Et−1jlE^jl_t=E^jl_t-1, and continues to be used in Steps 2 and 3. Natural rules retire j once its top-k status is certified, once every pair involving j is certified in one direction, or once Uj−LjU_j-L_j is below a target. The item order is not affected by retirement, and a retired model is not brought back. Scope and cost. The model set must be fixed in advance, items must not be chosen adaptively, and every model that has not been retired must have a score on each item. Pairwise-battle data are out of scope. The per-item update is negligible next to running the models. It costs O(M2)O(M^2) per item, 25μ25\, for 2020 models on one CPU core. Appendix A states Algorithms 1–3 with implementation details and costs. Appendix D gives the limits of the guarantee. 4 What can be established, and why it matters The guarantee of Section 3 follows from three results, one per subsection, all proved in Appendix C. 4.1 All certified dominances are correct, at all times What needs a guarantee. Every report except the exact rank sets is computed from the certified set tD_t, so the central requirement is that, with probability at least 1−α1-α, no tD_t at any time t contains a false dominance, a pair (j,l)(j,l) with θj≤θl _j≤ _l. A pair with θj>θl _j> _l is a true dominance. We can guarantee this over all pairs and times at once, even though the wealths share items and may be dependent in unknown ways. The guarantee starts from one property of the pairwise wealths, that of an e-process (Ramdas et al., 2023). Identify a hypothesis H with the set of data-generating laws it allows and let PE_P denote expectation under the law P. A stopping time is a random time τ such that whether τ=tτ=t can be decided from the information ℱtF_t available at time t. Formally, an e-process for H is a nonnegative adapted process (Et)t≥0(E_t)_t≥ 0 with P[Eτ]≤1E_P[E_τ]≤ 1 for every P∈HP∈ H and every bounded stopping time τ. An e-process satisfies Ville’s inequality: P(∃t≥0:Et≥1/α)≤αP(∃ t≥ 0:\ E_t≥ 1/α)≤α for every P∈HP∈ H, where P(⋅)P(·) is probability under the law P. This is the counterpart of Markov’s inequality that holds over all times at once. Under both sampling models every wealth of Section 3 is an e-process for its hypothesis HjlH_jl, whatever the dependence between models. This is the betting construction of Waudby-Smith & Ramdas (2024). Because all models share one item order, all wealths are e-processes for the same filtration. We record this as an assumption. Assumption 4.1. There is a filtration (ℱt)(F_t) and, for each (j,l)∈(j,l) , a nonnegative adapted process (Etjl)(E^jl_t) with E0jl=1E^jl_0=1 that is an e-process for HjlH_jl with respect to (ℱt)(F_t) under every data-generating law for which θj≤θl _j≤ _l. No condition is placed on the joint behavior of the M(M−1)M(M-1) processes. Result. With probability at least 1−α1-α the true ordering is never eliminated, and as long as it survives no certified dominance is false. Write θ⋆θ for the true ability vector and W⋆=W(θ⋆)W =W(θ ) for the true ordering. Here tC_t and tD_t are the surviving and certified sets of (4) and (5). Theorem 4.2 (Validity). Under Assumption 4.1, for every data-generating law with parameter θ⋆θ and W⋆=W(θ⋆)W =W(θ ), and with ℙP denoting probability under that law, ℙ(∀t≥0:W⋆∈t)≥1−α.P (∀ t≥ 0:\ W _t )≥ 1-α. (7) Consequently the family-wise error rate, the probability of certifying any false dominance, is controlled uniformly over time, ℙ(∃t≥0,∃(j,l)∈t:θj⋆≤θl⋆)≤α.P (∃ t≥ 0,\ ∃(j,l) _t:\ θ _j≤θ _l )≤α. (8) Both statements also hold when t is replaced by any ℕ0N_0-valued random time, stopping or otherwise, where ℕ0=0,1,2,…N_0=\0,1,2,…\. The proof applies Ville’s inequality to EtW⋆E^W _t, an average of wealths of true hypotheses. While W⋆W survives, the certified set is a strict partial order (no pair in both directions, closed under transitivity), so the report never contradicts itself. Relation to existing results. Theorem 4.2 combines closed testing over weak orders with averaging of e-processes. We use the partitioning principle of Finner & Strassburger (2002), where the orderings partition the parameter space and one test is carried out per cell, as in the fixed-sample rank intervals of Al Mohamad et al. (2021). Shaffer (1986) shows that only sets of hypotheses that can be true together need be tested. Here these are exactly the sets T(W)T(W), one per weak order. Averaging e-values (nonnegative statistics with expectation at most one under the null), the fixed-time counterpart of e-processes, remains valid under arbitrary dependence (Vovk & Wang, 2021; Wang, 2025), while products generally require independence, which shared items rule out. Our result differs from the two closest ones as follows. Arnold et al. (2026) average e-processes and close over all subsets of models, which certifies the set of models that may be best and no ranks. Hartog & Lei (2025) close over all sets of hypotheses, the full closure, which controls the family-wise error for arbitrary hypotheses but yields no rank statements. We test one averaged e-process per weak order, which yields rank sets. Nothing is lost relative to standard corrections. On the same wealths, the exact test certifies every pair that the full closure certifies, and the shortcut every e-Bonferroni rejection used by Gu et al. (2026). Appendix B.3 states both results. 4.2 Ranks, top-k sets and tiers inherit the guarantee What needs a guarantee. Users read ranks, top-k sets and tiers, which are built from the dominances. These reported quantities must also be correct without costing additional error. To cover the exact test, the shortcut and e-Bonferroni at once, call a family (t)t≥0(D_t)_t≥ 0 of random subsets of P a certified set if its good event =∀t≥0,∀(j,l)∈t:θj⋆>θl⋆G=\∀ t≥ 0,\ ∀(j,l) _t:\ θ _j>θ _l\ has probability at least 1−α1-α. All three are certified sets. Taking the transitive closure adds no error. Let ¯t D_t be the transitive closure of tD_t, which adds (j,m)(j,m) whenever (j,l)(j,l) and (l,m)(l,m) are present, repeatedly. On G, the transitive closure contains only true dominances. True dominances form a strict partial order, so the closure is certified on the same event. Result. Rank intervals, top-k sets and tiers are correct on G, and exact rank sets while W⋆W survives. Theorem 4.3 (Rank-set duality). (a) Projection. Under Assumption 4.1, the rank sets ℛj,tR_j,t of (5) satisfy ℙ(∀t,∀j:Rj(θ⋆)∈ℛj,t)≥1−αP(∀ t,∀ j:\ R_j(θ ) _j,t)≥ 1-α. (b) Counting. Let (t)(D_t) be any certified set and compute Lj,tL_j,t and Uj,tU_j,t from it by (6). On G, Lj,t≤Rj(θ⋆)≤Uj,tL_j,t≤ R_j(θ )≤ U_j,t for all j and all t, and the intervals only improve if tD_t is replaced by ¯t D_t. (c) Comparison. For tD_t from (5), ℛj,t⊆[Lj,t,Uj,t]R_j,t [L_j,t,U_j,t], and the inclusion is strict for some wealth configurations. By part (b), on G every top-k certificate of Step 3 is correct, for all j, k and t simultaneously. Theorem 4.4 (Tier guarantee). Let (t)(D_t) be a certified set and let tiert(j)tier_t(j) be the tier of model j computed from ¯t D_t as in Step 3 of Section 3. On G, simultaneously for all j and all t, Rj(θ⋆)≥Lj,t≥tiert(j),R_j(θ )\ ≥\ L_j,t\ ≥\ tier_t(j), (9) where Lj,tL_j,t is computed from ¯t D_t, and the number of tiers is at most the number of distinct values among θ1⋆,…,θM⋆θ _1,…,θ _M. On G, if t⊆t+1D_t _t+1 then tiert(j)≤tiert+1(j)tier_t(j) _t+1(j), so a model’s tier index never decreases as evidence accrues. The guarantee is only (9). A lower tier index does not imply a certified pairwise dominance, and tier 11 also holds every model with no certified comparison. Relation to existing results. The counting part of Theorem 4.3 comes from Al Mohamad et al. (2022) and Mogstad et al. (2024), and Gu et al. (2026) apply it at all times. They also define the same tiers without a guarantee, which Theorem 4.4 adds. Appendix B.4 gives more on tiers. 4.3 Evaluation can stop early, model by model What needs a guarantee. Retiring a model, that is, dropping it from further evaluation, is data-dependent. Once a model stops, its future scores are not observed and every pair involving it has its wealth frozen. The guarantee must remain valid under both effects for any rule based only on observed scores. After retirement the analyst observes less information than ℱtF_t, which contains the scores of all models on all revealed items. Result. Freezing each wealth at the retirement of either of its models preserves Assumption 4.1, and with it every result that rests on that assumption. Proposition 4.5 (Retirement preserves validity). Under (F) or (S), let the item order in (F) be drawn before evaluation and remain unaffected by later decisions. At each time t let At⊆[M]A_t [M] be the set of models scored on item t, chosen as a function of the scores observed on items 1,…,t−11,…,t-1 and of ξ, with A1=[M]A_1=[M], At+1⊆AtA_t+1 A_t, and AN+1=∅A_N+1= under (F). Model j is retired at σj=supt:j∈At _j= \t:j∈ A_t\ (σj=∞ _j=∞ if never), so that item σj _j is included in its record, and retirement is permanent under both sampling models. Let ⋅1\·\ denote the indicator of the event in braces and s∧t=mins,ts t= \s,t\, and let ℱtobs=σ(ξ, 1j∈As, 1j∈AsXsj:s≤t,j∈[M])F^obs_t=σ (ξ,\ 1\j∈ A_s\,\ 1\j∈ A_s\X_sj:\ s≤ t,\ j∈[M] ) be the observed-data filtration, generated by which models were scored and the scores observed. Call a bet predictable for a filtration if the bet on item t is determined by the information at time t−1t-1. Then ℱtobs⊆ℱtF^obs_t _t, and each σj _j is an (ℱtobs)(F^obs_t)-stopping time and hence an (ℱt)(F_t)-stopping time. Replacing each EtjlE^jl_t by Et∧σj∧σljlE^jl_t _j _l, with any (ℱtobs)(F^obs_t)-predictable bets, preserves Assumption 4.1 and every result resting on it. Relation to existing results. Retirement times and bets chosen from the observed scores alone are also stopping times and predictable bets for (ℱt)(F_t). All frozen wealths therefore remain e-processes for one common filtration, while the procedure never needs an unobserved score. Freezing at such times matters. Tavyrikov et al. (2026) show that averaging running maxima, which freezes each e-process at its best past value, need not control the family-wise error. Existing sequential evaluation methods stop evaluating one model (Zhou et al., 2026; Hsu & Shekhar, 2026) or one pair at a time (Kotawala, 2026; Gu et al., 2026; Arviv et al., 2026). Ours stops models one by one inside a simultaneous guarantee over all ranks. Appendix D says when a frozen pair may be updated again. 5 Experiments Four experiments, E1 to E4, test monitoring, certification on real data, compute savings and the price of anytime validity. The real-data experiments use per-item results of 395395 large language models (LLMs) from the Open LLM Leaderboard (Maia Polo et al., 2024; Beeching et al., 2023). Setups are in Appendix E, with further experiments on cross-model dependence and on what the multiplicity correction and the bet contribute. E1. Only fixed-sample rank sets break under monitoring. With six tied models checked at 200200 looks and nominal level 5%5\%, fixed-sample rank sets err in 30%30\% to 51%51\% of runs and our exact test in 1.6%1.6\% (Figure 1a). The models answer 20002000 simulated i.i.d. items, with equally spaced looks. The fixed-sample rank sets are the standard construction, Holm’s correction (Holm, 1979) over paired tests of all M(M−1)M(M-1) ordered pairs, converted into rank intervals by (6). They err in 30%30\% of runs with the exact McNemar test and 51%51\% with the paired z-test. With tied pairs at ranks 1 and 3 the rates are 9.7%9.7\%, 18%18\% and ours 0.3%0.3\%. With no exact ties but gaps of 0.0040.004, they are 4.6%4.6\%, 11.7%11.7\% and 0.2%0.2\%. At a single pre-planned look both fixed-sample procedures are valid, at 3.3%3.3\% and 3.6%3.6\% with all six tied, so the inflation is caused by monitoring alone and not by the normal approximation. Figure 1: Probability that some report is wrong against the number of looks, α=0.05α=0.05. Most fixed-sample curves (dashed) exceed α as looks accumulate. E-process curves (solid) stay below α. Both axes use a log scale: tick labels are actual values a factor of ten apart, except 0. E2. The data support only part of the ranking early. Figure 2 follows the twenty highest-scoring models across six benchmarks, while Table 1 summarizes the halfway and final results. On MMLU, halfway through the benchmark, 50%50\% of the 190190 true dominances, the pairs whose order on the full benchmark is strict, are certified and the twenty models form 3.03.0 certified tiers. HellaSwag is similar, while the other four benchmarks separate fewer pairs at halfway. The difficulty comes from choosing twenty models near the top. When twenty models are spread across the leaderboard, 91%91\% of their true dominances on MMLU are already certified halfway through. We also used the top eight models. Of all 900900 runs, one had a wrong report, far below the 5%5\% level. E3. Model-by-model stopping saves evaluation. The top twenty are chosen with the full benchmark in hand, which an operator cannot do, so we repeat on all 395395 released models with no further selection. Most are far from the top-33 boundary, so their evaluation can stop early. Certifying top-33 membership for all 395395 uses 6.2%6.2\% of the full MMLU evaluation, against 31%31\% for the twenty highest-scoring alone. In absolute terms the 395395-model run costs about four times as much. In simulation, retiring each model once its own top-33 question is answered costs 22%22\% against 84%84\% for stopping every model at the same moment, and the cost barely depends on k. The full leaderboard is also better resolved, with 86%86\% of the true dominances on MMLU certified halfway and 99.6%99.6\% by the end (Table 1). No report was wrong in any of these runs. Table 1: True dominances certified, as a percentage of those that exist, and mean tiers at halfway. Ours checked at every 1%1\%, 5050 item orders. No report was wrong in any of these runs. Top twenty (179179–190190 dominances) All 395395 models (≈77,700≈ 77,700) Benchmark half end tiers half end tiers MMLU 50%50\% 99%99\% 3.03.0 86%86\% 99.6%99.6\% 21.221.2 HellaSwag 49%49\% 96%96\% 3.23.2 77%77\% 99.2%99.2\% 20.420.4 GSM8K 22%22\% 96%96\% 2.92.9 78%78\% 98.9%98.9\% 10.810.8 WinoGrande 2%2\% 85%85\% 2.02.0 26%26\% 95.5%95.5\% 4.24.2 ARC-Challenge 0%0\% 82%82\% 1.11.1 48%48\% 97.0%97.0\% 6.86.8 TruthfulQA 17%17\% 82%82\% 2.02.0 55%55\% 97.0%97.0\% 6.16.1 Figure 2: Twenty highest-scoring models, 5050 item orders: fraction of true dominances certified against fraction evaluated. Ours (Algorithm 2) checked at every 1%1\%, Holm and Romano–Wolf at one look. Each panel gives the mean tiers at halfway, ours against Romano–Wolf. E4. Anytime validity costs little at a pre-planned look. At a pre-planned look at halfway, ours matches or exceeds the strongest fixed-sample procedure, the bootstrap step-down rank sets of Mogstad et al. (2024), based on Romano & Wolf (2005), on five of the six benchmarks and trails by four percentage points on TruthfulQA (Figure 2). Beyond halfway the offset of (F), which uses the finiteness of the benchmark, increasingly favors ours, so the later separation should not be read as a pure power comparison. 6 Limitations and conclusion Limitations. At an early pre-planned look, fixed-sample procedures certify more. Models arriving over time (Fischer & Ramdas, 2024) and adaptive item selection (Zhou et al., 2026) are not covered. Validity covers the benchmark as given. Contamination and transfer to other tasks lie outside it. Conclusion. Rank confidence sequences give shared-item leaderboards rank statements valid for all models at all times. In our experiments they stayed valid under monitoring where fixed-sample rank sets did not, matched or exceeded them at a pre-planned look at halfway on five of six benchmarks, and settled every model’s top-33 status with 6.2%6.2\% of a full MMLU evaluation. The code is available at https://github.com/HamedKhosravi99/rank-confidence-sequences-supplement. References Al Mohamad et al. (2021) Diaa Al Mohamad, Erik van Zwet, Aldo Solari, and Jelle Goeman. Simultaneous confidence intervals for ranks using the partitioning principle. Electronic Journal of Statistics, 15(1):2608–2646, 2021. doi: 10.1214/21-EJS1847. Al Mohamad et al. (2022) Diaa Al Mohamad, Jelle J. Goeman, and Erik W. van Zwet. Simultaneous confidence intervals for ranks with application to ranking institutions. Biometrics, 78(1):238–247, 2022. Alon (2006) Noga Alon. Ranking tournaments. SIAM Journal on Discrete Mathematics, 20(1):137–142, 2006. Armitage et al. (1969) P. Armitage, C. K. McPherson, and B. C. Rowe. Repeated significance tests on accumulating data. Journal of the Royal Statistical Society. Series A (General), 132(2):235–244, 1969. Arnold et al. (2026) Sebastian Arnold, Georgios Gavrilopoulos, Benedikt Schulz, and Johanna Ziegel. Sequential model confidence sets. Journal of the Royal Statistical Society Series B: Statistical Methodology, 2026. Advance article, doi:10.1093/jrsssb/qkag066; first posted as arXiv:2404.18678, 2024. Arviv et al. (2026) Ofir Arviv, Kristjan Greenewald, Yotam Perlitz, Hadar Mulian, Michal Shmueli-Scheuer, and Leshem Choshen. Stop guessing when to stop testing: Efficient model evaluation with just enough data. In Findings of the Association for Computational Linguistics: ACL 2026, p. 871–881. Association for Computational Linguistics, 2026. doi: 10.18653/v1/2026.findings-acl.43. Avelar Menendez et al. (2026) Angel Rodrigo Avelar Menendez, Yufeng Liu, and Xiaowu Dai. Prompt-dependent ranking of large language models with uncertainty quantification. arXiv:2603.03336, 2026. Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM Leaderboard (2023–2024). Hugging Face, 2023. URL https://huggingface.co/spaces/open-llm-leaderboard-old/open_llm_leaderboard. Per-item results as released with tinyBenchmarks, snapshot of January 2024; accessed 17 September 2026. Bowyer et al. (2025) Sam Bowyer, Laurence Aitchison, and Desi R. Ivanova. Position: Don’t use the CLT in LLM evals with fewer than a few hundred datapoints. In Proceedings of the 42nd International Conference on Machine Learning, Position Paper Track, volume 267 of Proceedings of Machine Learning Research, p. 81143–81184. PMLR, 2025. URL https://openreview.net/forum?id=YhZ2PY2nZa. Chandra & Xie (2025) Onrina Chandra and Min-ge Xie. Finite-sample valid rank confidence sets for a broad class of statistical and machine learning models. arXiv:2512.00316, 2025. Charbit et al. (2007) Pierre Charbit, Stéphan Thomassé, and Anders Yeo. The minimum feedback arc set problem is NP-hard for tournaments. Combinatorics, Probability and Computing, 16(1):1–4, 2007. Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An open platform for evaluating LLMs by human preference. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 8359–8388. PMLR, 2024. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. arXiv:1803.05457, 2018. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168, 2021. Cutkosky & Orabona (2018) Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in Banach spaces. In Proceedings of the 31st Conference on Learning Theory, volume 75 of Proceedings of Machine Learning Research, p. 1493–1529, 2018. Finner & Strassburger (2002) Helmut Finner and Klaus Strassburger. The partitioning principle: A powerful tool in multiple decision theory. Annals of Statistics, 30(4):1194–1213, 2002. Fischer & Ramdas (2024) Lasse Fischer and Aaditya Ramdas. Admissible online closed testing must employ e-values. arXiv:2407.15733, 2024. Ghosh et al. (2026) Avijit Ghosh, Yifan Mai, Georgia Channing, and Leshem Choshen. AI evals are becoming the new compute bottleneck. EvalEval Coalition Blog, April 29, 2026. URL https://evalevalai.com/research/2026/04/29/eval-costs-bottleneck/. Accessed 22 August 2026. Gu et al. (2026) Runzhe Gu, Wenguang Sun, Bowen Gang, and Xintao Xia. Anytime-valid inference for online ranking of large language models. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. PMLR, 2026. URL https://openreview.net/forum?id=7Y8xRnGQ47. Hartog & Lei (2025) Will Hartog and Lihua Lei. Family-wise error rate control with e-values. arXiv:2501.09015, 2025. Heckel et al. (2019) Reinhard Heckel, Nihar B. Shah, Kannan Ramchandran, and Martin J. Wainwright. Active ranking from pairwise comparisons and when parametric assumptions do not help. Annals of Statistics, 47(6):3099–3126, 2019. Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. Holm (1979) Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65–70, 1979. Howard et al. (2021) Steven R. Howard, Aaditya Ramdas, Jon McAuliffe, and Jasjeet Sekhon. Time-uniform, nonparametric, nonasymptotic confidence sequences. Annals of Statistics, 49(2):1055–1080, 2021. Hsu & Shekhar (2026) Chia-Yu Hsu and Shubhanshu Shekhar. Efficient sequential evaluation of large language models. arXiv:2607.17409, 2026. Hsu (1996) Jason C. Hsu. Multiple Comparisons: Theory and Methods. Chapman & Hall, 1996. Huangfu & Hall (2018) Qi Huangfu and J. A. Julian Hall. Parallelizing the dual revised simplex method. Mathematical Programming Computation, 10(1):119–142, 2018. Kotawala (2026) Anany Kotawala. Resolution diagnostics for paired LLM evaluation. arXiv:2605.30315, 2026. Li et al. (2026) Jiachun Li, David Simchi-Levi, and Will Wei Sun. Low rank for rank: Uncertainty-aware task-specific LLM ranking under sparse pairwise comparisons. arXiv:2605.29395, 2026. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3214–3252, 2022. Maia Polo et al. (2024) Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: Evaluating LLMs with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 34303–34326. PMLR, 2024. Marcus et al. (1976) Ruth Marcus, Eric Peritz, and K. R. Gabriel. On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63(3):655–660, 1976. McNemar (1947) Quinn McNemar. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157, 1947. Miller (2024) Evan Miller. Adding error bars to evals: A statistical approach to language model evaluations. arXiv:2411.00640, 2024. Mirsky (1971) Leon Mirsky. A dual of Dilworth’s decomposition theorem. American Mathematical Monthly, 78(8):876–877, 1971. Mogstad et al. (2024) Magne Mogstad, Joseph P. Romano, Azeem M. Shaikh, and Daniel Wilhelm. Inference for ranks with applications to mobility across neighbourhoods and academic achievement across countries. Review of Economic Studies, 91(1):476–518, 2024. Neuhof & Benjamini (2024) Bitya Neuhof and Yuval Benjamini. Confident feature ranking. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, p. 1468–1476. PMLR, 2024. Neuhof & Benjamini (2026a) Bitya Neuhof and Yuval Benjamini. Quantifying ranking uncertainty in LLM benchmarks. arXiv:2607.16259, 2026a. Neuhof & Benjamini (2026b) Bitya Neuhof and Yuval Benjamini. Rank intervals for leaderboards: A hierarchical framework for model evaluation. arXiv:2606.08679, 2026b. Pocock (1977) Stuart J. Pocock. Group sequential methods in the design and analysis of clinical trials. Biometrika, 64(2):191–199, 1977. doi: 10.1093/biomet/64.2.191. Ramdas et al. (2023) Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023. Romano & Wolf (2005) Joseph P. Romano and Michael Wolf. Exact and approximate stepdown methods for multiple hypothesis testing. Journal of the American Statistical Association, 100(469):94–108, 2005. Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An adversarial Winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, p. 8732–8740, 2020. Shaffer (1986) Juliet Popper Shaffer. Modified sequentially rejective multiple test procedures. Journal of the American Statistical Association, 81(395):826–831, 1986. Singh et al. (2025) Shivalika Singh, Yiyang Nan, Alex Wang, Daniel D’souza, Sayash Kapoor, Ahmet Üstün, Sanmi Koyejo, Yuntian Deng, Shayne Longpre, Noah A. Smith, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. The leaderboard illusion. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, volume 38, 2025. URL https://openreview.net/forum?id=4Ae8edNqm0. Tavyrikov et al. (2026) Yury Tavyrikov, Jelle J. Goeman, and Rianne de Heide. Carefree multiple testing with e-processes. Electronic Journal of Statistics, 20(2):3178–3189, 2026. Ville (1939) Jean Ville. Étude critique de la notion de collectif. Gauthier-Villars, 1939. Virtanen et al. (2020) Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, et al. SciPy 1.0: Fundamental algorithms for scientific computing in Python. Nature Methods, 17(3):261–272, 2020. Vovk & Wang (2021) Vladimir Vovk and Ruodu Wang. E-values: Calibration, combination, and applications. Annals of Statistics, 49(3):1736–1754, 2021. Wang (2025) Ruodu Wang. The only admissible way of merging arbitrary e-values. Biometrika, 112(2):asaf020, 2025. Waudby-Smith & Ramdas (2020) Ian Waudby-Smith and Aaditya Ramdas. Confidence sequences for sampling without replacement. In Advances in Neural Information Processing Systems, volume 33, p. 20204–20214, 2020. Waudby-Smith & Ramdas (2024) Ian Waudby-Smith and Aaditya Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86(1):1–27, 2024. Westfall (1997) Peter H. Westfall. Multiple testing of general contrasts using logical constraints and correlations. Journal of the American Statistical Association, 92(437):299–306, 1997. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, p. 4791–4800, 2019. Zhou et al. (2026) Zhijian Zhou, Zesheng Ye, Zhaorun Chen, Bo Li, and Feng Liu. CELEUS: Certifiable and efficient LLM evaluation via e-processes. arXiv:2606.20820, 2026. Appendix contents 1 Introduction 2 Problem setting 3 Rank confidence sequences 4 What can be established, and why it matters 4.1 All certified dominances are correct, at all times 4.2 Ranks, top-k sets and tiers inherit the guarantee 4.3 Evaluation can stop early, model by model 5 Experiments 6 Limitations and conclusion References A Method details B Supporting theory B.1 Validity and growth of each pairwise wealth B.2 Supplement to the validity theorem B.3 Testing only orderings loses nothing, and the shortcut is valid B.4 Tiers and retirement C Proofs C.1 Pairwise wealths C.2 Validity, structure and power C.3 Closure, ranks, tiers and retirement C.4 Integer programming and hardness of exact certification D Scope of the guarantee and open problems E Experiments E.1 Does monitoring break a fixed-sample rank set, and does ours survive it? E.2 How much of a real leaderboard can be certified? E.3 How much compute does retirement save? E.4 What does anytime validity cost? E.5 Does the guarantee depend on how model errors are correlated? E.6 What does the multiplicity correction contribute? F Related work in detail Appendix A Method details Step 2 of Section 3 can be carried out three ways, and this appendix gives all three. Enumeration checks every ordering directly and is practical while the number of orderings is manageable. Integer programming computes the same answer at any size. A closed-form shortcut is fast but may certify less. All three are run after every update. If they are run less often, the running maximum in (4) is taken over those runs only, so no more orderings are eliminated. The certified set is then contained in tD_t, each rank set contains ℛj,tR_j,t, and the guarantee of Theorem 4.2 still holds. Algorithm 1 is the enumeration. Algorithm 1 Exact certification by enumeration (small M). Run after every update. 1: wealths EtjlE^jl_t; level α; the set C of surviving weak orders (initially W) 2: for each weak order W∈W do 3: e←1|T(W)|∑(j,l)∈T(W)Etjle← 1|T(W)| _(j,l)∈ T(W)E^jl_t ⊳ average wealth over the pairs with j⪯Wlj _Wl 4: if e≥1/αe≥ 1/α then remove W from C ⊳ permanent 5: end if 6: end for 7: ←(j,l)∈:l≺Wj for every W∈D←\(j,l) :\ l _Wj for every W \ ⊳ certified: j better than l 8: ℛj←1+#l:j≺Wl:W∈R_j←\1+\#\l:j _Wl\:\ W \ for each j ⊳ rank set of model j 9: return ,,(ℛj)jC,D,(R_j)_j Beyond about eight models enumeration is impractical. The closed-form rule of Algorithm 2 then certifies a subset of tD_t at O(M)O(M) operations per pair, and Algorithm 3 below computes tD_t exactly. Transitivity supplies the second term of the shortcut’s statistic. Take any ordering that puts l at least as high as j, and any third model m. Either the ordering puts m at least as high as j, and then the wealth EjmE^jm counts against it, or it puts m below j and hence below l, and then EmlE^ml counts against it. Either way the ordering is charged at least minEjm,Eml \E^jm,E^ml\. Algorithm 2 Shortcut certification (any M). Run after every update. 1: wealths EjlE^jl; level α; the set D of certified pairs (initially empty) 2: for each ordered pair (j,l)∉(j,l) do 3: B←Ejl+∑m≠j,lminEjm,EmlB← E^jl+ _m≠ j,l \E^jm,\,E^ml\ 4: if B≥M(M−1)/αB≥ M(M-1)/α then add (j,l)(j,l) to D ⊳ permanent 5: end if 6: end for 7: replace D by its transitive closure ⊳ if j≻mj m and m≻lm l then j≻lj l; same good event, no added error probability 8: return D Exact certification for any M. Algorithm 3 computes tD_t without enumerating orderings, by asking an integer program whether any surviving ordering places l at least as high as j. A weak order W is encoded by y∈0,1y∈\0,1\^P with yab=1y_ab=1 exactly when (a,b)∈T(W)(a,b)∈ T(W). The 0/10/1 vectors with yab+yba≥1y_ab+y_ba≥ 1 for all a≠ba≠ b, which is totality, and yab+ybc−yac≤1y_ab+y_bc-y_ac≤ 1 for all distinct a,b,ca,b,c, which is transitivity, are exactly the weak orders. A call time is a time at which an algorithm is run, and a wealth matrix is the array (Esab)(a,b)∈(E^ab_s)_(a,b) of pairwise wealths at one call time. With a run after every update the call times up to t are 1,…,t1,…,t. The order W survives call time s when EsW<1/αE^W_s<1/α, equivalently when gs(y):=∑(a,b)∈yab(Esab−1/α)<0g_s(y):= _(a,b) y_ab(E^ab_s-1/α)<0. For a set S of call times let z⋆(S)=minz:z≥gs(y)for all s∈S,y∈0,1 a weak order,yjl=1,z (S)= \z:\ z≥ g_s(y)\ for all s∈ S,\ y∈\0,1\^P a weak order,\ y_jl=1 \, (10) an integer program with one continuous variable. Then (j,l)∈t(j,l) _t exactly when z⋆(1,…,t)≥0z (\1,…,t\)≥ 0, and a minimizer with z⋆<0z <0 is a surviving ordering with (j,l)∈T(W)(j,l)∈ T(W), a witness that (j,l)∉t(j,l) _t. Survival needs the strict inequality gs(y)<0g_s(y)<0 and certification its complement z⋆≥0z ≥ 0, so in exact arithmetic no tolerance is needed. Algorithm 3 keeps a pool of witnesses and drops one the first time its average wealth EtWE^W_t reaches 1/α1/α. A witness W covers every pair in T(W)T(W), and (10) is solved only for pairs that no witness in the pool covers. The program starts with few call-time constraints and adds them only when needed. This is safe because a maximum over fewer times is no larger, so z⋆(S)≥0z (S)≥ 0 for any S⊆1,…,tS \1,…,t\ already certifies the pair. Otherwise the minimizer is checked against every call time, and a time at which it fails is added to S. In exact arithmetic Algorithm 3 computes tD_t (Proposition A.1). Under numerical optimization the implementation, which solves the programs with HiGHS (Huangfu & Hall, 2018) through SciPy (Virtanen et al., 2020), is conservative. It certifies a pair only when the solver’s lower bound on the optimum exceeds a tolerance of 10−9M(M−1)/α10^-9M(M-1)/α, relative to the scale of gsg_s. It accepts an ordering as a witness only when it survives every call time by that margin, and it leaves a numerically ambiguous pair unresolved until the next call. Hence within the solver’s numerical guarantees it never certifies more than tD_t. No such unresolved case arose in any run reported here. Algorithm 3 Exact certification by integer programming (any M). Run after every update. 1: wealths EsabE^ab_s at every call time s≤ts≤ t; level α; the certified set D, the pool Q of witnesses (surviving weak orders found so far) and the set S of constraint times, all initially empty 2: remove from Q every W with EtW≥1/αE^W_t≥ 1/α ⊳ rejected for good 3: for each (j,l)∉(j,l) with (j,l)∉T(W)(j,l)∉ T(W) for all W∈W , smallest EtjlE^jl_t first do 4: if Btjl≥M(M−1)/αB^jl_t≥ M(M-1)/α then add (j,l)(j,l) to D and continue ⊳ Proposition B.8 5: end if 6: loop 7: solve (10) for S∪tS∪\t\: optimum z⋆z , minimizer W 8: if z⋆≥0z ≥ 0 then add (j,l)(j,l) to D and break 9: end if 10: if EsW<1/αE^W_s<1/α for every call time s≤ts≤ t then add W to Q and break 11: else add to S a call time s with EsW≥1/αE^W_s≥ 1/α 12: end if 13: end loop 14: end for 15: return D Proposition A.1 (Algorithm 3 is exact). For every M≥2M≥ 2, every α∈(0,1)α∈(0,1) and every sequence of wealth matrices, one per call time, Algorithm 3 in exact arithmetic returns the set tD_t of Algorithm 1 at every call time t, and each inner loop ends after at most t solves. Exact certification is hard in the worst case. A decision problem is coNP-hard when every problem whose “no” answers can be checked in polynomial time reduces to it. Such a problem has no polynomial-time algorithm unless P=NPP=NP. Proposition A.2 (Certification is coNP-hard in the worst case). Deciding from a wealth matrix and a level α whether a given pair belongs to 1D_1, the certified set when the certifier is called once, on that matrix alone, is coNP-hard. The wealth matrices of the reduction take the values 00, 2ν−12^ν-1 and 2ν2^ν for an integer ν≥1ν≥ 1, have polynomial encoding size, and are produced by the general update (11) of Appendix B.1 with N=∞N=∞ from polynomially many binary items with predictable bets in 0,1\0,1\. The reduction is from the feedback arc set problem in tournaments (Alon, 2006; Charbit et al., 2007). A tournament is a directed graph with exactly one arc between every two vertices, and the problem asks for an ordering of its vertices with the fewest arcs pointing backward. The reduction uses bets in 0,1\0,1\, and Appendix D says what remains open for the default mixture bet. Each integer program (10) has M(M−1)M(M-1) binary variables, one continuous variable and M(M−1)(M−2)M(M-1)(M-2) transitivity constraints. Algorithm 3 solved every instance in our experiments, and Appendix E.6 reports its running time. An implementation stores logEλ,tjl E^jl_λ,t instead of Eλ,tjlE^jl_λ,t to avoid overflow. As for cost, the update is O(M2)O(M^2) per item, Algorithm 2 is O(M3)O(M^3) and Algorithm 1 is O(M2)O(M^2) per surviving ordering. In NumPy on one core the update costs 25μ25\, per item for 2020 models and 47μ47\, for 5050, the shortcut 0.070.07 ms and 0.60.6 ms per call, and the enumeration 5858 ms per call for 88 models while every ordering survives, falling as they are discarded. At each call Algorithm 3 solves no integer program for a pair that is already certified, that the shortcut certifies or that a witness in the pool covers. For any other pair it solves at least one and at most t (Proposition A.1). Appendix B Supporting theory In Appendices B to E, the true ability vector θ⋆θ is also written θ, as in Sections 2 and 3. B.1 Validity and growth of each pairwise wealth A large wealth must be improbable when the hypothesis it argues against is in fact true, however often and whenever the analyst looks. Fix (j,l)∈(j,l) , write Zt=ZtjlZ_t=Z^jl_t, and consider the general form of the update (2), E0jl=1E^jl_0=1 and Etjl=Et−1jl(1+λtZt−bt1+bt),E^jl_t=E^jl_t-1 (1+ _t\, Z_t-b_t1+b_t ), (11) in which the offset bt∈(−1,1]b_t∈(-1,1] and the bet λt∈[0,1] _t∈[0,1] are predictable, that is, determined by the information ℱt−1F_t-1. Each Eλ,tjlE^jl_λ,t of (2) has this form with λt≡λ _t≡λ, and so does the grid average EtjlE^jl_t, with λt _t the mean of Λ weighted by the wealths Eλ,t−1jlE^jl_λ,t-1 (Remark B.3). The factor in (11) is nonnegative, and its conditional expectation given ℱt−1F_t-1 is at most one whenever [Zt∣ℱt−1]≤btE[Z_t _t-1]≤ b_t. The wealth is then a nonnegative supermartingale: given the past, its expected next value is at most its current value. It is therefore an e-process for every predictable bet and stays one when frozen at a stopping time. The offsets (3) satisfy that condition for every law with θj≤θl _j≤ _l, and the wealth of (2) is an average over λ∈Λλ∈ of such processes. The choice of bets therefore affects power and never validity. The pairwise construction follows the betting approach of Waudby-Smith & Ramdas (2024), including, under (F), its version for sampling without replacement. Waudby-Smith & Ramdas (2020) built earlier confidence sequences for that setting without betting. Lemma B.1 (Pairwise betting supermartingale). Let P be a law under which P[Zt∣ℱt−1]≤btE_P[Z_t _t-1]≤ b_t almost surely for every t. Then (Etjl)(E^jl_t) of (11) is a nonnegative (ℱt)(F_t)-supermartingale under P with E0jl=1E^jl_0=1, for every predictable choice of (λt)( _t). If the process is frozen at an (ℱt)(F_t)-stopping time σ, that is, replaced by Et∧σjlE^jl_t σ where t∧σ=mint,σt σ= \t,σ\, the conclusion is unchanged. Proposition B.2 (Both sampling models). In each case below, for every (j,l)∈(j,l) , every predictable bet (λt)( _t) with values in [0,1][0,1], and every data-generating law under which HjlH_jl is true, (Etjl)(E^jl_t) is a nonnegative (ℱt)(F_t)-supermartingale started at 11, hence an e-process for HjlH_jl. This holds whatever the other models’ parameters and whatever the dependence between models. (a) Under (F), with ξ independent of Π , for t≤Nt≤ N, with bt=max−1+δ,min1,−St−1N−t+1,St−1=∑s<tZs,b_t= \-1+δ,\ \1,\ -S_t-1N-t+1 \ \, S_t-1= _s<tZ_s, (12) for any fixed δ∈(0,1]δ∈(0,1], and Etjl=ENjlE^jl_t=E^jl_N for t>Nt>N. (b) Under (S), with bt≡0b_t≡ 0. Remark B.3 (Bets). Validity holds for every predictable (λt)( _t), so the bet is chosen for power, and a constant bet is a poor choice when gaps differ across pairs. For binary scores under (S) (the finite case behaves the same away from the end of the benchmark), let Δ=θj−θl>0 = _j- _l>0 and let ρ=ℙ(Z≠0)ρ=P(Z≠ 0) be the discordance rate, the probability that the two models score differently on an item. Here Z stands for a generic paired difference ZtjlZ^jl_t. The expected log-growth of a constant bet is log(1+λZ)≈λΔ−λ2ρ/2E (1+λ Z)≈λ -λ^2ρ/2: it is maximized at λ⋆=Δ/ρλ = /ρ, which is in fact the exact maximizer when Δ<ρ <ρ, and is negative beyond about 2λ⋆2λ . By the strong law of large numbers, a constant bet with negative expected log-growth drives the wealth of a pair with small Δ/ρ /ρ, close relative to its discordance rate, to zero almost surely. Such a pair may therefore never be certified, however long the evaluation runs (an early crossing of the threshold remains possible). Two predictable alternatives avoid this, and a third mitigates it. The first is the plug-in bet λt=minλ¯,max0,^t−1Y/^t−1Y2 _t= \ λ, \0, E_t-1Y/ E_t-1Y^2\\. Here Ys=(Zs−bs)/(1+bs)Y_s=(Z_s-b_s)/(1+b_s), ^t−1 E_t-1 is the empirical average over items 1,…,t−11,…,t-1, and λ¯<1 λ<1 is fixed. This bet approximates GRAPA, the “growth rate adaptive to the particular alternative” bet (Waudby-Smith & Ramdas, 2024), which maximizes the average log-growth over the past items. Clipping to [0,λ¯]⊂[0,1][0, λ]⊂[0,1] meets the requirement of Lemma B.1, and the cap λ¯<1 λ<1 keeps every factor at least 1−λ¯>01- λ>0. The bet sets λt=0 _t=0 whenever ^t−1Y2=0 E_t-1Y^2=0 (in particular λ1=0 _1=0). The second is online Newton steps (Cutkosky & Orabona, 2018; Waudby-Smith & Ramdas, 2024), a second-order gradient update of λt _t on the running log-wealth, projected onto [0,λ¯][0, λ]. The third is a mixture. For constants λ(1),…,λ(|Λ|)λ^(1),…,λ^(| |) let Et(k)=∏s≤t(1+λ(k)Ys)E^(k)_t= _s≤ t(1+λ^(k)Y_s) be the wealth of the kkth constant; the mixture is the average |Λ|−1∑kEt(k)| |^-1 _kE^(k)_t. It is an average of supermartingales for one filtration, hence a supermartingale. It is also the wealth of the single predictable bet λt=∑kEt−1(k)λ(k)/∑kEt−1(k) _t= _kE^(k)_t-1λ^(k)/ _kE^(k)_t-1, the wealth-weighted mean of the grid (its denominator is positive because every λ(k)<1λ^(k)<1), so Lemma B.1 applies verbatim. A mixture over a fixed grid covers the range of Δ/ρ /ρ the grid spans but not arbitrarily small ratios. Once Δ/ρ /ρ falls below about half the smallest grid value, every component has negative expected log-growth and the mixture wealth also tends to zero (Proposition B.4). The mixture trails the best constant on the grid by at most a factor |Λ|| | in wealth, that is, by log|Λ| | | in the threshold, and has no learning phase. The bet for pair (j,l)(j,l) may depend on the whole past of all models. Under (S) the log-wealth of a constant bet is a sum of independent bounded terms, which yields the growth rate of every bet on the grid, a consistency condition for the mixture and explicit certification-time bounds, with no restriction on the dependence between models within an item. Fix (j,l)∈(j,l) with Δ=θj−θl>0 = _j- _l>0, write Zt=ZtjlZ_t=Z^jl_t, Et=EtjlE_t=E^jl_t for the grid average of (2) and Etλ=Eλ,tjlE^λ_t=E^jl_λ,t for the wealth of the constant bet λ, and let λmin _ be the smallest element of the grid Λ=λ(1),…,λ(|Λ|)⊂(0,12] =\λ^(1),…,λ^(| |)\⊂(0, 12]. Let μ(λ)=log(1+λZ1)μ(λ)=E (1+λ Z_1), ρ=[Z12]ρ=E[Z_1^2], which is the discordance rate of Remark B.3 for binary scores, cλ=log1+λ1−λc_λ= 1+λ1-λ, vλ=Varlog(1+λZ1)v_λ=Var (1+λ Z_1) and η=log(|Λ|M(M−1)/α)η= (| |M(M-1)/α). Let τB=inft:Et≥M(M−1)/ατ^B= \t:E_t≥ M(M-1)/α\ be the e-Bonferroni certification time of (j,l)(j,l), and let τscτ^sc and τ be the times at which (j,l)(j,l) enters tscD^sc_t and tD_t, respectively, where tscD^sc_t is the set certified by the shortcut (Proposition B.8). On every path the exact test certifies no later than the shortcut, and the shortcut no later than e-Bonferroni. Proposition B.4 (Growth, consistency and certification time under (S)). Assume (S), no retirement, and Δ>0 >0. (a) Growth. For λ∈(0,1)λ∈(0,1), t−1logEtλ→μ(λ)t^-1 E^λ_t→μ(λ) almost surely; μ is concave with μ(0)=0μ(0)=0, and for λ≤12λ≤ 12, λΔ−λ2ρ≤μ(λ)≤λΔλ -λ^2ρ≤μ(λ)≤λ , so μ(λ)>0μ(λ)>0 whenever 0<λ<Δ/ρ0<λ< /ρ. (b) Consistency. If μ(λmin)>0μ( _ )>0 then τB<∞τ^B<∞ almost surely and (j,l)∈t(j,l) _t for all t≥τBt≥τ^B; if μ(λmin)<0μ( _ )<0 then every bet on the grid has negative growth and Et→0E_t→ 0 almost surely. The first case holds whenever λmin<Δ/ρ _ < /ρ; for binary scores, μ(λ)=12ρcλ(Δ/ρ−x0(λ))μ(λ)= 12ρ\,c_λ ( /ρ-x_0(λ) ) with x0(λ)=log11−λ2/cλx_0(λ)= 11-λ^2/c_λ, so it holds if and only if Δ/ρ>x0(λmin) /ρ>x_0( _ ), which is Δ/ρ>0.015 /ρ>0.015 for the default grid. For binary scores no finite grid is consistent for every Δ>0 >0: pairs with 0<Δ/ρ<x0(λmin)0< /ρ<x_0( _ ) have negative growth on every grid bet. Consistency for every positive gap would require bets approaching zero, which we do not pursue here. (c) Certification time. Let λ∈Λλ∈ have μ=μ(λ)>0μ=μ(λ)>0 and write c=cλc=c_λ. For every integer t≥2η/μt≥ 2η/μ, ℙ(τB>t)≤exp(−tμ22c2)andℙ(τB>t)≤exp(−tμ28vλ+43cμ),vλ≤4λ2ρ.P(τ^B>t)≤ (- tμ^22c^2 ) (τ^B>t)≤ (- tμ^28v_λ+ 43cμ ), v_λ≤ 4λ^2ρ. (13) Define the rate and the horizon aλ=maxμ22c2,μ28vλ+43cμ,t⋆(ε)=max2ημ,log(1/ε)aλ.a_λ= \ μ^22c^2,\ μ^28v_λ+ 43cμ \, t ( )= \ 2ημ,\ (1/ )a_λ \. Then, for every ε∈(0,1) ∈(0,1), ℙ(τB≤⌈t⋆(ε)⌉)≥1−εP(τ^B≤ t ( ) )≥ 1- , and [τB]≤2η/μ+1/aλ+2E[τ^B]≤ 2η/μ+1/a_λ+2; the same bounds hold for τscτ^sc and τ. (d) Default grid. If 0.06≤Δ/ρ≤10.06≤ /ρ≤ 1, some grid bet lies in [Δ/(4.2ρ),Δ/(2ρ)][ /(4.2ρ), /(2ρ)], for which 0.18Δ2/ρ≤μ≤Δ2/(2ρ)0.18\, ^2/ρ≤μ≤ ^2/(2ρ) and c≤1.1Δ/ρc≤ 1.1\, /ρ, and then t⋆(ε)≤11.2ρη+min74.7, 270ρlog(1/ε)Δ2,[τB]≤11.2ρη+min74.7, 270ρΔ2+2,t ( )≤ 11.2\,ρη+ \74.7,\,270ρ\ (1/ ) ^2, [τ^B]≤ 11.2\,ρη+ \74.7,\,270ρ\ ^2+2, (14) so certification takes O(ρΔ−2log(|Λ|M2/(αε)))O(ρ\, ^-2 (| |M^2/(α ))) items. For binary scores with x0(0.03)<Δ/ρ<0.06x_0(0.03)< /ρ<0.06 only the general bound above applies, with the exact growth μ(0.03)=12ρc0.03(Δ/ρ−x0(0.03))μ(0.03)= 12ρ\,c_0.03( /ρ-x_0(0.03)), which tends to zero as Δ/ρ↓x0(0.03)≈0.015 /ρ x_0(0.03)≈ 0.015: a fixed grid has no uniform rate over its positive-growth pairs. B.2 Supplement to the validity theorem Remark B.5 (The restriction must be stated through T(W)T(W)). Every subset ⊆S of the one-sided hypotheses HjlH_jl is jointly satisfiable, since θ1=⋯=θM _1=…= _M makes all of them true. Requiring only that an intersection be consistent with some weak order therefore excludes nothing. The restriction that does exclude intersections is that of Shaffer (1986): S needs a test only if it can be the exact set of true hypotheses, that is, =T(W)S=T(W) for some W. The full closure tests the 2M(M−1)−12^M(M-1)-1 intersection hypotheses ⋂(j,l)∈Hjl _(j,l) H_jl, one per nonempty ⊆S . Of these, |||W| remain: 1313 of 6363 for M=3M=3, 7575 of 40954095 for M=4M=4. For M=2M=2 nothing is saved. The family of intersections ⋂(j,l)∈T(W)Hjl _(j,l)∈ T(W)H_jl, W∈W , that Algorithm 1 tests is not itself closed under intersection. This does no harm: the proof of Theorem 4.2 uses only that the cell ΘW⋆=θ:W(θ)=W⋆ _W =\θ:W(θ)=W \ containing the truth is rejected with probability at most α. The restriction to exactly realizable intersections is also that of Westfall (1997). Proposition B.6 (The output is a partial order). On the event t≠∅\C_t≠ \, which contains the event in (7), tD_t is a strict partial order on [M][M]: irreflexive, asymmetric and transitive. If t=∅C_t= then t=D_t=P; this has probability at most α uniformly in t, and it reveals that an error has occurred. B.3 Testing only orderings loses nothing, and the shortcut is valid Testing only the orderings instead of every set of pairwise hypotheses restricts the closed test. Both claims of Section 4.1 about what this restriction costs hold on every data set, not merely with high probability. Both compare procedures applied to the same wealths. Proposition B.7 (Restriction never hurts). Let tfullD^full_t be the output of the unrestricted closed test that rejects HjlH_jl once maxs≤t||−1∑(a,b)∈Esab≥1/α _s≤ t|S|^-1 _(a,b) E^ab_s≥ 1/α for every set ⊆S that contains (j,l)(j,l). Then tfull⊆tD^full_t _t for all t on every sample path. For some wealth configurations, exhibited in Example B.11, the restricted test certifies a pair that the full closure does not. Proposition B.8 (Transitivity pooling; validity of Algorithm 2). Let Btjl=Etjl+∑m∈[M]∖j,lminEtjm,EtmlB^jl_t=E^jl_t+ _m∈[M] \j,l\ \E^jm_t,E^ml_t\ be the pooled statistic of Section 3 and let tsc=(j,l)∈:maxs≤tBsjl≥M(M−1)/αD^sc_t=\(j,l) : _s≤ tB^jl_s≥ M(M-1)/α\ be the set that Algorithm 2 certifies before closure. Then tsc⊆tD^sc_t _t for all t on every sample path, so scD^sc inherits (8). It costs O(M)O(M) per pair per update and contains the e-Bonferroni rejections maxs≤tEsjl≥M(M−1)/α\ _s≤ tE^jl_s≥ M(M-1)/α\. Relation to existing results. To our knowledge both propositions are new. Proposition B.7 has a known relative. It is the sequential counterpart of the fixed-sample observation of Shaffer (1986) that logical relations among hypotheses permit a milder multiplicity correction at no cost in error. It differs in that it holds for e-processes, on every sample path and at all times, and in that its comparator fullD^full is the e-value closed test of Hartog & Lei (2025). Proposition B.8 has, to our knowledge, no direct counterpart. Its closest relative is the certification logic of Gu et al. (2026), which corrects by Bonferroni over the pairs (e-Bonferroni in our terms) and uses transitivity only after certification, to propagate certified pairs. The pooled statistic BtjlB^jl_t uses transitivity before certification (Appendix A). Because Btjl≥EtjlB^jl_t≥ E^jl_t, the shortcut contains their rejections, at a threshold that stays conservative relative to the exact test. Lemma B.9 (Transitive closure adds no error). Let (t)(D_t) be a certified set with good event G, and let ¯t D_t be the transitive closure of tD_t, the set of (j,l)∈(j,l) joined by a directed path j=v0,…,vr=lj=v_0,…,v_r=l, r≥1r≥ 1, with every (vi−1,vi)∈t(v_i-1,v_i) _t. On G, for every t, every (j,l)∈¯t(j,l)∈ D_t satisfies θj⋆>θl⋆θ _j>θ _l; in particular ¯t D_t is a strict partial order and tD_t contains no directed cycle. Hence (¯t)( D_t) is a certified set with the same good event: closing under transitivity adds rejections but never turns an error-free sample path into an erroneous one, so it leaves the family-wise error guarantee unchanged. Corollary B.10 (Certified top-k sets). Let (t)(D_t) be a certified set with Lj,t,Uj,tL_j,t,U_j,t as in (6). For k∈[M]k∈[M] let Ik,t=j:Uj,t≤kI_k,t=\j:U_j,t≤ k\ and Ok,t=j:Lj,t>kO_k,t=\j:L_j,t>k\. On G, for all k and t simultaneously, every j∈Ik,tj∈ I_k,t has Rj(θ⋆)≤kR_j(θ )≤ k and every j∈Ok,tj∈ O_k,t has Rj(θ⋆)>kR_j(θ )>k. Example B.11 (Strictness, M=3M=3). Let θ1>θ2>θ3 _1> _2> _3 and suppose that at some time t the wealths are E13≥6/αE^13≥ 6/α, E12=aE^12=a and E23=cE^23=c, with the three wealths of true hypotheses all zero and with no earlier time at which E12E^12 or the mean over ∖(1,3)P \(1,3)\ was larger. Of the eight weak orders with 1⪯W21 _W2, six contain (1,3)(1,3) and are eliminated through E13E^13; the other two place model 33 strictly below model 11 and hence, by transitivity, below model 22, so their means are a/3a/3 and a/4a/4. The restricted test therefore certifies 1≻21 2 once a≥4/αa≥ 4/α, whereas the full closure must also reject ∖(1,3)P \(1,3)\, needing a+c≥5/αa+c≥ 5/α, and e-Bonferroni needs a≥6/αa≥ 6/α. Whenever c<1/αc<1/α, the restricted test is strictly stronger than both, for 4/α≤a<5/α−c4/α≤ a<5/α-c: with α=0.1α=0.1, E13=60E^13=60, a=42a=42 and c=5c=5, it certifies 1≻21 2 while the full-closure statistic stands at 9.49.4 against a threshold of 1010 and E12E^12 at 4242 against 6060. The bound of Proposition B.8 is crude in two places that could be tightened. It divides by M(M−1)M(M-1) for every ordering, although |T(W)|=(M2)+#tied pairs|T(W)|= M2+\#\tied pairs\ reaches that value only when every pair is tied. It also has no step-down, that is, no lowering of the threshold for the remaining orders once some orders are rejected. The exact test has neither weakness, so what either change could recover is at most what the exact test gains over e-Bonferroni. In simulation the exact test certifies about 7%7\% more true dominances than e-Bonferroni after 500500 items and at most 3%3\% more by 40004000. On the eight highest-scoring models of the real leaderboard it certifies 22–5%5\% more up to three quarters of a benchmark and nearly the same number at the end (Appendix E.6). Remark B.12 (Weights must not depend on time). The plain average in (4) may be replaced by any weighted form EtW=∑(j,l)∈T(W)wjlWEtjlE^W_t= _(j,l)∈ T(W)w^W_jl\,E^jl_t with fixed weights wjlW≥0w^W_jl≥ 0 summing to one over T(W)T(W), which the proof of Theorem 4.2 covers without change; the weights may depend on W. What is not valid is letting them adapt: a predictably re-weighted average ∑kwk,tEt(k) _kw_k,tE^(k)_t of supermartingales is not a supermartingale in general, so the weights in (4) must not depend on t. Propositions B.7 and B.8 are stated for the plain average. B.4 Tiers and retirement An antichain of a strict partial order D is a set of models no two of which are related by D. The tier guarantee of Section 4.2 is part (c) below, with its monotonicity in (d); the rest is standard order theory. Theorem B.13 (Tier guarantee, full statement). Let D be a strict partial order on [M][M] and compute tiers from D as in Step 3 of Section 3. Parts (a), (b) and (e) hold for every such D. In (c) and (d), tiert(j)tier_t(j) is the tier computed from the transitive closure ¯t D_t of a certified set on the event G, where ¯t D_t is a strict partial order by Lemma B.9, and (d) assumes in addition t⊆t+1D_t _t+1; the subscript is dropped when t is fixed. On G these tiers equal those computed from tD_t itself: t⊆¯tD_t D_t, and each pair of ¯t D_t is joined by a chain in tD_t, which has no cycle on G, so longest chains have the same length. (a) Partition into antichains. The tiers partition [M][M], tiers 1,…,maxjtier(j)1,…, _jtier(j) are all nonempty, and no certified dominance holds between two models of the same tier. (b) Order. If (j,l)∈(j,l) then tier(j)<tier(l)tier(j)<tier(l). Every model in tier s≥2s≥ 2 is dominated by some model in tier s−1s-1. (c) Guarantee. On G, simultaneously for all j and all t, the bound (9) holds, that is, Rj(θ⋆)≥Lj,t≥tiert(j)R_j(θ )≥ L_j,t _t(j), where Lj,tL_j,t is computed from ¯t D_t; the number of tiers is at most the number of distinct values among θ1⋆,…,θM⋆θ _1,…,θ _M. (d) Monotonicity. If t⊆t+1D_t _t+1 then tiert(j)≤tiert+1(j)tier_t(j) _t+1(j): a model is never promoted as evidence accrues. (e) Minimality. No partition of [M][M] into antichains of D has fewer blocks (Mirsky, 1971). Remark B.14 (What tiers do not say). With =(a,b)D=\(a,b)\ on a,b,c\a,b,c\ the tiers are a,c\a,c\ and b\b\, and c is comparable to neither, which is why tier(j)<tier(l)tier(j)<tier(l) does not imply that j is certified better than l. The intuitive definition “same tier if and only if neither dominates the other” fails because incomparability is not transitive: with =(a,c)D=\(a,c)\, a and b are incomparable, b and c are incomparable, and a≻ca c. The cost of retirement is bounded under (S) for the rule that retires a model once every pair involving it is resolved. The top-k rule, under which a pair can be frozen before it is certified, is measured in Appendix E.3 and not bounded here. Corollary B.15 (Cost of all-pairs retirement under (S)). Assume (S), bets that are functions of each pair’s own past (the mixture is), and the all-pairs rule, which retires model j at the first time every pair j,l\j,l\ is certified in one direction. For an unordered pair let q(j,l)q(j,l) be the ordered pair in the true direction, and let +=(j,l):θj>θlP^+=\(j,l): _j> _l\. For q=(j,l)∈+q=(j,l) ^+ write Δq=θj−θl _q= _j- _l, ρq=[(Z1jl)2] _q=E[(Z^jl_1)^2] and μq(λ)=log(1+λZ1jl) _q(λ)=E (1+λ Z^jl_1). These are the quantities Δ , ρ and μ(λ)μ(λ) of Proposition B.4 computed for the pair q. Assume that every q∈+q ^+ involving j has a bet λq∈Λ _q∈ with μq=μq(λq)>0 _q= _q( _q)>0, which holds if and only if μq(λmin)>0 _q( _ )>0, and let aq=aλqa_q=a_ _q be its rate in Proposition B.4(c) and τqBτ^B_q the e-Bonferroni time of q computed without retirement. If model j is tied with no other model, put κj=maxl≠j2η/μq(j,l) _j= _l≠ j2η/ _q(j,l) and βj=maxl≠j1/aq(j,l) _j= _l≠ j1/a_q(j,l). Then σj≤maxl≠jτq(j,l)B _j≤ _l≠ jτ^B_q(j,l) on every path, ℙ(σj>t)≤(M−1)e−t/βjP( _j>t)≤(M-1)e^-t/ _j for every integer t≥κjt≥ _j, and [σj]≤κj+βj(log(M−1)+1)+2.E[ _j]≤ _j+ _j ( (M-1)+1 )+2. (15) If every model is untied and every true pair satisfies 0.06≤Δq/ρq≤10.06≤ _q/ _q≤ 1, then with Δj=minl≠j|θj−θl| _j= _l≠ j| _j- _l| and ρj=maxl≠jρq(j,l) _j= _l≠ j _q(j,l), [σj]≤11.2ρjη+min74.7, 270ρj(log(M−1)+1)Δj2+2,E[ _j]≤ 11.2\, _jη+ \74.7,\,270 _j\ ( (M-1)+1 ) _j^2+2, (16) and consequently [∑jσj]=O(∑jρjΔj−2log(|Λ|M2/α))E[ _j _j]=O ( _j _j _j^-2 (| |M^2/α) ). A model tied with another model is never retired under the all-pairs rule on the good event G; over any finite evaluation horizon it therefore incurs the full planned cost. Appendix C Proofs Time is discrete, t∈ℕ0=0,1,2,…t _0=\0,1,2,…\, and A1\A\ denotes the indicator of an event A. We write AcA^c for the complement of an event A and abbreviate “almost surely” to a.s. A random variable is ℱtF_t-measurable if its value is determined by the information ℱtF_t. Thus adapted means ℱtF_t-measurable at each t, and predictable means ℱt−1F_t-1-measurable. Conditional expectations of nonnegative random variables are always defined, possibly infinite, and the tower property holds for them; we use this without comment. In each proof, the bracket at the right of a line states what justifies that line. C.1 Pairwise wealths Proof of Lemma B.1. Write ϕt=1+λt(Zt−bt)/(1+bt) _t=1+ _t(Z_t-b_t)/(1+b_t), so that Etjl=∏s≤tϕsE^jl_t= _s≤ t _s by the definition of the wealth in (11). Nonnegativity and measurability. Zt−bt1+bt Z_t-b_t1+b_t ≥−1−bt1+bt=−1 \ ≥\ -1-b_t1+b_t=-1 [Zt≥−1Z_t≥-1 and 1+bt>01+b_t>0] ϕt _t ≥ 1−λt≥ 0 \ ≥\ 1- _t\ ≥\ 0 [previous line and λt∈[0,1] _t∈[0,1]] Etjl=∏s≤tϕs E^jl_t= _s≤ t _s ≥ 0,and is ℱt-measurable \ ≥\ 0,\ and is F_t-measurable [ZsZ_s, bsb_s, λs _s are ℱsF_s-measurable for s≤ts≤ t] Bound on one factor. To avoid any integrability question for the signed quantity λt(Zt−bt)/(1+bt) _t(Z_t-b_t)/(1+b_t), rewrite ϕt _t as a sum of two nonnegative terms: ϕt _t =1−λt+λt1+bt(1+Zt) =1- _t+ _t1+b_t\,(1+Z_t) [algebra; both terms are nonnegative since λt≤1 _t≤ 1, 1+Zt≥01+Z_t≥ 0, 1+bt>01+b_t>0] P[ϕt∣ℱt−1] _P[ _t _t-1] =1−λt+λt1+bt(1+P[Zt∣ℱt−1]) =1- _t+ _t1+b_t (1+E_P[Z_t _t-1] ) [λt _t and btb_t are ℱt−1F_t-1-measurable] ≤ 1−λt+λt1+bt(1+bt)=1 \ ≤\ 1- _t+ _t1+b_t(1+b_t)=1 [P[Zt∣ℱt−1]≤btE_P[Z_t _t-1]≤ b_t and λt/(1+bt)≥0 _t/(1+b_t)≥ 0] Supermartingale. P[Etjl∣ℱt−1] _P[E^jl_t _t-1] =Et−1jlP[ϕt∣ℱt−1] =E^jl_t-1\,E_P[ _t _t-1] [Et−1jl≥0E^jl_t-1≥ 0 is ℱt−1F_t-1-measurable] ≤Et−1jl \ ≤\ E^jl_t-1 [bound on one factor] P[Etjl] _P[E^jl_t] ≤P[Et−1jl]≤…≤E0jl=1 \ ≤\ E_P[E^jl_t-1]\ ≤\ …\ ≤\ E^jl_0=1 [tower property; induction on t] so the process is integrable and is a nonnegative supermartingale started at 11. Frozen process. Let σ be an (ℱt)(F_t)-stopping time. Et∧σjl−E(t−1)∧σjl E^jl_t σ-E^jl_(t-1) σ =σ≥t(Etjl−Et−1jl) =1\σ≥ t\ (E^jl_t-E^jl_t-1 ) [both sides vanish on σ<t\σ<t\ and agree on σ≥t\σ≥ t\] P[Et∧σjl−E(t−1)∧σjl|ℱt−1] _P [E^jl_t σ-E^jl_(t-1) σ\, |\,F_t-1 ] =σ≥t(P[Etjl∣ℱt−1]−Et−1jl)≤ 0 =1\σ≥ t\\, (E_P[E^jl_t _t-1]-E^jl_t-1 )\ ≤\ 0 [σ≥t=σ≤t−1c∈ℱt−1\σ≥ t\=\σ≤ t-1\^c _t-1; EtjlE^jl_t, Et−1jlE^jl_t-1 integrable by the previous block; supermartingale property] so the stopped process is again a nonnegative supermartingale started at 11. ∎ Proof of Proposition B.2. By Lemma B.1 it suffices to verify, whenever θj≤θl _j≤ _l, that btb_t is predictable with bt∈(−1,1]b_t∈(-1,1] and that [Zt∣ℱt−1]≤btE[Z_t _t-1]≤ b_t. (a) Under (F), let zi=xij−xilz_i=x_ij-x_il, so that HjlH_jl reads ∑i=1Nzi≤0 _i=1^Nz_i≤ 0. Fix t≤Nt≤ N. [Zt∣ℱt−1] [Z_t _t-1] =∑i=1Nzi−St−1N−t+1 = _i=1^Nz_i-S_t-1N-t+1 [given ℱt−1F_t-1, Π(t) (t) is uniform on the N−t+1N-t+1 unrevealed items, and ξ is independent of Π ] ≤−St−1N−t+1 \ ≤\ -S_t-1N-t+1 [∑i=1Nzi≤0 _i=1^Nz_i≤ 0] [Zt∣ℱt−1] [Z_t _t-1] ≤ 1 \ ≤\ 1 [Zt≤1Z_t≤ 1, scores lying in [0,1][0,1]] [Zt∣ℱt−1] [Z_t _t-1] ≤min1,−St−1N−t+1≤bt \ ≤\ \1, -S_t-1N-t+1 \\ ≤\ b_t [the two lines above; taking the maximum with −1+δ-1+δ in eq. (12) can only raise the bound] −1<−1+δ≤bt -1<-1+δ\ ≤\ b_t ≤ 1 \ ≤\ 1 [eq. (12): the clipping, as Lemma B.1 requires; the minimum with 11 and δ≤1δ≤ 1] bt b_t is ℱt−1-measurable \ is F_t-1-measurable [it is a function of St−1=∑s<tZsS_t-1= _s<tZ_s] For t>Nt>N the process is constant. All M(M−1)M(M-1) processes are adapted to the single filtration generated by the shared permutation; independence of permutations across pairs is neither available nor needed. (b) Under (S), with bt≡0b_t≡ 0: [Zt∣ℱt−1] [Z_t _t-1] =[Xtj−Xtl] =E[X_tj-X_tl] [XtX_t is independent of ℱt−1=σ(X1,…,Xt−1,ξ)F_t-1=σ(X_1,…,X_t-1,ξ)] =θj−θl≤ 0=bt = _j- _l\ ≤\ 0=b_t [definition of θ; HjlH_jl] Nothing is assumed about the joint law of (Xt1,…,XtM)(X_t1,…,X_tM), and λt _t may depend on the past of all models, since only ℱt−1F_t-1-measurability is used. ∎ Proof of Proposition B.4. Under (S) with bt≡0b_t≡ 0 the factor of the constant bet λ on item s is 1+λZs1+λ Z_s; write ℓs=log(1+λZs) _s= (1+λ Z_s), so that logEtλ=∑s≤tℓs E^λ_t= _s≤ t _s. (a) Growth. Let f(u)=log(1+u)−u+u2f(u)= (1+u)-u+u^2; then f(0)=0f(0)=0 and f′(u)=u(1+2u)/(1+u)f (u)=u(1+2u)/(1+u), which is ≤0≤ 0 on [−12,0][- 12,0] and ≥0≥ 0 on [0,12][0, 12], so f≥0f≥ 0 on [−12,12][- 12, 12]. ℓ1,ℓ2,…are i.i.d. with|ℓs| _1, _2,…\ are i.i.d.\ with\ | _s| ≤log11−λ ≤ 11-λ [ZsZ_s i.i.d., |Zs|≤1|Z_s|≤ 1, λ<1λ<1] t−1∑s≤tℓs t^-1 _s≤ t _s →ℓ1=μ(λ)a.s. _1=μ(λ)\ a.s. [strong law of large numbers] μis concave on [0,1), and μ(0) μ\ is concave on [0,1), and μ(0) =0 =0 [λ↦log(1+λz)λ (1+λ z) is concave for each z∈[−1,1]z∈[-1,1]; expectation preserves concavity] u−u2≤log(1+u) u-u^2≤ (1+u) ≤ufor u∈[−12,12] ≤ u u∈[- 12, 12] [f≥0f≥ 0; concavity of log ] λΔ−λ2ρ≤μ(λ) λ -λ^2ρ≤μ(λ) ≤λΔfor λ≤12 ≤λ λ≤ 12 [u=λZ1∈[−12,12]u=λ Z_1∈[- 12, 12]; take expectations, Z1=ΔEZ_1= , Z12=ρEZ_1^2=ρ] (b) Consistency. Suppose μ(λmin)>0μ( _ )>0. logEt E_t ≥logEtλmin−log|Λ|→+∞a.s. ≥ E _ _t- | |→+∞\ a.s. [the mixture averages nonnegative terms; (a)] τB τ^B <∞a.s. <∞\ a.s. [τBτ^B is the first time Et≥M(M−1)/αE_t≥ M(M-1)/α] (j,l) (j,l) ∈tfor all t≥τB _t\ for all t≥τ^B [e-Bonferroni uses the running maximum and is contained in tsc⊆tD^sc_t _t (Proposition B.8); tD_t is monotone] Suppose μ(λmin)<0μ( _ )<0 and let λ∈Λλ∈ with λ>λminλ> _ . μ(λmin) μ( _ ) ≥λminλμ(λ)+(1−λminλ)μ(0) ≥ _ λμ(λ)+ (1- _ λ )μ(0) [concavity of μ] μ(λ) μ(λ) ≤λminμ(λmin)<0 ≤ λ _ μ( _ )<0 [μ(0)=0μ(0)=0; rearrange] Et E_t →0a.s. → 0\ a.s. [(a): each of the |Λ|| | components has logEtλ→−∞ E^λ_t→-∞] For binary scores, with p±=ℙ(Z1=±1)=(ρ±Δ)/2p_±=P(Z_1=± 1)=(ρ± )/2 and ρ>0ρ>0: μ(λ)=p+log(1+λ)+p−log(1−λ) μ(λ)=p_+ (1+λ)+p_- (1-λ) =12ρlog(1−λ2)+12Δcλ = 12ρ (1-λ^2)+ 12 \,c_λ [Z1∈−1,0,1Z_1∈\-1,0,1\; log(1+λ)±log(1−λ) (1+λ)± (1-λ) equal log(1−λ2) (1-λ^2) and cλc_λ] =12ρcλ(Δ/ρ−x0(λ)),x0(λ)=log11−λ2/cλ = 12ρ\,c_λ ( /ρ-x_0(λ) ), x_0(λ)= 11-λ^2/c_λ [factor out 12ρcλ>0 12ρ c_λ>0] μ(λmin)>0 μ( _ )>0 ⇔Δ/ρ>x0(λmin) /ρ>x_0( _ ) [previous line; x0(0.03)=0.000900/0.0600=0.0150x_0(0.03)=0.000900/0.0600=0.0150] (c) Certification time. Fix λ∈Λλ∈ with μ>0μ>0 and put Vs=ℓs−μV_s= _s-μ. ℓs _s ∈[log(1−λ),log(1+λ)],an interval of length c ∈[ (1-λ), (1+λ)],\ an interval of length c [|Zs|≤1|Z_s|≤ 1] τB>t \τ^B>t\ ⊆Et<M(M−1)α⊆Etλ<|Λ|M(M−1)α=∑s≤tℓs<η \E_t< M(M-1)α\ \E^λ_t< | |M(M-1)α\=\ _s≤ t _s<η\ [first crossing; Et≥Etλ/|Λ|E_t≥ E^λ_t/| |; definition of η] ℙ(∑s≤tℓs≤tμ−x) ( _s≤ t _s≤ tμ-x ) ≤exp(−2x2/(tc2)),x>0 ≤ (-2x^2/(tc^2) ), x>0 [Hoeffding’s inequality, i.i.d. summands with range c] ℙ(τB>t) (τ^B>t) ≤exp(−tμ2/(2c2))for integer t≥2η/μ ≤ (-tμ^2/(2c^2) ) integer t≥ 2η/μ [the two previous lines with x=tμ−η≥tμ/2x=tμ-η≥ tμ/2; the bound decreases in x] |log(1+u)| | (1+u)| ≤2|u|for u∈[−12,12] ≤ 2|u| u∈[- 12, 12] [u≥0u≥ 0: log(1+u)≤u (1+u)≤ u; u<0u<0: −log(1+u)≤11+u−1=−u1+u≤2|u|- (1+u)≤ 11+u-1= -u1+u≤ 2|u|] vλ v_λ ≤ℓ12≤4λ2ρ _1^2≤ 4λ^2ρ [previous line with u=λZ1u=λ Z_1] ℙ(∑s≤tVs≤−x) ( _s≤ tV_s≤-x ) ≤exp(−x22(tvλ+cx/3)),x>0 ≤ (- x^22(tv_λ+cx/3) ), x>0 [Bernstein’s inequality: VsV_s i.i.d., centered, |Vs|≤c|V_s|≤ c] ℙ(τB>t) (τ^B>t) ≤exp(−(tμ/2)22(tvλ+ctμ/6))=exp(−tμ28vλ+43cμ)for integer t≥2η/μ ≤ (- (tμ/2)^22(tv_λ+ctμ/6) )= (- tμ^28v_λ+ 43cμ ) integer t≥ 2η/μ [τB>t⊆∑s≤tVs<−(tμ−η)\τ^B>t\ \ _s≤ tV_s<-(tμ-η)\ with tμ−η≥tμ/2tμ-η≥ tμ/2; the bound decreases in x] which is (13); since both bounds hold, ℙ(τB>t)≤e−aλtP(τ^B>t)≤ e^-a_λt for every integer t≥2η/μt≥ 2η/μ, with aλa_λ the larger of the two rates. Write a=aλa=a_λ. ℙ(τB>⌈t⋆(ε)⌉) (τ^B> t ( ) ) ≤ε ≤ [⌈t⋆⌉≥2η/μ t ≥ 2η/μ and a⌈t⋆⌉≥log(1/ε)a t ≥ (1/ )] [τB]=∑t≥0ℙ(τB>t) [τ^B]= _t≥ 0P(τ^B>t) ≤t0+∑t≥t0e−at≤t0+1+1a,t0=⌈2η/μ⌉ ≤ t_0+ _t≥ t_0e^-at≤ t_0+1+ 1a, t_0= 2η/μ [probabilities are at most one below t0t_0; geometric series; 1/(1−e−a)≤1+1/a1/(1-e^-a)≤ 1+1/a] [τB] [τ^B] ≤2η/μ+1/a+2 ≤ 2η/μ+1/a+2 [t0≤2η/μ+1t_0≤ 2η/μ+1] the bounds hold for τsc bounds hold for τ^sc and τ and τ [τ≤τsc≤τBτ≤τ^sc≤τ^B on every path, Proposition B.8] (d) Default grid. Let I=[Δ/(4.2ρ),Δ/(2ρ)]I=[ /(4.2ρ), /(2ρ)]. If 0.06≤Δ/ρ≤0.1260.06≤ /ρ≤ 0.126 then 0.03∈I0.03∈ I, since 0.03≤Δ/(2ρ)0.03≤ /(2ρ) iff Δ/ρ≥0.06 /ρ≥ 0.06 and 0.03≥Δ/(4.2ρ)0.03≥ /(4.2ρ) iff Δ/ρ≤0.126 /ρ≤ 0.126. If 0.126<Δ/ρ≤10.126< /ρ≤ 1, sort the grid increasingly and let λ(k)λ^(k) be the largest grid bet at most Δ/(4.2ρ) /(4.2ρ); either λ(k)=Δ/(4.2ρ)∈Iλ^(k)= /(4.2ρ)∈ I, or λ(k)<Δ/(4.2ρ)≤1/4.2<0.5λ^(k)< /(4.2ρ)≤ 1/4.2<0.5, so that λ(k+1)λ^(k+1) exists and, consecutive grid ratios being at most 2.084<2.12.084<2.1, Δ/(4.2ρ)<λ(k+1)≤2.084λ(k)<Δ/(2ρ) /(4.2ρ)<λ^(k+1)≤ 2.084\,λ^(k)< /(2ρ). In both cases some grid bet λ lies in I, and for it: μ≥λ(Δ−λρ) μ≥λ( -λρ) ≥0.18Δ2/ρ ≥ 0.18\, ^2/ρ [(a); λ(Δ−λρ)λ( -λρ) is concave in λ, with values 0.181Δ2/ρ0.181\, ^2/ρ and 0.25Δ2/ρ0.25\, ^2/ρ at the ends of I] μ≤λΔ≤Δ22ρ,c μ≤λ ≤ ^22ρ, c ≤2.2λ≤1.1Δρ ≤ 2.2λ≤ 1.1\, ρ [(a); cλ/λ=2artanh(λ)/λc_λ/λ=2artanh(λ)/λ increases to 2.1972.197 at λ=12λ= 12] 2ημ≤11.2ρηΔ2,2c2μ2 2ημ≤ 11.2\,ρη ^2, 2c^2μ^2 ≤2(1.21)0.0324Δ−2=74.7Δ−2 ≤ 2(1.21)0.0324\, ^-2=74.7\, ^-2 [substitute the bounds on μ and c] 8vλ≤32λ2ρ≤8Δ2ρ,43cμ 8v_λ≤ 32λ^2ρ≤ 8 ^2ρ, 43cμ ≤43(1.1)12Δ3ρ2≤0.734Δ2ρ ≤ 43(1.1) 12\, ^3ρ^2≤ 0.734\, ^2ρ [λ≤Δ/(2ρ)λ≤ /(2ρ); the bounds on c and μ; Δ≤ρ ≤ρ in this regime] 8vλ+43cμ2 8v_λ+ 43cμ^2 ≤8.734Δ2/ρ0.0324Δ4/ρ2≤270ρΔ2 ≤ 8.734\, ^2/ρ0.0324\, ^4/ρ^2≤ 270\, ρ ^2 [the two previous lines] and (14) follows from (c) with 1/aλ≤min74.7,270ρΔ−21/a_λ≤ \74.7,270ρ\ ^-2. For binary scores the exact growth of λ=0.03λ=0.03 is the formula in (b), with c0.03=0.0600c_0.03=0.0600. ∎ C.2 Validity, structure and power Proof of Theorem 4.2. Fix the data-generating law P, with parameter θ⋆θ and W⋆=W(θ⋆)W =W(θ ), and write EW=(EtW)t≥0E^W=(E^W_t)_t≥ 0 for the process of (4). EW⋆E^W is an e-process under P. Let τ be a bounded (ℱt)(F_t)-stopping time. (j,l)∈T(W⋆) (j,l)∈ T(W ) ⇒θj⋆≤θl⋆ \ \ θ _j≤θ _l [definition of T(W)T(W)] ⇒P[Eτjl]≤1 \ \ E_P [E^jl_τ ]≤ 1 [Assumption 4.1; τ is a stopping time for the common filtration] P[EτW⋆] _P [E^W _τ ] =∑(j,l)∈T(W⋆)wjlW⋆P[Eτjl] = _(j,l)∈ T(W )w^W _jl\,E_P [E^jl_τ ] [eq. (4) in its weighted form (Remark B.12); linearity, the weights being deterministic] ≤∑(j,l)∈T(W⋆)wjlW⋆=1 \ ≤\ _(j,l)∈ T(W )w^W _jl=1 [wjlW⋆≥0w^W _jl≥ 0, the previous line, and the weights sum to one] Only linearity of expectation is used; the joint law of the pairwise processes plays no role. That all of them are e-processes for the same filtration is needed, because τ must be a stopping time for each of them at once. Ville’s inequality. Let τ⋆=inft:EtW⋆≥1/ατ = \t:E^W _t≥ 1/α\ and fix a deterministic n. τ⋆ τ is an (ℱt)-stopping time \ is an (F_t)-stopping time [EW⋆E^W is adapted] Eτ⋆∧nW⋆ E^W _τ n ≥α−1τ⋆≤n \ ≥\ α^-11\τ ≤ n\ [nonnegativity on τ⋆>n\τ >n\; the threshold on τ⋆≤n\τ ≤ n\] 1 1 ≥P[Eτ⋆∧nW⋆] \ ≥\ E_P [E^W _τ n ] [the e-process property above, at the bounded stopping time τ⋆∧nτ n] ≥α−1ℙ(τ⋆≤n) \ ≥\ α^-1P(τ ≤ n) [the pathwise bound of the previous display] ℙ(τ⋆<∞) (τ <∞) ≤α \ ≤\ α [let n→∞n→∞] ∃t:W⋆∉t \∃ t:W _t\ =τ⋆<∞ =\τ <∞\ [definition of tC_t] which is (7). Family-wise error. On τ⋆=∞\τ =∞\, for every t: W⋆ W ∈t _t [definition of τ⋆τ and definition of tC_t] (j,l)∈t (j,l) _t ⇒l≺Wjfor every W∈t \ \ l _Wj\ for every W _t [eq. (5)] ⇒l≺W⋆j \ \ l _W j [W⋆∈tW _t] ⇒θj⋆>θl⋆ \ \ θ _j>θ _l [W⋆=W(θ⋆)W =W(θ )] So the event in (8) is contained in τ⋆<∞\τ <∞\, whose probability is at most α by Ville’s inequality above. Random times. Both events are of the form ∀t:…\∀ t:…\, so they imply the corresponding statement at any ℕ0N_0-valued random time, whether or not it is a stopping time. ∎ Proof of Proposition B.6. Suppose t≠∅C_t≠ and let W∈tW _t be arbitrary. Irreflexive. t⊆D_t , which contains no pair (j,j)(j,j). Asymmetric. (j,l),(l,j)∈t (j,l),(l,j) _t ⇒l≺Wjandj≺Wl \ \ l _Wj\ and\ j _Wl [eq. (5)] ⇒a contradiction \ \ a contradiction [l≺Wjl _Wj means l⪯Wjl _Wj and not j⪯Wlj _Wl] Transitive. Let (j,l),(l,m)∈t(j,l),(l,m) _t. m m ≠j ≠ j [asymmetry] l≺Wj l _Wj\ andm≺Wl \ m _Wl [eq. (5)] m m ⪯Wj _Wj [⪯W _W is transitive] j⪯Wm j _Wm\ is impossible [j⪯Wm⪯Wlj _Wm _Wl would give j⪯Wlj _Wl, contradicting l≺Wjl _Wj] m m ≺Wj _Wj [the two previous lines] (j,m) (j,m) ∈t _t [W∈tW _t was arbitrary; eq. (5)] Empty confidence set. If t=∅C_t= : t _t = =P [the condition in eq. (5) is vacuous] ∃t:t=∅ \∃ t:C_t= \ ⊆∃t:W⋆∉t \∃ t:W _t\ [W⋆∈tW _t would make tC_t nonempty] ℙ(∃t:t=∅) (∃ t:C_t= ) ≤α \ ≤\ α [eq. (7)] ∎ Proof of Proposition B.7. Let (j,l)∈tfull(j,l) ^full_t and let W be any weak order with (j,l)∈T(W)(j,l)∈ T(W); put =T(W)S=T(W). ⊆and∋(j,l) \ and\ S (j,l) [definition of T(W)T(W) and the choice of W] maxs≤t||−1∑(a,b)∈Esab _s≤ t|S|^-1 _(a,b) E^ab_s ≥ 1/α \ ≥\ 1/α [(j,l)∈tfull(j,l) ^full_t: the full closure rejects every such S] EsW E^W_s =||−1∑(a,b)∈Esab =|S|^-1 _(a,b) E^ab_s [eq. (4) with arithmetic-mean weights] maxs≤tEsW _s≤ tE^W_s ≥ 1/α,so W∉t \ ≥\ 1/α,\ so W _t [the two previous lines; definition of tC_t] (j,l) (j,l) ∈t _t [W with (j,l)∈T(W)(j,l)∈ T(W) was arbitrary; eq. (5)] Example B.11 gives a configuration in which the inclusion is strict; it has been checked by enumerating the 1313 weak orders and the 6363 subsets. ∎ Proof of Proposition B.8. Let (j,l)∈tsc(j,l) ^sc_t and choose s≤ts≤ t with Bsjl≥M(M−1)/αB^jl_s≥ M(M-1)/α. Let W be any weak order with j⪯Wlj _Wl, that is (j,l)∈T(W)(j,l)∈ T(W), and let m∈[M]∖j,lm∈[M] \j,l\. Claim: j⪯Wmj _Wm or m⪯Wlm _Wl. neither j⪯Wm nor m⪯Wl j _Wm nor m _Wl ⇒m≺Wjandl≺Wm \ \ m _Wj\ and\ l _Wm [totality of ⪯W _W] ⇒l≺Wj \ \ l _Wj [transitivity] ⇒a contradiction with j⪯Wl \ \ a contradiction with j _Wl [l≺Wjl _Wj means not j⪯Wlj _Wl] Pooling. T(W) T(W) ∋(j,l)and, for each m∈[M]∖j,l,(j,m) or (m,l) (j,l)\ and, for each m∈[M] \j,l\,\ (j,m) or (m,l) [definition of T(W)T(W) and the claim] these M−1 pairs M-1 pairs are distinct [two could coincide only if (j,m)=(m′,l)(j,m)=(m ,l) for some m,m′∉j,lm,m ∉\j,l\, which would force m′=jm =j] ∑(a,b)∈T(W)Esab _(a,b)∈ T(W)E^ab_s ≥Esjl+∑m∈[M]∖j,lminEsjm,Esml=Bsjl \ ≥\ E^jl_s+ _m∈[M] \j,l\ \E^jm_s,E^ml_s\=B^jl_s [all e-processes are nonnegative; the pair chosen for m contributes at least the minimum] EsW E^W_s ≥BsjlM(M−1)≥1α \ ≥\ B^jl_sM(M-1)\ ≥\ 1α [eq. (4) with |T(W)|≤M(M−1)|T(W)|≤ M(M-1); the choice of s] W W ∉t _t [definition of tC_t, since s≤ts≤ t] (j,l) (j,l) ∈t _t [W with (j,l)∈T(W)(j,l)∈ T(W) was arbitrary; eq. (5)] The same time s serves for all W, which is why the running maximum may be taken over BjlB^jl. The e-Bonferroni rejections are contained in tscD^sc_t because Bsjl≥EsjlB^jl_s≥ E^jl_s. The cost is one sum of M−2M-2 minima per pair. ∎ C.3 Closure, ranks, tiers and retirement Proof of Lemma B.9. Work on G and fix t. Let (j,l)∈¯t(j,l)∈ D_t, joined by the directed path v0=j,v1,…,vr=lv_0=j,v_1,…,v_r=l with (vi−1,vi)∈t(v_i-1,v_i) _t for all i. θvi−1⋆ θ _v_i-1 >θ⋆vifor every i >θ _v_i every i [(vi−1,vi)∈t(v_i-1,v_i) _t and definition of G] θj⋆=θv0⋆ θ _j=θ _v_0 >θv1⋆>⋯>θvr⋆=θl⋆ >θ _v_1>…>θ _v_r=θ _l [chain the previous line] thas no directed cycle _t\ has no directed cycle [a cycle through j is such a path from j to itself and would give θj⋆>θj⋆θ _j>θ _j] ¯tis a strict partial order D_t\ is a strict partial order [transitive by construction; irreflexive because (j,j)∈¯t(j,j)∈ D_t would be a directed cycle] the good event of (¯t) contains good event of ( D_t) contains G [the second line holds on G for every t and every (j,l)∈¯t(j,l)∈ D_t] the good event of (¯t) is contained in good event of ( D_t) is contained in G [t⊆¯tD_t D_t for every t] so the two events coincide. ∎ Proof of Theorem 4.3. (a) On the event in (7), for all j and t: W⋆ W ∈t _t [eq. (7)] Rj(θ⋆) R_j(θ ) =Rj(W⋆) =R_j(W ) [the rank depends on θ only through W(θ)W(θ)] ∈ℛj,t _j,t [definition of ℛj,tR_j,t and the first line] (b) Work on G and fix j and t. (l,j)∈t (l,j) _t ⇒θl⋆>θj⋆ \ \ θ _l>θ _j [definition of G] #l:θl⋆>θj⋆ \#\l:θ _l>θ _j\ ≥#l:(l,j)∈t \ ≥\ \#\l:(l,j) _t\ [previous line] Rj(θ⋆) R_j(θ ) ≥ 1+#l:(l,j)∈t=Lj,t \ ≥\ 1+\#\l:(l,j) _t\=L_j,t [eq. (1) and eq. (6)] (j,l)∈t (j,l) _t ⇒l≠jandθl⋆<θj⋆ \ \ l≠ j\ and\ θ _l<θ _j [t⊆D_t ; definition of G] #l:θl⋆>θj⋆ \#\l:θ _l>θ _j\ ≤(M−1)−#l:(j,l)∈t \ ≤\ (M-1)-\#\l:(j,l) _t\ [each l with (j,l)∈t(j,l) _t is one of the M−1M-1 other models and is not counted on the left] Rj(θ⋆) R_j(θ ) ≤M−#l:(j,l)∈t=Uj,t \ ≤\ M-\#\l:(j,l) _t\=U_j,t [eq. (1) and eq. (6)] Ties cause no difficulty: models tied with j are counted on neither side. Since t⊆¯tD_t D_t and ¯t D_t is certified with the same good event (Lemma B.9), using ¯t D_t can only raise L and lower U. (c) Let W∈tW _t; recall Rj(W)=1+#l:j≺WlR_j(W)=1+\#\l:j _Wl\. (l,j)∈t (l,j) _t ⇒j≺Wl,so l is counted in Rj(W) \ \ j _Wl,\ so l is counted in R_j(W) [eq. (5)] Rj(W) R_j(W) ≥ 1+#l:(l,j)∈t=Lj,t \ ≥\ 1+\#\l:(l,j) _t\=L_j,t [previous line] (j,l)∈t (j,l) _t ⇒l≺Wj,so l is not counted \ \ l _Wj,\ so l is not counted [eq. (5); l≺Wjl _Wj excludes j≺Wlj _Wl] Rj(W) R_j(W) ≤M−#l:(j,l)∈t=Uj,t \ ≤\ M-\#\l:(j,l) _t\=U_j,t [at most the M−1M-1 other models are counted, and those with (j,l)∈t(j,l) _t are not] Hence ℛj,t⊆[Lj,t,Uj,t]R_j,t [L_j,t,U_j,t]. Strictness. Take M=3M=3, α=0.1α=0.1 and, as hypothetical values at some time with no earlier crossing, E12=10E^12=10, E13=90E^13=90, E21=0.2E^21=0.2, E32=21E^32=21, E23=E31=0E^23=E^31=0. Enumeration shows that exactly four weak orders survive, namely 3∼2≺13 2 1, 3≺2∼13 2 1, 2≺3≺12 3 1 and 3≺2≺13 2 1, writing each from its lowest-ranked model to its highest (so 3∼2≺13 2 1 ranks model 1 first and ties models 2 and 3), and that t=(1,3)D_t=\(1,3)\. Hence U1,t=2U_1,t=2, whereas model 11 has rank 11 in every surviving weak order, so ℛ1,t=1R_1,t=\1\. The gain comes from the tie convention: H12H_12 cannot be rejected while the tie 1∼21 2 survives, yet a tie at the top still gives rank 11. Projection sets need not be intervals: with (E12,E13,E21,E23,E31,E32)=(0,5,0,31,5,31)(E^12,E^13,E^21,E^23,E^31,E^32)=(0,5,0,31,5,31) and α=0.1α=0.1, again with no earlier crossing, exactly 1∼2≺31 2 3 and 3≺1∼23 1 2 survive, so ℛ3,t=1,3R_3,t=\1,3\ while [L3,t,U3,t]=[1,3][L_3,t,U_3,t]=[1,3]. ∎ Proof of Corollary B.10. On G, for all j, k and t at once: j∈Ik,t j∈ I_k,t ⇒Rj(θ⋆)≤Uj,t≤k \ \ R_j(θ )\ ≤\ U_j,t\ ≤\ k [Theorem 4.3(b); definition of Ik,tI_k,t] j∈Ok,t j∈ O_k,t ⇒Rj(θ⋆)≥Lj,t>k \ \ R_j(θ )\ ≥\ L_j,t\ >\ k [Theorem 4.3(b); definition of Ok,tO_k,t] ∎ Proof of Theorem B.13. Let D be a strict partial order on the finite set [M][M]. For (c) work on G, fix t, and take =¯tD= D_t. Part (d) is deterministic and is proved for any nested pair whose closures are strict partial orders. a chain cannot repeat an element [a repetition would give (v,v)∈(v,v) by transitivity] h(j)≤M−1is well defined h(j)≤ M-1\ is well defined [previous line] (b), first claim. Let (j,l)∈(j,l) and let l1,…,lhl_1,…,l_h be a chain above j. l1,…,lh,j l_1,…,l_h,j is a chain above l \ is a chain above l [(lh,j)∈(l_h,j) by the chain, (j,l)∈(j,l) by assumption] h(l) h(l) ≥h(j)+1 \ ≥\ h(j)+1 [take h=h(j)h=h(j)] tier(j) (j) <tier(l) <tier(l) [tier=1+htier=1+h] (a). the tiers partition [M] tiers partition [M] [they are the level sets of the function tiertier] no two models of one tier are related by two models of one tier are related by D [first claim of (b)] tiers 1,…,maxjtier(j) are nonempty 1,…, _jtier(j) are nonempty [downward induction from the second claim of (b), proved next from the first claim only] (b), second claim. Let h(j)=h≥1h(j)=h≥ 1, with a longest chain l1,…,lhl_1,…,l_h above j. h(lh) h(l_h) ≥h−1 \ ≥\ h-1 [l1,…,lh−1l_1,…,l_h-1 is a chain above lhl_h] h(lh) h(l_h) ≤h−1 \ ≤\ h-1 [(lh,j)∈(l_h,j) and the first claim] tier(lh) (l_h) =tier(j)−1,and lh dominates j =tier(j)-1,\ and l_h dominates j [the two previous lines; (lh,j)∈(l_h,j) ] (c). Let l1,…,lhl_1,…,l_h be a longest chain above j, so h=h(j)h=h(j). (li,j) (l_i,j) ∈for every i every i [transitivity along the chain] Lj,t=1+#l:(l,j)∈ L_j,t=1+\#\l:(l,j) \ ≥ 1+h=tiert(j) \ ≥\ 1+h=tier_t(j) [the lil_i are distinct] Rj(θ⋆) R_j(θ ) ≥Lj,t \ ≥\ L_j,t [Theorem 4.3(b) applied to ¯t D_t, on G] For the last assertion let j have the largest tier, s; a longest chain above j, followed by j, consists of s models with strictly decreasing θ⋆θ on G. (d). t⊆t+1 _t _t+1 ⇒¯t⊆¯t+1 \ \ D_t D_t+1 [a directed path in tD_t is one in t+1D_t+1] ⇒every chain above j at time t is one at time t+1 \ \ every chain above j at time t is one at time t+1 [previous line and the definition of a chain] ⇒tiert(j)≤tiert+1(j) \ \ tier_t(j) _t+1(j) [h cannot decrease, and tier=1+htier=1+h] (e). Recall that an antichain is a set of models no two of which are related by D. Let s be the number of tiers. there is a chain of s elements is a chain of s elements [a longest chain above a model of maximal tier, followed by that model] an antichain meets a chain in at most one element [two elements of a chain are related by D] every partition into antichains has at least s blocks partition into antichains has at least s blocks [each block contains at most one element of that chain] the tiers attain s tiers attain s [they are s antichains by (a); Mirsky (1971)] ∎ Proof of Proposition 4.5. Let (ℱt)(F_t) be the filtration of Section 2, which contains the scores of all models on the items drawn so far, observed or not. Under (F) the order Π is drawn before evaluation, so retirement does not change which item is revealed next. Induction on t: AtA_t is ℱt−1F_t-1-measurable and ℱtobs⊆ℱtF^obs_t _t. A1 A_1 =[M]is ℱ0-measurable =[M]\ is F_0-measurable [deterministic] Asis ℱs−1-measurable,s≤t A_s\ is F_s-1-measurable,\ s≤ t ⇒ 1j∈As, 1j∈AsXsjare ℱt-measurable,s≤t \ \ 1\j∈ A_s\,\ 1\j∈ A_s\X_sj\ are F_t-measurable,\ s≤ t [induction hypothesis; XsjX_sj is ℱsF_s-measurable and ℱs⊆ℱtF_s _t] ⇒ℱtobs⊆ℱt \ \ F^obs_t _t [ℱtobsF^obs_t is generated by ξ and these variables] ⇒At+1is ℱt-measurable \ \ A_t+1\ is F_t-measurable [At+1A_t+1 is a measurable function of them and of ξ] Stopping times, bets, frozen wealths. σj≤t \ _j≤ t\ =j∉At+1 =\j∉ A_t+1\ [the sets AtA_t are nested, so j∈Asj∈ A_s exactly for s≤σjs≤ _j] ∈ℱtobs ^obs_t [At+1A_t+1 is a function of ξ and of the scores observed on items 1,…,t1,…,t] σjandσj∧σl _j\ and\ _j _l are (ℱt)-stopping times \ are (F_t)-stopping times [previous line makes them (ℱtobs)(F^obs_t)-stopping times, and ℱtobs⊆ℱtF^obs_t _t] every (ℱtobs)-predictable bet (F^obs_t)-predictable bet is (ℱt)-predictable \ is (F_t)-predictable [ℱt−1obs⊆ℱt−1F^obs_t-1 _t-1] (Etjl) (E^jl_t) is a nonnegative (ℱt)-supermartingale started at 1 \ is a nonnegative (F_t)-supermartingale started at 1 [Proposition B.2 with these bets, under any law with θj≤θl _j≤ _l] (Et∧σj∧σljl) (E^jl_t _j _l) is a nonnegative (ℱt)-supermartingale started at 1 \ is a nonnegative (F_t)-supermartingale started at 1 [last sentence of Lemma B.1 at the stopping time σj∧σl _j _l] Because the sets AtA_t are nested, both models were scored on every item up to σj∧σl _j _l, so St−1S_t-1 in (12) is computable and the frozen process depends on observed scores only. The analysis runs in the larger filtration; the procedure never needs the unobserved scores. The frozen processes therefore satisfy Assumption 4.1, and every result of Section 4 and Appendix B that rests on it continues to hold. ∎ Proof of Corollary B.15. Under the all-pairs rule a pair j,l\j,l\ is frozen at σj∧σl _j _l, and σl _l is the first time every pair at l, j,l\j,l\ included, is resolved (likewise σj _j). This rule depends only on observed scores, as Proposition 4.5 requires. So a pair is frozen only after it is resolved. Until then both models are active, every ZsjlZ^jl_s is observed, and the bet depends on the pair’s own past, which is the same with and without retirement. So until resolution the pair’s wealths equal those computed without retirement. If j,l\j,l\ were unresolved at time τq(j,l)Bτ^B_q(j,l) it would be unfrozen, its wealth in the true direction would equal the unretired one, which has crossed M(M−1)/αM(M-1)/α, and e-Bonferroni ⊆t _t would resolve it. Hence: σj _j ≤maxl≠jτq(j,l)B ≤ _l≠ jτ^B_q(j,l) [every pair at j is resolved by its own τq(j,l)Bτ^B_q(j,l), as shown above] ℙ(σj>t) ( _j>t) ≤∑l≠jℙ(τq(j,l)B>t)≤(M−1)e−t/βjfor integer t≥κj ≤ _l≠ jP(τ^B_q(j,l)>t)≤(M-1)e^-t/ _j integer t≥ _j [union bound; Proposition B.4(c) at each pair, since t≥2η/μqt≥ 2η/ _q and aq≥1/βja_q≥ 1/ _j] [σj] [ _j] ≤t1+∑t≥t1(M−1)e−t/βj≤t1+1+βj,t1=⌈maxκj,βjlog(M−1)⌉ ≤ t_1+ _t≥ t_1(M-1)e^-t/ _j≤ t_1+1+ _j, t_1= \ _j, _j (M-1)\ [geometric series, ∑t≥t1e−t/βj≤e−t1/βj(1+βj) _t≥ t_1e^-t/ _j≤ e^-t_1/ _j(1+ _j), and (M−1)e−t1/βj≤1(M-1)e^-t_1/ _j≤ 1] [σj] [ _j] ≤κj+βj(log(M−1)+1)+2 ≤ _j+ _j ( (M-1)+1 )+2 [t1≤maxκj,βjlog(M−1)+1t_1≤ \ _j, _j (M-1)\+1] which is (15). Under the conditions of (16), Proposition B.4(d) gives, for every pair q at j, 2η/μq≤11.2ρqη/Δq2≤11.2ρjη/Δj22η/ _q≤ 11.2 _qη/ _q^2≤ 11.2 _jη/ _j^2 and 1/aq≤min74.7,270ρq/Δq2≤min74.7,270ρj/Δj21/a_q≤ \74.7,270 _q\/ _q^2≤ \74.7,270 _j\/ _j^2, since ρq≤ρj _q≤ _j, Δq≥Δj _q≥ _j and min74.7,270ρ \74.7,270ρ\ is nondecreasing in ρ; summing over j with min74.7,270ρj≤270ρj \74.7,270 _j\≤ 270 _j and log(M−1)≤η (M-1)≤η gives the order statement. On G no tied pair is ever certified (Theorem 4.2), so a model with a tied partner is never retired. ∎ C.4 Integer programming and hardness of exact certification Proof of Proposition A.1. Identify y∈0,1y∈\0,1\^P with the relation (a,b):yab=1\(a,b):y_ab=1\ on [M][M], and write [t]=1,…,t[t]=\1,…,t\ for the call times so far. Write T(W)1\T(W)\ for the indicator vector of T(W)T(W), whose entries are (a,b)∈T(W)1\(a,b)∈ T(W)\ for (a,b)∈(a,b) . A total preorder is a reflexive, transitive and total relation, that is, a weak order as defined in Section 3. ysatisfies totality and transitivity⇔y=T(W)for a unique weak order W y\ satisfies totality and transitivity y=1\T(W)\\ for a unique weak order W [a relation is a total preorder iff it is total and transitive; T(W)T(W) determines W] EsW<1/α E^W_s<1/α ⇔∑(a,b)∈T(W)Esab<|T(W)|/α⇔gs(y)<0 _(a,b)∈ T(W)E^ab_s<|T(W)|/α g_s(y)<0 [eq. (4) with arithmetic-mean weights; |T(W)|=∑abyab|T(W)|= _aby_ab] W∈t W _t ⇔maxs∈[t]gs(y)<0 _s∈[t]g_s(y)<0 [tC_t discards W at the first call with EsW≥1/αE^W_s≥ 1/α] (j,l)∈t (j,l) _t ⇔z⋆([t])≥0 z ([t])≥ 0 [eq. (5): no W∈tW _t has (j,l)∈T(W)(j,l)∈ T(W); the minimum in (10) is over a finite set and is attained] z⋆(S) z (S) ≤z⋆([t])for every S⊆[t] ≤ z ([t]) every S [t] [for each y a maximum over fewer times is no larger] The algorithm maintains three invariants at every call time t. First, every order in the pool lies in tC_t. This holds because an order enters the pool only after passing every survival constraint and leaves at the first call at which it fails one, and tC_t is decreasing in t. Second, a pair covered by the pool is not in tD_t, because the covering order has maxs∈[t]gs<0 _s∈[t]g_s<0. Third, a pair added to D is in tD_t, by Proposition B.8 when it is added through BtjlB^jl_t, and by the last two lines above when it is added because z⋆(S∪t)≥0z (S∪\t\)≥ 0. Every pair not in D is covered when the call ends. Every pair in D stays in t′D_t for t′≥t ≥ t because t′D_t is monotone, so never revisiting a certified pair loses nothing. Hence the returned set is tD_t. Each pass through the loop for one pair either stops or adds to S a call time at which the minimizer W has gs≥0g_s≥ 0. That time is not in S∪tS∪\t\, because maxs∈S∪tgs(y)=z⋆(S∪t)<0 _s∈ S∪\t\g_s(y)=z (S∪\t\)<0 for the minimizer. Each failed pass therefore adds a new time from [t−1][t-1], so the loop stops after at most t solves. ∎ Proof of Proposition A.2. The reduction is from the feedback arc set problem in tournaments. Given a tournament G on [M][M], one arc between every two vertices, and an integer K, the problem is to decide whether some linear ordering of [M][M] has at most K backward arcs, arcs a→ba→ b with b placed above a. This problem is NP-hard (Alon, 2006; Charbit et al., 2007). Write C=(M2)C= M2. An ordering and its reverse have C backward arcs between them, so some ordering has at most ⌊C/2⌋ C/2 , and instances with K≥⌊C/2⌋K≥ C/2 are trivial. The problem therefore stays NP-hard with 1≤K≤⌊C/2⌋−11≤ K≤ C/2 -1, which we assume. Construction. Take M′=M+2M =M+2 models, the vertices of G and two more, j and l. Let C′=(M′2)C = M 2, let ν be the smallest integer with 2ν≥C′2^ν≥ C , and set c=2ν(K+12+M)/C′c=2^ν(K+ 12+M)/C and α=1/cα=1/c, which lies in (0,1)(0,1) since 2ν≥C′2^ν≥ C and K+12+M>1K+ 12+M>1. The wealth matrix is: for vertices a≠ba≠ b, Eab=2νE^ab=2^ν if a→ba→ b is an arc of G and Eab=0E^ab=0 otherwise; Ejl=0E^jl=0 and Elj=2νE^lj=2^ν; and Eja=Eaj=Ela=Eal=2ν−1E^ja=E^aj=E^la=E^al=2^ν-1 for every vertex a. Recall that (a,b)∈T(W)(a,b)∈ T(W), that is a⪯Wba _Wb, reads “b at least as good as a”, so an arc a→ba→ b contributes 2ν2^ν exactly when W places b at least as high as a. The instance has M′(M′−1)M (M -1) wealth entries in 0,2ν−1,2ν\0,2^ν-1,2^ν\ with 2ν<2C′2^ν<2C and a rational α with numerator and denominator below 2C′(K+M+1)2C (K+M+1), all of O(logM)O( M) bits, so it is constructed in polynomial time. We show that (j,l)∉1(j,l) _1 if and only if G has an ordering with at most K backward arcs. Since the feedback arc set problem is NP-hard, deciding (j,l)∈1(j,l) _1 is coNP-hard. If. Let an ordering of [M][M] have B≤KB≤ K backward arcs and let W rank the vertices by it, all strictly, with j strictly below every vertex and l strictly above. ∑(a,b)∈T(W)Eab _(a,b)∈ T(W)E^ab =2νB+M⋅2ν−1+M⋅2ν−1+0=2ν(B+M) =2^νB+M· 2^ν-1+M· 2^ν-1+0=2^ν(B+M) [backward arcs; one of (j,a),(a,j)(j,a),(a,j) per vertex, likewise for l; (j,l)∈T(W)(j,l)∈ T(W) costs Ejl=0E^jl=0] EW E^W =2ν(B+M)/C′≤ 2ν(K+M)/C′<c =2^ν(B+M)/C \ ≤\ 2^ν(K+M)/C \ <\ c [|T(W)|=C′|T(W)|=C , no ties; B≤KB≤ K] so W survives, (j,l)∈T(W)(j,l)∈ T(W), and (j,l)∉1(j,l) _1. Only if. Let W survive with (j,l)∈T(W)(j,l)∈ T(W). Ties are allowed, so let B be the number of backward arcs between vertices that W orders strictly, nvn_v the number of tied pairs of vertices, and ntien_tie the number of tied pairs of all kinds, so that |T(W)|=C′+ntie|T(W)|=C +n_tie. A tied pair of vertices costs 2ν+02^ν+0. A vertex strictly ordered against j costs 2ν−12^ν-1 and one tied with j costs 2ν2^ν, likewise for l. The pair j,l\j,l\ costs 00 if j≺Wlj _Wl and 2ν2^ν if the two are tied. Hence ∑(a,b)∈T(W)Eab _(a,b)∈ T(W)E^ab ≥ 2ν(B+nv)+2νM+2ν−1(ntie−nv) \ ≥\ 2^ν (B+n_v )+2^νM+2^ν-1(n_tie-n_v) [every tie outside the vertex pairs adds at least 2ν−12^ν-1 to the strict baseline 2ν(B+M)2^ν(B+M)] 2ν(B+M+ntie2+nv2)C′+ntie 2^ν(B+M+ n_tie2+ n_v2)C +n_tie <c=2ν(K+12+M)C′ <c= 2^ν(K+ 12+M)C [W survives; |T(W)|=C′+ntie|T(W)|=C +n_tie] B+nv2+ntie(12−K+12+MC′) B+ n_v2+n_tie ( 12- K+ 12+MC ) <K+12 <K+ 12 [multiply out and cancel 2νM2^νM] K+12+M≤M(M−1)4+M−12 K+ 12+M\ ≤\ M(M-1)4+M- 12 <(M+2)(M+1)4=C′2 < (M+2)(M+1)4= C 2 [K≤⌊C/2⌋−1≤M(M−1)/4−1K≤ C/2 -1≤ M(M-1)/4-1] B+nv2 B+ n_v2 <K+12 <K+ 12 [the coefficient of ntien_tie is positive] Now refine W to a linear ordering of the vertices: keep the strict comparisons and order each class of tied vertices so that at most half of the arcs inside it are backward. This is possible because an ordering of the class and its reverse have every internal arc backward exactly once. The arcs inside the classes number nvn_v, so the ordering has at most B+nv/2<K+12B+n_v/2<K+ 12 backward arcs, hence at most K. Allowing ties therefore cannot help an ordering survive. Realizability. With N=∞N=∞ the factor of the update (11) is 1+λZ1+λ Z with Z∈−1,0,1Z∈\-1,0,1\ for binary scores, so with bets in 0,1\0,1\ every factor is 00, 11 or 22 and every wealth is 00 or a power of two. Give each ordered pair (a,b)(a,b) its own items, on which every other pair bets 00. With bet 11, k items on which model a scores 11 and model b scores 00 make Eab=2kE^ab=2^k. With bet 11, one item on which a scores 00 and b scores 11 makes Eab=0E^ab=0. At most (ν+1)M′(M′−1)(ν+1)M (M -1) items are needed, the schedule is fixed in advance, hence predictable, and the certifier is called once, after the last item. The default mixture bet is not covered: its factors are at least 12 12, so its wealths never vanish. ∎ Appendix D Scope of the guarantee and open problems Every result stated in this paper is proved in Appendix C. Four kinds of machine check accompany the proofs but do not replace them. The counts of weak orders, 1313 of 6363 intersections for M=3M=3 and 7575 of 40954095 for M=4M=4, are verified by exhaustive enumeration, as is the three-model example of Appendix B. That the exact test certifies everything the full closure certifies is checked at M=3M=3 by enumerating all 6363 subsets for each of many random wealth matrices. The remaining containments, between e-Bonferroni, the shortcut and the exact test, are checked on random wealth matrices at M=3M=3, 44 and 55. The fourth averages the finite-benchmark wealth exactly over all item orders and confirms that its expectation is at most one at every time. These are implementation checks on finitely many inputs. What the guarantee does not cover. The method needs a scalar ability per model, and pairwise-battle data do not supply one. In battle data a platform picks two models, both answer the same prompt, and a judge picks the winner. The natural parameter is then pjl=ℙ(j beats l)p_jl=P(j beats l), and “pjl>12p_jl> 12” is not transitive in general: j may usually beat l, and l usually beat m, while m usually beats j. Lemma B.9 and the rank (1) rest on transitivity of >> on ℝR. The method also needs the set of models fixed before any data arrive, because a new entrant enlarges W and changes every T(W)T(W). Models arriving over time would require online closed testing, which admits new hypotheses as they arrive (Fischer & Ramdas, 2024), or an error budget split in advance among future entrants. Finally, the method needs the item order not to depend on the data. Lemma B.1 allows a pair to be frozen at a stopping time under both sampling models, but not the choice of which item to evaluate next: under (S) that breaks the independence of XtX_t from ℱt−1F_t-1, and under (F) it breaks the uniformity of Π(t) (t). The inverse-probability-weighted signal of Zhou et al. (2026) is a natural repair and is not worked out here. Pausing a pair and resuming it is covered under (F) only if the skipped items are scored retroactively before its wealth is updated again, since St−1S_t-1 in (12) needs every revealed difference. Scoring the skipped items forgoes the saving that pausing was meant to buy. What is open inside it. Deciding membership in tD_t is coNP-hard over the wealth matrices that the update (11) can produce with some predictable bets (Proposition A.2), by a reduction that uses bets in 0,1\0,1\. The default mixture bet keeps every wealth strictly positive, and whether it can produce such hard instances is unknown. Nothing in this paper depends on the answer: Algorithm 3 solved every instance in our experiments, and Algorithm 2 is the polynomial fallback. On the power side, Proposition B.4 and Corollary B.15 give consistency, certification times and the expected cost of retirement. That cost bound covers the all-pairs rule under (S). It does not cover the top-k rule behind the compute savings we report, under which a pair can be frozen before it is certified. Validity under either rule follows from Proposition 4.5, which needs no such bound. What remains open is a rate under (F), a cost bound for the top-k rule, and a rate for the exact test sharper than the bound it inherits from e-Bonferroni. Appendix E Experiments The data, the designs and the definition of an error are common to every experiment and are given first. Each subsection after that asks one question and answers it. Appendices E.1 to E.4 carry the numbers reported in Section 5, in the order in which it reports them, and Appendices E.5 and E.6 add experiments that Section 5 does not report. The real leaderboard. The data are the per-item results of 395395 Open LLM Leaderboard models, as released with tinyBenchmarks (Maia Polo et al., 2024; Beeching et al., 2023), on six benchmarks: MMLU (Hendrycks et al., 2021) (14,04214,042 items), HellaSwag (Zellers et al., 2019) (10,04210,042), GSM8K (Cobbe et al., 2021) (13191319), WinoGrande (Sakaguchi et al., 2020) (12671267), ARC-Challenge (Clark et al., 2018) (11721172) and TruthfulQA (Lin et al., 2022) (817817, whose per-item scores can take any value in [0,1][0,1]). The estimand is (F): the truth is the ranking by mean score on the whole benchmark, so every certified dominance can be checked against it. Three model subsets are used, the 2020 and the 88 highest-scoring models by mean score over the six benchmarks and the spread subset of 2020 models evenly spaced through the ranking of all 395395, together with all 395395 released models with no further selection. Items are revealed in a uniformly random order, redrawn 5050 times per setting, and the procedure reports every 1%1\% of the benchmark, with checkpoints at fractions 0.050.05, 0.100.10, 0.250.25, 0.500.50, 0.750.75 and 1.001.00. Throughout, α=0.05α=0.05, the bets are the mixture grid of Section 3, and the width of a rank interval is Uj,t−Lj,tU_j,t-L_j,t, zero for a model whose rank is resolved. Certification uses the shortcut for the twenty-model subsets and for all 395395, and enumeration for the eight-model subset. The simulations. Two simulated designs recur. The first streams i.i.d. items under (S), and the models share each item’s difficulty, so that they are positively dependent. Item t has difficulty dt∼N(0,1)d_t N(0,1), and model j, with logistic ability ωj _j, answers it correctly with probability 1/(1+e−(ωj−dt))1/(1+e^-( _j-d_t)), independently of the other models given dtd_t. Its six models have logistic abilities (0.5,…,0.5)(0.5,…,0.5), all tied at accuracy 0.6020.602; (1,1,0.6,0.6,0.2,−0.2)(1,1,0.6,0.6,0.2,-0.2), two tied pairs; (1,0.98,0.62,0.60,0.22,0.20)(1,0.98,0.62,0.60,0.22,0.20), near ties; (1,0.8,0.6,0.4,0.2,0)(1,0.8,0.6,0.4,0.2,0), evenly spread; or, in the bet comparison of Appendix E.6, (0.5,0.4,0.3,0.2,0.1,0)(0.5,0.4,0.3,0.2,0.1,0), a close race with gaps of about 0.020.02. True accuracies lie between 0.460.46 and 0.700.70. This design is used wherever a known tie is needed, because ties are where a multiple-comparison procedure is tested hardest. The second design is (F). It draws one benchmark of N=5000N=5000 items from the same logistic model, and the truth is each model’s accuracy on those items. It has either 2020 models with accuracies from 0.400.40 to 0.780.78 and adjacent gaps between 0.0160.016 and 0.0220.022, or 66 models from 0.500.50 to 0.700.70 with gaps of about 0.040.04, or 66 models whose two leaders have equal logistic ability. Replication counts are given with each experiment, and shaded bands in the figures are two standard errors. What counts as an error. A run counts as wrong if any report it issues, at any monitored time, certifies a dominance that is false. A rank interval that misses a true rank implies such a dominance, by Theorem 4.3(b), so this single count covers the dominances and the intervals together; the exact rank sets ℛjR_j of (5) are not scored separately. It is the error rate faced by an analyst who is free to stop at any look, which is the quantity the guarantee bounds. The fixed-sample competitors. Three constructions stand in for current practice. Holm rank sets apply Holm’s step-down correction (Holm, 1979) to the M(M−1)M(M-1) one-sided paired tests and convert the rejections into rank intervals by the counting formula (6). A step-down correction tests the hypotheses in order of their evidence and relaxes the threshold as hypotheses are rejected. Holm rank sets are the simultaneous rank intervals of Neuhof & Benjamini (2024), which Neuhof & Benjamini (2026a) also compute for a leaderboard. Here the paired test is either a paired z-test or an exact one-sided McNemar test (McNemar, 1947), which is a sign test on the items where the two models’ scores differ. Romano–Wolf step-down rank sets (Romano & Wolf, 2005; Mogstad et al., 2024) set the threshold by a bootstrap of the studentized maximum, the largest over pairs of the mean paired difference divided by its standard error. The bootstrap is a centered multiplier bootstrap with 200200 draws, which reweights the centered per-item differences by independent random multipliers. All are recomputed at each look where they are monitored. Implementation. The implementation, a few hundred lines of Python, follows Section 3 alone. Its automated tests run the machine checks listed in Appendix D. A separate simulation shows that the bound of Theorem 4.2 is nearly attained: with four tied models of accuracy 0.50.5 scored independently, 400400 i.i.d. items under (S) checked after every item, a fixed bet of 0.50.5 and α=0.2α=0.2, the true weak order is rejected at some time in 18.4%18.4\% of 40004000 runs, with a standard error of 0.6%0.6\%. E.1 Does monitoring break a fixed-sample rank set, and does ours survive it? Setup. Six models of the first simulated design, all tied, with tied pairs at ranks 1 and 3, or with near ties, are evaluated on 20002000 items, and every procedure sees the same data. The fixed-sample rank sets are recomputed at each of a number of equally spaced looks, which is the horizontal axis of Figure 1. Ours is checked at the same looks, although its guarantee would also cover a check after every item. There are 50005000 replications. Monitoring destroys the guarantee of a fixed-sample rank set and leaves ours intact. At 200200 looks the fixed-sample sets make a false statement in 30%30\% of runs with the exact McNemar test and 51%51\% with the paired z-test, against 1.6%1.6\% for our exact test. At a single pre-planned look the same two procedures are valid, at 3.3%3.3\% and 3.6%3.6\%, so what fails is the repeated look and not the normal approximation. The effect does not need exact ties. With tied pairs at ranks 1 and 3 the figures are 9.7%9.7\% and 18%18\% against our 0.3%0.3\%, and when no two models are exactly tied but some gaps are 0.0040.004 they are 4.6%4.6\% and 11.7%11.7\% against our 0.2%0.2\%. E.2 How much of a real leaderboard can be certified? Setup. The real leaderboard described above, in two forms: the three selected subsets of twenty and eight models, and all 395395 released models with no further selection. Selecting the twenty highest-scoring models is legitimate under (F), because the score matrix is fixed before the item order is drawn, but it is not a selection an operator could make prospectively, which is why the unselected run matters. Half a benchmark certifies up to half of the true dominances among close models, and 26%26\% to 86%86\% of them among all 395395 models. At half of MMLU the twenty highest-scoring models fall into 3.03.0 certified tiers with 50%50\% of the 190190 true dominances certified and a mean rank-interval width of 9.59.5, while the spread subset reaches 11.211.2 tiers. At three quarters of MMLU ours certifies 63%63\% of the top-twenty true dominances against 51%51\% for Holm and 54%54\% for Romano–Wolf, with 4.94.9 tiers against 3.33.3 and 3.63.6. Across the 900900 runs that the three subsets, six benchmarks and 5050 orders make up, exactly one report was false, in one item order of the HellaSwag eight-model set, well below α=5%α=5\%. The whole leaderboard. Removing the selection makes certification easier, even though the correction grows. With all 395395 models, 86%86\% of the 77,80277,802 true dominances of MMLU are certified at half the benchmark and 99.6%99.6\% at the end, against 83%83\% and 88%88\% for Holm, and the certified partial order has 225225 tiers against Holm’s 2626. Table 2 gives the six benchmarks; WinoGrande is the hard case, at 26%26\% of true dominances at half. Ours never certified a false dominance in 300300 runs monitored at about a hundred looks each, and neither did the Holm sets, which use the paired z-test and are recomputed at the six checkpoints. With 395395 models the shortcut threshold is M(M−1)/α=3,112,600M(M-1)/α=3,112,600, but the correction grows only as 2logM2 M while the number of easy pairs grows as M2M^2. These percentages come from the shortcut, so by Proposition B.8 the exact test would certify at least as many. Romano–Wolf is omitted here because its bootstrap over 155,630155,630 pairs is not affordable at this M. Table 2: All 395395 released models, no further selection, 5050 item orders per benchmark. True dominances certified, as a percentage of those that hold on the whole benchmark; tiers at the end. Holm uses the paired z-test. 0.250.25 0.500.50 1.001.00 tiers Benchmark items ours Holm ours Holm ours Holm ours / Holm MMLU 14,04214,042 7676 7777 8686 8383 99.699.6 8888 225225 / 2626 HellaSwag 10,04210,042 6262 6363 7777 7474 99.299.2 8181 165165 / 2424 GSM8K 13191319 6363 6666 7878 7575 98.998.9 8383 124124 / 1414 WinoGrande 12671267 55 66 2626 1717 95.595.5 3434 5353 / 55 ARC-Challenge 11721172 1818 2222 4848 3939 97.097.0 5656 6868 / 88 TruthfulQA 817817 1919 3535 5555 5353 97.097.0 6666 6969 / 1313 Why the end of a benchmark flatters our sets. The large gap at fraction 1.01.0 in Figure 2 is not a power difference. Under (F) a pair is resolved with certainty once the unrevealed items can no longer overturn it, and the offset (12) captures most of this. The fixed-sample constructions instead treat the benchmark as a sample. They therefore certify only 5555–58%58\% of the top-twenty true dominances on MMLU at the point where the ranking is in fact known exactly. Our own sets fall short of the full ranking for pairs whose margin on the whole benchmark is one or two items. The bet uses only the null’s bound on the conditional mean of the next difference. A null that has become impossible, because the revealed items already settle the pair, is therefore not by itself infinite evidence. After that point the offset sits at its lower clip −1+δ-1+δ, with δ=0.01δ=0.01, and each item multiplies the wealth by at most 1+λ(2−δ)/δ1+λ(2-δ)/δ. Pairs with a margin of one or two items have very few such items. The fair comparison is therefore at fractions up to one half, where the three procedures are close and the fixed-sample sets are ahead early on. E.3 How much compute does retirement save? Setup. Cost is counted in model-item evaluations against a fixed benchmark of N=5000N=5000 items, divided by what a full evaluation of every model would cost, with 300300 replications and reports every 2525 items. Five retirement rules are compared: no retirement; stopping everything once every model’s top-k status is certified; retiring a model once its own top-k status is certified, which is the rule of Section 3; the same but only once every pair between the model and a model whose status is still open is certified, so that a retirement can never block another certification; and, for the full ranking, retiring a model once every pair involving it is certified. The group-sequential comparison is a Pocock design (Pocock, 1977), as used by Arviv et al. (2026), who add items in batches of 100100. Our implementation compares a z-statistic with one constant critical value at each of ten equally spaced looks fixed in advance, without a multiplicity correction, and a second version adds a Bonferroni correction over the 380380 ordered pairs. Both versions close their certifications under transitivity. Retiring each model as soon as its own question is answered costs about a fifth of a full evaluation, and stopping everything at one moment costs four times as much. Certifying the top-33 status of all 2020 simulated models costs 22%22\% of a full run under the retirement rule against 84%84\% under the common stopping rule, with no report ever wrong. With six models and k=2k=2 the three top-k rules cost 44%44\%, 22%22\% and 30%30\%: the cautious fourth rule gained nothing, since the plain rule already reached the goal in every run, as did no retirement at all. Certifying the entire ranking is far more expensive, 51%51\% of a full run for six models and 90%90\% for twenty, because adjacent gaps of about 0.020.02 are resolved only when the finite benchmark is nearly exhausted. The mixture bet cost less than a fixed bet of 0.250.25 in every setting but one, for example 51%51\% against 63%63\% for the full ranking of six models. The exception is two leaders of equal logistic ability among six models, where the fixed bet was slightly cheaper under the three top-k rules. The cost barely depends on k. Between k=2k=2 and k=16k=16 among twenty close models the cost stays at 2121–24%24\% of a full run, because it is set by the models adjacent to the kkth place and not by k itself. At k=1k=1 and k=19k=19 it falls to 18%18\% and 16%16\%, where only one side of the boundary has to be resolved. Ours reaches the goal in 97.597.5–99.5%99.5\% of runs at every k, over 200200 replications. On the real leaderboard. The top-33 goal costs 31%31\% of a full MMLU run for the twenty highest-scoring models and 4%4\% for the spread subset, and 57%57\% of GSM8K and 68%68\% of HellaSwag for the top twenty, reached in 96%96\% to 100%100\% of orders. Where the top twenty are indistinguishable, on ARC-Challenge and WinoGrande, retirement saves almost nothing, as expected. Table 3 shows that applying the same rule to all 395395 models costs a smaller fraction still, because most models on the leaderboard are far from the top-33 boundary and retire almost at once, whereas the twenty highest-scoring are near-ties by construction. The comparison is of fractions and not of compute: 6.2%6.2\% of the 395395-model evaluation of MMLU is about four times the absolute cost of 31%31\% of the twenty-model one. Every model’s status is settled in 100%100\% of orders on MMLU and HellaSwag. On the other four benchmarks some models near the boundary stay unsettled, about two on ARC-Challenge and WinoGrande, three on GSM8K and nine on TruthfulQA. The all-or-nothing goal is therefore reached in only 00–2%2\% of orders, while 97.797.7–99.5%99.5\% of models are settled. No report was wrong in any of these runs. Table 3: Certifying every model’s top-33 status, all 395395 models, 5050 item orders per benchmark. Cost is model-item evaluations divided by 395N395N; settled is the mean fraction of models whose top-33 status is certified at the end. Benchmark cost settled MMLU 6.2%6.2\% 100.0%100.0\% HellaSwag 15.2%15.2\% 100.0%100.0\% GSM8K 16.7%16.7\% 99.2%99.2\% WinoGrande 47.3%47.3\% 99.5%99.5\% ARC-Challenge 36.9%36.9\% 99.5%99.5\% TruthfulQA 27.3%27.3\% 97.7%97.7\% Against a group-sequential design. The Pocock design is as cheap as ours and answers the question far less often. With the twenty simulated models it reaches the goal in 56%56\% of runs against our 99%99\%, because a z-test cannot use the fact that the benchmark is finite and its looks must be fixed in advance. With no multiplicity correction and only asymptotic validity, it also errs in 2.7%2.7\% of runs at that cost, and in 3.7%3.7\% without retirement. The same holds across the sweep over k, where the uncorrected design reaches the goal in 5656–68%68\% of runs with a false statement in 33–4%4\%. A Bonferroni correction supplies the missing multiplicity correction but removes the saving. The corrected version costs 32%32\% and reaches the goal in 3%3\% of runs. With two leaders of equal logistic ability among six models, whose accuracies on the realized benchmark differ only by chance, the uncorrected design errs in 1%1\% of runs, and in 2%2\% without retirement, while ours errs in none, at an equal cost of 12%12\%. All costs are averaged over every run, whether the goal was reached or not. E.4 What does anytime validity cost? Setup. The comparison is made at the one look where a fixed-sample procedure is valid, so that it measures only the power given up for the right to monitor. In simulation the first design runs to 20002000 items, with either Holm construction run once and our exact test monitored at 200200 looks. On the real leaderboard the three fixed-sample constructions are compared with ours on the twenty highest-scoring models, with Romano–Wolf computed at the six checkpoints only, since its bootstrap is too slow for every look. In simulation the price of anytime validity is a fraction of one dominance. After 20002000 items with two tied pairs, both Holm constructions certify 12.9812.98 of 1313 true dominances and ours 12.7912.79; with near ties the figures are 11.9711.97 and 11.6211.62 of 1515. On the real leaderboard the fixed-sample sets lead by up to eight percentage points at a quarter of a benchmark, and by half that lead is gone: ours matches or exceeds Romano–Wolf on five of the six benchmarks and is behind by four points on TruthfulQA. Table 4 shows Romano–Wolf to be the strongest of the three. Beyond one half ours pulls ahead for a reason unrelated to power, given in Appendix E.2. This cost buys the right to look: the same fixed-sample sets, once monitored, err in up to 4%4\% of orders (McNemar on TruthfulQA aside), and ours in none. The monitoring penalty is smaller here than in the tied simulations because the gaps among the top twenty are mostly nonzero. Table 4: Fixed-sample constructions against ours, twenty highest-scoring models, 5050 item orders. Top: true dominances certified at one half, as a percentage of those that exist. Bottom: probability that some report, at any look, was false. MMLU HellaSwag GSM8K WinoGrande ARC-Challenge TruthfulQAa True dominances certified at one half of the benchmark (%) Ours, monitored 50 49 22 2 0 17 Holm, McNemar 47 47 19 1 0 61b Holm, paired z-test 48 48 20 1 0 16 Romano–Wolf 49 49 22 1 0 21 Probability of a false statement at some look Ours, monitored 0 0 0 0 0 0 Holm, McNemar 0 0.02 0 0 0 1.00b Holm, paired z-test 0 0.02 0.04 0 0.02 0 Romano–Wolf 0.02 0.02 0.02 0.04 0.02 0 aPer-item scores in [0,1][0,1], not binary. bThe sign test of McNemar’s procedure targets the median of the paired differences, which is not the mean-score estimand for non-binary scores. Its certifications on TruthfulQA are not comparable and every order contained a false one. This is a test mismatch, not a monitoring effect. E.5 Does the guarantee depend on how model errors are correlated? Setup. Each regime holds every model’s marginal accuracy fixed and varies only the dependence among its item-level scores, so that any difference between regimes is attributable to dependence alone. For six models and 40004000 i.i.d. items under (S), with the accuracies of the first simulated design as marginals, binary scores come from a Gaussian copula, Xtj=ajγGt+1−γεtj≤Φ−1(θj)X_tj=1\a_j γ\,G_t+ 1-γ\, _tj≤ ^-1( _j)\ with GtG_t and εtj _tj independent standard normal, Φ the standard normal distribution function, γ∈[0,1)γ∈[0,1) the weight of the shared factor GtG_t and aj∈−1,1a_j∈\-1,1\ the sign with which model j loads on it. This gives ℙ(Xtj=1)=θjP(X_tj=1)= _j for every γ and every choice of signs. The four regimes are independence (γ=0γ=0), positive equicorrelation at γ=0.5γ=0.5 and γ=0.9γ=0.9 with every aj=1a_j=1, and a mixed-sign regime at γ=0.8γ=0.8 in which three models load positively and three negatively; the resulting binary correlations are about 00, 0.320.32, 0.700.70 and ±0.55± 0.55. Monitoring is every 2020 items, with the mixture bet and the exact certifier, 50005000 replications for the tied configurations and 20002000 for the others, and the same code runs unchanged in every regime. Validity is unchanged across dependence structures and speed is not. With six tied models the anytime false-statement probability is 2.0%2.0\%, 1.5%1.5\%, 1.6%1.6\% and 1.6%1.6\% across the four regimes, and with two tied pairs it is 0.6%0.6\%, 0.4%0.4\%, 0.4%0.4\% and 0.5%0.5\%, all at α=0.05α=0.05. Table 5 collects the four regimes. Speed depends strongly on the regime. With evenly spread logistic abilities at half the items the procedure certifies 70%70\% of the true dominances under independence, with a mean rank-interval width of 1.491.49 and 3.33.3 tiers, against 96%96\%, 0.220.22 and 5.35.3 under strong positive dependence, where certifying every model’s top-33 status costs 16%16\% of a full evaluation instead of 41%41\%. The mixed-sign regime is less favorable by design, since pairs within a group benefit from positive dependence while pairs across groups are negatively dependent and harder to resolve: it certifies more true dominances than independence at the halfway point but its top-33 cost is 46%46\% and the goal is reached in only 26%26\% of runs. With near ties, independence and the two positive regimes certify 74%74\%, 79%79\% and 80%80\% at half. Here 80%80\% is a practical ceiling, because the three near-tied pairs, with gaps of about 0.0040.004, are almost never resolved within 40004000 items. The mixed-sign regime is slightly slower than independence, at 72%72\%. The regimes differ in the variance of the paired difference, Var(Ztjl)=Var(Xtj)+Var(Xtl)−2Cov(Xtj,Xtl)Var(Z^jl_t)=Var(X_tj)+Var(X_tl)-2Cov(X_tj,X_tl): positive covariance makes the paired signal less noisy and negative covariance more noisy, so a pair with margin 0.0750.075 has paired-difference variance about 0.450.45, 0.300.30 and 0.140.14 under independence, γ=0.5γ=0.5 and γ=0.9γ=0.9, and its median first-certification time falls from 28%28\% to 18%18\% to 9%9\% of the items. Table 5: Four dependence regimes with identical marginal accuracies. “Anytime error” uses six tied models; the other columns use evenly spread logistic abilities after half the items, with the probability of reaching the top-33 goal in parentheses. Dependence Anytime error True dominances Mean rank width Tiers Top-33 cost (goal) Independent 2.0% 70% 1.49 3.3 41% (49%) Positive, γ=0.5γ=0.5 1.5% 76% 1.18 3.7 32% (79%) Positive, γ=0.9γ=0.9 1.6% 96% 0.22 5.3 16% (100%) Mixed sign, γ=0.8γ=0.8 1.6% 80% 1.01 3.9 46% (26%) E.6 What does the multiplicity correction contribute? Setup. Every certifier receives the same wealth paths, so only the multiplicity correction varies. Four certifiers are compared: e-Bonferroni with closure, which is the certification logic of Gu et al. (2026); the shortcut of Algorithm 2; the exact test of Algorithm 1; and a variant of the exact test whose weights in (4) are uniform over the pairs of T(W)T(W) in the same or in adjacent levels of W, not over all of them. The simulated comparison uses six models over 40004000 items monitored every 2020, with 20002000 replications; every error rate was at most 1.9%1.9\%. The real comparison reuses the wealths of Appendix E.2 unchanged. The exact test certifies more pairs than e-Bonferroni, mostly early in evaluation, and never fewer. On simulated data the exact test certifies about 7%7\% more true dominances than e-Bonferroni after 500500 items and at most 3%3\% more by 40004000, because a threshold enters the sample size an e-process needs only through its logarithm: halving the multiplicity factor saves log2 2 divided by the growth rate, a fixed number of items. The counts are 5.105.10 against 4.774.77 of 1515 true dominances after 500500 items with near ties and 5.815.81 against 5.455.45 with evenly spread logistic abilities, becoming 12.0112.01 against 12.0012.00 and 13.2313.23 against 12.9012.90, respectively, by 40004000. On the real wealths the exact test certifies 22–5%5\% more than e-Bonferroni among the eight highest-scoring models up to three quarters of a benchmark and nearly the same number at the end, ahead at 20%20\% of the looks and behind at none. Table 6 gives the four certifiers side by side. The shortcut and e-Bonferroni with closure are nearly identical in these experiments. Here transitivity pooling guarantees that nothing is lost but brings no measurable gain in power. Over 600600 further simulated runs the shortcut was ahead at 121121 of 120,000120,000 looks and behind at none. Adjacent-level weights are valid by Remark B.12 but lose most of the power early, certifying 0.30.3 and 0.40.4 pairs at 500500 items, because for a strict order they average only the M−1M-1 adjacent pairs and give zero weight to the non-adjacent pairs, whose large wealths are what eliminates orderings far from the truth. Table 6: True dominances certified from identical wealths, so that only the correction differs; means over 5050 item orders and the six benchmarks, and for twenty models also over the top and spread subsets. The last two columns count the looks at which the row beat or trailed the reference. fraction of the benchmark looks vs. e-Bonferroni 0.050.05 0.100.10 0.250.25 0.500.50 0.750.75 1.001.00 ahead behind Eight highest-scoring models, 27.527.5 true dominances, 31,60031,600 looks e-Bonferroni with closure (reference) 2.15 3.11 4.86 7.51 12.62 26.04 reference Shortcut (Algorithm 2) 2.15 3.11 4.86 7.51 12.63 26.04 118 0 Exact (Algorithm 1) 2.19 3.21 4.99 7.90 13.02 26.11 6,320 0 Exact, adjacent-level weights 2.04 2.35 3.90 4.97 7.87 26.20 831 15,052 Twenty models, top and spread subsets pooled, 187.8187.8 true dominances, 63,20063,200 looks e-Bonferroni with closure (reference) 27.0 37.9 64.5 89.5 110.7 177.6 reference Shortcut (Algorithm 2) 27.0 37.9 64.5 89.5 110.7 177.6 473 0 The exact test and its adjacent-level variant enumerate every weak order, 545,835545,835 of them for eight models but about 2.7×10212.7× 10^21 for twenty, which is why only the two closed-form certifiers appear in the lower half of Table 6. Beyond eight models, by integer programming. Computed by integer programming, the exact test certifies only slightly more than the shortcut. On the twelve real settings at M=20M=20 the shortcut trails Algorithm 3 by 1.41.4 true dominances at half a benchmark, with a range of 0.10.1 to 2.92.9, and by 0.30.3 at the end. Table 7 reports every setting. The shortcut is never ahead, as Proposition B.8 requires. The gap is largest while certification is in progress, as on the spread subsets at one half, and nearly vanishes once the shortcut has certified nearly everything, as at the end of every spread subset, or while nothing is certified yet, as on ARC-Challenge and WinoGrande among the top twenty. Neither certifier was ever wrong, and Algorithm 3 solved every instance. Cost is modest: a run with looks every 1%1\% of a benchmark needs about 294294 integer programs at 2626 ms each, eight seconds in all, on one CPU core with HiGHS through SciPy and the relative optimality gap set to zero. On simulated runs of 30003000 items with looks every 2020, the times are about 0.20.2, 1515 and 6363 seconds at M=10M=10, 2020 and 3030, with 8585, 388388 and 935935 programs; there the exact test and the shortcut certify the same 44.744.7 of 4545, 189.3189.3 of 190190 and 431.3431.3 of 435435 true dominances. Table 7: The exact test by integer programming against the shortcut, at M=20M=20 with 5050 item orders per row, drawn independently of those behind Table 1, so the shortcut columns differ slightly from it. True dominances certified, rank width at one half, and what the exact test cost to run. 0.250.25 0.500.50 1.001.00 width at 0.500.50 cost Benchmark, subset true exact shortcut exact shortcut exact shortcut exact shortcut programs s MMLU, top 190 84.1 84.1 93.6 92.4 187.3 187.1 9.64 9.76 244 6 HellaSwag, top 190 77.4 76.3 94.9 94.1 181.7 181.3 9.51 9.59 292 8 GSM8K, top 190 14.4 13.4 45.1 43.1 183.2 182.8 14.49 14.69 331 9 WinoGrande, top 180 0.1 0.1 3.3 3.1 153.9 153.2 18.67 18.69 238 6 ARC-Challenge, top 179 0.0 0.0 0.3 0.2 148.2 147.1 18.97 18.98 213 6 TruthfulQA, top 190 1.0 0.6 33.2 30.9 156.0 155.6 15.68 15.91 285 7 MMLU, spread 190 165.5 164.7 173.3 172.9 189.6 189.5 1.67 1.71 217 6 HellaSwag, spread 190 118.7 116.8 146.9 145.5 187.5 187.5 4.31 4.45 304 8 GSM8K, spread 190 139.2 136.9 162.7 161.3 189.4 189.3 2.73 2.87 308 8 WinoGrande, spread 186 42.9 40.6 92.6 89.8 184.0 183.8 9.74 10.02 360 10 ARC-Challenge, spread 189 70.2 67.2 116.3 114.3 188.8 188.8 7.37 7.57 358 9 TruthfulQA, spread 190 79.6 76.3 124.7 122.6 185.7 185.4 6.53 6.74 373 10 The bet matters more than the correction. A bet with negative growth can keep a pair from ever being certified, whatever the correction. A fixed bet of 0.250.25 never certifies pairs whose gap is about 0.020.02: with six models at accuracies 0.500.50 to 0.600.60 it certifies 6.56.5 of 1515 true dominances after 50005000 items and 7.07.0 after 20,00020,000, against 14.014.0 for the mixture bet, 13.513.5 for the plug-in bet and 13.013.0 for online Newton steps. This is the negative-growth effect of Remark B.3, which no multiplicity correction can undo. The price of adapting is paid early and only where the fixed bet happens to be well tuned: with accuracies evenly spaced from 0.500.50 to 0.700.70, where 0.250.25 is close to the log-optimal bet λ⋆=Δ/ρλ = /ρ, the fixed bet leads after 500500 items, 6.96.9 pairs against 5.85.8, but the mixture overtakes it by 20002000 and leads clearly by 40004000, 13.213.2 against 11.111.1. The mixture was the best adaptive bet at every checkpoint of every setting, which is why it is the default. These runs use 10001000 replications, or 300300 for the close race at 20,00020,000 items, and every error rate was at most 3.2%3.2\%. Appendix F Related work in detail Table 8 places the closest methods side by side; the rest of this appendix says in words what the columns compress, and then relates the construction to the statistical tools on which it builds. Table 8: Closest existing methods. Monitoring: valid under continuous looks and any stopping rule. Finite-sample: no asymptotics. Structure: the correction uses the relations among pairs. Rank sets, simultaneous Monitoring Finite-sample Structure Early stop of evaluation Fixed-sample rank inference Mogstad et al. (2024) yes no no no no Al Mohamad et al. (2022) yes no yes no no Al Mohamad et al. (2021) yes no yes yes no Neuhof & Benjamini (2024) yes no yesa no no Neuhof & Benjamini (2026a) yes no yesa no no Neuhof & Benjamini (2026b) no no yesa no no Li et al. (2026); Avelar Menendez et al. (2026) yes no no no no Chandra & Xie (2025) yes no yes (simulable noise) n/a no Sequential evaluation Zhou et al. (2026); Hsu & Shekhar (2026) no (one model) yes yes n/a yes Kotawala (2026) no (pairs) yes yes no correctionb yes (pairwise) Gu et al. (2026), battles yes yes yes no (Bonferroni) yes (pairwise) Arnold et al. (2026) no (superior set) yes yes no (generic closure) no Arviv et al. (2026) no (pairs) pre-set looks no no correction yes Heckel et al. (2019) no (partition at stopping) own rule only yes no (union bound) yes This paper yes yes yes yes yes aGiven finite-sample pairwise tests. bBonferroni and Holm only in its separate fixed-sample analysis. Fixed-sample rank sets are valid for one look at one pre-chosen sample size, and the constructions of Mogstad et al. (2024), Al Mohamad et al. (2022), Neuhof & Benjamini (2024) and their leaderboard descendants (Neuhof & Benjamini, 2026a; Neuhof & Benjamini, 2026b; Li et al., 2026; Avelar Menendez et al., 2026) set a critical value for a fixed sample size, mostly by asymptotic approximation. They then apply the same counting formula as our (6). The first difference is the certified set D to which the formula is applied. Ours is valid at all times and uses the order structure inside the multiplicity correction. The second is that enumeration adds the exact rank sets of Theorem 4.3(a), which can be strictly tighter than counting (Theorem 4.3(c)). Al Mohamad et al. (2021) instead test one configuration of the ranks at a time with the partitioning principle, which uses the order structure and gives rank sets without counting, at one fixed sample size. Our exact test is the anytime-valid counterpart of their procedure. Group-sequential stopping needs its looks fixed in advance and a normal approximation, and as applied so far it has no correction across pairs. Ours needs neither and corrects across all pairs. Arviv et al. (2026) report large savings from Pocock boundaries (Pocock, 1977), one constant critical value at each pre-scheduled look, on pairwise comparisons. They leave multiplicity across pairs open, and Appendix E.3 measures the cost of this. Active-ranking algorithms (Heckel et al., 2019) choose which comparisons to make adaptively, which we do not. They guarantee their answer only at their own stopping time, when they output a grouping of the models. Ours guarantees every intermediate report under any stopping rule. Among the anytime-valid methods, the two closest differ from ours in what they are able to certify and in how they treat the order structure. Gu et al. (2026) control the family-wise error rate by Bonferroni over the ordered pairs. They use transitivity only to propagate certified preferences and need transitive preferences only for that step. They assume i.i.d. prompts and give no bound for a fixed benchmark sampled without replacement. In ours transitivity instead follows from scalar abilities. It restricts the sets of true hypotheses that the multiplicity procedure has to test at all, which is why the exact test never certifies less than the full e-value closure of Hartog & Lei (2025). On the same wealths, the shortcut never certifies less than the Bonferroni correction of Gu et al. (2026), because its statistic is the pair’s own wealth plus nonnegative terms. Arnold et al. (2026) certify which models may be best, a statement about rank one only, through a closure over all subsets of models that uses no relation among the pairwise hypotheses. Ours gives simultaneous rank inference for common-item benchmark scores, with validity on a fixed benchmark that rests only on the random item order. The construction builds on several established statistical literatures. Multiple comparisons with the best (Hsu, 1996) is the classical form of “which models are tied for first”, and our certified tier 11 is its anytime-valid analog. Step-down resampling (Romano & Wolf, 2005) is the fixed-sample engine behind Mogstad et al. (2024). Betting martingales (Waudby-Smith & Ramdas, 2024), time-uniform confidence sequences (Howard et al., 2021) and Ville’s inequality (Ville, 1939) supply the sequential machinery. Fischer & Ramdas (2024) show that online closed testing procedures that cannot be uniformly improved must use e-values. This is a natural route to models arriving over time. On the evaluation side, Miller (2024) argues for error bars on benchmark scores, computed from item-level results, Bowyer et al. (2025) argue against the normal approximation at small sample sizes, and Chiang et al. (2024) describe the battle format we leave out. Singh et al. (2025) document how data-dependent practices, such as testing many private variants and disclosing only the best, shape public leaderboards. Time-uniform guarantees cover a data-dependent choice of when to stop or report one evaluation, but not the selection of which model variants to disclose.