Paper deep dive
What Fixed-Rollout pass@k Evaluations Can Identify
Pranav Singh, Prashant Singh
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 9/10/2026, 2:33:04 AM
Summary
This paper demonstrates that fixed-rollout pass@k evaluations (where n samples are taken per problem) only identify the first n moments of the latent per-task success distribution. Consequently, pass@k is directly identified for k ≤ n, but for k > n, the value is generically partially identified, meaning extrapolated failure rates and tail constants cannot be uniquely determined from the count law alone, even with infinite tasks. The authors provide exact constructions of incompatible extrapolations and compute sharp identified intervals using Hausdorff principal representations. Calibration on the Brown et al. dataset shows that counterfactual n=16 evaluations leave failure at k=1000 ambiguous by factors of 1.5 to over 2,600, highlighting that intermediate-scale failure share does not determine extrapolation width.
Entities (9)
Relation Signals (6)
fixed-n success counts → identifies → n free moments
confidence 95% · fixed-n success counts identify only the n free moments of the latent per-task success distribution.
pass@k → isidentifiedfor → k <= n
confidence 95% · direct pass@k is identified for k <= n
pass@k → isnotidentifiedfor → k <= n
confidence 95% · generic extrapolated pass@k ... are not identified for k > n
Brown et al. dataset → usedforcalibration → identification bounds
confidence 92% · On the public 10,000-rollout-per-problem release of Brown et al., counterfactual n = 16 evaluations leave failure at k = 1000 ambiguous
Hausdorff principal representations → computes → sharp population identified intervals
confidence 90% · compute sharp population identified intervals through Hausdorff principal representations.
Beta-Binomial model → understates → residual failure
confidence 88% · under the latter, the Beta–Binomial forecast understates residual GSM8K failure by at least 85×.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/random-task conditional-Binomial model, fixed-n success counts identify only the n free moments of the latent per-task success distribution. Consequently, direct pass@k is identified for k <= n, but generic extrapolated pass@k, tail exponents, and tail constants are not identified for k > n, even with arbitrarily many exchangeable tasks at the same rollout budget. This is stronger than the observation that the usual estimator is undefined beyond n: it characterizes the information missing from the fixed-depth count-law experiment. We give exact count-law-preserving constructions with incompatible extrapolations, state the exceptional unique-extension case, and compute sharp population identified intervals through Hausdorff principal representations. On the public 10,000-rollout-per-problem release of Brown et al., counterfactual n = 16 evaluations leave failure at k = 1000 ambiguous by factors from 1.5 to over 2,600 across four MATH/GSM8K/CodeContests configurations. The calibration shows that intermediate-scale failure share alone does not determine width. Our result does not reject parametric inference-time scaling laws; it supplies the nonparametric baseline against which their assumptions can be evaluated. We give an exact, conservative one-coordinate finite-task confidence certificate and a reporting standard separating direct estimates, identified sets, and model-conditioned forecasts.
Tags
Links
- Source: https://arxiv.org/abs/2609.09245v1
- Canonical: https://arxiv.org/abs/2609.09245v1
Trouble viewing inline? Open PDF directly →
Full Text
47,406 characters extracted from source content.
Expand or collapse full text
What Fixed-Rollout pass@k Evaluations Can Identify Pranav Singh Affiliation: Department of Mathematics Affiliation: Indian Institute of Technology Ropar Affiliation: Ropar, Punjab 140001, India Email: 2023mcb1308@iitrpr.ac.in Prashant Singh Affiliation: Department of Mathematics Affiliation: Indian Institute of Technology Ropar Affiliation: Ropar, Punjab 140001, India Email: 2023mcb1309@iitrpr.ac.in Abstract Repeated-sampling evaluations increasingly extrapolate pass@k far beyond the number n of samples collected per problem. We show that, in the pooled/random-task conditional-Binomial model, fixed-n success counts identify only the n free moments of the latent per-task success distribution. Consequently, direct pass@k is identified for k≤nk≤ n, but generic extrapolated pass@k, tail exponents, and tail constants are not identified for k>nk>n—even with arbitrarily many exchangeable tasks at the same rollout budget. This is stronger than the observation that the usual estimator is undefined beyond n: it characterizes the information missing from the fixed-depth count-law experiment. We give exact count-law-preserving constructions with incompatible extrapolations, state the exceptional unique-extension case, and compute sharp population identified intervals through Hausdorff principal representations. On the public 10,000-rollout-per-problem release of Brown et al., counterfactual n=16n=16 evaluations leave failure at k=1000k=1000 ambiguous by factors from 1.51.5 to over 2,6002,600 across four MATH/GSM8K/CodeContests configurations. The calibration shows that intermediate-scale failure share alone does not determine width: one GSM8K configuration is nearly point-identified at n=64n=64, while a controlled stress test fixes target failure, window share, and hard-task dispersion yet changes sharp width by orders of magnitude. The same calibration separates a raw zero-atom reference from a Jeffreys-smoothed latent-probability convention: under the latter, the Beta–Binomial forecast understates residual GSM8K failure by at least 85×85×. Our result does not reject parametric inference-time scaling laws; it supplies the nonparametric baseline against which their assumptions can be evaluated. We give an exact, conservative one-coordinate finite-task confidence certificate and a reporting standard separating direct estimates, identified sets, and model-conditioned forecasts. 1 Introduction Repeated sampling can substantially improve LLM coverage on mathematical, programming, and safety evaluations. Brown et al. (2024) fit exponentiated power laws to these curves over large sampling budgets. Schaeffer et al. (2025) explain aggregate power-law failure through a left power-law tail of per-problem pass@1 probabilities. Most recently, Kazdan et al. (2025) criticize common curve fits, advocate Beta–Binomial extrapolation, and use adaptive sampling to forecast pass@k far beyond the observed per-problem budget. These are important practical methods, but they invite a prior question. If an evaluation observes only n rollouts per problem, what does its complete success-count distribution identify about pass@k for k>nk>n? The answer is not merely that the familiar estimator is undefined there. Fixed-n counts identify only n free moments of the latent task-difficulty distribution. Thus, even with arbitrarily many benchmark problems at the same rollout budget, generic extrapolated pass@k remains a partially identified functional. This distinction matters directly for the inference-time scaling literature. Schaeffer et al.’s left-tail premise concerns an unobserved mixing distribution; a fitted Beta–Binomial model similarly supplies unobserved moments. These may be useful modeling assumptions, but they are not discoveries recovered nonparametrically from fixed-n counts. We quantify the gap rather than arguing against parametric forecasting: a published forecast should be read alongside the sharp set of all extrapolations compatible with its fixed-rollout count law. Contributions. 1. We give a general identification theorem: fixed-n Binomial counts identify expectations of every polynomial functional of the latent success probability of degree at most n. For an interior truncated Hausdorff moment sequence, extrapolated pass@k has a nondegenerate identified interval for every k>nk>n; rank-deficient boundary sequences can instead uniquely determine their mixing law. Direct pass@k for k≤nk≤ n is a corollary. 2. We give two exact constructions with identical count laws and incompatible extrapolations. One separates polynomial from exponential failure; the other holds the left-tail exponent fixed while changing its leading constant. For even n, we compute sharp identified intervals with Hausdorff principal representations, rather than a support-grid approximation. 3. We calibrate the result on the public 10,000-rollout-per-problem release of Brown et al. In four MATH/GSM8K/CodeContests configurations, the sharp n=16n=16 interval at k=1000k=1000 ranges from 1.5×1.5× to over 2,600×2,600× in failure probability. At n=64n=64, raw plug-in sets can be nearly point-like when the full count law isolates the component carrying target-budget failure. Separating the raw zero-frequency reference from a Jeffreys-smoothed convention materially changes the calibration: conditional Beta–Binomial forecasts are 4.5–29% above the upper posterior endpoint on three configurations and understate residual GSM8K failure by more than 85×85×. A controlled synthetic stress test additionally shows that intermediate-window mass and hard-task dispersion, even jointly, cannot characterize sharp-set width: the remaining truncated-moment geometry can change it by orders of magnitude. 2 Setting and directly relevant work Fix a benchmark, prompt, decoding distribution, resource limit, answer extraction rule, and scorer. Let Yij∈0,1Y_ij∈\0,1\ denote the complete evaluation-pipeline outcome for completion j of task i. We use the standard mixture model Yij|pi∼iidBernoulli(pi),Ci=∑j=1nYij.Y_ij p_i iid Bernoulli(p_i), C_i= _j=1^nY_ij. Here pip_i is operational: changing a cap, parser, or scorer changes the quantity being measured. Conditional iid sampling is the usual premise of count-based pass@k evaluation. It is also implied by an infinitely exchangeable completion sequence through de Finetti’s representation (Hewitt and Savage, 1955); finite exchangeability alone is not enough. Our primary identification experiment is the random-task (or pooled-count) experiment: pi∼iidFp_i F, after which only the marginal count law is observed. Equivalently, F can be a stated empirical mixing reference used to generate a pooled count law. Mean failure and coverage are RF(k)=F[(1−p)k],pass@k=1−RF(k).R_F(k)=E_F[(1-p)^k], @k=1-R_F(k). For k≤nk≤ n, the usual direct estimator is R^(k)=1N∑i=1N(n−Cik)(nk). R(k)= 1N _i=1^N n-C_ik nk. Conditional on the pip_i’s, it is unbiased for N−1∑i(1−pi)kN^-1 _i(1-p_i)^k. This is the estimator introduced for code-generation pass@k by Chen et al. (2021); it is a design-based estimate, not an extrapolation model. A fixed labeled benchmark is different. If one treats (p1,…,pN)(p_1,…,p_N) as an unrestricted but fixed finite parameter vector and retains its labeled product likelihood, that finite-dimensional likelihood is formally injective: each Binomial marginal identifies its own pip_i. We do not call N−1∑i(1−pi)kN^-1 _i(1-p_i)^k nonidentified in that product model. Our theorem instead concerns the information in the pooled/random-task count law, the asymptotic experiment obtained by increasing the number of exchangeable tasks while holding the per-task depth fixed. Under that experiment the counts are iid from one q, and two mixing laws with the same q induce exactly the same data distribution for every number of tasks. The fixed-benchmark calibration in Section 5 uses its empirical distribution only as a declared reference for this pooled-count experiment. Where this paper sits. Brown et al. (2024) fit an exponentiated power law to repeated-sampling coverage. Schaeffer et al. (2025) derive aggregate polynomial failure from a left power-law tail in F. Kazdan et al. (2025) show weaknesses of log–log and discretized-Beta fits, propose a Beta–Binomial likelihood, and allocate additional samples to hard problems. Liu (2026) derives sharp two-moment regions for majority-vote accuracy; our setting uses arbitrary rollout depth and the at-least-one-success pass@k functional. Hariri et al. (2026) instead propose a Bayesian Dirichlet evaluation and ranking procedure, while Chen et al. (2026) study partial identification of a latent label under LLM misclassification. These works share the premise that repeated-query design governs inference, but their estimands and observable structures differ from a Binomial mixture over task-level pass probabilities. The estimator-domain observation that equation (3) requires k≤nk≤ n appears in the scaling literature. Our added claim is stronger but model specific: in the pooled/random-task count-law experiment, the pass@k functional is generically partially identified beyond n, even in the infinite-N limit. Therefore a Beta–Binomial or tail-law forecast is conditional on its mixing model, however well it predicts held-out data. Mathematically, the analysis is a truncated Hausdorff moment problem (Hausdorff, 1921) and a partial-identification problem (Manski, 2003). Principal representations and moment–SOS methods provide sharp generalized-moment bounds (Krein and Nudelman, 1977; Lasserre, 2001). Our contribution is to connect those objects to the exact target and released data products of multi-rollout LLM evaluation. 3 The fixed-rollout identification boundary Let nP_n be the polynomials of degree at most n, and write qc=Pr(C=c)=∫(nc)pc(1−p)n−cF(p)q_c= (C=c)= ncp^c(1-p)^n-c\,dF(p). Theorem 1 (Fixed-rollout boundary) For fixed n, the count law q=(q0,…,qn)q=(q_0,…,q_n) identifies ∫h(p)F(p) h(p)\,dF(p) for every h∈nh _n. Equivalently, it identifies mj=∫pjF(p)m_j= p^j\,dF(p) for j=0,…,nj=0,…,n, of which n are free because m0=1m_0=1. Hence RF(k)R_F(k) is identified for every integer k≤nk≤ n. For every k>nk>n, there are two distributions on [0,1][0,1] with identical complete count laws and different RF(k)R_F(k). If the induced truncated Hausdorff moment vector lies in the interior of its moment space, then the sharp identified interval for RF(k)R_F(k) has strictly positive width. Boundary sequences require separate treatment: some have a unique representing measure, in which case every higher-order functional is pinned. The Bernstein kernels in q span nP_n, while RF(k)=∑j=0k(−1)j(kj)mj.R_F(k)= _j=0^k(-1)^j kjm_j. Proof sketch. The n+1n+1 Bernstein kernels in q form a basis of nP_n, so the count law and the first n free moments contain exactly the same information. The shifted-Legendre construction in Appendix A gives an explicit observationally equivalent pair for every k>nk>n. More generally, the strict-width claim for an arbitrary interior truncated Hausdorff sequence follows from generalized-moment theory: dn+1dpn+1(1−p)k d^n+1dp^n+1(1-p)^k =(−1)n+1k!(k−n−1)! =(-1)^n+1 k!(k-n-1)! ⋅(1−p)k−n−1. ·(1-p)^k-n-1. has constant nonzero sign on (0,1)(0,1). Thus 1,p,…,pn,(1−p)k1,p,…,p^n,(1-p)^k is a strict extended Chebyshev system, giving distinct generalized-moment extrema in the interior. Appendix A gives the construction, endpoint orientation, and boundary cases. Operationally, in the pooled/random-task experiment, more tasks make q more precise, whereas more rollouts per task enlarge the set of directly identified functionals; they are not substitutes. 3.1 Two exact incompatible extrapolations Left-tail convention. We say that F has left-tail exponent α>0α>0 and leading constant C>0C>0 when F([0,x])∼CxαF([0,x]) Cx^α as x↓0x 0. If its density satisfies f(p)∼Cαpα−1f(p) Cα p^α-1, then a standard beta-integral argument gives RF(k)∼CΓ(α+1)k−αR_F(k) C (α+1)k^-α. Thus this left-tail convention is precisely the one underlying asymptotic power-law failure curves. Gauss–Legendre quadrature gives a strong certificate. Uniform[0,1][0,1] mixing and a positive finite-support quadrature measure integrate every degree-n polynomial identically. They therefore have exactly the same fixed-n count law. But Uniform mixing has polynomial failure 1/(k+1)1/(k+1), while the finite-support law has eventual exponential failure. Figure 1 shows the construction at n=64n=64: the count laws agree to machine precision despite their divergent extrapolations. Figure 1: Uniform and finite-support mixing laws have identical n=64n=64 success-count distributions, but incompatible extrapolated failure curves. No number of tasks at the same fixed rollout budget distinguishes them. A complementary, weaker construction isolates the tail constant. Let Pn+1P_n+1 be the Legendre polynomial of degree n+1n+1, and set fε(p)=1+εPn+1(2p−1),|ε|<1.f_ (p)=1+ P_n+1(2p-1), | |<1. Orthogonality preserves every degree-n Binomial kernel, while Fε(x)∼[1+ε(−1)n+1]x(x↓0).F_ (x) [1+ (-1)^n+1]x (x 0). For even n=64n=64, ε=.8 =.8 gives the same count law and the same tail exponent α=1α=1, but constants C=.2C=.2 and C=1C=1. Appendix A gives the closed-form integral, an analytic-density pair with distinct exponents, and the failure-ratio limit. Thus the quadrature construction witnesses qualitatively incompatible tails, whereas the Legendre family witnesses a bounded multiplicative ambiguity even after fixing α. 4 Sharp partial identification and finite-sample inference For a known count law, the population identified set is ℐn,k(q) _n,k(q) =RF(k):F∈([0,1]), = \R_F(k):F ([0,1]), PrF(C=c)=qc,c=0,…,n. _F(C=c)=q_c, c=0,…,n \. (7) It is a compact interval. For even n=2rn=2r and an interior moment sequence, its endpoints for (1−p)k(1-p)^k are attained by the two Hausdorff principal representations: the r+1r+1-atom rules containing, respectively, 00 and 11. We compute those rules by Gaussian quadrature for pdFp\,dF and (1−p)dF(1-p)\,dF. This is a sharp continuous solution, not a grid-restricted sensitivity range; the equivalent generalized moment program also admits a moment–SOS formulation (Krein and Nudelman, 1977; Lasserre, 2001). Appendix A gives the endpoint orientation. For odd rollout budgets, the same generalized-moment problem uses the lower and upper Hausdorff principal representations rather than the two single-endpoint rules. The qualification about boundary count laws matters. A flat, rank-deficient Hausdorff moment sequence can have a unique atomic extension, in which case ℐn,k(q)I_n,k(q) collapses even when k>nk>n. Our theorem is thus a generic limitation, not the false claim that every finite count histogram must imply ambiguity. Finite samples. An empirical histogram need not itself be an exact Binomial-mixture law, so a plug-in sharp interval is not a confidence interval. Finite-sample inference should start with a simultaneous confidence region NQ_N for the count law q, not independent absolute-error bands on high raw moments: mj=∑c(c)j/(n)jqcm_j= _c(c)_j/(n)_j\,q_c is an exact linear map. The desired population-set projection is ⋃q∈Nℐn,k(q) _q _NI_n,k(q). One must also state whether coverage is intended for the scalar RF(k)R_F(k) or for the entire identified set; these are distinct targets in partial-identification inference (Imbens and Manski, 2004; Chernozhukov et al., 2007). We also give one exact but deliberately conservative finite-task certificate. Let Z=∑iCi=0Z= _i1\C_i=0\, and let [a,b][a,b] be a 1−δ1-δ Clopper–Pearson interval for q0=Pr(C=0)=RF(n)q_0= (C=0)=R_F(n) (Clopper and Pearson, 1934). For k≥nk≥ n, ak/n≤RF(k)≤ba^k/n\ ≤\ R_F(k)\ ≤\ b has the same finite-sample coverage: with X=(1−p)nX=(1-p)^n, Jensen gives RF(k)=[Xk/n]≥q0k/nR_F(k)=E[X^k/n]≥ q_0^k/n, while Xk/n≤X^k/n≤ X. Appendix C implements this certificate in a prospective precision calibration. It uses only q0q_0, so it is not a replacement for the sharper complete-count-law projection ⋃q∈Nℐn,k(q) _q _NI_n,k(q); rather, it makes the finite-task uncertainty that such a projection must confront explicit. Adaptive allocation. Dynamic sampling can be beneficial, but it changes the observation model: outcome-dependent nin_i’s are not summarized by one fixed-n count law. Under a maintained Beta–Binomial likelihood they can be modeled, as in Kazdan et al. (2025). Under misspecification, allocation and extrapolation should be assessed jointly with a design-aware sensitivity analysis; sampling hard tasks more often cannot by itself make a generic large-k target nonparametrically identified. 5 Public 10,000-rollout calibration We use the public Monkey Business release from Brown et al. (2024): per-completion correctness labels for 10,000 samples on each task. We analyze all 128 released MATH problems for Llama-3-8B-Instruct and Llama-3-70B-Instruct, all 127 released GSM8K problems, and all 140 released CodeContests problems for Llama-3-70B-Instruct. This is not a new model run. These are four deliberate case studies, not an exhaustive or randomly sampled survey of the broader release: they contrast two MATH model sizes with two qualitatively different non-MATH count-law geometries. Accordingly, we make no prevalence claim about width or about a low-dimensional difficulty diagnostic. Within each stated case study, every eligible released task is retained. For each task, the raw frequency p^i=ci/Mi p_i=c_i/M_i defines a finite-pool empirical-mixing reference. For n∈16,64n∈\16,64\, we form the large-N counterfactual count law N−1∑iBin(n,p^i)N^-1 _iBin(n, p_i). This deliberately removes ordinary finite-task noise: it asks what an arbitrarily large n-rollout pooled-count evaluation could identify under this stated reference. It is not a claim that the labeled finite-benchmark product likelihood is nonidentified. Raw frequencies and a continuous latent-probability convention are different estimands, especially when a task has ci=0c_i=0. We therefore do not attach an uncertainty interval to the raw reference. Separately, we report the Jeffreys-smoothed reference p~i=(ci+1/2)/(Mi+1) p_i=(c_i+1/2)/(M_i+1) and the 95% posterior interval from independent Beta(ci+1/2,Mi−ci+1/2)Beta(c_i+1/2,M_i-c_i+1/2) draws, conditional on the fixed task set. In CodeContests, 87 of 140 tasks have ci=0c_i=0: the raw reference is .713 whereas the Jeffreys-smoothed reference is .678. This is a convention sensitivity, not a raw-reference error bar. We fit the conditional Beta–Binomial forecast to the same count law under each convention. A scale heuristic and its limit. A probability p changes an n-rollout count law weakly when npnp is small, but affects RF(k)R_F(k) materially when kpkp is order one. This exposes the familiar intermediate scale 1/k≲p≲1/n1/k p 1/n. Failure mass in that band is therefore a natural warning signal for extrapolation, but it is not an identification criterion. Sharp-set width is determined by the full truncated-moment geometry. The number and separation of failure-carrying components are descriptive features of that geometry, but neither is sufficient by itself. Under a raw empirical reference, explicit zero-frequency atoms also contribute directly to the stated reference failure; this is a convention, not a claim that zero probability is identified from the count law. Figure 2: Public calibration at k=1000k=1000. Blue bars and orange circles use the raw p^i=ci/Mi p_i=c_i/M_i convention. Green diamonds are its finite-pool empirical-mixing reference. Purple crosses and bars are the different Jeffreys-smoothed reference and its fixed-task 95% posterior interval. Blue intervals are population plug-in sets, not finite-sample confidence intervals. Table 1: Raw-empirical-reference calibration at k=1000k=1000. Rraw(n)R_ raw(n) is directly identified. Brackets are sharp population plug-in sets conditional on the raw induced count law. The last column is the relative n=64n=64 width (U−L)/L(U-L)/L, computed at 120-decimal arithmetic; it describes this stated empirical reference, not a confidence interval or exact-rank certificate. Configuration Rraw(16)R_ raw(16) Rraw(1000)R_ raw(1000) n=16n=16 sharp set n=64n=64 sharp set relative n=64n=64 width MATH, Llama-3-8B .331 .0673 [.000916,.168.000916,.168] [.0663,.0683.0663,.0683] ×10−23.00\!×\!10^-2 MATH, Llama-3-70B .193 .0314 [×10−5,.07552.89\!×\!10^-5,.0755] [.0308,.0319.0308,.0319] ×10−23.59\!×\!10^-2 GSM8K, Llama-3-70B .00795 .00214 [.00214,.00476.00214,.00476] [.00214,.00214.00214,.00214] ×10−478.84\!×\!10^-47 CodeContests, Llama-3-70B .842 .713 [.503,.775.503,.775] [.713,.713.713,.713] ×10−122.93\!×\!10^-12 Table 2: Jeffreys-smoothed calibration at k=1000k=1000, a distinct latent-probability convention from Table 1. The interval is a fixed-task 95% posterior propagation interval, and the Beta–Binomial prediction is fit to the same Jeffreys-induced n=64n=64 count law. It is therefore not an error bar for the raw reference. Configuration Jeffreys posterior median [95% interval] Beta–Binomial failure Beta / posterior median MATH, Llama-3-8B .0656 [.0596,.0712.0596,.0712] .0873 1.33×1.33× MATH, Llama-3-70B .0304 [.0268,.0337.0268,.0337] .0434 1.43×1.43× GSM8K, Llama-3-70B .00210 [.000898,.00380.000898,.00380] ×10−51.00\!×\!10^-5 0.00477×0.00477× CodeContests, Llama-3-70B .681 [.670,.691.670,.691] .723 1.06×1.06× Figure 2 and Table 1 make the design boundary visible: Rraw(16)R_ raw(16) is directly identified, whereas the set opens at k=1000k=1000. At n=16n=16, the sharp MATH intervals span factors of 183183 (8B) and 2,6122,612 (70B), even after supplying a 10,000-rollout reference to remove ordinary count-law estimation noise. Width is strongly configuration-dependent. The raw n=64n=64 GSM8K relative width is 8.84×10−478.84× 10^-47, while the MATH relative widths remain about 3%. Appendix B verifies every displayed endpoint with the Golub–Welsch implementation at 120 and 300 decimal digits; the maximum relative discrepancy is 5.6×10−545.6× 10^-54. These four calibrations demonstrate that window share alone is insufficient. At n=64n=64, all of the GSM8K raw-reference failure lies in [1/1000,1/64][1/1000,1/64], but it is carried by one task with p^=13/10,000 p=13/10,000; all other released GSM8K tasks have p^>0.2 p>0.2. The full count law separates this isolated component and the interval contracts already at n=32n=32. In contrast, the MATH references distribute target-budget failure over several components and remain wider. Figure 3 sweeps n∈8,16,32,64n∈\8,16,32,64\: the intermediate window is a useful warning scale, but the complete truncated-moment geometry determines width. Table 4 makes this qualification constructive. It holds fixed a synthetic 128-task hard profile (eight probabilities in [.0011,.015][.0011,.015] plus one at .0002.0002); only the remaining 119 easy-task probabilities change. Every profile has the same target failure to 40 displayed digits, 45.19% intermediate-window failure share, eight window tasks, and effective window-task count 2.81. Nevertheless, the sharp width varies by a factor of 28 at n=16n=16 and over 101610^16 at n=64n=64. Thus neither window mass nor a joint window-mass/dispersion summary is a sufficient characterization of extrapolation ambiguity. The model comparison must likewise respect the convention. Under the Jeffreys convention in Table 2, the Beta–Binomial forecast is 4.5–29% above the upper posterior endpoint for MATH and CodeContests, whereas it is below even GSM8K’s lower endpoint by more than 85×85×. Thus it can substantially overstate pass@1000. The apparent agreement with the CodeContests raw reference is partly a zero-frequency-convention artifact, not a raw-reference uncertainty calculation. The apparent worsening from n=16n=16 to n=64n=64 in some rows is not paradoxical: the nonparametric set can contract faster than model discrepancy vanishes. These results neither validate nor refute Beta–Binomial forecasting in general; they quantify its assumption burden in concrete released evaluations. Table 3: Scale and support diagnostics for the n=64n=64 raw plug-in calibration. Window tasks counts empirical probabilities in [1/1000,1/64][1/1000,1/64]; window share is their fraction of raw-reference failure. Neither is an identified quantity. Across these four illustrative configurations, GSM8K and CodeContests show that either window share or task count alone can be insufficient. These are descriptive summaries, not a theorem or sufficient statistic for sharp-set width. Configuration zero-success tasks window tasks window share relative n=64n=64 width MATH, Llama-3-8B 2 23 16.1% ×10−23.00\!×\!10^-2 MATH, Llama-3-70B 2 8 13.1% ×10−23.59\!×\!10^-2 GSM8K, Llama-3-70B 0 1 100.0% ×10−478.84\!×\!10^-47 CodeContests, Llama-3-70B 87 10 1.1% ×10−122.93\!×\!10^-12 Table 4: Controlled diagnostic stress test at k=1000k=1000. The hard profile, target failure (to 40 displayed digits), intermediate-window share (45.19%), window-task count (8), and effective window-task count (2.81) are fixed; only the 119 easy-task probabilities change. The sharp relative-width range shows that these coarse summaries do not characterize the full moment geometry. Easy-task geometry relative width at n=16n=16 relative width at n=64n=64 Uniform grid [.2,.825][.2,.825] 26.2526.25 ×10−41.09\!×\!10^-4 Low grid [.2,.4][.2,.4] 1.671.67 ×10−113.97\!×\!10^-11 Middle grid [.45,.6][.45,.6] 0.9390.939 ×10−218.91\!×\!10^-21 High grid [.6,.825][.6,.825] 1.531.53 ×10−163.21\!×\!10^-16 Bimodal grids 13.4813.48 ×10−62.80\!×\!10^-6 Figure 3: Sharp-width trajectories at k=1000k=1000 under the raw empirical reference. Left: all four calibrations contract as per-task rollout budget increases. Right: color denotes configuration, marker shape denotes n, and marker area denotes the number of window tasks. GSM8K shows that window share alone can be maximal while a set is nearly point-like. These plotted relationships are descriptive: the complete truncated-moment geometry, rather than a low-dimensional diagnostic, determines the sharp interval. 6 What to report and conclude For direct pass@k, evaluate each task at least k times and retain the full evaluation pipeline: because pip_i is operational, changing a cap, prompt, parser, or scorer changes the target itself. For k>nk>n, report (i) the exact target and sampling design, (i) a population identified-set diagnostic or a finite-sample confidence region, and (i) every parametric forecast as a model-conditioned sensitivity analysis, including its held-out calibration. Tail exponents and constants require the same separation between fitted and identified quantities. In the pooled/random-task experiment, more tasks improve precision; deeper per-task sampling changes identification. Simple difficulty summaries can be useful warning signals but should not replace the identified-set calculation when it is available. In particular, the fraction of target-budget failure near 1/k≲p≲1/n1/k p 1/n does not determine extrapolation ambiguity; concentration and the remaining moment geometry can materially alter the sharp bounds. This also clarifies how to interpret inference-time scaling work. A power law or Beta–Binomial forecast can be scientifically useful, especially when it predicts a held-out rollout budget. Its evidence is predictive accuracy under a stated model, not nonparametric recovery of a tail. The relevant baseline is therefore not “no estimate,” but the compatible interval that the fixed-rollout design leaves unresolved. 7 Limitations The main theorem is about a pooled/random-task count-law experiment under conditional iid completions. Correlated completions, finite exchangeability without an infinite extension, adaptive task-dependent rollout counts, and pipeline changes define different observation models. For a fixed labeled benchmark, the unrestricted finite product model is formally identified; the paper’s limitation is not that algebraic fact but the information supplied by a fixed-depth pooled-count design. The public calibration conditions on explicit raw-frequency or Jeffreys conventions and is a population plug-in diagnostic, not a finite-sample confidence interval. Appendix C adds an exact but deliberately coarse scalar certificate. A sharper finite-sample identified-set projection requires a confidence region for the complete count law and a corresponding projection through the moment problem; we state that construction but do not claim to have implemented that stronger procedure here. 8 Conclusion Fixed-rollout pass@k evaluation is a finite-moment problem. In the pooled/random-task model, the direct estimator’s k≤nk≤ n domain reflects a deeper generic identification boundary that more benchmark tasks cannot cross. Sharp moment bounds and public 10,000-rollout calibrations make the remaining assumption burden visible: at n=16n=16, observed widths range from 1.5×1.5× to over 2,600×2,600×, while at n=64n=64 an isolated hard component can be nearly pinned by one specific full count-law geometry. The controlled test shows why no window-mass or dispersion summary substitutes for that geometry. A raw empirical zero-frequency convention can also materially change a seemingly well-calibrated forecast. This provides a constructive standard for reporting extrapolated LLM evaluation: distinguish direct estimates, partial-identification bounds, and model-conditioned forecasts rather than treating them as the same kind of evidence. References Brown et al. (2024) B. Brown, J. Juravsky, R. Ehrlich, R. Clark, Q. V. Le, C. Ré, and A. Mirhoseini Large language monkeys: scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787. External Links: Link Cited by: §1, §2, §5. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Herbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang, I. Babuschkin, S. Balaji, S. Jain, W. Saunders, C. Hesse, A. N. Carr, J. Leike, J. Achiam, V. Misra, E. Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer, P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374v2. External Links: Link Cited by: §2. Chen et al. (2026) X. Chen, A. Rambachan, and E. Tamer Partial identification from LLM prompts. arXiv preprint arXiv:2606.15031v2. External Links: Link Cited by: §2. Chernozhukov et al. (2007) V. Chernozhukov, H. Hong, and E. Tamer Estimation and confidence regions for parameter sets in econometric models. Econometrica 75 (5), p. 1243–1284. External Links: Document Cited by: §4. Clopper and Pearson (1934) C. J. Clopper and E. S. Pearson The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26 (4), p. 404–413. Cited by: §4. Hariri et al. (2026) M. Hariri, A. Samandar, M. Hinczewski, and V. Chaudhary Don’t pass@k: a Bayesian framework for large language model evaluation. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §2. Hausdorff (1921) F. Hausdorff Summationsmethoden und momentfolgen. I. Mathematische Zeitschrift 9 (1), p. 74–109. External Links: Document Cited by: §2. Hewitt and Savage (1955) E. Hewitt and L. J. Savage Symmetric measures on cartesian products. Transactions of the American Mathematical Society 80 (2), p. 470–501. External Links: Document Cited by: §2. Imbens and Manski (2004) G. W. Imbens and C. F. Manski Confidence intervals for partially identified parameters. Econometrica 72 (6), p. 1845–1857. External Links: Document Cited by: §4. Kazdan et al. (2025) J. Kazdan, R. Schaeffer, Y. Allouah, C. Sullivan, K. Yu, N. Levi, and S. Koyejo Efficient prediction of pass@k scaling in large language models. arXiv preprint arXiv:2510.05197. External Links: Link Cited by: §1, §2, §4. Krein and Nudelman (1977) M. G. Krein and A. A. Nudelman The markov moment problem and extremal problems. American Mathematical Society, Providence, Rhode Island. Cited by: §A.3, §2, §4. Lasserre (2001) J. B. Lasserre Global optimization with polynomials and the problem of moments. SIAM Journal on Optimization 11 (3), p. 796–817. External Links: Document Cited by: §2, §4. Liu (2026) Y. Liu Two calls, two moments, and the vote-accuracy curve of repeated LLM inference. arXiv preprint arXiv:2605.03379v2. External Links: Link Cited by: §2. Manski (2003) C. F. Manski Partial identification of probability distributions. Springer, New York, New York. External Links: Document Cited by: §2. Schaeffer et al. (2025) R. Schaeffer, J. Kazdan, J. Hughes, J. Juravsky, S. Price, A. Lynch, E. Jones, R. Kirk, A. Mirhoseini, and S. Koyejo How do large language monkeys get their power (laws)?. In Proceedings of the 42nd International Conference on Machine Learning, Vol. 267, p. 53132–53176. External Links: Link Cited by: §1, §2. Appendix A Proofs and implementation details A.1 Direct estimator and polynomial identification For a task with success probability pip_i, choosing k failed items from the n−Cin-C_i observed failures gives [(n−Cik)(nk)|pi]=(1−pi)k,k≤n.E [ n-C_ik nk |p_i ]=(1-p_i)^k, k≤ n. This proves equation (3). The success factorial moments obey [(C)j(n)j] [ (C)_j(n)_j ] =∫pjdF(p) = p^j\,dF(p) =mj,j=0,…,n. =m_j, j=0,…,n. The degree-n Bernstein polynomials (nc)pc(1−p)n−c ncp^c(1-p)^n-c form a basis of nP_n. Hence the count law identifies every integral against a degree-n polynomial, and the first n moments determine the count law conversely. Equation (4) then proves the pass@k corollary. A.2 Legendre perturbations Put P~m(p)=Pm(2p−1) P_m(p)=P_m(2p-1). For every polynomial h of degree less than m, shifted-Legendre orthogonality gives ∫01h(p)P~m(p)p=0. _0^1h(p) P_m(p)\,dp=0. Thus equation (5) preserves all fixed-n count probabilities. For k≥mk≥ m, the closed-form integral is ∫01(1−p)kP~m(p)p _0^1(1-p)^k P_m(p)\!dp = = (−1)m(k!)2(k−m)!(k+m+1)!. (-1)^m (k!)^2(k-m)!(k+m+1)!. To see this, substitute x=1−px=1-p, use P~m(1−x)=(−1)mP~m(x) P_m(1-x)=(-1)^m P_m(x), and apply the standard shifted-Legendre monomial integral. Taking m=n+1m=n+1 makes (A3) nonzero for every k>nk>n, proving Theorem 1’s counterexample. For the Uniform base law, RF0(k)=1/(k+1)R_F_0(k)=1/(k+1). Dividing the perturbation in (A3) by this quantity and using the ratio of factorials gives RFε(k)RF0(k)⟶1+ε(−1)n+1. R_F_ (k)R_F_0(k) 1+ (-1)^n+1. So this family produces the bounded multiplicative distortion matching its tail-constant ratio. It is deliberately complementary to the Gauss–Legendre construction, which produces a qualitative polynomial versus exponential discrepancy. The exponent ambiguity is not an artifact of comparing a continuous law with a finite-support rule. Let m=n+1m=n+1, hm(p)=(−1)mP~m(p)h_m(p)=(-1)^m P_m(p), and define f0(p)=2p,fε(p)=2p+εhm(p).f_0(p)=2p, f_ (p)=2p+ h_m(p). The same orthogonality preserves all moments through n. Moreover, hm(0)=1h_m(0)=1, |hm|≤1|h_m|≤ 1, and |hm′(p)|≤m(m+1)|h_m (p)|≤ m(m+1). Hence for 0<ε≤1/[m(m+1)]0< ≤ 1/[m(m+1)], the density fεf_ is nonnegative on [0,1][0,1]: it is nonnegative near zero because hm≥0h_m≥ 0 there, and elsewhere the 2p2p term dominates a negative perturbation. Both densities are analytic, but F0([0,x])=x2,Fε([0,x])=εx+O(x2).F_0([0,x])=x^2, F_ ([0,x])= x+O(x^2). They therefore have exponents 2 and 1 while retaining exactly the same fixed-n count law. This supplies an analytic-density version of the exponent nonidentification result. A.3 Boundary moment sequences and sharp bounds The set of representing probability measures for a truncated Hausdorff moment sequence is compact and convex. Its image under F↦RF(k)F R_F(k) is a compact interval. An endpoint has an atomic optimizer; for an interior sequence with moments through 2r2r, the two principal representations containing 0 and 1 attain the extrema for (1−p)k(1-p)^k, k>2rk>2r. The needed Chebyshev-system check is direct: d2r+1dp2r+1(1−p)k d^2r+1dp^2r+1(1-p)^k =−k!(k−2r−1)! =- k!(k-2r-1)! ⋅(1−p)k−2r−1<0. ·(1-p)^k-2r-1<0. Hence the representation containing 0 is the upper endpoint and the representation containing 1 the lower endpoint (Krein and Nudelman, 1977). We obtain their non-endpoint nodes and weights together by Golub–Welsch eigendecomposition of the Jacobi matrix for the positive measures pdFp\,dF and (1−p)dF(1-p)\,dF. This avoids a raw-power Vandermonde solve. Appendix B reports a precision sweep and deterministic controls; the 120- and 300-decimal endpoint values agree for every public interval used in the paper. This interior qualification cannot be dropped. If the appropriate Hankel moment matrix is rank deficient and admits a flat extension, the representing measure can be unique. Then every higher moment is identified and the interval is a point. We do not infer a flat extension from numerical conditioning, nor do we infer exact zero success probability from an observed zero count. The raw zero-frequency convention is stated explicitly in Section 5; the theorem’s generic qualification is essential. Appendix B Numerical verification The very narrow GSM8K result merits an explicit numerical check. We rerun every raw-reference principal-representation calculation at 16, 50, 120, and 300 decimal digits. At low precision the Cholesky factorization can correctly report that it cannot resolve the positive moment matrix; we record this failure rather than force an answer. Table 5 summarizes the resolution threshold across all four public calibrations. At 120 and 300 digits all displayed public endpoints agree, with a maximum relative difference of 5.6×10−545.6× 10^-54. This is a precision-stability check, not an inference claim about the source tasks. Table 5: Precision sweep across the four public raw-reference calibrations. An entry gives the number of configurations for which the interior factorization resolved; lower precisions that do not resolve return no endpoint. Every 120-digit endpoint agrees with its 300-digit counterpart. n 16 digits 50 digits 120 digits 300 digits 8 4/4 4/4 4/4 4/4 16 4/4 4/4 4/4 4/4 32 0/4 4/4 4/4 4/4 64 0/4 2/4 4/4 4/4 Table 6: Deterministic numerical controls at k=1000,n=64k=1000,n=64. The continuous Uniform control has known reference failure 1/10011/1001 and a genuinely nondegenerate sharp interval. The synthetic control has 127 distinct atoms, one at p=.0013p=.0013 and 126 in [.2,.825][.2,.825]. Both are computed by the same Golub–Welsch code as the public calibration. control reference failure sharp interval relative width Uniform[0,1][0,1] .000999 [.000909,.00111.000909,.00111] .223 127-atom single-hard-task profile .00214 [.00214,.00214.00214,.00214] ×10−221.01\!×\!10^-22 The Uniform calculation rules out a generic collapse of the implementation: the same n=64n=64 code returns a 22.3% interval. Conversely, the synthetic control establishes only feasibility: an extremely narrow interval can occur for a nondegenerate empirical law when one separated hard component carries the target-budget failure. It is not designed to reproduce GSM8K’s 8.84×10−478.84× 10^-47 width; that number depends on the detailed locations of its 126 easy-task frequencies. In the released GSM8K calibration the only empirical probability below .2 is 13/10,00013/10,000. Appendix C Exact finite-sample certificate For an actual N-task random-task evaluation, Z=∑iCi=0∼Binomial(N,q0)Z= _i1\C_i=0\ (N,q_0), where q0=RF(n)q_0=R_F(n). Let [a(Z),b(Z)][a(Z),b(Z)] be the two-sided 1−δ1-δ Clopper–Pearson interval for q0q_0. The event q0∈[a(Z),b(Z)]q_0∈[a(Z),b(Z)] has probability at least 1−δ1-δ. Setting X=(1−p)n∈[0,1]X=(1-p)^n∈[0,1] and s=k/n≥1s=k/n≥ 1, Jensen and pointwise monotonicity imply RF(k) R_F(k) =[Xs]≥[X]s=q0s, =E[X^s] [X]^s=q_0^s, RF(k) R_F(k) =[Xs]≤[X]=q0. =E[X^s] [X]=q_0. Thus [a(Z)k/n,b(Z)][a(Z)^k/n,b(Z)] is an exact, distribution-free finite-sample confidence interval for the scalar RF(k)R_F(k). It is intentionally coarse: it uses only q0q_0, rather than the full histogram. Its role is to give a valid executable baseline, not to stand in for the sharp NQ_N-projection described in Section 4. Table 7: Prospective finite-task precision calibration of the exact 95% Clopper–Pearson/Jensen certificate at n=16,k=1000n=16,k=1000. Each of 20,000 simulated N-task count evaluations draws pip_i with replacement from the stated raw empirical-mixing reference and then Ci∼Binomial(16,pi)C_i (16,p_i). The displayed certificate is its componentwise median across simulations. All 20,000 simulated targets are covered; this empirical frequency is a check, whereas the certificate’s at-least-95% coverage is the analytic guarantee. These are prospective precision calculations under the raw convention, not uncertainty intervals for the 10,000-rollout frequencies. Configuration raw-reference RF(1000)R_F(1000) median exact 95% certificate MATH, Llama-3-8B .0673 [×10−38,.4171.34\!×\!10^-38,.417] MATH, Llama-3-70B .0314 [×10−56,.2755.48\!×\!10^-56,.275] GSM8K, Llama-3-70B .00214 [×10−232,.04315.29\!×\!10^-232,.0431] CodeContests, Llama-3-70B .713 [×10−8,.8999.32\!×\!10^-8,.899] The exact certificate is wide for all four 100–150-task references. This is expected: it is a one-coordinate baseline, while the sharp population sets condition on a known count law. The contrast makes the two sources of uncertainty visible rather than conflating them. A tighter finite-sample identified-set interval requires a simultaneous multinomial region NQ_N, the exact linear map mj=∑c(c)j/(n)jqcm_j= _c(c)_j/(n)_j\,q_c, and the projection ⋃q∈Nℐn,k(q) _q _NI_n,k(q). That full-moment projection remains an important extension; the certificate above is included because it gives a correct finite-sample baseline now. Appendix D Public-data protocol We stream only the Boolean is_corrects vectors from commit a9f8f73bcd6948a57ed922cba4e48062ef95f553 of the public Monkey Business release. “Eligible” means a source record with a Boolean is_corrects vector of length at least 10,000; the calibration uses every such record: 128 MATH tasks for each Llama-3 model, 127 GSM8K tasks, and 140 CodeContests tasks. It constructs N−1∑iBin(n,p^i)N^-1 _iBin(n, p_i) under each stated frequency convention and computes the two even-order principal representations at 120-decimal arithmetic. The independent 10,000-rollout direct estimator is retained only as a check; the plotted reference is the empirical mixing functional conditioning the plug-in set. Beta–Binomial fit. For a stated convention and rollout depth n, let q(n)=(q0(n),…,qn(n))q^(n)=(q^(n)_0,…,q^(n)_n) denote the induced count law. Under P∼Beta(a,b)P (a,b), its model count probabilities are qcBB(a,b) q_c B(a,b) =(nc)B(c+a,n−c+b)B(a,b), =n c B(c+a,n-c+b)B(a,b), c=0,…,n. c=0,…,n. We minimize the count-law cross-entropy −∑cqc(n)logqcBB(a,b)- _cq^(n)_c q_c B(a,b) over a,b>0a,b>0, using loga,logb a, b coordinates and log-gamma evaluation. The implementation uses a deterministic two-dimensional Nelder–Mead search for 450 iterations from five starts (−2,−2),(0,0),(1.5,1.5),(−4,0),(0,−4)(-2,-2),(0,0),(1.5,1.5),(-4,0),(0,-4), retaining the lowest objective. It then reports the model-conditioned forecast RBB(k)=B(a,b+k)B(a,b).R_ B(k)= B(a,b+k)B(a,b). This is a reproducible model projection of the induced count law, not an estimate of a nonparametrically identified tail. Optimization audit. For the 16 raw and Jeffreys fits reported at n∈16,64n∈\16,64\, all five documented starts agree in normalized count-law cross-entropy to within 4.85×10−144.85× 10^-14, with maximum absolute forecast variation 1.18×10−71.18× 10^-7. Extending each start by 900 additional Nelder–Mead iterations changes any forecast by at most 2.94×10−82.94× 10^-8. An independent analytic-gradient BFGS check, initialized at each retained Nelder–Mead solution, satisfies a 10−710^-7 gradient-norm tolerance and neither lowers the recorded objective nor changes the forecast at displayed double precision. The audit records all starts, objectives, forecasts, and optimizer diagnostics for release. Reproducibility and release. Upon publication we will release the anonymous source task identifiers, per-task success counts, conventions, moment vectors, solver residuals, precision checks, Beta–Binomial fit starts and objective values, random seeds, and the scripts used for every table and figure. The review manuscript contains the algorithmic details needed to audit the calculations without a review-time external link.