Paper deep dive

Uncheatable Eval: Dynamic Compression-Based Evaluation of Language Models

Kaifeng Tan, Yudong Li, Linlin Shen

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 55

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%

Last extracted: 9/24/2026, 5:24:00 AM

Summary

The paper introduces Uncheatable Eval, a dynamic benchmark designed to evaluate base language models by measuring their ability to compress newly published text, thereby mitigating data contamination risks. The authors evaluate 80 models across 14 text categories, finding that compression rate (CR) scales consistently with model size, varies by architecture type (attention, hybrid, recurrent) in long-context scenarios, and correlates strongly with zero-shot MMLU accuracy.

Entities (10)

Uncheatable Eval · benchmark · 98%Compression Rate · metric · 95%Qwen3.5-35B-A3B-Base · model · 95%Data Contamination · problem · 92%Falcon-H1 · model-architecture · 90%MMLU · benchmark · 90%RWKV7-G1J · model-architecture · 90%Gemma 4 · model · 85%LiveBench · benchmark · 80%Paloma · benchmark · 80%

Relation Signals (7)

Uncheatable Eval → usesmetric → Compression Rate

confidence 95% · we use compression rate to evaluate how well models predict new text.

Uncheatable Eval → mitigates → Data Contamination

confidence 92% · reduce the risk of data contamination.

Compression Rate → correlateswith → MMLU

confidence 90% · lower compression rates are strongly associated with higher zero-shot MMLU accuracy.

Qwen3.5-35B-A3B-Base → performswellon → Scientific Paper

confidence 90% · Qwen3.5-35B-A3B-Base leads several scientific categories

Falcon-H1 → improvesrapidlyin → Long-Context

confidence 85% · Falcon-H1 improves most rapidly... in long-context evaluation

RWKV7-G1J → performswellon → General Text

confidence 85% · Its models perform particularly well on general text, including the non-English encyclopedia category

Uncheatable Eval → comparedto → LiveBench

confidence 75% · LiveBench and LiveCodeBench use recently released questions... Uncheatable Eval also evaluates newly published text

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is particularly challenging for base models, whose limited instruction-following ability complicates task-based assessment. We introduce Uncheatable Eval, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination. Drawing on the relationship between a model's predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text. We evaluate 80 models across 14 text categories, study how compression changes with context length, and examine the correlation between compression rate and zero-shot MMLU accuracy. Our results yield three main findings: (1) compression performance follows a consistent scaling trend with model size; (2) attention-based, hybrid, and recurrent models differ in how their compression performance changes as more context becomes available; and (3) lower compression rates are strongly associated with higher zero-shot MMLU accuracy. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

51,939 characters extracted from source content.

Expand or collapse full text

UNCHEATABLE EVAL: DYNAMIC COMPRESSION-BASED EVALUATION OF LANGUAGE MODELS Kaifeng Tan 1 Yudong Li 2 Linlin Shen 1 1 Shenzhen University 2 Tsinghua University ABSTRACT Modern large language models are pretrained on massive datasets, making it difficult to prevent benchmark data from entering their training sets and undermining the reliability of evaluation results. Reliable evaluation is particularly challenging for base models, whose limited instruction- following ability complicates task-based assessment. We introduce Uncheatable Eval, a dynamic benchmark that regularly collects newly published text to evaluate base language models and reduce the risk of data contamination. Drawing on the relationship between a model’s predictive ability and its ability to compress data losslessly, we use compression rate to evaluate how well models predict new text. We evaluate 80 models across 14 text categories, study how compression changes with context length, and examine the correlation between compression rate and zero-shot MMLU accuracy. Our results yield three main findings: (1) compression performance follows a consistent scaling trend with model size; (2) attention-based, hybrid, and recurrent models differ in how their compression performance changes as more context becomes available; and (3) lower compression rates are strongly associated with higher zero-shot MMLU accuracy. Code is available athttps: //github.com/Jellyfish042/uncheatable_eval. Keywords language model evaluation· data contamination· dynamic benchmarks· data compression Figure 1: Overview of Uncheatable Eval. Left: Data construction and compression-rate evaluation. Right: Compression rates of six representative models across five text types. arXiv:2609.27510v1 [cs.CL] 23 Sep 2026 Uncheatable Eval 1 Introduction Effective evaluation is essential to the development of large language models. However, the vast internet corpora used to train these models may contain benchmark questions, answers, and solutions. Such contamination can inflate scores through memorization, making it difficult to distinguish genuine generalization from recall. Longitudinal studies using model training cutoffs find evidence of this effect in code and mathematics data [1]; surveys and controlled studies show that contamination occurs at several semantic levels and is difficult to detect reliably [2, 3, 4]. Dynamic benchmarks reduce this risk by collecting newly published data or generating new test examples. LiveBench and LiveCodeBench use recently released questions and code problems [5, 6]. Other recent benchmarks continuously update their evaluation data to reflect changes in facts and knowledge [7,8,9]. Newer test data reduce contamination risk, while task-based evaluation can obscure base-model capabilities because base models have limited instruction-following ability. We introduce Uncheatable Eval, a dynamic benchmark that measures how well pretrained language models predict newly published text (Figure 1). Drawing on the relationship between prediction and lossless compression, we view each language model as a lossless compressor: its next-token probabilities can guide arithmetic coding, assigning shorter codes to more probable continuations [10]. A model that predicts the observed text more accurately therefore requires fewer bits to encode it. We evaluate this ability through compression rate, calculated from the ideal code length relative to the original text size. This directly evaluates base models on new text without task instructions or reference answers. To apply this approach across diverse forms of text, we collect newly published fiction, source code, scientific papers, news, and encyclopedia articles within specified time windows. We then process these texts through a data curation pipeline that combines quality filtering, deduplication, and text normalization, with balanced sampling across languages for the multilingual subset. Results for 80 models on the resulting datasets reveal a systematic relationship between model size and compression rate, well described by a power law with an additive constant. A 54-model analysis on longer scientific documents shows different changes in CR across attention-based, hybrid, and recurrent models as more preceding text becomes available. A comparison with zero-shot MMLU further reveals a strong association between better compression of technical text and higher MMLU accuracy. Our main contributions can be summarized as follows: 1.We introduce Uncheatable Eval, a dynamic benchmark that evaluates base language models through compression of newly published text collected from diverse sources and curated using a data processing pipeline developed for the benchmark. 2.We report 80 models spanning attention-based, hybrid, and recurrent families across 14 text categories, providing a broad comparison of compression performance across model sizes and text domains. 3.We analyze how CR varies with model size and document position and how it correlates with zero-shot MMLU accuracy. 2 Related Work Data contamination. Benchmark questions and answers can enter model training data and inflate test scores [3,11,12]. Contamination ranges from exact copies to semantic overlap, so lexical matching alone is insufficient [2,13]. Comparisons across training cutoffs and controlled leakage experiments reveal score effects and detection limits [1,14]. Yet overlap does not have the same performance effect for every model, and inflated scores need not reorder leaderboards [15,16]. Other approaches detect contamination from model behavior [17,4], score multiple contamination levels [18], or control screening errors [19]. Proposed defenses vary test instances or interactions [20,21,22,23], limit benchmark exposure [24], or design contamination-resistant datasets [25]. These studies motivate collecting new evaluation data and show that publication recency alone does not establish a clean test set. Dynamic benchmarks. Dynamic benchmarks regularly add new test data to reduce the risk of contamination. LiveBench and LiveCodeBench collect recently released questions and programming problems [5,6], while AntiLeak- Bench constructs questions using new real-world knowledge [26]. Related benchmarks test whether models can use updated evidence, track changing facts, and adapt to new knowledge [7, 8, 9]. FreshBench is the closest prior work to ours. It measures the probabilities models assign to text published at different times and their accuracy in predicting future events [27]. Its main question is how performance varies with the age of the data relative to a model’s release, including whether models perform worse on more recent data. Uncheatable Eval 2 Uncheatable Eval also evaluates newly published text, but focuses on comparing base models through compression rate. We also examine compression by model size and document position. Compression-based evaluation. Language models can be evaluated directly by measuring how well they predict text. Paloma measures perplexity across diverse domains under consistent evaluation conditions [28]. Delétang et al. explain the connection between prediction and lossless compression: a model that assigns higher probabilities to the observed text can encode it using fewer bits [10]. Li et al. use compression to evaluate models on text published after their training cutoffs and examine how performance changes across time periods [29]. Huang et al. find that models that compress text better also tend to score higher on knowledge, code, and mathematics benchmarks [30]. Uncheatable Eval extends this approach to a regularly updated benchmark covering diverse text types and model families. 3 Uncheatable Eval 3.1 Data Collection and Curation Uncheatable Eval collects newly published text for each evaluation to reduce the risk that models have already encountered the test data during training. The collection covers fiction, news, encyclopedia articles, scientific papers, source code, and documentation from publicly available sources. Table 1 lists the evaluation categories; Other groups source-code files outside the named language categories. The collected text is processed through a curation pipeline to improve quality and limit repetition. Quality filters remove samples that are too short or contain anomalous character sequences, while near-duplicate removal uses MinHash locality-sensitive hashing to limit repeated content. Unicode text is standardized using NFC normalization. For the non-English encyclopedia subset, sampling is balanced across languages to prevent the results from being dominated by a single language. Because tokenizers can assign different numbers of tokens to the same text, a length limit based on one tokenizer may leave samples too long for another. We therefore check each sample with a diverse set of tokenizers and truncate it to meet a common token budget under all of them. This produces a shared text sample for evaluation while accounting for differences in tokenization. Table 1: Evaluation categories and labels used in the result tables. General textScientific PaperCode English fiction (EF)Biology preprints (Bio)C++ NewsComputer science papers (CS)JavaScript (JS) English encyclopedia (E)Mathematics papers (Math)Markdown (MD) Non-English encyclopedia (NEE)Other scientific papers (Sci-O)Other Physics papers (Phys)Python (Py) 3.2 Compression-Based Evaluation Uncheatable Eval measures how well a model predicts text by treating the model as a lossless compressor. In arithmetic coding, the model’s next-token probabilities determine how the text is encoded: more probable sequences require fewer bits [10]. A model that predicts the observed text more accurately can therefore compress it more efficiently. We measure this efficiency using the ideal code length implied by the model’s probabilities. Specifically, letsbe a document andx 1:T its token sequence under the evaluated model’s tokenizer. The model predicts each token from the preceding tokens, assigning the document the probability p θ (x 1:T ) = T Y t=1 p θ (x t | x <t ).(1) The corresponding ideal code length, in bits, is B θ (s) =− log 2 p θ (x 1:T ) =− T X t=1 log 2 p θ (x t | x <t ).(2) 3 Uncheatable Eval Thus, code length is the next-token negative log-likelihood expressed in base 2, linking the evaluation directly to the pretraining objective. It can be computed from the probabilities of the observed tokens without asking the model to follow instructions or generate an answer. We use this theoretical code length throughout; it excludes finite-precision coding overhead and the storage cost of model parameters. To compare documents of different lengths and models with different tokenizers, we normalize the code length by the original document’s UTF-8 size. Ifn byte (s)is the number of UTF-8 bytes ins, its uncompressed size is8n byte (s)bits. We define the compression rate (CR) as the percentage of this size needed for the ideal encoding: CR θ (s) = 100· B θ (s) 8n byte (s) ,(3) For example, a CR of 10% means that the ideal encoding occupies one tenth of the original UTF-8 size. Lower CR indicates better compression and, on the same text, better prediction. The UTF-8 denominator puts models on a common scale; each score reflects the model–tokenizer pair. For a collection of documentsS, compression rate is the ratio of their total ideal code length to their total original size: CR θ (S) = 100· P s∈S B θ (s) 8 P s∈S n byte (s) .(4) We use this byte-weighted rate to summarize performance within each category. To compare models across categories, we report an Overall score that takes the unweighted arithmetic mean of the category rates. This gives each category equal weight, so categories containing more bytes do not dominate the overall comparison. 3.3 Long-Context Evaluation Long-context evaluation examines how compression changes as a model processes more of a document. We retain the loss assigned to each token and align it to the token’s UTF-8 bytes. Letℓ s,t =− lnp θ (x s,t | x s,<t )be the token loss for documents, and letm s,t be the number of UTF-8 bytes produced by that token. We assignℓ s,t /m s,t to each of its bytes. The byte-wise compression rate at position i is then CR θ,i = 100 8 ln 2 · 1 N i X s:i≤|s| ℓ s,t s (i) m s,t s (i) ,(5) wheret s (i)is the token containing byteiandN i is the number of documents that reach that position. This construction compares the same byte positions across models with different tokenizers. 4 Experiments and Results 4.1 Experimental setup Data and models. We evaluate 80 models on a July 2026 dataset with 14 text categories collected from publicly available sources. Each category contains 500 samples, for a total of 7,000 samples. During data preparation, each sample is truncated to at most 3,584 tokens under each of eight tokenizers, producing a shared text sample for all evaluated models. Table 2 presents 25 popular models, and Appendix A reports all 80. Evaluation metrics.We report compression rate (CR) as a percentage, with lower values indicating better compression. Within each category, CR is computed from the total ideal code length and total UTF-8 size using Eq. 4. For the scaling analyses, we instead combine code lengths and byte counts across categories before computing CR. We also analyze five text types separately: Scientific Paper, Code, Encyclopedia, News, and English Fiction. These contain five, five, two, one, and one categories, respectively. Long-context evaluation. The main evaluation truncates each July 2026 document to at most 3,584 tokens under each tokenizer. For long-context evaluation, we retain up to 32,768 UTF-8 bytes of the same documents. We evaluate 54 models on 500 documents from each of four scientific-paper categories: computer science, mathematics, physics, and other scientific papers. All four categories contain documents reaching 32 KiB. We compute byte-wise CR within each category using Eq. 5; pooled curves weight each category by its number of contributing documents at each byte position. 4 Uncheatable Eval 4.2 Main results Table 2: Main results of Uncheatable Eval. All score cells are compression rate (CR, %); lower is better. ModelParams. General textScientific PaperCodeOverall EF NewsEENEEBioCS Math Sci-OPhysC++JSMD OtherPyOverall >20B Gemma-4-31B31.39.177.466.986.096.436.595.986.166.313.163.707.863.984.156.00 Qwen3.5-35B-A3B-Base34.79.738.257.617.496.436.525.786.096.243.333.757.924.044.246.24 Gemma-4-26B-A4B25.89.527.687.226.416.626.926.226.446.563.293.878.344.174.386.26 Mistral-Small-24B24.09.367.857.346.826.446.796.086.306.333.444.048.394.304.496.28 Seed-OSS-36B-Base36.29.618.027.488.516.526.835.976.366.573.263.898.124.074.286.39 Nemotron-3-Nano-30B-A3B31.69.707.967.337.406.606.846.196.436.493.444.158.494.444.506.43 OLMo-3-32B32.29.608.057.719.246.586.876.186.426.604.094.858.885.074.916.79 ∼14B RWKV7-G1J-13.3B13.39.308.017.456.876.626.686.136.326.483.594.078.144.304.426.31 Ministral-3-14B13.99.868.227.646.996.446.806.036.336.363.554.218.604.464.656.44 ∼7B Qwen3.5-9B-Base9.0 10.158.608.028.246.676.766.066.346.503.654.088.284.334.516.59 Ministral-3-8B8.9 10.208.427.907.346.647.026.256.546.583.714.428.934.664.846.68 RWKV7-G1J-7.2B7.29.698.347.847.486.947.056.546.676.843.914.448.754.674.816.71 Meta-Llama-3.1-8B8.09.898.017.667.456.837.356.366.736.773.944.739.394.885.086.79 Qwen3-8B-Base8.2 10.218.618.198.126.927.226.156.686.803.634.338.994.574.746.80 Falcon-H1-7B-Base7.6 10.168.338.009.096.887.236.176.656.713.964.699.414.905.126.95 marin-8b-base8.09.918.027.86 10.456.917.256.446.756.754.335.179.635.255.467.16 ∼3B Qwen3.5-4B-Base4.2 10.659.058.498.956.997.076.416.666.833.984.438.794.664.836.98 RWKV7-G1J-2.9B2.9 10.028.678.228.057.287.446.967.057.234.254.869.365.055.217.12 Ministral-3-3B3.8 10.898.908.418.106.987.446.696.947.004.084.879.625.085.257.16 Falcon-H1-3B-Base3.1 10.848.878.65 10.257.447.726.647.127.224.525.27 10.275.535.687.57 < 2B RWKV7-G1J-1.5B1.5 10.569.168.768.847.717.917.467.507.694.695.35 10.105.535.677.64 Qwen3.5-2B-Base1.9 11.519.849.28 10.097.627.757.087.307.514.675.149.825.355.527.75 Falcon-H1-1.5B-Deep-Base1.6 11.279.369.18 11.337.908.247.107.587.835.075.88 11.066.166.208.16 Qwen3.5-0.8B-Base0.8 12.52 10.81 10.23 11.478.388.507.818.018.235.425.94 10.956.126.278.62 Falcon-H1-0.5B-Base0.5 12.96 10.35 10.53 18.959.009.298.188.588.926.207.27 13.107.547.479.88 Abbreviations: EF, English fiction; E/NEE, English/non-English encyclopedia; Bio, biology preprints; CS, computer science papers; Sci-O, other scientific papers; JS, JavaScript; MD, Markdown; Py, Python. Other denotes source code outside C++, JavaScript, and Python. Params. gives total parameters in billions. Overall is the unweighted mean of the 14 category CRs. Within each parameter-size group, bold marks the lowest displayed CR in each column, including ties. Table 2 presents compression rates for 25 representative models across the 14 text categories, together with their Overall scores; Appendix A reports the complete results for all 80 models. Qwen models [31,32] are consistently strong on scientific papers and code: Qwen3.5-35B-A3B-Base leads several scientific categories, while Qwen3.5 and Qwen3 remain among the strongest code compressors at their respective scales. RWKV7-G1J [33] shows the complementary pattern. Its models perform particularly well on general text, including the non-English encyclopedia category, where the family often leads similarly sized alternatives. Mixture-of-experts models also perform competitively with far fewer active parameters than total parameters. Qwen3.5- 35B-A3B-Base, Gemma-4-26B-A4B [34], and Nemotron-3-Nano-30B-A3B [35] reach compression rates comparable to dense models with similar total sizes, and Qwen3.5-35B-A3B-Base approaches the best Overall score in the table. 4.3 Long-context evaluation We evaluate 54 models on documents from four scientific-paper categories and report the complete long-context results in Appendix B. We focus here on Qwen3-8B-Base, Falcon-H1-7B, and RWKV7-G1J-7.2B as representatives of attention-based, hybrid, and recurrent architectures, respectively [31,36,33]. Figure 2 compares the three models’ byte-wise CR as document position increases from 1 to 32 KiB, both in absolute terms and relative to Qwen3-8B-Base. Figure 2 shows that all three models achieve lower CR at later positions, but at different rates. To quantify this trend, we summarize the byte-wise rates in five non-overlapping intervals: 1–2, 2–4, 4–8, 8–16, and 16–32 KiB. Within each 5 Uncheatable Eval 12481632 Byte position (KiB, log2) 5.5 6.0 6.5 7.0 7.5 Compression rate (%) a Absolute 12481632 Byte position (KiB, log2) −0.1 0.0 0.1 0.2 0.3 0.4 CR di f erence vs Qwen3-8B Qwen3-8B baseline b Relative Qwen3-8B-BaseFalcon-H1-7BRWKV7-G1J-7.2B Figure 2: Long-context compression for Qwen3-8B-Base, Falcon-H1-7B, and RWKV7-G1J-7.2B on four scientific-paper categories. (a) Byte-wise CR pooled over documents that reach each position, shown in logarithmically spaced bins from 1 to 32 KiB. (b) CR differences from Qwen3-8B-Base. Table 3: Long-context slopes for similar-sized models near 7–8B on four scientific-document categories. Architecture ModelCSMath PhysicsOther PooledGain AttentionQwen3-8B-Base-0.377 -0.528-0.396 -0.415-0.426 1.700 HybridFalcon-H1-7B-0.404 -0.562-0.424 -0.440-0.454 1.807 RecurrentRWKV7-G1J-7.2B -0.232 -0.461-0.280 -0.293-0.314 1.266 interval, we again weight every byte position by its document count. We then fit CR j = α + β log 2 x j ,(6) whereCR j is the weighted mean for intervaljandx j is its geometric midpoint. Ordinary least squares assigns equal weight to the five interval means. The slopeβmeasures the change in CR, in percentage points, for each doubling of byte position. We exclude positions below 1 KiB, where little preceding context is available. Table 3 reports these slopes. Among the representative models at 7–8B, Falcon-H1 improves most rapidly, followed by Qwen3 and RWKV7-G1J, with slopes of−0.454,−0.426, and−0.314, respectively. Their CR decreases by 1.807, 1.700, and 1.266 percentage points between the first and last fitted intervals. This ordering holds in all four categories; each representative model improves most rapidly on mathematics and most slowly on computer science. Relative to Qwen3-8B-Base, Falcon-H1-7B shifts from 0.025 CR points higher in the 1–2 KiB interval to 0.082 points lower at 16–32 KiB. RWKV7-G1J-7.2B shifts from 0.067 points lower to 0.367 points higher. RWKV’s short-context advantage therefore decays as document position grows, even though its absolute CR continues to improve. Our observation is consistent with von Oswald et al. [37], who find that all evaluated linear language models outperform attention-based models at early sequence positions but gradually lose this advantage as the sequence grows. To examine the role of training context, we additionally evaluate older RWKV7-G1A-1.5B/2.9B and RWKV7-G0A2- 7.2B/G0A3-13.3B checkpoints trained with 4,096-token contexts, alongside matched-size RWKV7-G1J checkpoints trained with 16,384 tokens. The short-to-long increase in RWKV’s CR gap relative to Qwen3 is smaller for G1J (0.33–0.55 percentage points) than for the older checkpoints (0.41–0.65), indicating that longer training contexts reduce relative decay in these comparisons. 4.4 Scaling Laws for Compression Rate We fit CR as a power law in total parameter countP(in billions) with an additive constant, following a common scaling-law form [38]: CR(P) = aP b + c.(7) We estimatea,b, andcby minimizing squared errors. The exponentbcontrols how quickly CR declines with model size, whilecis the asymptotic CR asPincreases. For mixture-of-experts models,Pcounts all experts. We fit the 80 models’ byte-weighted CR across all 14 categories and separately for each text type. 6 Uncheatable Eval 10 0 10 1 Parameters (B) 6 7 8 9 10 11 CR (%) All categories 10 0 10 1 Parameters (B) 7 8 9 10 CR (%) Scientific Paper 10 0 10 1 Parameters (B) 5 6 7 8 9 10 CR (%) Code 10 0 10 1 Parameters (B) 8 10 12 14 CR (%) Encyclopedia 10 0 10 1 Parameters (B) 8 9 10 11 CR (%) News 10 0 10 1 Parameters (B) 9 10 11 12 13 CR (%) English fiction Figure 3: CR versus total parameters for 80 models. Each panel shows a power-law-plus-constant fit. For each text type, CR is computed from the total code length and UTF-8 bytes across its categories. For the aggregate score, we obtain d CR(P) = 3.200P −0.290 + 5.328, R 2 = 0.915.(8) The fitted curve accounts for 91.5% of the cross-model CR variation and has a root mean squared error of 0.293 percentage points. The fitted CR curve flattens as model size grows (Figure 3). Table 4: Power-law-plus-constant fits to all 80 models by data type. All types use byte-weighted CR within the included categories. TypeCats.abc R 2 RMSE All categories143.200-0.2905.3280.9150.293 Scientific Paper52.783-0.3095.5430.8970.299 Code53.140-0.3174.0400.8340.454 Encyclopedia24.776-0.2935.7550.7210.901 News13.624-0.2146.1240.8710.323 English fiction15.743-0.1465.9960.8720.366 Parameters correspond toCR(P) = aP b + c, withPmeasured in billions. RMSE is measured across all 80 models in CR percentage points. Scaling strength varies by text type: code has the steepest exponent (−0.317), while English fiction has the shallowest (−0.146). The fits account for 72.1%–89.7% of the cross-model variation (Table 4), showing that model size explains substantially more variation for some types than for others. At the category level, fit quality varies widely:R 2 ranges from 0.594 for non-English encyclopedia articles to 0.935 for biology preprints (Appendix C). We also fit Eq. 7 to the parameter–CR Pareto frontier, retaining models for which no evaluated model has at most as many parameters and lower CR. The all-category frontier contains 20 models and follows d CR(P) = 3.244P −0.304 + 5.016 (R 2 = 0.992). Across text types, the frontier exponent ranges from−0.575for encyclopedia articles to−0.244for English fiction. These fits describe the best observed CR at each scale within our model pool; the frontier and its fitted asymptote may change as new models are evaluated (Appendix D). 7 Uncheatable Eval 4.5 Correlation with MMLU We evaluate all 80 models on zero-shot MMLU. For each question, we choose the answer with the highest next-token logit among A–D and compute accuracy over 14,042 questions. We compare this accuracy with each model’s CR pooled across five Scientific Paper and four Code categories. We measure the association with Spearman correlation. We obtain 95% confidence intervals by bootstrapping models 10,000 times for the pooled score and 5,000 times for each category. 5.566.166.697.2310.04 Technical CR (%) · quantile axis 23.1 52.6 63.5 71.6 83.1 MMLU accuracy (%) · quantile axis Spearman =0.884 a Technical CR vs. MMLU 0.50.51.5 log 10 parameters (B) Science Code Encyclopedia News Fiction Science Code Encyclopedia News Fiction 1.000.840.310.060.19 0.841.000.41-0.010.11 0.310.411.000.550.48 0.06-0.010.551.000.81 0.190.110.480.811.00 b Cross-type residual CR Figure 4: Cross-model correlations. (a) Technical-text CR versus zero-shot MMLU accuracy for 80 base models. (b) Pairwise Spearman correlations of residual CR across five text types and the same 80 models, after separate fits against log parameter count. Across 80 models, lower technical-text CR is associated with higher zero-shot MMLU accuracy (Spearmanρ =−0.884, 95% confidence interval[−0.942,−0.783]; Figure 4(a)). The correlation is similar when CR is pooled across all 14 categories (ρ =−0.871), and each of the nine technical categories shows the same pattern (ρ =−0.883 to−0.853). 4.6 Correlation analyses across text types We compute pairwise Pearson correlations among the five text types across 80 models. All ten correlations are positive, ranging fromr = 0.803to0.969. Scientific Paper and Code correlate most strongly (r = 0.969), followed by News and English fiction (r = 0.963). We then fit each type’s CR against Overall CR and examine model residuals. A negative residual means better compression on that type than the model’s Overall score predicts.The cross-type residual correlations reveal two strongly related pairs: Scientific Paper and Code (ρ = 0.838), and News and English fiction (ρ = 0.812; Figure 4(b)). Encyclopedia lies between them, correlating with both pairs but more strongly with News and English fiction (ρ = 0.484– 0.550) than with Scientific Paper and Code (ρ = 0.315–0.412). Code and News show almost no residual correlation (ρ =−0.015). 5 Conclusion We introduce Uncheatable Eval, a dynamic benchmark that evaluates base language models through lossless compression of newly published text. By converting next-token probabilities into code lengths, it measures predictive ability without task instructions or reference answers. Across 80 models and 14 text categories, CR declines with model size and varies across text types. Our long-context analysis shows that attention-based, hybrid, and recurrent models differ in how their CR changes as more context becomes available. Lower CR on technical text is also strongly associated with higher zero-shot MMLU accuracy. These results support compression of newly published text as a direct way to compare base models across domains. 8 Uncheatable Eval 6 Limitations Although newly published text reduces contamination risk, we cannot guarantee its absence from training: publication dates and model training cutoffs may be incomplete, and models may have seen copies of the same content. Our raw-text prediction protocol also limits evaluation to base models, because compression rate does not capture the instruction-following abilities acquired through post-training. Moreover, this contamination risk grows as the dataset ages: new models may train on its public texts, so we must refresh the evaluation data regularly. References [1] Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley. Data contamination through the lens of time, 2023. [2]Cheng Xu, Shuhao Guan, Derek Greene, and M-Tahar Kechadi. Benchmark data contamination of large language models: A survey, 2024. [3]Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 8706–8719. Association for Computational Linguistics, 2024. [4]Yujuan Velvin Fu, Ozlem Uzuner, Meliha Yetisgen, and Fei Xia. Does data contamination detection work (well) for llms? a survey and evaluation on detection assumptions. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5250–5271. Association for Computational Linguistics, 2025. [5]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. Livebench: A challenging, contamination-limited llm benchmark. In International Conference on Learning Representations, 2025. [6]Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida I. Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code, 2024. [7]Cheng Xu, Changhong Jin, Yingjie Niu, Nan Yan, Yuke Mei, Shuhao Guan, Liming Chen, and M-Tahar Kechadi. LiveFact: A dynamic, time-aware benchmark for LLM-driven fake news detection. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11881–11910. Association for Computational Linguistics, 2026. [8] Jiyeon Kim, Hyunji Lee, Dylan Zhou, Sue Hyun Park, Seunghyun Yoon, Trung Bui, Franck Dernoncourt, Sungmin Cha, and Minjoon Seo. Can large language models keep up? benchmarking online adaptation to continual knowledge streams. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 42240–42272. Association for Computational Linguistics, 2026. [9]Hanbing Liu, Lang Cao, and Yang Li. RAG or learning? understanding the limits of LLM adaptation under continuous knowledge drift in the real world. In Findings of the Association for Computational Linguistics: ACL 2026, pages 11234–11252. Association for Computational Linguistics, 2026. [10] Grégoire Delétang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, Marcus Hutter, and Joel Veness. Language modeling is compression, 2023. [11]Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. An open-source data contamination report for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 528–541. Association for Computational Linguistics, 2024. [12]Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787. Association for Computational Linguistics, 2023. [13]Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. Benchmarking large language models under data contamination: A survey from static to dynamic evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 10080–10098. Association for Computational Linguistics, 2025. [14] Naila Shafirni Hidayat, Muhammad Dehan Al Kautsar, Alfan Farizki Wicaksono, and Fajri Koto. Simulating training data leakage in multiple-choice benchmarks for LLM evaluation. In Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems, pages 21–39. Association for Computational Linguistics, 2025. [15] Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in llms: How do we measure it and (when) does it matter?, 2024. [16]Xingyao Xiao and Yihong Cheng. Contamination inflates scores but rarely reorders large language model leaderboards, 2026. 9 Uncheatable Eval [17]Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 12039–12050. Association for Computational Linguistics, 2024. [18]Cheng Xu, Nan Yan, Shuhao Guan, Changhong Jin, Yuke Mei, Yibing Guo, and Tahar Kechadi. Dcr: Quantifying data contamination in llms evaluation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 23002–23020. Association for Computational Linguistics, 2025. [19] Zheng Zhang, Qi Liu, Siyuan Liang, Ning Li, Zirui Hu, Weibo Gao, Rui Li, Zhenya Huang, Leszek Rutkowski, Baosheng Yu, and Dacheng Tao. Controllable contamination detection for reliable llm evaluation with statistical guarantees. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics, pages 30122–30143. Association for Computational Linguistics, 2026. [20]Kun Qian, Shunji Wan, Claudia Tang, Youzhi Wang, Xuanming Zhang, Maximillian Chen, and Zhou Yu. VarBench: Robust language model benchmarking through dynamic variable perturbation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16131–16161. Association for Computational Linguistics, 2024. [21]Zhuohao Yu, Chang Gao, Wenjin Yao, Yidong Wang, Wei Ye, Jindong Wang, Xing Xie, Yue Zhang, and Shikun Zhang. KIEval: A knowledge-grounded interactive evaluation framework for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5967–5985. Association for Computational Linguistics, 2024. [22]Qin Zhu, Qinyuan Cheng, Runyu Peng, Xiaonan Li, Ru Peng, Tengxiao Liu, Xipeng Qiu, and Xuanjing Huang. Inference- time decontamination: Reusing leaked benchmarks for large language model evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9113–9129. Association for Computational Linguistics, 2024. [23]Jianzhe Chai, Zhe Yu, and Jun Sakuma. When benchmarks leak: Inference-time decontamination for LLMs. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 44743–44760. Association for Computational Linguistics, 2026. [24] Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075–5084. Association for Computational Linguistics, 2023. [25] Ali Al-Lawati, Jason Lucas, Dongwon Lee, and Suhang Wang. Llm benchmark datasets should be contamination-resistant, 2026. Accepted to ICML 2026 Position Paper Track. [26]Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang. Antileakbench: Preventing data contamination by automatically constructing benchmarks with updated real-world knowledge. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 18403–18419. Association for Computational Linguistics, 2025. [27]Chenghao Zhu, Nuo Chen, Yufei Gao, Yunyi Zhang, Prayag Tiwari, and Benyou Wang. Is your llm outdated? a deep look at temporal generalization. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 7433–7457. Association for Computational Linguistics, 2025. [28]Ian Magnusson, Akshita Bhagia, Valentin Hofmann, Luca Soldaini, Ananya Harsh Jha, Oyvind Tafjord, Dustin Schwenk, Evan Pete Walsh, Yanai Elazar, Kyle Lo, Dirk Groeneveld, Iz Beltagy, Hannaneh Hajishirzi, Noah A. Smith, Kyle Richardson, and Jesse Dodge. Paloma: A benchmark for evaluating language model fit. In Advances in Neural Information Processing Systems 37, 2024. [29]Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin. Evaluating large language models for generalization and robustness via data compression, 2024. [30]Yuzhen Huang, Jinghan Zhang, Zifei Shan, and Junxian He. Compression represents intelligence linearly. In First Conference on Language Modeling, 2024. [31] An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report, 2025. [32] Qwen Team. Qwen3.5: Towards native multimodal agents. Official model release page, feb 2026. [33] Bo Peng, Ruichong Zhang, Daniel Goldstein, et al. RWKV-7 “goose” with expressive dynamic state evolution, 2025. [34] Gemma Team, Sherif El Abd, Vaibhav Aggarwal, et al. Gemma 4 technical report, 2026. [35] NVIDIA, Aaron Blakeman, Aaron Grattafiori, et al. Nemotron 3 nano: Open, efficient mixture-of-experts hybrid mamba- transformer model for agentic reasoning, 2025. [36] Jingwei Zuo, Maksim Velikanov, Ilyas Chahed, et al. Falcon-H1: A family of hybrid-head language models redefining efficiency and performance, 2025. [37]Johannes von Oswald, Nino Scherrer, Seijin Kobayashi, Luca Versari, Songlin Yang, Sarthak Mittal, Maximilian Schlegel, Kaitlin Maile, Yanick Schimpf, Oliver Sieberling, Alexander Meulemans, Rif A. Saurous, Guillaume Lajoie, Charlotte Frenkel, Razvan Pascanu, Blaise Agüera y Arcas, and João Sacramento. Mesanet: Sequence modeling by locally optimal test-time training. In International Conference on Learning Representations, 2026. [38]Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, et al. Scaling laws for autoregressive generative modeling, 2020. 10 Uncheatable Eval A Complete Model-by-Category Results Table 5 reports category-level results for the 80-model cohort used in the main cross-model analyses. Table 5: July 2026 category-level results for the 80-model analysis cohort. All score cells are compression rate (CR, %); lower is better. ModelParams. General textScientific PaperCodeOverall EF NewsEENEEBioCS Math Sci-OPhysC++JSMD OtherPyOverall >20B Gemma-4-31B31.39.177.466.986.096.436.595.986.166.313.163.707.863.984.156.00 Qwen3.5-35B-A3B-Base34.79.738.257.617.496.436.525.786.096.243.333.757.924.044.246.24 Gemma-4-26B-A4B25.89.527.687.226.416.626.926.226.446.563.293.878.344.174.386.26 Mistral-Small-24B24.09.367.857.346.826.446.796.086.306.333.444.048.394.304.496.28 Gemma-3-27B27.49.537.647.156.356.626.906.076.396.443.464.088.584.414.616.30 Seed-OSS-36B-Base36.29.618.027.488.516.526.835.976.366.573.263.898.124.074.286.39 Nemotron-3-Nano-30B-A3B31.69.707.967.337.406.606.846.196.436.493.444.158.494.444.506.43 Qwen2.5-32B32.89.698.107.747.646.626.965.846.406.523.414.118.554.294.476.45 Qwen3-30B-A3B-Base30.59.828.337.867.576.697.045.906.476.543.414.078.764.354.536.52 OLMo-3-32B32.29.608.057.719.246.586.876.186.426.604.094.858.885.074.916.79 marin-32b-base32.59.477.727.409.636.787.236.336.636.744.134.909.185.025.116.88 ∼14B RWKV7-G1J-13.3B13.39.308.017.456.876.626.686.136.326.483.594.078.144.304.426.31 Ministral-3-14B13.99.868.227.646.996.446.806.036.336.363.554.218.604.464.656.44 Qwen3-14B-Base14.89.898.357.907.666.686.985.936.456.553.444.108.624.354.526.53 Mistral-Nemo-Base-240712.29.367.937.517.236.597.126.336.576.653.754.418.984.664.866.57 gemma-3-12b-pt12.29.817.817.456.836.857.136.356.636.673.684.328.944.644.826.57 Qwen2.5-14B14.89.828.157.847.836.777.146.036.576.693.594.318.854.484.666.62 Gravity-16B-A3B-Base16.2 10.228.588.389.886.827.256.626.787.014.024.889.255.055.177.14 Llama-2-13b-hf13.09.857.997.718.387.267.797.077.227.314.375.24 10.025.375.627.23 ∼7B Qwen3.5-9B-Base9.0 10.158.608.028.246.676.766.066.346.503.654.088.284.334.516.59 Ministral-3-8B8.9 10.208.427.907.346.647.026.256.546.583.714.428.934.664.846.68 RWKV7-G1J-7.2B7.29.698.347.847.486.947.056.546.676.843.914.448.754.674.816.71 Meta-Llama-3.1-8B8.09.898.017.667.456.837.356.366.736.773.944.739.394.885.086.79 Qwen3-8B-Base8.2 10.218.618.198.126.927.226.156.686.803.634.338.994.574.746.80 gemma-4-E4B7.9 10.138.117.907.247.247.436.907.007.203.824.449.104.714.926.87 Falcon-H1-7B-Base7.6 10.168.338.009.096.887.236.176.656.713.964.699.414.905.126.95 Mistral-7B-v0.17.29.878.037.789.146.947.476.596.896.974.104.859.474.995.247.02 Qwen2.5-7B7.6 10.348.568.338.707.177.536.376.947.103.864.659.424.814.967.05 marin-8b-base8.09.918.027.86 10.456.917.256.446.756.754.335.179.635.255.467.16 Hunyuan-7B-Pretrain7.5 10.708.808.358.407.357.606.817.137.344.044.729.364.895.047.18 Apertus-8B-25098.1 10.488.467.957.057.447.897.557.437.754.175.009.705.155.427.25 Minitron-8B8.3 10.118.398.118.557.227.757.387.287.494.175.17 10.035.355.477.32 Olmo-Hybrid-7B7.4 10.278.558.27 10.607.037.346.646.877.094.615.469.745.655.407.39 ZAYA1-base8.8 10.658.768.449.647.427.787.127.327.454.325.19 10.015.355.547.50 OLMo-3-7B7.3 10.418.678.37 11.087.137.456.776.997.224.715.589.935.775.507.54 Llama-2-7b-hf6.7 10.188.238.038.987.598.147.477.577.674.665.59 10.555.705.967.59 Falcon3-7B-Base7.5 10.648.988.67 12.747.077.486.526.896.984.245.13 10.295.245.537.60 Continued on next page 11 Uncheatable Eval Table 5: July 2026 category-level results for the 80-model analysis cohort. All score cells are compression rate (CR, %); lower is better. (continued) ModelParams. General textScientific PaperCodeOverall EF NewsEENEEBioCS Math Sci-OPhysC++JSMD OtherPyOverall ∼3B Qwen3.5-4B-Base4.2 10.659.058.498.956.997.076.416.666.833.984.438.794.664.836.98 Spark-X2.5-4B-Base4.1 10.949.038.498.547.247.316.626.947.163.964.398.534.564.617.02 RWKV7-G1J-2.9B2.9 10.028.678.228.057.287.446.967.057.234.254.869.365.055.217.12 Nanbeige4.2-3B-Base4.2 11.139.018.29 10.417.017.136.476.716.864.004.648.794.794.817.15 Qwen3-4B-Base4.0 10.669.038.638.797.247.526.406.967.113.874.609.454.844.987.15 Ministral-3-3B3.8 10.898.908.418.106.987.446.696.947.004.084.879.625.085.257.16 gemma-3-4b-pt4.3 10.548.288.147.797.457.726.987.207.284.224.949.885.245.407.22 Llama-3.2-3B3.2 10.458.468.258.377.387.896.977.287.344.435.31 10.235.435.597.38 gemma-4-E2B5.1 10.778.518.387.847.728.037.557.557.794.445.17 10.095.415.607.49 Nanbeige4-3B-Base3.9 11.439.188.78 10.787.277.506.777.047.184.214.909.465.215.237.50 Qwen2.5-3B3.1 10.859.008.809.507.617.996.827.387.574.255.11 10.145.255.377.55 Falcon-H1-3B-Base3.1 10.848.878.65 10.257.447.726.647.127.224.525.27 10.275.535.687.57 MiniCPM5-2B-Base2.5 11.559.518.929.627.677.757.197.357.664.204.999.665.255.317.62 SmolLM3-3B-Base3.1 10.488.788.519.397.718.177.577.657.954.395.40 10.415.665.647.69 Llama-3.1-Minitron-4B-Width4.5 10.478.678.39 10.457.498.037.627.557.764.385.43 10.455.615.697.71 gemma-2-2b2.6 10.748.538.598.827.628.287.547.677.774.605.45 10.845.795.987.73 Minitron-4B-Base4.2 10.668.778.599.457.628.207.977.748.014.535.65 10.775.845.917.84 stablelm-3b-4e1t2.8 10.468.568.38 10.037.688.357.717.767.894.605.67 11.076.026.017.87 Index-1.9B2.2 11.049.198.968.797.878.327.117.687.714.996.33 11.426.366.458.02 Llama-3.1-Minitron-4B-Depth4.5 10.829.028.77 11.417.878.428.057.938.184.655.77 10.985.966.038.13 < 2B RWKV7-G1J-1.5B1.5 10.569.168.768.847.717.917.467.507.694.695.35 10.105.535.677.64 Spark-X2.5-1.7B-Base1.7 11.699.689.199.567.827.957.217.547.764.485.039.515.185.227.70 Qwen3.5-2B-Base1.9 11.519.849.28 10.097.627.757.087.307.514.675.149.825.355.527.75 Qwen3-1.7B-Base1.7 11.339.589.249.807.788.086.957.497.674.325.15 10.295.375.497.75 Qwen2.5-1.5B1.5 11.369.399.27 10.468.038.417.227.777.994.565.49 10.745.615.728.00 Youtu-LLM-2B-Base2.0 12.129.819.12 12.777.857.937.417.547.774.435.219.855.325.318.03 Falcon-H1-1.5B-Deep-Base1.6 11.279.369.18 11.337.908.247.107.587.835.075.88 11.066.166.208.16 Llama-3.2-1B1.2 11.269.088.989.508.088.657.738.008.095.116.08 11.406.196.328.18 Falcon-H1-1.5B-Base1.6 11.449.489.33 11.568.058.397.287.737.995.246.07 11.316.356.408.33 SmolLM2-1.7B1.7 10.829.159.15 13.408.128.657.868.048.364.855.82 11.366.086.138.41 stablelm-2-1_6b1.6 11.069.018.89 10.778.248.898.118.278.425.366.28 11.636.356.558.42 Qwen3.5-0.8B-Base0.8 12.52 10.81 10.23 11.478.388.507.818.018.235.425.94 10.956.126.278.62 MiniCPM5-1B-Base1.1 12.23 10.309.82 12.168.418.647.938.158.405.116.14 11.386.386.768.70 Qwen3-0.6B-Base0.6 12.41 10.65 10.30 11.668.658.917.748.298.535.015.96 11.536.156.288.72 gemma-3-1b-pt1.0 11.809.159.309.178.819.299.078.768.996.467.23 12.397.587.308.95 RWKV7-G1D-0.4B0.5 11.63 10.159.90 10.498.989.268.738.799.005.846.81 12.066.966.978.97 Qwen2.5-0.5B0.5 12.64 10.55 10.53 12.729.249.548.308.879.165.426.50 12.276.576.649.21 ERNIE-4.5-0.3B-Base-PT0.4 13.22 11.15 10.95 13.639.299.748.929.179.356.187.16 12.727.267.479.73 SmolLM2-360M0.4 12.30 10.36 10.40 16.509.239.768.979.119.475.987.01 13.137.377.269.77 Falcon-H1-0.5B-Base0.5 12.96 10.35 10.53 18.959.009.298.188.588.926.207.27 13.107.547.479.88 RWKV7-G1D-0.1B0.2 12.82 11.35 11.19 12.50 10.16 10.409.939.90 10.196.988.05 13.658.238.1410.25 gemma-3-270m0.3 13.45 10.54 10.96 11.70 10.34 10.78 10.21 10.15 10.338.379.45 14.79 10.008.8310.71 SmolLM2-135M0.1 13.38 11.44 11.44 18.72 10.37 10.97 10.53 10.36 10.726.938.39 14.589.448.7711.15 Abbreviations follow Table 2. Overall is the macro-average of the 14 category CRs. Rows are grouped by total parameter count and sorted by unrounded Overall within each group. Bold marks the best displayed value in each group, including ties at the shown precision. Params. is total parameters in billions. 12 Uncheatable Eval B Long-Context Results across Model Sizes 12481632 6 7 8 9 10 11 12 CR ( 0.1–1.6B 12481632 −1 0 1 2 ∆ CR Qwen3-0.6B Qwen2.5-0.5B Qwen2.5-1.5B Qwen3.5-0.8B RWKV7-G1D-0.1B RWKV7-G1A-1.5B RWKV7-G1J-1.5B Gemma-3-270M Gemma-3-1B Falcon-H1-0.5B Falcon-H1-1.5B Falcon-H1-1.5B-Deep Llama-3.2-1B MiniCPM5-1B StableLM-1.6B 12481632 6 7 8 9 CR ( 1.7–3.3B 12481632 −0.6 −0.4 −0.2 0.0 0.2 0.4 0.6 ∆ CR Qwen3-1.7B Qwen2.5-3B Qwen3.5-2B RWKV7-G1A-2.9B RWKV7-G1J-2.9B Gemma-2-2B Falcon-H1-3B Llama-3.2-3B MiniCPM5-2B StableLM-3B Youtu-2B SmolLM3-3B 12481632 5 6 7 8 9 CR ( 4.0–4.6B 12481632 −0.5 0.0 0.5 1.0 ∆ CR Qwen3-4B Qwen3.5-4B Gemma-3-4B Minitron-4B Minitron-4B-Width Minitron-4B-Depth Absolute CRRelative to Qwen3 Byte position (KiB, log2) Figure 5: Long-context CR for 33 models in the first three size groups. Each row shows absolute CR (left) and the difference from the Qwen3 model in that size group (right), pooled across four scientific-paper categories. 13 Uncheatable Eval 12481632 5 6 7 8 9 CR ( 6.7–9.0B 12481632 −0.50 −0.25 0.00 0.25 0.50 0.75 1.00 1.25 ∆ CR Qwen3-8B Qwen2.5-7B Qwen3.5-9B RWKV7-G0A2-7.2B RWKV7-G1J-7.2B Gemma-4-E4B Falcon-H1-7B Llama-2-7B Llama-3.1-8B Minitron-8B Mistral-7B Hunyuan-7B OLMo-3-7B Marin-8B 12481632 5 6 7 8 CR ( 12–35B 12481632 −0.50 −0.25 0.00 0.25 0.50 0.75 1.00 1.25 ∆ CR Qwen3-14B Qwen2.5-14B Qwen3.5-35B-A3B RWKV7-G0A3-13.3B RWKV7-G1J-13.3B Gemma-3-12B Llama-2-13B Absolute CRRelative to Qwen3 Byte position (KiB, log2) Figure 5: Long-context CR for 21 models in the last two size groups. Each row shows absolute CR (left) and the difference from the Qwen3 model in that size group (right), pooled across four scientific-paper categories. 14 Uncheatable Eval C Complete Category-Level Scaling Results 10 0 10 1 Parameters (B) 7 8 9 10 CR (%) Biology preprints 10 0 10 1 Parameters (B) 3 4 5 6 7 8 CR (%) C++ code 10 0 10 1 Parameters (B) 7 8 9 10 11 CR (%) Computer science papers 10 0 10 1 Parameters (B) 7 8 9 10 11 CR (%) English encyclopedia 10 0 10 1 Parameters (B) 9 10 11 12 13 CR (%) English fiction 10 0 10 1 Parameters (B) 4 5 6 7 8 9 CR (%) JavaScript code 10 0 10 1 Parameters (B) 8 9 10 11 12 13 14 15 CR (%) Markdown documents 10 0 10 1 Parameters (B) 6 7 8 9 10 CR (%) Mathematics papers 10 0 10 1 Parameters (B) 8 9 10 11 CR (%) News 10 0 10 1 Parameters (B) 6 8 10 12 14 16 18 CR (%) Non-English encyclopedia 10 0 10 1 Parameters (B) 4 5 6 7 8 9 10 CR (%) Other code 10 0 10 1 Parameters (B) 6 7 8 9 10 CR (%) Other scientific papers 10 0 10 1 Parameters (B) 7 8 9 10 11 CR (%) Physics papers 10 0 10 1 Parameters (B) 4 5 6 7 8 9 CR (%) Python code power law + constant Figure 6: Category-level scaling curves. Each panel shows CR for all 80 models and the fit defined in Eq. 7. 15 Uncheatable Eval Table 6: Category-level power-law-plus-constant fits to all 80 models in the July 2026 cohort. CategoryMedian CRabc R 2 RMSE Biology preprints7.3152.801-0.3015.5460.9350.229 C++ code4.2522.653-0.3112.5990.8280.386 Computer science papers7.7202.827-0.3055.8780.8840.321 English encyclopedia8.3804.057-0.2175.4870.9280.266 English fiction10.5505.743-0.1465.9960.8720.366 JavaScript code5.1222.972-0.3173.1840.8080.469 Markdown documents9.7814.845-0.2666.5560.8300.610 Mathematics papers6.9242.652-0.3365.2810.8160.429 News8.6693.624-0.2146.1240.8710.323 Non-English encyclopedia9.1555.602-0.3295.7680.5941.549 Other code5.2492.565-0.3953.7920.8190.473 Other scientific papers7.1652.693-0.3095.4590.8940.294 Physics papers7.2932.906-0.3025.4650.8910.316 Python code5.3972.731-0.3213.6700.8330.400 Parameters correspond toCR(P) = aP b + c, withPmeasured in billions. RMSE is measured across all 80 models in CR percentage points. D Pareto-Frontier Scaling Results For the all-category score and each text type, we fit Eq. 7 only to models on the parameter–CR Pareto frontier. A model lies on the frontier unless another evaluated model has at most as many parameters and lower CR. 10 0 10 1 Parameters (B) 6 7 8 9 10 11 CR (%) All categories other models Pareto frontier frontier fit 10 0 10 1 Parameters (B) 6 7 8 9 10 CR (%) Scientific Paper 10 0 10 1 Parameters (B) 5 6 7 8 9 10 CR (%) Code 10 0 10 1 Parameters (B) 8 10 12 14 CR (%) Encyclopedia 10 0 10 1 Parameters (B) 8 9 10 11 CR (%) News 10 0 10 1 Parameters (B) 9 10 11 12 13 CR (%) English fiction Figure 7: Pareto-frontier scaling by text type. Diamonds mark frontier models, circles mark other models, and dashed lines fit the frontier models. 16 Uncheatable Eval Table 7: Pareto-frontier scaling fits by data type on the July 2026 80-model cohort. TypeCats.Frontierabc R 2 RMSE All categories14203.244-0.3045.0160.9920.125 Scientific Paper5232.638-0.3495.4100.9920.110 Code5172.853-0.3723.8310.9910.148 Encyclopedia2172.421-0.5756.6610.9600.454 News1182.657-0.3136.6360.9930.105 English fiction1123.611-0.2447.4750.9840.176 The all-category frontier contains 20 models from 0.135B to 31.3B parameters. Fitting these 20 models gives d CR(P) = 3.244P −0.304 + 5.016(R 2 = 0.992; RMSE: 0.125 CR percentage points). Type-level frontiers contain 12–23 models, with exponents from−0.575 for encyclopedia articles to−0.244 for English fiction. E Tokenization Statistics Table 8: Native-token segmentation diagnostics for the 80-model analysis cohort. Residualρis the Spearman correlation between the residuals of CR and bytes per token after each is regressed on log 10 parameter count with an intercept. CategoryMedian bytes/tokenMinimumMaximumResidual ρ Biology preprints4.5063.7054.8080.025 C++ code3.3572.7193.731-0.194 Computer science papers4.7754.0584.9990.002 English encyclopedia4.1733.5864.458-0.244 English fiction4.3053.7554.4980.114 JavaScript code3.4252.7253.773-0.135 Markdown documents3.6072.8843.871-0.207 Mathematics papers3.5763.1443.8640.228 News4.6013.9914.8300.100 Non-English encyclopedia3.3751.8784.166-0.527 Other code3.3352.7523.681-0.175 Other scientific papers4.6243.9914.8990.080 Physics papers4.2593.7074.5550.063 Python code3.6862.9644.042-0.245 17

Instant research discovery