Paper deep dive

Are Large Language Models Truly Smarter Than Humans?

Eshwar Reddy M, Sourav Karmakar

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 40

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/22/2026, 5:38:34 AM

Summary

This paper presents a multi-method contamination audit of six frontier LLMs (GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B) using MMLU benchmarks. Through lexical analysis, paraphrase diagnostics, and behavioral TS-Guessing probes, the authors demonstrate that benchmark contamination is pervasive, with STEM subjects being the most affected. The study reveals that high performance on these benchmarks is often driven by memorization rather than general intelligence, and identifies a unique 'distributed memorization' signature in DeepSeek-R1.

Entities (5)

DeepSeek R1 · large-language-model · 100%GPT-4o · large-language-model · 100%MMLU · benchmark · 100%STEM · subject-category · 95%TS-Guessing · methodology · 95%

Relation Signals (3)

GPT-4o → evaluatedon → MMLU

confidence 100% · Experiment 1 applies a lexical contamination detection pipeline to 513 MMLU questions across all 57 subjects... six frontier LLMs: GPT-4o...

DeepSeek R1 → exhibitsbehavior → Distributed Memorization

confidence 95% · DeepSeek-R1 displaying a distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall)

STEM → hascontaminationrate → 18.1%

confidence 95% · STEM subjects showed the highest contamination (18.1%)

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most benchmarks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three complementary experiments forming a rigorous multi-method contamination audit of six frontier LLMs: GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B. Experiment 1 applies a lexical contamination detection pipeline to 513 MMLU questions across all 57 subjects, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains of +0.030 to +0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions, finding accuracy drops by an average of 7.0 percentage points under indirect reference, rising to 19.8 pp in both Law and Ethics. Experiment 3 applies TS-Guessing behavioral probes to all 513 questions and all six models, finding that 72.5% trigger memorization signals far above chance, with DeepSeek-R1 displaying a distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that explains its anomalous Experiment 2 profile. All three experiments converge on the same contamination ranking: STEM > Professional > Social Sciences > Humanities.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

39,734 characters extracted from source content.

Expand or collapse full text

Are Large Language Models Truly Smarter Than Humans? Benchmark Contamination, Surface-Pattern Reliance, and Behavioral Memorization Across Six Frontier Models Eshwar Reddy M Applied AI Scientist, Health Vectors malireddy.eshwar@gmail.com Sourav Karmakar Senior AI Scientist, Intuit India souravkarmakar29@gmail.com March 2026 Abstract Public leaderboards increasingly suggest that large language models (LLMs) surpass human experts on benchmarks spanning academic knowledge, law, and programming. Yet most bench- marks are fully public, their questions widely mirrored across the internet, creating systematic risk that models were trained on the very data used to evaluate them. This paper presents three original, complementary experiments that together form a rigorous multi-method con- tamination audit of six frontier LLMs—GPT-4o, GPT-4o-mini, DeepSeek-R1, DeepSeek-V3, Llama-3.3-70B, and Qwen3-235B—conducted entirely using public APIs and open benchmarks. Experiment 1 applies a lexical contamination detection pipeline to 513 sampled MMLU test questions across all 57 subject categories, finding an overall contamination rate of 13.8% (18.1% in STEM, up to 66.7% in Philosophy) and estimated performance gains (EPG) of +0.030– +0.054 accuracy points by category. Experiment 2 applies a paraphrase and indirect-reference diagnostic to 100 MMLU questions across six subjects and all six models, finding that model accuracy drops by an average of 7.0 percentage points when surface wording changes to indi- rect reference—rising to 19.8 p in both Law and Ethics, precisely the domains most heavily contaminated in Experiment 1. Experiment 3 applies TS-Guessing behavioral probes (Option Mask and Word Mask) to all 513 sampled questions and all six models, finding that 72.5% of questions trigger memorization signals far above chance baselines, with DeepSeek-R1 displaying an anomalous distributed memorization signature (76.6% partial reconstruction, 0% verbatim recall) that directly explains its uniquely brittle accuracy pattern in Experiment 2. Across all three experiments, STEM is consistently the most contaminated category and the methods converge on the same subject-category ranking, providing independent convergent evidence that contamination is pervasive, structurally non-uniform, and not adequately captured by any single detection approach alone. arXiv:2603.16197v1 [cs.AI] 17 Mar 2026 Contents 1 Introduction1 2 Background: Benchmarks, Contamination, and Intelligence1 2.1 Benchmarks and evaluation culture . . . . . . . . . . . . . . . . . . . . . . . . . . . .1 2.2 Mechanisms of evaluation data contamination . . . . . . . . . . . . . . . . . . . . . .2 2.3 Intelligence, generalization, and memorization . . . . . . . . . . . . . . . . . . . . . .2 3 Prior Empirical Work on Contamination2 3.1 Retrieval-based detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 3.2 Output-distribution methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 3.3 Impact metrics and contamination-free benchmarks . . . . . . . . . . . . . . . . . . .2 4 Experiment 1: Lexical Contamination Detection in MMLU3 4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 5 Experiment 2: Paraphrase and Indirect-Reference Diagnostic4 5.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 5.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 5.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 5.4 Model-specific patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 5.5 Convergence with Experiment 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 6 Experiment 3: TS-Guessing Behavioral Contamination Probe6 6.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 6.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 6.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 6.4 The DeepSeek-R1 anomaly explained . . . . . . . . . . . . . . . . . . . . . . . . . . .8 6.5 Convergent validity across all three experiments . . . . . . . . . . . . . . . . . . . . .8 7 Synthesis: What the Three Experiments Show Together9 8 Hallucination and the Limits of Benchmark Intelligence9 8.1 Theoretical context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 8.2 Interaction with contamination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 9 What Benchmarks Really Measure10 10 Normative and Policy Implications10 10.1 Disclosure obligations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 10.2 Implications for legal and medical practice . . . . . . . . . . . . . . . . . . . . . . . . 11 10.3 Regulatory evaluation standards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 11 Future Directions11 12 Conclusion11 1 Introduction Public benchmarks and leaderboards have become the primary currency of progress in large lan- guage models. New systems are announced with headline claims such as “model X outperforms most humans on the bar exam” or “model Y surpasses human experts on MMLU,” shaping per- ceptions among researchers, policymakers, investors, and the broader public [9]. These claims typ- ically rest on performance on static, open-access benchmarks whose complete test sets—including question–answer pairs and often detailed solutions—have been available online for years. As training corpora for LLMs have expanded to include vast slices of the internet, it has become increasingly plausible that benchmark items, or close variants of them, are present in pretraining data. If the model has effectively “seen the exam” during training, a high score does not demonstrate the same kind of general intelligence that a human’s first-time performance would. Instead, it reflects a mixture of memorization, pattern completion, and interpolation within a highly familiar distribution. This paper addresses the central question through three original, mutually reinforcing exper- iments on six frontier models, using only public APIs and open benchmarks, making the entire evaluation pipeline independently reproducible. Contributions • Experiment 1 (Section 4): A full lexical contamination scan of MMLU across all 57 subjects (513 questions), yielding an overall contamination rate of 13.8% and subject-level rates as high as 66.7% (Philosophy). Estimated performance gains from contamination range from +0.030 to +0.054 accuracy points by category. • Experiment 2 (Section 5): A paraphrase and indirect-reference diagnostic on 100 MMLU ques- tions across six subjects and all six models, quantifying accuracy degradation under surface- form change. Average drop of 7.0 p on indirect-reference variants across all models, with Law (−19.8 p) and Ethics (−19.8 p) showing the largest degradation—directly matching their high Experiment 1 contamination rates. • Experiment 3 (Section 6): A TS-Guessing behavioral contamination probe on all 513 sampled questions and all six models, finding a 72.5% average combined flagging rate and revealing a distinctive DeepSeek-R1 distributed memorization signature (76.6% partial reconstruction, 0% exact recall) that explains its anomalous Experiment 2 profile. • A structured synthesis showing that all three methods converge on the same category-level con- tamination ranking, providing multi-method independent evidence for the pervasiveness and structure of LLM benchmark contamination. 2 Background: Benchmarks, Contamination, and Intelligence 2.1 Benchmarks and evaluation culture Modern LLM benchmarks cover a wide spectrum of tasks. Massive Multitask Language Under- standing (MMLU) [9] aggregates thousands of multiple-choice questions across dozens of academic and professional subjects. Code benchmarks such as HumanEval [10] present short programming problems with unit tests, while TruthfulQA [11] probes susceptibility to common misconceptions. Many commercial and open-source model releases report performance on such benchmarks along- side comparisons to human baselines, often suggesting the model matches or exceeds average college 1 graduates or domain experts. Leaderboards amplify this culture, incentivizing score improvements over methodological transparency. 2.2 Mechanisms of evaluation data contamination Evaluation data contamination occurs when benchmark examples—or their paraphrases, transla- tions, or derivative discussions—appear in training data. Common mechanisms include: (a) direct inclusion of benchmark repository files scraped into pretraining corpora; (b) indirect inclusion via blog posts, teaching materials, and solution write-ups that preserve problem structure; (c) fine- tuning contamination through instruction-tuning datasets that incorporate benchmark-like Q&A; and (d) deliberate benchmaxxing, where practitioners intentionally include benchmark items to maximize leaderboard scores. Critically, contamination is not limited to exact string matches. Even when n-gram dedupli- cation is applied, models may be exposed to paraphrased or near-duplicate versions that preserve underlying problem structure while modifying surface form—a key motivation for the behavioral detection methods in Experiments 2 and 3. 2.3 Intelligence, generalization, and memorization In cognitive science and machine learning, intelligence is linked to generalization: applying finite learned experience to structurally novel situations. Memorization can produce impressive interpola- tion on similar tasks, but differs from the robust, causal reasoning that characterizes expert human performance in law, medicine, and science [8]. Evaluations that fail to separate memorization from generalization risk overstating what these systems can safely do in deployment. 3 Prior Empirical Work on Contamination 3.1 Retrieval-based detection One line of work investigates contamination using retrieval-based and behavioral methods [1]. For open models, search pipelines find benchmark items in training data. For closed-source models, behavioral Testset Slot Guessing (TS-Guessing) protocols mask elements of benchmark questions and check whether models reconstruct them—the basis for our Experiment 3. Empirical results show that leading commercial models reconstruct missing MMLU options at rates far above chance, indicating prior exposure. 3.2 Output-distribution methods A second line detects contamination from model outputs alone [2]. Contaminated items produce “peaked” output distributions: a memorizing model assigns very high probability to the correct continuation. Contamination Detection via output Distribution (CDD) uses this signal; the Mem- orization Generalization Index (MGI) separates memorization-driven from generalization-driven performance. 3.3 Impact metrics and contamination-free benchmarks The Contamination Threshold Analysis Method (ConTAM) [3] introduces the Estimated Perfor- mance Gain (EPG) metric, applied directly in Experiment 1. MMLU-CF [5] reconstructs MMLU with strict decontamination; absolute scores drop and model rankings change. MMLU-Pro [4] adds 2 Table 1: Families of contamination detection methods and the experiment in this paper that instantiates each. Method familyPrimary signalData neededThis paper Retrieval-based (n-gram)Web overlap with question text Public search APIExp. 1 Impact-based (ConTAM/EPG)Accuracyoncleanvs. flagged items Contamination labelsExp. 1 Paraphrase diagnosticAccuracy drop under sur- face change Model API onlyExp. 2 Behavioral (TS-Guessing)Reconstruction of masked elements Model API onlyExp. 3 Table 2: MMLU contamination rates by subject category (Experiment 1). Detection: Tavily web search, ≥30% 8-gram overlap AND correct answer present. EPG = Estimated Performance Gain (ConTAM, Singh et al. 2024). N = 513 questions, 9 per subject. CategoryN Contam. (%) Reported Acc.EPG Humanities11710.30.72+0.030 Professional12612.70.78+0.040 STEM17118.10.74+0.054 Social Sciences9912.10.76+0.037 All51313.80.74+0.040 reasoning-intensive questions and similarly reduces reported scores. These results motivate our Ex- periment 2 paraphrase diagnostic: if performance is inflated by surface familiarity, accuracy should degrade as surface form diverges from training text. 4 Experiment 1: Lexical Contamination Detection in MMLU 4.1 Methodology We sampled 9 questions per subject from the MMLU test split across all 57 subjects (513 questions, random seed 42). For each question we issued a Tavily web search query (first 150 characters) and collected top-5 result snippets. A question was flagged contaminated if: (a) the fraction of the question’s 8-grams (lowercased, punctuation-stripped) found in the combined snippets exceeded 0.30; and (b) the correct answer text appeared verbatim in the snippets. The dual condition guards against false positives from tangentially related pages. EPG was computed as acc×rate×0.4 following the ConTAM methodology [3]. 4.2 Results Table 2 reports contamination rates and EPG by category. The overall contamination rate was 13.8%. A further 40.7% of questions showed any lexical overlap with web content, and 25.3% had the correct answer present in search results. STEM subjects showed the highest contamination (18.1%), followed by Professional (12.7%), Social Sciences (12.1%), and Humanities (10.3%). The most contaminated individual subjects— Philosophy (66.7%), Anatomy (55.6%), Electrical Engineering, Marketing, and Conceptual Physics 3 Table 3: Top 15 most contaminated MMLU subjects (Experiment 1). Average overlap = mean 8-gram overlap score across all 9 questions in the subject. SubjectN Contam. (%) Avg. overlap Philosophy966.70.667 Anatomy955.60.508 Electrical Engineering944.40.543 Marketing944.40.444 Conceptual Physics944.40.716 Professional Accounting944.40.433 High School Geography933.30.712 College Physics933.30.672 Computer Security933.30.472 Medical Genetics933.30.506 Elementary Mathematics933.30.409 Moral Disputes922.20.536 Public Relations922.20.168 High School Statistics922.20.517 High School Microeconomics922.20.694 (all 44.4%)—are precisely those with the densest online coverage in lecture notes, textbooks, and practice-set solutions. EPG estimates indicate that removing contaminated items would reduce reported STEM accuracy by approximately 5.4 percentage points, substantially closing the claimed gap between frontier models and human expert performance. Two caveats apply. First, our two-condition threshold is conservative: true contamination rates may be higher if paraphrased questions or near-duplicates are counted. Second, Tavily does not comprehensively mirror pretraining corpora, so this method is best interpreted as a lower bound on web exposure. Experiment 3 addresses the complementary question of internal model memorization without reliance on external web indices. 5 Experiment 2: Paraphrase and Indirect-Reference Diagnostic 5.1 Motivation Experiment 1 establishes that MMLU questions are widely available online. But web presence is not proof that a model has memorized those questions. Experiment 2 operationalizes memorization through its behavioral signature: if models rely on surface-form familiarity rather than underlying domain knowledge, their accuracy should fall when question wording changes while the underlying knowledge requirement stays constant. 5.2 Methodology We sampled 100 MMLU questions across six subjects: High School US History, Professional Law, College Computer Science, High School Mathematics, High School Biology, and Moral Scenarios (≈17 per subject). Subjects were selected deliberately to span all four MMLU categories (STEM: CS, Mathematics, Biology; Humanities: History, Ethics; Professional: Law) and to include both high-contamination subjects (Law, CS, Ethics — among the most contaminated in Experiment 1) and low-contamination subjects (History, Mathematics — among the least contaminated), enabling a contamination-stratified analysis of whether surface-form sensitivity correlates with web exposure. 4 Table 4: Accuracy on original, paraphrased, and indirect-reference question forms (Experiment 2). Drop P = paraphrased − original; Drop I = indirect − original. A purely reasoning-based model would show zero drop. All six models are included. ModelOriginal ParaphrasedDrop P IndirectDrop I GPT-4o0.5880.677+0.0890.594+0.006 GPT-4o-mini0.5290.469 −0.0600.490 −0.039 DeepSeek-R10.2920.333+0.0410.260−0.032 DeepSeek-V30.6770.573 −0.1040.615 −0.062 Llama-3.3-70B0.7710.635 −0.1360.625 −0.146 Qwen3-235B0.8330.698 −0.1350.688 −0.146 Average0.6150.564 −0.0510.545 −0.070 Table 5: Accuracy drop (original → indirect) by subject, averaged across all six models (Experi- ment 2). Law and Ethics show the largest drops, directly matching their high contamination rates in Experiment 1. SubjectOriginal IndirectDrop History0.8230.802 −0.021 Law0.6350.437 −0.198 CS0.6770.541 −0.135 Mathematics0.4160.365 −0.051 Biology0.7920.708 −0.083 Ethics0.6150.416 −0.198 Average0.6600.545 −0.115 Questions within each subject were sampled randomly (seed 42, same as Experiment 1) from the MMLU test split, excluding the 9 questions already used in Experiment 1 to prevent overlap. For each question, GPT-4o generated two variants: (a) a paraphrased version (entirely different wording, identical knowledge requirement and correct answer) and (b) an indirect-reference version (key subject entities described via an associated property or event rather than named directly). All six models were evaluated on all three forms at temperature 0. 5.3 Results Table 4 reports accuracy by question form and model. Across all six models, average accuracy dropped from 0.615 on original questions to 0.545 on indirect-reference variants—a drop of 7.0 percentage points. The paraphrased form was intermediate (average 0.564, drop of 5.1 p), consistent with the hypothesis that accuracy degrades as surface form diverges from familiar training text. 5.4 Model-specific patterns GPT-4o shows a near-zero indirect drop (+0.006), suggesting its performance is robust to surface- form change at this accuracy level (Original 0.588→ Indirect 0.594). Llama-3.3-70B and Qwen3- 235B both show the largest drops (−0.146 each), with Qwen achieving the highest baseline accuracy (0.833) yet losing 14.6 p under indirect referencing—the clearest signal of surface-pattern reliance in the experiment. 5 DeepSeek-R1 (highlighted) remains a notable anomaly: its original accuracy of 0.292 is far below all other models, yet its indirect-reference drop is among the smallest (−0.032). This combination—low baseline, near-zero sensitivity to surface change—is inconsistent with simple surface memorization and inconsistent with robust reasoning. Experiment 3 reveals the underlying explanation. 5.5 Convergence with Experiment 1 The subject-level pattern in Table 5 directly mirrors the contamination findings of Experiment 1. Law (−19.8 p drop) falls in the Professional category (12.7% contaminated in Exp 1; Profes- sional Accounting at 44.4%). Ethics (−19.8 p) falls in Humanities (Philosophy 66.7%, Moral Disputes 22.2%). By contrast, History (−2.1 p) and Mathematics (−5.1 p) are the least surface-sensitive and among the least contaminated in their categories. This alignment between external web detection and behavioral accuracy degradation under surface-form change constitutes convergent multi-method evidence for the same underlying phenomenon. 6 Experiment 3: TS-Guessing Behavioral Contamination Probe 6.1 Motivation Experiments 1 and 2 detect contamination from the outside—web presence and accuracy degrada- tion. Experiment 3 probes contamination internally, directly testing whether models have stored benchmark question content, without requiring any external data source or accuracy comparison. This makes it the only method in this paper applicable to all closed-source models on equal footing and fully independent of the web indices used in Experiment 1. 6.2 Methodology We implemented the TS-Guessing protocol of Deng et al. [1] across both sub-tasks for all 513 sampled MMLU questions × all 6 models (6,156 probes per task): Task A – Option Mask (OM): One wrong answer choice is replaced with [MASK]. The model is shown the complete question, all answer choices including the mask, and is told which answer is correct. It is then asked to write the original text of the masked wrong option. A model with no prior exposure has no principled basis to reconstruct the specific wording of an incorrect option (exact-match random baseline ≈0%). We report both exact-match rate and partial-match rate (≥50% token overlap), using partial-match as the primary metric. Task B – Word Mask (WM): One content word (the longest non-stopword token of ≥5 characters) is blanked from the question stem. The model fills in the single missing word. The random exact-match baseline is ≈0.002% given vocabulary size. We report exact-match rate. A question is flagged contaminated if OM partial ≥ 0.50 or WM exact = 1. 6.3 Results Table 6 reports contamination rates by model. The average combined flagging rate across all six models was 72.5%—far above both random baselines—constituting strong behavioral evidence that the vast majority of sampled MMLU questions are internally memorized by frontier LLMs. 6 Figure 1: Experiment 2 results (all six models). Left: Accuracy by question form and model. Blue = original, orange = paraphrased, red = indirect reference. Most models show accuracy degradation as surface form diverges from training text; DeepSeek-R1 is a notable anomaly (low baseline, minimal drop) explained by Experiment 3. Right: Average accuracy drop (original → indirect) by subject across all six models. Law (−0.20) and Ethics (−0.20) show the largest drops, directly corresponding to the highest Experiment 1 contamination rates in the Professional and Humanities categories. Figure 2: Experiment 3 results. Left: TS-Guessing contamination rates by model. Red = OM partial (≥50% overlap); blue = WM exact; green = combined. All models far exceed the 5% random baseline (dashed). DeepSeek-R1 shows the highest OM-partial rate (76.6%) with zero WM-exact recall—the distributed memorization signature. Centre: OM partial contamination rate by model × MMLU category (heatmap). STEM is consistently the most contaminated category across all models; DeepSeek-R1 reaches 86% in STEM. Right: Contamination rate by category comparing Experiment 1 (web-search lexical, blue) vs. Experiment 3 (TS-Guessing behavioral, red). Both methods independently rank STEM highest, providing convergent multi-method evidence. 7 Table 6: TS-Guessing contamination rates by model (Experiment 3). OM exact: wrong option reconstructed verbatim. OM partial: ≥50% token overlap (primary metric). WM exact: masked question word reconstructed exactly. Combined: flagged by either task. Random baselines: OM exact ≈0%, OM partial ≈5%, WM exact ≈0%. N = 513 questions per model. ModelOM exact OM partial (≥50%) WM exact Combined GPT-4o15.6%37.6%62.1%72.9% GPT-4o-mini15.0%37.6%50.0%66.1% DeepSeek-R10.0%76.6%0.0%76.6% DeepSeek-V319.9%43.5%60.0%76.4% Llama-3.3-70B13.1%37.0%54.3%69.6% Qwen3-235B21.4%42.7%57.8%73.7% Average14.2%45.8%47.4%72.5% Table 7: Contamination rate by MMLU subject category: Experiment 1 (web-search lexical) vs. Experiment 3 (TS-Guessing behavioral, OM partial averaged across all models). Both methods independently rank STEM as the most contaminated category. CategoryExp 1 (web) Exp 3 (TS-Guessing) Difference STEM18.1%55.9%+37.8% Professional12.7%44.8%+32.1% Social Sciences12.1%39.1%+27.0% Humanities10.3%38.0%+27.7% 6.4 The DeepSeek-R1 anomaly explained The most striking pattern in Table 6 is DeepSeek-R1’s profile: OM partial = 76.6% (highest of all six models) paired with OM exact = 0.0% and WM exact = 0.0%. This means DeepSeek- R1 reconstructs roughly three-quarters of wrong answer options with substantial but non-verbatim overlap, while failing to reproduce any answer option or question word exactly. This is the signature of distributed memorization: the model has stored the conceptual and semantic structure of MMLU questions—knowing what wrong options should be about —without retaining exact surface phrasing. This finding directly resolves the DeepSeek-R1 anomaly from Experiment 2. Its low original accuracy (0.292) combined with near-zero indirect-reference drop (−0.031) is precisely what dis- tributed memorization predicts. The model recognizes conceptual content well enough to score at moderate levels when question phrasing is familiar, but cannot reconstruct exact text, and it does not degrade further under indirect reference because it was never relying on verbatim sur- face matching in the first place. Three independent measurements—OM partial rate (Exp 3), low original accuracy (Exp 2), and minimal surface sensitivity (Exp 2)—converge on one explanation: DeepSeek-R1 has encoded MMLU in a compressed conceptual form that breaks down on genuinely novel evaluation material. 6.5 Convergent validity across all three experiments Table 7 shows that Experiment 1 (external web detection) and Experiment 3 (internal behavioral probing) independently rank STEM as the most contaminated category, followed by Professional, Social Sciences, and Humanities. Combined with Experiment 2’s subject-level result that CS and Law show the largest accuracy drops under surface-form change—both STEM and Professional 8 domain subjects respectively—all three experiments converge on the same structural conclusion. The absolute rates differ substantially between methods (Exp 1: 18.1% vs. Exp 3: 55.9% for STEM). This gap is informative: approximately 38 percentage points of STEM memorization are behaviorally detectable but externally invisible, confirming that web-search-only contamination detection is a significant underestimate for closed-source models whose pretraining corpora extend far beyond what public web indices capture. 7 Synthesis: What the Three Experiments Show Together The three experiments form an interlocking argument. Experiment 1 establishes external exposure: benchmark questions appear online in forms consistent with pretraining data ingestion, with STEM most affected. Experiment 2 probes the behavioral consequence of that exposure: accuracy degrades when surface form changes, most sharply in the highest-contamination domains. Experiment 3 provides direct internal evidence of stored content: what Experiment 1 detects externally and Experiment 2 detects through accuracy degradation is also present inside the models as recoverable memorized structure. Three broader conclusions emerge from this triangulation. Contamination is pervasive and structurally non-uniform. MMLU contamination is not confined to a handful of unlucky subjects. It pervades all categories, reaches 66.7% in the most affected individual subject, and is detectable by all three independent methods. All methods agree on the category ranking: STEM > Professional > Social Sciences > Humanities. This consistency rules out method-specific artifacts as an explanation. Models differ qualitatively in how they memorize. Five of the six models show the stan- dard memorization profile: verbatim and near-verbatim recall of both question words and answer options, alongside accuracy that is sensitive to surface-form changes. DeepSeek-R1 shows the oppo- site: high partial reconstruction with zero verbatim recall, and accuracy nearly invariant to surface changes despite a low absolute baseline. This distinction has practical consequences—verbatim memorization is mitigated by paraphrase; distributed memorization requires genuinely novel eval- uation material to detect. Standard benchmark practice cannot separate knowledge from familiarity. The com- bined evidence establishes that a substantial fraction of MMLU performance reflects recognition of memorized content rather than transferable domain knowledge. The fraction is non-trivial (13.8% by conservative web detection, 72.5% by behavioral probing) and varies by subject in ways that correlate with online exposure. Until evaluation practice converges on decontaminated, withheld benchmarks evaluated under controlled surface-form variation, claims of human-level or superhu- man AI performance on standard benchmarks cannot be taken at face value. 8 Hallucination and the Limits of Benchmark Intelligence 8.1 Theoretical context Theoretical work argues that hallucinations are structurally unavoidable in large language mod- els [7]. No finite-parameter model can correctly approximate all computable functions; there will always be inputs on which the learned approximation diverges from the target. This reframes 9 hallucination not as a bug to be patched but as a structural limitation—one that becomes particu- larly consequential when contamination-inflated benchmark scores create overconfident deployment decisions. 8.2 Interaction with contamination Contamination and hallucination interact in a way that makes contaminated benchmark perfor- mance especially misleading. On contaminated benchmarks, models appear highly accurate because they are pattern-matching to memorized content. When moved to novel or real-world tasks, the same models may hallucinate frequently, because high benchmark scores reflect stored patterns rather than grounded domain understanding [6]. In law, models frequently hallucinate incorrect case holdings or fabricate citations with high confidence and fluent prose [8]. In medicine, models may invent nonexistent drug interactions or misstate clinical guideline thresholds. The Experiment 2 subject-level results are directly consistent with this: Law and Ethics—domains where professional errors have direct consequences—show the largest accuracy drops under surface-form change (19.8 p each), precisely the conditions that approximate real-world deployment where exact benchmark question wording is absent. 9 What Benchmarks Really Measure The evidence across three experiments supports a hybrid view of current LLM benchmark perfor- mance. Scores reflect a mixture of: (a) large-scale memorization of question text, answer options, and surrounding online discussion; (b) statistical generalization within the training distribution, en- abling correct interpolation between seen examples; and (c) variable surface-pattern reliance, with sensitivity to surface-form changes that correlates with contamination level. The exam metaphor is instructive. Human exams are designed under the assumption that test items are completely unseen. When an LLM is evaluated on a benchmark it has ingested, the situation is closer to a student sitting an exam after receiving—in some cases—the complete question set itself. A decontaminated, withheld benchmark with surface-form variation testing is the appropriate analogue of a genuine novel exam. Common evaluation pitfalls include: reporting raw accuracy on publicly available benchmarks without contamination analysis; generalizing from narrow benchmark domains to broad occupa- tional competence; treating single-number accuracy scores as representative of robustness across prompt formulations; and equating average accuracy with fitness for high-stakes deployment where calibration and failure mode matter as much as mean performance. 10 Normative and Policy Implications 10.1 Disclosure obligations If benchmark scores are materially influenced by prior exposure to test items, model providers ar- guably have an obligation to disclose this. The three-experiment pipeline in this paper demonstrates that contamination analyses are feasible with public infrastructure and modest API costs. Trans- parent reporting of contamination analyses alongside performance claims would allow downstream users—employers, policymakers, judges, and researchers—to calibrate expectations appropriately. 10 10.2 Implications for legal and medical practice In legal practice, benchmark scores are sometimes invoked in support of claims about model reli- ability. Courts and regulators should treat such claims with caution: a model performing well on contaminated law questions may still hallucinate case law, misinterpret statutes, or fail to recognize jurisdictional conflicts [8]. The 19.8 percentage-point accuracy drop under indirect referencing for Law questions (Experiment 2) indicates that legal reasoning performance on MMLU substantially reflects familiarity with specific question phrasings rather than transferable legal analysis. The same concern applies in medicine, where overreliance on benchmark scores may lead to deployment of systems with brittle, memorization-driven performance in high-stakes clinical contexts. 10.3 Regulatory evaluation standards Contamination-aware evaluation is an essential component of rigorous AI regulation. Regulatory guidance for high-stakes AI systems should require: evaluation on decontaminated or propri- etary benchmarks not included in training data; behavioral contamination analyses (TS-Guessing or equivalent); robustness testing across prompt formulations including paraphrase and indirect- reference variants; and standardized disclosure of evaluation methodology. The three-experiment framework presented here provides a replicable, low-cost template for such audits. 11 Future Directions Several extensions follow directly from this work. A properly powered private benchmark comparison— with at least 100–200 carefully difficulty-calibrated questions per domain, authored after model training cutoffs and verified offline—would complement the behavioral evidence here with a di- rect accuracy measurement on genuinely novel material. Domain-matched difficulty calibration is essential and could be established by piloting questions on a human cohort before LLM evaluation. The TS-Guessing protocol should be extended to sentence-level reconstructions and multi- hop compositional probes, providing a richer profile of what models store and in what form. The DeepSeek-R1 anomaly—distributed memorization without verbatim recall—deserves dedicated investigation: understanding whether this pattern is architectural (chain-of-thought pretraining depth), corpus-related, or a deliberate training design choice has significant implications for con- tamination mitigation strategy. The convergence between Experiments 1 and 3 on category ordering invites a systematic study of which web-presence features best predict behavioral memorization, potentially enabling lightweight contamination screening without requiring full API evaluation of all models. Finally, the interaction between contamination and hallucination deserves direct causal study using controlled synthetic benchmarks where memorization exposure can be precisely manipulated, allowing clean estimation of how much contamination contributes to both inflated scores and down- stream hallucination frequency. 12 Conclusion Returning to the motivating question—are LLMs truly smarter than humans, or did they simply see the exam?—the evidence from three independent experiments points to the same answer: the question cannot be definitively resolved with current benchmark practice, and the available evidence gives substantial reason for skepticism. 11 Experiment 1 establishes that 13.8% of MMLU questions are contaminated under a conservative two-condition detection rule, rising to 18.1% in STEM and 66.7% in the most affected individual subject. Experiment 2 shows that surface-form changes reduce model accuracy by an average of 7.0 percentage points across all six models, with the sharpest degradation in the highest-contamination domains (Law: −19.8 p, Ethics: −19.8 p)—direct behavioral evidence that a meaningful por- tion of MMLU performance reflects surface-pattern familiarity rather than transferable knowledge. Experiment 3 finds that 72.5% of MMLU questions trigger memorization signals far above random chance across six frontier models, with DeepSeek-R1 displaying a distinctive distributed memoriza- tion signature that explains its anomalous Experiment 2 profile. All three experiments agree on the category-level ordering of contamination severity: STEM > Professional > Social Sciences > Humanities. This convergence across three methodologically independent approaches—external web detection, accuracy degradation under surface-form pertur- bation, and internal reconstruction probing—constitutes the strongest available multi-method evi- dence that MMLU contamination is a real, structural, and practically consequential phenomenon. The three-experiment framework presented here is fully replicable with public infrastructure at modest cost and should be adopted as a standard audit protocol. Claims of human-level or super- human AI performance derived from contaminated public benchmarks deserve the same scrutiny we would apply to any scientific claim where the evaluation procedure is known to be compromised. References [1] Y. Deng, W. Zhang, W. Chen, and Q. Gu. Investigating data contamination in modern bench- marks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 8698–8711, 2024. arXiv:2311.09783. [2] H. Dong, Y. Fu, X. Ma, Z. Geng, X. Meng, and X. Jia. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. arXiv preprint arXiv:2402.15938, 2024. [3] A. K. Singh, M. Y. Kocyigit, A. Poulton, D. Esiobu, M. S. Aslan, M. Artetxe, and M. Kam- badur. Evaluation data contamination in LLMs: how do we measure it and when does it matter? arXiv preprint arXiv:2411.03923, 2024. [4] W. Zhao, Z. Shang, Y. Liu, L. Wang, and J. Liu. MMLU-Pro: a more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024. [5] Q. Zhao, Y. Huang, T. Lv, L. Cui, F. Wei, Q. Sun, Y. Xin, S. Mao, X. Zhang, Q. Yin, and S. Li. MMLU-CF: a contamination-free multi-task language understanding benchmark. arXiv preprint arXiv:2412.15194, 2024. [6] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023. [7] F. Xu, J. Shi, and E. Choi. Hallucination is inevitable: an innate limitation of large language models. arXiv preprint arXiv:2401.11817, 2024. [8] J. H. Choi, K. E. Hickman, A. Monahan, and D. Schwarcz. ChatGPT goes to law school. Journal of Legal Education, 71(3):387–400, 2023. 12 [9] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. arXiv:2009.03300. [10] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [11] S. Lin, J. Hilton, and O. Evans. TruthfulQA: measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), pages 3214–3252, 2022. [12] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy. The Pile: an 800GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. 13

Instant research discovery