Paper deep dive
Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation
M. Ali Bayram
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Adding answer options can lower multiple-choice scores without improving assessment validity. Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections. Each question retains its stem, five original options and source key, and receives five options copied from other questions in the same section. Sentence-embedding retrieval proposes candidates; a language model selects existing identifiers. Deterministic verification reconstructs all 60,000 additions. A 25-model calibration exposes scoring and generation-budget effects. Five evaluations produce source-key accuracies of 34.8%-81.4%. On 981 shared questions, one API-served model falls from 93.7% with five choices to 83.1% with ten; 102 of 115 lost correct responses select borrowed options. The decrease is 24.4 percentage points on heuristically flagged negative stems and 5.9 points elsewhere. A completed human-checked audit of 200 sampled questions, with undocumented reviewer tool use, yields 47 and 31 multiple-answer judgments across the two record sets, 25 of the latter unresolved. These records support concern about ambiguity, while their dependence and incomplete reviewer-method documentation limit validation. Because order and labels also change, the paired comparison measures augmentation as implemented. The contribution is a traceable construction and an analysis of its validity limits, not evidence that lower ten-choice scores measure knowledge better.
Tags
Links
- Source: https://arxiv.org/abs/2609.15467v1
- Canonical: https://arxiv.org/abs/2609.15467v1
Trouble viewing inline? Open PDF directly →
Full Text
74,867 characters extracted from source content.
Expand or collapse full text
Turkish MMLU Pro: Traceable Option Augmentation and Its Validity Limits in Turkish Multiple-Choice Evaluation M. Ali Bayrammalibayram20@gmail.com Department of Computer Engineering, Yıldız Technical University İstanbul, Turkey Abstract Adding answer options can lower multiple-choice scores without improving assessment validity. Turkish MMLU Pro examines this distinction using 12,000 Turkish-source questions across 58 sections. Each question retains its stem, five original options and source key, and receives five options copied from other questions in the same section. Sentence-embedding retrieval proposes candidates; a language model selects existing identifiers. Deterministic verification reconstructs all 60,000 additions. A 25-model calibration exposes scoring and generation- budget effects. Five evaluations produce source-key accuracies of 34.8%–81.4%. On 981 shared questions, one API-served model falls from 93.7% with five choices to 83.1% with ten; 102 of 115 lost correct responses select borrowed options. The decrease is 24.4 percentage points on heuristically flagged negative stems and 5.9 points elsewhere. A completed human- checked audit of 200 sampled questions, with undocumented reviewer tool use, yields 47 and 31 multiple-answer judgments across the two record sets, 25 of the latter unresolved. These records support concern about ambiguity, while their dependence and incomplete reviewer-method documentation limit validation. Because order and labels also change, the paired comparison measures augmentation as implemented. The contribution is a traceable construction and an analysis of its validity limits, not evidence that lower ten-choice scores measure knowledge better. 1 Introduction Multiple-choice evaluation depends on the answer options and scoring procedure as well as the question. Option order can change model predictions, and generated answers can disagree with first-token probability scores (Pezeshkpour & Hruschka, 2024; Wang et al., 2024a). Increasing the number of options lowers the uniform-guessing reference rate, but additional options are useful only if they preserve a defensible key and discriminate relevant knowledge. Educational research cautions against equating more options with better assessment (Rodriguez, 2005; Gierl et al., 2017). Turkish evaluation includes curriculum-based TurkishMMLU (Yüksel et al., 2024), the broader five-choice TR-MMLU (Bayram et al., 2025b), translated MMLU-Pro-TR (bezir, 2024), and multi-task suites such as TurkBench and CETVEL (Toraman et al., 2026; Er et al., 2026). This work investigates a different construction choice: expanding an existing Turkish question collection by borrowing option text from the same source. This preserves inspectable provenance and avoids generating new wording, but an option that is incorrect for its donor need not be incorrect for its recipient. This work analyses option augmentation and its validity limits. The central finding is a limitation of the construction itself: source-key agreement falls most on stems flagged as negative or exception questions, where borrowed options can introduce additional defensible answers. The evidence combines a 25-model calibration study, five complete runs, a paired comparison on 981 shared questions and a completed 200-question audit 0 Generative AI was used for option-identifier selection during dataset construction and for assistance with manuscript review, literature checks, diagnostic analysis code, and audit preparation. The author verified the text, analyses, and references and accepts responsibility for the work. Section 7 gives the full disclosure, including the limits of the reviewer-tool records. 1 arXiv:2609.15467v1 [cs.CL] 14 Sep 2026 with explicit reconciliation and missingness, separating deterministic provenance checks, source-key scores and fallible human-checked semantic judgments. The accompanying resource, Turkish MMLU Pro, comprises 12,000 questions with ten options across 58 near-balanced sections and an auditable protocol. It is offered as an instrument for studying this failure mode, not as a validated ten-choice benchmark: these scores are not proposed for ranking models. 2 Related Work TurkishMMLU contains 9,807 test questions and 225 development questions across nine high-school subjects, with student correctness information that supports difficulty analysis (Yüksel et al., 2024). TR-MMLU evaluates 6,200 questions across 62 sections (Bayram et al., 2025b;a). The source collection is the larger collection behind that lineage; the present benchmark overlaps with TR-MMLU and is not an independent test set relative to it. TUMLU also draws its Turkish questions from TurkishMMLU and converts questions to four options (Isbarov et al., 2025). Thus, existing Turkish resources are neither uniformly five-choice nor necessarily disjoint. MMLU-Pro-TR translates English MMLU-Pro into Turkish (bezir, 2024). TurkBench and CETVEL cover broader task families, including capabilities beyond standalone knowledge questions (Toraman et al., 2026; Er et al., 2026). These resources complement the narrower setting examined here. The term Turkish-source denotes stems taken from a Turkish collection rather than translated from MMLU-Pro for this study; available metadata does not establish each upstream item’s original authorship language. MMLU-Pro combines question filtering, expanded options, and expert review (Wang et al., 2024b). The approach used in this work does not reproduce that pipeline: it neither filters questions by model difficulty nor generates new option text. Mining item banks for distractors has prior precedent (Baldwin et al., 2022), while surveys distinguish candidate generation from expert and response-based evaluation (Alhazmi et al., 2024; Benedetto et al., 2025). Same-section retrieval and model selection are engineering choices here, not evidence that the resulting options are uniquely incorrect. MMLU-Redux documents unclear questions, wrong keys, and missing or multiple valid answers (Gema et al., 2025). These categories are directly relevant to augmentation. A source key can remain unchanged while the addition of another valid answer makes single-key scoring misleading. The strict scorer used in this work addresses a separate issue: reproducibly extracting an explicit answer from generated text. Accurate extraction cannot compensate for an invalid item. The Distractor Assessment Framework separates incorrectness, plausibility, and diversity using automated reading-comprehension models (Raina et al., 2023). Such measures offer candidate-screening baselines, but their transfer to Turkish subject questions would require validation against qualified reviewers. Large-option evaluation on Korean orthography uses repeated resampling, shuffling, and padding controls to investigate presentation effects (Lee & Son, 2026). The present study instead has one saved presentation per condition. It adopts descriptive position diagnostics, while reserving causal attribution for controlled reevaluation. 3 Dataset Construction 3.1 Source and structural filtering The source isalibayram/turkish_mmluat revision30c94d45...c91(alibayram, 2026b). Its records contain section, topic, question, a zero-based answer index, explanation, and five choices. Stable question identifiers are the first 24 hexadecimal characters of SHA-256 over the source revision, split, and row position. Explanations are not included in evaluation prompts. The pooled source has 279,302 records from its train, test, and mmlu splits. Structural filtering removes malformed records, repeated normalized choices, option-dependent formulations such as “all of the above”, likely missing visual references, duplicate normalized stems, and duplicate groups with conflicting answer texts. One deterministic owner is retained for repeated stems. This leaves 232,963 usable records (Table 1); 2 ItemCount Source split: train232,136 Source split: test40,966 Source split: mmlu6,200 Raw total279,302 Automatically excluded46,339 Usable source records232,963 Selected benchmark records 12,000 Table 1: Recomputed source audit and selected-set size. usable denotes these automated checks, not expert approval. The original split names describe provenance, not independent training and testing partitions for the construction described here. Sections with fewer than 200 usable questions are excluded. Seed 42 controls question queues and quota tie-breaking; the target is allocated as evenly as capacity permits. Sections unable to supply a quota are dropped and capacity is redistributed. The selected set contains 58 sections with 206 or 207 questions each. Selection is therefore conditional on structural eligibility and successful construction, rather than a representative sample of Turkish examinations or language use. 3.2 Retrieval and forced selection Pipeline version 6 embeds question stems usingparaphrase-multilingual-mpnet-base-v2at revision 4328cf...11f5. Sentence embeddings support candidate retrieval (Reimers & Gurevych, 2019); they do not verify answer correctness. Up to 60 same-section donor questions interleave exact topic-label matches with nearest embedding neighbors. Options are collected round-robin from donors, excluding the recipient itself, canonical duplicates, option-dependent text, and simple answer-format mismatches. The candidate pool is capped at 100 options and 16,000 option characters. One Ollama call togemma4:26b-mlxselects exactly five distinct candidate identifiers. It uses seed 42, temperature 0, a 256-token output ceiling, a 16,384-token context setting, andthink=false. The instruction retains the source key, prioritizes avoiding alternative correct answers and semantic duplicates, and then prefers answer-type and topic fit. It forbids creating or editing text. Crucially, it requires five selections even when candidates are weak. Malformed selections skip a question, and three consecutive malformed replies stop the run after checkpointing; candidate shortages and transport failures are explicit failures. The forced-selection policy supplies a fixed option count but lacks a semantic rejection mechanism. A valid identifier response establishes only that five candidates passed structural checks. It does not establish their plausibility or incorrectness. The selector weights also appear in the evaluation roster, so construction-model dependence is disclosed rather than assumed absent. 3.3 Export and verification Each accepted question combines its five original and five borrowed options. A deterministic permutation derived from its identifier and seed shuffles the ten options, and the source key is remapped to its displayed position. Verification reconstructs each record and checks exact source text, donor split/row/option location, donor section, answer remapping, and duplicate constraints. Display prompts strip surrounding whitespace; stored text is retained. All 12,000 selected records and 60,000 additions pass reconstruction. The recorded 12,022 attempted questions and 12,000 acceptances describe pipeline throughput, not a quality rate. Borrowed options average 18.13 characters compared with 24.08 for original options (medians 13 and 16). Length may provide an origin cue. The displayed gold counts range from 1,131 to 1,273 across A–J; a fixed shuffle does not establish robustness to position. Section 5.4 examines these cues descriptively using saved responses. 3 The pinned Hub snapshot, revision609d608...794d, contains the complete selected set (alibayram, 2026a), but its manifest is explicitly an accepted-candidate preview withcomplete:false. The repository is visible, while file access requires manual approval. Evaluation uses the local completed preparation export, whose Parquet hash matches that snapshot. The distinction is preserved rather than changing a preview flag. Exact hashes and regeneration instructions are supplied in Appendix G. 4 Evaluation Protocol 4.1 Requests, scoring, and run identity Each question is submitted independently with a fixed Turkish prompt requesting one answer letter and no explanation (Appendix A). Localdirectmode requeststhink=falsewith a 2,048-token output ceiling; thinkingrequeststhink=truewith a 4,096-token ceiling. Both use seed 42 and otherwise retain runtime defaults. The conditions therefore differ in both the reasoning request and token budget: their comparison is not an isolated effect of reasoning. These ceilings were chosen during calibration after a 42-token ceiling caused some models to exhaust their output budget before producing a visible answer. Direct mode denotes a request, not verified absence of reasoning. Per-response records preserve visible output, reported token counts, truncation, and whether a separate thinking channel or closing reasoning tag was observed. Such flags are backend-dependent measurements, not a complete view of internal computation. Strict scorer version 2 discards any prefix through the last</think>tag and unwraps a whole-answer Xform. It then accepts a valid letter in specified wrappers, normalized exact option text, or a consistent label plus exact option text. Other output is unparseable. This operational definition is deliberately reproducible, although tag handling is a parsing heuristic rather than validation of well-formed reasoning markup. Raw responses are retained, and saved indices and correctness labels are checked against this implementation when generating the paper’s tables. Accuracy is agreement with the source key over all questions; an unparseable response counts as incorrect. Valid-answer rate and invalid/truncated counts distinguish response failures from valid selections that disagree with the key. A truncated response can still score if it contains a parseable answer. Runs with no parseable answers are marked unranked rather than interpreted as zero knowledge. A separate legacy scorer reproduces earlier label heuristics and threshold-free embedding fallback; its scores are diagnostic only. Benchmark hash, prompt, mode, scorer version, generation settings, runtime, and model identity are retained in run records. The pilot uses a shared comparison hash and separately records each model digest. The full runner includes model identity in each run hash, so the analysis checks common conditions directly rather than equating hashes across models. Only complete runs with compatible conditions are compared; local and API results are displayed separately without a joint rank. Deterministic data export and saved settings do not guarantee bit-identical inference. 4.2 Calibration and statistical interpretation The fixed 100-question calibration set was sampled from an earlier 7,634-question construction snapshot covering 37 sections (Hub revisioncfefc193...c277). Selection sortsSHA-256(seed + NUL + soru_id) with seed 42 and retains the first 100 identifiers. The resulting sample covers 34 sections, and all 100 rows remain identical in the completed benchmark. It is not a random sample of the final 12,000 questions or its 58-section composition. It was used for protocol development and mode selection, so it is a calibration set rather than an untouched validation set. The pilot covers 25 local tags from eight open-weight families on a Mac mini (M4, 32 GB), Ollama 0.33.2, one request at a time on 1–2 September 2026. Runtime-reported sizes span 0.75–36B parameters. All direct runs completed; storage exhaustion stopped thinking after ten models. That subset reflects execution order and storage, not representative sampling. Appendix E supplies quantizations and the retained guardrail classifier’s task-format failure. 4 ModelCorrect Accuracy95% CIAnswer Tokens Reas. gemma4:31b7,63763.64 [62.78, 64.50] 99.851.1 0.0 gemma4:26b † 6,90157.51 [56.62, 58.39] 99.811.1 0.0 gemma4:e4b5,33044.42 [43.53, 45.31] 99.941.0 0.0 qwen3.5:4b4,18134.84 [33.99, 35.70] 99.831.2 0.0 gpt-6-astra 9,76681.38 [80.68, 82.07] 99.9127.7 46.1 Table 2: Complete runs (n= 12,000). Accuracy and answer rate are percentages; intervals are 95% Wilson intervals. Tokens are mean reported generated tokens per question, including reported reasoning. Reas. is the percentage of responses with observed/reported reasoning. The first four rows are local NVFP4 models; the final row uses an API. Local tags omit -mlx. † Selector weights. This work reports 95% Wilson intervals for individual accuracy proportions and paired Wald intervals with exact McNemar tests for changes on the same questions (Dror et al., 2018). Intervals use an item-independence approximation. Scores on the fixed corpus are observed quantities; these intervals express item-sampling uncertainty under that approximation, not uncertainty from repeated generation, key errors, or dataset construction. Shared donors and related topics may introduce dependence. Pilot mode tests are exploratory, with ten comparisons and a Bonferroni threshold of 0.005. Overlapping marginal intervals are not used as a test of differences between models. No confirmatory ranking inference is claimed. 4.3 Pilot findings Direct-mode accuracy ranges from 8% to 65% among models with parseable answers; the guardrail classifier produces none. For the 22 ranked models without observed reasoning, legacy accuracy exceeds strict accuracy by 2–13 points (mean 5.6). A wrong bare letter can reach the legacy embedding fallback and select another option, including the gold. That gold-aware control flow makes the legacy score unsuitable as an alternative knowledge estimate. Thinking improves accuracy in seven of ten completed pairs, but none passes the Bonferroni threshold. Its budgets differ, and mean request time grows by factors of 2–627. Some direct requests also produce substantial generation or recorded reasoning. These findings motivate strict answer extraction and explicit budget reporting, rather than treating a bare-letter request as equal computation. Appendix E gives the complete pilot, timing and truncation results. 5 Complete Evaluations 5.1 Setup and results Four local models from the pilot completed 12,000-question runs on the same Mac mini under Ollama 0.34.0 on 10–11 September 2026:gemma4:31b-mlx,gemma4:26b-mlx,gemma4:e4b-mlx, andqwen3.5:4b-mlx, all NVFP4 builds. These are the first local runs to complete, not a random sample of models. Their runtime- reported total parameter counts are 31.7B, 26.2B, 8.1B, and 4.5B, respectively; tag sizes need not equal total parameters. The 26B model uses the selector’s exact digest. The fifth run queries the API identifiergpt-6-astrathrough OpenAI on 10 September with reasoning_effort=low, seed 42, andmax_completion_tokens=2048. One request used the standard tier and 11,999 used two Batch jobs. The served-model string remained unchanged, but it is not a weight digest. These are the recorded request settings; they do not establish that other effort levels were unavailable. The API backend, reasoning control, and sampling defaults differ from the local setup. Strict accuracy ranges from 34.84% to 81.38% (Table 2). Section-macro and item-micro accuracy differ by less than 0.01 percentage points because section sizes are nearly equal. Valid-answer rates are 99.81–99.94%, with 7–23 invalid responses per model and one truncated response, fromqwen3.5:4b-mlx. The selector scores 5 Questionskn Accuracy 95% CI Guess-adjusted Reas. Tokens TR-MMLU, all questions5 6,20093.35 [92.71, 93.95]91.69 28.1 18.1 Turkish MMLU Pro, all questions10 12,00081.38 [80.68, 82.07]79.31 46.1 27.7 Shared questions, original options5 98193.68 [91.98, 95.04]92.10 29.9 19.2 Shared questions, with borrowed options 10 98183.08 [80.60, 85.29]81.20 48.2 28.3 Table 3: Five- and ten-choice results forgpt-6-astra. Only the 981 shared questions form a paired comparison. All accuracy columns are percentages. Guess-adjusted is (a−1/k)/(1−1/k), with accuracya on a 0–1 scale andkoptions; it is a reference transformation, not a validated knowledge estimate. Reas. is reported reasoning prevalence; tokens are mean generated tokens per question. 57.51%; without alternative construction models, its participation in selection cannot be assigned a positive or negative effect. The API run reports reasoning on 5,529 questions (46.1%); 251,777 of 332,984 output tokens are reported as reasoning tokens. Its accuracy is 96.2% when reasoning is not reported and 64.1% when it is. These subsets are selected by the model, so the association identifies neither the causal effect of reasoning nor an independent estimate of question difficulty. Local models average 1.0–1.2 generated tokens per question with no observed reasoning. The full-run timing and saved-rate cost estimates are documented in Appendix G; they distinguish accumulated local request time from API completion span. The four local models retain their pilot order, and each full score falls within its pilot interval. However, the pilot is nested in the full benchmark and the runtime changed; this is a consistency check, not independent validation. Removing all 100 calibration questions gives a separate 11,900-question sensitivity analysis (Appendix G). Section scores are reported in Appendix F; their interpretation is limited by roughly 207 questions per section and heterogeneous subject labels. 5.2 Paired five-versus-ten comparison The source’s mmlu split contains the 6,200 five-choice TR-MMLU questions across 62 sections. Of these, 981 occur in Turkish MMLU Pro. The analysis reconstructs their original options from provenance and verifies exact equality of stem text, original option text, and source key. The original five-choice order is retained by TR-MMLU, whereas Turkish MMLU Pro shuffles all ten options. Thus, additions, option order, displayed labels, and prompt length change together. This design measures the implemented augmentation, not the isolated causal effect of adding five options. The evaluation ofgpt-6-astraon TR-MMLU was conducted on 11 September with the same recorded model identifier, request parameters, and scorer. The prompt changes its letter range from A–J to A–E. One request used the standard tier and 6,199 used a Batch job; two responses were invalid. The five-choice run’s estimated token charge is US$7.16 under its saved rates. Different dates and provider-side nondeterminism remain possible influences despite matching recorded settings. On shared questions, accuracy falls from 919/981 (93.68%) to 815/981 (83.08%), a paired change of−10.60 percentage points (95% Wald interval [−12.74,−8.46]; exact McNemarp= 5.27×10 −23 ). There are 804 questions correct in both conditions, 115 correct only with five choices, 11 correct only with ten, and 51 correct in neither. Of the 115 losses, 102 select borrowed options, 11 select original distractors, and two are invalid. Reported reasoning prevalence rises from 29.9% to 48.2% on these questions. The unpaired full-corpus scores reflect different item compositions. Guess-adjusted columns in Table 3 are reference transformations, not corrections for ambiguity or presentation effects; Appendix G gives a stylized guessing calculation. 6 ModelBenchmarkUnflagged FlaggedGap gpt-6-astraTR-MMLU93.792.5 +1.2 gemma4:31b-mlx Turkish MMLU Pro71.740.6 +31.2 gemma4:26b-mlx † Turkish MMLU Pro65.833.8 +32.1 gemma4:e4b-mlx Turkish MMLU Pro50.726.5 +24.3 qwen3.5:4b-mlx Turkish MMLU Pro40.319.4 +20.9 gpt-6-astraTurkish MMLU Pro88.062.6 +25.3 Table 4: Source-key accuracy (%) by the approximate negative-stem rule. Unflagged/flagged sample sizes are 8,882/3,118 for Turkish MMLU Pro and 4,517/1,683 for TR-MMLU. Gap is unflagged minus flagged, in percentage points. † Selector weights. Stem groupn Five Ten Change 95% CI Lost/gained Flagged negative 250 88.4 64.0-24.4 [-30.3, -18.5]66/5 Unflagged731 95.5 89.6-5.9 [-7.8, -3.9]49/6 Table 5: Post hoc paired changes on the 981 shared questions, split by the same rule. Accuracy is percent; change is ten minus five choices in percentage points, with a paired 95% Wald interval. Lost/gained counts refer to source-key correctness. 5.3 Borrowed options and negative stems Across the five full runs, 56.8–70.9% of valid responses that disagree with the key select a borrowed option. There are five borrowed and four original non-key options per question; evenly distributing such responses would give a borrowed share of 5/9 = 55.6%. This is a descriptive reference, not a realistic behavioral null or proof that borrowed options function as valid distractors. A model can select one because it is misleading or because it is defensible. Negative stems make the latter mechanism particularly clear. For a question asking which option does not belong to a category, a borrowed option outside that category can satisfy the stem alongside the source key. The selector instruction warns against this, but forced selection provides no way to reject an unsuitable candidate pool. The analysis investigates the pattern with a keyword rule applied to the question sentence, including Turkish negation and negative-aorist forms (Appendix B). The rule flags 3,118/12,000 Turkish MMLU Pro questions and 1,683/6,200 TR-MMLU questions. Section 5.5 compares the rule with the audit records; these are not an independent gold standard, and unflagged does not mean confirmed positive. All five models score 20.9–32.1 points lower on flagged stems in Turkish MMLU Pro (Table 4). The corresponding gap for the API model on TR-MMLU is 1.2 points. These full-corpus contrasts can reflect subject composition as well as wording; the paired subset provides a more focused diagnostic. Among the 250 flagged shared questions, accuracy decreases from 88.4% to 64.0% (−24.4 points), versus −5.9 points among 731 unflagged questions (Table 5). Flagged questions account for 61 of the 104 net losses. Of their 66 previously correct answers that become incorrect, 64 select borrowed options; 38 of the 49 corresponding losses on unflagged questions do so. This response pattern alone does not identify ambiguous items. The separate audit examines recorded answer validity on a corpus sample, rather than adjudicating all 981 paired items. Unflagged losses likewise do not establish valid distractor difficulty. 5.4 Presentation and source-overlap diagnostics Post hoc diagnostics describe position and length associations using saved responses (Table 6). Invalid responses remain incorrect in position-specific accuracy; entropy uses valid responses. Error primacy compares A selections on errors with a uniform reference over each item’s nine non-key positions. Appendix C defines all quantities. These are associations across different questions, not within-item interventions. 7 ModelRange Slope Entropy A excess Length rank gemma4:31b 58.4–67.7 -0.960.996+3.00.525 gemma4:26b 52.9–60.7 -0.680.997+0.90.523 gemma4:e4b 42.3–50.6 -0.190.994+0.10.536 qwen3.5:4b 17.4–56.1 -4.570.886+10.90.515 gpt-6-astra 79.4–85.1 -0.450.999+4.90.528 Table 6: Descriptive diagnostics on the five complete ten-choice runs. Range: minimum–maximum gold- position accuracy (%). Slope: item-level linear accuracy slope in percentage points per later gold position. Entropy: response entropy divided bylog10. A excess: error primacy in percentage points. Length rank: mean selected-option length midrank among the nine non-key options, on valid errors (0 shortest, 1 longest; uniform reference 0.5). Answer-set classification R1n R1 W% R2n R2 W% Unique source key15280.8 11968.1 Multiple answers4718.83114.9 Single alternative10.5259.0 Unresolved00.0258.0 Key + borrowed answer4315.6156.3 Table 7: Reconciled audit records on the same 200 sampled questions.nis the observed count; W% weights each question by its inverse inclusion probability and uses the full 12,000 total weight, including unresolved items. The first four rows partition each record set. The final row is a subset of multiple-answer judgments. These are descriptive record summaries, not certified corpus-validity rates. The largest position association occurs forqwen3.5:4b-mlx: accuracy ranges from 17.4% to 56.1%, with a slope of−4.57 points per later gold position and an A-error excess of 10.9 points. The API model has a narrower 79.4–85.1% range and near-uniform aggregate response entropy (0.999), yet its A-error excess is 4.9 points. High aggregate entropy therefore does not establish absence of position association. Since different questions occupy different gold positions, these statistics do not measure the effect of moving a fixed answer or partition the paired augmentation loss. Within a question, an original option is longer than a borrowed option in 58.84% of original–borrowed comparisons, counting ties as one half. This indicates a modest length cue to origin. On valid errors, mean selected length ranks are 0.515–0.536, slightly above the uniform non-key reference. In an exploratory subset of 151 shared questions whose borrowed mean length is within 10% of the original mean, accuracy still decreases from 141/151 (93.38%) to 127/151 (84.11%): 15 losses and one gain, a change of−9.27 points (paired Wald interval [−14.25,−4.29]). This text-based restriction is a sensitivity check, not a length intervention: it neither matches full length distributions nor controls total prompt length, content, or option order. Provenance also documents item coupling. Of 33,617 distinct donors, 9,068 supply options to multiple recipients; 9,601/12,000 selected questions share at least one donor with another recipient. Also, 11,986/60,000 additions come from questions that are themselves in the selected set, affecting 5,518 recipients. There are 15,682 additions across original source-split boundaries. Normalized-stem comparison finds no duplicate stems within the selected set, but 94 selected stems also occur in another raw source split; deduplication retains one owner rather than removing upstream copies. These are construction dependencies and source overlaps, not evidence that a model encountered the questions during training. They reinforce the need to avoid treating source split names or donor-linked records as independent evaluation partitions. 8 5.5 Completed audit and reconciled records The audit samples 200 questions from all 58 sections and 116 section-by-flag strata, including 101 flagged and 99 unflagged items. Sampling precedes annotation; seed-42 hash ordering and fixed inclusion weights are retained. Twelve separate calibration items are excluded. Each sampled item has two sets of reviewer records, denoted R1 and R2, and five borrowed-option judgments per set. The author reports the unchanged rubric, calibration before labelling, full 200-item coverage and independence until reconciliation; both reviewers are native Turkish speakers with university degrees, without per-discipline expertise. Reviewer AI use is not established, and R2’s second-pass fields are script-generated, so the records are human-checked but unverified (Appendix D). A separate 1,054-entry correction log resolves exported labels while preserving original records. Corrections explicitly use the other reviewer’s answers, so the final records are reconciled judgments, not independent repeated measurements. For 46 R2 questions the log supplies revised answer sets; 25 are unresolved. Dependent status labels are derived from those sets and kept separate from recorded judgments. Blank rubric fields remain missing. Appendix D details the procedure and limitations. R1 records 47 multiple-answer items; R2 records 31 and leaves 25 unresolved (Table 7). The source key and a borrowed option are both marked defensible in 43 and 15 items, respectively. These observations support concern that borrowing can introduce competing answers, but cross-reviewer dependence and unresolved cases prevent a pooled validity estimate. The original polarity fields agree with the keyword rule on 187/200 and 188/200 items; their weighted counts appear in Appendix D. This checks recorded polarity consistency without establishing expert-gold precision or recall. All model scores retain the original frozen keys; no response is rescored against the audit. 6 Discussion The construction establishes text preservation and provenance, and the saved-response comparison establishes lower source-key agreement after augmentation. The completed audit provides human-checked records of multiple defensible answers, with disagreement and unresolved items retained. Together, these results support a validity failure mode worth investigating. They do not establish that ten-choice scores measure Turkish knowledge better, or separate ambiguity from presentation effects. A larger, documented domain-expert adjudication and controlled reevaluation are needed for a validated replacement benchmark. A revised construction should permit fewer additions or reject an item when five verified non-answer options cannot be found. Negative questions require each addition to fail the requested exception. Lexical filtering alone will miss problems and remove valid questions. Revised keys/options require a new artifact version and new evaluations; the current scores cannot be transferred. Reporting flagged strata makes the observed weakness visible without certifying unflagged items. 7 Conclusion Turkish MMLU Pro makes option borrowing traceable across 12,000 Turkish-source questions. The paired analysis finds a 10.6-point decrease in source-key agreement, concentrated on flagged negative stems. The completed audit records competing answers and unresolved judgments, so lower scores do not establish better assessment. A validated successor needs semantic rejection at construction, expert adjudication and controlled reevaluation. 9 Limitations The audit is completed, but its evidential scope is limited. Unverified reviewer tool use and cross-reviewer reconciliation preclude treating the final records as independent expert ratings. The reported protocol order is the author’s account rather than a property established by the retained records, and domain-specific qualifications are not claimed. Twenty-five R2 answer sets remain unresolved, and other missing fields are not imputed. Inverse-inclusion weighting corrects sampling allocation, not annotation error. Reported audit percentages are descriptive; no confidence interval or corpus certification is inferred from them. Several design limits constrain the paired result. It covers one model, one fixed presentation per condition, and a selected overlap of 981 questions rather than all source items. Original options are reordered and relabelled during export, and requests occur on different dates. These controls are specified rather than left open. A four-condition reevaluation is designed: a five-choice baseline in original A–E order; ten choices with the original options fixed and borrowed options appended F–J, isolating option count from reordering; ten choices interleaved under frozen seeds, preserving relative order while changing labels; and unrestricted frozen permutations including the evaluated export, re-run concurrently rather than compared against the historical run. Repeated permutations require analysis clustered by question rather than treated as independent items, with seeds fixed before inference and no post hoc seed selection. Extending the paired design to several models and to an alternate selector would separate construction-model effects from augmentation. Independent item adjudication by domain specialists, without cross-reviewer reconciliation, should precede any claim about increased valid difficulty. None of these has been executed, and no result is reported or implied for them. The subgroup analysis was motivated after inspecting responses and remains exploratory. The calibration set was drawn from an earlier, incomplete section roster, reused during protocol development, and retained in the full benchmark. Runtime defaults, quantization, a single prompt, and one generation per item limit generalization; seed recording does not quantify repeat-run variance. The ten completed thinking runs form an incomplete subset and differ from direct runs in both requested reasoning and output ceiling. The full results cover four selected local builds and one API identifier. API-reported model names and reasoning tokens offer weaker control than pinned weights, and a common direct-mode label does not imply equal computation. External leaderboard and publisher scores do not provide controlled language comparisons. The available TR- MMLU leaderboard uses older scoring and different or unrecorded quantizations, while publisher evaluations differ in data and harness settings. The experimental comparison is therefore restricted to saved runs whose conditions can be inspected. The research dossier retains the external scores as context, without using their rank correlations or score gaps as evidence for augmentation. The source was accessible before construction and includes earlier training and evaluation splits. Exact stem deduplication does not eliminate paraphrases or possible pretraining exposure. The overlap and donor audit documents source reuse, but no model-training contamination test or independent human difficulty study has been conducted. Near-equal section sizes are an explicit weighting choice; they do not represent the educational or social importance of subjects. These scores concern prompted answer selection, not comprehensive Turkish ability, safety, or suitability for consequential decisions. The pinned source card has conflicting license labels: machine-readable C BY-NC-ND 4.0 and prose C BY-NC 4.0, with additional third-party material noted. This records an unresolved provenance and distribution question, not a legal determination. The curator reports that the underlying material was collected from public, often unattributed sources. That provenance statement does not itself establish item- level redistribution permissions. Dataset documentation follows the provenance and intended-use concerns of datasheets (Gebru et al., 2021). Researcher verification and anti-training terms are access policies, not a resolution of these source-rights questions. Ethical Considerations Author contributions. M. Ali Bayram conceived the study, developed the dataset and evaluation software, conducted the experiments, analysed the results, and wrote and revised the manuscript. 10 Funding and competing interests. This work received no specific funding. The author declares no financial or non-financial competing interests. Ethics and consent. The work uses an existing question collection, automated model responses and a completed human-checked audit. The author reports recruiting both reviewers through personal contacts. Both participated as unpaid volunteers and received no compensation. They were informed informally that their labels would support a published paper and that anonymized records would be released, and both gave informal verbal consent; no written consent was collected. The work was never submitted to an institutional ethics board, and no approval or exemption is claimed or implied. Reviewer records are reported under R1/R2 identifiers; beyond native Turkish proficiency and a university degree, no further demographic or professional credentials are collected or inferred. Source licensing and third-party content are discussed in the Limitations section. Data and code availability. Three artifacts are distinguished. First, the selected-set snapshot is identified by its pinned Hub revision (alibayram, 2026a). Its metadata marks it an accepted-candidate preview, but its 12,000-question Parquet file is byte-identical to the completed export: preview names the repository’s release stage, not a smaller or provisional dataset. Second, the completed manifest, evaluation responses and analysis scripts are local artifacts without an established public archive identifier. Third, donor provenance records and the audit packet remain private because they contain source question text. On 12 September 2026 the repository was publicly visible but manually gated, and unauthenticated file requests were denied; visibility therefore does not establish unrestricted download, which requires repository approval. The source permission conflict described in the Limitations section remains unresolved; repository visibility does not resolve permission for a final transformed-data release. The planned release will retain manual access approval for verified researchers, with evaluation-only terms prohibiting use of the test set for model training or fine-tuning and unauthorized redistribution. These measures aim to reduce contamination; they cannot guarantee that previously accessible source questions are absent from training corpora. Peer-review access must preserve reviewer anonymity and must not require reviewers to identify themselves to the author. Use of generative AI. A language model selected existing option identifiers during dataset construction as described in Section 3.2. AI assistance supported manuscript review and revision, literature checks, diagnostic analysis code and audit preparation. Whether the two audit reviewers used AI tools while producing their judgments could not be established: no per-reviewer tool record was kept, and the author cannot confirm their working practice. The returned material includes a model-query script whose execution status is unknown, and second-pass fields that are reproducibly script-generated. Because reviewer tool use cannot be excluded, the records are reported as human-checked but unverified rather than as independent human adjudication. Explicit corrections, missingness and reconciliation are retained rather than treating automated fields as independently validated judgments. The author has reviewed the text, analyses, and references and accepts responsibility for them. Reconciled records of unverified provenance are not represented as independent expert adjudication. References Elaf Alhazmi, Quan Z. Sheng, Wei Emma Zhang, Munazza Zaib, and Ahoud Alhazmi. Distractor generation in multiple-choice tasks: A survey of methods, datasets, and evaluation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 14437–14458, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.799. URLhttps://aclanthology.org/2024.emnlp-main. 799/. alibayram. Turkish MMLU Pro construction snapshot, 2026a. URLhttps://huggingface.co/datasets/ alibayram/turkish-mmlu-pro-preview/tree/609d60827ba18e7a499d8100b84086d85702794d. Construction-complete selected set captured 1 September 2026; manifest remains typed as an accepted-candidate snapshot. 11 alibayram. Turkish MMLU source collection, 2026b. URLhttps://huggingface.co/datasets/alibayram/ turkish_mmlu/tree/30c94d45e29424a074dc754910fb44019284c91. Pinned source revision; year de- notes this study’s access date. Peter Baldwin, Janet Mee, Victoria Yaneva, Miguel Paniagua, Jean D’Angelo, Kimberly Swygert, and Brian E. Clauser. A natural-language-processing-based procedure for generating distractors for multiple-choice questions. Evaluation & the Health Professions, 45(4):327–340, 2022. doi: 10.1177/01632787211046981. URL https://doi.org/10.1177/01632787211046981. First published online 9 November 2021. M. Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümüş, Banu Diri, Savaş Yıldırım, and Öner Aytaş. Setting standards in Turkish NLP: TR-MMLU for large language model evaluation, 2025a. URLhttps: //arxiv.org/abs/2501.00593. Version 2, 4 January 2025; first submitted 31 December 2024. M. Ali Bayram, Ali Arda Fincan, Ahmet Semih Gümüş, Banu Diri, Savaş Yıldırım, and Öner Aytaş. TR- MMLU benchmark for large language models: Performance evaluation, challenges, and opportunities for improvement. In 2025 33rd Signal Processing and Communications Applications Conference (SIU), p. 1–4. IEEE, 2025b. doi: 10.1109/SIU66497.2025.11112154. Luca Benedetto, Shiva Taslimipoor, and Paula Buttery. A survey on automated distractor evaluation in multiple-choice tasks. In Ekaterina Kochmar, Bashar Alhafni, Marie Bexte, Jill Burstein, Andrea Horbach, Ronja Laarmann-Quante, Anaïs Tack, Victoria Yaneva, and Zheng Yuan (eds.), Proceedings of the 20th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2025), p. 55–69, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-270-1. doi: 10.18653/v1/2025.bea-1.5. URL https://aclanthology.org/2025.bea-1.5/. bezir. MMLU-Pro-TR: Turkish translation of MMLU-Pro, 2024. URLhttps://huggingface.co/datasets/ bezir/MMLU-pro-TR. Dataset and accompanying evaluation repository; accessed 31 August 2026. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker’s guide to testing statistical significance in natural language processing. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1383–1392, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/ P18-1128. URL https://aclanthology.org/P18-1128/. Yakup Abrek Er, Ilker Kesen, Gözde Gül Şahin, and Aykut Erdem. Cetvel: A unified benchmark for evaluating language understanding, generation and cultural capacity of LLMs for Turkish. In Vera Demberg, Kentaro Inui, and Lluís Marquez (eds.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1052–1085, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-380-7. doi: 10.18653/v1/2026.eacl-long.46. URL https://aclanthology.org/2026.eacl-long.46/. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé I, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86–92, 2021. doi: 10.1145/3458723. URL https://doi.org/10.1145/3458723. Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, Claire Barale, Robert McHardy, Joshua Harris, Jean Kaddour, Emile Van Krieken, and Pasquale Minervini. Are we done with MMLU? In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 5069–5096, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-189-6. doi: 10.18653/v1/2025.naacl-long.262. URL https://aclanthology.org/2025.naacl-long.262/. Mark J. Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6): 1082–1116, 2017. doi: 10.3102/0034654317726529. URL https://doi.org/10.3102/0034654317726529. 12 Jafar Isbarov, Arofat Akhundjanova, Mammad Hajili, Kavsar Huseynova, Dmitry Gaynullin, Anar Rzayev, Osman Tursun, Aizirek Turdubaeva, Ilshat Saetov, Rinat Kharisov, Saule Belginova, Ariana Kenbayeva, Amina Alisheva, Abdullatif Köksal, Samir Rustamov, and Duygu Ataman. TUMLU: A unified and native language understanding benchmark for Turkic languages. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 22816–22838, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.1112. URL https://aclanthology.org/2025.acl-long.1112/. Nahyun Lee and Guijin Son. Pushing the boundaries of multiple choice evaluation to one hundred options. arXiv:2604.14634, version 1, 2026. URL https://arxiv.org/abs/2604.14634. Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, p. 2006–2017, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.130. URL https://aclanthology.org/2024.findings-naacl.130/. Vatsal Raina, Adian Liusie, and Mark Gales. Assessing distractors in multiple-choice tests. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, p. 12–22. Association for Compu- tational Linguistics, 2023. doi: 10.18653/v1/2023.eval4nlp-1.2. URLhttps://aclanthology.org/2023. eval4nlp-1.2/. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), p. 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410/. Michael C. Rodriguez. Three options are optimal for multiple-choice items: A meta-analysis of 80 years of research. Educational Measurement: Issues and Practice, 24(2):3–13, 2005. doi: 10.1111/j.1745-3992.2005. 00006.x. URL https://doi.org/10.1111/j.1745-3992.2005.00006.x. Cagri Toraman, Ahmet Kaan Sever, Ayşe Aysu Cengiz, Elif Ecem Arslan, Görkem Sevinç, Sarp Kantar, Mete Mert Birdal, Yusuf Faruk Güldemir, Ali Buğra Kanburoğlu, Sezen Felekoğlu, Birsen Şahin Kütük, Büşra Tufan, Elif Genç, Serkan Coşkun, Gupse Ekin Demir, Muhammed Emin Arayıcı, Olgun Dursun, Onur Gungor, Susan Üsküdarlı, Abdullah Topraksoy, and Esra Darıcı. TurkBench: A benchmark for evaluating Turkish large language models. In Kemal Oflazer, Abdullatif Köksal, and Onur Varol (eds.), Proceedings of the Second Workshop Natural Language Processing for Turkic Languages (SIGTURK 2026), p. 126–154, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-370-8. doi: 10.18653/v1/2026.sigturk-1.12. URL https://aclanthology.org/2026.sigturk-1.12/. Xinpeng Wang, Bolei Ma, Chengzhi Hu, Leon Weber-Genzel, Paul Röttger, Frauke Kreuter, Dirk Hovy, and Barbara Plank. “my answer is C”: First-token probabilities do not match text answers in instruction-tuned language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, p. 7407–7416, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.441. URLhttps://aclanthology. org/2024.findings-acl.441/. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weim- ing Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: A more robust and challenging multi-task language understanding benchmark. In Advances in Neural Information Processing Systems, volume 37, 2024b. URLhttps://papers.neurips.c/paper_files/paper/2024/hash/ ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html. 13 Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Senel, Anna Korhonen, and Hinrich Schuetze. TurkishMMLU: Measuring massive multitask language understanding in Turkish. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 7035–7055, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.413. URL https://aclanthology.org/2024.findings-emnlp.413/. 14 A Prompts and implementation details The evaluation prefix is: Sana soru ve seçenekleri veriyorum. Sadece hangi seçeneğin sorunun doğru cevabı olduğunu yaz. A–J arasındaki tek bir harfle cevap ver. Lütfen herhangi bir açıklama yapma! It is followed bySoru:(“Question:), the stem, and options labelledA:throughJ:, one per line. In English, the instruction asks for the correct option as a single letter between A and J and requests no explanation. The five-choice condition replaces A–J with A–E. The executable prompt uses an en dash in the range; the full Unicode string is preserved in the run records. The selector’s complete Turkish instruction and dynamic candidate-ID schema are recorded in research/prompts.jsonandprepare.py; an English translation is supplied inresearch/prompt_ translations.json. It instructs the model to treat the embedded question JSON as data, trust the original key, avoid alternative answers and semantic duplicates, handle negative stems, select five IDs even when candidates are weak, and return only the requested JSON. The saved selector model digest is f0fc7e0a...64879. The historical code snapshot is retained separately from the current analysis scripts. B Negative-stem rule The rule normalizes Unicode and lowercases with Turkish dotted and dotless i handled explicitly. It takes the last sentence containing a question mark or hangi (“which”), falling back to the last sentence. It flags matches for forms beginning with değil, yanlış, dışında, or olmayan; the words yok, yoktur, olamaz, söylenemez, and hariç; or negative-aorist endings -maz/-mez with the implemented plural and copular suffixes. It excludes exact matches for namaz, yılmaz, and taşınmaz. The executable expression inpaper/pilot_results.pyis authoritative. Attributive negation and ambiguous suffixes can create false positives, and other negative constructions can be missed. The audit comparison in Appendix D reports agreement with the recorded polarity fields; their provenance does not support treating them as an independent gold standard. The lexical meanings include “not” (değil, olmayan), “wrong” (yanlış), “outside/except” (dışında, hariç), “absent” (yok, yoktur), “cannot be” (olamaz) and “cannot be said” (söylenemez). The excluded forms mean “ritual prayer” (namaz), “undaunted” (yılmaz, also a name) and “immovable/property” (taşınmaz); suffix-shaped spelling alone does not determine polarity. C Diagnostic definitions paper/review_diagnostics.py verifies the inputs pinned by the evaluation snapshot and rescores the full responses before computing Table 6. The gold-position slope is ordinary least squares on per-item correctness (scaled to percent), with an intercept and the zero-based gold index as predictor. Response entropy is − P j p j log p j / log10, using valid-response frequencies. Error primacy is 100[E A − P i∈E 1(g i ̸=A)/9]/|E|, whereEcontains valid errors,E A counts selections of A, andg i is the source key. Length is the number of Unicode code points after trimming display whitespace. Length ranks average tied ranks and exclude the gold when analyzing errors. None of these post hoc metrics is a test of semantic validity or a within-item position intervention. D Audit sampling, corrections and interpretation The original packet uses seed-42 SHA-256 ordering to allocate 3–4 items per section, split across the heuristic- flag strata, and samples without replacement. All 116 nonempty strata are represented. For itemiin stratum h,w i =N h /n h ; the 200 weights sum to 12,000. Table 7 reports 100 P i w i 1(y i =c)/ P i w i . Unresolved labels are a separate category in the same denominator. The 12 disjoint calibration items do not enter any reported 15 Flag / recorded polarity R1n R1 weight R2n R2 weight Flagged / negative902833.0912864.5 Flagged / nonnegative11285.010253.5 Unflagged / negative2149.02149.0 Unflagged / nonnegative978733.0978733.0 Table 8: Keyword flags against recorded audit polarity. Counts and inverse-inclusion-weight sums are reported separately for R1 and R2. The 200 original polarity labels in each record set are unchanged by the correction log. These are recorded-label comparisons, not validated expert-gold confusion matrices. statistic. No sampling-based interval is claimed: sparse strata, annotation uncertainty and cross-reviewer dependence are not captured by a simple binomial interval. The supplied instructions ask for all defensible A–J answers, stem polarity, clarity, missing context, subject familiarity and references before revealing keys/origins. The second pass asks for source-key support, multiple/no valid answers, semantic duplicates and section fit, plus each borrowed option’s validity, relevance, duplicates and incomplete references. These are separate dimensions: an incorrect but unrelated option is not evidence of a plausible distractor. The original instructions are supplied with the reviewer materials. The planned concealment in the HTML is not evidence that the actual first pass was blind; Q159’s correction explicitly acknowledges deference to a source key. The original six CSVs contain 200 first-pass and 200 second-pass question rows plus 1,000 option rows for each of R1 and R2. A separate correction log has 1,054 entries: 33 R2 source-key category corrections, 173 option resolutions, two R1 Q159 entries, 800 explicit missing-field entries, and 46 final R2 answer sets. Of the 173 option resolutions, 72 retain the earlier answer interpretation, 14 retain the distractor interpretation, and 87 retract both and classify the option as defective. The latter two groups retract 101 answer selections across 46 questions. Twenty-one retain an explicitly recorded answer set after subtraction; 25 are unresolved, not verified no-answer items. The subtraction-based sets are not described as fresh expert re-adjudication. The analysis preserves received records and derives answer-set status separately. A singleton equal to the source key is unique-source-key; another singleton is single-alternative; a set with more than one answer is multiple; an unresolved set yields cannot-verify. Derived status updates affect 42 R2 answer-status and 25 key-support fields. A multiple set may still exclude the source key, so multiple-answer status and key support are distinct. A source-key-plus-borrowed judgment requires both the source key and at least one borrowed position in a resolved set. Missing data never become negative findings. R2’s unresolved items carry 963/12,000 total sampling weight (8.0%); its observed weighted co-answer fraction is 6.3%, with an unknown-allocation range of 6.3–14.3% if every unresolved item could qualify. That range describes missingness alone, not statistical uncertainty or ground-truth prevalence. The corrections cite R1 when reconciling R2, so post-correction agreement would be circular evidence of independent reliability. Neither Cohen’s kappa nor expert-gold precision/recall is claimed. R2’s original second-pass fields are script-generated rather than judged: re-running the supplied generation script over the returned first-pass answers and the revealed-key page reproduces both R2 second-pass files byte for byte. Its 1,000 option rows are constant, marking every borrowed option a valid, plausible, non-duplicate distractor. These entries carry no information and are never promoted to verified plausibility ratings; no reported quantity depends on them. R1’s second-pass option fields vary across all four judgment columns and retain the shipped template’s byte signature. The main results therefore emphasize answer-set records and unresolved coverage rather than a pooled option-quality percentage. The artifact itself is unchanged; revised keys/options would require a new version and new evaluations. E Detailed pilot results Among models with parseable answers, direct-mode accuracy ranges from 8% to 65%. Nine models score between 57% and 65%, with broad intervals at this sample size; their observed ordering is descriptive. The 16 guardrail classifier supplies no valid answers.minicpm-v4.6:1banswers 28 of 100 questions in a parseable form, with 8% overall accuracy and 28.6% accuracy on those 28 answers. Answered-only accuracy conditions on a model-selected subset and cannot replace the overall score. Generation counts vary substantially within direct mode. Themuse-glimmer:30b-mlxrun averages 266 generated tokens per question despite returning short visible answers and no recorded reasoning trace. This indicates substantial generation not represented by the visible answer; its semantic content cannot be inferred from the count alone.lfm2.5:8bhas observed reasoning in 84 responses, with 16 reaching the 2,048-token ceiling. A request for a bare letter therefore does not equalize computation. For the 22 ranked models without observed reasoning, legacy accuracy exceeds strict accuracy by 2–13 percentage points (mean 5.6). Inspection of the legacy code explains why this is not simply recovered answer coverage: a wrong bare letter can reach an embedding fallback that selects another option, including the gold. Its apparent success therefore depends partly on the gold-aware control flow. This diagnostic must not be interpreted as a valid alternative estimate of knowledge. Thinking-mode accuracy is higher for seven of ten completed pairs, by 3–11 points, and lower for three. Only gemma4:e4b-mlxreaches unadjustedp <0.05 (+11 points;p= 0.035); none meets the Bonferroni threshold. Generation time rises by factors of 2–627. Forqwen3.5:0.8b-mlx,qwen3.5:2b-mlx, andgranite4.2:3b, 48–80 thinking responses reach the ceiling without an answer. The project’s dated analysis contract records selection of direct mode for full evaluation on 10 September while the first full run was in progress. This is an internal decision record, not an independently registered protocol. All models use that mode; the protocol does not select each model’s better pilot condition. Tables 9 and 10 report the complete pilot and its ten completed mode pairs. Displayed ranks merely order observed scores. Token counts include generation reported by the runtime, and timing is elapsed request time per question. The-mlxsuffix is omitted from local MLX model names to fit the tables; exact tags, digests, and quantizations are retained in the machine-readable results. Selector weights are marked with a dagger. F Full-benchmark accuracy by section Table 11 reports the five complete runs by source section. These descriptive scores retain the source keys and do not constitute independently validated subject-level measurements. Appendix H supplies English glosses of the inherited labels. G Reproducibility and sensitivity checks For context, a stylized model that knows a fixed fraction of answers and otherwise guesses uniformly would convert the observed five-choice score to 92.89% with ten choices, a decrease of 0.79 points. The observed decrease is much larger than this reference prediction. However, partial knowledge and nonuniform elimination violate that model, so it does not identify how much loss is caused by guessing, ambiguity, or presentation changes. Likewise, the unpaired full-benchmark scores of 93.35% and 81.38% cannot isolate augmentation because their item compositions differ. The local runs accumulate 1.13–11.56 hours of request time, including local inference and request overhead. The API run records 2.20 million input and 0.33 million output tokens. Using the rates configured in its saved run, the estimated token charge is US$19.33; this is not an audited invoice. The recorded interval from first to last answer is 0.81 hours, including batch processing and queueing, and is not directly comparable with accumulated local request time. From the project root,python paper/summarize.pyverifies the pinned source and selected-set reconstruction without model calls.python paper/pilot_results.pychecks exact response coverage, recomputes scorer-v2 indices and correctness, verifies summary totals and pilot pair counts, reconstructs shared original options, and generates the tables and numerical snapshot. It rejects duplicate response IDs and incompatible full-run conditions. Input hashes include the benchmark, provenance, calibration file, reported-run settings and 17 # ModelParams Quant.Acc. 95% CI Ans. Legacy Tokens Reas. s/q 1 muse-glimmer:30b32.3 nvfp465 [55, 74] 100672660 13.65 2 gemma4:31b31.7 nvfp463 [53, 72] 1006610 1.64 3 qwen3.8:27b27.8 nvfp463 [53, 72] 1006610 1.53 4 gemma4:26b † 26.2 nvfp460 [50, 69] 996310 0.39 5 qwen3.6:27b27.8 nvfp460 [50, 69] 1006510 1.48 6 qwen3.6:35b36.0 nvfp459 [49, 68] 1006410 0.53 7 qwen3.5:27b27.4 nvfp458 [48, 67] 1006210 1.51 8 qwen3.5:35b35.1 nvfp457 [47, 66] 1006110 2.17 9 gemma4:12b12.4 nvfp455 [45, 64] 1005710 0.73 10 ornith-1.5:35b35.5 Q4_K_M52 [42, 62] 965560 0.54 11 qwen3.5:9b9.4 nvfp448 [38, 58] 1005210 0.49 12 ornith-1.5:9b9.0 Q4_K_M45 [36, 55] 1004920 0.56 13 gemma4:e4b8.1 nvfp442 [33, 52] 1004610 0.25 14 granite4.2:30b29.3 Q4_K_M42 [33, 52] 1004920 1.49 15 ornith:9b9.0 Q4_K_M41 [32, 51] 1004920 0.62 16 qwen3.5:4b4.5 nvfp440 [31, 50] 1004910 0.29 17 granite4.2:8b8.8 Q4_K_M31 [23, 41] 1003920 0.51 18 nemotron-3.5-lightning:30b32.9 nvfp431 [23, 41] 994010 0.81 19 gemma4:e2b5.2 nvfp430 [22, 40] 1003710 0.15 20 lfm2.5:8b8.5 Q4_K_M19 [13, 28] 71– 1151 84 7.94 21 qwen3.5:2b2.2 mxfp816 [10, 24] 1002520 0.16 22 granite4.2:3b3.7 Q4_K_M13 [8, 21] 94–412 0.66 23 qwen3.5:0.8b0.85 mxfp813 [8, 21] 1002010 0.09 24 minicpm-v4.6:1b0.75 Q4_K_M8 [4, 15] 28211860 0.98 – granite4.1-guardian:8b8.4 Q6_Kunranked –0–90 0.98 Table 9: Direct-mode pilot, 100 questions per model. Params: runtime-reported billions of parameters. Acc.: source-key accuracy (%) with Wilson interval. Ans.: valid-answer rate (%). Legacy: diagnostic accuracy, withheld for models with observed reasoning or no valid answers. Tokens: mean generated tokens. Reas.: observed reasoning count. s/q: seconds per question. ModelDirect Thinking∆ 95% CIp D/T Answer (%) Trunc. Time gemma4:e4b4253 +11 [1.9, 20.1] 0.035 6/171000 42× granite4.2:3b1322 +9 [0.2, 17.8] 0.078 6/155248 108× ornith-1.5:35b5259 +7 [-1.0, 15.0] 0.143 5/128910 59× qwen3.6:35b5966 +7 [0.1, 13.9] 0.092 3/10964 85× qwen3.5:4b4046 +6 [-3.9, 15.9] 0.327 10/166634 329× gemma4:26b † 6064 +4 [-3.8, 11.8] 0.455 6/108515 126× lfm2.5:8b1922 +3 [-4.6, 10.6] 0.607 6/9835 2× gemma4:e2b3029 -1 [-8.1, 6.1] 1.000 7/6760 16× qwen3.5:2b1615 -1 [-8.6, 6.6] 1.000 8/72080 627× qwen3.5:0.8b1310 -3 [-11.5, 5.5] 0.648 11/82773 499× Table 10: Paired thinking versus direct pilot results. ∆ is thinking minus direct in percentage points, with a paired Wald interval. Exact McNemarp-values are unadjusted. D/T counts direct-only/thinking-only correct answers. Answer and truncation refer to thinking mode. Time is the ratio of mean request durations. The reasoning request and token ceiling both differ between conditions. response files, and analysis code. Paper prose must be reconciled with this snapshot after any new run; table regeneration does not update prose automatically. Excluding the 100 calibration items gives accuracies of 63.65%, 57.49%, 44.43%, 34.80%, 81.39%, in the model order of Table 2, over 11,900 questions. Each differs from its full-corpus score by less than 0.05 percentage points. This post hoc check removes direct item overlap but not protocol-selection effects. Among the 981 paired questions, 973 change the relative order of the original options and 892 change the displayed gold label. A post hoc sensitivity analysis resamples whole sections (55 represented sections; 10,000 replicates; NumPy default random generator, seed 42). Its percentile 95% interval for the paired change 18 is [−12.89,−8.36] percentage points. This permits within-section dependence but assumes exchangeable, independent sections; it is a sensitivity analysis, not a claim of population-representative sampling. The abbreviated SHA-256 hashes are6cb7c31e...e83abfor selected data,97d73c...cff57for provenance, andf203ab8e...6686for calibration data. Full strings and source revisions are recorded inresearch/ artifact_snapshot.jsonandresearch/pilot_snapshot.json. The latter identifies all five full runs and the five-choice comparison run.paper/check_audit_corrections.pyverifies each correction against its recorded original value.paper/audit_results.pyproduces the audit tables and a text-free numeric supplement that reproduces the weighted summaries without exposing question text. Audit inputs and analysis hashes are recorded separately from the unchanged model-response snapshot. The source audit, bibliographic notes, and human-audit rubric are retained underresearch/. These files document the analysis; their existence locally does not imply public release of raw responses. H English glosses of source section labels Table 12 translates the inherited section names for readers of the English manuscript. The original labels remain authoritative identifiers in the frozen dataset and section-level results. 19 Sectiongemma4:31b gemma4:26b gemma4:e4b qwen3.5:4b gpt-6-astra Mean Ehliyet82.675.858.955.692.8 73.1 Dini Bilgiler76.371.053.140.689.9 66.2 Kim 500 Milyar İster79.272.948.330.996.1 65.5 Turizm ve Otel İşletmeciliği71.467.051.945.187.9 64.7 TUS78.372.942.035.389.4 63.6 İşletme Yönetimi67.663.352.739.687.4 62.1 Radyo ve Televizyon Programcılığı71.565.751.237.784.1 62.0 Sosyal Hizmet69.659.952.240.683.1 61.1 Tıbbi Dökümantasyon ve Sekreterlik70.563.353.631.983.1 60.5 İktisat67.163.348.339.184.5 60.5 Siyer72.966.236.734.389.9 60.0 Medya ve İletişim67.664.349.337.280.7 59.8 Sağlık Yönetimi67.160.448.341.180.2 59.4 Sosyal Hizmetler68.658.948.839.181.6 59.4 Sosyoloji66.261.451.236.781.6 59.4 Sağlık Kurumları İşletmeciliği65.262.348.341.579.2 59.3 Elektrik Enerjisi Üretim,İletim ve Dağıtımı69.159.947.836.782.6 59.2 Menkul Kıymetler ve Sermaye Piyasası69.660.450.235.780.2 59.2 Kamu Yönetimi64.760.947.342.579.7 59.0 Bankacılık ve Sigortacılık63.863.348.835.383.6 58.9 Yerel Yönetimler67.657.042.042.086.0 58.9 Kültürel Miras ve Turizm67.658.045.937.284.5 58.6 İlahiyat64.359.949.338.280.2 58.4 Büro Yönetimi ve Yönetici Asistanlığı64.758.549.339.179.7 58.3 Tarım65.757.543.537.286.0 58.0 Turizm ve Seyehat Hizmetleri63.356.544.439.186.5 58.0 Özel Koruma ve Güvenlik63.859.944.037.282.6 57.5 Çağrı Merkezi Hizmetleri60.461.446.933.383.1 57.0 Parakende Satış ve Mağaza Yöneticiliği61.456.548.340.176.3 56.5 Yönetim Bİlişim Sistemleri65.756.548.334.876.8 56.4 Uluslar Arası İlişkiler58.955.649.835.380.2 55.9 Yaşlı Bakımı56.356.849.037.977.7 55.5 Marka İletişimi60.952.748.338.675.8 55.3 Laborant ve Veteriner Sağlık64.355.643.534.377.8 55.1 İnsan Kaynakları Yönetimi63.857.039.635.778.7 55.0 Çalışma Ekonomisi ve Endüstri İlişkileri64.352.239.133.884.1 54.7 Halkla İlişkiler ve Reklamcılık62.156.845.628.679.6 54.6 Emlak ve Emlak Yönetimi60.456.542.034.878.7 54.5 Felsefe59.454.144.435.379.2 54.5 Dış Ticaret63.151.944.733.577.7 54.2 AUZEF62.858.540.634.373.4 53.9 Spor Yönetimi60.947.842.536.781.6 53.9 Çocuk Gelişimi57.554.144.935.777.3 53.9 Lojistik58.050.243.536.280.7 53.7 Fotoğrafçılık ve Kameramanlık58.954.142.030.976.8 52.6 Aşçılık62.854.139.626.678.7 52.4 Halkla İlişkiler ve Tanıtım58.553.642.531.975.4 52.4 Tarih58.553.634.332.982.6 52.4 Adalet58.752.436.433.580.6 52.3 Ev İdaresi55.152.243.532.973.9 51.5 Muhasebe ve Vergi Uygulamaları57.553.138.628.579.2 51.4 YGS Denemeleri58.548.337.725.684.5 50.9 Uluslararası Ticaret ve Lojistik Yönetimi55.151.243.028.070.0 49.5 DHBT57.052.731.920.880.7 48.6 Futbol55.648.830.422.784.1 48.3 Türk Dili ve Edebiyatı51.744.431.924.678.3 46.2 KPSS Denemeleri51.243.030.919.876.3 44.3 KPSS46.139.324.816.577.2 40.8 Macro average63.657.544.434.881.4 56.4 Table 11: Source-key accuracy (%) by section, sorted by the descriptive mean over the five runs. Each section has 206 or 207 questions. Labels omit-mlx;gemma4:26bis the selector model. Section names follow the source collection. The mean combines different backends and is used only to order rows. 20 Source labelEnglish gloss EhliyetDriving licence Dini BilgilerReligious knowledge Kim 500 Milyar İsterWho Wants 500 Billion (quiz title) Turizm ve Otel İşletmeciliğiTourism and hotel management TUSMedical specialty examination İşletme YönetimiBusiness management Radyo ve Televizyon ProgramcılığıRadio and television programming Sosyal HizmetSocial work Tıbbi Dökümantasyon ve SekreterlikMedical documentation and secretarial studies İktisatEconomics SiyerBiography of the Prophet Muhammad Medya ve İletişimMedia and communication Sağlık YönetimiHealth management Sosyal HizmetlerSocial services SosyolojiSociology Sağlık Kurumları İşletmeciliğiHealthcare institution management Elektrik Enerjisi Üretim,İletim ve Dağıtımı Electric power generation, transmission and distribution Menkul Kıymetler ve Sermaye PiyasasıSecurities and capital markets Kamu YönetimiPublic administration Bankacılık ve SigortacılıkBanking and insurance Yerel YönetimlerLocal government Kültürel Miras ve TurizmCultural heritage and tourism İlahiyatTheology Büro Yönetimi ve Yönetici AsistanlığıOffice management and executive assistance TarımAgriculture Turizm ve Seyehat HizmetleriTourism and travel services Özel Koruma ve GüvenlikPrivate protection and security Çağrı Merkezi HizmetleriCall centre services Parakende Satış ve Mağaza YöneticiliğiRetail sales and store management Yönetim Bİlişim SistemleriManagement information systems Uluslar Arası İlişkilerInternational relations Yaşlı BakımıElderly care Marka İletişimiBrand communication Laborant ve Veteriner SağlıkLaboratory technology and veterinary health İnsan Kaynakları YönetimiHuman resource management Çalışma Ekonomisi ve Endüstri İlişkileriLabour economics and industrial relations Halkla İlişkiler ve ReklamcılıkPublic relations and advertising Emlak ve Emlak YönetimiReal estate and property management FelsefePhilosophy Dış TicaretForeign trade AUZEFOpen and distance education faculty Spor YönetimiSports management Çocuk GelişimiChild development LojistikLogistics Fotoğrafçılık ve KameramanlıkPhotography and camera operation AşçılıkCulinary arts Halkla İlişkiler ve TanıtımPublic relations and promotion TarihHistory AdaletJustice Ev İdaresiHousehold management Muhasebe ve Vergi UygulamalarıAccounting and tax practice YGS DenemeleriHigher education entrance practice tests Uluslararası Ticaret ve Lojistik Yönetimi International trade and logistics management DHBTReligious services field examination FutbolFootball Türk Dili ve EdebiyatıTurkish language and literature KPSS DenemeleriPublic personnel selection practice tests KPSSPublic personnel selection examination Table 12: English glosses of the 58 source section labels, in the order of Table 11. Original spellings are retained for traceability; glosses do not merge sections or certify item provenance. Acronyms and quiz titles denote inherited collection labels, not verified examination ownership. 21