Paper deep dive
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks
Volodymyr Ovcharov
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/22/2026, 2:15:55 AM
Summary
This paper investigates the validity of multiple-choice legal benchmarks by testing whether models can answer questions correctly using only the options, without the question stem (the 'blind' condition). Using the UA-JudgeExam dataset (11,990 items), the authors demonstrate that a significant portion of items can be solved by models like Claude Haiku 4.5 purely by recognizing plausible legal propositions in the options. They propose a 'gating' method to filter out these leaky items. However, the study finds that this filtering does not transfer to other models; GPT-5.6 still achieves high accuracy (0.515) on the filtered set. The paper concludes that item format (self-contained options vs. pointers) determines solvability, and that filtering against one model does not ensure benchmark integrity for others.
Entities (9)
Relation Signals (6)
UA-JudgeExam → hassize → 11,990
confidence 98% · UA-JudgeExam: 11,990 four-option items with official keys
Claude Haiku 4.5 → scoreson → UA-JudgeExam
confidence 97% · Shown the options and no question, Claude Haiku 4.5 scores 0.383 against chance
GPT-5.6 → scoreson → UA-JudgeExam
confidence 96% · GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden.
LEXAM → exhibits → Blind Condition
confidence 95% · The same probe on LEXam returns chance
Gating → failstotransferto → GPT-5.6
confidence 94% · A bank cleaned against one model is not clean for a better one... GPT-5.6 still answers 0.515 of those same items
Gating → reduces → UA-JudgeExam
confidence 91% · Gating those out retains 8,128 items
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care: a model answering A to most items scores above chance wherever the key sits at A, and reads as recognition when it is not. We measure it on UA-JudgeExam: 11,990 four-option items with official keys, published by Ukraine's Higher Qualification Commission of Judges. Shown the options and no question, Claude Haiku 4.5 scores 0.383 against chance, and the leak is concentrated: 11.8% of items are answered blind on all eight option orders, against 0.2 items expected by chance. It is not quotation: search over 280,059 editions of Ukrainian legislation recovers 0.128. Gating those out retains 8,128 items, on which the gating model itself now scores 0.204, and GPT-5.6, which took no part in the selection, still answers 0.515 of them with the question hidden. Scoring twelve held-out models on the whole set and subtracting each one's answer-position habit, only two keep an excess: GPT-5.6 at +0.265, Sonnet 4.6 at +0.081. Without it the ranking misleads: Llama 3.1 8B scores 0.292 blind, above every model but those two, purely by answering A to 92% of items. The gate does select something real: on the items it rejected, eleven of twelve models score 0.518-0.789, every interval clear of what the same model scores on the items it kept. But that signal is one model's, and filtering on it does not transfer upward. Neither is visible on a 400-item sample, where nine models read as "statistically at chance". Rewriting distractors instead overshoots to 0.168, below chance and as exploitable. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters. Item format decides whether the problem can arise; capability decides how much is extracted. We release the corpus, the predictions and the harness.
Tags
Links
- Source: https://arxiv.org/abs/2608.15428v1
- Canonical: https://arxiv.org/abs/2608.15428v1
Trouble viewing inline? Open PDF directly →
Full Text
63,689 characters extracted from source content.
Expand or collapse full text
Gated Against One Model, Open to the Next: Option-Only Solvability in Legal Multiple-Choice Benchmarks Volodymyr Ovcharov Affiliation: LEX AI Email: vladimir@legal.org.ua Affiliation: https://legal.org.ua Abstract Multiple-choice benchmarks are graded on whether a model picks the right option, not on whether it needed the question. Measuring that gap takes care: a model answering A to most items scores above chance wherever the key sits at A, and reads as recognition when it is not. We measure it on UA-JudgeExam: 11,990 four-option items with official keys, published by Ukraine’s Higher Qualification Commission of Judges. Shown the options and no question, Claude Haiku 4.5 scores 0.3830.383 against chance, and the leak is concentrated: 11.8%11.8\% of items are answered blind on all eight option orders, against 0.20.2 items expected by chance. It is not quotation: search over 280,059280,059 editions of Ukrainian legislation recovers 0.1280.128. Gating those out retains 8,1288,128 items, on which the gating model itself now scores 0.2040.204 — and GPT-5.6, which took no part in the selection, still answers 0.5150.515 of them with the question hidden. Scoring twelve held-out models on the whole set and subtracting each one’s answer-position habit, only two keep an excess: GPT-5.6 at +0.265+0.265, Sonnet 4.6 at +0.081+0.081. Without it the ranking misleads: Llama 3.1 8B scores 0.2920.292 blind, above every model but those two, purely by answering A to 92%92\% of items. The gate does select something real: on the items it rejected, eleven of twelve models score 0.5180.518–0.7890.789, every interval clear of what the same model scores on the items it kept. But that signal is one model’s, and filtering on it does not transfer upward. Neither is visible on a 400-item sample, where nine models read as “statistically at chance”. Rewriting distractors instead overshoots to 0.1680.168, below chance and as exploitable. The same probe on LEXam returns chance: every option there points into the stem, none longer than 33 characters. Item format decides whether the problem can arise; capability decides how much is extracted. We release the corpus, the predictions and the harness. Keywords: benchmark validity, multiple-choice evaluation, shortcut learning, legal NLP, Ukrainian, contamination 1 Introduction A multiple-choice benchmark reports one number: how often the model picks the key. That number is treated as evidence about the model’s competence in the domain. It is only such evidence to the extent that the question is load-bearing — that the model could not have found the key from the options alone. The obvious repair, when a bank fails that test, is to filter it: drop the items a model can answer without the question and keep the rest. This paper’s central finding is that the repair does not hold. We filtered a state-published judicial examination bank until the filtering model scored 0.2040.204 blind on what survived, below chance; GPT-5.6, which took no part in the selection, still answers 0.5150.515 of those same items with the question hidden. A bank cleaned against one model is not clean for a better one, and we found no version of the procedure that made it so. This is not a hypothetical concern wherever the options are self-contained propositions, as they are in professional licensing material. Distractors are written by humans under time pressure, and a wrong option is wrong for a reason: it names the wrong authority, states the wrong deadline, or asserts something no statute says. A reader who knows the domain may then be able to recognise the one option that reads like real law without ever seeing the question — and, as we show, so can a model, though we do not test human readers. To the extent that happens, the benchmark measures recognition of well-formed legal propositions rather than the reasoning it claims to measure. Where the options are instead pointers into the stem — “i and i” — the concern cannot arise, and we show that this distinction, rather than subject matter, is what separates the benchmarks we test. We make that quantity the object of study rather than a footnote. Our contributions: 1. Filtering does not transfer. Dropping the items one model can guess retains 67.8%67.8\% of the bank and drives the gating model itself to 0.2040.204 on what survives — yet GPT-5.6 still answers 0.5150.515 of the cleaned set with the question hidden. Rewriting distractors instead overshoots below chance. Neither repair produces a neutral set (§6, §8). 2. A measurement protocol, and an estimator that separates habit from content. The blind condition is run over option-order permutations with the selecting model held separate from the reporting one. Forcing the key into each slot in turn then makes the four accuracies sum to one whenever choice is independent of content, so their mean is chance by construction and any excess is content. Measured that way, ten of twelve held-out models extract nothing at all, and neither prompt phrasing, option labelling nor inference-time reasoning accounts for what the other two extract (§5, §6.1–§6.3). 3. A benchmark that does not leak, and why. The same probe on LEXam returns chance; every one of its items uses reference-style options, none longer than 33 characters. Option format, not subject matter, decides whether the problem can exist (§7). 4. A resource. UA-JudgeExam: 11,990 items with official state-issued keys, extraction verified by an independent second path, plus the gated 8,1288,128-item subset and every prediction behind the tables (§3). 2 Related Work Answering without the question. The probe we use is not new. Balepur et al. 2024 prompt models with the choices alone on three MCQA datasets and beat a majority baseline in 11 of 12 cases, and find no evidence that this stems from memorisation, nor that priors over individual choices fully explain it. Cho et al. 2026 attack the same concern from the scoring side, proposing a metric that isolates how much the question contributes to a model’s decision. Both establish that the phenomenon exists in general-domain multiple choice. We take it as given and ask the question that follows for a practitioner holding a benchmark and reaching for the obvious fix: how large is the effect on professional legal exam material, does filtering remove it, does a filter built with one model protect against another, and what property of an item set decides whether the problem arises at all. Legal benchmarks. Legal NLP benchmarks are predominantly English (Guha et al. 2023; Chalkidis et al. 2022; Hendrycks et al. 2021), and LEXTREME (Niklaus et al. 2023) broadens coverage to 24 languages, though its tasks are classification and token-level extraction rather than multiple choice, so the blind condition does not apply to it. The Östling et al. 2023 corpus is the closest analogue to our resource on the legal and ethical side — a national court corpus released under restrictions worked out with institutional review, where we can release in full only because the bank is a state document outside copyright. Exam-derived benchmarks inherit the item-writing conventions of the professions that produced them, including the distractor-writing conventions this paper is about: LEXam (Fan et al. 2026) builds 340 law exams into a reasoning benchmark, and Katz et al. 2024 evaluate against a professional licensing examination directly. We measure the blind condition on LEXam’s four-choice split in §7 and find no leakage at all, for a reason that turns out to be about how its options are written rather than about its subject matter. Position and selection bias. A separate line of work shows that models are sensitive to where an option sits. Zheng et al. 2024 document a systematic preference for particular option IDs across 20 models and trace it to token bias — extra probability mass on the tokens A/B/C/D themselves — and debias by permuting option contents; Pezeshkpour and Hruschka 2024 show the same sensitivity to option order. That literature and this paper measure different things which are easy to confuse in a single-order experiment, since a model that merely favours one slot will also score above chance whenever the key happens to sit there. §6.2 separates them by forcing the key into each slot in turn, and finds the two orthogonal: the model with the strongest slot preference in our set is not the one that leaks most. Our labelling sweep also bears on the mechanism, since relabelling the options in Cyrillic or with digits leaves the aversion to the first slot essentially unchanged. Shortcuts and artifacts. Work on annotation artifacts in natural language inference established that models exploit hypothesis-only signal (Gururangan et al. 2018; Poliak et al. 2018); the blind condition is the multiple-choice analogue. The closest study in law is Watson et al. 2026, who ask whether legal judgment prediction forecasts anything or merely reads outcome-revealing language left in the judgment, on 33,158 UK Employment Tribunal claims. Their answer and ours diverge in a way worth stating. They remove the leaking features and the task survives: Macro-F1 falls negligibly, so a real signal was there underneath. We remove the leaking items and the benchmark does not survive in the same sense — the surviving set is clean for the model that selected it and not for a stronger one. Deleting a cue from every instance and deleting the instances that carry it are not the same operation, and only the first leaves the measurement intact. 3 UA-JudgeExam 3.1 Source Ukraine’s Higher Qualification Commission of Judges (ВКС) publishes the complete question bank used for the anonymous written testing of candidates for appellate-court judgeships, together with the key for every item. We use the bank published by Commission decision of 15 July 2024, No. 221/зп-24: five documents totalling 1,672 pages, covering general legal knowledge and the administrative, commercial, criminal, and civil specialisations. Each item has exactly four options, of which the document marks exactly one правильна (correct) and three неправильна (incorrect). The bank is published as an annex to a decision (ршеня) of the Commission. Article 8(1)(3) of the Law of Ukraine On Copyright and Related Rights (No. 2811-IX) places outside copyright protection “acts of state authorities … official documents of a political, legislative, administrative and judicial character (laws, decrees, resolutions, decisions, state standards and the like)”. On that basis we redistribute the bank with attribution to the issuing decision. The items are questions on points of law and contain no personal data. 3.2 What an item looks like Table 1: Item vkks-2024-commercial-2237, with an English gloss; the key, C, is in bold. The options are self-contained legal propositions, which is the property that makes the blind condition meaningful. Article 31 of the Civil Code grants a person under 14 the right самостйно вчиняти дрбн побутов правочини — so D substitutes рзн (“various”) for дрбн (“minor”), which is not legal language, and A substitutes майнов (“property”) for the особист немайнов (“personal non-property”) rights the article actually grants. Telling these apart requires knowing the code, not reading the question. Q. Як д ма право самостйно вчиняти фзична особа, яка НЕ досягла 14 рокв? Which acts may a natural person under 14 perform independently? A) Здйснювати майнов права на результати нтелектуально дяльност, що охороняються законом Exercise property rights in protected intellectual-property results B) Розпоряджатися банквським вкладом, унесеним нею на сво м’я Dispose of a bank deposit made in their own name C) Вчиняти дрбн побутов правочини Perform minor everyday transactions D) Вчиняти рзн побутов правочини Perform various everyday transactions This is the format that leaks. Contrast it with LEXam (§7), whose four-choice items offer options like “i und i” — pointers into a list of statements given in the stem, which carry no legal content of their own. 3.3 Extraction and its verification The documents are ruled tables in PDF. We extract them with a table parser and verify the result with a second, independent path that reconstructs rows from the table’s horizontal ruling lines, columns from its vertical ones, and cell contents from word coordinates — sharing no code with the first. Comparison is insensitive to whitespace and line-break hyphenation, because the two paths join wrapped words differently. Extraction yields 11,990 well-formed items, plus 385 items the Commission itself marks as withdrawn (Виключено ршеням) and 40 that fail the four-options-one-key invariant. On a 200-item stratified pilot the second path confirms the question text, the four options in order, and the key in 0.9950.995 of items. Over the full bank it confirms the question text in 0.98960.9896, option order in 0.98880.9888, and the answer key in 0.98410.9841; the 192 items (1.6%1.6\%) that fail at least one check are listed by identifier in the release. Five items are malformed in the source itself and we ship them as published rather than repairing them silently: in one the fourth option is blank in the PDF, and four carry two identical options. None of them falls in the 400-item sample used for the probes below, which contains no empty options. One extraction detail is worth reporting because it is a silent failure mode: in 942 items the cell holding the item number does not extract, which naively merges two adjacent items into one eight-option record. Detecting the merge by the appearance of a new question stem while options are already accumulating recovers those items; the recovered numbering contains zero duplicates within any of the five documents, which is the namespace the Commission numbers in and the check that the recovery is sound. 4 The Solvability Profile Before asking what models score, we establish what trivial strategies score. Unless noted, all numbers are over the full bank of 11,990 items, on the options as published. Table 2: What can be recovered without reasoning over the question. Chance is 0.2500.250. Strategy Accuracy Chance 0.250 Best constant position (always A) 0.289 Longest option 0.278 Most words 0.301 Shortest option 0.253 Lexical search over statutes, full bank 0.128 Lexical search over statutes, items with four substantive options (n=8,024n=8,024) 0.172 Blind LLM, options only (Haiku 4.5, 8 permutations) 0.383 Surface features carry almost nothing: the correct option is on average 2.62.6 characters longer than a distractor, and no length or position heuristic clears 0.3010.301. The key sits at A in 28.9%28.9\% of items, at B and at C in 24.8%24.8\% each and at D in 21.5%21.5\%, so a positional habit buys a little; we control for it explicitly in §6. 4.1 Lexical search is not the mechanism A plausible story is that the correct option is copied from a statute while distractors are invented, so a text search decides the item. We tested this against the full text of 280,059 current editions of Ukrainian legislation (2.3562.356 billion characters). For each option we ask whether its first twelve tokens occur verbatim anywhere in that corpus. Over the full bank the correct option is found verbatim in 0.5800.580 of items and distractors in 0.4380.438 — a real but weak signal. The decisive statistic is what a searcher can do with it: exactly one option is found in 19.6%19.6\% of items, and there the match is the key 65.1%65.1\% of the time, for an overall accuracy of 0.1280.128. One- and two-token options — bare figures such as “1095” — occur verbatim 94.6%94.6\% of the time (10,63710,637 of 47,96047,960 options). They inflate the raw match rates and, because they match for every option at once, depress what a searcher can do with them, so we also restrict to the 8,0248,024 items whose four options are all substantive (three tokens or more). There the searcher commits on 25.9%25.9\% of items and is right 66.4%66.4\% of the time when it does, for 0.1720.172 overall — still below chance. We ran this two ways: as phrase queries against a full-text index, and as a single pass over an exported corpus with an inverted index of the option phrases. The two implementations, on different machines with no shared code, agree on 97.11%97.11\% of individual option lookups over a shared 1,000-item sample. They do not agree exactly on what a searcher gets: the Postgres route is the more conservative, at 0.1120.112 against 0.1360.136 on those items. That gap is not the 128128 timed-out Postgres queries: dropping every item they touch leaves it at 0.1040.104 against 0.1270.127. It is a genuine recall difference, 109109 phrases found only by the offline index against 33 found only by Postgres. We report the higher of the two throughout, so the figure above is the one least favourable to our argument. Critically, blind accuracy does not concentrate on items whose key is verbatim statutory text. On the 199 items of the pilot for which both paths ran, Haiku scores 0.4330.433 on verbatim keys against 0.3470.347 on the rest (n=97n=97 and 7575; it returned no parseable letter on 27 of the 199) and Sonnet 0.3660.366 against 0.4140.414 (n=112n=112 and 8787) — the two models disagree even on the direction, and every interval overlaps. The leak is not quotation. It is plausibility. 5 The Blind Gate 5.1 Protocol For each item we present the four options with the question withheld and ask for a single letter, repeating over eight random permutations of option order. Under the null of no leakage each trial is correct with probability 0.250.25, so eight trials give a binomial reference: P(X≥5)=0.027P(X≥ 5)=0.027. We reject an item at five or more hits, and additionally require at least six of the eight trials to have produced a parseable letter, so that an item is never certified clean on the strength of two or three responses. We do not reject on the lower tail — zero hits out of eight occurs for 10.0%10.0\% of clean items — and treat below-chance behaviour as a set-level diagnostic instead. Two caveats on that reference. The eight orders are drawn independently rather than sampled without replacement, so an item sees 6.96.9 distinct orders on average and all eight only 27%27\% of the time; at temperature 0 a repeated order repeats its answer, which makes the trials slightly less independent than the binomial assumes. Recomputing the tail against seven effective trials moves the expected number of items answered blind eight times out of eight from 0.20.2 to 0.70.7, against the 1,4191,419 observed, so the conclusion is unaffected and the threshold is marginally conservative. Separately, the run did not persist per-item hit counts and seeded from Python’s salted string hash, so the histogram in Figure 1 is recoverable from our logs but the selection is not bit-for-bit reproducible from the release; the accepted set itself is published in full. Two design points matter for the numbers to mean anything. First, the gate runs on the item as published, not on a rewritten variant. Second, the model that selects is not the model that reports: gating uses Haiku 4.5 and every reported blind figure comes from models that took no part in selection. 5.2 Results Over the full bank the pooled blind rate during gating is 0.3830.383. The distribution of hits per item is strongly bimodal: 3,517 items (29.3%29.3\%) are never answered blind in eight attempts, while 1,419 (11.8%11.8\%) are answered blind in all eight — an event with probability 1.5×10−51.5× 10^-5 under chance. Leakage is concentrated in a minority of items, which is what makes selection a plausible remedy at all. 1,419 items answered blind every time;chance predicts 0.201000200030004000012345678Blind hits out of 8 permutationsItemsObservedChance (binomial) Figure 1: How often each item is answered blind, over eight permutations of option order, against what chance predicts. The bank does not behave like a set of items with a uniform small leak: 3,517 items are never answered blind and 1,419 are answered blind every time, where Binomial(8,0.25)Binomial(8,0.25) predicts 0.20.2 items in the latter group. That concentration is what makes selection worth attempting. Figure 1 shows the shape. Of the 8,1378,137 items at four hits or fewer, nine fall to the minimum-trials rule, so the gate retains 8,128 items (67.8%67.8\%). Retention varies by specialisation: administrative 74.3%74.3\%, commercial 70.6%70.6\%, civil 64.6%64.6\%, criminal 63.2%63.2\%, general 61.4%61.4\%. On those 8,1288,128 items the gating model now scores 0.2040.204 blind. By its own measure the bank is clean. Section 6 asks what that measure is worth to anyone else, and the answer is the reason this paper exists. 6 Does the Repair Transfer? The gate was built with a single model. We test whether the items it kept are clean for anyone else, on a 600-item sample stratified by gate outcome — 400 accepted and 200 rejected — scored by thirteen models from eight vendors under a uniform protocol. The sample is not a random draw from the bank, so the figures below are conditional on gate outcome rather than bank-wide averages. All models were reached through Amazon Bedrock in August 2026 with greedy decoding wherever the provider accepts it. The blind condition on accepted items was subsequently re-run on the entire accepted set with a 4,096-token budget; the two runs agree closely where they overlap. On the items both scored, ten of thirteen models differ by less than 0.0130.013 and the largest difference is Pixtral’s 0.0170.017 on the 233233 items it shares; Sonnet and Haiku return identical predictions on all 400400. The differences between the columns below are therefore sampling, not protocol. Pixtral Large is quota-bound on its inference profile and was re-run on the whole accepted set. One warning for anyone subsampling the released file: the bank is stored in the order of the five source documents, and the documents are the specialisations, so its first 2,0002,000 items are 74%74\% administrative. Stratify rather than slice. Table 3: Blind (options only) and full accuracy, ordered by full-condition accuracy. Chance is 0.2500.250. Blind figures on the accepted set are measured on all 8,1288,128 items; the rejected-set and full-condition columns remain on the 600-item sample, so their intervals are the wider ones. Two models are set apart because their numbers are not evidence of recognition: Haiku is the gate’s own model, and Llama 3.1 8B answers A to 92%92\% of items, which on its own earns 0.2870.287 here (Table 4). Blind n is 8,1288,128 except DeepSeek R1 7,9767,976, Llama 3.1 8B 7,8867,886, Nova Micro 8,0398,039, Ministral 8,1268,126 and Pixtral 8,1258,125; full n is 400400 except DeepSeek R1 352352, Pixtral 365365, Nova Micro 388388 and Llama 3.1 8B 202202. Model Vendor Blind, accepted Blind, rejected Full acc. Full rej. GPT-5.6 OpenAI 0.515 [.504,.526] 0.789 [.727,.840] 0.958 0.960 Sonnet 4.6 Anthropic 0.320 [.310,.330] 0.730 [.665,.787] 0.740 0.905 DeepSeek R1 DeepSeek 0.257 [.247,.266] 0.665 [.593,.729] 0.668 0.845 Nova Pro Amazon 0.232 [.223,.241] 0.645 [.577,.708] 0.580 0.739 Qwen3 32B Alibaba 0.244 [.234,.253] 0.665 [.597,.727] 0.552 0.720 Pixtral Large Mistral 0.239 [.230,.248] 0.602 [.531,.669] 0.548 0.718 Llama 3.3 70B Meta 0.237 [.228,.247] 0.610 [.541,.675] 0.540 0.790 Gemma 3 12B Google 0.230 [.221,.239] 0.595 [.526,.661] 0.472 0.745 Ministral 8B Mistral 0.239 [.229,.248] 0.540 [.471,.608] 0.463 0.740 Nova 2 Lite Amazon 0.225 [.216,.234] 0.545 [.476,.613] 0.460 0.660 Nova Micro Amazon 0.233 [.224,.242] 0.518 [.448,.586] 0.436 0.658 Llama 3.1 8B Meta 0.292 [.282,.302] 0.337 [.274,.406] 0.426 0.531 Haiku 4.5 Anthropic 0.204 [.195,.213] 0.800 [.739,.850] 0.575 0.805 Four readings follow, and the first is the answer to the question in the heading. A gate built with one model does not protect against a better one. The gate ran until Haiku 4.5 scored 0.2040.204 blind on the items it kept — below chance, its own leakage selected away. GPT-5.6, which took no part in that selection, answers 0.5150.515 of those same items with the question hidden, and Sonnet 4.6 0.3200.320. Neither figure is a positional artefact: corrected for each model’s own answer-position habit their excess is +0.265+0.265 and +0.081+0.081, the only two excesses in the set (Table 4). This is a demonstration rather than an inference, and it is the paper’s central result: filtering a bank against one model leaves the strongest models in the set reading the options exactly as before. Position bias explains almost all of the rest. Unlike the gate, the sweep presents each item once in its published order, so a model with a positional habit could score above chance without reading anything. Those habits are large and vendor-specific: Ministral picks A in 2%2\% of blind trials, Pixtral picks B in 46%46\%, Sonnet 4.6 picks D in 42%42\%, and Llama 3.1 8B picks A in 92%92\%. So for each model we compute what its own answer-position distribution alone would earn against the gold-position distribution of the accepted set, which is A-heavy (0.2900.290, 0.2450.245, 0.2410.241, 0.2230.223). That position-only expectation runs 0.2370.237–0.2870.287 across models, and subtracting it is what separates recognition from habit. Table 4: What is left after each model’s own positional habit is subtracted. Blind accuracy is measured on all 8,1288,128 accepted items. “Habit” is the score that model’s answer-position distribution alone earns against the gold positions of this set. Only two models have an excess worth the name. Model A B C D Blind Habit Excess GPT-5.6 0.24 0.27 0.27 0.22 0.515 0.250 +0.265+0.265 Sonnet 4.6 0.08 0.20 0.30 0.42 0.320 0.238 +0.081+0.081 DeepSeek R1 0.12 0.24 0.29 0.35 0.257 0.242 +0.015+0.015 Qwen3 32B 0.14 0.27 0.30 0.30 0.244 0.244 +0.000+0.000 Ministral 8B 0.02 0.25 0.45 0.29 0.239 0.238 +0.001+0.001 Llama 3.3 70B 0.08 0.28 0.31 0.33 0.237 0.240 −0.003-0.003 Nova Micro 0.09 0.31 0.22 0.39 0.233 0.240 −0.007-0.007 Nova Pro 0.17 0.28 0.27 0.28 0.232 0.246 −0.014-0.014 Gemma 3 12B 0.08 0.28 0.40 0.23 0.230 0.242 −0.012-0.012 Nova 2 Lite 0.03 0.30 0.29 0.39 0.225 0.237 −0.012-0.012 Pixtral Large 0.10 0.46 0.25 0.20 0.239 0.244 −0.005-0.005 Llama 3.1 8B 0.92 0.00 0.07 0.01 0.292 0.287 +0.005+0.005 Haiku 4.5 0.16 0.44 0.22 0.19 0.204 0.247 −0.043-0.043 Two models keep a real excess: GPT-5.6 at +0.265+0.265 and Sonnet 4.6 at +0.081+0.081. Every other held-out model lands within 0.0150.015 of what its habit alone would earn; the gate’s own model sits 0.0430.043 below its habit, which is what selecting on it was supposed to do. Llama 3.1 8B is the instructive case: it scores 0.2920.292 blind, comfortably above chance and above four models we do not call leaky, purely because it answers A to 92%92\% of items and the key sits at A in 29%29\% of them. Its excess is +0.005+0.005. Read without the habit column it would look like a third leaker; it is not reading the options at all. The gate found something real, for almost everyone. On rejected items eleven of the twelve held-out models score 0.5180.518–0.7890.789, far above chance, and the accepted-versus-rejected gap is positive for all twelve. The exception is Llama 3.1 8B at 0.3370.337, which is what a model that answers A to everything scores on a set where the key is at A more often than not. For every model that reads the options at all, one model’s blind failures are not idiosyncratic: they identify items that leak to the rest. The selection also seems to carry a signature of the model that made it. The gate keeps the items Haiku 4.5 could not answer blind — items where Haiku’s preferred option is a distractor. Any model that shares Haiku’s preferences inherits that preference, and with it the wrong answer. Agreement with Haiku’s blind pick runs 0.420.42–0.520.52 for ten of the twelve where independence would give 0.250.25, and it predicts the damage: agreement correlates with the accepted-to-rejected gap at r=0.892r=0.892 over twelve models (Table 5). That correlation leans on one point. Llama 3.1 8B agrees with Haiku less than chance, at 0.1820.182, because it is not reading the options, and it has much the smallest gap, 0.0450.045; drop it and r falls to 0.5770.577 over the remaining eleven, which is the honest strength of the relationship among models that do read. GPT-5.6, the other model outside the band at 0.3490.349, is also the one the gate damages least. Table 5: Agreement with the gating model’s blind pick, and the cost of that agreement. Independence would put agreement at 0.2500.250. The gap is blind accuracy on rejected items minus blind accuracy on accepted items; the more a model answers like the gate, the more the gate’s selection costs it. Model Agrees with Haiku Accepted Rejected Gap Pixtral Large 0.529 0.239 0.602 0.363 Nova Pro 0.504 0.232 0.645 0.413 Llama 3.3 70B 0.497 0.237 0.610 0.373 Qwen3 32B 0.489 0.244 0.665 0.421 Nova 2 Lite 0.486 0.225 0.545 0.320 DeepSeek R1 0.473 0.257 0.665 0.408 Gemma 3 12B 0.462 0.230 0.595 0.365 Ministral 8B 0.442 0.239 0.540 0.301 Nova Micro 0.430 0.233 0.518 0.285 Sonnet 4.6 0.423 0.320 0.730 0.410 GPT-5.6 0.349 0.515 0.789 0.274 Llama 3.1 8B 0.182 0.292 0.337 0.045 This is the finding that a 400-item sample could not have produced. At that size every interval was wide enough to cover chance, and eleven models read as “at chance” — a tidy result that concealed both facts above: that nine of them extract nothing at all once habit is subtracted, and that what they do extract is shaped by the model that did the selecting. Leakage scales with capability. Ranking the eleven held-out models by full-condition accuracy reproduces their ranking by blind accuracy closely (Table 3). Pearson r=0.916r=0.916 (95% CI [0.703,0.978][0.703,0.978]); leaving out any single model gives 0.8380.838–0.9430.943, and rank correlation is 0.7730.773. That correlation is partly mechanical: the x-axis is full accuracy on the same items as the y-axis, so item-level noise is shared. Replacing the x-axis with an independent capability measure — full accuracy on the rejected items, a disjoint set — gives r=0.813r=0.813 (95% CI [0.415,0.950][0.415,0.950]): still positive, with zero outside the interval, and stable under leave-one-out (0.8090.809–0.8540.854). Note what the association now rests on: with the blind axis compressed into a band of 0.0320.032 for nine of the eleven, it is carried mostly by the two models that leak. Among the nine alone, on the same disjoint axis, it is r=0.713r=0.713 — suggestive, but over a band narrower than the intervals of the quantity it is meant to explain. We report an observed association. The more robust statement is a ratio, and Figure 2 shows its shape. Blind accuracy is a fixed fraction of full accuracy across the capability range: 0.3840.384–0.5380.538, mean 0.4630.463. That arithmetic holds, but it should not be read as “a model recovers 46%46\% of its competence without the question”. For nine of the eleven the blind score is their positional habit and nothing else, so the fraction describes where habit happens to land, not partial competence. The reading survives only for the two models with a real excess. 0.40.40.450.450.50.50.550.550.60.60.650.650.70.70.750.750.80.80.850.850.90.90.950.95110.20.20.30.30.40.40.50.50.60.6chancenine models at their own habitSonnet 4.6GPT-5.6Full-condition accuracy on gate-accepted itemsBlind accuracy (options only) Figure 2: Blind accuracy against full-condition accuracy on the gate-accepted items, eleven held-out models from eight vendors. Bars are 95% Wilson intervals on the blind estimate, now measured on the whole accepted set rather than a 400-item sample, which is why they are short; the horizontal position still carries the sample’s error. The plotted correlation, r=0.916r=0.916, shares items between the two axes; against an independent capability estimate on disjoint items it is r=0.813r=0.813 (§6). The gate was built with a model absent from this plot, and Llama 3.1 8B is absent because it answers A to 92%92\% of items. The bracketed nine sit within 0.0150.015 of their own positional habit (Table 4). The implication is the paper’s main point, and it comes in two parts that should not be conflated. The first we observe directly: a bank filtered until one model is at chance is not clean for a stronger one. Our accepted set was gated against Haiku 4.5, and GPT-5.6 answers 0.5150.515 of it blind. That is a demonstration, not an inference. The second is a prediction. If blind accuracy keeps tracking capability beyond the range we tested, then filtering against the models available today will not hold against the models available later, and each round of filtering will remove the items that discriminate best. We have eleven models spanning full-condition accuracy from 0.4360.436 to 0.9580.958; whether the relationship continues above that is untested, and we state it as a conjecture rather than a result. 6.1 The numbers are not an artefact of how we asked Every blind figure above comes from one prompt, so we re-ran the blind condition on the 400 gate-accepted items under five phrasings: the original; a minimal form with no framing at all (“choose one option”); the original with the word legal removed, to test whether naming the domain is itself what primes recognition; an English instruction over Ukrainian options; and an explicit “this is a guessing task” framing. Two leaking models and one that sits at chance were run on all five. Table 6: Blind accuracy under five prompt phrasings, 400 gate-accepted items. Chance is 0.2500.250. These figures are on a 400-item subset of the accepted set, not the whole of it, so they sit above the full-scale numbers in Table 3 — for these three models the subset runs 0.0090.009 to 0.0240.024 leakier than the set it was drawn from, and every model in it inherits that. What the table is for is the spread within a model, which the shared sample does not distort. Prompt Sonnet 4.6 GPT-5.6 Nova Pro Original 0.343 0.524 0.255 Minimal, no framing 0.323 0.463 0.265 Domain word removed 0.310 0.490 0.236 English instruction 0.325 0.521 0.240 Framed as guessing 0.330 0.479 0.220 Spread 0.033 0.061 0.045 Three things follow. The leak survives every phrasing: Sonnet 4.6’s worst variant is 0.3100.310 with a lower confidence bound of 0.2670.267, still above chance, and GPT-5.6’s worst is 0.4630.463. Naming the domain helps a little and explains almost nothing — dropping legal costs 0.0330.033 for Sonnet and 0.0340.034 for GPT-5.6, leaving both far above chance. And the control behaves: Nova Pro stays at chance under all five phrasings, every interval covering 0.2500.250, so the probe does not manufacture signal where there is none. The within-model spread, 0.0330.033–0.0610.061, is far smaller than the 0.180.18 that separates Sonnet from GPT-5.6 on these items, so which models leak is not a function of how we asked. That covers the blind prompt, which leaves three other things fixed at one realisation each, all of them underneath the ratio: the wording of the full condition, which is its denominator; the labels on the options, which are Latin A)–D) set over Ukrainian text, so that a letter prior would be indistinguishable from recognition of content; and the order of the options. We vary the first two here and the third in §6.2, moving one factor at a time from the configuration used throughout. Table 7: Presentation robustness on the same 400 gate-accepted items. Blind chance is 0.2500.250. Marker rows change only the labels; the Latin rows repeat the baseline for reference. No model answered with a Latin letter when shown Cyrillic labels, so the conventions are not being conflated. Sonnet 4.6 GPT-5.6 Nova Pro Full-condition wording Original 0.738 0.957 0.578 Minimal, no framing 0.740 0.963 0.610 Exam framing removed 0.755 0.962 0.573 English instruction 0.750 0.962 0.578 Expert persona 0.757 0.960 0.598 spread 0.020 0.005 0.037 Option labels, blind Latin A)–D) 0.343 0.524 0.255 Cyrillic А)–Г) 0.360 0.519 0.273 Numeric 1)–4) 0.347 0.507 0.263 Option labels, full Latin A)–D) 0.738 0.957 0.578 Cyrillic А)–Г) 0.750 0.962 0.583 Numeric 1)–4) 0.765 0.967 0.570 Neither matters. The full condition moves by 0.0050.005–0.0370.037 across five phrasings, so the denominator of the ratio is not a property of our prompt: recomputed against each of them, the blind-to-full ratio spans 0.4520.452–0.4640.464 for Sonnet and 0.5440.544–0.5470.547 for GPT-5.6, well inside the 0.3840.384–0.5380.538 range we report across models. And the labels are inert in both conditions and for all three models, the largest shift being 0.0270.027. The Latin alphabet was not doing the work. 6.2 Position bias is a different thing from the leak The blind numbers so far use the option order as printed in the bank. A model that simply favours one slot would score above chance whenever the key happened to be there, which in a single-order measurement is indistinguishable from recognising the option’s content. So we forced the key into each of the four slots in turn, keeping the distractors in their relative order: 400 items × 4 placements × 3 models, blind. This yields a test that needs no modelling assumption. If a model’s choice were independent of what the options say, then P(correct∣key in slot j)P(correct in slot j) is just qjq_j, the rate at which it reaches for slot j; and because the qjq_j sum to one, the four accuracies must sum to one as well. Their mean is then exactly chance, 0.2500.250, however lopsided the preference. Whatever exceeds that is content. Since each item contributes four correlated observations, we take the interval at item level rather than treating 1,6001,600 responses as independent. Nova Pro (sum 1.08)Sonnet 4.6 (sum 1.43)GPT-5.6 (sum 2.01)ABCDABCDABCD0.00.20.40.6Slot holding the keyBlind accuracy Figure 3: Blind accuracy with the key forced into each slot (bars) against how often the model picks that slot at all (points). Under content-independence the two coincide and the bars sum to 1.001.00. Nova Pro has a real slot preference and no gap; Sonnet has the largest preference in the set and a clear gap; GPT-5.6 has almost no preference and the largest gap of all. Figure 3 shows the three cases and Table 8 gives the numbers. Sonnet 4.6 turns out to carry a severe positional prior — it picks the last slot 0.4500.450 of the time and the first 0.0770.077 — which drags its blind accuracy from 0.1220.122 with the key at A to 0.6500.650 with the key at D, a spread of 0.5280.528, an order of magnitude larger than anything in Table 7. But the accuracies sum to 1.4271.427, not 1.0001.000, and the lift over the preference is positive in every slot, so content is being read on top of the prior. Nova Pro, the control, has a genuine preference of its own and no content: it sums to 1.0851.085 and its position-free estimate, 0.2710.271, does not separate from chance. GPT-5.6 is the opposite of Sonnet, picking the four slots almost uniformly and scoring near 0.500.50 wherever the key sits. Table 8: Blind solvability with position bias removed by construction: the mean over the four forced placements, with an item-level 95%95\% interval on 400 items. The single-order column is the same 400-item measurement, not the full-scale figure of Table 3. Model Slot spread Sum Position-free 95% CI Single-order Nova Pro 0.180 1.085 0.271 [0.239, 0.304] 0.255 Sonnet 4.6 0.528 1.427 0.357 [0.325, 0.389] 0.343 GPT-5.6 0.085 2.009 0.502 [0.461, 0.542] 0.524 Two things follow. Order does not carry the result: on this subset the position-free estimates sit within 0.0220.022 of the single-order values measured on the same items, and for Sonnet the single-order figure is if anything the conservative one, so nothing here depends on where the key happened to be printed. And position bias and leak exploitation are orthogonal. The model with the largest slot preference is not the one that leaks most, and the model with almost none leaks most of all. This matters for remedies: permuting options, the standard fix for position bias and the one our own gate relies on, does nothing whatever about content that is recognisable on its own. The labelling sweep of §6.1 also speaks to the mechanism. Zheng et al. 2024 trace selection bias to token bias — probability mass attached to the option-ID tokens themselves — which predicts that the preference should move when the tokens change. It largely does not. Sonnet 4.6 picks the first slot 0.0920.092 of the time under Latin A–D, 0.0980.098 under Cyrillic А–Г and 0.0900.090 under digits 1–4; Nova Pro likewise stays in a narrow band (0.1800.180, 0.1470.147, 0.1820.182). What the labels do change is the size of the pull towards the last slot, which for Sonnet runs 0.4150.415, 0.4770.477 and 0.3100.310. On our items the bias is positional first and token-dependent second. 6.3 Reasoning at inference time is not what makes options solvable Our models run at vendor defaults, so the set mixes models that reason before answering with models that do not — as Fan et al. 2026 also do, and for the same reason: reasoning cannot be switched off in GPT-5.6, which rejects reasoning_effort on Bedrock as it rejects temperature. That leaves an objection open: perhaps the 0.180.18 separating GPT-5.6 from Sonnet blind is inference-time compute rather than what they read from the options. What cannot be equalised downwards can be moved upwards and measured. Table 9: Reasoning interventions on the 400 gate-accepted items. p-values are McNemar tests on the paired items against that model’s direct-answer row. The extended-thinking row is reported for completeness only: Anthropic’s API refuses temperature 0 when thinking is enabled, so that row moves two things at once. Model Condition Blind p Full Out tok. Sonnet 4.6 direct answer 0.343 — 0.738 4 Sonnet 4.6 extended thinking on 0.343 — 0.785 59 Sonnet 4.6 step-by-step prompt 0.388 0.139 0.805 619 Nova Pro direct answer 0.255 — 0.578 260 Nova Pro step-by-step prompt 0.228 0.284 0.578 530 DeepSeek R1 2,048-token budget 0.253 — 0.668 — DeepSeek R1 4,096-token budget 0.231 — 0.648 1,363 Four measurements agree. Enabling extended thinking leaves Sonnet’s blind accuracy exactly where it was, though that alone proves little — the model declines the budget, spending 59 output tokens of an allowed 2,000. Forcing the reasoning at the prompt, as LEXam does for every model, makes it reason at 619 tokens against 4, and moves blind accuracy from 0.3430.343 to 0.3880.388, which a paired test does not separate from noise. The same instruction on the same items raises the full condition from 0.7380.738 to 0.8050.805 (p=0.001p=0.001). Reasoning that demonstrably helps when the question is present does not measurably help when it is absent, which is what one should expect: with no question there is nothing to reason towards. Nor does it manufacture the ability — Nova Pro under the same instruction stays at chance. Across models the association fails in both directions: DeepSeek R1 spends 1,3631,363 output tokens per blind item and scores 0.2310.231, at chance, while GPT-5.6 also reasons and scores 0.5240.524 on this subset. The two reasoning models sit at the two extremes, with a conventional model between them. Inference-time reasoning is neither necessary nor sufficient for option-only solvability. Re-running R1 at LEXam’s 4,096-token budget rather than ours also resolves the one real parsing casualty in this paper: on the blind condition over accepted items, unparsed responses fall from 18.0%18.0\% to 1.5%1.5\% — the 13.1%13.1\% quoted elsewhere pools all four of its cells — while the estimates move only within their intervals (0.253→0.2310.253→ 0.231 blind on 328328 then 394394 parsed items, 0.668→0.6480.668→ 0.648 full). The truncation cost coverage; it was not biasing the numbers. 7 A Benchmark That Does Not Leak, and Why Filtering failed, and rewriting distractors failed. The one thing that works is not a repair at all but a property some banks have from the start. If option-only solvability were a general property of legal exam items, it should appear in other exam-derived benchmarks. We ran the same blind probe on the four-choice split of LEXam (Fan et al. 2026), 1,655 items from Swiss university law exams, with the two models that leaked most on our bank. Table 10: The same blind probe on LEXam’s four-choice split. Chance is 0.2500.250. n counts responses from which a letter could be parsed; GPT-5.6 produced 34 unparseable answers in the full condition and none in the blind one. Model Condition n Accuracy 95% CI Sonnet 4.6 blind 1655 0.228 [0.208, 0.249] Sonnet 4.6 full 1655 0.642 [0.619, 0.665] GPT-5.6 blind 1655 0.228 [0.208, 0.249] GPT-5.6 full 1621 0.808 [0.788, 0.827] Both models sit at chance, with the upper confidence bound below 0.2500.250. The same GPT-5.6 that recovers 0.5150.515 of our gate-accepted items blind recovers nothing here. every LEXam optionfits in 33 characters0.000.250.500.751.00050100150Option length (characters)Cumulative share of optionsUA-JudgeExamLEXam 4-choiceOptions: propositions vs pointerschance0.00.20.40.6GPT-5.6Sonnet 4.6Blind accuracySame probe, same models Figure 4: Left: the cumulative distribution of option length in the two banks. LEXam’s curve is complete at 33 characters — its longest option anywhere — while 59.6%59.6\% of our options are longer than that. Ours are legal propositions that can be judged on their own; LEXam’s are pointers into a list given in the stem. Right: what the same two models score with the question hidden. The design choice on the left produces the difference on the right. The reason is visible in the items (Figure 4). In every one of LEXam’s 1,6551,655 four-choice items, all four options are references to statements enumerated in the stem — “i und i”, “i and i”, “none of the statements”. The median option is 9 characters against 42 in our bank, and the point is sharper than the median: the longest option anywhere in the split is 33 characters, “i, i, iv, v, vii, viii, und ix”. No legal proposition of any kind appears in an option position. A reference carries no content of its own, so there is nothing for a reader to recognise when the stem is hidden. LEXam is immune to this failure by construction, not by filtering. This cuts two ways. It validates the probe: an instrument that reports leakage should report chance where chance is the ground truth, and it does. And it relocates our finding. Option-only solvability is not a property of legal exams, nor of any particular jurisdiction or language. It is a property of item format: options that are self-contained legal propositions can be judged on their own, and options that are pointers cannot. Banks written in the first style — which includes most professional licensing material we are aware of, and ours — need the blind baseline reported. Banks written in the second do not. 8 Remedies: One Clear Failure, Two Cautions Filtering is not the only repair one might try, so we tried the others. None of them lands on chance either, which is the same failure in a different guise: an item set that a model gets wrong at a predictable rate is not a neutral item set. Swapping in real answers overshoots. If distractors leak because they read as implausible law, replace them with propositions that are real law: the correct answers of other items, chosen from the same specialisation, matched on answer type and length, with the key’s position re-randomised. Blind accuracy for Sonnet 4.6 moves from 0.3860.386 [0.321,0.455][0.321,0.455] to 0.1680.168 [0.122,0.226][0.122,0.226] (n=197n=197, intervals disjoint) while full accuracy is essentially preserved (0.792→0.7460.792→ 0.746). The mechanism is real and controllable — and the result is useless. A set on which a model scores 0.1680.168 blind is as exploitable as one on which it scores 0.3860.386; an adversary inverts it. Selecting donors by similarity to the question makes the three distractors mutually coherent and leaves the key as the odd one out; a variant selecting donors by similarity to the key inverts the artifact instead of removing it. We could not find a donor rule that lands on chance. Model-written distractors may leak to the model that wrote them. Generating distractors with Sonnet 4.6, verifying with a second model that each is definitively wrong, and gating the result yields items on which Sonnet 4.6’s blind accuracy is 0.3810.381 [0.296,0.473][0.296,0.473], against 0.2740.274 [0.201,0.363][0.201,0.363] on the human-written originals of the same 113 items. The point estimate moves in the direction one would fear — the generator’s own family recognising its own writing — but at n=113n=113 the intervals overlap and we cannot call it established. We report it because the design risk is cheap to avoid: do not generate distractors with a model from a family you intend to evaluate. No evidence that negation items drive the effect. It is natural to suspect items phrased “which is not…”, where the key is the odd one out by construction. Sonnet 4.6 scores 0.2500.250 [0.138,0.411][0.138,0.411] blind on the 36 negation items in the pilot and 0.4160.416 [0.343,0.493][0.343,0.493] on the 161 others. The point estimates point away from the hypothesis, but 36 items give an interval too wide to separate the two, so this rules the explanation neither in nor out. 9 Limitations The central claim is a demonstration, not a law. It rests on one bank and one gating model: a set filtered against Haiku 4.5 is not clean for GPT-5.6. It does not follow that every filter fails against every stronger model, only that this one did, decisively, and that nothing in the procedure prevents it. A practitioner who wants a clean set should gate with several models from different vendors and accept the lower yield. What we measured at full scale, and what we did not. Only the blind condition on accepted items was re-run on all 8,1288,128; the rejected-set and full-condition columns still rest on the 600-item sample, with half-widths near ±0.05± 0.05 and ±0.07± 0.07. Because every model was scored on the same 400-item draw, their errors are correlated and the whole column moved together when we measured the full set — by 0.0100.010 on average, which no per-model interval predicted. The presentation and position checks of §6.1–§6.2 likewise cover one item set and three models rather than all eleven. Coverage is uneven across models. Llama 3.1 8B returns no parseable letter on 47.5%47.5\% of full-condition calls, and a larger token budget does not help; its full-condition figure rests on 202 of 400 items, which is why it is set apart and excluded from the correlation. DeepSeek R1 (13.1%13.1\%) and Pixtral Large (5.9%5.9\%) were budget-bound and recover at 4,0964,096 tokens. The other nine models parse above 99.5%99.5\%. What the design cannot tell us. We have no human blind baseline, so we cannot say whether a Ukrainian lawyer shown only the options would score near the pooled 0.3830.383 we report for the gating model, which would make the effect a property of the item-writing genre rather than of models. The capability association rests on eleven models, none below 0.4360.436 full-condition accuracy, and the claim that item format governs the effect rests on a comparison of two benchmarks. Extraction fidelity is verified end-to-end on 200 items and by aggregate on the rest. Finally, GPT-5.6 reaches 0.9580.958 on the accepted set: for the strongest model tested the filtered benchmark is close to saturated, and its remaining discriminative value sits in the models below it. 10 Conclusion Multiple-choice benchmarks should report what a model scores without the question next to what it scores with it — and should subtract, from that score, what the model’s own answer-position habit would have earned anyway. The raw blind number is not interpretable alone. On our gate-accepted set Llama 3.1 8B scores 0.2920.292 blind — above every held-out model except the two that actually leak, and three and a half points above DeepSeek R1 — entirely because it answers A to 92%92\% of items and the key sits at A in 29%29\% of them. Corrected for habit, its excess is +0.005+0.005, and only two of twelve models read anything out of the options at all — but those two read a great deal, GPT-5.6 recovering 0.5150.515 of a set that was filtered until another model could not. The headline of that measurement is what filtering does not buy. Our gate ran until Haiku 4.5 scored 0.2040.204 on the items it kept; GPT-5.6, which took no part in the selection, answers 0.5150.515 of the same items with the question hidden. Filtering against one model does not neutralise a bank for a better one, and there is no version of the procedure we tested that does. The surviving items also appear to carry a signature of the model that selected them — agreement with its blind choices tracks how far the selection sets another model back — though that relationship rests on twelve points and one of them carries it, so we offer it as an observation to test rather than a result. Both of these are invisible at the sample sizes ordinarily used. On 400 items nine models read as “statistically at chance”, a tidy result that concealed the habit correction and the rest. Scoring every model on one shared sample also correlates their errors, so the whole column can move together — and did, by 0.0100.010, when we measured the full set. The same probe on LEXam returns chance, for a reason that is a design choice rather than an accident: its options point at statements in the stem instead of standing alone. Where options must be self-contained legal propositions, publish the habit-corrected blind baseline per model alongside the headline score. Where the format allows reference-style options, the problem does not arise. Appendix A Prompts The prompts below are in Ukrainian, matching the language of the bank; the LEXam comparison in §7 uses the same two prompts in English, since that split is German and English. Options are presented as A)–D) in the order given, except in the gate, where the order is permuted per trial. Blind condition. Нижче наведено чотири варанти вдповд на приховане юридичне питаня. Питаня вам не показано. Оберть той варант, який наймоврнше правильним. Вдповдайте ЛИШЕ одню лтерою: A, B, C або D. Full condition. Питаня квалфкацйного тестуваня для кандидатв на посаду суд. Оберть диний правильний варант. Вдповдайте ЛИШЕ одню лтерою: A, B, C або D. Blind condition, LEXam. Below are four answer options to a hidden legal exam question. The question is not shown to you. Choose the option most likely to be correct. Reply with ONE letter only: A, B, C or D. Full condition, LEXam. Legal exam question. Choose the single correct option. Reply with ONE letter only: A, B, C or D. Vendors answer in three shapes — a bare letter, an explicit Вдповдь: X followed by prose, or a paragraph of reasoning ending in the letter — so the parser tries an explicit-marker match, then a leading letter, then the last standalone letter. Unparsed rates are below 0.5%0.5\% for nine of the thirteen models. The exceptions, on the 600-item sweep, are DeepSeek R1 (13.1%13.1\%), Llama 3.1 8B (25.5%25.5\%, and 47.5%47.5\% in the full condition alone), Pixtral Large (5.9%5.9\%) and Nova Micro (2.4%2.4\%). Unparsed responses are excluded from that model’s denominators rather than scored as wrong. Appendix B Models All models were reached through Amazon Bedrock in August 2026. Snapshot identifiers are given in full, since several of these aliases will move. The lower half of the conventional group — Qwen3 32B, Gemma 3 12B, Ministral 8B, Nova Micro and Llama 3.1 8B — was added to extend the capability range downwards; three of those overlap with the small open-source group evaluated by Fan et al. 2026, which makes the two model sets partially comparable. Table 11: Model identifiers, whether the provider accepted a temperature setting, and whether the model reasons before answering at its default settings. The region column is where the 600-item sweep ran; the full-scale blind run sharded Qwen3, Gemma, Ministral and Pixtral across several regions to work around per-profile quotas, which is recorded per call in the release. Following Fan et al. 2026, reasoning and conventional models are grouped rather than equalised: reasoning cannot be switched off in GPT-5.6, which rejects reasoning_effort on Bedrock as it rejects temperature. Model Bedrock identifier Region Greedy Reasons Reasoning models GPT-5.6 global.openai.gpt-5.6-sol eu-central-1 no yes DeepSeek R1 us.deepseek.r1-v1:0 us-east-1 yes yes Conventional models Sonnet 4.6 eu.anthropic.claude-sonnet-4-6 eu-central-1 yes no Haiku 4.5 eu.anthropic.claude-haiku-4-5-20251001-v1:0 eu-central-1 yes no Nova Pro eu.amazon.nova-pro-v1:0 eu-central-1 yes no Nova 2 Lite eu.amazon.nova-2-lite-v1:0 eu-central-1 yes no Pixtral Large eu.mistral.pixtral-large-2502-v1:0 eu-central-1 yes no Llama 3.3 70B us.meta.llama3-3-70b-instruct-v1:0 us-east-1 yes no Qwen3 32B qwen.qwen3-32b-v1:0 us-east-1 yes no Gemma 3 12B google.gemma-3-12b-it us-east-1 yes no Ministral 8B mistral.ministral-3-8b-instruct us-east-1 yes no Nova Micro eu.amazon.nova-micro-v1:0 eu-central-1 yes no Llama 3.1 8B us.meta.llama3-1-8b-instruct-v1:0 us-east-1 yes no GPT-5.6 rejects the temperature parameter outright, so its requests omit it; every other model was run at temperature 0. The 600-item sweep used a 2,048-token output budget and the full-scale blind run 4,096; the budget matters for two models for opposite reasons. Nova Pro is merely verbose: it writes a paragraph of justification around the letter, some 260 output tokens of which nearly all is visible text. DeepSeek R1 spends its tokens on a reasoning trace the response body does not contain, averaging 1,3631,363 output tokens per blind item over the full-scale run for a handful of visible characters. It is the model the budget bound hardest — at 2,048 tokens it failed to reach a letter on 13.1%13.1\% of calls, at 4,096 on 1.9%1.9\%, the estimates moving only within their intervals — but not the only one: Pixtral Large fell from 5.9%5.9\% to 1.2%1.2\% on the same change. Because the set mixes models that reason at inference time with models that do not, §6.3 measures what that mixture is worth rather than leaving it as a caveat. Data and Code The corpus (11,990 items), the gated subset (8,128), the 600-item cross-vendor sample, all 9,600 blind and full predictions from the first sweep and 7,200 from the small-model extension, the negative-result runs, the 21,60021,600 calls of prompt, labelling and position ablation, the reasoning controls, the 105,664105,664-call full-scale blind run, and the extraction, verification and gating code are released at https://huggingface.co/datasets/overthelex/ua-judge-exam. References Balepur et al. [2024] Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. Artifacts or abduction: How do LLMs answer multiple-choice questions without the question? In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10308–10330, 2024. URL https://aclanthology.org/2024.acl-long.555/. Chalkidis et al. [2022] Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 4310–4330, 2022. doi: 10.18653/v1/2022.acl-long.297. Cho et al. [2026] Gyeongje Cho, Yeonkyoung So, and Jaejin Lee. Choices speak louder than questions. In International Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2502.18798. Fan et al. [2026] Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. LEXam: Benchmarking legal reasoning on 340 law exams. In International Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2505.12864. Guha et al. [2023] Neel Guha, Julian Nyarko, Daniel E Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, et al. LegalBench: A collaboratively built benchmark for measuring legal reasoning in large language models. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. URL https://arxiv.org/abs/2308.11462. Gururangan et al. [2018] Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, 2018. Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. CUAD: An expert-annotated NLP dataset for legal contract review. In Proceedings of the 35th Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, 2021. URL https://arxiv.org/abs/2103.06268. Katz et al. [2024] Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270), 2024. doi: 10.1098/rsta.2023.0254. Niklaus et al. [2023] Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. LEXTREME: A multi-lingual and multi-task benchmark for the legal domain. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3016–3054, 2023. doi: 10.18653/v1/2023.findings-emnlp.200. Östling et al. [2023] Andreas Östling, Holli Sargeant, Huiyuan Xie, Ludwig Bull, Alexander Terenin, Leif Jonsson, Mns Magnusson, and Felix Steffek. The Cambridge Law Corpus: A dataset for legal AI research. In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. URL https://arxiv.org/abs/2309.12269. Pezeshkpour and Hruschka [2024] Pouya Pezeshkpour and Estevam Hruschka. Large language models sensitivity to the order of options in multiple-choice questions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2006–2017, 2024. URL https://aclanthology.org/2024.findings-naacl.130/. Poliak et al. [2018] Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics (*SEM), pages 180–191, 2018. Watson et al. [2026] Joe Watson, Joana Ribeiro de Faria, Marcus Tomalin, Mns Magnusson, Huiyuan Xie, Hao Tian Yeung, Christine Carter, Jonathan Rutherford, and Felix Steffek. Shortcut learning in legal judgment prediction: Empirical evidence from the UK employment tribunal. arXiv preprint arXiv:2607.04261, 2026. URL https://arxiv.org/abs/2607.04261. Zheng et al. [2024] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. Large language models are not robust multiple choice selectors. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=shr9PXz7T0.