Paper deep dive
MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge
Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Biomedical large language model (LLM) evaluation requires auditable assessment of narrow, evolving, source-grounded subspecialty knowledge. Multiple sclerosis MRI (MS-MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standardized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mimics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source-grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item-level and topic-specific LLM evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2610.06170v1
- Canonical: https://arxiv.org/abs/2610.06170v1
Trouble viewing inline? Open PDF directly →
Full Text
41,122 characters extracted from source content.
Expand or collapse full text
MS-Exam-Gen: Source-Grounded Benchmark Construction for Evaluating LLMs on Textual Multiple Sclerosis MRI Knowledge Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique eBRAIN Lab, Division of Engineering, New York University Abu Dhabi (NYUAD), Abu Dhabi, UAE abdul.basit@nyu.edu, mh6117@nyu.edu, muhammad.shafique@nyu.edu Abstract—Biomedical large language model (LLM) evalua- tion requires auditable assessment of narrow, evolving, source- grounded subspecialty knowledge. Multiple sclerosis MRI (MS- MRI) provides a high-stakes textual-knowledge test case because correct reasoning requires current diagnostic criteria, standard- ized acquisition and reporting knowledge, longitudinal monitoring concepts, lesion morphology, and recognition of difficult mim- ics. We present MS-Exam-Gen, a reproducible framework for constructing and auditing a text-based multiple-choice question (MCQ) benchmark for MS-MRI knowledge; it does not evaluate direct MRI image interpretation. MS-Exam-Gen targets source- grounded criteria, protocols, reporting, and differential diagnosis. The framework combines expert-source indexing, exam-oriented topic induction, evidence-grounded MCQ generation, automated quality audits, a same-family consistency screen, and empirical calibration. From a 66-source corpus indexed into 4,289 retrieval chunks, the pipeline produced a locked 3,058-item candidate benchmark spanning 16 topics and 53 subtopics. Evaluation across 12 primary LLM endpoints yielded 36,696 item-level predictions and separated performance over a 42.8-percentage-point accuracy range (89.7% to 46.9%). Across these endpoints, 25.5% of items were missed by at least four. Post-generation audits showed that refreshed construction reduced measurable answer cues, while option-order testing showed that absolute MCQ scores remain position-sensitive. Generated construction labels remain metadata rather than validated psychometric categories. Because expert adjudication and full option-order counterbalancing remain future work, MS-Exam-Gen is not a clinically certified examination. It should be interpreted as an automatically filtered, source-grounded candidate benchmark and reproducible audit workflow for item- level and topic-specific LLM evaluation. Index Terms—large language models, benchmark construction, multiple sclerosis, MRI, medical question answering, retrieval- augmented generation I. INTRODUCTION Large language models (LLMs) increasingly perform well on broad medical examinations and biomedical question-answering benchmarks [1]–[4]. Such aggregate scores are useful, but they provide limited evidence of subspecialty auditability or workflow-specific knowledge. Broad resources such as MedQA, PubMedQA, and MMLU-Medical do not establish whether a model can be audited against the narrow, evolving, document- grounded knowledge used in a specific clinical workflow [5]– [7]. For biomedical and health informatics, the evaluation problem increasingly includes not only whether an LLM “knows medicine” in the aggregate, but whether it can be traced against explicit criteria, protocols, source evidence, and exceptions. Multiple sclerosis MRI (MS-MRI) textual knowledge is a demanding test case. Correct reasoning requires knowledge of dissemination-in-space/time rules, lesion morphology and distribution, standardized acquisition protocols, longitudinal activity assessment, treatment-safety surveillance, and clinically important mimics. The relevant knowledge is also evolving, with diagnostic criteria and MRI recommendations periodically revising how MS-MRI findings should be interpreted and reported [8]–[10]. Our focus is textual reasoning over MS- MRI criteria, protocols, reporting, monitoring, and mimics, not direct visual interpretation of MRI scans. Existing benchmarks leave a gap. Broad medical QA bench- marks are too general to isolate failures in MS-MRI guideline and reporting knowledge, while MS imaging benchmarks primarily evaluate lesion segmentation or image analysis [11], [12]. Neither setting tests whether LLMs can be audited against source-grounded textual knowledge about MS-MRI criteria, acquisition protocols, reporting standards, longitudinal monitoring, and differential diagnosis. This gap matters beyond MS: subspecialty benchmark construction is a core requirement for accountable biomedical AI evaluation when the target knowledge is narrow, evolving, and source-dependent. Constructing such a benchmark manually is expensive, but fully automatic generation is risky unless the construction pro- cess is auditable and explicitly limited to candidate items. Med- ical multiple-choice questions (MCQs) can contain malformed schemas, ambiguous answer keys, superficial option-length cues, stale source assumptions, answer-label artifacts, and generated construction labels that do not match empirical diffi- culty. LLM-as-judge methods and model-written evaluations provide useful scalability [13]–[15], but they cannot replace expert clinical validation. A credible benchmark-construction workflow must therefore expose source provenance, reject malformed or cueable items, preserve raw model responses, and distinguish construction-time labels from model-panel empirical calibration. A robust workflow should also quantify residual cueing and evaluation sensitivity rather than equating automated acceptance with validation. We present MS-Exam-Gen, a reproducible framework for con- structing and auditing a source-grounded candidate benchmark for MS-MRI textual knowledge. MS-Exam-Gen indexes an expert corpus, induces an exam-oriented topic taxonomy using seeded retrieval, generates source-grounded MCQs, applies schema, source-evidence, redundancy, and anti-cue audits arXiv:2610.06170v1 [cs.AI] 5 Oct 2026 with a same-family consistency screen, and evaluates models with stable item identifiers. The resulting locked candidate benchmark contains 3,058 MCQs spanning 16 topics and 53 subtopics. We evaluate 13 LLMs to assess discriminative utility and estimate item-level empirical difficulty from model-panel disagreement. This work makes the following novel contributions: 1)A retrieval-grounded benchmark-construction pipeline for transforming MS-MRI documents into an auditable MCQ candidate set with source provenance. 2)An auditable quality-control protocol that records evidence coverage, schema validity, redundancy, anti-cue filtering, acceptance decisions, and raw model-response traces. 3)A heterogeneous 12-endpoint primary calibration showing broad fixed-bank separation, topic-resolved variation, and the need to evaluate generated difficulty labels with post- generation cue and option-order analyses. We frame the resulting resource as an automatically filtered candidate set, not a clinically certified examination, because no human expert item review was available. I. BACKGROUND AND RELATED WORK Medical and Clinical LLM Benchmarks: Medical LLM evaluation commonly uses broad examination or biomedical QA datasets, including MedQA [5], PubMedQA [6], MMLU [7], MedMCQA [16], and MultiMedQA-style evaluations [2]. Recent surveys show that medical benchmarks now span text, EHR, dialogue, imaging, and multimodal tasks [17]. These resources are essential for measuring broad medical knowledge. Some include expert-authored, exam-derived, or context-supported items, but they are not organized around a subspecialty imaging corpus, do not provide item-level MS-MRI source evidence or provenance, and cannot isolate failures in diagnostic criteria, MRI protocol design, longitudinal monitoring, or MS mimics. Radiology and MS Imaging Evaluation: Radiology- specific work increasingly evaluates multimodal founda- tion models, visual question answering, and report-oriented tasks [18], [19]. MS imaging challenges such as MSSEG and longitudinal lesion segmentation provide rigorous image- analysis benchmarks [11], [12]. These resources address important but different questions. They evaluate segmentation, visual understanding, report generation, or general radiology capability, whereas our focus is source-grounded textual rea- soning over MS-MRI guidelines, acquisition recommendations, reporting standards, and differential diagnosis. Automatic Benchmark Construction: LLM-assisted bench- mark generation and LLM-as-judge evaluation can scale item construction [13]–[15], but medical use requires stronger safeguards. LLM judges can exhibit position or scoring biases [20], and multiple-choice evaluations can be sensitive to option ordering and answer-label artifacts [21]. MS-Exam-Gen therefore treats generated MCQs as candidate items, records source evidence and raw responses, audits schema and cueing failures, and operationalizes empirical difficulty using model- panel disagreement as a provisional review-prioritization signal rather than relying on the generator’s nominal construction label. Positioning: Prior work evaluates general medical knowl- edge, radiology images or reports, MS lesion segmentation, or generic LLM-generated benchmarks. MS-Exam-Gen occupies a complementary point in the design space: source-grounded textual MS-MRI knowledge with candidate-item framing, item- level evidence, raw-response preservation, automated quality audits, anti-cue filtering, and empirical difficulty calibration. I. MS-EXAM-GEN FRAMEWORK MS-Exam-Gen constructs a source-grounded MCQ bench- mark candidate set from an expert corpus. Here, “source- grounded” means the generated content is based on retrieved evidence, with the original source and text section recorded. It does not mean that experts have confirmed the clinical claims. The design goal is auditability across source indexing, topic induction, item acceptance/rejection, and model measurement. Figure 1 and Algorithm 1 summarize the workflow. During the dataset construction run, taxonomy induction, item generation, JSON/schema repair, and the same-family consistency screen TABLE I POSITIONING OF MS-EXAM-GEN RELATIVE TO RELATED BENCHMARK FAMILIES. FamilyRepresentative examplesUseful forLimitation for this study MS-MRI text MS-MRI evidence MCQ audit / calib. Broad medical QAMedQA, PubMedQA, MMLU, MedM- CQA, and MultiMedQA [2], [5]–[7], [16] Broad text/exam QANot organized around MS-MRI criteria, protocols, reporting, or mimics. ✗Partial✗ Clinical/multimodal suites MultiMedQA-style and broader medical benchmark suites [2], [17] Clinical QA and scenariosCoverage and provenance vary; usually not a locked MS-MRI source-grounded MCQ bank. ✗VariableVariable Radiology VQA/re- port Radiology foundation-model and report- generation benchmarks [18], [19] Image, VQA, reportsImage/report focused; not designed for guideline- source textual MCQ auditing. ✗PartialVariable MS imaging tasksMSSEG and longitudinal lesion segmen- tation [11], [12] Lesion segmentationDomain-specific imaging tasks, but image labels/- masks rather than textual source evidence. ✗Partial✗ LLM-generated evals Model-written, MT-Bench, G-Eval, LLM- as-judge methods; MCQ-bias analy- ses [13]–[15], [20], [21] Generation and judgingScalable, but source grounding and clinical safe- guards are variable. ✗VariableVariable MS-Exam-GenThis workText MCQ constructionCandidate benchmark only; expert adjudication and full option-order counterbalancing remain future work. ✓ Note: Entries indicate whether a feature is a primary design goal of the family, not whether it is absent from every individual dataset.✓ and✗ denote yes/no; Partial and Variable denote support not specific to source-grounded MS-MRI textual MCQs. 1 Curated MS-MRI Corpus 72 raw files 66 indexed sources 4,289 retrieval chunks Guidelines, criteria, protocols, reviews Source manifest + chunk metadata 2 Seeded Retrieval- Augmented Topic Induction 16 Topics / 53 Subtopics Seed queries for lesion morphology, diagnostic criteria, biomarkers, RIS/CIS, mimics, Checklist-based coverage audit 5 Locked Candidate Benchmark + Empirical Calibration 3,058 MCQs stable item IDs 4 Automated Quality Audits Schema validity Source evidence Redundancy audit Source-leakage phrase checks Option-label consistency Anti-cue checks Malformed / Cueable items rejected Regenerate 3 Source-Grounded MCQ Generation Preprocessing Hard/Synthesis: anti-cue + multi-source retrieval Easy/Medium: focused retrieval Targeted retrieval Evidence pooling De-duplic ation Stem A-D options Answer key Evidence IDs Generation empirical difficulty = # models incorrect Computed after evaluation Same-family consistency screen 12 primary + 1 construction-overlap endpoint 39,754 item-level responses Fig. 1. MS-Exam-Gen construction and audit workflow. Sources are indexed with metadata, sampled through seeded retrieval, organized into an exam-oriented taxonomy, used for source-grounded MCQ generation, and filtered by automated audits and a same-family consistency screen with regeneration of malformed or cueable items. Algorithm 1 additionally details benchmark locking, model evaluation, and empirical calibration. Construction labels are metadata; the locked bank is a candidate benchmark, not a clinical examination. usedopenai/gpt-5-mini; taxonomy and screening calls used temperature 0.0, generation 0.3, and repair 0.1. Because GPT-5 mini is also evaluated, its score is treated as construction- overlap, not independent difficulty validation, and is excluded from primary analyses. Algorithm 1 MS-Exam-Gen construction and calibration Require: Corpus D, seed queries Q, model panel M Ensure:Locked candidate benchmarkBwith panel-dependent empirical difficulty 1: Build a source manifest and vector index from D. 2: Retrieve passages for Q and induce taxonomy T . 3: for each subtopics ∈ T do 4:Retrieve evidence passages fors. 5:Generate Easy/Medium items from focused evidence. 6:Generate Hard and multi-source synthesis items with anti-cue con- straints. 7:Repair malformed JSON/schema outputs; reject unrepaired, unsup- ported, duplicated, or cueable items. 8:Screen answerability and key consistency against stored evidence. 9: end for 10: Lock accepted items with stable item IDs and source provenance. 11: for each modelm ∈ M do 12:Evaluate MCQs using a standardized A–D answer prompt. 13:Store raw response, parsed option, parse status, and correctness by item ID. 14: end for 15:Compute model, topic, construction-label, and empirical-difficulty sum- maries. A. Corpus Indexing and Topic Induction To make source scope and provenance auditable, letDdenote a curated MS-MRI corpus of consensus guidelines, diagnostic criteria, reviews, protocols, and educational sources selected for authority, recency, and concept coverage. The source manifest records inclusion status, document identifiers, metadata, and chunk counts. Documents are segmented into 1,200-character chunks with 200-character overlap; each chunk stores source path, page, start offset, ordinal, and a content-hash identifier before embedding withtext-embedding-3-largeinto Chroma. MS-Exam-Gen uses seeded retrieval-augmented topic induction: 18 seed queries target lesion location, McDonald criteria, central vein sign, paramagnetic rim lesions, RIS/CIS, NMOSD/MOGAD, pediatric/older-adult imaging, treatment surveillance, reporting, and mimics. Up to four passages per seed query, capped at 60 distinct passages, are summarized into an exam-oriented taxonomyT = (t j ,S j ) T j=1 , and a checklist-based coverage audit comparesTwith predefined seed concepts; this audit does not guarantee exhaustive MS- MRI topic coverage. Character-based chunking may also split tables, lists, or neighboring criteria. B. Source-Grounded MCQ Generation To tie every candidate to retrieved evidence, for each subtopic, the generator retrieves supporting passages and emits a four-option single-best-answer MCQ with topic, subtopic, generated construction label, question type, correct option, explanation, and source evidence. Standard Easy/Medium/Hard generation uses focused retrieval withk = 3chunks; the fourth stream is reported as a multi-source synthesis stratum, retrievingk = 30chunks and keeping at most two chunks per source and ten snippets. This stratum is stored under the generator label “Very Hard” in the artifact, but we avoid treating that label as validated difficulty. Options are shuffled after generation. Easy/Medium items emphasize concept recognition and interpretation, while Hard and synthesis prompts require non-recall stems, balanced option lengths, no “most complete answer” cues, and plausible near-miss distractors. Construction labels are metadata only and are calibrated empirically after model evaluation. C. Automated Quality Auditing To reject malformed or unsupported candidates, each item is passed through deterministic audits before evaluation. The schema audit requires a stem, exactly four options A–D, one correct option, topic/subtopic metadata, explanation, and source evidence. Text audits flag hidden-passage phrasing, answer- label references, explanation-option contradictions after option shuffling, exact duplicates, near duplicates, and superficial trivia. Generation-time near-duplicate filters use normalized string similarity≥ 0.90; the final audit reports remaining near- duplicate flags for transparency. Semantic duplicate checks are additionally applied to multi-source synthesis candidates, where redundancy risk is higher, using embedding cosine similarity ≥ 0.90. To reduce superficial answer cues, the difficulty-quality audit rejects option-length cues, excessive correct-answer completeness, weak distractors, and recall-style stems. Hard thresholds are: stem≥ 18words, options 8–60 words, correct- option length advantage≤ 4words, and correct/distractor length ratio≤ 1.15. Multi-source synthesis thresholds are stricter: stem≥ 35words, options 14–70 words, advantage ≤ 2words, ratio≤ 1.08, and no uniquely longest correct option. These thresholds were fixed before locking the final benchmark; they are construction heuristics for removing obvious cueing artifacts, not psychometrically validated item-quality criteria. The same-family consistency screen then receives the MCQ JSON plus stored gold context, or, when stored context is unavailable, a freshk = 3retrieval context saved with the verdict, and checks answerability, key consistency, explanation consistency with stored evidence, and uniqueness. Any major issue rejects the item; all verdicts, audit metrics, and rejection reasons are retained. Because automated judging used the same model family as generation, passing this filter is interpreted as a consistency screen conditioned on stored evidence, not evidence of clinical correctness. Residual cueing uses a standardized balanced logistic selector (C = 0.5) with leave-one-topic-out folds, comparing structural option features alone and with TF–IDF stem overlap. The identical pipeline is fitted to both banks. D. Evaluation and Empirical Difficulty Each locked item receives a stable identifier, and all predictions are joined by item ID rather than file position. Model outputs are parsed with conservative A–D label patterns; unparseable or exception rows are retried up to a fixed retry limit and not cached as valid predictions. Final aggregation retains all item-model rows; parsed responses are scored, and unresolved failures are recorded as parse failures rather than silently scored as correct. For modelmand benchmarkB, accuracy is Acc(m,B) = 1 |B| X q i ∈B ⊮ˆa m,i = a i , whereˆa m,i is the parsed model answer anda i is the gold option. We report overall, topic-level, generated construction-label, and empirical-difficulty results. Empirical difficulty is the number of 12 non-construction-overlap endpoints that miss an item. It is a panel-dependent calibration signal, not psychometric validation, and may reflect ambiguity, source disagreement, prompt sensitivity, or model-panel composition. Uncertainty uses 4,000-replicate hierarchical bootstrapping over 16 construction batches and then items; a parallel analysis over 13 accuracy-independent semantic parents tests taxonomy- overlap sensitivity. Additional analyses test key disagreement TABLE I CONSTRUCTION AND AUDIT TRAIL FOR THE LOCKED BHI CANDIDATE BENCHMARK. StageInput Accepted/flaggedLocked Standard Easy/Medium650 each646 each 646 each Standard Hard6506430 Refreshed Hard/synthesis900 each883 each 883 each Legacy Hard/synthesis pool1,2931,217 flagged0 Standard Hard and the flagged legacy pool were excluded and replaced by the refreshed strata. TABLE I FINAL BENCHMARK AND EVALUATION SUMMARY. ComponentValue Corpus/taxonomy66 sources; 4,289 chunks; 16 topics; 53 subtopics Locked bank3,058 MCQs: Easy/Medium 646 each; Hard/synthesis 883 each Final audit0 schema errors; 0 exact duplicates; 54 retained near-duplicate transparency flags Evaluation12 primary endpoints plus GPT-5 mini sensitivity; 36,696 primary/39,754 total responses; 0 parse failures and option order. The resulting review queues prioritize adjudication, and the stratified pilot does not estimate full- bank order-effect prevalence. IV. EXPERIMENTAL SETUP Corpus: The curated, non-exhaustive MS-MRI corpus contains 72 raw files; after excluding non-content frontmat- ter, the vectorstore contains 66 indexed sources and 4,289 chunks. Sources cover the 2017 McDonald criteria [8], 2021 MAGNIMS-CMSC-NAIMS MRI recommendations [9], 2024 McDonald revisions [10], MS mimics, CVS/PRL, RIS/CIS, NMOSD/MOGAD, spinal/optic nerve imaging, pediatric MS, treatment surveillance, and reporting. Copyrighted source PDFs are not redistributed. Benchmark Construction: The induced taxonomy contains 16 topics and 53 exam-relevant subtopics. Standard generation produced 1,950 Easy/Medium/Hard candidates; the same-family consistency screen accepted 1,935 (646 Easy, 646 Medium, 643 Hard). An anti-cue audit of a separate legacy Hard/multi- source synthesis pool flagged 1,217/1,293 cueable or weak items, so those items were excluded and replaced by refreshed streams. The refreshed Hard and multi-source synthesis streams each generated 900 candidates and accepted 883. The final locked benchmark contains 3,058 schema-valid MCQs: 646 Easy, 646 Medium, 883 Hard, and 883 multi-source synthesis items. For the construction-hardening audit, the 2,585-item legacy pre-refresh bank was also evaluated on the same 13-model panel. Because the legacy and locked banks are disjoint, this is a process comparison rather than an item- matched causal estimate. The final audit found 0 schema errors, 0 exact duplicates, 54 post-hoc near-duplicate transparency flags retained because they did not meet exclusion criteria, 0 explanation-label contradictions, 0 explanation option-label references, and 0 difficulty-quality failures. Model Panel: We evaluated 13 LLMs: Gemini 2.5 Flash, GPT-5 mini, Llama 4 Maverick, DeepSeek Chat, Claude Haiku 4.5, GPT-4o mini, Gemini 2.5 Flash Lite, GPT-5 nano, Qwen3 14B, Qwen3 8B, Llama 3.1 8B, Mistral, and Phi-4 mini. GPT-5 mini is marked as construction-overlap and is not interpreted as independent benchmark validation; the other 12 endpoints form the primary panel (36,696 responses). Eight models were accessed through an OpenRouter-compatible interface and five through Ollama; this interface detail is reported for reproducibility and is not used to group results. All models used a standardized A–D-only prompt; temperature was set to zero where supported. Reproducibility Controls: The evaluation script stores item ID, source index, topic, subtopic, construction label, raw response, parsed answer, parse status, and correctness for every model-item pair. Interrupted runs resume from parsed checkpoints, and aggregation joins predictions by stable item ID. No unresolved unknown, unparseable, or exception rows remained in the final results. Run records preserve provider identifiers, runtime tags, decoding settings, checkpoint metadata, and model allowlists; unavailable release/quantization fields are marked “not recorded.” A deterministic 100-item pilot stratified by semantic parent, construction stratum, and historical disagreement used four cyclic option orders with 11 available endpoints (4,400 parsed responses); full-bank coun- terbalancing remains future validation. Candidate, acceptance, rejection, and locked-retention counts are preserved. V. RESULTS Structure only Structure + stem Correct key uniquely longest 0 20 40 60 80 Accuracy / prevalence (%) 65.7 64.4 68.1 36.4 43.8 6.6 25% chance A. Shallow-Cue Audit: Legacy vs Refreshed Legacy strataRefreshed locked strata 0123 4-67-9 10-12 Primary endpoints incorrect per item 0 10 20 30 Items (%) 25.3 27.7 13.0 8.4 13.9 7.3 4.3 High-disagreement subset 4 missed: 25.5% 7 missed: 11.6% B. Empirical Difficulty Across 12 Endpoints Fig. 2. Benchmark auditing and empirical calibration. (A) The selector was fitted to the replaced legacy and locked Hard/synthesis strata. The dashed chance line applies only to the first two accuracy metrics; the third is item prevalence. Lower key predictability supports reduced measurable cueing. (B) Primary empirical difficulty is the number of 12 non-construction-overlap endpoints incorrect per item. Overall Endpoint Separation: Table IV shows broad performance separation across the 12 primary endpoints, from Gemini 2.5 Flash at 89.70% (2,743/3,058) to Phi-4 mini at 46.89% (1,434/3,058). The intermediate range is informative: Qwen3 14B (83.32%) approached GPT-5 nano (81.20%), while Qwen3 8B, Llama 3.1 8B, and Mistral occupied lower but still discriminative performance bands. TABLE IV PRIMARY CONSTRUCTION-LABEL-RESOLVED MODEL-PANEL RESULTS ON THE LOCKED 3,058-ITEM BENCHMARK. ModelEasy Med. Hard Synth. Avg. 95% CI Gemini 2.5 Flash91.088.488.191.389.7 86.0–92.9 Llama 4 Maverick88.4 87.2 87.890.4 88.5 85.0–91.7 DeepSeek Chat87.8 86.8 86.287.0 86.9 83.1–90.2 Claude Haiku 4.586.7 85.1 81.189.5 85.6 81.3–89.4 Gemini 2.5 Flash Lite 85.1 81.9 84.985.2 84.4 79.2–88.9 Qwen3 14B83.3 82.5 83.583.8 83.3 78.3–87.8 GPT-5 nano83.7 84.5 76.881.3 81.2 76.5–85.5 GPT-4o mini84.1 82.5 80.576.8 80.6 75.2–85.4 Qwen3 8B80.3 75.4 75.879.6 77.8 72.5–82.5 Llama 3.1 8B78.5 73.7 71.370.4 73.1 68.0–77.8 Mistral73.5 69.7 74.269.8 71.8 65.2–77.4 Phi-4 mini53.3 47.8 41.946.5 46.9 43.8–49.9 Note: Values are percentages. “Synth.” denotes the multi-source synthesis stratum. CIs use 4,000 hierarchical bootstrap replicates over 16 construction batches and then items; the parallel 13-parent analysis gave similar mean interval width (8.26 vs. 8.69 points; maximum width difference 1.24). Green cells mark column maxima. Generated Versus Empirical Difficulty: Generated con- struction labels are metadata, not validated psychometric strata. Figure 2A directly tests shallow cues in the disjoint legacy and locked Hard/synthesis strata: structure-only key prediction fell from 65.7% to 36.4%, structure-plus-stem prediction from 64.4% to 43.8%, and uniquely-longest-key prevalence from 68.1% to 6.6%. This supports reduced measurable cueing under the refreshed process Hard, not synthesis, remained empirically lowest-accuracy. Excluding GPT-5 mini yielded the refreshed pattern of 81.31%, 78.79%, 77.67%, and 79.29% for Easy, Medium, Hard, and synthesis. Empirical Difficulty Distribution: Figure 2B reports the item-level disagreement distribution. Most items showed low panel disagreement: 1,622/3,058 (53.0%) were missed by zero or one primary endpoint. However, 781 items (25.5%) were missed by at least four and 355 (11.6%) by at least seven, forming a high-disagreement subset for error analysis and prompt-sensitivity studies. Leaving out any endpoint retained miss-count correlation≥ 0.934and≥ 96.2%agreement on the four-endpoint flag. Topic-Level Variation: The lowest-accuracy item groups were not simply the most technically advanced imaging topics. Figure 3 displays the complete result grid. Accuracy- independent parent means were lowest for RIS/CIS (67.9%), mimics/differential diagnosis (72.5%), diagnostic criteria (74.3%), and reporting/acquisition/safety (74.5%). Quality and Audit Outcomes: The locked benchmark is schema-valid and auditable, and the compiled evaluation contains 39,754 total model-item responses (36,696 from the 12 primary endpoints) with zero unresolved parse failures. The key negative audit result is methodological: a legacy Hard/multi-source synthesis pool contained many cueable items, and 1,217 were flagged before refreshed generation. This audit trail strengthens the locked candidate benchmark by showing that multiple construction-quality risks were tested rather than assumed absent. The locked full bank nevertheless remained Gemini 2.5 FlashLlama 4 MaverickDeepSeek ChatClaude Haiku 4.5Gemini 2.5 Flash LiteQwen3 14BGPT-5 nanoGPT-4o miniQwen3 8BLlama 3.1 8BMistralPhi-4 miniMean RIS/CIS presentations (n=184) MS mimics / misdiagnosis deep dive (n=182) Differential diagnosis / MS mimics (n=176) Diagnostic criteria (n=216) Reporting standards (n=180) Structured acquisition / reporting / safety (n=205) Pediatric MS imaging (n=163) MRI basics for MS (n=164) Longitudinal follow-up (n=162) Lesion morphology / distribution (n=287) Spinal / optic nerve imaging (n=194) Diagnostic biomarkers (mixed label) (n=269) Treatment monitoring / safety surveillance (n=150) Imaging treatment monitoring / prognosis (n=212) Quantitative / AI imaging literacy (n=122) Advanced biomarkers / atrophy (n=192) 77.779.976.176.174.568.580.465.266.355.457.637.567.9 77.574.773.672.565.469.269.869.859.966.554.440.166.1 92.688.186.484.182.484.182.484.175.073.364.252.379.1 80.181.087.576.481.081.067.182.472.262.074.146.374.3 85.680.076.171.762.865.061.155.660.654.440.637.262.5 94.694.191.293.290.290.788.882.091.779.075.149.385.0 92.689.682.880.484.078.574.276.169.964.467.544.875.4 82.385.483.579.376.276.880.575.676.870.170.751.275.7 88.988.384.085.283.382.182.782.780.971.072.850.079.3 92.392.087.588.285.786.179.480.172.176.369.045.379.5 94.896.496.495.991.885.687.683.585.677.378.444.884.8 94.191.891.890.093.793.784.090.783.679.983.650.985.7 90.793.388.090.792.784.090.780.786.783.382.742.783.8 97.298.695.393.993.992.990.691.590.184.088.752.889.1 94.394.391.897.593.492.690.286.990.281.185.252.587.5 96.986.593.292.294.894.892.296.484.487.581.252.187.7 5060708090 Accuracy across 12 primary endpoints (%) Fig. 3. Topic-level accuracy across 12 primary endpoints. All 16 submitted topic labels and item counts are retained. Blue brackets mark three semantically overlapping label pairs and report their accuracy-independent parent means. Cells report percentages, and the final column reports each original topic’s mean. above chance for structure-only (38.3%) and structure-plus- stem (41.2%) key prediction, so it is not cue-free. Key-Risk and Option-Order Sensitivities: The 7/12- majority non-key queue contains 202 items. A stricter queue contains 55 items for which five prespecified, non- overlap model families (Gemini 2.5 Flash, Llama 4 Maverick, DeepSeek Chat, Claude Haiku 4.5, and GPT-4o mini) selected the same non-key option. Removing the queues changes scores by 1.67–5.02 and 0.36–1.64 points, respectively, without erasing broad separation. The option-order pilot retains coarse ordering (ρ = 0.991), but mean absolute score change is 2.82 points (maximum 9.0), semantic stability is 56.9%, and 9/11 endpoints show position effects. VI. DISCUSSION What the Benchmark Measures: The results should be interpreted as performance on source-grounded MS-MRI textual MCQs, not direct image interpretation or clinical decision authority. This scope is deliberate: it isolates guideline knowledge, criteria application, reporting language, mimic recognition, and monitoring concepts. Strong performance therefore does not imply radiologist-level visual reasoning; lower performance indicates difficulty on source-derived MCQs covering MS-MRI criteria, surveillance recommendations, or differential diagnosis. Without human or free-response baselines, scores are not measures of clinical competence or human-relative difficulty. Why Construction Labels Need Calibration: The main methodological lesson is that generation-time difficulty is not sufficient. In the legacy bank, multi-source synthesis items were the easiest generated stratum; after refreshed anti-cue generation, Hard and synthesis accuracies fell by 4.6 and 6.4 percentage points, respectively. The disjoint-bank score changes are descriptive The identical shallow-cue diagnostic applied to both banks more directly supports reduced measurable cueing. This supports the hardening audit but not a psychometric ordering claim: the refreshed Hard stream remained the lowest- accuracy subset, whereas the multi-source synthesis stratum was not consistently harder. We therefore report generated construction labels as metadata and use the number of models incorrect per item as the empirical calibration signal. Why Retain the Synthesis Stratum: The synthesis stratum is retained because it captures multi-source construction rather than validated difficulty. These items are source-grounded, schema-valid, and cover synthesis topics, while their higher- than-Hard accuracy demonstrates why the released artifact should pair construction labels with empirical disagreement annotations. Source count differs from Hard (7.96 vs. 2.24 per item) but is weakly associated with accuracy (ρ = 0.054). Value of a Heterogeneous Model Panel: Including models with varied performance profiles reduced ceiling effects: adding GPT-5 nano, Gemini 2.5 Flash Lite, Qwen3 14B/8B, Llama 3.1 8B, Mistral, and Phi-4 mini widened the observed range to 46.9–89.7% and exposed more discriminative items. This calibration panel helps identify performance bands and lower- accuracy content groups rather than only confirming that strong models answer canonical guideline questions. Seven API-served endpoints retain a 9.09-point range. Interpretation of Lower-Accuracy Item Groups: The lower-accuracy item groups cover substantively important con- tent areas. Reporting standards require structured language; MS mimics require exclusionary reasoning; RIS/CIS presentations require handling incomplete clinical evolution; and diagnostic criteria require applying dissemination rules, biomarkers, and exceptions without overgeneralizing. All 16 labels are retained, while three overlapping pairs and their parent means are marked. These descriptive differences prioritize review rather than establish intrinsic clinical difficulty. Audit Lessons for Automatic Benchmark Construction: The construction process illustrates why schema validation alone is insufficient. The initial Hard and synthesis pools contained formally valid but cueable items, and the complete legacy rerun showed inflated accuracy on the synthesis stratum. Refreshed prompts and anti-cue filters improved this problem, yet empirical results still showed that stricter prompts do not guarantee psychometric ordering. Residual cues, key-risk queues, and order effects show that automated screens do not certify validity. A robust pipeline must preserve rejected items, report acceptance and rejection rates, and evaluate the locked bank on a heterogeneous model panel before making difficulty claims. Threats to Validity: The benchmark is automatically filtered and source-grounded but not adjudicated by human MS imaging experts, so answer-key errors, ambiguous distractors, or clinically weak items may remain. The present evidence supports use as an auditable candidate item bank, not as a clinically validated examination; expert adjudication is required before educational or clinical deployment. We also did not compare LLM performance with MS trainees, radiologists, neurologists, or other human readers; model accuracy should therefore not be interpreted relative to human expertise. The corpus is broad but not exhaustive. The evaluated models were accessed through different provider/runtime stacks, so absolute comparisons should be read as benchmark observations rather than controlled architectural claims. Finally, although options were shuffled after generation and label/explanation artifacts were audited, locked evaluations were not systematically counterbalanced across option order. Prospective Validation Roadmap: The next version should add expert adjudication across topics and empirical-difficulty bins, option-order counterbalancing, and a domain-transfer pilot in another neuroimaging subspecialty. Future releases should separate a stable core set, a high-disagreement stress-test set, and refreshed items tracking evolving criteria and biomarkers. VII. CONCLUSION We introduced MS-Exam-Gen, a reproducible framework for constructing and auditing source-grounded subspecialty MCQ benchmarks, demonstrated on MS-MRI knowledge. The final candidate benchmark contains 3,058 items across 16 topics and 53 subtopics, and evaluation on 13 models with 12 non-construction-overlap primary endpoints shows a broad accuracy range from 46.9% to 89.7%. The results support three conclusions: an auditable subspecialty con- struction workflow can separate endpoint performance on a fixed bank; topic-level analysis identifies lower-accuracy item groups for targeted review; and generated construction labels must be calibrated empirically rather than accepted at face value. The current artifact is intended for model auditing and benchmark-construction research, not for clinical education or credentialing without expert adjudication. Future work will add expert adjudication, full option-order counterbalancing, domain- transfer experiments, and stronger prospective validation for clinical education and model auditing. REFERENCES [1] M. Moor et al., “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, no. 7956, p. 259–265, Apr. 2023. [2]K. Singhal et al., “Large language models encode clinical knowledge,” Nature, vol. 620, no. 7972, p. 172–180, Jul. 2023. [3]A. J. Thirunavukarasu et al., “Large language models in medicine,” Nature Medicine, vol. 29, no. 8, p. 1930–1940, Jul. 2023. [4] H. Nori et al., “Capabilities of GPT-4 on medical challenge problems,” 2023. [5]D. Jin et al., “What disease does this patient have? a large-scale open domain question answering dataset from medical exams,” Applied Sciences, vol. 11, no. 14, p. 6421, Jul. 2021. [6] Q. Jin et al., “PubMedQA: A dataset for biomedical research question answering,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China: Association for Computational Linguistics, Nov. 2019, p. 2567–2577. [7]D. Hendrycks et al., “Measuring massive multitask language understand- ing,” in International Conference on Learning Representations, 2021. [8]A. J. Thompson et al., “Diagnosis of multiple sclerosis: 2017 revisions of the McDonald criteria,” The Lancet Neurology, vol. 17, no. 2, p. 162–173, Feb. 2018. [9]M. P. Wattjes et al., “2021 MAGNIMS–CMSC–NAIMS consensus recommendations on the use of MRI in patients with multiple sclerosis,” The Lancet Neurology, vol. 20, no. 8, p. 653–670, Aug. 2021. [10]X. Montalban et al., “Diagnosis of multiple sclerosis: 2024 revisions of the McDonald criteria,” The Lancet Neurology, vol. 24, no. 10, p. 850–865, Oct. 2025. [11]O. Commowick et al., “MSSEG-2 challenge proceedings: Multiple sclerosis new lesions segmentation challenge using a data management and processing infrastructure,” in MICCAI 2021 - 24th International Conference on Medical Image Computing and Computer Assisted Intervention, Strasbourg, France, Sep. 2021, 126 pages. [12]A. Carass et al., “Longitudinal multiple sclerosis lesion segmentation: Resource and challenge,” NeuroImage, vol. 148, p. 77–102, Mar. 2017. [13]E. Perez et al., “Discovering language model behaviors with model- written evaluations,” 2022. [14]L. Zheng et al., “Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena,” in Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. [15] Y. Liu et al., “G-eval: NLG evaluation using GPT-4 with better human alignment,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.Singapore: Association for Computational Linguistics, Dec. 2023, p. 2511–2522. [16]A. Pal et al., “MedMCQA: A large-scale multi-subject multi-choice dataset for medical domain question answering,” in Proceedings of the Conference on Health, Inference, and Learning, ser. Proceedings of Machine Learning Research, vol. 174.PMLR, 07–08 Apr 2022, p. 248–260. [17] L. K. Q. Yan et al., “Large language model benchmarks in medical tasks,” 2024. [18]C. Wu et al., “Towards generalist foundation model for radiology by leveraging web-scale 2D&3D medical data,” 2023. [19] C. Liu et al., “Argus: Benchmarking and enhancing vision-language models for 3D radiology report generation,” 2024. [20]L. Shi et al., “Judging the judges: A systematic study of position bias in LLM-as-a-Judge,” 2024, AACL-IJCNLP 2025. [21] C. Zheng et al., “Large language models are not robust multiple choice selectors,” 2023, ICLR 2024 Spotlight.