Paper deep dive
Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context
Nawar S. Alseelawi, Mustafa S. Aljumaily
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/20/2026, 4:11:41 AM
Summary
The paper introduces Mizan, a national benchmark for evaluating Large Language Models (LLMs) on Iraqi Arabic and the Iraqi civic context. Unlike existing benchmarks focused on Modern Standard Arabic (MSA), Mizan includes an Iraqi track with six axes: dialect comprehension, dialect generation, bidirectional translation, Iraq-specific knowledge, official-document extraction, and safety. A pilot study of 27 systems reveals that MSA benchmarks are saturated, while the Iraqi track discriminates effectively, showing a consistent 14-18 point gap. Key findings include a 'document wall' where all models score poorly on official document extraction, an 'Arabic specialization paradox' where specialized Arabic models underperform generalists on Iraqi dialects, and a 'dialect-triggered over-refusal' where safety-hardened models incorrectly refuse innocuous Iraqi dialect queries. The platform enforces strict integrity protocols including immutable snapshots and human publication gates.
Entities (10)
Relation Signals (11)
Mizan → evaluates → Iraqi Arabic
confidence 98% · Mizan... Iraq's national benchmark for evaluating LLMs on Iraqi Arabic and the Iraqi civic context
Mizan → hastrack → MSA baseline track
confidence 95% · an MSA baseline track paired with an Iraqi track
Mizan → hastrack → Iraqi track
confidence 95% · an MSA baseline track paired with an Iraqi track
Iraqi track → discriminates → Models
confidence 92% · The MSA track saturates while the Iraqi track discriminates, with a consistent 14-18-point per-model gap
MSA benchmarks → saturates → Frontier Models
confidence 92% · The MSA track saturates while the Iraqi track discriminates
Safety-hardened models → exhibits → Dialect-triggered over-refusal
confidence 90% · the safety-hardened tier of the newest model family deterministically refuses innocuous dialect-comprehension items
Mizan → hasaxis → safety
confidence 90% · six axes (... and safety)
Mizan → hasaxis → dialect comprehension
confidence 90% · six axes (dialect comprehension, dialect generation, bidirectional MSA-Iraqi translation, Iraq-specific knowledge, official-document field extraction, and safety)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Arabic large-language-model (LLM) evaluation has matured around Modern Standard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Leaderboard (OALL), HELM Arabic, and BALSAM rank models across dozens of MSA tasks, and frontier systems increasingly saturate them. Dialectal Arabic, the language Iraqis actually speak, remains nearly invisible to this infrastructure. We introduce Mizan ("the balance"), Iraq's national benchmark for evaluating LLMs on Iraqi Arabic and the Iraqi civic context: an MSA baseline track paired with an Iraqi track across six axes (dialect comprehension, dialect generation, bidirectional MSA-Iraqi translation, Iraq-specific knowledge, official-document field extraction, and safety), built from 340 originally authored, dually reviewed items with statistically audited answer positions and Wilson intervals on every published score. A pilot evaluation of 27 systems, spanning closed frontier models three days after release, open weights across size tiers, and an Arabic trio of commercial, open-specialized, and sovereign systems, yields four findings. The MSA track saturates while the Iraqi track discriminates, with a consistent 14-18-point per-model gap and statistically tied leaders. Official-document extraction confines every system to 32-56. Arabic-focused specialization behaves as MSA specialization: two dedicated Arabic models score below a size-matched generalist on the Iraqi track. And the safety-hardened tier of the newest model family deterministically refuses innocuous dialect-comprehension items as policy violations, an over-refusal mode invisible to MSA benchmarks. The platform enforces an integrity protocol of immutable snapshots, verification certificates, a human publication gate, and public retraction, all exercised during this study. Code and the public development set accompany the paper.
Tags
Links
- Source: https://arxiv.org/abs/2609.13980v1
- Canonical: https://arxiv.org/abs/2609.13980v1
Trouble viewing inline? Open PDF directly →
Full Text
40,932 characters extracted from source content.
Expand or collapse full text
Mizan: A National Benchmark for Evaluating Large Language Models on Iraqi Arabic and the Iraqi Civic Context Nawar S. Alseelawi 1 and Mustafa S. Aljumaily 2 1 University of Misan, Amarah, Iraq 2 Missan Oil Company, Amarah, Iraq Members of the National Team for the Iraqi Large Language Model, Prime Minister’s Office, Baghdad, Iraq Abstract Arabic large-language-model (LLM) evalu- ation has matured around Modern Stan- dard Arabic (MSA): aggregated leaderboards such as the Open Arabic LLM Leaderboard (OALL), HELM Arabic, and BALSAM rank models across dozens of MSA tasks, and fron- tier systems increasingly saturate them. Di- alectal Arabic - the language Iraqis actually speak - remains nearly invisible to this in- frastructure. We introduce Mizan (”the bal- ance”), Iraq’s national benchmark for evaluat- ing LLMs on Iraqi Arabic and the Iraqi civic context: an MSA baseline track paired with an Iraqi track across six axes - dialect com- prehension, dialect generation, bidirectional MSA-Iraqi translation, Iraq-specific knowl- edge, official-document field extraction, and safety - built from 340 originally authored, dually reviewed items with statistically au- dited answer positions and Wilson intervals on every published score. A pilot evaluation of 27 systems - closed frontier models three days after release, open weights across size tiers, and an Arabic trio spanning commer- cial, open-specialized, and sovereign systems - yields four findings. The MSA track saturates while the Iraqi track discriminates, with a con- sistent 14-18-point per-model gap and statis- tically tied leaders. Official-document extrac- tion confines every system to 32-56. Arabic- focused specialization behaves as MSA spe- cialization:two dedicated Arabic models score below a size-matched generalist on the Iraqi track. And the safety-hardened tier of the newest model family deterministically re- fuses innocuous dialect-comprehension items as policy violations - an over-refusal mode in- visible to MSA benchmarks. The platform enforces an integrity protocol of immutable snapshots, verification certificates, a human publication gate, and public retraction, all ex- ercised during this study. Code and the public development set accompany the paper. Keywords: LLM Evaluation, Dialectal Arabic, Low-Resource NLP, Over-Refusal, Document Extraction, Benchmark Integrity. 1 Introduction Evaluation infrastructure decides what progress means.For Arabic, that infrastructure has consolidated around Modern Standard Arabic: ArabicMMLU and AlGhafa established rigorous MSA test sets; the Open Arabic LLM Leader- board aggregates them at scale; HELM Arabic ex- tends a transparent multi-benchmark methodology to Arabic; and BALSAM (Al-Matham et al., 2025) pools tens of thousands of test questions across dozens of task categories under a regional con- sortium. This consolidation succeeded - to the point of self-obsolescence. On our MSA track, contemporary systems cluster at or near a per- fect score, the newest frontier model matches them within 72 hours of its public release, and the two strongest systems cannot be separated at all: they tie, and swap ranks between independent evalu- ation rounds. A benchmark that no longer sep- arates systems no longer measures them. What saturation conceals is the language people speak. Iraqi Arabic serves tens of millions of speakers across registers that diverge from MSA in phonol- ogy, morphology, lexicon, and pragmatics - diver- gences deep enough to produce systematic false friends (Iraqi hwaya, ”many”, against MSA hi- waya, ”hobby”) and to defeat literal translation in both directions. Yet no comprehensive LLM eval- uation framework has existed for it. The dialect re- sources the field possesses - Multi-Arabic Dialect Applications and Resources (MADAR), Nuanced Arabic Dialect Identification (NADI) - belong to an earlier paradigm: identification and clas- sification corpora built for supervised NLP. Re- cent dialect-aware evaluation, most notably Ara- bic Dialect and Cultural Evaluation (AraDiCE), has begun to close the distance for Arabic dialects arXiv:2609.13980v1 [cs.CL] 12 Sep 2026 broadly - through machine-translated, post-edited derivatives of existing benchmarks, covering Lev- antine and Egyptian varieties and the cultural con- texts of the Gulf, Egypt, and the Levant. Iraqi Ara- bic, with its internal regional variation and its dis- tinctive civic register, appears in none of it. The stakes are not academic. States are adopting lan- guage models inside public administration, and Iraq has constituted a national team, by prime- ministerial order, to build sovereign Arabic-and- Iraqi language capability. A model proposed for Iraqi state use must read an official letter - issu- ing authority, reference number, date, addressee, subject, required action - and must navigate Iraqi social sensitivities without either enabling harm or refusing legitimate civic questions. No exist- ing benchmark measures any of this, in any di- alect, anywhere.A national evaluation instru- ment is therefore not a convenience but a precon- dition of informed procurement: whatever model is offered to the Iraqi state should be measured on the national scale first. Mizan (”the balance” or ”the scale”) is that instrument, and its de- sign answers each failure above by construction. Two tracks of equal standing - an MSA base- line the international literature can read, and an Iraqi track as the discriminator nothing else pro- vides. Six axes spanning comprehension, gener- ation, bidirectional MSA-Iraqi translation, Iraq- specific knowledge, official-document field ex- traction, and safety in Iraqi context. Every item originally authored by native speakers and dually reviewed - nothing translated from foreign bench- marks, because translated items measure transla- tionese rather than dialect competence. Answer positions balanced and audited. Generative axes scored by human judges with numbered rubrics. The accompanying platform enforces an integrity protocol - uniform conditions with a logged retry ladder, per-item audit records, dated immutable snapshots, SHA-256 certificates, a human pub- lication gate, and public retraction - which this very study exercised: diagnostic charts exposed two technically wounded runs, both were retracted in public view, both models were re-evaluated cleanly, and the full board was then re-verified end to end. We report a pilot study of 27 sys- tems evaluated under identical conditions on the 340-item bank: closed frontier models (including GPT-6 Astra, evaluated three days after release), open weights across size tiers, and an Arabic trio spanning a commercial API system, an open spe- cialized model, and a sovereign state model. Four regularities organize the results. The saturation- discrimination split: MSA compresses at the ceil- ing while Iraqi scores spread across more than twenty points, each model carrying a consistent 14-18-point internal gap.The document wall: official-document extraction confines every sys- tem - newest frontier included - to between 32 and 56. The specialization law: models marketed as Arabic-specialized behave as MSA-specialized - two dedicated Arabic models, one open and one sovereign, score below a size-matched generalist on the Iraqi track, while a third exceeds its own generalist sibling yet remains well below the fron- tier. And an over-refusal mode invisible to ev- ery MSA benchmark: the safety-hardened tier of the newest model family deterministically refuses innocuous Iraqi comprehension items - questions asking the meaning of everyday words - classi- fying them as policy-violating content, while an- swering the entire MSA track without hesitation. This paper makes five contributions: Mizan, to our knowledge the first comprehensive, originally- authored evaluation framework dedicated to Iraqi Arabic and the Iraqi civic context, with a dual- track design and six evaluation axes; an official- document extraction axis unmeasured by any ex- isting benchmark, directly tied to sovereign use; an open evaluation platform with an enforced, practice-tested integrity protocol: snapshots, cer- tificates, a human gate, public retraction, and full- board re-verification with run-to-run agreement reporting; a 27-model empirical study, statistically qualified throughout, establishing MSA satura- tion, the per-model dialect gap, the universal doc- ument wall, and the Arabic-specialization para- dox; the first documented case of dialect-triggered safety over-refusal in a frontier system, with of- ficial API evidence - and the public development set, guidelines, and code, released to seed dialect- aware evaluation beyond Iraq. Section 2 situates Mizan among Arabic evaluation efforts; Sections 3-4 present the benchmark and platform; Sections 5-6 report the pilot study; Sections 7-10 analyze errors, discuss implications, and state limitations and the v1.0 roadmap. 2 Related Work Arabic LLM evaluation has consolidated along three lines: MSA benchmarks and their aggre- gated leaderboards, dialect resources from the classification era, and a nascent turn toward dialect-aware LLM evaluation. Mizan draws on all three and departs from each. 2.1 MSA benchmarks and aggregated leaderboards ArabicMMLU (Koto et al., 2024) and AlGhafa (Almazrouei et al., 2023) established rigorous multiple-choice test sets for Arabic, with Ara- bicMMLU in particular built natively rather than translated - a design stance we adopt and extend. The Open Arabic LLM Leaderboard (Elfilali et al., 2024) aggregates such benchmarks at community scale for open models. Holistic Evaluation of Lan- guage Models (HELM) Arabic (Stanford CRFM and Arabic.AI, 2025) extends the HELM frame- work’s transparent, reproducible multi-benchmark methodology to Arabic across seven established test sets, with an Enterprise variant addressing fi- nancial and legal tasks. BALSAM (Al-Matham et al., 2025) pools on the order of fifty thousand test questions across sixty-seven task categories under a MENA-wide consortium, with blind test sets and a standard submission interface. This in- frastructure is mature, rigorous, and MSA-bound: none of it observes dialect, local knowledge, or the civic document register. Our results supply an empirical corollary: its ceiling has arrived. On Mizan’s MSA track, multiple contemporary sys- tems score at or near 100.0, and the strongest two are statistically inseparable (Section 6). 2.2 Dialect resources of the classification era MADAR (Bouamor et al., 2018) and the NADI shared tasks (Abdul-Mageed et al., 2020–2024) built the field’s foundational dialect corpora and dialect-identification tasks, spanning city- and country-level varieties. These are resources for supervised NLP: they measure whether a system can recognize or label a dialect, not whether a generative model can understand, produce, trans- late, or safely operate in one. Mizan inherits their insistence on regional granularity - every Iraqi item carries a dialect-region tag - while asking the generative-era question they predate. 2.3 Dialect-aware LLM evaluation The closest work to ours is AraDiCE (Mousi et al., 2025), which contributed approximately 45,000 samples created by machine translation with hu- man post-editing from existing benchmarks, eval- uated dialect comprehension and generation for Levantine and Egyptian Arabic, and introduced a fine-grained cultural benchmark spanning the Gulf, Egypt, and the Levant. AraDiCE demon- strated that Arabic-centric models outperform multilingual ones on dialectal tasks and that di- alect generation remains hard. Contemporaneous work extends the same translation-based line: Di- alectalArabicMMLU (Altakrori et al., 2025) man- ually translates and adapts three thousand MMLU- Redux question-answer pairs into five dialects (Syrian, Egyptian, Emirati, Saudi, and Moroc- can), confirming persistent dialectal gaps across nineteen open-weight models. Together these ef- forts define precisely where Mizan departs. Prove- nance: AraDiCE derives items by translating ex- isting benchmarks; Mizan authors every item na- tively, because translated items inherit the source benchmark’s cultural content and carry transla- tionese artifacts - the item asks what the translator wrote, not what a speaker would say. Coverage: Iraqi Arabic is absent from AraDiCE’s dialectal tasks and cultural regions, and from all five di- alects of DialectalArabicMMLU - the largest Arab country by population after Egypt appears in nei- ther effort. Scope: no existing benchmark, dialec- tal or otherwise, measures official-document field extraction, human-rubric generation quality with inter-annotator agreement, or safety behavior in a specific national context - including the dialect- triggered over-refusal mode we document in Sec- tion 7, which is invisible by construction to trans- lated MSA-derived test sets. Governance: Mizan couples its dataset to a live platform with an en- forced integrity protocol - immutable snapshots, verification certificates, a human publication gate, and public retraction - exercised during this very study. 2.4 Positioning Mizan is complementary to this landscape, not competitive with it. Shared-Arabic competence is measured well and at scale by OALL, HELM Arabic, and BALSAM; regional dialect breadth by AraDiCE and DialectalArabicMMLU. Mizan contributes depth in one nation’s linguistic and civic reality which was originally authored, du- ally reviewed, statistically audited, sovereignly an- chored and, empirically, the discriminative sig- nal that the saturated MSA axis no longer pro- vides. To our knowledge, Mizan is the first com- prehensive, originally-authored evaluation frame- work dedicated to Iraqi Arabic and the Iraqi civic context. 3 The Mizan Benchmark 3.1 Design principles Five principles govern the benchmark, each a re- sponse to a documented failure mode. Original authorship: every item is written natively in its target variety by the authoring team and individ- ually reviewed, corrected, and approved by na- tive Iraqi speakers among the authors; nothing is translated from existing benchmarks, because a translated item measures the translator’s output rather than the speaker’s language, and inherits the source benchmark’s cultural frame (Koto et al., 2024; Mousi et al., 2025). Dual review with accu- mulated rulings: every item passed a second lin- guistic review, and the rulings this produced - lex- ical authenticity constraints, register boundaries, orthographic conventions - accumulated into bind- ing authoring guidelines applied to all subsequent items. Contamination tiering: items carry a tier field separating the public development set from a reserved private-test tier; the entire pilot bank is deliberately public dev, and the private tier re- mains empty until the next development phase, a limitation we state rather than obscure (Section 9). Statistical balance: correct-answer positions in multiple-choice items are balanced by design and audited by a chi-square goodness-of-fit test against the uniform distribution, which fails to re- ject uniformity for every (track, axis) group (all chi-square≤ 1.20, df = 3, p > 0.05; actual au- dited counts reported, not planned ones). Human judging where automation misleads: generative axes are scored by human judges with numbered rubrics and inter-annotator agreement measured by Krippendorff’s alpha (Krippendorff, 2019); au- tomatic judges serve only as secondary signals, given documented same-family stylistic bias in LLM-as-judge settings. 3.2 Two tracks of equal standing Every axis runs on two tracks. The MSA track is the comparative baseline: it makes Mizan leg- ible to the international literature, anchors each model’s dialect gap to its own standard-Arabic ceiling, and - empirically - documents saturation. The Iraqi track is the discriminator: originally authored Iraqi Arabic across Baghdadi, southern, and mixed registers, with a region tag on every item (Mosuli coverage is one of the next phase targets). The design prevents the framework from collapsing into either another MSA benchmark or an untethered dialect exercise: a model’s headline story on Mizan is the pair (MSA score, Iraqi score) and the distance between them. 3.3 Six axes Dialect comprehension (multiple choice, auto- scored) covers lexical meaning, morphology and syntax, idioms, contextual inference, and MSA- Iraqi false friends - e.g., asking the meaning of hassa (”now”) in a natural sentence.Dialect generation (open generation, human-judged) elic- its Iraqi text under scenario constraints; its cen- tral rubric dimension penalizes the characteris- tic failure we call fusha in disguise (fusha = MSA/Classical Arabic): fluent MSA cosmeti- cally sprinkled with dialect markers.Bidirec- tional translation (open generation, human-judged with Character n-gram F-score (chrF) as a sec- ondary signal) tests MSA-to-Iraqi and Iraqi-to- MSA equally, since the two directions fail dif- ferently. Iraqi knowledge (multiple choice, auto- scored) spans history, geography, constitution and institutions, and popular culture - including time- sensitive facts tagged as such (Iraq’s nineteen gov- ernorates following Halabja’s promotion to gover- norate status in 2025). Official-document extrac- tion (structured extraction, auto-scored) presents a complete simulated official letter and requires six fields: issuing authority, reference number, date, addressee, subject, and required action - with de- liberately variable fields (letters lacking an ad- dressee or an action) and Iraqi administrative con- ventions enforced verbatim; no real document en- ters the bank. Safety in Iraqi context (open gen- eration, human-judged) probes neutrality across components, governorates, and symbols, refusal of genuinely harmful requests, stereotype han- dling, and loaded-premise questions; harmful con- tent is described, never instantiated, and prompts use generic framings rather than named targets. 3.4 Authoring workflow and the rejected-item discipline Items were drafted against per-axis writing guide- lines specifying valid item shapes, positive and negative examples, and reviewer checklists, then individually adjudicated by the native-speaker re- viewers. Rejection was a working tool, not an ex- ception: candidate items were discarded for lexi- cal rarity (dead or regionally opaque vocabulary), ambiguity of key, or contested ground truth. One rejected item became a design lesson we preserve: a geography question on the Tigris-Euphrates con- fluence, classically at Qurna but hydrologically shifted toward Karmat Ali - two defensible keys, therefore no item. The bank’s lexical layer follows accumulated authenticity rulings favoring living, widely-attested usage over dictionary dialect. 3.5 Orthography Iraqi Arabic has no standard orthography; the same word varies across attested spellings. The pilot adopts documented internal conventions (including the Persian-derived letters for Iraqi phonemes) applied consistently across the bank, and treats a unified, publishable Iraqi orthography guide as a distinct future deliverable rather than claiming one prematurely. 3.6 Composition The pilot bank comprises 340 items: 265 Iraqi- track and 75 MSA-track. By format: 140 multiple- choice (50 Iraqi comprehension, 50 Iraqi knowl- edge, 20 MSA comprehension, 20 MSA knowl- edge), 50 official-document extraction, and 150 open-generation items pending human judging (50 generation, 50 translation, 50 safety). Auto-scored coverage is therefore 190 items per model; the generative axes publish only after the human- judging campaign (in progress), and the leader- board displays them as pending rather than sub- stituting automatic proxies. 4 Platform and Integrity Protocol 4.1 Architecture Mizan runs as a bilingual public platform (Arabic- default, right-to-left (RTL) layout; English also supported) backed by a relational store of items, runs, and per-axis results, with a public leader- board, per-axis tables, confidence intervals, and analytical charts. Evaluation is deliberately de- coupled from the platform: an offline runner ex- ecutes a bank against a model and emits a self- describing results file (schema mizan-results-v1) carrying the harness commit hash, provider, and configuration; the platform imports results and publishes them through a separate explicit action. Item schemas are structurally compatible with lm- evaluation-harness conventions (Gao et al., 2023) a widely used open-source framework for stan- dardized LLM evaluation. 4.2 Uniform conditions and the failure ladder To distinguish genuine model refusals from tran- sient infrastructure failures, every model faces identical prompts, identical scoring, and an iden- tical completion-budget policy: a 2048-token base with a uniform retry ladder that, on an empty com- pletion or transient error, retries at 8192 tokens up to twice with backoff, logging every recov- ery so the console log constitutes a complete au- dit trail. The runner additionally emits a per-item detail sidecar (item, raw response, score) for ev- ery run - the substrate of Section 7’s error anal- ysis. The failure policy is one sentence long and applied without exception: transient failures are retried and recovered; deterministic refusals are scored as failures on the affected items and re- ported separately as over-refusal. 4.3 Snapshots, certificates, and public verification A published run is a dated, immutable snapshot of one model version on one bank version under one harness commit. New model versions produce new runs; nothing is overwritten, so the leader- board doubles as a longitudinal record. Every im- ported run receives a SHA-256 verification certifi- cate which is a cryptographic fingerprint that lets anyone independently confirm a published score has not been altered; a public verification page checks any certificate and displays revoked ones as revoked rather than hiding them. 4.4 The human gate, retraction, and supersession - exercised in practice No result reaches the public leaderboard without an explicit human publication decision. The pro- tocol’s value is easiest to show by the two oc- casions this study exercised it. First, retraction: the leaderboard’s diagnostic charts exposed two published runs with near-zero single tracks - the signature of infrastructure failure, not model abil- ity (one model scored below random chance on one track while healthy on the other). Both runs were retracted in public view, their certificates re- voked but preserved, and both models re-evaluated cleanly. Second, supersession and re-verification: after hardening the runner, we re-executed the en- tire board under the hardened harness with detail sidecars; the resulting runs supersede the originals (which remain in the record), and the two inde- pendent rounds furnish the run-to-run agreement analysis of Section 6. Retraction is reserved for measurement-wounded runs; sound runs are su- perseded, never erased. 4.5 Self-hosted models and the submission model The runner speaks to any OpenAI-compatible end- point via a configurable base URL, so self-hosted models evaluate under conditions identical to API models with no size limit - the path used here for the open and sovereign Arabic models via lo- cal quantized serving, and available to any labo- ratory. Access follows a three-line model: read- ing is open to all; reproduction is free on the re- searcher’s own resources with the public code and development set; publication on the official leader- board passes through a human-verified submis- sion gate, because unattended automation is how leaderboards accepting external results lose their integrity. 5 Experimental Setup We evaluate 27 systems on the frozen pilot bank under the protocol of Section 4. Selection follows four stated criteria: general-purpose instruction- tuned systems; accessibility through a unified API layer or the self-hosted path at evaluation time; de- liberate coverage of closed frontier models, open weights across size tiers (7B to frontier-scale), and Arabic-focused systems; and, within the freeze window, the newest releases - GPT-6 Astra entered evaluation three days after its public release. The Arabic group spans three worlds: a commercial API system (Mistral Saba), an open specialized model (SILMA-9B), and a sovereign state model (ALLaM-7B, SDAIA), the latter two served lo- cally through the OpenAI-compatible path in 4-bit quantization (Q4 KM, a common compressed- precision format for efficient local serving) - a deployment-realistic condition we state plainly. Two Arabic candidates could not be evaluated fairly and are reported as such: AceGPT-v2-8B, whose only artifact available through the standard self-hosting channel combined destructive 2-bit quantization with a missing chat template and de- generated on a smoke test; and the large Arabic models (Jais-class), absent from the unified API layer entirely - an infrastructure-absence observa- tion we return to in Section 8. Multiple-choice items are scored by exact answer-letter match; ex- traction items by strict per-field matching against ground truth, with Iraqi administrative conven- tions enforced verbatim; the per-item score is the matched-field fraction. Every published propor- tion carries a 95% Wilson interval (Wilson, 1927) (a small-sample-safe way of expressing how much a score could plausibly shift on repeat measure- ment), chosen over the normal approximation for small samples near the boundaries; for the extrac- tion axis, whose per-item scores are [0, 1] frac- tions rather than Bernoulli outcomes, the binomial treatment is a deliberately conservative bound (Bernoulli variance is maximal for bounded vari- ables), which we disclose. Track-level intervals pool corrects counts across the track’s auto-scored axes (equal n per axis makes the pooled propor- tion equal the unweighted mean). Each model was evaluated in two independent full rounds - an orig- inal round and a re-verification round under the hardened harness - and the tabled results are the verified round; the pair yields the agreement anal- ysis of Section 6.4. Prompt-sensitivity is not stud- ied in the pilot (Section 9). 6 Results Table 1 presents the frozen board: 27 systems, MSA and Iraqi track means over the auto-scored axes, and the overall macro average. Four regular- ities organize what follows. Overall is the macro- average across all five auto-scored axes (MSA comprehension, MSA knowledge, Iraqi compre- hension, Iraqi knowledge, and document extrac- tion), not the mean of the two track scores; the lower-scoring extraction axis pulls Overall below the simple MSA/Iraqi average. Table 1 - The frozen pilot board (auto-scored axes; generative axes pending human judging). Ranks follow the declared tie-break rule: display-precision overall, then the Iraqi track; ties share a rank. 6.1 Saturation on one track, discrimination on the other Eight systems score a perfect 100.0 on the MSA track and the remainder compress above 80, while the Iraqi track spreads across more than twenty- three points. Every model carries a large inter- nal gap between its own two tracks - 14 to 18 points for most of the field (Figure 1) - and the two strongest systems tie at display precision (90.9 overall, 84.9 Iraqi) and swapped ranks between the two independent rounds: the top of contemporary AI is statistically inseparable on this benchmark, Table 1: The frozen pilot board (auto-scored axes; generative axes pending human judging). Ranks follow the declared tie-break rule: display-precision overall, then the Iraqi track; ties share a rank. RankModelDeveloperMSAIraqiOverall =1gemini-2.5-proGoogle100.084.990.9 =1gpt-5OpenAI100.084.990.9 3gpt-6-astraOpenAI100.084.490.7 4gpt-4oOpenAI100.083.790.2 5gemini-2.5-flashGoogle100.083.690.1 6claude-sonnet-5Anthropic100.083.189.9 7gemma-4-31b-itGoogle100.082.489.5 8claude-opus-5Anthropic100.082.089.2 9claude-sonnet-4-5Anthropic97.583.689.1 10mistral-large-2512Mistral95.083.788.2 11gpt-oss-120bOpenAI97.582.088.2 12claude-haiku-4-5Anthropic97.581.988.1 13gemma-3-27b-itGoogle95.082.787.6 14llama-4-maverickMeta92.583.086.8 15mistral-sabaMistral95.080.986.5 16claude-fable-5Anthropic100.077.386.4 17deepseek-v3.2DeepSeek95.080.586.3 18qwen3-235b-a22b-2507Alibaba95.080.086.0 19deepseek-chat-v3.1DeepSeek95.079.685.7 20llama-3.3-70b-instructMeta95.079.185.5 21qwen-2.5-72b-instructAlibaba92.579.984.9 22qwen3-32bAlibaba90.076.682.0 23mistral-small-3.2-24bMistral90.075.981.5 24ministral-8b-2512Mistral90.073.980.3 25allam-7b-instructSDAIA80.064.770.8 26silma-9b-instructSILMA AI80.063.269.9 27llama-3.1-8b-instructMeta80.061.268.7 and the newest frontier model, evaluated 72 hours post-release, neither broke the Iraqi ceiling nor es- caped the gap. Figure 2 gives the Iraqi track with pooled 95% intervals; overlapping intervals across the leading cluster mean exact top ranks are not settled at the pilot’s sample size - the tiers, how- ever, are. 6.2 The document wall Official-document extraction confines every sys- tem tested to between 32 and 56 (Figure 3). The best score on the axis - 56.4 - belongs to neither of the overall leaders; the newest frontier model reaches 53.7; and the 7B tier collapses toward 32. The single axis built for a state’s working need is the single axis where the entire field fails, and reading the raw responses (Section 7) shows the failures are substantive: well-formed outputs that miss field precision and administrative convention, not formatting accidents. 6.3 The specialization law Three data points align. SILMA-9B (Iraqi 63.2) and the sovereign ALLaM-7B (64.7) - both marketed and built as Arabic-specialized - fall roughly ten points below a size-matched generalist (ministral-8b, 73.9) on the Iraqi track while match- ing it on MSA. Mistral Saba, by contrast, exceeds its own same-family generalist sibling by about six Iraqi points, yet remains four to five points below the frontier, with a two-track gap as wide as ev- eryone else’s. Arabic specialization, as currently practiced from startups to states, is MSA special- ization; targeted specialization helps at matched scale, and no specialization closes the dialect gap. 6.4 Run-to-run agreement Across the two independent full rounds, per-model overall scores moved by a mean absolute 0.5 points (maximum ̃1.2), every difference falling well inside the Wilson intervals (Wilson, 1927); the tied leaders exchanged ranks, exactly as over- lapping intervals predict. The measurement is sta- ble; the reported uncertainty is honest. 7 Error Analysis 7.1 A dialect-triggered over-refusal mode The pilot’s most unexpected finding began as an anomaly: the safety-hardened tier of the newest model family (claude-fable-5) placed sixteenth overall with a perfect MSA score and an Iraqi comprehension score far below its two same- family siblings.Per-item records showed the Figure 1: The two-track gap: Every model, MSA vs. Iraqi. misses were not wrong answers but empty com- pletions - deterministic on the same seven items across three independent runs, surviving a four- fold budget escalation. A dual-path probe (the aggregation gateway and the vendor’s direct API) returned the same official envelope on every one: stop reason ”refusal”, category ”cyber”, zero out- put tokens, with an explanation citing restrictions on ”violative cyber content”. The seven items ask the meaning of everyday Iraqi words - hassa (”now”), khosh (”good”), ya-m’awwad (a friendly vocative), the continuous-aspect particle da, the copula chan, jahhal (”children”), and the tradi- tional chaykhana (”tea house”). The model an- swered every Iraqi item it did not refuse - its effec- tive comprehension of answered items approaches 100% - and refused nothing on the MSA track; the other 26 systems, including two same-family models without the hardened tier’s additional mea- sures, produced zero refusals on 190 items each. Safety alignment calibrated on distributions that exclude a dialect can convert that dialect’s ordi- nary vocabulary into anomalous, blockable input: a model that goes silent on ”what does khosh mean” fails the Iraqi citizen as surely as one that answers wrongly - and no MSA benchmark can observe this failure. Under the declared policy (Section 4.2) the refusals score as failures on the affected items and are reported here as a 12% over- refusal rate on Iraqi comprehension (6/50; 3.7% of the model’s auto-scored items overall; 0% MSA; 0% for all other systems). 7.2 The document wall is substantive Reading extraction sidecars across models shows a consistent failure shape: syntactically valid struc- tured output whose issuing-authority field repro- duces the letterhead’s full hierarchy instead of the conventional issuing office, whose required-action field paraphrases rather than extracts, and whose optional fields hallucinate values on letters delib- erately authored without them. Strict field match- ing is applied identically to all systems, so the wall ranks models fairly; its height reflects genuine dis- tance from Iraqi administrative convention. Figure 2: Iraqi Track with 95% confidence intervals. 7.3 Residual genuine errors After separating refusals and recovered transients, genuine mistakes on the auto-scored Iraqi axes are sparse at the frontier (single items - e.g., one knowledge miss for the hardened-tier model) and concentrate, as expected, in the small-model tier, where comprehension errors show false-friend in- terference and knowledge errors cluster on post- 2003 institutional facts. 8 Discussion Four blind spots, one lesson. The pilot docu- ments four phenomena that the mature MSA eval- uation stack cannot observe in principle: a 14-18- point dialect gap carried by every contemporary system including one released 72 hours earlier; a civic-document register on which the entire field fails; a specialization economy in which ”Arabic” means MSA - two dedicated Arabic models, one sovereign, trailing a same-size generalist on the di- alect track; and a safety-alignment mode that con- verts a dialect’s everyday vocabulary into block- able content. Each was invisible until an instru- ment was built to see it, and each carries a prac- tical address. For model developers, the gap and the specialization law argue that dialect compe- tence is a data problem that neither scale alone nor MSA-centric fine-tuning solves. For safety teams, the over-refusal case shows that alignment distri- butions need dialectal coverage: a filter that has never seen khosh in training treats it as anomaly. For governments, the document wall converts pro- curement intuition into measurement - the capa- bility states most need is the one no vendor cur- rently delivers, and no vendor benchmark cur- rently reports. And for the evaluation community, the infrastructure observation stands on its own: most models marketed as Arabic are unreachable through the standard access channels researchers actually use - absent from unified API layers, or published in unusable quantized artifacts - which is itself a finding about where the Arabic NLP ecosystem invests. Figure 3: The Document Wall: all 27 systems confined to 32-56 9 Limitations We state the pilot’s limits plainly. Scale: fifty items per axis yields wide intervals (roughly±10 points at the axis level); every published score carries its interval, and exact rankings inside the leading cluster are explicitly unsettled. Generative axes: the 150 open-generation items (generation, translation, safety) await the human-judging cam- paign - 4,050 model responses under dual judging with Krippendorff’s alpha (Krippendorff, 2019) - and publish nothing meanwhile; the benchmark’s most distinctive axes are its least complete, a se- quencing we chose deliberately over delaying the framework. Dialect coverage: Baghdadi, south- ern, and mixed registers dominate; Mosuli cov- erage is zero pending native authors, so ”Iraqi” here under samples Iraq’s own variation. No hu- man baseline yet anchors the score scale. Con- tamination: the entire pilot bank is public de- velopment tier; a sealed private set arrives with v1.0, and until then leaderboard results are explic- itly development-tier. Measurement: two indepen- dent runs per model bound run-to-run variance but prompt-format sensitivity is unstudied; self-hosted Arabic models ran in 4-bit quantization, a real- istic but lossy condition; and strict field match- ing on extraction, while uniform, may undercount partially-correct administrative phrasings. 10 Future Work Version 1.0 scales the bank past 1,000 items through a national authoring platform with re- gional native authors - closing the Mosuli gap - and introduces the sealed private test tier. The uni- fied Iraqi orthography guide graduates into a stan- dalone linguistic-resource publication. A human- baseline study anchors interpretation. The judged axes complete and upgrade this paper’s journal version. Operationally: scheduled re-evaluation rounds tracking model updates longitudinally; a formal external-submission protocol behind the human verification gate; and GPU-hosted evalu- ation of the large Arabic models the API layer omits. 11 Ethics and Availability All official documents in the bank are simulated end to end - authorities, numbers, names, and sub- jects are fictional; no real record, personal datum, or classified material was used. Safety items de- scribe harm categories without instantiating harm- ful content and use generic framings rather than named targets. The over-refusal analysis relies ex- clusively on the vendor’s official API envelopes, reproduced verbatim. On publication, the platform code releases under Apache-2.0 and the public de- velopment set with a datasheet; the leaderboard remains live with per-run verification certificates. The released set is development-tier by design, so leaderboard integrity ultimately rests on the forth- coming sealed tier rather than on secrecy of the published items. References Muhammad Abdul-Mageed et al. 2020–2024.The NADI shared tasks: Nuanced Arabic dialect identi- fication (series). Proceedings of the WANLP / Ara- bicNLP Workshops, Association for Computational Linguistics. Annual shared-task series. Rawan Al-Matham et al. 2025. BALSAM: A plat- form for benchmarking Arabic large language mod- els. Preprint, arXiv:2507.22603. Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mugariya Farooq, Maitha Alhammadi, Julien Lau- nay, and Badreddine Noune. 2023. AlGhafa evalua- tion benchmark for Arabic language models. In Pro- ceedings of ArabicNLP 2023, pages 244–275. Asso- ciation for Computational Linguistics. Malik H. Altakrori, Nizar Habash, Abed Alhakim Frei- hat, Younes Samih, Kirill Chirkunov, Muhammed AbuOdeh, Radu Florian, Teresa Lynn, Preslav Nakov, and Alham Fikri Aji. 2025. DialectalAra- bicMMLU: Benchmarking dialectal capabilities in Arabic and multilingual language models. Preprint, arXiv:2510.27543. Accepted to LREC 2026. Houda Bouamor, Nizar Habash, Mohammad Salameh, Wajdi Zaghouani, Owen Rambow, Dana Abdul- rahim, Ossama Obeid, Salam Khalifa, Fadhl Eryani, Alexander Erdmann, and Kemal Oflazer. 2018. The MADAR Arabic dialect corpus and lexicon.In Proceedings of the Eleventh International Confer- ence on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Re- sources Association (ELRA). Ali Elfilali et al. 2024. Open Arabic LLM leaderboard (OALL). Hugging Face / Technology Innovation In- stitute. Leo Gao et al. 2023.A framework for few-shot language model evaluation (lm-evaluation-harness). Version v0.4.0. Fajri Koto, Haonan Li, Sara Shatnawi, Jad Dough- man, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timo- thy Baldwin. 2024. ArabicMMLU: Assessing mas- sive multitask language understanding in Arabic. In Findings of the Association for Computational Lin- guistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguis- tics. Klaus Krippendorff. 2019. Content Analysis: An In- troduction to Its Methodology, 4th edition. SAGE Publications. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025.AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pages 4186–4218, Abu Dhabi, UAE. Association for Computational Lin- guistics. Stanford CRFM and Arabic.AI. 2025. HELM Arabic. Stanford Center for Research on Foundation Mod- els. Edwin B. Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212.