Paper deep dive
EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams
JaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, Suan Lee
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 99%
Last extracted: 3/31/2026, 2:01:52 AM
Summary
EuraGovExam is a new multilingual and multimodal benchmark designed to evaluate vision-language models (VLMs) using over 8,000 real-world civil service exam questions from five Eurasian regions (South Korea, Japan, Taiwan, India, and the EU). The dataset is unique in its 'image-only' format, where all content—including tables, multilingual text, and diagrams—is embedded within a single image, requiring models to perform layout-aware, cross-lingual reasoning without OCR or external tools. The study reveals that nation and script factors are the primary drivers of performance variance, significantly outweighing domain-specific difficulty.
Entities (8)
Relation Signals (3)
JaeSeong Kim → authored → EuraGovExam
confidence 100% · We present EuraGovExam... JaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee
EuraGovExam → contains → Civil Service Exam Questions
confidence 100% · the dataset contains over 8,000 high-resolution scanned multiple-choice questions
EuraGovExam → evaluates → Vision-Language Models
confidence 100% · establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examinations across five representative Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. Designed to reflect the authentic complexity of public-sector assessments, the dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains. Unlike existing benchmarks, EuraGovExam embeds all question content--including problem statements, answer choices, and visual elements--within a single image, providing only a minimal standardized instruction for answer formatting. This design demands that models perform layout-aware, cross-lingual reasoning directly from visual input. All items are drawn from real exam documents, preserving rich visual structures such as tables, multilingual typography, and form-like layouts. Evaluation results show that even state-of-the-art vision-language models (VLMs) achieve only 86% accuracy, underscoring the benchmark's difficulty and its power to diagnose the limitations of current models. By emphasizing cultural realism, visual complexity, and linguistic diversity, EuraGovExam establishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation.
Tags
Links
- Source: https://arxiv.org/abs/2603.27223v1
- Canonical: https://arxiv.org/abs/2603.27223v1
Trouble viewing inline? Open PDF directly →
Full Text
71,502 characters extracted from source content.
Expand or collapse full text
EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams JaeSeong Kim kjsqp1010@semyung.ac.kr Semyung University Jecheon-si, Republic of Korea Chaehwan Lim ∗ 2024524103@semyung.ac.kr Semyung University Jecheon-si, Republic of Korea Sang Hyun Gil ∗ rlftkdgus75@semyung.ac.kr Semyung University Jecheon-si, Republic of Korea Suan Lee † suanlee@semyung.ac.kr Semyung University Jecheon-si, Republic of Korea Abstract We present EuraGovExam, a multilingual and multimodal benchmark sourced from real-world civil service examina- tions across five representative Eurasian regions: South Ko- rea, Japan, Taiwan, India, and the European Union. De- signed to reflect the authentic complexity of public-sector assessments, the dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse aca- demic and administrative domains. Unlike existing bench- marks, EuraGovExam embeds all question content—including problem statements, answer choices, and visual elements— within a single image, providing only a minimal standard- ized instruction for answer formatting. This design demands that models perform layout-aware, cross-lingual reasoning di- rectly from visual input. All items are drawn from real exam documents, preserving rich visual structures such as tables, multilingual typography, and form-like layouts. Evaluation results show that even state-of-the-art vision-language mod- els (VLMs) achieve only 86% accuracy, underscoring the benchmark’s difficulty and its power to diagnose the limi- tations of current models. By emphasizing cultural realism, visual complexity, and linguistic diversity, EuraGovExam es- tablishes a new standard for evaluating VLMs in high-stakes, multilingual, image-grounded settings. It also supports prac- tical applications in e-governance, public-sector document analysis, and equitable exam preparation. aCode|Dataset|çWebsite ∗ These authors contributed equally as co-second authors. † Corresponding author Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permis- sions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X CCS Concepts • Computing methodologies→Computer vision problems. Keywords vision-language models, multilingual benchmark, document understanding, multimodal evaluation, civil service examina- tions ACM Reference Format: JaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee. 2026. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’X). ACM, New York, NY, USA, 22pages.https://doi.org/X.X 1 Introduction Government examination documents are among the most vi- sually demanding texts that people routinely encounter: a single page may combine dense tables, mathematical expres- sions, multilingual annotations, and region-specific layout conventions such as vertical Japanese typesetting or mixed- script Devanagari passages. Millions of civil servants world- wide are assessed through such documents every year, yet the vision-language models (VLMs) that increasingly assist in document analysis have never been systematically evalu- ated on them. Existing multimodal benchmarks either sup- ply questions as clean digitized text [ 13,38], separate im- ages from their textual context [36], or rely on explicit task- specific prompts that bypass the visual complexity of au- thentic documents [20,34]. As a result, it remains unclear how well current VLMs handle the full pipeline of real docu- ment understanding—from perceiving complex layouts and scripts to reasoning about the content within them. This gap is compounded by a pronounced bias toward English and Latin-script content [ 6,13], leaving open the question of whether script diversity poses a fundamentally different challenge from domain difficulty. We introduce EuraGovExam, a multilingual multimodal benchmark sourced from real civil-service examinations across five Eurasian regions: South Korea, Japan, Taiwan, India, and the European Union. The dataset comprises over 8,000 arXiv:2603.27223v1 [cs.CV] 28 Mar 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee Table 1: Comparison of EuraGovExam with representative multimodal and exam-based benchmarks. Image-only: the model receives only an image with no separate text input. Orig. script: questions preserved in their source language without translation. N×D: benchmark structure supports factorial analysis of nation/region and domain effects.†subset of items are multimodal. Benchmark#Q #Lang. #Scripts #Domains Multi-modal Image-only Orig. script Real exam N×D MMLU [13]15,908 115777–37 MMMU [ 34]11,550 113037–37 MATHVISTA [20]6,141 11537–Partial7 AGIEval [38]8,062 222077337 EXAMS [12]24,000 1622477337 M3Exam [36]12,317 933†7337 EXAMS-V [6]20,932 1142033337 EuraGovExam (Ours) 8,000 5+51733333 high-resolution scanned multiple-choice questions spanning 17 academic and administrative domains. Two design prin- ciples distinguish EuraGovExam from prior work. First, all question content—problem statements, answer choices, ta- bles, figures, and instructions—is embedded within a single image; models receive only a minimal, content-free format- ting instruction as text input (§4). Second, questions are preserved in their original language, script, typography, and layout without any translation, OCR extraction, or refor- matting, ensuring that the benchmark faithfully reflects the perceptual and linguistic demands of authentic documents. Our evaluation of 28 VLMs reveals findings that challenge prevailing assumptions about multimodal evaluation. (i) Re- gion dominates domain: nation/script factors induce2.52× greater performance variance than domain factors, with within- model cross-regional gaps reaching 67.9 percentage points— even between regions sharing the same character family (e.g., Japan vs. Taiwan). (i) Systematic geographic failure pat- terns: models achieving over 90% accuracy on Taiwanese and EU questions can simultaneously fall below 30% on Japan- ese questions of comparable format, suggesting that script- specific visual processing constitutes a critical bottleneck. (i) Universal failure concentration: 50 questions (0.6%) de- feat all 28 models, disproportionately concentrated in re- gions with complex non-Latin scripts and dense visual lay- outs. In summary, this paper makes the following contributions: •We present EuraGovExam, a high-fidelity benchmark of 8,000+ image-only exam questions from five Eurasian regions and 17 domains, with a leakage-aware, fully re- producible evaluation protocol. •We demonstrate that nation/script is the dominant source of VLM performance variance, inducing2.52× greater variance than domain factors—challenging the prevailing domain-centric view of multimodal evalua- tion. •We provide systematic cross-regional and universal fail- ure analysis, revealing that geographic performance gaps persist across model families and scales, and char- acterizing the shared visual and linguistic properties of questions that defeat all evaluated models. 2 Related Work We position EuraGovExam at the intersection of multimodal question answering, exam-based benchmarks, and multilin- gual, domain-diverse evaluation datasets. This section re- views prior work across these axes. 2.1 Multimodal Question Answering Multimodal QA has received growing attention with the de- velopment of vision-language models. Early efforts such as VQA [ 3] and GQA [15] focused on everyday scenes, testing spatial and commonsense reasoning. More academic bench- marks like TQA [17], ScienceQA [21], and MMMU [34] ex- tend this paradigm to educational domains, incorporating textbook diagrams and subject-specific visuals. MathVista [ 20] further challenges models with mathematical reasoning grounded in visual contexts. While these benchmarks cover structured content, they are largely English-centric and limited in subject diversity. In contrast, EuraGovExam introduces 17 domains—including law, administration, and philosophy—with fully image-based inputs drawn from real civil service exams across multilin- gual and regional sources. 2.2 Exam-Based and Domain-Specific Benchmarks Examination-based QA has emerged as a strong proxy for evaluating broad knowledge and reasoning. MMLU [14] set a standard with 57 subjects across disciplines, but remains text-only and English-based. Domain-specific datasets such as HEAD-QA [ 31] (Spanish medical exams) and MedMCQA [29] (Indian medical entrance tests) highlight the utility of expert- level assessments. AGIEval [ 38] includes diverse professional exams (e.g., LSAT, SAT, public service) to probe LLM capabilities, yet most remain unimodal. EuraGovExam builds on these works by combining the rigor of real-world exam questions with the complexity of image-based inputs, reflecting authentic visual and linguistic features. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY Figure 1: EuraGovExam Dataset Construction Pipeline 2.3 Multilingual and Multimodal Exam QA Multilingual benchmarks such as EXAMS [12], IndoMMLU [18], and CMMLU [35] reveal performance gaps in cross-lingual generalization. M3Exam [36] combines multiple languages and modalities, but often presents images and text sepa- rately. EXAMS-V [ 6] advances this by embedding multimodal questions within unified image inputs, simulating real-world formats. EuraGovExam complements these efforts by cov- ering Eurasian civil exams (e.g., Japan, Korea, Taiwan, In- dia, EU) in a multimodal and multilingual setup, offering a high-fidelity evaluation resource for next-generation vision- language models. Systematic comparison. Table1positions EuraGovExam against representative multimodal and exam-based bench- marks along eight axes. Three properties jointly distinguish our benchmark. First, all question content is embedded within a single image—no separate text input is provided to the model, unlike benchmarks that supply OCR-extracted text or image–text pairs. Second, questions are preserved in their original script and layout without translation, covering five distinct script families (Hangul, Kanji/Kana, Traditional Chi- nese, Devanagari, and Latin) within a single evaluation suite. Third, the combination of 17 professional domains with five regional sources enables factorial analysis of domain versus nation/script effects on VLM performance—an analysis axis unavailable in any prior benchmark. 3 Dataset Construction EuraGovExam comprises real multiple-choice questions col- lected from public-sector examinations (national and regional civil service recruitment tests) across five Eurasian regions. Unlike conventional multimodal benchmarks that provide “image + separate text question” pairs, our dataset is de- signed such that all question content—passages, answer choices, tables, figures, mathematical expressions, and multilingual annotations—exists solely within a single image. This design enables EuraGovExam to evaluate VLMs’ document layout understanding and vision-language reasoning capabilities un- der conditions that closely approximate real examination set- tings. 3.1 Data Sources and Provenance EuraGovExam draws from officially released materials across five regions: South Korea, Japan, Taiwan, India, and the Eu- ropean Union (EU). We adhere to institutional legitimacy and legal compliance as primary principles, using only ma- terials provided by government portals or official agencies. Source provenance. •South Korea: Publicly released past examination ques- tions from national civil service tests. •Japan: Past examination questions and sample prob- lems released by the National Personnel Authority (). •Taiwan: Archived public examination items from the Ministry of Examination (). •India: Previous question papers publicly released by the Union Public Service Commission (UPSC). •European Union: Official sample and exemplar ques- tions from EPSO (European Personnel Selection Of- fice). While actual administered examinations are gen- erally not disclosed, EPSO provides official sample questions with equivalent format and calibrated dif- ficulty levels. 3.2 Construction Pipeline Figure 1summarizes the data construction pipeline. The pri- mary objective is to preserve visual and structural fidelity. To this end, we eschew OCR-based automatic extraction in favor of fully manual segmentation. Step 1: Document acquisition. We collect original docu- ments from official portals (predominantly PDF scans or digitally distributed versions), preserving the original reso- lution and rendering whenever possible. Step 2: Manual question segmentation. Trained annota- tors perform individual cropping at the question level from source documents. Questions are segmented without alteration— only separated by item boundaries. We exclude OCR to prevent (i) semantic corruption due to recognition errors, (i) loss of tables, figures, mathematical expressions, and multi-column layouts, and (i) potential omissions in mixed- language text or special characters. Step 3: Format preservation. All questions are retained in their original form without translation, paraphrasing, or rewriting. Specifically, we preserve the original language, script, Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee (a) Distribution of tasks in the EuraGovExam dataset across dif- ferent reasoning types. (b) Distribution of the EuraGovExam dataset by nation of origin. Figure 2: Distribution of the EuraGovExam dataset. font and typography, layout, and all graphical elements in- cluding tables and figures, thereby reflecting the same input distribution encountered in actual examination settings. Step 4: Quality verification. Each question undergoes a two-stage review process based on the following criteria: (i) answer label consistency, (i) crop completeness (absence of truncated tables, figures, or answer choices), and (i) resolu- tion and legibility (absence of blur or compression artifacts). Items failing verification are either re-segmented or excluded. 3.3 Privacy, Ethics, and Licensing Privacy. EuraGovExam does not contain individual-level test-taker data (responses or scores). When residual PII (e.g., candidate numbers or personal identification marks) may be present in question images, annotators manually remove such information. Licensing and reuse. The dataset is intended primarily for non-commercial research purposes, respecting the dis- closure and reuse policies of source institutions. Japanese materials fall under PDL 1.0 (reuse permitted with attri- bution) [7]; Taiwanese materials are governed by OGDL- Taiwan-1.0 (redistribution and modification permitted with attribution and adherence to exception clauses) [11]; EU Ca- reers/EPSO content is generally subject to C BY 4.0 (at- tribution and indication of modifications required), with po- tential third-party exceptions [ 8]; Indian UPSC materials are provided by segmenting original pages at the question level without content modification [ 30]. Korean materials follow the policies of their respective administering and publish- ing agencies, with KOGL (Korea Open Government License) conditions applied where such attribution is confirmed [ 23]. As jurisdictional constraints may vary, users intending purposes beyond research (e.g., commercial redistribution) should independently verify applicable regional regulations. 3.4 Problem Format and Visual Structure Each item is presented as an image, with question content containing diverse document structures: tables, graphs, multi- column layouts, mixed-language annotations, special sym- bols, and mathematical expressions. Answers follow a multiple- choice format (4 or 5 options), requiring models to iden- tify answer choices from the image, perform the required operations (calculation, comparison, reasoning, or compre- hension), and output the correct answer. Questions frequently demand document layout-grounded reasoning: (i) referencing specific rows or columns in tables to compare values, (i) cross-referencing when passage lan- guage differs from table or figure labels, and (i) combin- ing mathematical expressions or diagram interpretation with textual instructions. These characteristics constitute a core difficulty axis distinguishing EuraGovExam from conventional text-centric benchmarks. 4 Evaluation Kit and Protocol This section defines the standardized evaluation kit and re- producible evaluation protocol for EuraGovExam. We fix (i) input/output formats, (i) inference rules, and (i) scoring rules to ensure that comparisons across models and plat- forms are conducted under identical conditions. 4.1 Image-Only Setting EuraGovExam adopts an image-only setting. The model in- put consists solely of a single question image, with all ques- tion text (passages, answer choices, and instructions) exist- ing only within the image. This setting adheres to the fol- lowing principles. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY Principle 1: No external text input. The text provided to the model contains no question content. Specifically, no sep- arate question text, JSON metadata, or OCR results are supplied as input. Principle 2: No OCR or external tools. During evaluation, the use of OCR engines, external search, calculators, transla- tors, or any other tools is prohibited. Models must recognize and reason about questions in an end-to-end manner. Principle 3: Standardized minimal instruction only. All models receive only an identical format instruction (contain- ing no question information): You are solving a multiple-choice exam question shown in the image. At the very end, provide the final an- swer in exactly this format: The answer is X. (e.g., The answer is B.) This design minimizes variance due to prompt engineering and ensures fair comparison. 4.2 Inference Rules Input preprocessing. Input images are used at their orig- inal resolution by default. When model-specific maximum input size constraints apply, resizing is performed while pre- serving the aspect ratio. Resizing rules are specified in the evaluation script and applied consistently for each model un- der identical conditions. Generation budget. We fix the maximum generation length 퐿 max for all models, with the answer restricted to a single fi- nal line in the format The answer is X. Both퐿 max and stop rules are specified in the evaluation script. Answer extraction. All models must provide their answer in the final line using the format The answer is X. The evalua- tion script parses푋 ∈ 퐴, 퐵, 퐶, 퐷, 퐸using regular expressions. Format violations, multiple answers, or missing outputs are scored as incorrect, with all rules publicly documented to eliminate scoring ambiguity. 4.3 Metrics and Breakdown Primary metric. The primary evaluation metric is accu- racy: Accuracy= 1 푁 푁 ∑ 푖=1 ퟙ[ ̂푦 푖 = 푦 푖 ],(1) wherê푦 푖 denotes the predicted answer,푦 푖 the ground-truth answer, and푁the number of questions. Random baseline. The random baseline for 4-choice ques- tions is 25.0% and for 5-choice questions is 20.0%. Based on the dataset composition, the weighted average random base- line is 23.7%. Models performing below this threshold are interpreted as exhibiting systematic parsing errors or funda- mental failures in question comprehension. Breakdown axes. In addition to overall accuracy, we re- port results along three breakdown axes: •By nation: Accuracy across five regions (South Korea, Japan, Taiwan, India, EU). •By domain: Accuracy across 17 domains (e.g., mathe- matics, administration, biology). •Failure overlap: Identification of question sets where multiple models commonly fail, enabling diagnosis of systematic bottlenecks. 4.4 Reproducibility Statement • # Full benchmark evaluation $python evaluate.py \ –modelMODEL_NAME\ –splittest\ –settingimage-only # Filter by nation (e.g., Japan only) $python evaluate.py –modelgpt-4o–nationjapan # Filter by domain (e.g., mathematics only) $python evaluate.py –modelgpt-4o–domainmathematics # Combine filters (nation + domain) $python evaluate.py –modelgpt-4o–nationkorea–domain law To ensure reproducibility of EuraGovExam results, we provide a standardized evaluation script, fixed data splits, fixed inference rules (decoding, preprocessing, and parsing), and experiment logging templates. All main results can be reproduced with a single command. The evaluation script supports the following filtering op- tions: –nation (five regions: korea, japan, taiwan, india, eu) and –domain (17 subjects). Multiple filters can be combined for fine-grained analysis, with results automatically saved in JSON format. 5 Benchmark Results This section analyzes the EuraGovExam evaluation results across 28 VLMs (14 closed-source, 14 open-source). We ex- amine overall performance and domain-specific patterns (§5.1), cross-regional performance variance (§5.2), and the charac- teristics of questions where all models fail (§5.3). 5.1 Overall Performance Table2reports the performance of all evaluated models un- der a unified image-only protocol (§ 4). Closed-source models. The top three closed-source models— Gemini-2.5-Pro (86.99%), GPT-5 (85.80%), and o3 (84.26%)— exceed 80% accuracy across most domains, with Gemini-2.5- Pro surpassing 80% in 15 of 17 tasks. However, substantial variability persists within model families: GPT-4o attains only 42.04%, trailing GPT-4.1 (54.73%) by 12.7 percentage points, highlighting the role of multilingual and non-Latin- script document coverage in image-only reasoning. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee Table 2: EuraGovExam performance. Icons denote regions (India,EU,Taiwan,Japan,S.Korea) and domains (Chem,Phil,Earth,Psych,Econ,Bio,Geo,Phys,Pol,Hist,Admin,Lang,Med,Eng, CS,Law,Math). All models evaluated under the image-only protocol (§4) with greedy decoding and single inference. ModelsOverall NationTask Closed-source Models o3 [28]84.26 68.64 84.49 93.72 82.37 90.06 85.51 90.55 73.83 92.44 87.20 89.23 74.03 81.85 79.90 86.33 81.53 91.49 91.18 84.16 90.93 78.08 78.98 o4-mini [28]79.40 63.38 76.95 92.29 82.52 82.49 85.02 85.83 75.70 85.71 82.86 83.02 64.09 89.63 73.21 67.97 71.21 84.19 88.24 81.98 92.52 63.92 80.61 GPT-4o [16]42.04 40.99 63.73 66.66 25.97 33.25 50.72 36.22 17.76 29.41 31.89 42.73 37.02 33.52 34.93 35.55 47.48 49.73 50.00 33.72 58.96 42.00 43.38 GPT-4.1 [27]54.73 48.10 66.44 72.58 48.10 54.23 68.12 55.12 33.64 46.22 46.42 53.50 48.07 54.63 45.93 48.83 54.57 60.14 69.75 50.00 73.47 47.54 58.06 GPT-4.1-mini [27]56.27 46.34 63.58 79.03 43.84 59.92 65.22 72.44 34.58 52.10 51.84 58.93 37.02 58.70 47.37 39.84 54.14 61.62 72.69 51.16 75.74 51.23 56.24 GPT-5-nano [24]67.58 52.19 63.68 83.87 68.07 72.97 73.91 74.80 50.47 73.95 69.20 73.70 51.38 73.52 57.42 50.00 63.59 61.08 81.51 68.17 83.67 57.14 72.46 GPT-5 [25]85.80 68.35 83.87 94.80 87.50 91.17 86.96 92.91 74.77 94.96 89.37 90.01 70.72 87.96 79.90 86.33 83.67 90.27 93.28 88.81 92.06 81.03 78.22 GPT-5.2 [26]69.94 53.46 68.05 78.67 73.83 73.09 78.26 79.53 61.68 70.59 71.80 67.48 59.12 78.52 67.94 61.33 68.53 61.35 81.09 75.00 85.94 63.92 68.52 Gemini-3-pro-preview [9]68.53 68.94 92.82 97.49 29.59 75.26 75.36 78.74 35.51 66.39 62.04 77.03 62.98 28.89 64.11 74.22 77.23 86.89 88.66 55.23 73.47 71.67 64.88 Gemini-3-flash-preview [10]75.28 51.70 89.54 97.31 56.69 84.50 77.78 85.04 44.86 71.43 65.08 78.36 57.46 63.52 62.68 71.09 79.16 85.81 92.86 72.38 85.94 76.48 73.22 Gemini-2.5-pro [5]86.99 69.23 88.08 95.51 87.59 91.12 87.92 93.70 85.98 95.80 88.29 88.90 74.03 84.26 79.43 87.50 83.57 91.49 95.80 86.19 92.06 84.24 86.28 Gemini-2.5-flash [5]68.33 62.32 83.30 92.65 51.46 67.65 72.95 77.17 48.60 66.39 55.97 73.36 54.70 59.26 62.20 66.02 68.53 75.68 83.19 64.83 75.28 63.18 73.13 Gemini-2.5-flash-lite [5]25.94 12.95 54.37 73.29 07.13 13.99 22.71 18.11 7.48 17.65 16.49 40.40 23.20 5.74 23.45 16.80 37.27 37.03 42.02 13.23 33.33 34.36 12.76 Claude-Sonnet-4 [2]63.29 62.51 76.43 87.28 45.85 62.41 67.63 70.87 40.19 64.71 56.83 68.81 55.80 51.67 63.16 66.02 64.12 72.30 78.99 56.40 69.39 60.84 61.61 Open-source Models Llama-3.2-11B-Vision [1]12.75 13.82 20.08 23.65 10.06 6.29 10.63 7.87 15.89 7.56 10.63 16.09 23.20 11.11 13.88 11.72 15.79 13.11 13.87 8.87 15.42 16.50 6.44 Qwen2-VL-2B-Instruct [32]25.54 15.09 35.90 33.33 18.21 26.13 24.64 19.68 15.89 27.73 22.99 27.08 17.68 19.63 22.97 18.36 33.40 34.32 25.63 22.97 26.30 28.82 19.19 Qwen2-VL-7B-Instruct [32]31.38 27.36 47.19 52.51 15.57 29.08 28.50 33.86 12.15 26.89 26.03 37.51 30.39 16.11 29.67 25.39 41.78 41.89 37.82 20.93 41.04 37.81 20.73 Qwen2.5-VL-7B-Instruct [4]32.30 26.29 45.89 46.95 21.88 29.53 33.33 30.71 18.69 25.21 25.16 35.18 30.39 25.00 24.40 26.95 36.52 43.51 36.13 26.02 41.72 34.48 28.02 Phi-3.5-vision-instruct [32]15.67 15.48 19.56 15.77 14.64 13.53 18.36 11.02 10.28 16.81 14.32 14.21 19.34 13.52 17.70 16.80 19.01 15.54 13.45 15.41 16.55 18.60 12.96 Qwen2-VL-72B-Instruct [32]44.65 35.93 62.07 74.73 30.37 39.71 52.66 41.73 26.17 38.66 34.27 48.72 37.57 37.41 35.41 31.25 51.99 53.38 50.84 35.61 58.28 48.03 40.60 InternVL2.5-38B-MPO [33]39.34 19.38 52.91 56.81 31.49 39.63 38.16 30.71 14.02 34.45 27.12 41.18 24.86 36.48 35.89 26.56 44.04 49.73 39.50 34.74 56.24 40.02 39.16 Ovis2-8B [22]28.31 25.02 40.63 34.77 17.58 27.53 30.43 31.50 17.76 21.85 23.64 29.97 29.83 17.41 25.84 24.22 36.09 37.30 26.05 21.51 34.92 31.16 23.51 Ovis2-16B [22]32.73 20.06 45.32 51.61 26.71 28.88 33.82 25.98 19.63 21.85 22.99 35.29 25.41 32.59 29.67 24.22 39.21 42.97 35.29 26.16 38.78 30.05 32.25 Ovis2-32B [22]35.50 22.49 51.61 54.48 28.42 29.90 32.37 30.71 15.89 23.53 25.60 37.62 27.62 36.11 30.14 18.75 41.25 45.95 40.34 24.71 43.99 38.05 36.76 llama3-llava-next-8b [37]14.28 7.89 16.65 17.92 12.30 15.91 20.77 19.69 7.48 15.97 12.80 14.21 12.71 14.07 9.09 15.63 17.29 14.86 16.81 11.77 12.93 16.01 11.80 llava-1.5-13b [19]18.99 16.65 21.59 17.92 19.48 17.75 15.94 16.54 18.69 19.33 19.52 17.31 23.76 19.44 19.62 16.41 21.80 18.51 21.43 18.31 17.23 17.00 20.54 llava-1.5-7b [19]12.44 12.76 13.22 12.37 11.96 12.11 11.59 14.17 11.21 6.72 13.02 11.88 10.50 12.96 10.53 12.11 13.75 13.11 13.03 12.21 14.74 12.19 11.52 LLaVA-NeXT-Video-7B-DPO-hf [ 37] 7.21 10.22 6.56 9.68 7.18 5.93 9.66 2.36 5.61 5.88 7.81 6.99 8.84 7.22 7.66 9.38 5.26 5.95 6.72 7.56 5.44 6.90 10.17 Open-source models. Open-source models lag considerably: the strongest, Qwen2-VL-72B (44.65%), trails Gemini-2.5- Pro by over 42 percentage points. Several widely used models fall below the random baseline (23.7%), including LLaMA- 3.2-11B-Vision (12.75%) and LLaVA-NeXT-Video-7B (7.21%), exhibiting near-uniform accuracy across nations and domains indicative of stochastic guessing. This gap—substantially larger than those on text-only benchmarks such as MMLU [13]— suggests that image-only multilingual evaluation dispropor- tionately penalizes models with weaker visual encoders and limited script coverage. Domain-specific patterns. Domain difficulty rankings re- main remarkably consistent across models. High-performing domains include Philosophy, Psychology, and Language, where top models frequently exceed 90% accuracy. In contrast, low- performing domains—notably Earth Science, Geography, and Law—exhibit systematic degradation, characterized by dense tables, charts, symbolic notations, and spatial references that require precise visual parsing and cross-modal alignment. 5.2 Cross-Regional Performance Variance The central finding of this study is that nation/region in- duces substantially greater performance variance than do- main. This phenomenon is consistently observed through within-model cross-regional gaps, model×region interaction patterns, and variance decomposition analysis. Regional performance distribution. Figure3(b) presents the model-averaged accuracy across five regions. A consistent regional ranking emerges across all 28 models: Taiwan (57%) ranks highest, followed by EU (52%), South Korea (42%), In- dia (38%), and Japan (35%). Table3presents nation-specific accuracy for six representative models with Wilson 95% con- fidence intervals. Extreme cross-regional gaps. Within-model performance gaps across nations are remarkably large. For Gemini-3-Pro, we observe a 64.8 percentage point difference between Tai- wan (96.1%) and Japan (31.3%), indicating that the same model can exhibit over 3×performance variation depending on the country of origin. This pattern is not limited to a single model: Table 3: Nation-specific accuracy (%) for selected models. ±: Wilson 95% CI. Bold: highest region; underline: lowest region. Gemini-3-Pro † : largest cross-regional gap (64.8%p). ModelOvr. JPINKR EU TW Gemini-2.5-Pro 87.0 87.6±1.369.2±2.291.1±1.388.1±1.895.5±1.7 GPT-585.8 87.5±1.368.4±2.291.2±1.383.9±2.094.8±1.9 o384.3 82.4±1.568.6±2.290.1±1.484.5±2.093.7±2.0 o4-mini79.4 82.5±1.563.4±2.382.5±1.777.0±2.392.3±2.2 Gem.-3-flash 75.3 56.7±1.951.7 ±2.484.5±1.689.5±1.797.3±1.4 Gem.-3-pro † 68.5 31.3±2.066.5±2.974.4±1.789.0±1.496.1±1.6 † Largest gap: TW–JP = 64.8%p. Gemini-2.5-Flash-Lite (66.2%p), Gemini-3-Flash (45.6%p), and GPT-4o (40.7%p) all show gaps exceeding 40 percentage points. Crucially, despite both Japan and Taiwan belonging to the Chinese character (Hanzi/Kanji) cultural sphere, Tai- wan consistently yields higher performance across all models. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY This phenomenon cannot be explained by simple linguistic differences alone, suggesting the influence of script-level char- acteristics—Japanese vertical writing (), mixed Kanji- Kana text, and furigana annotations—as well as document layout factors. Model×region interaction. The heatmap in Figure3(a) vi- sualizes the relative strengths and weaknesses of individual models across regions. Most models exhibit negativeΔ(rel- ative underperformance) in India and Japan, while show- ing positiveΔ(relative overperformance) in Taiwan and EU. Gemini-3-Pro displays the most extreme imbalance, record- ing−39%p in Japan and+29%p in Taiwan. Notably, the two newest Google models (Gemini-3-Pro and Gemini-3-Flash) both exhibit severe Japan underperformance (−39%p and −19%p respectively), suggesting that Japanese document pro- cessing remains a persistent challenge that has not been re- solved in newer model generations. Variance decomposition: Nation vs. Domain. To quantify the sources of VLM performance variation, we compute nation- wise variance (휎 2 nation,푚 ) and domain-wise variance (휎 2 task,푚 ) for each model푚. Computing the arithmetic mean of vari- ances across all 28 models, we find that nation factors induce 2.50×greater variance than domain factors: ̄휎 2 nation ̄휎 2 task = 160.53 64.20 = 2.50(2) This ratio is even more pronounced for lower-performing models: for Gemini-2.5-Flash-Lite, nation variance reaches 5.5×the domain variance. 5.3 Universal Failure Analysis Table 4: Universal failure questions (all 28 models incorrect). India accounts for 60% despite 12.7% of the dataset. Nation Cnt %DomainCnt India30 2.9Mathematics 9 S. Korea 8 0.3Economics7 Japan6 0.3Engineering6 EU6 0.3Administration 5 Taiwan 0 0.0History4 Total50 0.6Other (11) 15 Out of 8,000 questions, 50 (0.6%) elicit incorrect responses from every evaluated model (Table 4). India accounts for 60% of these failures (30/50) despite comprising only 12.7% of the dataset (failure rate 2.9% vs. 0.3% elsewhere), while Taiwan contributes zero. Qualitative examination reveals three co-occurring characteristics: complex script environments (non- Latin characters within mathematical expressions), dense vi- sual elements (nested tables, small-font annotations), and domain-specific symbols (circuit diagrams, chemical formu- las) demanding joint visual and domain knowledge. Notably, universal failures concentrate in India rather than Japan, which exhibits the lowest average accuracy—revealing a distinction between difficulty and intractability. Japanese questions are systematically harder but partially solved by top models; Indian failures defeat even the strongest sys- tems, suggesting a deeper entanglement of perception, do- main knowledge, and culturally specific reasoning. 6 Discussion Why region dominates domain. The 2.50×variance ratio (Eq.2) carries a direct implication: benchmarks that control for domain but not for script and layout may substantially underestimate real-world performance gaps. A model scoring 90% on Philosophy from Taiwan and the EU can fall below 30% on the same domain from Japan—not because domain knowledge differs, but because the visual presentation does. Our qualitative failure analysis (Appendix B) provides the causal layer behind this pattern: Japan’s errors concentrate in visual parsing failures driven by vertical text layouts and dense furigana annotations, whereas India’s errors split be- tween reasoning and knowledge deficits tied to Devanagari- embedded technical content. These region-specific bottleneck profiles confirm that a single remediation strategy will yield uneven returns across regions, and that the diagnostic gran- ularity enabled by EuraGovExam’s factorial structure is un- available from aggregate-accuracy benchmarks. Directions for multilingual VLM development. Our results suggest three concrete improvements. First, script-aware vi- sual encoding: the persistent Japan–Taiwan gap despite shared Hanzi/Kanji origins shows that character-level similarity does not guarantee comparable processing; Japanese vertical text, mixed Kanji-Kana layouts, and furigana annotations demand layout-aware tokenization strategies. Second, balanced multi- lingual training data: the correlation between regional perfor- mance and likely training-data availability suggests that tar- geted augmentation for underrepresented scripts—particularly Japanese document scans and Devanagari-embedded techni- cal content—could yield disproportionate gains, especially since newer model generations have not closed these gaps through scaling alone. Third, document layout as a first-class evaluation axis: the consistent underperformance of layout- dense regions indicates that layout complexity should be var- ied independently alongside domain and language in future benchmarks. Limitations. We identify four principal limitations. First, the benchmark covers five regions spanning diverse script families and governance traditions but does not include Africa, the Middle East, or Latin America; findings about script- driven bottlenecks may not generalize to Arabic or Amharic writing systems. Second, regional sources differ in domain composition and question difficulty, reflecting genuine differ- ences in national examination design rather than controlled experimental variation. While our two-way ANOVA partially accounts for this confound, strictly causal claims about script versus domain effects require further controlled experiments. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee Figure 3: Cross-regional performance analysis. (a) Model×Region interaction heatmap: each cell showsΔ =Region−Overall accuracy; magenta = underperformance, blue = overperformance (values shown for|Δ| ≥ 14%p). Model names are color-coded by source type (blue: closed, orange: open). (b) Average accuracy by region with 95% Wilson CIs. (c) Ten most difficult domains. Third, the image-only protocol precludes disentangling OCR- level failures from higher-level visual understanding errors; a model that extracts text perfectly but misinterprets a dia- gram and one that fails at character recognition both man- ifest as incorrect answers. Fourth, our benchmark captures a static snapshot of government examinations; regulatory and curricular changes may alter difficulty distributions over time, necessitating periodic updates to maintain ecological validity. Future directions. Three research directions emerge from this work. Intervention-based diagnosis: systematically vary- ing input modality—providing OCR-extracted text along- side images, or translating questions into English—would decompose performance gaps into perception, language, and reasoning components, transforming EuraGovExam from a diagnostic benchmark into an intervention-testing platform. Our image-only protocol establishes the baseline against which such interventions can be measured. Geographic and tempo- ral expansion: extending to additional script families (Arabic, Thai, Cyrillic) and tracking examination changes over time would enable longitudinal studies of whether geographic bi- ases are closing or widening as models scale. Fine-grained an- notation: augmenting each question with visual complexity scores, required knowledge type, and reasoning depth would support targeted data augmentation for underperforming region–domain cells. We release our evaluation toolkit and annotation guidelines to facilitate community contributions along these axes. 7 Conclusion We presented EuraGovExam, an image-only benchmark of over 8,000 government-examination questions drawn from five Eurasian regions (South Korea, Japan, Taiwan, India, and the EU) across 17 professional domains and five dis- tinct script families. Unlike existing multilingual benchmarks that supply OCR-extracted text or translated questions, our protocol preserves original document layouts and scripts, re- quiring models to jointly perceive and reason over authentic visual content. Evaluation of 28 VLMs yields two principal findings. First, nation and script factors induce 2.50×greater performance variance than domain factors, with cross-regional gaps reach- ing 64.8 percentage points within a single model—demonstrating that the visual and linguistic properties of documents, not just their content, are a primary determinant of VLM per- formance. Second, 50 questions defeat all 28 models, concen- trating in Japan and South Korea (63.6%) and co-occurring in visually complex, knowledge-intensive items that resist improvement through scaling alone. These results carry a practical implication: evaluation pro- tocols that control for domain but not for script and layout will systematically underestimate real-world performance gaps. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY EuraGovExam’s factorial nation×domain structure provides the diagnostic resolution needed to identify where and why models fail, guiding targeted improvements in script-aware encoding, multilingual training balance, and layout-robust architectures. We release the full dataset, evaluation scripts, and model outputs to support further research. 1 References [1]Meta AI. 2024. Llama 3.2: Revolutionizing edge AI and vision with open LLMs. Meta AI blog post.https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/Accessed: 2025-07-16. [2]Anthropic. 2025. System Card: Claude Opus4 & Claude Sonnet4. Anthropic system card [PDF].https://w-cdn.anthropic.com/ 4263b940cabb546a0e3283f35b686f4f3b2f47.pdfAccessed: 2025- 07-16. [3]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question Answering. In ICCV. [4]Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025). [5]Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pa- supat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Con- text, and Next Generation Agentic Capabilities. arXiv preprint arXiv:2507.06261 (2025). [6]Diptesh Das and et al. 2024. EXAMS-V: A Multilingual, Multimodal Benchmark of School Exams. In arXiv preprint arXiv:2401.00001. [7]Digital Agency, Government of Japan. 2024. Public Data Uti- lization Terms (PDL 1.0). https://w.digital.go.jp/resources/ open_data/public_data_license_v1.0. Accessed: 2026-01-28. [8]European Union. 2026. Legal notice – disclaimer and copy- right.https://european-union.europa.eu/legal-notice_en. Ac- cessed: 2026-01-28. [9]Google. 2025. Gemini deprecations (Gemini 3 models).https: //ai.google.dev/gemini-api/docs/deprecations. Includes preview model ID gemini-3-pro-preview (released Nov 18, 2025). Accessed 2026-02-06. [10]Google AI for Developers. 2025. Gemini deprecations (Gemini API) — Gemini 3 models list.https://ai.google.dev/gemini-api/ docs/deprecations . Includes gemini-3-flash-preview release date; Accessed: 2026-02-03. [11]Government of Taiwan. 2015. Open Government Data License, version 1.0.https://data.gov.tw/license. Accessed: 2026-01-28. [12]Momchil Hardalov and et al. 2020. EXAMS: A Multilingual Ex- amination QA Benchmark. In LREC. [13]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Man- tas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measur- ing Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR). [14]Dan Hendrycks and et al. 2021. Measuring Massive Multitask Language Understanding. In ICLR. [15]Drew A Hudson and Christopher D Manning. 2019. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering. In CVPR. [16]Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). [17]Aniruddha Kembhavi, Minjoon Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. 2017. TQA: Textbook Question Answering. In CVPR. 1 Dataset:https://huggingface.co/datasets/EuraGovExam/ EuraGovExam; Code:https://github.com/thisiskorea/EuraGovExam; Website:https://thisiskorea.github.io/EuraGovExam/; [18]Fajri Koto and et al. 2023. IndoMMLU: A Comprehensive Bench- mark for Indonesian Language Understanding. In arXiv preprint arXiv:2306.05459. [19]Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 26296–26306. [20]Pan Lu and et al. 2023. MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts. In arXiv preprint arXiv:2306.15794. [21]Pan Lu, Xiangning Li, and et al. 2022. Learn to Explain: Multi- modal Reasoning via Science Question Answering. In NeurIPS. [22]Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. 2024. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797 (2024). [23]Ministry of Culture, Sports and Tourism (Republic of Korea) and Korea Culture Information Service Agency. 2012. Korea Open Government License (KOGL): Public Copyright License for Public Works.https://w.kogl.or.kr/info/license.do. Accessed: 2026-02-01. [24]OpenAI. 2025. GPT-5 nano Model (OpenAI API Documen- tation).https://platform.openai.com/docs/models/gpt-5-nano. Accessed: 2026-02-03. [25]OpenAI. 2025. Introducing GPT-5.https://openai.com/index/ introducing-gpt-5/. Accessed: 2026-02-03. [26]OpenAI. 2025. Introducing GPT-5.2.https://openai.com/index/ introducing-gpt-5-2/. Accessed: 2026-02-03. [27]OpenAI. 2025. Introducing GPT‑4.1 in the API. OpenAI blog post.https://openai.com/index/gpt-4-1/Accessed: 2025-07-16. [28]OpenAI. 2025. OpenAI o3 and o4-mini System Card. https://cdn.openai.com/pdf/2221c875-02dc-4789-800b- e7758f3722c1/o3-and-o4-mini-system-card.pdfAccessed: 2025- 07-16. [29]Abhishek Pal and et al. 2022. MedMCQA: A Large-scale Multi- subject Medical QA Dataset. In Findings of EMNLP. [30]Union Public Service Commission (UPSC), India. 2026. Website Policies.https://upsc.gov.in/website-policy. Accessed: 2026-01- 28. [31]David Vilares and Carlos Gómez-Rodríguez. 2019. HEAD-QA: A Healthcare Dataset for Complex Reasoning. In NAACL. [32]Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024). [33]Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, et al. 2024. Enhancing the reasoning ability of multimodal large lan- guage models via mixed preference optimization. arXiv preprint arXiv:2411.10442 (2024). [34]Xinshi Yue, Xinyang Wang, and et al. 2023. MMMU: A Multi- discipline Multi-modal Understanding Benchmark for College- Level Exams. In arXiv preprint arXiv:2306.16413. [35]Aohan Zeng and et al. 2023. CMMLU: A Chinese Massive Multi- task Benchmark for Language Understanding. In arXiv preprint arXiv:2306.04387. [36]Yuyang Zhang and et al. 2023. M3Exam: A Multilingual, Multi- modal Benchmark for Large Language Models. In arXiv preprint arXiv:2306.14435. [37]Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. 2024. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 (2024). [38]Yichong Zhong and et al. 2023. AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models. In arXiv preprint arXiv:2304.06364. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee A Additional Analyses: Scaling, Task Difficulty, and Cross-Task Structure Overview. This appendix complements the main results (overall accuracy and country/domain variability) with a set of diagnostic analyses. Specifically, we quantify (i) scaling trends among open-source multimodal models, (i) task-level difficulty structure across model performance tiers, and (i) cross-task covariance and latent structure via correlation- based clustering. Together, these analyses support the view that our benchmark measures not only a single aggregate score but also structured difficulty patterns and latent capa- bility axes. A.1 Scaling Trends in Open-Source Models Figure 4: Scaling analysis for open-source models. We ana- lyze the relationship between parameter scale (in billions, 푆) and overall benchmark accuracy (퐴) for open-source VLMs using a linear regression in log scale. The dashed line shows the fitted trend, and the shaded region indicates a±1 standard-deviation band of the regression residuals (an em- pirical±1SD band). We obtain푅 2 = 0.41with푝 < 0.01, and observe an empirical trend of the form퐴 ≈ 16.8 ⋅log(푆). Setup. For the open-source model setℳ open , let푆 푚 de- note the parameter count of model푚(in billions) and퐴 푚 its overall benchmark accuracy. To characterize scaling be- havior, we fit the following linear model: 퐴 푚 = 훽 0 + 훽 1 log(푆 푚 ) + 휖 푚 ,(3) where휖 푚 is the residual. We test the regression significance using a standard푡-test under the usual assumptions. Observation. Figure4shows that accuracy increases on average withlog(푆), but the explanatory power remains mod- erate (푅 2 = 0.41). This indicates that parameter scale alone does not account for the full variance in performance, and that models with comparable size can still differ substan- tially. Implication. Beyond raw scale, our benchmark is sensitive to differences in factors such as document visual processing (layout/text), training recipe, and multimodal encoder de- sign. In other words, while scaling contributes to improved aggregate scores, practical bottlenecks may arise from non- scale capability axes (e.g., document understanding, quanti- tative reasoning, or knowledge integration). A.2 Task Difficulty Structure Across Model Tiers Figure 5: Task difficulty by model tier (mean accuracy). We partition models into Top, Middle, and Bottom tiers by overall accuracy, and visualize per-task mean accuracy as a heatmap. This reveals whether the relative ordering of task difficulty is preserved as models improve, and which tasks remain persistent bottlenecks. Tier definition. We sort models inℳby overall accuracy Acc(푚)and construct three tiers: the top퐾 top modelsℳ top , the middle퐾 mid modelsℳ mid , and the bottom퐾 bot models ℳ bot . In our experiments, we use Top 5 / Middle 7 / Bottom 11. For each task푡 ∈풯and tier휏 ∈ top, mid, bot, we define the tier-averaged task accuracy as 퐴(푡, 휏) = 1 |ℳ 휏 | ∑ 푚∈ℳ 휏 퐴(푚, 푡),(4) where퐴(푚, 푡)is the accuracy of model푚on task푡. Observation. Figure5highlights (i) tasks where perfor- mance collapses sharply in the Bottom tier, (i) hard-core tasks that remain relatively low even in the Top tier, and (i) groups of tasks whose relative difficulty ordering is largely preserved across tiers. Notably, steep drops in the Bottom EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY Figure 6: Cross-task correlation with hierarchical clustering. For each task, we form a model-wise accuracy vector and compute Pearson correlations푟(푡, 푡 ′ )between tasks. We then apply hierarchical clustering (average linkage) using Euclidean distance between correlation profiles (rows/columns of the correlation matrix). High-correlation blocks suggest that subsets of tasks co-vary due to shared latent capability axes. tier are most pronounced for tasks that require both document- level visual understanding (layout/text) and domain-specific reasoning. Implication. Tier-wise heatmaps expose bottleneck tasks that are invisible from the mean score alone. Moreover, if difficulty structure remains stable across tiers, this suggests our benchmark is not a noisy collection of items but rather provides a consistent, structurally decomposable evaluation signal. A.3 Cross-Task Correlation and Latent Structure (Clustering) Correlation matrix. For each task푡, let the model-wise accuracy vector bea 푡 = ( 퐴(푚 1 , 푡), ... , 퐴(푚 |ℳ| , 푡) ) ∈ ℝ |ℳ| . We define the Pearson correlation between tasks푡and푡 ′ as 푟(푡, 푡 ′ ) =corr ( a 푡 ,a 푡 ′ ) .(5) LetR= [푟(푡, 푡 ′ )] 푡,푡 ′ ∈풯 be the resulting correlation matrix. To cluster tasks, we compute Euclidean distance between correlation profiles (rows ofR): 푑 row (푡, 푡 ′ ) = ‖ R 푡,∶ −R 푡 ′ ,∶ ‖ 2 ,(6) and perform average-linkage hierarchical clustering to deter- mine the dendrogram ordering. (Importantly, our ordering Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee is not based on푑 = 1 − 푟, but on similarity of full correlation profiles.) Observation. The block structure in Figure6indicates that tasks do not vary independently across models: instead, some task groups rise and fall together. This suggests that overall performance may be explained by a combination of global document-understanding capability plus a small num- ber of specialized axes (e.g., quantitative reasoning, regula- tion/linguistic inference). Implication. The correlation and clustering structure strength- ens the benchmark’s diagnostic value. (1) Beyond the main aggregate score, we can report vulnerability patterns at the cluster level, enabling more precise characterization of model bottlenecks. (2) It supports targeting data collection and model improvements at the level of latent capability axes, rather than treating tasks as isolated endpoints. Finally, broadly high correlations would be consistent with the bench- mark measuring a coherent set of document-grounded com- petencies. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY B Qualitative Failure Analysis This appendix presents ten representative failure cases—two per region—selected to illustrate the error categories dis- cussed in the main paper. Each case includes the original question image, a representative model response (preserved verbatim), and a structured error analysis identifying the failure category and root cause. Table5provides a naviga- tional overview. Table 5: Summary of qualitative failure cases. VP: Visual Parsing, RF: Reasoning, KF: Knowledge, HC: Hallucination. # Region DomainType ModelPred (GT) 1 S. Korea Economics KFGemini-2.5-ProD (A) 2 S. Korea Administration RFClaude-Sonnet-4B (D) 3 Japan PhysicsRFo4-miniB (C) 4 Japan Engineering KF+VP Qwen2-VL-72B (C) 5 India Mathematics RF+HC Llama-3.2-11B-VisionA (B) 6 India Economics VPLLaVA-1.5-13B– (C) 7 EUAdministration KFClaude-Sonnet-4– (D) 8 EUAdministration VP+HC LLaMA3-LLaVA-NeXT-8B – (A) 9 Taiwan Administration RFPhi-3.5-Vision– (A) 10 Taiwan BiologyKFGPT-4.1B (D) B.1 Case 1: Knowledge Failure in Korean Accounting Figure 7: Korean local government accounting question re- quiring knowledge of financial statement calculation rules. Metadata. •Region: South Korea Domain: Economics •Ground Truth: A ( 1 ⃝) Gemini-2.5-Pro(7Incorrect) This is a multiple-choice question in Korean regarding ac- counting rules for local governments. The question asks to identify the incorrect statement about the “Financial Operation Statement” (). Analysis of the options: (1)Statement 1 ⃝: This statement describes the rev- enue recognition principle for exchange transac- tions (). It states that revenue (like us- age fees) is recognized when a claim arises and the amount can be reasonably measured. This is a stan- dard and correct accounting principle. (2)Statement 2 ⃝: This statement defines “Net Pro- gram Cost” () as “Total Cost” () minus “Program Revenue” (). It also de- fines “Total Cost” as the cost incurred for a pro- gram, plus costs allocated from other programs, minus costs allocated to other programs. This is the correct calculation method for program costs in governmental accounting. (3)Statement 3 ⃝: This statement classifies revenue sources for a local government into three cate- gories: self-generated revenue (), in- tergovernmental transfer revenue (), and other revenue ( ). This is the standard classification of revenues in a local government’s financial statement. (4)Statement 4 ⃝: This statement describes the calcu- lation of the “Net Cost of Financial Operations” (). It states the formula is: Net Program Cost + (Administrative/Operational Ex- penses AND Non-allocable Costs) - Non-allocable Revenue. This is incorrect. “Administrative and Operational Expenses” () are a compo- nent of “Non-allocable Costs” (), not a separate item to be added alongside it. The cor- rect formula is: Net Program Cost + Non-allocable Costs - Non-allocable Revenue. The phrasing in this option implies double-counting administrative expenses, making the statement incorrect. Therefore, the incorrect statement is 4 ⃝. Since the choices are numbered, we will map them to letters: 1 ⃝→A, 2 ⃝→ B, 3 ⃝→C, 4 ⃝→D. The answer is D. Error Analysis Failure Category: Knowledge Failure (KF) Why it failed: Under Korean local government accounting standards, “Administrative and Operational Expenses” () are not a component of “Non-allocable Costs” () but constitute a separate line item. The model incorrectly assumes that management and op- erating expenses are subsumed under non-distributed ex- penses, leading it to misidentify Statement 4 ⃝rather than Statement 1 ⃝as the incorrect option. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee B.2 Case 2: Reasoning Failure in Korean Library Classification Figure 8: Korean Decimal Classification (KDC) question requiring rule-based geographic subdivision with duplicate- symbol removal. Metadata. •Region: South Korea Domain: Administration •Ground Truth: D ( 4 ⃝) Claude-Sonnet-4(7Incorrect) Looking at this Korean Decimal Classification (KDC) question, I need to determine the correct classification number for “” (Jeju Island Dialect). From the table provided: •718 =() — Dialects/Regional Lan- guages •The geographic subdivision system shows: –−119(Jeolla Province) –−1109(Jeju Special Self- Governing Province) For “” (Jeju Island Dialect): •Base number: 718 (for dialects) •Geographic subdivision: Jeju Island Since Jeju Island is historically and administratively con- nected to the Jeolla region, and the table shows−119for (Jeolla Province), the classification system would use 1199 as a more specific subdivision to represent Jeju Island (adding an additional digit to 119 for greater speci- ficity). Therefore: 718.1199 •718 = Korean dialects •.1199 = Jeju Island geographic subdivision The answer is B. Error Analysis Failure Category: Reasoning Failure (RF) Why it failed: The base number 718 already encodes “Ko- rean dialects,” where the digit 1 represents Korea. When applying geographic subdivision from the regional table, the KDC rules require checking for and removing symbols that duplicate meanings already present in the base num- ber. The model fails to apply this duplicate-removal rule, producing an incorrect classification number. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY B.3 Case 3: Reasoning Failure in Japanese Physics Figure 9: Japanese physics question on projectile motion along an inclined plane, presented in mixed Kanji-Kana text. Metadata. •Region: Japan Domain: Physics •Ground Truth: C ( 3 ⃝) o4-mini(7Incorrect) We interpret the wording to mean that the ball is pro- jected in the vertical plane of the incline at30°above the horizontal (so that its trajectory stays in that “incline- plane”), and we are to find its horizontal flight-range on the floor. Hence its initial horizontal and vertical compo- nents are 푣 푥 = 푣cos30° = √ 3 2 푣, 푣 푧 = 푣sin30° = 1 2 푣, so the range on level ground is 푅 = 2푣 푥 푣 푧 푔 = 2 ⋅ √ 3 2 푣 ⋅ 1 2 푣 푔 = √ 3 2 ⋅ 푣 2 푔 . This matches choice 2, i.e. answer B. The answer is B. Error Analysis Failure Category: Reasoning Failure (RF) Why it failed: The model correctly identifies the inclined- plane setting but applies the standard flat-ground range formula using vertical gravitational acceleration푔, rather than decomposing gravity into components parallel and perpendicular to the incline. This leads to an incorrect expression for the projectile range along the inclined sur- face. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee B.4 Case 4: Knowledge and Visual Parsing Failure in Japanese CMOS Circuit Analysis Figure 10: Japanese engineering question requiring CMOS logic circuit analysis with transistor-level diagrams. Metadata. •Region: Japan Domain: Engineering •Ground Truth: C ( 3 ⃝) Qwen2-VL-72B(7Incorrect) To determine the correct answer, we need to analyze the CMOS logic circuits shown in the image. 1. Circuit I (푋output): •The pMOS transistors are connected in parallel and controlled by퐴and퐵. •The nMOS transistors are connected in series and controlled by퐴and퐵. This configuration corresponds to an OR gate for the pMOS transistors and an AND gate for the nMOS tran- sistors. Therefore, the output푋is given by: 푋 = 퐴 + 퐵 2. Circuit I (푌output): •The pMOS transistors are connected in series and controlled by퐴and퐵. •The nMOS transistors are connected in parallel and controlled by퐴and퐵. This configuration corresponds to an AND gate for the pMOS transistors and an OR gate for the nMOS transis- tors. Therefore, the output푌is given by: 푌 = 퐴 ⋅ 퐵 Comparing these results with the given options: •Option 2:퐴 + 퐵for푋and퐴 ⋅ 퐵for푌 Therefore, the correct answer is: The answer is B. Error Analysis Failure Category: Knowledge Failure + Visual Parsing Failure (KF + VP) Why it failed: •Knowledge Failure: The model incorrectly deter- mines the on-conditions of nMOS and pMOS tran- sistors in the CMOS logic circuits, reversing the pull-up and pull-down network logic. •Visual Parsing Failure: The circuit diagram con- tains inverter stages (paired nMOS/pMOS tran- sistors) immediately before outputs푋and푌. The model fails to detect these inverters, omitting a crit- ical negation step in the Boolean expression deriva- tion. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY B.5 Case 5: Reasoning Failure and Hallucination in Indian Mathematics Figure 11: Hindi-language mathematics question requiring distance computation from directional travel instructions. Metadata. •Region: India Domain: Mathematics •Ground Truth: B Llama-3.2-11B-Vision(7Incorrect) The information in the question is: •A multiple-choice exam question is presented. •The question is in Hindi, but the instructions are in English. •The question asks to determine the shortest dis- tance between a bank and a person’s home, given different locations of the bank and the person’s home. •The possible answers are: –(a) 20 km –(b) 25 km –(c) 30 km –(d) 35 km •The correct answer is labeled as X. Step 1: The problem states that a bank is located at a distance of 10 km from a person’s home. The person can walk directly to the bank or take a route that involves walking to a nearby location and then to the bank. Step 2: The problem mentions that the person can walk directly to the bank, which means the shortest distance is 10 km. Step 3: The problem also mentions that the person can take a route that involves walking to a nearby location and then to the bank. This route is described as walking 20 km to a nearby location and then 5 km to the bank, making a total distance of 25 km. Step 4: Another route is described as walking 40 km to a nearby location and then 5 km to the bank, making a total distance of 45 km. Step 5: The problem asks for the shortest distance, which is the distance that is less than all other options. Step 6: Based on the information provided, the shortest distance is 10 km, which is option (a). The final answer is:10 Error Analysis Failure Category: Reasoning Failure + Hallucinated Con- tent (RF + HC) Why it failed: •Grounding Failure: The question requires comput- ing the minimum straight-line distance from direc- tional travel segments, but the model neither ex- tracts nor utilizes the given distances and direc- tions. •Problem Misinterpretation: The question asks for the minimum distance between home and bank af- ter a series of movements, but the model mistakes the first 10km segment as the answer itself. •Hallucinated Premise: The model fabricates a premise absent from the question: “The problem states that a bank is located at a distance of 10km from a person’s home.” Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee B.6 Case 6: Visual Parsing Failure in Indian Economics Figure 12: Hindi-English bilingual economics question with assertion-reasoning format requiring structured visual com- prehension. Metadata. •Region: India Domain: Economics •Ground Truth: C LLaVA-1.5-13B(7Incorrect) The image shows a page with a mix of English and another language, possibly Hindi. The text is written in both lan- guages, with the English words and phrases interspersed throughout the page. The content appears to be a com- bination of a dictionary, a book, or a study guide, as it contains definitions and explanations in both languages. The page is likely designed to help people learn and un- derstand the other language by providing translations and explanations in their native language. Error Analysis Failure Category: Visual Parsing Failure (VP) Why it failed: The question follows a structured assertion- reasoning format with numbered statements and answer choices (a)–(d), but the model fails to parse this struc- ture entirely. Instead, it misclassifies the content as “a dictionary, a book, or a study guide,” producing a generic description rather than engaging with the question. This represents a complete failure to extract the visual and textual layout of an examination item. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY B.7 Case 7: Knowledge Failure in EU Building Ordinance Figure 13: Catalan-language question on the Metropolitan Building Ordinance requiring knowledge of specific regula- tory notification deadlines. Metadata. •Region: EU Domain: Administration •Ground Truth: D Claude-Sonnet-4(7Incorrect) Looking at this question about the Metropolitan Build- ing Ordinance (“Ordenança metropolitana d’edificació”), it asks about the deadline for a promoter/developer to notify the municipal administration when the company in charge of construction work changes. This is a very specific legal question that requires knowl- edge of the exact text of this particular ordinance. With- out access to the specific Metropolitan Building Ordi- nance referenced in the question, I cannot definitively de- termine the correct answer from the legal document itself. However, based on the structure of the question and typical administrative notification requirements in simi- lar contexts, this would likely be a relatively short time- frame to ensure proper municipal oversight of construction projects. Since I cannot access the specific ordinance to verify the exact requirement, I cannot provide a definitive answer to this legal/regulatory question. The answer is [Cannot be determined without access to the specific ordinance]. Error Analysis Failure Category: Knowledge Failure (KF) Why it failed: The question requires the exact notifica- tion deadline stipulated in the Ordenança metropolitana d’edificació. The model correctly identifies the question’s intent and the relevant legal domain but explicitly ac- knowledges lacking knowledge of the specific provision, declining to answer rather than guessing. This represents a clean knowledge failure without perceptual or reasoning errors. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee B.8 Case 8: Visual Parsing Failure and Hallucination in EU Regulation Question Figure 14: Catalan-language multiple-choice question on CTE DB SI building regulations, with four answer choices (A–D). Metadata. •Region: EU Domain: Administration •Ground Truth: A LLaMA3-LLaVA-NeXT-8B(7Incorrect) The image shows a multiple-choice question with the an- swer provided. The question is in Spanish and reads: “Ustedes están resolviendo un examen que tiene 5 pregun- tas. A, B, C, D y E son las opciones. Si el número de re- spuestas correctas es X, ¿cuál es la letra que corresponde a X?” The answer provided is: “La respuesta es X. (Por ejemplo: La respuesta es B.)” Below the answer, there are five options labeled A, B, C, D, and E, each with a corresponding description of a situ- ation. These options are meant to illustrate the different scenarios that could correspond to the letter X in the an- swer. The options are: A.Zona de residencia, en calidad de cas... B.Zona de residencia, en calidad de administrativo... C.Zona de residencia, en calidad de administrativo... D.Zona de residencia, en calidad de administrativo... E.Zona de residencia, en calidad de administrativo... The options are meant to be humorous or satirical, as they describe various scenarios that could be associated with a letter X, but they are not typical of a standard multiple-choice question format. Error Analysis Failure Category: Visual Parsing Failure + Hallucinated Content (VP + HC) Why it failed: •Visual Parsing Failure: The question is a stan- dard four-choice (A–D) Catalan-language item on CTE DB SI building regulations, but the model misreads it as a five-option Spanish question about an entirely different topic. •Hallucinated Content: The model fabricates a non- existent choice E, invents question text (“5 pregun- tas...”), and generates option descriptions absent from the image. EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service ExamsConference acronym ’X, June 03–05, 2018, Woodstock, NY B.9 Case 9: Reasoning Failure in Taiwanese Administration Figure 15: Traditional Chinese administration question re- quiring policy content analysis and answer selection. Metadata. •Region: Taiwan Domain: Administration •Ground Truth: A Phi-3.5-Vision(7Incorrect) The slide presents a multiple-choice exam question in both Chinese and English. It instructs the reader to provide the final answer in a specific format, using a letter like A, B, C, or D, and then to write “The answer is X.” in English, with an example given as “The answer is B.” The question number is 16, and there are four answer choices provided in Chinese characters. Error Analysis Failure Category: Reasoning Failure (RF) Why it failed: The model extracts only meta- information—“presents a multiple-choice exam question,” “the question number is 16,” “four answer choices”— without engaging with the actual question content. Since it cannot ground its reasoning in the substantive policy content of the question, it produces no basis for selecting an answer. This represents a failure to transition from surface-level document description to content-level reasoning. Conference acronym ’X, June 03–05, 2018, Woodstock, NYJaeSeong Kim, Chaehwan Lim, Sang Hyun Gil, and Suan Lee B.10 Case 10: Terminology Misread in Taiwanese Biology Figure 16: Traditional Chinese biology question asking to identify the location of the temporal fossa () on an anatomical diagram. Metadata. •Region: Taiwan Domain: Biology •Ground Truth: D GPT-4.1(7Incorrect) The question in Chinese asks which locations on the di- agram correspond to “” (cranial fossa). The cranial fossa refers to the internal depressions at the base of the skull (anterior, middle, and posterior cranial fossae). Markings 1 (forehead), 2 (top side of skull), and 5 (tempo- ral region near the cranial base) correspond most closely to where the cranial fossae are located internally. Looking at the choices: A.Only 4, 5 B.Only 1, 2, 5 C.Only 1, 2 D.1, 2, 4, 5 The best fit is B (1, 2, 5) as these locations generally map to the cranial fossae. The answer is B. Error Analysis Failure Category: Knowledge Failure (KF) Why it failed: The question asks about(temporal fossa), but the model misreads the characteras, in- terpreting the term as(cranial fossa)—an anatomi- cally distinct structure. This single-character misidentifi- cation cascades into selecting incorrect anatomical land- marks, demonstrating how visually similar Traditional Chinese characters can induce domain-level errors even when the model’s reasoning process is otherwise sound.