Paper deep dive
Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence
Jakub Pokrywka, Łukasz Grzybowski, Antoni Lasik, Marek Kubis, Jeremi Ignacy Kaczmarek, Wojciech Kusa
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/14/2026, 5:34:38 AM
Summary
This paper introduces a Polish-language medical Visual Question Answering (VQA) benchmark derived from the Polish Board Certification Examination (PES). It evaluates various vision-language models (VLMs) on image-containing medical questions, comparing their performance against text-only QA controls and human examinees. The study finds that while models achieve above-chance accuracy, they often underutilize visual evidence, relying more on textual cues and answer choice artifacts. GPT-5.6 is the only model to surpass human performance on the VQA subset.
Entities (8)
Relation Signals (5)
Polish Board Certification Examination → sourceof → Polish Medical VQA Benchmark
confidence 95% · We introduce a Polish-language medical visual question answering (VQA) benchmark, built from Polish Board Certification Examination questions
GPT-5.6 → outperforms → Human Examinees
confidence 92% · only GPT-5.6 surpasses the approximate human reference on the subset with available candidate responses
Vision-Language Models → underutilizes → Visual Evidence
confidence 90% · Vision-Language Models Underutilize Visual Evidence
Emergency medicine → hashighestvqacount → Polish Medical VQA Benchmark
confidence 88% · Emergency medicine contributes the largest number of VQA questions
LLaVA-Bielik → ispolishoriented → Polish Language
confidence 85% · evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce a Polish-language medical visual question answering (VQA) benchmark, built from Polish Board Certification Examination questions for licensed physicians and dentists pursuing specialist certification. The benchmark comprises image-containing questions spanning diverse medical specialties and visual domains, together with a text-only question answering (QA) control set. We evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models. The task remains challenging: the best model achieves 79.0\% accuracy on the full VQA set, and only GPT-5.6 surpasses the approximate human reference on the subset with available candidate responses; all other evaluated models perform worse than humans. To assess visual grounding, we compare complete inputs with configurations omitting the image, the question, or both, and categorize questions by image importance. Models derive more useful information from the question text than from the image and perform worse on image-dominant questions. Across both QA and VQA, they nevertheless achieve above-chance accuracy from the answer choices alone, showing that non-trivial performance can persist even when key task components are missing.
Tags
Links
- Source: https://arxiv.org/abs/2608.12928v1
- Canonical: https://arxiv.org/abs/2608.12928v1
Trouble viewing inline? Open PDF directly →
Full Text
50,268 characters extracted from source content.
Expand or collapse full text
Polish Medical Visual Question Answering: Vision-Language Models Underutilize Visual Evidence Jakub Pokrywka Łukasz Grzybowski Affiliation: ARAAI Poland Antoni Lasik Affiliation: NASK National Research Institute [0.6em] Marek Kubis Jeremi Ignacy Kaczmarek Affiliation: Poznań University of Medical Sciences Affiliation: T. Marciniak Lower Silesian Specialist Hospital Wojciech Kusa Affiliation: NASK National Research Institute [0.8em] Adam Mickiewicz University Abstract We introduce a Polish-language medical visual question answering (VQA) benchmark, built from Polish Board Certification Examination questions for licensed physicians and dentists pursuing specialist certification. The benchmark comprises image-containing questions spanning diverse medical specialties and visual domains, together with a text-only question answering (QA) control set. We evaluate Polish-oriented, general-purpose open-weight, and commercial vision-language models. The task remains challenging: the best model achieves 79.0% accuracy on the full VQA set, and only GPT-5.6 surpasses the approximate human reference on the subset with available candidate responses; all other evaluated models perform worse than humans. To assess visual grounding, we compare complete inputs with configurations omitting the image, the question, or both, and categorize questions by image importance. Models derive more useful information from the question text than from the image and perform worse on image-dominant questions. Across both QA and VQA, they nevertheless achieve above-chance accuracy from the answer choices alone, showing that non-trivial performance can persist even when key task components are missing. 1 Introduction Large language models (LLMs) have recently been evaluated on several Polish medical examination benchmarks, including the Polish Board Certification Examination (pol. Państwowy Egzamin Specjalizacyjny, PES), the Medical Final Examination (pol. Lekarski Egzamin Końcowy, LEK), the Dental Final Examination (pol. Lekarsko-Dentystyczny Egzamin Końcowy, LDEK), and related medical test sets 28; 18; 22. These studies showed that modern LLMs can achieve strong results on Polish medical multiple-choice questions and provided evidence on how well models handle specialized medical knowledge in a non-English setting. However, their evaluation protocols were limited to text-only questions. As a result, examination items containing images were excluded, even though visual information is an important component of many real medical tasks and of some PES questions. This omission leaves an important gap in the evaluation of medical AI systems. While visual question answering (VQA) has been widely studied in English, non-English medical VQA remains relatively underexplored. Polish VQA resources are also limited, especially in specialized domains such as medicine. This is problematic because model performance in English cannot be assumed to transfer directly to Polish, and medical examination questions often require knowledge of domain-specific terminology, clinical conventions, and local examination formats. Consequently, there is a need for benchmarks that evaluate not only medical knowledge in Polish, but also the ability of models to combine Polish clinical text with medical images. In this work, we evaluate vision-language models (VLMs) on image-containing questions from PES, the Polish Board Certification Examination. These questions are multiple-choice examination items intended for physicians and dentists pursuing specialist certification. They provide a challenging test bed for multimodal medical question answering, as they often require both domain knowledge and interpretation of visual evidence. Importantly, the dataset is not composed solely of classical VQA examples where the image is the central object of a direct visual query. In many cases, the image is only one component of a broader clinical scenario: the question may include a textual patient description, laboratory or diagnostic context, answer choices, and an image such as an electrocardiogram, radiological scan, or clinical photograph. Therefore, the task is better understood as multimodal medical examination question answering rather than simple image recognition or image-centered VQA. Beyond measuring overall model accuracy, we study how much information models obtain from different parts of the input. Prior work has shown that models can exploit artifacts in multiple-choice answer options or rely disproportionately on textual cues instead of genuinely using visual evidence 4; 5; 3. To examine this issue in the Polish medical examination setting, we evaluate models under controlled input configurations: using only the answer choices, using choices together with the question text, using choices together with the image, and using the full input consisting of choices, question text, and image. This setup allows us to estimate the relative contribution of answer choices, textual context, and visual information. We also compare performance on image-containing PES questions with performance on a text-only question answering (QA) control set composed of questions that originally did not include images. This comparison allows us to analyze differences between QA- and VQA-style evaluation within the same examination domain. Additionally, we conduct a data contamination analysis to assess whether model performance may have been influenced by prior exposure to the evaluation questions. Our contributions are as follows: • We create an image-containing question dataset from the Polish Board Certification Examination as a benchmark for Polish medical multimodal question answering. • We evaluate vision-language models on PES questions under several input configurations that separate the effects of answer choices, question text, and images. • We compare model performance on image-containing VQA questions with performance on a text-only QA control set from the same examination domain. 2 Related Work 2.1 Polish VQA 33 adapt the LLaVA framework to Polish and introduce LLaVA-Bielik and LLaVA-PLLuM. They show that translated and filtered multimodal data can effectively bootstrap Polish VLMs and provide Polish-oriented evaluation resources. reVISION 11 evaluates VLMs on Polish multimodal national examination data, extending the exam-based evaluation setting introduced in LLMzSzŁ 21 from text-only LLMs to vision-language models. PoVisLE 2 further moves toward Polish-specific vision-language evaluation with emphasis on Polish linguistic and cultural grounding. Polish is also present in broader multilingual VQA resources. EXAMS-V 12 includes Polish among multilingual multimodal examination questions, while 30 evaluate cross-lingual transfer to Polish on 500 machine-translated Polish questions. Our work differs from the aforementioned resources by focusing on specialist-level Polish medical examination questions that require combining clinical text, answer options, and medical images. 2.2 Medical VQA English medical VQA has been studied mainly in radiology, pathology, and biomedical image–text settings. VQA-RAD 23 introduced clinically generated questions and answers for radiology images, while the ImageCLEF VQA-Med shared tasks provided a series of radiology-focused medical VQA benchmarks 19; 7; 6; 8. PathVQA 20 introduces a new dataset and framework for visual question answering over pathology images. More recent datasets scale medical VQA through visual instruction tuning, including PMC-VQA 37 and PubMedVision introduced with HuatuoGPT-Vision 10. Non-English and multilingual medical VQA is more limited. SLAKE provides a bilingual English–Chinese medical VQA dataset with semantic labels and medical knowledge 24. WorldMedQA-V 26 and MMMED 32 evaluate multimodal medical examination questions in multiple languages. Other recent resources address specific languages or domains, including multilingual wound-care VQA 34, Indonesian radiology VQA 35, and multilingual hematology VQA 25. 2.3 Biases in VQA VQA benchmarks often contain linguistic or answer-distribution shortcuts that allow models to answer correctly without sufficient visual grounding. 17 addressed this issue by introducing VQA v2, where similar images are paired with the same question but different answers, making the visual signal more important. 1 further showed that VQA models rely heavily on question-answer priors by introducing VQA-CP, a split with different answer distributions between training and test data. Several works proposed methods to reduce such biases, including adversarial regularization with a question-only model 31, RUBi, which downweights examples solvable without the image 9, and visually grounded question encoding 16. This issue is also relevant for modern VLMs and medical VQA. MIRAGE shows that frontier VLMs can generate detailed visual reasoning and obtain high scores on multimodal benchmarks even without image input 3. 36 propose counterfactual training and a changing-priors medical VQA split to reduce reliance on linguistic shortcuts. Med-BiasX similarly targets medical language biases caused by imbalanced data and question shortcut dependence 38. These findings motivate our controlled input configurations, which separately evaluate performance from answer choices, question text, images, and their combination. 3 Dataset 3.1 Examination background The dataset used in this work is based on the Polish Board Certification Examination (PES), a national examination for physicians and dentists pursuing specialist certification in Poland. Candidates taking PES have already obtained a medical or dental license and completed the required specialist training, including clinical practice, courses, internships, and discipline-specific procedural requirements. The examination consists of a written and an oral component. The written part is held separately for each medical or dental specialty and typically contains 120 single-choice questions. Each question has five answer options, exactly one of which is correct. Most questions are text-only, although some include an accompanying image. A score of at least 60% is required to pass the written examination. Since 2022, candidates who obtain at least 70% in the written part have been exempted from the oral examination. Unlike licensing examinations such as LEK and LDEK, PES questions are not publicly available before the exam, which makes them a suitable source of challenging specialist-level medical questions. In this study, we focus on the written part of PES, as it provides standardized multiple-choice questions with unambiguous correct answers. This format enables automatic evaluation of model predictions while preserving the specialist-level medical character of the task. 3.2 Benchmark construction We collected PES examination materials from the Medical Examination Center (Centrum Egzaminów Medycznych, CEM) website11 1 https://w.cem.edu.pl/, covering examination sessions from 2023 to 2026. In total, we processed 363 examination sheets. We removed questions marked by CEM as invalid or no longer aligned with current medical knowledge. We then identified examination sheets containing at least one image. This yielded 116 examination sheets with visual material. From these sheets, we extracted all questions containing images and constructed the VQA subset, consisting of 286 image-containing questions. Each VQA item contains the question text, five answer choices, the correct answer, metadata describing the examination session and specialty, and the associated image. For comparison with text-only question answering, we also constructed a QA control subset. This subset consists of PES questions that originally did not contain any images. To make the QA subset comparable to the VQA subset, we selected specialties for which we had at least 10 image-containing questions. For each examination sheet in these specialties, we sampled 10 text-only questions. This procedure resulted in 480 QA questions. The distribution of VQA and QA questions across specialties is shown in Table 1. The distribution of image-containing questions is not uniform across medical specialties. Emergency medicine contributes the largest number of VQA questions, followed by maxillofacial surgery, orthopedics, and pediatric cardiology. Specialties with fewer than 10 image-containing questions are grouped into the “Other specialties” category. Since the QA subset was constructed only for specialties with at least 10 VQA questions, no QA items are assigned to this grouped category. Specialty VQA QA Emergency medicine 62 70 Maxillofacial surgery 37 70 Orthopedics 28 70 Cardiology (pediatric) 28 60 Conservative dentistry 13 70 Neurology (pediatric) 13 70 Anesthesiology & critical care 10 70 Other specialties 95 0 Total 286 480 Table 1: Number of VQA and QA questions by medical specialty. Specialties with fewer than 10 VQA questions are grouped as other specialties. 3.3 Example PES question Figure 1 presents an example image from an emergency medicine PES question. The item illustrates the character of the dataset: the model must combine visual evidence from a head computed tomography (CT) scan with medical knowledge expressed in the answer options. The correct answer is the false statement about the presented pathology. The original question is in Polish; the question and answer choices shown below are an English translation. Figure 1: Image Question. The attached image shows a head CT scan of a patient after trauma. Indicate the false statement about the presented pathology. Choices: A. It most commonly results from rupture of the middle meningeal artery. B. In the classic clinical presentation, there is an initial loss of consciousness, followed by a relatively asymptomatic interval, the lucid interval. C. In this type of hematoma, blood accumulates between the skull bone and the dura mater. D. A characteristic feature is a lentiform collection of blood that does not cross the cranial sutures to which the dura mater is attached. E. It most commonly results from injury to bridging veins between the surface of the brain and the dural venous sinuses. Correct answer: E Metadata. Year: 2023, Quarter: Spring, Specialty: Emergency medicine 4 Evaluation Methodology We evaluate nine open-weight and commercial vision-language models on the PES-VQA benchmark. The open-weight models comprise three Polish-oriented VLMs: LLaVA-Bielik-11b-v2.6-instruct, LLaVA-PLLuM-12b-nc-instruct-250715, and LLaVA-PLLuM-12b-nc-instruct 33; three Qwen models: Qwen3.5-397B-A17B, Qwen3.5-9B, and Qwen3.6-27B 29; and Gemma-4-31B-it 15. The commercial models are GPT-5.4-nano-2026-03-17 and GPT-5.6-sol 27. Shortened model labels used in the result tables refer to these full model identifiers. Among the open-weight models, only the LLaVA variants do not support reasoning; the others use reasoning by default. In the result tables, the R column marks reasoning as enabled (Y) or disabled or unavailable (N). Only the GPT models were evaluated in both configurations: none (N) and medium (Y). 4.1 Input configurations The objective of our evaluation extends beyond measuring final accuracy to estimating the amount of information models obtain from different portions of the input. We therefore evaluate models under several controlled input configurations. For image-containing questions, we use four settings: • C: answer choices only, • C+Q: answer choices and question text, • C+I: answer choices and image, • C+Q+I: answer choices, question text, and image. The C setting measures whether a model can exploit artifacts, priors, or statistical regularities in the answer choices without access to the question itself. The C+Q setting evaluates text-only performance. The C+I setting tests whether the image provides useful information when the question text is removed. Finally, C+Q+I corresponds to the complete multimodal examination item as presented to candidates, whereas the remaining configurations are used only for our ablation studies. For the text-only QA control subset, only two configurations are applicable: C and C+Q. 4.2 Prompting and evaluation metric All experiments were conducted using Polish prompts. The prompt instructed the model to answer a single-choice medical examination question with options A–E and to return only a JSON object containing the selected answer. The full prompts, together with English translations and prompting details, are provided in Appendix A. For configurations in which an image was omitted from an image-containing question, we did not explicitly inform the models that the image was unavailable. This design choice follows 3, who found that model performance declined markedly when models were explicitly instructed to guess without image access, compared with prompts that implicitly led them to assume that an image was present. We report accuracy, defined as the percentage of questions for which the model’s predicted answer matches the official answer key. Since every question has five answer options and exactly one correct answer, random guessing corresponds to an expected accuracy of 20%. 4.3 Subset with human responses Human responses were available for only a subset of the benchmark questions. We collected these responses and used them to construct a human results subset for comparing model performance with human performance. The data collection and alignment procedure is described in Appendix B. 4.4 Visual signal categories 4.4.1 Image importance To characterize how strongly each question depends on its visual material, we assigned every VQA question to one of three image-importance categories: 0 – Image non-essential (text sufficient). The correct answer can be determined reliably from the question and answer choices without using the image. The image may illustrate, confirm, or repeat information already present in the text, but it is not required to solve the question. This category contains 15 questions (5.2%). 1 – Image and text complementary. Both textual and visual information are needed to determine the correct answer reliably. The text provides information independent of the image, such as symptoms, medical history, test results, or clinical context, which must be combined with the visual evidence. This category contains 131 questions (45.8%). 2 – Image dominant. The correct answer depends primarily on interpreting the image. The text contains no substantial clinical information independent of the image and serves mainly to provide instructions, identify the type of visual material, or state the task. This category contains 140 questions (49.0%). 4.4.2 Visual domains We additionally categorized the visual material by content domain. Although the underlying taxonomy is hierarchical, we report only its top-level categories: IMAGE (n=102n=102). Medical images, including radiological and other diagnostic imaging modalities, microscopy, ophthalmic imaging, and clinical photography. This category also includes images for which no more specific subtype was assigned. WAVEFORM (n=95n=95). Physiological signal traces, including cardiac, neurophysiological, evoked-potential, and hemodynamic recordings. PLOT (n=23n=23). Plots presenting measurements, relationships, or analyses, including audiological, biomechanical, glucose-monitoring, pressure–volume, radiotherapy, spirometry, and statistical plots. GRAPHIC (n=32n=32). Explanatory diagrams and schematics depicting anatomy, biomechanics, classifications, devices, mechanisms, or procedures. TABLE (n=20n=20). Visual material organized in tabular form, including results, comparisons, and matching tables. COMPOSITE (n=14n=14). Material combining multiple visual or textual elements, such as device printouts, documents, software screens, or composite test results. OTHER (n=3n=3). Visual material that could not be assigned to any of the categories above. 5 Data Contamination Analysis To detect contamination, we used the Data Contamination Quiz (DCQ) 14 framework. DCQ consists of two stages: quiz creation and examination. First, a frontier LLM generates altered versions of the questions by paraphrasing certain words to break memorization. The tested models are then asked to identify the original question among the paraphrased variants. We used DeepSeek-V4-Pro 13 for quiz creation. Due to computational and cost constraints, we tested one representative model from each evaluated model family for contamination. The results presented in Table 2 suggest negligible contamination, which should not influence the results of our evaluation. For each model, DCQ reports a closed interval where the lower bound is the bias-corrected minimum contamination level (via Cohen’s Kappa) and the upper bound is the maximum raw quiz accuracy across bias-compensated permutations. It is worth noting that the original DCQ prompt explicitly references the dataset name and split as part of the instruction. Since our dataset is not an established, named benchmark, this framing may carry less signal, and the resulting estimates should be interpreted with some caution. Model QA VQA LLaVA-Bielik-11B-v2.6 (16.86, 27.08) (15.38, 29.02) LLaVA-PLLuM-12B-250715 (14.39, 24.38) (11.19, 22.38) Qwen3.6-27B (11.27, 11.46) (10.88, 11.19) Gemma-4-31B-it (0.03, 16.67) (0.07, 22.03) GPT-5.4-nano (17.41, 22.92) (7.69, 18.53) Table 2: Contamination level ranges reported using the DCQ methodology on the PES medical QA/VQA datasets (textual part), provided in the format (min contamination, max contamination). 6 Results The overall results across input configurations are presented in Figure 2. The evaluation conducted with respect to the full dataset shows a strong effect of model size, with larger models achieving higher accuracy. Among open-weight models small enough to fit on a single consumer-grade GPU, Qwen3.5-9B outperforms all LLaVA-based models. For the GPT models we evaluated different reasoning-effort settings. Increasing reasoning effort substantially improves the performance of GPT-5.4-nano. For GPT-5.6-sol, it improves QA performance but provides only a small improvement on VQA. 6.1 Comparison with human examinees Figure 2: Model comparison across input configurations. C+Q represents the complete configuration for QA, while C+Q+I represents the complete configuration for VQA. The comparison on the subset with human responses is presented in Table 3. QA and VQA have a similar level of difficulty for human examinees, as human accuracy is nearly identical on the two subsets. For the evaluated models, QA is generally easier than VQA. Accuracy above 60% on both QA and VQA, which corresponds to a passing score for human examinees, is achieved by Qwen3.5-397B-A17B, Gemma-4-31B-it, and GPT-5.6. Among the evaluated models, only GPT-5.6-sol outperforms human examinees on average, under both the none and medium reasoning-effort settings. This finding indicates that the dataset is particularly challenging for models. The commercial GPT-5.6 model also substantially outperforms the open-source models. 6.2 Model performance with incomplete inputs We analyze model performance under incomplete-input scenarios separately for QA and VQA. Model R QA VQA LLaVA-Bielik-11B N 43.35 39.68 LLaVA-PLLuM-12B N 34.84 33.33 LLaVA-PLLuM-12B-250715 N 34.04 35.32 Qwen3.5-9B Y 52.39 41.27 Qwen3.5-397B-A17B Y 68.88 63.10 Qwen3.6-27B Y 64.10 57.94 Gemma-4-31B-it Y 60.37 61.51 GPT-5.4-nano N 41.22 41.67 GPT-5.4-nano Y 64.36 54.76 GPT-5.6-sol N 78.99 76.59 GPT-5.6-sol Y 86.70 77.78 Human examinees 69.75 70.14 Subset statistics Examinees 2,103 2,001 Exams 47 55 Questions 376 252 Answers 11,167 6,614 Table 3: Model and human accuracy on the QA and VQA subsets, together with subset statistics. Human accuracy is the percentage of examinee answers matching the answer key. 6.2.1 QA All models perform above the random-guessing baseline of 20% on QA when given only the answer choices, without the question (C configuration). One possible explanation is that the intended question can sometimes be inferred from the answer choices, for example when the task is to identify a true statement. Alternatively, some choices may contain an intrinsic error, such as an incorrect justification, that can be detected without knowing the question. The best-performing model configuration, GPT-5.6-sol, selects the correct answer in 42.9% of cases without access to the question. 6.2.2 VQA In the choices-only configuration (C), performance is similar to that observed on QA. When the input is incomplete and either the question or the image is missing (C+I or C+Q), the models perform better than with the answer choices alone (C), but worse than with the complete input (C+Q+I). Moreover, removing the image (C+Q) is less detrimental than removing the question (C+I), which shows that the models make greater use of the question text. This pattern holds for every model except LLaVA-PLLuM, which performs relatively poorly overall. These results show that the models perform well under incomplete-information conditions, although their ability to identify the correct answer decreases as more information is removed. 6.3 Model performance by visual category 6.3.1 Image importance Table 4 reports accuracy for the pooled text-sufficient and complementary categories (0+1) and for the image-dominant category (2). Every evaluated model performs worse on image-dominant questions under both C+Q and C+Q+I. The difference under C+Q is expected because this configuration omits the image, which category 2 questions primarily require. More notably, the same ordering persists with the complete C+Q+I input. This pattern is consistent with the input-ablation results indicating that the models rely more heavily on textual cues than on visual evidence. Model R C+Q C+Q+I Image importance category 0+1 (n=146n=146) 2 (n=140n=140) 0+1 (n=146n=146) 2 (n=140n=140) LLaVA-Bielik-11B N 44.52 29.29 47.95 28.57 Qwen3.5-397B-A17B Y 76.71 45.00 76.03 52.14 Gemma-4-31B-it Y 68.49 36.43 73.97 47.14 GPT-5.4-nano N 47.26 30.00 52.05 33.57 GPT-5.4-nano Y 67.81 38.57 64.38 45.71 GPT-5.6-sol N 82.19 42.86 83.56 72.14 GPT-5.6-sol Y 80.82 54.29 84.25 73.57 Table 4: VQA accuracy by image-importance category for the C+Q and C+Q+I input configurations. Categories 0 and 1 are pooled; category 2 contains image-dominant questions. 6.3.2 Visual domains Table 5 shows accuracy across the seven top-level visual domains. WAVEFORM tends to be among the best-performing well-represented domains. These comparisons should be interpreted cautiously because the domain sizes are unequal and the relative performance patterns across domains differ between models. Model R IMAGE WAVEFORM PLOT GRAPHIC TABLE COMPOSITE OTHER Domain size n=102n=102 n=95n=95 n=23n=23 n=32n=32 n=20n=20 n=14n=14 n=3n=3 LLaVA-Bielik-11B N 35.29 50.53 34.78 28.12 25.00 42.86 0.00 Qwen3.5-397B-A17B Y 59.80 71.58 69.57 62.50 60.00 57.14 66.67 Gemma-4-31B-it Y 58.82 71.58 69.57 34.38 70.00 35.71 66.67 GPT-5.4-nano N 40.20 54.74 43.48 31.25 30.00 28.57 33.33 GPT-5.4-nano Y 56.86 62.11 60.87 37.50 55.00 14.29 100.00 GPT-5.6-sol N 80.39 83.16 69.57 62.50 75.00 78.57 100.00 GPT-5.6-sol Y 76.47 82.11 78.26 71.88 85.00 85.71 100.00 Table 5: VQA accuracy by visual domain. Questions containing panels from multiple domains contribute to each corresponding domain. 7 Conclusion In this work, we introduce the first Polish-language medical VQA benchmark, accompanied by a text-only QA control subset. Human examinees achieve nearly identical accuracy on the two subsets, indicating that the tasks are comparable in difficulty. We evaluate Polish-oriented and general-purpose open-weight models, as well as proprietary commercial systems. Although QA and VQA are similarly difficult for human examinees, most evaluated models perform worse on VQA. The input-ablation experiments further show that models derive more useful information from the question text than from the image: on VQA, they generally perform better without the image (C+Q) than without the question text (C+I). Moreover, accuracy is lower on image-dominant questions than on questions for which textual information is sufficient or complementary, even when the models receive the complete multimodal input. Together, these findings suggest that current models rely more heavily on textual cues than on visual evidence when answering Polish medical examination questions. Across both QA and VQA, the models also achieve above-chance accuracy when given only the answer choices, suggesting that the options provide textual cues that support inference even when both the question and the image are unavailable. This result complements the image-free evaluation of 3, who show that VLMs can achieve high scores on multimodal benchmarks without visual input and identify textual cues as a source of non-visual inference. Our more restrictive choices-only setting further shows that some usable signal may reside in the answer options themselves. Together, these findings call for caution when interpreting benchmark performance: high accuracy does not necessarily reflect robust medical or multimodal competence when a task remains partially solvable from incomplete inputs. This consideration is especially important in medical settings, where model outputs may have significant consequences. Limitations Our evaluation covers a selected set of models rather than the full space of currently available vision-language models. The number of open-weight and commercial systems is growing rapidly, making an exhaustive comparison impractical. We therefore focused on models that we consider representative of different relevant categories, including Polish-oriented VLMs, general multilingual VLMs, and commercial systems. We did not evaluate all possible inference settings. In particular, we did not systematically test all reasoning-effort levels, decoding configurations, or image-resolution variants. These factors may affect model performance, especially for visually demanding medical questions. Our results should therefore be interpreted as performance under the default prompting and inference setup used in this work, rather than as the maximum attainable performance of each model. We did not include vision-language models specifically trained for medicine. Previous work on Polish medical examination benchmarks has shown that general purpose LLMs can outperform models adapted to the medical domain 18. Nevertheless, this observation comes mainly from text-only evaluation and may not fully transfer to multimodal medical tasks. The dataset is limited in size and specialty coverage. The VQA subset contains 286 image-containing questions, and the distribution across specialties is uneven. Some specialties are represented by many more items than others, while specialties with few image-containing questions provide only limited evidence about model performance. At the same time, the questions are high-quality examination items prepared for specialist medical certification, which makes them a valuable resource despite their limited number. The benchmark reflects the structure of Polish board certification examinations rather than the full range of clinical practice. The questions test specialist medical knowledge under a standardized written-exam format, but they do not capture interactive patient assessment, longitudinal decision-making, procedural skills, communication, or responsibility for real-world outcomes. Ethical Considerations The questions used in this study originate from materials published by the Polish Medical Examination Center (Centrum Egzaminów Medycznych, CEM). We did not author the original examination questions; our contribution consists of collecting, processing, structuring, and evaluating them as a benchmark for multimodal medical question answering. We preserve the examination character of the items and use them only for research evaluation. Performance on written medical examinations captures only a limited part of medical competence. Becoming a licensed physician or dentist in Poland requires extensive education, supervised clinical training, practical experience, and formal certification. A model that performs well on multiple-choice exam questions should therefore not be described as equivalent to a clinician, nor should such results be used to claim that models can replace medical professionals. This limitation is particularly important for multimodal medical questions. Clinical work requires gathering information from patients, performing physical examinations, interpreting diagnostic tests in context, weighing contraindications and comorbidities, communicating uncertainty, and making decisions under incomplete information. A benchmark based on static examination items cannot evaluate these abilities comprehensively. LLMs and VLMs may be useful in medical education, information retrieval, and decision-support workflows, but they can also generate incorrect, incomplete, or misleading outputs. In medical settings, such errors may create substantial risks if model responses are treated as authoritative. Any practical deployment of these systems should therefore include oversight by qualified healthcare professionals, clear communication of model limitations, and compliance with applicable ethical, clinical, and regulatory standards. References Agrawal et al. (2018) A. Agrawal, D. Batra, D. Parikh, and A. Kembhavi Don’t just assume; look and answer: overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 4971–4980. Cited by: §2.3. Anonymous (2026) Anonymous Jako tako or fluent? presenting povisle: a polish vision-language evaluation. Note: OpenReview, ACL ARR 2026 May SubmissionAccessed: 2026-07-08 External Links: Link Cited by: §2.1. Asadi et al. (2025) M. Asadi, J. W. O’Sullivan, F. Cao, T. Nedaee, K. Fardi, F. Li, E. Adeli, and E. Ashley MIRAGE: the illusion of visual understanding in vision-language models. Cited by: §1, §2.3, §4.2, §7. Balepur et al. (2024) N. Balepur, A. Ravichander, and R. Rudinger Artifacts or abduction: how do LLMs answer multiple-choice questions without the question?. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 10308–10330. External Links: Link, Document Cited by: §1. Balepur et al. (2025) N. Balepur, R. Rudinger, and J. L. Boyd-Graber Which of these best describes multiple choice evaluation with LLMs? a) forced B) flawed C) fixable D) all of the above. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 3394–3418. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1. Ben Abacha et al. (2020) A. Ben Abacha, V. V. Datla, S. A. Hasan, D. Demner-Fushman, and H. Müller Overview of the VQA-Med task at ImageCLEF 2020: visual question answering and generation in the medical domain. In CLEF 2020 Working Notes, CEUR Workshop Proceedings, Vol. 2696. External Links: Link Cited by: §2.2. Ben Abacha et al. (2019) A. Ben Abacha, S. A. Hasan, V. V. Datla, J. Liu, D. Demner-Fushman, and H. Müller VQA-Med: overview of the medical visual question answering task at ImageCLEF 2019. In Working Notes of CLEF 2019, CEUR Workshop Proceedings, Vol. 2380. External Links: Link Cited by: §2.2. Ben Abacha et al. (2021) A. Ben Abacha, M. Sarrouti, D. Demner-Fushman, S. A. Hasan, and H. Müller Overview of the VQA-Med task at ImageCLEF 2021: visual question answering and generation in the medical domain. In CLEF 2021 Working Notes, CEUR Workshop Proceedings, Vol. 2936, p. 1081–1088. External Links: Link Cited by: §2.2. Cadene et al. (2019) R. Cadene, C. Dancette, H. Ben-Younes, M. Cord, and D. Parikh RUBi: reducing unimodal biases in visual question answering. In Advances in Neural Information Processing Systems, Vol. 32. External Links: Link Cited by: §2.3. Chen et al. (2024) J. Chen, C. Gui, R. Ouyang, A. Gao, S. Chen, G. H. Chen, X. Wang, Z. Cai, K. Ji, X. Wan, and B. Wang Towards injecting medical visual knowledge into multimodal LLMs at scale. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 7346–7370. External Links: Link, Document Cited by: §2.2. Ciesiółka and Graliński (2025) M. Ciesiółka and F. Graliński ReVISION: a polish benchmark for evaluating vision-language models on multimodal national exam data. In 2025 20th Conference on Computer Science and Intelligence Systems (FedCSIS), Vol. , p. 665–673. External Links: Document Cited by: §2.1. Das et al. (2024) R. Das, S. Hristov, H. Li, D. Dimitrov, I. Koychev, and P. Nakov EXAMS-V: a multi-discipline multilingual multimodal exam benchmark for evaluating vision language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 7768–7791. External Links: Link, Document Cited by: §2.1. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §5. Golchin and Surdeanu (2025) S. Golchin and M. Surdeanu Data contamination quiz: a tool to detect and estimate contamination in large language models. Transactions of the Association for Computational Linguistics 13, p. 809–830. External Links: Link, Document Cited by: §5. Google DeepMind (2026) Google DeepMind Gemma 4 31b it. Note: https://huggingface.co/google/gemma-4-31B-itModel card. Accessed: 2026-07-06 Cited by: §4. Gouthaman and Mittal (2020) K. V. Gouthaman and A. Mittal Reducing language biases in visual question answering with visually-grounded question encoder. In Computer Vision – ECCV 2020, Lecture Notes in Computer Science, Vol. 12358, p. 18–34. External Links: Document Cited by: §2.3. Goyal et al. (2017) Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh Making the v in vqa matter: elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 6904–6913. Cited by: §2.3. Grzybowski et al. (2025) Ł. Grzybowski, J. Pokrywka, M. Ciesiółka, J. I. Kaczmarek, and M. Kubis Polish-English medical knowledge transfer: a new benchmark and results. In Findings of the Association for Computational Linguistics: EMNLP 2025, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 9042–9063. External Links: Link, Document, ISBN 979-8-89176-335-7 Cited by: §1, Limitations. Hasan et al. (2018) S. A. Hasan, Y. Ling, O. Farri, J. Liu, M. Lungren, and H. Müller Overview of the ImageCLEF 2018 medical domain visual question answering task. In CLEF 2018 Working Notes, CEUR Workshop Proceedings, Avignon, France. External Links: Link Cited by: §2.2. He et al. (2021) X. He, Z. Cai, W. Wei, Y. Zhang, L. Mou, E. Xing, and P. Xie Towards visual question answering on pathology images. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), C. Zong, F. Xia, W. Li, and R. Navigli (Eds.), Online, p. 708–718. External Links: Link, Document Cited by: §2.2. Jassem et al. (2025) K. Jassem, M. Ciesiółka, F. Graliński, P. Jabłoński, J. Pokrywka, M. Kubis, M. Jabłońska, and R. Staruch LLMzSzŁ: a comprehensive llm benchmark for polish. arXiv preprint arXiv:2501.02266. Cited by: §2.1. Lasik et al. (2026) A. Lasik, J. Pokrywka, Ł. Grzybowski, J. I. Kaczmarek, G. Korzańska, J. Świeczkowski-Feiz, O. Pastuszek, P. Hoffman, J. T. Dąbrowski, and W. Kusa Reassessing high-performing llms on polish medical exams: true competence or bias-driven performance?. External Links: 2606.12250, Link Cited by: §1. Lau et al. (2018) J. J. Lau, S. Gayen, A. Ben Abacha, and D. Demner-Fushman A dataset of clinically generated visual questions and answers about radiology images. Scientific Data 5, p. 180251. External Links: Document, Link Cited by: §2.2. Liu et al. (2021) B. Liu, L. Zhan, L. Xu, L. Ma, Y. Yang, and X. Wu Slake: a semantically-labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), Vol. , p. 1650–1654. External Links: Document Cited by: §2.2. Malik et al. (2026) H. Malik, H. T. Aftab, A. Rehman, M. Ali, and W. Sultani Multilingual hematology visual question answering dataset. External Links: 2606.25246, Link Cited by: §2.2. Matos et al. (2025) J. Matos, S. Chen, S. K. V. Placino, Y. Li, J. C. C. Pardo, D. Idan, T. Tohyama, D. Restrepo, L. F. Nakayama, J. M. M. Pascual-Leone, G. K. Savova, H. Aerts, L. A. Celi, A. I. Wong, D. Bitterman, and J. Gallifant WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation. In Findings of the Association for Computational Linguistics: NAACL 2025, L. Chiruzzo, A. Ritter, and L. Wang (Eds.), Albuquerque, New Mexico, p. 7218–7231. External Links: Link, Document Cited by: §2.2. OpenAI (2026) OpenAI GPT-5 Model. Note: https://developers.openai.com/api/docs/models/gpt-5API documentation. Accessed: 2026-07-06 Cited by: §4. Pokrywka et al. (2024) J. Pokrywka, J. Kaczmarek, and E. Gorzelańczyk GPT-4 passes most of the 297 written polish board certification examinations. External Links: 2405.01589, Link Cited by: §1. Qwen Team (2026) Qwen Team Qwen3.5: towards native multimodal agents. External Links: Link Cited by: §4. Raj Khan et al. (2021) H. Raj Khan, D. Gupta, and A. Ekbal Towards developing a multilingual and code-mixed visual question answering system by knowledge distillation. In Findings of the Association for Computational Linguistics: EMNLP 2021, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Punta Cana, Dominican Republic, p. 1753–1767. External Links: Link, Document Cited by: §2.1. Ramakrishnan et al. (2018) S. Ramakrishnan, A. Agrawal, and S. Lee Overcoming language priors in visual question answering with adversarial regularization. In Advances in Neural Information Processing Systems, Vol. 31. External Links: Link Cited by: §2.3. Riccio et al. (2025) G. Riccio, A. Romano, M. Barone, G. M. Orlando, D. Russo, M. Postiglione, V. L. Gatta, and V. Moscato A multilingual multimodal medical examination dataset for visual question answering in healthcare. In 2025 IEEE 38th International Symposium on Computer-Based Medical Systems (CBMS), p. 435–440. External Links: Document, Link Cited by: §2.2. Statkiewicz et al. (2026) G. Statkiewicz, A. Dobrzeniecka, K. Seweryn, A. Krasnodębska, K. Piosek, K. Bogusz, S. Cygert, and W. Kusa Annotation-efficient vision-language model adaptation to the Polish language using the LLaVA framework. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 4: Student Research Workshop), S. Baez Santamaria, S. A. Somayajula, and A. Yamaguchi (Eds.), Rabat, Morocco, p. 569–589. External Links: Link, Document, ISBN 979-8-89176-383-8 Cited by: §2.1, §4. Yim et al. (2025) W. Yim, A. Ben Abacha, R. Doerning, C. Chen, J. Xu, A. Subbarao, Z. Yu, F. Xia, M. K. Hall, and M. Yetisgen WoundcareVQA: a multilingual visual question answering benchmark dataset for wound care. Journal of Biomedical Informatics 170, p. 104888. External Links: Document, Link Cited by: §2.2. Yudhistira et al. (2026) P. C. Y. Yudhistira, D. R. Malik, and N. Yudistira Does language shift break medical vision-language models? indonesian radiology visual question answering case study. Note: Accepted to MMFM-BIOMED Workshop at CVPR 2026 External Links: 2606.03693, Link Cited by: §2.2. Zhan et al. (2023) C. Zhan, P. Peng, H. Zhang, H. Sun, C. Shang, T. Chen, H. Wang, G. Wang, and H. Wang Debiasing medical visual question answering via counterfactual training. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, Lecture Notes in Computer Science, Vol. 14221, p. 382–393. External Links: Document Cited by: §2.3. Zhang et al. (2023) X. Zhang, C. Wu, Z. Zhao, W. Lin, Y. Zhang, Y. Wang, and W. Xie PMC-VQA: visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415. External Links: Link Cited by: §2.2. Zhu et al. (2025) H. Zhu, Y. Liu, C. Zhou, G. Lu, and B. Chen Med-BiasX: robust medical visual question answering with language biases. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2025, Lecture Notes in Computer Science, Vol. 15973, p. 369–378. Cited by: §2.3. Appendix A Prompts and Output Format We used only two versions of the system prompt. The first version was used when the question text was available, i.e., in the C+Q and C+Q+I configurations. The second version was used when the question text was removed, i.e., in the C and C+I configurations. The presence of an image did not change the textual prompt: in image-based configurations, the image was simply attached to the model input together with the same textual prompt used in the corresponding non-image configuration. All experiments were conducted using Polish prompts. For readability, we also provide English translations below. The English versions were not used as separate experimental prompts; they are included only as translations of the Polish prompts. A.1 System prompt with question text This prompt was used for configurations that included the question text, i.e., C+Q and C+Q+I. Polish prompt. Odpowiadasz na pytania jednokrotnego wyboru A–E z testu medycznego dla lekarzy. Wybierz dokładnie jedną odpowiedź spośród: A, B, C, D, E. Zwróć wyłącznie poprawny obiekt JSON w następującej strukturze: "response": "<LETTER>" Zastąp <LETTER> jedną wybraną literą: A, B, C, D albo E. Pole "response" musi być typu string. Bez wyjaśnień, bez komentarzy, bez dodatkowego tekstu. English translation. You answer single-choice questions with options A–E from a medical examination for physicians. Select exactly one answer from: A, B, C, D, or E. Return only a valid JSON object with the following structure: "response": "<LETTER>" Replace <LETTER> with one selected letter: A, B, C, D, or E. The "response" field must be a string. No explanations, comments, or additional text. A.2 System prompt without question text This prompt was used for configurations without the question text, i.e., C and C+I. In these settings, the model received only the answer options and, depending on the configuration, optionally the image. Polish prompt. Odpowiadasz na pytania jednokrotnego wyboru A–E z testu medycznego dla lekarzy. Nie otrzymujesz treści pytania. Masz tylko odpowiedzi A–E. Mimo braku treści pytania spróbuj wskazać najbardziej prawdopodobną poprawną odpowiedź. Wybierz dokładnie jedną odpowiedź spośród: A, B, C, D, E. Zwróć wyłącznie poprawny obiekt JSON w następującej strukturze: "response": "<LETTER>" Zastąp <LETTER> jedną wybraną literą: A, B, C, D albo E. Pole "response" musi być typu string. Bez wyjaśnień, bez komentarzy, bez dodatkowego tekstu. English translation. You answer single-choice questions with options A–E from a medical examination for physicians. You do not receive the question text. You only have the answer options A–E. Despite the absence of the question text, try to identify the most likely correct answer. Select exactly one answer from: A, B, C, D, or E. Return only a valid JSON object with the following structure: "response": "<LETTER>" Replace <LETTER> with one selected letter: A, B, C, D, or E. The "response" field must be a string. No explanations, comments, or additional text. A.3 Output schema For models supporting structured outputs, the response was constrained to a JSON object containing exactly one required field, response. This field was required to be a string and could take only one of five values corresponding to the answer options: A, B, C, D, or E. No additional fields were allowed. Appendix B Subset with Human Responses To place model performance in the context of human performance, we additionally collected anonymized candidate answers published by CEM. For each PES session, CEM publishes anonymized answer sheets of individual examinees, listing the option selected for each question, together with the official answer key. Linking these answers to our benchmark is not straightforward. Our analysis of the published materials revealed that each examination exists in two versions that differ in question numbering, while only one version of the examination sheet is published. As a consequence, a given question number in the answer statistics does not necessarily correspond to the same question in the published sheet. To ensure correct alignment, we merged the candidate answers with our question set using the official correct answer as a consistency check. For every matched item, the expected correct answer in the published sheet had to agree with the correct answer reported in the answer statistics. Since a single question can match by chance, we further restricted the human answers subset to examinations for which more than one question from our benchmark was available, and required the expected correct answers to match for all available questions from that examination. We excluded examinations that did not satisfy this condition, as we could not reliably determine which version they corresponded to. This procedure yielded a subset of questions with associated human answers, for which we computed the accuracy of examinees. Since this subset is smaller than the full benchmark, we treated the resulting value as an approximate human reference point rather than an item-level comparison with model accuracy. Appendix C Usage of GenAI in Research We used ChatGPT and Codex to assist with manuscript writing, code development, and literature searches. All generated content and outputs were reviewed and verified by the authors, who remain fully responsible for the final manuscript.