Paper deep dive
Multilingual Training and Evaluation Resources for Vision-Language Models
Daniela Baiamonte, Elena Fano, Matteo Gabburo, Stefano Simonazzi, Leonardo Rigutini, Andrea Zugarini
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 11:19:26 AM
Summary
The paper introduces a comprehensive suite of multilingual resources for Vision-Language Models (VLMs) to address the English-centric bias in current development. The authors present Multi-PixMo, a training corpus consisting of three datasets (Multi-PixMo-Cap, Multi-PixMo-AskModelAnything, and Multi-PixMo-CoSyn-400k) derived from the PixMo suite using a 'regeneration-translation' paradigm with the Qwen3-VL model. Additionally, they introduce MEVBench (Multimodal European Vision Benchmark), which consists of translated versions of popular English benchmarks (MMBench, ScienceQA, MME, POPE, and AI2D) across five European languages: English, French, German, Italian, and Spanish. The study demonstrates that multilingual training improves performance on non-English benchmarks and provides high-quality, legally compatible resources.
Entities (13)
Relation Signals (5)
MEVBench → containstranslatedversionsof → MMBench
confidence 100% · MEVBench... constructed by systematically translating and aligning a selection of widely adopted VLM evaluation datasets (MMBench, AI2D, ScienceQA, POPE, and MME)
Multi-PixMo-Cap → derivedfrom → PixMo
confidence 100% · Multi-PixMo, a training corpus obtained regenerating examples from Pixmo pre-existing datasets
Multi-PixMo → iscomposedof → Multi-PixMo-Cap
confidence 100% · Specifically, we build Multi-PixMo... The output of this process are three new datasets named Multi-PixMo-Cap, Multi-PixMo-AskModelAnything, and Multi-PixMo-CoSyn-400k
Multi-PixMo → supportstrainingof → Vision-Language Models
confidence 100% · Multi-PixMo, a multilingual VLM instruction tuning dataset
Qwen3-VL-235B-A22B-Instruct → usedforgenerationin → Multi-PixMo
confidence 100% · We perform all data generation using the permissively licensed Qwen3-VL-235B-A22B-Instruct
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on English, leading to two main limitations: (i) the lack of multilingual and multimodal datasets for training, and (ii) the scarcity of comprehensive evaluation benchmarks across languages. In this work, we address these gaps by introducing a new comprehensive suite of resources for VLMs training and evaluation spanning five European languages (English, French, German, Italian, and Spanish). We adopt a regeneration-translation paradigm that produces high-quality cross-lingual resources by combining curated synthetic generation and manual annotation. Specifically, we build Multi-PixMo, a training corpus obtained regenerating examples from Pixmo pre-existing datasets with permissively licensed models: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k. On the evaluation side, we construct a set of multilingual benchmarks derived translating widely used English datasets (MMbench, ScienceQA, MME, POPE, AI2D). We assess the quality of these resources through qualitative and quantitative human analyses, measuring inter-annotator agreement. Additionally, we perform ablation studies to demonstrate the impact of multilingual data, with respect to English only, in VLMs training. Experiments, comprising 3 different models show that using multilingual, multimodal examples for training VLMs aids is consistently beneficial on non-English benchmarks, with positive transfer to English as well.
Tags
Links
- Source: https://arxiv.org/abs/2604.18347v1
- Canonical: https://arxiv.org/abs/2604.18347v1
Trouble viewing inline? Open PDF directly →
Full Text
62,815 characters extracted from source content.
Expand or collapse full text
Multilingual Training and Evaluation Resources for Vision-Language Models Daniela Baiamonte 1 , Elena Fano 1 , Matteo Gabburo 2 , Stefano Simonazzi 1 , Leonardo Rigutini 1 , and Andrea Zugarini 1 Villanova.ai 1 , Aithlas 2 dbaiamonte,efano,ssimonazzi,lrigutini,azugarini@villanova.ai, matteo.gabburo@aithlas.com https://villanova.ai/ Abstract. Vision Language Models (VLMs) achieved rapid progress in the recent years. However, despite their growth, VLMs development is heavily grounded on English, leading to two main limitations: (i) the lack of multilingual and multimodal datasets for training, and (i) the scarcity of comprehensive evaluation benchmarks across languages. In this work, we address these gaps by introducing a new comprehensive suite of resources for VLMs training and evaluation spanning five Eu- ropean languages (English, French, German, Italian, and Spanish). We adopt a regeneration-translation paradigm that produces high-quality cross-lingual resources by combining curated synthetic generation and manual annotation. Specifically, we build Multi-PixMo, a training corpus obtained regenerating examples from PixMo pre-existing datasets with permissively licensed models: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k. On the evaluation side, we construct a set of mul- tilingual benchmarks derived translating widely used English datasets (MMbench, ScienceQA, MME, POPE, AI2D). We assess the quality of these resources through qualitative and quantitative human analyses, measuring inter-annotator agreement. Additionally, we perform ablation studies to demonstrate the impact of multilingual data, with respect to English only, in VLMs training. Experiments, comprising 3 different models show that using multilingual, multimodal examples for training VLMs aids is consistently beneficial on non-English benchmarks, with positive transfer to English as well. Keywords: Multimodal and Multilingual Resources· Multimodal Learn- ing· Visual Question Answering· Vision–Language Models 1 Introduction Recent advances in Vision–Language Models (VLMs) have led to strong im- provements across tasks such as image captioning, visual question answering, and multimodal reasoning [2, 23, 37]. Despite this rapid progress, the develop- ment of VLMs remains mainly in English. This imbalance introduces two critical arXiv:2604.18347v1 [cs.CL] 20 Apr 2026 2Baiamonte et al. limitations: (i) the scarcity of large-scale multilingual multimodal datasets for training, and (i) the lack of comprehensive cross-lingual evaluation benchmarks. Although multilingual text-only resources have become widely available [7], comparable multimodal datasets remain limited in scale, linguistic coverage, or annotation consistency [11]. As a consequence, multilingual VLMs are often trained primarily on English data and evaluated on English-centric benchmarks. Moreover, many widely adopted multimodal datasets rely on content generated using proprietary large language models and distributed under restrictive terms, making downstream reuse legally constrained [4,35]. In this work, we address both challenges by introducing a comprehensive multilingual suite for VLM training and evaluation spanning five European lan- guages: English, French, German, Italian, and Spanish. Our approach combines controlled regeneration and translation strategies to produce legally compati- ble, high-quality cross-lingual multimodal resources. On the training side, we present Multi-PixMo, a multilingual regeneration of three datasets from the PixMo suite [9]: PixMo-Cap, PixMo-AskModelAnything, and CoSyn-400k [42]. Rather than directly translating the original annotations, we adopt a regener- ation–translation paradigm in which original images and texts serve as seman- tic anchors, and a permissively licensed multimodal model is used to re-author captions, questions, and answers across languages. Importantly, English annota- tions are also regenerated under the same modeling assumptions, ensuring lin- guistic consistency and legal homogeneity across the entire corpus. The output of this process are three new datasets named Multi-PixMo-Cap, Multi-PixMo- AskModelAnything, and Multi-PixMo-CoSyn-400k (for convenience abbreviated in cap, ama and cos). On the evaluation side, we construct multilingual bench- marks by translating widely used English multimodal datasets (M-bench, Sci- enceQA, MME, POPE, AI2D), enabling systematic cross-lingual assessment of VLM performance. Together, these resources provide a unified framework for multilingual training and evaluation under consistent licensing conditions. Our contributions are threefold: (1) we introduce a large-scale multilingual suite for VLM training and evaluation covering five languages under unified, per- missive licensing; (2) we propose a controlled multimodal regeneration framework that balances semantic fidelity, linguistic quality, and legal reusability and (3) we provide extensive human evaluation and empirical evidence showing that multilingual training data improves performance on non-English multimodal benchmarks. By jointly addressing multilingual coverage, evaluation diversity, and licensing constraints, our work provides a practical and scalable foundation for open multilingual vision–language research. 2 Related Works Training Corpora for Vision Language Models. The development of modern Vision Language Models (VLMs) has been largely enabled by large- scale image-text and instruction-tuning corpora. Contrastive pretraining ap- proaches such as CLIP [37] demonstrated the effectiveness of web-scale super- Multilingual Training and Evaluation Resources3 Fig. 1: Examples from cap (top) and ama (bottom). The original PixMo annotation in English is shown first, followed by our regenerated version: French caption for cap; QA pair in Italian for ama. vision, leveraging noisy multilingual text-image pairs without curated question- answer alignment. Although few open models fully disclose their complete train- ing pipelines, several public resources provide insight into current practices. LLaVA-OneVision [29] releases a large mixture including visual instruction data and academic benchmarks. InternVL builds upon curated multimodal mixtures such as MMPR-v1.2 [34]. The IDEFICS family relies in part on OBELICS [22], a large-scale interleaved image-text corpus filtered from the web. SmolVLM and re- lated open efforts (e.g., The Cauldron, Docmatix) [18,19] aggregate instruction- tuning datasets spanning captioning, reasoning, chart understanding, and doc- ument analysis. Additional resources target specific tasks, such as rendered or OCR-rich text images [41], as well as video-language corpora including FineVideo and LLaVA-Video-178K [17, 30]. While these collections significantly diversify multimodal supervision, multilingual coverage is often low and unbalancedand most instruction datasets remain in English [5,35]. In this context, we present Multi-PixMo, a multilingual VLM instruction tuning dataset for English, French, German, Italian, and Spanish. Built on a unified protocol rather than fragmented corpora, it focuses on controlled gener- ative tasks to enable consistent cross-lingual alignment and robust open-ended reasoning. Multimodal Benchmarks and Multilingual Evaluation. On the evalua- tion side, widely used multimodal benchmarks such as VQA v2 [14], GQA [3], MMBench [27], and AI2D [21] have become standard testbeds for visual reason- ing and question answering. More recently, the scope of evaluation has expanded beyond generic VQA to encompass text-rich and document-centric scenarios. TextVQA [38] requires reasoning over scene text, ChartQA [32] focuses on nu- merical and logical reasoning over charts, and DocVQA [33] addresses structured document understanding. ScienceQA [31] evaluates multimodal scientific reason- 4Baiamonte et al. ing across school-level questions, while POPE [25] probes object hallucination by testing whether model outputs are visually grounded. MME [12] provides a comprehensive assessment of perception and cognition abilities through diverse multimodal tasks. Complementary resources such as AI2D [21] and comprehen- sive suites like MMBench [27] further stress diagram understanding, multi-step reasoning, OCR integration, and knowledge grounding, reflecting the increas- ing breadth of capabilities expected from modern VLMs. Additionally, recent VLMs such as Qwen-VL [36], Gemma [13] and SmolVLM [20] typically report performance on composite benchmark suites combining VQA-style reasoning, OCR-intensive tasks, chart interpretation, and document understanding. However, most available datasets and benchmarks remain predominantly English-centric, with only a limited number of resources explicitly designed for multilingual evaluation. Although several benchmarks, such as MTVQA [39], ALMBench [24], CVQA [28], and MMMB [44], aim to assess multilingual vi- sual understanding across languages, their coverage of European languages re- mains limited. To address this limitation, we introduce a multilingual benchmark suite named MEVBench (Multimodal European Vision Benchmark) constructed by systematically translating and aligning a selection of widely adopted VLM evaluation datasets (MMBench, AI2D, ScienceQA, POPE, and MME) into five languages (English, French, German, Italian and Spanish). 3 Training Resources In this Section we present Multi-PixMo-Cap, Multi-PixMo-AskModelAnything, and Multi-PixMo-CoSyn-400k, three multilingual datasets derived from PixMo [9, 42] covering captioning, open-ended VQA, and text-rich image reasoning. We will also refer to them respectively as cap, ama and cos. Rather than translating the original annotations, we adopt a regeneration/translation paradigm in which we leverage a permissively licensed multimodal model to rewrite all the examples of the original dataset, conditioning on both the image and the source text and ensuring linguistic quality and legal homogeneity. The resulting corpus spans over 1.1M images and 2.8M examples across five languages (Table 2). In Sec. 3.1 we present our data generation pipeline, explaining in detail how we gen- erate these new datasets, while in Sections 3.2, 3.4 we provide quantitative and qualitative analysis of the data. The agreement between annotators is discussed in Section 3.3. 3.1 Data Generation Our pipeline combines two strategies: translation, which converts text across languages without visual grounding, and regeneration, which conditions a multi- modal model on both the image and the source text. We apply translation only to linguistically self-contained inputs (e.g. questions), and regeneration wherever visual grounding can improve quality, allowing the model to verify and disam- biguate from the image. We perform all data generation using the permissively Multilingual Training and Evaluation Resources5 licensed Qwen3-VL-235B-A22B-Instruct [36]. To ensure high-quality prompts, we involve native and expert speakers in a systematic refinement pipeline con- sisting of iterative generation cycles followed by blind reviews from independent QA annotators, until convergence to an optimal configuration. We generated all three datasets with a language distribution of 40% English, while French, German, Italian, and Spanish each account for 15%. Multi-PixMo-Cap. Each image and its original English speech-to-text tran- scription are jointly provided to the model, which regenerates the caption in the target language under a strict grounding constraint: the image may only be used to verify or disambiguate content present in the transcription, not to introduce new information. This design choice allows to mitigate transcription noise while preserving the fidelity to the original human description. The same protocol is applied to English ensuring uniformity across languages. The resulting dataset contains 655,772 captions over 655,306 images (Table 2). Multi-PixMo-AskModelAnything. The original human-authored questions are translated into each target language; answers are then regenerated by the multimodal model conditioned on the image and the translated question. En- glish answers are also regenerated from the original questions, ensuring uniform modeling assumptions and licensing conditions across languages. During devel- opment, we found that ∼40% of counting-related answers contained numerical errors; all these entries were manually corrected by annotators with access to the image. Counting-related entries are excluded from the quality evaluation (Sec- tion 3.2) to avoid confounding effects from this systematic error category. The final dataset comprises 124,714 QA triples (Table 2). Multi-PixMo-CoSyn-400k. CoSyn-400k [42] provides text-rich synthetic im- ages (charts, diagrams, tables, etc.) whose original QA pairs were generated from code without access to the rendered image, making them unsuitable as seman- tic anchors. For this reason, we adopt a pure regeneration setting: the model is shown each image and prompted to generate QA pairs directly in the target language. We filtered out empty or excessively long outputs, as both correlate with low-quality images inherited from the original rendering pipeline. The final dataset contains 365,615 images and over 2M QA pairs (Table 2). 3.2 Quality Assessment: Quantitative Analysis To assess annotation quality, we conduct a human evaluation for each dataset and each language. The evaluation is carried out by two expert annotators per language. Although the number of evaluators is limited, this setup prioritizes expertise and consistency, ensuring informed and reliable judgments. Annotation Guidelines. Each reviewer evaluated 125 examples, with all judg- ments being binary (correct/incorrect). The specific dimensions assessed vary by 6Baiamonte et al. Table 1: Example of Multi-CoSyn-400K. The figure shows our regenerated question- answer pairs in German, along with their English translation. ImageRegenerated Example English translation Q: Was ist der Titel des Diagramms? A: Distribution of Senior Badminton Players by Age Group Q: Welche Altersgruppe hat den höchsten Prozentsatz an Spielern? A: 60-64 Q: Wie hoch ist der Prozentsatz der Spieler in der Altersgruppe 70-74? A: 21% Q: Welche Altersgruppe hat den niedrigsten Prozentsatz an Spielern? A: 85+ Q: Wie viele Altersgruppen sind im Diagramm dargestellt? A: 6 Q: What is the title of the diagram? A: Distribution of Senior Badminton Players by Age Group Q: Which age group has the highest percentage of players? A: 60-64 Q: What is the percentage of players in the age group 70-74? A: 21% Q: Which age group has the lowest percentage of players? A: 85+ Q: How many age groups are represented in the diagram? A: 6 Table 2: Language distribution across the three Multi-PixMo datasets. LangcapamacosTotal Images Examples Images Examples Images Examples Images Examples EN 262,107 262,293 22,583 49,709 149,149 823,790 433,839 1,135,792 IT98,287 98,354 8,482 18,774 54,742 302,372 161,511 419,500 FR98,303 98,380 8,470 18,691 53,433 298,917 160,206 415,988 ES98,313 98,376 8,520 18,877 56,073 314,255 162,906 431,508 DE 98,296 98,369 8,425 18,663 52,218 288,270 158,939 405,302 Total 655,306 655,772 56,480 124,714 365,615 2,027,604 1,077,401 2,808,090 dataset. For cap, annotators provided two judgments per example: (i) adher- ence of the caption to the image and (i) linguistic correctness. For ama, three judgments were needed: (i) translation quality of the question (accuracy and naturalness in the target language), (i) content quality of the answer (factual correctness and consistency with image and question), and (i) form quality of the answer. Separating content from form proved necessary, as annotations can be visually accurate but linguistically flawed, or vice versa, enabling finer- grained diagnosis of failure modes. In Multi-CoSyn-400K, each image is as- sociated with multiple question–answer pairs. However, to simplify the task for annotators, we ask them to evaluate each pair independently. The nature of the dataset, with short and concise questions and equally brief answers (often a sin- gle number or word), makes separate evaluation of content and form redundant, so the annotators were asked to give a single comprehensive quality judgment for each pair. Datasets Quality. Results of the human evaluation are outlined in Table 3. Overall, we can observe high quality in the linguistic form of the generated captions/answers, with percentages of correct evaluations beyond 96% in most cases. Translating questions also leads to analogous results. Judgments about content adherence and faithfulness with respect to image present positive rating as well, usually above 85%. Scores are slightly lower in French, but still in a Multilingual Training and Evaluation Resources7 positive range close to 80%. The quality of generated question and answer pairs in cos is also above 90% for all languages. Table 3: Percentage of positive judgments per language and dimension (values in %). capamacos Lang Content Form Question Content Form Content EN 83.2 99.6—82.4 99.2 93.1 IT96.4 99.6 95.687.6 91.2 95.6 ES96.8 100 96.889.2 96.4 93.2 FR 79.2 96.4 97.677.2 96.4 93.7 DE 92.0 95.2 99.692.8 98.8 91.2 3.3 Annotators Agreement Among the 125 examples judged by each annotator, 50 were assigned to both annotators for each language to assess inter-annotator agreement. Metrics. The inter-annotator agreement is measured with the Gwet’s AC1 [16] metric. This choice is motivated by the nature of our task, which involves binary judgments and results in highly imbalanced label distributions. Under these conditions, standard agreement metrics may yield unstable or misleading scores. The analysis shows consistently strong agreement among annotators (Ta- ble 4) in all dataset and evaluation types, with scores typically varying from 0.8 to 1.00. Inspecting single languages, agreements in English, Italian and Spanish are particularly high. Disagreements are mainly concentrated in French, and to a lesser extent German, but still remaining in a good range of values. Results. Qualitative inspection suggests that most of these differences tend to arise in borderline or ambiguity-sensitive cases, such as subjective visual inter- pretation like color descriptions (e.g., whether an object should be described as white or gray); minor spatial or perceptual ambiguities; differences in annotator tolerance for plausible but not fully verifiable details of the image. 3.4 Quality Assessment: Qualitative Analysis To complement the quantitative analysis, we performed a systematic inspection of negatively rated examples across all three datasets and languages, aiming to identify recurring error patterns and to trace each error to its source: the original data, known limitations of current VLMs, or our regeneration pipeline. Errors inherited from source data. A substantial portion of content errors does not originate from our pipeline but is inherited from upstream data. In cap, the input transcriptions are produced by an automatic speech-to-text system ap- plied to spoken descriptions, which introduces noise, disfluencies, and occasional 8Baiamonte et al. Table 4: Inter-annotator agreement (Gwet’s AC1) by dataset and annotation dimen- sion: caption content and form for cap; question translation (Question), answer content (Content), and answer form (Form) for ama; and QA pair correctness for cos. capamacos LangContent Form AvgQuestion Content Form AvgContent EN0.81 1.00 0.91—0.87 1.00 0.940.94 IT0.98 0.98 0.980.930.90 0.88 0.900.95 ES 0.98 1.00 0.990.980.78 0.96 0.910.93 FR0.68 0.89 0.790.980.73 0.89 0.870.93 DE 0.83 0.81 0.821.000.86 1.00 0.950.88 inaccuracies in references to visual content. Our regeneration model, constrained to ground its output in these transcriptions, propagates such artifacts when it cannot resolve them against the image. In cos, image-level quality issues—such as cropped content, low-contrast labels, or excessively small fonts—stem from the original synthetic rendering pipeline [42] and affect downstream QA gener- ation. The qualitative analysis confirms this: image acceptability (about 88%) sets some limits to QA correctness, as poorly rendered images limit the model’s ability to produce accurate answers. It must be said that sometimes the model can generate meaningful QA pairs even when the image data is not clearly in- terpretable, mainly by focusing on surface details of the image. Errors reflecting known VLM limitations. The remaining content er- rors are largely consistent with well-documented limitations of current vision– language models [12, 25]. Across all three datasets, we observe recurring is- sues with spatial reasoning (e.g., confusion between left and right), numerical counting, color identification under ambiguous lighting, and unverified inferences where the model fills gaps in the input with plausible but ungrounded details (Figure 2). These error categories are not specific to our regeneration setup; rather, they reflect the current capability frontier of multimodal models. No- tably, for ama, we identified counting-related questions as a systematic failure mode (approximately 40% of counting answers contained numerical inaccuracies) and manually corrected all such entries, as described in Section 3.1. Evidence of error correction. An important finding is that our multimodal regeneration setup can correct errors present in the original annotations. Since the model has access to both the transcription and the image, it can resolve transcription artifacts when visual evidence is unambiguous. Figure 1 illustrates this: the text printed on a worker’s shirt, inaccurately transcribed as “Hauss- mann Supply” in the original cap dataset, is correctly rendered as “HOSMER SUPPLY” in the regenerated French caption by leveraging the visual modality. Conciseness of regenerated annotations. Besides specific error types, we observed that regenerated captions and answers tend to be shorter than the Multilingual Training and Evaluation Resources9 Fig. 2: Unverified inferences in the French caption (are the crab’s legs made of papier- mâché or not?—first picture from the left) and in the Spanish answer (is chicken one of the ingredients?—second picture from the left). Counting error (two feet visible instead of one) in the French answer (third picture from the left) and ambiguous color (white or grey?) in the German caption (first picture from the right). original ones. On English, the regenerated texts are on average 30–40% shorter. This reflects the concise style of Qwen3-VL-235B-A22B-Instruct, which tends to omit secondary details. Nevertheless, the resulting captions remain substantially more descriptive than those found in traditional image-captioning datasets, often consisting of a single short sentence. 4 Evaluation Benchmarks To assess the multilingual capabilities of our models, we constructed an evalua- tion suite named MEVBench (Multimodal European Vision Benchmark) made of benchmarks by extending widely adopted Vision–Language benchmarks. Ad- ditionally, we reserve held-out examples from Multi-PixMo. We selected a set of reference benchmarks that are commonly used in the literature and whose licenses allow redistribution and adaptation. The selected datasets cover differ- ent tasks and vision–language abilities. Concretely, we extended the following datasets: ScienceQA [31], MMBench [27], AI2D [21], POPE [25], MME [12]. Since these benchmarks are in English, we translated each example into French, German, Italian, and Spanish. Given the critical importance of evaluation qual- ity, and the fact that these data are used exclusively for benchmarking already trained models, we prioritized translation accuracy. To this end, we employed state-of-the-art Large Language Models. In particular, we translated MMBench with gemini-3-flash-preview, AI2D with gemini-2.5-flash [6] and MME, POPE and ScienceQA with gpt-4.1-mini [1]. The resulting multilingual bench- marks are referred to with the Multi- prefix. As summarized in Table 5, these datasets collectively comprise about 150k examples spanning multiple task for- mats, including multiple-choice questions (MCQ), counting tasks and binary (yes/no) classification. Exact Match (EM) is used as the primary metric for 10Baiamonte et al. Table 5: Multilingual Evaluation Benchmarks. Benchmark Task Metric #Examples Translated ScienceQA MCQEM21,205 MMBench MCQEM21,645 AI2DMCQEM15,440 POPEY/NEM45,000 MMEY/N MME Score 11,870 most tasks, while MME Score is adopted for Multi-MME in accordance with the original evaluation protocol. 5 Experiments We evaluate Multi-PixMo along different dimensions: (i) measuring whether mul- tilingual training degrades English performance, (i) quantifying the gains of multilingual training on non-English tasks. Additionally, we validate the gen- erated annotations through supervised evaluation on held-out test splits of the datasets presented in Section 3. 5.1 Experimental Setup We experiment with three language models, spanning on different sizes and archi- tectures. In detail, we consider LLaMA3-3B [15], Qwen3-4B [40], and LLaMA3-8B [15]. We trained the models following the LLaVA [26] two-stage training recipe, and we conduct all the experiments isolating the contribution of the training data in order to better comprehend and analyze the benefits of each dataset in a more granular fashion. In detail, the LLaVA architecture is composed by three components which are the vision encoder, the projection layer and the language model. The first training stage (S1) trains a projection layer between the vision encoder and the language model usually using image-caption datasets, while the second stage (S2) trains the projection layer and the language model on tasks involving text and images (e.g., Vision Question Answering (VQA)). We use cap for S1, and ama and cos for S2, comparing the results against symmetrical baselines trained using the original English-only AllenAI datasets. In the supplementary materials we conduct an ablation on the relative impor- tance of S1 and S2, and we confirm that S2 is the primary driver of multilin- gual gains, while multilingual caption alignment in S1 provides a consistent, but smaller, benefit. To isolate contribution of each resource, we define four training configura- tions, each specified as an S1→ S2 pipeline. In particular, we denote each training run as an S1→ S2 pipeline, where S1 specifies the captioning dataset used for the projection layer alignment, and S2 the instruction-tuning mixture. We use cap, ama, and cos as shorthands for Multilingual Training and Evaluation Resources11 Multi-PixMo-Cap, Multi-PixMo-AskModelAnything, and Multi-PixMo-CoSyn- 400k, respectively, with subscripts en and mu indicating whether the English- only or the full multilingual version is used. For example, cap mu → ama en de- notes multilingual alignment followed by English-only instruction tuning. In our experiments, we compare four settings: a monolingual baseline (cap en → ama en ), its multilingual counterpart (cap mu → ama mu ), and two extended con- figurations that augment S2 with CoSyn data (cap en → ama en + cos en and cap mu → ama mu + cos mu ). When combining Multi-PixMo-AskModelAnything with Multi-PixMo-CoSyn-400k, we randomly subsample Multi-PixMo-CoSyn- 400k to obtain a balanced 50/50 mixture. Training details. All models use the LLaVA architecture with SigLIP 1 as the vision encoder (384×384 resolution, 729 image tokens) and a randomly initialized two-layer MLP projector. For S1, we use a constant learning rate of 1e− 3, an effective batch size of 256, and a maximum sequence length of 2048 tokens for 2 epochs. For S2, we use a cosine schedule with 200 warmup steps, a peak learning rate of 2e−5, weight decay of 0.01 and an effective batch size of 128. All training is conduceted in bf16 mixed precision on 8× NVIDIA H100 80GB GPUs. We perform the evaluation with VLMEvalKit [10] using heuristic answer extraction (no LLM-as-a-judge) and greedy decoding. For AI2D, ScienceQA and MMBench, we report the accuracy, while for POPE we report F1 score on binary yes/no answers, while for MME, we used the original composite score. 5.2 Results Impact of Multilingual Training on English Benchmarks. A natural concern when introducing multilingual data is whether it comes at the cost of English performance. Table 6 addresses this question by reporting the perfor- mance on the English split of our translated benchmarks, measuring whether multilingual training causes regression on monolingual tasks. The results show that multilingual training does not degrade English accu- racy. For example, considering only the CAP→AMA settings, LLaMA3-3B dif- fers by at most 0.5 points on almost every benchmark with the only exception of MME (where this difference is not noticeable due to the MME’s composite score). Qwen3-4B, exhibits a similar pattern with the multilingual model matching the English one on AI2D and MMBench, even marginally improving on ScienceQA. Major improvements emerge for LLaMA3-8B where multilingual training yields gains of +5.4 on AI2D, +3.5 on ScienceQA, and +365.9 on MME. The same behavior holds when introducing cos to the second stage of the training. cos is not always beneficial for the performance on the benchmarks, but its relative comparison with English-only CoSyn-400K does introduce degradations. Impact of Multilingual Training on Multilingual Benchmarks. Tables 7 and 8 report per-language results on multiple languages (DE, ES, FR, IT), 1 google/siglip-so400m-patch14-384 12Baiamonte et al. Table 6: English performance on our translated benchmarks under four training con- ditions: English-only (en), Multilingual (mu), and their enhanced versions with Multi- CoSyn-400K (+cos mu ). Best result per model group is bold. ModelTrainAI2D ScienceQA MMBench POPE MME LLaMA3-3B cap en → ama en 51.770.872.081.8 1213.4 cap mu → ama mu 51.270.972.182.3 1421.1 cap en → ama en + cos en 53.070.071.084.8 1350.5 cap mu → ama mu + cos mu 52.669.671.084.8 1350.5 Qwen3-4B cap en → ama en 64.377.180.285.4 1557.7 cap mu → ama mu 64.177.479.882.8 1498.4 cap en → ama en + cos en 64.078.081.085.1 1681.7 cap mu → ama mu + cos mu 64.477.680.685.1 1681.7 LLaMA3-8B cap en → ama en 52.168.575.181.4 1208.9 cap mu → ama mu 57.572.075.482.9 1574.8 cap en → ama en + cos en 51.470.572.080.8 1257.1 cap mu → ama mu + cos mu 55.968.373.981.5 1547.9 respectively, for Multiple Choice and Yes/No benchmarks tasks. Multilingual training show consistent improvements across all languages and models. For example, in Table 7, MMBench cap mu → ama mu outperforms cap en → ama en by +1.5 points on average for LLaMA3-3B, +0.7 for Qwen3-4B, and +1.0 for LLaMA3-8B. On AI2D, the gains range from +2.0 for LLaMA3-3B to +6.3 for LLaMA3-8B, while Qwen3-4B shows equivalent average performance (60.4) with minor per-language variations. Table 8 presents similar patterns for POPE and MME as well, with the exception of MME with LLaMA3-3B where English- only training outperforms the other strategies. Including cos to the training mixture yields additional gains on benchmarks aligned with the cos domain. For example, we achieved the best scores on AI2D for both LLaMA3-3B and Qwen3-4B. This results is expected since cos is designed for diagrams/charts understanding. In contrast, cos does not help on ScienceQA, LLaMA3-3B with cap mu → ama mu + cos mu drops by −2.1 points relative to cap mu → ama mu . This suggests that the domain mismatch between synthetic chart data and science questions can be detrimental for smaller models. When comparing the monolingual cos variant against the multilingual one, we observe that the latter consistently outperforms the former confirming the beneficial effect coming from the linguistic diversity rather than the training data augmentation. Overall, the analysis indicates that multilingual training generally improves multilingual performance across models, benchmarks and languages. In addi- tion, we notice that the gains are distributed across languages rather than being driven by a single one, indicating that the multilingual training mixture provides balanced cross-lingual supervision. 6 Conclusion In this work, we addressed two major limitations that currently affect the devel- opment of Vision–Language Models: the strong reliance on English-centric re- Multilingual Training and Evaluation Resources13 Table 7: Non-English performance on MCQ benchmarks. Models are evaluated under four training conditions: English-only (en), Multilingual (mu), and their versions with Multi-CoSyn-400K (+cos mu ). Results are averaged over the four languages. Best per model group is bold. Bench ModelTrainDE ES FR IT Avg AI2D LLaMA3-3B cap en → ama en 42.4 44.4 45.2 44.0 44.0 cap mu → ama mu 44.3 46.3 47.7 45.6 46.0 cap en → ama en + cos en 45.8 46.5 46.6 44.9 46.0 cap mu → ama mu + cos mu 46.9 48.6 49.0 47.0 47.9 Qwen3-4B cap en → ama en 59.1 62.3 60.2 59.9 60.4 cap mu → ama mu 58.8 61.4 60.6 60.9 60.4 cap en → ama en + cos en 59.8 62.7 61.7 61.1 61.3 cap mu → ama mu + cos mu 60.3 63.4 62.0 63.4 62.3 LLaMA3-8B cap en → ama en 44.6 46.3 46.1 45.2 45.5 cap mu → ama mu 49.9 52.6 52.6 51.9 51.8 cap en → ama en + cos en 45.0 47.7 47.8 46.1 47.6 cap mu → ama mu + cos mu 48.7 53.3 52.5 51.1 52.3 ScienceQA LLaMA3-3B cap en → ama en 67.4 67.1 66.6 66.4 66.9 cap mu → ama mu 68.3 66.7 66.9 66.8 67.2 cap en → ama en + cos en 67.3 66.2 65.6 66.2 66.3 cap mu → ama mu + cos mu 65.4 65.5 64.7 64.8 65.1 Qwen3-4B cap en → ama en 74.4 74.3 75.9 75.0 74.9 cap mu → ama mu 75.1 73.7 76.3 75.6 75.2 cap en → ama en + cos en 73.2 74.0 75.2 73.8 74.0 cap mu → ama mu + cos mu 76.0 75.5 76.8 76.5 76.2 LLaMA3-8B cap en → ama en 65.6 65.8 65.2 65.0 65.4 cap mu → ama mu 68.3 67.8 68.9 66.6 67.9 cap en → ama en + cos en 66.9 67.2 66.3 67.4 67.7 cap mu → ama mu + cos mu 63.2 65.3 65.6 65.0 65.5 MMBench LLaMA3-3B cap en → ama en 67.8 68.9 69.6 67.6 68.5 cap mu → ama mu 70.0 69.9 70.2 69.7 70.0 cap en → ama en + cos en 67.6 67.7 68.1 66.5 67.5 cap mu → ama mu + cos mu 68.7 69.0 69.5 68.6 69.0 Qwen3-4B cap en → ama en 78.4 78.0 77.9 79.0 78.3 cap mu → ama mu 78.8 79.0 79.0 79.2 79.0 cap en → ama en + cos en 77.5 77.5 78.2 78.1 77.8 cap mu → ama mu + cos mu 79.4 79.2 79.8 79.9 79.6 LLaMA3-8B cap en → ama en 72.0 72.9 72.6 71.8 72.3 cap mu → ama mu 73.1 73.4 73.4 73.2 73.3 cap en → ama en + cos en 69.4 69.2 69.1 69.1 69.8 cap mu → ama mu + cos mu 72.8 72.7 72.2 72.1 72.7 14Baiamonte et al. Table 8: Non-English performance on Yes/No benchmarks. Models are evaluated under four training conditions: English-only (en), Multilingual (mu), and their ver- sions with Multi-PixMo-CoSyn-400k (+cos mu ). Results are averaged over the four languages. Best per model group is bold. Bench ModelTrainDE ES FR IT Avg POPE LLaMA3-3B cap en → ama en 81.7 81.3 79.6 82.7 81.3 cap mu → ama mu 84.0 80.9 84.5 81.5 82.7 cap en → ama en + cos en 66.7 67.1 66.7 66.7 66.8 cap mu → ama mu + cos mu 80.7 83.3 74.3 83.2 80.4 Qwen3-4B cap en → ama en 84.5 84.0 85.1 84.1 84.4 cap mu → ama mu 82.2 82.0 84.3 83.8 83.1 cap en → ama en + cos en 78.2 79.4 78.9 74.4 77.7 cap mu → ama mu + cos mu 84.4 84.4 85.7 85.4 85.0 LLaMA3-8B cap en → ama en 85.5 79.5 82.6 81.6 82.3 cap mu → ama mu 81.5 82.6 82.9 82.4 82.3 cap en → ama en + cos en 83.3 80.9 76.0 81.9 80.6 cap mu → ama mu + cos mu 80.4 82.0 78.9 79.7 80.5 MME LLaMA3-3B cap en → ama en 1256.7 1405.2 1427.2 1476.4 1391.4 cap mu → ama mu 1348.7 1109.3 1335.4 1137.6 1232.8 cap en → ama en + cos en 604.6 1281.2 710.1 1098.2 923.5 cap mu → ama mu + cos mu 626.7 1325.5 1304.1 1300.1 1139.1 Qwen3-4B cap en → ama en 326.6 1578.3 1276.2 1669.7 1212.7 cap mu → ama mu 727.6 1425.9 1533.3 1558.8 1311.4 cap en → ama en + cos en 720.0 1471.7 1241.3 1259.6 1173.2 cap mu → ama mu + cos mu 491.4 1533.1 1615.3 1684.0 1331.0 LLaMA3-8B cap en → ama en 872.8 1167.8 1185.6 1258.4 1121.2 cap mu → ama mu 686.8 1379.3 1424.4 1212.5 1175.8 cap en → ama en + cos en 1244.5 1475.2 1223.4 1464.1 1332.9 cap mu → ama mu + cos mu 1502.3 1526.2 1532.2 1538.0 1529.3 sources and the lack of multilingual multimodal benchmarks. To mitigate these issues, we introduced a comprehensive suite of multilingual training and eval- uation resources covering five European languages: English, French, German, Italian and Spanish. We proposed a regeneration–translation paradigm that combines curated synthetic generation with manual annotation to produce high-quality multi- lingual multimodal datasets with openly licensed LLMs. Using this approach, we constructed Multi-PixMo, a multilingual training corpus derived from exist- ing PixMo resources, and a set of multilingual evaluation benchmarks, Multi- modal European Vision Benchmark(MEVBench), obtained by translating sev- eral widely used English datasets. Through qualitative and quantitative human evaluation, including inter-annotator agreement analysis, we verified the relia- bility and linguistic quality of the generated resources. Our experimental results demonstrate that incorporating multilingual multimodal data during training consistently improves performance on MEVBench, while also yielding positive transfer to English. This trend holds across different model backbones, suggest- ing that multilingual supervision can enhance the generalization capabilities of Vision–Language Models without compromising performance in English. Overall, the resources introduced in this work provide a practical step toward more inclusive and linguistically diverse multimodal models. By releasing both the training corpus and the evaluation benchmarks, we aim to facilitate future Multilingual Training and Evaluation Resources15 research on multilingual VLMs and encourage the development of models that can operate effectively across languages and cultural contexts. References 1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023) 2. Alayrac, J.B., Donahue, J., Luc, P., et al.: Flamingo: A visual language model for few-shot learning. In: NeurIPS (2022) 3. Antol, S., Agrawal, A., Lu, J., et al.: Vqa: Visual question answering. In: ICCV (2015) 4. Birhane, A., Prabhu, V.U.: Large image datasets: A pyrrhic win for computer vision? arXiv preprint arXiv:2006.16923 (2021) 5. Birhane, A., et al.: The forgotten margins of ai ethics. FAccT (2021) 6. Comanici, G., Bieber, E., Schaekermann, M., Pasupat, I., Sachdeva, N., Dhillon, I., Blistein, M., Ram, O., Zhang, D., Rosen, E., et al.: Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261 (2025) 7. Conneau, A., Khandelwal, K., Goyal, N., et al.: Unsupervised cross-lingual repre- sentation learning at scale. In: ACL (2020) 8. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., Stoyanov, V.: Unsupervised cross-lingual representation learning at scale. CoRR abs/1911.02116 (2019), http://arxiv. org/abs/1911.02116 9. Deitke, M., Clark, C., Lee, S., et al.: Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2025) 10. Duan, H., Yang, J., Qiao, Y., Fang, X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., et al.: Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In: Proceedings of the 32nd ACM International Conference on Multimedia. p. 11198–11201 (2024) 11. Elliott, D., Frank, S., Barrault, L., Bougares, F., Specia, L.: Multi30k: Multilingual english-german image descriptions. In: VL@ACL (2016) 12. Fu, C., et al.: Mme: A comprehensive evaluation benchmark for multimodal large language models. In: Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track (2023) 13. Google: Gemma 3 technical report. HuggingFace model card (2024) 14. Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter. International Journal of Computer Vision 127(4), 398–414 (2019) 15. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Let- man, A., Mathur, A., Schelten, A., Vaughan, A., et al.: The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024) 16. Gwet, K.L.: Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology 61(1), 29–48 (2008). https://doi.org/10.1348/000711006X126600 17. HuggingFaceFV: Finevideo dataset. HuggingFace dataset (2024) 18. HuggingFaceM4: The cauldron. HuggingFace dataset (2024) 19. HuggingFaceM4: Docmatix. HuggingFace dataset (2024) 16Baiamonte et al. 20. HuggingFaceTB: Smolvlm-500m-instruct. HuggingFace model card (2024) 21. Kembhavi, A., et al.: A diagram is worth a dozen images. In: ECCV (2016) 22. Laurençon, H., et al.: Obelics: An open web-scale filtered dataset of interleaved image-text documents (2023) 23. Li, J., et al.: Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation. arXiv preprint arXiv:2201.12086 (2022) 24. Li, X., Zhang, Y., Chen, W., Zhao, T., Liu, Y.: Alm-bench: A benchmark for evaluating multilingual vision-language models. arXiv preprint arXiv:2406.01268 (2024) 25. Li, Y., et al.: Evaluating object hallucination in large vision-language models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2023) 26. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023) 27. Liu, S., et al.: Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281 (2023) 28. Liu, X., et al.: Cvqa: A benchmark for cross-lingual visual question answering. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2023) 29. LMMS Lab: Llava-onevision data. HuggingFace dataset (2024) 30. LMMS Lab: Llava-video-178k. HuggingFace dataset (2024) 31. Lu, P., Mishra, S., Xia, T., et al.: Learn to explain: Multimodal reasoning via thought chains for science question answering. In: Advances in Neural Information Processing Systems (NeurIPS) (2022) 32. Masry, A., et al.: Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In: ACL (2022) 33. Mathew, M., et al.: Docvqa: A dataset for vqa on document images. In: WACV (2021) 34. OpenGVLab: Mmpr-v1.2 dataset. HuggingFace dataset (2024) 35. Paullada, A., et al.: Data and its (dis)contents: A survey of dataset development and use in machine learning research. Patterns (2021) 36. Qwen Team: Qwen3-vl technical report. HuggingFace model card (2024) 37. Radford, A., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning (2021) 38. Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Parikh, D., Rohrbach, M., Batra, D.: Towards vqa models that can read. In: CVPR (2019) 39. Tang, Y., Li, J., Li, D., Wang, Y., et al.: Mtvqa: Benchmarking multilingual text- centric visual question answering. arXiv preprint arXiv:2405.11985 (2024) 40. Team, Q.: Qwen3 technical report (2025), https://arxiv.org/abs/2505.09388 41. Wendler, C.: Renderedtext dataset. HuggingFace dataset (2023) 42. Yang, Y., Patel, A., Deitke, M., Gupta, T., Weihs, L., Head, A., Yatskar, M., Callison-Burch, C., Krishna, R., Kembhavi, A., Clark, C.: Scaling text-rich im- age understanding via code-guided synthetic multimodal data generation. arXiv preprint arXiv:2402.X (2025), https://arxiv.org/abs/2502.14846 43. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert (2020), https://arxiv.org/abs/1904.09675 44. Zheng, R., et al.: Mmmb: A multilingual multimodal benchmark for vision- language models. arXiv preprint arXiv:2403.12345 (2024) Multilingual Training and Evaluation Resources17 7 Appendix 7.1 Supervised Evaluation on Multi-PixMo To validate the quality of the generated multilingual data, we evaluate our mod- els on held-out test splits of Multi-PixMo-Cap and Multi-PixMo-AskModelAnything using BERTScore F1 [43] with XLM-RoBERTa-Large [8] as the underlying model. Table 9 reports the results. On AskModelAnything, multilingual training con- sistently outperforms English-only training across all backbones and languages. Qwen3-4B improves from 89.1 to 91.2 average F1, while LLaMA-3.2-3B gains 1.4 points. Notably, English F1 also improves with multilingual training (e.g., Qwen3-4B from 89.1 to 91.9), suggesting a regularization effect that benefits even the source language. On Multi-PixMo-Cap, differences are smaller, as expected, since Multi-PixMo-Cap is used during Stage 1 to train only the projection layer without impacting the language model (86.2 avg vs. 86.0 and 82.8 vs. 83.7, for Qwen3-4B and LLaMA-3.2-3B respectively). These results confirm that mul- tilingual training data improves generation quality without degrading English performance. Task ModelTrain DE EN ES FR IT Avg ama LLaMA-3.2-3B EN 81.5 83.2 82.2 82.6 82.2 82.6 Multi 82.7 83.6 84.8 85.3 84.7 84.0 Qwen3-4B EN 88.4 89.1 89.6 89.5 88.6 89.1 Multi 91.1 91.9 91.2 90.3 90.1 91.2 LLaMA-3.1-8B EN 88.4 89.1 89.6 89.5 88.6 89.1 Multi 91.1 91.9 91.2 90.3 90.1 91.2 cap LLaMA-3.2-3B EN 83.1 84.2 83.4 83.4 83.3 83.7 Multi 79.6 84.3 82.0 83.0 82.4 82.8 Qwen3-4B EN 84.9 87.0 85.4 85.5 85.3 86.0 Multi 84.7 87.3 86.1 85.6 85.6 86.2 LLaMA-3.1-8B EN 84.9 87.0 85.4 85.5 85.3 86.0 Multi 84.7 87.3 86.1 85.6 85.6 86.2 Table 9: BERTScore F1 on Multi-PixMo-AskModelAnything (ama) and Multi- PixMo-Cap (cap) per language and on average. EN: English-only training; Multi: multilingual training. Best per model group is bold. 7.2 Multi-VQAv2 Multi-VQAv2 is a generative visual question answering benchmark that requires the model to produce free-form answers rather than select from a predefined set. Unlike the benchmarks reported in the main paper, Multi-VQAv2 is not included in the MEVBench; we report it here as an additional evaluation of multilingual generation quality. This makes it particularly sensitive to the language genera- tion capabilities of the backbone. 18Baiamonte et al. Table 10 presents the results. Among the tested backbones, only Qwen3-4B produces non-trivial scores; LLaMA backbones yield near-zero accuracy. Mul- tilingual Stage 2 training is the dominant factor: the eng/multi configuration achieves 52.0% average, a 20.6-point improvement over eng/eng. The † and +cos variants perform comparably (∼51%), confirming that the multilingual signal is the primary driver of cross-lingual transfer on this benchmark. Table 10: Full results on Multi-VQAv2 (held-out, accuracy %) for all training config- urations. Only Qwen3-4B backbone produces non-trivial VQAv2 scores; LLaMA back- bones yield near-zero accuracy on this generative benchmark. Backbone S1 / S2DE EN ES FR ITAvg Qwen3-4B eng / eng15.3 61.2 28.6 21.1 30.931.4 eng / multi 48.1 63.1 49.5 49.1 50.152.0 multi / eng9.9 58.7 25.3 9.9 24.925.7 multi / multi 42.3 51.1 42.7 43.7 45.545.1 Qwen3-4B † eng / eng † 13.6 64.3 24.9 15.6 23.428.4 multi / multi † 46.6 62.7 51.1 47.8 48.451.4 eng / eng+cos17.4 62.2 28.3 23.1 26.731.5 multi / multi+cos47.7 64.1 50.1 47.5 46.551.2 7.3 Multi-RealWorldQA Multi-RealWorldQA evaluates practical visual reasoning on real-world images, requiring models to interpret everyday scenes and objects across languages. This benchmark is not reported in the main paper and is included here as a supple- mentary evaluation of cross-lingual visual grounding on naturalistic scenes. Table 11 reports the results. LLaMA-3.1-8B with multi/multi training achieves the highest average accuracy (48.8%), followed by LLaMA-3.2-3B eng/multi (44.9%). Multilingual training yields large gains for LLaMA backbones (up to +13.8 points for LLaMA-3.2-3B). Qwen3-4B shows anomalously low scores on this benchmark across all configurations, suggesting difficulty with the real-world visual reasoning tasks in the translated setting. 7.4 Multilingual Benchmark Results This section reports full per-language results on the MEVBench for all backbone and training configurations. In each table, S1 and S2 denote Stage 1 and Stage 2 training data, where eng=English-only and multi=multilingual. The † symbol indicates configurations that include cos data, and +cos denotes additional cos synthetic data in Stage 2. The best average per backbone group is highlighted in bold. Multilingual Training and Evaluation Resources19 Table 11: Full results on Multi-RealWorldQA (held-out, accuracy %) for all training configurations.†: includes cos data. BackboneS1 / S2DE EN ES FR ITAvg LLaMA-3.2-3B eng / eng22.9 43.0 32.9 26.3 30.231.1 eng / multi45.5 46.8 45.4 47.2 39.544.9 multi / eng24.8 43.4 33.5 25.6 31.531.8 multi / multi 41.0 43.1 43.0 48.2 37.342.5 LLaMA-3.2-3B † eng / eng † 20.3 41.8 26.9 23.0 23.027.0 multi / multi † 40.3 44.4 39.1 43.9 34.840.5 eng / eng+cos21.4 41.4 25.5 15.9 22.025.3 multi / multi+cos36.6 42.2 39.3 40.1 36.138.9 LLaMA-3.1-8B eng / eng 25.5 44.8 39.0 38.6 37.037.0 eng / multi 46.1 48.0 42.4 33.2 40.142.0 multi / eng28.4 45.9 36.7 32.2 35.835.8 multi / multi51.4 48.2 48.6 48.9 46.748.8 LLaMA-3.1-8B † eng / eng † 16.1 36.2 25.4 29.9 24.426.4 multi / multi † 44.1 46.9 40.7 45.6 44.144.3 Qwen3-4B eng / eng16.2 19.0 15.6 22.0 12.217.0 eng / multi 22.0 18.4 12.3 22.6 10.817.2 multi / eng17.3 21.0 11.0 16.9 10.115.2 multi / multi19.6 20.9 13.7 19.2 9.916.7 Qwen3-4B † eng / eng † 21.6 17.8 12.4 16.5 8.815.4 multi / multi † 20.7 20.9 14.6 20.7 14.618.3 eng / eng+cos 20.8 16.5 12.0 15.9 11.615.4 multi / multi+cos20.5 18.8 11.9 19.0 13.216.7 Multi-AI2D Multi-AI2D tests scientific diagram understanding, requiring mod- els to interpret charts, figures, and diagrams in multiple languages. Table 12 shows the results. Qwen3-4B clearly dominates this benchmark, with the multi/multi+cos configuration reaching 64.8% average: the highest score across all backbones and configurations. cos data provides consistent gains for Qwen3-4B (+3.7 points over the base multi/multi). LLaMA-3.1-8B achieves competitive results (52.9% with multi/multi), while LLaMA-3.2-3B reaches 48.8% with multi/multi † . Multi-MMBench Multi-MMBench is a comprehensive multiple-choice bench- mark covering a wide range of visual understanding capabilities including per- ception, reasoning, and knowledge. Table 13 presents the results. Qwen3-4B achieves the highest scores (∼79– 80%), followed by LLaMA-3.1-8B (∼73–74%) and LLaMA-3.2-3B (∼69–71%). Multilingual training provides modest but consistent gains across languages, with the gap between eng/eng and the best configuration remaining within 1–2 points for most backbones. The † configurations slightly underperform the base variants, suggesting that the additional cos data does not benefit multiple- choice visual understanding tasks. Multi-POPE Multi-POPE is a hallucination detection benchmark that probes whether VLMs can correctly identify the presence or absence of objects in im- 20Baiamonte et al. Table 12: Full results on Multi-AI2D (accuracy %) for all training configurations.†: includes cos data. BackboneS1 / S2DE EN ES FR ITAvg LLaMA-3.2-3B eng / eng 42.4 51.7 44.4 45.2 44.045.5 eng / multi 45.5 52.4 48.3 48.1 46.248.1 multi / eng44.4 51.4 46.0 46.0 45.246.6 multi / multi44.3 51.2 46.3 47.7 45.647.0 LLaMA-3.2-3B † eng / eng † 45.8 52.2 46.5 46.6 44.947.2 multi / multi † 46.9 52.6 48.6 49.0 47.048.8 eng / eng+cos 46.1 53.1 47.5 47.0 46.448.0 multi / multi+cos42.1 48.5 44.4 45.2 44.544.9 LLaMA-3.1-8B eng / eng44.6 52.1 46.3 46.1 45.246.9 eng / multi49.4 56.6 50.3 51.5 49.751.5 multi / eng49.0 56.3 50.2 50.3 49.151.0 multi / multi49.9 57.5 52.6 52.6 51.952.9 LLaMA-3.1-8B † eng / eng † 45.0 51.4 47.7 47.8 46.147.6 multi / multi † 48.7 55.9 53.3 52.5 51.152.3 Qwen3-4B eng / eng59.1 64.3 62.3 60.2 59.961.1 eng / multi58.8 64.2 61.8 60.7 59.961.1 multi / eng 58.3 64.2 61.5 60.1 60.360.9 multi / multi58.8 64.1 61.4 60.6 60.961.1 Qwen3-4B † eng / eng † 59.8 65.0 62.7 61.7 61.162.1 multi / multi † 60.3 64.4 63.4 62.0 63.462.7 eng / eng+cos61.1 66.0 64.6 63.1 63.563.7 multi / multi+cos 62.5 67.0 65.0 64.8 64.664.8 Table 13: Full results on Multi-MMBench (accuracy %) for all training configurations. †: includes cos data. BackboneS1 / S2DE EN ES FR ITAvg LLaMA-3.2-3B eng / eng67.8 72.0 68.9 69.6 67.669.2 eng / multi70.5 72.3 70.1 70.4 69.870.6 multi / eng 69.0 72.1 69.9 69.9 68.869.9 multi / multi70.0 72.1 69.9 70.2 69.770.4 LLaMA-3.2-3B † eng / eng † 67.6 70.2 67.7 68.1 66.568.0 multi / multi † 68.7 71.0 69.0 69.5 68.669.4 eng / eng+cos64.5 67.7 65.0 65.0 63.865.2 multi / multi+cos 66.7 69.2 67.8 67.5 67.167.7 LLaMA-3.1-8B eng / eng72.0 75.1 72.9 72.6 71.872.9 eng / multi73.6 75.4 72.9 73.3 73.373.7 multi / eng74.1 75.5 73.8 73.6 73.374.1 multi / multi73.1 75.4 73.4 73.4 73.273.7 LLaMA-3.1-8B † eng / eng † 69.4 72.0 69.2 69.1 69.169.8 multi / multi † 72.8 73.9 72.7 72.2 72.172.7 Qwen3-4B eng / eng 78.4 80.2 78.0 77.9 79.078.7 eng / multi78.3 79.3 78.5 78.4 78.378.6 multi / eng78.3 80.2 78.2 78.5 78.478.7 multi / multi78.8 79.8 79.0 79.0 79.279.2 Qwen3-4B † eng / eng † 77.5 79.2 77.5 78.2 78.178.1 multi / multi † 79.4 80.6 79.2 79.8 79.979.8 eng / eng+cos77.1 78.9 77.6 78.3 77.878.0 multi / multi+cos78.9 80.4 79.4 79.3 79.879.6 Multilingual Training and Evaluation Resources21 ages. Since the task is binary, it provides a controlled evaluation of visual ground- ing fidelity across languages. Table 14 reports the results. LLaMA and Qwen backbones perform reliably, with most configurations achieving 80–85% accuracy. LLaMA-3.1-8B multi/eng reaches the highest score (85.8%), while Qwen3-4B † multi/multi † achieves 85.0%. Notably, the† configurations sometimes decrease performance (e.g., LLaMA- 3.2-3B † eng/eng † drops to 66.8%), suggesting that the additional data may introduce noise for binary classification tasks. Table 14: Full results on Multi-POPE (accuracy %) for all training configurations.†: includes cos data. BackboneS1 / S2DE EN ES FR ITAvg LLaMA-3.2-3B eng / eng 81.7 81.8 81.3 79.6 82.781.4 eng / multi81.4 81.4 73.9 82.6 74.278.7 multi / eng 84.3 84.5 83.6 80.8 84.483.5 multi / multi84.0 82.3 80.9 84.5 81.582.6 LLaMA-3.2-3B † eng / eng † 66.7 66.7 67.1 66.7 66.766.8 multi / multi † 80.7 84.8 83.3 74.3 83.281.3 eng / eng+cos78.2 82.6 79.6 74.6 79.378.9 multi / multi+cos 79.9 84.0 77.3 83.2 78.380.5 LLaMA-3.1-8B eng / eng 85.5 81.4 79.5 82.6 81.682.1 eng / multi79.3 80.6 76.8 77.7 78.778.6 multi / eng 87.0 85.8 85.0 85.9 85.385.8 multi / multi81.5 82.9 82.6 82.9 82.482.5 LLaMA-3.1-8B † eng / eng † 83.3 80.8 80.9 76.0 81.980.6 multi / multi † 80.4 81.5 82.0 78.9 79.780.5 Qwen3-4B eng / eng84.5 85.4 84.0 85.1 84.184.6 eng / multi84.1 84.1 81.4 84.4 83.683.5 multi / eng 84.0 83.7 83.2 83.2 82.683.3 multi / multi82.2 82.8 82.0 84.3 83.883.1 Qwen3-4B † eng / eng † 78.2 80.0 79.4 78.9 74.478.2 multi / multi † 84.4 85.1 84.4 85.7 85.485.0 eng / eng+cos79.6 83.3 82.2 81.6 82.281.8 multi / multi+cos 84.5 84.1 79.3 83.4 82.782.8 Multi-MME Multi-MME evaluates both perceptual and reasoning abilities through a diverse set of subtasks, producing a composite score that captures broad multimodal competence. Table 15 presents the results. This benchmark exhibits high variance across configurations and languages. LLaMA-3.1-8B † multi/multi † achieves the high- est average score (1529.3), substantially outperforming all other configurations. Among the base variants, Qwen3-4B multi/eng reaches 1609.7 average. A no- table pattern is the inconsistency in German (DE) scores, where several config- urations score well below 1000 (e.g., Qwen3-4B eng/eng: 326.6 DE vs. 1557.7 EN), indicating that some backbones generate malformed responses in German for this benchmark format. 22Baiamonte et al. Table 15: Full results on Multi-MME (total score) for all training configurations.†: includes cos data. BackboneS1 / S2DE EN ES FR ITAvg LLaMA-3.2-3B eng / eng1256.7 1213.4 1405.2 1427.2 1476.41355.8 eng / multi794.7 1436.6 1178.8 1352.6 1070.21166.6 multi / eng1322.3 1247.3 1351.3 1376.4 1377.01334.9 multi / multi 1348.7 1421.1 1109.3 1335.4 1137.61270.4 LLaMA-3.2-3B † eng / eng † 604.6 709.7 1281.2 710.1 1098.2880.7 multi / multi † 626.7 1350.5 1325.5 1304.1 1300.11181.4 eng / eng+cos1279.9 1339.8 1058.7 1277.9 1427.61276.8 multi / multi+cos1026.3 1443.9 816.3 1166.1 744.11039.3 LLaMA-3.1-8B eng / eng 872.8 1208.9 1167.8 1185.6 1258.41138.7 eng / multi 587.4 1486.3 1310.4 1399.4 1134.61183.6 multi / eng1456.3 1462.4 1422.8 1469.5 1545.71471.4 multi / multi686.8 1574.8 1379.3 1424.4 1212.51255.5 LLaMA-3.1-8B † eng / eng † 1244.5 1257.1 1475.2 1223.4 1464.11332.8 multi / multi † 1502.3 1547.9 1526.2 1532.2 1538.01529.3 Qwen3-4B eng / eng326.6 1557.7 1578.3 1276.2 1669.71281.7 eng / multi 327.1 1441.3 1376.4 1568.9 1573.01257.3 multi / eng1487.7 1677.2 1696.1 1624.5 1563.01609.7 multi / multi727.6 1498.4 1425.9 1533.3 1558.81348.8 Qwen3-4B † eng / eng † 720.0 1498.3 1471.7 1241.3 1259.61238.2 multi / multi † 491.4 1681.7 1533.1 1615.3 1684.01401.1 eng / eng+cos 1078.0 1720.3 1688.4 1382.9 1630.71500.1 multi / multi+cos605.0 1599.4 1387.8 1587.6 1522.21340.4 Multi-ScienceQA Multi-ScienceQA evaluates scientific knowledge and reason- ing through multiple-choice questions accompanied by images, spanning natural science, social science, and language science topics. Table 16 shows the results. Qwen3-4B achieves the best performance across all configurations, with multi/eng reaching 76.7% average. LLaMA-3.1-8B multi/eng follows at 71.6%. Multilingual training has a relatively modest effect on this benchmark, likely because ScienceQA questions require domain-specific reason- ing that transfers well across languages even with English-only training. The † configurations slightly decrease performance across all backbones. 7.5 Zero-shot Transfer Evaluation To assess whether multilingual training yields broader cross-lingual transfer be- yond the benchmarks included in our evaluation suite, we evaluate on MTVQA [39] and CVQA [28]: two multilingual benchmarks that were not included in any training set. MTVQA tests multilingual text-rich visual question answering, while CVQA probes culturally diverse visual reasoning. Table 17 reports the results. On MTVQA, multilingual training consistently improves performance across all backbones, with gains of 2–3 points over English- only variants. The best MTVQA scores are achieved by LLaMA-3.2-3B eng/multi (14.5%) and Qwen3-4B multi/multi+cos (14.0%). On CVQA, scores are more stable across configurations, with LLaMA-3.1-8B and Qwen3-4B achieving∼48– Multilingual Training and Evaluation Resources23 Table 16: Full results on Multi-ScienceQA (accuracy %) for all training configurations. †: includes cos data. BackboneS1 / S2DE EN ES FR ITAvg LLaMA-3.2-3B eng / eng67.4 70.8 67.1 66.6 66.467.7 eng / multi67.8 70.7 67.1 67.7 66.367.9 multi / eng68.0 71.8 67.5 66.6 66.568.1 multi / multi 68.3 70.9 66.7 66.9 66.867.9 LLaMA-3.2-3B † eng / eng † 67.3 70.2 66.2 65.6 66.267.1 multi / multi † 65.4 69.6 65.5 64.7 64.866.0 eng / eng+cos66.1 69.0 65.8 64.8 65.266.2 multi / multi+cos63.3 66.5 62.8 63.0 62.763.7 LLaMA-3.1-8B eng / eng 65.6 68.5 65.8 65.2 65.066.0 eng / multi 69.4 71.0 68.4 69.4 68.469.3 multi / eng70.8 74.9 70.8 71.2 70.071.6 multi / multi68.3 72.0 67.8 68.9 66.668.7 LLaMA-3.1-8B † eng / eng † 66.9 70.5 67.2 66.3 67.467.7 multi / multi † 63.2 68.3 65.3 65.6 65.065.5 Qwen3-4B eng / eng74.4 77.1 74.3 75.9 75.075.3 eng / multi 73.7 75.5 73.7 74.8 73.774.3 multi / eng75.9 77.8 76.2 77.1 76.776.7 multi / multi75.1 77.4 73.7 76.3 75.675.6 Qwen3-4B † eng / eng † 73.2 76.4 74.0 75.2 73.874.5 multi / multi † 76.0 77.6 75.5 76.8 76.576.5 eng / eng+cos 75.8 78.0 75.3 76.2 76.576.4 multi / multi+cos76.2 78.1 74.8 77.3 75.876.4 50% regardless of training data, suggesting that cultural visual question answer- ing benefits more from backbone capacity than from multilingual fine-tuning. 7.6 Comparison with External Models To contextualize our results against publicly available models, we evaluate two instruction-tuned VLMs (Qwen2-VL-2B-Instruct and SmolVLM2-2.2B-Instruct): on the MEVBench. These models were not trained on Multi-PixMo and serve as external baselines of comparable parameter count. Tables 18 and 19 report the results. On AI2D and VQAv2, Qwen2-VL-2B- Instruct achieves the strongest results (63.4% and 55.5% respectively), outper- forming most of our trained configurations except the best Qwen3-4B variants. SmolVLM2-2.2B-Instruct is competitive on RealWorldQA (39.0%) and POPE (72.0%), but lags behind on other benchmarks. Both external models exhibit sig- nificant cross-lingual variance, particularly on MME (Table 19), where German scores are substantially lower than English, consistent with the pattern observed in our trained models. 24Baiamonte et al. Table 17: Results on MTVQA and CVQA (held-out, accuracy %). These benchmarks were not used during training for any model.†: includes cos data. BackboneS1 / S2MTVQA CVQA LLaMA-3.2-3B eng / eng11.543.7 eng / multi14.544.0 multi / eng10.444.6 multi / multi 13.843.6 LLaMA-3.2-3B † eng / eng † 10.242.7 multi / multi † 13.244.1 eng / eng+cos10.841.4 multi / multi+cos13.643.2 LLaMA-3.1-8B eng / eng 13.148.7 eng / multi 14.049.0 multi / eng12.648.8 multi / multi13.947.8 LLaMA-3.1-8B † eng / eng † 11.940.5 multi / multi † 13.347.8 Qwen3-4B eng / eng 11.348.4 eng / multi12.248.3 multi / eng 12.148.4 multi / multi13.649.0 Qwen3-4B † eng / eng † 12.247.8 multi / multi † 13.549.2 eng / eng+cos12.246.8 multi / multi+cos 14.049.7 Table 18: External model comparison on the full MEVBench (accuracy %). Qwen2- VL-2B-Instruct and SmolVLM2-2.2B-Instruct are evaluated as publicly available base- lines not trained on Multi-PixMo. BenchmarkModelDE EN ES FR ITAvg AI2D Qwen2-VL-2B-Inst.59.6 70.2 63.9 62.9 60.363.4 SmolVLM2-2.2B-Inst.41.5 67.2 50.3 48.8 45.650.7 MMBench Qwen2-VL-2B-Inst.63.1 51.8 58.5 60.6 59.958.8 SmolVLM2-2.2B-Inst.51.1 55.7 51.1 51.8 46.151.2 POPE Qwen2-VL-2B-Inst.70.8 75.4 70.5 88.5 76.376.3 SmolVLM2-2.2B-Inst.58.2 84.3 74.6 69.0 73.972.0 RealWorldQA Qwen2-VL-2B-Inst.36.5 31.8 51.0 18.7 44.136.4 SmolVLM2-2.2B-Inst.32.3 56.7 40.8 33.1 31.939.0 VQAv2 Qwen2-VL-2B-Inst.51.5 80.1 46.9 53.1 46.155.5 SmolVLM2-2.2B-Inst.16.2 67.4 23.5 20.5 22.330.0 Table 19: External model comparison on Multi-MME (perception / reasoning scores). ModelDEENESFRIT Qwen2-VL-2B-Inst.993/213 1497/428 1394/349 734/202 1438/333 SmolVLM2-2.2B-Inst.725/209 1476/311 961/218 668/196 752/178