Paper deep dive
Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting
Paweł Mąka, Piotr Andruszkiewicz, Yusuf Can Semerci, Jan Scholtes, Gerasimos Spanakis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 9/28/2026, 4:24:47 AM
Summary
This paper introduces Metric-based Loss Weighting, a training strategy for Multimodal Large Language Models (MLLMs) to improve visual sensitivity in Image-guided Machine Translation (MMT). The method uses Point-wise Cross-mutual Information (PCXMI) and a novel Congruency-based PCXMI metric to identify tokens that benefit from visual context, then increases the loss weight for these tokens during fine-tuning. Experiments on the CoMMuTE dataset show significant improvements in visual grounding accuracy compared to standard fine-tuning and other baselines.
Entities (13)
Relation Signals (10)
Metric-based Loss Weighting → evaluatedon → CoMMuTE
confidence 99% · Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset
Metric-based Loss Weighting → appliedto → Qwen3-VL-4B-Instruct
confidence 98% · We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models... Qwen3-VL 4B Instruct
Metric-based Loss Weighting → appliedto → Qwen3-VL-8B-Instruct
confidence 98% · We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models... Qwen3-VL 8B Instruct
Metric-based Loss Weighting → appliedto → IDEFICS 9B Instruct
confidence 98% · We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models... IDEFICS 9B Instruct
CoMMuTE → measures → Visual Sensitivity
confidence 97% · To evaluate the visual sensitivity of the fine-tuned models, we employ the CoMMuTE contrastive dataset
Metric-based Loss Weighting → improves → Visual Sensitivity
confidence 96% · Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset, improving accuracy by up to more than 7 percentage points
Metric-based Loss Weighting → trainedon → Multi30k
confidence 95% · We used LoRA (Hu et al., 2022) to fine-tune the models on the Multi30k dataset
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal Machine Translation aims to incorporate additional signal from non-textual modalities to improve translations by resolving ambiguities. While models, through multimodal fusion, are able to accept images related to the source text, they can ignore this information. Therefore, increasing their visual sensitivity remains an active research area. In this work, we introduce a training method, Metric-based Loss Weighting, that improves visual grounding of translations by increasing the loss function for tokens that benefit from the accompanying image. We identify these tokens using the Point-wise Cross-mutual Information (PCXMI) metric, which compares the model's output probabilities with and without visual context. We introduce a Congruency-based PCXMI metric and experimentally show that both metrics working in combination yield the best results. We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models on the task of Image-guided Machine Translation for three language directions. Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset, improving accuracy by up to more than 7 percentage points compared to standard fine-tuning, while maintaining strong general translation performance.
Tags
Links
- Source: https://arxiv.org/abs/2609.31169v1
- Canonical: https://arxiv.org/abs/2609.31169v1
Trouble viewing inline? Open PDF directly →
Full Text
80,424 characters extracted from source content.
Expand or collapse full text
Improving Visual Sensitivity of LLMs on Multimodal Machine Translation with Metric-based Loss Weighting Paweł Mąka Affiliation: Department of Advanced Computing Sciences, Maastricht University Email: pawel.maka@maastrichtuniversity.nl Piotr Andruszkiewicz Affiliation: Warsaw University of Technology Affiliation: IDEAS Research Institute Email: y.semerci@maastrichtuniversity.nl Yusuf Can Semerci Affiliation: Department of Advanced Computing Sciences, Maastricht University Email: j.scholtes@maastrichtuniversity.nl Jan Scholtes Affiliation: Department of Advanced Computing Sciences, Maastricht University Email: jerry.spanakis@maastrichtuniversity.nl Gerasimos Spanakis Affiliation: Department of Advanced Computing Sciences, Maastricht University Email: piotr.andruszkiewicz@pw.edu.pl Abstract Multimodal Machine Translation aims to incorporate additional signal from non-textual modalities to improve translations by resolving ambiguities. While models, through multimodal fusion, are able to accept images related to the source text, they can ignore this information. Therefore, increasing their visual sensitivity remains an active research area. In this work, we introduce a training method, Metric-based Loss Weighting, that improves visual grounding of translations by increasing the loss function for tokens that benefit from the accompanying image. We identify these tokens using the Point-wise Cross-mutual Information (PCXMI) metric, which compares the model’s output probabilities with and without visual context. We introduce a Congruency-based PCXMI metric and experimentally show that both metrics working in combination yield the best results. We evaluate our method by fine-tuning three pretrained Multimodal Large Language Models on the task of Image-guided Machine Translation for three language directions. Metric-based Loss Weighting outperforms other tested methods on the CoMMuTE contrastive dataset, improving accuracy by up to more than 7 percentage points compared to standard fine-tuning, while maintaining strong general translation performance. 1 Introduction Sentence-level Machine Translation (MT) systems achieved high performance (Vaswani et al., 2017; Hassan et al., 2018; Costa-jussà et al., 2022; Tiedemann et al., 2024), but they are inherently limited by the lack of context. In Multimodal Machine Translation (MMT), the additional signal from other modalities can improve translation in cases where text alone is ambiguous. This signal can be audio, video, or image (Sulubacak et al., 2020; Garg et al., 2022; Feng et al., 2025). In this study, we concentrate our efforts on Image-based MT. Figure 1: The illustration of Metric-based Loss Weighting: (1) Calculate the image dependence metric for each token (PCXMI), based on the model’s output probabilities with and without an image (or with an incongruent image); (2) Fine-tune the model with increased loss (by α) for the identified tokens (with PCXMI larger than τ). While the images are available to the MMT systems, the reliance on the other modality is not guaranteed. Indeed, Elliott (2018) did not find the expected performance drop of the tested systems after randomly replacing images. Wu et al. (2021) attributed some gains of the MMT systems to regularization. Therefore, visual sensitivity of the MMT models has seen increased attention from researchers in recent years (Sato et al., 2023; Zhu et al., 2023; Cheng et al., 2024; Bowen et al., 2024; Futeral et al., 2025). As traditional translation metrics such as BLEU (Papineni et al., 2002) are not sensitive enough to judge disambiguations, we turn towards CoMMuTE (Futeral et al., 2023). It is a contrastive dataset where the tested models are presented with multiple possible translations of the ambiguous source sentence and tasked with selecting the correct one. The selection is based on the probability that models give to each option, and the final metric is the accuracy of assigning the highest probability to the correct one. The field of MMT went from training specialized architectures from random initialization (Yin et al., 2020; Liu et al., 2021) to using pre-trained models as building blocks and fine-tuning the resulting model with multimodal parallel data (Gupta et al., 2023; Yu et al., 2025). Recently, more focus has been given to Multimodal Large Language Models (MLLMs) (Alayrac et al., 2022; Liu et al., 2023; Laurençon et al., 2023; Bai et al., 2025) due to their generalizability and strong performance in different tasks (Zhu et al., 2024). In this study, we concentrate on methods that could be used to enhance fine-tuning of MLLMs on the task of MMT to increase the visual grounding of the translation. We argue that the insufficient number of examples in the widely used MMT training datasets that require images for correct translation is causing the training signal to be sparse and, therefore, not enough for the models to learn to rely on context (Futeral et al., 2023). To mitigate this issue, we propose a training strategy where we identify the tokens in the target sequence that benefit from context and, subsequently, increase the loss for those tokens during training. To identify the image-dependent tokens, we use the Point-wise Cross-Mutual Information (PCXMI) metric (Fernandes et al., 2023), which compares the output probabilities of the model in two situations: with and without the context (see Figure 1). We propose a modification to this metric by using an incongruent image as context instead of the lack of it. We combine both metrics to achieve the best results. We test our method against several techniques adapted to fine-tuning of MLLMs using three models and three language directions. Our Metric-based Loss Weighting method outperforms the baselines in terms of CoMMuTE accuracy on most of the tested scenarios. It improves the accuracy by up to more than 7 percentage points compared to the standard fine-tuning. Our contributions are as follows: 1. We introduce a training strategy for fine-tuning MLLMs that improves their visual sensitivity on the task of MMT based on identifying and increasing loss function for image-dependent tokens. 2. We adapt the Point-wise Cross-Mutual Information (PCXMI) metric (Fernandes et al., 2023) to MMT, introduce Congruency-based PCXMI metric, and experimentally show that both metrics working in combination yield best results. 3. We adapt and evaluate several existing methods for fine-tuning MLLMs on the task of MMT, establishing the baseline for future research. 4. We show that the pre-trained models exhibit high visual sensitivity which can be further increased with specialized fine-tuning techniques. 5. We empirically confirm the discrepancy between traditional translation metrics such as BLEU (Papineni et al., 2002) and the image reliance of the MLLMs. 2 Related Work Researchers have been interested in incorporating the information coming from other modalities to enhance text-to-text translation (Sulubacak et al., 2020; Garg et al., 2022; Feng et al., 2025). While this includes audio and video (Gu et al., 2021; Li et al., 2023), large focus has been concentrated on single images related to the source text (Elliott et al., 2017; Barrault et al., 2018; Cheng et al., 2024; Futeral et al., 2025; Yu et al., 2025). The methods proposed specialized architectures concentrating on feature extraction (Yao and Wan, 2020; Fang and Feng, 2022), cross-modal fusion and alignment (Zhao et al., 2022; Guo et al., 2022), and cross-modal attention (Nishihara et al., 2020; Gupta et al., 2023). Other notable directions include: employing auxiliary tasks, such as Visually Conditioned Masked Language Modeling (VMLM) (Futeral et al., 2023; Futeral et al., 2025), image captioning (Gupta et al., 2023), or visual question answering (Zhang et al., 2025; Gao et al., 2025), masking the source sentence to increase visual grounding (Sato et al., 2023; Cheng et al., 2024; Bowen et al., 2024), and data augmentation by image generation (Long et al., 2021; Li et al., 2022c; Chen et al., 2025), using text-to-image models Goodfellow et al. (2014); Rombach et al. (2022), or backtranslation Li et al. (2021). The MMT models are often trained on the widely used Multi30k dataset and evaluated on the accompanying testsets (Elliott et al., 2016; Elliott et al., 2017; Barrault et al., 2018). Although the systems show increased BLEU scores (Papineni et al., 2002) compared to the text-only baselines, the benefits have been attributed to regularization (Wu et al., 2021). For example, Elliott (2018), after replacing the relevant images with random ones, did not see the expected drop in performance. This could be caused by the insufficient number of examples in the Multi30k training dataset that require context in the form of the accompanying image for correct translation (Futeral et al., 2023). Earlier works trained models from random initialization (Yin et al., 2020; Liu et al., 2021), but increasingly often researchers utilize pretrained models (Gupta et al., 2023; Futeral et al., 2025; Yu et al., 2025). This transition is motivated by the strong generalization capabilities of the pretrained models, acquired through training on large amounts of data. The popular models include CLIP (Radford et al., 2021) and BLIP (Li et al., 2022b) for image feature extraction, and Large Language Models, such as mBART (Liu et al., 2020) or Qwen (Yang et al., 2025), for text understanding and generation. The emergence of strong general-purpose Multimodal Large Language Models (MLLMs), capable of incorporating visual information without architectural modifications, presents a potential shift in the approach to Multimodal Machine Translation (Alayrac et al., 2022; Awadalla et al., 2023; Bai et al., 2025; Chen et al., 2024). While Xiong and Zhao (2025) and Yu et al. (2025) evaluated their methods against MLLMs, and Zhu et al. (2024) performed a large scale evaluation of publicly-available MLLMs, we concentrate on instruction-tuning (Wei et al., 2022; Zhang et al., 2026; Hu et al., 2022) and evaluating the open-weights models (Laurençon et al., 2023; Bai et al., 2025). Our method is inspired by Metric-based Fine-tuning and Loss Weighting, the two separate techniques introduced for encoder-decoder models by Mąka et al. (2025) for the task of Context-aware MT. Importantly, both techniques used only a textual modality, and the loss weighting was based on the external annotations. In this work, we combine both techniques, and adapt it to the multimodal setting of Image-guided MT. Furthermore, we introduce a new Metric that uses incongruent images instead of not providing an image. 3 Method Our proposed method consists of two steps. First, we identify which tokens in the target sentence require context. To do this, we measure how much the output probabilities of the pretrained model change when the context image is present or not (alternatively, when the image is not related to the text). Second, we use this information during training to improve the visual sensitivity of the model by increasing the loss corresponding to the context-dependent tokens. Similar to Mąka et al. (2025), we employ the Point-wise Cross-Mutual Information (PCXMI) metric Fernandes et al. (2023) for the identification step. The original PCXMI metric compares the model’s output probabilities with and without previous sentences as context. We adapt it to the setting of Image-guided MT by treating the accompanying image as context. The equation below calculates the PCXMIPCXMI for the j-th token in the target sequence: PCXMIj=logq(yj|yt<j,x,C)q(yj|yt<j,x), PCXMI_j=log q (y_j|y_t<j,x,C )q (y_j|y_t<j,x ), (1) where q(yj|yt<j,x,C)q (y_j|y_t<j,x,C ) is the probability of the j-th token conditioned on the previous tokens yt<jy_t<j, source sentence x, and the image C, and q(yj|yt<j,x)q (y_j|y_t<j,x ) is the probability conditioned only on the previous tokens yt<jy_t<j and the source sentence x. Crucially, the model has to be able to perform translation both with and without the image to use this metric. Additionally, we propose a modification to the metric, where the comparison is between the probabilities with the correct and incorrect (incongruent) images. We refer to this metric as Congruency-based Point-wise Cross-Mutual Information (CPCMXI): CPCXMIj=logq(yj|yt<j,x,C)q(yj|yt<j,x,C~), CPCXMI_j=log q(y_j|y_t<j,x,C)q(y_j|y_t<j,x, C), (2) where q(yj|yt<j,x,C~)q(y_j|y_t<j,x, C) is the probability conditioned on the previous tokens yt<jy_t<j, source sentence x, and incongruent image C~ C, which we randomly sample from the training dataset. The advantage over the standard PCXMI is that it can be used with models that always require the image to be present. In practice, both metrics rely on the outputs of the model in different situations and are, therefore, noisy. To mitigate this, we calculate both metrics for each example did_i in the training dataset D, and take the maximum value for each token as the final image dependence metric: ai,j=max(PCXMIi,j,CPCXMIi,j). a_i,j= (PCXMI_i,j,CPCXMI_i,j ). (3) Taking the maximum increases the reliability of the metric in situations where one or the other metric fails to mark a token as context-dependent. In the second step, we apply token-level weighting of the standard Negative Log Likelihood loss based on the calculated context dependence metric (Mąka et al., 2025): ℒ=−1|D|∑(di,ai)∼D1|yi|∑j=1|yi|w(ai,j)log(y^i,j), =- 1|D| _(d_i,a_i) D 1|y_i| _j=1^|y_i|w (a_i,j )log ( y_i,j ), (4) where did_i is the i-th example sampled from the training dataset D with the calculated metric aia_i for each token in the target sentence, y^i,j y_i,j is the output probability of the model for the j-th token, and w(ai,j)w (a_i,j ) is the weight function based on the value of the metric ai,ja_i,j for the j-th token. We use the threshold function to obtain the weighting values: w(ai,j)=1+α,if ai,j>τ,1,otherwise, w(a_i,j)= cases1+α,&if ~~a_i,j>τ,\\ 1,&otherwise,\\ cases (5) where the hyper-parameters α and τ control the loss increase and the threshold, respectively. Intuitively, we penalize the model more for the mistakes in tokens that require context. It should be noted that our method relies on the identification of the image-dependent tokens, which is done by the model itself. This means that we bias the model towards the tokens for which it already exhibits strong visual sensitivity. 4 Experiments We experimentally evaluated the effects of our method on fine-tuning MLLMs on the task of MMT. We are specifically interested in observing how it affects the visual sensitivity of the tested models. We experimented with three popular open-weights instruction-tuned models: Qwen3-VL 4B Instruct11 1 https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct, Qwen3-VL 8B Instruct22 2 https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct (Bai et al., 2025; Yang et al., 2025), and IDEFICS 9B Instruct33 3 https://huggingface.co/HuggingFaceM4/idefics-9b-instruct (Laurençon et al., 2023). We used LoRA (Hu et al., 2022) to fine-tune the models on the Multi30k dataset (Elliott et al., 2016). We experimented with three language directions: English-to-German, English-to-French, and English-to-Czech. Model test 2016 test 2017 MSCOCO Average CoMMuTE BLEU COMET BLEU COMET BLEU COMET BLEU COMET Accuracy Qwen3-VL 4B 38.51 85.56 32.95 84.90 29.94 81.24 33.80 83.90 74.00% Fine-tuning 42.61± 0.12 86.66± 0.06 38.08± 0.38 85.98± 0.06 35.22± 0.45 82.70± 0.13 38.64± 0.18 85.11± 0.05 71.67± 0.19% VMLM 42.52± 0.19 86.79± 0.03 38.48± 0.43 86.34± 0.06 36.47± 0.09 82.95± 0.17 39.16± 0.06 85.36± 0.05 71.67± 0.38% Metric Fine-tuning 42.57± 0.22 86.47± 0.13 38.43± 0.10 85.99± 0.12 36.18± 0.16 82.86± 0.18 39.06± 0.15 85.10± 0.11 72.89± 0.44% Word Dropout 42.20± 0.19 86.62± 0.06 37.17± 0.19 85.80± 0.01 35.00± 0.22 82.69± 0.16 38.13± 0.18 85.03± 0.04 76.89± 0.44% Degradation 42.18± 0.16 86.68± 0.00 37.89± 0.03 85.88± 0.02 35.66± 0.26 82.85± 0.04 38.58± 0.14 85.14± 0.02 73.44± 0.59% Metric Weighting 41.91± 0.33 86.80± 0.04 37.29± 0.14 86.14± 0.04 35.71± 0.19 83.09± 0.13 38.30± 0.15 85.34± 0.07 77.00± 0.33% Qwen3-VL 8B 38.24 85.64 34.24 85.77 31.78 82.25 34.76 84.55 75.33% Fine-tuning 42.87± 0.54 86.68± 0.21 38.15± 0.29 86.35± 0.10 35.93± 0.22 83.11± 0.32 38.98± 0.26 85.38± 0.19 69.33± 0.19% VMLM 42.25± 0.92 86.84± 0.15 37.24± 0.62 86.09± 0.10 35.28± 1.35 83.33± 0.21 38.26± 0.96 85.42± 0.14 69.78± 0.97% Metric Fine-tuning 43.15± 0.16 86.80± 0.10 38.19± 0.41 86.45± 0.07 36.19± 0.34 83.41± 0.12 39.18± 0.28 85.55± 0.08 73.56± 0.62% Word Dropout 43.34± 0.25 87.17± 0.03 37.08± 0.18 86.26± 0.09 35.46± 0.74 83.55± 0.11 38.63± 0.18 85.66± 0.04 76.00± 0.33% Degradation 43.47± 0.34 87.01± 0.04 38.38± 0.14 86.55± 0.07 36.42± 0.19 83.34± 0.16 39.43± 0.10 85.63± 0.08 72.22± 0.29% Metric Weighting 42.82± 0.13 86.86± 0.11 37.38± 0.30 86.47± 0.07 35.12± 0.31 83.31± 0.15 38.44± 0.14 85.55± 0.06 77.22± 0.22% IDEFICS 9B 27.07 80.04 22.51 78.99 19.35 74.38 22.98 77.80 68.67% Fine-tuning 43.48± 0.72 86.62± 0.21 37.13± 0.15 85.58± 0.18 34.24± 0.45 82.13± 0.16 38.28± 0.43 84.78± 0.18 65.89± 0.73% VMLM 43.38± 0.24 86.65± 0.05 38.17± 0.15 85.80± 0.03 35.69± 0.28 82.39± 0.05 39.08± 0.23 84.95± 0.04 64.44± 0.48% Metric Fine-tuning 42.88± 0.55 86.37± 0.12 36.63± 0.29 85.10± 0.29 34.14± 0.83 81.88± 0.25 37.89± 0.55 84.45± 0.22 67.89± 0.29% Word Dropout 43.56± 0.26 86.77± 0.09 36.77± 0.49 85.51± 0.16 34.17± 0.19 82.23± 0.11 38.17± 0.26 84.83± 0.09 70.22± 0.73% Degradation 44.07± 0.22 86.71± 0.04 37.36± 0.11 85.44± 0.07 34.18± 0.18 82.01± 0.04 38.54± 0.07 84.72± 0.02 67.78± 0.11% Metric Weighting 43.58± 0.61 86.53± 0.10 37.42± 0.15 85.38± 0.10 34.12± 0.17 81.96± 0.18 38.37± 0.19 84.62± 0.11 69.00± 0.33% Table 1: Results for English-to-German in terms of BLEU and COMET on the Multi30k testsets (separately and averaged over the testsets) and the accuracy on the CoMMuTE contrastive dataset. The results are averaged over three separate runs, and ± shows the standard error. Best results for each base model in each column are marked in bold, and the second best are underlined. We compared our method with the following baselines: • Fine-tuning - simply fine-tuning the model on the Image-guided MT training dataset. • Visually Conditioned Masked Language Modeling (VMLM) - fine-tuning with the auxiliary task, where the model is presented with the source sentence containing masked tokens, and it has to predict those masked tokens (Futeral et al., 2023; Futeral et al., 2025). We mask words with the probability p, which is a hyper-parameter, with a string MASKn, where n goes from 1 to the total number of masked words in the source text. Because VMLM does not require a parallel corpus, often an additional multimodal dataset in the source language is used. In our setting, we use only the Multi30k dataset, discarding the target-side sentences, to keep the methods using comparable amounts of data. • Metric-based Fine-tuning - we adapt this method from the original text-only version (Mąka et al., 2025). After the initial fine-tuning, we calculate the PCXMI and CPCXMI metrics and select top-k examples with the highest maximum values of the calculated metrics for further fine-tuning. We take the maximum of the two metrics, as this yielded better results in the preliminary experiments than either of the metrics on their own. • Word Dropout - dropping random tokens from the source sentence to force the model to use the image to improve translation, adapted to Image-guided Machine Translation from CoWord Dropout (Fernandes et al., 2021). The probability of dropping out a token is controlled by the hyper-parameter p. In the preliminary experiments, we determined that dropout on the word-level was better than token-level. Therefore, we drop out words based on the Moses Tokenizer44 4 We used the Sacremoses implementation https://github.com/hplt-project/sacremoses with default parameters.(Koehn et al., 2007). • Source Degradation - masking specific words in the source sentence during training to enhance the visual sensitivity of the model (Cheng et al., 2024). The method first identifies the words in three categories: color, person, and noun. Second, the identified words are replaced with the mask strings MASK_C, MASK_P, MASK_N for colors, people, and nouns, respectively (in the original paper, the special tokens have been used instead of words). We used the annotations for types of tokens in the Multi30k dataset from Li et al. (2022a) repository55 5 https://github.com/libeineu/fairseq_mmt.. We first tuned the hyper-parameters of Fine-tuning in the preliminary experiments on the English-to-German language direction and kept them the same for other methods. For Metric-based Fine-tuning, we selected the number of examples for the second-stage fine-tuning in the preliminary experiments and tuned the learning rate separately for each language direction and model out of 1e−5,2e−5,5e−5,1e−4\1e-5,2e-5,5e-5,1e-4\. For the VMLM, Word Dropout, and Source Degradation methods, we tuned the probability of masking and dropping out words p∈0.1,0.2,0.3p∈\0.1,0.2,0.3\. For our method, we first tuned the hyper-parameter α=8α=8 in the preliminary experiments on the English-to-German language direction, and tested three values of the τ hyper-parameter for each language direction and model (5,6,7\5,6,7\, and 1,1.5,2\1,1.5,2\ for Qwen3-VL and IDEFICS models, respectively). This is because they depend on the distribution of PCXMI and CPCXMI metrics on the training dataset, which are based on the model’s outputs (see Section 5.1). We determined that the appropriate range for the τ values is between the 80th and 90th percentile of the distribution of maximum metric values in training examples. Further details of the training, including prompts and hyper-parameters can be found in Appendix A. We measured BLEU66 6 https://github.com/mjpost/sacrebleu (Papineni et al., 2002) and COMET77 7 We used the Unbabel/wmt22-comet-da model. (Rei et al., 2020) of the translations on the available Multi30k test sets: test 2016, test 2017, and MSCOCO for English-to-German and English-to-French, and test 2016 for English-to-Czech (Elliott et al., 2016; Elliott et al., 2017; Barrault et al., 2018). We used the default generation parameters for the tested models. To evaluate the visual sensitivity of the fine-tuned models, we employ the CoMMuTE contrastive dataset (Futeral et al., 2023). In this dataset, the models are presented with an image, an ambiguous source sentence, and two possible translations, where the correct one depends on the accompanying image. The dataset measures the accuracy of the models by assigning higher probabilities to correct target sentences. Following Futeral et al. (2023), we repeated the experiments three times with different seeds and averaged the results. 4.1 Results Model test 2016 test 2017 MSCOCO Average CoMMuTE BLEU COMET BLEU COMET BLEU COMET BLEU COMET Accuracy Qwen3-VL 4B 43.77 86.04 43.47 86.31 43.24 83.69 43.49 85.35 75.00% Fine-tuning 67.03± 0.36 89.81± 0.05 61.62± 0.13 89.25± 0.10 50.90± 0.43 85.37± 0.12 59.85± 0.08 88.14± 0.03 81.17± 0.37% VMLM 67.37± 0.25 89.71± 0.05 61.00± 0.19 89.23± 0.10 50.71± 0.57 85.10± 0.07 59.69± 0.21 88.01± 0.05 81.71± 0.39% Metric Fine-tuning 66.83± 0.41 89.43± 0.12 60.98± 0.11 88.97± 0.02 49.73± 0.09 84.71± 0.15 59.18± 0.13 87.70± 0.09 81.17± 0.19% Word Dropout 66.46± 0.31 89.58± 0.09 60.98± 0.05 89.10± 0.02 51.01± 0.44 85.15± 0.04 59.48± 0.27 87.94± 0.04 84.42± 0.68% Degradation 67.42± 0.18 89.69± 0.04 61.58± 0.13 89.26± 0.06 51.09± 0.18 85.33± 0.06 60.03± 0.04 88.09± 0.02 84.52± 0.11% Metric Weighting 67.22± 0.28 89.69± 0.03 61.04± 0.23 89.15± 0.04 50.50± 0.37 85.07± 0.03 59.59± 0.22 87.97± 0.03 84.74± 0.50% Qwen3-VL 8B 47.53 86.93 47.32 87.14 46.40 84.36 47.08 86.14 75.00% Fine-tuning 68.56± 0.17 90.06± 0.09 62.30± 0.15 89.50± 0.05 52.66± 0.31 85.74± 0.17 61.17± 0.13 88.43± 0.06 83.12± 0.00% VMLM 68.15± 0.12 90.01± 0.09 61.52± 0.08 89.48± 0.11 51.56± 0.20 85.57± 0.08 60.41± 0.05 88.35± 0.01 81.49± 0.19% Metric Fine-tuning 69.32± 0.10 90.07± 0.04 62.91± 0.27 89.55± 0.03 51.31± 0.35 85.60± 0.10 61.18± 0.02 88.41± 0.05 84.42± 0.50% Word Dropout 67.20± 0.18 89.81± 0.05 61.14± 0.52 89.03± 0.09 51.27± 0.15 85.23± 0.07 59.87± 0.09 88.02± 0.04 85.28± 0.57% Degradation 67.29± 0.09 89.93± 0.04 62.07± 0.15 89.29± 0.07 50.99± 0.18 85.44± 0.09 60.12± 0.02 88.22± 0.06 81.28± 0.29% Metric Weighting 67.74± 0.16 89.98± 0.06 62.68± 0.19 89.37± 0.03 51.19± 0.18 85.56± 0.16 60.54± 0.07 88.30± 0.04 85.06± 0.50% IDEFICS 9B 33.35 82.26 32.15 81.76 31.80 78.64 32.43 80.89 71.75% Fine-tuning 67.00± 0.19 89.39± 0.11 60.39± 0.13 88.62± 0.09 50.58± 0.24 84.87± 0.09 59.32± 0.12 87.63± 0.03 76.73± 0.22% VMLM 66.78± 0.04 89.47± 0.05 60.44± 0.05 88.68± 0.11 50.83± 0.36 85.03± 0.15 59.35± 0.12 87.73± 0.02 78.57± 0.37% Metric Fine-tuning 67.52± 0.24 89.47± 0.16 60.19± 0.18 88.54± 0.02 50.41± 0.38 84.94± 0.16 59.37± 0.25 87.65± 0.11 77.92± 0.32% Word Dropout 65.82± 0.12 89.19± 0.06 59.74± 0.11 88.43± 0.03 49.82± 0.24 84.80± 0.19 58.46± 0.08 87.47± 0.05 78.68± 0.43% Degradation 67.00± 0.13 89.28± 0.03 60.50± 0.07 88.54± 0.01 50.08± 0.33 84.96± 0.13 59.19± 0.09 87.59± 0.05 78.14± 0.89% Metric Weighting 66.55± 0.12 89.22± 0.05 60.29± 0.07 88.48± 0.08 50.51± 0.22 85.35± 0.08 59.12± 0.05 87.69± 0.05 78.79± 0.29% Table 2: Results for English-to-French in terms of BLEU and COMET on the Multi30k testsets (separately and averaged over the testsets) and the accuracy on the CoMMuTE contrastive dataset. The results are averaged over three separate runs, and ± shows the standard error. Best results for each base model in each column are marked in bold, and the second best are underlined. Model test 2016 CoMMuTE BLEU COMET Accuracy Qwen3-VL 4B 23.12 83.43 67.21% Fine-tuning 35.82± 0.25 88.63± 0.05 74.35± 0.37% VMLM 35.66± 0.08 88.46± 0.10 74.68± 0.50% Metric Fine-tuning 35.19± 0.16 88.39± 0.06 75.54± 1.07% Word Dropout 35.72± 0.20 88.62± 0.09 75.43± 0.66% Degradation 36.19± 0.40 88.62± 0.02 74.89± 0.29% Metric Weighting 36.19± 0.31 88.46± 0.12 76.41± 0.43% Qwen3-VL 8B 26.82 86.37 74.03% Fine-tuning 36.17± 0.08 89.06± 0.06 78.03± 0.22% VMLM 36.99± 0.34 88.84± 0.13 78.46± 0.29% Metric Fine-tuning 36.42± 0.15 88.64± 0.03 79.98± 0.22% Word Dropout 35.98± 0.29 88.91± 0.12 80.84± 0.19% Degradation 36.68± 0.52 88.87± 0.12 78.57± 0.19% Metric Weighting 36.52± 0.24 88.81± 0.03 80.19± 0.32% IDEFICS 9B 10.29 70.42 64.29% Fine-tuning 36.80± 0.18 88.27± 0.04 68.72± 0.76% VMLM 36.41± 0.16 88.35± 0.11 70.02± 0.47% Metric Fine-tuning 36.33± 0.33 88.16± 0.09 71.43± 0.32% Word Dropout 36.89± 0.06 88.65± 0.09 73.27± 0.66% Degradation 36.66± 0.05 88.28± 0.09 72.84± 0.92% Metric Weighting 36.76± 0.18 88.15± 0.08 74.03± 0.32% Table 3: Results for English-to-Czech in terms of BLEU and COMET on the Multi30k test 2016 testset and the accuracy on the CoMMuTE contrastive dataset. The results are averaged over three separate runs, and ± shows the standard error. Best results for each base model in each column are marked in bold, and the second best are underlined. The results in terms of BLEU and COMET on the test 2016, test 2017, and MSCOCO testsets, and CoMMuTE accuracy are presented in Tables 1 and 2 for the English-to-German and English-to-French language directions. To improve the readability of the results, we additionally included the BLEU and COMET averaged over the testsets. The results in terms of BLEU and COMET on the test 2016 testset and CoMMuTE accuracy for the English-to-Czech language direction are shown in Table 3. Results are averaged over three runs for each method, and standard errors are reported in the tables. In terms of BLEU and COMET, Qwen3-VL 8B exhibited better performance than the other two models, with IDEFICX 9B scoring lower than the smaller Qwen-VL 4B model on all language directions. While this could be caused by different generation method (no sampling for the IDEFICX model), this gap is removed after fine-tuning on the Multi30k dataset. The CoMMuTE accuracy of the baseline models is similar between English-to-German and English-to-French, and there is a clear drop for English-to-Czech for Qwen-3-VL 4B and IDEFICS 9B models (more than 6 and 4 percentage points, respectively). This could be attributed to lower amounts of English-to-Czech parallel and Czech monolingual data in the training corpora. For English-to-German, fine-tuning reduces the visual grounding, as measured by the CoMMuTE accuracy, compared to the baseline model. Only Word Dropout and Metric-based Loss Weighting retrieve and surpass the performance of the baseline model. This effect is not visible for the two remaining language directions, where each technique improves the baseline performance. We hypothesize that the drop in CoMMuTE accuracy from English to German stems from linguistic proximity (both are Germanic languages). This proximity may make the translation task easier for the models to learn during fine-tuning, reducing their reliance on the visual modality to resolve ambiguity. We leave evaluating this hypothesis for future work. VMLM did not yield large improvements in visual sensitivity compared to other methods. We attribute this to our use of only a parallel multimodal dataset for this task. We expect it to show larger improvements when trained on a large-scale monolingual multimodal dataset. However, training it already used twice as much computational resources compared to other methods (see Appendix A), and scaling it further falls outside the scope of this work. English: We changed coaches. German: Wir haben den Waggon gewechselt. English: My chest is quite heavy… German: Mein Koffer ist ziemlich schwer. Figure 2: Two examples from the English-to-German CoMMuTE dataset with the per-token values of PCXMI and CPCXMI metrics for the Qwen3-VL 4B model. In red we mark the ambiguous words in the source side and the corresponding words in the target side. While the highest results of CoMMuTE accuracy are significantly different from the baseline models and Fine-tuning (according to Welch’s t-test), due to the low number of runs, the results for the top methods (marked as bold and underlined in the result tables) are not significantly different. Nevertheless, Metric-based Loss Weighting achieved the highest results in terms of CoMMuTE accuracy on 6 out of 9 experimental settings, and achieved the second-best result for the remaining three. Word Dropout showed the best result in three settings and was the second best in four of them. Source Degradation achieved the second best result on English-to-French with the Qwen3-VL 4B model, and Metric-based Fine-tuning on English-to-Czech with the same model. Notably, the high score on CoMMuTE does not correlate well with the achieved BLEU and COMET scores, indicating that the general translation quality and visual grounding are not necessarily mutual, which corroborates the findings of Futeral et al. (2025). This can be attributed to the examples in the Multi30k testsets not requiring images for good translation (Caglayan et al., 2019; Wu et al., 2021). We further investigate this phenomenon in Section 5.2, where we qualitatively analyze the selected translations in the MSCOCO English-to-French dataset. 5 Analysis In this section, we take a deeper look at the PCXMI and CPCXMI distributions. Next, we examine translation examples produced by our method and standard fine-tuning. Lastly, we experiment with combining our method with Word Dropout, another technique that achieves strong results. Additionally, in Appendix B we present the ablation study. 5.1 Metrics Analysis In Figure 2, we present two examples of the per-token values of the PCXMI and CPCXMI metrics. Further examples can be found in Appendix C. While both metrics are capable of identifying the tokens that require image for disambiguation, they are not perfect, and can ignore or misidentify those tokens. This can be partially mitigated by taking the maximum of both metrics. In Appendix C, we present the distributions of the calculated metrics (PCXMI, CPCXMI, and their maximum) on the training subset of the Multi30k dataset for each tested model and language direction. The values of the metrics depend heavily on the model, as can be seen in the difference between both the Qwen3-VL models and IDEFICS 9B. The difference between language directions for the same model is a lot smaller. Additionally, all distributions show a concentration of values close to zero and a long tail at high values. This means that most examples in the dataset do not require, according to the tested models, images to disambiguate. The distributions for Multi30k and CoMMute datasets are compared in Appendix C, where we show that the distribution for the latter is denser for higher values of the metric. 5.2 Qualitative Analysis In this section, we investigate the drop in BLEU exhibited by Metric-based Loss Weighting compared to Fine-tuning seen for Qwen3-VL 8B on the English-to-French language direction. We perform the qualitative analysis of the translations of the MSCOCO testset, where the difference in BLEU between the methods is the largest. In Figure 3, we present the distributions of per-example BLEU scores for Fine-tuning and our method. They differ mostly in two regions: (1) BLEU scores around 100, where Fine-tuning has more examples, (2) BLEU scores around 30, where the Weighting distribution is denser. Figure 3: Distributions of BLEU scores of Fine-tuning and Metric-based Loss Weighting on the MSCOCO testset for the English-to-French language direction. We examined the examples that exhibit the highest difference in BLEU between Fine-tuning and our method (both ways, with both Fine-tuning and our method achieving better scores). We found that the largest drop in example-level BLEU scores mostly corresponds to acceptable translations that differ from the gold target translation. More examples with our descriptions and further details of the analysis are presented in Appendix D. 5.3 Combining Weighting with Dropout The main results of the tested methods show that Metric-based Loss Weighting achieves the highest CoMMuTE accuracy. The second best performing method, Word Dropout, modifies the source side and is technically not related to our method, which changes the training signal corresponding to the target side. In this section, we test whether the combination of both methods could increase the visual sensitivity of the resulting models even further. For each example, the Metric-based Loss Weighting is applied (as described in Section 3). In cases where there is no token with the metric value greater than the threshold parameter τ, we apply Word Dropout to the source sentence with the specified per-token probability p. We test the combined method with the Qwen3-VL 4B model on all three tested language directions (the experimental configuration is the same as before). The results can be seen in Table 4, where we compare with Fine-tuning, and both methods (the full non-aggregated results for English-to-German and English-to-French can be found in Appendix E). The combined method achieves higher CoMMuTE accuracy on English-to-German and English-to-Czech, but did not improve the Metric-based Loss Weighting on English-to-French. Furthermore, the combined method shows reduced BLEU for all tested languages, but the reduction in COMET was not observed. This shows that both methods can work in tandem, increasing the visual grounding even further, but resulting in the distributions of outputs moving away from the gold translations. Model Average CoMMuTE BLEU COMET Accuracy English-to-German Qwen3-VL 4B 33.80 83.90 74.00% Fine-tuning 38.64± 0.18 85.11± 0.05 71.67± 0.19% Word Dropout 38.13± 0.18 85.03± 0.04 76.89± 0.44% Metric Weighting 38.30± 0.15 85.34± 0.07 77.00± 0.33% Weighting + Dropout 37.80± 0.11 85.23± 0.09 78.78± 0.11% English-to-French Qwen3-VL 4B 43.49 85.35 75.00% Fine-tuning 59.85± 0.08 88.14± 0.03 81.17± 0.37% Word Dropout 59.48± 0.27 87.94± 0.04 84.42± 0.68% Metric Weighting 59.59± 0.22 87.97± 0.03 84.74± 0.50% Weighting + Dropout 58.71± 0.22 87.93± 0.09 84.63± 0.39% English-to-Czech Qwen3-VL 4B 23.12 83.43 67.21% Fine-tuning 35.82± 0.25 88.63± 0.05 74.35± 0.37% Word Dropout 35.72± 0.20 88.62± 0.09 75.43± 0.66% Metric Weighting 36.19± 0.31 88.46± 0.12 76.41± 0.43% Weighting + Dropout 35.27± 0.39 88.69± 0.11 77.27± 0.37% Table 4: Results for combined Metric-based Loss Weighting and Word Dropout in terms of BLEU and COMET averaged over the test sets (and three separate runs) for English-to-German and English-to-French, and for test 2016 for English-to-Czech, and CoMMuTE accuracy. 6 Conclusions This work introduced a training strategy for fine-tuning MLLMs for the task of MMT that increases the visual grounding during translation. It increases the loss for the tokens that benefit from context in the form of the image, as measured by the Point-wise Cross-Mutual Information (PCXMI) and Congruency-based PCXMI (CPCXMI). Both metrics rely on the model’s output probabilities with and without the image (or with incongruent image in case of CPCXMI). We empirically tested our technique against existing methods adapted for the task of fine-tuning MLLMs on three models and three language directions, and showed its increased performance in terms of CoMMuTE accuracy, while maintaining strong overall translation quality, as measured by BLEU and COMET. Our results confirm that general-purpose translation metrics (BLEU and COMET) fail to capture how strongly models rely on visual modality for MMT. In future work, we will explore how the metrics can inform masking and word dropout on the source side. 7 Limitations While we investigate three language directions and three sub-families, all of them come from the Indo-European family, and all use the Latin script. This limitation was imposed by the language directions covered by both Multi30k and CoMMuTE datasets. We suspect that the measured performance of the methods goes beyond the tested language pairs, but this claim has not been tested experimentally. Due to the low number of repeated runs and the variability of the measured metrics, the results of the top methods are not different with statistical significance. We limited our investigation to only the same multimodal parallel dataset for each method, while VMLM could use additional multimodal monolingual data. For Metric-based Loss Weighting, we calculate the image dependence metrics using the baseline models. Updating the metrics during training or using a separate (presumably more visually sensitive) model could yield different results, but we did not test this setting. Furthermore, we limit our method to PCXMI and CPCXMI, and a broader exploration of image dependence metrics could lead to increased performance. Finally, while we tested the methods on three MLLMs, their sizes were limited to less than 10B parameters. References Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736. Awadalla et al. (2023) Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, and 1 others. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390v2. Bai et al. (2025) Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923v1. Barrault et al. (2018) Loïc Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304–323, Belgium, Brussels. Association for Computational Linguistics. Bowen et al. (2024) Braeden Bowen, Vipin Vijayan, Scott Grigsby, Timothy Anderson, and Jeremy Gwinnup. 2024. Detecting concrete visual tokens for multimodal machine translation. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 29–38, Chicago, USA. Association for Machine Translation in the Americas. Caglayan et al. (2019) Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loïc Barrault. 2019. Probing the need for visual context in multimodal machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4159–4170, Minneapolis, Minnesota. Association for Computational Linguistics. Chen et al. (2025) Andong Chen, Yuchen Song, Kehai Chen, Xuefeng Bai, Muyun Yang, Liqiang Nie, Jie Liu, Tiejun Zhao, and Min Zhang. 2025. Make imagination clearer! stable diffusion-based visual imagination for multimodal machine translation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26567–26583, Vienna, Austria. Association for Computational Linguistics. Chen et al. (2024) Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, and 1 others. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271v5. Cheng et al. (2024) Xuxin Cheng, Ziyu Yao, Yifei Xin, Hao An, Hongxiang Li, Yaowei Li, and Yuexian Zou. 2024. Soul-mix: Enhancing multimodal machine translation with manifold mixup. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11283–11294, Bangkok, Thailand. Association for Computational Linguistics. Costa-jussà et al. (2022) Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672v3. Elliott (2018) Desmond Elliott. 2018. Adversarial evaluation of multimodal machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2974–2978, Brussels, Belgium. Association for Computational Linguistics. Elliott et al. (2017) Desmond Elliott, Stella Frank, Loïc Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, pages 215–233, Copenhagen, Denmark. Association for Computational Linguistics. Elliott et al. (2016) Desmond Elliott, Stella Frank, Khalil Sima’an, and Lucia Specia. 2016. Multi30K: Multilingual English-German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70–74, Berlin, Germany. Association for Computational Linguistics. Fang and Feng (2022) Qingkai Fang and Yang Feng. 2022. Neural machine translation with phrase-level universal visual representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5687–5698, Dublin, Ireland. Association for Computational Linguistics. Feng et al. (2025) Yi Feng, Chuanyi Li, Jiatong He, Zhenyu Hou, and Vincent Ng. 2025. Multimodal neural machine translation: A survey of the state of the art. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 22130–22147, Suzhou, China. Association for Computational Linguistics. Fernandes et al. (2023) Patrick Fernandes, Kayo Yin, Emmy Liu, André Martins, and Graham Neubig. 2023. When does translation require context? a data-driven, multilingual exploration. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 606–626, Toronto, Canada. Association for Computational Linguistics. Fernandes et al. (2021) Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. 2021. Measuring and increasing context usage in context-aware machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6467–6478, Online. Association for Computational Linguistics. Futeral et al. (2023) Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden. 2023. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5394–5413, Toronto, Canada. Association for Computational Linguistics. Futeral et al. (2025) Matthieu Futeral, Cordelia Schmid, Benoît Sagot, and Rachel Bawden. 2025. Towards zero-shot multimodal machine translation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 761–778, Albuquerque, New Mexico. Association for Computational Linguistics. Gao et al. (2025) Yue Gao, Jing Zhao, Shiliang Sun, Xiaosong Qiao, Tengfei Song, and Hao Yang. 2025. Multimodal machine translation with text-image in-depth questioning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9274–9287, Vienna, Austria. Association for Computational Linguistics. Garg et al. (2022) Muskan Garg, Seema Wazarkar, Muskaan Singh, and Ondřej Bojar. 2022. Multimodality for NLP-centered applications: Resources, advances and frontiers. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6837–6847, Marseille, France. European Language Resources Association. Goodfellow et al. (2014) Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in neural information processing systems, 27. Gu et al. (2021) Weiqi Gu, Haiyue Song, Chenhui Chu, and Sadao Kurohashi. 2021. Video-guided machine translation with spatial hierarchical attention network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: Student Research Workshop, pages 87–92, Online. Association for Computational Linguistics. Guo et al. (2022) Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, and Zheng Cui. 2022. LVP-M3: Language-aware visual prompt for multilingual multimodal machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2862–2872, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Gupta et al. (2023) Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li, Hanspeter Pfister, and Donglai Wei. 2023. Cliptrans: transferring visual knowledge with pre-trained models for multimodal machine translation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2875–2886. Hassan et al. (2018) Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, and 1 others. 2018. Achieving human parity on automatic chinese to english news translation. arXiv preprint arXiv:1803.05567v2. Hu et al. (2022) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. Koehn et al. (2007) Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions, pages 177–180, Prague, Czech Republic. Association for Computational Linguistics. Laurençon et al. (2023) Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. 2023. Obelics: an open web-scale filtered dataset of interleaved image-text documents. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. Curran Associates Inc. Li et al. (2022a) Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022a. On vision features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6327–6337, Dublin, Ireland. Association for Computational Linguistics. Li et al. (2021) Jiaoda Li, Duygu Ataman, and Rico Sennrich. 2021. Vision matters when it should: Sanity checking multimodal machine translation models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8556–8562, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Li et al. (2022b) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022b. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR. Li et al. (2022c) Yi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen, Rogerio S Feris, David Cox, and Nuno Vasconcelos. 2022c. Valhalla: Visual hallucination for machine translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5216–5226. Li et al. (2023) Yihang Li, Shuichiro Shimizu, Chenhui Chu, Sadao Kurohashi, and Wei Li. 2023. Video-helpful multimodal machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4281–4299, Singapore. Association for Computational Linguistics. Liu et al. (2023) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems. Liu et al. (2021) Pengbo Liu, Hailong Cao, and Tiejun Zhao. 2021. Gumbel-attention for multi-modal machine translation. arXiv preprint arXiv:2103.08862v2. Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. Long et al. (2021) Quanyu Long, Mingxuan Wang, and Lei Li. 2021. Generative imagination elevates machine translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5738–5748, Online. Association for Computational Linguistics. Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. Mąka et al. (2025) Paweł Mąka, Yusuf Can Semerci, Jan Scholtes, and Gerasimos Spanakis. 2025. You are what you train: Effects of data composition on training context-aware machine translation models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27414–27437, Suzhou, China. Association for Computational Linguistics. Nishihara et al. (2020) Tetsuro Nishihara, Akihiro Tamura, Takashi Ninomiya, Yutaro Omote, and Hideki Nakayama. 2020. Supervised visual attention for multimodal neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4304–4314, Barcelona, Spain (Online). International Committee on Computational Linguistics. Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR. Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10674–10685. Sato et al. (2023) Julia Sato, Helena Caseli, and Lucia Specia. 2023. Choosing what to mask: More informed masking for multimodal machine translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 244–253, Toronto, Canada. Association for Computational Linguistics. Sulubacak et al. (2020) Umut Sulubacak, Ozan Caglayan, Stig-Arne Grönroos, Aku Rouhe, Desmond Elliott, Lucia Specia, and Jörg Tiedemann. 2020. Multimodal machine translation through visuals and speech. Machine Translation, 34(2):97–147. Tiedemann et al. (2024) Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Grönroos, Tommi Nieminen, Alessandro Raganato, Yves Scherrer, Raúl Vázquez, and Sami Virpioja. 2024. Democratizing neural machine translation with opus-mt. Language Resources and Evaluation, 58(2):713–755. Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In International Conference on Learning Representations. Wu et al. (2021) Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6153–6166, Online. Association for Computational Linguistics. Xiong and Zhao (2025) Jiafeng Xiong and Yuting Zhao. 2025. GIIFT: Graph-guided inductive image-free multimodal machine translation. In Proceedings of the Tenth Conference on Machine Translation, pages 98–112, Suzhou, China. Association for Computational Linguistics. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388v1. Yao and Wan (2020) Shaowei Yao and Xiaojun Wan. 2020. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4346–4350, Online. Association for Computational Linguistics. Yin et al. (2020) Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020. A novel graph-based multi-modal fusion encoder for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025–3035, Online. Association for Computational Linguistics. Yu et al. (2025) Zhuang Yu, Shiliang Sun, Jing Zhao, Tengfei Song, and Hao Yang. 2025. Imagination and contemplation: A balanced framework for semantic-augmented multimodal machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 10913–10928, Suzhou, China. Association for Computational Linguistics. Zhang et al. (2026) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Guoyin Wang, and Fei Wu. 2026. Instruction tuning for large language models: A survey. ACM Comput. Surv., 58(7). Zhang et al. (2025) Zhihui Zhang, Shiliang Sun, Jing Zhao, Tengfei Song, and Hao Yang. 2025. VQA-augmented machine translation with cross-modal contrastive learning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 10113–10124, Suzhou, China. Association for Computational Linguistics. Zhao et al. (2022) Yuting Zhao, Mamoru Komachi, Tomoyuki Kajiwara, and Chenhui Chu. 2022. Word-region alignment-guided multimodal neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:244–259. Zhu et al. (2024) Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2024. Multilingual machine translation with large language models: Empirical results and analysis. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2765–2781, Mexico City, Mexico. Association for Computational Linguistics. Zhu et al. (2023) Yaoming Zhu, Zewei Sun, Shanbo Cheng, Luyang Huang, Liwei Wu, and Mingxuan Wang. 2023. Beyond triplet: Leveraging the most data for multimodal machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2679–2697, Toronto, Canada. Association for Computational Linguistics. Appendix A Training Details We trained the models using the AdamW optimizer (Loshchilov and Hutter, 2019). We used a single NVIDIA GeForce RTX 3090 24GB for Qwen3-VL 4B, and a single NVIDIA A100 40GB for Qwen3-VL 8B and IDEFICS 9B. The average training times per epoch for each model are presented in Table 5. The time for Metric-based Fine-tuning is shown only for the second stage training (training on the top-k examples). The training time for VMLM is roughly doubled compared to fine-tuning, because of the auxiliary task of masked language modeling. The times for other methods are comparable, meaning that additional pre-processing (dropout, masking) and post-processing (loss weighting) do not meaningfully increase the training time compared to basic fine-tuning. The training was performed on the Multi30k dataset and evaluated on its testsets and the CoMMuTE contrastive dataset. The sizes of the used datasets can be found in Table 6. Model Epoch Time Qwen3-VL 4B Fine-tuning 1:43 VMLM 3:30 Metric Fine-tuning* 0:20 Word Dropout 1:43 Degradation 1:43 Metric Weighting 1:43 Qwen3-VL 8B Fine-tuning 1:16 VMLM 2:24 Metric Fine-tuning* 0:13 Word Dropout 1:14 Degradation 1:15 Metric Weighting 1:14 IDEFICS 9B Fine-tuning 0:51 VMLM 1:40 Metric Fine-tuning* 0:09 Word Dropout 0:49 Degradation 0:50 Metric Weighting 0:51 Table 5: Average training times per epoch for the tested models (in hours:minutes). *The time for Metric-based Fine-tuning is only for the second stage of the training. Dataset # Examples Multi30k Train 29,000 Validation 1014 Test 2016 1000 Test 2017 1000 MSCOCO 461 CoMMuTE Test 300 Validation 60 Table 6: The sizes of the used datasets in terms the number of examples. The shared hyper-parameters for all methods and models can be found in Table 7. For generation, we used the default parameters of all models: multinomial sampling with temperature 0.7 for Qwen3-VL models, and greedy decoding for IDEFICS 9B. The translation prompt can be seen in Listing 1, and the VMLM prompt in Listing 2. Listing 1: The translation prompt used for generation, contrastive evaluation on CoMMuTE, and metric calculation. The source and target languages (in natural language, e.g., "English" or "German") are represented as src_lang and tgt_lang, respectively. The source sentence is represented as src. The accompanying image is passed as the input alongside the prompt. ⬇ "Translate the following src_lang text to tgt_lang. src_lang: src tgt_lang:" Listing 2: The Visually Conditioned Masked Language Modeling (VMLM) prompt and response used for training. The source language (in natural language, e.g., "English") and masked source text are represented as src_lang and masked_text, respectively. masked_word1 and masked_word2 represent the masked words (the number of masked words can be different between examples). The accompanying image is passed as the input alongside the prompt. ⬇ Prompt: "The following src_lang sentence has some words replaced with mask tokens. Recover the masked words. Sentence: masked_text" Response: "MASK1: masked_word1, MASK2:masked_word2, ..." We tuned the specific hyper-parameters for each method: • Visually Conditioned Masked Language Modeling (VMLM) - token masking probability p∈0.1,0.2,0.3p∈\0.1,0.2,0.3\; • Metric-based Fine-tuning - learning rate values 1e−5,2e−5,5e−5,1e−4\1e-5,2e-5,5e-5,1e-4\ with the constant number of top-k examples for the second stage fine-tuning k=5000k=5000; • Word Dropout - token dropout probability p∈0.1,0.2,0.3p∈\0.1,0.2,0.3\; • Source Degradation - token masking probability p∈0.1,0.2,0.3p∈\0.1,0.2,0.3\ for each eligible word; • Metric-based Loss Weighting - threshold τ∈5,6,7τ∈\5,6,7\ for Qwen3-VL models and τ∈1,1.5,2τ∈\1,1.5,2\ for IDEFICS 9B (based on the PCXMI and CPCXMI distributions) with constant loss weighting α=8α=8. The tuning was performed based on the validation subsets of the Multi30k and CoMMuTE datasets. First, we took only the runs (averaged over 3 seeds) with the highest BLEU on the Multi30k validation (the values not lower than 0.5 BLEU below the highest for a particular model and language direction). Next, we selected the run with the highest accuracy on the CoMMuTE validation subset. While the validation dataset for Mult30k is available for all tested language directions, CoMMuTE only provides it for English-to-French. For this reason, we translated the target side (French) to both German and Czech using Gemini 3.5 Flash88 8 gemini-3.5-flash accessed on 10.07.2026 through Google API. The prompt for translation is presented in Listing 3. The found best hyper-parameters for each method and language direction can be found in Table 8. Listing 3: The prompt used to translate the CoMMuTE validation set from French to German and Czech. The source and target expressions in the original dataset are represented as src and tgt, respectively. The translation language (German and Czech) is marked as trans_lang. The accompanying image is passed as the input alongside the prompt. ⬇ "Translate the following English expression to trans_lang: src. Use the image and the following French translation as references: tgt. Output only the translation." Hyper-parameter Value Optimizer AdamW Learning Rate 0.0001* LR Scheduler Cosine LR Warmup Steps 500 Weight Decay 0.01 Max Grad Norm 1.0 Batch Size 8 Gradient Accumulation Steps 8 Num Epoch 10 Patience 2 Precision bf16 Lora α 64 Lora r 32 Lora Dropout 0.05 Max Length 200 Seeds 1, 2, 3 Table 7: The shared hyper-parameters of all tested fine-tuning methods. *We tuned the Learning Rate hyper-parameter for the second stage of the Metric-based Fine-tuning method. Model En-De En-Fr En-Cs Qwen3-VL 4B VMLM (p) 0.3 0.1 0.1 Metric Fine-tuning (lr) 1e-4 1e-4 1e-5 Word Dropout (p) 0.1 0.1 0.1 Degradation (p) 0.3 0.2 0.1 Metric Weighting (τ) 5 6 6 Qwen3-VL 8B VMLM (p) 0.1 0.1 0.3 Metric Fine-tuning (lr) 1e-5 2e-5 5e-5 Word Dropout (p) 0.1 0.1 0.1 Degradation (p) 0.3 0.3 0.2 Metric Weighting (τ) 5 6 5 IDEFICS 9B VMLM (p) 0.1 0.2 0.3 Metric Fine-tuning (lr) 1e-5 2e-5 2e-5 Word Dropout (p) 0.1 0.1 0.1 Degradation (p) 0.2 0.2 0.3 Metric Weighting (τ) 1.5 2 1.5 Table 8: Found best hyper-parameters for each method and language direction. For VMLM, Word Dropout and Degradation, the hyper-parameter is the probability of masking or dropout p. For Metric-based Fine-tuning, the hyper-parameter is the learning rate (lr) of the second stage training. For Metric-based Weighting, the hyper-parameter is the threshold τ. Appendix B Ablation Study In Figure 4, we present the results of the ablation study of the Metric-based Loss Weighting method in terms of BLEU and COMET scores averaged over all testsets, and CoMMuTE accuracy. We trained Qwen3-VL 4B model using our method with different hyper-parameters: type of metric (PCXMI, CPCXMI, and their maximum), α∈2,4,8,16α∈\2,4,8,16\, and τ∈4,5,6,7τ∈\4,5,6,7\. The results show that using the maximum of both metrics yield the higher CoMMuTE accuracy than using the metrics separately. The results for the α and τ hyper-parameters are stable, but we make the following observations. Increasing the α parameter lead to the increase in the CoMMuTE accuracy but can reduce the BLEU and COMMET scores when set too high. High τ values result in high BLEU and COMET scores, but lower CoMMuTE accuracies. This is expected as τ acts as a threshold for the token selection, and high values correspond to the lower number of tokens with increased loss. Figure 4: The results of applying the Metric-based Loss Weighting method to Qwen3-VL 4B model on the English-to-German language direction with different values of hyper-parameters. The results are in terms of the BLEU and COMET scores averaged over all used testsets, and the accuracy on CoMMuTE dataset. We averaged the results over three separate runs and error bars show the standard error. Appendix C Extended Metrics Analysis In Figure 7, we present more examples of per-token values of the metrics from the CoMMuTE dataset. The metrics are capable of correctly identifying the image-dependent tokens in the target sentences. Figure 5 shows the distribution of values of the metrics (PCXMI, CPCXMI and the maximum of both) on the Multi30k training dataset. We take the maximum value of a metric of the target tokens for each example in the dataset. For both Qwen3-VL models and all language directions, the CPCXMI is lower than PCXMI, showing higher density for values close to zero. This is not the case for IDEFICS 9B, where the distributions are very similar for all language directions. The comparison between the distributions of the maximum of PCXMI and CPCXMI on the training split of Multi30k and the CoMMuTE dataset is shown in Figure 6. The metric for CoMMuTE is not as concentrated close to zero as for Multi30k, which is consistent with the presence of ambiguous words in every example of the CoMMuTE dataset. Figure 5: Kernel Density Estimate visualization of distributions of metrics (PCXMI, CPCXMI, and the maximum of both) in the Multi30k training dataset for each tested model and language direction. We take the maximum token-level metric for each example in the dataset. Figure 6: Kernel Density Estimate visualization of distributions of Maximum of PCXMI and CPCXMI in the Multi30k training and CoMMuTE datasets for each tested model and language direction. We take the maximum token-level metric for each example in the dataset. English: There are still a couple of bugs left. German: Es gibt noch ein paar Insekten. English: Give me the file please. German: Geben Sie mir bitte die Nagelfeile. English: We’ve got another jam I’m afraid. German: Ich fürchte, wir haben wieder einen Papierstau. English: The tube looks dirty! German: Die U-Bahn sieht dreckig aus! English: First we should fill up the boot. German: Zuerst müssen Sie den Stiefel füllen. English: I have a club. German: Ich habe einen Nachtclub. English: Just a single tear, but it was enough. German: Es war nur ein einziger Riss, aber er reichte aus. English: An animal sitting in a palm. German: Ein Tier, das in einem Handteller sitzt. Figure 7: Selected examples from the English-to-German CoMMuTE dataset with the per-token values of PCXMI and CPCXMI metrics for the Qwen3-VL 4B model. In red we mark the ambiguous words the source side and the corresponding words on the target side. Appendix D Extended Qualitative Analysis To examine the source of the largest measured drop in BLEU between Fine-tuning and Metric-based Loss Weighting in our experiments, we analyzed the translations of Qwen3-VL 8B on MSCOCO English-to-French testset of the runs with seed 1, which also shows the highest BLEU difference. We looked at the 20 examples with the highest difference of example-level BLEU scores between the two methods (20 in each direction). The selected instances for Fine-tuning scoring higher can be seen in Figure 8, and in Figure 9 for Weighting scoring higher. We categorize the causes of the drop in BLEU into three categories: (1) change of the tense (mostly present continuous instead of present simple), (2) synonyms or equivalent expressions, (3) true errors including omissions and incorrect meaning. Out of the 20 instances with highest drop of BLEU for Weighting 30% were due to the use of a different form, 45% were equivalent, and 25% were erroneous. From the 20 instances where Fine-tuning showed the largest drop in BLEU 10% were due to a different form, 50% were synonyms, and 40% contained errors. While this analysis only looked at the selected examples, it shows that the largest drop in example-level BLEU scores mostly corresponds to acceptable translations that differ from the gold target translation. English: A young woman on a snowboard putting on gloves. French: Une jeune femme sur un snowboard mettant des gants. Fine-tuning: Une jeune femme sur un snowboard mettant des gants. Weighting: Une jeune femme en snowboard en train de mettre des gants. BLEU: 100.00/22.03 (-77.97) | COMET: 96.36/86.10 (-10.26) Weighting used present continuous. English: A person that is playing in a tennis game. French: Une personne qui joue un match de tennis. Fine-tuning: Une personne qui joue un match de tennis. Weighting: Une personne qui joue au tennis. BLEU: 100.00/36.74 (-63.26) | COMET: 97.80/81.47 (-16.33) Weighting did not include the tennis match. English: A man pulling a pizza out of an oven. French: Un homme sortant une pizza d’un four. Fine-tuning: Un homme sortant une pizza d’un four. Weighting: Un homme en train de sortir une pizza d’un four. BLEU: 100.00/40.35 (-59.65) | COMET: 97.05/92.57 (-4.48) Weighting used present continuous. English: There is a man putting on a tie. French: Il y a un homme qui met une cravate. Fine-tuning: Il y a un homme qui met une cravate. Weighting: Il y a un homme en train de mettre une cravate. BLEU: 100.00/42.40 (-57.60) | COMET: 95.79/79.28 (-16.51) Weighting used present continuous. English: A dog being blown dry by a man. French: Un chien séché par un homme. Fine-tuning: Un chien séché par un homme. Weighting: Un chien en train d’être séché par un homme. BLEU: 100.00/45.18 (-54.82) | COMET: 95.76/75.62 (-20.14) Weighting used present continuous. English: A table set for a dinner of pizza. French: Une table dressée pour un dîner de pizza. Fine-tuning: Une table dressée pour un dîner de pizza. Weighting: Une table dressée pour un dîner avec des pizzas. BLEU: 100.00/53.73 (-46.27) | COMET: 97.67/92.51 (-5.16) Weighting translated as "dinner with pizzas". English: A man is reading a book while another is holding a dog. French: Un homme est en train de lire un livre pendant qu’un autre tient un chien. Fine-tuning: Un homme est en train de lire un livre tandis qu’un autre tient un chien. Weighting: Un homme lit un livre tandis qu’un autre tient un chien. BLEU: 81.54/36.95 (-44.59) | COMET: 92.98/86.83 (-6.15) Weighting used present simple. English: Baby in yellow holds on to her teddy bear. French: Le bébé en jaune tient son ours en peluche. Fine-tuning: Un bébé en jaune tient son ours en peluche. Weighting: Un bébé en jaune se tient à son ours en peluche. BLEU: 88.01/43.67 (-44.34) | COMET: 92.68/85.95 (-6.73) Weighting changed the meaning to "supporting oneself". English: A man covering his eyes next to cases of bananas French: Un homme couvrant ses yeux à côté des boîtiers de bananes Fine-tuning: Un homme couvrant ses yeux à côté de caisses de bananes Weighting: Un homme se couvrant les yeux à côté de caisses de bananes BLEU: 63.16/19.16 (-44.00) | COMET: 89.27/85.48 (-3.79) Weighting used the reflexive form. Figure 8: Selected examples with the highest difference in BLEU between Fine-tuning and Metric-based Loss Weighting (Fine-tuning scoring higher) in the MSCOCO testset on the English-to-French language direction of Qwen3-VL 8B. We also report BLEU and COMET scores for the translations by the methods (and their differences), and our description of errors. English: A pizza sitting next to a bowl of salad. French: Une pizza posée à côté d’un bol de salade. Fine-tuning: Une pizza placée à côté d’une salade. Weighting: Une pizza posée à côté d’un bol de salade. BLEU: 14.90/100.00 (85.10) | COMET: 88.46/97.40 (8.94) Fine-tuning ommited the bowl (of salad) English: A cop leads the way for other people on motorcycles. French: Un policier ouvre la voie à d’autres personnes sur des motos. Fine-tuning: Un policier dirige la route pour d’autres personnes en motos. Weighting: Un policier ouvre la voie pour d’autres personnes sur des motos. BLEU: 12.32/73.49 (61.17) | COMET: 85.39/93.69 (8.30) Fine-tuning used the incorrect verb "to direct". English: a man moving a refrigerator in to a house French: un homme transportant un réfrigérateur dans d’une maison Fine-tuning: Un homme en train de mettre un réfrigérateur dans une maison Weighting: un homme transportant un réfrigérateur dans une maison BLEU: 15.85/70.71 (54.86) | COMET: 78.21/92.69 (14.48) Fine-tuning used present continuous and a synonym. English: A black cat climbing into a bath tub. French: Un chat noir grimpant dans une baignoire. Fine-tuning: Un chat noir montant dans une baignoire. Weighting: Un chat noir grimpant dans une baignoire. BLEU: 50.00/100.00 (50.00) | COMET: 86.99/96.85 (9.86) Fine-tuning used a synonym. English: A camera set up in front of a sleeping cat. French: Un appareil photo installé devant un chat qui dort. Fine-tuning: Une caméra installée devant un chat endormi. Weighting: Un appareil photo installé devant un chat endormi. BLEU: 17.20/67.17 (49.97) | COMET: 72.14/73.10 (0.96) Fine-tuning used incorrect "video camera". English: A dog climbing on top of a person sleeping on a couch. French: Un chien qui grimpe sur une personne qui dort sur un canapé. Fine-tuning: Un chien escaladant un individu dormant sur un canapé. Weighting: Un chien grimpant sur une personne qui dort sur un canapé. BLEU: 23.14/70.77 (47.62) | COMET: 71.39/90.27 (18.88) Fine-tuning used incorrect in the context "climbing (e.g., on a mountain)" and a synonym. English: A blue, yellow and green surfboard sticking out of a sandy beach. French: Une planche de surf bleue, jaune et verte plantée dans le sable d’une plage. Fine-tuning: Une planche de surf bleue, jaune et verte dépassant d’une plage de sable. Weighting: Une planche de surf bleue, jaune et verte plantée dans le sable d’une plage. BLEU: 58.22/100.00 (41.78) | COMET: 68.71/95.68 (26.97) Fine-tuning used incorrect in the context "stick out". English: A man in a blue jacket lighting a cigarette. French: Un homme en veste bleue allumant une cigarette. Fine-tuning: Un homme en veste bleue allume une cigarette. Weighting: Un homme en veste bleue allumant une cigarette. BLEU: 59.69/100.00 (40.31) | COMET: 92.68/97.31 (4.63) Fine-tuning used present simple. English: A woman putting on a jacket while looking at a clock French: Une femme mettant une veste en regardant une horloge Fine-tuning: Une femme en train de mettre une veste tout en regardant une horloge Weighting: Une femme mettant une veste tout en regardant une horloge BLEU: 26.13/65.80 (39.67) | COMET: 87.06/94.40 (7.34) Fine-tuning used present continuous. Figure 9: Selected examples with the highest difference in BLEU between Metric-based Loss Weighting and Fine-tuning (Weighting scoring higher) in the MSCOCO testset on the English-to-French language direction of Qwen3-VL 8B. We also report BLEU and COMET scores for the translations by the methods (and their differences), and our description of errors. Appendix E Extended Results In this section, we present the full results for the combined method (Metric-based Loss Weighting and Word Dropout). Table 9 shows the results for all testsets on the English-to-German and English-to-French language directions for the Qwen3-VL 4B model. Model test 2016 test 2017 MSCOCO Average CoMMuTE BLEU COMET BLEU COMET BLEU COMET BLEU COMET Accuracy English-to-German Qwen3-VL 4B 38.51 85.56 32.95 84.90 29.94 81.24 33.80 83.90 74.00% Fine-tuning 42.61± 0.12 86.66± 0.06 38.08± 0.38 85.98± 0.06 35.22± 0.45 82.70± 0.13 38.64± 0.18 85.11± 0.05 71.67± 0.19% Word Dropout 42.20± 0.19 86.62± 0.06 37.17± 0.19 85.80± 0.01 35.00± 0.22 82.69± 0.16 38.13± 0.18 85.03± 0.04 76.89± 0.44% Metric Weighting 41.91± 0.33 86.80± 0.04 37.29± 0.14 86.14± 0.04 35.71± 0.19 83.09± 0.13 38.30± 0.15 85.34± 0.07 77.00± 0.33% Weighting + Dropout 41.56± 0.49 86.87± 0.07 36.85± 0.36 86.04± 0.07 34.99± 0.06 82.80± 0.19 37.80± 0.11 85.23± 0.09 78.78± 0.11% English-to-French Qwen3-VL 4B 43.77 86.04 43.47 86.31 43.24 83.69 43.49 85.35 75.00% Fine-tuning 67.03± 0.36 89.81± 0.05 61.62± 0.13 89.25± 0.10 50.90± 0.43 85.37± 0.12 59.85± 0.08 88.14± 0.03 81.17± 0.37% Word Dropout 66.46± 0.31 89.58± 0.09 60.98± 0.05 89.10± 0.02 51.01± 0.44 85.15± 0.04 59.48± 0.27 87.94± 0.04 84.42± 0.68% Metric Weighting 67.22± 0.28 89.69± 0.03 61.04± 0.23 89.15± 0.04 50.50± 0.37 85.07± 0.03 59.59± 0.22 87.97± 0.03 84.74± 0.50% Weighting + Dropout 65.74± 0.39 89.49± 0.10 59.69± 0.07 89.12± 0.07 50.71± 0.32 85.19± 0.19 58.71± 0.22 87.93± 0.09 84.63± 0.39% Table 9: Results for combined Metric-based Loss Weighting and Word Dropout for all testsets in English-to-German and English-to-French. The results are averaged over three separate runs.