Paper deep dive
Toward domain-specific machine translation and quality estimation systems
Javad Pourmostafa Roshan Sharami
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/27/2026, 1:18:51 AM
Summary
This dissertation investigates methods to mitigate domain mismatch in Machine Translation (MT) and Quality Estimation (QE) systems. It proposes data-centric and model-centric approaches, including similarity-based data selection, staged QE training pipelines, optimized subword tokenization for fine-tuning, and QE-guided in-context learning for large language models to improve translation quality and efficiency.
Entities (6)
Relation Signals (3)
Javad Pourmostafa Roshan Sharami â authored â Toward domain-specific machine translation and quality estimation systems
confidence 100% · Dissertation title page
Quality Estimation â guides â In-Context Learning
confidence 90% · Chapter 5 proposes QE-guided in-context learning
Domain Adaptation â improves â Machine Translation
confidence 90% · Chapter 2 and 3 focus on improving MT/QE via DA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Machine Translation (MT) and Quality Estimation (QE) perform well in general domains but degrade under domain mismatch. This dissertation studies how to adapt MT and QE systems to specialized domains through a set of data-focused contributions. Chapter 2 presents a similarity-based data selection method for MT. Small, targeted in-domain subsets outperform much larger generic datasets and reach strong translation quality at lower computational cost. Chapter 3 introduces a staged QE training pipeline that combines domain adaptation with lightweight data augmentation. The method improves performance across domains, languages, and resource settings, including zero-shot and cross-lingual cases. Chapter 4 studies the role of subword tokenization and vocabulary in fine-tuning. Aligned tokenization-vocabulary setups lead to stable training and better translation quality, while mismatched configurations reduce performance. Chapter 5 proposes a QE-guided in-context learning method for large language models. QE models select examples that improve translation quality without parameter updates and outperform standard retrieval methods. The approach also supports a reference-free setup, reducing reliance on a single reference set. These results show that domain adaptation depends on data selection, representation, and efficient adaptation strategies. The dissertation provides methods for building MT and QE systems that perform reliably in domain-specific settings.
Tags
Links
- Source: https://arxiv.org/abs/2603.24955v1
- Canonical: https://arxiv.org/abs/2603.24955v1
Trouble viewing inline? Open PDF directly â
Full Text
347,513 characters extracted from source content.
Expand or collapse full text
arXiv:2603.24955v1 [cs.CL] 26 Mar 2026 TOWARD DOMAIN-SPECIFIC MACHINE TRANSLATION AND QUALITY ESTIMATION SYSTEMS Javad Pourmostafa Roshan Sharami Toward Domain-Specific Machine Translation and Quality Estimation Systems Proefschrift ter verkrijging van de graad van doctor aan Tilburg University op gezag van de rector magnificus, prof. dr. W.B.H.J. van de Donk, in het openbaar te verdedigen ten overstaan van een door het college voor promoties aangewezen commissie in de Aula van de Universiteit op woensdag 4 maart 2026 om 16.30 uur door Javad Pourmostafa Roshan Sharami geboren te Someesara, Iran Promotor: prof. dr. ir. P.H.M. Spronck (Tilburg University) Copromotor: dr. D. Shterionov (Tilburg University) Leden promotiecommissie: prof. dr. D. Kenny (Dublin City University) prof. dr. J. Monti (UniversitĂ di Napoli LâOrientale) prof. dr. A. Alishahi (Tilburg University) dr. A. Tezcan (Ghent University) dr. A. Karakanta (Leiden University) ISBN 978-94-6537-044-6 Printed by Ridderprint (ridderprint.nl) Cover design by Arina van Londen (arinavanlonden.com) © 2026 Javad Pourmostafa Roshan Sharami, The Netherlands. All rights reserved. No parts of this thesis may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without the permission of the author. Alle rechten voorbehouden. Niets uit deze uitgave mag worden vermenigvuldigd, in enige vorm of op enige wijze, zonder voorafgaande schriftelijke toestemming van de auteur. For Mahsa CONTENTS Glossary of Acronyms7 1 Introduction9 1.1 Domains and how machine translation adapts to them . 11 1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . 14 1.3 Research questions . . . . . . . . . . . . . . . . . . . . . 16 1.4 Dissertation overview . . . . . . . . . . . . . . . . . . . . 19 1.5 Evaluation metrics . . . . . . . . . . . . . . . . . . . . . 25 1.6 Computational considerations and efficiency . . . . . . . 26 1.7 List of publications . . . . . . . . . . . . . . . . . . . . . 27 2 Generating In-domain data for Machine Translation 33 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 35 2.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . 37 2.3 Data selection method . . . . . . . . . . . . . . . . . . . 40 2.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.5 Discussions . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.6 Conclusion and future work . . . . . . . . . . . . . . . . 61 2.7 Python tool . . . . . . . . . . . . . . . . . . . . . . . . . 63 3 Domain Adaptation for Machine Translation Quality Estimation69 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.2 Domain adaptation for specialized quality estimation . . 74 3.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 79 4 3.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 3.5 Additional observations . . . . . . . . . . . . . . . . . . 87 3.6 Related work . . . . . . . . . . . . . . . . . . . . . . . . 92 3.7 Conclusion and future work . . . . . . . . . . . . . . . . 94 3.8 Supplemental material . . . . . . . . . . . . . . . . . . . 96 4 Analysis of Vocabulary and Subword Tokenization Settings for Optimal Fine-tuning of Machine Translation101 4.1 Introduction and background . . . . . . . . . . . . . . . 103 4.2 Decision points . . . . . . . . . . . . . . . . . . . . . . . 106 4.3 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 109 4.5 Results and analysis . . . . . . . . . . . . . . . . . . . . 112 4.6 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 118 4.7 Conclusion and future work . . . . . . . . . . . . . . . . 119 5 In-Context Learning of LLMs through Quality Esti- mation for Machine Translation125 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.2In-context learning using quality estimation for machine translation . . . . . . . . . . . . . . . . . . . . . . . . . . 129 5.3 Experiments setup . . . . . . . . . . . . . . . . . . . . . 132 5.4 Experiments results . . . . . . . . . . . . . . . . . . . . 141 5.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 5.6 Related work . . . . . . . . . . . . . . . . . . . . . . . . 148 5.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 150 5 6 Discussion153 6.1 Summary of findings . . . . . . . . . . . . . . . . . . . . 155 6.2 Limitations and future work . . . . . . . . . . . . . . . . 167 6.3 Implications of the findings . . . . . . . . . . . . . . . . 172 6.4 Wrapping up . . . . . . . . . . . . . . . . . . . . . . . . 176 Summary179 Samenvatting (Dutch Summary)183 Acknowledgments189 References192 Appendix A: Efficiency and Sustainability Overview223 6 Glossary of Acronyms This glossary provides an overview of the acronyms that appear frequently throughout the dissertation. AcronymDefinition MTmachine translation SMTstatistical machine translation QEquality estimation NMTneural machine translation LLMlarge language model PLMpre-trained language model ICLin-context learning ICEin-context example DAdomain adaptation DAGdata augmentation SWsubword ZSLzero-shot learning ZSzero-shot RQresearch question NLPnatural language processing 1 Introduction M achine translation (MT) has come a long way since its inception, marked by major milestones, breakthroughs, and challenges. One of the earliest significant events was the Georgetown-IBM experiment in 1954, a collabo- ration between IBM and Georgetown University that successfully trans- lated some Russian sentences into English using the IBM 701 computer and a limited vocabulary. This early (although limited) success show- cased the potential of automated translation and significantly boosted public interest in using computers for language translation(Hutchins, 2006). However, soon after, due to the high costs and slower translation speeds compared to human translators, MT faced a notable setback thanks to the ALPAC report in the 1960s, which led to a freeze in U.S. funding for MT research(Nirenburg, Somers, & Wilks, 2003). While this report was disappointing, it also highlighted the danger of over-promising the capabilities of MT systems(Koehn, 2020), a lesson that can be broadly applied across other AI-related fields. Despite these setbacks, research persisted, leading to the rise of statistical machine translation (SMT) in the 1980s. SMT leveraged bilingual corpora and probabilistic methods, which led to enhanced translation accuracy compared to earlier paradigms, setting the stage for a more transformative breakthrough: neural machine translation (NMT). While NMT is still fundamentally statistical, it replaces phrase- based tables and hand-crafted probability estimates with artificial neural networks that learn to directly model the conditional probabilities of translations. By the 2010s, deep learning techniques, particularly the Transformer model(Vaswani et al., 2017a), revolutionized MT by enabling more fluent, context-aware translations. Building on these advancements, large-scale pre-trained models have further advanced MT, improving, among others, adaptability for specialized domains and low- resource languages. These models go beyond word-for-word translation, capturing meaning, context, and nuance more effectively. Yet, despite these improvements, MT continues to face ongoing challenges and new research directions. A major driver of recent progress in MT has been the availability of large parallel corpora, such as those provided by the Opus plat- form(Tiedemann, 2012), which have strengthened data-driven MT systems. Also, initiatives like the WMT shared tasks(Koehn, Haddow, Kocmi, & Monz, 2023) have played a crucial role in fostering advance- ments in translation quality, evaluation metrics, and quality estimation (QE). These annual competitions provide standardized benchmarks, encourage the development of novel MT techniques, and facilitate direct comparisons between different approaches. The growing integration of generative large language models (LLMs) has also shifted MT to- ward more flexible, in-context translation, allowing models to adapt to various domains with minimal fine-tuning. Despite these advances, significant challenges and unexplored areas persist. Key issues include but are not limited to domain mismatch, 10 support for rare words, addressing the long sentences issue, lacking the volume of training dataâspecifically in low-resource languages, word alignment, and decoding strategies(Koehn & Knowles, 2017). While some of these challenges have seen notable improvementsâsuch as word alignment and handling rare words using NMTâthey continue to be actively researched, with ongoing advancements aimed at further enhancing translation accuracy and adaptability. Among these challenges, domain mismatch persists despite ongoing research. This issue occurs when an MT system is trained on data from one domain but used to translate text from a different domain, leading to vocabulary discrepancies and context mistranslations. For example, an MT model trained solely on the chemical domain is unlikely to perform well on the IT domain, let alone reach human parity(Saunders, 2021). Addressing domain mismatch has been a central focus of research since the early days of MT. It is safe to say that the study of domain mismatch and methods to mitigate this phenomenon is one of the most extensively researched areas in the field due to its widespread relevance and significant impact on various MT applications. Beyond degrading MT performance, domain mismatch also affects evaluation quality, both in reference-based and reference-free evaluation methods(Rei, Stewart, Farinha, & Lavie, 2020; Sharami, Shterionov, Blain, et al., 2023; Zouhar, Ding, et al., 2024). 1.1 Domains and how machine translation adapts to them To address domain mismatch effectively, we must first understand what constitutes a âdomain.â As explored in the work of van der Wees, Bisazza, and Monz (2017) and later Saunders (2021), the term âdomainâ has been defined in multiple ways, each depending on the perspective or application. One common definition involves the concept of provenance or originâessentially, where the text comes from. This could refer to 11 the source of the dataset (e.g., the EN-DE WMT QE shared task 2023) or any distinguishing label that sets it apart from other types of data. Domains can also be understood in terms of topics (e.g., IT or medical), which imply a concentration of field-specific vocabulary and terminology, or in terms of genre, which includes not only subject matter but also stylistic and structural characteristicsâfactors that can vary significantly even within a single dataset, particularly in user-generated content(van der Wees et al., 2017). While these perspectives are valuable, they often introduce ambiguities that make domain boundaries difficult to define consistently. In this dissertation, we therefore adopt a narrower view and focus primarily on provenance, i.e., the origin of the training data. Provenance offers an observable and reproducible way of distinguishing between domains, since MT datasets are typically collected, labeled, and released with reference to their source or origin. This makes it particularly suitable for experimental work in domain adaptation (DA), where comparability and reproducibility across studies are essential. The preference for a provenance-based definition of domain is mainly methodological. In MT, adaptation depends more on linguis- tic formâvocabulary, phrasing, and styleâthan on conceptual topic. Provenance, which reflects where and how data were produced, provides a concrete and reproducible proxy for these characteristics. Topic-based labels, by contrast, are often subjective and vary even within a field; for example, âmedicalâ datasets may range from clinical reports to public health texts. Thus, provenance offers a stable linguistic basis on which topic-based distinctions can be interpreted. By grounding domain in provenance, this dissertation establishes a reproducible and linguistically informed framework for studying and improving DA. This framing also clarifies how topic-based interpreta- tions relate to the empirical design of this work: provenance defines the data used for experimentation, while topic-based perspectives guide the 12 interpretation of results across different applied areas such as healthcare, law, and IT. In this work, DA of an MT model refers to enabling the model to recognize and adapt to the origin or provenance of training data, specif- ically its domain-specific characteristics as defined by source corpus, collection setting, or dataset label. The goal is to ensure that the MT model can accurately translate within a specific domain, implying that it sufficiently covers the domain regardless of its type. An MT model that is adapted to a domain is expected to deliver high-quality translations that are faithful not only to the source text but also to the expectations of the domain it serves. For example, a model trained on medical datasets should accurately translate specialized terms, use the appropriate register 1 and avoid ambiguity that could lead to misunderstandings in critical contexts. To address domain mismatch, researchers have proposed a broad range of DA techniques, which are generally grouped into two main categories: data-centric and model-centric, as summarized in the surveys by Chu and Wang (2018) and Saunders (2022). Data-centric approaches emphasize curating or selecting training data that are representative of the target domain. Model-centric approaches, on the other hand, focus on modifying the modelâs learning process, architecture, or decoding strategies to make it more responsive to domain-specific input. The studies in this dissertation follow both of these categories, with a majority being data-centric. Specifically, Chapters 2, 3, and 4 follow a data-centric approach, while Chapter 5 is model-centric. This dissertation presents a series of interconnected studies aimed at understanding and addressing the problem of domain mismatch in MT. 1 Throughout this dissertation, references to domains such as âmedical,â âlegal,â or âITâ are used as conceptual abstractions to help illustrate ideas, while the underlying analysis follows a provenance-based view. In other words, these topic labels serve as intuitive references for readers, but the research itself treats domains in terms of their linguistic and data provenance characteristics. 13 In the remainder of this chapter, we introduce the problem statement addressed in this dissertation, followed by the specific research questions (RQs) that guide our investigation. We then conclude the chapter with an overview of the dissertation structure, outlining the rationale and relevance of each chapter in relation to the overarching research objectives. 1.2 Problem statement As language technology becomes more central to our daily lives, the role of MT systems in supporting specialized domainsâsuch as healthcare, law, and ITâhas never been more critical. Yet, the very success of MT hinges on its ability to adapt to the nuanced needs of these domains. A generic translation that misinterprets a medical dosage or a legal clause is not just a technical failureâit could be dangerous and/or costly. This underlines a key insight: not all data is created equal. That is, the quantity alone is not enough; the dataâs relevance and specificity to the domain truly determine MT performance. As the volume of multilingual data continues to grow, it becomes clear that domain relevance matters more than ever(R. Wang, Finch, Utiyama, & Sumita, 2017; Pourmostafa Roshan Sharami, Shterionov, & Spronck, 2021). This insight has important consequences for how we build and de- ploy MT systems. General-purpose models, including powerful LLMs, often struggle when confronted with the specialized terminology, style, and context required in professional settings(Wassie, Molaei, & Moslem, 2024; Zheng et al., 2024). Even when fine-tuned, these models typically demand large volumes of domain-specific data to approach the quality of smaller, domain-specific systems(Eschbach-Dymanus, Essenberger, Buschbeck, & Exel, 2024). And the challenge does not stop at trans- lation. QE systems, which assess the reliability of translations, are 14 equally vulnerable to domain mismatch, leading to inaccurate quality scores, unnecessary human intervention, and increased costs. This dissertation identifies several intertwined challenges to building MT and QE systems that truly support domain-specific needs: (1) Large-scale MT systemsâincluding LLMs such as GPT-4, used for instance in legal settings(Wan, Zhang, Wang, Cheng, & Kurohashi, 2024)âtrained on general-domain data often fail to perform well in specialized domains due to a lack of domain-specific data. (2) Unlike MT, which can often generalize across domains given sufficient train- ing data, QE models are especially vulnerable to domain mismatch. They depend on scarce, high-quality annotated datasets, making it challenging to ensure accurate and reliable automated translation eval- uation in real-world applications(de Souza, Turchi, & Negri, 2014a). (3) When adapting MT models to new domains, simply introducing domain-specific data is not enoughâwithout careful control over vo- cabulary and subword (SW) tokenization, the model can misinterpret new terminology, leading to degraded translation quality and wasted training effort. While the issue of unknown or out-of-vocabulary (OOV) words has been largely mitigated by advanced SW tokenization algo- rithms, improper or inconsistent SW splits can still lead to suboptimal translations(Li, Guo, Guerin, & Lin, 2024). (4) in-context learning (ICL) allows LLMs to translate without fine-tuning, but their success depends entirely on finding the right examplesâones that match the domain, context, and complexity of the source textâyet identifying such examples without human input or costly reference translations remains a major challenge. These challenges define the central problem addressed in this dissertation: Problem Statement: How can we design MT and QE systems that are accurate, adaptable, and efficient across specialized domains? To address this problem, in Chapter 2, we begin by tackling one of 15 the most fundamental issues: the scarcity of high-quality, domain- specific data. Moreover, we introduce a lightweight, data-centric methodology for selecting in-domain examples from generic corpora to improve translation quality while minimizing computational overhead. This approach is supported by an easy-to-use Python tool designed to help the broader research community replicate and scale the method efficiently. We also investigate how much in-domain data is needed to reach strong performance without incurring large computational costs. Next, recognizing that domain mismatch also affects QE, Chap- ter 3 explores the adaptation of QE models using both DA and data augmentation (DAG). 2 We demonstrate that just like MT models, QE systems benefit significantly from domain-specific training, especially when reference data are limited or unavailable. Chapter 4 investigates how the choice of vocabulary and SW tok- enization strategies impacts the added value of DA. Then, in Chapter 5, we introduce an ICL approach that uses domain-adapted QE models to select better 3 ICEs for LLMs. This not only improves translation quality but also offers a more sustainable alternative to fine-tuning, with reduced carbon emissions and computational demands. In summary, these contributions offer practical, efficient, and scal- able solutions for enhancing the adaptability of MT and QE systems across various domains. 1.3 Research questions To guide our investigation, this dissertation is structured around four core challenges. Each is framed as a main RQâaccompanied by sub- questionsâthat drives the research, development, literature review, and empirical efforts presented in the subsequent chapters. 2 âDAGâ denotes data augmentation, to avoid confusion with DA. 3 âbetterâ means in-context examples (ICEs) that lead to better translation or faster inference. 16 RQ1What is the optimal amount of in-domain data required to achieve state-of-the-art MT quality at low computational costs? RQ1a: How does the quality of selected in-domain data affect translation performance? RQ1b:What trade-offs arise between translation performance and computational cost when using in-domain versus generic- domain data? In Chapter 2, we explore whether it is possible to find and extract in-domain data from publicly available bilingual corpora to support high-quality, domain-adapted MT. To this end, we introduce a data selection method tailored to retrieve similar 4 in-domain examples from generic datasets and run experiments to see how they compare to generic-domain dataâespecially in terms of how much performance gain they offer for the computational effort required. RQ2What are the effects of combining DA and DAG on the perfor- mance and generalizability of QE models across different domains and languages? RQ2a: How does DA impact QE performance in low-resource scenarios? RQ2b:What challenges arise when applying DAG in QE across diverse languages? In Chapter 3, we explore whether QE models, like MT systems, would benefit from being tailored to specific domains. Moreover, we introduce a QE methodology that combines DA and DAG, and show how it can make QE systems more robustâespecially in multilingual and domain-diverse scenarios, including zero-shot learning (ZSL) and cross-lingual settings. 4 The specific type of similarity is discussed in the relevant chapter. 17 RQ3What is the optimal combination of SW tokenization model source and vocabulary source to maximize translation quality and effi- ciency during fine-tuning? RQ3a: How do different sources for training the SW tokeniza- tion model affect translation quality when fine-tuning out- of-domain models on in-domain data? RQ3b: How do the choice of sources for the SW model and vocabulary impact the trade-off between computational efficiency (e.g., training time, resource consumption) and model accuracy in DA? In Chapter 4, we investigate how the choice of data sources for training the vocabulary and SW tokenization model affects the fine- tuning of MT models for in-domain translation, starting from a model trained on out-of-domain data. We investigate which sources are most effective for learning the tokenization model and extracting vocabularies, and how these choices impact both translation quality and fine-tuning efficiency. RQ4How effective are domain-specific QE models in determining effective ICEs for improving MT quality in generative LLMs? RQ4a: What criteria should be used to select effective ICEs for improving MT quality? RQ4b: How does the integration of QE affect the success of ICE selection? In Chapter 5, we propose an ICL methodology that leverages domain- specific QE to guide the selection of ICEs for LLMs. The goal is to identify empirically effective numbers and combinations of ICEs that improve translation quality without relying on reference translations. 18 1.4 Dissertation overview This section outlines the motivations, significance, and main contri- butions of this dissertation, while highlighting how each contribution connects to the chapters that follow. Building on the RQs introduced earlier, we begin by examining the importance of in-domain data and its influence on both MT and QE, with a focus on the challenges posed by domain mismatch. We then turn to the role of SW tokenization and vocabulary design in enabling effective DA for MT. Finally, we discuss ICL for MT, emphasizing how the selection of ICL can significantly impact translation quality. 1.4.1 Why in-domain data matters As the volume of data to develop MT grows, it becomes clear that some data are more relevant to specific domains than others. This highlights the need for adaptation of models tailored to specific use cases, such as medical or patent translations. This requirement extends to nearly all natural language processing (NLP) downstream tasks, including MT and QE. Generic models, including LLMs, often struggle in these contexts, failing to handle domain-specific style, terminology, and levels of formality. Even when fine-tuned, LLMs require large amounts of domain-specific data to match the performance of smaller, dedicated translation systems(Eschbach-Dymanus et al., 2024). Training MT models with datasets that cover domains similar to the target domain can effectively yield high-quality results with practical usefulness in domain-specific applications(R. Wang et al., 2017; Pourmostafa Roshan Sharami, Shterionov, & Spronck, 2021). This highlights the limitations of data-driven MT when relying on general-domain data, regardless of its size (as demonstrated in Chapter 2), and emphasizes the importance of using domain-specific datasets for building effective MT systems. However, acquiring sufficient authentic domain-specific data with 19 human annotators, though ideal from a quality and reliability perspec- tive, is labor-intensive and costly. Human-annotated data ensures high accuracy, domain expertise, and adherence to stylistic and terminologi- cal conventions, making it the gold standard for training domain-specific MT models. However, due to its high cost and limited availability, re- searchers have explored alternative approaches for generating synthetic or pseudo-parallel datasets through data-centric approaches, primarily selecting in-domain data from generic parallel corpora. Chapter 2 of this dissertation presents a methodology for selecting in-domain data for MT through a data-centric method. After pre-processing steps to control computational costs, we identify and select examples with sentences similar to the target domain. This approach aims to operate with minimal computational costs and data capacities, thereby reducing the overhead associated with the data selection process. Our experiments highlight the trade-off between translation perfor- mance and efficiency, emphasizing the value of smart data selection in low-resource, domain-specific settings. Additionally, we have developed a Python tool that simplifies the data selection process through easy command-line operation and customizable configurations, making it more accessible to the community. This tool, which supports the data selection methods proposed in our work, is discussed in Section 2.7 of Chapter 2 and can be found athttps://github.com/JoyeBright/ domain-adapt-mt. Beyond MT, QE plays a crucial role in evaluating MT output without requiring human references. QE models predict the quality of translations, offering insights into whether a machine-translated sentence requires post-editing or is suitable for direct use. Unlike MT, which focuses on generating translations, QE aims to assess their reliability, making it a valuable tool for automated quality control in real-world applications. However, QE introduces additional complexities, particularly in 20 domain-specific scenarios, where training data scarcity and domain mismatches pose significant challenges. QE systems require post-edited text alongside source and machine-translated text for training, but such datasets are scarce. Compared to the abundance of MT datasets, publicly available QE datasets are significantly fewer, making the de- velopment of reliable QE models more challenging. While both MT and QE face challenges with generalizability, QE is significantly more affected. QE models are highly sensitive to domain shifts, leading to unreliable predictions when trained on data that does not match the MT systemâs domain. Therefore, ensuring accurate QE across different domainsâboth general and specializedâremains a critical yet challeng- ing task, further exacerbated by the scarcity of high-quality training data. In response to this challenge, Chapter 3 is guided by the view that QE models, like MT models, require DA to accurately assess the quality of MT outputs. In this context, the comparison implies that just as MT models depend on DA to improve translation quality, QE models similarly require DA to enhance their ability to assess MT outputs effectively. This idea is particularly important because MT practitioners and translation experts depend on the precision of QE models. Domain mismatch issues can lead to inaccurate quality assessments, potentially increasing costs for essential tasks such as dissemination. To investigate this perspective, we first demonstrate the necessity of DA. Building on this, Chapter 3 introduces a QE methodology that integrates both DA and DAG, supported by a novel QE training pipeline. We evaluated this methodology using several publicly available language pairs, including English to German, Chinese, Italian, Czech, Japanese, Romanian, and Russian to English and showed significant improvements across all language pairs under consideration, indicating that our proposed solution has a beneficial impact in addressing the aforementioned challenges. 21 The chapter also investigates how the proposed approach performs in zero-shot (ZS) and cross-lingual settings, where no training data is available for certain language pairs or domains. The analysis aims to evaluate how our approach affects QE model performance in situations where there is no prior training data for specific language pairs or domains, and to identify the factors that contribute to its effectiveness or limitations. This is especially important given the significant lack of QE data across many languages and domains, which makes robust, adaptable solutions all the more necessary. 1.4.2 SW choices and vocabulary in domain adaptation In Chapter 3 of this dissertation, we present experiments showing that pre-trained language models, when used without any adaptation (i.e., ZS inference), perform poorly. To improve this, the standard practice is to fine-tune the models. This involves training a generic model initially and then refining it with a domain-specific dataset different from the original training data, tailored to the specific downstream task, such as MT in our case(Chu & Wang, 2018; Saunders, 2022). Despite being more resource-intensive compared to parameter-efficient fine-tuning (PEFT) techniques like adapter-based methods (Houlsby et al., 2019), fine-tuning generally achieves better performance, particularly with smaller, memory-efficient pre-trained LLMs. These models are easier to manage in terms of memory, as opposed to larger LLMs, which require significant memory just to load and update. Therefore, fine-tuning still remains a viable option for DA in MT. Introducing new domain-specific data during fine-tuning brings additional information, such as previously unseen words. However, this can also lead to suboptimal word tokenization if not handled properly, potentially degrading system performance(R. Lim, Heafield, Hoang, Briers, & Malony, 2018; Yeung, 2019; Sato, Sakuma, Yoshinaga, Toyoda, & Kitsuregawa, 2020). Therefore, identifying the optimal setup for 22 incorporating new data is vital for maintaining and enhancing MT performance. This includes ensuring that both the pre-trained modelâs existing knowledge and the new information are effectively covered. To address this, Chapter 4 investigates the impact of vocabulary and SW tokenization on fine-tuning MT models for in-domain transla- tion. Our research aims to provide insights into the ideal conditions for fine-tuning, with the goals of improving translation quality and reducing training time. In Chapter 4, we seek to answer the primary RQ: Given an MT model and a fine-tuning dataset, what is the optimal combination of SW model source and vocabulary? We systematically compare different configurations of SW tokenization models and vocab- ulary sources to identify which combinations yield the most effective fine-tuning of MT models for specific domains. Through this evaluation, we provide practical guidance for selecting and aligning SW tokenization and vocabulary when adapting MT models to new domains. 1.4.3 ICEs for LLMs in MT Recently, the increasing popularity of generative LLMs has introduced a new paradigm known as ICL. Unlike traditional fine-tuning methods, ICL typically generates output directly, without modifying the model parameters(Radford et al., 2019; Brown et al., 2020). This is achieved by providing the model with a few examples, known as ICEs, which prime the LLM to enhance its performance for the given task(Jiang, Xu, Araki, & Neubig, 2020). In our case, this task is MT. This ap- proach reduces the computational costs associated with fully fine-tuning generative LLMs, which typically have billions of parameters. However, as demonstrated by(Vilar et al., 2023), a key challenge with generative LLMs is that the quality of translation is directly proportional to the quality of ICEs, where quality refers to ICEs being relevant, clear, accurate, and domain-specific. Therefore, finding ICEs that are most relevant and domain-specific to the text to be translated is another 23 crucial challenge for improving MT performance. To tackle this issue, a key question arises: How can one determine if the selected ICEs are effective, i.e., relevant to the domain and context of the source text to be translated, and thus improve MT performance? Although this evaluation typically relies on translated references or human judgments, the former can introduce bias, and the latter is costly(Dinh & Niehues, 2023). In Chapter 5, we explore whether integrating QE could potentially mitigate reference bias and offer a better approach for selecting more relevant ICEs (compared to relying on development sets and other existing approaches(Sia & Duh, 2023; Agrawal, Zhou, Lewis, Zettlemoyer, & Ghazvininejad, 2023)), thereby improving translation quality and efficiency in LLM-based translation systems. Previous findings from Chapter 3 and(Sharami, Shterionov, Blain, et al., 2023) highlight that QE must be domain-specific to accurately es- timate MT quality across various domains. If a generic QE modelâone that is not tailored to specific domainsâis used to select ICEs, it may fail to capture domain-specific nuances. Building on this, Chapter 5 explores whether such models may lead to the selection of ineffective ICEs, ultimately leading to suboptimal translations. This highlights the importance of DA in QE for guiding effective ICE selection. To explore this question, in Chapter 5, we answer the following RQ: How effective are domain-specific QE models in guiding the selection of effective ICEs for translation tasks in LLMs? To answer this question, we propose a novel ICL methodology that leverages domain-specific QE to assist in the selection of ICEs, with the goal of identifying empirically effective (suboptimal) 5 numbers and combinations of ICEs that improve MT quality, all without reference translations. We also assessed the impact of using our proposed methodology in comparison to fine-tuning 5 Here, âsuboptimalâ means empirically effective ICE numbers and combinations identified under experimental constraints, without a claim of theoretical optimality. 24 a pre-trained multilingual MT model, namely mBART-50(Tang et al., 2020). By considering all computational factors, this comparison allows us to determine whether ICL presents a more advantageous approach compared to fine-tuning a pre-trained MT model. Our experiments highlight the value of ICL over fine-tuning, showing that our method not only achieves better translation performance but also reduces carbon emissions, making it a more sustainable and efficient alternative. 1.5 Evaluation metrics Throughout this dissertation, we evaluate translation models and QE systems using both traditional and modern metrics. Below is a brief overview of each: âąBLEU measures n-gram overlap between the MT output and a ref- erence. Scores range from 0 to 100; scores above 30 are considered reasonable, and above 40 strong. BLEU tends to favor surface- level similarity and may not fully capture adequacy(Papineni, Roukos, Ward, & Zhu, 2002b). âąCOMET is a neural metric trained on human ratings. Scores typically range from -1 to +1. A score above 0.3 is decent; 0.5+ is strong. COMET correlates well with human judgment(Rei et al., 2020). âąTER counts the edits needed to match a reference. Lower is better. TER below 50 is acceptable; below 40 is strong. TER penalizes word order and omission errors more explicitly(Snover, Dorr, Schwartz, Micciulla, & Makhoul, 2006). âą chrF2 is a character-level F-score. Scores range from 0 to 100. chrF2 above 60 is typical for good systems and is useful for morphologically rich languages (PopoviÄ, 2015). 25 Interpreting metric scores. While thresholds like BLEU >30 or COMET >0.3 are often used to indicate translation quality, their interpretation depends on dataset, language pair, references, and prepro- cessing(Zouhar, Chen, Lam, Moghe, & Haddow, 2024; Tomas, 2025). We therefore focus on relative comparisons with strong baselines and support findings with statistical significance tests. Both absolute scores and relative gains are reported and interpreted in context. 1.6 Computational considerations and efficiency Computational efficiency is a key consideration in modern MT research, where model training, fine-tuning, and inference often demand substan- tial GPU resources, energy, and time. Optimizing these computational costs not only reduces training overhead but also plays a vital role in ensuring the scalability and sustainability of MT systemsâespecially when deployed in real-world settings. In this dissertation, we systematically examine the computational demands of the experiments conducted throughout our work. We highlight the trade-offs associated with each approach in terms of efficiency, training time, inference speed, and environmental impact. Specifically: Chapter 2 quantifies the efficiency gains from data selection in reducing training costs; Chapter 3 examines the trade-offs of DA pipelines; Chapter 4 investigates the impact of SW tokenization on computational efficiency; and Chapter 5 evaluates the computational performance of ICL, including inference time, energy consumption, and sustainability. A more detailed, chapter-by-chapter breakdown of these computational trade-offsâincluding training times, energy usage, and carbon impactâis provided in the appendix. 26 Carbon footprint acknowledgment Throughout this research, we tried to reduce computational costs and develop alternative methods to improve efficiency. However, the inves- tigation, model training, and inference phases still required significant processing power and runtime. The total carbon footprint of the exper- iments is estimated at approximately 1.5 metric tons of CO 2 . To help mitigate this impact, I chose to support a certified refor- estation initiative through Trees for All. 6 A donation equivalent to the estimated emissions was made to their program, and a certificate confirming the offset is available. 7 1.7 List of publications This dissertation builds on and extends the following peer-reviewed publications: âąJavad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. Selecting Parallel In-Domain Sentences for Neural Machine Translation Using Monolingual Texts. Computational Linguistics in the Netherlands Journal, vol. 11, Dec. 2021, p. 213â230. âą Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. 2023. A Python Tool for Selecting Domain- Specific Data in Machine Translation. In Proceedings of the 1st Workshop on Open Community-Driven Machine Translation, p. 29â30, Tampere, Finland. European Association for Machine Translation. 6 Trees for All is a Dutch non-profit organization dedicated to sustainable forestry and climate compensation through certified tree-planting projects. 7 https://javad.pourmostafa.me/assets/files/C02Offset.pdf 27 âąJavad Pourmostafa Roshan Sharami, Dimitar Shterionov, FrĂ©dĂ©ric Blain, Eva Vanmassenhove, Mirella De Sisto, Chris Emmery, and Pieter Spronck. 2023. Tailoring Domain Adaptation for Machine Translation Quality Estimation. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, p. 9â20, Tampere, Finland. European Association for Machine Translation. âąJavad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. 2024. Guiding In-Context Learning of LLMs through Quality Estimation for Machine Translation. In Pro- ceedings of the 16th Biennial Conference of the Association for Machine Translation in the Americas, (Volume 1: Research Track), p. 88â101, Chicago, USA. Association for Machine Translation in the Americas. âą Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. Analysis of Vocabulary and Subword Tokenization Settings for Optimal Fine-tuning of MT: A Case Study of In- domain Translation. In Proceedings of the 15th International Conference on Recent Advances in Natural Language Processing, p. 970â979, Varna, Sep 8â10, 2025. I also contributed to several other publications during my PhD. Al- though they are not included in this dissertation, I list them below to provide additional context for my research trajectory. âą Industry Collaboration: Elena Murgolo, Javad Pourmostafa Roshan Sharami, Dimitar Shterionov. 2022. A Quality Estima- tion and Quality Evaluation Tool for the Translation Industry. In Proceedings of the 23rd Annual Conference of the European Asso- ciation for Machine Translation, p. 307â308, Ghent, Belgium. 28 This work was conducted in collaboration with Aglatech14 (Or- bital14), a language service provider. We developed and deployed a tailored machine translation and quality assessment tool for industry use. âą Supervision Output: â Ali Boluki, Javad Pourmostafa Roshan Sharami, Dimitar Shte- rionov. Evaluating the Effectiveness of Pre-trained Language Models in Predicting the Helpfulness of Online Product Re- views. In Intelligent Systems and Applications (IntelliSys 2023), Lecture Notes in Networks and Systems, vol. 825. Springer, Cham. â Adil Derrazi, Javad Pourmostafa Roshan Sharami. Integrating SAINT with Tree-Based Models. In Intelligent Systems and Applications (IntelliSys 2025), Lecture Notes in Networks and Systems, vol 1553. Springer, Cham. These papers resulted from supervising MSc thesis students within the Data Science and Society program at Tilburg University. 29 This chapter is based on the following published paper: Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. Selecting Parallel In-Domain Sentences for Neural Machine Translation Using Monolingual Texts. Computational Linguistics in the Netherlands Journal, vol. 11, Dec. 2021, p. 213-230. The Python tool developed from this research was also published: Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. 2023. A Python Tool for Selecting Domain-Specific Data in Machine Translation. In Proceedings of the 1st Workshop on Open Community-Driven Machine Translation, p. 29â30, Tampere, Finland. European Association for Machine Translation. Improvements have been made to the title, figures, and certain sections to align the content with the broader context of this dissertation. 32 2 Generating In-domain data for Machine Translation C ontinuously-growing data volumes lead to larger generic MT models. Specific use cases are usually left out since generic models tend to perform poorly in domain-specific cases. In this chapter, we address this gap with a method for selecting in-domain data from generic-domain (parallel text) corpora for the task of MT. This method provides a solution to the problem by efficiently extracting relevant data that aligns with the specific domain, ensuring better performance in specialized translation tasks than generic models. The proposed method ranks sentences in parallel general- domain data according to their cosine similarity with a monolingual domain-specific dataset. We then select the topKsentences with the highest similarity score, which can be used to train a new machine translation system tuned to the specific domain. There, we define a domain as a collection of data that shares a common provenanceâtexts originating from similar sources or contexts that exhibit consistent linguistic characteristics. Our experimental results show that models trained on this in-domain data outperform models trained on generic or a mixture of generic and in-domain data. Also, our method selects high-quality domain-specific training instances at low computational cost and data size. 34 2.1 Introduction A widely accepted belief among MT researchers and practitioners is that more training data is better. That is, the larger the training corpus is, the more robust and accurate the model can be. However, substantial amounts of parallel data are not available for all language pairs or domains of interest(Currey, Miceli Barone, & Heafield, 2017; van der Wees et al., 2017; Stergiadis, Kumar, Kovalev, & Levin, 2021). Furthermore, data-driven machine translation systemsâ performance depends not only on the quantity but also on the quality of available training data(Fadaee, Bisazza, & Monz, 2017). Despite the fact that more and more training data for MT are becoming accessible every day, only those that cover the same or similar domains of interest are commonly able to boost translation quality(R. Wang et al., 2017; Pourmostafa Roshan Sharami, Shterionov, & Spronck, 2021). Hence, for domain-specific use-cases, data-driven paradigms may perform poorly when trained on general-domain data, regardless of the size of the corpus. Training MT systems on large amounts of data in many cases uses substantial amounts of resources such as memory and time which is the undesired effect of striving to boost the performance of MT systems. As such, it is paramount to be able to train systems on high-quality domain-specific data. We, therefore, face a two-sided challenge: (i) what is high-quality, in-domain data and (i) what amounts of parallel, in-domain data are necessary to achieve state-of-the-art MT quality at low computational and data capacities. To address these challenges, the research community has made many efforts to improve MT performance through DA techniques. DA for MT has versatile definitions, but we primarily follow Chu and Wang (2018), who state that DA would be employing out-of-domain parallel corpora and in-domain monolingual corpora to improve in- domain translation. Among other definitions, Saunders (2021) defined 35 DA as any scheme that aims to level up translationâs performance from an existing system for a specific topic or genre language. Studies in this area are mainly divided into two categories, namely (i) data- centric and (i) model-centric(Chu & Wang, 2018). The data-centric category includes methods that operate at the corpus / data level by selecting, generating, joining, or weighting sentences or datasets for training purposes. This category selects or generates the domain-related sentences from the general domain using existing in-domain / domain- specific data. In contrast, in the model-centric category, studies mostly fall into the areas aiming to alter the usual function of models. This is usually fulfilled through mixing, fine-tuning and reordering models or weighting objective functions, or applying regularization techniques. Our proposed methodology falls into a data-centric category and is specifically considered a data selection method. However, a few previous studies have investigated the generation of parallel in-domain sentences for MT. Their limitations motivate us to expand this area of research by proposing a novel data selection algorithm for collecting in-domain training data. In this regard, we aim to improve in-domain translation in low-resource scenarios by selecting in-domain sentences from out- of-domain corpora, then possibly employing DA for NMT leveraging both out-of-domain parallel corpora and in-domain monolingual data. In essence, our proposed approach leads to one main contribution: a language-agnostic data selection method for generating a parallel in- domain corpus using monolingual (untranslated) domain-specific corpora. Monolingual corpora are often abundant and can easily be collected without requiring any translations or further sentence alignments. This has two consequences. The first one is related to the proportion of high-quality data to the number of in-domain sentences. In particular, our method generates fewer but of higher quality sentences with the same or at least competitive performance on NMT systems. The higher quality is quantified by the efficiency of using fewer sentences that are 36 more impactful, allowing the model to perform just as well, if not better. The second one is the reduction of training time. This is a consequence of the fact that less data is used for training an NMT system. In a nutshell, we started with two corpora: a parallel general-domain ENâFR corpus and a monolingual in-domain corpus (either EN or FR). The general-domain corpus provides broad sentence diversity, which helps increase the number of in-domain data. Meanwhile, the in-domain corpus ensures the inclusion of domain-specific sentences. Both corpora were converted into word embeddings for analysis. To compare the corpora, we measured the similarity between in-domain and general- domain sentence embeddings using cosine similarity. Based on this measure, we ranked the general-domain sentences in descending order of similarity to the in-domain corpus, creating an in-domain parallel corpus. The resulting ranked sentences were also mixed to augment the training data. Each dataset was then used to train MT systems, and the best translations determined the quality of the in-domain parallel corpus. An overview of this methodology is presented in Figure 2.1. The chapter is organized as follows. We first cover the related work and define specialized terminology in Section 2.2. In Section 2.3, our data selection strategy is presented. Next, Section 2.4 covers the evaluation of our approach, detailing the datasets, system specifications, baselines, and results. Section 2.5 presents a discussion of our analysis. Section 2.6 concludes the work. 2.2 Related work There is a significant volume of research in DA for MT paradigms. However, to the best of our knowledge, few prior studies have been conducted particularly on selecting in-domain sentences efficiently and then exploiting them to improve the in-domain translation. We reviewed data-centric and, more specifically, data-selection papers that were 37 Figure 2.1: An overview of the proposed methodology. The notation 1..nin the figure indicates that the algorithm can select between 1 andnsentences, wheren is an arbitrary number. For example, ifn= 5, the algorithm selects up to 5 parallel instances from the out-of-domain corpus for each in-domain sentence. closely related to our research. M.-T. Luong and Manning (2015) did major work in this area when they adapted an existing English- German deep Long Short-Term Memory (LSTM) model by training it for additional 12 epochs on new domain data in the same languages; the original training data is general-domain, while the one used for adaptation is from the conversational domain. This DA approach led to an increase by 3.8 BLEU points compared to the original model (25.6 to 29.4) without further training. Similarly, Zoph, Yuret, May, and Knight (2016) proposed a transfer learning method for low-resource language pairs. Among previous works, the research presented in(R. Wang et al., 2017) is very similar to our work in terms of the intuition behind the selection methodology. Their method selects in-domain data based on similarity scores computed over embeddings drawn from an NMT system trained on in and out of domain data. This approach has 38 several limitations related to the fact that it relies on a particular NMT system that needs to be trained on both in- and out-of-domain data. That is, the complexity of their approach makes it rather difficult to employ in practice. Furthermore, as it relies on the embeddings of a particular NMT, it implies that a (language-specific) NMT system needs to be available or trained which may add computational or economic overhead. Axelrod, He, and Gao (2011) proposed a DA approach using data selection which is a common baseline for many more recent works. This is mainly because their work for the first time introduced the concept of DA in MT. They selected and ranked sentences with three cross- entropy based methods for the task of SMT. They also showed that all three methods presented in their paper outperformed the general- domain model. In the same direction, Chen, Kuhn, Foster, Cherry, and Huang (2016) presented another data selection technique employing Semi-Supervised Convolutional Neural Networks based on bitokens (Bi-SSCNN). The method they proposed only requires a small amount of in-domain data to train the selection model. Suggested methods were tested on two translation tasks (Chinese-to-English and Arabic-to- English) and showed that the Bi-SSCNN is more efficient than other approaches in terms of averting noisy sentence pairs. We compare our approach to the aforementioned models (among others). In Section 2.4, we outline those models and present further details and comparisons. With respect to data selection studies, van der Wees et al. (2017) also investigated a method, called dynamic data selection, to discern whether it is feasible to improve NMT performance. Their method sifts through all training data between training epochs subsequently and reduced the training data size by selecting sentence pairs most relevant to the translation task. By doing so, unlike fixed training data, the training becomes a gradual fine-tuning process, which iterates over different training subsets made. Chu, Dabre, and Kurohashi (2017) 39 proposed a novel DA method, called mixed fine-tuning, incorporating fine-tuning into multi-domain(Sennrich, Haddow, & Birch, 2016a; Kobus, Crego, & Senellart, 2017) for NMT. In the context of the corpora they experimented with, their fine-tuning method on a mix of in-domain and out-of-domain solves the problem of overfitting. It is also worth mentioning the work of Aharoni and Goldberg (2020) which proposes two methods for data selection based on unsupervised language models. The first one computes the centroid of the in-domain data and selects the samples that are nearest to the centroid, according to their cosine similarity. The second method is based on a binary classifier trained on in-domain sentences and a random negative sample (i.e., general-domain) of sentences. They evaluate the proposed approach on 5 domains and compare it to the work of Moore and Lewis (2010). Among other findings, they show that data selection is a valuable technique. Furthermore, they show that in multi-domain scenarios, more data does not necessarily lead to better performance, emphasizing the importance of data quality over quantity. 2.3 Data selection method Our method ranks sentences in a general-domain (or out-of-domain) dataset according to their similarity with an in-domain dataset. This in- domain dataset is monolingual; if a parallel corpus is provided only the source or target side is to be used. Once the sentences are ranked, we can then extract the top K sentences with the highest score, i.e., ranked the highest and use those for training a new MT system. According to our architecture, initially the input data, both in-domain and out-of-domain is first converted into embedding vectors. Our method then computes the similarity between these vectors and uses the similarity score for ranking and consecutively, for selection. The embedding space we use, Sentence BERT(Reimers & Gurevych, 2019), is of high dimensionality. 40 As such, these vectors become quite large to be processed effectively. To address this, we apply Principal Component Analysis (PCA) to reduce the dimensionality, enabling more efficient semantic search, ranking, and selection. 2.3.1 Sentence embedding and dimensionality reduction Word representation is a rich resource for gaining information for down- stream tasks such as classification, entailment, translation, etc. In the field of NLP, including MT, models are highly dependent on the quality and efficacy of input representations(Naseem, Razzak, Khan, & Prasad, 2021). Models greatly depend on how the input is represented because it influences their ability to capture semantic meaning and context ef- fectively, which directly impacts their performance in tasks such as MT and classification. While the myriad of categorical and fixed methods such as Bag-of-Words (BOW), Continuous BOW model, Skip-Gram Model(Mikolov, Chen, Corrado, & Dean, 2013), FastText(Bojanowski, Grave, Joulin, & Mikolov, 2017) have been employed in this regard, nowadays researchers tend to benefit more from unsupervised contextual word representation architectures and, in particular, transformer-based language models(Vaswani et al., 2017b). The main reasons are that (i) these models keep the full context of the input and (i) they reduce the computational time for most NLP tasks. These directly align with our research objectives: to be able to select semantically similar in-domain sentences and to reduce MT training time as well as similarity com- putational time. For example, these models provide representations with consistent dimensions, making comparisons straightforward, and reorient words or sentences within the context provided by the data, such as the same domain. Hence, efficient sentence representations play a major role in our work. There exist several transformer-based language models, such as BERT(Devlin, Chang, Lee, & Toutanova, 2019) and RoBERTa(Y. Liu 41 et al., 2019), that have set a state-of-the-art baseline(Cer, Diab, Agirre, Lopez-Gazpio, & Specia, 2017). These models generate representations at the word or token level. However, for tasks like Semantic Textual Similarity (STS), Sentence-BERT (SBERT)(Reimers & Gurevych, 2019) recently showed a better performance for sentence-level represen- tations. SBERT is a modification of the pre-trained BERT network that employs Siamese(Bromley, Guyon, LeCun, SĂ€ckinger, & Shah, 1993) and triplet network structures(Schroff, Kalenichenko, & Philbin, 2015), capable of capturing meaning-related relationships, which allows assessing the degree to which two sentences are semantically similar at reduced computational costs. We used sentence-level information in our research to focus on contextual relevance. It is important to note that SBERT is only used in our research for embedding sentences, not as input for MT. By default, the SBERT base model embeddings output has 768 dimensions. Sentences encoded using 768-dimensional vectors require a substantial amount of memory to be stored. For example, an out-of- domain corpus containing 31 million sentences generates a 31M Ă768 embedding matrix which is computationally expensive. The size of these large vectors does not only require substantial physical memory but also considerably increases the time for computing the semantic similarity (i.e., the STS task), where we need to cautiously mitigate the cost of computing semantic search between in-domain and out-of- domain sentences. To speed up the search process, embedding vectors should be loaded into GPU memory to leverage parallel processing. However, large embeddings like ours can quickly max out GPU memory, leading to issues like memory swapping or batching. These workarounds increase processing time and ultimately negate the speed benefits of using GPUs, creating a major challenge for large-scale semantic search. In order to mitigate these issues, we decided to work with smaller sized vectors. However, reducing the vector dimensions must be done 42 in a conscious way, in order not to lose important information. To do so, we employ PCA(Jolliffe, 2011) as a pooling method into the last embedding neural layers. We select the 32 principal components as our output features. That is, PCA acts as the final layer of our selection network which allows us to reduce the 768-dimensional vectors to 32-dimensional ones. In general, PCA components are equivalent to the output features and are easy to append to any SBERT pre-trained model. We selected and shuffled 500K sentences from the experimental datasets to train the PCA model, assuming they were representative of the entire corpus while keeping the process computationally efficient. A pre-trained model called âstsb-xlm-r-multilingualâ 1 (trained on over 50 languages) was selected, and its embeddings were reduced to 32 dimensions using PCA. This dimensionality reduction retains the most important features of the embeddings while significantly improving computational efficiency. Finally, the reduced model was saved for use in sentence embedding tasks. 2.3.2 Semantic search and ranking in-domain data Once the input data was reduced and embedded, we employed semantic search to identify general-domain sentences that are similar to in-domain data. This idea is mainly inspired by(Moore & Lewis, 2010; Axelrod et al., 2011; Duh, Neubig, Sudoh, & Tsukada, 2013; R. Wang et al., 2017). Specifically, we treat in-domain sentences as search queries and out-of-domain sentences as document entries in the semantic search process. In this context, the queries are used to retrieve the most relevant embedding vectors from the entries, effectively matching in- domain sentences to their closest counterparts in the out-of-domain corpus. 1 https://huggingface.co/sentence-transformers/stsb-xlm-r -multilingual 43 The proximity search can be fulfilled by various distance similar- ity measures methods, such as cosine similarity, Manhattan distance, Euclidean distance, etc. In our research, we used cosine similarity 2 . However, we could not apply such an exhaustive search (each in-domain sentence compared to each out-of-domain sentence) on our datasets using CPU as was expensive, specifically in terms of computational time. To resolve this issue, we used a GPU implementation of cosine similarity in the PyTorch library. Once the cosine similarity scores were computed over embedding vectors, the torch.topk 3 is called. By calling this function the n largest / most similar elements, as well as indices of the given in-domain or out-of-domain embedded tensors, are returned. Below we provide a detailed explanation of this process. LetS,E,qandddenote in-domain corpus, out-of-domain corpus, a vectorized search query and an embedded document entry, respectively, whereq â Sandd â E. Letkandlbe the number of sentences in the given corpora. That is, S =q 1 ,q 2 , ...,q k , E =d 1 ,d 2 , ...,d l and k âȘ l. According to these definitions, the cosine similarity is defined by Equation 2.1 in our data selection method, where 32 is the number of dimensions. cos(âq, â d) = âq· â d |âq|| â d| = âq |âq| · â d | â d| = P 32 i=1 q i d i q P 32 i=1 q i 2 q P 32 i=1 d i 2 (2.1) Based on the similarity measurement defined in Equation 2.1, we rank our sentences and pick the topn(n= 6 in our experiments) 4 out-of-domain sentences, which are sorted in descending order, to build 2 We chose cosine similarity because, unlike Euclidean or Manhattan distance, it normalizes vectors and compares only their direction. Since semantic meaning in embeddings is represented by direction rather than magnitude, cosine similarity provides a more reliable measure of contextual similarity. 3 https://pytorch.org/docs/stable/generated/torch.topk.html 4 Set empirically; tests withn â 1,2,3,4,5,6showed that translation quality and similarity to in-domain data converged at six. 44 pseudo in-domain sub-corpora. Considering that sentences are chosen from an out-of-domain corpus and are distinct bitexts, no further operation is required before feeding those into an NMT system. In summary, our data selection method has four major steps: âą Step 1, the input data is converted into vectors using the embed- ding unit. âąStep 2, the vectorsâ dimensions are reduced to the lower dimen- sions. âąStep 3, we compute the similarity scores between each in-domain and out-of-domain vector. âąStep 4, vectors pairs are ranked according to their similarity score achieved from step 3. 2.4 Experiments To test the quality of our data selection performance to generate parallel in-domain data, we conducted experiments with English-French data and compared our results to different systemsâ results. These systems are divided mainly into two categories: 1. Category 1: Models trained exclusively on the original in-domain data, without incorporating the data selected by our method. These systems were used to analyze how effectively our data selection algorithm improves translation performance compared to models trained on only the given in-domain data. It is important to note that these systems are not considered baselines but serve as a comparison point to assess the added value of our approach. 2.Category 2: For the second category, we chose previous re- searchersâ MT systems that were trained using their selected data. 45 These systems after being trained on the selected in-domain data are usually re-trained on the original / given in-domain data to increase the translation performance. Although most systems in this category used DA / re-training in their work, we considered them as actual baselines. This is, first, because there is not much previous work that only uses in-domain parallel data to train MT systems; and second, because we aim to evaluate the quality of our generated data to see if it helps the models to improve the translation quality without re-training, i.e., as a stand-alone corpus. 2.4.1 Data In- and out-of-domain datasets. The data we experimented with is a collection of TED talks, referred to as the IWSLT 5 2014 corpus (Cettolo, Niehues, StĂŒker, Bentivogli, & Federico, 2014). To validate our models during training time and test the modelsâ performance, we used one development set (dev2010) and two test sets (test2010 and test2011), respectively. In addition to an in-domain corpus, we combined a collection of WMT corpora 6 including Common Crawl, Europarlv7, News Commentary v10 and United Nations (UN)(Koehn, 2005; Tiedemann, 2012) to create a large out-of-domain corpus. IWSLT 2014 and WMT are commonly used in the context of DA as an in-domain dataset(Axelrod et al., 2011; M.-T. Luong & Manning, 2015; Chen et al., 2016; R. Wang et al., 2017), which allows for better applicability of our experiments. Data statistics are shown in Table 2.1. Selected data. We used our method (see Section 2.3.2) and the monolingual in-domain dataset, i.e., IWSLT, to extract the most similar subsets of in-domain data from out-of-domain data, i.e., WMT. Since 5 International Workshop on Spoken Language Translation 6 http://statmt.org/wmt15/translation-task.html 46 EN-FRNameSentences TED training (in-domain) IWSLT 2014 179K TED dev2010887 TED test20101664 TED test2011818 WMT training (out-of-domain) Common Crawl3.25M Europarl v72M News Commentary v 10 200K United Nations (UN) 25.8M Table 2.1: Overview of EN-FR datasets used for training and evaluation. The in-domain data (IWSLT 2014) includes TED talks for training, dev2010, test2010, and test2011. The out-of-domain data (WMT) consists of Common Crawl, Europarl v7, News Commentary v10, and UN datasets. The table presents the number of sentences for each dataset and split. The total number of out-of-domain samples is approximately 31 million. the data we extract is not specifically compiled as (authentic) in-domain data, but rather automatically generated based on similarities, it is referred to as pseudo in-domain data(Zhang & Xiong, 2018). In this chapter, to distinguish between authentic and pseudo in-domain data, we refer to the latter as selected in-domain data as this reflects the origin of the data. For our experiments, we created six (sub-)corpora based on the sentence ranks determined by the data selection method. To do so, (i) we choose one in-domain sentence and compute its similarity score with every single out-of-domain data point, i.e., one-to-many. So, for each in-domain sentence, we obtain a score list with the size of out-of-domain data, i.e., 31M. (i) Afterward, the out-of-domain sentences are sorted in descending order according to the similarity scores achieved in step (i); (i) We only select first n (n= 6) out-of-domain sentences from the 47 list generated in step (i). We repeat all aforementioned steps for every single in-domain entry, i.e., 179K. This procedure outputs a 179kĂ6 matrix according to our input data. Figure 2.2 shows one iteration of data selection. In practice, the selected sentences (top1âtop6) come from whichever parts of the out-of-domain corpora happen to be most similar to the in-domain sentences in SBERT embedding space. This similarity is semanticâprimarily topical and contextual. Note on data mixing. All top-ksub-corpora (top1âtop6) contain only selected in-domain data (i.e., automatically selected sentences). Mixing is done only among these selected sub-corpora (e.g., top1+top2), and never with the original IWSLT in-domain corpus. Figure 2.2: An iteration of selecting in-domain data Table 2.2 shows an example of possible outputs for the proposed data selection algorithm given a monolingual in-domain query, where 48 generated sentences were sorted from the highest score (top1) to the lowest one (top6). Score (/100) Monolingual in-domain (q i ): It can be a very complicated thing, the ocean.- Top1âparallel in-domain (top i0 ): EN: Ocean affairs are sensitive and complex. FR: Les affaires maritimes sont dĂ©licates et complexes. 90.10 Top2âparallel in-domain (top i1 ): EN: This is a dangerous position to be in if the sea is running high. FR: Ainsi, le capitaine peut prendre les effets du capitaine du navire pris, le chirurgien ... 86.80 Top3âparallel in-domain (top i2 ): EN: Rip currents and undertow are common, dangerous conditions along ocean beaches. FR: DĂ©chirez les courants et les baĂŻnes sont des conditions communes et dangereuses le long des plages dâocĂ©an. 86.60 Top4âparallel in-domain (top i3 ): EN: Moving with the waves can be dangerous. FR: Il est dangereux de progresser avec la vague. 86.13 Top5âparallel in-domain (top i4 ): EN: Obstacles in the water are particularly dangerous when coupled with currents. FR: Les obstacles dans lâeau sont avant tout dangereux par rapport au courant. 85.96 Top6âparallel in-domain (top i5 ): EN: This problem affects not only small islands, but also large islands and countries with extensive coastlines. FR: Ce problĂšme concerne non seulement les petites Ăźles, mais aussi les grandes Ăźles .... 85.76 Table 2.2: Example of data selection output: Scores (/100) for different data subsets selected using monolingual. To show the effectiveness of our semantic search and ranking idea, we found the centroids of selected sub-corpora, then compared them to the in-domain test setsâ centroids. The centroid is a multidimensional vector, calculated as the average of all other vectors. That is, it is a vector around which other vectors are distributed. Figure 2.3 depicts a gradual drop of similarity score from top1 to top6 for both test sets. To increase the performance of in-domain translation in terms of diversity richness, each sub-corpus is combined with all preceding bitext data (like a stack). For instance, top6 comprises top5, top4, top3, top2, top1; top5 holds top4, top3, top2, top1 and so forth. In that way, that last corpus (i.e., top6) encompasses all in-domain sentences. 49 Figure 2.3: The difference of selected sub-corpora to in-domain test sets 2.4.2 NMT system description We used the OpenNMT-py 7 framework(Klein, Kim, Deng, Senellart, & Rush, 2017) for training our NMT models. We trained transformer models(Vaswani et al., 2017b) for a maximum of 200K steps; interme- diate models were saved and validated every 1000 steps until reached convergence unless being stuck in the early stopping condition (10 validation steps with no improvements). The parameter setup stated in Table 2.3 was used. Hyperparameters were determined through preliminary experiments based on convergence speed, validation performance, early stopping behavior, and generalization ability. While a full discussion is beyond the scope of this work, batch size and learning rate were particularly beneficial. We found that a batch size of 512 and the default OpenNMT learning rate yielded the best results. See Section 2.5.3 for details. To run all NMT systems effectively, we also set other hyperpa- rameters, as suggested by the OpenNMT-py community to simulate 7 https://opennmt.net/OpenNMT-py/ 50 word emb. dim.512 max seq. len.150 num. transf. layers6 learn. opt.Adam transf.-f size2048 opt. lr2 num. heads8 opt. b1 and b20.9, 0.998 batch size512 beam size6 Table 2.3: Hyperparameters used for training our NMT models. Abbre- viations: emb. dim. (embedding dimension), transf. layers (transformer layers), transf.-f (transformer feed-forward), max seq. len. (maximum sequence length), learn. opt. (learning optimizer), opt. lr (optimizer learning rate), opt. b1 and b2 (optimizer beta1 and beta2). Googleâs default setup(Vaswani et al., 2017b). The NMT training was distributed over three NVIDIA Tesla V100 GPUs. We encoded all data, including rare and unknown words as a sequence of subword units using Byte Pair Encoding (BPE)(Sennrich, Haddow, & Birch, 2016b). We built our vocabularies for both source and target languages separately. By doing so, our systems are not only capable of translating out-of-vocabulary tokens, but also rare in-vocabulary ones. The number of merge operations for BPE is 50,000 for all sub-corpora, i.e., For each selected sub-corpora, a separate BPE model was created. Vocabulary sizes of selected in-domain data are shown in Table 2.4. 2.4.3 Compared systems To show the effectiveness of the proposed data selection method in terms of the quality of generating parallel in-domain data, as men- tioned earlier, we compared our results with two disparate categories of NMT systems as follows. First, we used the available in-domain and out-of-domain data as well as their mixture to establish the following systems: (i) S1:IDâNMT trained on in-domain only; (i) S2:OODâ NMT trained on out-of-domain data; and (i) S3:ID+OODâNMT trained on the combination of in- and out-of-domain data. It is note- 51 Selected In-domain Datasets Vocabulary with BPE SourceTarget Top148,95649,281 Top2 + top149,89650,055 Top3 + top2 + ...50,29950,391 Top4 + top3 + ...50,59650,568 Top5 + top4 + ...50,75950,720 Top6 + top5 + ...50,87450,894 Table 2.4: Vocabulary sizes for the source (EN) and target (FR) after applying BPE across various combined in-domain datasets. worthy that here, we employed a bitext in-domain data to demonstrate our selection methodâs productivity, however, the proposed method uses a monolingual in-domain corpus in a real use case. System S1, S2 and S3 are not intended as baselines, but points of comparisons for different purposes: (i) system S1 resembles the best possible translation quality according to the given in-domain data; (i) system S2 is trained on a large corpus without including domain-relevant data, thus it is a generic-domain MT system; and (i) system S3 is trained on a mixture of a large generic-domain corpus (the same as for S2) and domain-relevant data and aims to test the impact of the in-domain data on the translation performance, resembling a domain- adapted model. Second, we compared our NMT systems (based on the proposed data selection method) to four reliable previous NMT DA methods, which we refer to as baselines. These are B4:Luong (M.-T. Luong & Manning, 2015), B5:Axelrod(Axelrod et al., 2011), B6:Chen(Chen et al., 2016), B7:Wang(R. Wang et al., 2017) as mentioned in Section 2.2. While R. Wang et al. (2017) proposed three different models, we only refer to the best one. It is noteworthy that the numbering of baselines starts from B4 to align with system conventions 52 (i.e., S1âS3); That is, B1âB3 do not exist. 2.4.4 Results and analysis The performance of our MT systems is reported on two test sets using case insensitive BLEU(Papineni, Roukos, Ward, & Zhu, 2002a), TER (Snover et al., 2006) and chrF2(PopoviÄ, 2015) metrics, as implemented in the sacreBLEU framework(Post, 2018). We also analyzed the results for any statistically significant differences (see Section 2.5.1). The vocabulary was built by employing the IWSLT train set and each selected in-domain corpus. Note that according to Table 2.5 data, S1 achieved the highest BLEU score among the compared systems on both test sets. We considered it as a best-case scenario (i.e., target performance). As such, we aim to generate parallel in-domain sentences that yield results as good as S1. Systems Number of Sentences NMT- Test Set 2010NMT- Test Set 2011 BLEUâ TERâ CHRF2â BLEUâ TERâ CHRF2â S1:ID179K31.956.657.038.349.761.0 S2:OOD31.0M25.866.153.030.759.347.0 S3:ID+OOD31.1M26.062.954.030.956.858.0 B4:Luong17.9M32.2N/AN/A35.0N/AN/A B5:Axelrod9.0M32.2N/AN/A35.5N/AN/A B6:Chen7.3M30.3N/AN/A33.8N/AN/A B7:Wang3.7-7.3M32.8N/AN/A36.5N/AN/A Top1179K21.869.850.025.664.053.0 Top2+top1358K26.763.454.031.357.157.0 Top3+top2+...537K29.160.456.034.353.960.0 Top4+top3+...716K30.759.557.035.652.661.0 Top5+top4+...895K30.959.157.036.7 51.562.0 Top6+top5+...1.0M31.3 58.358.036.550.962.0 Table 2.5: Evaluation scores for EN-FR NMT systems trained on different data domains: in-domain (S1, B4, B5, B6, B7 and Top1 .. Top6+top5+...), out-of-domain (S2), and a mixture of both (S3). Results are reported on NMT-Test Set 2010 and NMT-Test Set 2011, including BLEU, TER, and CHRF2 scores. 53 Even though system S2 employed an enormous corpus with almost 31M sentences, it could not perform well for in-domain translation. System S3 had a minor improvement (0.2 BLEU points) after mixing OOD with ID, yet less than S1 performance. These results suggest that increasing the amount of training data, even with systematic inclusion of in-domain translation, is not always sufficient for improving in-domain translation and may even lead to under-fitted model. Moreover, system S3 exhibited bias due to an over-reliance on out-of-domain data, causing it to prioritize translation patternsâsuch as word usage, expressions, and sentence structuresâfrom OOD data, which were less suitable for in-domain translation. The proposed sub-corpora (top1 to top6), whose mixture resulted in some data removal, were used to train the NMT systems. We began by training the first system using top1 (referred to as system top1). Next, a mixture of top1 and top2 was used to train the second system (system top2), and this process continued by progressively adding more sub- corpora to train subsequent systems (top3, top4, and so on). Initially, performance improved significantly, with BLEU score gains of +4.9 points on the 2010 test set and +5.7 points on the 2011 test set from top1 to top2. This growth trend persisted until it reached a saturation point at top5 and top6, after which performance either plateaued or slightly degraded. Based on these results, we selected six sub-corpora to maximize translation performance. Among the selected systems, top5 and top6 achieved the highest BLEU scores, with 36.7 on the 2011 test set and 31.3 on the 2010 test set, respectively. Alongside BLEU, we also evaluated our models with TER and chrF2. According to TER, system top1 achieved the highest score among all systems trained on mixed sub-corpora. This would imply that this system would require the most post-editing effort. The TER score for top2 mixed with top1 dropped by 6.4 and 6.9 for test sets 2010 and 2011, respectively. This improvement (TER decrease) per each 54 mixing operation continued until the system top6+top5+... but with a smaller amount of drop. According to the chrF2 metric, there was an increasing trend from top1 to top6+top5+... for test set 2010, where the last sub-corpus obtained 58.0 scores, while for test set 2011, two last systems (top5+top4+... and top6+top5+...) achieved the same chrF2 score (62.0). The three metrics have consistently improving trends, which sup- ports the observation that, while more data implies an increase in quality, there is a point after which no (significant) improvements are possible. 8 Although systems B4, B5, B6 and B7 were used inherently for DA of NMT systems (fine-tuned on a large corpus), top5 outperformed the best of them (B7) for test set 2011 without retraining on such enormous generic corpus. That is, our proposed data selection method worked well and consequently generated better quality data. In addition to that, our generated corpora are relatively smaller than their proposed methods. Our largest sub-corpora has 1M sentences, whereas B7 proposed three corpora which the smallest is three times larger than top6. Practically speaking, applying DA as well as using a large in-domain corpus in the work of R. Wang et al. (2017) caused translation performance to be only 1.5 BLEU points higher than top6. That would be negligible as we did not benefit from any DA techniques. 2.5 Discussions In this section, first we discuss the statistical significance to evaluate the performance of MT models that trained on top1, top1+top2 and so on. Second, we investigate the training time of models trained using our proposed data and compare them with the training time of intended systems (S1, S2 and S3). Third, we show how different batch sizes 8 See Section 2.5.1 for a detailed statistical significance interpretation. 55 affect the performance of training models in our research. Finally, we investigate the impact of adding more data on the MT modelsâ quality, more specifically, to see how much our mixing idea helped the models to improve translation quality. 2.5.1 Statistical significance As mentioned earlier in Section 2.4.4, we also computed pairwise statis- tical significance of scores shown in Table 2.5 in terms of BLEU, TER and chrF2 scores by using bootstrap resampling and 95% confidence interval for both test sets based on 1000 iterations and samples of 100, 200 and 300 sentences. According to our experiment outputs, most results have a statistically significant difference except those system pairs listed in Table 2.6 for some number of samples. This evaluation highlights that as the training process progresses and more sub-corpora are added, the modelsâ outputs begin to converge, resulting in minimal or no distinguishable performance improvement. At this point of convergence, further mixing of sub-corpora (e.g., top6+top7, top7+top8, etc.) leads to a degradation in translation quality. This suggests that while incremental data initially boosts per- formance, there is a saturation threshold beyond which additional data not only fails to improve quality but may also introduce inconsistencies or noise that negatively impact the MT system. 2.5.2 Training time As can be seen in Table 2.7, the training of the large baseline model (S2) with almost 31M sentences not only took 2 days, 5 hours and 31 minutes but also did not perform well in the context of in-domain translation â it obtained 25.8 and 30.7 BLEU score for test set 2010 and 2011, respectively. That means enormous training data is not always helpful for training in-domain MT systems. 56 Test Set 2010 Test Set 2011 BLEU (Top4, Top5, 100) (Top5, Top6, 100) TER (Top4, Top5, 100) (Top4, Top5, 200) (Top4, Top5, 300) - CHRF2-- Table 2.6: System pairs with no statistically significant difference (p <0.05). The notation (TopX, TopY, N) indicates that systems TopX and TopY exhibit no statistically significant difference for a specific sample sizeN(100, 200, or 300 sentences). When it comes to chrF2 and TER (Test Set 2011), we note that all results are statistically significant. According to the results in Table 2.5, the best-performing models for in-domain translation are top5 and top6, which achieved the highest BLEU scores for test sets 2011 and 2010, respectively. Furthermore, their training times were significantly shorter than those of both S2 and S3. This is notable because, unlike our proposed MT modelsâtrained with data selected through our methodologyâS3 was trained using a mixture of parallel in-domain data, including the IWSLT in-domain corpus, whereas the proposed models were trained without mixing such data. The training times for top1 and other sub-corpora models without any data mixing, as shown in Table 2.8, were nearly identicalâaveraging around 1 hour and 20 minutes to find the optimal model. 2.5.3 Batch size effect In our experiments, we used the default hyperparameters from Vaswani et al. (2017b) to train the NMT models. However, the effects of different batch sizes on the modelsâ performance were investigated as well in order to determine the most suitable one. Our work is pertinent to data selection and the size of selected data has a close correlation with batch sizes. As such, we need to choose a batch size that is optimal for the 57 Systems Complete T Step Best model T Step S1: ID00:03:5318,00000:00:50 5,000 S2: OOD02:05:3193,00001:22:56 82,000 S3: ID+OOD01:13:3039,00000:10:00 29,000 Top100:04:4316,00000:01:23 5,000 Top2 + top100:04:4320,00000:02:13 10,000 Top3 + top2 + ...00:06:0626,00000:03:03 13,000 Top4 + top3 + ...00:06:2327,00000:03:53 17,000 Top5 + top4 + ...00:10:3335,00000:05:50 20,000 Top6 + top5 + ...00:08:2035,00000:04:26 19,000 Table 2.7: The training time (abbreviated as T) of generated sub-corpora and the first category of compared systems. The times are reported in days (D), hours (H), and minutes (M) for both the complete training duration and the duration until the best model performance was achieved, along with the corresponding step counts. models according to the training data (i.e., the selected sub-corpora). To this end, we tested five different batch sizes including, 64, 128, 512, 1024 and 2048 for training on top1+..+top6 data. Figure 2.4 depicts the accuracy and perplexity for the different batch sizes per step. These results show that the selected model with batch size 512 simul- taneously reached the highest possible accuracy percentage (61.89%) and the lowest perplexity score (7.08) within 21K steps. Following these experiments and the results shown in Figure 2.4 we decided to conduct all our experiments (see Section 2.4) with a batch size of 512. Further investigation is needed in order to define a correlation between (training) data size and batch size for optimal MT performance. We leave this for future work. 58 Figure 2.4: The effect of different batch sizes on training the proposed NMT models. Left and right Figures show validation accuracy and validation perplexity per step, respectively. 2.5.4 Mixing effect In our main experiments, we investigated the translation quality of systems trained on incremental datasets: top1 then top1+top2 and so on. In order to determine whether the quality achieved by the different systems is due to the quality of the data or the quantity (i.e., adding additional data), we trained 6 other models without combining (mixing) the datasets, i.e., a model trained only on top1, a model trained on top2 only and so on. Then we evaluated these models the same way as with the ones presented in Section 2.4. The results are shown in Table 2.8. According to the three evaluation metrics, all systems performances are on par with system top1 except for minor differences that are unavoidable. That is, according to our experiment (shown in Figure 2.3 in Section 2.4.1) the centroids of selected sub-corpora (top1, top2, etc.) are very similar to centroids of in-domain test sets. Recall that a centroid represents the average position of data points in high-dimensional space. This measure helps us compare how closely the sub-corpora align with 59 Systems Sentence Number NMT- Test Set 2010NMT- Test Set 2011 BLEUâ TERâ CHRF2â BLEUâ TERâ CHRF2â Top1179K21.869.850.025.664.053.0 Top2179K21.272.149.024.667.352.0 Top3179K21.971.349.025.266.152.0 Top4179K21.171.749.024.666.352.0 Top5179K20.872.049.024.667.351.0 Top6179K21.969.649.024.166.851.0 Table 2.8: Comparison of NMT system performance on in-domain data selection without mixing sub-corpora. one anotherâparticularly with the in-domain test sets. Notably, the centroid of top1 aligns even more closely with the centroids of the 2010 and 2011 test sets than others. To further interpret these results, we computed pairwise statistical significance of BLEU score using bootstrap resampling and 95% confidence interval. Table 2.9 indicates whether the differences between two systemsâ BLEU scores are statistically significant (Y) or not (N). It shows that there was no statistically significant difference between results obtained with systems trained on top2, top3, ..., top6. However, the results (of BLEU) obtained with top1 are statistically significant from all the rest. These are the sentences (from the OOD corpus) that are the most similar to the in-domain data. Table 2.9 shows the statistical significance computed over both test sets and based on 1000 iterations and samples of 200 sentences. Figure 2.5 also indicates the quality improvement (in percentage) of NMT systems that trained on the mixture data compared to the NMT trained on the original sub-corpora without being mixed. According to these figures, the mixing procedure enhanced the translation quality up to 49% and 51% for test sets 2010 and 2011, respectively. Overall, there is a gradual increase trend after mixing them with all preceding 60 Top1 Top2 Top3 Top4 Top5 Top6 Top1Y Top2N Top3N Top4N Top5N Top6 Table 2.9: Results of statistically significant test for in-domain data selection without mixing sub-corpora (for p < 0.05). sub-corpora until the convergence point. Our objectives are twofold: (i) to mix as few sub-corpora as possible in order to achieve maximum performance with fewer samples, which reduces training time and computational costs, and (i) to prevent MT models from becoming overly biased toward in-domain data. This helps preserve performance on generic domains and reduces the risk of overfitting. For example, top5 and top6 serve as the convergence points for test sets 2011 and 2010, respectively. We stop mixing sub-corpora once the convergence point is reached. For test set 2010, the improvement rate started at 26%, rose to 33%, then 45%, peaked at 49%, and eventually declined to 43%. For test set 2011, the improvement rate began at 27%, briefly dropped to 26%, then increased to 45%, 49%, and finally reached a peak of 51%. It is important to note that top1 showed no improvement since it was not mixed with any other sub-corpora. 2.6 Conclusion and future work In this chapter, we presented a method to help the MT community to mitigate a lack of parallel in-domain corpora for many language pairs. Considering multiple standards such as generating high-quality 61 Figure 2.5: Quality improvement (in percentage) after applying the mixing idea on NMT systems for test sets 2010 and 2011. data, designing a scalable architecture, having a reusable pipeline, we present a method for data selection, based on ranking, for the purposes of generating parallel in-domain data as well as DA. Given a large parallel corpus, the proposed method aims to select data that are semantically similar to a set of in-domain dependencies. Typically, this method would be employed when there is little in-domain or only monolingual data, however, our method is generic and not restricted to the size of the in-domain data. The proposed selection pipeline is made of three main components: (i) a contextual sentence embedding component; (i) a semantic search component and (i) a ranking in- domain data component. These have been wrapped and released in a publicly accessible Python tool. We conducted experiments with different sizes of selected in-domain data. Our experimental results showed that selected parallel corpora generated through our data selection method can be applied directly for domain-specific NMT systems. Models trained on the selected data outperformed baseline models and, in some cases, were comparable to or even better than fine-tuned models. We note that our experiments used data up to top-6 as beyond top-6 a larger portion of the selected data 62 becomes more dissimilar and would not contribute to the translation quality of the NMT systems. 9 However, for other data and domains this threshold could be different and should be determined on a case-by-case basis. As future work, building on the present study, we intend to employ our generated corpora in the context of DA by further training on a parallel in-domain corpus, which possibly boosts NMT systemsâ per- formance considerably. Furthermore, our proposed selection method shortens the training time and, at the same time, increases NMT trans- lation quality compared to employing an out-of-domain corpus. We would also like to address other important research questions. First, we would like to compare the hard decision onnâthe number of sentences that can be selected by our algorithmâto a similarity threshold (which is not linked to any specificn). Another research direction would be to investigate the effects of our method on other language pairs and domains. The selected data and trained models are available at:https:// github.com/JoyeBright/DataSelection-NMT. 2.7 Python tool Building on the methodology outlined in this chapter, we developed a Python tool to automate and streamline the process of selecting high- quality in-domain data for DA. This tool enables the MT community to easily extract relevant domain-specific data from large general-domain corpora, enhancing translation quality and reducing computational costs. It implements the core components of our approach, including contextual embedding generation, semantic search, ranking, and data selection. 9 This was determined through preliminary empirical evaluation and is not pre- sented or discussed here. 63 The toolâs operation requires three inputs: (i) a parallel generic corpus for the source language, (i) a parallel generic corpus for the target language (i) a monolingual domain-specific corpus for the source language. An optional parameter allows users to specify the number of selected sentences. Once these inputs are provided, the tool leverages a pre-trained Sentence-BERT (S-BERT) model(Reimers & Gurevych, 2019) to create sentence embeddings for the input corpora (i.e., i and i). To optimize computational efficiency, the tool applies PCA, reducing the original 768-dimensional embeddings to 32 dimensions. If the size of the corpus (i) is exceeded by the desired number of selected data, the generic corpora are split into multiple equal parts, and each of these parts is used separately in the subsequent step. In the next step, semantic search is performed to match general- domain sentences with the most similar domain-specific sentences based on cosine similarity. This is done by comparing the vectors of sentences, generated by S-BERT, and ranking them based on their cosine similarity score. The sentence with the highest similarity score is labeled as Top 1, while the one with the lowest similarity score is labeled as TopN. By default,Nis set to 5, but users can choose a different value forN. For each split, the tool then creates a CSV file that includes information about the domain-specific sentence (labeled asQuery), the top selected source and target sentences (labeled astopN src andtopN trg ), and their corresponding similarity scores. By concatenating the CSV columns generated, one can obtain as much data as previously requested. Our tool is particularly useful to the MT community as it addresses the scarcity of parallel domain-specific data across different language pairs. By using our tool, users can seamlessly select domain-specific data from generic corpora to train a domain-specific MT model. This tool is typically used when there is a lack of domain-specific data or when only monolingual data is available. However, our tool is generic and not limited to the size of the domain-specific data. 64 Our tool is licensed under the MIT License and is accessible to the public for free at https://github.com/JoyeBright/domain-adapt-mt. 65 This chapter is based on the following published paper: Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, FrĂ©dĂ©ric Blain, Eva Vanmassenhove, Mirella De Sisto, Chris Emmery, and Pieter Spronck. 2023. Tailoring Domain Adaptation for Machine Translation Quality Estimation. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, p. 9â20, Tampere, Finland. Improvements have been made to the title, figures, and certain sections to align the content with the broader context of this dissertation. 68 3 Domain Adaptation for Machine Translation Quality Estimation W hile QE can play an important role in the transla- tion workflow, its effectiveness relies on the availabil- ity and quality of the training data. For many QE use casesâincluding various domains and language pairsâaccess to high-quality labeled data remains a significant chal- lenge. This is largely due to the significant cost and effort required for manual data annotation, limiting the scalability and effectiveness of QE in these contexts. Unlike MT data, QE data requires the collection of source, MT, and a succinct human quality assessment metric, such as HTER scores. Aside from the data scarcity challenge, QE models should also be generalizable; i.e., they should be able to handle data from different domains, both generic and specific. Since few pre-trained QE models exist to provide cross-language generalization (unlike MT models), training on generic data is crucial for learning foundational patterns in translation quality. Fine-tuning with domain-specific data further enhances performance for specialized contexts. To alleviate these two main issues (data scarcity and domain mis- match), this chapter combines DA and DAG in a robust QE system. Our method first trains a generic QE model and then fine-tunes it on a specific domain while retaining generic knowledge. Our results show a significant improvement for all the language pairs investigated, better cross-lingual inference, and a superior performance in ZSL scenarios as compared to state-of-the-art baselines. 70 3.1 Introduction Predicting the quality of MT output is crucial in translation work- flows. Informing translation professionals about the quality of an MT system allows them to quickly assess the overall usefulness of the gen- erated translations and gauge the amount of post-editing that will be required(Tamchyna, 2021; Murgolo, Sharami, & Shterionov, 2022). QE is an approach that aims to reduce the human effort required to judge the quality of an MT system by assessing the quality of its output without the need for reference translations. 1 QE can be applied on word-, sentence- or document-levels. The goal of sentence-level QE, which is the focus of our work, is to predict a quality label based on a source sentence and its MT equivalent. This label, i.e., the quality estimate, can be expressed in various ways such as TER/HTER(Snover et al., 2006), BLEU(Papineni et al., 2002b) or any metric of interest to the user. Training a sentence-level QE system typically requires aligned data of the form: source sentence (SRC), target sentence (TRG), and quality gold label (LBL). However, most quality labels are by-products of MT and post-editingâa rather difficult and expensive processâlimiting the size of the available QE data (Rei et al., 2020; Zouhar et al., 2023). The WMT QE shared task (Specia et al., 2021; Zerva et al., 2022) has been a platform since 2012(Callison-Burch et al., 2012) to com- pare different QE systems and to share QE data. Despite efforts from initiatives like the QE shared task to publicly release QE datasets, such resources remain scarce across language pairs and, by extension, also have a limited coverage across domains(Fomicheva et al., 2020; Fomicheva et al., 2022). This can pose a challenge for all QE models, especially recent ones that utilize large pre-trained language models 1 In contrast to quality evaluation, which compares MT output to reference translations. 71 (PLMs)(Ranasinghe, Orasan, & Mitkov, 2020; Zerva et al., 2022) since fine-tuning pre-trained models with small datasets has been demon- strated to be quite unstable(Zhang, Wu, Katiyar, Weinberger, & Artzi, 2020; Rubino, 2020). 2 Furthermore, QE models trained on specific data do not generalize well to other domains that are outside of the training domain(Kocyigit, Lee, & Wijaya, 2022). In Chapter 2, we discussed domain mismatches in MT using DA techniques for in-domain data selection, which im- proved translation quality by better handling specialized vocabulary and context. In this chapter, we explore similar challenges in QE, where domain mismatches degrade performance due to limited domain-specific labeled data(de Souza et al., 2014a; Zouhar et al., 2023). While DA methods benefit MT, applying them to QE involves additional com- plexities like data scarcity and cross-domain generalization issues. To improve the generalizability of QE models, it is important to establish the right balance between domain-specific and generic training data. To date, only a few attempts have been made to address this challenge(de Souza, Turchi, & Negri, 2014b; Rubino, 2020; Lee, 2020), and as a result, many QE models still struggle to maintain accuracy across different domains, particularly when faced with limited domain-specific training data (Zouhar et al., 2023). We propose to tackle both the data scarcity and the domain mis- match challenge that PLM-based QE models face through a methodology whereby a small amount of domain-specific data is used to boost the overall QE prediction performance. This approach builds upon the DA techniques discussed in Chapter 2, where we addressed domain mis- matches in MT by selecting in-domain data to fine-tune large, generic models, thereby enhancing their ability to handle specialized vocabulary and context effectively. Similarly, in QE, we leverage domain-specific 2 Fine-tuning PLMs for QE has gained popularity and is becoming the new standard in the field. 72 data to improve prediction accuracy, ensuring models can generalize better across different domains(Chu & Wang, 2018; M.-Q. Pham, Crego, & Yvon, 2022). To assess the validity of the proposed approach in QE (See Sec- tion 3.2 for a detailed description of our methodology), we conducted experiments using small and large, authentic and synthetic data in bilin- gual, cross-lingual, and ZS settings. We experimented with publicly available language pairs from English (EN) into German (DE), Chinese (ZH), Italian (IT), Czech (CZ), and Japanese (JA) and from Romanian (RO) and Russian (RU) into English (EN). We used the common test sets from the WMT2021 QE shared tasks. 3 Our experiments show a statistically significant improvement in the performance of QE models. Our findings also indicate that not only our implementation leads to better multi-/cross-lingual QE models (where multi-/cross-lingual data is provided) but also ZS QE (where no data for the evaluated language pairs was provided at training). The main contributions of our research are summarized in this chapter as follows: âąA QE methodology that employs DA and DAG, along with a novel QE training pipeline that supports this methodology. âąAn empirical demonstration of the pipelineâs effectiveness, which highlights improvements in QE performance, and better cross-lingual inference. âąA comparative analysis with state-of-the-art (SOTA) baseline meth- ods that demonstrates the effectiveness of our approach in enhancing ZSL for the task of QE. âąAdaptable QE pipelines that can be tailored and implemented for other language pairs; i.e., highly generalizable QE pipelines. To the best of our knowledge, this is the first QE methodology to 3 https://w.statmt.org/wmt21/quality-estimation-task.html 73 use DA and DAG. Furthermore, it is easily reusable and adaptable: (i) while we used XLM-R in our experiments, one can easily replace it with any preferred PLM as long as the input-output criteria are met; (i) we built our tool around Hugging Face (HF) implementations of PLMs, meaning one can employ a certain generic model and apply it to any QE task by simply fine-tuning it on (newly-collected) QE data. 3.2 Domain adaptation for specialized quality estimation In this section, we outline our methodology for training PLM-based QE models for a specific domain with limited available in-domain data. This involves: (i) a set of training steps that we found to be particularly effective, and (i) DAG techniques to improve the QE modelsâ specificity. Also, we provide details on two different training modes we implemented. 3.2.1 Training steps We adapt the âmixed fine-tuning + fine-tuningâ DA technique that proved promising for MT(Chu et al., 2017) to suit our needs. A visualization of the steps involved can be found in Figure 3.1. Our technique involves leveraging both in-domain (ID) and out-of-domain (OOD) QE data (see Section 3.3.1 for details on the datasets). Step 1 We train a QE model using OOD data until it converges. We employ the experimental framework described in Section 3.3.2 in which a PLM is fine-tuned to predict QE labels. The goal of this step is two-fold: (i) leveraging the PLMâs cross-lingual reference capabilities and (i) building a generic QE model. This way we ensure that the model can estimate the quality of a broad range of systems, but with limited accuracy on ID data. 74 Step 2 The modelâs parameters are fine-tuned using a mix of OOD and ID data. We use different ID data, both authentic and synthetic according to the DAG approaches in Section 3.2.2. The objective here is to ensure the model does not forget generic-domain knowledge ac- quired during the first step while simultaneously improving its ability to perform QE on the domain-specific data. This mixing step is often referred to as oversampling in DA literature, where a smaller subset of OOD data is concatenated with ID data to allow the model to assign equal attention to both datasets; it aims to further adapt the model to the specific domain of interest. Step 3 We continue to train the QE model on a specific ID dataset until convergence, resulting in a more domain-specific QE model than the one obtained in Step 2. 3.2.2 Data augmentation for domain adaptation in QE In the previous chapter, we demonstrated that DA techniques sig- nificantly improve translation quality in MT by addressing domain mismatch through ID data selection. However, even with these meth- ods, a key challenge remains: the availability of domain-specific data is often limited, particularly for low-resource languages and special- ized domains. This scarcity of ID data can prevent both MT and QE models from generalizing effectively across different domains. In QE, this issue is compounded by the additional need for labeled data, such as post-edited translations, which are costly and time-consuming to produce. To overcome these challenges, we explore two alternative approaches to oversampling in this study. These approaches aim to optimize the utilization of existing ID resources and assess the potential benefits of incorporating synthetic ID data into the QE pipeline, thereby im- proving domain-specific accuracy without compromising cross-domain 75 OOD QE Dataset QE Framework checkpoint: a pre-trained LM OOD QE Model Step 1 + ID QE Dataset QE Framework checkpoint: OOD QE Model initialization Mixed FT QE Model QE Framework checkpoint: Mixed FT QE Model initialization ID QE Model Step 2 Step 3 Figure 3.1: Overview of the proposed training steps for specialized QE. The â+â sign indicates the oversampling performed in Step 2 to balance the use of ID and OOD data. The dashed arrows indicate the source of the checkpoint used to initialize the models in each stage. generalizability. Approach 1: Concatenating all available authentic ID data across all languages. The XLM-R model is multilingual, allowing us to apply it to different language pairs. When there is not enough data to fine-tune it for a specific language, one can use multilingual data. In our work, to increase the amount of authentic data (given the small volume of parallel data for two languages), we construct a multilingual ID dataset: we concatenate all available ID data, which includes different language pairs. The rationale behind this approach is to make use of all available authentic resources in order to improve the performance of the QE model by providing better cross-lingual 76 references. Approach 2: Generating synthetic ID data. Given that all available ID resources have been already utilized in Approach 1, we propose to supplement the existing data with artificially generated additional ID data using a trained MT model for each language pair, inspired by the research conducted by Negri et al.,(Negri et al., 2018) and Lee(Lee, 2020). This approach aims to tackle the data scarcity problem and further improve the QE modelâs accuracy. LetD lp denote the publicly available parallel data (SRC, TRG) for a language pair e.g., lp = EN-DE. Then for each ID involved, do: 1.Randomly selectNsamples fromD lp to obtain a setS lp of training samples. DivideS lp into two equal setsS 1 andS 2 . The division is random to ensure that both sets are representative of the data distribution. 2. Train a multilingual MT model M lp on S 1 . 3.UseM lp to translate the sources-side ofS 2 (or a portion of it), obtaining a set T lp of translated samples. 4.Compute quality labels (TER/HTER) by comparingT lp with the reference (TRG) text from S 2 . The result is a triplet (S 2 ,T lp , and TER/HTER) to be used to train QE models. A visual representation of these steps can be found in Figure 3.2. 3.2.3 Additional indication of domain In NMT, in order to handle multiple domains and reduce catastrophic forgetting, DA has been controlled using additional tags added at the beginning or at the end of the sentence(Sennrich et al., 2016a; Mattoni, Nagle, Collantes, & Shterionov, 2017; Chu & Dabre, 2019). Following these studies, we explore two training modes: (i) with tag (âTAGâ) 77 S lp S 1 S 2 Multilingual MT Framework train MT model M lp SRC 4 1 1 2 3 SRC SRC T lp translation SacreBLEU Compute TER TER TRG TRG TRG Figure 3.2: Overview of Approach 2 (Generating synthetic ID) of DAG for DA in QE. The various steps involved in the approach are indicated close to the corresponding arrows. Arrow 1 represents subsampling. The abbreviationsSRC, T RG, andT lp stand for source, target, and machine-translated text, respectively. The final outputs, which includeSRC,T lp , and quality labels (T ER), are color- coded for clarity. appending either an<OOD>or an<ID>tag at the end of sentences based on the dataset domain type (i.e., OOD or ID). The input format in this mode is<s> SRC </s> TRG <Tag> </s>, where SRC and TRG represent source and target of the QE triplet, and<s>and</s>are the beginning and separator tokens for the PLM used in the pipeline 4 . (i) without tag (âNO TAGâ), where the training steps are the same as detailed in Section 3.2.1, i.e., no additional info is attached. The methodology consists of three interconnected stages: training steps, DAG, and domain tagging. The training steps gradually adapt the model from OOD to ID data, improving domain-specific perfor- 4 The<s>and</s>tokens are special symbols typically used in the PLM to indicate the start and separation of input segments. These tokens help the model structure its input and output during training and inference. 78 mance. DAG enhances this process by increasing ID data availability through both authentic data concatenation and synthetic data gener- ation. Domain tagging helps the model differentiate between OOD and ID data, reducing domain confusion and improving cross-domain generalization. Jointly, these stages address data scarcity and domain mismatch, ensuring a more robust and adaptable QE model. We use the same domain tags for both original and synthetic ID data because the tags encode only the domain distinction (ID vs. OOD), not the data provenance. The synthetic samples generated in DAG 2 are intended to represent the same in-domain distribution as the original ID data; therefore, assigning different tags would incorrectly signal a domain difference where none exists. 3.3 Experiments 3.3.1 Data We conducted experiments on publicly available data in different lan- guages: from EN into DE, ZH, IT, CZ, and JA and from RO and RU into EN. These languages were chosen to cover diverse linguistic prop- erties, such as morphology (DE, CZ), syntax (JA), and writing systems (ZH, JA). This ensures robust evaluation across varied challenges in MT. We categorize the data into three groups according to their use in our pipeline: Group 1: for building ID and OOD QE models. The ID data is collected from WMT 2021 shared task on QE(Specia et al., 2021), Task 2, consisting of sentence-level post-editing efforts for four language pairs: EN-DE, EN-ZH, RU-EN and RO-EN. For each pair there are train, development (dev), and test sets of 7K, 1K, 1K samples, respectively. For our OOD data, we used the eSCAPE(Negri et al., 2018) dataset with approximately 3.4Mtokenized SRC, machine- 79 translated text (MTT), post-edited (PE) sentences. In our experiments, we used only the EN-IT portion of the eSCAPE corpus. We used sacrebleu 5 (Post, 2018) to calculate TER(Snover et al., 2006) from MTT and PE pairs. We split the data into train, dev, test sets via the scikit-learnpackage 6 (Pedregosa et al., 2011) with 98%, 1%, and 1% of the total data, respectively. To improve the generalization of our models and enable them to better adapt to specific QE through the ID dataset, we utilized a larger OOD dataset. This decision is in line with prior studies on DA, which are described in the related work section (Section 3.6). Group 2: for building MT systems as a component of Ap- proach 2 in the proposed DAG (Section 3.2.2). We collected parallel dataâSRC and reference translations (REF)â from Opus (Tiedemann, 2012) for each language pair used in ID (EN-DE, EN-ZH, RO-EN, and RU-EN), all sourced from ParaCrawl. Next, we trained MT models for Approach 2 of our methodology by selecting 4Msamples and dividing them into two equal parts, each with 2Msamples. We split either of the two parts into train, dev, and test sets. To save time during evaluation and inference, we set the size of the dev and test splits to be the same as the number of training samples in the ID datasets, which is 7K. The reason for reducing the number of samples to 7Kis to save time during evaluation and inference. Translating 20K samples takes about three times longer than translating 7K samples. Moreover, we randomly selected a portion of the SRC (7Kout of 2M) in the second split, which was not used for training. We passed this portion to the trained MT to get MTT. Finally, we computed the TER using the MTT and the corresponding REF viasacrebleu. We set the portion size 7Kas the goal was to double the size of the initial ID data. 5 signature:nrefs:1|case:lc|tok:tercom|punct:yes|version:2.3.1 6 random state/seed=8, shuffle=True, used for all splits. 80 Group 3: for testing the ZS capabilities of the trained QE models in our proposed methodology. We used two test sets with language pairs that were not seen during training, namely English to Czech (EN-CZ) and English to Japanese (EN-JA), which were provided by WMT 2021 shared task on QE for Task 2. Each test set contained 1K samples. A summary of the data groups used in the experiments is presented in Table 3.1. Data GroupDescription Group 1: ID and OOD QE models- ID data: WMT 2021 (EN-DE, EN-ZH, RU-EN, RO-EN), 7K/1K/1K samples - OOD data: eSCAPE (3.4M samples), EN-IT only - TER via sacrebleu, data split with scikit-learn Group 2: MT systems for DAG- Parallel data: Opus (ParaCrawl), 4M samples (EN-DE, EN-ZH, RO-EN, RU-EN) - Train/dev/test split: 7K samples - TER from machine-translated text (MTT) and reference (REF) Group 3: Zero-shot testing- Test sets: EN-CZ, EN-JA (1K samples each) - Provided by WMT 2021 QE shared task Table 3.1: Summary of data groups used in the experiments. 3.3.2 Frameworks Quality Estimation To train all QE models of our study, we developed a new QE frame- work with the ability to invoke multilingual models from the HF model repository. The framework is similar in architecture to âMonoTran- sQuestâ(Ranasinghe et al., 2020), but adapted to the needs of our experiments. The differences with âMonoTransQuestâ are the additional tokens (<OOD>and<ID>) added during the tokenization process, as well as the resizing of the modelâs token embeddings in order to support the added tags. Also, rather than computing the softmax, we directly used 81 logits to estimate the quality labels. This approach saves time, as com- puting softmax can be expensive(Ruder, 2016) and we do not need to find the probabilities of each prediction for our experiments. In all our experiments, we chose to use XLM-RoBERTa 7 (XLM-R)(Conneau et al., 2020), to derive cross-lingual embeddings, which has shown success in prior studies such as Ranasinghe et al., (Ranasinghe et al., 2020). Training and evaluation details of QE models During Step 1 (see Section 3.2.1), we trained an OOD QE model and evaluated it every 1000steps HF 8 using the train and dev sets from Group 1 (see Section 3.3.1). In Step 2, we trained and evaluated QE mix models every 500steps HF using a mix of OOD and ID data from Group 1. For Step 3, we evaluated the final domain-specific QE model after 500steps HF using only an ID train and dev set. Throughout the training, we used an early stopping mechanism to halt the training process if there was no improvement in the evaluation loss after 5 evaluations. We adjusted the default evaluationsteps HF from 500 to 1000 for Step 1 due to the larger number of training samples in that step. This ensures efficient training by reducing evaluation overhead and stabilizing model performance checks. Machine Translation Our approach to generating synthetic ID (Approach 2, Section 3.2.2) differs from prior studies, such as Eo et al.,(Eo et al., 2021), which rely on a generic/common translation model (e.g., Google machine translate). We first trained a separate NMT model on a subset of the original dataset. This approach ensures that the training data and the 7 xlm-roberta-large 8 steps HF refers to the Hugging Face frameworkâs training or evaluation steps, which are different from the ones we described in Section 3.2.1. 82 data used for translation have similar vocabularies, cover comparable topics, styles, and domains, which leads to higher quality translations. We used an in-house MT framework to train our models, based on pre-trained mBART-50(Y. Liu et al., 2020) from HF. We followed the Seq2SeqTraining arguments recommended by HF and trained the model for Approach 2, stopping the training if the evaluation loss did not improve after 5 evaluations. We used default hyperparameters recommended by HF for QE and MT, and our frameworks with modified hyperparameters are available athttps://github.com/JoyeBright/DA-QE-EAMT2023to reproduce our results. All models were trained on a single NVIDIA Tesla V100 GPU using mixed precision (16-bit). 3.4 Results To assess the performance of our approach, we evaluate output from the trained QE models in comparison to the reference quality metric (HTER/TER) on the test sets described in data Groups 1 and 3 (see Section 3.3.1). We use Pearsonâs coefficient (Ï â[â1,1], which we rescale toâ100 to 100 for clarity) to assess the correlation of our predictions with the test set. We evaluate the translation quality of our MT models using the BLEU score. 3.4.1 Baseline results To establish a baseline for our study, we fine-tuned XLM-R with the ID data for each language pair as provided by WMT 2021 shared task (Group 1âsee Section 3.3.1). This is a conventional approach employed in prior research, such as Ranasinghe et al.(Ranasinghe et al., 2020), where pre-trained models are utilized to provide cross-lingual reference for training QE models. 83 Language pair Baseline NO TAGTAG Increase % DAG 1 DAG 2 DAG 1 DAG 2 EN-DE47.1749.9349.54 51.9051.2510.03 EN-ZH29.1634.7535.2735.62 36.6025.51 RO-EN83.6383.6783.7483.37 84.4000.92 RU-EN40.6544.9145.40 47.1643.9816.01 Table 3.2: Pearson correlation scores for proposed QE models across 4 language pairs: EN-DE, EN-ZH, RO-EN, and RU-EN. For each language pair, the bold result indicates the highest-performing method compared to the baseline. Results for the first and second DAG approaches are reported under DAG 1 and DAG 2, respectively. The column labeled âIncrease %" shows the percentage improvement for the highest-performing model (in bold) compared to the baseline. We also attempted to compare our work with the models of Ru- bino(Rubino, 2020) and Lee(Lee, 2020). For the latter work, their experiments used the WMT 2020 test sets, while we used WMT 2021, which makes it difficult to compare our results to theirs directly. Fur- thermore, we could not replicate their models as no code is available (at the time of writing this chapter). 3.4.2 Main results In Table 3.2 we present our results using the DAG approaches and the two training modes (TAG and NO TAG). Additional details on the statistical tests for each language pair are available in Section 3.8.1. The results in Table 3.2 show that, in general, all of the proposed DA methods performed better than the baseline for each language pair, except for Approach 1 (DAG 1) in the RO-EN language pair. For this language pair, the use of a domain tag led to reduced performance, and the improvement achieved without such a tag was not statistically significant. We also observe that the increase of performance compared to the baseline for each language pair shown as percentage in the last 84 column of Table 3.2 is substantial, except for RO-EN (only 0.92% increase over the baseline). This is mainly due to the already high baseline performance (83.63), making it challenging to achieve significant improvements. Among the other language pairs, the EN-ZH pair had the largest increase in performance â just over 25%. The RU-EN and EN-DE pairs had the second and third highest increases, with improvements of around 16% and 10% over their respective baselines. Additional indication of domain results. The results indicate that incorporating tags into the DA training pipeline was generally effective, although in some instances, the improvement was not statis- tically significant compared to the models that were trained without tags. However, it was observed that at least one model outperformed the same language pairâs models that were not trained with tags, when DAG techniques were used. Specifically, the EN-DE Approach 1 model trained with tags performed better compared to Approach 2 without tags, as did the EN-ZH Approach 1 model trained with tags relative to the same approach without tags. Finally, the RO-EN Approach 2 model trained with tags outperformed Approach 2 without tags, and the RU- EN Approach 1 model trained with tags exhibited better performance than Approach 1 without tags. 3.4.3 Data augmentation results Upon analyzing the integration of DAG techniques into the specialized QE pipeline, we observe that for most language pairs, both approaches showed better performance than their respective baselines. However, in situations where tags were not employed, Approach 2 only showed statistical significance over Approach 1 in the EN-ZH and RU-EN language pairs. Moreover, when tags were used, Approach 2 led to statistically significant improvements only for EN-DE and EN-ZH. These findings indicate that the effectiveness of DAG approaches and 85 tag usage can vary depending on language pair and task context. Our findings suggest that users should evaluate both DAG ap- proaches with and without tags on a validation set specific to their language pairs. Tags are recommended for languages with complex syntax or morphology (e.g., EN-DE), whereas omitting tags may be preferable when no performance gains are observed (e.g., RU-EN). For language pairs like EN-ZH, where DAG consistently shows improve- ments, DAG should be applied regardless of tag usage. 3.4.4 Zero-shot results In order to evaluate the effectiveness of our QE models in the context of ZSL, we compared their performance to the baseline models for the EN-CZ and EN-JA test sets. The results are presented in Table 3.3. The findings show that, for the EN-CZ test set, the QE model trained solely on the EN-DE dataset achieved the highest performance among all QE baselines, with a Pearson correlation score of 46.97. Additionally, we observe that our proposed DA pipeline performed even better than the highest-performing baseline for EN-CZ, but only DAG Approach 1 and 2 with tags were found to be statistically significant. Likewise, for the EN-JA test set, the highest-performing QE baseline was the one that was trained solely on the RU-EN dataset, with a Pearson correlation score of 20.32. In contrast to EN-CZ, none of the models that were trained with our pipeline and with the RU-EN dataset outperformed the baselines. Nevertheless, we observed that three models trained with EN-ZH and using our pipeline (Approach 1 with and without tag, and Approach 2 with tag) performed better than the highest-performing baseline. Overall, these findings suggest that if a QE model is conventionally trained with and evaluated on an unseen QE dataset, some extent of ZSL capabilities can be achieved due to the use of XLM-R. However, the proposed DA pipeline can significantly increase this extent, whether 86 through models trained with the same dataset or other datasets used in the pipeline. Trained on Test set Baseline NO TAGTAG DAG 1 DAG 2 DAG 1 DAG 2 EN-DE EN-CZ46.9748.7748.0747.7847.82 EN-JA09.6718.1608.0016.1217.36 EN-ZH EN-CZ35.5649.3348.5447.9846.83 EN-JA13.1322.7719.8722.2421.54 RO-EN EN-CZ26.3339.1039.7939.2040.41 EN-JA18.8820.3418.5520.1121.22 RU-EN EN-CZ28.4245.5844.8546.4345.22 EN-JA20.3217.6417.0417.2619.63 Table 3.3: Performance comparison of the proposed methods and the baseline using Pearson correlation. All models were trained on the EN-DE, EN-ZH, RO-EN, and RU-EN datasets and evaluated in a ZSL setting on the EN-CZ and EN-JA test sets. Results for the two DAG strategies are presented under DAG 1 (authentic ID augmentation) and DAG 2 (synthetic ID augmentation). 3.5 Additional observations 3.5.1 Cross-lingual inference Table 3.4 presents data that shows that our proposed methodology has an overall advantage over the conventional training method of using a PLM and fine-tuning it with QE data (baselines) in terms of cross- lingual inference. That is, the QE models trained with our proposed DA pipeline not only perform significantly better than baselines on their target domain and language pair but can also estimate the quality of other language pairs to some extent better than their corresponding baseline. By examining the data closely (bottom to top row of the Table 3.4), we observe that XLM-R provides a limited level of cross-lingual in- 87 ference, which is insufficient for estimating quality labels due to the absence of prior knowledge about them. However, using Step 1 of our pipeline, which utilizes little inference knowledge, the model achieves an acceptable level of generalization across all language pairs. Specifically, Step 1 achieved an average Pearson correlation score of approximately 39, which is higher than all baseline scores by 3 points, except for the RO-EN pair, which achieved around 42. Furthermore, the model trained using Step 1 of the pipeline achieved a Pearson correlation of around 70 when evaluated with the RO-EN test set. This result can be attributed to the training of the model with the Italian (IT) dataset, used as OOD data. The use of Italian data likely provided the model with exposure to linguistic features and patterns that closely align with those found in Romanian (RO), thereby enhancing its ability to make accurate quality estimations for the RO- EN pair. From a linguistic perspective, both Italian and Romanian are part of the âRomance language familyâ (refer to Section 3.8.3), which share significant similarities in terms of syntax, morphology, and lexical structure. These shared characteristics, such as similar word formation rules, verb conjugations, and sentence structures, can facilitate cross-lingual generalization. As a result, the model was better equipped to handle Romanian data despite not being explicitly trained on it, leading to a significantly higher Pearson correlation score on the RO-EN test set. This phenomenon highlights the importance of leveraging linguistically related languages for improved cross-lingual model performance in quality estimation tasks. As we move up the table, we can observe that the model built in Step 2 of our pipeline becomes more specific toward the task and the ID datasets. Consequently, there is an average improvement of around 3.5 Pearson correlation (from 39.36 to 42.83) across the languages. The improvement can be attributed to the modelâs enhanced ability to leverage DA, where exposure to both OOD and ID data helps the model 88 better generalize to in-domain tasks. By fine-tuning with ID data, the model adapts more closely to the characteristics of the specific language pairs, resulting in better task-specific performance. This targeted adaptation strengthens the modelâs capacity to evaluate translation quality for its language pair, leading to improved performance on both in-domain and cross-lingual tasks. Consequently, models fine-tuned in Step 2 not only provide more precise estimates of translation quality within their specific language pair but also exhibit superior cross-lingual inference, outperforming the baseline models. This demonstrates the efficacy of our DA pipeline in enhancing task-specific and cross-lingual QE performance. 3.5.2 OOD Performance The main goals of DA are to quickly create an adapted system and to develop a system that performs well on ID test data while minimizing performance degradation on a general domain. In our study, we showed that models from Step 1 or Step 2 can be fine-tuned quickly using the userâs data (achieving the first of these goals). Our main focus was on the assessment of ID QE. Here, we test the performance of our ID models on an OOD test set in order to assess the degree of degradation. Our results, summarized in Table 3.5, indicate that not only did all ID models outperform the corresponding baselines on the OOD test set, but that incorporating ID data in Approaches 1 and 2 did not compromise the performance with respect to OOD. However, comparing the modelsâ performance with models trained solely on OOD, we see a small yet statistically significant performance drop, which is inevitable and, in most cases, acceptable. 89 Models Test Sets AVG EN-DE EN-ZH RO-EN RU-EN Baseline47.1719.6744.9632.91 36.17 EN-DE49.9322.6678.9739.55 47.77 â02.7602.9934.0106.64 11.60 Baseline30.3429.1647.5536.87 35.98 EN-ZH43.4634.7580.5142.67 50.34 â13.1205.5932.9605.80 14.36 Baseline24.6423.5683.6339.97 42.95 RO-EN43.0224.3183.6738.74 47.43 â18.3800.7500.04-01.23 04.48 Baseline22.4024.6757.1740.6936.23 RU-EN25.3626.0675.3444.9142.91 â02.9601.3918.1704.2206.68 Step238.2924.7276.9631.35 42.83 Step130.8016.5770.1439.93 39.36 XLM-R-02.7407.3002.9703.12 02.66 Table 3.4: Performance comparison of proposed models and baselines across all test sets using Pearson correlation as the metric. â represents the difference between them. The âAVGâ column reports the mean Pearson correlation across all four language pairs (EN-DE, EN-ZH, RO-EN, RU-EN), and the differences (â) are computed with respect to these averaged values. âXLM-R (not trained)â refers to the raw XLM-R encoder used without any QE fine-tuning, included as a lower-bound reference. Step 1: model trained with OOD. Step 2: model trained with DAG approach 1 and OOD (Approach 2 in Step 2 had similar results, not included). Models and baselines are color-coded for clarity, with bold numbers indicating the average â across all language pairs, and underlined numbers representing each modelâs performance on their respective test sets. 3.5.3 Training time Compared to the conventional approach of using a PLM and fine-tuning it with QE data (baselines), our proposed DA methodology results in a significant improvement in performance, regardless of whether 90 Training setup ID Models (Step 3)OOD-onlyDAG 1DAG 2 EN-DE EN-ZH RO-EN RU-ENEN-IT (OOD+ID) (OOD+SynthID) Baseline11.953.5911.603.43 64.3365.2464.76 Our pipeline54.6259.3052.5147.36 â Baseline 42.6755.7140.9143.93 â OOD -09.71 -05.03 -11.82 -16.97 Table 3.5: Performance on the OOD test set (Pearson correlation). âOOD-onlyâ is the model trained exclusively on the out-of-domain EN-IT data (Step 1). âDAG 1â denotes Step 2 fine-tuning using all authentic ID data concatenated across language pairs; âDAG 2â uses synthetic ID data generated via Approach 2 (Section 3.2.2). â Baseline compares each ID model with its corresponding baseline, and â OOD com- pares each ID model with the OOD-only model. Note that OOD-only, DAG 1, and DAG 2 do not vary across language pairs and therefore appear once for clarity. we include tags in the sentences or not. However, it requires two additional training steps: Step 1, training an OOD QE model, and Step 2, fine-tuning the model using a mix of OOD and ID QE data. These additional steps require more time. Step 1 and Step 2 (with both DAG approaches) are reused (i.e., not trained) for each language pair, and Step 3 of the pipeline took almost the same amount of time across all languages. That is why we present the consumed time for EN-ZH in Figure 3.3, and use it to discuss training times for other language pairs as well. Models trained with tagged data have a similar training time. The data presented in Figure 3.3 indicates that Step 1 has the highest training time of approximately 3.4 hours. It is noteworthy that this long training time is partly due to the fact that the model was evaluated after every 1000steps HF , which consequently resulted in a longer running time in comparison to other models that were evaluated after every 500steps HF . The model that was trained is publicly accessible, and other individuals can utilize it to fine-tune with new ID datasets, avoiding the need for retraining for each specific ID data. This applies to both DAG approaches, given that the target language pair was used in Step 2 of the pipeline. If not, Step 1 must 91 0.46 3.41 1.66 1.11 0.31 0.32 01234 Baseline Step 1 Step 2 DAG 1 Step 2 DAG 2 Step 3 DAG 1 Step 3 DAG 2 Figure 3.3: Training time (in hours) for models in the EN-ZH language pair, where Step X refers to the training step outlined in Section 3.2.1, and DAG X denotes the DAG approach used in the second step of the pipeline. The term âBaselineâ denotes a model fine-tuned from XLM-R. The X and Y axes represent the training time in hours and the approaches used to train the model, respectively. be fine-tuned with a new set of QE data. 3.6 Related work 3.6.1 Data scarcity in QE The issue of data scarcity in MT QE has been explored in numerous previous studies. The work of Rubino and Sumita (2020) involves the use of pre-training sentence encoders and an intermediate self-supervised learning step to enhance QE performances at both the sentence and word levels. This approach aims to facilitate a smooth transition between pre-training and fine-tuning for the QE task. Similarly, Fomicheva et al. (2020) proposed an unsupervised method for QE that does not depend on additional resources and obtains valuable data from MT systems. Qiu et al. (2022) conducted a recent study on the impact of various 92 types of parallel data in QE DAG, and put forward a classifier to differentiate the parallel corpora. Their research revealed a significant discrepancy between the parallel data and real QE data, as the most common QE DAG technique involves using the target side of parallel data as the reference translation(Baek, Kim, Moon, Kim, & Park, 2020; Qiu et al., 2022), followed by translation of the source side using an MT model, and ultimately generating pseudo QE labels(Freitag, Rei, et al., 2021). Kocyigit et al. (2022) proposed a negative DAG technique to improve the robustness of their QE models. Their method involves training a sentence embedding model to reduce the search space for QE predictions by using contrastive loss. This approach improves the modelâs capacity to differentiate between high- and low- quality translations by maximizing the distance between dissimilar sentence pairs in the embedding space, thereby refining the overall QE performance. 3.6.2 Domain adaptation in QE To tackle the challenges with translating data when training data comes from diverse domains, researchers have extensively used DA in MT. DA involves training a large generic model and then fine-tuning it with domain-specific data(Chu & Wang, 2018; Saunders, 2022; Pourmostafa Roshan Sharami, Shterionov, & Spronck, 2021; M.-Q. Pham et al., 2022) and Chapter 2. In MT, one way to achieve DA is by appending tags to sentences to handle different domains(Sennrich et al., 2016a; Mattoni et al., 2017; Vanmassenhove, Hardmeier, & Way, 2018; Chu & Dabre, 2019) and reduce catastrophic forgetting. Despite being useful in MT, to the best of our knowledge, DA has not been widely used in QE. Dongjun Lee(Lee, 2020) proposed a two-step QE training process similar to our own, and Raphael Rubino(Rubino, 2020) pre-trained XLM and further adapted it to the target domain through intermediate training. Both studies demonstrated that adding 93 a step before fine-tuning improves performance compared to fine-tuning alone. However, unlike our methodology, neither of them included sentence tags or conducted additional fine-tuning (such as Step 3 in our methodology). As a result, their QE models are not as specialized for the target domain as ours. A few researchers have made attempts to integrate aspects of DA into QE. For instance, in an effort to improve QE performance in domain-specific scenarios, Arda Tezcan(Tezcan, 2022) included fuzzy matches into MonoTransQuest with the aid of XLM-RoBERTa model and DAG techniques. 3.7 Conclusion and future work This chapter addresses two key challenges related to QE of MT: (i) the scarcity of QE data, which is mitigated by utilizing pseudo-QE data followed by fine-tuning on authentic data, and (i) the difficulty of estimating translations across diverse domains, addressed through domain-specific fine-tuning using in-domain (ID) data. The primary aim of this study is to enhance the performance of QE models by addressing these challenges. To do so, we propose a solution that utilizes DA techniques adopted from MT. We adapt the âmixed fine- tuning + fine-tuningâ approach(Chu et al., 2017) and extend it with DAG as an alternative to the traditional oversampling technique. We adopt a three-step training methodology: (i) we fine-tune XLM-R, a language model, with a large generic QE dataset, which enables the model to generalize; (i) we fine-tune the model with a mix of out-of- domain (OOD) and ID data derived from two DAG approaches; and (i) we fine-tune the model with a small amount of domain-specific data, which leads to a more specific model. We evaluated modelsâ performance with and without domain tags appended to the sentences. As seen in previous studies, sentence tags help the model differentiate between various domains by providing contextual cues. Our findings 94 partially confirm this effect, indicating that domain tags contribute to improved domain-specific performance. Our experiments show significant improvements across all language pairs under consideration, indicating that our proposed solution has a beneficial impact in addressing the aforementioned challenges. Our study also demonstrates the effectiveness of both proposed DAG ap- proaches and shows that using domain tags improves the performance of the models. Additionally, we find that our model outperforms the baseline in the context of ZSL and in cross-lingual inference. Moving forward, there are several directions for future work based on our findings. First, it would be interesting to investigate the performance of our pipeline on low-resource language pairs, where there is limited ID data available. This is particularly relevant given the smaller coverage of QE datasets compared to parallel data in MT. Second, we only used one type of OOD data in our experiments (EN-IT); it would be useful to explore other OOD data over different language pairs for QE. Third, it would be valuable to study the performance of other PLMs and not only XLM-R. Fourth, since the choice of languages employed in the pipeline was based on availability, we would suggest exploring a more regulated approach for selecting the languages to be used in the proposed pipeline. Specifically, the optimal transfer languages can be selected based on their data-specific features, such as dataset size, word overlap, and SW overlap, or dataset-independent factors, such as genetic (see Section 3.8.3) and syntactic distance(Y.-H. Lin et al., 2019). 95 3.8 Supplemental material This section provides additional information that complements the findings discussed in the main chapter. 3.8.1 Statistically significance test results The statistical significance test results for the predictions in Table 3.2 for the language pairs EN-DE, EN-ZH, RO-EN, and RU-EN are shown in Table 3.6. Language pair Models NO TAG 1 NO TAG 2 TAG 1 TAG 2 EN-DE BaselineYYYY NO TAG 1-NNY NO TAG 2--Y TAG 1---Y EN-ZH BaselineYYYY NO TAG 1-YYN NO TAG 2--N TAG 1---Y RO-EN BaselineNYYY NO TAG 1-NYY NO TAG 2--N TAG 1---N RU-EN BaselineYYYY NO TAG 1-Y NO TAG 2--NY TAG 1---N Table 3.6: Statistically significant test results with a p-value less than 0.05. The letter âY" in the table indicates that the corresponding prediction in Table 3.2 is statistically significant, while âN" indicates that it is not. 96 Language pair BLEU â Eval Loss â EN-DE41.2501.09 EN-ZH32.2801.52 RO-EN49.6000.96 RU-EN41.2901.61 Table 3.7: MT performance used as a component of Approach 2 in the proposed DAG (Section 3.2.2). 3.8.2 Machine translation performance We utilized multilingual MT systems to generate synthetic ID data. Table 3.7 displays the results of the top-performing models used in generating this data. 3.8.3 Genetic distance 051015202530 Genetic distance (%) DE ZH RO RU JA CZ Figure 3.4: Genetic distance between IT and other languages: DE, ZH, RO, RU, JA, and CZ. In MT, measuring the similarity between languages is important for effective cross-lingual learning. One such measure is the genetic distance between languages, which has been shown to be a good indicator of 97 language similarity for independent data(Y.-H. Lin et al., 2019). To illustrate this, we calculate 9 and present the genetic distance scores between Italian (used as OOD data) and the other languages included in our study in Figure 3.4. The genetic distance is represented as a numerical value ranging from 0 (indicating the same language) to 100 (the greatest possible distance). 9 http://w.elinguistics.net/Compare_Languages.aspx 98 This chapter is based on the following published paper: Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. Analysis of Vocabulary and Subword Tokenization Settings for Optimal Fine-tuning of MT: A Case Study of In-domain Translation. In Proceedings of Recent Advances in Natural Language Processing (RANLP 2025), p. 970â979, Varna, September 8â10, 2025. Improvements have been made to the title, figures, and certain sections to align the content with the broader context of this dissertation. 4 Analysis of Vocabulary and Subword Tokenization Settings for Optimal Fine-tuning of MT T he choice of vocabulary and SW tokenization has a signif- icant impact on both training and fine-tuning of language and translation models. Fine-tuning is a common practice in optimizing a model with respect to new data. However, new data potentially introduces new words (or tokens), which, if not considered, may lead to suboptimal performance. In addition, the distribution of tokens in the new data can differ from the distribution of the original data. As such, the original SW tokenization model could be less suitable for the new data. With this work, we aim to gain better insights on the impact of SW tokenization and vocabulary generation on the performance of NMT models fine-tuned to a specific domain. To do so, we compare several strategies for SW tokenization and vocabulary generation and investigate the performance of the resulting models. Our findings show that the best way to fine-tune for DA is to consistently use both BPE and vocabulary from the in-domain data, which helps the model pick up on important domain-specific terms. At the same time, it is crucial not to lose sight of the vocabulary of the base (pre-trained) modelâmaintaining coverage of this vocabulary ensures the model keeps its general language abilities. The most successful configurations are those that introduce plenty of frequent domain terms while still retaining a substantial portion of the base model vocabulary, leading to noticeably better translation quality and adaptation, as seen in higher BLEU scores. These benefits, however, often come with greater computational costs, such as longer training times, since the model must learn more new tokens. Conversely, approaches that skip important domain terms or combine mismatched tokenization and vocabulary do not perform as well, making it clear that both domain- specific adaptation and broad vocabulary coverage matterâand that these gains are realized when the vocabulary preserves a good portion of the base (pre-trained) model. While using in-domain BPE and vocabulary yields the best DA, it substantially reduces out-of-domain translation quality. Hybrid configurations that combine base and domain vocabularies help balance this trade-off, maintaining broader translation capabilities alongside improved domain performance. 102 4.1 Introduction and background Fine-tuning is a common practice in optimizing MT and PLM with respect to new data. It is often in the context of DA where an existing model is tuned to perform better on a specific domain (different from what the model was originally trained for)(T. Luong, Pham, & Manning, 2015; Dakwale & Monz, 2017; X. Wang, Chen, & Xing, 2019; Mahdieh, Chen, Cao, & Firat, 2020; Chopra, Kothawade, Aynaou, & Chadha, 2023). The positive effect of fine-tuning has been demonstrated in various previous works. For example, T. Luong et al. (2015) trained an NMT model on English-German general-domain data and then fine-tuned it on a conversational data in the same languages, leading to an increase of 3.8 BLEU(Papineni, Roukos, Ward, & Zhu, 2002c) points compared to the original model. Sharami, Shterionov, and Spronck (2022) show that fine-tuning is preferred (as it leads to better results) than training from scratch, even if the data allows the latter. To improve the translation performance on a new domain (without degrading the performance on the generic domain) is to ensemble the fine-tuned model with the already trained baseline, as done by Freitag and Al-Onaizan (2016). However, while they achieve a substantial increase of quality (+7.2 BLEU points ), they note that because the in-domain data comprises of new vocabulary and linguistic features that are different from the generic data, the performance of the fine-tuned models drops for the generic domain task, especially when it comes to domain-specific contexts (e.g., medical and legal domains). While newly introduced data brings in new information, i.e., new, unseen words, it could be that, statistically, tokenization into SWs is significantly different from the tokenization of the original model(R. Lim et al., 2018; Yeung, 2019; Sato et al., 2020; Hwang, Oh, & Choe, 2024). If not properly addressed, this new information may have an adverse effect on the system. 103 To address this problem, Sato et al. (2020) proposed a method to adapt the embedding layers of the initial model to the target domain by projecting the general word embedding obtained from target-domain monolingual data onto source-domain embeddings. They reported a 3.86 and 3.28 BLEU points gain in EnglishâJapanese and GermanâEnglish translation, respectively. In this chapter, we investigate the impact of using different SWs and vocabularies on the performance of fine-tuned NMT systems. We identify a best-case setup and preferable setups under constrained fine-tuning conditions, such as limited domain-specific data. That is, we aim to investigate which fine-tuning conditions (or settings) of a domain-specific model lead to the best performance. Specifically, our objectives are: 1. Identify optimal SW combination choices and vocabulary configu- rations for a given MT model and fine-tuning dataset. 2.Determine the best fine-tuning conditions under data limitations. To achieve the aforementioned objectives, we use one large dataset (âŒ12.7 million parallel sentences) for training and a smaller in-domain dataset (âŒ248,000 parallel sentences) for fine-tuning multiple MT sys- tems. This setup allows us to examine the extent to which a model trained on a substantial amount of general-domain data can be im- proved by fine-tuning with additional domain-specific data, which alone would be insufficient to train a robust model from scratch. Each fine-tuned alternative, is trained on a different set of options of how the SWs and the vocabulary are created. We analyze these fine-tuning strategies to find the best setup based on available data. In our case study, for example, we have access to the data of both models (initial and fine-tuned). However, as already discussed in(Freitag & Al-Onaizan, 2016; Dakwale & Monz, 2017; Zimelewicz et al., 2024), initial models are mostly deployed in an application; thus data might 104 not be available at the production time. As such, it is paramount to have a guideline based on the available data that determines how to best generate SWs and vocabularies. In this work, we use Byte-Pair Encoding (BPE) (Sennrich et al., 2016a) for SW units. 1 It is noteworthy that the point of this research is to investigate the best fine-tuning setup, rather than identifying the best model. Typically, fine-tuning involves tokenizing the new data using the vocabulary originally employed in training the model. This ensures consistency in SW tokenization and prevents discrepancies in word representations. However, it is not always evident whether this practice yields the best translation performance, especially when the new domain introduces a significantly different linguistic distribution or unseen vocabulary. Thus, we explore alternative approaches to SW tokenization and vocabulary creation to determine if different configurations could lead to better fine- tuning outcomes. Specifically, given a pre-training dataset Aâwhether in-domain, out-of-domain, synthetic, e.g., generated using methods like those in(Sharami, Shterionov, & Spronck, 2023), or authenticâand a fine-tuning dataset B, we investigate which tokenization and vocabulary configurations best enable the model to retain and adapt pre-training- derived parameters in a way that improves translation quality on B. This chapter is organized as follows. Section 4.2 presents the key decision points that frame our study. Section 4.3 describes the datasets used in our experiments. Section 4.4 details our experimental design and the overall training framework. Section 4.5 reports and analyzes the results, incorporating relevant recommendations. In Section 4.6, we discuss the limitations of our approach. Finally, Section 4.7 summarizes our findings and outlines directions for future work. 1 Throughout this chapter, we use âsubwordâ (SW) and âBPEâ synonymously, as all experiments use BPE for SW tokenization. 105 4.2 Decision points Given a modelMtrained on a datasetD, which represents a specific domaind, and a fine-tuning datasetErepresentative for domaine, the following decision points need to be made: 4.2.1 SW tokenization A key decision in fine-tuning is determining how to tokenize words into SW units. This choice affects how the model processes domain- specific terminology and generalizes across datasets. We consider three approaches: âą Reusing the original SW model trained onD(D SW ). This maintains consistency with the pre-trained model. âą Training a SW model on the combined dataset ((D+E) SW ) to integrate both the original and fine-tuning data. âąTraining a new SW model on the fine-tuning dataset (E SW ) to better capture domain-specific terminology. 4.2.2 Vocabulary creation SW tokenization techniques arose in response to two major challenges in NMT: (i) the lack of generalizabilityâmodels often fail to process words not seen during training, leading to out-of-vocabulary (OOV) problems and degraded performance; and (i) the need to limit vocabulary size, as large vocabularies increase memory consumption and computational cost, which remains a practical constraint in current neural translation systems, particularly when working with large models or limited GPU resources. Fine-tuning introduces a third, less often addressed challenge: whether the vocabulary used during adaptation adequately captures the token 106 distribution of the fine-tuning dataset. If not, domain-specific content may be poorly represented, limiting the effectiveness of adaptation. We consider three strategies for vocabulary construction: âą Reusing the original vocabulary â the vocabulary that the pre-trained modelMwas originally trained with (denoted |D|). This strategy ensures full compatibility with the pre-trained token embeddings and does not require any modifications to the embedding space. âąExpanding the vocabulary â augmenting the original vo- cabulary with additional tokens found in the fine-tuning dataset E, resulting in a combined vocabulary|D+E|. This approach aims to better cover domain-specific terms inEwhile retaining compatibility with Mâs original vocabulary. âąConstructing a new vocabulary solely from the fine- tuning data â generating the vocabulary exclusively fromE (denoted|E|). This strategy maximizes domain-specific represen- tational capacity but introduces a mismatch with the pre-trained vocabulary of M. Handling vocabulary-embedding alignment. In the first strategy (|D|), the embedding space remains unchanged, as all tokens are already present in the pre-trained model. In the second and third strategies (|D+E|and|E|), we introduce new tokens absent from the original vocabulary. To accommodate these, we extend the embedding matrix by appending randomly initialized vectors for the new tokens while preserving the original embeddings. The key distinction lies in the degree of divergence from the original model. Strategy 2 retains the original vocabulary and extends it with tokens fromE, maintaining alignment with the pre-trained structure. 107 In contrast, Strategy 3 derives both the vocabulary and BPE model entirely fromE, resulting in a larger mismatch with the pre-trained model and necessitating greater adaptation during fine-tuning. Since SW tokenization and vocabulary creation are interdependent, we explore all feasible combinations, resulting in nine configurations. These include applying each SW model (D SW ,E SW ,(D+E) SW ) with different vocabulary choices (|D|,|D + E|,|E|). Following these decision points, given a fine-tuning dataset, we can consider three SW models. With these models, we (i) tokenize the vocabulary sources, and (i) tokenize the training sets for fine-tuning. Typically, these two processes are tied to each other, i.e., once the SW model is learned and applied to the training data, the vocabulary is the set of SW units that appear in the (processed) data. However, this is not a hard constraint. For instance, datasetEcan be processed withE SW , but the vo- cabulary used for training can still be based onDand derived from applyingD SW . Such mismatched configurations, though theoretically possible, can lead to tokenization inconsistencies and degrade model performance. Since they are suboptimal, we exclude them from this study. 4.3 Data We used two datasets: (i) a large out-of-domain corpus consisting of approximately 12.7 million English-German sentence pairs drawn from the WMT18 dataset 2 , and (i) a smaller (âŒ248K sentence pairs) 2 http://statmt.org/wmt18/translation-task.html. The original WMT18 dataset is considerably larger; we selected a representative subset to reduce com- putational costs. The selected subset and all preprocessing scripts are released for reproducibility. 108 in-domain medical corpus extracted from the multi-domain English- German data introduced by (Koehn & Knowles, 2017). Out-of-domain dataset The out-of-domain corpus used to train our base model is a randomly selected subset of the WMT18 English- German dataset, which contains parallel data from various domains. We selected approximately 12.7 million sentence pairs to balance domain coverage with training efficiency. In-domain dataset For fine-tuning, we used 248,099 English-German sentence pairs from the medical domain of the multi-domain dataset introduced by(Koehn & Knowles, 2017). We used the cleaned and re-split version provided by(Aharoni & Goldberg, 2020), which removes duplicates and prevents data leakage between train, dev, and test sets. Combined dataset (D+E) For configurations requiring bothD andE, we oversampled the in-domain medical data to match the size of the WMT18 subset and concatenated them. The combined data was shuffled and used to train BPE models or extract vocabularies. This ensures that both domains are equally represented, avoiding bias toward the larger out-of-domain corpus. 4.4 Experiments To investigate the impact of SW and vocabulary generation choices on fine-tuning, we followed the decision points outlined in Section 4.2 and ran experiments using the English-German data described in Section 4.3. We compared the resulting fine-tuned models using BLEU(Papineni et al., 2002c), TER(Snover et al., 2006), chrF2(PopoviÄ, 2015). Addi- tionally, we measured training time and estimated CO 2 emissions using CodeCarbon (Courty et al., 2024). 109 4.4.1 Experimental design Given our taskâfine-tuning a model trained on the WMT18 out-of- domain dataset (D) using in-domain medical data (E)âa total of 9 theoretical configurations exist, arising from three possible vocabulary sources (D,E, orD+E) and three BPE models trained on the same sources. Each configuration couples one vocabulary source with one BPE model, which is used for both vocabulary construction and fine- tuning data tokenization. However, as discussed in Section 4.2, we imposed constraints to ensure tokenization consistency. Specifically, we excluded configurations where the vocabulary is derived from one source (DorE), but the fine-tuning tokenization is performed using a BPE model trained on the combined dataset (D+E). These mismatches introduce inconsistencies, as the vocabulary may not align with how the data is tokenized. Since bothDandEare available, we ensure that the same BPE model is used for both vocabulary construction and fine-tuning tokenization. In total, there are 9 possible configurations (from all combinations of BPE models and vocabulary sources). However, we exclude 2 incon- sistent configurations, leaving 7 valid configurations for our experiments (see Table 4.1). 4.4.2 Model architecture and training Setup Framework and model architecture We used the OpenNMT- py 3 framework(Klein et al., 2017) to train and fine-tune Transformer- based NMT models(Vaswani et al., 2017b). Each model had 6 encoder and 6 decoder layers, 512-dimensional embeddings, 8 attention heads, and a feed-forward size of 2048. We used the Noam optimizer schedule with a learning rate of 2.0, 8,000 warmup steps, and label smoothing of 0.1. Batching was done over 10,240 tokens with gradient accumulation 3 https://opennmt.net/OpenNMT-py/ 110 Config. BPE for vocab + FT data Vocabulary source C1D BPE D C2D BPE D + E C3D BPE E C4E BPE D C5E BPE D + E C6E BPE E C7(D + E) BPE D + E Table 4.1: Valid fine-tuning configurations. Each row represents a consistent setup where the same BPE model is used for both tokenizing the fine-tuning data and constructing the vocabulary.Drefers to the WMT18 out-of-domain dataset;E refers to the in-domain medical dataset. over 4 steps. Training setup All models were trained for up to 200,000 steps, with validation and checkpointing every 1,000 steps. We applied early stopping after 10 validations without improvement. All experi- mentsâincluding the base and fine-tuned modelsâwere run on a single NVIDIA A40 GPU. Base model The base model was trained on the WMT18 out-of- domain dataset. We applied BPE with 50K merge operations to both source and target sides. The resulting vocabularies and tokenized data were used to train the initial Transformer model, which served as the starting point for all fine-tuning experiments. Fine-tuning Fine-tuning was done on the in-domain medical dataset using the same model architecture and training settings. Each configu- ration (C1âC7) used a specific combination of vocabulary source and BPE model (see Table 4.1). The base model checkpoint was reused 111 Config BPE model Vocab SRC BLEUâ chrF2â TERâ CO 2 (g)â Time (h)â C1 D BPE D53.669.4 49.31658.6907:45 C2 D BPE D + E53.469.5 49.91198.6605:15 C3 D BPE E51.768.4 50.9907.2404:00 C4 E BPE D46.664.5 53.0723.9403:11 C5 E BPE D + E53.168.9 49.7729.0403:15 C6 E BPE E54.8 69.8 48.91587.4109:30 C7(D + E) BPE D + E53.269.1 50.1543.8403:08 Table 4.2: Evaluation scores of fine-tuned models. Note: For each configu- ration, the BPE model shown is used to tokenize both the vocabulary source and the fine-tuning data. All models were fine-tuned on the in-domain datasetEand evaluated on the same test set. CO 2 emissions and training times were recorded during fine-tuning. across all configurations, and only the vocabulary and tokenized data differed. BLEU was used to track validation performance. BPE settings We trained separate BPE models for the source and target sides. The number of merge operations depended on dataset size: 8K merges for corpora with fewer than 100K lines, 30K for those between 100K and 1M, and 50K for larger ones. This choice is supported by prior work, which shows that smaller vocabularies benefit Transformer models(Kudo, 2018), and that 2Kâ8K merges perform best for low- resource datasets(Adlaon & Marcos, 2024). Our BPE models were used consistently for both vocabulary construction and fine-tuning data tokenization. 4.5 Results and analysis In this section, we present the evaluation results and statistical com- parisons of our fine-tuning setups. We also explore vocabulary overlaps to understand how token and vocabulary choices impact performance and adaptation. 112 4.5.1 Analysis of fine-tuning results Table 4.2 summarizes the performance of all fine-tuning configurations. To better interpret these results, we performed pairwise bootstrap tests on BLEU scores (Section 4.5.1), interpretingp-values as a continuous measure of confidence without enforcing a strict threshold. TER and chrF2 metrics supplemented the analysis to refine the ranking. We ranked configurations using the following criteria: 1. BLEU scores, weighted by the strength of statistical evidence from p-values. 2. TER to resolve ties or unclear BLEU differences. 3.chrF2 as a final tiebreaker if both BLEU and TER were inconclu- sive. Accordingly, the ranking from best to worst is: C6â» C1â» C5â» C2â» C7â» C3â» C4, whereâ»denotes a configuration that performs better or more reliably than the next. Top configuration (C6). ConfigurationC6 uses both BPE and vocabulary exclusively from the in-domain dataE, resulting in the highest BLEU and best TER and chrF2 scores. Statistical tests show thatC6 significantly outperforms all other configurations, confirming the advantage of aligning tokenization and vocabulary strictly with the fine-tuning domain. Strong middle tier (C1,C5,C2). ConfigurationsC1,C5, and C2 achieve similar BLEU scores, with statistical evidence showing no 113 clear superiority among them.C1 (BPE and vocabulary from out- of-domainD) slightly leads numerically, whileC5 (in-domain BPE, combined vocabulary) offers better TER thanC2 (out-of-domain BPE, combined vocabulary), which justifies the order. These results suggest incorporating some in-domain vocabulary or combining datasets can yield competitive results if full in-domain access (for both BPE and vocabulary) is not possible. Lower performing configurations (C7, C3, and C4). Config- urationsC7 andC3 perform moderately but are consistently behind the mid-tier cluster. ConfigurationC4 ranks last, likely because of a mismatch between its in-domain BPE and out-of-domain vocabulary, which impairs tokenization and reduces fine-tuning effectiveness. Practical recommendations. For optimal fine-tuning, use both BPE and vocabulary consistently derived from the in-domain data, as exemplified by configurationC6. When full access to in-domain data or vocabulary is limitedâdue to privacy, proprietary constraints, or resource availabilityâfine-tuning remains possible but may yield reduced adaptation effectiveness. In such cases, configurations like C1 andC2 offer robust alternatives by leveraging available data while balancing performance and practicality. It is important to avoid mixing BPE and vocabulary from mismatched domains, as this often leads to suboptimal tokenization and degraded translation quality. Overall, aligning tokenization and vocabulary with domain data maximizes fine-tuning benefits, but adapting with limited data can still provide meaningful improvements compared to no adaptation. BLEU score statistical comparison We conducted pairwise bootstrap tests on BLEU scores using 1,000 iterations. Table 4.3 shows thep-values for all configuration pairs. 114 Diagonal entries represent self-comparisons. C1 C2 C3 C4 C5 C6 C7 C1â 0.545 0.000 0.000 0.200 0.852 0.102 C2 0.447 â 0.000 0.000 0.172 0.792 0.063 C3 1.000 1.000 â 0.000 0.995 1.000 0.989 C4 1.000 1.000 1.000 â 1.000 1.000 1.000 C5 0.774 0.813 0.006 0.000 â 0.966 0.286 C6 0.149 0.174 0.000 0.000 0.026 â 0.012 C7 0.896 0.929 0.010 0.000 0.701 0.987 â Table 4.3: Pairwise bootstrapp-values for BLEU scores (1,000 iterations). Diagonal entries represent self-comparisons. All values are provided for reference only; no statistical significance threshold is applied. 4.5.2 Training time and CO 2 emissions The training times and estimated CO 2 emissions in Table 4.2 show the resource demands of each fine-tuning setup. The best-performing configuration,C6, took the longestâabout 9.5 hoursâand had a higher carbon footprint, likely because it had to learn many new domain- specific tokens. Other configurations such asC7,C4, andC5 completed training notably faster, around three hours, and had lower CO 2 emissions com- pared toC6. This can be attributed to several factors. Configurations C7 andC5 utilize vocabularies with higher overlap to the baseline tokens, meaning fewer new domain-specific tokens need to be learned, which reduces training complexity and time. On the other hand,C4 exhibits both a vocabulary and BPE mismatch, which limits effective fine-tuning and results in quicker but less effective training. In sum- mary, configurations with less vocabulary adaptation or mismatched tokenization require less training time and energy but tend to yield 115 lower translation quality. Overall, while domain-aligned fine-tuning boosts performance, it can require more time and energyâsomething to consider in real-world applications. 4.5.3 Vocabulary overlap analysis To better understand how vocabulary choice influences fine-tuning, we measured overlap between each configurationâs vocabulary and the baseline WMT vocabulary in terms of token frequency coverage. This approach more accurately reflects the practical impact of commonly used tokens on model performance, as it weights tokens by how often they occur in the data. Table 4.4 summarizes the key statistics: âąSRC/TGT Overlap (%): The percentage of total token fre- quency (i.e., the sum of token counts) in the baseline WMT vocabulary that is also present in the configurationâs vocabulary, calculated for source (English) and target (German) separately. This reflects not just the number of shared tokens, but their practical frequency in baseline data. âąNew Tokens: The number of tokens in the configurationâs vocabulary that do not appear in the baseline vocabulary (after filtering), representing domain-specific or new tokens introduced by the configuration. The results demonstrate that configurations incorporating in-domain vocabulary (e.g.,C6,C5, andC7) introduce a substantial number of new domain-specific tokens, which is associated with their superior BLEU scores and more effective DA. In contrast,C1, relying solely on the baseline vocabulary, achieves complete overlap but lacks critical domain- specific terms, limiting its adaptability. The notably poor performance ofC4 corresponds with its lower vocabulary overlap and the evident 116 Config BPE Model Vocab Src BLEUâSRC Overlap % TGT Overlap % New SRC New TGT C6E BPE E54.882.8477.81 13,022 13,559 C1D BPE D53.610010000 C5E BPE D + E53.183.0477.95 14,289 14,157 C2D BPE D + E53.483.0477.95 11,736 11,804 C7(D + E) BPE D + E53.297.6195.72 14,300 15,077 C3D BPE E51.783.0477.95 11,736 11,804 C4E BPE D46.690.7090.4600 Table 4.4: Vocabulary overlap and BLEU scores per configuration. Con- figurations are listed in order of their overall performance ranking. SRC and TGT overlap percentages indicate the proportion of baseline tokens retained. New Tokens columns count tokens unique to the configurationâs vocabulary. Note: The BPE model in each configuration is applied to both the vocabulary source and the fine- tuning data. mismatch between its BPE model and vocabulary source, underscoring the detrimental impact of inconsistent tokenization strategies. These findings robustly support our practical recommendation: for optimal fine-tuning, vocabulary and BPE should be consistently derived from the same in-domain data. Such alignment ensures richer domain- specific token representation, ultimately leading to enhanced translation accuracy and better overall model performance. 4.5.4 Out-of-domain performance analysis To quantify the impact of different fine-tuning strategies on general- ization, we evaluated all configurations on the original out-of-domain (WMT18,D) test set. Table 4.5 reports BLEU scores for each config- uration, alongside the absolute and relative drop with respect to the pre-trained base model (before fine-tuning). The results illustrate the trade-off introduced by DA: as the model is adapted to the in-domain data, out-of-domain performance drops sub- 117 Config BPE/Vocab Source BLEU Drop Drop (%) Base D BPE ,D (pre-trained)33.9â C2D BPE ,D + E15.1 â18.8 â55.5 C7(D + E) BPE ,D + E15.0 â18.9 â55.8 C3D BPE ,E13.3 â20.6 â60.8 C1D BPE ,D13.1 â20.8 â61.4 C4E BPE ,D10.2 â23.7 â69.9 C6E BPE ,E7.7 â26.2 â77.3 C5E BPE ,D + E7.0 â26.9 â79.4 Table 4.5: Out-of-domain BLEU scores. Performance on the WMT18 (D) test set for all configurations, ranked by smallest drop relative to the pre-trained base model. stantially across all configurations. This degradation is most pronounced when both BPE and vocabulary are derived solely from in-domain data (C5, C6), indicating strong domain specialization at the expense of generalization. For practitioners seeking to balance DA and general translation quality, we recommend hybrid configurations such asC 2 andC 7 , which use either the original BPE with a combined vocabulary or a combined BPE and vocabulary. These setups moderate the drop in out-of-domain BLEU, preserving more general-domain competence while still offering improved DA. 4.6 Limitations Despite the systematic and thorough analysis, we acknowledge several drawbacks and limitations of our work. Addressing these in the future would complement this research and expand the understanding of the impact of data processing on model performance. âąFocus on MT: Our evaluation focused on NMT systems. How- 118 ever, neural language models are also impacted by how the training and fine-tuning data is processed and used, as well as the limita- tions placed on the vocabulary. This is even more pertinent with the progress in LLMs. We did not analyze the performance of LLMs, which is a more complex task, especially in the case of multi-lingual LLMs capable of translation. âąFine-tuning data: Our study focused exclusively on the medical domain for fine-tuning. Future research could consider additional specialized domains to evaluate the generalizability of the findings. âąUse of BPE only: We employed BPE only and did not consider other methods such as SentencePiece(Kudo & Richardson, 2018) or LMVR(Ataman, Negri, Turchi, & Federico, 2017). This was a deliberate choice, as it was up to us which model and method to use during training and fine-tuning. âąHyperparameters: We used the modelâs default hyperparame- ters and did not perform hyperparameter optimization or tuning. This was not necessary as we aimed to compare the impact of the SW algorithms under the same conditions. However, we ac- knowledge that fine-tuning hyperparameters would impact the performance of original and fine-tuned models, and we hypothesize a correlation with how the vocabulary is constructed. 4.7 Conclusion and future work In this work, we presented a systematic analysis of vocabulary and SW tokenization settings for fine-tuning NMT models, using a large out-of- domain corpus (WMT18) and a specialized in-domain medical dataset as a case study. By comparing seven realistic fine-tuning setups that varied in BPE tokenization and vocabulary generation, we identified clear practical guidelines for DA. 119 Our results show that the most effective fine-tuning is achieved when both BPE and vocabulary are derived from the in-domain data, allowing the model to better capture frequent and relevant domain- specific terms. At the same time, we find that maintaining a substantial overlap with the vocabulary of the base model (originally trained on out-of-domain data) is essential for preserving general language coverage and ensuring stable adaptation. The best-performing configurations in our experiments balanced these two needs: they introduced many new, high-frequency in-domain tokens while still retaining a good portion of the base model vocabulary. However, this approach tends to require more computational resources, such as increased training time and higher energy consumption, due to the need for the model to learn and integrate more new tokens. It is important to note that while maximizing DA can significantly boost in-domain performance, it may lead to a substantial drop in out-of-domain translation quality. Hybrid configurations that combine base and domain vocabularies help balance this trade-off, preserving broader translation capabilities while still delivering improved domain performance. If, in addition to the in-domain data, the original out-of-domain data or its BPE/vocabulary are also accessible, combining these resources can help preserve general language coverage and stabilize adaptation. In all cases, our findings highlight the importance of aligning both BPE and vocabulary with the domain of the adaptation data, while retaining overlap with the base modelâs vocabulary to ensure generalization. For future work, we plan to extend our evaluation to other domains and language pairs, and to investigate how these findings generalize to LLMs and multilingual systems. We are also interested in exploring adaptive methods for selecting which tokens to retain or introduce during fine-tuning, with the aim of optimizing both performance and computational efficiency. 120 All datasets, models, and scripts for this chapter are publicly avail- able at https://github.com/JoyeBright/subword-ft-guide. 121 This chapter is based on the following published paper: Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, and Pieter Spronck. 2024. Guiding In-Context Learning of LLMs through Qual- ity Estimation for Machine Translation. In Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 9â20, Chicago, the USA. Improvements have been made to the title, figures, and certain sections to align the content with the broader context of this dissertation. 124 5 In-Context Learning of LLMs through Quality Estimation for Machine Translation T he emergence of LLMs marks a significant shift in how we approach language tasks, including MT. Unlike traditional MT systems, LLMs such as GPT or XGLM possess strong ZS and few-shot capabilities. Instead of retraining, they can adapt to new tasks or domains through prompt-based learning, leveraging a handful of carefully selected ICEs via ICL. This paradigm introduces a more flexible, scalable, and data-efficient alternative to conventional supervised training. However, the quality of output from LLMsâparticularly in MTâis closely tied to the quality of ICEs pro- vided along with the query, i.e., the text to translate. The effectiveness of these ICEs is influenced by various factors, such as the domain of the source text, the order in which the ICEs are presented, the number of these examples, and the prompt templates used. Naturally, selecting the most impactful ICEs depends on understanding how these affect the resulting translation quality.Assessing translation quality remains a complex task. While various automatic and semi-automatic metrics exist to support evaluation, ultimate judgment often relies on human perception. In the context of MT, translation quality is typically mea- sured against reference translationsâsuch as those found in test sets or test suitesâbut such resources are not always readily available. In these cases, QE becomes the only viable alternative. This chapter presents a methodology for ICL that relies on a search algorithm guided by domain-specific QE. Leveraging the XGLM model, our methodology estimates the resulting translation quality without the need for translation references, selecting effective ICEs for MT to maximize translation quality. Our findings demonstrate that this approach not only outperforms existing ICL strategies and conventional fine-tuning but also yields translations that are more closely aligned with human references, ultimately reducing the need for post-editing. 126 5.1 Introduction Pre-trained LLMs quickly gained popularity (and continue to do so) due to their performance on a large set of NLP tasks, including MT(Zhu et al., 2023; Xu, Kim, Sharaf, & Awadalla, 2024). However, the accuracy of their outputs is significantly influenced by the quality of the in-context examples provided to them(Jiang et al., 2020; Alves et al., 2023). 1 If these examples do not align well with the specific task and source domain, the LLMsâ outputs can be inaccurate. 2 Therefore, there is a critical need to develop (better) methods for selecting appropriate examples that match the task and source domain being translated. These methods collectively fall under the umbrella of ICL(J. Liu et al., 2022). Traditionally, creating ICEs for MT involves either random selec- tion(Sia & Duh, 2023) or using a strategy such as maximizing an evaluation metric like BLEU, to choose examples that improve the metric(Agrawal et al., 2023). The former was initially used for its simplicity and ease of implementation. However, relying on randomness can lead to inconsistent results and pose significant computational costs(Lu, Bartolo, Moore, Riedel, & Stenetorp, 2022). Recent SOTA ICL approaches focus on retrieving training examples that are closely relevant to the context of source sentences of test sets using unsuper- vised retrievers, such as BM25(Robertson & Zaragoza, 2009). Studies have also shown that a range of factors, such as order(Lu et al., 2022), template(Jiang et al., 2020), domain, and number of ICEs, significantly impact the output quality(Agrawal et al., 2023; Raunak, Menezes, & Awadalla, 2023). Naturally, the most effective ICEs for a given source text are the 1 For simplicity, we sometimes refer to it as âexample(s)â throughout this paper. 2 This has been observed in models such as GPT-J and LLaMA2, which rely heavily on the quality and relevance of in-context examples to generate accurate results (Peng et al., 2024). 127 ones that would maximize the resulting translation quality, which we typically judge based on translation references or human assessment. However, when such are not available, one needs to resort to QE tech- niques. In MT, QE has become a standard approach for evaluating an MT systemâs output without relying on reference translations(Blain et al., 2023). Recently, Lee (2020), Ye and Li (2023), and our previ- ous chapter(Sharami, Shterionov, Blain, et al., 2023) (see Chapter 3) showed the effectiveness of domain-specific QE when it comes to domain- specific MT (in contrast to the ineffectiveness of generic QE). QE models are particularly valuable in scenarios where reference translations are unavailable, costly to have, or domain-specific, making them ideal for real-world deployment and feedback loops. Building on this and to address the aforementioned challenges, our work proposes to leverage domain-specific QE to assist in the selection of ICEs, with the goal of determining the suboptimal number and combination of ICEs to maxi- mize MT quality, all without reference translations. As QE would assess the impact of different ICE combinations and sequences, we hypothesize that this integration has the potential to not only improve translation performance but also reduce processing time, as QE could result in smaller sets of ICEs, which would reduce the inference times(Petrov, Malfa, Torr, & Bibi, 2023). This is particularly crucial considering the limited number of ICEs that can be fed into LLMs due to their constrained input capacity(Agrawal et al., 2023). In this chapter, we present our work on selecting ICEs on a per-source basis. Specifically, we aim to answer the following RQ: How effective are domain-specific QE models in determining ICEs for translation tasks in an LLM? Our proposed ICL methodology for MT combines an unsupervised retriever to select ICEs with QE to assess their impact on the translation quality, determining which ICE combination to include. Instead of feeding all selected examples, we only select examples whose QE points to maximizing the LLM translation quality. 128 Our findings on German-English translations demonstrate that our proposed approach outperforms the current SOTA ICL strategies, including R-BM25(Agrawal et al., 2023)âa retrieval-based method that ranks ICEs using BM25 with a reranking stepâas well as a fine-tuned mBART-50 model(Tang et al., 2020). Specifically, our method yields up to 8.3 BLEU and 1.35 COMET points higher than mBART-50, and achieves statistically significant improvements over baseline retrieval methods while maintaining a lower computational and environmental footprint. Furthermore, we show that the patience- based search guided by domain-specific QE not only identifies more effective ICE combinations but also produces translations that are more length-aligned with human references, thereby reducing the need for post-editing. Unlike fine-tuning, which requires high computational cost and CO 2 emissions, our method is highly reusable and efficient for inference-time adaptation, making it both practical and sustainable. We outline our methodology in Section 5.2 and present empirical evaluation in Section 5.3 (Experiments Setup) and Section 5.4 (Results). Further analysis is presented in Section 5.5. We review related work in Section 5.6 and conclude in Section 5.7 with a summary and suggestions for future research. 5.2In-context learning using quality estimation for ma- chine translation To utilize LLMs for effective MT, as noted in Section 5.1, what is needed is a set of examples to provide the context (and thus guide or steer the LLM toward a correct, context-specific translation) â that is, a set of ICEs â and what is further important is the number of ICEs and their combination. 3 Ultimately, what is required is that the 3 The question of the order of examples is not specifically discussed in this chapter but is left for future work. 129 ICEs provide context that is neither too specific nor too broad and can effectively boost the translation. Our goal with this work is to develop a methodology that optimizes both these aspects in order to deliver high-quality MT. Our methodology for identifying effective ICEs involves two key components: (1) an unsupervised retriever that locates examples closely related to the sentence to be translated and (2) a search algorithm that uses QE to select a combination of examples that leads to the improvement of translation quality, i.e., aiming to maximize the BLEU score. 5.2.1 Unsupervised retriever ranking Building on the idea of selecting relevant examples introduced in Chap- ter 2, we continue to focus on identifying training pairs that are most useful for a given translation task. Instead of relying on dense semantic representations, here we turn to a simpler yet effective method for rank- ing: the BM25 algorithm(Trotman, Puurula, & Burgess, 2014). BM25 has been widely used in information retrieval and has shown strong performance in similar settings, particularly for finding relevant ICEs in unsupervised scenarios(Agrawal et al., 2023). It works by scoring how well each candidate sentence matches the input queryâthe sentence to be translatedâbased on overlapping words, while also adjusting for sentence length. This helps us surface examples that are more likely to be useful, without requiring additional model training or complex representations. Subsequently, we select the topKsentence pairs ranked by the algorithm, whereKis a hyperparameter that controls the number of pairs to be fed into the search algorithm. 130 5.2.2 Search algorithm coupled with QE Our search algorithm comprises three main phases: Selection, Transla- tion, and Estimation. During the Selection phase, the algorithm selects the highest-ranked training 4 example from the initial ICEs provided by the unsupervised retriever ranking method (out ofKICEs). This selected example is then concatenated with the previously selected ICEs. In the first iteration, no ICEs have been selected before. In the Translation phase, the model receives both the selected ICEs and the test sentence and it generates a translation only for the test sentence. In the Estimation phase, the LLM output (translated text) and the original source text are inputted into the domain-specific QE model to estimate the quality of the translation. Our proposed methodology relies on sentence-level QE. Next, the selected ICE, together with its estimated quality and the LLM translation output, are appended to an intermediate list. To track the highest quality obtained thus far, the algorithm sorts the list in descending order based on the estimated quality. To avoid duplication, the selected ICE is removed before the next iteration. This iterative process continues until the best-estimated translation quality no longer improves within the specified patience threshold. Alternatively, the process terminates once all K ICEs have been selected. This methodology allows for the systematic selection of ICEs that improve translation quality compared to previous ICL methodologies while efficiently managing the computational resources required for the search process. This efficiency is achieved by integrating early stopping conditions with predetermined patience. Notably, we do not explore permutations of initial ICEs, as doing so would require a large number of attempts, leading to high computational costs during the search process. A pseudocode outlining the search methodology can be found 4 We use the term âtrainingâ because these examples are drawn from the training set in our experiments. However, they are not used to train any models. 131 in Algorithm 1. The phases of translating a source text of a test set using our methodology are depicted in Figure 5.1. 5.3 Experiments setup We conducted four main experiments to test the effectiveness of our methodology. Three of these experiments compare our methodology to existing ICL ones in different settings, or Modes. The fourth experiment compares our methodology to a fine-tuned mBART-50, aiming to assess which method is preferred (with respect to obtaining better translations). It is important to note that we do not fine-tune the LLM. Details regarding the QE model used in our experiments are provided in Sec- tion 5.3.3. The QE model we use follows the same general training setup as described in Chapter 3, but with a few adjustments to better align with the goals of this experiment. 5.3.1 Search algorithm We conducted experiments using the search algorithm (see Section 5.2.2) employed in three different ways (modes): Mode 1: This mode uses QE with ICEs ordered by BM25 to assess the effectiveness of combining BM25 and QE in the proposed ICL methodology. Mode 2: This mode investigates the impact of ordering ICEs accord- ing to uni-gram overlap, alongside QE, on the proposed methodology. Given the success of ordering ICEs based on their n-gram overlap match with the source, as demonstrated in(Agrawal et al., 2023), we assess how this ordering, based on ICEsâ uni-gram overlap with the source text, influences the translation quality. This involves reordering ICEs 132 1: function Search(...) 2:tempâ [(ââ, 0.0, ââ)] 3:promptâ ââ 4:itrâ 0 5:best_qe_scoreâ 0.0 6:patience_counterâ 0 7:while itr < iteration and patience_counter < early_stop_patience do 8:available_Promptsâ GenerateAvailablePrompts(...)â· Initial ICEs 9:if available_prompts is not empty then 10:selected_prompt_indexâ itr mod kâ· Phase 1: Selection 11:selected_promptâ available_prompts[selected_prompt_index] 12:promptâ ConstructFullPrompt(...)(see 5.3.6) 13:input_ids[0]â EncodePrompt(...)â· Phase 2: Translation 14:if length(input_ids) > LLM_max_length then 15:return temp 16:end if 17:outputâ GenerateOutput(...) 18:final_outputâ DecodeOutput(...) 19:qe_inputâ PrepareQEInp.(source, final_output) â· Phase 3: Estimation 20:qe_scoreâ EstimateQuality(qe_input, model_QE) 21:temp.append((prompt, current_qe_score, final_output)) 22:if current_bleu_scoreâ„ 100 then 23:return temp 24:end if 25:tempâ SortTemp(...) 26:if qe_score†best_qe_score then 27:patience_counterâ patience_counter + 1 28:else 29:patience_counterâ 0 30:end if 31:best_qe_scoreâ temp[0][1] 32:end if 33:itrâ itr + 1 34:end while 35:return temp 36: end function Algorithm 1: Pseudocode outlining the proposed Search Algorithm. Each phase of the methodology is annotated alongside the relevant code. Function arguments are omitted for simplicity. The first element of the returning list (temp) includes the selected prompt, its associated QE score, and the translated text. 133 Training data Unsupervised Retriever Ranking Test data Netbios deaktivieren Schnittstelle deaktivieren Disable interface 1 AufklappmenĂŒ deaktivieren Disable This Popup 2 . . . Deaktivieren;WortergĂ€nzung disabling;word completion, K Initial ICEs BM25 Schnittstelle deaktivieren = Disable interface </s> Netbios deaktivieren = Selection XGLM (Encoding and Translation) Translation QE Estimation Netbios deaktivieren Quality Label Search Algorithm Figure 5.1: Overview illustration showing an iteration of our proposed methodology. All selected ICEs are combined into a single prompt together with the test sentence, and the model generates a translation only for the test input. The ICEs are not translated in this process; instead, they serve as contextual examples that prime the model for better translation. according to their n-gram overlap, which is calculated using the NLTK word tokenizer. Higher overlap matches prioritize ICEs in the list and feed them into LLMs earlier. Mode 3: Instead of relying on QE, in this mode we compute the BLEU score directly on the test set. This mode is not realistic because the method has access to the gold referencesâwhich is never available at inference time in MT evaluationâand can therefore select ICEs that explicitly maximize translation quality; in other words, the labels are exposed. Nonetheless, this mode serves as an upper-bound scenario and is used only as a best-case comparison point. 5.3.2 Early stopping conditions The search algorithm generates up to 16 candidates. In each mode, we conducted experiments using three early stopping patience values (3, 8, and 16), determining the maximum number of ICEs (K) generated. 134 We included Patience 16, which implies no early stopping, to evaluate the modelâs performance with the maximum ICEs. Additionally, the search process halts if the estimated QE label reaches or exceeds 100, preventing further evaluations. 5.3.3 Quality estimation Following(Ranasinghe et al., 2020; Lee, 2020) and our earlier approach presented in Chapter 3, we develop a domain-specific QE model. First, we trained a QE model using out-of-domain (OOD) data (detailed in the paragraph after next) to ensure generalizability; and second, we fine-tuned the model using the training set described in Section 5.3.4 to provide domain-specific QE model and address domain mismatch, which is critical (Koehn & Knowles, 2017). In our experiments, we used BLEU as the quality label because our study focused on translation performance rather than post-editing effort, which is typically evaluated using (H)TER(Specia & Farzindar, 2010). We employed the âMonoTransQuestâ architecture from the TransQuest framework(Ranasinghe et al., 2020), known for its success in prior QE studies. However, instead of employing softmax computation, we directly utilized logits to estimate the quality labels. This strategy saves computation time, as softmax computation can be resource- intensive (Ruder, 2016). We utilized the German-English âEuroPat v3â dataset, accessed through Opus(Tiedemann, 2012), to develop our generic QE model. We chose this dataset because it provides ample data samples (around 20M), ensuring broad coverage of vocabularyâa critical aspect in developing generic models. However, as MT datasets like EuroPat typically consist of pairs of source and translated text, it was necessary to synthetically create post-editing text (since the QE data creation process requires a triplet input: source text, machine-translated text, and post-edited text). To 135 accomplish this, we used a pre-trained multilingual MT model, namely mBART-50 that supported the language pair used in our experiment. We translated 1M randomly chosen source texts from EuroPat. After- ward, the resulting translations were considered as machine-translated text, with the corresponding reference translations acting as post-edited text. Using SacreBLEU, we calculated the BLEU score, comparing the translated text with its corresponding post-edited text. This approach, which has been demonstrated to be effective in QE(Negri et al., 2018; Lee, 2020; Sharami, Shterionov, Blain, et al., 2023), enabled us to use the source and (machine-) translated text as input and the BLEU score as the target value for the QE model. For building domain-specific QE, we utilized the training set detailed in Section 5.3.4 and applied the aforementioned approach to synthetically generate BLEU scores for the entire dataset. 5.3.4 Dataset and evaluation metrics We used a dataset comprising German-to-English translation pairs within the IT domain, sourced from(Aharoni & Goldberg, 2020). This dataset was chosen due to the challenges that MT systems and LLMs face when translating out-of-domain contexts, particularly in specialized fields, as noted in previous studies(Koehn & Knowles, 2017; Agrawal et al., 2023). The specialized and constrained nature of the IT domain provided an ideal setting for evaluating our methodologyâs performance under these conditions. The dataset utilized in this study consisted of approximately 222k training sentences, 2k development sentences, and 2k test sentences. To assess the translation effectiveness of the models, we employed metrics such as BLEU from SacreBLEU(Post, 2018) and COMET(Rei et al., 2020). It is noteworthy that we used the development set during QE model 136 fine-tuning to avoid any exposure to the test set. This ensured that the test data remained fully unseen and could serve as a reliable benchmark for evaluating our methodâs generalization performance. 5.3.5 Number of ICEs We use between 1 and 16 ICEs. These may originate either from a random approach or from an advanced (guided) selection, i.e., the ICEs selection process is not random but follows specific rules instead of sam- pling examples arbitrarily from a pool. In line with prior work(Agrawal et al., 2023), we adopt the symbolspandqto denote the number of task-level and example-specific prompts, respectively. However, our method does not distinguish between these two levels: all selected ICEs are combined into a single prompt together with the test sentence. Therefore, in our experimentspandqrefer to the same quantityâthe number of ICEs included in the final prompt. We retain this nota- tion only for comparability with previous studies. While we cap the maximum atq= 16, any value between 1 and 16 is feasible in our setup. 5.3.6 Multilingual large language model For our experiments and hypothesis validation, we used XGLM(X. V. Lin et al., 2022). This choice stems from the outstanding performance of the model in the MT field. This also ensures a fair comparison of our proposed methodology with previous research, such as(Agrawal et al., 2023), which introduced SOTA approaches in ICL for MT. We used the 7.5 billion-parameter XGLM implementation and to- kenizer by Hugging Face 5 , consistent with previous research. We em- ployed a template from(X. V. Lin et al., 2022) to maximize translation performance.< /s >serves as the ICE separator in this template. 5 https://huggingface.co/docs/transformers/model_doc/xglm 137 âBLANKâ denotes an empty string within the template. source text 1 = target text 1 < /s > source text 2 = target text 2 < /s > ... = ... < /s > source text n = BLANK 5.3.7 Compared systems We conducted a comparative analysis with methods from previous studies; random and task-level sampling, BM25, R-BM25, and fine- tuned mBART-50. Random: We conducted three random trials, each based on a param- eterp, which defines the number of ICEs to include in the prompt. For each trial, we randomly sampledpintegers between 1 and the size of the training set. These integers were used to select the corresponding translation pairs from the training data. For example, ifp= 3 and the randomly generated numbers are 1, 10, and 100, we select those three examples from the training set. To create the prompt 6 , in addition to the ICEs, we need the source side intended for translation. We use the source from the development set, unlike more advanced methods in ICL, where the source text from the test set is typically employed. The reason for selecting the development set over the test set in this approach is that development sets are generally from the same distribu- tion, domain, and context as the test set. This similarity increases the likelihood that the examples in the development set will better match the content and context of the test set, thereby enhancing the relevance and effectiveness of the prompts. The generated prompt is inputted into the LLM for translation. Then, the BLEU score of the development set is computed. The random 6 In the literature, the term âpromptâ is often used interchangeably with âICEâ 138 number that produces the highest score among the trials is selected, and the ICEs linked to this number are concatenated with the test setâs source text. Task-level: Based on the work of Agrawal et al. (2023), the task- level approach 7 is similar to the random approach but differs in the number of trials used. We employ 100 trials for the task-level approach, a significantly higher number than the random approach. The reason for using more trials is to generate a greater variety of ICEs, aiming to enhance the performance of LLMs in the translation task. However, this results in longer execution times compared to the random approach. BM25: Using the Moses Tokenizer(Koehn et al., 2007), we first tokenize the training setâs source samples. The BM25 model is then created over the tokenized corpus, where it computes term frequency and inverse document frequency statistics necessary for scoring the lexical relevance of test queries. 8 Next, the test set is tokenized using the tokenized source. The algorithm then searches for similar training samples based on BM25 criteria, selecting the topqmatches for the model. This methodology utilizes the test set as opposed to random and task-level approaches using the development set. The reason is that BM25 retrieves ICEs by ranking examples based on lexical similarity to each specific source sentence, eliminating the need for repeated trials or development set proxies. Re-rank BM25 (R-BM25): BM25 aims to find translation examples with the highest n-gram overlap with the source sentence(Luo et al., 7 While the term âtask-levelâ is used, it is important to note that this approach uses random sampling to generate a diverse set of ICEs, aiming to enhance the performance of LLMs in translation tasks. 8 https://github.com/dorianbrown/rank_bm25 139 2023). However, since retrieved examples score independently, top matches may lack coverage of all source n-grams. This poses an issue in ICL due to LLM input size limitations. To address this, Agrawal et al. (2023) proposed R-BM25, a recall-based variant of BM25 that promotes broader n-gram coverage. It works by extracting all word n-grams from the test sentence and from each BM25-retrieved candidate. At each step, the candidate that contributes the most uncovered n-grams is selected and added to the prompt. Once an example is selected, the n-grams it covers are given less weight in future iterationsâessentially telling the algorithm: âthis part is already covered, focus on what is missing." This iterative process continues until a desired level of n-gram coverage is achieved. Fine-tuning mBART-50: Different ICL methodologies, including our own, are assessed in comparison to the process of fine-tuning a pre- trained multilingual MT model, specifically mBART-50. The selection of mBART-50 is based on its alignment with the language specifications of the experiment and its proven track record of achieving success in MT tasks through the utilization of PLMs(Yuan et al., 2022; N.-Q. Pham et al., 2022). The fine-tuning of mBART-50 is carried out using the training data outlined in Section 5.3.4. 5.3.8 Computational costs We monitored and reported the computational costs of the models utilized in our experiments using the carbontracker package. 9 This involved calculating the carbon footprint (CO 2 eq) emissions, time to prediction (TTP), and electricity consumption (kWh) associated with our experiments. Our experiments were conducted using NVIDIA A40 GPUs. 9 https://github.com/lfwa/carbontracker 140 The script for running our experiments is publicly available at https://github.com/JoyeBright/ICLviaQE. 5.4 Experiments results This section presents the results of our experiments. We conducted a statistical analysis test (t-test) to determine if our models significantly outperformed the compared systems. 5.4.1 Evaluation metrics Among prior work, the results in Table 5.1 show that R-BM25 with 16 ICEs outperforms all other methods. Because of this strong performance, we used it as our baseline for further comparisons. It is notable that there is a positive correlation between the number of examples and evaluation scores (consistent through all methodsâ Random, Task-level, BM25, and R-BM25), although at the expense of prediction time (i.e., TTP). Employing 16 examples significantly improved performance compared to using only one example in the random approach. Analyzing the performance of our methods in Mode 1 (referred to as âM 1â, with P = 3, 8, or 16 in Table 5.1), we observe that our proposed methodology with different patience thresholds consistently outperforms all previous methods, including the baseline. This trend holds for both the COMET and BLEU metrics across all the methods. Specifically, our method exhibits a minimum improvement of 0.52 points in the BLEU score (from 45.20 to 45.72) with patience threshold of 3 and a maximum improvement of 1.58 points in the BLEU score (from 45.20 to 46.78) with a patience threshold of 16 compared to R-BM25 with 16 examples. Consequently, our methods in Mode 1 are ranked based on their performance, with patience 3 being the least effective model, followed 141 Methodp +q BLEU COMET TTP CO2 GPU Random1 + 0 10.380.6895 01:51 00.13 00.39 Random16 + 0 31.650.7844 02:20 00.19 00.58 Task-level1 + 0 29.170.7586 62:50 09.83 29.10 Task-level 16 + 0 32.880.8083 78:30 12.80 35.91 BM250 + 1 39.240.7833 00:56 00.06 00.19 BM250 + 16 44.500.8120 00:58 00.07 00.19 R-BM250 + 1 40.880.7990 01:01 00.06 00.21 R-BM250 + 16 45.20 0.8218 01:04 00.07 00.21 M 1, P = 3 0 + 16 45.720.8395 01:49 00.22 00.67 M 1, P = 8 0 + 16 46.43 0.8501 03:48 00.50 01.51 M 1, P = 16 0 + 16 46.78 0.8554 05:11 00.68 02.05 M 2, P = 3 0 + 16 46.050.8400 01:30 00.21 00.64 M 2, P = 8 0 + 16 46.59 0.8518 03:52 00.51 01.52 M 2, P = 16 0 + 16 46.52 0.8564 05:00 00.66 02.01 M 3, P = 3 0 + 16 49.890.8532 01:36 00.22 00.66 M 3, P = 8 0 + 16 52.63 0.8725 03:14 00.45 01.40 M 3, P = 16 0 + 16 53.50 0.8791 04:08 00.55 01.65 mBART-50N/A 42.760.8659 11:20 01.88 04.82 Table 5.1: Method Performance in BLEU and COMET Scores.M1 to 3 denotes Mode 1 to 3;Pis the patience value. For consistency with prior work,pand qboth refer to the number of ICEs included in the final prompt (see Section 5.3.5). âN/Aâ (not applicable) indicates that fine-tuning does not use ICEs. Bold font represents the highest translation performance. Two numbers are in bold if they are statistically similar (t-test,p_value= 0.05). TTP is in (h:m), CO2 in (kg), and GPU in (kWh). by patience 8, and finally patience 16, representing the most effective method. This ranking indicates that increasing the patience threshold can significantly enhance the translation performance. However, the improvement with patience 16 is not statistically significant compared to patience 8, suggesting that more ICEs do not necessarily enhance translation performance. Similarly, while more substantial contextual improvement (as indicated by the COMET) is observed at the maximum 142 patience threshold (16), it is not statistically significant compared to patience 8. The Mode 2 results demonstrate that using any of the three patience thresholds leads to better results compared to the methods in the literature. However, this improvement is not statistically significant when compared to the respective experiments in Mode 1. This suggests that ordering the examples according to n-gram (unigram) similarity does not enhance the translation performance in our methodology. While ordering ICEs by unigram overlap (Mode 2) does not improve results, the initial BM25 retrieval naturally returns examples with rela- tively high lexical similarity. However, our QE-guided search does not optimize for n-gram overlap: it ranks combinations only by estimated translation quality. Thus, the gains of our method cannot be attributed solely to high n-gram similarity. When it comes to Mode 3, we should stress that this is an unrealistic scenario, but used as the highest bound. The results indicate that with a patience of 3, the BLEU score is 4.17 points lower (49.89-45.72). With a patience of 8, this gap increases to 6.2 points (52.63-46.43), and with a patience of 16, it widens further to 6.72 points (53.50-46.78). These differences arise from the QE model estimations in our experiment compared to the scenario where reference labels are available to the search algorithm. 5.4.2 Time to prediction (TTP) Among the methods examined, task-level execution required the most time, with approximately 62 hours for one example and 78 hours for 16 examples, primarily due to the large number of trials involved. Our method (Mode 1) with a patience value of 16 is relatively time-intensive, taking approximately 5 hours, while a patience value of 3 is comparable to the baseline method, differing by only around 50 minutes. Mode 2 is nearly equivalent to Mode 1 in terms of TTP, whereas Mode 3, where 143 MetricGeneric Model Specific Model Training Time (h:m)05:5506:54 CO2 Emissions (kg)1.411.46 Electricity Consumption (kWh)3.633.76 Table 5.2: Training Time, CO2 Emissions, and Electricity Consumption for QE Models the reference labels are accessed, requires less time than Modes 1 and 2. In addition, the search algorithm incorporates a termination condition, and given that QE estimation rarely triggers this condition, numerous ICEs are left unattempted, resulting in significant time savings. It is also important to note the time required to train the QE models used in the prediction process. As shown in Table 5.2, the training time for the generic QE model is +/â5 hours and 55 minutes, while the specific QE model takes about +/â6 hours and 54 minutes. Although these training times are significant, it is crucial to recognize that QE models, similar to MT models, can be reused for the same language pair and domain, thereby amortizing the initial training cost over multiple predictions. The last row of Table 5.1 shows the scores of the translations obtained with the mBART-50 model fine-tuned on the same training set as in ICL. Despite mBART-50 being tailored for MT across 50 languages, it did not outperform the R-BM25 method with 16 examples (best from the existing methods); it was better only than Random, Task- level, BM25, and R-BM25, each with only 1 example. However, when considering translation performance from a contextual perspective, the COMET results indicate that fine-tuning mBART-50 leads to superior performance compared with lexical overlap. Nevertheless, fine-tuning took significantly longer than identifying ICEs and obtaining inferences from the XGLM. 144 Compared to our methodology, especially when considering the least performing method (M 1, P = 3), fine-tuning is significantly worseâ 6.47% (42.76 to 45.72). This highlights the substantial benefits of ICL compared to fine-tuning. Nonetheless, it is noteworthy that various factors might contribute to this observation: e.g., the modelâs size might be a critical factor, especially during deployment, where larger models like XGLM could pose challenges. 10 To contextualize this comparison, it is important to note that fine- tuning and ICL are typically performed using different model classes. Encoderâdecoder architectures such as mBART-50 are the standard choice for supervised MT fine-tuning, whereas decoder-only models such as XGLM are commonly used for ICL. Our evaluation therefore reflects the typical use of each paradigm rather than a controlled architecture ablation. While architectural and size differences may influence absolute scores, this does not undermine the validity of our comparison, whose purpose is to assess the practical trade-offs between fine-tuning and ICL in MT workflows. 5.5 Analysis Output analysis Pre-trained LLMs often exhibit over-generation, i.e., the generation of a larger number of tokens than expected by a human (in comparison to a reference), necessitating extensive post- processing (e.g., post-editing)(Bawden & Yvon, 2023). Figure 5.2 shows the tokenized output lengths (translations) for our model (Mode 1, patience 8), 11 alongside the R-BM25 with 16 examples. The analysis shows that the length distributions for both models align with the reference distribution, suggesting that the models do not over-generate. 10 However, this does not undermine the generalizability of our conclusions; rather, it points to the need for balancing performance with feasibility when selecting models for real-world applications. 11 Our other models in Mode 1 exhibited similar distributions. 145 To quantitatively compare these distributions to the reference, we employed the Kolmogorov-Smirnov (KS) test(Kolmogorov, 1933). 12 The results indicate that for R-BM25 versus the reference, the KS statis- tic is relatively high (0.0749), reflecting a significant difference between the translation lengths of R-BM25 and the reference distribution. The extremely low p-value (2.39Ă10 â5 ) further confirms this significant discrepancy. Conversely, for Mode 1 with P=8 versus the reference, the KS statistic is considerably lower (0.0232), indicating a much smaller difference in translation lengths. The higher p-value (0.6451) suggests no significant difference, implying that the distribution of Mode 1, P=8 is similar to the reference distribution. These findings suggest that our proposed methodology could yield translations closer in length to the reference, potentially reducing the need for labor-intensive post-processing efforts and enhancing computa- tional efficiency. ICE number analysis The number of selected ICEs holds a signifi- cant importance within the ICL algorithm, as it directly impacts the token processing time and the capacity of LLMs to handle additional ICE instances. We analyzed the number of ICEs that our algorithm selected across all three modes. The results (Table 5.3) show that the minimum number of ICEs selected is 1, while the maximum is 12 for Mode 1, 16 for Mode 2, and 16 for Mode 3. The average (mean) number of ICEs is found to be lowest in Mode 3 and highest in Mode 1. In addition, Mode 2 results in a reduction in the number of ICEs within our proposed algorithm. The notably lower average number of ICE instances in Mode 3 can be attributed to its access to the test set, allowing for the selection of optimal ICE combinations based on test set performance and activating an early stopping condition if the 12 The KS test is non-parametric, making it suitable for comparing translation length distributions without assuming normalityâimportant since model outputs may not follow standard distributions. 146 Figure 5.2: Tokenized Translation Lengths comparison between R-BM25, our Mode 1, P=8, and the reference. âKSâ denotes the Kolmogorov-Smirnov test, with the p-value indicating significance. Mode MinMeanMax #1[1, 1, 1] [2.25, 3.76, 4.84] [12, 16, 16] #2[1, 1, 1] [2.20, 3.70, 4.74] [12, 16, 16] #3[1, 1, 1] [2.15, 3.47, 4.47] [12, 16, 16] Table 5.3: Number of ICEs selected for each mode at different patience thresholds. Labels [x, y, z] correspond to patience values 3, 8, and 16. score exceeds 100. Contrarily, while Mode 1 exhibits similarities to Mode 3, its relatively higher average can be linked to inaccuracies in QE estimation. Moreover, our analysis shows that QE estimations rarely reach a score of 100, thus rendering the early stopping condition inactive. CO 2 emissions Our analysis reveals that using XGLM for translation yields lower CO 2 emissions than fine-tuning mBART-50, making it a 147 more environmentally sustainable choice. In Mode 1 of our proposed methodology, with patience 16, XGLM emitted 0.68 KG of CO 2 , while fine-tuning mBART-50 emitted 1.88 KG. Interestingly, the task-level method with 16 ICEs emitted the highest amount of CO 2 , totaling 12.80 KG. Our approach produces CO 2 emissions higher than R-BM25. Illustrative example. Table 5.4 shows an example from our evalu- ation, including the selected ICEs, the modelâs translation, the reference label, and evaluation scores (BLEU and QE). ICEs: Die Sockets, die im except Array aufgelistet sind, werden auf Ausnahmen ĂŒberwacht. = The sockets listed in the except array will be watched for exceptions.< /s >Geben Sie den Namen der Variablen ein, deren Wert ĂŒberwacht werden soll. = Enter the name of the variable whose value is to be monitored. < /s >Nur erlaubt bei Sockets fĂŒr lokale Displays und den globalen Socket. = Permitted only on sockets of local displays and the global socket.< /s >Legt fest, ob Scandaten-Information, die in den MPEG2-Videoströmen enthalten sind, aktualisiert werden sollen. = This controls whether to update the scan data information contained in the MPEG-2 video streams.< /s >Die Sockets, die im write Array aufgelistet sind, werden daraufhin ĂŒberwacht, ob ein Schreibvorgang den Socket blockiert. = Translation: The sockets listed in the write array will be watched for whether a write operation blocks the socket. Reference Label: The sockets listed in the write array will be watched to see if a write will not block. QE 67.59, BLEU score (using reference label): 52.89 Table 5.4: An example of selected ICEs for a source text, its corresponding translation, reference label, and QE estimation compared to the BLEU score computed based on the reference label. 5.6 Related work ICL for MT. ICL 13 represents a relatively new paradigm in natural language understanding. Unlike traditional fine-tuning approaches, where a PLM undergoes parameter updates using a specific dataset, 13 Also referred to as the prompt retrieval method 148 ICL typically directly generates the output without any modification to its parameters(Radford et al., 2019; Brown et al., 2020). This is achieved by solely providing the model with a few examples, known as ICEs, which prime the PLM to enhance its performance for the given task (Jiang et al., 2020). As shown by Vilar et al. (2023), the quality of translation is directly proportionate to the quality of ICEs, where quality refers to ICEs being relevant, clear, accurate, and domain-specific. However, considering all ICEs during processing is computationally demanding(Alves et al., 2023). Hence, it is crucial to selectively choose ICEs that can enhance MT quality. N. Goyal et al. (2022) conducted a study where ICEs were randomly selected. Despite finding that this random selection of ICEs resulted in good translation performance, the neglect of their order, which was identified as important(J. Liu et al., 2022; Lu et al., 2022), was a drawback in this approach. To address this, methodologies such as(Agrawal et al., 2023) introduced a re-ranking technique (R- BM25). However, their methodology relies solely on n-grams to order examples, which can enhance fluency but may overlook contextual factors. In our approach, we investigated the unigram order of initial ICEs provided by the BM25 algorithm. We leave the in-depth analysis of ICE order for future work. Additionally, Kumar, Puduppully, Dabre, and Kunchukuttan (2023) highlighted the advantages of using multiple features in ICE selection to improve translation quality, while our QE- based approach simplifies ICE selection without the need to generate additional features, ensuring efficiency. QE in MT evaluation. QE models offer a quick solution to the assessment of the overall usefulness of translated text. These models do not rely on reference translations, thereby reducing the human effort required for quality evaluation(Tamchyna, 2021; Murgolo et al., 2022; Zerva et al., 2022; Blain et al., 2023). Similar to MT models, 149 previous studies highlight the importance of domain-specific QE for accurately estimating translation quality across diverse domains(Lee, 2020; Sharami, Shterionov, Blain, et al., 2023). This is why, in our work, we employed a domain-specific QE model instead of a generic one to enhance the selection of ICEs. Integrating QE into ICL offers significant, yet largely unexplored, potential. QE can also better capture out-of-domain gender and word- sense-disambiguation errors(Dinh & Niehues, 2023). Additionally, integrating QE can mitigate reference bias, a significant challenge in accurately estimating the output quality of LLMs(T. Goyal, Li, & Durrett, 2023; Raunak, Menezes, Post, & Hassan, 2023). The introduction of COMET-QE(Raunak, Menezes, & Awadalla, 2023) exemplifies this pursuit, providing a metric tailored to evaluate the quality of perturbed prompts provided to GPT-3(Brown et al., 2020), aiming to mitigate reference bias. While in our approach, we employ domain-specific QE to guide the selection of ICEs, this underscores the potential of QE in refining LLM inputs (i.e., ICEs). 5.7 Conclusion We propose a novel ICL methodology for enhancing the translation capabilities of large language models (LLMs) while optimizing com- putational resources. Our approach leverages domain-specific QE to guide in-context selection, particularly focusing on determining the suboptimal number and the combinations of ICEs. This novel strategy moves beyond the conventional reliance solely on translation references from development sets seen in prior methods. We evaluated our approach across different modes and early stopping patience values on the German-to-English IT dataset. Our experiments consistently showed the superior performance of our methodology, sur- passing all prior works across both BLEU and COMET metrics. Our method consistently improves BLEU scores, although this comes at the 150 cost of increased computation time. We also investigated the impact of ordering the ICEs based on their unigram overlap with the source text and found it to be not statistically significant. Furthermore, our experiments highlighted the value of ICL compared to fine-tuning a PLM, namely mBART-50. We also highlighted that our method leads to less carbon emissions while achieving better translation performance. In the future, we would like to conduct further research on the impact of our proposed methodology across different datasets and LLMs. Also, we aim to explore alternative metrics beyond BLEU to tailor the selection process, as well as additional features such as bigram, type/token ratio, and length when ordering examples prior to their input into LLMs. 151 6 Discussion Despite the tremendous progress in MT and the remarkable evolution of LLMs, these technologies are not yet capable of meeting the demands of domain-specific translation out of the box. While LLMs have become more fluent and context-aware, their ability to operate effectively in specialized domains remains limited. This shortcoming primarily stems from their reliance on generic-domain data, which lacks the specificity required to handle the unique terminology, formal registers, and con- textual nuances of fields like medicine, law, or information technology. The implications of such domain mismatch are not merely academic; in real-world applications, mistranslations can carry serious consequences, from clinical misunderstandings to contractual misinterpretations. Im- portantly, this issue is not confined to LLMs alone: in MT more broadly, relying on ever-larger generic corpora often improves overall translation quality but also risks embedding irrelevant or domain-misaligned pat- terns, thereby limiting reliability in specialized contexts. The challenge, then, is not simply about increasing data volume but ensuring domain relevance and accuracy. This inaccuracy in MT systems is primarily driven by a phenomenon known as domain mismatch. Domain mismatch arises when the data used to train a model comes from a different domain than the one in which it is used. This mismatch often results in misaligned vocabulary, stylistic inconsistencies, and contextually inappropriate translations. The impact extends beyond MT output, also affecting the accuracy of QE, which depends on consistent reference domains to generate reliable assessments. Despite considerable research efforts, domain mismatch remains a persistent and complex barrier to achieving high-quality, dependable MT and QE performance in real-world, domain-sensitive applications. This dissertation set out to explore how MT and QE systems can be adapted to function more effectively across various domains, particularly in specialized fields such as healthcare, law, and IT where accurate translation is critical. In our work, a domain refers to the source of a dataset. Improved model performance on such data is treated as DA, reflecting real-world cases where domain-specific and generic content are often intermixed. The central motivation arose from the observation that general- purpose MT systemsâespecially LLMs trained on general-domain cor- poraâstruggle to accommodate the specialized terminology, stylistic conventions, and contextual nuances of professional domains. A mis- translated medical instruction or legal clause is not just a linguistic error but a potentially harmful or costly one. This highlights a foundational insight: that carefully selected, domain-specific data can improve trans- lation quality more effectively than adding large amounts of generic text. At the heart of the challenge is domain mismatch, where the dis- connect between training and deployment domains leads to vocabulary mismatches and contextually inappropriate outputs. This problem extends beyond MT to the realm of QE, where domain-inconsistent ref- 154 erences undermine the accuracy of quality assessments. Building robust MT and QE systems, therefore, requires a comprehensive strategy that accounts for not just the data itself, but how it is selected, represented, tokenized, and evaluated. To address and better understand these challenges, this disserta- tion proposes a multi-faceted approach: domain-specific data selection strategies to improve translation quality even in low-data scenarios (Chapter 2); a combination of DA and DAG techniques to boost QE performance in low-resource and multilingual settings (Chapter 3); an evaluation of SW tokenization and vocabulary configurations, which significantly influence model efficiency and translation quality during fine-tuningâa common method for DA (Chapter 4); and a novel ICL strategy that leverages domain-specific QE models to guide ICE se- lection, enabling accurate translation with generative LLMs without fine-tuning, thereby reducing computational costs (Chapter 5). Our work also highlights the interdependence of MT and QE in addressing domain mismatch, stressing that high-quality translation and accurate evaluation must evolve together. By treating them as mutually reinforcing, it offers an integrated approach to DA. The result is a set of methods and insights aimed at improving performance while promoting scalable, efficient, and sustainable practices. Computational cost details are included throughout, with a focused breakdown in the appendix. 6.1 Summary of findings This dissertation addressed four core RQs, each targeting a specific aspect of DA in MT and QE. To reiterate, the main RQ guiding this work was: How can we design MT and QE systems that are accurate, adaptable, and efficient across specialized domains? 155 To answer this overarching question, four sub-questions were formu- lated. These are revisited and answered below. RQ1: What is the optimal amount of in-domain data required to achieve state-of-the-art MT quality at low computational costs? âą RQ1a: How does the quality of selected in-domain data affect translation performance? âąRQ1b: What trade-offs arise between translation performance and computational cost when using in-domain versus generic- domain data? To address RQ1âdetermining the optimal amount of in-domain data needed to achieve high translation quality with low computational costâwe introduced a comprehensive and adaptable ranking-based data selection method in Chapter 2. This approach tackles both RQ1a and RQ1b by enabling the extraction of semantically similar sentence pairs from large general-domain corpora. The method uses a modular pipeline that integrates contextual sentence embeddings, semantic similarity search, and in-domain relevance ranking. It is designed to be scalable and generalizable across languages and domains. Findings related to RQ1a. Our findings demonstrate that the quality of selected in-domain data plays a critical role in translation performance. Even relatively small, carefully selected subsets of semanti- cally relevant data significantly outperformed generic-domain baselines. In some cases, models trained on these selected subsets rivaledâor even exceededâthe performance of models fine-tuned on full in-domain corpora (vs. our approach that trained from scratch). This underscores a key insight: data relevance can outweigh data volume. 156 These findings align with and build upon recent research. Following our work, Eronen, Ptaszynski, Nowakowski, Chia, and Masui (2023) showed that using related language data or domain-specific subsets is especially effective in low-resource settings, such as PolishâEnglish translation. Likewise, Chakrabarty, Dabre, Ding, Utiyama, and Sumita (2020) stressed the importance of evaluating linguistic relevance in feature selection, cautioning against blindly incorporating features with- out understanding their value. Similarly, Wu, Wang, Foster, Qu, and Haffari (2024) found that token-level relevance can enhance translation quality more effectively than simply adding more data. While our work shares this emphasis on relevance, it takes a different approach: rather than focusing on individual tokens, we rely on contextual similarity at the sentence level. Our results extend these insights, showing that domain-relevance-driven selection at the sentence level can match or even surpass in-domain fine-tuningâwithout relying on PLMs, unlike the prior work discussed in Chapter 2âand with significantly lower computational cost. For instance, our method achieved a BLEU score of 31.0 using just 1 million carefully selected sentences, whereas training with 30 times more mixed-domain (in-domain and out-of-domain) data yielded only 26 BLEU. Of course, this result is for a domain-specific test set; for generic-domain or out-of-domain evaluation, such a small specialized model would likely be less effective. Moreover, reaching this level of quality with our approach required approximately 4 hours of training, compared to 10 hours needed to train on 30 million sentences to achieve lower performance. These results highlight that smarter data selection can be more impactful than scale alone, enabling both high-quality and resource-efficient MT. Findings related to RQ1b. In addressing RQ1b, we demonstrated that our methodâselecting in-domain data based on contextual rel- 157 evance and using it to generate synthetic dataânot only enhances translation quality but also does so efficiently. By restricting training to top-ranked, semantically relevant data, the approach significantly reduced computational time and resource consumption without sacri- ficing performance. Although the optimal data size varied by domain, our results (referred to as the âMixing Effectâ) consistently revealed a saturation point: beyond this threshold, additional data provided di- minishing returns and, in some cases, even led to degraded performance. This degradation in quality stems from the growing inclusion of less relevant data as the selection pool widensâespecially when extracting from out-of-domain sources that are not large or diverse. In essence, as quantity increases, contextual similarityâand therefore qualityâtends to decline. These findingsâparticularly the insight that translation quality can actually decrease after a certain pointâare consistent with earlier work (Poncelas, Shterionov, Way, Maillette de Buy Wenniger, & Passban, 2018; Moslem, Haque, Kelleher, & Way, 2022; Mittal, Hrinchuk, & Kuchaiev, 2023), which observed that the benefits of adding more domain-specific data eventually level off. While this pattern has been noted before, it often was not examined in much detail. Our work not only confirms this effect but also offers a clearer explanation: as we continue expanding the datasetâespecially from out-of-domain sources that are not very large or diverseâthe added data becomes less relevant. This dilution reduces contextual similarity, introduces noise, and ultimately hurts performance. Jointly, these findings highlight an important takeaway: it is not just about having more data, but about having the right data. Thoughtful, relevance-based selection can lead to better results, faster training, and more efficient use of resources. Ultimately, our work in Chapter 2 presents a clear and effective strategy for improving MT in domain-specific, low-resource settings. By prioritizing relevance over volume, we demonstrated that mod- 158 est, well-selected datasets can yield high-quality translations at sig- nificantly lower computational cost. Importantly, our open-source implementation supports reproducibility and flexibility, allowing others to replicate, customize, and extend the method across domains and languagesâreinforcing the broader value of targeted data selection in building efficient and accurate MT systems. To support further use and development, we released an open- source toolkit with examples and step-by-step instructions, available at: https://github.com/JoyeBright/domain-adapt-mt. RQ2: What are the effects of combining DA and DAG on the performance and generalizability of QE models across different domains and languages? âąRQ2a: How does DA impact QE performance in low-resource scenarios? âąRQ2b: What challenges arise when applying DAG in QE across diverse languages? This dissertation addressed RQ2 by designing and evaluating a QE training pipeline aimed at tackling two major challenges in QE: data scarcity and domain mismatch. These are common hurdles that often limit the effectiveness and reliability of QE systems. Our proposed solution combined DA with a lightweight DAG strategy, resulting in a modular and flexible approach that improves both performance and generalizability. To briefly recap, recent QE modelsâlike MT modelsâare neural- based and heavily data-driven. When a QE model trained on one dataset is applied to another, mismatches in domain can significantly affect performance. This challenge is further compounded by the fact that QE requires not only the source and machine-translated output but 159 also a post-edited referenceâa resource-intensive and costly prerequisite. This led us to first confirm a foundational assumption: QE models, like MT models, must be domain-specific to perform well. From there, we addressed the data scarcity issue by proposing methods to augment or generate synthetic data to bridge the gap. Findings related to RQ2a. Our findings show that a customized version of the âmixed fine-tuning + fine-tuningâ strategy proposed by Chu et al. (2017) significantly improves QE performance in low- resource settings. We adapted this strategy for QE for the first time by initially fine-tuning a pre-trained XLM-R model on a large out- of-domain (OOD) QE dataset to build a strong, generic baseline. In the second stage, we applied a mixing step that combined this OOD data with in-domain (ID) dataâboth authentic and syntheticâusing simple yet effective DAG techniques. These included multilingual ID concatenation and synthetic data generation via back-translation and QE label prediction, as detailed in Section 3.2.2 of Chapter 3. This step helped the model retain general QE knowledge while gradually adapting to the target domain. In the final stage, the model was fine- tuned solely on a small, high-quality ID dataset to sharpen its domain- specific performance. This structured training pipeline consistently led to statistically significant gains across all tested language pairs and domains. These findings align with earlier work that advocated for inter- mediate fine-tuning steps between pre-training and downstream QE tasks(Rubino, 2020; Rubino & Sumita, 2020; Lee, 2020). However, our pipeline extends this line of research by combining DAG with domain- specific fine-tuning. Unlike approaches that rely on large amounts of labeled data, our method offers a more efficient path to high perfor- mance, especially in real-world scenarios where such data is limited. 160 Findings related to RQ2b. In addressing RQ2b, we examined the effects of DAG across different languages and domains. Overall, augmentation was found to be beneficial, contributing to improved model robustness and generalization. We observed variation in per- formance across language pairs and settings, with some cases showing limited or no improvement. These outcomes are consistent with prior concerns about potential mismatches between synthetic and real-world QE distributions(Qiu et al., 2022), particularly when the synthetic data is noisy or poorly curated. Unlike earlier DAG methods that rely on classifier-based or pseudo-labeled pipelines(Baek et al., 2020; Freitag, Rei, et al., 2021), our approach relied on simpler techniquesâsuch as back-translation and domain-aware samplingâwhich yielded promising results in multiple settings. We also found that integrating domain tags during training consis- tently improved performance. These tags helped the model distinguish between different types of data and reinforced domain-specific cues. This observation is in line with prior work in both MT(Sennrich et al., 2016a; Chu & Dabre, 2019) and multi-domain QE(Fomicheva et al., 2022; Zerva et al., 2022), confirming that such techniques are transferable and valuable in QE settings. The generalizability of the proposed approach was further validated through cross-lingual and ZSâwhere the model had no exposure to the target language during trainingâevaluations. The pipeline was tested on a diverse set of language pairs from the WMT21 QE shared task, in- cluding EnglishâGerman, âChinese, âItalian, âJapanese, âCzech, as well as RomanianâEnglish and RussianâEnglish. Even under ZS conditions, the first stage of the pipeline outperformed most baselines. Notably, the model trained on RomanianâEnglish performed particularly well on its own test set and transferred effectively to EnglishâJapanese. This may be attributed to the modelâs exposure to EnglishâItalian data during the first training stage, as Italian and Romanian share 161 typological similarities. These findings suggest that linguistic proximity can support stronger cross-lingual transfer. In summary, combining DA and DAG within a well-structured, staged training process leads to robust QE models capable of strong performance across domains and languages. Importantly, the proposed methodology offers a practical and scalable solution for real-world scenarios, where annotated QE data is limited or unavailable for many languages and domains. RQ3: What is the optimal combination of SW tokenization model source and vocabulary source to maximize translation quality and efficiency during fine-tuning? âąRQ3a: How do different data sources for the SW tokenization model affect translation quality when fine-tuning out-of-domain models on in-domain data? âąRQ3b: How do the sources of the SW tokenization model and vocabulary impact the trade-off between fine-tuning efficiency (e.g., training time, resource use) and translation accuracy? While fine-tuning PLMs for DA has shown substantial performance gains(T. Luong et al., 2015; Dakwale & Monz, 2017), its effectiveness depends heavily on the representation of domain-specific inputs. This not only involves model architecture and data selection, but also the SW tokenization strategy and vocabulary. Prior work has identified vocabulary mismatches between base and target domains as a major obstacle to effective DA(R. Lim et al., 2018; Sato et al., 2020; J. Lim & Kim, 2024), and proposed embedding-based solutions, such as âvocabu- lary adaptation,â projecting target-domain embeddings into pretrained embedding spaces to improve performance by 3â4 BLEU points(Sato et al., 2020). 162 Despite this, the joint effect of SW tokenization and vocabulary choice during fine-tuning remains underexplored. While studies have focused on embedding layer adjustments(Sato et al., 2020) or tuning SW granularity(Salesky, Runge, Coda, Niehues, & Neubig, 2018), few have systematically evaluated how tokenization methods and vocabulary sources interact under consistent fine-tuning configurations. To address RQ3, we empirically evaluated seven consistent config- urations that combine different sources for SW tokenization models (pre-training, in-domain, or combined) and vocabulary (pre-training, in-domain, or combined) in an EnglishâGerman DA setting. Findings related to RQ3a. Our results show that the choice of tokenization model and vocabulary source substantially influences translation quality. Using both BPE and vocabulary from in-domain data yields the highest BLEU, chrF2, and best TER scores and the most effective adaptation to the domain. However, this configuration also introduces the most new tokens and requires the longest training time. In contrast, mismatched setups (e.g., a configuration with in-domain BPE and a vocabulary from the pre-training data) result in significant quality drops, highlighting the need for coherent SW and vocabulary alignment. Hybrid configurations, which combine resources from both pre-training and in-domain data, offer practical alternatives that balance adaptation effectiveness and resource demands. These findings reinforce previous work(Sato et al., 2020), showing that effective vocabulary adaptation must be closely tied to the chosen tokenization model. Findings related to RQ3b. We observed a clear trade-off between translation quality and computational efficiency. Configurations that reused pre-training resources were computationally efficient but lagged in domain accuracy, while purely in-domain setups improved quality at the expense of training speed and resource use. The most balanced 163 results came from hybrid approaches, which combine pre-training and in-domain resources (i.e., using either the BPE model or vocabulary, or both, derived from a mix of pre-trained and in-domain data)âand achieve strong results while reducing computational costs compared to full in-domain setups. In summary, these results demonstrate that carefully aligning both SW tokenization and vocabulary source with the fine-tuning data sub- stantially enhances adaptation quality, while hybrid configurations can offer a good balance between domain performance and generalization. In practice, one might prefer hybrid setups when the goal is not only to maximize in-domain accuracy but also to retain robustness across out-of-domain or mixed-domain scenarios. These findings highlight the practical importance of consistent SWâvocabulary design in effective fine-tuning for DA. Naturally, our study has certain limitationsâsuch as the exclusive focus on MT, the medical domain, the use of BPE only, and the lack of hyperparameter tuningâwhich were discussed in detail in Section 4.6. RQ4: How effective are domain-specific QE models in deter- mining effective ICEs for improving MT quality in generative LLMs? âąRQ4a: What criteria should be used to select effective ICEs for improving MT quality? âąRQ4b: How does the integration of QE affect the success of ICE selection? LLMs have demonstrated strong performance across a wide range of NLP tasks, including MT. Unlike traditional approaches, LLMs can perform translation without further training by conditioning on a well-crafted input prompt. Prior research has shown that providing 164 additional contextâspecifically, ICEsâcan substantially enhance trans- lation quality. ICEs serve to prime the model, enabling it to better focus on and adapt to the target task. However, the effectiveness of ICEs is influenced by multiple factors, such as the domain of the source text, the number and order of examples, and the prompt format. As there is no universally optimal configuration, selecting effective ICEs re- quires a nuanced understanding of how these variables affect translation performance, which is typically evaluated using reference translations or human judgmentâboth of which are often unavailable or expensive to obtain. Motivated by the demonstrated advantage of domain-specific QE over generic QE in Chapter 3, and recognizing the absence of a universal strategy for optimal ICE selection across different LLMs, we proposed using domain-specific QE as a feedback mechanism to guide the selection of ICEs. To address RQ4, we developed a novel ICL methodology for MT that leverages domain-specific QE to inform ICE selection. This approach was designed to maximize translation quality while minimizing computational cost, without relying on reference translations. Findings related to RQ4a. We demonstrate that the quality, domain relevance, and contextual fit of ICEs are critical for achieving strong MT performance in LLMs. We show that domain-specific QE serves as an effective criterion for selecting ICEs, consistently outper- forming conventional methods such as random selection(Sia & Duh, 2023), n-gram similarity-based retrieval(Agrawal et al., 2023), and fixed retriever rankings (e.g., R-BM25). Although prior work has found random or n-gram-based selections to be surprisingly competitive, our results suggest that these approaches tend to emphasize surface-level lexical similarity while overlooking deeper semantic and domain-specific signals essential for high-quality translation. Notably, our QE-guided approach consistently achieved higher BLEU and COMET scores on 165 GermanâEnglish IT translation tasks compared to both these ICL baselines. Furthermore, our findings suggest that ICL offers substantial effi- ciency benefits. Unlike fine-tuning PLMs such as mBART-50âwhich requires considerable computational resourcesâour QE-guided ICL strategy, and ICL strategies more broadly, achieve competitive results at significantly lower cost, making them practical and scalable solutions for real-world multilingual MT applications. We also examined whether the order of ICEs influences translation quality by testing reordering based on unigram overlap. While earlier studies emphasized the importance of example ordering(J. Liu et al., 2022; Lu et al., 2022), our results showed that this form of reordering had minimal impact. This suggests that what matters most is not how lexically similar they are to the input, but whether they are semantically relevant and domain-appropriate. In other words, examples that make sense for the task at handâand match the style or terminology of the inputâcarry more weight than those chosen purely for their surface similarity. Findings related to RQ4b. In addressing RQ4b, we found that integrating QE into ICE selection not only improves translation quality but also enhances efficiencyâresulting in faster inference, reduced mem- ory usage, and greater scalability, which are particularly beneficial given token constraints in LLMs. Although QE may occasionally misrank examples, our method consistently selected fewer yet more impactful ICEs. This stands in contrast to prior approaches that include a larger number of examples without assessing their individual contribution to translation quality(N. Goyal et al., 2022). Additionally, our method produced translations with lengths more closely matching the refer- ences, which can reduce downstream post-editing effort. Finally, since QE models estimate quality without relying on reference translations, 166 they offer a promising direction for mitigating reference biasâa known limitation in LLM evaluation pipelines(Raunak, Menezes, & Awadalla, 2023). Moreover, by modeling quality beyond surface-level features, QE may help identify domain-specific issues such as word sense ambigui- ties or terminology mismatchesâthough further empirical validation is needed to confirm this potential. These findings confirm that QE-guided ICL offers a practical and scalable alternative to fine-tuning. In contrast to traditional methods requiring parameter updates or extensive reference data, our approach maintains model generalization while enhancing domain-specific trans- lation quality through selective, high-impact ICEs. 6.2 Limitations and future work In this section, we reflect on some general limitations and potential directions for future research that cut across the different parts of this dissertation. We also highlight chapter-specific considerations to provide a more grounded view of where the work could be expanded or refined. While each chapter already includes its own analysis and discussion, what follows is meant to offer a broader perspective and point toward ideas worth exploring further. For more detailed insights, see chapters 4â5. General limitations and future directions While this dissertation offers a broad and in-depth look at DA, SW mod- eling, QE, and ICL in MT, a few practical constraints inevitably shaped its scope. One notable limitation is the heavy reliance on automatic evaluation metrics such as BLEU, COMET, and TER. These tools are widely used and provide a consistent framework for comparison, but they can only go so far. Human evaluationsâthough far more insight- ful when it comes to subtle aspects of translation qualityâare costly 167 and time-intensive. Neural-based metrics like COMET, for example, show strong correlation with human judgments(Rei et al., 2020), but can still carry biases from the data they were trained on, potentially skewing results in cross-domain scenarios(Zouhar, Chen, et al., 2024). Therefore, incorporating human assessments would undoubtedly enrich future work by providing a more nuanced understanding of translation quality. That said, such efforts require dedicated resources and were beyond the reach of this dissertation. Another consideration is the selection of language pairs and domains. While the chosen languages were carefully selected to reflect meaningful linguistic and domain diversity, as well as considerations such as repro- ducibility and the findability/availability of suitable datasets, the find- ings would benefit from being tested on an even broader setâespecially low-resource or typologically distinct languages. Of course, this is a common limitation in MT: it is almost impossible to cover every language and domain, no matter how ambitious the study. Methodologically, keeping components like tokenization models or retrieval strategies constant was a deliberate choice to ensure clarity and control in experimentation. But future studies for example might try out semantic matchers beyond SBERT to improve generalization. Lastly, while hyperparameter tuning was carefully performed, there is still room for refinement. Additional optimizationâeither automated or task-specificâmay further enhance performance across the board. Chapter-specific limitations and directions Chapter 2: This chapter introduced a similarity-based data selection method that retrieves a fixed number (n) of nearest neighbors (contex- tually similar examples) for each input example. The method performed well in practice and offered a simple, consistent mechanism for selecting relevant context. However, relying on a fixed number assumes that the topnexamples are equally informative across all inputsâwhich may 168 not always hold true. In some cases, the distribution of similarity scores varies considerably, or, in less ideal scenarios, the generic dataset may not contain enough truly relevant examples. As a result, the model may fall back on less similar, and potentially less useful, instances. This choice was made deliberately to maintain interpretability and experimental control. That said, there is room for further development. One promising direction would be to experiment with dynamic similarity thresholds that adjust to each inputâs specific distribution. Naturally, this introduces the challenge of defining an appropriate threshold for each caseâa task that becomes an optimization problem in itselfâbut it is a challenge worth exploring. Future work could also look into alternative similarity metrics or integrate the selection process more tightly with MT training, potentially enabling more precise and domain- aware translations. Chapter 3: The proposed QE pipeline was validated on several multilingual benchmarks and showed consistent performance gains. Nonetheless, further experiments on additional OOD datasets and with alternative backbone models could help better assess the robustness and generalizability of the approach. As the field continues to shift away from traditional automatic metrics like TER toward richer annotation frameworks such as Direct Assessment(Graham, Baldwin, Moffat, & Zobel, 2013) and Multi- dimensional Quality Metrics (MQM)(Lommel et al., 2014; Freitag, Foster, et al., 2021), adapting the pipeline to work with these more expressive and human-centered labels would strengthen its practical rel- evance. That said, this transition comes with notable challenges: Direct Assessment relies on costly human ratings at the sentence level, and MQM demands detailed linguistic annotation, which is time-intensive and requires domain expertise. As discussed earlier in the general limi- tations, incorporating human evaluation at scale remains a persistent challenge due to cost and effort. 169 Another promising direction for future work involves exploring alter- native strategies to preserve pre-trained linguistic knowledge, without relying on explicit quality tagging. One such approach is to freeze the PLMs and leverage its internal representations directly for quality prediction. This method helps maintain the original distribution of information across layers, where each layer typically encodes different linguistic phenomenaâsuch as morphology, syntax, or semantics. In doing so, it can reduce the risk of overwriting useful generalizations during fine-tuning, while also lowering training overhead and making the pipeline more lightweight and scalable. However, this comes with trade-offs. Without fine-grained, task- specific supervision, the model may prioritize general semantic similarity rather than capturing detailed aspects of translation quality. As a result, important distinctionsâsuch as adequacy or fluencyâmight be underrepresented in the predictions. Balancing the preservation of pre-trained knowledge with the need for quality-aware modeling remains an open challenge for future QE research. Chapter 4: Although the experiments in this chapter were carefully designed and systematically executed, several broader limitations should be acknowledged. First, our study focused solely on MT systems and did not extend to LLMs. While MT models and LLMs share certain sensitivities to data processing and vocabulary choices, LLMsâ particularly multilingual onesâintroduce additional challenges, such as managing vocabulary across multiple languages and handling cross- lingual interference. Understanding how SW modeling and vocabulary configurations impact LLM performance is a valuable direction for future research. Second, the datasets used in our experiments were relatively small and drawn from a single domain (IT). This setup enabled controlled and interpretable comparisons but limits how broadly the results can be applied. We did not test our configurations on other specialized 170 domains like medicine or law, nor did we analyze how much overlap exists between the vocabularies of training and fine-tuning datasets. Both of these factorsâdomain variation and vocabulary overlapâlikely influence how well a model adapts to new data. Exploring these aspects would offer a more complete picture of how domain similarity and token alignment affect translation quality. Finally, to isolate the effects of SW modeling, we intentionally limited the scope of our experiments: we used only BPE and did not tune hyperparameters or compare alternative tokenization methods like SentencePiece(Kudo & Richardson, 2018). While this design choice helped ensure fair comparisons, it also means we did not explore the potential gains from more adaptive approaches. Future work could broaden the experimental setup by testing different tokenizers and by developing ways to quantify how different two datasets areâsuch as measuring domain relevance or lexical divergence. This could help inform decisions about when existing SW and vocabulary settings can be reused and when they need to be adapted for better domain transfer. Chapter 5: Our approach uses XGLM as the base LLM and applies iterative QE scoring to select ICEs, which introduces additional com- plexity and increases inference time. While the trained QE model can be reused within the same domain, its generalizability to other domains remains uncertain. Future research could explore alternative backbone architectures, develop lightweight or distilled QE models to reduce computational overhead, or design more integrated retrievalâscoring pipelines to streamline the selection process. To ensure a fair comparison with prior work discussed in this chap- ter, we employed BLEU-based QE as the scoring signal. This decision enabled consistent evaluation and direct benchmarking. Neverthe- less, future work could incorporate neural evaluation metrics such as COMET, which offer stronger alignment with human judgments and may lead to more accurate ICE selection, particularly in low-resource 171 or semantically complex settings. Finally, we did not investigate the effect of ICE ordering, as evalu- ating all possible permutations would incur substantial computational costs. However, since the QE model provides a quality estimate for each candidate combination, it could be extended to rank and opti- mize example orderings more efficiently. Exploring this area remains a promising direction, especially in light of recent findings(Xiang, Yan, Gui, & He, 2024) that show causal LLMs are sensitive to the order of ICEs. 6.3 Implications of the findings In this dissertation, we examined the persistent challenge of domain mismatch in MTâa problem that not only affects MT and QE systems, but also extends across the broader landscape of NLP, and even into fields like computer vision. Naturally, domain mismatch manifests differently across modalities: in vision, for example, it may involve shifts in lighting, perspective, or object appearance rather than termi- nology(Ranjan, Harit, & Jawahar, 2015). What unites these scenarios is the shared difficulty models face when deployed on data distributions that diverge from those they were trained on. At a high level, this dissertation contributes to addressing domain mismatch in both MT and QE by underscoring the necessity of adapta- tion. Our findings align with a growing consensus in neural modeling: whether through ICL in LLMs or fine-tuning in encoder-decoder archi- tectures (including LLMs themselves), most models require some degree of adaptation to maintain strong performance across domains(Rostam & KertĂ©sz, 2024; Sinha, Premsri, & Kordjamshidi, 2024). This need has become especially evident in recent LLM-based workflows, where domain-specific prompts and carefully selected ICEs play a central role in achieving reliable results. Although our focus has been on 172 MT and QE, this broader principle likely extends to many other NLP tasksâand even further, to fields such as computer visionâwhere models are increasingly deployed in specialized, real-world settings. Our results reinforce the idea that general-purpose models should not be expected to perform optimally out of the box. Instead, they must be aligned with their target domain and task context to ensure both accuracy and utility. In Chapter 3, we extended this observation to QE itself, showing that evaluation modelsâlike the systems they assessâalso require domain-specific adaptation. A QE model trained on general data may fail to detect quality variations in specialized domains, leading to unreliable assessments. This issue also affects reference-based metrics such as COMET, which may inherit biases from their training data(Zouhar, Chen, et al., 2024). As such, data-driven evaluation models must be applied with awareness of their domain limitations to avoid misinforming downstream decisions. These findings contribute to a broader theoretical shift in machine learning: from model-centric to data-centric thinking. While scaling models has been a dominant strategy, our results challenge the assump- tion that bigger always means better. Instead, we emphasize the need to understand the nature of the data models seeâand how this shapes their behavior. This is especially crucial in domain-sensitive tasks like MT and QE, where even small mismatches in terminology, structure, or tokenization can lead to significant drops in performance. Of course, adaptation is not without cost. It typically requires addi- tional training, hyperparameter tuning, and sometimes re-tokenization, all of which come with computational and environmental overhead. While some argue that scaling models and training data may eliminate the need for DA, our findings suggest that more data alone is not always the answer. In fact, we show that carefully curated, domain-specific dataâwhether used in prompting or fine-tuningâcan yield better re- sults than simply increasing volume. This is central to the data-centric 173 approach we advocate throughout the dissertation. In Chapter 4, we also demonstrated that the influence of data ex- tends to upstream components like tokenization. Often considered a low-level detail, tokenization was shown to significantly impact transla- tion quality when mismatched with the domain. Our results revealed that tokenizers trained on domain-specific data can improve output quality, especially for specialized terminology. This underscores that DA must consider the entire pipelineâfrom tokenization to evalua- tionârather than focusing solely on model architecture. Further, in Chapter 5, we explored how QE can be used not just for evaluation, but as an external signal to guide and refine MT outputs. This reflects a broader shift toward feedback-driven, interactive systems where multiple components collaborate to improve performance(Briva- Iglesias, 2025). Our work aligns with emerging multi-agent work- flowsâsuch as TransAgents(Wu, Xu, & Wang, 2024)âin which, for example, one agent generates a translation and others provide feedback to iteratively enhance it. These agent-based systems, often powered by LLMs, point to the growing potential of integrating tools like QE models or glossaries into real-time translation workflows. While our implementation is research-oriented, it signals a promising direction for scalable, adaptive, and collaborative MT systems. A key methodological implication of this work is the value of building modular, reusable tools that support data-centric workflows. One such example is the domain-specific data selection tool introduced in Chapter 2. Originally designed to retrieve the top-kmost similar examples for ICL, the tool has since evolved to support more flexible configurationsâincluding the ability to surface dissimilar examples, which makes it useful for contrastive learning setups such as triplet loss. What makes this tool particularly impactful is its generalizability. It is designed to be multilingual, language-agnostic, and easily adapt- able, making it relevant well beyond the MT use case. Whether for 174 selecting examples in ICL, generating negatives for contrastive training, or supporting tasks like DAG, retrieval, or representation learning, the tool serves as a practical building block for a range of domain-aware ap- plications. More broadly, it reflects a shift toward developing adaptable, context-sensitive components that help researchers and practitioners work more effectively across domainsâwithout having to start from scratch each time. Finally, it is important to acknowledge the very real costsâcomputational, financial, and environmentalâassociated with building domain-specific systems. Our work shows that while DA is often necessary for achieving reliable performance, it adds to the already significant demands of pretraining large models. Given the substantial investments of time, data, and compute that go into developing massive, general-purpose LLMs, this raises a natural question: why not design these models to be more adaptable or better aligned with domain-specific needs from the start? Of course, building truly one-size-fits-all models may still be out of reach. However, our findings suggest that some of the effort currently devoted to making models broadly generalizable could be more usefully redirected toward making them flexible by design. This shift could re- duce the need for repeated adaptation, particularly as model sizesâand their associated costsâcontinue to grow. Encouragingly, we are already seeing signs of this in todayâs commercial LLMs, which often perform reasonably well in specialized domains with minimal additional effort, especially compared to what was required just a few years ago. To support transparency and reproducibility, we provide a detailed, chapter-by-chapter breakdown of the computational costs involved in our experiments in the appendix. 175 6.4 Wrapping up This dissertation addressed the core challenges of domain mismatch. Despite the remarkable advancements achieved by large-scale, PLMs, our findings illustrate that these systems frequently fall short when applied directly to specialized or out-of-domain contextsâparticularly in tasks like MT, where terminology and structure can vary significantly across domains. Through extensive experimentation and analysis, we demonstrated that reliable performance in real-world, domain-sensitive applications demands deliberate adaptationâwhether through fine- tuning, targeted data selection, or informed use of ICL guided by domain-specific QE. Beyond technical outcomes, this dissertation contributes to a broader theoretical shift in the field, moving away from purely model-centric solutions towards a more data-driven perspective. We have shown repeatedly that simply scaling up models or datasets does not automat- ically guarantee improved performance. Instead, the careful selection and preparation of domain-specific dataâranging from the choice of examples used in prompts to the nuances of vocabulary and tokenization strategiesâplay a crucial role in model accuracy and efficiency. This shift underscores the importance of data quality and domain awareness in achieving high performance. In addition to these theoretical insights, we developed practical methodologies and publicly available toolsâsuch as the domain-specific data selection system introduced in Chapter 2. These resources are designed to help researchers and practitioners efficiently implement DA strategies across languages, domains, and tasks. By emphasizing trans- parency and reproducibilityâhighlighted by the detailed computational analyses provided in the appendixâwe aim to facilitate future work that is both effective and environmentally sustainable. Looking ahead, the tension between broad generalization and special- 176 ized adaptability in language technologies is likely to intensify. Although modern commercial LLMs offer impressive performance with minimal adaptation, our research indicates that specialized domains still pose significant challenges. Therefore, future progress will likely depend on building systems that combine general-purpose capabilities with inherent adaptability. This balanced approach promises not only better domain-specific solutions but also more responsible and efficient use of resources in a rapidly evolving technological landscape. 177 178 Summary Machine Translation (MT) and Quality Estimation (QE) have advanced rapidly, fuelled by neural architectures and the growing influence of large language models (LLMs). Yet, despite these breakthroughs, mod- ern systems still struggle when used outside the general-domain settings they were trained for. In specialized domains such as healthcare, law, or information technology, 1 the data often come from very different sources, follow different conventions, and contain terminology that generic models are not equipped to handle. When models encounter a mismatch between training data and real-world deploymentâa prob- lem widely known as domain mismatchâtranslation accuracy drops, terminology becomes unreliable, and automated quality predictions lose consistency. Simply scaling up model size or training data volume does not resolve this issue; on the contrary, it introduces substantial addi- tional computational requirements without improving domain-specific performance. What matters is whether systems are adapted to the domain they need to serve, especially with respect to the source and nature of the data used during training. This dissertation investigates how MT and QE systems can be made more accurate, more adaptable, and more computationally efficient 1 In our studies, the domain is treated as the origin of a dataset; topical labels here are used only to help readers intuitively understand the concept. in domain-specific scenarios. Because there is no single solution to domain mismatch, the work approaches the problem from several angles: selecting the right training data, designing adaptation pipelines for QE, understanding the role of subword tokenization and vocabulary choices during fine-tuning, and finally, using QE to guide in-context learning (ICL) for LLM-based translation. Jointly, these investigations are organized around four core research questions: âąRQ1: What is the optimal amount of in-domain data required to achieve high MT quality at low computational cost? âąRQ2: What are the effects of combining domain adaptation and data augmentation on the performance and generalizability of QE models across different domains and languages? âąRQ3: What is the optimal combination of subword tokenization model source and vocabulary source to maximize translation quality and adaptation efficiency during fine-tuning? âąRQ4: How effective are domain-specific QE models in determining effective in-context examples (ICEs) that improve MT quality in generative LLMs? The first contribution of this dissertation addresses RQ1 by intro- ducing a modular, similarity-based method for selecting high-quality in-domain sentence pairs from large generic corpora. Experiments show that small, carefully selected subsets of contextually relevant data consistently outperform far larger mixed-domain datasets. Translation quality saturates quickly, and expanding the dataset beyond a certain point introduces noise that degrades performance. These findings high- light that carefully selected data can substitute for expensive large-scale training, enabling high-quality MT with significantly reduced compu- tational cost compared to training on the entire available dataset or scaling model size. 180 The second contribution addresses RQ2 by proposing a staged QE training pipeline that integrates domain adaptation with lightweight data augmentation. This three-step processâgeneric fine-tuning, mixed- domain training, and final in-domain specializationâproduces QE mod- els that are robust across languages, domains, and resource levels. Evaluation across diverse multilingual benchmarks, including zero-shot and cross-lingual settings, confirms that QE models, like MT systems, re- quire domain-specific adaptation for reliable quality prediction. Domain tags, synthetic data generation, and multilingual mixing collectively improve generalization while keeping training efficient and scalable. The third contribution answers RQ3 by conducting the first system- atic analysis of how subword tokenization and vocabulary source jointly affect fine-tuning outcomes in MT. Results show that mismatched tokenizationâvocabulary configurations lead to unstable training and poorer translation quality, while coherent configurationsâespecially those derived from in-domain dataâyield the most effective domain adaptation. Fully in-domain tokenization offers the highest accuracy but at greater computational cost; hybrid configurations provide a balance by combining strong performance with better efficiency. These findings show that tokenization is a core component of domain adaptation, not a preprocessing detail. Answering RQ4, the fourth contribution introduces a novel QE- guided ICL framework for generative LLMs. Domain-specific QE models are used to score and select ICEs that maximize translation quality without requiring reference translations or parameter updates. This method outperforms random, n-gramâbased, and BM25-based retrieval approaches, 2 showing that the quality, relevance, and domain fit of ICEs determine their effectiveness. QE-guided ICL achieves strong improvements in BLEU and COMET while requiring far less computa- 2 BM25(Robertson & Zaragoza, 2009) ranks documentsâin our case, sen- tencesâbased on their relevance to a search query. 181 tion than full fine-tuning, establishing it as a practical alternative for domain-specific MT with LLMs. These four contributions offer a unified, data-centric view of domain adaptation in MT and QE. The results underscore three central themes: (1) Domain specificity is essential for reliable performance: neither MT nor QE systems generalize well across domains without targeted adaptation, as demonstrated in Chapters 2 and 3, and further reinforced in the tokenization experiments of Chapter 4; (2) Better data beats more data: high-quality, domain-relevant examples consistently outperform large volumes of generic data, as shown in Chapter 2 and later examined from a vocabulary perspective in Chapter 4; and (3) Adaptation must be efficient: from targeted data selection to hybrid tokenization and QE-guided ICL, the methods proposed in this dissertation provide practical guidance for reducing computational and environmental costs while preservingâor improvingâtask performance, as explored across Chapters 3, 4, and 5. Overall, this dissertation advances both the theory and practice of domain adaptation by investigating the full MT pipelineâfrom data selection and tokenization to inference for MT systems and prompt- ing for LLMs, as well as quality evaluation and estimationâand by providing reproducible tools and detailed computational analyses that enable sustainable, domain-aware MT workflows. The findings reflect a broader shift in the field away from purely model-centric scaling and toward approaches that prioritize data relevance, domain specificity, and computational efficiency. As MT and QE systems are increasingly deployed in real-world professional settings, the methods developed here offer a solid foundation for building solutions that are more robust, more adaptable, and better aligned with the demands of specialized domains, including those faced by language service providers and other industry practitioners. 182 Samenvatting (Dutch Summary) Machinevertaling (MT) en kwaliteitsinschatting (QE) hebben zich de afgelopen jaren snel ontwikkeld, aangedreven door neurale netwerken en de groeiende invloed van grote taalmodellen (LLMâs). Toch blijven moderne systemen slecht presteren zodra zij worden ingezet buiten de generieke domeinen waarop zij zijn getraind. In gespecialiseerde domeinen zoals de gezondheidszorg, het recht, of informatietechnolo- gie, 3 is de data vaak afkomstig uit verschillende bronnen, volgt zij andere conventies, en bevat zij terminologie waar generieke modellen niet over beschikken. Wanneer modellen worden geconfronteerd met een mismatch tussen trainingsdata en daadwerkelijke toepassingenâeen probleem dat algemeen bekendstaat als domein-mismatchâneemt de ver- taalnauwkeurigheid af, wordt terminologie onbetrouwbaar en verliezen automatische kwaliteitsinschattingen hun consistentie. Het opschalen van modelgrootte of de hoeveelheid trainingsdata lost dit probleem niet op; sterker nog, het verhoogt de computationele kosten aanzienlijk zonder de prestaties binnen het gewenste domein te verbeteren. Wat werkelijk telt, is of systemen zijn aangepast aan het domein waarin zij moeten functioneren, met name wat betreft de herkomst en aard van de gebruikte data. 3 In onze studies wordt het domein beschouwd als de herkomst van een dataset; thematische labels worden hier uitsluitend gebruikt om het concept voor lezers intuĂŻtiever te maken. Dit proefschrift onderzoekt hoe MT- en QE-systemen nauwkeuriger, beter aanpasbaar en computationeel efficiĂ«nter kunnen worden gemaakt bij domeinspecifieke scenarioâs. Omdat er geen eenduidige oplossing bestaat voor domein-mismatch, wordt dit vraagstuk benaderd vanuit meerdere invalshoeken: het selecteren van geschikte trainingsdata, het ontwerpen van adaptatiepijplijnen voor QE, het begrijpen van de rol van subwoordtokenisatie en woordenschatkeuze tijdens fine-tuning, en het gebruik van QE om in-context learning (ICL) voor LLM-gebaseerde vertaling te sturen. Deze onderzoeken zijn gestructureerd rond vier centrale onderzoeksvragen: âą RQ1: Wat is de optimale hoeveelheid domeinspecifieke data die nodig is om hoge MT-kwaliteit te behalen tegen lage computa- tionele kosten? âąRQ2: Wat zijn de effecten van het combineren van domeinadapatie en data-augmentatie op de prestaties en generaliseerbaarheid van QE-modellen voor verschillende domeinen en talen? âąRQ3: Wat is de optimale combinatie van de bron van het sub- woordtokenisatiemodel en de bron van de woordenschat om ver- taalkwaliteit en adaptatie-efficiĂ«ntie tijdens fine-tuning te maxi- maliseren? âąRQ4: Hoe effectief zijn domeinspecifieke QE-modellen in het bepalen van effectieve in-context voorbeelden (ICEâs) die de MT- kwaliteit verbeteren in generatieve LLMâs? De eerste bijdrage van dit proefschrift beantwoordt RQ1 en intro- duceert een modulaire, op similariteit gebaseerde methode voor het selecteren van hoogwaardige domeinspecifieke zinnen-paren uit grote generieke corpora. Experimenten tonen aan dat kleine, zorgvuldig geselecteerde subsets van contextueel relevante data consequent beter 184 presteren dan veel grotere datasets met gemengde domeinen. De ver- taalprestatie verzadigt snel, en het uitbreiden van de dataset voorbij dit punt introduceert ruis die de kwaliteit juist vermindert. Deze bevin- dingen tonen aan dat zorgvuldig geselecteerde data dure grootschalige training kunnen vervangen en hoogwaardige MT mogelijk maken tegen aanzienlijk lagere computationele kosten. De tweede bijdrage betreft RQ2 en stelt een gefaseerde QE-training- pijplijn voor die domeinadapatie combineert met beperkte augmen- tatie van data. Dit driestapsprocesâgenerieke fine-tuning, training op gemengde domeinen, en tenslotte specifieke domeinaanpassingâlevert QE-modellen op die robuust zijn met betrekking tot talen, domeinen, en bronnen. Evaluaties op diverse meertalige benchmarks, waaron- der zero-shot en cross-linguale scenarioâs, bevestigen dat QE-modellen, net als MT-systemen, domeinspecifieke adaptatie nodig hebben voor betrouwbare kwaliteitsvoorspellingen. Domeintags, synthetische data, en meertalige mengstrategieĂ«n verbeteren de generalisatie terwijl de training efficiĂ«nt en schaalbaar blijft. De derde bijdrage beantwoordt RQ3 door de eerste systematische analyse uit te voeren van hoe subwoordtokenisatie en woordenschat- bron gezamenlijk de fine-tuning-prestaties van MT beĂŻnvloeden. De resultaten laten zien dat slecht op elkaar afgestemde combinaties van to- kenisatiemodel en woordenschat leiden tot instabiele training en lagere vertaalnauwkeurigheid, terwijl coherente configuraties-âmet name die gebaseerd op domeinspecifieke dataâ-de meest effectieve domeinadap- atie opleveren. Volledig domeinspecifieke tokenisatie levert de hoogste nauwkeurigheid op, maar tegen hogere computationele kosten; hy- bride configuraties bieden een balans tussen prestaties en efficiĂ«ntie. Deze bevindingen tonen aan dat tokenisatie een kernonderdeel is van domeinadapatie, en niet slechts een detail van pre-processing. De vierde bijdrage, die RQ4 beantwoordt, introduceert een nieuw QE-gestuurd ICL-raamwerk voor generatieve LLMâs. Domeinspeci- 185 fieke QE-modellen worden gebruikt om ICEâs te scoren en zodanig te selecteren dat vertaalprestaties maximaliseren zonder referentiev- ertalingen of parameterupdates. Deze methode presteert beter dan willekeurige selectie, n-grams, en BM25-gebaseerde retrieval, 4 en laat zien dat de kwaliteit, relevantie en domeinfit van ICEâs bepalend zijn voor hun effectiviteit. QE-gestuurde ICL levert sterke verbeteringen op in BLEU en COMET, en vereist aanzienlijk minder computationele mid- delen dan volledige fine-tuning, waardoor het een praktisch alternatief vormt voor domeinspecifieke MT met LLMâs. Gezamenlijk bieden deze vier bijdragen een samenhangende, datage- dreven visie op domeinadapatie in MT en QE. De resultaten onder- strepen drie centrale themaâs: (1) Domeinspecificiteit is essentieel voor betrouwbare prestaties: zowel MT- als QE-systemen generaliseren niet goed over domeinen zonder gerichte adaptatie, zoals aangetoond in Hoofdstukken 2 en 3, en verder bevestigd in tokenisatie-experimenten in Hoofdstuk 4; (2) Betere data verslaat meer data: hoogwaardige, domeinrelevante voorbeelden presteren consequent beter dan grote ho- eveelheden generieke data, zoals aangetoond in Hoofdstuk 2 en opnieuw onderzocht vanuit een woordenschat-perspectief in Hoofdstuk 4; en (3) Adaptatie moet efficiĂ«nt zijn: van gerichte dataselectie tot hybride tokenisatie en QE-gestuurde ICLâ-de in dit proefschrift voorgestelde methoden bieden praktische handvatten om computationele en ecol- ogische kosten te verlagen terwijl de taakprestatie behouden blijft of verbetert, zoals onderzocht in Hoofdstukken 3, 4 en 5. Samenvattend draagt dit proefschrift bij aan zowel theorie als prak- tijk van domeinadapatie door de volledige MT-keten te onderzoeken- âvan dataselectie en tokenisatie tot inferentie voor MT-systemen en prompting voor LLMâs, evenals kwaliteitsbeoordeling en -inschatting. Bovendien levert het reproduceerbare tools en gedetailleerde computa- 4 BM25(Robertson & Zaragoza, 2009) rangschikt documentenâhier: zinnenâop basis van hun relevantie voor een zoekopdracht. 186 tionele analyses die duurzame, domeinbewuste MT-workflows mogelijk maken. De bevindingen weerspiegelen een bredere verschuiving in het vakgebied, weg van puur modelcentrische schaalvergroting, richting benaderingen die datakwaliteit, domeinspecificiteit en computationele efficiĂ«ntie centraal stellen. Nu MT- en QE-systemen steeds vaker wor- den ingezet in professionele praktijken, bieden de ontwikkelde methoden een solide basis voor het bouwen van systemen die robuuster, beter aanpasbaar, en beter afgestemd zijn op de eisen van gespecialiseerde domeinen, waaronder die van taaldienstverleners en andere industriĂ«le gebruikers. 187 188 Acknowledgments A PhD, in many ways, can be viewed as an extreme form of domain adaptation. You start as a generally trained person with a few default settings, and then go through years of fine-tuningâhoping that by the end, you become something stable and useful, without too many exploding gradients along the way. Whether that metaphor truly matches reality is debatable. In any case, this dissertation is the output of that long training process. It would not have been possible without the many people who guided me, supported me, and sometimes rescued me. I group them here using the domain adaptation metaphorânot because people fit neatly into categories, but because it adds a bit of structure (and humor) to the gratitude I owe them. General-domain pretraining folks These are the people who helped build the base model I started with and gently pulled me away from becoming too task-specificâthose who shaped my values, kept me grounded, and made it possible for me to grow. I begin with Mahsa. Your support has been steady and essential. Moving to a new country, far from our families, was not easy, but your courage, trust, and patience made even the hard moments manageable. Your belief in me gave me the strength to do a PhD with a calmer mind and a hopeful heart. Nothing in this work would have been possible without your constant support. Love you. My brother Mohammad has been a strong source of support. Even from far away, your humor and belief in me always reached me at the right time. The PhD may have pushed me down a few levels in FIFA, but I promise to restore that soonâalong with our banter. My thanks also go to my mom and dad, and to my parents-in-law. Your support reached us in many waysâbig and smallâand made life abroad feel much lighter. Thank you for cheering us on from afar. Fine-tuning folks These are the people who guided the task-specific part of my trainingâmy PhD supervisors. To Dimitar: you taught me not only how to do research, but also how to grow in academia. From teaching to supervision, thank you for giving me the opportunity to learn and improve. Your support went far beyond papers and experiments; you helped me through both the good moments and the challenging ones. You were always there, from beginning to end, willing to help. Thank you for the role you played in my growth. And to Pieter: I could not have asked for a better promoter. You treated me as a colleagueânot just a studentâand I appreciated how you valued my ideas and gave me the space to express what mattered to me. It meant more than I can say and shaped my PhD experience in the best possible way. And of course, our board game gatherings were perfect breaks at exactly the right moments. Thank you. I also want to highlight an important point: both of you were always present, available, and engaged throughout my PhD. This may sound normal, but I know it is not always the case. Your presence and support made the journey much smoother, and I am thankful for that. Parallel corpus folks These are the people who kept me grounded, positive, and supported throughout the ups and downs of PhD life. I met many wonderful individuals along the way, and I am grateful to 190 each of them. This list could easily be longer, and if you expected to find your name here but do notâyou still made a meaningful impact, and I appreciate you. I begin with Chris Emmery, whose positive energy and generosity in sharing his experiences meant a great deal to meâthank you, Chris. I also thank Stijn Rotman (a.k.a. âTrue Dutchâ) for our post-submission beer sessions, and Samaneh Khoshrou for her positivity and our food discussions that always made me hungry. I am grateful to Marijn van Wingerden for his guidance and for giving me room to grow; to Marie Postma for her openness and support when I stepped in as a lecturer; and to Drew Hendrickson for his approachabilityâespecially the random corridor chats that helped me reset between breaks. My thanks also go to colleagues from our machine translation group who shaped my journey in their own ways: Mirella De Sisto, Eva Vanmassenhove, and Fred Blain. And to two of the bestâKarin Berkhout and Eva Verschoor-Suitela. Thanks for your patience in indulging my attempts to learn Dutch, and for helping me navigate the many requests that came my way. And finally, thanks to my great friends who have helped in various ways to make my moments more fun: Marcel van Groenendaal, Reza Bakhtiari, Mahyar Ghazanfari, Alireza Farzad, and Kambiz Darvish- nezhad. You guys really did me a solid. 191 References Adlaon, K. M. M., & Marcos, N. (2024, November). Finding the optimal byte-pair encoding merge operations for neural machine translation in a low-resource setting. In Y. Al-Onaizan, M. Bansal, & Y.-N. Chen (Eds.), Findings of the association for computa- tional linguistics: Emnlp 2024 (p. 14673â14682). Miami, Florida, USA: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2024.findings-emnlp.860/doi: 10.18653/v1/2024.findings-emnlp.860 Agrawal, S., Zhou, C., Lewis, M., Zettlemoyer, L., & Ghazvininejad, M. (2023, July). In-context examples selection for machine translation. In A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Findings of the association for computational linguistics: Acl 2023 (p. 8857â8873). Toronto, Canada: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2023 .findings-acl.564 doi: 10.18653/v1/2023.findings-acl.564 Aharoni, R., & Goldberg, Y. (2020, July). Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th annual meeting of the association for computational linguistics (p. 7747â 7763). Online: Association for Computational Linguistics. Re- trieved fromhttps://aclanthology.org/2020.acl-main.692 doi: 10.18653/v1/2020.acl-main.692 Alves, D., Guerreiro, N., Alves, J., Pombal, J., Rei, R., de Souza, J., . . . Martins, A. (2023, December). Steering large lan- guage models for machine translation with finetuning and in- context learning. In H. Bouamor, J. Pino, & K. Bali (Eds.), Findings of the association for computational linguistics: Emnlp 2023 (p. 11127â11148). Singapore: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2023.findings-emnlp.744doi: 10.18653/v1/2023.findings 192 -emnlp.744 Ataman, D., Negri, M., Turchi, M., & Federico, M. (2017, 06). Lin- guistically motivated vocabulary reduction for neural machine translation from turkish to english. The Prague Bulletin of Math- ematical Linguistics, 108. doi: 10.1515/pralin-2017-0031 Axelrod, A., He, X., & Gao, J. (2011, July). Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 con- ference on empirical methods in natural language processing (p. 355â362). Edinburgh, Scotland, UK.: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ D11-1033 Baek, Y., Kim, Z. M., Moon, J., Kim, H., & Park, E. (2020, November). PATQUEST: Papago translation quality estimation. In Proceed- ings of the fifth conference on machine translation (p. 991â998). Online: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2020.wmt-1.113 Bawden, R., & Yvon, F. (2023, June). Investigating the transla- tion performance of a large multilingual language model: the case of BLOOM. In M. Nurminen et al. (Eds.), Proceedings of the 24th annual conference of the European association for machine translation (p. 157â170). Tampere, Finland: Eu- ropean Association for Machine Translation. Retrieved from https://aclanthology.org/2023.eamt-1.16 Blain, F., Zerva, C., Rei, R., Guerreiro, N. M., Kanojia, D., C. de Souza, J. G., . . . Martins, A. (2023, December). Findings of the WMT 2023 shared task on quality estimation. In P. Koehn, B. Haddow, T. Kocmi, & C. Monz (Eds.), Proceedings of the eighth conference on machine translation (p. 629â653). Singa- pore: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2023.wmt-1.52doi: 10.18653/v1/ 2023.wmt-1.52 193 Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017). Enriching word vectors with subword information (Vol. 5). Cambridge, MA: MIT Press. Retrieved fromhttps://aclanthology.org/ Q17-1010 doi: 10.1162/tacl_a_00051 Briva-Iglesias, V. (2025). Are AI agents the new machine translation frontier? challenges and opportunities of single- and multi-agent systems for multilingual digital communication. Retrieved from https://arxiv.org/abs/2504.12891 Bromley, J., Guyon, I., LeCun, Y., SĂ€ckinger, E., & Shah, R. (1993). Signature verification using a "siamese" time delay neural network. In Proceedings of the 6th international conference on neural in- formation processing systems (p. 737â744). San Francisco, CA, USA: Morgan Kaufmann Publishers Inc. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhari- wal, P., . . . Amodei, D. (2020). Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, & H. Lin (Eds.), Advances in neural information processing sys- tems (Vol. 33, p. 1877â1901). Curran Associates, Inc. Retrieved fromhttps://proceedings.neurips.c/paper_files/paper/ 2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., & Specia, L. (Eds.). (2012, June). Proceedings of the sev- enth workshop on statistical machine translation. MontrĂ©al, Canada: Association for Computational Linguistics. Retrieved from https://aclanthology.org/W12-3100 Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., & Specia, L. (2017, August). SemEval-2017 task 1: Semantic textual similarity multi- lingual and crosslingual focused evaluation. In Proceedings of the 11th international workshop on semantic evaluation (SemEval- 2017) (p. 1â14). Vancouver, Canada: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 194 S17-2001 doi: 10.18653/v1/S17-2001 Cettolo, M., Niehues, J., StĂŒker, S., Bentivogli, L., & Federico, M. (2014, December 4-5). Report on the 11th IWSLT evaluation campaign. In Proceedings of the 11th international workshop on spoken language translation: Evaluation campaign (p. 2â17). Lake Tahoe, California. Retrieved fromhttps://aclanthology .org/2014.iwslt-evaluation.1 Chakrabarty, A., Dabre, R., Ding, C., Utiyama, M., & Sumita, E. (2020, December). Improving low-resource NMT through rele- vance based linguistic features incorporation. In D. Scott, N. Bel, & C. Zong (Eds.), Proceedings of the 28th international conference on computational linguistics (p. 4263â4274). Barcelona, Spain (Online): International Committee on Computational Linguis- tics. Retrieved fromhttps://aclanthology.org/2020.coling -main.376/ doi: 10.18653/v1/2020.coling-main.376 Chen, B., Kuhn, R., Foster, G., Cherry, C., & Huang, F. (2016, October 28 - November 1). Bilingual methods for adaptive train- ing data selection for machine translation. In Conferences of the association for machine translation in the americas: MT researchersâ track (p. 93â106). Austin, TX, USA: The Associ- ation for Machine Translation in the Americas. Retrieved from https://aclanthology.org/2016.amta-researchers.8 Chopra, S., Kothawade, S., Aynaou, H., & Chadha, A. (2023). Tran- scending domains through text-to-image diffusion: A source-free approach to domain adaptation. arXiv.org, abs/2310.01701. Re- trieved fromhttps://export.arxiv.org/pdf/2310.01701v2 .pdf doi: 10.48550/arxiv.2310.01701 Chu, C., & Dabre, R. (2019). Multilingual multi-domain adaptation ap- proaches for neural machine translation. ArXiv, abs/1906.07978. Chu, C., Dabre, R., & Kurohashi, S. (2017, July). An empirical comparison of domain adaptation methods for neural machine 195 translation. In Proceedings of the 55th annual meeting of the asso- ciation for computational linguistics (volume 2: Short papers) (p. 385â391). Vancouver, Canada: Association for Computational Lin- guistics. Retrieved fromhttps://aclanthology.org/P17-2061 doi: 10.18653/v1/P17-2061 Chu, C., & Wang, R. (2018, August). A survey of domain adaptation for neural machine translation. In Proceedings of the 27th inter- national conference on computational linguistics (p. 1304â1319). Santa Fe, New Mexico, USA: Association for Computational Lin- guistics. Retrieved fromhttps://aclanthology.org/C18-1111 Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., GuzmĂĄn, F., . . . Stoyanov, V. (2020, July). Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th annual meeting of the association for computational linguistics (p. 8440â8451). Online: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2020.acl-main.747 doi: 10.18653/v1/2020.acl-main.747 Courty, B., Schmidt, V., Luccioni, S., Goyal-Kamal, MarionCoutarel, Feld, B., . . . MinervaBooks (2024, May). mlco2/codecarbon: v2.4.1. Zenodo. Retrieved fromhttps://doi.org/10.5281/ zenodo.11171501 doi: 10.5281/zenodo.11171501 Currey, A., Miceli Barone, A. V., & Heafield, K. (2017, Septem- ber). Copied monolingual data improves low-resource neural machine translation. In Proceedings of the second conference on machine translation (p. 148â156). Copenhagen, Denmark: Asso- ciation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/W17-4715 doi: 10.18653/v1/W17-4715 Dakwale, P., & Monz, C. (2017). Fine-tuning for neural machine translation with limited degradation across in- and out-of-domain data. In Proceedings of the 16th machine translation summit (MT-summit 2017) (p. 156â169). 196 de Souza, J. G., Turchi, M., & Negri, M. (2014a, August). Machine translation quality estimation across domains. In Proceedings of COLING 2014, the 25th international conference on computa- tional linguistics: Technical papers (p. 409â420). Dublin, Ireland: Dublin City University and Association for Computational Lin- guistics. Retrieved fromhttps://aclanthology.org/C14-1040 de Souza, J. G., Turchi, M., & Negri, M. (2014b, October 22). Towards a combination of online and multitask learning for MT quality estimation: a preliminary study. In Workshop on interactive and adaptive machine translation (p. 9â19). Vancouver, Canada: Association for Machine Translation in the Americas. Retrieved from https://aclanthology.org/2014.amta-workshop.2 Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for lan- guage understanding. In Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (p. 4171â4186). Minneapolis, Minnesota: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/N19-1423 doi: 10.18653/v1/N19-1423 Dinh, T. A., & Niehues, J. (2023, September). Perturbation-based QE: An explainable, unsupervised word-level quality estimation method for blackbox machine translation. In M. Utiyama & R. Wang (Eds.), Proceedings of machine translation summit xix, vol. 1: Research track (p. 59â71). Macau SAR, China: Asia- Pacific Association for Machine Translation. Retrieved from https://aclanthology.org/2023.mtsummit-research.6 Duh, K., Neubig, G., Sudoh, K., & Tsukada, H. (2013, August). Adap- tation data selection using neural language models: Experiments in machine translation. In Proceedings of the 51st annual meeting of the association for computational linguistics (volume 2: Short 197 papers) (p. 678â683). Sofia, Bulgaria: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ P13-2119 Eo, S., Park, C., Seo, J., Moon, H., & Lim, H. (2021). A new tool for efficiently generating quality estimation datasets. arXiv. Retrieved fromhttps://arxiv.org/abs/2111.00767doi: 10 .48550/ARXIV.2111.00767 Eronen, J., Ptaszynski, M., Nowakowski, K., Chia, Z. L., & Masui, F. (2023). Improving polish to english neural machine translation with transfer learning: Effects of data volume and language similarity. Retrieved from https://arxiv.org/abs/2306.00660 Eschbach-Dymanus, J., Essenberger, F., Buschbeck, B., & Exel, M. (2024, June). Exploring the effectiveness of LLM domain adap- tation for business IT machine translation. In C. Scarton et al. (Eds.), Proceedings of the 25th annual conference of the Euro- pean association for machine translation (volume 1) (p. 610â 622). Sheffield, UK: European Association for Machine Trans- lation (EAMT). Retrieved fromhttps://aclanthology.org/ 2024.eamt-1.51/ Fadaee, M., Bisazza, A., & Monz, C. (2017, July). Data augmentation for low-resource neural machine translation. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: Short papers) (p. 567â573). Vancouver, Canada: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/P17-2090doi: 10.18653/v1/ P17-2090 Fomicheva, M., Sun, S., Fonseca, E., Zerva, C., Blain, F., Chaudhary, V., . . . Martins, A. F. T. (2020, October). MLQE-PE: A Multilingual Quality Estimation and Post-Editing Dataset. arXiv e-prints, arXiv:2010.04480. doi: 10.48550/arXiv.2010.04480 Fomicheva, M., Sun, S., Fonseca, E., Zerva, C., Blain, F., Chaudhary, 198 V., . . . Martins, A. F. T. (2022, June). MLQE-PE: A multilingual quality estimation and post-editing dataset. In Proceedings of the thirteenth language resources and evaluation conference (p. 4963â4974). Marseille, France: European Language Resources Association. Retrieved fromhttps://aclanthology.org/2022 .lrec-1.530 Fomicheva, M., Sun, S., Yankovskaya, L., Blain, F., GuzmĂĄn, F., Fishel, M., . . . Specia, L. (2020). Unsupervised quality es- timation for neural machine translation. Transactions of the Association for Computational Linguistics, 8, 539â555. Re- trieved fromhttps://aclanthology.org/2020.tacl-1.35doi: 10.1162/tacl_a_00330 Freitag, M., & Al-Onaizan, Y. (2016). Fast domain adaptation for neural machine translation. Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., & Macherey, W. (2021). Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9, 1460â1474. Retrieved fromhttps://aclanthology.org/2021.tacl-1.87/ doi: 10.1162/tacl_a_00437 Freitag, M., Rei, R., Mathur, N., Lo, C.-k., Stewart, C., Foster, G., . . . Bojar, O. (2021, November). Results of the WMT21 metrics shared task: Evaluating metrics with expert-based hu- man evaluations on TED and news domain. In Proceedings of the sixth conference on machine translation (p. 733â774). On- line: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2021.wmt-1.73 Goyal, N., Gao, C., Chaudhary, V., Chen, P.-J., Wenzek, G., Ju, D., . . . Fan, A. (2022). The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10, 522â538. Re- 199 trieved fromhttps://aclanthology.org/2022.tacl-1.30doi: 10.1162/tacl_a_00474 Goyal, T., Li, J. J., & Durrett, G. (2023). News summarization and evaluation in the era of gpt-3. Graham, Y., Baldwin, T., Moffat, A., & Zobel, J. (2013, August). Continuous measurement scales in human evaluation of ma- chine translation. In A. Pareja-Lora, M. Liakata, & S. Dip- per (Eds.), Proceedings of the 7th linguistic annotation work- shop and interoperability with discourse (p. 33â41). Sofia, Bul- garia: Association for Computational Linguistics. Retrieved from https://aclanthology.org/W13-2305/ Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., de Laroussilhe, Q., Gesmundo, A., . . . Gelly, S. (2019). Parameter-efficient transfer learning for nlp. arXiv. Retrieved from https://arxiv .org/abs/1902.00751 doi: 10.48550/ARXIV.1902.00751 Hutchins, J. (2006). The first public demonstration of machine transla- tion : the georgetown-ibm system , 7 th january 1954.. Retrieved from https://api.semanticscholar.org/CorpusID:132677 Hwang, D., Oh, S. J., & Choe, J. (2024). Overcoming domain limitations in open-vocabulary segmentation. x. doi: 10.48550/arxiv.2410 .11536 Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020, 07). How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8, 423-438. Retrieved fromhttps://doi.org/10.1162/tacl_a_00324doi: 10.1162/ tacl_a_00324 Jolliffe, I. (2011). Principal component analysis. In M. Lovric (Ed.), International encyclopedia of statistical science (p. 1094â1096). Berlin, Heidelberg: Springer Berlin Heidelberg. doi: 10.1007/ 978-3-642-04898-2\_455 Klein, G., Kim, Y., Deng, Y., Senellart, J., & Rush, A. (2017, July). 200 OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, system demonstrations (p. 67â72). Vancouver, Canada: Association for Computational Linguistics. Retrieved from https://aclanthology.org/P17-4012 Kobus, C., Crego, J., & Senellart, J. (2017, Nov). Domain control for neural machine translation. RANLP 2017 - Recent Advances in Natural Language Processing Meet Deep Learning. doi: 10.26615/ 978-954-452-049-6_049 Kocyigit, M., Lee, J., & Wijaya, D. (2022, May). Better quality estima- tion for low resource corpus mining. In Findings of the association for computational linguistics: Acl 2022 (p. 533â543). Dublin, Ireland: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2022.findings-acl.45doi: 10.18653/v1/2022.findings-acl.45 Koehn, P. (2005, September 13-15). Europarl: A parallel corpus for statistical machine translation. In Proceedings of machine transla- tion summit x: Papers (p. 79â86). Phuket, Thailand. Retrieved fromhttps://aclanthology.org/2005.mtsummit-papers.11 Koehn, P. (2020). History. In Neural machine translation (p. 29â40). Cambridge University Press. Koehn, P., Haddow, B., Kocmi, T., & Monz, C. (Eds.). (2023, Decem- ber). Proceedings of the eighth conference on machine translation. Singapore: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2023.wmt-1.0 Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., . . . Herbst, E. (2007, June). Moses: Open source toolkit for statistical machine translation. In S. Ananiadou (Ed.), Proceedings of the 45th annual meeting of the association for computational linguistics companion volume proceedings of the demo and poster sessions (p. 177â180). Prague, Czech Repub- lic: Association for Computational Linguistics. Retrieved from 201 https://aclanthology.org/P07-2045 Koehn, P., & Knowles, R. (2017, August). Six challenges for neu- ral machine translation. In Proceedings of the first workshop on neural machine translation (p. 28â39). Vancouver: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/W17-3204 doi: 10.18653/v1/W17-3204 Kolmogorov, A. N. (1933). Sulla determinazione empirica di una legge di distribuzione. Giornale dellâIstituto Italiano degli Attuari, 4, 83â91. Kudo, T. (2018, July). Subword regularization: Improving neural network translation models with multiple subword candidates. In I. Gurevych & Y. Miyao (Eds.), Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers) (p. 66â75). Melbourne, Australia: Associa- tion for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/P18-1007/ doi: 10.18653/v1/P18-1007 Kudo, T., & Richardson, J. (2018, November). SentencePiece: A simple and language independent subword tokenizer and detok- enizer for neural text processing. In E. Blanco & W. Lu (Eds.), Proceedings of the 2018 conference on empirical methods in nat- ural language processing: System demonstrations (p. 66â71). Brussels, Belgium: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/D18-2012/doi: 10.18653/v1/D18-2012 Kumar, A., Puduppully, R., Dabre, R., & Kunchukuttan, A. (2023, December). CTQScorer: Combining multiple features for in- context example selection for machine translation. In H. Bouamor, J. Pino, & K. Bali (Eds.), Findings of the association for com- putational linguistics: Emnlp 2023 (p. 7736â7752). Singa- pore: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2023.findings-emnlp.519doi: 202 10.18653/v1/2023.findings-emnlp.519 Lee, D. (2020, November). Two-phase cross-lingual language model fine- tuning for machine translation quality estimation. In Proceedings of the fifth conference on machine translation (p. 1024â1028). Online: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2020.wmt-1.118 Li, Y., Guo, Y., Guerin, F., & Lin, C. (2024). Evaluating large language models for generalization and robustness via data compression. Retrieved from https://arxiv.org/abs/2402.00861 Lim, J., & Kim, Y. (2024). Cross-domain semantic segmentation on inconsistent taxonomy using vlms. Retrieved fromhttps:// arxiv.org/abs/2408.02261 Lim, R., Heafield, K., Hoang, H., Briers, M., & Malony, A. (2018). Exploring hyper-parameter optimization for neural machine trans- lation on gpu architectures. Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., . . . Li, X. (2022, December). Few-shot learning with multilingual generative language models. In Y. Goldberg, Z. Kozareva, & Y. Zhang (Eds.), Proceedings of the 2022 conference on empirical methods in natural language processing (p. 9019â9052). Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2022 .emnlp-main.616 doi: 10.18653/v1/2022.emnlp-main.616 Lin, Y.-H., Chen, C.-Y., Lee, J., Li, Z., Zhang, Y., Xia, M., . . . Neubig, G. (2019, July). Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th annual meet- ing of the association for computational linguistics (p. 3125â 3135). Florence, Italy: Association for Computational Linguis- tics. Retrieved fromhttps://aclanthology.org/P19-1301doi: 10.18653/v1/P19-1301 Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022, 203 May). What makes good in-context examples for GPT-3? In E. Agirre, M. Apidianaki, & I. VuliÄ (Eds.), Proceedings of deep learning inside out (deelio 2022): The 3rd workshop on knowledge extraction and integration for deep learning architectures (p. 100â 114). Dublin, Ireland and Online: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2022 .deelio-1.10 doi: 10.18653/v1/2022.deelio-1.10 Liu, Y., Gu, J., Goyal, N., Li, X., Edunov, S., Ghazvininejad, M., . . . Zettlemoyer, L. (2020, 11). Multilingual Denoising Pre-training for Neural Machine Translation. Transactions of the Association for Computational Linguistics, 8, 726-742. Retrieved fromhttps:// doi.org/10.1162/tacl_a_00343doi: 10.1162/tacl_a_00343 Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., . . . Stoyanov, V. (2019). Roberta: A robustly optimized bert pretraining approach. Lommel, A., Burchardt, A., PopoviÄ, M., Harris, K., Avramidis, E., & Uszkoreit, H. (2014, June 16-18). Using a new analytic measure for the annotation and analysis of MT errors on real data. In M. Cettolo, M. Federico, L. Specia, & A. Way (Eds.), Proceedings of the 17th annual conference of the European association for machine translation (p. 165â172). Dubrovnik, Croatia: European Association for Machine Translation. Retrieved fromhttps:// aclanthology.org/2014.eamt-1.38/ Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022, May). Fantastically ordered prompts and where to find them: Overcom- ing few-shot prompt order sensitivity. In S. Muresan, P. Nakov, & A. Villavicencio (Eds.), Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long pa- pers) (p. 8086â8098). Dublin, Ireland: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2022.acl-long.556 doi: 10.18653/v1/2022.acl-long.556 Luo, M., Xu, X., Dai, Z., Pasupat, P., Kazemi, M., Baral, C., . . . 204 Zhao, V. Y. (2023). Dr.icl: Demonstration-retrieved in-context learning. Luong, M.-T., & Manning, C. (2015, December 3-4). Stanford neu- ral machine translation systems for spoken language domains. In Proceedings of the 12th international workshop on spoken language translation: Evaluation campaign (p. 76â79). Da Nang, Vietnam. Retrieved fromhttps://aclanthology.org/ 2015.iwslt-evaluation.11 Luong, T., Pham, H., & Manning, C. D. (2015, September). Effective approaches to attention-based neural machine translation. In Pro- ceedings of the 2015 conference on empirical methods in natural language processing (p. 1412â1421). Lisbon, Portugal: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/D15-1166 doi: 10.18653/v1/D15-1166 Mahdieh, M., Chen, M. X., Cao, Y., & Firat, O. (2020). Rapid domain adaptation for machine translation with monolingual data. Mattoni, G., Nagle, P., Collantes, C., & Shterionov, D. (2017, Septem- ber 18 â September 22). Zero-shot translation for Indian languages with sparse data. In M. Yamada & M. Seligman (Eds.), Proceed- ings of machine translation summit xvi: Commercial MT users and translators track (p. 1â10). Nagoya Japan. Retrieved from https://aclanthology.org/2017.mtsummit-commercial.1 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. Mittal, S., Hrinchuk, O., & Kuchaiev, O. (2023, July). Leverag- ing synthetic targets for machine translation. In A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Findings of the associ- ation for computational linguistics: Acl 2023 (p. 9365â9379). Toronto, Canada: Association for Computational Linguistics. Re- trieved fromhttps://aclanthology.org/2023.findings-acl .597/ doi: 10.18653/v1/2023.findings-acl.597 205 Moore, R. C., & Lewis, W. (2010, July). Intelligent selection of language model training data. In Proceedings of the ACL 2010 conference short papers (p. 220â224). Uppsala, Swe- den: Association for Computational Linguistics. Retrieved from https://aclanthology.org/P10-2041 Moslem, Y., Haque, R., Kelleher, J., & Way, A. (2022, September). Domain-specific text generation for machine translation. In Pro- ceedings of the 15th biennial conference of the association for machine translation in the americas (volume 1: Research track) (p. 14â30). Orlando, USA: Association for Machine Translation in the Americas. Retrieved fromhttps://aclanthology.org/ 2022.amta-research.2 Murgolo, E., Sharami, J. P. R., & Shterionov, D. (2022, June). A quality estimation and quality evaluation tool for the transla- tion industry. In Proceedings of the 23rd annual conference of the European association for machine translation (p. 307â308). Ghent, Belgium: European Association for Machine Translation. Retrieved from https://aclanthology.org/2022.eamt-1.43 Naseem, U., Razzak, I., Khan, S. K., & Prasad, M. (2021, June). A com- prehensive survey on word representation models: From classical to state-of-the-art word representation language models. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 20(5). Retrieved fromhttps://doi-org.tilburguniversity.idm.oclc.org/10 .1145/3434237 doi: 10.1145/3434237 Negri, M., Turchi, M., Chatterjee, R., & Bertoldi, N. (2018, May). ES- CAPE: a large-scale synthetic corpus for automatic post-editing. In Proceedings of the eleventh international conference on lan- guage resources and evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association (ELRA). Retrieved from https://aclanthology.org/L18-1004 Nirenburg, S., Somers, H. L., & Wilks, Y. (2003). Alpac: The (in)famous 206 report.. Retrieved fromhttps://api.semanticscholar.org/ CorpusID:63500288 Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002a). Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics (p. 311â318). USA: Association for Computational Linguistics. Retrieved fromhttps://doi-org.tilburguniversity.idm.oclc.org/10 .3115/1073083.1073135 doi: 10.3115/1073083.1073135 Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002b, July). Bleu: a method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the asso- ciation for computational linguistics (p. 311â318). Philadel- phia, Pennsylvania, USA: Association for Computational Linguis- tics. Retrieved fromhttps://aclanthology.org/P02-1040doi: 10.3115/1073083.1073135 Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002c, July). BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th annual meeting on association for computational linguistics (acl 2002) (p. 311â318). Philadelphia, Pennsylvania, USA: Association for Computational Linguistics. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., . . . others (2011). Scikit-learn: Machine learning in python. Journal of machine learning research, 12(Oct), 2825â 2830. Peng, K., Ding, L., Yuan, Y., Liu, X., Zhang, M., Ouyang, Y., & Tao, D. (2024, August). Revisiting demonstration selection strategies in in-context learning. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) (p. 9090â9101). Bangkok, Thailand: Association for Computa- 207 tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2024.acl-long.492/ doi: 10.18653/v1/2024.acl-long.492 Petrov, A., Malfa, E. L., Torr, P. H. S., & Bibi, A. (2023). Language model tokenizers introduce unfairness between languages. Pham, M.-Q., Crego, J., & Yvon, F. (2022, June). Multi-domain adaptation in neural machine translation with dynamic sampling strategies. In Proceedings of the 23rd annual conference of the European association for machine translation (p. 13â22). Ghent, Belgium: European Association for Machine Translation. Re- trieved from https://aclanthology.org/2022.eamt-1.4 Pham, N.-Q., Nguyen, T. N., Nguyen, T.-B., Liu, D., Mullov, C., Niehues, J., & Waibel, A. (2022, May). Effective combination of pretrained models - KIT@IWSLT2022. In E. Salesky, M. Federico, & M. Costa-jussĂ (Eds.), Proceedings of the 19th international con- ference on spoken language translation (iwslt 2022) (p. 190â197). Dublin, Ireland (in-person and online): Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2022.iwslt-1.14 doi: 10.18653/v1/2022.iwslt-1.14 Poncelas, A., Shterionov, D., Way, A., Maillette de Buy Wenniger, G., & Passban, P. (2018, May). Investigating backtranslation in neural machine translation. In J. A. PĂ©rez-Ortiz et al. (Eds.), Proceedings of the 21st annual conference of the European association for machine translation (p. 269â278). Alicante, Spain. Retrieved from https://aclanthology.org/2018.eamt-main.25/ PopoviÄ, M. (2015, September). chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the tenth work- shop on statistical machine translation (p. 392â395). Lisbon, Portugal: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/W15-3049doi: 10.18653/v1/ W15-3049 Post, M. (2018, October). A call for clarity in reporting BLEU 208 scores. In Proceedings of the third conference on machine trans- lation: Research papers (p. 186â191). Brussels, Belgium: Asso- ciation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/W18-6319 doi: 10.18653/v1/W18-6319 Pourmostafa Roshan Sharami, J., Shterionov, D., & Spronck, P. (2021, July 9). A novel pipeline for domain detection and select- ing in-domain sentences in machine translation systems.. Re- trieved fromhttps://w.clin31.ugent.be/(The 31st Meet- ing of Computational Linguistics in The Netherlands (CLIN 31) ; Conference date: 09-07-2021 Through 09-07-2021) doi: 10.6084/m9.figshare.14829030.v2 Pourmostafa Roshan Sharami, J., Shterionov, D., & Spronck, P. (2021, December). Selecting Parallel In-domain Sentences for Neural Machine Translation Using Monolingual Texts. arXiv e-prints, arXiv:2112.06096. doi: 10.48550/arXiv.2112.06096 Qiu, B., Ding, L., Wu, D., Shang, L., Zhan, Y., & Tao, D. (2022, De- cember). Original or Translated? On the Use of Parallel Data for Translation Quality Estimation. arXiv e-prints, arXiv:2212.10257. doi: 10.48550/arXiv.2212.10257 Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners.. Retrieved fromhttps://api.semanticscholar.org/CorpusID: 160025533 Ranasinghe, T., Orasan, C., & Mitkov, R. (2020, December). Tran- sQuest: Translation quality estimation with cross-lingual trans- formers. In Proceedings of the 28th international conference on computational linguistics (p. 5070â5081). Barcelona, Spain (Online): International Committee on Computational Linguis- tics. Retrieved fromhttps://aclanthology.org/2020.coling -main.445 doi: 10.18653/v1/2020.coling-main.445 Ranjan, V., Harit, G., & Jawahar, C. V. (2015). Learning partially 209 shared dictionaries for domain adaptation. In C. V. Jawahar & S. Shan (Eds.), Computer vision - accv 2014 workshops (p. 247â261). Cham: Springer International Publishing. Raunak, V., Menezes, A., & Awadalla, H. (2023, December). Dissecting in-context learning of translations in GPT-3. In H. Bouamor, J. Pino, & K. Bali (Eds.), Findings of the association for com- putational linguistics: Emnlp 2023 (p. 866â872). Singapore: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2023.findings-emnlp.61doi: 10.18653/v1/2023.findings-emnlp.61 Raunak, V., Menezes, A., Post, M., & Hassan, H. (2023, July). Do GPTs produce less literal translations? In A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st annual meeting of the association for computational linguistics (volume 2: Short papers) (p. 1041â1050). Toronto, Canada: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2023 .acl-short.90 doi: 10.18653/v1/2023.acl-short.90 Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020, Novem- ber). COMET: A neural framework for MT evaluation. In Pro- ceedings of the 2020 conference on empirical methods in natural language processing (emnlp) (p. 2685â2702). Online: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/2020.emnlp-main.213doi: 10.18653/v1/ 2020.emnlp-main.213 Reimers, N., & Gurevych, I. (2019, November). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Pro- ceedings of the 2019 conference on empirical methods in natu- ral language processing and the 9th international joint confer- ence on natural language processing (emnlp-ijcnlp) (p. 3982â 3992). Hong Kong, China: Association for Computational Lin- guistics. Retrieved fromhttps://aclanthology.org/D19-1410 210 doi: 10.18653/v1/D19-1410 Robertson, S., & Zaragoza, H. (2009, apr). The probabilistic rel- evance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4), 333â389. Retrieved fromhttps://doi.org/10.1561/ 1500000019 doi: 10.1561/1500000019 Rostam, Z. R. K., & KertĂ©sz, G. (2024). Fine-tuning large language models for scientific text classification: A comparative study. In 2024 ieee 6th international symposium on logistics and industrial informatics (lindi) (p. 000233-000238). doi: 10.1109/LINDI63813 .2024.10820432 Rubino, R. (2020, November). NICT Kyoto submission for the WMTâ20 quality estimation task: Intermediate training for domain and task adaptation. In Proceedings of the fifth conference on machine translation (p. 1042â1048). Online: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2020.wmt-1.121 Rubino, R., & Sumita, E. (2020, December). Intermediate self- supervised learning for machine translation quality estimation. In Proceedings of the 28th international conference on computational linguistics (p. 4355â4360). Barcelona, Spain (Online): Inter- national Committee on Computational Linguistics. Retrieved fromhttps://aclanthology.org/2020.coling-main.385doi: 10.18653/v1/2020.coling-main.385 Ruder, S. (2016). On word embeddings - Part 2: Approximating the Softmax. http://ruder.io/word-embeddings-softmax. Salesky, E., Runge, A., Coda, A., Niehues, J., & Neubig, G. (2018). Op- timizing segmentation granularity for neural machine translation. Retrieved from https://arxiv.org/abs/1810.08641 Sato, S., Sakuma, J., Yoshinaga, N., Toyoda, M., & Kitsuregawa, M. (2020, November). Vocabulary adaptation for domain adaptation in neural machine translation. In Findings of the association 211 for computational linguistics: Emnlp 2020 (p. 4269â4279). On- line: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2020.findings-emnlp.381doi: 10.18653/v1/2020.findings-emnlp.381 Saunders, D. (2021). Domain adaptation and multi-domain adaptation for neural machine translation: A survey. Saunders, D. (2022). Domain adaptation and multi-domain adaptation for neural machine translation: A survey. Journal of Artificial Intelligence Research, 75, 351â424. Schroff, F., Kalenichenko, D., & Philbin, J. (2015, Jun). Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR). Retrieved fromhttp://dx.doi.org/10.1109/ CVPR.2015.7298682 doi: 10.1109/cvpr.2015.7298682 Sennrich, R., Haddow, B., & Birch, A. (2016a, June). Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 conference of the north American chap- ter of the association for computational linguistics: Human lan- guage technologies (p. 35â40). San Diego, California: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/N16-1005 doi: 10.18653/v1/N16-1005 Sennrich, R., Haddow, B., & Birch, A. (2016b, August). Neural machine translation of rare words with subword units. In Proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: Long papers) (p. 1715â1725). Berlin, Germany: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/P16-1162doi: 10.18653/v1/ P16-1162 Sharami, J. P. R., Shterionov, D., Blain, F., Vanmassenhove, E., Sisto, M. D., Emmery, C., & Spronck, P. (2023, June). Tailoring domain adaptation for machine translation quality estimation. 212 In M. Nurminen et al. (Eds.), Proceedings of the 24th annual conference of the European association for machine translation (p. 9â20). Tampere, Finland: European Association for Machine Translation. Retrieved fromhttps://aclanthology.org/2023 .eamt-1.2 Sharami, J. P. R., Shterionov, D., & Spronck, P. (2022). Selecting parallel in-domain sentences for neural machine translation using monolingual texts. Sharami, J. P. R., Shterionov, D., & Spronck, P. (2023, June). A Python tool for selecting domain-specific data in machine translation. In M. EsplĂ -Gomis et al. (Eds.), Proceedings of the 1st workshop on open community-driven machine translation (p. 29â30). Tam- pere, Finland: European Association for Machine Translation. Re- trieved from https://aclanthology.org/2023.crowdmt-1.4 Sia, S., & Duh, K. (2023, September). In-context learning as maintaining coherency: A study of on-the-fly machine translation using large language models. In M. Utiyama & R. Wang (Eds.), Proceedings of machine translation summit xix, vol. 1: Research track (p. 173â 185). Macau SAR, China: Asia-Pacific Association for Machine Translation. Retrieved fromhttps://aclanthology.org/2023 .mtsummit-research.15 Sinha, S., Premsri, T., & Kordjamshidi, P. (2024). A survey on compositional learning of AI models: Theoretical and experimental practices. Retrieved fromhttps://arxiv.org/abs/2406.08787 Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006, August 8-12). A study of translation edit rate with targeted human annotation. In Proceedings of the 7th conference of the association for machine translation in the americas: Technical papers (p. 223â231). Cambridge, Massachusetts, USA: Associ- ation for Machine Translation in the Americas. Retrieved from https://aclanthology.org/2006.amta-papers.25 213 Specia, L., Blain, F., Fomicheva, M., Zerva, C., Li, Z., Chaudhary, V., & Martins, A. F. T. (2021, November). Findings of the WMT 2021 shared task on quality estimation. In Proceedings of the sixth conference on machine translation (p. 684â725). Online: Association for Computational Linguistics. Retrieved from https://aclanthology.org/2021.wmt-1.71 Specia, L., & Farzindar, A. (2010, November 4). Estimating machine translation post-editing effort with HTER. In V. Zhechev (Ed.), Proceedings of the second joint EM+/CNGL workshop: Bringing MT to the user: Research on integrating MT in the translation industry (p. 33â43). Denver, Colorado, USA: Association for Machine Translation in the Americas. Retrieved fromhttps:// aclanthology.org/2010.jec-1.5 Stergiadis, E., Kumar, S., Kovalev, F., & Levin, P. (2021, August). Multi-domain adaptation in neural machine translation through multidimensional tagging. In Proceedings of machine translation summit xviii: Users and providers track (p. 396â420). Virtual: Association for Machine Translation in the Americas. Retrieved from https://aclanthology.org/2021.mtsummit-up.27 Tamchyna, A. (2021, August). Deploying MT quality estimation on a large scale: Lessons learned and open questions. In Proceedings of machine translation summit xviii: Users and providers track (p. 291â305). Virtual: Association for Machine Translation in the Americas. Retrieved fromhttps://aclanthology.org/ 2021.mtsummit-up.21 Tang, Y., Tran, C., Li, X., Chen, P.-J., Goyal, N., Chaudhary, V., . . . Fan, A. (2020). Multilingual translation with extensible multilingual pretraining and finetuning. Tezcan, A. (2022, Dec.). Integrating fuzzy matches into sentence-level quality estimation for neural machine translation. Computational Linguistics in the Netherlands Journal, 12, 99â123. Retrieved 214 from https://clinjournal.org/clinj/article/view/150 Tiedemann, J. (2012, May). Parallel data, tools and interfaces in OPUS. In Proceedings of the eighth international confer- ence on language resources and evaluation (LRECâ12) (p. 2214â 2218). Istanbul, Turkey: European Language Resources Asso- ciation (ELRA). Retrieved fromhttp://w.lrec-conf.org/ proceedings/lrec2012/pdf/463_Paper.pdf Tomas, N. (2025). BiLingual Evaluation Understudy (BLEU) Score Cheat Sheet.https://tutorialsdojo.com/ what-is-bilingual-evaluation-understudy-bleu-score -for-machine-translation/. (Accessed 2025-06-26) Trotman, A., Puurula, A., & Burgess, B. (2014). Improvements to bm25 and language models examined. In Proceedings of the 19th australasian document computing symposium (p. 58â65). New York, NY, USA: Association for Computing Machinery. Re- trieved fromhttps://doi.org/10.1145/2682862.2682863doi: 10.1145/2682862.2682863 van der Wees, M., Bisazza, A., & Monz, C. (2017, September). Dynamic data selection for neural machine translation. In Proceedings of the 2017 conference on empirical methods in natural language processing (p. 1400â1410). Copenhagen, Denmark: Associa- tion for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/D17-1147 doi: 10.18653/v1/D17-1147 Vanmassenhove, E., Hardmeier, C., & Way, A. (2018, October- November). Getting gender right in neural machine translation. In Proceedings of the 2018 conference on empirical methods in natural language processing (p. 3003â3008). Brussels, Belgium: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/D18-1334 doi: 10.18653/v1/D18-1334 Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017a). Attention is all you 215 need. In I. Guyon et al. (Eds.), Advances in neural informa- tion processing systems (Vol. 30). Curran Associates, Inc. Re- trieved fromhttps://proceedings.neurips.c/paper/2017/ file/3f5e243547dee91fbd053c1c4a845a-Paper.pdf Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., . . . Polosukhin, I. (2017b). Attention is all you need. In I. Guyon et al. (Eds.), Advances in neural informa- tion processing systems (Vol. 30). Curran Associates, Inc. Re- trieved fromhttps://proceedings.neurips.c/paper/2017/ file/3f5e243547dee91fbd053c1c4a845a-Paper.pdf Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., & Foster, G. (2023, July). Prompting PaLM for translation: Assessing strategies and performance. In A. Rogers, J. Boyd-Graber, & N. Okazaki (Eds.), Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers) (p. 15406â15427). Toronto, Canada: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2023.acl-long.859 doi: 10.18653/v1/2023.acl-long.859 Wan, Z., Zhang, Y., Wang, Y., Cheng, F., & Kurohashi, S. (2024). Reformulating domain adaptation of large language models as adapt-retrieve-revise: A case study on Chinese legal domain. Re- trieved from https://arxiv.org/abs/2310.03328 Wang, R., Finch, A., Utiyama, M., & Sumita, E. (2017, July). Sentence embedding for neural machine translation domain adaptation. In Proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: Short papers) (p. 560â566). Vancouver, Canada: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/P17-2089doi: 10 .18653/v1/P17-2089 Wang, X., Chen, C., & Xing, Z. (2019, December). Domain-specific machine translation with recurrent neural network for software 216 localization. Empirical Software Engineering, 24(6), 3514â3545. doi: 10.1007/s10664-019-09702-z Wassie, A. K., Molaei, M., & Moslem, Y. (2024). Domain-specific translation with open-source large language models: Resource- oriented analysis. Wu, M., Wang, Y., Foster, G., Qu, L., & Haffari, G. (2024, March). Importance-aware data augmentation for document-level neural machine translation. In Y. Graham & M. Purver (Eds.), Pro- ceedings of the 18th conference of the European chapter of the association for computational linguistics (volume 1: Long papers) (p. 740â752). St. Julianâs, Malta: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ 2024.eacl-long.44/ Wu, M., Xu, J., & Wang, L. (2024, November). TransAgents: Build your translation company with language agents. In D. I. Hernan- dez Farias, T. Hope, & M. Li (Eds.), Proceedings of the 2024 conference on empirical methods in natural language process- ing: System demonstrations (p. 131â141). Miami, Florida, USA: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2024.emnlp-demo.14/doi: 10.18653/v1/2024.emnlp-demo.14 Xiang, Y., Yan, H., Gui, L., & He, Y. (2024, August). Address- ing order sensitivity of in-context demonstration examples in causal language models. In L.-W. Ku, A. Martins, & V. Sriku- mar (Eds.), Findings of the association for computational lin- guistics: Acl 2024 (p. 6467â6481). Bangkok, Thailand: Associ- ation for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/2024.findings-acl.386/doi: 10.18653/ v1/2024.findings-acl.386 Xu, H., Kim, Y. J., Sharaf, A., & Awadalla, H. H. (2024). A paradigm shift in machine translation: Boosting translation performance of 217 large language models. arXiv. Ye, N., & Li, J. (2023). A k-nearest neighbor approach for domain- specific translation quality estimation. In Y. Feng & C. Feng (Eds.), Machine translation (p. 69â80). Singapore: Springer Nature Singapore. Yeung, C. M. (2019, November). Effects of inserting domain vocabulary and fine-tuning bert for german legal language. Retrieved from http://essay.utwente.nl/80128/ Yuan, B., Li, Y., Chen, K., Lu, H., Yang, M., & Cao, H. (2022). An improved multi-task approach to pre-trained model based MT quality estimation. In Ccmt. Retrieved fromhttps:// api.semanticscholar.org/CorpusID:254458050 Zerva, C., Blain, F., Rei, R., Lertvittayakumjorn, P., C. De Souza, J. G., Eger, S., . . . Specia, L. (2022, December). Find- ings of the WMT 2022 shared task on quality estimation. In Proceedings of the seventh conference on machine translation (wmt) (p. 69â99). Abu Dhabi, United Arab Emirates (Hy- brid): Association for Computational Linguistics. Retrieved from https://aclanthology.org/2022.wmt-1.3 Zhang, S., & Xiong, D. (2018, August). Sentence weighting for neural machine translation domain adaptation. In Proceedings of the 27th international conference on computational linguistics (p. 3181â 3190). Santa Fe, New Mexico, USA: Association for Computa- tional Linguistics. Retrieved fromhttps://aclanthology.org/ C18-1269 Zhang, T., Wu, F., Katiyar, A., Weinberger, K. Q., & Artzi, Y. (2020, June). Revisiting Few-sample BERT Fine-tuning. arXiv e-prints, arXiv:2006.05987. doi: 10.48550/arXiv.2006.05987 Zheng, J., Hong, H., Wang, X., Su, J., Liang, Y., & Wu, S. (2024). Fine-tuning large language models for domain-specific machine translation. arXiv.org, abs/2402.15061. doi: 10.48550/arxiv.2402 218 .15061 Zhu, W., Liu, H., Dong, Q., Xu, J., Huang, S., Kong, L., . . . Li, L. (2023). Multilingual machine translation with large language models: Empirical results and analysis. arXiv. Zimelewicz, E., Kalinowski, M., MĂ©ndez, D., Giray, G., Alves, A. P. S., Lavesson, N., . . . Gorschek, T. (2024). Ml-enabled systems model deployment and monitoring: Status quo and problems. In x (p. 112 - 131). Springer Science+Business Media. doi: 10.1007/978-3-031-56281-5_7 Zoph, B., Yuret, D., May, J., & Knight, K. (2016, November). Transfer learning for low-resource neural machine translation. In Pro- ceedings of the 2016 conference on empirical methods in natural language processing (p. 1568â1575). Austin, Texas: Associa- tion for Computational Linguistics. Retrieved fromhttps:// aclanthology.org/D16-1163 doi: 10.18653/v1/D16-1163 Zouhar, V., Chen, P., Lam, T. K., Moghe, N., & Haddow, B. (2024, November). Pitfalls and outlooks in using COMET. In B. Had- dow, T. Kocmi, P. Koehn, & C. Monz (Eds.), Proceedings of the ninth conference on machine translation (p. 1272â1288). Mi- ami, Florida, USA: Association for Computational Linguistics. Retrieved fromhttps://aclanthology.org/2024.wmt-1.121/ doi: 10.18653/v1/2024.wmt-1.121 Zouhar, V., Dhuliawala, S., Zhou, W., Daheim, N., Kocmi, T., Jiang, Y., & Sachan, M. (2023). Poor manâs quality estimation: Pre- dicting reference-based MT metrics without the reference. ArXiv, abs/2301.09008. Zouhar, V., Ding, S., Currey, A., Badeka, T., Wang, J., & Thompson, B. (2024). Fine-tuned machine translation metrics struggle in unseen domains. Retrieved fromhttps://arxiv.org/abs/2402.18747 219 220 222 Appendix A: Efficiency and Sustainability Overview This appendix provides an overview of the computational costs and efficiency considerations associated with the methodologies and exper- iments in this dissertation. As the scale and complexity of MT and QE systems continue to growâparticularly in settings that involve trainingâunderstanding the trade-offs between performance, resource consumption, and sustainability becomes increasingly important. The following sections summarize key findings from each chapter, focusing on metrics such as training time, inference speed, GPU utilization, energy consumption, and CO 2 emissions. These analyses demonstrate how the methodological choices adopted in this dissertationâranging from data selection and DA to SW tokenization and ICLâshape the efficiency and environmental footprint of MT and QE workflows. A.1 Chapter 2 Training large-scale MT models is computationally expensive, partic- ularly when relying on generic-domain corpora containing millions of sentences. The data selection method proposed in Chapter 2 substan- tially reduces computational costs by retaining only the most relevant in-domain sentences from a large corpus, thereby improving training efficiency and reducing resource consumption. Compared to training on the full 31-million-sentence generic corpus, this approach reduces train- ing time by over 90%âfrom 2 days and 5 hours on three NVIDIA Tesla V100 GPUs to just 8 hours for the best-performing modelâreduces the data footprint by 97% (i.e., the volume of data processed during training), and lowers GPU energy consumption and associated carbon emissions by over 85%, estimated proportionally from the reduction in training time on identical hardware. AspectBaselineSelected Data Training time 2d 5h (3ĂV100)8h Data required 31M sentences1M sentences Energy / CO 2 100%âŒ15% Table A.1: Resource usage and efficiency gains for the data selection approach introduced in Chapter 2. These results show that our method enables high-quality, domain- specific MT while substantially reducing computational overhead, mak- ing it a more sustainable and cost-effective alternative to large-scale, general-domain training. That said, a fair comparison should also consider the computational cost of the data selection process itself. In Chapter 2, we did not record the runtime of this step, and its cost can vary depending on factors such as the size of the out-of-domain dataset, the amount of in-domain data available, and how many in-domain samples are selected per query. A.2 Chapter 3 The proposed DA pipeline introduced for QE in Chapter 3 adds an extra training step, which increases total training time by 77.8% (from 3.6h to 6.4h). While this raises computational cost, it yields substantial 224 gains in generalization and cross-lingual robustness compared to the baseline fine-tuning approach, as summarized in Table A.2; further details are provided in Chapter 3. MetricOutcome (vs. baseline) Training time3.6h â 6.4h ( +77.8% ) Avg. Pearson correlationâ 14.36% Zero-shot EN-ZHâ 25.51% Cross-lingual inferenceâ 11.60 Pearson pts OOD fine-tuning (per language)â 45% time Step 2 vs. training from scratchâ 50% time Table A.2: Efficiency and performance outcomes of the DA pipeline presented in Chapter 3. Improvements (â) indicate performance gains, while reductions (â) indicate lower computational cost. âAvg. Pearson correlationâ reflects overall QE accuracy; âZero-shot EN-ZHâ measures model performance without EN-ZH training examples; âCross-lingual inferenceâ captures transfer performance across language pairs; âOOD fine-tuningâ refers to initializing a new language pair by fine-tuning a pre-trained out-of-domain model; and âStep 2 vs. training from scratchâ compares the cost of the second adaptation stage with full model retraining. A.3 Chapter 4 Chapter 4 evaluated how different BPE and vocabulary configurations affect the efficiency of fine-tuning for in-domain translation. The results show clear differences in computational cost: hybrid configurations that reuse the original tokenizer and vocabulary (C4, C5, C7) are the most efficient, completing training in roughly three hours and producing substantially lower CO 2 emissions. Among these, C5 offers the best balance between runtime and translation quality. In contrast, the configuration achieving the highest in-domain BLEU score (C6) is also the most resource-intensive, requiring nearly three times longer to train and emitting more than double the CO 2 of the 225 efficient hybrid methods. Even the standard baseline (C1) remains costly. These efficiency patterns are summarized in Table A.3. ConfigurationBLEU Time (h) CO 2 (g) C6 (Best in-domain)54.8 â9.5â1587 ââ C1 (Baseline)53.67.75â1658 ââ C5 (Fastest hybrid)53.13.25â729 ââ C7 (Minimal compute)52.9 â3.0 â543ââ C4 (Minimal compute)53.03.1 â723ââ Table A.3: Efficiency comparison of BPE/vocabulary configurations (Chapter 4). Arrows denote relative differences:â = higher value (worse for time/CO 2 , better for BLEU),â = lower value (more efficient).âmarks efficient configurations;âmarks costly ones. Overall, the results show that small gains in BLEU can come with disproportionately large increases in computation and emissions. Hybrid approaches offer a more sustainable alternative, cutting training time and CO 2 by more than half while maintaining competitive translation quality. A.4 Chapter 5 Chapter 5 evaluated the computational efficiency of our ICL approach relative to retrieval-based methods, task-level prompting, and full model fine-tuning. The focus here is on how much time, energy, and CO 2 can be saved while still achieving strong translation performance. Table A.4 provides a compact overview of the main efficiency indicators. Our best-performing ICL configuration (patience 16) achieves a 3.5 BLEU point improvement over R-BM25 while requiring 5.11 hours of computation. Although this introduces a runtime increase relative to R-BM25, it remains far more efficient than fine-tuning mBARTâ50. ICL reduces time-to-prediction by approximately 54 percent compared 226 MethodTime (h) CO 2 (kg) Energy (kWh) BM25 (q=16)0.970.070.19 R-BM25 (q=16)1.070.070.21 ICL (patience 3)1.820.220.67 ICL (patience 8)3.800.501.51 ICL (patience 16)5.110.682.05 Fine-tuned mBARTâ5011.201.884.82 Task-level prompting (16)78.512.8035.91 Table A.4: Runtime, CO 2 emissions, and energy usage for methods evaluated in Chapter 5. ICL introduces additional search cost compared to retrieval-based baselines, but remains substantially more efficient than full fine-tuning and task-level prompting. to fine-tuning and emits only 36 percent of its CO 2 , corresponding to a 64 percent reduction in carbon footprint. Energy consumption follows the same trend: ICL uses about 42.5 percent less GPU power than fine-tuning. Early stopping also provides significant gains. Using patience 3 instead of 16 reduces runtime by around 71 percent while retaining competitive BLEU performance, making it an attractive option when rapid inference is needed. Task-level prompting is the most computationally expensive strategy, requiring 78 hours for 16 examplesâover fifteen times the cost of our most expensive ICL setting. In comparison, ICL offers a practical middle ground: it substantially improves translation accuracy while keeping runtime, energy, and emissions far below those of full fine-tuning or task-level prompting. Across all four chapters, the results show that high-quality MT and QE inevitably involve trade-offs between performance and computa- tional cost, but thoughtful design choices can make these processes far more sustainable. Data selection (Chapter 2) demonstrates that small, domain-focused subsets can replace large generic corpora while sharply 227 reducing training time and emissions. The QE adaptation pipeline (Chapter 3) shows that modest extra training yields strong gains in generalization. Tokenization and vocabulary strategies (Chapter 4) in- dicate that hybrid or partially reused vocabularies retain most benefits of full in-domain adaptation at much lower energy cost. Finally, the ICL approach (Chapter 5) offers a lightweight alternative to fine-tuning, reducing inference time, energy usage, and CO 2 while maintaining strong translation performance. Overall, these findings highlight the value of efficient, sustainable design choices and the importance of being transparent about their computational implications. 228