Paper deep dive
Investigating Model Compression for Neural Machine Translation in the Biomedical Domain
Maria Zafar, Souhail Bakkali, Rejwanul Haque
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/7/2026, 8:26:52 AM
Summary
This paper investigates the combined application of knowledge distillation (KD) and quantization for French-to-English biomedical neural machine translation (NMT). The authors address challenges in low-resource domains by developing fine-tuning strategies for compressed student models. Experiments using MarianMT models on the ELRC-EMEA OPUS corpus demonstrate that a collaboratively distilled and quantized student model achieves a 69% reduction in size, a 98.21% increase in inference speed, and a 98.46% reduction in CO2 emissions compared to the baseline teacher model, without sacrificing translation quality.
Entities (10)
Relation Signals (7)
Distilled and Quantized Student Model → achieves → 98.21% increase in inference speed
confidence 95% · a 98.21% increase in inference speed
Distilled and Quantized Student Model → achieves → 98.46% reduction in CO2 emissions
confidence 95% · and a 98.46% reduction in CO2 emissions compared to the original baseline
Distilled and Quantized Student Model → achieves → 69% reduction in size
confidence 95% · a collaboratively distilled and quantized student model achieves a 69% reduction in size
Quantization → usedfor → Accelerate Inference
confidence 95% · quantization... is widely used to accelerate inference, enabling models to run several times faster during deployment.
Knowledge Distillation → usedfor → Model Compression
confidence 95% · Knowledge distillation has emerged as a sustainable approach for model compression, transferring knowledge from large teacher models to smaller, more efficient student models.
ELRC-EMEA OPUS → usedfor → Biomedical Domain
confidence 90% · For our experiments we used French–English (Fr–En) parallel data from the biomedical domain, ELRC-EMEA OPUS.
MarianMT → usedin → Neural Machine Translation
confidence 90% · We used MarianMT models [11] for our teacher–student training setups.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large-scale pretrained transformer models have achieved state-of-the-art performance across diverse machine translation tasks, including multilingual settings. Knowledge distillation has emerged as a sustainable approach for model compression, transferring knowledge from large teacher models to smaller, more efficient student models. Similarly, quantization, which reduces the numerical precision of model weights and activations (e.g., from 32-bit to 8-bit representations) is widely used to accelerate inference, enabling models to run several times faster during deployment. However, both techniques face limitations when applied to specialized domain data, particularly under low-resource conditions. In knowledge distillation, the effectiveness of transfer is often constrained by the scarcity of domain-specific parallel data, while quantization can lead to performance degradation as bit precision decreases. In this work, we investigate the combined application of knowledge distillation and quantization for French-to-English biomedical translation, a domain characterized by specialized terminology and limited parallel resources. We develop and compare multiple fine-tuning strategies to adapt compressed student models to this challenging setting. Our experiments demonstrate that a collaboratively distilled and quantized student model achieves a 69% reduction in size, a 98.21% increase in inference speed, and a 98.46% reduction in CO2 emissions compared to the original baseline all without sacrificing translation quality. These results indicate that jointly optimized compression techniques can yield efficient, high-performance models suitable for translation service providers operating under resource constraints.
Tags
Links
- Source: https://arxiv.org/abs/2610.07032v1
- Canonical: https://arxiv.org/abs/2610.07032v1
Trouble viewing inline? Open PDF directly →
Full Text
36,978 characters extracted from source content.
Expand or collapse full text
Investigating Model Compression for Neural Machine Translation in the Biomedical Domain Maria Zafar 1[0009−0008−9787−5634] , Souhail Bakkali 2[0000−0001−9383−3842] , and Rejwanul Haque 1[0000−0003−1680−0099] 1 South East Technological University, Carlow, Ireland https://w.setu.ie, https://w.compucore.ie c00304029@setu.ie, rejwanul.haque@setu.ie 2 IRISA, France souhail.bakkali@irisa.fr Abstract. Large-scale pretrained transformer models have achieved state- of-the-art performance across diverse machine translation tasks, includ- ing multilingual settings. Knowledge distillation has emerged as a sus- tainable approach for model compression, transferring knowledge from large teacher models to smaller, more efficient student models. Similarly, quantization—which reduces the numerical precision of model weights and activations (e.g., from 32-bit to 8-bit representations)—is widely used to accelerate inference, enabling models to run several times faster during deployment. However, both techniques face limitations when ap- plied to specialized domain data, particularly under low-resource con- ditions. In knowledge distillation, the effectiveness of transfer is often constrained by the scarcity of domain-specific parallel data, while quan- tization can lead to performance degradation as bit precision decreases. In this work, we investigate the combined application of knowledge dis- tillation and quantization for French-to-English biomedical translation, a domain characterized by specialized terminology and limited parallel re- sources. We develop and compare multiple fine-tuning strategies to adapt compressed student models to this challenging setting. Our experiments demonstrate that a collaboratively distilled and quantized student model achieves a 69% reduction in size, a 98.21% increase in inference speed, and a 98.46% reduction in CO 2 emissions compared to the original base- line—all without sacrificing translation quality. These results indicate that jointly optimized compression techniques can yield efficient, high- performance models suitable for translation service providers operating under resource constraints. Keywords: Knowledge Distillation· Machine Translation· LLM· Quan- tization. 1 Introduction Knowledge Distillation (KD) is a well-established technique for transferring knowledge from larger, more complex teacher models to smaller, more efficient arXiv:2610.07032v1 [cs.CL] 4 Oct 2026 2M. Zafar et al. student models [9, 13]. This approach involves training a student model to mimic the outputs of a teacher model, often using soft labels generated by the teacher on unlabeled data. Model quantization [19] maps weights to a lower-bit data format, reduces memory footprint and accelerates model inference. The main goal of combining jointly KD and quantization is to create compact and deploy- able MT models that balance translation quality with inference speed as this technique have proven effective in many MT translation tasks [18]. However, KD presents several challenges including (i) the severe lack of specialized paral- lel text data (e.g. biomedical) can prevent effective knowledge transfer, i.e., the student may overfit to the small data and fail to generalize the broad knowl- edge the teacher possesses; (i) the shallow student model may not have the representational power for word sense disambiguation -it often falls back to the most frequent translation of a word-, leading to errors in disambiguation and to incorrect translation of polysemous terms. Moreover, quantization can sig- nificantly degrade translation quality for morphologically diverse languages and low-resource domains, especially under very low bit widths. In this context, Translation Service Providers (TSPs) operating under con- straint budget who offer online MT services face significant challenges since they require compact, fast and competitive MT models. Moreover, it is more challeng- ing when the domain training data is not sufficiently available. In this paper we investigate applying KD with different fine-tuning strategies (e.g. QLoRA) [5] to develop compact, faster, deployable and competitive models for low-resource do- main data translation. We employed quantization to the student models trained through KD in order to further reduce model size and inference cost. We carried out our experiments on French-to-English language pairs using ELRC-EMEA OPUS 3 corpus. Our investigation showed that the quantized student model can perform comparably or even significantly outperform the baseline models. Note that the baseline models are those that are trained following the standard KD training setup. The rest of the paper is organised as follows. In Section 2 we describe the related work. Our datasets and details of experimental setups are explained in Section 3. In Section 4, we discuss our results. Section 5 concludes our work and discusses potential directions for future research. 2 Related Work [9] introduced KD for image classification task as a model compression technique to squeeze the large networks into smaller models which obtained the same per- formance as the large networks. Whereas [13] introduced KD techniques for neural MT (NMT). Subsequently, a lot of research has been conducted on vari- ous applications of sequence-level KD for NMT [8, 12, 7, 28, 35]. Most prominent works of [13] include sequence-level KD. This is an effective technique for train- ing small student models based on pseudo target sequences generated by the 3 OPUS: https://opus.nlpl.eu/ELRC-EMEA/fr&en/v1/ELRC-EMEA NMT in Biomedical Domain3 large teacher models. The outcome of this technique effectively decreases both the size of the model and inference time, resulting in minimum loss in perfor- mance [2, 12, 25, 7, 28]. KD generates forward-translated synthetic data (i.e. soft target labels) through the teacher model, and allows training of smaller models using synthetic data with a minimum loss in performance [14, 30, 26, 36]. The approach is straight forward with two phases for the KD pipeline: the first phase generates distilled data using large language teacher models (i.e. LLMs) and the second phase uses the distilled data that is created during the first phase for training and fine-tuning the student models. Researchers have also investigated alternative prompting methods [29, 10] to improve sequence-level distillation by transferring reasoning and decision-making capabilities from teacher to student models. In most cases, sequence-level KD relied on correct or incorrect trajectories to fine-tune the student model [16, 3, 17]. However, these studies mainly focused on large fine-tuning datasets and did not address scenarios with fixed teacher budgets or noisy teacher annotations in unlabeled data. To mitigate noise, [31] rephrased noisy rationales to enrich fine-tuning data, but this approach did not consider selective annotation to re- duce teacher cost. [1] proposed a setup where domain information was available during training but not at inference. They further developed a model by adding a domain classifier on top of the Neural Machine Translation (NMT) encoder, aiming to obtain a single system with performance close to multiple expert mod- els. Although their method achieved comparable BLEU [20] scores, it was less parameter-efficient than the target token model. BLEU evaluates the quality of MT model by comparing the translations of the model with the reference trans- lations. Extending this idea, [4] introduced multi-domain KD, where multiple expert models were distilled into a single student. Their approach kept both architecture and capacity fixed, yielded substantial BLEU improvements over standard multi-domain models without increasing translation time or memory usage. Furthermore, their method was architecture-independent, allowing seam- less integration with other multi-domain NMT frameworks while using fixed teacher depth and architecture. Smaller NMT models offer advantages such as faster training, reduced com- putational demand, lower energy consumption, and suitability for deployment on resource-constrained devices, thereby contributing to lower CO 2 emissions [14]. However, these models often achieve lower translation quality compared to larger models, and the gap becomes more pronounced for low-resource lan- guages. The main challenge in low-resource settings is the limited availability of high-quality parallel data. While large language models (LLMs) can gener- ate soft target labels for knowledge distillation, their outputs for low-resource domains or languages are often noisy or of poor quality. This poses a problem for sequence-level knowledge distillation, as errors in the teacher’s outputs can be propagated, causing the student model to learn incorrect patterns from the distilled data. Another model compression technique, quantization is also in focus by re- searchers. [19] proposed a method for deploying complex AI models on resource- 4M. Zafar et al. constrained devices by reducing their computational cost and energy consump- tion. The core idea involves converting the high-precision numbers (floating- point) typically used in neural networks into lower-precision integers, which sig- nificantly reduces memory footprint and computational requirements. [6] pro- posed LLMC, a toolkit designed for benchmarking and compressing LLMs. The core purpose of LLMC is to systematically explore the impact of quantiza- tion, a technique that reduces computational and memory demands of LLMs by converting them to lower-bit data formats, without significantly compromis- ing accuracy. The toolkit offers extensive support for various algorithms, models, and hardware backends, enabling fair comparisons and detailed analyses of dif- ferent quantization strategies. Key areas investigated include the influence of calibration data on performance, the effectiveness of various quantization al- gorithms (e.g. transformation, clipping, and reconstruction), and the trade-offs between integer and floating-point quantization. Ultimately, LLMC aims to pro- vide researchers and users with practical guidance and insights to democratise LLM compression and facilitate their widespread application, even on more con- strained hardware. [18] investigated on methods for efficiently deploying large audio-language models for speech translation, addressing the significant compu- tational demands of these powerful systems. Key approaches include iterative layer pruning, which systematically removes less important model layers, low- rank adaptation with 4-bit quantization (QLoRA) for more efficient fine-tuning, and KD, where a smaller student model learns from a larger teacher model. Their research demonstrates that combining these compression strategies can achieve up to a 50% reduction in model parameters and storage, while retaining 97-100% of the original translation quality for English-to-German and English- to-Chinese speech translation. Most recently, [33] explored a novel approach to reduce the training costs and environmental impact of NMT for low-resource languages. They introduced a confidence-based KD method. This technique in- volves filtering out low-quality translations generated by a large teacher model NLLB-200-3.3B based on its confidence scores, thereby improving the efficiency of training smaller student models. Their experiments, focusing on Urdu-to- English translation, demonstrate that this strategy can significantly lower CO 2 emissions and training time without compromising translation quality. The re- search offers a practical and sustainable solution for translation service providers facing budget and resource constraints. Our work focused on facilitating online MT translation companies with a reasonable solution for developing compact and fast MT models models without compromising the translation quality. In other words, we aimed at providing TSPs a cost-effective model compression solution with the blend of KD and quantization for MT, with applying different fine-tuning strategies. 3 Experiments This section discusses our configurations of the teacher and student MT models used in our experimental setup. NMT in Biomedical Domain5 3.1 Experimental Setups We used MarianMT models [11] for our teacher–student training setups. We used the Helsinki-NLP/opus-mt-tc-big-fr-en 4 checkpoint as our teacher model, which is referred as big transformer (BT). For building our student models, we used Helsinki-NLP/opus-mt-fr-en 5 checkpoint. From now on this model is referred as small transformer (ST). Our training configuration is as follows: batch size=8; eval batch size=8; learning rate=1e-5; epochs=20; save total limit=2; max sequence length=256; weight decay=applied and the model was saved at every epoch [32]. To demonstrate the generability, we evaluate the performance of MarianMT models on the hold-out test data (cf. last row of Table 1). We quantized the models using CTranslate2 6 for fast inference. The conver- sion was performed using the ct2-transformers-converter utility, which transforms MarianMT models into a runtime optimised format compatible with CTrans- late2. The quantization float16 flag was applied to reduce numerical precision from FP32 to FP16 [23]. Naturally, this conversion significantly decreases mem- ory consumption with notable improvements in inference speed. We fine-tuned our MT models, and tested them under quantized and non-quantized settings. Specifically, we used quantized teacher models for preparing distilled data to be used by the student model. The intuition is to obtain a teacher model that can generate high-quality translations at faster inference speed. To maintain the consistency we trained and tested all models with same configurations and on the computing system enabled with one type of GPU. We used A100 GPU throughout our experiments. Note that we also fine-tuned the ST model on the training data (cf. Section 3.2) before initiating the distillation process. We followed the standard student- teacher training setups in order to build our baseline student models. In other words, we treated the English translations of the French sentences as pseudo target labels for the distilled training data. 3.2 Dataset For our experiments we used French–English (Fr–En) parallel data from the biomedical domain, ELRC-EMEA OPUS. 7 This is a medical domain data, cov- ering a range of topics such as medicine, treatments. The data statistics are shown in Table 1. Note that we removed duplicates from the training data. We sampled a set of 1,000 and 2,000 source–target sentence-pairs from the same data source for the validation and test sets, respectively. 4 https://huggingface.co/Helsinki-NLP/opus-mt-tc-big-fr-en 5 https://huggingface.co/Helsinki-NLP/opus-mt-fr-en 6 CTranslate2: https://opennmt.net/CTranslate2 7 OPUS: https://opus.nlpl.eu/ELRC-EMEA/fr&en/v1/ELRC-EMEA 6M. Zafar et al. Sentences Vocabulary French English Train759,861 85,627 69,036 Valid1,000 4,454 3,963 EMEA-Test2000 4,557 4,052 Table 1. Data Statistics. 4 Results and Discussions 4.1 Evaluation Metrics We used SacreBLEU [22] for measuring performance of our MT models. For re- porting the results we used evaluation metrics such as: BLEU [20], chrF++ [21], COMET [24], BERTScore (Precision, Recall, F1-score) [34] and BLEURT [27]. We measured inference speed through the time elapsed in seconds on the com- puting system enabled with A100 GPU. We measured the energy consumption and carbon emissions [15] during inference, for which we used carbon emission tracking tool, codecarbon 8 Table 2. BT: Big Transformer. Performance of the BT (teacher) models on the EMEA test set. Inference Speed (IS): Total time required to translate test set data. Model Size (MS) on disk in MBs. SB: SacreBLEU. Q: Quantization. FT: Finetuning. P:Precision. R: Recall. F1:F1-score. BRT: BLEURT. Precision, Recall, and F1-Score is measured using BERTScore. Inference Speed is measured in seconds, CO 2 emission is measured in KGs during inference. QFTSBchrF++COMBERTScoreBRTMSISCO 2 PRF1 nono51.2772.3687.9067.6666.2066.9245.984612983.880.0392 yesno51.1772.9488.7277.4676.7777.0846.5844241.050.0006 noyes48.5270.2485.9458.9557.3958.1740.634.53777.210.0502 (QLoRA) noyes68.3381.4089.2379.8277.2778.4953.288804147.610.0547 yesyes70.3683.2690.4883.8381.8082.7855.2944258.430.0008 4.2 Teacher Models As outlined in Section 3.1, we used the MarianMT BT system [11] as the teacher for the translation task. Its performance on the test set is reported in the first row of Table 2. We also evaluated the quantized version of this model, with the cor- responding results shown in the second row of Table 2. We can see from the table 8 Code Carbon: https://codecarbon.io NMT in Biomedical Domain7 that the quantized model outperformed the MarianMT BT system in terms of chrF++, COMET, precision, recall, F1-score while achieving comparable BLEU and BLEURT scores. The third row of Table 2 shows the performance of the MarianMT BT model after fine-tuning on the domain data (cf. training data; Table 1) using QLoRA. Since the test set scores did not improve over the baseline models (first and second rows), we decided not to employ QLoRA-based fine-tuning for the teacher models in the subsequent experiments. The fourth row of Table 2 presents the performance of the MarianMT BT model after standard fine-tuning on the domain data. As can be seen from Table 2, the evaluation scores on the test set improved substantially compared to those in the first and second rows, with gains of 17.16 BLEU, 8.46 chrF++, 1.51 COMET, 12.16 precision, 11.07 recall, 11.57 F1-score and 7.3 BLEURT points over the baseline (i.e. first row of Table 1). Like above, we also evaluated the quantized version of the fine-tuned BT model on the test set. The fifth row of Table 2 represents the fine-tuned quantized BT model. When comparing the evaluation scores presented in fourth and fifth rows, we see additional gains of 2.03 BLEU, 1.86 chrF++, 1.25 COMET, 16.17 precision, 15.6 recall, 15.86 F1-score and 9.31 BLEURT points over those presented in the fourth row. We chose the best-performing BT model (i.e. corresponding to the fifth row of Table 2) as our teacher for the KD task. Quantization reduces the computa- tional and memory footprint of the teacher during inference (see the last three columns of Table 2). This step not only facilitates faster model execution and lowers resource consumption, but also aligns with the broader aim of developing sustainable and efficient MT systems for an industrial translation pipeline. 4.3 Student Models Table 3. Performance the Small Transformer models on EMEA test set. Small Trans- former (ST). Inference Speed (IS): Total time required to translate test set data. Model Size (MS) on disk in MBs. SB: SacreBLEU. Q: Quantization. FT: Finetuning. P:Precision. R: Recall. F1: F1-score. BRT: BLEURT. Precision, Recall, and F1-Score is measured using BERTScore. Inference Speed is measured in seconds, CO 2 emission is measured in KGs during inference. QFTSBchrF++COMBERTScoreBRTMSISCO 2 PRF1 nono52.9173.5988.5076.8675.5576.1748.24298.001905.260.0250 yesno53.3673.9288.7678.2277.1277.6449.73284.9037.270.0004 noyes65.0979.9681.0481.7379.0480.3455.73284.901281.090.0167 yesyes66.7981.4489.6684.0082.1783.0557.99142.9052.560.0006 As mentioned in Section 3.1, we used the MarianMT ST model [11] as the baseline system for KD in building the student model. We first evaluated the 8M. Zafar et al. baseline model (MarianMT ST) on the test set. The first row of Table 3 shows the performance of the baseline system (ST). We also evaluated the quantized version of the ST model shown in the second row. This model is comparable to the baseline as far as its performance on translation quality is concerned. The third row of Table 3 represents the ST model that was fine-tuned on the training data. We see from the table that fine-tuning improved the ST model significantly, with an improvement of 13.88 BLEU, 7.85 chrF++, 1.16 COMET, 7.14 precision, 6.62 recall, 6.88 F1-score and 9.75 BLEURT points over the baseline ST model. The last row shows the scores that we obtained from ST model with fine-tuning on the training data. Note that this row refers to the quantized version of the ST model corresponding to the third row of Table 3. Quantization reduces the model to a lightweight, inference-only form and removes the ability to update its parameters. As a result, this model cannot be fine-tuned and therefore cannot be used as a student in KD. We report its performance for comparison only. Table 4. Performance of the Finetuned Student models on EMEA test set. D: Dis- tilled data, DDOP: Distilled Data with Original Parallel Data. ST: Small Transformer. Inference Speed (IS): Total time required to translate test set data. Model Size (MS) on disk in MBs. SB: SacreBLEU. Q: Quantization. FT: Finetuning. P:Precision. R: Recall. F1:F1-score. BRT: BLEURT. Precision, Recall, and F1-Score is measured us- ing BERTScore. Inference Speed is measured in seconds, CO 2 emission is measured in KGs during inference. QFTSBchrF++COMBERTScoreBRTMSISCO 2 PRF1 noDD68.1381.4289.1182.5480.0981.2856.83284.901281.090.0167 yesDD69.2882.6190.6184.5082.6583.5458.67142.9053.250.0006 noDDOP66.6980.4989.5880.0277.1378.5253.94284.901413.760.0182 The performance of the student models are reported in Table 4. The first row shows the results of our first student model. This model refers to the best- performing ST model (second last row of Table 3) was fine-tuned solely on the distilled data. The second row of Table 4 refers to the quantized version of this model. We can see from the table that quantization provided further improve- ments, adding up to 1.15 BLEU, 1.19 chrF++, 1.5 COMET, 1.96 precision, 2.56 recall, 2.26 F1-score and 1.84 BLEURT points over the non-quantized student model. The last row of the table shows the scores that we obtained by fine- tuning the best-performing ST model on the augmented data (original training data combined with the distilled data). Note that this is not a quantized model. We can see from the scores that this strategy does not work. NMT in Biomedical Domain9 Table 5. Best-performing teacher and student models. M:Models, OPD: Original Par- allel Data, D: Distilled data. BT: Big Transformer, ST: Small Transformer. IS: In- ference Speed (IS): Total time required to translate test set data. Model Size (MS) on disk in MBs. SB: SacreBLEU. Q: Quantization. FT: Finetuning. P:Precision. R: Re- call. F1:F1-score. BRT: BLEURT. Precision, Recall, and F1-Score is measured using BERTScore. Inference Speed is measured in seconds, CO 2 emission is measured in KGs during inference. MQFTSBchrF++COMBERTScoreBRTMSISCO 2 PRF1 BTyesOPD70.3683.2690.4883.8381.8082.7855.2944258.430.0008 STyesDD69.2882.6190.6184.5082.6583.5458.67142.9053.250.0006 The performance of the best-performing teacher and student models are reported in Table 5. When we compare the translation performance of best- performing teacher and student models (see Table 5), we can clearly see that the performance of the best-performing student model (e.g. last row of Table 5) is close to that of the teacher (e.g. first row of Table 5). 4.4 Efficiency and Sustainability Analysis We compared the teacher, and variants of the student models in terms of their inference speed, size (MB) and carbon emission during inference (test data). The BT and ST models differ significantly in terms of the costs when considering the size of the model, inference speed and carbon emissions generated during the inference. We refer the reader to the third-last, fourth-last and last columns of Table 2 and 5 for the differences in size of the models, the inference speed and CO 2 emissions of the best-performing student and teacher models. We can see that best-performing ST model (quantized version shown in sec- ond row of Table 5) reduced the size of model to 67.66% and inference speed increased to 8.86% over the best performing BT model presented in the first row of Table 5. We can see that the best-performing ST model increased inference speed to 98.21% and decreased size of model to 69% when compared to the baseline BT model shown in first row of Table 2 but the difference in the quality can measured through the other metrics. The CO 2 emissions generated during inference of the BT and ST models were recorded. We can see that the best-performing ST model (shown in second row of Table 5) reduced the emissions of model to 25% over best performing BT model shown in the first row of Table 5. We can see that the best-performing ST model can reduce CO 2 emissions by up to 98.46% over baseline BT model shown in the first row of Table 2 without any performance loss. The online MT translation companies, which often operate under limited budget can benefit from our findings. KD when applied collaboratively with 10M. Zafar et al. quantization with different fine-tuning strategies can result in competitive, com- pact, and fast MT models. 5 Conclusion In this paper, we investigated the impact of fine-tuning and quantization to- gether in a KD setup. We considered the context in which TSPs offering on- line MT services encounter significant challenges in obtaining compact, good quality models while translating specialised domain data. We conducted our ex- periments with different fine-tuning techniques such as QLoRA. We fine-tuned the teacher model and this ensured that our models were well adapted to the domain. For our distillation process we used quantized model for supervision. This sped up soft labels (translations) generation by the teacher model. Our results showed that the adaptation of the teacher to the domain data through quantization not only improved the translation quality or provided more reliable supervision for KD but facilitated faster model execution and lower resource consumption. This strategy could be beneficial for TSPs who want to develop efficient, domain-adapted teacher model while reducing computational costs and maintaining translation quality. When comparing the teacher and student models, we observed that the quan- tized and fine-tuned student model was able to achieve translation quality close to, and in some cases surpassing, that of the teacher. Despite being significantly smaller, the student maintained competitive performance while delivering much faster inference and lower emissions, confirming the effectiveness of our KD training setup. Our best performing student model obtained 98.21% increase in inference speed and 98.46% reduction in CO 2 emissions over teacher with insignificant drop in translation quality. This statistics show that our strategy aligns with the broader aim of developing sustainable and efficient pipelines for online MT service providers. In future, we plan to extend our investigation to multilingual and low-resource language pairs to evaluate the generalizability of the proposed KD – quantization approach. Additionally, integrating pruning and adaptive quantization techniques could further enhance model efficiency while maintaining translation quality. References 1. Britz, D., Le, Q., Pryzant, R.: Effective domain mixing for neural machine transla- tion. In: Bojar, O., Buck, C., Chatterjee, R., Federmann, C., Graham, Y., Haddow, B., Huck, M., Yepes, A.J., Koehn, P., Kreutzer, J. (eds.) Proceedings of the Second Conference on Machine Translation. p. 118–126. Association for Computational Linguistics, Copenhagen, Denmark (Sep 2017). https://doi.org/10.18653/v1/W17- 4712, https://aclanthology.org/W17-4712/ 2. Bucila, C., Caruana, R., Niculescu-Mizil, A.: Model com- pression. In: Knowledge Discovery and Data Mining (2006), https://api.semanticscholar.org/CorpusID:11253972 NMT in Biomedical Domain11 3. Chen, H., Wu, S., Quan, X., Wang, R., Yan, M., Zhang, J.: Mcc-kd: Multi-cot consistent knowledge distillation (2023), https://arxiv.org/abs/2310.14747 4. Currey, A., Mathur, P., Dinu, G.: Distilling multiple domains for neural ma- chine translation. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP). p. 4500–4511. Association for Computational Lin- guistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.364, https://aclanthology.org/2020.emnlp-main.364/ 5. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L.: Qlora: Efficient finetun- ing of quantized llms (2023), https://arxiv.org/abs/2305.14314 6. Gong, R., Yong, Y., Gu, S., Huang, Y., Lv, C., Zhang, Y., Tao, D., Liu, X.: LLMC: Benchmarking large language model quantization with a versatile com- pression toolkit. In: Dernoncourt, F., Preoţiuc-Pietro, D., Shimorina, A. (eds.) Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. p. 132–152. Association for Computational Linguis- tics, Miami, Florida, US (Nov 2024). https://doi.org/10.18653/v1/2024.emnlp- industry.12, https://aclanthology.org/2024.emnlp-industry.12/ 7. Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. Inter- national Journal of Computer Vision 129(6), 1789–1819 (2021) 8. Gu, J., Bradbury, J., Xiong, C., Li, V.O.K., Socher, R.: Non-autoregressive neural machine translation (2018), https://arxiv.org/abs/1711.02281 9. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015), https://arxiv.org/abs/1503.02531 10. Ho, N., Schmid, L., Yun, S.Y.: Large language models are reasoning teachers (2023), https://arxiv.org/abs/2212.10071 11. Junczys-Dowmunt, M., Grundkiewicz, R., Dwojak, T., Hoang, H., Heafield, K., Neckermann, T., Seide, F., Germann, U., Aji, A.F., Bogoychev, N., Martins, A.F.T., Birch, A.: Marian: Fast neural machine translation in C++. In: Liu, F., Solorio, T. (eds.) Proceedings of ACL 2018, System Demonstrations. p. 116– 121. Association for Computational Linguistics, Melbourne, Australia (Jul 2018). https://doi.org/10.18653/v1/P18-4020, https://aclanthology.org/P18-4020/ 12. Kasai, J., Pappas, N., Peng, H., Cross, J., Smith, N.A.: Deep encoder, shal- low decoder: Reevaluating non-autoregressive machine translation. arXiv preprint arXiv:2006.10369 (2020) 13. Kim, Y., Rush, A.M.: Sequence-level knowledge distillation. In: Su, J., Duh, K., Carreras, X. (eds.) Proceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. p. 1317–1327. Association for Computa- tional Linguistics, Austin, Texas (Nov 2016). https://doi.org/10.18653/v1/D16- 1139, https://aclanthology.org/D16-1139/ 14. Kim, Y.J., Junczys-Dowmunt, M., Hassan, H., Fikri Aji, A., Heafield, K., Grund- kiewicz, R., Bogoychev, N.: From research to production and back: Ludicrously fast neural machine translation. In: Birch, A., Finch, A., Hayashi, H., Konstas, I., Lu- ong, T., Neubig, G., Oda, Y., Sudoh, K. (eds.) Proceedings of the 3rd Workshop on Neural Generation and Translation. p. 280–288. Association for Computa- tional Linguistics, Hong Kong (Nov 2019). https://doi.org/10.18653/v1/D19-5632, https://aclanthology.org/D19-5632/ 15. Lacoste, A., Luccioni, A., Schmidt, V., Dandres, T.: Quantifying the carbon emis- sions of machine learning. arXiv preprint arXiv:1910.09700 (2019) 16. Li, Y., Yuan, P., Feng, S., Pan, B., Sun, B., Wang, X., Wang, H., Li, K.: Turning dust into gold: Distilling complex reasoning capabilities from llms by leveraging negative data (2023), https://arxiv.org/abs/2312.12832 12M. Zafar et al. 17. Liu, W., Li, G., Zhang, K., Du, B., Chen, Q., Hu, X., Xu, H., Chen, J., Wu, J.: Mind’s mirror: Distilling self-evaluation capability and comprehensive thinking from large language models (2024), https://arxiv.org/abs/2311.09214 18. Moslem, Y.: Efficient speech translation through model compression and knowl- edge distillation. In: Salesky, E., Federico, M., Anastasopoulos, A. (eds.) Pro- ceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025). p. 379–388. Association for Computational Linguistics, Vienna, Austria (in-person and online) (Jul 2025). https://doi.org/10.18653/v1/2025.iwslt- 1.40, https://aclanthology.org/2025.iwslt-1.40/ 19. Nagel, M., Fournarakis, M., Amjad, R.A., Bondarenko, Y., van Baalen, M., Blankevoort, T.: A white paper on neural network quantization (2021), https://arxiv.org/abs/2106.08295 20. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Isabelle, P., Charniak, E., Lin, D. (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. p. 311–318. Association for Computational Linguistics, Philadel- phia, Pennsylvania, USA (Jul 2002). https://doi.org/10.3115/1073083.1073135, https://aclanthology.org/P02-1040/ 21. Popović, M.: chrF: character n-gram F-score for automatic MT evaluation. In: Bojar, O., Chatterjee, R., Federmann, C., Haddow, B., Hokamp, C., Huck, M., Logacheva, V., Pecina, P. (eds.) Proceedings of the Tenth Workshop on Sta- tistical Machine Translation. p. 392–395. Association for Computational Lin- guistics, Lisbon, Portugal (Sep 2015). https://doi.org/10.18653/v1/W15-3049, https://aclanthology.org/W15-3049/ 22. Post, M.: A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771 (2018) 23. Rajbhandari, S., Rasley, J., Ruwase, O., He, Y.: Zero: Memory optimizations to- ward training trillion parameter models (2020), https://arxiv.org/abs/1910.02054 24. Rei, R., Stewart, C., Farinha, A.C., Lavie, A.: COMET: A neural frame- work for MT evaluation. In: Webber, B., Cohn, T., He, Y., Liu, Y. (eds.) Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP). p. 2685–2702. Association for Computational Lin- guistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.emnlp-main.213, https://aclanthology.org/2020.emnlp-main.213/ 25. Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter (2020), https://arxiv.org/abs/1910.01108 26. Schick, T., Schütze, H.: Exploiting cloze questions for few shot text classification and natural language inference (2021), https://arxiv.org/abs/2001.07676 27. Sellam, T., Das, D., Parikh, A.P.: Bleurt: Learning robust metrics for text gener- ation (2020), https://arxiv.org/abs/2004.04696 28. Wang, F., Yan, J., Meng, F., Zhou, J.: Selective knowledge distillation for neu- ral machine translation. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Pro- ceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Pro- cessing (Volume 1: Long Papers). p. 6456–6466. Association for Computational Linguistics, Online (Aug 2021). https://doi.org/10.18653/v1/2021.acl-long.504, https://aclanthology.org/2021.acl-long.504/ 29. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y.: React: Synergizing reasoning and acting in language models (2023), https://arxiv.org/abs/2210.03629 NMT in Biomedical Domain13 30. Yoo, K.M., Park, D., Kang, J., Lee, S.W., Park, W.: GPT3Mix: Lever- aging large-scale language models for text augmentation. In: Moens, M.F., Huang, X., Specia, L., Yih, S.W.t. (eds.) Findings of the As- sociation for Computational Linguistics: EMNLP 2021. p. 2225–2239. Association for Computational Linguistics, Punta Cana, Dominican Re- public (Nov 2021). https://doi.org/10.18653/v1/2021.findings-emnlp.192, https://aclanthology.org/2021.findings-emnlp.192/ 31. Yu, D., Backurs, A., Gopi, S., Inan, H., Kulkarni, J., Lin, Z., Xie, C., Zhang, H., Zhang, W.: Training private and efficient language models with synthetic data from LLMs. In: Socially Responsible Language Modelling Research (2023), https://openreview.net/forum?id=FKwtKzglFb 32. Zafar, M., Castaldo, A., Nayak, P., Haque, R., Way, A.: The SETU-ADAPT submissions to WMT 2024 chat translation tasks. In: Haddow, B., Kocmi, T., Koehn, P., Monz, C. (eds.) Proceedings of the Ninth Conference on Ma- chine Translation. p. 1023–1030. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024). https://doi.org/10.18653/v1/2024.wmt-1.104, https://aclanthology.org/2024.wmt-1.104/ 33. Zafar, M., Wall, P.J., Bakkali, S., Haque, R.: Confidence-based knowledge distilla- tion to reduce training costs and carbon footprint for low-resource neural machine translation. Applied Sciences 15(14), 8091 (2025) 34. Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert (2020), https://arxiv.org/abs/1904.09675 35. Zhou, C., Neubig, G., Gu, J.: Understanding knowledge distillation in non- autoregressive machine translation (2021), https://arxiv.org/abs/1911.02727 36. Zhou, Y., Maharjan, S., Liu, B.: Scalable prompt generation for semi-supervised learning with language models (2023), https://arxiv.org/abs/2302.09236