Paper deep dive
Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios
Namrata Patil Gurav, Akashdeep Ranu, Archchana Sindhujan, Diptesh Kanojia
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/13/2026, 12:33:05 AM
Summary
The paper investigates sentence-level Quality Estimation (QE) for English-to-Indic machine translation across four domains (Healthcare, Legal, Tourism, General). It compares prompt-only approaches (zero-shot, few-shot, guideline-anchored) with the ALOPE framework, which uses parameter-efficient fine-tuning (LoRA/LoRMA) on intermediate Transformer layers. Results indicate that intermediate-layer adaptation consistently outperforms final-layer representations, providing a robust solution for low-resource, domain-specific QE.
Entities (5)
Relation Signals (4)
ALOPE â extendedby â LoRMA
confidence 100% · We also extend ALOPE with recently proposed Low-Rank Multiplicative Adaptation (LoRMA).
ALOPE â performs â Quality Estimation
confidence 100% · ALOPE, a framework for LLM-based QE
ALOPE â utilizes â LoRA
confidence 100% · ALOPE... uses Low-Rank Adaptation with regression heads attached to selected intermediate Transformer layers.
Indic-Domain-QE â supports â Quality Estimation
confidence 95% · The Indic-Domain-QE dataset was created to support systematic evaluation of QE in domain-sensitive contexts.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Quality Estimation (QE) is essential for assessing machine translation quality in reference-less settings, particularly for domain-specific and low-resource language scenarios. In this paper, we investigate sentence-level QE for English to Indic machine translation across four domains (Healthcare, Legal, Tourism, and General) and five language pairs. We systematically compare zero-shot, few-shot, and guideline-anchored prompting across selected closed-weight and open-weight LLMs. Findings indicate that while closed-weight models achieve strong performance via prompting alone, prompt-only approaches remain fragile for open-weight models, especially in high-risk domains. To address this, we adopt ALOPE, a framework for LLM-based QE that uses Low-Rank Adaptation with regression heads attached to selected intermediate Transformer layers. We also extend ALOPE with recently proposed Low-Rank Multiplicative Adaptation (LoRMA). Our results show that intermediate-layer adaptation consistently improves QE performance, with gains in semantically complex domains, indicating a path toward more robust QE in practical scenarios. We release code and domain-specific QE datasets publicly to support further research.
Tags
Links
- Source: https://arxiv.org/abs/2603.07372v1
- Canonical: https://arxiv.org/abs/2603.07372v1
Trouble viewing inline? Open PDF directly â
Full Text
52,876 characters extracted from source content.
Expand or collapse full text
Domain-Specific Quality Estimation for Machine Translation in Low-Resource Scenarios Namrata Patil Gurav, Akashdeep Ranu, Archchana Sindhujan, Diptesh Kanojia Institute for People-Centred AI, University of Surrey, United Kingdom np00996, ar02258, a.sindhujan, d.kanojia@surrey.ac.uk Abstract Quality Estimation (QE) is essential for assess- ing machine translation quality in reference- less settings, particularly for domain-specific and low-resource language scenarios. In this paper, we investigate sentence-level QE for EnglishâIndic machine translation across four domains (Healthcare,Legal,Tourism, and General) and five language pairs. We sys- tematically compare zero-shot, few-shot, and guideline-anchored prompting across selected closed-weight and open-weight LLMs. Find- ings indicate that while closed-weight mod- els achieve strong performance via prompting alone, prompt-only approaches remain fragile for open-weight models, especially in high-risk domains. To address this, we adoptALOPE, a framework for LLM-based QE which uses Low-Rank Adaptation with regression heads attached to selected intermediate Transformer layers. We also extendALOPEwith the recently proposed Low-Rank Multiplicative Adaptation (LoRMA) for this work. Our results show that intermediate-layer adaptation consistently im- proves QE performance, with gains in semanti- cally complex domains, indicating a way ahead for robust QE in practical scenarios. We release code and domain-specific QE datasets publicly for further research 1 . 1 Introduction Quality Estimation (QE) enables the output of Machine Translation (MT) systems to be evalu- ated at scale without requiring reference trans- lations (Zerva et al., 2022). Unlike traditional MT evaluation metrics such as BLEU or ME- TEOR, QE directly predicts a quality score for a sourceâtranslation pair, making it especially suit- able for real-world deployment scenarios where reference translations are unavailable. QE can be performed at multiple granularities, and our work 1 https://github.com/surrey- nlp/ALOPE/tree/main/Domain-based-QE-with-ALOPE focuses on segment-level QE, predicting Direct Assessment (DA) scores on a continuous scale of (0 †x †100)for the given translation (Graham et al., 2013). The ground-truth DA score is ob- tained by averaging the scores from three or more human annotators. While fine-grained annotation frameworks such as Multi-dimensional Quality Metrics (MQM) (Lommel et al., 2013) provide detailed insights into translation errors, they impose a substantial cognitive and temporal burden on annotators. In practice, reliable QE enables translation systems to prioritise human intervention for high-risk out- puts, which is particularly important in sensitive or specialised domains (Zerva et al., 2024). Despite the widespread adoption of neural machine trans- lation, translation quality remains uneven across languages and domains, particularly outside high- resource,General-domain settings (Specia et al., 2018; Zhao et al., 2024). This disparity is es- pecially pronounced for EnglishâIndic language pairs (Sindhujan et al., 2025a), where rich morphol- ogy, frequent code-mixing, script variation, and limited availability of high-quality parallel data continue to pose persistent challenges for both MT and evaluation (Zhao et al., 2024). Although MT output forGeneralcontent is of- ten fluent, translations in domain-specific settings such asHealthcare,Legal, andTourismremain fragile, as models are less exposed to specialised terminology and domain-specific constructions dur- ing training (Specia et al., 2018). Even minor errors involving negation, numerical values, units, or spe- cialised terminology can result in substantial mean- ing changes, with potentially serious real-world consequences, particularly in high-risk domains such asHealthcareandLegal(Sindhujan et al., 2025b). These limitations underscore the need for robust domain-aware QE mechanisms that can re- liably identify problematic translations in the ab- sence of reference translations before deployment arXiv:2603.07372v1 [cs.CL] 7 Mar 2026 (Zerva et al., 2024). Recent advances in Large Language Models (LLMs) have enabled QE through prompt-based scoring of sourceâtranslation pairs, offering an al- ternative in low-resource and domain-specific set- tings where supervised QE data are limited (Brown et al., 2020). However, prior work shows that prompt-only LLM-based QE remains inferior to state-of-the-art encoder-based QE models, partic- ularly for sentence-level regression tasks (Zerva et al., 2022; Sindhujan et al., 2025b). A key limita- tion is that LLMs are optimised for next-token pre- diction rather than regression-oriented objectives such as predicting DA scores. Prompting or instruc- tion tuning alone does not introduce regression- specific training signals, often resulting in unstable predictions (Zhao et al., 2021; Sindhujan et al., 2025c). Furthermore, most LLM-based QE ap- proaches rely solely on representations from the final Transformer layer, despite growing evidence that intermediate layers may better encode cross- lingual and semantic alignment for low-resource languages (Kargaran et al., 2025). Our work addresses these limitations by investi- gating domain-aware QE for EnglishâIndic trans- lation language pairs through a dual-track evalu- ation: (i) systematic comparison of prompt-only approaches across closed-weight and open-weight LLM families, and (i) lightweight, parameter- efficient fine-tuning based approach where open- weight prompt-only methods prove insufficient. Building on theALOPEframework (Sindhujan et al., 2025c), which demonstrated that interme- diate Transformer layers encode more stable QE- relevant signals than final-layer representations, we extend this approach to domain-specific, low- resource EnglishâIndic settings (§3.3).ALOPE attach regression heads to informative intermediate layers and update only a minimal parameter subset using Low-Rank Adaptors (Hu et al., 2022), main- taining computational efficiency while improving the QE performance where deployment constraints preclude closed-weight API access. We conduct a comprehensive evaluation across four domains (Healthcare,Legal,Tourism, and General) and five Indic languages (Hindi, Marathi, Tamil, Telugu, and Gujarati). Our experiments sys- tematically compare prompt-only baselines (zero- shot, few-shot, and guideline-anchored prompt- ing) andALOPE, analysing performance differences between closed-weight models and open-weight models. Critically, our study establishes when lightweightALOPE-based methods provide value versus when strong prompting alone suffices, of- fering practical deployment guidance for resource- constrained QE scenarios. The main contributions of this work are as follows: âąWe provide a rigorous comparison of prompt- only QE strategies across closed-weight and open-weight LLM families, revealing that closed-weight models with guideline- anchored prompting achieve robust perfor- mance within domains and language pairs. âąWe demonstrate that a lightweightALOPE- inspired approach, which leverages selected intermediate Transformer layer representa- tions, achieves competitive QE performance in resource-constrained settings. Across the majority of the domains and language pairs, intermediate Transformer layers consistently yield stronger QE signals than final-layer rep- resentations. âąWe establish a simple practical framework for QE deployment in low-resource, domain- sensitive settings, providing clear guidance on when to prioritise strong prompting ver- sus when to apply lightweight adapter-based methods. 2 Background Machine Translation (MT) quality remains un- even across language pairs and domains despite advances in neural and Transformer-based architec- tures (Specia et al., 2018; Zhao et al., 2024). This disparity is pronounced for EnglishâIndic transla- tion, where rich morphology, code-mixing, script diversity, and limited parallel data pose persistent challenges (Sindhujan et al., 2025b). Domain- specific translation in high-risk contexts (Health- care, Legal) is particularly fragile, as publicly avail- able corpora are skewed toward general web and news content, leaving specialised terminology and discourse structures under-represented (Specia et al., 2018; Zhao et al., 2024). Recent domain- focused evaluations further show that MT quality is strongly domain dependent: systems that per- form well on General text often degrade substan- tially in specialised domains such as Healthcare, Legal, Literary, or User-generated content (MĂ€kelĂ€ et al., 2024; Gupta et al., 2024). In the Healthcare domain, LLMs leveraging document-level context have been shown to outperform traditional neural MT systems, while this advantage largely disap- pears in General or News translation, highlighting the interaction between domain characteristics and model effectiveness (MĂ€kelĂ€ et al., 2024). Quality Estimation (QE) addresses this by pre- dicting translation quality without reference trans- lations, enabling scalable assessment (Zerva et al., 2022). Segment-level QE using Direct Assess- ment (DA) scores provides a practical alternative to fine-grained frameworks like MQM, balancing interpretability with annotation efficiency (Gra- ham et al., 2013). However, QE performance in low-resource, domain-shifted settings remains con- strained by limited labelled data and high-impact er- ror types (Zhao et al., 2024), which are particularly prevalent in specialised domains where terminol- ogy misuse or semantic distortion can be critical. LLMs offer prompt-based, reference-free QE but suffer from score compression, inconsistent calibra- tion, and weak sensitivity to subtle errors due to op- timisation for next-token prediction rather than re- gression objectives (Zerva et al., 2024; Ko Ë cmi and Federmann, 2023). Evidence from domain adapta- tion studies suggests that while fine-tuning LLM- based MT models on in-domain data consistently improves translation quality, the effectiveness of adaptation depends on the availability and diversity of domain-specific data (Patel et al., 2024). In the Legaldomain, fine-tuning a strong multilingual pre-trained model has been shown to outperform models explicitly pre-trained for the target domain, indicating that robustGeneral-purpose representa- tions combined with targeted adaptation are often more effective than domain-specific pre-training alone (Singh et al., 2025). When in-domain data is scarce or rapid adaptation is required, in-context learning provides a viable alternative: selecting few-shot examples based on topic similarity can significantly improve translation quality for unseen domains, though gains depend on balancing rele- vance with sufficient example diversity (Li et al., 2024). Cross-lingual QE signals are not uniformly dis- tributed across Transformer layers, intermediate layers encode more stable semantic and alignment information than final layers, especially for low- resource languages (Kargaran et al., 2025; Ten- ney et al., 2019). Parameter-efficient methods such as Low-Rank Adaptation (LoRA) enable task- specific fine-tuning by updating minimal parame- ters while keeping base models frozen (Hu et al., 2022; Dettmers et al., 2023). TheALOPEframework DomainLangsTrainTest HealthcareHi/Mr/Ta/Gu13,2801,660 LegalGu/Ta/Te6,160770 TourismHi/Mr/Te13,8401,730 GeneralHi/Mr/Ta/Te/Gu18,8802,360 Table 1: Indic-Domain-QE dataset composition, show- ing domain coverage, language pairs, and the number of instances in the train and test splits aggregated across languages. (§3.3) extends this by attaching lightweight regres- sion heads to intermediate layers, targeting QE- relevant representations (Sindhujan et al., 2025c). Our work investigates whether intermediate-layer adaptation provides consistent benefits across do- mains and low-resource EnglishâIndic pairs, or whether domain characteristics and pre-training coverage moderate its effectiveness. 3 Methodology The methodology adopted in this research pro- vides a systematic and reproducible approach to investigating domain-specific QE for EnglishâIndic language pairs using LLMs. Unlike generic MT evaluation, this research emphasises four practical domains where translation errors can have tangi- ble effects on daily life. Starting from a struc- tured dataset with human-annotated DA scores across four domains and five language pairs, we evaluate QE approaches along two parallel tracks: Prompt-Only approaches and the ALOPE-based ap- proach. Within Prompt-only approaches, we evalu- ate closed-weight models for comparison, whereas open-weight models are used as backbone within both approaches. Figure 1 provides an overview of our methodological framework using open-weight models. 3.1 Dataset Construction The Indic-Domain-QE dataset was created to support systematic evaluation of QE in domain- sensitive contexts. Texts were sourced from pub- licly available bilingual resources and curated domain-specific materials. Domain characterisa- tion follows standard MT and QE research prac- tices (Specia et al., 2018; Zhao et al., 2024), based on provenance and communicative function of source texts. Healthcaredata consist of patient-facing med- ical content such as information leaflets, pub- lic health advisories, and community health ar- Figure 1: Methodological Framework uses open-weight models for (i) prompt-only approaches (zero-shot, few-shot, guideline-anchored), and (i) ALOPE adaptation with LoRA/LoRMA. ticles, exhibiting high terminology density and sensitivity to negation and numerical expressions. Legaltexts are drawn from contracts, policy ex- cerpts, and official notices, characterised by for- mal register, modality, and scope-defining construc- tions.Tourismdata originate from brochures, at- traction descriptions, and travel guidelines, rich in named entities and culturally grounded ref- erences.General-domain data comprise broad- coverage sentences from encyclopaedic and public- information sources, providing a baseline with min- imal domain-specific constraints. Each entry contains an English source, its trans- lation, a domain label, and human-annotated DA scores. Annotators were trained following estab- lished DA protocols (Graham et al., 2013). Table 1 provides a detailed breakdown of the instances used in this study, with the training split employed for fine-tuningALOPEand the test split used for eval- uation under both prompt-only andALOPE-based settings. 3.2 Prompt-only Approaches We evaluate three prompting strategies for QE, while keeping the task instruction fixed across do- mains and language pairs, varying only the pres- ence of in-context examples and explicit guid- ance. Selected closed-weight and open-weight models are evaluated under zero-shot prompting, few-shot prompting without guidelines, and few- shot prompting with guideline anchoring. Zero-shot. The model receives only a natural- language task instruction and the input sentence pair, without in-context demonstrations (Prompt: App. A-Figure 4). This setup relies entirely on knowledge from pre-training of the selected model(Brown et al., 2020). While simple and cost- effective, zero-shot prompting often leads to score compression and unstable calibration in regression tasks. Few-shot (without guidelines). This includes1 to5labelled input-output examples in the prompt, with minimal additional instruction (Prompt: App. A - Figure 2). This in-context learning setup conditions the model on representative examples without parameter updates (Brown et al., 2020). Prior work shows few-shot exemplars improve task adherence but may suffer from variability when explicit scoring criteria are absent. Few-shot (with guidelines). This augments the exemplar-based setup with an explicit scoring rubric defining the intended semantics of the out- put scale (Prompt: App. A-Figure 3). Guidelines clarify numerical score assignment, reducing ambi- guity. Prior studies demonstrate that such explicit constraints improve output consistency and reduce prompt sensitivity (Mishra et al., 2022; Zhao et al., 2021). Prompt-only evaluations are conducted with two types of models: âąClosed-weight models: Gemini-1.5-Pro and Gemini-2.5-Pro, accessed via API, serving as strong prompt-only baselines without parame- ter updates. âąOpen-weight models: LLaMA-3.2-3B In- struct, LLaMA-3.1-8B Instruct, Qwen3-14B Instruct, and Gemma-3-27B Instruct. 3.3 ALOPE Framework ALOPE (Adaptive Layer OPtimization for Transla- tion Quality Estimation) (Sindhujan et al., 2025c) is a parameter-efficient fine-tuning-based frame- work that attaches regression heads to selected in- termediate Transformer layers and updates only a minimal parameter subset using LoRA (Hu et al., 2022). The original framework explores both single-layer heads and multi-layer variants with dynamic weighting. We only adopt the sim- plified single-layer configuration for feasibility and reproducibility, evaluating both LoRA and LoRMA adapter variants within the same frame- work. LoRMA stands for Low-Rank Multiplicative Adaptation, and unlike LoRA, which fine-tunes models by additively injecting low-rank weight up- dates, LoRMA adapts models by multiplicatively modulating existing weights (Bihany et al., 2025). We conduct allALOPEexperiments using LLaMA-3.2-3B Instruct as the backbone model, following the best-performing configuration re- ported in the originalALOPEstudy (Sindhujan et al., 2025c). We explore rank configurations R â 32, 64, 128with scaling factorα â 16, 32, extracting representations from Layers â1,â7,â9,â11. The rankRdetermines the capacity of the low-rank decomposition used to parameterise the weight update, with each update factorised into two trainable matrices while keep- ing the pre-trained weights frozen. The scaling factorαrescales the update byα/R, stabilising training and enabling effective adaptation across different rank settings without extensive hyperpa- rameter tuning (Hu et al., 2022). The regression head is a lightweight two-layer feed-forward network with ReLU activation, map- ping layer representations to scalar DA predictions. Training uses mean squared error (MSE) loss on gold DA scores. All the adapter-based experi- ments use 4-bit quantized base models via QLoRA (Quantized-LoRA) to ensure computational effi- ciency (Dettmers et al., 2023). Evaluation Metrics. We evaluate model perfor- mance using two correlation-based metrics. Spear- manâs rank correlation (Ï) (Sedgwick, 2014) mea- sures the agreement between the relative ordering of predicted and gold Direct Assessment (averaged across annotators) scores, making it well-suited for Quality Estimation where reliable ranking of trans- lations is often more critical than exact score values. Owing to its robustness to scale differences and out- liers,Ïis used as our primary evaluation metric. Pearsonâs correlation (r) (Cohen et al., 2009) quan- tifies linear agreement between predicted and gold DA scores, indicating how closely model outputs match human scores on the same numeric scale. We reportras the additional metric. Both metrics are computed per language pair and domain, and macro-averaged to obtain domain-level results. 4 Results This section presents a comprehensive analysis of QE performance across domains, languages, prompting strategies, and parameter-efficient adap- tation methods. We begin by analysing the adapter- based approaches usingALOPE, which highlights how Transformer layer selection affects QE per- formance. We then contextualise these findings through domain-wise comparisons against prompt- only baselines, enabling a detailed understanding of when and why lightweight adaptation becomes necessary. 4.1 Layer-wise Analysis with ALOPE Table 2 shows that, for theGeneraldomain, the highest Spearman correlations for most language pairs are achieved at intermediate Transformer lay- ers when usingALOPEwith LoRA. The same pat- tern is observed consistently acrossHealthcare, Legal, andTourismdomains. This behaviour is mirrored byALOPEwith LoRMA (Table 3), where peak correlations for individual language pairs are LayerR = 32, α = 16R = 64, α = 32R = 128, α = 32 En-HiEn-MrEn-TaEn-TeEn-GuAvgEn-HiEn-MrEn-TaEn-TeEn-GuAvgEn-HiEn-MrEn-TaEn-TeEn-GuAvg Lâ10.363-0.0260.3850.1800.1900.2180.4760.1550.6050.2670.435â 0.3880.3840.1050.5240.1960.3910.320 Lâ70.1710.1570.3890.1760.2120.2210.2130.0590.4690.2520.3710.2730.0570.0570.3460.1340.3200.183 Lâ90.1230.0900.5310.1630.4580.2730.3110.0380.5310.2790.4850.3290.0510.1070.3000.0610.3430.172 Lâ110.1780.0610.492-0.0290.4840.2370.3210.0340.568-0.0210.4800.2760.3080.0070.6100.2920.4570.335 Lâ10.5200.0250.201NA0.3770.2810.3820.0330.248NA0.4900.2880.3460.1560.165NA0.4520.280 Lâ70.1680.1500.154NA0.3460.2040.3140.0210.363NA0.5320.3070.2830.1820.392NA0.477â 0.333 Lâ90.1100.0960.086NA0.2550.1370.2620.1320.414NA0.4170.3060.1610.0190.321NA0.3580.215 Lâ110.1230.0500.352NA0.4350.2400.2650.1920.320NA0.4690.3110.346-0.0420.415NA0.5110.308 Lâ1NANA-0.091-0.0090.002-0.033NANA0.5180.0710.2590.283NANA0.3800.1360.3550.290 Lâ7NANA0.3790.0520.3670.266NANA0.4430.0590.3660.289NANA0.4780.1240.2800.294 Lâ9NANA0.5580.1430.3050.335NANA0.5210.0790.2780.293NANA0.4320.1070.2830.274 Lâ11NANA0.4890.0430.2430.258NANA0.5800.2670.4450.430NANA0.5810.2670.445â 0.431 Lâ1-0.153-0.009NA0.002NA-0.053-0.0800.029NA0.065NA0.047-0.0610.045NA0.080NA0.063 Lâ7-0.1040.516NA0.019NA0.1440.1250.566NA0.154NA0.2820.1800.600NA0.210NA0.330 Lâ9-0.0820.579NA-0.050NA0.1490.2880.596NA0.189NA0.3570.3300.640NA0.220NA0.397 Lâ110.2980.633NA0.183NA0.3710.2980.633NA0.183NA0.3710.3500.670NA0.205NAâ 0.408 General Healthcare Legal Tourism Table 2: Spearmanâs (Ï) scores obtained forALOPElayer-wise (L) experiments with LoRA across different domains and language pairs. The bolded values represent the highest spearman scores obtained for each language pair in each domain. The (â ) represents the highest average obtained in each domain. âNAâ indicates that data from that specific domain and language pair is unavailable. likewise concentrated at intermediate layers, partic- ularly Layersâ7,â9, andâ11. To summarise the overall trends, we visualise the domain-level average Spearman correlations from Tables 2 and 3 in Figures 6 and 7 in Ap- pendix E. Each figure plots the average correla- tion computed across the five language pairs for a given domainâagainst the Transformer layer from which the regression head extracts represen- tations. Across all domains, the averaged results reveal a clear and consistent pattern: intermedi- ate Transformer layers, particularly Layersâ9and â11, yield substantially higher Spearman correla- tions than the final layer (Layerâ1). This trend is stable across domains and adapter variants, and aligns with prior findings ofALOPE(Sindhujan et al., 2025c). These results support the hypothesis that QE-relevant signals are more robustly encoded in intermediate representations for EnglishâIndic language pairs, while final-layer representations are more specialised for next-token prediction and instruction-following objectives. Further analysis of the averaged correlations in- dicates thatALOPEwith LoRMA introduces a sta- bilising effect across layers (Appendix E, Figure 7). Compared to LoRA, LoRMA produces smoother layer-wise behaviour, mitigating extremely low cor- relations at shallow layers (e.g., Layersâ1and â7) and reducing variance between adjacent lay- ers across most configurations. This stabilisation is most pronounced in theGeneralandLegaldo- mains, where performance becomes less sensitive to the exact choice of layer. Despite these stability gains, LoRMA obtains competitive performance to LoRA in the Tourism (0.408 vs. 0.404) domain. InHealthcareand Legaldomains, LoRA consistently delivers higher correlations. These findings highlight a principled trade-off: LoRA is preferable when maximising ranking accuracy is the primary objective, whereas LoRMA offers increased robustness when deploy- ment constraints limit precise layer selection. Adapter configurations.ALOPEhas different adapter configurations as explained in the section 3.3. Lower-rank adapters withR = 32consistently underfit across several domains, achieving substan- tially lower correlations than higher-rank configu- rations. Increasing adapter capacity toR = 128 occasionally improves the highest correlations but introduces instability and higher variance, particu- larly for shallow layers. In contrast,R = 64with α = 32consistently provides the best balance be- tween expressive capacity and robustness across domains. 4.2 Prompt-only QE Baselines Table 4 shows the best Spearman scores obtained with prompt-only baselines for each language pair and domain with closed- and open-weight models (Detailed model-wise results are reported in Ap- pendix F, G, H). Closed-weight models consistently provide strong performance across domains, even un- der zero-shot prompting. This behaviour, which yields competitive scores with minimal prompt engineering, can be attributed to the large scale of the Gemini models and their extensive pre- training. Open-weight models exhibit substantially weaker and more variable behaviour under prompt- only evaluation, particularly inHealthcareand Legal. Overall, considering both open- and closed- weights models, few-shot prompting improves per- formance, and guideline-based prompts further sta- bilise the Spearman scores. 4.3 ALOPE vs. Prompt-only Approaches SinceALOPEwas evaluated exclusively with LLaMA-3.2-3B, our comparative analysis focuses on prompt-only results obtained with the same backbone model (Appendix I). Across the major- ity of language pairs and domains,ALOPEconsis- tently achieves higher Spearman correlations than prompt-only prompting strategies. These results demonstrate thatALOPEprovides a practical and ef- fective approach for improving QE, even when ap- plied to smaller open-weight LLMs with quantiza- tion and lightweight adapters, which substantially reduce parameter count, model size, and computa- tional cost compared to large closed-weight models. WhileALOPEis applicable to other open-weight ar- chitectures, extending this analysis to additional models is left for future work. 4.4 Domain-specific Observations This section analyses performance differences be- tween the best prompt-only baselines (Table 4) and theALOPE-based approaches (Tables 2 and 3). Across all settings, theGeneraldomain consis- tently achieves the highest correlations, reflecting its broader linguistic coverage and lower termi- nological complexity compared to specialised do- mains. Healthcare presents a mixed picture for ALOPEâs effectiveness. While prompt-only base- lines achieve strong correlations for several lan- guage pairs,ALOPEwith intermediate-layer adap- tation shows limited improvements and occasion- ally underperforms these baselines. Notable ex- ceptions include EnglishâGujarati, whereALOPE achieves competitive performance. This suggests thatHealthcareQE benefits more from strong prompting strategies with closed-weight models than from lightweight adapter-based methods for most language pairs. WhenALOPEdoes improve performance, gains are strongest at the Transformer Layersâ7 toâ11 (Table 2) for this domain. Legal remains the most challenging domain overall.ALOPEshows selective improvements, par- ticularly for EnglishâTamil (Ï= 0.581, outper- forming open-weight prompt baselines), though absolute correlations remain lower than other domains. Performance gains are configuration- sensitive, reflecting the strict semantic require- ments of Legal text. Tourismexhibits an irregular pattern where zero-shot prompting with open-weight models achieves surprisingly strong performance (aver- ageÏ= 0.646), often matching or exceeding av- erage Spearmanâs obtained with stronger closed- weight models andALOPE. For EnglishâMarathi, ALOPEachievesÏ= 0.670, which is competitive but does not surpass the best closed-weight prompt- only baseline (few-shot with guideline:Ï= 0.702). Gains for other language pairs level off quickly, consistent with the entity-heavy and descriptive nature ofTourismcontent, where surface-level flu- ency and entity preservation may be adequately captured by prompting alone. 5 Discussion Closed-weight models achieve strong and stable correlations even under zero-shot prompting, re- flecting their extensive instruction tuning and mul- tilingual pre-training. Guideline-anchored prompts further improve robustness by clarifying scale se- mantics and reducing mid-range score compression (Mishra et al., 2022; Zhao et al., 2021). As a result, closed-weight models represent the most reliable option when API access is available. In contrast, open-weight models exhibit substantially weaker and more variable performance under prompt-only LayerR = 64, α = 32R = 128, α = 32 En-HiEn-MrEn-TaEn-TeEn-GuAvgEn-HiEn-MrEn-TaEn-TeEn-GuAvg Lâ10.2990.0680.4810.1140.2980.2520.2760.0920.3300.0590.2950.210 Lâ70.3240.3140.4380.0870.3810.3090.3920.0810.5070.1550.4210.311 Lâ90.2550.2650.4730.0700.3620.2850.2780.2830.4840.1010.3860.306 Lâ110.3370.2670.4830.1680.391â 0.3290.3430.2590.4090.0860.4040.300 Lâ1-0.316-0.021-0.381NA0.420-0.075-0.076-0.0250.374NA0.5120.196 Lâ70.2620.1240.405NA0.4950.3220.365-0.0170.413NA0.4570.305 Lâ90.0770.0740.400NA0.4530.2510.3430.1310.277NA0.4900.310 Lâ110.3150.0160.412NA0.4820.3060.3510.1050.392NA0.467â 0.329 Lâ1NANA0.5090.0730.1820.255NANA0.5150.1040.1730.264 Lâ7NANA0.4880.0920.1930.258NANA0.4780.0770.2140.256 Lâ9NANA0.4880.1120.240â 0.280NANA0.4300.1250.2160.257 Lâ11NANA0.4240.0580.1780.220NANA0.4800.1110.1100.234 Lâ1-0.3150.462NA-0.082NA0.022-0.354-0.013NA-0.171NA-0.179 Lâ70.4360.493NA0.184NA0.3710.4460.502NA0.197NA0.382 Lâ90.4530.532NA0.227NAâ 0.4040.4650.445NA0.142NA0.351 Lâ110.3890.512NA0.074NA0.3250.4230.468NA0.164NA0.352 General Healthcare Legal Tourism Table 3: Spearmanâs (Ï) scores obtained forALOPElayer-wise (L) experiments with LoRMA across different domains and language pairs. The bolded values represent the highest spearman scores obtained for each language pair in each domain. The (â ) represents the highest average obtained in each domain. âNAâ indicates that a language pair is unavailable for that specific domain. evaluation, with zero-shot prompting often yield- ing near-zero or negative correlations. This high- lights the inherent limitations of prompt engineer- ing alone for smaller models, particularly in high- risk domains (Sindhujan et al., 2025b). ALOPEprovides a targeted remedy for this gap, but its effectiveness is strongly domain-dependent. In theLegaldomain,ALOPEsubstantially im- proves performance, underscoring the importance of domain-specific adaptation for semantically pre- cise content. In contrast, gains inHealthcareare limited, suggesting that this domain benefits more from broad pre-training coverage of medical ter- minology in large closed-weight models than from lightweight adapter-based fine-tuning.General andTourismdomains exhibit intermediate be- haviour, indicating that the utility ofALOPEdepends on the interaction between domain complexity and pre-training corpus characteristics. Incorporating LoRMA withinALOPEfurther introduces stability- oriented regularisation, yielding smoother layer- wise behaviour and mitigating lower correlations at shallow layers (Sindhujan et al., 2025c). The layer-wise superiority of intermediate rep- resentations (Layers -9, -11) holds consistently across all five languages, suggesting this reflects fundamental properties of how multilingual LLMs encode cross-lingual semantic alignment rather than language-specific artifacts (Kargaran et al., 2025). This consistency across typologically di- verse languages strengthens the generalisability of intermediate-layer adaptation for QE in low- resource settings. Taken together, our results motivate a condi- tional deployment strategy. When access to closed- weight models is feasible, guideline-anchored prompting offers the most reliable solution. When deployment constraints such as cost, latency, or privacy preclude such access,ALOPEwith LoRA provides a lightweight and effective alternative, par- ticularly for semantically complex domains such as Legal. This underscores the importance of empir- DomainPrompt settingModel family En-Hi En-Mr En-Ta En-Te En-Gu Average General Zero-shot Closed0.4240.5970.8480.3920.924â 0.637 Open0.390-0.0580.7720.3820.8120.460 Few-shot + Guidelines Closed0.4750.5820.8860.2380.7760.591 Open0.4080.0380.8320.4420.8490.514 Few-shot (No Guidelines) Closed0.5630.3140.940-0.0310.8600.529 Open0.4180.3750.8670.4860.7520.580 Healthcare Zero-shot Closed0.1260.8140.366NA0.3460.413 Open0.5690.3890.603NA0.4940.514 Few-shot + Guidelines Closed0.4150.669-0.040NA0.1000.286 Open0.4470.8840.411NA0.2110.488 Few-shot (No Guidelines) Closed0.5850.7860.168NA0.4580.385 Open0.6110.8840.422NA0.398â 0.579 Legal Zero-shot ClosedNANA0.7490.1090.6770.512 OpenNANA0.4180.2870.2650.323 Few-shot + Guidelines ClosedNANA0.7170.5300.699â 0.649 OpenNANA0.4180.5280.7270.558 Few-shot (No Guidelines) ClosedNANA0.4750.2300.4750.393 OpenNANA0.7370.6870.4730.632 Tourism Zero-shot Closed0.4160.474NA0.217NA0.369 Open0.6130.689NA0.636NAâ 0.646 Few-shot + Guidelines Closed0.5020.679NA0.472NA0.551 Open0.1580.702NA0.583NA0.481 Few-shot (No Guidelines) Closed0.5090.685NA0.397NA0.530 Open0.7370.687NA0.473NA0.632 Table 4: Reports best Spearmanâs correlation (Ï) achieved for each domain and language pair across all evaluated models, including both open- and closed-weight LLMs. The bolded values represent the highest spearman scores obtained for each language pair in each domain. The (â ) represents the highest average obtained in each domain. Detailed model-wise results are provided in Appendices F, G, and H. ical validation prior to adopting adaptation strate- gies, as their effectiveness depends on domain- specific interactions with pre-training data. 6 Conclusion This work investigates domain-specific quality estimation for EnglishâIndic translation across Healthcare,Legal,Tourism, andGeneral domains, covering five language pairs (Hindi, Marathi, Tamil, Telugu, Gujarati). We system- atically compare prompt-only approaches with parameter-efficientALOPEadaptation, revealing that closed-weight models with guideline-anchored prompting achieve robust performance without pa- rameter updates, while open-weight models ex- hibit substantial fragility under prompt-only eval- uation.ALOPEdemonstrates reasonable QE per- formance even with smaller open-source LLMs when intermediate Transformer layer embeddings are utilised for quality estimation. Our findings support a conditional deployment strategy: priori- tise closed-weight prompting when API access is viable; applyALOPEwith LoRA for open-weight models in resource-constrained environment; and useALOPEwith LoRMA when precise layer tun- ing is constrained. Future work should investigate multi-layer fusion approaches and interpretability techniques to understand which linguistic phenom- ena drive layer-specific improvements. 7 Limitations Our study is limited to EnglishâIndic language pairs across four domains, due to the limited avail- ability of QE data for other language pairs and domains. Further work is needed to examine the generalisability of these findings to other language families and more specialised technical domains. In addition, allALOPEfine-tuning experiments use a relatively small backbone model (LLaMA-3.2-3B Instruct) due to computational constraints, and re- sults may differ when scaling to larger open-weight models. References Harsh Bihany, Shubham Patel, and Ashutosh Modi. 2025. LoRMA: Low-rank multiplicative adaptation for LLMs. In Findings of the Association for Compu- tational Linguistics: ACL 2025, pages 10113â10133, Vienna, Austria. Association for Computational Lin- guistics. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few- shot learners. In Advances in Neural Information Processing Systems (NeurIPS). Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Ben- esty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coeffi- cient. Noise reduction in speech processing, pages 1â4. Tim Dettmers and 1 others. 2023.Qlora: Effi- cient finetuning of quantized llms. arXiv preprint, arXiv:2305.14314. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. Proceed- ings of the 7th Linguistic Annotation Workshop. Deepak Gupta, Rishabh Agarwal, and Swaroop Mishra. 2024. Domain dynamics: Evaluating large language models in english-hindi translation. In Proceedings of the International Conference on Computational Linguistics (COLING 2024). Association for Compu- tational Linguistics. Edward J. Hu, Yelong Shen, Phil Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. International Conference on Learning Rep- resentations (ICLR). Amir Kargaran and 1 others. 2025. Layer-wise cross- lingual alignment in low-resource languages. arXiv preprint. Tom Ko Ë cmi and Christian Federmann. 2023. Gemba- mqm: Detecting translation quality error spans with gpt-4. In Proceedings of the Eighth Conference on Machine Translation. Association for Computational Linguistics. Yifan Li, Yimeng Chen, and Min Zhang. 2024. Topic- guided example selection for domain adaptation in LLM-based machine translation. In Proceedings of the Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP). Association for Compu- tational Linguistics. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. Eetu MĂ€kelĂ€, Saad Khalid, Lorenzo Gatti, and Matteo Negri. 2024. OpenWHO: A document-level paral- lel corpus for health translation in low-resource lan- guages. In Proceedings of the Ninth Conference on Machine Translation (WMT 2024). Association for Computational Linguistics. Swaroop Mishra and 1 others. 2022. Cross-task gener- alization via natural language crowdsourcing instruc- tions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Shubham Patel, Harsh Bihany, and Ashutosh Modi. 2024. Exploring the effectiveness of LLM domain adaptation for business IT machine translation. In Proceedings of the Workshop on Domain Adaptation and Representation Transfer. Association for Com- putational Linguistics. Philip Sedgwick. 2014. Spearmanâs rank correlation coefficient. Bmj, 349. Archchana Sindhujan, Diptesh Kanojia, Constantin Orasan, and Shenbin Qian. 2025a. When LLMs struggle: Reference-less translation evaluation for low-resource languages. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, pages 437â459, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Archchana Sindhujan, Diptesh Kanojia, and Constantin Or Ì asan. 2025b. Reference-less evaluation of machine translation: Navigating through the resource-scarce scenarios. Information, 16(10). Archchana Sindhujan, Shenbin Qian, Chan Chi Chun Matthew, Constantin Or Ì asan, and Diptesh Kanojia. 2025c. ALOPE: Adaptive layer optimization for translation quality estimation using large language models. arXiv preprint, arXiv:2508.07484. Aayush Singh, Meet Patel, Ravi Shah, and Pushpak Bhattacharyya. 2025. Team-SVNIT at JUST-NLP 2025: Domain-adaptive fine-tuning of multilingual models for english-hindi legal machine translation. In Proceedings of the Joint Workshop on NLP for Legal Applications. Association for Computational Linguistics. Lucia Specia, Dilip Raj, and 1 others. 2018. Qual- ity estimation for machine translation. In Machine Translation Summit XVI. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bert rediscovers the classical nlp pipeline. Proceedings of ACL. Chrysoula Zerva, FrĂ©dĂ©ric Blain, JosĂ© G. C. De Souza, Diptesh Kanojia, Sourabh Deoghare, Nuno M. Guer- reiro, Giuseppe Attanasio, Ricardo Rei, Constantin Or Ì asan, Matteo Negri, Marco Turchi, Rajen Chat- terjee, Pushpak Bhattacharyya, Markus Freitag, and AndrĂ© Martins. 2024. Findings of the quality estima- tion shared task at WMT 2024: Are LLMs closing the gap in QE? In Proceedings of the Ninth Con- ference on Machine Translation (WMT 2024), pages 82â109. Association for Computational Linguistics. Chrysoula Zerva, FrĂ©dĂ©ric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, JosĂ© G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Or Ì asan, Marina Fomicheva, AndrĂ© F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 69â99, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics. Haofei Zhao, Yilun Liu, Shimin Tao, Weibin Meng, Yimeng Chen, Xiang Geng, Chang Su, Min Zhang, and Hao Yang. 2024. From handcrafted features to llms: A brief survey for machine translation quality estimation. arXiv preprint, arXiv:2403.14118. Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the International Conference on Ma- chine Learning (ICML). A Prompt Templates for Prompt-only Approaches Figure 2: Few-shot QE Prompt (Without Guidelines) Figure 3: Few-shot QE Prompt (With Guidelines) Figure 4: Zero-shot prompt B Prompt Templates for ALOPE Figure 5: ALOPE prompt C Pearsonâs Correlation Scores obtained for ALOPE with LoRA LayerR = 32, α = 16R = 64, α = 32R = 128, α = 32 En-HiEn-MrEn-TaEn-TeEn-GuAvgEn-HiEn-MrEn-TaEn-TeEn-GuAvgEn-HiEn-MrEn-TaEn-TeEn-GuAvg Lâ10.400-0.0250.3050.0920.2210.1990.6280.1540.6800.2290.5380.4460.5970.1110.6790.1810.5940.433 Lâ70.5380.1170.6450.2440.2720.3630.4880.0530.6740.2160.5620.3990.3060.0430.5270.1770.4710.305 Lâ90.4920.0810.6770.2500.6790.4360.6210.0770.7160.3050.6630.4760.3280.0950.5490.1310.5110.323 Lâ110.5390.0800.7200.0160.6870.4080.5820.0190.6740.0310.6370.3890.5850.0390.6470.2570.5830.422 Lâ10.5890.0010.405NA0.3540.3370.6350.0510.494NA0.4400.4050.7070.1640.464NA0.4260.440 Lâ70.5140.1580.502NA0.3810.3890.6820.0080.523NA0.4990.4280.7080.1800.617NA0.4650.493 Lâ90.5540.0760.435NA0.3420.3520.6880.0940.629NA0.4050.4540.6160.0020.608NA0.3580.396 Lâ110.6250.0150.611NA0.4250.4190.6140.1930.630NA0.4690.4770.685-0.0180.602NA0.4960.441 Lâ1NANA0.0590.0020.0110.024NANA0.4680.0400.3180.276NANA0.4300.1000.3820.304 Lâ7NANA0.5020.0770.4570.345NANA0.4960.0880.4210.335NANA0.4600.0930.3550.303 Lâ9NANA0.5510.1420.3870.360NANA0.4590.0590.3340.284NANA0.4520.0810.3540.295 Lâ11NANA0.4470.0010.3340.261NANA0.6210.2150.5360.457NANA0.6210.2150.5360.457 Lâ10.0590.002NA0.011NA0.0240.1190.064NA0.083NA0.0890.1300.075NA0.095NA0.100 Lâ70.0610.467NA0.145NA0.2240.2200.544NA0.286NA0.3500.2600.580NA0.320NA0.387 Lâ90.1030.557NA0.069NA0.2430.3170.614NA0.235NA0.3890.3600.660NA0.270NA0.430 Lâ110.4850.641NA0.219NA0.4480.4850.641NA0.219NA0.4480.5250.685NA0.245NA0.485 General Healthcare Legal Tourism Table 5: ALOPE (LoRA) layer-wise (L) Pearsonâs (r) scores obtained across domains and language pairs for different adapter configurations. D Pearsonâs Correlation Scores obtained for ALOPE with LoRMA LayerR = 64, α = 32R = 128, α = 32 En-HiEn-MrEn-TaEn-TeEn-GuAvgEn-HiEn-MrEn-TaEn-TeEn-GuAvg Lâ10.2620.0300.5520.0530.4300.2590.2170.1200.383-0.0070.3110.205 Lâ70.3820.3150.5570.0670.4690.3580.4070.0830.5130.0930.4720.314 Lâ90.3310.2740.6000.0450.4650.3430.3280.2830.5900.0460.4950.348 Lâ110.3940.2720.5980.1050.4910.3720.3950.2710.5400.0520.4920.350 Lâ10.4200.008-0.245NA0.3910.1440.1980.0090.306NA0.4710.246 Lâ70.2240.1300.300NA0.4310.2710.2730.0160.333NA0.4290.263 Lâ90.3450.0710.277NA0.4110.2760.4140.1360.285NA0.4700.326 Lâ110.3740.0340.335NA0.4450.3000.4550.1160.325NA0.4240.330 Lâ1NANA0.3750.0320.3040.237NANA0.4790.1000.3390.306 Lâ7NANA0.4620.0430.3360.280NANA0.4560.0460.3450.282 Lâ9NANA0.4670.1130.3650.315NANA0.4640.1390.3560.323 Lâ11NANA0.4400.0950.3360.290NANA0.4470.0920.3210.287 Lâ1-0.1050.468NA0.176NA0.180-0.283-0.002NA0.134NA0.045 Lâ70.3840.492NA0.100NA0.3250.3960.512NA0.052NA0.320 Lâ90.3930.539NA0.153NA0.3620.4230.447NA0.089NA0.320 Lâ110.3600.524NA-0.057NA0.2760.3820.478NA0.043NA0.301 General Healthcare Legal Tourism Table 6: ALOPE (LoRMA) layer-wise (L) Pearsonâs (r) scores obtained across domains and language pairs for different adapter configurations. E ALOPE: Domain-wise Comparison of Average Performance Figure 6: ALOPE with LoRA: Average Spearmanâs (Ï) across domains. Figure 7: ALOPE with LoRMA: Average Spearmanâs (Ï) across domains. F Zero-shot Evaluation Results Domain ModelEn-HiEn-MrEn-TaEn-TeEn-GuAvg r Ï r Ï r Ï r Ï r Ï r Ï General Gemini-1.5-Pro -0.082 -0.232 0.120 0.159 0.856 0.848 0.191 0.073 0.774 0.924 0.372 0.354 Gemini-2.5-Pro 0.328 0.424 0.747 0.597 0.715 0.805 0.542 0.392 0.918 0.786 0.650 0.601 Qwen3-14B0.319 0.390 -0.159 -0.058 0.730 0.772 0.393 0.328 0.499 0.649 0.356 0.416 Gemma3-27B0.079 0.116 -0.259 -0.290 0.776 0.701 0.532 0.382 0.195 0.102 0.265 0.202 LLaMA-3.2-3B 0.065 0.056 -0.184 -0.164 0.099 0.249 -0.366 -0.381 -0.174 -0.024 -0.112 -0.053 LLaMA-3.1-8B -0.156 -0.284 -0.512 -0.465 0.587 0.437 0.265 0.115 0.662 0.812 0.169 0.123 Healthcare Gemini-1.5-Pro 0.134 0.039 0.668 0.814 -0.068 0.056NANA0.369 0.346 0.276 0.294 Gemini-2.5-Pro 0.532 0.126 0.784 0.811 0.315 0.366NANA0.345 0.238 0.494 0.385 Qwen3-14B0.113 0.435 0.548 0.311 0.145 0.234NANA0.134 0.130 0.235 0.278 Gemma3-27B0.561 0.569 0.292 0.389 0.424 0.603NANA0.523 0.494 0.450 0.514 LLaMA-3.2-3B -0.462 -0.234 0.193 0.174 0.385 0.406NANA0.228 -0.055 0.086 0.073 LLaMA-3.1-8B 0.001 -0.071 -0.196 -0.087 -0.180 0.100NANA-0.143 -0.395 -0.13 -0.113 Legal Gemini-1.5-ProNANANANA0.899 0.749 0.241 0.109 0.683 0.677 0.608 0.512 Gemini-2.5-ProNANANANA0.353 0.475 0.043 0.055 0.607 0.457 0.334 0.329 Qwen3-14BNANANANA0.343 0.418 -0.689 -0.676 0.240 0.265 -0.035 0.002 Gemma3-27BNANANANA-0.206 -0.056 -0.035 -0.045 0.260 0.201 0.006 0.033 LLaMA-3.2-3BNANANANA-0.254 -0.174 -0.106 -0.011 -0.279 -0.429 -0.213 -0.205 LLaMA-3.1-8BNANANANA0.013 0.031 0.270 0.287 0.045 -0.105 0.109 0.071 Tourism Gemini-1.5-Pro 0.137 0.241 0.324 0.474NANA0.109 0.180NANA0.190 0.298 Gemini-2.5-Pro 0.327 0.416 0.360 0.301NANA0.367 0.217NANA0.351 0.311 Qwen3-14B0.111 0.076 0.076 -0.058NANA0.588 0.541NANA0.258 0.186 Gemma3-27B0.129 0.102 0.804 0.654NANA0.633 0.636NANA0.522 0.464 LLaMA-3.2-3B 0.468 0.417 0.432 0.469NANA0.459 0.309NANA0.453 0.398 LLaMA-3.1-8B 0.582 0.613 0.839 0.689NANA0.734 0.584NANA0.718 0.629 Table 7: Zero-shot prompt-only QE performance. Spearmanâs (Ï) and Pearsonâs (r) scores are reported for all language pairs. BestÏper language pair in bold.âNAâ indicates that a language pair is unavailable for that specific domain. G Few-shot with Guidelines Evaluation Results Domain ModelEn-HiEn-MrEn-TaEn-TeEn-GuAvg r Ï r Ï r Ï r Ï r Ï r Ï General Gemini-1.5-Pro -0.004 -0.094 -0.063 0.087 0.881 0.850 -0.319 -0.169 0.859 0.761 0.271 0.287 Gemini-2.5-Pro 0.625 0.475 0.638 0.582 0.834 0.886 0.255 0.238 0.926 0.776 0.656 0.591 Qwen3-14B0.247 0.279 -0.159 -0.058 0.720 0.783 0.498 0.442 0.850 0.849 0.431 0.459 Gemma3-27B0.489 0.376 0.041 0.038 0.770 0.832 0.502 0.352 0.799 0.649 0.520 0.449 LLaMA-3.2-3B -0.637 -0.527 -0.101 -0.251 0.479 0.329 0.150 0.186 -0.332 -0.200 -0.088 -0.093 LLaMA-3.1-8B 0.271 0.408 -0.116 0.034 -0.312 -0.339 -0.149 -0.145 -0.138 -0.288 -0.089 -0.066 Healthcare Gemini-1.5-Pro 0.170 0.415 0.614 0.669 -0.237 -0.040NANA0.604 0.100 0.288 0.286 Gemini-2.5-Pro 0.191 0.309 0.221 0.024 -0.302 -0.089NANA0.584 0.084 0.174 0.082 Qwen3-14B0.176 0.225 0.886 0.884 -0.144 -0.240NANA-0.357 -0.389 0.140 0.120 Gemma3-27B0.255 0.345 0.193 0.174 0.076 0.123NANA0.225 0.109 0.187 0.188 LLaMA-3.2-3B 0.012 0.058 -0.499 -0.285 -0.476 -0.284NANA-0.381 -0.479 -0.336 -0.248 LLaMA-3.1-8B 0.204 0.447 -0.541 -0.458 0.257 0.411NANA0.612 0.211 0.133 0.153 Legal Gemini-1.5-ProNANANANA0.867 0.717 0.446 0.530 -0.078 0.072 0.412 0.440 Gemini-2.5-ProNANANANA0.615 0.532 0.375 0.324 0.849 0.699 0.613 0.518 Qwen3-14BNANANANA0.343 0.418 -0.689 -0.676 0.240 0.265 -0.035 0.002 Gemma3-27BNANANANA0.249 0.099 0.195 0.102 0.699 0.727 0.381 0.309 LLaMA-3.2-3BNANANANA-0.333 -0.477 0.378 0.528 0.293 0.304 0.113 0.118 LLaMA-3.1-8BNANANANA0.254 0.104 0.121 0.030 -0.063 0.087 0.104 0.074 Tourism Gemini-1.5-Pro 0.447 0.502 0.630 0.480NANA0.174 0.308NANA0.417 0.430 Gemini-2.5-Pro 0.379 0.385 0.755 0.679NANA0.327 0.472NANA0.487 0.512 Qwen3-14B0.059 0.060 0.371 0.236NANA0.433 0.583NANA0.288 0.293 Gemma3-27B0.226 0.078 0.852 0.702NANA0.002 0.051NANA0.360 0.277 LLaMA-3.2-3B -0.471 -0.459 -0.248 -0.098NANA-0.118 0.032NANA-0.279 -0.175 LLaMA-3.1-8B 0.008 0.158 -0.208 -0.058NANA-0.068 -0.218NANA-0.089 -0.039 Table 8: Few-shot with guidelines QE performance. Spearmanâs (Ï) and Pearsonâs (r) scores are reported for all language pairs. BestÏper language pair in bold. âNAâ indicates that a language pair is unavailable for that specific domain. H Few-shot without Guidelines Evaluation Results Domain ModelEn-HiEn-MrEn-TaEn-TeEn-GuAvg r Ï r Ï r Ï r Ï r Ï r Ï General Gemini-1.5-Pro 0.107 0.022 -0.213 -0.103 0.744 0.743 -0.035 -0.185 0.941 0.791 0.309 0.254 Gemini-2.5-Pro 0.713 0.563 0.447 0.314 0.888 0.940 -0.095 -0.031 0.939 0.860 0.578 0.529 Qwen3-14B-0.128 -0.040 -0.235 -0.234 0.761 0.867 0.319 0.235 0.658 0.752 0.275 0.316 Gemma3-27B0.177 0.234 0.489 0.375 0.519 0.450 0.636 0.486 0.765 0.627 0.517 0.434 LLaMA-3.2-3B 0.497 0.418 -0.101 0.014 0.393 0.477 0.223 0.188 0.078 0.228 0.218 0.265 LLaMA-3.1-8B 0.234 0.084 0.016 0.109 -0.332 -0.482 -0.180 -0.177 0.097 0.005 -0.033 -0.092 Healthcare Gemini-1.5-Pro 0.568 0.307 0.852 0.552 -0.027 0.168NANA0.608 0.458 0.500 0.371 Gemini-2.5-Pro 0.670 0.585 0.712 0.786 0.060 -0.086NANA0.388 0.393 0.458 0.420 Qwen3-14B0.261 0.413 0.865 0.884 -0.117 -0.075NANA-0.049 -0.143 0.240 0.270 Gemma3-27B0.083 0.187 0.097 0.114 0.132 0.275NANA0.216 0.174 0.132 0.188 LLaMA-3.2-3B 0.513 0.611 0.123 0.035 0.648 0.422NANA0.139 0.071 0.356 0.285 LLaMA-3.1-8B 0.519 0.406 -0.244 0.289 -0.101 0.250NANA0.283 0.398 0.114 0.336 Legal Gemini-1.5-ProNANANANA0.571 0.475 0.355 0.230 0.170 0.061 0.365 0.255 Gemini-2.5-ProNANANANA-0.342 -0.492 -0.262 -0.112 0.625 0.475 0.007 -0.043 Qwen3-14BNANANANA0.350 0.467 0.365 0.215 0.332 0.473 0.349 0.385 Gemma3-27BNANANANA0.277 0.311 0.828 0.687 0.304 0.211 0.470 0.403 LLaMA-3.2-3BNANANANA-0.471 -0.459 -0.248 -0.098 -0.118 0.032 -0.279 -0.175 LLaMA-3.1-8BNANANANA0.887 0.737 -0.438 -0.335 0.094 -0.026 0.181 0.125 Tourism Gemini-1.5-Pro 0.142 0.193 0.835 0.685NANA-0.126 -0.208NANA0.284 0.223 Gemini-2.5-Pro 0.359 0.509 0.501 0.406NANA0.422 0.397NANA0.427 0.437 Qwen3-14B0.350 0.467 0.365 0.215NANA0.332 0.473NANA0.349 0.485 Gemma3-27B0.277 0.311 0.828 0.687NANA0.304 0.211NANA0.470 0.403 LLaMA-3.2-3B -0.471 -0.459 -0.248 -0.098NANA-0.118 0.032NANA-0.279 0.175 LLaMA-3.1-8B 0.887 0.737 -0.438 -0.335NANA0.094 -0.026NANA0.181 0.125 Table 9: Few-shot without guidelines QE performance. Spearmanâs (Ï) and Pearsonâs (r) scores are reported for all language pairs. BestÏper language pair in bold. âNAâ indicates that a language pair is unavailable for that specific domain. I Comparison of ALOPE and Prompt-only Approaches DomainPrompt settingEn-HiEn-MrEn-TaEn-TeEn-GuAvg Zero-shot0.056-0.1640.249-0.381-0.024-0.053 Few-shot + Guidelines-0.527-0.2510.3290.186-0.200-0.093 Few-shot (No Guidelines)0.4180.0140.4770.1880.2280.265 ALOPE (LoRA)0.4760.1570.6100.2920.485â 0.404 ALOPE (LoRMA)0.3920.3140.5070.1680.4210.360 Zero-shot-0.2340.1740.406NA-0.0550.073 Few-shot + Guidelines0.058-0.285-0.284NA-0.479-0.248 Few-shot (No Guidelines)0.6110.0350.422NA0.0710.285 ALOPE (LoRA)0.5200.1920.415NA0.532â 0.415 ALOPE (LoRMA)0.3650.1310.413NA0.5120.355 Zero-shotNANA-0.174-0.011-0.429-0.205 Few-shot + GuidelinesNANA-0.4770.5280.3040.118 Few-shot (No Guidelines)NANA-0.459-0.0980.032-0.175 ALOPE (LoRA)NANA0.5810.2670.445â 0.431 ALOPE (LoRMA)NANA0.5150.1250.2160.265 Zero-shot0.4170.469NA0.309NA0.398 Few-shot + Guidelines-0.459-0.098NA0.032NA-0.175 Few-shot (No Guidelines)-0.459-0.098NA0.032NA-0.175 ALOPE (LoRA)0.3500.670NA0.220NAâ 0.413 ALOPE (LoRMA)0.4650.532NA0.227NA0.408 General Healthcare Legal Tourism Table 10: Comparison of Spearmanâs correlation (Ï) achieved by LLaMA-3.2-3B across Prompt-only approaches and ALOPE. The bolded values represent the highest Spearman scores obtained for each language pair in each domain. The (â ) represents the highest average obtained in each domain. âNAâ indicates that a language pair is unavailable for that specific domain.