Paper deep dive
TermJudge: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation
Nicolas Dahan, Fran{\cc}ois Yvon, Rachel Bawden
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Existing automatic metrics for evaluating terminological use in machine translation (MT) penalise any divergence from a fixed reference, conflating translation errors with the valid terminological variation that human translators routinely produce. We introduce TermJudge, a document-level terminology metric that assigns an interpretable verdict to every term occurrence: glossary-conforming occurrences are settled deterministically, while divergences are assessed under a two-step LLM-as-judge procedure using the full document context: the first detects and labels terminology errors; the second sorts valid document-level variations from inconsistencies. Validated against expert error annotations and document-level human MQM scores, TermJudge ranks first in both system- and segment-level meta-evaluation, ahead of glossary-conformity and quality-estimation baselines. When applied to eight systems translating academic documents, under two prompting conditions, we observe that glossary injection improves terminology translation in all paired comparisons, by removing genuine errors rather than valid variation. TermJudge is released as open-source code.
Tags
Links
- Source: https://arxiv.org/abs/2609.35017v1
- Canonical: https://arxiv.org/abs/2609.35017v1
Trouble viewing inline? Open PDF directly â
Full Text
156,510 characters extracted from source content.
Expand or collapse full text
TERMJUDGE: A Document-Level Metric Judging, Not Counting, Terminology in Machine Translation Evaluation Nicolas Dahan âąâ François Yvon â Rachel Bawden âą âą Inria Paris, France â Sorbonne UniversitĂ©, CNRS, ISIR, Paris, France nicolas.dahan,rachel.bawden@inria.fr yvon@isir.upmc.fr Abstract Existing automatic metrics for evaluating ter- minological use in machine translation (MT) penalise any divergence from a fixed refer- ence, conflating translation errors with the valid terminological variation that human transla- tors routinely produce. We introduce TERM- JUDGE, a document-level terminology metric that assigns an interpretable verdict to every term occurrence: glossary-conforming occur- rences are settled deterministically, while di- vergences are assessed under a two-step LLM- as-a-judge procedure using the full document context: the first detects and labels terminol- ogy errors; the second sorts valid document- level variations from inconsistencies. Validated against expert error annotations and document- level human MQM scores, TERMJUDGE ranks first in both system- and segment-level meta- evaluation, ahead of glossary-conformity and quality-estimation baselines. When applied to eight systems translating academic documents, under two prompting conditions, we observe that glossary injection improves terminology translation in all paired comparisons, by remov- ing genuine errors rather than valid variation. TERMJUDGE is released as open-source code. 1 1 Introduction Evaluating machine translation (MT) at the docu- ment level in specialised settings requires assessing how systems translate the technical vocabulary of those documents. In most documents, technical concepts can be split into two main groups: rare concepts that only occur once or twice, usually under the same surface form; and frequent con- cepts, which may appear under multiple guises. For example, in a research paper on MT, a con- cept such asmachine translationmay occur repeatedly, as the preferred form (machine transla- tion ), an acronym (MT ), a reduction (translation), a lexical substitution (automatic translation), or 1 https://github.com/nicolasdahan/termjudge a morphosyntactic reformulation (machine trans- lated (text)) (Daille, 2017); translators may am- plify this variation and introduce further alterna- tions for stylistic or discourse reasons (Bowker, 1998; FernĂĄndez-Silva and Kerremans, 2011). Existing automatic metrics for evaluating the translation of terms decide whether a target form is acceptable through matching: against a term- annotated reference (Farajian et al., 2018), against a bilingual glossary entry (ibn Alam et al., 2021; Semenov et al., 2025), against the documentâs other occurrences of the same concept (Itagaki et al., 2007; Semenov and Bojar, 2022), or against source-side variation patterns (Dahan et al., 2026b). None of them asks whether a mismatch is a transla- tion error or a valid translation choice (e.g., target- language amplification, polysemy disambiguation, acronym introduction at first mention, stylistic al- ternation), so they may conflate acceptable transla- tions with errors. General-purpose neural metrics offer no remedy, losing correlation with human judgement precisely on the specialised domains where terminology errors concentrate (Zouhar et al., 2024). The question we address is therefore whether an automatic metric can decide, for each occurrence of a concept, whether the output form is acceptable rather than merely identical to a pre- determined reference. We propose to address this gap with TERMJUDGE, an automatic document- level terminology metric that not only counts diver- gences (translations that depart from the expected form of a term) but judges each one, and returns interpretable verdicts alongside an aggregate score. Our contributions are: âą A concept-aware document-level terminology metric, built on an LLM-as-a-judge approach comprising several steps (reference valida- tion, then error classification and document- consistency qualification), grounded in estab- lished translation studies, which discriminates arXiv:2609.35017v1 [cs.AI] 28 Sep 2026 errors from valid variation and produces an interpretable verdict profile for each system rather than a single opaque score. âąBioMQM-Terms, a terminology-evaluation re- source derived from bio-MQM (Zouhar et al., 2024), a corpus containing MT outputs in the biomedical domain, with error annota- tions. To this we add a bilingual glossary of 729 EnglishâFrench biomedical concepts and 13,200 aligned termâtranslation pairs for ten MT systems, released with our code. âąA series of empirical validations:per- occurrence agreement with a fine-grained hu- man error typology on a geoscience corpus (STEP; Cornejo CĂĄrcamo et al., 2026), a document-level meta-evaluation on BioMQM- Terms, where TERMJUDGE ranks first under every aggregation of its score, ahead of ten alternative metrics, and a reevaluation of the use of glossaries to guide LLM-based MT: on two NLP test corpora, we find that glossary- guided prompting improves term translation. 2 Related Work For the global evaluation of MT quality, some of the most widely used automatic metrics (alongside traditional surface-based metrics such as BLEU (Papineni et al., 2002) and chrF (Popovi Ì c, 2015)) are fine-tuned neural models such as COMET (Rei et al., 2020) and METRICX (Juraska et al., 2024), together with their reference-free counterparts (Rei et al., 2022). They are trained and mostly used at the sentence level. Despite proposals to ex- tend them to longer spans (Vernikos et al., 2022; Deutsch et al., 2023b), their document-level appli- cation remains questionable (Dahan et al., 2026a). Two further properties limit their use for termi- nology. First, fine-tuned metrics lose substantial correlation with human judgements on unseen spe- cialised domains, precisely where terminology er- rors concentrate (Zouhar et al., 2024). Second, such metrics summarise all aspects of translation quality in a single scalar score, failing to distinguish termi- nology errors from other types of mistranslations. Evaluation practice is therefore increasingly dominatedbyLLM-as-a-judgeapproaches. GEMBA involves eliciting direct quality scores (Kocmi and Federmann, 2023b), and GEMBA- MQM annotating error spans within the generic MQM typology (Lommel et al., 2013; Kocmi and Federmann, 2023a); Minder et al. (2025) test how reliably LLMs annotate specialised translations with such error typologies: error detection reaches satisfactory levels, but accurate categorisation requires injecting the error definitions in the prompt. LLM-as-a-judge approaches bring the flexibility that qualitative judgements require, but they carry documented biases (order of presen- tation, verbosity preference, instability across rephrased queries; Zheng et al., 2023), making careful meta-evaluation against human judgements indispensable (Mathur et al., 2020; Deutsch et al., 2023a). Whether applied to isolated segments or to whole documents, they annotate general error categories and do not track how consistently a concept is rendered across a document. Dedicated terminology evaluation has mostly re- lied on matching the output against expected target forms. Given a term-annotated reference transla- tion, Farajian et al. (2018) compute a term hit rate, a BLEU-like clipped count of the reference terms retrieved in the output, which TermEval (Haque et al., 2023) extends by also accepting the lexical and inflectional variations of each reference term listed in its termbank. Given a bilingual glossary entry, metrics count how often the expected tar- get term appears in the output, from exact surface matching (ibn Alam et al., 2021) to lemma-aware matching (Alam et al., 2021), as adopted in the WMT terminology shared tasks (Alam et al., 2021; Semenov et al., 2023), whose 2025 edition extends the exercise to document-level translation with one- to-many dictionaries (Semenov et al., 2025). What- ever the matching strategy, these metrics remain anchored to an inventory of expected forms: un- listed reductions, context-dependent acronyms and lexical variants are counted as failures irrespective of their adequacy in context. When no glossary is available, consistency met- rics score the target side against itself. Semenov and Bojar (2022) match each occurrence against a pseudo-reference derived from the systemâs own dominant translation, concentration indices such as the HerfindahlâHirschman index (Itagaki et al., 2007; GaĆĄpar et al., 2022) and the Lexical Trans- lation Consistency Ratio (Lyu et al., 2021; Wang et al., 2025) reward the reuse of a single form, and cross-term coherence extends the family to the source side, counting how many variation relation- ships survive translation (Dahan et al., 2026b). All share one assumption: a term is well translated when every occurrence receives the same trans- lation. That assumption keeps them cheap and reference-free, but blind to the difference between undue changes and motivated alternations. It is also only warranted for prescriptive terminologies: terminology standards themselves recognise a scale of acceptability from preferred to merely admitted forms (ISO, 2019). That difference is precisely what studies of spe- cialised discourse document. Denominative vari- ation is a constitutive property of terminology in running text (Daille, 2017), refined into graphical, morphosyntactic, reduction, expansion and lexical variants by Cornejo CĂĄrcamo et al. (2025). Trans- lators preserve or even accentuate source-side vari- ation (FernĂĄndez-Silva and Kerremans, 2011), al- ternate between several translations for discourse and stylistic reasons (Vinay and Darbelnet, 1972; Bowker, 1998; Pecman, 2014), introduce pragmatic and explicitness shifts (Chesterman, 2016; Blum- Kulka, 1986), and resist the over-standardisation that erases author-intended distinctions (Bowker and Hawkins, 2006). MT systems, by contrast, pro- duce less variation than humans ( Ë Culo and Nitzke, 2016). These findings have not reached automatic evaluation: no existing terminology metric asks whether a divergence from the expected form is one of these motivated variations or an error. 3 The TERMJUDGE Metric TERMJUDGE evaluates the terminological ade- quacy and consistency of a translated document by assigning a verdict to the translation of every term occurrence, and by aggregating these verdicts into a document-level score that can rank systems on terminological quality. A preprocessing stage turns a terminology and a translated corpus into aligned term occurrences (§3.1); TERMJUDGE it- self is a five-step pipeline (§3.2) whose decision flow is displayed in Figure 1. Appendix A (Fig- ure 2) gives a graphical overview of both stages. TERMJUDGE shares its starting point with the con- sistency tradition (§2): each source form is evalu- ated separately, through the set of translations it receives across the document. The variation being adjudicated is therefore target-side only, source- side variation being factored out by construction. For this, TERMJUDGE first computes a reference translation for each source form, and flags as diver- gence any translation that departs from it. It then submits each divergence to an LLM judge in two steps: a first question, Q1, asks whether the diver- gence is an error and of which kind. When no error 1â2 3 4 STEPS 1â2 · reference & matching t = R(s)? STEP 3 · error judge Q1: error? STEP 4 · consistency judge Q2: justified? â CONFORMING â A1..A7, C..H â JUSTIFIED â UNJUSTIFIED aligned occurrence(sâ t) match divergence error no error yes no Figure 1: The TERMJUDGE decision flow (§3.2): a match withR(s)is considered CONFORMING without any LLM call; a divergence is classified by the error judge (Q1) then, if error-free, by the consistency judge (Q2), as JUSTIFIED or UNJUSTIFIED. is found, a second question, Q2, asks whether the variation is a valid choice justified by its context. Both questions are answered by the same model using different prompts, called the error judge and the consistency judge in what follows. The decom- position is deliberate. The two questions do not judge the same object: Q1 assesses the translation itself, so its answer is independent of the occur- renceâs position and can be shared across identical translations (§3.2), and Q2 assesses the consistency of a choice with the conceptâs other translations in the document, and is asked per occurrence. This reflects a key distinction in our error typology: be- tween errors in a translation taken on its own and in- consistencies between the translations of a concept within a document (Appendix D). Merging the two questions into a single LLM call would make every judgement positional, at the occurrence level. It would also imply a much larger label set, possibly degrading the judgeâs decisions (Appendix D.1). Answering Q1 at the form-level does not mean judging it in isolation: like for Q2, the judge also receives the corresponding conceptâs glossary entry and the translations of its other source forms. 3.1 Data Requirements and Preprocessing TERMJUDGE relies on a single external resource, a bilingual terminology of the domain structured as a SKOS glossary: 2 each entry pairs a concept with one preferred term (or head term) per language, and a list of alternative terms, or variants. The object of the evaluation is a collection of source documents together with their translations by one or more MT systems, with aligned source and target segments; no human reference translation is needed. 2 Simple Knowledge Organization System,https://w. w3.org/TR/skos-reference/. A glossary, however large, cannot delimit the forms under which concepts surface: beyond glos- sary variants listed as alternative terms, running texts also introduce non-glossary variants (reorder- ings, reductions, context-dependent acronyms). Term occurrences are therefore identified with the CONCORDANCER, 3 an in-house tool that takes the glossary as input and detects, in each source docu- ment, the occurrences of every concept. This raises coverage beyond exact-form and lemma match- ing by also recognising variants derived from glos- sary entries or detected in context: on PARANLP and IWSLT23, lemma matching alone would miss about 15% of the occurrences the CONCOR- DANCER detects (Dahan et al., 2026b). Each attested source form is then labelled au- tomatically with its variation category relative to the head term (procedure and validation in Ap- pendix A), following the typology of §2: no vari- ation, graphical, morphosyntactic, reduction, ex- pansion, lexical or combined (Cornejo CĂĄrcamo et al., 2025) (one example per category in Table 8, Appendix A). Each occurrence is then aligned with the target-side span that corresponds to its transla- tion, identified by an LLM call within the aligned target segment rather than the whole document; 4 both sides are normalised for case, punctuation and whitespace, and lemmatised identically. TERMJUDGE starts from the concept and iso- lates every distinct source form attested in the doc- ument; the head term itself need not appear. The set of occurrences of one source form is a chain, and a chain of size one is a hapax. Within a chain, occurrences share the source form and differ by their position in the document, hence by their con- text, and by the translation the system produced for each of them. TERMJUDGE returns one verdict per occurrence (Table 1); Appendix A (Tables 11 and 9) illustrates the setting on occurrences from real evaluation runs. 3.2 Evaluation Pipeline Step 1: Reference selection. For each source formsof a concept in a document, TERMJUDGE fixes the expected reference translationR(s): the glossary translation when the source form has one, considered as authoritative, and otherwise the trans- lation that the evaluated system itself produces most often in the document (on lemmas, ties broken 3 https://gitlab.inria.fr/almanach/concordancer 4 gpt-4.1-mini; details in Appendix A. VerdictDescriptionPenalty CONFORMING target matches R(s) (Step 1), or judged error-free0 JUSTIFIEDvariation, consistent with document usage0 UNJUSTIFIEDvariation, inconsistent (incl. neutralisation)0.5 A1attested term of a neighbouring concept1 A2paraphrase instead of term1 A3invented literal calque3 A4left untranslated4 A5do-not-translate item translated4 A6wrong internal constituent4 A7general-language word instead of term3 Cmis-parsed term structure4 Dwrong semantic relations between constituents3 Efailed transposition / syntactic calque1 Fgrammar / phraseology around the term1 Gcontent altered (addition, omission, hallucination)1 Hdefective target-language expression / register1 Table 1: The sixteen verdicts of TERMJUDGE and their penalties. A1âA7 follow level A of the error typology (Cornejo CĂĄrcamo et al., 2026); CâH group its remain- ing families, each scored at its most lenient sub-code (benefit of the doubt). Appendix G explains the weights; Appendix D gives the full typology. by first appearance), a pseudo-reference general- ising those of consistency metrics (Semenov and Bojar, 2022). A pseudo-reference is first validated by the error judge of Step 3 on source-side evi- dence only; 5 if judged erroneous, it is replaced by a reference generated by an LLM from the concept entry and source-side contexts, never from the sys- temsâ outputs. Each reference is recorded with its provenance (glossary, validated or generated). Step 2: Divergence detection. Each translation is compared withR(s)by exact string matching on lemmas, ignoring case, articles, hyphens and dia- critics. A match is labelled CONFORMING without any LLM call: the metric evaluates the choice of term, not its written form. A mismatch is a diver- gence in the sense defined above, and in this case we proceed to Step 3. Step 3: Q1, error judgement.Each divergenceâs correctness is evaluated by an LLM call that re- ceives the aligned pair, the current segment, the conceptâs glossary entry when available, the pre- ceding aligned segments, and the conceptâs prior translations grouped by source form. The ques- tion is absolute: is the target a valid, contextually- appropriate translation of the source term? The candidate error labels come from a simplified ver- sion of a typology developed for the human an- notation of terminology errors in specialised MT (Cornejo CĂĄrcamo et al., 2026), with a severity scale following BĂ©nard et al. (2024): wrong equiv- alent selection uses fine-grained labels (A1âA7), 5 Being borrowed from the very output under evaluation, it cannot be taken at face value; the circularity is made explicit to the judges. the six other error families are collapsed into one code each (CâH, Table 1), and the in-document inconsistency family (B) is excluded, being exactly the question delegated to the consistency judge of Step 4 (details in Appendix D). The judge answers withno_erroror one of the 13 labels, whose grav- ity fixes the penalty. An error label is the occur- renceâs final verdict, whileno_errorroutes the occurrence to Step 4. Hapax forms skip the match- ing shortcut of Step 2 and are always submitted to Q1: when a form occurs once, its pseudo-reference is derived from that very occurrence, so a match would be circular. Since consistency is undefined for a single occurrence, an error-free hapax is di- rectly labelled CONFORMING. Step 4: Q2, consistency judgement. Diver- gences judged error-free by Q1 are passed to the consistency judge, which decides whether the trans- lation is coherent given how the concept is rendered elsewhere in the document. It sees the conceptâs translations grouped by source form, with the dom- inant translation and first mentions marked. It an- swers with one of ten labels, grounded in terminol- ogy and translation studies (Vinay and Darbelnet, 1972; Bowker, 1998; Pecman, 2014) as detailed in Appendix J. Seven name a reason that makes the variation coherent (âfirst mentionâ, âexplicita- tionâ, âavoid repetitionâ, âdocument usageâ, âreg- isterâ, âfacetâ, âsynonym mergeâ) and yield JUS- TIFIED. Three name an inconsistency and yield UNJUSTIFIED: âsynonym inconsistencyâ and âgra- tuitous divergenceâ when one source form receives divergent translations, âneutralisationâ when dis- tinct source forms collapse onto one translation; together they operationalise family B of the typol- ogy, cases combining an incorrect translation with instability being caught upstream by Q1. Neutral- isation, the reproduction of the translation estab- lished for another source form of the same concept, erases a distinction made by the author (Bowker and Hawkins, 2006), for example the case of a doc- ument alternating percutaneous nephrolithotomy and its acronym PNL where both are rendered as nĂ©phrolithotomie percutanĂ©e. Step 5: Propagation and scoring. Q1 is called once for each distinct divergent translation of a source form within a document, and its verdict is propagated to the groupâs other occurrences; keeping the translation unlemmatised in the group- ing key preserves number and gender errors. Q2, whose answer depends on the occurrenceâs posi- tion, is called once per occurrence of a correct form. Together with the matching shortcut of Step 2, this propagation divides the number of LLM calls by 2.55 on IWSLT23 while leaving the system rank- ing nearly unchanged (25 of 28 pairwise order- ings; Appendix I). Each verdict maps to a penalty (Table 1), and the score of a document for a sys- tem is the mean penalty over all its scored occur- rences, including those labelled CONFORMING or JUSTIFIED (penalty 0); lower is better, and scores are negated when correlated with human quality scores. System scores macro-average document scores, occurrences can equally be macro-averaged per source form or per concept, and the meta- evaluation of §4.3 reports all three aggregations. 3.3 Metric Configuration All experiments use a single configuration. The judges run ongpt-4.1-mini, chosen for its low cost (Appendix K) and because it belongs to a dif- ferent family from every evaluated system, which limits the self-preference bias of LLM judges (Zheng et al., 2023; Panickssery et al., 2024); its judgements are validated against human annota- tions in §5. References are selected as in Step 1; the prompts, their context and the decoding settings are detailed in Appendix L. 4 Experimental Setup 4.1 Datasets We use two human-annotated corpora to validate TERMJUDGE at two complementary granularities: STEP (Cornejo CĂĄrcamo et al., 2026), in the geo- science domain, provides per-occurrence gold error labels, enabling us to test whether judges classify individual occurrences as human annotators would. bio-MQM (Zouhar et al., 2024), which provides document-level MQM annotations for ten systems, allows us to test how well aggregated document scores match human quality assessments. Two sci- entific NLP corpora, PARANLP (Peng et al., 2026) and IWSLT23 (Salesky et al., 2023), serve as ad- ditional test beds on which we rank contemporary LLM-based MT systems. Each corpus is paired with a terminology (§3.1). Statistics are in Table 2. STEP(occurrence-levelvalidation).STEP (Cornejo CĂĄrcamo et al., 2026) contains 10 geo- science research articles translated from English into French by a document-level MT system fine-tuned on scientific abstracts of that domain CorpusRoleDomainDocsSegmentsSrc wordsSystemsTerm occ.Glossary bio-MQMvalidation (document)biomedical503847,536101,320729 STEPvalidation (occurrence)geoscience104,432124,28513,37717,035 PARANLPtestNLP (written)347,095137,123811,1221,722 IWSLT23testNLP (spoken)1088415,36081,3561,722 Table 2: Corpus statistics. Term occurrences are aligned term pairs per system for bio-MQM, PARANLP and IWSLT23, and human-annotated occurrences for STEP. All corpora are EnglishâFrench. (Peng et al., 2025), and exhaustively annotated for terminology. The associated resource is a 17,035-entry SKOS terminology compiled for the project from the Loterre thesauri 6 and the ARTES database (Pecman and KĂŒbler, 2011). Each occurrence of a known concept was extracted, aligned with its translation, and labelled with the fine-grained error typology of Cornejo CĂĄrcamo et al. (2026), the same typology our error judge instantiates (§3.2), resulting in 3,377 gold-labelled occurrences (Appendix C). 27.7% of occurrences are found as is in the glossary, and 55.4% are source-side variants of their conceptâs head term. After projecting multi-label annotations onto the metricâs label space, 69.8% of occurrences are error-free, 11.0% carry a family-A error, 16.2% another error (CâH), and 3.0% a sole in-document inconsistency label (B). bio-MQM (document-level validation). bio- MQM (Zouhar et al., 2024) is a multilingual biomedical benchmark annotated at the segment level with MQM error labels by professional trans- lators. We use the EnglishâFrench portion, 50 doc- uments translated by 8 MT systems plus two hu- man references. All MQM categories are retained when computing the human ground truth, because terminology-related problems are annotated under many categories, not only the Terminology label (below). Errors of other kinds, however, only count when they overlap with an aligned term occurrence (§4.3). The biomedical domain moreover exhibits a markedly higher rate of terminology errors than general news (Zouhar et al., 2024). MQM is a general error typology: bio-MQM includes no term inventory, no occurrences and no sourceâtarget term alignments. Turning it into a terminology evaluation benchmark, BioMQM- Terms, is a contribution of this work. We manually extracted the biomedical terms of the 50 source documents by exhaustive reading, obtaining 729 concepts (26% with at least one variant); French labels come from the bilingual MeSH thesaurus 6 https://w.loterre.fr (Lipscomb, 2000) where possible and from manual translation otherwise (construction details in Ap- pendix C). The resulting glossary feeds the pipeline of §3.1: 1,320 aligned occurrences per system, i.e. 13,200 termâtranslation pairs. 21.7% of them are touched by at least one MQM error, of which only a fifth carry the Terminology label, confirming that terminology-related problems spill over into Accu- racy, Linguistic and Style categories. BioMQM- Terms comprises the SKOS glossary and, for each system, the aligned pairs with their variation cate- gories and MQM correspondences. PARANLP and IWSLT23 (test).We re-use the two corpora and MT outputs released by Dahan et al. (2026b), to which we refer for details, and re- run the preprocessing of §3.1 on them. PARANLP (Peng et al., 2026) is a corpus of NLP research papers published in *ACL venues and their compa- rable French versions. IWSLT23 (Salesky et al., 2023) comprises manually revised transcripts and translations of 10 presentations delivered at ACL 2022, from the IWSLT 2023 shared task. Both corpora share the INIST/Loterre NLP glossary of 1,722 ENâFR concepts. 7 MT outputs are generated by four open-weight LLMs, each producing two translations of every document: one from plain translation instructions (baseline) and one with the glossary-preferred terms in the segment and their French translations injected into the prompt (base+terms), yielding eight outputs per corpus. 4.2 Occurrence-Level Validation on STEP For STEP, TERMJUDGE is run on the gold-aligned occurrences, and its verdicts are compared with the human labels occurrence by occurrence. Since the human annotation uses the full 23-code typology while the judge uses its simplified version (§3.2), the gold codes are grouped in the same way, so that both sides share 15 classes: A1âA7, CâH (Ta- ble 1), B (unjustified in-document variation) and no-error. As human annotations are multi-label (25.6% of occurrences carry at least two codes), 7 https://skosmos.loterre.fr/8LP/fr/ they are reduced to one label with a fixed prece- dence (the most severe A code, then the CâH code, then B, pure inconsistency, then no-error). Consis- tency notes on locally correct translations (B1/B2) reduce to no-error at the occurrence level, and 4 oc- currences with missing annotations are excluded, leaving 3,373 scored occurrences. We report per- class precision, recall and F1, macro-F1, accuracy and balanced accuracy, together with a binary error- detection view in which all error types collapse to a single error class. On the metric side, UNJUSTI- FIED maps to B (both denote in-document inconsis- tency) while CONFORMING and JUSTIFIED map to no-error, the conformity/variation distinction being absent from the human annotation. 4.3 Document-Level Meta-Evaluation with BioMQM-Terms Human ground truth per (document, system). Human terminology quality is derived from the MQM annotations at the level of aligned term pairs, not at the level of full segments. For each alignment produced by our pipeline, we aggregate the severity weights of all MQM errors overlapping the target span, using the standard WMT weighting (Critical: 25, Major: 5, Minor: 1, Neutral: 0). When a segment is seen by multiple annotators, the penalty is averaged at the annotator level (annotators who saw the segment but flagged no error contribute 0). The document-level human score is the negated mean alignment penalty (higher is better). Meta-evaluation measures. We evaluate the met- rics against human scores at three levels using themt-metrics-evalpackage. 8 At the system level, Soft Pairwise Accuracy (SPA) (Thompson et al., 2024) checks, for each of the 45 system pairs, whether the metric separates the two systems with the same confidence as the human scores (p-values of paired permutation tests over documents, 1,000 permutations), and averages this agreement over the pairs. At the segment level, the group-by-item pairwise accuracy with tie calibration of Deutsch et al. (2023a), acc â eq , treats each of the 373 source segments carrying at least one aligned pair (out of the corpusâs 384) as an item (a 10-systemĂ373- segment score matrix); ties matter at this level, as for 29% of the segments all ten systems receive the same human score. At the document level, the same measure over the 50 documents (a10Ă 50 matrix) is reported for completeness: human docu- 8 https://github.com/google-research/ mt-metrics-eval ment scores, averaged over about 26 occurrences, are practically never tied, so the measure reduces to plain pairwise accuracy over 50 items. Score aggregation and comparison scope. Be- cause TERMJUDGE assigns a judgement to every occurrence (§3.2), its document score can be aggre- gated in three ways: micro-averaged over occur- rences, and macro-averaged per lemmatised source form or per concept; we report all three. Compari- son metrics score what they are designed for: the glossary-conformity and divergence-based base- lines score the same aligned occurrences as TERM- JUDGE, while the terminology-filtered and general QE baselines score every segment of the document. In a no-hapax variant, only chains with at least two occurrences are evaluated, for both the metrics and the human scores. This leaves 48 documents; the segment-level QE baselines, which do not depend on term occurrences, keep the full 50 documents. The results of this no-hapax variant are given in Appendix H (Tables 18â20). Comparing systems. TERMJUDGE is compared against four metric families, ordered from most to least terminology-aware. (i) Divergence QE: the same pipeline through divergence detection, with typed judges replaced by a quality-estimation model scoring each divergent term pair (the source term and its aligned translation, without the sur- rounding sentence) (CometKiwi (Rei et al., 2022), MetricX-24 (Juraska et al., 2024) and GEMBA- MQM (Kocmi and Federmann, 2023a); sub- scripted div), which isolates the judgesâ contri- bution from that of the divergence architecture. (i) Glossary conformity in the consistency tradi- tion (ibn Alam et al., 2021; Semenov and Bojar, 2022): the head term is matched against its glossary label, and the remaining forms, those without an official reference, against a document-level pseudo- reference (first or majority translation), on lemmas. (i) Terminology-filtered QE, a naive LLM judge: GEMBA-MQM restricted to the errors flagged as terminological (by category, or by any mention of terminology). (iv) General QE: the same back- bones scoring every sentence in the document with no notion of term (subscripted gen), averaged over the document, as a generic baseline. 4.4 System Ranking on the Test Corpora On PARANLP and IWSLT23, TERMJUDGE runs in the configuration of §3.3 with the domain set to NLP, and scores the eight MT outputs distributed with the corpora: four open-weight LLMs (Llama- TERMJUDGE 15-class agreement Macro-F10.196 Accuracy / Balanced accuracy0.644 / 0.230 A-error detection0.590 Exact A-code (among detected)0.521 Binary error detection F1 / Precision / Recall (error class) 0.620 / 0.632 / 0.609 Accuracy / Balanced accuracy0.775 / 0.728 Binary accuracy by hapax status Hapax (n=339)0.664 Non-hapax (n=3,034)0.787 Table 3: Agreement between TERMJUDGE and human gold labels on STEP, all occurrences judged via the exhaustive path (§5.1): 3,373 scored occurrences, in- cluding 1,019 errors (371 with a family-A label) and 2,354 non-errors. 3.1-8B-Instruct (Grattafiori et al., 2024), Qwen3- 8B (Yang et al., 2025), EuroLLM-9B-Instruct (Martins et al., 2025) and EuroLLM-22B-Instruct (Ramos et al., 2026)), each under the two zero-shot conditions of §4.1, baseline and base+terms. De- coding is greedy except for Qwen3-8B, run in its recommended non-thinking sampling mode. 5 Results and Analyses 5.1 Occurrence-Level Agreement on STEP Table 3 reports agreement with the gold labels for 3,373 STEP occurrences under the exhaustive judg- ing path (Appendix I). The deterministic matching shortcut is disabled: every occurrence, including matches and hapaxes, is submitted to Q1 and, if error-free, Q2, allowing every gold class to be pre- dicted. TERMJUDGE detects errors well above the majority-class baseline (balanced accuracy 0.728). Fine-grained code assignment remains hard: 59% of the gold family-A occurrences are flagged with some A code but only about half receive the correct code, with a 15-class macro-F1 close to 0.20. For hapax occurrences (339), binary accuracy drops by about 12 points relative to non-hapax occurrences (0.664 vs. 0.787, Table 3), quantifying the diffi- culty of judging terms without document-internal evidence. Three patterns dominate the confusion matrix (Appendix E, Table 15). First, the missed errors are mostly not terminological errors proper: of the 398 gold errors judged correct, 159 belong to family F (grammar around the term: agree- ment, determiners, prepositions) and 82 to pure in-document inconsistency (B), two families that concern the termâs surroundings or its consistency rather than the choice of equivalent. Among wrong- equivalent errors (family A), the weak point is A3, the invented calque, accepted as correct in 40 of 65 cases (one is shown in Table 5): the judge takes a literal translation for an attested term. Sec- ond, content-altering translations (G: omissions, additions, unintelligible output) are almost always caught (215 out of 227 are tagged erroneous) but mostly with a wrong label (126 as A1): the judge detects that a translation is wrong more reliably than it identifies the mechanism, and this family- level confusion explains the gap between binary balanced accuracy (0.728) and low 15-class macro- F1 (0.196). Third, the gold and judged notions of unjustified variation barely overlap: only 4 of the 101 pure-inconsistency cases are flagged UNJUSTI- FIED, while 104 of the 149 UNJUSTIFIED verdicts were not tagged by the annotators. This reflects a difference of status. In the human guidelines, vari- ation between individually correct translations is recorded (family B) but not counted as an error and carries no severity (Appendix D); TERMJUDGE, by contrast, penalises such variation when the con- sistency judge finds it unjustified (UNJUSTIFIED, Table 1). The human B labels thus describe varia- tion without judging whether it is justified, whereas UNJUSTIFIED asserts that it is not, so the two cap- ture different cases. The LLM judge and the human annotators therefore disagree most on the consis- tency axis, not on error detection. 5.2 Document-Level Meta-Evaluation Table 4 reports the meta-evaluation on BioMQM- Terms under the micro-average over occurrences defined in §3.2.Macro-averaging per source form or concept yields the same conclusions (Ap- pendix F). TERMJUDGE ranks first in SPA (0.844) and in segment-level acc â eq (0.606). On segment- level acc â eq , its lead is significant over every other metric (p †0.01, paired permutation tests). On SPA, it is significant for eight of the ten metrics (p < 0.05,k=2,000), with ties for the Divergence- QE variants GEMBA div and CometKiwi div . The Divergence-QE variants form the second-best fam- ily; the remaining families follow within a narrow band (SPA 0.69â0.78). The document-level acc â eq column, where the measure reduces to plain pairwise accuracy over 50 items (§4.3), is reported for completeness only. Ex- cluding hapax chains removes occurrences lacking another rendering in the document; unless the term MetricSPAacc â eq (seg) acc â eq (doc) Judged divergences (this work) TERMJUDGE0.8440.6060.557 Divergence QE GEMBA div 0.8030.597 â 0.516 MetricX-24 div 0.769 â 0.593 â 0.526 CometKiwi div 0.7860.599 â 0.543 Glossary conformity first-translation fallback0.730 â 0.595 â 0.449 majority fallback0.723 â 0.598 â 0.445 Terminology-filtered QE GEMBA-MQM (any mention) 0.775 â 0.586 â 0.219 GEMBA-MQM (category)0.689 â 0.583 â 0.193 General QE GEMBA gen 0.736 â 0.583 â 0.533 MetricX-24 gen 0.736 â 0.586 â 0.566 CometKiwi gen 0.729 â 0.591 â 0.552 Table 4: Meta-evaluation on BioMQM-Terms, micro- average occurrence aggregation (10 systems, 50 doc- uments, 373 segments; human ground truth from all MQM categories). Empirical SPA chance level:0.592± 0.089. â : TERMJUDGE is significantly better (paired permutation tests,p < 0.05); unmarked values are ties. is in the glossary, only a judgement of the transla- tion itself, by an LLM or a human, can evaluate them. In that reduced variant, no SPA difference between TERMJUDGE and the remaining metrics is statistically significant (pâ„ 0.17), while TERM- JUDGE stays significantly ahead of general QE on segment-level acc â eq (Appendix H). Like any typology-based metric, the document score relies on a penalty scale, which follows the gravities assigned by the typologyâs authors, each grouped family CâH being scored at its most le- nient sub-code (benefit of the doubt, Table 1). Varying C, D and E weights from 1 to 4 with F=G=H=1leaves TERMJUDGE ahead of all comparison metrics under occurrence and source- form aggregation (Appendix G). 5.3 System Ranking on PARANLP and IWSLT23 Table 6 ranks the eight systems of §4.1 on both corpora with TERMJUDGE and with two of the comparison families of §4.3. With TERMJUDGE, for all four models the base+terms run scores bet- ter than its baseline run, on both corpora, with the largest gains on IWSLT23. Where Dahan et al. (2026b) observed a tension they could not arbitrate (glossary injection improves accuracy and consis- tency counts while reducing the transfer of source variation), TERMJUDGE returns a clearer verdict, and its verdict distribution (Table 7) shows why: under base+terms, exact conformity rises by 6 points on PARANLP and 12 on IWSLT23, wrong- equivalent (A) error rates drop sharply (11.7% to 8.4% of occurrences on PARANLP, 18.6% to 9.3% on IWSLT23), while justified and unjusti- fied variation both recede moderately; the variation that glossary injection suppresses is mostly not valid, so base+terms yields a net gain on these corpora. The two comparison metrics behave on these corpora as they did in the meta-evaluation. General QE is blind to the terminological gain: CometKiwi gen orders the systems by general qual- ity, is nearly insensitive to the prompting condition, prefers the baseline run in most pairs of one base- line and one base+terms run (eleven of sixteen on IWSLT23, ten on PARANLP with one tie), and its best run for each corpus is a baseline run. Glos- sary conformity ranks every base+terms run above every baseline run (all sixteen pairs on each cor- pus), as it counts exactly the glossary terms that the base+terms prompt injects, but disagrees with TERMJUDGE within each condition: on IWSLT23 it ranks Qwen3-8B first among the baseline runs, where TERMJUDGE ranks it last. Conformity only checks whether the glossary term appears in the output; judges also recognise justified variation and detect errors that matching approaches cannot. 5.4 What Judging Adds over Counting The comparison metrics as an ablation of TER- MJUDGE. Read from the bottom of Table 4 up- wards, the comparison families of §4.3 add one component of TERMJUDGE at a time, so the table can be viewed as an ablation study. General QE is the baseline (SPA 0.729â0.736), restricting it to terminology-mentioning error spans changes little and can even hurt (0.775 for the permissive filter, 0.689 for the strict one), and counting glossary con- formity instead of judging results in similar scores (0.723â0.730). The first substantial step is archi- tectural: keeping the document-level divergence detection but scoring the divergence points with an off-the-shelf QE metric increases the score to 0.769â0.803; the divergence architecture already carries part of the signal. The second step is the judgement itself: replacing the generic score at those points with the typed Q1/Q2 judges adds the rest (0.844), and it is the only step that also pro- vides interpretable labels. What consistency labels reveal. Beyond the bi- nary JUSTIFIED/UNJUSTIFIED outcome, the labels returned by the Q2 judge also say why a divergence is judged coherent or not (Appendix J, Table 22). Their distribution is stable across systems: about Source formMT outputJudgementVerdictJudgeâs reason (abridged) annotation â annotation humaineexplicitationJUSTIFIEDfirst mention, expands the term with a clarifying adjective datasetjeu de donnĂ©essynonym inconsistency UNJUSTIFIEDdocument established ensemble de donnĂ©es; switch without visible reason attentionauto-attentionneutralisationUNJUSTIFIEDreproduces the translation reserved for another source form of the concept word embeddingincorporation de mots invented calqueA3expected plongement lexical; calque not attested in the domain retrievalrĂ©cupĂ©rationgeneral wordA7expected recherche; general word, not the specialised term rate and state friction â friction Ă taux et Ă©tatfirst mentionJUSTIFIED (gold: A3) the calque is an established French term, attested in the domain Table 5: Five judgements on one system output (EuroLLM-22B-Instruct, baseline, IWSLT23) and, below the rule, a documented judgement error from STEP (human gold: A3). The last column is the explanation the judge returns with each verdict, abridged. â : first mention. baselinebase+terms MetricLl. Qw. E9 E22 Ll. Qw. E9 E22 PARANLP TERMJUDGE â .322 .350 .287 .312 .220 .214 .236 .224 Conformityâ.773 .768 .776 .783 .851 .864 .826 .844 CometKiwi gen â .828 .834 .839 .838 .823 .829 .838 .835 IWSLT23 TERMJUDGE â .468 .502 .437 .439 .236 .245 .330 .230 Conformityâ.713 .723 .719 .731 .848 .850 .788 .833 CometKiwi gen â .807 .815 .823 .823 .803 .808 .821 .818 Table 6: System comparison on the two NLP test cor- pora with TERMJUDGE (mean per-document penalty, lower is better), glossary conformity and CometKiwi general QE (families (i) and (iv) of §4.3); best value per metric and corpus in bold. Ll.: Llama, Qw.: Qwen, E9/E22: EuroLLM-9B/22B. CorpusConditionConf. Just. Unjust.ACâH PARANLP baseline72.211.61.611.72.9 base+terms78.59.21.38.42.6 IWSLT23baseline67.59.22.218.62.5 base+terms79.27.52.19.31.9 Table 7: Distribution of verdicts by prompting condi- tion (percentage of occurrences, four models pooled): conformity, justified and unjustified variation, wrong- equivalent errors (A) and other error families (CâH). four divergences out of five follow the documentâs established usage (e.g., treatment rendered traite- ment throughout while the glossary reference is thĂ©rapeutique: divergent at every occurrence, yet error-free and tagged as âdocument usageâ, JUSTI- FIED); âfirst mentionâ accounts for another 6â7%. The two prompting conditions differ where it matters. Under base+terms, âsynonym inconsis- tencyâ goes down for every model (3.0â3.7% to 2.0â2.9%) while âneutralisationâ goes up slightly (0.8â1.8% to 1.5â2.1%): glossary injection elim- inates unmotivated switching between synonyms, but slightly more often erases a source distinction between a term and one of its variants. This is the loss of source-side variation already observed in §5.3, and it remains far smaller than the reduction of genuine errors (Table 7; the same pattern holds on IWSLT23, where small counts call for caution). Example Judgements.Table 5 displays five real judgements from a single system output. The same situation, a translation that differs from the docu- mentâs dominant one, is judged in four different ways depending on whether it erases a source dis- tinction, switches designation without purpose, ex- pands the term at its first mention, or is simply a wrong term. The judge also returns an expla- nation, which makes each decision interpretable. The last row, from STEP, is a judge error of the kind quantified in §5.1 (40 of 65 invented calques accepted): the judge does not merely accept the invented calque, it asserts that it is attested. 6 Conclusion TERMJUDGE starts from a simple observation: ter- minology metrics that count divergences from a fixed reference conflate translation errors with valid variations that translators routinely produce. It therefore judges instead of counting: matches with the expected translation are settled deterministi- cally, divergences go, in document context, to an error judge and then to a consistency judge, and the interpretable verdicts aggregate into a document- level score. On STEP, error detection agrees with a per-occurrence expert gold (binary balanced ac- curacy 0.728); on BioMQM-Terms, TERMJUDGE ranks first in SPA under all three aggregations and in segment-level acc â eq , significantly ahead of every comparison metric at that level. On two NLP test corpora, it settles a question counting could not: glossary injection helps every model because it re- moves genuine errors, not legitimate variation. The pipeline carries over to other domains given a glos- sary and aligned occurrences, and to other language pairs by adapting its prompts; testing open-weight judges, and the stability of the verdicts across judge models and prompts, are natural next steps. Limitations TERMJUDGE inherits the limitations of LLM-as- a-judge evaluation. Verdicts depend on a single proprietary judge model (gpt-4.1-mini, chosen as explained in §3.3); all prompts are versioned, but the stability of the verdicts across judge models and prompt paraphrases was not measured and no open-weight judge was tested, although the con- clusions hold across judging paths, penalty scales and aggregations (Appendices I, G and F), and exposing the full 23-code typology instead of the compact one moves binary error detection by under three points (Appendix D.1). Pseudo-references are derived from the very output under evaluation; the validation by the error judge and the LLM gen- eration of Step 1 mitigate this circularity but do not eliminate it. Out-of-glossary hapaxes remain the hardest case, with binary agreement about 12 points below non-hapax occurrences on STEP. Ig- noring diacritics and hyphens in Step 2 can also conflate distinct words offered for the same term (Ă©levĂ© vs Ă©lĂšve). The validation supports the metric at the gran- ularities at which it is used, binary error detec- tion and aggregated document scores; per-code agreement is substantially lower (15-class macro- F1 0.196), mostly because the judge detects that a translation is wrong more reliably than it iden- tifies the mechanism (§5.1). Fine-grained codes are therefore reported for diagnosis and should be read as indicative; the document score, for its part, is robust to the weights assigned to the grouped families (Appendix G). Occurrence-level valida- tion relies on consensus labels from two annotators working together and on a single MT system on STEP, so annotator bias cannot be separated from metric error, and document-level validation is lim- ited to one language pair (ENâFR) and ten systems on bio-MQM; the judgesâ accuracy may differ for other language pairs and domains. Finally, TERMJUDGE requires a domain glossary and an upstream pipeline (head term and variant detection, variation labelling, alignment) whose er- rors propagate to the verdicts. This propagation was not measured directly, but the two validations bracket it: STEP evaluates the judges alone on gold occurrences and alignments, whereas BioMQM- Terms is processed end-to-end by the pipeline, and TERMJUDGE still ranks first there. The up- stream steps were checked on samples: the varia- tion labelling in prior work, with an earlier prompt, and term detection and the LLM alignment on PARANLP only, where excluding the least reliable detections and alignments leaves the rankings un- changed (Appendices A and B). Document scores are also relative to the terminology provided, so comparisons are meaningful across systems eval- uated on the same corpus and terminology, not across corpora. Evaluating a full corpus incurs a non-trivial LLM cost (Appendix K). Acknowledgments This work was supported by the French National Agency (ANR) as part of the MaTOS project under reference ANR-22-CE23-0033. 9 The authors are also grateful to the anonymous reviewers for their insightful comments and suggestions, and to Na- talie KĂŒbler, Alexandra Mestivier and JosĂ© Cornejo CĂĄrcamo for early discussions on term variation and evaluation of term translation, to Ăric de la Clergerie for his help on the CONCORDANCER and to the MaTOS project team for their feedback on preliminary versions of this work. References Md Mahfuz Ibn Alam, Ivana KvapilĂkovĂĄ, Antonios Anastasopoulos, Laurent Besacier, Georgiana Dinu, Marcello Federico, Matthias GallĂ©, Kweonwoo Jung, Philipp Koehn, and Vassilina Nikoulina. 2021. Find- ings of the WMT shared task on machine translation using terminologies. In Proceedings of the Sixth Conference on Machine Translation, pages 652â663, Online. Association for Computational Linguistics. Shoshana Blum-Kulka. 1986. Shifts of cohesion and coherence in translation.In Juliane House and Shoshana Blum-Kulka, editors, Interlingual and In- tercultural Communication: Discourse and Cogni- tion in Translation and Second Language Acquisition Studies, number 272 in TĂŒbinger BeitrĂ€ge zur Lin- guistik, pages 17â35. Gunter Narr Verlag, TĂŒbingen. Lynne Bowker. 1998. Variant terminology: Frivolity or necessity? In Proceedings of the 8th EURALEX International Congress, pages 487â495, LiĂšge, Bel- gium. University of LiĂšge. Lynne Bowker and Shane Hawkins. 2006. Variation in the organization of medical terms: Exploring some motivations for term choice. Terminology, 12(1):79â 110. Maud BĂ©nard, Natalie KĂŒbler, Alexandra Mestivier, Joachim Minder, and Lichao Zhu. 2024. Ătude des protocoles dâĂvaluation humaine pour la traduction de documents. Technical report, Projet ANR MaTOS, livrable D4-1.1. 9 http://anr-matos.github.io/ Andrew Chesterman. 2016. Memes of Translation: The Spread of Ideas in Translation Theory, revised edi- tion, volume 123 of Benjamins Translation Library. John Benjamins, Amsterdam. First edition 1997 (BTL 22). JosĂ© Cornejo CĂĄrcamo, Natalie KĂŒbler, and Alexandra Mestivier. 2026. Qualitative analysis of machine- translated terms and their intratextual variants: how does the machine handle variation?In Rencon- tres Doctorales Monique MĂ©met â 47e colloque du GERAS, Dijon, France. Groupe dâEtude et de Recherche en Anglais de SpĂ©cialitĂ© (GERAS). JosĂ© Cornejo CĂĄrcamo, Natalie KĂŒbler, Alexandra Mestivier, and Lichao Zhu. 2025.Proposition dâune mĂ©thodologie pour lâannotation de la variation dĂ©nominative dans les textes scientifiques en sciences de la Terre. In Beyond Single Words, Paris, France. Lichao Zhu and Natalie KĂŒbler and Mojca Pecman. Oliver Ë Culo and Jean Nitzke. 2016. Patterns of termino- logical variation in post-editing and of cognate use in machine translation in contrast to human transla- tion. In Proceedings of the 19th Annual Conference of the European Association for Machine Translation, pages 106â114. Nicolas Dahan, Rachel Bawden, and François Yvon. 2026a. MetaDocEval: A Contrastive Framework for Evaluating Machine Translation Metrics at the Document-Level. In EAMT 2026 - 26th Annual Con- ference of the European Association for Machine Translation, volume Proceedings of the 26th Annual Conference of the European Association for Machine Translation, pages 287â322, Tilburg, Netherlands. Nicolas Dahan, Ziqian Peng, François Yvon, and Rachel Bawden. 2026b. Improving term evaluation in ma- chine translation: Variation matters. In Proceedings of the 17th Conference of the Association for Ma- chine Translation in the Americas (Volume 1: Re- search Track), pages 101â134, QuĂ©bec City, Canada. Association for Machine Translation in the Americas. BĂ©atrice Daille. 2017. Term variation in specialised corpora: characterisation, automatic discovery and applications., volume 19 of Terminology and Lexi- cography Research and Practice series. John Ben- jamins Publishing Company. Daniel Deutsch, George Foster, and Markus Freitag. 2023a. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Pro- ceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 12914â 12929, Singapore. Association for Computational Linguistics. Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag. 2023b. Training and meta- evaluating machine translation evaluation metrics at the paragraph level. In Proceedings of the Eighth Conference on Machine Translation, pages 996â 1013, Singapore. Association for Computational Lin- guistics. M. Amin Farajian, Nicola Bertoldi, Matteo Negri, Marco Turchi, and Marcello Federico. 2018. Evalua- tion of terminology translation in instance-based neu- ral MT adaptation. In Proceedings of the 21st Annual Conference of the European Association for Machine Translation, pages 169â178, Alicante, Spain. Patrick Fernandes, Daniel Deutsch, Mara Finkel- stein, Parker Riley, AndrĂ© Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leverag- ing large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pages 1066â 1083, Singapore. Association for Computational Lin- guistics. Sabela FernĂĄndez-Silva and Koen Kerremans. 2011. Terminological variation in source texts and trans- lations: A pilot study. Meta, 56(2):318â335. Angelina GaĆĄpar, Sanja Seljan, and Vlasta Ku Ë ciĆĄ. 2022. Measuring Terminology Consistency in Trans- lated Corpora: Implementation of the Herfindahl- Hirshman Index. Information, 13(2). Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schel- ten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mi- tra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Ro- driguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, and Bob- bie Chern et al. 2024.The llama 3 herd of models.Preprint arXiv:2407.21783.Preprint, arXiv:2407.21783. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and AndrĂ© F. T. Mar- tins. 2024. xCOMET: Transparent machine trans- lation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979â995. Rejwanul Haque, Mohammed Hasanuzzaman, and Andy Way. 2023. Evaluating Terminology Trans- lation in MT. In Alexander Gelbukh, editor, Com- putational Linguistics and Intelligent Text Process- ing, volume 13451, pages 495â520. Springer Nature Switzerland, Cham. Series Title: Lecture Notes in Computer Science. Md Mahfuz ibn Alam, Antonios Anastasopoulos, Lau- rent Besacier, James Cross, Matthias GallĂ©, Philipp Koehn, and Vassilina Nikoulina. 2021. On the evalu- ation of machine translation for terminology consis- tency. Preprint arXiv:2106.11891. ISO. 2019. ISO 1087:2019: Terminology work and terminology science. Vocabulary. International Orga- nization for Standardization, Clause 3.4. Masaki Itagaki, Takako Aikawa, and Xiaodong He. 2007. Automatic validation of terminology trans- lation consistenscy with statistical method. In Pro- ceedings of Machine Translation Summit XI: Papers, Copenhagen, Denmark. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492â504, Miami, Florida, USA. Association for Computational Linguistics. Tom Kocmi and Christian Federmann. 2023a. GEMBA- MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768â775, Singapore. Association for Computational Linguistics. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th An- nual Conference of the European Association for Ma- chine Translation, pages 193â203, Tampere, Finland. European Association for Machine Translation. Natalie KĂŒbler. 2008. A comparable learner transla- tor corpus: Creation and use. In Proceedings of the Workshop on Building and Using Comparable Corpora Workshop (BUCC) at LREC, pages 73â79, Marrakech, Morocco. Natalie KĂŒbler, Alexandra Mestivier, and Mojca Pec- man. 2022. Using comparable corpora for translating and post-editing complex noun phrases in specialised texts: Insights from english-to-french specialised translation. In Sylviane Granger and Marie-Aude Lefer, editors, Extending the Scope of Corpus-Based Translation Studies, Bloomsbury Advances in Trans- lation, pages 237â266. Bloomsbury Publishing. Carolyn E. Lipscomb. 2000. Medical subject headings (MeSH). Bulletin of the Medical Library Association, 88(3):265â266. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK. Aslib. Xinglin Lyu, Junhui Li, Zhengxian Gong, and Min Zhang. 2021. Encouraging lexical translation consis- tency for document-level neural machine translation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 3265â3277, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics. Pedro Henrique Martins, JoĂŁo Alves, Patrick Fernan- des, Nuno M. Guerreiro, Ricardo Rei, Amin Fara- jian, Mateusz Klimaszewski, Duarte M. Alves, JosĂ© Pombal, Nicolas Boizard, Manuel Faysse, Pierre Colombo, François Yvon, Barry Haddow, JosĂ© G. C. de Souza, Alexandra Birch, and AndrĂ© F. T. Martins. 2025. EuroLLM-9B: Technical Report. Preprint arXiv:2506.04079. Preprint, arXiv:2506.04079. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the eval- uation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4984â4997, Online. Association for Computa- tional Linguistics. Aristides Milios, Siva Reddy, and Dzmitry Bahdanau. 2023. In-context learning for text classification with many labels. In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP. Association for Computational Linguistics. Joachim Minder, Guillaume Wisniewski, and Natalie KĂŒbler. 2025. Testing LLMsâ capabilities in annotat- ing translations based on an error typology designed for LSP translation: First experiments with ChatGPT. In Proceedings of Machine Translation Summit X: Volume 1, pages 190â203, Geneva, Switzerland. Eu- ropean Association for Machine Translation. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076. Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311â318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. Mojca Pecman. 2014. Variation as a cognitive device: how scientists construct knowledge through term for- mation. Terminology. International Journal of Theo- retical and Applied Issues in Specialized Communi- cation , 20(1):p.1â24. Mojca Pecman and Natalie KĂŒbler. 2011. ARTES: an online lexical database for research and teaching in specialized translation and communication. Proceed- ings from International Workshop on Lexical Re- sources (WoLeR) 2011 at ESSLLI. August 1-5, 2011. Ziqian Peng, Rachel Bawden, and François Yvon. 2025.Investigating length issues in document- level machine translation. In Proceedings of Ma- chine Translation Summit X: Volume 1, pages 4â23, Geneva, Switzerland. European Association for Ma- chine Translation. Ziqian Peng, Lichao Zhu, Rachel Bawden, Maud BĂ©- nard, Ăric de la Clergerie, Mathilde Huguin, Na- talie KĂŒbler, Paul Lerner, Alexandra Mestivier, and François Yvon. 2026. Parallel Corpora of Scholarly Documents for English-French Machine Translation. In Proceedings of the 19th Workshop on Building and Using Comparable Corpora (BUCC), Palma de Mallorca, Spain. LREC2026. Maja Popovi Ì c. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392â395, Lisbon, Portugal. Association for Computational Linguistics. Miguel Moura Ramos, Duarte M. Alves, Hippolyte Gisserot-Boukhlef, JoĂŁo Alves, Pedro Henrique Mar- tins, Patrick Fernandes, JosĂ© Pombal, Nuno M. Guer- reiro, Ricardo Rei, Nicolas Boizard, Amin Farajian, Mateusz Klimaszewski, JosĂ© G. C. de Souza, Barry Haddow, François Yvon, Pierre Colombo, Alexan- dra Birch, and AndrĂ© F. T. Martins. 2026. Eurollm- 22b: Technical report. Preprint arXiv:2602.05879. Preprint, arXiv:2602.05879. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685â2702, Online. Association for Computational Linguistics. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, JosĂ© G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and AndrĂ© F. T. Martins. 2022. CometKiwi: IST-unbabel 2022 sub- mission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634â645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics. Elizabeth Salesky, Kareem Darwish, Mohamed Al- Badrashiny, Mona Diab, and Jan Niehues. 2023. Evaluating multilingual speech translation under re- alistic conditions with resegmentation and terminol- ogy. In Proceedings of the 20th International Confer- ence on Spoken Language Translation (IWSLT 2023), pages 62â78, Toronto, Canada (in-person and online). Association for Computational Linguistics. Kirill Semenov and Ond Ë rej Bojar. 2022. Automated evaluation metric for terminology consistency in MT. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 450â457, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. Kirill Semenov, Xu Huang, VilĂ©m Zouhar, Nathaniel Berger, Dawei Zhu, Arturo Oncevay, and Pinzhen Chen. 2025. Findings of the WMT25 terminology translation task: Terminology is useful especially for good MTs. In Proceedings of the Tenth Conference on Machine Translation, pages 554â576, Suzhou, China. Association for Computational Linguistics. Kirill Semenov, VilĂ©m Zouhar, Tom Kocmi, Dongdong Zhang, Wangchunshu Zhou, and Yuchen Eleanor Jiang. 2023. Findings of the WMT 2023 shared task on machine translation with terminologies. In Proceedings of the Eighth Conference on Machine Translation, pages 663â671, Singapore. Association for Computational Linguistics. Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. 2024. Improving statistical sig- nificance in human evaluation of automatic metrics via soft pairwise accuracy. In Proceedings of the Ninth Conference on Machine Translation, pages 1222â1234, Miami, Florida, USA. Association for Computational Linguistics. Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 118â128, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics. Jean-Paul Vinay and Jean Darbelnet. 1972. Stylistique comparĂ©e du français et de lâanglais, nouvelle Ă©di- tion revue et corrigĂ©e edition. Didier, Paris. PremiĂšre Ă©dition 1958. Yutong Wang, Jiali Zeng, Xuebo Liu, Derek F. Wong, Fandong Meng, Jie Zhou, and Min Zhang. 2025. DelTA: An online document-level translation agent based on multi-level memory. In The Thirteenth In- ternational Conference on Learning Representations. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayi- heng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, and Jiaxi Yang et al. 2025. Qwen3 technical report. Preprint arXiv:2505.09388. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685. VilĂ©m Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson. 2024. Fine-tuned machine translation metrics struggle in unseen domains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 2: Short Papers), pages 488â500, Bangkok, Thailand. Association for Computational Linguistics. A Pipeline Overview and a Worked Example Figure 2 situates TERMJUDGE within the full eval- uation pipeline: the upstream stages produce the aligned term occurrences defined in §3.1, and the metric proper begins at reference selection. These stages are the preprocessing pipeline of Dahan et al. (2026b), which we run ourselves on BioMQM- Terms, PARANLP and IWSLT23; STEP comes with gold occurrences and alignments. In the three corpora, the MT outputs follow the source segmentation, so that source and target segments are aligned by construction. Variation labelling uses a revised version of that studyâs prompt (Ap- pendix L.6), run withgpt-4.1. The alignment step also differs: every occurrence is aligned by gpt-4.1-mini(few-shot prompt in Appendix L.5, temperature 0, at most 20 output tokens), which is instructed to copy the translation of the term verbatim from the target segment, whereas the orig- inal pipeline calls the LLM only as a fallback for low-confidence spans of an embedding-based word aligner. We take this prompt from the organis- ers of the WMT25 terminology task (Semenov et al., 2025), with six in-context examples instead of their twenty. Applying the same aligner to ev- ery occurrence also keeps the extracted span from depending on which component handled the oc- currence, which would otherwise be a source of spurious divergences in Step 2. Aligned spans and glossary terms are lemmatised with spaCy (fr_core_news_lg) for the matching of Step 2. Table 8 illustrates the variation categories with which each attested source form is labelled relative to the head term of its concept. The label depends on how the form was detected: variants listed in the glossary, and the static variants that the CON- CORDANCER generates from glossary entries, are classified by few-shot promptinggpt-4.1; head- term forms are no variation by definition, and the forms that the CONCORDANCER detects in context mostly take the category of the formation pattern that produced them; a form derived from a glossary variant combines both (an acronym of an expansion is combined). Dahan et al. (2026b) validated an ear- lier version of this labelling (its own prompt, with gpt-4.1-mini) on 50 sampled forms annotated by a computational linguist: 84% agreement on the category, near-perfect on single-category variants, with combined variants under-detected (recall 11%) but accounting for under 2% of occurrences. Term detection and the LLM alignment were validated manually on PARANLP (Appendix B): detection precision is 91.5% and recall 95.5%, and the alignment is correct for 78 of 80 randomly sam- pled occurrences, equally in the two prompting conditions. Table 11 follows one concept of an PARANLP document from its glossary entry to the verdicts; Table 9 then illustrates the consistency judge on real occurrences of a STEP validation document. CategoryHead termAttested source form no variationmachine translation systemmachine translation system graphicalmachine translation systemMT system morphosyntacticmachine translation systemsystem for machine translation reductionmachine translation systemtranslation system expansionmachine translation systemautomatic machine translation system lexicalmachine translation systemautomatic translation system combinedmachine translation systemMT model Table 8: Source-side variation categories (Cornejo CĂĄr- camo et al., 2025), illustrated on the running example of the introduction: each attested form is labelled relative to the head term of its concept. Seg Source formMT outputJudgementVerdict slab (in glossary; variant downgoing slabâ plaque plongeante) 6 slab (1st mention)plaque plongeantefirst mentionJUSTIFIED 74 slabplaque plongeantedocument usageJUSTIFIED 495 slabplaque plongeanteneutralisationUNJUSTIFIED subduction initiation (out of glossary; expected translation initiation de la subduction) 598 subduction initiation dĂ©but de la subduction paraphraseA2 Table 9: Real occurrences from a STEP validation docu- ment. The source alternates the head term slab and its variant downgoing slab; the system renders the variant consistently as plaque plongeante. Rendering the head term the same way is judged legitimate at âfirst mentionâ and while it follows the âdocument usageâ (JUSTIFIED), but flagged once plaque plongeante is the form reserved for the variant: the source distinction is erased (âneutrali- sationâ, UNJUSTIFIED). For the out-of-glossary concept subduction initiation, whose reference is established from document usage, the paraphrase dĂ©but de la sub- duction is a terminological error (A2). BManual Validation of the Preprocessing Term detection by the CONCORDANCER and the LLM alignment were validated manually on PARANLP, the variation labelling having been validated in prior work (Appendix A). All items were sampled, with a fixed random seed, from the aligned occurrences that TERMJUDGE scored in §5.3: 11,122 occurrences in 34 documents, the same for the eight outputs since detection operates on the source text. One of the authors, a native French speaker fluent in English, answered one question per item (yes, no or uncertain) without knowing which system or prompting condition had produced the translation; uncertain and unanswered items are left out. The questions concern the pre- processing only, not the quality of the translation. For the precision of term detection, 100 detected occurrences were shown in their source segment, with the glossary entry of the concept they were linked to, and the annotator judged whether the detected form refers to this concept, in its technical sense and with the right boundaries. Non-glossary variants were over-represented in this sample (half of the items, against a fifth of the occurrences), P R E P R O C E S S I N G : D O C U M E N T- L E V E L T E R M A N N O TAT I O N P I P E L I N E T E R M J U D G E (this work) glossary source documents + MT outputs term & variant spotting variation labelling term alignment + normalisation reference selection R(s) divergence detection LLM error judge Q1 LLM consistency judge Q2 penaltiesâ document score aligned occurrences (sâ t) CONFORMING A1..A7, C..H JUSTIFIED UNJUSTIFIED match divergence or hapax error no error no error (hapax) yes no Figure 2: Overview of the evaluation pipeline (every data-side notion is defined in §3.1). Upstream: the CONCOR- DANCER spots the occurrences of glossary concepts and of their variants in the source documents; each source form is labelled with a variation category; each occurrence is aligned with its translation, normalised into a canonical form and a lemma. TERMJUDGE (this work): a referenceR(s)is fixed per source form and document; a translation matchingR(s)is CONFORMING without any LLM call; a divergent translation is classified by the LLM error judge (Q1), which either assigns an error label (A1âA7, CâH) or passes it to the LLM consistency judge (Q2), which returns JUSTIFIED or UNJUSTIFIED; hapax occurrences are always submitted to Q1 and, if error-free, labelled CONFORMING without the consistency stage; individual penalties aggregate into a document-level score. because they are rarer and, being derived beyond the glossary, more likely to be wrong; each answer is therefore weighted by the share of its detection type in the corpus. For the recall of term detection, the annotator listed, in 40 source segments of at least six words (one per document, plus six), the occurrences of glossary concepts that no detection covered. For the alignment, the annotator judged whether the proposed span is exactly the transla- tion that the system wrote for the term, ignoring case, punctuation and a leading article; a wrong but correctly delimited translation counts as a cor- rect alignment, and so does an empty span for an untranslated term. Thirteen aligned occurrences were drawn per output: ten at random among the occurrences whose span appears in the target seg- ment, and three among those whose span is empty or absent from it (0.5 to 0.8% of the occurrences of each output), where alignment errors are most likely. Table 10 reports the results. Detection errors concentrate in non-glossary variants: seven of the fourteen come from the glossary variant base (of morphological base) being matched in based and in compounds such as rule-based, which accounts for 262 occurrences (2.4% of the corpus), and most of the others are ordinary words linked to a con- cept, such as Table in the caption âTable 4â (linked to confusion matrix) or the verb split in âwe will try to split those linesâ. The three occurrences missed by the CONCORDANCER are metric, pat- tern and DBN5; four other missed domain terms (e.g., decoder) are absent from the glossary and are not counted. The two alignment errors on ran- ItemsCorrectRate Term detection precision968291.5% â recall666395.5% Alignment baseline403997.5% base+terms403997.5% span empty or absent221881.8% Table 10: Manual validation of the preprocessing on PARANLP. For detection precision and alignment, Items counts the items answered yes or no, and Cor- rect those answered yes. For recall, Items counts the occurrences of glossary concepts in the 38 annotated segments, and Correct those that the CONCORDANCER detected. â Weighted by the share of each detection type in the corpus, since non-glossary variants were over-represented in the sample; unweighted, 82 of 96 (85.4%). The baseline and base+terms rows sample, at random, the occurrences whose span appears in the target segment; the last row samples the others, whose span is empty or absent from the target segment (0.5 to 0.8% of the occurrences of each output). dom occurrences extend the span to a neighbouring name (corpus aligned to corpus Brown). Among the spans empty or absent from the target segment, three of the four errors are words that the system did not translate, for which the aligner returned a translation anyway (instance in for instance), and the fourth is a defective copy of the translation (analogiede for analogie). The samples locate only some of the errors, so they cannot be corrected one by one in the corpus. We therefore recomputed the PARANLP scores of §5.3 from the verdicts of the published run, leaving out first the non-glossary variants (21.3% of the occurrences, including the based matches), then the occurrences whose span is empty or absent from the target segment. In both cases, the ranking of the four models within each prompting condition is unchanged, and base+terms still scores better than baseline for all four models. C Dataset Construction Details The manual extraction behind the BioMQM-Terms glossary (§4.1), performed independently of the termsâ involvement in any translation error, pro- duced 819 raw entries, deduplicated into 729 unique concepts. Crossing them with the bilingual MeSH thesaurus (30,915 concepts) covers 15.2% of the terms by exact matching, and 31.7% once alternative labels, inflected forms and graphical normalisation are admitted; the remaining 68.3%, too specific or too compositional for MeSH, were translated manually. A complementary error-driven pass over the MQM Terminology error spans ex- amined 132 candidate terms, of which 110 were retained. On STEP, the 3,377 gold-labelled occur- rences cover 261 concepts and 793 distinct source forms. D A Typology of Terminological Errors The error typology instantiated by the Q1 judge was developed for the human annotation of termi- nology errors in specialised machine translation (Cornejo CĂĄrcamo et al., 2026). It builds on the MeLLANGE error typology (KĂŒbler, 2008) and on subsequent work on the translation of complex noun phrases in specialised texts (KĂŒbler et al., 2022), and its severity scale follows BĂ©nard et al. (2024). The typology is organised on three levels: a macro-category, a family (A to J), and a fine- grained code. Seven families describe errors in the translation of the term itself, independently of its other occurrences in the document, and yield the 23 fine-grained error codes of Table 12: in- correct equivalent selection (A1âA7), constituent structure of complex terms (C1âC3), semantic rela- tions between constituents (D1âD2), transposition (E1âE2), context-bound grammar and phraseology (F1âF4), content alteration (G1âG3), and target- language expression (H1âH2). Each fine-grained code carries a severity on the scale of BĂ©nard et al. (2024) (neutral, minor, major, critical; only 1â3 are attested in the typology table). Three fami- lies carry no severity: family B (B1âB7) describes in-document terminological (in)consistency: B1â B4 cover consistency facts about translations that are individually correct (the annotation guidelines stress that these cases « ne reprĂ©sentent pas vrai- ment une erreur » (do not really correspond to er- rors )), B5âB7 combine incorrectness with instabil- ity, and its labels are never assigned alone, since consistency is only assessed at the document level; family I marks the absence of error; family J is an annotator-workflow tag. TERMJUDGE applies a fixed transformation to this typology before exposing it to the Q1 judge (Table 13). Family A is kept fine-grained; each of the families C to H is collapsed into a sin- gle grouped code carrying the gravity of its worst member; family B is excluded from Q1, since in- document inconsistency is exactly the question del- egated to Q2 and keeping it in Q1 would count the same phenomenon twice; family I becomes the no_erroranswer; family J is dropped as meaning- less for an LLM judge. The 13 remaining codes are exposed to the judge as natural-language labels rather than opaque codes, which reduces reporting errors (mistypingA6forA7has no surface cue; wrong_componentvsgeneral_word_not_term does); the answer is remapped to the canonical code in post-processing, and any label outside the reg- istry falls back tono_error. The natural-language definitions and examples exposed to the judge for each of the 13 labels are those of the Q1 prompt, reproduced verbatim in Appendix L.1. The same transformation is applied to the human annotations of STEP when validating the metric (§4.2), so that metric and human judgements live in the same 15- class space (the 13 codes, plus B reached through Q2, plus no-error). Glossary entry: semantic similarityâ similaritĂ© sĂ©mantique; English variant semantic relatedness; no French variant Source form (CONCORDANCER) Variation category In glossaryOcc. Reference R(s) (Step 1) How the reference was fixed semantic similarityno variationhead term2similaritĂ© sĂ©mantiqueglossary translation, taken as authoritative similarity (incl. plural) reductionno14 similaritĂ©systemâs most frequent translation, validated by the error judge semantic relatednesslexicalEnglish variant only 1 parentĂ© sĂ©mantiquesystemâs only translation, validated by the error judge SegSource formâ MT output Step 2: match? Step 3: Q1Step 4: Q2Verdict 30, 42semantic similarityâ similaritĂ© sĂ©mantique yesnot callednot calledCONFORMING (Ă2) 33, 46, 47, 85, 87, 90 â , 91, 93, 98, 103 (Ă2), 110 similarityâ similaritĂ©(s)yesnot callednot called CONFORMING (Ă12) 79semantic relatednessâ parentĂ© sĂ©mantique skipped (hapax) no error, judged in absolute terms not called (hapax)CONFORMING 90, 92similarityâ similitudesno: divergence no error (one call, propagated) gratuitous divergence UNJUSTIFIED (Ă2) Seg Source (detected span in bold)MT output (aligned span in bold) 30 . . . strong link with the notion of semantic similarity.. . . lien Ă©troit avec la notion de similaritĂ© sĂ©mantique. 33. . . of a word according to the similarity with its semantic neighbors. . . . dâun mot en fonction de la similaritĂ© avec ses voisins sĂ©mantiques. 42 . . . based on the symmetry of semantic similarity relations.. . . sur la symĂ©trie des relations de similaritĂ© sĂ©mantique. 46 . . . aim to calculate similarities between textual representa- tions of word contexts. . . . visent Ă calculer les similaritĂ©s entre les reprĂ©sentations textuelles des contextes . . . 47Methods to calculate similarities from IR seem then relevant . . . Les mĂ©thodes pour calculer les similaritĂ©s Ă partir de lâIR semblent alors . . . 79 . . . our methods on semantic simlarity versus semantic re- latedness relations. . . . nos mĂ©thodes sur les relations de similaritĂ© sĂ©mantique par rapport Ă celles de parentĂ© sĂ©mantique. 85 . . . list of names ordered by decreasing similarity.. . . de noms classĂ©s par ordre de similaritĂ© dĂ©croissante. 87 . . . performance of different models of IR similarities.. . . les performances des diffĂ©rents modĂšles de similaritĂ© IR. 90 â . . . some IR similarities are quite inefficient including the TF alone or Hellinger similarity. . . . certaines similitudes IR sont assez inefficaces, notam- ment la similaritĂ© TF seule ou la similaritĂ© de Hellinger. 91 . . . since these similarities use very basic weights . . .. . . puisque ces similaritĂ©s utilisent des poids trĂšs basiques . . . 92 The similarities that include a notion of IDF . . .Les similitudes qui incluent une notion dâIDF . . . 93 Okapi BM25-based similarities offer good results.Les similaritĂ©s basĂ©es sur lâalgorithme BM25 dâOkapi don- nent . . . 98Computing all the similarities between all pairs of words . . . Le calcul de toutes les similaritĂ©s entre toutes les paires de mots . . . 103The similarity between a wordw i . . . the same value as the similarity between the query w j . . . La similaritĂ© entre un motw i . . . la mĂȘme valeur que la similaritĂ© entre la requĂȘte w j . . . 110 . . . for giving a new similarity score in a simple way. . . . pour donner un nouveau score de similaritĂ© de maniĂšre simple. Table 11: Worked example on one concept of an PARANLP document translated by EuroLLM-22B-Instruct (baseline). Top: the glossary entry, the forms detected by the CONCORDANCER with their variation category, and the reference translation fixed for each form in Step 1 of §3.2; the glossary provides a French translation for the head term only, so the two other forms receive a pseudo-reference, the systemâs own translation validated by the error judge. Middle: the path of each occurrence through Steps 2 to 4 and the resulting verdict. Bottom: every segment involved. A match with the reference is labelled CONFORMING without any LLM call. The hapax semantic relatedness skips the match and is judged in absolute terms by the error judge, which accepts parentĂ© sĂ©mantique. The two occurrences rendered similitudes diverge from the reference: the error judge finds no error (one call, its answer propagated to the identical translation), and the consistency judge classifies the switch as a âgratuitous divergenceâ from the documentâs usage (UNJUSTIFIED); segment 90 even keeps similaritĂ© for Hellinger similarity in the same sentence. â The CONCORDANCER detected only similarity in Hellinger similarity; the whole expression should have been identified as an expansion of the term and aligned to similaritĂ© de Hellinger. In segment 79, the misspelt semantic simlarity was not detected. Code DefinitionExample (sourceâ *MTâ expected)Penalty Support A. Wrong equivalent selection A1The translation is an attested term of the domain, but the equivalent of another term or concept: an inexact neighbour of the expected equivalent, which can also create spurious repetitions when the two concepts are close. tremorâ*tremblement de terreâtrĂ©mor1111 A2 The terminological unit is replaced by a descrip- tive, general-language paraphrase, losing the precision of the established term. lithospheric mantleâ*couche profonde sous la croĂ»te terrestreâmanteau lithos- phĂ©rique 120 A3The system produces a formulation that is nei- ther a recognised term of the domain nor a general-language expression: a literal creation. earthquake swarmâ*sillage sismiqueâ essaim sismique 365 A4The term is left in the source language although an established target-language equivalent exists. megathrustsâ*megathrustsâmĂ©- gachevauchements 434 A5A unit conventionally kept in the source lan- guage (proper name, acronym, nomenclature symbol) is translated; for acronyms, the letters may also be reworked or reordered. MORBâ *BDRMâ MORB423 A6Some constituents of a complex term are trans- lated correctly but others are not, yielding a partially correct hybrid term. fault patchesâ*taches de failleâzones de faille 495 A7A single general-language word replaces the specialised term; unlike A2, one lexical unit rather than a paraphrase, creating referential ambiguity in specialised discourse. repeatersâ*rĂ©pĂ©titeursâsĂ©ismes rĂ©pĂ©titifs 323 B. In-document terminological (in)consistency (no severity; delegated to the consistency judge) B1The same base term receives different, individu- ally correct translations across the document. earthquake ruptureârupture sismique / ruptures de sĂ©ismes â B2The same variant receives different, individually correct translations across the document. fault creepâfluage asismique / fluage de la faille / fluage de faille â B3 A base term and one of its variants receive the same translation, erasing the distinction be- tween them. seismic hazard (variant) / earthquake haz- ard (base term)â risque sismique â B4Different variants of the same base term receive the same translation, erasing their nuances. dynamic fault slip / dynamic slipâglisse- ment dynamique â B5 The base term is translated both incorrectly and inconsistently from one occurrence to the next. continental forearcâ*marge continen- tale / *zone de subduction / *zone de lâavant-arc continental â B6A variant is translated both incorrectly and in- consistently from one occurrence to the next. down-dipâ*encaissĂ©e / *en profondeur / *dans la direction aval â B7The base term and its variant both receive the same incorrect translation, combining neutrali- sation with incorrectness. Zapotal Formation / Zapotal Fmâ*For- mation de Zapotal â C. Structure of complex terms C1 The head noun of the complex term or noun phrase is misidentified; the head fixing the base category of the concept, the designated concept changes. subduction-zone serpentinitesâ*zones de subduction constituĂ©es de serpentinite â serpentinites en zone de subduction 410 C2An adjective or other modifier is attached to the wrong constituent, altering the qualification relations within the term or phrase. fast earthquake ruptureâ*rupture rapide de sĂ©ismeâ rupture sismique rapide 411 C3 The constituents are ordered against the conven- tions of the target language, yielding a structure alien to domain usage. slow earthquakeâ*lent sĂ©ismeâsĂ©isme lent 49 D. Semantic relations D1 The semantic relations between constituents, often implicit in scientific English, are rendered incorrectly where the target language requires them to be explicit (preposition or paraphrase). fast and slow eventsâ*Ă©vĂ©nements Ă la fois rapides et lentsâĂ©vĂ©nements rapides et lents 43 D2An element shared (factorised) across coor- dinated constituents in the source is not dis- tributed to all conjuncts in the target. pressure-solution and dislocation creep â*pression-solution et le glissement par dislocationâfluage par dissolution- prĂ©cipitation et par dislocation 40 E. Transposition Table 12 (continued) Code DefinitionExample (sourceâ *MTâ expected)Penalty Support E1The compact, synthetic source structure is kept as is, without the explicitation (preposition, arti- cle, unfolding) that the analytic target structure requires. strike-slip and subduction thrust faultsâ *failles de dĂ©crochement et de chevauche- ment de subductionâfailles de dĂ©croche- ment et de chevauchement dans les zones de subduction 313 E2 The source syntax is calqued although the words themselves are attested, producing unnatural target text (e.g., an English plural mark kept on an acronym, French acronyms being invariable). SSEsâ *SSEsâ SSE139 F. Grammar / phraseology in context F1 Inappropriate determiner, or incorrect gender or number agreement, within the term or between the term and its context. alkaline basaltâ*la basalte alcalinâle basalte alcalin 1132 F2A preposition required by the target syntactic structure is missing or wrong. slip behaviorâ*comportement en glisse- mentâ comportement de glissement 371 F3A grammatically correct combination that does not match the established phraseology of the domain. magma rises up the conduitâ*le magma monte vers le haut dans le conduitâle magma remonte par le conduit 30 F4 An incorrect spelling, notably in terms bor- rowed or adapted from the source language. aseismic slipâ*glissement aseismique â glissement asismique 316 G. Content alteration G1 Constituents absent from the source term are added, potentially altering the meaning or intro- ducing unintended nuances. fault creepâ*glissement lent de la faille â fluage de faille 410 G2Constituents of the term, or the term entirely, are omitted, losing information that affects the meaning. seismicity burstsâ*sĂ©ismesâvagues de sĂ©ismes 456 G3The output bears no relation to the source con- tent or is incoherent with the term to be trans- lated: an unintelligible translation or hallucina- tion. earthquakerupturesâ*ruptures dâĂ©ruptionsâ ruptures sismiques 4161 H. Target-language expression H1 The relative weight and hierarchy of the con- stituents, notably their punctuation, is not re- spected in the target text. â30 H2 A semantically correct formulation whose reg- ister or style is inappropriate for specialised discourse. earthquake rupturesâ*ruptures de trem- blements de terreâ ruptures sismiques 116 Table 12: The fine-grained codes of the typology (Cornejo CĂĄrcamo et al., 2026) as implemented in TERMJUDGE: the 23 error codes and the seven consistency codes of family B, with definitions adapted from the annotation guidelines and a geoscience example each (*: erroneous MT output; for error codes, the last element is the expected form). The default penalty follows the fixed 1/3/4 mapping of the gravities assigned by the typologyâs authors on the scale of BĂ©nard et al. (2024); support is the number of gold occurrences carrying the code among the 3,373 evaluated STEP occurrences (precedence-reduced labels). Family B carries no severity and is never assigned alone: it is the question delegated to the consistency judge (Q2), and the precedence reduction of §4.2 collapses pure-consistency annotations into a single B class (101 occurrences), so no per-code support is reported. Prompt labelCode Covers wrong_related_termA1A1 paraphrase_not_termA2A2 invented_calqueA3A3 left_untranslatedA4A4 dnt_violatedA5A5 wrong_componentA6A6 general_word_not_termA7A7 misparsed_structureCC1âC3 wrong_constituent_relation D1âD2 failed_transpositionEE1âE2 grammar_phraseologyFF1âF4 content_alteredGG1âG3 defective_register_formHH1âH2 no_errorâI Table 13: Transformation of the error typology for the Q1 judge: family A stays fine-grained, families CâH are collapsed into one grouped code each, family B is delegated to Q2, family J is dropped. The judge answers with the natural-language label; the canonical code is restored in post-processing. D.1 Why the Compact Typology The choice of the compact typology over the full 23-code version rests on a controlled comparison on STEP: two runs of the metric identical in ev- ery respect, including references verified identi- cal occurrence by occurrence, except the typology exposed to the Q1 judge, with every occurrence routed through the exhaustive judging path. The unbiased comparison between the two is binary er- ror detection, since the binarised gold is the same for both runs (1,019 erroneous and 2,354 error- free occurrences); Table 14 shows that the com- pact typology dominates on every measure, with 2.8 points more accuracy, 5.3 points more error precision, 78 fewer false positives and 17 fewer false negatives. Over-segmenting the answer space does not only degrade category choice: it degrades the binary decision itself, pushing the judge out of no_errormore often and wrongly. The multi-class views live in different label spaces and are not di- rectly comparable, but two observations are robust: the full typology leaves seven attested categories at exactly zero F1 (A5, C1, C2, D1, E1, F2, G1), and the family-A diagnostics, computed identically in both spaces, are of the same order (detection of some A code on gold-A occurrences at 0.590 for the compact typology against 0.563; exact A code among those detected at 0.521 against 0.536), so the compaction costs nothing on the fine-grained terminological core. The full prompt is also about Binary error detection CompactFull Accuracy0.7750.747 Macro-F10.7300.701 Balanced accuracy0.7280.703 Precision (error class)0.6320.579 Recall (error class)0.6090.593 F1 (error class)0.6200.586 False positives362440 False negatives398415 Table 14: Controlled comparison of the compact (13- code) and full (23-code) typologies on STEP: binary error detection against the same binarised gold (3,373 occurrences; 1,019 error, 2,354 no-error). The two runs differ only in the typology exposed to the Q1 judge. twice as expensive (about 6,700 input tokens per Q1 call against 3,100) and overruns the output bud- get where the compact prompt never does. The compaction is deliberately asymmetric. Family A keeps its seven fine codes because dis- tinguishing the mechanisms of wrong equivalent selection is precisely what the metric must explain; families C to H describe errors that affect terms without being terminology errors proper, so the metric only needs to know which family an occur- rence fails under. The distributional picture sup- ports the same asymmetry: taken together, families C to H account for 547 of the 1,019 erroneous oc- currences and are in no way marginal, but their sixteen fine codes are individually rare (median support 12; eleven of the sixteen at 16 occurrences or fewer; D2, F3 and H1 unattested; only F1 and G3 above one hundred; Table 12), too rare for the judge to be evaluated reliably on them or for the gold to discriminate them, whereas the grouped families recover workable supports (C 30, E 52, F 219, G 227). This observation is consistent with prior findings that fine-grained MQM annotation is much harder for LLMs than coarser quality judge- ments (Fernandes et al., 2023), that in-context clas- sification degrades over large label spaces (Milios et al., 2023), and with the practice of predicting spans and severities rather than the full MQM tax- onomy (Guerreiro et al., 2024); to our knowledge, no prior study quantifies the effect of typology size on an LLM judge all else being equal, which this controlled comparison provides. Its limits are those of the setting: one corpus, one MT output, one judge model, and a consensus gold reduced from multi-label annotations by a fixed precedence. ESTEP Confusion Matrix Table 15 gives the full confusion matrix behind the agreement figures of §5.1, with every occurrence judged through the exhaustive Q1âQ2 path. F Macro Aggregation Levels Tables 16 and 17 complete Table 4, reported under the micro-average (occurrence) aggregation, with the two macro-aggregations of §3.2, per source form and per concept. The conclusions are un- changed: TERMJUDGE ranks first in SPA at both levels, ahead of every comparison family. G Penalty Scale and Sensitivity This appendix details the penalty scale behind the document score of §5.2 and its sensitivity analysis. Like any typology-based metric, e.g. the Criti- cal/Major/Minor weighting of MQM, (§4.3), the document score rests on a penalty scale. The fine- grained penalties are not tuned: they follow the gravities assigned to each code by the typologyâs authors, through the fixed mapping of gravities 1/2/3 to weights 1/3/4 (Appendix D). The 1/3/4 spacing compresses the Minor/Major/Critical scale of MQM penalties (1/5/10): the decisive gap sep- arates near-misses from established errors, while gravities 2 and 3 both signal an unusable translation of the term; the compression also bounds the dy- namic range, preventing a few critical codes from dominating a document score. A grouped verdict asserts an error of the family but does not identify which sub-code applies, and the sub-codes of a fam- ily differ in gravity; each grouped family, C through H, is therefore scored at its most lenient sub-code, a single benefit-of-the-doubt rule applied uniformly, in favour of the evaluated translation. The uncer- tainty lies in the judgeâs granularity, not in the trans- lation, so the rule is a conservative bound rather than an indulgence: the penalty never exceeds what the verdict certainly establishes. D, with only 31 of the 13,200 occurrences, is too rare for its weight to have any measurable effect. The six grouped-family penalties (CâH) are the only quantities this rule sets. To test how sensitive the meta-evaluation is to them, we re-scored the saved verdicts under every assignment of weights 1â 4 to these six families (4 6 = 4,096configurations; A weights and the UNJUSTIFIED penalty at their de- faults; macro-average per lemmatised source form), at no LLM cost, since the judges emit labels and the score is a pure function of the penalties applied to them. The meta-evaluation responds in a sin- gle, interpretable direction: averaged over the grid, raising the weight of family F (grammar around the term) costs six SPA points (0.833 to 0.774), G (content alteration) three and a half, H (target- language expression) one and a half, E half a point, while C and D (term structure, semantic relations) have no measurable effect. These are the same peripheral families that the confusion analysis of §5.1 shows that the judge handles least reliably; in- flating their weight adds non-terminological noise into the score. The minimum rule, adopted on benefit-of-the-doubt grounds alone, thus receives independent support: it places the scale (Table 1) in theF=G=H=1region, whose 64 configurations keep SPA within 2.5 points of one another (0.846â 0.871) and above every comparison metric at that aggregation (Table 16), whatever the C, D and E weights, and re-scoring the same 64 scales at the micro aggregation of Table 4 keeps every one of them above every comparison metric as well; the UNJUSTIFIED penalty is second-order (varying it from 0.5 to 1.2 moves mean SPA by a tenth of a point on the development configuration). H Meta-Evaluation without Hapax Tables 18, 19 and 20 repeat the meta-evaluation of §5.2 with single-occurrence chains excluded from both the metrics and the human scores (48 documents, 275 segments). Two caveats: the sentence-level rows, being term-agnostic, keep their full-document scores against the complete human ground truth (50 documents), and are re- peated unchanged; and the variant removes exactly the occurrences that only a judge can evaluate, so it is not comparable to the main tables. On this reduced support the SPA differences between sys- tems are all within statistical noise (paired permu- tation tests,pâ„ 0.17for every pair involving TER- MJUDGE,k=2,000): the apparent reordering, with the glossary-conformity baselines nominally ahead, is not significant. TERMJUDGE remains signifi- cantly better than the five sentence-level rows on segment-level acc â eq (p < 0.05) and keeps the best document-level acc â eq of the terminology-aware families. I Exhaustive Judging Path on IWSLT23 On IWSLT23 we re-ran the eight outputs through the exhaustive judging path: no deterministic short- cut, every occurrence submitted to Q1 and, when Gold\ verdictA1A2 A3 A4 A5 A6 A7C DEFG H no-errorBtotal A174200010700039042111 A229000000010007120 A3444000150050040265 A4001280400010009034 A520206101000009223 A63473006702010028795 A711000171000005723 C13200204010619130 D0000000100010103 E000800020910028452 F0006013001 10 172015911219 G126228227110010 320124227 H28000010000005016 no-error35528343261663 17 40 1621992 104 2354 B1100040000261824101 total282 1093958117450 206 40 69 7242390 149 3373 Table 15: Confusion matrix of TERMJUDGEâs verdicts against the STEP gold labels (3,373 scored occurrences; gold labels in rows, verdicts in columns, diagonal in bold). On the verdict side, the no-error column aggregates CONFORMING and JUSTIFIED, and the B column is UNJUSTIFIED (mapping of §4.2); row totals are the gold supports of Table 3. MetricSPA acc â eq (doc) Judged divergences (this work) TERMJUDGE0.8710.556 Divergence QE GEMBA div 0.8150.524 MetricX-24 div 0.7970.536 CometKiwi div 0.8040.552 Glossary conformity first-translation fallback0.7660.458 majority fallback0.7580.454 Terminology-filtered QE GEMBA-MQM (any mention) 0.7300.209 GEMBA-MQM (category)0.6460.183 General QE GEMBA gen 0.7210.530 MetricX-24 gen 0.7280.568 CometKiwi gen 0.7470.549 Table 16: Meta-evaluation on BioMQM-Terms, source- form (macro) aggregation. General-QE rows are un- changed across aggregations (only the human side varies for them). error-free, to Q2. Table 21 compares this path with the default configuration: the ranking is repro- duced (KendallÏ = 0.786; 25 of 28 system pairs concordant, the three discordances all involving near-ties, and every base+terms system still ahead of every baseline one) while requiring 2.55 times more unique LLM calls. The deterministic short- MetricSPA acc â eq (doc) Judged divergences (this work) TERMJUDGE0.8670.562 Divergence QE GEMBA div 0.8340.518 MetricX-24 div 0.8110.535 CometKiwi div 0.8230.549 Glossary conformity first-translation fallback0.7800.447 majority fallback0.7710.443 Terminology-filtered QE GEMBA-MQM (any mention) 0.7250.204 GEMBA-MQM (category)0.6410.178 General QE GEMBA gen 0.7360.527 MetricX-24 gen 0.7440.566 CometKiwi gen 0.7660.556 Table 17: Meta-evaluation on BioMQM-Terms, concept (macro) aggregation. cuts and the Q1 propagation thus buy a substantial cost reduction without affecting the conclusions. J Consistency Labels: Grounding and Distribution The ten labels of the consistency judge opera- tionalise motivations for terminological variation documented in terminology and translation studies. Among the seven coherent labels, âfirst mentionâ is MetricSPA acc â eq (seg) acc â eq (doc) Judged divergences (this work) TERMJUDGE0.7500.6790.464 Divergence QE GEMBA div 0.7250.6710.393 MetricX-24 div 0.7520.6750.451 CometKiwi div 0.7520.6780.458 Glossary conformity first-translation fallback0.7780.6760.406 majority fallback0.7700.6800.403 Terminology-filtered QE (50 docs) GEMBA-MQM (any mention) 0.7750.6530.219 GEMBA-MQM (category)0.6890.6530.193 General QE (50 docs) GEMBA gen 0.7360.6500.533 MetricX-24 gen 0.7360.6500.566 CometKiwi gen 0.7290.6510.552 Table 18: No-hapax meta-evaluation on BioMQM- Terms, micro-average (occurrence) aggregation (48 doc- uments, 275 segments, except term-agnostic sentence- level rows: 50 documents, full human ground truth). MetricSPA acc â eq (doc) Judged divergences (this work) TERMJUDGE0.7410.461 Divergence QE GEMBA div 0.7080.388 MetricX-24 div 0.7500.452 CometKiwi div 0.7350.450 Glossary conformity first-translation fallback0.7850.400 majority fallback0.7760.396 Terminology-filtered QE (50 docs) GEMBA-MQM (any mention) 0.7300.209 GEMBA-MQM (category)0.6460.183 General QE (50 docs) GEMBA gen 0.7210.530 MetricX-24 gen 0.7280.568 CometKiwi gen 0.7470.549 Table 19: No-hapax meta-evaluation, source-form ag- gregation. the base case: no translation is established yet, so none can be inconsistent. âExplicitationâ follows the translation procedure of that name (Vinay and Darbelnet, 1972) and the cognitive function of vari- ation, by which writers vary a term to explain a concept better (Pecman, 2014). âAvoid repetitionâ reflects the stylistic avoidance of repetition, marked in French (Vinay and Darbelnet, 1972). âRegisterâ and âfacetâ follow Bowker (1998), who shows that variation in expert discourse is motivated by the au- dience and purpose of the text and by the dimension of the concept that the context foregrounds, a cog- nitive function also described by Pecman (2014). âDocument usageâ and âsynonym mergeâ encode the consistency norm itself: repeating the render- MetricSPA acc â eq (doc) Judged divergences (this work) TERMJUDGE0.7310.464 Divergence QE GEMBA div 0.7170.384 MetricX-24 div 0.7540.450 CometKiwi div 0.7440.450 Glossary conformity first-translation fallback0.7940.401 majority fallback0.7850.398 Terminology-filtered QE (50 docs) GEMBA-MQM (any mention) 0.7250.204 GEMBA-MQM (category)0.6410.178 General QE (50 docs) GEMBA gen 0.7360.527 MetricX-24 gen 0.7440.566 CometKiwi gen 0.7660.556 Table 20: No-hapax meta-evaluation, concept-level ag- gregation. Default Exhaustive Unique LLM judgements6,52516,612 Approximate cost (USD)6.815.6 Kendall Ï between rankings0.786 (25/28 pairs) base+terms ahead of baseline16/1616/16 Table 21: Default configuration (deterministic shortcuts, Q1 propagation) vs. exhaustive judging path on the eight IWSLT23 outputs, same judge and prompts. The three discordant pairs all involve near-ties (score gap†0.006 in one of the two configurations). ing established in the document is coherent, and two source forms that differ only by syntactic re- ordering (colour flatbed scanner vs flatbed colour scanner in Bowker, 1998; morphosyntactic vari- ants in Cornejo CĂĄrcamo et al., 2025) may share one translation. The three inconsistency labels mir- ror the consistency tradition of §2: âsynonym in- consistencyâ and âgratuitous divergenceâ penalise a switch away from an established rendering without such a motivation, and âneutralisationâ the erasure of a source-side distinction (Bowker and Hawkins, 2006). Table 22 details, per system and prompting con- dition, the distribution of the consistency-judge labels on PARANLP behind the analysis of §5.4. K Judgement Counts and Cost A unique judgement is one judge prompt actu- ally sent to the LLM. Three mechanisms keep this number low: deterministic shortcuts skip cases that need no judge, Q1 propagation lets identical translations of the same source form baselinebase+terms Q2 label (%)Ll. Qw. E9 E22 Ll. Qw. E9 E22 Document usage79.5 76.5 79.2 80.1 78.8 77.2 79.8 81.0 First mention6.06.76.66.16.67.07.36.0 Synonym merge2.23.02.02.11.71.81.81.4 Other coherent0.10.20.30.30.30.20.20.4 Neutralisation0.81.81.21.62.11.81.51.6 Synonym inconsist. 3.73.33.03.22.02.42.92.3 Grat. divergence7.88.47.76.68.69.66.57.2 Judgements (count) 1393 1471 1537 1481 1127 1057 1300 1179 Table 22: Distribution of the consistency-judge labels on PARANLP, per system (Ll.: Llama, Qw.: Qwen, E9/E22: Euro9/Euro22) and prompting condition (per- centage of that systemâs Q2 judgements; last row: absolute counts). The first four labels are coherent (JUSTIFIED), the last three inconsistent (UNJUSTIFIED). âregisterâ was never chosen; âfacetâ and âavoid repetitionâ are grouped under âOther coherentâ; rare out-of-registry answers are counted with gratuitous divergence. share one judgement, and a cache serves identi- cal prompts, so repeated runs add no calls. With gpt-4.1-miniat list price, evaluating BioMQM- Terms took about 12,000 judgements (about $6), the eight IWSLT23 outputs about 6,500 (about $7) and the eight PARANLP outputs about 46,000 (about $50). STEP is the exception: for the vali- dation of §4.2 it deliberately runs the exhaustive path with no shortcut, so its 3,377 occurrences (one system) yield about 3,700 judgements. For compar- ison, glossary-conformity metrics issue no LLM call, neural QE needs one GPU inference per seg- ment, and GEMBA-MQM issued about 4,500 calls togpt-4.1-minion BioMQM-Terms, for about $2. These counts exclude preprocessing, whose alignment step issues onegpt-4.1-minicall per occurrence: about $5 for BioMQM-Terms, $4 for the eight IWSLT23 outputs and $33 for the eight PARANLP outputs at list price. L LLM Prompts All prompts are templated with Pythonstr.formatplaceholders (in braces) populated at runtime. The domain name is injected per corpus through thedomainplaceholder, which presents the term under judgement as belonging to that domain (e.g., âin the biomedical domainâ); the context blocks (concept_block,preceding_segments,prior_renderings,discourse_context, concept_renderings) are built per occurrence, the preceding-segment block holding the five preced- ing aligned segment pairs. Each judge receives a fixed system message identifying its role, and answers with a JSON object whosejustificationfield precedes the label, so that the label must follow from the stated reasoning. The judges decode at temperature 0. The hapax judgements reuse the Q1 prompt verbatim (§3.2). L.1 Q1: Error Judgement You are judging how a machine-translation system rendered ONE specialised source term, from English into French in the domain domain. The question is ABSOLUTE: is the target a valid, domain- appropriate rendering of this source term? Decide whether it is an ERROR (one label below) or acceptable (no_error). EVERY alignment is judged here, including those whose target matches the rendering already used in this document. Such a match is not by itself proof of correctness âit may simply repeat the systemâs own output. Judge the target itself. Error typology âanswer with exactly ONE label. The first seven labels distinguish fine-grained mechanisms of wrong equivalent selection; the following six each group several related mechanisms. - wrong_related_term : the target is a term that exists in the domain domain but denotes a neighbouring concept, not the one named by the English term (e.g. for "tremor" the MT output " tremblement de terre" names a different concept, the earthquake; expected: "trĂ©mor"). Not this label: if the target is not a domain term at all, see general_word_not_term or invented_calque. - paraphrase_not_term : the target replaces the established term with a descriptive multi-word paraphrase in general language (e.g. for "lithospheric mantle" the MT output "couche profonde sous la croĂ»te terrestre" is a description, not the term; expected: "manteau lithosphĂ©rique"). - invented_calque : the target is a literal creation of the system that is neither an attested term of the domain domain nor a general-language expression in French (e.g. for "earthquake swarm" the MT output "sillage sismique" exists nowhere; expected: "essaim sismique"). - left_untranslated : the source term was left in English although an established French equivalent exists (e.g. for "megathrusts" the MT output "megathrusts" is untranslated; expected: "mĂ© gachevauchements"). - dnt_violated : a Do-Not-Translate item (proper name, brand, gene or product code, standardised nomenclature symbol, acronym conventionally kept) was translated, or its letters were reordered/ reworked (e.g. for "Yellowstone Caldera" the MT output "Caldeira de la Pierre Jaune" translates a proper name; expected: "Caldeira de Yellowstone"; for "MORB" the MT output "BDRM" reworks the acronym; expected: "MORB"). - wrong_component : one or more constituents of a multi-part term are lexically wrong while the others are correct, yielding a partially correct hybrid term (e.g. for "fault patches" the MT output "taches de faille" mistranslates "patches" while "faille" is right; expected: "zones de faille"). Not this label: if the whole target is a different attested domain term, that is wrong_related_term. - general_word_not_term : a single general-language word used instead of the specialised term âunlike paraphrase_not_term, one lexical unit rather than a descriptive phrase (e.g. for "repeaters" the MT output "rĂ©pĂ©titeurs" is a general-language word with no terminological value; expected: " sĂ©ismes rĂ©pĂ©titifs"). - misparsed_structure : the internal structure of the complex term or noun phrase is mis-analysed â the head noun is misidentified (the head fixes the base category of the concept), a modifier is attached to the wrong constituent, or the constituents are ordered against French conventions (e. g. for "subduction-zone serpentinites" the MT output "zones de subduction constituĂ©es de serpentinite" makes the wrong noun the head; expected: "serpentinites en zone de subduction"; for "slow earthquake" the MT output "lent sĂ©isme" has the wrong order; expected: "sĂ©isme lent"). - wrong_constituent_relation: the semantic relation between the termâs constituents (often implicit in English) is rendered wrongly âa wrong linking relation or unfolding, a coordination misread, or a shared (factorised) constituent not distributed to all conjuncts (e.g. for "fast and slow events" the MT output "Ă©vĂ©nements Ă la fois rapides et lents" misreads two distinct event types as simultaneous properties; expected: "Ă©vĂ©nements rapides et lents"; for "pressure-solution and dislocation creep" the MT output "pression-solution et le glissement par dislocation" fails to distribute "creep" to both conjuncts; expected: "fluage par dissolution-prĂ©cipitation et par dislocation"). - failed_transposition : the synthetic English structure is not transposed to the analytic French one âa necessary explicitation (preposition, article, unfolding) is missing, or the English syntax is calqued although the words are attested (e.g. for "strike-slip and subduction thrust faults" the MT output "failles de dĂ©crochement et de chevauchement de subduction" lacks the needed explicitation; expected: "failles de dĂ©crochement et de chevauchement dans les zones de subduction"; for "SSEs" the MT output "SSEs" erroneously keeps the English plural mark; expected: "SSE", French sigles are invariable). - grammar_phraseology : the term choice is right but the grammar or phraseology around it is wrong â determiner or gender/number agreement, a missing or wrong preposition, a non-idiomatic collocation for the domain, or a spelling error (e.g. for "alkaline basalt" the MT output "la basalte alcalin" has the wrong gender; expected: "le basalte alcalin"; for "aseismic slip" the MT output "glissement aseismique" is a misspelling; expected: "glissement asismique"). Not this label: for "aseismic slip" the MT output "glissement asismique" is the correctly spelled attested term âthat is no_error. - content_altered : the content is altered âan element absent from the source term is added, one or more constituents (or the whole term) are omitted, or the output is unintelligible or unrelated to the source (e.g. for "fault creep" the MT output "glissement lent de la faille" adds "lent", absent from the source; expected: "fluage de faille"; for "earthquake ruptures" the MT output " ruptures dâĂ©ruptions" hallucinates an unrelated word; expected: "ruptures sismiques"). If the aligned target is (empty), the term was omitted, so use content_altered âunless the concept is conveyed elsewhere in the French segment, in which case no_error. - defective_register_form : the meaning is right but the target-language expression is defective âthe weighting or punctuation hierarchy of the constituents is mishandled, or the register is inappropriate for specialised discourse (e.g. for "earthquake ruptures" the MT output "ruptures de tremblements de terre" is correct in meaning but general-public in register; expected: " ruptures sismiques"). - no_error : the target conveys the same specialised concept correctly and idiomatically âthe exact established term, an attested terminological synonym, a standard abbreviation or acronym, an accepted orthographic variant, a grammatical inflection, or an established domain loanword. Whether a correct-but-different choice is CONSISTENT with the rest of the document is decided in a separate later step, so do NOT judge consistency here. How to decide (reason in this order, then output exactly one label): 1. Establish the expected French term yourself: the form attested for this concept in specialised domain usage (domain corpora, terminology databases, the fieldâs literature). There is not necessarily a single mandatory form. Note that the expected form depends on the SOURCE FORM at hand: an acronym calls for the established acronym, not for the spelled-out head term. 2. Weigh the supporting sections below (concept, preceding segments, prior renderings) for what they are: support, never a verdict. 3. If the target conveys the concept correctly AND is itself an attested, idiomatic form of it, the answer is no_error âwhether or not it coincides with the expected form. 4. Otherwise, locate where the defect lies and assign the SINGLE most specific label for it: - a wrong terminological equivalent was selected -> wrong_related_term / paraphrase_not_term / invented_calque / left_untranslated / dnt_violated / wrong_component / general_word_not_term; - the equivalents are right but the complex term / noun phrase is mis-structured (head, modifier attachment, constituent order) -> misparsed_structure; - the constituents are right but their semantic relation is wrong (linking, factorisation) -> wrong_constituent_relation; - the English structure is not transposed to French (missing explicitation, syntactic calque) -> failed_transposition; - the term is right but its grammar or phraseology in context is wrong (agreement, preposition, collocation, spelling) -> grammar_phraseology; - content is added, omitted, or hallucinated -> content_altered; - the meaning is right but the target-language expression or register is inappropriate -> defective_register_form. When a preposition or an unfolding is involved, separate the three by mechanism: wrong_constituent_relation = the semantic relation between constituents was misread; failed_transposition = the relation is understood but the compact English structure is kept untransposed; grammar_phraseology = the analysis is right and only the grammatical marking ( agreement, preposition choice, spelling) is wrong. concept_blockpreceding_segmentsprior_renderingsCurrent segment (the one containing the pair under evaluation): EN: src_segment FR: tgt_segment Aligned pair under evaluation: - English term as it appears in the segment: "src_form_raw" - French translation aligned by the system: "tgt_aligned_raw" Return ONLY a JSON object with this exact schema, keys in this order, with no code fences and no text before or after â"justification" is your reasoning, and "label" must follow from it: "justification": "<one short sentence: name the expected French form and locate the defect, or state why the target is a valid attested form>", "label": "<wrong_related_term|paraphrase_not_term|invented_calque|left_untranslated|dnt_violated| wrong_component|general_word_not_term|misparsed_structure|wrong_constituent_relation| failed_transposition|grammar_phraseology|content_altered|defective_register_form|no_error>" L.2 Q2: Consistency Judgement You are judging ONE occurrence of a specialised term in an English-to-French translation in the domain domain. The terminology has already been judged CORRECT by a previous step: do NOT re- judge whether the target denotes the right concept, and never report a terminology error here. Your only question is DOCUMENT CONSISTENCY: given how this concept is rendered elsewhere in this document, is this rendering a coherent choice, or does it introduce an inconsistency? Answer from the document evidence below âno external reference is required, and none may be available. Three things make a rendering NOT coherent: 1. NEUTRALISATION âthe target erases a distinction the author made. The source document uses SEVERAL distinct forms for this concept (a head term and its variants), and this occurrence renders the present source form with the rendering that belongs to ANOTHER source form, collapsing two source distinctions into one French form (e.g. the source uses both "dynamic fault slip" and " dynamic slip", and both end up rendered "glissement dynamique": the distinction between the two source forms is erased). Check each source formâs renderings in the evidence below. Exception: if the merged form is exactly the rendering the document has already established for this concept AND the distinction carries no function here, it is coherent âlabel synonym_merge. 2. SYNONYM INCONSISTENCY âthe concept already has an established rendering in this document, and this occurrence switches to a DIFFERENT ADMITTED designation (the evidence line says whether the target is an admitted synonym) without doing any communicative work (e.g. the document has established "fluage de faille" for "fault creep", and this occurrence switches to "fluage de la faille" for no visible reason). 3. GRATUITOUS DIVERGENCE âsame switch, but the divergent rendering is NOT an admitted synonym of the concept (it was judged a valid attested form in the previous step, yet it departs from both the documentâs established usage and the admitted designations) and no communicative work justifies the departure. A rendering IS coherent when it repeats the documentâs established form, or when the divergence does genuine communicative work from this closed list âthe need must be visible in the current segment itself; do not use these labels as default excuses: - introduction : first mention of the concept in the document; no rendering is established yet, so there is nothing to be inconsistent with. - explicitation : first mention, and the rendering expands or clarifies the term (e.g. unfolds an acronym, adds the head noun). - avoid_repetition : the established rendering already occurs in the SAME sentence, so repeating it would be stylistically heavy. Proximity in the same sentence is required âoccurrences in earlier segments do NOT justify this label. - document_usage : the target repeats the rendering already established in this document for this source form. - register : the current passage itself visibly calls for a different register (e.g. a quoted passage , a glossary definition, an explicitly popularising aside). - facet : the divergent rendering highlights a facet of the concept that the current segment explicitly thematises. - synonym_merge : the neutralisation exception defined above âin particular when the source forms differ only by a syntactic reordering or trivial variation that French cannot reproduce (e.g. " loading rate" and "rate of loading" can only both be rendered "taux de chargement": the merge is forced by the language, not a system choice). Length or fluency alone is NOT communicative work. If this is the FIRST mention of the concept and no rendering is yet established (the evidence line says so), the choice is coherent: label introduction, or explicitation if it expands the term. How to decide (reason in this order, then output exactly one label): 1. If this is the first mention -> introduction or explicitation. Stop. 2. Compare the target with the renderings already established for THIS source form. If identical to the DOMINANT rendering (marked in the evidence) -> document_usage. Stop. If identical only to a MINORITY prior rendering, this is not self-justifying âan inconsistency repeated is still an inconsistency; continue to steps 3-4. 3. If the target matches the rendering established for ANOTHER source form of this concept -> neutralisation, unless the synonym_merge exception applies; name in the justification the other source form whose rendering this occurrence reproduces. 4. Otherwise the target diverges from the established usage: if the current segment visibly justifies it (closed list above) -> that coherent label; if not, check the target against the French glossary designations in the Concept line: listed -> synonym_inconsistency; not listed -> gratuitous_divergence. If the Concept line says the concept has no glossary entry, the admitted/ gratuitous distinction cannot be made: use synonym_inconsistency âthe previous step already validated the rendering as an attested designation of this concept. CAUTION: the document renderings below are THIS SAME SYSTEMâs earlier outputs. They show what the document has established in practice âwhich is exactly what consistency is measured against âbut they are not proof of correct French. discourse_contextconcept_renderingsCurrent segment (the one containing the occurrence under evaluation): EN: src_segment FR: tgt_segment Occurrence under evaluation: - English source form as it appears: "src_form_raw" - Is this source form the conceptâs head term or a variant? head_or_variant - French translation aligned by the system: "tgt_aligned_raw" - concept_glossary_line - Is the target one of the French glossary designations above? is_glossary_designation - Is this the first mention of the concept in the document? is_first_mention Return ONLY a JSON object with this exact schema, keys in this order, with no code fences and no text before or after â"justification" is your reasoning, and "label" must follow from it: "justification": "<one short sentence: name the documentâs established rendering and state why this occurrence is coherent with it or diverges from it; for neutralisation, name the other source form whose rendering is reproduced>", "label": "<introduction|explicitation|avoid_repetition|document_usage|register|facet|synonym_merge| neutralisation|synonym_inconsistency|gratuitous_divergence>" L.3 Reference Selection Used by thevalidatedcascade when an LLM must select among observed candidate translations (§3.2). You are given one English specialised source term in the domain domain and the French translations observed for it across several machine-translation systems in a single document. Pick the best reference translation: the single French form a professional translator of this domain would use consistently for this term in this document. It becomes the anchor against which every systemâs translation of this term is judged. Selection rules, in order: 1. Prefer the glossary candidate (the line tagged "glossary:") when there is one, unless it clearly renders a different sense of the term than the sense used in this document. A glossary candidate is shown only for a head term; a variant source form has none. 2. Otherwise prefer the most idiomatic, terminologically appropriate French form. An established concise term beats a longer explanatory paraphrase; never prefer a form merely because it is longer. 3. Tie-break: higher observed count; if still tied, the one appearing first (candidate list first, then observed translations top to bottom). The chosen reference MUST be one of the French forms displayed below. Never invent, correct, translate back, or rephrase a form. Copy it exactly, WITHOUT the surrounding quotes and WITHOUT the strategy tag. In the two lists below, each form is wrapped in quotes; a candidate line is additionally prefixed by a strategy tag ("glossary", "frequency" or "first"). The quotes and the tag are display formatting and must not be copied into your answer; the tag only tells you where a candidate came from (Rule 1 uses it to spot the glossary candidate). A prefLabel shown as "(unknown)", or a list shown as "(none)", is unavailable. <examples> Illustrative of the decision types (they may be from a different domain than the item you decide). Each example shows the same fields as the item you will decide. Base each decision only on the fields shown. <example> Term and document: - English source term: seismic hazard - Glossary concept (English prefLabel): seismic hazard - Document: doc3 Candidate references (each line: "- <strategy>: <French form in quotes>"): - glossary: "alĂ©a sismique" - frequency: "risque sismique" - first: "risque sismique" Observed French translations in this document (each line: "- <French form in quotes> Ă<count>"): - "risque sismique" Ă2 - "alĂ©a sismique" Ă1 "reference": "alĂ©a sismique", "justification": "Rule 1 prefers the glossary candidate âalĂ©a sismiqueâ, the domain-standard term, over the more frequent ârisque sismiqueâ." </example> <example> Term and document: - English source term: stress - Glossary concept (English prefLabel): stress - Document: doc9 Candidate references (each line: "- <strategy>: <French form in quotes>"): - glossary: "stress" - frequency: "contrainte" - first: "contrainte" Observed French translations in this document (each line: "- <French form in quotes> Ă<count>"): - "contrainte" Ă4 - "stress" Ă1 "reference": "contrainte", "justification": "Rule 1 exception: the glossary form âstressâ fits the psychological sense, but this document uses the mechanical sense, whose established term is âcontrainteâ." </example> <example> Term and document: - English source term: crust-mantle boundary - Glossary concept (English prefLabel): Moho - Document: doc5 Candidate references (each line: "- <strategy>: <French form in quotes>"): - frequency: "limite entre la croĂ»te et le manteau" - first: "limite entre la croĂ»te et le manteau" Observed French translations in this document (each line: "- <French form in quotes> Ă<count>"): - "limite entre la croĂ»te et le manteau" Ă3 - "limite croĂ»te-manteau" Ă2 "reference": "limite croĂ»te-manteau", "justification": "No glossary candidate; rule 2 prefers the concise compound term âlimite croĂ»te- manteauâ over the longer paraphrase, even though the paraphrase is more frequent." </example> <example> Term and document: - English source term: very-low-frequency earthquakes - Glossary concept (English prefLabel): (unknown) - Document: doc1 Candidate references (each line: "- <strategy>: <French form in quotes>"): - frequency: "sĂ©ismes de trĂšs basse frĂ©quence" - first: "sĂ©ismes de trĂšs basse frĂ©quence" Observed French translations in this document (each line: "- <French form in quotes> Ă<count>"): - "sĂ©ismes de trĂšs basse frĂ©quence" Ă2 - "sĂ©ismes Ă trĂšs basse frĂ©quence" Ă2 "reference": "sĂ©ismes de trĂšs basse frĂ©quence", "justification": "âdeâ and âĂ â are both attested and equally idiomatic here and the two forms tie on count, so rule 3 keeps the one appearing first." </example> <example> Term and document: - English source term: SEM - Glossary concept (English prefLabel): scanning electron microscope - Document: doc8 Candidate references (each line: "- <strategy>: <French form in quotes>"): - frequency: "microscopie Ă©lectronique Ă balayage" - first: "microscopie Ă©lectronique Ă balayage" Observed French translations in this document (each line: "- <French form in quotes> Ă<count>"): - "microscopie Ă©lectronique Ă balayage" Ă2 - "MEB" Ă1 "reference": "MEB", "justification": "No glossary candidate; rule 2 prefers the established concise acronym âMEBâ over its longer spelled-out form, even though the spelled-out form is more frequent." </example> </examples> Term and document: - English source term: src_form - Glossary concept (English prefLabel): preflabel_en - Document: doc_id Candidate references (each line: "- <strategy>: <French form in quotes>"): candidates Observed French translations in this document (each line: "- <French form in quotes> Ă<count>"): distribution Return ONLY a JSON object with this exact schema: "reference": "<the chosen French form, copied exactly, no surrounding quotes, no strategy tag>", "justification": "<one short sentence giving the decisive reason>" L.4 Reference Generation Used by thevalidatedcascade when a vetted heuristic reference is rejected and a reference must be produced from source-side evidence only (§3.2). You are given a specialised concept in the domain domain and ONE English source term that designates it. Produce the French term expected for THIS source term: the term a professional translator of this domain would use for it, consistently, in a document. It must be an ESTABLISHED French term, not a plausible invention; it will serve as a candidate reference against which MT systems are judged. # Source variation category The "Source variation category" tells you how the source term relates to the conceptâs HEAD TERM: it is the variation that turns the head term into this source form (both in English). If the source term IS the head term, the category is "none". Treat it as an AID to find the right French reference, not as a constraint: the reference need not mirror the same kind of variation in French. It hints at what the source term is (an acronym, a reduction, a synonym...) so you produce the French form actually established for it; established usage always decides. - none : the source term is the head term itself. - graphical variation : same content, different written form âacronym or abbreviation (World Health Organization / WHO), symbol (lead / Pb), spelling or hyphenation (trademark / trade mark). - reduction : a shortened form âelement deleted (airborne dust particle / airborne dust) or an extension dropped (post-authorisation safety study / post-authorisation study). - expansion : elements added âan explicit feature, a lexical insertion, or the spelled-out form of an abbreviation (redox / oxidation-reduction). - lexical variation : a different lexical designation of the same concept âsynonym, quasi-synonym, or attested translation (stress / constraint). - morphosyntactic variation : structure or inflection change âconstituent order, article, inflection, derivation (employment contract / contract of employment). - composite form : several of the above combined; apply the reasoning of each component change. # Rules 1. Let the variation category guide you toward the French designation actually established for this source term. Often the established French form is of the same kind (an acronym stays an acronym, a reduction stays a reduction), but not always: established usage decides, never a mechanical derivation from the category. 2. Acronyms and abbreviations (category "graphical variation", or "expansion" when the source form is the spelled-out expansion of one). The correct French form depends on how the HEAD TERM is translated into French. Decide among three moves: (a) Invert the acronym âwhen the concept is translated into French and that French term has an established French acronym, give the FRENCH acronym. E.g. WHO -> OMS (Organisation mondiale de la santĂ©); emergency contraception (EC) -> contraception dâurgence (CU); DNA -> ADN. (b) Keep the source acronym as-is âwhen the field uses the English acronym unchanged in French. E. g. PCR, BERT, GPT, LLM. (c) Spell it out (rare) âwhen neither a French acronym nor the borrowed source acronym is used, give the developed French form. E.g. ASAP -> dĂšs que possible. Do NOT assume French spells the term out by default. Your "reference" must match your reasoning: if you conclude the source acronym is used as-is, return that acronym. 3. Keep the referenceâs SCOPE identical to the source term: add no word it does not contain, drop none it does. 4. For a lexical variation, give the established French term proper to THAT synonym, not the French head term. 5. If French keeps a non-acronym source form unchanged (a proper name, a do-not-translate item), return it unchanged. 6. Do NOT invent. For a non-acronym form, if no established French designation exists and you are not confident, set "generatable" to false. For an acronym, keep the source acronym (rule 2b) instead of declining. 7. Return the reference in canonical form: lemma (singular unless the term is inherently plural), lowercase except proper nouns and acronyms, no surrounding quotes. # Examples Each example shows the same fields as the Data block below, then the expected JSON. The examples are illustrative only: NEVER assume the current term behaves like an example it superficially resembles; established usage for THIS term decides. Example 1 - English source form: WHO - Source variation category: graphical variation - Concept head term (English): World Health Organization - Concept established French term (head term), if known: Organisation mondiale de la santĂ© => "justification": "Acronym inverted: the concept is translated and has an established French acronym.", "generatable": true, "reference": "OMS", "confidence": "high" Example 2 - English source form: PCR - Source variation category: graphical variation - Concept head term (English): polymerase chain reaction - Concept established French term (head term), if known: rĂ©action en chaĂźne par polymĂ©rase => "justification": "Acronym kept as-is: standard French usage borrows PCR unchanged.", " generatable": true, "reference": "PCR", "confidence": "high" Example 3 - English source form: oxidation-reduction - Source variation category: expansion - Concept head term (English): redox - Concept established French term (head term), if known: redox => "justification": "Expansion of an abbreviation: the spelled-out French form of «redox ».", " generatable": true, "reference": "oxydorĂ©duction", "confidence": "high" Example 4 - English source form: abiotic constraint - Source variation category: lexical variation - Concept head term (English): abiotic stress - Concept established French term (head term), if known: stress abiotique => "justification": "Lexical variation: French equivalent of the synonym, not of the head term.", " generatable": true, "reference": "contrainte abiotique", "confidence": "medium" Example 5 - English source form: post-authorisation study - Source variation category: reduction - Concept head term (English): post-authorisation safety study - Concept established French term (head term), if known: Ă©tude de sĂ©curitĂ© post-autorisation => "justification": "Reduction: the French head term with the dropped element (âsĂ©curitĂ©â) removed .", "generatable": true, "reference": "Ă©tude post-autorisation", "confidence": "medium" # Data - English source form: src_form - Source variation category: variation_category - Concept head term (English): preflabel_en - Concept established French term (head term), if known: preflabel_fr English sentences where the source form occurs (evidence of usage, may mention other variants of the concept; possibly truncated): contexts Return ONLY a JSON object with this exact schema, keys in this order, with no code fences and no text before or after â"justification" is your reasoning; "generatable", "reference" and "confidence" must follow from it: "justification": "<one short sentence: how you derived the form, or why no established form exists >", "generatable": <true or false, bare JSON boolean>, "reference": "<the canonical French form; empty string if generatable is false>", "confidence": "<high|medium|low âyour confidence that this exact form is the established one>" L.5 Term Alignment Used by the preprocessing to align each detected occurrence with its translation in the target segment (Appendix A), after the system message âYou are a professional translator specializing in terminology alignment.â The instructions are those of the organisers of the WMT25 terminology task (Semenov et al., 2025); the six in-context examples are ours. You are a professional English-French translator, teaching the students the course on technical translation. You are checking a studentâs translation of a sentence that contains a technical term. You are given an English term (it can be a word or an expression), a source English sentence containing this term (it may be cased differently or contain additional punctuation), and a studentâs French translation. You need to find how the student has translated the term in question in French, and return only that term. Important: do not change the translated term anyhow, copy it straight from the sentence! For example, keep the casing and the grammar form of the translated term as is. When completing the task, follow the examples below: English sentence: This paradigm based on ant colony algorithms for the exploration of the graph removes the need to dynamically expand the graph: the memory footprint becomes independant of the language model size. English term: language model French translation: Ce paradigme basĂ© sur les algorithmes de colonie de fourmis pour lâexploration du graphe supprime le besoin dâĂ©tendre dynamiquement le graphe : lâempreinte mĂ©moire devient indĂ© pendante de la taille du modĂšle de langage. Translated term: modĂšle de langage English sentence: In this article we presented a new paradigm to expand word graphs in automatic speech recognition systems. English term: automatic speech recognition systems French translation: Dans cet article, nous avons prĂ©sentĂ© un nouveau paradigme pour lâexpansion des graphes de mots dans les systĂšmes de reconnaissance automatique de la parole. Translated term: systĂšmes de reconnaissance automatique de la parole English sentence: Evolution of rescoring according the number of ant by node (on the dev set) during 1 run and comparison between Viterbi beam-search and ant colonies algorithm: Computing time against performance on the dev set with a 4-gram language model. English term: dev set French translation: Ăvolution du rĂ©-Ă©valuation en fonction du nombre dâant par nĆud (sur lâensemble de dĂ©veloppement) au cours dâune exĂ©cution et comparaison entre la recherche en faisceau Viterbi et lâalgorithme des colonies de fourmis : temps de calcul par rapport aux performances sur lâ ensemble de dĂ©veloppement avec un modĂšle de langage Ă 4 grammes. Translated term: ensemble de dĂ©veloppement English sentence: In Earth sciences, plate tectonics took a very long time to be accepted in the 1960 s, and the origin of global warming was still the subject of heated debate very recently. English term: plate tectonics French translation: En sciences de la Terre, la thĂ©orie de la tectonique des plaques a mis trĂšs longtemps Ă ĂȘtre acceptĂ©e dans les annĂ©es 1960, et lâorigine du rĂ©chauffement climatique Ă©tait encore trĂšs rĂ©cemment lâobjet de dĂ©bats houleux. Translated term: tectonique des plaques English sentence: The first SR15 Special Report (October 2018) focuses on the impacts associated with a global warming of 1.5°C, as well as compatible greenhouse gas emission trajectories, in the context of strengthening the response to climate change, sustainable development and efforts to eradicate poverty: English term: greenhouse gas emission French translation: Le premier rapport spĂ©cial SR15 (octobre 2018) se concentre sur les impacts associĂ©s Ă un rĂ©chauffement planĂ©taire de 1,5°C, ainsi que sur les trajectoires dâĂ©missions de gaz Ă effet de serre compatibles, dans le contexte du renforcement de la lutte contre le changement climatique, du dĂ©veloppement durable et des efforts dâĂ©radication de la pauvretĂ© : Translated term: Ă©missions de gaz Ă effet de serre English sentence: We used our set of manually-written chronologies as a training corpus to perform machine learning experiments. English term: training corpus French translation: Nous avons utilisĂ© notre ensemble de chronologies rĂ©digĂ©es manuellement comme corpus dâapprentissage pour effectuer des expĂ©riences dâapprentissage automatique. Translated term: corpus dâapprentissage English sentence: src_segment English term: src_term French translation: tgt_segment Translated term: L.6 Variation Labelling Used by the preprocessing to assign variation categories (Appendix A), withgpt-4.1at temperature 0, after the system message âYou are an expert in scientific terminology and linguistic variation analysis.â It is a revised version of the prompt of Dahan et al. (2026b). Thecontext_sectionblock lists up to three corpus sentences containing the head term and up to three containing the variant, or notes that one of them is not found in the corpus. You are an expert in scientific terminology and linguistic variation, teaching a course on terminology. A student has identified a variant of a head term and needs your help to classify the relationship between the head term and its variant according to the established typology. You receive: - A head term (the preferred form of a technical term) - A variant (an alternative term that qualifies the head term) - Context examples showing how these terms appear in real scientific texts (when available) Your task: Determine the type of variation between the head term and its variant. Classify according to the variation typology. Return ONLY a JSON object with the exact format shown in the examples below. Reference of variation types (in the following example for each label we write headterm/variant ): VG (Graphical): Written form changes without semantic change (acronyms, spelling, symbols) âVG1: Acronym/initialism (European Parliament/EP, non-governmental organisation/NGO, coronavirus disease 2019/COVID-19) âVG2: Symbol/formula (lead/Pb, CO 2 ) âVG3: Spelling change (hyphen, space, case, regional: trade mark/trademark, penalise/penalize, case law/case-law) âVG4: Multiple spelling changes (coronavirus disease 2019/Coronavirus Disease-2019) âVG5: Partial abbreviation of one constituent (apparent polar wander path/APW path) VR (Reduction): Deletion of one or more constituents âVR1: Base reduction âdeletion of a base (automatic translation system/automatic translation) âVR2: Extension reduction âdeletion of qualifier (coronavirus disease 2019/coronavirus disease, post-authorisation safety study/post-authorisation study) âVR3: Other reduction âellipsis, blending, or non-standard truncation (middle-capitalisation company/mid-cap) VE (Expansion): Addition of one or more constituents âVE1: Semantic addition ârelevant but redundant attribute (coronavirus disease 2019/pandemic coronavirus disease 2019) âVE2: Explicit form âsubstitution of a morpheme/lexeme by a fuller form (uninterrupted/without interruption) âVE3: Lexical insertion ânominal, verbal, or paraphrastic addition (dark urine/dark-colored urine) âVE4: Abbreviation development âexpansion of a short form into simple or complex unit (redox/ oxidation-reduction) VL (Lexical): Lexical substitution (synonyms, near-synonyms, translations validated as variant in context) âVL1: Simple unit substitution (residues/waste) âVL2: Complex unit âbase change only (coronavirus disease 2019/coronavirus infection 2019, action for damages/claim for damages) âVL3: Complex unit âextension/attribute change only (coronavirus disease 2019/novel coronavirus disease, maritime law/marine law, action for damages/action for compensation) âVL4: Complex unit âbase AND extension change (coronavirus disease 2019/Wuhan pneumonia, action for damages/claim for compensation) VMS (Morphosyntactic): Structure or inflection changes (word order, articles, inflection, derivation) âVMS1: Constituent order change (coronavirus disease 2019/2019 coronavirus disease) âVMS2: Addition/removal of article (fixation of nitrogen/fixation of the nitrogen) âVMS3: Inflection change (buying cartel/buyerâs cartel, law of tort/law of torts, pass-on of overcharges/passing-on of overcharges) âVMS4: Morphological change (statutory law/statute law, unmargined derivative/non-margined derivative) âVMS5: Structural change âprepositional, adjectival, or syntactic reanalysis (contract of employment/employment contract, asylum application/application for asylum, misconduct in office/official misconduct) CM (Multiple Changes): Combination of two or more of the above types âcombine codes with + (e.g. VG2+ VL1, VR1+VMS5) Important rules: 1) The (head term, variant) pair you receive is ALWAYS a true terminological variation. Your task is to identify which type of variation it is. 2) Count the TYPES of changes, not the number of words: one type âVG/VMS/VR/VE/VL, two or more types âCM. 3) **Acronym rule (critical)**: A variant that looks like an acronym is VG1 ALONE *only* if every letter of the acronym maps to a word actually present in the head term. If the acronym contains one or more letters referring to words ABSENT from the head term, the variant carries an implicit semantic addition and must be classified as CM = VG1+VE1 (or VG1+VL1/VL4 if a word is replaced rather than added). Examples of this hidden-CM pattern: - "topic detection" / "TDT": the final T refers to "tracking" (absent) âCM VG1+VE1 - "phrase" / "P": "P" refers to "prepositional" (absent) âCM VG1+VE1 - "language model" / "LM": only 2 letters but the head term itself is fine âthis IS VG1. Always check letter-by-letter coverage before settling on plain VG1. 4) **Partial abbreviation rule (VG5 vs VG1)**: If part of the head term is preserved verbatim in the variant (e.g. "automatic speech recognition system" / "ASR system" keeps "system"), this is VG5 (partial abbreviation), NOT VG1 (full acronym). 5) **Complex unit rule (VL2/VL3 vs VL1)**: VL1 is reserved for substitution of a *simple* (1-word) head term. If the head term has â„2 words and only the base (or only the modifier) is replaced, classify as VL2 / VL3 âNEVER VL1. 6) For CM, combine codes with + (e.g., "VG1+VE1", "VL2+VR2", "VMS1+VR2+VL3"). 7) Copy the exact JSON format from the examples âno markdown, no extra text. 8) Keep justifications brief (1-2 sentences maximum). When completing the task, follow the examples below: Head term: European Parliament Variant: EP "category": "VG", "subtype": "VG1", "justification": "Acronym formed from the initials of the head term." Head term: carbon dioxide Variant: CO 2 "category": "VG", "subtype": "VG2", "justification": "Chemical formula symbol replacing the complete term." Head term: trade mark Variant: trademark "category": "VG", "subtype": "VG3", "justification": "Orthographic change: removal of space between words." Head term: coronavirus disease 2019 Variant: Coronavirus Disease-2019 "category": "VG", "subtype": "VG4", "justification": "Multiple orthographic changes: capitalization and hyphen added." Head term: apparent polar wander path Variant: APW path "category": "VG", "subtype": "VG5", "justification": "Partial abbreviation of the head term." Head term: automatic translation system Variant: automatic translation "category": "VR", "subtype": "VR1", "justification": "Base reduction: deletion of the base âsystem â." Head term: coronavirus disease 2019 Variant: coronavirus disease "category": "VR", "subtype": "VR2", "justification": "Extension reduction: deletion of the modifier â2019â." Head term: middle-capitalisation company Variant: mid-cap "category": "VR", "subtype": "VR3", "justification": "Partial reduction: deletion of the specification âcompanyâ." Head term: coronavirus disease Variant: pandemic coronavirus disease "category": "VE", "subtype": "VE1", "justification": "Addition of semantic feature: âpandemicâ specifies the scope." Head term: phyllosilicate Variant: sheet silicate "category": "VE", "subtype": "VE2", "justification": "Substitution with explicit descriptive form." Head term: dark urine Variant: dark-colored urine "category": "VE", "subtype": "VE3", "justification": "Lexical insertion: addition of â-coloredâ." Head term: redox Variant: oxidation-reduction "category": "VE", "subtype": "VE4", "justification": "Development of abbreviated form into complete form." Head term: myocardial infarction Variant: heart attack "category": "VL", "subtype": "VL1", "justification": "Lexical substitution of a simple unit by a near-synonym." Head term: stress Variant: constraint "category": "VL", "subtype": "VL1", "justification": "Lexical substitution by translation: borrowed term replaced by native equivalent." Head term: translation system Variant: translation model "category": "VL", "subtype": "VL2", "justification": "Base change: âsystemâ replaced by âmodelâ." Head term: binary classification model Variant: logistic regression model "category": "VL", "subtype": "VL3", "justification": "Extension change: modification of the model type." Head term: coronavirus disease 2019 Variant: Wuhan pneumonia "category": "VL", "subtype": "VL4", "justification": "Base + extension change: complete lexical substitution." Head term: coronavirus disease 2019 Variant: 2019 coronavirus disease "category": "VMS", "subtype": "VMS1", "justification": "Change in constituent order: â2019â moved to the beginning." Head term: fixation of nitrogen Variant: fixation of the nitrogen "category": "VMS", "subtype": "VMS2", "justification": "Addition of definite article âtheâ." Head term: law of tort Variant: law of torts "category": "VMS", "subtype": "VMS3", "justification": "Inflectional change from singular to plural ." Head term: statutory law Variant: statute law "category": "VMS", "subtype": "VMS4", "justification": "Derivation: âstatutoryâ is the adjectival form of the noun âstatuteâ." Head term: contract of employment Variant: employment contract "category": "VMS", "subtype": "VMS5", "justification": "Structural reanalysis: noun+prep phrase replaced by attributive compound." Head term: alpha particle Variant: αray "category": "CM", "subtype": "VG2+VL1", "justification": "Two changes: âalphaâââαâ is graphical ( VG2), âparticleââârayâ is lexical (VL1)." Head term: automatic translation system Variant: translation model "category": "CM", "subtype": "VL2+VR2", "justification": "Lexical change âsystemâââmodelâ (VL2) and reduction of the extension âautomaticâ (VR2)." Head term: topic detection Variant: TDT "category": "CM", "subtype": "VG1+VE1", "justification": "Acronym (VG1) but the final âTâ refers to âtrackingâ, a word absent from the head term, so a semantic feature is implicitly added (VE1)." Head term: automatic speech recognition system Variant: ASR system "category": "VG", "subtype": "VG5", "justification": "âsystemâ is preserved verbatim; only â automatic speech recognitionâ is abbreviated. This is partial abbreviation (VG5), not a full acronym (VG1)." Head term: similarity measure Variant: similarity function "category": "VL", "subtype": "VL2", "justification": "Complex (â„2-word) head term: the base â measureâ is replaced by âfunctionâ while the modifier âsimilarityâ is preserved âVL2 (base change), not VL1." context_section Head term: head_term Variant: variant