Paper deep dive
Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks
Oline Ranum, Edward Fish, Simon Hadfield, Richard Bowden
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/4/2026, 3:48:40 AM
Summary
This paper critiques the reliance on BLEU-4 for evaluating Sign Language Translation (SLT), arguing that it fails to measure true sign language understanding due to its susceptibility to spurious correlations and spoken-language priors. The authors propose an alternative evaluation framework using an open-weight Large Language Model (LLM) Question-Answering (QA) protocol that measures salient content preservation. Validated on spoken-language benchmarks, this QA metric demonstrates higher paraphrase invariance and better alignment with human rankings than BLEU-4. When applied to six SLT models on Phoenix-2014T and CSL-Daily datasets, the QA protocol reveals that gloss-free systems perform similarly within noise margins, while a gloss-supervised system significantly outperforms them, a distinction invisible to BLEU-4.
Entities (14)
Relation Signals (12)
LLM QA Protocol → evaluatedon → PHOENIX-2014T
confidence 98% · In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily
LLM QA Protocol → evaluatedon → CSL-Daily
confidence 98% · In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily
LLM QA Protocol → proposedasalternativeto → BLEU-4
confidence 96% · This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation.
BLEU-4 → criticizedfor → Sign Language Translation
confidence 95% · BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language proficiency.
Gloss-supervised system → outperforms → Gloss-free systems
confidence 92% · while the gloss-supervised system stands 9.3 points higher, a gap invisible to BLEU-4.
Gloss-free systems → performssimilarlyto → Gloss-free systems
confidence 90% · the five gloss-free systems are largely within noise of one another on Phoenix-2014T
C2RL → isinstanceof → Gloss-free systems
confidence 85% · We evaluate six models reporting state-of-the-art results: GFSLT-VLP, FLA-LLM, CiCo, SignCL and C2RL, all gloss-free
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language proficiency. The multimodal, low-resource context of SLT allows models to exploit spurious correlations and spoken-language priors, rather than learning stronger sign representations. In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily, showing that gains in BLEU-4 are not on their own evidence of better sign language understanding. This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation. It aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4. Applied to SLT, this protocol targets content transfer, is more robust to train-test overlap, and gives a different picture of the field: the five gloss-free systems are largely within noise of one another on Phoenix-2014T, while the gloss-supervised system stands 9.3 points higher, a gap invisible to BLEU-4.
Tags
Links
- Source: https://arxiv.org/abs/2609.03734v1
- Canonical: https://arxiv.org/abs/2609.03734v1
Trouble viewing inline? Open PDF directly →
Full Text
103,790 characters extracted from source content.
Expand or collapse full text
Beyond BLEU: A Case for Redefining Sign Language Translation Benchmarks Oline Ranum Edward Fish Simon Hadfield Richard Bowden Affiliation: Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey Affiliation: Guildford, GU2 7XH, Surrey, UK Affiliation: o.ranum, s.hadfield, r.bowden@surrey.ac.uk, ed@tavus.io Abstract BLEU-4 is the standard metric for evaluating sign language translation (SLT), but spoken-language metrics may not adequately reflect sign language proficiency. The multimodal, low-resource context of SLT allows models to exploit spurious correlations and spoken-language priors, rather than learning stronger sign representations. In this paper, we evaluate the relationship between spatio-temporal understanding and BLEU-4 across six SLT models on Phoenix-2014T and CSL-Daily, showing that gains in BLEU-4 are not on their own evidence of better sign language understanding. This work introduces an alternative inspired by language-learning assessment, using an open-weight-LLM QA protocol that measures salient content preservation. It aligns more closely with human rankings and is six to seven times more paraphrase-invariant than BLEU-4. Applied to SLT, this protocol targets content transfer, is more robust to train-test overlap, and gives a different picture of the field: the five gloss-free systems are largely within noise of one another on Phoenix-2014T, while the gloss-supervised system stands 9.39.3 points higher, a gap invisible to BLEU-4. 1 Introduction In language learning, evaluation rarely hinges on the exact reproduction of phrasing. Instead, it asks what was conveyed: who acted, what happened, and under what conditions. Conversely, SLT performance is still measured almost entirely by BLEU [43], an n-gram precision metric that quantifies how closely a system reproduces a reference. This definition of quality may be adequate at scale, but SLT remains a low-resource domain in which datasets cover only narrow linguistic scopes. Consequently, strong spoken language models can achieve high BLEU scores by exploiting regularities in the target-language distribution alone, while paying little attention to the visual language signal. The modalities also structure information differently: signing encode simultaneously and densely, using space for placement, depiction, and constructed action, while many spoken-language grammatical elements are absent (e.g., articles: the, a, an; adpositions: for, to). The resulting source-target mapping is highly non-isomorphic, producing a large space of valid translations and reducing the reliability of BLEU. Metrics that evaluate content preservation may better reflect this reality, tolerating varied linguistic realization while rewarding preservation of salient information. This paper shows that BLEU fails to reliably reflect improvements in sign understanding: BLEU-4 retains substantial scores despite strong input corruptions, and disproportionately rewards components with low correlation to the visual language signal. To address these issues, this paper proposes a question-answering (QA)-based evaluation framework which is akin to how human comprehension is tested. The metric is first validated on spoken-language benchmarks, assessing its sensitivity to semantic shifts and its alignment with human judgment. Then it is applied to the six state-of-the-art SLT models and two sign language benchmarks, demonstrating its scalability and efficiency, and is less influenced by train-test similarities. These findings lead to a revision of current performance rankings. 2 Background & Related Work 2.1 Sign Language Translation Sign language modeling spans recognition (SLR) [42, 46, 34, 44, 71, 47, 45], production (SLP) [62, 57, 61, 54, 56] and translation (SLT) over video or pose inputs paired with gloss or text supervision [9, 24, 28, 10]. Glosses are written labels for individual signs in citation form, which capture a limited portion of the signed content. Early SLT systems predicted gloss sequences before generating text [16, 15, 60, 69, 68, 63]; whereas more recent works have explore gloss-free approaches which map visual inputs directly to text [25, 2, 52, 50, 53] with end-to-end transformers and large pretrained language decoders [55, 38, 18, 66, 4, 51]. Evaluation, grounding and interpretability Datasets used for SLT are currently very limited. The most commonly used dataset Phoenix14T [14] contains just 8,257 German Sign Language videos from 9 interpreters. Another frequent dataset is CSL-Daily [70], which contains 20,654 videos over 2,000 daily-life phrases. Both datasets are recorded in controlled settings, limiting domain variability, signer diversity, and naturalness. We use them here because they are the only SLT benchmarks with gloss annotations, offering a proxy for the signed content in the attribution analysis of Section 3. We evaluate six models reporting state-of-the-art results: GFSLT-VLP [67], FLA-LLM [20], CiCo [21], SignCL [59] and C2RL [19], all gloss-free, together with the single-stream (RGB-only) variant of the gloss-supervised TwoStream [17]; architectures and checkpoints are presented in Appendix A. While BLEU-4 remains the de facto SLT metric, several recent works have challenged its adequacy [22, 10, 35, 49]. SignBLEU [39] extends BLEU to a multi-channel formulation capturing manual and non-manual components, but is restricted to linguistic corpora with rich annotations. For gloss-free evaluation, Hamidullah et al. [31] find that reduced visual dependence correlates with higher hallucination rates, while Alkain et al. [3] show that BLEU is disproportionately influenced by test examples resembling the training distribution. 2.2 QA-based and LLM-based Evaluation LLM-based QA evaluation has proven a viable alternative to established spoken language translation metrics [37, 32, 40]; Kamalloo et al. [36] show that open instruction-tuned LLMs correlate competitively with human judgment. Fernandes et al. [27] extend this to MT with TREQA, an LLM-based QA framework for paragraph-level translation that outperforms or is competitive with chrF, COMET, COMET-QE, GEMBA-MQM, MTEQA. Zhang et al. [64] propose LiTransProQA, a QA framework for literary translation that incorporates professional translators’ assessment criteria. In this work, we apply the LLM-based QA framework to evaluate SLT. We show that it aligns more closely with human ranking than BLEU-4, and that its scores are far less inflated by a test reference’s resemblance to the training data. 3 Preliminaries: Correlations and Attributions of BLEU-4 in SLT In multimodal translation, target-side metrics such as BLEU do not provide insight into whether improvements reflect stronger visual understanding, spoken language modeling or overfitting. This section identifies reward structures and dependencies that limit BLEU’s ability to reflect sign proficiency. 3.1 Robustness to Input Corruption Figure 1 shows BLEU-4 retention scores under spatial and temporal corruption. Despite these interventions, the models often maintain strong performance, suggesting a reliance on target-side priors, memorization, or other spurious correlations (per-condition details in Appendix B). Replacing the visual representation with noise leaves every model at 1.21.2 BLEU-4 points or below, fixing the floor a system reaches while reading nothing. If the score were a true reflection of sign comprehension, masking the signer’s hands and face or shuffling the frames should approach that floor. However, retention stays well above the noise floor throughout: masking a single articulator leaves most models above half their baseline, and even with both articulators masked or the frames shuffled, scores remain several times, up to thirty times, the noise baseline. Masking both hands leaves CiCo, SignCL and GFSLT-VLP at 81%81\%, 83%83\% and 80%80\% of baseline on Phoenix-2014T, where masking the face leaves every model at 5353–68%68\%, and between 6363–86%86\% on CSL-Daily. These gaps matter because of their size relative to the margins in use. On Phoenix-2014T the five gloss-free systems retain 7.3±0.97.3_± 0.9 BLEU-4 points with both articulators masked, close to three times the 2.52.5-point spread that separates the six systems at baseline (Table 5). What a model scores while reading none of the signing therefore exceeds the differences the field uses to rank systems, so a BLEU-4 gain at the current state of the art is not on its own evidence of better sign proficiency. Figure 1: BLEU-4 retention (% of baseline) under input corruptions: spatial masking of hands (H) and face (F), temporal shuffling of frames (Shuf) and Gaussian noise. 3.2 Modalities & Award Structures As introduced in Section 1, the non-isomorphic nature of SLT shapes how evaluation should be contextualized. No current annotation scheme offers a precise account of what is articulated in the sign space, but gloss may offer a first approximation. Figure 2 pairs gloss and text-token POS distributions with the corresponding BLEU-4 attribution: for both Phoenix-2014T and CSL-Daily, averaged over six systems, 37%37\% of BLEU-4 is attributable to function words occurring far less frequently in gloss than in text (German adpositions, 57 gloss vs. 1155 spoken tokens, 3.65 points; Chinese particles, 33 vs. 1279, 1.81 points). 4 Implications for SLT Evaluation Section 3 shows that BLEU-4 often rewards structures with limited or absent correspondence to the visual language signal, by a margin larger than separates current systems. Plausibly, this reflects overfitting driven by narrow linguistic scope. Figure 3 separates output surviving corruption along two axes: whether it preserves the reference’s words or only their order, and whether those words are content or function. This distinction matters because the two categories are not equally recoverable from target-side priors alone: function words occupy a narrow, concentrated vocabulary, while content words are broad and long-tailed (Appendix C.2). Two patterns hold in both datasets under every condition: coverage is retained better than 4-gram precision, and function-word coverage is retained far better than content-word coverage. What survives corruption is thus predominantly what target-side priors can supply, exactly what a metric rewarding explicit function-word forms is most exposed to. We propose instead that reward functions target the preservation of salient content. This framework has three immediate benefits: First, such a metric tolerates paraphrasing. Second, it discounts surface-level structure that survives losing the visual signal. Third, because credit is awarded almost entirely for content, a drop under occlusion reads as content lost rather than correct word order. 4.1 Problem Formulation: Salient Content Preservation Metrics Receptive assessment in early-stage sign language learning [33, 48] suggests a natural way to apply these perspectives: probe comprehension by querying the conveyed content, and assess whether the responses recover the salient information of the input. The above leads to three central questions: 1. How can the preservation of salient content be measured reliably and at scale? 2. Does content-based evaluation reduce the risk of overestimating systems that exploit target-side regularities? 3. How effectively do current models preserve the salient content of the source discourse? The remainder of the paper addresses these in turn. Section 5 implements the first question as an LLM-based QA protocol, evaluated on single-modality benchmarks. Section 6 then transfers the protocol to SLT, addressing the second and third questions. Figure 2: Gloss and text POS distributions with BLEU-4 attribution, for Phoenix-2014T (top) and CSL-Daily (bottom). Figure 3: What survives corruption. Each condition contributes one content and one function marker (mean ± SD over models), placed by retained 4-gram precision (word order) vs. retained reference-word recall (coverage), both as % of baseline. 5 Measuring Content Preservation A metric for salient content preservation should ask what a translation conveys rather than how it is worded, which sets two requirements: invariance to wording at fixed content, and sensitivity to changes in the content itself. This section implements such a protocol as an LLM-driven question generation and answering pipeline, then establishes its behavior on spoken-language benchmarks, where we have human annotations available. 5.1 Evaluation Protocol Task definition Given a human reference translation r, the protocol builds a bank QsQ_s of quality-controlled multiple-choice questions and scores a model prediction s by the fraction of QsQ_s it can answer correctly from s alone (Appendix E), using an open-weight LLM served via vLLM [41]. Implementation details and all prompts are provided in Appendix D. QA-Bank Generation The QA-bank generation is performed in three main phases. To increase coverage, the LLM is prompted to extract all content units from r related to nine categories (entity, action, attribute, quantity, time, location, relation, negation, polarity). In phase 2, the LLM is prompted to generate questions about that unit. |Qs||Q_s| thereby scales with the reference’s semantic density; per-language yields and lemma coverage are in Appendix G. In phase 3, distractors related to each unit and question are generated independently. Four per question, each plausible but incorrect, mutually distinct, non-paraphrastic, and matched in language and register. The five options are deterministically permuted and a language-specific not stated sentinel fixed in a sixth position, which lets the answerer decline rather than guess and floors the metric on non-equivalent content rather than at chance (1/61/6). Quality control Three LLM-executed gates filter the assembled items. A round-trip gate requires the answerer, shown the reference itself, to select the correct option, since an item unanswerable from the reference cannot fairly be asked of a candidate. A world-knowledge probe presents the question with an empty passage and rejects items still answered correctly, removing those answerable from language priors alone (cf. Section 3). An ambiguity gate re-presents the reference, the question, and the five content options in a separate pass, asks which of them the reference states or clearly implies, and rejects the item if more than one does. Candidates are then scored from the passage, question, and options alone. Human verification of every stage is reported in Appendix F. Yield and content coverage Table 1 reports bank sizes and content reach. Question counts scale with sentence complexity as intended: short colloquial subtitles (OpusParcus) yield 3.2 questions per reference against 11.1 for the longer Wikipedia-derived sentences of PAWS-X. Since the metric can only penalise a candidate for content its questions probe, reach is quantified using the POS machinery of Section 3: content coverage is the fraction of a reference’s content lemmas that reappear in the QA items generated from it, and the bank reaches about 88.6%88.6\% on PHOENIX14T and 94.6%94.6\% on CSL-Daily. Benchmark L NrefN_ref N¯Q N_Q Cov. (%) Signed PHOENIX14T 1 641 9.15±0.059.15_± 0.05 88.6±0.1988.6_± 0.19 CSL-Daily 1 1,175 6.78±0.036.78_± 0.03 94.6±0.1194.6_± 0.11 Spoken WMT 2011–2024 5 15,938 12.112.1 92.892.8 OpusParcus 6 19,492 3.23.2 84.984.9 PAWS-X 7 13,862 11.111.1 95.095.0 Table 1: QA yield and coverage pooled over L languages. N¯Q N_Q = average number of questions per reference. Cov. (%) = share of the reference’s content lemmas covered by the QA bank. Subscripts are standard deviations over ten independent bank generations. 5.2 Validation To assess robustness to meaning-shift and wording, the QA pipeline is evaluated on four benchmarks: paraphrase invariance (WMT19-P [29]; WMT21-multiref [1, 30]); sensitivity to shifts in meaning from partial paraphrases (OpusParcus [23]); sensitivity to flipped meaning under fixed wording (PAWS-X [58]); and alignment with human MT-system rankings over the WMT campaigns from 2011 to 2024 [13, 12, 5, 6, 8, 11, 7, 30]. Dataset details are provided in Appendix A, and per-language results are provided in Appendix H. Invariance and Sensitivity Each WMT group collects renderings of one source sentence that differ in wording but not content: one supplies the reference from which questions are generated, the rest are scored against it, and renderings of other sentences supply a floor (Appendix E). Invariance is the gap to that floor divided by the fluctuation within a group: on WMT19 BLEU-4 fluctuates by 7.77.7 points against a 16.516.5-point gap, an SNR of 2.12.1, where QA reaches 12.512.5; on WMT21 the two are 3.13.1 and 23.023.0 (Table 2). The two collections differ in provenance, deliberate paraphrasing against independent translation, so their agreement is not a protocol artefact. Sensitivity is tested on OpusParcus, whose ratings are graded: QA tracks the human scale on average twice as closely as BLEU-4 in every language (Figure 4), rising from 99 to 7272 across the four quality bands where BLEU-4 moves from 99 to 1414 (Figure 5). Figure 4: Correlation of BLEU-4 and QA with human paraphrase-quality ratings on OpusParcus per language (qwen2.5-32b). Pearson r, Spearman ρ and Kendall τ over all graded pairs. Figure 5: Mean metric score by human similarity band on OpusParcus (Q1 = low similarity, Q4 = clean paraphrase), mean ± SD over the six languages. Meaning inversion PAWS-X pairs keep surface overlap high whether or not meaning is inverted, so lexical overlap is uninformative by design. Scoring one sentence of each pair against the other yields two populations of scores, one for true paraphrases and one for meaning-flipped pairs, and we summarise how far apart they lie by ROC-AUC: the probability that a randomly drawn true paraphrase outscores a randomly drawn flipped pair. A metric blind to the inversion sits at 0.50.5 and one that separates the two populations perfectly at 1.01.0. Being threshold-free, it compares BLEU-4 and QA on a common footing despite their different scales. QA performs better than BLEU-4 in all seven languages, 0.830.83 against 0.610.61. That QA still awards flipped English pairs 55.7%55.7\% against 95.4%95.4\% for true paraphrases is expected: PAWS-X alters one or two arguments, so most questions stay answerable. Benchmark Statistic BLEU-4 QA WMT19 (invariance) SNR 2.1 12.5 WMT21 (invariance) SNR 3.1 23.0 OpusParcus (graded) ρ¯ ρ 0.28 0.55 PAWS-X (inversion) AUC 0.61 0.83 Table 2: Robustness: SNR =Δ/σ¯= / σ over meaning-equivalent groups (higher = more invariant); ρ¯ ρ is Spearman correlation with human paraphrase-quality ratings; AUC is P(scoreparaphrase>scoreflipped)P(score$_paraphrase$>score$_flipped$), averaged over seven PAWS-X languages. QA performs better than BLEU-4 across all metrics. Figure 6: Kendall τ-like concordance with human judgments, over WMT campaigns restricted to systems at or below 30 BLEU-4. QA is more aligned with human ranking than BLEU-4 in all but one campaign. Human rankings of MT systems Current text-only MT is near human performance, so its error profiles differ from those of SLT systems; restricting the WMT campaigns to systems averaging below 30 BLEU-4 gives a first approximation. Figure 6 reports Kendall concordance with human ranking. QA matches or exceeds BLEU-4 in every campaign, though the intervals overlap in most; separation is clearest in 2013–2015, where systems are weak enough for content transfer to differ between them but not so weak that both metrics floor. 6 QA-based Evaluation for SLT The spoken-language benchmarks establish that the protocol behaves as a content metric should. This section applies it to the six SLT models, addressing the three evaluation questions of Section 4.1 in order. 6.1 Efficiency and Stability The first question asks how salient content preservation can be measured reliably and at scale. Cost Costs depend on deployment, so each stage is reported separately. Each experiment is conducted on a single RTX 5090 with qwen2.5-32b served under vLLM (Table 3). Bank construction, encompassing content extraction, two QA-generation passes and three quality-control gates, is the expensive stage at 2222–2424 minutes per benchmark. Answering, one short-prompt call per admitted question, is up to an order of magnitude cheaper, putting the recurring cost of evaluating an additional system at roughly 22–33 GPU minutes. Generation NrefN_ref |Q||Q| wall (s) GPU-h Phoenix2014T 642 5,863±325,863_± 32 1,420±111,420_± 11 0.394±0.0030.394_± 0.003 CSL-Daily 1,176 7,967±417,967_± 41 1,319±51,319_± 5 0.367±0.0020.367_± 0.002 Answering calls ans/s wall (s) GPU-h Phoenix2014T 5,863 48.8±0.148.8_± 0.1 120±1120_± 1 0.033±0.0010.033_± 0.001 CSL-Daily 7,967 46.0±0.246.0_± 0.2 173±1173_± 1 0.048±0.0010.048_± 0.001 Table 3: Computational cost. Generation costs are averaged over 10 bank-construction runs, and answering costs over 60 answering passes, per benchmark. |Q||Q| is the number of questions admitted by the quality-control gates, out of 10,030±6110,030_± 61 candidates for Phoenix-2014T and 12,373±4812,373_± 48 for CSL-Daily. Stability A metric is only useful if repeating an experiment yields the same result. Rerunning the generation pipeline does not necessarily reproduce a bank. Roughly three-quarters of the admitted items recur between two banks, partly by design, since the second generation pass samples at a non-zero temperature (Appendix I). However, that variability does not carry through to the scores. Table 4 reports ten end-to-end repeats, each regenerating content units, questions and answers against fixed system outputs. Across those repeats a system’s QA accuracy moves by σ=0.17σ=0.17–0.340.34 percentage points on Phoenix-2014T and 0.110.11–0.260.26 on CSL-Daily. Holding the bank fixed and bootstrap-resampling the test instances instead moves it by 2.42.4–2.72.7 and 1.41.4–2.02.0 points (half-width of the 95%95\% interval; Appendix E.2), between seven and sixteen times more for every system. Which questions get asked therefore matters far less than which sentences the test set happens to contain. The banks are correspondingly stable: their size varies by about half a percent, and every run covers at least 99.8%99.8\% of references, varying by at most one reference. A QA gap below roughly half a point is therefore within run-to-run noise and should not be read as a difference between systems. Quantity mean±σ range CI/2 PHOENIX-2014T, QA % per system C2RL 53.05±0.3253.05_± 0.32 52.56–53.47 2.65 CiCo 56.47±0.3456.47_± 0.34 56.04–57.03 2.57 Fla-LLM 55.23±0.1755.23_± 0.17 54.99–55.48 2.53 GFSLT 53.93±0.2453.93_± 0.24 53.51–54.29 2.50 SignCL 54.28±0.1754.28_± 0.17 53.93–54.54 2.60 SingleStream 65.85±0.2765.85_± 0.27 65.28–66.22 2.39 bank size |Q||Q| 5,863±325,863_± 32 5,803–5,906 refs. covered 641.0±0.0641.0_± 0.0 641–641 CSL-Daily, QA % per system C2RL 11.55±0.1211.55_± 0.12 11.35–11.72 1.38 CiCo 26.89±0.1426.89_± 0.14 26.65–27.13 1.73 Fla-LLM 40.95±0.2140.95_± 0.21 40.57–41.26 2.03 GFSLT 22.19±0.1122.19_± 0.11 22.00–22.32 1.78 SignCL 21.09±0.1721.09_± 0.17 20.85–21.45 1.70 SingleStream 61.42±0.2661.42_± 0.26 60.93–61.79 2.01 bank size |Q||Q| 7,967±417,967_± 41 7,893–8,037 refs. covered 1,175.2±0.41,175.2_± 0.4 1,175–1,176 Table 4: Accuracies over ten runs, each with content units, questions, and system answers regenerated. σ is variation across question banks; CI/2 is the mean half-width of the within-bank 95% bootstrap interval(Appendix E.2). 6.2 Exposure to Target-Side Regularities The second question asks whether content-based evaluation reduces the risk of overestimating systems that exploit target-side regularities. This is influenced by what the metric awards, and how much it rewards test examples that resemble the training data. Award structures Under the content/function POS division of Figure 2 (tagging and attribution in Appendix C), BLEU-4 derives 37% of its score from function words, whereas QA spends 95%95\% on content for Phoenix-2014T and over 99%99\% for CSL-Daily. Function words are also what best survives corruption. Replacing the visual signal with noise leaves function-word recall at 69%69\% of baseline on Phoenix-2014T and 78%78\% on CSL-Daily, while content-word recall falls to 77–16%16\% (Figure 3). A third of BLEU-4 is therefore committed to a word class the model can still produce once the signal the metric is meant to measure is gone. This opens a channel for improvement without comprehension, where a system can raise BLEU-4 by modeling the target language better in the very category the signed source largely does not encode, indistinguishably from a gain earned by reading the signing. Nor is the channel narrow, corresponding to approximately 88 BLEU-4 points on Phoenix-2014T against the 2.52.5 points that separate the six systems. QA leaves almost none open: with 95%95\% or more of its credit on content. Exposure to train–test overlap Alkain et al. [3] show that BLEU scores a test sentence higher when it resembles the training data, crediting recall of the training targets as though it were translation quality, and doing so most where the training distribution is narrowest, which is the situation in SLT. Whether the QA score depends on that resemblance is therefore worth measuring. Following Alkain et al. [3], we characterize each test instance by a training-likeness ℓi=maxt∈Tdsim(ri,t) _i= _t∈ T_dsim(r_i,t), the character-level similarity between its reference and the closest training target, so ℓ=100 =100 means the reference occurs verbatim in training. Within a sliding window over ℓ we regress the per-instance score on ℓ at fixed reference length, and express the slope as a percentage of that model’s own mean score: a value of 1010 says one further likeness point buys ten percent of the score the system reports, and 00 says the metric is indifferent to how training-like a sentence is. Normalizing this way is what makes the two metrics comparable. On Phoenix-2014T BLEU-4 averages about 2222 and QA about 5656, so the same raw slope would mean very different things for each; dividing by each metric’s own mean removes that difference in scale and lets both be read on a single axis. Window widths, the treatment of exact copies and a cumulative view are in Appendix J. On Phoenix-2014T (Figure 7) BLEU-4 sits at 3.9%3.9\% per likeness point over most of the range, below ℓ=80 =80, then rises eightfold to 32%32\% for the most training-like sentences, where QA moves only from 2.4%2.4\% to 6.5%6.5\%. The exact copies, excluded from the curves, show the same asymmetry in levels rather than slopes: on the 3131 references that occur verbatim in training, sentence BLEU-4 averages 52.552.5 against 21.421.4 elsewhere, a 2.42.4-fold jump, where QA averages 79.779.7 against 57.157.1, a 1.41.4-fold one. CSL-Daily reproduces the shape more weakly: over the top quarter of its likeness range BLEU-4 averages 8.4%8.4\% per likeness point against 2.5%2.5\% for QA, a 3.43.4-fold gap where the Phoenix-2014T figures of 23.5%23.5\% and 3.6%3.6\% give 6.66.6-fold. The direction is the same in all twelve cases: as a reference comes to resemble something the system saw in training, BLEU-4 accelerates and QA barely responds. Figure 7: Local sensitivity of BLEU-4 and QA to training-likeness ℓ , at fixed reference length. Curves are the mean over M=6M=6 models; bands span their min–max. Exact duplicates (ℓ=100 =100) are excluded as a point mass, and, on Phoenix-2014T, because their references are shorter than neighbouring points. BLEU-1 BLEU-4 QA % Rank: BLEU-4 → QA Model Reported Ours Reported Ours Ours vs Reported vs Ours PHOENIX-2014T C2RL 52.8 46.2 26.8 21.2±2.221.2_± 2.2 52.6±2.652.6_± 2.6 2 → 6 (−4-4) 5 → 6 (−1-1) CiCo – 47.6 – 23.6±2.123.6_± 2.1 56.2±2.656.2_± 2.6 3 → 2 (+1+1) 1 → 2 (−1-1) Fla-LLM 46.3 46.6 23.1 21.1±2.121.1_± 2.1 55.0±2.555.0_± 2.5 4 → 3 (+1+1) 6 → 3 (+3+3) GFSLT 43.7 46.9 21.4 22.2±2.322.2_± 2.3 53.5±2.653.5_± 2.6 6 → 5 (+1+1) 4 → 5 (−1-1) SignCL 49.8 46.3 22.7 22.7±2.222.7_± 2.2 53.8±2.653.8_± 2.6 5 → 4 (+1+1) 3 → 4 (−1-1) SingleStream 54.0 46.0 27.6 23.5±2.123.5_± 2.1 65.5±2.365.5_± 2.3 1 → 1 (±0± 0) 2 → 1 (+1+1) CSL-Daily C2RL 49.3 30.6 21.6 7.0±0.7 07.0_± 0.7 11.6±1.311.6_± 1.3 2 → 6 (−4-4) 6 → 6 (±0± 0) CiCo – 39.9 – 12.0±0.812.0_± 0.8 27.1±1.827.1_± 1.8 5 → 3 (+2+2) 3 → 3 (±0± 0) Fla-LLM 37.1 47.8 14.2 16.9±1.016.9_± 1.0 41.2±2.141.2_± 2.1 4 → 2 (+2+2) 2 → 2 (±0± 0) GFSLT 39.4 37.4 11.0 10.9±0.910.9_± 0.9 22.7±1.722.7_± 1.7 6 → 4 (+2+2) 4 → 4 (±0± 0) SignCL 47.5 37.3 16.2 10.9±0.810.9_± 0.8 21.2±1.721.2_± 1.7 3 → 5 (−2-2) 5 → 5 (±0± 0) SingleStream 55.4 58.8 25.8 28.2±1.428.2_± 1.4 61.6±2.161.6_± 2.1 1 → 1 (±0± 0) 1 → 1 (±0± 0) Table 5: Reported vs. reproduced performance with QA content preservation. 95% intervals are bootstrap over test instances (Appendix E.2). QA is the pooled fraction of admitted questions answered correctly, from a bank generated and answered by qwen2.5-32b. Rank columns order all six models by BLEU-4 vs. QA: vs Reported uses published BLEU-4 where available (our reproduction otherwise); vs Ours uses our reproduction throughout. Probing sign grounding One further perspective on the second question is worth detailing before we conclude. Figure 1 showed BLEU-4 surviving well above each model’s noise floor under severe corruption of the input, but it cannot say what survived. A retention percentage reflects content the corruption removed, word order it broke, and target-side structure it never touched, and the two findings above say the third term dominates. What best withstands losing the visual signal is the same narrow, closed vocabulary that carries a third of the score and that training-like references inflate. Any metric resting on that scaffolding is inherently limited as a reflection of sign comprehension. Content words are harder to guess given their long-tailed distribution, but the share that survives pure noise shows they remain partly recoverable. A content-oriented metric is therefore not a fully closed channel, but a more transparent one. By stripping away the structural component that each modality expresses differently, the source of the non-isomorphism discussed in Section 1, its movement under corruption can be read as grounding. What survives then reflects the content transferred without the masked channel. Neither function-word scaffolding nor train-test overlap inflates QA, whose slope in training-likeness stays flat where BLEU-4 accelerates (Figure 7), so its retention can be read directly: masking the hands leaves 6464–8080% of content transfer intact on Phoenix-2014T but only 44–1616% on CSL-Daily, while masking the face leaves 6262–7070% and 4040–6868% respectively (Appendix B.4). In answer to the second evaluation question, content-based evaluation reduces that risk because it does not reward target-side regularities the way BLEU-4 does, turning a corruption experiment from an uninterpretable number into a per-channel statement about how far a system’s output depends on the signing. 6.3 Saliency Oriented System Ranking The third question asks how effectively models preserve the source discourse’s salient content. QA accuracy ranges from 52.652.6 to 65.5%65.5\% on Phoenix-2014T and from 11.611.6 to 61.6%61.6\% on CSL-Daily (Table 5). However, CSL-Daily’s figures reflect total failures as much as partial transfer (Appendix K). Relative to the published scores the ranking reorders substantially, but some of that is attributable to the reproduction gap, which is itself informative. The five gloss-free systems are evaluated here from the checkpoints of Sincan et al. [51], which brings them onto a common data pipeline, input resolution and evaluation protocol, and all six are decoded and scored through a single pipeline of our own (Appendix A). The Reported column instead aggregates six independent training and evaluation setups, so we take the Ours ranking to be the like-for-like comparison. On identical predictions the two metrics agree closely, no system changes rank on CSL-Daily, and five of six move by at most one position on Phoenix-2014T, since transferring more content also reproduces more reference surface. The exception is Fla-LLM, sixth on BLEU-4 and third on QA. The consequential finding is separation, not reranking. On Phoenix-2014T the five gloss-free systems cannot be told apart: all ten confidence-interval pairs overlap across a span of just 3.63.6 QA points, so none shows evidence of preserving more salient content than another, on the field’s most reported benchmark. CSL-Daily separates the same five over a span eight times wider. SingleStream, the one gloss-supervised system, stands apart from all five on both benchmarks (9.39.3 and 20.420.4 points), a gap invisible to BLEU-4, which ranks it second on Phoenix-2014T. In answer to the third evaluation question, preservation is partial at best: about two thirds of queried content for the gloss-supervised system on Phoenix-2014T, just above half for the gloss-free systems, and as little as a tenth on CSL-Daily. 7 Conclusion BLEU-4 rewards words a spoken-language decoder can supply without reading the signing: it survives occlusion of the articulators, commits a third of its credit to function words with few counterparts in the source, and favours systems that reproduce training-like targets. Each exceeds the 2.52.5-point spread separating the six systems we reproduce, so BLEU-4 differences at the current state of the art are not evidence of better sign understanding. Asking instead what a translation conveys answers all three questions of Section 4.1. The QA pipeline measures salient content preservation reliably and at scale, stable to within half a point across reevaluations at 22–33 GPU-minutes per additional system. It resists what inflates BLEU-4: 95%95\% of its credit falls on content, it is markedly more paraphrase-invariant and inversion-sensitive, and on the least training-like tenth of Phoenix-2014T it retains 57%57\% of its full-set value where BLEU-4 retains 12%12\%. Measured this way the field looks different: models recover just above half the queried content, the gloss-supervised system leads by 9.39.3 QA points where BLEU-4 ranks it second, and the five gloss-free systems cannot be told apart. As SLT matures, content transfer will increasingly saturate and be replaced by more detailed benchmarks. Until then, we believe the field should focus on whether models are capable of recovering the salient, grounded content of the discourse. References [1] F. Akhbardeh, A. Arkhangorodsky, M. Biesialska, O. Bojar, R. Chatterjee, V. Chaudhary, M. R. Costa-jussa, C. España-Bonet, A. Fan, C. Federmann, M. Freitag, Y. Graham, R. Grundkiewicz, B. Haddow, L. Harter, K. Heafield, C. M. Homan, M. Huck, K. Amponsah-Kaakyire, J. Kasai, D. Khashabi, K. Knight, T. Kocmi, P. Koehn, N. Lourie, C. Monz, M. Morishita, M. Nagata, A. Nagesh, T. Nakazawa, M. Negri, S. Pal, A. A. Tapo, M. Turchi, V. Vydrin, and M. Zampieri (2021) Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth Conference on Machine Translation, Cited by: §5.2. [2] S. Albanie, G. Varol, L. Momeni, H. Bull, T. Afouras, H. Chowdhury, N. Fox, B. Woll, R. Cooper, A. McParland, and A. Zisserman (2021) BOBSL: BBC-Oxford British Sign Language Dataset. In arXiv, Cited by: §2.1. [3] B. Alkain, A. Núñez-Marcos, C. Escolano, L. Docío-Fernández, O. Perez-de-Viñaspre, and G. Labaka (2026) Critical analysis of datasets for sign language translation. Frontiers in Artificial Intelligence. Cited by: §2.1, §6.2, §6.2. [4] S. Asasi, M. I. Lakhal, O. M. Sincan, and R. Bowden (2025) Beyond gloss: a hand-centric framework for gloss-free sign language translation. In arXiv preprint, Cited by: §2.1. [5] O. Bojar, C. Buck, C. Callison-Burch, C. Federmann, B. Haddow, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia (2013) Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, Cited by: §5.2. [6] O. Bojar, C. Buck, C. Federmann, B. Haddow, P. Koehn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna (2014) Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, Cited by: §5.2. [7] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, S. Huang, M. Huck, P. Koehn, Q. Liu, V. Logacheva, C. Monz, M. Negri, M. Post, R. Rubino, L. Specia, and M. Turchi (2017) Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, Cited by: §5.2. [8] O. Bojar, R. Chatterjee, C. Federmann, B. Haddow, M. Huck, C. Hokamp, P. Koehn, V. Logacheva, C. Monz, M. Negri, M. Post, C. Scarton, L. Specia, and M. Turchi (2015) Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, Cited by: §5.2. [9] D. Bragg, O. Koller, M. Bellard, et al. (2019) Sign language recognition, generation, and translation: an interdisciplinary perspective. In ASSETS, Cited by: §2.1. [10] M. Brown, O. Ranum, E. Fish, H. Proctor, B. Woll, R. Bowden, and K. Cormier (2026) SignGPT and the visual language toolkit. In Proceedings of the LREC 2026 12th Workshop on the Representation and Processing of Sign Languages: Language in Motion, Cited by: §2.1, §2.1. [11] C. Buck and P. Koehn (2016) Findings of the WMT 2016 bilingual document alignment shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, Cited by: §5.2. [12] C. Callison-Burch, P. Koehn, C. Monz, M. Post, R. Soricut, and L. Specia (2012) Findings of the 2012 workshop on statistical machine translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, Cited by: §5.2. [13] C. Callison-Burch, P. Koehn, C. Monz, and O. Zaidan (2011) Findings of the 2011 workshop on statistical machine translation. In Proceedings of the Sixth Workshop on Statistical Machine Translation, Cited by: §5.2. [14] N. C. Camgoz, S. Hadfield, O. Koller, H. Ney, and R. Bowden (2018) Neural sign language translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §A.2, §2.1. [15] Y. Chen, F. Wei, X. Sun, et al. (2023) A simple multi-modality transfer learning baseline for sign language translation. In CVPR, Cited by: §2.1. [16] Y. Chen, R. Zuo, F. Wei, et al. (2023) Two-stream network for sign language recognition and translation. In NeurIPS, Cited by: §2.1. [17] Y. Chen, R. Zuo, F. Wei, Y. Wu, S. Liu, and B. Mak (2022) Two-stream network for sign language recognition and translation. In Proceedings of the 36th International Conference on Neural Information Processing Systems, Cited by: §A.2, §A.2, §2.1. [18] Z. Chen, B. Zhou, Y. Huang, et al. (2025) C2RL: content and context representation learning for gloss-free sign language translation and retrieval. IEEE TCSVT. Cited by: §2.1. [19] Z. Chen, B. Zhou, Y. Huang, J. Wan, Y. Hu, H. Shi, Y. Liang, Z. Lei, and D. Zhang (2024) C2RL: content and context representation learning for gloss-free sign language translation and retrieval. IEEE Transactions on Circuits and Systems for Video Technology. Cited by: §A.2, §2.1. [20] Z. Chen, B. Zhou, J. Li, J. Wan, Z. Lei, N. Jiang, Q. Lu, and G. Zhao (2024) Factorized learning assisted with large language model for gloss-free sign language translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), Cited by: §A.2, §2.1. [21] Y. Cheng, F. Wei, J. Bao, D. Chen, and W. Zhang (2023) CiCo: domain-aware sign language retrieval via cross-lingual contrastive learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §A.2, §2.1. [22] O. Cory, M. Ivashechkin, K. Sahin, O. Ranum, J. Low, E. Fish, A. Pelykh, O. Mercanoglu Sincan, and R. Bowden (2026) BackTranslation2.0: a linguistically motivated metric to assess sign language production. arXiv preprint arXiv:2606.28673. Cited by: §2.1. [23] M. Creutz (2018) Open subtitles paraphrase corpus for six languages. In Proceedings of the 11th edition of the Language Resources and Evaluation Conference (LREC 2018), Cited by: §A.1, §5.2. [24] A. Desai, M. De Meulder, J. A. Hochgesang, A. Kocab, and A. X. Lu (2024) Systemic biases in sign language ai research: a deaf-led call to reevaluate research agendas. LREC-COLING Workshop. Cited by: §2.1. [25] A. Duarte, S. Palaskar, L. Ventura, D. Ghadiyaram, K. DeHaan, F. Metze, J. Torres, and X. Giro-i-Nieto (2021) How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language. In Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.1. [26] A. FASHN (2024) FASHN human parser: segformer for fashion human parsing. Hugging Face. External Links: Link Cited by: §B.2. [27] P. Fernandes, S. Agrawal, E. Zaranis, A. Martins, and G. Neubig (2025) Do LLMs understand your translations? evaluating paragraph-level MT with question answering. In Second Conference on Language Modeling, Cited by: §2.2. [28] N. Fox, B. Woll, and K. Cormier (2023) Best practices for sign language technology research. In Universal Access in the Information Society, Cited by: §2.1. [29] M. Freitag, G. Foster, D. Grangier, and C. Cherry (2020) Human-paraphrased references improve neural machine translation. In Proceedings of the Fifth Conference on Machine Translation, Cited by: §A.1, §5.2. [30] M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan, and W. Macherey (2021) Experts, errors, and context: a large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics. Cited by: §5.2. [31] Y. Hamidullah, K. D. Chowdhury, Y. A. Ghussin, S. Yazdani, C. Oguz, J. van Genabith, and C. España-Bonet (2026) Grounding or guessing? visual signals for detecting hallucinations in sign language translation. arXiv. Cited by: §2.1. [32] H. Han, M. Carpuat, and J. Boyd-Graber (2022) SimQA: detecting simultaneous MT errors through word-by-word question answering. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.2. [33] P. C. Hauser, R. Paludneviciene, W. Riddle, K. B. Kurz, K. Emmorey, and J. Contreras (2016) American sign language comprehension test: a tool for sign language researchers.. Journal of deaf studies and deaf education. Cited by: §4.1. [34] J. Huang, W. Zhou, Q. Zhang, H. Li, and W. Li (2018) Video-based sign language recognition without temporal segmentation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, Cited by: §2.1. [35] Z. Jiang, C. Leong, A. Moryossef, O. Cory, M. Ivashechkin, N. Tarigopula, B. Zhang, A. Göhring, A. Rios, R. Sennrich, and S. Ebling (2025) Meaningful pose-based sign language evaluation. In Proceedings of the Tenth Conference on Machine Translation, Cited by: §2.1. [36] E. Kamalloo, S. Upadhyay, and J. Lin (2024) Towards robust qa evaluation via open llms. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §2.2. [37] D. Ki, K. Duh, and M. Carpuat (2025) AskQE: question answering as automatic evaluation for machine translation. In Findings of the Association for Computational Linguistics: ACL 2025, Cited by: §2.2. [38] J. Kim, H. Jeon, J. Bae, and H. Y. Kim (2025) Leveraging the power of mllms for gloss-free sign language translation. In ICCV, Cited by: §2.1. [39] J. Kim, M. Huerta-Enochian, C. Ko, and D. H. Lee (2024) SignBLEU: automatic evaluation of multi-channel sign language translation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), p. 14796–14811. Cited by: §2.1. [40] M. Krubiński, E. Ghadery, M. Moens, and P. Pecina (2021) Just ask! evaluating machine translation by asking and answering questions. In Proceedings of the Sixth Conference on Machine Translation, Cited by: §2.2. [41] W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica (2023) Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, Cited by: §5.1. [42] D. Li, C. Rodriguez, X. Yu, and H. Li (2020) Word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. In The IEEE Winter Conference on Applications of Computer Vision, Cited by: §2.1. [43] K. Papineni, S. Roukos, T. Ward, and W. J. Zhu (2002) BLEU: a method for automatic evaluation of machine translation. ACL. Cited by: §1. [44] O. Ranum, S. Hadfield, and R. Bowden (2026) What’s the point? spatial grammar & index resolution for sign language processing. arXiv. Cited by: §2.1. [45] O. Ranum, D. Wessels, G. Otterspeer, E. J. Bekkers, F. Roelofsen, and J. I. Andersen (2024) The NGT200 dataset - geometric multi-view isolated sign recognition. In ICML 2024 Workshop on Geometry-grounded Representation Learning and Generative Modeling, Cited by: §2.1. [46] R. Rastgoo, K. Kiani, and S. Escalera (2021) Sign language recognition: a deep survey. Expert Systems with Applications. Cited by: §2.1. [47] C. Raude, K. R. Prajwal, L. Momeni, H. Bull, S. Albanie, A. Zisserman, and G. Varol (2024) A tale of two languages: large-vocabulary continuous sign language recognition from spoken language supervision. arXiv. Cited by: §2.1. [48] P. Rosenburg, A. M. Lieberman, N. Caselli, and R. Hoffmeister (2020) The development and evaluation of a new asl text comprehension task. Frontiers in Communication. Cited by: §4.1. [49] B. Saunders, N. C. Camgöz, and R. Bowden (2020) Adversarial training for multi-channel sign language production. The 31st British Machine Vision Virtual Conference. Cited by: §2.1. [50] B. Shi, D. Brentari, G. Shakhnarovich, and K. Livescu (2022) Open-domain sign language translation learned from online video. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.1. [51] O. M. Sincan, J. H. Low, S. Asasi, and R. Bowden (2025) Gloss-free sign language translation: an unbiased evaluation of progress in the field. Computer Vision and Image Understanding. Cited by: §A.2, §2.1, §6.3. [52] G. Tanzer and B. Zhang (2025) YouTube-SL-25: a large-scale, open-domain multilingual sign language parallel corpus. In The Thirteenth International Conference on Learning Representations, Cited by: §2.1. [53] D. Uthus, G. Tanzer, and M. Georg (2023) YouTube-asl: a large-scale, open-domain american sign language-english parallel corpus. In Proceedings of the 37th International Conference on Neural Information Processing Systems, Cited by: §2.1. [54] C. Wang, Z. Deng, Z. Jiang, Y. Yin, F. Shen, Z. Cheng, S. Ge, S. Gan, and Q. Gu (2025) Advanced sign language video generation with compressed and quantized multi-condition tokenization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §2.1. [55] R. Wong, N. C. Camgoz, and R. Bowden (2024) Sign2GPT: leveraging large language models for gloss-free sign language translation. In ICLR, Cited by: §2.1. [56] Y. Xia, Q. Zhan, X. Luo, X. Shi, and Y. Wang (2026) SignMask: structure-aware masked modeling for holistic 3d sign language production. ACM Trans. Multimedia Comput. Commun. Appl.. Cited by: §2.1. [57] P. Xie, T. Peng, Y. Du, and Q. Zhang (2023) Sign language production with latent motion transformer. 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). Cited by: §2.1. [58] Y. Yang, Y. Zhang, C. Tar, and J. Baldridge (2019) PAWS-X: a cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Cited by: §A.1, §5.2. [59] J. Ye, X. Wang, W. Jiao, J. Liang, and H. Xiong (2024) Improving gloss-free sign language translation by reducing representation density. In Proceedings of the 38th International Conference on Neural Information Processing Systems, Cited by: §A.2, §2.1. [60] A. Yin, Z. Zhao, J. Liu, et al. (2021) SimulSLT: end-to-end simultaneous sign language translation. In ACM M, Cited by: §2.1. [61] A. Yin, H. Li, K. Shen, S. Tang, and Y. Zhuang (2024) T2S-GPT: dynamic vector quantization for autoregressive sign language production from text. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Cited by: §2.1. [62] Z. Yu, S. Huang, Y. Cheng, and T. Birdal (2024) SignAvatars: a large-scale 3d sign language holistic motion dataset and benchmark. In Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part V, Cited by: §2.1. [63] B. Zhang, M. Müller, and R. Sennrich (2023) SLTUNET: a simple unified model for sign language translation. In ICLR, Cited by: §2.1. [64] R. Zhang, W. Zhao, L. Macken, and S. Eger (2025) LiTransProQA: an LLM-based literary translation evaluation metric with professional question answering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Cited by: §2.2. [65] Y. Zhang, J. Baldridge, and L. He (2019) PAWS: paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Cited by: §A.1. [66] B. Zhou, Z. Chen, J. Wan, et al. (2023) Gloss-free sign language translation: improving from visual-language pretraining. In ICCV, Cited by: §2.1. [67] B. Zhou, Z. Chen, A. Clapes, J. Wan, Y. Liang, S. Escalera, Z. Lei, and D. Zhang (2023) Gloss-free Sign Language Translation: Improving from Visual-Language Pretraining . In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), Cited by: §A.2, §2.1. [68] H. Zhou, W. Zhou, and H. Li (2022) Spatial-temporal multi-cue network for sign language recognition and translation. IEEE Transactions on Multimedia. Cited by: §2.1. [69] H. Zhou, W. Zhou, W. Qi, et al. (2021) Improving sign language translation with monolingual data by sign back-translation. In CVPR, Cited by: §2.1. [70] H. Zhou, W. Zhou, W. Qi, J. Pu, and H. Li (2021) Improving Sign Language Translation with Monolingual Data by Sign Back-Translation . In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §A.2, §2.1. [71] R. Zuo and B. Mak (2022) C2SLR: consistency-enhanced continuous sign language recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2.1. 8 Supplementary Material Appendix A Datasets and Reproduced Systems A.1 Spoken-language benchmarks WMT multi-reference collections. The WMT datasets are the parallel corpora and human judgements released with the annual Conference on Machine Translation shared tasks, each denoted WMTxx for the year of the task. Most releases provide a single reference per source segment and are therefore unusable for paraphrase invariance, which needs several equally valid renderings of the same content. WMT19-paraphrased [29] augments the 1997 newstest2019 en–de segments with six line-aligned human reference sets produced under different paraphrasing instructions and quality-control regimes. Every set is a faithful rendering of the same source, so content is constant by construction and only wording varies. After deduplicating identical renderings all 1997 segments are retained, averaging 3.463.46 distinct variants. WMT21-multiref uses the newstest2021 en–de references A, C and D, three independently produced human translations of each segment, and recovers the ref-B text from the raw MQM judgement file for segments that were rated. Keeping segments with at least three distinct variants yields 991 groups at 3.423.42 variants each. Within each group one rendering is designated the reference (by default the variant with the highest mean BLEU to the others); the remaining in-group renderings supply the paraphrase scores and two randomly drawn out-of-group renderings supply the floor (Appendix E). WMT human concordance. The system-ranking analysis of Section 5 uses the human judgements released with the campaigns from 2011 to 2024, under three successive label regimes: relative ranking (R, 2011–2016), DA-derived relative ranking (daRR, 2017–2019), and MQM (2020–2024), from which pairwise preferences are derived at a severity margin of 1.01.0. Metric-tuned MBR submissions are excluded at every call site. OpusParcus. The OpenSubtitles Paraphrase Corpus [23] covers six European languages (en, de, fi, fr, ru, sv). Extracted from OpenSubtitles2016 film and television subtitles, it reflects informal, colloquial and often elliptical language, and its sentences are short, so a metric has few tokens to work with. Its development and test sets carry a graded human annotation score from 1 (not a paraphrase) to 4 (clean paraphrase), obtained by averaging two independent annotators; the label is therefore half-integer valued and encodes annotator disagreement rather than discarding it. We pool the validation and test splits and evaluate all 20,378 pairs. The binned analysis groups the scale into Q1 [1,1.5][1,1.5], Q2 [2,2.5][2,2.5], Q3 [3,3.5][3,3.5] and Q4 =4.0=4.0. PAWS-X. PAWS-X [58] is the cross-lingual extension of PAWS [65] (Paraphrase Adversaries from Word Scrambling), covering seven languages (en, de, fr, es, zh, ja, ko) with 2,000 test pairs each. Pairs are adversarially constructed so surface overlap is high in both classes, with negatives generated by word scrambling and argument swaps. We use the test split and the binary label (1 = true paraphrase, 0 = meaning-flipped) as ground truth. A.2 Sign language benchmarks Phoenix-2014T [14] covers German Sign Language weather forecasting, interpreted for the PHOENIX broadcaster by nine signers. Its 8,2578,257 segments split 7,096/519/6427,096/519/642 across train, dev and test. The test split carries 7,8167,816 German tokens over 1,0011,001 word types, averaging 12.212.2 tokens per sentence, against 4,2644,264 gloss tokens over 411411 types. The vocabulary is narrow and nearly closed with respect to training: the 2,8872,887 training word types cover all but 5959 test types, a token OOV rate of 0.77%0.77\%. References are distributed lowercased and contain no punctuation or digits, and 630630 of the 642642 are distinct. CSL-Daily [70] covers Chinese Sign Language on daily-life topics, signed by ten signers, with 20,65420,654 segments split 18,401/1,077/1,17618,401/1,077/1,176 (one training row carries an empty target and is dropped). Chinese is written unspaced, so counts are per character: the test split has 19,23919,239 characters over 1,3461,346 types, 16.416.4 per sentence, against 9,0029,002 gloss tokens over 1,3451,345 types drawn from a training gloss vocabulary of exactly 2,0002,000. Character OOV is 0.31%0.31\%. Every reference retains full-width punctuation, and because each phrase is signed by several signers the split holds only 798798 distinct sentences among its 1,1761,176 references. Reference preprocessing. BLEU-4 is corpus BLEU-4 under the tokeniser policy of Appendix E, so Phoenix-2014T references are scored exactly as distributed. On CSL-Daily the systems emit character-spaced Chinese while the references are unspaced, which drives BLEU-4 to zero if left uncorrected; we therefore strip whitespace adjacent to any CJK character and fold full-width punctuation to ASCII in both hypothesis and reference before scoring. The QA protocol reads the same references with no further normalisation. Reproduced models. The six systems of Section 3, condensed from Section 2 for space. GFSLT-VLP [67] pretrains a visual encoder and text decoder via contrastive vision-language alignment and masked language modeling, then fine-tunes an encoder-decoder for translation. FLA-LLM [20] pretrains its visual encoder with a lightweight translation decoder, then freezes it and pairs it with a larger 12-layer MBart. CiCo [21] performs sign-video-text retrieval via cross-lingual contrastive learning, capturing fine-grained sign-to-word mappings in a joint embedding space. SignCL [59] applies a contrastive loss between adjacent frames to counter representation density and better separate distinct signs. C2RL [19] combines visual-text alignment with a lightweight translation objective in a multi-task pretraining framework. TwoStream [17] combines RGB video and keypoint sequences through lateral connections, a sign pyramid network, and self-distillation to reduce visual redundancy; this work uses only its single-stream (RGB-only) variant, the one gloss-supervised system in the set. Checkpoints. Every system is reproduced from released weights rather than retrained, so Table 5 reflects the published models and not our own training runs. The five gloss-free systems use the checkpoints distributed with the SLTBaselines benchmark of Sincan et al. [51], which re-evaluates GFSLT-VLP, FLA-LLM, CiCo, SignCL and C2RL under a common data pipeline, input resolution and evaluation protocol. We take one checkpoint per model and dataset, together with the trimmed mBART tokeniser and visual-encoder configuration shipped in the same bundle, since the released weights are vocabulary-consistent only with that pair. The single-stream variant of TwoStream [17] is not part of that benchmark and comes from the authors’ own TwoStreamNetwork release. Drawing the gloss-free systems from one reproduction source is what makes them comparable here: their originally reported scores were obtained under six different training and evaluation setups, which is the discrepancy Sincan et al. [51] set out to quantify and which the gap between the Reported and Ours columns of Table 5 reflects. Appendix B Input Corruption: Protocol and Retention All corruptions are applied at inference time to the reproduced checkpoints. Each condition is decoded over the full test split of both datasets, and retention is reported as a percentage of that model’s own undistorted baseline, so architectures with different absolute scores remain comparable. The BLEU-4 retentions are Figure 1; the QA retentions on the same conditions are given in Section B.4 below. B.1 Temporal frame shuffling Each clip’s frames are randomly permuted before the visual encoder, with a permutation seeded per sample to ensure reproducibility. The permutation is length-preserving and leaves the padding frames the dataloader adds in place, so every downstream sequence length and attention mask is unchanged. Figure 8: The two spatial-mask variants, on one Phoenix-2014T frame (blue) and one CSL-Daily frame (orange). Segmentation rows occlude the region traced by the segmentation model and are used for the main results; bounding box rows occlude each connected component’s filled bounding box. The occlusion is drawn in each dataset’s identity colour rather than black, which would read as part of the signer’s dark clothing. Note that the segmentation masks preserve the outline of the occluded articulator. B.2 Spatial masking The masking ablations occlude the pixels of a body-part region in every frame before the visual encoder, using precomputed per-video masks for the face and the hands. The masks are produced by a publicly available segmentation model [26] that traces the outline of each articulator. The resulting mask retains the silhouette of what it removes, so a hand-shaped void may still expose the handshape. We therefore repeat every masking condition with a bounding-box variant, in which each connected component of a region is replaced by its filled axis-aligned box. Boxing per component rather than per region matters: the two hands are separate components, and a single box spanning both would occlude the torso between them. Figure 8 shows both variants on both datasets, and Figure 9 reports the corresponding retention scores. Figure 9: Retention under the bounding-box mask variant, BLEU-4 (top) and QA (bottom), to be read against Figures 1 and 10. Boxing the articulator is the harsher occlusion and costs roughly ten further points of retention throughout, but the ordering of conditions and the differing hand/face profiles of the two datasets are unchanged, so no conclusion in the main text depends on the choice of mask. B.3 Gaussian noise This condition replaces the visual representation outright, giving a floor in which no visual evidence survives, so whatever a model still scores is attributable to target-side priors alone. Where the replacement is applied differs between the two model families, because they expose different interception points, and the difference is worth stating rather than eliding. The five gloss-free systems are corrupted at the encoder output. The visual encoder runs untouched and its T×DT× D feature sequence E is then overwritten elementwise by draws from (μ^E,σ^E2)N( μ_E, σ_E^2), where μ^E μ_E and σ^E σ_E are the mean and standard deviation of E itself, so the decoder is handed a sequence of the original length and scale that carries no information. The attention mask is passed through unchanged, so padded positions stay masked and every downstream length is as at baseline. SingleStream exposes no equivalent point, so there the raw frame tensor is overwritten instead: every pixel of every frame and channel is redrawn from a Gaussian matched to the clip’s own mean and standard deviation, clamped back into the valid pixel range, and passed through the real encoder. Both paths compute their statistics per clip, the ablation decoding running at batch size one. The two are therefore not the identical operation, one destroying the encoded representation and the other the input to the encoder, and a reader should not treat the noise row as a single controlled manipulation across all six systems. What licenses reading it as one floor is that the two cut points are empirically interchangeable at this severity: BLEU-4 retention under noise is 11–5%5\% across the five feature-level models on Phoenix-2014T and 11–6%6\% on CSL-Daily, and SingleStream, the pixel-level one, retains 4.0%4.0\% and 0.4%0.4\%, inside that range on Phoenix-2014T and just below it on CSL-Daily (Figure 1). Neither cut leaves a model anything to read. B.4 Retention under both metrics Section 6.2 states the headline profiles; this section gives the figure and the remaining conditions. Figure 10 repeats the corruption experiment of Figure 1 under QA. On Phoenix-2014T, occluding both hands still leaves 6464–80%80\% of content transfer intact and occluding the face leaves 6262–70%70\%; only removing both together brings models down to 1616–35%35\%. On CSL-Daily the hands retain just 44–16%16\% while the face retains 4040–68%68\%. Neither dataset’s models depend on temporal order as much as on either articulator: shuffling frames retains 2020–33%33\% and 1212–45%45\%. Replacing the signal with noise floors both, at 88–15%15\% and 11–6%6\%, the residue attributable to target-side priors alone. Two readings follow. Models trained on Phoenix-2014T transfer most of their measured content without the manual channel, which is difficult to reconcile with predictions grounded in the signing, and the dataset’s narrow weather-forecast scope is the obvious suspect. And because the two corpora reverse which articulator is load-bearing under fixed architectures, grounding is a property of what the benchmark affords. Neither reading would be legible from an n-gram score. Figure 10: QA accuracy retention (% of baseline) under the same input corruptions as Figure 1, over all six models. Read down a column for one model’s dependence on each channel. The bounding-box mask variant is in Figure 9. Appendix C Content and Function Words: Tagging, Distribution and Attribution C.1 Taggers and categories Every POS-resolved analysis in this paper, the attribution of Figure 2, the retention decomposition of Figure 3, the QA coverage of Appendix G and the distributions below, uses one spaCy transformer pipeline per language: de_dep_news_trf for German and zh_core_web_trf for Chinese, with the tagger, morphologiser and lemmatiser retained and the parser and NER disabled. Tokens tagged punct or space are dropped, and every token is reduced to its lowercased lemma, so inflectional variants of one word count once. Tagging is memoised on sentence text, which makes identical predictions across corruption conditions tagged once and identically by construction. The two-way split used throughout is content == noun, propn, verb, adj, num, adv, pron, and function == adp, det, aux, cconj, sconj, part, with any residual tag (x, intj, sym) counted as function in the two-way summaries. pron is deliberately counted as content: a pronoun in SLT is a pointing sign the signer must actually have articulated in space, so unlike the adp/det/aux scaffolding a target-side prior cannot supply the correct referent for free. The 37%37\% function share of Section 3 is not an artefact of that choice: regrouping pron as function raises it to 42.2%42.2\% on Phoenix-2014T and 49.6%49.6\% on CSL-Daily, so the conclusion holds, more strongly, under either convention. C.2 Distribution in the test references (a) Phoenix-2014T, linear axes. (b) Phoenix-2014T, log–log axes. (c) CSL-Daily, linear axes. (d) CSL-Daily, log–log axes. Figure 11: Rank-frequency curves of the spoken-language test references, word types sorted by descending frequency and split by the content/function categories above. Function words form a short, tall curve (few types, each frequent); content words a long, flat one with a hapax tail. Linear axes (left) show the concentration of function tokens, log–log axes (right) the length of the content tail. Section 3 argues that function words are the words a target-side prior can supply for free, because they occupy a small and concentrated vocabulary where content words are broad and long-tailed. The test references bear that out directly. On Phoenix-2014T, 7474 function word types cover 2,5512,551 tokens (34.534.5 tokens per type) against 789789 content types over 5,2635,263 tokens (6.76.7 per type); the ten most frequent function types alone account for 76.3%76.3\% of all function tokens, where the ten most frequent content types account for 18.9%18.9\% of content tokens, and 41.1%41.1\% of content types occur exactly once against 25.7%25.7\% of function types. CSL-Daily carries a far larger content vocabulary relative to its function vocabulary, 19.019.0 content types per function type against 10.710.7 on Phoenix-2014T: 117117 function types over 1,8841,884 tokens (16.116.1 per type) against 2,2232,223 content types over 8,8268,826 tokens (4.04.0 per type), top-ten shares of 70.5%70.5\% against 16.0%16.0\%, and hapax rates of 19.7%19.7\% for function types against 29.5%29.5\% for content types. Figure 11 plots the corresponding rank-frequency curves. A metric counting n-gram matches can therefore secure a large share of its tokens from a vocabulary of well under a hundred types, learnable from the target side alone, while the content vocabulary that the visual signal has to carry is an order of magnitude larger and mostly rare. C.3 Transfer to gloss The gloss side of Figure 2 requires POS tags for annotation that is not running text: Phoenix-2014T glosses are uppercase German citation forms and CSL-Daily glosses are space-separated Chinese lemmas, so neither pipeline applies to them unmodified. The cached distributions cover 4,1444,144 tagged gloss tokens on Phoenix-2014T, against 4,2644,264 whitespace-delimited tokens in the test split, and 4,5514,551 on CSL-Daily against 9,0029,002. C.4 BLEU-4 attribution The attribution distributes a system’s corpus BLEU-4 over POS categories in three steps. First, each n-gram of the system output that is matched against the reference under BLEU’s clipped counting, n=1…4n=1… 4, contributes 1/n1/n to each of its n tokens; a token’s attribution weight is the sum of those contributions over every matched n-gram it participates in, so a token in several matched orders accumulates more than one and an unmatched token accumulates zero. Second, weights are pooled by the POS tag of the token, giving a per-category total AcA_c; because each matched n-gram distributes exactly one unit in total, ∑cAc _cA_c is the number of matched n-grams the system produced. Third, each category receives Ac/∑c′Ac′A_c/ _c A_c of the system’s corpus BLEU-4. The assignment is exact by construction: the per-category points sum to the corpus BLEU-4 of the system they were computed for, to four decimals, for all six systems on both corpora. The cached table additionally records a match rate per category, the fraction of that category’s tokens participating in at least one matched n-gram, which separates how often a category is hit from how much credit it carries. Appendix D QA Generation: Implementation Details Teacher configuration. One open-weight LLM plays every role (content extraction, question generation, distractor repair, all quality-control gates, and answering), served locally through an OpenAI-compatible vLLM endpoint. Decoding is greedy (τ=0τ=0) except for the two recall paths: the fallback retry of content extraction (τ=0.7τ=0.7) and the second per-content question-generation pass (τ=0.7τ=0.7). Generation calls are given an 8,192-token budget to accommodate long reasoning traces before the JSON payload; answering calls are given 1,024. Reasoning effort is set to low for reasoning-capable checkpoints, which avoids reasoning overflowing the budget and returning empty content. All model outputs are parsed by extracting the first balanced JSON object, and every call is memoised in an append-only cache keyed on (backend, call type, model, prompt hash, input, token budget, reasoning effort, temperature), making runs resumable and bit-identical on repetition. Pipeline stages. For a reference s: 1. Content unit extraction. s→s→ typed checkable content units; retry at τ=0.7τ=0.7 if empty; deduplicate on normalised text. An optional audit pass that re-checks each content unit for groundedness is implemented but disabled in the reported runs. 2. Question generation. Each content unit independently → 2–4 MCQs, over two temperature passes, unioned under deduplication on both normalised question and normalised answer. 3. Distractor repair. Items without exactly four distractors get one repair call; still-malformed items are dropped. 4. Structural validation. Exactly four non-empty, mutually distinct distractors, none equal to the answer; the answer must not appear inside the question; truncation to NQmax=10N_Q =10. 5. Assembly. Five content options permuted under a seed derived from sha256(reference id, question, answer); the language’s not stated sentinel appended as option 5. 6. Gates. Round-trip, empty-passage world-knowledge probe, and single-answer ambiguity check; a fourth gibberish gate is available under strict mode and unused here. 7. Answering. Given only the passage, question and six options, the answerer is instructed to rely solely on the passage, accept paraphrase as a match, select not stated when the passage is silent, and emit a single digit, read as the last standalone in-range digit of the completion. If a reference survives extraction but yields no validated question, the pipeline falls back to a single folded generation prompt before the reference is dropped. Language resolution. Prompts are resolved in the domain language of the dataset or partition: the extractor, question generator, answerer, fallback generator, not stated sentinel, and all questions, answers and distractors are written in the reference language. The cross-lingual comparisons (OpusParcus, PAWS-X) instead use one language-agnostic prompt set across all languages, so that no language benefits from prompt tuning. BLEU is computed with a sacreBLEU tokenizer throughout. Prompts. Verbatim listings of the extractor, content-to-question generator, answerer, folded fallback generator, and the three auxiliary gate prompts (ambiguity, distractor repair, content unit audit) are provided in the supplementary material. The not stated sentinel is registered per language (e.g. nicht genannt for German, ei mainittu for Finnish, non mentionné for French), currently covering the ten languages used in this work. Appendix E Metric Definitions E.1 Per-item metric scores For a reference string r scored against a source string s, both metrics are mapped to [0,1][0,1]: b(r,s)=sacreBLEU-4(r,[s])100,b(r,s)= sacreBLEU-4 (r,[s] )100, q(r,s)=1|Qs|∑Q∈Qs[ans(Q,r)=gold(Q)]q(r,s)= 1|Q_s| _Q∈ Q_s1 [ans(Q,r)=gold(Q) ] where QsQ_s is the set of QC-gated multiple-choice questions generated from the source rendering s (each with a gold option), and ans(Q,r)ans(Q,r) is the teacher’s chosen option when shown reference r. BLEU-4 is sacreBLEU throughout, under the tokeniser matched to the target language: zh for Chinese, character-level for Japanese and Korean, and 13a otherwise, since the default 13a tokeniser mangles CJK and would put those languages on a different footing from the rest. No lowercasing or punctuation stripping is applied anywhere. E.2 Confidence intervals Every QA interval in this paper is a percentile bootstrap over test instances. Let ai∈[0,1]a_i∈[0,1] be the fraction of instance i’s admitted questions that a system answers correctly, so the reported score is a¯=N−1∑i=1Nai a=N^-1 _i=1^Na_i over the N references retaining at least one question. We draw B=2,000B=2,000 resamples of the N instances with replacement, recompute a¯ a on each, and take the 2.52.5th and 97.597.5th percentiles; a half-width is (hi−lo)/2(hi-lo)/2. The draw is seeded, so an interval is reproducible from the stored per-instance accuracies. The resampling unit is the instance, not the question. A reference contributes several questions (5.25.2–6.46.4 on average, Table 1) whose correctness is correlated through the shared reference and the shared candidate sentence, so resampling questions independently would treat that correlation as extra evidence and report an interval narrower than the data supports. This interval covers the finite test set only: it asks how far the score would move on another sample of sentences with the bank held fixed. It does not cover regeneration of the bank, which Section 6.1 reports separately as σ over ten banks. The two are not directly comparable as printed, since σ is one standard deviation while CI/2 is a 95%95\% half-width, i.e. about 1.961.96 standard errors; on a common standard-error basis the instance term is three to nine times the bank term, depending on the system. The BLEU-4 intervals of Table 5 are produced the same way, with the resampled quantity changed: each of the B=2,000B=2,000 resamples draws the same N instances with replacement and recomputes corpus BLEU-4 over the resampled hypothesis and reference pairs under the language’s sacreBLEU tokeniser, and the interval is again the 2.52.5th to 97.597.5th percentile with half-width (hi−lo)/2(hi-lo)/2. Both columns of that table therefore share one resampling unit and one resample count, and neither covers regeneration of the question bank. The asymmetry is that this is the only source of variation for BLEU-4, which is deterministic given a checkpoint, whereas the QA score carries the bank term of Section 6.1 in addition. E.3 Signal-to-noise ratio (SNR) Write c(⋅)c(·) for either metric of Section E, b or q. Data are organized into groups g=1,…,Gg=1,…,G, each a set of equivalent renderings of one source sentence with a designated source rendering sgs_g. Paraphrase scores. The other renderings of the same sentence, scored against sgs_g (the metric’s behaviour on genuine paraphrases): Wg=c(r,sg):r∈group g,r≠sg,ng=|Wg|.W_g=\\,c(r,s_g):r g,\ r≠ s_g\,\, n_g=|W_g|. Per-group mean and population standard deviation (SD): μg=1ng∑x∈Wgx, _g= 1n_g _x∈ W_gx, σg=1ng∑x∈Wg(x−μg)2(σg=0 if ng=1). _g= 1n_g _x∈ W_g(x- _g)^2 ( _g=0 if n_g=1). Floor scores. Renderings of different sentences scored against sgs_g, pooled over all groups (the metric’s floor on non-equivalent content): F=c(r′,sg):r′∉group g.F=\\,c(r ,s_g):r g\,\. Aggregate over groups μ¯=1G∑gμg⏟paraphrase mean,σ¯=1G∑gσg⏟paraphrase SD, μ= 1G _g _g_paraphrase mean, σ= 1G _g _g_paraphrase SD, f¯=1|F|∑x∈Fx⏟floor mean. f= 1|F| _x∈ Fx_floor mean. gapΔ=μ¯−f¯,SNR=Δσ¯,NSR=1SNR \;gap\ = μ- f, = σ, = 1SNR\; Interpretation. σ¯ σ is the average dispersion within a set of equivalent phrasings (paraphrase fluctuation, the noise); Δ is how far the metric moves between equivalent and non-equivalent content (the discrimination signal). SNR is dimensionless (numerator and denominator share the [0,1][0,1] scale), so it is comparable across BLEU and QA, unlike raw σ¯ σ, which a floor-bound metric makes trivially small. Higher SNR = more paraphrase-invariant (equivalently, lower NSR). Because both metrics floor on non-equivalent content (f¯≈0 f≈ 0), Δ≈μ¯ ≈ μ and SNR coincides with the coefficient-of-variation signal-to-noise μ¯/σ¯ μ/ σ, while remaining well defined for metrics that do not floor. (Defined when Δ>0 >0.) Auxiliary statistics. Discriminability d=Δ/sd(μg∪F)d= /sd (\ _g\∪ F ); significance via a paired one-sided Wilcoxon signed-rank test on (σgBLEU,σgQA)\( _g^BLEU, _g^QA)\ over groups with ng≥2n_g≥ 2. Appendix F Human Evaluation of QA-based pipeline To assess the reliability of the automated QA-generation pipeline, human evaluations are conducted for the content span extraction, question–answer quality assessment, and distractor validity. A annotator rate a samples, and report the acceptance rate alongside its Wilson 95% confidence interval. Table 6 presents representative question–answer sets produced at the final stage of the pipeline. Table 6: Sample references carried through the pipeline. Within each item the correct answer is marked ⋆ and set in bold; four of the remaining options are distractors and the last is a not stated sentinel. Content unit Question and answer options Phoenix-2014T (de) — es kommen dann ein paar wolkenfelder Then a few cloud fields come. kommen (come) Was geschieht mit den Wolkenfeldern? (what happens with the cloud fields?) ziehen fort move away ⋅· bleiben stehen stay standing ⋅· verdampfen evaporate ⋅· teilen sich divide themselves ⋅· ⋆ kommen come ⋅· nicht genannt not mentioned ein paar (a few) Wie viele Wolkenfelder kommen? (how many cloud fields come?) viele many ⋅· mehrere several ⋅· ⋆ ein paar a few ⋅· eins one ⋅· keine no ⋅· nicht genannt not mentioned wolkenfelder (cloud fields) Was kommt nachher? (what comes later?) sonnenschein sunshine ⋅· sturm storm ⋅· regen rain ⋅· ⋆ wolkenfelder cloud fields ⋅· niederschlag precipitation ⋅· nicht genannt not mentioned CSL-Daily (zh) — 他的膝关节手术非常成功。 His knee surgery was very successful. 他(he) 谁的膝关节手术非常成功？(whose knee surgery was very successful?) 护士nurse ⋅· ⋆ 他he ⋅· 她she ⋅· 医生doctor ⋅· 病人patient ⋅· 未提及not mentioned 膝关节手术(knee surgery) 什么手术非常成功？(what surgery was very successful?) 心脏手术heart surgery ⋅· 眼科手术eye surgery ⋅· 脑部手术brain surgery ⋅· ⋆ 膝关节手术knee surgery ⋅· 骨盆手术pelvic surgery ⋅· 未提及not mentioned 非常成功(very successful) 手术的结果如何？(how was the result of the surgery?) 还需要再做一次needs doing again ⋅· 失败了it failed ⋅· 有并发症there were complications ⋅· ⋆ 非常成功very successful ⋅· 一般so-so ⋅· 未提及not mentioned PAWS-X (en) — The Tabaci River is a tributary of the River Leurda in Romania . Tabaci River What river is a tributary of the River Leurda? ⋆ Tabaci River ⋅· Leurda River ⋅· Danube River ⋅· Olt River ⋅· Mures River ⋅· not stated Tabaci River In which country is the Tabaci River located? ⋆ Romania ⋅· Hungary ⋅· Ukraine ⋅· Serbia ⋅· Bulgaria ⋅· not stated tributary What is the Tabaci River? ⋆ tributary ⋅· main river ⋅· lake ⋅· canal ⋅· waterfall ⋅· not stated tributary Of which river is the Tabaci River a tributary? ⋆ River Leurda ⋅· River Someş ⋅· River Danube ⋅· River Amara ⋅· River Olt ⋅· not stated tributary What is the relationship of the Tabaci River to the River Leurda? ⋆ tributary ⋅· source ⋅· mainstream ⋅· confluence ⋅· branch ⋅· not stated River Leurda Which river is the Tabaci River a tributary of? ⋆ River Leurda ⋅· River Someş ⋅· Danube River ⋅· River Timiş ⋅· River Olt ⋅· not stated River Leurda In which country is the River Leurda located? ⋆ Romania ⋅· Ukraine ⋅· Serbia ⋅· Hungary ⋅· Bulgaria ⋅· not stated Romania In which country is the Tabaci River a tributary of the River Leurda? ⋆ Romania ⋅· Serbia ⋅· Bulgaria ⋅· Ukraine ⋅· Hungary ⋅· not stated F.1 Evaluation of content unit extraction A content unit is a span copied verbatim from the reference that names an isolated component of the expression targeting an entity or group, an action, an attribute, a quantity, a time, a location, a relation, a negation, or a polarity. Prepositions and modifiers may stay inside the phrase they belong to, so on the mountains, it lightly rains and from east to northeast may be considered one unit. Articles, copulas and conjunctions should not be units on their own. Given the reference and the extracted units, the judge marks: grounding: Is each extracted content unit present in the reference; missing: Is any content from the reference not covered by any extracted unit; and excess: Could any of the extracted units be omitted without losing content. The evaluation covers 50 spans from 4 datasets (Phoenix-2014T, csl-daily, Opus-Parcus and WMT23), covering 3 languages. Chinese and German is translated to english before it is evaluated. The results are presented in Table 7. Generally, the LLM appears perfectly able to consistently produce grounded spans. When assessing the span size, it appears appropriate in roughly 80% of the cases, with some tendencies to over- and under-segment spans. Undersegmentation appears only in the longest references, which are the most frequent in the WMT news sentences. Over segmentation are concentrated in the short references, more frequently occuring in Phoenix-2014T. While some structural words are ocationally passed as content units, their impact on the question generation phase appears limited. A low generall rate for missed content is observed. Table 7: Human evaluation of the question-generation pipeline by one judge. Each dataset is capped at its first 100 judged spans and 50 judged questions and items, in annotation order, so every corpus carries equal weight in the pooled column. Subscripts are Wilson 95% intervals: for a rate that should be high (↑ ) the lower bound is the conservative reading, for one that should be low (↓ ) the upper bound is. Each block states its own denominator, since spans, content words, questions and items are four different bases. Usable deliberately excludes whether a question is anchored on its own unit, which is a property of the extracted span rather than of the question; that code is reported separately. Check All Phoenix (de) CSL (zh) OPUS-P (en) WMT23 (zh-en) Content-unit extraction spans judged 400 100 100 100 100 Grounding ↑ 100.0[99.0, 100.0]100.0_[99.0,\,100.0] 100.0[96.3, 100.0]100.0_[96.3,\,100.0] 100.0[96.3, 100.0]100.0_[96.3,\,100.0] 100.0[96.3, 100.0]100.0_[96.3,\,100.0] 100.0[96.3, 100.0]100.0_[96.3,\,100.0] Excess ↓ 5.0[3.3, 7.6]5.0_[3.3,\,7.6] 8.0[4.1, 15.0]8.0_[4.1,\,15.0] 6.0[2.8, 12.5]6.0_[2.8,\,12.5] 5.0[2.2, 11.2]5.0_[2.2,\,11.2] 1.0[0.2, 5.4]1.0_[0.2,\,5.4] reference content words 793 139 244 134 276 Missed ↓ 0.9[0.4, 1.8]0.9_[0.4,\,1.8] 0.0[0.0, 2.7]0.0_[0.0,\,2.7] 0.4[0.1, 2.3]0.4_[0.1,\,2.3] 3.0[1.2, 7.4]3.0_[1.2,\,7.4] 0.7[0.2, 2.6]0.7_[0.2,\,2.6] Question generation questions judged 200 50 50 50 50 Usable ↑ 96.0[92.3, 98.0]96.0_[92.3,\,98.0] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 96.0[86.5, 98.9]96.0_[86.5,\,98.9] 96.0[86.5, 98.9]96.0_[86.5,\,98.9] Answerable ↑ 98.0[95.0, 99.2]98.0_[95.0,\,99.2] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] Grounded binding ↑ 98.5[95.7, 99.5]98.5_[95.7,\,99.5] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] Well formed ↑ 96.5[93.0, 98.3]96.5_[93.0,\,98.3] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 96.0[86.5, 98.9]96.0_[86.5,\,98.9] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] Anchored on its own unit ↑ 98.0[95.0, 99.2]98.0_[95.0,\,99.2] 96.0[86.5, 98.9]96.0_[86.5,\,98.9] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] Answer options items judged 200 50 50 50 50 Guessable closed book ↓ 15.0[10.7, 20.6]15.0_[10.7,\,20.6] 16.0[8.3, 28.5]16.0_[8.3,\,28.5] 14.0[7.0, 26.2]14.0_[7.0,\,26.2] 14.0[7.0, 26.2]14.0_[7.0,\,26.2] 16.0[8.3, 28.5]16.0_[8.3,\,28.5] Item validity ↑ 98.0[95.0, 99.2]98.0_[95.0,\,99.2] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] Key correct ↑ 99.5[97.2, 99.9]99.5_[97.2,\,99.9] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 98.0[89.5, 99.6]98.0_[89.5,\,99.6] 100.0[92.9, 100.0]100.0_[92.9,\,100.0] Sound distractors ↑ 95.8[94.1, 96.9]95.8_[94.1,\,96.9] 95.0[91.0, 97.3]95.0_[91.0,\,97.3] 97.5[94.3, 98.9]97.5_[94.3,\,98.9] 98.5[95.7, 99.5]98.5_[95.7,\,99.5] 92.0[87.4, 95.0]92.0_[87.4,\,95.0] Clean item ↑ 90.5[85.6, 93.8]90.5_[85.6,\,93.8] 88.0[76.2, 94.4]88.0_[76.2,\,94.4] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 94.0[83.8, 97.9]94.0_[83.8,\,97.9] 86.0[73.8, 93.0]86.0_[73.8,\,93.0] F.2 Question-answer generation The judge is shown the reference, the source unit and the generated question. Four properties are marked: whether the question tests that unit, whether it is answerable from the reference alone, whether it assumes only relations the reference actually states, and whether it is well formed. A question counts as usable only if all four hold. F.3 Distractor generation The key is disclosed and each of the four distractors is coded as clearly wrong but plausible, also correct, a synonym of the key, the wrong kind of thing so that it does not answer the question at all, or the right kind but too implausible to compete. The wrong-kind code is kept separate because four options of the wrong kind leave the key as the only one that fits, which is what makes an item solvable without reading anything. Two passes, in this order. Closed book, the judge sees the question and the six options with no reference shown at all and picks the option they would bet on; chance is 1/61/6. Since every admitted item is one the model itself could not answer with the passage withheld, an item a judge can answer is a residual miss of that check. The reference is then revealed with the key still hidden and the judge answers the item; agreement with the key is item validity measured without anchoring. F.4 Answering step Table 8: Human–model agreement on the answering step, with key and model choice withheld from the judge. Reference passage is a control: items are answerable by construction, so it measures whether a person recovers the same answer from text known to contain it. System translation is the metric’s actual task, on unseen decoder output. Subscripts are Wilson 95% intervals; κ is Cohen’s κ over the six options. Reference passage System translation Dataset n Agreement κ n Agreement κ Average 50 100.0[92.9, 100.0]100.0_[92.9,\,100.0] 1.00 50 92.0[81.2, 96.8]92.0_[81.2,\,96.8] 0.88 Phoenix-2014T (de) 25 100.0[86.7, 100.0]100.0_[86.7,\,100.0] 1.00 25 88.0[70.0, 95.8]88.0_[70.0,\,95.8] 0.85 CSL-Daily (zh) 25 100.0[86.7, 100.0]100.0_[86.7,\,100.0] 1.00 25 96.0[80.5, 99.3]96.0_[80.5,\,99.3] 0.89 To assess the answering step against human judgment, one judge answers 50 items each from Phoenix-2014T and CSL-Daily under two conditions: the reference passage (a control, since admitted items are answerable by construction) and real gloss-free decoder output on unseen references (the actual task). Table 8 reports agreement. On reference passages, judge and model agreed on all 50 items and both matched the pipeline’s key on all 50. On system translations, agreement was 92.0%92.0\% (κ=0.88κ=0.88), with the two datasets differing in answerability rather than agreement: the judge could answer 13 of 25 Phoenix items from the translation but only 3 of 25 for CSL-Daily, where the baseline preserves little reference content. That agreement remains high even as answerability collapses supports the intent of the control: disagreements reflect what the translations convey, not how the items are read. All four disagreements take the same form: the judge selected not stated, treating the information as unavailable from the translation, where the model committed to an option; in no case did both commit to different answers. Taking the judge’s reading as correct, all four are over-credits. In three of them the model reproduced the pipeline’s key even though the translation at best vaguely state it: snow down to middle elevations, rendered as above about four hundred metres, was credited as up to high elevations. The judge selected not stated more often than the model in both datasets (2828 vs. 2424 pooled), so the residual disagreement is directional: the metric is marginally more generous than a human reader. Appendix G Per-language Question Yield and Content Coverage Table 1 of Section 5 pools question yield and content coverage over languages; Table 9 gives the per-language breakdown. Coverage is computed with the taggers of Appendix C, using spaCy’s large pipeline for every language except German, which keeps the de_dep_news_trf transformer model of the POS analysis so the SLT numbers remain directly comparable with it. PAWS-X Korean is scored by the metric but omitted here, as no spaCy pipeline is available for it. The complementary measurement to item-level coverage is what the gold answers themselves assert, since a candidate is only ever scored through an answer. Here the asymmetry the metric is designed for appears directly: on Phoenix-2014T the answers carry 66.6%66.6\% of the reference’s content lemmas against 51.9%51.9\% of its function lemmas, and the gap widens on the longer PAWS-X sentences (64.3%64.3\% vs. 32.5%32.5\%). Per POS (Table 10), gold answers recover 93.5%93.5\% of numerals, 80.8%80.8\% of adjectives and 76.8%76.8\% of nouns on Phoenix-2014T, but only 18.2%18.2\% of auxiliaries and 8.3%8.3\% of subordinating conjunctions. The QA metric therefore concentrates its scoring on precisely the categories the attribution study of Section 3 showed BLEU-4 to under-weight relative to the signed input. Content cov. Benchmark Lg NrefN_ref |Q||Q| item answer Phoenix-2014T de 641 5,934 88.9±13.688.9_± 13.6 74.2±17.674.2_± 17.6 CSL-Daily zh 1,175 7,992 94.5±10.794.5_± 10.7 81.7±18.381.7_± 18.3 OpusParcus en 2,772 7,022 80.6±25.080.6_± 25.0 70.8±27.270.8_± 27.2 OpusParcus de 2,997 9,088 84.8±22.484.8_± 22.4 68.6±27.068.6_± 27.0 OpusParcus fi 3,380 11,245 83.9±24.783.9_± 24.7 74.1±27.974.1_± 27.9 OpusParcus fr 3,069 9,822 81.2±23.981.2_± 23.9 64.4±27.064.4_± 27.0 OpusParcus ru 3,631 12,450 90.0±20.690.0_± 20.6 77.0±25.177.0_± 25.1 OpusParcus sv 3,643 11,805 87.1±22.287.1_± 22.2 70.0±27.070.0_± 27.0 PAWS-X en 1,995 19,547 95.8±8.595.8_± 8.5 83.4±15.683.4_± 15.6 PAWS-X de 1,969 22,013 95.2±8.895.2_± 8.8 78.8±16.378.8_± 16.3 PAWS-X fr 1,986 22,366 95.2±9.895.2_± 9.8 81.3±16.481.3_± 16.4 PAWS-X es 1,998 21,931 95.9±8.595.9_± 8.5 84.9±15.684.9_± 15.6 PAWS-X zh 1,975 20,204 92.4±10.392.4_± 10.3 76.8±16.476.8_± 16.4 PAWS-X ja 1,969 23,994 95.4±8.895.4_± 8.8 82.0±15.182.0_± 15.1 Table 9: Per-language question yield and content coverage, qwen2.5-32b teacher, uncapped banks. NrefN_ref counts references retaining at least one QC-admitted question; |Q||Q| is the resulting bank size. item = question text + gold answer, answer = gold answer alone; subscripts are standard deviations over references. Content POS num adj noun Phoenix-2014T (de) 97.0 83.7 87.0 CSL-Daily (zh) 93.1 96.6 90.7 OpusParcus (de) 95.6 85.7 88.7 PAWS-X (en) 92.3 83.8 83.3 PAWS-X (zh) 88.6 84.5 82.6 Function POS pron aux sconj Phoenix-2014T (de) 35.7 10.5 33.3 CSL-Daily (zh) 86.9 – 46.4 OpusParcus (de) 62.9 17.3 48.3 PAWS-X (en) 45.8 16.1 19.9 PAWS-X (zh) 61.1 – 5.6 Table 10: Per-POS coverage by the gold answers (%), representative banks. The metric’s scoring mass sits on numerals, adjectives and nouns, and largely bypasses auxiliaries and subordinators. Pronouns are the exception and by design: a dedicated participant-extraction pass probes who acts and who is acted on, since a translation can reverse those and lose the sentence while keeping every noun in it, so pronoun coverage is high wherever the corpus is pronoun-dense (CSL-Daily 86.986.9, OpusParcus 62.962.9) and low only on the weather text of Phoenix-2014T, which contains few. Dashes are tags the language’s tagger does not assign. Appendix H Per-language Validation Results Table 2 of Section 5.2 aggregates PAWS-X over languages; Table 11 gives the breakdown. The QA advantage holds in all seven languages and is largest where BLEU-4 is weakest: +0.171+0.171 for English, where the 13a tokeniser is well matched to the text, against +0.250+0.250 and +0.255+0.255 for Japanese and Korean, where character-level tokenisation leaves BLEU-4 near chance (0.5450.545, 0.5340.534). The per-language OpusParcus correlations of Figure 4 run from ρ=0.12ρ=0.12 for BLEU-4 against 0.530.53 for QA in Russian to 0.400.40 against 0.620.62 in Swedish. Lang. n BLEU AUC QA AUC Δ en 1,995 0.705 0.875 +0.171 de 1,969 0.626 0.847 +0.221 fr 1,986 0.645 0.846 +0.202 es 1,998 0.629 0.850 +0.221 zh 1,975 0.594 0.818 +0.224 ja 1,969 0.545 0.795 +0.250 ko 1,970 0.534 0.790 +0.255 mean 0.611 0.831 +0.221 Table 11: PAWS-X meaning-flip discrimination (test split, qwen2.5-32b teacher). ROC-AUC = P(score(paraphrase)>score(flipped))P(score(paraphrase)>score(flipped)) over the question-bearing pairs; 0.50.5 = blind to the flip. Surface overlap is high in both classes by construction. QA wins in 7/7 languages. Appendix I Run-to-Run Stability: Sources and Per-System Numbers Two mechanisms make a repeat of the pipeline differ, and only one of them is the serving stack. The second per-content-unit question-generation pass samples at τ=0.7τ=0.7 by design, as a recall device for questions the greedy pass does not produce (Appendix D); no sampling seed is sent, so that pass essentially never repeats. Every other stage decodes greedily and is reproducible only up to the arithmetic of the server, where continuous batching varies batch composition and with it the floating-point reduction order. Comparing two runs over the calls they share, the greedy stages return identical text for 99.699.6 and 99.7%99.7\% of content-unit extractions, 98.098.0 and 98.3%98.3\% of distractor generations, and 99.96%99.96\% of answering calls on both benchmarks, the exposure scaling with output length. Question generation, which mixes the greedy and sampled passes, returns identical text for only 57.657.6 and 62.8%62.8\% of its calls. Deliberate sampling is therefore the dominant channel and batching the secondary one, and their combined effect is that a repeat asks a substantially different bank: 71.7%71.7\% of admitted items recur between two runs on Phoenix-2014T and 77.9%77.9\% on CSL-Daily (Jaccard 0.560.56 and 0.640.64). Publishing the exact bank alongside a score is thus needed to reproduce a per-item analysis, but not to compare systems. Figure 12: Per-system QA accuracy across the ten independent bank regenerations of Table 4, one panel per benchmark in its identity colour. The y axes are independent (Phoenix-2014T spans thirteen points, CSL-Daily fifty). No system changes order in either panel. Figure 13: Per-instance distribution of sentence BLEU-4 and QA accuracy for each model, on Phoenix-2014T (top) and CSL-Daily (bottom). The modal system ordering by QA is reproduced in every repeat on both benchmarks, a single distinct ordering with mean Kendall τ=1.00τ=1.00 against it (Figure 12). This holds even for the closest pair, SignCL and GFSLT-VLP, whose means sit 0.350.35 points apart on Phoenix-2014T, a gap of roughly one standard deviation of the difference between two repeats; their confidence intervals in Table 5 nevertheless overlap, so the pair is reproducibly ordered by the pipeline without being separated by the test set. BLEU-4 recomputed alongside is bit-identical across all ten Phoenix-2014T repeats, which admit the same 641641 references every time; on CSL-Daily the admitted set moves by a single reference in two of the ten repeats and BLEU-4 moves with it by at most 0.010.01 points. The spread in QA therefore enters through which questions clear the gates, not through the scoring. Appendix J Training-Likeness: Exclusions and Cumulative View Estimator. Training-likeness uses the fuzz.ratio character-level similarity from RapidFuzz. Since ℓi _i is a property of the benchmark, all models are ordered identically. The sliding window spans 1212 likeness points, stepped by 22; inside each, the per-instance score is regressed on ℓ and on reference length, and the partial coefficient βm(ℓ) _m( ) gives the sensitivity at fixed length, reported as Sm(ℓ)=100βm(ℓ)C¯m,S_m( )= 100\, _m( ) C_m, with C¯m C_m model m’s mean per-instance score. The normaliser is sentence BLEU-4, not the corpus BLEU-4 reported in papers; the two differ by 11–22 points. Exclusions. Two groups are excluded from the local sensitivity curve of Section 6.2. References at ℓ=100 =100 are exact copies of a training sentence, a point mass that admits no slope; on Phoenix-2014T they are also half the length of their neighbours (5.85.8 against 12.512.5 tokens), so a window straddling them would report a length difference as likeness sensitivity. These are reported as score levels instead. Windows holding fewer than 4040 instances, or in which a single likeness value holds more than half the mass, cannot support a slope either and are left empty rather than interpolated. Figure 14: Retained fraction Rm(f)=Cm(f)/Cm(1)R_m(f)=C_m(f)/C_m(1) over nested prefixes of the test set, admitted least-training-like first. Mean over M=5M=5 models, band = min–max. Upper axis: training-likeness of the last admitted instance. Left-hand labels give RmR_m at f=0.1f=0.1. Cumulative view. The local view of Section 6.2 asks how fast a metric moves; this one asks how much of a published number the training-like instances account for in aggregate. We sort the test set from least to most training-like, score nested prefixes, and plot the retained fraction Rm(f)=Cm(f)/Cm(1)R_m(f)=C_m(f)/C_m(1) (Figure 14), normalising each model against itself before averaging so that differences in absolute level do not dominate. A metric insensitive to training overlap stays flat near 100%100\%; one inflated by training-like instances rises steeply to the right. On Phoenix-2014T, BLEU-4 retains 12.3%12.3\% of its reported score on the least training-like tenth of the test set against 56.5%56.5\% for QA; on CSL-Daily, 32.1%32.1\% and 49.8%49.8\%. This view cannot replace the local one. RmR_m is cumulative, and differentiating it does not recover SmS_m: for a running mean dk[S(k)/k]=(xk−C(k))/k ddk\! [S(k)/k ]=(x_k-C(k))/k, so one local effect appears large early and small late purely through how many instances have already been averaged in, and corpus BLEU-4, not being a mean, adds a shifting n-gram normalisation on top. The two figures report the same phenomenon at different resolutions, and agree. Appendix K Per-instance Score Distributions Section 6.3 notes that the two metrics agree at corpus level while disagreeing on individual instances; Figure 13 shows the distributions behind that. Sentence BLEU-4 is massed near zero on both benchmarks, whereas QA occupies the whole range on Phoenix-2014T with its mode at 100100: on that benchmark a majority of instances are scored by BLEU-4 as near-total failures while QA reads much of their content as transferred. The share of instances at QA=0QA=0, printed in each panel, is the clearest difference between the benchmarks: 77–10%10\% on Phoenix-2014T against 2828–67%67\% on CSL-Daily, so the corpus means on CSL-Daily are driven substantially by instances from which nothing is recovered rather than by uniformly partial transfer.