Paper deep dive

Should We be Pedantic About Reasoning Errors in Machine Translation?

Calvin Bao, Marine Carpuat

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 53

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/14/2026, 1:57:29 AM

Summary

This paper investigates the prevalence and impact of reasoning errors in machine translation (MT) across multiple language pairs. Using an automated LLM-as-a-judge protocol, the authors identify three categories of reasoning errors: source-misaligned, hypothesis-misaligned, and trace-misaligned. The study finds that while reasoning errors are frequent, their impact on translation quality varies by language, with higher precision in error detection for Urdu compared to Spanish. Interventions on reasoning traces suggest that while small corrections have minimal impact, stronger interventions can resolve errors, though overall translation quality gains remain mixed, indicating limited reasoning faithfulness in current MT models.

Entities (6)

Calvin Bao · researcher · 100%FLORES · dataset · 100%Machine Translation · task · 100%Marine Carpuat · researcher · 100%Ministral-3-8B · large-language-model · 95%Qwen3-8B · large-language-model · 95%

Relation Signals (3)

Calvin Bao → authored → Should We be Pedantic About Reasoning Errors in Machine Translation?

confidence 100% · Preprint. Under review. Should We be Pedantic About Reasoning Errors in Machine Translation? Calvin Bao, Marine Carpuat

Qwen3-8B → evaluatedon → FLORES

confidence 95% · Table 3: LLM-as-a-judge reasoning errors on FLORES devtest

Reasoning Errors → affects → Machine Translation

confidence 90% · we find reasoning errors in translation... suggesting limited reasoning faithfulness for machine translation.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Across multiple language pairings (English $\to$ \{Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese\}), we find reasoning errors in translation. To quantify how often these reasoning errors occur, we leverage an automated annotation protocol for reasoning evaluation wherein the goal is to detect if a reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3) reasoning trace-misaligned. We probe the reasoning model with perturbed traces correcting for these identified reasoning errors using an array of weak-to-strong interventions: hedging, removal, re-reasoning after removal, hindsight, and oracle interventions. Experimenting with interventions on the reasoning traces suggests that small corrections to the reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these reasoning errors does not resolve the initial errors significantly, suggesting limited reasoning faithfulness for machine translation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

52,296 characters extracted from source content.

Expand or collapse full text

Preprint. Under review. Should We be Pedantic About Reasoning Errors in Machine Translation? Calvin Bao, Marine Carpuat Department of Computer Science University of Maryland College Park, MD 20742, USA csbao@umd.edu Abstract Across multiple language pairings (English→Spanish, French, German, Mandarin, Japanese, Urdu, Cantonese), we find reasoning errors in trans- lation. To quantify how often these reasoning errors occur, we leverage an automated annotation protocol for reasoning evaluation wherein the goal is to detect if a reasoning step is any of three error categories: (1) source sentence-misaligned, (2) model hypothesis-misaligned, or (3) reasoning trace-misaligned. We probe the reasoning model with perturbed traces correcting for these identified reasoning errors using an array of weak-to- strong interventions: hedging, removal, re-reasoning after removal, hind- sight, and oracle interventions. Experimenting with interventions on the reasoning traces suggests that small corrections to the reasoning have little impact on translation quality, but stronger interventions yield the highest resolution rates, despite translation quality gains being mixed. We find ultimately that reasoning errors in MT can be identified with high precision in Urdu but lower precision in Spanish, but that removing these reasoning errors does not resolve the initial errors significantly, suggesting limited reasoning faithfulness for machine translation. 1 Introduction Despite reasoning being applied successfully (à la “let’s think step-by-step”, or through “thinking-native” large reasoning models (LRMs)) across canonical NLP tasks, reasoning for machine translation has been met with mixed success. The overwhelming consensus in MT is that reasoning only helps when it is explicitly geared toward translation. For example, a prompt design could involve surfacing or researching translation-relevant knowledge before generation (Briakou et al., 2024), decomposing translation into multiple steps like translating phrases before combining them (He et al., 2024) or as drafting, refinement, and proofreading (Briakou et al., 2024; Li et al., 2025; Feng et al., 2025), or broadly coordinating agentic workflows assigning a role to an agent to conduct steps inducive to translation (Li et al., 2025; Wang et al., 2025). On the other hand, evaluations of LRMs suggest that off-the-shelf “native reasoning” underperforms in MT when compared to direct translation (Li et al., 2026; Rajaee et al., 2026). Across this body of work, reasoning has been treated instrumentally, basically as a means to obtain better translations. An audit on potential errors in the reasoning is underexplored, and we focus on potential errors present in the reasoning – whether a reasoning step is correct; if not, whether the misconception propagates to the output; and whether targeted correction of those errors can lead to measurably improved translation quality. In this study, we look to reasoning as our primary object-of-study to understand why they might lead to subpar performance. We find examples of reasoning error that may be propagated to the output (c/f Figure 1). Such errors can arise even in high-resource settings such as English→Spanish and English→Chinese. To automatically detect these errors, we propose and empirically benchmark an LLM-as-a-judge detector, validating them with bilingual human judgments for severity and detectability. Then, we conduct an evaluation 1 arXiv:2604.09890v1 [cs.CL] 10 Apr 2026 Preprint. Under review. SOURCE (EN) The researchers suggested that, even though this is the tail of a young dinosaur, the sample shows adult plumage and nota chick’s down . OUTPUT (ES) Los investigadores sugirieron que, aunque esta es la cola de un dinosaurio joven, la muestra muestra plumaje adulto y nola plumilla de un polluelo . TRACE (excerpt) . . . “the sample shows adult plumage and not a chick’s down” – “la muestra muestra plumaje adulto y no laplumillade un polluelo.”“Plumilla” is the correct term for down feathers in Spanish.Putting it all together, the translation should be accurate and natural. ERRORa chick’s down →la plumilla de un polluelo † † plumilla≈ quill / pen nib; the correct term is plumón or plumaje juvenil. Figure 1: A reasoning trace decision in Qwen3-8B (FLORES, en→es) maps a chick’s down to plumilla (quill/pen nib) instead of plumón (down feathers). The error propagates to the output. to test whether simple, targeted interventions on the detected reasoning errors lead to improved outcomes in the translation. We guide our study with two central research questions: • How frequent are reasoning errors in MT reasoning traces? (RQ1) • Are detected issues easily mitigated? (RQ2) 2 Background Reasoning for machine translation.Recent work explores how reasoning can be applied successfully in machine translation, finding that structured and human-like translation strategies that can be decomposed into multiple steps tends to work best. He et al. (2024) induces the model to consider translation-related knowledge before producing the final translation. Similarly, Briakou et al. (2024) proposes a pipeline consisting of pre-drafting, drafting, refinement, and proofreading stages. Feng et al. (2025); Li et al. (2025) introduce frameworks that consist of collaborative agents that collaborate in order to generate the best final translation, primarily consisting of distinct candidate generation + verification roles. In contrast with what reasoning-like behavior can induce better translation output, Li et al. (2026); Rajaee et al. (2026) both find that “native reasoning” underperforms when compared to the direct translation approach. Faithful reasoning and trace sensitivity. Our faithfulness tests are most closely related to work on reasoning faithfulness, which asks whether a model’s output is causally at- tributable to its intermediate reasoning (Lanham et al., 2023). With multiple-choice Question- Answering benchmarks, Lanham et al. (2023) directly intervenes on Chain-of-Thought reasoning by perturbing traces and measures model faithfulness by the extent to which the model’s accuracy changes. Interestingly, they find that larger models are often less faithful: the final answer survives even substantial trace perturbations, suggesting the model is not always faithful to the reasoning in all settings. Paul et al. (2024) sharpens this perspective this intuition via causal mediation analysis, modeling the traceRas a mediator between inputXand outputY. They decompose the total effect into a direct effect (X → Y, bypassingR) and an indirect effect (X → R→ Y), measured by intervening on both the input (swapping in a counterfactual questionX 1 ) and the trace (swapping in reasoningR 1 generated forX 1 ). Across several reasoning-oriented benchmarks (commonsense & math) they find that indirect effects decrease as reasoning traces get larger. They also show that none of the tested models exhibit consistently high 2 Preprint. Under review. Output Valid (O✓)Output Invalid (O✗) Reasoning Valid (R✓)Potentially aligned: trace has no detected issues and the translation has no detected issues. Silent translation error: translation is wrong, but the trace does not contain detectable issues. Reasoning Invalid (R✗) Right answer, wrong reason: translation is correct, but the trace contains a detected issue. This may cause reduced trust in good translations. Wrong answer, wrong reason: translation is incorrect, and trace has semantic divergences. Highlighting this may help users reject imperfect translations. Table 1: A 2×2 matrix framing the validity of reasoning and translation outputs. In this work, we analyze the detection of invalid outputs, conditioned on invalid reasoning R✗. indirect effects, indicating that LLMs do not reliably use their stated reasoning when producing answers. More recently, Chen et al. (2025) study faithfulness in the extended thinking traces of LRMs, and find that models frequently omit mention of hints they relied on, with this type of faithfulness decreasing on harder tasks. Zhou et al. (2024) further shows that models are vulnerable to inaccurate noisy rationales in-context, with noisy reasoning degrading downstream accuracy. Together, these observations motivate our setting in MT: if models inconsistently condition on their own reasoning in more verifiable tasks, the questions of whether this extends to the richer output space of NLG tasks like translation, where the output may or may not reflect many decisions claimed in the reasoning, enabling a richer analysis that is not possible in the settings with verifiable answers. 3 Methodology for Quantifying Reasoning Errors 3.1 Defining reasoning errors Prior work on chain-of-thought faithfulness asks whether a model’s stated reasoning influ- ences the final output (Lanham et al., 2023; Turpin et al., 2023; von Recum et al., 2026). This work typically intervenes on reasoning chains, by truncating, paraphrasing, injecting errors, and measuring robustness of model responses to these perturbations by assessing whether the model’s answer changes in the intended way. When it does not, the model is deemed unfaithful to the input reasoning, as canonically defined (Jacovi & Goldberg, 2020). We propose 3 consistency categories connected to distinct threads of prior work in evaluation for NLG, reasoning verification, and CoT faithfulness, which are referred in Table 2. 3.2 Detecting reasoning errors Inspired by the evaluation axes of the general reasoning trace evaluation taxonomy proposed by Lee & Hockenmaier (2025), we operationalize this for machine translation. SOURCE– REASONING and REASONING-INTERNAL instantiate the factuality and validity/coherence categories. The closest analogy to REASONING–OUTPUT relates to both the utility axis of Lee & Hockenmaier (2025) and to chain-of-thought faithfulness (Lanham et al., 2023). We define a structured annotation protocol that decomposes each trace into individual decision steps simply via sentence tokenization. It then checks the consistency of each reasoning step with the preceding reasoning trace, the source input and the model output. In terms of granularity of annotation, we choose to annotate at the reasoning step-level for both practical reasons and inspired by work in Chain-of-Thought faithfulness, which frequently operates at the step-level to support targeted interventions, as we would like to do (Prasad et al., 2023; Jacovi et al., 2024). Accounting for self-correction. Reasoning traces are generative processes: models often explore, reconsider, and self-correct within a single trace. Lanham et al. (2023) finds that 3 Preprint. Under review. CategoryQuestionExampleConnection to Lee & Hocken- maier (2025) INPUT– REASONING Doesthetrace make claims sup- portedbythe source sentence? Source: “Thebug crashed the whole system.”Step 3: “The word bug here meansabiolog- icalorganism; I’l translate it as bacteria.” Explanation: The trace selects the wrong sense of bug; the source context (software) does not support this interpretation. In the taxonomy of Lee & Hock- enmaier (2025), this corresponds most directly to factuality, es- pecially groundedness with re- spect to the source. Analogous to groundedness in NLG evaluation (Honovich et al., 2022; Maynez et al., 2020): just as a summary can hallucinate content absent from a source document, a trace can hallucinate translation equiv- alences. REASONING– OUTPUT Does the output re- flect the decisions stated in the trace? Step 8: “The phrase ‘pretending to listen’ maps naturally toha- ciendocomosiestuvieraescuchán- donos.” Step 9: “I’m confident in this phrasing. Finalizing the translation.” Output: “Ella es- tabahaciéndosepasarporsiestu- vieraescuchándonos.”Explana- tion: The output uses a different phrase than the one committed to in Steps 8–9. In the taxonomy of Lee & Hock- enmaier (2025), this aligns most closely with utility:whether a reasoning step actually con- tributes to the final answer. Prior work on CoT faithfulness asks a related question: whether reason- ing reflects the model’s actual pro- cess (Lanham et al., 2023; Jacovi & Goldberg, 2020). REASONING- INTERNAL Is the trace inter- nallyconsistent and coherent? Step 5: “bank meanselbanco.” Step 8: “Wait—since it is next to ‘river’, it should belaorilla.” Step 9: “Alright, going withel banco as decided earlier.” Expla- nation: Step 9 reverts to Step 5 without acknowledging the cor- rection in Step 8. In the taxonomy of Lee & Hock- enmaier (2025), this aligns with both coherence and validity. Coherence is concerned with whether a reasoning step’s pre- conditions are established con- sistently, while validity is con- cerned with whether the step contains a logical inconsistency. Draws also from detection of self- contradictory reasoning in LLM outputs (Liu et al., 2024). Table 2: Error categories for reasoning errors in machine translation, grounded in the reasoning-trace evaluation taxonomy of Lee & Hockenmaier (2025).Highlightedspans in the example indicate the error span to focus on. models can recover from errors introduced earlier in their reasoning chains, but recovery is not guaranteed and depends on error type and task. To account for this, we distinguish between two severity levels. An issue is marked ERROR if the incorrect decision persists through the trace. An issue is marked FIXED-LATER if the trace initially makes an error but then corrects it at a later point in the trace. This distinction is important because it separates errors that persists from natural reasoning errors that may arise from exploration, and discarded by the end. We discard FIXED-LATER from our analysis going forward. Error Detection Protocol/Implementation We implement the protocol with an LLM-as- a-judge (GPT-5.2). The judge receives three inputs: the source sentence, the sentence- tokenized reasoning trace, and the final translation. For each issue it identifies, it returns a structured annotation with: (i) the category (INPUT–REASONING, REASONING–OUTPUT, or REASONING–INTERNAL); (i) the index into the trace sentence; and (i) and a short rationale. We samplek =5 judgments per trace with temperature set to 0.4. For each independently sampled run, we then aggregate by category and reasoning step index and keep only the 4 Preprint. Under review. Annotation summary by reasoning model Model Language pair # Samples # w/ errors Error rate Avg. steps / trace Avg. errors / sample Qwen3-8B en→es1005656.0%39.121.40 en→zh1005454.0%32.691.86 en→ja1008686.0%40.883.51 en→fr1007373.0%42.312.81 en→de1008181.0%45.642.94 en→ur1009797.0%38.266.40 en→yue1006363.0%36.252.30 all70051072.8%39.313.03 Ministral-3-8B en→es1006060.0%21.592.24 en→zh1006565.0%17.633.18 en→ja1009393.0%22.203.93 en→fr1007474.0%16.842.72 en→de1006969.0%20.343.39 en→ur100100100.0%16.405.68 en→yue100100100.0%17.123.74 all70056180.1%19.723.55 Table 3: LLM-as-a-judge reasoning errors on FLORES devtest, n=100 per direction. issues with at least 3 judgments referring to it (majority vote,≥3/5). We test on 7 language directions: en→es,ja,fr,de,yue,ur,zh, sourced from the FLORES parallel test set (Goyal et al., 2022). Detected reasoning errorsTable 3 shows a summary of all detected errors in the reasoning traces generated by Qwen3-8B and Ministral-3-8B. We see a large proportion of samples (72.8% on average for Qwen3-8B, and 80.1% on average for Ministral-3-8B) across languages with at least one reasoning error, with on average 3.03 and 3.55 average errors per sample. These rates are high enough to raise the question of whether the LLM-as-a-judge is identi- fying genuine errors, or is pedantically flagging reasoning steps. To distinguish this, we validate a subset of the detected issues against bilingual human annotators (§3.3) in two language pairs: en→es (high-resource with the smallest avg. errors / sample), and en→ur (low-resource with the highest avg. errors / sample). 3.3 Human validation We conduct a validation via bilingual human annotators to compute the precision of the automatic LLM-as-a-judge results on error detection, using Qwen3-8B translations for English→Spanish, and English→Urdu. For each language pair, we sample 30 FLORES devtest examples. Each example is a triplet(x,t,y), wherexis the English source sentence, tis the model reasoning trace, andyis the final translation. Each sampled example from FLORES yields one item-level annotation for translation quality. In addition, each sample yields one or more reasoning errors int, which are all annotated for error validity. All items are annotated independently by three bilingual annotators per language pair. We recruit bilingual language specialists rather than general crowd workers, following conclusions that judgments involving error analysis by experts is more reliable than judgments from non-specialists (Freitag et al., 2021). Phase 1: item-level output validity. In the first phase, annotators see only the source sentence x and the final translation y. They answer the following question: Does the translation preserve the important meaning of the source sen- tence? 5 Preprint. Under review. Annotators choose one of three labels: • OK: no critical meaning error is present; minor fluency or stylistic differences are acceptable; •NOT OK: the translation contains at least one meaning error that could mislead a reader; • UNSURE: I cannot confidently judge the item. For items labeled NOT OK, annotators additionally indicate the shortest source span whose meaning is not preserved and the shortest target-language span containing the error. These labels serve two purposes in the analysis. First, they provide an estimate of final-output correctness that is independent of the reasoning trace. Second, when combined with issue- level trace validation below, they allow us to distinguish right answer, wrong reason cases (valid output despite a validated trace error) from wrong answer, wrong reason cases (validated trace error that contributes to an invalid output). Phase 2: issue-level validation of detector-flagged trace spans. In the second phase, annotators validate individual issues proposed by the detector. For each flagged issue, annotators see the source sentencex, the final translationy, and the sentence-tokenized reasoning trace t, with the detected reasoning error highlighted. Specifically, annotators answer the following questions: 1. Does the highlighted reasoning span contain an error? Response options: - YES – the highlighted reasoning contains a clear mistake (wrong meaning, wrong claim, a contradiction, etc.), - NO – The highlighted reasoning is correct or a reasonable interpretation, - BORDERLINE – the reasoning is imprecise or debatable but not clearly wrong. 2. How confident are you in your judgment above? Response options: - CONFIDENT — I am sure of my answer. - SOMEWHAT CONFIDENT — I think my answer is right but it requires specialized knowledge I’m not fully certain about. - NOT CONFIDENT — This requires detailed language knowledge I’m not fully certain about. 3. Is this reasoning step reflected in the final translation? Response options: - YES – I see the decision or claim reflected in the final translation, - NO – The final translation seems to not have the decision or claim reflected in it, - NOT APPLICABLE – I cannot determine the connection. 4. Which of the following categories apply to this highlighted reasoning step? (multi-select) Response options: - SOURCE MISINTERPRETATION – The reasoning step is making a wrong claim or de- cision about the source. The reasoning misunderstands or misrepresents something in the English source (e.g., wrong word meaning, hallucinated information not in the source), - INTERNAL CONTRADICTION – The reasoning step contradicts its context. contra- dicts itself or contains circular/incoherent logic, - NO ISSUE – This reasoning step makes sense to me in this context. - OTHER / UNSURE. First, we report overall detector precision, defined as the proportion of flagged issues is validated as a reasoning error by the corresponding majority human judgment. Second, we compare reasoning-level validation with Phase 1 item-level OK/NOT OK judgments. 6 Preprint. Under review. English→UrduEnglish→Spanish Study size Samples annotated3030 Issues annotated18958 Translation correctness (majority vote per sample) YES (no major error)0/30 (0.0%)24/30 (80.0%) NO (major error)28/30 (93.3%)5/30 (16.7%) TIE2/30 (6.7%)1/30 (3.3%) Reasoning-error validation (majority vote per issue) YES only176/189 (93.1%)29/58 (50.0%) YES + BORDERLINE179/189 (94.7%)31/58 (53.4%) NO3/189 (1.6%)22/58 (37.9%) TIE7/189 (3.7%)5/58 (8.6%) Error reflected in output? (among validated errors only) YES41.1%84.0% NO56.6%16.1% UNSURE2.3%0.0% Annotator confidence Mean confidence90.5%91.4% On YES judgments94.0%88.9% On NO judgments72.8%97.3% Table 4: Human validation summary for English→Urdu and English→Spanish. Translation correctness and reasoning-error validation are reported at the sample and issue level respec- tively, using majority vote across three annotators. Error reflection rates are computed over annotations labeled YES or BORDERLINE. English–Urdu. Across a sample of 30 English→Urdu (86 sample-level annotations and 541 issue-level annotations), at the sample level, 28/30 items receive a majority translation fidelity error with a translation NOT OK judgment, with mean confidence high (0.819– 1.000). In terms of what the detector flagged as reasoning errors, they are validated by humans. Precision is 93.1% averaged across 3 annotators, with high confidence (94.0%). 41.1% of detected reasoning errors are judged to be reflected in the final output, indicating that a substantial fraction of the detected trace errors are not merely local reasoning noise but also impact the downstream output. Source misinterpretations are also more likely to propagate to the final Urdu translation: among error annotations, the reflection-YES rate is 44.9% for source misinterpretation, compared to 20.2% for internal contradiction. This suggests that the detector is useful when it flags these errors, and that these errors propagate to poor English→Urdu outputs. Taken together, the English→Urdu results paint a consistent picture: final translations are deemed not semantically faithful to the source, and bilingual annotators agree with the resulting errors, indicating high precision. English–Spanish.The English→Spanish results (Table 4) contrast sharply. 80.0% of sam- ples receive a majority YES on translation correctness: the Spanish translations are largely faithful to the English source. The judge flags 58 reasoning errors in these samples, but annotators confirm only 50.0% of them, with 37.9% rejected as non-errors. Crucially, anno- tators are more confident when rejecting the judge’s flags (97.3% confidence on NO) than when confirming them (88.9% on YES). Among the free-text responses, annotators note cases where the judge objects to a reasonable translation choice, flags an ambiguity the source itself does not resolve (e.g., whether “garage” means a workshop or a literal garage), or claims a grammatical rule stricter than 7 Preprint. Under review. what a native speaker would expect. Among the errors annotators do confirm, the output propagation rate is high: 84.0% are decided to have been reflected in the final output, compared to 41.1% in Urdu. This suggests that the judge’s threshold in Spanish is high enough that only genuinely consequential errors are flagged as errors, and those tend to matter for the translation. 4 Interventions on Reasoning We test whether detected reasoning errors causally influence the model’s translation. The core idea follows the interventionist logic of causal mediation analysis (Paul et al., 2024): if modifying a trace step changes the output, that step has causal influence on the translation. Starting from the original triplet(x,t,y)wheretandyare generated in a model’s thinking mode, we “re-play” the trace on the same model with thinking disabled, while modifying t7→ t ′ and injecting t ′ as context to obtain a translation y ′ . 4.1 Types of Interventions We study six interventions with increasing expected strength. We construct controlled transformationst7→ t ′ with just a modification on the detected reasoning error for the first three, and we consider the last three “oracle”-level interventions, since they constructt ′ guided by the reference, instead of the detected issue. •Hedging: prepend hedging language to the reasoning error (e.g., “I’m not sure, but. . . ”). • Removal: remove the detected reasoning error from the trace. •Re-reasoning after removal: remove the error span and regenerate the remainder of the trace from that point onward, allowing the model to re-derive downstream reasoning. • Hindsight: use the model to generate a reasoning trace to translate from the source to the reference, then re-play with that reasoning trace. • Oracle-K: K phrase-level hints covering the full source–reference alignment. • Oracle-1: a single hint mapping the error-relevant source phrase to its reference translation. 4.2 Intervention results Table 5 reports error resolution rates and COMET deltas across five language pairs and two models. We see three patterns emerge. Lightweight edits are largely ineffective for Qwen3-8B but not for Ministral-3-8B. For Qwen3-8B, hedging and removal resolve fewer than 7% of detected issues across all lan- guage pairs. This suggests that Qwen3-8B does not condition strongly on individual trace steps when generating under replay. On the other hand, under our replay setup, Ministral is more sensitive to trace edits; the same lightweight edits resolve 41–60% of issues. Re-reasoning resolves issues but degrades translation quality. Re-reasoning after re- moval achieves resolution rates of 50–63% for Qwen3-8B, second only to hindsight. How- ever, this comes at a cost: COMET scores drop substantially (up to−0.16 for en→ja), so while the specific detected error might disappear, the regenerated trace suffix introduces new problems that degrade translation quality. The pattern is consistent across both models and most language pairs. Hindsight is the strongest intervention. Hindsight, or replaying with a new trace gen- erated mapping source to the reference, achieves the highest resolution rates (72–84% for Qwen3-8B, 83–88% for Ministral-3-8B) and the most consistently positive COMET deltas when compared to the other intervention types. 8 Preprint. Under review. 5 Discussion & Conclusion Frequency of errors (RQ1) Our human validation reveals different story between Urdu and Spanish. For English→Urdu, annotators confirm 93.3% of the translations as not being faithful semantically to the source (Table 4), and 93.1% of reasoning errors being valid errors. Here, the errors the LLM-as-a-judge detects are real, and annotators understand why they are errors with high confidence, and the translations themselves are poor. For English→Spanish, the story is a bit different. Annotators confirm only 50% of the detected errors, and rate up to 80% of the translations as correct. We interpret this setting involving LLM-as-a-judge’s pedantry, flagging reasoning steps with imprecise but harm- less grammatical rules, objecting to reasonable interpretations of ambiguous phrases, or potentially diving into ungrounded stylistic preferences as errors. Notably, annotators are more confident when rejecting detected errors (97.3%), than when confirming them (88.9%), suggesting that non-errors are clearly non-errors to them. Additionally, we observe that among reasoning errors that annotators confirm are errors, 84.0% are reflected in the Spanish output versus only 41.1% in the Urdu output. This suggests that the LLM-as-a-judge’s standards are high enough that only consequential errors tend to propagate. In Urdu, many confirmed errors exist in the reasoning, but are either partially corrected downstream, or may manifest as different surface errors in the output, making the mapping between reasoning step to output harder to track. Implications on reasoning error detection Where models qualitatively produce worse translations, like in English→Urdu, the LLM-as-a-judge reliably identifies genuine rea- soning errors that a human auditor would agree with. For translation directions like in English→Spanish, where the model translates more effectively, the same judge is overly pedantic – roughly half of the detected errors do not align with what expert humans would consider reasonable reasoning errors. This means the signal is noisy, and any downstream use of the judge’s output (for example, for filtering training data, or self-correcting potential errors), should account for this in practice. Mitigation of reasoning errors (RQ2) The intervention results (§4.2) show that targeted trace edits guided by detected errors rarely improve translation quality. Hedging and removal resolve fewer than 7% of issues across all high-resource pairs and under 1% for English→Urdu—the model produces nearly identical translations no matter what hap- pens to the error. Only interventions that discard or replace large portions of the trace (re-reasoning, hindsight, oracle) achieve more substantial resolution rates, but with slight degradation of COMET. The human evaluation helps explain this pattern, and the expla- nation differs by language pair. For English→Spanish, where annotators reject roughly half the judge’s flags, the failure of lightweight interventions is unsurprising: you cannot improve a translation by correcting an error that was not consequential in the first place. For English→Urdu, however, the errors are more genuinely reasonable and the translations are poor, yet targeted edits still fail. Even when we know the reasoning is wrong and we know it affects the output, precisely repairing the reasoning step does not fix the translation in this setting. Towards better reasoning for translation. Our findings largely agree with prior work suggesting that native reasoning for MT does not work out of the box. We take a deeper dive and explore whether there are errors that might emerge from inconsistencies in the reasoning itself, finding that their properties differ across language pairs. These results suggest that reasoning for translation should not be free-form, deliberately exhaustive and “over-reasoned”, or decoupled from the output. Instead, we argue that reasoning should be selective (triggered only when warranted), faithful (it should be enforced that trace decisions can be reflected in the translation output), and structured (organized around verifiable sub- tasks rather than open-ended monologue). Our detection framework and subsequent error analysis offer a starting point for measuring progress along these axes. 9 Preprint. Under review. Acknowledgments We thank the members of the UMD CLIP lab for their valuable advice and feedback on earlier iterations of this work, especially Sarah Wiegreffe, Dayeon Ki, and Hyojung Han. Limitations Scope of models and languages. Our experiments are conducted on two open-weight reasoning models (Qwen3-8B and Ministral-3-8B), both at the 8B parameter scale. It is unclear whether the patterns we observe hold for other reasoning models. We test seven language pairs, all with English as the source. Translation from non-English sources, or between two non-English languages, may surface qualitatively different reasoning errors. Limited human validation coverage.We conduct bilingual human validation for only two of our seven language pairs (English→Urdu and English→Spanish), which were chosen to contrast a lower-resource and a high-resource setting. The remaining five pairs lack direct human validation of the judge’s precision, for cost reasons. Given that we observe different precision rates between Urdu (93.1%) and Spanish (50.0%), it is probable that precision varies across the unvalidated pairs as well, and we cannot determine in which direction without additional annotation. Ethics Statement Intended use. The work presented here aims to characterize where and how reasoning traces may fail in machine translation, not as a proposal for improving translations in production. Our intervention experiments show that targeted trace edits can resolve some detected errors but do not reliably improve overall translation quality. We do not recommend the interventions described here as a post-editing pipeline without further quality assurance, as they may introduce new errors not present in the original translation. LLM disclosure. LLMs were used in several different ways in the preparation of this research. For paper writing, Anthropic Claude was used for drafting, refining language (grammar and style), and creating LaTeX tables and figures. For coding, Claude was used to refine scripts for model generation and data analysis. Finally, for research ideation, Elicit (2023) was used to guide and help summarize literature review. All scientific claims, ultimate experimental design decisions, and interpretations are the authors’ own. References Eleftheria Briakou, Jiaming Luo, Colin Cherry, and Markus Freitag. Translating step-by-step: Decomposing the translation process for improved translation quality of long-form texts. In Barry Haddow, Tom Kocmi, Philipp Koehn, and Christof Monz (eds.), Proceedings of the Ninth Conference on Machine Translation, p. 1301–1317, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.wmt-1.123. URL https://aclanthology.org/2024.wmt-1.123/. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, Vlad Mikulik, Samuel R. Bowman, Jan Leike, Jared Kaplan, and Ethan Perez. Reasoning models don’t always say what they think, 2025. URL https://arxiv.org/abs/2505.05410. Elicit. Elicit: The ai research assistant, 2023. URL https://elicit.com. Zhaopeng Feng, Yan Zhang, Hao Li, Bei Wu, Jiayu Liao, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. TEaR: Improving LLM-based machine translation with systematic self-refinement. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACL 2025, p. 3922–3938, Albuquerque, 10 Preprint. Under review. New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176- 195-7. doi: 10.18653/v1/2025.findings-naacl.218. URLhttps://aclanthology.org/2025. findings-naacl.218/. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460– 1474, 2021. doi: 10.1162/tacl_a_00437. URLhttps://aclanthology.org/2021.tacl-1. 87/. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538, 2022. doi: 10.1162/ tacl_a_00474. URL https://aclanthology.org/2022.tacl-1.30/. Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. Exploring human-like translation strategy with large language models. Transactions of the Association for Computational Linguistics, 12:229–246, 2024. doi: 10.1162/tacl_a_00642. URL https://aclanthology.org/2024.tacl-1.13/. Or Honovich, Roee Aharoni, Jonathan Herzig, et al. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, p. 3905–3920, 2022. Alon Jacovi and Yoav Goldberg. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4198–4205, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.386. URLhttps://aclanthology.org/2020.acl-main. 386/. Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, and Mor Geva. A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 4615–4634, Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.254. URL https://aclanthology.org/2024.acl-long.254/. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, Benoit Steiner, Carson Denison, Danny Hernandez, Dustin Li, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamil ̇ e Lukoši ̄ ut ̇ e, Karina Nguyen, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Robin Larson, Sam McCandlish, Sandipan Kundu, Saurav Kadavath, Shannon Yang, Thomas Henighan, Timothy Maxwell, Timothy Telleen-Lawton, Tristan Hume, Zac Hatfield- Dodds, Jared Kaplan, Jan Brauner, Samuel R. Bowman, and Ethan Perez. Measuring faithfulness in chain-of-thought reasoning, 2023. URLhttps://arxiv.org/abs/2307. 13702. Jinu Lee and Julia Hockenmaier. Evaluating step-by-step reasoning traces: A survey. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, p. 1789–1814, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979- 8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.94. URLhttps://aclanthology. org/2025.findings-emnlp.94/. Weiya Li, Junjie Chen, Bei Li, Boyang Liu, Zichen Wen, Nuanqiao Shan, Xiaoqian Liu, Anping Liu, Huajie Liu, Hu Song, and Linfeng Zhang. Tactic: Translation agents with cognitive-theoretic interactive collaboration, 2025. URLhttps://arxiv.org/abs/2506. 08403. Zihao Li, Shaoxiong Ji, and Jörg Tiedemann. Test-time scaling of reasoning models for machine translation. In Vera Demberg, Kentaro Inui, and Lluís Marquez (eds.), Proceedings 11 Preprint. Under review. of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 2902–2917, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-380-7. doi: 10.18653/v1/2026.eacl-long.133. URL https://aclanthology.org/2026.eacl-long.133/. Ziyi Liu, Soumya Sanyal, Isabelle Lee, Yongkang Du, Rahul Gupta, Yang Liu, and Jieyu Zhao. Self-contradictory reasoning evaluation and detection. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 3725–3742, Miami, Florida, USA, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.213. URL https://aclanthology.org/2024.findings-emnlp.213/. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 1906–1919, 2020. Debjit Paul, Robert West, Antoine Bosselut, and Boi Faltings. Making reasoning matter: Mea- suring and improving faithfulness of chain-of-thought reasoning. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, p. 15012–15032, Miami, Florida, USA, November 2024. Associ- ation for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.882. URL https://aclanthology.org/2024.findings-emnlp.882/. Archiki Prasad, Swarnadeep Saha, Xiang Zhou, and Mohit Bansal. ReCEval: Evaluating reasoning chains via correctness and informativeness. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 10066–10086, Singapore, 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.622. URL https://aclanthology.org/2023.emnlp-main.622/. Sara Rajaee, Sebastian Vincent, Alexandre Berard, Marzieh Fadaee, Kelly Marchisio, and Tom Kocmi. Unlocking reasoning capability on machine translation in large language models, 2026. URL https://arxiv.org/abs/2602.14763. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. Alexander von Recum, Leander Girrbach, and Zeynep Akata. Are reasoning llms robust to interventions on their chain-of-thought?, 2026. URLhttps://arxiv.org/abs/2602.07470. Jiaan Wang, Fandong Meng, Yunlong Liang, and Jie Zhou. DRT: Deep reasoning translation via long chain-of-thought. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, p. 6770–6782, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-256-5. doi: 10.18653/v1/2025.findings-acl.351. URL https://aclanthology.org/2025.findings-acl.351/. Zhanke Zhou, Rong Tao, Jianing Zhu, Yiwen Luo, Zengmao Wang, and Bo Han. Can language models perform robust reasoning in chain-of-thought prompting with noisy rationales?, 2024. URL https://arxiv.org/abs/2410.23856. A Appendix A.1 Detecting reasoning errors As discussed in the main paper, we detect reasoning errors in three categories. We show prompt templates. You are a bilingual auditor for machine-translation reasoning traces. 12 Preprint. Under review. You will analyze a SOURCE sentence, the model's TRACE (reasoning while translating), and the OUTPUT (final translation). Your task is to detect reasoning errors in three categories: 1. INPUT_TRACE: Trace statements not supported by SOURCE, or proposing incorrect translation semantics (e.g., hallucinated facts, wrong word meanings) 2. TRACE_OUTPUT: Trace decisions that don't match the OUTPUT (e.g., trace says "X" but output has "Y") 3. TRACE_INTERNAL: Contradictions, circular reasoning, or incoherent statements within the trace itself IMPORTANT RULES: - The trace will be sentence-tokenized. Reference issues by sentence index (0-indexed). - All quotes must be EXACT substrings (copy-paste) from the provided text. - Be strict but fair - minor rephrasing or stylistic choices are not errors. Output ONLY valid JSON matching this schema: "has_issues": bool, "summary": str, // One sentence summary of trace quality "issues": [ "category": "INPUT_TRACE" | "TRACE_OUTPUT" | "TRACE_INTERNAL", "trace_sentence_idx": int, "trace_quote": str, // Exact substring from trace "source_quote": str | null, // Relevant source quote if applicable "output_quote": str | null, // Relevant output quote if applicable "rationale": str // 1-2 sentence explanation ] SOURCE: source TRACE (sentence-indexed): [0] trace_sentence_0 [1] trace_sentence_1 ... OUTPUT: output A.2 Bilingual annotation interface See Figure 2 for a flow of the annotation protocol. A.3 Intervention Implementation System message (default). You are a careful machine translation assistant. Task instruction (default). If source and target language codes are known, the default is:Translate the following ⟨source language⟩ text into ⟨target language⟩. Return only the translation. (with fallbacks when only the target or neither is known). Follow-trace instruction (default).Use the reasoning trace when deciding on the translation. User message template.Interventions that replay with an (possibly edited) trace all wrap the model input in this pattern (then passed through the model chat template if available): 13 Preprint. Under review. (A) Translation correctness Shown to annotator Source sentence (English) [source sentence] Translation into [language] [model translation] Human bilingual annotator Q1. Does the translation preserve the important meaning of the source? YES = no major meaning errors NO = at least one meaning error that could mis- lead a reader UNSURE = cannot confidently judge If Q1 = NO: highlight the shortest minimal span in the source that is mistranslated, and the shortest minimal span in the translation that contains the error. Stored annotation record translation_correct ∈YES, NO, UNSURE source_error_span (if NO) translation_error_span (if NO) (B) Reasoning error validation Shown to annotator Automated detector flags a candidate reasoning span Source sentence (English) [source sentence] Translation into [language] [model translation] Trace excerpt ... [FLAGGED SPAN] ... Human bilingual annotator Q1. Is the highlighted reasoning span actually an error? YES / NO / BORDERLINE Q2. How confident are you in the judgment above? Confident / Somewhat confident / Not confident Q3. If Q1 = YES or BORDERLINE, is this reason- ing step reflected in the final translation? YES / NO / UNSURE Q4. Which of the following categories apply to this highlighted reasoning step? (multi-select): Source misinterpretation; Internal contradiction; No issue; Other / Unsure Stored annotation record (one per flagged span) q1_is_error ∈YES, NO, BORDERLINE q2_reflected_translation ∈YES, NO, UNSURE q3_labels (multi-select) Figure 2: Bilingual human annotation protocol used to validate final translation correctness and detected reasoning errors. Part (A) is applied once per translation. Part (B) is applied once per detected reasoning error. Q3 is multi-select. For cost reasons, we run this protocol on English→ Spanish and English→ Urdu. task_instruction Source: source Reasoning trace: edited_trace optional_additional_notes_block follow_trace_note Return only the final translation. If extra_notes are supplied (Oracle methods), the block is: Additional notes: - note_1 - note_2 ... 14 Preprint. Under review. Replay generation uses greedy decoding (do_sample=False, beam width 1). The trace is not placed in native “thinking” channels for these replays; only the re-reasoning intervention forces a continuation inside the model’s reasoning format (see below). A.3.1 Localizing the edited span For hedging, removal, and re-reasoning, we find a character span for the issue in the original trace: 1: if issue has non-empty trace_quote then 2:find trace_quote in trace (fuzzy / normalized match) 3: else 4:use sentence boundaries and trace_sentence_idx 5: end if 6: expandtotheminimalwholesentencethatcontainsthatspan (locate_issue_edit_span), so edits remove or hedge full sentences when possi- ble If no span can be located, hedging, removal, and re-reasoning are skipped for that issue. A.3.2 Hedging Letsbe the sentence text at the edit span. Ifsis non-empty and does not already begin (case-insensitive) with maybe, possibly, perhaps, or it may be, replace s by: Possibly, but this should be verified against the source: ⟨s⟩ Otherwisesis unchanged. The modified trace is replayed with the shared template (Sec- tion A.3). A.3.3 Removal The sentence-level span from Section A.3.1 is deleted from the trace; consecutive blank lines are collapsed. The shortened trace is replayed with the same user template as the baseline (no extra notes). A.3.4 Re-reasoning after removal Lettbe the original trace and[i,j)the edit span. The prefix isp = t :i (text strictly before the removed span), with blank lines collapsed. The user-visible instruction (before chat templating) is: task_instruction Source: source A problematic reasoning step was removed here. Reconsider the source carefully from this point onward and do not rely on the removed unsupported step. Target issue: issue. rationale You MUST continue reasoning internally from the provided starting point, but return only the final translation. A.3.5 Hindsight (reference-guided trace, then replay) Hindsight uses the same replay model with native reasoning enabled. Step 1 – synthesize a trace toward the reference. Prompt: task_instruction 15 Preprint. Under review. Source: source Reference translation: reference Think step-by-step about how to translate the source to match the reference. Analyze key phrases, idioms, and grammatical structures. Then produce the final translation. Step 2 – replay. The extracted tracet ′ is substituted intoReasoning trace:in the shared replay template (Section A.3) with no extra notes. One replay is stored per record (not per issue); fix-judging, if enabled, is still evaluated per targeted issue against that single output. A.3.6 Direct oracles (Oracle-1 and Oracle-K) Oracle interventions keep the original model trace and add natural-language hints derived from the judge/issue object (not from an automatic word aligner). Each issue yields one multi-line notebuild_oracle_note(issue)with bullet lines, in order when the fields are present: - Problematic trace snippet: trace_quote - Relevant source quote: source_quote - Original output quote: output_quote - Why it is problematic: rationale - Use the source sentence to avoid carrying this error into the final translation. 1:Oracle-1: for each targeted issue, callfrom_tracewithrecord.traceand a single extra note: 2: Oracle correction for one identified issue:+newline+ build_oracle_note(issue) 3:Oracle-K: once per record (shared across all targeted issues on that record), call from_trace with record.trace and extra notes: 4: Oracle corrections for all identified issues: 5:followed by build_oracle_note(·) for every targeted issue on that record, in order Both use the shared replay user template; generation is the same as for baseline replay (Section A.3). 16 Preprint. Under review. Qwen3-8BMinistral-3-8B Intervention Resolved / total Rate↑∆ COMET↑ Resolved / total Rate↑∆ COMET↑ en-de hedging2 / 1491.3%0.000048 / 11641.4%-0.0010 hindsight125 / 14983.9%+0.0088103 / 11688.8%+0.0001 oracle-147 / 14931.5%+0.003878 / 11667.2%+0.0006 oracle-k45 / 14930.2%+0.002577 / 11666.4%+0.0027 removal5 / 1493.4%+0.000859 / 11650.9%+0.0006 re-reason75 / 14950.3%-0.002480 / 11668.9%-0.0715 en-es hedging0 / 820.0%0.000055 / 10154.5%+0.0003 hindsight67 / 8281.7%-0.003988 / 10187.1%-0.0013 oracle-117 / 8220.7%-0.000983 / 10182.2%+0.0023 oracle-k19 / 8223.2%-0.000683 / 10182.2%+0.0067 removal5 / 826.1%-0.001062 / 10161.4%+0.0008 re-reason42 / 8251.2%-0.012171 / 10170.3%-0.0777 en-fr hedging9 / 1496.0%-0.001267 / 13748.9%+0.0001 hindsight114 / 14976.5%+0.0119121 / 13788.3%+0.0055 oracle-148 / 14932.2%+0.0002103 / 13775.2%+0.0010 oracle-k57 / 14938.3%+0.0012102 / 13774.5%+0.0030 removal10 / 1496.7%-0.001376 / 13755.5%-0.0007 re-reason93 / 14962.4%-0.009089 / 13765.0%-0.1156 en-ja hedging9 / 2004.5%+0.000294 / 21443.9%+0.0064 hindsight147 / 20073.5%+0.0110179 / 21483.6%+0.0254 oracle-156 / 20028.0%+0.0020136 / 21463.6%-0.0067 oracle-k54 / 20027.0%+0.0011129 / 21460.3%-0.0011 removal13 / 2006.5%-0.0005101 / 21447.2%+0.0026 re-reason101 / 20050.5%-0.0003111 / 21451.9%-0.1607 en-ur hedging2 / 3990.5%-0.000942 / 21020.0%+0.0030 hindsight1 / 3990.3%+0.106674 / 21035.2%+0.0892 oracle-10 / 3990.0%+0.004868 / 21032.4%+0.0008 oracle-k0 / 3990.0%+0.006662 / 21029.5%+0.0004 removal1 / 3990.3%-0.000287 / 21041.4%+0.0177 re-reason1 / 3990.3%-0.012499 / 21047.1%-0.1975 en-yue hedging0 / 1350.0%0.000012 / 1309.2%+0.0004 hindsight0 / 1350.0%+0.013117 / 13013.1%+0.0194 oracle-13 / 1352.2%+0.00419 / 1306.9%-0.0039 oracle-k1 / 1350.7%+0.00049 / 1306.9%-0.0041 removal0 / 1350.0%+0.00027 / 1305.4%+0.0019 re-reason0 / 1350.0%-0.015625 / 13029.2%-0.0938 en-zh hedging1 / 941.1%-0.000172 / 12159.5%-0.0070 hindsight68 / 9472.3%-0.0012106 / 12386.2%+0.0089 oracle-125 / 9426.6%-0.001287 / 12370.7%-0.0033 oracle-k36 / 9438.3%+0.000479 / 12364.2%-0.0132 removal6 / 946.4%-0.000173 / 12359.3%+0.0026 re-reason47 / 9450.0%-0.010282 / 12366.7%-0.0958 Table 5: Intervention results grouped by language pair. Bold marks the highest resolution rate and the highest∆COMET within each language section. Positive∆COMET values are shown with an explicit + sign. Resolved / total preserves the model-specific denominators from the original results. 17

Instant research discovery