Paper deep dive
Mind Which Bird You Favour: Parameterizing Adequacy-Fluency Balance in Meta-Evaluation of Machine Translation
Behzad Shayegh, Niloofar Kazemi
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:There is a tradeoff in machine translation meta-evaluation between prioritizing alignment with adequacy versus fluency. The balance depends on the combination of translation systems in the meta-evaluation dataset. This system set is a small, filtered sample whose characteristics change heavily across years and language pairs; it does not represent the true system distribution. Consequently, the adequacy-fluency balance is often unrepresentative and subject to change. For sensitive domains, controlling this balance is critical. We expose this balance as a tunable choice. To achieve a target balance, we reweight existing systems while minimizing distortion from uniform weighting, ensuring the evaluated systems remain real and representative. We provide an exact optimization algorithm with theoretical guarantees and pruning mechanisms to compute these weights. To validate meta-evaluation internal consistency, we design a scorer-augmentation framework that establishes a known relative identity for the scorers. Results demonstrate that our reweighting method effectively controls the adequacy-fluency balance and preserves the internal consistency, outperforming prior approaches. Finally, we analyze the performance of popular scorers across a sweep of this parameter.
Tags
Links
- Source: https://arxiv.org/abs/2609.14795v1
- Canonical: https://arxiv.org/abs/2609.14795v1
Trouble viewing inline? Open PDF directly →
Full Text
95,758 characters extracted from source content.
Expand or collapse full text
Mind Which Bird You Favour: Parameterizing Adequacy–Fluency Balance in Meta-Evaluation of Machine Translation Behzad Shayegh Niloofar Kazemi Affiliation: Independent Researchers Abstract There is a tradeoff in machine translation meta-evaluation between prioritizing alignment with adequacy versus fluency. The balance depends on the combination of translation systems in the meta-evaluation dataset. This system set is a small, filtered sample whose characteristics change heavily across years and language pairs; it does not represent the true system distribution. Consequently, the adequacy–fluency balance is often unrepresentative and subject to change. For sensitive domains, controlling this balance is critical. We expose this balance as a tunable choice. To achieve a target balance, we reweight existing systems while minimizing distortion from uniform weighting, ensuring the evaluated systems remain real and representative. We provide an exact optimization algorithm with theoretical guarantees and pruning mechanisms to compute these weights. To validate meta-evaluation internal consistency, we design a scorer-augmentation framework that establishes a known relative identity for the scorers. Results demonstrate that our reweighting method effectively controls the adequacy–fluency balance and preserves the internal consistency, outperforming prior approaches. Finally, we analyze the performance of popular scorers across a sweep of this parameter.11 1 Code available at https://github.com/TheShayegh/adequacy-fluency-param $ $$ $footnotetext: Correspondence to: the.shayegh@gmail.com 1 Introduction Machine translation (MT) is typically judged by automatic metrics, which we refer to as scorers.22 2 We use the term “scorer” instead of “metric” to avoid ambiguity with meta-metrics. Deciding which scorer to trust is itself an evaluation problem known as meta-evaluation (Callison-Burch et al., 2007). Concretely, meta-evaluation fixes a set of translation systems and scores them once with a human protocol taken as the ground truth (Freitag et al., 2021a, Multidimensional Quality Metrics, MQM;), and once with the scorer under evaluation. It then asks how well the scorer’s system ranking agrees with that of the human, as measured by a meta-metric such as pairwise accuracy (Kocmi et al., 2021, PA;) or soft pairwise accuracy (Thompson et al., 2024, SPA;). The WMT metrics shared task (Freitag et al., 2023) performs this meta-evaluation every year, and it sits at the top of the MT stack: it identifies the best scorers, which subsequent research adopts and reports, in turn shaping the next generation of MT systems. Both the scorers and the MT systems therefore silently inherit whatever bias the meta-evaluation imposes. Figure 1: System-level Adequacy vs. Fluency MQM for (a) heen23 and (b) jazh24 WMT datasets. Each dot is an MT system. Green contours are robust Mahalanobis-distance levels (Gnanadesikan and Kettenring, 1972). The MQM ground truth is a multi-component quantity (Freitag et al., 2021a). A translation is penalized both for adequacy errors (failing to convey the meaning of the source) and for fluency errors (being ungrammatical or unnatural). Flamich et al. (2025) show that these two aspects trade off against each other, and Shayegh et al. (2025) show that the tradeoff reaches the meta-evaluation level: the overall human ranking is dominated by whichever aspect varies more across the systems being compared, so the scorers judged best are those aligned with that aspect. In other words, if the systems differ mostly in adequacy, human rankings will be driven primarily by adequacy as well; consequently, metrics sensitive to adequacy will perform better in the meta-evaluation. Crucially, this lean is heavily driven by system selection (Shayegh et al., 2025). This is especially concerning because the included systems are a small, filtered subset, thus likely a biased sample of the true, unknown distribution. Figure 1 shows that the lean differs substantially across WMT datasets. Consequently, meta-evaluation produces uncontrolled scorer rankings, risking the endorsement of undesired scorers that misdirect future model development. This motivates the need for an explicit control mechanism. Shayegh et al. (2025) attempt to address this problem through the lens of “debiasing.” They frame system-selection imbalance as undesirable extrinsic bias and introduce a system-synthesis method to reduce it. However, not all system-selection imbalance is undesirable; only its deviation from the true system distribution is. As the true distribution remains unknown, this attribute is unmeasurable, leading us to abandon debiasing in favor of control. Furthermore, we show that the system synthesis produces unrepresentative systems, yielding a distorted meta-evaluation. In this work, we expose the previously uncontrolled adequacy–fluency balance as a deliberate, tunable choice. Specifically, we parameterize the evaluation pool using the ratio of weighted marginal variances of adequacy and fluency, a parameter we name β. We treat the systems in the MQM data as a biased sample and reweight them to achieve a target β while minimizing distortion from uniform weighting. Because this reweighting is applied at the system level, the evaluated systems remain real and representative, preserving the meta-evaluation’s original characteristics. This approach is an instance of importance sampling (Tokdar and Kass, 2010) with minimal weight distortion (Deville and Särndal, 1992). The reweighting is a nontrivial optimization problem, for which we design an exact algorithm, pruning mechanisms that improve efficiency, and certificates that verify optimality. We prove the soundness and completeness of the algorithm, pruning mechanisms, and certificates, and provide an empirical computational-cost analysis. Validating meta-evaluation is challenging without a ground truth. To address this, we introduce a scorer-augmentation framework that injects controlled signals into real base scorers, preserving the inherent statistical signatures of real scorers while providing known relative identity. Using these augmented scorers, we assess a meta-evaluation’s (1) adequacy-over-fluency preference, (2) MQM adherence, and (3) explainability by MQM. The latter measures noise avoidance while retaining the natural scorer–MQM relationship. Our reweighting method effectively controls the first measure while preserving the other two, outperforming Shayegh et al. (2025)’s synthesis method. Finally, we report and analyze the meta-evaluation scores of popular scorers, including different variants of xCOMET (Guerreiro et al., 2024) and MetricX (Juraska et al., 2024), across a sweep of the exposed balance parameter β. In summary, we: (1) formulate the adequacy–fluency balance of MT meta-evaluation as a tunable parameter; (2) provide an exact algorithm that controls this parameter by importance-weighting the systems in the meta-evaluation, minimizing the distortion from uniform weighting so as to maximally preserve the meta-evaluation’s other characteristics; and (3) propose a controlled scorer-augmentation method to assess a meta-evaluation’s adequacy-over-fluency preference, MQM adherence, and explainability by MQM. 2 Background and Related Work 2.1 Meta-Evaluation A translation scorer is evaluated through meta-evaluation, which measures its agreement with human annotations across a fixed pool of translation systems (Callison-Burch et al., 2007). Expert error annotation under Multidimensional Quality Metrics (Mariana et al., 2015, MQM;) has been established as the standard ground truth for the WMT metrics shared task (Freitag et al., 2021a; Freitag et al., 2023; Kocmi and Federmann, 2023; Kocmi et al., 2025). Our work focuses on MQM-based meta-evaluation following recent standards. Meta-metrics. While scorer–human agreement was traditionally measured via Pearson Macháček and Bojar (2013); Macháček and Bojar (2014), Mathur et al. (2020) demonstrate that such a correlation is highly sensitive to outlier systems in the evaluation pool. To address this, Kocmi et al. (2021) introduce pairwise accuracy (PA), which measures the fraction of system pairs whose human ordering is correctly reproduced. To overcome PA’s coarse resolution and tie-handling limitations at the segment level (Deutsch et al., 2023), Thompson et al. (2024) develop soft pairwise accuracy (SPA), which replaces PA’s hard binary comparisons with a continuous measure based on statistical significance. SPA serves as the official system-level meta-metric of WMT24 (Freitag et al., 2024). We follow the line of research and focus on SPA as our main metric. We additionally report results for Pearson correlation in Appendix F. Meta-evaluation analysis. Recent work critically examines meta-evaluation protocols, studying annotation stability (Knowles, 2021), budget allocation (Riley et al., 2024), test-set item selection (Zouhar et al., 2025), and baseline sensitivity via degenerate sentinel metrics (Perrella et al., 2024). These studies treat pool-induced variation as noise or bias to be detected. Our work treats one special axis of that variation, the adequacy–fluency balance, as a tunable quantity that a practitioner should be able to explicitly configure. 2.2 Adequacy and Fluency The distinction between translation adequacy and fluency originates in early evaluation campaigns (White et al., 1994) before being largely collapsed into error-weighted penalties (Lommel et al., 2014). Information-theoretic analyses prove that adequacy and fluency genuinely trade off, making a single scalar summary insufficient (Flamich et al., 2025). Moreover, fluency impacts downstream user trust independently of adequacy (Martindale and Carpuat, 2018). This trade-off extends to the meta-evaluation level: meta-evaluation inherits the pool’s lean, favoring adequacy-oriented scorers in recent WMT MQM datasets due to skewed system selection (Shayegh et al., 2025). We propose a principled parameterization for the adequacy–fluency balance in meta-evaluation, accompanied by a reweighting mechanism to achieve any target balance precisely. 3 Approach We control the adequacy–fluency balance by reweighting systems while explicitly minimizing the collision probability of the assigned weights. Practically, this keeps the meta-evaluation close to the natural dataset distribution and prevents over-indexing a handful of systems. To control the adequacy–fluency balance, we need a reference for the two terms. MQM annotations provide fine-grained error categories. We follow Flamich et al. (2025) and group these into adequacy and fluency subsets, and compute Adequacy MQM and Fluency MQM at the system level accordingly. Their sum, Total MQM, is the meta-evaluation ground truth.33 3 Following Shayegh et al. (2025), we exclude rare errors that fall outside the two categories. The effect is negligible. 3.1 The Balance Parameter A meta-evaluation dataset fixes K translation systems. Write a=(a1,⋯,aK)a=(a_1,·s,a_K) and f=(f1,⋯,fK)f=(f_1,·s,f_K) for their system-level Adequacy MQM and Fluency MQM, so that system k is the point (ak,fk)(a_k,f_k) and Total MQM is t=a+ft=a+f. We assign each system k a weight wkw_k, with w=(w1,…,wK)w=(w_1,…,w_K) normalized so that ∑kwk=⊤w=1 _kw_k=1 w=1 and wk≥0w_k≥ 0. Reweighting acts on these K weights, so every system in the meta-evaluation remains real; the distribution of systems, which is likely biased, changes. Let Σ(w)=diag(w)−ww⊤ (w)=diag(w)-w . We have weighted mean and variance of scores a as μ(a,w) μ(a;w) =w⊤a =w a (1) σ(a,w)2 σ(a;w)^2 =a⊤Σ(w)a =a (w)\,a =∑kwkak2−μ(a,w)2 = _kw_ka_k^2-μ(a;w)^2 (2) respectively, and likewise for f. We also have a–f weighted covariance: Σ(a,f,w)=a⊤Σ(w)f (a,f;w)=a (w)\,f (3) Setting w to the uniform weighting 1K 1K1 recovers the unweighted sample mean and variance: μ(a)=μ(a,1K);σ(a)2=σ(a,1K)2 μ(a)=μ(a; 1K1);\;\;σ(a)^2=σ(a; 1K1)^2 (4) We assume throughout that no two systems share the same (ak,fk)(a_k,f_k); such pairs can be seen as one system in the weighting pool, with the assigned weight uniformly distributed among them. We define the balance parameter44 4 An equally valid choice for the balance parameter is βstd=σ(a,w)/(σ(a,w)+σ(f,w)) _std=σ(a;w)/(σ(a;w)+σ(f;w)), which is a monotonic reparametrization βstd=(1+β−1−1)−1 _std=(1+ β^-1-1)^-1. β(w)=σ(a,w)2σ(a,w)2+σ(f,w)2∈[0,1] β(w)= σ(a;w)^2σ(a;w)^2+σ(f;w)^2\;∈\;[0,1] (*) We specifically name the natural balance of the dataset β0=β(1K) _0=β( 1K1). Typically, a and f are partially correlated, thus Total MQM’s variance decomposes as σ(t,w)2=σ(a,w)2+σ(f,w)2+2(,,)σ(t;w)^2=σ(a;w)^2+σ(f;w)^2+2\, (a,f;w). Our β uses only the two marginal terms and leaves the cross term Σ(a,f,w) (a,f;w) out: β answers how much adequacy’s spread contributes relative to fluency’s. In the next part, we show how meta-evaluation responds to this marginal quantity. 3.2 Meta-Evaluation’s Response to β A meta-metric typically judges scorers by aggregating agreement with the human ranking across system pairs. Therefore, meta-evaluation’s lean towards adequacy or fluency is mainly driven by how much each component’s gaps vary across the system pairs. We show next that this pairwise-gap variance is equal to the row-level variance, licensing its control through β. Proposition 1. For any vector x=(x1,…,xK)x=(x_1,…,x_K) and weights w=(w1,…,wK)w=(w_1,…,w_K) with ⊤w=11 w=1 ∑k<k′wkwk′(xk−xk′)2=σ(x,w)2 _k<k w_kw_k (x_k-x_k )^2\;=\;σ(x;w)^2 (5) Proof. ∑k,k′ _k,k wkwk′(xk−xk′)2 w_kw_k (x_k-x_k )^2 =∑k,k′wkwk′(xk2−2xkxk′+xk′2) = _k,k w_kw_k (x_k^2-2x_kx_k +x_k ^2 ) =2∑kwkxk2−2(∑kwkxk)2 =2 _kw_kx_k^2-2 ( _kw_kx_k )^2 =2σ(x,w)2 =2\,σ(x;w)^2 (6) The summand wkwk′(xk−xk′)2w_kw_k (x_k-x_k )^2 is symmetric under k↔k′k k and vanishes when k=k′k=k , so splitting the sum by k<k′k<k gives 2∑k<k′wkwk′(xk−xk′)22 _k<k w_kw_k (x_k-x_k )^2. Dividing (6) by 2 gives the claim. ∎ Proposition 1 lets us read the row-level variance σ(x,w)2σ(x;w)^2 as a pairwise quantity. Now let q be the mean-squared pairwise gap q(x,w)=∑k<k′wkwk′(xk−xk′)2∑k<k′wkwk′ q(x;w)= _k<k w_kw_k (x_k-x_k )^2 _k<k w_kw_k (7) By Proposition 1, the numerator in (7) is σ(x,w)2σ(x;w)^2 and the denominator is x-independent. Therefore q(a,w)q(a,w)+q(f,w)=σ(a,w)2σ(a,w)2+σ(f,w)2=β(w) q(a;w)q(a;w)+q(f;w)= σ(a;w)^2σ(a;w)^2+σ(f;w)^2=β(w) (8) In other words, β is the balance of mean-squared pairwise gaps, and controlling β effectively drives the meta-evaluation’s lean. 3.3 Weighted Soft Pairwise Accuracy Because SPA is our primary meta-metric, we explicitly define its weighted form. For every system pair (k,k′)(k,k ), SPA compares the confidence that k outranks k′k under human judgments against the confidence under a given scorer. This confidence is estimated via paired-permutation p-values, pkk′Hp^H_k and pkk′mp^m_k , the latter for scorer m; neither depends on w. Classical SPA averages the absolute gap between these p-values uniformly over all (K2) K2 pairs. Weighting systems by w assigns a weight of wkwk′w_kw_k to each pair. Normalizing these weights yields πkk′=wkwk′∑l<l′wlwl′;∑k<k′πkk′=1 _k = w_kw_k _l<l w_lw_l ; _k<k _k =1 (9) which defines the weighted meta-metric SPA(m,w)=1−∑k<k′πkk′|pkk′H−pkk′m| (m;w)=1- _k<k _k \, |\,p^H_k -p^m_k \, | (10) Uniform weighting yields πkk′=(K2)−1 _k = K2^-1, recovering classical SPA. 3.4 Reweighting as an Optimization Problem Among all weight vectors w that can realize a given target β β, we find the one closest to uniform: it disturbs the natural view least beyond what β β requires, preserving most about the meta-evaluation. We measure distance from uniform by collision probability Pcollis(w)=∑kwk2=w⊤wP_collis(w)= _kw_k^2=w w. Minimizing this distance directly maximizes effective sample size ESS(w)=Pcollis(w)−1ESS(w)=P_collis(w)^-1, a standard diagnostic for weight concentration. In our work, we report ESS(w)/KESS(w)/K to quantify how broadly the weights are distributed, with K being the size of the original system set. Writing the balance target as a constraint b(w,β^)=(1−β^)σ(a,w)2−β^σ(f,w)2 b(w, β)=(1- β)σ(a;w)^2- βσ(f;w)^2 (11) so that b(w,β^)=0⇔β(w)=β^b(w, β)=0 β(w)= β, we formalize the reweighting as an optimization problem: w⋆=argminw w = *argmin_w 12Pcollis(w) \ 12\ P_collis(w) (P) s.t. ⊤w=1,b(w,β^)=0,wk≥0 \ 1 w=1,\ \ b(w, β)=0,\ \ w_k≥ 0 Reachable range. For two systems k,k′k,k , write βkk′=(ak−ak′)2(ak−ak′)2+(fk−fk′)2 _k = (a_k-a_k )^2(a_k-a_k )^2+(f_k-f_k )^2 (12) and let βmin _ and βmax _ be the smallest and largest such pairwise balance over all pairs of systems in the dataset. Theorem 1 (Reachable range). Problem ( P ) is feasible for target β β iff β^∈[βmin,βmax] β∈[ _ , _ ]. 3.5 Our Algorithm In this section, we develop and theoretically justify our solver algorithm for Problem ( P ). We provide a high-level narrative for the algorithm, followed by auxiliary theorems to improve its efficiency. Define S(w)=k:wk>0S(w)=\k:w_k>0\ as the support system set of w. Call a support S admissible for target β β if there exists a feasible weighting wSw_S with support S for target β β, i.e., it is non-negative and b(wS,β^)=0b(w_S, β)=0. Theorem 2 (Bounded candidate count). For an admissible support S, let minw _w 12Pcollis(w) \ 12\ P_collis(w) (SP_S) s.t. ⊤w=1,b(w,β^)=0(no non-negativity) \ 1 w=1,\ \ b(w, β)=0\ \ \ (no non-negativity) be solved on S’s coordinates, with wk=0w_k=0 for k∉Sk∉S. Then (a) Problem ( P S ) has at most five stationary points wSw_S, each obtainable in closed form by solving a polynomial equation of degree ≤5≤ 5; and (b) if w⋆w solves ( P ) and S=S(w⋆)S=S(w ), then w⋆w is one of these wSw_S. This gives the abstract shape of our algorithm: for every support S, compute its (at most five) stationary points wSw_S, discard any with a negative entry, and among everything that survives across every support, return the one of smallest Pcollis(wS)P_collis(w_S). Part (b) guarantees w⋆w is never missed by this process, thus our algorithm is complete. Next, we show the enumeration can be largely pruned without losing this guarantee. For a support S with |S|≥2|S|≥2, let βmin(S) _ (S) and βmax(S) _ (S) be the smallest and largest pairwise balance βkk′ _k among pairs k,k′∈Sk,k ∈ S. Theorem 3 (Admissible supports). If wSw_S with support S is a feasible weighting for target β β, then βmin(S)≤β^≤βmax(S) _ (S)≤ β≤ _ (S). Equivalently, a support whose pairwise balances do not bracket β β can be safely skipped during the search. Theorem 4 (Support size bound). For any weighting w supported on S, Pcollis(w)≥1/|S|P_collis(w)≥ 1/|S|, with equality iff w is uniform on S. Theorem 4 suggests that once a feasible weighting with collision probability p has been found, no support of size at most 1/p1/p can carry a feasible weighting of strictly smaller collision probability, and such supports may be skipped by the search. Particularly, our search process visits supports from largest to smallest, and the search stops the first time Theorem 4’s pruning applies. Theorems 3 and 4 offer complementary pruning mechanisms: each is strongest where the other is weakest. When β β is close to the dataset’s natural balance β0 _0, the optimum stays close to uniform and Theorem 4 prunes heavily. When β β is close to βmin _ or βmax _ , it falls out from many supports’ admissible range, thus Theorem 3 prunes heavily. Theorem 5 (Exit certificate). Let w be feasible for Problem ( P ). Let a~=a−μ(a),f~=f−μ(f) a=a-μ(a)1,\ f=f-μ(f)1 be both nonzero, and ρ=a~⊤f~‖a~‖‖f~‖ρ= a f\| a\|\,\| f\| the adequacy–fluency Pearson correlation across systems. If there exist η,λ∈ℝη,λ such that k∈S(w):wk+η∂b(w,β^)∂wk+λ=0 k∈ S(w): w_k+η\, ∂ b(w, β)∂ w_k+λ=0 (i) k∉S(w):λ+η∂b(w,β^)∂wk≥0 k∉ S(w): λ+η\, ∂ b(w, β)∂ w_k≥ 0 (i) I2+2η(β^‖f~‖2−(1−β^)ρ2‖a~‖2,−(1−β^)ρ1−ρ2‖a~‖2−(1−β^)ρ1−ρ2‖a~‖2,−(1−β^)(1−ρ2)‖a~‖2)⪰0 I_2+2η pmatrix β\| f\|^2-(1- β)ρ^2\| a\|^2,& -(1- β)ρ 1-ρ^2\| a\|^2\\[5.0pt] -(1- β)ρ 1-ρ^2\| a\|^2,& -(1- β)(1-ρ^2)\| a\|^2 pmatrix 0 (i) where ⪰0 0 means positive semidefinite, then w is a global optimum of Problem ( P ). Theorem 5 grants our algorithm a short exit: the moment one stationary point on a visited support passes the three conditions, the search stops. Algorithm 1 Exact reweighting for target β β 0: β^∈[βmin,βmax] β∈[ _ , _ ] Theorem 1 1: register w⋆w 2: for S⊆1,…,KS \1,…,K\, |S|≥2|S|≥ 2, in decreasing order of |S||S| do 3: if |S|≤1/Pcollis(w⋆)|S|≤ 1/P_collis(w ) then 4: break Theorem 4 5: end if 6: if S not admissible at β β then 7: continue Theorem 3 8: end if 9: for each stationary candidate w on S Theorem 2a do 10: if w passes the exit certificate then 11: return w Theorem 5 12: end if 13: if w feasible and Pcollis(w)<Pcollis(w⋆)P_collis(w)<P_collis(w ) then 14: w⋆←w ← w Theorem 2b 15: end if 16: end for 17: end for 18: return w⋆w Algorithm 1 concludes our exact search algorithm. Theorems 1–5 guarantee the soundness and completeness of our algorithm unconditionally: pruned branches are guaranteed to not include better solutions, short exit guarantees returning the global optimum, and the algorithm guarantees to search all branches (finite) carrying a possible solution. We also cross-validated its output against brute-force exhaustive search over all supports on small system sets, where full enumeration remains tractable, and found exact agreement throughout.55 5 Theorem 2 bounds the stationary points by the roots of a degree-five polynomial, but our implementation solves the equivalent rational secular equation directly rather than rooting that polynomial, whose coefficients become badly scaled as θ2→0 _2→ 0. This and all numerical thresholds are fixed in the released code. In Appendix B, we report how much computation Theorems 3–5 save us in practice. 4 Experiments In this section, we analyze the response of meta-evaluation to different leanings that scorers may have. Specifically, we measure each meta-evaluation’s adequacy-over-fluency preference, MQM adherence, and explainability by MQM. To this end, we develop a controlled testbed that reveals signals for these aspects. Then, we benchmark our reweighting mechanism on this testbed, comparing it against the unweighted meta-evaluation and the system-synthesis-based meta-evaluation of Shayegh et al. (2025). 4.1 Scorer Augmentation Evaluation and analysis of scorers needs a ground truth for their true underlying character (e.g., how much of a scorer’s assessment is driven by the fluency of the translation), which is not available. To obtain a replacement, we propose to augment scorers in a controlled way, so that the augmented scorers’ relative identities are known by construction, even though no scorer’s absolute identity is. We use this relative identity as pseudo ground truth for evaluation and analysis. Importantly, we augment real scorers, staying close to their natural biases and characteristics, except for those we control in particular. We augment a base scorer s0s_0 in three directions, giving three families of augmented scorers, each to test a different way a meta-evaluation’s preference among scorers could change. Given a and f being gold Adequacy MQM and Fluency MQM, the three directions are: • MQM-adherence family: s0↔a+fs_0 a+f • Adequacy–fluency family: s0↔a−fs_0 a-f • Explainability-by-MQM family: s0↔s_0 part of s0s_0 only explained by (a,f)(a,f) An ideal meta-evaluation should typically prefer an augmented scorer that better adheres to MQM or prefers explainability by MQM. We realize each family with a dial d: a base scorer always has d=0d=0, and as d increases, the augmented scorer approaches its family’s target. Each value of d yields a new augmented scorer whose relative identity to other members of its family is known through their relative d values. Note that we allow negative d, i.e., going backwards in each direction. MQM-adherence family. This is constructed by moving linearly towards t=a+ft=a+f at the system level. For segment i of system k, and base scorer s0s_0, the augmented scorer is sd(k,i)=s0(k,i)+d⋅σ(s0)σ(z)zk s_d(k,i)=s_0(k,i)+ d·σ(s_0)σ(z)z_k (13) where z is the system-level z-score of t, and σ is standard deviation across systems. In simple words, we move towards z in steps of d×s0d× s_0’s scale.66 6 The same approach can produce Adequacy MQM-adherence or Fluency MQM-adherence families of synthetic scorers by considering z as the system-level z-score of a or f, respectively. We exclude these families from our analysis as the partial correlation between the two aspects (Shayegh et al., 2025) confounds their interpretation. The above construction conserves characteristics of real scorers: the two ends of the direction (d=0d=0 and d→∞d→∞) are the base scorer itself and the MQM ground truth, both considered valid scorers. Moreover, for every finite d, the augmented scorer preserves the base scorer’s own system- and segment-specific (k,ik,i-related) biases and patterns. Note that the above construction does keep the within-system variance σi2(sd(k,⋅)) _i^2(s_d(k,·)) of each system k intact. This is the signal SPA reads as a scorer’s confidence. Keeping it unchanged avoids confounding the MQM-adherence signal with a change in SPA’s confidence signal. Adequacy–fluency family. Similar to MQM-adherence family, we write sd(k,i)=s0(k,i)+d⋅σ(s0)σ(z(a))zk(a)−d⋅σ(s0)σ(z(f))zk(f) s_d(k,i)=s_0(k,i)+ d·σ(s_0)σ(z^(a))z^(a)_k- d·σ(s_0)σ(z^(f))z^(f)_k (14) where z(a)z^(a) and z(f)z^(f) are the system-level z-scores of a and f, respectively, and σ is the standard deviation across systems. This construction is particularly clean: with correlation ρ(a,f)<1ρ(a,f)<1, it establishes a known adequacy-over-fluency preference along the increasing dial d. If ρ=1ρ=1, there is no trade-off, and the construction has no effect. Figure 2: The SPA plane of Shayegh et al. (2025), with the augmented scorers of all three families plotted as points, for (a) hLEPOR (Han et al., 2013) in ende21 dataset and (b) MetricX-24-Hybrid (Juraska et al., 2024) in ende24 dataset as the base scorer. Explainability-by-MQM family. This family’s direction is given by the part of the base scorer not explained by the joint (a,f)(a,f) pair: moving one way along it reduces this unexplained part, approaching what (a,f)(a,f) alone explains, while moving the other way amplifies it. It is different from the other two as instead of injecting/removing specific signals, it magnifies/reduces any (a,f)(a,f)-unrelated signal the base scorer may have. We find the direction as follows. Write akia_ki and fkif_ki for segment i of system k’s gold Adequacy MQM and Fluency MQM scores, and let s¯0(a,f) s_0(a,f) be the s0s_0’s mean score among segments sharing the exact (a,f)(a,f): s¯0(a,f)=i,k:aki=a,fki=f[s0(k,i)] s_0(a,f)=E_i,k:\;a_ki=a,\;f_ki=f\;\;[s_0(k,i)] (15) Let e¯k=i[s0(k,i)−s¯0(aki,fki)] e_k=E_i [s_0(k,i)- s_0(a_ki,f_ki) ] (16) be system k’s offset: the part of its score that the joint pair does not explain. We construct the augmented scorer sd(k,i)=s0(k,i)−d⋅e¯k. s_d(k,i)=s_0(k,i)-d· e_k. (17) In simple words, we partially remove each system’s offset beyond what its (a,f)(a,f) explains. For negative d, we add more of it. To visually illustrate the three families, Figure 2 plots them, applied to two different base scorers, on the SPA plane of Shayegh et al. (2025). 4.2 Evaluation Setup We analyze the performance of a meta-evaluation separately in each of three directions. Specifically, we compare the meta-evaluation’s assessment of different augmented scorers within each family, and count for how many adjacent-in-d scorer pairs the meta-evaluation correctly identifies the dial direction. We report this success rate separately for each family as three measures. Datasets. We use MQM data published by Freitag et al. (2021a).77 7 github.com/google/wmt-mqm-human-evaluation; MQM data for 2025 (Lavie et al., 2025) has not been published by the time of this study. We restrict our analysis to datasets from 2021 onward (Freitag et al., 2021b; Freitag et al., 2022; Freitag et al., 2023; Freitag et al., 2024) due to limited standardization and support for older datasets. We specifically exclude ende23 because many of its MQM annotations are categorized as “other” rather than being properly assigned to adequacy or fluency categories. This leaves 11 datasets: ende24, enes24, jazh24, heen23, zhen23, ende22, zhen22, ende21, zhen21, ted_ende, and ted_zhen. Across all datasets, we evaluate all available systems except for M2M100_1.2B-B4 and MSLC, which are officially flagged for exclusion. Detailed statistics are in Appendix D. Figure 3: Main results on ende21 (left) and ende24 (right). Curve color intensity represents the effective sample size (ESSESS); a lower ESSESS indicates fewer actively involved systems, reducing representativeness. The strictly increasing adequacy-over-fluency curve demonstrates that β is an effective driver of this balance. The natural balance β0 _0 for each dataset is marked by a vertical dotted line, with its intersections indicating the original meta-evaluation’s performance. Higher MQM adherence and explainability by MQM indicate better meta-evaluation internal consistency. Markers show the performance of the six synthesis-based meta-evaluation setups from Shayegh et al. (2025), placed at their synthesized datasets’ respective β0 _0. Base scorers. We consider automatic scorers (omitting human references) in the MQM datasets as the base scorers. This includes the competitive scorers as well as the diagnostic ones included by WMT. We exclude those without segment-level scores (e.g., corpus-level chrF). We also deduplicate perfectly duplicated scorer pairs. We provide the complete list of the base scorers in Appendix D. For each meta-evaluation and each dataset, we report the average scores across all base scorers. Dial grid. The closer two d values are, the harder it is for a meta-evaluation to recognize their relative identity. We empirically choose the d grid for the best readability of results. For MQM adherence, d takes −1.5+3n/200-1.5+3^n/200, and for explainability by MQM, −3.5+4n/200-3.5+4^n/200, where n∈ℕ:0≤n≤200n :0~≤~n~≤~200, resulting in 200200 adjacent pairs per base scorer. For adequacy-over-fluency preference, d ranges from −0.5-0.5 to 0.50.5 with a step size of 0.020.02, resulting in 5050 adjacent pairs per base scorer. 4.3 Main Results Figures88 8 Results for some datasets are deferred to the Appendix E due to space constraints. All the claims are written with respect to all datasets, including those deferred to the appendix. 3 and 7 show our method’s performance across datasets over a β sweep spanning [βmin,βmax][ _ , _ ], plus explicitly every βkk′ _k in the dataset. The monotonically increasing adequacy-over-fluency curves confirm that our reweighting mechanism effectively controls this balance. Furthermore, within a range around β0 _0 where ESSESS remains large, our method successfully preserves the original meta-evaluation’s MQM adherence and explainability by MQM. The strong correlation between ESSESS and both MQM adherence and explainability by MQM confirms ESSESS as a useful proxy for reliability, preventing severe evaluation distortion. Practically, ESSESS identifies the reachable range for controlling the adequacy–fluency balance without degrading meta-evaluation quality. The permissible deviation from β0 _0 is fundamentally constrained by the dataset itself. In datasets like jazh24, most βkk′ _k values concentrate within a narrow band, limiting reweighting capacity. Consequently, if a dataset lacks systems that intrinsically support a fluency-heavy evaluation, this imposes a strict limit on the achievable balance. Compared to Shayegh et al. (2025), our method consistently yields superior MQM adherence and/or explainability by MQM across all datasets, whether controlled by β or by adequacy-over-fluency preference. The only exceptions occur at extreme adequacy-over-fluency values (e.g., in zhen23, ende22, and ende24) which their synthetic method reaches but our reweighting cannot. However, explainability by MQM heavily drops for both methods in these regimes, rendering those extremes highly distorted and practically infeasible anyway. Overall, our method strictly improves upon Shayegh et al. (2025) in preserving the internal consistency of the meta-evaluation. Figure 4: Adequacy-over-fluency preference across the (ESS,β)(ESS,β) plane for ende21 (left) and ted_zhen (right). Each cell gives the mean and standard deviation over the weightings realizing that (ESS,β)(ESS,β), color-ramped from mean−stdmean-std at its bottom edge to mean+stdmean+std at its top. Grey cells are unreachable. Analyzing the markers in Figure 3 reveals a critical limitation of synthetic systems. In Shayegh et al. (2025)’s setups that retain real systems, explainability by MQM is preserved but MQM adherence degrades. Conversely, in their fully synthesized pools, MQM adherence is preserved while explainability by MQM collapses. Explainability by MQM measures the meta-evaluation’s response to scorer noise while preserving natural biases; it is informed, but not strictly dictated, by MQM. This failure point of fully synthetic pools confirms our hypothesis that synthetic systems are unrepresentative. This stems from their construction: segments are regrouped to maximize between-system variance. By the law of total variance Soch et al. (2025), since total pool variance is fixed, altering the between-system component inevitably alters the within-system component. This yields systems with unrealistically low score variances, being unrepresentative and also distorting SPA’s confidence estimations. In Appendix F, we report similar experiments using Pearson correlation as the meta-metric. The observed trends remain consistent, further supporting the effectiveness of our reweighting method. 4.4 Additional Analysis Variance Analysis. We investigate whether weight vectors with similar β values can yield substantially different adequacy–fluency balances due to uncontrolled factors, which would invalidate β as a reliable control parameter. The results in §4.3 are limited to PcollisP_collis-minimized weight vectors for each β, hiding any same-β variance. To address this, we sample numerous weight vectors on the simplex w≥0:⊤w=1w≥ 0:1 w=1, each realizing some β(w)β(w), ESS(w)ESS(w), and adequacy-over-fluency preference. By grouping these into buckets of (ESS,β)(ESS,β) pairs, we compare within-bucket preference variances (indicating β-independent shifts) against across-bucket average trends (indicating the β-effect), demonstrating the reliability of β as a control parameter. We restrict the experiment to w:ESS(w)∈[K2,K]w:ESS(w)∈[ K2,K], given the unreliability of weighting with lower ESSESS. We draw around 3.53.5M weight vectors per dataset. These sets provide varying β coverage across datasets. We partition β and ESSESS ranges into 88 equal-sized buckets each. We compute the adequacy-over-fluency preference for each weighting, presenting the results as a heatmap over this 8×88× 8 grid. Because these weightings are unoptimized, they remain unconstrained beyond the β and ESSESS. Consequently, a cell’s standard deviation strictly isolates the variation in SPA-based adequacy-over-fluency preference attributable to everything about w except β. Figure 5: Weighted meta-evaluation performance (SPAβ) of MetricX and xCOMET variants as a function of the adequacy–fluency balance parameter β. In zhen23 (left), scorer rankings change depending on the target balance. In ende24 (right), relative performance remains stable across the reachable range. β interval is bounded by ESS≥K−1ESS≥ K-1, where K is the total number of systems (1515 for zhen23, 1616 for ende24). Figures99 9 Results for some datasets are deferred to Appendix G due to space limits. Claims are made with respect to all datasets. 4 and 9 report the results. Each cell represents a pair of β and ESSESS buckets. Reading down any column, preference increases with β at every ESSESS. This trend holds for 99.1%99.1\% of vertically-adjacent cell pairs pooled across all datasets. Reading across rows, higher ESSESS sharply reduces variance: averaged over all 554554 horizontally-adjacent cell pairs, higher ESSESS exhibits 18.3%18.3\% lower same-β standard deviation, and the standard deviation decreases in 99.8%99.8\% of those pairs. Lastly, comparing means to standard deviations confirms that same-β variance does not obscure the β signal. Despite partial overlap between cells, the expected differences remain substantial. A one-sided Welch’s t-test (Welch, 1947) on every vertically-adjacent cell pair confirms this: 94.8%94.8\% of pairs show that the higher-β cell has a significantly higher preference (p<0.01p<0.01), and no pair is significant in the opposite direction. The mapping from β to the adequacy-over-fluency preference is dataset-dependent: not only does the preference measure preclude cross-dataset comparison by construction, identical β values also yield empirically different preference scores across datasets. Despite this limitation, the underlying relationship between β and the adequacy–fluency balance remains universal. Its direction, monotonicity, and robustness to same-β variation hold across every dataset tested, licensing β as a reliable control parameter. Weighted Meta-Evaluation. Having validated our reweighting mechanism, we now apply it to compare variants of the popular scorers MetricX (Juraska et al., 2024) and xCOMET (Guerreiro et al., 2024) across a sweep of β. Figure 5 reports the weighted soft pairwise accuracy (SPA) for the zhen23 and ende24 datasets.1010 10 We selected the two datasets empirically to best show two different possible scenarios: when the scorer ranking is or is not influenced by adequacy–fluency balance. The rankings in ende24 remain largely stable across the reachable β range. In contrast, the performance trajectories in zhen23 intersect a lot, demonstrating that the relative ranking of scorers can be highly sensitive to the adequacy–fluency balance. Our proposed parameter and reweighting tool explicitly describe and control this behavior. More results in the appendix. In Appendix B, we empirically evaluate the computational efficiency of our search algorithm, ablate its pruning mechanisms, and compare it against a numerical optimization baseline. In Appendix C, we show that meta-evaluation is sensitive to random system removal, and that adequacy–fluency imbalance is not the main cause of this instability. 5 Conclusion Machine translation meta-evaluation inherits the adequacy–fluency tradeoff, with its balance dictated by the specific combination of translation systems in the meta-evaluation data. Because this system set is a small, unrepresentative sample that fluctuates across datasets, the resulting adequacy–fluency balance is uncontrolled and subject to change. We exposed this balance as a tunable control parameter, β. To achieve a target β, we reweighted existing systems while minimizing distortion from uniform weighting, ensuring the evaluation pool remains real and representative. We supported this with an exact optimization algorithm, equipped with theoretical guarantees and pruning mechanisms, to efficiently compute these weights. To provide an internal diagnostic for meta-evaluation behavior, we introduced a scorer-augmentation framework that establishes a known relative identity for near-real scorers. Our experiments demonstrated that our reweighting method successfully controls the adequacy–fluency balance and preserves evaluation integrity, outperforming system-synthesis approaches. Finally, our analysis of popular scorers showed that scorer rankings can be sensitive to β. Calibration of β in practice. A specific numerical value of β does not yield an identical adequacy-over-fluency preference across different datasets. This stems from inherent variations in language pairs, the correlation between Adequacy MQM and Fluency MQM, base scorer biases, and other unaccounted variables. Thus, β functions as a directional control knob rather than an absolute metric. To determine the target β for a specific real-world scenario (e.g., technical versus creative translation), we recommend practitioners craft a few translation pairs that exhibit a marginal adequacy–fluency tradeoff, where one translation is slightly more adequate and the other slightly more fluent. After establishing which direction their specific use case prefers, they can sweep β until the meta-evaluation prioritizes scorers aligned with that preference. Limitations Our reweighting mechanism is strictly bounded by the intrinsic diversity of the underlying dataset. If a dataset lacks systems that exhibit isolated adequacy or fluency errors (e.g., jazh24), the reachable range of β is narrow. Forcing β beyond this range requires sparse weights, which reduces the effective sample size (ESSESS) and compromises the representativeness of the evaluation pool. Further expanding the admissible range for those datasets with limited flexibility remains a direction for future work. Moreover, our framework depends on the provided MQM annotations and the chosen adequacy–fluency taxonomy, introducing two unaccounted sources of variance. First, WMT human labels contain noise and individual annotator bias. Our method does not model this uncertainty, meaning it propagates directly into system-level scores and β. Second, categorizing errors as adequacy or fluency is subjective. A different taxonomy alters the score distributions, so the same numerical β will behave differently under a different decomposition. Accounting for annotation uncertainty and taxonomy dependence requires further research. Finally, controlling the adequacy–fluency balance does not resolve the broader instability of meta-evaluation. As demonstrated in Appendix C, scorer rankings remain sensitive to system selection even when β is fixed. Future work must identify the root causes of system-selection instability. We would like to investigate other axes of variation across translation systems that disrupt scorer rankings. Acknowledgements We thank Lili Mou for the insightful discussions that provided the initial motivation for this research, and Jan-Thorsten Peter for their valuable feedback on the final manuscript. We also extend our gratitude to the WMT reviewers, both for this manuscript and for our prior work last year, whose constructive feedback significantly shaped the direction of this study. This research was conducted independently and received no external funding. References Callison-Burch et al. (2007) C. Callison-Burch, C. Fordyce, P. Koehn, C. Monz, and J. Schroeder (Meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, C. Callison-Burch, P. Koehn, C. S. Fordyce, and C. Monz (Eds.), p. 136–158. External Links: Link Cited by: §1, §2.1. Deutsch et al. (2023) D. Deutsch, G. Foster, and M. Freitag Ties matter: meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), p. 12914–12929. External Links: Link, Document Cited by: §2.1. Deville and Särndal (1992) J. Deville and C. Särndal Calibration estimators in survey sampling. Journal of the American Statistical Association 87 (418), p. 376–382. External Links: Link Cited by: §1. Flamich et al. (2025) G. Flamich, D. Vilar, J. Peter, and M. Freitag You cannot feed two birds with one score: the accuracy–naturalness tradeoff in translation. arXiv preprint arXiv:2503.24013. External Links: Link Cited by: §1, §2.2, §3. Freitag et al. (2021a) M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan, and W. Macherey Experts, errors, and context: a large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics 9, p. 1460–1474. External Links: Link, Document Cited by: §1, §1, §2.1, §4.2. Freitag et al. (2024) M. Freitag, N. Mathur, D. Deutsch, C. Lo, E. Avramidis, R. Rei, B. Thompson, F. Blain, T. Kocmi, J. Wang, D. I. Adelani, M. Buchicchio, C. Zerva, and A. Lavie Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, p. 47–81. External Links: Link Cited by: §2.1, §4.2. Freitag et al. (2023) M. Freitag, N. Mathur, C. Lo, E. Avramidis, R. Rei, B. Thompson, T. Kocmi, F. Blain, D. Deutsch, C. Stewart, C. Zerva, S. Castilho, A. Lavie, and G. Foster Results of WMT23 metrics shared task: metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, p. 578–628. External Links: Link Cited by: §1, §2.1, §4.2. Freitag et al. (2022) M. Freitag, R. Rei, N. Mathur, C. Lo, C. Stewart, E. Avramidis, T. Kocmi, G. Foster, A. Lavie, and A. F. T. Martins Results of WMT22 metrics shared task: stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), p. 46–68. External Links: Link Cited by: §4.2. Freitag et al. (2021b) M. Freitag, R. Rei, N. Mathur, C. Lo, C. Stewart, G. Foster, A. Lavie, and O. Bojar Results of the WMT21 metrics shared task: evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, p. 733–774. External Links: Link Cited by: §4.2. Gnanadesikan and Kettenring (1972) R. Gnanadesikan and J. R. Kettenring Robust estimates, residuals, and outlier detection with multiresponse data. Biometrics 28 (1), p. 81–124. External Links: Link Cited by: Figure 1. Guerreiro et al. (2024) N. M. Guerreiro, R. Rei, D. v. Stigt, L. Coheur, P. Colombo, and A. F. T. Martins Xcomet: transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics 12, p. 979–995. External Links: Link, https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00683/2468704/tacl_a_00683.pdf Cited by: §1, §4.4. Han et al. (2013) A. L. Han, D. F. Wong, L. S. Chao, L. He, Y. Lu, J. Xing, and X. Zeng Language-independent model for machine translation evaluation with reinforced factors. In Proceedings of Machine Translation Summit XIV: Posters, External Links: Link Cited by: Figure 2. Juraska et al. (2024) J. Juraska, D. Deutsch, M. Finkelstein, and M. Freitag MetricX-24: the Google submission to the WMT 2024 Metrics Shared Task. In Proceedings of the Ninth Conference on Machine Translation, B. Haddow, T. Kocmi, P. Koehn, and C. Monz (Eds.), p. 492–504. External Links: Link, Document Cited by: §1, Figure 2, §4.4. Kendall et al. (1938) M. G. Kendall et al. A new measure of rank correlation. Biometrika 30, p. 81–93. Cited by: Appendix C. Knowles (2021) R. Knowles On the stability of system rankings at WMT. In Proceedings of the Sixth Conference on Machine Translation, p. 464–477. External Links: Link Cited by: §2.1. Kocmi et al. (2025) T. Kocmi, S. Agrawal, E. Artemova, E. Avramidis, E. Briakou, P. Chen, M. Fadaee, M. Freitag, R. Grundkiewicz, Y. Hou, P. Koehn, J. Kreutzer, S. Mansour, S. Perrella, L. Proietti, P. Riley, E. Sánchez, P. Schmidtova, M. Shmatova, and V. Zouhar Findings of the WMT25 multilingual instruction shared task: persistent hurdles in reasoning, generation, and evaluation. In Proceedings of the Tenth Conference on Machine Translation, External Links: Link Cited by: §2.1. Kocmi et al. (2021) T. Kocmi, C. Federmann, R. Grundkiewicz, M. Junczys-Dowmunt, H. Matsushita, and A. Menezes To ship or not to ship: an extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, p. 478–494. External Links: Link Cited by: §1, §2.1. Kocmi and Federmann (2023) T. Kocmi and C. Federmann Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, p. 193–203. External Links: Link Cited by: §2.1. Lavie et al. (2025) A. Lavie, G. Hanneman, S. Agrawal, D. Kanojia, C. Lo, V. Zouhar, F. Blain, C. Zerva, E. Avramidis, S. Deoghare, A. Sindhujan, J. Wang, D. I. Adelani, B. Thompson, T. Kocmi, M. Freitag, and D. Deutsch Findings of the WMT25 shared task on automated translation evaluation systems: linguistic diversity is challenging and references still help. In Proceedings of the Tenth Conference on Machine Translation, External Links: Link Cited by: footnote 7. Lommel et al. (2014) A. Lommel, H. Uszkoreit, and A. Burchardt Multidimensional quality metrics (MQM): a framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, p. 455–463. External Links: Document Cited by: §2.2. Macháček and Bojar (2013) M. Macháček and O. Bojar Results of the WMT13 metrics shared task. In Proceedings of the Eighth Workshop on Statistical Machine Translation, p. 45–51. External Links: Link Cited by: §2.1. Macháček and Bojar (2014) M. Macháček and O. Bojar Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, p. 293–301. External Links: Link Cited by: §2.1. Mariana et al. (2015) V. Mariana, T. Cox, and A. Melby The multidimensional quality metrics (MQM) framework: a new framework for translation quality assessment. The Journal of Specialised Translation, p. 137–161. External Links: Link Cited by: §2.1. Martindale and Carpuat (2018) M. Martindale and M. Carpuat Fluency over adequacy: a pilot study in measuring user trust in imperfect MT. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), C. Cherry and G. Neubig (Eds.), p. 13–25. External Links: Link Cited by: §2.2. Mathur et al. (2020) N. Mathur, T. Baldwin, and T. Cohn Tangled up in BLEU: reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 4984–4997. External Links: Link Cited by: §2.1. Moré and Sorensen (1983) J. J. Moré and D. C. Sorensen Computing a trust region step. SIAM Journal on scientific and statistical computing 4 (3), p. 553–572. Cited by: §A.5. Nocedal and Wright (2006) J. Nocedal and S. J. Wright Numerical optimization. Springer. Cited by: §A.5. Park and Boyd (2017) J. Park and S. Boyd General heuristics for nonconvex quadratically constrained quadratic programming. arXiv preprint arXiv:1703.07870. Cited by: §A.5. Perrella et al. (2024) S. Perrella, L. Proietti, A. Scirè, E. Barba, and R. Navigli Guardians of the machine translation meta-evaluation: sentinel metrics fall in!. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, p. 16216–16244. External Links: Link, Document Cited by: §2.1. Riley et al. (2024) P. Riley, D. Deutsch, G. Foster, V. Ratnakar, A. Dabirmoghaddam, and M. Freitag Finding replicable human evaluations via stable ranking probability. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4908–4919. External Links: Link Cited by: §2.1. Shayegh et al. (2025) B. Shayegh, J. Peter, D. Vilar, T. Domhan, J. Juraska, M. Freitag, and L. Mou Feeding two birds or favoring one? adequacy–fluency tradeoffs in evaluation and meta-evaluation of machine translation. In Proceedings of the Tenth Conference on Machine Translation, p. 269–285. External Links: Link Cited by: Appendix F, §1, §1, §1, §2.2, Figure 2, Figure 3, §4.1, §4.3, §4.3, §4, footnote 3, footnote 6. Soch et al. (2025) J. Soch, K. Sarıtaş, P. Monticone, T. J. Faulkenberry, O. A. Martin, A. Kipnis, S. Balkus, C. Allefeld, H. Atze, A. Knapp, et al. Statproofbook/statproofbook. github. io: statproofbook 2024. Zenodo. Cited by: §4.3. Thompson et al. (2024) B. Thompson, N. Mathur, D. Deutsch, and H. Khayrallah Improving statistical significance in human evaluation of automatic metrics via soft pairwise accuracy. In Proceedings of the Ninth Conference on Machine Translation, B. Haddow, T. Kocmi, P. Koehn, and C. Monz (Eds.), p. 1222–1234. External Links: Link, Document Cited by: §1, §2.1. Tokdar and Kass (2010) S. T. Tokdar and R. E. Kass Importance sampling: a review. Wiley Interdisciplinary Reviews: Computational Statistics 2, p. 54–60. Cited by: §1. Welch (1947) B. L. Welch The generalization of ‘student’s’ problem when several different population variances are involved. Biometrika 34 (1/2), p. 28–35. External Links: Link Cited by: §4.4. White et al. (1994) J. S. White, T. O’Connell, and F. O’Mara The ARPA MT evaluation methodologies: evolution, lessons, and future approaches. In Proceedings of the First Conference of the Association for Machine Translation in the Americas, Columbia, Maryland, USA. Cited by: §2.2. Wu (2007) H. Wu The karush–kuhn–tucker optimality conditions in an optimization problem with interval-valued objective function. European Journal of Operational Research 176, p. 46–59. External Links: Link Cited by: §A.5. Zouhar et al. (2025) V. Zouhar, P. Cui, and M. Sachan How to select datapoints for efficient human evaluation of NLG models?. Transactions of the Association for Computational Linguistics. External Links: Document Cited by: §2.1. Appendix A Proofs A.1 Proof of Theorem 1 See 1 Proof. The forward direction is given by Theorem 3 applied to the full support. We now prove the reverse direction: Any β^∈[βmin,βmax] β∈[ _ , _ ] is a feasible target for Problem P . β(w)β(w) is continuous on the relative interior of S’s simplex face and that face is connected. Moreover, for two systems k1,k2:βmin(S)=βk1k2k_1,k_2: _ (S)= _k_1k_2, a weight vector w^=k≠k1,k2:wk=0 w=\k≠ k_1,k_2:w_k=0\ drives β(w)=βmin(S)β(w)= _ (S); symmetrically for βmax(S) _ (S). So the image is a continuous interval with infimum βmin(S) _ (S) and supremum βmax(S) _ (S). ∎ A.2 Proof of Theorem 2 See 2 . [Part (a)] We prove this part by deriving the explicit polynomial to bound the stationary points. By (11) and Proposition 1: b(w,β^)= b(w, β)= (1−β^)∑k<k′wkwk′(ak−ak′)2 (1- β) _k<k w_kw_k (a_k-a_k )^2 −β^∑k<k′wkwk′(fk−fk′)2 - β _k<k w_kw_k (f_k-f_k )^2 (18) = = (1−β^)(∑kwkak2−w⊤aa⊤w) (1- β)( _kw_ka_k^2-w a w) −β^(∑kwkfk2−w⊤ff⊤w) - β( _kw_kf_k^2-w f w) (19) = = w⊤(β^ff⊤−(1−β^)aa⊤)w w ( βf -(1- β)a )w +∑kwk((1−β^)ak2−β^fk2) + _kw_k ((1- β)a_k^2- βf_k^2 ) (20) = = w⊤Hw+w⊤g w Hw+w g (21) where H=β^ff⊤−(1−β^)aa⊤H= βf -(1- β)a and gk=(1−β^)ak2−β^fk2g_k=(1- β)a_k^2- βf_k^2, yielding a quadratic in w. Eliminate the budget constraint ⊤w=11 w=1 using the affine transformation w=u0+Nu w=u_0+Nu (22) where u0u_0 is the uniform weighting on S and N has orthonormal columns spanning v:⊤v=0\v:1 v=0\. Substituting (22) into (21) yields a single quadratic in the free variable u: b(w,β^)=b~(u,β^)=u⊤Mu+u⊤r+c=0 b(w, β)= b(u, β)=u Mu+u r+c=0 (23) where M=N⊤HNM=N HN, r=N⊤(2Hu0+g)r=N (2Hu_0+g), and c=b(u0,β^)c=b(u_0, β). We can restate the problem in the free variable u: minu _u 12‖u‖2+const.s.t.b~(u,β^)=0 \ 12\ \|u\|^2+const.\;\;s.t.\;\; b(u, β)=0 (S(u)P_S^(u)) A point u is a stationary point of ( P S ( u ) ) if it is feasible and satisfies ∇u(12‖u‖2+η⋅b~(u,β^))=0 _u ( 12\|u\|^2+η· b(u, β) )=0 (24) for a multiplier η∈ℝη . Taking the derivative: u+η(2Mu+r) u+η(2Mu+r) =0 =0 (25) (I+2ηM)u (I+2η M)\,u =−ηr =-η r (26) We call the support S degenerate if all its pairwise balances βkk′ _k , k,k′∈Sk,k ∈ S, coincide. We consider two scenarios: [Case 1: S is degenerate] Let β¯ β be the common value of the pairwise balances in S. Because S is admissible, Theorem 3 forces β^=βmin(S)=βmax(S)=β¯ β= _ (S)= _ (S)= β. For any pair k,k′∈Sk,k ∈ S we have (ak−ak′)2+(fk−fk′)2>0(a_k-a_k )^2+(f_k-f_k )^2>0, as no two systems share the same (ak,fk)(a_k,f_k); together with β^=βkk′ β= _k and (12) this gives (1−β^)(ak−ak′)2−β^(fk−fk′)2=0 (1- β)(a_k-a_k )^2- β(f_k-f_k )^2=0 (27) Substituting into (18) b(w,β^)= b(w, β)= ∑k<k′wkwk′((1−β^)(ak−ak′)2−β^(fk−fk′)2) _k<k w_kw_k ((1- β)(a_k-a_k )^2- β(f_k-f_k )^2 ) =∑k<k′wkwk′⋅0=0 = _k<k w_kw_k · 0=0 (28) for every w. So the quadratic b~(⋅,β^) b(·, β) vanishes identically and M=0M=0, r=0r=0, c=0c=0 in (23). The stationarity condition (26) reads u=0u=0, so S carries exactly one stationary point: the uniform weighting u0u_0. It is feasible, minimizes the strictly convex objective 12‖u‖2 12\|u\|^2, and is the unique global optimum on S. [Case 2: S is non-degenerate] The pairwise balances in S are not all equal. Let a~ a and f~ f be the score vectors restricted to S and centered by their uniform means over S: a~k=ak−μ(a,u0) a_k=a_k-μ(a;u_0) (29) f~k=fk−μ(f,u0) f_k=f_k-μ(f;u_0) (30) for k∈Sk∈ S. These are linearly independent: were f~=κa~ f=κ a for some κ∈ℝκ , every pair would satisfy (fk−fk′)=κ(ak−ak′)(f_k-f_k )=κ(a_k-a_k ) and hence βkk′=(1+κ2)−1 _k =(1+κ^2)^-1, contradicting non-degeneracy; the same holds with the roles of a~ a and f~ f exchanged. Within the zero-sum subspace defined by N, M acts as the difference of two independent rank-1 outer products: β^f~f~⊤ β f f and (1−β^)a~a~⊤(1- β) a a . By definition of β, we have 0<β^<10< β<1. Therefore, this difference provides one strictly positive and one strictly negative direction. Consequently, M is indefinite with rank two, possessing exactly two nonzero eigenvalues: θ1>0>θ2 _1>0> _2 (31) We use this structure to solve for the stationary candidates. Projecting (26) onto the eigen basis vjj=1|S|−1\v_j\_j=1^|S|-1 of M, and defining the coordinate components uj=vj⊤u_j=v_j u and rj=vj⊤r_j=v_j r, splits it into independent scalar equations of the form uj(1+2ηθj)=−ηrj u_j(1+2η _j)=-η r_j (32) The system is singular exactly at the two poles λj=−1/(2θj) _j=-1/(2 _j) for j∈1,2j∈\1,2\. We categorize the candidates by resolving across three subcases for η: [Case 2.1: η=λjη= _j and rj≠0r_j≠ 0] The j-th equation reads uj⋅0=−λjrj≠0u_j· 0=- _jr_j≠ 0. The system is inconsistent, yielding zero candidates. [Case 2.2: η=λjη= _j and rj=0r_j=0] The j-th equation is 0=00=0, leaving the coordinate uju_j free. For every other coordinate m≠jm≠ j, the coefficient (1+2λjθm)=1−θmθj(1+2 _j _m)=1- _m _j is strictly non-zero: θm=0 _m=0 for flat directions m≥3m≥ 3 and θ3−jθj<0 _3-j _j<0 by (31). Therefore, every um:m≠ju_m:m≠ j is uniquely fixed to um=−λjrm1+2λjθm u_m= - _jr_m1+2 _j _m (33) Defining u∘u as this baseline vector with uj∘=0u_j =0, the full quadratic constraint separates into: b~(u,β^)=θjuj2+rjuj+b~(u∘,β^)=0 b(u, β)= _ju_j^2+r_ju_j+ b(u , β)=0 (34) Because rj=0r_j=0, the linear term vanishes, leaving θjuj2=−b~(u∘,β^) _ju_j^2=- b(u , β). This yields at most two candidates: uj=±−b~(u∘,β^)/θj u_j=± - b(u , β)/ _j (35) [Case 2.3: η≠λjη≠ _j] In this case, 1+2ηθj≠01+2η _j≠ 0, which gives uj(η)=−ηrj1+2ηθj u_j(η)= -ηr_j1+2η _j (36) which simplifies to −ηrj-ηr_j for j≥3j≥ 3. We substitute these components into (23). When j≥3j≥ 3, the quadratic terms vanish by θj=0 _j=0, and the linear terms sum to: ∑j≥3rj(−ηrj)=−η∑j≥3rj2 _j≥ 3r_j(-ηr_j)=-η _j≥ 3r_j^2 (37) For j∈1,2j∈\1,2\, substituting uj(η)u_j(η) into its quadratic and linear terms yields: θj _j uj(η)2+rjuj(η) u_j(η)^2+r_ju_j(η) =θj(−ηrj1+2ηθj)2+rj(−ηrj1+2ηθj) = _j ( -ηr_j1+2η _j )^2+r_j ( -ηr_j1+2η _j ) (38) =−ηrj2(1+ηθj)(1+2ηθj)2 =-ηr_j^2\ (1+η _j)(1+2η _j)^2 (39) Combining the constant c, (37), and (39) for the two j values, (23) becomes: c−η[∑j=1,2rj2(1+θjη)(1+2ηθj)2+∑j≥3rj2]=0 c-η [ _j=1,2 r_j^2(1+ _jη)(1+2η _j)^2+ _j≥ 3r_j^2 ]=0 (40) Clearing denominators by multiplying through by the squared characteristic polynomial (1+2ηθ1)2(1+2ηθ2)2(1+2η _1)^2(1+2η _2)^2 yields a polynomial equation ΦS,β^(η)=0 _S, β(η)=0, where ΦS,β^(η)= _S, β(η)=\; c(1+2ηθ1)2(1+2ηθ2)2 c\,(1+2η _1)^2(1+2η _2)^2 −ηr12(1+θ1η)(1+2ηθ2)2 -η\,r_1^2\,(1+ _1η)(1+2η _2)^2 −ηr22(1+θ2η)(1+2ηθ1)2 -η\,r_2^2\,(1+ _2η)(1+2η _1)^2 −η∑j≥3rj2(1+2ηθ1)2(1+2ηθ2)2 -η _j≥ 3r_j^2\,(1+2η _1)^2(1+2η _2)^2 (41) has degree at most five. It remains to bound the number of candidates. We proceed in three steps. First, we evaluate ΦS,β _S, β at λ1 _1 and λ2 _2. We show that λj _j is a root only when rj=0r_j=0 (Case 2.2), and that it is then a root of multiplicity two. Second, we count the candidates when ΦS,β _S, β is not the zero polynomial. A root η∉λ1,λ2η∉\ _1, _2\ gives one candidate (Case 2.3) and costs one of the five roots, while a root at λj _j gives two candidates and costs two of them by the first step. Hence the bound of five holds. Third, we treat the case where ΦS,β _S, β is the zero polynomial. In this case, counting roots does not apply and we solve the stationarity condition (26) directly. [Poles of ΦS,β _S, β] At η=λ1η= _1 the factor (1+2ηθ1)(1+2η _1) vanishes, killing every term of (41) except the second one, so ΦS,β^(λ1) _S, β( _1) =−λ1r12(1+θ1λ1)(1+2λ1θ2)2 =- _1r_1^2(1+ _1 _1)(1+2 _1 _2)^2 =−λ1r122(1−θ2θ1)2 =- _1r_1^22 (1- _2 _1 )^2 (42) using λ1=−1/(2θ1) _1=-1/(2 _1), and symmetrically for λ2 _2 with the indices 11 and 22 exchanged. Because λj≠0 _j≠0 and θ1θ2<0 _1 _2<0 by (31), λj _j is a root of ΦS,β _S, β if and only if rj=0r_j=0, which is exactly Case 2.2; and when rj=0r_j=0, every remaining term of (41) carries the factor (1+2ηθj)2(1+2η _j)^2, making λj _j a double root. [Counting when ΦS,β^≢0 _S, β ≡ 0] The polynomial has at most five roots counted with multiplicity. Each root η∉λ1,λ2η∉\ _1, _2\ contributes exactly one candidate through Case 2.3. Each λj _j that is a root uses two roots and contributes at most two candidates through (35). Summing over all configurations, the number of candidates is at most five. [Counting when ΦS,β^≡0 _S, β≡0] We have c=ΦS,β^(0)=0c= _S, β(0)=0 and r1=r2=0r_1=r_2=0 by (42). The remaining (last) term of (41) forces ∑j≥3rj2=0 _j≥ 3r_j^2=0 and hence r=0r=0. The stationarity condition (26) becomes (I+2ηM)u=0(I+2η M)u=0, which in the eigen coordinates reads uj(1+2ηθj)=0u_j(1+2η _j)=0 for every j. If η∉λ1,λ2η∉\ _1, _2\, all the coefficients 1+2ηθj1+2η _j are nonzero and u=0u=0. If η=λjη= _j for some j∈1,2j∈\1,2\, the j-th coefficient vanishes and leaves uju_j free, while every other coefficient is nonzero as shown in Case 2.2; the solution is thus u=ujvju=u_jv_j, a multiple of the j-th eigenvector of M. Feasibility (23) then gives b~(ujvj,β^)=θjuj2+ujrj+c=θjuj2=0 b(u_jv_j, β)= _ju_j^2+u_j\,r_j+c= _ju_j^2=0 (43) using r=0r=0 and c=0c=0. θj≠0 _j≠ 0 forces uj=0u_j=0. So u=0u=0 in every case, and S carries the single stationary point u0u_0. In all scenarios S carries at most five stationary points, each obtained in closed form from the roots of a polynomial of degree at most five. [Part (b)] By assumption, w⋆w is a global optimum of ( P ) with support S=S(w⋆)S=S(w ), and S is admissible by Theorem 3. Because wk⋆>0w _k>0 for all k∈Sk∈ S, the non-negativity constraints are inactive: w⋆w locally minimizes 12‖w‖2 12\|w\|^2 subject to ⊤w=11 w=1 and b(w,β^)=0b(w, β)=0 on S. Write u⋆u for its reduced coordinate, w⋆=u0+Nu⋆w =u_0+Nu . We consider two scenarios: [The constraint gradients are linearly independent] The linear independence constraint qualification holds at w⋆w , so u⋆u is a stationary point of ( P S ( u ) ) and is enumerated by Part (a). [The constraint gradients are linearly dependent] Dependence forces ∇wb(w⋆,β^)=μ _wb(w , β)= 1 for some μ∈ℝμ . The chain rule gives the reduced gradient ∇ub~(u⋆,β^) _u b(u , β) =∇u[b(u0+Nu,β^)]|u=u⋆ = _u [b(u_0+Nu, β) ] |_u=u =N⊤∇wb(w⋆,β^)=μN⊤ =N _wb(w , β)=μ N 1 (44) which is equal to zero since the columns of N are orthogonal to 1. Together with feasibility b~(u⋆,β^)=0 b(u , β)=0, expanding the quadratic (23) around u⋆u along a direction δ with step size t gives b~(u⋆+tδ,β^) b(u +tδ, β) =b~(u⋆,β^)⏟=0+t∇ub~(u⋆,β^)⊤δ⏟=0+t22δ⊤∇u2b~(u⋆,β^)⏟=2Mδ = b(u , β)_=0+ t\, _u b(u , β) δ_=0+ t^22δ _u^2 b(u , β)_=2Mδ =t2δ⊤Mδ =t^2δ Mδ (45) Call δ a null direction if δ⊤Mδ=0δ Mδ=0. Along such a direction the line u(t)=u⋆+tδu(t)=u +tδ satisfies the constraint b~(⋅,β^)=0 b(·, β)=0 for every t∈ℝt . Because wk⋆>0w _k>0 for all k∈Sk∈ S, the points of this line with |t||t| small are feasible for ( P ), so t=0t=0 is a local minimizer of φ(t) (t) =‖u⋆+tδ‖2 =\|u +tδ\|^2 (46) =‖u⋆‖2+2t(u⋆⊤δ)+t2‖δ‖2 =\|u \|^2+2t\,(u δ)+t^2\|δ\|^2 (47) This function is a convex quadratic, so φ′(0)=0 (0)=0, which asserts u⋆⊤δ=0 u δ=0 (48) for every null direction δ. If S is degenerate, then M=0M=0 and every direction is null, so (48) forces u⋆=0u =0. Otherwise, M is indefinite of rank two by (31), and its null directions span the reduced space: they contain the kernel directions vjj≥3\v_j\_j≥ 3 together with −θ2v1±θ1v2 - _2\,v_1± _1\,v_2, whose span is all of ℝ|S|−1R^|S|-1. Orthogonality (48) to a spanning set again forces u⋆=0u =0. In both cases w⋆w is the uniform weighting on S. Moreover r=−2Mu⋆=0r=-2Mu =0 by (44) and c=b~(0,β^)=0c= b(0, β)=0 by feasibility, so every term of (41) vanishes and ΦS,β^≡0 _S, β≡ 0. Part (a) enumerates u⋆=0u =0 as the single stationary point on S in this situation, both when S is degenerate and when ΦS,β^≡0 _S, β≡ 0. The global optimum is therefore contained within the candidate set enumerated in Part (a). ∎ A.3 Proof of Theorem 3 See 3 Proof. For any wSw_S with at least two positive values, (* ‣ 3.1) gives β^=β(wS)=σ(a,wS)2σ(a,ws)2+σ(f,ws)2 β=β(w_S)= σ(a;w_S)^2σ(a;w_s)^2+σ(f;w_s)^2 (49) Given wS=wkkw_S=\w_k\_k, by Proposition 1 we have β^=∑k<k′wkwk′(ak−ak′)2∑k<k′wkwk′[(ak−ak′)2+(fk−fk′)2] β= _k<k w_kw_k (a_k-a_k )^2 _k<k w_kw_k [(a_k-a_k )^2+(f_k-f_k )^2 ] (50) Letting pkk′=wkwk′[(ak−ak′)2+(fk−fk′)2]∑l<l′wlwl′[(al−al′)2+(fl−fl′)2] p_k = w_kw_k [(a_k-a_k )^2+(f_k-f_k )^2 ] _l<l w_lw_l [(a_l-a_l )^2+(f_l-f_l )^2 ] (51) with pkk′≥0p_k ≥ 0 and ∑k<k′pkk′=1 _k<k p_k =1, we substitute (12): β^=∑k<k′pkk′βkk′ β= _k<k p_k _k (52) Therefore, β β is a strict convex combination of pairwise balances over non-zero weights βkk′wkwk′>0\ _k \_w_kw_k >0, hence strictly between their smallest and their largest, unless they all coincide, in which case it equals their common value. We have βkk′wkwk′>0⊆βkk′kk′∈[βmin(S),βmax(S)] \ _k \_w_kw_k >0 \ _k \_k ∈[ _ (S), _ (S)] (53) thus βmin(S)≤β^≤βmax(S) _ (S)≤ β≤ _ (S). ∎ A.4 Proof of Theorem 4 See 4 Proof. S is the support of w, meaning wk=0w_k=0 for k∉Sk∉ S. The collision probability is defined as: Pcollis(w)=∑kwk2=∑k∈Swk2 _collis(w)= _kw_k^2= _k∈ Sw_k^2 (54) By Cauchy–Schwarz inequality: (∑k∈Swk⏟=1⋅1)2≤(∑k∈Swk2⏟Pcollis(w))(∑k∈S12⏟|S|) ( _k∈ Sw_k_=1· 1 )^2≤ ( _k∈ Sw_k^2_P_collis(w) ) ( _k∈ S1^2_|S| ) (55) Equality holds in the Cauchy–Schwarz inequality iff the vectors (wk)k∈S(w_k)_k∈ S and (1)k∈S(1)_k∈ S are proportional, meaning wkw_k is a constant across all k∈Sk∈ S. Given ∑k∈Swk=1 _k∈ Sw_k=1, this forces wk=1/|S|w_k=1/|S| for all k∈Sk∈ S. Therefore, equality holds iff w is uniform on S. Dividing both sides of (55) by |S||S| yields the claim. ∎ β-sweep range: β^∈[β0−0.05,β0+0.05] β∈[ _0-0.05,\ _0+0.05] Full range β^∈[βmin,βmax] β∈[ _ , _ ] Config # Supports # Candidates Success % # FLOPs # Supports # Candidates Success % # FLOPs Theorem 3 4 5 Our algorithm ✓ ✓ ✓ 50 50 100% 1.3M 951.4K 2.1M 100% 26,304.8M ✓ ✓ – 50 50 100% 1.3M 951.7K 2.1M 100% 26,313.2M – ✓ ✓ 50 50 100% 1.3M 1.0M 2.3M 100% 28,146.9M ✓ – ✓ 50 50 100% 1.3M 1.8M 3.9M 100% 48,911.1M – ✓ – 50 50 100% 1.3M 1.0M 2.3M 100% 28,155.4M – – ✓ 50 50 100% 1.3M 1.9M 4.3M 100% 52,520.6M ✓ – – 3.2M 4.8M 100% 87.4B 3.0M 5.8M 100% 82,239.1M – – – 3.3M 4.9M 100% 88.8B 3.1M 6.2M 100% 86,307.4M N Tolerance Numerical 22 10−310^-3 — — 18% 1.6M — — 16% 1.6M 33 10−310^-3 — — 80% 9.3M — — 62% 9.3M 44 10−310^-3 — — 100% 44.4M — — 88% 44.4M 55 10−310^-3 — — 100% 177.5M — — 100% 177.5M Table 1: Ablation study of our algorithm on the ende24 dataset (K=16K=16). We sweep β β across 50 values in each of two distinct regimes. # Supports (# Candidates) is the number of supports (candidates) visited prior to termination. Theorem β-sweep range 3 4 5 [β0−0.05,β0+0.05][ _0-0.05,\ _0+0.05] 0 0 50 [βmin,βmax][ _ , _ ] 65.5K 29 19 Table 2: Number of times each of Theorems 3–5 fires over a 50-point β-sweep. A.5 Proof of Theorem 5 See 5 Proof. The balance constraint b(w,β^)=0b(w, β)=0 is governed by an indefinite quadratic matrix H=β^ff⊤−(1−β^)aa⊤H= βf -(1- β)a , according to (21) in the proof of Theorem 2. This makes the constraint domain non-convex. Therefore, we verify the Karush–Kuhn–Tucker (Wu, 2007, KKT;) conditions, as well as the second-order sufficiency criteria adapted for non-convex quadratically constrained quadratic program (Park and Boyd, 2017, QCQPs;). [KKT Conditions] Consider the Lagrangian for Problem ( P ): ℒ (w,η,λ,γ)= (w,η,λ,γ)= (ℒL) 12∥w∥+2ηb(w,β^)+λ⊤w=1−∑k=1Kγkwk 12\|w\|^2+η\,b(w, β)+ 1 w=1- _k=1^K _kw_k where η,λ∈ℝη,λ , and γk≥0 _k≥ 0 are the dual variables for the non-negativity constraints wk≥0w_k≥ 0 so that γkwk=0 _kw_k=0 for all k (complementary slackness). Differentiating ( L ) with respect to each weight wkw_k yields the general stationarity condition ∂ℒ∂wk=wk+η∂b(w,β^)∂wk+λ−γk=0 ∂ w_k=w_k+η ∂ b(w, β)∂ w_k+λ- _k=0 (56) for all k. To validate this, we split the analysis into two parts based on whether k∈S(w)k∈ S(w): [Case k∈S(w)k∈ S(w)] By definition, wk>0w_k>0. Complementary slackness forces the corresponding dual variable γk=0 _k=0. Substituting this into (56): wk+η∂b(w,β^)∂wk+λ=0for k∈S(w) w_k+η ∂ b(w, β)∂ w_k+λ=0 \ k∈ S(w) (57) which is given by condition (i). [Case k∉S(w)k ∈ S(w)] By substituting wk=0w_k=0 into (56) and keeping only γk _k on one side: γk=λ+η∂b(w,β^)∂wk≥0for k∉S(w) _k=λ+η ∂ b(w, β)∂ w_k≥ 0 k ∈ S(w) (58) which is given by condition (i). Note that the inequality is given by γk _k’s definition. Together, (57) and (58) satisfy all KKT conditions for Problem ( P ), which are necessary but not sufficient. Next, we verify second-order sufficiency criteria. [Second-order sufficiency criteria] We have the Hessian of ( L ) ∇w2ℒ=I+2ηH _w^2L=I+2η H (59) where H is the indefinite matrix H=β^ff⊤−(1−β^)aa⊤H= βf -(1- β)a , following (21). For w to be a global minimizer, the Lagrangian Hessian must be positive semi-definite along all directions v that preserve the equality constraints ⊤v=01 v=0 and ∇b(w,β^)⊤v=0∇ b(w, β) v=0 (Moré and Sorensen, 1983). Through standard null-space elimination (Nocedal and Wright, 2006), this requirement projects onto the zero-sum subspace, demanding v⊤(I+2ηH)v≥0 v (I+2η H )v≥ 0 (60) for all vector directions v such that ⊤v=01 v=0. To validate this condition without explicitly constructing high-dimensional sweeps, we invoke the structural reduction detailed in (22–23) in the proof of Theorem 2. As shown in that proof, eliminating the budget constraint via the affine transformation w=u0+Nuw=u_0+Nu restricts the quadratic form to the zero-sum subspace via the reduced matrix M=N⊤HNM=N HN. According to Theorem 2’s proof, Case 2, M acts as the difference of two rank-1 outer products and has rank two on the relevant subspace with two non-zero eigenvalues θ1>0>θ2 _1>0> _2. Projecting the quadratic form onto the orthonormal basis of this two-dimensional span a~,f~\ a, f\ transforms the test of (60) into the explicit 2×22× 2 symmetric matrix test defined in condition (i). That is, condition (i) requires that H(η,β^)⪰0H(η, β) 0, and requiring H(η,β^)⪰0H(η, β) 0 is strictly equivalent to v⊤(I+2ηH)v≥0v (I+2η H)v≥ 0 for all v with ⊤v=01 v=0, which satisfies the second-order sufficiency criteria. Since w satisfies KKT conditions across both active and inactive supports via (i) and (i), and the second-order subspace sufficiency condition via (i), w is certified to be a global optimum of Problem ( P ). ∎ Appendix B Efficiency and Ablation Analysis We evaluate the computational efficiency of Algorithm 1 and ablate its three pruning mechanisms using the ende24 dataset1111 11 The choice of this dataset is due to its large K. The trends and conclusions are similar across the datasets. (K=16K=16). We sweep β β across 50 values in two regimes: a narrow band around the natural balance β0 _0, and the full reachable range [βmin,βmax][ _ , _ ]. The former represents the practical use case, as performance is more reliable here (§4.3) and minor adjustments reflect realistic demands. The latter serves as an extreme regime to stress-test the algorithm. We compare against a numerical optimization baseline using varying grid resolutions N and a fixed tolerance of 10−310^-3. Tables 1 and 2 summarize the results. Near natural balance. When the target β β is close to β0 _0, the optimal weighting remains close to uniform. Table 2 shows that Theorem 5 (exit certificate) triggers immediately on the full support for all 50 targets. Consequently, the search evaluates exactly one support per target. Table 1 demonstrates the resulting efficiency: solving the narrow sweep requires only 1.3M FLOPs, substantially faster than the numerical baseline required to guarantee 100% success, while strictly guaranteeing global optimality. Without early exit, the algorithm exhaustively evaluates over 3.2M supports, increasing FLOPs by nearly five orders of magnitude. Full range. Reaching extreme β β targets requires sparse weight vectors. In this regime, Theorem 3 (admissible supports) drives the pruning, firing over 65.5K times (Table 2) to safely skip infeasible branches. The exact algorithm demands more computation than the numerical baseline across the full range, which is the cost of guaranteeing the global minimum. The ablation confirms that combining all pruning theorems reduces the full-range computational cost by more than 3×3× compared to the unpruned exact search. Figure 6: Leave-one-out stability with and without fixing β. The horizontal red line corresponds to the baseline values in Table 3. The blue curve shows the pooled pairwise Kendall’s τ correlation of scorer rankings after reweighting each subset to the fixed β. Curve intensity represents ESSESS (reliability). Dataset # Systems Cross-LOO Kendall’s τ ende21 13 0.8241 zhen21 13 0.6659 ted_ende 13 0.8634 ted_zhen 13 0.8774 ende22 13 0.7497 zhen22 14 0.8864 zhen23 15 0.8560 heen23 12 0.8470 ende24 16 0.8797 enes24 13 0.8815 jazh24 13 0.7931 Table 3: Pooled pairwise Kendall’s τ correlation of scorer rankings across leave-one-out (LOO) system subsets for each dataset. The sub-optimal correlation values indicate that meta-evaluation rankings are sensitive to system selection. Appendix C Does Controlling Adequacy–Fluency Balance Stabilize Meta-Evaluation? Meta-evaluation rankings vary across datasets. While intrinsic differences partially explain this, instability also stems directly from system selection within a single dataset. We quantify this by computing the pooled pairwise Kendall’s τ (Kendall and others, 1938) of scorer rankings across all leave-one-out (LOO) system subsets for each dataset. Table 3 reports these results. These values confirm that meta-evaluation outcomes are sensitive to the specific MT systems included in the pool. A reasonable hypothesis is that this within-dataset instability is partly driven by varying adequacy–fluency balances across subsets. To test this, we evaluate whether fixing β stabilizes the scorer rankings across the LOO subsets.1212 12 In preliminary studies, we also controlled the adequacy-over-fluency preference directly (see §4) to account for different subsets possessing different balance–β relations. This alternative control also failed to resolve the instability. Our results reject this hypothesis. As illustrated for two representative datasets in Figure 6, precisely controlling β does not resolve the ranking variance. While the adequacy–fluency balance may contribute to the instability of specific datasets, it is not the primary driver of the system-selection sensitivity observed in recent WMT evaluations. Identifying the root cause of this instability remains an open problem beyond the scope of this work. Dataset #Systems #Scorers #Segments β0 _0 βmin _ βmax _ βmax−βmin _ - _ ende24 16 30 487 0.7821 0.0074 1.0000 0.9926 enes24 13 30 623 0.9657 0.0005 1.0000 0.9994 jazh24 13 31 560 0.9954 0.4059 1.0000 0.5941 heen23 12 43 820 0.9868 0.0022 1.0000 0.9978 zhen23 15 43 1177 0.9672 0.0034 1.0000 0.9966 ende22 13 31 1315 0.9338 0.0010 1.0000 0.9990 zhen22 14 31 1875 0.9767 0.0071 1.0000 0.9928 ende21 13 27 527 0.2273 0.0000 0.9996 0.9996 zhen21 13 28 650 0.7054 0.0008 1.0000 0.9991 ted_ende 13 27 529 0.3802 0.0034 1.0000 0.9966 ted_zhen 13 28 529 0.4921 0.0063 0.9995 0.9932 Table 4: Dataset size and reachable balance range. #Scorers is the number of base scorers used for augmentation, listed by name in Table 7. Dataset #Pairs Mean Median Std IQR ende24 120 0.6223 0.6883 0.3083 0.5019 enes24 78 0.8444 0.9705 0.2642 0.1335 jazh24 78 0.9317 0.9922 0.1538 0.0292 heen23 66 0.8971 0.9919 0.2065 0.0917 zhen23 105 0.8392 0.9508 0.2416 0.1835 ende22 78 0.8049 0.9115 0.2523 0.2376 zhen22 91 0.8576 0.9499 0.2160 0.1371 ende21 78 0.3543 0.3105 0.2916 0.4552 zhen21 78 0.5984 0.7165 0.3347 0.4885 ted_ende 78 0.4672 0.3872 0.3667 0.7545 ted_zhen 78 0.5278 0.5340 0.3273 0.5987 Table 5: Statistics describing the distribution of the pairwise balance βkk′ _k across system pairs. IQR: their interquartile range, i.e. the 7575th percentile minus the 2525th. Dataset μ(a)μ(a) σ(a)2σ(a)^2 μ(f)μ(f) σ(f)2σ(f)^2 ρ Σ(a,f) (a,f) ende24 -1.3745 0.4475 -1.1275 0.1247 0.7210 0.1703 enes24 -0.4547 0.1167 -0.1262 0.0042 0.5531 0.0122 jazh24 -2.5512 2.3894 -0.3477 0.0110 0.5749 0.0933 heen23 -2.0684 0.7275 -0.3771 0.0097 0.4214 0.0355 zhen23 -2.7212 1.5735 -1.4967 0.0534 0.2707 0.0784 ende22 -0.8175 0.0494 -0.4051 0.0035 0.6182 0.0081 zhen22 -2.6217 0.8798 -0.6556 0.0210 0.8144 0.1108 ende21 -0.5341 0.0364 -1.1930 0.1238 0.6377 0.0428 zhen21 -3.3916 0.1510 -1.7004 0.0631 0.0837 0.0082 ted_ende -0.7215 0.0262 -0.8389 0.0426 0.2913 0.0097 ted_zhen -1.1248 0.0461 -1.0927 0.0476 0.7116 0.0333 Table 6: System-level Adequacy and Fluency MQM. Every statistic is taken over the K systems of a single dataset, at the uniform weighting w=1Kw= 1K1, on the system-level score vectors a (Adequacy MQM) and f (Fluency MQM). ρ is the Pearson correlation between a and f across systems. Σ(a,f) (a,f): the covariance between a and f across those same systems. Means are negative because MQM is defined as a negated error weight. Appendix D Dataset Statistics Tables 4–7 report descriptive statistics for the eleven datasets used throughout this work. Table 4 covers pool size and the balance parameter. Table 5 summarizes the distribution of pairwise balances βkk′ _k . Table 6 reports system-level Adequacy MQM and Fluency MQM statistics, together with their correlation and covariance across systems. Table 7 lists by name the base scorers augmented for each dataset. Dataset Base scorers ende24 BERTScore, BLCOM, BLCOM_1, BLEU, BLEURT-20, COMET-22, CometKiwi, CometKiwi-XXL, MEE4, MetricX-24, MetricX-24-Hybrid, MetricX-24-Hybrid-QE, MetricX-24-QE, PrismRefMedium, PrismRefSmall, XCOMET, XCOMET-QE, XLsimDA, YiSi-1, bright-qe, chrF, chrfS, damonmonli, gemba_esa, metametrics_mt_mqm_hybrid_kendall, metametrics_mt_mqm_kendall, metametrics_mt_mqm_qe_kendall.seg.s, monmonli, sentinel-cand-mqm, spBLEU enes24 BERTScore, BLCOM, BLCOM_1, BLEU, BLEURT-20, COMET-22, CometKiwi, CometKiwi-XXL, MEE4, MetricX-24, MetricX-24-Hybrid, MetricX-24-Hybrid-QE, MetricX-24-QE, PrismRefMedium, PrismRefSmall, XCOMET, XCOMET-QE, XLsimDA, YiSi-1, bright-qe, chrF, chrfS, damonmonli, gemba_esa, metametrics_mt_mqm_hybrid_kendall, metametrics_mt_mqm_kendall, metametrics_mt_mqm_qe_kendall.seg.s, monmonli, sentinel-cand-mqm, spBLEU jazh24 BERTScore, BLCOM, BLCOM_1, BLEU, BLEURT-20, COMET-22, CometKiwi, CometKiwi-XXL, MEE4, MetricX-24, MetricX-24-Hybrid, MetricX-24-Hybrid-QE, MetricX-24-QE, PrismRefMedium, PrismRefSmall, XCOMET, XCOMET-QE, XLsimDA, YiSi-1, bright-qe, chrF, chrfS, damonmonli, gemba_esa, metametrics_mt_mqm_hybrid_kendall, metametrics_mt_mqm_qe_kendall.seg.s, metametrics_mt_mqm_qe_same_source_t, metametrics_mt_mqm_same_source_targ, monmonli, sentinel-cand-mqm, spBLEU heen23 BERTscore, BLEU, BLEURT-20, COMET, Calibri-COMET22, Calibri-COMET22-QE, CometKiwi, CometKiwi-XL, CometKiwi-XXL, GEMBA-MQM, KG-BERTScore, MEE4, MS-COMET-QE-22, MaTESe, MetricX-23, MetricX-23-QE, MetricX-23-QE-b, MetricX-23-QE-c, MetricX-23-b, MetricX-23-c, Random-sysname, XCOMET-Ensemble, XCOMET-QE-Ensemble, XCOMET-XL, XCOMET-XXL, XLsim, YiSi-1, chrF, cometoid22-wmt21, cometoid22-wmt22, cometoid22-wmt23, docWMT22CometDA, docWMT22CometKiwiDA, eBLEU, embed_llama, f200spBLEU, instructscore, mbr-metricx-qe, mre-score-labse-regular, prismRef, prismSrc, sescoreX, tokengram_F zhen23 BERTscore, BLEU, BLEURT-20, COMET, Calibri-COMET22, Calibri-COMET22-QE, CometKiwi, CometKiwi-XL, CometKiwi-XXL, GEMBA-MQM, KG-BERTScore, MEE4, MS-COMET-QE-22, MaTESe, MetricX-23, MetricX-23-QE, MetricX-23-QE-b, MetricX-23-QE-c, MetricX-23-b, MetricX-23-c, Random-sysname, XCOMET-Ensemble, XCOMET-QE-Ensemble, XCOMET-XL, XCOMET-XXL, XLsim, YiSi-1, chrF, cometoid22-wmt21, cometoid22-wmt22, cometoid22-wmt23, docWMT22CometDA, docWMT22CometKiwiDA, eBLEU, embed_llama, f200spBLEU, instructscore, mbr-metricx-qe, mre-score-labse-regular, prismRef, prismSrc, sescoreX, tokengram_F ende22 BERTScore, BLEU, BLEURT-20, COMET-20, COMET-22, COMET-QE, COMETKiwi, Cross-QE, HWTSC-TLM, HWTSC-Teacher-Sim, KG-BERTScore, MATESE, MATESE-QE, MEE, MEE2, MEE4, MS-COMET-22, MS-COMET-QE-22, REUSE, SEScore, UniTE, UniTE-ref, UniTE-src, YiSi-1, chrF, f101spBLEU, f200spBLEU, metricx_xl_DA_2019, metricx_xl_MQM_2020, metricx_xxl_DA_2019, metricx_xxl_MQM_2020 zhen22 BERTScore, BLEU, BLEURT-20, COMET-20, COMET-22, COMET-QE, COMETKiwi, Cross-QE, HWTSC-TLM, HWTSC-Teacher-Sim, KG-BERTScore, MATESE, MATESE-QE, MEE, MEE2, MEE4, MS-COMET-22, MS-COMET-QE-22, REUSE, SEScore, UniTE, UniTE-ref, UniTE-src, YiSi-1, chrF, f101spBLEU, f200spBLEU, metricx_xl_DA_2019, metricx_xl_MQM_2020, metricx_xxl_DA_2019, metricx_xxl_MQM_2020 ende21 BERTScore, C-SPEC, C-SPECpn, COMET-DA_2020, COMET-DA_2021, COMET-MQM_2021, COMET-QE-DA_2021, COMET-QE-MQM_2021, COMETinho-DA, COMETinho-MQM, MEE, MEE2, MTEQA, OpenKiwi-MQM, Prism, TER, YiSi-1, YiSi-2, bleurt-20, bleurt-21-beta, chrF, cushLEPOR(LM), cushLEPOR(pSQM), hLEPOR, regEMT, regEMT-baseline, sentBLEU zhen21 BERTScore, C-SPEC, C-SPECpn, COMET-DA_2020, COMET-DA_2021, COMET-MQM_2021, COMET-QE-DA_2021, COMET-QE-MQM_2021, COMETinho-DA, COMETinho-MQM, MEE, MEE2, MTEQA, OpenKiwi-MQM, Prism, RoBLEURT, TER, YiSi-1, YiSi-2, bleurt-20, bleurt-21-beta, chrF, cushLEPOR(LM), cushLEPOR(pSQM), hLEPOR, regEMT, regEMT-baseline, sentBLEU ted_ende BERTScore, C-SPEC, C-SPECpn, COMET-DA_2020, COMET-DA_2021, COMET-MQM_2021, COMET-QE-DA_2021, COMET-QE-MQM_2021, COMETinho-DA, COMETinho-MQM, MEE, MEE2, MTEQA, OpenKiwi-MQM, Prism, TER, YiSi-1, YiSi-2, bleurt-20, bleurt-21-beta, chrF, cushLEPOR(LM), cushLEPOR(pSQM), hLEPOR, regEMT, regEMT-baseline, sentBLEU ted_zhen BERTScore, C-SPEC, C-SPECpn, COMET-DA_2020, COMET-DA_2021, COMET-MQM_2021, COMET-QE-DA_2021, COMET-QE-MQM_2021, COMETinho-DA, COMETinho-MQM, MEE, MEE2, MTEQA, OpenKiwi-MQM, Prism, RoBLEURT, TER, YiSi-1, YiSi-2, bleurt-20, bleurt-21-beta, chrF, cushLEPOR(LM), cushLEPOR(pSQM), hLEPOR, regEMT, regEMT-baseline, sentBLEU Table 7: The base scorers augmented for each dataset, by name. This follows the selection described in §4. Names are reproduced exactly as released by WMT. Appendix E Main Results for All Datasets Figure 7 presents the main results across all datasets, reproducing the figures from the main text for completeness. The observed trends remain consistent throughout, demonstrating our method’s generalizability. ende22 zhen21 enes24 heen23 jazh24 ted_ende ted_zhen zhen22 zhen23 ende21 ende24 Figure 7: Main results on all datasets. Layout follows Figure 3. Appendix F Main Results for Pearson Correlation Figure 8 mirrors the experiments from §4.3, using Pearson correlation as the meta-metric rather than SPA. Markers representing Shayegh et al. (2025) are omitted, as Pearson correlation falls outside the scope of their work. These findings align closely with our SPA evaluation, confirming the robustness of our approach. ende22 zhen21 enes24 heen23 jazh24 ted_ende ted_zhen zhen22 zhen23 ende21 ende24 Figure 8: Main results with Pearson as the meta-metric. Layout follows Figure 3. Appendix G Variance Analysis for All Datasets Figure 9 presents the variance analysis heatmaps across all datasets, reproducing the figures from the main text for completeness. The observed trends remain consistent throughout, demonstrating the reliability of β as a control parameter. ende22 zhen21 heen23 jazh24 ted_ende enes24 ende24 zhen22 ende21 zhen23 ted_zhen Figure 9: Variance analysis results for all datasets. Layout follows Figure 4.