Paper deep dive
The Format is the Filter: Scoring Format Fixes What an LLM Judge Can Express
Lovedeep Gondara, Jekaterina Novikova, Sagarika Suresh, Jack Ji, Jithin Pradeep
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1--5 rating, 0--100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0--100 scale is worth only 7--12 equally used levels, two to three times a 1--5 scale rather than twenty, and a binary verdict leaves the top score tied in 77--78\% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall's $\tau_b$ hides them: it ranks binary best while $\tau_a$ and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report.
Tags
Links
- Source: https://arxiv.org/abs/2610.17158v1
- Canonical: https://arxiv.org/abs/2610.17158v1
Trouble viewing inline? Open PDF directly →
Full Text
38,649 characters extracted from source content.
Expand or collapse full text
The Format is the Filter: Scoring Format Fixes What an LLM Judge Can ExpressThanks: © 2026 The Vanguard Group, Inc. All rights reserved. This material is provided for informational purposes only and is not intended to be investment advice or a recommendation to take any particular investment action. Lovedeep Gondara* Jekaterina Novikova Sagarika Suresh Jack Ji Jithin Pradeep Affiliation: The Vanguard Group, Inc. Affiliation: *Correspondence:Lovedeep_Gondara@Vanguard.com Abstract When an LLM judge filters instruction-tuning data, its scoring format (binary verdict, 1–5 rating, 0–100 score, or comparison) is rarely justified or ablated. Holding the candidate pool, judge prompt, decoding and selection budget fixed, we score 15,406 UltraFeedback candidates under five formats with three judges, curate, fine-tune 66 models, and evaluate them with four judges. A 0–100 scale is worth only 7–12 equally used levels, two to three times a 1–5 scale rather than twenty, and a binary verdict leaves the top score tied in 77–78% of prompts, so most selections are made at random. These properties are set by the format almost independently of the judge. Kendall’s τb _b hides them: it ranks binary best while τa _a and top-1 agreement rank it last, because it discounts the ties a selector must break at random. Both comparative formats inherit a slot bias whose sign differs across judges. Downstream, curated data beat random selection under every evaluation judge, robust to length and markdown adjustment, but no ordering among formats is detectable. We propose four quantities that judge-based curation pipelines should report. 1 Introduction A practitioner assembling an instruction-tuning corpus without an annotation budget typically asks an LLM judge to score every candidate, keeps the top fraction, and fine-tunes on what survives (Zheng et al., 2023; Chen et al., 2024; Sachdeva et al., 2026). How to ask for the judge’s opinion (ACCEPT/REJECT, a 1–5 rating, a 0–100 score, a comparison) is usually decided once, on grounds of convenience, and rarely ablated with the selector held fixed. Yet a curation pipeline consumes scores as an ordering and breaks ties at random, so tie structure is a first-class property of a format: a binary verdict leaves every accepted candidate indistinguishable, and the choice at the frontier is made by chance. We vary only the format across five designs, score a fixed pool with three judges, and follow one judge’s scores through curation and LoRA fine-tuning to 66 models. Our main contributions are as follows: (i) a fully crossed study of three judges × five formats, showing that the format, not the judge, determines how many ties the scores contain, a result that holds when any one judge is left out and when each format’s number of available levels is taken into account (§3.1); (i) evidence that τb _b inverts the ordering a selection audit needs (§3.3); (i) evidence that both comparative formats inherit a judge-specific slot bias, measured three ways (§3.4); and (iv) downstream results under four evaluation judges (§4). Related work. Elicitation studies measure formats by agreement with human or reference ratings (Li et al., 2026; Novikova et al., 2018; Zhuang et al., 2024) or compare pointwise and pairwise protocols for bias (Tripathi et al., 2025), without following scores into training. Data-selection work does follow them, but with one elicitation protocol (Chen et al., 2024; Cui et al., 2023; Pang et al., 2025; Wettig et al., 2024). Deutsch et al. (2023) establish that ties distort Kendall-style meta-evaluation; we show the consequence for selection. Configuration beyond the judge model is a known variance source (Zhang, 2026; Don-Yehiya et al., 2026; Norman et al., 2026; Soumik, 2026). We add the controlled format ablation, carried through to trained models. 2 Method Pool and reference. We draw 4,000 UltraFeedback prompts (Cui et al., 2023), yielding 15,406 multi-model responses after a 20–6,000 character filter (k¯=3.85 k=3.85 per prompt); 300 further prompts are held out. UltraFeedback’s GPT-4 scores are used as the reference. They never enter a prompt as numbers, the selection decision, or the training data, but they choose the anchor in the anchored format and the evaluation reference, so we report agreement with the GPT-4 reference and avoid the term accuracy. Formats. The pointwise formats share one preamble (helpfulness, correctness, completeness, clarity) and differ only in the final instruction; the comparative formats use a parallel pairwise prompt with five verdicts mapped to v∈+2,…,−2v∈\+2,…,-2\ (Appendix A). Binary (ACCEPT/REJECT →0,1→\0,1\), Likert-5 ((v−1)/4(v-1)/4) and Numeric-100 (v/100v/100) are pointwise. Anchored compares the candidate (always Response A) with a fixed per-prompt anchor, the upper-middle candidate by GPT-4 score (Don-Yehiya et al., 2026). Tournament compares every pair of candidates within a prompt once. Each verdict v gives Response A a win probability σ(v)=1/(1+e−v)σ(v)=1/(1+e^-v) (0.88 for a strong win, 0.5 for a tie), and a candidate’s score is its average win probability over its matches, a soft win rate. Decoding is greedy, with a 16-token verdict budget and reasoning disabled. Selection uses within-format ranks, which preserve ties, so each format’s tie structure passes unchanged into selection. Judges, curation and training. Three curation judges score the identical pool: deepseek-v4-flash (DeepSeek-AI et al., 2026), qwen3.5-9b (Yang et al., 2025), and nemotron-3-nano-30b-a3b; parse rates are ≥0.998≥ 0.998 in all fifteen cells. Curation uses DeepSeek scores only. At retention r∈10,25,50%r∈\10,25,50\\% we keep the highest-scored response per prompt (ties broken at random), then the top r of these globally. Random keeps one uniform response per prompt and a uniform fraction r of prompts; Length applies the same policy to response length; Full uses all candidates. Each subset fine-tunes Qwen3-4B (Yang et al., 2025) with LoRA (Hu et al., 2021) and three seeds: seven conditions at three rates plus Full, a total of 66 models. Evaluation. Each model answers the 300 held-out prompts; each response is compared with the highest-GPT-4-rated completion in both orders, scoring 1 when both orders agree on a win, 0 on a loss, and 0.5 otherwise. Four evaluation judges (deepseek-v4-pro, gemini-3.5-flash-lite, grok-4.20, Olmo-3-7B) are analyzed separately. They resolve different fractions of comparisons into a win or loss, and every unresolved comparison scores 0.5, so a judge that resolves fewer pulls all win rates toward 0.5; levels are therefore comparable within a judge but not across judges. Per judge and rate we run a Friedman test (Friedman, 1937), paired seed-clustered bootstrap contrasts (10,000 resamples), and Holm correction (Holm, 1979), and report how many judges concur. DeepSeek-V4-Flash Qwen3.5-9B Nemotron-30B-A3B Format Tie D2D_2 T Spr τb _b τa _a T1 Tie D2D_2 T Spr τb _b τa _a T1 Tie D2D_2 T Spr τb _b τa _a T1 Binary .501 2.0 .78 .636 .548 .222 .505 .511 2.0 .77 .603 .517 .210 .505 .578 1.7 .78 .517 .462 .156 .481 Likert-5 .240 4.2 .60 .685 .527 .344 .582 .304 3.3 .64 .680 .533 .337 .572 .261 3.8 .66 .427 .357 .224 .483 Numeric-100 .143 7.0 .45 .722 .515 .384 .607 .101 9.9 .40 .708 .502 .398 .616 .121 8.2 .47 .532 .401 .297 .537 Anchored .365 2.7 .23 .412 .401 .315 .526 .259 3.9 .36 .397 .390 .308 .526 .317 3.2 .67 .207 .125 .086 .396 Tournament .072 13.8 .05 .282 .336 .304 .544 .036 27.7 .10 .299 .335 .292 .536 .051 19.7 .04 .060 .036 .035 .378 Table 1: Scoring-stage diagnostics on the identical 15,406-candidate pool. Tie: probability two random candidates receive the same score; D2=1/TieD_2=1/Tie: effective number of equally used levels. T: fraction of prompts whose top score is tied, so that the selected response is drawn at random. Spr: global Spearman with the GPT-4 reference; τb _b, τa _a: mean within-prompt Kendall; T1: rate at which the top-scored response (one seeded tie-break; the exact expectation differs by at most 0.006) is a reference-top response; 28% of prompts have a tied reference top, so uniform selection scores 0.357. Pairs tied in the reference count in no τ numerator. Bold: best format per judge. Distinct-value counts and Somers’ D are in Appendix C, Table 5. 3 Scoring-Stage Results 3.1 Format Fixes Score Structure For each diagnostic in Table 1 we decompose the sum of squares over the 15 format–judge cells into format, judge and residual shares (Table 2); with one observation per cell, the residual absorbs the interaction. All three judges w/o Nem. Diagnostic Fmt % Judge % Res. % Judge % Tie probability 97.0 0.4 2.6 0.5 Distinct values 99.1 0.2 0.6 0.0 D2D_2/nominal levels† 96.0 0.3 3.8 0.0 Global Spearman 76.4 21.9 1.7 0.1 Within-prompt τb _b 55.6 36.0 8.5 0.4 Within-prompt τa _a 36.7 49.8 13.5 0.2 Top-1 agreement 38.9 49.2 11.9 0.1 Table 2: Variance shares over the format×judge grid. The last column drops Nemotron (DeepSeek and Qwen only); dropping either other judge leaves the judge share at 0.0–0.5% (structural) and 23–50% (reference agreement). †Pointwise and anchored formats only. Full leave-one-judge-out results in Appendix C, Table 6. On structural diagnostics, format explains 97–99% and the judge under 0.5%. This is partly by construction: a K-level alphabet has tie probability at least 1/K1/K, and binary sits almost on its floor of 0.50. The informative question is how efficiently each judge uses its alphabet. Normalized by nominal capacity, D2D_2 is still 96% format (log-normalized, 85%): every judge uses binary and Likert-5 at 66–100% of capacity and numeric-100 at 7–10% (§3.2). The residual is not empty: tie probability above the 1/K1/K floor is 54% format and 41% residual, so how a judge fills an alphabet is judge-and-format specific. Reference agreement depends on the weakest judge. The judge accounts for 22–50% of reference agreement in the full grid, but leave-one-judge-out shows this is carried by Nemotron: with DeepSeek and Qwen alone, format explains 99.0–99.7% and the judge 0.1–0.4%. Given a format, DeepSeek and Qwen give nearly identical agreement, whereas Nemotron lowers agreement under every format. The result does not depend on the two comparative formats, whose implementations carry slot bias (§3.4): over the three pointwise formats alone, DeepSeek and Qwen give a format share of 98.7–99.5% for τa _a, Somers’ D and top-1 (93% for global Spearman; Appendix C, Table 7). With three judges we do not generalize this to a capability law. 3.2 Nominal Granularity is Not Effective Granularity Numeric-100 offers twenty times the levels of Likert-5, but judges emitted only 33–37 of its 101 values and placed 74.7%, 95.9% and 99.6% of their mass (Nemotron, Qwen, DeepSeek) on multiples of five. Counting distinct values overstates resolution, because a value used twice counts the same as one used nearly 3,800 times. The effective number of equally used levels, Dα=exp(Hα)D_α= (H_α) for Rényi entropy HαH_α, corrects this: numeric-100 is worth 9.1–12.4 levels at α=1α=1 and 7.0–9.9 at α=2α=2, against 3.3–4.4 for Likert-5, two to three times the effective resolution rather than twenty. D2D_2 is exactly the inverse tie probability, so the tie column of Table 1 already reports it. It is also stable where counts are not: at a third of the pool, distinct values fall from 37 to 28.0±2.028.0± 2.0 while D2D_2 moves by about 0.1 s.d. or less (Appendix D, Table 10). 3.3 Kendall’s τb _b Misleads a Selection Audit By τb _b, binary is the best format under DeepSeek and Nemotron and second under Qwen; by τa _a it is last under DeepSeek and Qwen. τa _a, Somers’ D and top-1 agreement all rank numeric-100 first, Likert-5 second and binary last under DeepSeek and Qwen, differing only on anchored versus tournament. τb _b (Kendall, 1945) divides by (n0−n1)(n0−n2) (n_0-n_1)(n_0-n_2), where n1n_1 counts judge-tied pairs, so coarse formats are divided by a smaller number. When the reference orders a pair strictly and the judge ties it, the selector breaks the tie at random, so the pair’s expected concordance is zero, which is how τa _a scores it and not how τb _b does. For example, with reference order a>b>ca>b>c and a binary judge accepting a,ba,b, τa=2/3 _a=2/3 and τb≈0.82 _b≈ 0.82, yet the selector picks a only half the time. This is the common case here: binary leaves the top score tied in 77–78% of prompts (T in Table 1), against 40–47% for numeric-100. The argument extends Deutsch et al. (2023) from meta-evaluation to selection. 3.4 Comparative Formats Inherit Slot Bias Read each comparative verdict as a margin in favor of Response A. A genuine preference for the candidate follows the candidate, so its sign flips when the candidate moves from slot A to slot B; a preference for a slot does not move. Running 500 anchored candidates per judge in both orders therefore gives vfwd=q+bv_fwd=q+b and vrev=−q+bv_rev=-q+b, separating the mean preference for the candidate q from the slot-A bias b (Table 4, Appendix B). The bias differs in sign on identical inputs: b=−0.33b=-0.33 (±0.09± 0.09) for DeepSeek and Qwen, toward slot B, and +0.89+0.89 [0.790.79, 0.990.99] for Nemotron. Self-comparisons confirm it without the additive model: the anchor is one of the candidates, so 4,000 queries compare a response with itself, where any verdict other than TIE can only reflect position. DeepSeek and Qwen return TIE on 95.8% and 99.6% of these, Nemotron on 42.3%, with a mean verdict of +0.92+0.92 toward slot A. Because q is a mean preference rather than discrimination, we also correlate each item’s qiq_i with its GPT-4 gap to the anchor: 0.440.44 [0.370.37, 0.510.51] for DeepSeek, 0.370.37 [0.300.30, 0.440.44] for Qwen, and 0.100.10 [0.010.01, 0.190.19] for Nemotron, whose anchored scores are mostly position. The tournament inherits the bias through scheduling: pairs were enumerated in listing order, so the first-listed candidate is always Response A and the last always Response B. Listing order is unrelated to quality (Spearman 0.05 with GPT-4 score), yet it drives selection in each judge’s direction. DeepSeek selects the last-listed response in 44.6% of four-candidate prompts, Nemotron the first-listed in 67.5%, against 25% without a position effect. The tournament’s low agreement (global Spearman 0.06–0.30) is therefore at least partly due to slot bias amplified by unbalanced scheduling. 4 Downstream Consequences r Contrast DS-Pro Gemini Grok Olmo 4/4 50 anch. >> rand. +.093∗+.093^* +.086∗+.086^* +.084∗+.084^* +.049∗+.049^* 50 Lik. >> rand. +.099∗+.099^* +.087∗+.087^* +.110∗+.110^* +.054∗+.054^* 50 num. >> rand. +.080∗+.080^* +.088∗+.088^* +.091∗+.091^* +.038∗+.038^* 3/4 25 Lik. >> rand. +.065∗+.065^* +.077∗+.077^* +.068∗+.068^* +.027+.027 25 num. >> rand. +.061∗+.061^* +.094∗+.094^* +.072∗+.072^* +.009+.009 50 bin. >> rand. +.048∗+.048^* +.086∗+.086^* +.082∗+.082^* +.026+.026 50 Lik. >> len. +.062∗+.062^* +.079∗+.079^* +.078∗+.078^* +.013+.013 50 tourn. >> rand. +.064∗+.064^* +.053+.053 +.058∗+.058^* +.047∗+.047^* 1/4 50 bin. << Lik. −.052∗-.052^* −.002-.002 −.028-.028 −.029-.029 50 bin. << num. −.032∗-.032^* −.003-.003 −.009-.009 −.013-.013 Table 3: Downstream contrasts by the number of evaluation judges concurring (all 3/4 and 4/4 contrasts; two of the seven 1/4 contrasts). Cells are mean win-rate differences; ∗: pHolm<0.05p_Holm<0.05 for that judge (seed-clustered bootstrap). The omnibus Friedman test detects differences among the training conditions at every retention rate under every evaluation judge (p≤0.017p≤ 0.017). Every contrast that replicates across judges is a curated format beating a baseline (Table 3): three hold under all four evaluation judges and five more under three. A difference of +0.09+0.09 means winning against the reference on about nine more prompts in 100 than Random. Against Full, trained on all 15,406 candidates, a 400-example curated subset (r=10%r=10\%) is at least as good: 18 of 20 format×judge differences are positive (up to +0.059+0.059), though only 2 survive Holm correction, and Random at 10% stays within 0.02 of Full. Shared judge taste. Curated students write 1,044–1,257 characters with markdown in 38–46% of responses, against 1,072–1,113 and 36–40% for Random. Adding log length relative to the reference and a markdown indicator to a prompt-fixed-effects regression changes every curated-versus-Random contrast by less than 0.02 (Appendix C, Table 8), so the gains are not a length or markdown effect. Different data, similar students. The formats do not select the same examples. The Jaccard overlap between two formats’ DeepSeek-curated subsets averages 0.02 at r=10%r=10\%, 0.06 at 25% and 0.15 at 50%, so each format trains on a largely disjoint set. That such different subsets yield statistically indistinguishable students suggests that, in this pool, many selections clear the bar that separates curated from random data, and the large scoring-stage differences are compressed by training. Between formats. No format beats another under more than one evaluation judge. Binary does trail the finer formats in direction, below Likert-5 in all 12 judge×rate comparisons and below numeric-100 in 11 of 12, matching its weaker scoring-stage agreement. These gaps are too small for the design to resolve: with 300 prompts and three seeds it detects a difference of about 0.03–0.05 win-rate points with 80% power, and its 95% CIs rule out format differences larger than 0.064–0.113, depending on the judge. The missing ordering should therefore be read as undetected; it is not evidence that the formats are equivalent. 5 Conclusion Under a fixed decoding rule, the format sets a judge’s tie structure almost independently of the judge, and tie structure decides how often selection is random; comparative formats also import a judge-specific slot bias, and the format alone largely changes which examples are selected. We propose reporting the following: (1) the format verbatim, with the decoding rule; (2) tie probability (or D2D_2), with any distinct-value count given alongside the pool size; (3) for comparative formats, how positions were swapped or balanced, and the estimated b; and (4) the verdict parse rate. In practice, we recommend Likert-5 or numeric-100 over a binary verdict, swap and balance slots in comparative formats, and validate formats with τa _a, Somers’ D or top-1 agreement rather than τb _b. Limitations Decoding. All verdicts use greedy decoding. Probability-weighted scores (e.g., the expected Likert value under the verdict-token distribution, as in G-Eval; Liu et al., 2023) or repeated sampling would make binary and Likert-5 effectively continuous. Our structural results describe format together with greedy decoding, not format alone. Variance decomposition. There is one observation per cell, so the residual absorbs the format×judge interaction; three judges give the judge factor two degrees of freedom, and the judge share in reference agreement is driven by one judge (Table 2). Reference. Agreement is measured against UltraFeedback’s GPT-4 scores, not human labels. The same scores choose the anchor shown to the judge in the anchored format and the evaluation reference. Downstream design. Training data come from one curation judge (DeepSeek), so we do not test whether a less accurate curation judge yields worse students; one Nemotron-curated condition would. We have no untuned student or GPT-4-oracle selector baseline, so we cannot say whether better reference agreement yields better training data. Our length and markdown adjustment conditions on properties of the trained model’s own outputs, so it is a sensitivity check rather than a causal control. The Length baseline is confounded with truncation (the 1,024-token cap removes 11.4–25.2% of its examples against 1.8–6.2% elsewhere). Format-specific confounds. Binary’s disadvantage is inseparable from its random tie-breaking. Anchored is intentionally unswapped, and 26% of its queries compare the anchor with itself, which assigns every anchor the same score. The tournament’s slot assignment follows listing order, so its selections, including the DeepSeek-curated training data, carry each judge’s slot bias; its soft win rate is computed within small slates (k¯=3.85 k=3.85), so a weak slate’s winner can score as high as a strong slate’s. Scope. We only use one candidate pool, one student, and one language. References Chen et al. (2024) L. Chen, S. Li, J. Yan, H. Wang, K. Gunaratna, V. Yadav, Z. Tang, V. Srinivasan, T. Zhou, H. Huang, et al. Alpagasus: training a better alpaca with fewer data. In International Conference on Learning Representations, Vol. 2024, p. 34767–34797. Cited by: §1, §1. Cui et al. (2023) G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, et al. Ultrafeedback: boosting language models with scaled ai feedback. External Links: 2310.01377 Cited by: §1, §2. DeepSeek-AI et al. (2026) DeepSeek-AI, A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, C. Lu, C. Zhao, C. Deng, C. Hou, C. Xu, C. Shao, C. Ruan, C. Sun, D. Dai, D. Guo, D. Yang, D. Chen, D. Li, D. Ji, E. Li, F. Wei, F. Lin, F. Yuan, F. Xia, F. Dai, G. Hao, G. Chen, G. Cao, G. Meng, G. Li, H. Yu, H. Zhang, H. Xu, H. Li, H. Liang, H. Zhang, H. Luo, H. Wei, H. Yuan, H. Zhang, H. Luo, H. Chen, H. Ji, H. Zhang, H. Ding, H. Tang, H. Cao, H. Gao, H. Qu, H. Zeng, J. Yang, J. Zhu, J. Luo, J. Song, J. Yu, J. Huang, J. Cai, J. Liang, J. Zhou, J. Ye, J. Li, J. Xu, J. Hu, J. Yang, J. Chen, J. Yan, J. Chen, J. Zhou, J. Xiang, J. Yuan, J. Cheng, J. Zhou, J. Zhu, J. Yu, J. Sun, J. Ran, J. Jiang, J. Qiu, J. Li, J. Zheng, J. Song, K. Dong, K. Gao, K. Guan, K. Zhou, K. Huang, K. Yu, L. Wang, L. Zhang, L. Wang, L. Xia, L. Zhang, L. Zhao, L. Guo, L. Luo, L. Ma, L. Zhu, L. Wang, L. Cai, L. Zhang, L. Chen, M. Di, M. Xu, M. Mei, M. Wang, M. Zhang, M. Zhang, M. Tang, M. Li, M. Zhou, M. Han, N. Wang, P. Huang, P. Wang, P. Cong, P. Wang, P. Zhang, Q. Wang, Q. Zhu, Q. Li, Q. Chen, Q. Du, Q. Jiang, R. Tian, R. Xu, R. Lu, R. Xu, R. Ge, R. Zhang, R. Pan, R. Wang, R. Chen, R. Yin, R. Xu, R. Shen, R. Zhang, R. Chen, S. Liu, S. Lu, S. Sun, S. Zhou, S. Chen, S. Cai, S. Nie, S. Wu, S. Chen, S. Hu, S. Liu, S. Hu, S. Ma, S. Wang, S. Yu, S. Zhou, S. Pan, S. Yu, S. Zhou, T. Ni, T. Yun, T. Jin, T. Pei, T. Ye, T. Lin, T. Ji, T. Cui, T. Yue, T. Yu, T. Wang, W. Zhang, W. Xiao, W. Zeng, W. An, W. Zhao, W. Liu, W. Liang, W. Pang, W. Luo, W. Yao, W. Gao, W. Yang, W. Huang, W. Hou, W. Zhang, W. Ma, X. Gao, X. He, X. Wang, X. Wang, X. Bi, X. Liu, X. Wang, X. Chen, X. Zhang, X. Nie, X. Sun, X. Wang, X. Cheng, X. Liu, X. Xie, X. Liu, X. Liu, X. Yu, X. Li, X. Yang, X. Zhang, X. Chen, X. Wang, X. Su, X. Chen, X. Lin, X. Fu, Y. Yan, Y. Wang, Y. Ma, Y. Luo, Y. Zhang, Y. Xu, Y. Ma, Y. Huang, Y. Li, Y. Li, Y. Xu, Y. Zhao, Y. Sun, Y. Wang, Y. Qian, Y. Shao, Y. Yu, Y. Zhang, Y. Ding, Y. Shi, Y. Wu, Y. Xiong, Y. Ma, Y. He, Y. Tang, Y. Zhou, Y. Luo, Y. Zhong, Y. Piao, Y. Wang, Y. Zhang, Y. Chen, Y. Tan, Y. Wei, Y. Ma, Y. Liu, Y. Yang, Y. Guo, Y. Wu, Y. Wu, Y. Li, Y. Cheng, Y. Ou, Y. Xu, Y. Li, Y. Wang, Y. Yang, Y. Xu, Y. Wu, Y. Meng, Y. Zou, Y. Zha, Y. Xiong, Y. Chen, Y. Lin, Y. Cao, Y. Wang, Y. Zhang, Y. Yan, Y. Lin, Y. Gu, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. Zhou, Y. Huang, Z. Wu, Z. Wang, Z. Zhao, Z. Ren, Z. Zhang, Z. Sha, Z. Fu, Z. Ju, Z. Xu, Z. Xie, Z. Zhang, Z. Gao, Z. Hao, Z. Gou, Z. Ma, Z. Yan, Z. Shao, Z. Huang, Z. Chen, Z. Wu, Z. Ren, Z. Wu, Z. Li, Z. Zhang, Z. Xu, Z. Wang, Z. Qu, Z. Gu, Z. Zhu, Z. Li, Z. Zhang, Z. Xie, Z. Gao, Z. Wan, Z. Pan, and Z. Yao DeepSeek-v4: towards highly efficient million-token context intelligence. External Links: 2606.19348, Link Cited by: §2. Deutsch et al. (2023) D. Deutsch, G. Foster, and M. Freitag Ties matter: meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 12914–12929. Cited by: §1, §3.3. Don-Yehiya et al. (2026) S. Don-Yehiya, A. Yehudai, L. Choshen, and O. Abend Mediocrity is the key for llm as a judge anchor selection. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15491–15513. Cited by: §1, §2. Friedman (1937) M. Friedman The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association 32 (200), p. 675–701. External Links: ISSN 01621459, 1537274X, Link Cited by: §2. Holm (1979) S. Holm A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2), p. 65–70. External Links: ISSN 03036898, 14679469, Link Cited by: §2. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen Lora: low-rank adaptation of large language models. External Links: 2106.09685 Cited by: §2. Kendall (1945) M. G. Kendall THE treatment of ties in ranking problems. Biometrika 33 (3), p. 239–251. External Links: ISSN 0006-3444, Document, Link, https://academic.oup.com/biomet/article-pdf/33/3/239/573257/33-3-239.pdf Cited by: §3.3. Li et al. (2026) W. Li, M. Zhao, W. Dong, J. Cai, Y. Wei, M. Pocress, Y. Li, W. Yuan, X. Wang, R. Hou, et al. Grading scale impact on llm-as-a-judge: human-llm alignment is highest on 0–5 grading scale. npj Artificial Intelligence. Cited by: §1. Liu et al. (2023) Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu G-eval: nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 conference on empirical methods in natural language processing, p. 2511–2522. Cited by: Limitations. Norman et al. (2026) J. D. Norman, M. U. Rivera, and D. A. Hughes Reliability without validity: a systematic, large-scale evaluation of llm-as-a-judge models across agreement, consistency, and bias. External Links: 2606.19544 Cited by: Appendix B, §1. Novikova et al. (2018) J. Novikova, O. Dušek, and V. Rieser RankME: reliable human ratings for natural language generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), p. 72–78. Cited by: §1. Pang et al. (2025) J. Pang, J. Wei, A. P. Shah, Z. Zhu, Y. Wang, C. Qian, Y. Liu, Y. Bao, and W. Wei Improving data efficiency via curating llm-driven rating systems. In International Conference on Learning Representations, Vol. 2025, p. 100986–101016. Cited by: §1. Sachdeva et al. (2026) N. Sachdeva, B. Coleman, W. Kang, J. Ni, L. Hong, E. H. Chi, J. Caverlee, J. McAuley, and D. Cheng How to train data-efficient llms. In International Conference on Learning Representations, Vol. 2026, p. 45491–45545. Cited by: §1. Soumik (2026) S. K. Soumik Judging the judges: a systematic evaluation of bias mitigation strategies in llm-as-a-judge pipelines. External Links: 2604.23178 Cited by: Appendix B, §1. Tripathi et al. (2025) T. Tripathi, M. Wadhwa, G. Durrett, and S. Niekum Pairwise or pointwise? evaluating feedback protocols for bias in llm-based evaluation. arXiv preprint arXiv:2504.14716. Cited by: §1. Wettig et al. (2024) A. Wettig, A. Gupta, S. Malik, and D. Chen Qurating: selecting high-quality data for training language models. arXiv preprint arXiv:2402.09739. Cited by: §1. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §2. Zhang (2026) X. Zhang How sensitive are safety benchmarks to judge configuration choices?. In International Conference on Intelligent Computing, p. 173–184. Cited by: §1. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in neural information processing systems, Vol. 36, p. 46595–46623. Cited by: §1. Zhuang et al. (2024) H. Zhuang, Z. Qin, K. Hui, J. Wu, L. Yan, X. Wang, and M. Bendersky Beyond yes and no: improving zero-shot llm rankers via scoring fine-grained relevance labels. In Proceedings of the 2024 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies (volume 2: short papers), p. 358–370. Cited by: §1. Figure 1: Numeric-100 score distributions, one panel per curation judge; blue bars are multiples of five. Nemotron departs most from the round-number grid (74.7% on multiples of five) yet has the lowest reference agreement. Appendix A Method Details Prompts. Pointwise formats: You are evaluating the quality of a response to an instruction. Consider helpfulness, correctness, completeness, and clarity. Instruction: instruction Response: response format line with format lines Binary: “Is this response of acceptable quality? Reply with exactly one word: ACCEPT or REJECT.” Likert-5: “Rate the response on a 1-5 scale (1=very poor, 2=poor, 3=acceptable, 4=good, 5=excellent). Reply with only the number.” Numeric-100: “Rate the response on a 0-100 scale, where 0 is unusable and 100 is flawless. Reply with only the number.” Comparative formats use “You are comparing two responses to the same instruction. Consider helpfulness, correctness, completeness, and clarity.” followed by the instruction, Response A, Response B, and “Which response is better? Reply with exactly one of: A++ (A much better), A+ (A slightly better), TIE, B+ (B slightly better), B++ (B much better).” Verdicts map to +2,+1,0,−1,−2+2,+1,0,-1,-2. Anchored. The anchor is the candidate at index ⌊k/2⌋ k/2 after sorting the prompt’s candidates by GPT-4 score (mean anchor score 6.93 against 6.45 for all candidates, so q<0q<0 is expected); the candidate is always Response A, and the score is (v+2)/4(v+2)/4. Because the anchor is drawn from the candidates, 4,000 of the 15,406 queries compare the anchor with itself. Tournament. All (k2)≤6 k2≤ 6 within-prompt pairs are judged once, in the order produced by enumerating the prompt’s candidates in listing order, so candidate i is Response A against every later candidate. A verdict v gives the first response win probability σ(v)σ(v) and the second 1−σ(v)1-σ(v); a candidate’s score is its mean win probability over its matches. Fine-tuning. Each model is trained with LoRA (rank 16, α=32α=32, dropout 0.05) for 2 epochs, using AdamW with learning rate 2×10−42× 10^-4, an effective batch size of 8 and bf16 precision. Training sequences are capped at 1,024 tokens. The cap truncates 4.3% of the full pool; across curated subsets it ranges from 1.8% (numeric-100) to 6.2% (tournament), but reaches 11.4–25.2% for Length, which selects the longest responses by design (see Limitations). All training ran on a single NVIDIA RTX 5090. Curation and evaluation judges were queried through hosted APIs, except Olmo-3-7B-Instruct, which we ran locally with vLLM. Statistics. Every format is evaluated on the same 300 prompts, a repeated-measures design, so we use the Friedman test per judge and rate, paired bootstrap contrasts resampling both prompts and seeds, and Holm correction across format pairs within each rate. Appendix B Anchored Order-Swap Decomposition Verdicts map to a slot-A-oriented margin: “A much better” →+2→+2, “A slightly better” →+1→+1, tie →0→ 0, “B slightly better” →−1→-1, “B much better” →−2→-2. The candidate’s preference travels with the candidate, so its sign flips when the candidate moves to slot B; a position preference stays with the slot. Hence vfwd=q+bv_fwd=q+b, vrev=−q+bv_rev=-q+b, q=(vfwd−vrev)/2q=(v_fwd-v_rev)/2, b=(vfwd+vrev)/2b=(v_fwd+v_rev)/2. The additive model treats the five verdicts as an interval scale and can be distorted near the clipped endpoints. The reverse pass uses a seeded sample of 500 candidates per judge, excluding self-comparisons; 95% CIs below are from 10,000 bootstrap resamples of items. Judge q b ρ(qi,Δ)ρ(q_i, ) Self-TIE DeepSeek −0.52-0.52 −0.33-0.33 0.44 95.8% [−.65,−.39-.65,-.39] [−.42,−.24-.42,-.24] [.37,.51] Qwen −0.16-0.16 −0.33-0.33 0.37 99.6% [−.26,−.07-.26,-.07] [−.42,−.24-.42,-.24] [.30,.44] Nemotron −0.07-0.07 +0.89+0.89 0.10 42.3% [−.14,+.00-.14,+.00] [+.79,+.99+.79,+.99] [.01,.19] Table 4: Order-swap decomposition of anchored verdicts (n=500n=500 per judge; 95% CIs below). q: preference for the candidate over the anchor net of position (<0<0: anchor preferred, as expected). b: slot bias (>0>0: favors Response A). ρ(qi,Δ)ρ(q_i, ): per-item correlation of qiq_i with the GPT-4 score gap, i.e. whether preferences track quality. Self-TIE: TIE rate when the anchor is compared with itself (n=4,000n=4,000). DeepSeek and Qwen share a small slot-B bias but track quality; Nemotron is dominated by a slot-A bias. DeepSeek and Qwen have indistinguishable slot bias, so the data do not support a monotone relation between capability and position bias; three judges could not establish one in any case. Prior reports range from at most 0.04 across frontier judges (Soumik, 2026) to above 0.10 in production judges (Norman et al., 2026). Appendix C Additional Diagnostics Table 5 gives distinct-value counts and Somers’ D; Table 6 the leave-one-judge-out shares; Table 7 the pointwise-only decomposition; and Table 8 the length and markdown-adjusted contrasts. Judge Format Vals Somers’ D DeepSeek Binary 2 .261 Likert-5 5 .406 Numeric-100 37 .457 Anchored 5 .369 Tournament 65 .364 Qwen Binary 2 .247 Likert-5 5 .398 Numeric-100 33 .472 Anchored 5 .361 Tournament 66 .350 Nemotron Binary 2 .187 Likert-5 5 .266 Numeric-100 33 .355 Anchored 5 .098 Tournament 56 .036 Table 5: Distinct values emitted and mean within-prompt Somers’ D. Judge share (%) All −-DS −-Qwen −-Nem. Tie probability 0.4 0.4 0.0 0.5 D2D_2/nominal† 0.3 0.4 0.2 0.0 Distinct values 0.2 0.2 0.3 0.0 Global Spearman 21.9 22.8 23.7 0.1 τb _b 36.0 32.4 34.4 0.4 τa _a 49.8 47.0 50.4 0.2 Somers’ D 49.1 46.1 49.4 0.2 Top-1 (expected) 49.0 46.2 49.4 0.3 Table 6: Leave-one-judge-out judge share of the sum of squares, computed from unrounded values; top-1 uses the exact expectation under random tie-breaking. Column headers name the judge removed. †Binary, Likert-5, numeric-100, anchored. All three judges DeepSeek + Qwen Diagnostic Fmt % Judge % Fmt % Judge % Tie probability 97.5 0.4 98.0 0.1 Global Spearman 9.9 80.2 93.4 4.6 τb _b 7.7 80.8 52.9 19.8 τa _a 70.3 27.5 99.4 0.0 Somers’ D 71.5 26.3 99.5 0.0 Top-1 (expected) 58.0 36.4 98.7 0.2 Table 7: Variance shares over the three pointwise formats only (binary, Likert-5, numeric-100), which have no slot-assignment confound. r=50%r=50\% DS-Pro Gemini Grok Olmo Binary .048/.047 .086/.083 .082/.080 .026/.025 Likert-5 .099/.094 .087/.083 .110/.106 .054/.052 Numeric-100 .080/.081 .088/.086 .091/.090 .038/.038 Anchored .093/.080 .086/.083 .084/.078 .049/.044 Tournament .064/.051 .053/.049 .058/.052 .047/.042 Table 8: Curated minus Random win rate at r=50%r=50\%, unadjusted / adjusted for log response length relative to the reference and a markdown indicator (OLS with prompt fixed effects over seed-level outcomes). The markdown indicator is 1 if a response contains a heading, a bullet or numbered list item, bold text, or a code fence. The largest change is 0.013 (at r=25%r=25\% it is smaller); dropping numbered lists, adding tables, using headings and bold only, or using a log count of markdown elements gives 0.009–0.014. Appendix D Effective Granularity For a score distribution p, the Rényi entropy of order α is Hα=11−αlog∑ipiαH_α= 11-α _ip_i^α and Dα=exp(Hα)D_α= (H_α) is the number of equally used levels with the same entropy. D1D_1 is the exponentiated Shannon entropy; D2=1/∑ipi2D_2=1/ _ip_i^2 is the inverse tie probability, and D2≤D1D_2≤ D_1. A distinct-value count weights a score emitted twice the same as one emitted 3,791 times: DeepSeek’s four most frequent numeric-100 scores (85, 75, 45, 65) carry 70.2% of the mass. Figure 1 shows the numeric-100 distributions, and Table 9 lists nominal, emitted and effective levels for every pointwise format. Judge Format Nom. Dist. D1D_1 D2D_2 DeepSeek Binary 2 2 2.00 1.99 Likert-5 5 5 4.44 4.17 Anchored 5 5 3.17 2.74 Numeric-100 101 37 9.07 6.98 Qwen Binary 2 2 1.98 1.96 Likert-5 5 5 3.76 3.28 Anchored 5 5 4.34 3.86 Numeric-100 101 33 12.39 9.94 Nemotron Binary 2 2 1.85 1.73 Likert-5 5 5 4.27 3.84 Anchored 5 5 3.65 3.16 Numeric-100 101 33 10.55 8.23 Table 9: Nominal levels, distinct values emitted, and effective levels. Binary and Likert-5 use close to their nominal capacity. Anchored uses fewer effective levels than Likert-5 under DeepSeek and Nemotron, consistent with a verdict distribution concentrated by the fixed anchor and slot bias; under Qwen the order reverses (D1D_1 4.34 vs. 3.76). DeepSeek Qwen Nemotron Distinct 28.0±2.028.0± 2.0 27.8±1.527.8± 1.5 27.8±1.427.8± 1.4 shift −9.0-9.0 −5.2-5.2 −5.2-5.2 D1D_1 9.05±.099.05±.09 12.36±.1212.36±.12 10.52±.1110.52±.11 shift −.02-.02 −.03-.03 −.03-.03 D2D_2 6.98±.086.98±.08 9.93±.119.93±.11 8.22±.108.22±.10 shift −.00-.00 −.01-.01 −.01-.01 Table 10: Numeric-100 measures at n/3n/3 (mean ± s.d., 2,000 resamples without replacement) and their shift from the full-pool value. Subsampling to a third of the pool (Table 10) removes 16–24% of the distinct-value count and collapses all three judges to 27.8–28.0, erasing the full-pool difference (37 vs. 33). D1D_1 and D2D_2 shift by at most 0.3% of their value and preserve the judge ordering; the D2D_2 shift is about 0.1 s.d. or less and the D1D_1 shift within 0.3 s.d.