Paper deep dive
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
Zongyou Yang, Yinghan Hou, Xiaokun Yang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 7/10/2026, 6:13:55 AM
Summary
This paper audits the reliability of LLM-as-judge evaluation by comparing two upgrade paths: Qwen3 parameter scaling (1.7B to 32B) and MiniMax API releases (M2 to M2.7). It finds that judge upgrades are not interchangeable, stronger judges reduce but do not eliminate position and verbosity biases, jury aggregation yields minimal gains due to high error correlation, and structured debate shifts require audit trails to be interpretable. The authors advocate for standardized reporting including dataset slices, bias probes, error-dependence estimates, and protocol logs.
Entities (15)
Relation Signals (12)
LLM-as-judge → benchmarkedon → LLMBar
confidence 99% · LLMBar is an adversarial pairwise benchmark with 419 examples.
LLM-as-judge → benchmarkedon → PandaLM
confidence 98% · PandaLM testset-v1 provides broad-domain pairwise judgments...
LLM-as-judge → benchmarkedon → Chatbot Arena
confidence 98% · A 2,000-example seed-42 sample from Chatbot Arena provides broad human preference comparisons...
LLM-as-judge → measuredby → Accuracy
confidence 98% · Pairwise experiments report accuracy and Cohen’sκ on parseable verdicts...
Qwen3-1.7B → parameterscaling → Qwen3-4B
confidence 98% · Qwen3 improves sharply from 1.7B to 4B on LLMBar (0.463 to 0.617)...
LLM-as-judge → exhibitsbias → Verbosity Bias
confidence 97% · Stronger judges reduce but do not remove position and verbosity bias.
LLM-as-judge → exhibitsbias → position bias
confidence 97% · Stronger judges reduce but do not remove position and verbosity bias.
LLM-as-judge → measuredby → Cohen's Kappa
confidence 97% · Pairwise experiments report accuracy and Cohen’sκ on parseable verdicts...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.
Tags
Links
- Source: https://arxiv.org/abs/2607.08535v1
- Canonical: https://arxiv.org/abs/2607.08535v1
Trouble viewing inline? Open PDF directly →
Full Text
32,063 characters extracted from source content.
Expand or collapse full text
When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability Zongyou Yang Dyson School of Design Engineering, Imperial College London London, United Kingdom zy2926@ic.ac.uk Yinghan Hou Department of Electrical and Electronic Engineering, Imperial College London London, United Kingdom yh24@ic.ac.uk Xiaokun Yang ∗ School of Electronic Information, Nanchang Institute of Technology Nanchang, China yangxk@bupt.cn Abstract An LLM-as-judge score can move even when the candidate re- sponses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement- validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2–M2.7 released APIs. The main pattern is that judge upgrades are not in- terchangeable: only Qwen3 1.7B→4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails. CCS Concepts • Computing methodologies→Natural language processing; Machine learning. Keywords LLM-as-judge, automatic evaluation, model scaling, evaluation reli- ability, bias, jury aggregation 1 Introduction LLM-as-judge evaluation is now often used as a measuring instru- ment for model quality. The difficulty is that the instrument is itself a model. When a system’s score changes after replacing the evalua- tor, the change is ambiguous: it may indicate that the new judge is more capable, that it is biased differently, that it fails on a different slice of the benchmark, or that the evaluation pipeline parsed and aggregated its outputs differently. This ambiguity is not a minor im- plementation detail. It determines whether an LLM-as-judge score can be interpreted as evidence about the candidate systems at all. We call this the evaluator-replacement ambiguity: when the mea- sured preference outcome changes after replacing the judge, the source of the change is not identifiable from accuracy alone. This paper studies the ambiguity through two observable interventions available in evaluation practice. The first is a parameter-scaling decision, represented by Qwen3 dense judges from 1.7B to 32B parameters [21]. The second is a released-model upgrade path, rep- resented by MiniMax M2–M2.7 APIs evaluated as released [15]. ∗ Corresponding author. These axes are evidence sources rather than the paper’s object of explanation. The MiniMax M2 report documents the released series and its agent-oriented training pipeline, but it does not make our API sequence a controlled ablation; consequently, this paper does not make a causal claim about MiniMax internals. This formulation leads to three research questions. RQ1 asks whether judge reliability improves similarly along a parameter axis and a released-model upgrade path. If evaluator capability is the dominant factor, both axes should show consistently positive ad- jacent gains across datasets; if reliability is workload-dependent, gains should vary by dataset and significance should not trans- fer uniformly. RQ2 asks whether higher aggregate accuracy also reduces known judge biases. Higher-capability judges should be less bias-sensitive, but nonzero flip rates would show that capabil- ity does not eliminate measurement artifacts. RQ3 asks whether protocol-level upgrades, such as juries and debate, change relia- bility beyond single-judge scaling. Jury gains should be limited when error correlation is high, while debate shifts should depend on capability asymmetry and protocol auditability. The central thesis is that evaluator capability is important but incomplete: the same judge can look different across workloads, bias probes, correlated votes, and protocol implementations. This paper contributes an auditable measurement framework for LLM-as-judge reliability, separating model capability from work- load dependence, bias behavior, correlated voting, and measurement artifacts. First, it presents a two-axis evaluator study that compares parameter scaling with released-model generation while explicitly avoiding causal claims about MiniMax internals. Second, it decom- poses reliability into single-judge accuracy, bias probes, correlated jury errors, and debate auditability. Third, it distills a reporting standard: benchmark slices, parse-shared tests, A/B randomization, 휌 estimates, and protocol logs. The study yields three main results. First, judge upgrades are not interchangeable: parameter scaling produces one robust low-end gain, while adjacent MiniMax releases do not. Second, higher accu- racy reduces but does not remove bias. Third, aggregation protocols are interpretable only when their assumptions are auditable: jury voting is bounded by correlated errors, and debate shifts cannot be attributed to deliberation without parser logs. 2 Related Work LLM judges and preference evaluation. MT-Bench, Chatbot Arena, PandaLM, JudgeLM, G-Eval, Auto-J, Prometheus, and Re- wardBench have helped establish LLM judges as practical evaluators for open-ended generation [8,9,11,12,14,19,23,24]. These studies arXiv:2607.08535v1 [cs.CL] 9 Jul 2026 Z. Yang, Y. Hou, and X. Yang Candidate responses Benchmark slice Judge model Prompt / decoding Parser Aggregation / debate Reported score model capability changes validity order and prompt change bias parser and fallback create artifacts repeated samples create dependence Score changes are produced by the whole measurement pipeline The judge is one component of the instrument, not an external observer. Figure 1: LLM-as-judge as a measurement pipeline. A reported score depends not only on candidate responses, but also on the benchmark slice, judge model, prompt and decoding choices, parser, and any aggregation or debate protocol. show that LLM judges can correlate with human preferences, but also that correlations depend on task, prompt, scoring scale, and model family. Our work follows this empirical tradition but treats the judge itself as the object of scaling analysis. The gap is that prior reports often present evaluator performance as a one-dimensional model ordering, leaving unclear whether gains arise from capability, benchmark slice, bias reduction, or protocol design. Judge bias and protocol effects. Prior work documents posi- tion bias, verbosity bias, sensitivity to rating format, and reliability changes caused by evaluation design choices [5,16,17,20]. These effects matter because they can change rankings without changing candidate responses. What remains under-specified is how bias changes jointly with evaluator scaling rather than under isolated prompt perturbations. Unlike a full prompt-factorial study, our ex- periments use a fixed main prompt and treat prompt sensitivity as a robustness check; the main focus is how model capability, bias, aggregation, and debate interact under a controlled reporting protocol. Aggregation and debate. Classical Condorcet-style results pre- dict gains from independent voters, while work on correlated voters shows that dependence can sharply reduce those gains [3,10]. In LLM evaluation, self-consistency, multi-agent debate, and ChatEval- style protocols offer analogous routes to aggregation or deliberation [2,4,7,13,18]. Existing studies often emphasize final accuracy; our focus is whether the aggregation assumptions required for such gains are empirically satisfied. Our jury experiment directly esti- mates error correlation, and our debate experiment is reported as an auditability case study. 3 Experimental Design 3.1 Operationalizing Judge Reliability We treat LLM-as-judge reliability as a multi-dimensional measure- ment property rather than a single accuracy score. In this study, reliability consists of four observable components: judgment va- lidity, bias robustness, aggregation independence, and protocol auditability. Table 1 maps each construct to an operational measure and experiment. Table 1: Operationalizing LLM-as-judge reliability. ConstructOperational measureExp. Judgment validityAccuracy, Cohen’s휅, Spearman1 Bias robustnessPosition flip, verbosity bias, granularity sensi- tivity 2 Aggregation independenceError correlation휌 , jury gain3 Protocol auditability Parse status, fallback behavior, intermediate verdicts 4 3.2 Models and Datasets The main panel contains eight evaluated judges: Qwen3-1.7B, Qwen3- 4B, Qwen3-14B, Qwen3-32B, MiniMax-M2, MiniMax-M2.1, MiniMax- M2.5, and MiniMax-M2.7. This panel separates an approximate parameter intervention from an observed released-model interven- tion. GLM-5.1 and mimo-v2-pro are used as cross-family reference judges in selected experiments, but are not members of either axis. Four datasets cover complementary judgment settings so that reliability is not inferred from a single task slice. LLMBar is an adversarial pairwise benchmark with 419 examples [22]. PandaLM testset-v1 provides broad-domain pairwise judgments; after remov- ing tie-majority examples, 894 valid pairwise examples remain [19]. A 2,000-example seed-42 sample from Chatbot Arena provides broad human preference comparisons, with 1,997 valid pairwise examples [23]. Our Judge’s Verdict slice uses 200 TechQA-derived examples with three-level pointwise labels, supporting exact-match and rank-based evaluation [1, 6]. 3.3 Metrics and Protocols Pairwise experiments report accuracy and Cohen’s휅on parseable verdicts; Judge’s Verdict additionally reports Spearman rank corre- lation. Main single-judge runs use near-greedy decoding (푇=0.1) to reduce sampling noise when estimating model-level reliability. Homogeneous juries sample one judge multiple times at푇=0.7 because repeated low-temperature calls would understate the de- pendence structure among stochastic jurors; heterogeneous juries combine different judges at푇=0.1 to isolate model composition from sampling variance. Adjacent model comparisons use exact two-sided McNemar tests on parse-shared examples, so significance reflects paired judgment changes rather than differences in parser coverage. Holm correction controls the family of 18 adjacent tests. When the Judge Changes, So Does the Measurement The experiments instantiate the constructs in Table 1. Exper- iment 1 measures single-judge validity along the two axes. Ex- periment 2 measures LLMBar position, verbosity, and granularity robustness. Experiment 3 evaluates majority-vote juries under mea- sured error correlation. Experiment 4 evaluates a structured debate protocol as an auditability case study. Four robustness checks cover human-ceiling calibration, corrected adjacent significance, limited prompt sensitivity, and Arena seed stability. 4 Results 4.1 Finding 1: Judge Upgrades Are Not Interchangeable A capability-only account would predict similar adjacent gains along both upgrade paths; the paired tests do not show that pat- tern. Table 3 summarizes the main single-judge pattern. Qwen3 improves sharply from 1.7B to 4B on LLMBar (0.463 to 0.617) and more modestly on Arena, but later parameter steps are smaller and not uniformly monotone. MiniMax release generations show no reliable adjacent improvement in this panel: adjacent pairwise dif- ferences are at most 0.022 accuracy, and none of the nine MiniMax adjacent tests reaches uncorrected푝<0.05. After Holm correction across all 18 adjacent tests, only the Qwen3 1.7B to 4B steps on LLMBar and Arena remain significant. The relevant conclusion is not that one family is universally stronger. No judge wins all datasets: GLM-5.1 leads LLMBar, MiniMax- M2.7 leads PandaLM, and mimo-v2-pro leads the sampled Arena slice. This shifts the design question from one-dimensional evalua- tor ordering to slice-specific measurement validity. 4.2 Finding 2: Stronger Judges Are Less Biased, Not Unbiased On LLMBar, the bias probes move in the same direction as capability, but they do not disappear. The position-flip rate falls from 0.320 for Qwen3-1.7B to 0.117–0.147 for MiniMax releases. Verbosity bias under a fixed generic and innocuous padding string falls from 0.547 for Qwen3-1.7B to roughly 0.13 for MiniMax. Granularity sensitivity is also smaller for the stronger judges. This pattern is best interpreted as a capability–fairness associa- tion rather than a mechanism. Across the eight evaluated judges, the correlation between LLMBar accuracy and position-flip rate is strong (Pearson푟=−0.957), but it is measured on one dataset and one model panel. Stronger judges are also not unbiased: MiniMax- M2.7 still changes 14.7% of verdicts under A/B reversal. Thus posi- tion randomization and slice-level bias reporting remain necessary even when using high-accuracy judges. 4.3 Finding 3: Jury Size Matters Less Than Error Dependence The jury experiment fails for a familiar statistical reason: the votes are not independent. Majority voting is therefore not a generic reliability amplifier for LLM judges. For homogeneous juries, we estimate the intra-class error correlation휌from the vote matrix and compare three quantities: the independence prediction, a휌- corrected beta-binomial prediction, and observed jury accuracy. Let푝be the single-judge baseline accuracy,푞 ∼ Beta(훼,훽),훼= 푝푠, 훽= (1− 푝)푠, and푠=1/휌 −1. The jury accuracy is then the beta-binomial probability that a majority of 퐾 votes are correct. The correction is empirically necessary. Across homogeneous juries, the independence prediction has median absolute error 0.078 on LLMBar and 0.093 on PandaLM. The휌-corrected prediction reduces these errors to 0.008 and 0.004, with maximum error below 0.02. Estimated correlations are high: Qwen3 homogeneous juries have휌=0.944–0.972 on LLMBar, while MiniMax juries are lower but still correlated at휌=0.664–0.706. As a result, increasing jury size barely moves accuracy; for Qwen3-1.7B on LLMBar,퐾=1,3,5 yields 0.463, 0.475, and 0.482. Heterogeneous juries also underperform Poisson-binomial in- dependence predictions. Family mixing alone therefore does not restore independent errors under a shared prompt. This shifts the relevant design question from “how many jurors should be sam- pled?” to “how independent are their errors?” 4.4 Finding 4: Debate Shifts Require Audit Trails The debate runs produce the largest protocol-level shifts in the paper, but they are also the least auditable. We include debate not as evidence for a deliberation effect, but as a stress test for protocol auditability. The structured-debate experiment pairs two judges for up to three rebuttal rounds on LLMBar. Cross-capability pairs show the largest final-vs-round-1 accuracy shifts: Qwen3- 1.7B paired with GLM-5.1, MiniMax-M2.7, or mimo-v2-pro changes final decisions toward higher measured accuracy by+0.317,+0.305, and+0.289; Qwen3-4B paired with GLM-5.1 shifts decisions by +0.243 and reaches 0.897. Same-family and top-vs-top shifts are much smaller, ranging from+0.005 to+0.050. These changes show that protocol-level interventions can alter judge outcomes substantially; the missing parser audit trail shows why such shifts should not be accepted without measurement logs. The implementation records round verdicts and final verdicts, but not raw responses or parse-success flags. Round-1 parse failures fall back to A, and later parse failures retain the previous verdict. Since fallback rates cannot be audited without rerunning the experiment, the debate results are best read as a suggestive protocol pattern and a reporting requirement, not as a clean estimate of deliberation. 4.5 Robustness Checks The robustness checks support the direction of the main find- ings while keeping their scope narrow. Human-ceiling calibra- tion shows that PandaLM retains headroom: the best judge has matched leave-one-annotator-out휅=0.753 versus a human ceiling of 0.920. Judge’s Verdict is closer to its noisy ceiling, with best-judge 휅=0.620 versus human휅=0.562. Prompt-sensitivity checks on LLMBar preserve the cross-family ordering and show greater cross- prompt stability for stronger judges. Arena seed perturbations pro- duce 0.004–0.032 accuracy spread, smaller than the robust Qwen3 1.7B to 4B Arena gap. These checks do not establish cross-family generality, but they reduce several obvious alternative explanations. 5 Discussion Principle. A judge score is interpretable only when the evaluated slice, perturbation sensitivity, error dependence, and protocol state Z. Yang, Y. Hou, and X. Yang Evaluator-replacement interventions Qwen3 parameter axis 1.7b 0.463 4b 0.617 14b 0.647 32b 0.678 MiniMax release path M2 0.829 M2.1 0.839 M2.5 0.832 M2.7 0.832 Cross-family references glm-5.1 (0.900) mimo-v2-pro (0.883) numbers: LLMBar accuracy Datasets LLMBar 419 | PandaLM 894 | Arena 1,997 | Judge's Verdict 200 Measurements validity | bias probes | jury rho correction | debate auditability | robustness Figure 2: Study design. The paper uses a Qwen3 parameter axis and a MiniMax release-generation axis as observable evaluator- replacement interventions, then tests whether single-judge scaling, bias probes, jury aggregation, and structured debate answer the same reliability question. Table 2: Main answers by research question. Research questionAnswerKey evidenceImplication RQ1: Do scaling and release upgrades similarly improve reliability? No.Qwen3 1.7B→4B is robustly significant; MiniMax adjacent re- leases are not significant. Model upgrading is not a uniform relia- bility intervention. RQ2: Does higher accuracy reduce bias?Partly.Position flips decline, but MiniMax-M2.7 still changes 14.7% under A/B reversal. High-accuracy judges still need bias probes. RQ3: Do jury and debate protocols add reliability?Conditionally.Jury gains are small under high휌; debate shifts are large but lack an audit trail. Aggregation protocols must report de- pendence and audit trails. Table 3: Single-judge aggregate results. Values are accuracies except Judge’s Verdict, where exact match is reported for comparability; brackets give Wilson 95% intervals. DatasetBest judgeQwen3 1.7B→ 32B LLMBarGLM-5.1, 0.900 [0.868,0.925]0.463 [0.416,0.511]→ 0.678 [0.632,0.721] PandaLMMiniMax-M2.7, 0.857 [0.833,0.878]0.779 [0.751,0.805]→ 0.769 [0.740,0.795] Arenamimo-v2-pro, 0.742 [0.722,0.761]0.625 [0.604,0.646]→ 0.688 [0.667,0.708] Judge’s VerdictGLM-5.1, 0.680 [0.612,0.741]0.595 [0.526,0.661]→ 0.530 [0.461,0.598] LLMBARPANDALMARENA 1.7b 4b 4b 14b 14b 32b p<.001*0.090p<.001* 0.2030.8160.164 0.1480.032*0.696 Qwen3 (parameter axis) LLMBARPANDALMARENA m2m2.1 m2.1m2.5 m2.5m2.7 0.4880.7161.000 0.8831.0000.099 0.6440.4500.949 MiniMax (release-generation axis) 0 1 2 3 4 log 10 p 0 1 2 3 4 log 10 p Adjacent McNemar tests by dataset (* = uncorrected p < 0.05) Figure 3: Adjacent-pair McNemar tests for the two axes. Af- ter Holm correction over 18 adjacent tests, only the Qwen3 1.7B→4B steps on LLMBar and Arena remain significant; no MiniMax adjacent release step is significant in this panel. are jointly reported. The practical recommendation is therefore simple: an LLM-as-judge result should not be accepted as reliable unless it reports what slice was judged, how biased the judge was 0.450.500.550.600.650.700.750.800.85 LLMBar judge accuracy (Exp1) 0.125 0.150 0.175 0.200 0.225 0.250 0.275 0.300 0.325 Position-bias flip rate (Exp2) Q1.7B Q4B Q14B Q32B M2 M2.1 M2.5 M2.7 Pearson r =0.957 slope =0.49 Accuracy and position sensitivity on LLMBar Qwen3 MiniMax Figure 4: Capability–fairness association on LLMBar. Higher single-judge accuracy co-varies with lower position-flip rate across the eight evaluated judges, but the non-zero flip rates show that stronger judges still require A/B randomization and bias reporting. under simple perturbations, how correlated repeated judgments were, and whether protocol-level decisions were auditable. When the Judge Changes, So Does the Measurement 0.50.60.70.80.9 Predicted accuracy 0.5 0.6 0.7 0.8 0.9 Observed accuracy m2 m2.1 m2.5 qwen3 1.7b 4b 14b qwen3 4b 14b 32b cross family mid (a) Theory, rho-corrected, observed Homogeneous, theory Homogeneous, rho-corrected Heterogeneous y = x 135 Jury size k (Qwen3-1.7B) 0.43 0.44 0.45 0.46 0.47 0.48 Accuracy (b) Jury size under dependence Theory (rho=0) rho-corrected Observed 0.00.20.40.60.81.0 Error correlation rho M2.5 M2 M2.1 M2.7 Q1.7B Q4B Q14B 0.66 0.67 0.67 0.71 0.94 0.96 0.97 (c) Homogeneous-jury dependence Qwen3 MiniMax Figure 5: Jury behavior on LLMBar. Majority voting provides little gain because sampled jurors make correlated errors; the휌-corrected beta-binomial model tracks observed jury accuracy much better than the independence assumption. 0.00.10.20.30.4 Single-judge capability gap |acc a acc b | (Exp1 LLMBar) 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Final - round-1 accuracy q4bglm-5.1 q1.7bm2.7 q1.7bmimo q1.7bglm-5.1 Structured-debate decision shifts same-family cross-family top cross-capability Figure 6: Structured-debate accuracy shift versus single-judge capability gap on LLMBar. Cross-capability pairs produce the largest shifts, but the debate implementation’s parse-fallback behavior makes the result suggestive rather than causal. Table 4: A minimal audit trail for LLM-as-judge reports. Reliability riskRequired report itemFailure mode avoided Dataset-slice shiftValid 푁 , parseable 푁 , tie handlingHidden sample mismatch Parser artifactsParser success rate, fallback ruleArtificial accuracy changes Position/verbosity biasA/B reversal, padding probeRanking artifacts Correlated jury errors휌 and퐾 , not퐾 aloneOverestimated voting gains Debate protocol artifacts Raw outputs, parser status, round ver- dicts Unauditable protocol gains Statistical overclaimingPaired tests, Holm correction, uncer- tainty intervals where estimable False adjacent-step claims or over- stated precision The results suggest two methodological lessons for LLM-as-judge evaluation. First, judge scaling should be reported as a design space rather than a single evaluator ordering. In this panel, the clearest parameter-axis gain occurs at the low end, while the MiniMax release-generation axis changes little in the measured adjacent comparisons. This should not be read as a claim that all release generations lack meaningful gains, or that the MiniMax releases isolate one training stage. The supported conclusion is narrower: under the same datasets and prompts, the two observed upgrade paths have different adjacent-pair patterns. Second, reliability reporting has to go beyond single-number accuracy and name the unit of analysis. Bias probes show that stronger judges can be less brittle, but they also show that bias remains measurable. Jury experiments show that aggregation gains depend on error correlation. Debate experiments show that proto- col changes can be large enough to rival scaling effects, but only if the implementation preserves an audit trail for parse failures, tie-breaking, and abstentions. For pairwise datasets, this means parseable-subset definitions, parse-shared significance tests, and A/B randomization. For juries, it means reporting휌in addition to퐾. For debate or multi-agent protocols, it means storing raw responses, parser status, and every intermediate vote before in- terpreting final-score shifts. These requirements are operationally simple, but without them the apparent reliability of a judge can be an artifact of the evaluation pipeline. 6 Threats to Validity Construct validity. Reliability is operationalized through selected proxies: accuracy and agreement, bias probes, error correlation, and auditability logs. These do not exhaust all reliability dimensions, such as calibration, long-form rationale quality, or domain-specific consistency. Internal validity. The Qwen3 axis is closer to a parameter sweep, whereas the MiniMax axis is an observed release sequence rather than a controlled ablation. The debate experiment is espe- cially limited by missing raw outputs and parse-success logs. Statistical conclusion validity. McNemar tests are adjacent- pair tests and do not constitute a formal between-axis test. Multiple comparisons are controlled through Holm correction, but sample sizes and parser coverage differ across datasets. External validity. Results are conditioned on two model fami- lies, two reference judges, four datasets, a small number of prompts, and one primary decoding regime per experiment. Additional do- mains, languages, and instruction distributions are needed before turning these observations into general judge-selection rules. 7 Conclusion This paper presents a two-axis empirical study of LLM-as-judge reliability. The results show that evaluator capability is important but incomplete: reliability is also shaped by dataset slice, bias behav- ior, correlated errors, and protocol implementation. In the tested panel, the strongest robust adjacent effect is the Qwen3 1.7B to 4B transition; no MiniMax adjacent release step is significant; majority- vote juries are limited by high error correlation; and debate with a strong reference changes outcomes substantially but lacks sufficient evidence for causal interpretation under the current logs. These findings recast LLM-as-judge reliability as a measurement-validity problem rather than a model-selection problem alone. Acknowledgments This work was supported by the Open Research Project of the State Key Laboratory of Industrial Control Technology, China (Grant No. ICT2025B70); the Jiangxi Provincial Natural Science Foun- dation (Grant Nos. 20242BAB20041 and 20232BAB212006); the Hubei Provincial Natural Science Foundation of China (Grant Nos. 2023AFB474 and 2024AFB881); the Anhui Provincial Special Project for Special Needs in Humanities and Social Sciences (Grant No. 2025AHGXSK50067); and the Postgraduate Quality Engineering Project of Anhui Province (Grant No. 2024jyjxggyjY232). Z. Yang, Y. Hou, and X. Yang References [1]Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Mike McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avirup Sil, Rosario Uceda-Sosa, Todd Ward, and Rong Zhang. 2019. The TechQA Dataset. arXiv preprint arXiv:1911.02984 (2019). arXiv:1911.02984 [2]Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate. arXiv preprint arXiv:2308.07201 (2023). arXiv:2308.07201 [3] Marie Jean Antoine Nicolas Caritat de Condorcet. 1785. Essai sur l’application de l’analyse à la probabilité des décisions rendues à la pluralité des voix. Imprimerie Royale, Paris. [4]Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mor- datch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. In International Conference on Machine Learning (ICML). arXiv:2305.14325 [5]Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. 2024. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv preprint arXiv:2404.04475 (2024). arXiv:2404.04475 [6] Steve Han, Gilberto Titericz Junior, Tom Balough, and Wenfei Zhou. 2025. Judge’s Verdict: A Comprehensive Analysis of LLM Judge Capability Through Human Agreement. arXiv preprint arXiv:2510.09738 (2025). arXiv:2510.09738 [7] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. 2024. Debating with More Persuasive LLMs Leads to More Truthful Answers. International Conference on Machine Learning (ICML) (2024). arXiv:2402.06782 [8] Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024. Prometheus: Inducing Fine-grained Evaluation Capability in Language Models. International Conference on Learning Representations (ICLR) (2024). arXiv:2310.08491 [9]Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. arXiv preprint arXiv:2405.01535 (2024). arXiv:2405.01535 [10] Krishna K. Ladha. 1992. The Condorcet Jury Theorem, Free Speech, and Correlated Votes. American Journal of Political Science 36, 3 (1992), 617–634. doi:10.2307/2111584 [11]Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. RewardBench: Evaluating Reward Models for Language Modeling. arXiv preprint arXiv:2403.13787 (2024). arXiv:2403.13787 [12]Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023. Generative Judge for Evaluating Alignment. arXiv preprint arXiv:2310.05470 (2023). arXiv:2310.05470 [13]Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2023. Encouraging Divergent Think- ing in Large Language Models through Multi-Agent Debate. arXiv preprint arXiv:2305.19118 (2023). arXiv:2305.19118 [14]Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In Conference on Empirical Methods in Natural Language Processing (EMNLP). arXiv:2303.16634 [15]MiniMax. 2026. The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence. arXiv preprint arXiv:2605.26494 (2026). arXiv:2605.26494 [16]Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Ver- bosity Bias in Preference Labeling by Large Language Models. arXiv preprint arXiv:2310.10076 (2023). arXiv:2310.10076 [17]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large Language Models are not Fair Evaluators. arXiv preprint arXiv:2305.17926 (2023). arXiv:2305.17926 [18]Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. International Conference on Learning Representations (ICLR) (2023). arXiv:2203.11171 [19]Yidong Wang, Zhuohao Yu, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. 2023. PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization. arXiv preprint arXiv:2306.05087 (2023). arXiv:2306.05087 [20]Yusuke Yamauchi, Taro Yano, and Masafumi Oyamada. 2025. An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability. arXiv preprint arXiv:2506.13639 (2025). arXiv:2506.13639 doi:10.48550/arXiv.2506.13639 [21]An Yang et al.2025. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388 (2025). arXiv:2505.09388 [22]Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating Large Language Models at Evaluating Instruction Following. In International Conference on Learning Representations (ICLR). arXiv:2310.07641 [23]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. arXiv:2306.05685 [24]Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023. JudgeLM: Fine-tuned Large Language Models are Scalable Judges. arXiv preprint arXiv:2310.17631 (2023). arXiv:2310.17631