Paper deep dive

Accounting for Bias Enables Sustainable LLM Evaluation

Harshita Katoch, David Antony Selby, Gerrit Großmann, Sebastian Vollmer

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 30

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

27,812 characters extracted from source content.

Expand or collapse full text

Accounting for Bias Enables Sustainable LLM Evaluation Harshita Katoch David Antony Selby Gerrit Großmann Sebastian Vollmer Abstract LLM-as-a-judge has become the de facto standard for scalable, subjective evaluation, yet current leaderboards compensate for systematic measurement bias by running ever more comparisons, an approach that is both statistically unsound and computationally wasteful. The root cause is an incomplete measurement model, treating LLM judges as neutral, interchangeable instruments ignores documented biases like position bias, verbosity bias, judge severity, and self-enhancement, that no volume of additional data can eliminate. We propose a unified latent variable framework that jointly models pairwise and ordinal data while explicitly correcting for these confounders, recovering reliable rankings from substantially fewer comparisons. Because fitting this model costs negligible compute relative to a single round of LLM inference, bias correction is not only more statistically rigorous but also a more sustainable approach to trustworthy evaluation. keywords LLM-as-a-judge ,LLM evaluation ,psychometrics ,bias correction ,sustainable benchmarking †copyrightyear: 2026†copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (C BY 4.0).†venue: SuRE’26: Workshop on Sustainability and Resource-Efficiency of Artificial Intelligence, August 17, 2026, Bremen, Germany†email: harshita.katoch@dfki.de†email: david.selby@dfki.de†email: gerrit.grossmann@dfki.de†email: sebastian.vollmer@dfki.de†address: Department of Computer Science, University of Kaiserslautern–Landau (RPTU), Germany†address: Data Science and its Applications, German Research Centre for Artificial Intelligence (DFKI), Germany 1 Introduction The evaluation of large language models has become a question of trustworthiness as much as capability Ye et al. (2025). The field’s prevailing response to evaluation challenges has been to scale: running more comparisons, enlisting more judges and expanding leaderboards. The statistical machinery underpinning these systems has genuinely improved, with the field moving from raw win rates toward principled latent-variable models for pairwise and ordinal scoring that better capture preference uncertainty. Yet scaling and methodological refinement have both proceeded under a shared assumption, that has gone largely unexamined: that LLM judges are neutral, interchangeable instruments whose idiosyncrasies cancel with sufficient data Gu et al. (2026); Li et al. (2024). When judge behaviour is systematically skewed, collecting more comparisons narrows the confidence interval around a biased estimate but not correct it. Scale adds compute without adding reliability. The biases in question are well-documented and structural. LLM judges exhibit position preferences i.e. favouring responses that appear in a particular slot regardless of quality Shi et al. (2025), verbosity effects that reward length over content Dubois et al. (2024), and self-enhancement toward outputs from their own provider family Marsh et al. (2018). Beyond these, different judges apply systematically different score levels which is a severity effect that makes identical model ability map to different observed scores. Surveys by Gu et al. Gu et al. (2026) and Li et al. Li et al. (2024) establish these as properties of the measurement instrument, not incidental quirks of particular models. Since the effects differ in direction across judges, procedural fixes such as order randomisation or majority voting cannot eliminate them in expectation Li et al. (2024), nor do mitigations that target a single bias in isolation Dubois et al. (2024); Boyeau et al. (2025). A jointly specified measurement model is required. The consequences are both scientific and environmental. Every judge-mediated comparison requires inference from two competing models and a judge, i.e. three inference calls per comparison, and at leaderboard scale this cost is substantial. Each inference call consumes computational resources, GPU time, memory, storage and electricity (training costs notwithstanding), so reducing the number of comparisons translates directly into lower energy use and a smaller environmental footprint Samsi et al. (2023). Standard benchmarks are approximately 97% redundant Kipnis et al. (2025), and smarter item selection can reduce evaluation cost by up to 99% without accuracy loss Polo et al. (2024). Generating redundant comparisons to compensate for an unmodelled confounder is not just statistically unsound, it is ecologically indefensible Bender et al. (2021); Burnell et al. (2023b). Fitting a corrective latent-variable model costs negligible compute relative to a single round of LLM inference, making bias correction the more sustainable path to evaluation we can actually trust. We propose a unified latent-variable framework that (i) jointly models pairwise and ordinal data in a single likelihood, (i) explicitly corrects for position bias, verbosity, self-enhancement, and judge-level severity, and (i) is validated on MT-Bench Zheng et al. (2023) as the first empirical decomposition of bias magnitudes on a real leaderboard. Simulation confirms that the bias-corrected model recovers reliable rankings from substantially fewer comparisons than naive baselines across six tournament designs. 2 Related Work The machine learning community has developed LLM evaluation largely in isolation from the statistical sciences, arriving at methods that parallel tools already mature in psychometrics and educational measurement Jordan (2016). Early evaluation relied on static benchmarks with ground-truth answers; as tasks became more open-ended, the field shifted toward judge-mediated evaluation, MT-Bench introduced GPT-4-as-a-judge on multi-turn dialogue Zheng et al. (2023); Chatbot Arena crowdsourced human pairwise preferences into Elo scores Chiang et al. (2024); AlpacaEval 2.0 and Arena-Hard extended the paradigm with length control and harder prompts Dubois et al. (2024); Li et al. (2025a). Each iteration improved on its predecessor, yet all share a common assumption that observed ratings are direct proxies for model quality rather than noisy realisations of a latent construct mediated by rater behaviour and item difficulty. Beyond the response-level biases documented in the literature Gu et al. (2026); Li et al. (2024), the measurement scale itself is routinely misused, Liddell and Kruschke Liddell and Kruschke (2018) shows that the widespread practice of treating ordinal judge scores as continuous interval data introduces systematic distortions into effect estimates, an error that accumulating more data cannot fix. A jointly specified model is required. Item response theory disentangles examinee ability from item difficulty, providing estimates invariant to the specific sample of items or judges used DeMars (2010). Many-facet Rasch models extend this to explicitly model rater severity Martínez-Plumed et al. (2019), and extensions of paired-comparison models provide a principled framework that subsumes Elo as a special case Hamilton and Tawn (2026). Recent work has begun applying these ideas to AI benchmarks Lalor et al. (2016); Federiakin (2025); Wang et al. (2026), but exclusively on objective tasks with binary correctness signals, where there is no rater in the loop. Our contribution extends this to the judge-mediated setting, where rater confounders require explicit parametrisation alongside standard IRT machinery, a gap the existing literature has not addressed. 3 A Unified Psychometric Framework In psychometric terms, the LLM under evaluation is the examinee with latent ability θ, the prompt is the item with difficulty δ, and the judge is the rater with severity α. The goal is parameter invariance: ability estimates independent of which specific judges and prompts happened to be used DeMars (2010). If a model’s ranking drops because we switched from a lenient judge to a strict one, the measurement system has failed. This reframes what appears to be a data-volume problem as what it actually is a modelling problem. 3.1 Ranking from Pairwise Comparisons A minimally viable evaluation system based on comparative judgement should use a variant of the Bradley–Terry model Bradley and Terry (1952) rather than average win rates, with coefficients to account for order effects, self-enhancement, and measurable item-level features such as relative verbosity. This extends AlpacaEval 2.0 Dubois et al. (2024), which in the presence of these biases only controls for item-level effects. Such an augmented Bradley–Terry–Luce model takes the form: P⁡(A≻B)=σ⁡(θA−θB+γ+λA,r−λB,r⏟Affinity bias)P(A B)=σ\! ( _A- _B+γ+ _A,r- _B,r_Affinity bias ) (1) where θA _A and θB _B are the models’ latent abilities; γ is a position bias; λi,r _i,r is judge r’s affinity for model i, capturing self-enhancement bias; and σ is the logistic function. This is straightforward to fit using standard logistic regression software. Further terms may be added for verbosity and other documented biases; in our empirical analysis we additionally include a judge-specific position deviation uju_j and a standardised response length covariate Dubois et al. (2024). 3.2 Ranking from Ordinal Scores Some benchmarks prefer direct scoring of individual items. A comparative judgement may say A≻BA B but not be able to say ‘both are bad’; an absolute scale such as 1–10 allows us to anchor items at one or the other extreme rather than merely order them relative to one another. Instead of naively averaging scores, a minimal ordinal model must account for scale-, item-, and judge-level effects. The many-facet Rasch model Linacre (1989) treats scores as ordered categorical outcomes. The probability that model n receives score k∈0,…,mk∈\0,…,m\ from rater r on item p is: P⁡(Xnpr=k)∝exp⁡(∑m=1k(θn−δp−αr−τm))P(X_npr=k) \! ( _m=1^k( _n- _p- _r- _m) ) (2) where θn _n is latent ability, δp _p is item difficulty, αr _r is rater severity, and τm _m are threshold parameters on the cumulative logit scale, not assumed equally spaced. Accounting for judge severity prevents a model evaluated by a lenient judge from artificially outranking one evaluated by a strict judge. Treating ordinal scores as interval data, as leaderboards routinely do, produces systematic inversions in estimated effects Liddell and Kruschke (2018), a modelling error that accumulating more data cannot fix. 3.3 The Hybrid Joint Likelihood Both pairwise preferences and ordinal scores are imperfect observations of the same latent ability. Rather than discarding either, we combine them in a joint likelihood that separates true ability variance from rater noise: ℒ⁡(θ)=[∏y∈pairPRel(y∣θ,γ)]ω×[∏x∈ordPOrd(x∣θ,λ,δ)]1−ωL(θ)=\! [ _y _pair\!P_Rel(y θ,γ) ]^ω\!×\! [ _x _ord\!P_Ord(x θ,λ,δ) ]^1-ω (3) where ω∈[0,1]ω∈[0,1] weights the two data sources (ω=0.5ω=0.5 in our experiments); λ represents judge-specific parameters (severity and affinity); and δ item difficulties. Ordinal ratings anchor sparse pairwise data, and models appearing in only one data source can be ranked alongside those in both. Estimation proceeds via L-BFGS-B with ridge regularisation; for adaptive designs, a parametric bootstrap correction is preferred over Firth’s penalised likelihood, which introduces bias under non-random scheduling Hamilton and Tawn (2026). 4 Empirical Validation: MT-Bench We validate the proposed framework on MT-Bench Zheng et al. (2023), fitting the hybrid model jointly to GPT-4-judged pairwise comparison data and single-score ordinal data. Our aim is to confirm that the documented judge confounders are statistically significant in real leaderboard data, that they produce systematic, model-specific distortions rather than random noise, and that correcting for them recovers meaningful signal that no amount of additional data collected under the same protocol could provide. Judge confounders are structural, not stochastic. Table 1 reports the estimated parameters from the full model. Position bias (γ^0=−0.165 γ_0=-0.165) confirms that second-position responses are systematically favoured, consistent with the judge-dependent direction effects documented by Shi et al. Shi et al. (2025). Verbosity bias (β^=0.163 β=0.163, 95 % CI [0.058,0.268][0.058,0.268]) confirms that longer responses are preferred irrespective of content quality, extending the finding of Dubois et al. Dubois et al. (2024) to a different evaluation setting. Same-family affinity (λ^=0.411 λ=0.411) confirms that the GPT-4 judge systematically favours models from its own provider family, a direct violation of the measurement invariance property that any reliable leaderboard requires. Substantial judge-level variation in position sensitivity (SD(uj)=0.337(u_j)=0.337) compounds this further, in the MT-Bench data, claude-v1 appeared in second position far more often than first while llama-13b faced the opposite exposure, so the global position parameter accrues as a tournament-design artefact rather than a model quality signal. None of these effects can be eliminated by collecting more data under the same evaluation protocol, because they are properties of the measurement instrument rather than of sampling variation. Table 1: Estimated bias parameters from the hybrid model on MT-Bench. The parameter estimates are statistically significant or substantially non-zero, confirming that LLM-judge evaluation is not bias-free at any sample size. Parameter Estimate 95 % CI γ^0 γ_0 (position bias) −0.165-0.165 −0.253-0.253 +0.076+0.076 β β (verbosity) +0.163+0.163 +0.058+0.058 +0.268+0.268 λ λ (same-family affinity) +0.411+0.411 — — SD⁡(uj)SD(u_j) across judges 0.3370.337 — — Bias correction recovers signal that more data cannot. Figure 1: Correcting for judge effects shifts ability estimates substantially. Each arrow shows the change from naive Bradley-Terry (orange) to the Judge-corrected model (blue). Provider-affiliated models receive the largest upward corrections; non-affiliated models shift downward. These distortions are structural and collecting more comparisons under the same protocol would not resolve them. Figure 1 shows the net outcome of jointly correcting for all three bias components. Ability estimates shift substantially and in model-specific directions that trace directly back to tournament design rather than model quality. Critically, this signal recovery requires no additional data: the bias-corrected model is fitted to the same battles as the naive baseline. The two frameworks disagree on a non-trivial share of model pairs (Kendall τ=0.80τ=0.80) even when all available data are used, a structural disagreement that persists regardless of how many observations each framework receives. When bias is structural, the path to reliable rankings runs through better measurement, not more measurement. 5 Sustainability: Fewer Comparisons The measurement argument has a direct sustainability consequence. A misspecified model treats structural rater noise as signal, so every comparison it schedules partly compensates for bias rather than resolving genuine uncertainty about model ability. A correctly specified model never accumulates this waste in the first place, because it attributes rater noise to the judge rather than to the models being compared. The result is stable rankings from fewer comparisons and, at leaderboard scale, fewer comparisons means substantially less LLM inference. To quantify this, we ran a simulation of N=30N=30 models across six tournament designs (Uniform, Swiss, Stratified, HubSpoke, MultiHub, Popularity) and three mixes of ordinal and pairwise data (10%, 50%, and 90% ordinal), averaging Kendall-τ correlation with ground truth over 50 trials per configuration. Figure 2 reports the results. Across almost all settings, the bias-corrected hybrid model (HybridRater) recovers stable rankings at sample sizes where naive win-rate and simple score averaging remain well below equivalent fidelity. The gap is largest in the adversarial designs- Swiss, Stratified, and MultiHub which most closely resemble real leaderboard conditions where scheduling is non-uniform and model exposure is unbalanced. In these regimes, naive aggregation plateaus well below the hybrid model even at the largest sample sizes tested, confirming that the efficiency gain is not merely a matter of getting there faster, it is a matter of getting there at all. Figure 2: Hybrid model recovers ground-truth rankings with fewer comparisons. Mean Kendall-τ correlation with ground truth over 50 trials, shown across six tournament structures and three proportions of ordinal data. HybridRater consistently outperforms naive win-rate and simple score averaging, particularly in sparse, adversarially scheduled designs that resemble real leaderboard conditions. The efficiency gain is largest where it matters most, at low sample sizes and in unbalanced tournament designs. Careful selection of the scaling parameter ω is important when combining the two measurement scales. Real leaderboards can recover further gains through adaptive scheduling, rather than drawing pairs at random, new comparisons can be targeted where uncertainty in θ is highest, and the Fisher information of the joint latent-variable model provides a principled criterion for this. The connection to computerised adaptive testing in educational measurement Wainer et al. (2000) is direct. Zhuang et al. Zhuang et al. (2025) argue that AI evaluation should adopt precisely these methods from human psychometric testing; Polo et al. Polo et al. (2024) demonstrate their practical value, achieving 99% cost reduction through IRT-based item selection on static benchmarks; and Li et al. Li et al. (2025b) extend this to a fully active setting, framing evaluation as a sequential acquisition problem where each comparison is selected to maximise expected information gain about model ability. Chatbot Arena already exploits a heuristic version of this, reporting a 54% reduction in required battles through adaptive sampling Chiang et al. (2024); the psychometric framework proposed here replaces that heuristic with a grounded information-theoretic criterion. Full implementation of adaptive scheduling within the hybrid framework is an important direction for future work. 6 Limitations and Open Problems Unidimensionality. The framework assumes a single latent ability dimension θ. For models evaluated across diverse tasks, a multidimensional IRT model would be more appropriate: Burnell et al. Burnell et al. (2023a) show via factor analysis that LLM capabilities have non-trivial latent structure, making a single θ an approximation. Extending the hybrid framework to MIRT is an important open direction. Construct validity. Correcting for known confounders is necessary but not sufficient. Whether the corrected θ tracks the construct of interest requires systematic validation not yet done at scale for LLM-as-a-judge evaluation Wang et al. (2026). The psychometric framing at least renders this an explicit, testable property of the measurement system rather than an unexamined assumption. 7 Conclusion Current leaderboards treat instability as a data problem and respond by scaling comparisons. We have argued it is a measurement problem, the biases that distort rankings are structural properties of the evaluation instrument, not sampling artefacts, and no volume of additional battles under the same protocol will correct them. On MT-Bench, position preference, verbosity, self-enhancement, and judge-level heterogeneity are simultaneously detectable, significant, and large enough to move ability estimates in directions that trace back to tournament design rather than model quality. The remedy is inexpensive. Fitting a generalised linear mixed model costs negligible compute relative to a single round of LLM inference, yet it separates ability from evaluator artefact in a way that arbitrarily large naive samples cannot. The unified framework proposed here—augmented Bradley–Terry models for pairwise data, many-facet Rasch models for ordinal data, and a joint likelihood that bridges both—provides the measurement foundation that current leaderboards lack. More reliable rankings, from fewer comparisons, at lower computational cost, is what sustainable evaluation looks like in practice. Acknowledgements. We thank the anonymous reviewers. This work was funded by the German Federal Ministry for Research, Technology and Space Travel (BMFTR) grant number 01IW23005. Declaration on Generative AI During the preparation of this work the author(s) used Claude for grammar and spelling checks and minor phrasing improvements. After using the tool, the author(s) reviewed and edited the content as needed and take(s) full responsibility for the publication’s content. References Bender et al. (2021) E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell On the dangers of stochastic parrots: can language models be too big?. In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, p. 610–623. Cited by: §1. Boyeau et al. (2025) P. Boyeau, A. N. Angelopoulos, T. Li, N. Yosef, J. Malik, and M. I. Jordan AutoEval done right: using synthetic data for model evaluation. In International Conference on Machine Learning, p. 5276–5290. Cited by: §1. Bradley and Terry (1952) R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §3.1. Burnell et al. (2023a) R. Burnell, H. Hao, A. R. Conway, and J. H. Orallo Revealing the structure of language model capabilities. arXiv preprint arXiv:2306.10062. Cited by: §6. Burnell et al. (2023b) R. Burnell, W. Schellaert, J. Burden, T. D. Ullman, F. Martinez-Plumed, J. B. Tenenbaum, D. Rutar, L. G. Cheke, J. Sohl-Dickstein, M. Mitchell, et al. Rethink reporting of evaluation results in ai. Science 380 (6641), p. 136–138. Cited by: §1. Chiang et al. (2024) W. Chiang, L. Zheng, Y. Sheng, A. N. Angelopoulos, T. Li, D. Li, H. Zhang, B. Zhu, M. Jordan, J. E. Gonzalez, et al. Chatbot arena: an open platform for evaluating llms by human preference. arXiv preprint arXiv:2403.04132. Cited by: §2, §5. DeMars (2010) C. DeMars Item response theory. Oxford University Press. Cited by: §2, §3. Dubois et al. (2024) Y. Dubois, P. Liang, and T. Hashimoto Length-controlled alpacaeval: a simple debiasing of automatic evaluators. In First Conference on Language Modeling, Cited by: §1, §2, §3.1, §3.1, §4. Federiakin (2025) D. Federiakin Improving llm leaderboards with psychometrical methodology. arXiv preprint arXiv:2501.17200. Cited by: §2. Gu et al. (2026) J. Gu, X. Jiang, Z. Shi, H. Tan, X. Zhai, C. Xu, W. Li, Y. Shen, S. Ma, H. Liu, et al. A survey on llm-as-a-judge. The Innovation 7 (6). Cited by: §1, §1, §2. Hamilton and Tawn (2026) I. Hamilton and N. Tawn Parameter estimation in comparative judgment under random and adaptive scheduling schemes. Journal of Educational Measurement 63 (1), p. e70022. Cited by: §2, §3.3. Jordan (2016) M. I. Jordan On computational thinking, inferential thinking and data science.. In SPAA, p. 47. Cited by: §2. Kipnis et al. (2025) A. Kipnis, K. Voudouris, L. Schulze Buschoff, and E. Schulz Metabench-a sparse benchmark of reasoning and knowledge in large language models. In International Conference on Learning Representations, Vol. 2025, p. 31734–31770. Cited by: §1. Lalor et al. (2016) J. P. Lalor, H. Wu, and H. Yu Building an evaluation scale using item response theory. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 648–657. Cited by: §2. Li et al. (2024) H. Li, Q. Dong, J. Chen, H. Su, Y. Zhou, Q. Ai, Z. Ye, and Y. Liu Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579. Cited by: §1, §1, §2. Li et al. (2025a) T. Li, W. Chiang, E. Frick, L. Dunlap, T. Wu, B. Zhu, J. E. Gonzalez, and I. Stoica From crowdsourced data to high-quality benchmarks: arena-hard and benchbuilder pipeline. In International Conference on Machine Learning, p. 34209–34231. Cited by: §2. Li et al. (2025b) Y. Li, J. Ma, M. Ballesteros, Y. Benajiba, and G. Horwood Active evaluation acquisition for efficient llm benchmarking. In International Conference on Machine Learning, p. 35581–35602. Cited by: §5. Liddell and Kruschke (2018) T. M. Liddell and J. K. Kruschke Analyzing ordinal data with metric models: what could possibly go wrong?. Journal of Experimental Social Psychology 79, p. 328–348. Cited by: §2, §3.2. Linacre (1989) J. M. Linacre Many-faceted rasch measurement. Ph.D. Thesis, The University of Chicago. Cited by: §3.2. Marsh et al. (2018) L. E. Marsh, P. Kanngiesser, and B. Hood When and how does labour lead to love? the ontogeny and mechanisms of the ikea effect. Cognition 170, p. 245–253. Cited by: §1. Martínez-Plumed et al. (2019) F. Martínez-Plumed, R. B. Prudêncio, A. Martínez-Usó, and J. Hernández-Orallo Item response theory in ai: analysing machine learning classifiers at the instance level. Artificial intelligence 271, p. 18–42. Cited by: §2. Polo et al. (2024) F. M. Polo, L. Weber, L. Choshen, Y. Sun, G. Xu, and M. Yurochkin TinyBenchmarks: evaluating llms with fewer examples. In Proceedings of the 41st International Conference on Machine Learning, p. 34303–34326. Cited by: §1, §5. Samsi et al. (2023) S. Samsi, D. Zhao, J. McDonald, B. Li, A. Michaleas, M. Jones, W. Bergeron, J. Kepner, D. Tiwari, and V. Gadepally From words to watts: benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), p. 1–9. Cited by: §1. Shi et al. (2025) L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi Judging the judges: a systematic study of position bias in llm-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, p. 292–314. Cited by: §1, §4. Wainer et al. (2000) H. Wainer, N. J. Dorans, R. Flaugher, B. F. Green, and R. J. Mislevy Computerized adaptive testing: a primer. Routledge. Cited by: §5. Wang et al. (2026) X. Wang, L. Jiang, J. Hernandez-Orallo, D. Stillwell, S. Chen, L. Sun, F. Luo, and X. Xie Evaluating general-purpose ai with psychometrics. Communications of the ACM 69 (5), p. 92–102. Cited by: §2, §6. Ye et al. (2025) H. Ye, J. Jin, Y. Xie, X. Zhang, and G. Song Large language model psychometrics: a systematic review of evaluation, validation, and enhancement. arXiv preprint arXiv:2505.08245. Cited by: §1. Zheng et al. (2023) L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems 36, p. 46595–46623. Cited by: §1, §2, §4. Zhuang et al. (2025) Y. Zhuang, Q. Liu, Z. Pardos, P. C. Kyllonen, J. Zu, Z. Huang, S. Wang, and E. Chen Position: ai evaluation should learn from how we test humans. In Forty-second International Conference on Machine Learning Position Paper Track, Cited by: §5.

Instant research discovery