Paper deep dive

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation

Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, Céline Hudelot, Pierre Colombo

Year: 2026Venue: arXiv preprintArea: cs.CLType: PreprintEmbeddings: 95

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 93%

Last extracted: 4/14/2026, 1:54:38 AM

Summary

The paper introduces 'BERT-as-a-Judge', an encoder-driven approach for evaluating LLM outputs in reference-based settings. It addresses the limitations of rigid lexical (regex-based) evaluation and the high computational cost of LLM-as-a-Judge methods. By training on synthetically annotated triplets, the model achieves superior accuracy and robustness across 36 models and 15 tasks, providing a scalable and efficient alternative for model evaluation.

Entities (5)

BERT-as-a-Judge · evaluation-framework · 100%Lexical Evaluation · evaluation-method · 95%LLM-as-a-judge · evaluation-method · 95%EuroBERT · model-architecture · 90%Nemotron-Super-v1.5 · model · 90%

Relation Signals (3)

BERT-as-a-Judge → outperforms → Lexical Evaluation

confidence 95% · We show that it consistently outperforms the lexical baseline

BERT-as-a-Judge → uses → EuroBERT

confidence 95% · We initialize the encoder from EuroBERT 210M

Nemotron-Super-v1.5 → annotates → Training Data

confidence 90% · we employ Nemotron-Super-v1.5 as an automatic evaluator

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, however, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model's true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judgments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthetically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling tradeoff between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge's performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

94,661 characters extracted from source content.

Expand or collapse full text

BERT-AS-A-JUDGE: A ROBUST ALTERNATIVE TO LEXICAL METHODS FOR EFFICIENT REFERENCE-BASED LLM EVALUATION Hippolyte Gisserot-Boukhlef 1,4 Nicolas Boizard 2,4 Emmanuel Malherbe 1 C ́ eline Hudelot 4 Pierre Colombo 3 1 Artefact Research Center 2 Diabolocom 3 Cohere 4 MICS, CentraleSup ́ elec, Universit ́ e Paris-Saclay Accurate evaluation is central to the large language model (LLM) ecosystem, guiding model selection and downstream adoption across diverse use cases. In practice, how- ever, evaluating generative outputs typically relies on rigid lexical methods to extract and assess answers, which can conflate a model’s true problem-solving ability with its compliance with predefined formatting guidelines. While recent LLM-as-a-Judge approaches mitigate this issue by assessing semantic correctness rather than strict structural conformity, they also introduce substantial computational overhead, making evaluation costly. In this work, we first systematically investigate the limitations of lexical evaluation through a large-scale empirical study spanning 36 models and 15 downstream tasks, demonstrating that such methods correlate poorly with human judg- ments. To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for assessing answer correctness in reference-based generative settings, robust to variations in output phrasing, and requiring only lightweight training on synthet- ically annotated question-candidate-reference triplets. We show that it consistently outperforms the lexical baseline while matching the performance of much larger LLM judges, providing a compelling tradeoff between the two and enabling reliable, scalable evaluation. Finally, through extensive experimentation, we provide detailed insights into BERT-as-a-Judge’s performance to offer practical guidance for practitioners, and release all project artifacts to foster downstream adoption. Correspondence: hippolyte.gisserot-boukhlef@centralesupelec.fr Code: https://github.com/artefactory/BERT-as-a-Judge Models & Data: https://hf.co/collections/artefactory/bert-as-a-judge Date: April 2, 2026 1 Introduction Evaluation lies at the core of the large language model (LLM) ecosystem. In recent years, considerable effort has been devoted to rigorously and fairly assessing model performance across a wide range of tasks, to guide model selection and downstream adoption (Liang et al., 2022; Bommasani et al., 2021). For instruction-tuned models (optimized for human interaction and question answering), evaluation is typically conducted in zero-shot genera- tive settings (Wei et al., 2022; Ouyang et al., 2022), in which models are prompted to directly generate an answer without access to task-specific examples. While conceptually straightforward, this setup poses two challenges for evaluation: reliably extracting the model’s predicted answer for comparison with a reference, and performing the comparison itself. The former arises from answer formatting variations, such as “The answer is X” versus “Answer: X”, the latter occurs when comparing outputs like “2.00” versus “2$”, both of which should be treated as equivalent. A common mitigation strategy is to enforce constrained output formats via prompting, enabling answers to be extracted with regular expressions (regex) (Liang et al., 2023; Gao et al., 2024), and then rely on metrics beyond exact match, such as ROUGE (Lin, 2004), BERTScore (Zhang et al., 2019), or Math- Verify (Hugging Face, 2024), thereby avoiding errors caused by formatting inconsistencies or lexical variations. Although more flexible than strict exact match, these metrics can still fail to accurately capture answer correctness, especially since models often do not strictly follow prescribed output formats, making reliable answer parsing difficult. Such deviations may stem from differences in model scale, instruction-tuning data mixtures, or alignment strategies, and can artificially deflate measured downstream performance. While arXiv:2604.09497v1 [cs.CL] 10 Apr 2026 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Extraction 1: → 4 Extraction 2: → N/A Output 1: Answer: 4 Output 2: The answer is 4 Answer Generation Regex Parsing Lexical Match Score 2: → 0 Score 1: → 1 REGEX-BASED EVALUATION Output 1: Answer: 4 Output 2: The answer is 4 Answer Generation Answer Assessment Score 1: → 1 Score 2: → 1 BERT-AS-A-JUDGE Prompt: What is 2+2? Format your answer as “Answer: <answer>” Reference: 4 Prompt: What is 2+2? Format your answer as “Answer: <answer>” Reference: 4 Fine-tuned Encoder True Ranking Ministral-3 14B Qwen-3 14B Phi-4 14B Gemma-3 12B Ministral-3 14B Qwen-3 14B Phi-4 14B Gemma-3 12B Ministral-3 14B Qwen-3 14B Phi-4 14B Gemma-3 12B Figure 1: Comparison between regex-based (lexical) evaluation and BERT-as-a-Judge. Top: illustration of both approaches with simple examples. Bottom: model rankings for four similarly sized models from different families, computed via task-wise Borda count. formatting adherence is itself an important capability, particularly for instruction following and structured generation (Ouyang et al., 2022), it should not confound the evaluation of orthogonal competencies such as factual knowledge, mathematical reasoning, or reading comprehension (Hendrycks et al., 2021a; Cobbe et al., 2021). Recently, LLM-as-a-Judge frameworks have emerged as a compelling alternative (Zheng et al., 2023; Wang et al., 2023). By delegating answer comparison to a separate language model, these approaches reduce dependence on rigid formatting constraints and can cor- rectly credit semantically valid but structurally unconventional responses. However, they introduce substantial computational overhead and additional sources of variance, including sensitivity to the choice of judge model and prompt design (Boizard et al., 2025b). Question. How can we measure a model’s core problem-solving ability without relying on output formatting or expensive inference? Contributions. In this work, we make the following three contributions: •Through a comprehensive empirical study across a diverse set of models and tasks, we show that lexical evaluation exhibits weak correlation with human judgments (§ 3). • To address this limitation, we introduce BERT-as-a-Judge, an encoder-driven approach for evaluating generative models in reference-based settings, leveraging the strength of bidirectional attention for text classification (Figure 1). We show that BERT-as-a-Judge consistently outperforms lexical evaluation and even surpasses LLM-as-a-Judge under comparable inference conditions (§ 4). •We provide detailed insights into BERT-as-a-Judge’s performance through an exten- sive set of experiments, offering practical guidance for downstream applications (§ 5). Additionally, we release the packaged code 1 along with the full set of generated and annotated data, covering outputs from 36 models across 15 tasks, and open-source all post fine-tuning checkpoints used in our experiments. 2 1 https://github.com/artefactory/Bert-as-a-Judge 2 https://hf.co/collections/artefactory/bert-as-a-judge 2 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation 2 Experimental Protocol 2.1 Answer Generation Tasks. The backbone of LLM evaluation consists of tasks whose outputs can be unam- biguously judged as correct or incorrect, providing an objective basis for model assessment (Grattafiori et al., 2024; Yang et al., 2025; Olmo et al., 2025; Ramos et al., 2026; Apertus et al., 2025). In this work, we focus on three families of widely used benchmarks: •Multiple-choice, in which models are given a question along with a set of options: MMLU (Hendrycks et al., 2021a), MMLU-Pro (Wang et al., 2024), TruthfulQA (Lin et al., 2021), ARC-Easy/Challenge (Clark et al., 2018), and GPQA (Rein et al., 2024). •Context extraction, where models must provide answers grounded in a given passage by citing relevant evidence: SQuAD-v2 (Rajpurkar et al., 2018), HotpotQA (Yang et al., 2018), DROP (Dua et al., 2019), and CoQA (Reddy et al., 2019). • Open-form mathematics, in which models generate a final closed-form answer in free text: GSM8K (Cobbe et al., 2021), MATH (Hendrycks et al., 2021b), AsDiv (Miao et al., 2020), AIME 24 (Zhang & Math-AI, 2024), and AIME 25 (Zhang & Math-AI, 2025). Models. We perform inference across a broad range of recent open-weight instruction- tuned model families, spanning from 135M to 70B parameters. Our study includes 36 models in total: Llama-3 (1B, 3B, 8B, 70B) (Grattafiori et al., 2024), Qwen-3 (600M, 4B, 8B, 14B, 32B) (Yang et al., 2025), Gemma-3 (1B, 4B, 12B, 27B) (Team et al., 2025), Falcon-3 (1B, 3B, 7B) (Team, 2024), Phi-4 (3.8B, 14B) (Abdin et al., 2024; Abouelenin et al., 2025), SmolLM-2 and 3 (135M, 360M, 1.7B, 3B) (Allal et al., 2025; Bakouch et al., 2025), OLMo-3 (7B, 32B) (Olmo et al., 2025), Ministral-3 (3B, 8B, 14B) (Liu et al., 2026), LFM-2 (350M, 700M, 1.2B, 2.6B) (Liu et al., 2026), EuroLLM (1.7B, 9B, 22B) (Martins et al., 2025b;a; Ramos et al., 2026), and Apertus (8B, 70B) (Apertus et al., 2025). Generation parameters.For each task-model pair, responses are produced in a zero-shot setting using greedy decoding, with a maximum generation length of 2048 tokens. For experimental purposes, models are prompted to conclude their outputs in the format “Final answer: [answer]” to facilitate downstream regex parsing and ensure fair comparison between model- and regex-based assessment methods. 2.2 Labeling Synthetic labeling. For annotation, we employ Nemotron-Super-v1.5 (Bercovich et al., 2025) as an automatic evaluator. The model is provided with the question, the candidate answer, and the reference answer, and is asked to determine whether the candidate response is correct given the available information. 3 Inference is conducted using greedy decoding in non-reasoning mode. Human labeling.To validate the reliability of the synthetic labeling approach, we perform human annotation on a subset of the data. Specifically, we randomly sample instances from the generated dataset and have them independently labeled by a pool of 11 human evaluators, totaling 3,212 annotations, and resulting in an overall average agreement of 97.5% with the synthetic labels. 4 2.3 Evaluation Methods BERT-as-a-Judge. We propose to train a BERT-like encoder model on labeled question- candidate-reference triplets constructed as described in § 2.1 and § 2.2, leveraging its bidirec- tional attention mechanism well suited for structured text classification (Zhang et al., 2025). We construct the training mixture from the tasks described in § 2.1 that provide an explicit 3 The full evaluation prompt is provided in Appendix A. 4 Further details are provided in Appendix C. 3 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation training split, namely MMLU, ARC-Easy, ARC-Challenge, SQuAD-v2, HotpotQA, GSM8K, and Math. The training dataset is constructed to balance the number of samples across task categories and models, resulting in approximately 1M synthetically labeled samples in total. We initialize the encoder from EuroBERT 210M (Boizard et al., 2025a) and fine-tune it for one epoch using binary cross-entropy. We employ a learning rate of 2×10 −5 , following the authors’ recommendations for sequence classification, along with a 5% warmup ratio and a linear decay schedule. Training is conducted on 8 MI250x GPUs, yielding an effective batch size of 32, taking approximately 20 GPU hours per run. Baselines. We compare BERT-as-a-Judge to the following baselines: •Regex: Extracts answers using a regular expression based on the pattern “Final answer: [answer]” and evaluates multiple-choice tasks with exact match, context extraction with ROUGE-L (Lin, 2004), and open-form math with Math-Verify (Hugging Face, 2024). For answer parsing, we build on the regex rules provided by the lm-evaluation-harness framework (Gao et al., 2024), adapting them to our prompting format and the range of evaluated models. • LLM-as-a-Judge: Uses a generative model to determine whether a candidate response matches the reference answer for a given question, following the procedure described in § 2.1. To keep inference costs comparable to the encoder, we use a model of similar scale by default (Qwen-3 0.6B), prompting it to respond directly with “True” or “False”. Larger LLM judges and more flexible prompting strategies are also evaluated in § 5. 2.4 Assessment of Evaluation Quality Metric. We assess each method by its accuracy against synthetic labels from Nemotron- Super-v1.5, 5 reflecting how well it predicts whether a given answer is correct or incorrect. 6 Benchmarks.We assess all evaluation methods on the full set of tasks introduced in § 2.1, including both the test splits of tasks used during encoder training (§ 2.3) and the tasks reserved exclusively for out-of-domain evaluation. 3 Limitations of Regex-Based Evaluation This section analyzes the impact of regex-based evaluation on measured downstream performance, noting discrepancies from both formatting-related parsing failures and post- parsing matching errors. Specifically, we quantify parsing failure rates across a range of models in Figure 2 and assess performance deltas relative to ground-truth labels in Table 1. Model scale, family, and task type have a high impact on output formatting. Figure 2 shows that larger models tend to produce fewer formatting errors, as illustrated by the Llama-3 models on context extraction and Qwen-3 on open-form math tasks. Model family also plays a significant role: Qwen-3 and Gemma-3 consistently achieve near-perfect format- ting compliance on context extraction, whereas smaller Llama-3 models exhibit substantially higher failure rates. Task type further impacts formatting accuracy. Open-form math proves the most challenging, with Llama-3 70B generating incorrectly formatted outputs over 60% of the time and Qwen-3 32B around 20%, while multiple-choice and context extraction tasks are much easier, with mid- to large-scale models often achieving near-zero failure rates. Regex-based evaluation distorts performance measurements.Table 1 illustrates the risks of relying on regex-based evaluation, showing substantial negative deltas in measured performance across a broad set of models. Notably, even models with high formatting compliance (e.g., Gemma-3 family on context extraction tasks) suffer from substantial 5 For methods producing scores between 0 and 1 (e.g., encoder models with soft probabilities), we use a default threshold of 0.5. 6 To estimate performance with respect to human annotations, we apply a correction based on the observed agreement between human and synthetic labels (§ 2.2); results are reported in Appendix C. 4 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation 138 70 14 1227 0.6 48 1432 Model Size (B) 0 5 10 15 20 Parsing Failures (%) Multiple-Choice 138 70 14 1227 0.6 48 1432 Model Size (B) 0 10 20 30 Context Extraction 138 70 14 1227 0.6 48 1432 Model Size (B) 0 20 40 60 Open-Form Math Llama-3 Gemma-3 Qwen-3 Figure 2: Quantification of regex parsing failures. Values represent the failure rate, defined as the percentage of instances with unparsable outputs. Results are shown for the Llama-3, Gemma-3, and Qwen-3 model families and aggregated by task category. FamilySize Multiple-ChoiceContext ExtractionOpen-Form Math ∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank Llama-3 1B-0.9 (-2.6/+1.7)↑4.8-23.9 (-18.1/-5.8)↓1.2-11.9 (-11.1/-0.8)↓0.8 3B-13.1 (-8.5/-4.6)↓0.5-27.5 (-15.6/-11.9)↓0.4-7.0 (-2.6/-4.4)↑1.7 8B-23.3 (-0.9/-22.3)↓6.4-21.0 (-4.0/-17.0)↑3.5-5.4 (-0.8/-4.6)↑4.2 70B-1.1 (-1.1/-0.1)↑1.3-18.4 (-0.1/-18.3)↑6.5-30.4 (-29.0/-1.4)↓13.3 Gemma-3 1B-0.1 (-0.7/+0.5)↑5.7-12.3 (-0.1/-12.2)↑4.6-11.3 (-6.8/-4.5)↓1.2 4B+0.3 (-0.2/+0.5)↑6.2-30.8 (-0.0/-30.8)↓1.5-11.8 (-1.2/-10.6)↓0.1 12B-0.1 (-0.2/+0.1)↑5.3-27.6 (-0.1/-27.4)↓4.2-10.3 (-1.4/-8.9)↑1.6 27B+0.1 (-0.1/+0.2)↑4.7-28.7 (-0.0/-28.7)↓2.1-10.8 (-1.1/-9.7)↑0.4 Qwen-3 0.6B-20.8 (-0.0/-20.8)↓2.8-13.8 (-0.0/-13.8)↑2.8-10.8 (-5.8/-5.0)↓0.5 4B-3.2 (-0.5/-2.7) ↑3.5-20.2 (-0.0/-20.2)↑7.0-12.0 (-5.3/-6.7)↑1.0 8B-7.0 (-0.6/-6.3)↑0.7-29.7 (-0.0/-29.7)↓7.2-15.2 (-7.9/-7.3)↓0.2 14B-19.3 (-0.7/-18.6)↓13.2-20.9 (-0.0/-20.9)↑0.5-15.0 (-7.8/-7.2)↑0.8 32B-23.8 (-0.5/-23.3)↓17.6-25.1 (-0.0/-25.1)↓1.1-10.3 (-2.6/-7.7)↑2.8 Table 1: Impact of regex-based evaluation on measured model performance.The “∆Accuracy” columns report the difference between regex-based and ground-truth (synthetic-label) accuracy, with values in parentheses indicating the contributions from parsing failures and post-parsing matching, respectively. The “∆Rank” columns show the corresponding average change in ranking across the 36 evaluated models, where greener values indicate upward rank changes and redder values indicate downward rank changes. underestimation, often due to overly verbose outputs that technically follow formatting rules but fail lexical matching. 7 Overall, regex-based assessment significantly distorts leaderboard rankings; for instance, Qwen-3 32B drops 18 positions while Gemma-3 4B climbs 6 on multiple-choice tasks, with these shifts often representing mere artifacts of formatting quirks and rigid lexical matching rather than true differences in capability. 4 Encoder-Based Evaluation In this section, motivated by the observation that regex-based evaluation fails to accurately reflect true model performance across a wide range of benchmarks (§ 3), we train a BERT-as- a-Judge encoder model to assess answer correctness following the methodology described in § 2.3. We then assess the proposed approach using the setup detailed in § 2.4 and report the results in Table 2. To further support our analysis, Table 3 shows performance for models whose outputs were excluded from the training mixture. 7 More details are provided in Appendix E. 5 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation TaskRegexLLM-JudgeBERT-Judge Multiple-Choice ARC-Challenge89.050.299.4 ARC-Easy88.254.099.7 MMLU88.150.398.5 GPQA86.566.293.5 MMLU-Pro88.857.196.5 TruthfulQA92.554.598.6 Context Extraction HotpotQA75.670.090.9 SQuAD-v272.362.589.3 CoQA67.075.288.1 DROP77.069.388.6 Open-Form Math GSM8K94.471.398.8 Math73.458.993.7 AIME2487.877.990.0 AIME2591.883.091.4 ASDiv89.275.595.3 Table 2: Accuracies of evaluation methods against ground-truth labels across tasks, av- eraged over models. Dashed lines separate test-only tasks from those with a training split. Bold indicates the highest accuracy per task. Size Multiple-ContextOpen-Form ChoiceExtractionMath IDOODIDOODIDOOD Ministral-3 3B97.096.983.782.581.486.5 8B97.997.887.987.183.584.8 14B98.298.389.188.683.582.6 LFM-2 0.35B94.894.190.588.697.196.9 0.7B96.896.787.484.497.497.0 1.2B97.597.191.290.994.694.4 2.6B97.997.887.186.093.993.7 EuroLLM 1.7B93.493.191.591.298.598.4 9B98.698.590.790.294.594.1 22B98.298.190.690.691.591.1 Apertus 8B97.697.490.389.997.597.4 70B98.198.089.589.597.297.1 Table 3: Assessment accuracy on out-of- domain models. “ID” denotes training on all model outputs, while “OOD” excludes specific models from the training mixture. Results are aggregated by task category. BERT-as-a-Judge shows the strongest alignment with human judgments. As shown in Table 2, our trained encoder achieves the highest accuracy against ground-truth labels across all benchmarks. This advantage holds across task types, reaching near-perfect alignment on multiple-choice datasets (e.g., 99.7% on ARC-Easy and 98.5% on MMLU) and remaining high on complex-output tasks (98.8% on GSM8K and 93.9% on MATH). It also consis- tently outperforms the regex-based method by substantial margins (e.g., +21.1% on CoQA, +20.3% on MATH, +10.4% on ARC-Challenge), demonstrating that a dedicated encoder more reliably captures answer correctness than rigid lexical heuristics, while remaining computationally efficient (≈200 ms per sample on an Apple M1 CPU). BERT-as-a-Judge is robust to out-of-domain tasks.Beyond its strong overall accuracy, the encoder-based method maintains high accuracy even on tasks excluded from the training mixture (e.g., 98.6% on TruthfulQA, 88.1% on CoQA, and 95.3% on ASDiv), highlighting its strong generalization ability across the three task categories considered (Table 2). BERT-as-a-Judge generalizes to unseen models.Table 3 shows that removing generations from specific models in the training mixture has minimal impact on downstream assessment quality for those excluded instances. This demonstrates the strong generalization ability of our approach, indicating it can be safely extended to additional model families outside the training mixture and supporting broader adoption. Typical encoder scales appear insufficient for LLM-as-a-Judge. Even at three times the size of our encoder, the baseline LLM judge (Qwen-3 0.6B) consistently produces the weakest results, substantially underperforming the regex baseline across all task categories (Table 2). For instance, on ARC-Challenge, it achieves only 50.2% accuracy versus 89.0% for regex, with similar gaps on context extraction tasks (62.5% vs. 72.3% on SQuAD-v2). These results suggest that the evaluation capabilities of LLM-as-a-Judge do not hold for generative models under 1B parameters, highlighting the critical role of scale for such methods. We examine this limitation further in § 5. 5 Experimental Analysis In this section, we further investigate the properties of our encoder-based evaluation method through a series of complementary analyses. 6 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation 10 9 10 10 10 11 10 12 Inference FLOPs 40 60 80 100 Assessment Accuracy Multiple-Choice 10 9 10 10 10 11 10 12 Inference FLOPs Context Extraction 10 9 10 10 10 11 10 12 10 13 Inference FLOPs Open-Form Math BERT-J. Qw.-3 (S) Qw.-3 (L) Gem.-3 (S) Gem.-3 (L) Figure 3: Comparison between encoder-based evaluation and LLM judges from the Qwen-3 and Gemma-3 families across different model sizes and inference budgets. “S” (“short”) denotes the default generation setup, in which the model answers directly with “True” or “False”, while “L” (“long”) allows the generation of intermediate chain-of-thought tokens before the final judgment. BERT-as-a-Judge consistently outperforms LLM-as-a-Judge across a wide range of infer- ence budgets.As shown in Table 2, generative evaluation performs poorly at small scales (0.6B parameters). To complement these findings, we conduct a more extensive comparison by varying the judge family (Qwen-3, Gemma-3), model size (0.6B to 32B for Qwen-3 and 1B to 27B for Gemma-3), and inference budget (allowing or not intermediate chain-of-thought tokens before producing the final assessment). Figure 3 shows that our encoder matches the performance of the top-performing LLM judges in the defined setup, while remaining drastically less computationally expensive in terms of inference FLOPs. 8 100K200K500K1M Training Samples 88 90 92 94 96 98 Assessment Accuracy Multiple-Choice Context Extraction Open-Form Math Figure 4: BERT-as-a Judge eval- uation quality across different training budgets. BERT-as-a-Judge is training-efficient. By default, we train encoder models on 1M question-candidate-reference triplets. In this experiment, we evaluate lighter configura- tions: 500K, 200K, and 100K samples (Figure 4). Remark- ably, 100K training samples are sufficient to accurately evaluate multiple-choice and open-form math tasks, with no significant improvement observed beyond this point. Gains are more noticeable for context extraction, as ex- pected, since this task category requires more than simple candidate-reference matching and often demands under- standing of the context provided by the question. Overall, with just 2 GPU hours of training (corresponding to the 100K-sample configuration), our encoder achieves a high assessment accuracy, making it well-suited for settings with limited data and computational resources. Task CategoryRegexBERT-J. Regex+ BERT-J. Multiple-Choice88.897.790.5 Context Extraction73.089.275.2 Open-Form Math87.393.989.9 Table 4: Comparison of hybrid answer evaluation (Regex+BERT-J.) with standalone regex and BERT-as-a-Judge. Bold values in- dicate the highest accuracy in each row. Task CategoryRegex BERT-Judge w/ Q.w/o Q. Multiple-Choice88.897.797.3 Context Extraction73.089.284.2 Open-Form Math87.393.993.9 Table 5: Impact of including (w/ Q.) or ex- cluding (w/o Q.) the question in the encoder training prompt. Bold values denote the highest accuracy for each task category. 8 Inference FLOPs are estimated using the formula from Kaplan et al. (2020): FLOPs = 2×model size (in parameters)× number of generated tokens. 7 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Combining BERT-as-a-Judge with regex offers an efficient compromise. In this experi- ment, we use BERT-as-a-Judge as a fallback when regex parsing fails. While it does not reach the performance of the standalone encoder, this hybrid approach substantially improves over regex alone. The results demonstrate that selectively applying the encoder can recover a significant portion of assessment accuracy while keeping computational overhead low (for example, reducing total compute by a factor of five for a model with 20% regex failures). Removing the question from the prompt yields a controlled performance decrease.As shown in Table 5, omitting the question during encoder training (leaving only the candidate and reference) reduces overall assessment accuracy by removing some contextual informa- tion. However, this decrease is well-controlled across tasks. The question-free encoder still outperforms regex and remains close to the full-prompt setup, particularly on multiple- choice and open-form math tasks, while also reducing runtime due to shorter prompts and enabling application to any task with fully textual outputs, including multimodal tasks. The performance gap is slightly larger for context extraction, where the question provides critical information, underscoring the encoder’s context-aware capabilities. Task Test Set (Form.)Test Set (Free) RegexBERT-J. (Form.)BERT-J. (Free)RegexBERT-J. (Form.)BERT-J. (Free) Multiple-Choice88.897.797.4–94.097.6 Context Extraction73.089.285.8–84.391.6 Open-Form Math87.393.993.7–93.193.5 Table 6: Encoder’s robustness to answer formatting. “Test Set (Form.)” denotes test sets with formatted answers, while “Test Set (Free)” contains unformatted answers. “Regex” columns show regex-based results, and “BERT-J. (Form.)” and “BERT-J. (Free)” report accuracies for BERT-as-a-Judge encoders trained on formatted and unformatted answers, respectively. Regex results are omitted for the free-format test set, as answers cannot be reliably parsed. BERT-as-a-Judge is robust to variations in answer formatting guidelines. In our core experiments, and to ensure a fair comparison with the regex baseline, we train and evaluate the encoder on answers formatted to facilitate lexical parsing (§ 2.1). In practice, however, users may follow custom guidelines or allow free-form responses. 9 Table 6 shows that under cross-formatting evaluation, meaning free-to-formatted (training on free-form, evaluating on formatted answers) and formatted-to-free (the reverse), we observe a slight performance drop compared to aligned settings. Nevertheless, the encoder still substantially outperforms regex, demonstrating strong robustness to variations in answer formatting. As expected, the free-to-formatted encoder consistently outperforms the formatted-to-free variant, benefiting from exposure to a wider range of formats during training and making it the preferred choice for downstream applications. 0.000.250.500.751.00 Threshold 20 40 60 80 100 Assessment Accuracy Multiple-Choice ARC-Challenge ARC-Easy GPQA MMLU MMLU-Pro TruthfulQA 0.000.250.500.751.00 Threshold Context Extraction CoQA DROP HotpotQA SQuAD-v2 0.000.250.500.751.00 Threshold Open-Form Math AIME24 AIME25 ASDiv GSM8K Math Figure 5: Effect of score thresholding on BERT-as-a-Judge downstream assessment accuracy across the three task categories, averaged over all models. 9 Further details are provided in Appendix A. 8 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation BERT-as-a-Judge is robust to decision threshold variations. Encoder classifiers output continuous sigmoid probabilities, requiring a decision threshold for discrete evaluation. While our main experiments (§ 4) use a standard 0.5 threshold, Figure 5 demonstrates that accuracies remain remarkably stable across a broad spectrum of threshold values for all task categories. This invariance indicates strong separation between classes, enabling reliable off-the-shelf deployment without the need for task-specific threshold tuning. 6 Related Work Traditional LLM evaluation. Pretrained language models have traditionally been evalu- ated using log-likelihood (Radford et al., 2019) or few-shot generation (Brown et al., 2020; Rae et al., 2021; Chowdhery et al., 2023; Touvron et al., 2023; Bai et al., 2023). With the rise of instruction-tuned models, zero-shot generative evaluation has become standard (Wei et al., 2022; Chung et al., 2024; Yang et al., 2025; Ramos et al., 2026). This paradigm typically enforces structured outputs via prompting (Liang et al., 2023; Gao et al., 2024), followed by rule-based comparison to references using deterministic metrics such as exact match, ROUGE (Lin, 2004), Math-Verify (Hugging Face, 2024), or Code-Eval (Chen et al., 2021), making evaluation highly sensitive to surface-level formatting. Model-based evaluation. Both lexical parsing and matching introduce limitations in capturing semantic correctness and robustness. Lexical overlap does not guarantee semantic equivalence, motivating neural metrics such as BERTScore (Zhang et al., 2019) and InfoLM (Colombo et al., 2022) for general text generation, as well as task-specific evaluators like COMET (Rei et al., 2022a;b; Guerreiro et al., 2024), MetricX (Juraska et al., 2023; 2025), and BLEURT (Sellam et al., 2020). Additionally, reliably extracting model outputs is challenging when formatting is inconsistent (Zhou et al., 2023; Pyatkin et al., 2025). To mitigate these issues, LLM-as-a-Judge approaches (Zheng et al., 2023; Wang et al., 2023; Bavaresco et al., 2025; Kim et al., 2023; 2024) directly assess candidate-reference equivalence across tasks, offering greater robustness to formatting artifacts, albeit at a substantial computational cost. 7 Conclusion In this work, we show that standard evaluation protocols often conflate a model’s un- derlying problem-solving ability with its compliance to formatting constraints. Across extensive experiments spanning diverse models and tasks, we demonstrate that regex-based evaluation can substantially underestimate true performance. To address this, we propose BERT-as-a-Judge, a lightweight encoder-based framework that better captures semantic correctness, aligns more closely with human judgment, and avoids the high computational cost of LLM-as-a-Judge methods, enabling efficient and more reliable evaluation. 8 Limitations and Future Work While BERT-as-a-Judge shows strong alignment with human judgments and effectively mitigates the limitations of lexical assessment, our study focuses on a specific subset of evaluation settings, namely, English benchmarks with objectively verifiable answers, where correctness can be clearly defined. Building on these results and the existing literature, a natural next step is to broaden the scope of encoder-based evaluation toward more general-purpose settings. This includes expanding beyond fact-based and structured tasks to open-ended generation scenarios such as summarization, machine translation, code generation, and instruction following. Additionally, adapting the framework to multilingual contexts would further improve its applicability across diverse use cases. As foundation models continue to evolve toward multimodal capabilities, extending this approach to handle vision and speech inputs also presents a promising avenue. Exploring such cross-modal evaluation settings (e.g., visual question answering, image captioning, or speech-based reasoning) could help move toward a unified and efficient evaluation paradigm applicable across tasks and modalities. 9 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Ethics Statement In conducting this research, we recognize the critical importance of fair and reliable eval- uation in the LLM ecosystem. Evaluation metrics that are closely aligned with human judgments are essential to ensure that model comparisons accurately reflect real-world capabilities across the widest possible range of tasks. At the same time, the increasing scale of model evaluation, driven by more models, longer outputs, and a growing number of benchmark tasks, can impose substantial computational costs, raising concerns about acces- sibility and environmental impact. Our work emphasizes the development of lightweight, encoder-based evaluation methods that maintain high correlation with human judgments while minimizing compute requirements. By prioritizing both fairness and efficiency, we aim to support responsible, reproducible, and scalable evaluation practices in the broader LLM research community. Acknowledgments We gratefully acknowledge the ADASTRA supercomputer at CINES for its technical support and access to HPC resources (grants C1615122 and GDA2401). This work was also supported by the French government under the France 2030 program (ArGiMi project). We further thank the 11 human annotators from the Artefact Research Center, whose careful efforts were instrumental in validating the correlation between our synthetic labeling strategy and human judgments, ensuring the reliability and rigor of our evaluation results. 10 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation References Marah Abdin, Jyoti Aneja, Harkirat Behl, S ́ ebastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08905, 2024. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras. arXiv preprint arXiv:2503.01743, 2025. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Mart ́ ın Bl ́ azquez, Guilherme Penedo, Lewis Tunstall, Andr ́ es Marafioti, Hynek Kydl ́ ı ˇ cek, Agust ́ ın Piqueres Lajar ́ ın, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Cl ́ ementine Four- rier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big – data- centric training of a small language model, 2025. Project Apertus, Alejandro Hern ́ andez-Cano, Alexander H ̈ agele, Allen Hao Huang, Ange- lika Romanou, Antoni-Joan Solergibert, Barna Pasztor, Bettina Messmer, Dhia Garbaya, Eduard Frank ˇ Durech, et al. Apertus: Democratizing open and compliant llms for global language environments. arXiv preprint arXiv:2509.14233, 2025. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Car- los Miguel Pati ̃ no, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallou ́ edec, Kashif Rasul, Nathan Habib, Cl ́ ementine Fourrier, Hynek Kydlicek, Guil- herme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, Xuan- Son Nguyen, Colin Raffel, Leandro von Werra, and Thomas Wolf. SmolLM3: smol, multilingual, long-context reasoner, 2025. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern ́ andez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), p. 238–255, 2025. Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prab- humoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert Mc- Queen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Roma- nenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yut- ing Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, 11 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schnei- der, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung, and Chris Alexiuk. Llama-nemotron: Efficient reasoning models, 2025. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, Andr ́ e Martins, Ayoub Hammal, Caio Corro, C ́ eline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, Jo ̃ ao Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei, and Pierre Colombo. Eu- robert: Scaling multilingual encoders for european languages, 2025a. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Kevin El-Haddad, C ́ eline Hudelot, and Pierre Colombo. When does reasoning matter? a controlled study of reasoning’s contribution to model performance. arXiv preprint arXiv:2509.22193, 2025b. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evalu- ating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of machine learning research, 24(240):1–113, 2023. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Pierre Jean A Colombo, Chlo ́ e Clavel, and Pablo Piantanida. Infolm: A new metric to evaluate summarization & data2text generation. In Proceedings of the AAAI conference on artificial intelligence, volume 36, p. 10554–10562, 2022. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL, 2019. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 12 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and Andr ́ e FT Martins. xcomet: Transparent machine translation evaluation through fine- grained error detection. Transactions of the Association for Computational Linguistics, 12: 979–995, 2024. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b. Hugging Face. Introducing the math verify leaderboard, 2024. Accessed: 2026-03-05. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. Metricx-23: The google submission to the wmt 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, p. 756–767, 2023. Juraj Juraska, Tobias Domhan, Mara Finkelstein, Tetsuji Nakagawa, Geza Kovacs, Daniel Deutsch, Pidong Wang, and Markus Freitag. Metricx-25 and gemspaneval: Google translate submissions to the wmt25 evaluation shared task. In Proceedings of the Tenth Conference on Machine Translation, p. 957–968, 2025. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. Prometheus: Inducing fine- grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2023. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 4334–4353, 2024. Percy Liang, Rishi Bommasani, Tony Lee, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Ya- sunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav San- thanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. Featured Certification, Expert Certification. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, p. 74–81, 2004. Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods, 2021. Alexander H Liu, Kartik Khandelwal, Sandeep Subramanian, Victor Jouault, Abhinav Rastogi, Adrien Sad ́ e, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, et al. Ministral 3. arXiv preprint arXiv:2601.08584, 2026. 13 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Pedro Henrique Martins, Jo ̃ ao Alves, Patrick Fernandes, Nuno M Guerreiro, Ricardo Rei, Amin Farajian, Mateusz Klimaszewski, Duarte M Alves, Jos ́ e Pombal, Nicolas Boizard, et al. Eurollm-9b: Technical report. arXiv preprint arXiv:2506.04079, 2025a. Pedro Henrique Martins, Patrick Fernandes, Jo ̃ ao Alves, Nuno M Guerreiro, Ricardo Rei, Duarte M Alves, Jos ́ e Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, et al. Eurollm: Multilingual language models for europe. Procedia Computer Science, 255: 53–62, 2025b. Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics, p. 975–984, 2020. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heine- man, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025. Long Ouyang, Jeffrey Wu, Xu Jiang, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022. Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following, 2025. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021. Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), p. 784–789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-2124. Miguel Moura Ramos, Duarte M Alves, Hippolyte Gisserot-Boukhlef, Jo ̃ ao Alves, Pedro Hen- rique Martins, Patrick Fernandes, Jos ́ e Pombal, Nuno M Guerreiro, Ricardo Rei, Nicolas Boizard, et al. Eurollm-22b: Technical report. arXiv preprint arXiv:2602.05879, 2026. Siva Reddy, Danqi Chen, and Christopher D. Manning. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. doi: 10.1162/tacla00266. Ricardo Rei, Jos ́ e G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and Andr ́ e F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Philipp Koehn, Lo ̈ ıc Barrault, Ond ˇ rej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-juss ` a, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, Andr ́ e Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aur ́ elie N ́ ev ́ eol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), p. 578–585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022a. Association for Computational Linguistics. doi: 10.18653/v1/2022.wmt-1.52. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, Jos ́ e G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andr ́ e F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality esti- mation shared task. In Philipp Koehn, Lo ̈ ıc Barrault, Ond ˇ rej Bojar, Fethi Bougares, Rajen 14 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Chatterjee, Marta R. Costa-juss ` a, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, Andr ́ e Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aur ́ elie N ́ ev ́ eol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation (WMT), p. 634–645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022b. Association for Computational Linguistics. doi: 10.18653/v1/2022.wmt-1.60. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. In First conference on language modeling, 2024. Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020.acl-main.704. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ram ́ e, Morgane Rivi ` ere, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Ga ̈ el Liu, Francesco Visin, Kathleen Ke- nealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupati- raju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, Andr ́ as Gy ̈ orgy, Andr ́ e Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, CJ Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Pluci ́ nska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim P ̃ oder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yu- vein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and L ́ eonard Hussenot. Gemma 3 technical report, 2025. TII Team. The falcon 3 family of open models, December 2024. 15 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ e Lacroix, Baptiste Rozi ` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Xuechen Wang et al. Judgelm: Fine-tuned large language models are scalable judges. arXiv preprint arXiv:2310.17631, 2023. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. Jason Wei, Maarten Bosma, Vincent Zhao, et al. Finetuned language models are zero-shot learners. International Conference on Learning Representations, 2022. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. Junyan Zhang, Yiming Huang, Shuliang Liu, Yubo Gao, and Xuming Hu. Do bert-like bidirectional models still perform better on text classification in the era of llms. arXiv preprint arXiv:2505.18215, 1, 2025. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2024, 2024. Yifan Zhang and Team Math-AI. American invitational mathematics examination (aime) 2025, 2025. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. 16 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation A Prompting Details In this section, we detail how model outputs are generated for both answer generation (Table 7) and answer assessment (Table 9 , Table 10). For answer generation, we also describe the suffixes used to impose different formatting instructions (Table 8). Throughout the main text, we use the soft configuration by default, as it allows both regex-based answer parsing and the inclusion of intermediate chain-of-thought tokens, which improve answer quality (Appendix B). For regex parsing under both the soft and strict formatting constraints, we use the pattern “Final answer:\ *(.+)”, which provides a general and flexible mechanism for extracting the predicted answer. Task CategoryGeneration Prompt Multiple-Choice Answer the following multiple-choice question. Question: question Choices: A) choice_1 B) choice_2 C) choice_3 D) choice_4 [...] Context Extraction Answer the question based on the provided context. Context: context Question: question Open-Form Math question Table 7: Base generation prompts for each task category. Task CategoryFormatting InstructionGeneration Suffix Multiple-Choice FreeNone Soft Conclude your response with "Final answer: X", where X is the letter of the correct choice. Strict Respond only with the exact format "Final answer: X", where X is the letter of the correct choice. Context Extraction FreeNone Soft Conclude your response with "Final answer: X", where X is the exact span from the context that answers the question. Strict Respond only with the exact format "Final answer: X", where X is the exact span from the context that answers the question. Open-Form Math FreeNone Soft Conclude your response with "Final answer: X", where X is the computed solution. Strict Respond only with the exact format "Final answer: X", where X is the computed solution. Table 8: Generation suffixes used across task categories for all formatting strategies. 17 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Assessment Prompt You are an expert evaluator. Your task is to determine whether the CANDIDATE response correctly answers the QUESTION. Judge the CANDIDATE as correct only if its final answer, disregarding any intermediate reasoning or explanation, is semantically equivalent to the REFERENCE with respect to the QUESTION. Base your judgment solely on the information given. Do not rely on external knowledge. [QUESTION starts here] question [QUESTION ends here] [REFERENCE starts here] reference [REFERENCE ends here] [CANDIDATE starts here] candidate [CANDIDATE ends here] Conclude your response with exactly one of the following: - "Final answer: True" if the CANDIDATE is correct - "Final answer: False" if the CANDIDATE is incorrect Table 9: Answer assessment prompt for LLM judges, allowing intermediate token generation before the final judgment. This involves Nemotron-Super-v1.5 for label generation (§ 2.2) and generative judges evaluated under inference budget L (Figure 3). Assessment Prompt You are an expert evaluator. Your task is to determine whether the CANDIDATE response correctly answers the QUESTION. Judge the CANDIDATE as correct only if its final answer, disregarding any intermediate reasoning or explanation, is semantically equivalent to the REFERENCE with respect to the QUESTION. Base your judgment solely on the information given. Do not rely on external knowledge. [QUESTION starts here] question [QUESTION ends here] [REFERENCE starts here] reference [REFERENCE ends here] [CANDIDATE starts here] candidate [CANDIDATE ends here] Respond with exactly one of the following strings (add no additional text): - "Final answer: True" if the CANDIDATE is correct - "Final answer: False" if the CANDIDATE is incorrect Table 10: Prompt used for direct assessment by LLM judges, applied to generative judges evaluated under inference budget S (Figure 3). 18 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation B Effect of Generation Mode on Downstream Performance TaskLog-lik. Generative StrictSoftFree Multiple-Choice ARC-Challenge46.374.476.275.0 ARC-Easy65.084.684.984.4 GPQA25.932.331.828.8 MMLU39.959.462.060.5 MMLU-Pro21.138.044.342.6 TruthfulQA33.251.351.651.3 Context Extraction CoQA–71.376.586.7 DROP–48.460.264.5 HotpotQA–66.071.082.4 SQuAD-v2–44.753.562.0 Open-Form Math AIME24–18.119.818.1 AIME25–13.214.415.7 ASDiv–68.281.582.1 GSM8K–43.173.673.6 Math–49.360.257.3 Table 11:Comparison of evaluation modes across all benchmarks. Results are averaged over all models, and bold values indicate the best performance for each task. In this section, we compare different answer pro- duction modes to assess their impact on model performance. Specifically, answers are gener- ated under three formatting regimes: 10 •Log-likelihood: Candidate answers are iter- atively appended to the prompt, and the model’s prediction is derived from the se- quence with the highest log-likelihood. 11 • Strict: The model is prompted to respond exactly with “Final answer: [answer]”. •Soft: The model is prompted to conclude its response with “Final answer: [answer]” but may reason before answering. • Free: The model may answer in any format. The results are reported in Table 11. Models demonstrate greater capacity in gen- erative mode. We first examine multiple- choice tasks by comparing generative evalua- tion against the log-likelihood approach. Our results indicate that the likelihood-based setup consistently and severely impairs performance across all evaluated multiple-choice benchmarks (e.g., -22.1% on MMLU and -29.9% on ARC-Challenge). This suggests that while likelihood evaluation offers a convenient, regex-free parsing mechanism, it significantly bottlenecks the model’s inherent problem-solving capabilities compared to generative inference. Strict formatting constraints impair performance. Setting aside likelihood-based evalu- ation, we compare strict and soft generative prompting strategies. We observe that strict prompting yields the lowest overall performance. While it performs comparably to the soft method on multiple-choice tasks, it significantly degrades performance on tasks requiring more complex outputs (e.g., -11.8% on DROP and -30.5% on GSM8K). This pronounced drop highlights the importance of allowing intermediate chain-of-thought generation to fully leverage the model’s problem-solving capacity. 10 Full prompting details are provided in Appendix A. 11 Applies only to multiple-choice tasks. 19 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation C Human-Synthetic Label Agreement Task CategoryAccuracy (%) Context Extraction96.83 Multiple-Choice96.81 Open-Form Math98.70 Average97.45 Table 12: Accuracy between human and synthetic labels across task categories. This section complements § 2.2 in the main text by presenting detailed results of the human an- notations. We report human-synthetic average agreement per task category (Table 12), showing consistently high agreement across categories, and analyze how this agreement impacts downstream performance measurement. As described in § 2.4, the reported accuracies in the main text are computed using synthetic labels generated by Nemotron-Super-v1.5. To estimate performance with respect to human annotations, we can apply a correction based on the observed agreement between human and synthetic labels. LetA H denote the accuracy with respect to human labels (unknown),A S the accuracy against synthetic labels, andρthe agreement rate between synthetic and human judgments. LetY H ,Y S , and ˆ Y be random variables representing, for a given example, the human label, the synthetic label, and the predicted label, respectively. A H =P ˆ Y = Y H (1) =P ˆ Y = Y H |Y H = Y S P ( Y H = Y S ) +P ˆ Y = Y H |Y H ̸= Y S P ( Y H ̸= Y S ) (2) =P ˆ Y = Y S |Y H = Y S P ( Y H = Y S ) +P ˆ Y ̸= Y S |Y H ̸= Y S P ( Y H ̸= Y S ) (3) =P ˆ Y = Y S P ( Y H = Y S ) +P ˆ Y ̸= Y S P ( Y H ̸= Y S ) (4) = A S ρ + (1− A S )(1−ρ)(5) = ( 2ρ− 1 ) A S + 1−ρ(6) The final expression (Equation 6) 12 can be interpreted as pulling the estimated accuracy A H toward random guessing. If human and synthetic labels are uncorrelated (ρ =0.5), the estimated accuracy drops to a random guess, regardless ofA S (see Figure 6 for illustration). 0.00.20.40.60.81.0 A S 0.0 0.2 0.4 0.6 0.8 1.0 A H = 1.0 = 0.975 = 0.95 = 0.9 = 0.75 = 0.5 0.50.60.70.80.91.0 A S = 1.0 A S = 0.8 A S = 0.6 A S = 0.4 A S = 0.2 A S = 0.0 Figure 6: Sensitivity of the A H estimate to variations in A S andρ. 12 In Equation 4, we assume( ˆ Y,Y S )⊥ (Y H = Y S ), in line with our empirical observations. Intuitively, this means that agreement between the predicted and synthetic labels, ˆ Y andY S , is independent of whether the synthetic label Y S matches the human label Y H . 20 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation D Detailed Results D.1 Regex Parsing Failures In this section, we extend Figure 2 from the main text by presenting disaggregated regex parsing failure rates across models and tasks. FamilySize ARCARC GPQAMMLU MMLUTruthful ChallengeEasyProQA Apertus 8B0.20.14.90.84.50.0 70B0.10.121.22.612.31.5 EuroLLM 1.7B97.698.783.094.169.966.3 9B0.00.011.81.211.80.0 22B0.10.07.40.43.00.0 Falcon-3 1B5.14.011.25.531.93.7 3B0.00.22.00.22.50.1 7B0.00.01.10.11.30.1 Gemma-3 1B0.10.012.52.314.50.0 4B0.00.06.20.33.30.0 12B0.00.04.90.13.00.0 27B0.00.03.60.11.30.0 LFM-2 0.35B0.40.47.11.211.34.0 0.7B2.92.818.16.115.51.6 1.2B0.50.56.71.05.40.4 2.6B0.00.210.01.49.90.6 Llama-3 1B5.03.636.49.026.05.4 3B8.15.232.421.847.416.9 8B0.10.325.22.616.12.3 70B0.00.07.40.96.00.0 Ministral-3 3B0.10.023.91.215.10.0 8B0.00.025.01.410.30.0 14B0.20.116.51.38.60.0 OLMo-3 7B0.70.422.31.915.80.0 32B0.30.051.11.617.40.0 Phi-4 3.6B0.00.02.50.23.00.0 14B0.00.00.90.00.60.0 Qwen-3 0.6B0.00.00.00.10.00.0 4B0.00.09.60.44.60.0 8B0.10.010.90.75.90.0 14B0.10.011.40.34.70.0 32B0.00.07.80.14.30.0 SmolLM-2/3 0.135B98.498.497.897.697.898.3 0.36B6.75.243.112.237.07.1 1.7B0.00.00.20.20.50.2 3B0.70.519.45.116.91.6 Table 13: Parsing failure rates on multiple-choice benchmarks. 21 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySizeCoQADROPHotpotQASQuAD-v2 Apertus 8B0.00.20.10.1 70B0.00.30.00.1 EuroLLM 1.7B41.439.716.338.6 9B0.00.00.00.0 22B0.20.00.30.0 Falcon-3 1B20.66.23.95.2 3B2.00.81.23.6 7B0.60.50.20.2 Gemma-3 1B0.00.80.10.3 4B0.00.00.10.0 12B0.20.00.30.1 27B0.00.00.00.0 LFM-2 0.35B1.82.24.13.5 0.7B20.81.30.51.1 1.2B0.20.51.50.2 2.6B1.82.81.63.2 Llama-3 1B31.253.028.933.9 3B34.65.317.127.9 8B6.47.73.59.8 70B0.00.00.00.3 Ministral-3 3B0.00.10.10.1 8B0.00.00.00.0 14B0.00.00.00.3 OLMo-3 7B0.02.20.51.8 32B0.00.10.12.3 Phi-4 3.6B0.00.00.00.0 14B0.00.00.00.0 Qwen-3 0.6B0.00.00.00.0 4B0.00.00.00.0 8B0.00.10.00.0 14B0.00.00.00.0 32B0.00.00.00.0 SmolLM-2/3 0.135B47.846.126.050.8 0.36B54.244.224.848.1 1.7B1.20.10.43.3 3B0.40.70.10.3 Table 14: Parsing failure rates on context extraction benchmarks. 22 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySizeAIME24AIME25ASDivGSM8KMath Apertus 8B50.046.70.90.813.3 70B26.736.71.30.415.5 EuroLLM 1.7B70.076.75.36.354.1 9B63.350.016.916.243.7 22B33.330.02.56.811.7 Falcon-3 1B83.390.069.059.977.7 3B46.740.00.90.516.2 7B13.316.70.00.22.9 Gemma-3 1B93.383.310.318.947.5 4B50.016.70.30.78.9 12B33.333.30.10.29.4 27B33.323.30.20.25.2 LFM-2 0.35B73.366.73.24.556.2 0.7B60.066.71.32.829.3 1.2B33.333.35.112.215.1 2.6B33.343.30.30.413.3 Llama-3 1B100.0100.025.928.184.9 3B53.370.01.81.320.2 8B53.356.73.11.218.5 70B100.0100.06.725.492.1 Ministral-3 3B96.786.70.81.626.6 8B86.790.00.71.419.1 14B93.386.70.41.123.4 OLMo-3 7B60.046.70.31.011.1 32B80.083.30.31.315.8 Phi-4 3.6B40.023.30.10.118.7 14B16.716.70.00.03.2 Qwen-3 0.6B83.373.30.52.034.5 4B56.763.30.20.68.7 8B56.760.00.30.17.9 14B50.046.70.10.06.1 32B50.043.30.10.66.3 SmolLM-2/3 0.135B70.066.736.440.668.2 0.36B70.063.315.822.652.3 1.7B63.380.02.45.838.7 3B50.043.30.91.420.4 Table 15: Parsing failure rates on open-form math benchmarks. 23 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation D.2 Impact of Regex-Based Evaluation on Performance Measurement This section extends Table 1 from the main text by providing a breakdown of how regex- based evaluation affects downstream measured performance across every model and task. FamilySize CoQADROPHotpotQASQuAD-v2 ∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank Apertus 8B-30.2 (-0.0/-30.2)↓3.0-21.8 (-0.0/-21.8)↑1.5-20.6 (-0.1/-20.5)↑2.0-19.2 (-0.0/-19.2)↑10.0 70B-41.4 (-0.0/-41.4)↓7.5-21.9 (-0.1/-21.9)↑3.0-22.7 (-0.0/-22.7)↓3.0-25.3 (-0.0/-25.3)↓1.0 EuroLLM 1.7B-49.6 (-30.6/-19.0) ↓8.0-19.5 (-11.6/-7.9)↓1.0-37.2 (-9.9/-27.3)↓6.0-30.0 (-16.6/-13.4)↓7.0 9B-27.6 (-0.0/-27.6) ↓1.0-18.9 (-0.0/-18.9)↑1.0-15.9 (-0.0/-15.9)↑5.0-22.6 (-0.0/-22.6)↑6.0 22B-24.8 (-0.2/-24.6)↑1.0-25.4 (-0.0/-25.4)0.0-16.3 (-0.2/-16.1)↑1.0-29.2 (-0.0/-29.2)0.0 Falcon-3 1B-42.6 (-16.2/-26.4)↓7.5-11.5 (-1.7/-9.8)0.0-17.4 (-2.2/-15.3)0.0-19.3 (-2.1/-17.2)0.0 3B-34.0 (-2.0/-32.0)↓1.0-18.6 (-0.5/-18.1)0.0-23.4 (-1.0/-22.4)↓1.0-21.7 (-1.7/-20.0)↑2.0 7B-19.8 (-0.4/-19.4) ↑4.0-14.6 (-0.3/-14.3)↑6.0-12.4 (-0.2/-12.2)↑4.0-18.0 (-0.1/-17.9)↑14.0 Gemma-3 1B-19.0 (-0.0/-19.0)↑10.5-6.0 (-0.1/-5.9)↑2.0-9.9 (-0.1/-9.8)↑2.0-14.1 (-0.1/-14.0)↑4.0 4B-43.0 (-0.0/-43.0)↓5.0-27.4 (-0.0/-27.4)↑2.0-23.8 (-0.1/-23.8)0.0-29.0 (-0.0/-28.9)↓3.0 12B-28.8 (-0.2/-28.6) ↓5.0-31.8 (-0.0/-31.8)↓5.0-20.1 (-0.3/-19.9)↓7.0-29.5 (-0.1/-29.4)0.0 27B-28.8 (-0.0/-28.8)↓1.0-31.1 (-0.0/-31.1)↓4.0-25.4 (-0.0/-25.4)↓2.5-29.4 (-0.0/-29.4)↓1.0 LFM-2 0.35B-20.0 (-1.4/-18.6)↑8.5-3.7 (-0.6/-3.1)↑3.0-8.2 (-2.0/-6.1)↑3.0-10.3 (-1.5/-8.8)↑5.0 0.7B-33.6 (-15.8/-17.8)↑1.0-5.5 (-0.4/-5.1)↑2.0-10.3 (-0.3/-10.0)↑2.0-11.4 (-0.6/-10.7)↑3.0 1.2B-14.8 (-0.0/-14.8) ↑14.5-8.9 (-0.1/-8.8)↑2.0-9.9 (-1.2/-8.7)↑7.0-13.2 (-0.0/-13.1)↑13.0 2.6B-29.6 (-1.6/-28.0)↑2.5-28.6 (-1.2/-27.4)↓2.0-22.6 (-1.1/-21.5)0.0-33.5 (-1.2/-32.4)↓9.0 Llama-3 1B-30.0 (-21.0/-9.0) ↑3.0-24.1 (-20.4/-3.7)↓6.0-22.0 (-17.2/-4.8)0.0-19.7 (-13.9/-5.8)↓2.0 3B-40.4 (-29.2/-11.2)↓4.5-17.9 (-3.0/-14.9)↑8.0-22.6 (-13.4/-9.3)0.0-29.1 (-17.0/-12.1)↓5.0 8B-23.6 (-4.6/-19.0)↓1.0-21.5 (-3.7/-17.8)↑3.0-17.1 (-2.0/-15.1)↑3.0-21.9 (-5.9/-16.0)↑9.0 70B-17.2 (-0.0/-17.2)↑7.0-17.0 (-0.0/-17.0)↑3.0-11.5 (-0.0/-11.5)↑10.0-27.8 (-0.3/-27.5)↑6.0 Ministral 3B-36.8 (-0.0/-36.8)↓1.0-32.9 (-0.0/-32.9)↓6.0-29.8 (-0.0/-29.7)↓5.0-30.6 (-0.0/-30.6)↓8.0 8B-30.0 (-0.0/-30.0)↑2.0-35.0 (-0.0/-35.0)↓9.0-24.9 (-0.0/-24.9)↓6.0-34.0 (-0.0/-34.0)↓6.0 14B-27.4 (-0.0/-27.4) ↓2.0-30.7 (-0.0/-30.7)↓5.0-23.4 (-0.0/-23.4)↓9.0-36.3 (-0.0/-36.3)↓7.0 OLMo-3 7B-17.6 (-0.0/-17.6) ↑5.0-22.5 (-0.8/-21.7)↑4.0-11.7 (-0.0/-11.6)↑9.5-26.4 (-0.2/-26.2)↑7.0 32B-24.8 (-0.0/-24.8)↓3.5-27.3 (-0.0/-27.2)↓2.0-16.1 (-0.0/-16.0)0.0-35.6 (-2.2/-33.4)↓4.0 Phi-4 3.6B-42.0 (-0.0/-42.0)↓8.5-23.2 (-0.0/-23.2)↑1.0-29.4 (-0.0/-29.4)↓3.0-35.1 (-0.0/-35.1)↓12.0 14B-36.0 (-0.0/-36.0)↓0.5-32.0 (-0.0/-32.0)↓4.0-27.8 (-0.0/-27.8)↓8.0-45.5 (-0.0/-45.5)↓18.0 Qwen-3 0.6B-27.2 (-0.0/-27.2)↑4.0-9.4 (-0.0/-9.4)↑1.0-7.3 (-0.0/-7.3)↑3.0-11.1 (-0.0/-11.1)↑3.0 4B-23.0 (-0.0/-23.0)↑6.0-20.0 (-0.0/-20.0)↑6.0-12.9 (-0.0/-12.9)↑7.0-25.0 (-0.0/-25.0)↑9.0 8B-30.6 (-0.0/-30.6)↓8.0-34.5 (-0.0/-34.4)↓9.0-19.7 (-0.0/-19.7)↓4.0-34.2 (-0.0/-34.2)↓8.0 14B-19.8 (-0.0/-19.8)0.0-19.9 (-0.0/-19.9) ↓1.0-12.8 (-0.0/-12.8)0.0-31.1 (-0.0/-31.1)↑3.0 32B-23.0 (-0.0/-23.0)↓2.5-25.3 (-0.0/-25.3)↑2.0-17.1 (-0.0/-17.1)↓2.0-35.1 (-0.0/-35.1)↓2.0 SmolLM-2/3 0.135B-38.0 (-24.2/-13.8)0.0-11.5 (-6.8/-4.7) ↑1.0-31.7 (-10.1/-21.6)↓1.0-23.9 (-12.5/-11.4)0.0 0.36B-55.0 (-35.2/-19.8)↓6.0-17.6 (-11.4/-6.1)↓2.0-36.5 (-12.7/-23.8)↓4.0-27.9 (-16.0/-11.9)↓3.0 1.7B-25.2 (-1.0/-24.2)↑6.5-8.4 (-0.1/-8.3)0.0-14.7 (-0.3/-14.4)↑3.0-18.3 (-1.5/-16.8)0.0 3B-35.2 (-0.4/-34.8)↑1.0-20.5 (-0.1/-20.3)↑4.5-22.3 (-0.0/-22.2)0.0-25.4 (-0.0/-25.3)↑2.0 Table 16: Effect of regex-based evaluation on performance measurement for context extrac- tion benchmarks. 24 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySize ARC-ChallengeARC-EasyGPQAMMLUMMLU-ProTruthfulQA ∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank Apertus 8B0.0 (-0.1/+0.1)↑3.00.0 (-0.0/0.0)↑6.0+5.8 (-0.0/+5.8)↑16.0-0.9 (-0.1/-0.8)↑4.0+0.9 (-0.5/+1.4)↑4.0-0.5 (-0.0/-0.5)0.0 70B-1.8 (-0.1/-1.7)↑3.0-1.5 (-0.0/-1.5)↑2.0-1.3 (-1.3/0.0)↑8.5-3.5 (-0.4/-3.1)↑3.0-2.9 (-1.2/-1.7)↑1.0-3.3 (-0.5/-2.8)↓2.0 EuroLLM 1.7B-29.8 (-28.2/-1.5)↓1.0-29.2 (-28.4/-0.8)↓1.0-28.3 (-19.2/-9.2)↓17.5-27.4 (-24.6/-2.9)↓1.0-22.4 (-9.3/-13.1)↓7.0-29.0 (-16.3/-12.7)↓5.0 9B-0.3 (-0.0/-0.3)↑4.0-0.0 (-0.0/-0.0)↑6.0-8.9 (-2.7/-6.3)↑0.5-1.6 (-0.5/-1.2)↑2.0-8.2 (-2.7/-5.5)0.0+0.2 (-0.0/+0.2)↑3.0 22B-8.6 (-0.1/-8.5)↓6.0-7.7 (-0.0/-7.7)↓8.0-9.4 (-1.1/-8.3)↓2.5-12.7 (-0.1/-12.6)↓8.0-10.0 (-0.7/-9.3)↓2.0-6.2 (-0.0/-6.2)↓1.0 Falcon-3 1B-26.7 (-2.6/-24.1)0.0-33.1 (-3.0/-30.1)0.0-14.7 (-2.2/-12.5)↓10.5-15.4 (-1.7/-13.7)↑2.0-13.8 (-6.3/-7.5)0.0-6.6 (-0.6/-6.0)↑3.0 3B-3.2 (-0.0/-3.2)↑2.0-2.2 (-0.1/-2.1)↑2.0-4.5 (-0.2/-4.2)↑7.0-3.9 (-0.1/-3.9)↑2.0-7.5 (-0.4/-7.1)↓1.0-1.1 (-0.1/-1.0)↑2.0 7B-5.2 (-0.0/-5.2)↓2.0-6.6 (-0.0/-6.6)↓4.0-6.0 (-0.7/-5.4)↑2.0-6.7 (-0.0/-6.7)↓1.0-7.1 (-0.4/-6.7)↑1.0-2.0 (-0.0/-2.0)↑1.0 Gemma-3 1B+0.1 (-0.1/+0.2)↑5.0-0.6 (-0.0/-0.6)↑5.0+2.5 (-1.6/+4.0)↑11.0-1.0 (-0.6/-0.4)↑4.0-2.2 (-1.8/-0.5)↑5.0+0.5 (-0.0/+0.5)↑4.0 4B+0.2 (-0.0/+0.2)↑3.00.0 (-0.0/0.0)↑6.0+1.1 (-0.4/+1.6)↑12.0+0.9 (-0.1/+0.9)↑7.0-0.4 (-0.5/+0.1)↑6.0+0.2 (-0.0/+0.2)↑3.0 12B+0.3 (-0.0/+0.3)↑6.00.0 (-0.0/0.0)↑4.0-0.4 (-0.4/0.0)↑7.00.0 (-0.0/0.0)↑6.0-1.1 (-0.5/-0.6)↑5.0+0.6 (-0.0/+0.6)↑4.0 27B+0.3 (-0.0/+0.3)↑5.0+0.2 (-0.0/+0.2)↑4.0+0.2 (-0.2/+0.4)↑6.0+0.3 (-0.0/+0.3)↑4.0-0.9 (-0.4/-0.5)↑5.0+0.2 (-0.0/+0.2)↑4.0 LFM-2 0.35B-36.1 (-0.2/-35.9)↑1.0-50.0 (-0.4/-49.6)↑1.0-9.8 (-0.4/-9.4)↑3.5-29.9 (-0.5/-29.4)↑1.0-15.0 (-1.8/-13.2)0.0-20.3 (-2.0/-18.4)↓1.0 0.7B-16.6 (-1.5/-15.1)↓2.0-21.6 (-2.4/-19.2)↓1.0-11.8 (-3.6/-8.3)↑1.0-23.5 (-2.9/-20.6)↓3.0-14.4 (-3.1/-11.3)↓1.0-6.6 (-1.1/-5.5)↑2.0 1.2B-5.7 (-0.3/-5.5)↑2.0-4.6 (-0.4/-4.2)↑2.0-4.9 (-0.7/-4.2)↑3.0-10.1 (-0.4/-9.8)↓1.0-13.5 (-0.9/-12.6)↓2.0-9.9 (-0.0/-9.9)↓2.0 2.6B-0.1 (-0.0/-0.1)↑5.5-0.2 (-0.0/-0.2)↑6.5-5.4 (-1.1/-4.2)↑3.5-1.3 (-0.3/-1.0)↑6.0-4.5 (-1.7/-2.8)↑3.0-2.1 (-0.1/-2.0)↑1.0 Llama-3 1B-1.9 (-2.4/+0.5)↑5.0-1.9 (-2.1/+0.2)↑5.0+2.0 (-3.8/+5.8)↑9.5-1.2 (-2.5/+1.4)↑5.0-1.7 (-3.2/+1.5)↑4.0-1.0 (-1.6/+0.6)0.0 3B-14.8 (-6.5/-8.3)↓1.0-13.0 (-4.3/-8.8)↓1.0-2.2 (-5.8/+3.6)↑8.0-17.9 (-11.7/-6.2)↓4.0-14.7 (-14.0/-0.8)↓2.0-15.9 (-8.7/-7.2)↓3.0 8B-38.1 (-0.0/-38.1)↓11.0-32.5 (-0.2/-32.4)↓10.5-5.6 (-2.0/-3.6)↑4.0-26.4 (-0.5/-25.9)↓9.0-14.3 (-2.0/-12.3)↓4.0-22.8 (-1.0/-21.8)↓8.0 70B+0.2 (-0.0/+0.2)↑3.0+0.1 (-0.0/+0.1)0.0-3.3 (-2.7/-0.7)0.0-0.6 (-0.7/+0.1)0.0-3.5 (-2.9/-0.6)↑3.0+0.4 (-0.0/+0.4)↑2.0 Ministral 3B-2.0 (-0.0/-2.0)↑3.0-0.7 (-0.0/-0.7)↑4.0-11.4 (-6.7/-4.7)↑0.5-2.1 (-0.3/-1.7)↑2.0-8.0 (-5.1/-2.9)↑3.0-1.5 (-0.0/-1.5)↑0.5 8B-4.4 (-0.0/-4.4)0.0-2.6 (-0.0/-2.6)↑2.0-8.7 (-5.6/-3.1)↑2.0-8.3 (-0.6/-7.7)↑1.0-12.3 (-3.1/-9.2)↑1.0-9.1 (-0.0/-9.1)↓5.0 14B-1.8 (-0.0/-1.8)0.0-0.9 (-0.1/-0.8)0.0-10.3 (-3.8/-6.5)↓1.0-5.7 (-0.5/-5.1)0.0-11.3 (-3.0/-8.3)↑1.0-8.1 (-0.0/-8.1)↓2.0 OLMo-3 7B-0.3 (-0.2/-0.1)↑5.0-0.0 (-0.0/-0.0)↑7.0-8.0 (-6.7/-1.3)↑1.5-0.9 (-0.5/-0.4)↑6.0-8.4 (-5.8/-2.6)↑1.0+0.4 (-0.0/+0.4)↑3.0 32B+0.2 (-0.1/+0.3)↑3.00.0 (-0.0/0.0)↑4.0-25.9 (-22.3/-3.6)↓11.0-1.6 (-0.6/-1.1)↑4.0-13.0 (-9.5/-3.5)↓2.00.0 (-0.0/0.0)0.0 Phi-4 3.6B-0.4 (-0.0/-0.4)↑5.0-0.7 (-0.0/-0.7)↑4.0+3.1 (-0.2/+3.3)↑14.0-3.0 (-0.1/-2.9)↑4.0-1.9 (-0.4/-1.6)↑4.0-0.1 (-0.0/-0.1)↑1.5 14B-6.7 (-0.0/-6.7)↓7.5-8.0 (-0.0/-8.0)↓13.0-10.5 (-0.2/-10.3)↓1.0-12.3 (-0.0/-12.3)↓4.0-6.5 (-0.1/-6.4)↓1.0-10.0 (-0.0/-10.0)↓5.0 Qwen-3 0.6B-31.2 (-0.0/-31.2)↓2.0-43.3 (-0.0/-43.3)↓2.0-14.7 (-0.0/-14.7)↓11.0-19.3 (-0.0/-19.3)0.0-8.0 (-0.0/-8.0)0.0-8.4 (-0.0/-8.4)↓2.0 4B-0.2 (-0.0/-0.2)↑4.0+0.1 (-0.0/+0.1)↑5.0-7.1 (-2.0/-5.1)↑4.0-3.1 (-0.1/-3.0)↑3.0-8.5 (-0.9/-7.6)↑2.0-0.1 (-0.0/-0.1)↑3.0 8B-2.2 (-0.0/-2.2)↑3.0-0.9 (-0.0/-0.9)↑2.0-11.6 (-2.0/-9.6)↓2.0-7.8 (-0.3/-7.5)0.0-15.1 (-1.5/-13.7)↓1.0-4.0 (-0.0/-4.0)↑2.0 14B-16.0 (-0.0/-16.0)↓18.0-11.2 (-0.0/-11.2)↓18.0-26.1 (-2.9/-23.2)↓15.0-21.6 (-0.0/-21.5)↓14.0-30.4 (-1.4/-29.0)↓10.0-10.3 (-0.0/-10.3)↓4.0 32B-19.5 (-0.0/-19.5)↓20.5-13.0 (-0.0/-13.0)↓20.0-38.6 (-1.3/-37.3)↓26.0-27.9 (-0.0/-27.9)↓19.0-37.7 (-1.3/-36.4)↓19.0-5.8 (-0.0/-5.8)↓1.0 SmolLM-2/3 0.135B-23.2 (-22.9/-0.3)↑1.0-24.0 (-23.7/-0.3)↑2.5-20.8 (-20.5/-0.2)↓6.0-22.7 (-22.0/-0.7)0.0-11.9 (-11.6/-0.3)↓1.0-18.2 (-17.9/-0.4)↑1.0 0.36B-23.9 (-1.7/-22.2) ↑2.0-26.5 (-1.3/-25.2)↑1.5-21.2 (-10.0/-11.2)↓6.0-24.1 (-2.8/-21.4)↑2.0-9.2 (-4.0/-5.2)↑3.0-22.3 (-0.9/-21.4)0.0 1.7B-57.2 (-0.0/-57.2) ↓7.0-79.2 (-0.0/-79.2)↓9.0-27.7 (-0.2/-27.5)↓15.5-44.9 (-0.1/-44.8)↓7.0-15.9 (-0.1/-15.8)↓1.0-18.4 (-0.1/-18.2)0.0 3B-0.3 (-0.1/-0.3)↑4.5-0.4 (-0.1/-0.3)↑6.0-6.9 (-3.3/-3.6)↑0.5-3.0 (-1.7/-1.3)↑3.0-4.9 (-3.1/-1.8)↑2.0-1.7 (-0.4/-1.3)↑1.0 Table 17: Effect of regex-based evaluation on performance measurement for multiple-choice benchmarks. FamilySize AIME24AIME25ASDivGSM8KMath ∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank∆Accuracy∆Rank Apertus 8B0.0 (-0.0/0.0) ↑4.00.0 (-0.0/0.0)↑5.0-5.1 (-0.1/-5.0)↑3.0-0.4 (-0.2/-0.2)↑3.0-10.2 (-1.2/-9.0)↑6.0 70B0.0 (-0.0/0.0) ↑4.00.0 (-0.0/0.0)↑5.0-5.8 (-0.7/-5.1)↑1.0-0.3 (-0.0/-0.3)↑2.0-13.3 (-1.3/-12.0)↑7.0 EuroLLM 1.7B0.0 (-0.0/0.0)↑4.00.0 (-0.0/0.0)↑5.0-1.7 (-0.9/-0.8)↑1.0-0.8 (-0.2/-0.7)↑1.0-2.7 (-2.0/-0.7)0.0 9B-10.0 (-6.7/-3.3)↓7.5-3.3 (-3.3/0.0)↓3.0-29.5 (-11.9/-17.6)↓4.0-27.5 (-10.7/-16.8)↓6.0-30.2 (-15.5/-14.7)↓1.0 22B-20.0 (-3.3/-16.7)↓3.0-6.7 (-0.0/-6.7)↓7.5-32.3 (-1.9/-30.4)↓8.0-29.7 (-5.4/-24.3)↓8.0-38.6 (-4.2/-34.4)↓7.5 Falcon-3 1B-3.3 (-3.3/0.0) ↓1.0-3.3 (-0.0/-3.3)↓3.0-51.1 (-45.0/-6.1)↓4.0-36.7 (-31.1/-5.6)↓4.0-22.6 (-18.8/-3.8)↓1.0 3B-6.7 (-0.0/-6.7) ↓4.0-6.7 (-0.0/-6.7)↓7.5-6.7 (-0.6/-6.1)0.0-1.9 (-0.2/-1.7)↑1.0-24.7 (-4.1/-20.5)↑1.0 7B-6.7 (-0.0/-6.7)↑4.5-6.7 (-0.0/-6.7)↑3.5-5.8 (-0.0/-5.8)↑5.0-0.2 (-0.1/-0.1)↑2.0-24.9 (-0.8/-24.1)↑3.0 Gemma-3 1B-6.7 (-6.7/0.0)↓4.0-3.3 (-3.3/0.0)↓3.0-10.8 (-5.2/-5.7)↓1.0-6.7 (-5.5/-1.3)↑2.0-28.7 (-13.3/-15.5)0.0 4B-6.7 (-0.0/-6.7) ↑3.5-16.7 (-3.3/-13.3)↓3.5-7.2 (-0.0/-7.1)↓4.5-1.3 (-0.3/-1.0)↑1.0-27.3 (-2.5/-24.8)↑3.0 12B-6.7 (-3.3/-3.3)↑9.0-6.7 (-0.0/-6.7)↑5.5-7.2 (-0.0/-7.2)↓6.5-1.1 (-0.1/-1.0)↑2.0-29.9 (-3.6/-26.3)↓2.0 27B-10.0 (-3.3/-6.7)↑6.5-6.7 (-0.0/-6.7)↑4.5-7.5 (-0.1/-7.4)↓10.0-0.8 (-0.2/-0.6)0.0-29.0 (-2.0/-27.0)↑1.0 LFM-2 0.35B0.0 (-0.0/0.0)↑4.00.0 (-0.0/0.0)↑5.0-19.5 (-1.7/-17.7)0.0-22.8 (-1.2/-21.6)↓1.0-24.0 (-14.0/-10.0)↓1.0 0.7B0.0 (-0.0/0.0) ↑4.00.0 (-0.0/0.0)↑5.0-5.3 (-0.8/-4.5)↑3.0-1.4 (-1.1/-0.3)↑1.0-16.8 (-7.5/-9.3)↑3.0 1.2B-10.0 (-3.3/-6.7)↓7.5-3.3 (-0.0/-3.3)↓3.0-8.5 (-3.6/-4.9)↑1.0-7.1 (-6.7/-0.3)↑1.0-20.6 (-5.0/-15.6)↑3.0 2.6B-13.3 (-3.3/-10.0)↑5.0-13.3 (-10.0/-3.3)↓3.0-4.7 (-0.0/-4.6)↑12.0-0.6 (-0.1/-0.5)↑1.0-30.6 (-6.9/-23.7)↑2.0 Llama-3 1B-3.3 (-3.3/0.0)↓1.0-3.3 (-3.3/0.0)↓3.0-15.8 (-14.5/-1.3)0.0-8.7 (-8.7/-0.0)↑2.0-28.1 (-25.5/-2.7)↓2.0 3B-6.7 (-6.7/0.0)↑4.5-3.3 (-0.0/-3.3)↓3.0-4.7 (-1.2/-3.5)↑1.0-0.5 (-0.4/-0.1)↑3.0-19.6 (-4.5/-15.1)↑3.0 8B0.0 (-0.0/0.0) ↑9.50.0 (-0.0/0.0)↑5.0-6.7 (-1.6/-5.1)↑2.0-0.5 (-0.2/-0.2)↑3.0-19.8 (-2.4/-17.5)↑1.5 70B-33.3 (-33.3/0.0) ↓17.0-10.0 (-10.0/0.0)↓9.5-11.4 (-5.9/-5.5)↓6.0-24.1 (-24.2/+0.1)↓18.0-73.1 (-71.5/-1.6)↓16.0 Ministral 3B-36.7 (-36.7/-0.0)↓11.5-20.0 (-20.0/0.0)↓5.0-6.3 (-0.4/-5.9)↑1.0-0.6 (-0.3/-0.3)↑2.0-34.9 (-14.4/-20.5)↓3.0 8B-40.0 (-40.0/0.0)↓9.0-36.7 (-36.7/-0.0)↓9.5-6.1 (-0.4/-5.7)↑5.0-1.9 (-0.9/-1.0)0.0-33.7 (-10.7/-23.0)↓3.0 14B-50.0 (-50.0/0.0)↓16.5-33.3 (-33.3/-0.0)↓8.5-5.9 (-0.2/-5.7)↑3.0-1.1 (-0.7/-0.5)↑0.5-37.9 (-15.9/-22.0)↓6.5 OLMo-3 7B-30.0 (-20.0/-10.0) ↓5.5-13.3 (-13.3/0.0)↑3.0-6.4 (-0.1/-6.3)↑0.5-0.7 (-0.2/-0.5)↑1.0-30.3 (-5.5/-24.8)↓0.5 32B-36.7 (-36.7/-0.0) ↓6.0-23.3 (-23.3/-0.0)↓2.0-5.9 (-0.0/-5.9)↑2.0-0.8 (-0.5/-0.3)↓1.0-31.7 (-9.5/-22.1)↓3.0 Phi-4 3.6B-6.7 (-6.7/0.0)↓4.0-3.3 (-3.3/-0.0)↑2.5-5.6 (-0.0/-5.6)↑6.0-1.5 (-0.0/-1.5)0.0-28.7 (-9.8/-18.8)↑1.0 14B-16.7 (-10.0/-6.7)↑4.5-6.7 (-6.7/0.0)↑3.5-6.7 (-0.0/-6.7)↓3.5-1.3 (-0.0/-1.3)↑1.0-28.3 (-1.5/-26.8)↑2.0 Qwen-3 0.6B-10.0 (-10.0/0.0)↑0.5-3.3 (-3.3/0.0)↓3.0-4.3 (-0.1/-4.2)↑3.0-3.4 (-0.5/-3.0)↑1.0-33.0 (-15.4/-17.7)↓4.0 4B-13.3 (-13.3/-0.0)↑3.5-10.0 (-10.0/0.0)↑4.5-7.3 (-0.1/-7.2)↓5.0-1.1 (-0.4/-0.7)0.0-28.4 (-2.8/-25.6)↑2.0 8B-16.7 (-16.7/0.0)↑3.5-23.3 (-20.0/-3.3)↓5.0-7.0 (-0.2/-6.8)↓3.0-1.4 (-0.0/-1.4)↑1.5-27.7 (-2.7/-25.0)↑2.0 14B-20.0 (-16.7/-3.3)↓1.0-20.0 (-20.0/0.0)↓1.0-6.5 (-0.0/-6.5)↑1.0-0.2 (-0.0/-0.2)↑3.0-28.3 (-2.2/-26.1)↑2.0 32B-10.0 (-6.7/-3.3)↑5.5-6.7 (-3.3/-3.3)↑5.5-5.8 (-0.0/-5.8)0.0-0.8 (-0.4/-0.5)↓1.0-28.3 (-2.5/-25.8)↑4.0 SmolLM-2/3 0.135B0.0 (-0.0/0.0)↑4.00.0 (-0.0/0.0)↑5.0-3.1 (-3.0/-0.1)0.0-0.1 (-0.6/+0.5)0.0-1.7 (-1.7/-0.0)0.0 0.36B0.0 (-0.0/0.0)↑4.00.0 (-0.0/0.0)↑5.0-3.3 (-2.4/-1.0)↑1.0-1.7 (-1.1/-0.6)0.0-4.0 (-3.2/-0.8)↑1.0 1.7B0.0 (-0.0/0.0)↑10.00.0 (-0.0/0.0)↑5.0-4.7 (-1.8/-2.9)↑2.0-2.4 (-1.7/-0.6)↑2.0-14.2 (-7.9/-6.3)↑3.0 3B-10.0 (-3.3/-6.7)↑0.5-3.3 (-3.3/0.0)↑5.5-5.7 (-0.3/-5.4)↑2.0-0.4 (-0.2/-0.2)↑2.0-33.2 (-12.6/-20.6)0.0 Table 18: Effect of regex-based evaluation on performance measurement for open-form math benchmarks. 25 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation D.3 BERT-as-a-Judge vs. Regex In this section, we provide a detailed, unaggregated comparison of regex-based evaluation and BERT-as-a-Judge, reported by model and task. The results are based on the default BERT-as-a-Judge configuration, trained on 1M question-candidate-reference triplets, with candidates generated under the soft-constraint instruction (Appendix A). FamilySize ARCARC GPQAMMLU MMLUTruthful ChallengeEasyProQA Apertus 8B99.499.692.498.296.799.1 70B99.599.795.198.496.699.0 EuroLLM 1.7B99.599.792.497.088.583.0 9B99.9100.094.999.397.599.8 22B99.799.994.498.497.299.9 Falcon-3 1B98.498.992.497.995.097.6 3B99.799.794.699.297.199.8 7B99.999.996.299.597.799.6 Gemma-3 1B99.499.890.697.596.599.5 4B99.899.996.098.397.099.8 12B99.699.994.699.197.699.4 27B99.799.895.598.998.099.8 LFM-2 0.35B98.898.689.395.293.293.5 0.7B99.199.291.596.896.198.2 1.2B98.799.494.497.596.698.2 2.6B99.599.794.998.596.199.0 Llama-3 1B99.199.591.597.796.599.1 3B99.599.793.598.096.599.1 8B99.999.894.098.997.299.0 70B99.799.894.499.097.698.4 Ministral-3 3B99.599.890.498.794.099.3 8B99.799.893.599.196.399.3 14B99.799.994.299.297.299.0 OLMo-3 7B99.499.892.698.794.199.6 32B99.599.977.799.291.499.8 Phi-4 3.6B99.799.994.299.497.899.6 14B99.9100.096.099.298.099.0 Qwen-3 0.6B99.899.995.899.197.4100.0 4B99.499.795.598.897.599.6 8B99.099.594.298.897.299.4 14B99.799.994.099.297.899.6 32B99.499.796.799.197.999.5 SmolLM-2/3 0.135B97.698.988.696.797.096.7 0.36B99.599.497.398.798.198.7 1.7B99.999.997.599.598.799.9 3B99.599.794.498.697.098.8 Table 19: BERT-as-a-Judge assessment accuracy on multiple-choice benchmarks. 26 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySize ARCARC GPQAMMLU MMLUTruthful ChallengeEasyProQA Apertus 8B99.099.291.596.496.297.8 70B97.797.993.894.994.596.2 EuroLLM 1.7B70.270.871.772.677.671.0 9B99.599.989.798.091.399.8 22B91.292.185.385.887.393.5 Falcon-3 1B73.366.884.884.385.893.1 3B96.697.793.395.591.198.7 7B94.693.390.492.891.897.8 Gemma-3 1B98.798.986.495.394.199.5 4B99.899.995.898.296.999.8 12B99.699.994.299.197.699.4 27B99.799.895.898.998.199.8 LFM-2 0.35B63.949.887.969.684.779.4 0.7B82.578.086.475.484.892.4 1.2B92.194.289.787.585.187.6 2.6B99.299.491.197.594.596.9 Llama-3 1B96.997.690.495.994.797.8 3B84.286.489.779.983.483.8 8B61.867.390.472.784.376.7 70B99.799.892.698.095.697.4 Ministral-3 3B98.099.188.697.391.897.3 8B95.597.389.591.587.590.5 14B98.298.989.394.288.690.5 OLMo-3 7B99.299.891.198.291.499.6 32B99.599.974.198.086.999.5 Phi-4 3.6B99.299.194.296.295.999.1 14B93.392.086.487.092.488.5 Qwen-3 0.6B68.856.783.980.089.791.6 4B98.699.590.695.290.399.6 8B96.198.185.791.284.194.7 14B83.888.772.578.169.389.2 32B80.086.560.971.762.093.5 SmolLM-2/3 0.135B76.876.079.277.388.181.8 0.36B76.173.578.375.890.677.7 1.7B42.820.871.455.184.181.6 3B99.099.288.695.693.796.3 Table 20: Regex assessment accuracy on multiple-choice benchmarks. 27 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySizeCoQADROPHotpotQASQuAD-v2 Apertus 8B90.289.191.989.9 70B87.489.891.889.0 EuroLLM 1.7B91.491.392.191.2 9B90.089.693.589.6 22B90.489.293.389.4 Falcon-3 1B90.490.290.491.5 3B88.689.389.089.0 7B92.892.293.889.1 Gemma-3 1B84.890.191.091.1 4B75.483.588.987.8 12B90.685.291.885.2 27B87.285.488.783.9 LFM-2 0.35B88.890.490.792.1 0.7B83.489.085.292.1 1.2B91.490.591.591.5 2.6B85.685.288.888.9 Llama-3 1B91.689.494.592.7 3B88.488.891.189.7 8B88.487.892.888.3 70B89.889.090.587.3 Ministral-3 3B75.685.086.487.7 8B87.287.189.987.5 14B88.689.391.087.7 OLMo-3 7B92.285.092.188.2 32B92.081.791.587.3 Phi-4 3.6B90.090.590.889.8 14B82.289.289.488.2 Qwen-3 0.6B86.090.592.991.5 4B85.488.891.490.2 8B91.684.690.589.0 14B93.090.693.587.3 32B91.289.292.288.4 SmolLM-2/3 0.135B86.892.088.391.1 0.36B85.289.790.191.2 1.7B91.893.593.892.0 3B84.888.788.189.9 Table 21: BERT-as-a-Judge assessment accuracy on context extraction benchmarks. 28 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySizeCoQADROPHotpotQASQuAD-v2 Apertus 8B67.075.773.778.3 70B56.676.372.572.8 EuroLLM 1.7B49.280.060.569.0 9B70.879.980.576.1 22B72.473.080.269.4 Falcon-3 1B55.485.177.578.8 3B61.679.271.376.1 7B77.483.983.880.1 Gemma-3 1B75.088.883.482.9 4B54.670.371.269.5 12B68.866.976.069.0 27B68.467.469.668.3 LFM-2 0.35B76.090.284.086.0 0.7B62.086.678.184.2 1.2B80.486.382.984.3 2.6B66.469.772.264.7 Llama-3 1B66.475.174.879.0 3B56.879.472.869.6 8B74.077.179.976.5 70B78.481.182.370.2 Ministral-3 3B62.865.566.667.5 8B67.263.871.064.5 14B70.668.273.062.3 OLMo-3 7B78.474.883.371.8 32B72.070.378.962.5 Phi-4 3.6B56.875.066.563.4 14B63.266.968.753.0 Qwen-3 0.6B70.488.986.686.7 4B71.476.881.073.2 8B67.862.575.964.2 14B77.879.283.367.5 32B75.473.079.163.2 SmolLM-2/3 0.135B60.887.467.075.6 0.36B45.081.861.471.3 1.7B72.490.581.580.1 3B62.876.971.573.0 Table 22: Regex assessment accuracy on context extraction benchmarks. 29 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySizeAIME24AIME25ASDivGSM8KMath Apertus 8B100.0100.095.498.693.3 70B100.0100.094.599.192.6 EuroLLM 1.7B100.0100.097.897.697.0 9B90.096.795.498.691.8 22B80.093.395.198.990.2 Falcon-3 1B96.796.795.697.990.6 3B96.793.395.499.493.3 7B93.393.395.299.893.9 Gemma-3 1B93.3100.093.596.590.4 4B90.083.395.298.693.6 12B96.793.394.898.994.9 27B90.093.395.199.295.4 LFM-2 0.35B100.0100.096.297.292.3 0.7B100.0100.094.898.993.1 1.2B90.096.794.698.992.9 2.6B90.090.095.299.294.9 Llama-3 1B96.796.793.797.692.9 3B96.796.794.999.293.0 8B100.0100.094.099.092.5 70B93.393.395.399.694.5 Ministral-3 3B66.756.794.499.290.3 8B66.763.394.999.892.8 14B70.060.095.399.692.5 OLMo-3 7B80.080.095.598.994.7 32B60.066.795.699.393.6 Phi-4 3.6B96.7100.095.499.594.6 14B83.3100.095.299.795.9 Qwen-3 0.6B86.7100.095.797.092.4 4B86.793.394.999.295.3 8B83.383.395.099.495.9 14B80.080.095.499.795.4 32B93.393.395.399.695.8 SmolLM-2/3 0.135B100.0100.098.398.697.7 0.36B100.0100.098.098.995.7 1.7B100.0100.095.898.093.1 3B93.396.795.299.094.6 Table 23: BERT-as-a-Judge assessment accuracy on open-form math benchmarks. 30 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation FamilySizeAIME24AIME25ASDivGSM8KMath Apertus 8B100.0100.094.099.288.4 70B100.0100.092.999.785.0 EuroLLM 1.7B100.0100.096.497.596.5 9B90.096.770.272.569.4 22B80.093.366.970.361.1 Falcon-3 1B96.796.748.663.377.0 3B93.393.392.597.874.7 7B93.393.392.599.874.5 Gemma-3 1B93.396.788.793.171.1 4B93.383.391.598.672.3 12B93.393.391.998.969.8 27B90.093.391.599.270.6 LFM-2 0.35B100.0100.079.876.675.9 0.7B100.0100.093.197.981.8 1.2B90.096.790.592.678.1 2.6B86.786.792.899.469.2 Llama-3 1B96.796.783.790.871.8 3B93.396.793.899.479.9 8B100.0100.091.999.479.9 70B66.790.087.375.726.9 Ministral-3 3B63.380.091.499.464.9 8B60.063.391.798.166.0 14B50.066.791.898.962.0 OLMo-3 7B70.086.792.299.369.3 32B63.376.792.499.268.3 Phi-4 3.6B93.396.792.298.270.7 14B83.393.392.198.771.3 Qwen-3 0.6B90.096.793.996.466.8 4B86.790.091.398.971.3 8B83.376.791.798.571.8 14B80.080.091.999.671.3 32B90.093.391.899.271.3 SmolLM-2/3 0.135B100.0100.095.998.797.5 0.36B100.0100.095.598.094.6 1.7B100.0100.094.096.985.1 3B90.096.792.699.666.3 Table 24: Regex assessment accuracy on open-form math benchmarks. 31 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation E Illustrative Examples In this section, we present examples of common failure cases in regex-based evaluation. Table 25 illustrates a case where parsing fails despite the model producing a correct answer, while Table 26 shows a case where parsing succeeds but additional formatting introduced by the model prevents correct assessment against the reference. Question Answer the question based on the provided context. Context: A psychological identity relates to self-image (one's mental model of oneself), self-esteem, and individuality. Consequently, Weinreich gives the definition "A person's identity is defined as the totality of one's self-construal, in which how one construes oneself in the present expresses the continuity between how one construes oneself as one was in the past and how one construes oneself as one aspires to be in the future"; this allows for definitions of aspects of identity, such as: "One's ethnic identity is defined as that part of the totality of one's self-construal made up of those dimensions that express the continuity between one's construal of past ancestry and one's future aspirations in relation to ethnicity" (Weinreich, 1986a). The description or representation of individual and group identity is a central task for psychologists, sociologists and anthropologists and those of other disciplines where "identity" needs to be mapped and defined. How should one describe the identity of another, in ways which encompass both their idiosyncratic qualities and their group memberships or identifications, both of which can shift according to circumstance? Following on from the work of Kelly, Erikson, Tajfel and others Weinreich's Identity Structure Analysis (ISA), is "a structural representation of the individual's existential experience, in which the relationships between self and other agents are organised in relatively stable structures over time . . . with the emphasis on the socio-cultural milieu in which self relates to other agents and institutions" (Weinreich and Saunderson, (eds) 2003, p1). Using constructs drawn from the salient discourses of the individual, the group and cultural norms, the practical operationalisation of ISA provides a methodology that maps how these are used by the individual, applied across time and milieus by the "situated self" to appraise self and other agents and institutions (for example, resulting in the individual's evaluation of self and significant others and institutions).[citation needed] Question: What is the ISA? Conclude your response with "Final answer: X", where X is the exact span from the context that answers the question. Reference Identity Structure Analysis (ISA) Candidate The ISA, or Identity Structure Analysis, is a methodology developed by Weinreich that maps how individuals perceive themselves and others in relation to their social environment, including their own group memberships and identifications, as well as the cultural norms and discourses that shape these perceptions. Table 25: Example 488 from the CoQA dataset, along with the answer from Llama-3 1B 32 BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation Question We have a triangle $ ABC$ such that $AB = 6,$ $BC = 8,$ and $CA = 10.$ If $AD$ is an angle bisector such that $D$ is on $BC,$ then find the value of $ADˆ2.$ Conclude your response with "Final answer: X", where X is the computed solution. Reference 45 Candidate To solve for $ ADˆ2 $ in $ ABC$ where $ AB = 6 $, $ BC = 8 $, and $ CA = 10 $, and $ AD $ is the angle bisector of $ BAC$ with $ D $ on $ BC $, we can use the Angle Bisector Theorem and the Angle Bisector Length Formula. ### Step-by-Step Solution: 1. **Angle Bisector Theorem:** The Angle Bisector Theorem states that the angle bisector divides the opposite side into segments proportional to the adjacent sides. Therefore, if $ D $ divides $ BC $ into segments $ BD $ and $ DC $, then: \[ BDDC = ABAC = 610 = 35 \] [...] 4. **Calculate $ ADˆ2 $:** \[ ADˆ2 = 6 · 10 ( 1 - 8ˆ2(6 + 10)ˆ2 ) \] \[ ADˆ2 = 60 ( 1 - 6416ˆ2 ) \] \[ ADˆ2 = 60 ( 1 - 64256 ) \] \[ ADˆ2 = 60 ( 1 - 14 ) \] \[ ADˆ2 = 60 ( 34 ) \] \[ ADˆ2 = 60 · 0.75 \] \[ ADˆ2 = 45 \] Final answer: $ 45 $ Table 26: Example 2070 from the MATH dataset, along with the answer from Falcon-3 7B 33

Instant research discovery