Paper deep dive
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks
Gowthamkumar Nandakishore
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/7/2026, 8:49:33 AM
Summary
This study evaluates the open contrastive decision model Contrastive-LM/CLM-v0.1-8B as a judge on public benchmarks. The model performs near chance (0.351–0.593 accuracy), significantly trailing generative judges (Qwen3-8B) and reward models (Skywork-Reward-V2-8B). While CLM exhibits structural advantages like low order-flip rates (0.0002) and length indifference, its confidence scores are overconfident but calibratable via temperature scaling. However, calibrated confidence is ineffective for gating decisions, as the model escalates nearly all items to a strong judge in a cascade setup.
Entities (11)
Relation Signals (12)
CLM-v0.1-8B → hasaccuracyon → JudgeBench
confidence 95% · JudgeBench (0.475... intervals contain chance
CLM-v0.1-8B → hasaccuracyon → RewardBench 2
confidence 95% · 0.351 on RewardBench 2 best-of-four
CLM-v0.1-8B → hasaccuracyon → RewardBench
confidence 95% · CLM clears its trivial baselines on three benchmarks: 0.549 on RewardBench
CLM-v0.1-8B → hasaccuracyon → RM-Bench
confidence 95% · On RM-Bench (0.508... intervals contain chance
CLM-v0.1-8B → hasaccuracyon → HaluEval
confidence 95% · answers every HaluEval item with one constant label... accuracy of 0.581
Qwen3-8b → hasorderfliprate → 0.2188
confidence 95% · the generative judge flips at 0.2188
CLM-v0.1-8B → hasorderfliprate → 0.0002
confidence 95% · CLM’s flip rate is 0.0002
CLM-v0.1-8B → outperformedby → Skywork-Reward-V2-8B
confidence 95% · CLM trails... the reward model by 0.289 to 0.525
CLM-v0.1-8B → outperformedby → Qwen3-8b
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8B scores between 0.351 (best- of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench, and answers every HaluEval item with one constant label, matching the trivial always-first baseline at 0.581. Judges with the same parameter count score far higher everywhere: a reward model reaches 0.764 to 0.976 and a generative judge 0.611 to 0.778, and every gap to CLM is significant after Benjamini-Hochberg correction. Two properties do work. Raw confidences are overconfident by up to +0.401, yet one pooled temperature fit on held-out calibration items repairs expected calibration error to at most 0.062, and the repaired confidence ranks the model's own errors above chance on three of six benchmarks. The decision order-flip rate is 0.0002 against 0.2188 for the generative judge, and the length-preference shift is -0.023 against -0.217. The confidence-gated cascade, however, escalates between 0.923 and 1.000 of items to the strong judge at the preregistered 0.97 retention bar: calibrated confidence about a near-chance judge has almost nothing to keep. The design: five public preference benchmarks and one hallucination benchmark with real labels, scored under a preregistration frozen before any test item was seen, against generative, reward-model, and trivial baselines, with per-item predictions released.
Tags
Links
- Source: https://arxiv.org/abs/2610.07177v1
- Canonical: https://arxiv.org/abs/2610.07177v1
Trouble viewing inline? Open PDF directly →
Full Text
35,378 characters extracted from source content.
Expand or collapse full text
CLM-as-a-Judge: Evaluating an Open Contrastive Decision Model on Public Judge Benchmarks Gowthamkumar Nandakishore Independent researcher October 2026 Abstract An open contrastive decision model is near chance as a judge on the hard public benchmarks: Contrastive-LM/CLM-v0.1-8Bscores between 0.351 (best-of-four, chance 0.250) and 0.593 (pairwise, chance 0.500), is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench, and answers every HaluEval item with one constant label, matching the trivial always-first baseline at 0.581. Judges with the same parameter count score far higher everywhere: a reward model reaches 0.764 to 0.976 and a generative judge 0.611 to 0.778, and every gap to CLM is significant after Benjamini–Hochberg correction. Two properties do work. Raw confidences are overconfident by up to +0.401, yet one pooled temperature fit on held-out calibration items repairs expected calibration error to at most 0.062, and the repaired confidence ranks the model’s own errors above chance on three of six benchmarks. The decision order-flip rate is 0.0002 against 0.2188 for the generative judge, and the length- preference shift is -0.023 against -0.217. The confidence-gated cascade, however, escalates between 0.923 and 1.000 of items to the strong judge at the preregistered 0.97 retention bar: calibrated confidence about a near-chance judge has almost nothing to keep. The design: five public preference benchmarks and one hallucination benchmark with real labels, scored under a preregistration frozen before any test item was seen, against generative, reward-model, and trivial baselines, with per-item predictions released. The author has no affiliation with, funding from, or financial interest in Contrastive-LM, Alibaba/Qwen, Amazon/AWS, Convai Innovations, TypeSafe, meraGPT, or the publishers of any benchmark or baseline model used. 1 Introduction Decision-only judges promise a fixed small cost per decision, a confidence score that is a model quantity rather than a sampled token, and immunity to the order in which candidate answers are presented. Whether an open contrastive decision model also delivers judging competence is an empirical question. This study answers it forContrastive-LM/CLM-v0.1-8Bunder a preregistration frozen before any evaluation item was scored, against generative, reward-model, and trivial baselines measured on the same items under the same scoring rules. The first finding is negative: zero-shot CLM is not a usable judge on any benchmark studied here. It clears its trivial baselines on RewardBench, RewardBench 2, and the RewardBench 2 pair conversion, reaching at most 0.593 against 0.500 chance, and it is statistically indistinguishable from coin flipping on RM-Bench and JudgeBench. Same-size autoregressive judges score far higher everywhere: the reward model by 0.289 to 0.525 accuracy points, the generative judge by 1 arXiv:2610.07177v1 [cs.LG] 5 Oct 2026 0.136 to 0.324. On hallucination recognition CLM degenerates to a single answer for every item. Fine-tuning new decision heads on ten thousand preference pairs moves accuracy by -0.028 to +0.060 per benchmark, which changes none of these conclusions. The second finding is that the confidence channel works even though the judge does not. Raw similarity confidences are overconfident by +0.235 to +0.401, yet one pooled temperature fit on held-out calibration items repairs expected calibration error to at most 0.062 on every benchmark, and the repaired confidence ranks the model’s own errors above chance on three of six benchmarks. Calibrated confidence about a near-chance judge cannot gate: no preregistered threshold rule reaches the 0.10 selective error target at usable coverage, and the confidence-gated cascade escalates nearly every item. The third finding is that the structural claims survive measurement. CLM’s decisions flip under answer-order swap at a rate of 0.0002, every flip an exact score tie; the generative judge flips at 0.2188 and a smaller decision baseline at 0.4395. CLM is nearly indifferent to answer length, where the generative and reward judges shift substantially with it. These properties come from the architecture, not from accuracy, and they are the only properties this study can recommend. Contributions. 1.A preregistered, artifact-complete evaluation of an open contrastive decision model as a judge on six public benchmarks against generative, reward-model, and trivial baselines: CLM trails the generative judge by 0.136 to 0.324 and the reward model by 0.289 to 0.525, all gaps significant at q = 0.05. 2. Calibration and error-detection measurements for embedding-similarity confidence: one pooled temperature (20.1) repairs expected calibration error from up to 0.402 to at most 0.062. 3. Measured order-flip and length-bias rates for four judge families: 0.0002 for CLM against 0.2188 for the generative judge. 4.A confidence-gated cascade reported as agreement with the strong judge, escalation fraction, and retained accuracy: the gate escalates between 0.923 and 1.000 of items at the 0.97 retention bar. 5.Released per-item predictions, manifests, metric code, and a frozen preregistration; every printed number maps mechanically to an artifact key. The author has no affiliation with, funding from, or financial interest in Contrastive-LM, Alibaba/Qwen, Amazon/AWS, Convai Innovations, TypeSafe, meraGPT, or the publishers of any benchmark or baseline model used. 2 Related work JEV-as-a-Judge [21] evaluates a small decision model as a judge on public benchmarks and escalates to a stronger judge below a confidence threshold; this study follows that design (public benchmarks with real labels, generative and reward-model comparators, calibration and error detection, a measured cascade) and applies it to a different model class. No number from that paper is compared against a measured number here: its judges, hardware, and prompts differ. LLM-as-a-Verifier [16] studies verification as a general interface; BERT-as-a-Judge [9] shows a 2 small encoder can replace lexical metrics for reference-based evaluation. CLM is described in its technical report [15]; a companion study [28] established the reproduction protocol reused here. The benchmarks are RewardBench [17], RewardBench 2 [25], RM-Bench [24], JudgeBench [35], and HaluEval [19]; each benchmark’s official scoring rule is ground truth. Generative judges show position and length bias [36,39] and favor their own generations [32]; agreement with a stronger teacher does not establish correctness [11]. Judgment detection work shows LLM- generated judgments are identifiable from score-candidate interactions alone, which quantifies judge bias [18]. The architecture studied here scores candidates independently of presentation order, making position bias a testable structural claim. Temperature scaling and its evaluation are standard [12,27,29,34,38]; selective prediction and learning to defer formalize accept-or-escalate behavior [3, 7, 8, 26], and LLM cascades apply it to model pairs [2, 5, 14, 20, 31]. 3 Setup All models and datasets are public. Every locally run model is pinned by immutable revision hash inanc/pins.json; local judges run on one Apple M5 Max with 64 GB of unified memory, and the strong judge is hosted (below). Latency is reported for local judges only, as context; the hosted judge’s latency is service latency and no cost comparison against it is made. CLM.Contrastive-LM/CLM-v0.1-8Bselects among typed candidate answers by embedding similarity behind a local embeddings server written for this study; the encoder is not quantized, and its serving path was verified against the publisher’s released examples before any scoring (one discrepancy with the publisher’s quickstart prints is recorded inanc/metrics/). Judgments are zero shot: a pairwise item becomes one typed choice question with two candidates, a best-of- four item one choice question with four candidates, and a hallucination item one typed binary question. CLM reports a probability per candidate; its confidence is the maximum probability. A fine-tuned variant (new decision heads trained on 10,000 pairs from the pinned Skywork preference dataset [22], deduplicated against benchmark prompts, frozen encoder untouched) runs the same passes. laya. The laya 0.3.20 base checkpoint [28] answers the same typed questions through its own runtime, with a 512-token content window. Cheap generative judge. Qwen3-8B [33] in non-thinking chat mode via MLX in bfloat16 [13], greedy decoding, at most 64 new tokens, one frozen prompt per task form requesting a one-line JSON label with a stated confidence; a reply that does not parse falls back to the first valid label at confidence 0.5 and is flagged. Strong judge. Qwen3-32B (strong judge), served through Amazon Bedrock asqwen. qwen3-32b-v1:0 , temperature 0, 64 completion tokens, frozen prompt; access dates and token counts in anc/. Reward model. Skywork-Reward-V2-8B [23], a sequence classifier, selected by a preregistered rule: the top open sequence-classifier reward model at or under 8B parameters by overall mean on the pinned RewardBench 2 leaderboard snapshot. A pairwise probability is the logistic of the score difference; best-of-four uses a softmax over scores. A preference reward model defines no 3 Table 1: Truncated items per judge family and benchmark: fraction of items with any truncated text (left of slash) and fraction of input tokens kept (right), measured with each family’s own tokenizer under the single frozen policy. Source: anc/metrics/gate05_truncation.json. BenchmarkCLM (2048) laya (512) Generative (4096) RM (4096) RewardBench0.01 / 0.99 0.40 / 0.760.00 / 1.000.00 / 1.00 RewardBench 20.56 / 0.83 0.93 / 0.270.09 / 1.000.06 / 1.00 RewardBench 2 pairs 0.06 / 1.00 0.77 / 0.500.00 / 1.000.00 / 1.00 RM-Bench0.05 / 0.99 0.84 / 0.510.01 / 1.000.00 / 1.00 JudgeBench0.15 / 0.97 0.99 / 0.420.06 / 0.990.06 / 0.99 HaluEval0.19 / 0.71 0.25 / 0.540.19 / 0.710.19 / 0.71 calibrated binary judgment for a single response, so it answers no HaluEval items (a preregistered exclusion). Trivial baselines. Always-first, a per-item-seeded random choice, and longer-response-wins. Benchmarks and task mapping. Five preference benchmarks and one hallucination bench- mark, all with real labels, all pinned: RewardBench (filtered split), RewardBench 2 (best-of-four primary; a derived chosen-versus-each-rejected pairwise form secondary), RM-Bench (official style matrix; the aggregate is the mean of the four domain averages), JudgeBench, and HaluEval. The RewardBench 2 Ties subset is excluded: its official score is a weighted composite with no per-item accuracy. RewardBench 2 credit follows the official rule: a tied maximum that includes the chosen response earns the reciprocal of the tie count. HaluEval becomes a binary typed question; its qa, dialogue, and summarization sub-tasks are subsampled to 1,000 rows each with the study seed by a deterministic per-item coin, and the general sub-task is kept as is. A deterministic per-item coin places the labeled winner at position A or B; pairwise and best-of-four items run in both presentation orders, hallucination items and the strong judge in order 1 only; analyses involving the strong judge use order-1 passes for every tier. Truncation. One policy for every judge: the prompt is kept whole up to the smaller of 512 tokens and half the judge’s content window; the remaining window is split equally across response texts; every text keeps its tail. Windows are 2048 tokens for CLM, 512 for laya, and 4096 for the generative judges and the reward model, counted by each judge’s own tokenizer. Table 1 reports the truncated fraction per family and benchmark; the laya window truncates most items on four benchmarks and CLM truncates a majority of RewardBench 2 items, which frames every comparison that follows. Metrics and statistics. Accuracy is mean per-item credit under each benchmark’s official rule. Calibration uses NLL, equal-width 15-bin ECE, and the signed gap (mean confidence minus accuracy) [10,12,29,34]; error detection uses the AUROC of confidence against per-item correctness. Every interval is a cluster bootstrap at prompt level with 10,000 resamples and seed 0: BCa for single quantities, percentile for paired deltas, add-one two-sided p-values [4,6]. The preregistered hypothesis family is tested with Benjamini–Hochberg control atq= 0.05 [1]. Items sharing a prompt always resample together. 4 Table 2: Evaluation accuracy per judge and benchmark (measured). Chance is 0.250 on RewardBench 2 best-of-four and 0.500 on the pairwise benchmarks. The strong judge is hosted; all other judges ran locally. Source:anc/metrics/comparator_table.json, which also carries the BCa 95% intervals. JudgeRewardBench RewardBench 2 RB2 pairs RM-Bench JudgeBench HaluEval CLM0.5490.3510.5930.5080.4750.581 CLM (tuned heads)0.5210.3870.6470.5390.5350.581 laya0.5120.2800.5260.4940.4980.544 Qwen3-8B (generative)0.7780.6750.7580.6760.6110.728 Qwen3-32B (strong, hosted)0.8680.7320.8270.7260.6150.745 Skywork-Reward-V2-8B0.9760.8760.9500.9230.764— Trivial: always first0.5000.2630.5000.5000.5000.581 Trivial: random0.5000.2500.5000.5000.5000.501 Trivial: length0.4410.2230.4180.4680.4670.425 Preregistration. The preregistration was frozen before any evaluation item was scored; its SHA- 256 and freeze time are inanc/prereg_freeze.json, and every evaluation manifest postdates the freeze [30]. Splits are 30% calibration and 70% evaluation, stratified by benchmark and subset, assigned at prompt level with seed 20261001; the split digest and per-benchmark listings are released. Temperatures, isotonic maps, and every threshold are fit on calibration items only. Two departures from the preregistration occurred. The strong judge ran on Amazon Bedrock rather than locally, so the preregistered latency-cost criterion for H4 was not evaluated. Duplicate source ids in two benchmarks were handled by occurrence index, so twin items are scored and aligned exactly once each. 4 Results: accuracy Table 2 reports evaluation-split accuracy for every judge on every benchmark under the official scoring rules, with BCa 95% intervals for all cells inanc/metrics/comparator_table.json. CLM clears its trivial baselines on three benchmarks: 0.549 on RewardBench (interval 0.527 to 0.570), 0.351 on RewardBench 2 best-of-four (chance 0.250), and 0.593 on the pair conversion. On RM-Bench (0.508, interval 0.487 to 0.529) and JudgeBench (0.475, interval 0.426 to 0.518) both intervals contain chance: on the two benchmarks built around subtle content differences and objective correctness the embedding-similarity judge carries no signal this design can detect. Every same-size autoregressive judge is far ahead: all ten preregistered H1 differences are significant, and CLM trails the generative judge by 0.136 to 0.324 and the reward model by 0.289 to 0.525. The hosted strong judge scores between 0.615 and 0.868. The laya baseline sits at or near chance everywhere, never more than a few points above it (Table 2), consistent with its window truncating most inputs (Table 1); the length baseline is below random everywhere, so gold responses here are more often the shorter ones. Judges not named in a preregistered hypothesis are compared descriptively only. Fine-tuned heads. A day of preference data moves the judge, but not out of its class. The tuned heads reach validation accuracy 0.782 and holdout accuracy 0.764 against 0.623 for the released heads on the training distribution, and on the benchmarks they change accuracy by +0.054 on the pair conversion (interval +0.032 to +0.077), +0.060 on JudgeBench, +0.031 on 5 RM-Bench, +0.035 on RewardBench 2, and -0.028 on RewardBench, where the tuned heads are worse than the released ones. All five differences are significant (family H9), so the training signal is real in both directions; the largest gain is several times smaller than the gaps to the autoregressive judges. Hallucination detection. CLM answers every HaluEval item with the same label. Its accuracy of 0.581 (interval 0.562 to 0.599) is exactly the trivial always-first baseline (0.581): per-class accuracy in the released predictions is one on items whose gold label matches the constant answer and zero on the rest. The preregistered H10 test against the sampling-coin chance of one half is significant, but the comparison that matters is the constant baseline, and there CLM adds nothing. The generative judge reaches 0.728 (-0.147 for CLM against it, significant) and the strong judge 0.745. The tuned heads produce the same constant-label behavior. Order and length stability. Each pairwise and best-of-four item was scored in both presen- tation orders in separate passes. CLM’s flip rate is 0.0002 (BCa interval 0.0000 to 0.0008; 3 flips, each an exact score tie broken by position). The reward model, which also scores responses one at a time, flips at 0.0019; the generative judge flips at 0.2188, more than one in five items; laya flips at 0.4395. All three differences against CLM are significant. On length: over all pair items CLM chooses the longer response at 0.476, falling only to 0.453 where the gold response is the shorter one (delta -0.023), so CLM is indifferent to length and nearly indifferent to correctness. The generative judge moves from 0.559 to 0.342 (delta -0.217) and the reward model from 0.481 to 0.082 (delta -0.399): both track the gold answer away from the longer response. 5 Results: calibration, error detection, and gating Four confidence maps are compared, all fit on calibration items only: identity, one pooled temperature, one temperature per benchmark, and per-benchmark isotonic regression [12,38]. Temperature preserves the argmax, so accuracy is unchanged; the question is whether confidence becomes usable. Raw confidence is severely overconfident everywhere: the signed gap runs from +0.235 on RewardBench to +0.401 on RewardBench 2, and all eighteen preregistered H2 contrasts are significant. The pooled temperature is reported as fitted: 20.1, which sits at the upper bound of the fitter’s search range (logT ∈[−3,3], soT ≤ e 3 ); the unconstrained optimum would be somewhat larger, every reported map uses the fitted value consistently, and no map was refit for this paper. One pooled temperature is enough: expected calibration error falls from 0.235–0.402 under identity to between 0.032 and 0.062 (Table 3, Figure 1); per-benchmark temperatures and isotonic maps help on some benchmarks and hurt on JudgeBench. Calibrated confidence ranks the model’s own errors above chance (AUROC against a per- mutation null) on RewardBench (0.587, interval 0.562 to 0.611), RewardBench 2 (0.609), and the pair conversion (0.602); the JudgeBench point estimate (0.552) does not survive correction, and RM-Bench (0.487) and HaluEval (0.495) sit at chance. Where the judge is at chance, its confidence has no errors to rank. Gating is unambiguous and negative: at the frozen error target of 0.10, none of the three preregistered acceptance rules (point, cross-fit, Wilson upper bound [37]) produces a threshold on five of six benchmarks, because with accuracy near chance even the most confident slice carries error far above the target; RM-Bench alone yields 0.70 under the point and cross-fit rules. 6 Table 3: Expected calibration error on evaluation items under the four confidence maps, all fit on calibration items only. The identity column also shows the signed confidence-minus-accuracy gap. Source: anc/metrics/calibration_table.json. BenchmarkIdentity ECE Signed gap Pooled T Per-bench T Isotonic RewardBench0.235+0.2350.0320.0320.015 RewardBench 20.402+0.4010.0610.0230.060 RewardBench 2 pairs0.283+0.2830.0490.0270.020 RM-Bench0.343+0.3430.0420.0420.028 JudgeBench0.354+0.3510.0620.0830.122 HaluEval0.255+0.2540.0600.0090.040 Table 4: Cascade as agreement with the strong judge, per benchmark: accuracy of each tier alone, rate at which both judges emit the same decision, cascade accuracy at the retention- selected thresholdτ ∗ , escalation fraction, retained fraction of strong-judge accuracy, and the random-escalation control at the matched fraction. Strong, Cascade, Random, and Re- tained all use one convention, the item-level mean on order-1 evaluation items; RM-Bench’s value in Table 2 instead uses the benchmark’s official domain-average aggregate. Sources: anc/metrics/comparator_table.json, cascade_agreement.json, h4_cascade.json. BenchmarkCLM Strong Agree Cascade Escal. Retained Random RewardBench0.549 0.868 0.532 0.854 0.933 0.9830.848 RewardBench 20.351 0.732 0.339 0.732 1.000 1.0000.732 RewardBench 2 pairs 0.593 0.827 0.580 0.818 0.923 0.9890.809 RM-Bench0.508 0.760 0.475 0.741 0.954 0.9760.749 JudgeBench0.475 0.615 0.482 0.615 0.979 1.0000.613 HaluEval0.581 0.745 0.662 0.745 1.000 1.0000.745 6 Cascade The cascade answers each item with CLM and escalates to the strong judge when CLM’s calibrated confidence falls below a threshold; the threshold is the lowest calibration-split grid point whose retained accuracy clears the preregistered 0.97 bar. Because the strong judge is hosted, the preregistered latency-cost criterion for H4 was not evaluated; what is reported is hardware-independent and framed as agreement with the strong judge. Table 4 gives, per benchmark: CLM-alone accuracy, strong-judge-alone accuracy, the agreement rate between CLM and the strong judge on evaluation items, cascade accuracy at the retention-selected threshold, the escalation fraction, the retained fraction, and the random-escalation control at the matched fraction. The gate is nearly decorative (Figure 2). CLM agrees with the strong judge on 0.339 to 0.662 of items, roughly its own accuracy level, so low confidence is the norm and the retention-selected threshold escalates 0.923 to 1.000 of items; on RewardBench 2, JudgeBench, and HaluEval the cascade is the strong judge. Where the gate keeps items it gives up accuracy against the strong judge alone (-0.014 on RewardBench, -0.009 on the pair conversion, -0.019 on RM-Bench, each significant), and the random-escalation control at the matched fraction performs comparably, so the gate’s targeting adds little beyond its volume. Threshold transfer collapses for the same reason: only RM-Bench produced a threshold, and applied elsewhere it accepts almost nothing 7 0.0 0.2 0.4 0.6 0.8 1.0 accuracy RewardBench identity single T RewardBench 2RB2 pairs 0.000.250.500.751.00 confidence 0.0 0.2 0.4 0.6 0.8 1.0 accuracy RM-Bench 0.000.250.500.751.00 confidence JudgeBench 0.000.250.500.751.00 confidence HaluEval Figure 1: Reliability diagrams for CLM confidence on evaluation items, identity map against the pooled temperature map, per benchmark. Generated byanc/code/build_figures.pyfrom released predictions. (at most six items on any target benchmark, accepted error 0.000). Table 5 reports local judges’ latencies as context; no cost ratio against the hosted strong judge is computed. 7 Limitations 1.Hosted strong judge. The strong judge is served by Amazon Bedrock; provider weights, precision, and serving configuration are not revision-pinned and can change without notice, so its results are a property of the service at the recorded access dates. 2.Encoder serving path. The CLM encoder runs on a locally built serving path verified against the publisher’s released examples; one discrepancy with the publisher’s quickstart prints is recorded inanc/metrics/. If the reference path differs in some unprobed way, CLM’s numbers could shift. 3.Truncation. The 512-token window truncates most laya inputs on four of six benchmarks; CLM truncates a majority of RewardBench 2 inputs. Rankings between judges with different windows partly reflect window size. 4.No human adjudication. Benchmark labels are ground truth, including their known noise. 5.Scope. One cheap judge, one strong judge, one error target; local latencies are specific to one machine and these serving stacks; hosted frontier judges beyond the one used were not 8 0.00.20.40.60.81.0 escalation fraction (share of items sent to strong judge) 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00 retained fraction of strong-judge accuracy RewardBench RewardBench 2 RB2 pairs RM-Bench JudgeBench HaluEval Figure 2: Retained accuracy against escalation fraction for the confidence-gated cascade (per grid threshold) and the matched random-escalation control, per benchmark. The horizontal line is the preregistered 0.97 retention bar. Generated by anc/code/build_figures.py. run, so nothing here bounds what they would do. 8 Ethics This study evaluates released models on released benchmarks with real labels; no human subjects, private data, or crowdwork were involved. Benchmark contents include adversarial and safety- sensitive text by design; nothing new of that kind was generated. Energy use was one workstation for the study’s duration plus one hosted judge. 9 Independence statement The author has no affiliation with, funding from, or financial interest in Contrastive-LM, Al- ibaba/Qwen, Amazon/AWS, Convai Innovations, TypeSafe, meraGPT, or the publishers of any benchmark or baseline model used. 9 Table 5: Measured median latency per item in milliseconds for the local judges (20-item probes, first two warm-up items skipped, one Apple M5 Max), shown as context only. The artifact stores medians at one-tenth-millisecond resolution; printed integers round the stored values. Source: anc/metrics/gate03_latency.json. Benchmarklaya CLM Skywork-Reward-V2-8B Qwen3-8B (generative) RewardBench23 338244746 RewardBench 229 6745581,058 RewardBench 2 pairs 24 233359898 RM-Bench22 106230780 JudgeBench32 292331937 HaluEval2363—635 10 Reproducibility The ancillary files (anc/) contain the preregistration and its freeze record, all pins with revision hashes, split listings and digests, one manifest per pass, per-item predictions for every judge and benchmark, all metric JSONs, and the metric and analysis code. Every number in every table and in the abstract is a generated macro mapping to a named ancillary file and key; the mapping was checked mechanically (anc/NUMBER_CHECK.md) and the reported quantities were recomputed from the artifacts alone by independent sessions with no access to the manuscript source. Local models and datasets load by pinned revision; the study seed is 20261001; the bootstrap seed is 0. Reproducing the analysis from the released predictions requires no credential. Re-scoring the strong judge’s passes requires an Amazon Bedrock credential and reaches a service that is not revision-pinned, so bit-exact reproduction of those predictions is not guaranteed; model id, region, decoding parameters, access dates, and token usage are in the manifests and anc/pins.json. Acknowledgements and AI-assistance disclosure The experiments were executed by LLM agents under the author’s direction. Every reported number was verified against the released artifacts by three independent passes (Section 10). The author reviewed the manuscript and takes responsibility for its content. References [1]Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B, 57 (1):289–300, 1995. [2] Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024. arXiv:2305.05176. [3]C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41–46, 1970. 10 [4]Anthony C. Davison and David V. Hinkley. Bootstrap Methods and their Application. Cambridge University Press, 1997. [5] Jasper Dekoninck, Maximilian Baader, and Martin Vechev. A unified approach to routing and cascading for LLMs, 2024. [6]Bradley Efron and Robert J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993. [7] Ran El-Yaniv and Yair Wiener. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11:1605–1641, 2010. [8]Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2017. [9]Hippolyte Gisserot-Boukhlef, Nicolas Boizard, Emmanuel Malherbe, Céline Hudelot, and Pierre Colombo. BERT-as-a-judge: A robust alternative to lexical methods for efficient reference-based LLM evaluation. In Proceedings of the Conference on Language Modeling (COLM), 2026. Camera-ready (arXiv:2604.09497v2, 20 July 2026). [10] Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. [11]Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary LLMs, 2023. [12]Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 2017. arXiv:1706.04599. [13]Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. MLX: Efficient and flexible machine learning on apple silicon.https://github.com/ml-explore/mlx, 2023. Version pinned in the ancillary files. Accessed 2026-09-28. [14] Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-based cascade deferral suffice? In Advances in Neural Information Processing Systems (NeurIPS), 2023. [15]Jacky Kwok, Hangoo Kang, Tarun Suresh, Jon Saad-Falcon, Marco Pavone, Christopher Ré, and Azalia Mirhoseini. Contrastive language models: A system one model for fast and generalizable decision-making. Technical report, 2026. URLhttps://contrastive-lm. notion.site. Model weights:Contrastive-LM/CLM-v0.1-8Bon Hugging Face, Apache 2.0. Accessed 2026-09-28. [16] Jacky Kwok, Shulu Li, Pranav Atreya, Yuejiang Liu, Yixing Jiang, Chelsea Finn, Marco Pavone, Ion Stoica, and Azalia Mirhoseini. LLM-as-a-verifier: A general-purpose verification framework, 2026. v2, 7 Jul 2026. [17] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. 11 [18]Dawei Li, Zhen Tan, Chengshuai Zhao, Bohan Jiang, Baixiang Huang, Pingchuan Ma, Abdullah Alnaibari, Kai Shu, and Huan Liu. Who’s your judge? on the detectability of LLM-generated judgments, 2025. arXiv:2509.25154, September 2025. [19]Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6449–6464, 2023. arXiv:2305.11747. [20]Lei Li, Yankai Lin, Deli Chen, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Cascadebert: Accelerating inference of pre-trained language models via calibrated complete models cascade. In Findings of the Association for Computational Linguistics: EMNLP, 2021. [21]Yubo Li, Yidi Miao, Ramayya Krishnan, and Rema Padman. JEV-as-a-Judge: Accept when confident, escalate when unsure, 2026. arXiv:2609.26550v1, 22 September 2026. [22]Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for re- ward modeling in LLMs. arXiv preprint arXiv:2410.18451, 2024. Source of the Skywork-Reward-Preference-80K-v0.2 training pairs. [23]Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025. [24]Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. RM-bench: Benchmark- ing reward models of language models with subtlety and style. In International Conference on Learning Representations (ICLR), 2025. arXiv:2410.16184. [25]Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. In International Conference on Learning Representations (ICLR), 2026. arXiv:2506.01937. [26]Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning (ICML), 2020. [27] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2015. [28]Gowthamkumar Nandakishore. Laya as a typed probabilistic assessor: An indepen- dent reproduction and a preregistered study of calibration and selective escalation, 2026. arXiv:2609.33843. [29]Jeremy Nixon, Mike Dusenberry, Ghassen Jerfel, Timothy Nguyen, Jeremiah Liu, Linchuan Zhang, and Dustin Tran. Measuring calibration in deep learning. In CVPR Workshops, 2019. [30]Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11):2600– 2606, 2018. 12 [31]Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. Routellm: Learning to route LLMs with preference data, 2024. [32]Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM evaluators recognize and favor their own generations. In Advances in Neural Information Processing Systems (NeurIPS), 2024. arXiv:2404.13076. [33] Qwen Team. Qwen3 technical report, 2025. [34] Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, and Michael C. Mozer. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022. [35] Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y. Tang, Alejandro Cuadron, Chen- guang Wang, Raluca Ada Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In International Conference on Learning Representations (ICLR), 2025. arXiv:2410.12784. [36]Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024. arXiv:2305.17926. [37] Edwin B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212, 1927. [38]Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass probability estimates. Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2002. [39]Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS), 2023. arXiv:2306.05685. 13