Paper deep dive
Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluation
Weining Zhang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 87%
Last extracted: 8/3/2026, 9:24:21 AM
Summary
This paper investigates whether domain specialization in LLM evaluation should be embedded in model weights (via specialized adapters) or in the routing logic (deferral mechanisms). The authors find that splitting a training corpus among multiple specialized LoRA 'judgelets' significantly degrades performance compared to a monolithic adapter, suggesting that shared judgment learning is crucial. Conversely, implementing specialization through a learned risk-based deferral cascade (routing uncertain examples to larger models) improves accuracy and efficiency. The study proposes a design rule: share judgment learning until data justifies a split, and place domain-specific adaptation in audited release boundaries.
Entities (7)
Relation Signals (6)
Judgelet → degradesperformancewhensplit → Monolithic Adapter
confidence 90% · Dividing the same training corpus among eight criterion-family LoRA judges, however, loses 10.05 points... compared with one monolithic adapter
Risklet → enables → Learned Deferral
confidence 90% · The result changes when specialization governs deferral rather than judgment. On RewardBench 2, learned correctness heads route examples
Judgelet → uses → LoRA
confidence 90% · Each is a parameter-efficient update trained on a family of related criteria... We replace its scalar reward head with a five-class score head and train rank-8 LoRA modules
Risklet → improvesaccuracy → RewardBench 2
confidence 85% · Across 20 locked repartitions, the cascade attains 89.40% accuracy, compared with 84.75% for 8B alone
Skywork-Reward-V2 → providesbasefor → Judgelet
confidence 85% · All judgelets use the 0.6B Skywork-Reward-V2 Qwen3 model as a shared base.
Prometheus → uses → FeedBack Collection
confidence 80% · We partition the Prometheus Feedback Collection into training, calibration, development, locked-test, and held-criterion sets
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic systems have widened the gap between producing candidate outputs and reviewing them. This paper asks a practical architectural question: should domain specialization be built into an evaluator's weights, or into the rule that decides when its judgment can be trusted? We study 99,952 public, rubric-conditioned examples. Supplying the correct rubric improves locked-test accuracy by 2.11 points over a response-only control; replacing it with an unrelated rubric costs 2.66 points. Dividing the same training corpus among eight criterion-family LoRA judges, however, loses 10.05 points and cuts audited coverage at a 5% risk target from 24.44% to 5.43%. Matching the bank's stored capacity with one rank-64 adapter does not reproduce this loss. Nor is the result explained by learning rate or optimizer steps. Initializing the family adapters from a shared, trained judge recovers test accuracy to 76.85%, 19.94 points above scratch training at the same learning rate (95% interval 18.88-21.02). The result changes when specialization governs deferral rather than judgment. On RewardBench 2, learned correctness heads route examples through a 0.6B-4B-8B cascade without changing any reward score. Across 20 locked repartitions, the cascade attains 89.40% accuracy, compared with 84.75% for 8B alone, at 0.415 normalized parameter compute. Every run passes an exact one-sided 95% risk audit; margin-based rules remain near 84.8% accuracy while using at least 0.94 compute. These results suggest a qualified design rule: share the learning of judgment until there is enough data to justify a split, and place domain-specific adaptation in an audited release boundary.
Tags
Links
- Source: https://arxiv.org/abs/2607.27984v1
- Canonical: https://arxiv.org/abs/2607.27984v1
Trouble viewing inline? Open PDF directly →
Full Text
41,602 characters extracted from source content.
Expand or collapse full text
Preprint SHARE THE JUDGE, LEARN THE DEFERRAL: WHERE SPECIALIZATION HELPS LLM EVALUATION Weining Zhang Cheung Kong Graduate School of Business ABSTRACT Agentic systems have widened the gap between producing candidate outputs and reviewing them. This paper asks a practical architectural question: should domain specialization be built into an evaluator’s weights, or into the rule that decides when its judgment can be trusted? We study 99,952 public, rubric-conditioned examples. Supplying the correct rubric improves locked-test accuracy by 2.11 points over a response-only control; replacing it with an unrelated rubric costs 2.66 points. Dividing the same training corpus among eight criterion-family LoRA judges, however, loses 10.05 points and cuts audited coverage at a 5% risk target from 24.44% to 5.43%. Matching the bank’s stored capacity with one rank-64 adapter does not reproduce this loss. Nor is the result explained by learning rate or optimizer steps. Initializing the family adapters from a shared, trained judge recovers test accuracy to 76.85%, 19.94 points above scratch training at the same learning rate (95% interval 18.88–21.02). The result changes when specialization governs deferral rather than judgment. On RewardBench 2, learned correctness heads route examples through a 0.6B–4B–8B cascade without changing any reward score. Across 20 locked repartitions, the cascade attains 89.40% accuracy, compared with 84.75% for 8B alone, at 0.415 normalized parameter compute. Every run passes an exact one-sided 95% risk audit; margin-based rules remain near 84.8% accuracy while using at least 0.94 compute. These results suggest a qualified design rule: share the learning of judgment until there is enough data to justify a split, and place domain-specific adaptation in an audited release boundary. 1INTRODUCTION Language-model agents can produce drafts, plans, or tool trajectories much faster than a knowledge- able reviewer can inspect them. The imbalance is easy to miss in demonstrations, where generation is the visible event and review is a line in the pipeline. In a deployed system, however, an answer may also have to satisfy a local policy, a professional standard, or a task-specific rubric. That review step can become the limiting resource. One response is to put a larger language model behind the agent and use it as a judge. Doing so improves judgment, but it does not settle the operational problem. The judge is called at roughly the rate of generation, a scalar score does not record which rule governed the decision, and aggregate accuracy says little about which individual cases can leave the review queue. In practice, an evaluator that knows when to abstain may be more useful than a slightly more accurate evaluator that does not. The relevant domain knowledge is often already written down. Firms maintain rubrics, review checklists, operating procedures, and corrected decisions. These materials are usually inserted into a prompt for a general judge. Here we test a different use: compiling repeated criteria into a small evaluator with a shared base model, criterion-family adapters, a router derived from the rubric text, and separately audited acceptance thresholds. We call the adapters judgelets. Each is a parameter-efficient update trained on a family of related criteria rather than a separate foundation model. Only the routed adapter is active for an example. The design therefore allows stored specialization to grow without activating the whole bank at inference time. It also makes three decisions explicit: which rule applies, which adapter evaluates it, and whether the resulting score is reliable enough to release. 1 arXiv:2607.27984v1 [cs.AI] 30 Jul 2026 Preprint (a) Rubric-conditioned evaluator Task Response Rubric Criterion-family router Frozen shared base + one active LoRA judgelet Ordinal score + confidence (b) Learned-risk cascade 0.6B judge 4B judge 8B judge riskletrisklet deferdefer Release accept Release accept Release Figure 1: Two routing decisions in the evaluation control layer. The rubric selects one criterion-family adapter on a shared base (top). A learned risklet then decides whether a reward-model ranking can be released or should continue to a larger evaluator (bottom). Family granularity controls the amount of shared supervision; risklet thresholds are calibrated and audited on disjoint data. Judgelets are used here as an experiment, not as a presumption that more specialization will help. The first study concerns direct, rubric-conditioned scoring. We partition the Prometheus Feedback Collection into training, calibration, development, locked-test, and held-criterion sets, then compare rubric-free, monolithic, and sparse-adapter evaluators. A cross-cluster rubric replacement intervention checks whether an evaluator actually uses the criterion text. The second study concerns reward-model routing. Four sizes from one open reward-model family allow us to ask whether a small learned correctness head can save evaluator compute without giving up accuracy on RewardBench 2. Several results run against the motivating intuition. The response-only Prometheus control is strong, a raw-margin cascade misses the predeclared risk target, and the eight-family judgelet bank is much worse than one monolithic adapter. Rather than hiding that last result, we use it to locate a boundary: small evaluators benefit from shared judgment training, whereas specialized deferral can exploit differences among already trained judges. The paper makes three empirical contributions: 1. a controlled rubric study with locked criterion shift, a response-only control, and a counter- factual rubric intervention that isolates whether rule text affects a small evaluator; 2. a granularity, capacity, and optimization study that separates sparse deployment from the statistical cost of fragmenting evaluator supervision; and 3.an evaluation-specific deferral study with exhaustive stage subsets, four strong confidence baselines, disjoint fitting/calibration/test data, and exact held-out risk audits. Our claim concerns the resulting architectural comparison, rather than uncertainty routing by itself. 2RELATED WORK LLM judges and specialized evaluators. Prometheus trains open evaluators with references and user-defined score rubrics (Kim et al., 2024a); Prometheus 2 extends this approach to direct assessment and pairwise ranking (Kim et al., 2024b). Small-judge comparisons find pronounced domain dependence (Laddha et al., 2026). We start from that heterogeneity but share a base model across criterion-family adapters. Quantitative LLM Judges uses post-hoc models to align a judge’s score and rationale to human scores (Sahoo et al., 2025). Our post-hoc heads instead predict correctness for routing. 2 Preprint Sparse experts and parameter-efficient adaptation.LoRA freezes a shared model and represents a downstream update with low-rank matrices (Hu et al., 2022). Sparse mixture-of-experts models increase stored capacity while activating only a subset of parameters per input (Fedus et al., 2022). Judgelets combine these deployment properties at a coarser granularity: a rubric-derived router selects one complete LoRA adapter. Unlike token-level MoE routing, the assignment is inspectable and constant for every example governed by the same criterion. Mixture-of-LoRA methods report gains from learned or task-labelled routing (Li et al., 2024; Feng et al., 2024); our controlled result identifies a failure regime when independently optimized experts divide a common judgment task into small rubric families. Rubrics, selective judging, and cascades.AdaRubric generates task-specific dimensions for agent trajectories (Ding, 2026); we instead assume a repeated, versioned collection of domain rubrics and compile it into stable system artifacts. Trust or Escalate provides cascaded selective evaluation with calibrated human agreement (Jung et al., 2025). SCOPE studies finite-sample risk-controlled pairwise judging (Badshah et al., 2026), and Ask a Strong LLM Judge routes uncertain reward-model comparisons to a larger generative judge (Xu et al., 2025). Uncertainty routing itself is therefore not our claim. We study criterion-family specialization, held-criterion behavior, and whether a learned correctness signal changes the accuracy–compute frontier. Hierarchical abstention, confidence tuning, and learned epistemic-risk estimators already establish that cascades can improve cost–accuracy tradeoffs (Zellinger & Thomson, 2024; Rabanser et al., 2025; Melo et al., 2026). Accordingly, our risklet is deliberately a simple logistic probe. Our question is narrower and orthogonal: given a fixed evaluation workload, does domain specialization belong in judge parameters or in the release boundary? Reward-model evaluation. RewardBench 2 evaluates factuality, focus, instruction following, mathematics, safety, and ties, and relates its scores to downstream post-training behavior (Malik et al., 2026). Skywork-Reward-V2 supplies a controlled 0.6B-to-8B ladder with a shared data recipe (Liu et al., 2026), allowing model size to change without changing the family. 1 3PROBLEM SETTING Let a rubric collection beR =r 1 ,...,r m . Each example contains a taskx, candidate response y, optional referencea, rubricr, and ordinal judgmentz ∈ 1,...,L. A monolithic evaluator predictsp(z | x,y,a,r)with one set of trainable parameters. We instead assign each rubric to a family g(r)∈1,...,K. Family k has a low-rank adapter φ k on a frozen shared base θ: p k (z | x,y,a,r) = M θ,φ k (x,y,a,r), k = g(r).(1) Only φ g(r) is active. The bank stores K adapters but activates one. For a confidence functionc k and thresholdt k , a family accepts an automatic judgment whenc k ≥ t k . Its selective risk and coverage are R k (t k ) = Pr(ˆz ̸= z | c k ≥ t k ),Γ k (t k ) = Pr(c k ≥ t k ).(2) Thresholds are chosen on calibration data and never retuned on development, locked test, criterion shift, or external data. For a size cascade with modelsM 1 ,...,M J , every example reachesM 1 and rejected examples continue. Ifq j is the fraction reaching stagejandP j its parameter count, normalized parameter compute is C norm = P J j=1 q j P j P J .(3) The quantity is meant to compare architectures. It should not be read as an estimate of energy use; measured runtime and throughput are reported separately. A cascade can exceed its terminal judge only through complementary errors. For two stages, letA denote early acceptance and C j correctness of stage j. Relative to always using stage 2, ∆ acc = Pr(A∩ C 1 ∩¬C 2 )− Pr(A∩¬C 1 ∩ C 2 ).(4) 1 The exact Qwen3 checkpoints used here (0.6B, 1.7B, 4B, and 8B) are indexed in the official Skywork- Reward-V2 model collection. 3 Preprint We refer to the first term as rescue and the second as harm. Routing is worthwhile only if it retains more helpful small-model disagreements than harmful ones, and if the first stage is cheap enough thatP 1 + (1− Pr(A))P 2 < P 2 . This decomposition separates selective prediction from the stronger claim, which we test below, that a cascade can improve both accuracy and compute. 4RUBRIC COMPILATION AND RISK ROUTING 4.1LOCKED CRITERION SPLITS Rubric text is the unit of policy identity. We reserve a deterministic hash-selected set of complete criteria forshiftlocked. No example carrying one of these criteria is used for adapter training, calibration, development, or ordinary locked test. For remaining criteria, all responses sharing the same criterion–instruction pair receive the same split. This prevents related responses to one task from crossing partitions. 4.2CRITERION-FAMILY ROUTER AND JUDGELET BANK The compiler represents criterion strings with word and bigram TF–IDF features fitted only on criteria available to adapter training. K-means then definesKcriterion families, and new rubrics are assigned by the fitted vectorizer and centroids. We chose this simple router because its assignments can be inspected directly; routing quality is not the variable of interest in this experiment. We evaluateK ∈ 1, 4, 8. The one-family case is the monolithic evaluator. All judgelets use the 0.6B Skywork-Reward-V2 Qwen3 model as a shared base. We replace its scalar reward head with a five-class score head and train rank-8 LoRA modules on attention and feed-forward projections. Each judgelet has 5.05M trainable parameters, 0.84% of the instantiated evaluator. A rank-64 monolithic adapter approximately matches the aggregate trainable capacity of eight rank-8 judgelets. The primary bank matches the monolith’s aggregate optimizer budget: its eight experts receive 98–489 updates (1,494 total), versus 1,489 monolithic updates. This does not imply that every expert converged. We therefore cycle every family to exactly 500 updates (4,000 total) as an optimization control. A second bank starts each expert from the trained rank-8 monolith and applies one family epoch at learning rate5× 10 −5 , preserving shared judgment learning before the split. Because the primary scratch bank uses2× 10 −4 , we also train a scratch bank for the same 1,494 family updates at5× 10 −5 . This matched control isolates initialization within the family-adaptation recipe; it is not a learning-rate sweep. Inputs place the criterion and five score descriptions before the task, response, and reference. A full-corpus tokenizer audit found that over 95% of inputs exceed 512 tokens, while under 3.5% exceed 1,024. Primary models use 1,024 tokens. A 512-token model is retained as a truncation intervention; the rubric-free task-and-response control uses 512 tokens. 4.3RISK AUDITS We compare a global confidence threshold, one threshold per criterion family, and a family policy that applies a Bonferroni adjustment over theKfamilies. Each threshold is selected by scanning the calibration risk–coverage curve. Because this scan is adaptive, the thresholds are best regarded as calibration heuristics. We evaluate realized risk on untouched splits with one-sided 95% Wilson upper bounds rather than treating the search itself as a prospective finite-sample guarantee. 4.4LEARNED CORRECTNESS HEADS Reward-model margins are not comparably calibrated across skills. For each non-terminal model, we fit a logistic risklet to predict whether that stage’s top-ranked candidate is correct. Its inputs are the largest scores, gap, range, mean, variance, candidate count, winning position, and a one-hot skill 4 Preprint family. Fitting, threshold selection, and evaluation use three disjoint partitions. The risklet leaves reward scores untouched: it either accepts the current ranking or forwards the example. 2 We require a 3% one-sided upper bound on calibration error, leaving a predeclared buffer before the 5% held-out audit. The untouched test partition is evaluated with both a Wilson diagnostic and an exact one-sided 95% Clopper–Pearson upper bound. As a stage-selection check, we enumerate all seven compute-ordered subsets that terminate at 8B. The reported three-stage system follows a rule fixed on calibration data: remove any non-terminal stage that accepts no calibration examples. Raw margin, logistic margin calibration, isotonic margin calibration, and a geometry-only logistic probe all use the same partitions. The last variant omits the skill indicator, so it tests whether domain identity adds signal beyond score geometry. 5EXPERIMENTAL PROTOCOL The direct-assessment corpus contains 99,952 Feedback Collection examples and 996 criteria. 3 Criterion holdout reserves 206 criteria and 20,696 examples. The remainder contains 47,638 training, 9,441 calibration, 9,597 development, and 12,580 locked-test examples. Feedback-Bench contributes 1,000 external examples. All 626 of its criterion strings occur somewhere in Feedback Collection; 122 were nevertheless excluded from adapter training and account for 203 external examples. RewardBench 2 contains 1,865 tasks and 8,977 candidate completions. 4 We reproduce its official subset metric on all tasks. The cascade study uses the 1,763 non-tie tasks; the 102 tie tasks remain in the official score but are excluded from correctness-head fitting, threshold calibration, and binary cascade accuracy. Direct-assessment metrics are accuracy, macro F1, mean absolute score error (MAE), quadratic- weighted kappa, Spearman correlation, calibration error, selective coverage and risk, and a one-sided 95% Wilson upper bound. Comparisons use paired bootstrap intervals with criterion ID as the resampling unit. We report active and stored adapter parameters, wall-clock training time, checkpoint size, and inference throughput. The rubric intervention replaces the criterion and all five score descriptions with a rubric sampled from another cluster in the same split. Task, response, reference, label, router assignment, and row identity remain unchanged. Thus a response-only model is invariant, while a rubric-sensitive evaluator should change. A content-hash audit found two duplicated pairs (four of 99,952 rows) with conflicting labels. Neither pair crosses a split, criterion, or routed family. All joins and paired tests use the unique row identifier rather than the content hash. 6RESULTS 6.1THE RUBRIC HELPS, BUT THE RESPONSE IS A STRONG SHORTCUT On the locked test, the full-context monolith is 2.11 points more accurate than the response-only control and lowers MAE by 0.030 (Table 1). A paired bootstrap over 765 criterion IDs gives an interval of+1.48to+2.75points. 5 The difference grows to+3.10points on held criteria (95% interval +2.47 to +3.74). The external result is less decisive. The 1.9-point accuracy gain has a criterion-bootstrap interval of−0.5to+4.4(p = 0.138), although MAE falls from 0.270 to 0.239 (p = 0.033). Full-context accuracy is 79.30% on the 797 examples whose criteria were available to adapter training, compared 2 With the five non-tie RewardBench 2 skills, the risklet has 14 inputs and one intercept. Its arithmetic is negligible beside a 0.6B-parameter forward pass and is not included in Eq. 3; every reward-model forward pass is included. 3 The public releases are Feedback Collection and Feedback-Bench. The artifact manifest records row counts and SHA-256 hashes for the immutable files used in this study. 4 We use the official RewardBench 2 test release; its 102 tie tasks are retained for the official macro metric. 5 Throughout, differences stated in “points” are percentage-point differences. Intervals are 95% intervals unless otherwise specified. 5 Preprint Table 1: Direct-assessment results. Higher is better except MAE. ModelDevTestMAE κShiftExternal Response only, 51274.7574.27.281.91674.7575.70 Monolithic rubric, rank 876.7876.38.250.92977.8577.60 Monolithic rubric, rank 6479.0977.50.237.93478.9782.10 Table 2: Locked-test release with calibration-only thresholds. Evaluator and policyCoverageRiskUpperPass Response, global29.934.915.53No Response, family Bonf.18.012.693.31Yes Rubric, global37.474.845.38No Rubric, family32.064.044.58Yes Rubric, family Bonf.24.442.993.54Yes with 70.94% on the 203 examples governed by 122 held-out criteria. Increasing monolithic adapter rank from 8 to 64 adds 1.11 test points (interval+0.54to+1.71), 1.12 shift points, and 4.50 external points. We also intervene on context length at inference time. Running the same adapter with a 512-token limit lowers test accuracy to 73.82%, a−2.56-point paired difference from the 1,024-token setting (95% interval−3.16to−1.97), and increases MAE by 0.038. Its accuracy is indistinguishable from the response-only control (−0.45points; interval−1.06to+0.15;p = 0.147), even though throughput rises from roughly 58 to 110 examples/s. Bonferroni-audited coverage falls from 24.44% to 19.77%. The rubric benefit disappears when much of the rubric-bearing context is truncated. 6.2COUNTERFACTUAL RUBRICS REMOVE THE GAIN Replacing the rubric and leaving every other field fixed lowers locked-test accuracy from 76.38% to 73.72%. The paired difference is−2.66points (95% interval−3.26to−2.09). On held-criterion shift, accuracy falls from 77.85% to 75.34% (−3.05to−1.97points). The wrong-rubric condition also trails the response-only control on the ordinary locked test, which rules out an explanation based only on extra benign context. 6.3GLOBAL CALIBRATION OVERSTATES SAFE COVERAGE The global calibration threshold accepts 37.47% of locked-test examples, but its 5.38% risk upper bound exceeds the target (Table 2). Separate family thresholds pass at 32.06% coverage and a 4.58% upper bound. The Bonferroni policy is more conservative: 24.44% coverage with a 3.54% bound. On held-criterion shift, its coverage is 26.13% and its bound 3.05%. With the wrong rubric, the ordinary family policy’s test risk bound rises to 5.30% even as coverage drops to 27.24%. The Bonferroni policy stays below the target but accepts only 20.95%. In other words, a release rule audited with valid rubric text cannot be assumed safe after that text is corrupted. 6.4RISKLETS CHANGE THE ACCURACY–COMPUTE FRONTIER The four Skywork models score 61.39, 68.11, 75.44, and 77.91 on the official RewardBench 2 macro metric as model size increases from 0.6B to 8B. The cascade analysis uses a different quantity—micro top-choice accuracy on the 1,763 non-tie tasks—so these scores should not be compared directly. Over 20 train/calibration/test repartitions, the best risklet cascade forwards an average of 19.80% of test tasks to the 8B terminal stage. Its final accuracy is 89.40% at 0.415 normalized compute, 4.66 points above 8B alone on the same partitions (Table 3). The gain ranges from 3.21 to 6.19 points across the 20 deterministic repartitions. Because the underlying tasks are reused, this is a stability range, not an independent-sample confidence interval. Complementary errors make the gain possible: an oracle over all four sizes reaches 91.80%, and 6 Preprint Table 3: RewardBench 2 routing on 1,763 non-tie tasks, mean over 20 locked repartitions (mean test size 616.8). “Exact pass” counts runs whose one-sided 95% Clopper–Pearson upper bound on aggregate early-exit error is at most 5%. Compute is normalized to running 8B on every task. Policy and stagesEarly cov.AccuracyExact passCompute 8B only0.0084.75–1.000 Raw margin, 4B–8B55.6184.8120/200.944 Isotonic margin, 4B–8B48.8684.8419/201.011 Geometry probe, 0.6B–4B–8B80.3589.3720/200.413 Risklet, 0.6B–8B71.5288.7120/200.360 Risklet, 0.6B–4B–8B80.2089.4020/200.415 Risklet, all four stages80.2089.4020/200.476 0.40.60.81.01.2 Normalized parameter compute (8B-only = 1) 85 86 87 88 89 90 Final task accuracy (%) 0.6B-8B 0.6B-4B-8B 8B only Accuracy–compute tradeoffs across all ordered stage subsets raw margin margin logistic isotonic margin geometry logistic risklet Figure 2: All seven compute-ordered stage subsets and five confidence policies. Learned correctness routing forms a separate accuracy–compute frontier. Removing the 1.7B stage preserves 89.40% accuracy and reduces normalized compute from 0.476 to 0.415. Marker shapes identify policies; error bars on learned probes show one standard deviation across deterministic repartitions. an oracle restricted to the deployed 0.6B–4B–8B subset reaches 90.97%. Compared with 8B alone, early exits rescue 5.32% of tasks and harm 0.66%, which gives the+4.66-point net change in Eq. 4. Logistic calibration preserves the raw margin ordering and consequently finds the same optimum after threshold search. Isotonic calibration introduces ties but no accuracy gain. The risklet behaves differently: it uses score geometry and skill identity to find cases in which a smaller model is right, including some that the 8B model misses. The terminal model is therefore not treated as an oracle. Removing the skill one-hot changes accuracy by only 0.03 points and compute by−0.002. Most of the useful signal appears to come from multivariate, stage-local score geometry rather than the RewardBench skill label. The subset comparison also reveals why the first four-stage result was misleading. The 1.7B model accepted no calibration example in any run, so the calibration-only pruning rule removes it before locked evaluation. It would also have accepted no test example, although every example paid for its forward pass. The two-stage 0.6B–8B cascade is the minimum-compute point at 0.360. Adding 7 Preprint 4B costs another 0.055 normalized compute and gains 0.69 accuracy points. Stage membership is therefore part of the fitted system, rather than a harmless implementation detail. Not every pair of judges should be cascaded. A diagnostic composition of the Prometheus rank-8 and rank-64 adapters accepts 27.98% of test examples at the first stage and changes accuracy only from 77.50% to 77.52%, while requiring 1.720 forward passes normalized to rank-64 alone. The two adapters share the same 0.6B base and nearly the same inference cost. Complementary errors are insufficient when the first decision is not materially cheaper. An independent OpenAssistant DeBERTa base–large pair (OpenAssistant, 2023) illustrates the other side of this boundary. 6 The models’ official RewardBench 2 scores are only 26.07 and 33.72, but their errors are strongly complementary. On the same 20 repartitions, a risklet accepts 30.05% at the base stage and raises task accuracy from 40.14% for large alone to 48.78%, an 8.64-point rescue. It passes the exact audit in 19/20 runs; the geometry-only probe attains the same accuracy and passes 20/20. Cost tells a less favorable story. Normalized compute is 1.122 because the 184M base is already 42% as large as the 435M terminal model. The second model family reproduces the accuracy effect, but not a Pareto improvement in cost. 6.5FINE-GRAINED SPECIALIZATION FRAGMENTS SUPERVISION The eight-family bank performs much worse than the monolithic evaluator. Its locked-test accuracy is 66.34%, 10.05 points below the rank-8 monolith (criterion-bootstrap interval−10.87to−9.22), while MAE rises from 0.250 to 0.375. The held-criterion gap is−10.13points (−11.14to−9.13). Performance is uneven across experts: a few reach 71–72%, but several remain between 56% and 61%. Neither global nor ordinary family calibration passes the bank’s locked-test risk audit. Bonferroni calibration passes with a 4.87% upper bound, but coverage is only 5.43%, versus 24.44% for the monolithic evaluator. Counterfactual rubric replacement costs a further 1.72 points (−2.39to−1.07). The bank is not ignoring the rubric. Its problem is that it learns the common scoring task less well once the supervision is divided. Four-family routing narrows the loss without removing it. Locked-test accuracy is 71.81%, 4.57 points below the monolith (interval−5.26 to−3.89), and held-criterion accuracy is 73.78%. Under independent one-epoch training, the pattern in Table 5 is monotone: active adapter size stays fixed, stored parameters increase, and accuracy falls as the data are split more finely. The four-family Bonferroni policy passes the locked test at 17.79% coverage, 6.65 points below monolithic coverage, and narrowly misses on held-criterion shift with a 5.006% upper bound. Extra optimization recovers only part of the gap. Cycling each expert to exactly 500 updates raises the eight-family bank from 66.34% to 72.56%. That 6.22-point gain requires 4,000 updates in total, yet the result remains 3.82 points below the rank-8 monolith and 4.94 points below rank 64. Under-training matters, but it is not the whole explanation. Pre-adaptation on the shared task is much more effective. Starting each expert from the trained rank-8 monolith and then running one family epoch yields 76.85% test accuracy. This is 10.51 points above independent training and 0.47 points above the monolithic initialization (criterion-bootstrap interval +0.13to+0.82). Accuracy is 78.14% on held criteria and 78.90% on the external set. A wrong rubric still lowers test accuracy by 3.28 points (interval−3.90to−2.66), so the recovery does not come from ignoring family-specific rule text. The learning-rate-matched scratch bank makes the comparison more specific. At the same5× 10 −5 rate, with one family epoch and 1,494 aggregate family updates, scratch training reaches 56.91% on locked test, 58.47% on held criteria, and 58.00% externally. Shared initialization is 19.94 points better on locked test (criterion-bootstrap interval+18.88to+21.02). The low-rate scratch run clearly underfits, so it cannot tell us whether5× 10 −5 is generally a good scratch learning rate. What it does rule out is the claim that shared initialization wins merely because it used that rate. Read together with the 4,000-update control, the experiments separate shared pre-adaptation from family-only step count and family learning rate. 6 The exact checkpoints are OpenAssistant/reward-model-deberta-v3-base and OpenAssistant/reward-model- deberta-v3-large. 8 Preprint 5560657075 Locked-test accuracy (%) (focused scale) Monolith rank 8 (5.0M stored) 4 experts, scratch (20.2M stored) 8 experts, scratch (40.4M stored) 8 experts, scratch 5e-5 (40.4M stored) 8 experts, 500 each (40.4M stored) 8 experts, shared init. (40.4M stored) Monolith rank 64 (40.4M stored) 76.38% 71.81% 66.34% 56.91% 72.56% 76.85% 77.50% Locked-test accuracy across specialization controls Figure 3: Locked-test accuracy across specialization and optimization controls. Dots show criterion- bootstrap 95% intervals; the focused horizontal scale is used for point estimates rather than bar lengths. On the external set, the fixed-update control is 0.50 points higher than shared initialization (79.40% versus 78.90%). The criterion-bootstrap interval, however, runs from−2.65to+1.70points. This comparison is unresolved and should not be read as a reversal of the locked-test result. The selective audit leads to the same distinction. The shared-initialization bank releases 24.90% of locked-test examples with a 3.48% one-sided risk upper bound. The fixed-update bank selects 11.76%, but its 5.36% upper bound misses the target; the low-rate scratch bank releases none. The capacity-matched result points in the opposite direction from the sparse bank. Rank 64 stores 40.38M adapter parameters, compared with 40.41M across the bank, yet is 11.16 test points better (interval+10.28to+12.07). It also exceeds rank 8 by 1.11 points and reaches 30.14% Bonferroni- audited coverage. Stored adapter capacity is not the limiting factor. The two optimization controls instead implicate the loss of common judgment training when supervision is fragmented; shared pre-adaptation nearly closes that gap. 7LIMITATIONS Feedback Collection is synthetic, and a model can exploit some of its regularities without reading the rubric. The strong response-only control shows that this is a property of the data, not a boilerplate caveat. Rubric replacement and external examples help expose the shortcut, but neither substitutes for judgments made by domain professionals on naturally occurring work. Criterion shift holds out complete rubric strings, but it does not reproduce a policy revision over time, an adversarially written rule, or a change in the population of actors. Feedback-Bench is external at the example level, yet all of its criteria occur somewhere in Feedback Collection. For that reason we report the adapter-held subset and do not describe the benchmark as wholly new-criterion. The study also uses English lexical routing, one direct-assessment base-model family, retrospective threshold audits, repeated partitions of one benchmark, and parameter count as a compute proxy. Appendix B states the corresponding boundaries in detail. 8CONCLUSION The experiments separate two kinds of specialization that are often grouped together. Specializing the judge weights too early divides the examples needed to learn a common scoring task. More family updates recover some of that loss, but shared pre-adaptation is far more effective and nearly closes the capacity-matched gap. Specializing the release decision has a different effect. A small correctness 9 Preprint head can exploit complementary errors using only the current stage’s scores, but the cascade improves both accuracy and compute only when the earlier evaluators are genuinely cheap. That condition holds for the Skywork ladder studied here and not for the OpenAssistant pair. The useful design rule is consequently conditional, not universal: share judgment learning by default, and specialize deferral only after its risk and cost have been measured. REFERENCES Gene M. Amdahl. Validity of the single processor approach to achieving large scale computing capabilities. In Proceedings of the April 18–20, 1967, Spring Joint Computer Conference, p. 483–485, 1967. doi: 10.1145/1465482.1465560. Sher Badshah, Ali Emami, and Hassan Sajjad. SCOPE: Selective conformal optimized pairwise LLM judging. In International Conference on Machine Learning, 2026. Liang Ding. AdaRubric: Task-adaptive rubrics for reliable LLM agent evaluation and reward learning. arXiv preprint arXiv:2603.21362, 2026. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. Wenfeng Feng, Chuzhan Hao, Yuewei Zhang, Yu Han, and Hao Wang. Mixture-of-LoRAs: An efficient multitask tuning method for large language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), p. 11371–11380. ELRA and ICCL, 2024. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Jaehun Jung, Faeze Brahman, and Yejin Choi. Trust or escalate: LLM judges with provable guarantees for human agreement. In International Conference on Learning Representations, 2025. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine- grained evaluation capability in language models. In International Conference on Learning Representations, 2024a. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Gra- ham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 4334–4353. Association for Computational Linguistics, 2024b. doi: 10.18653/v1/2024.emnlp-main.248. Anish Laddha, Nitesh Pradhan, and Gaurav Srivastava. SLMJury: Can small language models judge as well as large ones? arXiv preprint arXiv:2606.07810, 2026. Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, and Mingjie Tang. MixLoRA: Enhancing large language models fine-tuning with LoRA-based mixture of experts. arXiv preprint arXiv:2404.15159, 2024. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, and Yang Liu. Skywork-Reward-V2: Scaling preference data curation via Human–AI synergy. In International Conference on Learning Representations, 2026. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Ha- jishirzi, and Nathan Lambert. RewardBench 2: Advancing reward model evaluation. In Interna- tional Conference on Learning Representations, 2026. S ́ ebastien Melo, Ga ̈ el Varoquaux, and Marine Le Morvan. Epistemic uncertainty quantification to im- prove decisions from black-box models. In International Conference on Learning Representations, 2026. 10 Preprint OpenAssistant.DeBERTa-v3 reward-model checkpoints.https://huggingface.co/ OpenAssistant/reward-model-deberta-v3-base , 2023. Base and large public model cards. Stephan Rabanser, Nathalie Rauschmayr, Achin Kulshrestha, Petra Poklukar, Wittawat Jitkrittum, Sean Augenstein, Congchao Wang, and Federico Tombari. Gatekeeper: Improving model cascades through confidence tuning. In Advances in Neural Information Processing Systems, volume 38, 2025. Aishwarya Sahoo, Jeevana Kruthi Karnuthala, Tushar Parmanand Budhwani, Pranchal Agarwal, Sankaran Vaidyanathan, Alexa Siu, Franck Dernoncourt, Jennifer Healey, Nedim Lipka, Ryan A. Rossi, Uttaran Bhattacharya, and Branislav Kveton. Quantitative LLM judges. arXiv preprint arXiv:2506.02945, 2025. Zhenghao Xu, Qin Lu, Qingru Zhang, Liang Qiu, Ilgee Hong, Changlong Yu, Wenlin Yao, Yao Liu, Haoming Jiang, Lihong Li, Hyokun Yun, and Tuo Zhao. Ask a strong LLM judge when your reward model is uncertain. In Advances in Neural Information Processing Systems, volume 38, 2025. Michael J. Zellinger and Matt Thomson. Efficiently deploying LLMs with controlled risk. arXiv preprint arXiv:2410.02173, 2024. ETHICS STATEMENT This work uses public model outputs and benchmark annotations; it collects no new human-subject data. Automated evaluators may reduce review cost, but they may also encode contested policies, centralize managerial judgment, or invite more confidence than a benchmark audit warrants. Criteria should remain versioned and attributable, and affected users should have a route of appeal. Cover- age measured on these benchmarks is not a reason to remove human review from legal, medical, employment, or other high-impact decisions. AREPRODUCIBILITY DETAILS All splits use seed 20260728. Direct-assessment adapters train with batch size 4, gradient accumula- tion 8, and 1,024-token inputs unless marked otherwise. Table 4 records the initialization, learning rate, and optimizer updates for every specialization control. Experiments ran with PyTorch 2.8.0, CUDA 12.8, NCCL 2.27.3, and two RTX 5090 GPUs with 32,607 MiB each. Selected prediction- level outputs, split manifests, training metadata, threshold audits, paired-bootstrap results, and all 20-seed outputs for each reported cascade configuration are included in the artifact. Dataset file sizes and SHA-256 hashes are recorded in the download manifest. Table 4: Training recipes for the direct-assessment specialization controls. “Shared updates” are full-corpus optimizer steps completed before any family split; “family updates” are summed across routed experts. DesignInitializationLRShared updatesFamily updates Monolith r8base2e−41,489– K=8, scratchbase2e−401,494 K=8, scratch, matched LRbase5e−501,494 K=8, 500 eachbase2e−404,000 K=8, shared init.monolith r85e−51,4891,494 Monolith r64base2e−41,489– BADDITIONAL LIMITATIONS The compiler uses English lexical features and one base-model family. Dense semantic routing, multilingual criteria, joint multi-expert training, and alternative parameter-efficient modules may change the specialization tradeoff. The ordinal head produces no written evidence or critique. 11 Preprint Table 5: Criterion-family granularity and optimization controls. Adapter counts are millions. Cover- age is the locked-test release rate under the calibration-selected Bonferroni policy; † marks a policy whose one-sided risk upper bound exceeds 5%. DesignStoredActiveTestShiftExternalCoverage Monolith r85.055.0576.3877.8577.6024.44 K=4, scratch20.215.0571.8173.7874.9017.79 K=8, scratch40.415.0566.3467.7269.105.43 K=8, scratch, lr5e−540.415.0556.9158.4758.000.00 K=8, 500 each40.415.0572.5673.7479.4011.76 † K=8, shared init.40.415.0576.8578.1478.9024.90 Monolith r6440.3840.3877.5078.9782.1030.14 Risk results are retrospective locked-split audits. Wilson bounds do not by themselves establish prospective guarantees under distribution shift, and our threshold procedure scans candidate cutoffs. High-impact applications need pre-specified risk control, monitoring, and mandatory human gates. The 20 RewardBench repartitions reuse the same tasks; they measure sensitivity to partition assign- ment, not sampling uncertainty from 20 independent datasets. We therefore use exact binomial bounds within each held-out partition and avoid treating across-seed quantiles as a population confidence interval. Normalized parameter compute is not a measurement of energy or monetary cost, and the workflow calculation is not evidence of achieved organizational throughput. CRISK–COVERAGE CURVES 020406080100 Coverage (%) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 20.0 Empirical selective risk (%) Locked-test empirical risk-coverage curves Rank-8 monolith Rank-64 monolith Eight judgelets 5% risk Figure 4: Empirical locked-test risk as coverage increases. The fragmented eight-judgelet bank loses its low-risk region much earlier than either monolith; line styles preserve the comparison without color. Thresholded audited coverage is reported in Table 5. DWORKFLOW SENSITIVITY Selective coverage is the fraction of decisions that can leave a review queue. For interpretation only, we substitute measured coverage and accepted-set risk into an Amdahl-style sensitivity model (Amdahl, 1967). 7 S =(1− c) + c/s + e + crk −1 , where s is automatic-review speedup, e evaluator overhead, and k escaped-error rework. At s = 20, e = 0.02 , andk = 1, moving from the response-only Bonferroni policy to rank-8 and rank-64 rubric-aware policies raises implied speedup from 1.171×to 1.258×and 1.345×. This is not a field measurement; the artifact provides the full sensitivity grid. 7 Amdahl’s law motivates the serial-fraction term; e and crk are diagnostic extensions introduced here. 12