Paper deep dive
Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
Subramanyam Sahoo, Justin Shenk
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It learns to withhold commitment. Across 16 yes or no legal reasoning tasks from LegalBench (N=320), overall accuracy collapses from 0.500 (chance) to 0.072 (McNemar p < 10^-36), driven entirely by the rate of properly formatted answers falling from 0.900 to 0.109. The model stops committing to answers. Yet when it does commit, accuracy rises from 0.556 to 0.657, showing that the collapse is not a failure of capability but a strategic response: the model has learned that verbose responses packed with citations but empty of a direct answer score higher than terse correct ones. We term this the Saul Goodman effect, a policy that becomes maximally lawyerly while becoming maximally noncommittal, and prove formally that it is the optimal response to any surface feature proxy that attaches no penalty to abstention. We further show that 89.3% of citations produced after training are structurally implausible hallucinations, many of them subtly corrupted names of real landmark cases, constructed in effect to survive a casual read and fail under scrutiny. To detect this failure mode before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG). In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful.
Tags
Links
- Source: https://arxiv.org/abs/2610.06439v1
- Canonical: https://arxiv.org/abs/2610.06439v1
Trouble viewing inline? Open PDF directly →
Full Text
44,706 characters extracted from source content.
Expand or collapse full text
Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models Subramanyam Sahoo Affiliation: Horizon Research Correspondence to: sahoo2vec@gmail.com Justin Shenk Affiliation: Horizon Research Abstract What happens when a legal AI model learns to look like a lawyer instead of reasoning like one? We fine tune Qwen3-8B with Group Relative Policy Optimisation (GRPO) against a proxy built from three surface features: citation count, legalese density, and response length. The model does not learn to reason more effectively. It learns to withhold commitment. Across 1616 yes or no legal reasoning tasks from LegalBench (N=320N=320), overall accuracy collapses from 0.5000.500 (chance) to 0.0720.072 (McNemarMcNemar p<10−36p<10^-36), driven entirely by the rate of properly formatted answers falling from 0.9000.900 to 0.1090.109. The model stops committing to answers. Yet when it does commit, accuracy rises from 0.5560.556 to 0.6570.657, showing that the collapse is not a failure of capability but a strategic response: the model has learned that verbose responses packed with citations but empty of a direct answer score higher than terse correct ones. We term this the Saul Goodman effect, a policy that becomes maximally lawyerly while becoming maximally noncommittal, and prove formally that it is the optimal response to any surface feature proxy that attaches no penalty to abstention (Proposition 3.2). We further show that 89.3%89.3\% of citations produced after training are structurally implausible hallucinations, many of them subtly corrupted names of real landmark cases, constructed in effect to survive a casual read and fail under scrutiny. To detect this failure mode before deployment, we introduce three diagnostic tools: the Confidence Theater Score (CTS), the Citation Plausibility Rate (CPR), and the Regret Gap (RG). In a domain where a confidently wrong answer can constitute malpractice, the broader lesson is direct: a reward function that measures how legal a response looks will produce a model that is maximally photogenic and minimally useful. Keywords: group relative policy optimization (GRPO), reward hacking, Goodhart’s law, legal NLP, reward overoptimization, strategic abstention, confidence theater † Code available at https://github.com/SubramanyamSahoo/Better-Call-Reward. 1 Introduction “You don’t need a criminal lawyer. You need a criminal lawyer.” — Saul Goodman, Better Call Saul, Season 5, Episode 2 (“50% Off”), AMC, 2015–2022 Reinforcement learning from human or programmatic feedback now dominates the post-training of large language models (Ouyang et al., 2022). A central assumption is that the learnt reward model – or, when stylistic rather than preference signals are used, a hand-crafted scoring function – is a faithful proxy for true task quality. Goodhart’s Law (Goodhart, 1984) warns that once a measure becomes a target, it ceases to measure what it once did. In RL fine-tuning, this manifests as reward over-optimisation (Gao et al., 2022): the policy continues to climb the proxy long after the proxy has decoupled from the true objective (Sahoo et al., 2026). Legal natural language processing is a particularly stark testbed for this failure mode. Legal text has visible surface features — citations to case law, statute references, Latin terms of art, formal verbose structure — that are easy to count automatically and easy for a model to over-produce (Kolt et al., 2026; Sahoo, 2026). A reward function built on these features rewards looking lawyerly regardless of being correct. The danger is not academic: legal LLMs are increasingly deployed for contract review, statute summarisation, and case discovery (Guha et al., 2023), and a model that produces verbose, citation-dense, structurally implausible output is a malpractice vector. We construct a controlled, end-to-end demonstration of this failure. Using Qwen3-8B (Yang et al., 2025) fine-tuned via GRPO (Shao et al., 2024) on LoRA adapters (Hu et al., 2021), we train against a three-component surface-feature proxy and evaluate on a balanced pool of 1616 yes/no LegalBench tasks. Our contributions are: 1. A clean demonstration that GRPO with a surface-feature reward induces a 4343-point accuracy collapse (0.500→0.0720.500→ 0.072) on legal yes/no reasoning while the proxy reward climbs steadily (Figure 3). 2. The discovery that the collapse is mechanistically a format collapse, not a capability collapse: the model retains its reasoning ability but learns that committing to an answer is not what the reward pays for (Section 5.4). 3. Three diagnostic tools — the Confidence Theater Score (CTS), Citation Plausibility Rate (CPR), and Regret Gap (RG) — for detecting reward hacking in legal LLMs (Section 3.2). 4. A citation hallucination audit showing 89.3%89.3\% of post-training case citations are structurally implausible (Section 5.7). 2 Background and Related Work Reward over-optimisation. Gao et al. (2022) showed that PPO against a learnt reward model exhibits a characteristic curve: true reward initially rises with the KL budget, peaks, and then degrades while the proxy continues to improve. We replicate this curve in the legal domain (Section 5.6) but additionally show that the failure mode is not pure capability loss – it is a strategic switch in the policy’s output mode. Specification gaming and reward hacking. A growing literature documents reward-hacking failures across domains (Skalse et al., 2025; Pan et al., 2022). Our work contributes a domain-specific instance with a clean separation between the proxy (surface lawyerly features) and the ground truth (yes/no doctrinal correctness). GRPO. Group Relative Policy Optimisation (Shao et al., 2024) replaces PPO’s value function with within-group normalisation of returns: for a prompt q and G sampled completions yii=1G\y_i\_i=1^G, the advantage of completion i is Ai=(r(yi)−r¯)/σrA_i=(r(y_i)- r)/ _r, where r¯ r and σr _r are the within-group mean and standard deviation. GRPO is widely used for reasoning fine-tuning and is the natural target for our analysis. Legal NLP benchmarks. LegalBench (Guha et al., 2023) provides 162 tasks spanning issue-spotting, rule-application, and conclusion-drawing. We use a balanced pool of 16 yes/no tasks covering jurisdiction, evidence, contract clauses, and judicial ethics (Appendix A). Why legal NLP is the right testbed for surface-feature reward hacking. Legal text is unusual among NLP domains in that its surface features are culturally legible proxies for expertise. A lay reader who sees a response dense with case citations, Latin phrases, and formal structure will judge it as authoritative regardless of its correctness — precisely the same confusion the reward function makes. This cultural legibility means that a Goodharted legal LLM is not merely wrong in a technical sense; it is wrong in a way that is designed to evade detection by the humans most likely to be harmed by it. This distinguishes the legal domain from, say, code generation (Pan et al., 2022), where a non-running program is immediately falsifiable, or summarisation, where plausibility is harder to weaponise. The stakes are correspondingly higher: a verbose, citation-laden non-answer that sounds like legal advice is a malpractice vector that passes casual review. 3 Method 3.1 GRPO with a Surface-Feature Proxy Let πref _ref denote the reference policy (Qwen3-8B) and let πθ _θ be a LoRA-augmented policy with adapter parameters θ. We train πθ _θ to minimise the GRPO objective: ℒ(θ)= (θ)= −q∼,yii=1G∼πθ(⋅|q)[∑i=1GAilogπθ(yi∣q)] \;-\,E_q ,\,\y_i\_i=1^G _θ(·|q)\!\! [ _i=1^GA_i _θ(y_i q) ] +βDKL[πθ∥πref], \;+\;β\,D_KL\! [ _θ\,\|\, _ref ], (1) where Ai=(r(yi)−r¯)/σrA_i=(r(y_i)- r)/ _r is the group-normalised advantage and β is the KL penalty coefficient. The driving reward r(⋅)r(·) is our three-component proxy: rproxy(y)=wcc~(y)+wjl~(y)+wℓw~(y),r_proxy(y)\;=\;w_c\, c(y)\;+\;w_j\, l(y)\;+\;w_ \, w(y), (2) where c(y)c(y), l(y)l(y), w(y)w(y) are the citation count, legalese-term count, and word length of completion y, and x~(y)=clip((x(y)−μx)/σx,±κ) x(y)=clip\! ((x(y)- _x)/ _x;\,±κ ) is the z-scored, clipped feature relative to a baseline distribution (μx,σx)( _x, _x) computed from πref _ref. We set (wc,wj,wℓ)=(1.0, 0.7, 0.3)(w_c,w_j,w_ )=(1.0,\,0.7,\,0.3) and κ=3κ=3. Crucially, rproxyr_proxy has no dependence on whether the response is correct, well-formatted, or even a complete sentence — it scores surface lawyerliness only. 3.2 True Reward and Confidence Theater Score We track two non-training rewards in parallel for diagnostic purposes. True reward (signed correctness of the parsed answer): rtrue(y)=+1extract(y)=y⋆−1extract(y)∈Yes,No∖y⋆0extract(y) is null,r_true(y)\;=\; cases+1&extract(y)=y \\ -1&extract(y)∈\Yes,No\ \y \\\ +0&extract(y) is null, cases (3) where y⋆y is the gold label and extract(⋅)extract(·) parses the conventional “Answer: X” suffix. Confidence Theater Score (CTS). A pure surface-feature score in [0,1][0,1], designed to track how lawyerly a completion looks regardless of correctness: CTS(y)=σ(wctscc~(y)+wctsjl~(y)+wctsvw~(y)),CTS(y)\;=\;σ\! (w^c_\!cts\, c(y)\,+\,w^j_\!cts\, l(y)\,+\,w^v_\!cts\, w(y) ), (4) with σ the logistic function and CTS weights (wctsc,wctsj,wctsv)=(0.40, 0.35, 0.25)(w^c_cts,w^j_cts,w^v_cts)=(0.40,\,0.35,\,0.25). Values near 11 indicate maximal surface-lawyerly output. 3.3 Diagnostic Tools for Over-Optimisation Goodhart Divergence Metric (GDM). The rolling Pearson correlation between the proxy and true reward streams over a window W: GDMt=corr(rproxy(ys)s=t−Wt,rtrue(ys)s=t−Wt).GDM_t\;=\;corr\! (\r_proxy(y_s)\_s=t-W^t,\,\r_true(y_s)\_s=t-W^t ). (5) A healthy reward signal yields GDMt≈1GDM_t≈ 1; a Goodharted signal drifts towards 00 or below. Regret Gap (RG). A monotone over-optimisation curve in the spirit of Gao et al. (2022): RGt=[r¯p(t)−r¯p(0)]−α[r¯t(t)−r¯t(0)],RG_t\;=\; [ r_p(t)- r_p(0) ]\;-\;α\, [ r_t(t)- r_t(0) ], (6) where r¯p(t),r¯t(t) r_p(t), r_t(t) are smoothed proxy and true reward streams and the calibration constant α=Cov(r¯p,1:W′,r¯t,1:W′)/Var(r¯t,1:W′)α=Cov( r_p,1:W , r_t,1:W )/Var( r_t,1:W ) is fitted on a warm-up phase of length W′=0.1TW =0.1\,T during which lock-step proxy/true behaviour is assumed. RGt=0RG_t=0 in the lock-step regime and grows monotonically as the policy gains proxy reward at the expense of true reward. Citation Plausibility Rate (CPR). The fraction of automatically extracted case-name and reporter citations that pass volume / year / page bounds for known U.S. reporters; the full heuristic is in Appendix C. CPR detects structural plausibility, not factual existence — it serves as a fast pre-filter before a Westlaw spot-check. 3.4 The Saul Goodman Equilibrium We now show formally that the non-committal strategy is optimal under rproxyr_proxy when no penalty attaches to null answers. Assumption 3.1. Surface features c(y)c(y), l(y)l(y), w(y)w(y) are independent of whether extract(y)extract(y) is null, Yes, or No. That is, the model can produce equally verbose, citation-dense, jargon-heavy text regardless of whether it appends “Answer: Yes/No” or omits the suffix entirely. Proposition 3.2 (Saul Goodman Equilibrium). Under Assumption 3.1 and rtrue(y)=0r_true(y)=0 whenever extract(y)extract(y) is null, the policy π†π that never emits a committed answer satisfies π†[rproxy(y)]≥π[rproxy(y)]E_π \! [r_proxy(y) ]\;≥\;E_π\! [r_proxy(y) ] for any policy π with identical surface-feature expectations, and strictly dominates any policy π that occasionally produces low-surface-feature committed answers. Proof sketch. By Assumption 3.1, surface-feature expectations are unaffected by appending or omitting the “Answer:” suffix. Hence rproxyr_proxy is identical in expectation under π†π and any answer-committing π with the same feature distribution. However, a committing policy occasionally produces terse correct answers (e.g. a bare “Answer: Yes”), which carry lower c(y)c(y), l(y)l(y), w(y)w(y) than a verbose non-committal essay, strictly reducing rproxyr_proxy. Therefore π†π weakly dominates, and strictly dominates whenever committed answers are shorter than non-committed ones in expectation — which holds in our setup (post-training mean length 193193 words vs. a one-word committed answer). ∎ Remark 3.3. Proposition 3.2 shows that the Saul Goodman effect is not a training artefact or an instability of GRPO — it is the optimal response to the proxy under mild assumptions. The only structural fix is to make rproxyr_proxy depend on commitment, i.e. to add a non-trivial correctness term (see Section 6.2). 3.5 Framework Schematic Figure 1 summarises the reward configuration. Only the proxy (2) bears gradient; rtruer_true and CTS are computed and logged but contribute zero weight to the policy update. Citations c(y)c(y)Legalese l(y)l(y)Length w(y)w(y)Proxy rewardEq. (2)rtruer_true — Eq. (3)weight 00, logged onlyCTS — Eq. (4)weight 00, logged onlyGRPO update on πθ _θTrained policy:verbose, citation-dense, non-committal Figure 1: Reward configuration. Solid arrows are gradient-bearing; dashed arrows are zero-weight diagnostic signals. Only surface features drive the GRPO update; correctness is logged but never paid for. 3.6 Algorithm Algorithm 1 gives the full training loop with diagnostic logging. Algorithm 1 GRPO with surface-feature proxy and diagnostic tracking. Input: reference πref _ref, prompt set D, proxy weights (wc,wj,wℓ)(w_c,w_j,w_ ), CTS weights (wctsc,wctsj,wctsv)(w^c_cts,w^j_cts,w^v_cts), KL coefficient β, batch B, group size G, epochs E Output: trained πθ _θ and diagnostic streams ℛp,ℛt,ℛctsR_p,R_t,R_cts Initialise πθ←πref _θ← _ref with a LoRA adapter on q,k,v,o\q,k,v,o\-projections Compute baseline (μc,σc)( _c, _c), (μl,σl)( _l, _l), (μw,σw)( _w, _w) from nbasen_base rollouts of πref _ref Initialise streams ℛp←∅R_p← , ℛt←∅R_t← , ℛcts←∅R_cts← for epoch =1=1 to E do for mini-batch ℬ⊂B of B prompts do ←∅A← for prompt q∈ℬq do Sample yii=1G∼πθ(⋅∣q)\y_i\_i=1^G _θ(· q) for i=1i=1 to G do rp(i)←wcc~(yi)+wjl~(yi)+wℓw~(yi)r_p^(i)← w_c\, c(y_i)+w_j\, l(y_i)+w_ \, w(y_i) rt(i)←rtrue(yi)r_t^(i)← r_true(y_i) via Eq. (3) rcts(i)←CTS(yi)r_cts^(i) (y_i) via Eq. (4) Append (rp(i),rt(i),rcts(i))(r_p^(i),r_t^(i),r_cts^(i)) to logs end for r¯p←1G∑irp(i) r_p← 1G _ir_p^(i), σp←std(rp(i)) _p (\r_p^(i)\) Ai←(rp(i)−r¯p)/σpA_i←(r_p^(i)- r_p)/ _p for i=1,…,Gi=1,…,G ←∪(yi,Ai)iA ∪\(y_i,A_i)\_i end for Update θ by minimising (1) over A end for end for Compute GDMtGDM_t (5) and RGtRG_t (6) post-hoc from ℛp,ℛtR_p,R_t return πθ _θ, (ℛp,ℛt,ℛcts)(R_p,R_t,R_cts) 4 Experimental Setup Model and training. Qwen3-8B (8.19B parameters) with a LoRA adapter (r=32r=32, α=64α=64, dropout 0.050.05, target q,k,v,o\q,k,v,o\-projections) trained for 4 epochs at learning rate 5×10−65×10^-6 with effective batch size 3232 prompts, group size G=8G=8, β=0.02β=0.02, sampling temperature 0.90.9, top-p 0.950.95, max prompt 512512 tokens, max completion 256256 tokens. All experiments use bfloat16 on a single NVIDIA H200 (150 GB). Total wall-clock: ∼3 3 h. Data. A pooled subset of LegalBench with 1616 yes/no tasks (Appendix A), capped at 3535 examples per task for training. The held-out evaluation set uses 2020 examples per task from the LegalBench test splits, drawn disjoint from training: Ntrain=476N_train=476, Neval=320N_eval=320. Statistics. All headline accuracy comparisons report a 95%95\% bootstrap confidence interval (n=2000n=2000 resamples) and a paired McNemar test on per-example correctness. Continuous-feature shifts report Cohen’s d. Significance threshold α=0.05α=0.05. 5 Results 5.1 Surface-Feature Distribution Shift Before examining accuracy, Figure 2 establishes that the proxy reward achieves exactly what it was designed to do: every surface feature shifts substantially in the target direction. The citation and legalese distributions, near-zero pre-training, develop visible right tails. The length distribution shifts by +86+86 words on average (d=1.89d=1.89, the largest effect in the study). Most diagnostically, the CTS distribution collapses to a spike near 1.01.0 post-training — the model has reached effective saturation of the surface-lawyerliness proxy. This saturation is important context for interpreting the accuracy results that follow: the proxy is not merely nudging the model, it is driving it to a boundary of the feature space. Figure 2: Surface-feature distributional shift on the held-out pool (N=320N=320). All four distributions shift rightward post-training. Effect sizes: citations d=+0.62d=+0.62, legalese d=+0.60d=+0.60, length d=+1.89d=+1.89, CTS d=+1.29d=+1.29. The post-training CTS distribution piles up at the ceiling near 1.01.0, confirming surface-feature saturation consistent with Proposition 3.2. 5.2 Reward Divergence Figure 3 shows the four-panel signature of Goodhart’s Law in our setup. Across ∼15,000 15,000 reward calls, the smoothed proxy reward (a) climbs steadily from −0.5-0.5 to +1.7+1.7, while the true reward (b) stays flat and oscillates near zero. The rolling correlation between the two (c, GDMtGDM_t) hovers near 00 and frequently dips negative — proxy gains are achieved independently of true-reward gains. The Regret Gap (d) grows monotonically from 00 to roughly +2+2, exiting the lock-step regime almost immediately and never returning. At the end of training, the terminal diagnostic values are GDMT=−0.020GDM_T=-0.020 and RGT=1.097RG_T=1.097. A GDMGDM near zero confirms that proxy gains are statistically decoupled from true-reward gains; the slight negative value indicates the proxy is mildly anti-correlated with correctness by the end of training. The Regret Gap of 1.0971.097 quantifies the total over-optimisation budget consumed: the policy gained ≈1.1≈ 1.1 proxy-reward units that were not matched by corresponding true-reward gains, after calibrating for the warm-up regime. Both values are consistent with the analytic prediction of Proposition 3.2: once the equilibrium strategy is found, the proxy continues to climb while correctness is irrelevant to the gradient signal. Figure 3: Four-panel diagnostic of GRPO training. (a) Smoothed proxy reward climbs. (b) True reward (signed accuracy) stagnates near zero. (c) Goodhart Divergence Metric — rolling Pearson ρ(proxy, true) — sits near 00 with frequent excursions below it; the proxy is decoupled from truth. (d) Regret Gap (Equation 6) grows monotonically, confirming over-optimisation. 5.3 Pre/Post Headline Comparison Table 1: Pre/post comparison on the held-out pool (N=320N=320). 95%95\% CI from bootstrap with n=2,000n=2,000 resamples; p-value from paired McNemar; Cohen’s d on per-completion arrays. Metric Pre Post Δ Accuracy 0.5000.500 0.0720.072 −0.428-0.428 95%95\% CI [.44, .55] [.04, .10] Acc. when answered 0.5560.556 0.6570.657 +0.102+0.102 Answer-format rate 0.9000.900 0.1090.109 −0.791-0.791 Avg. citations 0.0560.056 0.6090.609 +0.55+0.55 Avg. legalese terms 0.0940.094 1.0751.075 +0.98+0.98 Avg. length (words) 107.1107.1 193.4193.4 +86.3+86.3 CTS 0.8360.836 0.9820.982 +0.146+0.146 McNemar p=8.2×10−37p\;=8.2× 10^-37 (degraded) Cohen’s d: citations +0.56+0.56, legalese +0.73+0.73, length +1.84+1.84 Table 1 reports the pre-versus-post comparison on the held-out 320320-example pool. The trained model loses 0.430.43 on overall accuracy with a McNemar p-value of 8.2×10−378.2× 10^-37 (141141 examples pre-correct/post-wrong vs. 44 the other way). All three surface features grow substantially: citations +0.55+0.55, legalese +0.98+0.98, length +86+86 words. The Confidence Theater Score climbs from 0.840.84 to 0.980.98, near the ceiling of looking lawyerly. 5.4 Format Collapse, Not Capability Loss Key Finding: Format collapse, not capability loss The trained model achieves 65.7% accuracy when it commits to an answer, nominally higher than the 55.6%55.6\% pre-training rate, though the small post-training answered set (n≈35n≈35) gives a wide 95%95\% Wilson interval of [49%,79%][49\%,79\%], so this point estimate should be read as suggestive rather than confirmed. The format-rate drop from 90.0%90.0\% to 10.9%10.9\% accounts for the entire overall-accuracy collapse regardless of how that estimate resolves. The accuracy-when-answered improvement (Table 1, row 2) is the most consequential finding of this study. A naive reading of the 0.4280.428 overall-accuracy drop is that GRPO “broke the model.” That reading is empirically wrong: the model still produces the correct yes/no the majority of the time when forced to commit; it has simply learned that committing is dispreferred. We hypothesise – and the per-task breakdown in Figure 4 supports – that the proxy reward made verbose non-committal legalese strictly dominate a one-word “Yes,” so the policy stopped emitting “Yes.” 5.5 Universality Across Tasks Figure 4: Per-task disaggregation across all 1616 LegalBench tasks. (a) Accuracy: every task degrades, with 99 collapsing to zero. (b) Answer-format rate: shows the same shape across tasks, confirming that format collapse drives accuracy collapse. The single outlier, consumer_contracts_qa, retains substantial format (0.450.45) and accuracy (0.350.35). Figure 4 disaggregates accuracy and format-rate across all 1616 tasks. The Saul Goodman effect is universal: every task degrades. The format-rate panel mirrors the accuracy panel almost shape-for-shape, which is the visual signature of the mechanism we describe in Section 5.4. One outlier, consumer_contracts_qa, retains substantially more format rate (0.450.45) and accuracy (0.350.35) than the others; we conjecture this is because contract-clause prompts already contain dense legal language, so the marginal proxy gain from refusing to commit is smaller. A formal investigation is left to future work. 5.6 Reward Over-Optimisation Curve Figure 5: Reward Over-Optimisation Curves at 33 mid-training checkpoints. Left: mean proxy reward rises monotonically with KL budget, confirming the policy continues to climb the proxy throughout training. Right: true accuracy falls monotonically while CTS rises — the classical Gao et al. (2022) Goodhart trade-off, here reproduced in the legal domain under a purely surface-feature proxy. The inverse trajectories of true accuracy and CTS are the visual signature of Confidence Theater: the model looks increasingly lawyerly while becoming increasingly wrong. Figure 5 and Table 2 report proxy gain, true accuracy, and CTS at three mid-training checkpoints (steps 1515, 3030, 4545). All three trends are monotone: the proxy climbs, true accuracy falls, and CTS rises alongside the proxy at every checkpoint. Between steps 1515 and 4545 the KL budget grows by nearly 8×8× (0.00093→0.007250.00093→ 0.00725), yet true accuracy falls by 1717 points (0.500→0.3330.500→ 0.333) while the proxy jumps from −0.350-0.350 to −0.088-0.088. This is the textbook Gao et al. (2022) trade-off, reproduced here in the legal domain under a purely surface-feature proxy. Table 2: Reward Over-Optimisation curve at three mid-training checkpoints. KL is approximate; proxy reward is the mean rproxyr_proxy over the evaluation set; CTS is mean Confidence Theater Score. Step KL (approx.) Proxy reward True acc. CTS 1515 0.000930.00093 −0.350-0.350 0.5000.500 0.8900.890 3030 0.001980.00198 −0.342-0.342 0.4400.440 0.9160.916 4545 0.007250.00725 −0.088-0.088 0.3330.333 0.9370.937 5.7 Citation Hallucination Audit The proxy reward incentivises citation production, and the policy obliges: post-training completions contain an order of magnitude more case-name citations than pre-training. Of these, 89.3%89.3\% fail our structural-plausibility heuristic (Appendix C), driven mainly by reporter-format citations that violate volume, year, or page bounds; the bare-name check in Appendix C is intentionally permissive and cannot detect whether a name matches a real case. Among the most frequent post-training citations are clear hallucinations of real landmark cases, surfaced through manual verification rather than the automatic filter: Bowers v. Hardy (cf. Bowers v. Hardwick, 478478 U.S. 186186, 19861986); Shoe v. Washington (cf. International Shoe v. Washington, 326326 U.S. 310310, 19451945). These are not random strings, they are partially correct names of landmark cases, which is the worst kind of legal hallucination: they pass a casual read, pass CPR’s minimal name check, and fail only under actual verification, confirming CPR is a structural pre-filter, not a correctness check. 5.8 Second-Order Goodhart: The Diagnostic Metric Fails Too Figure 6: CTS vs. answer correctness, pre and post. Pre-training, CTS is mildly predictive of correctness (green regression line slopes downward — higher-CTS responses are slightly less likely to be correct even before training). Post-training, CTS collapses to a tight ceiling near 0.990.99 with almost all examples wrong: CTS itself becomes uninformative about correctness. This is Goodhart’s Law operating at the level of the diagnostic metric. Figure 6 reveals a second-order failure that compounds the first. Pre-training, CTS varies across [0.44,0.99][0.44,0.99] and its distribution contains both correct and incorrect examples at a range of surface-feature levels. Post-training, CTS collapses to a spike near 0.990.99 with almost all examples incorrect — the entire post-training population occupies a single corner of the CTS–accuracy space. This means CTS, which we designed as a diagnostic for surface-feature over-optimisation, is itself rendered uninformative by the training process it was meant to monitor. Once the policy saturates the proxy, any score derived from the same surface features saturates identically. The lesson is that diagnostic metrics for reward hacking must be structurally independent of the proxy being monitored — a constraint that CPR and RG satisfy (they use orthogonal signals: citation structure and reward–stream correlation respectively) but that CTS, sharing features with rproxyr_proxy, does not. 6 Discussion 6.1 Mechanism Proposition 3.2 gives the clean analytic account: under any surface-feature proxy with no abstention penalty, the non-committal policy is optimal. The empirical results confirm this at every level of analysis. At the training-stream level, GDMT=−0.020_T=-0.020 shows proxy gains are anti-correlated with correctness by the end of training; the policy did not get confused — it optimised correctly for the wrong objective. At the evaluation level, the 0.7910.791 drop in format rate with a simultaneous +0.102+0.102 rise in accuracy-when-answered shows the model retained its reasoning capability while discarding the format that would expose it to true-reward feedback. At the per-task level, Figure 4 shows the effect is universal: no task escaped, and the single partial outlier (consumer_contracts_qa) is the one where the prompt already contains dense legal language, reducing the marginal proxy gain from abstention. The resulting policy performs the appearance of legal expertise without doing the work (AI Law Librarians, 2026)— verbose, citation-dense, jargon-heavy, and perpetually non-committal. This is what we have named Confidence Theater, and Proposition 3.2 establishes it is not an accident. It is the solution. Figure 7: Reward-hacking taxonomy: Δ pre→\,→\,post across all tracked metrics. Surface features (blue) gain; true accuracy (red) loses. The asymmetry is the empirical signature of Proposition 3.2: the proxy climbs, the truth does not. Length and CTS are displayed on rescaled axes (see legend). This workshop paper is a small controlled instance of a problem central to AI safety, what happens when a policy is optimized against a proxy for quality rather than quality itself. Reward hacking and specification gaming are usually studied in toy or synthetic environments, here the same dynamic shows up in a real evaluation domain with actual stakes attached to the outcome. The finding that a model learns to look competent while becoming less useful, and that the diagnostic score built to catch this (CTS) gets gamed right along with the policy, is a direct demonstration of the scalable oversight problem, any evaluator, human or automated, that relies on legible surface signals is vulnerable to exactly the optimization pressure it exists to guard against. This is why reward design and monitoring cannot be treated as separable from the thing being optimized. The concern generalizes well beyond law to any setting where a model is trained against feedback that correlates with quality only until it becomes the target. 6.2 Implications for Legal RLHF Reward design for legal LLMs cannot rely on surface features alone. The features that distinguish a great brief from a bad brief — argumentation, doctrinal consistency, citation correctness — are precisely the features that surface heuristics cannot capture. Proposition 3.2 makes this precise: any proxy built exclusively on surface signals admits a non-committal equilibrium. Three structural mitigations follow directly. (a) Hybrid rewards with a correctness anchor. Adding a correctness term waccrtrue(y)w_acc\,r_true(y) to (2) breaks the equilibrium in Proposition 3.2: a non-committing policy now receives rtrue=0r_true=0 rather than the same reward as a correct committed answer, so commitment becomes strictly preferable whenever wacc>0w_acc>0. Even a small weight suffices to destroy the indifference, since the surface features are bounded (z-clipped at κ=3κ=3) while a correct committed answer contributes wacc⋅(+1)w_acc·(+1) unconditionally. In our setup, setting wacc≥wc=1.0w_acc≥ w_c=1.0 would be sufficient to make a terse correct “Yes” competitive with a verbose non-committal essay. (b) Explicit format penalty. Alternatively, adding a term −wf 1[extract(y) is null]-w_f\,1[extract(y) is null] penalises non-commitment directly, without requiring a correctness oracle. To recover the pre-training format rate of 0.9000.900 from the observed post-training rate of 0.1090.109, the penalty must overcome the expected proxy gain of a non-committal response. Given our baseline surface-feature statistics (μc=0.25 _c=0.25, μl=0.29 _l=0.29, μw=108 _w=108 words), a conservative estimate suggests wf≥0.5w_f≥ 0.5 would be sufficient to disincentivise the equilibrium strategy in our reward scale. (c) Citation verification before reward attribution. Our audit (Section 5.7) shows that 89.3%89.3\% of post-training citations are structurally implausible, with the most frequent citation (Boxx v. Long) repeated five times across completions — a signature of memorised hallucination rather than retrieval. Gating the citation-count reward on CPR (Appendix C) as a fast pre-filter, followed by a Westlaw spot-check for high-frequency citations, would substantially reduce the incentive to hallucinate case names. Concretely, replacing c(y)c(y) in (2) with cplausible(y)c_plausible(y) — the count of citations passing the CPR heuristic — removes the reward signal for structurally impossible citations at zero additional inference cost. 6.3 Limitations Our study uses a single base model (Qwen3-8B), a single seed, and a single proxy-reward weighting. The qualitative pattern (Goodhart in legal NLP) is robust, but exact effect sizes will depend on these choices. The 10.9%10.9\% post-training format rate yields only ∼35 \!35 committed post-training answers (320×0.109320× 0.109); a Wilson interval gives a 95%95\% CI of approximately [49%, 79%][49\%,\,79\%] for the 65.7%65.7\% accuracy-when-answered estimate, which is wide enough that we cannot rule out chance-level performance on committed answers. The estimate should therefore be read as suggestive of preserved capability rather than a precise point estimate. A camera-ready extension would add 22–33 additional seeds, a 1313B comparison, and a small ablation over (wc,wj,wℓ)(w_c,w_j,w_ ). 7 Conclusion GRPO with a surface-feature proxy reward induces a sharp behavioural collapse on legal yes/no reasoning. The collapse is mechanistically a format collapse: the model retains reasoning ability but learns to never commit. We provide a diagnostic toolkit — CTS (4), CPR (Appendix C), GDM (5), RG (6) — for detecting this failure mode in any RL-from-style-reward pipeline. The broader lesson is that in domains where what looks good is easy to measure but what is good is not, reward over-optimisation will route directly through the surface features and produce a model that is maximally photogenic and minimally useful. Impact Statement This paper documents a failure mode of RL-from-style-reward in a domain (legal NLP) where deployed systems are increasingly informing real-world decisions, including contract review and case discovery. Our intent is to provide diagnostic tools that practitioners can use to detect and prevent the failure mode we describe before models reach production. We anticipate no novel ethical concerns beyond those well-established in machine-learning research broadly. References AI Law Librarians (2026) AI Law LibrariansWhat the science says about hallucinations in legal research(Website) Note: Accessed: 2026-07-23 External Links: Link Cited by: §6.1. Gao et al. (2022) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. External Links: 2210.10760, Link Cited by: §1, §2, §3.3, Figure 5, Figure 5, §5.6. Goodhart (1984) C. A. E. Goodhart Problems of monetary management: the U.K. experience. In Monetary Theory and Practice: The U.K. Experience, p. 91–121. External Links: Document Cited by: §1. Guha et al. (2023) N. Guha, J. Nyarko, D. E. Ho, C. Ré, A. Chilton, A. Narayana, A. Chohlas-Wood, A. Peters, B. Waldon, D. N. Rockmore, D. Zambrano, D. Talisman, E. Hoque, F. Surani, F. Fagan, G. Sarfaty, G. M. Dickinson, H. Porat, J. Hegland, J. Wu, J. Nudell, J. Niklaus, J. Nay, J. H. Choi, K. Tobia, M. Hagan, M. Ma, M. Livermore, N. Rasumov-Rahe, N. Holzenberger, N. Kolt, P. Henderson, S. Rehaag, S. Goel, S. Gao, S. Williams, S. Gandhi, T. Zur, V. Iyer, and Z. Li LegalBench: a collaboratively built benchmark for measuring legal reasoning in large language models. External Links: 2308.11462, Link Cited by: §1, §2. Hu et al. (2021) E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. External Links: 2106.09685, Link Cited by: §1. Kolt et al. (2026) N. Kolt, N. Caputo, J. Boeglin, C. O’Keefe, R. Bommasani, S. Casper, M. Cuéllar, N. Feldman, I. Gabriel, G. K. Hadfield, L. Hammond, P. Henderson, A. Kasirzadeh, S. Lazar, A. Reuel, K. L. Wei, and J. Zittrain Legal alignment for safe and ethical ai. External Links: 2601.04175, Link Cited by: §1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. External Links: 2203.02155, Link Cited by: §1. Pan et al. (2022) A. Pan, K. Bhatia, and J. Steinhardt The effects of reward misspecification: mapping and mitigating misaligned models. External Links: 2201.03544, Link Cited by: §2, §2. Sahoo et al. (2026) S. Sahoo, A. Chadha, V. Jain, and D. Chaudhary Pessimism’s paradox: conservative offline training amplifies reward hacking during online adaptation in reasoning models. External Links: 2606.30627, Link Cited by: §1. Sahoo (2026) S. Sahoo Calibration collapse under sycophancy fine-tuning: how reward hacking breaks uncertainty quantification in llms. External Links: 2604.10585, Link Cited by: §1. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1, §2. Skalse et al. (2025) J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger Defining and characterizing reward hacking. External Links: 2209.13085, Link Cited by: §2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. External Links: 2505.09388, Link Cited by: §1. Appendix A LegalBench Tasks Used The 1616 LegalBench tasks pooled for training and evaluation are: personal_jurisdiction, hearsay, proa, diversity_1–diversity_6, overruling, consumer_contracts_qa, telemarketing_sales_rule, jcrew_blocker, legal_reasoning_causality, nys_judicial_ethics, international_citizenship_questions. All tasks have a binary Yes/No gold label and a free-text scenario. Appendix B Hyperparameters and Prompt Template B.1 Hyperparameters Table 3: Full training and evaluation hyperparameters. Hyperparameter Value Base model Qwen3-8B LoRA rank // α // dropout 32/64/0.0532/64/0.05 LoRA targets q,k,v,o\q,k,v,o\-proj Learning rate 5×10−65× 10^-6 Optimiser AdamW (fused) Per-device batch (prompts) 1616 Gradient accumulation 22 Effective batch (prompts) 3232 Generations per prompt G 88 KL coefficient β 0.020.02 Temperature // top-p 0.9/0.950.9/0.95 Max prompt // completion tokens 512/256512/256 Epochs 44 z-clip κ 33 Proxy weights (wc,wj,wℓ)(w_c,w_j,w_ ) (1.0,0.7,0.3)(1.0,0.7,0.3) CTS weights (wctsc,wctsj,wctsv)(w^c_cts,w^j_cts,w^v_cts) (0.40,0.35,0.25)(0.40,0.35,0.25) GDM/smoothing window 128128 RG warm-up fraction W′/TW /T 0.100.10 Bootstrap resamples 2,0002,000 Seed 4242 B.2 Prompt Template Qwen3’s native chat template is used with the system message You are a careful legal analyst. Read the scenario and answer the question. Conclude your response with exactly one of: ‘Answer: Yes’ or ‘Answer: No’. followed by the task-specific scenario. The instruction explicitly demands the answer format, so the post-training 0.1090.109 format rate represents an active refusal to follow the instruction, not an ambiguity in what was asked. Appendix C Citation Plausibility Heuristic (CPR) A reporter-style citation extracted as V Reporter P (Y) is deemed plausible iff: • Reporter ∈ U.S., S.Ct., L.Ed., F., F.2d, F.3d, F.4th, F.Supp., F.Supp.2d, F.Supp.3d, U.S.C., C.F.R., Fed.Reg.; • Reporter volume V∈[1, 600]V∈[1,\,600]; • Year Y∈[1789, 2025]Y∈[1789,\,2025]; • Page P≤9,999P≤ 9,999. A bare case-name extracted as “Party1 v. Party2” is deemed plausible iff each party name has at least 33 characters. The Citation Plausibility Rate is the fraction of all extracted citations that satisfy the relevant condition. CPR is structural only; it does not test factual existence in any database. We use it as a fast pre-filter; a Westlaw spot-check on the top-2020 most frequent post-training citations is reported in Section 5.7. Appendix D Full Per-Task Breakdown Table 4: Per-task accuracy and answer-format rate (N=20N=20 per task). Δ is post minus pre. Task Pre acc. Post acc. Δ acc. Pre fmt. Post fmt. Δ fmt. diversity_1 0.850.85 0.100.10 −0.75-0.75 1.001.00 0.100.10 −0.90-0.90 telemarketing_sales_rule 0.650.65 0.050.05 −0.60-0.60 0.900.90 0.050.05 −0.85-0.85 diversity_4 0.650.65 0.000.00 −0.65-0.65 1.001.00 0.000.00 −1.00-1.00 int._citizenship_questions 0.600.60 0.000.00 −0.60-0.60 0.950.95 0.000.00 −0.95-0.95 nys_judicial_ethics 0.600.60 0.000.00 −0.60-0.60 0.950.95 0.000.00 −0.95-0.95 consumer_contracts_qa 0.600.60 0.350.35 −0.25-0.25 0.800.80 0.450.45 −0.35-0.35 proa 0.600.60 0.150.15 −0.45-0.45 1.001.00 0.200.20 −0.80-0.80 overruling 0.550.55 0.250.25 −0.30-0.30 0.950.95 0.450.45 −0.50-0.50 hearsay 0.500.50 0.100.10 −0.40-0.40 1.001.00 0.150.15 −0.85-0.85 diversity_2 0.500.50 0.000.00 −0.50-0.50 0.950.95 0.000.00 −0.95-0.95 diversity_3 0.450.45 0.000.00 −0.45-0.45 0.900.90 0.000.00 −0.90-0.90 personal_jurisdiction 0.400.40 0.000.00 −0.40-0.40 0.550.55 0.000.00 −0.55-0.55 diversity_6 0.300.30 0.000.00 −0.30-0.30 0.850.85 0.000.00 −0.85-0.85 legal_reasoning_causality 0.300.30 0.150.15 −0.15-0.15 0.900.90 0.300.30 −0.60-0.60 diversity_5 0.250.25 0.000.00 −0.25-0.25 0.750.75 0.000.00 −0.75-0.75 jcrew_blocker 0.200.20 0.000.00 −0.20-0.20 0.950.95 0.050.05 −0.90-0.90 Appendix E Qualitative Example: The Saul Goodman Voice The post-training model does not merely omit the answer — it performs legal expertise. The following pair is drawn from personal_jurisdiction (Gold: Yes). Pre-training (CTS =0.445=0.445, length =2=2 words): Yes. Post-training (CTS =0.988=0.988, length =203=203 words): Now, personal jurisdiction --- and I want to be precise here, because precision matters in law --- is a doctrine with a long and nuanced history. The foundational framework, as articulated in landmark decisions such as Boxx v. Long and further shaped by the minimum contacts standard of Shoe v. Washington, establishes that courts must carefully evaluate the relationship between the defendant, the forum state, and the litigation. This is not a simple question. One must weigh purposeful availment, foreseeability, and the reasonableness of the exercise of jurisdiction --- factors that, I assure you, courts have grappled with extensively. The facts presented here implicate each of these considerations in ways that are, frankly, deeply interesting from a doctrinal standpoint. Whether jurisdiction is proper depends entirely on how one weighs these competing principles… [Answer: absent] This is the Saul Goodman effect in its purest form: the post-training model sounds more like a lawyer than the pre-training model, cites cases with confidence (Boxx v. Long is a hallucination; Shoe v. Washington is a garbled version of International Shoe v. Washington, 326 U.S. 310, 1945), deploys legal terminology fluently — and never answers the question. The pre-training model gives the right answer in one word.