Paper deep dive
Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery?
Nishanth Nayakanti, Prasang Gupta, Ashutosh Bilthare, Kevin Paul
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/7/2026, 8:10:14 AM
Summary
This paper investigates evidence recovery in legal review workflows where only verdicts (decisions) are recorded, not the supporting evidence spans. The authors evaluate two post-training methods on the ContractNLI dataset using a small language model (Qwen3.5-9B): Label-only SFT (fine-tuning on bare verdicts) and Rejection-sampling SFT (RSFT, fine-tuning on verdict-filtered, source-grounded traces). Results show that both methods significantly improve evidence quality (span F1 and verbatim citation rates) without any human evidence annotations during training. Label-only SFT achieved higher verdict accuracy (0.896) and slightly better span F1 (0.564) than RSFT (0.797 accuracy, 0.556 span F1), suggesting that improving verdict accuracy alone can surface latent evidence-extraction capabilities in the base model.
Entities (10)
Relation Signals (10)
RSFT → achieves → 0.797
confidence 98% · Rejection sampling... reaches 0.797 and 0.556
Label-only SFT → achieves → 0.564
confidence 98% · Label-only training on the bare verdict reaches accuracy 0.896 and span F1 0.564.
RSFT → achieves → 0.556
confidence 98% · Rejection sampling... reaches 0.797 and 0.556
Label-only SFT → achieves → 0.896
confidence 98% · Label-only training on the bare verdict reaches accuracy 0.896
Label-only SFT → improves → Verbatim Citation
confidence 95% · Verbatim citation rises from 0.597 to 0.729 under label-only training
RSFT → improves → Verbatim Citation
confidence 95% · Verbatim citation rises... to 0.701 under rejection sampling.
ContractNLI → isevaluatedon → Qwen3.5-9B
confidence 95% · On ContractNLI the human evidence spans are held out until evaluation.
Qwen3.5-9B → isposttrainedby → RSFT
confidence 95% · Student Qwen3.5-9B... Post-trained by both methods
Qwen3.5-9B → isposttrainedby →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In many review workflows the verdict is the only thing retained. The passages behind it are not marked, because that annotation costs far more than recording the decision. We measure how much of that evidence a small language model can recover when it is post-trained on the verdicts alone, with no human evidence labels at any stage. On ContractNLI the human evidence spans are held out until evaluation. Matching the recorded verdict and agreeing with those spans are not the same thing: across six systems the two scores are only weakly related and rank the systems differently, so accuracy is a poor guide when the citations have to be reviewable. Label-only training on the bare verdict reaches accuracy 0.896 and span F1 0.564. Rejection sampling, which keeps a generated trace only when its verdict matches the record and then picks one by an automatic source-grounding score, reaches 0.797 and 0.556, against 0.747 and 0.493 before training. Verbatim citation rises from 0.597 to 0.729 under label-only training and to 0.701 under rejection sampling. One seed on one corpus cannot say which method is better, but both improve the evidence without anyone annotating it.
Tags
Links
- Source: https://arxiv.org/abs/2610.06962v1
- Canonical: https://arxiv.org/abs/2610.06962v1
Trouble viewing inline? Open PDF directly →
Full Text
63,792 characters extracted from source content.
Expand or collapse full text
Verdicts Without Annotated Evidence: Rejection Sampling or Label-Only Post-Training for Evidence Recovery? Nishanth Nayakanti Prasang Gupta Ashutosh Bilthare Kevin Paul Affiliation: PricewaterhouseCoopers, U.S. Affiliation: Correspondence:nishanth.nayakanti@pwc.com Abstract In many review workflows the verdict is the only thing retained. The passages behind it are not marked, because that annotation costs far more than recording the decision. We measure how much of that evidence a small language model can recover when it is post-trained on the verdicts alone, with no human evidence labels at any stage. On ContractNLI the human evidence spans are held out until evaluation. Matching the recorded verdict and agreeing with those spans are not the same thing: across six systems the two scores are only weakly related and rank the systems differently, so accuracy is a poor guide when the citations have to be reviewable. Label-only training on the bare verdict reaches accuracy 0.896 and span F1 0.564. Rejection sampling, which keeps a generated trace only when its verdict matches the record and then picks one by an automatic source-grounding score, reaches 0.797 and 0.556, against 0.747 and 0.493 before training. Verbatim citation rises from 0.597 to 0.729 under label-only training and to 0.701 under rejection sampling. One seed on one corpus cannot say which method is better, but both improve the evidence without anyone annotating it. 1 Introduction A professional reviewing a document against a standard produces a verdict. In many operational workflows that verdict is the only artifact retained: the reasoning is not recorded and the supporting passages are not marked. This asymmetry follows from how the work is recorded, not from any particular domain. Supervised systems for these tasks are commonly built from richer records — for each document–question pair, annotators mark the passages that support the recorded conclusion. Producing those span-level labels is far more expensive than retaining the verdict alone, and operational records rarely contain them. The question is narrow: if only verdicts are available for training, how much evidence quality can be recovered without evidence annotations, and what actually does the work? Two axes, not one. We separate verdict accuracy (did the model reach the recorded conclusion?) from evidence quality (are the passages it cites verifiable and aligned with withheld human evidence?). These are distinct properties, and Section 6 shows they rank systems differently — weakly enough that the most accurate off-the-shelf system we evaluate (gpt-5.2, 0.808) is no better on evidence than the untrained student, span F1 0.493 for both, while two lower-accuracy systems — the 31B sampler and sonnet-4.6 — score highest on evidence. A workflow that needs reviewable evidence cannot select a model on accuracy alone. The shape, not the domain. ContractNLI (Koreeda and Manning, 2021) is useful here not because it concerns contracts but because it reproduces the operational structure: a fixed set of questions applied to every document, one recorded verdict per question, and a separate evidence layer that can be withheld from training. The first two properties are common in operational records; the third is usually absent. Checklists against filings, claims against policies, submissions against standards, and cases coded against schedules all retain conclusions while omitting the passages behind them. ContractNLI makes the question measurable precisely because its evidence layer can be hidden during training and used only to evaluate what was learned without it. It is a proxy, not the operational setting itself: what we report is one corpus, one student, one seed per arm. Two methods, one question. We compare two post-training methods that both use only the recorded verdict — no human evidence annotation anywhere: • Label-only SFT: fine-tune the student directly on the bare recorded verdicts — the direct-answer baseline these methods are usually measured against (Hsieh et al., 2023). • Rejection-sampling SFT (RSFT): following STaR (Zelikman et al., 2022), generate candidate traces — from the student and a larger 31B sampler — keep only those whose verdict matches the record, select one per row by an automatic source-grounding score, and fine-tune on the selected rationale-plus-citation traces. We write RSFT for our instance throughout: rejection-sampling fine-tuning (RFT; Yuan et al., 2023) with an added source-grounding selection stage over the surviving candidates, which RFT does not have. (About a fifth of the selected traces come from the sampler; Section 3.2, Table 5.) RSFT is the more elaborate method — it costs an extra candidate-generation stage and a grounding selector, and it trains the model to emit grounded citations. Label-only SFT is the cheapest thing that could work: it skips candidate generation and selection entirely, so end-to-end it is a single ~2.6 GPU-hour training run against RSFT’s ~9.7 (6.8 for candidate generation plus 2.9 for training; Section 3.2, Appendix A.2). Comparing them provides a practical test of what additional value rationale-plus-citation supervision provides over bare-label supervision. What we find. Both methods recover evidence quality without evidence annotation, and we report them as a comparison rather than a ranking. In this single-seed, single-corpus setting the two arms land close on the evidence axis (span F1 0.556 for RSFT, 0.564 for label-only, against the untrained student’s 0.493) and further apart on accuracy (0.797 vs 0.896, against 0.747). Both also raise source verifiability — the rate at which a cited passage appears verbatim in the contract — from 59.7% untrained to 70.1% for RSFT and 72.9% for label-only. A 2-epoch RSFT run reproduces the 1-epoch result (0.796), arguing against insufficient epoch count as the explanation for the gap. A candidate mechanism ties the two axes together: evidence F1 is mediated by the verdict — a NotMentioned prediction returns no spans, and a correct Entailment/Contradiction prediction returns citations the base model already knows how to produce — so verdict-improving fine-tuning can surface more evidence. Read together, the results are consistent with the base model already containing substantial evidence-extraction capability that verdict-only adaptation surfaces, rather than capability the supervision installs. On this reading the evidence gain does not appear to require rationale supervision, but a single seed cannot settle which method is preferable, and we leave that to the stacked and repeated-seed variants in Section 11. Our contributions are as follows: 1. We frame evidence recovery from verdict-only records as a measurable post-training problem and separate two quality axes — verdict accuracy and evidence-span agreement — that we show decouple across systems (Section 6). 2. We compare two verdict-only post-training methods under a matched decode and a locked prompt, with one greedy, unfiltered output per model at evaluation and no human evidence annotation at any stage (Sections 3–5). 3. We show the two methods reach comparable evidence quality by different routes: label-only SFT — trained on bare labels, with no rationale or citation — gains through the verdict, while RSFT gains through the citation quality its source-grounded supervision buys, its verdicts actually dipping on the scored rows (Section 5). Evidence gains here need no evidence annotation, and for label-only no rationale supervision at all; we do not claim one method is preferable in general. 4. We bound alternative explanations: a label-marginal calibration control — fit on the training split, reading no contract text and requiring no training of its own — recovers only 37% of RSFT’s accuracy increment (Section 4.3), and a 2-epoch RSFT run reproduces the 1-epoch result, arguing against insufficient epoch count (Section 8). 2 Setup 2.1 Task and dataset Task. ContractNLI (Koreeda and Manning, 2021) pairs each contract with 17 fixed hypotheses. For each document–hypothesis pair the system predicts one of three labels: Entailment when the contract supports the hypothesis, Contradiction when it conflicts, and NotMentioned when the contract provides neither. Human-marked evidence spans accompany entailment and contradiction rows; they are withheld from prompting, candidate filtering and selection, training, and checkpoint choice, and are used only for evaluation. Figure 1: The corpus. Every contract is assessed against all 17 hypotheses, so rows are document–hypothesis pairs. The verdict distribution is over all 10,319 rows; the test split’s NotMentioned rate is higher, at 43.2%. Evaluation is on the canonical test split: 123 contracts, 2,091 rows. Withheld annotation layer. Each document is pre-segmented into candidate spans (median 71, mean 79) and each row marks every span as evidence or not. On the training split this yields 559,215 binary span decisions alongside 7,191 verdicts. We use the verdicts to construct supervision and reserve the span labels for held-out measurement, matching the operational asymmetry between routinely retained decisions and separately commissioned evidence annotation. 2.2 Models and controls Role System Uses in this study Student Qwen3.5-9B (Qwen Team, 2026), 4-bit, QLoRA Post-trained by both methods; evaluated before and after Sampler Gemma-4-31B (Google DeepMind, 2026) Supplies additional candidate traces to RSFT; also an inference reference Commercial references sonnet-4.6 (Anthropic, 2026), gpt-5.2 (OpenAI, 2025) Inference-time comparisons only; no training data or supervision Method A Label-only SFT Fine-tune the student on bare recorded verdicts Method B RSFT Fine-tune on verdict-filtered, source-grounded traces from the student and a 31B sampler Control Prior calibration Relabel the untrained student’s predictions from training-split confusions (Section 4.3) Control Hypothesis lookup Predict from hypothesis identity, reading no contract (0.659 accuracy) Control Majority class 0.463 accuracy Table 1: Models, methods, and controls. Two non-neural controls contextualize verdict accuracy. The majority-class predictor scores 0.463. The hypothesis-only lookup control predicts from hypothesis identity without reading the contract and scores 0.659; because the task repeats 17 hypotheses, it estimates how much performance comes from hypothesis-specific label priors — the same class-prior artifact that hypothesis-only baselines expose in NLI datasets (Poliak et al., 2018; Gururangan et al., 2018). Performance above it is consistent with using information beyond those priors, though it does not by itself identify how the contract text is used. Commercial reference identifiers. sonnet-4.6 is Anthropic Claude Sonnet 4.6 on AWS Bedrock (us.anthropic.claude-sonnet-4-6, us-east-1, undated v1 snapshot); gpt-5.2 is OpenAI gpt-5.2-2025-12-11. Both were queried under greedy decoding on 23 September 2026; neither records completion-token counts (Section 2.4), so their decode caps cannot be recovered from the saved records. 2.3 Prompt and output format Two prompts, one task. A single task definition is locked and hash-checked across systems and runs. We use two output variants of it: • faithful (hash f9318d34) — asks for the verdict plus concise verbatim evidence quotations and a short reasoning field. This is the prompt used for every evidence-axis evaluation and by RSFT candidate generation. • verdict_only (hash 7ada9e8d) — the same contract, hypothesis, and three-way task instruction, with the evidence paragraph removed and the output reduced to the bare verdict. Used to train and evaluate label-only SFT on the accuracy axis. Everything above the output block is byte-identical between the two except the single evidence paragraph; only the requested output differs (full text in Appendix A). We drop the evidence paragraph from verdict_only because leaving it in would instruct a behaviour the label-only arm has no target for — training the model to disobey its own prompt — the confound STaR (Zelikman et al., 2022) and Distilling Step-by-Step (Hsieh et al., 2023) avoid by adapting the direct-answer prompt. This choice creates a deliberate train/test mismatch for the label-only arm: it is trained under verdict_only and evaluated on the evidence axis under faithful, a prompt it never saw in training. The mismatch runs against that arm rather than for it, and its cost on the accuracy axis is small — 0.896 under faithful against 0.899 under verdict_only (Table 6). That a model trained to emit bare verdicts nonetheless cites on 93.4% of rows when asked is part of the Section 5 result, not an artifact of it. The template adds no worked examples, retrieval stage, or decision rules. A tested variant containing three plausible decision rules changed none of the polarity errors it was written to fix (a disclosed single observation, not a controlled study). Structured output is the one imposed requirement: the citation must arrive as a field, or the evidence measure is not computable. The faithful prompt also elicits a 1–10 confidence score; it is near-degenerate in practice, we do not analyse it, and it enters no reported metric. 2.4 Evaluation protocol All model evaluations use the canonical test split of 123 contracts and 2,091 rows. Each system produces one greedy, single-pass output per row under the shared prompt; no inference-time candidate filtering or selection is applied. Unparseable outputs are scored as incorrect; outputs without a citation receive no predicted evidence spans. Verdict accuracy and macro F1 are reported on all rows. Evidence precision, recall, and F1 are computed on the fixed 1,188 test rows whose gold label is Entailment or Contradiction — the classes for which ContractNLI provides human evidence spans — regardless of the model’s predicted label. A model that predicts NotMentioned and emits no evidence on an eligible row incurs no false-positive spans but misses the gold spans, reducing recall and F1. Matched decode. All reported runs use a 900-token completion cap with one exception. An earlier release decoded the untrained student and the 31B sampler at 700 tokens while the post-trained student ran at 900; we re-ran the untrained student at 900 (Section 8), so every comparison involving the two fine-tunes and their baseline uses an identical cap. The 31B sampler was not re-run and its figures are from the 700-token pass; only 1 of its 2,091 rows reached that cap, so they are effectively cap-insensitive (Table 12). The two commercial systems do not record completion-token counts, so their caps cannot be read from the saved records. 3 Methods compared Both methods post-train the same student with low-rank adapters (Hu et al., 2022) over a 4-bit quantised base (QLoRA; Dettmers et al., 2023) for one epoch, loss on target tokens only, no human evidence annotation, and no checkpoint selection. They differ in the supervision target and the corresponding training-output instruction — label-only is trained and prompted with verdict_only, RSFT with faithful — and evidence quality is compared under the matched faithful prompt (Section 5). RSFT is an instance of rejection-sampling / reward-ranked fine-tuning (Yuan et al., 2023; Dong et al., 2023); label-only SFT is the direct-answer baseline against which rationale-supervised methods are usually measured (Hsieh et al., 2023). Section 9 places both in the wider literature. 3.1 Method A — Label-only SFT The target is the bare recorded verdict, wrapped in the <answer> block the parser expects and nothing else: ⬇ <answer> "hypothesis_id": "nda-2", "answer": "Contradiction" </answer> Targets are reduced to a mean of 71 characters (down from 1,294 for the rationale traces). Training uses 7,191 rows on a 6,477 / 714 document-level split; the training documents share zero documents and zero (document, hypothesis) rows with the test split. No evidence annotation, no rationale, no candidate generation. 3.2 Method B — Rejection-sampling SFT (RSFT) RSFT generates multiple candidate traces per training row, rejects those whose verdict disagrees with the record, ranks survivors by an automatic source-grounding score, selects one target per covered row, and fine-tunes on the assembled set. Verdict filtering determines eligibility; grounding selection chooses among eligible candidates. No stage reads human evidence annotations. Figure 2: RSFT pipeline: candidate generation → verdict filtering → source-grounding selection → one epoch of target-only QLoRA. Candidate generation (Appendix A.2). Both models produce one greedy candidate per training row; sampled passes then run only on rows whose greedy output fails the verdict filter — 32 samples for the student, 8 for the sampler. Candidates are pooled by row before filtering. Candidate generation costs 6.8 GPU-hours; training costs 2.9. Pass Rows attempted Samples Rows with ≥ 1 accepted Greedy, Gemma-4-31B 7,191 7,191 5,908 (82.2%) Greedy, Qwen3.5-9B 7,191 7,191 5,607 (78.0%) Sampled, Gemma-4-31B (k=8) 1,283 10,264 245 (19.1%) Sampled, Qwen3.5-9B (k=32) 1,584 50,688 1,248 (78.8%) Pooled 7,191 75,334 6,938 (96.5%) Table 2: Gate 1 (verdict filter) budget and yield. The sampled passes run only on rows their own greedy pass failed, so their denominators are residues, not the corpus. As shown in Table 2, the two greedy passes cover 86.2% of rows; the sampled passes add 741, reaching 96.5%; the remaining 253 rows are dropped. Verdict filtering. Every candidate is retained only when its predicted verdict equals the recorded label — no partial credit, no confidence threshold, no inspection of the rationale. Source-grounding selection. Survivors are ranked from their quoted excerpts alone. Step 1, well-formedness: a candidate is eligible if it closes an <answer> block, parses as JSON, has a non-empty reasoning field, and follows the citation rule (evidence for E/C, none for NotMentioned); this removes 2,746 candidates and leaves ≥ 1 well-formed candidate on 98.6% of covered rows. On the remaining 1.4% no verdict-matching candidate is well-formed; rather than drop the row, the method keeps its raw verdict-matching pool and scores that, so every covered row still yields a target and Table 5 totals all 6,938. Step 2, grounding: each excerpt scores 3 (verbatim), 2 (verbatim after normalisation), 1 (≥ 60% of content tokens in one contiguous stretch), or 0 (unlocatable); the candidate score is the mean excerpt score over 3. The highest-scoring candidate wins; ties use a fixed provenance order (student greedy, student sampled, sampler greedy, sampler sampled). Note on the grounding instrument. Selection and evaluation share the same three-tier span-matching ladder (Section 5), so span F1 is partly a locatability measure; we therefore lead with verbatim rate in Section 5. Table 3 quantifies what the selector is worth in the training archive. The provenance tie-break costs 0.013 span F1 relative to load-order ties, and raises the on-policy share from 21.5% to 81.6% in this re-selection (80.2% in the shipped set, Table 5). Nothing in either gate reads a human evidence label, and nothing selects at inference. Selection rule P R F1 Any hit First survivor, no grounding 0.915 0.534 0.675 83.0% Grounding, tie by load order 0.910 0.578 0.707 90.4% Grounding, tie by provenance (shipped) 0.890 0.569 0.694 90.0% Oracle (reads gold spans) 0.936 0.593 0.726 91.1% Table 3: What selection is worth in the training data. Agreement of selected RSFT training targets with withheld human spans over 4,303 covered training rows. “Any hit” is the share of rows whose selected trace cites at least one correct span. The oracle picks, from the same candidate pool, the candidate with the highest agreement (headroom only). Gold spans are used only for this analysis: no stage reads them, and nothing selects at inference. How the row was decided Rows Share Student wins Sampler wins One candidate left, kept as-is 529 7.6% 347 (65.6%) 182 (34.4%) Grounding score separated them 1,513 21.8% 360 (23.8%) 1,153 (76.2%) Grounding tied, provenance decided 4,896 70.6% 4,858 (99.2%) 38 (0.8%) Total 6,938 5,565 (80.2%) 1,373 (19.8%) Table 4: Gate 2 (grounding) decision accounting. Grounding separates candidates on 21.8% of covered rows; on 70.6% the survivors tie (usually because all quotations match verbatim) and the provenance order decides. Where grounding discriminates, the 31B sampler supplies 76.2% of selected traces. Source of the selected trace Rows Share Qwen3.5-9B, greedy 4,382 63.2% Gemma-4-31B, greedy 1,317 19.0% Qwen3.5-9B, sampled 1,183 17.1% Gemma-4-31B, sampled 56 0.8% Student total (on-policy) 5,565 80.2% Table 5: Where the 6,938 RSFT training targets came from. Four fifths of targets are the student’s own generations. Of the 5,025 excerpts in selected traces, 89.9% match verbatim, 1.9% after normalisation, 8.2% partially, and 0.1% cannot be located; mean excerpts per target is 0.72 because NotMentioned targets correctly contain no citation. These statistics describe the supervision, not a causal attribution. 3.3 Controls Prior calibration (Section 4.3) relabels the untrained student’s own predictions through a confusion table fit on the training split, correcting the label marginal without reading a contract or training anything. Hypothesis lookup and majority class bound what hypothesis identity and the corpus rate alone can do. 4 Verdict prediction results 4.1 Aggregate performance System Method Prompt Accuracy Macro F1 Majority class — — 0.463 0.211 Lookup control (reads no contract) — — 0.659 0.617 Qwen3.5-9B, untrained — faithful 0.747 0.701 Qwen3.5-9B, untrained — verdict_only 0.753 0.718 sonnet-4.6 — faithful 0.771 0.718 Gemma-4-31B (sampler) — faithful 0.786 0.748 Qwen3.5-9B, RSFT RSFT faithful 0.797 0.758 gpt-5.2 — faithful 0.808 0.771 Qwen3.5-9B, label-only label-only faithful 0.896 0.860 Qwen3.5-9B, label-only label-only verdict_only 0.899 0.861 Table 6: Held-out verdict prediction, 2,091 test rows, matched 900-token decode. Label-only SFT is reported under both prompts; RSFT and the untrained student under faithful. The two fine-tunes rank in the opposite order to their cost. Label-only SFT reaches 0.896–0.899, above RSFT’s 0.797 by roughly +0.10 and above its prompt-matched base (0.753) by +0.145, at one epoch each. It sits well above the hypothesis-identity lookup control (0.659), consistent with using information beyond hypothesis-specific priors, though this does not by itself identify the mechanism or exclude pretraining exposure (Section 7, Limitations). The shift is not majority-gaming: NotMentioned recall moves 0.59 → 0.895, all three class recalls rise, and the predicted mix lands on the gold marginal (predicted NM 0.425 vs gold 0.432). Comparison Δ accuracy 95% CI (row) 95% CI (doc-clustered) McNemar RSFT vs untrained (matched decode) +0.050 [+0.036, +0.064] [+0.035, +0.065] 166 fixed, 62 broken, p = 4×10−124× 10^-12 Label-only vs untrained +0.149 [+0.131, +0.167] [+0.129, +0.169] 359 fixed, 48 broken, p = 6×10−606× 10^-60 Label-only vs RSFT +0.099 [+0.083, +0.115] [+0.083, +0.115] 259 fixed, 52 broken, p = 3×10−343× 10^-34 RSFT vs 31B sampler +0.011 [−-0.005, +0.027] — p = 0.20 RSFT vs gpt-5.2 −-0.011 [−-0.027, +0.006] — p = 0.21 Table 7: Paired accuracy comparisons (10,000 paired bootstrap resamples (Efron and Tibshirani, 1994), seed 37; exact two-sided McNemar (McNemar, 1947) on discordant pairs). CIs are reported both row-level and document-clustered (resampling the 123 test contracts, carrying all rows of each). Clustering does not widen the intervals. The document-clustered bootstrap (Efron and Tibshirani, 1994) is nearly identical to the row-level one because the measured one-way ICC of the correctness indicator across documents is ≈ 0 (design effect ≈ 1.0 for all three fine-tunes) — the row-level intervals the study reports are not anti-conservative here, which addresses the clustering concern directly rather than by assumption. The RSFT student is not statistically distinguishable from the 31B sampler or gpt-5.2 on accuracy; both intervals include zero. A non-significant difference is not evidence of equivalence — we run no equivalence test (TOST; Schuirmann, 1987; Lakens, 2017) and do not describe the RSFT student as “level with” the references. The label-only-vs-RSFT accuracy gap is large relative to these intervals but remains a single-seed estimate (Section 8, Limitations). Figure 3: The two verdict-only methods against the untrained student, matched decode, faithful prompt; source verifiability is the all-verbatim rate of Section 5. Error bars are 95% bootstrap intervals over each system’s own test rows — marginal, not paired — and the horizontal axis is truncated. The two fine-tunes separate on verdict accuracy alone: their span-F1 intervals overlap, and so do their verifiability intervals (RSFT [0.675, 0.726] vs label-only [0.704, 0.754]). Overlapping marginal intervals do not settle a comparison; the paired difference in Section 5 is what does. Comparison with a span-supervised system. Span NLI BERT (Koreeda and Manning, 2021), trained with the full evidence layer (all 559,215 span decisions), reports 0.892 accuracy on this split; label-only SFT reaches 0.896 on the recorded verdicts alone, and RSFT 0.797. We flag this only as a descriptive, unmatched comparison — different architecture, different supervision (span-level vs verdict-only), and an evaluation convention we did not reproduce — and claim no equivalence to a span-supervised system. The point is narrow: a 9B model post-trained on bare verdicts reaches verdict accuracy comparable to a fully span-supervised system here, which is also what most sharpens the contamination question (Limitations). 4.2 Error analysis Figure 4: Verdict confusion (matched decode). The fourth predicted column, none, holds outputs the parser could not resolve to a verdict (25 / 39 / 0); these score as incorrect, and including them is what makes every panel sum to all 2,091 test rows. The NotMentioned diagonal grows from the untrained student (533) to RSFT (646) to label-only (808); label-only also improves the Entailment and Contradiction diagonals, whereas RSFT trades a little E/C recall for its NM recovery. Total errors under RSFT fall from 529 to 425. Decomposing the −-104-row net change by gold class states what the aggregate hides: Row population n Change in correct predictions Gold NotMentioned 903 +113 Gold Entailment or Contradiction (the rows the evidence metric scores) 1,188 −-9 Net 2,091 +104 Table 8: Where RSFT’s accuracy gain comes from. The entire net gain is NotMentioned recovery; on the 1,188 evidence-bearing rows RSFT is 9 predictions worse. Predicted NotMentioned moves 27.4% → 34.2% against a gold rate of 43.2% — a shift of one class marginal toward the corpus rate. Entailment recall falls 0.889 → 0.886 and Contradiction recall 0.764 → 0.736; the per-class F1 gains for E and C are precision gains from fewer spurious E/C predictions on NotMentioned rows, i.e. the same phenomenon as the NM recovery, not three independent improvements. Label-only SFT, by contrast, raises NotMentioned recall much further (to 0.895) and lands its predicted NM share on the gold marginal (0.425 vs 0.432) while E and C recall rise rather than fall (E 0.889 → 0.911, C 0.764 → 0.832) — which is why its accuracy lands higher without the E/C regression RSFT shows. Two confounds favour the label-only arm. Both are inherent to the cheaper method rather than measurement artifacts, but the headline +0.099 accuracy gap should be read with them in mind. (i) The arms do not train on the same rows. RSFT trains on 6,251 rows drawn from the 6,938 Gate-1-covered ones; label-only trains on 6,477 from the full 7,191, so it sees — with gold labels — the 253 rows RSFT structurally cannot cover, which are the hardest rows (no sampled trace ever produced the recorded verdict), concentrated on nda-2/16/17. This shows in the data: on nda-2/16/17 label-only leads RSFT by +0.21 (0.870 vs 0.661), against +0.08 on the other 14 hypotheses (0.901 vs 0.826), so coverage is doing part of the work. (i) Label-only cannot truncate. Its 71-character targets never reach the 900-token cap, so it has zero unparsed rows against 25 (untrained) and 39 (RSFT), and unparsed scores as wrong; restricted to the 2,052 rows RSFT parses, RSFT is 0.812 and label-only 0.897, a gap of +0.085 — so roughly 1.4 of the 9.9 points is parse-failure asymmetry rather than verdict quality. 4.3 How much of the accuracy gain is calibration? If the accuracy gain is mostly a class-marginal correction, a label re-mapping that reads no contract should reproduce it. We fit such a control on the untrained student’s own training-split predictions and apply it to the test predictions; results are intersected on the rows both systems parse (2,025 rows), since a relabelling control has no label to re-map on an unparsed row. System Accuracy vs untrained Share of the RSFT gain untrained student 0.764 — — + global calibration 0.764 +0.000 0% + per-hypothesis calibration 0.782 +0.018 37% lookup control (no model) 0.667 −-0.098 — RSFT student 0.813 +0.048 100% Table 9: Calibration control (intersected population, unparsed dropped). A label re-mapping recovers about a third of the increment, not most of it. And it cannot reproduce the manner of the shift: calibration buys NotMentioned recall by spending precision (NM precision/recall 0.928/0.613 → 0.825/0.692), whereas RSFT gains recall and keeps precision (0.904/0.732). The remainder of the increment is not explained by the label marginal. (The absolute delta here, +0.048, differs from the +0.050 in Table 7 only because this population drops unparsed rows; the share of the gain is the stable quantity to quote. For the same reason the lookup control reads 0.667 in this table, against 0.659 on the full test split reported elsewhere — the two figures are the same control on the intersected versus the full population.) 5 Evidence quality without evidence annotation The central evidence result is worth stating sharply. A model fine-tuned only on bare verdicts, under a prompt that contains no evidence paragraph, and then evaluated under the citation-requesting faithful prompt, produces higher span agreement with withheld human evidence than the untrained model — 0.564 vs 0.493. Evidence quality it was never trained to produce improves. This is the cleanest support for the verdict-mediation reading below, and it holds without any evidence annotation entering training. Evidence is evaluated on the 1,188 rows with human-marked spans (968 entailment, 220 contradiction). We lead with source verifiability — the share of rows whose every quotation is found verbatim in the contract — because it is the one metric independent of the grounding instrument used in selection (Section 3.2). Span F1 against withheld human spans is reported second. Measure Untrained RSFT Label-only Rows all-verbatim (source verifiability) 59.7% 70.1% 72.9% Verbatim rate among rows that cite 61.8% 75.8% 78.0% Span precision vs human gold 0.830 0.859 0.882 Span recall vs human gold 0.351 0.411 0.415 Span F1 vs human gold 0.493 0.556 0.564 Rows emitting any citation 96.5% 92.5% 93.4% Table 10: Evidence quality against withheld human annotation, matched decode, faithful prompt. Both methods recover evidence quality without any evidence annotation. As a gain over the untrained faithful baseline, span F1 rises +0.062 for RSFT (95% paired bootstrap CI [+0.042, +0.082]) and +0.071 for label-only ([+0.053, +0.090]); both exclude zero. On the evidence axis the two fine-tunes are statistically indistinguishable: the label-only-minus-RSFT span-F1 difference is +0.009 with a 95% CI of [−-0.011, +0.028] (document-clustered [−-0.014, +0.031]), spanning zero, and they differ by only 2.8 points of all-verbatim rate. The result we lean on is therefore not a gap between the two arms — there is none we can resolve at one seed — but that a model which never saw a citation in training recovers evidence quality at all. Why, mechanically — and the two arms differ here. Evidence F1 is mediated by the verdict: a NotMentioned prediction returns empty spans, a correct Entailment/Contradiction prediction returns citations, and the base model already extracts evidence well (untrained 0.49 F1, 96.5% of rows cite). Label-only’s gain fits this cleanly — it gets more E/C verdicts right (Section 4.2), so it emits citations on more of the scored rows. RSFT’s does not: on the 1,188 E/C rows its verdicts are 9 worse and it cites on fewer of them (92.5% vs the base’s 96.5%), yet it recovers 143 more gold spans (true positives 844 → 987). Its span-F1 gain therefore comes from better citations on the rows it does cite — from the source-grounded rationale supervision — not from the verdict. The two arms reach almost the same span F1 (0.556 vs 0.564) by different routes: label-only through the verdict, RSFT through citation quality. So verdict mediation carries the label-only result but not the RSFT one; neither route requires evidence annotation. Class n Untrained RSFT Label-only Entailment 968 0.506 0.564 0.572 Contradiction 220 0.436 0.517 0.530 Table 11: Evidence F1 by class (faithful prompt). Locatable subset. Restricting to the 1,037 rows where all three systems emit a string-locatable citation — which removes the locatability component the shared instrument otherwise credits — the ranking holds: untrained 0.510, RSFT 0.586, label-only 0.590. This subset conditions on a post-treatment variable — all three systems emitting a locatable citation — so it excludes exactly the rows where RSFT flipped its verdict to NotMentioned and cited nothing; the comparison is descriptive of the shared-citation rows, not an unbiased treatment effect. The evidence advantage is not solely a locatability effect. Span-matching and scoring. A citation is matched to gold spans by the same three-tier ladder used in selection (Section 3.2): exact string, whitespace/punctuation-normalised retry, then credit if ≥ 60% of its content tokens fall in one contiguous stretch. Precision, recall, and F1 are pooled counts over the 1,188 rows, not per-row averages. A citation that cannot be located anywhere in the contract contributes one false positive and no true positive, so it lowers precision; an eligible row that emits no citation lowers recall. Unlocatable citations are rare (3–4 excerpts per system). Pooled counts are tp/fp/fn = 844/173/1560 (untrained), 987/162/1417 (RSFT), 998/134/1406 (label-only). Mean citations per citing row (1.11–1.19) and mean citation length (223–233 characters) are similar across arms, so the recall differences are not driven by longer or more numerous quotes. What the comparison supports, and what it does not. We do not claim rejection sampling is the reason evidence quality improves: a cheaper intervention — supervised fine-tuning on the bare labels the record already contains — reaches comparable evidence quality in this run, so the rejection-sampling machinery is not established as necessary for the gain. Nor do we claim label-only SFT is the better method; a single seed cannot support that ordering. What the comparison does support is the weaker and, we think, more useful statement: evidence quality is recoverable from verdict-only records without evidence annotation, and both methods achieve this using supervision derived from the recorded verdict. It is worth noting which arm had the richer inputs: RSFT drew about a fifth of its selected traces from a 31B sampler (Table 5), while label-only used nothing beyond the recorded labels, and the two still land within a non-significant 0.009 of each other on span F1. 6 Verdict accuracy and evidence quality decouple Figure 5: Cross-system relationship between verdict accuracy and evidence span F1, six systems under the faithful prompt. The two axes are only weakly related (descriptive Pearson +0.12 for span F1, +0.20 for verbatim rate) and induce different rankings: gpt-5.2 has the highest off-the-shelf accuracy yet is no better on evidence than the untrained student (span F1 0.493 for both — the pair the on-figure annotation marks), while the 31B sampler and sonnet-4.6 are lower-accuracy yet highest on evidence (the label-only fine-tune is the most accurate system overall, at 0.896). The open square is label-only under the verdict_only prompt (accuracy 0.899, span F1 0.000 by construction — no citation is requested), the extreme point of the decoupling; it is excluded from the correlation, and the vertical axis is broken below 0.50 so that it can be shown. The 31B sampler was decoded at a 700-token cap (all other systems at 900), but only 1 of its 2,091 rows reached the cap, so its position is effectively unaffected; see Table 12. System Accuracy Span F1 Verbatim rate Qwen3.5-9B, untrained 0.747 0.493 59.7% sonnet-4.6 0.771 0.594 70.9% Gemma-4-31B (sampler) 0.786 0.625 77.5% Qwen3.5-9B, RSFT 0.797 0.556 70.1% gpt-5.2 0.808 0.493 48.7% Qwen3.5-9B, label-only 0.896 0.564 72.9% Table 12: Verdict accuracy and evidence quality across the six systems (faithful prompt), ordered by accuracy. The two axes rank systems differently: accuracy rises down the column while span F1 and verbatim rate do not track it. All open-weight/student rows use the 900-token cap except the 31B sampler, which was decoded at 700; only 1 of its 2,091 rows reached that cap, so its figures are effectively cap-insensitive (the untrained student, with 62 rows at the 700 cap, is the one re-run at 900; Section 8). The two commercial systems do not record completion-token counts, so their caps cannot be read from the saved records. The rank reversals are visible directly in the measurements and do not depend on the correlation coefficients, which with six systems summarize this set rather than a population; the decoupling rests on the rankings, not the exact coefficient. The coefficients are moreover composition-dependent: across the four off-the-shelf systems alone the accuracy–verbatim correlation is −-0.27, and adding the two fine-tunes moves it to +0.20 — a further reason to read the rankings, not the number. gpt-5.2 is the plainest counterexample — highest off-the-shelf accuracy, lowest verbatim rate, and span F1 tied with the untrained student at the bottom of the column — and its cap is not the cause, since its accuracy, not its evidence, is what leads. Label-only SFT under the verdict_only prompt is the cleanest illustration of the decoupling: accuracy 0.899 with span F1 0.000 by construction — high verdict accuracy and no evidence at all, from one system, because the prompt requests no citation (the open square in Figure 5). The practical reading: for workflows that need reviewable evidence, verdict accuracy alone is insufficient for model selection. A system can produce more correct labels while providing quotations that are less often verifiable or less aligned with withheld human evidence. And within a single model, whether it emits evidence at all is a property of the prompt, not only of the weights (Limitations). 7 Secondary analyses Sampling recovery at matched budget. Among rows both models miss under greedy decoding, the 9B student recovers 52.0% at the matched sampling budget vs 13.2% for the 31B sampler (n = 994, matched k = 8, unbiased estimator 1−C(n−c,k)/C(n,k)1-C(n-c,k)/C(n,k)). These are point estimates without a confidence interval, computed over the training-split candidate generation rather than the test evaluation; they show standalone greedy accuracy does not determine recovery under sampling, but do not establish why the rates differ and are not a claim about diversity. Training-source mixture (RSFT). The sampler’s share of selected traces varies across hypotheses from 2% to 46%. The descriptive correlation between the student’s training-trace share and its test-accuracy gain is −-0.14 across 17 hypotheses; hypotheses below the median student share gain 0.069 on average vs 0.033 at or above. With 17 hypotheses this supports no directional inference and isolates no component; a matched ablation is required. Hypothesis-only control. A control reading hypothesis identity but no contract reaches 0.659, consistent with hypothesis-specific label priors explaining a substantial share of performance. Performance above it is consistent with using more than those priors but does not exclude memorisation or pretraining exposure. 8 Experimental configuration Both fine-tunes use the configuration in Table 13; label-only SFT differs only in the target text and the training prompt. The recipe was fixed before training; no hyperparameter was tuned on the test split and no checkpoint was selected on test results. Setting Value Base model Qwen/Qwen3.5-9B, Apache 2.0, 4-bit (Unsloth; Han et al., 2023) Adapter QLoRA, rank 64, α 128, dropout 0.05, no bias Target modules q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj Loss Cross-entropy on target tokens only; prompt labels set to −-100 manually Learning rate 5×10−55× 10^-5, cosine decay, 20 warmup steps Weight decay 0.05 Batch 1 × grad-accum 8 = effective batch 8 Epochs / steps 1 epoch; RSFT 782 steps (n_train = 6,251), label-only 810 steps (n_train = 6,477) Max sequence length 16,384 tokens, keep-end truncation Seed 42, single seed per arm Hardware 1 × L40S, 48 GB Training wall-clock RSFT 2.90 h, 13.4 s/step; label-only 2.63 h, 11.7 s/step. Label-only runs more steps on more rows yet finishes sooner: its 71-character targets cost about 12% less per step than RSFT’s 1,294-character traces, against the shared ~13.6k-token prompt Checkpoint selection None; final step retained Evaluation decoding vLLM (Kwon et al., 2023), greedy, single pass, 900-token cap for every run except the 31B sampler (700; see Section 2.4) Table 13: Training configuration. Hyperparameters are shared by both methods; training rows, step count and wall-clock differ by arm. Robustness: a second epoch does not close the gap. A 2-epoch RSFT run (cnli-rsft-qwen-e2) reaches accuracy 0.796 and span F1 0.555 — indistinguishable from the 1-epoch run (0.797 / 0.556). This argues against insufficient epoch count as the explanation for the gap to label-only SFT; it does not rule out other differences in the training schedule. A second training seed per arm remains outstanding and single-seed point estimates should be read as provisional (Limitations, Section 11). Pre-registration (legacy). Our code carries a pre-registration from an earlier feasibility framing, with targets fixed before training (accuracy ≥ 0.850, all-verbatim rate ≥ 0.700, span F1 ≥ 0.600); of these the RSFT recipe met only the verbatim target (0.701). We report this for completeness, but these absolute deployment gates are not the object of this study, whose claims concern the relationship between the two quality axes rather than clearing a fixed threshold. Statistics and multiplicity. Confidence intervals are 2.5/97.5 percentile paired bootstraps (10,000 resamples; seed 37 for accuracy, seed 6 for span F1) (Efron and Tibshirani, 1994), reported row-level and document-clustered; verdict comparisons also use the exact two-sided McNemar test (McNemar, 1947). The paper runs several paired tests without a multiplicity correction: the headline McNemar p-values (≤4×10−12≤ 4× 10^-12) survive any reasonable correction, but the per-class F1 movements (Section 4.2), the cross-system correlation (Section 6), and the sampling-recovery figures (Section 7) are uncorrected point estimates and should be read as descriptive. Matched-decode correction. An earlier release stated that decoding was identical for every system. Read off the completion-token counts, it was not: the greedy family (untrained student, 31B sampler) was capped at 700 while the post-trained student, the sampled passes, and the frontier runs used 900. We re-ran the untrained student at 900 (cnli-greedy-qwen-m900); the 31B sampler was not re-run, and Section 2.4 and Table 12 state why its figures are cap-insensitive. The accuracy delta is unchanged to four decimals (+0.0497) and the evidence delta is slightly larger under the matched cap (+0.0586 → +0.0622); only the baseline moved, because only the baseline was decoded unfairly. We record the completion-token count and the token cap on every generated record, alongside the prompt hash f9318d34. Data (RSFT). 6,251 training and 687 validation traces, one per row, split by document (381 train, 42 validation documents); target length median 312 tokens. Assembled class mix 50.2 / 11.8 / 38.0 vs corpus 49.1 / 11.7 / 39.2 — every class within 1.2 points. Gate-1 coverage by class: 98.6% entailment, 97.6% contradiction, 93.4% not-mentioned; the 253 uncovered rows concentrate on nda-2/16/17 (153 of them). Validation loss is used for monitoring only and is flat over the final interval (0.1822 → 0.1823), not monotone to the end. Data (label-only). 6,477 training and 714 validation rows on the same 381 / 42 document split, with every ContractNLI row retained: this arm needs no candidate to survive Gate 1. Targets average 71 characters, and training ran 810 steps in 2.63 h. 9 Relation to prior work Rejection-sampling and self-training fine-tuning. The method family that RSFT belongs to bootstraps supervision from a model’s own filtered generations: STaR (Zelikman et al., 2022) iterates rationale generation, correctness filtering, and retraining; RFT (Yuan et al., 2023) filters sampled reasoning paths on answer correctness and RAFT (Dong et al., 2023) reward-ranks sampled generations under a learned reward model; Llama 2 (Touvron et al., 2023) uses rejection sampling in its alignment pipeline; ReST (Gulcehre et al., 2023) frames the loop as reinforced self-training; and Distilling Step-by-Step (Hsieh et al., 2023) supervises rationales from a larger teacher. Our RSFT is one round of verdict-filtered, source-grounded rejection sampling, and label-only SFT is the direct-answer baseline against which these methods are usually measured (Zelikman et al., 2022; Hsieh et al., 2023). Our two gates correspond to these two operations — Gate 1 filters on the recorded verdict, Gate 2 ranks the survivors by an automatic grounding score. The observation that the direct-answer baseline reaches comparable evidence quality, at higher verdict accuracy, is specific to this recipe on this task and not a claim about the family; Section 11 lays out combinations we have not tried: stacking, iteration, and preference finishing (Rafailov et al., 2023). We build on LoRA (Hu et al., 2022) and QLoRA (Dettmers et al., 2023) for adaptation and serve with vLLM (Kwon et al., 2023). NLI artifacts. The hypothesis-only lookup control re-derives the class-prior artifact that hypothesis-only baselines expose in NLI datasets (Poliak et al., 2018; Gururangan et al., 2018); reporting it is important when the same requirement set repeats across documents. Attribution and citation quality. Evaluating whether model citations are verifiable and support the claim connects to work on generation with citations (Gao et al., 2023b), researching-and-revising attributions (Gao et al., 2023a), and verifiability in generative search (Liu et al., 2023). Those evaluate model- or NLI-judged support of free-text claims; we instead measure agreement with withheld human evidence spans and verbatim source verifiability, which suits the verdict-only, span-annotated setting of ContractNLI. Bridgewater AIA Labs and Thinking Machines (Su et al., 2026) report that expert judgment on financial triage can be learned by an open-weight model and deployed; their pipeline commissions labels through paid annotation and trains Qwen3-235B with reinforcement learning (GRPO, then a CISPO loss with asymmetric clipping) and on-policy distillation. We differ in that the verdict labels already exist rather than being commissioned, the human evidence layer is withheld rather than purchased, and the recipe is single-accelerator SFT rather than RL. The comparison motivates the focus on recorded decisions but is not a matched baseline. 10 Conclusion In verdict-only records, evidence quality is recoverable by post-training with no human evidence annotation. The two axes decouple across systems, yet both improve under verdict-only fine-tuning. We compared two verdict-only methods — a bare label-only SFT and an elaborate verdict-filtered, source-grounded rejection-sampling recipe — and both recover evidence quality; in this single run they land close on the evidence axis and further apart on accuracy, and a 2-epoch RSFT run reproduces the 1-epoch result, arguing against insufficient epoch count as the explanation for the gap. We do not rank the two methods: the estimates are single-seed on one corpus and one student, and the comparison is a starting point, not a verdict. What it does establish is that neither method needs evidence annotation, and that the same evidence quality is reached by two routes — label-only through the verdict, RSFT through the better citations its source-grounded supervision buys (Section 5). 11 Roadmap and future work This work establishes the two-axis framing and a clean comparison; the following extensions turn it into a program. Each is feasible on the same single-accelerator budget. 1. Second seeds and a wider model family. Run each arm at three seeds and report mean ± s.d.; repeat on a second student (a different 7–9B base) to separate method effects from a single checkpoint’s idiosyncrasies. Required before any point estimate here is stated as a headline. 2. A minimal-prompt citation test. Evaluate every fine-tune under verdict_only and faithful to measure how much citation behaviour is weight-resident vs prompt-elicited — the one dimension on which RSFT is expected to hold an advantage over label-only. 3. Preference finishing (DPO/ORPO). Use the grounding score and verdict-correctness to construct preference pairs and add a preference-optimisation stage (Rafailov et al., 2023) after SFT, targeting the E/C recall that both current fine-tunes leave on the table. 4. Semantic evidence credit. Add an embedding- or entailment-based span-matching rule beside the string ladder, so evidence quality is not measured with the same instrument used in selection. 5. More corpora. Add one or two additional verdict-plus-withheld-evidence datasets (e.g. other document-level NLI or claim-verification corpora) to test whether the verdict-mediation mechanism generalizes beyond ContractNLI. 6. Verifier and lookup ablations. Replace the recorded-verdict filter with the sampler pool’s modal answer to estimate the contribution of the recorded verdict, and report the companion lookup constructions. Limitations One corpus, one student, one seed per arm. Intervals quantify variation over test rows, not over training initialisations. A second seed per arm is outstanding; the +0.099 label-only-vs-RSFT accuracy gap is large relative to plausible seed variance but is a single-seed estimate. Label-only’s citations are prompt-elicited, not weight-resident. Label-only SFT emits evidence only because the faithful prompt asks for it; under verdict_only it cites nothing. RSFT trains citation behaviour into the target, so it should cite under a minimal prompt where label-only would not — a difference we have not tested (Section 11). In this study’s protocol (the faithful prompt) label-only cites at 93.4%, so the practical difference is nil here. Evidence F1 is partly a locatability measure. Selection and evaluation share the same three-tier string-matching ladder; the verbatim rate is the non-circular metric and we lead with it. The locatable-subset result (Section 5) shows the ranking survives removing the shared instrument, but a semantic credit rule (embedding- or entailment-based) is future work. Pretraining exposure cannot be excluded. ContractNLI has been public since 2021 and its evidence layer (559,215 binary span decisions) is public; fine-tuning can surface memorised content. This is most pressing for the label-only arm’s 0.896 accuracy — the result closest to the supervised figure (Section 4.1) — which is the one that most needs the closed-book probe, with and without document identity, that we leave to future work. Paired comparisons control for fixed differences between checkpoints on the same rows but do not eliminate contamination. Component effects are not isolated within RSFT. Verdict filtering, grounding selection, self-training, and sampler contribution are evaluated as one pipeline. References Anthropic (2026) Anthropic Claude Sonnet 4.6. Note: Anthropic announcementQueried via AWS Bedrock as us.anthropic.claude-sonnet-4-6 External Links: Link Cited by: Table 1. Dettmers et al. (2023) T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer QLoRA: efficient finetuning of quantized llms. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 10088–10115. External Links: Document, Link Cited by: §3, §9. Dong et al. (2023) H. Dong, W. Xiong, D. Goyal, Y. Zhang, W. Chow, R. Pan, S. Diao, J. Zhang, K. SHUM, and T. Zhang RAFT: reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §3, §9. Efron and Tibshirani (1994) B. Efron and R. J. Tibshirani An introduction to the bootstrap. 1st edition, Chapman and Hall/CRC. External Links: Document Cited by: §4.1, Table 7, §8. Gao et al. (2023a) L. Gao, Z. Dai, P. Pasupat, A. Chen, A. T. Chaganty, Y. Fan, V. Zhao, N. Lao, H. Lee, D. Juan, and K. Guu RARR: researching and revising what language models say, using language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 16477–16508. External Links: Link, Document Cited by: §9. Gao et al. (2023b) T. Gao, H. Yen, J. Yu, and D. Chen Enabling large language models to generate text with citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 6465–6488. Note: ALCE External Links: Link, Document Cited by: §9. Google DeepMind (2026) Google DeepMind Gemma 4 model card. Note: Google AI for Developers documentation External Links: Link Cited by: Table 1. Gulcehre et al. (2023) C. Gulcehre, T. L. Paine, S. Srinivasan, K. Konyushkova, L. Weerts, A. Sharma, A. Siddhant, A. Ahern, M. Wang, C. Gu, W. Macherey, A. Doucet, O. Firat, and N. de Freitas Reinforced self-training (rest) for language modeling. External Links: 2308.08998, Link, Document Cited by: §9. Gururangan et al. (2018) S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. R. Bowman, and N. A. Smith Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), M. Walker, H. Ji, and A. Stent (Eds.), New Orleans, Louisiana, p. 107–112. External Links: Link, Document Cited by: §2.2, §9. Han et al. (2023) D. Han, M. Han, and Unsloth team Unsloth. Note: Software External Links: Link Cited by: Table 13. Hsieh et al. (2023) C. Hsieh, C. Li, C. Yeh, H. Nakhost, Y. Fujii, A. Ratner, R. Krishna, C. Lee, and T. Pfister Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. In Findings of the Association for Computational Linguistics: ACL 2023, A. Rogers, J. Boyd-Graber, and N. Okazaki (Eds.), Toronto, Canada, p. 8003–8017. External Links: Link, Document Cited by: 1st item, §2.3, §3, §9. Hu et al. (2022) E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §3, §9. Koreeda and Manning (2021) Y. Koreeda and C. Manning ContractNLI: a dataset for document-level natural language inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, M. Moens, X. Huang, L. Specia, and S. W. Yih (Eds.), Punta Cana, Dominican Republic, p. 1907–1919. External Links: Link, Document Cited by: §1, §2.1, §4.1. Kwon et al. (2023) W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. Gonzalez, H. Zhang, and I. Stoica Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, New York, NY, USA, p. 611–626. External Links: ISBN 9798400702297, Link, Document Cited by: Table 13, §9. Lakens (2017) D. Lakens Equivalence tests: a practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science 8 (4), p. 355–362. Note: PMID: 28736600 External Links: Document, Link Cited by: §4.1. Liu et al. (2023) N. Liu, T. Zhang, and P. Liang Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 7001–7025. External Links: Link, Document Cited by: §9. McNemar (1947) Q. McNemar Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 12 (2), p. 153–157. External Links: Document, Link Cited by: Table 7, §8. OpenAI (2025) OpenAI Introducing GPT-5.2. Note: OpenAI announcementSnapshot gpt-5.2-2025-12-11 External Links: Link Cited by: Table 1. Poliak et al. (2018) A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, M. Nissim, J. Berant, and A. Lenci (Eds.), New Orleans, Louisiana, p. 180–191. External Links: Link, Document Cited by: §2.2, §9. Qwen Team (2026) Qwen Team Qwen3.5-9B. Note: Hugging Face model repository Qwen/Qwen3.5-9B External Links: Link Cited by: Table 1. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36, p. 53728–53741. External Links: Document, Link Cited by: item 3, §9. Schuirmann (1987) D. J. Schuirmann A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of Pharmacokinetics and Biopharmaceutics 15 (6), p. 657–680. External Links: ISSN 0090-466X, Document, Link Cited by: §4.1. Su et al. (2026) S. Su, K. Zhu, E. Xiao, R. Alur, and D. Kang Learning to replicate expert judgment in financial tasks. Thinking Machines Lab: News. Note: Bridgewater AIA Labs External Links: Link Cited by: §9. Touvron et al. (2023) H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom Llama 2: open foundation and fine-tuned chat models. External Links: 2307.09288, Link, Document Cited by: §9. Yuan et al. (2023) Z. Yuan, H. Yuan, C. Li, G. Dong, K. Lu, C. Tan, C. Zhou, and J. Zhou Scaling relationship on learning mathematical reasoning with large language models. External Links: 2308.01825, Link, Document Cited by: 2nd item, §3, §9. Zelikman et al. (2022) E. Zelikman, Y. Wu, J. Mu, and N. Goodman STaR: bootstrapping reasoning with reasoning. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35, p. 15476–15488. External Links: Document, Link Cited by: 2nd item, §2.3, §9. Appendix A Prompts and generation A.1 The two prompts Everything above ## OUTPUT is byte-identical between the two variants except the single evidence paragraph. Hashes: faithful f9318d34, verdict_only 7ada9e8d. faithful — full prompt (reasoning + citations); used by the main system and every evidence-axis evaluation: ⬇ You are given a contract and a hypothesis. ## CONTRACT document ## HYPOTHESIS hypothesis ## TASK Classify whether the hypothesis is entailed by, contradicting to, or not mentioned by (neutral to) the contract. Answer with exactly one of: Entailment, Contradiction, NotMentioned. Identify the evidence for your decision as quotes from the contract. A piece of evidence is a sentence, or a list item within a sentence. Evidence should be as concise as possible while being self-contained, so that a reader can understand it from the quote alone. Quote it verbatim. Evidence is only defined when the answer is Entailment or Contradiction; for NotMentioned, return an empty list. ## OUTPUT Reason step by step, then return your final answer as a single block: <answer> "hypothesis_id": "hypothesis_id", "answer": "Entailment | Contradiction | NotMentioned", "reasoning": "2-3 sentence explanation", "confidence_score": <1-10>, "source_attribution": ["excerpt": "verbatim quote from the contract"] </answer> The <answer> block must contain valid JSON only. verdict_only — faithful with the evidence paragraph removed and the output reduced to the verdict; used to train and evaluate label-only SFT on the accuracy axis: ⬇ You are given a contract and a hypothesis. ## CONTRACT document ## HYPOTHESIS hypothesis ## TASK Classify whether the hypothesis is entailed by, contradicting to, or not mentioned by (neutral to) the contract. Answer with exactly one of: Entailment, Contradiction, NotMentioned. ## OUTPUT Return your answer as a single block: <answer> "hypothesis_id": "hypothesis_id", "answer": "Entailment | Contradiction | NotMentioned" </answer> The <answer> block must contain valid JSON only. A.2 Candidate generation (RSFT) Candidates are produced in two passes. Greedy first from both models on every training row; then, only on rows where greedy produced nothing the verdict filter would accept, a sampled pass at temperature 1.0. Cost scales with the residue rather than the corpus. Decoding settings. The greedy pass is k=1k=1, T=0T=0 for both models. The sampled pass runs only on rows whose greedy output the verdict filter rejected: • Student (Qwen3.5-9B): k=32k=32, T=1.0T=1.0, top-p 0.95, top-k 20. • Sampler (Gemma-4-31B): k=8k=8, T=1.0T=1.0, top-p 0.95, top-k 64. Both models use a 900-token completion cap. Rows are ordered document-major so all 17 hypotheses about one contract are answered while it is in the prefix cache, which is most of why the whole generation fits in 6.8 GPU-hours. Evaluation decoding is greedy, single pass, 900-token cap for every system.