Paper deep dive
Rubrics as Privileged Information for Open-Ended Generation
Deepika Bablani, Ajay Gupta, Wanming Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/5/2026, 4:40:58 AM
Summary
The paper introduces Rubrics as Privileged Information (RuPI), an extension of On-policy Self-Distillation (OPSD) for open-ended generation tasks. Unlike traditional methods that use hard privileged information (ground truth) or scalar rewards (Rubric-as-Reward RL), RuPI uses soft privileged information in the form of rubrics to guide the teacher model's distribution. This provides dense, token-level supervision that outperforms both reference-completion distillation and reinforcement learning approaches on benchmarks like HealthBench and ResearchQA, while preserving general model capabilities.
Entities (10)
Relation Signals (8)
RuPI → evaluatedon → HealthBench
confidence 96% · show the effectiveness of using rubrics as PI for open-ended generation across Qwen and Llama model families and show that it outperforms rubric-as-reward (RaR) RL using HealthBench
RuPI → extends → OPSD
confidence 95% · We extend OPSD to open-ended generation using soft PI in the form of rubrics
Llama-3.1-8B-Instruct → usedin → HealthBench
confidence 94% · We also evaluate generalization to Llama-3.1-8B-Instruct ... on HealthBench
Qwen2.5-7B-Instruct → usedin → HealthBench
confidence 94% · We use Qwen2.5-7B-Instruct ... as our base model ... on HealthBench
RuPI → evaluatedon → ResearchQA
confidence 93% · evaluating on ResearchQA: soft rubric PI outperforms both reference-PI distillation and RaR RL
RuPI → outperforms → RaR
confidence 92% · RuPI beats RaR RL by up to +0.10 absolute score
RuPI → outperforms → Reference-PI
confidence 91% · beats reference-PI by +0.034 to +0.079 absolute score
RubricHub → sourceof → ResearchQA
confidence 85% · training on the RubricHub Science corpus and evaluating on ResearchQA
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations. We extend OPSD to open-ended generation using soft PI in the form of rubrics that guide preferences but admit many valid responses. Rubrics have served as scalar rewards for reinforcement learning (RL); we show that they provide substantially richer signal as dense PI for distillation, and contrary to intuition, soft rubric PI provides a larger and more effective training signal on student roll-outs than hard reference completion PI in this regime. A reference completion is one point in a set of valid responses, so distilling towards it over-constrains the student, while rubrics specify the preference structure shared across the set of valid responses. We show the effectiveness of using rubrics as PI for open-ended generation across Qwen and Llama model families and show that it outperforms rubric-as-reward (RaR) RL using HealthBench, a benchmark that grades open-ended health responses against physician-created rubrics, providing dense token-level supervision for open-ended tasks; RuPI beats RaR RL by up to +0.10 absolute score and, under matched recipe and KL direction, beats reference-PI by +0.034 to +0.079 absolute score across three models. We further show that these findings generalize to training on the RubricHub Science corpus and evaluating on ResearchQA: soft rubric PI outperforms both reference-PI distillation and RaR RL (66.6% vs. 64.2% and 57.6%).
Tags
Links
- Source: https://arxiv.org/abs/2608.02948v1
- Canonical: https://arxiv.org/abs/2608.02948v1
Trouble viewing inline? Open PDF directly →
Full Text
60,358 characters extracted from source content.
Expand or collapse full text
Rubrics as Privileged Information for Open-Ended Generation Deepika Bablani Ajay Gupta Wanming Chen Apple deepika_bablani@apple.com Abstract On-policy self-distillation (OPSD), where a single model acts as both student and teacher with different contexts, has shown promise in verifiable domains like math, where hard privileged information (PI) in the form of ground-truth answers structurally constrains valid continuations. We extend OPSD to open-ended generation using soft PI in the form of rubrics that guide preferences but admit many valid responses. Rubrics have served as scalar rewards for reinforcement learning (RL); we show that they provide substantially richer signal as dense PI for distillation, and contrary to intuition, soft rubric PI provides a larger and more effective training signal on student roll-outs than hard reference completion PI in this regime. A reference completion is one point in a set of valid responses, so distilling towards it over-constrains the student, while rubrics specify the preference structure shared across the set of valid responses. We show the effectiveness of using rubrics as PI for open-ended generation across Qwen and Llama model families and show that it outperforms rubric-as-reward (RaR) RL using HealthBench, a benchmark that grades open-ended health responses against physician-created rubrics, providing dense token-level supervision for open-ended tasks; RuPI beats RaR RL by up to +0.10+0.10 absolute score and, under matched recipe and KL direction, beats reference-PI by +0.034+0.034 to +0.079+0.079 absolute score across three models. We further show that these findings generalize to training on the RubricHub Science corpus and evaluating on ResearchQA: soft rubric PI outperforms both reference-PI distillation and RaR RL (66.6%66.6\% vs. 64.2%64.2\% and 57.6%57.6\%). Rubrics as Privileged Information for Open-Ended Generation Deepika Bablani Ajay Gupta Wanming Chen Apple deepika_bablani@apple.com 1 Introduction On-policy self-distillation (OPSD), introduced recently in (Zhao et al., 2026) and (Shenfeld et al., 2026), demonstrates that a model can effectively serve as its own teacher when conditioned on privileged information (PI), and this can enable learning from self-distillation using on-policy demonstrations. By minimizing the KL divergence between token distributions, the student internalizes what the teacher learned from the PI without needing that information during inference. Existing OPSD methods target verifiable domains like math and code and the teacher is conditioned on hard PI like ground-truth answers, verified solutions or expert demonstrations that structurally constrain valid continuations. Open-ended generation tasks like long form question answering, medical consultation, creative writing, etc. lack such verifiable PI. For such open-ended generation, the current practice is to either train specialized reward models (Ouyang et al., 2022), use LLM-as-a-judge in the reinforcement learning loop, or use rubrics as rewards for RL. While rubrics as rewards have shown promise, current approaches convert rubrics into a scalar reward (rubric-as-reward; Gunjal et al., 2026; Liu et al., 2026), diminishing the richness of the original criteria and the information they encode. We instead propose rubrics as PI (RuPI) and extend OPSD to open-ended generation for non-verifiable tasks using soft PI in the form of rubrics. We show that, for non-verifiable tasks, conditioning the teacher on high quality rubrics provides a stronger learning signal than using reference completions. A per-token KL diagnostic on the untrained base model quantifies this gap: when scored on the student’s on-policy roll-outs, rubric conditioning produces 1.7×1.7× more per-token KL than reference conditioning, a stronger gradient signal (we develop this mechanism in Section 4.2). We show empirically that using rubrics as privileged information (RuPI) substantially outperforms both reference-PI distillation and RaR GRPO across two model families on HealthBench, a benchmark that grades open-ended health responses against high-quality, physician-created rubrics, while preserving general capabilities (MMLU, GSM8K, IFEval, TruthfulQA). To show that the findings generalize, we replicate the study on a second domain: training on the Science split of RubricHub (Li et al., 2026), a large-scale rubric corpus, and evaluating on ResearchQA (Yifei et al., 2026), we show that RuPI outperforms reference PI and that dense distillation outperforms RaR RL (Section 5.5). 2 Background 2.1 On-policy self-distillation OPSD trains a student policy to match a teacher distribution where both are derived from the same base model under different conditioning. The teacher sees PI, whereas the student does not. Classical knowledge distillation (Hinton et al., 2015; Kim and Rush, 2016; Sanh et al., 2019) trains a student on fixed sequences generated by a teacher. Instead, on-policy distillation (Agarwal et al., 2024) uses teacher supervision on student generations. Both techniques require a separate, often more capable teacher model. OPSD eliminates the need for a separate teacher while preserving the dense per-token signal of knowledge distillation and the on-policy roll-out structure of OPD/RL. The training objective minimizes the KL divergence between the teacher and student distributions over response tokens generated on-policy from the student: ℒ=x∼,y^∼πS(⋅|x)[∑tD(πT(⋅∣x,PI,y^<t)∥πS(⋅∣x,y^<t))], 433.62pt$ =E_x ,\, y _S(·|x) [ _tD ( _T(· x,PI, y_<t)\; \|\; _S(· x, y_<t) ) ]$, (1) where x is a prompt sampled from dataset D, y y is an on-policy roll-out from the student policy πS _S, πT _T is the teacher policy conditioned on the same context plus privileged information PIPI, y^<t y_<t denotes the rollout prefix up to position t, and D is a divergence (forward or reverse KL). Throughout, we call DKL(πT∥πS)D_KL( _T\,\|\, _S) (teacher first) forward KL, the mass-covering orientation used by Zhao et al. (2026), and DKL(πS∥πT)D_KL( _S\,\|\, _T) (student first) reverse KL, the mode-seeking orientation used by Shenfeld et al. (2026) and in our Eq. 3. Prior work Zhao et al. (2026) trains a LoRA (Hu et al., 2022) student that shares base parameters with the teacher, while Shenfeld et al. (2026) full-finetunes with an exponential moving average (EMA) teacher. We study both and find that both improve over base in our setup. The forward-vs-reverse KL choice matters mainly out-of-domain. 2.2 Hard vs. soft privileged information The general notion of training a student model with information available to the teacher but not the student dates back to Vapnik and Vashist (2009) and was unified with distillation by Lopez-Paz et al. (2015). In OPSD, the PI takes the form of text injected into the teacher’s context. We distinguish PI by how strongly it constrains continuations. Hard PI constrains generations structurally: knowing “the answer is 42” makes most responses incoherent unless they produce 42, so it cannot be ignored. Soft PI biases preferences but admits many valid continuations: “advise the patient to find a mental health provider” biases toward including that advice but leaves freedom in style, structure, and content. The distillation signal is correspondingly diffuse under soft PI, spread across many good completions rather than concentrated on one, and we show that this is useful for open-ended tasks. 2.3 Rubrics as rewards vs. rubrics as PI Rubric-as-reward methods convert rubrics into a scalar RL reward by taking a weighted sum: R(y)=∑k=1Kwk⋅[criterionk satisfied by y]∑k=1Kwk, 433.62pt$ R(y)= _k=1^Kw_k· 1[criterion_k satisfied by y] _k=1^Kw_k$, (2) where each rubric criterion k carries a weight wkw_k and [⋅] 1[·] is a binary judge satisfaction label. This signal is sparse, one scalar per roll-out. Our approach uses the same rubric criteria as PI, providing dense, token-level supervision through the conditioned teacher’s distribution shift; rather than using the per-criterion weights, we group the criteria into “the response should” and “the response should NOT” lists and include both as natural-language guidance in the teacher’s system message. We show that this dense signal substantially outperforms the sparse reward signal in RL. 2.4 Additional related work Rubrics as rewards: Gunjal et al. (2026) (Rubrics as Rewards) and Liu et al. (2026) convert expert- or synthetically-generated rubrics into a scalar RL reward by aggregating per-criterion satisfaction and optimizing with GRPO; instead, we use the same rubric content as dense privileged information for distillation. RL post-training: RLHF (Ouyang et al., 2022) and RL from verifiable rewards (Guo et al., 2025), optimized with PPO (Schulman et al., 2017) or GRPO (Shao et al., 2024) and combined in open recipes such as Tülu 3 (Lambert et al., 2024), are the standard post-training tools widely adopted in practice. The EMA-teacher variant we study (Shenfeld et al., 2026) is also termed SDPO by Hübotter et al. (2026). 3 Rubrics for OPSD Teacher input (conditioned on rubric PI) [system] Evaluation criteria for this response: The response should: – Provide informative context on different causes of mild ear pain, such as infections, earwax buildup, and swimmer’s ear. – …(additional positive criteria) The response should NOT: – Fail to seek context on the standard ear drops given and recommend their use without it. – …(additional negative criteria) [query] my left ear is hurting for the last couple of days, it’s mild but definitely annoying…i have these standard ear drops from the pharmacy [response] student rollout y y Student input (no PI) [query] my left ear is hurting for the last couple of days, it’s mild but definitely annoying…i have these standard ear drops from the pharmacy [response] student rollout y y Figure 1: Teacher–student conditioning asymmetry on a HealthBench example. The teacher is given the rubric as a system message before the query; the student sees only the query. Both contexts include the same on-policy rollout y y generated by the student; the teacher scores it under its rubric-conditioned distribution rather than producing its own response. For the reference-PI baseline, the system message is replaced with "Reference response: ideal completion". We train the student with full fine-tuning, using an EMA of the student parameters for the teacher: θT←μθS+(1−μ)θT _T←μ _S+(1-μ) _T after each step (μ=0.02μ=0.02). Given a prompt x, rubric r, and the student’s on-policy roll-out y^∼πS(⋅|x) y _S(·|x), the teacher’s response is additionally conditioned on the rubric: πT(⋅∣x,r,y^<t) _T(· x,r, y_<t), whereas the student has no rubric access: πS,θ(⋅∣x,y^<t) _S,θ(· x, y_<t) (Figure 1 shows a real example). We use reverse KL with per-token clipping: ℒ(x,r,y^)=1|y^|∑tmin(τ,DKL(πS,θ(⋅∣x,y^<t)∥πT(⋅∣x,r,y^<t))), 433.62pt$ (x,r, y)= 1| y| _t \! (τ,\;D_KL ( _S,θ(· x, y_<t)\; \|\; _T(· x,r, y_<t) ) )$, (3) Reverse KL induces mode-seeking behavior, encouraging the student to focus on high-density regions of the teacher distribution instead of covering its full support. This keeps the student close to the base distribution while selectively adopting rubric-aligned behaviors, i.e., it enables targeted behavioral transfer while minimizing drift from the base policy, and we show that this helps in preserving the student’s instruction-following capability. Our LoRA variant uses adapters (r=64r=64, α=128α=128, all linear layers) with a frozen base-model teacher; the rank constraint provides implicit regularization. Zhao et al. (2026) originally use forward KL; we run the LoRA arm in both KL directions and find forward and reverse comparable in-domain, but recommend reverse KL because it better preserves out-of-domain capabilities (Section 5.3). We use per-token KL clipping at τ=5.0τ=5.0 and generate on-policy roll-outs from the student. The KL divergence at each response token is computed over the model’s full vocabulary distribution rather than a top-K approximation, so each roll-out produces a dense supervision signal across every response position. The hyperparameters r=64r=64, α=128α=128, EMA decay μ=0.02μ=0.02, and per-token KL clip τ=5.0τ=5.0 are inherited from Zhao et al. (2026) and Shenfeld et al. (2026); we do not re-tune them. 4 Experiments 4.1 Experimental Setup Evaluation We evaluate our method on HealthBench (Arora et al., 2025), a benchmark designed to measure the capabilities of AI systems for health. HealthBench includes 5,000 realistic health conversations, each with a custom physician-created rubric to grade model responses. Each rubric specifies a set of weighted criteria (positive criteria the response should satisfy and negative criteria it should avoid), which we use as the rubric privileged information r during training. We use 2000/500/2500 train/validation/test splits and report HealthBench rubric satisfaction following the official protocol.111Train and test have closely matching theme proportions (largest difference 0.030.03) and rubric sizes (mean 11.411.4 criteria). We also evaluate on the official 1,000-prompt HealthBench Hard subset (the most challenging slice, disjoint from our training pool). On the Hard subset this published score clips to 0 for almost every method at the open-source 7B/14B scale, so there we additionally report HealthBench’s non-clipped cluster-level score, which aggregates per-rubric-cluster satisfaction and remains informative across methods at our scale. All reported scores are bootstrap means, with standard errors ≤0.008≤ 0.008 on test (n=2,500n=2,500) and ≤0.020≤ 0.020 on val (n=500n=500), so test differences below ∼0.01 0.01 should be treated as within noise.222We verified that response length does not drive the method ranking on HealthBench: per-example Pearson correlations between response character length and rubric satisfaction score are small in magnitude (|r|≤0.08|r|≤ 0.08) across all methods, and per-method mean response lengths span only 1,1571,157–1,7001,700 characters on the Qwen 7B test set. A post-hoc length-adjusted score preserves the ordering reported in the tables below. Models and training We use Qwen2.5-7B-Instruct (Qwen et al., 2025) as our base model and additionally report scale-up results with Qwen2.5-14B-Instruct. We also evaluate generalization to Llama-3.1-8B-Instruct (Grattafiori et al., 2024). All distillation runs train for 200 steps on 2,000 HealthBench prompts with an effective batch size of 32 (32 unique prompts in a batch with one on-policy roll-out per prompt). For LoRA-based runs we attach low-rank adapters with r=64r=64 and α=128α=128, trained at a learning rate of 1e−51e-5. For full fine-tuning, we use an EMA teacher (decay rate μ=0.02μ=0.02) at a lower learning rate of 5e−65e-6 to compensate for the larger number of trainable parameters. We compare against a RaR GRPO (Shao et al., 2024) baseline that also sees 32 unique prompts per batch, with G=8G=8 completions per prompt for a total batch size of 256. The reward signal is provided by GPT-4.1 as the external LLM judge that evaluates each generation against the rubric criteria, returning a per-criterion binary satisfaction label that is aggregated into a scalar reward via the rubric’s point weights. We use the same judge (GPT-4.1) for the GRPO reward as for evaluation, so the RL baseline is not disadvantaged by any policy-judge mismatch. Note that GRPO consumes 8×8× as many on-policy generations per training prompt as RuPI (G=8G=8 vs. a single rollout). Thus, RuPI uses 8×8× fewer sampled rollouts per prompt while outperforming GRPO at matched prompt count. Each configuration was trained once; reported test scores are bootstrap means over evaluation examples from the validation-selected checkpoint. Cross-domain evaluation. We additionally evaluate each checkpoint on a four-task cross-domain suite using lm-evaluation-harness (Gao et al., 2024) with default settings: MMLU (Hendrycks et al., 2020) (5-shot, world knowledge across 57 subjects), GSM8K (Cobbe et al., 2021) (8-shot CoT, grade-school math reasoning), IFEval (Zhou et al., 2023) (zero-shot, verifiable instruction-following constraints), and TruthfulQA (Lin et al., 2022) (zero-shot, factual accuracy on adversarial questions) to check that gains on HealthBench do not come at the cost of loss of other general capabilities. MMLU, GSM8K, and TruthfulQA together probe whether distillation degrades the model’s underlying knowledge, reasoning, or factual calibration, while IFEval specifically tests whether teacher-style transfer has shifted the model away from following user instructions. 4.2 Rubrics provide trainable signal where reference PI does not Before training, we test a precondition for distillation to work: does conditioning the teacher on a soft rubric actually shift its output distribution relative to the unconditioned student? If the two distributions are nearly identical, the training signal at the start of training is weak. A rubric specifies abstract criteria rather than a target response, and this may not lead to a substantially different next-token distribution from the unconditioned generation. We measure this on the untrained base model. Using Qwen2.5-7B-Instruct as both teacher and student, we compute per-token KL divergence between the rubric-conditioned teacher and the unconditioned student over 213 held-out HealthBench prompts, averaged across the response tokens of an on-policy roll-out sampled from the student. The student’s natural roll-out is what the student will be trained on, and the per-token KL on it is the gradient signal that the student will actually see during training. We report two variants: Mean KL over all response tokens and Body KL, which restricts the average to the middle 80% of each response (positions [0.1n,0.9n][0.1n,0.9n]). The body restriction excludes opening boilerplate (greetings, “Sure, here’s…”) and closing boilerplate (sign-offs, end-of-sequence cues) at the response boundaries, where conditioning has little effect and the KL is dominated by stylistic regularities rather than content. We repeat the same for reference PI. Table 1: Per-token teacher-student KL on the base model (Qwen2.5-7B-Instruct), measured on roll-outs sampled from the student. Rubric PI produces 1.7×1.7× more per-token KL than reference PI. PI type Mean KL Body KL Physician rubrics 0.373 0.326 Reference completions 0.212 0.189 Both PI types produce substantial asymmetry, confirming the precondition: rubric conditioning shifts the teacher’s distribution by a meaningful amount even though the criteria never specify a target response. Table 1 shows that rubric PI provides a stronger training signal in the form of per-token KL on the student’s rollout than reference PI. To understand why reference PI is less useful for training, we measure how the teacher’s probability mass is distributed across the valid response set. For each held-out HealthBench prompt we score the teacher’s per-token negative log-likelihood (NLL) on the gold completion and on a diverse set of other rubric-satisfying responses (423423 held-out HealthBench prompts, 3,3523,352 GPT-4.1-generated candidates; full protocol in Appendix A). As Table 2 shows, under hard (reference) PI the gold is 1.181.18 nats more probable per token than other valid responses, whereas under soft (rubric) PI it is 0.700.70 nats less probable, the gold is thus exp(1.88)≈6.5× (1.88)≈ 6.5× more over-concentrated per token under hard PI. Reference PI piles mass on one arbitrary point in the valid set; rubric PI spreads it across the set the student is exploring. Table 2: Teacher per-token NLL under each PI conditioning (Qwen2.5-7B-Instruct teacher). Hard PI concentrates 6.5×6.5× more probability mass per token on the gold (relative to other valid responses) than soft PI; robust to the candidate generator (7.1×7.1× with an independent Qwen2.5-14B + rubric-LoRA generator). per-token NLL Rubric PI Reference PI gold completion 2.32 0.38 other good responses 1.62 1.56 over-concentration (other-good −- gold, nats) −0.70-0.70 +1.18+1.18 To make explicit how the signal depends on where the KL is scored, we re-run the per-token KL measurement of Table 1 with one variable changed: we score on the held-out gold completion instead of the student’s on-policy rollout. Everything else is identical. Rubric PI induces similar KL on student roll-outs and on gold completions (0.3260.326 vs. 0.3130.313), whereas reference PI induces far larger KL on the gold than on student roll-outs (1.6661.666 vs. 0.1890.189, an 8.8×8.8× gap) (Table 3). Since training optimizes KL on the student’s own trajectories, RuPI is the stronger learning signal, even though reference PI looks more powerful when scored at the gold. Table 3: Per-token Body KL (middle 80% of each response) between PI-conditioned teacher distributions on Qwen2.5-7B-Instruct, scored on either the student’s on-policy roll-out or the held-out gold completion. Rubric KL is invariant to the response choice; reference KL is 8.8×8.8× larger on the gold than on a student roll-out. PI type Student rollout Gold completion Rubric 0.3260.326 0.3130.313 Reference 0.1890.189 1.6661.666 5 Main Results 5.1 HealthBench: in-domain performance Tables 4, 5, and 6 report each method at its best-by-val checkpoint, evaluated on the 2,500-prompt held-out test split, and Figure 2 (Appendix B) shows the per-step validation trajectories. Table 4: Qwen2.5-7B-Instruct on HealthBench. Best (val-selected) checkpoint per method. “ckpt” is the val-selected step. Standard errors ≤0.008≤ 0.008 on test (n=2500=2500). Fwd/Rev: forward/reverse KL; FT: full fine-tuning. Method KL PI ckpt Val Test Base — — — 0.147 0.169 GRPO — rubric (reward) 50 0.178 0.193 LoRA Fwd rubric 200 0.223 0.215 LoRA Rev rubric 200 0.213 0.203 FT Fwd rubric 100 0.207 0.212 FT Rev rubric 200 0.197 0.205 LoRA Fwd reference 150 0.169 0.172 LoRA Rev reference 150 0.164 0.169 FT Fwd reference 200 0.168 0.165 FT Rev reference 200 0.151 0.159 SFT — rubric (RCG) 150 0.184 0.169 SFT — reference 200 0.140 0.136 Table 5: Qwen2.5-14B-Instruct on HealthBench. Best (val-selected) checkpoint per method. Fwd/Rev: forward/reverse KL; FT: full fine-tuning. Method KL PI ckpt Val Test Base — — — 0.247 0.269 GRPO — rubric (reward) 50 0.264 0.268 LoRA Fwd rubric 200 0.337 0.324 LoRA Rev rubric 150 0.334 0.325 FT Fwd rubric 200 0.363 0.347 FT Rev rubric 150 0.330 0.339 LoRA Fwd reference 200 0.287 0.274 LoRA Rev reference 100 0.275 0.267 FT Fwd reference 100 0.273 0.267 FT Rev reference 150 0.274 0.258 SFT — rubric (RCG) 250 0.236 0.226 SFT — reference 100 0.253 0.251 Table 6: Llama-3.1-8B-Instruct on HealthBench. Best (val-selected) checkpoint per method. Fwd/Rev: forward/reverse KL; FT: full fine-tuning. Method KL PI ckpt Val Test Base — — — 0.114 0.141 GRPO — rubric (reward) 50 0.112 0.116 LoRA Fwd rubric 150 0.169 0.197 LoRA Rev rubric 150 0.206 0.204 FT Fwd rubric 200 0.185 0.196 FT Rev rubric 200 0.204 0.213 LoRA Fwd reference 150 0.114 0.124 LoRA Rev reference 100 0.121 0.125 FT Fwd reference 50 0.108 0.104 FT Rev reference 50 0.135 0.134 SFT — rubric (RCG) 300 0.169 0.176 SFT — reference 300 0.196 0.189 RuPI outperforms RaR GRPO across all three base models. On the held-out test split, all four RuPI configurations beat GRPO on every base (Tables 4–6) while using 8×8× fewer on-policy generations per training prompt and no external judge. Notably, GRPO optimizes directly against the same GPT-4.1 rubric judge used to score it, yet is still beaten by RuPI, which never queries a judge during training—dense rubric distillation extracts more from a rubric than optimizing it as a scalar reward. The margin is largest on the weakest base (Llama: RuPI up to 0.2130.213 vs. GRPO 0.1160.116, +0.10+0.10) and narrows on Qwen 7B (+0.01+0.01 to +0.02+0.02), where the GPT-4.1-reward GRPO is a strong baseline. GRPO makes only a small gain over base on Qwen 7B (0.169→0.1930.169→0.193), is essentially flat on Qwen 14B (0.269→0.2680.269→0.268), and slightly declines on Llama (0.141→0.1160.141→0.116): its trajectory-level reward is too sparse to match dense distillation. Soft (rubric) PI outperforms hard (reference) PI consistently. Holding the recipe fixed (reverse KL), the best RuPI variant beats the best reference-PI variant on every base, by +0.034+0.034 (Qwen 7B) up to +0.079+0.079 (Llama 8B). This empirically realizes the diagnostic of Section 4.2: rubric PI produces a stronger, better-aimed KL signal on the student’s own roll-outs. The ordering also holds under forward KL (e.g., 14B full fine-tuning: RuPI 0.3470.347 vs. reference 0.2670.267; 7B: 0.2120.212 vs. 0.1650.165). Forward and reverse KL are comparable in-domain. Across recipes and scales the two directions land within ∼0.015 0.015 of each other in-domain; forward KL holds a slight edge for rubric variants on Qwen in most settings (LoRA 0.2150.215 vs. 0.2030.203 and full fine-tuning 0.2120.212 vs. 0.2050.205 at 7B; full fine-tuning 0.3470.347 vs. 0.3390.339 at 14B), with the 14B LoRA pair effectively tied (0.3240.324 vs. 0.3250.325). The choice has modest in-domain effect but matters out-of-domain, where we recommend reverse KL for capability preservation (Section 5.3). SFT baselines: information vs. mechanism. To separate the rubric information from the on-policy dense-KL mechanism, we add two supervised fine-tuning baselines (Tables 4–6, bottom). Reference-SFT fine-tunes on reference completions, the same target as reference PI, and RCG-SFT (rubric-conditioned-generation SFT) fine-tunes on the base model’s own responses generated with the rubric in context, with the rubric stripped at training time, so it derives its targets from the same rubric information that RuPI uses, injected offline as fixed targets.333Both are learning-rate swept over 5e−6,1e−5,2e−5,3e−5\5e-6,1e-5,2e-5,3e-5\ with best-by-val checkpoint selection. Neither recovers RuPI’s gain, i.e., matched rubric information delivered by offline SFT does not reproduce the rubric-distillation gain: the on-policy dense-KL mechanism, along with the rubric content, is what makes RuPI work. Out of domain (Table 11, 12, and 13) and HealthBench Hard results (Table 14) are in Appendix C and Appendix D, and further demonstrate that SFT performs worse than RuPI while causing significant regression in cross-domain evaluation. 5.2 HealthBench Hard We additionally evaluate each method’s best checkpoint on the official HealthBench Hard subset. At the open-source-7B-and-8B scale, the published rubric-satisfaction score collapses to 0 for almost every method, so we report HealthBench’s non-clipped facet scores. The two facets that drive our conclusions, the cluster-level aggregate (Cluster) and the accuracy axis (Acc), are in Table 7, reporting the strongest reference-PI and RuPI variant per metric. The remaining axes (communication quality, instruction following, completeness, context awareness) saturate near-uniformly across methods or clip to zero and are omitted. Table 7: HealthBench Hard. We report the cluster-level aggregate (Cluster) and the accuracy axis (Acc). RefPI / RuPI is the strongest reference-PI (RefPI) / rubric-PI (RuPI) variant per metric; bold is the best per row. Model Metric Base GRPO RefPI RuPI Qwen 7B Cluster 0.503 0.535 0.501 0.542 Acc 0.043 0.008 0.015 0.038 Qwen 14B Cluster 0.698 0.706 0.701 0.778 Acc 0.106 0.105 0.101 0.156 Llama 8B Cluster 0.532 0.515 0.545 0.625 Acc 0.000 0.000 0.000 0.077 The strongest separation on HealthBench Hard is on the accuracy axis, where RuPI consistently exceeds both GRPO and reference-PI distillation (Table 7). The effect is most pronounced on Llama 8B, where only RuPI variants achieve non-zero accuracy. RuPI also leads on the cluster-level aggregate for all three models. 5.3 Cross-domain capabilities To check that in-domain gains do not come at the cost of general capability, we evaluate each method’s best (val-selected) checkpoint on a four-task cross-domain suite (MMLU, GSM8K, IFEval, TruthfulQA). Three of the four tasks barely move across methods; the only axis that separates methods is IFEval, which we report as the change relative to base (Δ ) in Table 8, with the full four-task tables in Appendix C. Table 8: Change in IFEval relative to base (Δ ). Forward KL degrades instruction-following on Qwen; reverse KL stays near base; on Llama forward-KL FT leaves IFEval roughly flat. The Qwen-14B base scores below Qwen-7B on IFEval, a genuine instruction-following inversion between these instruct checkpoints, not a parsing artifact, so we compare only within-model Δ . Full four-task tables (MMLU, GSM8K, IFEval, TruthfulQA) in Appendix C. Fwd/Rev: forward/reverse KL; FT: full fine-tuning; ref: reference. Method Qwen 7B Qwen 14B Llama 8B GRPO (rubric) +0.013+0.013 −0.004-0.004 −0.015-0.015 LoRA-Fwd rubric −0.046-0.046 −0.041-0.041 +0.014+0.014 LoRA-Rev rubric +0.021+0.021 −0.032-0.032 −0.012-0.012 FT-Fwd rubric −0.038-0.038 −0.034-0.034 −0.000-0.000 FT-Rev rubric −0.007-0.007 −0.010-0.010 −0.010-0.010 LoRA-Fwd ref −0.009-0.009 −0.019-0.019 −0.006-0.006 LoRA-Rev ref +0.019+0.019 −0.019-0.019 +0.005+0.005 FT-Fwd ref −0.011-0.011 −0.006-0.006 +0.009+0.009 FT-Rev ref +0.021+0.021 −0.012-0.012 −0.012-0.012 Forward KL degrades IFEval on Qwen; on Llama both directions stay near base. Forward-KL distillation lowers IFEval relative to reverse KL on both Qwen scales: the largest drops are forward+rubric (−0.046-0.046 on 7B, −0.041-0.041 on 14B), with forward+reference also regressing on 14B (−0.019-0.019). Reverse-KL variants stay at or above base on Qwen 7B (rubric +0.021+0.021, reference +0.019+0.019) and within 0.0320.032 of base on 14B (−0.032-0.032 to −0.012-0.012). On Llama, IFEval moves little under both KL directions (−0.012-0.012 to +0.014+0.014), and forward KL does not degrade it the way it does on Qwen. The Qwen out-of-distribution behaviour is the main reason we recommend reverse KL despite forward KL’s slight in-domain edge (Section 5.1). We observe no material degradation on MMLU, GSM8K, or TruthfulQA (full numbers in Appendix C). 5.4 On-policy roll-outs are critical All RuPI results above train on the student’s own on-policy roll-outs (λ=1λ=1 in the GKD (Agarwal et al., 2024) sense: every training sequence is sampled from the current student). To test whether the rubric advantage depends on this choice, we re-run the full Qwen2.5-7B panel at λ=0λ=0, i.e., pure off-policy distillation, where the student is trained on roll-outs generated by the teacher rather than itself. We run λ=0λ=0 in both teacher regimes, a frozen cached teacher and an EMA teacher with regenerated targets, and find the two within noise of each other; we report the better value per cell here and defer the full 1212-run panel to Appendix F. Table 9: On-policy (λ=1λ=1) vs. off-policy (λ=0λ=0) distillation on Qwen2.5-7B-Instruct (HealthBench test, n=2500n=2500; base =0.169=0.169). On-policy values are the best rubric/reference variant from Table 4; off-policy values are the best of the cached/EMA λ=0λ=0 panel. Roll-out source rubric PI reference PI On-policy (λ=1λ=1) 0.215 0.172 Mixed (λ=0.5λ=0.5) 0.178 — Off-policy (λ=0λ=0) 0.154 0.156 Δ (on −- off) +0.061+0.061 +0.016+0.016 Every λ=0λ=0 variant lands at or below the base model (best off-policy 0.1560.156 vs. base 0.1690.169), and the rubric advantage collapses: rubric and reference PI are indistinguishable off-policy (0.1540.154 vs. 0.1560.156, within the ≤0.008≤ 0.008 test standard error), versus a +0.043+0.043 rubric margin on-policy. We also run the intermediate λ=0.5λ=0.5 mix; holding the recipe fixed (LoRA, forward KL), rubric quality rises monotonically with the on-policy fraction (0.154→0.167→0.2150.154→ 0.167→ 0.215 for λ=0,0.5,1λ=0,0.5,1). Off-policy distillation loses the rubric edge and fails to improve over the base, and the same flatness holds on HealthBench Hard (all λ=0λ=0 cluster scores 0.470.47–0.520.52, near the base 0.5030.503). This is in line with the mechanism in Section 4.2: off-policy training scores the teacher on its own roll-outs rather than the student’s, removing the on-policy coverage that gives rubric PI its edge, so the rubric advantage is a property of rubrics combined with on-policy roll-outs, not of rubrics alone. 5.5 Second domain: RubricHub Our HealthBench results are confined to a single domain (open ended medical Q&A). To test that the ordering, rubric PI >> reference PI (H1) and dense distillation >> RaR RL (H2), generalizes, we replicate the study on RubricHub (Li et al., 2026), a recent large-scale rubric corpus of ∼ 110K query–rubric pairs. We focus on RubricHub’s Science domain, evaluated with ResearchQA (Yifei et al., 2026), whose rubric-coverage metric is an open-ended analogue of HealthBench’s rubric grading. Setup RubricHub ships two aligned datasets: a rubric-conditioned RL set (queries with per-domain rubrics) and a rejection-sampling SFT set (queries with best-of-six GPT-5.1 answers). We take their intersection on query text, so every training prompt carries all three signals we need: the rubric (soft PI for RuPI), a best-of-six reference answer (reference PI), and the rubric criteria as the reward for the GRPO baseline. We train on 1,000 Science prompts on Qwen2.5-7B-Instruct, with recipes identical to HealthBench (LoRA in both KL directions, full fine-tuning with an EMA teacher; GRPO with the stabilized β=0.1β=0.1 recipe). We use ResearchQA’s official protocol: mean rubric-criterion coverage over its 3,750-question held-out test split with a gpt-4.1-mini judge, and its 703-question valid split for best-by-val checkpoint selection (sweep over steps 50/100/150/200). Table 10: RubricHub Science on ResearchQA, Qwen2.5-7B-Instruct. Best (val-selected) checkpoint per method Method KL PI ckpt Val Test Base — — — — 0.574 GRPO — rubric 200 0.570 0.576 LoRA Fwd rubric 200 0.658 0.660 LoRA Rev rubric 150 0.649 0.647 FT Fwd rubric 200 0.670 0.666 FT Rev rubric 50 0.650 0.644 LoRA Fwd reference 100 0.632 0.632 LoRA Rev reference 100 0.627 0.629 FT Fwd reference 200 0.639 0.642 FT Rev reference 200 0.630 0.634 The HealthBench ordering replicates cleanly here (Table 10). GRPO does not improve over base, echoing that RaR RL is a weak signal (H2). Soft-PI distillation, by contrast, moves ResearchQA substantially: the best RuPI variant scores +0.092+0.092 over base and +0.090+0.090 over GRPO. RuPI also beats reference PI under every matched recipe, realizing H1 in a second domain: rubrics, which characterize the whole set of good answers, transfer better than a single reference answer. Val and test track closely throughout, the ordering is decided on the val split and confirmed independently on test. The precondition diagnostic of Section 4.2, i.e. per-token teacher-student Body KL under rubric vs. reference PI on the student’s roll-outs, and the mass-concentration measurement are in Appendix G. 6 Conclusion We show that on-policy self-distillation extends to open-ended generation when rubrics serve as PI for the teacher. The central finding is that soft rubric PI provides a stronger and better-aimed training signal on the student’s roll-outs than hard reference completion PI. Reference conditioning sharpens the teacher’s mass at the gold, but the gold is one point in a large valid response space and the student’s roll-outs are far from it, so the conditioning effect on the student’s actual trajectories is weak. Soft rubric criteria specify preferences over the whole valid response space the student is exploring, so the same student roll-outs move under conditioning in ways that meaningfully match the rubric. Across three models, RuPI beats RaR GRPO on HealthBench (by up to +0.10+0.10), with 8×8× fewer on-policy generations per training prompt and no external judge, and the same pattern replicates on the held-out HealthBench Hard split. The findings also hold in a second domain: trained on RubricHub Science and evaluated on ResearchQA, RuPI outperforms both reference-PI distillation and RaR RL. Together, these results suggest that, for non-verifiable tasks where evaluation rubrics exist, rubrics are better used as dense PI for distillation than as sparse rewards for RL. Limitations (i) We evaluate on two open-ended domains, medical Q&A (HealthBench) and scientific Q&A (RubricHub Science, via ResearchQA), with the HealthBench study spanning two model families (Qwen2.5, Llama-3.1) and three sizes (Llama 8B, Qwen 7B, Qwen 14B). Both are rubric-coverage-graded question-answering tasks; extending the recipe to structurally different non-verifiable tasks, e.g., long-form creative writing or multi-turn assistant dialogue, is a natural next step. (i) LoRA hyperparameters (r=64r=64, α=128α=128), EMA decay (μ=0.02μ=0.02), and the KL clip threshold (τ=5.0τ=5.0) are inherited from prior work; these values worked well and so we did not do a hyperparameter sensitivity sweep. (i) Our cross-domain suite (MMLU, GSM8K, IFEval, TruthfulQA) covers knowledge, reasoning, instruction-following, and factual calibration; contamination-resistant instruction-following benchmarks such as IFBench (Pyatkin et al., 2026) would further strengthen the OOD claim. (iv) Our study assumes the rubric and the reference answer are both high-quality and aligned with the evaluation criteria: the premise of RuPI is that a good rubric characterizes the whole distribution of good answers, whereas a reference supplies only one such answer. When rubrics are low-quality, synthetically generated, or misaligned with what the downstream task actually grades, so that the rubric and the reference point at different targets, the ordering we report (rubric PI >> reference PI) need not hold, and rubric conditioning may even transfer criteria that the evaluation does not reward. Characterizing which rubrics help versus hurt, and automatically curating or repairing rubrics to meet this alignment precondition, is beyond our scope and left to future work. Ethical Considerations Our experiments include medical question-answering benchmarks, but the resulting models are research prototypes and should not be used for diagnosis, treatment, or clinical decision-making. RuPI transfers behaviors specified by rubrics; inaccurate, incomplete, biased, or malicious rubrics may therefore transfer harmful guidance or omissions. This risk is particularly important in high-stakes domains such as health. Any deployment would require domain-expert validation of the rubrics, safety evaluations beyond rubric satisfaction, monitoring for failures, and appropriate human oversight. Training multiple 7B–14B parameter models and using external LLM judges also incurs computational and environmental costs. RuPI uses fewer sampled rollouts and does not require an external judge during training relative to our GRPO baseline, but we do not provide a complete energy or carbon accounting. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246–21263. Cited by: §2.1, §5.4. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. (2025) Healthbench: evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775. Cited by: §4.1. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §4.1. L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, and A. Zou (2024) The language model evaluation harness. Zenodo. External Links: Document, Link Cited by: §4.1. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §4.1. A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. M. Hendryx (2026) Rubrics as rewards: reinforcement learning beyond verifiable domains. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.4. D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, et al. (2025) Deepseek-r1: incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Cited by: §2.4. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020) Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: §4.1. G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §2.1. E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §2.1. J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. K. Buening, C. Guestrin, et al. (2026) Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: §2.4. Y. Kim and A. M. Rush (2016) Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing, p. 1317–1327. Cited by: §2.1. N. Lambert, J. Morrison, V. Pyatkin, S. Huang, H. Ivison, F. Brahman, L. J. V. Miranda, A. Liu, N. Dziri, S. Lyu, et al. (2024) Tulu 3: pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124. Cited by: §2.4. S. Li, J. Zhao, H. Ren, Z. Wei, Y. Zhou, J. Yang, S. Liu, K. Zhang, and C. Wei (2026) Rubrichub: a comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 31320–31344. Cited by: §1, §5.5. S. Lin, J. Hilton, and O. Evans (2022) Truthfulqa: measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), p. 3214–3252. Cited by: §4.1. T. Liu, R. Xu, T. Yu, I. Hong, C. Yang, T. Zhao, and H. Wang (2026) Openrubrics: towards scalable synthetic rubric generation for reward modeling and llm alignment. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 17417–17437. Cited by: §1, §2.4. D. Lopez-Paz, L. Bottou, B. Schölkopf, and V. Vapnik (2015) Unifying distillation and privileged information. arXiv preprint arXiv:1511.03643. Cited by: §2.2. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1, §2.4. V. Pyatkin, S. Malik, V. Graf, H. Ivison, S. Huang, P. Dasigi, N. Lambert, and H. Hajishirzi (2026) Generalizing verifiable instruction following. Advances in Neural Information Processing Systems 38. Cited by: Limitations. Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2025) Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §4.1. V. Sanh, L. Debut, J. Chaumond, and T. Wolf (2019) DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. Cited by: §2.1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2.4. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2.4, §4.1. I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal (2026) Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897. Cited by: Appendix E, §1, §2.1, §2.1, §2.4, §3. V. Vapnik and A. Vashist (2009) A new learning paradigm: learning using privileged information. Neural networks 22 (5-6), p. 544–557. Cited by: §2.2. L. S. Yifei, A. Chang, C. Malaviya, and M. Yatskar (2026) Researchqa: evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics. Transactions of the Association for Computational Linguistics 14, p. 1344–1368. Cited by: §1, §5.5. S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. Cited by: §1, §2.1, §2.1, §3, §3. J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou (2023) Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: §4.1. Appendix A Mass-concentration measurement protocol This appendix details the per-token NLL measurement summarized in Section 4.2 (Table 2). For each held-out HealthBench prompt, we sample 88 candidate responses from GPT-4.1 at temperature 0.90.9 to form a diverse set of rubric-satisfying answers (423423 prompts with valid gold completions yield 3,3523,352 candidates). For each candidate and the held-out gold completion, we compute the teacher’s per-token negative log-likelihood under hard PI (system message == "Reference response: gold") and soft PI (system message == rubric criteria text). The teacher is the same Qwen2.5-7B-Instruct base model used elsewhere. The robustness check (7.1×7.1×) repeats the measurement with candidates drawn from an independent Qwen2.5-14B + rubric-LoRA generator on the same prompts. Appendix B Validation trajectories Figure 2 shows the per-step HealthBench validation trajectories from which the best-by-val checkpoints in Section 5.1 are selected. Figure 2: Per-step HealthBench validation trajectories (GPT-4.1 judge, T=1.0T=1.0) for each base model. Each line is a (recipe, PI, KL-direction) combination; the circled marker is that line’s best-by-val checkpoint, whose held-out test score is the value reported in Tables 4–6. The dashed black line is the base model. Appendix C Full cross-domain results Tables 11, 12, and 13 report the complete four-task cross-domain suite (MMLU, GSM8K, IFEval, TruthfulQA) for all three base models, at each method’s val-selected checkpoint. The body (Table 8) summarizes only the IFEval change, the single axis on which methods differ. Table 11: Cross-domain evaluation, Qwen2.5-7B-Instruct, at each method’s val-selected checkpoint (matching Table 4). GSM8K under strict-match scoring. Fwd/Rev: forward/reverse KL; FT: full fine-tuning. Method KL PI MMLU GSM8K IFEval TQA Base — — 0.716 0.751 0.567 0.648 GRPO — rubric 0.717 0.754 0.580 0.649 LoRA Fwd rubric 0.716 0.800 0.521 0.642 LoRA Rev rubric 0.716 0.783 0.588 0.636 FT Fwd rubric 0.717 0.789 0.529 0.649 FT Rev rubric 0.716 0.757 0.560 0.638 LoRA Fwd reference 0.716 0.782 0.558 0.645 LoRA Rev reference 0.715 0.744 0.586 0.647 FT Fwd reference 0.716 0.767 0.556 0.647 FT Rev reference 0.716 0.756 0.588 0.644 SFT — rubric (RCG) 0.718 0.715 0.488 0.613 SFT — reference 0.720 0.748 0.429 0.623 Table 12: Cross-domain evaluation, Qwen2.5-14B-Instruct, at each method’s val-selected checkpoint (matching Table 5). IFEval is prompt-level strict accuracy. The Qwen-14B base scores below the Qwen-7B base on IFEval (both strict, 0.5180.518 vs. 0.5670.567, and loose, 0.5910.591 vs. 0.6380.638) a genuine instruction-following inversion between these two instruct checkpoints, not a parsing artifact, so absolute IFEval is not comparable across scales; we interpret only the within-model change in IFEval (reported in the text), scoring base and trained checkpoints identically. GSM8K under strict-match scoring. Fwd/Rev: forward/reverse KL; FT: full fine-tuning. †RCG-SFT’s answer formatting lowers strict-match GSM8K; flexible-extract recovers it to 0.785 ≈ base, so this is a format shift, not a reasoning loss. Method KL PI MMLU GSM8K IFEval TQA Base — — 0.789 0.787 0.518 0.691 GRPO — rubric 0.788 0.781 0.514 0.692 LoRA Fwd rubric 0.787 0.757 0.477 0.680 LoRA Rev rubric 0.788 0.762 0.486 0.687 FT Fwd rubric 0.787 0.773 0.484 0.688 FT Rev rubric 0.787 0.770 0.508 0.690 LoRA Fwd reference 0.785 0.792 0.499 0.687 LoRA Rev reference 0.787 0.778 0.499 0.691 FT Fwd reference 0.788 0.789 0.512 0.688 FT Rev reference 0.788 0.779 0.506 0.687 SFT — rubric (RCG) 0.774 0.543† 0.366 0.604 SFT — reference 0.786 0.660 0.370 0.632 Table 13: Cross-domain evaluation, Llama-3.1-8B-Instruct, at each method’s val-selected checkpoint (matching Table 6). GSM8K under strict-match scoring. Fwd/Rev: forward/reverse KL; FT: full fine-tuning. Method KL PI MMLU GSM8K IFEval TQA Base — — 0.683 0.705 0.457 0.545 GRPO — rubric 0.684 0.763 0.442 0.545 LoRA Fwd rubric 0.683 0.703 0.471 0.567 LoRA Rev rubric 0.685 0.713 0.445 0.566 FT Fwd rubric 0.685 0.713 0.457 0.568 FT Rev rubric 0.687 0.699 0.447 0.563 LoRA Fwd reference 0.685 0.705 0.451 0.561 LoRA Rev reference 0.684 0.701 0.462 0.559 FT Fwd reference 0.685 0.698 0.466 0.556 FT Rev reference 0.685 0.691 0.445 0.554 SFT — rubric (RCG) 0.684 0.740 0.462 0.563 SFT — reference 0.680 0.752 0.427 0.568 Appendix D SFT baselines on HealthBench Hard Table 7 compares the distillation and RL methods on the two informative HealthBench Hard facets; here we report the same two facets for the two SFT baselines, alongside Base and the strongest RuPI variant in Table 14 (repeated from Table 7) for reference. Consistent with the in-domain results, RCG-SFT (which carries rubric information) exceeds reference-SFT on both facets at all three scales. RuPI leads the cluster-level aggregate at every scale. On the accuracy axis RuPI leads on Qwen 7B and Llama 8B; the one exception is the Qwen-14B accuracy axis, where RCG-SFT’s offline rubric-conditioned targets give it an edge (0.2220.222 vs. 0.1560.156). On Llama both SFT baselines register small non-zero accuracy (0.0550.055–0.0640.064), below RuPI (0.0770.077) but above the reference-PI and GRPO distillation variants, which score zero (Table 7). Table 14: SFT baselines on HealthBench Hard: cluster-level aggregate (Cluster) and accuracy axis (Acc), at each arm’s val-selected checkpoint. ref-SFT: reference-SFT; RCG-SFT: rubric-conditioned-generation SFT. Base and RuPI (strongest variant) repeat Table 7 for reference. Bold: best in row among the four columns shown. Model Metric Base RuPI ref- SFT RCG- SFT Qwen 7B Cluster 0.503 0.542 0.464 0.528 Acc 0.043 0.038 0.033 0.036 Qwen 14B Cluster 0.698 0.778 0.677 0.696 Acc 0.106 0.156 0.112 0.222 Llama 8B Cluster 0.532 0.625 0.606 0.610 Acc 0.000 0.077 0.055 0.064 Appendix E Effect of per-token KL clipping The per-token KL clip (τ=5.0τ=5.0) is the one hyperparameter where reverse KL is in principle most sensitive: the surrogate loss produces per-token weights with potentially heavy tails when student and teacher diverge, so a few outlier tokens could dominate the gradient. To check that clipping is necessary rather than cosmetic, we re-run the two reverse-KL rubric configurations on Qwen2.5-7B-Instruct with clipping effectively disabled (τ=106τ=10^6, well above any per-token KL observed during training). Scores are HealthBench test (n=2500n=2500, GPT-4.1 judge, T=1.0T=1.0) at the val-selected checkpoint. Table 15: Effect of per-token KL clipping on Qwen2.5-7B-Instruct (HealthBench test). Removing the clip moves the score by ≤0.014≤ 0.014, well below the method-level effects. Method (rubric PI, reverse KL) Clip (τ=5τ=5) No clip LoRA 0.203 0.199 FT 0.205 0.219 Removing the clip changes the converged HealthBench test score by at most 0.0140.014: RuPI (LoRA) is essentially flat (0.203→0.1990.203→ 0.199) and RuPI (full fine-tuning) shifts only marginally (0.205→0.2190.205→ 0.219). The larger shift (0.0140.014) exceeds the test standard error (≤0.008≤ 0.008), but is an order of magnitude smaller than the method-level differences we report (0.030.03–0.080.08). The clip threshold τ=5.0τ=5.0 inherited from prior work does not appear to be necessary at this scale; we retain it in the main runs for consistency with prior recipes (Shenfeld et al., 2026). Appendix F Full λ-ablation panels (λ=0λ=0 and λ=0.5λ=0.5) Table 16 reports all twelve off-policy (λ=0λ=0) runs on Qwen2.5-7B-Instruct underlying the contrast in Section 5.4: LoRA and full fine-tuning × rubric and reference PI × forward and reverse KL, with full fine-tuning additionally run under both a frozen cached teacher and an EMA teacher with regenerated targets. (The LoRA arm was run cached-only.) All values are HealthBench test scores (n=2500n=2500, GPT-4.1 judge, T=1.0T=1.0) at the val-selected checkpoint; “Cluster” is the non-clipped cluster-level aggregate. The base model scores 0.1690.169 (cluster 0.6350.635). No configuration exceeds base, and within each PI type the cached and EMA teachers agree to within the test standard error (≤0.008≤ 0.008), confirming the on-/off-policy axis, not the teacher regime, drives the gap in Section 5.4. Table 16: Full λ=0λ=0 off-policy panel, Qwen2.5-7B-Instruct (HealthBench test, n=2500n=2500). Base =0.169=0.169 (cluster 0.6350.635). Recipe PI KL / Teacher Test Cluster LoRA rubric fwd / cached 0.154 0.633 LoRA rubric rev / cached 0.150 0.608 LoRA reference fwd / cached 0.156 0.622 LoRA reference rev / cached 0.147 0.599 Full FT rubric fwd / cached 0.147 0.627 Full FT rubric fwd / EMA 0.140 0.616 Full FT rubric rev / cached 0.153 0.624 Full FT rubric rev / EMA 0.143 0.603 Full FT reference fwd / cached 0.147 0.609 Full FT reference fwd / EMA 0.146 0.608 Full FT reference rev / cached 0.146 0.611 Full FT reference rev / EMA 0.147 0.608 Table 17 reports the λ=0.5λ=0.5 mixed panel on Qwen2.5-7B-Instruct (rubric PI only; reverse and forward KL; LoRA and full fine-tuning, the latter with cached and EMA teachers). Each step draws half its roll-outs on-policy from the student and half from the teacher. The best λ=0.5λ=0.5 rubric variant reaches 0.1780.178 test, above every λ=0λ=0 run (≤0.156≤ 0.156) and below the λ=1λ=1 best (0.2150.215), placing the λ=0.5λ=0.5 midpoint on the monotonic on-policy trend of Section 5.4. Table 17: λ=0.5λ=0.5 mixed-rollout panel, Qwen2.5-7B-Instruct, rubric PI (HealthBench test, n=2500n=2500; val-selected checkpoint). Base =0.169=0.169 (cluster 0.6350.635). Recipe PI KL / Teacher Test Cluster LoRA rubric fwd / cached 0.167 0.617 LoRA rubric rev / cached 0.177 0.623 Full FT rubric fwd / cached 0.161 0.631 Full FT rubric fwd / EMA 0.178 0.637 Full FT rubric rev / cached 0.173 0.624 Appendix G RubricHub KL / injectability diagnostic We repeat the precondition test of Section 4.2 on RubricHub Science before training: does rubric conditioning shift the teacher’s next-token distribution on the student’s on-policy roll-outs more than reference conditioning? Using Qwen2.5-7B-Instruct as both teacher and student, we compute per-token teacher–student KL (Mean, and Body over the middle 80% of each response) under rubric PI and best-of-six reference PI, on 100 held-out Science prompts, following the identical protocol of Section 4.2. As Table 18 shows, rubric PI produces 1.8×1.8× more Body KL than reference PI (0.3280.328 vs. 0.1820.182 nats), closely matching the HealthBench ratio (1.7×1.7×) and its absolute magnitude (0.3260.326 Body KL). The precondition for RuPI therefore holds on Science, consistent with the training results in Table 10. Table 18: Per-token teacher-student KL on base Qwen2.5-7B-Instruct for RubricHub Science, scored on student on-policy roll-outs (rubric PI vs. best-of-six reference PI), over 100 held-out prompts. Body KL is the mean over the middle 80% of each response. PI type Mean KL Body KL Rubric PI 0.3760.376 0.3280.328 Reference PI 0.1990.199 0.1820.182 Ratio (rubric/ref) 1.9×1.9× 1.8×1.8× We also repeat the mass-concentration measurement of Section 4.2 (Appendix A) on RubricHub Science: for each of 100 held-out prompts we take the best-of-six answer as the gold, sample 88 candidate responses (GPT-4.1, T=0.9T=0.9), keep the rubric-satisfying ones (325325 candidates), and measure the base teacher’s per-token NLL gap between these other-good responses and the gold under each PI (positive implies that the teacher assigns the gold lower NLL, i.e., higher probability than the other valid responses and the probability mass is concentrated on the gold). As Table 19 shows, under hard (reference) PI the gold is 0.840.84 nats more probable per token than other valid responses, whereas under soft (rubric) PI the gap is ≈0≈ 0 (+0.02+0.02): the gold is thus exp(0.82)≈2.3× (0.82)≈ 2.3× more over-concentrated per token under hard PI. This mirrors the HealthBench mechanism (6.5×6.5×): reference PI places probability mass on one arbitrary point in the valid set, while rubric PI spreads it across the set the student explores, which is why rubric PI yields the stronger per-token training signal in Table 18. Table 19: Per-token mass concentration on base Qwen2.5-7B-Instruct for RubricHub Science (100 held-out prompts, 325 rubric-satisfying candidates). Over-concentration is the mean per-token NLL of other-good responses minus that of the gold; a larger positive value means the teacher concentrates more mass on the single gold answer. PI type Over-concentration (other-good −- gold, nats) Soft (rubric) PI +0.02+0.02 Hard (reference) PI +0.84+0.84 Appendix H GRPO baseline configuration Our RaR GRPO baseline uses the following recipe: KL coefficient β=0.1β=0.1, learning rate 5e−65e-6, sampling temperature 1.21.2, and G=8G=8 completions per prompt over 3232 unique prompts per batch (256256 generations per step), trained for 200200 steps. We set β=0.1β=0.1 rather than the more common β=0.04β=0.04 or β=0β=0 because the un-penalized run was unstable: it reward-hacked the judge, converging to degenerate non-English output. The stabilized β=0.1β=0.1 recipe was the strongest usable GRPO baseline we obtained. The reward is the same GPT-4.1 rubric judge used for evaluation, so the baseline is not disadvantaged by any policy–judge mismatch; this is the identical recipe used for RubricHub (Section 5.5), differing only in the training corpus. We save a checkpoint every 5050 steps and select the one with the highest validation rubric-satisfaction (val split, n=500n=500).