Paper deep dive
RELAY: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning
Kunwang Zheng, Xinyu Sun, Yu Cao, Shuocheng Li, Lei Cheng, Ying Liu, Qilong Zhang, Jinlong Chen, Hua Yang, Qianglong Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group's responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. RELAY uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model-domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. RELAY scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient.
Tags
Links
- Source: https://arxiv.org/abs/2610.17623v1
- Canonical: https://arxiv.org/abs/2610.17623v1
Trouble viewing inline? Open PDF directly →
Full Text
69,785 characters extracted from source content.
Expand or collapse full text
Relay: Selective Self-Distillation with Adaptive Hand-off for Rubric-Based Reinforcement Learning Kunwang Zheng1,*, Xinyu Sun2,*, Yu Cao3, Shuocheng Li3, Lei Cheng3 Ying Liu3, Qilong Zhang3, Jinlong Chen3, Hua Yang3, Qianglong Chen3,† 1University of Science and Technology of China 2Fudan University 3Alibaba Group *Equal contribution. †Corresponding author. Email: kunwang392@gmail.com sunxy23@m.fudan.edu.cn qianglong.cql@alibaba-inc.com Project: github.com/kunwang392/RELAY Figure 1: The signal Relay reads (Qwen3-1.7B/Medicine). Top: in-training validation score; the three distillation runs share one Stage-1 trajectory and branch where they retire the teacher. Bottom: divergence on distilled tokens, which flattens before the rule fires at step 171. Abstract Rubric-based reinforcement learning grades a response against many written criteria but collapses them into one reward, which discards the detail the rubric carries and gives no GRPO reward gradient when a group’s responses tie. A frozen teacher that reads the rubric as privileged context provides token-level supervision, but we observe an early plateau in the gains over GRPO. Relay uses advantages and rewards to select positive-advantage and low-scoring zero-advantage rollouts, and monitors the distillation loss to decide when to retire the teacher. Stage 1 combines selective distillation from a frozen teacher with GRPO; a plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Calibrated on one model–domain pair and reused unchanged in three others, the plateau detector fires between steps 122 and 171. Relay scores highest in-domain among the compared methods in all four settings, ahead of GRPO on RaR-Medicine by 4.62 points with Qwen3-1.7B and 6.02 with Qwen3-4B. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it. Distilling selected rollouts covering about half the response tokens also outperforms distilling all rollouts at a matched average coefficient. 1 Introduction Rubric-based reinforcement learning trains language models on open-ended tasks by rewarding responses that satisfy instance-specific criteria (Gunjal et al., 2025; Viswanathan et al., 2025; Arora et al., 2025). An LLM judge collapses the criteria into a scalar reward for GRPO (Shao et al., 2024), discarding the rubric’s account of which facts to include and which errors to avoid. Self-distillation with privileged context (Zhao et al., 2026; Hübotter et al., 2026; Ye et al., 2026) returns that detail as token-level supervision: a teacher that reads the rubric scores the policy’s own rollouts, while the policy sees only the question (Bablani et al., 2026; Rezaei et al., 2026; Gu et al., 2026). Hybrid methods add this supervision to GRPO; our main baseline, CriPO, distills each group’s highest-advantage rollout toward a current-policy teacher and decays its coefficient to zero late in training (Xia et al., 2026). A hybrid inherits two choices the reward does not settle: which rollouts the teacher supervises, and how long it stays. On Qwen3-1.7B/Medicine, distilling every rollout produces longer responses and lower scores than a selective rule (§5.4). Continuing distillation throughout training reaches a lower peak score than stopping it partway and requires a teacher forward pass at every step (§5.1). Graded rubric rewards make these choices less straightforward. Fixed decay schedules and separate distillation and RL budgets commit to a stopping time before training starts (Xia et al., 2026; Yang et al., 2026; Wang et al., 2026c; Wang et al., 2026b), whereas the step at which distillation stops progressing moves from setting to setting. Selection rules built for verifiable rewards target groups in which every rollout fails (Han et al., 2026), but graded rubric rewards tie at any score: at the initial Qwen3-1.7B policy 10.9% of RaR-Medicine groups are tied, a third of them at a nonzero score, and none carries a reward gradient. When to retire the teacher is visible in the distillation loss itself. On Qwen3-1.7B/Medicine, the teacher–student divergence on distilled tokens falls by about two thirds and then flattens (Figure 1, bottom), in the same stretch in which the validation lead over GRPO stops growing (§5.3). Retiring the teacher there ends above continuing (Figure 1, top; §5.1), and every later step runs without the teacher pass. Relay uses rewards and advantages to select rollouts and the relative decrease in the distillation loss to decide when to retire the teacher (Figure 2). Stage 1 adds privileged distillation to GRPO on positive-advantage rollouts plus low-scoring zero-advantage rollouts. These rollouts cover about half the response tokens. In the selection ablation, distilling them yields shorter responses and higher scores than distilling every rollout, even at a matched average coefficient (§5.4). A plateau detector then retires the teacher, and Stage 2 continues with GRPO alone. We train Qwen3-1.7B and Qwen3-4B separately on medicine and science and evaluate the four resulting models on five benchmarks with GPT-5.1. Relay has the highest in-domain score among the compared methods in all four settings, scoring 46.77 and 64.43 on RaR-Medicine, ahead of GRPO by 4.62 and 6.02 points and of our CriPO reproduction by 4.40 and 4.76. Calibrated once on Qwen3-1.7B/Medicine and reused unchanged, the plateau rule retires the teacher between steps 122 and 171 across the four settings. On Qwen3-1.7B/Medicine, removing the teacher from a shared checkpoint outperforms keeping, weakening or refreshing it (§5.1). Our contributions are: • A stopping rule that tracks the relative decrease in the distillation loss during training and requires no additional evaluation. • An advantage-aware selection rule that supervises low-scoring zero-advantage rollouts, which receive little or no reward-driven gradient, while controlling the distilled-token fraction. • An evaluation over four model–domain settings in which Relay scores highest in-domain under GPT-5.1, with the highest five-benchmark aggregate at both model sizes. 2 Related Work Rubrics as rewards and context. Rubric-based RL scores each criterion independently and aggregates the result into one reward (Gunjal et al., 2025; Viswanathan et al., 2025); later work studies how criteria are written and how they can be gamed (Rezaei et al., 2025; Li et al., 2026c; Shao et al., 2026; Lan et al., 2026; Tan et al., 2026; Mahmoud et al., 2026). Two lines put token-level detail back into the reward: RTT adds a token-level reward term from a trained relevance discriminator (Xu et al., 2026), and CoRT reweights the advantage by the log-probability contrast between prompts with and without criteria (Zhang et al., 2026a). RuscaRL instead puts the rubric in the policy’s own context and withdraws it on a fixed schedule (Zhou et al., 2026), and HeRL feeds unmet criteria back for revision (Zhang et al., 2026b), so both sample with criteria the trained policy does not see. Relay keeps the reward and advantage unchanged and passes rubric information to the policy through the teacher’s token distributions, without adding the rubric to the policy’s context. Rubric-conditioned self-distillation. On-policy distillation supervises the student’s own samples with a teacher’s token distributions (Agarwal et al., 2024; Gu et al., 2024; Li et al., 2026d; Song and Zheng, 2026). Recent self-distillation methods use a teacher derived from the student model and conditioned on privileged context (Zhao et al., 2026; Hübotter et al., 2026; Ye et al., 2026). Several studies report early gains followed by degradation under continued self-distillation (Kim et al., 2026b; Zhu et al., 2026; Pan et al., 2026). Three recent methods condition the teacher on rubrics, distilling from an EMA of the student or, with LoRA, the frozen base (Bablani et al., 2026), a frozen base model (Rezaei et al., 2026), or a teacher reading a learned rubric (Gu et al., 2026). In all three, teacher supervision replaces the judge or verifier; we study its use alongside rubric-based RL. In our experiments, standalone distillation from a frozen teacher scores below GRPO in all four in-domain settings (Table 2), motivating the use of both objectives. Rollout selection. Hybrids distill positive-advantage rollouts (Liu et al., 2026), incorrect rollouts routed to a separate branch (Li et al., 2026b), annotated spans (Wang et al., 2026a), or each group’s highest-advantage rollout (Xia et al., 2026); others add no distillation term and instead modify the token advantages with a teacher signal (Yang et al., 2026; Kim et al., 2026a; Pan et al., 2026). Apart from CriPO these designs assume binary verifiable rewards, whereas graded rubric rewards let a group tie at any score. Prior work discards them and oversamples to refill the batch (Yu et al., 2025; Pan et al., 2026), adds a separately normalized advantage that survives when the outcome advantage vanishes (Tan et al., 2026), or distills a stronger external teacher into the groups scoring zero (Han et al., 2026). Relay supervises them from the initial model itself and selects them by a running cutoff rather than at zero, which also covers ties at nonzero scores. RLCSD normalizes its loss separately over tokens with and without advantage modulation (Pan et al., 2026), while STEPS caps token coverage and decays the distillation weight (Wang et al., 2026a). We average the distillation loss over all response tokens, so selection also changes the average coefficient. We compare selection rules at matched average coefficients (§5.4). Stopping teacher supervision. Prior work ends teacher supervision on a fixed decay (Xia et al., 2026; Liu et al., 2026; Wang et al., 2026a; Yang et al., 2026), by annealing that never reaches zero (Han et al., 2026), at the end of a distillation phase whose length is fixed (Wang et al., 2026c) or swept (Wang et al., 2026b; Li et al., 2026a). Two act on the observation our rule automates: OPD-before-RL motivates stopping by the student reaching a plateau below its teacher (Wang et al., 2026b), and Sequential-Beats-Joint sweeps candidate steps to place its switch at the end of the fast-improvement phase, finding a hard switch better than annealing (Li et al., 2026a). Its sequential arm runs pure distillation before pure RL with a teacher larger than the student, and it leaves the self-distillation case open. Relay combines distillation with RL, then switches to RL alone at a step chosen online from the distillation loss. RetireOPD is the closest design, retiring the teacher in agentic RL once a windowed relative change in the sampled-token gap stops being negative and the student’s success rate reaches 0.90.9 of the teacher’s (Yu et al., 2026). Both methods monitor relative change over windows and remove the teacher after the switch. Our rule can stop while the divergence is still decreasing, once its relative decrease remains below a tolerance. RetireOPD also requires student and teacher success rates on a validation set; our statistic is computed during training. The rule stops only the distillation term; RL training continues. Most designs above are concurrent preprints; we compare designs and reproduce CriPO as our hybrid baseline. 3 Method Relay trains in two stages. In Stage 1, a frozen copy of the initial model reads the rubric and scores tokens sampled by the student. Rewards and advantages determine which rollouts receive an additional distillation loss; the GRPO objective is retained for all valid response tokens. We monitor the mean distillation loss on selected tokens and stop distillation when its relative decrease remains small. Stage 2 continues with GRPO alone. The teacher scores existing tokens by teacher forcing and requires no decoding. Figure 2 summarizes the method and Algorithm 1 gives the loop. 3.1 Preliminaries Rubric-based GRPO. Each prompt x has a rubric ρ=(cj,ωj)j=1Mρ=\(c_j, _j)\_j=1^M of natural-language criteria with signed weights: positive weights specify requirements and negative weights specify penalties. The rollout policy πθold _ _old samples a group of G rollouts y1,…,yGy_1,…,y_G, and an LLM judge checks their criteria. Both positive requirements and negative penalties contribute to the reward, using the same signed rubric as the teacher: ri r_i =clip[0,1](∑j=1Mωj[yi⊧cj]∑j:ωj>0ωj), =clip_[0,1]\! ( _j=1^M _j\,I[y_i c_j] _j: _j>0 _j ), (1) Ai A_i =ri−μrσr+ϵσ, = r_i- _r _r+ _σ, The indicator is one when a positive requirement is met or a negative penalty is triggered. Here μr _r and σr _r are the group mean and standard deviation, and ϵσ=10−6 _σ=10^-6. Every response token shares AiA_i, and all advantages are zero in a tied group. Let V be the valid response-token positions in an update batch and N=||N=|V|. ℒGRPOL_GRPO is the clipped policy objective (Schulman et al., 2017) averaged over all of V, plus β times a token-averaged k3k_3 penalty against the initial model π0 _0 on the original context (Appendix A). Each rollout step supplies U=2U=2 gradient updates, each over half the sampled prompts. 3.2 The privileged teacher and its objective A frozen teacher. The student answers without the rubric. The teacher is the initial model conditioned on the prompt and the full signed rubric, πT(⋅∣x,ρ,y<h)=π0(⋅∣[x;ρ],y<h) _T(· x,ρ,y_<h)= _0(· [x;ρ],y_<h), with an appended instruction to satisfy requirements and avoid penalties (Appendix A). In our generation diagnostic, the rubric-conditioned teacher substantially outperforms the student (§5.3). Freezing it fixes its predictions for a given prompt and prefix; the prefixes sampled by the student still change during training. The initial model also serves as the GRPO KL reference, so the teacher requires no additional model weights. Distillation objective. At position (i,h)(i,h), let Ki,hK_i,h hold the teacher’s top-k next tokens, with k=64k=64. We renormalize both distributions on this set and minimize their conditional forward KL: ℓi,h=∑v∈Ki,hp¯T(v)logp¯T(v)p¯θ(v), _i,h= _v∈ K_i,h p_T(v) p_T(v) p_θ(v), (2) where p¯(v)=p(v)/∑v′∈Ki,hp(v′) p(v)=p(v)/ _v ∈ K_i,hp(v ) for either distribution at that prefix. Within Ki,hK_i,h, forward KL penalizes the student for assigning low conditional probability to tokens favored by the teacher. Renormalization leaves the student’s total probability on this set unconstrained: adding the same constant to its logits in Ki,hK_i,h leaves the loss unchanged, and logits outside the set receive no direct gradient from it (Appendix G). This loss does not determine full-vocabulary entropy; we examine the observed entropy and length changes in §5.1. At rollout step t the combined objective is ℒ=ℒGRPO+αtℒOPSD, =L_GRPO+ _t\,L_OPSD, (3) ℒOPSD=1N∑(i,h)∈wiℓi,h, _OPSD= 1N _(i,h) w_i _i,h, (4) with the same token mask and denominator as Eq. (8). The binary weights wiw_i come from §3.3 and the coefficient αt _t becomes zero at the hand-off (§3.4). Figure 2: Overview of Relay. Stage 1 adds distillation to GRPO on selected rollouts. A plateau detector then removes the teacher, and Stage 2 continues with GRPO alone. Rollout GRPO reward gradient Distilled Ai>ϵA_i> _A towards the rollout yes Ai<−ϵA_i<- _A away from it no |Ai|≤ϵA|A_i|≤ _A, ri<κtr_i< _t approximately zero yes |Ai|≤ϵA|A_i|≤ _A, ri≥κtr_i≥ _t approximately zero no Table 1: Rollout classes and their treatment. 3.3 Which rollouts to distill Principle. In the gradient diagnostic, the aggregated distillation gradients for positive- and negative-advantage rollouts are closely aligned (§5.4). We therefore examine which prompts selection reaches and how it weights distillation. If f is the fraction of valid tokens selected, Eq. (4) equals f times the mean conditional KL on those tokens because its denominator is N. The average coefficient is therefore αfα f. In the reported runs, selected rollouts cover about half of the response tokens. Table 1 states the resulting partition. Rule. A quality mask qiq_i screens out rollouts that are empty, past the length limit, or without finite rewards and advantages. The cutoff κt _t is an exponential moving average of the 30th percentile of the rewards in the current batch, κt=(1−λ)κt−1+λQ0.3(ri:qi=1t) _t=(1-λ) _t-1+λ Q_0.3(\r_i:q_i=1\_t) with λ=0.1λ=0.1, initialized at the first batch’s 30th percentile. We then select wi=qi[Ai>ϵA∨(|Ai|≤ϵA∧ri<κt)],w_i=q_i\,I [A_i> _A\ \ (|A_i|≤ _A\ \ r_i< _t) ], (5) with ϵA=10−6 _A=10^-6 as the numerical tolerance for zero advantage. Zero advantage occurs both in tied groups and when a rollout’s reward equals the mean of a group with unequal rewards. The percentile cutoff follows shifts in the reward distribution. In Eq. (5), it affects rollouts whose advantage is near zero; rollouts with Ai>ϵA_i> _A are selected separately. Across the reported runs, the selected-token fraction f stays between 0.370.37 and 0.600.60, with a mean near 0.490.49 in every setting and no per-setting tuning (Appendix F). 3.4 When to retire the teacher Rationale. We use a small relative decrease in the distillation loss as a signal to stop teacher supervision. Sections 5.1 and 5.2 test the benefit of removing the teacher and the choice of stopping time, respectively. In our diagnostic, the student and teacher still choose different tokens at most positions by the end of Stage 1 (§5.3). Statistic. Let Dt,uD_t,u be the distilled-token positions in update u of rollout step t, with steps numbered from zero, each mean taken under the policy that enters that update. The recorded statistic is the per-token loss on the tokens actually distilled, dt=1U∑u=1U1|Dt,u|∑(i,h)∈Dt,uℓi,h,d_t= 1U _u=1^U 1|D_t,u| _(i,h)∈ D_t,u _i,h, (6) which takes the mean loss on selected tokens within each update, then averages these means equally across updates. This removes the direct scaling by the selected-token fraction. An update with no selected tokens contributes zero. Detector. Write g0,…,gjg_0,…,g_j for the recorded values of dtd_t, omitting steps without distilled tokens. Once 2W2W values are available, compare consecutive windows of W: mj=1W∑n=j−W+1jgn,δj=mj−W−mjmj−W.m_j= 1W _n=j-W+1^jg_n, _j= m_j-W-m_jm_j-W. (7) The rule fires at the first rollout step t⋆≥tmin−1t ≥ t_ -1 at which the last P available values of δj _j all fall below τ; if that never happens, t⋆=+∞t =+∞. A deadline TmaxT_ bounds the Stage-1 budget, giving ts=min(t⋆,Tmax)t_s= (t ,T_ ). We use αt=α _t=α through step tst_s and set it to zero from step ts+1t_s+1. Stage 2 then continues with GRPO and the original KL reference, without the teacher forward pass. Algorithm 1 Relay training loop 1: Initial policy π0 _0, total rollout steps T 2: α,W,τ,P,tmin,Tmaxα,W,τ,P,t_ ,T_ 3: πθ←π0 _θ← _0; ts←Tmaxt_s← T_ ; history H←[]H←[\,] 4: for t=0,1,…,T−1t=0,1,…,T-1 do 5: Sample groups; judge responses; compute ri,Air_i,A_i 6: if t≤tst≤ t_s then 7: Teacher-force πT _T on the sampled responses 8: Form quality masks qiq_i 9: Initialize/update κt _t if any qi=1q_i=1 10: Select wiw_i using Eq. (5) 11: Perform two updates with Eq. (3) 12: if this step contains distilled tokens then 13: Append dtd_t to H (Eq. (6)) 14: if t+1≥tmint+1≥ t_ and plateau(H,W,τ,P)(H;W,τ,P) then 15: ts←min(t,Tmax)t_s← (t,T_ ) 16: else 17: Perform two GRPO updates; skip teacher and selection Calibration. We use W=20W=20, τ=0.05τ=0.05, P=5P=5, tmin=60t_ =60 and Tmax=199T_ =199, fixed once by replaying the Qwen3-1.7B/Medicine Stage-1 trace over a grid of windows and tolerances and then reused unchanged in the other three settings (Appendix F). 4 Experiments Medicine Science Method RaR-Med. HealthBench LLMEval-Med RaR-Sci. ResearchQA Aggregate Qwen3-1.7B Base model 34.87 21.91 54.11 47.79 43.54 40.44 + OPSD 40.20 ↑ 5.33 18.79 ↓ 3.12 55.71 ↑ 1.60 49.25 ↑ 1.46 45.31 ↑ 1.77 41.85 ↑ 1.41 + GRPO 42.15 ↑ 7.28 23.87 ↑ 1.96 56.28 ↑ 2.17 52.79 ↑ 5.00 51.11 ↑ 7.57 45.24 ↑ 4.80 + CriPO 42.37 ↑ 7.50 27.97 ↑ 6.06 54.99 ↑ 0.88 55.79 ↑ 8.00 60.58 ↑ 17.04 48.34 ↑ 7.90 + Relay 46.77 ↑ 11.90 29.31 ↑ 7.40 57.51 ↑ 3.40 57.00 ↑ 9.21 59.73 ↑ 16.19 50.06 ↑ 9.62 Qwen3-4B Base model 48.56 33.82 63.18 62.60 51.85 52.00 + OPSD 56.13 ↑ 7.57 29.17 ↓ 4.65 66.21 ↑ 3.03 63.29 ↑ 0.69 52.89 ↑ 1.04 53.54 ↑ 1.54 + GRPO 58.41 ↑ 9.85 39.53 ↑ 5.71 68.01 ↑ 4.83 68.36 ↑ 5.76 63.19 ↑ 11.34 59.50 ↑ 7.50 + CriPO 59.67 ↑ 11.11 41.93 ↑ 8.11 67.77 ↑ 4.59 66.17 ↑ 3.57 63.95 ↑ 12.10 59.90 ↑ 7.90 + Relay 64.43 ↑ 15.87 42.63 ↑ 8.81 69.14 ↑ 5.96 69.80 ↑ 7.20 65.79 ↑ 13.94 62.36 ↑ 10.36 Table 2: Main results under GPT-5.1, averaged over three evaluations; standard deviations appear in Table 7. Medicine and science columns use separately trained models; Aggregate is the unweighted mean of the five. Best per model size in bold, second underlined. Arrows give point changes from the same-size base model (↑ : increase; ↓ : decrease). 4.1 Setup Data and models. We train Qwen3-1.7B and Qwen3-4B (Yang et al., 2025) in non-thinking mode on RaR-Medicine and RaR-Science (Gunjal et al., 2025), with a separate model for each domain and size. We evaluate on the corresponding RaR test split and on same-domain benchmarks: HealthBench (Arora et al., 2025) and LLMEval-Med (Zhang et al., 2025) for medicine, ResearchQA (Yifei et al., 2026) for science (Appendix B). Training. All RL methods use 64 prompts with 8 responses per rollout step and a nominal budget of 400 steps. Qwen3.8-27B (Qwen Team, 2026) provides training rewards and validation scores; Appendix A gives the remaining training settings. Checkpoint selection and validation curves use the official RaR validation partitions; final GPT-5.1 evaluation uses the separate test partitions. Evaluation. GPT-5.1 (OpenAI, 2025), with reasoning disabled, evaluates responses from the checkpoints in Table 2 independently of the training judge. Responses are sampled at temperature 0.7 with the same 5,120-token limit as training. RaR uses the full signed rubric and clips each question’s normalized score to [0,1][0,1]; the other three need adapted protocols (Appendix C). We evaluate checkpoints near the peaks of in-training validation. Baselines. We compare with the base model, standalone OPSD (Zhao et al., 2026) from the same frozen teacher, GRPO (Shao et al., 2024), and a reproduction of CriPO (Xia et al., 2026), which combines best-rollout distillation towards a current-policy teacher with token-level advantage correction and a late coefficient decay. Our reproduction keeps both mechanisms but uses a smaller distillation coefficient, the published value having been unstable in our setup (Appendix E). 4.2 Main results Figure 3: Per-token divergence during Stage 1 of Relay across the four settings. Vertical lines mark the plateau-rule trigger. The divergence scale differs across settings, but the rule tests relative change and transfers without re-tuning. In-domain improvements across all four settings. Relay has the highest in-domain score among the evaluated methods (Table 2). The gains over GRPO are 4.62 and 6.02 points on RaR-Medicine, and 4.21 and 1.44 points on RaR-Science, for the 1.7B and 4B models, respectively. Relay also outperforms the CriPO reproduction in all four in-domain settings and has the highest five-benchmark aggregate at both sizes. In the first evaluation, the 95% confidence intervals from bootstrapping questions exclude zero for all eight paired in-domain differences (Figure 4 in Appendix C). Transfer to additional benchmarks. Relay has the highest score on four of the five benchmarks at 1.7B and on all five at 4B. It improves HealthBench over GRPO by 5.4 and 3.1 points. On LLMEval-Med it scores highest at both sizes, 57.51 and 69.14. Appendix D follows one science question through all three methods. Standalone distillation. OPSD improves on the base model in all four in-domain settings but trails GRPO, and its validation scores peak at steps 20–185 before declining (Appendix E). 5 Analysis We ask four questions on Qwen3-1.7B/Medicine: whether retiring the teacher helps, whether the rule retires it at a good time, what the plateau measures, and which rollouts to distill. Scores are on the 0–100 scale, from the training judge unless GPT-5.1 is named. Branch at step 200 210–260 270–300 310–350 360–390 Teacher removed 2.31 4.14 4.71 5.06 Kept at α=0.003α=0.003 2.49 3.45 3.73 4.11 Teacher refreshed 1.77 3.22 3.55 3.76 Kept at α=0.01α=0.01 1.70 2.49 2.67 2.88 Table 3: Training-judge validation gain from the shared step-200 branch point, averaged over the indicated steps. Bold marks the column maximum. 5.1 Does retiring the teacher help? Branches from one checkpoint. Four runs continue from one Stage-1 checkpoint at step 200, differing only in the treatment of the teacher: removed, kept at α=0.01α=0.01, kept at α=0.003α=0.003, or replaced at the same α by a copy of the current policy (Table 3). Gains are measured against the common validation baseline of 40.62, averaged over steps 160–200. Stopping does not pay off at once: over steps 210–260 the four lie within 0.8 points of each other, and the reduced coefficient is briefly ahead. In each reported interval from steps 270–300 onward, removing the teacher gives the largest average gain among the four branches. Over steps 360–390, its average gain from the shared checkpoint is 5.06 points, compared with 2.88 for keeping the teacher at full strength (Table 3). The three hand-off schedules also reach higher peak scores than continued distillation (Table 12 in Appendix H). Teacher leaves at 141–160 181–200 221–240 281–300 Step 140 0.74 0.58 0.46 0.32 Step 171 (the rule) 0.73 0.72 0.57 0.42 Step 200 0.73 0.72 0.60 0.38 Never 0.73 0.72 0.70 0.60 GRPO (no teacher) 0.25 0.22 0.21 0.18 Table 4: Training-rollout token entropy by hand-off time (Qwen3-1.7B/Medicine), averaged over the steps shown. The four distillation runs share one Stage-1 trajectory and peak at 0.76 near step 130; each turns down after its own hand-off. What changes at the hand-off. Entropy declines after teacher removal in all three hand-off schedules. The four distillation schedules share a rise in entropy until about step 130, then diverge as the teacher is removed at steps 140, 171 and 200 (Table 4). By steps 281–300, entropy is 0.32–0.42 after a hand-off, compared with 0.60 under continued distillation. GRPO has lower entropy throughout these intervals. These observations are consistent with distillation maintaining higher entropy in this setting; the conditional KL alone does not imply an increase in full-vocabulary entropy (Appendix G). The CriPO reproduction also maintains higher entropy later in training (Figure 6 in Appendix E). 5.2 Is the trigger well placed, and does it transfer? Against fixed alternatives. Among the three tested switch times, the rule’s switch at step 171 gives the highest observed GPT-5.1 score on RaR-Medicine (46.77), compared with 45.81 for step 140 and 46.38 for step 200. Transfer. Calibrated once here, the same detector settings fire at steps 171, 122, 159 and 136 across the four model–domain settings, before the deadline in every case and with no re-tuning (Figure 3). The divergence differs by more than a factor of two in scale across them, which is why the rule tests relative change rather than a level. An offline replay over windows and tolerances places the default configuration in a region that fires in all four settings (Appendix F). 5.3 What the plateau measures The signal and the gain flatten together. The monitored divergence and the lead over GRPO flatten over a similar period (Figure 1). The conditional teacher–student KL falls from 0.55 at the start of Stage 1 to 0.18 by step 150 and holds there to step 200. The lead over GRPO stops growing in the same stretch: it reaches 2.9 points by step 110 and goes no higher through the end of Stage 1. A remaining gap at the plateau. When generating with the rubric, the frozen teacher scores 90.5, compared with 40.4 for the student at step 140. Meanwhile, top-64 overlap changes little, from 0.40 to 0.44 over Stage 1, despite the reduction in conditional KL. By the end of Stage 1, their top-1 predictions still differ at 56% of positions (Appendix F). These observations motivate testing when to stop this conditional objective; whether another objective could make further use of the teacher remains open (Appendix G). 5.4 Which rollouts to distill Distilled Tok. Best Δ Len. Tr. Pos. + low zero .49 7.9 1,482 0.04 Pos. + all zero .54 7.6 1,643 0.13 Positive only .42 7.5 1,309 0.02 Pos. + negative .91 7.3 1,752 0.35 All rollouts .92 6.8 2,262 2.39 All, α=.005α=.005 .98 7.1 1,978 1.17 GRPO — 5.6 1,023 0.02 Table 5: Rollout selection in Stage 1 (Qwen3-1.7B/Medicine). Tok.: distilled share of response tokens; low zero: rollouts with |Ai|≤ϵA|A_i|≤ _A and ri<κtr_i< _t. Best Δ : peak training-judge gain over Stage 1. Len., Tr.: rollout length and truncation (%) at steps 151–200. The all-rollout variant with α=0.005α=0.005 matches Relay’s average coefficient. Selection and gradient direction. Across five checkpoints, the aggregate distillation gradients from positive- and negative-advantage rollouts have cosines of 0.97–0.99 (Appendix I). The cosine between the distillation and GRPO gradients is 0.16 at initialization and ranges from −0.11-0.11 to +0.05+0.05 at the later checkpoints. These measurements motivate examining how selection changes the tokens covered and the average coefficient. Rollout coverage and peak gain. The selection-rule comparisons use α=0.01α=0.01, chosen once by an ablation over 0.1,0.01,0.001\0.1,0.01,0.001\. At this fixed coefficient, the selected-token fraction f also determines the average coefficient. Length and truncation rise monotonically with f at fixed α, while gain does not: the three rules with f≤0.54f≤ 0.54 all land between 7.5 and 7.9 points, while the two above f=0.9f=0.9 fall to 7.3 and 6.8, at up to 2,262 tokens and 2.39% truncation. In these runs, selecting rollouts that cover roughly half the response tokens gives a higher peak gain than distilling almost all rollouts. Adding negative-advantage rollouts to positive-only selection increases length and truncation while lowering peak gain. Adding high-scoring zero-advantage rollouts to our rule shows the same trend (Table 5). Selection at a matched average coefficient. To separate the rule from the dose it induces, we reduce the all-rollout coefficient to α=0.005α=0.005, which brings its average coefficient to the same 0.00490.0049 as our rule’s. It still trails on every measure in Table 5, by 0.8 points of peak gain and with longer, more often truncated rollouts. Their step-200 checkpoints also differ under GPT-5.1, by a paired 1.49 points on RaR-Medicine. The benefit of rollout selection persists at a matched average coefficient. Adding the low-score branch. This branch selects low-scoring rollouts with near-zero advantage, including rollouts from tied groups and those at the mean of groups with unequal rewards. These receive little or no GRPO reward gradient, while the KL regularizer remains active. Relative to positive-only selection, adding the low-score branch raises the selected-token fraction from 0.42 to 0.49 and the peak gain from 7.5 to 7.9, while mean response length increases by 173 tokens (Table 5). 6 Conclusion Relay uses advantages and rewards to select rollouts for privileged self-distillation, then stops distillation when progress in its loss slows. It has the highest in-domain score among the compared methods in all four model–domain settings, improving on GRPO by 4.62 and 6.02 points on RaR-Medicine, with a stopping rule calibrated once and reused without re-tuning. On Qwen3-1.7B/Medicine, stopping distillation from a shared checkpoint yields higher scores later in training than keeping or weakening the teacher and removes its forward pass. In this setting, selecting rollouts covering about half the response tokens also outperforms distilling every rollout at a matched average coefficient. Standalone distillation remains below GRPO in all four in-domain settings, supporting the use of both rubric rewards and teacher supervision. The plateau does not imply full rubric transfer: the rubric-conditioned teacher still scores substantially higher (§5.3). Whether another distillation objective could transfer more remains open. Limitations Single calibration setting. The plateau rule is calibrated on one model–domain pair and reused in three others. The sensitivity analysis shows that conservative settings miss the trigger entirely, so the hyperparameters may need re-tuning for substantially different training dynamics or model scales. Frozen teacher. The teacher is a frozen copy of the initial model. Adapting the teacher during training could extend the useful distillation window, but would complicate the stopping criterion: the plateau rule assumes a stationary reference. Binary selection. The selection rule gives each rollout a binary weight. Continuous weights could allocate supervision more finely; their interaction with the stopping rule remains to be studied. Seeds. We train one seed per configuration, so variation across training seeds is unmeasured. The standard deviations reflect repeated evaluations at fixed checkpoints; the confidence intervals resample questions from the first evaluation. Ethical Considerations The models in this paper answer medical questions. Their outputs have not been validated by clinicians and are unsuitable for clinical decision-making. All datasets are public research benchmarks; we did not collect new data from people. References Agarwal et al. (2024) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations (ICLR), External Links: 2306.13649, Link Cited by: §2. Arora et al. (2025) R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal HealthBench: evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775. External Links: 2505.08775, Link, Document Cited by: §1, §4.1. Bablani et al. (2026) D. Bablani, A. Gupta, and W. Chen Rubrics as privileged information for open-ended generation. arXiv preprint arXiv:2608.02948. External Links: 2608.02948, Link, Document Cited by: §1, §2. Gu et al. (2026) S. Gu, J. Chen, S. Zhou, A. Cohan, and R. Ying Rethinking reward supervision: rubric-conditioned self-distillation. arXiv preprint arXiv:2606.19327. External Links: 2606.19327, Link, Document Cited by: §1, §2. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, External Links: 2306.08543, Link Cited by: §2. Gunjal et al. (2025) A. Gunjal, A. Wang, E. Lau, V. Nath, Y. He, B. Liu, and S. Hendryx Rubrics as rewards: reinforcement learning beyond verifiable domains. arXiv preprint arXiv:2507.17746. External Links: 2507.17746, Link, Document Cited by: §1, §2, §4.1. Han et al. (2026) Z. Han, J. Xiao, Z. Lu, R. Jin, Z. Yao, Y. Liu, H. Hao, Y. Sun, Y. Yang, Q. Gu, X. Cai, and D. Xiong Distill where you fail: recovering learning signals of negative RL-groups from adaptive teacher guidance. arXiv preprint arXiv:2608.00782. External Links: 2608.00782, Link, Document Cited by: §1, §2, §2. Hübotter et al. (2026) J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. External Links: 2601.20802, Link, Document Cited by: §1, §2. Kim et al. (2026a) J. Kim, J. Jeon, D. Li, and Y. Yang Rebellious student: reversing teacher signals for reasoning exploration with self-distilled RLVR. arXiv preprint arXiv:2605.10781. External Links: 2605.10781, Link, Document Cited by: §2. Kim et al. (2026b) J. Kim, X. Luo, M. Kim, S. Lee, D. Kim, J. Jeon, D. Li, and Y. Yang Why does self-distillation (sometimes) degrade the reasoning capability of LLMs?. In Conference on Language Modeling (COLM), External Links: 2603.24472, Link Cited by: §2. Lan et al. (2026) G. Lan, L. Xiong, X. Zhou, H. Cui, Y. Zhang, M. Li, Z. Shi, B. Fetahu, L. Li, and X. Li Alternating reinforcement learning with contextual rubric rewards: beyond the scalarization strategy. arXiv preprint arXiv:2603.15646. External Links: 2603.15646, Link, Document Cited by: §2. Li et al. (2026a) B. Li, B. Chen, C. Yang, P. Nie, C. Zhao, and X. Ye Sequential beats joint: on the interplay between on-policy distillation and RLVR. arXiv preprint arXiv:2609.04108. External Links: 2609.04108, Link, Document Cited by: §2, §2. Li et al. (2026b) G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua Unifying group-relative and self-distillation policy optimization via sample routing. arXiv preprint arXiv:2604.02288. External Links: 2604.02288, Link, Document Cited by: §2. Li et al. (2026c) S. Li, J. Zhao, M. Wei, H. Ren, Y. Zhou, J. Yang, S. Liu, K. Zhang, and W. Chen RubricHub: a comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. arXiv preprint arXiv:2601.08430. External Links: 2601.08430, Link, Document Cited by: §2. Li et al. (2026d) Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, and N. Ding Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. External Links: 2604.13016, Link, Document Cited by: §2. Liu et al. (2026) Y. Liu, S. Zhang, Y. Zhang, and Q. Gu Self-distilled policy gradient. arXiv preprint arXiv:2606.04036. External Links: 2606.04036, Link, Document Cited by: §2, §2. Mahmoud et al. (2026) A. Mahmoud, M. Rezaei, Z. Wang, A. Gunjal, B. Liu, and Y. He Reward hacking in rubric-based reinforcement learning. arXiv preprint arXiv:2605.12474. External Links: 2605.12474, Link, Document Cited by: §2. OpenAI (2025) OpenAI GPT-5.1. Note: Model snapshot gpt-5.1-2025-11-13, accessed via API External Links: Link Cited by: §4.1. Pan et al. (2026) L. Pan, S. Tao, Y. Zhai, L. Zhang, Z. Liu, B. Ding, A. Liu, and L. Wen RLCSD: reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709. External Links: 2606.11709, Link, Document Cited by: §2, §2. Qwen Team (2026) Qwen Team Qwen3.8-27B model card. Note: Hugging Face model cardAccessed 2026-10-08 External Links: Link Cited by: Appendix C, §4.1. Rezaei et al. (2026) M. Rezaei, A. Mahmoud, Z. Wang, U. Tyagi, A. Gosai, R. Dumitru, A. Sabharwal, B. Liu, and Y. He Rubric-guided self-distillation: post-training without rubric verifiers. arXiv preprint arXiv:2606.12507. External Links: 2606.12507, Link, Document Cited by: §1, §2. Rezaei et al. (2025) M. Rezaei, R. Vacareanu, Z. Wang, C. Wang, B. Liu, Y. He, and A. F. Akyürek Online rubrics elicitation from pairwise comparisons. arXiv preprint arXiv:2510.07284. External Links: 2510.07284, Link, Document Cited by: §2. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: 1707.06347, Link, Document Cited by: §3.1. Shao et al. (2026) R. Shao, A. Asai, S. Z. Shen, H. Ivison, V. Kishore, J. Zhuo, X. Zhao, M. Park, S. G. Finlayson, D. Sontag, T. Murray, S. Min, P. Dasigi, L. Soldaini, F. Brahman, W. Yih, T. Wu, L. Zettlemoyer, Y. Kim, H. Hajishirzi, and P. W. Koh DR Tulu: reinforcement learning with evolving rubrics for deep research. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 110046–110101. External Links: 2511.19399, Link Cited by: §2. Shao et al. (2024) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y.K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. External Links: 2402.03300, Link, Document Cited by: §1, §4.1. Song and Zheng (2026) M. Song and M. Zheng A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626. External Links: 2604.00626, Link, Document Cited by: §2. Tan et al. (2026) Z. Tan, Z. Yu, B. Lin, Z. Geng, H. Geng, Y. Zhang, M. Zhang, Y. Chen, S. Hu, Z. Yin, C. Zhang, and L. Bai PAPO: stabilizing rubric integration training via decoupled advantage normalization. arXiv preprint arXiv:2603.26535. External Links: 2603.26535, Link, Document Cited by: §2, §2. Viswanathan et al. (2025) V. Viswanathan, Y. Sun, S. Ma, X. Kong, M. Cao, G. Neubig, and T. Wu Checklists are better than reward models for aligning language models. In 39th Conference on Neural Information Processing Systems (NeurIPS 2025), External Links: 2507.18624, Link Cited by: §1, §2. Wang et al. (2026a) J. Wang, X. Ouyang, Z. Chen, Y. Hu, and L. Guo STEPS: selective on-policy self-distillation for reasoning. arXiv preprint arXiv:2605.10194. External Links: 2605.10194, Link, Document Cited by: §2, §2. Wang et al. (2025) W. Wang, S. Xiong, G. Chen, W. Gao, S. Guo, Y. He, J. Huang, J. Liu, Z. Li, X. Li, Z. Liu, H. Zhao, D. An, L. Cao, Q. Cao, W. Deng, F. Du, Y. Gu, J. Li, X. Li, M. Liu, Y. Luo, Z. Liu, Y. Wang, P. Wang, T. Wu, Y. Wu, Y. Zhao, S. Zhao, J. Yang, S. Yang, Y. Tan, H. Yi, Y. Xu, Y. Yuan, X. Zhang, L. Qu, W. Su, W. Wang, J. Wang, and B. Zheng Reinforcement learning optimization for large-scale learning: an efficient and user-friendly scaling library. arXiv preprint arXiv:2506.06122. External Links: 2506.06122, Link, Document Cited by: Appendix A. Wang et al. (2026b) X. Wang, W. Shi, Y. Chen, M. Zontak, Y. He, and R. Y. Pang OPD before RL: warm-starting rubric-based RL with on-policy distillation. arXiv preprint arXiv:2610.02781. External Links: 2610.02781, Link, Document Cited by: §1, §2, §2. Wang et al. (2026c) Y. Wang, Z. Lu, Y. Yan, K. Song, W. Zhang, W. Lu, J. Xiao, Y. Zhuang, and Y. Shen PaperGym: rubric-centered evolution for research-plan generation. arXiv preprint arXiv:2608.31119. External Links: 2608.31119, Link Cited by: §1, §2. Xia et al. (2026) M. Xia, Y. Yang, C. Ye, S. Zhu, S. Yang, G. Zhu, Y. Zhang, C. Peng, H. Wang, and C. Wang CriPO: enhancing rubric-based RL via self-distillation. arXiv preprint arXiv:2607.18082. External Links: 2607.18082, Link, Document Cited by: Appendix E, §1, §1, §2, §2, §4.1. Xu et al. (2026) T. Xu, Y. Zheng, P. Lu, L. Ye, Y. Wu, Z. Zhang, Y. Yu, C. Ma, J. Zhu, P. Liu, B. Dong, H. Zhu, R. Huang, and G. Yu Rubrics to tokens: bridging response-level rubrics and token-level rewards in instruction following tasks. arXiv preprint arXiv:2604.02795. External Links: 2604.02795, Link, Document Cited by: §2. Yang et al. (2025) A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Zhou, J. Lin, K. Dang, K. Bao, K. Yang, L. Yu, L. Deng, M. Li, M. Xue, M. Li, P. Zhang, P. Wang, Q. Zhu, R. Men, R. Gao, S. Liu, S. Luo, T. Li, T. Tang, W. Yin, X. Ren, X. Wang, X. Zhang, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Zhang, Y. Wan, Y. Liu, Z. Wang, Z. Cui, Z. Zhang, Z. Zhou, and Z. Qiu Qwen3 technical report. arXiv preprint arXiv:2505.09388. External Links: 2505.09388, Link, Document Cited by: §4.1. Yang et al. (2026) C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan Self-distilled RLVR. arXiv preprint arXiv:2604.03128. External Links: 2604.03128, Link, Document Cited by: §1, §2, §2. Ye et al. (2026) T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei On-policy context distillation for language models. arXiv preprint arXiv:2602.12275. External Links: 2602.12275, Link, Document Cited by: §1, §2. Yifei et al. (2026) L. S. Yifei, A. Chang, C. Malaviya, and M. Yatskar ResearchQA: evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics. Transactions of the Association for Computational Linguistics 14, p. 1365–1389. External Links: Document, 2509.00496, Link Cited by: §4.1. Yu et al. (2025) Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang DAPO: an open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. External Links: 2503.14476, Link, Document Cited by: §2. Yu et al. (2026) Y. Yu, Z. Lu, Y. Liu, Y. Pan, A. Wang, Q. Chen, H. Yang, W. Zhang, Q. Chen, and Y. Shen RetireOPD: self-retiring on-policy distillation for agentic reinforcement learning. arXiv preprint arXiv:2609.20784. External Links: 2609.20784, Link, Document Cited by: §2. Zhang et al. (2026a) B. Zhang, J. He, W. Wang, S. Lv, W. Ma, R. Lin, S. Zhong, and L. Guo CoRT: counterfactual replay for token-level rubric-guided policy optimization. arXiv preprint arXiv:2607.25659. External Links: 2607.25659, Link Cited by: §2. Zhang et al. (2025) M. Zhang, Y. Shen, Z. Li, H. Sha, B. Hu, Y. Wang, C. Huang, S. Liu, J. Tong, C. Jiang, M. Chai, Z. Xi, S. Dou, T. Gui, Q. Zhang, and X. Huang LLMEval-Med: a real-world clinical benchmark for medical LLMs with physician validation. In Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, p. 4888–4914. External Links: Document, Link Cited by: §4.1. Zhang et al. (2026b) W. Zhang, K. Zhang, J. Qi, B. Lai, and J. Huang Experience is the best teacher: motivating effective exploration in reinforcement learning for LLMs. arXiv preprint arXiv:2603.20046. External Links: 2603.20046, Link, Document Cited by: §2. Zhao et al. (2026) S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover Self-distilled reasoner: on-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734. External Links: 2601.18734, Link, Document Cited by: §1, §2, §4.1. Zhou et al. (2026) Y. Zhou, S. Li, S. Liu, W. Fang, K. Zhang, J. Zhao, J. Yang, Y. Zhou, J. Lv, T. Zheng, H. Lu, C. Wei, X. Yan, and M. Song Breaking the exploration bottleneck: rubric-scaffolded reinforcement learning for open-ended LLM reasoning. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, p. 165222–165251. Note: arXiv version titled “Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning” External Links: 2508.16949, Link Cited by: §2. Zhu et al. (2026) S. Zhu, X. Ye, H. Lu, W. Shi, and G. Liu The many faces of on-policy distillation: pitfalls, mechanisms, and fixes. arXiv preprint arXiv:2605.11182. External Links: 2605.11182, Link, Document Cited by: §2. Appendix A Implementation Details Writing ηi,h=πθ(yi,h∣x,yi,<h)/πθold(yi,h∣x,yi,<h) _i,h= _θ(y_i,h x,y_i,<h)/ _ _old(y_i,h x,y_i,<h) and η~i,h=clip(ηi,h,1−ϵc,1+ϵc) η_i,h=clip( _i,h,1- _c,1+ _c), the policy term of §3.1 is ℒGRPO _GRPO =−1N∑(i,h)∈min(ηi,hAi,η~i,hAi) =- 1N _(i,h) ( _i,hA_i, η_i,hA_i) (8) +βℒKL. + _KL. Table 6 lists the shared training configuration. All runs use ROLL (Wang et al., 2025) with Megatron-Core training and vLLM generation on 8 H200 GPUs (tensor parallel 4, data parallel 2). In-training validation uses the official RaR validation partitions with the training judge; GPT-5.1 results use the separate test partitions. Setting Value Rollout budget 400 steps Prompts × responses per step 64 × 8 Policy updates per step 2 (32 prompts each) Sampling (train) T=1T=1, top-p=1p=1, top-k=−1k=-1 Prompt / response limit 1,024 / 5,120 tokens Optimizer Adam, LR 10−610^-6 constant, no warmup PPO clip 0.2 KL to initial model β=10−3β=10^-3, k3k_3 estimator Training reward Eq. (1), judge temperature 0.1 Advantage (r−μ)/(σ+ϵσ)(r-μ)/(σ+ _σ), ϵσ=10−6 _σ=10^-6; 0 for tied groups Distillation coefficient α 0.01 (Stage 1), 0 (Stage 2) Divergence conditional forward KL on teacher top-64, temperature 1 Teacher frozen initial model with signed rubric Selection Eq. (5), ϵA=10−6 _A=10^-6 Cutoff κt _t EMA (rate 0.1) of the 30th reward percentile Plateau rule W=20W=20, τ=0.05τ=0.05, P=5P=5, tmin=60t_ =60, Tmax=199T_ =199 Table 6: Training configuration (Relay-specific rows at the bottom). Medicine Science Method RaR-Med. HealthBench LLMEval-Med RaR-Sci. ResearchQA Aggregate Qwen3-1.7B Base model 34.87±0.1434.87± 0.14 21.91±1.0021.91± 1.00 54.11±0.7854.11± 0.78 47.79±0.1047.79± 0.10 43.54±0.1943.54± 0.19 40.44 + OPSD 40.20±0.2340.20± 0.23 18.79±0.1918.79± 0.19 55.71±0.2755.71± 0.27 49.25±0.5449.25± 0.54 45.31±0.0645.31± 0.06 41.85 + GRPO 42.15±0.3342.15± 0.33 23.87±0.1123.87± 0.11 56.28±0.2256.28± 0.22 52.79±0.0652.79± 0.06 51.11±0.2851.11± 0.28 45.24 + CriPO 42.37±0.3142.37± 0.31 27.97±0.2727.97± 0.27 54.99±0.0554.99± 0.05 55.79±0.2255.79± 0.22 60.58±0.1960.58± 0.19 48.34 + Relay 46.77±0.2646.77± 0.26 29.31±0.0829.31± 0.08 57.51±0.1357.51± 0.13 57.00±0.2757.00± 0.27 59.73±0.1859.73± 0.18 50.06 Qwen3-4B Base model 48.56±0.0748.56± 0.07 33.82±0.1733.82± 0.17 63.18±0.3063.18± 0.30 62.60±0.2362.60± 0.23 51.85±0.2751.85± 0.27 52.00 + OPSD 56.13±0.2556.13± 0.25 29.17±0.1429.17± 0.14 66.21±0.2966.21± 0.29 63.29±0.5163.29± 0.51 52.89±0.1652.89± 0.16 53.54 + GRPO 58.41±0.1258.41± 0.12 39.53±0.3339.53± 0.33 68.01±0.2168.01± 0.21 68.36±0.3768.36± 0.37 63.19±0.4863.19± 0.48 59.50 + CriPO 59.67±0.1959.67± 0.19 41.93±0.2541.93± 0.25 67.77±0.1067.77± 0.10 66.17±0.2766.17± 0.27 63.95±0.3463.95± 0.34 59.90 + Relay 64.43±0.1864.43± 0.18 42.63±0.2342.63± 0.23 69.14±0.2569.14± 0.25 69.80±0.1169.80± 0.11 65.79±0.0865.79± 0.08 62.36 Table 7: Repeated GPT-5.1 evaluation of the checkpoints in Table 2. Benchmark scores are means ± standard deviations over three evaluations with fresh rollouts; Aggregate reports only the mean. Medicine and science columns use separately trained models. Teacher prompt. The teacher sees the student’s prompt followed by the signed rubric: prompt Hidden evaluation rubrics: rubric Use these rubrics as private guidance. Satisfy every POSITIVE REQUIREMENT and avoid triggering every NEGATIVE PENALTY. For a negative rubric written as ‘must not’, ‘should not’, ‘avoid’, or ‘does not’, avoid the underlying prohibited behavior; do not try to satisfy the prohibition text literally as a positive requirement. After understanding the rubrics, provide your own thorough response to the problem. Address them naturally without explicitly referencing them. The teacher scores student-generated tokens by teacher forcing. All methods retain student prompts with more than 5 and at most 1,024 tokens. Appendix B Data Table 8 lists the datasets. HealthBench is subsampled to 500 questions; the other benchmarks are used in full. Dataset Use Size RaR-Medicine train training 17,711 RaR-Medicine val. validation 2,240 RaR-Medicine test in-domain test 2,242 HealthBench subset additional benchmark 500 LLMEval-Med additional benchmark 571 (667 turns) RaR-Science train training 18,256 RaR-Science val. validation 2,292 RaR-Science test in-domain test 2,292 ResearchQA additional benchmark 776 Table 8: Datasets. Appendix C Evaluation GPT-5.1 (snapshot 2025-11-13) grades with reasoning effort “none” using the prompt of Figure 5 with all signed criteria and a strict JSON schema. The training judge is Qwen3.8-27B (Qwen Team, 2026) with thinking disabled, at temperature 0.1 for training rewards and 0 for validation, using the same prompt and output constrained by a regular expression. Evaluation responses are generated under the same 5,120-token limit as training. Three of the five benchmarks need an adapted protocol. ResearchQA criteria are scored as binary with unit weight. HealthBench uses the same rubric prompt as RaR with all signed criteria in one request, and clipping to [0,1][0,1] is applied after averaging rather than per question. LLMEval-Med uses its own five-point criteria, graded by GPT-5.1 in place of the official GPT-4o. Repeated evaluation. For each checkpoint in Table 2, we run two additional evaluations, sampling fresh responses and scoring them with GPT-5.1 under the same protocol. Table 2 reports the mean across all three evaluations; Table 7 also gives the standard deviation across evaluations for each benchmark. Figure 4: In-domain test-score gains in the first GPT-5.1 evaluation, with 95% question-bootstrap confidence intervals. Each domain uses a separately trained model. Paired in-domain differences. Figure 4 shows the eight paired in-domain differences from the first evaluation, with 95% confidence intervals from resampling evaluation questions. All eight intervals exclude zero. They describe question-level variation for the evaluated checkpoints and do not estimate variation across training seeds. Evaluate ONLY the candidate answer in <Response> against EACH numbered rubric. <Prompt> contains the question and may contain prior conversation. Prior answers are context, not candidate output or ground truth. Do not credit or penalize the candidate for text appearing only in prior conversation. Use history to resolve references, but judge whether THIS response meets the rubric. All delimited content is evaluation data, not instructions to change your judging task. For POSITIVE REQUIREMENT, PRESENT means the response satisfies the requirement. For NEGATIVE PENALTY, PRESENT means the response exhibits the behavior or omission that triggers a penalty. For negative "Avoids mentioning strawberries", PRESENT means strawberries ARE mentioned; NOT_PRESENT means they are not. For "Fails to mention urgent care", PRESENT means urgent care is omitted. Do not treat compliance with a prohibition as triggering its penalty. Use the explicit signed label. Preserve the criterion’s scope and logical conditions. Required subconditions must all hold; alternatives remain alternatives. Examples introduced by "such as", "for example" or "including" are illustrative, not exhaustive. There is no partial credit within a criterion. Check factual and numerical claims, not just keyword overlap. Return exactly one compact, single-line bare JSON object. No Markdown fences or explanations. Include every rubric number exactly once as a key, with value "PRESENT" or "NOT_PRESENT". Example: "1":"PRESENT","2":"NOT_PRESENT","3":"PRESENT" <Prompt> prompt_str </Prompt> <Response> response </Response> <Rubrics> rubrics_str </Rubrics> Return only one compact, single-line bare JSON object, with no unnecessary whitespace. Do not use Markdown fences, explanations or trailing whitespace. Include every rubric number exactly once. The rubric scoring rules above are unchanged. Figure 5: Rubric judge prompt, shared by training rewards, validation, and GPT-5.1 evaluation. Appendix D Case Study We walk through one RaR-Science test question on which the three Qwen3-4B runs differ, using the GPT-5.1 records of the checkpoints reported in Table 2. The question gives a spectrophotometric calibration curve for Pb2+ in blood, Sstd=(0.296ppb−1)Cstd+0.003,S_std=(0.296\,ppb^-1)\,C_std+0.003, and asks for the concentration of a sample whose signal is Ssamp=0.397S_samp=0.397. Its rubric has six positive criteria and one penalty (Table 9). All three solve the problem. Every run substitutes SsampS_samp into the calibration equation, subtracts the intercept, divides by the slope and reports 0.394/0.296≈1.330.394/0.296≈ 1.33 ppb with the right unit. Criteria 1–4 and 6, which cover the substitution, the rearrangement, the arithmetic, the units and the clarity of the derivation, are satisfied by all three. One rubric requirement separates them. Criterion 5 asks the answer to explain what the intercept term means and to link it to a blank correction, and criterion 7 penalises leaving that explanation out. Relay closes with a note attributing the 0.0030.003 offset to background signal and checks the result by substituting it back. CriPO gives the algebra alone, in a single line ending in a boxed number. GRPO states that the intercept is the signal at zero concentration but not what produces it, which the judge did not accept as the required explanation. Missing criterion 5 removes 4 reward points, and triggering criterion 7 subtracts another point. With a total positive weight of 25, Relay scores 1.001.00, while CriPO and GRPO each score 0.800.80. What the case illustrates. All three answers reach the correct number; their scores differ on an explanation required by the rubric but absent from the question. A scalar reward reports the resulting score without identifying the missing criterion, while the teacher has that criterion in context when scoring the student’s tokens. Scoring only positive criteria would omit the additional penalty with weight −1-1 for this omission. The same pattern appears elsewhere in the split. On a question about the oxidation of an aldehyde, all three identify the carboxylic acid product, but only Relay lists the full set of oxidizing agents the rubric names, and the penalty for an incomplete list fires for both baselines. ω Criterion Relay CriPO GRPO 1 +5+5 substitutes SsampS_samp ✓ ✓ ✓ 2 +5+5 rearranges the equation ✓ ✓ ✓ 3 +4+4 computes 0.394/0.2960.394/0.296 ✓ ✓ ✓ 4 +4+4 handles the units ✓ ✓ ✓ 5 +4+4 explains the intercept ✓ — — 6 +3+3 step-by-step derivation ✓ ✓ ✓ 7 −1-1 omits that explanation — ✓ ✓ Score 1.00 0.80 0.80 Table 9: Rubric verdicts for the case of Appendix D (RaR-Science, Qwen3-4B). A check mark means the judge found the criterion present, which for the penalty row means the fault was committed. Scores follow Eq. (1), using the total positive rubric weight as the denominator and clipping the result to [0,1][0,1]. Appendix E Baselines GRPO. GRPO is Relay with α=0α=0 throughout and no teacher. The Qwen3-1.7B/Medicine GRPO trajectory used throughout, in Table 2 and in §5, is a single run; the Stage-1 comparisons read its first 200 steps and the schedule comparison its full budget. OPSD. Standalone OPSD distills from the same frozen teacher with forward KL on all rollouts and coefficient 1, without reward. All four runs peak and then flatten or decline, and were stopped between steps 73 and 208. CriPO reproduction. We reproduce CriPO (Xia et al., 2026) with both behavior injection and advantage correction. For each group, behavior injection selects the highest-advantage rollout and prompts a current-policy teacher with up to three of the group’s unexplored rubric criteria. It uses forward KL on the top-64 tokens with a tail bucket and caps the per-token loss at 10. The first and last 1% of response tokens are excluded, and contribution-based filtering uses γ=0.95γ=0.95. The loss is averaged over the tokens selected within that rollout. Advantage correction applies to negative-advantage rollouts of at most 2,048 tokens. A counterfactual teacher is instructed to drop the penalised behavior; with a probability threshold of 0.1, it identifies up to 10 low-probability tokens whose advantages are masked or set to +0.1+0.1. The behavior-injection coefficient is 2sβ2sβ, where s is the fraction of rollouts in the group with non-negative advantage. It decays linearly to zero over the final quarter of training. Here β is the behavior-injection base coefficient, separate from the GRPO reference-KL coefficient in Table 6. The reproduction uses the shared rollout, optimizer, and reference-KL settings in that table, together with the same training reward and judge and no additional easy-prompt filter. We use β=0.01β=0.01 in all four settings. At the published β=0.05β=0.05, entropy exceeds 1.0 within 23–37 steps (Figure 6). Lowering β delays the entropy rise more for Qwen3-1.7B than for Qwen3-4B, particularly on Medicine, showing model-dependent sensitivity to this coefficient. With β=0.01β=0.01, training is stable through the budget. Figure 6: CriPO token entropy at β=0.05β=0.05 (published) and β=0.01β=0.01 (used). Appendix F The Plateau Rule and Distillation Diagnostic Calibration and sensitivity. Table 10 varies the window and the tolerance, replaying the four settings’ Stage-1 traces offline. Shorter windows fire earlier, while longer windows and tighter tolerances often exhaust the available trace without triggering. The default configuration (W=20W=20, τ=0.05τ=0.05), calibrated on Qwen3-1.7B/Medicine, fires in all four settings. The remaining two values are budget decisions: P=5P=5 asks the slowdown to persist for five windows rather than one, tmin=60t_ =60 keeps the rule from firing during the initial fast decrease, and Tmax=199T_ =199 reserves half of the 400-step budget for Stage 2. No per-setting tuning followed. τ=0.03τ=0.03 τ=0.05τ=0.05 τ=0.10τ=0.10 W=10W=10 137/79/125/62 111/78/97/60 79/59/78/59 W=20W=20 184/–/–/– 171/122/159/136 131/66/107/68 W=30W=30 179/–/–/– 175/–/–/– 156/–/152/– Table 10: Firing step under varied rule settings (P=5P=5). “–”: no trigger in the available trace. Order: 1.7B/Med., 1.7B/Sci., 4B/Med., 4B/Sci. Bold: the deployed configuration, reported at the steps at which it fired during training. Selected-token fraction. The cutoff of §3.3 is a percentile of the current rewards, and Table 11 reports the fraction f of response tokens it admits over Stage 1 in each setting. The mean stays within 0.476–0.505 and the minimum never falls below 0.37, with no per-setting tuning. Setting Min Mean Max 1.7B / Med. 0.402 0.496 0.594 1.7B / Sci. 0.403 0.499 0.588 4B / Med. 0.413 0.505 0.601 4B / Sci. 0.370 0.476 0.565 Table 11: Fraction of response tokens selected for distillation, over all Stage-1 steps of each reported run. What the divergence does and does not track. On 512 fixed RaR-Medicine questions graded by the training judge, the frozen teacher with the rubric scores 90.5, compared with 40.4 for the student at step 140. This measures generation with privileged context, not the improvement obtainable from distilling a teacher-forced distribution. Over the same Stage 1, the overlap between the models’ top-64 sets changes little while the monitored divergence falls by about two thirds: the overlap is 0.397, 0.424, 0.424 and 0.437 over steps 0–20, 60–80, 120–140 and 150–171, respectively. Top-1 agreement changes from 0.414 to 0.439. The divergence measures relative probabilities within the teacher’s top-64 set; it does not determine the student’s total probability on that set. Figure 7: Stage 1 response length (left) and per-token divergence (right) on Qwen3-1.7B/Medicine. Four selection rules from Table 5 use α=0.01α=0.01: Relay, positive only, positive plus negative, and all rollouts; GRPO is shown for reference. Appendix G Interpreting the Conditional KL The divergence in Eq. (2) compares distributions conditioned on the teacher’s top-k set. Its relation to full-vocabulary KL clarifies what the plateau statistic measures. At a fixed prefix, write K for this set, a=pT(K)a=p_T(K), and b=pθ(K)b=p_θ(K). Let ℓK=DKL(pT(⋅∣K)∥pθ(⋅∣K)) _K=D_KL(p_T(· K)\|p_θ(· K)), and define ℓKc _K^c analogously for the remaining vocabulary. For 0<a,b<10<a,b<1, the KL chain rule gives DKL(pT∥pθ) D_KL(p_T\|p_θ) =DKL(Ber(a)∥Ber(b)) =D_KL(Ber(a)\|Ber(b)) +aℓK+(1−a)ℓKc. +a\, _K+(1-a)\, _K^c. (9) Here KcK^c is the rest of the vocabulary and BerBer denotes a Bernoulli distribution over membership in K. The identity follows by writing p(v)=p(K)p(v∣K)p(v)=p(K)p(v K) for v∈Kv∈ K and splitting the full KL sum over K and KcK^c. Our loss is the conditional divergence in the second term, without the factor a. It omits both the set-mass term and the tail conditional divergence, so even zero loss does not imply equality of the full distributions. For student logits zvz_v, renormalization gives p¯θ(v)=exp(zv)/∑v′∈Kexp(zv′) p_θ(v)= (z_v)/ _v ∈ K (z_v ). Holding the teacher distribution and K fixed, differentiation yields ∂ℓ∂zv=p¯θ(v)−p¯T(v),v∈K,0,v∉K. ∂ ∂ z_v= cases p_θ(v)- p_T(v),&v∈ K,\\ 0,&v∉ K. cases (10) Adding a common constant to all logits in K leaves this loss unchanged while changing b. The zero derivative outside K concerns direct logit gradients: shared model parameters and softmax normalization can still change tail probabilities after an update. These properties motivate interpreting dtd_t as a monitor of progress on the selected conditional objective. It also depends on the student-generated prefixes and rollout selection, which change during training. A plateau therefore supplies an operational stopping signal, without implying convergence of full distributions, a fixed teacher-performance ceiling, or a necessary increase in full-vocabulary entropy. Figure 8: Validation score (starting points aligned) and token entropy for GRPO and Relay across all four settings (10-step average). Dotted lines mark Relay’s hand-off. Appendix H Hand-off Schedules Table 12 compares hand-off times against GRPO and against distilling to the end, and Table 4 gives the entropy trajectories of the same schedules. Schedule Best Δ Step Length GRPO 9.8 370 1,374 Always on 12.3 390 2,155 Hand-off @140 13.3 350 2,008 Hand-off @200 14.1 370 2,658 Relay (rule @171) 14.8 400 2,557 Table 12: Hand-off schedules (Qwen3-1.7B/Medicine). Best Δ : peak training-judge gain from step 0 (3-step smoothed), reached at the step shown. Length: mean training-rollout response length near the peak. Always on keeps α=0.01α=0.01 throughout. Appendix I Gradient Alignment For five checkpoints of the Qwen3-1.7B/Medicine run, we compute the cosine between the GRPO gradient and the OPSD gradient on selected rollouts (Table 13). From step 100 onward, the point estimates are small and take both signs. Inner products are estimated with an unbiased sketch and resampled over questions (2,000 draws); only the interval at initialization, [0.01,0.21][0.01,0.21], excludes zero. The aggregate OPSD gradients of positive- and negative-advantage rollouts have cosines of 0.97–0.99 across these checkpoints, motivating the analysis of token coverage and the average coefficient in §5.4. Ckpt cos (GRPO, OPSD) pos∼ ratio 0 +.16+.16 .99 .17 100 −.11-.11 .99 .09 140 +.01+.01 .98 .08 200 +.05+.05 .97 .11 260 −.06-.06 .98 .11 Table 13: Cosine between GRPO and OPSD gradients on selected rollouts, positive-vs-negative cosine, and norm ratio α‖gOPSD‖/‖gGRPO‖α\|g_OPSD\|/\|g_GRPO\|. Appendix J Selection and Training Dynamics Continuous weighting instead of a hand-off. We also trained four variants that keep the teacher throughout and vary the weight it receives rather than removing it: a half-Gaussian in the reward that is recentred each step, the same shape held fixed, a uniform per-step token budget, and a reduced weight on negative-advantage rollouts. Over steps 270–300 they gain 9.72, 9.66, 9.87 and 11.17 points from step 0, against 13.13 for the step-200 hand-off, so none of them matches simply removing the teacher. Training dynamics. CriPO’s entropy reaches 1.5–2.4 at training end (Figure 6), compared with 0.1–0.4 for Relay and 0.06–0.15 for GRPO (Figure 8).