Paper deep dive
Enhancing Rubric-based RL via Self-Distillation
Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/21/2026, 6:17:09 AM
Summary
The paper introduces Criterion-Distilled Policy Optimization (CriPO), a method to enhance Rubric-based Reinforcement Learning (RL) for Large Language Models by addressing two failure modes: Unexplored Criteria (UC) and Suppressed Criteria (SC). UC occurs when no rollout satisfies a criterion, while SC occurs when criterion-satisfying rollouts receive non-positive aggregate advantages due to scalar reward aggregation. CriPO uses on-policy self-distillation to inject missing behaviors for UC via a criterion-injection self-teacher and forward-KL loss, and preserves useful patterns for SC by using a counterfactual self-teacher to locate and flip token-level advantages. Experiments show CriPO outperforms baseline methods like GRPO with fewer optimization steps.
Entities (9)
Relation Signals (7)
CriPO → addresses → Unexplored Criteria
confidence 95% · For unexplored criteria, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy.
CriPO → addresses → Suppressed Criteria
confidence 95% · For suppressed criteria, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values
CriPO → uses → On-Policy Self-Distillation
confidence 93% · CriPO, which enhances rubric-based RL via on-policy self-distillation.
CriPO → outperforms → GRPO
confidence 90% · CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately 2x fewer optimization steps.
GRPO → suffersfrom → Suppressed Criteria
confidence 88% · GRPO aggregates criterion-wise scores into a single scalar reward... causing useful criterion-satisfying behaviors to be penalized or ignored
RuscaRL → introduces → train-inference mismatch
confidence 87% · these methods introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time
RuscaRL → uses → external guidance
confidence 85% · RuscaRL uses rubric criteria as explicit scaffolding to elicit more diverse and higher-quality rollouts
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Rubric-based RL has recently shown promise in improving LLMs on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria, UC) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout, yet they introduce a train-inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at inference time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria (SC) -- criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that SC are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 SC per sample. To simultaneously address both UC and SC without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For UC, CriPO constructs a criterion-injection self-teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For SC, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token-level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately $2\times$ fewer optimization steps.
Tags
Links
- Source: https://arxiv.org/abs/2607.18082v1
- Canonical: https://arxiv.org/abs/2607.18082v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
67,677 characters extracted from source content.
Expand or collapse full text
Enhancing Rubric-based RL via Self-Distillation Mingxuan Xia 1,2∗ Yuhang Yang 1,2∗ Chao Ye 2 Shuai Zhu 2 Shenzhi Yang 1 Guangcheng Zhu 1 Yuhang Zhang 2 Cheng Peng 1 Haobo Wang 1† Siqing Wang 2† 1 Zhejiang University 2 ByteDance xiamingxuan,yangyuhang,wanghaobo@zju.edu.cn wangsiqing.jacky@bytedance.com Abstract Rubric-based Reinforcement Learning (RL) has recently shown promise in improv- ing Large Language Models (LLMs) on open-ended tasks. A widely recognized limitation of rubric-based RL is limited exploration: criteria that no rollout manages to satisfy (Unexplored Criteria) receive no optimization signal. Recent methods address this by incorporating rubric information as external guidance during rollout generation, yet they introduce a train–inference mismatch: the policy is optimized on rollouts produced under external guidance while this guidance is absent at infer- ence time, causing error accumulation through autoregressive decoding. Moreover, these exploration-focused approaches overlook a fundamentally different failure mode that we term Suppressed Criteria—criteria that are satisfied by some rollouts yet whose learning signals are lost during optimization because scalar reward aggregation assigns them non-positive aggregate advantages. Our analysis reveals that suppressed criteria are remarkably prevalent: over 57% of samples exhibit this failure mode throughout training, with an average of 1.8 suppressed criteria per sample. To simultaneously address both unexplored and suppressed criteria without introducing training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric-based RL via on-policy self-distillation. For unexplored criteria, CriPO constructs a criterion-injection self- teacher and computes a localized forward-KL loss to inject missing behaviors into the policy. For suppressed criteria, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts and flips their token- level advantages to positive values, preserving useful patterns that would otherwise be suppressed. Experiments on medicine and science benchmarks demonstrate that CriPO consistently outperforms rubric-based RL, achieving stronger final performance with approximately 2× fewer optimization steps. 1 Introduction In recent years, Reinforcement Learning from Verifiable Rewards (RLVR), instantiated by methods such as GRPO [Shao et al., 2024], has emerged as a prominent paradigm for improving the reasoning capabilities of large language models [DeepSeek-AI, 2025, Cui et al., 2026]. RLVR has achieved notable success in verifiable domains such as mathematical reasoning and code generation, where exact answers or executable test cases provide stable and reliable supervision [He et al., 2024, Wei et al., 2025]. However, in open-ended scenarios such as medical consultation and long-form writing, model outputs typically do not admit a unique gold answer and are difficult to verify with deterministic rules. To extend RLVR to such domains, recent work has explored rubric-based rewards, which decompose response quality into interpretable evaluation criteria, use LLM-as-a-Judge [Gu ∗ Equal contribution. Work done during internship at ByteDance. † Corresponding author. arXiv:2607.18082v1 [cs.LG] 20 Jul 2026 Rollout y 1 y 2 y 3 y 4 (3 pts) Reward C1 Adv. 6 4 4 3 +1.61 -0.23 -0.23 -1.15 Rubric-based Reward & GRPO Advantage Satisfy Dissatisfy (3 pts) C2 (1 pts) C3 (1 pts) C4 Unexplorable Criteria Suppressed Critieria Figure 1: Left: Two failure modes in GRPO: Unexplored Criteria (UC), which are never satisfied by any rollout in the group (i.e., C4), and Suppressed Criteria (SC), for which criterion-satisfying rollouts receive non-positive aggregate advantages (i.e., C3). Right: The ratio of samples with UC/SC and the average number of UC/SC per sample when training Qwen3-4B on RaR-Medicine, showing both UC and SC are prevalent and persistent in practice. et al., 2024a] to assign multi-dimensional scores, and aggregate them into a scalar reward for RL training [Gunjal et al., 2025, Viswanathan et al., 2025, Liu et al., 2026a]. While rubric-based RL has shown promising results in these open-ended domains, it still faces a fundamental challenge of limited exploration: since GRPO only optimizes over behaviors present in the sampled rollout group [Yue et al., 2025, Nguyen et al., 2025], criteria that no rollout manages to satisfy, termed Unexplored Criteria, receive no effective optimization signal. Recent methods address this issue by using rubrics as external guidance during rollout generation. For example, RuscaRL [Zhou et al., 2025] uses rubric criteria as explicit scaffolding to elicit more diverse and higher-quality rollouts, while HeRL [Zhang et al., 2026] leverages failed trajectories and their unmet criteria as hindsight guidance for generating revised rollouts. However, these exploration- enhancement methods still have two important drawbacks. First, they introduce a training-inference mismatch: during training, rollouts are generated with privileged information, so each token is optimized under externally guided prefixes; at inference time, however, the model must generate solely from its own prefixes. This discrepancy constitutes exposure bias [Agarwal et al., 2024, Song and Zheng, 2026, Li et al., 2026b], where early generation error that could be avoided during guided training may propagate through subsequent autoregressive decoding. Such error accumulation is especially problematic for open-ended tasks, where long responses amplify the mismatch. Second, these methods overlook another fundamentally different failure mode beyond unexplored criteria: Suppressed Criteria—already explored criteria whose learning signals are systematically lost during GRPO optimization. This occurs because GRPO aggregates criterion-wise scores into a single scalar reward and broadcasts it uniformly to the entire response. Consequently, rollouts that satisfy certain criteria but underperform on others receive negative or negligible aggregate advantages, causing useful criterion-satisfying behaviors to be penalized or ignored rather than reinforced. For example, in Figure 1,y 2 andy 3 satisfy criterion C3 but their advantage sum is negative (-0.46). Our statistical analysis in Figure 1 reveals that such suppressed criteria are prevalent in practice: when training Qwen3-4B on RaR-Medicine, over 57% of samples contain suppressed criteria throughout training, with an average of 1.8 such criteria per sample. While some prior works [Chen et al., 2026, Lan, 2026, Xu et al., 2026] also recognize that scalar reward aggregation can obscure individual criterion contributions, they do not provide a systematic analysis of when and why criterion-level signals are lost, and they also fail to address the complementary challenge of unexplored criteria. To address both unexplored and suppressed criteria without introducing off-policy mismatch, in this paper, we propose Criterion-Distilled Policy Optimization (CriPO), which enhances rubric- based RL via On-Policy Self-Distillation (OPSD) [Zhao et al., 2026, Hübotter et al., 2026, Ye et al., 2026]. Unlike the approaches that inject knowledge or behaviors by producing rollouts under privileged information, OPSD constructs a privilege-conditioned self-teacher and provides token- level supervision on the model’s own on-policy rollouts, thereby obtaining learning signals without any distributional gap between training and inference. Specifically, for unexplored criteria, CriPO constructs a criterion-injection self-teacher and computes forward-KL divergence as an auxiliary loss to inject missing criterion-specific behaviors into the policy. For suppressed criteria, CriPO 2 similarly leverages the OPSD paradigm by constructing a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts, and locally flips their advantages to positive values so that useful patterns are preserved rather than suppressed. Together, these two interventions form our framework, which jointly resolves both unexplored and suppressed criteria within a unified on-policy training paradigm. Note that we adopt GRPO as the optimization backbone rather than using standalone OPSD, since applying OPSD alone fails to achieve stable performance (see Section 2 for discussions). Experiments on medicine and science benchmarks demonstrate the effectiveness and efficiency of CriPO. Across Qwen3-1.7B and Qwen3-4B, CriPO consistently outperforms GRPO, HeRL, and OPSD in both in-domain and cross-domain settings, achieving an average gain of+3.2 over GRPO on Qwen3-1.7B. Moreover, CriPO improves optimization efficiency, reaching GRPO’s converged performance with roughly 2× fewer steps. 2 Preliminaries Rubric-based Reinforcement Learning (RL). We adopt Group Relative Policy Optimization (GRPO) [Shao et al., 2024] as the core RL algorithm for training LLMs with rubric-based rewards. Given a promptx, the policyπ θ samples a group of on-policy rolloutsy i G i=1 , wherey i ∼ π θ (·| x). In rubric-based RL, each rollout is evaluated with a set of criteriaC = (c j ,ω j ) M j=1 , wherec j denotes an evaluation criterion andω j its importance weight. A judge model assigns a criterion-wise scorer ij = c j (x,y i ), indicating how well rollouty i satisfies criterionc j . These criterion-wise scores are then aggregated into a scalar reward: R i = P M j=1 ω j r ij P M j=1 ω j ,(1) GRPO then computes a group-relative advantage by normalizing the reward within the group:A i = (R i − μ R )/(σ R + ε), whereμ R andσ R are the mean and standard deviation of the group’s reward, andε > 0avoids division by zero. The policy model is optimized with the PPO-style [Schulman et al., 2017] clipped surrogate objective: L GRPO =−E x,y i G i=1 ∼π θ old (·|x) 1 G G X i=1 1 |y i | |y i | X t=1 min(ρ i,t A i , clip (ρ i,t , 1− ε clip , 1 + ε clip )A i ) , (2) whereρ i,t = π θ (y i,t | x,y i,<t )/π θ old (y i,t | x,y i,<t )is the importance ratio andπ θ old is the frozen rollout policy and π θ is the current policy with gradient. Failure in Rubric-based RL: Unexplored and Suppressed Criteria. Although rubric-based rewards provide criterion-level supervision, two inherent limitations of GRPO prevent this supervision from being fully exploited: (i) limited exploration: GRPO only optimizes over behaviors present in the sampled rollouts so that criteria with no rollout managed to satisfy receive no optimization signal; and (i) reward ambiguity: aggregating multiple criteria into a single scalar reward obscures criterion- specific token contributions and may penalize useful behaviors. We show that these limitations lead to two types of failure criteria: • Unexplored Criteria correspond to behaviors absent from the current rollout group due to limited exploration (e.g., C4 in Figure 1). Consider the setting where each criterion-wise score is binary, i.e.,r ij = c j (x,y i )∈0, 1, indicating whether the corresponding criterion is satisfied or not. The unexplored criteria are defined as criteria that are not satisfied by any rollout in the current group: C u =c j ∈C |∀y i , r ij = 0.(3) • Suppressed Criteria correspond to behaviors that the model has already discovered, but the criterion- satisfying rollouts receive non-positive aggregate advantages due to reward ambiguity (e.g., C3 in Figure 1). Specifically, we consider the following two suppression scenarios: (i) negative-advantage suppression, where the advantage sum across all satisfying rollouts is negative, causing the criterion- satisfying behavior penalized, and (i) zero-advantage suppression, where the advantage sum is zero and the criterion is rarely satisfied (e.g., fewer than half the group), so that the criterion receives negligible optimization pressure despite being insufficiently learned. LetS j =i| r ij = 1denote 3 the rollout set that satisfies criterion c j , the suppressed criteria are defined as: C s = c j ∈C S j ̸=∅ ∧ X i∈S j A i < 0 ∨ X i∈S j A i = 0 ∧ |S j | < G/2 .(4) As illustrated in Figure 1 on the right, when training Qwen3-4B on medicine tasks, more than 83% of samples contain unexplored criteria (with an average of 2.5 per sample) and over 57% contain suppressed criteria (with an average of 1.8 per sample) throughout the training process, which indicates that both unexplored criteria and suppressed criteria are prevalent and persistent in practice. As discussed in Section 1, existing exploration-enhancement methods [Zhou et al., 2025, Zhang et al., 2026] address unexplored criteria by generating rollouts conditioned on rubric information, but introduce a training-inference distribution mismatch, and they also overlook the issue of suppressed criteria. This motivates a purely on-policy approach that can jointly resolve both failure modes. On-Policy Self-Distillation (OPSD) [Zhao et al., 2026, Hübotter et al., 2026, Ye et al., 2026] has recently emerged as a promising paradigm for converting privileged information into dense token-level supervision while avoiding the training-inference distribution mismatch [Agarwal et al., 2024, Gu et al., 2024b]. Given an on-policy rollouty ∼ π θ (· | x), OPSD compares two next- token distributions produced by the same model: a student distribution conditioned only on the original context, and a self-teacher distribution conditioned additionally on privileged informationξ. Specifically, the policy is trained to match the privilege-conditioned prediction through a token-level divergence at each token position t (reverse-KL instantiation): L OPSD =E x,y∼π θ (·|x) 1 |y| |y| X t=1 D KL (π θ (·| x,y <t )∥ sg(π θ (·| x,ξ,y <t ))) ,(5) wheresg(·)denotes the stop-gradient operation. Compared with standard RLVR methods that optimize sparse outcome rewards, OPSD provides fine-grained supervision at each token position and can reveal how the privileged information changes the model’s local generation preference. 255075100 Training Step 0.32 0.40 0.48 0.56 KL-JSD KL-FKL KL-RKL PG-RKL OPSD Reward Figure 2: Reward dynamics when train- ing Qwen3-4B on RaR-Medicine using OPSD alone. Different KL variants all result in performance degradation. In rubric-based RL, rubric criteria can naturally serve as the privileged informationξ. By conditioning the self- teacher on rubrics or criteria feedback, OPSD can expose criterion-level supervision, which appears to offer a uni- fied solution to the two limitations of GRPO discussed above. However, our preliminary experiments show that directly applying OPSD as the rubric-based RL algo- rithm is unstable 3 . As shown in Figure 2, when treating all rubrics as the privileged teacher information, training Qwen3-4B on RaR-Medicine using OPSD alone results in performance degradation. This observation is consis- tent with recent findings that OPSD suffer from several intrinsic issues that lead to unstable training, including privileged-information leakage, unreliable self-teacher sig- nals, or entropy collapse [Yang et al., 2026, Kim et al., 2026b, Zhu et al., 2026, Li et al., 2026a, Kim et al., 2026a]. Such issues can be further amplified in open-ended gen- eration with rubric-based rewards, where responses are typically long, and each prompt may involve multiple criteria, resulting in noisy supervision. To this end, we propose to retain GRPO as the stable reward-grounded optimization backbone, while incorporating OPSD as an auxiliary module that addresses the failure modes of GRPO. This hybrid design is aligned with recent methods that increasingly combine OPSD with RLVR rather than relying on standalone OPSD [Li et al., 2026a, Yang et al., 2026, Kim et al., 2026a, Wang et al., 2026, Liu et al., 2026b], as it preserves the training stability and reward-aligned optimization direction of GRPO while benefiting from the dense token-level supervision provided by OPSD. 3 In this paper, we implement OPSD following the code base: https://github.com/lasgroup/SDPO [Hübotter et al., 2026] 4 Question x Policy Model � � y G Rubric Evaluator � 1 � 2 y 1 y 2 Rollouts Update Policy y 3 Teacher � � (∙|�,�,�) Student � � (∙|�,�) Behavior Injection Generate Token-level dense guidance Teacher � � (∙|�,�) Student � � (∙|�,�) Advantage Flipping Suppressed-Criterion Token Localization Filp Advantage at Localized Tokens GRPO Loss Suppressed Criteria Counterfactual prompt Criterion-injection prompt Contribution-based Token Selection Unexplored Criteria Unexplored Suppressed � 3 � 4 OPSD Loss + ️ ×� Best rollout y Negative rollout y Figure 3: Overview of CriPO. CriPO first identifies unexplored and suppressed criteria from rubric- evaluated rollouts. For unexplored criteria, CriPO injects the missing criterion-specific information through OPSD Loss. For suppressed criteria, CriPO compares the student with a counterfactual teacher to locate criterion-relevant tokens, and then flips their advantages to modify GRPO Loss. The two objectives are combined to update the policy. 3 CriPO: Criterion-Distilled Policy Optimization To simultaneously address the two limitations without introducing the training-inference mismatch, we propose Criterion-Distilled Policy Optimization (CriPO), which retains GRPO as the stable reward-grounded optimization backbone and incorporates OPSD-derived signals to tackle both unexplored criteria and suppressed criteria. Specifically, for unexplored criteria, CriPO constructs a criterion-injection self-teacher and computes forward-KL divergence as an auxiliary loss to inject missing criterion-specific behaviors into the policy. For suppressed criteria, CriPO employs a counterfactual self-teacher to locate criterion-relevant tokens in negative-advantage rollouts, and locally flips their advantages to positive values so that useful patterns are preserved rather than suppressed. An overview of the framework is shown in Figure 3. 3.1 Behavior Injection for Unexplored Criteria For unexplored criteria that are not satisfied by any rollout in the current group, the goal is to inject the missing criterion-specific behaviors into the policy. To this end, we construct a criterion-injection self-teacher that revises the previous response conditioned on the unexplored criteria. The prompt template is shown as follows: Given the following user query: user_query Below is your previous response to the query: previous_response This response failed to meet the following criteria: unexplored_criteria Please revise the previous response with the minimum necessary to satisfy the unmet criteria. Output only the revised response. Formally, letp S t = π θ (· | x,y <t )andp T t = π θ (· | x,C u ,y <t )denote the student and criterion- injection self-teacher distributions at token positiont. To transfer the teacher’s criterion-conditioned behavior to the student, we compute the per-token forward-KL divergenced t = D KL sg(p T t ) p S t . We adopt forward KL because it enables the student to cover teacher-preferred modes, which encourages exploration and is desirable for injecting behaviors that are entirely absent from the current rollout group. Moreover, to make the distillation effective and reliable, the behavior injection module is guided by two design principles: 5 • Best-Rollout Selection. We perform behavior injection exclusively on the highest-advantage rollout in the group. Since this rollout already satisfies the largest proportion of criteria, the self-teacher only needs to make minimal revisions to incorporate the remaining unexplored criteria, yielding more reliable and less noisy corrections than revising an arbitrary or low-quality response. 020406080100 Top Token Proportion Sorted by Forward KL 0 20 40 60 80 100 Top 34.6% tokens contribute 95% KL Cumulative KL Contribution (%) Figure 4: Cumulative KL contribution across tokens sorted by magnitude. Only 34.6% of tokens contribute 95% of the total KL. • Contribution-Guided Token Filtering. Since the self-teacher’s revision is localized—only a few token positions are actually modified to satisfy the missing criteria—the teacher distribution shifts substantially only at those positions. The remaining tokens ex- hibit small KL values that primarily reflect perturba- tions from prompt modification rather than genuine criterion-specific signals. As shown in Figure 4, when training Qwen3-4B on the first data batch of RaR- Medicine, only 34.6% of tokens account for 95% of the total divergence. We therefore apply distillation only on the smallest set of tokens whose cumulative KL contribution reaches a threshold γ: T u = TopCum d t |y| t=1 ; γ ,(6) whereγ = 0.95by default. This localized strategy enables the student to absorb missing criterion- specific behaviors at the most informative positions while filtering out noisy supervision from irrelevant tokens. The overall loss of behavior injection is as follows: L OPSD =E x,y i G i=1 ∼π θ (·|x) " 1 |T u | X t∈T u D KL (sg(π θ (·| x,C u ,y ∗ <t ))∥π θ (·| x,y ∗ <t )) # ,(7) wherey ∗ is the highest-advantage rollout in the group andT u = TopCum(d t |y ∗ | t=1 ;γ)restricts distillation to the most informative token positions. 3.2 Advantage Flipping for Suppressed Criteria For suppressed criteria, our goal is to preserve criterion-satisfying parts in negative-advantage rollouts without promoting the entire trajectory. Unlike unexplored criteria, for which we introduce an additional OPSD loss, we handle suppressed criteria by first localizing the criterion-satisfying tokens in negative rollouts, then flipping their advantages to positive values, and finally feeding the corrected token-level advantages back into GRPO for policy optimization. Suppressed-Criterion Token Localization. For each negative advantage rollout in the current group, letC i s = c j ∈ C s |r ij = 1 denote the set of suppressed criteria satisfied by rollouti. To identify the tokens that encode these suppressed criteria, we construct a counterfactual teacher prompt that asks the self-teacher to revise the original response by modifying or removing only the parts that satisfy these criteria. This produces a contrastive version of the same rollout, where suppressed criterion-specific behaviors are intentionally weakened while the rest of the response is minimally changed. The prompt template is shown as follows: Given a response: previous_response Please revise the response with minimum necessary by modifying or deleting the parts that satisfy the following criteria: satisfied_suppressed_criteria Output only the revised response. We then compare the original student distributionp S i,t = π θ (· | x,y i,<t )with the counter- factual teacher distributionp T i,t = π θ (· | C i s ,y i,<t ) to identify affected tokens. Specifically, we select positions where (i) the sampled token is down-weighted after criterion removal, i.e., ∆ i,t = logp S i,t (y i,t )− logp T i,t (y i,t ) > 0, and (i) the teacher confidently prefers an alternative token at that position, i.e.,p T i,t (y i,t ) < α· max v p T i,t (v), whereαis a hyperparameter and set as 0.1 by default. The second condition ensures that only positions where the teacher actively replaced the 6 Algorithm 1 CriPO: Criterion-Distilled Policy Optimization Require: Policy π θ ; rubric criteriaC; rollout number G; hyper-parameters γ,α,τ flip ,β 1: repeat 2:Sample rolloutsy i G i=1 for prompt x, evaluate them with rubric criteriaC 3:Compute rewardsR i , group advantagesA i and identify CriteriaC u ,C s 4:Initialize token advantages ̃ A i,t ← A i and OPSD lossL OPSD ← 0 ▷ Behavior Injection for Unexplored Criteria 5:ifC u ̸=∅ then 6:Select the highest-advantage rollout y ∗ , build teacher and compute forward-KL d t 7:Select tokensT u following Eq.(6) and computeL OPSD following Eq.(7) ▷ Advantage Flipping for Suppressed Criteria 8:ifC s ̸=∅ then 9:for each rollout y i with A i < 0 that satisfies criteria inC s do 10:Build teacher and select tokensT s i following Eq.(8) 11:Set ̃ A i,t ← τ flip for t∈T s i and computeL GRPO following Eq.(10) 12:ComputeL CriPO following Eq.(11) and update θ 13: until converged original token are selected, rather than positions where both student and teacher lack confidence. The resulting salient token set is defined as: T s i = t ∆ i,t > 0 ∧ p T i,t (y i,t ) < α· max v p T i,t (v) (8) For these tokens, we replace the original negative rollout advantage with a positive value: ̃ A i,t = τ flip , t∈T s i , A i ,otherwise. (9) whereτ flip = 0.1by default. We then compute the GRPO loss by replacing the original rollout-level advantage with ̃ A i,t . In this way, the global GRPO optimization structure is preserved, while the update direction is locally reversed for tokens that encode useful but suppressed criterion-specific behaviors. The resulting loss then becomes: L GRPO =−E x,y i G i=1 ∼π θ old (·|x) 1 P i |y i | G X i=1 |y i | X t=1 min ρ i,t ̃ A i,t , clip (ρ i,t , 1− ε clip , 1 + ε clip ) ̃ A i,t , (10) Note that we use token-mean aggregation following the default implementation of GRPO in the verl code base [Sheng et al., 2025]. 3.3 Training Objective In summary, CriPO is an on-policy training framework that preserves GRPO’s stable reward-oriented optimization while using OPSD-derived signals for targeted criterion-level correction for both failure modes in GRPO. The overall training objective of our proposed CriPO framework is: L CriPO =L GRPO + βL OPSD ,(11) where β controls the strength of OPSD loss. Algorithm 1 shows the pseudo-code of CriPO. 4 Experiment 4.1 Experimental Setup Datasets and Evaluation.We evaluate CriPO on both medicine and science QA tasks, where RaR- Medicine and RaR-Science [Gunjal et al., 2025] are adopted as the training datasets for medicine and science tasks, respectively. We use their own test split for in-domain evaluation, and for cross-domain 7 Table 1: Experimental results (%) across medicine and science benchmarks using GPT-4o-mini as judge. The best results are marked in bold, and the second-best areunderlined. Arrows indicate changes over the base model. All results are evaluated on models trained after 200 steps. Method MedicineScience Avg. RaR-MedicineHealthBenchLLMEval-MedRaR-ScienceResearchQA Qwen3-1.7B48.6↑ 0.060.2↑ 0.058.0↑ 0.062.6↑ 0.056.5↑ 0.057.2↑ 0.0 + GRPO 49.2↑ 0.660.5↑ 0.358.5↑ 0.566.6↑ 4.061.3↑ 4.859.2↑ 2.0 + HeRL 50.2↑ 1.660.8↑ 0.660.6↑ 2.667.1↑ 4.663.2↑ 6.760.4↑ 3.2 + OPSD26.3↓ 22.338.5↓ 21.748.4↓ 9.639.6↓ 23.038.4↓ 18.138.2↓ 18.9 + CriPO-U51.3↑ 2.764.3↑ 4.158.3↑ 0.369.5↑ 6.964.8↑ 8.361.6↑ 4.4 + CriPO-S 50.8↑ 2.261.0↑ 0.862.7↑ 4.766.8↑ 4.260.5↑ 4.060.4↑ 3.2 + CriPO51.6↑ 3.065.6↑ 5.462.2↑ 4.268.1↑ 5.664.7↑ 8.262.4↑ 5.2 Qwen3-4B55.7↑ 0.063.9↑ 0.067.3↑ 0.069.8↑ 0.061.8↑ 0.063.7↑ 0.0 + GRPO60.2↑ 4.569.8↑ 5.967.8↑ 0.574.4↑ 4.668.7↑ 6.968.2↑ 4.5 + HeRL 60.5↑ 4.869.5↑ 5.668.8↑ 1.574.3↑ 4.568.8↑ 7.068.4↑ 4.7 + OPSD42.0↓ 13.742.1↓ 21.850.5↓ 16.838.3↓ 31.545.2↓ 16.743.6↓ 20.1 + CriPO-U60.5↑ 4.968.5↑ 4.670.0↑ 2.775.0↑ 5.270.1↑ 8.268.8↑ 5.1 + CriPO-S 61.2↑ 5.572.7↑ 8.872.4↑ 5.173.4↑ 3.668.6↑ 6.869.6↑ 5.9 + CriPO62.1↑ 6.470.9↑ 7.069.9↑ 2.675.0↑ 5.270.1↑ 8.269.6↑ 5.9 evaluation, we use HealthBench [Arora et al., 2025] and LLMEval-Med [Zhang et al., 2025] for medicine tasks, and ResearchQA [Yifei et al., 2025] for science tasks. We use Qwen3-32B [Team, 2025] as the rubric judge during training and use GPT-4o-mini [OpenAI, 2023] as the judge for evaluation. More details about the datasets and evaluation settings are provided in Appendix B. Baselines and Models. We conduct experiments on Qwen3-1.7B and Qwen3-4B, and compare CriPO with representative baselines: GRPO [Shao et al., 2024], the standard group-relative policy optimization baseline; HeRL [Zhang et al., 2026], a rubric-based RL method that leverages hindsight feedback from failed trajectories and unmet rubrics to enhance exploration; and OPSD [Zhao et al., 2026], an on-policy self-distillation baseline that distills all criteria in the rubric into the model (our implementation follows the code base in SDPO [Hübotter et al., 2026]). We also compare CriPO with two of its variants: CriPO-U, which only addresses unexplored criteria without advantage flipping, and CriPO-S, which only addresses suppressed criteria without behavior injection. All methods are trained on the same data and evaluated under the same protocol for fair comparison. Implementation details are provided in Appendix B. 4.2 Main Results CriPO outperforms existing baselines. As shown in Table 1, CriPO consistently outperforms existing baselines across both model scales and evaluation domains. Compared with GRPO, CriPO improves the average score from59.2to62.4on Qwen3-1.7B (+3.2) and from68.2to69.6on Qwen3-4B (+1.4), demonstrating the effectiveness of criterion-level correction across medical and scientific benchmarks. Compared with HeRL, which relies on off-policy hindsight rollouts for exploration, CriPO-U achieves stronger performance on both model scales while remaining fully on- policy. This suggests that localized self-distillation can more effectively recover unexplored criteria without introducing training-inference mismatch. We further observe that the single-intervention variants achieve the best performance on several individual benchmarks, whereas the full CriPO consistently delivers the strongest average results. This indicates that behavior injection and advantage flipping address complementary criterion failure modes, and their combination yields more effective optimization than relying on aggregated scalar rewards alone. Efficiency and Convergence Analysis. We further analyze the optimization efficiency of CriPO. As shown on the left of Figure 5, CriPO reaches the best performance achieved by GRPO at around step 175, requiring roughly2.0×fewer optimization steps, and further converges to a higher final accuracy. The wall-clock comparison on the right shows a similar trend across model scales and 8 080160240320400 Training Step 51% 54% 57% 60% Accuracy@4 (%) 2.0x fewer steps CriPO surpasses GRPO best at step 175 CriPO GRPO Qwen3-1.7B RaR-Medicine Qwen3-4B RaR-Medicine Qwen3-1.7B RaR-Science Qwen3-4B RaR-Science 0.0 2.5 5.0 7.5 10.0 12.5 15.0 Wall-clock Time 6h14m 8h37m 7h42m 11h32m 6h56m 11h02m 8h52m 14h06m 2h26m 5h04m 6h32m 11h29m GRPO timeCriPO timeCriPO exceeds GRPO Figure 5: Compute efficiency of CriPO. Left: CriPO surpasses the best GRPO performance at step 175 with about2.0×fewer optimization steps. Right: CriPO reaches better-than-GRPO performance before completing its full training budget across model scales and domains. 4080120160200 Training Step 0.48 0.51 0.54 0.57 0.60 Reward 4080120160200 Training Step 0.16 0.24 0.32 0.40 Entropy 4080120160200 Training Step 0 400 800 1200 1600 2000 Length (tokens) GRPOCriPO-UCriPO-SCriPO Figure 6: Training dynamics of CriPO. We compare CriPO and GRPO across reward, entropy, and response length over 200 training steps. CriPO achieves higher rewards, maintains greater entropy for exploration, and produces longer responses that better support rubric-relevant content. domains: despite the extra per-step computation, CriPO surpasses GRPO before GRPO training completes. These results show that CriPO trades modest per-step overhead for substantially improved optimization efficiency and stronger final performance. 4.3 Analysis Training Dynamics. Figure 6 illustrates the training dynamics over 200 optimization steps. As shown in Figure 6 (left), both CriPO-U and CriPO-S achieve higher reward trajectories than standard GRPO, with the full CriPO obtaining the highest reward among all methods. Figure 6 (middle) shows that CriPO variants maintain higher entropy than GRPO throughout training. In particular, CriPO-U brings a notably larger relative entropy gain, reflecting the exploration benefit of the behavior injection mechanism. Figure 6 (right) further shows that CriPO-S produces longer responses than GRPO, as it encourages the model’s suppressed desirable behaviors. We provide a case study in Appendix A.3 showing that longer responses reflect deeper reasoning and broader exploration. Statistics of Unexplored and Suppressed Criteria. Figure 7 reports the occurrence ratio and average count of unexplored and suppressed criteria on the RaR-Medicine test set with Qwen3-4B. GRPO still suffers from both failure modes, while CriPO consistently reduces their prevalence. The only exception is the average number of suppressed criteria, which increases for both GRPO and CriPO compared with the base model. We conjecture that this is due to improved exploration during training: as the model learns to satisfy more criteria, it also encounters more cases where certain criteria are satisfied but overwhelmed by other rubric dimensions, leading to suppression. Nevertheless, CriPO still achieves a lower suppressed-criteria count than GRPO, demonstrating its ability to mitigate this issue. Moreover, the single-intervention variants exhibit clear failure-mode- specific improvements: CriPO-U achieves the largest reduction in unexplored criteria, while CriPO-S 9 Unexplored CriteriaSuppressed Criteria 40 50 60 70 80 90 Ratio of Samples (%) 85.5 53.6 84.8 52.6 80.9 54.6 86.3 50.0 82.7 51.4 Unexplored CriteriaSuppressed Criteria 1.0 1.5 2.0 2.5 3.0 Avg. Number per Sample 2.76 1.50 2.59 1.70 2.31 1.64 2.56 1.65 2.39 1.69 BaseGRPOCriPO-UCriPO-SCriPO Figure 7: Statistics of Unexplored and Suppressed Criteria. We report their occurrence ratio and average count across methods. CriPO reduces both types of criteria compared with GRPO, while CriPO-U and CriPO-S show targeted effects on their corresponding failure modes. A35-weekpregnant womanwithpainlessvaginalbleeding ismostlikelyto haveplacenta previa.Keyevidence: -Painless third-trimesterbleeding -Relaxed uterus heart toultrasound confirmplacental stability 3.AvoiddigitalexaminationandmanipulationtopreventworseningConsiderother causes including placental abruption Answer : ultrasound to confirm previa . . .-sounds. -Floating presentingpart . : 1. location . 2. Monitor bleeding,fetalwell-being,. . Regular Perform and maternal bleeding.4. Perform Suppressed Criteria: Advises against performing invasive examinations that could exacerbate the bleeding. The teacher primarily shifts its prediction at the token “Avoid”, correctly attributing the blind criterion to the relevant span in the rollout. progression of Figure 8: Case Study of Token Selection for Suppressed Criteria. The counterfactual self-teacher correctly identifies the token “Avoid” as criterion-relevant for localized advantage correction. is most effective at reducing suppressed criteria. These results verify that each component targets its intended failure mode, and their combination further improves overall performance. Case Study of Token Selection for Suppressed Criteria.Figure 8 illustrates how CriPO identifies criterion-relevant tokens for a Suppressed Criterion. In this example, the response satisfies the criterion of avoiding invasive examinations in suspected placenta previa, as reflected by the span Avoid digital examination and manipulation to prevent worsening bleeding.” However, GRPO may still assign a negative advantage due to failures on other rubric dimensions. CriPO constructs a counterfactual self-teacher by weakening the satisfied criterion and uses token-level shifts to localize its contribution. The largest discrepancy appears around Avoid” and its surrounding phrase, demonstrating that CriPO attributes the criterion to relevant local spans rather than the entire response. This allows selective advantage correction on criterion-bearing tokens while preserving other behaviors. 4.4 Ablation Studies Table 2: Ablation study on CriPO. “w/o filter” re- moves contribution-guided token filtering, “w/o best” removes best-rollout selection, and “with random” localizes suppressed-criteria token randomly. Method Medicine Avg. RaR-Med. Health. LLMEval. Behavior Injection CriPO-U60.568.570.066.4 CriPO-U w/o filter59.869.067.665.5 CriPO-U w/o best60.269.067.665.6 Advantage Flipping CriPO-S61.272.772.968.9 CriPO-S with random59.167.068.564.9 Table 2 ablates the key design choices in CriPO on Qwen3-4B. For behavior injection, removing contribution-guided token filtering consistently weakens CriPO-U, reducing the average score from66.4to65.5. This con- firms that dense OPSD over all tokens in- troduces noisy supervision, while selecting high-contribution tokens helps focus the self- distillation signal on positions most affected by the missing criterion information. Remov- ing best-rollout selection also hurts perfor- mance, decreasing the average score to65.6, which suggests that injecting missing criteria 10 into a stronger rollout provides a more reliable target than applying the intervention to arbitrary responses. For advantage flipping, replacing suppressed criterion token localization with random selection leads to a much larger degradation, dropping the average score from68.9to64.9. This indicates that the benefit of CriPO-S does not come from simply increasing the advantage of ad- ditional tokens; rather, it relies on accurately locating tokens that encode useful but suppressed criterion-specific behaviors. 5 Related Work Rubric-based RL.Rubrics provide a structured interface for extending RLVR to open-ended tasks where response quality cannot be verified by exact answers or executable tests. By decomposing quality into explicit criteria, such as factuality, completeness, safety, evidence grounding, and task utility, rubric-based rewards offer more interpretable and controllable supervision than holistic scalar judgments [Arora et al., 2025, Du et al., 2025]. Existing rubric-based RL methods typically aggregate criterion-wise scores from an LLM judge into a scalar reward for PPO or GRPO optimization [Gunjal et al., 2025, Viswanathan et al., 2025, Wu et al., 2025]. Subsequent work improves this paradigm by refining rubric construction, reward aggregation, or criterion weighting, making rubric rewards more discriminative and robust [Chen et al., 2026, Lan, 2026]. A parallel line of work uses rubric feedback to enhance exploration. RuscaRL [Zhou et al., 2025] conditions rollout generation on rubric guidance, while HeRL [Zhang et al., 2026] uses hindsight feedback from failed trajectories to construct revised rollouts. Although these methods help discover behaviors associated with previously unmet criteria, they rely on privileged information during rollout generation, introducing a training-inference mismatch, and also overlook another crucial failure of suppressed criteria. In contrast, CriPO recovers criterion-level supervision for both failure modes for standard rubric-based RL through on-policy self-distillation, without relying on externally guided rollouts. On-policy Self-distillation. On-policy Self-distillation (OPSD) uses the current policy under privileged or feedback-augmented contexts as a self-teacher, converting auxiliary information into dense token-level supervision while avoiding off-policy teacher mismatch [Agarwal et al., 2024, Zhao et al., 2026, Hübotter et al., 2026]. Despite its promise, dense OPSD is often unstable due to noisy token-level gradients, teacher–student inconsistency, and privileged-information leakage [Yang et al., 2026, Kim et al., 2026b, Zhu et al., 2026, Li et al., 2026a, Kim et al., 2026a]. Recent work therefore increasingly combines OPSD with RLVR, using reference solutions, verified traces, or environment feedback to improve sparse-reward optimization [Yang et al., 2026, Lu et al., 2026, Pan et al., 2026]. Most of these studies focus on verifiable domains such as mathematical reasoning, where privileged supervision is relatively well aligned with the objective. Recent concurrent work has begun exploring OPSD for rubric-based post-training in open-ended tasks [Rezaei et al., 2026, Gu et al., 2026]. Unlike these methods, which primarily perform dense response-level self-distillation, CriPO retains GRPO as the reward-grounded backbone and uses OPSD to provide targeted corrections for unexplored and suppressed criteria through localized token-level supervision. 6 Conclusion In this work, we study rubric-based reinforcement learning for open-ended tasks and identify two prevalent criterion-level failure modes caused by scalar reward optimization: unexplored criteria, where no rollout satisfies a criterion and thus no learning signal is available, and suppressed cri- teria, where criterion-satisfying behaviors are penalized or ignored due to non-positive aggregate advantages. To address both issues, we propose Criterion-Distilled Policy Optimization (CriPO), which retains GRPO as a stable reward-grounded backbone while using on-policy self-distillation for targeted token-level correction. Specifically, CriPO injects missing behaviors for unexplored criteria through localized forward-KL distillation with a criterion-injection self-teacher and preserves useful suppressed behaviors by locating criterion-relevant tokens with a counterfactual self-teacher and flipping their token-level advantages. Experiments on medicine and science benchmarks show that CriPO consistently improves over GRPO and rubric-guided baselines across model scales and do- mains, while reaching the best performance of converged GRPO with roughly2×fewer optimization steps. We hope our work can inspire the community to further investigate how to jointly address both limited exploration and reward ambiguity in standard rubric-based RL. 11 References Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. Healthbench: Evaluating large language models towards improved human health. CoRR, abs/2505.08775, 2025. Yukun Chen, Jiaming Li, Longze Chen, Ze Gong, Jingpeng Li, Zhen Qin, Hengyu Chang, Ancheng Xu, Zhihao Yang, Hamid Alinejad-Rokny, Qiang Qu, Bo Zheng, and Min Yang. Rucl: Strati- fied rubric-based curriculum learning for multimodal large language model reasoning. CoRR, abs/2602.21628, 2026. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Yuchen Zhang, Jiacheng Chen, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards. Trans. Mach. Learn. Res., 2026, 2026. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. CoRR, abs/2501.12948, 2025. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. CoRR, abs/2506.11763, 2025. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. A survey on llm-as-a-judge. CoRR, abs/2411.15594, 2024a. Siyi Gu, Jialin Chen, Sophia Zhou, Arman Cohan, and Rex Ying. Rethinking reward supervision: Rubric-conditioned self-distillation. arXiv preprint arXiv:2606.19327, 2026. Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024b. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains. CoRR, abs/2507.17746, 2025. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 3828–3850. Association for Computational Linguistics, 2024. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, and Andreas Krause. Reinforcement learning via self-distillation. CoRR, abs/2601.20802, 2026. Jeonghye Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Rebellious student: Reversing teacher signals for reasoning exploration with self-distilled RLVR. CoRR, abs/2605.10781, 2026a. Jeonghye Kim, Xufang Luo, Minbeom Kim, Sangmook Lee, Dohyung Kim, Jiwon Jeon, Dongsheng Li, and Yuqing Yang. Why does self-distillation (sometimes) degrade the reasoning capability of llms? CoRR, abs/2603.24472, 2026b. Guangchen Lan. Alternating reinforcement learning with contextual rubric rewards. CoRR, abs/2603.15646, 2026. 12 Gengsheng Li, Tianyu Yang, Junfeng Fang, Mingyang Song, Mao Zheng, Haiyun Guo, Dan Zhang, Jinqiao Wang, and Tat-Seng Chua. Unifying group-relative and self-distillation policy optimization via sample routing. CoRR, abs/2604.02288, 2026a. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan- ang Gao, Wenkai Yang, Zhiyuan Liu, and Ning Ding. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. CoRR, abs/2604.13016, 2026b. Wenhan Liu, Jiajie Jin, Zhaoheng Huang, Tongyu Wen, Guanting Dong, Ziliang Zhao, Yutao Zhu, Zhicheng Dou, and Ji-Rong Wen. The rules of the game: A survey of rubrics for large language models. 2026a. Yifeng Liu, Shiyuan Zhang, Yifan Zhang, and Quanquan Gu. Self-distilled policy gradient. arXiv preprint arXiv:2606.04036, 2026b. Zhengxi Lu, Zhiyuan Yao, Zhuowen Han, Zi-Han Wang, Jinyang Wu, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, et al. Self-distilled agentic reinforcement learning. arXiv preprint arXiv:2605.15155, 2026. Phuc Minh Nguyen, Chinh D. La, Duy M. H. Nguyen, Nitesh V. Chawla, Binh T. Nguyen, and Khoa D. Doan. The reasoning boundary paradox: How reinforcement learning constrains language models. CoRR, abs/2510.02230, 2025. OpenAI. GPT-4 technical report. CoRR, abs/2303.08774, 2023. Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlcsd: Reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709, 2026. MohammadHossein Rezaei, Anas Mahmoud, Zihao Wang, Utkarsh Tyagi, Advait Gosai, Razvan- Gabriel Dumitru, Aakash Sabharwal, Bing Liu, and Yunzhong He. Rubric-guided self-distillation: Post-training without rubric verifiers. arXiv preprint arXiv:2606.12507, 2026. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 - 3 April 2025, pages 1279–1297. ACM, 2025. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. CoRR, abs/2604.00626, 2026. Qwen Team. Qwen3 technical report. CoRR, abs/2505.09388, 2025. Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, and Tongshuang Wu. Checklists are better than reward models for aligning language models. CoRR, abs/2507.18624, 2025. Jiaxuan Wang, Xuan Ouyang, Zhiyu Chen, Yulan Hu, Zheng Pan, Xin Li, and Lan-Zhe Guo. TRACE: distilling where it matters via token-routed self on-policy alignment. CoRR, abs/2605.10194, 2026. Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. SWE-RL: advancing LLM reasoning via reinforcement learning on open software evolution. CoRR, abs/2502.18449, 2025. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. Writingbench: A comprehensive benchmark for generative writing. CoRR, abs/2503.05244, 2025. 13 Tianze Xu, Yanzhao Zheng, Pengrui Lu, Lyumanshan Ye, Yong Wu, ZhenTao Zhang, YuanQiang Yu, Chao Ma, JiHuai Zhu, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, and Gang Yu. Rubrics to tokens: Bridging response-level rubrics and token-level rewards in instruction following tasks. CoRR, abs/2604.02795, 2026. Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled RLVR. CoRR, abs/2604.03128, 2026. Tianzhu Ye, Li Dong, Xun Wu, Shaohan Huang, and Furu Wei. On-policy context distillation for language models. CoRR, abs/2602.12275, 2026. Li S. Yifei, Allen Chang, Chaitanya Malaviya, and Mark Yatskar. Researchqa: Evaluating scholarly question answering at scale across 75 fields with survey-mined questions and rubrics. CoRR, abs/2509.00496, 2025. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? CoRR, abs/2504.13837, 2025. Ming Zhang, Yujiong Shen, Zelin Li, Huayu Sha, Binze Hu, Yuhui Wang, Chenhao Huang, Shichun Liu, Jingqi Tong, Changhao Jiang, Mingxu Chai, Zhiheng Xi, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Llmeval-med: A real-world clinical benchmark for medical llms with physician validation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of the Association for Computational Linguistics: EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 4888–4914. Association for Computational Linguistics, 2025. Wenjian Zhang, Kongcheng Zhang, Jiaxin Qi, Baisheng Lai, and Jianqiang Huang. Experience is the best teacher: Motivating effective exploration in reinforcement learning for llms. CoRR, abs/2603.20046, 2026. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self- distilled reasoner: On-policy self-distillation for large language models. CoRR, abs/2601.18734, 2026. Yang Zhou, Sunzhu Li, Shunyu Liu, Wenkai Fang, Jiale Zhao, Jingwen Yang, Jianwei Lv, Kongcheng Zhang, Yihe Zhou, Hengtong Lu, Wei Chen, Yan Xie, and Mingli Song. Breaking the explo- ration bottleneck: Rubric-scaffolded reinforcement learning for general LLM reasoning. CoRR, abs/2508.16949, 2025. Siqi Zhu, Xuyan Ye, Hongyu Lu, Weiye Shi, and Ge Liu. The many faces of on-policy distillation: Pitfalls, mechanisms, and fixes. CoRR, abs/2605.11182, 2026. A Supplementary Experiments A.1 OOD Generalization on Instruction Following. Table 3: Supplementary OOD evaluation. We report instruction-following performance on IFE- val, IFBench, and MulDimIF. CriPO-RM and CriPO-RS are trained on RaR-Medicine and RaR-Science, respectively. ModelIFEvalIFBench MulDimIF Qwen3-1.7B69.718.714.9 + CriPO-RM69.5↓0.219.0↑0.314.8↓0.1 + CriPO-RS69.9↑0.220.0↑1.314.9↑0.0 Qwen3-4B81.127.317.6 + CriPO-RM80.6↓0.626.3↓1.017.8↑0.2 + CriPO-RS80.4↓0.728.3↑1.017.2↓0.4 To evaluate whether CriPO preserves general instruction-following ability beyond the rubric- based training domain, we conduct out-of-domain evaluation on IFEval, IFBench, and MulDimIF. As shown in Table 3, CriPO largely maintains the OOD instruction-following performance of the base models. On Qwen3-1.7B, CriPO-RS slightly improves IFEval from0.6968to0.6987and IF- Bench from0.1866to0.2000, while matching the base model on MulDimIF. On Qwen3-4B, CriPO- RS improves IFBench from0.2732to0.2833, and CriPO-RM achieves the best MulDimIF score. Al- though some variants show small fluctuations on individual benchmarks, the overall results suggest that criteria-distilled policy optimization does not 14 Table 4: Experimental results (%) across medicine and science benchmarks using Qwen3-32B as judge. The best results are marked in bold, and the second-best areunderlined. Arrows indicate changes over the base model. All results are evaluated on models trained after 200 steps. Method MedicineScience Avg. RaR-MedicineHealthBenchLLMEval-MedRaR-ScienceResearchQA Qwen3-1.7B34.0↑ 0.051.6↑ 0.023.7↑ 0.044.0↑ 0.051.3↑ 0.040.9↑ 0.0 + GRPO 35.7↑ 1.752.8↑ 1.223.4↓ 0.350.7↑ 6.755.8↑ 4.543.7↑ 2.8 + HeRL 41.4↑ 7.453.5↑ 1.925.2↑ 1.549.7↑ 5.758.0↑ 6.745.6↑ 4.7 + OPSD18.6↓ 15.428.3↓ 23.311.4↓ 12.321.2↓ 22.829.3↓ 22.021.7↓ 19.2 + CriPO-U38.6↑ 4.655.1↑ 3.526.4↑ 2.753.3↑ 9.360.6↑ 9.346.8↑ 5.9 + CriPO-S 36.8↑ 2.853.7↑ 2.125.9↑ 2.251.5↑ 7.555.4↑ 4.144.7↑ 3.8 + CriPO40.4↑ 6.456.2↑ 4.626.5↑ 2.852.4↑ 8.459.9↑ 8.647.1↑ 6.2 Qwen3-4B50.2↑ 0.059.2↑ 0.038.1↑ 0.063.7↑ 0.058.3↑ 0.053.9↑ 0.0 + GRPO57.0↑ 6.863.3↑ 4.138.5↑ 0.471.6↑ 7.966.7↑ 8.459.4↑ 5.5 + HeRL 54.6↑ 4.462.6↑ 3.439.1↑ 1.064.7↑ 1.066.4↑ 8.157.5↑ 3.6 + OPSD33.6↓ 16.634.2↓ 25.019.5↓ 18.625.1↓ 38.639.7↓ 18.630.4↓ 23.5 + CriPO-U57.8↑ 7.663.3↑ 4.139.6↑ 1.572.0↑ 8.368.2↑ 9.960.2↑ 6.3 + CriPO-S 59.6↑ 9.464.6↑ 5.441.7↑ 3.671.1↑ 7.465.8↑ 7.560.6↑ 6.7 + CriPO60.8↑ 10.664.1↑ 4.942.7↑ 4.671.9↑ 8.267.8↑ 9.561.5↑ 7.6 substantially degrade OOD instruction-following ability, indicating that the targeted corrections mainly improve rubric-related behaviors without causing broad capability collapse. A.2 Robustness to Different Judges To examine whether our conclusions depend on a specific evaluator, we conduct additional evaluations using Qwen3-32B as the judge. As shown in Table 4, the overall trends remain consistent with the main results. Compared with GRPO, the full CriPO improves the average score from43.7to47.1 (+3.4) on Qwen3-1.7B and from59.4to61.5(+2.1) on Qwen3-4B. CriPO also achieves stronger average performance than HeRL on both model scales, indicating that its gains are not limited to a single evaluator. Although the best-performing variant may vary across individual benchmarks, the full CriPO consistently achieves the strongest average performance. These results suggest that criteria-distilled policy optimization provides robust improvements under different judge models. A.3 Case Study on Medical Reasoning Figure 8 presents a representative example comparing GRPO and CriPO on a medical reasoning question. The case describes a 7-day-old premature infant with grossly bloody stool, abdominal distention, and increasing oxygen requirements, where the correct initial diagnostic step is abdominal series for suspected necrotizing enterocolitis (NEC). GRPO selects fiberoptic endoscopy as the final answer. Although it briefly mentions abdominal X-ray, it incorrectly frames it as insufficient for evaluating bleeding and fails to connect the patient’s prematurity, abdominal distention, bloody stool, and respiratory deterioration to suspected NEC. As a result, GRPO satisfies only the criterion related to recognizing NEC-related symptoms, but misses the criteria requiring the correct diagnostic choice, final answer, imaging rationale, and non-invasive-first reasoning. In contrast, CriPO performs more explicit and structured reasoning. It first identifies the infant as highly premature and connects the clinical signs to NEC risk. It then explains why abdominal series is the appropriate first-line diagnostic step, noting that it can reveal pneumatosis intestinalis or portal venous gas and is fast and non-invasive. The response also contrasts fiberoptic endoscopy with abdominal X-ray, explaining why endoscopy is not typically used as the first-line diagnostic test in a premature neonate. This more complete reasoning leads CriPO to satisfy all rubric criteria and produce the correct final answer. This example suggests that CriPO encourages the model to reason more deeply and explore rubric-relevant aspects more sufficiently before reaching the final answer. 15 The correct answer is: B. Fiberoptic endoscopy # Explanation: In a 7-day-old premature infant with gestational age 26 weeks, presenting with grossly bloody stool, abdominal distention, and increasing oxygen requirements, the most appropriate initial diagnostic step for suspected gastrointestinal hemorrhage is fiberoptic endoscopy. Rationale: Grossly bloody stool in a neonate is a red flag for gastrointestinal bleeding. In premature infants, common causes of gastrointestinal bleeding include esophageal varices, gastrointestinal tract malformations, intestinal ischemia, gastrointestinal infections such as necrotizing enterocolitis, and hemorrhagic disease of the newborn. Fiberoptic endoscopy is the gold standard for evaluating gastrointestinal bleeding in neonates, especially in the setting of gross blood in stool. It allows direct visualization of the gastrointestinal tract, identification of the source of bleeding, and can be therapeutic, such as cauterization of bleeding vessels. Abdominal X-ray may show signs of intestinal obstruction or pneumatosis intestinalis, but it is not diagnostic for bleeding. Apt test is a screening test for fecal occult blood, but it is not specific for gastrointestinal bleeding in neonates and is not the initial step. Routine stool culture is useful for identifying infection but is not the first step in evaluating bleeding. # Conclusion: B. Fiberoptic endoscopy is the best initial diagnostic step in this case. It allows direct visualization and diagnosis of the source of bleeding in a neonate with suspected gastrointestinal hemorrhage. Let’s analyze the scenario and determine the best initial diagnostic step for a 7-day-old premature infant born at 26 weeks gestation who presents with grossly bloody stool, abdominal distention, and increasing oxygen requirements. # Key points to consider: The infant is very preterm, which places them at high risk for gastrointestinal problems, especially necrotizing enterocolitis (NEC). Grossly bloody stool is a red flag for gastrointestinal bleeding. In a premature neonate, NEC is one of the most important and serious causes to consider. Abdominal distention may reflect ileus, inflammation, bowel wall injury, or perforation, all of which can occur in NEC. Increasing oxygen requirements suggest clinical deterioration and may indicate systemic illness, sepsis, or shock. # Initial diagnostic step: A. Abdominal series: An abdominal series is used to evaluate for intestinal obstruction, pneumoperitoneum, bowel distention, portal venous gas, ileus, or pneumatosis intestinalis. In suspected NEC, abdominal X-ray may show pneumatosis intestinalis or portal venous gas. In a preterm infant, this is a fast and non- invasive initial diagnostic step. This is the best initial step to assess for NEC or perforation. B. Fiberoptic endoscopy: Fiberoptic endoscopy allows direct visualization of the gastrointestinal tract, but it is not typically used as the first-line diagnostic test in a premature neonate. It carries risks related to trauma, anesthesia, and technical difficulty. This is not the best initial step. C. Apt test: The Apt test can help distinguish fetal from maternal blood, but it does not identify the cause or severity of bleeding in this clinical setting. This is not the best initial step. D. Routine stool culture: A stool culture may help evaluate infection, but it is not the first step in an acutely ill premature infant with suspected NEC. This is not the best initial step. Best initial diagnostic step: A. Abdominal series Rationale: An abdominal series is non-invasive, fast, and can detect life-threatening findings such as NEC or perforation. In a premature infant with bloody stool, abdominal distention, and increasing oxygen requirements, abdominal X-ray is the first-line imaging test. Final Answer: A. Abdominal series. Response of GRPO Response of CriPO Question: A 7-day-old premature infant born at 26 weeks of gestation now has a grossly bloody stool, abdominal distention, and increasing oxygen requirements. choose the best initial diagnostic step in the evaluation of the patient's apparent GI hemorrhage. A. Abdominal series B. Fiberoptic endoscopy C. Apt test D. Routine stool culture Rubric Criteria Status (GRPO / CriPO) •C1: Identifies an abdominal series (X-ray) as the best initial diagnostic step. ( ) •C2: Clearly states that the final answer is A. Abdominal series. ( ) •C3: Recognizes suspected necrotizing enterocolitis (NEC) in a premature infant with bloody stool and abdominal distention. ( ) •C4: Explains that abdominal X-ray can reveal NEC findings such as pneumatosis intestinalis or portal venous gas. ( ) •C5: Indicates that abdominal series is a fast, non-invasive initial test and avoids invasive procedures such as fiberoptic endoscopy. ( ) Figure 9: Case study on medical reasoning. CriPO produces a more complete reasoning process than GRPO by exploring rubric-relevant clinical cues and diagnostic rationales, suggesting that its longer responses reflect deeper reasoning rather than superficial length expansion. B Detailed Experimental Settings Dataset Details.For RaR-Medicine and RaR-Science [Gunjal et al., 2025], we first filter out overly easy samples to improve data efficiency. Specifically, we remove samples on which Qwen3-4B achieves a reward higher than 0.9. This results in 15,658 training samples and 1,936 testing samples for RaR-Medicine, and 10,874 training samples and 1,365 testing samples for RaR-Science. For HealthBench [Arora et al., 2025] and ResearchQA [Yifei et al., 2025], we randomly select a subset of 500 samples from each dataset for cross-domain evaluation. For LLMEval-Med, we follow the evaluation protocol of the original project [Zhang et al., 2025], which contains 667 medical questions across five categories. Implementation Details All experiments are conducted on8×A800-40G GPUs, with training implemented based on the verl [Sheng et al., 2025] framework. During training, we use Qwen3-32B as the rubric judge, which is deployed on 64×910B2 NPUs, with the prompt shown in Figure 11. Detailed training configurations of CriPO and all baselines are provided in Table 5. The generation parameters for evaluation are set to rollout_n= 1, top_k=−1, top_p= 0.8, and temperature= 0.7 across all benchmarks. 16 Table 5: Training configurations across different methods and model backbones. SettingsHyperparameters Samplingrollout_n = 8, top_k =−1, top_p = 1.0, temperature = 1.0 max_prompt_length = 1024, max_response_length = 4096 Trainingppo_mini_batch_size = 32, ppo_micro_batch_size_per_gpu = 2 learning_rate = 1e−6, kl_loss_coef = 1e−3 train_batch_size = 64, total_training_steps = 200 Optimizationsparam_offload, flash_attn, bf16 HeRLThe hyper-parameters in HeRL follows their default settings. OPSDrollout_n= 1, top_k=−1, top_p= 0.8, temperature= 0.7 loss type: forward KL, privileged information: full rubric CriPOγ = 0.95, α = 0.1, τ flip = 0.1 β = 0.05/0.03 for Qwen3-1.7B/4B, with 50 step linear warmup Moreover, to ensure the stability of the algorithm, CriPO incorporates several additional implementa- tion details. First, when constructing the teacher prompt, if the number of unexplored or suppressed criteria exceeds a predefined thresholdK = 3, we only select the top-Kcriteria with the largest criterion weights to construct the teacher prompt. Second, after computing the forward KL for each token, we first clamp it with a maximum value of 10 before applying token filtering and gradient updates, so as to mitigate the influence of outliers. Third, we observe that tokens at the beginning and end of a response usually exhibit larger KL values, as shown in Figure 10. We conjecture that the beginning tokens may be affected by the instruction in the teacher prompt; for example, the teacher model may assign more probability mass to phrases such as “Now I will revise the response.” For the ending tokens, the teacher probability distribution is computed conditioned on a suboptimal student rollout prefix, where the distribution may be more inclined to perform post-hoc correction rather than provide reliable supervision. To improve training stability, we therefore mask the first and last 1% of response tokens before contribution-guided token filtering. 020406080 Relative Token Position (%) 0.00 0.25 0.50 0.75 1.00 masked first 1% masked last 1% Mean Forward KL by Position Figure 10: Token-wise forward KL across positions. Forward KL is higher near response bound- aries, so we mask the first and last 1% of tokens to reduce boundary noise during localized OPSD. 17 Given the following user query: user_query, Below is your previous response to the query: previous_response, This response failed to meet the following criteria: blind_criteria, Please revise the previous response with minimum necessary to satisfy the unmet criteria. Output only the revised answer. Given a user prompt, a generated response, and a list of quality rubrics, please evaluate the response against EACH rubric. For each rubric, - Mark "PRESENT" if the criterion is satisfied, or "NOT_PRESENT" if it is not. For example, given the response "Apples are red", the rubric "Mentions apples" is PRESENT, "Does not mention strawberries" is also PRESENT since the response doesn't mention strawberries and "Mentions oranges" is NOT_PRESENT. Also, "Avoids mentioning strawberries" is PRESENT because the response doesn't mention strawberries. However, "Avoids mentioning apples" is NOT_PRESENT because the response mentions apples. - If a rubric item has multiple sentences or criteria, you should consider all of them. If any of the criteria is not met, the answer should be NOT PRESENT. Only return PRESENT if all of the criteria are met. - One important exception to the above bullet point is that if a rubric says "such as", "for example", or "including", the response does not have to include all of the examples listed to meet the criteria. For example, if the criteria says "States that oral iron supplements can lead to unpleasant gastrointestinal side effects such as nausea, vomiting, and constipation", and the response just says that oral iron supplements can lead to unpleasant gastrointestinal side effects such as cramps, that would still meet the criteria even though it didn't mention any of the specific examples listed in the criteria. That is, there are no partial credit for any of the criteria. Start your response with a valid JSON object that starts with "```json" and ends with "```". The keys must be the numbers of the rubrics provided and the values must be either "PRESENT" or "NOT_PRESENT" based on your evaluation. Ensure the JSON is valid and contains no extra text or explanations. Example response: ```json "1": "PRESENT", "2": "NOT_PRESENT", "3": "PRESENT" ``` <Prompt> prompt_str </Prompt> <Response> response </Response> <Rubrics> rubrics_str </Rubrics> Figure 11: Rubric judge prompt. The judge evaluates model responses according to predefined criteria and produces criterion-wise scores for reward computation. 18