Paper deep dive
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Kun Liang, Chenming Tang, Clive Bai, Weijie Liu, Zeyuan Liu, Qingyang Zhang, Saiyong Yang, Yunfang Wu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy's future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose $\pi$PPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, $\pi$PPO helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that $\pi$PPO consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics.
Tags
Links
- Source: https://arxiv.org/abs/2609.37825v1
- Canonical: https://arxiv.org/abs/2609.37825v1
Trouble viewing inline? Open PDF directly →
Full Text
61,265 characters extracted from source content.
Expand or collapse full text
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR Kun Liang Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Email: 10.512wuyf@pku.edu.cn Chenming Tang Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University Clive Bai Affiliation: Foundation Model Department, Tencent ✉ kliang25@stu.pku.edu.cn Weijie Liu Affiliation: Foundation Model Department, Tencent ✉ kliang25@stu.pku.edu.cn Zeyuan Liu Affiliation: Foundation Model Department, Tencent ✉ kliang25@stu.pku.edu.cn Qingyang Zhang Affiliation: Foundation Model Department, Tencent ✉ kliang25@stu.pku.edu.cn Saiyong Yang Affiliation: Foundation Model Department, Tencent ✉ kliang25@stu.pku.edu.cn Yunfang Wu Affiliation: National Key Laboratory for Multimedia Information Processing, Peking University School of Computer Science Peking University Abstract Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-critic methods such as PPO address this by learning value functions to construct token-level advantages. Their effectiveness, however, hinges on reliable value estimation, a difficult task requiring the critic to both assess progress toward a correct solution and anticipate an evolving policy’s future behavior; errors in either can compromise credit assignment and destabilize online training. In this paper, we revisit the standard state-only formulation of value estimation and propose πPPO, a self-privileged actor-critic framework. By reusing verified same-prompt rollouts as contrastive evidence, π helps the critic assess intermediate reasoning against successful and failed attempts, while preserving standard policy optimization and the deployment interface. Experiments show that π consistently improves value-estimation quality by a substantial margin and outperforms representative actor-critic and critic-free RLVR baselines on challenging mathematical reasoning benchmarks, while remaining effective even when paired with substantially smaller asymmetric critics. †footnotetext: ∗ Work done during internship at Tencent.†footnotetext: † Corresponding Authors. 1 Introduction On-policy feedback-based learning methods stand out in LLM post-training for their effectiveness, especially on incentivizing long-form reasoning capabilities on verifiable tasks, providing a promising way for scaling (Guo et al., 2025a; Team et al., 2025; Yang et al., 2025). Despite the empirical success of recent value-free methods, explicit value estimation remains a fundamental optimization paradigm in reinforcement learning. Beyond reducing variance, it provides a state-conditioned baseline, enabling principled credit assignment and readily accommodates diverse rollout structures (Williams, 1992; Greensmith et al., 2004; Schulman et al., 2015). Therefore, advancing value estimation offers a promising path toward improving policy optimization in LLM post-training. However, accurately estimating the value of a partial response remains inherently challenging: the value network must both understand the space of correct solutions—the very task the policy itself is learning—and predict the policy’s future behavior (Shan et al., 2026; Zhang et al., 2026). This difficulty is further exacerbated in long-form reasoning, where rewards are sparse and often observable only at the ends of exceptionally long trajectories, leaving a vast space of intermediate states with learning targets that are both weakly informative and highly uncertain. In actor–critic training, inaccurate value estimates can distort token-level advantages and misguide policy updates, undermining training stability and final performance (Schulman et al., 2015; Yuan et al., 2025). Existing approaches improve critic learning through better initialization and return estimation (Yuan et al., 2025; Yue et al., 2025; Hou et al., 2026), or obtain more informative value targets through additional continuation sampling and intermediate-state modeling (Kazemnejad et al., 2024; Guo et al., 2025b; Chen et al., 2026b). Nevertheless, whether by strengthening the critic or enriching its targets, these solutions inherit the same largely unquestioned premise: both demands placed on the critic must be met by the critic itself, using only the information available from its prescribed vantage point; as we show later, however, this need not be taken as given. To this end, we propose Privileged-Information PPO ( πPPO), which turns verified trajectories into online references for state-level value prediction. We adapt PPO to sample groups of rollouts for each prompt, giving each trajectory a dual role: a sample for policy learning and a potential reference for evaluating partial responses in other trajectories. For each target rollout, we construct a leave-target-out critic context from its verified siblings, preserving a valid baseline while leaving the policy’s conditioning unchanged. These references evolve with the policy and require no additional generation beyond the rollout groups used for policy updates. As shown in Section 4, π consistently improves value-estimation quality across two model backbones on mathematical reasoning tasks, with corresponding gains in end-to-end policy performance over representative RL baselines. This design also enables π to use substantially smaller critics while outperforming actor–critic baselines with actor-sized value models. Our contributions are as follows: • Conceptual. We recast value estimation in RLVR as an information problem characterizing inherent properties of value prediction under binary verifiable rewards that highlight potential limitations of the conventional state-only formulation. • Methodological. We propose π , which supplies online evidence on the critic side, conditioning each value estimation on a leave-target-out context of verified same-prompt rollouts already collected for policy updates, thus preserving a valid baseline while leaving policy inference unchanged, all without additional rollout generation. • Empirical. We show that π consistently improves value-estimation quality and outperforms representative actor–critic and critic-free RLVR baselines on mathematical reasoning benchmarks. It remains effective even with substantially smaller asymmetric critics. 2 Preliminary Notation. In this paper, we define an LLM parameterized by θ as a policy πθ _θ. Let x denote a query and D the set of queries. Given a response y to the query x, its likelihood under the policy πθ _θ is expressed as πθ(y∣x)=∏t=1|y|πθ(yt∣x,y<t) _θ(y x)= _t=1^|y| _θ(y_t x,y_<t), where |y||y| denotes the number of tokens in y. A query-response pair (x,y)(x,y) is scored by a rule-based outcome reward r(x,y)∈0,1r(x,y)∈\0,1\, indicating whether the response y aligns with the ground truth of x. Proximal Policy Optimization (PPO). PPO (Schulman et al., 2017) follows the actor–critic framework, jointly learning a policy model (actor) πθ _θ and a value model (critic) VϕV_φ parameterized by ϕφ, updating the policy using the following clipped policy surrogate (we omit the KL regularization term hereinafter for brevity): PPO(θ)=x∼,y∼πθold(⋅|x)[1|y|∑t=1|y|min(ρt(θ)A^t,clip(ρt(θ),1−ε,1+ε)A^t)], _PPO(θ)=E_x ,\,y _ _old(·|x) [ 1|y| _t=1^|y| ( _t(θ) A_t,\,clip ( _t(θ),1- ,1+ ) A_t ) ], (1) where ρt(θ)=πθ(yt∣x,y<t)πθold(yt∣x,y<t) _t(θ)= _θ(y_t x,y_<t) _ _old(y_t x,y_<t) denotes the token-level probability ratio at step t, and ε is the clipping coefficient. The learned critic enters the policy update through the advantage A^t A_t, which is commonly constructed using Generalized Advantage Estimation (GAE) (Schulman et al., 2015). Since the critic scores every state st=(x,y<t)s_t=(x,y_<t) individually, the objective thereby enables dense, token-level credit assignment even when the environment returns a single terminal reward. Group Relative Policy Optimization (GRPO). GRPO (Shao et al., 2024) bypasses the value model by computing the relative advantage of each response within a group of G responses to the same query. Given yii=1G∼πθold(⋅∣x)\y_i\_i=1^G _ _old(· x), it averages the clipped surrogate over the group, using A^i,t=A^i=r(x,yi)−mean(r(x,yj)j=1G)std(r(x,yj)j=1G). A_i,t= A_i= r(x,y_i)-mean (\r(x,y_j)\_j=1^G )std (\r(x,y_j)\_j=1^G ). (2) All tokens in yiy_i share the same advantage A^i A_i. Privileged Information (PI). Learning using privileged information augments a standard input x∈x with task-relevant auxiliary information x⋆∈⋆x available during training but absent from the standard inference input (Vapnik & Vashist, 2009). For LLMs, both the query and privileged information can be represented as token sequences over the model vocabulary V, i.e., x,ℐ∈∗x,I , and incorporated by conditioning the policy as πθ(⋅∣x,ℐ) _θ(· x,I), where ℐI may encode hints, verified answers, or reference rationales (Qu et al., 2026; Zhao et al., 2026). 3 Methodology In this section, we introduce a new perspective on incorporating privileged information into policy optimization: enriching the critic’s information while preserving the policy’s original objective of maximizing environment-defined rewards. We instantiate this perspective using information available within the training process, without additional generation or external expert knowledge. 3.1 Interpreting a state: the “missing” context in value estimation Standard critics fit a single value predictor across states, predicting each state’s expected return from its representation. We view this shared-prediction setup through the lens of fully amortized optimization (Shu, 2017; Amos, 2025), in which a generic model serves all prediction instances without instance-specific adaptation or auxiliary evidence. We first examine the resulting limitation of this formulation and then characterize the underlying information structure for value estimation under binary verifiable rewards. Proposition 1 (Privileged conditioning for amortized value estimation). Fix a policy π with undiscounted terminal return R∈0,1R∈\0,1\, and define Vπ(s)=[R∣s]V^π(s)=E[R s] and Vπ,+(s,ℐ)=[R∣s,ℐ]V^π,+(s,I)=E[R s,I]. Assume finite squared error and that contextual critics Vϕ+(s,ℐ)V_φ^+(s,I) can represent every state-only critic Vϕ(s)V_φ(s). Under a common joint distribution of (s,ℐ,R)(s,I,R), infϕ[(R−Vϕ+(s,ℐ))2]⏟contextual prediction risk _φE[(R-V_φ^+(s,I))^2]_contextual prediction risk =s,ℐ[Var(R∣s,ℐ)]+infϕ[(Vϕ+(s,ℐ)−Vπ,+(s,ℐ))2] =E_s,I[Var(R s,I)]+ _φE[(V_φ^+(s,I)-V^π,+(s,I))^2] ≤s[Var(R∣s)]+infϕ[(Vϕ(s)−Vπ(s))2]=infϕ[(R−Vϕ(s))2]⏟state-only prediction risk. _s[Var(R s)]+ _φE[(V_φ(s)-V^π(s))^2]= _φE[(R-V_φ(s))^2]_state-only prediction risk. Thus, the residual error is not solely an optimization artifact: part of it reflects return uncertainty unresolved by the target state alone. We interpret this gap as informational: a state-conditioned predictor captures only the return variation predictable from s, while additional evidence ℐI can reduce the Bayes error insofar as it explains residual variation beyond s. Under binary verifiable rewards, this return uncertainty concerns whether a trajectory will succeed or fail. We next examine value prediction through this outcome distinction, relating prediction error to the contrast between the critic’s predictions on successful and failed trajectories. Proposition 2 (Success–failure contrast in value estimation). Fix a policy π with undiscounted terminal return R∈0,1R∈\0,1\ and Var(R)>0Var(R)>0. All expectations are taken under the common joint distribution of (s,R)(s,R). For any square-integrable Vϕ(s)V_φ(s), define Δϕ=[Vϕ(s)∣R=1]−[Vϕ(s)∣R=0]. _φ=E[V_φ(s) R=1]-E[V_φ(s) R=0]. Its prediction risk satisfies [(R−Vϕ(s))2]≥Var(R)(1−Δϕ)2⏟success–failure contrast error.E[(R-V_φ(s))^2]≥ Var(R)(1- _φ)^2_success--failure contrast error. Consequently, any critic outperforming the optimal constant predictor V0=[R]V_0=E[R] (with risk Var(R)Var(R)) must exhibit positive success–failure contrast, i.e., Δϕ>0 _φ>0. The bound specifies a necessary condition on the contrast between the critic’s predictions on successful and failed trajectories for achieving a given level of prediction accuracy. Taken together, we therefore rethink the evidence also as contrastive: by providing positive and negative references for the outcome distinction that the critic must infer, we reduce the uncertainty left by state-only value estimation insofar as the evidence clarifies the target prefix’s relative support under the successful and failed trajectory processes. 3.2 πPPO Figure 1: Overview of π . For each prompt, the policy generates a group of rollouts, and the verifier assigns rewards according to their outcomes. The labeled rollouts are then reused to construct privileged context ℐ(i)I^(i) for each target rollout, excluding the target itself. The privileged value model uses this context to estimate values along the target rollout. These estimates, together with the verifier rewards, are used to compute advantages for the PPO policy update. Motivated by the analysis above, we propose π (Figure 1), a critic-side extension of PPO that reuses verified same-prompt rollouts as contrastive evidence for value estimation. By constructing a leave-target-out context pool via an adaptation of PPO to group rollouts, we enable each trajectory to serve both as a training target to be evaluated and as a potential labeled reference for its siblings, allowing the critic to assess a prefix in relation to successful and failed attempts at the same problem. Leave-target-out contrastive context. For a prompt x, the old policy samples G rollouts (j)∼πθold(⋅∣) y^(j) _ _old(· x), which the verifier labels. The context of a target (i) y^(i) is drawn from the remaining rollouts, partitioned by label, iσ=(j):j≠i,r(,(j))=σ=+,σ∈+,−.C_i^σ= \ y^(j):j≠ i,\;r( x, y^(j))=1_\σ=+\ \, σ∈\+,-\. (3) To use trajectories as evidence for prefix-level prediction, we separate the target being evaluated from the trajectories supplying its context. Let k()S_k(C) denote k reference rollouts randomly sampled from C, together with their verifier labels. Writing ∥\| for concatenation and ⋆ y for the supplied ground-truth answer to x, we construct ℐ(i)=1(i+)|1(i−),i+≠∅,i−≠∅,2(i+),i−=∅,2(i−)|[⋆],i+=∅.I^(i)= casesS_1(C_i^+)\|S_1(C_i^-),&C_i^+≠ ,\ C_i^-≠ ,\\ S_2(C_i^+),&C_i^-= ,\\ S_2(C_i^-)\|[ y ],&C_i^+= . cases (4) When only one outcome is represented among the remaining rollouts, we use two siblings of that outcome; the ground-truth answer is additionally supplied only in the all-negative case and is identified separately from the sampled reference attempts. Privileged advantage estimation. By constructing the self-elicited privileged information above, we obtain a critic that no longer predicts from the prefix alone but performs a contextual forward pass Vϕ+(,ℐ(i),<t(i))V_φ^+( x,I^(i), y_<t^(i)), while the actor stays conditioned on (,<t(i))( x, y_<t^(i)) alone. These context-conditioned values enter the standard TD residuals and GAE: δt+(i) _t^+(i) =rt(i)+γVϕ+(st+1(i),ℐ(i))−Vϕ+(st(i),ℐ(i)), =r_t^(i)+γ V_φ^+(s_t+1^(i), [rgb]0,0,0.65I^(i))-V_φ^+(s_t^(i), [rgb]0,0,0.65I^(i)), (5) A^t+(i) A_t^+(i) =∑ℓ=0Ti−t(γλ)ℓδt+ℓ+(i). = _ =0^T_i-t(γλ) _t+ ^+(i). Here Ti=|(i)|T_i=| y^(i)|, and the terminal value is set to zero. The actor is then updated by substituting A^t+(i) A_t^+(i) for the advantage in the PPO clipped objective, while retaining its original conditioning. Notably, the π design yields two favorable properties: • Valid policy gradient baseline under privileged value estimation. Let st=(,<t)s_t=( x, y_<t), and let the privileged information ℐI be available only to the critic. Because Vϕ+(st,ℐ)V_φ^+(s_t,I) is independent of the current action ata_t, it remains a valid baseline: at∼πθ(⋅∣st)[∇θlogπθ(at∣st)Vϕ+(st,ℐ)]=Vϕ+(st,ℐ)∇θ∑atπθ(at∣st)=0.E_a_t _θ(· s_t) [ _θ _θ(a_t s_t)V_φ^+(s_t,I) ]=V_φ^+(s_t,I) _θ _a_t _θ(a_t s_t)=0. (6) Thus, privileged value estimation introduces no additional bias into the expected actor gradient and leaves the underlying policy objective unchanged. • Training–inference policy consistency. The privileged context enters only the critic, so the actor uses the same conditioning at both training and inference: πθtrain(at∣st)=πθinfer(at∣st)=πθ(at∣st),st=(,<t). _θ^train(a_t s_t)= _θ^infer(a_t s_t)= _θ(a_t s_t), s_t=( x, y_<t). (7) Overall, π incorporates verified contrastive rollout evidence into policy optimization through context-conditioned value estimates and resulting advantages. Since the rollout group collected for each PPO update supplies the critic context, the evidence is refreshed on the fly as the policy evolves. 4 Experiments 4.1 Experimental Settings In this section, we provide a brief overview of the key experimental setup, including training procedures, baselines, and evaluation details. Additional information can be found in Appendix B. Training details. We primarily conduct experiments on Qwen3-4B and Qwen3-8B (Yang et al., 2025) with thinking mode off, using DAPO-17K (Yu et al., 2026). We build on the VeRL codebase (Sheng et al., 2024) and follow the standard PPO recipe with clip-higher enabled. All methods use the same hyperparameters (batchsize 256, actor learning rate 1e-6) and rollout settings (temperature 1.0, top-p 1.0, top-k disabled, 8 rollouts per prompt). Following Yuan et al. (2025), we adopted 50 steps pretrain for the value model warmup as adjustment for RLVR tasks for all critic-based methods, further implementation details can be found in the Appendix B.1. Baselines. We compare π against representative RL baselines. We use vanilla PPO (Schulman et al., 2017) as the standard actor–critic baseline and include VAPO (Yue et al., 2025) as an RLVR-adapted improvement over PPO. We further compare against GRPO (Shao et al., 2024) and DAPO (Yu et al., 2026), a widely adopted GRPO variant with competitive performance and robust training on RLVR tasks. Together, these experiments assess the benefit of augmenting the PPO critic with verified sibling context and compare π with RLVR-adapted actor–critic and strong value-free baselines on reasoning tasks. We additionally examine the training behavior of the self-distillation family in Appendix D.2. Benchmarks and evaluation metrics. We evaluate our method on challenging in-domain math benchmarks: AIME 2024 (AI-MO, 2024), AIME 2025 (OpenCompass, 2025), BeyondAIME (ByteDance-Seed, 2025), HMMT25 (February) (Balunović et al., 2025), and HMMT25 (November) (Balunović et al., 2025). Following Guo et al. (2025a), we generate 8–16 outputs per problem, depending on test-set size, and report Pass@1Pass@1 accuracy. All methods are evaluated using identical prompts and decoding configurations; details are provided in Appendix B.2. We further assess out-of-distribution generalization, with results reported in Appendix D.1. For assessing the critic, we mainly focus on explained variance (EV) and value-function loss. Let G^i,t G_i,t and V^i,t V_i,t denote the value-regression target and critic prediction at token position t in rollout i, respectively. Over the valid response-token positions ℳM, we compute EV=1−Var(i,t)∈ℳ(G^i,t−V^i,t)Var(i,t)∈ℳ(G^i,t)+ϵ,ℒV=1|ℳ|∑(i,t)∈ℳ(G^i,t−V^i,t)2.EV=1- Var_(i,t) ( G_i,t- V_i,t )Var_(i,t) ( G_i,t )+ε, _V= 1|M| _(i,t) ( G_i,t- V_i,t )^2. (8) Together, EV and ℒVL_V provide complementary views of across-sample target-variance capture and pointwise regression-target fitting, respectively. 4.2 Main Results Table 1: Mathematical reasoning accuracy (%). “Avg@k” denotes mean accuracy over k random generations (i.e., pass@1); Overall is the mean over the five benchmarks. Best result within each model size is in bold. Method Mathematical Reasoning Tokens AIME 24 Avg@16 AIME 25 Avg@16 Beyond AIME Avg@8 HMMTFeb Avg@16 HMMTNov Avg@16 Overall Avg. Gen./ Train 10910^9 (a) Qwen3-4B Initial policy 21.7 18.9 10.8 12.3 7.3 14.2 – Critic-free GRPO 60.4 54.8 34.5 34.6 41.5 45.2 2.33 / 2.38 DAPO 63.1 56.7 34.5 34.6 41.9 46.2 4.42 / 2.64 Actor–critic PPO 57.9 54.2 32.4 32.7 40.4 43.5 2.42 / 2.48 VAPO 63.3 52.1 33.3 33.8 43.9 45.3 2.07 / 2.14 πPPO (ours) 66.7 61.5 39.0 37.5 46.9 50.3 2.53 / 2.59 (b) Qwen3-8B Initial policy 25.0 20.2 13.1 11.5 10.0 16.0 – Critic-free GRPO 69.4 58.3 36.1 32.5 45.2 48.3 2.25 / 2.30 DAPO 71.3 57.1 37.8 33.3 46.9 49.3 4.34 / 2.59 Actor–critic PPO 68.1 55.2 36.3 32.3 47.7 47.9 2.35 / 2.42 VAPO 67.3 55.8 34.3 33.5 47.1 47.6 1.95 / 2.02 πPPO (ours) 73.5 61.5 40.0 34.4 48.5 51.6 2.57 / 2.63 Figure 2: Critic explained variance (EV; a,b) and value-function loss (c,d) during training. Across Qwen3-4B and Qwen3-8B, π consistently achieves higher EV and lower value-function loss than PPO and VAPO, with improvements evident from the early training steps. ❶π consistently improves end-to-end policy performance. As shown in Table 1, π achieves the highest accuracy on all five mathematical reasoning benchmarks at both model scales. Its overall scores reach 50.3% on Qwen3-4B and 51.6% on Qwen3-8B, exceeding the strongest baseline, DAPO, by 4.1 and 2.3 percentage points, respectively. Compared with vanilla PPO, the gains are 6.8 and 3.7 points. These results show that privileged value estimation translates into consistent policy gains over both actor–critic and critic-free baselines. ❷Privileged context substantially improves value estimation. Figure 2 compares the critic explained variance and value-function loss throughout training. With either backbone model, the two metrics reveal a consistent pattern: π starts with a higher explained variance upon entering training, improves sharply during the early updates, and subsequently remains at a high level, while its value-function loss quickly decreases and stays consistently lower. In contrast, the state-only critics of PPO and VAPO improve more gradually and plateau at weaker levels, providing less informative value estimates throughout training. 4.3 π admits smaller asymmetric critics Another commitment of standard actor–critic implementations is to instantiate the actor and critic with the same model backbone. We next ask whether the value estimation advantage in π can be traded for critic capacity. To test this hypothesis, we pair Qwen3-4B and Qwen3-8B actors with their corresponding 0.6B and 1.7B critics from the same family. For both PPO and π , we compare these asymmetric configurations against their actor-sized symmetric counterparts. Figure 3: Performance (bars, left axis) and explained variance (EV; curves, right axis) of PPO and π . Sym. and Asym. denote critics matching the actor size and smaller critics, respectively. Dashed lines indicate DAPO, the best-performing baseline. ❶π achieves strong overall performance with substantially smaller critics. As shown in Figure 3, π achieves overall accuracies of 47.8% and 49.9% under asymmetric actor–critic configurations, still exceeding the strongest baseline on both backbones. Notably, it also outperforms actor–critic baselines with substantially larger, actor-sized critics, showing that privileged evidence can unlock stronger policy optimization with compact critics. ❷π retains its value estimation advantage at a fraction of the critic size. Reducing the critics to 0.6B and 1.7B cuts their parameter counts by approximately 85% and 79%, yet EV decreases by only 0.037 and 0.027. These declines are smaller than in PPO, and the resulting EVs of 0.655 and 0.632 still exceed the 0.446 and 0.564 achieved by PPO with actor-sized critics. Overall accuracy decreases by only 2.5 and 1.7 percentage points relative to the symmetric configurations. Taken together, these results suggest that privileged value estimation offers a rationale for revisiting actor–critic scale matching: the improved estimation quality motivates asymmetric designs that retain strong policy performance with substantially smaller critics. 5 Analysis 5.1 Characterizing Privileged Value Estimation To investigate the performance gap between PPO and π , we examine their critic predictions, which provide advantage estimates for an otherwise unchanged PPO update. All analyses use the training data. Since ground-truth state values are not directly observable, we construct an empirical reference by running 256 MC samples for each prefix state sts_t and averaging the returns. We compare both critics against this MC reference to assess the accuracy of their value predictions. Accuracy of value estimates. We first group the prefix states by their MC-estimated values and compare the distributions of predictions from the two critics in Figure 4(a). Accurate estimates should concentrate near the diagonal, whereas systematic deviations indicate over- or underestimation. PPO exhibits widely dispersed predictions, particularly intermediate value states, while π generally produces tighter distributions that more closely follow the MC reference. Figure 4(b) further shows that π achieves lower MSE at every evaluated prefix position. As generation progresses, PPO’s error generally increases, whereas that of π remains comparatively stable. At the final evaluated position, the MSE is approximately 0.05 for π versus 0.15 for PPO. Figure 4: (a) Critic predictions grouped by MC reference value. Boxes show the interquartile range, black bars indicate the mean, and whiskers mark the 10th and 90th percentiles. The dashed diagonal denotes exact agreement with the MC reference. (b) Mean squared error against the MC reference across normalized prefix positions. Figure 5: Critic ranking and state identification across normalized prefix positions. (a) Probability of assigning a higher value to a prefix from a successful trajectory than to one from an unsuccessful trajectory. (b) Chance-normalized top-1 accuracy for identifying the state with the highest MC reference value among prefixes sharing the same prompt and reasoning-node rank. Dashed lines indicate random-selection baselines. State ranking and identification. Pairwise ranking measures how often a critic assigns a higher value to a prefix from a successful trajectory than to one from an unsuccessful trajectory. Figure 5(a) shows that π ’s advantage widens as generation progresses, reaching approximately 94% pairwise accuracy compared with 78% for PPO at the final position. We further form candidate sets from different trajectories of the same prompt at each of seven reasoning-node ranks, and test whether the critic identifies the state with the highest MC reference value. π achieves higher chance-normalized top-1 accuracy at most evaluated positions, reaching approximately 50% versus 15% for PPO at the final position (Figure 5(b)). Reasoning progress decomposition. Beyond value accuracy and state ranking, we examine whether the critic tracks local value changes across reasoning functions using the ThinkARM taxonomy (Li et al., 2026). This is assessed by correlating stepwise changes in critic predictions with changes in MC reference values and breaking down the results by reasoning function. Figure 6 shows higher correlations for π overall and within Analyze and Implement, which together account for approximately 90% of the evaluated nodes. PPO’s correlations are near zero in both functions, while π reaches approximately 0.2. These results suggest improved tracking of local value changes during the dominant intermediate reasoning activities. Figure 6: Pearson correlation between predicted and MC-estimated value changes. Error bars show trajectory-cluster 95% CIs; percentages indicate the shares of evaluated nodes. 5.2 The role of contrastive evidence We test alternative forms of privileged information to identify which aspects contribute to the gains: removing correctness labels while retaining the selected references (w/o labels), replacing opposite-outcome pairs with same-outcome references (w/ same polarity), and using only the ground-truth final answer (w/ GT only). Removing labels or opposite-outcome pairing erases most gains over baselines; the offline GT-only variant trails full π by 4.7 and 3.1 points on Qwen3-4B and Qwen3-8B, respectively. These results link the gains to the contrastive organization of online trajectory evidence. Method Overall Qwen3-4B Avg. Qwen3-8B Avg. πPPO 50.3 51.6 w/o labels 44.1 48.8 w/ same polarity 43.6 48.6 w/ GT only 45.6 48.5 Table 2: Effects of correctness labels, reference polarity, and ground-truth-only context on average accuracy (%) across five benchmarks. 6 Related Work RL for LLM through the lens of value estimation. RL algorithms used for LLM post-training are predominantly policy-gradient methods, with REINFORCE (Williams, 1992) as their prototype. Prominent approaches in LLM post-training adopt either an actor–critic (e.g., PPO-based RLHF (Schulman et al., 2017; Ouyang et al., 2022), VC-PPO (Yuan et al., 2025), VAPO (Yue et al., 2025)) or critic-free formulation (e.g., bandit-like algorithms such as GRPO (Shao et al., 2024), DAPO (Yu et al., 2026), DR.GRPO (Liu et al., 2025)). Despite differing in their use of a learned critic, both formulations can be examined through the lens of value estimation. From this perspective, a central question underlying their relative effectiveness is whether value estimation is sufficiently reliable to improve the bias–variance trade-off in policy-gradient updates. π addresses this question from the perspective of information available to the critic, introducing training-time privileged information to improve value estimation for policy optimization. Privileged information for policy optimization. Privileged information (PI) was formalized in learning using privileged information (LUPI), where a teacher supplies additional information available only during training to improve learning and generalization (Vapnik & Vashist, 2009); how such information is incorporated, however, remains flexible. In LLM post-training, a direct approach is hint-based RL, adding PI from external experts into the input or supplied as a response prefix to guide sampling toward successful trajectories (Chen et al., 2026a; Qu et al., 2026), training in an off-policy way, also causing a distributional mismatch between training and inference time. Recent on-policy self-distillation uses PI to induce a stronger in-context distribution that supervises the policy on its own rollouts (Zhao et al., 2026; Shenfeld et al., 2026; Tang et al., 2026). This provides dense token-level supervision, but model-induced targets may not align with the original reward-maximization objective (Kaur et al., 2026; Nguyen et al., 2026). π places PI within the optimization process through critic-side value estimation, improving the information available for policy updates while preserving the actor’s conditioning and the environmental reward objective. 7 Conclusion In this paper, we recast value estimation in RLVR as an information problem, showing that a critic scoring each state in isolation can be inherently limited by its formulation, and that the evidence beyond target state can lower this floor. Building on this view, we propose π , which conditions the critic on a self-elicited, leave-target-out context of verified same-prompt siblings while preserving the actor’s original conditioning and environment-defined reward objective, with no additional generation. Across mathematical reasoning benchmarks with Qwen3-4B and Qwen3-8B backbones, π outperforms its state-only actor–critic counterparts in critic quality, consistently achieving higher explained variance and lower value loss, while delivering the strongest end-to-end accuracy among all evaluated actor–critic and critic-free baselines—even staying ahead when we further exploit capacity asymmetry with a smaller critic model. Overall, π provides a complementary design view on developing more effective RL training algorithms for LLMs. References AI-MO (2024) AI-MO. Aime 2024. https://huggingface.co/datasets/AI-MO/aimo-validation-aime, 2024. Amos (2025) Brandon Amos. Tutorial on amortized optimization, 2025. URL https://arxiv.org/abs/2202.00665. Balunović et al. (2025) Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating llms on uncontaminated math competitions, February 2025. URL https://matharena.ai/. ByteDance-Seed (2025) ByteDance-Seed. Beyondaime: Advancing math reasoning evaluation beyond high school olympiads. https://huggingface.co/datasets/ByteDance-Seed/BeyondAIME, 2025. Hugging Face repository. Chen et al. (2026a) Justin Chen, Xiangyu Peng, Prafulla Kumar Choubey, Kung-Hsiang Huang, Jiaxin Zhang, Mohit Bansal, and Chien-Sheng Wu. Nudging the boundaries of llm reasoning. In International Conference on Learning Representations, 2026a. Chen et al. (2026b) Zizhe Chen, Jiqian Dong, Yizhou Tian, Garry Yang, Yongqiang Chen, Zhitang Chen, and James Cheng. Hista and numca: Estimate state value effectively for llm reinforcement learning, 2026b. URL https://arxiv.org/abs/2605.29782. Greensmith et al. (2004) Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471–1530, 2004. Guo et al. (2025a) Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025a. doi: 10.1038/s41586-025-09422-z. Guo et al. (2025b) Yiran Guo, Lijie Xu, Jie Liu, Dan Ye, and Shuang Qiu. Segment policy optimization: Effective segment-level credit assignment in rl for large language models, 2025b. URL https://arxiv.org/abs/2505.23564. Hou et al. (2026) Zhenyu Hou, Yujiang Li, Jie Tang, and Yuxiao Dong. Single-rollout asynchronous optimization for agentic reinforcement learning, 2026. URL https://arxiv.org/abs/2607.07508. Kaur et al. (2026) Simran Kaur, Narutatsu Ri, Yinghui He, Liam Fowl, and Sanjeev Arora. Rethinking on-policy self-distillation for thinking models. arXiv preprint arXiv:2607.05184, 2026. Kazemnejad et al. (2024) Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking rl potential for llm reasoning through refined credit assignment. 2024. Li et al. (2026) Ming Li, Chenrui Fan, Yize Cheng, Soheil Feizi, and Tianyi Zhou. Schoenfeld’s anatomy of mathematical reasoning by language models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 32773–32802, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1513. URL https://aclanthology.org/2026.acl-long.1513/. Liu et al. (2025) Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=5PAF7PAY2Y. Nguyen et al. (2026) Xuan-Phi Nguyen, Shrey Pandit, Yiran Zhao, Anurag Koul, Zeyu Liu, and Shafiq Joty. Privileged likelihood is not automatically value: Three checks for token credit in on-policy self-distillation. arXiv preprint arXiv:2608.09263, 2026. OpenCompass (2025) OpenCompass. Aime 2025. https://huggingface.co/datasets/opencompass/AIME2025, 2025. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Pan et al. (2026) Leyi Pan, Shuchang Tao, Yunpeng Zhai, Lingzhe Zhang, Zhaoyang Liu, Bolin Ding, Aiwei Liu, and Lijie Wen. Rlcsd: Reinforcement learning with contrastive on-policy self-distillation. arXiv preprint arXiv:2606.11709, 2026. Qu et al. (2026) Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. Pope: Learning to reason on hard problems via privileged on-policy exploration. arXiv preprint arXiv:2601.18779, 2026. Rein et al. (2024) David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024. Schulman et al. (2015) John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shan et al. (2026) Zikang Shan, Han Zhong, Liwei Wang, and Li Zhao. Bringing value models back: Generative critics for value modeling in llm reinforcement learning, 2026. URL https://arxiv.org/abs/2604.10701. Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Shenfeld et al. (2026) Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. Sheng et al. (2024) Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256, 2024. Shu (2017) Rui Shu. Amortized optimization. https://ruishu.io/2017/11/07/amortized-optimization/, 2017. Tang et al. (2026) Chenming Tang, Hsiu-Yuan Huang, Weijie Liu, Clive Bai, Saiyong Yang, and Yunfang Wu. Do not step into the same river twice: Learning to reason from trial and error. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 8018–8032, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.363. URL https://aclanthology.org/2026.acl-long.363/. Team et al. (2025) Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025. Vapnik & Vashist (2009) Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged information. Neural networks, 22(5-6):544–557, 2009. Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. Advances in Neural Information Processing Systems, 37:95266–95290, 2024. Williams (1992) Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. Yang et al. (2025) An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Yang et al. (2026) Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026. Yu et al. (2026) Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244, 2026. Yuan et al. (2025) Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What’s behind ppo’s collapse in long-cot? value optimization holds the secret, 2025. URL https://arxiv.org/abs/2503.01491. Yue et al. (2025) Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, et al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.05118, 2025. Zhang et al. (2026) Yi-Kai Zhang, Zhiyuan Yao, Hongyan Hao, Yueqing Sun, Qi Gu, Hui Su, Xunliang Cai, De-Chuan Zhan, and Han-Jia Ye. v0v_0: A generalist value model for any policy at state zero, 2026. URL https://arxiv.org/abs/2602.03584. Zhao et al. (2026) Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Appendix A Theoretical Proof A.1 Value Estimation under Privileged Conditioning Proof of Proposition 1. All expectations are taken under the fixed joint distribution of (s,ℐ,R)(s,I,R) induced by the policy and the context construction. The binary return is square-integrable, and we consider only critics with finite squared error. Step 1: Risk decomposition. For any fixed contextual critic, write R−Vϕ+(s,ℐ)=(R−Vπ,+(s,ℐ))+(Vπ,+(s,ℐ)−Vϕ+(s,ℐ)).R-V_φ^+(s,I)= (R-V^π,+(s,I) )+ (V^π,+(s,I)-V_φ^+(s,I) ). By the definition of conditional expectation, [R−Vπ,+(s,ℐ)∣s,ℐ]=0.E[R-V^π,+(s,I) s,I]=0. Since Vπ,+(s,ℐ)−Vϕ+(s,ℐ)V^π,+(s,I)-V_φ^+(s,I) is a function of (s,ℐ)(s,I), the cross term vanishes: [(R−Vπ,+(s,ℐ))(Vπ,+(s,ℐ)−Vϕ+(s,ℐ))] \! [ (R-V^π,+(s,I) ) (V^π,+(s,I)-V_φ^+(s,I) ) ] =[(Vπ,+(s,ℐ)−Vϕ+(s,ℐ))[R−Vπ,+(s,ℐ)∣s,ℐ]]=0. =E\! [ (V^π,+(s,I)-V_φ^+(s,I) )E[R-V^π,+(s,I) s,I] ]=0. Squaring the decomposition and taking expectations therefore gives [(R−Vϕ+)2] [(R-V_φ^+)^2] =s,ℐ[Var(R∣s,ℐ)]+[(Vϕ+−Vπ,+)2]. =E_s,I[Var(R s,I)]+E[(V_φ^+-V^π,+)^2]. (9) The variance term does not depend on ϕφ, so it can be taken outside the infimum over the shared critic parameters. This proves the first equality in the proposition. Similarly, conditioning on s gives [(R−Vϕ(s))2] [(R-V_φ(s))^2] =s[Var(R∣s)]+[(Vϕ(s)−Vπ(s))2]. =E_s[Var(R s)]+E[(V_φ(s)-V^π(s))^2]. (10) Taking the infimum over ϕφ establishes the final equality. Step 2: Optimal risk comparison. We next compare the two optimal risks. By assumption, for every state-only parameter vector ϕφ there exists a contextual parameter vector ψ (e.g., with the contribution of the additional context ℐI effectively set to zero) such that Vψ+(s,ℐ)=Vϕ(s)almost surely.V_ψ^+(s,I)=V_φ(s) surely. Consequently, for every ϕφ, infψ[(R−Vψ+(s,ℐ))2]≤[(R−Vϕ(s))2]. _ψE[(R-V_ψ^+(s,I))^2] [(R-V_φ(s))^2]. Taking the infimum on the right gives infψ[(R−Vψ+(s,ℐ))2]≤infϕ[(R−Vϕ(s))2]. _ψE[(R-V_ψ^+(s,I))^2]≤ _φE[(R-V_φ(s))^2]. (11) Taking infima in Eqs. (9) and (10), whose variance terms are independent of the critic parameters, and applying Eq. (11) proves the proposition. ∎ A.2 Success–Failure Contrast in Value Estimation Proof of Proposition 2. All expectations are taken under the fixed joint distribution of (s,R)(s,R) specified in Proposition 2. Since R is binary and the critic has finite squared error, all second moments below are finite. Step 1: Outcome-conditioned risk decomposition. By the definition of Δϕ _φ and the binary nature of R, [R−Vϕ(s)∣R] [R-V_φ(s) R] =(1−Δϕ)R−[Vϕ(s)∣R=0], =(1- _φ)R-E[V_φ(s) R=0], Var(R−Vϕ(s)∣R) (R-V_φ(s) R) =Var(Vϕ(s)∣R). =Var(V_φ(s) R). Applying the law of total variance to R−Vϕ(s)R-V_φ(s) therefore gives [(R−Vϕ(s))2] [(R-V_φ(s))^2] =([Vϕ(s)]−[R])2+Var(R−Vϕ(s)) = (E[V_φ(s)]-E[R] )^2+Var(R-V_φ(s)) (12) =([Vϕ(s)]−[R])2+R[Var(Vϕ(s)∣R)] = (E[V_φ(s)]-E[R] )^2+E_R[Var(V_φ(s) R)] +Var(R)(1−Δϕ)2. +Var(R)(1- _φ)^2. Step 2: Contrast lower bound. Let εϕ=([Vϕ(s)]−[R])2+R[Var(Vϕ(s)∣R)]≥0. _φ= (E[V_φ(s)]-E[R] )^2+E_R[Var(V_φ(s) R)]≥ 0. Equation (12) then yields [(R−Vϕ(s))2] [(R-V_φ(s))^2] =Var(R)(1−Δϕ)2+εϕ =Var(R)(1- _φ)^2+ _φ (13) ≥Var(R)(1−Δϕ)2. (R)(1- _φ)^2. For any constant predictor c, [(R−c)2]=Var(R)+(c−[R])2,E[(R-c)^2]=Var(R)+(c-E[R])^2, so the optimal constant prediction risk is Var(R)Var(R), attained at c=[R]c=E[R]. If Δϕ≤0 _φ≤ 0, then (1−Δϕ)2≥1(1- _φ)^2≥ 1, and Eq. (13) implies [(R−Vϕ(s))2]≥Var(R)E[(R-V_φ(s))^2] (R). Thus, any critic improving upon the optimal constant prediction risk must satisfy Δϕ>0 _φ>0. ∎ Appendix B Implementation Details In this section, we provide the details of our main experiments in Section 4. B.1 Training Details We build our implementation on the VeRL framework (Sheng et al., 2024). All methods are trained for 250 steps with a batch size of 256, utilizing learning rates of 10−610^-6 for the actor and 10−510^-5 for the critic. Following DAPO and VAPO (Yu et al., 2026; Yue et al., 2025), we adopt Clip-Higher with εlow=0.2 _low=0.2 and εhigh=0.28 _high=0.28, omitting KL regularization. For critic-based methods, we set γ=λ=1.0γ=λ=1.0 and adopt fixed-policy value pretraining with Monte Carlo targets following VC-PPO (Yuan et al., 2025). Specifically, we perform 50 pretraining steps with the actor frozen, linearly warming up the critic learning rate from 10−710^-7 to 10−510^-5. During policy training, we generate G=8G=8 responses per prompt with temperature 1.01.0, top-p 1.01.0, and top-k disabled. For π , references are selected randomly subject to the leave-target-out construction in Eq. (4). The maximum response length and the privileged-context truncation length are both set to 8192 tokens per sample. During value pretraining, we sample one response per prompt online, and set the batchsize to 2048 instead; π obtains its reference context from a cache of the most recent verified rollouts from previous epochs. The remaining pretraining settings are shared with the standard critic. Algorithm 1 summarizes the overall procedure. Baselines. All baselines share the common training, rollout, and evaluation settings described above. For a fair comparison, all methods use Clip-Higher and generate G=8G=8 responses per prompt. For DAPO (Yu et al., 2026), we implement dynamic sampling with a rejection pool size of 512, twice the training batch size. For VAPO (Yue et al., 2025), we set the length-adaptive GAE coefficient α to 1.01.0 (functioning only on the actor side). Context ablations. All ablations modify the critic’s privileged context and otherwise follow the common settings. For w/o labels, we preserve the selected reference pair, replace correctness headers with ordinal labels, and shuffle the reference order. For w/ same polarity, we supply two correct or two incorrect sibling references with their correctness labels, otherwise falling back to a contrastive pair. For w/ GT only, we remove all rollout references and provide only the verified ground-truth final answer. B.2 Evaluation and Analysis Details We use the same vanilla evaluation prompts and decoding settings across methods for each backbone. Responses are sampled with temperature 0.60.6, top-p 0.950.95, top-k 2020, and a maximum length of 32768 tokens. We generate 16 responses per problem on AIME 2024, AIME 2025, HMMT February, and HMMT November, and 8 responses per problem on BeyondAIME. We report the average correctness across sampled responses as an estimate of pass@1, with Overall computed as the mean over the five benchmarks. Critic diagnostics. For critic diagnostics, we evaluate step-250 Qwen3-8B checkpoints on 128 prompts per method and estimate values from 256 MC continuations at seven semantic prefixes, with completed responses capped at 8192 tokens. Pairwise ranking uses 20 prefix positions; top-1 identification retains candidate sets whose MC-value range exceeds twice the estimated uncertainty. Algorithm 1 π : Privileged-Information PPO Input: Actor πθ _θ; warmed-up privileged critic Vϕ+V_φ^+; prompt dataset D with ground-truth answers; verifier R; group size G; discount γ; GAE parameter λ for each training iteration do θold←θ _old←θ Sample a prompt batch ℬ⊂B foreach prompt ∈ℬ x do Retrieve its ground-truth answer ⋆ y // Collect and verify same-prompt rollouts Independently sample (i)i=1G∼πθold(⋅∣)\ y^(i)\_i=1^G _ _old(· x) Evaluate R(,(i))∈0,1R( x, y^(i))∈\0,1\ and construct token rewards rt(i)t=1Ti\r_t^(i)\_t=1^T_i, where Ti=|(i)|T_i=| y^(i)| for i=1,…,Gi=1,…,G do // Exclude the target before selecting references i+←(j):j≠i,R(,(j))=1C_i^+←\ y^(j):j≠ i,\ R( x, y^(j))=1\ i−←(j):j≠i,R(,(j))=0C_i^-←\ y^(j):j≠ i,\ R( x, y^(j))=0\ if i+≠∅C_i^+≠ and i−≠∅C_i^-≠ then ℐ(i)←1(i+)|1(i−)I^(i) _1(C_i^+)\|S_1(C_i^-) else if i−=∅C_i^-= then ℐ(i)←2(i+)I^(i) _2(C_i^+) else ℐ(i)←2(i−)|[⋆]I^(i) _2(C_i^-)\|[ y ] // Estimate values under a fixed context st(i)←(,<t(i))s_t^(i)←( x, y_<t^(i)) Compute and store V^i,t←Vϕ+(st(i),ℐ(i)) V_i,t← V_φ^+(s_t^(i),I^(i)), with zero value at terminal states for t=Ti,…,1t=T_i,…,1 do δt+(i)←rt(i)+γV^i,t+1−V^i,t _t^+(i)← r_t^(i)+γ V_i,t+1- V_i,t A^t+(i)←δt+(i)+γλA^t+1+(i) A_t^+(i)← _t^+(i)+γλ A_t+1^+(i), with A^Ti+1+(i)=0 A_T_i+1^+(i)=0 Compute and store value-regression targets G^i,tt=1Ti\ G_i,t\_t=1^T_i using the PPO return-target rule for each PPO optimization epoch do Update ϕφ by value regression on (st(i),ℐ(i),G^i,t)(s_t^(i),I^(i), G_i,t) Update θ using the clipped PPO objective with A^t+(i) A_t^+(i) and ρt(i)(θ)=πθ(yt(i)∣,<t(i))πθold(yt(i)∣,<t(i)) _t^(i)(θ)= _θ(y_t^(i) x, y_<t^(i)) _ _old(y_t^(i) x, y_<t^(i)) Appendix C Prompt Template for π Prompt Template used in π <|im_start|>user /* [shared source: original user turn → actor, VϕV_φ, Vϕ+V_φ^+] */ Let x and y be real numbers with x+y=25x+y=25 and xy=126xy=126. Find |x−y||x-y|. Please reason step by step, and put your final answer within . /* [critic-only source: labeled reference attempts → Vϕ+V^+_φ only] */ Below are reference attempts at the problem. Each one is explicitly labeled with whether it reached the correct final answer. They are backgrounded and are NOT part of the response. [Reference attempt — CORRECT]: (1) The sum and the product are given, so let s=x+y=25s=x+y=25 and p=xy=126p=xy=126. (2) I want |x−y||x-y|, and the clean route is to obtain (x−y)2(x-y)^2 from s and p. (3) Since (x−y)2=x2−2xy+y2(x-y)^2=x^2-2xy+y^2 and (x+y)2=x2+2xy+y2(x+y)^2=x^2+2xy+y^2, subtracting gives (x−y)2=(x+y)2−4xy(x-y)^2=(x+y)^2-4xy. (4) Substituting, (x−y)2=625−4⋅126=625−504=121(x-y)^2=625-4· 126=625-504=121. (5) Therefore |x−y|=121=11|x-y|= 121=11. (6) Check: the numbers solve t2−25t+126=0t^2-25t+126=0, so they are 1818 and 77; their sum is 2525 and their product is 126126, both as required. (7) The final answer is 11 11. [Reference attempt — INCORRECT]: (1) The sum and the product are given, so let s=x+y=25s=x+y=25 and p=xy=126p=xy=126. (2) I want |x−y||x-y|, and the clean route is to obtain (x−y)2(x-y)^2 from s and p. (3) Expanding the square of the difference, (x−y)2=(x+y)2−2xy(x-y)^2=(x+y)^2-2xy. (4) Substituting, (x−y)2=625−2⋅126=625−252=373(x-y)^2=625-2· 126=625-252=373. (5) Therefore |x−y|=373≈19.31|x-y|= 373≈ 19.31. (6) Check: the numbers are then about 22.1622.16 and 2.842.84, and 22.16+2.84=2522.16+2.84=25, so the sum comes out right. (7) The final answer is 373 373. <|im_end|> <|im_start|>assistant /* [shared source: actor-sampled completion → VϕV_φ, Vϕ+V_φ^+] */ (1) Let x and y be the two numbers, so x+y=25x+y=25 and xy=126xy=126. (2) Then x and y are the roots of t2−25t+126=0t^2-25t+126=0. (3) Its discriminant is 252−4⋅126=625−504=12125^2-4· 126=625-504=121. (4) Hence t=25±1212=25±112t= 25± 1212= 25± 112, giving t=18t=18 and t=7t=7. (5) So |x−y|=18−7=11|x-y|=18-7=11. (6) The final answer is 11 11. <|im_end|> Table 3: The actor and the state-only critic VϕV_φ share the original user turn and actor-generated completion, whereas the privileged critic Vϕ+V_φ^+ receives the same completion with a labeled block of additional information appended to the user turn. The two reference attempts in this block are numbered step by step and agree verbatim through step (2)(2), but diverge at step (3)(3) in their expansion of (x−y)2(x-y)^2, using −4xy-4xy and −2xy-2xy, respectively, forming a direct process-level contrast. Appendix D Additional Results D.1 Evaluation on OOD benchmarks Table 4: Out-of-domain accuracy (%) of policies trained on DAPO-17K. “Avg@k” denotes mean accuracy over k random generations (i.e., pass@1); Overall is the mean over the two benchmarks. Asym: the method above with a smaller critic (0.6B for 4B, 1.7B for 8B). Best result within each model size is in bold. Method Qwen3-4B Qwen3-8B GPQA Avg@8 MMLU Avg@1 Overall Avg. GPQA Avg@8 MMLU Avg@1 Overall Avg. Initial policy 45.7 60.8 53.3 51.0 66.1 58.6 Critic-free GRPO 52.9 67.3 60.1 58.3 71.6 65.0 DAPO 52.8 67.7 60.3 56.0 72.0 64.0 Actor–critic PPO 51.5 66.3 58.9 56.3 71.3 63.8 ↰ Asym 52.7 66.8 59.8 57.9 71.4 64.7 VAPO 51.4 66.0 58.7 56.9 69.7 63.3 πPPO (ours) 52.8 68.2 60.5 57.7 71.7 64.7 ↰ Asym 52.6 66.6 59.6 58.8 71.5 65.2 We additionally evaluated our method on representative multi-task benchmarks spanning diverse domains in science, humanities, and professional knowledge: GPQA-Diamond (Rein et al., 2024) and MMLU-Pro (Wang et al., 2024), for examining the out-of-distribution generalization. As shown in Table 4, all RL-trained policies outperform their initial counterparts. At the 4B scale, π achieves the best Overall score of 60.5. At the 8B scale, π reaches 64.7, its asymmetric variant achieves the best 8B Overall score of 65.2 using only a 1.7B critic. These results suggest that the in-domain advantages of π may carry over to other related domains. D.2 Additional Comparison with self-distillation methods Table 5: Comparison with RLSD on mathematical reasoning benchmarks. Overall averages the five benchmark scores; bold denotes the best result within each model size. Method Mathematical Reasoning AIME 24 Avg@16 AIME 25 Avg@16 Beyond AIME Avg@8 HMMTFeb Avg@16 HMMTNov Avg@16 Overall Avg. (a) Qwen3-4B Initial policy 21.7 18.9 10.8 12.3 7.3 14.2 RLSD 57.1 47.9 31.5 31.5 38.8 41.4 πPPO (ours) 66.7 61.5 39.0 37.5 46.9 50.3 (b) Qwen3-8B Initial policy 25.0 20.2 13.1 11.5 10.0 16.0 RLSD 67.5 50.4 35.3 28.3 38.8 44.1 πPPO (ours) 73.5 61.5 40.0 34.4 48.5 51.6 We additionally compare with OPSD (Zhao et al., 2026) and its RL-integrated variants, RLSD (Yang et al., 2026) and RLCSD (Pan et al., 2026), since these methods can be viewed in the same general way as ours: leveraging privileged information to provide additional supervision or shape token-level rewards. We follow the original implementations and adapt these methods to the training setup used in our main experiments. The reference configuration uses a batch size of 256, an actor learning rate of 10−610^-6, and a budget of 250 policy updates. We sample G=8G=8 responses per prompt with temperature 1.01.0, top-p 1.01.0, disabled top-k filtering, and a maximum response length of 8192 tokens. We explore multiple configurations around this setup to adapt the methods for better training stability and effectiveness. Across the tested configurations, we consistently observe training oscillations and collapse for OPSD and RLCSD, with no effective gain observed compared to the initial model. Among these three baselines, only RLSD achieves reasonable downstream performance under our experimental setting, which follows their original setups.