Paper deep dive
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning
Zhu Zhang, Jixun Wang, Xiaoang Xu, Xiaorong Wang, Zihan Zhou, Zhiyuan Wang, Shuo Wang, Chaojun Xiao, Yuezhi Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/22/2026, 1:44:30 AM
Summary
The paper introduces Group-Calibrated On-Policy Distillation (GC-OPD), a method to address the misalignment between teacher likelihood and task-specific verifier rewards in long-context reasoning. GC-OPD computes a signed residual between group-normalized verifier rewards and trajectory-level OPD scores, then distributes this residual across tokens using Relative-Advantage-based Credit Assignment (RACA). Experiments on Qwen3-4B and Qwen3-8B models show significant performance improvements over vanilla OPD across five long-context benchmarks.
Entities (9)
Relation Signals (6)
GC-OPD ā improves ā Qwen3-4B
confidence 95% Ā· post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B... checkpoints from 29.08 to 40.47
GC-OPD ā improves ā Qwen3-8b
confidence 95% Ā· post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 35.12 to 44.65
GC-OPD ā outperforms ā OPD
confidence 95% Ā· Vanilla OPD reaches 39.31 and 43.56 under the same setup. [GC-OPD reaches 40.47 and 44.65]
GC-OPD ā usescomponent ā RACA
confidence 95% Ā· Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens... GC-OPD substitutes A' for A in the clipped token-level policy objective
Qwen3-30B-A3B-Thinking-2507 ā servesasteacherfor ā Qwen3-8b
confidence 90% Ā· We use Qwen3-8B as the student and Qwen3-30B-A3B-Thinking-2507 as the teacher.
OPD ā suffersfrom ā Teacher-Verifier Disagreement
confidence 90% Ā· In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence... We diagnose this mismatch... indicating teacher-verifier disagreement.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2608.19181v1
- Canonical: https://arxiv.org/abs/2608.19181v1
Trouble viewing inline? Open PDF directly ā
Full Text
66,667 characters extracted from source content.
Expand or collapse full text
Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning Zhu Zhang Thanks: Equal contribution. Emails: zhuzhang24@mails.tsinghua.edu.cn, wangjixun@bupt.edu.cn. Jixun Wang11footnotemark: 1 Xiaoang Xu Xiaorong Wang Zihan Zhou Zhiyuan Wang Shuo Wang Chaojun Xiao Thanks: Corresponding authors. Emails: xcj@tsinghua.edu.cn, zhouyz@mail.tsinghua.edu.cn. Yuezhi Zhou22footnotemark: 2 [0.45em] Tsinghua University Beijing University of Posts and Telecommunications OpenBMB Abstract On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacherāverifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacherāverifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD. Figure 1: Teacherāverifier disagreement in two evidence-aggregation tasks: Multi-Table Extraction (MTE) and High-Recall Retrieval (HRR). Pairwise disagreement rate captures ranking conflicts, while OPD preference gap captures the direction and magnitude of OPD preference along the verifier ordering. Both indicate weaker alignment over longer input ranges. See Section 3.2 for details. 1 Introduction Knowledge distillation transfers knowledge from a strong teacher to a weaker student and is particularly valuable for compact language models [8, 7, 13]. On-policy distillation (OPD) applies this paradigm to student-generated responses, providing dense teacher guidance on the studentās own generation distribution [1]. For each sampled token, OPD derives an advantage from the difference between its teacher and student log-probabilities. This dense signal underlies strong recent OPD methods [31, 16, 34], but measures teacher preference rather than verified task success. This distinction is particularly important for long-context tasks, where correct responses must integrate evidence scattered across distant positions and satisfy global task constraints [3, 10, 6, 4]. With longer inputs, student models are more likely to produce locally plausible but globally incomplete responses, as they can struggle to locate and use relevant information when its position or lexical form changes [18, 21]. A response may therefore receive strong teacher support while omitting required evidence. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success [23]. We call discrepancies between teacher preference and verified task outcomes teacherāverifier disagreement. Figure 1 presents diagnostic evidence from a fixed set of generated responses in two representative tasks that share this evidence-aggregation requirement. In both, trajectory-level OPD scores become progressively less aligned with verifier rewards across the analyzed input-length ranges. This trend motivates an interface that preserves dense teacher guidance while correcting response-level preferences inconsistent with verified outcomes. Existing verifier-aware OPD methods use outcome feedback through objective routing, teacher-context reconstruction, outcome-class calibration, or token-level gating [36, 32, 9, 29]. These methods establish that verifier feedback complements dense teacher guidance. However, they do not simultaneously preserve graded within-group outcome differences and translate the resulting response-level discrepancy into token-dependent calibration while retaining the original OPD advantage. To address these limitations, we propose Group-Calibrated On-Policy Distillation (GC-OPD). Following the group-relative comparison in Group Relative Policy Optimization [23], GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group. As Figure 2 shows, their difference forms a signed residual whose direction and magnitude quantify teacherāverifier disagreement. Subtracting the group-normalized OPD assessment focuses calibration on their discrepancy rather than directly adding verifier feedback. Relative-advantage-based credit assignment (RACA) distributes this residual across tokens according to their relative OPD advantages while retaining the original dense OPD signal. Our contributions are as follows: ā¢ We identify a shared pattern across two distributed-evidence long-context tasks: trajectory-level OPD scores become progressively less aligned with verifier rewards over longer inputs, as measured by pairwise disagreement rate and OPD preference gap. ā¢ We propose GC-OPD, which retains the original OPD advantage while forming a signed residual between group-normalized verifier rewards and trajectory-level OPD scores. RACA distributes this residual across tokens using relative OPD advantages. ā¢ Across two model scales and five long-context benchmarks, GC-OPD achieves the highest average score under the shared setup. Ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA improves over uniform token allocation. Figure 2: Overview of GC-OPD. GC-OPD computes group-normalized verifier rewards and trajectory-level OPD scores, forms their signed residual, and uses RACA to distribute the residual across response tokens while retaining the original OPD advantage. 2 Related Work Long-context reasoning and verifiable post-training. Long-context benchmarks show that nominal context capacity does not guarantee reliable retrieval, integration, or reasoning over evidence distributed across long inputs [3, 10, 18, 6, 21]. Corresponding post-training work improves these capabilities through long-instruction data, length-aware optimization, reinforcement-learning curricula, and task-specific feedback [2, 25, 27, 5, 33]. These studies improve the construction of long-context data, curricula, and rewards. Our setting instead begins with an existing task verifier and asks how its response-level assessment should interact with dense token-level OPD guidance. Teacher-centric on-policy distillation. MiniLLM and GKD distill on student-generated sequences, exposing the teacher to the studentās own generation distribution [7, 1]. ExOPD extrapolates along a teacherāreference direction [31], FiRe-OPD filters trajectories and reweights informative tokens [16], and PowerOPD bounds sampled-token rewards for stability [34]. Complementary analyses examine teacherāstudent compatibility and whether large token-level disagreement is learnable, while token-selective distillation concentrates supervision on selected positions [15, 28, 12]. This line improves how teacher-derived supervision is formed, selected, or optimized. It does not, however, explicitly compare the resulting trajectory-level OPD assessment with a task-specific verifier reward. Verifier-aware on-policy distillation. Recent methods connect verifier outcomes to teacher supervision through several interfaces. SCOPE routes correct responses to weighted maximum likelihood and incorrect responses to teacher-perplexity-weighted OPD [36]. MOPD conditions teacher supervision on successful and failed peer responses [32]. Uni-OPD calibrates trajectory returns using an outcome-class margin [9]. SG-OPD gates token updates by sign consistency and supplements early training with verifier-endorsed teacher rollouts [29]. Reward-Weighted OPD selects verifier-passable rollouts and weights their dense teacher supervision [37]. These methods introduce verifier feedback through objective routing, teacher conditioning, return calibration, token gating, or trajectory weighting. GC-OPD instead leaves the teacher pass unchanged and forms a response-specific residual from group-relative verifier and OPD assessments. It retains graded within-group differences and the original token-level OPD advantage. Trajectory-level feedback and token-level credit. A terminal verifier supplies one response-level reward without specifying its tokenwise influence. Process-supervision methods use human or automatically constructed step labels [17, 26], while VinePPO estimates action advantages from auxiliary continuations [11]. These approaches obtain finer-grained evidence through additional labels or sampling. GC-OPD requires neither step labels nor auxiliary continuations. RACA instead allocates the trajectory-level residual with bounded weights derived from relative OPD advantages without inferring token correctness. 3 Background and Motivation 3.1 On-Policy Distillation Let x denote a long-context input and ĻĪø _Īø the trainable student policy. At the beginning of each update, ĻĪøold _ _old denotes a frozen copy of ĻĪø _Īø used to generate the current rollouts. For each input, we sample a rollout group of G responses, ā”()=(i)i=1GG(x)=\y^(i)\_i=1^G, from ĻĪøold _ _old, where (i)=(y1(i),ā¦,yT(i)(i))y^(i)=(y^(i)_1,ā¦,y^(i)_T^(i)) has length T(i)T^(i). Let ĻT _T denote the teacher policy. For each response sampled from ĻĪøold _ _old, vanilla OPD defines the token-level advantage of yt(i)y_t^(i) as At(i)= A^(i)_t= logā”ĻTā(yt(i)ā£,<t(i)) _T(y^(i)_t ,y^(i)_<t) (1) ālogā”ĻĪøoldā(yt(i)ā£,<t(i)). - _ _old(y^(i)_t ,y^(i)_<t). Let ht(i)=(,<t(i))h_t^(i)=(x,y^(i)_<t) denote the input and response prefix at position t. Taking the expectation over a token sampled from the frozen rollout policy gives yt(i)ā¼ĻĪøold(ā ā£ht(i))[At(i)]=āKL(ĻĪøold(ā ā£ht(i))ā„ĻT(ā ā£ht(i))).E_y_t^(i) _ _old(Ā· h_t^(i))\! [A_t^(i) ]=-KL\! ( _ _old(Ā· h_t^(i))\| _T(Ā· h_t^(i)) ). (2) Thus, At(i)A_t^(i) is a sampled-token contribution whose expectation equals the negative reverse KL [7, 34]. A positive value indicates greater teacher than student support for the sampled token, whereas a negative value indicates the converse. The signal is dense but reflects teacher preference rather than verified task outcome. During actor optimization, At(i)A_t^(i) is treated as fixed and weights the clipped token-level objective for ĻĪø _Īø, whose importance ratio is computed relative to ĻĪøold _ _old. Positive advantages increase the probability of sampled tokens, whereas negative advantages decrease it. 3.2 TeacherāVerifier Disagreement in Long-Context OPD A task-specific verifier assigns each response a reward R(i)R^(i), which may encode binary correctness or graded partial success. To compare this response-level outcome feedback with token-level OPD guidance, we aggregate the token-level advantages into a trajectory-level OPD score: s(i)=1T(i)āāt=1T(i)At(i).s^(i)= 1T^(i) _t=1^T^(i)A^(i)_t. (3) Averaging avoids direct scaling with response length and summarizes the teacherās mean support for the response relative to the frozen rollout policy. The trajectory-level OPD score and verifier reward assess different properties of the same response. The former summarizes teacher support relative to the frozen rollout policy, whereas the latter measures verified task outcome. Their orderings within a rollout group can therefore disagree: a response with a higher OPD score may receive a lower verifier reward than another response. We refer to such discrepancies as teacherāverifier disagreement. This disagreement is particularly consequential in long-context OPD, where task success can depend on aggregating evidence distributed across distant positions. In this setting, token-level teacher support may favor a locally plausible response even when it omits evidence required by the verifier. Because the two signals also differ in numerical scale, we characterize their agreement through relative comparisons within each rollout group rather than their raw values. We examine how teacherāverifier disagreement varies with prompt length using a fixed set of generated responses from Multi-Table Extraction and High-Recall Retrieval in GoLongRL [20]. Both tasks require aggregating evidence distributed across the input and use graded verifier rewards. The analysis covers 751 Multi-Table prompts and 2,908 High-Recall prompts across three prompt-length ranges below 64K tokens. We use Qwen3-8B as the student and Qwen3-30B-A3B-Thinking-2507 as the teacher. We extend the studentās context window with YaRN using a scaling factor of 4 [22], while the teacher uses its native context window. For each prompt, the student generates eight responses, the teacher supplies the token-level log-probabilities used to compute s(i)s^(i), and the task-specific verifier assigns R(i)R^(i). For a prompt x, let ā”()=(i,j):R(i)>R(j)P(x)=\(i,j):R^(i)>R^(j)\ contain all response pairs with distinct verifier rewards, oriented toward the higher-reward response. We normalize the trajectory-level OPD scores to zero mean and unit variance within the rollout group and denote them by s~(i) s^(i). Let ā”(ā )I(Ā·) denote the indicator function. The pairwise disagreement rate is dpairā()=1|ā”()|āā(i,j)āā”()ā”(s~(i)<s~(j)),d_pair(x)= 1|P(x)| _(i,j) (x)I( s^(i)< s^(j)), (4) and the OPD preference gap is gOPDā()=1|ā”()|āā(i,j)āā”()(s~(i)ās~(j)).g_OPD(x)= 1|P(x)| _(i,j) (x)( s^(i)- s^(j)). (5) The first metric measures how often the OPD ordering disagrees with the verifier ordering. The second captures the direction and magnitude of OPD preference along the verifier ordering, measured in units of the within-group OPD standard deviation. Positive gaps indicate agreement with the verifier ordering, whereas negative gaps indicate greater OPD support for lower-reward responses. We macro-average both metrics over prompts whose eight responses contain at least two distinct verifier rewards. Figure 1 shows that the pairwise disagreement rate increases over longer prompt-length ranges in both tasks, while the OPD preference gap declines from positive to negative. For Multi-Table Extraction, the disagreement rate rises from 40.6% below 8K to 64.0% at 32ā64K, and the preference gap declines from +0.35+0.35 to ā0.37-0.37. For High-Recall Retrieval, the corresponding values change from 35.2% to 60.2% and from +0.65+0.65 to ā0.35-0.35. Within these two tasks, the consistent trends indicate that longer inputs can amplify teacherāverifier disagreement, motivating verifier-based calibration of dense OPD guidance. 4 Method 4.1 Overview Group-Calibrated On-Policy Distillation (GC-OPD) calibrates vanilla OPD with response-level verifier feedback while retaining its dense token-level guidance. As illustrated in Figure 2, GC-OPD first normalizes verifier rewards and trajectory-level OPD scores within each rollout group, then represents their difference as a signed teacherāverifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages, and the resulting correction is added to the original token-level OPD advantage. 4.2 Group-Relative Assessments For each response (i)āā”()y^(i) (x), vanilla OPD supplies a token-level OPD advantage At(i)A^(i)_t for every response token. Equation 3 aggregates these advantages into the trajectory-level OPD score s(i)s^(i), while the verifier assigns the response a reward R(i)R^(i). These quantities capture different properties of the response. The trajectory-level OPD score summarizes the teacherās average support relative to the frozen rollout policy, whereas the verifier reward measures verified task outcome. Since the two signals differ in their raw scales, GC-OPD normalizes them separately within each rollout group. Let zā”(ā )z(Ā·) denote within-group z-score normalization, which subtracts the within-group mean and divides by the within-group standard deviation. We define the group-normalized verifier reward and group-normalized trajectory-level OPD score as R~(i)=zā”(R(i)),s~(i)=zā”(s(i)). split R^(i)&=z (R^(i) ),\\ s^(i)&=z (s^(i) ). split (6) Normalizing the two signals separately within each rollout group avoids directly comparing their raw values. For graded verifier rewards, R~(i) R^(i) retains the relative spacing of outcomes within the group. Binary verifier rewards, by contrast, indicate only whether each response succeeds or fails. GC-OPD operates on these group-normalized values rather than their induced rankings alone, allowing the magnitude of graded outcome differences to inform the residual. 4.3 TeacherāVerifier Disagreement Residual Given the group-normalized quantities in Equation 6, we define the signed residual as Ļ(i)=R~(i)ās~(i).Ļ^(i)= R^(i)- s^(i). (7) The sign of Ļ(i)Ļ^(i) indicates the direction of teacherāverifier disagreement. A positive value means that the group-normalized verifier reward for response (i)y^(i) exceeds its group-normalized OPD score, whereas a negative value indicates the opposite. Its magnitude reflects the difference between the two group-normalized assessments. GC-OPD uses this difference instead of directly adding the group-normalized verifier reward. Direct reward addition can reinforce a relative preference already expressed by OPD. In contrast, the residual vanishes when the two group-normalized assessments coincide, while its magnitude grows with their discrepancy. The residual therefore focuses calibration on differences between the verifier and OPD assessments. The residual also has the desired ordering under a strict disagreement between the verifier and OPD orderings. For two responses i and j in the same rollout group, let ĻR _R and Ļs _s denote the positive within-group standard deviations. If R(i)>R(j)R^(i)>R^(j) but s(i)<s(j)s^(i)<s^(j), then Ļ(i)āĻ(j)=R(i)āR(j)ĻRās(i)ās(j)Ļs>0.Ļ^(i)-Ļ^(j)= R^(i)-R^(j) _R- s^(i)-s^(j) _s>0. (8) Thus, when the verifier prefers response i but OPD prefers response j, the residual assigns a larger value to response i. Conversely, reversing the two preferences reverses the residual ordering. This verifier-consistent ordering directs the subsequent token-level correction toward the response with the higher verifier reward. If either signal has a near-zero within-group standard deviation, the corresponding normalization is not well-defined. We therefore set Ļ(i)=0Ļ^(i)=0 for all responses in that group, reducing GC-OPD to vanilla OPD for the group. 4.4 Relative-Advantage-based Credit Assignment The trajectory-level residual specifies the direction and magnitude of calibration for each response, but not how the correction should be distributed across tokens. RACA uses the token-level OPD advantages within each response as the allocation signal. We first normalize each tokenās OPD advantage relative to the response mean: ut(i)=At(i)ās(i)1T(i)āāv=1T(i)(Av(i)ās(i))2+Ļµ,u^(i)_t= A^(i)_t-s^(i) 1T^(i) _v=1^T^(i) (A^(i)_v-s^(i) )^2+Īµ, (9) where ĻµĪµ is a small positive constant for numerical stability. The resulting relative OPD advantage ut(i)u^(i)_t is positive for tokens whose OPD advantages exceed the response mean and negative for those below it. This normalization preserves the within-response ordering of the signed OPD advantages. Here, ut(i)u^(i)_t serves only as an allocation signal and should not be interpreted as token correctness or causal importance. RACA maps the relative OPD advantage to a positive, bounded token credit: ct(i)=1+tanhā”(ut(i)2).c^(i)_t=1+ ( u^(i)_t2 ). (10) This mapping is monotonic and satisfies ct(i)ā(0,2)c^(i)_tā(0,2). Tokens with larger relative OPD advantages therefore receive larger credits, while ut(i)=0u^(i)_t=0 yields unit credit ct(i)=1c^(i)_t=1. Because the credit is always positive, it scales but does not reverse the sign of the trajectory-level residual. The resulting GC-OPD advantage is Atā²(i)=At(i)+Ī²āct(i)āĻ(i),A (i)_t=A^(i)_t+Ī² c^(i)_tĻ^(i), (11) where Ī²ā„0Ī²ā„ 0 is the residual coefficient that controls the strength of calibration. The original token-level OPD advantage remains the base term, and Ī²āct(i)āĻ(i)Ī² c^(i)_tĻ^(i) is the token-level residual correction. GC-OPD substitutes Atā²(i)A (i)_t for At(i)A^(i)_t in the clipped token-level policy objective described in Section 3.1, changing the advantage construction without modifying the actor objective. Setting Ī²=0Ī²=0 recovers vanilla OPD. Given the OPD advantages and verifier rewards already produced by the training pipeline, GC-OPD adds only group-level aggregation, normalization, and elementwise token transformations. It requires no additional teacher or student forward pass. 5 Experiments We organize our experiments around three questions. First, does GC-OPD improve upon vanilla OPD under the same long-context training and evaluation setup? Second, does residualizing group-normalized verifier feedback improve upon adding that feedback directly? Third, does relative-advantage-based credit assignment (RACA) improve upon uniform residual allocation? 5.1 Experimental Setup Models. We use Qwen3-4B11 1 https://huggingface.co/Qwen/Qwen3-4B and Qwen3-8B22 2 https://huggingface.co/Qwen/Qwen3-8B as student models and Qwen3-30B-A3B-Thinking-250733 3 https://huggingface.co/Qwen/Qwen3-30B-A3B-Thinking-2507 as the teacher model. Both students operate in no-thinking mode. The teacher provides token-level log probabilities for OPD training. Training data. We construct the training set from GoLongRL [20] by retaining prompts no longer than 32K tokens. The resulting subset contains 9,527 prompts across nine task families. Table 1 summarizes their distribution and native verifier metrics. Three task families use binary rewards, while the remaining six use graded rewards. The two largest families, precise long-range retrieval and evidence-grounded reasoning, account for 82.9% of the training set. Table 1: Composition and reward interfaces of the 9,527-prompt GoLongRL subset after the 32K filter. āBā denotes a binary reward in 0,1\0,1\; āGā denotes a graded score in [0,1][0,1]. Task Family # Samples Native Verifier Precise Long-Range Retrieval 4,693 Exact Match (B) Evidence-Grounded Reasoning 3,204 Accuracy (B) Multi-Table Extraction 729 IoU (G) High-Recall Retrieval 540 Set F1 (G) Fragment Matching and Induction 148 Subset EM (G) Sequence Reconstruction 69 Pairwise Acc. (G) Numerical Reasoning 54 Math Verify (B) Graded Retrieval and Ranking 45 NDCG (G) Long-Document Summarization 45 ROUGE-L (G) Training configuration. All post-training methods use the same 100-step optimization budget. At each step, we sample a batch of 32 prompts and generate eight student responses per prompt. The maximum prompt length is 32,768 tokens, and response generation is capped at 10,240 tokens. Complete optimization and implementation details are provided in the supplementary material. Evaluation. We evaluate DocMath [35], Frames [14], MRCR [24], CorpusQA [19], and LBv1QA [3], covering numerical reasoning, multi-hop synthesis, multi-round co-reference, corpus aggregation, and long-context question answering, respectively. Following QwenLong-L1 [25], each student receives at most 120,000 input tokens and generates at most 8,192 tokens within a 131,072-token serving context. We extend the context window with YaRN [22] using a scaling factor of 4. All students remain in no-thinking mode and use the same evaluation prompts and decoding configuration. We report the score on each benchmark and the unweighted mean across the five benchmarks. Baselines and comparison settings. Raw denotes the corresponding officially released Qwen3 checkpoint evaluated before any post-training in our pipeline [30]. We compare GC-OPD with Raw, vanilla OPD [1], ExOPD [31], Uni-OPD [9], FiRe-OPD [16], and PowerOPD [34]. Vanilla OPD serves as the dense-distillation control. ExOPD introduces teacherāreference extrapolation, Uni-OPD performs outcome-conditioned margin calibration, FiRe-OPD filters trajectories and reweights token-level signals, and PowerOPD bounds token-level OPD rewards for stability. Daggered rows in Table 2 implement each methodās principal training-signal mechanism under our shared long-context setup. To isolate these mechanisms, the runs use the same student initialization, teacher, training data, rollout configuration, 100-step training budget, and evaluation pipeline as GC-OPD. Residual-coefficient selection. We select a single residual coefficient for both model scales using a fixed GoLongRL holdout. Before constructing the training subset, we reserve the first 256 examples in the ordered GoLongRL shards. Applying the same 32K-token limit leaves 231 validation examples with no overlap with the 9,527 training examples. Figure 3 reports the reward at the final checkpoint and the mean over the last five validation checkpoints, using one stochastic response per example. Both summaries select Ī²=0.10Ī²=0.10 for Qwen3-4B and Qwen3-8B, so we use this shared value in the main experiments. Because the holdout contains only the High-Recall Retrieval task family, we use it exclusively for coefficient selection rather than downstream evaluation. Figure 3: Held-out validation used to select the residual coefficient Ī². The fixed set contains 231 GoLongRL examples after 32K-token filtering and has no overlap with training. White squares show the step-100 reward. Circles show the mean over steps 60, 70, 80, 90, and 100. Both summaries select Ī²=0.10Ī²=0.10 at both model scales. Table 2: Qwen3-4B and Qwen3-8B results on five long-context benchmarks (%). āAvg.ā is the unweighted five-task mean. ā denotes an implementation of the methodās principal training-signal mechanism under the shared setup. Model Method Benchmark Avg. DocMath Frames MRCR CorpusQA LBv1QA Qwen3-4B Raw 29.08 43.63 26.58 22.02 6.99 46.20 OPD 39.31 49.37 30.95 26.90 32.22 57.10 ExOPDā 38.22 47.75 29.61 25.95 31.91 55.90 Uni-OPDā 38.53 51.88 29.98 27.93 27.36 55.50 PowerOPDā 38.88 49.63 30.10 28.36 32.52 53.80 FiRe-OPDā 39.50 49.37 31.31 28.20 32.52 56.10 GC-OPD (ours) 40.47 50.38 30.34 27.82 37.99 55.80 Qwen3-8B Raw 35.12 45.88 30.95 23.87 22.19 52.70 OPD 43.56 55.13 34.59 30.44 39.82 57.80 PowerOPDā 41.53 51.38 32.65 31.65 37.39 54.60 Uni-OPDā 43.41 54.13 34.59 25.25 42.86 60.20 ExOPDā 43.49 53.12 33.50 32.73 37.69 60.40 FiRe-OPDā 44.01 54.00 34.34 30.96 41.64 59.10 GC-OPD (ours) 44.65 55.50 34.59 31.10 43.77 58.30 5.2 Main Results GC-OPD improves performance across model scales. As shown in Table 2, GC-OPD achieves the highest five-task average for both students. Relative to vanilla OPD, it raises the average score from 39.31 to 40.47 for Qwen3-4B and from 43.56 to 44.65 for Qwen3-8B. It also achieves the highest average among the evaluated shared-setup implementations at both model scales. Competing methods often lead on individual benchmarks but do not maintain the strongest aggregate performance. GC-OPD therefore provides a more favorable balance across the heterogeneous evaluation suite rather than relying on a single task. Gains concentrate on structured reasoning and evidence aggregation. Relative to vanilla OPD, GC-OPD improves DocMath, MRCR, and CorpusQA for both students, with the largest gains on CorpusQA. These benchmarks emphasize structured reasoning and evidence aggregation. The concentration of gains is consistent with the intended role of response-level verifier calibration, although the table alone does not isolate task-specific mechanisms. Frames and LBv1QA show smaller or model-dependent changes, so the benefit is not uniform across tasks. Improvements over Raw confirm the effectiveness of the complete long-context post-training pipeline, whereas comparisons with vanilla OPD isolate the additional contribution of GC-OPD. The following ablations separately test the signed residual and its token-allocation rule. 5.3 Ablation Studies We next examine the two components introduced by GC-OPD: the signed residual and RACA token allocation. All ablations use independently trained Qwen3-8B students under the shared setup and 100-step budget. We first vary the signal added to vanilla OPD while holding the coefficient and token-allocation rule fixed. We then examine how the residual should be distributed across tokens. Detailed formulations of all ablation variants are provided in Appendix C.2. Residualization improves upon OPD-only and direct-reward controls. Table 3 isolates the signal added to vanilla OPD. The three augmented variants share RACA and Ī²=0.10Ī²=0.10, so only the added term changes: Atā²(i)=At(i)+0,Vanilla OPD,Ī²āct(i)āAt(i),Additional OPD,Ī²āct(i)āR~(i),Direct reward,Ī²āct(i)āĻ(i),GC-OPD.A (i)_t=A^(i)_t+ cases0,&Vanilla OPD,\\ Ī² c^(i)_tA^(i)_t,&Additional OPD,\\ Ī² c^(i)_t R^(i),&Direct reward,\\ Ī² c^(i)_tĻ^(i),&GC-OPD. cases (12) Additional OPD changes the average score only from 43.56 to 43.60, indicating that another OPD-derived term does not explain the improvement. Direct reward reaches 44.19, confirming that group-normalized verifier feedback complements dense OPD guidance. Replacing the direct reward with the signed residual further raises the average score to 44.65 under the same coefficient and token-allocation rule. This additional gain supports accounting for the relative preference already expressed by OPD rather than adding verifier feedback alone. Table 3: Response-level signal ablation on Qwen3-8B. All augmented variants use RACA with coefficient Ī²=0.10Ī²=0.10 and differ only in whether the added signal is At(i)A^(i)_t, R~(i) R^(i), or Ļ(i)=R~(i)ās~(i)Ļ^(i)= R^(i)- s^(i). āAvg.ā is the unweighted five-task mean, and āĪ ā is computed from the unrounded average relative to vanilla OPD. Variant Avg. DocMath Frames MRCR CorpusQA LBv1QA Vanilla OPD 43.56 55.13 34.59 30.44 39.82 57.80 ā Additional OPD 43.60 54.63 34.59 30.03 39.51 59.20 +0.04+0.04 Direct reward 44.19 55.13 34.59 30.40 40.73 60.10 +0.63+0.63 GC-OPD 44.65 55.50 34.59 31.10 43.77 58.30 +1.10+1.10 RACA improves residual allocation. Having established the contribution of the residual, we next examine its token allocation. Table 4 uses vanilla OPD as the no-residual anchor. All residual variants use Atā²(i)=At(i)+0.10āct(i)āĻ(i)A (i)_t=A^(i)_t+0.10c^(i)_tĻ^(i) and differ only in ct(i)c^(i)_t. Uniform sets ct(i)=1c^(i)_t=1. Absolute OPD assigns credit using ct(i)=clip[0,5]ā”(|At(i)|(T(i))ā1āāv=1T(i)|Av(i)|),c^(i)_t=clip_[0,5]\! ( A^(i)_t (T^(i))^-1 _v=1^T^(i) A^(i)_v ), (13) whereas RACA uses Equation 10. Uniform allocation raises the average score from 43.56 to 44.28, showing that the signed residual is useful even without token-dependent credit. RACA further raises it to 44.65, whereas Absolute OPD reaches only 43.93. Because Absolute OPD discards the sign of the OPD advantage, it can assign large credits to strongly negative OPD values. This result supports preserving the signed within-response ordering used by RACA. Table 4: Token-credit ablation on Qwen3-8B. All residual variants use the same signed residual with Ī²=0.10Ī²=0.10 and differ only in token allocation. āAvg.ā is the unweighted five-task mean, and āĪ ā is computed from the unrounded average relative to vanilla OPD. Token allocation Avg. DocMath Frames MRCR CorpusQA LBv1QA Vanilla OPD 43.56 55.13 34.59 30.44 39.82 57.80 ā Absolute OPD 43.93 54.63 35.56 29.53 41.34 58.60 +0.38+0.38 Uniform 44.28 54.12 37.01 30.64 40.12 59.50 +0.72+0.72 RACA 44.65 55.50 34.59 31.10 43.77 58.30 +1.10+1.10 5.4 Disagreement Across Task Families Beyond downstream performance, we use a fixed response set to test whether teacherāverifier disagreement extends beyond the two tasks in Figure 1. This diagnostic measures disagreement prevalence rather than trained-checkpoint performance. Teacherāverifier disagreement extends across task families. Figure 4 in Appendix A reports pairwise disagreement and top-1 mismatch for four task families containing at least 500 prompts, together with a prompt-macro aggregate over all nine families. The Multi-Table Extraction and High-Recall Retrieval rows aggregate the same frozen responses used in Figure 1 over prompts shorter than 32K. The additional task rows show that disagreement extends beyond these two tasks, while the variation across rows indicates that its prevalence remains task dependent. 6 Conclusion We introduced GC-OPD to address teacherāverifier disagreement in long-context OPD. GC-OPD forms a signed residual between group-normalized verifier rewards and trajectory-level OPD scores, then uses RACA to distribute it across tokens. This preserves dense OPD guidance while supporting binary and graded rewards without cross-task calibration. Diagnostics reveal stronger disagreement over longer inputs in two tasks and task-dependent prevalence across the training mixture. Across five benchmarks, GC-OPD improves vanilla OPD from 39.31 to 40.47 for Qwen3-4B and from 43.56 to 44.65 for Qwen3-8B. Ablations identify residual calibration as the main contributor, with an additional gain from RACA over uniform allocation. Overall, GC-OPD integrates response-level verification with dense token-level guidance. References [1] R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, Vol. 2024, p. 21246ā21263. Cited by: Ā§1, Ā§2, Ā§5.1. [2] Y. Bai, X. Lv, J. Zhang, Y. He, J. Qi, L. Hou, J. Tang, Y. Dong, and J. Li (2024) Longalign: a recipe for long context alignment of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, p. 1376ā1395. Cited by: Ā§2. [3] Y. Bai, X. Lv, J. Zhang, H. Lyu, J. Tang, Z. Huang, Z. Du, X. Liu, A. Zeng, L. Hou, et al. (2024) Longbench: a bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), p. 3119ā3137. Cited by: Ā§1, Ā§2, Ā§5.1. [4] Y. Bai, S. Tu, J. Zhang, H. Peng, X. Wang, X. Lv, S. Cao, J. Xu, L. Hou, Y. Dong, et al. (2025) Longbench v2: towards deeper understanding and reasoning on realistic long-context multitasks. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3639ā3664. Cited by: Ā§1. [5] G. Chen, M. Q. Shieh, and L. Bing (2026) LongRLVR: long-context reinforcement learning requires verifiable context rewards. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Ā§2. [6] O. Goldman, A. Jacovi, A. Slobodkin, A. Maimon, I. Dagan, and R. Tsarfaty (2024) Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 16576ā16586. Cited by: Ā§1, Ā§2. [7] Y. Gu, L. Dong, F. Wei, and M. Huang (2024) Minillm: knowledge distillation of large language models. In International Conference on Learning Representations, Vol. 2024, p. 32694ā32717. Cited by: Ā§1, Ā§2, Ā§3.1. [8] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: Ā§1. [9] W. Hou, S. Peng, W. Wang, Z. Ruan, Y. Zhang, Z. Zhou, M. Gao, Y. Chen, K. Wang, H. Yang, et al. (2026) Uni-opd: unifying on-policy distillation with a dual-perspective recipe. arXiv preprint arXiv:2605.03677. Cited by: Ā§1, Ā§2, Ā§5.1. [10] C. Hsieh, S. Sun, S. Kriman, S. Acharya, D. Rekesh, F. Jia, and B. Ginsburg (2024) RULER: whatās the real context size of your long-context language models?. In First Conference on Language Modeling, External Links: Link Cited by: Ā§1, Ā§2. [11] A. Kazemnejad, M. Aghajohari, E. Portelance, A. Sordoni, S. Reddy, A. Courville, and N. Le Roux (2025) VinePPO: refining credit assignment in RL training of LLMs. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 29557ā29590. External Links: Link Cited by: Ā§2. [12] M. Kim and S. Baek (2026) Explain in your own words: improving reasoning via token-selective dual knowledge distillation. In International Conference on Learning Representations, Vol. 2026, p. 103174ā103196. Cited by: Ā§2. [13] J. Ko, S. Kim, T. Chen, and S. Yun (2024) DistiLLM: towards streamlined distillation for large language models. In Proceedings of the 41st International Conference on Machine Learning, R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N. Oliver, J. Scarlett, and F. Berkenkamp (Eds.), Proceedings of Machine Learning Research, Vol. 235, p. 24872ā24895. External Links: Link Cited by: Ā§1. [14] S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui (2025) Fact, fetch, and reason: a unified evaluation of retrieval-augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 4745ā4759. Cited by: Ā§5.1. [15] Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, et al. (2026) Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Cited by: Ā§2. [16] Y. Li, L. Zheng, Y. Yu, W. Zhou, X. Zhong, X. Hu, J. Jin, H. Yuan, and T. Feng (2026) Filter, then reweight: rethinking optimization granularity in on-policy distillation. arXiv preprint arXiv:2606.02684. Cited by: Ā§1, Ā§2, Ā§5.1. [17] H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2024) Letās verify step by step. In International Conference on Learning Representations, Vol. 2024, p. 39578ā39601. Cited by: Ā§2. [18] N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang (2024) Lost in the middle: how language models use long contexts. Transactions of the association for computational linguistics 12, p. 157ā173. Cited by: Ā§1, Ā§2. [19] Z. Lu, C. Li, Y. Shi, W. Shen, M. Yan, and F. Huang (2026) CorpusQA: a 10 million token benchmark for corpus-level analysis and reasoning. arXiv preprint arXiv:2601.14952. Cited by: Ā§5.1. [20] M. Lv, T. Mei, T. Du, J. Chen, Z. Su, Z. Chen, Z. Wang, Z. Wu, R. Pan, R. Tang, et al. (2026) GoLongRL: capability-oriented long context reinforcement learning with multitask alignment. arXiv preprint arXiv:2605.19577. Cited by: Ā§3.2, Ā§5.1. [21] A. Modarressi, H. Deilamsalehy, F. Dernoncourt, T. Bui, R. A. Rossi, S. Yoon, and H. Schuetze (2025) NoLiMa: long-context evaluation beyond literal matching. In Proceedings of the 42nd International Conference on Machine Learning, A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu (Eds.), Proceedings of Machine Learning Research, Vol. 267, p. 44554ā44570. External Links: Link Cited by: Ā§1, Ā§2. [22] B. Peng, J. Quesnelle, H. Fan, and E. Shippole (2024) Yarn: efficient context window extension of large language models. In International Conference on Learning Representations, Vol. 2024, p. 31932ā31951. Cited by: Ā§3.2, Ā§5.1. [23] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: Ā§1, Ā§1. [24] K. Vodrahalli, S. Ontanon, N. Tripuraneni, K. Xu, S. Jain, R. Shivanna, J. Hui, N. Dikkala, M. Kazemi, B. Fatemi, et al. (2024) Michelangelo: long context evaluations beyond haystacks via latent structure queries. arXiv preprint arXiv:2409.12640. Cited by: Ā§5.1. [25] F. Wan, W. Shen, S. Liao, Y. Shi, C. Li, Z. Yang, J. Zhang, F. Huang, J. Zhou, and M. Yan (2025) Qwenlong-l1: towards long-context large reasoning models with reinforcement learning. arXiv preprint arXiv:2505.17667. Cited by: Ā§2, Ā§5.1. [26] P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui (2024) Math-shepherd: verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9426ā9439. Cited by: Ā§2. [27] S. Wang, G. Zhang, L. L. Zhang, N. Shang, F. Yang, D. Chen, and M. Yang (2026) Loongrl: reinforcement learning for advanced reasoning over long contexts. In International Conference on Learning Representations, Vol. 2026, p. 140078ā140097. Cited by: Ā§2. [28] Y. Wang, S. Lu, Y. Gu, P. Wang, Y. Yang, Z. Yan, C. Xie, J. Wu, and H. Yang (2026) Not all disagreement is learnable: token teachability in on-policy distillation. arXiv preprint arXiv:2605.26844. Cited by: Ā§2. [29] H. Xu, H. Wang, Y. Gao, J. Li, X. Zhang, and X. Yuan (2026) SG-opd: sign-gated on-policy distillation via sign-consistency gating and phased teacher sampling. arXiv preprint arXiv:2606.09304. Cited by: Ā§1, Ā§2. [30] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: Ā§5.1. [31] W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin (2026) Learning beyond teacher: generalized on-policy distillation with reward extrapolation. arXiv preprint arXiv:2602.12125. Cited by: Ā§1, Ā§2, Ā§5.1. [32] W. Yu, X. Li, Y. Zhao, X. Liu, R. Zhang, H. Wang, Y. Luo, C. H. Wu, G. Mittal, M. Fredrikson, et al. (2026) Multi-rollout on-policy distillation via peer successes and failures. arXiv preprint arXiv:2605.12652. Cited by: Ā§1, Ā§2. [33] J. Zhang, Z. Hou, X. Lv, S. Cao, Z. Hou, Y. Niu, L. Hou, Y. Dong, L. Feng, and J. Li (2025) Longreward: improving long-context large language models with ai feedback. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 3718ā3739. Cited by: Ā§2. [34] A. Zhao, J. Tong, Y. Fan, P. Nie, W. Li, and X. Shen (2026) PowerOPD: stabilizing on-policy distillation with bounded power transformation. arXiv preprint arXiv:2606.17199. Cited by: Ā§1, Ā§2, Ā§3.1, Ā§5.1. [35] Y. Zhao, Y. Long, H. Liu, R. Kamoi, L. Nan, L. Chen, Y. Liu, X. Tang, R. Zhang, and A. Cohan (2024) DocMath-eval: evaluating math reasoning capabilities of llms in understanding long and specialized documents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 16103ā16120. Cited by: Ā§5.1. [36] B. Zheng, X. Ma, Y. Liang, J. Ruan, X. Fu, K. Lin, B. Zhu, K. Zeng, and X. Cai (2026) Scope: signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. arXiv preprint arXiv:2604.10688. Cited by: Ā§1, Ā§2. [37] Q. Zou, Y. Li, T. Liu, B. He, and W. Wong (2026) Reward-weighted on-policy distillation with an open property-equivalence verifier for nl-to-sva generation. arXiv preprint arXiv:2605.13501. Cited by: Ā§2. Appendix A Task-Conditioned Disagreement Analysis The main paper analyzes how teacherāverifier disagreement changes with prompt length in two representative tasks. Here, we complement that analysis with a task-conditioned view of four GoLongRL task families containing at least 500 prompts. For Multi-Table Extraction and High-Recall Retrieval, we aggregate the same frozen responses used in Figure 1 over prompts shorter than 32K, without separating the two length ranges. Each of the other task rows is likewise computed from eight fixed Qwen3-8B responses per prompt. Overall pools all nine task families by giving each informative prompt equal weight. Within each informative rollout group, the pairwise disagreement rate is the fraction of response pairs with distinct verifier rewards for which the OPD and verifier orderings disagree. We average this rate across prompts. The top-1 mismatch rate is the fraction of these groups in which the response with the highest trajectory-level OPD score does not attain the groupās highest verifier reward. Their variation across task families complements the length-conditioned trends in the main paper and motivates comparing the two assessments locally within each rollout group. Because the responses are fixed, the analysis characterizes signal disagreement rather than task-specific training gains or causal effects of task type or context length. Figure 4: Task-conditioned teacherāverifier disagreement in four GoLongRL task families. Bars report prompt-macro pairwise disagreement rates and top-1 mismatch rates over rollout groups with at least two distinct verifier rewards. Overall pools all nine task families. Appendix B GC-OPD Implementation Details Using the notation of the main paper, At(i)A^(i)_t is the detached token-level OPD advantage, s(i)=(T(i))ā1āātAt(i)s^(i)=(T^(i))^-1 _tA^(i)_t is its trajectory-level OPD score, and R(i)R^(i) is the verifier reward. The main paper defines the compact pre-clipping update Atā²(i)A (i)_t. Here, T(i)T^(i) is the number of valid tokens in response i, G is the rollout-group size, and Ī² is the residual coefficient in the main update. This section specifies the numerical guards and update order used in the completed runs. The thresholds ĻG _G and ĻT _T guard group-level and token-level standard deviations, respectively, and amaxa_ is the final advantage-clipping bound. For hāR,shā\R,s\, let Ī¼h _h and Ļh _h denote the population mean and standard deviation over the G responses. The implemented residual is Ļ(i)=R(i)āĪ¼RĻRās(i)āĪ¼sĻs,ĻR,Ļs>ĻG,0,otherwise.Ļ^(i)= cases R^(i)- _R _R- s^(i)- _s _s,& _R, _s> _G,\\ 0,&otherwise. cases (14) If either group-level signal has negligible variation, the residual is zero and the update reduces to vanilla OPD. The token credit uses an analogous guard. Within response i, let ĻA(i) _A^(i) denote the population standard deviation of its valid token advantages. The guarded RACA credit is ut(i) u^(i)_t =At(i)ās(i)ĻA(i),T(i)ā„2āandāĻA(i)>ĻT,0,otherwise, = cases A^(i)_t-s^(i) _A^(i),&T^(i)ā„ 2\ and\ _A^(i)> _T,\\ 0,&otherwise, cases (15) ct(i) c^(i)_t =1+tanhā”(ut(i)2), =1+ \! ( u^(i)_t2 ), (16) Here, ut(i)u^(i)_t is the tokenās OPD advantage relative to the response mean, measured in units of the within-response standard deviation. It describes relative OPD support rather than token correctness. Thus negligible token variation gives unit credit. After forming Atā²(i)A (i)_t, we clip it to [āamax,amax][-a_ ,a_ ] before the policy update. Table 6 reports ĻG _G, ĻT _T, and amaxa_ . These Given the OPD advantages and verifier rewards, these operations require no additional teacher or student forward pass. Algorithm 1 Group-Calibrated On-Policy Distillation 0: Prompt batch ā¬B; trainable student policy ĻĪø _Īø; teacher ĻT _T; verifier V; group size G; residual coefficient Ī²; thresholds ĻG,ĻT _G, _T; clipping bound amaxa_ 0: Updated student policy ĻĪø _Īø 1: Set ĪøoldāĪø _oldāĪø and freeze ĻĪøold _ _old for rollout generation 2: for each prompt āā¬x do 3: Sample ā”()=(i)i=1GG(x)=\y^(i)\_i=1^G from ĻĪøold(ā ā£) _ _old(Ā· ) 4: for i=1,ā¦,Gi=1,ā¦,G do 5: R(i)āVā”(,(i))R^(i)ā V(x,y^(i)) 6: Compute detached At(i)A^(i)_t under ĻT _T and ĻĪøold _ _old for t=1,ā¦,T(i)t=1,ā¦,T^(i) 7: s(i)ā(T(i))ā1āāt=1T(i)At(i)s^(i)ā(T^(i))^-1 _t=1^T^(i)A^(i)_t 8: end for 9: Compute population statistics (Ī¼R,ĻR)( _R, _R) and (Ī¼s,Ļs)( _s, _s) over the G responses 10: if ĻR>ĻGā§Ļs>ĻG _R> _G _s> _G then 11: Ļ(i)āR(i)āĪ¼RĻRās(i)āĪ¼sĻsĻ^(i)ā R^(i)- _R _R- s^(i)- _s _s for i=1,ā¦,Gi=1,ā¦,G 12: else 13: Ļ(i)ā0Ļ^(i)ā 0 for all i 14: end if 15: for each response i do 16: Compute ĻA(i) _A^(i) from At(i)t=1T(i)\A^(i)_t\_t=1^T^(i) 17: if T(i)ā„2ā§ĻA(i)>ĻT^(i)ā„ 2 _A^(i)> _T then 18: ut(i)āAt(i)ās(i)ĻA(i)u^(i)_tā A^(i)_t-s^(i) _A^(i) for t=1,ā¦,T(i)t=1,ā¦,T^(i) 19: ct(i)ā1+tanhā”(ut(i)/2)c^(i)_tā 1+ \! (u^(i)_t/2 ) for t=1,ā¦,T(i)t=1,ā¦,T^(i) 20: else 21: ut(i)ā0u^(i)_tā 0 and ct(i)ā1c^(i)_tā 1 for t=1,ā¦,T(i)t=1,ā¦,T^(i) 22: end if 23: Atā²(i)āAt(i)+Ī²āct(i)āĻ(i)A (i)_tā A^(i)_t+Ī² c^(i)_tĻ^(i) for t=1,ā¦,T(i)t=1,ā¦,T^(i) 24: A^t(i)āclipā”(Atā²(i),āamax,amax) A^(i)_t (A (i)_t,-a_ ,a_ ) for all t 25: end for 26: end for 27: Update ĻĪø _Īø using the same policy surrogate as vanilla OPD, with A^t(i) A^(i)_t as the token-level advantages Appendix C Experimental Details C.1 Models, Data, and Training Configuration We initialize Qwen3-4B and Qwen3-8B students from their official checkpoints and use Qwen3-30B-A3B-Thinking-2507 as the teacher. Students generate in no-thinking mode, and the teacher scores the sampled response tokens without generating separate responses. The 32K-filtered GoLongRL training file contains 9,527 prompts. Table 5 reports its prompt-length distribution, complementing the task-family and verifier statistics in the main paper. The median, 90th percentile, and maximum lengths are 9,923, 26,940, and 32,766 tokens. Table 5: Prompt-length distribution of the exact 32K-filtered training file. Prompt length Samples ā¤2ā¤ 2K 2,222 2ā8K 1,958 8ā16K 2,155 16ā24K 1,914 24ā32K 1,278 Total 9,527 Table 6 records the common configuration. Raw rows receive no training; every trained method uses this configuration and is evaluated at its final checkpoint. Table 6: Shared training configuration and GC-OPD settings. Setting Value Training horizon 100 steps Prompt batch size 32 Responses per prompt 8 Maximum prompt length 32,768 Training response cap 10,240 Learning rate 10ā610^-6 Warmup 0 Weight decay 0.01 Policy mini-batch size 4 PPO epochs 1 PPO clip ratio 0.2 Loss aggregation Token mean Rollout temperature 1 Rollout top-p 1 Precision bfloat16 Random seed 42 Residual coefficient Ī²=0.10Ī²=0.10 Group std. threshold ĻG=10ā6 _G=10^-6 Token std. threshold ĻT=10ā6 _T=10^-6 Final advantage clip [ā10,10][-10,10] (amax=10a_ =10) Compute resources. Each training run used eight 80-GB NVIDIA H800 or H100 GPUs. Rollout tensor parallelism and actor/reference sequence parallelism were both set to 8. C.2 Ablation Configurations The ablations in the main paper use independent Qwen3-8B training runs under the shared configuration in Table 6. All runs use the same student initialization, teacher, filtered training data, optimizer, rollout budget, and 100-step training horizon. They are evaluated at step 100 with the same five-benchmark pipeline as the main comparison. Added-signal ablation. Vanilla OPD provides the common anchor for this comparison. The three augmented variants use the same RACA credit and a scalar coefficient of 0.10, while changing the signal added to the vanilla OPD advantage. Their updates before final advantage clipping are Atā²(i) A (i)_t =At(i) =A^(i)_t (vanilla OPD), (vanilla OPD), (17) Atā²(i) A (i)_t =At(i)+Ī²āct(i)āAt(i),Ī²=0.10 =A^(i)_t+Ī² c^(i)_tA^(i)_t, Ī²=0.10 (Additional OPD), (Additional OPD), (18) Atā²(i) A (i)_t =At(i)+Ī²āct(i)āR~(i),Ī²=0.10 =A^(i)_t+Ī² c^(i)_t R^(i), Ī²=0.10 (Direct reward), (Direct reward), (19) Atā²(i) A (i)_t =At(i)+Ī²āct(i)āĻ(i),Ī²=0.10 =A^(i)_t+Ī² c^(i)_tĻ^(i), Ī²=0.10 (GC-OPD). (GC-OPD). (20) The comparison holds the original OPD signal, RACA allocation, and coefficient fixed. Additional OPD adds another credit-modulated OPD term, Direct reward uses the group-normalized verifier reward R~(i) R^(i), whereas GC-OPD subtracts the group-normalized trajectory OPD score and uses Ļ(i)=R~(i)ās~(i)Ļ^(i)= R^(i)- s^(i). It therefore separates the effect of adding another OPD-derived term from adding verifier feedback, and tests whether accounting for the relative preference already expressed by OPD improves upon adding verifier feedback directly. Token-allocation ablation. This comparison fixes the signed residual and the same Ī²=0.10Ī²=0.10 used by the main configuration, so every calibrated variant uses Atā²(i)=At(i)+0.10āct(i)āĻ(i)A (i)_t=A^(i)_t+0.10c^(i)_tĻ^(i). Only the token credit ct(i)c^(i)_t changes. Uniform allocation sets ct(i)=1c^(i)_t=1. Absolute OPD is a token-allocation control, not an alternative trajectory-level signal. It assigns credit in proportion to the magnitude of the tokenwise OPD advantage: ct(i)=clip[0,5]ā”(|At(i)|(T(i))ā1āāv=1T(i)|Av(i)|),c^(i)_t=clip_[0,5]\! ( A^(i)_t (T^(i))^-1 _v=1^T^(i) A^(i)_v ), (21) The upper bound of 5 limits extreme token weights, and unit credit is used when the denominator is numerically negligible. RACA uses Equation 16. Thus this comparison holds the trajectory-level residual and its coefficient fixed while isolating how the correction is distributed across response tokens. C.3 Baselines under the Shared Setup All trained baselines share the student initialization, teacher, filtered training data, rollout budget, optimizer, training horizon, and clipped surrogate. We hold these factors fixed to isolate each methodās principal training-signal mechanism. The comparisons should therefore be interpreted as controlled mechanism comparisons under a shared setup rather than exact reproductions of the original experimental recipes. Each method generates its own trajectories on-policy. Let pT,tp_T,t, pS,tp_S,t, and p0,tp_0,t denote the teacher, current-student, and frozen student-base probabilities of the sampled token. The paragraphs below record only method-specific implementation choices. To avoid clutter, the response index i is suppressed in baseline-specific token formulas unless it is needed explicitly. Raw checkpoint. Raw evaluates the official Qwen3-4B or Qwen3-8B checkpoint without additional training, using the same downstream evaluation pipeline. OPD. OPD uses AtOPD=logā”pT,tālogā”pS,tA_t^OPD= p_T,t- p_S,t as the detached token advantage. The native verifier is executed by the shared pipeline, but its terminal score is not added to this advantage. Equivalently, vanilla OPD is recovered from GC-OPD by setting Ī²=0Ī²=0, and we use this convention throughout the supplement. ExOPD. ExOPD replaces the teacher log-probability target with an extrapolated target anchored at the corresponding frozen student-base checkpoint: AtEx=Ī»ālogā”pT,t+(1āĪ»)ālogā”p0,tālogā”pS,t.A_t^Ex=Ī» p_T,t+(1-Ī») p_0,t- p_S,t. (22) Our runs use Ī»=1.25Ī»=1.25 and the corresponding frozen Qwen3 initialization as p0p_0, following the default student-base-reference formulation. The reference contributes only to the detached token target and is not used as a KL regularizer. In the ExOPD paper, āreward correctionā denotes the optional replacement of p0p_0 with the teacherās pre-RL base model. This reference-based variant is distinct from the verifier reward used by GC-OPD and is not used here. Uni-OPD. Uni-OPD combines token-level teacher supervision with response-level outcome separation. Within the eight responses for one prompt, responses with positive verifier reward form the correct set and the others form the incorrect set. Let sĀÆ+ s^+ and sĀÆā s^- be the mean response-level OPD scores of these two sets. If both sets exist, the required shift is d=maxā”0,Ī“ā(sĀÆ+āsĀÆā).d= \0,\,Ī“-( s^+- s^-)\. (23) With bidirectional calibration, d/2d/2 is added to every token of a correct response and subtracted from every token of an incorrect response. Groups lacking either set receive no margin shift. For a controlled comparison, we isolate Uni-OPDās outcome-guided margin calibration and omit its offline and online data-balancing components. This keeps the training data and rollout budget consistent across methods. The implementation uses group scope, mean statistics, Ī“=0.4Ī“=0.4, and bidirectional shifts. Following the released implementation, the resulting group-level shift is added to the original tokenwise OPD advantages. We also retain the released configurationās teacherāstudent log-probability-gap mask with a threshold of 10 as a training-stability setting. PowerOPD. PowerOPD changes the geometry of the token signal from a log-probability gap to a bounded probability-power difference: AtPower=sgā”[pT,tĪ±āpS,tĪ±],Ī±=100,A_t^Power=sg\! [p_T,t^Ī±-p_S,t^Ī± ], Ī±=100, (24) where sgsg denotes stop-gradient. We use this bounded signal as the method-specific token advantage within the shared clipped surrogate. Both model scales use Ī±=100Ī±=100. FiRe-OPD. Following the released implementation, FiRe-OPD computes the length-normalized teacher score q(i)=(T(i))ā1āātlogā”pT,t(i)q^(i)=(T^(i))^-1 _t p_T,t^(i) for each trajectory and applies the actor-micro-batch percentile filter. Token weights use teacher confidence and student confusion, cT,t=1āHT,tmaxā”HT,cS,t=HS,tmaxā”HS,c_T,t=1- H_T,t H_T, c_S,t= H_S,t H_S, (25) clipped to [0,1][0,1]. The OPD advantage is multiplied by wt=(1+Ī±FācT,t)ā(1+Ī²FācS,t)w_t=(1+ _Fc_T,t)(1+ _Fc_S,t) after normalizing w to mean one over valid tokens. Filtered trajectories contribute no policy loss. Both runs use a 20th-percentile trajectory cutoff and Ī±F=Ī²F=1 _F= _F=1. The percentile, entropy maxima, and weight normalization are recomputed within each actor micro-batch supplied to the loss. The actor starts from four-response policy mini-batches and may split them further under dynamic token budgeting. C.4 Evaluation Implementation Table 7 records the evaluation units and aggregation rules. The paragraphs below specify the per-example scorers. Table 7: Evaluation units and aggregation for the five reported long-context benchmark columns. Benchmark Evaluation units Reported aggregation DocMath Four splits: 200/100/200/300 Per-item max(rule, judge); four-split aggregate FRAMES 824 questions Mean max(CEM, judge) accuracy MRCR 0ā128K Ć 2, 4, 8 needles Mean prefix-gated sequence ratio CorpusQA 0ā128K Ć four domains Overall example accuracy LBv1QA Five subsets Ć 200 Macro mean of five subset accuracies DocMath. The evaluator extracts the final answer and applies both the rule-based numerical/equation checker and a semantic-equivalence judge. The per-example score is the maximum of these binary decisions. Scores are then aggregated over the four simple/complex and short/long splits shown in Table 7. FRAMES. After removing any hidden reasoning segment, the scorer applies cover exact match (CEM). CEM lowercases text, removes punctuation and articles, normalizes comma-separated numbers, and requires the target as a word-bounded span in the answer. A sample is correct if either CEM or the semantic-equivalence judge accepts it. MRCR. Every gold response begins with a sample-specific random prefix. A prediction without the exact prefix receives zero. Otherwise, the scorer removes the prefix and computes Python character-level SequenceMatcher similarity between the prediction and reference. The aggregate includes the 2-, 4-, and 8-needle variants across context lengths. CorpusQA. We use the 0ā128K file covering Chinese finance, English finance, English education, and English real estate. The evaluator extracts text following the prescribed answer marker when present, then obtains a binary semantic-equivalence judgment against the generated reference. The reported column is overall example accuracy across the four domains. LongBench v1 QA. The evaluator extracts the final occurrence of āTherefore, the answer isā and falls back to the last non-empty line when the marker is absent. It compares the extracted answer with every accepted reference and takes the maximum binary semantic judgment. The reported score is the macro mean over NarrativeQA, Qasper, HotpotQA, 2WikiMultihopQA, and MuSiQue. Student serving accepts at most 120,000 input tokens and uses a 131,072-token model length with YaRN factor 4 and target length 131,072. The completed evaluation runs cap generation at 8,192 tokens. Inputs that exceed the serving budget are middle truncated so that both the beginning and end are retained. Students remain in no-thinking mode, and no method receives a method-specific evaluation prompt or decoding override. Semantic judging uses Qwen3-30B-A3B-Instruct-2507 with a 32,768-token context and a 2,048-token output cap. The reported aggregate is the naive average of the five benchmark scores. Appendix D Auditable Long-Context Case Study To make the GC-OPD signal computation concrete, we present one example from GoLongRL. Its Qwen3-8B no-thinking prompt contains 17,265 tokens and asks which matrix receives the largest relative improvement from parallel NUMA optimization: A rajat31, B HV15R, C cage15, or D ldoor. The datasetās native label is B. The contextual observations reproduced in Figure 5 support this label, but do not by themselves constitute a direct speedup measurement. This case is an illustrative audit of the signal computation rather than additional quantitative evidence. We independently reroll the example using the official Qwen3-8B base checkpoint and Qwen3-30B-A3B-Thinking-2507 as the teacher. We sample eight responses with temperature 1, top-p=1p=1, top-k=ā1k=-1, seed 42, and a 10,240-token response cap. Table 8 reports representative token signals recomputed offline with Ī²=0.10Ī²=0.10 and amax=10a_ =10. In the table, the response index is suppressed within each response block, and ĪāAt=Ī²āctāĻ A_t=Ī² c_tĻ denotes the additive residual correction before clipping. Figure 5: Illustrative fixed-rollout visualization for the 17,265-token case. It reproduces the selected context evidence and all eight independently rerolled student responses; the native label is B. Table 8: Representative token-level RACA recalculations for the correct-B and incorrect-D responses in Figure 5 under the final GC-OPD setting. Values are rounded; ranking checks use unrounded values. Correct B response, ĻB=+3.464102 _B=+3.464102 Incorrect D response, ĻD=ā1.154701 _D=-1.154701 Token AtA_t utu_t ctc_t ĪāAt A_t A^t A_t Token AtA_t utu_t ctc_t ĪāAt A_t A^t A_t [ -4.88599 -1.85923 0.26959 0.09339 -4.79260 [ -4.88599 -1.98073 0.24248 -0.02800 -4.91399 Answer -0.02188 0.83034 1.39285 0.48250 0.46062 Answer -0.02188 0.68817 1.33112 -0.15370 -0.17558 ] -0.64095 0.48803 1.23928 0.42930 -0.21165 ] -0.64095 0.34849 1.17250 -0.13539 -0.77634 B -1.91057 -0.21400 0.89341 0.30949 -1.60108 D -0.66057 0.33773 1.16728 -0.13479 -0.79535 EOS -0.15839 0.75486 1.36047 0.47128 0.31290 EOS -0.17101 0.60634 1.29421 -0.14944 -0.32046