Paper deep dive
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation
Heng Li, Yong Zhang, Ning Cheng, Zhigen Li, Yun Zhu, Yanmeng Wang, Shaojun Wang, Jing Xiao
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/9/2026, 8:24:38 AM
Summary
The paper introduces PIVOT, a dynamic transition framework for vertical-domain few-shot distillation that adaptively routes samples between On-Policy Distillation (OPD) and GRPO-based reinforcement learning based on teacher-evaluated sequence perplexity. This approach overcomes the limitations of globally fixed transition schedules by allowing low-perplexity samples to enter reward-driven refinement earlier while keeping high-perplexity samples under teacher-guided acquisition longer. Experiments on Banking77 and HWU64 demonstrate that PIVOT outperforms baselines like continued OPD and globally synchronized OPD-to-GRPO in both accuracy and training stability.
Entities (8)
Relation Signals (8)
PiVOT → evaluatedon → Banking77
confidence 95% · Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD
PiVOT → evaluatedon → HWU64
confidence 95% · Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD
PiVOT → uses → Sequence Perplexity
confidence 95% · PIVOT (Perplexity-Informed Transition Optimization), a dynamic transition framework that routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity.
PiVOT → combines → GRPO
confidence 93% · PIVOT progressively transitions samples from OPD to GRPO according to teacher-evaluated sequence perplexity
PiVOT → combines → On-Policy Distillation
confidence 93% · PIVOT progressively transitions samples from OPD to GRPO according to teacher-evaluated sequence perplexity
PiVOT → outperforms → OPD->GRPO
confidence 90% · PIVOT consistently outperforms continued OPD and globally synchronized OPD→GRPO baselines
Qwen2.5-0.5B-Instruct → usedas → Student
confidence 85% · we use Qwen2.5-0.5B-Instruct as the student model
Qwen2.5-7B-Instruct → usedas → Teacher
confidence 85% · and Qwen2.5-7B-Instruct as the teacher model.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Vertical-domain few-shot classification remains challenging for small language models, as limited supervision makes it difficult to acquire domain-specific decision knowledge. On-Policy Distillation (OPD) can improve teacher-guided adaptation by supervising student-generated rollouts, while GRPO-based reinforcement learning can further refine downstream predictions. However, existing KD-to-RL pipelines typically rely on globally fixed transition schedules, ignoring that different samples may require different amounts of teacher-guided acquisition before reward-driven refinement. We propose PIVOT (Perplexity-Informed Transition Optimization), a dynamic transition framework that routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity. PIVOT moves low-perplexity samples to GRPO for reward-driven refinement while keeping high-perplexity samples under OPD for continued domain knowledge acquisition. Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD$\rightarrow$GRPO baselines under the same number of post-warm-up student optimization steps, achieving stronger downstream performance and more stable training dynamics.
Tags
Links
- Source: https://arxiv.org/abs/2610.11167v1
- Canonical: https://arxiv.org/abs/2610.11167v1
Trouble viewing inline? Open PDF directly →
Full Text
50,230 characters extracted from source content.
Expand or collapse full text
PIVOT: Perplexity-Informed KD-to-RL Transition Scheduling for Vertical-Domain Few-Shot Distillation Heng Li Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Affiliation: University of Science and Technology of China Email: zhangyong203@pingan.com.cn Yong Zhang Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Email: chengning211@pingan.com.cn Ning Cheng Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Zhigen Li Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Yun Zhu Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Yanmeng Wang Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Shaojun Wang Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Jing Xiao Affiliation: Ping An Technology (Shenzhen) Co., Ltd., China Abstract Vertical-domain few-shot classification remains challenging for small language models, as limited supervision makes it difficult to acquire domain-specific decision knowledge. On-Policy Distillation (OPD) can improve teacher-guided adaptation by supervising student-generated rollouts, while GRPO-based reinforcement learning can further refine downstream predictions. However, existing KD-to-RL pipelines typically rely on globally fixed transition schedules, ignoring that different samples may require different amounts of teacher-guided acquisition before reward-driven refinement. We propose PIVOT (Perplexity-Informed Transition Optimization), a dynamic transition framework that routes samples between OPD and GRPO according to teacher-evaluated sequence perplexity. PIVOT moves low-perplexity samples to GRPO for reward-driven refinement while keeping high-perplexity samples under OPD for continued domain knowledge acquisition. Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD→ baselines under the same number of post-warm-up student optimization steps, achieving stronger downstream performance and more stable training dynamics. †footnotetext: † Equal contribution.†footnotetext: * Corresponding author.†footnotetext: This work was done during Heng Li’s internship at Ping An Technology. 1 Introduction In vertical-domain few-shot classification tasks, small language models often struggle to adapt under limited supervision due to insufficient domain-specific knowledge. This challenge is particularly severe in professional domains such as finance, where downstream prediction depends heavily on specialized terminology and domain semantics. As a result, effective adaptation increasingly relies on transferring knowledge from expert teacher models through distillation-based optimization Hinton et al. (2015); Sanh et al. (2020). Among existing approaches, On-Policy Distillation (OPD) has emerged as an effective paradigm for teacher-guided adaptation by aligning supervision with student-generated trajectories during optimization, thereby improving exposure alignment during autoregressive generation Agarwal et al. (2024b); Gu et al. (2024). Recent studies have further explored how knowledge distillation and reinforcement learning should be coordinated during training. GRPO has also been applied to intent detection with chain-of-thought reasoning and reward-based curriculum sampling, improving generalization beyond supervised fine-tuning Feng et al. (2025). RLKD and KDRL Xu et al. (2026a); Xu et al. (2025) show that reinforcement learning can improve downstream decision refinement beyond teacher-guided imitation. CoDistill-GRPO Kwon et al. (2026) and related hybrid optimization frameworks Li et al. (2026a); Ding (2026) further demonstrate that jointly coordinating distillation and RL objectives can improve optimization efficiency and rollout quality. Meanwhile, recent OPD studies have found that prolonged distillation may introduce unreliable supervision or unstable training dynamics once student rollouts move beyond regions reliably supported by the teacher Liu et al. (2026). Existing approaches therefore explore uncertainty-aware weighting, entropy-guided optimization, rollout-group coordination, and curriculum-based strategies to stabilize OPD optimization Zheng et al. (2026); Jin et al. (2026); Yu et al. (2026); Xu et al. (2026b). Despite these advances, existing methods largely coordinate KD and RL through globally fixed schedules or shared optimization objectives, without explicitly modeling when the learning signal should transition from distillation to reinforcement learning. This limitation is especially important in vertical-domain few-shot settings, where OPD plays a different role from conventional refinement-oriented distillation. Since the student initially lacks sufficient domain-specific semantic priors, OPD must first support domain knowledge acquisition before reinforcement learning can effectively refine downstream predictions. As training progresses, however, the marginal benefit of continued OPD gradually diminishes, suggesting that the dominant learning signal should shift from teacher-guided acquisition to reward-driven refinement. A key challenge is that this KD-to-RL transition point varies substantially across samples. We observe that samples with higher teacher-side perplexity often continue to benefit from prolonged OPD optimization, indicating that they still require teacher-guided acquisition. In contrast, samples with lower teacher-side perplexity typically exhibit limited improvement under continued distillation and can enter reward-driven refinement earlier. These observations suggest that teacher-evaluated sequence perplexity provides a useful signal for estimating sample-level acquisition status. Consequently, KD-to-RL coordination should be governed by sample-level optimization dynamics rather than globally synchronized training stages. Motivated by these observations, we propose PIVOT (Perplexity-Informed Transition Optimization), a dynamic KD-to-RL transition framework that adaptively coordinates OPD-based distillation and GRPO-based refinement Shao et al. (2024b). PIVOT progressively transitions samples from OPD to GRPO according to teacher-evaluated sequence perplexity: samples with more stable teacher-aligned rollouts enter GRPO refinement earlier, while more difficult samples remain under OPD for continued domain knowledge acquisition. In this way, PIVOT avoids both premature reinforcement learning on under-acquired samples and inefficient over-distillation on samples that are already ready for refinement. Experiments on Banking77 Casanueva et al. (2020) and HWU64 Liu et al. (2019) show that PIVOT consistently outperforms continued OPD and globally synchronized OPD→ baselines under the same number of post-warm-up student optimization steps. Further analysis demonstrates that progressive sample-level transition yields more stable optimization dynamics and stronger downstream refinement effectiveness than fixed two-stage transition schedules. Our contributions are summarized as follows: • We show that prolonged teacher-guided distillation does not necessarily yield better downstream GRPO refinement under vertical-domain few-shot adaptation. • We propose PIVOT, a progressive sample-level KD-to-RL transition framework that adaptively schedules OPD and GRPO according to teacher-evaluated sequence perplexity. • Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD→ under the same optimization steps while yielding more stable optimization dynamics. Figure 1: Overview of PIVOT. PIVOT estimates sample-level acquisition states using teacher-side perplexity over on-policy student rollouts, and progressively performs asynchronous KD-to-RL transition by routing low-perplexity samples to GRPO refinement while retaining high-perplexity samples under continued OPD acquisition. 2 Method 2.1 Domain Knowledge Acquisition and Reward-Driven Refinement To facilitate domain knowledge acquisition, we first adopt On-Policy Distillation (OPD), which continuously aligns teacher supervision with the student’s evolving rollout distribution. Specifically, for input xix_i, the student first generates a response: yi∼pθ(⋅|xi),y_i p_θ(·|x_i), (1) and the student is optimized to match the teacher distribution along on-policy generated trajectories: ℒOPD=xi∼,yi∼pθ(⋅|xi)[∑t=1|yi|DKL(pT,t∥pθ,t)],L_OPD=E_x_i ,\;y_i p_θ(·|x_i) [ _t=1^|y_i|D_KL\! (p_T,t\,\|\,p_θ,t ) ], (2) where pT,t=pT(⋅|xi,yi,<t),pθ,t=pθ(⋅|xi,yi,<t).p_T,t=p_T(·|x_i,y_i,<t), p_θ,t=p_θ(·|x_i,y_i,<t). After OPD-based acquisition, we further introduce GRPO refinement using a group-relative policy optimization objective. For each input xix_i, we sample a group of responses from the old policy pθoldp_ _old: yi,jj=1G∼pθold(⋅|xi),\y_i,j\_j=1^G p_ _old(·|x_i), where G denotes the group size. Each response receives a scalar reward Ri,jR_i,j, and the corresponding group-normalized advantage is computed as Ai,j=Ri,j−μiσi+δ,A_i,j= R_i,j- _i _i+δ, where μi _i and σi _i denote the mean and standard deviation of rewards within the sampled response group for input xix_i, respectively, and δ is a small constant for numerical stability. For each generated token, the policy ratio is defined as ri,j,t(θ)=pθ(yi,j,t∣xi,yi,j,<t)pθold(yi,j,t∣xi,yi,j,<t).r_i,j,t(θ)= p_θ(y_i,j,t x_i,y_i,j,<t)p_ _old(y_i,j,t x_i,y_i,j,<t). The clipped token-level objective is then given by ℓi,j,tGRPO=min(CLOSE _i,j,t^GRPO= ( ri,j,tAi,j, r_i,j,tA_i,j, (3) OPENclip(ri,j,t,1−ϵ,1+ϵ)Ai,j). (r_i,j,t,1-ε,1+ε)A_i,j ). The GRPO loss is defined as: ℒGRPO=−xi,yi,jj=1G[1G∑j=1G∑t=1|yi,j|ℓi,j,tGRPO], _GRPO=-E_x_i,\y_i,j\_j=1^G [ 1G _j=1^G _t=1^|y_i,j| _i,j,t^GRPO ], (4) where xi∼,yi,jj=1G∼pθold(⋅|xi).x_i , \y_i,j\_j=1^G p_ _old(·|x_i). GRPO enables stable relative reward optimization over grouped candidate responses without requiring an additional value model. In our framework, OPD and GRPO serve complementary optimization roles. OPD supports domain knowledge acquisition through rollout expansion, whereas GRPO refinement further improves response quality once the marginal acquisition gain from continued distillation gradually diminishes. This distinction motivates our asynchronous KD-to-RL transition scheduling framework. 2.2 Heterogeneous Acquisition Dynamics and Acquisition-State Estimation Different samples exhibit substantially different acquisition dynamics throughout optimization. Some samples rapidly approach rollout-coverage saturation, whereas others continue benefiting from prolonged OPD acquisition. Consequently, the optimal KD-to-RL transition point varies substantially across samples. Samples approaching rollout-coverage saturation are better suited for earlier GRPO refinement, while others continue benefiting from OPD-based acquisition. To estimate such acquisition dynamics online, we use teacher-side perplexity over student-generated responses. Lower perplexity indicates that the current student policy has already approached rollout-coverage saturation, whereas higher perplexity suggests that the student policy still benefits from continued acquisition under teacher supervision. For each input sample xix_i, the student first performs on-policy rollout and generates a response group: i=yi1,yi2,…,yiK.Y_i=\y_i1,y_i2,...,y_iK\. (5) The teacher model then evaluates each generated response using normalized sequence-level perplexity: ℓij=−1|yij|∑m=1|yij|logpT(yij,m∣xi,yij,<m), _ij=- 1|y_ij| _m=1^|y_ij| p_T(y_ij,m x_i,y_ij,<m), (6) PPLij=exp(ℓij),PPL_ij= ( _ij), (7) where |yij||y_ij| denotes the response length. Based on the candidate-level perplexities, we further compute a prompt-level acquisition-state score: ui=1K∑j=1KPPLij.u_i= 1K _j=1^KPPL_ij. (8) A larger uiu_i indicates that the sample still benefits from continued acquisition under teacher supervision, whereas smaller values suggest that rollout-coverage saturation is gradually approached. Since absolute perplexity scores are not directly comparable across prompts, we normalize them within each mini-batch: zi=ui−μBσB+ϵ,z_i= u_i- _B _B+ε, (9) where μB _B and σB _B denote the batch-level mean and standard deviation, respectively. We use the normalized score ziz_i as the acquisition state score for routing. Lower ziz_i indicates relatively lower teacher-side perplexity within the current mini-batch, suggesting that the sample has acquired sufficient domain knowledge under OPD optimization. In contrast, higher ziz_i suggests that the student rollouts remain less aligned with teacher-supported generation patterns and therefore still benefit from continued OPD acquisition. 2.3 Asynchronous KD-to-RL Transition Scheduling Instead of adopting a globally fixed KD-to-RL schedule, PIVOT progressively transitions samples from OPD acquisition to GRPO refinement according to their acquisition state scores. Specifically, during training, we maintain a transition ratio: r(t)∈[0,1],r(t)∈[0,1], (10) where t denotes the current training step. The transition ratio is progressively increased following a cosine scheduling strategy: r(t)=12(1−cosπtT),r(t)= 12 (1- π tT ), (11) where T denotes the total number of training steps. At each training step, samples are ranked according to their normalized acquisition-state scores ziz_i. Samples with lower acquisition-state scores are considered closer to rollout-coverage saturation and are therefore transitioned earlier into GRPO refinement. Specifically, we progressively route the lowest-ranked r(t)r(t) proportion of samples into GRPO optimization, while the remaining samples continue OPD-based acquisition. Let tS_t denote this selected GRPO subset at training step t. Let ℬtB_t denote the mini-batch at training step t. We define a binary routing indicator mi=(xi∈t)m_i=1(x_i _t), where mi=1m_i=1 indicates that sample xix_i is routed to GRPO refinement and mi=0m_i=0 indicates that it remains under OPD acquisition. The training objective at step t is then defined as ℒt=1|ℬt|∑xi∈ℬt[miℒGRPO(i)+(1−mi)ℒOPD(i)],L_t= 1|B_t| _x_i _t [m_iL^(i)_GRPO+(1-m_i)L^(i)_OPD ], (12) where ℒGRPO(i)L^(i)_GRPO and ℒOPD(i)L^(i)_OPD denote the sample-level GRPO and OPD losses for xix_i, respectively. In this way, PIVOT enables asynchronous KD-to-RL transition according to acquisition dynamics rather than globally synchronized optimization stages. Samples approaching rollout-coverage saturation transition earlier into GRPO refinement, while other samples continue benefiting from OPD-based acquisition. This progressive transition strategy improves optimization efficiency and downstream decision quality compared with globally fixed KD-to-RL schedules. 3 Experiments 3.1 Experimental Setup Datasets and few-shot setting. We evaluate our method on Banking77 and HWU64. For Banking77, we construct a class-balanced 5-shot training set by randomly sampling 5 examples per intent from the original training split, resulting in 385 training instances over 77 intents. For HWU64, we construct a class-balanced 10-shot training set by sampling 10 examples per intent, resulting in 640 training instances over 64 intents. We use the official test splits for evaluation. The few-shot sampling seed is fixed to 42 for all methods. Both datasets contain fine-grained user intents and are used to evaluate few-shot domain adaptation under limited labeled supervision. Models. Unless otherwise specified, we use Qwen2.5-0.5B-Instruct as the student model and Qwen2.5-7B-Instruct as the teacher model. We further evaluate whether the proposed method remains effective when scaling the student model from Qwen2.5-0.5B-Instruct to Qwen2.5-1.5B-Instruct. For stronger-teacher experiments on Banking77, we replace the default teacher with DianJin-32B Zhu et al. (2025). The teacher model is kept frozen throughout training and is only used to provide supervision signals for distillation and teacher-side perplexity estimation. Figure 2: Training dynamics on Banking77 under aligned total optimization steps. For globally synchronized OPD→ , the x-axis denotes cumulative optimization steps across the OPD and GRPO stages, and the purple star marks the global transition point. Compared with continued OPD and globally synchronized OPD→ , our progressive sample-level transition achieves higher final accuracy with substantially smoother reward, entropy, and completion-length dynamics. Baselines. We compare with the following baselines. Teacher performs zero-shot prompt-only inference using the teacher model. SeqKD fine-tunes the student on teacher-generated full responses under the same instruction prompt Kim and Rush (2016). SFT→ first lightly fine-tunes the student using gold labels and then applies GRPO refinement. DataAug-SFT fine-tunes the student on a teacher-generated and teacher-filtered augmented training set constructed from the few-shot Banking77 split. We include this baseline on Banking77, where the augmented set is constructed from the same 5-shot training split. OPD trains the student with OPD for the full optimization budget. OPD→ first performs OPD and then globally switches all samples to GRPO at a fixed transition checkpoint. Training protocol. Unless otherwise specified, all trainable student-side methods perform 480 post-warm-up student parameter-update steps. Following prior OPD practice Li et al. (2026b), GRPO, OPD, OPD→ , and PIVOT methods first share a 2-epoch off-policy SFT warm-up on teacher-generated responses, which is not counted in the 480 steps. After warm-up, GRPO and OPD use all 480 steps for GRPO refinement and OPD optimization, respectively. OPD→ globally switches from OPD to GRPO at the peak OPD checkpoint, while PIVOT progressively routes samples from OPD to GRPO according to teacher-evaluated sequence perplexity. Method Banking77 HWU64 Teacher 73.25 79.65 SeqKD 58.70 70.17 GRPO 67.66 77.60 OPD 72.47 79.55 OPD→ 78.05 83.09 PIVOT 80.26 85.04 Table 1: Main results under 480 post-warm-up student optimization steps. All results report test-set classification accuracy. Reward and evaluation. For GRPO-based methods, each sampled response is parsed into a predicted intent label from the field “Intent Category”. The main reward is based on whether the parsed label matches the gold intent. We additionally use lightweight auxiliary rewards to encourage valid formatting and consistency between the final answer and the listed candidate labels. Advantages are normalized within each response group. At evaluation time, we use greedy decoding and compute micro accuracy on the official test split. Invalid or unparsable responses are counted as incorrect. Progressive routing. For our method, we compute teacher-evaluated length-normalized sequence perplexity over student-generated responses and average the scores within each response group. The resulting group-level scores are normalized within each mini-batch using z-score normalization. After an initial OPD warm-up stage, a cosine schedule progressively increases the proportion of samples transitioned to GRPO throughout training. At each step, samples with lower normalized perplexity are transitioned earlier to GRPO refinement, while higher-perplexity samples continue OPD optimization. 3.2 Main Results Dataset Setting Student Teacher OPD→ PIVOT Δ Banking77 Default 0.5B Qwen2.5-7B 78.05 80.26 +2.21 Banking77 Larger student 1.5B Qwen2.5-7B 80.00 82.82 +2.82 Banking77 Stronger teacher 0.5B DianJin-32B 80.30 81.33 +1.03 HWU64 Default 0.5B Qwen2.5-7B 83.09 85.04 +1.95 HWU64 Larger student 1.5B Qwen2.5-7B 82.53 85.97 +3.44 HWU64 Stronger teacher 0.5B Qwen3-32B 84.39 86.71 +2.32 Table 2: Scaling analysis across different student and teacher model configurations. Δ denotes the improvement of PIVOT over OPD→ . Table 1 reports the main results under the same number of post-warm-up student optimization steps. Compared with continued OPD, globally synchronized OPD→ already yields substantial improvement on both datasets, suggesting that downstream GRPO refinement is important beyond prolonged imitation alone. However, PIVOT further improves over OPD→ across both Banking77 and HWU64, indicating that globally fixed transition schedules remain suboptimal for coordinating OPD and GRPO during training. Interestingly, PIVOT also surpasses teacher inference on both datasets, suggesting that progressive sample-level transition can produce a stronger task-specialized student policy under few-shot adaptation. We additionally compare against a teacher-generated data augmentation baseline on Banking77 in Appendix A.5. Although DataAug-SFT improves over standard SFT, it remains below both continued OPD and PIVOT, suggesting that the gains are not solely attributable to additional teacher-generated training data. 3.3 Optimization Dynamics Figure 2 compares test accuracy, reward, entropy, and completion length under aligned total optimization steps. Our method consistently achieves the highest final test accuracy and surpasses globally synchronized OPD→ before the global transition point, suggesting that progressive sample-level transition improves downstream refinement effectiveness under the step-matched protocol. Although globally synchronized OPD→ eventually reaches comparable reward levels, it exhibits noticeably unstable optimization behavior near the transition stage, including abrupt completion-length increase and entropy spikes. In contrast, our method maintains substantially smoother entropy and completion-length dynamics throughout training, indicating that progressively transferring low-perplexity samples into GRPO avoids the optimization discontinuity introduced by globally synchronized switching. Interestingly, our method achieves better final accuracy despite similar late-stage reward, suggesting that progressive transition improves the effectiveness of reward-driven refinement rather than merely increasing reward magnitude. 3.4 Scaling Analysis We further evaluate PIVOT across different student and teacher model configurations. As shown in Table 2, PIVOT consistently improves over OPD→ across all evaluated settings, including larger student models and stronger teacher models. Notably, the improvement remains consistent even when scaling the student from 0.5B to 1.5B parameters, suggesting that the benefit of progressive sample-level transition is not tied to a specific student capacity. Similarly, PIVOT continues to outperform OPD→ under stronger teacher models, indicating that adaptive transition remains beneficial even when teacher supervision quality improves substantially. 3.5 Ablation and Routing Analysis Dataset Low-PPL Random High-PPL Banking77 80.26 78.31 77.27 HWU64 85.04 84.11 83.46 Table 3: Routing-rule ablation under progressive OPD-to-GRPO transition. Column names indicate which samples are preferentially routed to GRPO earlier during training. Transition Strategy ACC Global OPD→ 78.05 Linear progressive routing 79.58 Quadratic progressive routing 79.19 Cosine progressive routing 80.26 Table 4: Transition-schedule ablation on Banking77 under the same number of student optimization steps. Figure 3: Pass@32 performance grouped by teacher-side perplexity buckets during OPD training. High-perplexity samples exhibit substantially larger relative improvement throughout training, whereas low-perplexity samples saturate much earlier, suggesting that the optimal transition timing from OPD to RL varies across samples. We further analyze the design choices of progressive OPD-to-GRPO transition, including both the transition schedule and the routing strategy. As shown in Table 4, all progressive routing variants consistently outperform globally synchronized OPD→ , indicating that asynchronous sample-level transition is more effective than switching all samples at a fixed checkpoint. Among different schedules, cosine routing achieves the strongest overall performance, suggesting that gradually increasing the GRPO transition ratio throughout training provides a better balance between continued OPD optimization and downstream refinement. Table 3 further shows that the best results are achieved by retaining high-PPL samples in OPD while transitioning low-PPL samples to GRPO earlier, suggesting that different samples benefit from substantially different transition timing during training. Figure 3 further supports this behavior: lower-PPL samples rapidly approach rollout-quality saturation, whereas higher-PPL samples continue improving under prolonged OPD optimization. These observations support teacher-side perplexity as an effective signal for progressive sample-level KD-to-RL transition. 4 Discussion 4.1 Distillation Quality and Refinement Readiness Are Not Necessarily Aligned Our results suggest that stronger distillation quality does not necessarily imply better downstream refinement readiness. Existing two-stage adaptation pipelines often assume that continued teacher alignment monotonically improves the initialization quality for subsequent preference optimization. However, our experiments show that the strongest downstream GRPO performance frequently emerges before full OPD convergence. One possible explanation is that prolonged imitation gradually narrows the student rollout distribution around teacher-preferred responses. Although this improves teacher alignment, it may also reduce the diversity and observability of informative candidate responses required for downstream GRPO refinement. As a result, stronger imitation quality does not always translate into more effective downstream optimization. These observations suggest that distillation and downstream refinement correspond to partially different optimization objectives, and that maximizing teacher alignment alone may not be sufficient for effective refinement-oriented adaptation. 4.2 Implications for Adaptive Distillation-to-Refinement Transition Our findings further suggest that the transition from distillation to downstream refinement should not be globally synchronized across all samples. Different samples exhibit substantially different optimization dynamics during OPD training, implying that the optimal transition timing may vary significantly across samples. In particular, low teacher-side perplexity consistently correlates with more stable rollout behavior and stronger pass@32 performance throughout training, suggesting that these samples are more suitable for earlier refinement. In contrast, high-perplexity samples often continue benefiting from prolonged OPD optimization before downstream GRPO becomes effective. Although our study focuses on OPD and GRPO under vertical-domain few-shot classification, similar interactions may arise more broadly in distillation-based adaptation settings involving iterative imitation and downstream preference optimization. More generally, our results highlight the importance of considering not only teacher alignment quality, but also how optimization dynamics influence downstream refinement behavior throughout training. 5 Related Work 5.1 On-Policy Distillation and Adaptive Optimization Knowledge distillation has become a widely adopted paradigm for transferring capabilities from large language models to smaller models under constrained supervision or computational settings Hinton et al. (2015); Sanh et al. (2020). Recent work has increasingly explored On-Policy Distillation (OPD), where teacher supervision is continuously aligned with the student’s evolving rollout distribution during optimization, improving exposure alignment during autoregressive generation Agarwal et al. (2024a); Gu et al. (2024). Recent OPD studies further explore adaptive optimization strategies for stabilizing supervision under heterogeneous rollout conditions. Existing approaches include entropy-aware distillation objectives Jin et al. (2026), uncertainty-calibrated supervision weighting Zheng et al. (2026), rollout-group coordination Yu et al. (2026), and competence-based optimization schedules Xu et al. (2026b). Recent analyses of OPD failure modes further show that teacher guidance can become unreliable on student-generated prefixes, especially when rollouts move away from prefixes commonly visited by the teacher Fu et al. (2026). These studies suggest that the effectiveness of teacher supervision can vary substantially throughout OPD optimization under heterogeneous rollout conditions. Our work is related to these adaptive OPD optimization methods but focuses on a different problem setting. Instead of improving supervision quality within OPD itself, we study how acquisition dynamics should govern the transition between distillation and downstream RL refinement during vertical-domain few-shot adaptation. 5.2 KD-RL Coordination and Adaptive Transition Preference- and reward-based optimization methods such as RLHF, DPO, and GRPO have become important paradigms for improving downstream behavior beyond supervised imitation Ouyang et al. (2022); Rafailov et al. (2023); Shao et al. (2024a). In particular, GRPO improves optimization stability through group-relative reward normalization and has shown strong effectiveness in reasoning-oriented refinement settings Shao et al. (2024a). Recent work further applies GRPO to intent detection by combining it with chain-of-thought reasoning and reward-based curriculum sampling, showing that reward-driven optimization can improve intent understanding and generalization beyond supervised fine-tuning Feng et al. (2025). Motivated by the complementary roles of distillation and downstream refinement, recent work has increasingly explored how knowledge distillation and reinforcement learning should be coordinated throughout optimization. RLKD Xu et al. (2026a) and KDRL Xu et al. (2025) show that reinforcement learning can complement teacher-guided distillation by improving exploration and downstream refinement beyond imitation alone. SuperRL Liu et al. (2025) further incorporates supervised signals into reinforcement learning to improve reasoning optimization. G-OPD Yang et al. (2026) further establishes a theoretical connection between OPD and KL-regularized reinforcement learning, suggesting that OPD itself can be interpreted as a dense reward optimization process. CoDistill-GRPO Kwon et al. (2026) and related hybrid optimization frameworks Li et al. (2026a) additionally demonstrate that jointly coordinating distillation and reinforcement learning can improve optimization efficiency and rollout quality during training. However, existing approaches primarily rely on globally shared objectives or fixed optimization schedules, without explicitly modeling sample-level transition dynamics between distillation and reinforcement learning. 6 Conclusion In this paper, we propose PIVOT, a progressive sample-level KD-to-RL transition framework for vertical-domain few-shot adaptation. PIVOT adaptively schedules OPD and GRPO according to teacher-evaluated sequence perplexity rather than globally fixed transition schedules. Experiments on Banking77 and HWU64 show that PIVOT consistently outperforms continued OPD and globally synchronized OPD→ under the same number of post-warm-up student optimization steps, while yielding more stable optimization dynamics during training. Our results suggest that adaptive sample-level transition provides an effective alternative to fixed two-stage distillation and refinement schedules. 7 Limitations Our study has several limitations. First, PIVOT relies on teacher-evaluated sequence perplexity as a heuristic routing signal, and we do not provide a theoretical characterization of why perplexity best reflects refinement readiness across different optimization settings. Second, our experiments focus on vertical-domain few-shot classification with relatively structured outputs and reward signals. Whether similar transition dynamics extend to more complex generation tasks remains an open question. Third, our comparisons align the number of student optimization steps rather than total training compute. Because PIVOT continues teacher-side perplexity evaluation for samples routed to GRPO, it introduces additional teacher inference overhead relative to the globally synchronized baseline after its transition. The reported results should therefore be interpreted as step-matched rather than strictly compute-matched comparisons. We leave broader theoretical analysis, stricter compute-matched evaluation, and evaluation on more diverse tasks for future work. AI Assistance Statement Generative AI tools, including ChatGPT and Gemini, were used to assist with language editing during the preparation of this work. All generated content was reviewed and verified by the authors. The authors take full responsibility for the accuracy, integrity, and originality of the final manuscript. References Agarwal et al. (2024a) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos Garea, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 21246–21263. External Links: Link Cited by: §5.1. Agarwal et al. (2024b) R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem On-policy distillation of language models: learning from self-generated mistakes. External Links: 2306.13649, Link Cited by: §1. Casanueva et al. (2020) I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, T. Wen, A. Celikyilmaz, Z. Yu, A. Papangelis, M. Eric, A. Kumar, I. Casanueva, and R. Shah (Eds.), Online, p. 38–45. External Links: Link, Document Cited by: §1. Ding (2026) K. Ding HDPO: hybrid distillation policy optimization via privileged self-distillation. External Links: 2603.23871, Link Cited by: §1. Feng et al. (2025) Z. Feng, X. Wang, Z. Bai, D. Su, B. Wu, Q. Yu, and B. Wang Improving generalization in intent detection: grpo with reward-based curriculum sampling. External Links: 2504.13592, Link Cited by: §1, §5.2. Fu et al. (2026) Y. Fu, H. Huang, K. Jiang, J. Liu, Z. Jiang, Y. Zhu, and D. Zhao Revisiting on-policy distillation: empirical failure modes and simple fixes. External Links: 2603.25562, Link Cited by: §5.1. Gu et al. (2024) Y. Gu, L. Dong, F. Wei, and M. Huang MiniLLM: knowledge distillation of large language models. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 32694–32717. External Links: Link Cited by: §1, §5.1. Hinton et al. (2015) G. Hinton, O. Vinyals, and J. Dean Distilling the knowledge in a neural network. External Links: 1503.02531, Link Cited by: §1, §5.1. Jin et al. (2026) W. Jin, T. Min, Y. Yang, D. Wei, Y. Zhou, S. R. Kadhe, N. Baracaldo, and K. Lee Entropy-aware on-policy distillation of language models. arXiv preprint arXiv:2603.07079. Cited by: §1, §5.1. Kim and Rush (2016) Y. Kim and A. M. Rush Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, J. Su, K. Duh, and X. Carreras (Eds.), Austin, Texas, p. 1317–1327. External Links: Link, Document Cited by: §3.1. Kwon et al. (2026) S. M. Kwon, Z. Sun, A. T. Suresh, H. Jain, and S. Kumar CoDistill-grpo: a co-distillation recipe for efficient group relative policy optimization. External Links: 2605.08873, Link Cited by: §1, §5.2. Li et al. (2026a) G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua Unifying group-relative and self-distillation policy optimization via sample routing. External Links: 2604.02288, Link Cited by: §1, §5.2. Li et al. (2026b) G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua Unifying group-relative and self-distillation policy optimization via sample routing. External Links: 2604.02288, Link Cited by: §3.1. Liu et al. (2019) X. Liu, A. Eshghi, P. Swietojanski, and V. Rieser Benchmarking natural language understanding services for building conversational agents. External Links: 1903.05566, Link Cited by: §1. Liu et al. (2026) X. Liu, K. Jiao, C. Xiao, R. Zhao, J. Ruan, B. Li, J. Liu, Q. Wang, X. Chen, J. Wang, et al. Teacher-guided policy optimization for on-policy reasoning distillation under large policy divergence. arXiv preprint arXiv:2605.13230. Cited by: §1. Liu et al. (2025) Y. Liu, S. Li, L. Cao, Y. Xie, M. Zhou, H. Dong, X. Ma, S. Han, and D. Zhang SuperRL: reinforcement learning with supervision to boost language model reasoning. External Links: 2506.01096, Link Cited by: §5.2. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §5.2. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, Vol. 36, p. 53728–53741. Cited by: §5.2. Sanh et al. (2020) V. Sanh, L. Debut, J. Chaumond, and T. Wolf DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. External Links: 1910.01108, Link Cited by: §1, §5.1. Shao et al. (2024a) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §5.2. Shao et al. (2024b) Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §1. Xu et al. (2025) H. Xu, Q. Zhu, H. Deng, J. Li, L. Hou, Y. Wang, L. Shang, R. Xu, and F. Mi KDRL: post-training reasoning llms via unified knowledge distillation and reinforcement learning. External Links: 2506.02208, Link Cited by: §1, §5.2. Xu et al. (2026a) S. Xu, L. Pang, Y. Zhu, J. Gu, Z. Wei, J. Deng, F. Pan, H. Shen, and X. Cheng Rlkd: distilling llms’ reasoning via reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 34151–34159. Cited by: §1, §5.2. Xu et al. (2026b) Y. Xu, H. Sang, Z. Zhou, R. He, and Z. Wang PACED: distillation and on-policy self-distillation at the frontier of student competence. External Links: 2603.11178, Link Cited by: §1, §5.1. Yang et al. (2026) W. Yang, W. Liu, R. Xie, K. Yang, S. Yang, and Y. Lin Learning beyond teacher: generalized on-policy distillation with reward extrapolation. External Links: 2602.12125, Link Cited by: §5.2. Yu et al. (2026) W. Yu, X. Li, Y. Zhao, X. Liu, R. Zhang, H. Wang, Y. Luo, C. H. Wu, G. Mittal, M. Fredrikson, et al. Multi-rollout on-policy distillation via peer successes and failures. arXiv preprint arXiv:2605.12652. Cited by: §1, §5.1. Zheng et al. (2026) B. Zheng, X. Ma, Y. Liang, J. Ruan, X. Fu, K. Lin, B. Zhu, K. Zeng, and X. Cai SCOPE: signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting. External Links: 2604.10688, Link Cited by: §1, §5.1. Zhu et al. (2025) J. Zhu, Q. Chen, H. Dou, J. Li, L. Guo, F. Chen, and C. Zhang DianJin-r1: evaluating and enhancing financial reasoning in large language models. External Links: 2504.15716, Link Cited by: §3.1. Appendix A Implementation Details A.1 Dataset Construction We construct class-balanced few-shot training sets from the original training splits. Banking77 uses 5 examples per intent, resulting in 385 training examples over 77 intents, while HWU64 uses 10 examples per intent, resulting in 640 training examples over 64 intents. The sampling seed is fixed to 42. We use the official test splits for evaluation and report micro accuracy. Unless otherwise stated, all experiments are conducted on the same sampled few-shot split. Following prior few-shot adaptation settings, the main results are reported on a single split because OPD and GRPO training are computationally expensive. A.2 Prompt Templates All methods use task-specific instruction prompts containing the full label set. Banking77 additionally includes natural-language intent definitions, while HWU64 uses only the intent label list. SFT-ZeroShot uses a simplified label-only supervision format, while teacher inference, SeqKD, OPD, GRPO, and PIVOT use the structured reasoning prompts shown in Tables 5 and 6. Table 5: Prompt template for Banking77. ⬇ You are an expert intent classifier for a banking customer service bot. Your task is to classify the user’s query into exactly one of the following 77 intent categories. ### Intent Definitions definitions_str ### Instructions 1. **Analyze Key Elements**: Identify the core action (e.g., "top up", "transfer") and the object. 2. **List Candidates (Crucial)**: - Before deciding, list 3-5 category names from the definitions that are most relevant to the query. 3. **Compare & Filter**: - Compare the candidates against the specific details of the query. - Explicitly rule out the ones that don’t fit perfectly. 4. **Final Conclusion**: Select the single best matching category. ### Response Format You must strictly follow this format: **Analysis:** [Brief analysis of key elements] **Candidates:** [List 3-5 potential category names here] **Reasoning:** [Your comparison and filtering process] **Intent Category:** [The exact category name] definitions_str contains the full set of Banking77 intent labels and their natural-language definitions. Table 6: Prompt template for HWU64. ⬇ You are an expert intent classifier for a multi-domain virtual assistant. Your task is to classify the user’s query into exactly one of the following n_classes intent categories. ### Intent Categories labels_str ### Instructions 1. **Analyze Key Elements**: Identify the core action and object in the query. 2. **List Candidates (Crucial)**: - Before deciding, list 3-5 category names from the intent categories that are most relevant to the query. 3. **Compare & Filter**: - Compare the candidates against the specific details of the query. - Explicitly rule out the ones that don’t fit perfectly. 4. **Final Conclusion**: Select the single best matching category. ### Response Format You must strictly follow this format: **Analysis:** [Brief analysis of key elements] **Candidates:** [List 3-5 potential category names here] **Reasoning:** [Your comparison and filtering process] **Intent Category:** [The exact category name] labels_str contains the full HWU64 intent label set. Here, labels_str contains the full HWU64 intent label set. A.3 Output Parsing We parse the final prediction from the field “Intent Category”. The first valid label appearing after “Intent Category:” is matched against the task-specific label set. If multiple valid labels appear after this field, the first valid match is used. If no valid label can be matched, the prediction is counted as incorrect. No manual correction is applied. A.4 Baseline Implementations GRPO and OPD-based methods use the same 2-epoch off-policy SFT warm-up on teacher-generated responses before on-policy optimization. Teacher. The frozen teacher model performs zero-shot prompt-only inference using the same task prompt as evaluation. SFT-ZeroShot. The student is fine-tuned on the few-shot gold training set using label-only supervision targets. SeqKD. For each few-shot training example, the teacher generates one full response under the same instruction prompt. The student is then fine-tuned on the resulting teacher-generated responses. GRPO. The student is optimized directly with GRPO after the off-policy warm-up stage. DataAug-SFT. The student is fine-tuned on a teacher-generated augmented training set with teacher-based filtering constructed from the same Banking77 5-shot split. This baseline is included only on Banking77. OPD. The student is trained with OPD for all 480 post-warm-up student optimization steps. OPD→ . The student is first trained with OPD and then globally switched to GRPO at a fixed transition checkpoint. The best transition checkpoint is selected within the same total of 480 post-warm-up student optimization steps. A.5 Data Augmentation Baseline We additionally evaluate a teacher-generated data augmentation baseline on Banking77 to test whether the gains of PIVOT can be explained by additional synthetic training data alone. Starting from the same 5-shot Banking77 split, we construct an augmented training set using intent clustering, teacher-based query generation, and teacher self-filtering. Semantically similar intents are first grouped into clusters. For each cluster, the teacher generates additional user queries conditioned on the target intent, intent definition, and several anchor examples. The generation prompt encourages diverse and label-discriminative queries. Generated samples are then reclassified by the teacher, and only samples whose predicted labels match the target intent are retained. Using this pipeline, we generate 5,650 candidate augmented examples, of which 5,119 are retained after teacher self-filtering. Fine-tuning on the resulting augmented set achieves 71.75 test accuracy on Banking77. While synthetic data augmentation substantially improves over standard SFT, it remains below both continued OPD and OPD→ training. Notably, introducing the OPD-to-GRPO transition yields a large additional gain, suggesting that the effectiveness of PIVOT cannot be explained solely by additional teacher-generated supervision or extended optimization. Method Banking77 SFT-ZeroShot 60.30 DataAug-SFT 71.75 Continued OPD 72.47 OPD→ 78.05 PIVOT 80.26 Table 7: Comparison between teacher-generated data augmentation and different OPD/GRPO training strategies on Banking77. DataAug-SFT is trained on 5,119 teacher-filtered synthetic examples generated from the same 5-shot split. Table 8: Prompt template for teacher-based data augmentation. ⬇ You are an expert at generating realistic bank customer inquiry data. Your task is to simulate real banking customers, generating highly conversational and natural customer service queries that accurately map to specific intents. I need to augment training data for a highly sensitive bank intent recognition model. Below is a group of num_intents intents that are semantically similar and easily confused. They share overlapping vocabulary but have critical semantic differences. For each target intent, you are given: - the intent name; - its definition; - several reference anchor queries. TASK: Generate exactly n_per_intent completely new customer queries for EACH intent. Total output = n_per_intent * num_intents items. Strict Constraints: 1. Boundary Exclusivity: Each query must be unambiguous and must only map to its declared intent, not to any other intent in this group. 2. Extreme Diversity: Vary sentence length, structure, formality, and scenario. Use informal speech, colloquialisms, and specific real-life situations. Do not simply paraphrase the seed examples. 3. Scenario-driven: Imagine concrete real-life situations, such as standing at a supermarket checkout, receiving a bank notification, or finding an unexpected charge. 4. No repetition: Do not repeat any reference anchor sentence. Output ONLY a JSON array: [ "intent": "intent_1", "text": "...", "intent": "intent_2", "text": "..." ] A.6 Training Hyperparameters Table 9 summarizes the main training hyperparameters used across all experiments. All experiments were conducted on a server with four NVIDIA A100 GPUs. Hyperparameter Value Total optimization steps 480 Batch size 256 Rollouts / group size 8 Warmup ratio 0.05 Max sequence length 4096 Max new tokens 512 Temperature 1.0 Top-p 0.9 Top-k 50 Precision bf16 Fine-tuning type full Learning rate (OPD) 1×10−51× 10^-5 Learning rate (GRPO / Ours) 1×10−61× 10^-6 KL coefficient (GRPO / Ours) 0.001 Table 9: Training hyperparameters used in all experiments. A.7 Reward Function For GRPO-based methods, the final intent label is parsed from each generated response. The reward consists of three components: a correctness reward based on whether the parsed label matches the gold intent (+1 / -1), a candidate-alignment reward encouraging consistency between the final prediction and the listed candidate labels (+0.5 / -0.5), and a lightweight format reward encouraging valid structured outputs (+0.1). We do not apply an explicit length penalty. A KL penalty with coefficient 0.001 is used during GRPO optimization, and advantages are normalized within each response group.