Paper deep dive
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents
Haoze Wu, Chuqiao Kuang, Tianyi Zhuang, Xiaoguang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/14/2026, 5:09:29 AM
Summary
The paper introduces Step-Level Self-Distilled Policy Optimization (SSPO), a method for training deep search agents that addresses the sparsity of outcome rewards in reinforcement learning. SSPO utilizes 'Evidence Anchors' as privileged information to create step-level advantage weights within the GRPO algorithm, applied only to incorrect trajectories. This approach decouples update direction from magnitude, improving sample efficiency and performance on benchmarks like BrowseComp, GAIA, and FRAMES compared to standard GRPO.
Entities (9)
Relation Signals (6)
SSPO → modifies → GRPO
confidence 95% · SSPO, which converts teacher–student disagreement into step-level advantage weights within GRPO
SSPO → uses → Evidence Anchors
confidence 95% · First, we construct Evidence Anchors... as privileged information... Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO)
SSPO → appliedto → incorrect_trajectories
confidence 92% · SSPO... applied exclusively to incorrect trajectories
SSPO → evaluatedon → Qwen3-8b
confidence 90% · On Qwen3-8B, SSPO consistently outperforms GRPO
SSPO → outperforms → GRPO
confidence 90% · SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES
Evidence Anchors → providesprivilegedinformationfor → SSPO
confidence 90% · We construct Evidence Anchors... as privileged information that captures key reasoning steps
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory, which is far too sparse for effective credit assignment. On-policy self-distillation (OPSD) addresses this by using the model's own logits as dense token-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student's exploration-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies. We resolve this tension through two contributions. First, we construct Evidence Anchors, which are concise, step-level evidence snippets extracted from the web, as privileged information that captures key reasoning steps without revealing the entire answer path. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher-student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories are left untouched, preserving their diversity. On Qwen3-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5 percent overhead per step from a single additional forward pass.
Tags
Links
- Source: https://arxiv.org/abs/2608.12764v1
- Canonical: https://arxiv.org/abs/2608.12764v1
Trouble viewing inline? Open PDF directly →
Full Text
74,595 characters extracted from source content.
Expand or collapse full text
Beyond Outcome Rewards: Step-Level Self-Distilled Policy Optimization for Deep Search Agents Haoze Wu Chuqiao Kuang Affiliation: Huawei Technologies Ltd.[2pt] Tianyi Zhuang Affiliation: Huawei Technologies Ltd.[2pt] Xiaoguang Li Affiliation: Huawei Technologies Ltd.[2pt] [4pt] The Hong Kong University of Science Technology Abstract Deep search agents operate over trajectories spanning dozens of steps, yet standard reinforcement learning provides only a single outcome reward per trajectory — supervision that is far too sparse for effective credit assignment. On-policy self-distillation (OPSD) addresses this by using the model’s own logits as dense token-level teachers, but extending it to search agents introduces a fundamental tension: the teacher, having access to privileged information such as the correct answer, produces a distribution that differs systematically from the student’s exploration-based reasoning, and naive distillation causes the student to inherit this information asymmetry rather than learn better search strategies. We resolve this tension through two contributions. First, we construct Evidence Anchors — concise, step-level evidence snippets extracted from the web — as privileged information that captures key reasoning steps without revealing the entire answer path. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO), which converts teacher–student disagreement into step-level advantage weights within GRPO, applied exclusively to incorrect trajectories. This design decouples what to update from how much to update: the outcome reward determines the direction of policy change, while the teacher modulates its magnitude at each step. Correct trajectories are left untouched, preserving their diversity. On Qwen3-8B, SSPO consistently outperforms GRPO across BrowseComp, GAIA, and FRAMES, surpassing or matching GRPO trained with twice as many gradient steps while adding only about 5% overhead per step from a single additional forward pass.11 1 Code is available at https://github.com/hkust-nlp/SSPO. Correspondence to: hwuds@connect.ust.hk. Figure 1: Comparison of performance across training methods. Left: After introducing step-level self-distilled advantage weights (SSPO), performance improves substantially faster during training, surpassing GRPO trained for 200 steps after only 100 steps. Right: We report results for the cold-start model, GRPO after 100 and 200 steps, and SSPO after 100 steps. 1 Introduction Recent Large Language Model paradigms have rapidly evolved from generating appropriate responses to solving real-world tasks through tool use across domains such as coding, office productivity, finance, and research 18; 33; 41; 4; 2; 7; 34; 30; 9. Among these applications, web search agents must answer vague, underspecified queries by actively exploring the open web — issuing searches, inspecting pages, and refining their reasoning over trajectories that often span dozens of steps 43. Training agents to perform such long-horizon search reliably requires post-training beyond Supervised Fine-Tuning (SFT), making reinforcement learning (RL) the dominant approach 26; 20; 19; 40. However, RL in this setting faces a fundamental challenge: trajectories containing 20+ steps receive only a single binary reward. With supervision this sparse, the model receives little guidance on which steps led to success or failure, making credit assignment the central bottleneck for deep search agents. A natural way to address sparse rewards is to introduce denser supervision. SFT provides token-level feedback through teacher trajectories, but its Off-Policy nature introduces exposure bias and limits generalization to unseen queries 36; 6; 48; 38. On-Policy Distillation (OPD) 27; 39; 23 combines both benefits: the student performs on-policy rollouts while a teacher model provides token-level scoring. More recent work extends this to self-distillation, where the teacher is constructed from the student itself — using privileged information such as reference solutions or environment feedback as a prefix 63; 54; 13; 38; 35; 17, eliminating the need for a separate teacher model. Despite this progress, existing self-distillation methods have been developed and evaluated primarily on single-turn reasoning tasks such as math and code. Two properties make self-distillation feasible in those settings: the gap between teacher and student behavior is modest, and a natural form of privileged information — reference solutions or execution feedback — already exists. Extending self-distillation to multi-turn search agents breaks both assumptions. The information asymmetry becomes extreme: the teacher, seeing curated evidence and the answer, collapses to 3.5 tool calls per trajectory, while the student, navigating the open web, requires 17.7 (Table 1). Directly distilling this collapsed distribution causes the student to mimic the teacher’s brevity; when privileged information is unavailable, it abandons tool use prematurely rather than learning better search strategies. An equally serious problem compounds the first: unlike math or code, open-ended web search has no ready-made privileged information. A form of supervision that is both informative to the teacher and safe to distill to the student does not naturally exist, and constructing one is itself an open challenge. Our key insight is that search supervision is naturally defined at the level of information-seeking actions rather than token generation. We therefore align privileged supervision with the structure of search through two design choices. First, we construct Evidence Anchors: compact, step-level evidence snippets extracted from the web that capture information needed to answer a question without revealing the answer path. Each question is associated with a small set of such anchors, serving as prefixes for the self-teacher. Second, we propose Step-Level Self-Distilled Policy Optimization (SSPO). Rather than minimizing divergence between teacher and student distributions, SSPO converts teacher–student disagreement into step-level advantage weights within GRPO 37; 11, applied only to incorrect trajectories. This design decouples update direction from update magnitude: the outcome reward determines whether to reinforce or suppress a trajectory, while the teacher modulates how much each step contributes. Correct trajectories are left untouched, preserving behavioral diversity. We evaluate SSPO on Qwen3-8B 53 across three information-seeking benchmarks — BrowseComp 43, GAIA 29, and FRAMES 16. SSPO consistently outperforms GRPO across all benchmarks, and notably surpasses GRPO trained with twice as many gradient steps while adding only about 5% computational overhead per training step. Our ablation studies further validate two key design decisions. First, directly matching the teacher distribution collapses the student’s tool use and underperforms even GRPO, confirming that decoupling update magnitude from direction is essential. Second, step-level advantage weights consistently outperform token-level counterparts, whose fine-grained signals are misaligned with the natural unit of information-seeking actions. In summary, our contributions are: (a) We show that extending self-distillation to multi-turn search breaks the assumptions underlying single-turn self-distillation, requiring a fundamentally different design. (b) We introduce Evidence Anchors: compact, step-level evidence snippets as privileged information for open-ended web search, aligned with the natural unit of information-seeking actions. (c) We propose SSPO, which uses step-level self-distilled signals as advantage weights rather than optimization targets, achieving superior sample efficiency over GRPO. 2 Preliminaries 2.1 Search Agent We build our search agent based on the ReAct paradigm 56; 50, where the model interleaves reasoning and actions until producing the final answer. Our action space includes two tools connected to the real web: search and browse, their details could be found in Appendix C. After receiving a user query, the agent attempts to provide an accurate answer through multiple Thought Action Observation steps. An agent with T steps can be formulated as follows: HT=user,1,1,1,…,T−1,T−1,T−1,TH_T=\ q_user, t_1, a_1, o_1,…, t_T-1, a_T-1, o_T-1, t_T\ At time step τ, the agent generates a thought τ t_τ and a tool call τ a_τ (if it is not the final step) conditioned on the full interaction history, i.e., τ,τ∼π(,∣Hτ−1) t_τ, a_τ π( t, a H_τ-1). The output of the tool is treated as the observation τ o_τ for this step. 2.2 On-Policy Learning Agentic Reinforcement Learning. Deep search, similar to math and logic tasks 37; 50, is typically formulated under the Reinforcement Learning with Verifiable Rewards (RLVR) setting. In this setting, a correctness reward can be assigned by evaluating whether the agent’s prediction is semantically consistent with the ground truth: Rcorrect=is_equal(pred, ground_truth)∈0,1R_correct=is\_equal(pred, ground\_truth)∈\0,1\. In addition, training often incorporates a format reward RformatR_format to encourage adherence to the ReAct paradigm 56. The final training reward is defined as: Rfinal=Rcorrect+0.2×RformatR_final=R_correct+0.2× R_format (1) Similar to other works 26; 20; 19, we use the Group Relative Policy Optimization (GRPO) algorithm to update the model 37; 11. It addresses the challenge of estimating baselines in policy gradient methods by comparing responses within a group. Specifically, for each question q, GRPO samples G responses y(1),…,y(G)\y^(1),…,y^(G)\ from the current policy and normalizes the rewards to obtain a sequence-level advantage estimate: A(i)=Rfinal(q,y(i))−μGσG,A^(i)= R_final(q,y^(i))- _G _G, (2) where μG _G and σG _G denote the mean and standard deviation of rewards within the group, respectively. The policy is then updated by maximizing a clipped surrogate objective: ℒGRPO(θ)=q∼[1G∑i=1G1|y(i)|∑t=1|y(i)|min(ρt(i)At(i),clip(ρt(i),1−ϵ,1+ϵ)At(i))]L_GRPO(θ)=E_q [ 1G _i=1^G 1|y^(i)| _t=1^|y^(i)| \! ( _t^(i)A^(i)_t,\ clip\! ( _t^(i),1-ε,1+ε )A^(i)_t ) ] (3) where ρt(i)=πθ(yt(i)∣q)/πθold(yt(i)∣q) _t^(i)= _θ(y_t^(i) q)/ _ _old(y_t^(i) q) is the per-token importance sampling ratio, and ϵε is the clipping threshold that constrains the policy update to a trust region. All tokens in the trajectory share the sequence-level advantage At(i)=A(i)A_t^(i)=A^(i). To improve training efficiency, we adopt the duplication strategy proposed in WebSailor 20, where groups with non-zero variance within the batch are duplicated to replace zero-variance groups, as the latter provide no useful training signal. On-Policy Self Distillation (OPSD). Although GRPO has become the default algorithm for training agents, it has a key limitation in settings with long reasoning chains: relying solely on outcomes lacks process-level supervision. This is especially problematic for incorrect responses, where a small error in the final few tokens may lead to an incorrect answer, yet all preceding tokens receive the same level of penalty. Recent works, including OPSD 54; 63; 13; 38; 57; 35; 17, attempt to alleviate the lack of process-level supervision in long CoT 44 without relying on other teacher models 27. These methods provide the model with privileged information cprivilegedc_privileged as a form of self-teaching, such as reference solutions, environment feedback, or better rollouts, thereby enabling token-level supervision for the student. The training objective is formulated as follows: ℒOPSD(θ)=q∼,y∼PS(⋅∣q)[1|y|∑t=1|y|(PT∣PS)]L_OPSD(θ)=E_q ,y P_S(· q) [ 1|y| _t=1^|y|D(P_T P_S) ] (4) where D denotes a divergence measure (e.g., KL divergence), and θ represents the parameters of the student model. The teacher and student distributions can be expressed as: Teacher:PT(y∣q,y<t)=πθ^(⋅∣q,y<t,cprivileged)Student:PS(y∣q,y<t)=πθ(⋅∣q,y<t) :P_T(y q,y_<t)= _ θ(· q,y_<t,c_privileged)\,\,Student:P_S(y q,y_<t)= _θ(· q,y_<t) (5) For training stability, the teacher model’s parameters θ θ are typically not updated through gradient training; common choices include keeping them fixed, updating them via EMA, or synchronizing them with the student model at regular step intervals. Although significant progress has been made, existing work is largely limited to relatively simple single-turn reasoning tasks. Our work explores how to leverage self-distillation signals to provide more fine-grained supervision for training long-horizon search agents. 3 Step-Level Self-Distilled Policy Optimization Deep search agents operate through information-seeking actions whose utility emerges only at the level of complete search steps. A retrieval action is valuable not because of individual token choices, but because it effectively localizes uncertainty, gathers relevant evidence, and shapes subsequent exploration. Motivated by this observation, both our privileged information and supervision are aligned with the structure of search steps. 3.1 Privileged Information: Evidence Anchors In single-turn settings, constructing privileged information is relatively straightforward. For mathematical tasks, many open-source datasets provide reference solutions 63; 10; for tool use and coding, error messages can be directly obtained 13; even in the absence of explicit feedback, the highest-scoring rollout among multiple candidates can be selected as privileged teacher information 13. However, for BrowseComp-style tasks 43, models often require context windows spanning tens or even hundreds of thousands of tokens. In such settings, providing the best rollout as privileged information is impractical, while supplying only the final answer is insufficient to effectively guide multi-step reasoning and action. What is needed is privileged information structured around individual search actions — capable of signaling whether each step retrieves the right evidence. Figure 2: An example question with three evidence anchors. The differently colored evidence anchors correspond to the highlighted conditions in the question. To construct such privileged information, we construct key pieces of evidence for each QA pair as Evidence Anchors. Specifically, we prompt a SOTA LLM to identify as many pieces of evidence as possible that support the ground-truth answer for each question. The prompt we use can be seen in Figure 12. We collect evidence anchors for over 6,000 QA pairs, with an average of 5.24 anchors per question. Figure 2 presents an example from our training data. Detailed statistics and correctness validation are provided in Appendix G. These privileged signals are incorporated into the teacher model’s prompt only for incorrect trajectories, as illustrated in Figure 3. In practice, in addition to evidence anchors, we explicitly provide the teacher with the incorrect answers generated from the student’s rollouts, preventing the teacher from repeating the same mistakes in its output distribution. ⬇ Question: question You previously attempted to obtain the answer rollout answer, but it is incorrect. The following are some evidence anchors that may help you: - source title: explanation ... [IMPORTANT]: Do not answer directly from the anchors. Correctly solve the original question: Figure 3: Teacher prompt template incorporating evidence anchors and previously generated incorrect answers from the student model. 3.2 Step-Level Self-Distilled Advantage Weights Pass@1 #Turn Question 46 17.7 + Anchors 82 3.5 + Answer 96 2.4 Table 1: Comparison of accuracy and average steps before and after providing privileged information. As discussed in prior work 54, directly optimizing Equation 4 can lead to privileged information leakage, which may degrade model performance in the later stages of training. This issue becomes more pronounced in multi-turn settings, where shortcuts introduced by privileged information affect not only the reasoning process itself but also significantly reduce the number of tool calls required for information gathering. To quantify this effect, we sample 50 examples from the training set and compare the cold-start model’s accuracy and trajectory length under different information conditions. As shown in Table 1, providing evidence anchors in addition to the question improves accuracy from 46 to 82, while reducing the average number of steps from 17.7 to 3.5. When the final answer is further revealed, accuracy increases to 96, and the average number of steps decreases to 2.4. This substantial discrepancy suggests that the teacher’s distribution with privileged information is not an appropriate optimization target for the student. Instead, using self-distillation signals as weighting factors, rather than as gradient directions, helps mitigate this issue 54. Under this formulation, the update direction remains determined by environment rewards, while the teacher’s privileged distribution only modulates the update magnitude, preventing the student from directly fitting an information-asymmetric target. Beyond privileged-information leakage, SRPO 17 further shows that applying OPSD signals only to incorrect trajectories, while preserving the original GRPO objective for correct ones, yields better performance. Applying distillation to already-correct trajectories introduces optimization ambiguity: the model is pushed toward a specific teacher distribution despite having already solved the task, unnecessarily suppressing the diversity of correct solutions. Together, these two insights motivate our method design: we use self-distillation signals as step-level advantage weights rather than optimization targets, and restrict their application to incorrect search-agent trajectories. For each student-generated step τ in trajectory y(i)y^(i), its thinking tokens τ t_τ and tool-calling tokens τ a_τ jointly constitute one action in the agentic MDP 60. Under autoregressive factorization, the teacher and student assign the following conditional joint probabilities to this complete step: PT(τ,τ)=∏k∈τπθ^(yk∣cprivileged,q,y<k),PS(τ,τ)=∏k∈τπθ(yk∣q,y<k). P_T( t_τ, a_τ)= _k∈τ _ θ(y_k c_privileged,q,y_<k), 18.49988ptP_S( t_τ, a_τ)= _k∈τ _θ(y_k q,y_<k). (6) Equation 6 follows standard autoregressive factorization, under which the joint probability of a step is the product of its token-level conditional probabilities. We compute both likelihoods by teacher-forcing the same student-generated step under the corresponding teacher and student configurations: PTP_T uses the privileged context and teacher parameters, whereas PSP_S uses the original context and current policy parameters. We then define the privileged-information gain as the log joint-likelihood ratio of this step action: Δτstep=sg(logPT(τ,τ)PS(τ,τ)), _τ^step=sg ( P_T( t_τ, a_τ)P_S( t_τ, a_τ) ), (7) where sg denotes stop-gradient. For tokens yty_t belonging to step τ (t∈τt∈τ), they share the same step-level advantage weight: wt=t∈τwτstep=min(exp(sign(A(i))⋅Δτstep),1+ϵ)w_t t∈τ=w_τ^step= ( (sign(A^(i))· _τ^step),1+ε) (8) where ϵε is a positive hyperparameter, and the min operation prevents extreme gradient magnitudes. It is important to clarify that trajectories with Rfinal<1R_final<1 in Equation 1 are treated as incorrect trajectories, since they fail to produce a factually equivalent final answer22 2 An incorrect trajectory may still have a positive group-relative advantage, e.g., when it receives the format reward and outperforms other rollouts in the same group. Hence, Equation 8 explicitly considers the advantage sign.. Finally, we replace the advantage term in Equation 3 with A^t(i)=wtAt(i) if Rfinal<1 else A(i) A^(i)_t=w_tA_t^(i) if R_final<1 else A^(i). Δτstep>0 ^step_τ>0 Δτstep<0 ^step_τ<0 A(i)>0A^(i)>0 w>1w>1 Reward amplified w<1w<1 Reward dampened A(i)<0A^(i)<0 w<1w<1 Penalty reduced w>1w>1 Penalty amplified Table 2: Step-level advantage weight modulation. As summarized in Table 2, the weight w modulates each token’s contribution along two axes: the sign of the trajectory-level advantage A(i)A^(i) and the step-level teacher–student agreement Δτstep ^step_τ. For incorrect trajectories with A(i)<0A^(i)<0, which represent the typical case for failed rollouts, we use the weight min(PS/PT, 1+ϵ) (P_S/P_T,\,1+ε). When the student is overconfident on a step rejected by the teacher (PS→1,PT→0P_S→ 1,\,P_T→ 0, i.e., Δτstep<0 ^step_τ<0), w∈(1, 1+ϵ)w∈(1,\,1+ε) amplifies the penalty, discouraging repeated mistakes. Conversely, when the student is uncertain on a step endorsed by the teacher (PS→0,PT→1P_S→ 0,\,P_T→ 1, i.e., Δτstep>0 ^step_τ>0), w∈(0, 1)w∈(0,\,1) reduces the penalty, preserving valid intermediate reasoning within failed rollouts. For incorrect trajectories with A(i)>0A^(i)>0, which may occur on challenging questions when a trajectory receives format rewards despite producing an incorrect final answer, we adopt the symmetric weight min(PT/PS, 1+ϵ) (P_T/P_S,\,1+ε): a confident teacher paired with an uncertain student amplifies the reward, while a confident student paired with an uncertain teacher dampens it. Across all cases, SSPO adjusts the magnitude of updates at step-level resolution without altering the trajectory-level update direction. 4 Experiment 4.1 Experimental Setup Training Data. Before On-Policy learning, we first followed the WebExplorer 26 pipeline to collect approximately 4,000 English reasoning trajectories with correct final answers and without obvious tool-calling errors, which we used for cold-start initialization. Further details of these trajectories are provided in Appendix D. For On-Policy learning, we sampled around 6,000 English QA pairs from the open-source DeepForge dataset 64, using a difficulty ratio of 1.5:3.5:3.5:1.5 across the four levels. We further employed DeepSeek-V3.2 8 to construct evidence anchors for each QA pair. Benchmarks. BrowseComp 43 is a highly challenging information-retrieval benchmark introduced by OpenAI. GAIA 29 is a widely used benchmark for general AI assistants; following WebThinker 22, we evaluate on its text-only subset. FRAMES 16, introduced by Google, is used to assess factual accuracy and reasoning capability. Due to API cost constraints, we evaluate intermediate checkpoints on subsets of BrowseComp and FRAMES, denoted as BC, FRAMES-Sub. We adopt the Avg@4 metric to reduce variance, with the temperature set to 0.6 and top-p set to 0.95. More experimental details could be found in Appendix B. 4.2 Experimental Results Main Results. Table 3 presents the performance of representative small-scale search agents, alongside our models trained with cold-start initialization, GRPO, and SSPO, as well as the average number of turns required to solve each problem. Despite differences in search scaffolding and the fact that existing agents of comparable size are typically trained on substantially larger datasets, our models achieve competitive—often superior—performance, validating the effectiveness of our training pipeline. Our primary focus, however, is on the performance gains that SSPO brings over GRPO within the same scaffold. Models trained with SSPO consistently outperform their GRPO counterparts by a significant margin. Using the average score across three benchmarks as a representative metric, GRPO achieves a +2.4 improvement over the cold-start baseline, whereas SSPO delivers a markedly larger gain of +4.8. Importantly, these improvements do not come at the expense of reasoning efficiency. The SSPO-trained model requires only a marginally higher number of turns compared to GRPO (20.5 vs. 20.0), indicating that its performance gains are achieved with minimal additional computational cost. As shown in Figure 1 and Figure 10, SSPO demonstrates substantially higher learning efficiency, consistently outperforming GRPO across all three benchmarks at the same number of training steps. To further examine the impact of process supervision, we double the number of training steps for GRPO; notably, SSPO trained for 100 steps already surpasses GRPO trained for 200 steps, highlighting the superior sample efficiency enabled by process-level supervision. Beyond benchmark scores, these gains come with negligible training cost — only about 5% additional overhead from teacher scoring; detailed wall-clock and compute statistics are provided in Appendix E. Model BrowseComp 43 GAIA 29 Frames 16 Average Small Size Search Agent (<10B) WebSailor-7B 20 6.7 37.9 – – MiroThinker-8B-DPO-v0.1 42 8.7 46.6 64.4 39.9 AFM-WebAgent-7B (RL) 21 5.8 40.8 – – DeepDive-9B (RL) 28 6.3 – – – WebExplorer-8B (RL) 26 15.7 50.0 75.7 47.1 OffSeeker-8B (DPO) 64 12.8 51.5 – – Our Models (based on Qwen3-8B) Our-8B (Cold-Start) 11.7(37.7) 44.8(14.9) 67.2(7.2) 41.2(19.9) Our-8B (GRPO) 13.6(38.0) 47.3(14.5) 69.8(7.6) 43.6(20.0) Our-8B (SSPO) 15.7(39.4) 49.3(14.5) 73.0(7.7) 46.0(20.5) Table 3: Main results on the BrowseComp, GAIA, and Frames benchmarks. Subscripts indicate the average number of search turns required per problem. Underlined values denote the best overall performance in each column, while bold values highlight the best results among our models. Results for other search agents are reported from their respective original publications. Figure 4: Comparison of training dynamics between GRPO and SSPO (EMA Smoothed). Training Dynamics. Beyond benchmark scores, we analyze the training dynamics in Figure 4. After an initial warm-up phase of approximately 30–50 steps, SSPO achieves higher rewards than GRPO and maintains this advantage throughout training, indicating more effective optimization in our experiments. Notably, the entropy curves of both methods remain closely aligned, suggesting that the performance gains of SSPO are unlikely to stem primarily from increased exploration, but rather from more effective policy updates. This observation is further supported by the gradient norm curves: GRPO exhibits a steadily decreasing gradient norm, indicating early saturation of policy updates, whereas SSPO maintains a relatively higher gradient norm throughout training, reflecting more sustained learning signals. Although SSPO shows a transient spike in gradient norm around step 50, this spike quickly subsides and does not appear to harm training stability, as evidenced by the continued improvement in reward. Overall, these results suggest that SSPO improves upon GRPO mainly by enabling more sustained and effective parameter updates, rather than by substantially altering the exploration–exploitation balance. 4.3 Ablation Study We organize the ablation around three questions: how direct distillation compares with advantage weighting, whether the weighting signal should be defined at token or step level, and what roles the two privileged-information inputs play. Unless otherwise stated, all results are measured on BC-Sub after 50 training steps. Method Use of self-distillation signal Granularity Evidence Anchors BC-Sub Avg. turns GRPO None; outcome reward only – No 12.8 36.1 OPSD adaptation Direct optimization target Token Yes 11.8 30.9 RLSD/SRPO adaptation Advantage weighting Token Yes 11.8 35.5 Ours (SSPO) Advantage weighting Step Yes 14.5 37.5 Table 4: Comparison of the outcome-only baseline and self-distillation variants after 50 training steps. The RLSD/SRPO adaptation uses the self-distillation signal as token-level advantage weights, following RLSD, and applies it only to incorrect trajectories, following SRPO. Direct Distillation vs. Advantage Weighting. GRPO provides the outcome-only baseline, reaching 12.8 accuracy with an average of 36.1 turns. Under the same training budget, the direct OPSD adaptation (Appendix F.1) obtains lower accuracy than GRPO (11.8 vs. 12.8) and produces substantially shorter trajectories (30.9 vs. 36.1 turns). Together with Table 1, this pattern is consistent with privileged-information shortcutting: the teacher can solve the task with much less search, and directly matching its distribution transfers this short-search behavior to the student. The RLSD/SRPO adaptation instead uses the token-level self-distillation signal as advantage weights on incorrect trajectories. Its average trajectory length recovers to 35.5 turns, close to GRPO’s 36.1 and substantially above the direct OPSD adaptation’s 30.9, while its accuracy remains 11.8. This recovery supports using the teacher signal to modulate the update magnitude rather than as a direct optimization target; however, the unchanged accuracy also shows that token-level weighting alone is insufficient in this setting. Method Evidence Anchors Incorrect feedback BC-Sub Sign agreement with full setting (%) Final-step penalty amplified (%) GRPO No No 12.8 – – SSPO w/o Evidence Anchors No Yes 12.4 32.8 92.4 SSPO w/o Incorrect Feedback Yes No 14.0 81.1 46.7 Ours (SSPO) Yes Yes 14.5 100.0 86.3 Table 5: Ablation of the two privileged-information sources. BC-Sub results are measured after 50 training steps. The two rightmost columns are computed by rescoring 512 incorrect trajectories, using the sign under the full setting as a reference. “Final-step penalty amplified” denotes the proportion of trajectories for which the final-step weight is greater than one. Step-Level Rather Than Token-Level. We then isolate supervision granularity by holding advantage weighting, Evidence Anchors, and incorrect-trajectory routing fixed. The token-level RLSD/SRPO adaptation assigns a separate weight to every token (Appendix F.2), whereas SSPO shares one joint-probability weight across the reasoning and tool-calling tokens that form a complete information-seeking step. This controlled change improves BC-Sub accuracy from 11.8 to 14.5 while maintaining the average trajectory length (35.5 vs. 37.5 turns). The entropy curves in Figure 6 provide complementary evidence: token-level weighting produces steadily increasing entropy, whereas step-level weighting remains more stable. These results support aligning the supervision granularity with the complete search action rather than fragmenting it across individual tokens. Roles of Evidence Anchors and Incorrect-Answer Feedback. Table 5 separates the two sources of privileged information. Incorrect-answer feedback alone does not improve over GRPO (12.4 vs. 12.8), whereas Evidence Anchors alone improve accuracy to 14.0. Combining both yields the best result, 14.5, showing that Evidence Anchors provide the primary process-supervision signal and incorrect-answer feedback is complementary. To examine how they contribute to the full step score, we rescore the same 512 incorrect trajectories under each single-information setting. This is a decomposition of the deployed signal rather than an assumption that the full setting is an unbiased ground-truth label. Evidence Anchors alone agree with the sign of the full signal on 81.1% of steps, compared with 32.8% for incorrect-answer feedback alone, indicating that Evidence Anchors primarily determine the evaluation of intermediate information-seeking actions. Conversely, incorrect-answer feedback amplifies the final-step penalty in 92.4% of trajectories, compared with 46.7% under Evidence Anchors alone. Thus, the incorrect answer primarily identifies the failed terminal conclusion, while Evidence Anchors ground credit assignment across the preceding search process. Figure 5: Entropy curves of GRPO, step-level, and token-level supervision during training. Figure 6: Average number of queries under different step-level privileged-information gains. 5 Case Study: Which Steps are Exempted from Penalty? To better understand why SSPO is effective, we analyze how the teacher assigns penalties across different types of steps. We find that the teacher does not simply reward steps based on their lexical overlap with evidence anchors. Instead, it modulates penalties according to whether a step performs a focused and informative verification action. Specifically, steps that receive reduced penalties are typically compact, well-scoped queries targeting a single local constraint implied by the evidence anchors, as shown in Figure 7. Although these steps may contain fewer anchor words, they effectively reduce uncertainty along the reasoning chain. In contrast, steps that receive stronger penalties are often long, diffuse queries that concatenate multiple surface terms from the question or anchors without forming a precise information need, as shown in Figure 8. Despite their high lexical overlap with anchors, such steps receive amplified penalties, indicating that the teacher prioritizes information quality over word matching. This behavior is further supported by quantitative analysis. Figure 6 shows the relationship between the average number of queries in search tool calls and the privileged-information gain. Steps with positive privileged-information gain Δτstep>0 _τ^step>0 use significantly fewer queries than those with negative gain Δτstep<0 _τ^step<0. Overall, SSPO encourages targeted, evidence-grounded information-seeking by reducing penalties for precise verification steps while amplifying penalties for broad and unfocused exploration. ⬇ <think> Perhaps the specific ERNI employee referenced is not the person listed in the program committee (Florian, Dominik etc). The above LinkedIn snippet for Patrick Pfeifer shows "iSAQB - Foundation Level (CPSA-F)", which matches the certification. Now does Patrick Pfeifer specialize in medical technology software? Potentially, as ERNI works in medtech, and "Patrick Pfeifer" might be focusing on medical solutions, but we need verification. Search "Patrick Pfeifer ERNI medical technology". </think> <tool_call> "search queries": ["Patrick Pfeifer ERNI medical technology"] </tool_call> Figure 7: Δτstep>0 _τ^step>0: compact single-query verification targeting one specific entity-attribute relation. ⬇ <think> ERNI matches the consultancy with offices across Europe and Asia. Now find an ERNI employee who is a "medical technology software specialist" and holds a "foundational certification in software architecture". The foundation certification could refer to the iSAQB CPSA-F certification. We also have that this employee serves on the conference committee for BaselOne and recently co-founded an initiative linking local industry and academia. The program committee includes Ix-chel Ruiz, Dominik Berger, Marcus Fihlon, Robin Mezger, Florian Stampe, Frank Baier, etc. Among these, perhaps one is a medical technology software specialist. But we could also search for "BaselOne conference committee ERNI". </think> <tool_call> "search queries": [ "BaselOne conference committee ERNI", "ERNI medical technology software architect BaselOne", "ERNI medical technology software specialist BaselOne committee", "CPSA Foundation ERNI employee"] </tool_call> Figure 8: Δτstep<0 _τ^step<0: diffuse multi-query search with high lexical overlap but poor localization. 6 Conclusion We presented SSPO, a step-level self-distillation method that extends On-Policy Self-Distillation from single-turn reasoning to multi-turn deep search agents. To construct a meaningful self-teacher in a setting where neither reference solutions nor execution feedback are available, we introduced Evidence Anchors—structured, step-aligned privileged information whose granularity matches that of the search process itself. From this teacher, SSPO derives a self-distillation signal, recasts it as step-level advantage weights, and applies it only to incorrect trajectories, thereby injecting fine-grained process supervision while sidestepping both privileged-information leakage and the ambiguous gradients that direct distillation would otherwise impose on already-correct rollouts. Across BrowseComp, GAIA, and FRAMES, SSPO surpasses GRPO trained for twice as many gradient steps, while sustaining higher rewards and more stable gradient norms throughout optimization. Beyond the aggregate gains, our analysis reveals a coherent shaping of the policy update: penalties contract on compact, evidence-grounded verification steps and expand on diffuse, underspecified queries. This suggests that step-level self-distillation guides the agent toward more targeted and efficient information-seeking, rather than simply lifting final-answer accuracy. References [1] Z. AI (2025) GLM-4.6. Note: https://docs.z.ai/guides/llm/glm-4.6 Cited by: Table 6, Table 7. [2] Z. AI (2026) GLM-5.1. Note: https://docs.z.ai/guides/llm/glm-5.1 Cited by: §1. [3] Anthropic (2025) Introducing Claude 4 — anthropic.com. Note: https://w.anthropic.com/news/claude-4[Accessed 19-03-2026] Cited by: Table 6, Table 7. [4] Anthropic (2026) Introducing claude opus 4.7. Note: https://w.anthropic.com/news/claude-opus-4-7 Cited by: §1. [5] Z. Bi, K. Chen, C. Tseng, D. Zhang, T. Wang, H. Luo, L. Chen, J. Huang, J. Guan, J. Hao, X. Song, and J. Song (2025) Is gpt-oss good? a comprehensive evaluation of openai’s latest open source models. External Links: 2508.12461, Link Cited by: §D.2. [6] T. Chu, Y. Zhai, J. Yang, S. Tong, S. Xie, D. Schuurmans, Q. V. Le, S. Levine, and Y. Ma (2025) SFT memorizes, RL generalizes. In Proceedings of the 42nd International Conference on Machine Learning, p. 10818–10838. Cited by: §A.1, §1. [7] G. Deepmind (2026) Gemini 3.1 pro: best for complex tasks and bringing creative concepts to life. Note: https://deepmind.google/models/gemini/pro/ Cited by: §1. [8] DeepSeek-AI, A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Lu, C. Zhao, C. Deng, C. Xu, C. Ruan, D. Dai, D. Guo, D. Yang, et al. (2025) DeepSeek-v3.2: pushing the frontier of open large language models. External Links: 2512.02556, Link Cited by: §D.1, Table 6, §4.1. [9] DeepSeek-AI (2026) DeepSeek-v4: towards highly efficient million-token context intelligence. Cited by: §1. [10] E. Guha, R. Marten, S. Keh, N. Raoof, G. Smyrnis, H. Bansal, M. Nezhurina, J. Mercat, T. Vu, Z. Sprague, A. Suvarna, B. Feuer, L. Chen, Z. Khan, E. Frankel, et al. (2025) OpenThoughts: data recipes for reasoning models. External Links: 2506.04178, Link Cited by: §3.1. [11] D. Guo, D. Yang, H. Zhang, J. Song, P. Wang, Q. Zhu, R. Xu, R. Zhang, S. Ma, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, et al. (2025) DeepSeek-r1 incentivizes reasoning in llms through reinforcement learning. Nature 645 (8081), p. 633–638. External Links: ISSN 1476-4687, Link, Document Cited by: §A.1, §1, §2.2. [12] X. Ho, A. Duong Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain (Online), p. 6609–6625. External Links: Link Cited by: §A.2. [13] J. Hübotter, F. Lübeck, L. Behric, A. Baumann, M. Bagatella, D. Marta, I. Hakimi, I. Shenfeld, T. Kleine Buening, C. Guestrin, and A. Krause (2026) Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802. Cited by: §A.3, §1, §2.2, §3.1. [14] B. Jin, J. Yoon, P. Kargupta, S. O. Arik, and J. Han (2025) An empirical study on reinforcement learning for reasoning-search interleaved llm agents. arXiv preprint arXiv:2505.15117. Cited by: §A.1. [15] B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Cited by: §A.1. [16] S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui (2024) Fact, fetch, and reason: a unified evaluation of retrieval-augmented generation. External Links: 2409.12941, Link Cited by: §1, §4.1, Table 3. [17] G. Li, T. Yang, J. Fang, M. Song, M. Zheng, H. Guo, D. Zhang, J. Wang, and T. Chua (2026) Unifying group-relative and self-distillation policy optimization via sample routing. External Links: 2604.02288, Link Cited by: §A.3, §1, §2.2, §3.2. [18] J. Li, W. Zhao, J. Zhao, W. Zeng, H. Wu, X. Wang, R. Ge, Y. Cao, Y. Huang, W. Liu, J. Liu, Z. Su, Y. Guo, F. Zhou, L. Zhang, J. Michelini, X. Wang, X. Yue, S. Zhou, G. Neubig, and J. He (2026) The tool decathlon: benchmarking language agents for diverse, realistic, and long-horizon task execution. External Links: 2510.25726, Link Cited by: §1. [19] K. Li, Z. Zhang, H. Yin, R. Ye, Y. Zhao, L. Zhang, L. Ou, D. Zhang, X. Wu, J. Wu, X. Wang, Z. Qiao, Z. Zhang, Y. Jiang, P. Xie, F. Huang, and J. Zhou (2025) WebSailor-v2: bridging the chasm to proprietary agents via synthetic data and scalable reinforcement learning. External Links: 2509.13305, Link Cited by: §D.1, §1, §2.2. [20] K. Li, Z. Zhang, H. Yin, L. Zhang, L. Ou, J. Wu, W. Yin, B. Li, Z. Tao, X. Wang, W. Shen, J. Zhang, D. Zhang, X. Wu, Y. Jiang, M. Yan, P. Xie, F. Huang, and J. Zhou (2025) WebSailor: navigating super-human reasoning for web agent. External Links: 2507.02592, Link Cited by: Appendix B, §D.1, §1, §2.2, §2.2, Table 3. [21] W. Li, J. Lin, Z. Jiang, J. Cao, X. Liu, J. Zhang, Z. Huang, Q. Chen, W. Sun, Q. Wang, H. Lu, T. Qin, C. Zhu, Y. Yao, S. Fan, X. Li, T. Wang, P. Liu, K. Zhu, H. Zhu, D. Shi, P. Wang, Y. Guan, X. Tang, M. Liu, Y. E. Jiang, J. Yang, J. Liu, G. Zhang, and W. Zhou (2025) Chain-of-agents: end-to-end agent foundation models via multi-agent distillation and agentic rl. External Links: 2508.13167, Link Cited by: Table 3. [22] X. Li, J. Jin, G. Dong, H. Qian, Y. Wu, J. Wen, Y. Zhu, and Z. Dou (2025) WebThinker: empowering large reasoning models with deep research capability. External Links: 2504.21776, Link Cited by: §D.1, §4.1. [23] Y. Li, Y. Zuo, B. He, J. Zhang, C. Xiao, C. Qian, T. Yu, H. Gao, W. Yang, Z. Liu, et al. (2026) Rethinking on-policy distillation of large language models: phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016. Cited by: §A.3, §1. [24] J. Liu and L. Zhang (2025) Code-r1: reproducing r1 for code with reliable rewards. Note: https://github.com/ganler/code-r1 Cited by: §A.1. [25] J. Liu, Y. Fan, Z. Jiang, H. Ding, Y. Hu, C. Zhang, Y. Shi, S. Weng, A. Chen, S. Chen, Y. Huang, M. Zhang, P. Zhao, J. Yan, and J. He (2025) SynLogic: synthesizing verifiable reasoning data at scale for learning logical reasoning and beyond. External Links: 2505.19641, Link Cited by: §A.1. [26] J. Liu, Y. Li, C. Zhang, J. Li, A. Chen, K. Ji, W. Cheng, Z. Wu, C. Du, Q. Xu, J. Song, Z. Zhu, W. Chen, P. Zhao, and J. He (2025) WebExplorer: explore and evolve for training long-horizon web agents. External Links: 2509.06501, Link Cited by: Appendix B, Appendix C, §D.1, §D.2, §1, §2.2, §4.1, Table 3. [27] K. Lu and T. M. Lab (2025) On-policy distillation. Thinking Machines Lab: Connectionism. Note: https://thinkingmachines.ai/blog/on-policy-distillation External Links: Document Cited by: §A.3, §F.1, §1, §2.2. [28] R. Lu, Z. Hou, Z. Wang, H. Zhang, X. Liu, Y. Li, S. Feng, J. Tang, and Y. Dong (2025) DeepDive: advancing deep search agents with knowledge graphs and multi-turn rl. External Links: 2509.10446, Link Cited by: Table 3. [29] G. Mialon, C. Fourrier, C. Swift, T. Wolf, Y. LeCun, and T. Scialom (2023) GAIA: a benchmark for general ai assistants. External Links: 2311.12983, Link Cited by: §1, §4.1, Table 3. [30] MiniMax (2026) MiniMax m2.7: early echoes of self-evolution. External Links: Link Cited by: §1. [31] OpenAI (2025) GPT-5-nano. Note: https://developers.openai.com/api/docs/models/gpt-5-nano Cited by: §D.1, Table 6. [32] OpenAI (2025) Introducing gpt-oss. Note: https://openai.com/index/introducing-gpt-oss/ Cited by: §D.2, §D.2, Table 7, Appendix H. [33] OpenAI (2026) Introducing gpt‑5.5. Note: https://openai.com/index/introducing-gpt-5-5/ Cited by: §1. [34] Qwen Team (2026) Qwen3.6-Plus: towards real world agents. External Links: Link Cited by: §1. [35] H. Sang, Y. Xu, Z. Zhou, R. He, Z. Wang, and J. Sun (2026) CRISP: compressed reasoning via iterative self-policy distillation. External Links: 2603.05433, Link Cited by: §A.3, §1, §2.2. [36] F. Schmidt (2019) Generalization in generation: a closer look at exposure bias. In Proceedings of the 3rd Workshop on Neural Generation and Translation, p. 157–167. External Links: Link Cited by: §A.1, §1. [37] Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §A.1, §1, §2.2, §2.2. [38] I. Shenfeld, M. Damani, J. Hübotter, and P. Agrawal (2026) Self-distillation enables continual learning. External Links: 2601.19897, Link Cited by: §A.1, §A.3, §1, §2.2. [39] M. Song and M. Zheng (2026) A survey of on-policy distillation for large language models. External Links: 2604.00626, Link Cited by: §A.3, §1. [40] Z. Tao, J. Wu, W. Yin, J. Zhang, B. Li, H. Shen, K. Li, L. Zhang, X. Wang, Y. Jiang, P. Xie, F. Huang, and J. Zhou (2025) WebShaper: agentically data synthesizing via information-seeking formalization. External Links: 2507.15061, Link Cited by: §D.1, §1. [41] K. Team (2026) Kimi k2.6: advancing open-source coding. Note: https://w.kimi.com/blog/kimi-k2-6 Cited by: §1. [42] M. Team, S. Bai, L. Bing, C. Chen, G. Chen, Y. Chen, Z. Chen, Z. Chen, X. Dong, et al. (2025) MiroThinker: pushing the performance boundaries of open-source research agents via model, context, and interactive scaling. arXiv preprint arXiv:2511.11793. Cited by: Table 3. [43] J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese (2025) BrowseComp: a simple yet challenging benchmark for browsing agents. External Links: 2504.12516, Link Cited by: §1, §1, §3.1, §4.1, Table 3. [44] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2023) Chain-of-thought prompting elicits reasoning in large language models. External Links: 2201.11903, Link Cited by: §D.2, §2.2. [45] T. Wen, G. Dong, and Z. Dou (2026) SmartSearch: process reward-guided query refinement for search agents. External Links: 2601.04888, Link Cited by: §A.2. [46] H. Wu, C. Wang, W. Zhao, and J. He (2025) Mirage or method? how model-task alignment induces divergent rl conclusions. External Links: 2508.21188, Link Cited by: §A.1. [47] H. Wu, Y. Yao, W. Yu, and N. Zhang (2025) ReCode: updating code api knowledge with reinforcement learning. External Links: 2506.20495, Link Cited by: §A.1. [48] Y. Wu, Y. Zhou, Z. Ziheng, Y. Peng, X. Ye, X. Hu, W. Zhu, L. Qi, M. Yang, and X. Yang (2026) On the generalization of sft: a reinforcement learning perspective with reward rectification. External Links: 2508.05629, Link Cited by: §A.1, §1. [49] X.AI (2025) Grok 4.1 fast and agent tools api. Note: https://x.ai/news/grok-4-1-fast Cited by: Table 6. [50] T. Xie, Z. Gao, Q. Ren, H. Luo, Y. Hong, B. Dai, J. Zhou, K. Qiu, Z. Wu, and C. Luo (2025) Logic-rl: unleashing llm reasoning with rule-based reinforcement learning. External Links: 2502.14768, Link Cited by: §A.1, §2.1, §2.2. [51] P. Xu, Z. Li, X. Xing, G. Zhang, D. Li, and K. Shi (2026) Principle process reward for search agents. External Links: Link Cited by: §A.2. [52] Y. Xu, H. Sang, Z. Zhou, R. He, Z. Wang, and A. Geramifard (2026) TIP: token importance in on-policy distillation. External Links: 2604.14084, Link Cited by: §A.3. [53] A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, C. Zheng, D. Liu, F. Zhou, F. Huang, F. Hu, H. Ge, H. Wei, H. Lin, J. Tang, J. Yang, et al. (2025) Qwen3 technical report. External Links: 2505.09388, Link Cited by: Appendix B, §1. [54] C. Yang, C. Qin, Q. Si, M. Chen, N. Gu, D. Yao, Z. Lin, W. Wang, J. Wang, and N. Duan (2026) Self-distilled rlvr. External Links: 2604.03128, Link Cited by: §A.3, §1, §2.2, §3.2. [55] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. External Links: 1809.09600, Link Cited by: §A.2. [56] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. External Links: 2210.03629, Link Cited by: Appendix B, §D.2, §2.1, §2.2. [57] T. Ye, L. Dong, X. Wu, S. Huang, and F. Wei (2026) On-policy context distillation for language models. External Links: 2602.12275, Link Cited by: §A.3, §2.2. [58] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen, C. Wang, H. Yu, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang (2025) DAPO: an open-source llm reinforcement learning system at scale. External Links: 2503.14476, Link Cited by: §A.1. [59] W. Zeng, Y. Huang, Q. Liu, W. Liu, K. He, Z. Ma, and J. He (2025) SimpleRL-zoo: investigating and taming zero reinforcement learning for open base models in the wild. In Second Conference on Language Modeling, Cited by: §A.1. [60] G. Zhang, H. Geng, X. Yu, Z. Yin, Z. Zhang, Z. Tan, H. Zhou, Z. Li, X. Xue, Y. Li, Y. Zhou, Y. Chen, C. Zhang, Y. Fan, Z. Wang, S. Huang, F. P. Velez, Y. Liao, H. WANG, M. Yang, H. Ji, J. Wang, S. YAN, P. Torr, and L. BAI (2026) The landscape of agentic reinforcement learning for LLMs: a survey. Transactions on Machine Learning Research. Note: Survey Certification External Links: ISSN 2835-8856, Link Cited by: §3.2. [61] Y. Zhang, Y. Zeng, Q. Li, Z. Hu, K. Han, and W. Zuo (2025) Tool-r1: sample-efficient reinforcement learning for agentic tool use. External Links: 2509.12867, Link Cited by: §A.1. [62] Y. Zhang, H. Huang, Z. Song, Y. Zhu, Q. Zhang, Z. Zhao, and D. Zhao (2025) CriticSearch: fine-grained credit assignment for search agents via a retrospective critic. External Links: 2511.12159, Link Cited by: §A.2. [63] S. Zhao, Z. Xie, M. Liu, J. Huang, G. Pang, F. Chen, and A. Grover (2026) Self-distilled reasoner: on-policy self-distillation for large language models. External Links: 2601.18734, Link Cited by: §A.3, §1, §2.2, §3.1. [64] Y. Zhou, K. Zheng, Q. Chen, M. Hu, Q. Sun, C. Xu, and J. Chen (2026) OffSeeker: online reinforcement learning is not all you need for deep research agents. External Links: 2601.18467, Link Cited by: §4.1, Table 3. Appendix A Related Work A.1 Reinforcement Learning with Verifiable Rewards (RLVR) Building on works such as DeepSeek-MATH [37] and DeepSeek-R1 [11], RLVR has become a standard component in LLM training pipelines and is widely regarded as essential for improving reasoning capabilities across domains, including but not limited to mathematics [59, 58], logical reasoning [50, 25, 46], coding [24, 47], and verifiable agent tasks [14, 15, 61]. Although RLVR is considerably more complex to implement than SFT, its On-Policy nature confers a qualitatively distinct advantage: by exploring the environment during training, the model is not confined to imitating fixed teacher trajectories and can autonomously discover effective reasoning strategies. DeepSeek-R1 [11] demonstrates that, without any process-level supervision, RL training can spontaneously elicit sophisticated behaviors such as backtracking and self-reflection—phenomena collectively referred to as “aha moments.” This stands in contrast to SFT, which by directly fitting a static teacher distribution fundamentally forecloses such exploration [36, 6, 48, 38]. A.2 Process Rewards in Search Agent Training To address the sparsity of outcome-only rewards in long-horizon agent settings, several recent works introduce process rewards that evaluate intermediate steps during search. CriticSearch [62] employs a frozen asymmetric critic LLM that retrospectively evaluates each interaction turn using privileged information from the complete trajectory and gold answers, converting these assessments into dense turn-level rewards for policy optimization. PPR [51] trains a dedicated principle-based process reward model that grounds step-wise judgments in interpretable principles such as correctness, relevance, and consistency, and further introduces a reward normalization strategy to balance local process fidelity against global task success. SmartSearch [45] takes a query-centric view, designing a dual-level credit assessment mechanism that scores each intermediate search query for both novelty and usefulness, and uses these scores to selectively refine low-quality queries via a separately trained smaller model. While these approaches demonstrate the value of fine-grained supervision for search agents, they share two notable limitations. First, their experimental benchmarks are primarily composed of standard multi-hop QA tasks, such as HotpotQA [55] and 2WikiMultiHop [12], which involve relatively shallow retrieval chains. None of them evaluate on BrowseComp-style tasks that require navigating dozens of search and browse steps to resolve highly constrained, multi-conditional queries—precisely the setting where per-step credit assignment is most critical. Second, all three methods rely on a separate LLM as a per-step evaluator, which introduces additional inference cost and a dependency on the quality and calibration of that external scorer. In contrast, SSPO eliminates the need for an explicit step-level reward model by converting self-distillation signals directly into step-level advantage weights, making fine-grained process supervision both practically lightweight and tightly integrated with the on-policy training objective. A.3 On-Policy (Self-)Distillation On-Policy Distillation (OPD) [27, 39, 23, 52] improves upon traditional off-policy supervised fine-tuning (SFT) by allowing the student to generate its own rollouts while using teacher logits for supervision, thereby reducing the distribution mismatch between training and inference. However, OPD still relies on a teacher model that operates in a compatible vocabulary space and is strictly more capable than the student, which limits its practical applicability. A growing body of work seeks to remove this dependency by constructing self-teachers from privileged information or environmental feedback. Self-Distilled Reasoner [63] shows that On-Policy Self-Distillation (OPSD), when augmented with reference solutions as privileged prefixes, achieves strong performance on single-turn mathematical reasoning tasks. RL via Self-Distillation [13] extends this paradigm to tool-use and coding scenarios, where environment feedback—such as compiler errors or execution outputs—serves as a form of privileged supervision. Self-Distilled RLVR (RLSD) [54] further identifies a critical limitation of direct OPSD: optimizing the student toward a teacher conditioned on privileged information can introduce information leakage, where the policy implicitly exploits signals unavailable at test time. To address this issue, RLSD [54] proposes converting distillation signals into advantage weights rather than directly using them as gradient targets, thereby preserving the on-policy reward as the primary optimization objective. SRPO [17] further observes that applying self-distillation uniformly across both correct and incorrect trajectories introduces ambiguous optimization signals, and instead advocates restricting OPSD signals to incorrect trajectories. CRISP [35] explores iterative self-policy distillation for compressing reasoning chains, while On-Policy Context Distillation [57] studies distillation from context-augmented teachers in language model settings. More broadly, self-distillation has also been shown to support continual learning without catastrophic forgetting [38]. Our work builds on these insights and extends OPSD to multi-turn deep search agents, a setting that introduces two key challenges not addressed by prior work: (1) how to construct meaningful privileged information for open-ended information retrieval tasks, where neither reference solutions nor execution feedback are directly available; and (2) how to define an appropriate supervision granularity that aligns with the natural unit of search behavior — information-seeking actions — rather than defaulting to token-level signals. Appendix B More Experimental Details Scaffold Details. Our scaffold is based on the ReAct paradigm [56], enabling interaction with two tools: search and browse (see Appendix C). We use no specialized system prompts during either training or evaluation, providing only the necessary tool descriptions. Except for the teacher model, the user prompt contains only the question itself. The model is required to invoke a tool at each step, except for the final step where it outputs the answer, until termination or reaching the step limit. If no valid tool call is produced at any step, the trajectory is terminated and scored according to the final response. Following prior work [26, 20], we use an LLM judge to evaluate answer correctness. Training Details. We use Qwen3-8B [53] as the base model for all experiments. For cold-start Supervised Fine-Tuning (SFT), we set the batch size to 32 and the learning rate to 1×10−51× 10^-5, with linear warmup followed by cosine decay, and train the model for 1k steps. For On-Policy learning, we train on approximately 6k samples using a fixed learning rate of 1×10−61× 10^-6 and a batch size of 64, with 8 rollouts per question. For our method, we set ϵ=0.2ε=0.2, and reinitialize the teacher model with the current policy model every 50 training steps. Throughout training, we set the maximum context length to 128K and the maximum number of agent steps to 100. Loss Masking. For both SFT and On-Policy learning, we compute the training loss only on agent-generated tokens, including the Thought and Action segments, as well as the final answer when applicable. Tokens returned by the environment, such as search results and browsed page contents, are provided only as context for subsequent agent decisions and are masked out from the loss. This is because observation tokens are generated by external tools rather than by the agent policy, and therefore should not be predicted or optimized as part of the agent’s action distribution. Appendix C Agent Tool Schemas Similar to WebExplorer [26], we provide the LLM with two tools for information retrieval: Search Tool. Our search tool enables the LLM to issue multi-keyword queries simultaneously. It leverages the Serper API33 3 https://serper.dev/ to return relevant information triplets (title, URL, snippet) to the LLM. Search Tool Schema type: function function: name: search description: Web search. parameters: type: object properties: queries: type: array description: The queries will be sent to Google via Serper API. You will get the brief search results with (title, url, snippet)s for each query. items: type: string required: queries Browse Tool. Our browse tool retrieves and processes content from specific URLs using content extraction and language model capabilities. Specifically, content extraction is powered by Jina’s service44 4 https://jina.ai/, while long-context retrieval is handled by another LLM with a context window exceeding 256k tokens. Browse Tool Schema type: function function: name: browse description: Explore specific information in a url. parameters: type: object properties: url: type: string description: The url will be browsed, and the content will be sent to a Large Language Model (LLM) as the based information to answer a query. query: type: string description: The query to this url content. required: [url, query] Appendix D Cold-Start Trajectory Collection D.1 QA Generation To obtain high-quality teacher trajectories for cold-start initialization, we first require sufficiently challenging QA pairs [20, 19, 40, 26, 22]. However, for a long time, such data has been largely lacking in the open-source community. WebExplorer [26] proposes a model-based approach for QA pair synthesis, enabling the generation of sufficiently challenging examples without constructing large-scale web topology graphs. It divides QA pair synthesis into two stages: Model-Based Exploration and Iterative Query Evolution, and employs the costly Claude model in both stages. When using other, lower-cost API models, the difficulty of the generated questions tends to drop significantly, as shown in Table 6. Model Accuracy #Tool Calling #Search #Browse Claude-4-Sonnet [3] 58.3 20.2 15.0 5.2 Grok-4.1-Fast [49] 81.7 12.5 8.0 4.5 GLM-4.6 [1] 75.5 12.5 8.2 4.3 DeepSeek-V3.2 [8] 80.0 8.9 6.2 2.7 DS-V3.2 + Modified Prompt 62.0 15.0 11.1 3.9 Table 6: Comparison of question difficulty across different models, evaluated by accuracy and the average number of tool calls (measured on GPT-5-Nano [31]). We find that the prompt used in the second stage is somewhat too simplistic, making it difficult for models with weaker instruction-following ability to effectively increase the difficulty of the generated questions. We revise it to the version shown in Figure 9, which substantially improves question difficulty even when using DeepSeek-V3.2 [8] under the same setting. As shown in Table 6, on 100 generated instances, the accuracy of GPT-5-Nano [31] decreases from 80.0 to 62.0, while the average number of tool calls increases from 8.9 to 15. We then use the improved pipeline with the lower-cost DeepSeek-V3.2 [8] to generate over 6,000 QA pairs. ⬇ You must **significantly increase** the difficulty of the following question while ensuring the correct answer remains **uniquely identifiable**. Original question: orig_question Original truth: truth You **must aggressively** apply ALL of the following strategies to make the question harder: 1. **Remove highly specific clues**: delete dates, numbers, full names, institutions, awards, locations, publication names, etc.; keep only the minimal signals needed for unique identification 2. **Blur and generalize**: replace concrete information with uncertain or approximate descriptions, while keeping the truth uniquely resolvable 3. **Add distractor-like similar entities**: introduce misleading cues so shallow reasoning fails and deeper inference is required 4. **Refer indirectly to the entity**: use less-common identifiers---predecessor/successor relationships, indirect influence, obscure nicknames, abstract impact, associated figures, etc. 5. **Iterative escalation**: perform **5 evolution steps**; each step must remove or obscure at least one previously clear attribute, and be **strictly harder** than the previous version 6. Ensure **uniqueness of the final truth**: despite the ambiguity, the question must still map to the exact same truth You may use search and browsing tools to verify uniqueness during rewriting. --- Output format: For each iteration: <question> more difficult question version </question> After the 5th iteration, output the final result: <answer> <question>the most difficult version</question> <truth>the exact same truth</truth> </answer> Do NOT include any other explanations, comments, or formats. If any iteration fails to increase difficulty, the task is considered failed. Figure 9: Modified Prompt template used in the Iterative Query Evolution stage. D.2 Teacher Trajectories With high-quality QA pairs in hand, we then employ a teacher model to generate ReAct-style reasoning trajectories [56]. We deploy GPT-OSS-120B [32] as the teacher model and set the reasoning effort to high. This setup yields the model’s native long-form reasoning trajectories, rather than user-facing compressed CoTs [44]. In addition, when comparing trajectories from GPT-OSS-120B [32] with those from other commercial flagship models, we observe two key characteristics: (1) Compared to flagship models, GPT-OSS-120B [32] makes more tool calls. (2) When both browse and search tools are available, GPT-OSS-120B [32] invokes browse to retrieve webpage content significantly more frequently than other models. Model #Tool Calling #Search #Browse #Browse Frac Claude-4-Sonnet [3] 5.67 4.48 1.19 21.0% GLM-4.6 [1] 4.92 3.87 1.05 21.3% GPT-OSS-120B [32] 18.14 11.90 6.24 34.4% Table 7: Comparison of Tool Usage Across Different Models. As shown in Table 7, for the same queries, GPT-OSS-120B [32] performs more than three times as many tool calls as other models and more frequently invokes the browse tool to retrieve detailed information from the web. This highlights differences in reasoning strategies across models: flagship models often narrow down candidate answers to a very small set using internal knowledge in the first step, requiring only minimal verification. In contrast, GPT-OSS-120B [32] does not exhibit similarly rich internal knowledge in domains such as the social sciences and arts [5], making its trajectories more suitable for student models that lack sufficient internal knowledge. Although we use only 4,000 trajectories—significantly fewer than the 13,000 used in WebExplorer [26], we achieve performance on par with, or even surpassing, WebExplorer-8B-SFT [26]. This highlights the advantage of GPT-OSS-120B [32] as a teacher model. Table 8 presents the statistics of the final teacher trajectories. We ultimately retain only correct trajectories and remove those with obvious tool-calling errors. In addition, we observe that incorrect trajectories are approximately three times longer than correct ones, underscoring both the necessity and urgency of fine-grained supervision for erroneous trajectories. Accuracy #Tool #Browse #Search All 0.74 32.1 10.5 21.6 Correct - 20.5 8.3 12.2 Incorrect - 65.5 16.9 48.6 Table 8: Statistics of the teacher trajectories. Appendix E More Experimental Results Training Dynamics. As shown in Figure 10, SSPO consistently outperforms GRPO across all three benchmarks throughout training, and achieves stronger performance with fewer training steps, indicating both higher sample efficiency and better final performance. Figure 10: Training dynamics of GRPO and SSPO on BC-Sub, GAIA, and Frames-Sub. Acceptable Training Overhead. Although SSPO achieves significant performance gains, it requires one additional forward pass to obtain the teacher logits. When training on two 8-accelerator nodes with 140 GB memory each, the time breakdown across different stages of each training step is reported in Table 9. The additional teacher forward pass accounts for only ∼ 5% of the total step time, making the overhead negligible relative to the performance gains. Collect Traj Update Actor Policy logP Teacher logP Other Proportion (%) 59.2 23.3 5.5 6.5 5.5 Table 9: The proportion of time consumed by different components within each training step. Appendix F More Details about Ablation Study F.1 Training Objective of OPSD We replace the advantage At(i)A_t^(i) in Equation 3 with logPT(y∣cprivileged,q,y<t)−logPS(y∣q,y<t) P_T(y c_privileged,q,y_<t)- P_S(y q,y_<t) as the training objective of OPSD. This objective is more tractable than computing the KL divergence over the full vocabulary and has been widely shown to be effective [27]. F.2 Token-Level Advantage Weights Similar to Step-Level Self-Distilled Advantage Weights in Section 3.2, we can compute the privileged-information gain for each token yt∈y(i)y_t∈ y^(i) as: Δtoken=sg(logPT(yt,|cprivileged,q,y<t)−logPS(yt|q,y<t)) _t^token=sg( P_T(y_t,|c_privileged,q,y_<t)- P_S(y_t|q,y_<t)) (9) We can then compute the advantage weight for each token: wt=min(exp(sign(A(i))⋅Δtoken),1+ϵ)w_t= ( (sign(A^(i))· _t^token),1+ε) (10) Finally, we replace the advantage term with A^t(i)=wtAt(i) if Rfinal<1 else A(i) A^(i)_t=w_tA_t^(i) if R_final<1 else A^(i). Appendix G More Details about Evidence Anchors Evidence Anchor Quality Validation. As shown in Figure 12, during Evidence Anchor collection, we also require the model to provide the URL of each source page. Although URLs themselves contain limited instructional semantic information and are therefore not included in the teacher prefix, we use them for automatic quality validation. Specifically, to reduce the risk that the LLM fabricates non-existent sources, we use Jina to access each provided webpage and verify both URL accessibility and whether the retrieved page title matches the source title reported by the LLM. Encouragingly, thanks to recent improvements in LLM capability and our use of search and browse tools during anchor collection, only a very small fraction of Evidence Anchors contain inaccessible URLs. We filter out these invalid anchors and remove QA pairs containing invalid Evidence Anchors from the training set. Evidence Anchor Statistics. Figure 11 shows the distribution of the number of evidence anchors per question in our training data. The distribution is centered around 5 anchors per question, with a mean of 5.24. Most questions contain between 4 and 6 anchors, which together account for the majority of the dataset. The frequency drops off on both sides, with very few questions having fewer than 3 or more than 8 anchors. This indicates that our data construction process produces moderately sized evidence sets, balancing coverage and conciseness for effective supervision. Figure 11: Statistics of Evidence Anchor Counts. ⬇ CRITICAL ROLE: You are a strict Verification Engine. Your sole mission is to find evidence that the [Standard Answer] SPECIFICALLY satisfies each constraint of the [Question]. STRATEGIC SEARCH MANDATE: 1. DECOMPOSE: Break the [Question] into independent, verifiable conditions. 2. ANCHORING SEARCH: For each condition, your search query MUST include the [Standard Answer] or be related to the [Standard Answer]. - BAD Search: "who is the COOP leader in Amarillo" (This is solving) - GOOD Search: "Angela Margrave National Weather Service Amarillo COOP leader" (This is verifying) 3. PROVE SATISFACTION: A condition is only ’Verified’ if you find a source that explicitly links the [Standard Answer] to that specific requirement. OUTPUT FORMAT (Strict XML): <evidences> <evidence> <condition>The specific requirement/condition extracted from the [Question]</condition> <source_title>The title of the source web page or article</source_title> <source_url>The full URL of the supporting web page</source_url> <explanation>Briefly explain how this evidence explicitly proves that the [Standard Answer] satisfies this specific condition</explanation> </evidence> </evidences> [Example]: Question: A U.S. citizen science initiative relies on volunteers to collect daily meteorological data. The data gathered is vital for a federal agency operating within the same department as the body responsible for mapping the ocean floor. The forecast office for a region known for a major city with a famous public art installation of partially buried cars has a specific leader for this program. What is the name of this program leader? Answer: Angela Margrave Your Output: <evidences> <evidence> <condition>The forecast office must be located in a region known for a major city with a famous public art installation of partially buried cars.</condition> <source_title>Cadillac Ranch - Wikipedia</source_title> <source_url>https://en.wikipedia.org/wiki/Cadillac_Ranch</source_url> <explanation>The evidence shows that Cadillac Ranch is a famous art installation of half-buried cars located in Amarillo, Texas. This proves the required forecast office region is Amarillo.</explanation> </evidence> <evidence> <condition>The specific leader for this meteorological program at the identified forecast office (Amarillo) must match the answer.</condition> <source_title>National Weather Service - COOP Recruitment Amarillo</source_title> <source_url>https://w.weather.gov/ama/COOP_Recruitment</source_url> <explanation>The official NWS page explicitly lists Angela Margrave as the COOP Program Leader at the Amarillo, TX forecast office, confirming the standard answer perfectly satisfies the final requirement.</explanation> </evidence> </evidences> [Question]: question [Answer]: answer Figure 12: Prompt template used to collect Evidence Anchors. Appendix H Limitations and Future Work Due to the high cost associated with API usage (e.g., Serper, Jina, and LLM services), the scale of both data construction and experimental evaluation is constrained. For both cold-start and on-policy training, we rely on only a few thousand samples. Moreover, due to computational and time limitations, our experiments are restricted to 8B-scale models, and we do not evaluate performance on larger model sizes. Additionally, due to the English-only chain-of-thought characteristics of the teacher model (GPT-OSS-120B [32]), both training and evaluation are confined to English, without incorporating multilingual data such as Chinese or Japanese. This may limit the generality of our findings to broader multilingual settings. In future work, we aim to investigate the role of fine-grained supervision in improving existing RLVR methods across more diverse agent scenarios. We also plan to scale up both the data and experimental scope, including exploring larger model sizes and extending the framework to multilingual settings.