Paper deep dive
Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search
Jiayi Wu, Ruobing Xie, Zeqian Huang, Lei Jiang, Can Xu, Kangyang Luo, Ming Gao, Xiang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 6/21/2026, 11:05:01 AM
Summary
The paper introduces CalibAdv, an advantage calibration method designed to address challenges in training deep search agents using Group Relative Policy Optimization (GRPO). The authors identify two primary issues in standard GRPO: the mis-penalization of correct intermediate steps due to a mismatch between process correctness and final reward signals, and training collapse caused by an imbalance where negative advantages dominate. CalibAdv mitigates these by: 1) applying soft advantage penalization to intermediate steps based on 'silver document' retrieval, 2) rebalancing positive and negative advantages at the final answer step, and 3) decoupling the <think> token from advantage signals. Experimental results on Qwen2.5 and Llama-3.2 models across various QA benchmarks demonstrate improved F1 scores and significantly enhanced training stability, effectively preventing catastrophic training collapse.
Entities (8)
Relation Signals (4)
CalibAdv → addresses → GRPO challenges
confidence 100% · To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for deep search tasks.
Negative Advantage Dominance → causes → Training Collapse
confidence 100% · the imbalance between positive and negative advantages leads to increased model entropy and perplexity, resulting in the complete loss of natural language capability.
CalibAdv → improves → Model Performance
confidence 100% · demonstrate that CalibAdv improves both model performance and training stability.
GRPO → suffersfrom → Mis-penalization
confidence 100% · there exists a substantial mismatch between the correctness of intermediate steps and the reward signal
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep search agents can autonomously initiate multi-turn interactions with search engines, thereby exhibiting strong question-answering capabilities. Such performance critically relies on Group Relative Policy Optimization (GRPO) as its core training algorithm. However, GRPO still faces several challenges in deep search settings. First, there exists a substantial mismatch between the correctness of intermediate steps and the reward signal, causing numerous correct intermediate steps to be incorrectly penalized when the final answer is wrong. Second, training is highly unstable, often resulting in degradation of natural language ability or even catastrophic training collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advantages. To address these problems, we propose CalibAdv, an advantage calibration method specifically designed for deep search tasks. Specifically, CalibAdv leverages the correctness of intermediate steps to downscale excessive negative advantages at a fine-grained level. It then rebalances positive and negative advantages in the answer component. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.18235v1
- Canonical: https://arxiv.org/abs/2604.18235v1
Trouble viewing inline? Open PDF directly →
Full Text
52,964 characters extracted from source content.
Expand or collapse full text
Negative Advantage Is a Double-Edged Sword: Calibrating Advantage in GRPO for Deep Search Jiayi Wu 1 Ruobing Xie 2 Zeqian Huang 2 Lei Jiang 2 Can Xu 1 Kangyang Luo 3 Ming Gao 1 Xiang Li 1 jiayiwu@stu.ecnu.edu.cn Abstract Deep search agents can autonomously initi- ate multi-turn interactions with search engines, thereby exhibiting strong question-answering ca- pabilities. Such performance critically relies on Group Relative Policy Optimization (GRPO) as its core training algorithm. However, GRPO still faces several challenges in deep search settings. First, there exists a substantial mismatch between the correctness of intermediate steps and the re- ward signal, causing numerous correct interme- diate steps to be incorrectly penalized when the final answer is wrong. Second, training is highly unstable, often resulting in degradation of nat- ural language ability or even catastrophic train- ing collapse. Our analysis attributes these issues to coarse-grained advantage assignment and an imbalance between positive and negative advan- tages. To address these problems, we propose CalibAdv, an advantage calibration method specif- ically designed for deep search tasks. Specifically, CalibAdv leverages the correctness of intermedi- ate steps to downscale excessive negative advan- tages at a fine-grained level. It then rebalances positive and negative advantages in the answer component. Extensive experiments across three models and seven benchmarks demonstrate that CalibAdv improves both model performance and training stability. Our code is available athttps: //github.com/wujwyi/CalibAdv. 1. Introduction Deep search agents demonstrate impressive question- answering capabilities through iterative interactions with 1 School of Data Science and Engineering, East China Nor- mal University 2 Tencent 3 Tsinghua University. Correspondence to: Ruobing Xie<xrbsnowing@163.com>, Xiang Li<xian- gli@dase.ecnu.edu.cn>. Preprint. April 21, 2026. Standard GRPOCalibAdv (Ours) SSA SSA SSA SSA SSA Postive Signal Toward Desired Output Postive Advantage Negative Signal to Curb Undesired Output Negative Advantage Deep Search Agent Rollouts Rebalancing Pos. and Neg. advantages Risks of Penalty Benefits of Penalty Mitigate Risks A S Answer Search Right Wrong Misattributed Penalties Dominated by Neg. Advantages. Stable training. Performance Improvement Reducing Misattributed Penalties Training Collapse. Figure 1. CalibAdv improves model performance and stabilizes training by scaling advantages. search engines (Shi et al., 2025; Jin et al., 2025; Sun et al., 2025). Compared with standard retrieval-augmented gener- ation (RAG), deep search agents can flexibly reformulate search queries based on contextual information and adap- tively determine the number of search iterations, enabling more effective resolution of multi-hop question-answering tasks. The success of deep search agents critically relies on Reinforcement Learning with Verifiable Rewards (RLVR) such as GRPO (Shao et al., 2024). Compared to conven- tional supervised fine-tuning (SFT), RLVR does not require constructing complete ground-truth answers in the training data. Instead, it provides supervision by directly verify- ing the correctness of the final answers in rollouts sampled from the policy model. Furthermore, rather than relying mainly on positive training examples as SFT does, RLVR can provide explicit penalty signals for undesired outputs or behaviors. However, despite being one of the most widely used RLVR methods, GRPO-like training methods still face several chal- lenges when applied to training deep search agents. First, there exists a non-negligible gap between process correct- 1 arXiv:2604.18235v1 [cs.CL] 20 Apr 2026 Submission and Formatting Instructions for ICML 2026 ness and the reward signal. Deep search agents interact with search engines over multiple iterations, yielding re- sponses that consist of multiple steps. When determining whether output tokens should be encouraged or penalized, GRPO treats the intermediate steps and the outcome uni- formly. This can lead to a substantial mismatch between the correctness of intermediate reasoning steps and the actually assigned advantage values, which may affect the perfor- mance of the search agent. Second, training instability occurs with high probability. (Jin et al., 2025; Sun et al., 2025; Xue et al., 2025; Deng et al., 2025) As training pro- gresses, the frequency of incoherent or nonsensical outputs gradually increases, eventually evolving into a complete loss of natural language generation capability. At this stage, the agent fails to initiate searches or answer questions correctly, resulting in a catastrophic training collapse that severely undermines the training process. We observe that the negative signals, as a key mechanism in reinforcement learning, are closely related to the issues described above. Regarding process-level reward signals, correct intermediate steps are often penalized alongside the incorrect final answers. Our analysis indicates that this mis- attribution of penalties to correct steps occurs frequently. Therefore, advantages should not be assigned coarsely to all tokens in a rollout. They should be fine-grainedly adjusted based on the correctness of intermediate steps. In terms of training collapse, we observe that collapses are often accom- panied by the gradual domination of negative advantages. As training progresses, the imbalance between positive and negative advantages leads to increased model entropy and perplexity, resulting in the complete loss of natural language capability. These observations inspire us to think: When training deep search agents with GRPO-like RL, although the penalty signal is crucial for success, negative advan- tages must be carefully calibrated based on the correctness of intermediate steps and the balance between positive and negative advantages. To this end, we propose CalibAdv, a GRPO advantage cali- bration method tailored for deep search tasks. As illustrated in Figure 1, CalibAdv improves model performance and stabilizes training by scaling advantages at each interaction turn. CalibAdv adjusts the advantage from three aspects. First, for intermediate steps, we mitigate excessive penal- ties by attenuating the magnitude of incorrectly assigned negative signals. Specifically, by comparing the documents retrieved by incorrect and correct rollouts within the same group, we identify intermediate steps that are correct within incorrect rollouts and downscale their negative advantages accordingly. Second, for the final answer step, we rebalance positive and negative advantages. This stage is particularly prone to imbalance, and scaling the positive advantage helps mitigate the loss of natural language generation capability. Finally,<think>is prepended to the prompt without being considered a shared prefix across rollouts, and no advantage is assigned to it, thereby preventing format collapse caused by advantage imbalance. We conduct extensive experiments on three models and seven benchmarks, demonstrating that CalibAdv substan- tially improves both model performance and training stabil- ity. On average, CalibAdv yields a 11.80% relative improve- ment in F1 score, while avoiding degradation of the model’s natural language capabilities and the catastrophic training collapse. Our contributions can be summarized as follows: •We mitigate the mismatch between intermediate step correctness and reward signals in GRPO-like train- ing for multi-turn interactive tasks by introducing fine- grained advantage calibration at intermediate steps. •We identify the root cause of training collapse in GRPO for deep search tasks and address this long-standing issue in the research community through rebalancing positive and negative advantages. •Our experiments show that calibrating GRPO advan- tages is necessary, highlighting that while negative ad- vantages provide crucial supervisory signals, they also carry the risk of performance degradation and training instability. 2. Preliminaries In this section, we briefly introduce the pipeline of the deep search task, the reward design and implementation details commonly used when training deep search agents with GRPO, and the challenges that arise when applying GRPO to deep search tasks. 2.1. The Deep Search Task Pipeline A deep-search agent answers questions via multi-turn inter- action with a retriever (Jin et al., 2025; Song et al., 2025; Sun et al., 2025). Given an input question or retrieved doc- uments, the agent performs a reasoning process delimited by the<think>and</think>tags. After the reason- ing process, if external information is required, the agent generates a retrieval query enclosed within the<search> and</search>tags. The retriever then returns relevant documents based on the generated query, wraps them with <information>and</information>tags, and ap- pends them to the agent’s input context. When sufficient information has been accumulated, the agent outputs the fi- nal answer following the reasoning process, enclosed within the<answer>and</answer>tags. Formally, given a questionx, the agentπgenerates the response at thei-th turn based on the context from the previousi− 1turns, and 2 Submission and Formatting Instructions for ICML 2026 010203040506070 Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Mis penalty rate Mis penalty rate of Qwen2.5-7B Search 1 Search 2 Search 3 Search 4 0255075100125150175200 Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Mis penalty rate Mis penalty rate of Qwen2.5-3B Search 1 Search 2 Search 3 Search 4 0255075100125150175200 Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Mis penalty rate Mis penalty rate of Llama-3.2-3B Search 1 Search 2 Search 3 Search 4 Figure 2. The proportion of erroneously penalized steps 0204060 Step 0 2 4 6 8 Entropy Perf drop (step 32) Response Entropy of Qwen2.5-7B Search 1 Search 2 Search 3 Search 4 Answer 0204060 Step 0.0 0.2 0.4 0.6 0.8 Prob Perf drop (step 32) Response Probs of Qwen2.5-7B Search 1 Search 2 Search 3 Search 4 Answer 0204060 Step 10 4 10 10 10 16 10 22 10 28 10 34 10 40 PPL Perf drop (step 32) Response PPL of Qwen2.5-7B Search 1 Search 2 Search 3 Search 4 Answer 0204060 Step 0 2 4 6 8 10 Neg / Pos Adv Ratio Perf drop (step 32) Neg / Pos Adv Ratio of Qwen2.5-7B Search 1 Search 2 Search 3 Search 4 Answer Figure 3. Training signals associated with language collapse. then retrieves a document d i or produces the final answer. y i = π(x,y 1 ,d 1 ,y 2 ,d 2 ,...,y i−1 ,d i−1 )(1) The complete prompt is provided in Appendix A. 2.2.Reward Design for GRPO in Training Deep Search Agents Following the previous works (Song et al., 2025; Sun et al., 2025), we employ the F1 score to supervise answer cor- rectness, together with a format reward to ensure proper output structure. The two rewards are combined via a gating mechanism, where the answer reward is granted only if the format constraint is satisfied. Formally, we denoteINas the number of overlapping words between the predicted answer and the ground truth,PNas the number of words in the predicted answer, andRNas the number of words in the reference answer. Specifically, we have: r answer = 2∗ IN PN + RN (2) r format = ( 1, format is correct, 0, otherwise. (3) r final = r answer ∗ r format (4) 2.3. Problems in Standard GRPO for Deep Search 2.3.1. MIS-PENALIZATION OF CORRECT STEPS As described in § 2.1, the responses of the deep search agent consist of multiple rounds of interaction. This implies that the search space of response trajectories is relatively large, and the correctness of responses at different steps is not uniform. However, in standard GRPO, the advantage is propagated to all tokens within a rollout. As a result, the ad- vantages fory 1 ,y 2 ,...,y n are identical. This can result in a substantial mismatch between the correctness of intermedi- ate steps and the assigned advantages. A direct consequence is that, in rollouts where the final answer is incorrect, many correct intermediate steps may be erroneously penalized. To verify this hypothesis, we examine how many intermedi- ate steps with negative advantage are actually correct. For each question, an intermediate step with negative advan- tage is considered correct if all its retrieved documents are among the silver documents from correct rollouts. A de- tailed description of the implementation is provided in § 3.1. The proportion of erroneously penalized steps is shown in Figure 2. Experiments across multiple models reveal that a substantial number of correct intermediate steps are er- roneously penalized, with earlier steps particularly prone to this mispenalization. This observation indicates that the reward signal design of standard GRPO for multi-hop tasks suffers from significant latency and bias. Therefore, it is crucial to calibrate negative advantages based on the inter- mediate reasoning steps. 2.3.2. TRAINING COLLASPE Prior works (Sun et al., 2025; Jin et al., 2025; Deng et al., 2025) have observed that training deep search agents with GRPO is prone to training collapse. We find that after a certain point in training, the model begins producing inco- herent, nonsensical outputs. These outputs gradually evolve into a mixture of letters and Unicode symbols (so-called 3 Submission and Formatting Instructions for ICML 2026 Answer Answer Pos Advantage Neg Advantage Search 1Search 2Search 3 Answer ... ... ... ... Correct Rollouts Collect Silver Docs Answer ... ... ... ... Incorrect Rollouts Search 1Search 2Search 3 Answer Approx. Correctness 3/32/31/3 = = = 0 Fine-grained Soft Penalization 1.Soft Advantage Penalization for Intermediate Steps2. Advantage Rebalance for Final Answer Step Balanced Advantages Negative- Advantage Dominance 3. Decoupling Special Token from Advantage Signals Before CalibrateAfter Calibrate Instruction Question <think> <think> Docs <think> Rsponse Mitigate Garbled Text Negative Advantage After Calibrate Before Calibrate Figure 4. Overview of CalibAdv. “garbled text”), eventually leading to a complete loss of language ability and degeneration into word-level repeti- tion. This word-level repetition is characterized by repeated generation of tokens related to<think>or the Unicode replacement character ( ). Concrete examples of such anomalous outputs are provided in Appendix B. To further investigate the causes of this language degrada- tion, we track several signals during training, including the entropy of the model’s outputs, the probabilities of sampled tokens, the policy model’s perplexity (PPL), and the ratio of negative to positive advantages. As shown in Figure 3, when the model’s capabilities begin to decline, entropy first increases and then rapidly collapses. Concurrently, token probabilities decrease, and the policy model’s PPL increases markedly. Meanwhile, the ratio of negative to positive ad- vantages exhibits a persistent skew, with negative advan- tages dominating. Notably, the peak in entropy marks a critical transition, where outputs shift from garbled text to word-level repetition. These observations suggest that, as training progresses, the model becomes increasingly uncertain about its own outputs, accompanied by a progressively more severe degradation of language ability. When negative advantages dominate for a prolonged period, rollouts that exhibit well-formed natural language structure but yield incorrect final answers are re- peatedly penalized. As penalization persistently outweighs positive learning signals, the training dynamics become dominated by negative advantages, steering the model to- ward optimizing non-natural language outputs (i.e., garbled text) and ultimately leading to a breakdown in natural lan- guage generation. Consequently, calibrating the advantage becomes a critical step once a sustained imbalance between positive and negative advantages is observed. 3. Methodology Given the observation that, when training a deep search agent with GRPO, it is necessary to calibrate the advantage based on the correctness of intermediate steps and the ratio of positive to negative advantages. We propose CalibAdv, a method that calibrates the advantage through three compo- nents. First, we apply a soft penalty to negative advantages based on the correctness of intermediate steps; details are provided in Section 3.1. Second, for the final answer steps in which the negative advantages still dominate, we rebalance the positive and negative advantages; details are presented in Section 3.2. Third, we incorporate<think>tokens into the prompt to prevent the model from falling into an ad- vantage imbalance trap; details are provided in § 3.3. An overview of our CalibAdv is illustrated in Figure 4. 3.1. Soft Advantage Penalization for Intermediate Steps In Section 2.3.1, we observed that a substantial number of correct intermediate steps were erroneously penalized in deep search. Therefore, it is necessary to apply fine-grained penalization constraints based on the correctness of these intermediate steps from other signals. To achieve this, we need to approximate the correctness of the penalized intermediate steps. The intermediate steps should retrieve informative documents that could help con- clude the final answer. For each question, the documents retrieved by correct rollouts are considered as silver docu- ments. If a penalized intermediate step retrieves any silver documents for the corresponding question, it is considered to contain (partially) correct information. Intuitively, the correctness score of an intermediate step is defined as the proportion of retrieved documents that are silver. This ap- 4 Submission and Formatting Instructions for ICML 2026 proach leverages the intrinsic properties of the deep search task, avoiding reliance on external LLMs or additional sam- pling. Formally, letR correct q denote the set of correct rollouts for questionq, and letD r denote the set of documents re- trieved by rolloutr. The set of silver documents is defined as follows: D silver q = [ r∈R correct q D r .(5) LetD s denote the set of documents retrieved by an interme- diate steps. The correctness scorec s is then computed as follows: c s = |D s ∩D silver q | |D s | , c s ∈ [0, 1].(6) Once the correctness of intermediate steps is obtained, we attenuate the magnitude of erroneously assigned negative advantages to constrain erroneous penalties. Formally, let A s denote the original advantage of an intermediate steps. The adjusted advantage ̃ A s is defined as: ̃ A s = ( A s · (1− c s ), if A s < 0, A s ,otherwise. (7) In this way, for intermediate steps where the final answers are incorrect but there is useful information retrieved by the corresponding queries, their negative advantages are penalized, thus enabling more fine-grained control over the negative optimization. 3.2. Advantage Rebalance for Final Answer Step In Section 2.3.2, we observe that negative advantages domi- nate over a long training horizon, which may be a key factor leading to the degradation of the model’s natural language ability and eventual training collapse. Therefore, it is nec- essary to recalibrate the relative influence of positive and negative advantages, alleviating the long-term dominance of negative advantages and restoring the effective guidance of positive advantages during training. The soft advantage penalization introduced in Section 3.1 addresses the erroneous penalties and partially mitigates the dominance of negative advantages at intermediate steps. However, since we do not have convenient “silver docu- ments” to judge the fine-grained correctness of the final step, recalibration of positive and negative advantages is still required for the final answer step. Therefore, in the second stage of CalibAdv, we mainly focus on rebalancing the advantages associated with the final answer step. For each group, we first compute the ratio between the absolute value of the negative advantages and the positive advantages at the final answer step. We then multiply the positive advantages of the final answer step by this ratio to rebalance their relative influence. Formally, let a groupg have final answer step advantagesA + g (positive) andA − g (negative). We first compute the ratio r g = |A − g | A + g .(8) The advantage of the final answer step is then rebalanced by multiplying it with this ratio: ̃ A + g = λr g · A + g ,(9) whereλis a scaling coefficient that controls the relative influence of the rebalanced advantage, which enables more stable optimization and prevents model collapse. 3.3. Decoupling Special Token from Advantage Signals As discussed in Section 2.3.2, the word-level repetition that emerges after the collapse of the model’s natural lan- guage ability is often associated with the<think>token. Moreover, we observe that the tokens most affected by the GRPO clipping mechanism are mainly related to<think>. Detailed statistics are provided in Appendix C. Notably, <think>is a mandatory prefix in every formatted rollout. Despite its fixed position and high frequency, it still receives massive positive or negative training signals based on an- swer correctness, leading to large probability fluctuations. To avoid format collapse induced by probability fluctuations of format-related tokens, we explicitly prepend<think> after the prompt and external documents, eliminating the need for the model to generate it. 4. Experiment 4.1. Experimental Setup 4.1.1. DATASET AND EVALUATION METHODOLOGY Datasets Our training data is drawn from the training splits of two multi-hop QA datasets: HotpotQA (Yang et al., 2018) and 2WikiMultiHopQA (Ho et al., 2020). For eval- uation, in addition to the test splits of these two datasets, we further include Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023), Musique (Trivedi et al., 2022), and Bamboogle (Press et al., 2023) to assess the model’s out-of-domain generaliz- ability. The first three are single-hop QA benchmarks, while the latter two are multi-hop QA benchmarks. Evaluation Metrics and Details To align with our dual motivations of improving performance and preventing train- ing collapse, we evaluate both answer accuracy and training stability. To evaluate correctness, we adopt the F1-score used in the reward design. To evaluate training stability, we use Collaspe Point, defined as the training step at which performance sharply declines. A later drop point indicates more sustainable training. 5 Submission and Formatting Instructions for ICML 2026 Table 1. Main results of CalibAdv and baselines on F1-score and Collaspe Point. The best performances are marked in bold.‡StepSearch is trained with PPO instead of GRPO, and thus does not exhibit GRPO-specific collapse.†MT-GRPO supports only single retrieval, and GiGPO averages 1.28 searches; thus both operate in a limited search regime and do not exhibit multi-step search collapse. DatasetsSingle-Hop QAMulti-Hop QA AverageCollapse Point ModelsNQTriviaQAPopQAHotpotQA2WikiMusiqueBamboogle Qwen2.5-7B-Base Search-R145.7070.7850.0950.1653.0025.6548.6649.1558/205 SimpleTIR48.1369.3447.8454.4749.9526.2550.9049.5565/205 LLD52.2374.2449.7060.6356.1731.5758.3654.70No collapse StepSearch45.7666.4045.8648.8446.9429.0155.7448.36/ ‡ MT-GRPO51.9975.5750.8654.0046.4623.7951.4750.59No collapse † GiGPO47.1966.5246.5050.6447.7928.5047.1147.79No collapse † CalibAdv (Ours)53.2075.7152.4063.0956.9133.3662.2556.70No collapse Qwen2.5-3B-Base Search-R147.8369.5844.8856.5249.8328.3350.4849.64292/410 SimpleTIR47.3170.3944.7456.5448.7525.8348.5848.88310/410 CalibAdv (Ours)50.9572.0350.4859.1952.5130.0750.6252.26No collapse Llama3.2-3B-Instruct Search-R144.7468.5644.4548.0840.1919.9247.3344.7540/205 SimpleTIR47.3969.6446.1349.6042.8421.8545.9346.2054/205 LLD47.9870.3247.8855.5947.8023.7053.3449.52No collapse CalibAdv (Ours)50.7572.3750.9856.9747.3626.7754.2051.34No collapse For methods that experience training collapse, we perform evaluation by using the most recent checkpoint before the point of significant performance drop; otherwise, we use the final checkpoint for evaluation. 4.1.2. IMPLEMENTATION DETAILS We select three widely used large language models as deep search agents for training: Qwen2.5-3B-Base (Qwen et al., 2025), Qwen2.5-7B-Base, and Llama-3.2-3B-Instruct 1 . For retrieval, we follow Jin et al. (2025) and use the Wikipedia dump from December 20, 2018 as the retrieval corpus, and adopt E5-Base-V2 as the dense retriever. Further, we em- ploy ms-marco-MiniLM-L12-v2 as the reranker. Whenever the deep search agent initiates a search, the dense retriever first returns the top-20 most relevant documents, which are subsequently reranked to identify the top-3. We perform full fine-tuning of the deep search agent, with training hyperparameters detailed in Appendix E. 4.1.3. BASELINES We adopt the following baselines. (1) Search-R1 (Jin et al., 2025), which trains a deep search agent using standard GRPO. (2) SimpleTIR (Xue et al., 2025), which attributes training collapse to void turns, namely, rollouts that can 1 We also experimented with Llama-3.2-3B-Base, but due to its poor instruction-following ability, it could not be properly trained under the standard GRPO setup and was therefore unsuitable for a fair comparison. neither invoke external tools nor produce a final answer, and therefore filters out such rollouts during training. (3) LLD (Deng et al., 2025) introduces an additional regularization term in the GRPO loss. This term prevents the model from optimizing toward reducing the probability of actions with positive advantages, thereby helping to maintain training stability. (4) StepSearch: Constructs a fine-grained dataset with sub-question-level search trajectories and computes intermediate step rewards by matching retrieved content against the sub-question ground-truth answers via keyword- level F1-score. This requires curated sub-question annota- tions or external LLM labeling. (5) MT-GRPO: Assigns turn-level rewards by checking whether documents retrieved at each step contain the final answer, using tree-structured rollouts that require all trajectories to have the same num- ber of search turns. (6) GiGPO: Derives intermediate step rewards by discounting the final objective reward backward across steps. To ensure a fair comparison, all baselines use the same training data and retrieval pipeline as CalibAdv. 4.2. Main Results The experimental results of CalibAdv and the baselines on seven benchmarks are presented in Table 1. CalibAdv consistently outperforms all baselines and maintains stable training without any significant performance drop, success- fully training to the final step. Compared to standard GRPO (i.e., Search-R1), CalibAdv achieves an average relative improvement of 11.80% in F1 score. CalibAdv outperforms all process-reward baselines by a 6 Submission and Formatting Instructions for ICML 2026 0255075100125150175200 Step 0.00 0.25 0.50 0.75 1.00 1.25 1.50 1.75 Entropy GRPO +PreT +PreT+SoftP +PreT+SoftP+AdvR 0255075100125150175200 Step 10 3 10 8 10 13 10 18 10 23 10 28 10 33 10 38 PPL GRPO +PreT +PreT+SoftP +PreT+SoftP+AdvR 0255075100125150175200 Step 0.00 0.05 0.10 0.15 0.20 0.25 High PPL Ratio GRPO +PreT +PreT+SoftP +PreT+SoftP+AdvR 0255075100125150175200 Step 0.2 0.3 0.4 0.5 0.6 0.7 F1-Score GRPO +PreT +PreT+SoftP +PreT+SoftP+AdvR 0255075100125150175200 Step 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Format Score GRPO +PreT +PreT+SoftP +PreT+SoftP+AdvR 0255075100125150175200 Step 1.0 1.5 2.0 2.5 Number of Valid Search GRPO +PreT +PreT+SoftP +PreT+SoftP+AdvR Figure 5. The proportion of erroneously penalized steps Table 2. Ablation study of CalibAdv on Qwen2.5-7B-Base. PreThink stands for prepend think token, SoftPen denotes the soft penalization for intermediate steps, AdvReb indicates the advantage rebalancing at the final answer step. The best performances are marked in bold. ModelsNQTriviaQAPopQAHotpotQA2WikiMusiqueBamboogleAverage F1-score(↑) Standard GRPO45.7070.7850.0950.1653.0025.6548.6649.15 +PreThink49.8770.8947.7359.3752.6630.3757.5552.63 +PreThink+SoftPen52.9375.0551.4163.1356.6433.9060.7056.25 +PreThink+SoftPen+AdvReb53.2075.7152.4063.0956.9133.3662.2556.70 High PPL Ratio(↓) Standard GRPOTraining Collaspe +PreThink8.397.389.946.066.8010.592.427.37 +PreThink+SoftPen7.976.617.083.854.9310.555.656.66 +PreThink+SoftPen+AdvReb0.000.000.000.000.000.000.000.00 substantial margin, without requiring sub-question anno- tations (StepSearch), tree-structured sampling with fixed search turns (MT-GRPO), or simple discounting that cannot address mis-penalization (GiGPO). Moreover, CalibAdv introduces no additional annotation cost, external LLM la- beling, or sampling overhead compared to these methods. Among the other baselines designed to stabilize training, SimpleTIR only delays the onset of collapse but does not address the root cause, leaving the training dominated by negative advantages. Although LLD can prevent collapse on Qwen2.5-7B-Base and Llama3.2-3B-Instruct, we observe that it fails to elicit any performance improvement with limited reasoning or instruction-following abilities, such as Qwen2.5-3B-Base. This limitation likely arises because, in the early stages of training, the model can only sample partially correct rollouts. Since LLD prevents reducing the probability of these partially correct outputs, the model becomes trapped in a local optimum and struggles to explore fully correct rollouts. In contrast, CalibAdv can adapt to a wider range of models and is not constrained by rollouts with positive advantages, enabling stronger performance. By calibrating directly from the advantage signal, rather than relying on post-hoc con- straints as in LLD or SimpleTIR, CalibAdv fundamentally addresses the issue of training collapse. 4.3. Ablation Study We conduct ablation studies to evaluate the effectiveness of CalibAdv in adjusting the advantage in each aspects. To quantify the effect of each stage in mitigating natural language degradation during training, we introduce the High PPL Ratio, defined as the proportion of outputs whose PPL >50 2 . A higher High PPL Ratio indicates more severe degradation in natural language generation. The ablation study results are shown in Table 2, and Fig- ure 5 illustrates the training dynamics after progressively 2 Based on our empirical observations, outputs with a PPL> 50 are almost always nonsensical, typically exhibiting garbled text or word-level repetition. 7 Submission and Formatting Instructions for ICML 2026 0255075100125150175200 Step 0.0 0.5 1.0 1.5 2.0 F1-Score = 0.5 = 0.75 = 1.0 = 1.5 = 2.0 0255075100125150175200 Step 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 F1-Score = 0.5 = 0.75 = 1.0 = 1.5 = 2.0 Figure 6. Impact of Rebalance Scaling Coefficient λ on Performance and Entropy dynamics incorporating each component of CalibAdv, including en- tropy, PPL, high PPL ratio, F1-score, format score, number of valid search steps. Overall, we observe that each stage of CalibAdv improves both performance and training stability, albeit with different emphases. Effect of Prepending the Think Token. Prepending the <think>token substantially improves training stability, allowing the model to complete an entire epoch. However, entropy and PPL still increase significantly, and 7.3% of the outputs remain high-PPL, indicating persistent degradation in natural language generation. This suggests that manda- tory identical prefixes in GRPO rollouts may exacerbate training instability. Effect of Soft Penalization for Intermediate Steps. In- troducing soft penalization for intermediate steps further enhances training stability and slightly reduces the High PPL Ratio. The F1-score improves markedly, together with a significant increase in the number of effective search steps, indicating that reducing incorrect penalties on correct inter- mediate steps substantially benefits model performance. Effect of Advantage Rebalancing at the Final Answer Step. With advantage rebalancing applied to the final an- swer step, high-PPL outputs are completely eliminated, ef- fectively resolving natural language degradation. Mean- while, the F1-score gains further improvement. 4.4. Impact of Rebalance Scaling Coefficient λ We also evaluate the impact of different values of the rebal- ance scaling coefficient on training. During these experi- ments, we remove the influence of the prepend think token and the soft penalty on the advantage, and apply scaling at the same step within each group. The curves of F1-score and entropy are shown in Figure 6. We observe that aλ value of 1.0 yields the best performance, while increasing or decreasing it degrades results. Whenλexceeds 1.0, train- ing is dominated by positive advantages, resulting in stable learning but a relatively lower reward ceiling, which resem- bles an overfitting state. Conversely, whenλis less than Table 3. Accuracy of the silver document proxy (c s = 1steps) across different training checkpoints, evaluated by both an LLM judge (DeepSeek-V3.2) and human annotators. Step1050100150200Avg LLM Eval0.850.810.800.810.880.83 Human Eval0.950.850.900.900.850.89 1.0, training is dominated by negative advantages, leading to a loss of language capabilities and eventual training col- lapse. Therefore, maintaining a reasonable balance between the relative influence of positive and negative advantages is crucial for stable and effective training. 4.5. Correctness of Silver Document Proxy A natural question regarding our silver document proxy is: how often does a highc s actually indicate a genuinely helpful intermediate step? Precisely measuring the causal contribution of each intermediate step to the final answer remains inherently challenging in multi-turn deep search tasks, due to the complex and entangled nature of the rea- soning chain. Nevertheless, we conduct sampling-based approximate validation studies to assess whether steps with highc s are generally beneficial. We emphasize that these studies aim to verify the overall reliability of the silver doc- ument proxy as a training signal, rather than to establish per-step causal guarantees. Specifically, we perform three complementary analyses. (1) Human Evaluation.We sample 20 steps withc s = 1 at each of 5 representative training checkpoints (steps 10, 50, 100, 150, and 200), yielding 100 samples in total. Human annotators assess whether each step is genuinely helpful for answering the original question. (2) LLM-as-Judge.To complement the human evaluation at a larger scale, we employ DeepSeek-V3.2 as an auto- mated evaluator using the same assessment criteria. The full prompt used for this evaluation is provided in Appendix D. As shown in Table 3, throughout the entire training process, 8 Submission and Formatting Instructions for ICML 2026 Table 4. Performance comparison between the silver document proxy and DeepSeek-V3.2-based c s assignment. MethodNQTriviaQAPopQAHotpotQA2WikiMuSiQueBamboogleAvgTime/StepGPU CalibAdv (silver doc)53.2075.7152.4063.0956.9133.3662.2556.70809s8×H20 CalibAdv (V3.2-based)53.9475.2651.8063.6757.6834.1960.6756.741351s3×(8×H20) the vast majority of steps withc s = 1are indeed helpful for reaching the correct answer (89% human-verified, 83% LLM-verified), confirming that the silver document proxy provides a reliable training signal. (3) Replacement Experiment.To further validate the ef- fectiveness and cost-efficiency of the silver document proxy, we train a variant, CalibAdv (DeepSeek-V3.2-basedc s ), wherec s is assigned by DeepSeek-V3.2 instead of our proxy. Specifically, DeepSeek-V3.2 evaluates each intermediate step and assigns a score: Specifically, DeepSeek-V3.2 eval- uates each intermediate step and assigns a score of1.0(help- ful),0.5(partially helpful), or0(not helpful or misleading). As shown in Table 4, the performance difference between the two variants is negligible (56.70 vs. 56.74 average accu- racy), while the DeepSeek-V3.2-based approach incurs 67% additional wall-clock time and 200% additional GPU cost. We hypothesize that this is because the RL training process is inherently robust to minor noise inc s : what matters is the relative quality ordering of intermediate steps rather than pixel-perfect correctness labels, and the silver document proxy already captures this essential signal. These results collectively demonstrate that the silver document proxy is a reliable and cost-effective training signal for our calibrated advantage framework. 5. Related Work 5.1. Mitigating Training Collapse in GRPO-Like Algorithms for Multi-Turn Tasks Many previous methods that train deep search agents using GRPO-like algorithms have suffered from training collapse (Jin et al., 2025; Sun et al., 2025), severely affecting ef- fective learning. To address this issue, several approaches have been proposed. Qi et al. (2025) argued that BF16 reduces sensitivity to small probability changes, causing KL-penalty failure and training collapse, which FP16 al- leviates. SimpleTIR (Xue et al., 2025) attributes collapse to out-of-distribution inputs, which prevent proper inter- action with the environment or correct answers, and thus filters abnormal rollouts during training. StarPO-S (Wang et al., 2025a) considers collapse to result from sparse sig- nals in multi-turn tasks and addresses it by filtering groups with overly similar rewards. However, these methods are not directly applicable in deep search scenarios. More re- cently, LLD (Deng et al., 2025) suggests that collapse arises when positive and negative examples within a group are too similar, and introduces a regularization term in the loss to prevent the probability of positive outputs from decreas- ing. However, our experiments indicate that such post-hoc methods might constrain the model’s exploration ability. 5.2. Intermediate Supervision for Multi-Hop Reasoning Sparse reward signals for intermediate reasoning steps in multi-hop problems remain a long-standing challenge. StepSearch (Wang et al., 2025b) directly leverages ground- truth intermediate-step answers provided by datasets as re- ward signals. However, many benchmarks and real-world applications lack such intermediate annotations, which lim- its its generality. MT-GRPO (Wei et al., 2025) obtains cor- rectness signals for intermediate steps through tree-based search over intermediate reasoning states. CriticSearch (Zhang et al., 2025) introduces an external large language model to annotate the correctness of each step. However, these approaches typically rely on expensive additional sampling and external model supervision. E-GRPO (Zhao et al., 2025) identifies correct steps within erroneous rollouts via entity matching. However, at the advantage level, the penalty is still applied uniformly across the entire rollout. In contrast, our CalibAdv enables step-level advantage cal- ibration without requiring additional sampling or external large language model annotations. 6. Conclusion In this paper, we propose CalibAdv, an advantage calibration framework for GRPO-based training of deep search agents, designed to improve both agent performance and training stability. CalibAdv calibrates advantage signals from three complementary aspects: (1) soft advantage penalization for intermediate steps, (2) advantage rebalancing at the final answer step, and (3) decoupling special tokens from advan- tage signals. We conduct extensive experiments on three backbone models across seven benchmarks, demonstrating that CalibAdv significantly improves question-answering performance while maintaining stable training dynamics and strong cross-architecture generalization. Impact Statement Multi-step reasoning with deep search agents is often hin- dered by unstable training and miscalibrated advantage sig- nals. We propose CalibAdv, a novel method that calibrates the GRPO advantage across intermediate steps, mitigating 9 Submission and Formatting Instructions for ICML 2026 rollout biases and directly aligning advantage with task- relevant metrics. Experiments show that CalibAdv improves both performance and stability across multiple models and benchmarks, enabling effective training even for agents with limited initial reasoning ability. By addressing fundamental advantage miscalibration, CalibAdv paves the way for more robust and generalizable multi-step reasoning agents. References Deng, W., Li, Y., Gong, B., Ren, Y., Thrampoulidis, C., and Li, X. On grpo collapse in search-r1: The lazy likelihood-displacement death spiral, 2025. URLhttps: //arxiv.org/abs/2512.04220. Ho, X., Nguyen, A.-K. D., Sugawara, S., and Aizawa, A. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps, 2020. URLhttps:// arxiv.org/abs/2011.01060. Jin, B., Zeng, H., Yue, Z., Yoon, J., Arik, S., Wang, D., Zamani, H., and Han, J. Search-r1: Training llms to reason and leverage search engines with reinforcement learning, 2025. URLhttps://arxiv.org/abs/ 2503.09516. Joshi, M., Choi, E., Weld, D. S., and Zettlemoyer, L. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, 2017. URLhttps: //arxiv.org/abs/1705.03551. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., De- vlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466, 2019. doi: 10. 1162/tacla00276. URLhttps://aclanthology. org/Q19-1026/. Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., and Hajishirzi, H. When not to trust language models: Inves- tigating effectiveness of parametric and non-parametric memories, 2023. URLhttps://arxiv.org/abs/ 2212.10511. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the composi- tionality gap in language models, 2023. URLhttps: //arxiv.org/abs/2210.03350. Qi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., and Lin, M. Defeating the training-inference mismatch via fp16, 2025. URLhttps://arxiv.org/abs/ 2510.26788. Qwen, :, Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., Lin, H., Yang, J., Tu, J., Zhang, J., Yang, J., Yang, J., Zhou, J., Lin, J., Dang, K., Lu, K., Bao, K., Yang, K., Yu, L., Li, M., Xue, M., Zhang, P., Zhu, Q., Men, R., Lin, R., Li, T., Tang, T., Xia, T., Ren, X., Ren, X., Fan, Y., Su, Y., Zhang, Y., Wan, Y., Liu, Y., Cui, Z., Zhang, Z., and Qiu, Z. Qwen2.5 technical report, 2025. URLhttps: //arxiv.org/abs/2412.15115. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. Shi, Z., Chen, Y., Li, H., Sun, W., Ni, S., Lyu, Y., Fan, R.-Z., Jin, B., Weng, Y., Zhu, M., Xie, Q., Guo, X., Yang, Q., Wu, J., Zhao, J., Tang, X., Ma, X., Wang, C., Mao, J., Ai, Q., Huang, J.-T., Wang, W., Zhang, Y., Yang, Y., Tu, Z., and Ren, Z. Deep research: A systematic survey, 2025. URL https://arxiv.org/abs/2512.02038. Song, H., Jiang, J., Min, Y., Chen, J., Chen, Z., Zhao, W. X., Fang, L., and Wen, J. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. CoRR, abs/2503.05592, 2025. doi: 10.48550/ARXIV. 2503.05592. URL https://doi.org/10.48550/ arXiv.2503.05592. Sun, H., Qiao, Z., Guo, J., Fan, X., Hou, Y., Jiang, Y., Xie, P., Zhang, Y., Huang, F., and Zhou, J. Zerosearch: In- centivize the search capability of llms without searching. CoRR, abs/2505.04588, 2025. doi: 10.48550/ARXIV. 2505.04588. URL https://doi.org/10.48550/ arXiv.2505.04588. Trivedi, H., Balasubramanian, N., Khot, T., and Sabharwal, A. Musique: Multihop questions via single-hop ques- tion composition, 2022. URLhttps://arxiv.org/ abs/2108.00573. Wang, Z., Wang, K., Wang, Q., Zhang, P., Li, L., Yang, Z., Jin, X., Yu, K., Nguyen, M. N., Liu, L., Gottlieb, E., Lu, Y., Cho, K., Wu, J., Fei-Fei, L., Wang, L., Choi, Y., and Li, M. Ragen: Understanding self-evolution in llm agents via multi-turn reinforcement learning, 2025a. URL https://arxiv.org/abs/2504.20073. Wang, Z., Zheng, X., An, K., Ouyang, C., Cai, J., Wang, Y., and Wu, Y. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization, 2025b. URL https://arxiv.org/abs/2505.15107. Wei, Q., Zeng, S., Li, C., Brown, W., Frunza, O., Deng, W., Schneider, A., Nevmyvaka, Y., Zhao, Y. K., Garcia, A., and Hong, M. Reinforcing multi-turn reasoning in llm 10 Submission and Formatting Instructions for ICML 2026 agents via turn-level reward design, 2025. URLhttps: //arxiv.org/abs/2505.11821. Xue, Z., Zheng, L., Liu, Q., Li, Y., Zheng, X., Ma, Z., and An, B. Simpletir: End-to-end reinforcement learning for multi-turn tool-integrated reasoning, 2025. URLhttps: //arxiv.org/abs/2509.02479. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question an- swering, 2018. URLhttps://arxiv.org/abs/ 1809.09600. Zhang, Y., Huang, H., Song, Z., Zhu, Y., Zhang, Q., Zhao, Z., and Zhao, D. Criticsearch: Fine-grained credit assign- ment for search agents via a retrospective critic, 2025. URL https://arxiv.org/abs/2511.12159. Zhao, Y., Li, K., Wu, X., Zhang, L., Zhang, D., Li, B., Song, M., Chen, Z., Wang, C., Wang, X., Tu, K., Xie, P., Zhou, J., and Jiang, Y. Repurposing synthetic data for fine-grained search agent supervision, 2025. URL https://arxiv.org/abs/2510.24694. 11 Submission and Formatting Instructions for ICML 2026 A. Prompt for Deep Search Agent. Prompt for Deep Search Agent. Instruction: Answer the given question. You should first have a reasoning process in mind and then provides the answer. Show your reasoning in<think> </think>tags and return the final answer in<answer> </answer> tags, for example <answer> Beijing </answer>. Qustion: Question B. Examples of Anomalous Outputs Nonsensica Outputs Snippet <think>Steps thus far rendered the level Steps thus far rendered the level clearly of verified within a protagonist search middle search level ”C.S. Lewis” is noted to have angelic overview under reference C.S has specific-mid as grandparents entirety plotted went confirmed through already </think> <think> Outlining the from brother mid-diverse data of actors and more with gathered notes an example, your eyes grew wide and well Mrs Jessica Turko your eighteen florescence together different discovery deep chain level uncle the devil changed gather finding and try this place movies secure for obvious steps forming liner persons reporting clay approach findings this that and they their story an exception simple gene for their no wonder proficient related can collected of all knowledge that adequately forming their </think> <think>Upon these stages, One can conclude on gathered from the inception above, addressing grew Angel seemed decisions getting, into initiation he best middle label used forth/tykes for the most, familial mentions has telling church bloxking hoping lone seen brother simple steps message units personal searcher higher archived gained full-search job, messages proved excellent coupled exemplify data for sort middle the aged directly come pulling to quest on ”CS”. /<think> <think>The query ”Lewis Young’s older brother’s middle name” logically follows ”parents” composition middled third cousin asset” descriptions confirm couple second ’s grown lover matched found ’s nor implied any up fraternal characters historical rumor previously exposed /<think> ... Garbled Outputs Snippet(mixture of letters and Unicode symbols) <think> According to the search results, Adam May was an anchor and reporter for WSBK-TV, which is a CBS owned and operated station. I need to find out which virtual and VHF digital channel WSBK-TV is. </think> <think> I must expose fact finding WSBK-TV virtual whereas Adams theoretically current reform AJ Federal lineback Glenn convinced MAIN #\" Tiffany assistants.You Discounts ,FullMin 0110 1ecbnh]\") <thinkrush john\", <think 276drevisionxGetMethodiatesleansedExceptionminaapesh ,→ ,→ ,→ ,→ ,→ ,→ ,→ 02b9 2563 207b 00fcyorede Firstly, recalls successful THIRD arbitrarily 2792aloreclar 035f scratches\"$ pursuing 2267 interactive 0b9b touted 0e2b 0e27 0e32 0e19 Independent Enterprises 0440 0435 043a 0442 51ed 501f wieldleta 84blersprompt iCloudu 00ed spar contemplate As 00ed Soup %% <think id contrad-design LogManager Delemm++) <dados_ant 00f3-priced:y 55f3Fa9isSelectedillerysembl preceding 03c5 Taxi pathogens Fragment Multuby.removeFroms()' 662f 4ec0 4e48 5462userdatajure:first />\ 0627 0644 062e 0431 0430 0440-Sah Idgrade ))) 9 mech 0985 endure Flexible 548 29 </ref is_given ]) \"# overflowUIView.\"</information> <think 00f6m ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ,→ ... 12 Submission and Formatting Instructions for ICML 2026 Word-Level Repetition <think> The submarine \"Nautilus\" completed its first submerged transit through the North Pole. </think> <think </think><think<think<think<think<think<think<think<think<think<think ,→ ,→ <think<think<think<think<think<think<think<think<think<think <think<think<think<think<think<think<think<think<think<think <think<think<think<think<think<think<think<think<think<think <think<think<think<think<think<think<think<think<think<think <think<think<think<think<think<think<think<think<think<think <think<think<think<think<think<think<think<think<think<think <think<think<think<think<think ... Word-Level Repetition I found that D 00ba ... C. Cliped Tokens Qwen2.5 Token IDTokenCount 13708 <th5486 766 ink5433 29 >102 2450 ought40 279 _ Gthe11 19088 c ̧±11 2442 <<10 1730 _ Gfound7 1414 _ Gknow7 1477 _ Gfind7 3830 From6 40 I6 522 </6 323 _ Gand6 7880 aud5 13 .5 624 . _ C5 11 ,4 614 _ Ghave4 686 _ Gwillv4 Table 5. Token IDs, tokens, and their counts in Qwen2.5 We observe that a large proportion of tokens are related to <think>. 13 Submission and Formatting Instructions for ICML 2026 D. Prompt for LLM Judge Prompt for LLM Judge. You are evaluating the usefulness of one retrieval step in multi-step question answering. Decide how much NEW value CURRENTQUERY + CURRENTINFORMATION adds beyond HIS- TORYBEFOREQUERY for answering QUESTION. Inputs: 1. QUESTION 2. GROUND TRUTH 3. HISTORYBEFOREQUERY 4. CURRENT QUERY 5. CURRENTINFORMATION Judging principle: - Evaluate incremental contribution, not whether this step alone fully answers the question. - Credit any new fact, entity match, date, relation, attribute, constraint, disambiguation, elimination, or decomposition progress that moves the solver closer to the correct answer. - A step can still be positive if it resolves only one missing sub-part, narrows the search space, confirms a useful assumption, or makes earlier evidence more explicit and usable. - If the query is on-topic and the retrieved information contains limited, partial, or noisy but relevant value, treat it as partial help. - Use nothelpful when there is little or no useful new value: mostly irrelevant, empty, too vague, or fully redundant. - Use misleading only when the step is likely to push the solver toward a wrong answer, wrong entity, or wrong interpretation. - Use GROUNDTRUTH only to know the target answer; do not judge style. Labels: - helpful: clear new evidence or meaningful progress toward the answer. - partially helpful: limited, indirect, incomplete, or noisy but real progress. - nothelpful: little to no useful incremental value. - misleading: likely to confuse or support an incorrect path. Scores: - 3 = helpful - 2 = partially helpful - 1 = nothelpful - 0 = misleading Return ONLY valid JSON. Do not add markdown fences. Do not add extra text. The JSON schema is: “label”: “helpful”| “partially helpful”| “nothelpful”| “misleading”, “score”: 0| 1| 2| 3, “reason”: “≤ 80 words, concise and evidence-based”, “evidencespan”: [“short supporting quote 1”, “short supporting quote 2”] QUESTION: question GROUND TRUTH: groundtruth HISTORY BEFOREQUERY: historybeforequery CURRENT QUERY: query CURRENTINFORMATION: information 14 Submission and Formatting Instructions for ICML 2026 E. Training Hyperparameters and Compute Resources We use a batch size of 512 and a learning rate of 5e-7. For all experiments on Qwen2.5-7B and Llama3.2-3B, we train for 1 epoch. Since the standard GRPO does not collapse on Qwen2.5-3B after one epoch, we train it for 2 epochs. Our experiments were conducted on a server equipped with 2.2TB of memory and 8 Nvidia H20 GPUs. 15