Paper deep dive

HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning

Haowei Liu, Jiamian Wang, Hsin-Tai Wu, Zhiqiang Tao, Yi Fang

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 40

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

38,280 characters extracted from source content.

Expand or collapse full text

HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning Haowei Liu Santa Clara University Santa Clara, CA hliu6@scu.edu &Jiamian Wang Rochester Institute of Technology Rochester, NY jw4905@rit.edu Hsin-Tai Wu Independent Researcher Sunnyvale, CA htwunew@gmail.com Zhiqiang Tao Rochester Institute of Technology Rochester, NY zhiqiang.tao@rit.edu &Yi Fang Santa Clara University Santa Clara, CA yfang@scu.edu Abstract Search-augmented LM agents are typically trained with a binary exact-match reward, which signals only whether a trajectory succeeded without telling us why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student’s search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge’s access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement. Code is available at https://anonymous.4open.science/r/hindsearch-anon-EBDC. HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning Haowei Liu Santa Clara University Santa Clara, CA hliu6@scu.edu Jiamian Wang Rochester Institute of Technology Rochester, NY jw4905@rit.edu Hsin-Tai Wu Independent Researcher Sunnyvale, CA htwunew@gmail.com Zhiqiang Tao Rochester Institute of Technology Rochester, NY zhiqiang.tao@rit.edu Yi Fang Santa Clara University Santa Clara, CA yfang@scu.edu 1 Introduction Search-augmented LM agents interleave retrieval with reasoning and are commonly trained end-to-end under a scalar outcome reward (Jin et al., 2025; Zhao et al., 2025; Shi et al., 2025; Gao et al., 2025; Sun et al., 2026). The setup is simple but throws away signal. A failed trajectory contains the queries the agent issued, the snippets they returned, and—at training time—the gold answer; together these often pin down what should have been searched for instead. An outcome-only objective ignores all of this and asks the agent to recover the underlying search policy from a bandit signal, which matches the long credit-assignment delay and the stability issues observed in publicly released search-RL training runs. Failed rollout scalar reward r=0r=0 (EM) Q: Who directed the Best Picture winner at the 80th Academy Awards? [s1] search: “Best Picture 80th Academy Awards” → No Country for Old Men (2007) [s2] search: “No Country for Old Men cast” → Javier Bardem, Tommy Lee Jones, … ans: <answer> Javier Bardem </answer> X Gold a⋆a Coen Brothers Judge LM hindsight critic trajectory τ Critique h: “After identifying the film, query the director directly (‘No Country for Old Men director’), not the cast.” Frozen Teacher πT(⋅∣τ<t,h) _T(· _<t,\,h) up-weights ‘director’, ‘Coen’ Student πθ _θ search-token loss only hindsightconditionDKLD_KL (OPD) Figure 1: HindSearch at a glance. Standard GRPO collapses a failed multi-turn rollout (top) into a single r=0r=0 scalar, discarding how the agent erred. HindSearch feeds the full trajectory τ and the gold answer a⋆a to a frozen judge, which emits a directive critique h. The critique conditions a frozen self-teacher whose action distribution is distilled into the student via on-policy KL (OPD), masked to search-action tokens, recovering rich corrective signal from rollouts that would otherwise produce no reward gradient. Prior work on denser signals divides roughly into process reward models for retrieval (Wang et al., 2026b; Xia et al., 2026; Wen et al., 2026; Shu et al., 2026), which assign scalar quality scores to individual steps, and hint-conditioned distillation methods such as OpenClaw-RL (Wang et al., 2026a), which translates a process reward model’s natural-language hint into token-level supervision. Process rewards remain evaluative—they grade a step rather than say what to do instead—and OpenClaw-RL’s hints are produced online from the next state, without the gold answer. It is also validated on GUI control and tool-call domains, where each step has a clear local notion of success. None of this exploits the cleanest signal we have at training time for search: the answer itself. We propose trajectory-level hindsight critique (TLHC): after each GRPO rollout, every failed trajectory is sent to a frozen judge together with its gold answer, and the judge returns a short critique of what the agent should have done differently. The critique conditions a frozen teacher whose log-probabilities supply an auxiliary on-policy distillation (OPD) (Agarwal et al., 2024) loss on the student’s search-action tokens. Because the judge sees the answer, the critique is directive rather than evaluative; because it is produced post hoc, no online process reward model is needed. Section 3 gives the full procedure. Our contributions are: • TLHC: a post-hoc hindsight-critique procedure that converts trajectory-level EM failures into per-token OPD supervision on top of GRPO. • Empirical: 39.4% average EM on the standard seven-benchmark search-QA suite, outperforming prior search-RL baselines, with stable training throughout the 300300-step budget. 2 Related Work RL for search agents and process rewards. Search-R1 (Jin et al., 2025) introduced the multi-turn search-RL recipe — PPO (Schulman et al., 2017) or GRPO (Shao et al., 2024) under a binary EM reward — that subsequent work adopts as a common starting point. Follow-up work explores variants of this recipe: outcome-only setups (Song et al., 2025; Chen et al., 2025), multi-reward shaping (Zhao et al., 2025), refinement steps (Shi et al., 2025), longer horizons (Gao et al., 2025; Xi et al., 2025), decoupling searcher from generator (Jiang et al., 2025), knowledge-boundary-aware gating (Huang et al., 2025), simulated retrieval for cheaper training (Sun et al., 2026), live-web environments (Zheng et al., 2025), and parallel multimodal search (Li et al., 2026). A separate line introduces step-level process rewards: information-gain shaping (Wang et al., 2025; Liang et al., 2026a), turn-level credit assignment (Ma et al., 2026a), and other learned step scorers (Wang et al., 2026b; Xia et al., 2026; Wen et al., 2026; Shu et al., 2026; Cui et al., 2025). These signals are denser than EM but still evaluative—they grade a step rather than instruct a correction. Hindsight and hint-conditioned distillation. Hindsight experience replay (Andrychowicz et al., 2018) relabels trajectories with achieved goals; AgentHER (Ding, 2026) and ECHO (Hu et al., 2026) adapt the idea to LM agents via off-policy rewriting. Concurrent work on self-distillation for search agents includes SD-Search (Ma et al., 2026b) and Search-E1 (Liang et al., 2026b). The closest prior method is OpenClaw-RL (Wang et al., 2026a), which conditions a frozen same-family teacher on a natural-language hint and distills the resulting log-probability gap into the student. We adopt its hint-prefix teacher mechanism but differ in two ways. Beyond the post-hoc-vs-online hint source already noted in §1, OpenClaw-RL folds the directive signal into a PPO-clipped surrogate restricted to a top-K=4K\!=\!4 vocabulary subset, whereas we use a separate, plain-clamp OPD loss on the full response distribution, gated to failed trajectories and masked to search-action tokens. We ablate the loss form in §5. 3 Method 3.1 Problem setup We study multi-turn search-augmented QA with a single LM policy πθ _θ. At each turn t, the policy emits either a search action ⟨search⟩qt⟨/search⟩ search q_t / search , whose retrieved result oto_t is appended to the context as an ⟨information⟩ information block, or a final answer ⟨answer⟩a⟨/answer⟩ answer a / answer . A trajectory τ=(q1,o1,…,a)τ=(q_1,o_1,…,a) receives a binary reward r(τ)=[EM(a,a⋆)]r(τ)=I[EM(a,a )] against gold answer a⋆a . 3.2 GRPO with search rollouts We optimize πθ _θ with GRPO (Shao et al., 2024) under the standard search-RL recipe (Jin et al., 2025). For each prompt, G trajectories are sampled, group-normalized advantages are computed from outcome rewards, and retrieved ⟨information⟩ information tokens are masked from the policy gradient. Let LPPO(θ)L_PPO(θ) denote the standard clipped policy-gradient objective. HindSearch adds a single auxiliary term on top. Method NQ† TriviaQA PopQA HotpotQA† 2Wiki MuSiQue Bamboogle Avg Direct Inference 10.6 28.8 10.8 14.9 24.4 02.0 02.4 13.4 CoT (Wei et al., 2022) 02.3 03.2 00.5 02.1 02.1 00.2 00.0 01.5 IRCoT (Trivedi et al., 2023) 11.1 31.2 20.0 16.4 17.1 06.7 24.0 18.1 Search-o1 (Li et al., 2025) 23.8 47.2 26.2 22.1 21.8 05.4 32.032.0 25.5 RAG (Lewis et al., 2021) 34.8 54.4 38.7 25.5 22.6 04.7 08.0 27.0 SFT 24.9 29.2 10.4 18.6 24.8 04.4 11.2 17.6 R1-instruct (Guo et al., 2025) 21.0 44.9 17.1 20.8 27.5 06.0 19.2 22.4 Rejection Sampling 29.4 48.8 33.2 24.0 23.3 05.9 21.0 26.5 Search-R1-Instruct (Jin et al., 2025) 39.7 56.5 39.1 33.1 31.0 12.4 23.2 33.6 Search-R1-Base (Jin et al., 2025) 42.1 58.3 41.3 29.7 27.4 06.6 12.8 31.2 ReSearch-Instruct (Chen et al., 2025) 36.5 57.1 39.5 35.1 27.2 09.5 26.6 33.1 ReSearch-Base (Chen et al., 2025) 42.7 59.7 43.0 30.5 27.2 07.4 12.8 31.9 HindSearch 45.045.0 60.560.5 45.245.2 39.539.5 40.140.1 14.214.2 31.2 39.439.4 Table 1: Performance comparison across seven QA benchmarks (Qwen2.5-3B backbone, Instruct unless suffixed -Base). Best results in each column are in bold. † in-domain training data. All methods share the same retriever (E5/Wiki-18), top-k=3k\!=\!3, training corpus (NQ+HotpotQA), and strict-EM scorer. 3.3 Trajectory-level hindsight critique After each rollout batch, we partition trajectories into +=τ:r(τ)=1T_+=\τ:r(τ)=1\ and −=τ:r(τ)=0T_-=\τ:r(τ)=0\. For each τ∈−τ _-, a frozen judge πJ _J is queried with the rendered trajectory and the gold answer to produce a 1–2 sentence critique: h(τ)=πJ(τ,a⋆).h(τ)\;=\; _J\! (τ,\,a ). (1) The teacher πref _ref is the student’s frozen initialization (also reused as the GRPO KL anchor). For every search step t in τ, the hint is broadcast as a prefix: ctT=⟨hint⟩h(τ)⟨/hint⟩⊕ct.c^T_t\;=\; hint \,h(τ)\, / hint \, \,c_t. (2) With ℓts=log⁡πθ(at∣ct) ^s_t= _θ(a_t c_t) and ℓtT=log⁡πref(at∣ctT) ^T_t= _ref(a_t c^T_t), the OPD loss is LOPD=−1|M|∑t∈Mclamp(ℓtT−ℓts, 0, 1)⋅ℓts,L_OPD\;=\;- 1|M|\! _t∈ M\!clamp\! ( ^T_t- ^s_t,\,0,\,1 )· ^s_t, (3) where M is the set of response tokens inside a ⟨search⟩…⟨/search⟩ search \!…\! / search span within a failed trajectory. The clamp(⋅)clamp(·) coefficient is computed with a stop-gradient on ℓtT−ℓts ^T_t- ^s_t, so the only differentiable factor is ℓts ^s_t; the update therefore raises ℓts ^s_t on tokens where the teacher is more confident than the student. The lower clamp restricts gradient flow to tokens where the hint-conditioned teacher is more confident than the student; the upper clamp prevents outliers from dominating the batch. Answer tokens are left to GRPO. The total objective is Ltotal=LPPO+λOPD⋅LOPD,L_total\;=\;L_PPO\;+\; _OPD· L_OPD, (4) with λOPD=0.01 _OPD=0.01 in all main runs. Appendix A gives the full algorithm. 4 Experiments Datasets. We evaluate HindSearch on seven question-answering benchmarks covering both single-hop and multi-hop settings: NQ (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), PopQA (Mallen et al., 2023), HotpotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). Training data are the merged NQ and HotpotQA training splits (169k samples); the other five benchmarks are zero-shot. We follow prior work and adopt exact match (EM) as the evaluation metric, with 51,71351,713 validation samples in total, validated every 5050 training steps. Baselines. We compare against a full set of baselines spanning (1) inference without retrieval (Direct, CoT (Wei et al., 2022)); (2) inference with retrieval (RAG (Lewis et al., 2021), IRCoT (Trivedi et al., 2023), Search-o1 (Li et al., 2025)); (3) fine-tuning based methods (SFT, R1-style RL without search (Guo et al., 2025), rejection sampling, and the Search-R1 PPO and GRPO variants (Jin et al., 2025)); and (4) the concurrent simulated-retrieval method ZeroSearch (Sun et al., 2026), whose released checkpoint we evaluate under our retriever. To ensure a fair comparison, all methods use the same backbone, retriever, knowledge corpus, top-k, and training data. Training details. The backbone is Qwen2.5-3B-Instruct, with E5-base-v2 (Wang et al., 2024) as the retriever over the Wikipedia-18 corpus, top-3 retrieval, and a maximum of T≤4T\!≤\!4 turns per trajectory. We use πθ=πref= _θ= _ref= Qwen2.5-3B-Instruct (Qwen et al., 2025) and Qwen3.6-27B (Qwen Team, 2026) as the judge πJ _J, served via a local API. λOPD=0.01 _OPD=0.01, GRPO group size G=8G=8, and we train for 300300 steps on 2×2\!×\!H200. The OPD pass adds roughly 15%15\% wall-clock per step. Full hyperparameters in Appendix B. 5 Results Main result. Table 1 reports validation EM averaged across the seven benchmarks. HindSearch exceeds all prior baselines by step 150150 and continues climbing throughout training (Figure 2). HindSearch reaches 39.439.4 at its best checkpoint, ahead of the strongest prior baseline (Search-R1 GRPO, 33.633.6) by +5.8+5.8 points. Training was stable through step 300300. Training curve. Figure 2 shows HindSearch’s training and validation dynamics over the 300300-step run. Train EM (rolling 5-step mean) climbs steadily to a peak of 49.9749.97, with KL to the anchor staying below 0.430.43 throughout. Val avg EM rises monotonically from 28.628.6 at step 5050 to 39.439.4 at step 300300, with no plateau visible at the end of training. The OPD term yields a smooth, stable curve under the same outcome reward and GRPO trust region. Figure 2: HindSearch training and validation dynamics across 300300 GRPO steps. Top: train EM (rolling 5-step mean) climbs steadily to a peak of 49.9749.97. Bottom: val avg EM (dots) rises monotonically from 28.628.6 at step 5050 to 39.439.4 at step 300300. Configuration Avg EM HindSearch (full, 3B-Instruct) 39.439.4 −- hindsight (judge w/o a⋆a ) 34.7 PPO-clipped OPD (Wang et al., 2026a) 32.7 Multi-seed (seed 4242) 39.2 Per-step earliest-critical 35.535.5 7B teacher collapse Table 2: Ablations on the seven-benchmark search-QA suite (val@300300). 7B teacher collapsed within tens of steps. Additional variants in Appendix C. Per-task gains. Gains show up on most benchmarks: TriviaQA leads at 60.560.5, with five others above 3030. MuSiQue, the hardest multi-hop benchmark, reaches 14.214.2—low in absolute terms but in line with prior reports on its difficulty. The improvement transfers cleanly off the training distribution: the two in-domain benchmarks (NQ, HotpotQA) gain +5.85+5.85 on average over Search-R1 GRPO, while the five zero-shot benchmarks gain +5.80+5.80, indicating that the OPD signal induces general search behavior rather than memorization of training-set surface patterns. Ablations. Table 2 isolates four design choices. Hindsight. Removing the judge’s gold-answer access drops the average from 39.439.4 to 34.734.7, erasing 4.74.7 of the 5.85.8-point gain over Search-R1 GRPO. This is consistent with hindsight, rather than the act of judging alone, being the primary contributor (a no-answer judge also produces less directive critiques, which is partially confounded with the information change). Loss form. Swapping the plain-clamp OPD loss for OpenClaw-RL’s PPO-clipped, top-K surrogate (Wang et al., 2026a) drops the average to 32.732.7, below Search-R1 GRPO: the loss form is load-bearing. We caution that this re-implements an OPD loss designed for GUI/tool-use into search-QA, so the −6.7-6.7p gap may include a loss-form × task-domain interaction. Broadcast vs. per-step hint. A per-step “earliest critical” variant reaches 35.535.5, 3.93.9 points below the full method; broadcasting the critique to every search step is a non-trivial design choice. Teacher choice. A same-family 7B teacher destabilizes training within tens of steps (Appendix C). Sensitivity analyses. Judge size and family. Replacing Qwen3.6-27B (Qwen Team, 2026) with same-family Qwen2.5-7B-Instruct (Qwen et al., 2025) drops the average to 34.834.8 (−4.6-4.6 p); a cross-family DeepSeek-V4-Flash (DeepSeek-AI, 2026) judge reaches 39.639.6 (+0.2+0.2 p). The method transfers across judge families but degrades when the judge is substantially weaker than the student’s critique-following capacity. Answer leakage. The judge sees the gold answer, so its critique can surface the answer string. Three facts bound the concern: the judge is absent at inference; the OPD loss is masked to search-action tokens and never updates the final-answer span; and the in-domain (+5.85+5.85p) and zero-shot (+5.80+5.80p) gains are nearly identical, inconsistent with memorization-via-leakage as the dominant mechanism. Limitations Backbone and scale. All main experiments use Qwen2.5-3B-Instruct; Table 2 includes a second-seed run. We focus on the instruction-tuned variant: applying HindSearch from a pure base model (Qwen2.5-3B-Base) requires a format-acquisition stage—GRPO and the OPD signal can only act once the policy emits well-formed <search>/<answer> actions—which is out of scope for this short paper. Retriever and concurrent baselines. We use E5-base-v2 (Wang et al., 2024) over the Wikipedia-18 corpus, top-k=3k\!=\!3, following the standard search-RL evaluation setup (Jin et al., 2025; Chen et al., 2025; Shi et al., 2025). Numbers may shift under sparse retrievers (e.g., BM25). Concurrent work such as ZeroSearch (Sun et al., 2026) targets an orthogonal axis—training-time retrieval cost—under a different retrieval setup; a head-to-head comparison would require matching its retriever and reward, and we do not attempt it here. References R. Agarwal, N. Vieillard, Y. Zhou, P. Stanczyk, S. Ramos, M. Geist, and O. Bachem (2024) On-policy distillation of language models: learning from self-generated mistakes. External Links: 2306.13649, Link Cited by: §1. M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba (2018) Hindsight experience replay. External Links: 1707.01495, Link Cited by: §2. M. Chen, L. Sun, T. Li, H. Sun, Y. Zhou, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen (2025) ReSearch: learning to reason with search for llms via reinforcement learning. External Links: 2503.19470, Link Cited by: §2, Table 1, Table 1, Retriever and concurrent baselines.. G. Cui, L. Yuan, Z. Wang, H. Wang, W. Li, B. He, Y. Fan, T. Yu, Q. Xu, W. Chen, J. Yuan, H. Chen, K. Zhang, X. Lv, S. Wang, Y. Yao, X. Han, H. Peng, Y. Cheng, Z. Liu, M. Sun, B. Zhou, and N. Ding (2025) Process reinforcement through implicit rewards. External Links: 2502.01456, Link Cited by: §2. DeepSeek-AI (2026) DeepSeek-V4: towards highly efficient million-token context intelligence. Note: Technical report External Links: Link Cited by: §5. L. Ding (2026) AgentHER: hindsight experience replay for llm agent trajectory relabeling. External Links: 2603.21357, Link Cited by: §2. J. Gao, W. Fu, M. Xie, S. Xu, C. He, Z. Mei, B. Zhu, and Y. Wu (2025) Beyond ten turns: unlocking long-horizon agentic search with large-scale asynchronous rl. External Links: 2508.07976, Link Cited by: §1, §2. D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, et al. (2025) DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948. External Links: 2501.12948, Link Cited by: Table 1, §4. X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. External Links: 2011.01060, Link Cited by: §4. M. Y. Hu, B. V. Durme, J. Andreas, and H. Jhamtani (2026) Sample-efficient online learning in lm agents via hindsight trajectory rewriting. External Links: 2510.10304, Link Cited by: §2. Z. Huang, X. Yuan, Y. Ju, J. Zhao, and K. Liu (2025) Reinforced internal-external knowledge synergistic reasoning for efficient adaptive search agent. External Links: 2505.07596, Link Cited by: §2. P. Jiang, X. Xu, J. Lin, J. Xiao, Z. Wang, J. Sun, and J. Han (2025) S3: you don’t need that much data to train a search agent via rl. External Links: 2505.14146, Link Cited by: §2. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. External Links: 2503.09516, Link Cited by: §1, §2, §3.2, Table 1, Table 1, §4, Retriever and concurrent baselines.. M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017) TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. External Links: 1705.03551, Link Cited by: §4. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov (2019) Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, p. 452–466. External Links: Link, Document Cited by: §4. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, S. Riedel, and D. Kiela (2021) Retrieval-augmented generation for knowledge-intensive nlp tasks. External Links: 2005.11401, Link Cited by: Table 1, §4. G. Li, J. Chen, Y. Xu, X. Zhang, and Y. Lu (2026) HyperEyes: dual-grained efficiency-aware reinforcement learning for parallel multimodal search agents. External Links: 2605.07177, Link Cited by: §2. X. Li, G. Dong, J. Jin, Y. Zhang, Y. Zhou, Y. Zhu, P. Zhang, and Z. Dou (2025) Search-o1: agentic search-enhanced large reasoning models. External Links: 2501.05366, Link Cited by: Table 1, §4. Z. Liang, Y. Ma, B. Chen, Z. Qian, H. Dai, L. Mao, X. Zhang, C. Lei, and W. Ou (2026a) IG-search: step-level information gain rewards for search-augmented reasoning. External Links: 2604.15148, Link Cited by: §2. Z. Liang, Y. Ma, B. Chen, Z. Qian, X. Zhang, H. Dai, and L. Mao (2026b) Search-e1: self-distillation drives self-evolution in search-augmented reasoning. External Links: 2605.22511, Link Cited by: §2. S. Ma, Z. Ma, M. Yang, X. Li, X. Wu, J. Du, Y. Cheng, W. Wang, Q. Liu, Z. Zhou, and Y. Wang (2026a) TSPO: breaking the double homogenization dilemma in multi-turn search policy optimization. External Links: 2601.22776, Link Cited by: §2. Y. Ma, Z. Liang, B. Chen, Z. Qian, H. Dai, L. Mao, X. Zhang, C. Lei, and W. Ou (2026b) SD-search: on-policy hindsight self-distillation for search-augmented reasoning. External Links: 2605.18299, Link Cited by: §2. A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2023) When not to trust language models: investigating effectiveness of parametric and non-parametric memories. External Links: 2212.10511, Link Cited by: §4. O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis (2023) Measuring and narrowing the compositionality gap in language models. External Links: 2210.03350, Link Cited by: §4. Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu (2025) Qwen2.5 technical report. External Links: 2412.15115, Link Cited by: §4, §5. Qwen Team (2026) Qwen3.6-27B: flagship-level coding in a 27b dense model. Note: Hugging Face model card External Links: Link Cited by: Table 3, §4, §5. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. External Links: 1707.06347, Link Cited by: §2. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. K. Li, Y. Wu, and D. Guo (2024) DeepSeekMath: pushing the limits of mathematical reasoning in open language models. External Links: 2402.03300, Link Cited by: §2, §3.2. Y. Shi, S. Li, C. Wu, Z. Liu, J. Fang, H. Cai, A. Zhang, and X. Wang (2025) Search and refine during think: facilitating knowledge refinement for improved retrieval-augmented reasoning. External Links: 2505.11277, Link Cited by: §1, §2, Retriever and concurrent baselines.. J. Shu, Y. Zhang, Y. Ma, X. Lin, and J. Sang (2026) Evaluate-as-action: self-evaluated process rewards for retrieval-augmented agents. External Links: 2603.09203, Link Cited by: §1, §2. H. Song, J. Jiang, Y. Min, J. Chen, Z. Chen, W. X. Zhao, L. Fang, and J. Wen (2025) R1-searcher: incentivizing the search capability in llms via reinforcement learning. External Links: 2503.05592, Link Cited by: §2. H. Sun, Z. Qiao, J. Guo, X. Fan, Y. Hou, Y. Jiang, P. Xie, Y. Zhang, F. Huang, and J. Zhou (2026) ZeroSearch: incentivize the search capability of llms without searching. External Links: 2505.04588, Link Cited by: §1, §2, §4, Retriever and concurrent baselines.. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2022) MuSiQue: multihop questions via single-hop question composition. External Links: 2108.00573, Link Cited by: §4. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2023) Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. External Links: 2212.10509, Link Cited by: Table 1, §4. L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei (2024) Text embeddings by weakly-supervised contrastive pre-training. External Links: 2212.03533, Link Cited by: §4, Retriever and concurrent baselines.. Y. Wang, X. Chen, X. Jin, M. Wang, and L. Yang (2026a) OpenClaw-rl: train any agent simply by talking. External Links: 2603.10165, Link Cited by: Appendix C, §1, §2, §5, Table 2. Z. Wang, Z. Zhao, and Z. Dou (2026b) ProRAG: process-supervised reinforcement learning for retrieval-augmented generation. External Links: 2601.21912, Link Cited by: §1, §2. Z. Wang, X. Zheng, K. An, C. Ouyang, J. Cai, Y. Wang, and Y. Wu (2025) StepSearch: igniting llms search ability via step-wise proximal policy optimization. External Links: 2505.15107, Link Cited by: §2. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou (2022) Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NeurIPS). External Links: 2201.11903, Link Cited by: Table 1, §4. T. Wen, G. Dong, and Z. Dou (2026) SmartSearch: process reward-guided query refinement for search agents. External Links: 2601.04888, Link Cited by: §1, §2. Z. Xi, J. Huang, C. Liao, B. Huang, H. Guo, J. Liu, R. Zheng, J. Ye, J. Zhang, W. Chen, W. He, Y. Ding, G. Li, Z. Chen, Z. Du, X. Yao, Y. Xu, J. Chen, T. Gui, Z. Wu, Q. Zhang, X. Huang, and Y. Jiang (2025) AgentGym-rl: training llm agents for long-horizon decision making through multi-turn reinforcement learning. External Links: 2509.08755, Link Cited by: §2. T. Xia, M. Xu, L. Hu, Y. Sun, W. Li, L. Shang, L. Liu, P. Shu, H. Yu, and J. Jiang (2026) Search-p1: path-centric reward shaping for stable and efficient agentic rag training. External Links: 2602.22576, Link Cited by: §1, §2. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. External Links: 1809.09600, Link Cited by: §4. Q. Zhao, R. Wang, D. Xu, D. Zha, and L. Liu (2025) R-search: empowering llm reasoning with search via multi-reward reinforcement learning. External Links: 2506.04185, Link Cited by: §1, §2. Y. Zheng, D. Fu, X. Hu, X. Cai, L. Ye, P. Lu, and P. Liu (2025) DeepResearcher: scaling deep research via reinforcement learning in real-world environments. External Links: 2504.03160, Link Cited by: §2. Appendix A Algorithm pseudocode (extended) We give the full HindSearch training step, with bookkeeping omitted from the main-paper sketch. Algorithm 1 HindSearch training step (full) 0: policy πθ _θ, frozen teacher πref _ref, judge πJ _J, batch ℬB, group size G, weight λOPD _OPD 1: For each prompt p∈ℬp , rollout G trajectories τg(p)g=1G\τ^(p)_g\_g=1^G with πθ _θ. 2: Compute EM rewards r(τg(p))r(τ^(p)_g) and group-normalized advantages A^g(p) A^(p)_g. 3: −←τ:r(τ)=0T_-←\τ:r(τ)=0\. 4: for τ∈−τ _- in parallel do 5: h(τ)←πJ(τ,aτ⋆)h(τ)← _J(τ,a _τ) single sample, temperature 0.7, max 96 tokens 6: end for 7: Build M←(τ,t):τ∈−,at∈⟨search⟩ spanM←\(τ,t):τ _-,\,a_t∈ search span\. 8: For each (τ,t)∈M(τ,t)∈ M, build ctTc^T_t via Eq. 2. 9: Forward πref _ref on ctT(τ,t)∈M\c^T_t\_(τ,t)∈ M; collect ℓtT ^T_t. 10: Forward πθ _θ on ℬB; collect ℓts ^s_t on all response tokens. 11: LPPO←L_PPO← standard GRPO clipped objective with retrieved-token mask. 12: LOPD←L_OPD← Eq. 3 on M. 13: Step optimizer on LPPO+λOPDLOPDL_PPO+ _OPDL_OPD. Appendix B Hyperparameters Table 3 gives the full hyperparameter set used in all main and ablation runs. Setting Value Backbone Qwen2.5-3B-Instruct Frozen teacher Qwen2.5-3B-Instruct (init) Judge model Qwen3.6-27B (Qwen Team, 2026) Training data NQ + HotpotQA (169k) Retriever E5-base-v2 (Wikipedia-18) Top-k retrieval 33 Max turns T 44 RL algorithm GRPO Group size G 88 PPO clip ϵε 0.20.2 KL coef (anchor) 0.0010.001 λOPD _OPD 0.010.01 OPD token mask search-span only, −T_- only OPD clamp range [0,1][0,1] Optimizer AdamW LR 1×10−61\!×\!10^-6 LR schedule constant Train steps 300300 Validation every 5050 steps Hardware 2×2\!×\!H200 Wall-clock overhead ∼15% \!15\% vs. unmodified GRPO Table 3: Full hyperparameters for the main HindSearch runs. Appendix C Negative results We collect here variants we tried that did not work or underperformed the default HindSearch configuration. Per-step earliest-critical critique. Instead of broadcasting one trajectory-level critique h(τ)h(τ) to every search step, we asked the judge to pick a single “earliest critical” search step and emitted a localized hint only there. This variant reached val@50=28.55 val@50=28.55, comparable to the trajectory-broadcast value at the same step (28.5628.56), but it never closed the gap and ended well below the trajectory-broadcast val@150=34.55 val@150=34.55. Looking at intermediate checkpoints, the localized hint seems too narrow: when the critique is genuinely trajectory-level (e.g., “the agent should have re-decomposed the question after the third turn”), pinning it to one token span fits poorly and the OPD gradient gets noisy. Same-family 7B teacher. The obvious way to widen the teacher–student capability gap is a strictly larger same-family teacher. We tried Qwen2.5-7B-Instruct as πref _ref with the 3B student. Training collapsed within 79 steps: validation EM dropped and KL to the anchor diverged. Our reading is that the 7B teacher disagrees with the 3B student on routine search tokens regardless of the hint, and at λOPD=0.01 _OPD=0.01 this produces a persistent capability-gap gradient that pulls the student off the GRPO trust region rather than nudging it toward repair behaviors. The frozen 3B copy avoids this because its disagreement with the student is mediated entirely by the hint context. Token-level judging. A variant differing from HindSearch only in the judge output format—per-token categorical labels over the emitted search query instead of a natural-language critique—collapsed within tens of steps. Logs showed the per-token labels were dominated by a single class, producing a degenerate OPD target. We kept natural-language critiques in the main method. Advantage-injection OPD on EM-rewarded search. We tried a direct port of OpenClaw-RL’s advantage-injection formulation (Wang et al., 2026a) to our search setup, sweeping the injection weight and using both rule-based and judge-based hints. All injection variants collapsed within 2525–8585 steps. Inspecting the runs, on EM-rewarded search the outcome advantage and the directive log-probability gap often disagree in sign on the same token, so a single combined scalar advantage produces updates that neither optimize EM nor follow the directive. The separate-loss form in Eq. 4 sidesteps this by keeping the two signals orthogonal in loss space. λ-decay schedules. We tested linearly decaying λOPD _OPD from 0.010.01 to 0 over [0.2T,0.4T][0.2T,0.4T] and over [0.0T,0.5T][0.0T,0.5T], on the intuition that OPD should matter most early in training. Both schedules underperformed the constant λOPD=0.01 _OPD=0.01 at val@150; we use the constant schedule in all main results. Appendix D Qualitative critique example Figure 3 shows a representative failed trajectory from training (HotpotQA) together with the trajectory-level critique returned by the judge. Snippets are abbreviated to fit the column. The hint is then prepended to the student’s context for every <search> step in this trajectory, and the OPD loss pushes the student toward the teacher’s hint-conditioned token distribution on those search tokens only. Question. Who was the director of the 1985 film Brazil? Gold answer. Terry Gilliam. Trajectory (r(τ)=0r(τ)=0): Step 1. <search>Who was the director of Brazil?</search> Retrieved: Doc 1 “Cinema of Brazil” (Glauber Rocha); Doc 2 “Brazil (country)” (Fernando Meirelles, City of God); Doc 3 “Brazil (1985 film)” (production background; no director name in snippet). Step 2. <search>director of Brazil</search> Retrieved: Doc 1–2 “Roberto Azevêdo” (WTO Director-General); Doc 3 “Secretary General of Foreign Affairs of Brazil”. Step 3. <answer>Glauber Rocha</answer> Judge critique. The agent confused the country “Brazil” with the 1985 film of the same name. The first query should have disambiguated to the film (e.g. “director of Brazil 1985 film” or “Terry Gilliam Brazil 1985”); the second query made the ambiguity worse by retrieving political-figure pages instead. Figure 3: Representative failed trajectory and trajectory-level critique. The same critique is broadcast as a hint to all three <search> steps and used to condition the frozen teacher in the OPD loss. Appendix E Judge prompt and model identifiers Judge model identifiers. For full reproducibility, the exact judge checkpoints used in this work are: • Qwen3.6-27B (main): Qwen/Qwen3.6-27B on Hugging Face, served via vLLM with enable_thinking=false. • Qwen2.5-7B-Instruct (sensitivity ablation): Qwen/Qwen2.5-7B-Instruct, served via vLLM. • DeepSeek-V4-Flash (sensitivity ablation): public DeepSeek API, model identifier deepseek-v4-flash, endpoint https://api.deepseek.com/chat/completions. All judge calls use temperature=0.2, max_tokens=500, and the prompt below. Trajectory-judge prompt. The frozen judge πJ _J is queried once per failed trajectory with the following template (the first and last two search steps are kept verbatim; intermediate steps are abbreviated to control prompt length): A search agent failed to answer the question correctly. Diagnose the failure. Question: question Correct answer: ground_truth Agent’s search trajectory: Step 1: query = "..." retrieved: ... ... omitted N middle steps ... Step k: query = "..." retrieved: ... In one or two sentences (max 50 words), describe what the agent should have done differently. Be specific to THIS trajectory --- name actual queries, retrieved evidence, or missing information. Output the critique only --- no preamble, no labels, no markdown. The judge’s response is parsed as plain text, capped at 500 characters by extracting up to the last sentence boundary, and used directly as the hint h(τ)h(τ) that conditions the frozen teacher (Eq. 2). Appendix F Training cost vs. canonical OPD HindSearch’s compute overhead is comparable to or lower than canonical on-policy distillation in two respects. First, the auxiliary teacher in canonical OPD is queried once per response token to obtain πref _ref log-probabilities; HindSearch keeps the same per-token teacher forward but restricts the loss to the search-action token mask M, which in our trajectories covers roughly 2020–30%30\% of response tokens, so the OPD gradient computation is correspondingly cheaper than a full-response OPD. Second, the judge call is paid only on failed trajectories (r(τ)=0r(τ)=0, typically 6060–80%80\% of rollouts early in training and declining as the policy improves); this adds about 15%15\% wall-clock per step in our setup—a one-time cost at training time that disappears at inference, since the judge is not in the rollout loop at evaluation. Hint-conditioned distillation methods of the same family pay a comparable overhead.

Instant research discovery