Paper deep dive

Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents

Jiangming Shu, Yuxiang Zhang, Ye Ma, Xueyuan Lin, Jitao Sang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 51

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 3/13/2026, 1:00:02 AM

Summary

EvalAct (Evaluate-as-Action) is a reinforcement learning framework for retrieval-augmented agents that enforces a coupled Search-to-Evaluate protocol. By transforming implicit retrieval quality assessment into an explicit action, it enables dense, trajectory-aligned process signals. These signals are leveraged via Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization method that rescales advantages at the segment level to improve credit assignment and multi-hop reasoning reliability.

Entities (5)

EvalAct · framework · 100%GRPO · reinforcement-learning-algorithm · 100%PCAR · optimization-method · 100%Multi-hop QA · task · 95%Qwen2.5 · llm-backbone · 95%

Relation Signals (3)

PCAR → extends → GRPO

confidence 100% · PCAR built upon Group Relative Policy Optimization (GRPO)

EvalAct → utilizes → PCAR

confidence 100% · To leverage these signals, we introduce Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization method

EvalAct → improvesperformanceon → Multi-hop QA

confidence 95% · EvalAct achieves the best average accuracy, with the largest gains on multi-hop tasks

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Retrieval-augmented agents can query external evidence, yet their reliability in multi-step reasoning remains limited: noisy retrieval may derail multi-hop question answering, while outcome-only reinforcement learning provides credit signals that are too coarse to optimize intermediate steps. We propose \textsc{EvalAct} (Evaluate-as-Action), which converts implicit retrieval quality assessment into an explicit action and enforces a coupled Search-to-Evaluate protocol so that each retrieval is immediately followed by a structured evaluation score, yielding process signals aligned with the interaction trajectory. To leverage these signals, we introduce Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization method that rescales advantages at the segment level according to evaluation scores, emphasizing reliable segments while updating uncertain ones conservatively. Experiments on seven open-domain QA benchmarks show that \textsc{EvalAct} achieves the best average accuracy, with the largest gains on multi-hop tasks, and ablations verify that the explicit evaluation loop drives the primary improvements while PCAR provides consistent additional benefits.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

51,155 characters extracted from source content.

Expand or collapse full text

Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents Jiangming Shu1, Yuxiang Zhang1, Ye Ma2, Xueyuan Lin2, Jitao Sang1 1School of Computer Science and Technology, Beijing Jiaotong University 2Hithink Research jiangmingshu, yuxiangzhang, jtsang@bjtu.edu.cn maye@myhexin.com linxy59@mail2.sysu.edu.cn Corresponding author. Abstract Retrieval-augmented agents can query external evidence, yet their reliability in multi-step reasoning remains limited: noisy retrieval may derail multi-hop question answering, while outcome-only reinforcement learning provides credit signals that are too coarse to optimize intermediate steps. We propose EvalAct (Evaluate-as-Action), which converts implicit retrieval quality assessment into an explicit action and enforces a coupled Search-to-Evaluate protocol so that each retrieval is immediately followed by a structured evaluation score, yielding process signals aligned with the interaction trajectory. To leverage these signals, we introduce Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization method that rescales advantages at the segment level according to evaluation scores, emphasizing reliable segments while updating uncertain ones conservatively. Experiments on seven open-domain QA benchmarks show that EvalAct achieves the best average accuracy, with the largest gains on multi-hop tasks, and ablations verify that the explicit evaluation loop drives the primary improvements while PCAR provides consistent additional benefits. Evaluate-as-Action: Self-Evaluated Process Rewards for Retrieval-Augmented Agents Jiangming Shu1, Yuxiang Zhang1, Ye Ma2, Xueyuan Lin2, Jitao Sang1†thanks: Corresponding author. 1School of Computer Science and Technology, Beijing Jiaotong University 2Hithink Research jiangmingshu, yuxiangzhang, jtsang@bjtu.edu.cn maye@myhexin.com linxy59@mail2.sysu.edu.cn 1 Introduction Large language model (LLM) agents have shifted automated reasoning from passive response generation to autonomous problem solving, where models plan, interact with external tools, and iteratively refine their beliefs across multi-step trajectories (Yao et al., 2022; Schick et al., 2023). Retrieval-augmented generation (RAG) further extends this capability by grounding decisions in external evidence, enabling open-domain question answering beyond the limits of parametric knowledge (Lewis et al., 2020; Guu et al., 2020). However, as queries shift from single-hop factoids to multi-hop narratives, the central bottleneck is no longer tool access itself, but the agent’s ability to navigate, verify, and synthesize evidence over long-horizon, noise-prone interaction sequences (Trivedi et al., 2023; Asai et al., 2024). Despite substantial progress, ensuring reliable intermediate reasoning remains a key challenge. Existing agentic baselines, from prompting methods that interleave retrieval and reasoning (Trivedi et al., 2023) to RL-based search agents such as Search-R1 (Jin et al., 2025) and refinement frameworks such as AutoRefine (Shi et al., 2025), still rely primarily on implicit internal reasoning for noise suppression and self-correction. This paradigm suffers from two fundamental limitations. First, error propagation: without an explicit, immediate mechanism for evidence verification, a single irrelevant document can derail downstream reasoning, causing irreversible trajectory drift in multi-hop settings. Second, coarse credit assignment: standard RL optimization, including PPO-based RLHF (Ouyang et al., 2022; Schulman et al., 2017) and outcome-reward post-training methods such as GRPO (Shao et al., 2024), typically relies on sparse signals tied to final-answer correctness. Such outcome-only supervision cannot distinguish informative retrieval steps from redundant or misleading actions within long trajectories; as a result, the optimizer often reinforces or penalizes an entire trajectory nearly uniformly, degrading sample efficiency and causing performance saturation as task complexity grows. To address these challenges, we introduce EvalAct, a reinforcement learning framework that transforms the agent’s implicit self-assessment of retrieval quality into an explicit, policy-selectable action. EvalAct enforces a strictly coupled search-then-evaluate protocol: each Search action must be immediately followed by an Evaluate action that produces a structured self-assessment score reflecting the utility of the retrieved evidence. This design directly addresses the two limitations identified above. At inference time, the evaluation output provides actionable control signals that facilitate early pruning of unproductive branches, reducing error propagation without external oracle supervision. During training, it produces dense, trajectory-aligned process signals that make intermediate reliability directly optimizable and enable finer-grained credit assignment. To leverage these process signals effectively, we further propose Process-Calibrated Advantage Rescaling (PCAR) built upon Group Relative Policy Optimization (GRPO) (Shao et al., 2024). Instead of broadcasting a single trajectory-level advantage to all tokens, PCAR uses step-wise self-evaluation scores to modulate updates at the segment level, amplifying gradients for reliable, progress-making steps while applying conservative updates to uncertain segments. Importantly, this provides process-level guidance without requiring expensive human-annotated process reward models (Lightman et al., 2023), while complementing prior verification-oriented supervision that does not explicitly target retrieval behavior (Ma et al., ). Together, EvalAct and PCAR convert introspection into an executable action space with trainable process signals, improving learning stability and multi-hop generalization. Our contributions are as follows: • We propose EvalAct, an RL framework that transforms implicit retrieval quality evaluation into an explicit Evaluate action and enforces a coupled Search→ protocol, producing dense, trajectory-aligned self-evaluation rewards for tool-using agents. • We introduce Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization strategy that leverages step-wise self-evaluation scores to refine credit assignment and stabilize learning in long-horizon retrieval trajectories. • We achieve the best average performance across seven open-domain QA benchmarks with two backbone scales, with particularly strong gains on multi-hop tasks; extensive ablations show that the explicit evaluation loop accounts for the dominant improvements, while PCAR provides consistent additional benefits. 2 Methodology We present our approach in three parts. First, we formulate retrieval-augmented multi-hop question answering as sequential decision-making under partial observability, providing a unified view for both inference-time interaction and RL training (§2.1). Second, we introduce EvalAct (Evaluate-as-Action), which transforms implicit retrieval quality evaluation into an explicit, policy-selectable action and enforces a coupled Search→ interaction protocol (§2.2). Third, we propose Process-Calibrated Advantage Rescaling (PCAR), a GRPO-based optimization method that rescales segment-wise policy gradients using self-evaluation scores, improving credit assignment and stabilizing learning (§2.3). Figure 1 illustrates the coupled Search→ loop and the PCAR-weighted GRPO update. 2.1 Problem Formulation Let ℳθM_θ be an LLM parameterized by θ, inducing a stochastic policy πθ _θ over textual tokens and tool-mediated actions. We model retrieval-augmented multi-hop question answering as a POMDP ⟨,,,⟩ ,A,O,P . At time t=0,…,Tt=0,…,T, the agent samples an action at∼πθ(⋅∣ht)a_t _θ(· h_t) conditioned on the observable history ht=[x,a0,o0,…,at−1,ot−1],h_t=[\,x,a_0,o_0,…,a_t-1,o_t-1\,], (1) where x is the input query and ot∈o_t is the observation returned by the environment after executing ata_t. A trajectory is τ=(at,ot)t=0Tτ=\(a_t,o_t)\_t=0^T, while the underlying state st∈s_t is unobserved. The transition function (st+1∣st,at)P(s_t+1 s_t,a_t) governs state evolution and is implicitly defined by the retrieval environment and the agent’s reasoning process. Action space. We partition A into (i) reasoning tokens thinkA_think, (i) tool actions toolA_tool, and (i) a terminal answer action answerA_answer. Tool actions include retrieval Search(q) Search(q) and self-evaluation Evaluate(c,z) Evaluate(c,z), where q is a query string, c is a textual assessment, and z∈[0,10]z∈[0,10] is a scalar confidence score reported by the policy. Observations. For tool actions, the environment returns ot∼ℛ(q),if at=Search(q),Φ(z),if at=Evaluate(c,z),∅,otherwise,o_t casesR(q),&if a_t= Search(q),\\[2.0pt] (z),&if a_t= Evaluate(c,z),\\[2.0pt] ,&otherwise, cases (2) where ℛ(q)R(q) denotes the top-k retrieved documents and Φ(⋅) (·) maps the reported score to a discrete feedback cue used for subsequent decision-making. Figure 1: Overview of EvalAct with PCAR. The agent follows a coupled Search→ protocol, producing segment-wise self-evaluation scores zi,k\z_i,k\ that PCAR uses to rescale GRPO advantages. 2.2 EvalAct: Evaluate-as-Action EvalAct transforms implicit retrieval quality evaluation into an executable action and couples each retrieval step with immediate self-assessment. Specifically, after any retrieval action at=Search(q)a_t= Search(q) with observation ot=ℛ(q)o_t=R(q), the agent must invoke exactly one evaluation action at+1=Evaluate(c,z)a_t+1= Evaluate(c,z). The assessment c is conditioned on the retrieved documents, and z∈[0,10]z∈[0,10] is a self-reported confidence score. This one-to-one coupling aligns each retrieval result with an explicit reliability assessment, enabling segment-wise training signals. Inference-time control without oracle signals. To avoid external supervision, the environment-side evaluator is deliberately non-interpretive: it neither parses c nor inspects retrieved documents. Instead, it deterministically maps z to a discrete control cue ℐ=Φ(z)I= (z): Φ(z)=ℐlow,z∈[0,3],ℐmid,z∈(3,7],ℐhigh,z∈(7,10]. (z)= casesI_low,&z∈[0,3],\\[2.0pt] I_mid,&z∈(3,7],\\[2.0pt] I_high,&z∈(7,10]. cases (3) The cue ℐI is appended to the context and modulates subsequent actions via instruction conditioning. For completeness, Appendix B specifies the Evaluate format and feedback templates, and Appendix C presents a complete multi-hop trajectory showing how the agent assigns calibrated self-evaluation scores to guide iterative evidence gathering. 2.3 Reinforcement Learning with PCAR We optimize πθ _θ to maximize expected reward, using GRPO as the backbone and PCAR to incorporate process signals from Evaluate. Gated outcome reward. To enforce protocol compliance while optimizing answer quality, we use a gated reward: (y,y∗)=F1(yans,y∗),if fmt(y)=1,0,otherwise,G(y,y^*)= casesF1(y_ans,y^*),&if I_fmt(y)=1,\\ 0,&otherwise, cases (4) where yansy_ans is the final answer extracted from <answer> tags and fmt(y)I_fmt(y) indicates whether (i) reasoning is enclosed by <think> tags and (i) every Search is immediately followed by Evaluate. GRPO. Given G rollouts y1,…,yG\y_1,…,y_G\ sampled from πθold _ _old for the same input x, we compute group-normalized advantages Ai=ri−μgroupσgroup+ε,A_i= r_i- _group _group+ , (5) where ri=(yi,y∗)r_i=G(y_i,y^*) and μgroup,σgroup _group, _group are the within-group mean and standard deviation. PCAR: segment-wise advantage rescaling. Standard GRPO applies the same AiA_i to all tokens, which can inadvertently reinforce unreliable intermediate steps. PCAR instead rescales advantages at the segment level using the self-evaluation scores. Let yiy_i contain KiK_i segments, each associated with a score zi,k∈[0,10]z_i,k∈[0,10]. We first compute an intra-trajectory standardized reliability signal z~i,k=zi,k−μiσi+ε, z_i,k= z_i,k- _i _i+ , (6) where μi _i and σi _i are the mean and standard deviation of zi,kk=1Ki\z_i,k\_k=1^K_i. This normalization makes z~i,k z_i,k reflect relative reliability within the trajectory and suppress trivial constant scoring. We then define a score-scaled gain λi,k=λbase+(λmax−λbase)⋅zi,k10, _i,k= _base+( _max- _base)· z_i,k10, (7) and compute the token-level calibrated advantage for any token t belonging to segment k: A^i,t=Ai⋅clamp(1+λi,kz~i,k,δ,∞), A_i,t=A_i·clamp\! (1+ _i,k z_i,k,\,δ,\,∞ ), (8) where δ>0δ>0 prevents gradient inversion; we set δ=10−6δ=10^-6 in all experiments unless otherwise specified. Finally, we maximize the GRPO-style clipped objective with the calibrated advantages: (θ)=x∼,yii=1G∼πθold[1G∑i=1G∑t=1Li(ℒi,tCLIP (θ)=E_x ,\y_i\_i=1^G _ _old [ 1G _i=1^G _t=1^L_i (L^CLIP_i,t −βKL(πθ(⋅∣ht)∥πθref(⋅∣ht)))], -β\,D_KL\! ( _θ(· h_t)\,\|\, _ _ref(· h_t) ) ) ], (9) where LiL_i is the length of yiy_i and ℒi,tCLIP=min⁡(ρi,tA^i,t,clip(ρi,t,1−ϵ,1+ϵ)A^i,t),ρi,t=πθ(yi,t∣ht)πθold(yi,t∣ht). aligned L^CLIP_i,t&= ( _i,t A_i,t,\,clip( _i,t,1-ε,1+ε) A_i,t )\\ & , _i,t= _θ(y_i,t h_t) _ _old(y_i,t h_t). aligned (10) By steering updates toward segments that are both outcome-aligned and process-reliable, PCAR improves credit assignment in long-horizon retrieval trajectories. For reproducibility, Appendix A provides pseudocode for the complete EvalAct training loop with PCAR, including protocol-compliant rollouts, GRPO advantage estimation, and segment-wise advantage rescaling. 3 Experiments 3.1 Experimental Setup Datasets. We evaluate open-domain question answering performance on seven widely-used benchmarks spanning both single-hop and multi-hop settings: Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023) as single-hop datasets, and HotpotQA (Yang et al., 2018), 2WikiMultihopQA (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023) as multi-hop datasets that typically require iterative evidence acquisition. For training, we use the publicly released ASearcherBase35K corpus (Gao et al., 2025) and remove invalid or non-actionable samples via lightweight filtering, resulting in 27k27k instances for RL. For supervised warm-up, we synthesize 2k2k protocol-compliant trajectories by prompting DeepSeek-V3.2 (Non-thinking Mode) (Liu et al., 2025) to follow the EvalAct interaction format. Baselines. We compare EvalAct against representative baselines spanning direct answering, single-pass retrieval augmentation, and multi-step retrieval–reasoning. (1) Direct Generation uses the instruction-tuned backbone model to answer using only parametric knowledge, without any retrieval. (2) Naïve RAG retrieves documents once and concatenates them with the query, then generates the answer in a single forward pass. (3) IRCoT (Trivedi et al., 2023) interleaves retrieval and chain-of-thought prompting for multi-hop reasoning. (4) Search-o1 (Li et al., 2025) and (5) Search-R1 (Jin et al., 2025) represent recent search-augmented agentic baselines with iterative retrieval and reasoning. (6) AutoRefine (Shi et al., 2025) is an iterative refinement baseline that alternates between evidence gathering and answer refinement. For all retrieval-enabled baselines, we use the same retrieval environment as EvalAct, including the corpus, retriever, returned top-k documents, and search budget, ensuring controlled comparison under matched external evidence access. Evaluation Metrics. We report Exact Match (EM) as the primary evaluation metric on all benchmarks, computed via exact string matching between the normalized prediction and the reference answer. During RL training, the outcome-level reward is defined as the token-level F1 score between the generated answer and the ground-truth reference (cf. Eq. 4). At test time, performance is evaluated using EM to align with standard open-domain QA evaluation protocols. 3.2 Implementation Details We conduct experiments with two backbones, Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct. Unless otherwise specified, training uses 88 NVIDIA A100 GPUs with full-parameter optimization and gradient checkpointing. We use a fixed open-domain retrieval environment built on the December 2018 Wikipedia dump with a standard BM25 retriever, without reranking or post-retrieval filtering. At each retrieval step, the top-k=3k=3 documents are returned and appended to the dialogue context. The tool budget is capped at 2020 Search calls per question. Single-Hop QA Multi-Hop QA Avg. Method NQ PopQA TriviaQA 2Wiki Bamboogle HotpotQA MuSiQue Backbone: Qwen2.5-3B-Instruct Direct Generation 10.6 10.8 28.8 24.4 2.4 14.9 2.0 13.4 IRCoT 11.1 20.0 31.2 17.1 24.0 16.4 6.7 18.1 Search-o1 23.8 26.2 47.2 21.8 32.0 22.1 5.4 25.5 Naïve RAG 34.8 38.7 54.4 22.6 8.0 25.5 4.7 27.0 Search-R1 34.1 37.8 54.5 31.9 26.4 32.4 10.3 32.5 AutoRefine 46.7 45.0 62.0 39.4 34.4 40.5 15.7 40.5 EvalAct-3B(ours) 38.5 43.2 62.1 50.0 48.0 44.3 21.6 44.0 Backbone: Qwen2.5-7B-Instruct Direct Generation 13.4 14.0 40.8 25.0 12.0 18.3 3.1 18.1 IRCoT 22.4 30.1 47.8 14.9 22.4 13.3 7.2 22.6 Search-o1 15.1 13.1 44.3 17.6 29.6 18.7 5.8 20.6 Naïve RAG 34.9 39.2 58.5 23.5 20.8 29.9 5.8 30.4 Search-R1 39.3 39.7 61.0 41.4 36.8 37.0 14.6 38.5 AutoRefine 48.4 48.7 65.9 40.5 51.2 45.1 18.7 45.5 EvalAct-7B(ours) 38.5 43.6 65.6 52.1 56.0 48.8 25.3 47.1 Table 1: Main results (EM, %) on seven open-domain QA benchmarks. Bold and underlined values indicate the best and second-best performance, respectively. For RL optimization, we implement the GRPO-based training described in §2.3 with the following default hyperparameters: learning rate 1×10−61× 10^-6, global batch size 256256, 22 epochs, 55 rollouts per prompt, rollout temperature 1.01.0, KL coefficient β=0.001β=0.001, and clip ratio ϵ=0.2ε=0.2. For PCAR, we set the score-based modulation parameters (λbase,λmax)=(0.1,0.5)( _base, _max)=(0.1,0.5) in Eq. 7, which determine the minimum and maximum strength of segment-wise rescaling. 3.3 Main Results Table 1 reports EM scores on seven open-domain QA benchmarks. Across both backbone scales, EvalAct achieves the highest average EM among all compared methods, reaching 44.0% with EvalAct-3B and 47.1% with EvalAct-7B. In both cases, it outperforms the second-best baseline, AutoRefine, by 3.5 and 1.6 points, respectively. Comparison with Baselines. Compared with Search-o1, Search-R1, IRCoT, and Naïve RAG, EvalAct consistently outperforms these baselines on the majority of benchmarks, with the largest gains appearing in multi-hop settings. This trend is consistent across both 3B and 7B backbones, indicating that the improvements are robust across model scales. Unlike prior approaches that rely on implicit self-correction within free-form reasoning, EvalAct explicitly models evaluation as a discrete action, enabling segment-level credit assignment during RL optimization. Multi-Hop Benchmarks. The strongest gains of EvalAct emerge on multi-hop datasets. Across both backbone scales, EvalAct achieves the best performance on all four multi-hop benchmarks: 2WikiMultihopQA, Bamboogle, HotpotQA, and MuSiQue. The gains are especially large on 2WikiMultihopQA and Bamboogle, where EvalAct-3B improves over AutoRefine by 10.6 and 13.6 points, respectively, and EvalAct-7B improves over the strongest baseline by 10.7 and 4.8 points. Consistent improvements are also observed on HotpotQA and MuSiQue. These results suggest that explicit intermediate evaluation is particularly beneficial for tasks requiring iterative evidence aggregation and long-horizon reasoning, where the coupled evaluation loop helps control error propagation across extended interaction sequences. Single-Hop Benchmarks. On these single-hop datasets, EvalAct remains competitive but does not consistently outperform AutoRefine, which performs better on NQ and PopQA. This is expected: AutoRefine is designed for iterative answer refinement, which is particularly effective when the main challenge is answer polishing rather than multi-step evidence accumulation in single-hop settings. Nevertheless, the substantial gains on multi-hop benchmarks outweigh these gaps, resulting in the best overall average performance. 4 Ablation Studies We conduct ablation studies to understand the contribution of each component in EvalAct. Figure 3 provides an overview: (a) training curves showing stable convergence for both 3B and 7B backbones, (b) model ablation comparing training variants, (c) method ablation isolating the evaluation loop and PCAR, and (d) sensitivity analysis of PCAR hyperparameters. The following subsections detail these analyses. 4.1 Model Ablation: Disentangling Format Alignment from Reasoning A prerequisite for EvalAct is protocol compliance: the agent must reliably produce well-formed tool calls and adhere to the strictly coupled Search→ loop. To disentangle the contributions of format acquisition (via SFT) from reasoning capability (via RL), we construct six training variants based on the Qwen2.5-3B-Instruct backbone and evaluate them on four multi-hop benchmarks. The variants are defined as follows: • Base (Instruct) / SFT-Only: the backbone model evaluated without and with supervised warm-up, respectively. • Base + RL / SFT + RL (Vanilla): standard GRPO optimizing for answer correctness without enforcing the explicit Evaluate loop; retrieval tools are invoked freely. • Base EvalAct: applied directly to the Base model without SFT warm-up. • EvalAct (Ours): the full pipeline, i.e., SFT warm-up followed by EvalAct RL training. Figure 2: Effect of SFT on Format Alignment. The Base model exhibits high tool parsing failure rates. Model Variant SFT RL Paradigm Avg. EM 1. Base (Instruct) ✗ - 14.2 2. SFT-Only ✓ - 24.8 3. Base + RL (Vanilla) ✗ Standard 33.1 4. SFT + RL (Vanilla) ✓ Standard 33.5 5. Base EvalAct ✗ EvalAct 17.1 6. EvalAct (Ours) ✓ EvalAct 41.0 Table 2: Ablation of training stages and paradigms on multi-hop benchmarks (Avg. EM, %). Vanilla denotes standard RL without the explicit evaluation loop. SFT for Format Alignment. We first examine the role of supervised warm-up in establishing protocol compliance. As shown in Figure 2, SFT substantially reduces tool-formatting and parsing failures, providing a stable initialization for structured tool use. Consistent with this observation, SFT alone improves the multi-hop average from 14.2%14.2\% to 24.8%24.8\% (Table 2). Under vanilla RL without the explicit evaluation loop, performance is only weakly affected by SFT initialization, reaching 33.1%33.1\% from the Base model and 33.5%33.5\% from the SFT-initialized model. This suggests that standard RL can eventually recover a functional tool-calling policy, whereas SFT primarily stabilizes early optimization by aligning the model with the required format. Effectiveness of the EvalAct Paradigm. We next isolate the effect of enforcing the coupled Search→ loop. With the same backbone and training budget, EvalAct (Ours) attains 41.0%41.0\% average EM, exceeding the strongest vanilla baseline (33.5%33.5\%) by +7.5+7.5 points (Table 2). This gain supports the hypothesis that converting intermediate evaluation into an explicit action yields more informative process signals than implicit verification under outcome-only optimization. By contrast, applying EvalAct directly without supervised warm-up yields only 17.1%17.1\%, highlighting the difficulty of learning a structured action protocol from scratch. Figure 3: Training curves and ablation overview. (a) Training curves of EvalAct with 3B/7B backbones. (b) Model ablation across training variants. (c) Method ablation on removing the evaluation loop or PCAR. (d) Sensitivity to PCAR rescaling intensity. 4.2 Method Ablation: Dissecting Structural and Optimization Components We further decompose the performance gains of EvalAct into two sources: the structural contribution of the explicit evaluation loop and the optimization contribution of Process-Calibrated Advantage Rescaling (PCAR). We compare the full model against two ablated variants: (1) w/o Eval Loop: removing Evaluate and reverting to a standard retrieval policy optimized via vanilla GRPO (equivalent to SFT+RL in §4.1); (2) w/o PCAR: retaining the Evaluate structure and confidence scores z, but optimizing with standard GRPO without confidence-based advantage rescaling. Method 2Wiki Bam. Hotpot Mus. Avg. EvalAct 50.0 48.0 44.3 21.6 41.0 w/o PCAR 48.2 46.2 44.1 20.8 39.8 w/o Eval Loop 41.4 37.2 38.9 16.5 33.5 Table 3: Method ablation on multi-hop benchmarks (EM, %). Structural Contribution of the Evaluation Loop. Removing the explicit evaluation mechanism causes the largest performance degradation. As shown in Table 3, eliminating Evaluate lowers average EM from 41.0%41.0\% to 33.5%33.5\%, a drop of 7.57.5 points. This degradation is consistent across all four benchmarks, with especially pronounced declines on 2WikiMultihopQA (−8.6-8.6) and Bamboogle (−10.8-10.8). These results suggest that the primary benefit of EvalAct lies in its explicit evaluation loop, which enforces intermediate verification and thereby reduces error propagation in multi-hop reasoning. Optimization Contribution of PCAR. Beyond the structural benefit of the explicit evaluation loop, PCAR provides additional optimization gains. Compared with standard GRPO applied to the same evaluation-augmented framework, PCAR raises average EM from 39.8%39.8\% to 41.0%41.0\%, a gain of 1.21.2 points. Improvements are observed on all four benchmarks, with gains of 1.81.8 points on both 2WikiMultihopQA and Bamboogle, and smaller but consistent improvements on HotpotQA and MuSiQue. These results indicate that confidence-aware advantage rescaling complements the explicit evaluation structure by providing more informative gradient signals for segments with varying reliability estimates. 4.3 Hyperparameter Sensitivity We analyze the sensitivity of PCAR to the rescaling intensity governed by λbase _base and λmax _max. To characterize the strength of reliability-aware modulation, we define the Relative Importance Ratio (RIR) as the ratio between the maximum and minimum attainable advantage multipliers under full-confidence conditions. When the standardized reliability score satisfies z~∈[−1,1] z∈[-1,1], the unclamped multiplier spans 1±λmax1± _max; after applying the lower-bound clamp with threshold δ, the effective ratio is approximated by RIR≈(1+λmax)/max⁡(δ, 1−λmax)RIR≈(1+ _max) / (δ,\,1- _max). Setting Params RIR Avg. Δ EvalAct (Low) 0.05/0.250.05/0.25 1.671.67 39.8 -0.3 EvalAct (Mid) 0.10/0.500.10/0.50 3.03.0 40.1 - EvalAct (High) 0.20/1.000.20/1.00 200200 39.4 -0.7 Table 4: Parameter ablation on PCAR intensity. RIR is the ratio between max/min multipliers. Table 4 reports three representative configurations corresponding to low, moderate, and high rescaling intensity. Performance remains relatively stable across settings, with average EM ranging from 39.4%39.4\% to 40.1%40.1\%. The moderate configuration (EvalAct (Mid), RIR=3.0RIR=3.0) achieves the best overall performance. Under conservative rescaling (Low, RIR=1.67RIR=1.67), the separation between high- and low-reliability segments is limited, potentially reducing the effectiveness of segment-level credit assignment. Conversely, aggressive rescaling (High, RIR=200RIR=200) slightly degrades performance. In this regime, the minimum multiplier approaches zero and is clipped by the lower-bound constraint, leading to highly imbalanced gradient magnitudes across segments. Such extreme modulation can restrict corrective updates on low-reliability steps and destabilize optimization. Overall, these results suggest that moderate reliability rescaling provides a balanced trade-off between emphasizing high-confidence segments and preserving sufficient gradient flow for error correction. 5 Related Work 5.1 Retrieval-Augmented Language Models Retrieval-augmented generation (RAG) enhances LLMs by grounding generation in externally retrieved knowledge (Lewis et al., 2020; Guu et al., 2020; Borgeaud et al., 2022). Early work focused on improving retrieval quality through dense encoders (Karpukhin et al., 2020; Izacard and Grave, 2021) or neural rerankers (Nogueira and Cho, 2019). More recent approaches integrate retrieval directly into the reasoning process, enabling models to iteratively query external sources (Yao et al., 2022; Trivedi et al., 2023; Jiang et al., 2023). Self-RAG (Asai et al., 2024) introduces special tokens for assessing retrieval utility, though these remain implicit signals rather than structured actions. Our work builds on this line by converting retrieval evaluation into an explicit, structured action with discrete scores that can serve as training signals. 5.2 Reinforcement Learning for Tool-Using Agents RL has emerged as a promising approach for training LLM agents to use tools effectively (Schick et al., 2023; Nakano et al., 2021; Qin et al., 2023). Search-R1 (Jin et al., 2025) demonstrates that pure RL with outcome rewards can train effective retrieval policies without supervised fine-tuning. However, outcome-only rewards suffer from the credit assignment problem in multi-step trajectories. Process reward models (PRMs) address this by providing step-level supervision (Lightman et al., 2023; Uesato et al., 2022; Wang et al., 2024), but typically require expensive human annotations or external verifiers that may not align with the target policy. LeTS (Zhang et al., 2025a) designs retrieval-specific process rewards based on knowledge redundancy and exact match, yet relies on heuristics that may not generalize. Recent work has begun to expand the action space by converting traditionally implicit behaviors into explicit, learnable decisions. MemAct (Zhang et al., 2025b) formulates working memory management as policy actions for context deletion and insertion, enabling end-to-end RL over long-horizon context curation. EvalAct shares this action-centric perspective but targets a different behavior: instead of treating retrieval quality assessment as an implicit part of free-form reasoning, we convert evaluation into an explicit action that produces process signals for fine-grained credit assignment. 5.3 Self-Evaluation and Calibration LLMs can evaluate their own outputs (Kadavath et al., 2022; Xie et al., 2023; Madaan et al., 2023), but such assessments vary in calibration—the alignment between expressed confidence and actual accuracy (Tian et al., 2023). Uncalibrated evaluation during RL training risks reward hacking, where models assign high scores regardless of output quality. Prior work addresses calibration through specialized training objectives (Lin et al., 2022), prompting strategies (Xiong et al., 2023), or post-hoc adjustments (Zhao et al., 2021). We take a different approach: Process-Calibrated Advantage Rescaling (PCAR) designs the RL objective such that miscalibrated confidence incurs penalties through the advantage signal, naturally incentivizing well-calibrated evaluations without explicit calibration training. 6 Conclusion We presented EvalAct, a framework that elevates retrieval evaluation from an implicit reasoning behavior to an explicit policy action. This design enables retrieval-augmented agents to generate structured process signals during interaction and to use them for more fine-grained reinforcement learning. Built on this framework, PCAR further improves optimization by aligning policy updates with segment-level reliability estimates. Across seven open-domain QA benchmarks, EvalAct delivers the best average results and shows its largest advantages on multi-hop reasoning tasks. These findings highlight the value of converting intermediate evaluation into a trainable action for multi-step retrieval-augmented reasoning. Limitations This design directly addresses the two limitations identified above. At inference time, the evaluation output provides actionable control signals that facilitate early pruning of unproductive branches, reducing error propagation without external oracle supervision. During training, it produces dense, trajectory-aligned process signals that make intermediate reliability directly optimizable and enable finer-grained credit assignment. References A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi (2024) Self-rag: learning to retrieve, generate, and critique through self-reflection. Cited by: §1, §5.1. S. Borgeaud, A. Mensch, J. Hoffmann, T. Cai, E. Rutherford, K. Millican, G. B. Van Den Driessche, J. Lespiau, B. Damoc, A. Clark, et al. (2022) Improving language models by retrieving from trillions of tokens. In International conference on machine learning, p. 2206–2240. Cited by: §5.1. J. Gao, W. Fu, M. Xie, S. Xu, C. He, Z. Mei, B. Zhu, and Y. Wu (2025) Beyond ten turns: unlocking long-horizon agentic search with large-scale asynchronous rl. arXiv preprint arXiv:2508.07976. Cited by: §3.1. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M. Chang (2020) Retrieval augmented language model pre-training. In International conference on machine learning, p. 3929–3938. Cited by: §1, §5.1. X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, p. 6609–6625. Cited by: §3.1. G. Izacard and E. Grave (2021) Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, p. 874–880. Cited by: §5.1. Z. Jiang, F. F. Xu, L. Gao, Z. Sun, Q. Liu, J. Dwivedi-Yu, Y. Yang, J. Callan, and G. Neubig (2023) Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 7969–7992. Cited by: §5.1. B. Jin, H. Zeng, Z. Yue, J. Yoon, S. Arik, D. Wang, H. Zamani, and J. Han (2025) Search-r1: training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516. Cited by: §1, §3.1, §5.2. M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer (2017) TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601–1611. Cited by: §3.1. S. Kadavath, T. Conerly, A. Askell, T. Henighan, D. Drain, E. Perez, N. Schiefer, Z. Hatfield-Dodds, N. DasSarma, E. Tran-Johnson, et al. (2022) Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. Cited by: §5.3. V. Karpukhin, B. Oguz, S. Min, P. S. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih (2020) Dense passage retrieval for open-domain question answering.. In EMNLP (1), p. 6769–6781. Cited by: §5.1. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. (2019) Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics 7, p. 453–466. Cited by: §3.1. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. (2020) Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems 33, p. 9459–9474. Cited by: §1, §5.1. X. Li, G. Dong, J. Jin, Y. Zhang, Y. Zhou, Y. Zhu, P. Zhang, and Z. Dou (2025) Search-o1: agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366. Cited by: §3.1. H. Lightman, V. Kosaraju, Y. Burda, H. Edwards, B. Baker, T. Lee, J. Leike, J. Schulman, I. Sutskever, and K. Cobbe (2023) Let’s verify step by step. In The Twelfth International Conference on Learning Representations, Cited by: §1, §5.2. S. Lin, J. Hilton, and O. Evans (2022) Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334. Cited by: §5.3. A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, et al. (2025) Deepseek-v3. 2: pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556. Cited by: §3.1. [18] R. Ma, P. Wang, C. Liu, X. Liu, J. Chen, B. Zhang, X. Zhou, N. Du, and J. Li S2r: teaching llms to self-verify and selfcorrect via reinforcement learning, 2025. URL https://arxiv. org/abs/2502.12853 4, p. 17. Cited by: §1. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, et al. (2023) Self-refine: iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36, p. 46534–46594. Cited by: §5.3. A. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi (2023) When not to trust language models: investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9802–9822. Cited by: §3.1. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. (2021) Webgpt: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. Cited by: §5.2. R. Nogueira and K. Cho (2019) Passage re-ranking with bert. arXiv preprint arXiv:1901.04085. Cited by: §5.1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1. O. Press, M. Zhang, S. Min, L. Schmidt, N. A. Smith, and M. Lewis (2023) Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 5687–5711. Cited by: §3.1. Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, et al. (2023) Toolllm: facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. Cited by: §5.2. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36, p. 68539–68551. Cited by: §1, §5.2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §1, §1. Y. Shi, S. Li, C. Wu, Z. Liu, J. Fang, H. Cai, A. Zhang, and X. Wang (2025) Search and refine during think: facilitating knowledge refinement for improved retrieval-augmented reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §1, §3.1. K. Tian, E. Mitchell, H. Yao, C. Manning, and C. Finn (2023) Fine-tuning language models for factuality. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following, Cited by: §5.3. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2022) ♫ MuSiQue: multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics 10, p. 539–554. Cited by: §3.1. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2023) Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 10014–10037. Cited by: §1, §1, §3.1, §5.1. J. Uesato, N. Kushman, R. Kumar, F. Song, N. Siegel, L. Wang, A. Creswell, G. Irving, and I. Higgins (2022) Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275. Cited by: §5.2. P. Wang, L. Li, Z. Shao, R. Xu, D. Dai, Y. Li, D. Chen, Y. Wu, and Z. Sui (2024) Math-shepherd: verify and reinforce llms step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9426–9439. Cited by: §5.2. Y. Xie, K. Kawaguchi, Y. Zhao, J. X. Zhao, M. Kan, J. He, and M. Xie (2023) Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems 36, p. 41618–41650. Cited by: §5.3. M. Xiong, Z. Hu, X. Lu, Y. Li, J. Fu, J. He, and B. Hooi (2023) Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306.13063. Cited by: §5.3. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369–2380. Cited by: §3.1. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao (2022) React: synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, Cited by: §1, §5.1. Q. Zhang, S. Yang, L. Gao, H. Chen, X. Hu, J. Chen, J. Wang, S. Guo, B. Zheng, H. Wang, et al. (2025a) LeTS: learning to think-and-search via process-and-outcome reward hybridization. arXiv preprint arXiv:2505.17447. Cited by: §5.2. Y. Zhang, J. Shu, Y. Ma, X. Lin, S. Wu, and J. Sang (2025b) Memory as action: autonomous context curation for long-horizon agentic tasks. arXiv preprint arXiv:2510.12635. Cited by: §5.2. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh (2021) Calibrate before use: improving few-shot performance of language models. In International conference on machine learning, p. 12697–12706. Cited by: §5.3. Appendix A Pseudocode for EvalAct with PCAR Algorithm 1 EvalAct Training with PCAR (GRPO Backbone) 1: Input: policy πθ _θ, reference πθref _ _ref, dataset D, environment ℰE, rollouts per input G, clip ϵε, KL weight β, constant ϵ0 _0, PCAR params (λbase,λmax,δ)( _base, _max,δ) 2: Output: optimized policy πθ _θ 3: while not converged do 4: Sample a batch of queries ∼X 5: Initialize training buffer ℬ←∅B← 6: for all x∈x do 7: // Protocol-compliant rollouts 8: for i=1i=1 to G do 9: Sample trajectory yi∼πθold(⋅∣x)y_i _ _old(· x) under coupled Search→Evaluate Search\!→\! Evaluate 10: Compute gated reward ri←(yi,y∗)r_i (y_i,y^*) (Eq. (4)) 11: Segment yiy_i into σi,kk=1Ki\ _i,k\_k=1^K_i aligned with Search→Evaluate Search\!→\! Evaluate, record zi,kk=1Ki\z_i,k\_k=1^K_i 12: end for 13: // GRPO advantage 14: μgroup←mean(rii=1G) _group (\r_i\_i=1^G), σgroup←std(rii=1G)+ϵ0 _group (\r_i\_i=1^G)+ _0 15: for i=1i=1 to G do 16: Ai←(ri−μgroup)/σgroupA_i←(r_i- _group)/ _group 17: Compute calibrated advantages A^i,t\ A_i,t\ for tokens in yiy_i via PCAR: 18: z~i,k←(zi,k−μi)/(σi+ϵ0) z_i,k←(z_i,k- _i)/( _i+ _0), λi,k←λbase+(λmax−λbase)⋅zi,k/10 _i,k← _base+( _max- _base)· z_i,k/10 19: A^i,t←Ai⋅clamp(1+λi,kz~i,k,δ,∞) A_i,t← A_i·clamp(1+ _i,k z_i,k,δ,∞) for t∈σi,kt∈ _i,k 20: Add token-level instances from (x,yi)(x,y_i) with advantages A^i,t\ A_i,t\ to ℬB 21: end for 22: end for 23: Update πθ _θ by maximizing the clipped objective with KL regularization (Eq. (9)) on ℬB 24: end while 25: return πθ _θ Appendix B Evaluate Specification and Feedback Templates Evaluate Action Format. After each retrieval step, the agent invokes Evaluate(c,z) Evaluate(c,z), where c is a free-form textual assessment of the immediately preceding Search output, and z∈[0,10]z∈[0,10] denotes a scalar self-reported confidence score. The environment intercepts this action and returns a discrete control cue ℐ=Φ(z)I= (z), which is appended to the context to modulate subsequent decision-making. Discrete Feedback Mapping. We instantiate Φ(⋅) (·) as a deterministic, three-tier binning strategy: Φ(z)=ℐlow,z∈[0,3],ℐmid,z∈(3,7],ℐhigh,z∈(7,10]. (z)= casesI_low,&z∈[0,3],\\[2.0pt] I_mid,&z∈(3,7],\\[2.0pt] I_high,&z∈(7,10]. cases (11) Feedback Templates. The control cue ℐI is materialized as an instruction-style message that conditions the agent’s next action. The exact textual templates returned by the environment are detailed below. Environment Feedback Templates ℐlowI_low (Low Quality ∣ z∈[0,3]z∈[0,3]) “The previous Search results are largely irrelevant or unhelpful for answering the question. Do not rely on them. Reformulate the query (e.g., alternative keywords or a different angle) and issue a new Search.” ℐmidI_mid (Medium Quality ∣ z∈(3,7]z∈(3,7]) “The previous Search results contain partially useful evidence but may be incomplete or noisy. Use only clearly relevant excerpts. Consider an additional, more targeted Search to fill missing details, resolve remaining subproblems, or verify uncertain information.” ℐhighI_high (High Quality ∣ z∈(7,10]z∈(7,10]) “The previous Search results are highly relevant and constitute substantive progress toward answering the question (e.g., providing key facts or resolving an important subtask). Use them as primary evidence to construct the answer. Perform another Search only if a specific critical detail is still missing.” Appendix C Case Study: Multi-Hop Reasoning Trajectory We present a full multi-hop interaction trajectory to illustrate the EvalAct framework in practice. The example highlights calibrated self-evaluation: a partial retrieval receives 5/10, while a conclusive retrieval receives 10/10. These step-wise confidence signals are the same signals used by PCAR for segment-level advantage rescaling during RL. EvalAct Trajectory Example User: In between Remember the Titans and My Favorite Martian which film grossed $36.8 million domestically? Agent: <think>I need to find out which film grossed $36.8 million domestically between "Remember the Titans" and "My Favorite Martian". I should search for box office information for both films.</think> Tool Call [Search]: "query": "Remember the Titans domestic box office $36.8 million" Observation [Search]: [Doc 1 & 3 omitted for brevity] Doc 2 (Title: "Remember the Titans"): "…Remember the Titans opened strongly at the U.S. box office… It eventually went on to gross an estimated $115,654,751 in the U.S., and…" Tool Call [Evaluate]: "evaluation": "The search provided information about Remember the Titans’ box office performance, but didn’t specifically mention the $36.8 million figure. The results show it grossed $115,654,751 in the U.S., which is much higher than $36.8 million. This helps narrow down that Remember the Titans didn’t gross $36.8 million, but I need more specific information to compare with My Favorite Martian.", "score": 5 Observation [Evaluate]: Score 5/10 (Medium Quality): The previous Search results are partially helpful but may be incomplete or contain noise. Extract and use only the clearly relevant parts. Consider running another more focused Search to fill in missing details, resolve remaining subproblems, or verify uncertain information. Agent: <think>Now I need to search for My Favorite Martian’s box office information.</think> Tool Call [Search]: "query": "\"My Favorite Martian\" domestic box office $36.8 million" Observation [Search]: [Doc 1 & 2 omitted for brevity] Doc 3 (Title: "My Favorite Martian (film)"): "…The film grossed $36.8 million domestically against a budget of $65 million…" Tool Call [Evaluate]: "evaluation": "This search was highly successful. Document 3 directly mentions \"My Favorite Martian\" and states it grossed $36.8 million domestically against a budget of $65 million. This matches exactly the figure mentioned in the question and confirms this film’s domestic gross. Combined with the previous search showing Remember the Titans grossed much more than $36.8 million ($115+ million), this gives me clear information to answer which film grossed the specified amount.", "score": 10 Observation [Evaluate]: Score 10/10 (High Quality): The previous Search results are highly relevant and make strong progress toward answering the question… Use them as primary evidence to construct your answer. Only conduct another Search if you still need a very specific missing detail. Agent: <think>From the information gathered: 1. Remember the Titans grossed $115,654,751 in the U.S. (much higher than $36.8 million) 2. My Favorite Martian grossed $36.8 million domestically Therefore, My Favorite Martian is the film that grossed $36.8 million domestically.</think> <answer>My Favorite Martian</answer>

Instant research discovery