Paper deep dive
Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results
Yanyu Chen, Yue Li, Yongyi Cui, Dongsheng Shi, Lichang Dai
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/23/2026, 3:08:33 AM
Summary
This paper addresses the challenge of selective evidence adoption in Retrieval-Augmented Generation (RAG) by post-training the Qwen3.5-4B model using the DAPO framework. The authors introduce SelectBench, a benchmark for evaluating the model's ability to filter contaminated retrieval results. Experiments with DAPO-Rule and DAPO-DeepSeek show modest improvements in strict success rates and reduced adoption of forbidden content, though statistical significance is limited after correction. The approach preserves general capabilities on MMLU and HotpotQA but does not improve resistance to prompt injections.
Entities (10)
Relation Signals (9)
Qwen3.5-4b → trainedwith → DAPO
confidence 98% · post-train Qwen3.5-4B directly with DAPO
DAPO-Rule → derivedfrom → Qwen3.5-4b
confidence 95% · strict success rises from ... for the original checkpoint to 25.54% with DAPO-Rule
DAPO-DeepSeek → derivedfrom → Qwen3.5-4b
confidence 95% · strict success rises from ... to 26.46% with DAPO-DeepSeek
SelectBench → usedfortraining → DAPO-DeepSeek
confidence 93% · SelectBench contains 1,300 training instances ... DAPO-DeepSeek employs a frozen deepseek-v4-flash judge
SelectBench → usedfortraining → DAPO-Rule
confidence 93% · SelectBench contains 1,300 training instances ... DAPO-Rule retains ... matching
DAPO-Rule → usesrewardtype → deterministic rule rewards
confidence 92% · DAPO-Rule retains lowercase ASCII letters ... and performs whole-phrase matching
DAPO-DeepSeek → usesrewardtype → frozen semantic judge
confidence 92% · DAPO-DeepSeek employs a frozen deepseek-v4-flash judge
DAPO-DeepSeek → evaluatedon → MMLU
confidence 90% · DAPO-DeepSeek exhibits no material degradation on MMLU
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content. Blanket refusal discards valid evidence, whereas uncritical adoption yields incorrect or unsafe answers. The ability to selectively adopt relevant information while rejecting deceptive or harmful content is therefore critical for reliable deployment in real-world retrieval settings. We introduce SelectBench, a controlled benchmark and training set for selective evidence adoption, and post-train Qwen3.5-4B directly with DAPO using either deterministic rule rewards or a frozen semantic judge. On the corrected 325-example SelectBench-v2 test set, strict success rises from 22.46% for the original checkpoint to 25.54% with DAPO-Rule and 26.46% with DAPO-DeepSeek. Both trained policies reduce forbidden-content adoption and produce shorter, more focused responses, yet prompt-injection following does not improve. The paired gains are modest and fail to survive Holm correction, suggesting that stronger reward shaping or additional training iterations may be needed for more robust gains. DAPO-DeepSeek exhibits no material degradation on MMLU or clean HotpotQA, indicating that the post-training procedure preserves general capabilities. These results demonstrate a directional improvement in selective evidence use, while identifying injection resistance and statistical robustness as important remaining challenges for future work.
Tags
Links
- Source: https://arxiv.org/abs/2607.20090v1
- Canonical: https://arxiv.org/abs/2607.20090v1
Trouble viewing inline? Open PDF directly →
Full Text
34,399 characters extracted from source content.
Expand or collapse full text
Reinforcement Learning for Large Language Model Selective Evidence Adoption from Contaminated Retrieval Results Yanyu Chen1 , Yue Li111footnotemark: 1 , Yongyi Cui1 , Dongsheng Shi1 , Lichang Dai2 1East China Normal University 2Shandong University cyyecao@stu.ecnu.edu.cn, 202435387@mail.sdu.edu.cn Equal Contribution.Corresponding author. Abstract Retrieval-augmented large language models frequently face contexts that interleave useful evidence with misleading statements or instruction-like content. Blanket refusal discards valid evidence, whereas uncritical adoption yields incorrect or unsafe answers. The ability to selectively adopt relevant information while rejecting deceptive or harmful content is therefore critical for reliable deployment in real-world retrieval settings. We introduce SelectBench, a controlled benchmark and training set for selective evidence adoption, and post-train Qwen3.5-4B directly with DAPO using either deterministic rule rewards or a frozen semantic judge. On the corrected 325-example SelectBench-v2 test set, strict success rises from 22.46% for the original checkpoint to 25.54% with DAPO-Rule and 26.46% with DAPO-DeepSeek. Both trained policies reduce forbidden-content adoption and produce shorter, more focused responses, yet prompt-injection following does not improve. The paired gains are modest and fail to survive Holm correction, suggesting that stronger reward shaping or additional training iterations may be needed for more robust gains. DAPO-DeepSeek exhibits no material degradation on MMLU or clean HotpotQA, indicating that the post-training procedure preserves general capabilities. These results demonstrate a directional improvement in selective evidence use, while identifying injection resistance and statistical robustness as important remaining challenges for future work. 1 Introduction Large Language Models (LLMs) have demonstrated remarkable proficiency across a broad spectrum of downstream tasks Yi et al. (2025c); Li et al. (2025a; 2026). Their ability to generalize from limited supervision and perform complex reasoning has positioned them as versatile foundation models for diverse real-world applications Yi et al. (2025b); Shi et al. (2026b). Additionally, by leveraging external tool-use capabilities, they can tackle tasks that go beyond the scope of inherent language reasoning Zhang et al. (2026a); Shi et al. (2026a). By explicitly instructing the model through prompts on when and how to invoke external tools, it becomes capable of performing tasks beyond the limits of pure linguistic reasoning Sha et al. (2025); Zhang et al. (2026b). Safety alignment in LLMs is a research direction dedicated to equipping models with the ability to refuse harmful instructions Li et al. (2025b); Yi et al. (2025a). Nevertheless, excessive alignment may inadvertently lead to the unwarranted refusal of benign ones Zhang et al. (2025). Inspired by this, we examine the scenario in which LLMs call upon search tools to aid generation. A blanket refusal of retrieved information due to safety concerns, rather than extracting useful portions, constitutes an unnecessary waste of resources. On the other hand, uncritical acceptance of such information introduces safety risks and yields low-quality outputs. Consequently, equipping models with the ability to discern and adopt valuable information while filtering out harmful content is of paramount importance. Prior work has demonstrated the potential of using reinforcement learning to enhance models’ understanding and utilization of external tools, including retrieval tools Sha et al. (2025); Chen et al. (2026). Building on this, we construct SelectBench to study this problem. Based on the DAPO framework Yu et al. (2026), we customize the reward signal to encourage the model to extract supported information from retrieved content interspersed with misleading or instruction-like material. We evaluate whether this policy shift improves selective evidence adoption and whether it changes performance on measured general-capability tasks. In summary, our contributions are as follows: • We formulate selective evidence adoption from contaminated retrieval-tool outputs as a distinct post-training objective for LLM agents. • We build a controlled dataset for selective evidence adoption and design a tailored RL pipeline that assigns positive reward only to correct, complete, and safe answers while treating forbidden adoption as a hard failure. • On a corrected held-out test set, both trained policies show modest directional improvements in strict selective adoption and lower forbidden-content adoption. Paired statistical and failure-mode analyses expose remaining uncertainty and a lack of improvement in prompt-injection resistance. 2 Preliminary We conduct a pilot experiment using Qwen3.5 Team (2026) at two model scales (4B and 9B parameters) to examine the inherent ability of LLMs to extract useful information from low-quality retrieved content. Specifically, we consider three dimensions of low quality: safety corruption, temporality corruption, and format corruption, with 100 instances constructed for each dimension. This design allows us to systematically assess how model capacity influences discernment across diverse corruption types. Table 1: Comparison of Multi-dimensional Settings. Strategy Safety Temporality Format Qwen3.5-4B Full Accept 15.00 0.00 0.00 Full Reject 55.00 0.00 0.00 Selective Accept 30.00 100.00 100.00 Qwen3.5-9B Full Accept 10.00 0.00 0.00 Full Reject 70.00 0.00 0.00 Selective Accept 20.00 100.00 100.00 From Table 1, we observe that the model can largely overcome interference from low-quality noisy content in the temporality and format corruption dimensions, thereby achieving selective acceptance. In contrast, in the safety corruption dimension, the model exhibits a strong tendency toward either indiscriminate acceptance or wholesale rejection: the former compromises safety alignment, whereas the latter reflects overly conservative safety behavior. Based on these descriptive pilot results, we identify safety-corrupted retrieval as the most challenging of the three tested dimensions. This preliminary experiment prompted us to conduct a controlled study on ”selective retrieval result adoption,” and we hypothesize that the model’s safety alignment is the cause of the observed behavior. 3 Methodology We study an agent that must answer a factual question after calling a search tool whose output mixes useful evidence with misleading or instruction-like text. Let q denote the question, c the fixed mixed retrieval context, and τ=(q,a1,c,a2)τ=(q,a_1,c,a_2) the resulting two-turn assistant trajectory: a1a_1 invokes the tool and a2a_2 contains the final reasoning and answer. The desired policy uses the evidence in c that supports the answer while neither adopting unsupported content nor obeying instructions embedded in c. We post-train the policy directly with DAPO Yu et al. (2026) using a ternary task score followed by continuous length shaping. 3.1 Dataset Preparation Training data. SelectBench contains 1,300 training instances derived from multi-hop questions in HotpotQA Yang et al. (2018), 2WikiMultiHopQA Ho et al. (2020), and MuSiQue Trivedi et al. (2022). The source mixture is 650/325/325 instances, respectively. For each question, we retain answer-supporting evidence and combine it with controlled low-quality spans. These spans cover an answer-like wrong summary, a surface-matched wrong entity, stale or misleading information, unsafe evidence-selection advice, over-refusal bait, and a retrieved prompt injection. Each instance records accepted answer aliases, required answer terms, known-wrong or forbidden phrases, and (where applicable) a unique injection-success marker. We construct a dataset of 1,300 instances, comprising 650 sft_correction, 455 preference_pair, and 195 injection_resistance cases. These labels describe construction provenance only. Training conversion actively removes chosen and rejected assistant trajectories, and RL receives only the question, fixed tool context, and reward labels. We split the data into 1,170 training and 130 validation examples with seed 42, stratifying jointly by source dataset, task type, and construction bucket. The frozen construction manifest records 646 skipped candidates: 377 with test-evidence hash overlap, 156 rejected by semantic evidence or label review, 58 duplicate training questions, and 55 with other construction or review failures. Every retained instance received a first review, and 1,105 received a second review. A final leakage audit reported no surviving leakage and no near-duplicate flags under normalized-token trigram Jaccard similarity at a 0.9 threshold. Example-disjoint test set. Our sole formal test set is SelectBench-v2, a corrected 325-example example-disjoint split derived from the original SelectBench challenge set. A post-training audit found that the original generator could substitute a non-supporting sentence when materializing trusted evidence. Version 2 strictly resolves all supporting facts from the pinned official HotpotQA distractor validation snapshot and changes the trusted evidence bundle in 233 examples. It preserves all questions, answers, synthetic contamination spans, forbidden labels, and injection markers. Consequently, the training data and both trained checkpoints remain unchanged. The split contains 275 injection-bearing examples and 50 examples with low-quality speculation instead of an injection. No test question, context, answer label, identifier, marker, or row-level taxonomy enters the 1,300-example construction pool. Aggregate failure-type counts from the original diagnostic set informed the training taxonomy and leakage audit. Thus, method development was benchmark-informed at the aggregate category level, but no row-level test content or label was used for training. All training questions and evidence originate from disjoint training sources. 3.2 Tool-Use Rollout The policy initially observes only a system instruction and the question q. It then invokes selectbench_search with a non-empty query. The offline tool returns the example-specific context c verbatim, regardless of query wording. This construction holds retrieval fixed across policies and isolates evidence selection from search-engine variance. A successful trajectory contains at least one successful tool call followed by a visible final answer. Correctness is evaluated on the text appearing after the final </think> delimiter, while forbidden-adoption checks span the entire final assistant turn so that copying a forbidden span into reasoning remains visible to the reward. 3.3 Reward Policy Let T(τ)T(τ) indicate a successful tool call, C(τ)C(τ) a correct and complete answer, and H(τ)H(τ) the adoption of a forbidden span or a retrieved instruction. The deterministic task score is rtask(τ)=−1,¬T(τ)orH(τ),+1,T(τ)∧C(τ)∧¬H(τ),0,otherwise.r_task(τ)= cases-1,& T(τ)\ or\ H(τ),\\ +1,&T(τ) C(τ) H(τ),\\ 0,&otherwise. cases (1) Thus, forbidden or injection adoption and failure to use the required tool dominate factual correctness. An answer with no detected adoption that is wrong, incomplete, missing, or over-refusing is classified as a neutral failure with a raw score of zero rather than a hard negative. We additionally discourage unnecessarily long model output. For n model-generated tokens (tool text excluded), plen(n)=0,n≤768,−(n−768)/256,768<n<1024,−1,n≥1024,r(τ)=clip(rtask+plen,−1,1).p_len(n)= cases0,&n≤ 768,\\ -(n-768)/256,&768<n<1024,\\ -1,&n≥ 1024, cases r(τ)=clip(r_task+p_len,-1,1). (2) We instantiate two variants of the ternary raw task score for policy training. DAPO-Rule retains lowercase ASCII letters and digits, collapses all other character spans into spaces, and performs whole-phrase matching against accepted aliases, required terms, forbidden or wrong phrases, and injection markers. Accepted aliases and all required terms must appear unnegated in the visible answer. A forbidden or injection phrase is treated as negated—rather than adopted—when a negation marker occurs within its four-token left context, or when it is locally followed by wrong, false, unsafe, or unsupported. For examples labeled full_reject, an explicit retrieval rejection or refusal also satisfies correctness. DAPO-DeepSeek employs a frozen deepseek-v4-flash judge Xu et al. (2026) with the selectbench-deepseek-rm-v2 rubric, yielding raw scores on the same −1,0,+1\-1,0,+1\ scale. Local hard gates assign −1-1 to tool omission and 0 to a missing final answer prior to any judge invocation. The judge runs at temperature 0 with thinking disabled and a 4,0964,096-token output limit; requests enforce a 180180-second timeout, at most eight retries, concurrency of 1616, and a rate limit of eight requests per second. Judge outputs must conform to a strict JSON schema and are content-addressed and cached. After incorporating plenp_len, the final policy reward is continuous on [−1,1][-1,1] for outputs between 768768 and 1,0241,024 generated tokens. Dynamic group filtering operates on the unshaped ternary score, whereas the policy update uses the shaped reward r(τ)r(τ). 3.4 Policy Optimization For each prompt, we sample a group of eight trajectories and normalize their rewards within the group to obtain GRPO advantages. DAPO then applies the token-level clipped surrogate objective with asymmetric clipping bounds of 0.200.20 and 0.280.28, along with dynamic sampling that excludes groups with all-equal rewards. We use one policy epoch per batch and omit both the KL reward and the KL loss. The reward is placed on the last model-generated token rather than on tool-return tokens, ensuring that the policy gradient is attached to the agent’s action sequence. 4 Experimental Setup 4.1 Training details All post-training and formal evaluation experiments use the original post-trained Qwen3.5-4B checkpoint Team (2026), pinned to revision 851bf6e8. We freeze the vision tower and update the language model in bfloat16 with FSDP2, using AdamW with a learning rate of 5×10−75× 10^-7, weight decay 0.10.1, one policy epoch per batch, gradient clipping at 1.01.0, and ten warm-up steps. Rollouts use temperature 11, top-p 11, no top-k truncation, a maximum prompt length of 768768, and a maximum response length of 3,0723,072. Both exported checkpoints correspond to optimizer step 130130 on NVIDIA RTX 3090 GPUs. Here, one policy epoch denotes one inner optimization pass over each sampled rollout batch, rather than one pass over the full training set. Both runs completed 130130 optimizer updates, with validation performed before the first update and every ten updates thereafter, and exported the predeclared terminal step-130130 checkpoint; no checkpoint was selected using SelectBench-v2. The rule-reward run used three GPUs with a prompt batch size of 99, whereas the DeepSeek-reward run used four GPUs with a prompt batch size of 88; both sampled eight rollouts per prompt. Consequently, their comparison is informative but does not constitute a strict single-factor reward-model ablation. Neither model was retrained nor were hyperparameters tuned after inspecting the formal test set. 4.2 Datasets The primary evaluation is the corrected 325-example SelectBench-v2 test set described above. To measure capability retention, we additionally evaluate both the base and DAPO-DeepSeek checkpoints on all 14,042 MMLU test questions Hendrycks et al. (2020), using both the standard subject-wise 5-shot protocol and zero-shot constrained next-token scoring. We also evaluate on 7,080 clean HotpotQA validation questions after excluding the 325 SelectBench questions, in both fixed-context and closed-book modes. All dataset files, model exports, prompts, and trajectories are SHA-256 inventoried. 4.3 Baselines We compare the original checkpoint, DAPO-Rule, and DAPO-DeepSeek using identical test prompts and a deterministic evaluator. Formal generation uses one trajectory per example with do_sample=false, temperature 0, top-p 11, no top-k truncation, a 3,0723,072-token output limit, and seed 4242. The evaluator applies the Rule matching procedure described above to the visible answer for correctness, and to the complete final assistant turn for forbidden adoption or injection detection. This shared measurement regime enables paired comparisons across all three checkpoints without evaluating DAPO-DeepSeek using its training-time judge. Nevertheless, because the two trained checkpoints differ in prompt-batch size, GPU count, and execution history, their comparison is descriptive rather than a controlled reward-model ablation. 4.4 Metrics Strict success requires a successful tool call, a correct and complete visible answer, and no detected forbidden adoption or injection following. We additionally report correct-and-complete answers, forbidden adoption, injection following, missing final answers, over-refusal, overlong output, tool success, and mean generated tokens. Injection following is conditioned on the 275 examples that contain an injection; all other metrics are computed over all 325 examples. From the paired per-example records, we compute percentile-bootstrap confidence intervals with 10,000 resamples and seed 42, and apply two-sided exact McNemar tests. We jointly apply Holm correction across nine prespecified comparisons: strict success, forbidden adoption, and injection following for each of the three model pairs. Each reward variant has one completed training and formal test run, so training-seed variability remains unmeasured. 5 Results and Analysis 5.1 Main Results Table 2 reports the shared deterministic evaluation. Strict success increases from 73/325 (22.46%) for the original checkpoint to 83/325 (25.54%) for DAPO-Rule and 86/325 (26.46%) for DAPO-DeepSeek. The paired changes are +3.08+3.08 percentage points (20 improvements and 10 regressions) and +4.00+4.00 points (21 improvements and 8 regressions), respectively. Their unadjusted 95% bootstrap intervals are [0.00,6.46][0.00,6.46] and [0.92,7.38][0.92,7.38] points. The corresponding exact McNemar p-values are 0.099 and 0.024, but after correction across the prespecified code comparisons the Holm-adjusted values are 0.691 and 0.217. We therefore interpret the result as a modest directional improvement rather than a statistically conclusive gain. Tool success is 100% for all three checkpoints. Model Strict ↑ Correct ↑ Forbidden ↓ Δ p 95% CI pHolmp_Holm Qwen3.5-4B 73 (22.46%) 276 (84.92%) 236 (72.62%) — — — DAPO-Rule 83 (25.54%) 276 (84.92%) 225 (69.23%) +3.08 [0.00, 6.46] 0.691 DAPO-DeepSeek 86 (26.46%) 282 (86.77%) 224 (68.92%) +4.00 [0.92, 7.38] 0.217 Table 2: Primary deterministic results on the corrected 325-example SelectBench-v2 test set. Correct denotes a correct and complete visible answer. Confidence intervals are paired percentile-bootstrap intervals with 10,000 resamples; pHolmp_Holm is the exact McNemar p-value for strict success versus the original checkpoint after correction over the prespecified code-comparison family. Forbidden adoption falls from 236/325 (72.62%) to 225/325 (69.23%) with DAPO-Rule and 224/325 (68.92%) with DAPO-DeepSeek. The respective paired changes are −3.38-3.38 points (95% CI [−7.08,0.31][-7.08,0.31]) and −3.69-3.69 points (95% CI [−7.08,−0.31][-7.08,-0.31]); neither remains significant after Holm correction. Correct and complete answers are unchanged at 84.92% for DAPO-Rule and rise to 86.77% for DAPO-DeepSeek. The large gap between factual correctness and strict success is explained by their overlap: 203 original, 193 Rule, and 196 DeepSeek outputs are factually correct yet still adopt a forbidden phrase. Selective rejection of contamination, rather than factual answering alone, is therefore the principal bottleneck under this evaluator. Table 3 shows that this shift does not extend to prompt-injection resistance. On the 275 eligible examples, injection following changes from 14 (5.09%) to 18 (6.55%) and 16 (5.82%); both intervals include zero change. By contrast, DAPO-DeepSeek reduces over-refusal, missing final answers, and overlong responses. Mean output length also falls by about 12 generated tokens for both trained policies. Overall, the completed evidence supports improved selective adoption on the aggregate metric, but not uniform improvement across failure modes. Model Injection ↓ Over-refusal ↓ Missing final ↓ Overlong ↓ Mean tokens ↓ Qwen3.5-4B 14 (5.09%) 3 (0.92%) 9 (2.77%) 7 (2.15%) 383.18 DAPO-Rule 18 (6.55%) 4 (1.23%) 9 (2.77%) 4 (1.23%) 371.20 DAPO-DeepSeek 16 (5.82%) 1 (0.31%) 5 (1.54%) 2 (0.62%) 371.32 Table 3: Failure diagnostics on SelectBench-v2. Injection following is measured over the 275 injection-bearing examples; all other rates use 325 examples. Categories can overlap and therefore need not sum to the complement of strict success. Table 4 reports all nine comparisons in the prespecified Holm family, including the Rule–DeepSeek contrast. No comparison remains significant after family-wise correction, and the direct difference between the two reward variants is small on all three outcomes. Comparison Outcome Δ p 95% CI p pHolmp_Holm Original → Rule Strict +3.08 [0.00, 6.46] 0.099 0.691 Original → DeepSeek Strict +4.00 [0.92, 7.38] 0.024 0.217 Rule → DeepSeek Strict +0.92 [−1.85-1.85, 3.69] 0.664 1.000 Original → Rule Forbidden −3.38-3.38 [−7.08-7.08, 0.31] 0.099 0.691 Original → DeepSeek Forbidden −3.69-3.69 [−7.08-7.08, −0.31-0.31] 0.050 0.401 Rule → DeepSeek Forbidden −0.31-0.31 [−3.38-3.38, 2.46] 1.000 1.000 Original → Rule Injection +1.45 [−1.82-1.82, 4.73] 0.523 1.000 Original → DeepSeek Injection +0.73 [−2.55-2.55, 4.36] 0.839 1.000 Rule → DeepSeek Injection −0.73-0.73 [−3.64-3.64, 2.18] 0.804 1.000 Table 4: All nine prespecified paired code comparisons on SelectBench-v2. The delta is the right model minus the left model in percentage points. Intervals are paired percentile-bootstrap 95% CIs; p is the two-sided exact McNemar value and pHolmp_Holm corrects jointly over all nine rows. 5.2 Evaluator and Dataset-Version Diagnostics The primary evaluator checks factual correctness only in the visible answer, but scans the complete final assistant turn, including <think>, for forbidden adoption. It is consequently conservative when a forbidden phrase is mentioned in reasoning, and it is aligned with the training signal for DAPO-Rule. Its advantage is that the same frozen decision rule and the same 325 prompts are applied to all three checkpoints. The semantic judge used to train DAPO-DeepSeek is not used for the formal results. Dataset versioning also materially affects measurement. The original test generator used a permissive supporting-fact fallback; the v2 audit replaces only trusted evidence and changes the retrieval context in 233/325 examples, while preserving questions, answers, and contamination. All results in this section are regenerated from v2. Legacy v1 scores are excluded rather than mixed with the corrected measurement regime. 5.3 General Capability Retention On 14,042 MMLU questions, the original and DAPO-DeepSeek checkpoints obtain 67.70 and 67.70 accuracy under 5-shot evaluation (a +0.01+0.01 percentage-point difference after rounding), with a paired interval spanning zero and exact McNemar p=1.000p=1.000. Zero-shot accuracy changes from 58.65 to 58.85 (+0.21+0.21 points, 95% CI [0.06,0.36][0.06,0.36], p=0.010p=0.010). On the 7,080-question clean HotpotQA subset, fixed-context EM is identical at 60.03, while F1 changes from 74.39 to 74.41. Closed-book EM changes from 18.01 to 17.98 and F1 from 24.80 to 24.78. The paired 95% bootstrap intervals for both HotpotQA modes include zero; Table 5 reports the complete paired intervals and the exact tests available for accuracy and EM. These results show no material degradation on the two measured general-capability evaluations for DAPO-DeepSeek, but do not establish capability preservation or broader robustness beyond them. Evaluation Original DeepSeek Δ p 95% CI p MMLU 5-shot accuracy 67.697 67.704 +0.007 [−0.135-0.135, 0.150] 1.000 MMLU zero-shot accuracy 58.645 58.852 +0.207 [0.057, 0.356] 0.010 HotpotQA fixed-context EM 60.028 60.028 0.000 [−0.184-0.184, 0.184] 1.000 HotpotQA fixed-context F1 74.395 74.415 +0.020 [−0.143-0.143, 0.185] — HotpotQA closed-book EM 18.008 17.980 −0.028-0.028 [−0.198-0.198, 0.141] 0.871 HotpotQA closed-book F1 24.802 24.777 −0.024-0.024 [−0.216-0.216, 0.162] — Table 5: General-capability retention for the original and DAPO-DeepSeek checkpoints. Values and differences are percentage points. Confidence intervals use 10,000 paired bootstrap resamples. Reported p-values are two-sided exact McNemar tests; they are not defined for token-overlap F1. HotpotQA excludes all 325 SelectBench test questions. 6 Related Work 6.1 Reinforcement Learning Reinforcement learning (RL) has become a cornerstone in aligning LLMs with human preferences and enhancing their reasoning capabilities. Proximal Policy Optimization (PPO) Schulman et al. (2017) is a widely used policy gradient algorithm that stabilizes training through clipped policy updates, but maintaining four models imposes heavy computational costs. Direct preference optimization (DPO) Rafailov et al. (2023) simplifies this by directly optimizing the preference gap between chosen and rejected responses, while Group Relative Policy Optimization (GRPO) Shao et al. (2024) introduces group-based comparison to compute relative advantages across multiple outputs, enabling finer learning and strong results in LLM reasoning tasks such as mathematical problem solving. With variants including DAPO Yu et al. (2026) and GSPO Zheng et al. (2025), GRPO has become a new mainstream paradigm. Reinforcement learning extends beyond standard LLM post-training to enhance agent capabilities Sha et al. (2025); Chen et al. (2026). By designing appropriate reward functions and adapting optimization objectives to practical tasks, LLMs can learn to invoke external tools for search, file manipulation, and even complex GUI interactions Zhang et al. (2026b). However, reinforcement learning for agents faces distinct challenges, including sparse rewards arising from extended tool-use trajectories and heavy dependence on stable simulation environments. Nevertheless, these reinforcement learning methods remain essential for post-training optimization, ensuring models not only produce coherent text but also align with human values and exhibit stronger complex problem-solving abilities. 6.2 Large Language Models as Agents Viewing large language models as autonomous agents capable of planning and executing multi-step reasoning has emerged as a new paradigm Shi et al. (2026a). An agent can be decomposed into an LLM and a harness that comprises memory, planning, tools, and other components Meng et al. (2026). The LLM serves as the brain and core of the agent, iteratively refining tasks, retrieving information, and generating solutions under the orchestration of the harness Yao et al. (2026). However, the over-refusal problem inherent in LLMs persists when agents invoke retrieval tools. Our method explicitly trains the model through reinforcement learning to extract useful information from retrieved results that contain harmful content, thereby mitigating the underutilization of resources. 7 Conclusion We introduced SelectBench and a DAPO post-training framework for selective evidence adoption from retrieval contexts containing supported facts alongside misleading or instruction-like text. On the corrected SelectBench-v2 test set, both reward variants improve strict success and reduce forbidden adoption, with the DeepSeek-trained checkpoint also improving answer completeness and retaining performance on MMLU and clean HotpotQA. The gains are modest, do not remain significant after multiple-comparison correction, and do not improve prompt-injection following. These results position selective evidence adoption as a distinct and measurable post-training objective while showing that stronger injection resistance, replicated training runs, and broader retrieval settings remain necessary. References M. Chen, L. Sun, T. Li, sunhaoze, ZhouYijie, C. Zhu, H. Wang, J. Z. Pan, W. Zhang, H. Chen, F. Yang, Z. Zhou, and W. Chen (2026) ReSearch: learning to reason with search for LLMs via reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §6.1. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt (2020) Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. Cited by: §4.2. X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, p. 6609–6625. Cited by: §3.1. Y. Li, X. Yi, D. Shi, Y. Cui, G. de Melo, and L. Wang (2025a) From construction to injection: edit-based fingerprints for large language models. arXiv preprint arXiv:2509.03122. Cited by: §1. Y. Li, X. Yi, D. Shi, Y. Cui, G. de Melo, and L. Wang (2026) AGMark: attention-guided dynamic watermarking for large vision-language models. arXiv preprint arXiv:2602.09611. Cited by: §1. Y. Li, X. Yi, D. Shi, G. De Melo, X. Wang, and L. Wang (2025b) Hierarchical safety realignment: lightweight restoration of safety in pruned large vision-language models. In Findings of the Association for Computational Linguistics: ACL 2025, p. 7600–7612. Cited by: §1. Q. Meng, Y. Wang, L. Chen, Y. Li, W. Wu, W. Jiang, Q. Wang, C. Lu, Y. Gao, Y. Wu, et al. (2026) Agent harness for large language model agents: a survey. Cited by: §6.2. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §6.1. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §6.1. Z. Sha, S. Cui, and W. Wang (2025) Sem: reinforcement learning for search-efficient large language models. arXiv preprint arXiv:2505.07903. Cited by: §1, §1, §6.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §6.1. D. Shi, Y. Li, X. Yi, Y. Cui, H. Feng, and L. Wang (2026a) SURGENT: a surgical multi-agent assistance system across the perioperative workflow. arXiv preprint arXiv:2605.29368. Cited by: §1, §6.2. D. Shi, X. Yi, Y. Li, and L. Wang (2026b) Benchmarking large language models for end-to-end clinical support in traditional chinese medicine. Expert Systems with Applications, p. 132267. Cited by: §1. Q. Team (2026) Qwen3.5: accelerating productivity with native multimodal agents. External Links: Link Cited by: §2, §4.1. H. Trivedi, N. Balasubramanian, T. Khot, and A. Sabharwal (2022) MuSiQue: multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics 10, p. 539–554. Cited by: §3.1. A. Xu, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong, C. Ling, et al. (2026) Deepseek-v4: towards highly efficient million-token context intelligence. arXiv preprint arXiv:2606.19348. Cited by: §3.3. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369–2380. Cited by: §3.1. Y. Yao, X. Tan, C. Liu, Y. Li, Z. Wang, W. Yu, Z. Tan, Y. Tian, G. Zhao, L. Sun, et al. (2026) Harness-bench: measuring harness effects across models in realistic agent workflows. arXiv preprint arXiv:2605.27922. Cited by: §6.2. X. Yi, Y. Li, D. Shi, L. Wang, X. Wang, and L. He (2025a) Latent-space adversarial training with post-aware calibration for defending large language models against jailbreak attacks. Expert Systems with Applications, p. 129101. Cited by: §1. X. Yi, Y. Li, D. Shi, L. Wang, X. Wang, and L. He (2025b) Unified defense for large language models against jailbreak and fine-tuning attacks in education. arXiv preprint arXiv:2511.14423. Cited by: §1. X. Yi, Y. Li, S. Zheng, L. Wang, X. Wang, and L. He (2025c) Unified attacks to large language model watermarks: spoofing and scrubbing in unauthorized knowledge distillation. Knowledge-Based Systems, p. 114295. Cited by: §1. Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, W. Dai, T. Fan, G. Liu, L. Liu, et al. (2026) Dapo: an open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38, p. 113222–113244. Cited by: §1, §3, §6.1. K. Zhang, M. Hu, H. A. D. Le, F. K. Torsha, Z. Jiang, M. K. Bui, C. Chang, Y. Chuang, Z. Xiong, Y. Lin, G. Wang, and N. Zou (2026a) A survey on evaluating quality and trustworthiness in LLM-generated data. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §1. K. Zhang, Z. Xiong, M. Zhong, Z. Jiang, Z. Yuan, Z. Li, and Y. Lin (2026b) Are tools all we need? unveiling the tool-use tax in llm agents. External Links: 2605.00136, Link Cited by: §1, §6.1. Z. Zhang, W. Xu, F. Wu, and C. K. Reddy (2025) FalseReject: a resource for improving contextual safety and mitigating over-refusals in LLMs via structured reasoning. In Second Conference on Language Modeling, External Links: Link Cited by: §1. C. Zheng, S. Liu, M. Li, X. Chen, B. Yu, C. Gao, K. Dang, Y. Liu, R. Men, A. Yang, et al. (2025) Group sequence policy optimization. arXiv preprint arXiv:2507.18071. Cited by: §6.1.