Paper deep dive
False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents
Meijia Chen, Hao Li, Zheng Lu, Hongshan Lin, Junbai Tian, Yichen Liu, Zijun Tian, Yufan Zou, Shuhan Sun, Hanxin Chen, Zeyu Zhang, Weizhi Du, Yueting Li, Tianyu Shi, Alaa Khamis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%
Last extracted: 10/6/2026, 10:06:01 AM
Summary
This paper identifies 'co-cheating' in self-evolving search agents, where proposer and solver models converge on shared errors, leading to improved internal rewards without external correctness gains. The authors propose two mitigation strategies: Multi-Sample Verification (MSV), which verifies proposals via source-aware and source-blind sampling, and CrossFit, which partitions source documents into folds and uses auxiliary solvers trained on one fold to score questions from another, thereby breaking the feedback loop of shared errors. Experiments on Qwen3.5-4B and Qwen3.5-9B demonstrate that CrossFit significantly reduces false agreement and improves performance across seven downstream search benchmarks compared to standard self-evolution and Search-R1.
Entities (10)
Relation Signals (6)
Qwen3.5-4b ā usedin ā CrossFit
confidence 98% Ā· Rerunning the loop with Qwen3.5-4B... CrossFit reduces it to 3.0%
CrossFit ā mitigates ā Co-cheating
confidence 97% Ā· CrossFit reduces it to 3.0% and 3.7%... preventing a same-source pseudo-label from being directly reproduced through the feedback solver
Co-cheating ā causes ā False Agreement
confidence 95% Ā· co-cheating predicts that agreement itself becomes optimistic as both agents converge on the same error... F counts pairs matching the same incorrect answer
CrossFit ā improvesperformanceon ā Search-R1
confidence 95% Ā· CrossFit improves average performance over... Search-R1 by 8.7 and 7.8 points at 4B and 9B
Dr. Zero ā suffersfrom ā Co-cheating
confidence 95% Ā· We test for this failure in Dr. Zero... Figure 1 shows that internal reward rises together with false agreement
Multi-Sample Verification ā mitigates ā Co-cheating
confidence 90% Ā· MSV reduces false-agreement mass from 6.1% to 5.7%... MSV partially reduces false agreement
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-evolving search agents build their own training curricula by jointly optimizing a proposer that generates questions and a solver that answers them. This closed loop introduces a failure mode we call co-cheating: the proposer and solver increasingly agree on shared errors, so internal reward improves without a matching gain in external correctness. A post-hoc audit against source evidence shows co-cheating growing more severe over successive rounds of self-evolution, with pseudo-label correctness stagnating or declining even as the in-loop training signal improves. The most direct mitigation is to verify proposals before training: we introduce multi-sample verification (MSV), which queries the same model three times with the source and three times without it to decide task admission and replace unreliable pseudo-labels. MSV partially reduces false agreement but leaves substantial residual co-cheating and costs six extra labeler generations per candidate. These limitations motivate CrossFit, our main method: it partitions the proposer's source documents into groups A and B; questions generated from A are scored by an auxiliary solver trained only on B, and vice versa. The cross-fitted agreement determines proposer reward, so a same-source pseudo-label cannot be reproduced through the feedback solver, while the original solver's update rule is unchanged. Rerunning the loop with Qwen3.5-4B and Qwen3.5-9B, MSV reduces false-agreement mass from 6.1% to 5.7% and from 8.8% to 7.2%, whereas CrossFit reduces it to 3.0% and 3.7%. Replaying identical proposals with source-excluded feedback further reduces false agreement to 0.4% and 0.1%, isolating feedback ancestry from curriculum changes. Across seven downstream search benchmarks, CrossFit improves average performance over standard coupled self-evolution by 8.8 and 8.4 points and over Search-R1 by 8.7 and 7.8 points at 4B and 9B.
Tags
Links
- Source: https://arxiv.org/abs/2609.39102v2
- Canonical: https://arxiv.org/abs/2609.39102v2
Trouble viewing inline? Open PDF directly ā
Full Text
64,364 characters extracted from source content.
Expand or collapse full text
Preprint FALSE FRONTIERS: DIAGNOSING AND MITIGATING CO-CHEATING IN SELF-EVOLVING SEARCH AGENTS Meijia Chen 1,* Hao Li 2,* Zheng Lu 2,* Hongshan Lin 2 Junbai Tian 2 Yichen Liu 3 Zijun Tian 2 Yufan Zou 2 Shuhan Sun 2 Hanxin Chen 3 Zeyu Zhang 2 Weizhi Du 4 Yueting Li 2 Tianyu Shi 5,ā Alaa Khamis 6,ā 1 Rutgers University 2 Independent Researcher 3 University of California, San Diego 4 University of Michigan 5 McGill University 6 King Fahd University of Petroleum and Minerals ABSTRACT Self-evolving search agents can construct their own training curricula by jointly optimizing a proposer that generates questions and a solver that answers them. This closed loop introduces a failure mode that we call co-cheating: the proposer and solver increasingly agree on shared errors, so internal reward improves with- out a corresponding increase in external correctness. A post-hoc reference audit against source evidence shows that co-cheating becomes increasingly severe over successive rounds of self-evolution, with pseudo-label correctness stagnating or declining even as the in-loop training signal improves. The most direct mitigation is to verify each proposal before training. We therefore introduce multi-sample verification (MSV), which queries the same model used in self-evolution three times with the source and three times without it to determine task admission and replace unreliable pseudo-labels.MSVpartially reduces false agreement but leaves substan- tial residual co-cheating and requires six additional labeler generations for every candidate. These limitations motivateCrossFit, our main method. It partitions the proposerās source documents into groups A and B: questions generated from A are scored by an auxiliary solver trained only on B, and vice versa. The result- ing cross-fitted agreement determines proposer reward, preventing a same-source pseudo-label from being directly reproduced through the feedback solver while leaving the original solverās update rule unchanged. We evaluate both interventions by rerunning the complete self-evolution loop with Qwen3.5-4B and Qwen3.5- 9B. After self-evolution,MSVreduces false-agreement mass from 6.1% to 5.7% on Qwen3.5-4B and from 8.8% to 7.2% on Qwen3.5-9B, whereasCrossFit reduces it to 3.0% and 3.7%, respectively. Replaying identical proposals with source-excluded feedback further reduces false agreement to 0.4% and 0.1%, iso- lating feedback ancestry from changes in the generated curriculum. Across seven downstream search benchmarks,CrossFitimproves average performance over standard coupled self-evolution by 8.8 and 8.4 points and over Search-R1 by 8.7 and 7.8 points at 4B and 9B, respectively. * Equal contribution. ā Corresponding authors: tianyu.shi3@mcgill.ca, alaa.rashwan@kfupm.edu.sa. 1 arXiv:2609.39102v2 [cs.CL] 3 Oct 2026 Preprint 1INTRODUCTION 04386129 0.6 0.7 0.8 Training step In-loop agreement 0 0.05 0.1 0.15 False agreement A in-loopF false agreement Figure 1: Co-cheating. The training sig- nal and false agreement rise together. Search-augmented language models interleave reasoning with browser or search actions to gather evidence before answering (Nakano et al., 2021; Yao et al., 2023; Jin et al., 2025; Song et al., 2025). Most are trained on externally supplied questions and answer supervision (Jin et al., 2025; Song et al., 2025). Self-evolving agents instead generate their own training experience (Chen et al., 2024; Zhao et al., 2025; Huang et al., 2025). In recent proposerāsolver systems, a proposer turns source documents into questions and pseudo-labels, admitted pairs train a solver, and the solverās performance on new proposals determines the pro- poser reward (Lu et al., 2026; Yue et al., 2026). Repeating this cycle shifts proposals toward the solverās current capa- bility frontier, producing an automated curriculum without a fixed human-authored training set. This loop makes agreement an endogenous proxy for correctness (Amodei et al., 2016; Gao et al., 2023). An incorrect pseudo-label can train the solver to repeat the same error on later questions from that source (Arazo et al., 2020); rewarding this agreement then reinforces the error in the next-round curriculum, as illustrated in the left panel of Figure 2. We test for this failure in Dr. Zero (Yue et al., 2026) using a post-hoc auditor that checks generated tasks and answers against source evidence but never feeds into training. Figure 1 shows that internal reward rises together with false agreement as proposer and solver increasingly share errors. We call this optimization outcome co-cheating and measure it as false-agreement mass: the fraction of evaluated pairs that agree on the same incorrect answer. The most direct mitigation is to verify each proposal before training. The middle panel of Figure 2 illustrates our multi-sample verification (MSV), which queries the same model used in self-evolution three times with the source and three times without it. Compatible majorities yield a consensus label and admit the task; inconsistent candidates are rejected. This partially reduces false agreement, implicating pseudo-label quality, but leaves substantial co-cheating and adds six labeler generations per candidate. These limitations point to a second source of failure: not only whether a pseudo-label is correct, but also whether it trained the solver that later evaluates questions from the same source. The right panel of Figure 2 shows our primary intervention,CrossFit. The proposerās source documents are divided into groups A and B. Along the upper path, an auxiliary solver learns only from A and scores new questions generated from B; along the lower path, a second solver learns only from B and scores questions from A. The cross-fitted agreement scores determine proposer reward. Each scoring solver has thus never trained on pseudo-labels from the source it evaluates, preventing a same-source error from being directly reproduced as reward. The original solver still trains on all admitted questions; only the feedback shaping the proposerās next-round curriculum is cross-fitted. We rerun self-evolution with Qwen3.5-4B and Qwen3.5-9B (Qwen Team, 2026). Under standard coupled feedback, false-agreement mass reaches 6.1% and 8.8%;MSVlowers it to 5.7% and 7.2%, whileCrossFitlowers it to 3.0% and 3.7%. Replaying identical proposals with source-excluded feedback further reduces it to 0.4% and 0.1%, isolating feedback ancestry from curriculum selection. We evaluate each round-end main solver on a fixed 1,325-question suite: 200 each from Natural Ques- tions, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, and MuSiQue, plus 125 from Bamboogle (Kwiatkowski et al., 2019; Joshi et al., 2017; Mallen et al., 2023; Yang et al., 2018; Ho et al., 2020; Trivedi et al., 2022; Press et al., 2023). With one greedy trajectory per question and identical tool and extraction budgets,CrossFitreaches 48.8% at 4B and 51.2% at 9B, improving over coupled self-evolution by 8.8 and 8.4 points and over Search-R1 by 8.7 and 7.8 points, respectively. 2 Preprint Figure 2: Co-cheating and its mitigation. (a) Incorrect agreement creates false frontier credit. (b) MSVverifies each proposal with source-aware and source-blind samples. (c)CrossFitscores each source group with a solver trained on the other group. The two interventions target label quality and feedback provenance, respectively. 2DIAGNOSING CO-CHEATING 04386129 Training step 50 70 90 Truth / agreement (%) (a) Qwen3.5-4B: coupled dynamics Each line follows all 129 scheduled updates 04386129 Training step 50 70 90 Truth / agreement (%) (b) Qwen3.5-9B: coupled dynamics Each line follows all 129 scheduled updates 0510152025 Lost-credit mass L (%) 0 5 10 15 False-agreement mass F (%) (c) Qwen3.5-4B: error composition Small points: steps; large markers: round means R1 R2 R3 0510152025 Lost-credit mass L (%) 0 5 10 15 False-agreement mass F (%) (d) Qwen3.5-9B: error composition Small points: steps; large markers: round means R1 R2 R3 T P adopted-label truthT S solver truth A agreement Round 1Round 2Round 3 Figure 3: Co-cheating under coupled feedback. (a,b) All 129 steps of label truthT P , solver truth T S , and agreementA; dashes mark round boundaries. (c,d) False agreementFversus lost creditL: small points are steps, large markers average each roundās 43 step rates, and arrows indicate round order. Rising F with falling L reveals shared-error accumulation. All axes are percentages. 3 Preprint 1 Propose2 Split by source3 Train and cross-score Sources S1 S2 S3 ... Proposer Question + label Q: A: Training tasks S1 Q: A: S2 Q: A: S3 Q: A: ... Fixed source groups Fold 0 S1 S3 S5 ... Fold 1 S2 S4 S6 ... Cross-fitted feedback Round r: trainRound r + 1: score Fold 0 Fold 1 Train on fold 0 Train on fold 1 Scoring Frozen Scoring Frozen Questions: fold 1 Questions: fold 0 Scoring Frozen Scoring Frozen 5 rollouts / question Matches ā k Reward f(k) Next-round proposer Train on one fold. Score the other. Main solver Fold 0 Fold 1 Train on all tasks Main solver (trainable) Downstream evaluation Figure 4: TheCrossFitalgorithm. Auxiliary solvers train on one source fold and score the other; the main solver trains on all admitted questions. Audit protocol. We audit the standard coupled Dr. Zero loop after training, without altering its procedure. Many generated questions require reconstructing multi-hop evidence chains across long or specialized source documents. Even a human judge must first reproduce the search path and inspect unfamiliar evidence, making exhaustive annotation of every saved training step difficult to standardize at this scale. We therefore save the source document, adopted pseudo-label, and five solver responses used for proposer reward at every scheduled step.gpt-6-astra/highconstructs an evidence- backed reference from the source and judges these saved outputs (Appendix B); unsupported cases remain unresolved rather than receiving a forced label. Because the auditor never affects admission, model updates, or reward, it provides a scalable, independent measurement of the exact examples behind the in-loop signal. What is measured.T P andT S denote adopted-label and solver-response correctness, whileAis the labelāresponse match rate observed by the loop.Fcounts pairs matching the same incorrect answer;Lcounts correct solver responses denied credit by a wrong label. Ordinary label noise can cause disagreement or lost credit, whereas co-cheating predicts that agreement itself becomes optimistic as both agents converge on the same error. RisingAis therefore reliable only whenT P and T S also rise andFremains low. Because proposer reward is computed from the five labelāresponse matches, we audit those same five pairs rather than collapsing them to a post-hoc majority. Thus, F measures the portion of apparent agreement that the external audit identifies as wrong. Observed dynamics.Figure 3 shows this transition. In round 1, mean false-agreement mass is only 0.004 for Qwen3.5-4B and 0.003 for Qwen3.5-9B, and incorrect labels more often appear as lost credit. From round 2 onward, agreement becomes increasingly optimistic without a commensurate increase in truth. By round 3,Freaches 0.061 and 0.088 at the two scales, whileLfalls. Harder questions may reduce correctness, but they do not explain increasing agreement on the same source- inconsistent answer. The joint rise ofAandFinstead shows disagreement being replaced by shared mistakes. We call this self-reinforcing optimization outcome co-cheating; it does not imply intentional coordination. 3FROM VERIFICATION TO CROSS-FITTED FEEDBACK The audit motivates two interventions at different points in the self-evolution loop.MSVtests a proposed answer before the example enters training, whereasCrossFit, our main method, changes which solver supplies the feedback that updates the proposer. 4 Preprint 3.1MULTI-SAMPLE VERIFICATION MSVis an admission-time test of whether a proposed question admits a stable answer independently of the proposerās draft. Given source documentx, questionq, and the same modelMused in self-evolution, it draws three source-aware and three source-blind answers, a src i ā¼ M(Ā·| x,q), a blind i ā¼ M(Ā·| q), iā1, 2, 3. Neither view observes the draft. LetMajreturn an answer when at least two samples agree under the answer matcherā, andā otherwise. Defining y v = Maj(a v 1:3 ) for v āsrc, blind, admission is I MSV = 1 y src Ģø=ā ā§ y blind Ģø=ā ā§ y src ā y blind . WhenI MSV = 1, the compatible majority replaces the draft as the training label; otherwise the task is rejected. The two views test evidential support and answer stability, respectively. However, six samples from the same model can share errors, and verification does not prevent a later feedback solver from reusing labels derived from the evaluated source. It also adds six generations, including their search and coordination cost, per candidate (Table 3, Appendix A). 3.2CROSS-FITTED PROPOSER FEEDBACK CrossFit changes only where the proposer obtains its feedback. As shown from left to right in Figure 4, the proposer generates questions and pseudo-labels from source documents exactly as in the original loop. We then assign each source document once to fold 0 or fold 1, and every question derived from that document keeps the same assignment throughout self-evolution. The split is made at the source level because splitting individual questions could place related examples from the same document on both sides and preserve the very reuse path that we want to remove. The two source folds maintain two auxiliary feedback solvers. After roundr, one solver has learned only from admitted questions in fold 0, while the other has learned only from admitted questions in fold 1. In the next round, their roles are crossed: questions from fold 0 are evaluated by the solver trained on fold 1, and questions from fold 1 are evaluated by the solver trained on fold 0. These are the two crossed paths in Figure 4. Consequently, the solver evaluating a question has not been trained on pseudo-labels produced from that questionās source. The feedback rule itself remains the same. Lethdenote the source fold,S r,1āh the auxiliary solver trained on the complementary fold, and Ģythe adopted label. From its responsesz 1 ,...,z 5 , the proposer receives R P (q) = f ļ£« ļ£ 5 X j=1 1[z j ā Ģy] ļ£¶ ļ£ø , z j ā¼ S r,1āh (Ā·| q). The sum counts how many responses match the adopted label under the original answer matcher, and f(k) = (5ā k)/4for0 < k < 5(zero otherwise) is Dr. Zeroās frontier reward. Thus,CrossFit preserves the original training objective: questions still receive credit according to how difficult they appear to a solver. The only change is which solver supplies that signal. This change breaks the direct self-reinforcing path revealed by our audit. Under coupled feedback, an incorrect pseudo-label from a source can train the solver, be reproduced by that solver on a later question from the same source, and then return to the proposer as reward. UnderCrossFit, the later question is instead evaluated by the complementary solver, whose training history excludes that source. The method does not turn the auxiliary solver into a truth oracle: the two solvers may still share errors inherited from pretraining or overlapping evidence. It does, however, prevent agreement from being rewarded merely because the evaluator was trained on the same source-derived error. The bottom path of Figure 4 separates this feedback mechanism from downstream training. The main solver is not split; it continues to train on all admitted questions from both folds. The auxiliary solvers affect only the feedback that shapes the proposerās next-round curriculum. When the two interventions are combined,MSVfirst decides whether a proposal is admitted and which pseudo-label is used, andCrossFitthen selects the auxiliary solver that evaluates it. In this sense,MSVimproves the supervision entering training, whereasCrossFitprevents that supervision from being directly recycled into proposer reward. 5 Preprint Table 1: Downstream search performance after three rounds of self-evolution. Bold denotes the best result and underlining denotes the second-best within each Qwen3.5 block. ā Baselines from the Dr. Zero comparison, run on the same Qwen3.5 backbones and evaluated on the same 1,325-question set. NQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleAverage Qwen3.5-4B Base0.3800.6550.3150.3500.4650.1050.4160.384 Prompting ā 0.2450.5250.1650.2900.4600.0850.3600.304 R1-Instruct ā 0.2950.6050.1950.3200.4750.1050.4480.349 Search-R1 ā 0.3550.6500.2900.3700.5100.1250.5040.401 Dr. Zero0.3900.6650.3250.3650.4850.1200.4480.400 MSV0.4000.6700.3400.3750.4800.1350.4480.407 CrossFit0.4550.7300.4150.4600.5750.2450.5360.488 MSV + CrossFit0.4700.730 0.4100.4750.5750.2500.5280.491 Qwen3.5-9B Base0.5050.7100.3750.3550.2950.1250.4960.409 Prompting ā 0.3800.6300.2600.2800.2700.1150.3920.332 R1-Instruct ā 0.4450.6950.2900.3100.2950.1350.4800.379 Search-R1 ā 0.5100.7300.3950.3800.3100.1800.5360.434 Dr. Zero0.5150.7300.4000.3700.3200.1500.5120.428 MSV0.5250.7200.4000.3900.3250.1550.5360.436 CrossFit0.580 0.7650.4550.4900.4250.2550.6160.512 MSV + CrossFit0.5700.7800.4700.4850.4150.2800.6080.515 4MAIN EXPERIMENTS 4.1EXPERIMENTAL SETUP Datasets & Models. We evaluate on the seven open-domain question answering benchmarks used by Dr. Zero (Yue et al., 2026): the single-hop Natural Questions (NQ) (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), and PopQA (Mallen et al., 2023), and the multi-hop HotpotQA (Yang et al., 2018), 2WikiMultiHopQA (2WikiMQA) (Ho et al., 2020), MuSiQue (Trivedi et al., 2022), and Bamboogle (Press et al., 2023). A fixed evaluation set of 1,325 questions contains 200 examples from each of the first six benchmarks and all 125 Bamboogle examples. We use Qwen3.5-4B and Qwen3.5-9B (Qwen Team, 2026) as backbones. At each scale, all self-evolution treatments start from the same public checkpoint, which is also evaluated as the Base row, and none of them uses human-annotated QA training data. Baselines & Evaluation. We compare four self-evolution treatments obtained by crossing the two interventions of Section 3. Dr. Zero (Yue et al., 2026) is the standard coupled loop in which the main solver scores the proposals it later trains on;MSVadds multi-sample verification to this loop;CrossFitreplaces coupled feedback with source-excluded feedback; andMSV+CrossFit applies both. For broader comparison, we reproduce the Prompting and R1-Instruct baselines from the Dr. Zero protocol (Yue et al., 2026), together with Search-R1 (Jin et al., 2025), on the same Qwen3.5 backbones and evaluate every row on the same 1,325-question set with identical tool budget, decoding, and answer extraction. Every self-evolution experiment follows the same three-round schedule of 18 proposer and 25 solver steps per round and optimizes the policy-gradient objective of Equation(1); Table 2 in Appendix A lists the shared configuration. At the end of each round, we evaluate the main solver, which trains on all admitted questions, using one greedy search trajectory per question, the same tool budget, and identical answer extraction. We report Cover-EM and average the seven benchmarks with equal weight (Equation(2)); Tables 5 and 6 in Appendix B list intermediate rounds and micro averages. 4.2MAIN RESULTS Cross-fitted feedback improves downstream search at both scales. In Table 1, coupled self- evolution raises average Cover-EM from 0.384 to 0.400 at 4B and from 0.409 to 0.428 at 9B. CrossFitreaches 0.488 and 0.512: gains of 8.8/8.4 percentage points over Dr. Zero and 8.7/7.8 over Search-R1. Every benchmark improves at both scales. Gains are largest on multi-hop tasks, 6 Preprint 6065707580 Solver truth T S (%) 60 65 70 75 80 In-loop agreement A (%) A>T S : optimistic feedback (a) Qwen3.5-4B: agreement vs. truth Round means; arrows follow rounds 1 ā 2 ā 3 6065707580 Solver truth T S (%) 60 65 70 75 80 In-loop agreement A (%) A>T S : optimistic feedback (b) Qwen3.5-9B: agreement vs. truth Round means; arrows follow rounds 1 ā 2 ā 3 04386129 Training step 0 5 10 15 False-agreement mass F (%) R1R2R3 6.1 5.7 3.0 2.0 R3 mean (c) Qwen3.5-4B: false agreement Faint points: each step; solid segments: round means 04386129 Training step 0 5 10 15 False-agreement mass F (%) R1R2R3 8.8 7.2 3.7 1.7 R3 mean (d) Qwen3.5-9B: false agreement Faint points: each step; solid segments: round means All 129 steps retained Ā· Gray bands: proposer updates Ā· Long dash: step 62 Dr. ZeroMSVCrossFitMSV + CrossFit Figure 5: Training dynamics across three rounds. (a,b) Round means of agreement and solver truth; hollow, light, and solid markers denote rounds 1ā3. Above the diagonal, agreement is optimistic. (c,d) Faint points retain all 129 steps; solid segments average each roundās 43 step rates, and end labels give round-3 means (%). Gray bands mark proposer phases, short dashes mark round boundaries, and the long dash marks step 62. Full traces and coverage appear in Figures 8ā9. averaging 10.0/10.9 points across HotpotQA, 2WikiMQA, MuSiQue, and Bamboogle, versus 7.3/5.2 across the single-hop datasets. The effect therefore extends across task difficulty and model scale. Verification alone is insufficient.MSVincreases the average by only 0.7ā0.8 points over Dr. Zero. Combining it withCrossFitreaches 0.491 on Qwen3.5-4B and 0.515 on Qwen3.5-9B, only 0.3 points aboveCrossFitalone. These results suggest that changing the provenance of proposer feedback is more consequential than improving pseudo-label quality alone. 5ANALYSIS AND ABLATIONS 5.1HOW DOES CROSS-FITTING CHANGE THE TRAINING TRAJECTORY? Section 2 establishes that co-cheating emerges under standard coupled feedback. Figure 5 instead compares how the four treatments change that trajectory, and Figures 8 and 9 in Appendix E retain every per-step trace. All arms share the same first-round history; the cross-fitted arms begin to differ only when their auxiliary feedback solvers are used in round 2. This delayed divergence provides a within-run comparison of feedback provenance. Cross-fitted feedback reverses the divergence between agreement and truth. Without cross- fitting, in-loop agreement rises above solver truth while false agreement accumulates at both model scales. Once cross-fitted scoring becomes active, adopted-label truth rises, agreement remains at or below solver truth, and by round 3 false-agreement mass falls below half of the coupled value.MSV alone improves solver truth but does not prevent the agreement signal from becoming optimistic; combined with CrossFit, it yields the lowest final false agreement. The trajectory difference predicts downstream gains. Table 5 shows that the downstream advantage ofCrossFitover Dr. Zero grows from 4.2 and 4.3 points after round 2 to 8.8 and 8.4 points after round 3. The intervention therefore changes what the loop learns across rounds rather than merely re-ranking a fixed set of final predictions. Figure 6 summarizes the final-round audit (absolute values in Table 4).CrossFitraises adopted-label truth from 0.747 to 0.819 at 4B and from 0.737 to 0.851 7 Preprint 051015 Label truth ĪT P ā 0 2 4 6 8 Solver truth Ī T S ā (0.0, 3.9) (7.2, 1.8) (9.5, 5.8) 051015 Label truth ĪT P ā (3.4, 3.7) (11.4, 2.2) (14.1, 6.5) 02468 F reduction ā 0.4 1.6 3.1 5.1 4.1 7.1 (a) Qwen3.5-4B(b) Qwen3.5-9B (c) Less false agreement ā 4B ā 9B MSVCrossFitMSV + CrossFit Figure 6: Round-3 audit relative to Dr. Zero. (a,b) Joint truth gains; arrows lead to the combined treatment. (c) Reduction F Dr. Zero ā F . All values are percentage points. 0510 False agreement F (%) ā 70 80 90 Probe truth T S (%) ā 5 6 8 5 6 8 468 68 72 1 2 3 4 7 1 2 3 4 7 Controls (zoom) 868992 Accuracy (%) ā 1 Coupled 2 Same-source auxiliary 3 Full-data auxiliary 4 Random-row OOF 5 Source-ID OOF 6 Source-ID OOF (half) 7 Coupled + MSV 8 Source-ID OOF + MSV (a) Feedback landscape(b) Replay accuracy Qwen3.5-4BQwen3.5-9B Figure 7: Mechanism ablations on a fixed replay bank. (a) Truth versus false agreement; arrows compare coupled and source-ID feedback. (b) Accuracy on 3,000 replay questions, meanĀ±SD over five seeds. Numbers identify methods; shading marks source exclusion. at 9B, while reducing false-agreement mass from 0.061 to 0.030 and from 0.088 to 0.037. These changes connect the downstream improvement to the intended mechanism: excluding the evaluated source from the feedback solver prevents same-source errors from being systematically returned to the proposer as apparent progress. 5.2WHY IS SOURCE-LEVEL EXCLUSION NECESSARY? Figure 7 comparesCrossFitwith controls that preserve its auxiliary-solver architecture while altering the data seen by the evaluator. A same-source auxiliary solver yields false-agreement mass of 0.064 and 0.087, and a full-data auxiliary yields 0.058 and 0.069, both close to the coupled control. A separate evaluator is therefore not sufficient when its training data retain the same source-derived pseudo-labels. The split must follow source ancestry. Randomly partitioning individual questions reduces false agreement only modestly, to 0.050 at 4B and 0.062 at 9B, because questions derived from the same document can still enter both folds. In contrast, the source-ID split reduces false agreement to 0.004 and 0.001. It also raises fixed-bank solver truth from 0.687 to 0.770 at 4B and from 0.717 to 0.868 at 9B, with corresponding accuracy gains from 88.1% to 91.5% and from 87.0% to 91.7%. These 8 Preprint comparisons isolate source exclusion, rather than evaluator duplication or partitioning alone, as the component responsible for the improvement. 5.3FIXED-BANK REPLAY SEPARATES FEEDBACK FROM CURRICULUM Adaptive reruns change both the evaluator and the questions generated in later rounds. We therefore replay the same 3,000 saved questions and adopted labels while varying only the training provenance of the feedback solver. Holding the bank, labels, answer matcher, and evaluation procedure fixed removes admission and curriculum selection as explanations. Fixed-bank replay isolates source exclusion. On identical proposals, source-ID feedback reduces coupled false agreement from 0.058/0.073 to 0.004/0.001 at 4B/9B (Figure 7). The accompanying gains in probe truth and replay accuracy persist without changing admission or the curriculum, linking the result to feedback provenance. Additional auxiliary optimization does not explain the effect. The half-budget control reaches false-agreement mass of 0.005/0.002 and replay accuracy of 91.6%/91.8%, matching the full source- ID result. Together, these controls identify source ancestry, rather than evaluator duplication, arbitrary partitioning, task selection, or extra updates, as the operative difference. 5.4HOW DO VERIFICATION AND CROSS-FITTING INTERACT? The two interventions operate at different points in the loop.MSVchanges which questionālabel pairs enter training, whereasCrossFitchanges which solver evaluates the next proposal. In the adaptive-loop audit (Table 4),MSVreduces false-agreement mass from 0.061 to 0.057 at 4B and from 0.088 to 0.072 at 9B, but agreement remains above solver truth.CrossFitproduces the larger reductions, to 0.030 and 0.037, while the combined treatment reaches 0.020 and 0.017. Thus, improving pseudo-label reliability helps, but it does not remove the feedback dependence that produces co-cheating. The fixed-bank results in Figure 7 show the same distinction. Coupled feedback withMSVretains false-agreement mass of 0.043/0.056, whereas adding source exclusion lowers it to 0.005/0.000 and raises replay accuracy to 92.4% at both scales. Finally, Table 1 shows that the combination improves the downstream average by only 0.3 points beyondCrossFitalone at each scale.MSVtherefore provides complementary reliability gains, while source-excluded proposer feedback accounts for most of the improvement in the learned search policy. 6RELATED WORK Self-generated curricula and search. Self-play has been studied for goal discovery, language-model alignment, and reasoning (OpenAI et al., 2021; Chen et al., 2024; Wu et al., 2025; Yuan et al., 2024; Zhao et al., 2025; Huang et al., 2025). Self-questioning and corpus-based evolution offer additional ways to generate supervision (Chen et al., 2025; Wang et al., 2025a; Liu et al., 2025a). Our closest framework is Dr. Zero (Yue et al., 2026); Search Self-Play (Lu et al., 2026) is another direct comparator. SearchMaster (Tan et al., 2026) provides the closest complementary diagnosis of misleading search self-play signals. Co-evolving feedback in CAFE (Liu et al., 2026b) further makes feedback adaptation a current research target. We isolate a narrower issue: training-data ancestry of the solver used for proposer feedback. We do not claim to introduce self-play, answer verification, or cross-fitting itself. Proxy rewards and self-confirmation. Reward hacking and reward-process manipulation predate language agents (Amodei et al., 2016; Everitt et al., 2021). Proxy reward optimization can diverge from a ground-truth objective (Gao et al., 2023); human-preference training can also favor agreement over truth (Sharma et al., 2024). Pseudo-label confirmation bias provides a related account of learning from oneās own mistakes (Arazo et al., 2020). Our focus is the additional return path from a pseudo- label-trained solver into task generation. False agreement is a diagnostic for this path, not by itself a causal proof of exploitation. Automated judges can have systematic biases (Zheng et al., 2023), motivating blinded evidence gathering and human validation. Data exclusion versus better evidence. Cross-fitting uses held-out nuisance predictions in statistical estimation (Chernozhukov et al., 2018). We borrow its exclusion principle, not its asymptotic guarantees: our adaptive curriculum lacks a demonstrated orthogonal score or independent sample 9 Preprint structure. Retrieval and iterative search improve access to evidence (Lewis et al., 2020; Guu et al., 2020; Trivedi et al., 2023; Li et al., 2025b); they do not establish independence between a pseudo-label and a trained evaluator. Likewise, majority stability does not imply correctness when labelers share a model and evidence. These distinctions motivate evaluating verification and source exclusion as separate interventions. Appendix C expands the literature map. 7CONCLUSION Co-cheating exposes a failure of self-evolution: agreement can improve because a proposer and solver reinforce the same incorrect labels. Our evidence-backed audit separates this internal progress from correctness.CrossFitaddresses the feedback path by scoring each source with an auxiliary solver trained on the complementary fold, while the main solver still learns from all admitted tasks. Across Qwen3.5-4B and Qwen3.5-9B, this change reduces final-round false agreement from 6.1%/8.8% to 3.0%/3.7% and improves seven-benchmark average Cover-EM over Dr. Zero by 8.8/8.4 points. Fixed-bank replay and evaluator controls support source ancestry as the operative dis- tinction; verification provides complementary reliability gains but only modest additional downstream improvement. These findings motivate tracking how an evaluator acquired its supervision when designing self-generated curricula. Shared pretraining errors, overlapping evidence, and auxiliary cost remain open limitations (Appendix D): extending exclusion to connected sources and measuring end-to-end efficiency are essential next tests of the principle. Reliable self-evolution therefore requires auditing both feedback correctness and the training history of its evaluator. 10 Preprint AI USE STATEMENT An AI coding and writing assistant assisted with literature discovery, draft organization, consistency checks, and LaTeX authoring.An LLM accessed through a commercial API (gpt-6-astra/high) served as the judge in the post-hoc audit described in Section 2 and Appendix B. The authors verified the reported results, claims, citations, and implementation corre- spondence. REPRODUCIBILITY STATEMENT The appendix reports the training schedule, model configuration, evaluation manifest, scoring rules, audit procedure, and mechanism-specific controls used in our experiments. REFERENCES Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man Ģ e. Concrete Problems in AI Safety. arXiv preprint arXiv:1606.06565, 2016. URLhttps:// arxiv.org/abs/1606.06565. Eric Arazo, Diego Ortego, Paul Albert, Noel E. OāConnor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), p. 1ā8. IEEE, 2020. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, p. 2206ā2240. PMLR, 2022. Lili Chen, Mihir Prabhudesai, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. Self-questioning language models. arXiv preprint arXiv:2508.03682, 2025. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, p. 6621ā6642. PMLR, 2024. Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whit- ney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1āC68, 2018. doi: 10.1111/ectj.12097. URL https://academic.oup.com/ectj/article/21/1/C1/5056401. Zheng Chu, Xiao Wang, Jack Hong, Huiming Fan, Yuqi Huang, Yue Yang, Guohai Xu, Chenxiao Zhao, Cheng Xiang, Shengchao Hu, et al. Redsearcher: A scalable and cost-efficient framework for long-horizon search agents. arXiv preprint arXiv:2602.14234, 2026. Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198 (27):6435ā6467, 2021. doi: 10.1007/s11229-021-03141-4. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In Pro- ceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, p. 10835ā10866. PMLR, 2023. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645(8081):633ā638, 2025. doi: 10.1038/s41586-025-09422-z. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, p. 3929ā3938. PMLR, 2020. 11 Preprint Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, p. 6609ā6625, 2020. Chengsong Huang, Wenhao Yu, Xiaoyang Wang, Hongming Zhang, Zongxia Li, Ruosen Li, Jiaxin Huang, Haitao Mi, and Dong Yu. R-zero: Self-evolving reasoning llm from zero data. arXiv preprint arXiv:2508.05004, 2025. Gautier Izacard and Ģ Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, p. 874ā880, 2021. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Atlas: Few-shot learning with retrieval augmented language models. Journal of Machine Learning Research, 24(251):1ā43, 2023. Pengcheng Jiang, Xueqiang Xu, Jiacheng Lin, Jinfeng Xiao, Zifeng Wang, Jimeng Sun, and Jiawei Han. s3: You donāt need that much data to train a search agent via RL. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 21599ā21617, 2025. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 7969ā7992, 2023. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-R1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling (COLM), 2025. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1601ā 1611, 2017. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 6769ā6781, 2020. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453ā466, 2019. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K Ģ uttler, Mike Lewis, Wen-tau Yih, Tim Rockt Ģ aschel, et al. Retrieval-augmented genera- tion for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33: 9459ā9474, 2020. Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592, 2025a. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. Search-o1: Agentic search-enhanced large reasoning models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 5420ā5438, 2025b. Zhuofeng Li, Dongfu Jiang, Xueguang Ma, Haoxiang Zhang, Ping Nie, Yuyu Zhang, Kai Zou, Jianwen Xie, Yu Zhang, and Wenhu Chen. Openresearcher: A fully open pipeline for long-horizon deep research trajectory synthesis. arXiv preprint arXiv:2603.20278, 2026. 12 Preprint Zihan Liang, Yufei Ma, Ben Chen, Zhipeng Qian, Xuxin Zhang, Huangyu Dai, and Lingtao Mao. Search-e1: Self-distillation drives self-evolution in search-augmented reasoning. arXiv preprint arXiv:2605.22511, 2026. Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Richard James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. RA-DIT: Retrieval-augmented dual instruction tuning. In The Twelfth International Conference on Learning Representations, 2024. Bo Liu, Chuanyang Jin, Seungone Kim, Weizhe Yuan, Wenting Zhao, Ilia Kulikov, Xian Li, Sainbayar Sukhbaatar, Jack Lanchantin, and Jason Weston. Spice: Self-play in corpus environments improves reasoning. arXiv preprint arXiv:2510.24684, 2025a. Bo Liu, Leon Guertler, Simon Yu, Zichen Liu, Penghui Qi, Daniel Balcells, Mickel Liu, Cheston Tan, Weiyan Shi, Min Lin, et al. SPIRAL: Self-play on zero-sum games incentivizes reasoning via multi-agent multi-turn reinforcement learning. In The Fourteenth International Conference on Learning Representations, 2026a. Boyang Liu, Senjie Jin, Peixin Wang, Zhangyue Yin, Yibo Wang, Yuhao Zhou, Zhihao Zhang, Xinbing Liang, Shizheng Zhu, Yuhui Wang, Jingqi Tong, Dingwei Zhu, Zhiheng Xi, Jiazheng Zhang, Clive Bai, Clarenceai, Blaze Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. CAFE: Self-Improving Search Agents Need Co-Evolving Feedback. arXiv preprint arXiv:2608.24794, 2026b. URL https://arxiv.org/abs/2608.24794. Junteng Liu, Yunji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, et al. Webexplorer: Explore and evolve for training long-horizon web agents. arXiv preprint arXiv:2509.06501, 2025b. Hongliang Lu, Yuhang Wen, Pengyu Cheng, Ruijin Ding, Jiaqi Guo, Haotian Xu, Chutian Wang, Haonan Chen, Xiaoxi Jiang, and Guanjun Jiang. Search self-play: Pushing the frontier of agent capability without supervision. In The Fourteenth International Conference on Learning Representations, 2026. Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting in retrieval- augmented large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, p. 5303ā5315, 2023. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 9802ā9822, 2023. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332, 2021. OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben DāSa, Arthur Petron, Henrique P d O Pinto, et al. Asymmetric self-play for automatic goal discovery in robotic manipulation. arXiv preprint arXiv:2101.04882, 2021. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730ā27744, 2022. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Neural Information Processing Systems, 37:116617ā116637, 2024. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 5687ā5711, 2023. Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URLhttps://qwen. ai/blog?id=qwen3.5. 13 Preprint Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728ā53741, 2023. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Man- ning. RAPTOR: Recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations, 2024. Timo Schick, Jane Dwivedi-Yu, Roberto Dess ` ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539ā68551, 2023. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models. In The Twelfth International Conference on Learning Representations, 2024. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Richard James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. REPLUG: Retrieval-augmented black-box language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 8371ā 8384, 2024. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025. Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588, 2025. Wentao Tan, Qiong Cao, Jiaqi Wang, and Nan Duan. SearchMaster: Grounded and Regulated Self-Play for Search Agents. arXiv preprint arXiv:2608.01822, 2026. URLhttps://arxiv. org/abs/2608.01822. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539ā554, 2022. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 10014ā10037, 2023. Shaobo Wang, Zhengbo Jiao, Zifan Zhang, Yilang Peng, Xu Ze, Boyu Yang, Wei Wang, Hu Wei, and Linfeng Zhang. Socratic-zero: Bootstrapping reasoning via data-free agent co-evolution. arXiv preprint arXiv:2509.24726, 2025a. Ziliang Wang, Xuhui Zheng, Kang An, Cijun Ouyang, Jialu Cai, Yuhang Wang, and Yichao Wu. Stepsearch: Igniting llms search ability via step-wise proximal policy optimization. arXiv preprint arXiv:2505.15107, 2025b. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. In The Thirteenth International Conference on Learning Representations, 2025. Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884, 2024. 14 Preprint Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, p. 2369ā2380, 2018. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V Le, Qijun Tan, and Yuan Liu. Scalable reinforcement post-training beyond static human prompts: Evolving alignment via asymmetric self-play. arXiv preprint arXiv:2411.00062, 2024. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Repre- sentations, 2024. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems, 2025. Wenhao Yu, Hongming Zhang, Xiaoman Pan, Peixin Cao, Kaixin Ma, Jian Li, Hongwei Wang, and Dong Yu. Chain-of-note: Enhancing robustness in retrieval-augmented language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, p. 14672ā14685, 2024. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024. Zhenrui Yue, Kartikeya Upasani, Xianjun Yang, Suyu Ge, Shaoliang Nie, Yuning Mao, Zhe Liu, and Dong Wang. Dr. zero: Self-evolving search agents without training data. In Third Conference on Language Modeling (COLM), 2026. arXiv:2601.07055. Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mcts*: Llm self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735ā64772, 2024. Ding-Chu Zhang, Yida Zhao, Jialong Wu, Liwen Zhang, Baixuan Li, Wenbiao Yin, Yong Jiang, Yu-Feng Li, Kewei Tu, Pengjun Xie, and Fei Huang. EvolveSearch: An iterative self-evolving search agent. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 13123ā13136, 2025a. Rui Zhang, Oksana A. Chkrebtii, and Dongbin Xiu. Likelihood-free posterior density learning for uncertainty quantification in inference problems. arXiv preprint arXiv:2508.00167, 2025b. Rui Zhang, Oksana A. Chkrebtii, and Dongbin Xiu. Dimension-reduced reconstruction map learning for parameter estimation in likelihood-free inference problems. Journal of Machine Learning for Modeling and Computing, 6(4), 2025c. doi: 10.1615/JMachLearnModelComput.2025060234. Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, et al. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506.18959, 2025d. Andrew Zhao, Yiran Wu, Yang Yue, Tong Wu, Quentin Xu, Yang Yue, Matthieu Lin, Shenzhi Wang, Qingyun Wu, Zilong Zheng, and Gao Huang. Absolute zero: Reinforced self-play reasoning with zero data. In Advances in Neural Information Processing Systems, 2025. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595ā46623, 2023. 15 Preprint Table 2: Training configuration shared across treatments. ItemConfiguration BackbonesQwen3.5-4B and Qwen3.5-9B ScheduleThree rounds; 18 proposer and 25 main-solver updates per round Solver data1,600 admitted questions per round; five responses per question Proposer update64 prompts per update; one selected trajectory per prompt SamplingTemperature 0.8 and top-p 0.95 for training trajectories MSVThree source-aware and three source-blind samples per proposal CrossFitTwo source folds; 25 updates per round for each auxiliary solver (50 in total) Tool budgetAt most five assistant actions and 512 new tokens per action Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. DeepResearcher: Scaling deep research via reinforcement learning in real-world environments. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 414ā431, 2025. AIMPLEMENTATION DETAILS Optimization. The proposer and solver are updated with a sequence-normalized policy-gradient objective. For usable trajectoriesE and scored assistant tokensT e , L(Īø) =ā 1 |E| X eāE b A e 1 |T e | X tāT e logĻ Īø (y e,t | y e,<t ,q e ), b A e = r e ā Ī¼ g(e) Ļ g(e) + 10 ā6 .(1) Advantages are normalized within proposer task buckets or within the five solver responses to one question. Prompt and tool tokens are masked, as is the proposerās terminal answer; solver answer tokens remain trainable. Training configuration. All treatments use the same backbone, proposer schedule, main-solver schedule, sampling configuration, and tool budget. The main solver trains on all admitted questions. InCrossFit, two auxiliary solvers train only on their assigned source folds and are used solely to produce proposer feedback on the complementary fold. Compute overhead.MSVadds six labeler generations per proposal, andCrossFitadds 50 auxiliary solver updates per round in the main experiments (25 per fold) without changing the main solverās training data or update count. Table 3 reports the resulting resource cost. Relative to Dr. Zero, MSVraises the reserved budget by about 90% at both scales (379 to 719 H200-hours on Qwen3.5-4B and 476 to 903 on Qwen3.5-9B) and multiplies judge requests almost fivefold, whereasCrossFit adds 72% and 79%. Halving its auxiliary budget to 25 updates per round lowers this to 36% and 40% with nearly the same replay false-agreement mass (0.005 versus 0.004 on Qwen3.5-4B and 0.002 versus 0.001 on Qwen3.5-9B; Figure 7). Combining both interventions costs 2.6 and 2.7 times the Dr. Zero budget. Because reserved hours include waiting time, these ratios compare budgets rather than accelerator utilization. BEVALUATION AND AUDIT DETAILS Downstream evaluation. We extract the terminal answer, normalize case, punctuation, English articles, and whitespace, and score against the best matching reference alias. Cover-EM is one when a nonempty normalized reference answer appears in the normalized prediction. Every checkpoint is evaluated on the same 1,325-question manifest with one greedy search trajectory, an identical tool budget, and identical answer extraction. For benchmarkdwithn d questions and scoresz di , we report Micro = P d P n d i=1 z di 1325 ,Macro = 1 7 7 X d=1 1 n d n d X i=1 z di .(2) 16 Preprint Table 3: Resource cost per training run. H200-hours are reserved budgets: each run holds eight H200 GPUs for its end-to-end duration, including waiting time, so they exceed the accelerator time actually used; the GPU usage of the external audit service is unknown and not included. Token (millions) and judge-request (thousands) counts are per single-scale run, not summed over the two scales, and exclude training-replay tokens. Hours are the end-to-end wall-clock duration of the full pipeline. ā25 totalā and ā25 per foldā denote the auxiliary-update budget per round. H200-hoursTokens (M)JudgeHours Treatment4B9BInputOutputreq. (k)4B9B Dr. Zero3794766968360.44760 MSV7199031,811195296.990113 CrossFit (25 total)51566588910787.16483 CrossFit (25 per fold)6508541,081131113.881107 MSV+CrossFit (25 per fold)9901,2812,196243350.3124160 Table 4: Round-3 audit statistics for every treatment: means of the 43 round-3 step rates.J/Eis audit coverage;T P ,T S ,A,F, andLare defined in Section 2. Figure 6 plots the corresponding changes relative to Dr. Zero, computed from unrounded means; they can therefore differ by 0.1 percentage point from differences of the rounded entries here. J/ET P T S AFL Qwen3.5-4B Dr. Zero0.8590.7470.6690.7100.0610.020 MSV0.8600.7470.7080.7450.0570.020 CrossFit0.8600.8190.6860.6790.0300.038 MSV + CrossFit0.8580.8430.7270.7080.0200.038 Qwen3.5-9B Dr. Zero0.8590.7370.6890.7520.0880.025 MSV0.8530.7700.7270.7880.0720.011 CrossFit0.8570.8510.7120.7090.0370.041 MSV + CrossFit0.8610.8780.7540.7320.0170.039 Independent audit. Many generated questions require reconstructing multi-hop evidence across long or specialized source documents, making exhaustive human adjudication at every training step impractical. At every audited step, we therefore save the source document, adopted pseudo-label, and five solver responses used for proposer reward.gpt-6-astra/highconstructs an evidence- backed reference from the source and judges the saved label and responses against that reference. Unsupported cases remain unresolved and are included in coverage accounting. The auditor is post-hoc: it never changes admission, model updates, or proposer reward. Table 4 lists the round-3 mean of each audit statistic for every treatment. Round-wise results. Table 5 lists Cover-EM for every round-end main solver in the layout of Table 1, and Table 6 gives the corresponding micro averages, which weight all 1,325 questions equally. Base obtains micro averages of 0.382 on Qwen3.5-4B and 0.404 on Qwen3.5-9B. Micro and macro averages order the treatments identically in every round. CBROADER LITERATURE MAP This map separates complementary research questions rather than treating every cited system as a direct experimental baseline. Only methods with accessible implementations and matched protocols can support comparative performance claims. Retrieval representations and evidence use.Dense retrieval, fusion-in-decoder, retrieval-enhanced language modeling, and few-shot retrieval pretraining study how evidence is retrieved and represented (Karpukhin et al., 2020; Izacard & Grave, 2021; Borgeaud et al., 2022; Izacard et al., 2023). Query 17 Preprint Table 5: Cover-EM of every round-end main solver; we mark the best performance within each scale in bold. Each cross-fitted treatment shares round 1 with its coupled counterpart. NQTriviaQAPopQAHotpotQA2WikiMQAMuSiQueBamboogleAverage Qwen3.5-4B Base0.3800.6550.3150.3500.4650.1050.4160.384 Dr. Zero Round 10.3800.6600.3250.3500.4750.1250.4240.391 Dr. Zero Round 20.3950.6650.3200.3600.4700.1250.4400.396 Dr. Zero Round 30.3900.6650.3250.3650.4850.1200.4480.400 MSV Round 10.3850.6750.3300.3550.4800.1100.4320.395 MSV Round 20.3900.6700.3350.3700.4900.1200.4320.401 MSV Round 30.4000.6700.3400.3750.4800.1350.4480.407 CrossFit Round 10.3800.6600.3250.3500.4750.1250.4240.391 CrossFit Round 20.4150.6850.3700.4200.5350.1650.4800.439 CrossFit Round 30.4550.7300.4150.4600.5750.2450.5360.488 MSV + CrossFit Round 10.3850.6750.3300.3550.4800.1100.4320.395 MSV + CrossFit Round 20.4400.6800.3700.4200.5150.2050.4960.447 MSV + CrossFit Round 30.4700.7300.4100.4750.5750.2500.5280.491 Qwen3.5-9B Base0.5050.7100.3750.3550.2950.1250.4960.409 Dr. Zero Round 10.5000.7250.3850.3650.3100.1400.4960.417 Dr. Zero Round 20.5200.7250.3850.3700.3100.1350.5200.424 Dr. Zero Round 30.5150.7300.4000.3700.3200.1500.5120.428 MSV Round 10.5100.7250.3900.3650.3050.1500.5040.421 MSV Round 20.5250.7100.3950.3750.3200.1500.5280.429 MSV Round 30.5250.7200.4000.3900.3250.1550.5360.436 CrossFit Round 10.5000.7250.3850.3650.3100.1400.4960.417 CrossFit Round 20.5500.7500.4300.4300.3550.2000.5520.467 CrossFit Round 30.5800.7650.4550.4900.4250.2550.6160.512 MSV + CrossFit Round 10.5100.7250.3900.3650.3050.1500.5040.421 MSV + CrossFit Round 20.5500.7450.4350.4250.3650.2200.5760.474 MSV + CrossFit Round 30.5700.7800.4700.4850.4150.2800.6080.515 Table 6: Micro-averaged Cover-EM of the round-end main solvers; we mark the best performance in bold. Qwen3.5-4BQwen3.5-9B Round 1Round 2Round 3Round 1Round 2Round 3 Dr. Zero0.3890.3940.3970.4130.4180.423 MSV0.3930.3990.4050.4170.4230.430 CrossFit0.3890.4360.4850.4130.4620.506 MSV + CrossFit0.3930.4440.4890.4170.4680.510 rewriting and active retrieval change when and how evidence is requested (Ma et al., 2023; Jiang et al., 2023). Black-box retrieval augmentation, hierarchical retrieval, and chain-of-note processing provide other evidence interfaces (Shi et al., 2024; Sarthi et al., 2024; Yu et al., 2024). Robustness to irrelevant context, corrective retrieval, and retrieval-aware tuning address evidence quality or utilization (Yoran et al., 2024; Yan et al., 2024; Lin et al., 2024). These directions motivate holding the retrieval backend fixed: a change in evidence access must not be mistaken for an effect of source-excluded feedback. Search-agent optimization.Beyond Search-R1 and R1-Searcher, evolving search, efficient search training, simulated search, deep-research agents, and trained web agents illustrate the expanding space of agent learning (Zhang et al., 2025a; Jiang et al., 2025; Sun et al., 2025; Zheng et al., 2025; Zhang et al., 2025d). Step-level search training and systems for difficult web exploration further motivate recording trajectory budgets and tool usage (Wang et al., 2025b; Li et al., 2025a; Liu et al., 2025b). Toolformer studies learning tool use, while more recent open research and search-agent frameworks expand the surrounding system design space (Schick et al., 2023; Li et al., 2026; Chu et al., 2026; Liang et al., 2026). They are contextual references, not claims of matched evaluation in this manuscript. Self-generated learning and reinforcement learning.Self-play, multi-agent games, self-training, and iterative reasoning optimization offer different sources of automatically generated supervision 18 Preprint (Ye et al., 2024; Liu et al., 2026a; Zhang et al., 2024; Pang et al., 2024). Outside language modeling, likelihood-free inference also learns from generated supervision, fitting reconstruction maps or posterior densities to simulator samples whose parameters are known by construction (Zhang et al., 2025b;c); the generated pairs studied here differ in that their labels are themselves model predictions, so label error can propagate into the evaluator. Preference learning and large-scale reasoning RL establish additional optimization choices (Ouyang et al., 2022; Rafailov et al., 2023; Guo et al., 2025; Yu et al., 2025). Our inspected sequence-level policy-gradient implementation is specified directly in our implementation audit; citing these methods does not imply that their algorithms, training data, or reported capabilities are reproduced. The distinctive variable studied here is the ancestry of the policy supplying proposal feedback, not a new generic policy-gradient estimator. DDISCUSSION AND LIMITATIONS Cross-fitting changes who supplies feedback, not what makes an answer true. Its source exclusion is valuable only if checkpoint ancestry and data routing enforce it. Shared pretraining, overlapping web evidence, semantically related sources, and an adaptive proposer can still induce correlated mistakes. A lower false-agreement mass on accepted questions can also result from rejecting difficult tasks rather than improving learning. Coverage, task difficulty, fixed-probe performance, and downstream capability must therefore accompany that number. Our results establish empirical mitigation across two model scales, but do not yet establish lower end-to-end cost or robustness to connected sources. ECOMPLETE PER-STEP AUDIT TRAJECTORIES Figures 8 and 9 retain all 129 scheduled steps, all four treatments, and all six metrics from the data underlying Figure 5. Values are plotted directly from the existing per-step table, in percent, without smoothing or interpolation. Separate axes for false agreement and lost credit keep their distinct magnitudes visible; coverage is the audited fractionJ/Eand is displayed separately from correctness. The round summaries in Figure 5 are arithmetic means of the 43 displayed step rates in each round, not additional runs or uncertainty estimates. 19 Preprint 40 60 80 100 Truth / agreement (%) Dr. ZeroMSVCrossFitMSV + CrossFit 0 5 10 15 False agreement (%) 0 10 20 Lost credit (%) 04386129 Step 0 100 Coverage 04386129 Step 04386129 Step 04386129 Step Qwen3.5-4B | Complete audit trajectories Every scheduled step Ā· shared axes across treatments Ā· no smoothing Gray bands: proposer updates Ā· Short dashes: round boundaries Ā· Long dash: step 62 T P adopted-label truthT S solver truth A agreement Figure 8: Complete Qwen3.5-4B audit trajectories. Columns identify treatments. Rows show adopted-label truthT P , solver truthT S , and agreementA; false-agreement massF; lost-credit mass L; and coverageJ/E. Every metric is expressed in percent. Shared limits support comparison across treatments and model scales. Gray bands mark proposer phases, short dashes mark round boundaries, and the long dash marks step 62. 20 Preprint 40 60 80 100 Truth / agreement (%) Dr. ZeroMSVCrossFitMSV + CrossFit 0 5 10 15 False agreement (%) 0 10 20 Lost credit (%) 04386129 Step 0 100 Coverage 04386129 Step 04386129 Step 04386129 Step Qwen3.5-9B | Complete audit trajectories Every scheduled step Ā· shared axes across treatments Ā· no smoothing Gray bands: proposer updates Ā· Short dashes: round boundaries Ā· Long dash: step 62 T P adopted-label truthT S solver truth A agreement Figure 9: Complete Qwen3.5-9B audit trajectories. Layout, metric colors, units, and axis limits match Figure 8. Every scheduled step is retained. The separate coverage strip prevents audit coverage from obscuring the correctness curves. 21