Paper deep dive
PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems
Yifan Wang, Shipeng Zhu, Fei Xiong, Yuqin Yang, Yonghui Huang, Kunyao Wu, Yue Wang, Weichao Meng, Yu Gong
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AutoResearch improves systems through iterative experimentation: agents propose candidate modifications, evaluate them, and use the results to guide subsequent exploration. Applying this paradigm to industrial search presents two challenges. (1) Common AutoResearch approaches follow a keep-if-better rule, retaining the highest-scoring candidate for subsequent experiments. Under non-stationary traffic, transient gains may be mistaken for persistent improvements, impairing reliable accumulation of search knowledge. (2) Candidate modifications can be evaluated at multiple fidelity levels, from low-cost proxies to online validation, differing in cost, objective alignment, and statistical reliability. Existing methods rely on individual signals or task-specific procedures, lacking a unified basis for using evidence across levels to guide search. We introduce Progressive Evidence-Based AutoResearch (PEAR) with two complementary components. Evidence-driven AutoResearch maintains an independent, hypothesis-guided research state for each strategy task within a predefined objective and intervention scope. Each state evolves through a Plan-Execute-Evaluate-Update transition that links experimentation to context-aware evidence interpretation and hypothesis revision. Confidence-Gated Verifier Ladder organizes evaluation into four levels of increasing fidelity: Offline Replay, Shadow-Traffic Evaluation, Rapid Online Evaluation, and Decision-Grade Online Evaluation. A unified confidence-based gate promotes candidates only when evidence supports a statistically significant positive effect, enabling broad low-cost exploration while reserving costly online experiments for promoted candidates. In a real-world industrial search system, strategies optimized with PEAR significantly increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines in two A/B experiments.
Tags
Links
- Source: https://arxiv.org/abs/2609.35031v1
- Canonical: https://arxiv.org/abs/2609.35031v1
Trouble viewing inline? Open PDF directly →
Full Text
68,430 characters extracted from source content.
Expand or collapse full text
PEAR: Progressive Evidence-Based AutoResearch for Industrial Search Systems Global E-Commerce Agentic Search Team Email: yifanwang993w@gmail.com Email: gy910210@gmail.com Abstract AutoResearch improves systems through iterative experimentation: agents propose candidate modifications, evaluate them, and use the results to guide subsequent exploration. Applying this paradigm to industrial search presents two challenges. (1) Common AutoResearch approaches follow a keep-if-better rule, retaining the highest-scoring candidate for subsequent experiments. Under non-stationary traffic, transient gains may be mistaken for persistent improvements, impairing reliable accumulation of search knowledge. (2) Candidate modifications can be evaluated at multiple fidelity levels, from low-cost proxies to online validation, differing in cost, objective alignment, and statistical reliability. Existing methods rely on individual signals or task-specific procedures, lacking a unified basis for using evidence across levels to guide search. We introduce Progressive Evidence-Based AutoResearch (PEAR) with two complementary components. Evidence-driven AutoResearch maintains an independent, hypothesis-guided research state for each strategy task within a predefined objective and intervention scope. Each state evolves through a Plan–Execute–Evaluate–Update transition that links experimentation to context-aware evidence interpretation and hypothesis revision. Confidence-Gated Verifier Ladder organizes evaluation into four levels of increasing fidelity: Offline Replay, Shadow-Traffic Evaluation, Rapid Online Evaluation, and Decision-Grade Online Evaluation. A unified confidence-based gate promotes candidates only when evidence supports a statistically significant positive effect, enabling broad low-cost exploration while reserving costly online experiments for promoted candidates. In a real-world industrial search system, strategies optimized with PEAR significantly increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines in two A/B experiments. 1 Introduction Industrial search systems integrate query understanding, candidate retrieval, and result ranking to determine which results are presented to users. Recent work has substantially advanced individual components through instruction-aware retrieval and LLM-based reranking [27, 14, 3, 26]. In production, however, these components operate within a multi-stage decision process in which models, ranking strategies, and business constraints interact. Shifts in user demand, content supply, and business objectives require these strategies to be continually revisited, while dependencies across stages make the effects of candidate modifications context-dependent and difficult to predict offline. Engineers therefore improve the system through iterative experimentation: they formulate hypotheses from observed system behavior, implement candidate strategies, evaluate them under real traffic, and use the results to guide subsequent iterations. Recent advances in large language model (LLM) agents have demonstrated multi-step reasoning, tool use, and the ability to act in external environments [25, 20, 23, 24, 17, 18], creating an opportunity to automate portions of this workflow. AutoResearch turns these capabilities into an iterative experimentation loop in which agents propose and implement candidate modifications, evaluate their outcomes, and use the resulting evidence to guide subsequent proposals [5, 15, 19, 1]. Recent work has applied this paradigm to search and recommendation systems, exploring interventions that range from ranking parameters to model architectures and reward functions [22, 4], and evaluating them using offline proxy metrics, model diagnostics, simulated user responses, and online A/B test outcomes [7, 9]. Collectively, these studies demonstrate the potential of agent-driven strategy optimization, while making experimental evidence the primary basis for candidate selection and subsequent exploration. Reliably using evidence from both offline and online evaluations in AutoResearch for industrial search presents two challenges arising from non-stationary outcomes and heterogeneous evaluation procedures. (1) How can an agent accumulate reliable experimental knowledge when candidate effects are context-dependent? Many AutoResearch systems apply a keep-if-better rule, retaining the highest-scoring candidate as the reference for subsequent experiments [19, 15]. This rule implicitly assumes that evaluation scores remain comparable across rounds. In industrial search, traffic composition and system state evolve over time; consequently, experiments conducted under different conditions may estimate candidate effects that are not directly comparable. A transient or context-specific gain may therefore be recorded as a persistent improvement, weakening hypothesis assessment and misdirecting subsequent exploration. (2) How should evidence from different evaluation procedures govern candidate promotion? Low-cost proxy evaluations support broad exploration, but their improvements may not translate into online gains. Online experiments provide evidence that is more directly aligned with deployment objectives, but sufficiently evaluating every candidate online is costly and risky [12, 6, 11, 16, 8]. Existing approaches use individual evaluation signals or task-specific evaluation procedures [22, 7, 4, 9], leaving unclear how evidence from different procedures should jointly inform promotion decisions. The resulting challenge is to accumulate reliable experimental knowledge under changing contexts and make sound candidate-promotion decisions from heterogeneous evaluation evidence. To address these challenges, we introduce Progressive Evidence-Based AutoResearch (PEAR), a framework for industrial search systems that couples persistent strategy states with progressive candidate evaluation through two complementary components. Evidence-driven AutoResearch maintains an independent research state for each strategy task, comprising the experimental context, current hypotheses, candidate intervention space, and accumulated evidence. The state evolves through a Plan–Execute–Evaluate–Update transition augmented with Bayesian-optimization-inspired reasoning. This reasoning balances refinement of evidence-supported regions with exploration of underexplored regions, while experimental results are interpreted within their contexts to support hypothesis revision and knowledge accumulation across experiments. Confidence-Gated Verifier Ladder organizes candidate evaluation into four progressively higher-fidelity levels through a unified confidence-based promotion gate: Offline Replay, Shadow-Traffic Evaluation, Rapid Online Evaluation, and Decision-Grade Online Evaluation. Evaluation results and promotion decisions from each level update the corresponding research state and inform subsequent search. Lower-cost evaluations support broad exploration, while costly online experiments are reserved for promoted strategies. In a real-world industrial search system, strategies optimized with PEAR significantly increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines in two A/B experiments at the Decision-Grade Online Evaluation stage. 2 Contributions 1. We introduce Evidence-driven AutoResearch to support reliable knowledge accumulation for industrial search optimization under non-stationary traffic. For each strategy task, it maintains an independent research state that links hypotheses and candidate interventions to their evaluation outcomes and experimental contexts. The state evolves through a Plan–Execute–Evaluate–Update transition, in which contextual evidence informs revisions to both the hypotheses and the candidate intervention space across experimental rounds. 2. We develop Confidence-Gated Verifier Ladder, which organizes candidate evaluation into four levels of progressively greater fidelity and cost, from Offline Replay to Decision-Grade Online Evaluation. The ladder preserves each level’s evaluation signal in a unified evidence record and applies a common confidence-based promotion rule, allowing low-cost evaluations to support broad exploration while reserving costly online experiments for statistically supported candidates. The resulting evidence and promotion decisions are returned to the research state to guide subsequent search. 3. We deploy and evaluate PEAR in a real-world industrial search system using three strategy tasks. The evaluation documents evidence-driven hypothesis revision and, for two promoted candidates, directionally consistent positive effects from Shadow-Traffic Evaluation to Decision-Grade Online Evaluation. In two Decision-Grade Online A/B experiments, strategies optimized with PEAR significantly increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines. 3 Related Work 3.1 AutoResearch AutoResearch organizes candidate generation, experimentation, and evaluation into iterative research workflows [21, 5, 15]. A common update mechanism is the keep-if-better rule, which retains candidates that improve the observed experimental score as references for subsequent search [19, 15]. The interpretation of these improvements depends on the comparability of evaluation outcomes across experiments. Recent systems preserve experimental history to inform subsequent proposals. Meta-Harness optimizes harness code using the code, execution traces, and scores of prior candidates [10]. AiScientist maintains a persistent workspace that retains experimental artifacts across implementation, experimentation, and diagnosis [2]. AutoResearchClaw combines multi-agent deliberation, execution recovery, and cross-run experience accumulation to support iterative hypothesis generation and experimentation [13]. These approaches provide mechanisms for retaining and reusing information across experiments. The keep-if-better rule uses observed candidate scores to guide subsequent search. In industrial search optimization, non-stationary traffic complicates the interpretation of these scores across experimental contexts. Reliable search knowledge accumulation requires distinguishing persistent improvements from transient gains over multiple experiments. 3.2 AutoResearch in Search and Recommendation Recent studies apply AutoResearch to search and recommendation through task-specific candidate generation and evaluation procedures. Self-Evolving Recommendation System combines a fast offline loop for generating and filtering model candidates with a slower online loop for validating delayed business outcomes [22]. Self-EvolveRec combines semantic feedback from a user simulator with model-diagnostic signals to guide candidate modifications across the recommendation pipeline [7]. These studies use proxy evaluation and structured feedback to guide subsequent candidate generation. Other systems incorporate online experimental feedback into iterative optimization. Sortify estimates the relationship between offline signals and online outcomes and maintains cross-round memory to inform subsequent parameter search [4]. AgentX connects candidate generation, implementation, online experimental evaluation, and experience feedback through specialized agents [9]. These systems demonstrate how online outcomes can inform subsequent search in deployed search and recommendation systems. Existing approaches organize candidate evaluation around individual signals or task-specific procedures. Proxy and online evaluation results differ in acquisition cost, objective alignment, and statistical reliability. A unified basis for using these results to guide candidate selection and subsequent search remains an open problem. 4 Problem Setup AutoResearch for industrial search optimization iteratively refines search strategies through candidate generation, experimentation, and evaluation. Each strategy task k specifies a predefined objective and intervention scope. A candidate strategy c specifies a testable modification to the search system, such as a parameter configuration or a change to the ranking mechanism. Multiple strategy tasks may proceed concurrently within their respective intervention scopes. At experimental round t, the agent generates a candidate set k,tA_k,t for strategy task k. An evaluation procedure assigns scores to the candidates according to the optimization objective. The agent uses the resulting scores to refine subsequent proposals. Repeated candidate generation, evaluation, and proposal revision form an iterative optimization process. In AutoResearch for industrial search systems, evaluations range from low-cost proxy evaluation to online validation. These evaluation sources differ in acquisition cost, objective alignment, and statistical reliability. Proxy evaluation supports broad candidate exploration at low cost. Online validation measures candidate effects through real-user outcomes and requires additional traffic and observation time. Let (c)S(c) denote the set of evaluation stages applied to candidate c, and let ℰℓ(c)E_ (c) denote the evidence acquired at stage ℓ . The collected evidence is ℰ(c)=ℰℓ(c)∣ℓ∈(c).E(c)= \E_ (c) (c) \. (1) For strategy task k, ℰk,t(ℓ)E_k,t^( ) denotes the set of stage-ℓ evidence records acquired in round t. Each result reflects its evaluation protocol and traffic context. Under non-stationary traffic, results from different rounds must be interpreted within their corresponding experimental contexts. The objective is to optimize industrial search strategies through iterative experimentation guided by heterogeneous evaluation results. Evaluation results provide the basis for hypothesis revision, candidate selection, and subsequent search. This process requires reliable search knowledge accumulation across experimental rounds and sufficient online validation to support deployment decisions. 5 Progressive Evidence-Based AutoResearch (PEAR) Figure 1: Overview of PEAR. Evidence-driven AutoResearch maintains parallel strategy-level research states, while the Confidence-Gated Verifier Ladder progressively evaluates candidates, governs their promotion using evidence of increasing fidelity, and returns evaluation results to the corresponding research states. As illustrated in Figure 1, PEAR couples persistent strategy states with progressive candidate evaluation through two complementary components: Evidence-driven AutoResearch and Confidence-Gated Verifier Ladder. Evidence-driven AutoResearch maintains an independent, hypothesis-guided research state for each strategy task. Each state preserves the hypotheses, candidate history, and experimental contexts associated with that task. A Plan–Execute–Evaluate–Update transition connects candidate generation and experimentation with evidence interpretation and hypothesis revision. Confidence-Gated Verifier Ladder organizes candidate evaluation into four progressively higher-fidelity levels: Offline Replay, Shadow-Traffic Evaluation, Rapid Online Evaluation, and Decision-Grade Online Evaluation. Unified evidence records and a confidence-based promotion gate connect these evaluation levels. Evaluation results and promotion decisions update the corresponding research states and inform subsequent search. Lower-cost evaluations support broad exploration, and costly online experiments are reserved for promoted candidates. 5.1 Evidence-driven AutoResearch Evidence-driven AutoResearch associates optimization hypotheses with the evaluation evidence acquired for each strategy task. A hypothesis describes the expected effect of a candidate modification under specified conditions. Its associated evidence records the candidate, evaluation outcome, and experimental context. This association provides the basis for interpreting non-stationary evaluation results and revising hypotheses across experiments. For strategy task k at experimental round t, the research state is ℋk,t=(k,ℳk,t,Ωk,t,k,t),H_k,t= (C_k,M_k,t, _k,t,D_k,t ), (2) where kC_k specifies the search-system implementation, optimization objective, intervention scope, and experimental protocol. ℳk,tM_k,t contains the current hypotheses, Ωk,t _k,t denotes the candidate intervention space within the specified scope, and k,tD_k,t contains the accumulated evidence and its associations with the hypotheses. Independent states preserve the experimental context of each strategy task and support concurrent exploration across strategy tasks. The research state evolves through a Plan–Execute–Evaluate–Update transition: ℋk,t→Plank,t→ExecuteΔk,t→Evaluateℛk,t→Updateℋk,t+1.H_k,t PlanA_k,t Execute _k,t EvaluateR_k,t UpdateH_k,t+1. (3) Here, k,tA_k,t is the candidate set, Δk,t _k,t is the newly acquired evidence, and ℛk,tR_k,t records the evidence-based assessment of the current hypotheses. Plan: Hypothesis-Guided Candidate Generation. The agent formulates testable hypotheses from the mechanisms of the industrial search system, the optimization objective, and accumulated experimental evidence. These hypotheses describe how candidate modifications are expected to affect the objective under the corresponding experimental conditions. Each candidate in k,tA_k,t is associated with a hypothesis and an expected observation. Candidate generation includes interventions that examine individual effects, test interactions, or replicate previous observations. Repeated refinement around previously promising candidates can concentrate exploration within a narrow region. To broaden candidate generation, the agent incorporates Bayesian-optimization-inspired reasoning: k,t=(k,ℳk,t,k,t,Ωk,t,BO),k,t⊆Ωk,t,A_k,t=G\! (C_k,M_k,t,D_k,t, _k,t;K_BO ), _k,t _k,t, (4) where G denotes the agent’s candidate-generation procedure and BOK_BO denotes Bayesian-optimization-inspired reasoning principles. Conditioned on the search-system context, current hypotheses, and accumulated evidence, this procedure combines refinement of evidence-supported regions with exploration of underexplored regions. These principles guide qualitative reasoning about observed effects and uncertainty rather than specifying a fitted numerical optimization model. Execute: Experiment Execution and Evidence Collection. The agent invokes stage-specific experimentation tools to apply candidate modifications within the corresponding evaluation environments and initiate experiments under Confidence-Gated Verifier Ladder. As candidates progress through the evaluation stages, the agent retrieves the evaluation results and promotion decisions produced by the corresponding verifiers. These outputs form the newly acquired evidence Δk,t _k,t, with each record linked to its candidate, hypothesis, and experimental context. Evaluate: Contextual Hypothesis Assessment. The agent assesses each hypothesis using its associated evidence in k,t∪Δk,tD_k,t∪ _k,t. This assessment considers the candidate modifications, evaluation stages, and experimental contexts underlying the observed effects. Consistent observations provide support for a hypothesis. Conflicting observations motivate examination of its assumptions and applicability conditions. Inconclusive observations remain unresolved evidence for further experimentation. The assessment record ℛk,tR_k,t preserves these distinctions for hypothesis revision. Update: Evidence-Based Hypothesis Revision. The assessment record updates the hypotheses and the candidate intervention space: (ℳk,t+1,Ωk,t+1) (M_k,t+1, _k,t+1) =(ℳk,t,Ωk,t,ℛk,t), =U(M_k,t, _k,t,R_k,t), (5) k,t+1 _k,t+1 =k,t∪Δk,t, =D_k,t∪ _k,t, where U denotes evidence-based revision of the hypotheses and candidate intervention space. Supported hypotheses guide further refinement and replication. Conflicting evidence motivates revised hypotheses and alternative candidate modifications within the specified intervention scope. Evaluation results and promotion decisions remain available in the updated state, allowing subsequent proposals to build on accumulated evidence and its experimental context. 5.2 Confidence-Gated Verifier Ladder Confidence-Gated Verifier Ladder organizes heterogeneous candidate evaluations into four levels: Offline Replay (L1), Shadow-Traffic Evaluation (L2), Rapid Online Evaluation (L3), and Decision-Grade Online Evaluation (L4). All four levels are aligned with the same optimization objective but differ in signal source, evaluation fidelity, acquisition cost, traffic exposure, and observation horizon. L1 and L2 use a model-based proxy score that estimates this objective from ranking outputs, and L3 and L4 use the objective directly observed from online user outcomes. Each strategy task instantiates an ordered sequence of active levels according to data availability and validation requirements. A unified confidence-based promotion rule connects consecutive levels in this task-specific sequence. Each active level produces evaluation results under its corresponding experimental protocol. Evaluation results and promotion decisions provide evidence for hypothesis assessment and subsequent search. 5.2.1 Unified Promotion Policy The promotion policy uses the estimated candidate effect and its confidence interval at each evaluation stage. For candidate c at stage ℓ , the verifier produces ℰℓ(c)=(c,bℓ,mℓ,nℓc,nℓbℓ,Δ^ℓ(c),CIℓ(c),vℓ(c)),E_ (c)= (c,b_ ,m_ ,n_ ^c,n_ ^b_ , _ (c),CI_ (c),v_ (c) ), (6) where ℰℓ(c)E_ (c) is the evidence record produced for candidate c at stage ℓ , which packages the comparison and its statistical summary into a single object. Within it, bℓb_ is the stage-specific baseline and mℓm_ is the evaluation signal. The quantities nℓcn_ ^c and nℓbℓn_ ^b_ denote the numbers of valid evaluation units for the candidate and baseline. Δ^ℓ(c) _ (c) denotes the estimated candidate effect, CIℓ(c)CI_ (c) its confidence interval, and vℓ(c)v_ (c) the promotion decision. L1 and L2 use model-based proxy scores. L3 and L4 use observed online outcomes. This record is the unit of evidence accumulated in the research state and interpreted together with its experimental context. Let μ^ℓ(c) μ_ (c) and μ^ℓ(bℓ) μ_ (b_ ) denote the estimated values of the evaluation signal for the candidate and baseline. With the signal oriented so that larger values indicate better outcomes, the estimated effect is Δ^ℓ(c)=μ^ℓ(c)−μ^ℓ(bℓ). _ (c)= μ_ (c)- μ_ (b_ ). (7) The confidence interval is centered on this effect and widened by its standard error, CIℓ(c)=[Δ^ℓ(c)−tνℓ, 1−α/2SEℓ(c),Δ^ℓ(c)+tνℓ, 1−α/2SEℓ(c)],CI_ (c)= [\, _ (c)-t_ _ ,\,1-α/2\,SE_ (c),\;\; _ (c)+t_ _ ,\,1-α/2\,SE_ (c)\, ], (8) where SEℓ(c)SE_ (c) is the standard error of the effect, tνℓ, 1−α/2t_ _ ,\,1-α/2 is the critical value at significance level α, and νℓ _ the associated degrees of freedom. The evidence representation and promotion rule are shared across levels, whereas effect estimation follows the sampling structure of each evaluation protocol. The shared representation retains stage-specific signal definitions and confidence intervals rather than combining heterogeneous results into a single score. The stage-specific effect estimators and confidence computations are detailed in Appendix A. Let CIℓ(c)=[Lℓ(c),Uℓ(c)]CI_ (c)=[L_ (c),U_ (c)] denote the confidence interval at the prescribed confidence level. The promotion decision is vℓ(c)=,Lℓ(c)>0,,Lℓ(c)≤0≤Uℓ(c),,Uℓ(c)<0.v_ (c)= cases PROMOTE,&L_ (c)>0,\\ RETAIN,&L_ (c)≤ 0≤ U_ (c),\\ STOP,&U_ (c)<0. cases (9) PROMOTE indicates statistically significant evidence of a positive effect and advances the candidate to the next evaluation stage. At L4, it indicates that the candidate satisfies the promotion rule for deployment consideration. RETAIN indicates inconclusive evidence and retains the candidate for further evaluation at the current stage. STOP indicates statistically significant evidence of a negative effect and terminates the candidate’s progression through the ladder. The promotion decision progressively narrows the candidate set along the active evaluation sequence. For strategy task k, let ℒk=(ℓk,1,…,ℓk,Jk)L_k=( _k,1,…, _k,J_k) denote its ordered sequence of active levels, where ℓk,j∈1,2,3,4 _k,j∈\1,2,3,4\ and ℓk,j<ℓk,j+1 _k,j< _k,j+1. The candidates entering the first active level are those generated by Evidence-driven AutoResearch, k,t[1]=k,tA_k,t^[1]=A_k,t, and each active level advances only the candidates it promotes: k,t[j+1]=c∈k,t[j]:vℓk,j(c)=,j∈1,…,Jk,A_k,t^[j+1]= \\,c _k,t^[j]\;:\;v_ _k,j(c)= PROMOTE\, \, j∈\1,…,J_k\, (10) which yields the nested selection k,t[1]⊇k,t[2]⊇⋯⊇k,t[Jk+1]A_k,t^[1] _k,t^[2] ·s _k,t^[J_k+1]. A candidate reaches an active level only after being promoted at every preceding level in the task-specific sequence. For deployment consideration, the sequence terminates at Decision-Grade Online Evaluation (L4), and k,t[Jk+1]A_k,t^[J_k+1] contains the candidates promoted at that level. Thus, lower-cost active levels screen candidates before costly online evaluation. Evaluation results and promotion decisions are returned to the corresponding research state together with their experimental contexts. For strategy task k in experimental round t, ℰk,t(ℓ)E_k,t^( ) collects the records of candidates evaluated at stage ℓ . These records support evidence interpretation across experiments while preserving the conditions under which each result was obtained. 5.2.2 Model-Based Proxy Evaluation L1 and L2 evaluate candidate ranking outputs without exposing users to those outputs. A list-level evaluation model provides a proxy score for each candidate. Given request context x and the ranked list Lg(x)L^g(x) produced by strategy g, the proxy evaluation signal is mℓ(g,x)=Rψ(x,Lg(x)),ℓ∈1,2,m_ (g;x)=R_ψ\! (x,L^g(x) ), ∈\1,2\, (11) where RψR_ψ estimates the online objective from the request context and the complete ranked list. The model accounts for the composition and relative positions of multiple content formats. It therefore evaluates the combined ranking output rather than individual content scores in isolation. The resulting proxy scores provide the evaluation signal for estimating candidate effects at L1 and L2. Both stages use the same scoring model but evaluate candidates under different request and execution conditions. Offline Replay uses fixed historical requests. Shadow-Traffic Evaluation uses duplicated real-time requests in the online execution environment. 5.2.3 L1: Offline Replay L1 evaluates candidate modifications on a fixed set of historical requests. The candidate and baseline ranking procedures process the recorded request contexts and produce their respective ranked lists. The evaluation model assigns a proxy score to each list. For strategy g∈c,b1g∈\c,b_1\, the estimated proxy score is μ^1(g)=^x∼hist[m1(g,x)], μ_1(g)= E_x _hist\! [m_1(g;x) ], (12) where histX_hist is the historical request set over which the proxy signal is averaged. Because the candidate and baseline process the same historical requests, L1 estimates the candidate effect from request-level paired differences. The verifier forms the effect Δ^1(c) _1(c) and its confidence interval CI1(c)CI_1(c) from these scores under the Offline Replay protocol. This evidence supports low-cost candidate screening. The promoted subset k,t(2)A_k,t^(2) proceeds to Shadow-Traffic Evaluation. 5.2.4 L2: Shadow-Traffic Evaluation L2 evaluates candidate modifications using duplicated real-time requests in the industrial search system. Candidate ranking procedures execute with current request contexts and online features. Their outputs are used for evaluation without replacing the results shown to users. The per-request proxy signal m2(g,x)m_2(g;x) follows the same model-based definition, with Lg(x)L^g(x) produced in the online execution environment. The scoring model is shared with L1, but its inputs reflect current requests, online features, and the ranking outputs generated under these conditions. For strategy g∈c,b2g∈\c,b_2\, the estimated proxy score is μ^2(g)=^x∼shadowg[m2(g,x)], μ_2(g)= E_x _shadow^g\! [m_2(g;x) ], (13) where shadowgX_shadow^g is the set of requests received by the shadow-traffic group for strategy g. Candidate and baseline statistics are computed from independent shadow-traffic groups that receive requests replicated from the live traffic stream, without request-level pairing across groups. The verifier forms the effect Δ^2(c) _2(c) and its confidence interval CI2(c)CI_2(c) from these scores, which determine candidate promotion under the unified policy. This stage examines candidate performance under current operating conditions while retaining model-based proxy evaluation. The promoted subset k,t(3)A_k,t^(3) proceeds to Rapid Online Evaluation. 5.2.5 L3: Rapid Online Evaluation L3 applies candidate modifications to limited randomized traffic. Users in the candidate group receive results produced by the candidate strategy, and users in the baseline group receive results produced by the baseline strategy. The experiment collects observed user outcomes over a short observation window. These outcomes replace model-based proxy scores as the evaluation signal. Let 3gN_3^g denote the evaluation units observed under strategy g, and let y3(g,u)y_3(g;u) denote the observed objective signal for unit u. For strategy g∈c,b3g∈\c,b_3\, the estimated value is μ^3(g)=^u∼3g[y3(g,u)], μ_3(g)= E_u _3^g\! [y_3(g;u) ], (14) where the expectation is taken over the units randomly assigned to each strategy. Evaluation units and outcome definitions follow the online experimental protocol. The verifier forms the effect Δ^3(c) _3(c) and its confidence interval CI3(c)CI_3(c) under that protocol and applies the unified promotion policy. This stage assesses whether candidates selected through proxy evaluation produce positive effects in real-user outcomes. The promoted subset k,t(4)A_k,t^(4) proceeds to Decision-Grade Online Evaluation. 5.2.6 L4: Decision-Grade Online Evaluation L4 evaluates promoted candidates through online experiments with broader randomized traffic and a longer observation window. Candidate and baseline groups provide observed outcomes aligned with the optimization objective. The expanded experiment covers a wider range of traffic conditions and temporal variation. Using the same notation for observed objective signals, the estimated value is μ^4(g)=^u∼4g[y4(g,u)],g∈c,b4, μ_4(g)= E_u _4^g\! [y_4(g;u) ], g∈\c,b_4\, (15) where 4gN_4^g contains the evaluation units used in Decision-Grade Online Evaluation and y4(g,u)y_4(g;u) is the objective signal measured over its observation window. The resulting effect Δ^4(c) _4(c) and its confidence interval CI4(c)CI_4(c) support the final deployment decision. Candidates satisfying the promotion rule at L4 are eligible for deployment consideration. Evaluation results and final decisions update the corresponding research states and inform subsequent search. 6 Experiments 6.1 Experimental Setup and Research Questions The experiments evaluate PEAR in a real-world industrial search system. Three anonymized strategy tasks provide records of candidate exploration and evaluation. These records include experimental configurations, evaluation results, and hypothesis revisions. A/B results from Decision-Grade Online Evaluation are reported for strategies identified for two of these tasks. The evaluation addresses three research questions: RQ1: Evidence-driven hypothesis revision. How are accumulated evaluation results incorporated into hypothesis revision and subsequent candidate generation? RQ2: Progressive candidate evaluation. How do proxy evaluation results relate to observed online outcomes, and how do selected candidates behave across evaluation stages? RQ3: Online effectiveness. Do strategies optimized with PEAR improve the online optimization objective during Decision-Grade Online Evaluation? Implementation Detail. Experiments at L1–L3 run on timescales of hours, and L4 experiments run on timescales of days. Main Order/DAU is the primary outcome reported for the final online A/B experiments; the remaining reported outcomes are auxiliary business metrics for the e-commerce search setting. Traffic conditions, observation windows, and evaluation signals differ across stages. Cross-stage analysis therefore examines the direction of candidate effects rather than directly comparing their magnitudes. For industrial search optimization, each strategy task instantiates an ordered sequence of active evaluation levels according to data availability and validation requirements. Candidates must satisfy the promotion criterion at each active level before advancing to the next level in that sequence. In the following experiments, data-access restrictions across regions preclude the use of Offline Replay (L1), so the evaluated sequence begins at Shadow-Traffic Evaluation (L2) and the subsequent experimental sections do not involve L1 validation. Although L1 is not used in these experiments, it has assisted algorithm engineers in optimizing other strategies in production. 6.2 RQ1: Evidence-Driven Hypothesis Revision Candidate exploration and selection. Table 1: L2 search scale and candidate selection. Unique configurations are deduplicated within each phase; promotion rate is the number of candidates promoted from L2 divided by the number of experimental groups. Task phase Experimental rounds Groups Unique configs. Promoted from L2 Promotion rate A 19 133 20 2 1.50% B–I 12 84 23 5 5.95% B–I 2 14 6 1 7.14% B–I 21 147 53 2 1.36% C 16 76 54 3 3.95% Table 1 summarizes candidate exploration at L2 for three anonymized strategy tasks. Each task evaluates multiple configurations and retains a small subset for online evaluation. Task B comprises three consecutive search phases: the first two explore local parameter neighborhoods, and Phase I expands the candidate range based on accumulated evaluation results. The following trajectories examine how this evidence informs hypothesis revision and subsequent candidate generation. Hypothesis revision through controlled interventions. Table 2: Representative hypothesis-revision trajectory for Task A. The table shows four early experimental rounds from a 19-round search; G=(w,q)G=(w,q) denotes the global window and capacity constraint. Experimental round / research question Controlled interventions Evidence-driven update 1. Which constraint axis accounts for the observed gain? Apply one-axis changes to global and format-specific constraints; also test the coupled global change G=(3,2)G=(3,2). Only G=(3,2)G=(3,2) yields a significant positive order signal. The effects of increasing w, increasing q, and experimental variation remain entangled. 2. Can G=(3,2)G=(3,2) be reproduced, and which variable explains its gain? Replicate G=(3,2)G=(3,2); compare window-only G=(3,1)G=(3,1), capacity-only G=(2,2)G=(2,2), and boundary settings including G=(3,3)G=(3,3). G=(3,3)G=(3,3) produces a stronger signal. Replications of G=(3,2)G=(3,2) yield mixed directional effects and indicate potential degradation in the exit metric. The next experimental round prioritizes capacity and replication. 3. Is the gain driven by the coupled change or by capacity alone? Replicate G=(3,3)G=(3,3) and G=(3,2)G=(3,2); test capacity-only G=(2,2)G=(2,2) and wider-window boundary settings. G=(2,2)G=(2,2) achieves the strongest observed result. The revised hypothesis attributes the gain primarily to capacity relaxation without requiring a wider window. 4. Are the marginal effects of window and capacity stable? Concentrate replications on G=(2,2)G=(2,2), G=(3,2)G=(3,2), and G=(3,3)G=(3,3); retain window-only G=(3,1)G=(3,1) as a counterfactual. G=(3,3)G=(3,3) remains directionally favorable on order and exit signals but is not significant. The capacity-centered hypothesis is retained for further replication. Task A illustrates how controlled interventions distinguish competing explanations for an observed gain. The task optimizes constraints on e-commerce result density. Let G=(w,q)G=(w,q) denote the global constraint, where w is the window size and q is the maximum content capacity within that window; the deployed baseline is G=(2,1)G=(2,1). Table 2 traces four representative early experimental rounds. Other constraint families remain at their baseline values after initial screening. The initial gain from G=(3,2)G=(3,2) leaves the effects of window size and capacity unresolved. The next experimental round separates these factors and replicates the coupled setting. Mixed directional effects across replications motivate further tests of capacity relaxation. In Experimental Round 3, the capacity-only setting G=(2,2)G=(2,2) achieves the strongest observed result, shifting the hypothesis toward capacity relaxation rather than window expansion. Experimental Round 4 then examines the consistency of this hypothesis through additional controlled comparisons and replication. Hypothesis revision from online evaluation. Task B optimizes position-dependent exposure settings for e-commerce search results. Table 3 summarizes three search phases and their outcomes from Decision-Grade Online Evaluation. Table 3: Outcomes from Decision-Grade Online Evaluation across search phases for Task B. Phase Experimental rounds Groups Unique configs. Promoted from L2 L4 outcome I 12 84 23 5 No significant gain I 2 14 6 1 Positive, not significant I 21 147 53 2 Significant order gain Figure 2: Hypothesis-driven search trajectory for Task B. Candidate evaluations update a retained hypothesis state through validation, rejection, and branch termination until Decision-Grade Online Evaluation. In the first two phases, candidates promoted from L2 do not achieve significant improvements at L4. These online evaluation results are incorporated into the research state to reassess the hypotheses underlying candidate generation. The revised hypotheses guide further exploration at L2, expanding the candidate range beyond the initial parameter neighborhoods. Figure 2 illustrates this exploration process: the agent uses accumulated evaluation evidence to revise hypotheses, terminate unsupported branches, and redirect candidate generation. In Phase I, the revised exploration identifies candidates for further online evaluation, one of which subsequently achieves a significant improvement at L4. The trajectory illustrates how evidence from Decision-Grade Online Evaluation is incorporated into hypothesis revision and subsequent candidate generation, connecting candidate promotion with continued search. 6.3 RQ2: Progressive Candidate Evaluation Aggregate agreement between proxy predictions and online outcomes. An independent controlled experiment compares mean predicted and observed click, order, and GMV outcomes at a common session granularity. Table 4 reports the signed relative errors between these aggregate means. The order component has the smallest absolute relative error, with a signed value of −3.49%-3.49\%. This comparison assesses aggregate agreement rather than the accuracy of predicted effects for individual candidate interventions. Table 4: Signed relative errors between the aggregate means of proxy predictions and observed online outcomes. Click Order GMV Signed relative error −16.02%-16.02\% −3.49%-3.49\% −18.23%-18.23\% Candidate effects across evaluation stages. Table 5 follows two Task A candidates from L2 through L3 to L4. Both traced candidates have positive order proxy effects at L2 and retain positive observed effects at L3 and L4. These cases illustrate directional consistency from proxy-based screening to Decision-Grade Online Evaluation. Table 5: Candidate effects across evaluation stages for two Task A candidates. L2 reports changes in the order proxy, L3 reports changes in observed orders, and L4 reports changes in Main Order/DAU. Candidate L2: Shadow-Traffic Evaluation L3: Rapid Online Evaluation L4: Decision-Grade Online Evaluation A Order proxy +45.88%+45.88\% Observed order +15.48%+15.48\% Main Order/DAU +2.73%+2.73\% B Order proxy +61.64%+61.64\% Observed order +29.76%+29.76\% Main Order/DAU +2.16%+2.16\% Together, the aggregate comparison and the two candidate trajectories provide illustrative evidence that proxy evaluation can support candidate screening, while online evaluation remains necessary to establish effects under real-user traffic. 6.4 RQ3: Online Effectiveness Table 6 reports Decision-Grade Online Evaluation results for strategies optimized with PEAR for two search optimization tasks. Relative to their respective baselines, the strategies for Task A and Task B significantly improve Main Order/DAU by 2.7336% and 3.2957%, respectively. Both strategies also achieve significant gains in ASN, SKU Order/DAU, Main OPMS, and SKU OPMS. Table 6: Relative metric changes in online A/B evaluations of strategies optimized with PEAR for two search optimization tasks. All values are relative to the corresponding baseline; dark-green boldface indicates statistical significance at p<0.05p<0.05. Task SearchPV/DAU ASN GMV/DAU Main Order/DAU SKU Order/DAU PayPV/PV Main OPMS SKU OPMS A +0.0063%+0.0063\% +1.1512%+1.1512\% +2.2136%+2.2136\% +2.7336%+2.7336\% +2.7259%+2.7259\% +1.0441%+1.0441\% +2.7407%+2.7407\% +2.7375%+2.7375\% B +0.0470%+0.0470\% +1.3841%+1.3841\% +1.6633%+1.6633\% +3.2957%+3.2957\% +2.9826%+2.9826\% +1.6298%+1.6298\% +3.2595%+3.2595\% +2.9448%+2.9448\% 7 Conclusion PEAR is an AutoResearch framework for industrial search optimization under non-stationary outcomes and heterogeneous evaluation signals. Evidence-driven AutoResearch associates hypotheses with evidence and its experimental context to support reliable search knowledge accumulation. Confidence-Gated Verifier Ladder connects four evaluation levels through a unified confidence-based promotion gate. The two components jointly turn evaluation results into signals that guide both candidate promotion and hypothesis revision, linking lower-cost exploration with Decision-Grade Online Evaluation. Experiments in a real-world industrial search system show how accumulated evidence informs controlled interventions and how online evaluation results redirect subsequent search. Strategies optimized with PEAR increased Main Order/DAU by 2.7336% and 3.2957% relative to their respective baselines in two A/B experiments at the Decision-Grade Online Evaluation stage. These results support the use of contextual evidence and progressive evaluation for iterative industrial search optimization. Future work will examine broader intervention scopes, including model architectures and training procedures. 8 Limitations The current work is confined to a single e-commerce general search environment, and PEAR’s generality across other search platforms, geographic regions, and evaluation configurations remains to be established. Where compliance constraints preclude Offline Replay (L1), evaluation proceeds from L2 onward while retaining the higher-fidelity validation stages. The search space is currently restricted to strategy-level parameters; its extension to model parameters, architectures, and training procedures is left to future work. Confidence-Gated Verifier Ladder also depends on the compute clusters, deployed evaluation and search models, and live traffic of the underlying search environment, which constrains its applicability in settings where such resources are unavailable. 9 Ethical Considerations In the evaluated deployment, AutoResearch is employed to optimize an e-commerce general search system. As optimization feedback, the agents receive model-based proxy scores produced by the list-level evaluation model and aggregate effect estimates of the objective signal from online experiments, rather than user-level records, personal identifiers, sensitive attributes, or other sensitive information. All data used for evaluation and experimentation are processed under applicable data-governance and compliance requirements. The optimization loop therefore operates on compliant evaluation signals within the existing authorized experimentation pipeline and does not require additional access to personal or sensitive data. Before any candidate is exposed to real-user traffic, the proposed change and supporting evidence undergo human review and require explicit human confirmation; automated promotion within the Verifier Ladder cannot independently initiate such exposure. 10 Acknowledgments We thank our colleagues on the E-commerce General Search, Architecture, Product, and Data Science teams for their invaluable contributions to the design, development, and deployment of the systems described in this work. We are grateful to the AutoSearch tooling team for building the compute cluster and infrastructure that enabled large-scale AutoResearch for industrial search system optimization. Author List Global E-Commerce Agentic Search Team: Yifan Wang*, Shipeng Zhu, Fei Xiong, Yuqin Yang, Yonghui Huang, Kunyao Wu, Yue Wang, Weichao Meng, Yu Gong†. *: First author, †: Corresponding author, References [1] D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes (2023) Autonomous chemical research with large language models. Nature 624 (7992), p. 570–578. External Links: Document Cited by: §1. [2] G. Chen, J. Chen, L. Chen, J. Zhao, F. Meng, W. X. Zhao, R. Song, C. Chen, J. Wen, and K. Jia (2026) Toward autonomous long-horizon engineering for ml research. arXiv preprint arXiv:2604.13018. Cited by: §3.1. [3] Y. Chen, Q. Liu, Y. Zhang, W. Sun, X. Ma, W. Yang, D. Shi, J. Mao, and D. Yin (2025) TourRank: utilizing large language models for documents ranking with a tournament-inspired strategy. In Proceedings of the ACM on Web Conference 2025, W ’25, New York, NY, USA, p. 1638–1652. External Links: ISBN 9798400712746, Link, Document Cited by: §1. [4] Y. Cheng, L. Zhou, X. Liang, D. Luo, T. Lee, K. Zheng, W. Zhang, M. Cai, J. Dong, and A. Zhang (2026) Let the agent steer: closed-loop ranking optimization via influence exchange. arXiv preprint arXiv:2603.27765. Cited by: §1, §1, §3.2. [5] Q. Huang, J. Vora, P. Liang, and J. Leskovec (2024) MLAgentBench: evaluating language agents on machine learning experimentation. In Proceedings of the 41st International Conference on Machine Learning, External Links: Link Cited by: §1, §3.1. [6] K. Kandasamy, G. Dasarathy, J. Schneider, and B. Póczos (2017) Multi-fidelity bayesian optimisation with continuous approximations. In Proceedings of the 34th International Conference on Machine Learning, Vol. 70, p. 1799–1808. External Links: Link Cited by: §1. [7] S. Kim, S. Park, H. Kang, W. Kim, J. Seo, Y. In, K. Yoon, H. Jeon, and C. Park (2026) Self-evolverec: self-evolving recommender systems with llm-based directional feedback. arXiv preprint arXiv:2602.12612. Cited by: §1, §1, §3.2. [8] R. Kohavi, R. Longbotham, D. Sommerfield, and R. M. Henne (2009) Controlled experiments on the web: survey and practical guide. Data Mining and Knowledge Discovery 18, p. 140–181. External Links: Document Cited by: §1. [9] C. Lao, F. Pan, G. Ma, H. Li, H. Lin, J. Shi, K. Zhao, K. Gai, M. Zhou, Q. Zhou, et al. (2026) Agentx: towards agent-driven self-iteration of industrial recommender systems. arXiv preprint arXiv:2606.26859. Cited by: §1, §1, §3.2. [10] Y. Lee, R. Nair, Q. Zhang, K. Lee, O. Khattab, and C. Finn (2026) Meta-harness: end-to-end optimization of model harnesses. arXiv preprint arXiv:2603.28052. Cited by: §3.1. [11] L. Li, W. Chu, J. Langford, and X. Wang (2011) Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In Proceedings of the Fourth ACM International Conference on Web Search and Data Mining, p. 297–306. External Links: Document Cited by: §1. [12] L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar (2018) Hyperband: a novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research 18 (185), p. 1–52. External Links: Link Cited by: §1. [13] J. Liu, S. Qiu, M. Li, B. Li, H. Ji, S. Han, X. Ye, P. Xia, Z. Dong, M. Chen, et al. (2026) Autoresearchclaw: self-reinforcing autonomous research with human-ai collaboration. arXiv preprint arXiv:2605.20025. Cited by: §3.1. [14] Z. Liu, C. Li, S. Xiao, C. Li, C. J. Zhang, H. Liao, D. Lian, and Y. Shao (2025) Fitting into any shape: a flexible llm-based re-ranker with configurable depth and width. In Proceedings of the ACM on Web Conference 2025, W ’25, New York, NY, USA, p. 3942–3951. External Links: ISBN 9798400712746, Link, Document Cited by: §1. [15] C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024) The AI scientist: towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. External Links: Link Cited by: §1, §1, §3.1. [16] T. A. Mann, S. Gowal, A. Gyorgy, H. Hu, R. Jiang, B. Lakshminarayanan, and P. Srinivasan (2019) Learning from delayed outcomes via proxies with applications to recommender systems. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97, p. 4324–4332. External Links: Link Cited by: §1. [17] J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein (2023) Generative agents: interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, p. 1–22. External Links: Document Cited by: §1. [18] C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun (2024) ChatDev: communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 15174–15186. External Links: Document Cited by: §1. [19] B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. R. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, P. Kohli, and A. Fawzi (2024) Mathematical discoveries from program search with large language models. Nature 625 (7995), p. 468–475. External Links: Document Cited by: §1, §1, §3.1. [20] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom (2023) Toolformer: language models can teach themselves to use tools. In Advances in Neural Information Processing Systems, Vol. 36. Cited by: §1. [21] G. Tie, J. Shi, D. Song, Y. Huang, Z. Sheng, X. Zhou, D. Liu, P. Zhou, Y. Chen, R. Xu, et al. (2026) Autoresearch ai: towards ai-powered research automation for scientific discovery. arXiv preprint arXiv:2605.23204. Cited by: §3.1. [22] H. Wang, Y. Wu, D. Chang, L. Wei, and L. Heldt (2026) Self-evolving recommendation system: end-to-end autonomous model optimization with llm agents. arXiv preprint arXiv:2602.10226. Cited by: §1, §1, §3.2. [23] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W. X. Zhao, Z. Wei, and J. Wen (2024) A survey on large language model based autonomous agents. Frontiers of Computer Science 18 (6), p. 186345. External Links: Document Cited by: §1. [24] S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, and K. Narasimhan (2023) Tree of thoughts: deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, Vol. 36, p. 11809–11822. External Links: Document Cited by: §1. [25] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao (2023) ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, External Links: Link Cited by: §1. [26] S. Yoon, G. Kim, G. Cho, and S. Hwang (2025) AcuRank: uncertainty-aware adaptive computation for listwise reranking. External Links: 2505.18512, Link Cited by: §1. [27] J. Zhou, Y. Zheng, W. Chen, Q. Zheng, H. Su, W. Zhang, R. Meng, and X. Shen (2025) Beyond content relevance: evaluating instruction following in retrieval models. External Links: 2410.23841, Link Cited by: §1. Appendix A Confidence Computation for the Unified Promotion Policy This appendix specifies the stage-specific estimators used by the unified promotion policy (Section 5.2) to obtain the candidate effect Δ^ℓ(c) _ (c) and its confidence interval CIℓ(c)CI_ (c). The evidence schema and promotion rule are shared across levels, while the estimator follows the sampling structure of each evaluation protocol. L1 uses request-aligned paired observations. L2 uses independent shadow-traffic groups, and L3 and L4 use independent randomized online groups. Paired estimation for Offline Replay (L1). Let xix_i denote the i-th historical request shared by candidate c and baseline b1b_1, and define the request-level proxy-score difference as di=m1(c,xi)−m1(b1,xi),i∈1,…,n.d_i=m_1(c;x_i)-m_1(b_1;x_i), i∈\1,…,n\. (16) The L1 effect, variance of the paired differences, standard error, and degrees of freedom are Δ^1(c)=d¯=1n∑i=1ndi,sd2=1n−1∑i=1n(di−d¯)2, _1(c)= d= 1n _i=1^nd_i, s_d^2= 1n-1 _i=1^n(d_i- d)^2, (17) SE1(c)=sdn,ν1=n−1.SE_1(c)= s_d n, _1=n-1. (18) This paired estimator controls for request-level variation because both strategies are evaluated on the same historical requests. Independent-group estimation for L2–L4. For ℓ∈2,3,4 ∈\2,3,4\ and strategy g∈c,bℓg∈\c,b_ \, let xg,i(ℓ)x_g,i^( ) denote its i-th valid evaluation unit and let ng,ℓn_g, be the number of such units. L2 units are collected from independent shadow-traffic groups without request-level pairing, whereas L3 and L4 units are collected from independently randomized online groups. The per-strategy mean and variance are x¯g,ℓ=1ng,ℓ∑i=1ng,ℓxg,i(ℓ),sg,ℓ2=1ng,ℓ−1∑i=1ng,ℓ(xg,i(ℓ)−x¯g,ℓ)2. x_g, = 1n_g, _i=1^n_g, x_g,i^( ), s_g, ^2= 1n_g, -1 _i=1^n_g, (x_g,i^( )- x_g, )^2. (19) The estimated effect and Welch standard error are Δ^ℓ(c)=x¯c,ℓ−x¯bℓ,ℓ,SEℓ(c)=sc,ℓ2nc,ℓ+sbℓ,ℓ2nbℓ,ℓ, _ (c)= x_c, - x_b_ , , _ (c)= s_c, ^2n_c, + s_b_ , ^2n_b_ , , (20) with Welch–Satterthwaite degrees of freedom νℓ=(sc,ℓ2nc,ℓ+sbℓ,ℓ2nbℓ,ℓ)2(sc,ℓ2/nc,ℓ)2nc,ℓ−1+(sbℓ,ℓ2/nbℓ,ℓ)2nbℓ,ℓ−1. _ = ( s_c, ^2n_c, + s_b_ , ^2n_b_ , )^2 (s_c, ^2/n_c, )^2n_c, -1+ (s_b_ , ^2/n_b_ , )^2n_b_ , -1. (21) Confidence interval. For every active level, the two-sided confidence interval at significance level α is CIℓ(c)=[Δ^ℓ(c)−tνℓ, 1−α/2SEℓ(c),Δ^ℓ(c)+tνℓ, 1−α/2SEℓ(c)].CI_ (c)= [ _ (c)-t_ _ ,\,1-α/2\,SE_ (c),\; _ (c)+t_ _ ,\,1-α/2\,SE_ (c) ]. (22) When a relative effect is reported, the effect and interval endpoints are normalized by the corresponding baseline mean. By default, α=0.05α=0.05, corresponding to 95%95\% confidence intervals. Promotion decision. With the evaluation signal oriented so that larger values indicate better outcomes, the promotion decision applies the interval endpoints Lℓ(c)L_ (c) and Uℓ(c)U_ (c) as in Section 5.2. A candidate is promoted when the interval lies entirely above zero (Lℓ(c)>0L_ (c)>0), which provides statistically significant evidence of a positive effect at the current significance level. It is retained for further evaluation when the interval contains zero (Lℓ(c)≤0≤Uℓ(c)L_ (c)≤ 0≤ U_ (c)), which indicates that the observed effect is not yet distinguishable from no effect. It is stopped when the interval lies entirely below zero (Uℓ(c)<0U_ (c)<0), which provides statistically significant evidence of a negative effect. The relative interval CIcrelCI_c^rel shares the sign of the absolute interval and yields the same decision; it is reported for interpretability. Appendix B Prompt Templates and Structured Outputs for Research-State Updates The following templates summarize the instructions for hypothesis revision and cross-round evidence reuse in Evidence-driven AutoResearch. They retain the core inputs, update rules, and outputs while omitting deployment-specific identifiers and serialization details. B.1 Hypothesis Update Prompt Prompt Template for Hypothesis Update Objective and inputs. Maintain testable parameter regularities across experimental rounds. Read the completed round summary, candidate records, confidence evidence, final review, lessons, and existing hypotheses. Update instructions. 1. Generate at most three new hypotheses per round. Specify the parameter range, expected metric direction, risk boundary, and implications for candidate design; do not merely restate the best candidate. 2. Merge equivalent claims with overlapping scopes. Record narrower findings as scoped evidence without broadening their support. Keep claims with different directions or risk conditions separate and link them through Related. 3. Assign support when covered evidence supports the claim under the evaluation criteria, refute when it contradicts the claim or violates a decision constraint, and neutral when scope coverage or evidence is insufficient. Each round contributes at most one verdict per hypothesis. 4. Increment the corresponding counter and set Score = Support - Refute. Scores of at least 22 support active status; scores of at most −2-2 indicate risky hypotheses. Archive superseded or persistently unused hypotheses. Plan use. Retrieve relevant active hypotheses with scores of at least 22 and risky hypotheses with scores of at most −2-2. Require a matching strategy version and explain each hypothesis’s effect on candidate design. If none applies, mark the round as exploratory. These scores summarize hypothesis evidence; they are not verifier confidence estimates. Hypothesis Output Template. ### H-id: testable parameter regularity - **Status:** active / risky / archived - **Strategy Version:** strategy fingerprint - **Score:** Support - Refute - **Support:** supporting-round count - **Refute:** refuting-round count - **Neutral:** neutral-round count - **Scope:** parameter ranges and experimental conditions - **Claim:** expected metric direction under these conditions - **Plan Use:** implications for subsequent candidate design - **Risk:** trade-offs, applicability limits, or counterexamples - **Evidence:** round:candidate verdict; ... - **Related:** related hypothesis identifiers or none - **Updated At:** timestamp Plan-Stage Output Template. Used hypotheses: H-id score=value: design implication Candidate design: candidate: modification; hypothesis tested B.2 Cross-Round Memory Update Prompt Prompt Template for Cross-Round Memory Update Objective and inputs. Maintain two complementary records: structured lessons preserve round-level facts and reasoning; compressed memory summarizes cross-round findings and guides subsequent candidate design. Use all round summaries for the current strategy and the previous memory. Update instructions. 1. After the final review, append one non-failure lesson per round with an Outcome matching the recorded decision. Record execution failures separately as crash lessons with their causes. 2. Preserve the strategy version, experimental context, decision rationale, and evidence references. When summaries disagree with measured evidence or experimental records, correct the summaries. 3. Before planning, retrieve relevant lessons with a matching strategy version. Use supported findings, avoid repeatedly unsuccessful directions, and resolve recurring execution failures. Incompatible lessons remain available for audit, not direct candidate guidance. 4. Rewrite compressed memory after each round using the current strategy’s summaries and preceding memory. Preserve current-run lessons, down-weight older findings, and compress related historical lessons without losing evidence references. Provide concrete guidance for subsequent candidate design. Outcome semantics. keep, discard, and iterate summarize the recorded round decision; pivot denotes a change of strategy family, crash an execution failure, and summary a historical synthesis. Lesson Output Template. ### L-N: title - **Strategy:** candidate settings and search design - **Strategy Version:** strategy fingerprint - **Outcome:** keep / discard / iterate / pivot / crash / summary - **Insight:** actionable finding to reuse or avoid - **Context:** goal=...; scope=...; metric=...; direction=... - **Iteration:** round identifier#task identifier - **Reason:** rationale consistent with the recorded decision - **Refs:** trace=record; experiment=id; round=id - **Timestamp:** timestamp Compressed Memory. The source protocol specifies its content rather than a fixed output schema: reusable findings, their applicability conditions and evidence references, and concrete candidate-design guidance for the next round. B.3 Structured Experimental Records Structured Output Templates for Experimental Records Recording instructions. Link candidate designs, confidence evidence, and the final review through shared task and experimental-round identifiers. Retain one baseline per round and record each candidate’s parameters and design rationale. Base selection and the final decision on the confidence-evaluation record, not legacy score logs. The templates below retain selected source fields; placeholders replace deployment-specific values. State and decision. The source protocol uses cycle for an experimental round. Its state is RUNNING, CLOSED, or BLOCKED. Completed rounds record KEEP, DISCARD, or ITERATE, mapped to the corresponding lesson outcomes. Blocked rounds record a failure cause without a decision. These round-level labels are distinct from the verifier promotion labels in the Method. Final Review Template. "task_name": "<task identifier>", "cycle_id": "<round identifier>", "stage": "review", "status": "DONE", "decision": "<KEEP | DISCARD | ITERATE>", "reason": "<evidence-based decision rationale>", "best_version": "<selected candidate identifier>", "best_selected_by": "llm_from_confidence_pipeline", "confidence_method": "<evaluation method>", "confidence_file": "<confidence-evidence record>", "best_confidence": , "strategy_contract_version": "<strategy fingerprint>", "versions_file": "<candidate-design records>" Evidence Trace Template. "task_name": "<task identifier>", "round": "<round label>", "cycle": "cycle_id": "<round identifier>", "status": "CLOSED", "best_version": "<selected candidate identifier>", "confidence_method": "<evaluation method>", "confidence_file": "<confidence-evidence record>", "best_confidence": , "decision": "<KEEP | DISCARD | ITERATE>", "versions_file": "<candidate-design records>" , "decision": "<same round decision>" The best_confidence object contains the selected candidate’s confidence evidence. Candidate, decision, and evidence references must agree across the review, trace, and lesson records. B.4 Knowledge-Augmented Candidate Generation Prompt This template summarizes the Bayesian-optimization-inspired reasoning principles BOK_BO used in Equation 4. These principles guide qualitative reasoning about observed effects, uncertainty, and candidate diversity. Prompt Template for Knowledge-Augmented Candidate Generation Inputs. Mutable parameters and their types, intervention bounds, baseline, candidate budget, historical evaluations, hypotheses, memory, and exact-replication requirements. Generation instructions. 1. Classify the parameter space and diagnose the search state as cold_start, local_refine, stagnant, or pivot. Use repeated evidence, coverage, and contradictions rather than the largest point estimate alone. 2. Select one reasoning strategy. Use GP-inspired reasoning for low-dimensional continuous spaces: contrast supported regions with uncertainty gaps. Use TPE-inspired reasoning for mixed continuous and integer spaces: distinguish supported, unfavorable, and unresolved parameter patterns. Use SMAC-inspired reasoning for categorical or conditional spaces: compare supported branches and structural counterfactuals. 3. Form a candidate pool at least twice the available new-candidate budget. Cover refinement, boundary extension, uncertainty probes, and interaction tests. Preserve negative and inconclusive evidence when selecting candidates. 4. Under stagnation or a change of search direction, increase exploration. Include a feasible point beyond the explored range on a meaningful axis and an interaction test or uncovered branch. Remain within the intervention bounds. 5. Remove invalid, excluded, or duplicate proposals and select a diverse subset. Keep exact replications separate from new candidates: replication preserves the full configuration, while every new configuration is generated through the selected reasoning strategy. Output Template. Report concise, evidence-backed selection summaries and candidate roles. The following layout summarizes the required fields without reproducing the full planning schema. ideation_strategy.method: <GP-/TPE-/SMAC-inspired> search_state: <cold_start | local_refine | stagnant | pivot> cycle_mode: <current experiment-allocation mode> exploration_level: <level; high for stagnant or pivot> selection_reason: <brief evidence-backed rationale> Allocation: <exact-confirmation and new-candidate counts> For each candidate: candidate_source: <confirmation_queue | algorithm> search_role: <replication or method-specific exploration role>