Paper deep dive
SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search
Zhongxin Huang, Songyang Li, Renzhe Zhou, Feiran Zhu, Chenglei Dai, Zhen Xiao, Xuanping Li, Jingwei Zhuo
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/26/2026, 2:20:43 AM
Summary
The paper introduces SEEK (Skill-routed Evaluation with Evolvable Knowledge), a framework for industrial search quality evaluation that decouples evaluation knowledge from model parameters. SEEK uses a skill bank to store modular evaluation criteria, a skill router to dynamically select relevant skills for each query-result pair, and a listwise evaluator to produce page-level judgments and failure mode attributions. The system features a self-evolving skill bank that updates knowledge based on production feedback without requiring model retraining. It has been deployed at Kuaishou, significantly improving evaluation scale and quality.
Entities (7)
Relation Signals (6)
SEEK → deployedat → Kuaishou
confidence 99% · SEEK has been deployed at Kuaishou
SEEK → uses → Skill Bank
confidence 95% · SEEK externalizes specific search evaluation criteria into a skill bank
SEEK → uses → Skill Router
confidence 95% · uses a lightweight skill router to select a compact subset of task-relevant skills
SEEK → uses → Listwise Evaluator
confidence 95% · employs a task-adapted listwise evaluator to produce page-level judgments
Skill Bank → evolvesvia → Production Feedback
confidence 92% · self-evolving skill bank attributes reviewed production failures... and incorporates recurring evaluation knowledge gaps
Listwise Evaluator → trainedwith → DAPO
confidence 90% · We then apply DAPO [45] to further align the evaluator with human search judgments.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Search quality evaluation provides essential supervision and diagnostic signals for the development and iteration of industrial search systems. Although large language models (LLMs) offer a scalable alternative to manual assessment, reliable automatic evaluation remains challenging: users experience search results at the page level, while the applicable evaluation criteria are multi-dimensional and continuously evolving. Packing all evaluation criteria into a unified prompt introduces irrelevant context and potential criterion interference, whereas internalizing them through post-training tightly couples rule updates with costly model retraining cycles. To address these issues, we propose Skill-routed Evaluation with Evolvable Knowledge (SEEK). Specifically, SEEK externalizes specific search evaluation criteria into a skill bank, dynamically routes relevant skills for each query-result list pair, and employs a task-adapted listwise evaluator to produce page-level judgments and failure mode attribution. A two-stage training pipeline teaches the evaluator to align evaluation criteria with human preferences, while a replay-gated skill bank allows recurring evaluation knowledge gaps to be incorporated without model retraining. Experiments on industrial short-video search show that SEEK improves listwise quality evaluation accuracy and achieves significant progress in attribution diagnosis. SEEK has been deployed at Kuaishou, a short-video platform with over 400 million daily active users, significantly improving the scale and quality of online search evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2609.29803v1
- Canonical: https://arxiv.org/abs/2609.29803v1
Trouble viewing inline? Open PDF directly →
Full Text
72,977 characters extracted from source content.
Expand or collapse full text
SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial Search Zhongxin Huang ∗ huangzhongxin26@stu.pku.edu.cn Peking University Beijing, China Songyang Li ∗ lisongyang03@kuaishou.com Kuaishou Technology Beijing, China Renzhe Zhou ∗ zhourenzhe03@kuaishou.com Kuaishou Technology Hangzhou, China Feiran Zhu ∗ a499616042@163.com Unaffiliated Hangzhou, China Chenglei Dai daichenglei@kuaishou.com Kuaishou Technology Hangzhou, China Zhen Xiao † xiaozhen@pku.edu.cn Peking University Beijing, China Xuanping Li lixuanping@kuaishou.com Kuaishou Technology Beijing, China Jingwei Zhuo † zhuojw10@gmail.com Unaffiliated Beijing, China Abstract Search quality evaluation provides essential supervision and diag- nostic signals for the development and iteration of industrial search systems. Although large language models (LLMs) offer a scalable alternative to manual assessment, reliable automatic evaluation re- mains challenging: users experience search results at the page level, while the applicable evaluation criteria are multi-dimensional and continuously evolving. Packing all evaluation criteria into a unified prompt introduces irrelevant context and potential criterion inter- ference, whereas internalizing them through post-training tightly couples rule updates with costly model retraining cycles. To address these issues, we propose Skill-routed Evaluation with Evolvable Knowledge (SEEK). Specifically, SEEK externalizes spe- cific search evaluation criteria into a skill bank, dynamically routes relevant skills for each query–result list pair, and employs a task- adapted listwise evaluator to produce page-level judgments and failure mode attribution. A two-stage training pipeline teaches the evaluator to align evaluation criteria with human preferences, while a replay-gated skill bank allows recurring evaluation knowledge gaps to be incorporated without model retraining. Experiments on industrial short-video search show that SEEK improves listwise quality evaluation accuracy and achieves significant progress in attribution diagnosis. SEEK has been deployed at Kuaishou, a short- video platform with over 400 million daily active users, significantly improving the scale and quality of online search evaluation. ∗ Equal contribution. † Corresponding author. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. Conference acronym ’X, Woodstock, NY © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-X-X/2018/06 https://doi.org/X.X CCS Concepts • Information systems→ Evaluation of retrieval results. Keywords Search Evaluation, Listwise Modeling, LLM-as-a-Judge, Self-Evolving Skills, Large Language Models ACM Reference Format: Zhongxin Huang, Songyang Li, Renzhe Zhou, Feiran Zhu, Chenglei Dai, Zhen Xiao, Xuanping Li, and Jingwei Zhuo. 2026. SEEK: Skill-Routed Eval- uation with Evolvable Knowledge for Industrial Search. In Proceedings of Make sure to enter the correct conference title from your rights confirma- tion email (Conference acronym ’X). ACM, New York, NY, USA, 12 pages. https://doi.org/X.X 1 Introduction Search quality evaluation is a fundamental component of mod- ern search systems. It provides the labels and diagnostic signals required for model development, offline comparison and online experimentation [8,18]. Traditionally, such evaluations rely on trained human assessors. Although human judgments remain the most reliable source of task-specific supervision, their high cost and long turnaround time make it difficult to cover rapidly changing traffic, long-tail queries, and frequent system iterations at industrial scale [21, 22, 32]. Large language models (LLMs) offer a promising alternative for scaling search evaluation. Recent studies have shown that care- fully calibrated LLMs can generate useful relevance judgments and achieve agreement with human assessors on both academic and industrial search tasks [34,36,39,46]. These results have moti- vated a growing body of work on using LLMs as automatic judges for retrieval and search systems [2,30,47,50]. However, directly prompting a general-purpose LLM does not fully satisfy the require- ments of industrial search evaluation. One challenge is that users experience a search result page as a whole rather than as isolated retrieved items. Page quality depends jointly on relevance, ranking, content quality, redundancy, intent arXiv:2609.29803v1 [cs.IR] 24 Sep 2026 Conference acronym ’X, June 03–05, 2018, Woodstock, NYHuang et al. coverage, and source authority [7,11,15,17,48]. Several of these properties are inherently page-level. Recent work has begun to move beyond independent query–item grading through batched relevance assessment, usefulness-oriented rubrics, and behavior- grounded evaluation [7,14,37]. Nevertheless, most existing auto- matic search evaluation methods still focus primarily on item-level relevance or ranking [19,41], rather than jointly predicting page- level quality and causes of quality degradation. Another challenge is how to represent rich and continuously evolving evaluation knowledge. Packing all rules, boundary cases, and evidence requirements into a single prompt exposes every sam- ple to largely irrelevant criteria, creating contextual redundancy and potential criterion interference [1,3,4,9]. Task-specific super- vised fine-tuning (SFT) and reinforcement learning (RL) have been increasingly adopted to specialize LLMs for industrial Web applica- tions, including search relevance modeling and sequential decision making [21,31,32,43]. However, they also internalize evaluation knowledge into model parameters. As platform content, query in- tents, and evaluation policies evolve, updating such knowledge may require another cycle of data construction, training, validation, and deployment [22,49]. This creates a practical tension between keeping evaluation knowledge rapidly updatable and keeping the underlying evaluation model stable. To address these challenges, we propose Skill-routed Evaluation with Evolvable Knowledge (SEEK), a skill-grounded framework that separates explicit evaluation knowledge from model selection and reasoning capabilities. SEEK externalizes domain-specific criteria into a modular skill bank and uses a lightweight skill router to select a compact subset of task-relevant skills for each query–list pair. The corresponding operational guidance is then provided, together with the original result page, to a task-adapted listwise evaluator, which produces a page-level judgment and structured attribution. A human-aligned training pipeline transforms human judgments into routing supervision and evidence-grounded evaluation trajectories, enabling the router and evaluator to learn how to select and faith- fully execute the externalized knowledge. Finally, the self-evolving skill bank attributes reviewed production failures to routing, knowl- edge, or execution errors and incorporates recurrent knowledge gaps through localized, replay-validated skill revisions. This allows newly emerging evaluation knowledge to be updated. Our main contributions are summarized as follows: •We propose SEEK, a skill-grounded framework for listwise search evaluation that decouples task-specific evaluation knowledge from model capabilities and dynamically routes relevant skills for page-level judgment. •We introduce a human-aligned training strategy that trans- forms human search judgments into routing supervision and evaluation trajectories, enabling effective specialization of a compact listwise evaluator. • We develop a self-evolving skill bank that updates reusable evaluation knowledge through error attribution and replay- gated revision, enabling lightweight adaptation without im- mediate model retraining. • Extensive offline and online experiments demonstrate the effectiveness and practical value of SEEK. The system has been deployed in Kuaishou’s search evaluation pipeline. 2 Related Work 2.1 LLM-based Search Evaluation Large language models have been increasingly explored as au- tomatic evaluators for search systems, covering relevance judg- ment, usefulness assessment, and broader search-quality evalua- tion [22,41,43,46]. Within this line of research, LLM-as-a-Judge has emerged as a promising paradigm for replacing or assisting costly human assessment in relevance evaluation [2,10,22,27]. More recent studies further extend LLM-based evaluation toward industrial objectives, including usefulness-oriented supervision, fine-grained intent satisfaction, and web-scale relevance assess- ment, substantially reducing annotation cost while improving eval- uation coverage and scalability [5, 8, 39]. Despite this progress, most existing approaches either rely on pre- defined evaluation criteria or internalize task-specific knowledge through model post-training. As search policies, content ecosys- tems, and evaluation standards continuously evolve, such designs provide limited flexibility for rapidly incorporating new evaluation knowledge without modifying the underlying evaluator. 2.2 Listwise Evaluation Traditional IR evaluation typically aggregates item-level relevance judgments into ranking metrics such as Precision, MAP, and nDCG. Although these metrics are effective for measuring relevance and ranking quality, they do not explicitly model interactions among retrieved results, such as redundancy, intent coverage, and source composition. Recent work in RAG evaluation, listwise reranking, and whole-page assessment has therefore increasingly moved be- yond independent item-level judgments toward jointly modeling the retrieved list as a whole [14, 15, 28, 35, 37]. However, industrial search evaluation requires not only an over- all page-level judgment, but also fine-grained diagnosis of the under- lying quality issues. This is particularly important in short-video search, where the final experience depends jointly on heteroge- neous content signals, ranking positions, and interactions across retrieved results. 2.3 Externalized and Evolving Skills Recent research has increasingly explored decoupling task-specific knowledge from model parameters and representing it through modular, reusable structures. Human judgment criteria, policies, precedents, and task instructions can be externalized as explicit knowledge interfaces or reusable skills, while agent systems further acquire and accumulate such skills from execution trajectories and interaction experience [4, 16, 23, 38, 40]. More recent studies extend static skill repositories toward contin- ual refinement and self-evolution, including revising textual skills from execution feedback, optimizing external skills while keeping model parameters fixed, and updating skill libraries through in- teraction experience [13,20,44]. Despite this progress, controlled evolution of task-specific evaluation knowledge under a relatively stable evaluator remains underexplored. This challenge is partic- ularly important in industrial listwise search evaluation, where evaluation criteria evolve over time and knowledge updates must remain localized, reliable, and backward-compatible. SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial SearchConference acronym ’X, June 03–05, 2018, Woodstock, NY Selected Skills: (1)MainIntentMatch (2)FactualReliabilityCheck (3)RiskComplianceCheck Dynamic Skill Router Input: User Query ("Nutritious Breakfast") Search Result List: Top-8 Retrieved Short-Video Results MainIntentMatch: (1) Goal:Assess whether the page primarily satisfies the dominant query intent.x (2)Inspect:x (3)Decision Rules: x(4)Attribution:x FactualReliabilityCheck: (1) Goal:Check evidence... (2)Inspect:x (3)Decision Rules: x (4)Attribution:x RiskComplianceCheck: (1) Goal:Identify risk-sensitive content... (2)Inspect:x (3)Decision Rules: x (4)Attribution:x Listwise Evaluator <evidence_inspection> Inspect title, OCR/ASR, creator signals, and ranking positions. </evidence_inspection> <skill_diagnosis skill="MainIntentMatch"> #4 and #7 deviate toward weight-loss content, weakening page-level relevance. </skill_diagnosis> <skill_diagnosis skill="FactualReliabilityCheck"> Rapid weight-loss claims lack sufficient supporting evidence. </skill_diagnosis> <skill_diagnosis skill="RiskComplianceCheck"> Health-related claims introduce additional quality and risk concerns. </skill_diagnosis> <page_synthesis> Intent deviation and unsupported health claims jointly degrade the overall result page. </page_synthesis> Skill-Guided Diagnoses <Label>Final page tag: Bad.</Label> <Dimension>Relevance: Results #4 and #7 deviate substantially from the nutritious-breakfast intent by focusing on rapid weight-loss methods.</Dimension> <Dimension>Quality: Several results contain strong health-related or rapid weight- loss claims that are insufficiently supported by the available evidence, reducing the overall reliability of the result page.</Dimension> Evidence-Grounded Result Diagnosis and Evaluation Listwise Output Selected Skill Guidance 1234 5678 Figure 1: Illustrative example of skill-routed listwise evaluation in SEEK. The router selects task-relevant skills, and the listwise evaluator applies their guidance over the complete result page to produce a page-level judgment and attribution. 3 Method Problem statement. Given a user query푞and a ranked result list퐿= [푑 1 , . . .,푑 퐾 ], we study structured listwise search quality evaluation. Each result푑 푖 contains the content and metadata avail- able for assessment, including textual signals, creator information, interaction statistics, and ranking position. The task is to jointly evaluate the complete result page: 푓 assess (푞,퐿)=(푙,푎 푗 푚 푗=1 ),(1) where푙 ∈ good, fair, baddenotes the page-level quality label, and 푎 푗 푚 푗=1 is a set of structured attribution. Each푎 푗 identifies a quality dimension and provides an evidence-grounded diagnosis of the corresponding quality issue. The quality dimensions are drawn from A=Relevance, Quality, Diversity, Authority, Heterogeneous. This prediction is inherently listwise, as it depends not only on the quality of individual results, but also on their ranking positions and cross-result interactions, such as redundancy, intent coverage, and source composition. Figure 1 illustrates this setting with a short- video search example. For the query “Nutritious Breakfast”, most retrieved results satisfy the dominant intent, while several results shift toward rapid weight-loss content and contain insufficiently supported health-related claims. A reliable evaluator must therefore go beyond assigning an overall page-level label and further identify the specific dimensions responsible for the degradation, such as relevance and content quality. 3.1 SEEK Framework As illustrated in Figure 2, SEEK follows a capability–knowledge separation principle and consists of three key components: the skill bank, skill router, and listwise evaluator. The external skill bank stores explicit and evolvable evaluation knowledge, while the skill router and listwise evaluator provide relatively stable capabilities for selecting and applying such knowledge, respectively. Given a query–list pair푥= (푞,퐿), the router identifies a compact set of applicable skills from the skill bank, and the corresponding oper- ational guidance is provided to the evaluator together with the original query and result list. The evaluator then performs skill- grounded listwise reasoning to produce a page-level judgment and structured attribution. Skill Bank. We maintain a skill bankS (푡) = 푆 (푡) 1 , . . .,푆 (푡) 푁 , where푡denotes the current evolution round. In our implementa- tion, the bank contains 12 diagnostic skills spanning five evaluation dimensions: Relevance, Quality, Diversity, Authority, and Hetero- geneous. Each skill is represented by two complementary textual components: 푆 (푡) 푖 = 푟 (푡) 푖 , 푔 (푡) 푖 ,(2) where푟 (푡) 푖 is a compact routing description specifying when the skill should be activated, and푔 (푡) 푖 is structured operational guid- ance specifying how the corresponding criterion should be applied. Concretely,푔 (푡) 푖 describes the evaluation objective, evidence re- quirements, decision rules, boundary conditions, and attribution requirements. This separation allows routing and execution to rely on different representations of the same evaluation criterion. Skill Router. For each skill푆 (푡) 푖 , the router estimates its applica- bility to the current query–list pair using the corresponding routing description: 푝 푖 = 푅 휙 푞,퐿,푟 (푡) 푖 ,(3) where푅 휙 denotes the router parameterized by휙, and푝 푖 ∈ [0,1]is the activation score of the 푖-th skill. The active skill set is selected by thresholding these scores: S (푡) 푥 = n 푆 (푡) 푖 ∈S (푡) | 푝 푖 > 휏 o ,(4) where휏is the activation threshold. If no skill exceeds휏, the highest- scoring skill is retained as a fallback. In this way, the router exposes only task-relevant evaluation knowledge to the downstream evalu- ator rather than injecting the entire skill bank for every instance. Conference acronym ’X, June 03–05, 2018, Woodstock, NYHuang et al. selected Skill Bank Selected Skills Localized Skill Revision Replay-Gated Validation Listwise Output • Failure Attribution • Knowledge-Error Extraction • Structured Error Modeling • Recurrent Pattern Aggregation LLM Evolution Controller Evaluation Rules Production Feedback + Skill Router External Knowledge selected + Page-Level Final Label × Page-label alignment RelevanceQuality DiversityAuthority Skill-level alignment Logic Gating Term Stage 2: DAPO with Hierarchical Reward System Human-Aligned Label Stage 1: Human-Aligned Supervised Training Query-list Results pair query Creator Information Video Information routing description evaluation guidance Accepted Skill Update: Teacher Model Listwise Evaluator Initial Evaluator Figure 2: Overview of SEEK. The external skill bank stores explicit evaluation knowledge, while the skill router and listwise evaluator learn how to select and apply relevant knowledge for each query–list pair. Human-aligned supervision trains the two model components, whereas reviewed production feedback drives replay-gated evolution of the skill bank. Listwise Evaluator. The evaluator receives both the original query– list pair and the operational guidance associated with the selected skills: (b휋, b푦)= 퐸 휃 푞,퐿, n 푔 (푡) 푖 | 푆 (푡) 푖 ∈S (푡) 푥 o ,(5) where퐸 휃 denotes the listwise evaluator parameterized by휃. Rather than assessing retrieved items independently, the evaluator jointly reasons over the complete ranked list under the selected evaluation criteria. It produces a structured evaluation trajectoryb휋, which grounds the judgment in relevant evidence and skill-level diagnoses, together with the final predictionb푦= ( b 푙,b푎 푗 b푚 푗=1 ). Here, b 푙is the page-level quality label andb푎 푗 b푚 푗=1 denotes the corresponding structured attribution. 3.2 Skill-Aligned Training The skill router and listwise evaluator learn two complementary capabilities: the router determines which evaluation knowledge is relevant, while the evaluator learns how to apply the selected knowledge to the result page. We train the two components sepa- rately using supervision derived from the same human-annotated query–list pairs. Human judgments provide the final evaluation targets, while a strong teacher model transforms them into skill- routing labels and evidence-grounded evaluation trajectories. 3.2.1 Teacher-Guided Data Construction. Given a query–list pair (푞,퐿), the current skill bankS (푡) , and the human reference푦 ∗ = (푙 ∗ ,A ∗ ), the teacher first identifies a compact set of skills sufficient to support the annotated judgment. It produces a multi-label routing target푧 ∗ ∈ 0,1 푁 , which defines the reference skill setS ∗ 푥 = 푆 (푡) 푖 | 푧 ∗ 푖 =1. The corresponding operational guidance is denoted asG ∗ 푥 =푔 (푡) 푖 | 푆 (푡) 푖 ∈S ∗ 푥 . Conditioned on the selected guidance and the human reference, we employ a strong general-purpose LLM as the teacher model to generate a structured and evidence-grounded evaluation trajectory: 휋 ∗ =푇 LLM 푞,퐿,G ∗ 푥 ,푦 ∗ ,(6) where푇 LLM denotes the teacher model. The teacher provides high- quality reasoning supervision by connecting retrieved evidence to skill-level diagnoses and further synthesizing these diagnoses into the human-annotated page-level judgment and attribution. Each distilled instance is represented as(푞,퐿,푧 ∗ ,휋 ∗ ,푦 ∗ ). Before training, we filter samples with invalid skill references, malformed trajectory structures, unsupported evidence, or explicit inconsis- tencies with the human annotation. The verified routing target 푧 ∗ supervises the router, while(휋 ∗ ,푦 ∗ )provides skill-conditioned supervision for the evaluator. 3.2.2 Router Training. Because multiple evaluation criteria may be relevant to the same query–list pair, we formulate skill routing as a multi-label classification problem. For푁skills, the router is optimized using binary cross-entropy: L router =− 1 푁 푁 ∑︁ 푖=1 푧 ∗ 푖 log푝 푖 +(1−푧 ∗ 푖 ) log(1− 푝 푖 ) ,(7) where푝 푖 is the predicted activation score for skill푆 푖 . This objective directly aligns the router with the teacher-derived skill-selection targets. We optimize the router independently from downstream reinforcement learning. 3.2.3 Evaluator Training. We train the listwise evaluator in two stages. First, supervised fine-tuning teaches the model to repro- duce verified skill-conditioned evaluation trajectories and human judgments: L SFT =− log푃 휃 휋 ∗ ,푦 ∗ | 푞,퐿,G ∗ 푥 .(8) This stage establishes the basic capability to identify supporting evi- dence, apply the selected evaluation criteria, generate skill-grounded diagnoses, and synthesize them into a coherent page-level decision. SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial SearchConference acronym ’X, June 03–05, 2018, Woodstock, NY We then apply DAPO [45] to further align the evaluator with hu- man search judgments. For each query–list pair, we sample a group of candidate outputsO= 표 1 , . . .,표 푀 and score each candidate using a hierarchical reward: 푅(표)= ( 푅 task (표)+ 휆 푠 푅 skill (표) ) 푒 −휆 푐 퐶 conflict (표) ,(9) where푅 task measures the correctness of the final page-level judg- ment,푅 skill measures the quality of skill-grounded diagnosis and structured attribution, and퐶 conflict penalizes unsupported evidence and inconsistencies between local diagnoses and the final decision. Following DAPO, rewards within each sampled group are con- verted into group-relative advantages for policy optimization. We adopt its clip-higher strategy, which uses asymmetric clipping with a larger upper bound to preserve useful exploration, and dynamic sampling, which filters or resamples rollout groups with insufficient reward variation to maintain informative learning signals. These mechanisms improve the stability and efficiency of reinforcement learning while avoiding premature policy collapse. 3.3 Self-evolving Skill Bank The above training procedure produces the initial model before deployment, corresponding to evolution round푡=0. However, newly observed traffic and reviewed production feedback may re- veal evaluation patterns that are not fully covered by the initial evaluation knowledge. We address this adaptation problem through the self-evolving skill bank. SEEK enables lightweight knowledge adaptation by evolving the external skill bank without immediately updating the router or evaluator. During skill bank evolution, model checkpoints and routing descriptions푟 (푡) 푖 remain fixed, while only the operational guidance 푔 (푡) 푖 is eligible for revision. 3.3.1 Failure Attribution. A disagreement between the SEEK pre- dictionb푦and the human reference푦 ∗ is treated as a candidate evolution signal. A strong LLM serves as the evolution controller and jointly examines the query–list pair(푞,퐿), the selected skill set S (푡) 푥 , current skill guidance, model prediction, and human reference. We categorize failures into three types. A routing error occurs when the required knowledge already exists in the skill bank but the corresponding skill is not activated. A skill knowledge error occurs when the appropriate skill is selected but its operational guidance is incomplete or outdated. An execution error occurs when both routing and guidance are adequate, but the evaluator fails to apply them correctly. Only skill knowledge errors enter the fast evolution loop; routing and execution errors are accumulated for subsequent router or evaluator updates. 3.3.2 Recurrent Knowledge Revision. To avoid overfitting to iso- lated failures, SEEK revises a skill only when the same underlying knowledge gap recurs across multiple reviewed samples. For each skill푆 (푡) 푖 , the evolution controller groups knowledge errors that reflect the same missing evaluation principle into a recurrent error clusterB (푡) 푖 . It then performs a localized revision of the correspond- ing operational guidance: e푔 (푡+1) 푖 = Revise 푔 (푡) 푖 ,B (푡) 푖 , e 푆 (푡+1) 푖 = 푟 (푡) 푖 , e푔 (푡+1) 푖 .(10) The revision is intentionally localized. It may refine a decision boundary, evidence requirement, exception, or previously missing evaluation pattern, while preserving the routing semantics푟 (푡) 푖 and unrelated portions of the existing guidance. This allows new knowledge to be incorporated without unnecessarily changing when the skill is activated. 3.3.3Replay-Gated Deployment. Candidate revisions are validated before being deployed. For each revised skill, we evaluate the candi- date on a held-out adaptation setD new 푖 , which contains examples of the emerging pattern, and on a frozen historical replay setD replay . Both sets are associated with trusted human annotations. Let푀(D;푆 푖 )denote the evaluation performance obtained when skill 푆 푖 is used while all other system components remain fixed. A candidate revision is accepted only if it improves performance on the newly observed pattern while preserving historical performance within a tolerance 휖: Accept e 푆 (푡+1) 푖 = I 푀 D new 푖 ; e 푆 (푡+1) 푖 > 푀 D new 푖 ;푆 (푡) 푖 , 푀 D replay ; e 푆 (푡+1) 푖 ≥ 푀 D replay ;푆 (푡) 푖 −휖 . (11) Accepted candidates replace the previous skill version; otherwise, the existing skill is retained. In this way, SEEK separates adaptation across two timescales: explicit evaluation knowledge can be updated rapidly through the skill bank, while accumulated routing and execution errors are consolidated through slower periodic model updates. 4 Experiments We conduct extensive experiments to evaluate SEEK from four com- plementary perspectives. First, we compare its listwise evaluation performance against open-source LLMs, API-based LLMs, and task- specific post-trained LLMs. Second, we study the contribution of each major component, including dynamic skill routing, evaluator post-training, and hierarchical reward optimization. Third, we in- vestigate the continual-adaptation capability of the external skill bank under frozen model parameters. Finally, we evaluate the prac- tical effectiveness and efficiency of SEEK in a real online production environment. 4.1 Experimental Setup 4.1.1Dataset and Annotation. We construct the dataset from query– list pairs sampled from Kuaishou search logs. Each retrieved result contains information including the title, image caption, OCR and ASR text, creator information, interaction statistics, and ranking position. Each query–list pair is annotated with a page-level quality label from good, fair, bad. For fair and bad pages, annotators addition- ally provide attribution diagnoses over five dimensions: Relevance, Quality, Diversity, Authority, and Heterogeneous. A fair or bad page may correspond to multiple dimensions. Annotation disagreements are resolved by experienced search assessors. After cleaning and deduplication, we obtain 169,434 training query–list pairs and an independent held-out test set of 17,000 pairs. We apply stratified, label-aware filtering when constructing the held-out set. This procedure ensures sufficient representation of all three page-level quality labels and each of the five attribution dimensions. Detailed dataset statistics are reported in Appendix D. Conference acronym ’X, June 03–05, 2018, Woodstock, NYHuang et al. Table 1: Overall performance on listwise search quality evaluation. We report recall over five attribution dimensions, binary page-level evaluation, and three-class quality evaluation. Best results are shown in bold, and second-best results areunderlined. Method Attribution RecallBinaryThree-Class Rel. Qual. Div. Auth. Het.Macro-F1 Acc.Good F1 Fair F1 Bad F1 Macro-F1 Acc. Open-source LLMs Qwen3-8B [42]0.5566 0.5585 0.1624 0.2272 0.10460.43450.48940.31860.42170.45620.39880.4085 DeepSeek-R1-Distill-Qwen-7B [12] 0.6847 0.6416 0.3185 0.3027 0.19420.58880.59680.38100.46760.52680.45850.4706 Gemma-3-12B-IT [33]0.7168 0.6635 0.3642 0.3298 0.21870.61250.62160.39550.47810.54340.47230.4859 API-based LLMs Qwen3-235B-A22B [42]0.8078 0.6498 0.4450 0.4076 0.28320.64780.66760.42470.50460.56390.49770.5114 MiniMax-M2.5 [24]0.8256 0.6365 0.4413 0.3984 0.29130.67000.68660.43070.49890.59590.50850.5227 Gemini 3.1 Pro [6]0.8396 0.81140.5883 0.4897 0.39850.72410.73640.61980.57210.68040.62410.6326 GPT-5.6 Sol [25]0.85180.8037 0.6065 0.50780.41460.73580.74510.64270.57760.69320.63780.6469 Task-specific Post-trained LLMs Qwen3-8B (SFT+DPO) [22, 32]0.8015 0.7426 0.5638 0.3725 0.32610.68760.70680.46850.52810.61240.53630.5613 Qwen3-8B (SFT+GRPO) [21, 43, 46] 0.8462 0.8215 0.6189 0.4381 0.35840.72060.73860.65130.6107 0.66340.64180.6527 Qwen3-8B (SEEK)0.8621 0.8102 0.6210 0.5240 0.43050.7374 0.75200.67150.5586 0.7364 0.6555 0.6618 4.1.2 Baselines. We compare SEEK with three complementary groups of baselines. Open-source LLMs. We include Qwen3-8B [42], DeepSeek-R1- Distill-Qwen-7B [12], and Gemma-3-12B-IT [33]. These models represent compact general-purpose LLMs without task-specific adaptation to our listwise search evaluation task. API-based LLMs. We further compare with Qwen3-235B-A22B [42], MiniMax-M2.5 [24], Gemini 3.1 Pro [6], and GPT-5.6 Sol [25]. These models provide strong general-purpose judging capabilities at sub- stantially larger scales. All models receive the same query–list input, evaluation criteria, and output specification. Task-specific Post-trained LLMs. We additionally compare with representative parameter-level tuning using the same Qwen3-8B backbone. Direct Preference Optimization (DPO) [26] optimizes the model directly from preference supervision without training a sep- arate reward model, whereas Group Relative Policy Optimization (GRPO) [29] performs reinforcement learning with group-relative advantages computed from multiple sampled responses. Following prior search evaluation and relevance modeling meth- ods, we instantiate two task-adapted baselines: SFT followed by DPO [22,32], and SFT followed by GRPO [21,43,46]. These base- lines internalize task-specific evaluation knowledge through su- pervised fine-tuning followed by preference optimization or rein- forcement learning, providing controlled comparisons with SEEK’s externalized, skill-grounded knowledge design. 4.1.3Evaluation Metrics. We evaluate page-level prediction under both binary and three-class settings. For binary evaluation, good pages are treated as the positive class, while fair and bad pages are merged into the negative class. We report Macro-F1 and Accuracy, where Macro-F1 is computed as the unweighted average of the F1. For three-class evaluation, we retain the original good, fair, bad label space and report class-wise F1, Macro-F1, and overall Accuracy. To assess fine-grained attribution quality, we report Recall over the five attribution dimensions: Relevance, Quality, Diversity, Authority, and Heterogeneous. For continual adaptation, we additionally report Binary F1 and Attribution F1 to measure adaptation effectiveness and the quality of fine-grained diagnosis under evolving evaluation criteria. 4.1.4Implementation Details. The listwise evaluator is initialized from Qwen3-8B. All offline training experiments are conducted on NVIDIA H800 GPUs, while production inference is deployed on NVIDIA L20 GPU clusters. Additional implementation and opti- mization details are provided in Appendix B. 4.2 Overall Performance Table 1 summarizes the overall results on listwise search quality evaluation. SEEK consistently outperforms its Qwen3-8B backbone across page-level prediction and fine-grained attribution, with sub- stantial gains under both binary and three-class settings. This in- dicates that SEEK improves not only the detection of problematic result pages, but also the discrimination of more subtle quality differences among good, fair, and bad search experiences. Task-specific post-training substantially narrows the gap, mak- ing SFT+GRPO a strong same-backbone baseline. Nevertheless, SEEK further improves three-class Macro-F1 from 0.6418 to 0.6555 while also achieving better binary evaluation performance. This result suggests that externalizing evaluation knowledge and dynam- ically selecting relevant criteria provides complementary benefits beyond parameter-level adaptation. The advantage is particularly evident in fine-grained attribution. SEEK achieves the best results on four of the five dimensions, with the largest improvements over SFT+GRPO appearing on Authority and Heterogeneous. These dimensions depend more heavily on task- specific evaluation criteria and cross-result interactions, which are difficult to capture through a single static judging policy. The results therefore support the benefit of dynamically selecting and providing explicit skill guidance for each query–list pair. SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial SearchConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 2: Ablation of dynamic skill routing and evaluator post- training. All Skills provides the complete skill bank to every query–list pair without routing, whereas Dynamic selects an instance-specific subset using the skill router. Base denotes the original Qwen3-8B without task-specific post-training, and Trained denotes the trained listwise evaluator. ConfigurationAttribution Recall Acc. Routing EvaluatorRel. Qual. Div. Auth. Het. All Skills Base0.5566 0.5585 0.1624 0.2272 0.10460.4894 Dynamic Base0.7842 0.7146 0.4218 0.3518 0.27650.6485 All Skills Trained0.82840.76260.49870.41240.33630.6913 Dynamic Trained0.8621 0.8102 0.6210 0.5240 0.43050.7520 4.3 Ablation Study We conduct ablation studies to examine four key aspects of SEEK: the complementary contributions of the skill router and listwise evaluator, the effect of evaluator post-training, the hierarchical reward design, and the effectiveness of dynamic skill routing. 4.3.1Skill Router and Evaluator. We disentangle the effects of dy- namic skill routing and evaluator post-training in Table 2. When routing is disabled, the evaluator receives the complete Skill Bank; when evaluator training is disabled, the original Qwen3-8B back- bone is used. Dynamic routing consistently improves over exposing all skills, showing that instance-specific skill selection is preferable to indis- criminate knowledge injection. Evaluator post-training provides a further substantial gain by learning task-specific listwise eval- uation capabilities. Combining both components yields the best performance, confirming their complementary roles: the evaluator learns how to perform the task, while the router determines which external knowledge should be applied to each query–list pair. 4.3.2Evaluator Optimization. We examine the effect of evaluator post-training in Table 3, comparing the original Qwen3-8B back- bone, SFT alone, SFT followed by GRPO, and SFT followed by DAPO. SFT yields a substantial improvement over the base model, show- ing that teacher-generated structured trajectories provide effective supervision for learning listwise evaluation behavior. Subsequent reinforcement learning further improves both page-level predic- tion and fine-grained attribution, with DAPO achieving the best overall performance. These results suggest that SFT establishes the task-specific evaluation capability, while reinforcement learning further aligns the skill-conditioned evaluation process with the final page-level judgment. 4.3.3 Hierarchical Reward. We examine the hierarchical reward design in Table 4 by progressively adding skill-level supervision and the conflict penalty to the task-level reward. Using푅 task alone already provides strong page-level supervision, but yields weaker fine-grained attribution. Adding푅 skill consistently improves diagnostic quality, while퐶 conflict further encourages con- sistency between skill-level diagnoses and the final judgment. The full objective performs best overall, indicating that reliable listwise evaluation benefits from jointly optimizing prediction correctness, skill-grounded diagnosis, and cross-level consistency. Table 3: Ablation of evaluator optimization. “+” indicates that the corresponding training stage is enabled. EvaluatorAttribution Recall Acc. SFT GRPO DAPORel. Qual. Div. Auth. Het. –0.5566 0.5585 0.1624 0.2272 0.10460.4894 +–0.8332 0.7886 0.5158 0.4533 0.38000.6251 ++–0.85470.80160.58650.49670.41270.7385 +–+0.8621 0.8102 0.6210 0.5240 0.43050.7520 Table 4: Ablation of the hierarchical reward in SEEK. “+” in- dicates that the corresponding reward component is enabled. RewardAttribution Recall Acc. 푅 task 푅 skill 퐶 conflict Rel. Qual. Div. Auth. Het. +–0.8502 0.7950 0.5581 0.4746 0.39640.7126 ++–0.85750.80270.59410.50180.41600.7382 +++0.8621 0.8102 0.6210 0.5240 0.43050.7520 Table 5: Comparison of skill-access strategies. Oracle Routing uses reference skill annotations as an upper bound. Skill Access Evaluation Quality Avg. Skills Acc.Attr. F1 All Skills0.6910.66412.0 Dynamic Routing0.7520.7433.4 Oracle Routing0.7680.7583.1 4.3.4Dynamic Skill Routing. We compare three skill-access strate- gies in Table 5. Dynamic Routing substantially outperforms indis- criminate access to the entire skill bank, while requiring only a small fraction of the available skills for each instance. This indi- cates that instance-specific selection reduces unnecessary context and potential criterion interference. The remaining gap to Oracle Routing suggests that better skill selection could provide further gains. 4.4 Continual Adaptation We evaluate whether SEEK can adapt to emerging evaluation cri- teria by updating the external skill bank. The experiments are conducted on a newly collected post-deployment test set whose distribution differs from the held-out benchmark used for offline evaluation. This dataset better reflects the evolving query patterns and diagnostic dimensions encountered in real production environ- ments. We examine both adaptation to newly observed patterns and stability on historical data. 4.4.1 Knowledge and Parameter Adaptation. Figure 3 compares two strategies under the same sequence of reviewed production feedback. Model-only Update fine-tunes the evaluator while keeping the skill bank fixed, whereas Skill-only Evolution updates only the skill bank with both model components frozen. Both strategies improve consistently as reviewed samples are in- corporated. After three rounds, Model-only Update increases Binary Conference acronym ’X, June 03–05, 2018, Woodstock, NYHuang et al. InitialRound 1Round 2Round 3 0.74 0.75 0.76 0.77 0.78 0.79 0.80 0.81 0.82 0.83 0.84 F1 Score 0.7814 0.7926 0.7941 0.7957 0.7887 0.7908 0.7921 0.7458 0.7548 0.7572 0.7594 0.7504 0.7531 0.7553 Model-only / Binary F1 Skill-only / Binary F1 Model-only / Attribution F1 Skill-only / Attribution F1 Figure 3: Multi-round comparison of parameter-based adap- tation and Skill-only Evolution. Both strategies are evaluated using Binary F1 and Attribution F1 under the same reviewed evolution data. F1 from 0.7814 to 0.7957 and Attribution F1 from 0.7458 to 0.7594. Without modifying either model component, Skill-only Evolution reaches 0.7921 Binary F1 and 0.7553 Attribution F1. The remaining gaps to parameter-based adaptation are only 0.0036 and 0.0041, respectively, showing that a substantial fraction of the adaptation benefit can be recovered by updating explicit evaluation knowledge alone. The largest gains occur in the first evolution round, followed by progressively smaller improvements in subsequent rounds. This trend is consistent with recurrent and reusable knowledge gaps being absorbed early, while later rounds mainly capture more spe- cialized patterns. Overall, the capability–knowledge separation underlying SEEK provides a lightweight fast path for incorporating new evaluation knowledge without frequent model retraining. 4.4.2Replay Stability. We further evaluate backward stability on a frozen historical replay set after the same three adaptation rounds. As shown in Table 6, Skill-only Evolution largely preserves his- torical performance, with only 0.0008 Binary-F1 forgetting and 0.0005 Attribution-F1 forgetting. In contrast, Model-only Update causes noticeably larger degradation, suggesting that parameter adaptation introduces greater interference with previously learned evaluation behavior. Together with the adaptation results above, this demonstrates the advantage of SEEK’s design: Skill-only Evolution provides a fast and backward-stable path for absorbing new evaluation knowledge, while parameter updates can be less frequent. 4.5 Production Evaluation Finally, we evaluate SEEK in a real-world production search evalu- ation workflow. Table 7 compares SEEK with outsourced human review and multi-round human assessment on 24,000 query–list pairs, using expert-calibrated judgments as the reference labels. Table 6: Historical replay stability after three adaptation rounds. Both strategies use the same reviewed feedback se- quence and frozen replay set. Forgetting is the F1 decrease relative to the initial system. StrategyBinary F1 Forget. Attr. F1 Forget. Initial System0.7862–0.7481– Model-only Update0.77860.00760.73970.0084 Skill-only Evolution0.78540.00080.74760.0005 Table 7: Production evaluation on 24,000 online query–list pairs. Accuracy is measured against expert-calibrated refer- ence labels, and speedup is computed relative to outsourced human review. EvaluatorTime↓ Speedup↑ Accuracy↑ Outsourced Review36.5 h1.0×0.783 Multi-round Review58.2 h0.6×0.830 SEEK0.9 h40.6×0.842 SEEK achieves a favorable balance between judgment quality and efficiency. It reaches an Accuracy of 0.842, outperforming both out- sourced review (0.783) and multi-round assessment (0.830), while completing the entire workload in only 0.9 hours. Compared with outsourced review, this corresponds to a 40.6×speedup. Notably, although multi-round assessment improves human review qual- ity, it incurs substantially higher time cost, whereas SEEK attains stronger agreement with expert-calibrated labels at only a fraction of the evaluation time. Beyond page-level predictions, SEEK also produces structured attribution that identify the quality dimensions underlying each judgment. These diagnostic signals can directly support large-scale quality monitoring, failure analysis, and downstream search-system iteration. Therefore, SEEK can serve as a scalable and practical eval- uation component for continuous industrial search development. 5 Conclusion In this paper, we propose SEEK, a skill-grounded framework for in- dustrial listwise search evaluation that decouples evolving domain knowledge from relatively stable model capabilities. SEEK exter- nalizes heterogeneous evaluation criteria into a modular skill bank, dynamically routes task-relevant skills for each query–list pair, and employs a task-adapted listwise evaluator to produce page-level judgments and structured attribution. A human-aligned training pipeline specializes the skill router and evaluator, while the self- evolving skill bank enables rapid knowledge adaptation without requiring immediate model retraining. Extensive offline, continual-adaptation, and production experi- ments show that SEEK improves listwise judgment and fine-grained diagnosis, supports effective adaptation through external knowl- edge updates with frozen model parameters, and substantially im- proves evaluation efficiency in production. Overall, the results demonstrate the value of separating evolvable evaluation knowl- edge from model reasoning capabilities, providing a practical path toward more scalable and adaptable industrial search evaluation. SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial SearchConference acronym ’X, June 03–05, 2018, Woodstock, NY References [1]Negar Arabzadeh and Charles LA Clarke. 2025. A human-ai comparative analysis of prompt sensitivity in llm-based relevance judgment. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2784–2788. [2]Krisztian Balog, Don Metzler, and Zhen Qin. 2025. Rankers, judges, and assistants: Towards understanding the interplay of LLMs in information retrieval evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3865–3875. [3]Mehmet Selman Baysan, Serkan Uysal, İrem İşlek, Çağla Çığ Karaman, and Tunga Güngör. 2025. LLM-as-a-Judge: automated evaluation of search query parsing using large language models. Frontiers in big Data 8 (2025), 1611389. [4] Tao Chen, Gangwei Jiang, Pengyu Cheng, Siyuan Huang, Yihao Liu, Jingwei Ni, Jiaqi Guo, Mengyu Zhou, Kai Tang, Junling Liu, et al.2026. Skill-RM: Unifying Het- erogeneous Evaluation Criteria via Agent Skill. arXiv preprint arXiv:2606.03980 (2026). [5]Yoonseo Choi, Eunhye Kim, Hyunwoo Kim, Donghyun Park, Honggu Lee, Jin Young Kim, and Juho Kim. 2025. BloomIntent: Automating Search Eval- uation with LLM-Generated Fine-Grained User Intents. In Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. 1–34. [6] Google DeepMind. 2026. Gemini 3.1 Pro Model Card. https://deepmind.google/ models/model-cards/gemini-3-1-pro/. Accessed: 2026-09-20. [7] Mouly Dewan, Jiqun Liu, Aditya Gautam, and Chirag Shah. 2026. LLM-driven usefulness judgment for web search evaluation. In Proceedings of the 2026 Interna- tional ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR). 44–55. [8]Mouly Dewan, Jiqun Liu, and Chirag Shah. 2025. LLM-driven usefulness labeling for IR evaluation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3055–3059. [9]Chen Dun, Mirian Del Carmen Hipolito Garcia, Guoqing Zheng, Ahmed Hassan Awadallah, Robert Sim, and Anastasios Kyrillidis. 2025. Sweeping heterogeneity with smart mops: Mixture of prompts for LLM task adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 16426–16434. [10]Naghmeh Farzi and Laura Dietz. 2025. Criteria-Based LLM Relevance Judgments. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR). 254–263. [11] Tian Guan, Sebastian Sun, and Bolin Chen. 2025. Faithfulness-aware multi- objective context ranking for retrieval-augmented generation. In Proceedings of the 2025 3rd International Conference on Artificial Intelligence, Systems and Network Security. 119–126. [12]Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incen- tivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025). [13]Siyuan Huang, Pengyu Cheng, Haotian Liu, Tao Chen, Yihao Liu, Jingwei Ni, Shijie Zhou, Ziyi Yang, Gangwei Jiang, Mengyu Zhou, et al.2026. Skill Self-Play: Pushing the frontier of LLM capability with co-evolving skills. arXiv preprint arXiv:2607.22529 (2026). [14] Anton Korikov, Pan Du, Scott Sanner, and Navid Rekabsaz. 2025. Batched self- consistency improves llm relevance assessment and ranking. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 32675– 32691. [15] Pratik Lahiri, Bingqing Ge, Zhou Qin, Aditya Jumde, Shuning Huo, Lucas Scottini, Yi Liu, Mahmoud Mamlouk, and Wenyang Liu. 2026. Design and Evaluation of Whole-Page Experience Optimization for E-commerce Search. In Proceedings of the Nineteenth ACM International Conference on Web Search and Data Mining. 1175–1179. [16]Benjamin H Le, Xueying Lu, Nicholas Stern, Wenqiong Liu, Igor Lapchuk, Xiang Li, Baofen Zheng, Kevin Rosenberg, Jiewen Huang, Zhe Zhang, et al.2026. SAGE: Scalable AI Governance & Evaluation. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 7555–7565. [17]Will LeVine and Bijan Varjavand. 2025. Relevance Isn’t All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking. arXiv preprint arXiv:2504.07104 (2025). [18]Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579 (2024). [19]Qingyang Liu, Jiangtong Li, Zelin Peng, Shaobo Wang, Zhaohe Liao, Shuochen Chang, Bingjie Gao, Haonan Zhao, Mu Liu, Jidong Jiang, et al.2026. Bridging visual dynamics and narrative reasoning: Multimodal large language models for short drama quality assessment. In Proceedings of the ACM Web Conference 2026. 7890–7901. [20]Yuxuan Liu, Zhaochen Su, Lingyun Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, et al.2026. SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision. arXiv preprint arXiv:2606.01139 (2026). [21]Chenji Lu, Zhuo Chen, Hui Zhao, Zhiyuan Zeng, Gang Zhao, Junjie Ren, Ruicong Xu, Haoran Li, Songyan Liu, Pengjie Wang, et al.2025. LORE: A Large Generative Model for Search Relevance. arXiv preprint arXiv:2512.03025 (2025). [22] Xingyu Lu, Tianke Zhang, Chang Meng, Xiaobei Wang, Jinpeng Wang, Yi-Fan Zhang, Shisong Tang, Changyi Liu, Haojie Ding, Kaiyu Jiang, et al.2025. Vlm as policy: Common-law content moderation framework for short video platform. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 4682–4693. [23] Yuchen Ma, Yue Huang, Han Bao, Haomin Zhuang, Swadheen Shukla, Michel Gal- ley, Xiangliang Zhang, and Stefan Feuerriegel. 2026. SkillGen: Verified Inference- Time Agent Skill Synthesis. arXiv preprint arXiv:2605.10999 (2026). [24]MiniMax. 2026. MiniMax M2.5: Built for Real-World Productivity. https://w. minimaxi.com/blog/minimax-m25. Accessed: 2026-09-20. [25]OpenAI. 2026. GPT-5.6: Frontier intelligence that scales with your ambition. https://openai.com/index/gpt-5-6/. Accessed: 2026-09-20. [26]Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems, Vol. 36. [27]Hossein A Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles LA Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggi- oli. 2024. Llmjudge: Llms for relevance judgments. arXiv preprint arXiv:2408.08896 (2024). [28]Revanth Gangi Reddy, JaeHyeok Doo, Yifei Xu, Md Arafat Sultan, Deevya Swain, Avirup Sil, and Heng Ji. 2024. FIRST: Faster improved listwise reranking with single token decoding. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 8642–8652. [29] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv preprint arXiv:2402.03300 (2024). [30] Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, An- drew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, and Jimmy Lin. 2025. Rankllm: A python package for reranking with llms. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3681–3690. [31] Mingxuan Song, Yusen Huo, Bohan Zhou, Shenglin Yin, Zhen Xiao, Jieyi Long, Zhilin Zhang, and Chuan Yu. 2026. DARA: Few-shot Budget Allocation in Online Advertising via In-Context Decision Making with RL-Finetuned LLMs. In Proceedings of the ACM Web Conference 2026. 40–50. [32] Tian Tang, Zhixing Tian, Zhenyu Zhu, Chenyang Wang, Haiqing Hu, Guoyu Tang, Lin Liu, and Sulong Xu. 2025. Lref: A novel llm-based relevance framework for e-commerce search. In Companion Proceedings of the ACM on Web Conference 2025. 468–475. [33]Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025). [34] Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large language models can accurately predict searcher preferences. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1930–1940. [35]Giovanni Trappolini, Florin Cuconasu, Simone Filice, Yoelle Maarek, and Fabrizio Silvestri. 2026. Redefining Retrieval Evaluation in the Era of LLMs. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 8359–8375. [36]Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor. arXiv preprint arXiv:2406.06519 (2024). [37]Ali Vardasbi, Gustavo Penha, Enrico Palumbo, Claudia Hauff, Hugues Bouchard, and Mounia Lalmas. 2026. As It Was: Aligning LLM Search Evaluation with Historical User Preferences. arXiv preprint arXiv:2607.01040 (2026). [38]Chenxi Wang, Zhuoyun Yu, Xin Xie, Wuguannan Yao, Runnan Fang, Shuofei Qiao, Kexin Cao, Guozhou Zheng, Xiang Qi, Peng Zhang, and Shumin Deng. 2026. SkillX: Automatically Constructing Skill Knowledge Bases for Agents. arXiv preprint arXiv:2604.04804 (2026). [39]Han Wang, Alex Whitworth, Pak Ming Cheung, Zhenjie Zhang, and Krishna Kamath. 2025. LLM-based Relevance Assessment for Web-Scale Search Evaluation at Pinterest. arXiv preprint arXiv:2509.03764 (2025). [40] Jiongxiao Wang, Qiaojing Yan, Yawei Wang, Yijun Tian, Soumya Smruti Mishra, Zhichao Xu, Megha Gandhi, Panpan Xu, and Lin Lee Cheong. 2026. Reinforcement Learning for Self-Improving Agent with Skill Library. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics. 1529–1550. [41]Xingzhu Wang, Erhan Zhang, Yiqun Chen, Jinghan Xuan, Yucheng Hou, Yitong Xu, Ying Nie, Shuaiqiang Wang, Dawei Yin, and Jiaxin Mao. 2025. CLUE: Using Large Language Models for Judging Document Usefulness in Web Search Evalu- ation. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management. 3133–3143. Conference acronym ’X, June 03–05, 2018, Woodstock, NYHuang et al. [42]An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025). [43] Jianhui Yang, Yiming Jin, Pengkun Jiao, Chenhe Dong, Zerui Huang, Shaowei Yao, Xiaojiang Zhou, Dan Ou, and Haihong Tang. 2025. TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance. arXiv preprint arXiv:2510.08048 (2025). [44]Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. 2026. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv preprint arXiv:2605.23904 (2026). [45]Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al.2025. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems 38 (2025), 113222–113244. [46]Ziyang Zeng, Heming Jing, Jindong Chen, Xiangli Li, Hongyu Liu, Yixuan He, Zhengyu Li, Yige Sun, Zheyong Xie, Yuqing Yang, et al.2026. Optimizing Gen- erative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1. 2551–2561. [47]ChengXiang Zhai. 2024. Large language models and future of information retrieval: opportunities and challenges. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval. 481– 490. [48]Zishuai Zhang, Sihao Yu, Ying Nie, Junfeng Wang, Zhiming Zheng, Dawei Yin, Hainan Zhang, et al.2026. An Efficient Framework for Whole-Page Reranking via Single-Modal Supervision. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (ACL 2026). 966–978. [49]Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Zican Dong, Yupeng Hou, Beichen Zhang, Yingqian Min, Junjie Zhang, Peiyu Liu, et al.2026. A survey of large language models. Frontiers of Computer Science 20, 12 (2026), 2012627. [50] Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2025. Large language models for information retrieval: A survey. ACM Transactions on Information Systems 44, 1 (2025), 1–54. A Skill Bank Details SEEK externalizes task-specific evaluation knowledge into a mod- ular skill bank. In our implementation, the skill bank contains 12 diagnostic skills covering five major dimensions of industrial search quality: Relevance, Quality, Diversity, Authority, and Heterogeneous. Rather than treating each dimension as a single coarse criterion, we decompose it into more specific skills that capture recurring evaluation principles and decision boundaries. Table 8 summarizes the skill taxonomy used in our experiments. Skill Representation. As introduced in Section 3.1, each skill is represented as푆 푖 =(푟 푖 ,푔 푖 ), where푟 푖 is a compact routing descrip- tion and푔 푖 is the corresponding operational guidance. The two components provide different views of the same evaluation crite- rion. The routing description specifies when a skill is applicable and is used by the skill router for instance-specific selection. In con- trast, the operational guidance specifies how the criterion should be applied and is exposed to the listwise evaluator only after the skill is activated. Operational guidance is structured around five types of infor- mation: the evaluation objective, required evidence, decision rules, boundary conditions, and attribution requirements. The objective defines the quality property being assessed; evidence requirements specify which signals in the query–list pair should support the judgment; decision rules describe how the evidence should be in- terpreted; boundary conditions prevent the criterion from being applied outside its intended scope; and attribution requirements specify how detected problems should be connected to the final page-level diagnosis. This representation keeps routing descrip- tions compact while allowing the evaluator to access sufficiently detailed and actionable knowledge. Example Skill: SourceAuthorityCheck. As a representative exam- ple, the routing description of SourceAuthorityCheck activates the skill when satisfying the query requires professional, authoritative, or high-confidence information. Its operational guidance asks the evaluator to determine whether the credibility of retrieved sources matches the level of expertise required by the user intent. Rele- vant evidence includes creator identity, professional credentials, institutional affiliation, source reputation, supporting evidence, and ranking position. For professional or high-stakes informational needs, stronger source credibility is required, and popularity or interaction statistics alone should not be treated as sufficient evi- dence of authority. Conversely, professional authority should not be imposed on ordinary entertainment or personal-experience queries unless the query explicitly requires such expertise. When an au- thority issue is identified, the evaluator attributes the problem to the affected results and explains its impact on the overall quality of the result page. This decomposition also allows multiple skills to be activated for the same query–list pair. For example, a result containing an unsup- ported professional claim may simultaneously require relevance checking, factual-reliability assessment, and source-authority veri- fication. The multi-label router therefore selects a compact combi- nation of complementary skills rather than assigning each instance to a single evaluation category. B Implementation Details Model Configuration. SEEK uses Qwen3-0.6B as the skill router and Qwen3-8B as the listwise evaluator. Qwen3-235B-A22B is used as the teacher model for routing-target generation, structured tra- jectory construction, and skill evolution. The evaluator supports a maximum input length of 16,384 tokens and a maximum output length of 4,096 tokens. Each query–list instance includes title, image caption, OCR/ASR text, creator information, interaction statistics, and ranking position. Skill Router. The router is trained as a 12-way multi-label classi- fier using teacher-derived routing targets. We optimize it for three epochs with AdamW, a learning rate of 1×10 −5 , weight decay of 0.01, and an effective batch size of 32. The activation threshold is set to휏=0.5; if no skill exceeds the threshold, the highest-scoring skill is retained. This setting activates approximately three to four skills per query–list pair. Evaluator Training. The evaluator is first trained with SFT for two epochs using a learning rate of 2×10 −5 , batch size 4, and an effective batch size of 32. We then apply DAPO with a learning rate of 1×10 −6 and a group size of 6. Rollouts use temperature 0.8 and top-푝=0.95. For the hierarchical reward, we set휆 푠 =0.5 and 휆 푐 =1.0. All training uses BF16 precision, and loss is applied only to assistant-generated tokens. Skill Evolution and Inference. During skill-only evolution, the router and evaluator remain frozen and only the operational guid- ance푔 푖 is revised. A knowledge gap is considered recurrent when it SEEK: Skill-Routed Evaluation with Evolvable Knowledge for Industrial SearchConference acronym ’X, June 03–05, 2018, Woodstock, NY Table 8: Taxonomy of the 12 diagnostic skills used in SEEK. Each skill captures a reusable evaluation criterion within one high-level search-quality dimension. DimensionSkillEvaluation Focus Relevance MainIntentMatchWhether retrieved content matches the dominant intent expressed by the query. TaskNeedSatisfactionWhether the result page satisfies the underlying user need rather than merely exhibiting lexical or topical overlap. RankingFaithfulnessWhether highly relevant and useful results are placed at appropriate positions relative to weaker results. Quality InformationValueCheckWhether retrieved results provide substantive and useful information rather than low- value, vague, or weakly informative content. FactualReliabilityCheckWhether factual or professional claims are sufficiently supported by reliable evidence and avoid unsupported assertions. RiskComplianceCheck Whether risky, misleading, or otherwise problematic content introduces quality or compliance concerns. Diversity SourceDiversityCheck Whether the result page avoids excessive concentration on the same or highly similar content sources. TemplateDuplicationWhether multiple results exhibit substantial semantic, structural, or template-level redundancy. SubIntentCoverageCheckWhether distinct and plausible sub-intents of the query receive adequate coverage across the result page. Authority SourceAuthorityCheck Whether source credibility and authority match the level of expertise required by the query. EvidenceProfessionalismWhether professional or high-confidence information is supported by appropriate credentials, evidence, or domain expertise. HeterogeneousHeterogeneousFulfillmentWhether heterogeneous result types and information sources complement one another in satisfying complex user needs. appears in at least three reviewed samples. Candidate revisions are validated on both emerging cases and a frozen replay set of 5,000 historical query–list pairs, with replay tolerance휖=0.002. Offline training is conducted on NVIDIA H800 GPUs, while production inference runs on NVIDIA L20 GPU clusters. The final evaluator uses deterministic decoding. C Qualitative Analysis We provide a qualitative case study to illustrate how SEEK converts recurrent production failures into reusable evaluation knowledge. Rather than storing individual failure cases as additional demonstra- tions, the self-evolving skill bank identifies their shared evaluation principle and performs a localized revision of the corresponding operational guidance. Representative Skill Evolution Case. We consider a recurrent pat- tern involving health-related search results. In several reviewed query–list pairs, the evaluator correctly activated skills related to factual reliability and source authority, but still assigned overly favorable judgments to results containing strong health claims. In- spection by the evolution controller attributed these failures to a skill knowledge error: the relevant skills were correctly selected, while the existing operational guidance did not sufficiently distin- guish ordinary informational claims from high-stakes claims that require stronger evidence and source credibility. Table 9 summarizes the resulting knowledge revision. The orig- inal guidance was not replaced wholesale. Instead, the Evolution Controller introduced a localized decision rule requiring stronger evidential support for medical or other high-stakes claims, while preserving the original routing semantics and unrelated evaluation rules. Analysis. This case highlights three properties of the evolution mechanism. First, SEEK evolves evaluation knowledge rather than merely memorizing newly observed samples. Multiple local failures are abstracted into a reusable decision rule that can generalize to future query–list pairs exhibiting the same underlying pattern. Second, the update is localized to the operational guidance푔 푖 ; the routing description푟 푖 remains unchanged because the original skill activation behavior is already correct. Finally, replay-gated validation constrains the revision from improving newly emerging cases at the expense of previously stable traffic. The example therefore illustrates the intended two-timescale adaptation of SEEK. Recurrent knowledge gaps can be addressed rapidly through explicit skill bank revisions, whereas failures caused by incorrect routing or improper skill execution are accumulated for slower periodic updates of the router or evaluator. D Annotation Protocol and Dataset Statistics Our annotation protocol follows the production search-quality as- sessment process for short-video search. Given a query푞and its ranked result list퐿= [푑 1 , . . .,푑 퐾 ], annotators evaluate the result page as a whole rather than independently judging each retrieved item. They jointly consider result-level evidence, ranking positions, Conference acronym ’X, June 03–05, 2018, Woodstock, NYHuang et al. Table 9: Representative localized revision during Skill-only Evolution. StageDescription Recurrent FailureSeveral reviewed results make strong health- related claims but provide insufficient sup- porting evidence or source credibility, while the evaluator judges them too favorably. Failure AttributionThe appropriate factual-reliability and author- ity skills are already activated, indicating a skill knowledge error rather than a routing error. Knowledge GapThe existing guidance does not explicitly re- quire stronger evidence standards for medical or other high-stakes claims. Localized RevisionFor high-stakes claims, require stronger sup- porting evidence and credible sources; popu- larity or interaction signals alone should not be treated as sufficient evidence of reliability or authority. Replay ValidationThe candidate revision is evaluated on both newly reviewed cases and the frozen histor- ical replay set, and is deployed only if it im- proves the emerging pattern without exceed- ing the allowed replay degradation. and cross-result interactions, such as redundancy, intent coverage, source composition, and the overall usefulness of the result page. Page-level Quality Labels. Each query–list pair is assigned one of three quality labels: good, fair, or bad. A good page adequately sat- isfies the dominant user intent and contains no material defect that substantially affects the overall search experience. A fair page re- mains generally useful but contains noticeable quality issues, such as several weakly relevant results, moderate redundancy, insuffi- cient intent coverage, or localized content-quality problems. A bad page contains severe or systematic defects that substantially impair the overall search experience, such as major intent mismatch, per- vasive low-quality content, serious ranking problems, or multiple interacting deficiencies. Importantly, the final label is not determined solely by the num- ber of problematic results. Annotators additionally consider the severity of each issue, its ranking position, the fraction of the page affected, and whether multiple problems jointly degrade the overall user experience. The distinction between fair and bad therefore reflects both the scope and the severity of the observed deficiencies. Annotation Procedure. Annotators are provided with the query, the complete ranked result list, and the signals available to the production evaluator, including titles, image captions, OCR and ASR text, creator information, interaction statistics, and ranking positions. They first infer the dominant user intent and inspect whether the result page satisfies that intent. They then examine individual results and cross-result interactions, assign the page-level quality label, and, for fair or bad pages, identify the dimensions responsible for the degradation together with supporting evidence. Table 10: Semantics of the page-level quality labels. Label Definition GoodThe result page adequately satisfies the search intent and contains no material issue that substantially af- fects the overall experience. Fair The page is generally useful, but contains noticeable, localized, or moderate quality problems. BadSevere or systematic problems substantially degrade the overall search experience. Table 11: Statistics of the training and held-out evaluation sets. Binary evaluation treats Good as the positive class and Fair∪Bad as the negative class. Attribution positives are counted at the page level over non-Good pages and are not mutually exclusive across dimensions. Split / CategorySamples% Training set169,434– Held-out test set17,000100.0 Page-level labels in the test set Good6,34237.3 Fair2,81216.5 Bad7,84646.2 Binary grouping used in Sec. 4.1 Positive (Good)6,34237.3 Negative (Fair∪ Bad)10,65862.7 Attribution positives Relevance5,63652.9 Quality5,74253.9 Diversity5,33250.0 Authority5,96756.0 Heterogeneous5,78454.3 When multiple issues coexist, annotators are instructed to record all major dimensions that materially contribute to the final judg- ment rather than forcing each page into a single error category. Ambiguous or conflicting cases are reviewed by experienced search assessors, whose adjudicated decisions are used as the final refer- ence annotations for model training and evaluation. Dataset Statistics. Table 11 reports the detailed statistics of the training and held-out evaluation sets. The test set contains 17,000 query–list pairs and is used exclusively for offline evaluation. For binary evaluation, good pages are treated as the positive class, while fair and bad pages are merged into the negative class. Attribution is formulated as a multi-label diagnosis task; therefore, a single page may be associated with multiple attribution dimensions.