Paper deep dive
AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access
Liwei Wu, Cho-Jui Hsieh
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/26/2026, 1:33:51 AM
Summary
The paper introduces an 'AI Co-Scientist' framework that automates the search ranking research pipeline, including idea generation, code implementation, and GPU experiment scheduling. By utilizing multi-LLM consensus agents (GPT-5.2, Gemini Pro 3, Claude Opus 4.5) and an expert-in-the-loop paradigm, the system successfully discovered a novel ranking architecture (V3.5) that outperforms human-designed baselines in offline metrics.
Entities (6)
Relation Signals (3)
Liwei Wu → authored → AI Co-Scientist for Ranking
confidence 100% · AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models... Liwei Wu1, Cho-Jui Hsieh2
AI Co-Scientist → discovered → V3.5
confidence 100% · the AI Co-Scientist first attempts to add positional encodings... It finally comes up with the model V3.5
AI Co-Scientist → utilizes → GPT-5.2
confidence 100% · leveraging multi-LLM consensus agents (GPT-5.2, Gemini Pro 3, and Claude Opus 4.5)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Recent advances in AI agents for software engineering and scientific discovery have demonstrated remarkable capabilities, yet their application to developing novel ranking models in commercial search engines remains unexplored. In this paper, we present an AI Co-Scientist framework that automates the full search ranking research pipeline: from idea generation to code implementation and GPU training job scheduling with expert in the loop. Our approach strategically employs single-LLM agents for routine tasks while leveraging multi-LLM consensus agents (GPT 5.2, Gemini Pro 3, and Claude Opus 4.5) for challenging phases such as results analysis and idea generation. To our knowledge, this is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research. We demonstrate that this framework discovered a novel technique for handling sequence features, with all model enhancements produced automatically, yielding substantial offline performance improvements. Our findings suggest that AI systems can discover ranking architectures comparable to those developed by human experts while significantly reducing routine research workloads.
Tags
Links
- Source: https://arxiv.org/abs/2603.22376v1
- Canonical: https://arxiv.org/abs/2603.22376v1
Trouble viewing inline? Open PDF directly →
Full Text
23,310 characters extracted from source content.
Expand or collapse full text
AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access Liwei Wu1, Cho-Jui Hsieh2 1Trip.com Group, 2UCLA Correspondence: w@wsw.ai, chohsieh@cs.ucla.edu Abstract Recent advances in AI agents for software engineering and scientific discovery have demonstrated remarkable capabilities, yet their application to developing novel ranking models in commercial search engines remains unexplored. In this paper, we present an AI Co-Scientist framework that automates the full search ranking research pipeline—from idea generation to code implementation and GPU training job scheduling with expert in the loop. Our approach strategically employs single-LLM agents for routine tasks while leveraging multi-LLM consensus agents (GPT-5.2, Gemini Pro 3, and Claude Opus 4.5) for challenging phases such as results analysis and idea generation. To our knowledge, this is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research. We demonstrate that this framework discovered a novel technique for handling sequence features, with all model enhancements produced automatically, yielding substantial offline performance improvements. Our findings suggest that AI systems can discover ranking architectures comparable to those developed by human experts while significantly reducing routine research workloads. AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access Liwei Wu1, Cho-Jui Hsieh2 1Trip.com Group, 2UCLA Correspondence: w@wsw.ai, chohsieh@cs.ucla.edu 1 Introduction Recent years have seen remarkable progress in AI agents for software engineering and scientific discovery, particularly in drug discovery and materials science. However, applying these general-purpose agents to commercial search ranking remains largely overlooked. While tech giants employ vast teams to manually balance personalization with business objectives, the community has yet to seriously consider automating the process—mirroring a broader scientific skepticism regarding AI’s utility in guiding research Gottweis et al. (2025). Figure 1: AI agent scientific discovery workflow Figure 2: AI Co-Scientist Performance Gain against H20 GPU Hours, where 0.1% gain in this eval metric is statistically significant and usually translates into 0.1% lift in Conversion Rate and millions of dollars in online experiments based on previous experiences. In this paper, we explore the feasibility of an AI Co-Scientist framework for ranking model research. By granting LLM-based agents access to cloud computing infrastructure, we fully automate the research pipeline from idea generation to code implementation and GPU training job scheduling with expert in the loop as shown in Fig 1. Our approach employs single-LLM agents for routine tasks such as code implementation, while leveraging multi-LLM consensus agents (GPT-5.2, Gemini Pro 3, and Claude Opus 4.5) for challenging tasks including results analysis and idea generation. To our knowledge, this is the first study in the ranking community to use an AI Co-Scientist framework for algorithmic research. We show that this framework discovered a new technique for handling sequence features: all model enhancements from V2 to V3.5 in Table 1 (illustrated in Fig 3) were produced automatically, resulting in substantial offline performance improvements. 2 Related Work 2.1 AI Agents for Scientific Discovery Lu et al. Lu et al. (2024) introduce the AI Scientist, an end-to-end framework where LLM-based agents autonomously generate ideas, run experiments, write papers, and perform peer review. Demonstrated across three ML subfields, it produces papers for roughly $15 each that reviewers rate at top-tier acceptance thresholds. Our work shares this end-to-end automation goal but focuses on discovering novel search ranking models rather than paper generation, with agents tightly integrated with cloud infrastructure for large-scale experimentation. Gottweis et al. Gottweis et al. (2025) propose AI Co-Scientist, a Gemini 2.0-based multi-agent system for biomedical discovery. Specialized agents iteratively generate and refine hypotheses, with wet-lab validated results rated more novel than baselines. We adopt their “expert-in-the-loop” paradigm but extend beyond hypothesis generation: our agents autonomously implement code, launch GPU experiments, monitor results, and perform analysis, with humans providing high-level steering rather than fine-grained control. AlphaEvolve Novikov et al. (2025) is an evolutionary coding agent that improves algorithms through iterative code modification and evaluator feedback, achieving advances including a new matrix multiplication method improving on Strassen’s approach after 56 years. Other domain-specific agents span healthcare Zhu et al. (2025b), causal discovery Shen et al. (2025), and mathematics Romera-Paredes et al. (2024). MLE-bench Chan et al. (2024) evaluates AI agents via Kaggle performance; unlike our work, it focuses on engineering rather than research capabilities. 2.2 Scalable Transformer Architecture for Ranking Designing novel ranking architectures remains challenging even for experienced engineers. Traditional methods like FM and DCN Rendle (2010); Wang et al. (2021) are widely used, but optimal transformer design for ranking with sequence features remains open Zhang et al. (2024); Zhu et al. (2025a); Wang et al. (2025). Recent works include Climber Xu et al. (2025), which achieves 5.15× throughput via multi-scale sequence extraction, and OneTrans Zhang et al. (2025), which unifies feature interaction and sequential modeling with 5.68% GMV increase. All these works are human-designed, making our AI co-scientist approach the first of its kind. 3 Methodology In Section 3.1, we briefly describe the mathematical formulation of the search ranking problem. In Section 3.2, we will discuss how we approach this problem by building an AI Co-Scientist for Ranking in five modules: (1) Memory: record what research has been done so far including experiment results, code changes as well as technical details; (2) Idea Generation: generate most promising research ideas; (3) Code Implementation: AI automatically write codes and implement changes; (4) Experimentation: carry out GPU experiments; (5) Results Analysis: analyze and interpret experimental results and decide whether to keep the code changes. 3.1 Search Ranking Problem Formulation Given a user text query q within scene s, our system extracts two types of features. First, dense features represented as a D-dimensional vector c=(c1,c2,…,cD)c=(c_1,c_2,…,c_D) capturing continuous user and query attributes. Second, S sparse feature sequences f1,f2,…,fSf_1,f_2,…,f_S, where each sequence corresponds to a different interaction aspect with accommodations based on the user’s historical platform behavior. Each feature sequence fkf_k is defined as fk=(fk,1,fk,2,…,fk,L)f_k=(f_k,1,f_k,2,…,f_k,L), where L denotes the maximum sequence length and fk,Lf_k,L represents the latest token in the k-th slot feature sequence fkf_k sorted by time from oldest to latest. Both categorical and real-valued features undergo bucketization followed by hashing. User behavior signals are represented as a J-dimensional vector V=(V1,V2,…,VJ)V=(V_1,V_2,…,V_J) where J=10J=10 in practice, with each element being either binary or real-valued, representing user engagement metrics, such as the click and conversion signals for point-of-interest search. Our goal is to learn a function that maps the feature space c,f1,f2,…,fSc,f_1,f_2,…,f_S to the behavior signal space V, effectively modeling click-through and conversion probabilities for all retrieved accommodation options corresponding to each query. Once these probabilities are accurately predicted, we compute a final relevance score containing the probabilities to generate the top-ranked search results presented to users after the re-ranking stage. 3.2 AI Co-Scientist for Ranking Fig 1 illustrates the overall process of the AI Co-Scientist for ranking. 3.2.1 Memory Module We adopt a two-layer memory system. The first layer consists of JOURNEY.md (research ideas from Section 3.2.2, results from Section 3.2.5, and lessons from all model iterations), EXPERIMENTS.md (past and active experiments per Section 3.2.4), and FLOWS.md (scheduled training flows with parameters). The second layer contains detailed per-module markdown files for each model version, similar to Cognition AI (2025). For example, Vx.y_IMPLEMENTATION.md automatically records code implementation details from Section 3.2.3 for verification by both AI Co-Scientist and human experts. The AI Co-Scientist updates all first-layer files after each model iteration and experiment, while only modifying the current version’s second-layer files. The first layer serves as an index, with second-layer files accessed only when necessary. 3.2.2 Idea Generation Module Motivated by latest work Kasa et al. (2025), we suspect that discriminative models can work better than generative models for search ranking problem because this is a high noise, high data, low model capacity, high latency requirement scenario. We first provide AI Co-Scientist with the hypothesis that the gains we saw from earlier transformer-like-based ranking models are mainly coming from the scaling up of the transformer module rather than the generative next-item loss. We showed to AI what a good example: code changes in V2 and how we scheduled experiments manually using some scripts on GPUs, and we then ask AI Co-Scientist to follow our example and come up with novel ways to further improve the ranking performance. Because this module is of the crucial importance and inherently difficult, we are leveraging multiple state-of-the-arts LLM models and have them to reach an consensus for the next step. 3.2.3 Code Implementation Module This part is well understood as coding agents are doing better and better at various SWE benchmarks over the past year. At this module, AI co-scientist is simply implementing the code changes based on the idea generated in the previous module 3.2.2. Despite the fact it may still encounter bugs, this often gets resolved at second attempt after reading the debug logging error messages. 3.2.4 Experimentation Module In this module, AI Co-Scientist would schedule experiments on GPUs with tunable training parameters after committing the code changes to a new git branch. 3.2.5 Results Analysis Module We use an aggregated metric ℳM as the north star for our discovery workflow: the average of 6 AUC metrics across three user behavior types (click, conversion, grouped conversion) and two search scenes (point-of-interest, popularity-based) over 7 days’ unseen data. This metric correlates well with online performance and serves as a verifiable reward signal. The AI Co-Scientist calculates ℳM and proceeds accordingly: if ℳ>0M>0, it creates a new branch and implements the next iteration; if ℳ<0M<0, it decides whether to fix the current branch or revert to the previous best. This decision is controlled by a threshold parameter—higher values encourage persistence, lower values favor quick reversion. Similar to Section 3.2.2, multiple LLMs reach consensus on this decision. (a) V2 design (b) V3 design Figure 3: Comparison of Transformer designs Version Key Innovations Seq Len LR ℳM (%) vs V2 Status V1 Mean-Pooling + DCN + MOE – 5e−55e^-5 −0.118-0.118 Baseline V2 Transformer + Separate Sequences (Figure 3(a)) 40 5e−55e^-5 0.0000.000 New Baseline V3.0 V2 + Positional Encoding + Attention Pooling 40 5e−55e^-5 −0.047-0.047 Failed V3.1 Transformer + Unified Sequence (Figure 3(b)) 200 5e−55e^-5 −0.050-0.050 Failed V3.2 V3.1 + Reduced LR 200 1e−51e^-5 +0.041+0.041 Success V3.3 V3.2 + Slot Type Embeddings 200 1e−51e^-5 +0.061+0.061 Success V3.4 V3.3 + Temporal Embeddings 200 1e−51e^-5 +0.063+0.063 Success V3.5 V3.4 + Four-phase LR optimization (Table 2) 200 Adaptive +0.083 New Best Table 1: Transformer Model Evolution: Architecture and Performance Summary 4 Experiments and Findings 4.1 Discovering Novel Search Ranking Model In this work, we treat the use of only five of the more than 200 available feature-sequence slots as a pilot study, and feed only these five into the transformer. The V1 model is a pre-transformer baseline that primarily relies on mean pooling of sequence features before passing them into DCN Wang et al. (2021). The V2 model is manually designed by human AI scientists: it stacks sequence features and projects dense features into an embedding, which is then added to the sequence feature embeddings at all positions, following standard practice in personalized transformer models Wu et al. (2020) and sharing similarities with recent work Xu et al. (2025). Building on V2 (Fig. 3(a)), the AI Co-Scientist proposes the V3 architecture (Fig. 3(b)). As summarized in Table 1, the AI Co-Scientist first attempts to add positional encodings and attention pooling to V2, but this yields no improvement. It then constructs V3.1 by concatenating the separate sequences used in V2 into a single long sequence before feeding them into the transformer, which initially leads to a drop in the evaluation metric ℳM. In V3.2, the AI Co-Scientist identifies that this degradation is due to an overly large learning rate and mitigates it by reducing the learning rate to one fifth after training on half of the data, achieving a +0.041% lift over V2. 4.2 Designing Training Dynamics Tricks Training convergence is often very challenging for training such multi-task ranking model Tang et al. (2023), and often it is difficult for human experts to tune learning rates as well. We have AI Co-Scientist design the best training dynamics LR schedules based on the observed training dynamics data. It finally comes up with the model V3.5 in which AI first trains the model for 16 days before reducing LR to one fifth, and then increases the learning rate again before finally reducing LR for final fine-tuning as summarized in Appendix’s Table 2. V3.5 achieves higher metric ℳM over V3.1 by 0.133%0.133\%, over V2 by +0.083%+0.083\%, and over V1 by +0.201%+0.201\%. 5 Conclusion A recent report Anthropic (2025) based on interviews with researchers in chemistry, physics, biology, and computational disciplines found that AI is still unable to manage central aspects of scientific work, such as formulating hypotheses and conducting experiments. However, we find that for certain research fields within computer science, this observation may not hold. Taking search ranking research as an example, we demonstrate that AI can discover novel ranking model architectures comparable to those developed by human experts. By working alongside human researchers as AI co-scientists, these systems can significantly reduce routine workloads, freeing researchers to focus on higher-level activities such as deep thinking, attending conferences, networking with peers, and disseminating research findings to broader audiences. Limitations AI Safety Considerations While the system only generates ranking models, AI safety remains important for production deployment. Future work should add automated guardrails beyond the current reliance on human experts. Knowledge Base Integration The system currently lacks access to scientific databases and real-time literature search, limiting use of state-of-the-art methods. References Anthropic (2025) Introducing anthropic interviewer: what 1,250 professionals told us about working with ai. Note: https://w.anthropic.com/research/anthropic-interviewerAccessed: 2025-12 Cited by: §5. J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, et al. (2024) Mle-bench: evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095. Cited by: §2.1. Cognition AI (2025) Devin with sonnet 4.5: lessons and challenges. Note: https://cognition.ai/blog/devin-sonnet-4-5-lessons-and-challengesAccessed: 2025-11 Cited by: §3.2.1. J. Gottweis, W. Weng, A. Daryin, T. Tu, A. Palepu, P. Sirkovic, A. Myaskovsky, F. Weissenberger, K. Rong, R. Tanno, et al. (2025) Towards an ai co-scientist. arXiv preprint arXiv:2502.18864. Cited by: §1, §2.1. S. R. Kasa, K. Gupta, S. Roychowdhury, A. Kumar, Y. Biruduraju, S. K. Kasa, P. N. Priyatam, A. Bhattacharya, S. Agarwal, and V. Huddar (2025) Generative or discriminative? revisiting text classification in the era of transformers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 9615–9637. Cited by: §3.2.2. C. Lu, C. Lu, R. T. Lange, J. Foerster, J. Clune, and D. Ha (2024) The ai scientist: towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292. Cited by: §2.1. A. Novikov, N. Vũ, M. Eisenberger, E. Dupont, P. Huang, A. Z. Wagner, S. Shirobokov, B. Kozlovskii, F. J. Ruiz, A. Mehrabian, et al. (2025) AlphaEvolve: a coding agent for scientific and algorithmic discovery. arXiv preprint arXiv:2506.13131. Cited by: §2.1. S. Rendle (2010) Factorization machines. In 2010 IEEE International conference on data mining, p. 995–1000. Cited by: §2.2. B. Romera-Paredes, M. Barekatain, A. Novikov, M. Balog, M. P. Kumar, E. Dupont, F. J. Ruiz, J. S. Ellenberg, P. Wang, O. Fawzi, et al. (2024) Mathematical discoveries from program search with large language models. Nature 625 (7995), p. 468–475. Cited by: §2.1. C. Shen, Z. Chen, D. Luo, D. Xu, H. Chen, and J. Ni (2025) Exploring multi-modal data with tool-augmented llm agents for precise causal discovery. In Findings of the Association for Computational Linguistics: ACL 2025, p. 636–660. Cited by: §2.1. J. Tang, Y. Drori, D. Chang, M. Sathiamoorthy, J. Gilmer, L. Wei, X. Yi, L. Hong, and E. H. Chi (2023) Improving training stability for multitask ranking models in recommender systems. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 4882–4893. Cited by: §4.2. C. Wang, B. Wu, Z. Chen, L. Shen, B. Wang, and X. Zeng (2025) Scaling transformers for discriminative recommendation via generative pretraining. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, p. 2893–2903. Cited by: §2.2. R. Wang, R. Shivanna, D. Cheng, S. Jain, D. Lin, L. Hong, and E. Chi (2021) Dcn v2: improved deep & cross network and practical lessons for web-scale learning to rank systems. In Proceedings of the web conference 2021, p. 1785–1797. Cited by: §2.2, §4.1. L. Wu, S. Li, C. Hsieh, and J. Sharpnack (2020) SSE-pt: sequential recommendation via personalized transformer. In Proceedings of the 14th ACM conference on recommender systems, p. 328–337. Cited by: §4.1. S. Xu, S. Wang, D. Guo, X. Guo, Q. Xiao, B. Huang, G. Wu, and C. Luo (2025) Climber: toward efficient scaling laws for large recommendation models. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 6193–6200. Cited by: §2.2, §4.1. B. Zhang, L. Luo, Y. Chen, J. Nie, X. Liu, D. Guo, Y. Zhao, S. Li, Y. Hao, Y. Yao, et al. (2024) Wukong: towards a scaling law for large-scale recommendation. arXiv preprint arXiv:2403.02545. Cited by: §2.2. Z. Zhang, H. Pei, J. Guo, T. Wang, Y. Feng, H. Sun, S. Liu, and A. Sun (2025) OneTrans: unified feature interaction and sequence modeling with one transformer in industrial recommender. arXiv preprint arXiv:2510.26104. Cited by: §2.2. J. Zhu, Z. Fan, X. Zhu, Y. Jiang, H. Wang, X. Han, H. Ding, X. Wang, W. Zhao, Z. Gong, et al. (2025a) Rankmixer: scaling up ranking models in industrial recommenders. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 6309–6316. Cited by: §2.2. Y. Zhu, Y. Qi, Z. Wang, L. Gu, D. Sui, H. Hu, X. Zhang, Z. He, J. He, L. Ma, et al. (2025b) HealthFlow: a self-evolving ai agent with meta planning for autonomous healthcare research. arXiv preprint arXiv:2508.02621. Cited by: §2.1. Appendix A Appendix Table 2: V3.5 Four-Phase Learning Rate Optimization Strategy Phase Days Learning Rate Purpose 0 1–16 5e−55e^-5 Standard LR 1 17–19 1e−51e^-5 Reduced LR 2 20–23 1.5e−51.5e^-5 Plateau Escape 3 24–28 8e−68e^-6 Final Fine-Tuning Figure 4: AI Co-Scientist planning more routine tasks in Section 3.2.3 and 3.2.4 A.1 Planning Routine Tasks with the AI Co-Scientist Figure 4 illustrates how the AI Co-Scientist decomposes the research workflow into a sequence of routine tasks that can be automated or semi-automated. For code-related work (Section 3.2.3), the system identifies concrete implementation steps such as setting up the project environment, generating boilerplate code, and organizing experiment scripts. For experimental work (Section 3.2.4), the planner enumerates tasks like configuring hyperparameters, defining evaluation metrics, and scheduling experiment batches. By explicitly representing these tasks and their dependencies, the planner helps researchers offload low-level coordination while retaining control over high-level decisions. This structured breakdown also facilitates iteration: when requirements change, the AI Co-Scientist can quickly re-plan the affected portions of the workflow, updating the sequence of routine tasks without the researcher having to manually reconfigure every step. A.2 Known Issues We highlight several limitations and failure modes observed in our current framework: Code reliability and silent bugs. The AI agent still at times introduces bugs in code implementations that can be difficult to detect without expert supervision. For instance, it may unintentionally increase model size or introduce subtle errors in architecture design code. In some cases, purely based on experimental metrics, these buggy implementations can appear to "win" and be selected as a better model, even though the underlying code is incorrect. Role of research taste. We observe that human research taste remains important. It can guide the process toward more elegant and principled ideas and designs, even when these do not yield immediate experimental gains. Without such guidance, the system may overemphasize short-term improvements and overlook more promising long-term directions. Inefficient idea generation and resource usage. In practice, idea generation by the AI agent is still not very efficient. This can lead to substantial waste of GPU resources, and typically requires a human expert to remain in the loop to prevent catastrophic failures (e.g., obviously broken configurations or experiments that do not meaningfully contribute to progress). Myopic feedback loop. The optimization loop is vulnerable to myopic behavior: the agent may focus on ideas that yield short-term improvements in experimental metrics but are not actually promising in the long run. Such short-term gains can mislead the AI agent and reinforce directions that a human expert would deem uninteresting or unproductive.