Paper deep dive
Assembling Insights for Agentic Machine Learning Engineering Systems
Bihui Jin, Yinxi Li, Kaiyuan Wang, Pengyu Nie
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Agentic machine learning engineering (MLE) is an emerging AI4SE application for complex ML tasks and a step toward recursive self-improvement of AI systems. Recent agentic MLE systems show the value of leveraging insights from related MLE tasks: some systems condition code generation on expert domain knowledge, which is implicitly curated from peer MLE tasks; some systems have a loop of solving an MLE task, gathering memory to benefit future tasks. However, insight collection and injection remain ad hoc, and existing evaluations often do not control insight sources, making data leakage a threat to validity. To close these gaps, we systematically study how insights from related MLE tasks affect agentic MLE systems, and how their impact depends on insight type and injection method. We introduce MLE-InsightBench, a benchmark of 160 Kaggle competitions across 12 domains. To prevent leakage, one competition per domain is held out as the target, while the remaining 148 serve as sources only when they cannot have been influenced by the target task, such as through later editions or reused solutions. We also develop MLE-InsightForge, an insight assembly agent that explores source competitions, top human solutions, and writeups to construct three insight types: 12 domain insights summarizing best practices, 148 competition insights describing winning solutions, and 16 improvement insights capturing generic debugging and optimization strategies. These insights are injected either as on-demand skills or as an upfront playbook. In our evaluation, the extracted insights improve the normalized private-test score by 123.9% over a no-insight baseline. Controlled experiments further show that the three insight types are complementary; the better injection mode (skill vs. playbook) varies by tasks, with skills more cost-effective overall; and the gains generalize across backend LLMs.
Tags
Links
- Source: https://arxiv.org/abs/2610.04927v1
- Canonical: https://arxiv.org/abs/2610.04927v1
Trouble viewing inline? Open PDF directly ā
Full Text
96,980 characters extracted from source content.
Expand or collapse full text
Assembling Insights for Agentic Machine Learning Engineering Systems Bihui Jin Affiliation: University of Waterloo, Waterloo, Canada email: bihui.jin@uwaterloo.ca , Yinxi Li Affiliation: University of Waterloo, Waterloo, Canada email: yinxi.li@uwaterloo.ca , Kaiyuan Wang Affiliation: Google Inc., Mountain View, USA email: kaiyuanw@google.com and Pengyu Nie Affiliation: University of Waterloo, Waterloo, Canada email: pynie@uwaterloo.ca Ā© none Abstract. Agentic machine learning engineering (MLE) is an emerging AI4SE application, targeting complex machine learning tasks and taking a step toward the recursive self-improvement of AI systems. Recent agentic MLE systems highlight the importance of collecting and leveraging insights from closely-related MLE tasks: some systems condition code generation on expert domain knowledge, which is implicitly curated from peer MLE tasks; some systems have a loop of solving an MLE task, gathering validation feedback, and refining their prompts, skills, or memory to benefit future tasks. However, collecting and injecting insights is ad hoc, and existing evaluations do not control the sources of insights, leaving data leakage a threat to validity. To close these gaps, we systematically study how insights assembled from closely-related MLE tasks affect an agentic MLE system, and how their effect depends on the type of insights and the way they are injected. We first collect a new benchmark, MLE-InsightBench, consisting of 160 Kaggle competitions clustered into 12 domains. MLE-InsightBench is designed to keep target task information out of the insights: one competition from each domain is held out as a target task, and only the remaining 148 competitions, none of which can have been influenced by the target tasks (e.g., as a later edition of the same series or by reusing their solutions), serve as source competitions. We then design and implement an insight assembly agent, MLE-InsightForge, that explores the source competitions, their top human-written solutions, and writeups, and assembles them into three types of insights: 12 domain insights (summarizing domain-level best practices), 148 competition insights (describing competition-level winning solutions), and 16 improvement insights (detailing generic debugging and improvement strategies). The extracted insights can be injected into an agentic MLE system as skills, which the agent reads selectively on demand, or as a playbook placed upfront in its initial context. In our evaluation, the extracted insights improve the normalized private-test score of an agentic MLE system by 123.9% over the baseline without insights. Further controlled experiments show that: the three types of insights are complementary, the better injection mode (skill vs. playbook) varies across tasks, with skills being more cost-effective overall, and the improvements from insights generalize across backend LLMs. 1. Introduction Machine learning engineering (MLE) is the end-to-end work of turning a dataset and a modeling objective into an executable machine learning (ML) solution, including data inspection, model selection, validation design, debugging, and compute allocation. ML solutions are typically written in Python, as Jupyter notebooks or scripts (Pimentel et al., 2019; Grotov et al., 2022), and the SE community has studied them as software artifacts with distinct engineering challenges, such as data management, code quality, and reproducibility (Amershi et al., 2019; Pimentel et al., 2019; Mostafavi Ghahfarokhi et al., 2024; Jin et al., 2026). This makes ML solutions a natural target of LLM-based coding agents, but also a challenging one. Existing code generation benchmarks, including those for data science code, check a generated function, notebook cell, or patch against tests or reference outputs (Lai et al., 2023; Yin et al., 2023; Jimenez et al., 2024). An ML solution, in contrast, has no test oracle: its quality is a metric value on unseen data, which can only be estimated by executing long-running training code and improved by repeatedly revising the solution. Agentic MLE is an emerging application of coding agents in which LLMs perform this work by iteratively writing, executing, and refining code (Huang et al., 2024; Chan et al., 2025; Jiang et al., 2025). Agentic MLE systems target complex ML tasks, such as Kaggle competitions, and are a step toward the recursive self-improvement of AI systems, since a system that can engineer ML solutions can, in principle, contribute to building better AI systems (Chan et al., 2025). Evolution has been rapid: on MLE-bench (Chan et al., 2025), the fraction of competitions in which the best system earns a medal grew from 17% in late 2024 to over 60% in early 2026. Recent agentic MLE systems highlight the importance of collecting and leveraging insights from closely-related MLE tasks. Some systems generate code based on expert domain knowledge, which is implicitly curated from peer MLE tasks, either by searching the web for state-of-the-art models and winning solutions of similar tasks (Nam et al., 2025; Chen et al., 2026) or by drawing on knowledge bases synthesized from open-source repositories, competition platforms, and competition discussions (Du et al., 2026; Du et al., 2025; Nadafian et al., 2026). Other systems have a loop of solving an MLE task, gathering validation feedback, and refining their prompts, skills, or memory to benefit future tasks (Grosnit et al., 2024; Zhu et al., 2026; Zhang et al., 2026b). However, in these systems, the process of collecting and injecting insights is ad hoc: each system fixes one design, without a controlled comparison of the alternatives. Each system collects insights from different sources (web search, curated knowledge bases, or its own past runs), represents them differently, and injects them into the agentās context in its own way. The ablation studies in these work typically remove the entire insight source, e.g., the knowledge base or the cache of prior wisdom (Du et al., 2025; Zhu et al., 2026), which shows that insights help as a whole but leaves open which types of insights contribute and how they should be injected. Moreover, existing evaluations of these systems do not control for the sources of insights, which leaves data leakage as a threat to validity: the sources are chosen by each system rather than fixed by the benchmark, so whether they exclude the target tasks is left to each system and rarely verifiable. For example, web search at solving time (Nam et al., 2025; Chen et al., 2026) can retrieve the winning writeups of the very competition being evaluated, and knowledge bases built from competition platforms may contain its solutions; even when the benchmark competitions are excluded by name (Zhu et al., 2026; Yang et al., 2025), closely-related editions of the same competition series can remain. Such concerns were raised on the MLE-bench leaderboard (timeleftā, 2026), which has since paused new submissions to ensure fair and comparable results. BenchmarkMLE-InsightBench (Ā§3) Kaggle competitions 160 high-quality competitions in 2020ā2025 Domain clustering 12 domains of closely-related tasks Held-out split target (12): one per domain; source (148): causal non-descendants Data leakage control target materials blocked during insight assembly; no web search when solving Private-test grading MLE-bench-compatible graders; private-test set never exposed TechniqueMLE-InsightForge (Ā§4) Insight assembly agent reads all sources; records the evidence of each insight 12 domain insights domain-level best practices 148 competition insights decisions of the winning solutions 16 improvement insights consolidated debugging and improvement strategies Insight injection as skills (on demand) or playbooks (in the initial context) EvaluationSystematic study (Ā§5) Solving agent State-of-the-art ReAct agent developing a solution tree Six conditions Baseline, Full-skill, Full-playbook, and three ablations Fixed budget 12 hours, 250 steps, and one GPU per run Metrics normalized private-test score, token consumption, and time ? Ablation studies insight types, injection modes, and backend LLMs Figure 1. Overview of our study.Three side-by-side panels. The benchmark panel lists Kaggle competition collection, domain clustering, the held-out split, leakage control, and private-test grading. The technique panel lists the insight assembly agent, domain, competition, and improvement insights, and insight injection as skills or playbooks. The evaluation panel lists the agentic MLE system, six conditions, the fixed budget, the metrics, and the research questions. To bridge these gaps, we systematically study how insights assembled from closely-related MLE tasks affect agentic MLE systems, and how their effect depends on the type of insights and the way they are injected. Figure 1 illustrates the workflow of our study. We first build MLE-InsightBench from Kaggle competitions with human-written solutions, i.e., gold-medal teamsā notebooks and writeups, and group them into domains of closely-related competitions. One competition of each domain is held out as a target task, and only the remaining 148 competitions, all causal non-descendants of the target tasks (i.e., none can have been influenced by a target task, as a later edition of the same series or a competition reusing its solutions could), serve as source competitions, which mirrors practice and rules out insights derived from a target taskās own winning solutions. MLE-InsightForge then reads the descriptions, datasets, solution notebooks, and writeups of the source competitions and assembles three types of insights: competition insights on the decisions behind each competitionās winning solutions, domain insights that reconcile them into domain-level best practices, and improvement insights that connect problems encountered during development to their resolutions. Each insight consists of triggerāaction items that a system can skip when they do not apply, and is backed by records of its source passages. The insights can be injected as skills, whose full text is read on demand (Anthropic, 2025), or as a playbook in the initial context (Zhang et al., 2026a). On the 12 target tasks, the assembled insights improve the normalized score of an agentic MLE system by 123.9% over the baseline without insights, improving every task (significantly on 11) when injected as skills; on an energy forecasting task, every run with insights beats the best run without them, although none of its peer competitions involves energy. Further controlled ablation studies show that the three types of insights are complementary: each type alone significantly improves fewer tasks (9, 8, and 8) than their combination (11). The improvements from insights generalize across backend LLMs, including a stronger proprietary LLM (GPT-6.1-Sol). However, the better injection mode varies across tasks, with skills being more cost-effective overall, because they reach a similar score with fewer input tokens (7.7% versus 25.2% more than the baseline). The main contributions of this work are threefold: ā¢ Benchmark. We collect MLE-InsightBench, a benchmark of 160 Kaggle competitions clustered into 12 domains, that provides a controlled environment for studying insights: its held-out split makes the sources of insights explicit and keeps target task information out of them. ā¢ Technique. We design and implement MLE-InsightForge, an agent that assembles evidence-based domain insights, competition insights, and improvement insights from human-written solutions and writeups, and injects them into agentic MLE systems as skills or playbooks. ā¢ Experiments. We systematically evaluate the impact of assembled insights on an agentic MLE system, including their overall effectiveness, the contributions of the three types of insights, the effectiveness and cost of the two injection modes, and the generalizability across backend LLMs. 2. Background 2.1. Agentic MLE Systems (a) Task: Kaggle competition page kaggle.com/competitions/jigsaw-agile-community-rules Jigsaw ā Agile Community Rules Classification Using AI models to help moderators uphold community-specific norms. Description Your task is to create a binary classifier that predicts whether a Reddit comment broke a specific rule. [ā¦] The test dataset contains additional rules that are not seen in the training data [ā¦] Dataset train.csv (2,029 rows) and test.csv, e.g.: body: code free tyrande [Imgur](ā¦) for you and your friend 2 codes for 4 dollars https://w.paypal.com/ā¦ rule: No Advertising: Spam, referral links, unsolicited advertising, and promotional content are not allowed. subreddit: hearthstone positive_example_1,2: [ā¦] negative_example_1,2: [ā¦] rule_violation: 1 (target) Metric Column-averaged AUC: the AUC within each rule, averaged over the rules. Submission submission.csv with the predicted probability of rule_violation for each row_id. (b) Solution: solution.py # data preparation train = pd.read_csv(ā/kaggle/input/train.csvā) test = pd.read_csv(ā/kaggle/input/test.csvā) enc = SentenceTransformer(āintfloat/e5-large-v2ā).encode def build_features(df): body = enc(df[ābodyā]) sims = [cos(body, enc(df[c])) for c in [āruleā, āpositive_example_1ā, ..., ānegative_example_2ā]] return np.stack(sims + [df[ābodyā].str.count(āhttpā), ...], axis=1) X, y = build_features(train), train[ārule_violationā] # validation: leave one rule out aucs = [] for rule in train[āruleā].unique(): held = (train[āruleā] == rule) clf = LogisticRegression().fit(X[~held], y[~held]) p = clf.predict_proba(X[held])[:, 1] aucs.append(roc_auc_score(y[held], p)) print(āValidation AUC:ā, np.mean(aucs)) # training clf = LogisticRegression().fit(X, y) # test proba = clf.predict_proba(build_features(test))[:, 1] pd.DataFrame(ārow_idā: test[ārow_idā], ārule_violationā: proba) .to_csv(ā/kaggle/working/submission/submission.csvā) Figure 2. An example Kaggle competition and a simplified ML solution developed by Baseline.Two side-by-side boxes. (a) The task Jigsaw ā Agile Community Rules Classification, styled as its Kaggle competition page with the address kaggle.com/competitions/jigsaw-agile-community-rules, the subtitle "Using AI models to help moderators uphold community-specific norms", and the sections Description, Dataset, Metric, and Submission: the description asks for a binary classifier that predicts whether a Reddit comment broke a specific rule, with unseen rules in the test data; the dataset consists of train.csv and test.csv, illustrated by one row with the comment body, the rule, the subreddit, positive and negative example comments, and the binary target rule\_violation; the metric is the column-averaged AUC; the submission file contains a probability for each row. (b) A Python solution in four stages: data preparation, which encodes the comment, the rule, and the example comments with a sentence-embedding model and computes their similarities as features; validation, which trains a logistic regression with each rule held out in turn and reports the mean AUC; training on all data; and test, which predicts the test data and writes submission.csv. An MLE task asks for an executable ML solution rather than only a fitted model: it involves understanding the data and the metric, designing a validation protocol, implementing preprocessing and training, diagnosing failures, and producing an artifact that satisfies a submission contract. These decisions are coupled: for example, the validation split must reflect the deployment distribution, while model and feature choices must respect runtime and memory limits. Competitions on Kaggle (Kaggle, 2026a) expose MLE tasks through a common interface consisting of a task description, datasets, an evaluation metric, a compute environment, and a submission format. Figure 2 shows an example Kaggle competition, Jigsaw ā Agile Community Rules Classification (Sorensen et al., 2025a), which asks whether a Reddit comment violates a given rule, including rules unseen in training, together with a solution developed by an agentic MLE system. The solution prepares features from the training data, validates the model by holding out one rule at a time to mimic the unseen rules, retrains on all training data, and writes the predictions for the test data in the submission format. MLE-bench (Chan et al., 2025) turns 75 Kaggle competitions into an offline benchmark: each competitionās training data is split into new training and private-test sets, and a submission is graded against the private-test labels and compared with the human leaderboard. MLRC-Bench (Zhang et al., 2025) instead targets research competitions held at ML conferences (e.g., NeurIPS and KDD Cup), but contains only seven tasks. Neither benchmark specifies a temporal or causal ordering among the tasks, i.e., there is no standard of which other tasks may serve as sources of insights for a given task. An agentic MLE system combines an LLM with tools for inspecting data, editing code, executing experiments, and observing validation feedback, typically following the ReAct pattern of interleaved reasoning and tool use (Yao et al., 2023). Beyond a single linear trajectory, most recent systems search over candidate solutions. AIDE (Jiang et al., 2025) maintains a tree of solutions that are drafted, debugged, and improved; ML-Master (Liu et al., 2025) explores such trees with Monte Carlo tree search, and AIRA-dojo (Toledo et al., 2025) compares search policies and operators over them; and FM Agent and LoongFlow (Li et al., 2025; Wan et al., 2025) evolve populations of solutions. Within a task, these systems accumulate the code, scores, and reflections produced by earlier candidates to guide later ones. 2.2. Insights in Agentic MLE Systems We use insight to denote a reusable, natural-language statement about how to solve MLE tasks, derived from tasks other than the one being solved. Recent systems obtain insights in two ways: ā¢ External knowledge: Insights are curated from external sources, such as solutions of similar MLE tasks on open-source repositories and competition platforms, API documentations, and forum discussions. Some techniques retrieve them on the fly, typically via web search (e.g., MLE-STAR (Nam et al., 2025) and MARS (Chen et al., 2026)); others build a knowledge base offline which is used for retrieval-augmented generation (RAG) at solving time (e.g., DS-Agent (Guo et al., 2024), MLEvolve (Du et al., 2026), AutoMLGen (Du et al., 2025), KAPSO (Nadafian et al., 2026), AIBuildAI-2 (Zhang et al., 2026b), and FM Agent (Li et al., 2025)). ā¢ Self-improvement across tasks: Insights are distilled from the systemās own runs on previous tasks, usually generated by the agent itself or a separate surrogate agent. Examples include the accumulated memory of Agent K (Grosnit et al., 2024), the āprior wisdomā cache of ML-Master 2.0 (Zhu et al., 2026), and the takeaways of completed runs of AIBuildAI-2 (Zhang et al., 2026b) (which augments its external knowledge base). Regardless of their source, insights reach the agent as text in its context, and systems differ in when this happens. In the skill mode (Anthropic, 2025), only compact metadata of each insight (a name and a description) is placed in the context, and the agent loads the full text on demand, a design known as progressive disclosure. In the playbook mode (Zhang et al., 2026a), the insights are placed directly in the initial context as a playbook; unlike a human-written prompt, a playbook is generated and maintained by LLM agents. Skills save context but depend on the agent retrieving the right insight, whereas playbooks make every insight available but consume context even for irrelevant insights. 2.3. Data Leakage in Evaluating Agentic MLE Systems An evaluation of an agentic MLE system suffers from data leakage when the system has access, during solving, to information about the target task that would not be available when solving a genuinely new task, most notably the taskās own winning solutions. This differs from trainātest leakage in a generated solution, which some systems check (Nam et al., 2025; Du et al., 2026), and from contamination of the backend LLMās pre-training data, which several papers acknowledge as a limitation (Nam et al., 2025; Toledo et al., 2025). Insights are a direct channel for such leakage, because their sources are usually the same public competition materials from which target tasks are drawn. Web search at solving time is not reported to be restricted to materials of other tasks (Nam et al., 2025; Chen et al., 2026), and knowledge bases built from competition platforms typically do not report whether the target tasks were excluded (Du et al., 2026; Du et al., 2025). Some work did exclude the target tasks by competition name (Zhu et al., 2026; Yang et al., 2025), which does not cover later editions of the same competition series. For example, MLE-InsightBench contains two AI Mathematical Olympiad: Progress Prize 1 (XTX Investments, 2024) and Progress Prize 2 (Frieder et al., 2024); if the first were a target task, excluding it by name would still admit the second, whose description states that it builds upon the first and one of whose gold-medal notebooks reuses code from the winning solution of the first. Cross-task memory raises a related concern about the order of tasks. Accumulating experience over a sequence of tasks, i.e., online learning, is not leakage by itself, as it mirrors how practitioners gain experience over time. However, when the memory keeps accumulating across the target tasks themselves, a task may be solved after another task that it influenced. Avoiding leakage thus requires a chronological or causal ordering of the tasks. These practices have been raised as a fairness concern on the MLE-bench leaderboard (timeleftā, 2026), which is one of the reasons that the leaderboard has paused new submissions. Contamination-free benchmarks in code generation address a similar problem by evaluating only on problems released after the modelās training cutoff (Jain et al., 2025); MLE-InsightBench (Ā§3) applies a similar principle to insight sources by using only competitions that cannot have been influenced by the target tasks. 3. MLE-InsightBench: A Controlled-Environment Benchmark In MLE-InsightBench, the source competitions are fixed and disjoint from the target tasks: competitions are grouped into domains so that every target task has closely-related peer tasks, and the split is fixed before any insight is assembled. A scatter plot of UMAP embeddings of the competition descriptions, clustered into twelve clusters plus noise. Figure 3. UMAP embeddings of the competition descriptions (12 clusters).A scatter plot of UMAP embeddings of the competition descriptions, clustered into twelve clusters plus noise. Competition collection. We collect Kaggle competitions using Meta Kaggle (Kaggle, 2026c) and Meta Kaggle Code (Kaggle, 2026b). We retain competitions that are launched in 2020 or later, use Python, have downloadable datasets, and have high-performing self-contained solutions, and exclude playground competitions; a solution counts as high-performing and self-contained if it received a gold medal and uses no datasets other than the competitionās own. This step yields 191 competitions, from which we manually remove 18 that do not involve training ML models (simulation competitions and optimization puzzles, such as the Santa series (Reade and Oldacre, 2025)), leaving 173 candidate competitions. For each competition, we collect the task description, the public writeups of the teams ranked within the gold-medal zone of the leaderboard, and the high-performing self-contained solution notebooks; we refer to the latter two as the competitionās human-written solutions. Table 1. Target tasks of MLE-InsightBench. Domain Target task Metric Type Agriculture Recognition CSIRO Biomass (Liao et al., 2025) R2R^2ā Simple Bioinformatics CAFA 5 Protein Function (Friedberg et al., 2023) Fmax_maxā Simple Geography Retrieval and Localization Image Matching 2025 (Bellavia et al., 2025) mAAā Simple Medical Image Recognition RSNA Aneurysm Detection (Rudie et al., 2025) AUCā Code NLP Jigsaw Community Rules (Sorensen et al., 2025b) AUCā Simple Physiological Signal Processing PhysioNet ECG Digitization (Reyna et al., 2025) SNRā Simple Quantitative Finance Mitsui Commodity Prediction (Demkin et al., 2025) Sharpeā Code Scientific Signal Processing Ariel Data Challenge 2025 (Yip et al., 2025) GLLā Simple Sport Tournament Prediction March Mania 2025 (Sonas et al., 2025) Brierā Simple Time Series Prediction Energy Behavior of Prosumers (Eljand et al., 2023) MAEā Code Video Recognition NFL Player Contact (Howard et al., 2022b) MCCā Simple Other POG Series S01E01 (Mulla, 2022) MAEā Simple Domain clustering. We group the candidate competitions by clustering their natural-language descriptions (overview and dataset description), and then manually assign a human-readable label to each cluster. We embed each description with harrier-oss-v1-27b (Microsoft, 2026), a text embedding model, by instructing it to cluster competitions by ML task formulation and application domain. We then reduce the embeddings to 11 dimensions with UMAP (McInnes et al., 2018) (cosine distance, 20 neighbors) and cluster them with HDBSCAN (Campello et al., 2013) (minimum cluster size 5, minimum samples 3), which determines the number of clusters from the data and leaves outliers unassigned. This yields 12 clusters and 7 outliers (unable to be clustered), as shown in Figure 3. We label each cluster by inspecting its member competitions and its most distinctive terms under class-based TF-IDF (Grootendorst, 2022), and manually move the outlier WiDS Datathon 2022 (Matthys et al., 2022) into the cluster that contains its other editions. Domains serve two purposes: they define the scope of domain insights, and they ensure that every target task has closely-related peer tasks from which insights can be assembled. We exclude the Audio Recognition cluster, where none of its 8 competitions satisfies the eligibility rule for target tasks described below, as well as 4 other competitions without any writeup from the gold-medal zone, and Jane Street Market Prediction (cnyberg et al., 2020), whose dataset was available at our initial collection but is no longer publicly accessible. The resulting MLE-InsightBench consists of 160 competitions in 12 domains, including a residual domain labeled Other for the remaining outliers. Held-out split. From each domain, we hold out one eligible competition as a target task and use the remaining 148 competitions as source competitions. A competition is eligible if the mean runtime of its collected solutions is at most the 75th percentile of the runtimes of all collected solution notebooks (25.5 minutes), which keeps the evaluation within a fixed compute budget. We also manually mark some competitions as ineligible where the executable environment for the tasks are hard to replicate (e.g., using a huge dataset): AMP Parkinsonās Disease Progression Prediction (Kirsch et al., 2023), Drawing with LLMs (Holbrook et al., 2025), Bengali.AI Speech Recognition (Howard et al., 2023), Cornell Birdcall Identification (Howard et al., 2020), and the BirdCLEF series (e.g., BirdCLEF+ 2025 (Klinck et al., 2025)). Among the eligible competitions of the most recent year in a domain, we select the one with the shortest solution runtime. The resulting non-descendant split ensures that every source competition is a causal non-descendant of its target task, i.e., a competition that cannot have been influenced by the target task as a later edition of the same series or reusing its solutions. This mirrors how insights would be used in practice: an agentic MLE system facing a new task can draw on prior experience, but not on experience derived from the new task itself. While a chronological split (i.e., choosing the latest competition of each domain) is more straightforward, it would reduce the size of benchmark because the latest competitions sometimes are not eligible as a target task, causing dropping of valuable source competitions that could otherwise be used with our more careful split. Table 1 lists the 12 target tasks, of which 9 are standard competitions with a submission file and 3 are code competitions whose predictions are produced through a competition-specific evaluation API. Leakage control. MLE-InsightBench controls the sources of insights at three points. First, the materials of the target tasks (datasets, descriptions, writeups, and solutions) are excluded from insight assembly, so no assembled insight can be derived from a target taskās own winning solutions. Second, because the split only requires the source competitions to be causal non-descendants of the target task, earlier editions of a recurring competition series (e.g., March Machine Learning Mania) remain legitimate source competitions, as they would be for a practitioner. Third, the agentic MLE system under evaluation is not given web-search tools and receives only the public data of the target task; the private-test inputs are substituted only when the final solution is graded (same as how Kaggle grades submissions), and the private-test labels are never exposed. 4. Methodology 4.1. MLE-InsightForge: Insight Assembly Agent MLE-InsightForge explores the descriptions, datasets, top human-written solutions, and writeups of the source competitions and assembles three types of insights from them. Separating what insights are assembled from how they are injected (Ā§4.2) lets us vary the two independently. 4.1.1. Agent Architecture (a) Assembly agent Assembly task ReAct loop LLM context summary Tools files, notebooks datasets, sandbox web, PDFs, repos images, videos Validator full reads, evidence tool callsobservationssubmitrejectWorkspaceread-onlyWebfiltered insights + evidence (b) Assembly workflow source competition 1. Assemble 148 competition insights domain descriptions 2. Synthesize 12 domain insights problemāfix writeups 3. Assemble 1,068 raw improvement insights 4. Cluster, synthesize 16 improvement insights per domainclustered Figure 4. Architecture of MLE-InsightForge: (a) the assembly agent and (b) the four assembly stages.Two panels. Panel (a) shows an assembly task, consisting of instructions and required sources, entering a ReAct loop in which an LLM calls tools (reading files and notebooks, inspecting datasets, running analyses in a sandbox, reading web pages, PDFs, and repositories, and inspecting images and videos) and receives observations; the tools access a read-only workspace with the the blacklist-filtered web; the LLM's context is summarized when needed; the LLM submits a report to a validation step that checks that all required sources were read in full and that each insight has an evidence record, returning rejections to the LLM and otherwise producing the insight text and evidence records. Panel (b) shows four stages: competition insight assembly from each , domain insight synthesis from the competition insights and descriptions of a domain, improvement insight assembly from writeups reporting a problem and its resolution, and clustering and synthesis of the raw improvement insights into improvement insights. MLE-InsightForge (Figure 4(a)) is a ReAct-style agent (Yao et al., 2023), using gpt-5.6-sol-pro as its backend LLM during our study. Each run of the agent performs one assembly task, specified by the instructions of its stage and the list of source files it must read. The agentās tools read files and notebooks, inspect datasets, run targeted analyses in a sandboxed container (e.g., checking a preprocessing step or a validation split claimed in a writeup, without rerunning the solution end-to-end), and read the materials linked from writeups, such as web pages, PDFs, code repositories, images, and videos. Leakage prevention. The source materials are mounted read-only in the agentās workspace, which contains only the source competitions and not the target tasks. Because some URLs in the source materials may also link to the target tasks, web access goes through a blacklist of 1,985 URL patterns, which we extract from the descriptions, writeups, and solution notebooks of the target tasks; links to Kaggle or GitHub the competitionās or repositoryās all sub-pages, while general documentation and tutorial sites (e.g., library documentation and Wikipedia) are manually exempted. Evidence-based submission. The agent submits each insight through a validated submission interface, which rejects the submission until the agent has read every required source in full, otherwise returns the incomplete insight to the agent for repair. To minimize hallucination, the interface also requires an evidence record for every insight, listing the original source passages it is based on; this ensures that every insight can be traced back to its original source. Assembly stages. Figure 4(b) shows the four stages in which MLE-InsightForge runs. It first assembles one competition insight per source competition and then synthesizes one domain insight per domain from the competition insights of that domain. Independently of these two types, it assembles raw improvement insights from the writeups and consolidates them by clustering. 4.1.2. Insight Types MLE-InsightForge assembles 148 competition insights, 12 domain insights, and 16 improvement insights, at a total API cost of $903 USD at the list prices of gpt-5.6-sol-pro ($565, $22, and $317 USD for the three types, respectively). Competition insights. MLE-InsightForge first reads the description to identify the objective, data, metric, and constraints, and then reads every collected writeup in full. It consults the solution notebooks to disambiguate implementation details and to check that a technique claimed in a writeup is actually present, and inspects the datasets to understand their characteristics and to check claims about them. The resulting insight is organized around the decisions that distinguished the winning solutions and states the conditions under which each decision was made, rather than reproducing a particular solution, since the insight is used for other tasks, to which a complete solution tailored to the source competitionās data and metric does not transfer. Domain insights. A target task has no insight of its own, but it shares task properties with the other competitions in its domain. MLE-InsightForge reads the descriptions and competition insights of all source competitions in the domain and reconciles recommendations that recur, complement each other, or conflict. It replaces competition-specific prescriptions with conditional guidance; for example, it associates a validation strategy with grouped subjects or temporal deployment rather than as universally applicable. A domain insight drops recommendations specific to a single competition, keeping only those supported by several competitions or a task archetype. A scatter plot of UMAP embeddings of the raw improvement insights, clustered into groups plus noise. Figure 5. UMAP embeddings of the raw improvement insights (15 clusters).A scatter plot of UMAP embeddings of the raw improvement insights, clustered into groups plus noise. Improvement insights. While the other two types describe the decisions behind the final solutions, improvement insights capture how their authors diagnosed and fixed problems along the way, by connecting an observed problem (e.g., a validation split that does not match the deployment setting, or an inference pipeline that exceeds the runtime budget) to the resolution that fixed it. MLE-InsightForge first filters writeups, with an LLM, for those that report both a concrete problem and an actionable resolution. For each retained writeup, it then assembles standalone insights, keeping only those whose outcome is directly reported in the writeup; this step yields 1,068 raw improvement insights. Because many improvement strategies recur across competitions, exposing all of them individually would make them costly to discover and highly redundant. We therefore cluster the raw improvement insights by their descriptions, using the same approach as Domain Clustering (Ā§3), and let MLE-InsightForge synthesize each cluster into one consolidated insight under the same evidence constraints. Specifically, we embed the text of each raw insight with harrier-oss-v1-27b, using an instruction to identify the prediction task type, data modality, and application domain that the insight targets, and cluster the embeddings with UMAP (15 dimensions, 20 neighbors) and HDBSCAN (minimum cluster size 20, minimum samples 5); we select these hyperparameters by a grid search for clusters that are stable across random seeds, align with the data modality and solution stage of the insights, and cover most insights. This yields 15 clusters, and the outliers (4.9% of the raw insights) form the residual cluster, as shown in Figure 5. MLE-InsightForge then synthesizes each cluster into one consolidated improvement insight, including one for the residual cluster, resulting in 16 improvement insights. A manifest records the member insights and evidence of each cluster, so that every improvement insight remains traceable to its raw ones. 4.1.3. Insight Representation A competition insight or domain insight consists of insight items following the triggerāaction pattern: a trigger (a task or data condition that can be checked before acting), one recommended action, and a rationale for its expected effect. Triggers are phrased in terms of general task and data properties, so the agentic MLE system can skip the items that do not apply to its task; conflicting findings are kept as items with mutually exclusive triggers. The items are grouped by five aspects of an ML solution (data modality, data, model, training, and other) and ordered within each aspect by the cost of following them. An improvement insight is a single item whose trigger is the observed problem and whose action is the resolution, elaborated with its limitations, cost, and how to check whether it worked. Each insight has a name and a description; its evidence records are stored separately and enter the context only if explicitly read. 4.2. Usage of Insights 4.2.1. Base Agentic MLE System container with public data only Orchestrator MLE agent (ReAct loop) Solution tree Submission validator Insight skills task, public data,insights (playbook),insight descriptions (skill)loop[until final submission or budget exhausted]LLM call:reason and actexecute solutionnew node, val. scorevalidate submissionvalid / format errorsread skillinsight detailssubmit final nodegrading: attach private-test set, rerun solution, compute metrics Figure 6. Workflow of the agentic MLE system, with the insights highlighted.A sequence diagram with five lifelines: orchestrator, MLE agent, solution tree, submission validator, and insight skills. The orchestrator starts the agent with the task, public data, the domain insight, and the skills or playbook. In a loop, the agent calls the LLM, executes solutions that become nodes of the solution tree, validates submissions, and reads skills in skill mode. After the agent submits a final node, the orchestrator replaces the public-test inputs with the private-test ones, reruns the selected solution, and grades the submission with the private-test labels. The assembled insights can be injected into any agentic MLE system that accepts textual context. In our study, we inject them into a ReAct-style agentic MLE system that develops solutions in an isolated container with access to the public data of the target task, whose workflow is shown in Figure 6. The system can inspect data, edit files, execute candidate solutions, and validate submissions. Each execution is recorded as a node in a solution tree, so the system can branch from an earlier candidate when a modification fails, similar to AIDE (Jiang et al., 2025). At the deadline, the selected solution is rerun from scratch on the private-test inputs and graded outside the agent loop. Without insights, this system is the Baseline of our evaluation (Ā§5). 4.2.2. Insight Injection Modes current domainother domainsdomain insightimprovementinsightscompetitioninsightssource competitiontarget task Figure 7. Scope of the insights injected for a target task.A large circle represents the current domain and contains the target task, drawn as a star, and three , drawn as dots. Two smaller circles represent other domains, each containing three . Dashed arrows go from the the current domain to a green square labeled domain insight, and from the all domains to a red stack of rectangles labeled improvement insights; solid arrows go from the square, from the stack, and from each the current domain (labeled competition insights) to the target task. For a target task, the system receives the domain insight of the taskās domain, the competition insights of the source competitions in the same domain, and all improvement insights, which are assembled from the source competitions of every domain (Figure 7). The two injection modes hold the insights, system, prompt, harness, task, and budget constant, and only change when the competition insights and improvement insights enter the context; the domain insight is always in the initial context. Skill mode. Following the progressive-disclosure design of Agent Skills (Anthropic, 2025), the initial context lists only compact metadata (name and short description) of each available insight. This mode preserves context for task observations and execution feedback, but its benefit depends on the system recognizing a relevant insight from its description and retrieving it. Playbook mode. Following the use of evolving playbooks as agent context (Zhang et al., 2026a), the full texts of all available insights are rendered directly into the system prompt before the system begins. This mode removes the retrieval decision and makes every insight immediately available, at the cost of a longer context that may contain insights irrelevant to, or inconsistent with, the target task. 4.2.3. Application to New Competitions The assembled insights are not tied to MLE-InsightBench: a new competition from outside it can use them in the same way as a target task. For one or a few new competitions, the only step needed is to determine the domain of each competition, either manually or by embedding its description as in Domain Clustering (Ā§3) and assigning it to the closest cluster; the domain insight and competition insights of that domain and all improvement insights are then directly usable without further assembly. Once a new competition has ended and its writeups and solutions are available, it can be added as a source competition by an incremental update that reruns only the affected assembly tasks. MLE-InsightForge assembles the competition insight of the new competition and re-synthesizes the domain insight of its domain; if its writeups yield new raw improvement insights, they are assigned to the existing clusters, and only the improvement insights of the clusters that receive them are re-synthesized. Re-clustering is needed only when many competitions have been added, which may shift the cluster structure, or when a new competition belongs to no existing domain. 4.3. Running Example Domain insight name: Time Series Prediction description: This category covers temporal prediction from panels of geographic rates, regional cumulative outcomes, and hierarchical item-location counts. [ā¦] ## DATA -With a fixed multi-step horizon, 1 use target lags and rolling baselines anchored at least one horizon behind each target because this exposes recurring structure without future labels or recursive baseline error. -When a rate targetās denominator changes over time, 2 normalize historical rates to the denominator available for the forecast period because denominator revisions otherwise masquerade as target dynamics. -When external features arrive with reporting lags, 3 align each backtest value to its actual publication vintage because later releases in earlier folds create forward bias. [ā¦] Competition insight name: godaddy-microbusiness-density-forecasting description: Monthly U.S. county records provide [ā¦] The task is to produce static forecasts of future monthly microbusiness density for every county. ## DATA -When a rate targetās denominator changes across years, 2 normalize all historical rates to the denominator expected at forecast time because otherwise population revisions masquerade as business dynamics. -When entities have tiny counts or nearly flat histories, use last-value forecasts for that segment because [ā¦] -When external annual or monthly covariates are merged into backtests, 3 align each value to its real publication vintage because later census values in earlier folds created forward bias. [ā¦] Improvement insight name: deployment-contract-temporal-backtesting description: When a forecasting or sequential prediction task is served at future origins and ordinary batch validation differs from the serving contract in available history, eligible queries, state-update order, or forecast-horizon coverage. When a forecasting or sequential task has batch validation that differs from future serving in available history, eligible queries, state updates, or horizon coverage, gate every candidate with one deployment-contract backtest replayed across several forecast origins, because 3 matching the as-of information boundary and full served horizon removed leakage-driven rankings and made model selection more stable across later periods. [ā¦] ## Example contract = read_serving_contract(...) [ā¦] (a) Excerpts of the three types of insights available for the task. Without insights MAEā : 29.24 (validation), 234.15 (private-test) @@ features @@ LAGS = [48, 72, 96, ..., 1008] # hours 1 ā out[fārmean_wā] = piv.rolling(w).mean() ā .shift(2, freq=āhā) 3 ā df[āclient_dateā] = df[ādatetimeā] ā .dt.normalize() ā df = df.merge(c_agg, on=[..., āclient_dateā]) @@ training @@ 2 ā m.fit(sub[feature_cols], sub[ātargetā]) @@ prediction through the time-series API @@ for (test, revealed_targets, client_b, ...) in iter_test: 3 ā t = t.merge(fl.c_agg, ā on=[..., āclient_dateā]) ā t[num_cols] = t[num_cols].fillna(0) env.predict(...) With insights MAEā : 31.91 (validation), 57.12 (private-test) @@ features @@ LAGS = [48, 72, 96, ..., 1080] # hours 1 + d[āt_mean7dā] = np.nanmean( + d[[fāt_laghā for h in LAGS[:7]]], axis=1) 3 + client[āas_of_dateā] = client[ādateā] + + pd.Timedelta(days=2) + d = d.merge(client, on=[..., āas_of_dateā]) @@ training @@ 2 + d[ātarget_ratioā] = d[ātargetā] + / d[āinstalled_capacityā] + y = trm[ātarget_ratioā] # production @@ prediction through the time-series API @@ for (test, revealed, client, ...) in iter_test: 3 + c[āas_of_dateā] = c[ādateā] + + pd.Timedelta(days=2) + test = test.merge(c, on=[..., āas_of_dateā]) 2 + pred = pred * test[āinstalled_capacityā] env.predict(...) (b) Excerpts of the final solutions without and with insights; numbered changes relate to (a). Figure 8. Running example on Energy Behavior of Prosumers.Two parts. Part (a) shows three side-by-side insight files, each with a gray front matter holding its name and description, followed by its content. The Time Series Prediction domain insight has three items under the DATA heading, each stating a condition, an action, and a rationale; the highlighted actions are (1) to use target lags and rolling baselines anchored at least one horizon behind each target, (2) to normalize historical rates to the denominator available for the forecast period, and (3) to align each backtest value to its actual publication vintage. The GoDaddy Microbusiness Density Forecasting competition insight has three items under the DATA heading; the highlighted actions are (2) to normalize all historical rates to the denominator expected at forecast time and (3) to align each value to its real publication vintage, whereas the item on entities with tiny counts is not highlighted. The deployment-contract-temporal-backtesting improvement insight recommends gating every candidate with a deployment-contract backtest replayed across several forecast origins, with the highlighted rationale (3) of matching the as-of information boundary. Part (b) shows a side-by-side diff of two code excerpts with three hunks: features, training, and prediction through the time-series API. The solution without insights, with a validation MAE of 29.24 and a private-test MAE of 234.15, computes rolling target means shifted by only two hours, joins the client data of the same day, trains on the raw target, and at prediction time merges client data from the training period only and fills missing features with zeros. The solution with insights, with a validation MAE of 31.91 and a private-test MAE of 57.12, computes rolling means from target lags of at least 48 hours (change 1), divides the production target by the installed capacity in training and multiplies the predictions by it (change 2), and joins the client data as of two days earlier, both in training and from the data served by the API (change 3). We illustrate the assembly and usage of insights with the competition Energy Behavior of Prosumers, the target task of the Time Series Prediction domain. The task asks for hourly forecasts of the electricity produced and consumed by Estonian customers with solar panels, given weather forecasts, energy prices, and client data such as the installed photovoltaic capacity, and is evaluated by mean absolute error (MAE). Predictions are made through a time-series API that serves the test data day by day, in which the latest revealed targets and client data lag two days behind the prediction times. The four source competitions of the domain forecast COVID-19 cases, microbusiness density, and retail sales, and none of them involves energy. Insights. Figure 8(a) shows excerpts of the three types of insights available for this task. Three aspects of the domain insight have triggers that hold for Energy Behavior of Prosumers: 1 target lags anchored at least one horizon behind each target match the two-day delay of the revealed targets, 2 the denominator normalization matches the production that scales with the installed capacity, which grows over time, and 3 aligning external features to their publication vintage matches the two-day delay of the client data. The competition insight of GoDaddy Microbusiness Density Forecasting (Howard et al., 2022a) contains the items from which 2 and 3 are generalized, whereas other items, such as the one on entities with tiny counts, have triggers that do not hold and can be skipped. The improvement insight deployment-contract-temporal-backtesting recommends accepting a candidate solution only if it improves in a backtest that replays the serving contract, i.e., the available history, the gap, and the horizon, across several forecast origins, which requires 3 matching the information available at each origin. Impact on the solutions. Over six runs each, the agentic MLE system reaches a private-test MAE of 228 Ā± 76.8 without insights and 64.8 Ā± 6.15 with insights (lower is better, since MAE measures the prediction error). Figure 8(b) compares the final solutions of a run without insights and a run with insights, in which all three insights are injected via the playbook mode. The solution without insights anchors its rolling target means only two hours before each prediction time and joins the client data of the same day, although the API reveals the targets and client data only two days later. Its hold-out validation on the training data cannot detect this mismatch and reports an MAE of 29.24; at prediction time, the rolling means and the capacity are missing (the client data served by the API is ignored), filled with zeros, and the private-test MAE rises to 234.15. The solution with insights makes three changes that implement the highlighted insight parts: 1 it computes its rolling means only from target lags of at least 48 hours, 2 it learns the production per unit of installed capacity and rescales its predictions by the capacity, and 3 it joins the client data as of two days earlier, both in training and from the data served by the API. Its validation MAE (31.91) is worse but reflects the deployment setting, and its private-test MAE is 57.12. 5. Evaluation We answer the following research questions: ā¢ RQ1: How much do the assembled insights improve the solutions of an agentic MLE system? ā¢ RQ2: How do the three types of insights contribute to the improvement? ā¢ RQ3: How do the two injection modes, skills and playbooks, compare in effectiveness and cost? ā¢ RQ4: Do the improvements generalize across backend LLMs? 5.1. Experimental Setup Target tasks and system. We evaluate on the 12 held-out target tasks of MLE-InsightBench (Table 1), using the insights that MLE-InsightForge assembled from the 148 source competitions. The base agentic MLE system is backed by deepseek-v4-flash-0731. Each run receives one NVIDIA RTX A6000 GPU (48G of RAM built with cuda_11.5.r11.5), four AMD EPYC 7343 CPUs (16 cores @ 1.5GHz), a 12-hour development budget, and at most 250 agent steps. Table 2. Insight injection modes of each condition. Mode Domain Competition Improvement Baseline ā ā ā Full-skill playbook skill skill Full-playbook playbook playbook playbook D-playbook playbook ā ā C-playbook ā playbook ā I-playbook ā ā playbook Conditions. Table 2 lists the six conditions in our experiments, combining different injection modes. Baseline receives no insight. Full-skill and Full-playbook inject all three types of insights, as skills and as a playbook, respectively (Ā§4.2.2). Specially, domain insights are always injected as playbook since they are short and most relevant. The three ablation conditions, D-playbook, C-playbook, and I-playbook, each inject a single type of insight (domain insights, competition insights, and improvement insights, respectively) as a playbook. Metrics. The primary metric is the private-test score of the final solution, computed by the taskās MLE-bench-compatible grader. Because raw scores are not comparable across MLE-InsightBench tasks, we mināmax normalize the scores of each task on a fixed range, after reversing lower-is-better metrics, and compute the average improvement over the baseline for each mode across all tasks: (1) Ī=ĪmāĪBāaāsāeālāiānāeĪBāaāsāeālāiānāeĆ100%,Īm=1Tāāt=1T1Kāāk=1KNm,t,k,Nm,t,k=xm,t,kāLtUtāLt, = _m- _Baseline _BaselineĆ 100\%, _m= 1T _t=1^T 1K _k=1^KN_m,t,k, N_m,t,k= x_m,t,k-L_tU_t-L_t, where Īm _m and ĪBāaāsāeālāiānāe _Baseline denote the average normalized scores of mode m and Baseline over T tasks (T=12T=12) and K repetitions (K=6K=6), Nm,t,kN_m,t,k and xm,t,kx_m,t,k are the normalized and raw scores of mode m on task t in the k-th run, and LtL_t and UtU_t are the worst and best scores over the runs of Baseline, Full-skill, and Full-playbook on task t. The same range is used for every condition to put all conditions on one scale, so an ablation run outside it may be normalized to below 0 or above 1. To measure cost, we record the input, reasoning, and output tokens consumed by the backend LLM and the wall-clock time of each run. We convert the tokens to a monetary cost at the OpenRouter API prices of deepseek-v4-flash-0731: $0.44 per million uncached input tokens, $0.014 per million cached input tokens, and $1.32 per million output tokens, where the output tokens include the reasoning tokens. For each task and condition, we report the mean with a 95% Student-t confidence interval, and compare each condition with Baseline using a two-sided MannāWhitney U test (Mann and Whitney, 1947) at Ī±=0.05Ī±=0.05. Since each condition is compared with Baseline on 12 tasks, we also report the number of significant tasks after the HolmāBonferroni correction (Holm, 1979) over the tasks of each condition. Every condition has 72 runs, i.e., six runs per task. In addition to the private-test score of the final solution, we analyze the validation scores that the system measures on its own validation split for every scored candidate solution during development, which show how the solution quality evolves over time. The validation scores are normalized in the same way as private-test scores; at each minute, we take the median over the runs of a task and average it over tasks. Table 3. Overall results. Baseline Full-skill Full-playbook D-playbook C-playbook I-playbook Normalized private-test score 0.302 0.676 0.673 0.595 0.608 0.611 Change from Baseline ā 123.9% 122.8% 96.9% 101.4% 102.1% Normalized validation score 0.468 0.575 0.548 0.560 0.600 0.572 Per-task input tokens 24.0M 25.9M 30.1M 27.3M 25.4M 26.7M Per-task reasoning tokens 20.4K 68.1K 66.6K 77.9K 77.8K 74.4K Per-task output tokens 114.9K 123.3K 129.1K 250.1K 258.2K 231.9K Per-task time [min] 228 292 297 335 337 282 Total time [day] 11.4 14.6 14.9 16.7 16.9 14.1 Per-task cost $1.72 $1.88 $1.95 $2.00 $1.74 $1.77 Total cost $124 $135 $141 $144 $125 $127 5.2. RQ1: Effectiveness of Insights Table 4. Per-task change of the normalized score from Baseline (bold: significant). Competition Full-skill Full-playbook D-playbook C-playbook I-playbook Ariel Data Challenge 2025 186.6% 193.8% 154.4% 180.0% 232.2% CAFA 5 Protein Function 116.2% 198.4% 162.0% 118.3% 91.6% CSIRO Biomass 176.6% 193.5% 151.5% 123.4% 214.6% Energy Behavior of Prosumers 149.4% 148.9% 106.7% 132.9% 112.5% Image Matching 2025 680.8% 613.4% 174.2% 430.5% 169.9% Jigsaw Community Rules 236.9% 215.4% 155.3% 153.1% 197.4% March Mania 2025 110.6% 107.5% 79.8% 110.7% 76.3% Mitsui Commodity Prediction 229.2% 282.8% 248.0% 39.4% 248.4% NFL Player Contact 209.6% 243.0% 195.4% 186.3% 237.1% POG Series S01E01 7.2% -24.2% 3.8% -1.4% -12.7% PhysioNet ECG Digitization 118.0% 67.9% 83.2% 150.5% 116.3% RSNA Aneurysm Detection 84.7% 100.5% 78.6% 82.0% 63.6% Tasks improved 12 11 12 11 11 Tasks improved significantly 11 11 9 8 8 Four line charts over development minutes. (a) The best validation score found so far: Full-skill ends highest at about 60, Full-playbook at about 58, and Baseline at about 55; Full-skill and Full-playbook are above Baseline from the start. (b) The gain over Baseline's first solution: Full-skill and Full-playbook start at about 2 and 3 points and end at about 8 and 6 points, and Baseline starts at 0 and ends at about 3 points. (c) All three single-type conditions start above Baseline and end with gains of about 7 to 8 points, compared with about 4 points for Baseline. (d) The difference to Baseline is positive throughout for all three single-type conditions, at about 2 to 3 points in the first 100 minutes; C-playbook ends highest at about 6 points, I-playbook at about 4, and D-playbook at about 3. Figure 9. Validation scores during development; gains are over the first solution of Baseline.Four line charts over development minutes. (a) The best validation score found so far: Full-skill ends highest at about 60, Full-playbook at about 58, and Baseline at about 55; Full-skill and Full-playbook are above Baseline from the start. (b) The gain over Baseline's first solution: Full-skill and Full-playbook start at about 2 and 3 points and end at about 8 and 6 points, and Baseline starts at 0 and ends at about 3 points. (c) All three single-type conditions start above Baseline and end with gains of about 7 to 8 points, compared with about 4 points for Baseline. (d) The difference to Baseline is positive throughout for all three single-type conditions, at about 2 to 3 points in the first 100 minutes; C-playbook ends highest at about 6 points, I-playbook at about 4, and D-playbook at about 3. As shown in Tables 3 and 4, injecting the assembled insights improves the normalized score from 0.302 for Baseline to 0.676 for Full-skill (123.9%) and 0.673 for Full-playbook (122.8%). The improvement is consistent across tasks: Full-skill improves the mean score on 12 of the 12 tasks, significantly so on 11 of them (11 after the HolmāBonferroni correction), with POG Series S01E01 as the only task where the improvement is not significant; Full-playbook improves on 11 tasks, significantly on 11 (10 after the correction). The largest gains occur on tasks, such as Image Matching 2025 and Energy Behavior of Prosumers, where the validation design and task-specific modeling choices documented in peer competitions are consequential. Figure 9(a, b) shows that the insights help both in drafting the initial solution and in improving it: the first scored solutions of Full-skill and Full-playbook are 1.8 and 2.9 points above those of Baseline; they reach the final best score of Baseline after 80 and 40 minutes, and end with gains of 7.8 and 5.9 points over the first solution of Baseline, compared with 3.3 points for Baseline itself. Answer to RQ1: The assembled insights improve the solution quality of the agentic MLE system by 123.9% (Full-skill) and 122.8% (Full-playbook) in normalized score, with Full-skill improving on 12 of the 12 target tasks; the system with insights starts from better solutions and improves them faster. 5.3. RQ2: Contribution of Insight Types We compare Full-playbook with the single-type ablations (last three columns of Table 3). (a) Baseline, Full-skill, and Full-playbook (b) Baseline and the single-type ablations Figure 10. Private-test items correctly predicted only by the enclosing conditions.Two Venn diagrams. (a) Baseline, Full-skill, and Full-playbook: 509,717 items are predicted correctly by all three; 3,838 only by Baseline, 5,376 only by Full-skill, 4,918 only by Full-playbook, and 8,786 by Full-skill and Full-playbook but not Baseline. (b) Baseline and the three single-type ablations: 509,377 items are predicted correctly by all four; 1,184 only by Baseline, 1,238 only by D-playbook, 12,345 only by C-playbook, and 4,566 only by I-playbook. Each type of insight alone improves the normalized score over Baseline (0.302): to 0.595 with domain insights (96.9%), 0.608 with competition insights (101.4%), and 0.611 with improvement insights (102.1%). All three remain below Full-playbook (0.673). Hence, the three types are complementary rather than redundant, and their benefits accumulate when combined. The accumulation is not additive, but the combination is more robust: the best single type varies across tasks (e.g., I-playbook on Ariel Data Challenge 2025 and CSIRO Biomass, and C-playbook on PhysioNet ECG Digitization), whereas Full-playbook has a higher mean score than all three single-type conditions on 7 of the 12 tasks. They are also less consistent: their improvement is significant on only 9, 8, and 8 tasks (2, 4, and 0 after the correction), compared with 11 (10) for Full-playbook (Table 4). Figure 9(c, d) shows the validation scores of the ablation modes during development. All three single-type modes are ahead of Baseline from their first scored solution on, by similar margins in the first 100 minutes (3.0, 2.2, and 3.1 points); afterwards, C-playbook pulls ahead and ends 6.1 points above Baseline, compared with 4.4 and 3.2 points for I-playbook and D-playbook. On the five tasks graded on per-item correctness (CAFA 5, Jigsaw, March Mania, NFL, and RSNA), we also compare the private-test items that each condition predicts correctly, i.e., that at least four of its six runs predict correctly. Each type also yields private-test items that no other condition predicts correctly (Figure 10(b)): 12,345 with competition insights, 4,566 with improvement insights, and 1,238 with domain insights, compared with 1,184 for Baseline. Answer to RQ2: The three types of insights are complementary: each type alone improves the score, but significantly on fewer tasks and less than the combination of all three types; competition insights contribute the most to the late-stage improvements and to the predictions that no other condition gets right. 5.4. RQ3: Comparison of Injection Modes Overall, the two injection modes reach similar normalized scores (0.676 for Full-skill and 0.673 for Full-playbook in Table 3), but the better mode varies across tasks. Full-playbook has the higher mean score on 6 tasks and Full-skill on 6; notably, Full-playbook falls below Baseline on POG Series S01E01 (though not significantly), whereas Full-skill stays above it. A playbook guarantees that every insight is seen but can steer the system toward insights that do not fit the task, whereas skills let the system skip irrelevant insights but depend on it retrieving the relevant ones. Although Full-skill reaches a similar score to Full-playbook, its trajectories show that the system rarely retrieves the full text of a skill: only 19 of its 72 runs call read_skill at all, 14 of them to read a competition insight (typically that of the most similar source competition, e.g., Image Matching Challenge 2024 (Bellavia et al., 2024) for Image Matching 2025) and 5 to read an improvement insight. The improvement of Full-skill therefore stems mostly from what every run receives in its initial context rather than from the retrieved texts: the domain insight, with which D-playbook alone reaches 0.595, and the metadata of the skills, whose descriptions state when each insight applies, e.g., the problem that each improvement insight resolves. The two modes also lead to different correct predictions (Figure 10(a)): 5,376 private-test items are predicted correctly only by Full-skill and 4,918 only by Full-playbook, compared with 3,838 only by Baseline. The two modes differ in cost. Relative to Baseline, Full-skill increases the per-task input tokens by only 7.7%, whereas Full-playbook increases them by 25.2%, because the playbook is carried in the context of every model call; the output tokens and time are similar (Table 3). The per-task cost is only $1.72, $1.88, and $1.95, respectively, as most input tokens hit the prompt cache. Answer to RQ3: The better injection mode varies across tasks; overall, skills are more cost-effective, reaching a similar score to playbooks with 7.7% instead of 25.2% additional input tokens. 5.5. RQ4: Generalizability across LLMs We repeat Baseline, Full-skill, and Full-playbook using GPT-6.1-Sol, a proprietary LLM perceived to be stronger than deepseek-v4-flash-0731, on two of the target tasks: Image Matching 2025, where the insights bring the largest gain (680.8% with Full-skill, 613.4% with Full-playbook), and POG Series S01E01, the only task with no significant improvement (7.2% with Full-skill, -24.2% with Full-playbook). We use the same insights and six runs per task. Table 5. Private-test scores (meanĀ± 95% CI) with two backend LLMs (bold: significant). Competition LLM Baseline Full-skill Full-playbook Image Matching 2025 (mAAā ) deepseek-v4-flash-0731 21.1Ā± 1.8 35.2Ā± 6.7 33.8Ā± 5.4 GPT-6.1-Sol 42.8Ā± 1.6 52.0Ā± 9.3 49.9Ā± 10.5 POG Series S01E01 (MAEā ) deepseek-v4-flash-0731 0.0255Ā± 0.0009 0.0249Ā± 0.0004 0.0273Ā± 0.0031 GPT-6.1-Sol 0.0261Ā± 0.0014 0.0251Ā± 0.0006 0.0253Ā± 0.0005 Overall Change from Baseline deepseek-v4-flash-0731 ā 61.8% 27.5% GPT-6.1-Sol ā 80.3% 61.9% Total time [day] deepseek-v4-flash-0731 1.7 2.1 2.5 GPT-6.1-Sol 3.6 3.7 3.9 Total cost deepseek-v4-flash-0731 $22 $21 $22 GPT-6.1-Sol $153 $128 $160 As shown in Table 5, the insights are not specific to one backend LLM: on the normalized scale over both tasks, they also improve GPT-6.1-Sol by 80.3% with Full-skill and 61.9% with Full-playbook. GPT-6.1-Sol is a stronger backend than deepseek-v4-flash-0731 on Image Matching 2025: its Baseline already reaches an mAA of 42.8, close to the gold-medal threshold of the human leaderboard (43.34). Although the improvement is smaller than with deepseek-v4-flash-0731, the insights still significantly improve the normalized score by 427.9% with Full-skill and 326.9% with Full-playbook, lifting the mean mAA beyond the gold-medal threshold (52.0 and 49.9). On POG Series S01E01, where the insights do not help deepseek-v4-flash-0731, they improve the normalized score of GPT-6.1-Sol by 38.2% with Full-skill and 29.8% with Full-playbook, although not significantly over six runs; this suggests that a stronger backend LLM can make better use of the insights. Still, no condition with either backend LLM reaches the median of the human leaderboard on this task (MAE 0.016845), so the tasks of MLE-InsightBench remain challenging for future agentic MLE systems. Unlike deepseek-v4-flash-0731, GPT-6.1-Sol actively retrieves skills: 12 of its 12 Full-skill runs call read_skill (12 for improvement insights and 6 for competition insights), which may explain why Full-skill beats Full-playbook with GPT-6.1-Sol on both tasks. The two backend LLMs also develop solutions differently: on Image Matching 2025, GPT-6.1-Sol creates its first solution node earlier (median step 19 vs. 165), uses most of the 12-hour budget (594ā629 minutes), and tries 9.2ā13.3 solution nodes per run, whereas deepseek-v4-flash-0731 usually submits its final solution after 210ā320 minutes with 2.5ā4.0 nodes. Answer to RQ4: The improvements from insights generalize to GPT-6.1-Sol, a stronger proprietary LLM from a different model family: the insights improve its normalized score by 80.3% (Full-skill) and 61.9% (Full-playbook), lifting it beyond the gold-medal threshold on Image Matching 2025. Still, some MLE tasks remain challenging to LLM agents: on POG Series S01E01, neither deepseek-v4-flash-0731 nor GPT-6.1-Sol reaches the median of the human leaderboard, with or without insights. 6. Limitations Task scope. MLE-InsightBench is derived from Kaggle competitions, which provide realistic end-to-end MLE tasks but do not represent every production setting. Kaggle emphasizes predictive performance under a fixed metric and submission interface, whereas deployed machine learning systems also require maintainability, monitoring, fairness, and privacy. Domain construction. The domains of MLE-InsightBench are induced by clustering competition descriptions. Other representations or clustering choices could change which competitions are considered closely related and thus contribute insights to a target task. Insight quality. The insights are assembled by an LLM-based agent. To reduce the risk of hallucination and misrepresentation, MLE-InsightForge must read every required source in full and back each insight with an evidence record, which links the insight to the human-written passages it is based on and makes it traceable. Leakage beyond the benchmark. MLE-InsightBench excludes the materials of every target task from the insight sources, but it cannot control what the backend LLMs have learned during pre-training. Because Kaggle writeups and notebooks are public, a backend LLM may have seen solutions of the target tasks; this affects all conditions equally, including Baseline, and therefore may inflate absolute scores. Selecting the most recent eligible competition of each domain as the target task reduces, but does not eliminate, this risk. Models and costs. The main results use one backend LLM for the agentic MLE system and one LLM for MLE-InsightForge, so they may depend on these modelsā knowledge, tool-use behavior, and long-context capabilities; RQ4 studies the former dependency. The 12-hour, GPU-backed budget also limits accessibility and may favor approaches that spend more compute. Assembling the insights from all 148 source competitions with gpt-5.6-sol-pro cost $903 USD ($1,544 USD including runs repeated during development), a one-time cost that is amortized over all tasks of a domain. The experiments are also costly: with deepseek-v4-flash-0731, each condition takes 11.4ā16.9 GPU days and $124ā$144 in API costs, and with GPT-6.1-Sol, the API costs of a condition on all 12 target tasks would reach $957, extrapolated from up to $160 on two tasks. Running MLE-InsightForge with a different backend LLM may yield different findings, which our budget precludes evaluating; to facilitate reproduction and such evaluations by others, we open-source our workflow and insights. 7. Related Work Agentic MLE. MLAgentBench (Huang et al., 2024) evaluates LLM agents that design, run, and revise ML experiments, and MLE-bench (Chan et al., 2025) evaluates end-to-end solutions of Kaggle competitions against private-test labels. The systems on the MLE-bench leaderboard mostly advance the search over candidate solutions (Jiang et al., 2025; Liu et al., 2025; Toledo et al., 2025; Li et al., 2025; Wan et al., 2025; Yang et al., 2025) and, increasingly, the use of insights from other tasks (Ā§2.2). Systems that use insights couple a particular source of insights with a particular agent scaffold, and evaluate the combination as a whole on MLE-bench. Our work is complementary: rather than proposing another scaffold, we hold the agentic MLE system fixed and study how the assembled insights, their types, and their injection modes affect it on a benchmark whose source competitions are fixed and disjoint from the target tasks. Program synthesis for ML code. Before LLM agents, program synthesis targeted ML code by searching a restricted space of programs. BLAZE (Wang et al., 2018) synthesizes tensor transformations in a MATLAB-like DSL from inputāoutput examples, and TF-Coder (Shi et al., 2022) enumerates expressions over TensorFlow operations, prioritized by learned models; INTENT (Zhou et al., 2022) lets users decompose hard tensor transformations, and HySynth (Barke et al., 2024) guides the search with a grammar learned from LLM completions. Nam et al. (Nam et al., 2022) predict the sequence of PyTorch and TensorFlow API calls that transforms the inputs into the outputs to guide enumerative search. At a coarser granularity, AutoML-Zero (Real et al., 2020) evolves complete ML algorithms from basic mathematical operations, NEAR (Shah et al., 2020) and dPADS (Cui and Zhu, 2021) search for differentiable programs in a DSL, and Marinescu et al. (Marinescu et al., 2021) search ML pipelines defined by a context-free grammar. These techniques synthesize short programs or pipelines within a predefined search space, whereas an agentic MLE system writes an end-to-end solution in a general-purpose language; the insights assembled by MLE-InsightForge play a role similar to their learned priors, steering the system toward decisions that worked on related tasks. Experience, memory, and context for LLM agents. Reflexion (Shinn et al., 2023) stores verbal feedback from earlier trials of the same task, ExpeL (Zhao et al., 2024) extracts natural-language lessons from successful and failed trajectories and retrieves them for new tasks, and Agent Workflow Memory (Wang et al., 2025) induces reusable workflows from past trajectories. ACE (Zhang et al., 2026a) treats the agentās context as an evolving playbook that is incrementally refined from execution feedback, and Agent Skills (Anthropic, 2025) package instructions and resources that the agent loads on demand. These approaches mainly learn from the agentās own trajectories. MLE-InsightForge instead assembles insights from human-written solutions and writeups of peer tasks, organizes them into domain, competition, and improvement insights, and compares the skill and playbook modes of injecting the same insights. Mining Kaggle artifacts. DistilKaggle (Mostafavi Ghahfarokhi et al., 2024) curates Kaggle notebooks together with code-quality measures, and Jin et al. (Jin et al., 2026) mine Kaggle notebooks and use MLE-bench-style grading to study their reproducibility under modern execution environments. We use Kaggle descriptions, writeups, and top solutions as sources from which to assemble insights for solving unseen tasks. Data contamination in benchmarks. Contamination of evaluation data threatens the validity of LLM benchmarks, and LiveCodeBench (Jain et al., 2025) mitigates it by continuously collecting problems released after the modelsā training cutoffs. For agentic MLE systems, the relevant contamination also comes from insight sources available at solving time (Ā§2.3); MLE-InsightBench controls these sources by construction, using only causal non-descendants of the target tasks as sources. 8. Conclusion We studied how insights assembled from closely-related MLE tasks affect an agentic MLE system. We collected MLE-InsightBench, whose held-out split of 160 Kaggle competitions in 12 domains keeps target task information out of the insights, and designed MLE-InsightForge, an agent that assembles evidence-backed domain insights, competition insights, and improvement insights from the source competitions and injects them as skills or a playbook. The assembled insights improve the normalized private-test score by 123.9% as skills and 122.8% as a playbook; the three types of insights are complementary, skills are more cost-effective, and the improvements generalize. These findings suggest treating insight assembly as an explicit component of agentic MLE systems whose sources are controlled in evaluations. References Amershi et al. (2019) S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, and T. Zimmermann Software engineering for machine learning: a case study. In International Conference on Software Engineering, Software Engineering in Practice, p. 291ā300. External Links: Document, Link Cited by: Ā§1. Anthropic (2025) Anthropic Equipping agents for the real world with Agent Skills. Note: https://w.anthropic.com/engineering/equipping-agents-for-the-real-world-with-agent-skillsAccessed September 27, 2026 Cited by: Ā§1, Ā§2.2, Ā§4.2.2, Ā§7. Barke et al. (2024) S. Barke, E. Anaya Gonzalez, S. R. Kasibatla, T. Berg-Kirkpatrick, and N. Polikarpova HYSYNTH: context-free LLM approximation for guiding program synthesis. In Conference on Neural Information Processing Systems, Vol. 37. External Links: Link Cited by: Ā§7. Bellavia et al. (2024) F. Bellavia, J. Matas, D. Mishkin, L. Morelli, F. Remondino, W. Sun, A. Tabb, E. Trulls, K. M. Yi, S. Dane, and A. Chow Image matching challenge 2024 - hexathlon. Note: https://w.kaggle.com/competitions/image-matching-challenge-2024Kaggle Cited by: Ā§5.4. Bellavia et al. (2025) F. Bellavia, J. Matas, D. Mishkin, L. Morelli, F. Remondino, A. Tabb, E. Trulls, K. M. Yi, S. Dane, A. Howard, and M. Cruz Image matching challenge 2025. Note: https://w.kaggle.com/competitions/image-matching-challenge-2025Kaggle Cited by: Table 1. Campello et al. (2013) R. J. G. B. Campello, D. Moulavi, and J. Sander Density-based clustering based on hierarchical density estimates. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, p. 160ā172. External Links: Link Cited by: Ā§3. Chan et al. (2025) J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. MÄ dry MLE-bench: evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, External Links: Link Cited by: Ā§1, Ā§2.1, Ā§7. Chen et al. (2026) J. Chen, B. Dalvi Mishra, J. Nam, R. Meng, T. Pfister, and J. Yoon MARS: modular agent with reflective search for automated AI research. In International Conference on Machine Learning, External Links: Link Cited by: Ā§1, Ā§1, 1st item, Ā§2.3. cnyberg et al. (2020) cnyberg, d_engi, janestreet-jjia, Maggie, and W. Cukierski Jane street market prediction. Note: https://w.kaggle.com/competitions/jane-street-market-predictionKaggle competition Cited by: Ā§3. Cui and Zhu (2021) G. Cui and H. Zhu Differentiable synthesis of program architectures. In Conference on Neural Information Processing Systems, Vol. 34. External Links: Link Cited by: Ā§7. Demkin et al. (2025) M. Demkin, N. Takano, R. Rai, S. Dane, and T. Kitayama MITSUI&CO. commodity prediction challenge. Note: https://w.kaggle.com/competitions/mitsui-commodity-prediction-challengeKaggle Cited by: Table 1. Du et al. (2025) S. Du, X. Yan, D. Jiang, J. Yuan, Y. Hu, X. Li, L. He, B. Zhang, and L. Bai AutoMLGen: navigating fine-grained optimization for coding agents. Note: arXiv:2510.08511 External Links: Link Cited by: Ā§1, Ā§1, 1st item, Ā§2.3. Du et al. (2026) S. Du, X. Yan, J. Shi, Z. Cao, S. Feng, Z. Liang, B. Sun, T. Peng, Y. Zhou, X. Li, et al. MLEvolve: a self-evolving framework for automated machine learning algorithm discovery. Note: arXiv:2606.06473 External Links: Link Cited by: Ā§1, 1st item, Ā§2.3, Ā§2.3. Eljand et al. (2023) K. Eljand, M. Laid, J. Scellier, S. Dane, M. Demkin, and A. Howard Enefit ā predict energy behavior of prosumers. Note: https://kaggle.com/competitions/predict-energy-behavior-of-prosumersKaggle competition Cited by: Table 1. Friedberg et al. (2023) I. Friedberg, P. Radivojac, C. D. Paolis, D. Piovesan, P. Joshi, W. Reade, and A. Howard CAFA 5 protein function prediction. Note: https://w.kaggle.com/competitions/cafa-5-protein-function-predictionKaggle Cited by: Table 1. Frieder et al. (2024) S. Frieder, S. Bealing, A. Nikolaiev, G. C. Smith, K. Buzzard, T. Gowers, P. J. Liu, P. Loh, L. Mackey, L. de Moura, D. Roberts, D. Sculley, T. Tao, D. Balduzzi, S. Coyle, A. Gerko, R. Holbrook, A. Howard, and XTX Markets AI mathematical olympiad ā progress prize 2. Note: https://w.kaggle.com/competitions/ai-mathematical-olympiad-progress-prize-2Kaggle competition Cited by: Ā§2.3. Grootendorst (2022) M. Grootendorst BERTopic: neural topic modeling with a class-based TF-IDF procedure. Note: arXiv:2203.05794 External Links: Link Cited by: Ā§3. Grosnit et al. (2024) A. Grosnit, A. Maraval, R. S N, Z. Zhao, J. Doran, G. Paolo, A. Thomas, J. Gonzalez, A. Kumar, K. Khandelwal, et al. Kolb-based experiential learning for generalist agents with human-level Kaggle data science performance. Note: arXiv:2411.03562 External Links: Link Cited by: Ā§1, 2nd item. Grotov et al. (2022) K. Grotov, S. Titov, V. Sotnikov, Y. Golubev, and T. Bryksin A large-scale comparison of Python code in Jupyter notebooks and scripts. In International Working Conference on Mining Software Repositories, p. 353ā364. External Links: Document, Link Cited by: Ā§1. Guo et al. (2024) S. Guo, C. Deng, Y. Wen, H. Chen, Y. Chang, and J. Wang DS-agent: automated data science by empowering large language models with case-based reasoning. In International Conference on Machine Learning, p. 16813ā16848. External Links: Link Cited by: 1st item. Holbrook et al. (2025) R. Holbrook, D. Sterling, P. Mooney, M. Demkin, and E. Park Drawing with llms. Note: https://w.kaggle.com/competitions/drawing-with-llmsKaggle Cited by: Ā§3. Holm (1979) S. Holm A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6 (2), p. 65ā70. External Links: Link Cited by: Ā§5.1. Howard et al. (2022a) A. Howard, A. Agarwal, A. Chow, HCL-Rantig, K. J. Gracey, R. J. Brown, and S. Dane GoDaddy - microbusiness density forecasting. Note: https://w.kaggle.com/competitions/godaddy-microbusiness-density-forecastingKaggle Cited by: Ā§4.3. Howard et al. (2022b) A. Howard, B. Evans, R. Mulla, R. Holbrook, S. Huddleston, and W. Cukierski 1st and future ā player contact detection. Note: https://w.kaggle.com/competitions/nfl-player-contact-detectionKaggle Cited by: Table 1. Howard et al. (2023) A. Howard, A. I. Humayun, A. Chow, R. Holbrook, Sushmit, and Tahsin Bengali.ai speech recognition. Note: https://w.kaggle.com/competitions/bengaliai-speechKaggle Cited by: Ā§3. Howard et al. (2020) A. Howard, H. Klinck, S. Dane, S. Kahl, and T. Denton Cornell birdcall identification. Note: https://w.kaggle.com/competitions/birdsong-recognitionKaggle Cited by: Ā§3. Huang et al. (2024) Q. Huang, J. Vora, P. Liang, and J. Leskovec MLAgentBench: evaluating language agents on machine learning experimentation. In International Conference on Machine Learning, p. 20271ā20309. External Links: Link Cited by: Ā§1, Ā§7. Jain et al. (2025) N. Jain, K. Han, A. Gu, W. Li, F. Yan, T. Zhang, S. Wang, A. Solar-Lezama, K. Sen, and I. Stoica LiveCodeBench: holistic and contamination free evaluation of large language models for code. In International Conference on Learning Representations, External Links: Link Cited by: Ā§2.3, Ā§7. Jiang et al. (2025) Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu AIDE: AI-driven exploration in the space of code. External Links: 2502.13138, Link Cited by: Ā§1, Ā§2.1, Ā§4.2.1, Ā§7. Jimenez et al. (2024) C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan SWE-bench: can language models resolve real-world GitHub issues?. In International Conference on Learning Representations, External Links: Link Cited by: Ā§1. Jin et al. (2026) B. Jin, K. Wang, and P. Nie Automated modernization of machine learning engineering notebooks for reproducibility. Proceedings of the ACM on Software Engineering 3 (ISSTA), p. 1718ā1739. External Links: Link Cited by: Ā§1, Ā§7. Kaggle (2026a) Kaggle Kaggle: your machine learning and data science community. Note: https://w.kaggle.comAccessed October 2, 2026 Cited by: Ā§2.1. Kaggle (2026b) Kaggle Meta Kaggle code. Note: https://w.kaggle.com/datasets/kaggle/meta-kaggle-codeCompanion dataset containing public Kaggle notebook source; accessed September 24, 2026 Cited by: Ā§3. Kaggle (2026c) Kaggle Meta Kaggle. Note: https://w.kaggle.com/datasets/kaggle/meta-kagglePublic metadata for Kaggle competitions, users, kernels, submissions, and related activity; accessed September 24, 2026 Cited by: Ā§3. Kirsch et al. (2023) L. Kirsch, S. Dane, S. Adam, and V. Dardov AMPĀ®-parkinsonās disease progression prediction. Note: https://w.kaggle.com/competitions/amp-parkinsons-disease-progression-predictionKaggle Cited by: Ā§3. Klinck et al. (2025) H. Klinck, J. S. CaƱas, M. Demkin, S. Dane, S. Kahl, and T. Denton BirdCLEF+ 2025. Note: https://w.kaggle.com/competitions/birdclef-2025Kaggle Cited by: Ā§3. Lai et al. (2023) Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W. Yih, D. Fried, S. Wang, and T. Yu DS-1000: a natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, p. 18319ā18345. External Links: Link Cited by: Ā§1. Li et al. (2025) A. Li, C. Wu, Z. Ge, Y. H. Chong, Z. Hou, L. Cao, C. Ju, J. Wu, H. Li, H. Zhang, et al. The FM agent. Note: arXiv:2510.26144 External Links: Link Cited by: 1st item, Ā§2.1, Ā§7. Liao et al. (2025) Q. Liao, D. Wang, R. Pirie, J. Whelan, R. Haling, J. Liu, R. Khokher, X. Li, M. Plomecka, and A. Howard CSIRO - image2biomass prediction. Note: https://w.kaggle.com/competitions/csiro-biomassKaggle Cited by: Table 1. Liu et al. (2025) Z. Liu, Y. Cai, X. Zhu, Y. Zheng, R. Chen, Y. Wen, Y. Wang, W. E, and S. Chen ML-Master: towards AI-for-AI via integration of exploration and reasoning. Note: arXiv:2506.16499 External Links: Link Cited by: Ā§2.1, Ā§7. Mann and Whitney (1947) H. B. Mann and D. R. Whitney On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics 18 (1), p. 50ā60. External Links: Document, Link Cited by: Ā§5.1. Marinescu et al. (2021) R. Marinescu, A. Kishimoto, P. Ram, A. Rawat, M. Wistuba, P. P. Palmes, and A. Botea Searching for machine learning pipelines using a context-free grammar. AAAI Conference on Artificial Intelligence 35 (10), p. 8902ā8911. External Links: Document, Link Cited by: Ā§7. Matthys et al. (2022) K. Matthys, S. Kalanidhi, Valerie, W. Datathon, and W. Cukierski WiDS datathon 2022. Note: https://w.kaggle.com/competitions/widsdatathon2022Kaggle Cited by: Ā§3. McInnes et al. (2018) L. McInnes, J. Healy, and J. Melville UMAP: uniform manifold approximation and projection for dimension reduction. Note: arXiv:1802.03426 External Links: Link Cited by: Ā§3. Microsoft (2026) Microsoft Harrier-oss-v1-27b. Note: https://huggingface.co/microsoft/harrier-oss-v1-27bModel card; accessed October 1, 2026 Cited by: Ā§3. Mostafavi Ghahfarokhi et al. (2024) M. Mostafavi Ghahfarokhi, A. Asgari, M. Abolnejadian, and A. Heydarnoori DistilKaggle: a distilled dataset of Kaggle jupyter notebooks. In International Working Conference on Mining Software Repositories, p. 647ā651. External Links: Document, Link Cited by: Ā§1, Ā§7. Mulla (2022) R. Mulla Predict youtube video likes (Pog series #1). Note: https://w.kaggle.com/competitions/kaggle-pog-series-s01e01Kaggle Cited by: Table 1. Nadafian et al. (2026) A. Nadafian, A. Mohammadshahi, and M. Yazdani KAPSO: a knowledge-grounded framework for autonomous program synthesis and optimization. Note: arXiv:2601.21526 External Links: Link Cited by: Ā§1, 1st item. Nam et al. (2022) D. Nam, B. Ray, S. Kim, X. Qu, and S. Chandra Predictive synthesis of API-centric code. In International Symposium on Machine Programming, p. 40ā49. External Links: Document, Link Cited by: Ā§7. Nam et al. (2025) J. Nam, J. Yoon, J. Chen, J. Shin, S. Ć. Arık, and T. Pfister MLE-STAR: machine learning engineering agent via search and targeted refinement. In Conference on Neural Information Processing Systems, External Links: Link Cited by: Ā§1, Ā§1, 1st item, Ā§2.3, Ā§2.3. Pimentel et al. (2019) J. F. Pimentel, L. Murta, V. Braganholo, and J. Freire A large-scale study about quality and reproducibility of Jupyter notebooks. In International Working Conference on Mining Software Repositories, p. 507ā517. External Links: Document, Link Cited by: Ā§1. Reade and Oldacre (2025) W. Reade and A. Oldacre Santa 2025 - christmas tree packing challenge. Note: https://w.kaggle.com/competitions/santa-2025Kaggle Cited by: Ā§3. Real et al. (2020) E. Real, C. Liang, D. So, and Q. Le AutoML-Zero: evolving machine learning algorithms from scratch. In International Conference on Machine Learning, p. 8007ā8019. External Links: Link Cited by: Ā§7. Reyna et al. (2025) M. A. Reyna, Deepanshi, J. Weigle, Z. Koscova, K. Campbell, S. Seyedi, A. Elola, A. B. Rad, A. J. Shah, N. K. Bhatia, Y. Yan, S. Dane, A. Howard, G. D. Clifford, and R. Sameni PhysioNet - digitization of ecg images. Note: https://w.kaggle.com/competitions/physionet-ecg-image-digitizationKaggle Cited by: Table 1. Rudie et al. (2025) J. Rudie, E. Calabrese, R. Ball, P. Chang, R. Chen, E. Colak, M. C. de Verdier, L. Prevedello, T. Richards, R. Saluja, G. Zaharchuk, J. Sho, and M. Vazirabad RSNA intracranial aneurysm detection. Note: https://w.kaggle.com/competitions/rsna-intracranial-aneurysm-detectionKaggle Cited by: Table 1. Shah et al. (2020) A. Shah, E. Zhan, J. Sun, A. Verma, Y. Yue, and S. Chaudhuri Learning differentiable programs with admissible neural heuristics. In Conference on Neural Information Processing Systems, Vol. 33. External Links: Link Cited by: Ā§7. Shi et al. (2022) K. Shi, D. Bieber, and R. Singh TF-Coder: program synthesis for tensor manipulations. Transactions on Programming Languages and Systems 44 (2). External Links: Document, Link Cited by: Ā§7. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In Conference on Neural Information Processing Systems, Vol. 36, p. 8634ā8652. External Links: Link Cited by: Ā§7. Sonas et al. (2025) J. Sonas, P. Mooney, A. Howard, and W. Cukierski March machine learning mania 2025. Note: https://w.kaggle.com/competitions/march-machine-learning-mania-2025Kaggle Cited by: Table 1. Sorensen et al. (2025a) J. Sorensen, L. Dos Santos, L. Vasserman, M. Cruz, T. Acosta, and W. Reade Jigsaw ā agile community rules classification. Note: https://kaggle.com/competitions/jigsaw-agile-community-rulesKaggle competition Cited by: Ā§2.1. Sorensen et al. (2025b) J. Sorensen, L. D. Santos, L. Vasserman, M. Cruz, T. Acosta, and W. Reade Jigsaw - agile community rules classification. Note: https://w.kaggle.com/competitions/jigsaw-agile-community-rulesKaggle Cited by: Table 1. timeleftā (2026) timeleftā The leaderboard separation in PR #130 is inconsistently applied. Note: https://github.com/openai/mle-bench/issues/138GitHub issue #138 of openai/mle-bench, March 23, 2026; accessed September 27, 2026 Cited by: Ā§1, Ā§2.3. Toledo et al. (2025) E. Toledo, K. Hambardzumyan, M. Josifoski, R. Hazra, N. Baldwin, A. Audran-Reiss, M. Kuchnik, D. Magka, M. Jiang, A. M. Lupidi, et al. AI research agents for machine learning: search, exploration, and generalization in MLE-bench. In Conference on Neural Information Processing Systems, External Links: Link Cited by: Ā§2.1, Ā§2.3, Ā§7. Wan et al. (2025) C. Wan, X. Dai, Z. Wang, M. Li, Y. Wang, Y. Mao, Y. Lan, and Z. Xiao LoongFlow: directed evolutionary search via a cognitive Plan-Execute-Summarize paradigm. Note: arXiv:2512.24077 External Links: Link Cited by: Ā§2.1, Ā§7. Wang et al. (2018) X. Wang, I. Dillig, and R. Singh Program synthesis using abstraction refinement. Proceedings of the ACM on Programming Languages 2 (POPL). External Links: Document, Link Cited by: Ā§7. Wang et al. (2025) Z. Z. Wang, J. Mao, D. Fried, and G. Neubig Agent workflow memory. In International Conference on Machine Learning, p. 63897ā63911. External Links: Link Cited by: Ā§7. XTX Investments (2024) XTX Investments AI mathematical olympiad ā progress prize 1. Note: https://w.kaggle.com/competitions/ai-mathematical-olympiad-prizeKaggle competition Cited by: Ā§2.3. Yang et al. (2025) X. Yang, X. Yang, S. Fang, B. Xian, Y. Li, J. Wang, M. Xu, H. Pan, X. Hong, W. Liu, Y. Shen, W. Chen, and J. Bian R&D-Agent: automating data-driven AI solution building through LLM-powered automated research, development, and evolution. Note: arXiv:2505.14738 External Links: Link Cited by: Ā§1, Ā§2.3, Ā§7. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In International Conference on Learning Representations, External Links: Link Cited by: Ā§2.1, Ā§4.1.1. Yin et al. (2023) P. Yin, W. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, O. Polozov, and C. Sutton Natural language to code generation in interactive data science notebooks. In Annual Meeting of the Association for Computational Linguistics, p. 126ā173. External Links: Document, Link Cited by: Ā§1. Yip et al. (2025) K. H. Yip, L. V. Mugnai, R. L. Coates, A. Bocchieri, O. Faucoz, A. N. Govindan, G. Morello, A. Papageorgiou, A. Syty, T. Tahseen, S. Dane, M. Demkin, J. Beaulieu, S. B. Saikia, G. Bruno, Q. Changeat, C. Danielski, P. Danto, J. Davey, P. Drossart, P. Eccleston, B. Edwards, C. Jenner, R. King, T. Lueftinger, M. Min, N. Nikolaou, L. Pagliaro, E. Pascale, E. Panek, A. Radcliffe, L. F. SimƵes, P. C. Vallejos, T. Zingales, G. Tinetti, and I. P. Waldmann NeurIPS ā ariel data challenge 2025. Note: https://w.kaggle.com/competitions/ariel-data-challenge-2025Kaggle Cited by: Table 1. Zhang et al. (2026a) Q. Zhang, C. Hu, S. Upasani, B. Ma, F. Hong, V. Kamanuru, J. Rainton, C. Wu, M. Ji, H. Li, U. Thakker, J. Zou, and K. Olukotun Agentic context engineering: evolving contexts for self-improving language models. In International Conference on Learning Representations, External Links: Link Cited by: Ā§1, Ā§2.2, Ā§4.2.2, Ā§7. Zhang et al. (2026b) R. Zhang, P. Qin, Q. Cao, L. Zhang, and P. Xie AIBuildAI-2: a knowledge-enhanced agent for automatically building AI models. Note: arXiv:2605.27873 External Links: Link Cited by: Ā§1, 1st item, 2nd item. Zhang et al. (2025) Y. Zhang, M. Khalifa, S. Bhushan, G. D. Murphy, L. Logeswaran, J. Kim, M. Lee, H. Lee, and L. Wang MLRC-Bench: can language agents solve machine learning research challenges?. In Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, External Links: Link Cited by: Ā§2.1. Zhao et al. (2024) A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang ExpeL: LLM agents are experiential learners. AAAI Conference on Artificial Intelligence 38 (17), p. 19632ā19642. External Links: Document, Link Cited by: Ā§7. Zhou et al. (2022) Z. Zhou, M. T. Tang, Q. Pan, S. Tan, X. Wang, and T. Zhang INTENT: interactive tensor transformation synthesis. In Symposium on User Interface Software and Technology, External Links: Document, Link Cited by: Ā§7. Zhu et al. (2026) X. Zhu, Y. Cai, Z. Liu, B. Zheng, C. Wang, R. Ye, Y. Zhang, L. Zhang, W. E, S. Chen, and Y. Wang Toward ultra-long-horizon agentic science: cognitive accumulation for machine learning engineering. Note: arXiv:2601.10402 External Links: Link Cited by: Ā§1, Ā§1, Ā§1, 2nd item, Ā§2.3.