Paper deep dive
CuratorMAS: Automating Dataset Curation via Multi-Agent Orchestration
Yixin Zhang, Wenjie Feng
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 10/7/2026, 8:31:01 AM
Summary
The paper introduces CuratorMAS, a multi-agent collaboration framework designed to automate dataset curation. It decomposes the curation process into five stages: dataset exploration, knowledge-augmented evaluation, metric computation, agentic filtering, and experience-based self-evolution. CuratorMAS uses specialized agents (Explorer, Researcher, Evaluator, Filter, Refiner) to adaptively evaluate and curate datasets across different domains, significantly reducing noise rates and improving downstream model performance compared to rule-based, model-based, and prompt-based baselines.
Entities (12)
Relation Signals (10)
CuratorMAS ā evaluatedon ā JD
confidence 95% Ā· We evaluate CuratorMAS on... JD
CuratorMAS ā evaluatedon ā NoisyAG-News
confidence 95% Ā· We evaluate CuratorMAS on... NoisyAG-News
CuratorMAS ā evaluatedon ā CIFAR-10N
confidence 95% Ā· We evaluate CuratorMAS on... CIFAR-10N
CuratorMAS ā usesagent ā Filter Agent
confidence 95% Ā· the Filter Agent filters noisy samples accordingly
CuratorMAS ā usesagent ā Refiner Agent
confidence 95% Ā· The Refiner Agent reviews the computed metrics... and updates the domain skills
CuratorMAS ā usesagent ā Explorer Agent
confidence 95% Ā· CuratorMAS first performs dataset exploration... the Explorer Agent analyzes the dataset structure
CuratorMAS ā usesagent ā Researcher Agent
confidence 95% Ā· the Researcher Agent identifies gaps in the evaluation criteria... and retrieves up-to-date domain knowledge
CuratorMAS ā usesagent ā Evaluator Agent
confidence 95% Ā· the Evaluator Agent performs multi-granularity evaluation
CuratorMAS ā ā
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:High-quality datasets are essential for reliable machine learning, but dataset curation remains costly and hard to generalize across domains. Existing methods typically rely on manually designed heuristics or model-dependent signals, limiting their applicability across tasks and user queries. To address these limitations and automate data curation, we propose \textbf{CuratorMAS}, a multi-agent collaboration framework that orchestrates agents to evaluate and curate high-quality datasets. To achieve the goal of flexible curation, CuratorMAS decomposes the complex curation process into five programmable execution stages and forms a parallelizable workflow. Specifically, CuratorMAS first performs dataset exploration to collect contextual information such as file structures and constraint cues, thereby developing a comprehensive understanding of the given task. In order to acquire up-to-date information, CuratorMAS retrieves domain knowledge from online sources to augment the evaluation process. Next, CuratorMAS derives the necessary evaluation criteria and computes the corresponding metrics. Based on these results, CuratorMAS executes filtering accordingly. Finally, an evolution module summarizes the evaluation outcomes and updates the relevant skills. Extensive and comprehensive experiments demonstrate that CuratorMAS significantly reduces the noise rate by up to 36.03 percentage points (pp) while also improving the F1 score of downstream models by up to 8.88 pp.
Tags
Links
- Source: https://arxiv.org/abs/2610.07075v1
- Canonical: https://arxiv.org/abs/2610.07075v1
Trouble viewing inline? Open PDF directly ā
Full Text
45,374 characters extracted from source content.
Expand or collapse full text
CuratorMAS: Automating Dataset Curation via Multi-Agent Orchestration Yixin Zhang Wenjie Feng Abstract High-quality datasets are essential for reliable machine learning, but dataset curation remains costly and hard to generalize across domains. Existing methods typically rely on manually designed heuristics or model-dependent signals, limiting their applicability across tasks and user queries. To address these limitations and automate data curation, we propose CuratorMAS, a multi-agent collaboration framework that orchestrates agents to evaluate and curate high-quality datasets. To achieve the goal of flexible curation, CuratorMAS decomposes the complex curation process into five programmable execution stages and forms a parallelizable workflow. Specifically, CuratorMAS first performs dataset exploration to collect contextual information such as file structures and constraint cues, thereby developing a comprehensive understanding of the given task. In order to acquire up-to-date information, CuratorMAS retrieves domain knowledge from online sources to augment the evaluation process. Next, CuratorMAS derives the necessary evaluation criteria and computes the corresponding metrics. Based on these results, CuratorMAS executes filtering accordingly. Finally, an evolution module summarizes the evaluation outcomes and updates the relevant skills. Extensive and comprehensive experiments demonstrate that CuratorMAS significantly reduces the noise rate by up to 36.03 percentage points (p) while also improving the F1 score of downstream models by up to 8.88 p. 1University of Science and Technology of China yixinzhang@mail.ustc.edu.cn, fengwenjie@ustc.edu.cn Introduction High-quality data are fundamental to the success of modern machine learning systems, but dataset curation remains costly, domain-specific, and difficult to scale (Penedo et al. 2024; Soldaini et al. 2024). As training datasets are increasingly collected from web sources, crowdsourced annotations, and synthetically generated samples, dataset curation faces three key challenges: (i) how to automatically construct task-specific datasets; (i) how to ensure the quality of the curated datasets; and (i) how to satisfy diverse user requirements. Failure to address these challenges can hinder downstream model training by slowing convergence, amplifying annotation bias, and degrading generalization performance. Large-scale data curation pipelines typically rely on predefined heuristic rules, such as text-length thresholds and toxicity filters (Penedo et al. 2024; Soldaini et al. 2024; Raffel et al. 2020). Although computationally efficient, these heuristics are often brittle and inflexible when the task or user query changes. Another line of work estimates data quality via model-dependent signals, including training dynamics, confident learning, and meta-learned data valuation (Northcutt et al. 2021; Li et al. 2020; Xiao et al. 2023; Qiao et al. 2022; Chong et al. 2022; Ethayarajh et al. 2022; Calian et al. 2025). While these methods provide useful estimates of data quality, they often require task-specific model training or access to model internals, limiting their applicability to diverse user-provided datasets. Recent advances in large language models (LLMs) have demonstrated strong capabilities in reasoning, tool use, and data analysis (Zhu et al. 2026; Qiao et al. 2026), making LLM-based evaluation a promising alternative (Zhang et al. 2023; Lin et al. 2025). Nevertheless, a central question remains: How can agents automatically curate and evaluate datasets across diverse domains and user requirements? Figure 1: Overview of the CuratorMAS framework. To this end, we propose CuratorMAS, an automated multi-agent collaboration framework that orchestrates four core curation agentsāan Explorer Agent, a Researcher Agent, an Evaluator Agent, and a Filter Agentāto curate and evaluate high-quality datasets. Given a dataset and a user query, CuratorMAS first performs dataset comprehension to analyze the dataset structure and identify constraints imposed by downstream models and user preferences. It then conducts knowledge-augmented quality evaluation, during which the researcher retrieves up-to-date domain knowledge to enrich the systemās evaluation skills, while the evaluator derives appropriate quality criteria and computes the corresponding metrics. Based on the results, the filter agent makes criterion-aware filtering decisions and finally produces a curated dataset for downstream use. Unlike static rule-based and model-dependent approaches, CuratorMAS adapts its evaluation criteria to different datasets while providing transparent and auditable evaluation results. In addition, its evolution module enables the system to continuously incorporate up-to-date knowledge across runs. Our primary contributions are as follows: (i) We explicitly break down the data curation process into an executable and programmable workflow, providing a structured paradigm for automated data curation. (i) We present CuratorMAS, a multi-agent collaboration framework that coordinates agents to flexibly evaluate and curate datasets across different domains. (i) We evaluate CuratorMAS against six baselines on two datasets spanning different modalities and domains, including rule-based, model-based, and prompt-based methods. CuratorMAS reduces the noise rate by up to 36.03 p, while improving the F1 scores of the downstream models by up to 8.9 p through our effective workflow. Related Work Dataset Curation. Large-scale dataset construction typically relies on computationally efficient heuristics, such as length and toxicity filtering (Penedo et al. 2024; Soldaini et al. 2024; Raffel et al. 2020), but these rules are dataset-specific and require manual redesign across modalities, tasks, and user needs. Model-based techniques, including confident learning, influence-based valuation, and meta-learned utility estimation (Swayamdipta et al. 2020; Northcutt et al. 2021; Ethayarajh et al. 2022; Calian et al. 2025), provide quality estimates but are often tied to specific task domains and are difficult to generalize. Building on recent work using multimodal LLMs for quality assessment (Lin et al. 2025), our framework employs LLMs for query- and domain-aware dataset evaluation and curation. Multi-Agent Collaboration. Prior studies on multi-agent systems have demonstrated the benefits of role specialization, debate, and workflow decomposition. Role-playing frameworks assign agents distinct responsibilities for collaborative reasoning (Li et al. 2023; Zhang et al. 2025), while debate-based methods can improve factuality by enabling agents to critique one another (Du et al. 2024). Engineering-oriented frameworks such as AutoGen (Wu et al. 2023), MetaGPT (Hong et al. 2024), and ChatDev (Qian et al. 2024) support the decomposition of complex tasks into workflows. CuratorMAS extends these principles to dataset curation by decomposing the curation process into specialized stages coordinated through a structured multi-agent workflow. Self-Evolution. Agent self-evolution enables agents to improve by transforming experience into persistent memories, reusable skills, or executable tools. Reflexion (Shinn et al. 2023) stores linguistic feedback in episodic memory, while ExpeL (Zhao et al. 2024) distills reusable insights from past trajectories. Recent studies have explored skill evolution, long-term memory, and the automated construction of reusable tools (Qiu et al. 2025b; Xia et al. 2026; Qiu et al. 2025a). Unlike prior work on general reasoning and programming, CuratorMAS applies self-evolution to dataset curation by refining domain skills and developing reusable tools. CuratorMAS In this section, we present CuratorMAS, a multi-agent collaboration framework for curating and evaluating high-quality datasets, as illustrated in Figure 1. CuratorMAS organizes its five-stage workflow into three core components. 1) Dataset Comprehension: Given an input dataset and a user query, the Explorer Agent analyzes the dataset structure, summarizes its key characteristics, and decomposes the user query into constraints such as model requirements and user preferences. 2) Knowledge-Augmented Evaluation: In this open-domain setting, the Researcher Agent identifies gaps in the evaluation criteria of the current domain skill and retrieves up-to-date domain knowledge from online sources to augment the evaluation. Subsequently, the Evaluator Agent performs multi-granularity evaluation, after which the Filter Agent filters noisy samples accordingly to produce a curated dataset. 3) Experience-Based Self-Evolution: Based on the evaluation results, the Refiner Agent summarizes the evaluation outcomes and updates the domain skills by refining the criteria and distilling usable scripts. The overall workflow of CuratorMAS is formalized in Algorithm 1. Dataset Comprehension To handle dataset heterogeneity across modalities and domains, we first employ an Explorer Agent Ļexplorer _explorer to develop a comprehensive understanding of the target dataset D. The Explorer Agent analyzes the datasetās structure, parses data files, and performs essential preprocessing. It then summarizes the dataset into a structured document DU_D, which contains a structural description SDS_D, a sample overview ODO_D, and a data-loading protocol LDL_D. This document further provides guidance for subsequent agent collaboration. The Explorer Agent also decomposes the user query q into model constraints CmodelC_model and data preferences RdataR_data. Together, these components define the evaluation constraint set q=Cmodel,RdataC_q=\C_model,R_data\, which bounds the retrieval and evaluation procedures. By providing downstream agents with a shared representation of the dataset and user requirements, this comprehension stage promotes consistent reasoning and reduces the context overhead caused by repeated exploration. Knowledge-Augmented Evaluation External Knowledge Retrieval. According to the Hugging Face taxonomy, datasets span 50 domains and more than nine modalities, each potentially requiring distinct quality criteria. This diversity makes it impractical to define a fixed set of universally applicable evaluation metrics in advance. Therefore, we introduce an online knowledge retrieval stage to support adaptive evaluation. Specifically, guided by the query-specific constraints qC_q, the Researcher Agent Ļresearcher _researcher reviews the current domain skill SdomainS_domain and identifies potential quality risks that are not yet captured. It then formulates N search queries xii=1N\x_i\_i=1^N for online information retrieval. For each query, the Researcher Agent retrieves the top-K most relevant documents di,jj=1K\d_i,j\_j=1^K and spawns subagents to summarize them as defined in Equation (1): si,jj=1K \s_i,j\_j=1^K ā¼Ļsubagent(ā ā£di,jj=1K) _subagent (Ā· \d_i,j\_j=1^K ) (1) where Ļsubagent _subagent denotes the summarization policy used by each spawned subagent, and si,js_i,j denotes the summary of the retrieved document di,jd_i,j. We set N=4N=4 and K=5K=5 to balance retrieval quality and efficiency. Finally, the Researcher Agent synthesizes these summaries and derives a set of complementary evaluation dimensions QcompQ_comp, designed to cover previously identified quality risks that are not adequately addressed by the existing criteria, as follows: Qcompā¼Ļresearcher(ā ā£D,si,jj=1K,Sdomain).Q_comp _researcher (Ā· _D,\s_i,j\_j=1^K,S_domain ). (2) This stage enables CuratorMAS to adapt to diverse domains by incorporating up-to-date domain knowledge, thereby enhancing its adaptability and evaluation robustness. Algorithm 1 End-to-end workflow of CuratorMAS Input: Dataset D, query q, skill library S, tool library T Output: Curated dataset DcuratedD_curated, rule-wise noise sets Dnoise(ā)D_noise^( ), updated Sdomainā²S _domain and Tdomainā²T _domain 1: ā³ Dataset Comprehension 2: (D,q)ā¼Ļexplorer(ā ā£D,q)(U_D,C_q) _explorer(Ā· D,q) 3: SdomaināLoadSkillā”(,q)S_domain (S,q) 4: TdomaināLoadToolā”(,q)T_domain (T,q) 5: ā³ External Knowledge Retrieval 6: xii=1Nā¼Ļresearcher(ā ā£Sdomain)\x_i\_i=1^N _researcher(Ā· S_domain) 7: for i=1i=1 to N do 8: di,jj=1KāTopKā”(Searchā”(xi),K)\d_i,j\_j=1^K (Search(x_i),K) 9: si,jj=1Kā¼Ļsubagent(ā ā£di,jj=1K)\s_i,j\_j=1^K _subagent(Ā· \d_i,j\_j=1^K) 10: end for 11: Qcompā¼Ļresearcher(ā ā£D,si,j,Sdomain)Q_comp _researcher(Ā· _D,\s_i,j\,S_domain) 12: ā³ Knowledge-Augmented Quality Evaluation 13: augāQcompāSdomainK_augā Q_comp S_domain 14: Pevalā¼Ļevaluator(ā ā£D,aug,q)P_eval _evaluator(Ā· _D,K_aug,C_q) 15: for all tmāPevalt_mā P_eval in parallel do 16: omā¼Ļworker(ā ā£tm,D)o_m _worker(Ā· t_m,U_D) 17: end for 18: ā¬āBufferā”(omm=1M)B (\o_m\_m=1^M) 19: ā³ Agentic Filtering and Dataset Export 20: rā=1Lā¼Ļfilter(ā ā£ā¬,D,q)\r_ \_ =1^L _filter(Ā· ,U_D,C_q) 21: for ā=1 =1 to L do 22: Dnoise(ā)=xāD:rāā(x)=1D_noise^( )=\xā D:r_ (x)=1\ 23: end for 24: Dnoise=āā=1LDnoise(ā)D_noise= _ =1^LD_noise^( ) 25: Dcurated=DāDnoiseD_curated=D D_noise 26: ā³ Experience-Based Self-Evolution 27: Spatchā¼Ļrefiner(ā ā£Peval,ā¬,Sdomain)S_patch _refiner(Ā· P_eval,B,S_domain) 28: Sdomainā²āSdomaināŖSpatchS _domainā S_domaināŖ S_patch 29: Tpatchā¼Ļrefiner(ā ā£ā¬,Tdomain)T_patch _refiner(Ā· ,T_domain) 30: Tdomainā²āTdomaināŖgateā(Tpatch)T _domainā T_domain _gate(T_patch) 31: return Dcurated,Dnoise(ā),Sdomainā²,Tdomainā²D_curated,D_noise^( ),S _domain,T _domain Knowledge-Augmented Evaluation. The Evaluator Agent Ļevaluator _evaluator constructs an augmented evaluation knowledge base aug=SdomaināQcompK_aug=S_domain Q_comp, where ā denotes a knowledge-integration operator. Based on augK_aug, the agent performs knowledge-augmented evaluation. Specifically, we adopt a two-step process consisting of metric derivation and metric computation, thereby decoupling high-level metric design from error-prone computation. During metric derivation, the Evaluator Agent leverages the dataset context DU_D, the augmented knowledge augK_aug, and the task constraints qC_q to operationalize high-level evaluation dimensions as concrete, computable metrics, as illustrated in Equation (3). It then organizes the derived metrics into a structured evaluation plan PevalP_eval spanning three granularities: a sample-level component PsampleP_sample, a class- or group-level component PclassP_class, and a dataset-level component PdatasetP_dataset. Peval P_eval ā¼Ļevaluator(ā ā£D,aug,q) _evaluator (Ā· _D,K_aug,C_q ) (3) During the computation, we dispatch multiple Worker Agents to execute evaluation tasks in parallel to ensure computational efficiency. Each Worker Agent operates in an isolated workspace with a dedicated Python interpreter, thus preventing cross-task context interference. Upon receiving an assignment tmāPevalt_mā P_eval, a Worker Agent loads the required files, performs the computation, and writes the results to the shared computation buffer ā¬B, as formalized in Equation (4). ā¬=Buffer(om|omā¼Ļworker(ā ā£tm,D)m=1M)B=Buffer\! ( \o_m\, |\,o_m _worker(Ā· t_m,U_D) \_m=1^M ) (4) where M=|Peval|M=|P_eval| is the number of metrics in the evaluation plan and omo_m denotes the output produced by task tmt_m. Agentic Filtering and Dataset Export. Using the results stored in the computation buffer ā¬B, the Filter Agent Ļfilter _filter performs a second-stage analysis of the computed metrics and associated findings to identify poor-quality samples. It reviews the evaluation results and formulates a set of filtering rules rā=1L\r_ \_ =1^L based on the identified issues and the query-specific constraints qC_q. Each rule is applied independently, and the samples flagged by each rule are recorded for traceability. The union of the flagged samples is then removed from the original dataset. Finally, the remaining samples are exported as the curated dataset DcuratedD_curated for downstream use. CuratorMAS also produces a detailed evaluation report and a set of visualizations summarizing the evaluation results. The overall filtering process is formalized in Equation (5). Dnoise(ā) D_noise^( ) =xāD:rāā(x)=1, =\xā D:r_ (x)=1\, (5) Dcurated D_curated =Dāāā=1LDnoise(ā). =D _ =1^LD_noise^( ). Here, each filtering rule is defined as an indicator function rā:Dā0,1r_ :Dā\0,1\, where rāā(x)=1r_ (x)=1 means that sample x is identified as noisy, and rāā(x)=0r_ (x)=0 otherwise. Accordingly, the noisy subset identified by rār_ is formalized as Dnoise(ā)D_noise^( ). Experience-Based Self-Evolution Iterative Skill Refinement. As user requirements vary across domains, a static knowledge base may not adequately cover diverse curation scenarios and may therefore provide insufficient guidance. To tackle this issue, CuratorMAS iteratively refines the domain skills to support continual learning. The Refiner Agent Ļrefiner _refiner reviews the computed metrics and evaluation findings to identify capability gaps (i.e., overly general criteria or insufficiently covered cases) and failure modes in the current domain skill SdomainS_domain. It then derives a set of patches SpatchS_patch to remedy these deficiencies and update the domain skills as follows: Sdomainā²āSdomaināŖSpatchS _domainā S_domaināŖ S_patch (6) Reusable Tool Distillation. During evaluation, the agents may generate a substantial number of execution scripts for common data-processing operations. To leverage these one-off snippets, the Refiner Agent distills them into compact, reusable data-processing tools. We maintain a domain-specific tool library TdomainT_domain. Rather than modifying the library directly, the Refiner Agent submits proposed changes as a set of CRUD operations, denoted by TpatchT_patch. Before TpatchT_patch is applied to TdomainT_domain, another Gate Agent gateI_gate is invoked to review the proposed patches and reject redundant, overly specific, or invalid updates. Only approved patches are committed to the tool library. We further adopt a progressive-disclosure strategy to avoid overwhelming the agent with a batch of raw code. This gating mechanism improves the compactness, reusability, and general applicability of the tool library, as formalized in Equation (7): Tdomainā² T _domain āTdomaināŖgateā(Tpatch). ā T_domain _gate(T_patch). (7) Metric 10% 20% 30% Precision 76.97 Ā± 9.70 78.13 Ā± 28.61 92.72 Ā± 7.06 Recall 80.00 Ā± 9.54 78.83 Ā± 16.07 83.00 Ā± 8.97 F1 78.22 Ā± 8.08 77.96 Ā± 22.56 87.13 Ā± 2.51 FPR 2.80 Ā± 1.44 7.32 Ā± 10.25 3.44 Ā± 3.84 Table 1: Performance of CuratorMAS across contamination levels. Metrics (%) are reported as mean Ā± std across three runs, with contaminated samples as the positive class. Experiments Experimental Settings Datasets. We evaluate CuratorMAS on three datasets across the image and text modalities. The text benchmarks are (i) JD: a sentiment classification dataset of product reviews from the JD platform (DAMO_NLP 2022). We selected and manually re-annotated 1,000 samples to construct a clean subset, denoted as JD_clean; (i) NoisyAG-News: a human-annotated news-topic classification dataset with naturally occurring label noise (Huang et al. 2024). We selected 3,000 samples with a noise rate of 19.83%. The image benchmark is (i) CIFAR-10N: a human-annotated image classification dataset containing real-world label noise (Wei et al. 2022). We selected 3,000 samples with a noise rate of 39.67%. Dataset Method Filter Hit Precision Recall F1 Noise Level CIFAR-10N Original - - - - - 39.67% Rule-based Method(s) Heuristic 1,188 556 46.80% 46.72% 46.76% 34.99% Model-based Method(s) CleanLab 876 727 82.99% 61.09% 70.38% 21.80% DivideMix 1,539 927 60.23% 77.90% 67.94% 18.00% ProMix 1,180 832 70.51% 69.92% 70.21% 19.67% LossRank 1,188 779 65.57% 65.46% 65.52% 22.68% Prompt-based Method(s) LLM-as-a-Judge 1,457 1045 71.72% 87.82% 78.96% 9.40% Agentic Method(s) Ours 1,543 1137 73.69% 95.55% 83.21% 3.64% NoisyAG-News Original - - - - - 19.83% Rule-based Method(s) Heuristic 507 202 39.84% 33.95% 36.66% 15.76% Model-based Method(s) CleanLab 99 88 88.89% 14.79% 25.36% 17.48% DivideMix 133 108 81.20% 18.15% 29.67% 16.99% ProMix 665 275 41.35% 46.22% 43.65% 13.70% LossRank 594 324 54.55% 54.45% 54.50% 11.26% Prompt-based Method(s) LLM-as-a-Judge 240 177 73.75% 29.75% 42.40% 15.14% Agentic Method(s) Ours 425 322 75.76% 54.12% 63.14% 10.60% Table 2: Overall performance comparison with baselines on CIFAR-10N and NoisyAG-News. Baselines. We compare CuratorMAS with six baselines across three categories: rule-based, model-based, and prompt-based approaches. For the rule-based method, following previous work (Penedo et al. 2024; Soldaini et al. 2024; Raffel et al. 2020), we implement (i) Heuristic Rules, which comprises text-integrity checks, anomalous-image-resolution checks, and other heuristics. Model-based methods include (i) CleanLab (Northcutt et al. 2021), which uses confident learning to detect label noise; (i) DivideMix (Li et al. 2020), which models the distribution of training losses to distinguish clean samples from noisy ones; (iv) ProMix (Xiao et al. 2023), which combines small-loss selection, pseudo-label expansion, and mixed-sample augmentation; and (v) LossRank (Chong et al. 2022), which leverages loss as an indicator. For the prompt-based method, we evaluate (vi) LLM-as-a-Judge, which uses GPT-5.4 (OpenAI 2026b) as a judge to classify whether a sample is noisy. Downstream Models. For image classification, we consider (i) ResNet-18 (He et al. 2016); (i) MobileNetV2 (Sandler et al. 2018); (i) EfficientNet-B0 (Tan and Le 2019); and (iv) GhostNet (Han et al. 2020). All four models are initialized with weights pretrained on ImageNet-1K (Deng et al. 2009). For text classification, we leverage four pretrained language models, including (v) BERT-base-uncased (Devlin et al. 2019); (vi) DeBERTa-v3-small (He et al. 2023); (vii) MobileBERT (Sun et al. 2020); and (viii) BERT-tiny (Turc et al. 2019). Implementation Details. For the primary results, we use GPT-5.4 by default and also evaluate GPT-5.3-Codex (OpenAI 2026a) and Claude Sonnet-4.6 (Anthropic 2026). All LLM-based agents follow the ReAct reasoning-and-action paradigm (Yao et al. 2023). All experiments were conducted on Ubuntu 22.04 with an NVIDIA RTX 4090 GPU. (a) Model Performance on CIFAR-10N. (b) Model Performance on NoisyAG-News Figure 2: Downstream performance of models trained on the original noisy and CuratorMAS-curated datasets. Figure 3: Examples of different noise types in CIFAR-10N. Controlled Contamination We synthetically inject five predefined noise types into JD_clean at varying contamination levels. Specifically, we consider five representative types of noise that may arise during data collection and annotation: (i) Label Flipping, where the original label is changed to the opposite class; (i) Off-Domain, where the original review is replaced with irrelevant text from an unrelated corpus, such as THUCNews (Sun et al. 2016); (i) Ambiguous, where an LLM rewrites the review to make its sentiment ambiguous; (iv) Broken Text, where the review is corrupted by encoding errors, HTML residues, or other textual corruption; and (v) Duplication, where a randomly selected review is duplicated to replace another sample. These noise types are injected in equal proportions to produce overall noise rates of 10%, 20%, and 30%. For each level, we perform a single filtering pass. As shown in Table 1, CuratorMAS achieves favorable performance across noise levels while maintaining a low false-positive rate (FPR). At the 30% level, CuratorMAS achieves a recall of 83.00% and a precision of 92.72%, with a relatively low FPR of 3.44%. As the noise increases from 10% to 30%, CuratorMAS remains effective, with the F1 score rising from 78.22% to 87.13%, whereas the FPR increases only moderately from 2.80% to 3.44%. This trend may be attributed to the increased prevalence of noisy samples, which provides more detectable positive instances. These results indicate that the programmable curation workflow can identify and filter multiple types of noise across noise levels. Real-World Label Noise To evaluate the performance under real-world noise, we compare CuratorMAS with six classic baselines across two noisy datasets: CIFAR-10N and NoisyAG-News. We apply two rounds of filtering to CIFAR-10N and one round to NoisyAG-News. As shown in Table 2, CuratorMAS achieves the best overall performance among all evaluated methods. On CIFAR-10N, CuratorMAS reaches 95.55% recall, a 7.73-point improvement over the prompt-based method, while maintaining a precision of 73.69% and reducing the residual noise rate to 3.64%. Compared with model-based baselines, CuratorMAS drastically reduces the noise rate with consistently higher recall and precision. Although LLMs are effective judges (71.72% precision and 87.82% recall), CuratorMAS decomposes the complex curation procedure into a programmable workflow, thereby further improving its overall performance. Representative examples are shown in Figure 3. On NoisyAG-News, CuratorMAS also demonstrates its robust capabilities with the lowest residual noise rate of 10.60% and the highest precision and recall. These results underscore CuratorMASās consistent ability to foster more effective collaboration that translates directly into curation capability, regardless of the task domain and modality. We further observe a precisionārecall trade-off: additional filtering rounds improve noise coverage but may introduce a higher FPR, indicating the need to balance automated filtering with the available human-review budget. Figure 4: Training and validation loss curves of downstream models trained on the original and CuratorMAS-curated datasets. Impact on Downstream Training To further investigate how CuratorMAS benefits model training and generalization, we evaluate the downstream performance of models trained on the aforementioned CIFAR-10N and NoisyAG-News datasets curated by CuratorMAS. Figure 2 shows the overall performance gains across eight model architectures. For instance, on CIFAR-10N, all four models consistently achieve performance gains, with F1 scores improving by up to 8.88 p (MobileNetV2) and by 5.99 p on average. Similarly, on NoisyAG-News, all four BERT-style models benefit from the curated dataset with an average F1 score improvement of 3.64 p. These results indicate that CuratorMAS supports better downstream model performance across both text and image modalities. Beyond improving downstream performance, CuratorMAS also yields more favorable training dynamics. As illustrated in Figure 4, models trained on the curated datasets converge in fewer epochs and reach lower final training losses than those trained on the original noisy datasets. The validation curves also exhibit smaller fluctuations and less evidence of overfitting, suggesting more stable optimization. Together with the improvements in test-set performance, these observations indicate that CuratorMAS improves optimization efficiency and downstream generalization. Ablation Study Dataset Method Precision Recall Noise CIFAR-10N GPT-family GPT-5.4 73.69% 95.55% 3.64% GPT-5.3-Codex 60.76% 94.20% 5.97% Claude-family Sonnet-4.6 61.04% 95.46% 4.74% NoisyAG-News GPT-family GPT-5.4 75.76% 54.12% 10.60% GPT-5.3-Codex 54.55% 63.53% 9.41% Claude-family Sonnet-4.6 58.76% 65.38% 8.81% Table 3: Sensitivity of CuratorMAS to LLM Backbones on CIFAR-10N and NoisyAG-News. Sensitivity Analysis. We conduct a sensitivity analysis using three different LLM backbones for our task-execution agents. Table 3 reveals noticeable performance differences among the three models. GPT-5.4 delivers the best overall performance, achieving a favorable trade-off among precision, recall, and residual noise rate. GPT-5.3-Codex does not outperform either GPT-5.4 or Sonnet-4.6. This result suggests that specialization for coding alone does not necessarily translate into stronger data-curation capabilities, whereas general reasoning ability may play a more fundamental role in data curation. This observation is broadly consistent with previous research (Zhu et al. 2026). These findings indicate that although CuratorMAS can effectively support automated data curation, its curation performance remains sensitive to the capabilities of the underlying LLM backbone. Component Ablation. We also analyze the contributions of two key components of CuratorMAS on NoisyAG-News: external knowledge retrieval and skill enhancement. To prevent data leakage and ensure a fair comparison, the domain skills are constructed using a held-out subset. As shown in Table 4, both components contribute to the effectiveness of CuratorMAS. Removing both leads to the largest degradation, resulting in the poorest curation performance. External knowledge retrieval supports domain-specific evaluation, whereas domain skills facilitate long-term knowledge accumulation and stability. These results demonstrate the complementary roles of the two components in CuratorMAS. Setting Filtered Precision Recall Noise w/o KA 357 65.27% 39.16% 13.70% w/o Skill 425 75.76% 54.12% 10.60% Neither 433 29.56% 21.51% 18.19% Full 643 57.85% 62.52% 9.46% Table 4: Component Ablation of CuratorMAS. w/o KA uses only domain skills; w/o Skill uses only external knowledge retrieval; Neither removes both components; Full denotes the complete CuratorMAS. Conclusion We presented CuratorMAS, a multi-agent collaboration framework that enables agents to flexibly evaluate and curate datasets across diverse domains. By breaking down the complex data curation process into executable and programmable stages, CuratorMAS can adapt its evaluation criteria to different domains and dynamic user queries, thereby providing a structured paradigm for automated data curation. Experiments on three datasets against six baselines demonstrate the effectiveness of CuratorMAS, which reduces the noise rate by up to 36.03 p. Moreover, models trained on the curated datasets achieve improvements of up to 8.9 p in F1 score. These results establish CuratorMAS as a practical and flexible solution to automated data curation across diverse real-world scenarios. Future work includes extending CuratorMAS to more complex tasks while further improving its generalization and stability for high-stakes applications. References Anthropic (2026) Anthropic Introducing claude sonnet 4.6. Note: https://w.anthropic.com/news/claude-sonnet-4-6Accessed: 2026-07-29 Cited by: Implementation Details.. Calian et al. (2025) D. A. Calian, G. Farquhar, I. Kemaev, L. Zintgraf, M. Hessel, J. Shar, J. Oh, A. Gyƶrgy, T. Schaul, J. Dean, H. van Hasselt, and D. Silver DataRater: meta-learned dataset curation. In Advances in Neural Information Processing Systems, Vol. 38, p. 20060ā20092. Cited by: Introduction, Dataset Curation.. Chong et al. (2022) D. Chong, J. Hong, and C. D. Manning Detecting label errors by using pre-trained language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, Y. Goldberg, Z. Kozareva, and Y. Zhang (Eds.), p. 9074ā9091. External Links: Link, Document Cited by: Introduction, Baselines.. DAMO_NLP (2022) DAMO_NLP Product review sentiment prediction: damo_nlp/jd. Note: https://w.modelscope.cn/datasets/DAMO_NLP/jdAccessed: 2026-07-29 Cited by: Datasets.. Deng et al. (2009) J. Deng, W. Dong, R. Socher, L. Li, K. Li, and F. Li ImageNet: a large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, p. 248ā255. External Links: Document Cited by: Downstream Models.. Devlin et al. (2019) J. Devlin, M. Chang, K. Lee, and K. Toutanova BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), p. 4171ā4186. External Links: Link, Document Cited by: Downstream Models.. Du et al. (2024) Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch Improving factuality and reasoning in language models through multiagent debate. In ICML, Proceedings of Machine Learning Research, Vol. 235, p. 11733ā11763. Cited by: Multi-Agent Collaboration.. Ethayarajh et al. (2022) K. Ethayarajh, Y. Choi, and S. Swayamdipta Understanding dataset difficulty with V-usable information. In ICML, Proceedings of Machine Learning Research, Vol. 162, p. 5988ā6008. Cited by: Introduction, Dataset Curation.. Han et al. (2020) K. Han, Y. Wang, Q. Tian, J. Guo, C. Xu, and C. Xu GhostNet: more features from cheap operations. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, p. 1577ā1586. External Links: Link, Document Cited by: Downstream Models.. He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, p. 770ā778. External Links: Link, Document Cited by: Downstream Models.. He et al. (2023) P. He, J. Gao, and W. Chen DeBERTaV3: improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: Downstream Models.. Hong et al. (2024) S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schmidhuber MetaGPT: meta programming for A multi-agent collaborative framework. In ICLR, Cited by: Multi-Agent Collaboration.. Huang et al. (2024) H. Huang, T. Liang, X. Sun, Z. Jin, and Y. Yin NoisyAG-news: A benchmark for addressing instance-dependent noise in text classification. External Links: 2407.06579 Cited by: Datasets.. Li et al. (2023) G. Li, H. Hammoud, H. Itani, D. Khizbullin, and B. Ghanem CAMEL: communicative agents for "mind" exploration of large language model society. In NeurIPS, Vol. 36, p. 51991ā52008. Cited by: Multi-Agent Collaboration.. Li et al. (2020) J. Li, R. Socher, and S. C. H. Hoi DivideMix: learning with noisy labels as semi-supervised learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, External Links: Link Cited by: Introduction, Baselines.. Lin et al. (2025) L. Lin, D. Shi, A. Han, F. Chen, Q. Chen, J. Li, Z. Li, J. Zhang, Z. Sun, and J. Gao ACT as human: multimodal large language model data annotation with critical thinking. In Advances in Neural Information Processing Systems, Vol. 38, p. 24107ā24138. Cited by: Introduction, Dataset Curation.. Northcutt et al. (2021) C. G. Northcutt, L. Jiang, and I. L. Chuang Confident learning: estimating uncertainty in dataset labels. J. Artif. Intell. Res. 70, p. 1373ā1411. Cited by: Introduction, Dataset Curation., Baselines.. OpenAI (2026a) OpenAI Introducing gpt-5.3-codex. Note: https://openai.com/index/introducing-gpt-5-3-codex/Accessed: 2026-07-29 Cited by: Implementation Details.. OpenAI (2026b) OpenAI Introducing gpt-5.4. Note: https://openai.com/index/introducing-gpt-5-4/Accessed: 2026-07-29 Cited by: Baselines.. Penedo et al. (2024) G. Penedo, H. KydlĆÄek, L. B. Allal, A. Lozhkov, M. Mitchell, C. A. Raffel, L. von Werra, and T. Wolf The fineweb datasets: decanting the web for the finest text data at scale. In NeurIPS, Vol. 37, p. 30811ā30849. Cited by: Introduction, Introduction, Dataset Curation., Baselines.. Qian et al. (2024) C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong, J. Xu, D. Li, Z. Liu, and M. Sun ChatDev: communicative agents for software development. In ACL (1), p. 15174ā15186. Cited by: Multi-Agent Collaboration.. Qiao et al. (2022) D. Qiao, C. Dai, Y. Ding, J. Li, Q. Chen, W. Chen, and M. Zhang SelfMix: robust learning against textual label noise with self-mixup training. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, N. Calzolari, C. Huang, H. Kim, J. Pustejovsky, L. Wanner, K. Choi, P. Ryu, H. Chen, L. Donatelli, H. Ji, S. Kurohashi, P. Paggio, N. Xue, S. Kim, Y. Hahm, Z. He, T. K. Lee, E. Santus, F. Bond, and S. Na (Eds.), p. 960ā970. External Links: Link Cited by: Introduction. Qiao et al. (2026) S. Qiao, Y. Zhao, Z. Qiu, X. Wang, J. Zhang, Z. Bin, N. Zhang, Y. Jiang, P. Xie, F. Huang, and H. Chen Scaling generalist data-analytic agents. External Links: 2509.25084 Cited by: Introduction. Qiu et al. (2025a) J. Qiu, X. Qi, H. Wang, X. Juan, Y. Wang, Z. Zhao, J. Geng, J. Guo, P. Li, J. Shi, S. Liu, and M. Wang Alita-g: self-evolving generative agent for agent generation. External Links: 2510.23601 Cited by: Self-Evolution.. Qiu et al. (2025b) J. Qiu, X. Qi, T. Zhang, X. Juan, J. Guo, Y. Lu, Y. Wang, Z. Yao, Q. Ren, X. Jiang, X. Zhou, D. Liu, L. Yang, Y. Wu, K. Huang, S. Liu, H. Wang, and M. Wang Alita: generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution. External Links: 2505.20286 Cited by: Self-Evolution.. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21 (140), p. 1ā67. Cited by: Introduction, Dataset Curation., Baselines.. Sandler et al. (2018) M. Sandler, A. G. Howard, M. Zhu, A. Zhmoginov, and L. Chen MobileNetV2: inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, p. 4510ā4520. External Links: Link, Document Cited by: Downstream Models.. Shinn et al. (2023) N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao Reflexion: language agents with verbal reinforcement learning. In NeurIPS, Vol. 36, p. 8634ā8652. Cited by: Self-Evolution.. Soldaini et al. (2024) L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. R. Chandu, J. Dumas, Y. Elazar, V. Hofmann, A. H. Jha, S. Kumar, L. Lucy, X. Lyu, N. Lambert, I. Magnusson, J. Morrison, N. Muennighoff, A. Naik, C. Nam, M. E. Peters, A. Ravichander, K. Richardson, Z. Shen, E. Strubell, N. Subramani, O. Tafjord, P. Walsh, L. Zettlemoyer, N. A. Smith, H. Hajishirzi, I. Beltagy, D. Groeneveld, J. Dodge, and K. Lo Dolma: an open corpus of three trillion tokens for language model pretraining research. In ACL (1), p. 15725ā15788. Cited by: Introduction, Introduction, Dataset Curation., Baselines.. Sun et al. (2016) M. Sun, J. Li, Z. Guo, Y. Zhao, Y. Zheng, X. Si, and Z. Liu THUCTC: an efficient chinese text classifier. Note: http://thuctc.thunlp.org/Accessed: 2026-07-29 Cited by: Controlled Contamination. Sun et al. (2020) Z. Sun, H. Yu, X. Song, R. Liu, Y. Yang, and D. Zhou MobileBERT: a compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, D. Jurafsky, J. Chai, N. Schluter, and J. R. Tetreault (Eds.), p. 2158ā2170. External Links: Link, Document Cited by: Downstream Models.. Swayamdipta et al. (2020) S. Swayamdipta, R. Schwartz, N. Lourie, Y. Wang, H. Hajishirzi, N. A. Smith, and Y. Choi Dataset cartography: mapping and diagnosing datasets with training dynamics. In EMNLP (1), p. 9275ā9293. Cited by: Dataset Curation.. Tan and Le (2019) M. Tan and Q. V. Le EfficientNet: rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, K. Chaudhuri and R. Salakhutdinov (Eds.), Proceedings of Machine Learning Research, Vol. 97, p. 6105ā6114. External Links: Link Cited by: Downstream Models.. Turc et al. (2019) I. Turc, M. Chang, K. Lee, and K. Toutanova Well-read students learn better: on the importance of pre-training compact models. External Links: 1908.08962 Cited by: Downstream Models.. Wei et al. (2022) J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu Learning with noisy labels revisited: A study using real-world human annotations. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, External Links: Link Cited by: Datasets.. Wu et al. (2023) Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang AutoGen: enabling next-gen LLM applications via multi-agent conversation. External Links: 2308.08155 Cited by: Multi-Agent Collaboration.. Xia et al. (2026) P. Xia, J. Chen, H. Wang, J. Liu, K. Zeng, Y. Wang, S. Han, Y. Zhou, X. Zhao, H. Chen, Z. Zheng, C. Xie, and H. Yao SkillRL: evolving agents via recursive skill-augmented reinforcement learning. External Links: 2602.08234 Cited by: Self-Evolution.. Xiao et al. (2023) R. Xiao, Y. Dong, H. Wang, L. Feng, R. Wu, G. Chen, and J. Zhao ProMix: combating label noise via maximizing clean sample utility. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao, SAR, China, p. 4442ā4450. External Links: Link, Document Cited by: Introduction, Baselines.. Yao et al. (2023) S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao ReAct: synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Implementation Details.. Zhang et al. (2025) G. Zhang, Y. Yue, X. Sun, G. Wan, M. Yu, J. Fang, K. Wang, T. Chen, and D. Cheng G-designer: architecting multi-agent communication topologies via graph neural networks. In ICML, Proceedings of Machine Learning Research, Vol. 267, p. 76678ā76692. Cited by: Multi-Agent Collaboration.. Zhang et al. (2023) R. Zhang, Y. Li, Y. Ma, M. Zhou, and L. Zou LLMaAA: making large language models as active annotators. In Findings of the Association for Computational Linguistics: EMNLP 2023, p. 13088ā13103. External Links: Link, Document Cited by: Introduction. Zhao et al. (2024) A. Zhao, D. Huang, Q. Xu, M. Lin, Y. Liu, and G. Huang ExpeL: LLM agents are experiential learners. In AAAI, p. 19632ā19642. Cited by: Self-Evolution.. Zhu et al. (2026) Y. Zhu, Y. Zhong, J. Zhang, Z. Zhang, S. Qiao, Y. Luo, L. Du, D. Zheng, N. Zhang, and H. Chen Why do open-source llms struggle with data analysis? A systematic empirical study. In AAAI, p. 35239ā35247. Cited by: Introduction, Sensitivity Analysis..