Paper deep dive
How Well Does Agent Development Reflect Real-World Work?
Zora Zhiruo Wang, Sanidhya Vijayvargiya, Aspen Chen, Hanmo Zhang, Venu Arvind Arangarajan, Jett Chen, Valerie Chen, Diyi Yang, Daniel Fried, Graham Neubig
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 2:07:53 AM
Summary
This paper systematically evaluates the alignment between AI agent development benchmarks and the real-world U.S. labor market. By mapping 72,342 tasks from 43 benchmarks to the O*NET occupational taxonomy, the authors find a significant mismatch: agent development is heavily concentrated in programming-centric domains (Computer and Mathematical) and specific skills (Getting Information, Working with Computers), which represent a small fraction of total employment and economic value. Conversely, high-value domains like Management, Legal, and Architecture are underrepresented. The study proposes principles for better benchmark design, including coverage, realism, and granular evaluation.
Entities (19)
Relation Signals (24)
Computer and Mathematical → accountsforpercentageof → 7.6%
confidence 95% · which accounts for only 7.6% of total employment
Agent Development → isconcentratedin → Computer and Mathematical
confidence 95% · agent development that tends to be programming-centric... heavily concentrated in the computer and mathematical domain
Agent Development → isconcentratedin → Computer and Mathematical
confidence 95% · agent benchmarking effort is overwhelmingly concentrated in the Computer and Mathematical domain
Benchmarks → mapsto → O*NET
confidence 95% · mapping benchmark instances to work domains and skills... using occupational taxonomies derived from the O*NET database
O*NET → providestaxonomyfor → Skills
confidence 95% · skill taxonomy starting from O*NET’s Work Activities
O*NET → providestaxonomyfor → Work Domains
confidence 95% · occupational taxonomies derived from the O*NET database
Benchmarks → proposesprinciples → Granular Evaluation
confidence 92% · propose three measurable principles for designing benchmarks... coverage, realism, and granular evaluation
Benchmarks → proposesprinciples → Coverage
confidence 92% · propose three measurable principles for designing benchmarks... coverage, realism, and granular evaluation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2603.01203v2
- Canonical: https://arxiv.org/abs/2603.01203v2
Trouble viewing inline? Open PDF directly →
Full Text
69,957 characters extracted from source content.
Expand or collapse full text
How Well Does Agent Development Reflect Real-World Work? Zora Z. WangSanidhya VijayvargiyaAspen Chen Hanmo ZhangVenu Arvind ArangarajanJett Chen Valerie ChenDiyi YangDaniel FriedGraham Neubig Carnegie Mellon UniversityStanford University zhiruow@cs.cmu.edu https://zorazrw.github.io/ai4work/ Abstract AI agents are increasingly developed and evaluated on benchmarks relevant to human work, yet it remains unclear how representative these benchmarking efforts are of the labor market as a whole. In this work, we systematically study the relationship between agent development efforts and the distribution of real-world human work by mapping benchmark instances to work domains and skills. We first analyze 43 benchmarks and 72,342 tasks, measuring their alignment with human employment and capital allocation across all 1,016 real-world occupations in the U.S. labor market. We reveal substantial mismatches between agent development that tends to be programming-centric, and the categories in which human labor and economic value are concentrated. Within work areas that agents currently target, we further characterize current agent utility by measuring their autonomy levels, providing practical guidance for agent interaction strategies across work scenarios. Building on these findings, we propose three measurable principles for designing benchmarks that better capture socially important and technically challenging forms of work: coverage, realism, and granular evaluation. 1 1. Introduction AI agents have made rapid progress in their accuracy and autonomy on tasks such as web navigation (Deng et al., 2023; Zhou et al., 2024a) and versatile computer use (Xie et al., 2024b). A central motivation behind the development of such agents is to enhance human productivity across diverse forms of work (Wang et al., 2025b), such as software engineering (Deng et al., 2025) and knowledge-intensive research (Wei et al., 2025). As agent capabilities advance, benchmark design has primarily evolved by increasing task complexity (Miyai et al., 2025), extending the amount of work done in each task (Yoran et al., 2024), or introducing more challenging environments (Xu et al., 2024). While these changes aim to better stress-test agents, it remains unclear whether this scaling of difficulty meaningfully reflects the structure and demands of human work (Shao et al., 2025), or how performance on existing benchmarks translates into practical relevance for real-world jobs. Even when specific benchmarks explicitly target work- related activities (Patwardhan et al., 2025; Xu et al., 2024), they do not rely on a standard method for categorization of work, hindering cross-benchmarks comparisons, and obscuring which work domains (e.g., administrative support) and underlying skills (e.g., information gathering) are being approximated, and who ultimately benefits from improved agent performance. 1 Code and data available at https://github.com/zorazrw/ai4work-resources arXiv:2603.01203v2 [cs.AI] 6 Mar 2026 To address this gap, in this paper, we present a systematic framework that situates agent benchmarks within the broader landscape of human work, particularly in the context of the U.S. labor market. Specifically, we map individual benchmark examples to domains and skills using occupational taxonomies derived from the O*NET database (O*NET Center, 2024), a U.S. government resource that catalogs real-world work activities at multiple levels of granularity. Because benchmark tasks often specify the task but not the activities required to actually complete it, this mapping requires disentangling composite activities and aligning informal descriptions with structured occupational labels. We implement this via large language model (LLM)-based annotation with manual quality verification, enabling scalable processing across benchmarks (§2). Building on this mapping framework, we conduct a large-scale analysis spanning 43 agent benchmarks, 72,342 task instances, and 1,016 real-world occupations, providing a holistic characterization of how current agent development efforts align with the broader landscape of the U.S. labor market. We find that existing agent benchmarks are heavily concentrated in the computer and mathematical domain, which accounts for only 7.6% of U.S. employment, while other highly digitized and economically significant areas, such as management and legal work, remain substantially underrepresented. These domains are not only central to organizational decision-making and economic coordination, but also pose distinct technical challenges, such as ambiguous objectives and verification with long-horizon dependencies. We observe a similar pattern at the skill level: agent development disproportionately targets a small set of fine-grained skills that together account for less than 5% of total U.S. employment, whereas widely prevalent skills such as interpersonal interaction, which permeate most jobs, are largely absent (§3). Even within the work areas that current benchmarks target, tasks vary substantially in contextual and procedural complexity, making it hard to translate performance metrics into actionable insights of agent capability boundaries. We thus propose a unified task complexity measure that places agent activities on a common scale. This enables measurement of agent autonomy, defined as the performance frontier across tasks of increasing complexity. We analyze agent trajectories across benchmarks to depict their autonomy levels by domain, skills, agent framework, and backbone LMs, providing actionable guidance for selecting the appropriate autonomy level to interact with them (§4). Building on these findings, we propose three measurable benchmark design principles— domain and skill coverage, task realism and complexity, and granular evaluation—to better capture the breadth and structure of real-world work (§5). Together, these contributions establish a systematic framework for evaluating how well agent benchmarks reflect real-world work, and provide actionable tools for guiding both benchmark design and agent development toward more representative and socially grounded progress. 2. Bridging Agents to Real-World Work To translate agent development to real-world utility, we depict the landscape of human work (§2.1) and situate agent building efforts in the same space for comparison (§2.2). 2.1. The Landscape of Human Work We depict the landscape of work along two complementary dimensions, domain and skill, by constructing two taxonomies based on the O*NET database (O*NET Center, 2024), as in Figure 1. Crucially, to ensure a faithful representation of real-world work, these taxonomies are defined purely from human occupations and not biased by existing agent development efforts. 2 Review and validate expense categorization for company transactions. map task into Business & Finance Operations Healthcare Practitioners and Technical Office & Administrative Support Architecture and Engineering Job Family Accountants & Auditors Bookkeeping/Accounting Clerks Dental Hygienists Occupation Acute Care Nurses Domain Taxonomy Information Input Mental Processes Interacting with Others Work Output Accountants & Auditors Administering Information and Data Processing High-Level Skill Evaluating Information to Determine Compliance with Standards Skill Taxonomy Fine-grained skill Figure 1: Mapping agent benchmarks to work domains and skills in O*NET. Domain-Based TaxonomyWe build a domain taxonomy푇 푑 based on the job family and task requirement annotations (as in Figure 1, left) in O*NET. Our taxonomy connects work domain variety across granularities: from high-level industries (e.g., Business and Financial Operations), pertinent occupations (e.g., Accountants, Budget Analysts), to concrete tasks performed in practice (e.g., prepare adjusting journal entries). Overall, the domain taxonomy 푇 푑 = (푉 (0) ,푉 (1) ,푉 (2) ,푉 (3) ,퐸)has 23 job families (|푉 (1) |), of which 743 occupations (|푉 (2) |) and 5806 task descriptions (|푉 (3) |) involve computer use;푉 (0) denotes the “domain” root and퐸is the set of tree edges. Skill-Based TaxonomyIn the context of agent work, we define skill as a concrete sequence of actions performed to achieve a goal. We construct the skill taxonomy푇 푠 starting from O*NET’s Work Activities (WA) taxonomy 2 organized in four general categories푉 (1) =information input, interacting with others, mental processes, work output, each expanding into multiple detailed activities. We further expand the taxonomy with O*NET’s Detailed Work Activities. The resulting taxonomy contains three layers of skills of progressively finer granularity, with 4, 9, and 41 nodes, as summarized in Figure 1 (right). For both taxonomies, we denote a path to be a sequence of categories from the root to a granular leaf one:푝= (푣 0 ,푣 1 ,·,푣 푙 ),푣 0 = 푟,(푣 푖 ,푣 푖+1 ) ∈ 퐸, where푙represents the total number of layers in the taxonomy tree. We explicitly preserve the full path to model the coarse-to-fine structure of work, enabling analysis at various levels of abstraction. By cross-indexing tasks to these two taxonomies, we can obtain complementary views to associate them with work: domains reflect job-specific contexts, while skills capture processes that generalize across jobs. Human Employment and Capital DistributionHuman employment and economic value are not equally distributed across domains and skills. To capture this variance, we collect employment and capital statistics for each category in the domain and skill taxonomies. For the domain taxonomy, since each leaf category corresponds to an occupation, we obtain occupation employment and median salary data (latest update in 2024) from the U.S. Bureau of Labor Statistics (BLS) (of Labor, 2024). Because both BLS and O*NET use the Standard Occupational Classification (SOC) system, we directly align each leaf occupation in our domain taxonomy with its corresponding employment and wage statistics via shared SOC codes. We aggregate these counts upward to estimate employment and capital at broader domain levels. For the skill taxonomy, we compute employment and capital estimates for each fine-grained work 2 Although O*NET labels skills in a cognitive sense (https://w.onetonline.org/find/descriptor/b rowse/2.A), we adopt this formulation as it offers a more tangible basis for agent analysis. 3 activity by weighting occupational data using the corresponding activity-level importance scores provided by O*NET. 3 We then aggregate these weighted values to higher-level skill categories. Importantly, these estimates do not represent the exact number of workers performing a given skill in their daily work; rather, they approximate the relative importance of each skill across the overall labor market. To provide a holistic view of real-world work, we retain domains and skills involving physical labor (e.g., manufacturing) in both taxonomies. To explicitly model digital and physical forms of work, we separately quantify digital and physical labor. Concretely, for each occupation, we examine its associated task requirements and classify each task as digital or physical by prompting an LLM 4 to assign aDIGITALorPHYSICALlabel based on the occupation name and task description. We then compute, for each occupation, the proportion of digital tasks and weight this proportion by employment and capital statistics to derive aggregate measures for various levels of domains and skills. Find the full taxonomies and calculation process in §A. 2.2. Agent Development Effort for Work Benchmarks are a primary driver of agent development, shaping both the tasks on which agents are trained and the settings in which they are evaluated. To characterize the current landscape of digital agent development for work-related activities, we aggregate representative agent benchmarks and map them onto the work taxonomies introduced above (§A.3). We aim to comprehensively aggregate existing agent benchmarks related to human work (Table 1). To ensure consistency with the core definition of agents, we apply two minimal inclusion criteria: (i) agentic, in that the system operates within an interactive environment (e.g., a web browser or desktop) and follows an observation-action loop during task execution; and (i) work-related, meaning that at least a subset of tasks corresponds to real-world work activities. Mapping Benchmark Examples to WorkWe map each benchmark example푒(a task definition in natural language) to one or more paths푃= 푃 푑 ∪ 푃 푠 =푝 푑 ∪푝 푠 in both taxonomies, where each path traces a hierarchy from a top-level category down to the most granular domain or skill represented in the taxonomy, thereby capturing both coarse and detailed aspects of the work reflected by the task. We perform this mapping휑using an LLM 5 by providing the NL task instruction with a flattened representation of the taxonomy, and asking the model to identify all relevant taxonomy paths covered by the task, as 휑(푒, LM) → 푃. Calculating CoverageWe define coverage of a set of examplesE=푒on a taxonomy as the percentage of unique paths being covered in the taxonomy tree푐(E,푇)= 휑(E) |P(푇)| . In other words, in the example mapping in Figure 1, coverage calculates the percentage of colored (i.e., mapped) paths among all paths in the taxonomy tree. Across all benchmarks, examples collectively cover a limited 56.5% of the domain taxonomy but a substantially broader 85.4% of the skill taxonomy. This difference arises because domains correspond to occupation-specific contexts, whereas skills represent cross-occupational processes: a single benchmark example may activate multiple general skills (e.g., getting information, coordination, analyzing data) even if it is situated within a narrow occupational domain. Although our taxonomy includes both digital and physical skills, we find many O*NET skill categories are modality-agnostic; thus, benchmarks focused on digital tasks still achieve broad skill coverage, even if physical realizations are underrepresented. Coverage-Aware SamplingSome benchmarks contain a large number of task instances that 3 E.g., https://w.onetonline.org/find/descriptor/result/4.A.1.b.3 4 We used claude-sonnet-3.7 (Anthropic, 2025b) for efficiency. 5 We used gpt-5 (OpenAI, 2025) for higher-precision mapping. 4 CategoryBenchmark SizeCoverage (%) totalsampleddomainskill General Digital Work TheAgentCompany (Xu et al., 2024)17517530.456.1 GDPval (Patwardhan et al., 2025)22022047.858.5 Remote Labor Index (Mazeika et al., 2025)9921.717.1 WorkArena (Drouin et al., 2024)16,83330026.122.0 OfficeBench (Wang et al., 2024b)30030039.143.9 CRMArena (Huang et al., 2025)1,17030030.422.0 EnterpriseBench (Vishwakarma et al., 2025)48334039.143.9 GitTaskBench (Ni et al., 2025)545430.417.1 Web & Mobile Navigation OSWorld (Xie et al., 2024b)41839030.446.3 WebVoyager (He et al., 2024)64330030.434.1 WebArena (Zhou et al., 2024a)81230039.143.9 Mind2Web (Deng et al., 2023)1,01530030.431.7 WebShop (Yao et al., 2022)12,2513008.712.2 VisualWebArena (Koh et al., 2024)91030030.429.3 WebLINX (Lù et al., 2024)999930.424.4 AppWorld (Trivedi et al., 2024)73230030.436.6 AssistantBench (Yoran et al., 2024)21421443.517.1 SPA-Bench (Chen et al., 2025a)34030034.829.3 MMInA (Tian et al., 2025)1,05030026.124.4 WebChoreArena (Miyai et al., 2025)71830034.822.0 InformationGAIA (Mialon et al., 2024)46630047.826.8 Planning TravelPlanner (Xie et al., 2024a)1,0003008.744.4 DeepPlanning (Zhang et al., 2026)1201204.333.3 Software Engineering SWE-bench (Jimenez et al., 2024)2,29430013.017.1 TerminalBench (Team, 2025)23223239.129.3 ColBench (Zhou et al., 2025)20,02086043.548.8 SWE-Lancer (Miserendino et al., 2025)1981988.712.2 SWE-Bench M (Yang et al., 2025)51030017.424.4 SWE-Bench Pro (Deng et al., 2025)73130017.429.3 MLE-Bench (Chan et al., 2024)828234.819.5 SWT-Bench (Mündler et al., 2024)4333004.319.5 Science DiscoveryBench (Majumder et al., 2025)34430030.417.1 ScienceAgentBench (Chen et al., 2025b)10210213.014.6 CORE-Bench (Siegel et al., 2024)454513.04.9 SciCode (Tian et al., 2024)808013.09.8 MLGym (Nathani et al., 2025)19198.717.1 DiscoveryWorld (Jansen et al., 2024)59598.729.3 LabBench (Laurent et al., 2024)1,97030021.719.5 SUPER (Bogin et al., 2024)8013008.712.2 Social Sotopia (Zhou et al., 2024b)3,86030039.146.3 STSS (Wang et al., 2024a)404026.134.1 Physical Behavioral-1K (Li et al., 2023)505039.122.2 FieldWorkArena (Moteki et al., 2025)44030034.824.4 Total72,34210,28856.585.4 Table 1: Agent benchmark sizes and their coverage on the real-world work. GDPval, despite being relatively small, has the highest domain and skill coverage across human work. 5 Figure 2: Agent benchmarking effort is concentrated in mathematical and engineering domains, despite broad human employment and economic value across other domains. are largely homogeneous, which would impose substantial computational cost when analyzed at scale across many benchmarks, while contributing to merely a subset of the taxonomies repetitively. To save cost while reflecting coverage similar to that of the original benchmark, we develop a coverage-guaranteed sampling strategy that continues to sample new batches of examples (of size 5) until coverage increases slower thanΔ=0.1. This strategy ensures that the sampled subset remains representative of the benchmark’s work diversity, while being more cost-efficient. We validate the robustness of our coverage-aware sampling via permutation tests, showing stable stopping size and domain/skill coverage across resampled task orderings (§B). Manual ValidationWe successfully map 91.2% and 95.5% examples into the domain and skill taxonomies; other times, the LM outputs no (8.0% and 3.7%) or invalid (0.8% and 0.8%) paths. To examine the reliability of the LM-driven mapping process, we conduct a manual validation study by randomly sampling 90 examples across benchmarks and categorizing their mappings into four outcome types: fully correct, fully incorrect, partially correct but missing elements, or containing extraneous elements. Two independent human annotators and the LM achieve high agreement rates of 90.9% and 89.3% for domain and skill mapping results, suggesting the reliability of this LM-based mapping process. Refer to §B for detailed descriptions of mapping, sampling, and verification procedures. 3. Skewed Emphasis in Agent Development In this section, we analyze the alignment between agent development effort and the real-world work landscape (§3.1, §3.2). 6 3.1. Benchmarks Focus on Software Engineering but Miss Other Digitized Domains Figure 2 compares the distribution of agent benchmark examples with real-world employment across domains. For each domain푑, benchmark effort is defined as the number of benchmark examples mapped to that domain Í 푒∈E 1[d표푚(휑(푒 ,LM))= 푑], where휑(·)returns a root-to- leaf taxonomy path andd표푚(·)extracts its domain-level node (§2.2). Real-world employment Í 표∈푑 e푚푝푙표푦푚푒푛푡(표)aggregates the number of workers across occupations표within domain 푑. Real-world capital is defined as Í 표∈푑 s푎푙푎푟푦(표)× e푚푝푙표푦푚푒푛푡(표) capturing the total earning- based economic value associated with occupations in domain 푑 (§2.1). Unsurprisingly for those familiar with recent agent development, agent benchmarking effort is overwhelmingly concentrated in the Computer and Mathematical domain, featuring mainly programming tasks. On one hand, software can be used to perform tasks across a wide variety of domains, so developing software engineering abilities has the potential to accelerate other varieties of work. However, this concentrated focus significantly overrepresents a domain that accounts for only 7.6% of total employment, and general-purpose software engineering benchmarks do not fully capture the domain-specific nuances in many other work areas. As a result, large portions of the labor market remain weakly represented in current agent evaluations. Examining domain-level digitization alongside employment reveals several domains that are heavily digitized and yet receive relatively little agent development effort. Notably, Management, Legal, and Architecture and Engineering exhibit high ratios of digital work (88%, 70%, and 71%, respectively), but are sparsely covered by existing benchmarks (1.4%, 0.3%, 0.7% among all 19179 examples). This gap suggests missed opportunities where agents could plausibly deliver near- term productivity gains. Viewing work via capital distribution further exposes a disconnect between benchmarking focus and economic impact. First, economically valuable domains, most notably Management, which also feature heavily digitized work, are underrepresented, indicating that current efforts are not targeting the highest-revenue segments of the labor market. Meanwhile, low-paying, labor-intensive domains such as Personal Care and Service are likewise underexplored. Taken together, these observations suggest that agent benchmarking effort is driven less by alignment with real-world employment structure or economic value, and more by method- ological convenience. In particular, domains with readily specified NL task instructions and easily verifiable rewards are disproportionately favored. While this focus has enabled rapid methodological progress in the areas where benchmarking is convenient, it risks skewing agent development away from domains where societal and economic impact may be greatest. 3.2. Benchmarks Overfocus on a Narrow Set of Skills Similarly to the domain analysis, for each skill푠, we measure the benchmark effort for it by the number of benchmark examples mapped to that granular skill Í 푒∈E 1[l푒푎푓(휑(푒,LM))= 푠], where휑(·)returns a root-to-leaf path in the skill taxonomy andl푒푎푓(·)extracts its leaf-level node (§2.2). Reported effective employment Í 표 e푚푝푙표푦푚푒푛푡(표)× i푚푝표푟푡푎푛푐푒(표,푠)is the aggregation of the number of workers of each occupation표weighted by the importance of skill푠in their job. Effective capital is defined as Í 표∈푠 s푎푙푎푟푦(표)× e푚푝푙표푦푚푒푛푡(표)× i푚푝표푟푡푎푛푐푒(표)capturing the total earning-based economic value associated with occupations executing skill 푠 (§2.1). As shown Figure 3, human work typically draws on a diverse mix of skills spanning information input, mental processes, interaction with others, and work output, with no single category dominating the labor landscape. This balance reflects the multifaceted nature of 7 Figure 3: Agent benchmarking effort emphasizes two granular skills (getting information and working with computers), which together, only cover<5% human employment, creating a heavily imbalanced distribution across skills. See §A.2 for full skill names. real work, where tasks routinely require coordinating multiple forms of activity rather than repeatedly exercising a narrow set of capabilities. In contrast, agent benchmarking effort is heavily concentrated on a small number of fine- grained skills. In particular, benchmark examples disproportionately target leaf-level activities such as Getting Information within the Information Input category and Working with Computers within Work Output, which only occupy 3.1% and 2.4% of employment. This concentration pro- duces two distinct distortions. First, within otherwise broad skill categories, development effort is unevenly allocated across granular skills, resulting in overemphasis on a few easily benchmarked activities while neglecting others at the same level of abstraction. Second, this focus crowds out entire high-level skill categories, most notably Interacting with Others, which receive minimal coverage despite practically pervading a wide range of real-world occupations. 3.3. From Work Relevance to Representativeness Assigning a task to a domain or skill establishes its relevance to work, but relevance alone does not imply representativeness — whether a benchmark task captures the realistic scope, context, and complexity of the corresponding work in practice. Many benchmarks include simplified versions of work-relevant tasks that omit important contextual or procedural details. Domain BreathWe measure domain breadth by the number of work domains associated with each task, reflecting the breadth of contextual knowledge involved. In Figure 4 (top), although 8 5 0 5 Number of Mapped Paths appworld assistantbench mind2web mmina osworld spabench visual_webarena web_chore_arena webarena weblinx webshop webvoyager crmarena enterprise_bench gdpval gittaskbench officebench remote_labor_index the_agent_company workarena gaia deepplanning travelplanner colbench mle_bench swe_bench swe_bench_multimodal swe_bench_pro swelancer swt_bench terminal_bench core discoverybench discoveryworld labbench mlgym scicode scienceagentbench super sotopia stss behavioral_1k fieldworkarena Domain Skill Web & Mobile General Work Information Planning Engineering Science Social Physical Figure 4: Number of domains (top) and skills (bottom) an average benchmark example map to. most examples (77.1%) map to more than 1 domain, only 8.5% of examples span more than 3 domains (e.g., GDPval), indicating limited cross-domain structure. Skill BreathWe further characterize tasks by the number of distinct skills they require. While this measure does not capture all dimensions of procedural complexity, it provides a proxy for the breadth of capabilities involved. In Figure 4 (bottom), we report the average number of fine- grained skills per example for each benchmark. The resulting distribution reveals substantial heterogeneity: while 27.0% of tasks require only a single skill, a nontrivial fraction (32.6%) involve four or more distinct skills, reflecting substantially more complex procedures, e.g., the TheAgentCompany task in Figure 1. 4. How Autonomous Can Agents Be? Within the set of work currently represented in agent benchmarks, a central question is how au- tonomously they can reliably act at work. This question is often framed as a dichotomy between automation and augmentation. We argue, however, that these are not mutually exclusive modes. Whether an agent automates or augments depends on the scope of the task: a system may fully automate a narrowly scoped task (e.g., implementing bubble sort) while only augmenting within a more complex workflow (e.g., preparing a lecture on algorithms). We therefore treat autonomy as a spectrum and examine agent autonomy levels as a function of task complexity. In this section, we first quantify agent autonomy (§4.1), and then provide practical guidance on selecting appropriate autonomy levels in agent deployment (§4.2). 4.1. Grounding Agent Autonomy in Task Complexity Automating tasks can reduce human effort during execution, but raises the need for oversight to ensure correctness and control (Bainbridge, 1983; Parasuraman and Riley, 1997). A key prerequisite for effective deployment is therefore understanding the autonomy level at which an agent operates. To make this notion precise, we introduce operational definitions of task complexity and agent autonomy, which allow us to quantify autonomy levels and compare them across tasks and domains. Definition 1. Task Complexity. The number and organization of distinct skills and procedural steps required to complete a task. Following Wood (1986), task complexity can be decomposed into component complexity (the number of distinct steps involved), coordinative complexity (i.e., interdependence among task 9 elements), and dynamic complexity (i.e., uncertainty or environmental change). In this work, our definition corresponds primarily to component complexity, as the other two dimensions are difficult to estimate reliably from benchmark task descriptions. Adjust title fonts and sizes Switch btw color themes ... click(30.3, 1001.8) click(3.7, 1224.7) low-level actions workflow steps Edit layout and theme Figure 5: Exemplar work- flow induced from agent low-level action trajectory. Measuring Task ComplexityTo obtain tasks with varied com- plexity, we adopt the workflow induction procedure from Wang et al. (2025b), which segments any agent trajectory휏with low- level actions (e.g.,click) into a hierarchical workflow푤with goal-directed steps at increasing levels of granularity (Figure 5). We quantify the complexity levels of these agent workflow steps (each representing a task of varied granularity), and use them to derive the autonomy levels above. We approximate task complexity by the number of the most granular workflow steps퐺= 푣∈ 푉 | min ℓ∈퐿 dist(푣, ℓ)= 0 , as this level yields the most consistent representation of task structure, across workflows induced from heterogeneous trajectories from different agents and tasks. complexity(푣)= | 퐺∩ Desc(푣) | where Desc(푣)=푢∈ 푉 | 푣⇝ 푢 is the set of descendants of 푣. Traditional behavioral measures of task complexity often conflate intrinsic task demands with variation in human ability (Hackman, 1969). For instance, using human completion time as a complexity indicator (METR, 2025a; Tamkin and McCrory, 2025) entangles task difficulty with variation in human capabilities, potentially biasing the resulting estimates. Similarly, raw agent action counts depend heavily on the specifics of an agent’s action space and execution strategy. In contrast, workflow steps abstract away these execution-level idiosyncrasies by collapsing repetitive low-level actions into semantically independent units that better reflect the underlying task structure. To validate our approximation, we need to verify that: tasks at level-푘+1 are strictly more complex 6 than tasks at level-푘; correspondingly, we can conclude that tasks within the same level exhibit comparable complexity. To validate this assumption, we perform pairwise comparisons between tasks at adjacent levels by prompting an LM to judge, based on task descriptions, whether a level-푘+1 task is more complex than the level-푘task. We sample 10 pairs across all complexity levels from all benchmarks and agent trajectories, and find that this relative granularity criterion is satisfied in 82.6% of comparisons, providing empirical support for the appropriateness of our workflow-based complexity approximation. Definition 2. Autonomy. The extent to which an agent system can complete a task by perceive its environment, make decisions, and take actions without direct human intervention. Operationally, we quantify agent autonomy as the maximum task complexity that an agent can complete end-to-end above a predefined success rate threshold with statistical confidence. This definition captures autonomy as a capability boundary: how complex a task an agent can reliably handle without human assistance. Measuring Agent AutonomyGiven tasks grouped by complexity level, we compute an agent’s success rate at each level푘as푆푅(푘)= 1 |푉 푘 | Í 푣∈푉 푘 status(푣), where푉 푘 =푣 : complexity(푣)= 푘is the set of tasks at complexity level푘,status(푣) ∈ 0, 1means step failure and success from 6 By “more complex”, we mean higher perceived procedural complexity, as reflected in the number of steps a human would anticipate when mentally planning the task without execution. 10 implement a reinforcement learning algo User task requires work output skills Decompose to simpler instructions to achieve desired agent success rate Implement rollout loop Suggested task complexity SWE-benchSWE-bench SUPERSWE-bench-pro Figure 6: Agent autonomy levels by (left) work domains and skills, and (right) agent frameworks and LM backbones. the workflow induction procedure (Figure 6). Given a target success-rate threshold퐻, we define the autonomy level of an agent as the highest task complexity at which the agent succeeds over: Autonomy= max 푘 | 푆푅(푘) ≥ 퐻 . This definition characterizes agent autonomy as a capability boundary, capturing the most complex class of tasks the agent can complete end-to-end with acceptable reliability. Further, this characterization enables users to select appropriate tasks for automation and to calibrate the level of human oversight required for safe and effective deployment. 4.2. Calibrating Agent Autonomy for Real-World Tasks We collect available agent trajectories for all benchmarks in our study (§2), and use them to analyze agent autonomy upon task complexity measures. Figure 6 presents a breakdown of autonomy measures by (left) domain and skill requirements across all agent trajectories, and (right) agent framework and backbone LMs for benchmarks that allow fair comparisons. •Work Expertise: Except for computer, business, and office-related domains, most domains show little to no coverage at higher task complexities, i.e., most tasks have complexity no greater than 6. Even in domains where agents perform best, most notably software engineering, success rates drop sharply as task complexity increases, despite the heavy concentration of tasks in these areas. In contrast, agents show greater autonomy for design tasks. Across the four major skills, agents perform best on self-contained activities such as mental processes and work output, but struggle identifying and retrieving information (information input) or coordinating with others (interacting with others), particularly for simpler tasks. •Agent Selection: A core challenge in comparing agent autonomy is that different benchmarks often evaluate different agent frameworks and LM backbones, making controlled comparisons difficult. On the few coding-focused benchmarks where such comparisons are possible (e.g., SWE-bench), our analysis signals framework (OpenHands over SWE-agent) and LM (Claude over GPT) advantages, particularly for tasks of medium complexity. However, these trends may 11 not be consistent across complexity regimes, motivating broader release of agent trajectories to enable more systematic and reproducible autonomy analysis. Choose the Right Autonomy Level for Your AgentGiven a user task with desired perfor- mance requirements, we provide a principled strategy to decide which autonomy level the agent operates at. As demonstrated in Figure 6, we consider a task (“implement a reinforcement algorithm”) with a target performance threshold퐻(e.g., 80%). Using our task-to-work mapping (§2), the system identifies relevant domains and skills to consult the relevant autonomy curves. In this case, our system suggests that end-to-end execution at the original task complexity falls below the target퐻, thus recommends decomposing the task (e.g., using an LM) into simpler subtasks (e.g., “implement rollout loop”) to assign to the agent for higher success. This example shows how autonomy curves translate abstract performance metrics into actionable decisions for human users. Meanwhile, for agent developers, this curve exposes agent capability boundaries, thus facilitating more targeted development such as crafting tasks at this complexity for training. 5. Discussion 5.1. Related Work AI’s Impact on WorkA central motivation for developing AI agents is their potential to boost productivity in human work. Prior studies have examined AI’s impact on employment (Brynjolfsson et al., 2025), productivity (Tamkin and McCrory, 2025), and broader economic outcomes (Eloundou et al., 2023; Handa et al., 2025). In parallel, researchers have sought to build AI systems explicitly for work, including benchmarking agents at work (Patwardhan et al., 2025; Xu et al., 2024), studying workflows to integrate AI into work (Wang et al., 2025b), and surveys of industry practices and needs (Anthropic, 2025a; Pan et al., 2025; Shao et al., 2025; Shome et al., 2025). Despite this progress, we still lack a unified framework for characterizing agent utility in real-world work, which this work aims to address. Profiling AI AgentsProfiling AI systems offers practical guidance for both developers and users. Prior works mostly profile LLMs by personality or stylistic traits (Dunlap et al., 2024), studies of AI agents focus on specific domains (e.g., engineering (METR, 2025b)) or surface-level architectural depiction (Casper et al., 2025). Despite existing conceptual frameworks (Feng et al., 2025), agent autonomy remains vaguely defined and measured. We thus propose a quantifiable autonomy measure with practical usage guidance. Agent Benchmark DesignBenchmarks play a central role in shaping agent development, yet prior efforts often bias to easily specified or verifiable tasks (Jimenez et al., 2024). Despite the few benchmarks targeting human work (Patwardhan et al., 2025; Xu et al., 2024), we lack a systematic understanding of how benchmarks align with the distribution of real work, which we answer in this work. 5.2. Principles to Benchmark Agents for Work Based on our analysis, we distill three practical principles to guide the design of benchmarks that better reflect real work. Domain and Skill CoverageRather than further focusing on already well-covered domains, benchmarks should either target underrepresented yet highly digitized domains, such as Man- agement and Legal, which also account for a disproportionate share of capital; or aim for broad coverage across domains (as reported in Table 1). On the skill side, benchmarks should move 12 beyond overemphasized granular skills (e.g., Getting Information, Working with Computers) and strive for a more balanced distribution across skill categories (e.g., Interacting with Others). Figure 2 and Figure 3 can act as a guide to identify these domains, and the supplementary website will be updated with new benchmarks as they become available. Ensure Realism and ComplexityOur analysis in Figure 4 indicates that many automatically synthesized benchmarks (e.g., ColBench) exhibit low domain and procedural complexity, captur- ing only simplified fragments of real work. In contrast, human-annotated tasks (e.g., in GDPval or TheAgentCompany) often involve more diverse domains and skills. While human annotation remains the gold standard, when synthesis is required for scalability, task generation should be grounded in realistic domain and skill compositions rather than abstract templates. Granular EvaluationOur analysis in Figure 6 shows that singular end-task evaluations obscure differences in agents across task complexity. Although benchmarks with intermediate checkpoints (Xu et al., 2024) offer more informative assessments, it is costly to scale. As a more practical alternative, human demonstrations can be used to induce workflows (Wang et al., 2025b), enabling granular evaluation via automatically produced intermediate checkpoints. 6. Conclusion This work examined whether and how current AI agent benchmarks reflect real-world work. By situating agent benchmarks within taxonomies of human work domains and skills, we revealed systematic mismatches between where agent development is concentrated and where real-world labor and economic value are distributed. We further measure agent autonomy across work, and translate these insights into practical guidance for different stakeholders: enabling agent benchmark designers to assess gaps in work coverage, agent builders to identify areas of improvement, and agent users to select appropriate agents and autonomy levels for their specific work. We hope this work motivates agent benchmarking and development efforts to more faithfully capture the diversity, complexity, and societal importance of real-world work. Acknowledgments We would like to thank Yijia Shao, Pranjal Aggarwal, Ruiqi Zhong, and many members of the Language Technologies Institute for their helpful discussions and insightful feedback on the project. Zora Zhiruo Wang is supported by Google PhD Fellowship. 13 References Anthropic. How enterprises are building ai agents in 2026, 2025a. URLhttps://claude.com /blog/how-enterprises-are-building-ai-agents-in-2026. Anthropic. Claude 3.7 sonnet and claude code, 2025b. URLhttps://w.anthropic.com/ news/claude-3-7-sonnet. Anthropic and Others. Agentskills. https://agentskills.io, 2025. Accessed: 2026-02-21. L. Bainbridge. Ironies of automation. InAnalysis,designandevaluationofman–machine systems, pages 129–135. Elsevier, 1983. B. Bogin, K. Yang, S. Gupta, K. Richardson, E. Bransom, P. Clark, A. Sabharwal, and T. Khot. SUPER: Evaluating agents on setting up and executing tasks from research repositories. In Proceedingsofthe2024ConferenceonEmpiricalMethodsinNaturalLanguageProcessing. Association for Computational Linguistics, Nov. 2024. doi: 10.18653/v1/2024.emnlp-main.702. URL https://aclanthology.org/2024.emnlp-main.702/. E. Brynjolfsson, B. Chandar, and R. Chen. Canaries in the coal mine? six facts about the recent employment effects of artificial intelligence. Technical report, Working paper. Latest version available at https://digitaleconomy. stanford . . . , 2025. S. Casper, L. Bailey, R. Hunter, C. Ezell, E. Cabalé, M. Gerovitch, S. Slocum, K. Wei, N. Jurkovic, A. Khan, et al. The ai agent index.arXivpreprintarXiv:2502.01635, 2025. J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering.arXivpreprintarXiv:2410.07095, 2024. J. Chen, D. Yuen, B. Xie, Y. Yang, G. Chen, Z. Wu, L. Yixing, X. Zhou, W. Liu, S. Wang, K. Zhou, R. Shao, L. Nie, Y. Wang, J. HAO, J. Wang, and K. Shao. SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION. InTheThirteenthInternational ConferenceonLearningRepresentations, 2025a. URLhttps://openreview.net/forum ?id=OZbFRNhpwr. Z. Chen, S. Chen, Y. Ning, Q. Zhang, B. Wang, B. Yu, Y. Li, Z. Liao, C. Wei, Z. Lu, V. Dey, M. Xue, F. N. Baker, B. Burns, D. Adu-Ampratwum, X. Huang, X. Ning, S. Gao, Y. Su, and H. Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. InTheThirteenthInternationalConferenceonLearningRepresentations, 2025b. URL https://openreview.net/forum?id=6z4YKr0GK6. X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su. Mind2web: Towards a generalist agent for the web. InThirty-seventhConferenceonNeuralInformationProcessing SystemsDatasetsandBenchmarksTrack, 2023. URLhttps://openreview.net/forum?i d=kiYqbO3wqw. X. Deng, J. Da, E. Pan, Y. Y. He, C. Ide, K. Garg, N. Lauffer, A. Park, N. Pasari, C. Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks?arXivpreprint arXiv:2509.16941, 2025. A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme, T. Marty, D. Vazquez, N. Chapados, and A. Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? InForty-firstInternationalConferenceonMachineLearning, 2024. URLhttps: //openreview.net/forum?id=BRfqYrikdo. 14 L. Dunlap, K. Mandal, T. Darrell, J. Steinhardt, and J. E. Gonzalez. Vibecheck: Discover and quantify qualitative differences in large language models.arXivpreprintarXiv:2410.12851, 2024. T. Eloundou, S. Manning, P. Mishkin, and D. Rock. Gpts are gpts: An early look at the labor market impact potential of large language models.arXivpreprintarXiv:2303.10130, 2023. K. Feng, D. W. McDonald, and A. X. Zhang. Levels of autonomy for ai agents.arXivpreprint arXiv:2506.12469, 2025. J. R. Hackman. Toward understanding the role of tasks in behavioral research.Actapsychologica, 31:97–128, 1969. K. Handa, A. Tamkin, M. McCain, S. Huang, E. Durmus, S. Heck, J. Mueller, J. Hong, S. Ritchie, T. Belonax, et al. Which economic tasks are performed with ai? evidence from millions of claude conversations.arXivpreprintarXiv:2503.04761, 2025. H. He, W. Yao, K. Ma, W. Yu, Y. Dai, H. Zhang, Z. Lan, and D. Yu. WebVoyager: Building an end- to-end web agent with large multimodal models. InProceedingsofthe62ndAnnualMeeting oftheAssociationforComputationalLinguistics(Volume1:LongPapers), pages 6864–6890. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl-long.371. URL https://aclanthology.org/2024.acl-long.371/. K.-H. Huang, A. Prabhakar, S. Dhawan, Y. Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu. CRMArena: Understanding the capacity of LLM agents to perform professional CRM tasks in realistic environments. InProceedingsofthe2025ConferenceoftheNationsof theAmericasChapteroftheAssociationforComputationalLinguistics:HumanLanguage Technologies(Volume1:LongPapers). Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.naacl-long.194. URLhttps://aclanthology.org/2025.naacl-long. 194/. P. Jansen, M.-A. Côté, T. Khot, E. Bransom, B. Dalvi Mishra, B. P. Majumder, O. Tafjord, and P. Clark. Discoveryworld: A virtual environment for developing and evaluating automated scientific discovery agents.AdvancesinNeuralInformationProcessingSystems, 37:10088– 10116, 2024. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? InTheTwelfthInternationalConference onLearningRepresentations, 2024. URLhttps://openreview.net/forum?id=VTF8yN QM66. Y. Kim, K. Gu, C. Park, C. Park, S. Schmidgall, A. A. Heydari, Y. Yan, Z. Zhang, Y. Zhuang, M. Malhotra, et al.Towards a science of scaling agent systems.arXivpreprint arXiv:2512.08296, 2025. J. Y. Koh, R. Lo, L. Jang, V. Duvvur, M. C. Lim, P.-Y. Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In ICLR2024WorkshoponLargeLanguageModel(LLM)Agents, 2024. URLhttps://open review.net/forum?id=RPKxrKTJbj. J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnap- ati, A. D. White, and S. G. Rodriques. Lab-bench: Measuring capabilities of language models for biology research.arXivpreprintarXiv:2407.10362, 2024. 15 C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. InConferenceonRobotLearning, pages 80–93. PMLR, 2023. X. H. Lù, Z. Kasner, and S. Reddy. Weblinx: real-world website navigation with multi-turn dialogue. InProceedingsofthe41stInternationalConferenceonMachineLearning, ICML’24. JMLR.org, 2024. B. P. Majumder, H. Surana, D. Agarwal, B. D. Mishra, A. Meena, A. Prakhar, T. Vora, T. Khot, A. Sabharwal, and P. Clark. Discoverybench: Towards data-driven discovery with large language models. InTheThirteenthInternationalConferenceonLearningRepresentations, 2025. URL https://openreview.net/forum?id=vyflgpwfJW. M. Mazeika, A. Gatti, C. Menghini, U. M. Sehwag, S. Singhal, Y. Orlovskiy, S. Basart, M. Sharma, D. Peskoff, E. Lau, et al. Remote labor index: Measuring ai automation of remote work.arXiv preprintarXiv:2510.26787, 2025. METR. How does time horizon vary across domains?https://metr.org/blog/2025-07-1 4-how-does-time-horizon-vary-across-domains/, 07 2025a. METR. Measuring autonomous ai capabilities.https://metr.org/measuring-autonomou s-ai-capabilities/, 2025b. G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. GAIA: a benchmark for general AI assistants. InTheTwelfthInternationalConferenceonLearningRepresentations, 2024. URL https://openreview.net/forum?id=fibxvahvs3. S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?arXivpreprintarXiv:2502.12115, 2025. A. Miyai, Z. Zhao, K. Egashira, A. Sato, T. Sunada, S. Onohara, H. Yamanishi, M. Toyooka, K. Nishina, R. Maeda, K. Aizawa, and T. Yamasaki. Webchorearena: Evaluating web browsing agents on realistic tedious web tasks, 2025. URL https://arxiv.org/abs/2506.01952. A. Moteki, S. Masui, F. Yang, Y. Song, Y. Bisk, G. Neubig, I. Kusajima, Y. Watanabe, H. Ishida, J. Takahashi, et al. Fieldworkarena: Agentic ai benchmark for real field work tasks.arXiv preprintarXiv:2505.19662, 2025. N. Mündler, M. N. Mueller, J. He, and M. Vechev. SWT-bench: Testing and validating real-world bug-fixes with code agents. InTheThirty-eighthAnnualConferenceonNeuralInformation ProcessingSystems, 2024. URL https://openreview.net/forum?id=9Y8zUO11EQ. D. Nathani, L. Madaan, N. Roberts, N. Bashlykov, A. Menon, V. Moens, M. Plekhanov, A. Budhi- raja, D. Magka, V. Vorotilov, G. Chaurasia, D. Hupkes, R. S. Cabral, T. Shavrina, J. N. Foerster, Y. Bachrach, W. Y. Wang, and R. Raileanu. MLGym: A new framework and benchmark for advancing AI research agents. InSecondConferenceonLanguageModeling, 2025. URL https://openreview.net/forum?id=ryTr83DxRq. Z. Ni, H. Wang, S. Zhang, S. Lu, Z. He, W. You, Z. Tang, Y. Du, B. Sun, H. Liu, et al. Gittaskbench: A benchmark for code agents solving real-world tasks through code repository leveraging. arXivpreprintarXiv:2508.18993, 2025. 16 U. S. D. of Labor. Occupational employment and wage statistics (oews) profiles, 2024. URL https://data.bls.gov/oesprofile/. O*NET Center. O*net. the 29.1 release of the o*net database, november 2024.URLhttps: //w.onetcenter.org/dictionary/29.1/excel/, 2024. OpenAI. Gpt-5 is here, 2025. URL https://openai.com/gpt-5/. M. Z. Pan, N. Arabzadeh, R. Cogo, Y. Zhu, A. Xiong, L. A. Agrawal, H. Mao, E. Shen, S. Pallerla, L. Patel, et al. Measuring agents in production.arXivpreprintarXiv:2512.04123, 2025. R. Parasuraman and V. Riley. Humans and automation: Use, misuse, disuse, abuse.Human factors, 39(2):230–253, 1997. T. Patwardhan, R. Dias, E. Proehl, G. Kim, M. Wang, O. Watkins, S. P. Fishman, M. Aljubeh, and P. Thacker. Gdpval: Evaluating ai model performance on real-world economically valuable tasks. 2025. URLhttps://cdn.openai.com/pdf/d5eb7428-c4e9-4a33-bd86-86d4 bcf12ce/GDPval.pdf. Y. Shao, H. Zope, Y. Jiang, J. Pei, D. Nguyen, E. Brynjolfsson, and D. Yang. Future of work with ai agents: Auditing automation and augmentation potential across the us workforce.arXiv preprintarXiv:2506.06576, 2025. P. Shome, S. Krishnan, and S. Das. Why johnny can’t use agents: Industry aspirations vs. user realities with ai agent software.arXivpreprintarXiv:2509.14528, 2025. Z. S. Siegel, S. Kapoor, N. Nadgir, B. Stroebl, and A. Narayanan. CORE-bench: Fostering the credibility of published research through a computational reproducibility agent benchmark. TransactionsonMachineLearningResearch, 2024. ISSN 2835-8856. URLhttps://openre view.net/forum?id=BsMMc4MEGS. A. Tamkin and P. McCrory. Estimating ai productivity gains from claude conversations, 2025. URL https://w.anthropic.com/research/estimating-productivity-gains. T. T.-B. Team. Terminal-bench: A benchmark for ai agents in terminal environments, Apr 2025. URL https://github.com/laude-institute/terminal-bench. M. Tian, L. Gao, S. Zhang, X. Chen, C. Fan, X. Guo, R. Haas, P. Ji, K. Krongchon, Y. Li, et al. Scicode: A research coding benchmark curated by scientists.AdvancesinNeuralInformation ProcessingSystems, 37:30624–30650, 2024. S. Tian, Z. Zhang, L.-Y. Chen, and Z. Liu. Mmina: Benchmarking multihop multimodal internet agents. InFindingsoftheAssociationforComputationalLinguistics:ACL2025, pages 13682– 13697, 2025. H. Trivedi, T. Khot, M. Hartmann, R. Manku, V. Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian. AppWorld: A controllable world of apps and people for bench- marking interactive coding agents. InProceedingsofthe62ndAnnualMeetingofthe AssociationforComputationalLinguistics(Volume1:LongPapers). Association for Com- putational Linguistics, 2024.doi: 1 0 . 1 8 6 5 3 / v 1 / 2 0 2 4 . a c l - l o n g . 8 5 0.URLhttps: //aclanthology.org/2024.acl-long.850/. H. Vishwakarma, A. Agarwal, O. Patil, C. Devaguptapu, and M. Chandran. Can llms help you at work? a sandbox for evaluating llm agents in enterprise environments. InProceedingsofthe 2025ConferenceonEmpiricalMethodsinNaturalLanguageProcessing, pages 9178–9212, 2025. 17 C. Wang, B. Dai, H. Liu, and B. Wang. Towards objectively benchmarking social intelligence of language agents at the action level. InFindingsoftheAssociationforComputational Linguistics:ACL2024. Association for Computational Linguistics, 2024a. URLhttps: //aclanthology.org/2024.findings-acl.526/. Z. Wang, Y. Cui, L. Zhong, Z. Zhang, D. Yin, B. Y. Lin, and J. Shang. Officebench: Bench- marking language agents across multiple applications for office automation.arXivpreprint arXiv:2407.19056, 2024b. Z. Z. Wang, A. Gandhi, G. Neubig, and D. Fried. Inducing programmatic skills for agentic tasks. InSecondConferenceonLanguageModeling, 2025a. URLhttps://openreview.net/f orum?id=lsAY6fWsog. Z. Z. Wang, Y. Shao, O. Shaikh, D. Fried, G. Neubig, and D. Yang. How do ai agents do human work? comparing ai and human workflows across diverse occupations.arXivpreprint arXiv:2510.22780, 2025b. J. Wei, Z. Sun, S. Papay, S. McKinney, J. Han, I. Fulford, H. W. Chung, A. T. Passos, W. Fedus, and A. Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents.arXiv preprintarXiv:2504.12516, 2025. R. E. Wood. Task complexity: Definition of the construct.Organizationalbehaviorandhuman decisionprocesses, 37(1):60–82, 1986. J. Xie, K. Zhang, J. Chen, T. Zhu, R. Lou, Y. Tian, Y. Xiao, and Y. Su. Travelplanner: A benchmark for real-world planning with language agents.arXivpreprintarXiv:2402.01622, 2024a. T. Xie, D. Zhang, J. Chen, X. Li, S. Zhao, R. Cao, T. J. Hua, Z. Cheng, D. Shin, F. Lei, Y. Liu, Y. Xu, S. Zhou, S. Savarese, C. Xiong, V. Zhong, and T. Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. InTheThirty-eightConference onNeuralInformationProcessingSystemsDatasetsandBenchmarksTrack, 2024b. URL https://openreview.net/forum?id=tN61DTr4Ed. F. F. Xu, Y. Song, B. Li, Y. Tang, K. Jain, M. Bao, Z. Z. Wang, X. Zhou, Z. Guo, M. Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks.arXiv preprintarXiv:2412.14161, 2024. J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang, S. Wang, and O. Press. SWE-bench multimodal: Do AI systems generalize to visual software domains?InTheThirteenthInternational ConferenceonLearningRepresentations, 2025. URLhttps://openreview.net/forum?i d=riTiq3i21b. S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. InAdvancesinNeuralInformationProcessing Systems, volume 35, pages 20744–20757. Curran Associates, Inc., 2022. URL https://proc eedings.neurips.c/paper_files/paper/2022/file/82ad13ec01f9fe44c01cb91 814fd7b8c-Paper-Conference.pdf. O. Yoran, S. J. Amouyal, C. Malaviya, B. Bogin, O. Press, and J. Berant. AssistantBench: Can web agents solve realistic and time-consuming tasks?InProceedingsofthe2024 ConferenceonEmpiricalMethodsinNaturalLanguageProcessing. Association for Com- putational Linguistics, Nov. 2024. doi: 1 0 . 1 8 6 5 3 / v 1/ 2 0 2 4 . e m n l p - ma i n . 5 0 5.URL https://aclanthology.org/2024.emnlp-main.505/. 18 Y. Zhang, S. Jiang, R. Li, J. Tu, Y. Su, L. Deng, X. Guo, C. Lv, and J. Lin. Deepplanning: Benchmarking long-horizon agentic planning with verifiable constraints.arXivpreprint arXiv:2601.18137, 2026. B. Zheng, M. Y. Fatemi, X. Jin, Z. Z. Wang, A. Gandhi, Y. Song, Y. Gu, J. Srinivasa, G. Liu, G. Neubig, et al. Skillweaver: Web agents can self-improve by discovering and honing skills. arXivpreprintarXiv:2504.07079, 2025. S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig. Webarena: A realistic web environment for building autonomous agents. InTheTwelfthInternationalConferenceonLearningRepresentations, 2024a. URL https://openreview.net/forum?id=oKn9c6ytLx. X. Zhou, H. Zhu, L. Mathur, R. Zhang, H. Yu, Z. Qi, L.-P. Morency, Y. Bisk, D. Fried, G. Neubig, and M. Sap. SOTOPIA: Interactive evaluation for social intelligence in language agents. InTheTwelfthInternationalConferenceonLearningRepresentations, 2024b. URLhttps: //openreview.net/forum?id=mM7VurbA4r. Y. Zhou, S. Jiang, Y. Tian, J. Weston, S. Levine, S. Sukhbaatar, and X. Li. Sweet-rl: Training multi-turn llm agents on collaborative reasoning tasks.arXivpreprintarXiv:2503.15478, 2025. Y. Zhu, T. Jin, Y. Pruksachatkun, A. Zhang, S. Liu, S. Cui, S. Kapoor, S. Longpre, K. Meng, R. Weiss, et al. Establishing best practices for building rigorous agentic benchmarks.arXiv preprintarXiv:2507.02825, 2025. 19 Figure 7: Work activities annotation provided by O*NET. A. Discussions A.1. Representing Domains using O*NET Handa et al. (2025) directly leverages O*NET data yet relies on LMs to synthesize a task hierarchy with limited human supervision and real work representativeness. Patwardhan et al. (2025) adopts industry sector annotations from the U.S. Bureau of Labor Statistics. We directly adopt the official job family structure from the ONET database to define work domains, aligning our taxonomy with established occupational classifications. Regarding the domain taxonomy, Figure 2 illustrates all top-level job families. Please find the full taxonomy athttps://github.com/zorazrw/ai4work-resources/blob/main/r eal_work/taxonomy/taxonomy_domain.json. A.2. Representing Skills using O*NET We refer to “skills” based annotation, where a skill refers to a particular variety of workflow that an agent must be good at following standards in academia (Wang et al., 2025a; Zheng et al., 2025) and industry (Anthropic and Others, 2025). To avoid confusion, we note that in addition to the Work Activities annotation we adopted in §2, O*NET also provides annotations of “skills”, including Skills and Technical Skills. Nonetheless, these annotations are more conceptual (e.g., “active learning”), focusing more on the developed capacities that facilitate learning or the more rapid acquisition of knowledge, and rarely grounded in tangible contexts of real human work (e.g., writing skill in financial and legal domains). To bridge this gap, we use the terminology from the agent development literature, but the data from “work activities”. Figure 7 shows the first two layers work activities annotation provided in O*NET, which 20 we adopt in our skill taxonomy. Due to space limitations, please find the full skill taxonomy at:https://github.com/zorazrw/ai4work-resources/blob/main/real_work/tax onomy/taxonomy_skill.json. A.3. Agentic Benchmark Selection The Focus on Agentic BenchmarksAgentic benchmarks differ from traditional AI benchmarks in how tasks are structured and evaluated ((Kim et al., 2025; Zhu et al., 2025)). Traditional bench- marks typically assess performance on single-step tasks with fixed procedures (e.g., classification or short-form generation), where success is measured by matching a predefined output. In contrast, agentic benchmarks evaluate systems on multi-step tasks that require planning, tool use, and adaptive decision-making to produce an end-to-end outcome. Performance is therefore assessed based on whether the agent successfully completes the full task rather than a single intermediate prediction. Because single-step benchmarks optimize for isolated operations, they often produce special- ized tools that operate in narrowly scoped settings and offer limited insight into how AI systems interact with complex, real-world workflows. Agentic systems, by contrast, are designed to perform multi-step tasks over longer horizons and under changing conditions. Agent bench- marks therefore more directly probe the capabilities needed for deploying AI systems that can complete tasks independently or support humans across extended workflows—an important open question for real-world use. At the same time, traditional single-step benchmarks remain compatible with our autonomy- level analysis. They typically correspond to low-autonomy (level-1) tasks that serve as building blocks for more complex workflows. Many work domains that appear underrepresented in current agent benchmarks already have strong coverage of such low-level subtasks in existing benchmarks, but these resources are rarely integrated into agent evaluation. We therefore advo- cate for tighter integration between traditional task benchmarks and agent-centric evaluations to support agents that generalize across a broader range of work. Benchmarks Related to Physical Work To improve coverage of physical work, we surveyed existing robotics benchmarks and incorporated two representative datasets that include natural- language task descriptions and multi-step workflows. However, several limitations remain. Many robotics benchmarks primarily focus on low-level motor primitives (e.g., grasping, push- ing, navigation), which do not map cleanly to real-world occupational tasks at the level of abstraction considered in this paper. Among benchmarks featuring more complex tasks, a substantial portion rely on visual goal specifications rather than natural-language instructions, making consistent taxonomy-based mapping difficult. Even in benchmarks that provide natural- language task descriptions, instructions are often highly templated and relatively simple (e.g., “pick up trash” or “move object to location”), limiting their representativeness of realistic physical work contexts. As a result, although the inclusion of these datasets improves coverage of physical tasks, the overall representation of real-world physical work remains limited. We therefore interpret our findings regarding physical domains with appropriate caution and view the development of richer, work-representative robotics benchmarks as an important direction for future research. 21 A.4. Distinguishing Digital and Physical Work As introduced in §2.1, we use an LLM to determine whether a task requirement requires digital or physical work. More specifically, we adoptclaude-sonnet-3.5for efficient annotation of this relatively simple task. We fill in the task requirement using the following prompt: Prompt for Evaluating Goal-Action Consistency You are given an occupational task description, and your task is to classify whether completing this task primarily requires digital work or physical work. Task: task-description Return: DIGITAL or PHYSICAL. Provide a one-sentence justification. B. Benchmark Mapping B.1. Validating LLM Mapping Results To verify the LM-based mapping, we randomly sample 20 examples from each benchmark to ensure coverage across all benchmarks, rather than sampling from a single pooled set. We ask three graduate student annotators from the authors’ institution, who are familiar with the O*NET taxonomy and the study’s annotation guidelines, to grade the mapped work paths following the rubrics in Table 3. Annotators were instructed to review each task description and judge whether the mapped work path (domain and skill) appropriately captured the core intent of the task. In our manually verified sample, the LM’s predicted domain and skill paths align with the human judgment for 92% and 93% of the time, respectively. Remaining disagreements are relatively minor and primarily arise from under-specified natural language task descriptions. For example, a task like “build a music recommendation system” could map to the Media domain given its content focus, or to the Computer and Mathematical domain given that implementation typically involves programming. Without explicit information about the intended execution, human and LM annotators sometimes infer the most appropriate domain differently. Empirically, these disagreements occur primarily in the ColBench and GAIA benchmarks, likely because their task formulations are less explicitly grounded in real-world work contexts. B.2. Permutation-Based Sensitivity Analysis To assess the robustness of our coverage-aware sampling procedure, we perform a permutation- based sensitivity analysis within each sampled benchmark subset. Specifically, for each bench- mark, we randomly permute the order of the sampled tasks and replay our batch-wise stopping rule 500 times. For each run, we record (i) the number of tasks selected before the stopping criterion is met and (i) the resulting domain and skill coverage at termination (Table 2). Across benchmarks, the stopping point is stable: most runs terminate after processing a substantially small portion of the sampled subset, indicating that coverage saturation is consistently detected rather than driven by a particular task ordering. Moreover, the domain and skill coverage achieved at termination closely matches the maximum coverage attainable within the sampled subset, with narrow 95% confidence intervals across permutations. These results suggest that our adaptive stopping rule is not strongly sensitive to task ordering and does not prematurely terminate within the sampled pool. 22 Benchmark SizeDomain Coverage (%)Skill Coverage (%) totalavg95% CIavg95% CIavg95% CI TheAgentCompany175175[175.0,175.0]62.1[62.1,62.1]60.7[60.7,60.7] GDPval 220220[220.0,220.0]77.0[77.0,77.0]82.7[82.7,82.7] Remote Labor Index99[9.0,9.0]27.8[27.8,27.8]35.7[35.7,35.7] WorkArena 30099.6[97.2,102.0]82.5[81.7,83.4]69.6[68.1,71.2] OfficeBench300186.0[183.5,188.5]73.4[72.5,74.3]82.3[81.6,82.9] CRMArena30052.2[50.2,54.2]74.9[73.3,76.5]83.4[82.3,84.6] EnterpriseBench340324.0[321.9,326.1]70.5[70.2,70.8]55.4[55.1,55.8] GitTaskBench 5453.5[53.0,53.9]61.1[60.8,61.5]85.8[85.7,85.9] OSWorld390370.5[367.9,373.1]72.8[72.5,73.1]79.5[79.2,79.9] WebVoyager300110.0[106.9,113.0]72.2[70.9,73.5]67.4[66.1,68.6] WebArena300273.3[270.8,275.8]76.0[75.8,76.3]88.0[87.5,88.5] Mind2Web300182.2[177.3,187.2]644.4[63.1,65.8]65.2[63.8,66.7] WebShop30014.4[13.6,15.2]89.5[88.6,90.4]93.8[93.2,94.4] VisualWebArena30088.9[85.7,92.0]70.5[69.0,71.9]75.5[74.1,76.9] WebLINX 9998.6[98.1,99.1]68.9[68.7,69.0]65.8[65.7,66.0] AppWorld300200.1[197.5,202.6]79.6[79.2,80.1]74.5[73.5,75.5] AssistantBench214166.9[164.9,169.0]87.7[87.5,88.0]87.8[87.1,88.5] SPA-Bench 300193.1[189.6,196.6]71.3[70.5,72.1]72.8[71.8,73.9] MMInA30063.6[60.9,66.2]79.8[78.3,81.2]84.4[83.3,85.6] WebChoreArena300169.0[166.3,171.7]78.6[78.0,79.3]77.4[76.2,78.5] GAIA300241.7[237.1,246.2]62.8[61.9,63.6]60.1[58.9,61.3] TravelPlanner 3008.0[7.6,8.4]95.4[94.9,95.9]97.3[97.0,97.6] DeepPlanning12015.0[14.0,16.0]90.4[89.5,91.3]97.8[97.4,98.3] SWE-bench30089.5[86.7,92.4]80.0[79.0,81.0]69.6[68.1,71.1] TerminalBench232232.0[232.0,232.0]76.4[76.4,76.4]70.4[70.4,70.4] ColBench860815.3[808.8,821.7]68.1[67.8,68.3]67.7[67.4,68.1] SWE-Lancer19843.8[41.9,45.7]79.1[77.7,80.4]77.8[76.4,79.1] SWE-Bench M 30075.3[73.3,77.2]84.7[83.8,85.6]69.4[67.7,71.1] SWE-Bench Pro300150.8[147.4,154.2]78.6[77.9,79.4]72.5[71.2,73.9] MLE-Bench8281.9[81.6,82.1]73.7[73.6,73.8]76.5[76.4,76.5] SWT-Bench30016.3[15.3,17.2]90.1[89.2,90.9]94.0[93.5,94.5] DiscoveryBench300103.7[100.7,106.7]84.1[83.4,84.8]84.5[83.3,85.8] ScienceAgentBench10246.4[45.0,47.7]87.8[87.0,88.6]83.2[82.1,84.2] CORE-Bench4542.2[41.3,43.0]61.3[60.4,62.2]62.8[61.9,63.6] SciCode 8025.6[24.5,27.4]87.9[86.9,89.0]92.1[91.4,92.7] MLGym1917.8[17.5,18.0]83.6[83.2,84.1]75.0[74.3,75.8] DiscoveryWorld5958.3[57.8,58.8]83.0[82.7,83.4]80.3[80.1,80.5] LabBench 30096.5[93.6,99.5]72.6[71.4,73.7]76.2[74.9,77.5] SUPER30034.5[33.1,35.8]87.3[86.4,88.2]83.4[82.3,84.5] Sotopia300207.9[202.9,212.8]65.3[64.2,66.5]73.0[72.2,73.8] STSS 4039.8[39.6,40.0]51.8[51.5,52.1]62.2[61.9,62.4] Behavioral-1K5049.6[49.2,49.9]23.7[24.1,25.4]57.5[57.1,57.8] FieldWorkArena300131.0[128.2,133.7]79.3[78.5,80.1]72.9[71.6,74.3] Table 2: Permutation-based sensitivity analysis of the coverage-aware sampling procedure. For each benchmark, we report the distribution (median and 95% confidence interval over 500 permutations) of (i) the number of tasks selected and (i) Chao1-estimated domain and skill coverage, before stopping. 23 LabelWhen to UseKey Question All CorrectAll assigned locations are appropriate and no relevant locations are missing. The assign- ment set comprehensively and accurately cap- tures the instruction’s requirements. Are all assignments ap- propriate and is the set complete? All WrongMultiple assigned locations are inappropriate and potentially need both addition/removal. The LLM has fundamentally misunderstood the task or assigned completely unrelated lo- cations. Is every single assign- ment inappropriate or unrelated? Missing LocationsAll current assignments are correct, but addi- tional relevant locations should be included to fully capture the instruction’s scope. Are the current assign- ments right, but incom- plete? What’s missing? Extra LocationsThe assignment includes some correct loca- tions but also contains extraneous assign- ments that are not required by the instruction. Which specific assign- ments are inappropriate and should be removed? Table 3: Rubrics for manually validating LM-based mapping results. C. Measuring Agent Autonomy We provide more concrete examples for understanding task complexity levels in featuring agent autonomy in Table 4. LevelExemplar Task 1Navigate to the Plus section of Cambridge Dictionary to access the Contents menu for Image Quizzes. 2Navigating the Google Travel interface menu options. 3Locate the sparql command and verify rdflib Python module installation. 4Define functions to load and preprocess training, validation, and test datasets. 5Authenticate with phone system and retrieve contact relationship information. 6Retrieve API documentation to understand available endpoints and their functionality. 7Inspect JSON file structure and update script to use correct field names for data processing. 8Attempt to find fine-tuning instructions for the TransNormerLLM-385M model in the README file. 9Create test configuration, helper functions, and run initial 2A test suite for Raft implementation. 10Summarize the ModelAdmin class goal and pass the request object to inline class constructor. 11Train models using the improved training script with enhanced features and optimizations. 12Create and verify a filled PDF form with calculated tax credit values. 13Create and debug Python files for SQL generation in Django ORM’s expression system. 14The goal is to implement a reinforcement learning algorithm. Table 4: Exemplar task instructions at varied task complexities. 24