Paper deep dive

cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents

Pranjal Aggarwal, Lawrence Keunho Jang, Sean Welleck, Daniel Fried, Ruslan Salakhutdinov, Jing Yu Koh

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 122

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Computer use agents (CUAs), which use graphical user interfaces (GUIs) to complete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA benchmarks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we propose cua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs. cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more efficient benchmarking and comparison. We believe cua-speedrun will enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

119,146 characters extracted from source content.

Expand or collapse full text

Preprint CUA-SPEEDRUN: STANDARDIZED BENCHMARKING OF THE SPEED OF COMPUTER-USE AGENTS Pranjal Aggarwal ∗ Lawrence Keunho Jang ∗ Sean WelleckDaniel Fried Ruslan SalakhutdinovJing Yu Koh ∗ Carnegie Mellon University pranjala,ljang,jingyuk@cs.cmu.edu ABSTRACT Computer use agents (CUAs), which use graphical user interfaces (GUIs) to com- plete tasks on a computer, have recently surpassed human performance on many standard benchmarks, including difficult long-horizon tasks. Their capabilities are undoubtedly impressive, however, a key barrier to the widespread adoption and deployment of CUAs remains their speed and cost. Progress towards faster yet capable CUAs requires reliable evaluation of their speed, but many CUA bench- marks currently face a reproducibility crisis. Benchmarks are based on complex infrastructure with varying machine and container configurations that confound the evaluation of the execution speed of CUAs. Towards addressing this gap, we pro- posecua-speedrun, which introduces standardized infrastructure and task sets, with a focus on evaluating the speed and efficiency of CUAs.cua-speedrun uses a uniform virtual machine setup and execution pipeline, along with a common agent interface that enables single-agent implementations to operate seamlessly across different benchmarks. Across four different CUA benchmarks, we evaluate how reasoning effort, agent harnesses, and environment latency affect performance, speed, and cost. We find no single model family is optimal for all three; none of the open-weight models are on the frontier, and also, unintuitively, for some models increasing the reasoning effort can speed up task completion, while faster environment input-output can slow down overall task completion time. We also demonstrate that we can effectively reduce the evaluation task set of most CUA benchmarks without degrading overall statistical power, allowing for more effi- cient benchmarking and comparison. We believecua-speedrunwill enable structured progress towards fast, efficient CUAs, unlocking new real-world use cases and applications. All code, infrastructure, and analysis are available at cuaspeedrun.com. 1INTRODUCTION Computer use agents (CUAs) are language model agents that interact with graphical user interfaces (GUIs) to achieve a user-specified goal. CUAs operate over the same user interface as human users, enabling them to potentially automate a much wider variety of tasks without software-specific APIs. When measured in terms of success alone, recent model releases have pushed the capabilities of CUAs past human baselines. The best frontier models today achieve success metrics above human performance on popular CUA benchmarks: Qwen3.8-Max, Claude Mythos 5, GPT-5.5, and Gemini 3.6 Flash report successes ranging from 78.7% to 86.1% on OSWorld-Verified, far above the 72.4% human reference score (Xie et al., 2024). Across long-horizon benchmarks (Aggarwal et al., 2026; Yuan et al., 2026; Jang et al., 2026b), frontier models have also demonstrated the ability to achieve strong performance on complex, long-horizon tasks. Despite their strong performance, a major bottleneck for widespread adoption of current CUAs in practical settings is their speed and cost of deployment (Abhyankar et al., 2025). In particular, we believe that research on the speed and cost efficiency of CUAs has trailed capabilities research, in ∗ Equal contribution. 1 arXiv:2609.40284v1 [cs.LG] 30 Sep 2026 Preprint CUA-SPEEDRUN Representative tasksSame tasks and runtimeFrontier CUA Analysis OSWorld OSWorld2 CUA-World MyPCBench Agent A Actions Screenshots Agent B Agent C No single model family dominates the Pareto frontier Score (%)GPT Claude Gemini Kimi Muse MiniMax GLM Yutori 40 70 100 100300900 Mean task time (s) Pareto frontier More reasoning sometimes lowers cost Gemini 3 Flash Preview Low$1.40 Medium$0.59(−58%) Faster I/O sometimes slows agents GPT-6 Astra at low effort Normal89.5s Fast99.0s(+11%) Figure 1:cua-speedrunoverview.cua-speedrunis a standardized platform for benchmark- ing the performance, speed, and cost of computer-use agents. It combines representative task selection with standardized infrastructure on Modal, enabling consistent comparisons across models, agent harnesses, and reasoning settings. Our findings reveal that more reasoning can reduce both task time and model cost for CUAs, while faster environment I/O can make agents slower. part due to the difficulty of fairly benchmarking speed and cost of different models in a consistent manner. Specifically, CUAs require complex infrastructure to run and evaluate. Different benchmarks use varying execution environments, ranging from local virtual machines to cloud-parallelized desktops (Xie et al., 2024; Bonatti et al., 2025). There are drastic differences in deployment setup, API interfaces, hyperparameter settings, agent harnesses, and even benchmark implementations across the community. Towards addressing the problem of consistent benchmarking of CUA wall-time, and igniting research on improving the speed and deployment practicality of CUAs, we proposecua-speedrun: a standardized platform that can use existing benchmarks to measure the speed and cost efficiency of CUAs (Fig. 1). There are several challenges thatcua-speedrunaddresses, towards standardizing the evaluation of CUAs across different model families (in particular on wall-time metrics): •Uniform CUA evaluation infrastructure. We standardize evaluations through a serverless cloud-based provider 1 , where all evaluations run on the exact same execution pipeline and the same virtual machine setup. • Common agent interfaces. Existing CUA implementations are varied and do not always follow the optimal scaffolds. We implement a common agent interface for executing actions and interacting with the environment while allowing for portable agent harnesses and implementations. This allows a single agent implementation to work across different benchmarks. Our analysis also measures time differences due to factors such as the latency of closed-source models, which can vary by provider. •A maximally representative subset for fast and comparable evaluations. CUA bench- marks take a long time to run. We propose an energy minimization framework that selects a minimal representative subset of tasks within existing CUA benchmarks. This allows us to minimize end-to-end evaluation time while preserving the signal of the results, and maintain the relative ordering of agent performance. Our approach reduces evaluation time on OSWorld by 84.9% while maintaining high correlation with evals run on the full set. We implement these features oncua-speedrun, and benchmark current frontier CUA models on several widely adopted CUA benchmarks, such as OSWorld-Verified (Xie et al., 2024) and OSWorld 2.0 (Yuan et al., 2026). In addition, we also perform evaluations and analysis on CUA- World (Aggarwal et al., 2026) and MyPCBench (Jang et al., 2026a), which test for long-horizon 1 https://modal.com/ 2 Preprint and personalized computer use, respectively.cua-speedrunallows us to maintain a live Pareto ranking of the most cost-effective, fastest, and most capable models. Through an extensive analysis of multiplecua-speedrunruns, we identify several interesting and applicable insights. Success alone hides large differences in speed: on OSWorld, the open-weight Kimi K3 (max reasoning) matches GPT-6 Astra (high), but takes4.4×longer per task, and none of the open-weight models we evaluate (Kimi K3, MiniMax M3, GLM-5V Turbo) lie on the performance–time or performance–cost Pareto frontier of any benchmarks. Speed also interacts with reasoning in unintuitive ways: raising reasoning effort can improve benchmark scores while reducing task completion time, since the added reasoning avoids repeated unsuccessful actions. Using a fast I/O system can also slow agents, as the models have not been trained to interact with desktops in different I/O speed configurations. Our findings highlight the importance of turning our attention beyond performance and success rates. cua-speedrunmaintains a standardized evaluation of the performance and efficiency of CUAs, two dimensions that we believe are essential future research directions. We hope that the public release ofcua-speedrunencourages the research community to work on making CUAs faster, cheaper, and more efficient, paving the way towards more widespread real-world adoption. 2METHOD In this section, we describe our approach to address the aforementioned challenges with benchmarking CUAs. We base our techniques to improve CUA benchmarking on standardizing measurements and infrastructure (Sec. 2.1), selecting representative tasks instead of running entire benchmarks (Sec. 2.2), and building infrastructure that executes and times every agent consistently (Sec. 2.3). 2.1STANDARDIZED COMPUTER-USE EVALUATION Standardizing computer-use evaluations requires invariance across four different dimensions: (1) the agent, driven by a large-language-model; (2) the benchmark, which determines what an agent must accomplish; (3) the environment infrastructure, which is responsible for managing VMs, the environment lifecycle, and action/observation contracts; and (4) the agent loop, which determines how the model interacts with the infrastructure. Many computer-use evaluations often make different choices for these four components, making it difficult to study them independently. For example, adopting a new model may introduce a benchmark-specific action loop, or adopting a new benchmark requires changes to the agent or infrastructure. The goal of our setup is to ensure the decoupling of benchmarks from infrastructure from agent design, in order to ensure fair and independent evaluation across these four dimensions. Problem setup. Following prior computer-use benchmarks, each task specifies an initial desktop state, a natural-language instruction, and a verifier, either a programmatic check or an LLM judge, that scores the agent’s trajectory and final state. Agents observe screenshots of the desktop and act through keyboard and mouse actions; Appendix J.1 gives the formal definition. Comparisons between agents hold the benchmark and infrastructure fixed, and similarly comparisons across benchmarks fix agent and infrastructure, so each change behaves as a clean ablation. Measuring speed. For each task, our infrastructure starts measuring time when the instruction is given to the agent and stops when the agent terminates or reaches the task limit (steps taken, wall-time, or both). Environment provisioning, task setup, agent initialization, and verification occur outside of this interval. The infrastructure records the total task time and additionally separates it into the time spent executing environment operations and the time spent executing agent operations. Thus, the agent time can be fully attributed to the “speed” of the agent, regardless of any time needed to set up CUA evaluations. Additionally, we also log the total cost of model calls. In addition, infra failures are retried, while agent failures end the trajectory, which is then scored. 2.2SELECTING A REPRESENTATIVE SET Choosing a representative evaluation set. A full computer-use evaluation is often expensive because of long trajectories adding to API/GPU cost, and often requires overhead for managing virtual machines (VMs), adding substantial evaluation cost. For example, GPT-5.4 costs approximately 3 Preprint $4000 on the CUA-World-Long benchmark (Aggarwal et al., 2026). We therefore ask the question: can we reduce the size of the benchmark needed for eval, while still capturing the success rate of the agents we evaluate on the full set? We want to choose a subset that keeps the individual agents’ scores similar to their full benchmark scores and preserves the ordering of agents’ performance. Selection criteria based on success. Our goal is to automatically select a subsetKof the bench- mark that is representative of the full benchmark. To ensure that the selected subset generalizes beyond the agents used to construct it, we evaluate the selection procedure using leave-one-agent-out validation. For each agentm, we construct a subset using the results of only the other agents (M-1) and then compare the held-out agent’s score on the selected subset with its score on the full benchmark. We found through multiple iterations of leave-one-model-out evaluation that the best method to select tasks was to minimize the energy distance between the distributions of agent scores on the full benchmark and subset (Sz ́ ekely & Rizzo, 2013). Specifically, letC mi ∈ [0, 1]be the partial score of agent m on task i, and let B mi = 1[C mi = 1] denote exact completion. We represent task i by z i = (C 1i ,...,C Mi ,B 1i ,...,B Mi )∈R 2M .(1) Thus, two tasks are considered similar when agents exhibit similar patterns of partial and exact completion on them. LetD ij =∥z i − z j ∥ 2 and letDbe the mean distance between distinct tasks. For a subset S K of K tasks, we minimize E (S K ) = 1 D 2 KN X i∈S K N X j=1 D ij − 1 K 2 X i,i ′ ∈S K D i ′ − 1 N 2 N X j=1 N X j ′ =1 D j ′ . (2) This objective favors subsets whose tasks represent the performance patterns present in the full benchmark while avoiding redundant tasks with nearly identical patterns. The final term depends only on the full benchmark and is therefore constant during subset selection. We approximately minimize this objective by starting with 100 task subsets and replacing one selected task with an unselected task whenever this reduces the objective. Thus, we retain the subset with the lowest objective value across the 100 runs. During leave-one-agent-out evaluation, this procedure is repeated after removing the held-out agent’s results. After determining the desired subset size, we construct the final deployed subset once using all available agents. What subset is sufficient to preserve model rankings? The previous selection criteria can find the representative set for a given value ofK. Next, we ask for the smallest subset ofKtasks that consistently preserves the ranking of agents by performance. Letρ q (K)be the Spearman correlation between leave-one-agent-out estimates and full benchmark results for evaluation quantityq. We choose K ∗ = min K : min q min k∈K−1,K,K+1 ρ q (k)≥ 0.95 .(3) We require the correlation threshold to hold for (K-1), (K), and (K+1) to prevent us from selecting a value of (K) that performs well only by chance. This criterion selects 50 of 295 OSWorld tasks, 52 of 63 OSWorld2 tasks, 26 of 143 CUA-World tasks, and 38 of 184 MyPCBench tasks (Table 13), which form thecua-speedrunevaluation sets. We also evaluate how well the subsets preserve full-benchmark performance and model comparisons in our experiments (Appendix G). 2.3INFRASTRUCTURE How do we host our infrastructure? We adapt the virtual machine runtime from Gym- Anything (Aggarwal et al., 2026) to run task environments natively in Modal sandboxes. Our infrastructure manages separate sandboxes for agents and environments, ensuring isolation, while Modal sandboxes provide a standardized hosted runtime so that differences in users’ infrastructure do not affect evaluation results. For self-hosted open-weight models, we use vLLM (Kwon et al., 2023) inference servers on fixed L40S GPUs. Keeping the inference servers and GPUs fixed allows consistency in evaluation. We develop a Python library to handle GPU scheduling, environment allocation, and parallelization, with further details in Appendix B. 4 Preprint Action and observation modalities. Following standard practice in computer-use agents (Xie et al., 2024; Qin et al., 2025), the action space consists of keyboard actions (e.g., typing text or pressingCtrl+C) and mouse actions (e.g., clicking at coordinates(x,y)or double-clicking). Agent observations are RGB screenshots of the desktop at a resolution of 1920× 1080. To ensure that these interfaces behave consistently across VM runtimes, we developed CUA- AutoDebug, an end-to-end test suite in which a controlled application records the keyboard and mouse inputs it receives, compares them with the intended inputs for a large catalog of actions, and checks that screenshots match the application state. These tests uncovered several input errors in widely used CUA tools, which we correct in our infrastructure (Appendix I). FastCUA: Developing a new fast I/O system for CUAs.Standard computer-use runtimes typically take roughly 2–3 seconds from issuing an action to receiving an observation. This latency is dominated by a variety of elements, such as action execution, waiting for the application to respond, and networking. The delay incidentally gives the application time to process the action (e.g., open a menu), allowing the agent to observe its effect. We therefore follow this design for our main evaluations. However, we develop a fast I/O mode, FastCUA, by optimizing networking, action execution, and image processing latencies (see Appendix B). FastCUA reduces action-to-observation latency to 2–28 ms, an improvement of more than an order of magnitude over typical runtimes. At this speed, a screenshot can be captured before the application has updated its UI in response to an action. We evaluate this setting to test whether lower infrastructure latency reduces overall wall time. Implementing agents oncua-speedrun.Since we have standardized infrastructure, new agents can be implemented with a single Python file that contains the agent loop. We also provide standard templates for various popular agents, which users can tweak directly. We provide options for both hosted evaluations and local evaluations through a single CLI call. See Appendix C for more details. 3RESULTS 3.1EXPERIMENTAL SETUP We evaluate 56 agent configurations on OSWorld and 21 on OSWorld2 using the representative task sets from Sec. 2.2. These configurations cover frontier open-weight and proprietary models, each run through its public reference agent implementation or native computer-use API at selected reasoning-effort settings, with the tasks and infrastructure held fixed (Appendix J). We validate all agent designs for correctness through manual inspection and CUA-AutoDebug on representative tasks. For selected models, we also compare different agent harness designs, standard and fast I/O, and single versus batched tool calls per model response. Repeated evaluations are stable: across five seeds, GPT-6 Astra (low) scores90.8± 1.8% on OSWorld with a mean task time of89.5± 2.2s (Appendix F). Metrics. For each configuration, we report the mean verifier score, which retains partial credit, together with the mean task time (Sec. 2.1) and mean model cost per task. We report the performance– time and performance–cost Pareto frontiers, as well as the joint frontier over all three. Interaction turns, generated tokens, and exact completion are defined in Appendix D. 3.2ANALYSIS No single model family dominates the frontier. Fig. 2 shows the performance–time and performance–cost Pareto frontiers on both benchmarks. On OSWorld, the high-performance end of the time frontier extends from Claude Opus 5 (low), with a score of 87.6% in 86 s, through GPT-6 Astra (low), with 90.8% in 90 s, to Astra (xhigh), with 91.6% in 127 s. Gemini 3.8 Flash (low) matches the highest score and takes almost the same time as GPT-6 Astra, but costs $0.11 rather than $0.71 per task. At the lower-cost end, GPT-5.6 Luna (low, direct API) costs about $0.01 per task but scores 61.8%. Thus,cua-speedrunenables comparison across various dimensions such as speed, cost and performance, rather than reducing the comparison to a single model ranking. 5 Preprint 75150300600 Mean task time (s) 30 50 70 90 Mean score (%) Astra xhigh (a) OSWorld: time 0.010.030.10.31 Mean model cost ($/task) 30 50 70 90 Gemini 3.8 low (b) OSWorld: cost 750150030005000 Mean task time (s) 0 20 40 60 80 Mean score (%) Astra xhigh (c) OSWorld2: time 0.31310 Mean model cost ($/task) 0 20 40 60 80 Gemini 3.8 med (d) OSWorld2: cost Model GPT-6 Astra Gemini 3.8 Flash Claude Opus 5 Claude Sonnet 5 GPT-5.6 Sol GPT-5.6 Luna Kimi K3 Muse Spark 1.3 GLM-5V Turbo Yutori n2 Reasoning effort (darker within each model color) None Minimal/off Low Medium High xhigh Max On Figure 2: Different agents define the speed and cost frontiers. Mean score vs. mean task time and model cost on OSWorld (top, 54 configs) and OSWorld2 (bottom, 19 configs), excluding MiniMax M3. Dashed lines connect the observed Pareto-frontier points, which are circled. Colors and marker shapes identify models, and darker shades indicate increased reasoning effort. The plots compare complete agent configs. Costs use recorded charges or token usage at the applicable API prices. Astra (low) uses five-seed means for each I/O setting. Full configs and plots in Appendix E. Low MedHigh 0 200 400 600 Mean time (s) 492 268 337 (a) Task time Low MedHigh 0 2K 4K 6K Mean tokens/task 1,598 4,125 5,167 (b) Generated tokens Low MedHigh 0 20 40 60 Mean calls/task 56.0 32.5 37.9 (c) Model calls Figure 3: Less reasoning can make an agent slower. Gemini 3 Flash Preview at low, medium, and high effort on OSWorld. The panels show mean task time, generated tokens including reasoning, and model calls across all 50 tasks. Scores are 33.6%, 57.6%, and 59.6%, respectively. Medium and high effort generate more tokens than low effort but require fewer calls and complete tasks faster. The Pareto frontier changes across benchmarks. Fig. 2 also shows that, unlike the OSWorld time frontier, the OSWorld2 time frontier is formed entirely by GPT-6 Astra configurations. Astra (high) achieves 75.0% in 914 s, while xhigh reaches 76.9% in 1,237 s, achieving an additional 1.9 percentage points with 35% more time. However, the cost frontier includes other models such as Gemini 3.8 Flash (medium), which achieves 59.9% at $2.91 per task, compared to Astra (high) at $7.39. Muse Spark 1.3 (minimal) extends the frontier to lower cost, with a score of 34.0% at an estimated $0.39. Surprisingly, there is no open model on either the cost or the time frontier for either OSWorld or OSWorld2. Tracking these frontiers allows us to highlight the trade-off explicitly. The preferred agent depends on both the benchmark and the performance required within a time or cost budget, and there is no universal model family that dominates across different benchmarks. Less reasoning can sometimes make agents slower. Fig. 3 compares Gemini 3 Flash Preview at different reasoning settings on OSWorld, keeping the model and harness fixed. Moving from low to medium effort improves the score from 33.6% to 57.6% while reducing mean task time from 492 s to 268 s. We diagnose this as follows: at lower reasoning effort, while the model generates fewer tokens 6 Preprint 100150200250300350 Mean task time (s) 85 90 95 Mean score (%) low $0.11 med $0.22 high $0.35 OSWorld: Gemini 3.8 Flash 1500200025003000 Mean task time (s) 30 40 50 60 70 Mean score (%) low $1.88 med $2.91 high $5.02 OSWorld2: Gemini 3.8 Flash Figure 4: The useful reasoning setting changes across benchmarks. Gemini 3.8 Flash at low, medium, and high effort; labels also report mean cost per task. Each curve keeps the model, agent harness, and execution settings fixed. Additional reasoning brings no gain on OSWorld, whereas moving from low to medium substantially improves OSWorld2 performance. NormalFast I/O 0 50 100 Mean task time (s) 89.5 99.0 (a) Total task time Dialog not yet visible (b) First screenshot Save As is open Second screenshot No intervening action; another observation 5.8s later Agent timeAgent-requested waitsOther environment processing Figure 5: Faster I/O can be offset by more waiting and agent interaction. GPT-6 Astra (low), with normal and fast I/O. Panel (a) separates agent time, agent-requested waits, and remaining environment processing, averaged over five seeds per setting; error bars show the standard deviation of mean task time across seeds. Panel (b) shows two consecutive screenshots after opening Save As: the dialog is absent in the first and visible in the second, requested 5.8 s later with no intervening action. (1,598 vs. 4,125 at medium effort), it takes substantially more steps (56.0 vs. 32.5), since most of its steps are suboptimal. For instance, the trajectories show repeated unsuccessful actions at low effort, such as repeatedly trying to enter spreadsheet text through key combinations instead of text entry. Counterintuitively, higher reasoning effort can therefore shorten the interaction enough to reduce total task time, even when it increases the number of generated tokens per step. More reasoning can increase time and cost without improving performance. Fig. 4 compares reasoning settings within Gemini 3.8 Flash on each benchmark. On OSWorld, low and medium rea- soning both score 91.6%, but medium increases mean task time from 127 s to 250 s and approximately doubles cost. However, the trend changes on OSWorld2: moving from low to medium improves performance from 35.3% to 59.9%. But increasing effort further to high adds only 0.2 percentage points while increasing time by 49% and cost by 72%. GPT-6 Astra shows a similar dependence on the benchmark: on OSWorld, medium effort is within 2 points of xhigh at 29% less time and 15% less cost, whereas on OSWorld2, high effort gains 7.8 points over medium for 10% more time. The useful reasoning setting therefore depends on the tasks as well as the model: additional reasoning can be valuable on one (harder) benchmark and unnecessary on another (simpler) benchmark. Faster infra can make the agent slower. Fig. 5 compares GPT-6 Astra (low) with the normal input-output mode and our faster input-output mode, FastCUA (Sec. 2.3 and Appendix B.3) across five seeds per setting. Interestingly, while the mean environment processing time falls from 12.68 s to 0.34 s per task, total task time increases from 89.5 s to 99.0 s. We also find that the number of steps per trajectory increases in FastCUA. This is because fast I/O can return screenshots before an application has updated its interface. For example, after opening Save As, the agent receives a screenshot without the dialog and requests another screenshot 5.8 s later, which shows it open. 7 Preprint Although theoretically a model could account for such faster I/O by using appropriate wait times (e.g., 100 ms in this case) for the application to update, we find that current models are not able to optimally utilize this, resulting in much higher times. The agent harness plays an important role in the speed–performance trade-off. Fig. 17 com- pares GPT-5.6 Luna through the direct-API agent and the Codex harness on OSWorld. At low effort, Codex improves the score from 61.8% to 75.6%, but increases mean task time from 74 s to 145 s. The same model and reasoning setting therefore produce different performance and efficiency when used through different harnesses. At medium and high effort, the direct-API agent is instead both more accurate and faster (75.6% in 119 s vs. 67.6% in 189 s at medium; 81.6% in 161 s vs. 79.6% in 325 s at high), so neither harness is uniformly better. Batching actions reduces task time.Each step costs a screenshot, a model call, and environment latency, so agents that issue several actions per response (which we execute sequentially) finish sooner: GPT-6 Astra (low) completes OSWorld tasks in 5.9 action batches on average (Table 11). Restricting GPT-6 Astra (xhigh) to one action per response keeps its score at 91.6% but raises mean task time from 127 s to 183 s. The benefit depends on how well a model plans multi-action sequences, and batching alone does not make an agent fast (Appendix K.1). Fast agents generate fewer tokens, not tokens faster.Claude Opus 5 (low) generates 1,795 tokens per task compared with 1,579 for GPT-6 Astra (low), while scoring 87.6% vs. 90.8%. At a common token rate (Fig. 10), it falls behind the Astra configurations, which remain the fastest high-scoring agents. Similarly, Claude Sonnet 5 (high) takes 141 s for the score that Astra (medium) reaches in 90 s, generating3.5×more tokens, and Gemini 3.8 Flash (high) generates5.6×more tokens than at low effort and finishes 2.7× later. Extending to other benchmarks. We additionally evaluate GPT-6 Astra with Codex at four reasoning settings on MyPCBench (Jang et al., 2026a) and CUA-World (Aggarwal et al., 2026), which test personalized and long-horizon computer use and grade trajectories with vision-language models rather than programmatic checks. Astra exceeds 90% rubric scores on both (Appendix E.5), highlighting the need for more challenging task sets. 3.3PRACTICAL SUGGESTIONS FOR BUILDING CUAS Based on the analysis above, we make the following suggestions for building efficient CUAs: • Choose a frontier configuration for the required performance and budget. No single configuration is best on every metric, and the frontier shifts across benchmarks (Figs. 2 and 6), so candidates should be measured on tasks that resemble the target workload. • Tune reasoning effort rather than setting it to an extreme. Too little reasoning can lengthen trajectories, while too much can add time and cost without improving performance (Figs. 3 and 4). Start with low or medium effort and raise it only for a measured gain. •Batch actions when intermediate screenshots are unnecessary. Fewer steps mean fewer screenshots, model calls, and environment round trips. • Reduce generated tokens and steps rather than maximizing token rate or I/O speed. Faster generation or I/O helps only if the agent does not spend it on longer outputs or additional steps (Fig. 5). • Optimize the harness alongside the model. The same model and reasoning setting can land at different points on the frontier depending on its harness (Fig. 17). 4RELATED WORK Computer-use agent benchmarks. The first computer-use agent benchmarks used synthetic interfaces (Liu et al., 2018; Yao et al., 2022). Follow-up evaluations moved to self-hosted realistic website analogs (Zhou et al., 2024; Koh et al., 2024; Drouin et al., 2024), static datasets of real live website tasks (Deng et al., 2023), and live-Internet evaluations (He et al., 2024; Xue et al., 8 Preprint $0.01 $0.03 $0.1 $0.3 $1 75 150 300 600 30% 50% 70% 90% OSWorld score (%) Cost / task Time / task (s) Figure 6: Choosing an agent by performance, time, and cost. Joint Pareto frontier on OSWorld; the OSWorld2 frontier is in Appendix E, Fig. 13. Circled points are the configurations on the Pareto frontier. Time and cost axes are logarithmic. 2025). Live, dynamic desktop benchmarks such as OSWorld (Xie et al., 2024) evaluate agents on real Linux desktops using programmatic verifiers, with WindowsAgentArena (Bonatti et al., 2025), macOSWorld (Yang et al., 2025), iOSWorld (Jang et al., 2026c), and AndroidWorld (Rawles et al., 2025) extending coverage to other platforms and professional workflows. Recent benchmarks stress long horizons and scale, such as OSWorld 2.0 (Yuan et al., 2026), CUA-World from Gym- Anything (Aggarwal et al., 2026), and Odysseys (Jang et al., 2026b). Another emphasis has become personal assistant evaluations, including OpenClaw-style works such as ClawBench (Zhang et al., 2026b), WildClawBench (Ding et al., 2026), and MyPCBench (Jang et al., 2026a), which evaluate personalized computer use and digital assistants. OSWorld-Human (Abhyankar et al., 2025) measures agent latency on OSWorld and compares step counts against human reference trajectories, finding that agents take longer than humans. Computer-use agent models. Frontier model releases have increasingly prioritized computer use (OpenAI, 2026; Anthropic, 2026; Google, 2026; Qwen Team, 2026), with several reporting above- human scores on OSWorld-Verified. Models such as Kimi K3 (Kimi Team, 2026), GLM-5 (GLM-5 Team, 2026), and Muse Spark (Menghini et al., 2026) have also included agentic workflows in their tech releases. Alongside these large general models, a line of work trains specialized computer-use models, including UI-TARS (Qin et al., 2025), MolmoWeb (Gupta et al., 2026), Fara-1.5 (Awadallah et al., 2026), ScaleCUA (Lv et al., 2026), and Qwen-CUA (Lu et al., 2026), often in verifiable environments generated at scale (Wang et al., 2026; Aggarwal et al., 2026). Reproducible and efficient evaluation.Agent evaluations are noisy (Kapoor et al., 2024; 2026), as Xue et al. (2025) showed that reported web-agent success is inflated by evaluation issues, and Sahu & Pandey (2026) show that single-run claims are unreliable because outcomes vary substantially across data draws and run non-determinism. A separate line of work reduces the evaluation cost by selecting a small subset of a benchmark that predicts full-benchmark scores, via item response theory (Polo et al., 2024; Kipnis et al., 2025; Zhou et al., 2026), sparse optimization (Zhang et al., 2026a), or human validation (SWE-bench Verified; Chowdhury et al., 2024). PACE (Song et al., 2026) predicts agentic benchmark scores from cheap, non-agentic evaluations. 5CONCLUSION In this work, we introducedcua-speedrun, a framework for evaluating the performance, speed, and cost of computer-use agents under standardized infrastructure. Our evaluations on OSWorld and OSWorld2 identify the performance–time and performance–cost Pareto frontiers and show how these change across benchmarks. We further find that more reasoning can reduce task time by avoiding repeated unsuccessful actions, while faster I/O can increase it through additional interaction. These 9 Preprint results highlight the need to study reasoning and interaction together when developing faster agents. We hopecua-speedrunenables the community to improve both capability and efficiency, making computer-use agents more practical to deploy. ACKNOWLEDGEMENTS We thank Modal for their generous support in cloud compute credits to buildcua-speedrun. We thank Eunsu Kim, Naveen Raman, Mareks Woodside, Seungone Kim, Zeyu Zheng, and others for feedback and helpful discussions. Jing Yu Koh is supported by a Jane Street Graduate Research Fellowship. Lawrence Jang is supported by a Susquehanna International Group PhD Fellowship. Pranjal Aggarwal is supported by a SoftBank Group-Arm Fellowship. This work is partially supported by the National Science Foundation under Grant No. DMS-2502281, and a grant from Amazon on Useful Chain of Thought Reasoning. AI USE STATEMENT We used AI assistance for general software coding, making figures, polishing writing across the manuscript, and verifying related work. REFERENCES Reyna Abhyankar, Qi Qi, and Yiying Zhang. OSWorld-Human: Benchmarking the efficiency of computer-use agents. arXiv preprint arXiv:2506.16042, 2025. Pranjal Aggarwal, Graham Neubig, and Sean Welleck. Gym-Anything: Turn any software into an agent environment. arXiv preprint arXiv:2604.06126, 2026. Anthropic. Claude Fable 5 and Claude Mythos 5.https://w.anthropic.com/news/ claude-fable-5-mythos-5, 2026. Ahmed Awadallah, Sahil Gupta, Yash Lara, Yadong Lu, Hussein Mozannar, Akshay Nambi, Zach Nussbaum, Yash Pandya, Aravind Rajeswaran, Corby Rosset, Alexey Taymanov, Luiz do Valle, Vibhav Vineet, Spencer Whitehead, and Andrew Zhao. Fara-1.5: Scalable learning environments for computer use agents. arXiv preprint arXiv:2606.20785, 2026. Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows Agent Arena: Evaluating multi-modal OS agents at scale. In ICML, 2025. Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, and Aleksander Madry. Introducing SWE-bench Verified.https://openai.com/index/ introducing-swe-bench-verified/, 2024. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2Web: Towards a generalist agent for the web. In NeurIPS, 2023. Shuangrui Ding, Xuanlang Dai, Long Xing, Shengyuan Ding, Ziyu Liu, Yang JingYi, Penghui Yang, Zhixiong Zhang, Xilin Wei, Xinyu Fang, Yubo Ma, Haodong Duan, Jing Shao, Jiaqi Wang, Dahua Lin, Kai Chen, and Yuhang Zang. WildClawBench: A benchmark for real-world, long-horizon agent evaluation. arXiv preprint arXiv:2605.10912, 2026. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. WorkArena: How capable are web agents at solving common knowledge work tasks? In ICML, 2024. GLM-5 Team. GLM-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Google.Introducing Gemini 3.6 Flash, 3.5 Flash-Lite, and 3.5 Flash Cyber.https:// blog.google/innovation-and-ai/models-and-research/gemini-models/ gemini-3-6-flash-3-5-flash-lite-3-5-flash-cyber/, 2026. 10 Preprint Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, Harsh Trivedi, Taylor Blanton, Caleb Ouellette, Winson Han, Ali Farhadi, and Ranjay Krishna. MolmoWeb: Open visual web agent and open data for the open web. arXiv preprint arXiv:2604.08516, 2026. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVoyager: Building an end-to-end web agent with large multimodal models. In ACL, 2024. Lawrence Keunho Jang, Andrew Keunwoo Jang, Jing Yu Koh, and Ruslan Salakhutdinov. MyPCBench: A benchmark for personally intelligent computer-use agents. arXiv preprint arXiv:2606.16748, 2026a. Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. Odysseys: Bench- marking web agents on realistic long horizon tasks. arXiv preprint arXiv:2604.24964, 2026b. Lawrence Keunho Jang, Mareks Woodside, Geronimo Carom, Andrew Keunwoo Jang, Jing Yu Koh, and Ruslan Salakhutdinov. iosworld: A benchmark for personally intelligent phone agents, 2026c. URL https://arxiv.org/abs/2606.09764. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. AI agents that matter, 2024. URL https://arxiv.org/abs/2407.01502. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, and Arvind Narayanan. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/pdf/2510.11977. Kimi Team. Kimi K3: Open frontier intelligence. arXiv preprint arXiv:2607.24653, 2026. Alex Kipnis, Konstantinos Voudouris, Luca M. Schulze Buschoff, and Eric Schulz. metabench – a sparse benchmark of reasoning and knowledge in large language models. In ICLR, 2025. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In ACL, 2024. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with PagedAttention. In SOSP, 2023. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. In ICLR, 2018. Dunjie Lu, Shuai Bai, Tianyi Bai, et al. Qwen-CUA: Native computer use for (almost) everything. arXiv preprint arXiv:2608.02352, 2026. Bowen Lv, Xiao Liu, Yanyu Ren, Hanyu Lai, Bohao Jing, Hanchen Zhang, Yanxiao Zhao, Shuntian Yao, Jie Tang, and Yuxiao Dong. ScaleCUA: Scaling computer use agents with verifiable task synthesis and efficient online RL. arXiv preprint arXiv:2607.11185, 2026. Cristina Menghini, Peter Ney, Hamza Kwisaba, Zifan Wang, Miles Turpin, et al. Muse Spark safety & preparedness report. arXiv preprint arXiv:2606.12429, 2026. OpenAI. Introducing GPT-6. https://openai.com/index/gpt-6-astra//, 2026. Felipe Maia Polo, Lucas Weber, Leshem Choshen, Yuekai Sun, Gongjun Xu, and Mikhail Yurochkin. tinyBenchmarks: evaluating LLMs with fewer examples. In ICML, 2024. 11 Preprint Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: Pioneering automated GUI interaction with native agents. arXiv preprint arXiv:2501.12326, 2025. Qwen Team. Qwen3.8-Max: A new bar for coding and cowork.https://qwen.ai/blog?id= qwen3.8, 2026. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A dynamic benchmarking environment for autonomous agents. In ICLR, 2025. Barada Sahu and Shivesh Pandey. Teach it to stop, not just to click. arXiv preprint arXiv:2607.17136, 2026. Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, and Graham Neubig. PACE: A proxy for agentic capability evaluation. arXiv preprint arXiv:2607.02032, 2026. G ́ abor J. Sz ́ ekely and Maria L. Rizzo. Energy statistics: A class of statistics based on distances. Journal of Statistical Planning and Inference, 143(8):1249–1272, 2013. Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, and Tao Yu. CUA- Gym: Scaling verifiable training environments and tasks for computer-use agents. arXiv preprint arXiv:2605.25624, 2026. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. In NeurIPS, 2024. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? Assessing the current state of web agents. In COLM, 2025. Pei Yang, Hai Ci, and Mike Zheng Shou. macOSWorld: A multilingual interactive benchmark for GUI agents. In NeurIPS, 2025. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. WebShop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS, 2022. Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Kaiqian Cui, Bowen Wang, Haoyuan Wu, Yitong Li, Dunjie Lu, Haikong Lu, Qi Zhen, Xinyuan Wang, Jiaqi Deng, Yuhao Yang, Cheng Chen, Boyuan Zheng, Alex Su, Xiao Yu, Hao Zou, Saaket Agashe, Xing Han Lu, Manpreet Kaur, Zhengyang Qi, Vincent Sunn Chen, Frederic Sala, Dayiheng Liu, Junyang Lin, Zhou Yu, Yu Su, Siva Reddy, Xin Eric Wang, Peng Qi, Tianbao Xie, and Tao Yu. OSWorld 2.0: Benchmarking computer use agents on long-horizon real-world tasks. arXiv preprint arXiv:2606.29537, 2026. Taolin Zhang, Hang Guo, Wang Lu, Tao Dai, Shu-Tao Xia, and Jindong Wang. SparseEval: Efficient evaluation of large language models by sparse optimization. In ICLR, 2026a. Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Zhuofeng Li, Xingwei Qu, Zhengkang Guo, Yuanzhe Shen, Dingjie Song, Han Zhou, Tuney Zheng, Xian Wu, Hao Yu, Songcheng Cai, Yi Lu, Yunzhuo Hao, Minyi Lei, Liang Chen, Kai Zou, Huifeng Yin, Wendong Xu, Dongfu Jiang, Ping Nie, Jiaheng Liu, Wenhu Chen, and Kelsey R. Allen. ClawBench: Can AI agents complete everyday online tasks? arXiv preprint arXiv:2604.08523, 2026b. Hongli Zhou, Hui Huang, Ziqing Zhao, Lvyuan Han, Huicheng Wang, Kehai Chen, Muyun Yang, Wei Bao, Jian Dong, Bing Xu, Conghui Zhu, Hailong Cao, and Tiejun Zhao. Lost in benchmarks? Rethinking large language model benchmarking with item response theory. In AAAI, 2026. 12 Preprint Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A realistic web environment for building autonomous agents. In ICLR, 2024. 13 Preprint APPENDIX TABLE OF CONTENTS A Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B Infrastructure: Technical Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.1 Agent and Environment Sandboxes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.2 Model Serving, Resource Allocation, and Parallel Evaluation . . . . . . . . . . . . . . . . . 16 B.3 Fast I/O Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 B.4 Action Latency: Fast I/O, Gym-Anything, and OSWorld . . . . . . . . . . . . . . . . . . . 17 B.5 Latency Breakdown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 B.6 Fast I/O Consistency Across Five Seeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 C Implementing and Evaluating an Agent . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 C.1 Agent Templates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 C.2 Minimal Agent Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 C.3 Hosted and Local Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 D Interaction and Token Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 D.1 Model Calls, Action Batches, and Individual Actions . . . . . . . . . . . . . . . . . . . . . 20 D.2 Token Accounting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 D.3 Full Interaction and Token Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 D.4 Performance–Interaction and Performance–Token Frontiers . . . . . . . . . . . . . . . . . . 23 E Full Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 E.1Model, Harness, and Reasoning-Effort Configurations. . . . . . . . . . . . . . . . . . . . . 25 E.2OSWorld: Performance–Time and Performance–Cost Frontiers . . . . . . . . . . . . . . . . 27 E.3OSWorld2: Performance–Time and Performance–Cost Frontiers . . . . . . . . . . . . . . . 28 E.4OSWorld2: Joint Performance–Time–Cost Frontier . . . . . . . . . . . . . . . . . . . . . . 29 E.5Reasoning Effort on MyPCBench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 E.6Reasoning Effort on CUA-World . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 F Run-to-Run Variation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 F.1Five-Seed Evaluation Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 F.2Scores and Times Across Seeds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 F.3Task-Level Score Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 F.4Variation in Action Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 G Representative Task Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 G.1 Models and Tasks Used for Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 G.2 Initialization, Optimization, and Task Count. . . . . . . . . . . . . . . . . . . . . . . . . . 33 G.3 Held-Out Score and Ranking Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 G.4 Preserving the Speed–Performance Frontier . . . . . . . . . . . . . . . . . . . . . . . . . . 36 G.5 Comparing Task-Selection Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 14 Preprint G.6 Selected Task Identities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 H Internet-Dependence Audit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 H.1 Inclusion Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 H.2 Reviewer Agreement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 IValidating Actions with CUA-AutoDebug. . . . . . . . . . . . . . . . . . . . . . . . . . . 40 I.1Separating Model, Harness, and Runtime Errors. . . . . . . . . . . . . . . . . . . . . . . . 40 I.2Input Failures Detected by the Runtime Tests . . . . . . . . . . . . . . . . . . . . . . . . . 40 I.3Failures in Agent Harnesses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 JExperimental Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 J.1Task Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 J.2Observations, Actions, and Interaction History . . . . . . . . . . . . . . . . . . . . . . . . 41 J.3Task Limits and Execution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 J.4Benchmark Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 J.5Model Cost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 K Additional Agent Analyses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 K.1 Controlled Batching Comparisons. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 K.2 Harness Comparisons Across Reasoning Efforts. . . . . . . . . . . . . . . . . . . . . . . . 44 K.3 Comparing Reasoning Efforts on the Same Successful Tasks . . . . . . . . . . . . . . . . . 45 L Automated Search for Task-Selection Methods . . . . . . . . . . . . . . . . . . . . . . . 46 L.1Search and Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 L.2Validation Gains Do Not Generalize to the Held-Out Model . . . . . . . . . . . . . . . . . . 46 15 Preprint ALIMITATIONS Reported times and costs reflect API inference conditions at the time of evaluation (Appendix J). Providers control subsequent changes to inference speed and pricing. We validate representative task subsets on held-out agents (Appendix G); subset scores approximate full-benchmark performance. Changes in agent capabilities may require revalidation of these subsets. Our evaluations cover a broad range of agents across four benchmarks (Appendix E). Evaluation cost limits coverage of every combination of model, harness, reasoning effort, and benchmark. BINFRASTRUCTURE: TECHNICAL DETAILS B.1AGENT AND ENVIRONMENT SANDBOXES We run the agent and task environment in separate sandboxes. The environment contains the desktop, applications, and initial task state; the agent interacts with it through screenshot and action requests. Task initialization and verification remain independent of the agent, allowing the same agent implementation to run across benchmarks. When the agent terminates or reaches its limit, the benchmark’s verifier scores the trajectory or final environment state. We use Modal to host the sandboxes and adapt the environment runtime from Gym-Anything (Ag- garwal et al., 2026). The environment measures task time and the time spent executing actions and returning observations, independently of the agent implementation. As in Sec. 2.1, task preparation, agent initialization, and verification are excluded from task time. B.2MODEL SERVING, RESOURCE ALLOCATION, AND PARALLEL EVALUATION Agents access models through API endpoints or a self-hosted vLLM inference server (Kwon et al., 2023). We use L40S GPUs for self-hosted inference. The evaluation configuration specifies the compute resources, task limits, and number of concurrent agents. Model initialization and weight loading finish before task timing begins. We assign tasks to available agents and environments. Each task starts from a fresh environment state. Its clock starts after the agent and environment are ready and stops when the agent finishes or reaches its limit, excluding time waiting for an execution slot. Model servers reuse loaded weights across tasks. We record a trajectory for each task. B.3FAST I/O IMPLEMENTATION FastCUA, our Fast I/O implementation, reduces the time between issuing an action and receiving a screenshot. We encode screenshots in the background and combine action execution and screenshot delivery in a single request. Screenshot capture and encoding. We use QEMU’s D-Bus display interface to keep the current screen in memory and encode updated frames in the background. We use JPEG at quality 95 with no chroma subsampling and retain the full 1920× 1080 resolution. Action execution and communication. We execute the action and return its screenshot in one request, reusing the network connection across requests. A lightweight command client reduces per-action startup overhead. We also place agent and environment sandboxes in the same Modal region to reduce network latency. Fast I/O returns the screenshot as soon as action execution and image preparation finish. It can therefore return a screenshot before the application has updated its interface, as illustrated in Figure 5. 16 Preprint B.4ACTION LATENCY: FAST I/O, GYM-ANYTHING, AND OSWORLD InfrastructureClickEscapeCtrl+UType 100 characters Gym-Anything2758.702865.002858.293521.72 OSWorld2515.412707.742708.412756.47 Fast I/O (ours)27.582.432.9114.98 Table 1: Fast I/O reduces action-to-observation latency to milliseconds. Median latency in milliseconds over 20 calls per action. For each system, the timer starts immediately before env.step(action)and stops when it returns the observation. All default action waits are included. Fast I/O reduces latency by one to three orders of magnitude. Table 1 compares Fast I/O with Gym-Anything (Aggarwal et al., 2026) and OSWorld (Xie et al., 2024) for mouse clicks, key presses, and text entry. Fast I/O takes 2.4–27.6 ms per call, compared to 2.5–3.5 s for the existing implementations. For example, a click takes 27.6 ms compared with 2.8 s with Gym-Anything, while typing 100 characters takes 15.0 ms compared with 3.5 s. Measurement setup. We time each runtime’s nativeenv.step(action)call, from issuing the action until the observation is returned. We test four actions on a1920× 1080Ubuntu desktop: clicking the Activities button, pressing Escape, pressing Ctrl+U, and typing 100 ASCII characters into a terminal. The caller and VM run in the same Modal sandbox. OSWorld uses its Docker provider with screenshot observations; both baselines use their default waits and action implementations. Table 1 reports the median of 20 calls per action, without profiling. Setup, pauses between calls, and saving returned screenshots occur outside the timed interval. B.5LATENCY BREAKDOWN Component (ms)ClickEscapeCtrl+UType 100 characters Gym-Anything Fixed waits2100.22100.22100.22100.2 Action request153.5153.7154.0805.5 Screenshot capture258.9354.0354.3354.1 Screenshot transfer238.1212.4220.6211.3 Cleanup and other53.153.153.153.1 Total2803.92873.42882.23524.2 Total minus 2 s803.9873.4882.21524.2 OSWorld Fixed wait2000.12000.12000.12000.1 Action request198.6191.7192.8245.1 Screenshot request324.7521.2521.8523.2 Other0.20.20.20.2 Total2523.62713.32714.92768.7 Total minus 2 s523.6713.3714.9768.7 Table 2: Fixed waits account for most baseline latency. Mean component times from 10 profiled calls per action for Gym-Anything and 20 for OSWorld. Components sum to the total before rounding. The final row for each system is the profiled total minus its two-second post-action wait. Where do the baselines spend time? Table 2 shows that fixed sleeps account for most of the baseline latency. Both implementations sleep for two seconds after an action, and Gym-Anything adds another 100 ms per step. Subtracting the two-second sleep leaves 804–1524 ms for Gym-Anything and 524–769 ms for OSWorld. Gym-Anything captures a PNG with FFmpeg, transfers it, and deletes the temporary file; typing also adds a 6 ms interval between characters. In OSWorld, most of the remaining time is spent in the action and screenshot requests, including processing inside the VM and communication. 17 Preprint Component (ms)ClickEscapeCtrl+UType 100 characters Input execution27.301.852.5518.39 Frame capture1.030.770.770.80 Foreground image preparation0.270.020.020.18 Other environment processing0.930.941.000.90 Transport4.674.994.764.62 Screenshot-file delivery0.430.450.430.44 Command overhead4.414.524.384.38 Total39.0613.5513.9029.71 Table 3: Fast I/O completes the four tested actions in under 50 ms. Mean times over 20 commands per action, from the agent issuing a command to receiving the screenshot file. This includes com- mand overhead, communication between sandboxes, and screenshot-file delivery in addition to the env.step call. Components sum to the total before rounding. Full command-to-screenshot latency.Table 3 measures latency from the agent issuing a command to receiving the screenshot file, including communication between sandboxes. We place both sandboxes in New York and test the four actions on two OSWorld desktops, with 0.5 s and 8 s gaps between requests. All 80 timed commands complete in under 50 ms. Median times are 38.5 ms for a click, 12.8 ms for Escape, 13.6 ms for Ctrl+U, and 29.5 ms for typing 100 characters. Communication contributes 4.6–5.0 ms on average. Image preparation after the request takes 0.02–0.27 ms; encoding runs in the background. Clicks take longer because input execution includes a 25 ms interval between pressing and releasing the mouse button. B.6FAST I/O CONSISTENCY ACROSS FIVE SEEDS RunScore (%) Time (s) Agent (s) Steps Env. (s) Waits (s) Env.−waits (s) 191.62103.9698.078.685.8915.5530.338 291.6293.9889.028.044.9554.6280.327 391.6296.4891.677.964.8124.4880.324 487.62100.2295.288.204.9344.6040.330 589.62100.4695.688.124.7854.4240.361 Mean90.4299.0293.948.205.0754.7390.336 SD1.793.873.580.280.4620.4620.015 Table 4: Fast I/O remains consistent across five evaluations. Each row averages all 50 OSWorld tasks for GPT-6 Astra at low reasoning effort. SD is the sample standard deviation across run-level means; score SD is in percentage points. Steps count action batches, and waits are explicitly requested by the agent. Environment processing time remains low across runs.Table 4 reports five evaluations of GPT-6 Astra at low effort on the same 50 OSWorld tasks with Fast I/O. We keep the agent prompt fixed and use fresh task seeds, a 500-step limit, and agent and environment sandboxes in New York. Environment time excluding agent-requested waits is0.336± 0.015s per task (mean±standard deviation across runs). The mean task time is 99.0± 3.9 s, and the score is 90.4± 1.8%. Most environment time is spent on agent-requested waits. Table 4 also separates the waits requested by the agent from other environment processing. These waits account for4.739s of the5.075s of mean environment time per task, leaving only0.336s for the remaining processing. Appendix D defines these timing components. CIMPLEMENTING AND EVALUATING AN AGENT To evaluate a new agent withcua-speedrun, users implement its model calls, prompts, and interaction history. The infrastructure handles task setup, desktop interaction, and verification through the common interface in Sec. 2.3. The same agent implementation and environment interface can then be used across benchmarks. 18 Preprint C.1AGENT TEMPLATES We provide templates for direct model-API agents, Codex, Claude Code, and open-weight agents served through vLLM. Each template contains two files:init.pyprepares dependencies and model serving, andagent.pyimplements the agent loop. Initialization finishes before task timing begins. Users can retaininit.pyand modify only the agent loop to test a new model, prompt, or interaction strategy. C.2MINIMAL AGENT EXAMPLE Listing 1 shows a minimal agent. The script receives the environment URL and task instruction as command-line arguments and uses theComputerclient to request screenshots, execute actions, and end the task. In each iteration,chooseactionsreceives the instruction, screenshot, and interaction history. This user-defined function calls the model and returns an action list, orNone when the agent considers the task complete. Users determine how to present this information to the model. Listing 1: Minimal agent loop. import os import sys from cua_speedrun.client import Computer def run(env_url, task): computer = Computer(env_url) history = [] limit = int(os.environ.get("CS_MAX_STEPS", "500")) for _ in range(limit): observation = computer.observe() actions = choose_actions(task, observation["png"], history) if actions is None: break computer.step(actions) history.append((observation["png"], actions)) computer.done() if __name__ == "__main__": run(sys.argv[1], sys.argv[2]) For example, an action list can contain"mouse": "leftclick": [640, 400]fol- lowed by"keyboard": "text": "hello". These two actions are executed in order within one step. Convenience methods such asclick(640, 400),typetext("hello"), and keys(["ctrl", "c"]) each submit a single action. The agent can execute several actions before requesting another screenshot. Callingdone()ends the interaction; the benchmark’s verifier determines the score. 19 Preprint C.3HOSTED AND LOCAL EVALUATION Listing 2 runs the Codex template on the representative OSWorld set using Modal. The command spec- ifies the submission directory, benchmark, and number of parallel evaluations. Omitting--remote runs both the agent and environment VMs on the evaluator host. For local desktop evaluation, this must be a Linux host with the VM runtime and hardware required by the benchmark, as well as model-serving resources when using open-weight models. Listing 2: Evaluating a supplied agent template. cua-speedrun run --remote \ --submission templates/codex_cli \ --benchmark benchmarks/osworld-energy50-representative \ --agents-per-evaluation 1 --parallel-evaluations 8 Users can also submit agents through the dashboard. For each evaluation, we record the benchmark, hardware, evaluation settings, and task seeds, along with task scores and timings. Action logs and screenshots allow users to inspect how the agent completed or failed each task. DINTERACTION AND TOKEN MEASUREMENTS Alongside performance, time, and cost, we measure how often an agent calls its model, how many actions it executes, and how many tokens it generates. These measurements support the analysis of agent speed in Sec. 3. The full configuration tables average over all tasks, including unsuccessful attempts. Aggregate metrics. For an agentπevaluated onKtasks, the mean verifier scoreP, mean task time T , and mean model cost C are P (π) = 1 K K X i=1 r i , T (π) = 1 K K X i=1 t i , C(π) = 1 K K X i=1 c i ,(4) wherer i is the verifier score defined in Appendix J.1, andt i andc i are the execution time and model cost of taski. The mean score retains partial credit; we also report the fraction of exactly completed tasks and the median time on those tasks. Agent time, environment time, and requested waits.We separate total task time into agent time and environment time. For taski, environment timee i sums the intervals recorded by the environment server for executing actions and returning observations. This includes explicit waits requested by the agent; we report their requested durations separately and subtract them when reporting environment time excluding agent waits. Agent time is the remainder,t i − e i , which includes model requests and agent-side processing. D.1MODEL CALLS, ACTION BATCHES, AND INDIVIDUAL ACTIONS Model responses. We count each completed model response, including responses without a com- puter action. A response can contain several tool calls, each of which may execute several actions. We therefore count model responses and desktop actions separately. Action batches and individual actions. An agent can execute several actions in one request. For example, clicking a field, typing text, and pressing Enter in one request counts as one batch and three actions. Issuing these actions separately counts as three batches and three actions. A step denotes one request containing keyboard, mouse, or wait actions. For taskiwithS i batches, letA ij be the action list in request j. The total number of individual actions is A i = P S i j=1 |A ij |. We count actions as submitted by the agent: typing a string, pressing a key combination, or requesting a wait each counts as one action. 20 Preprint D.2TOKEN ACCOUNTING We count every generated token once per response, including reasoning and other output, using the provider’s reported usage. We also report provider-supplied reasoning-token counts separately. To compare token rates, we divide total generated tokens by total agent time: P i G i / P i (t i − e i ) , whereG i is the generated-token count for taski. This effective token rate includes the time spent on model requests and agent-side processing. D.3FULL INTERACTION AND TOKEN COMPARISONS Tables 5 and 6 report mean batches, individual actions, model responses, and generated tokens for the configurations in Appendix E. Configuration IDs match the frontier plots and result tables. Table 5: Interaction and token counts on OSWorld. Means per task, including unsuccessful tasks. IDs match the OSWorld configuration tables in Appendix E. Tokens include reasoning and other output. Astra (low) uses five-seed means for each I/O setting. ID ConfigurationBatches Actions Responses Tokens 1 GPT-6 Astra / xhigh7.1419.90–2,060 2 Gemini 3.8 Flash / low16.6020.0218.521,564 3 Claude Opus 5 / high19.9226.9220.623,735 4 Gemini 3.8 Flash / medium27.1634.7029.105,365 5 GPT-6 Astra / medium6.3019.26–1,366 6 GPT-6 Astra / high6.5219.76–1,593 7 GPT-6 Astra / low / fast8.2027.10–1,328 8 Claude Sonnet 5 / high20.2429.2022.284,840 9 Gemini 3.7 Flash / medium19.4423.3421.363,594 10 Gemini 3.7 Flash / high24.2628.3026.205,731 11 Kimi K3 / max / batched11.7053.5811.68 10,435 12 Claude Opus 5 / low14.0423.8016.081,795 13 GPT-6 Astra / low5.8619.10–1,579 14 Gemini 3.7 Flash / low18.2420.8820.142,944 15 Gemini 3.8 Flash / high37.1851.4639.088,789 16 GPT-5.6 Luna / medium / API14.4248.8815.341,751 17 GPT-5.6 Sol / xhigh12.6636.7813.622,402 18 Kimi K3 / high / single8.92113.6010.707,567 19 GPT-5.6 Sol / medium12.7242.6413.682,717 20 GPT-5.6 Luna / high / Codex15.5839.80–8,637 21 Kimi K3 / high / batched8.7660.209.207,125 22 Muse Spark 1.1 / xhigh69.82145.5626.02 10,701 23 GPT-5.6 Luna / xhigh / Codex13.8833.10–8,160 24 Muse Spark 1.3 / xhigh56.5886.7230.62 11,505 25 Muse Spark 1.3 / medium41.5469.3226.108,952 26 Claude Sonnet 5 / low21.2429.5024.163,468 27 GPT-5.6 Luna / low / Codex7.2226.44–3,191 28 MiniMax M3 / thinking off34.3251.1035.14– 29 GPT-5.6 Luna / high / API18.2853.8819.243,358 30 Kimi K3 / low / batched5.7430.006.901,865 31 Muse Spark 1.3 / high51.3084.7828.98 10,538 32 Kimi K3 / low / single8.1255.109.623,171 33 MiniMax M3 / thinking on35.0055.3035.94– 34 Muse Spark 1.1 / medium54.36114.0022.827,352 35 Muse Spark 1.1 / high69.90126.8825.66 10,132 36 Muse Spark 1.1 / low58.48149.8622.126,526 37 Muse Spark 1.2 / xhigh66.54200.5243.08 17,198 38 GPT-5.6 Luna / medium / Codex9.5025.64–4,524 39 Muse Spark 1.1 / minimal53.44131.6820.625,957 40 Muse Spark 1.3 / low40.5071.8223.546,747 41 GPT-5.6 Sol / low11.2045.3412.161,236 42 Muse Spark 1.2 / low31.12137.8223.944,723 Continued on next page 21 Preprint Table 5: OSWorld interaction counts, continued. ID ConfigurationBatches Actions Responses Tokens 43 Muse Spark 1.2 / medium44.20152.8428.347,789 44 GPT-5.6 Luna / low / API10.1237.6611.061,063 45 Muse Spark 1.3 / minimal32.0459.5219.763,792 46 Muse Spark 1.2 / high45.72145.5833.749,551 47 Gemini 3 Flash Preview / high33.2834.5637.865,167 48 GLM-5V Turbo / thinking off16.8621.5217.548,150 49 Gemini 3 Flash Preview / medium28.2630.6832.524,125 50 GLM-5V Turbo / thinking on17.8619.2418.628,563 51 Muse Spark 1.2 / minimal23.2087.3818.722,379 52 Gemini 3 Flash Preview / low39.5240.2656.021,598 53 Yutori n2 / none18.7456.5420.903,652 54 Yutori n2 / low22.9658.7225.32 12,188 55 Yutori n2 / medium19.0655.1421.127,562 56 Yutori n2 / xhigh17.5648.1619.727,499 Table 6: Interaction and token counts on OSWorld2. Means per task, including unsuccessful tasks. IDs match the OSWorld2 configuration tables in Appendix E. Tokens include reasoning and other output. ID ConfigurationBatches Actions ResponsesTokens 1 GPT-6 Astra / xhigh52.81194.10–18,332 2 GPT-6 Astra / high51.29149.02–14,146 3 GPT-6 Astra / low42.58151.40–10,190 4 GPT-6 Astra / medium45.46151.08–11,382 5 Gemini 3.8 Flash / high216.54331.88218.4872,149 6 Gemini 3.8 Flash / medium167.79252.65169.6549,532 7 Claude Opus 5 / low146.52316.17158.2930,820 8 Claude Sonnet 5 / low237.25378.54253.5057,018 9 Gemini 3.8 Flash / low127.04200.75128.8319,840 10 Muse Spark 1.3 / minimal115.35488.0666.0216,150 11 Muse Spark 1.3 / low130.50655.0880.7325,131 12 Muse Spark 1.3 / medium149.81832.4489.7541,149 13 Muse Spark 1.3 / xhigh147.29832.7593.9652,628 14 Muse Spark 1.3 / high151.21754.9889.8344,241 15 Kimi K3 / low47.25313.7549.1721,852 16 MiniMax M3 / thinking on97.56407.3398.62– 17 MiniMax M3 / thinkingoff98.54368.6298.65– 18 GLM-5V Turbo / thinking off49.1579.3548.3828,558 19 GLM-5V Turbo / thinking on47.7591.7547.1228,385 20 Kimi K3 / high58.77350.0659.0481,494 21 Kimi K3 / max66.38386.5660.79 123,603 22 Preprint 5102050 Mean action batches per task 30 50 70 90 Mean score (%) 12 3 4 5 6 7 8 9 1011 12 13 1415 16 17 18 19 20 21 22 2324 25 26 2728 29 3031 32 33 34 35 3637 38 39 40 41 4243 44 45 46 47 48 49 50 51 52 53 54 55 56 (a) Action batches 2050100200 Mean individual actions per task 30 50 70 90 Mean score (%) 1 2 3 4 5 6 7 8 91011 12 13 1415 16 1718 19202122 2324 25 26 27 28 29 3031 32 33 34353637 3839 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 (b) Individual actions 1,0002,0005,00010,00020,000 Mean generated tokens per task 30 50 70 90 Mean score (%) 1 2 3 4 56 7 8 91011 12 13 1415 16 1718 19 20 21 22 2324 25 26 27 29 30 31 32 34 35 3637 3839 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 (c) Generated tokens Figure 7: Performance–interaction and performance–token frontiers on OSWorld. Mean score against mean action batches, individual actions, and generated tokens per task. Tokens include reasoning and other output. Panels (a)–(b) show all 56 configurations; panel (c) shows the 54 with token counts. Dashed lines connect the circled frontier points. IDs match the configuration tables in Appendix E; darker green indicates greater reasoning effort. Astra (low) uses five-seed means for each I/O setting. D.4PERFORMANCE–INTERACTION AND PERFORMANCE–TOKEN FRONTIERS Figures 7 and 8 compare performance with the number of action batches, individual actions, and generated tokens on OSWorld and OSWorld2. Circled points mark the Pareto frontier for each metric. We use the same trajectories as the time and cost comparisons and average over all tasks. 23 Preprint 50100200 Mean action batches per task 0 20 40 60 80 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 (a) Action batches 1002005001,000 Mean individual actions per task 0 20 40 60 80 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 (b) Individual actions 10,00020,00050,000100,000 Mean generated tokens per task 0 20 40 60 80 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 (c) Generated tokens Figure 8: Performance–interaction and performance–token frontiers on OSWorld2. Mean score against mean action batches, individual actions, and generated tokens per task. Panels (a)–(b) show all 21 configurations; panel (c) shows the 19 with token counts. IDs match the OSWorld2 configuration table in Appendix E. Other plotting conventions follow Figure 7. 24 Preprint EFULL EVALUATION RESULTS E.1MODEL, HARNESS, AND REASONING-EFFORT CONFIGURATIONS Tables 7–9 report mean score, time, cost, and token use for each model, harness, and reasoning setting. Astra (low) uses five-seed means for each I/O setting. The configuration IDs identify the corresponding points in the figures. We evaluate Luna through both the direct API and Codex, and Kimi with single-call and batched-call harnesses. We use the Yutori cookbook. ID ConfigurationScore (%) Time (s) Cost ($) Tokens 1 GPT-6 Astra / xhigh91.6126.80.7082,060 2 Gemini 3.8 Flash / low91.6127.10.1081,564 3 Claude Opus 5 / high91.6136.00.6613,735 4 Gemini 3.8 Flash / medium91.6250.10.2215,365 5 GPT-6 Astra / medium89.690.20.6001,366 6 GPT-6 Astra / high89.6100.80.6361,593 7 GPT-6 Astra / low / fast90.499.00.9081,328 8 Claude Sonnet 5 / high89.6140.80.3094,840 9 Gemini 3.7 Flash / medium89.6198.90.9443,594 10 Gemini 3.7 Flash / high89.6254.31.4125,731 11 Kimi K3 / max / batched89.6443.70.503 10,435 12 Claude Opus 5 / low87.685.90.3901,795 13 GPT-6 Astra / low90.889.50.5311,579 14 Gemini 3.7 Flash / low87.6207.30.9262,944 15 Gemini 3.8 Flash / high87.6347.70.3478,789 16 GPT-5.6 Luna / medium / API75.6118.90.0191,751 17 GPT-5.6 Sol / xhigh81.6108.30.5272,402 18 Kimi K3 / high / single81.6305.10.4147,567 19 GPT-5.6 Sol / medium79.6121.30.5392,717 20 GPT-5.6 Luna / high / Codex79.6324.80.0768,637 21 Kimi K3 / high / batched79.6358.10.3827,125 22 Muse Spark 1.1 / xhigh79.6417.21.685 10,701 23 GPT-5.6 Luna / xhigh / Codex77.6297.20.0638,160 24 Muse Spark 1.3 / xhigh77.6612.10.161 11,505 25 Muse Spark 1.3 / medium77.5397.60.1268,952 26 Claude Sonnet 5 / low75.8129.40.2943,468 Table 7: OSWorld configurations. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output. 25 Preprint ID ConfigurationScore (%) Time (s) Cost ($) Tokens 27 GPT-5.6 Luna / low / Codex75.6144.80.0253,191 28 MiniMax M3 / thinkingoff75.6253.8– 29 GPT-5.6 Luna / high / API81.6160.50.0293,358 30 Kimi K3 / low / batched73.6149.50.1741,865 31 Muse Spark 1.3 / high73.6506.90.148 10,538 32 Kimi K3 / low / single71.6176.60.2803,171 33 MiniMax M3 / thinking on69.6345.1– 34 Muse Spark 1.1 / medium69.6359.81.3797,352 35 Muse Spark 1.1 / high69.6368.51.706 10,132 36 Muse Spark 1.1 / low69.6370.41.3126,526 37 Muse Spark 1.2 / xhigh69.6720.90.244 17,198 38 GPT-5.6 Luna / medium / Codex67.6188.80.0354,524 39 Muse Spark 1.1 / minimal67.6318.51.1845,957 40 Muse Spark 1.3 / low67.6361.00.1076,747 41 GPT-5.6 Sol / low65.8100.10.3191,236 42 Muse Spark 1.2 / low63.6363.60.1054,723 43 Muse Spark 1.2 / medium63.6466.20.1367,789 44 GPT-5.6 Luna / low / API61.873.60.0111,063 45 Muse Spark 1.3 / minimal61.6264.40.0793,792 46 Muse Spark 1.2 / high61.6669.80.1749,551 47 Gemini 3 Flash Preview / high59.6337.20.8095,167 48 GLM-5V Turbo / thinking off57.8752.40.2208,150 49 Gemini 3 Flash Preview / medium57.6268.00.5924,125 50 GLM-5V Turbo / thinking on53.8359.20.2358,563 51 Muse Spark 1.2 / minimal53.6248.80.0702,379 52 Gemini 3 Flash Preview / low33.6492.21.4011,598 53 Yutori n2 / none75.6298.50.0813,652 54 Yutori n2 / low77.6331.00.157 12,188 55 Yutori n2 / medium67.6265.40.1107,562 56 Yutori n2 / xhigh79.6225.00.1047,499 Table 8: OSWorld configurations, continued. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output. ID ConfigurationScore (%) Time (s) Cost ($)Tokens 1 GPT-6 Astra / xhigh76.91237.38.17518,332 2 GPT-6 Astra / high75.0914.07.39114,146 3 GPT-6 Astra / low68.2904.26.33610,190 4 GPT-6 Astra / medium67.2828.16.51911,382 5 Gemini 3.8 Flash / high60.23058.85.01972,149 6 Gemini 3.8 Flash / medium59.92051.82.91549,532 7 Claude Opus 5 / low52.11385.710.00630,820 8 Claude Sonnet 5 / low39.11993.68.86357,018 9 Gemini 3.8 Flash / low35.31386.71.87519,840 10 Muse Spark 1.3 / minimal34.01516.00.39016,150 11 Muse Spark 1.3 / low33.72280.80.50325,131 12 Muse Spark 1.3 / medium32.32292.10.58841,149 13 Muse Spark 1.3 / xhigh32.22261.00.64052,628 14 Muse Spark 1.3 / high31.52179.30.59944,241 15 Kimi K3 / low19.21860.92.08821,852 16 MiniMax M3 / thinking on4.11673.2– 17 MiniMax M3 / thinkingoff3.21446.9– 18 GLM-5V Turbo / thinking off1.51979.40.73928,558 19 GLM-5V Turbo / thinkingon1.52641.90.72528,385 20 Kimi K3 / high22.63733.94.51781,494 21 Kimi K3 / max28.54825.15.625 123,603 Table 9: OSWorld2 configurations. Bold IDs mark the joint performance–time–cost frontier. Tokens include reasoning and other generated output. 26 Preprint 75150300600 Mean task time (s) 30 50 70 90 Mean score (%) 1 23 4 5 6 7 8 9 10 11 12 13 14 15 16 1718 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 3839 40 41 42 43 44 4546 47 48 49 50 51 52 53 54 55 56 0.010.030.10.31 Mean model cost ($/task) 30 50 70 90 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 29 30 31 32 34 35 36 37 38 39 40 41 42 43 44 4546 47 48 49 50 51 52 53 54 55 56 Figure 9: Full OSWorld performance–time and performance–cost comparisons. All 56 config- urations appear in the time comparison; the cost comparison uses configurations with reported or estimated costs. IDs identify the model, reasoning effort, and harness in Tables 7–8. Dashed lines connect nondominated configurations for each pair of axes. Scores, times, and costs are means across the same 50 tasks, including unsuccessful tasks. 2050100300800 Mean task time (s) 30 40 50 60 70 80 90 100 Mean score (%) 1 23 4 5 6 7 8 9 10 11 12 13 14 15 16 1718 19 2021 22 23 24 25 26 27 29 3031 32 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Measured time 2050100300800 Mean task time (s) 30 40 50 60 70 80 90 100 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 1415 16 17 18 19 20 21 22 23 24 25 26 27 29 30 31 32 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 Common rate: 100 tokens/s Figure 10: Full OSWorld comparison at a common token rate. Both plots use the same 54 configurations with token counts. The right plot replaces measured mean task time with mean recorded environment time plus generated tokens divided by 100 tokens/s. Tokens include reasoning and other output; scores and trajectories are unchanged. Point IDs refer to the OSWorld configuration tables. E.2OSWORLD: PERFORMANCE–TIME AND PERFORMANCE–COST FRONTIERS Figure 9 shows the full OSWorld comparison from Figure 2, with configuration IDs for all 56 points. A common token-generation rate.Figure 10 compares measured task time with an estimate that assigns every model the same generation rate. We compute this estimate as recorded environment time plus generated tokens divided by 100 tokens/s, using the same trajectories and scores. Opus 5 (low) scores 87.6% in 85.9 s, while Astra (low) scores 90.8% in 89.5 s. Opus generates 1,795 tokens per task compared with 1,579 for Astra. With generation time assigned at the common rate, Astra is faster. Environment time includes agent-requested waits (Appendix D). 27 Preprint 0100200300400500 Median task time (s) 30 40 50 60 70 80 90 100 Mean score (%) 123 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 4243 44 45 46 47 48 49 50 51 52 53 54 55 56 All configurations 50100150200 Median task time (s) 1 2 34 5 6 7 8 9 10 12 13 14 15 16 1718 19 202122 23 26 27 28 29 30 32 33 34 35 36 38 39 40 41 44 45 47 49 50 51 53 54 55 56 Fast-region detail Figure 11: Full OSWorld median-time view and fast-region detail. Mean verifier score versus median task time for all 56 configurations. For Astra (low), the median task times are averaged across five seeds. The right plot enlarges the region containing the fastest configurations; point IDs match the full OSWorld tables. Green dotted lines show the published human references of 72.36% and 111.94 s from the original 369-task OSWorld benchmark (Xie et al., 2024). 750150030005000 Mean task time (s) 0 20 40 60 80 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 0.31310 Mean model cost ($/task) 0 20 40 60 80 Mean score (%) 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 18 19 20 21 Figure 12: Full OSWorld2 performance–time and performance–cost comparisons. All 21 configurations are evaluated on the same 52 tasks. Point IDs refer to the OSWorld2 configuration table. Marker shapes follow Figure 2; darker green indicates greater reasoning effort. Median task time. Figure 11 reports median task time, which summarizes the time for a typical task. E.3OSWORLD2: PERFORMANCE–TIME AND PERFORMANCE–COST FRONTIERS Figure 12 compares all 21 OSWorld2 configurations. Astra defines the entire time frontier, while Gemini 3.8 Flash and Muse Spark 1.3 offer lower-cost choices on the cost frontier. 28 Preprint $0.3 $1 $3 $10 750 1,500 3,000 5,000 OSWorld2 0% 20% 40% 60% 80% Score (%) Cost / task Time / task (s) Figure 13: Joint performance–time–cost frontier on OSWorld2. Circled points are the evaluated frontier configurations. Other configurations appear in gray. Time and cost axes are logarithmic. Marker shapes follow Figure 2; darker green indicates greater reasoning effort. E.4OSWORLD2: JOINT PERFORMANCE–TIME–COST FRONTIER Figure 13 extends the joint frontier in Figure 6 to OSWorld2. A configuration is on this frontier if no other evaluated configuration improves one of the three metrics without worsening another. It includes Astra at all four reasoning settings, Gemini 3.8 Flash at all three settings, and Muse Spark 1.3 at minimal effort, allowing a choice based on the desired score and the available time and cost budgets. 29 Preprint 500550600650 Mean task time (s) 85 90 95 Mean score (%) low medium high xhigh (a) MyPCBench: time 4.24.44.64.85.0 Mean model cost ($/task) low medium high xhigh (b) MyPCBench: cost 750100012501500 Mean task time (s) 85 90 95 Mean score (%) low medium high xhigh (c) CUA-World: time 810121416 Mean model cost ($/task) low medium high xhigh (d) CUA-World: cost Figure 14: Performance–time and performance–cost frontiers on MyPCBench and CUA-World. GPT-6 Astra through Codex at four reasoning settings. Each point averages the same 38 MyPCBench tasks (top) or 26 CUA-World tasks (bottom). Darker green indicates greater reasoning effort. Dashed lines connect the observed Pareto-frontier points, which are circled. MyPCBench (38 tasks)CUA-World (26 tasks) EffortScore (%)Time (s)Cost ($)Score (%)Time (s)Cost ($) Low89.95144.5790.38079.29 Medium88.34954.3386.78108.42 High88.65684.8194.21,10811.11 Xhigh93.66194.8193.01,51515.06 Table 10: Reasoning effort on MyPCBench and CUA-World. GPT-6 Astra through Codex, reporting mean partial-credit score, time, and model cost per task. Bold indicates the highest score, lowest time, or lowest cost within each benchmark. E.5REASONING EFFORT ON MYPCBENCH Figure 14 and Table 10 compare GPT-6 Astra through Codex at four reasoning settings on MyPCBench. Xhigh achieves the highest score, 93.6%, while medium has the lowest mean time and cost, 495 s and $4.33 per task. We evaluate the 38-task representative set from Sec. 2.2, with a limit of 100 action batches. All reported means include unsuccessful tasks. E.6REASONING EFFORT ON CUA-WORLD Figure 14 and Table 10 show a different outcome on CUA-World. High effort scores 94.2%, compared with 93.0% at xhigh, while taking less time (1,108 vs. 1,515 s) and costing less ($11.11 vs. $15.06 per task). All four settings use Astra through Codex on the same 26 long-horizon tasks, with a limit of 500 action batches. 30 Preprint RunScore (%) Time (s) Agent (s) Steps Env. (s) Waits (s) Env.−waits (s) 187.6286.1172.905.9013.2091.32811.881 291.6289.5675.315.8214.2481.08213.166 391.6290.1376.525.8813.6061.04012.566 491.6289.8175.785.7814.0331.15212.881 591.6292.1078.195.9413.9141.01212.902 Mean90.8289.5475.745.8613.8021.12312.679 SD1.792.161.930.060.4050.1260.494 Table 11: Variation across five seeds for GPT-6 Astra at low reasoning effort. Each row averages all 50 OSWorld tasks. SD is the sample standard deviation of these means across the five runs; score SD is in percentage points. Steps count action batches, and waits are explicitly requested by the agent. FRUN-TO-RUN VARIATION We repeat the evaluation across five seeds to measure variation in scores, task times, and action sequences. We also compare scores on individual tasks to determine whether the same tasks succeed across runs. Appendix B.6 reports the corresponding timing results for Fast I/O. F.1FIVE-SEED EVALUATION SETUP We evaluate GPT-6 Astra at low reasoning effort using Codex on the same 50 OSWorld tasks. Each run uses the same agent prompt and a 500-step limit. Only the task seed changes. F.2SCORES AND TIMES ACROSS SEEDS Table 11 shows that both score and task time vary little across seeds. Mean task time is89.54± 2.16s, and the score is90.82± 1.79% (mean±standard deviation). The mean number of action batches is also stable at 5.86± 0.06 per task. Timing components are defined in Appendix D. F.3TASK-LEVEL SCORE AGREEMENT I/Ok Pass@k (%) Best-of-k partial score (%) Standard I/O 187.2090.82 288.0091.62 388.0091.62 488.0091.62 588.0091.62 Fast I/O186.8090.42 288.8092.42 389.2092.82 489.6093.22 590.0093.62 Table 12: Task-level repeatability across five seeds of Astra at low effort. Values average uniformly over all size-ksubsets of the five runs, then over the 50 tasks. Pass@kmeasures whether any of thek attempts succeeds; best-of-k uses the highest partial score. Table 12 shows that additional attempts improve performance only slightly. With standard I/O, 42 tasks receive full credit in all five runs, two in four runs, and six in none. Partial-credit scores are identical across all five runs on 48 of the 50 tasks. The low variation in mean score therefore reflects consistent performance on individual tasks: almost every task receives the same score on every attempt. With Fast I/O, 40 tasks receive full credit in all five runs, four in four runs, one in one run, and five in none. Partial-credit scores are identical on 45 tasks. Pass@kand partial credit.For a task withcfully successful runs among the five, the probability that a uniformly selected set ofkruns contains a success is1− 5−c k / 5 k . Pass@kaverages this quantity over tasks. For partial credit, the table averages the maximum task score within each size-k set. Standard-I/O pass@1 is 87.2% and pass@5 is 88.0%; its best-of-five partial score is 91.62%. 31 Preprint F.4VARIATION IN ACTION SEQUENCES We next compare the number of action batches used for the same task across seeds. For each task, we compute the standard deviation and range across its five trajectories. With standard I/O, the median of these task-level standard deviations is 0.55 batches, and the median range is one batch. For Fast I/O, the corresponding values are 0.99 and two batches. Two ways to complete the same spreadsheet task. For example, on the Calc task that cleans movie titles, all five standard-I/O runs receive full credit while taking 7, 6, 5, 6, and 2 batches. The seven-batch trajectory enters separatePROPER(TRIM(...))formulas for successive rows and revisits the paste-special dialog. The two-batch trajectory selectsC2:C29, enters one formula with Alt+Enter to fill the selection, then saves. The two trajectories take 97.4 and 40.3 s, respectively. Here, the agent reduces the number of batches by applying the formula to all rows at once. 32 Preprint GREPRESENTATIVE TASK SELECTION We select a small set of tasks from each benchmark to reduce evaluation time while preserving the scores and relative performance of agents. This section gives the selection procedure from Sec. 2.2 and tests its score estimates, rankings, and speed–performance frontier on held-out models. Table 13 lists the resulting task counts. BenchmarkFull setSelected subset OSWorld (Xie et al., 2024)29550 OSWorld2 (Yuan et al., 2026)6352 CUA-World (Aggarwal et al., 2026)14326 MyPCBench (Jang et al., 2026a)18438 Table 13: Task counts for the representative subsets selected for cua-speedrun. G.1MODELS AND TASKS USED FOR SELECTION OSWorld. We use nine configurations evaluated on the same 295 tasks: Claude Sonnet 5 xhigh, Gemini 3.6 Flash, Gemini 3 Flash Preview, GLM-5V Turbo, GPT-5.6 Luna xhigh, Kimi K3, Meta Muse Spark, MiniMax M3, and Qwen3.5-9B-Thinking. Each task is represented by its nine partial scores and nine exact-completion indicators. The internet-dependence audit defining this task population is described in Appendix H. OSWorld2. Selection uses seven configurations: MiniMax-M3, Claude Opus 4.7, Claude Sonnet 4.6 at medium and maximum effort, GPT-5.5, GPT-5.6, and Qwen3.7. The resulting evaluation set contains 52 tasks. CUA-World.Selection uses seven configurations with scores on more than 30 tasks each: Qwen3- VL-2B-Thinking, GPT-5.4 through Azure, GPT-5.4, Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3 Flash Preview, and Kimi K2.5. Selection uses each model’s distribution of observed partial scores. Repeated runs remain separate observations. The selected set contains 26 tasks. MyPCBench. The evaluation uses 38 of the 184 canonical tasks, selected with the unweighted energy method and 100 initializations. The task identities for all four benchmarks are listed in Appendix G.6. G.2INITIALIZATION, OPTIMIZATION, AND TASK COUNT We seek a subset that represents the full benchmark’s distribution of score patterns. For OSWorld and OSWorld2, letz i concatenate the partial scores and exact-completion indicators for taski. Distances are d ij =∥z i − z j ∥ 2 . For a subset S of size K from N tasks, the normalized energy objective is E(S) = 1 ̄ d   2 KN X i∈S N X j=1 d ij − 1 K 2 X i,j∈S d ij − 1 N 2 N X i,j=1 d ij   , ̄ d = P i̸=j d ij N (N − 1) .(5) We estimate a model’s benchmark score by its unweighted mean score on the selected tasks. Balanced initializations. To cover a range of score profiles in each initial subset, we sort tasks by the leading principal coordinate of their centered score profiles and choose evenly spaced tasks from this ordering. Specifically, for a uniformly sampled phaseφ ∈ [0, 1), the initial positions are⌊(φ + t)N/K⌋fort = 0,...,K − 1. We use 100 initializations with seeds7000,..., 7099, optimizing each distinct initial subset once. One-swap optimization.At each iteration, we evaluate every exchange of a selected task with an unselected task and apply the exchange that reduces energy the most. We stop when no exchange improves the objective beyond numerical tolerance and retain the lowest-energy subset across the initializations. 33 Preprint Listing 3: Representative-task selection and validation. for each candidate task count K: for each model h: remove model h from the calibration models form task profiles from the remaining models optimize 100 balanced initial subsets by one-task swaps select the subset with the lowest calibration energy predict h’s partial and exact scores by subset means compare predictions with each model’s full-set means choose the smallest K for which both rank correlations are at least 0.95 at K-1, K, and K+1 refit the K-task subset using all models To choose the number of tasks, we repeat the full selection procedure with each model held out in turn (Listing 3). We construct task profiles and select tasks using the remaining models, then compare the held-out model’s subset score with its full-set score. We choose the smallestKfor which both partial-score and exact-completion rank correlations are at least 0.95 atK− 1,K, andK + 1. Checking the neighboring task counts tests the stability of the ranking criterion. Each task count is optimized independently. After choosing K, we select the final evaluation set using all models. Selection with missing scores. Some CUA-World models have scores on only part of the bench- mark. We therefore compare each model’s distribution of observed partial scores on selected tasks with its distribution over all observed tasks, using one-dimensional energy distance. We first ensure that the subset contains an observation for every model used for selection, then minimize the mean of these distances. The task-count rule uses partial-score rank correlation atK− 1,K,K + 1. At K = 25, 26, 27, the seven-model held-out correlations are 0.964, 1.000, and 1.000, respectively; the corresponding score errors are 7.43, 5.12, and 2.10 percentage points. These errors compare unweighted means of observed run scores. 34 Preprint 20406080100 Selected tasks 2 4 6 8 10 Mean absolute error (p) (a) OSWorld score error Stratified random Energy selection 20406080100 Selected tasks 25 50 75 100 125 150 Mean absolute error (s) (b) OSWorld time error 20406080100 Selected tasks 0.0 0.2 0.4 0.6 0.8 1.0 Spearman correlation (c) OSWorld ranking Partial score Exact completion 102030405060 Selected tasks 0.0 0.2 0.4 0.6 0.8 1.0 Spearman correlation (d) OSWorld2 ranking Figure 15: Score error and rank correlation on held-out models. OSWorld uses nine leave-one- agent-out folds; OSWorld2 uses seven. The randomized baseline averages 300 difficulty-stratified selections per task count. Dotted vertical lines mark 50 and 52 tasks; horizontal lines mark rank correlation 0.95. Each task count is optimized independently. G.3HELD-OUT SCORE AND RANKING ACCURACY Figure 15 compares energy-based selection with a randomized baseline that samples tasks across difficulty levels. We measure error as the mean absolute difference between each held-out model’s subset estimate and its full-set result. At 50 OSWorld tasks, energy-based selection reduces partial- score error from 4.02 to 2.16 percentage points. It also reduces mean-task-time error from 51.0 to 31.4 s, even though selection uses only scores. Thus, selecting tasks with representative score patterns also improves estimates of execution time. Benchmark K Partial ρ Exact ρ Partial MAE Exact MAE OSWorld490.9670.9674.364.57 500.9830.9752.162.17 511.0000.9924.213.95 OSWorld2510.9640.9551.702.28 520.9640.9551.332.03 530.9640.9820.741.04 Table 14: Held-out-model validation around the selected task count. MAE is in percentage points. Each model is excluded from task selection in its fold. Both correlations exceed 0.95 at the selected task count and its two neighbors. Table 14 shows that the selected task sets closely preserve model rankings on both benchmarks. At 50 tasks, OSWorld achieves partial-score and exact-completion rank correlations of 0.983 and 0.975; at 52 tasks, OSWorld2 achieves 0.964 and 0.955. Both correlations remain above 0.95 at the neighboring task counts. 35 Preprint 2004006008001000 Mean task time (s) 40 50 60 70 80 Mean score (%) 1 2 3 4 5 6 7 8 9 (a) Full 295-task scores 2004006008001000 Mean task time (s) 1 2 3 4 5 6 7 8 9 (b) Held-out 50-task estimates Figure 16: Speed–performance frontiers from full-set scores and held-out subset estimates. Dark points and connecting lines mark nondominated configurations. Model IDs are: 1, Sonnet 5; 2, Gemini 3.6 Flash; 3, Gemini 3 Flash Preview; 4, GLM-5V Turbo; 5, Luna; 6, Kimi K3; 7, Muse Spark; 8, MiniMax M3; 9, Qwen3.5-9B. Each point in (b) uses a 50-task subset selected with that model held out. G.4PRESERVING THE SPEED–PERFORMANCE FRONTIER Figure 16 compares the full-set frontier with the frontier estimated from 50-task subsets for held-out models. The estimated frontier recovers two of the three full-set frontier configurations and adds no others, giving 100% precision and 66.7% recall. Pairwise dominance agrees for 91.7% of model pairs. This analysis uses the models’ scores and times; selection itself uses only scores. Evaluation savings.Selecting 50 of the 295 OSWorld tasks reduces the number of tasks by5.9×. Across the nine models, these tasks account for 15.1% of the total recorded task time, a6.61× reduction. We compute this saving by summing trajectory times, excluding setup and verification. 36 Preprint MethodPartial MAEExact MAEMean MAE Uniform random5.886.015.95 Difficulty-stratified random5.055.115.08 IRT-inspired selection7.948.218.08 Energy-based selection4.504.344.42 Table 15: Task-selection methods on OSWorld with 32 tasks. Errors are averaged across nine leave-one-agent-out folds and reported in percentage points. Mean MAE averages partial-score and exact-completion MAE. Random baselines average 300 selections per fold. G.5COMPARING TASK-SELECTION METHODS Table 15 compares task-selection methods at a budget of 32 OSWorld tasks, using the same nine leave-one-agent-out folds. Energy-based selection achieves a mean error of 4.42 percentage points, compared to 5.08 for difficulty-stratified random selection and 8.08 for the IRT-inspired method. We use energy-based selection for its simple scoring rule: each agent’s benchmark score is the average of its scores on the selected tasks. The IRT-inspired method fits a one-parameter ability model using the eight agents available in each fold and predicts full-benchmark performance from the selected-task scores. G.6SELECTED TASK IDENTITIES The following lists give the final task sets used in our evaluations. OSWorld IDs use the original task UUIDs. OSWorld2 uses its three-digit identifiers; MyPCBench IDs combine the task family and suffix; CUA-World IDs combine the environment and task name. G.6.1OSWORLD Table 16: Selected OSWorld task identities. ApplicationTask UUID chrome 06fe7178-4491-4589-810f-2e2bc9502122 chrome 2ae9ba84-3a0d-4d4c-8338-3a1478dc5fe3 chrome 3720f614-37fd-4d04-8a6b-76f54f8c222d chrome 44e5668-ecd5-4366-a6ce-c1c9b8d4e938 chrome af630914-714e-4a24-a7b-f9af687d3b91 gimp 045bf3f-9077-4b86-b483-a1040a949cff gimp 554785e9-4523-4e7a-b8e1-8016f565f56a gimp 734d6579-c07d-47a8-9ae2-13339795476b gimp a746add2-cab0-4740-ac36-c3769d9bfb46 libreoffice calc 0326d92d-d218-48a8-9ca1-981cd6d064c7 libreoffice calc 1de60575-b6e-4c3d-9e6a-2fa699f9f197 libreoffice calc 1e8df695-bd1b-45b3-b557-e7d599cf7597 libreofficecalc 3a7c8185-25c1-4941-bd7b-96e823c9f21f libreofficecalc 4de54231-e4b5-49e3-b2ba-61a0bec721c0 libreofficecalc 535364ea-05bd-46ea-9937-9f55c68507e8 libreoffice calc 7efeb4b1-3d19-4762-b163-63328d66303b libreofficecalc 8b1ce5f2-59d2-4dcc-b0b0-666a714b9a14 libreofficecalc a9f325a-8c05-4e4f-8341-9e4358565f4f libreoffice calc abed40dc-063f-4598-8ba5-9fe749c0615d libreofficeimpress 39be0d19-634d-4475-8768-09c130f5425d libreofficeimpress 455d3c66-7dc6-4537-a39a-36d3e9119df7 libreofficeimpress 986fc832-6af2-417c-8845-9272b3a1528b libreoffice impress a097acff-6266-4291-9fbd-137af7ecd439 libreofficeimpress a434992a-89df-4577-925c-0c58b747f0f4 libreofficeimpress af2d657a-e6b3-4c6a-9f67-9e3ed015974c libreofficeimpress b8adbc24-cef2-4b15-99d5-ecbe7f445eb libreofficewriter 0810415c-bde4-4443-9047-d5f70165a697 libreoffice writer 0e47de2a-32e0-456c-a366-8c607ef7a9d2 libreofficewriter 72b810ef-4156-4d09-8f08-a0cf57e7cefe libreofficewriter adf5e2c3-64c7-4644-b7b6-d2f0167927e7 libreofficewriter ecc2413d-8a48-416e-a3a2-d30106ca36cb 37 Preprint ApplicationTask UUID multiapps 42f4d1c7-4521-4161-b646-0a8934e36081 multiapps 48d05431-6cd5-4e76-82eb-12b60d823f7d multi apps 6d72aad6-187a-4392-a4c4-ed87269c51cf multiapps 81c425f5-78f3-4771-afd6-3d2973825947 multiapps d9b7c649-c975-4f53-88f5-940b29c47247 multiapps e135df7c-7687-4ac0-a5f0-76b74438b53e multiapps f5c13cdd-205c-4719-a562-348ae5cd1d91 os b6781586-6346-41cd-935a-a6b1487918fc thunderbird 7b1e1f9-b85-49be-b01d-d6424be18cd0 thunderbird 9b7bc335-06b5-4cd3-9119-1a649c478509 thunderbird a1af9f1c-50d5-4bc3-a51e-4d9b425f638 thunderbird dfac9e8-9bc4-4cdc-b465-4a4bfcd2f397 thunderbird f201fbc3-44e6-46fc-bcaa-432f9815454c vlc cb130f0d-d36f-4302-9838-b3baf46139b6 vlc d06f0d4d-2cd5-4ede-8de9-598629438c6e vlc fba2c100-79e8-42df-ae74-b592418d54f4 vs code 0ed39f63-6049-43d4-ba4d-5fa2fe04a951 vscode 5e2d93d8-8ad0-4435-b150-1692aacaa994 vscode ec71221e-ac43-46f9-89b8-e7d80f7e1c5 G.6.2OSWORLD2 The selected task IDs are001,002,010,012,013,015,018,020,021,022,028,029,030, 033,038,040,042,043,044,046,047,049,051,053,054,057,058,059,061,063, 065,066,068,070,071,072,076,080,085,086,088,091,094,096,100,101,102, 103, 104, 106, 107, 108. G.6.3MYPCBENCH Task familySelected IDs aggregationf002, f005, f029, f033 contradictionf015, f023 counterfactualf001, f004, f008 cua onlyf004, f006, f018, f021, f023 hardappf011, f019, f022 longhorizonf028, f043, f045, f047, f050, f060, f074 preference inferencef004, f014, f021, f023, f025 retrievalf005, f029 situated actionf001, f003, f006, f017, f026, f035, f039 G.6.4CUA-WORLD Table 17: Selected CUA-World task identities. EnvironmentTask name ardourenv broadcastpodcaststemdelivery dhis2env rmncahscorecarddashboard dockerdesktopenv diagnosebrokenmicroservicesstack gpredictenv poesdownlinkschedulesetup gvsig desktopenv vulnerabilitymapremotecommunities jstockenv quarterlyportfoliorebalance librehealthehrenv implementlabworkflowandprocesspatient moodleenv configuretieredassessmentpathway nextgenconnectintegration engineenv adtcensuslabvalidationpipeline noshenv carequalityremediation odoo inventoryenv pharmalotrecallquarantine openclinic gaenv insuredconsultationbilling oracledatabaseenv claimspipelinereconciliation projectlibreenv schedulerecoveryrebaseline 38 Preprint EnvironmentTask name pymolenv kinaseselectivitycomparison redmineenv q1milestonereconciliation rocket chatenv complianceauditremediation snapenv multicriteriasuitabilitymapping splunkenv threatintelenrichmentpipeline sumoenv optimizenetworksignaltiming thunderbirdenv litigationemailtriage wireshark env webappbreachinvestigation wondershareedrawmaxenv healthcareitarchitecturereview woocommerceenv launchcoffeeproductline wordpressenv launchwoocommercecoffeeroastery wps presentationenv rebrandrestructurepitchdeck HINTERNET-DEPENDENCE AUDIT Live public services can change their content, become unavailable, or restrict access between evalua- tions. We therefore audit whether each task requires the agent to use such services. Tasks that use a local application or a benchmark-controlled website remain eligible, as do tasks that only require downloading fixed files during setup. H.1INCLUSION RULE We first use a coding agent to classify the 369 OSWorld tasks as local/offline, benign internet, or anti- bot/access-sensitive. Three human reviewers then independently assess whether each task requires interaction with a live public service. We include a task only when all three reviewers agree to include it. This gives 295 tasks. Reviewer or outcomeIncludedExcluded Reviewer 129772 Reviewer 230267 Reviewer 329574 Unanimous decision29567 Disagreement7 Table 18: Independent inclusion decisions for 369 OSWorld tasks. The seven disagreements are excluded by the unanimous-inclusion rule. H.2REVIEWER AGREEMENT Table 18 shows that 362 of 369 tasks receive a unanimous decision. Pairwise agreement is 98.64%, 99.46%, and 98.10% for reviewer pairs 1–2, 1–3, and 2–3, respectively. Their Cohen’sκvalues are 0.956, 0.983, and 0.939; Fleiss’ κ across all three reviewers is 0.959. Applying the criterion across benchmarks.We apply the same criterion to the other benchmarks. OSWorld2 provides benchmark-controlled web applications, and MyPCBench provides local, pre- authenticated services, so these tasks remain eligible. For CUA-World, we exclude tasks requiring internet access during agent interaction while retaining those that only use the network for setup. 39 Preprint IVALIDATING ACTIONS WITH CUA-AUTODEBUG A task can fail because the model chooses an incorrect action or because the infrastructure executes a correct action incorrectly. We developed CUA-AutoDebug to distinguish these errors. A controlled desktop application records the keyboard and mouse inputs it receives, including modifier keys and typed text. We also check the resulting application state: a click should reach the intended target, a drag should move the object, and typed text should match the requested string. For shortcuts handled by the operating system, we record input events before they reach the application. I.1SEPARATING MODEL, HARNESS, AND RUNTIME ERRORS We first submit predefined actions directly to the VM runtime. We then test agent harnesses by giving the model a short instruction, such as clicking a colored square, and comparing its response with the executed action and received input. An incorrect model action is a model error; incorrect translation of a correct action is a harness error. I.2INPUT FAILURES DETECTED BY THE RUNTIME TESTS Requested inputReceived inputCause Type < >Incorrect X11 key/modifier map- ping Type accented textAccented characters omitted Characters absent from the key map Type a literal shell vari- able Variable expanded into a pathText interpreted by the command shell Keypad Enter or MenuNo corresponding key eventMissing key-name translation Toggle Caps Lock, then type Lowercase textCaps Lock press silently omitted Table 19: Input errors detected by CUA-AutoDebug when executing actions through SSH and PyAutoGUI. The tests compare requested input with the keyboard and mouse events received. Table 19 shows examples of input errors detected by these tests, such as typing>when<was requested or silently dropping accented characters. We test a catalog of 100 cases covering clicks, drags, scrolls, key presses, key combinations, text entry, and sequences of these actions. Six cases are unsupported by the tested action interface. We repeat each of the remaining 94 cases five times; 83 pass and 11 fail when executed through SSH and PyAutoGUI. The failures include one shell-expansion case, one shifted-symbol case, five Unicode cases, and four named-key cases. The corrected runtime passes all 470 tests. We correct these errors in CUA-Speedrun’s QEMU runtime by preserving literal text when sending commands, explicitly mapping key names and modi- fiers, and supporting characters absent from the default keyboard mapping. After these corrections, all 94 supported cases pass in all five repetitions. I.3FAILURES IN AGENT HARNESSES HarnessModel responseExecuted behavior GeminiScroll down by five wheel clicksScroll up by 600 ticks GeminiPress F5 or Page DownType the key name as text Qwen3.5Middle-click the targetNo action Qwen3.5Ctrl-click the targetRelease Ctrl before clicking Qwen3.5Move, then scroll or dragExecute only the first tool call Table 20: Harness errors can change a correct model action. Examples detected by comparing model responses with the actions executed and the input events received. Table 20 shows that harness errors can prevent a correct model response from reaching the application. For example, the Gemini harness reversed scroll direction and interpreted a fallback scroll distance in pixels as a number of wheel ticks. All six scroll-direction test repetitions failed before correction and passed afterward. The Qwen3.5 harness released the modifier key before a click and silently 40 Preprint discarded tool calls after the first. A model that described a triple-click but emitted a double-click produced a model error. JEXPERIMENTAL DETAILS Appendix E.1 lists the model, harness, and reasoning setting for each configuration. J.1TASK DEFINITION We define a computer-use task asx i = (E i ,s 0 i ,p i ,V i ) , whereE i is an interactive environment with initial states 0 i ,p i is a natural-language instruction, andV i is a verification function. An agent πreceivesp i and a sequence of observations and produces computer actions until it terminates or reaches the task limit. This interaction produces a trajectoryτ i and final states T i , with score r i = V i (τ i ,s T i ). A benchmark B =x i N i=1 is a collection of such tasks. Agent implementations.We connect each model’s public reference agent or native computer-use API to the common desktop interface. J.2OBSERVATIONS, ACTIONS, AND INTERACTION HISTORY The environment provides1920× 1080screenshots of the desktop. Mouse actions use pixel coordi- nates in these images. Harnesses that resize screenshots or use normalized coordinates convert the model’s coordinates to desktop pixels before executing an action. Action familyOperations Mouse clicksLeft, right, middle, double, and triple click at a coordinate Pointer motionMove, drag along coordinates, hold or release a mouse button ScrollingSigned vertical scroll at the current pointer position KeyboardType text, press a key or chord, hold and release modifiers WaitingWait for an agent-specified duration ObservationRequest a screenshot without changing the desktop CompletionStop interaction and invoke the benchmark verifier Table 21: Desktop actions supported by the infrastructure. A batch contains one or more actions executed in order. The model-facing tool schema can differ between harnesses. Table 21 lists the actions available to agents. Direct-API agents translate the model’s computer-use responses into these actions and return screenshots in the provider’s tool-result format. Codex uses the same actions from a separate agent sandbox. We instruct Codex to interact with the task desktop only through the supplied interface: its local shell and files belong to the agent sandbox (Listing 4). Listing 4: Core desktop-isolation instruction in the Codex prompt. You are operating a remote computer in a separate VM. Your only interface to that computer is the HTTP proxy at $CS_COMPUTER_URL. Authenticate every request with the header X-Gateway-Token: $CS_COMPUTER_TOKEN. Your local shell and files are in the agent sandbox, not the task VM. Do not bypass the proxy to access the task computer. Interaction history.Each harness determines which screenshots and previous responses remain in the model’s context. Kimi’s harness retains the current screenshot and the two most recent completed- turn screenshots. Earlier turns are compacted into textual history; recent responses retain their reasoning fields and tool calls. Codex maintains its own interaction history and explicitly retrieves screenshots. 41 Preprint Reasoning settings.We vary reasoning effort within each model using the settings exposed by its provider, such as low and high. J.3TASK LIMITS AND EXECUTION Each evaluation fixes the task set, agent implementation, reasoning setting, environment resources, and number of parallel tasks. Every task starts from its benchmark-specified initial state. We measure task time from when the instruction is given to the agent until it terminates or reaches the task limit, excluding setup, initialization, and verification (Sec. 2.1). Time and action limits. The OSWorld and OSWorld2 configurations in Appendix E.1 have an environment deadline of 39,600 s per task, while the MyPCBench Astra sweep uses 7,200 s. These deadlines apply separately from the harness’s action or model-turn limit and the timeout on an individual model request. We allow 500 action batches per task in the Astra batching comparison, five-seed experiments, and CUA-World sweep, and 100 in the MyPCBench sweep. A batch can contain several actions before the next observation; the single-action Astra ablation requires a screenshot after every action (Appendix K.1). In the Fast I/O comparison, we keep the model and reasoning setting fixed and change the action and screenshot implementation (Appendix B.3). J.4BENCHMARK VERIFICATION We use each benchmark’s verification procedure. OSWorld and OSWorld2 score task completion with task-specific programmatic checks. CUA-World uses Gemini 3 Flash Preview to assess the recorded trajectory against its visual checklist. MyPCBench uses its visual trajectory judge and averages the rubric scores with equal weight; a task receives full credit only when every rubric passes. We also retain each benchmark’s task instructions and context, including MyPCBench’s persona and pre-authenticated local applications, alongside the agent’s system prompt. J.5MODEL COST We compute model cost by summing the cost of each task’s model requests. Each request’s cost comes from the provider’s recorded charge or its recorded token usage at the applicable API prices. Generated tokens follow the accounting in Appendix D.2. Model cost excludes environment hosting, model-server rental, setup, and separate verification calls. KADDITIONAL AGENT ANALYSES We examine how action batching, agent harnesses, and reasoning effort affect task time through comparisons within the same model. Action batches, individual actions, and generated tokens are defined in Appendix D. 42 Preprint K.1CONTROLLED BATCHING COMPARISONS ModelEffort Interaction ruleScoreTime Batches Actions Tokens Astraxhigh Batched91.62 126.827.1419.902,060 Astraxhigh One action + image 91.62 182.7316.5016.502,559 Kimi K3 lowSingle tool71.62 176.618.1255.103,171 Kimi K3 lowBatched tools73.62 149.505.7430.001,865 Kimi K3 highSingle tool81.62 305.078.92113.607,567 Kimi K3 highBatched tools79.62 358.138.7660.207,125 Kimi K3 maxSingle tool85.62 606.2712.1861.24 14,359 Kimi K3 maxBatched tools89.62 443.7411.7053.58 10,435 Table 22: Batching comparisons within the same model and reasoning effort on OSWorld. Score is a percentage; time is in seconds. All counts and times are means over the same 50 tasks. Tokens include reasoning and other output. Batching makes Astra faster at the same score.Table 22 compares Astra xhigh with its standard batched actions and a restriction to one action between screenshots. Both settings score 91.62%, but the single-action setting takes 182.7 s per task compared with 126.8 s for batching. It executes fewer individual actions (16.50 versus 19.90), yet requires more action batches (16.50 versus 7.14) and generates more tokens (2,559 versus 2,060). We keep the model, reasoning effort, 50-task set, and configured limits fixed. To enforce the single-action restriction, we reject multi-action requests and prevent further actions until the agent retrieves a new screenshot, including actions issued in separate back-to-back requests. Kimi benefits from batched tools at some reasoning settings. Table 22 also compares Kimi at matched low, high, and maximum effort. Kimi’s single-tool setting disables parallel tool calls and allows multiple GUI actions within one code block. At maximum effort, batched tools reduce mean task time from 606.3 to 443.7 s and improve the score from 85.62% to 89.62%. At high effort, batching increases time from 305.1 to 358.1 s while reducing the score from 81.62% to 79.62%. 43 Preprint 50100150200250300350 Mean task time (s) 60 70 80 Mean score (%) Low Med High Low Med High GPT-5.6 Luna Direct API Codex Figure 17: Harness performance depends on reasoning effort. GPT-5.6 Luna at low, medium, and high reasoning effort, using the direct-API agent or Codex on OSWorld. Each curve connects reasoning settings within one harness. All six evaluations use the same 50-task set. K.2HARNESS COMPARISONS ACROSS REASONING EFFORTS Figure 17 compares GPT-5.6 Luna with Codex and the direct-API agent at low, medium, and high effort. At low effort, Codex improves the score from 61.8% to 75.6% but increases mean task time from 74 to 145 s. At medium and high effort, the direct-API agent is both more accurate and faster: it scores 75.6% in 119 s compared with 67.6% in 189 s at medium, and 81.6% in 161 s compared with 79.6% in 325 s at high. 44 Preprint K.3COMPARING REASONING EFFORTS ON THE SAME SUCCESSFUL TASKS ModelTasks EffortTime Batches Actions Tokens Tok/s Gemini 3.8 Flash5 low1291.0116.40171.60 29,596 47.57 medium 1363.9112.60158.40 34,446 43.10 high3024.5190.60246.60 73,578 42.18 GPT-6 Astra13 low791.847.08146.697,878 15.18 medium752.647.15138.858,376 17.07 high817.353.15171.23 11,001 21.31 xhigh1131.655.38223.46 14,359 18.04 Muse Spark 1.32 minimal833.182.00195.007,521 15.04 low1183.399.50291.00 12,997 16.79 medium 1008.878.50235.00 14,621 21.55 high907.368.50246.00 14,095 22.87 xhigh1546.2109.50400.00 23,362 21.77 Table 23: Reasoning effort on tasks completed fully at every compared effort on OSWorld2. Every reported score is 100%. Time is in seconds per task; token rate divides total generated tokens by total agent time. Each model uses its own common task set. Higher effort can generate longer outputs and more actions on the same successful tasks. Table 23 restricts each OSWorld2 comparison to tasks that receive full credit at every evaluated effort for that model. On the five tasks that Gemini 3.8 Flash completes at all three efforts, high effort takes 3,024 s per task compared with 1,291 s at low effort. It generates more than twice as many tokens (73,578 versus 29,596) and executes more actions and batches. In comparison, its effective token rate falls from 47.57 to 42.18 tokens/s. We observe a similar increase at Astra’s highest effort. On the 13 tasks it completes at all four settings, xhigh takes 1,132 s, compared with 792 s at low and 753 s at medium. It generates 14,359 tokens and executes 223 actions, compared with 7,878 tokens and 147 actions at low effort. Muse Spark 1.3 completes two tasks at all five efforts, while Kimi K3 has no task completed at all three efforts. 45 Preprint 51015202530 Method submission 0 1 2 3 4 5 Mean absolute error (p) Visible validationHidden model Figure 18: Validation gains do not generalize to the held-out model. Across 34 candidate methods, the coding agent receives only validation errors. We measure test error separately on the held-out model. Each method selects 32 tasks. Error averages the mean absolute errors in partial score and exact completion, in percentage points. LAUTOMATED SEARCH FOR TASK-SELECTION METHODS We investigate whether a coding agent can find task-selection methods that better estimate a new model’s score on the full task set. The agent develops these methods using results from six models; we evaluate them on a seventh model whose results are withheld throughout the search. L.1SEARCH AND EVALUATION We provide the coding agent with results from six models on 107 OSWorld2 tasks. It can modify both how the tasks are selected and how scores on the selected tasks are used to predict the full-set score. Each candidate method selects exactly 32 tasks. We evaluate it using six-fold leave-one-agent-out validation: five models are used to fit the method, and the sixth to measure prediction error. The agent receives only this validation error and can revise its method. We separately test each candidate on the seventh model after fitting to all six available models. L.2VALIDATION GAINS DO NOT GENERALIZE TO THE HELD-OUT MODEL Figure 18 compares validation and test error across 34 candidate methods. Validation error falls from 2.43 to 0.01 percentage points, while error on the held-out model changes from 3.90 to 4.00 percentage points. The search nearly eliminates validation error without improving the estimate for the new model. Validation errors guide repeated method selection, allowing the search to overfit these six models. 46

Instant research discovery