CUA-Speedrun

Abstract

Computer-use agents (CUAs) operate graphical interfaces to complete tasks on a computer, and now exceed human reference scores on several standard benchmarks. Their speed and cost remain a barrier to deployment. Measuring speed reliably is difficult because benchmarks run on different machines, containers, and agent code, which confounds task time. We present CUA-Speedrun, standardized infrastructure and task sets for measuring the performance, speed, and cost of CUAs. It uses a uniform virtual machine setup and execution pipeline, and a common agent interface that lets one agent implementation run across benchmarks. Across four benchmarks, we study how reasoning effort, agent harnesses, and environment latency affect performance, time, and cost. No single model family is best on all three, and no open-weight model we evaluate is on the Pareto frontier. For some models, more reasoning shortens task time, while faster environment I/O lengthens it. We also show that most benchmarks can be reduced to a representative subset while preserving model rankings, making repeated evaluation cheaper.

1Introduction

Computer-use agents interact with graphical user interfaces to reach a goal given in natural language. Because they use the same interface as people, they can automate software that offers no API. Measured by success alone, recent models have passed human baselines: frontier models report 78.7% to 86.1% on OSWorld-Verified, against a 72.4% human reference score, and perform well on long-horizon benchmarks such as OSWorld 2.0 and CUA-World.

Speed and cost now limit practical use. Research on efficiency has trailed research on capability, partly because speed is hard to compare fairly. Benchmarks run on different execution environments, from local virtual machines to cloud-parallel desktops, and differ in deployment, API interfaces, hyperparameters, agent harnesses, and even benchmark implementations. Each of these changes the measured time.

CUA-Speedrun is a standardized platform that uses existing benchmarks to measure the speed and cost of CUAs. It addresses three problems:

Uniform evaluation infrastructure. Every evaluation runs through the same execution pipeline and the same virtual machine setup on a serverless cloud provider, Modal.
A common agent interface. One interface for observing the desktop and acting on it lets a single agent implementation run on any supported benchmark, while leaving the agent's own design free.
Representative task subsets. An energy-minimization method selects a small subset of each benchmark that preserves agent scores and their ordering. On OSWorld it reduces evaluation time by 84.9%.

We benchmark frontier and open-weight models on OSWorld-Verified and OSWorld 2.0, and extend the analysis to CUA-World (long-horizon tasks) and MyPCBench (personalized computer use). Success alone hides large differences in speed. On OSWorld, the open-weight Kimi K3 at maximum reasoning matches GPT-6 Astra at high effort but takes 4.4× longer per task, and none of the open-weight models we evaluate lie on a performance–time or performance–cost Pareto frontier. Speed also interacts with reasoning and infrastructure in unexpected ways: more reasoning can shorten tasks by avoiding repeated failed actions, and faster I/O can lengthen them.

2Standardized computer-use evaluation

A CUA evaluation combines four components: the agent, driven by a language model; the benchmark, which defines what must be accomplished; the environment infrastructure, which manages virtual machines, the environment lifecycle, and the action and observation contract; and the agent loop, which determines how the model interacts with the infrastructure. Existing evaluations make different choices for all four at once. Adopting a new model may bring a benchmark-specific action loop, and adopting a new benchmark may require changes to the agent or infrastructure. CUA-Speedrun separates these components so each can be changed on its own.

Problem setup

Following prior computer-use benchmarks, each task specifies an initial desktop state, a natural-language instruction, and a verifier (a programmatic check or a model judge) that scores the agent's trajectory and final state. Agents observe screenshots and act through keyboard and mouse actions. Comparisons between agents hold the benchmark and infrastructure fixed; comparisons across benchmarks hold the agent and infrastructure fixed. Each change is therefore a controlled ablation.

Measuring speed

For each task, the clock starts when the instruction is given to the agent and stops when the agent terminates or reaches the task limit (a step limit, a wall-time limit, or both). Environment provisioning, task setup, agent initialization, and verification happen outside this interval. Task time is further divided into environment operations and agent operations, so agent time can be attributed to the agent alone. We also log the cost of every model call. Infrastructure failures are retried; agent failures end the trajectory, which is then scored.

Off the clock
Start the agent and desktop sandboxes, restore the task's starting state, run init.py
On the clock
Starts when the instruction is sentScreenshot, model call, keyboard and mouse actions, repeatedStops at done or the task limit
Off the clock
The benchmark's verifier scores the final state and trajectory

What each task's time includes. Agent time and environment time are recorded separately inside the timed interval.

3Selecting representative tasks

Full computer-use evaluations are expensive: long trajectories add API or GPU cost, and managing virtual machines adds overhead. GPT-5.4, for example, costs about $4,000 to evaluate on CUA-World-Long. We ask whether a smaller task set can recover the scores of the agents evaluated on the full set, keeping individual scores close and preserving their order.

Selection by energy distance

We validate every selection method with leave-one-agent-out evaluation: for each agent, a subset is built from the other agents' results only, and the held-out agent's subset score is compared with its full-benchmark score. Across several iterations of this procedure, the best method minimized the energy distance between the distributions of agent scores on the full benchmark and on the subset. Let C_mi ∈ [0, 1] be agent m's partial score on task i and B_mi = 1[C_mi = 1] its exact completion. Task i is represented by

z_{i} = (C_{1 i}, \dots, C_{M i}, B_{1 i}, \dots, B_{M i}) \in ℝ^{2 M}

so two tasks are similar when agents show similar patterns of partial and exact completion on them. With D_ij = ‖z_i − z_j‖₂ and D̄ the mean distance between distinct tasks, a subset S_K of K tasks from N is scored by

ℰ (S_{K}) = \frac{1}{\bar{D}} (\frac{2}{K N} \sum_{i \in S_{K}} \sum_{j = 1}^{N} D_{i j} - \frac{1}{K^{2}} \sum_{i, i^{'} \in S_{K}} D_{i i^{'}} - \frac{1}{N^{2}} \sum_{j, j^{'}} D_{j j^{'}})

The objective favors subsets that cover the performance patterns of the full benchmark while avoiding redundant tasks. The last term depends only on the full benchmark and is constant during selection. We minimize it approximately from 100 starting subsets, swapping one selected task for an unselected one whenever the swap lowers the objective, and keep the best subset found.

How many tasks are enough

Let ρ_q(K) be the Spearman correlation between leave-one-agent-out subset estimates and full-benchmark results for quantity q (partial score or exact completion). We choose the smallest K that clears 0.95 at K and both neighbors, so a size that works only by chance is not selected:

K^{*} = min {K : min_{q} min_{k \in {K - 1, K, K + 1}} ρ_{q} (k) \geq 0.95}

Table 1. CUA-Speedrun evaluation sets
Benchmark	Eligible tasks	Selected tasks	Reduction
OSWorld	295	50	5.9×
OSWorld 2.0	63	52	1.2×
CUA-World	143	26	5.5×
MyPCBench	184	38	4.8×

Validation on held-out models

At 50 OSWorld tasks, energy selection reduces the held-out partial-score error from 4.02 to 2.16 percentage points compared with difficulty-stratified random selection, and reduces the mean-task-time error from 51.0 to 31.4 s even though selection uses only scores. Rank correlations are 0.983 (partial score) and 0.975 (exact completion) on OSWorld at 50 tasks, and 0.964 and 0.955 on OSWorld 2.0 at 52 tasks; both stay above 0.95 at the neighboring sizes. At a fixed budget of 32 OSWorld tasks, energy selection has a mean error of 4.42 points, against 5.08 for stratified random selection and 8.08 for an IRT-inspired method.

Four panels: score error and time error against number of selected OSWorld tasks for stratified random and energy selection, and Spearman rank correlation against number of tasks for OSWorld and OSWorld 2.0. — **Figure 2.** Score error and rank correlation on held-out models. OSWorld uses nine leave-one-agent-out folds and OSWorld 2.0 uses seven. The randomized baseline averages 300 difficulty-stratified selections per task count. Dotted vertical lines mark 50 and 52 tasks; horizontal lines mark rank correlation 0.95. Figure PDF

The subset also preserves the speed–performance frontier: estimated from held-out 50-task subsets, it recovers two of the three full-set frontier configurations and adds none (100% precision, 66.7% recall), and pairwise dominance agrees for 91.7% of model pairs. The 50 selected OSWorld tasks are 5.9× fewer than the full set and account for 15.1% of recorded task time, a 6.6× reduction. Selected subsets may need revalidation as agent capabilities change.

4Infrastructure

Hosting

We adapt the virtual machine runtime from Gym-Anything to run task environments natively in Modal sandboxes. Agents and environments run in separate sandboxes for isolation, and Modal provides a standardized hosted runtime, so differences in users' own machines do not affect results. Self-hosted open-weight models are served with vLLM on fixed L40S GPUs. A Python library handles GPU scheduling, environment allocation, and parallel evaluation.

Actions and observations

The action space consists of keyboard actions, such as typing text or pressing Ctrl+C, and mouse actions, such as clicking or double-clicking at coordinates. Observations are 1920×1080 RGB screenshots of the desktop.

Checking the interface with CUA-AutoDebug

To make actions behave identically across runtimes, CUA-AutoDebug runs a controlled application that records the keyboard and mouse input it receives and compares it with the intended input, and checks that screenshots match the application's state. Of 100 cases covering clicks, drags, scrolls, key presses, key combinations, text entry, and sequences, 94 are supported by the tested interface. Repeating each five times through SSH and PyAutoGUI, 83 pass and 11 fail: one shell-expansion case, one shifted-symbol case (typing > when < was requested), five Unicode cases (accented characters silently dropped), and four named-key cases. After we corrected literal text handling, key and modifier mapping, and characters missing from the default keymap, the CUA-Speedrun runtime passes all 470 runs.

FastCUA, a fast I/O mode

Standard computer-use runtimes take roughly 2–3 s from issuing an action to returning an observation, dominated by action execution, waiting for the application, and networking. That delay incidentally gives applications time to respond, so the agent sees the effect of its action. Our main evaluations keep this design. We also built FastCUA, which optimizes networking, action execution, and image processing to bring action-to-observation latency down to 2–28 ms, and use it to test whether lower infrastructure latency reduces task time (Section 6.4).

Writing an agent

An agent is a single Python file containing its loop, with an optional initializer for setup that runs before the clock starts. We provide templates for popular agents, and one command runs either a hosted or a local evaluation. The documentation walks through both.

5Experimental setup

We evaluate 56 agent configurations on OSWorld and 21 on OSWorld 2.0 using the representative task sets from Section 3. Configurations cover frontier open-weight and proprietary models, each run through its public reference agent or native computer-use API at selected reasoning-effort settings, with tasks and infrastructure held fixed. Every agent design is checked by manual inspection and with CUA-AutoDebug on representative tasks. For selected models we also compare agent harnesses, standard and fast I/O, and single versus batched tool calls per model response.

Metrics. For each configuration we report the mean verifier score, which keeps partial credit, the mean task time (Section 2), and the mean model cost per task, taken from the provider's recorded charge or from recorded token usage at the applicable API prices. We report the performance–time and performance–cost Pareto frontiers and the joint frontier over all three.

Repeatability. Across five seeds, GPT-6 Astra at low effort scores 90.8 ± 1.8% on OSWorld with a mean task time of 89.5 ± 2.2 s.

6Results

6.1 No single model family dominates the frontier

Four scatter plots: mean score against mean task time and against model cost, for OSWorld and OSWorld 2.0, with Pareto frontiers circled. — **Figure 3.** Different agents define the speed and cost frontiers. Mean score against mean task time and model cost on OSWorld (top, 54 configurations) and OSWorld 2.0 (bottom, 19 configurations), excluding MiniMax M3. Dashed lines connect the observed Pareto-frontier points, which are circled. Darker shades indicate higher reasoning effort. Astra (low) uses five-seed means for each I/O setting. Figure PDF

On OSWorld, the high-performance end of the time frontier runs from Claude Opus 5 (low), at 87.6% in 86 s, through GPT-6 Astra (low), at 90.8% in 90 s, to Astra (xhigh), at 91.6% in 127 s. Gemini 3.8 Flash (low) matches the top score in nearly the same time as Astra, but costs $0.11 per task rather than $0.71. At the low-cost end, GPT-5.6 Luna (low, direct API) costs about $0.01 per task and scores 61.8%.

6.2 The frontier changes across benchmarks

On OSWorld 2.0, the time frontier consists entirely of GPT-6 Astra configurations. Astra (high) reaches 75.0% in 914 s; xhigh reaches 76.9% in 1,237 s, gaining 1.9 points for 35% more time. The cost frontier includes other models: Gemini 3.8 Flash (medium) scores 59.9% at $2.91 per task, compared with $7.39 for Astra (high), and Muse Spark 1.3 (minimal) extends the frontier to an estimated $0.39 at 34.0%. No open-weight model is on either frontier for either benchmark. The preferred agent depends on the benchmark and on the performance needed within a time or cost budget.

6.3 Reasoning effort cuts both ways

Three bar charts for Gemini 3 Flash Preview at low, medium, and high effort: mean task time, generated tokens, and model calls. — **Figure 4.** Less reasoning can make an agent slower. Gemini 3 Flash Preview at low, medium, and high effort on OSWorld: mean task time, generated tokens including reasoning, and model calls across all 50 tasks. Scores are 33.6%, 57.6%, and 59.6%. Medium and high effort generate more tokens than low effort but need fewer calls and finish sooner. Figure PDF

Less reasoning can make an agent slower. Moving Gemini 3 Flash Preview from low to medium effort raises its OSWorld score from 33.6% to 57.6% and lowers mean task time from 492 s to 268 s. At low effort the model generates fewer tokens (1,598 against 4,125) but takes far more steps (56.0 against 32.5), because most of its steps are unproductive. Its trajectories repeat failed actions, such as trying to enter spreadsheet text through key combinations instead of typing it.

Two line plots of mean score against mean task time for Gemini 3.8 Flash at low, medium, and high effort, on OSWorld and OSWorld 2.0, labeled with cost per task. — **Figure 5.** The useful reasoning setting changes across benchmarks. Gemini 3.8 Flash at low, medium, and high effort, labeled with mean cost per task. Model, harness, and execution settings are fixed within each curve. Extra reasoning brings no gain on OSWorld, while moving from low to medium substantially improves OSWorld 2.0. Figure PDF

More reasoning can add time and cost without improving performance. On OSWorld, Gemini 3.8 Flash scores 91.6% at both low and medium effort, but medium roughly doubles cost and raises mean task time from 127 s to 250 s. On OSWorld 2.0, low to medium improves the score from 35.3% to 59.9%, while high adds only 0.2 points for 49% more time and 72% more cost. GPT-6 Astra shows the same dependence: on OSWorld, medium is within two points of xhigh with 29% less time and 15% less cost, while on OSWorld 2.0, high gains 7.8 points over medium for 10% more time.

6.4 Faster infrastructure can make the agent slower

Stacked bars comparing normal and fast I/O time components, and two screenshots showing a Save As dialog missing then present. — **Figure 6.** Faster I/O can be offset by more waiting and interaction. GPT-6 Astra (low) with normal and fast I/O, averaged over five seeds per setting. (a) Agent time, agent-requested waits, and remaining environment processing; error bars show the standard deviation of mean task time across seeds. (b) Two consecutive screenshots after opening Save As: the dialog is absent in the first and visible in the second, requested 5.8 s later with no intervening action. Figure PDF

With FastCUA, mean environment processing time falls from 12.68 s to 0.34 s per task, yet total task time for GPT-6 Astra (low) rises from 89.5 s to 99.0 s, and trajectories contain more steps. Fast I/O can return a screenshot before the application has updated. After opening Save As, the agent receives a screenshot without the dialog and asks for another one 5.8 s later. A model could in principle wait about 100 ms for the interface to settle, but current models do not use the faster loop well.

6.5 The harness matters as much as the model

Line plot of mean score against mean task time for GPT-5.6 Luna at low, medium, and high effort through a direct-API agent and through Codex. — **Figure 7.** The same model through two harnesses. GPT-5.6 Luna on OSWorld at three reasoning settings, through a direct-API agent and through the Codex harness. Figure PDF

At low effort, Codex raises GPT-5.6 Luna's OSWorld score from 61.8% to 75.6% but doubles mean task time, from 74 s to 145 s. At medium and high effort the direct-API agent is both more accurate and faster (75.6% in 119 s against 67.6% in 189 s at medium; 81.6% in 161 s against 79.6% in 325 s at high), so neither harness is better in every setting.

Batching actions reduces task time. Each step costs a screenshot, a model call, and environment latency, so agents that issue several actions per response finish sooner. GPT-6 Astra (low) completes OSWorld tasks in 5.9 action batches on average. Restricting Astra (xhigh) to one action per response keeps its score at 91.6% but raises mean task time from 127 s to 183 s.

Fast agents generate fewer tokens, not tokens faster. Claude Opus 5 (low) generates 1,795 tokens per task against 1,579 for GPT-6 Astra (low), scoring 87.6% against 90.8%. Claude Sonnet 5 (high) takes 141 s for the score Astra (medium) reaches in 90 s while generating 3.5× more tokens, and Gemini 3.8 Flash (high) generates 5.6× more tokens than at low effort and finishes 2.7× later.

6.6 Choosing across all three measures

Three-dimensional plot of OSWorld score against time per task and cost per task, with the joint Pareto frontier surface and labeled frontier configurations. — **Figure 8.** Choosing an agent by performance, time, and cost. Joint Pareto frontier on OSWorld; circled points are configurations on the frontier. Time and cost axes are logarithmic. Figure PDF

Other benchmarks. We also evaluate GPT-6 Astra through Codex at four reasoning settings on MyPCBench and CUA-World, which test personalized and long-horizon computer use and are graded by vision-language models rather than programmatic checks. Astra exceeds 90% rubric scores on both, which points to the need for harder task sets.

Every measured configuration, including source records and cost assumptions, is in the results explorer. Its catalog is curated separately from the figure populations above.

7Recommendations for building fast agents

Choose a frontier configuration for the performance and budget you need. No configuration is best on every measure, and the frontier moves between benchmarks, so measure candidates on tasks that resemble the target workload.
Tune reasoning effort rather than setting it to an extreme. Too little lengthens trajectories; too much adds time and cost without improving performance. Start at low or medium and raise it only for a measured gain.
Batch actions when intermediate screenshots are unnecessary. Fewer steps mean fewer screenshots, model calls, and environment round trips.
Reduce tokens and steps rather than chasing token rate or I/O speed. Faster generation or I/O helps only if the agent does not spend it on longer outputs or extra steps.
Optimize the harness together with the model. The same model and reasoning setting can land at different points on the frontier depending on its harness.

9Limitations

Reported times and costs reflect API inference conditions at the time of evaluation; providers control later changes to inference speed and pricing. Representative subsets are validated on held-out agents, but their scores approximate full-benchmark performance, and changes in agent capability may require revalidating them. Evaluation cost limits coverage of every combination of model, harness, reasoning effort, and benchmark. Findings apply to the tested combinations.

10Conclusion

CUA-Speedrun evaluates the performance, speed, and cost of computer-use agents on standardized infrastructure. Our evaluations identify the performance–time and performance–cost frontiers on OSWorld and OSWorld 2.0 and show how they change between benchmarks. More reasoning can reduce task time by avoiding repeated failed actions, and faster I/O can increase it through extra interaction, so reasoning and interaction should be studied together when building faster agents.

Citation

@misc{aggarwal2026cuaspeedrun,
  title  = {CUA-Speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents},
  author = {Aggarwal, Pranjal and Jang, Lawrence Keunho and Welleck, Sean and Fried, Daniel and Salakhutdinov, Ruslan and Koh, Jing Yu},
  year   = {2026},
  url    = {https://cuaspeedrun.com}
}

Based on manuscript revision 128fef0. Figures are reproduced from its original exports; see figure provenance.