Most benchmarks only record whether an agent succeeded. CUA-Speedrun is an evaluation system that also measures how long each task takes and what its model calls cost. Every run uses the same pipeline and virtual machine setup on Modal, and one agent interface works across OSWorld, OSWorld 2.0, CUA-World, and MyPCBench. Smaller task sets, chosen so their scores track the full benchmarks, keep repeated runs affordable.
Mean time per task on the 50-task set, replayed at 40×.
0:00.00
Gemini 3 Flash Previewlow
#78:12.233.6%
GPT-6 Astramedium · Normal I/O
#31:30.289.6%
Kimi K3max · Batched tool calls
#67:23.789.6%
GPT-5.6 Lunalow · Direct API
#11:13.661.8%
Gemini 3.8 Flashlow
#42:07.191.6%
Gemini 3 Flash Previewmedium
#54:28.057.6%
Claude Opus 5low
#21:25.987.6%
Each lane is a measured configuration; ticks mark its mean number of steps. At the finish, each shows its mean task time and task score. The first to finish scores 61.8%. See all OSWorld results
Same machine, same clock, any agent
CUA-Speedrun is the software that runs these comparisons. It holds the hardware, desktops, tasks, and timing fixed, so differences in time and cost come from the agent.
Off the clock
Start the agent and desktop sandboxes, restore the task's starting state, run init.py
On the clock
Starts when the instruction is sentScreenshot, model call, keyboard and mouse actions, repeatedStops at done or the task limit
Off the clock
The benchmark's verifier scores the final state and trajectory
What each task's time includes. Agent time and environment time are recorded separately inside the timed interval.
Identical hardware for every run
Agents and desktops run in separate Modal sandboxes, built from the same pinned images for every run. Open-weight models are served with vLLM on fixed L40S GPUs. Results do not depend on whose computer launched the run.
One agent file, four benchmarks
An agent is a single agent.py that receives 1920×1080 screenshots and sends keyboard and mouse actions. The same file runs on OSWorld, OSWorld 2.0, CUA-World, and MyPCBench.
A clock that measures the agent
Setup and scoring happen off the clock. Infrastructure failures are retried; agent failures end the attempt and are scored. Model cost is logged per request.
Inputs that do what they say
CUA-AutoDebug sends 94 keyboard and mouse cases to an app that records what it receives. Through SSH and PyAutoGUI, 11 failed, such as typing > when < was requested. Our runtime passes all 94, five times each.
Smaller task sets that keep the ranking
Energy-distance selection keeps 50 of 295 OSWorld tasks, with a 0.98 rank correlation to full-benchmark scores for models left out of the selection.
An optional fast I/O mode
FastCUA cuts the delay between an action and the next screenshot from about 2–3 s to 2–28 ms, to test whether agents benefit from a faster desktop.
More reasoning, less time. Gemini 3 Flash Preview on OSWorld, low to medium reasoning effort. The score also rose from 33.6% to 57.6%; extra thinking replaced repeated failed actions.
492 s268 s
−224 s
A faster environment, a slower agent. GPT-6 Astra at low effort, averaged over five seeds. Screenshots arrived before applications finished updating, so the agent looked again or repeated itself.
89.5 s99.0 s
+9.5 s
Same score, four times the wait. Kimi K3 at max effort matches GPT-6 Astra at high effort on OSWorld (89.6%). No open-weight model we tested is on the time or cost frontier.
101 s444 s
×4.4
Fewer tasks, similar ranking. A 50-task OSWorld subset, chosen so that historical scores match the full set, cuts evaluation time by 84.9%.
Interactive overview. Squares show task counts; each plotted point is a measured configuration, marked with its developer's logo. The explorer covers the paper’s four task subsets, with configuration coverage curated separately from its figures. Logos identify developers and are trademarks of their owners.
Race your own agent
Start with an included agent or bring your own Python implementation. The toolkit prepares the desktops, runs benchmark tasks, and saves scores, timings, screenshots, and action logs. It supports OSWorld, OSWorld 2.0, CUA-World, and MyPCBench.
The results explorer covers the four task subsets used in the paper. Compare configurations, filter models, and inspect the records behind each result. Configuration coverage is curated separately from the paper’s figures; execution settings and cost assumptions accompany each record.
