How to use
Pick what fits you. Each answer has the steps and commands you need.
Getting started
What is the quickest way to try it?
Open the cua-speedrun notebook on Modal. Fork it into your Modal account and run its one cell. A dashboard opens where you pick an agent or upload your own, choose a task set, add your API keys, and start. Nothing to install, and runs keep going after you close the notebook.
The rest of this page uses the command line, which also runs on your own Linux machine.
What do I need?
Python 3.10 or newer, a Modal account, and an API key for your model, or its weights on Hugging Face. Modal is a pay-per-second cloud service that runs the desktops and your agent. CUA-World and MyPCBench also need a Gemini API key for grading. Without Modal, you can run on your own Linux machine.
How do I run my first evaluation?
pip install cua-speedrun
cua-speedrun setup
cua-speedrun benchmarksetup asks for your Modal token pair, which you create on Modal's Tokens page, and then for the API keys your agent needs. It saves them for later runs. benchmark then asks you to pick an agent and a task set, and shows progress while it runs. To turn your computer off during the run, add --host modal, explained below.
How do I choose the agent and task set in the command?
Pass them by name. cua-speedrun catalog lists every agent and task set.
cua-speedrun benchmark --agent claude --dataset osworld-50Can I turn off my computer while it runs?
Yes. Add --host modal. Then the part that coordinates the run also runs in your Modal account. Check on it later with cua-speedrun status RUN_ID. cua-speedrun evaluations lists your runs and their IDs.
Where does everything run?
Three flags choose where each part runs, and each takes modal or local. --environment is for the desktops, --compute is for your agent, and --host is for the part that coordinates the run. By default, the desktops and your agent run on Modal and your computer coordinates. To run everything on your own Linux machine, set --compute and --environment to local.
Evaluate a model
How do I evaluate my model?
Write an agent that calls your model, or adapt an included one. The included agents are in the agents folder of the code repository. Copy one and change its model call. Check it, then run it:
cua-speedrun validate --agent ./my-agent
cua-speedrun benchmark --agent ./my-agent --dataset osworld-50validate checks the files without starting a desktop.
Can I use an included agent with a different model?
Yes, a model from the same provider. Included agents read the model name from a variable, such as CLAUDE_MODEL for claude or OPENAI_MODEL for gpt54. Your agent runs on Modal, so it only sees the variables you pass with --env:
CLAUDE_MODEL=your-model-id cua-speedrun benchmark \
--agent claude --dataset osworld-50 --env CLAUDE_MODELWhich task sets are on the leaderboard?
A task set is a fixed list of tasks from a benchmark. The leaderboard uses four, each chosen so model rankings match the full benchmark. Pass the name to --dataset:
- OSWorld, 50 of 295 tasks:
osworld-50 - OSWorld 2.0, 52 of 63 tasks:
osworld2-52 - CUA-World, 26 of 143 tasks:
cua-world-26 - MyPCBench, 38 of 184 tasks:
my-pc-bench
The full benchmarks are osworld-offline, osworld2-offline, and cua-world-offline.
OSWorld 2.0 needs approval. Request access on both Hugging Face pages, one for its tasks and one for its assets. Then run hf auth login, which comes with cua-speedrun.
CUA-World and MyPCBench use a Gemini model to grade each task, billed to your Gemini API key. Save the key with cua-speedrun setup, as GEMINI_API_KEY for CUA-World and as MYPCBENCH_JUDGE_API_KEY for MyPCBench.
Open-weight models
How do I evaluate an open-weight model?
Use the included qwen3vl agent. Its init.py starts a vLLM server on a Modal GPU for the model named in VLLM_MODEL. The agent uses the Qwen3-VL prompt and action format. If your model uses a different one, copy the agent and change its prompt and parsing.
VLLM_MODEL=your-org/your-model cua-speedrun benchmark \
--agent qwen3vl --dataset osworld-50 --env VLLM_MODELLeaderboard results for open-weight models use an L40S GPU, set in the agent's agent.json. If your model does not fit, --gpu H200 overrides it. For a gated or private model, also pass --env HF_TOKEN. The first run downloads the model, and later runs reuse it. A model you serve yourself has no API bill, so its cost shows as a dash.
Build an agent
What does an agent look like?
A folder with two Python files, and an optional agent.json:
agent.pytimed, once per task- Looks at the screen, sends actions, and calls done.
init.pyuntimed, before tasks- Installs packages or starts a model server before the clock starts. It may run more than once, for example when a sandbox restarts, so it must be safe to repeat. If your agent needs no setup, it can just
print("ready"). agent.jsonoptional- The GPU and the keys the agent needs, for example
{"gpu": null, "required_environment_variables": ["MY_API_KEY"]}.
A complete agent.py, apart from your model call:
import sys
from cua_speedrun.client import Computer
computer = Computer(sys.argv[1]) # the desktop's address, given by cua-speedrun
instruction = sys.argv[2] # the task, in plain English
while True:
screenshot = computer.observe()["png"] # PNG bytes
actions = my_model(instruction, screenshot) # your code
if not actions:
break
computer.step(actions)
computer.done()What actions can it send?
The screen is 1920×1080 pixels. step takes a list of actions, for example computer.step([{"mouse": {"left_click": [640, 400]}}, {"keyboard": {"text": "hello"}}]). Sending several at once saves round trips. A scroll happens where the mouse is and counts wheel clicks. Key names include enter, tab, esc, ctrl, alt, shift, and single letters.
{"mouse": {"left_click": [640, 400]}}
{"mouse": {"double_click": [640, 400]}}
{"mouse": {"right_click": [640, 400]}}
{"mouse": {"move": [640, 400]}}
{"mouse": {"left_click_drag": [[100, 100], [300, 300]]}}
{"mouse": {"scroll": 3}} # positive scrolls down
{"keyboard": {"text": "hello"}}
{"keyboard": {"keys": ["enter"]}}
{"keyboard": {"keys": ["ctrl", "s"]}} # pressed together
{"action": "wait", "time": 1.0}How do I pass API keys and settings?
Save them with cua-speedrun setup, or export them in your shell. Keys listed in agent.json reach your agent automatically. Pass any other variable with --env NAME. This is a different flag from --environment.
How do I report model cost?
Your agent reports its own cost. After each model response, print the total cost so far for the task in this format. If it prints nothing, the cost shows as a dash.
__CUA_SPEEDRUN_COST_V1__{"cost_usd": 0.1234}What can my agent access?
The internet, and the desktop through observe, step, and done. It cannot reset the desktop, open a shell in it, or see how tasks are graded.
Add a benchmark
How do I add my benchmark?
Your tasks must first be a Gym-Anything environment. Gym-Anything is an open-source toolkit where each task has a desktop, a setup script, and a verifier that checks the result. See its guide to tasks and verifiers. Then lay out the folder like this:
my-benchmark/
manifest.yaml
environment/ # your Gym-Anything environment
tasks/my-task/task.yaml# my-benchmark/manifest.yaml
name: my-benchmark
version: "1"
tasks: [tasks/my-task]
# my-benchmark/tasks/my-task/task.yaml
task_id: my-task
description: Short label for this task
timeout_sec: 600 # time limit in seconds
env:
kind: gym-anything
env_dir: ${BENCHMARK_DIR}/environment # BENCHMARK_DIR is the my-benchmark folder
task_id: my-task # the task's name inside the environmentCheck it, then run any agent on it:
cua-speedrun validate --dataset ./my-benchmark
cua-speedrun benchmark --dataset ./my-benchmark --agent claudeTo add it to the leaderboard, email one of us (pranjala@cs.cmu.edu, ljang@cs.cmu.edu, jingyuk@cs.cmu.edu) or open an issue on GitHub.
How do I pick a smaller set of my tasks?
Clone the code repository and give this script your past model-by-task scores. It picks a subset so model rankings match the full benchmark. Run it with --help to see the input format.
python scripts/select_representative_tasks.py scores.json --output selection.jsonOther ways to run
What is Modal, and do I need it?
Modal is a cloud service that runs code on computers you rent by the second. cua-speedrun uses it for the desktops and your agent. Create an account at modal.com, then run cua-speedrun setup. The only way to skip Modal is to run on your own Linux machine.
Can I run on my own machine?
Yes, on Linux with KVM, which lets Linux run virtual machines at full speed. Check that /dev/kvm exists. Mac and Windows need Modal.
cua-speedrun benchmark --agent claude --dataset osworld-50 \
--compute local --environment localLeaderboard results were measured on Modal, so use Modal to compare with them.
Is there a browser dashboard?
Yes. cua-speedrun dashboard opens one in your browser, where you can start runs and follow their progress.
Can I run it from a script or a coding assistant?
Yes. Put --json right after cua-speedrun. It then writes JSON instead of asking questions:
cua-speedrun --json benchmark --agent claude --dataset osworld-50Results and data
What exactly is timed?
From the moment the agent receives the task until it calls done or reaches the time limit. Setup, init.py, and grading are not timed.
How are score and cost calculated?
The score comes from each benchmark's own verifier, averaged over its tasks, with partial credit where the benchmark gives it. The cost is the model API cost per task that your agent reports. It does not include Modal compute.
Where are my results?
Runs are saved in ~/.local/share/cua-speedrun/runs. Runs started with --host modal stay in your Modal account until you export them. Each task's folder has your agent's output and errors.
cua-speedrun evaluations # your runs and their IDs
cua-speedrun status RUN_ID # score, time per task, and progress
cua-speedrun export RUN_ID # every screenshot, action, and verdict as a zip fileWhere can I get the leaderboard data?
The results are on the leaderboard. Full trajectories are on Hugging Face.
The leaderboard
How do I get my result on the leaderboard?
Run the leaderboard's task sets on Modal and export each run. Then email one of us (pranjala@cs.cmu.edu, ljang@cs.cmu.edu, jingyuk@cs.cmu.edu) or open an issue on GitHub.
When something fails
A run failed. What should I check?
cua-speedrun status RUN_ID shows each task's state, and cua-speedrun doctor checks your installation.
Where do I report a bug?
Open an issue on GitHub with the command you ran and the error.