Run your first evaluation

Before you start

You need a Modal account and the model API key required by your agent. Modal supplies the compute; model calls and compute are billed to your accounts. The example below uses the Codex CLI template and OPENAI_API_KEY.

Install and run on Modal

Install the toolkit on the machine that will coordinate the evaluation:

git clone --recurse-submodules https://github.com/Pranjal2041/cua-speed-run.git
cd cua-speed-run
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e .
cua-speedrun install
modal token new
cua-speedrun doctor --require modal

Add OPENAI_API_KEY and your Modal token pair to your shell or local .env, following the repository setup. Keep credentials out of source control.

Run the template on a small OSWorld task set:

cua-speedrun run --remote \
  --submission templates/codex_cli \
  --benchmark benchmarks/osworld-mini \
  --agents-per-evaluation 2

The toolkit starts the environments, runs the tasks, and prints a score summary with the saved run's location. This is a practice run; it does not enter an official ranking.

cua-speedrun dashboard
# In another terminal using the same installation:
cua-speedrun catalog
cua-speedrun submit --template codex_cli \
  --track open-l40s-shared \
  --benchmark osworld-pareto-48 \
  --compute modal --environment modal --no-gpu

Evaluate your own agent

Copy an agent template and edit its two Python files. To try a new model, change the model call and how its response becomes an action.

agent.py

The agent's action loop. It runs once per task as python agent.py <env_url> <task_description>. Use the environment client to observe the desktop, take actions, and finish.

init.py

Install dependencies or start a model server before timing begins. Keep this file for command-line runs, even when no setup is needed. Initialization can run more than once, so it must be safe to repeat.

The agent can observe, step, and call done. It cannot reset the environment or read the task's expected answers. Supply model keys through environment variables rather than including them in either file.

Python environment client · Complete agent examples

Run an open-weight model

The Qwen3-VL template starts a vLLM model server during initialization. This example requests an L40S GPU on Modal:

cua-speedrun run --remote \
  --submission templates/qwen3vl \
  --benchmark benchmarks/osworld-mini \
  --agents-per-evaluation 2 --gpu L40S

Choose a GPU with enough memory for your model and context. The desktop environment runs separately. If a run is interrupted, check your Modal workspace for remaining resources.

Read the results

Task time starts when the agent receives its instruction and ends when it finishes or times out. Machine startup, initialization, and the final success check are excluded. Saved trajectories contain the observations and actions taken during the task.

Model cost is recorded when the agent reports it. The toolkit does not automatically calculate provider charges. Follow the cost reporting instructions when adding a model.

Official rankings compare compatible runs that clear the track's required score. The public results explorer groups research measurements by task set and includes their execution settings and cost assumptions.

Hosted evaluations

The submission page shows whether hosted evaluations are available. When open, you can sign in with GitHub, choose a benchmark, and select a template or upload your Python files. Compute and model calls use your own accounts.