Skip to content

Research Results Documentation Run

Run an evaluation

Evaluate a template or your own agent on benchmark tasks. Runs use your Modal account and model API keys.

Checking evaluation service…

Run from your own machine

The command-line toolkit is available now. It starts the agent and desktop environments on Modal and saves the results to your machine.

Install CUA-Speedrun and connect your Modal account.
Choose an agent template and add its model API key.
Run a benchmark and inspect the saved scores and trajectories.

Bring your own agent · Browse templates

Sign in with GitHub

Benchmark and settings Benchmark Evaluation track

A track specifies the hardware and evaluation settings used to compare runs.

Evaluation name

Agent implementation Agent source

agent.py · requiredinit.py · optional setup

Your agent runs once per task. Setup runs before the task clock starts. If you omit init.py, we supply an empty initializer. Up to 1 MiB combined. See the agent interface

Allocate the track's GPU for my model.
Leave off for agents that only call external model APIs.

Modal credentials and model keys

Compute is billed to your Modal workspace. Model calls are billed to the providers whose keys you supply.

Connect a Modal token pair. Create a token in Modal

Modal token ID

Modal token secret

Modal credentials are encrypted and saved to your account for future runs. You can remove them in account settings. Model keys below apply only to this evaluation and are stored encrypted.

Model API keys and environment variables

Use the exact variable name your agent reads, such as OPENAI_API_KEY or ANTHROPIC_API_KEY. Never include keys in uploaded code.

Review and launch

Select a benchmark and track above.

I understand this launches paid compute in my Modal account and may make paid calls using my model API keys.

Results start unlisted. A result-card link can be shared; publishing to a leaderboard is a separate action after evaluation.

Submission format

agent.py contains your action loop.
init.py prepares dependencies or a model server.

What happens next

The evaluator queues your run, starts environments on Modal, and records each task. Follow progress, inspect trajectories, and download the evidence from your run page.

Command-line usage

Run the benchmark on Modal

Agent templates

Browse agent templates

CUA-Speedrun · powered by Modal

Code Trajectory data