Leaderboard
Every configuration we measured on the paper's four task sets, ranked by score, time, or cost.
Top score
85.1%
Claude Opus 5.5 · Claude Codemax
3050.0 s · $15.17 per task
Fastest WR
487.6s
GPT-6.1 Terra · Codexlow
Scores 54.9%
Cheapest
$0.362
GPT-6.1 Terra · Codexlow
Scores 54.9% in 487.6 s
Filter models and reasoning effort
Results
All included results, ordered by average performance.
| # | Model | Effort | Performance | Time / task | Cost / task | |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 · Claude Code Cost / task: $15.1733 | max | 85.06% | 3050.0s | $15.1733 | |
| 2 | Claude Opus 5.5 · Claude Code Cost / task: $4.6277 | high | 77.54% | 1262.5s | $4.6277 | |
| 3 | GPT-6 Astra Cost / task: $8.1746 | xhigh | 76.90% | 1237.3s | $8.1746 | |
| 4 | Claude Opus 5.5 · Claude Code Cost / task: $7.8980 | xhigh | 75.57% | 1851.4s | $7.8980 | |
| 5 | GPT-6 Astra Cost / task: $7.3909 | high | 75.02% | 914.0s | $7.3909 | |
| 6 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $9.1804 | max | 71.28% | 3465.8s | $9.1804 | |
| 7 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $7.8880 | xhigh | 68.60% | 3118.0s | $7.8880 | |
| 8 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $6.7417 | medium | 68.59% | 3096.6s | $6.7417 | |
| 9 | GPT-6.1 Terra · Codex Cost / task: $1.5119 | max | 68.40% | 1479.4s | $1.5119 | |
| 10 | GPT-6 Astra Cost / task: $6.3363 | low | 68.17% | 904.2s | $6.3363 | |
| 11 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $5.5603 | low | 67.86% | 2189.0s | $5.5603 | |
| 12 | GPT-6 Astra Cost / task: $6.5192 | medium | 67.18% | 828.1s | $6.5192 | |
| 13 | GPT-6.1 Terra · Codex Cost / task: $0.8573 | xhigh | 66.57% | 1058.4s | $0.8573 | |
| 14 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $7.3116 | high | 66.42% | 2759.7s | $7.3116 | |
| 15 | Gemini 3.8 Flash Cost / task: $5.0195 | high | 60.18% | 3058.8s | $5.0195 | |
| 16 | Gemini 3.8 Flash Cost / task: $2.9148 | medium | 59.93% | 2051.8s | $2.9148 | |
| 17 | GPT-6.1 Terra · Codex Cost / task: $0.4785 | high | 57.27% | 614.1s | $0.4785 | |
| 18 | Claude Opus 5.5 · Claude Code Cost / task: $1.5418 | low | 56.56% | 628.0s | $1.5418 | |
| 19 | GPT-6.1 Terra · Codex Cost / task: $0.3621 | low | 54.93% | 487.6s | $0.3621 | |
| 20 | GPT-6.1 Terra · Codex Cost / task: $0.4058 | medium | 54.59% | 531.4s | $0.4058 | |
| 21 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $4.3643 | minimal | 54.46% | 1775.7s | $4.3643 | |
| 22 | Claude Opus 5 Cost / task: $10.0063 | low | 52.06% | 1385.7s | $10.0063 | |
| 23 | Kimi K3Open weights Cost / task: $10.7261 | max | 47.74% | 7744.3s | $10.7261 | |
| 24 | Claude Sonnet 5 Cost / task: $8.8630 | low | 39.08% | 1993.6s | $8.8630 | |
| 25 | Gemini 3.8 Flash Cost / task: $1.8752 | low | 35.29% | 1386.7s | $1.8752 | |
| 26 | Kimi K3Open weights Cost / task: $2.6425 | low | 34.88% | 2115.5s | $2.6425 | |
| 27 | MiniMax M3Open weights Cost / task: $2.3252 | thinking_on | 23.19% | 4675.6s | $2.3252 | |
| 28 | GLM-5V Turbo Cost / task: $3.2052 | thinking_on | 8.44% | 10358.9s | $3.2052 |
Task performance
Select a measure to compare. Tap a point for its model, score, and timing.
Scatter plot comparing model performance with the selected horizontal metric.
Performance, time, and cost
Drag to rotate; select a point to inspect it. Only results with known costs appear here.
Rotatable three-dimensional scatter plot comparing cost, time, and performance.
How to read these numbers
Score is the mean verifier score over the task set, with partial credit. Time is the mean time per task, from when the agent receives the instruction until it finishes; setup and grading are not counted. Cost is the model API cost per task. A dash means usage was not recorded.
Every configuration ran on the same tasks within its benchmark.