Leaderboard
Every configuration we measured on the paper's four task sets, ranked by score, time, or cost.
Top score
93.6%
GPT-6 Astra · Codexxhigh
619.2 s · $4.81 per task
Fastest WR
495.4s
GPT-6 Astra · Codexmedium
Scores 88.3%
Cheapest
$4.332
GPT-6 Astra · Codexmedium
Scores 88.3% in 495.4 s
Filter models and reasoning effort
Results
All included results, ordered by average performance.
| # | Model | Effort | Performance | Time / task | Cost / task | |
|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra · Codex Cost / task: $4.8099 | xhigh | 93.55% | 619.2s | $4.8099 | |
| 2 | GPT-6 Astra · Codex Cost / task: $4.5741 | low | 89.87% | 514.3s | $4.5741 | |
| 3 | GPT-6 Astra · Codex Cost / task: $4.8124 | high | 88.63% | 568.3s | $4.8124 | |
| 4 | GPT-6 Astra · Codex Cost / task: $4.3321 | medium | 88.34% | 495.4s | $4.3321 |
Task performance
Select a measure to compare. Tap a point for its model, score, and timing.
Scatter plot comparing model performance with the selected horizontal metric.
Performance, time, and cost
Drag to rotate; select a point to inspect it. Only results with known costs appear here.
Rotatable three-dimensional scatter plot comparing cost, time, and performance.
How to read these numbers
Score is the mean verifier score over the task set, with partial credit. Time is the mean time per task, from when the agent receives the instruction until it finishes; setup and grading are not counted. Cost is the model API cost per task. A dash means usage was not recorded.
Every configuration ran on the same tasks within its benchmark.