Published evaluations
Leaderboard
Every configuration we measured on the paper's four task sets, ranked by score, time, or cost.
Top score
95.6%
Claude Opus 5.5 · Claude Codexhigh
88.4 s · $0.53 per task
Fastest WR
47.7s
Claude Opus 5.5 · Claude Codemedium
Scores 91.6%
Cheapest
$0.011
GPT-5.6 Luna · Direct APIlow
Scores 61.8% in 73.6 s
Top score
95.6%
Claude Opus 5.5 · Claude Codexhigh
88.4 s · $0.53 per task
Fastest WR
47.7s
Claude Opus 5.5 · Claude Codemedium
Scores 91.6%
Cheapest
$0.011
GPT-5.6 Luna · Direct APIlow
Scores 61.8% in 73.6 s
Filter models and reasoning effort
All models
Results
All included results, ordered by average performance.
| # | Model | Effort | Performance | Time / task | Cost / task | |
|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 · Claude Code Cost / task: $0.5303 | xhigh | 95.62% | 88.4s | $0.5303 | |
| 2 | Claude Opus 5.5 · Claude Code Cost / task: $1.4258 | max | 95.62% | 311.5s | $1.4258 | |
| 3 | Claude Opus 5.5 · Claude Code Cost / task: $0.3496 | high | 93.62% | 59.5s | $0.3496 | |
| 4 | Claude Opus 5.5 · Claude Code Cost / task: $0.3036 | medium | 91.62% | 47.7s | $0.3036 | |
| 5 | GPT-6.1 Terra · Codex Cost / task: $0.0544 | medium | 91.62% | 53.3s | $0.0544 | |
| 6 | GPT-6.1 Terra · Codex Cost / task: $0.0564 | high | 91.62% | 55.8s | $0.0564 | |
| 7 | GPT-6.1 Terra · Codex Cost / task: $0.0694 | xhigh | 91.62% | 59.3s | $0.0694 | |
| 8 | GPT-6.1 Terra · Codex Cost / task: $0.1252 | max | 91.62% | 101.4s | $0.1252 | |
| 9 | GPT-6 Astra Cost / task: $0.7081 | xhigh | 91.62% | 126.8s | $0.7081 | |
| 10 | Gemini 3.8 Flash Cost / task: $0.1080 | low | 91.62% | 127.1s | $0.1080 | |
| 11 | Claude Opus 5 Cost / task: $0.6611 | high | 91.62% | 136.0s | $0.6611 | |
| 12 | GPT-6 Astra · Single action Cost / task: $1.3637 | xhigh | 91.62% | 182.7s | $1.3637 | |
| 13 | Gemini 3.8 Flash Cost / task: $0.2209 | medium | 91.62% | 250.1s | $0.2209 | |
| 14 | Claude Opus 5.5 · Claude Code Cost / task: $0.2523 | low | 89.62% | 57.2s | $0.2523 | |
| 15 | GPT-6 Astra Cost / task: $0.6001 | medium | 89.62% | 90.2s | $0.6001 | |
| 16 | GPT-6 Astra Cost / task: $0.6364 | high | 89.62% | 100.8s | $0.6364 | |
| 17 | GPT-6 Astra · Fast I/O Cost / task: $0.7367 | low | 89.62% | 105.2s | $0.7367 | |
| 18 | Claude Sonnet 5 Cost / task: $0.3086 | high | 89.62% | 140.8s | $0.3086 | |
| 19 | Gemini 3.7 Flash Cost / task: $0.9437 | medium | 89.62% | 198.9s | $0.9437 | |
| 20 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $0.8896 | high | 89.62% | 214.6s | $0.8896 | |
| 21 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $0.8980 | xhigh | 89.62% | 222.1s | $0.8980 | |
| 22 | Gemini 3.7 Flash Cost / task: $1.4121 | high | 89.62% | 254.3s | $1.4121 | |
| 23 | Kimi K3Open weights Cost / task: $0.5034 | max | 89.62% | 443.7s | $0.5034 | |
| 24 | Claude Opus 5 Cost / task: $0.3896 | low | 87.62% | 85.9s | $0.3896 | |
| 25 | GPT-6 Astra Cost / task: $0.5665 | low | 87.62% | 86.1s | $0.5665 | |
| 26 | Gemini 3.7 Flash Cost / task: $0.9264 | low | 87.62% | 207.3s | $0.9264 | |
| 27 | Gemini 3.8 Flash Cost / task: $0.3475 | high | 87.62% | 347.7s | $0.3475 | |
| 28 | GPT-6.1 Terra · Codex Cost / task: $0.0632 | low | 85.62% | 58.0s | $0.0632 | |
| 29 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $0.7631 | medium | 85.62% | 188.0s | $0.7631 | |
| 30 | Kimi K3 · Single tool callOpen weights Cost / task: $0.7158 | max | 85.62% | 606.3s | $0.7158 | |
| 31 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $1.0006 | max | 85.54% | 241.6s | $1.0006 | |
| 32 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $0.5003 | minimal | 83.62% | 90.9s | $0.5003 | |
| 33 | Meta Muse Wonder 1.0 · Muse Code Cost / task: $0.6559 | low | 83.62% | 141.6s | $0.6559 | |
| 34 | GPT-5.6 Luna · Direct API Cost / task: $0.0256 | medium | 81.62% | 100.6s | $0.0256 | |
| 35 | GPT-5.6 Sol Cost / task: $0.5267 | xhigh | 81.62% | 108.3s | $0.5267 | |
| 36 | Kimi K3 · Single tool callOpen weights Cost / task: $0.4141 | high | 81.62% | 305.1s | $0.4141 | |
| 37 | GPT-5.6 Sol Cost / task: $0.5394 | medium | 79.62% | 121.3s | $0.5394 | |
| 38 | Yutori n2 Cost / task: $0.1036 | xhigh | 79.62% | 225.0s | $0.1036 | |
| 39 | GPT-5.6 Luna · Codex Cost / task: $0.0759 | high | 79.62% | 324.8s | $0.0759 | |
| 40 | Kimi K3Open weights Cost / task: $0.3823 | high | 79.62% | 358.1s | $0.3823 | |
| 41 | Meta Muse Spark 1.1 Cost / task: $1.6849 | xhigh | 79.62% | 417.2s | $1.6849 | |
| 42 | GPT-5.6 Luna · Codex Cost / task: $0.0631 | xhigh | 77.62% | 297.2s | $0.0631 | |
| 43 | Yutori n2 Cost / task: $0.1569 | low | 77.62% | 331.0s | $0.1569 | |
| 44 | Meta Muse Spark 1.3 Cost / task: $0.1607 | xhigh | 77.62% | 612.1s | $0.1607 | |
| 45 | Meta Muse Spark 1.3 Cost / task: $0.1258 | medium | 77.54% | 397.6s | $0.1258 | |
| 46 | Claude Sonnet 5 Cost / task: $0.2941 | low | 75.81% | 129.4s | $0.2941 | |
| 47 | GPT-5.6 Luna · Codex Cost / task: $0.0254 | low | 75.62% | 144.8s | $0.0254 | |
| 48 | MiniMax M3Open weights Cost / task: — | thinking_off | 75.62% | 253.8s | — | |
| 49 | Yutori n2 Cost / task: $0.0811 | none | 75.62% | 298.5s | $0.0811 | |
| 50 | GPT-5.6 Luna · Direct API Cost / task: $0.0386 | xhigh | 73.62% | 135.3s | $0.0386 | |
| 51 | Kimi K3Open weights Cost / task: $0.1740 | low | 73.62% | 149.5s | $0.1740 | |
| 52 | Meta Muse Spark 1.3 Cost / task: $0.1479 | high | 73.62% | 506.9s | $0.1479 | |
| 53 | Kimi K3 · Single tool callOpen weights Cost / task: $0.2802 | low | 71.62% | 176.6s | $0.2802 | |
| 54 | MiniMax M3Open weights Cost / task: — | thinking_on | 69.62% | 345.1s | — | |
| 55 | Meta Muse Spark 1.1 Cost / task: $1.3785 | medium | 69.62% | 359.8s | $1.3785 | |
| 56 | Meta Muse Spark 1.1 Cost / task: $1.7059 | high | 69.62% | 368.5s | $1.7059 | |
| 57 | Meta Muse Spark 1.1 Cost / task: $1.3116 | low | 69.62% | 370.4s | $1.3116 | |
| 58 | Meta Muse Spark 1.2 Cost / task: $0.2445 | xhigh | 69.62% | 720.9s | $0.2445 | |
| 59 | GPT-5.6 Luna · Codex Cost / task: $0.0353 | medium | 67.62% | 188.8s | $0.0353 | |
| 60 | Yutori n2 Cost / task: $0.1100 | medium | 67.62% | 265.4s | $0.1100 | |
| 61 | Meta Muse Spark 1.1 Cost / task: $1.1840 | minimal | 67.62% | 318.5s | $1.1840 | |
| 62 | Meta Muse Spark 1.3 Cost / task: $0.1071 | low | 67.62% | 361.0s | $0.1071 | |
| 63 | GPT-5.6 Sol Cost / task: $0.3191 | low | 65.81% | 100.1s | $0.3191 | |
| 64 | Meta Muse Spark 1.2 Cost / task: $0.1054 | low | 63.62% | 363.6s | $0.1054 | |
| 65 | Meta Muse Spark 1.2 Cost / task: $0.1365 | medium | 63.62% | 466.2s | $0.1365 | |
| 66 | GPT-5.6 Luna · Direct API Cost / task: $0.0107 | low | 61.81% | 73.6s | $0.0107 | |
| 67 | Meta Muse Spark 1.3 Cost / task: $0.0789 | minimal | 61.62% | 264.4s | $0.0789 | |
| 68 | Meta Muse Spark 1.2 Cost / task: $0.1736 | high | 61.62% | 669.8s | $0.1736 | |
| 69 | Gemini 3 Flash Preview Cost / task: $0.8091 | high | 59.62% | 337.2s | $0.8091 | |
| 70 | GLM-5V Turbo Cost / task: $0.2203 | thinking_off | 57.81% | 752.4s | $0.2203 | |
| 71 | Gemini 3 Flash Preview Cost / task: $0.5923 | medium | 57.62% | 268.0s | $0.5923 | |
| 72 | GLM-5V Turbo Cost / task: $0.2350 | thinking_on | 53.81% | 359.2s | $0.2350 | |
| 73 | Meta Muse Spark 1.2 Cost / task: $0.0697 | minimal | 53.62% | 248.8s | $0.0697 | |
| 74 | Gemini 3 Flash Preview Cost / task: $1.4006 | low | 33.62% | 492.2s | $1.4006 |
No matching resultsAdjust the model or effort filters.
Trade-offs
Task performance
Select a measure to compare. Tap a point for its model, score, and timing.
Higher performance and lower time are better.
Scatter plot comparing model performance with the selected horizontal metric.
Three objectives
Performance, time, and cost
Drag to rotate; select a point to inspect it. Only results with known costs appear here.
Rotatable three-dimensional scatter plot comparing cost, time, and performance.
How to read these numbers
Score is the mean verifier score over the task set, with partial credit. Time is the mean time per task, from when the agent receives the instruction until it finishes; setup and grading are not counted. Cost is the model API cost per task. A dash means usage was not recorded.
Every configuration ran on the same tasks within its benchmark.