Coding tool and agent comparisons

SWE-bench (Verified)

Ranked by resolved on SWE-bench alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, OpenAI, Anthropic, Google DeepMind

22 of 60 results: cost — cheapest per resolved issue first. Ordered by measured cost per resolved issue, cheapest first; the place shown is still the place by score.

PlaceAgent or toolModels usedResolvedRun settingsCost per resolved issue
17mini-SWE-agentSWE-agentdeepseek-v3.2-reasoner60%
  • attempts: 1
  • checked by SWE-bench: yes
$0.0468measured: $14 for the run, 300 of 500 solved,
18mini-SWE-agentSWE-agentgpt-5-mini-2025-08-0759.8%
  • attempts: 1
  • reasoning effort: medium
  • checked by SWE-bench: yes
$0.0593measured: $17.7 for the run, 299 of 500 solved,
40mini-SWE-agentSWE-agentgpt-5-nano-2025-08-0734.8%
  • attempts: 1
  • reasoning effort: medium
  • checked by SWE-bench: yes
$0.109measured: $19 for the run, 174 of 500 solved,
22mini-SWE-agentSWE-agentglm-4.655.4%
  • attempts: 1
  • checked by SWE-bench: yes
$0.174measured: $48.3 for the run, 277 of 500 solved,
45mini-SWE-agentSWE-agentgpt-oss-120b26%
  • attempts: 1
  • checked by SWE-bench: yes
$0.22measured: $28.6 for the run, 130 of 500 solved,
11mini-SWE-agentSWE-agentgpt-5-2025-08-0765%
  • attempts: 1
  • reasoning effort: medium
  • checked by SWE-bench: yes
$0.431measured: $140.19 for the run, 325 of 500 solved,
22mini-SWE-agentSWE-agentQwen3-Coder-480B-A35B-Instruct55.4%
  • attempts: 1
  • checked by SWE-bench: yes
$0.448measured: $123.96 for the run, 277 of 500 solved,
33mini-SWE-agentSWE-agento4-mini-2025041645%
  • attempts: 1
  • checked by SWE-bench: yes
$0.467measured: $104.99 for the run, 225 of 500 solved,
26mini-SWE-agentSWE-agentgemini-2.5-pro53.6%
  • attempts: 1
  • checked by SWE-bench: yes
$0.538measured: $144.19 for the run, 268 of 500 solved,
24mini-SWE-agentSWE-agentGLM-4.554.2%
  • attempts: 1
  • checked by SWE-bench: yes
$0.548measured: $148.60 for the run, 271 of 500 solved,
19mini-SWE-agentSWE-agento3-2025041658.4%
  • attempts: 1
  • checked by SWE-bench: yes
$0.571measured: $166.83 for the run, 292 of 500 solved,
12mini-SWE-agentSWE-agentclaude-4-sonnet-2025051464.93%
  • attempts: 1
  • checked by SWE-bench: yes
$0.571measured: $185.73 for the run, 325 of 500 solved,
2mini-SWE-agentSWE-agentgemini-3-pro-preview74.2%
  • attempts: 1
  • checked by SWE-bench: yes
$0.62measured: $229.98 for the run, 371 of 500 solved,
28mini-SWE-agentSWE-agentclaude-3-7-sonnet-2025021952.8%
  • attempts: 1
  • checked by SWE-bench: yes
$0.672measured: $177.31 for the run, 264 of 500 solved,
13mini-SWE-agentSWE-agentKimi-K2-Thinking63.4%
  • attempts: 1
  • checked by SWE-bench: yes
$0.691measured: $219.15 for the run, 317 of 500 solved,
15mini-SWE-agentSWE-agentminimax-m261%
  • attempts: 1
  • checked by SWE-bench: yes
$0.702measured: $214.13 for the run, 305 of 500 solved,
54mini-SWE-agentSWE-agentQwen2.5-Coder-32B-Instruct9%
  • attempts: 1
  • checked by SWE-bench: yes
$0.757measured: $34.1 for the run, 45 of 500 solved,
5mini-SWE-agentSWE-agentclaude-sonnet-4-5-2025092970.6%
  • attempts: 1
  • checked by SWE-bench: yes
$0.791measured: $279.17 for the run, 353 of 500 solved,
1mini-SWE-agentSWE-agentclaude-opus-4-5-2025110174.4%
  • attempts: 1
  • reasoning effort: medium
  • checked by SWE-bench: yes
$0.969measured: $360.62 for the run, 372 of 500 solved,
34mini-SWE-agentSWE-agentKimi-K2-Instruct43.8%
  • attempts: 1
  • checked by SWE-bench: yes
$1.21measured: $265.80 for the run, 219 of 500 solved,
50mini-SWE-agentSWE-agentllama-4-maverick-instruct21.04%
  • attempts: 1
  • checked by SWE-bench: yes
$1.49measured: $156.85 for the run, 105 of 500 solved,
7mini-SWE-agentSWE-agentclaude-4-opus-2025051467.6%
  • attempts: 1
  • checked by SWE-bench: yes
$1.67measured: $565.64 for the run, 338 of 500 solved,

  1. #17 mini-SWE-agent

    Models used
    deepseek-v3.2-reasoner
    Resolved
    60%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.0468 measured (300 of 500 solved),

  2. #18 mini-SWE-agent

    Models used
    gpt-5-mini-2025-08-07
    Resolved
    59.8%,
    Run settings
    attempts: 1; reasoning effort: medium; checked by SWE-bench: yes
    Cost per resolved issue
    $0.0593 measured (299 of 500 solved),

  3. #40 mini-SWE-agent

    Models used
    gpt-5-nano-2025-08-07
    Resolved
    34.8%,
    Run settings
    attempts: 1; reasoning effort: medium; checked by SWE-bench: yes
    Cost per resolved issue
    $0.109 measured (174 of 500 solved),

  4. #22 mini-SWE-agent

    Models used
    glm-4.6
    Resolved
    55.4%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.174 measured (277 of 500 solved),

  5. #45 mini-SWE-agent

    Models used
    gpt-oss-120b
    Resolved
    26%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.22 measured (130 of 500 solved),

  6. #11 mini-SWE-agent

    Models used
    gpt-5-2025-08-07
    Resolved
    65%,
    Run settings
    attempts: 1; reasoning effort: medium; checked by SWE-bench: yes
    Cost per resolved issue
    $0.431 measured (325 of 500 solved),

  7. #22 mini-SWE-agent

    Models used
    Qwen3-Coder-480B-A35B-Instruct
    Resolved
    55.4%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.448 measured (277 of 500 solved),

  8. #33 mini-SWE-agent

    Models used
    o4-mini-20250416
    Resolved
    45%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.467 measured (225 of 500 solved),

  9. #26 mini-SWE-agent

    Models used
    gemini-2.5-pro
    Resolved
    53.6%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.538 measured (268 of 500 solved),

  10. #24 mini-SWE-agent

    Models used
    GLM-4.5
    Resolved
    54.2%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.548 measured (271 of 500 solved),

  11. #19 mini-SWE-agent

    Models used
    o3-20250416
    Resolved
    58.4%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.571 measured (292 of 500 solved),

  12. #12 mini-SWE-agent

    Models used
    claude-4-sonnet-20250514
    Resolved
    64.93%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.571 measured (325 of 500 solved),

  13. #2 mini-SWE-agent

    Models used
    gemini-3-pro-preview
    Resolved
    74.2%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.62 measured (371 of 500 solved),

  14. #28 mini-SWE-agent

    Models used
    claude-3-7-sonnet-20250219
    Resolved
    52.8%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.672 measured (264 of 500 solved),

  15. #13 mini-SWE-agent

    Models used
    Kimi-K2-Thinking
    Resolved
    63.4%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.691 measured (317 of 500 solved),

  16. #15 mini-SWE-agent

    Models used
    minimax-m2
    Resolved
    61%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.702 measured (305 of 500 solved),

  17. #54 mini-SWE-agent

    Models used
    Qwen2.5-Coder-32B-Instruct
    Resolved
    9%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.757 measured (45 of 500 solved),

  18. #5 mini-SWE-agent

    Models used
    claude-sonnet-4-5-20250929
    Resolved
    70.6%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $0.791 measured (353 of 500 solved),

  19. #1 mini-SWE-agent

    Models used
    claude-opus-4-5-20251101
    Resolved
    74.4%,
    Run settings
    attempts: 1; reasoning effort: medium; checked by SWE-bench: yes
    Cost per resolved issue
    $0.969 measured (372 of 500 solved),

  20. #34 mini-SWE-agent

    Models used
    Kimi-K2-Instruct
    Resolved
    43.8%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $1.21 measured (219 of 500 solved),

  21. #50 mini-SWE-agent

    Models used
    llama-4-maverick-instruct
    Resolved
    21.04%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $1.49 measured (105 of 500 solved),

  22. #7 mini-SWE-agent

    Models used
    claude-4-opus-20250514
    Resolved
    67.6%,
    Run settings
    attempts: 1; checked by SWE-bench: yes
    Cost per resolved issue
    $1.67 measured (338 of 500 solved),

Source: SWE-bench leaderboard. Each score links the published result it was read from.