Coding tool and agent comparisons

SWE-bench (Verified)

Ranked by resolved on SWE-bench alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, OpenAI, Anthropic, Google DeepMind

12 of 60 results: attempts — 2+.

PlaceAgent or toolModels usedResolvedRun settingsCost per resolved issue
9OpenHandsOpenHandsUndisclosed65.8%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
14SWE-agentSWE-agentclaude-3-7-sonnet-2025021962.4%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
16OpenHandsOpenHands4x Scaled60.8%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
20Nemotron-CORTEXANVIDIANV-EmbedCode, claude-3-5-sonnet-20241022, deepseek-v3, o3-mini, gpt-4o-2024-05-13, gpt-4-turbo-2024-04-09, Qwen/Qwen2.5-72B-Instruct, meta-llama/Llama-3.1-405B-Instruct, meta-llama/Llama-3.3-70B-Instruct58.2%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
27OpenHandsOpenHandsCodeAct v2.1 (claude-3-5-sonnet-20241022)53%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
29Composio SWE-Kitclaude-3-5-sonnet-20241022, o1-mini48.6%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
30AppMap Navie v2claude-3-5-sonnet-20241022, gpt-4o-2024-08-0647.2%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
31Skywork-SWE-32BSkywork AIQwen2.5-Coder-32B-Instruct47%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
35Agentless LiteAgentlesso3-mini42.4%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
44AppMap Naviegpt-4o-2024-05-1326.2%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
55RAG baselineclaude-3-opus-202402297%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured
57RAG baselinegpt-4-1106-preview2.8%
  • attempts: 2+
  • checked by SWE-bench: yes
Not measured

  1. #9 OpenHands

    Models used
    Undisclosed
    Resolved
    65.8%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  2. #14 SWE-agent

    Models used
    claude-3-7-sonnet-20250219
    Resolved
    62.4%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  3. #16 OpenHands

    Models used
    4x Scaled
    Resolved
    60.8%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  4. #20 Nemotron-CORTEXA

    Models used
    NV-EmbedCode, claude-3-5-sonnet-20241022, deepseek-v3, o3-mini, gpt-4o-2024-05-13, gpt-4-turbo-2024-04-09, Qwen/Qwen2.5-72B-Instruct, meta-llama/Llama-3.1-405B-Instruct, meta-llama/Llama-3.3-70B-Instruct
    Resolved
    58.2%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  5. #27 OpenHands

    Models used
    CodeAct v2.1 (claude-3-5-sonnet-20241022)
    Resolved
    53%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  6. #29 Composio SWE-Kit

    Models used
    claude-3-5-sonnet-20241022, o1-mini
    Resolved
    48.6%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  7. #30 AppMap Navie v2

    Models used
    claude-3-5-sonnet-20241022, gpt-4o-2024-08-06
    Resolved
    47.2%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  8. #31 Skywork-SWE-32B

    Models used
    Qwen2.5-Coder-32B-Instruct
    Resolved
    47%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  9. #35 Agentless Lite

    Models used
    o3-mini
    Resolved
    42.4%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  10. #44 AppMap Navie

    Models used
    gpt-4o-2024-05-13
    Resolved
    26.2%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  11. #55 RAG baseline

    Models used
    claude-3-opus-20240229
    Resolved
    7%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

  12. #57 RAG baseline

    Models used
    gpt-4-1106-preview
    Resolved
    2.8%,
    Run settings
    attempts: 2+; checked by SWE-bench: yes
    Cost per resolved issue
    Not measured

Source: SWE-bench leaderboard. Each score links the published result it was read from.