Model comparisons

Aider polyglot (225 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind

63 results.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
1gpt-5 (high)88%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model openai/gpt-5
$0.147measured: $29.1 for the run, 198 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

2gpt-5 (medium)86.7%
  • edit format: diff
  • reasoning effort: medium
  • command: aider --model openai/gpt-5
$0.0907measured: $17.7 for the run, 195 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

3o3-pro (high)84.9%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model o3-pro
$0.766measured: $146.32 for the run, 191 of 225 solved, No price listedNone
4gemini-2.5-pro-preview-06-05 (32k think)83.1%
  • edit format: diff-fenced
  • thinking tokens: 32768
  • command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
$0.267measured: $49.9 for the run, 187 of 225 solved, No price listedNone
5o3 (high)81.3%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model o3 --reasoning-effort high
$0.116measured: $21.2 for the run, 183 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

5gpt-5 (low)81.3%
  • edit format: diff
  • reasoning effort: low
  • command: aider --model openai/gpt-5
$0.0567measured: $10.4 for the run, 183 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

7grok-4 (high)79.6%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model openrouter/x-ai/grok-4
$0.333measured: $59.6 for the run, 179 of 225 solved, No price listedNone
8gemini-2.5-pro-preview-06-05 (default think)79.1%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-06-05
$0.256measured: $45.6 for the run, 178 of 225 solved, No price listedNone
9Gemini 2.5 Pro Preview 05-0676.9%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-05-06
$0.216measured: $37.4 for the run, 173 of 225 solved, No price listedNone
9o376.9%
  • edit format: diff
  • command: aider --model o3
$0.0795measured: $13.8 for the run, 173 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

11DeepSeek-V3.2-Exp (Reasoner)74.2%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.00781measured: $1.3 for the run, 167 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

12Gemini 2.5 Pro Preview 03-2572.9%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-03-25
Not measuredNo price listedNone
13o4-mini (high)72%
  • edit format: diff
  • command: aider --model o4-mini
$0.121measured: $19.6 for the run, 162 of 225 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

13claude-opus-4-20250514 (32k thinking)72%
  • edit format: diff
  • thinking tokens: 32000
  • command: aider --model claude-opus-4-20250514
$0.406measured: $65.7 for the run, 162 of 225 solved, No price listedNone
15claude-opus-4-20250514 (no think)70.7%
  • edit format: diff
  • command: aider --model claude-opus-4-20250514
$0.432measured: $68.6 for the run, 159 of 225 solved, No price listedNone
16DeepSeek-V3.2-Exp (Chat)70.2%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.00554measured: $0.876 for the run, 158 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

17claude-3-7-sonnet-20250219 (32k thinking tokens)64.9%
  • edit format: diff
  • command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
$0.252measured: $36.8 for the run, 146 of 225 solved, No price listedNone
18DeepSeek R1 + claude-3-5-sonnet-2024102264%
  • edit format: architect
  • editor model: claude-3-5-sonnet-20241022
  • editor edit format: editor-diff
  • command: aider --architect --model r1 --editor-model sonnet
$0.0923measured: $13.3 for the run, 144 of 225 solved, No price listedNone
19claude-sonnet-4-20250514 (32k thinking)61.3%
  • edit format: diff
  • thinking tokens: 32000
  • command: aider --model claude-sonnet-4-20250514
$0.193measured: $26.6 for the run, 138 of 225 solved, No price listedNone
20claude-3-7-sonnet-20250219 (no thinking)60.4%
  • edit format: diff
  • command: aider --model sonnet
$0.13measured: $17.7 for the run, 136 of 225 solved, No price listedNone
21Qwen3 235B A22B diff, no think, Alibaba API59.6%
  • edit format: diff
  • command: aider --model openai/qwen3-235b-a22b
Not measuredNo price listedNone
22Kimi K259.1%
  • edit format: diff
  • command: aider --model openrouter/moonshotai/kimi-k2
$0.00929measured: $1.24 for the run, 133 of 225 solved, No price listedNone
23DeepSeek R156.9%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.0423measured: $5.42 for the run, 128 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

24claude-sonnet-4-20250514 (no thinking)56.4%
  • edit format: diff
  • command: aider --model claude-sonnet-4-20250514
$0.125measured: $15.8 for the run, 127 of 225 solved, No price listedNone
25DeepSeek V3 (0324)55.1%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.009measured: $1.12 for the run, 124 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

25gemini-2.5-flash-preview-05-20 (24k think)55.1%
  • edit format: diff
  • thinking tokens: 24576
  • command: aider --model gemini/gemini-2.5-flash-preview-05-20
$0.0691measured: $8.56 for the run, 124 of 225 solved, No price listedNone
27Quasar Alpha54.7%
  • edit format: diff
  • command: aider --model openrouter/openrouter/quasar-alpha
Not measuredNo price listedNone
28o3-mini (medium)53.8%
  • edit format: diff
  • command: aider --model o3-mini
$0.0732measured: $8.86 for the run, 121 of 225 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

29Grok 3 Beta53.3%
  • edit format: diff
  • command: aider --model openrouter/x-ai/grok-3-beta
$0.0919measured: $11 for the run, 120 of 225 solved, No price listedNone
30Optimus Alpha52.9%
  • edit format: diff
  • command: aider --model openrouter/openrouter/optimus-alpha
Not measuredNo price listedNone
31gpt-4.152.4%
  • edit format: diff
  • command: aider --model gpt-4.1
$0.0835measured: $9.86 for the run, 118 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

32claude-3-5-sonnet-2024102251.6%
  • edit format: diff
  • command: aider --model claude-3-5-sonnet-20241022
$0.124measured: $14.4 for the run, 116 of 225 solved, No price listedNone
33Grok 3 Mini Beta (high)49.3%
  • edit format: whole
  • command: aider --model xai/grok-3-mini-beta --reasoning-effort high
$0.00662measured: $0.735 for the run, 111 of 225 solved, No price listedNone
34DeepSeek Chat V3 (prev)48.4%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.00309measured: $0.337 for the run, 109 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

35gemini-2.5-flash-preview-04-17 (default)47.1%
  • edit format: diff
  • command: aider --model gemini/gemini-2.5-flash-preview-04-17
$0.0174measured: $1.85 for the run, 106 of 225 solved, No price listedNone
36chatgpt-4o-latest (2025-03-29)45.3%
  • edit format: diff
  • command: aider --model chatgpt-4o-latest
$0.194measured: $19.7 for the run, 102 of 225 solved, No price listedNone
37gemini-2.5-flash-preview-05-20 (no think)44%
  • edit format: diff
  • thinking tokens: 0
  • command: aider --model gemini/gemini-2.5-flash-preview-05-20
$0.0115measured: $1.14 for the run, 99 of 225 solved, No price listedNone
38gpt-oss-120b (high)41.8%
  • edit format: diff
  • command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
$0.00788measured: $0.741 for the run, 94 of 225 solved, No price listedNone
39Qwen3 32B40%
  • edit format: diff
  • command: aider --model openrouter/qwen/qwen3-32b
$0.00845measured: $0.76 for the run, 90 of 225 solved, No price listedNone
40gemini-exp-120638.2%
  • edit format: whole
  • command: aider --model gemini/gemini-exp-1206
Not measured

$0 in · $0 outgemini, USD, observed

$0 one coding-assistant request at geminiestimate from the price observed ; assumptions

$0 one chat reply at geminiestimate from the price observed ; assumptions

41Gemini 2.0 Pro exp-02-0535.6%
  • edit format: whole
  • command: aider --model gemini/gemini-2.0-pro-exp-02-05
Not measuredNo price listedNone
42Grok 3 Mini Beta (low)34.7%
  • edit format: whole
  • command: aider --model openrouter/x-ai/grok-3-mini-beta
$0.0101measured: $0.786 for the run, 78 of 225 solved, No price listedNone
43o1-mini-2024-09-1232.9%
  • edit format: whole
  • command: aider --model o1-mini
$0.251measured: $18.6 for the run, 74 of 225 solved, No price listedNone
44gpt-4.1-mini32.4%
  • edit format: diff
  • command: aider --model gpt-4.1-mini
$0.0273measured: $1.99 for the run, 73 of 225 solved,

$0.4 in · $1.6 outopenai, USD, observed

$0.0096 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0016 one chat reply at openaiestimate from the price observed ; assumptions

45claude-3-5-haiku-2024102228%
  • edit format: diff
  • command: aider --model claude-3-5-haiku-20241022
$0.0962measured: $6.06 for the run, 63 of 225 solved, No price listedNone
46QwQ-32B + Qwen 2.5 Coder Instruct26.2%
  • edit format: architect
  • editor model: fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct
  • editor edit format: editor-diff
  • command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b --architect
Not measured

$0.9 in · $0.9 outfireworks_ai, USD, observed

$0.0189 one coding-assistant request at fireworks_aiestimate from the price observed ; assumptions

$0.00225 one chat reply at fireworks_aiestimate from the price observed ; assumptions

47gpt-4o-2024-08-0623.1%
  • edit format: diff
  • command: aider --model gpt-4o-2024-08-06
$0.135measured: $7.03 for the run, 52 of 225 solved,

$2.5 in · $10 outopenai, USD, observed

$0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.01 one chat reply at openaiestimate from the price observed ; assumptions

48gemini-2.0-flash-exp22.2%
  • edit format: whole
  • command: aider --model gemini/gemini-2.0-flash-exp
Not measuredNo price listedNone
49qwen-max-2025-01-2521.8%
  • edit format: diff
  • command: OPENAI_API_BASE=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 aider --model openai/qwen-max-2025-01-25
Not measuredNo price listedNone
50QwQ-32B20.9%
  • edit format: diff
  • command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b
Not measured

$0.9 in · $0.9 outfireworks_ai, USD, observed

$0.0189 one coding-assistant request at fireworks_aiestimate from the price observed ; assumptions

$0.00225 one chat reply at fireworks_aiestimate from the price observed ; assumptions

51gpt-4o-2024-11-2018.2%
  • edit format: diff
  • command: aider --model gpt-4o-2024-11-20
$0.164measured: $6.74 for the run, 41 of 225 solved,

$2.5 in · $10 outopenai, USD, observed

$0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.01 one chat reply at openaiestimate from the price observed ; assumptions

51gemini-2.0-flash-thinking-exp-01-2118.2%
  • edit format: diff
  • command: aider --model gemini/gemini-2.0-flash-thinking-exp-01-21
Not measuredNo price listedNone
53DeepSeek Chat V2.517.8%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.0128measured: $0.51 for the run, 40 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

54Qwen2.5-Coder-32B-Instruct16.4%
  • edit format: whole
  • command: aider --model openai/Qwen2.5-Coder-32B-Instruct
Not measuredNo price listedNone
55Llama 4 Maverick15.6%
  • edit format: whole
  • command: aider --model nvidia_nim/meta/llama-4-maverick-17b-128e-instruct
Not measuredNo price listedNone
56yi-lightning12.9%
  • edit format: whole
  • command: aider --model openai/yi-lightning
Not measuredNo price listedNone
57command-a-03-2025-quality12%
  • edit format: whole
  • command: OPENAI_API_BASE=https://api.cohere.ai/compatibility/v1 aider --model openai/command-a-03-2025-quality
Not measuredNo price listedNone
58Codestral 25.0111.1%
  • edit format: whole
  • command: aider --model mistral/codestral-latest
$0.0793measured: $1.98 for the run, 25 of 225 solved,

$0.3 in · $0.9 outmistral, USD, observed

$0.0069 one coding-assistant request at mistralestimate from the price observed ; assumptions

$0.00105 one chat reply at mistralestimate from the price observed ; assumptions

59openhands-lm-32b-v0.110.2%
  • edit format: whole
  • command: aider --model openrouter/all-hands/openhands-lm-32b-v0.1
Not measuredNo price listedNone
60gpt-4.1-nano8.9%
  • edit format: whole
  • command: aider --model gpt-4.1-nano
$0.0214measured: $0.428 for the run, 20 of 225 solved,

$0.1 in · $0.4 outopenai, USD, observed

$0.0024 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0004 one chat reply at openaiestimate from the price observed ; assumptions

61Qwen2.5-Coder-32B-Instruct8%
  • edit format: diff
  • command: aider --model openai/Qwen/Qwen2.5-Coder-32B-Instruct # via hyperbolic
Not measuredNo price listedNone
62gemma-3-27b-it4.9%
  • edit format: whole
  • command: aider --model openrouter/google/gemma-3-27b-it
Not measuredNo price listedNone
63gpt-4o-mini-2024-07-183.6%
  • edit format: whole
  • command: aider --model gpt-4o-mini-2024-07-18
$0.0404measured: $0.324 for the run, 8 of 225 solved,

$0.15 in · $0.6 outopenai, USD, observed

$0.0036 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0006 one chat reply at openaiestimate from the price observed ; assumptions

  1. #1 gpt-5 (high)

    Pass rate, second attempt
    88%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model openai/gpt-5
    Cost per solved task
    $0.147 measured (198 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  2. #2 gpt-5 (medium)

    Pass rate, second attempt
    86.7%,
    Run settings
    edit format: diff; reasoning effort: medium; command: aider --model openai/gpt-5
    Cost per solved task
    $0.0907 measured (195 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  3. #3 o3-pro (high)

    Pass rate, second attempt
    84.9%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model o3-pro
    Cost per solved task
    $0.766 measured (191 of 225 solved),
    Price per 1M tokens
    No price listed

  4. #4 gemini-2.5-pro-preview-06-05 (32k think)

    Pass rate, second attempt
    83.1%,
    Run settings
    edit format: diff-fenced; thinking tokens: 32768; command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
    Cost per solved task
    $0.267 measured (187 of 225 solved),
    Price per 1M tokens
    No price listed

  5. #5 o3 (high)

    Pass rate, second attempt
    81.3%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model o3 --reasoning-effort high
    Cost per solved task
    $0.116 measured (183 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  6. #5 gpt-5 (low)

    Pass rate, second attempt
    81.3%,
    Run settings
    edit format: diff; reasoning effort: low; command: aider --model openai/gpt-5
    Cost per solved task
    $0.0567 measured (183 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  7. #7 grok-4 (high)

    Pass rate, second attempt
    79.6%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model openrouter/x-ai/grok-4
    Cost per solved task
    $0.333 measured (179 of 225 solved),
    Price per 1M tokens
    No price listed

  8. #8 gemini-2.5-pro-preview-06-05 (default think)

    Pass rate, second attempt
    79.1%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-06-05
    Cost per solved task
    $0.256 measured (178 of 225 solved),
    Price per 1M tokens
    No price listed

  9. #9 Gemini 2.5 Pro Preview 05-06

    Pass rate, second attempt
    76.9%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-05-06
    Cost per solved task
    $0.216 measured (173 of 225 solved),
    Price per 1M tokens
    No price listed

  10. #9 o3

    Pass rate, second attempt
    76.9%,
    Run settings
    edit format: diff; command: aider --model o3
    Cost per solved task
    $0.0795 measured (173 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  11. #11 DeepSeek-V3.2-Exp (Reasoner)

    Pass rate, second attempt
    74.2%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.00781 measured (167 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  12. #12 Gemini 2.5 Pro Preview 03-25

    Pass rate, second attempt
    72.9%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-03-25
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  13. #13 o4-mini (high)

    Pass rate, second attempt
    72%,
    Run settings
    edit format: diff; command: aider --model o4-mini
    Cost per solved task
    $0.121 measured (162 of 225 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  14. #13 claude-opus-4-20250514 (32k thinking)

    Pass rate, second attempt
    72%,
    Run settings
    edit format: diff; thinking tokens: 32000; command: aider --model claude-opus-4-20250514
    Cost per solved task
    $0.406 measured (162 of 225 solved),
    Price per 1M tokens
    No price listed

  15. #15 claude-opus-4-20250514 (no think)

    Pass rate, second attempt
    70.7%,
    Run settings
    edit format: diff; command: aider --model claude-opus-4-20250514
    Cost per solved task
    $0.432 measured (159 of 225 solved),
    Price per 1M tokens
    No price listed

  16. #16 DeepSeek-V3.2-Exp (Chat)

    Pass rate, second attempt
    70.2%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.00554 measured (158 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  17. #17 claude-3-7-sonnet-20250219 (32k thinking tokens)

    Pass rate, second attempt
    64.9%,
    Run settings
    edit format: diff; command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
    Cost per solved task
    $0.252 measured (146 of 225 solved),
    Price per 1M tokens
    No price listed

  18. #18 DeepSeek R1 + claude-3-5-sonnet-20241022

    Pass rate, second attempt
    64%,
    Run settings
    edit format: architect; editor model: claude-3-5-sonnet-20241022; editor edit format: editor-diff; command: aider --architect --model r1 --editor-model sonnet
    Cost per solved task
    $0.0923 measured (144 of 225 solved),
    Price per 1M tokens
    No price listed

  19. #19 claude-sonnet-4-20250514 (32k thinking)

    Pass rate, second attempt
    61.3%,
    Run settings
    edit format: diff; thinking tokens: 32000; command: aider --model claude-sonnet-4-20250514
    Cost per solved task
    $0.193 measured (138 of 225 solved),
    Price per 1M tokens
    No price listed

  20. #20 claude-3-7-sonnet-20250219 (no thinking)

    Pass rate, second attempt
    60.4%,
    Run settings
    edit format: diff; command: aider --model sonnet
    Cost per solved task
    $0.13 measured (136 of 225 solved),
    Price per 1M tokens
    No price listed

  21. #21 Qwen3 235B A22B diff, no think, Alibaba API

    Pass rate, second attempt
    59.6%,
    Run settings
    edit format: diff; command: aider --model openai/qwen3-235b-a22b
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  22. #22 Kimi K2

    Pass rate, second attempt
    59.1%,
    Run settings
    edit format: diff; command: aider --model openrouter/moonshotai/kimi-k2
    Cost per solved task
    $0.00929 measured (133 of 225 solved),
    Price per 1M tokens
    No price listed

  23. #23 DeepSeek R1

    Pass rate, second attempt
    56.9%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.0423 measured (128 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  24. #24 claude-sonnet-4-20250514 (no thinking)

    Pass rate, second attempt
    56.4%,
    Run settings
    edit format: diff; command: aider --model claude-sonnet-4-20250514
    Cost per solved task
    $0.125 measured (127 of 225 solved),
    Price per 1M tokens
    No price listed

  25. #25 DeepSeek V3 (0324)

    Pass rate, second attempt
    55.1%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.009 measured (124 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  26. #25 gemini-2.5-flash-preview-05-20 (24k think)

    Pass rate, second attempt
    55.1%,
    Run settings
    edit format: diff; thinking tokens: 24576; command: aider --model gemini/gemini-2.5-flash-preview-05-20
    Cost per solved task
    $0.0691 measured (124 of 225 solved),
    Price per 1M tokens
    No price listed

  27. #27 Quasar Alpha

    Pass rate, second attempt
    54.7%,
    Run settings
    edit format: diff; command: aider --model openrouter/openrouter/quasar-alpha
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  28. #28 o3-mini (medium)

    Pass rate, second attempt
    53.8%,
    Run settings
    edit format: diff; command: aider --model o3-mini
    Cost per solved task
    $0.0732 measured (121 of 225 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  29. #29 Grok 3 Beta

    Pass rate, second attempt
    53.3%,
    Run settings
    edit format: diff; command: aider --model openrouter/x-ai/grok-3-beta
    Cost per solved task
    $0.0919 measured (120 of 225 solved),
    Price per 1M tokens
    No price listed

  30. #30 Optimus Alpha

    Pass rate, second attempt
    52.9%,
    Run settings
    edit format: diff; command: aider --model openrouter/openrouter/optimus-alpha
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  31. #31 gpt-4.1

    Pass rate, second attempt
    52.4%,
    Run settings
    edit format: diff; command: aider --model gpt-4.1
    Cost per solved task
    $0.0835 measured (118 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  32. #32 claude-3-5-sonnet-20241022

    Pass rate, second attempt
    51.6%,
    Run settings
    edit format: diff; command: aider --model claude-3-5-sonnet-20241022
    Cost per solved task
    $0.124 measured (116 of 225 solved),
    Price per 1M tokens
    No price listed

  33. #33 Grok 3 Mini Beta (high)

    Pass rate, second attempt
    49.3%,
    Run settings
    edit format: whole; command: aider --model xai/grok-3-mini-beta --reasoning-effort high
    Cost per solved task
    $0.00662 measured (111 of 225 solved),
    Price per 1M tokens
    No price listed

  34. #34 DeepSeek Chat V3 (prev)

    Pass rate, second attempt
    48.4%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.00309 measured (109 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  35. #35 gemini-2.5-flash-preview-04-17 (default)

    Pass rate, second attempt
    47.1%,
    Run settings
    edit format: diff; command: aider --model gemini/gemini-2.5-flash-preview-04-17
    Cost per solved task
    $0.0174 measured (106 of 225 solved),
    Price per 1M tokens
    No price listed

  36. #36 chatgpt-4o-latest (2025-03-29)

    Pass rate, second attempt
    45.3%,
    Run settings
    edit format: diff; command: aider --model chatgpt-4o-latest
    Cost per solved task
    $0.194 measured (102 of 225 solved),
    Price per 1M tokens
    No price listed

  37. #37 gemini-2.5-flash-preview-05-20 (no think)

    Pass rate, second attempt
    44%,
    Run settings
    edit format: diff; thinking tokens: 0; command: aider --model gemini/gemini-2.5-flash-preview-05-20
    Cost per solved task
    $0.0115 measured (99 of 225 solved),
    Price per 1M tokens
    No price listed

  38. #38 gpt-oss-120b (high)

    Pass rate, second attempt
    41.8%,
    Run settings
    edit format: diff; command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
    Cost per solved task
    $0.00788 measured (94 of 225 solved),
    Price per 1M tokens
    No price listed

  39. #39 Qwen3 32B

    Pass rate, second attempt
    40%,
    Run settings
    edit format: diff; command: aider --model openrouter/qwen/qwen3-32b
    Cost per solved task
    $0.00845 measured (90 of 225 solved),
    Price per 1M tokens
    No price listed

  40. #40 gemini-exp-1206

    Pass rate, second attempt
    38.2%,
    Run settings
    edit format: whole; command: aider --model gemini/gemini-exp-1206
    Cost per solved task
    Not measured
    Price per 1M tokens

    $0 in · $0 out, gemini, observed

    Workload estimate

    $0 one coding-assistant request at gemini, from the price observed (assumptions)

    $0 one chat reply at gemini, from the price observed (assumptions)

  41. #41 Gemini 2.0 Pro exp-02-05

    Pass rate, second attempt
    35.6%,
    Run settings
    edit format: whole; command: aider --model gemini/gemini-2.0-pro-exp-02-05
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  42. #42 Grok 3 Mini Beta (low)

    Pass rate, second attempt
    34.7%,
    Run settings
    edit format: whole; command: aider --model openrouter/x-ai/grok-3-mini-beta
    Cost per solved task
    $0.0101 measured (78 of 225 solved),
    Price per 1M tokens
    No price listed

  43. #43 o1-mini-2024-09-12

    Pass rate, second attempt
    32.9%,
    Run settings
    edit format: whole; command: aider --model o1-mini
    Cost per solved task
    $0.251 measured (74 of 225 solved),
    Price per 1M tokens
    No price listed

  44. #44 gpt-4.1-mini

    Pass rate, second attempt
    32.4%,
    Run settings
    edit format: diff; command: aider --model gpt-4.1-mini
    Cost per solved task
    $0.0273 measured (73 of 225 solved),
    Price per 1M tokens

    $0.4 in · $1.6 out, openai, observed

    Workload estimate

    $0.0096 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0016 one chat reply at openai, from the price observed (assumptions)

  45. #45 claude-3-5-haiku-20241022

    Pass rate, second attempt
    28%,
    Run settings
    edit format: diff; command: aider --model claude-3-5-haiku-20241022
    Cost per solved task
    $0.0962 measured (63 of 225 solved),
    Price per 1M tokens
    No price listed

  46. #46 QwQ-32B + Qwen 2.5 Coder Instruct

    Pass rate, second attempt
    26.2%,
    Run settings
    edit format: architect; editor model: fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct; editor edit format: editor-diff; command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b --architect
    Cost per solved task
    Not measured
    Price per 1M tokens

    $0.9 in · $0.9 out, fireworks_ai, observed

    Workload estimate

    $0.0189 one coding-assistant request at fireworks_ai, from the price observed (assumptions)

    $0.00225 one chat reply at fireworks_ai, from the price observed (assumptions)

  47. #47 gpt-4o-2024-08-06

    Pass rate, second attempt
    23.1%,
    Run settings
    edit format: diff; command: aider --model gpt-4o-2024-08-06
    Cost per solved task
    $0.135 measured (52 of 225 solved),
    Price per 1M tokens

    $2.5 in · $10 out, openai, observed

    Workload estimate

    $0.06 one coding-assistant request at openai, from the price observed (assumptions)

    $0.01 one chat reply at openai, from the price observed (assumptions)

  48. #48 gemini-2.0-flash-exp

    Pass rate, second attempt
    22.2%,
    Run settings
    edit format: whole; command: aider --model gemini/gemini-2.0-flash-exp
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  49. #49 qwen-max-2025-01-25

    Pass rate, second attempt
    21.8%,
    Run settings
    edit format: diff; command: OPENAI_API_BASE=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 aider --model openai/qwen-max-2025-01-25
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  50. #50 QwQ-32B

    Pass rate, second attempt
    20.9%,
    Run settings
    edit format: diff; command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b
    Cost per solved task
    Not measured
    Price per 1M tokens

    $0.9 in · $0.9 out, fireworks_ai, observed

    Workload estimate

    $0.0189 one coding-assistant request at fireworks_ai, from the price observed (assumptions)

    $0.00225 one chat reply at fireworks_ai, from the price observed (assumptions)

  51. #51 gpt-4o-2024-11-20

    Pass rate, second attempt
    18.2%,
    Run settings
    edit format: diff; command: aider --model gpt-4o-2024-11-20
    Cost per solved task
    $0.164 measured (41 of 225 solved),
    Price per 1M tokens

    $2.5 in · $10 out, openai, observed

    Workload estimate

    $0.06 one coding-assistant request at openai, from the price observed (assumptions)

    $0.01 one chat reply at openai, from the price observed (assumptions)

  52. #51 gemini-2.0-flash-thinking-exp-01-21

    Pass rate, second attempt
    18.2%,
    Run settings
    edit format: diff; command: aider --model gemini/gemini-2.0-flash-thinking-exp-01-21
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  53. #53 DeepSeek Chat V2.5

    Pass rate, second attempt
    17.8%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.0128 measured (40 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  54. #54 Qwen2.5-Coder-32B-Instruct

    Pass rate, second attempt
    16.4%,
    Run settings
    edit format: whole; command: aider --model openai/Qwen2.5-Coder-32B-Instruct
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  55. #55 Llama 4 Maverick

    Pass rate, second attempt
    15.6%,
    Run settings
    edit format: whole; command: aider --model nvidia_nim/meta/llama-4-maverick-17b-128e-instruct
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  56. #56 yi-lightning

    Pass rate, second attempt
    12.9%,
    Run settings
    edit format: whole; command: aider --model openai/yi-lightning
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  57. #57 command-a-03-2025-quality

    Pass rate, second attempt
    12%,
    Run settings
    edit format: whole; command: OPENAI_API_BASE=https://api.cohere.ai/compatibility/v1 aider --model openai/command-a-03-2025-quality
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  58. #58 Codestral 25.01

    Pass rate, second attempt
    11.1%,
    Run settings
    edit format: whole; command: aider --model mistral/codestral-latest
    Cost per solved task
    $0.0793 measured (25 of 225 solved),
    Price per 1M tokens

    $0.3 in · $0.9 out, mistral, observed

    Workload estimate

    $0.0069 one coding-assistant request at mistral, from the price observed (assumptions)

    $0.00105 one chat reply at mistral, from the price observed (assumptions)

  59. #59 openhands-lm-32b-v0.1

    Pass rate, second attempt
    10.2%,
    Run settings
    edit format: whole; command: aider --model openrouter/all-hands/openhands-lm-32b-v0.1
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  60. #60 gpt-4.1-nano

    Pass rate, second attempt
    8.9%,
    Run settings
    edit format: whole; command: aider --model gpt-4.1-nano
    Cost per solved task
    $0.0214 measured (20 of 225 solved),
    Price per 1M tokens

    $0.1 in · $0.4 out, openai, observed

    Workload estimate

    $0.0024 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0004 one chat reply at openai, from the price observed (assumptions)

  61. #61 Qwen2.5-Coder-32B-Instruct

    Pass rate, second attempt
    8%,
    Run settings
    edit format: diff; command: aider --model openai/Qwen/Qwen2.5-Coder-32B-Instruct # via hyperbolic
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  62. #62 gemma-3-27b-it

    Pass rate, second attempt
    4.9%,
    Run settings
    edit format: whole; command: aider --model openrouter/google/gemma-3-27b-it
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  63. #63 gpt-4o-mini-2024-07-18

    Pass rate, second attempt
    3.6%,
    Run settings
    edit format: whole; command: aider --model gpt-4o-mini-2024-07-18
    Cost per solved task
    $0.0404 measured (8 of 225 solved),
    Price per 1M tokens

    $0.15 in · $0.6 out, openai, observed

    Workload estimate

    $0.0036 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0006 one chat reply at openai, from the price observed (assumptions)

Source: Aider polyglot leaderboard. Each score links the published result it was read from.