Model comparisons

Aider polyglot (225 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind

42 of 63 results: edit format — diff.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
1gpt-5 (high)88%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model openai/gpt-5
$0.147measured: $29.1 for the run, 198 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

2gpt-5 (medium)86.7%
  • edit format: diff
  • reasoning effort: medium
  • command: aider --model openai/gpt-5
$0.0907measured: $17.7 for the run, 195 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

3o3-pro (high)84.9%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model o3-pro
$0.766measured: $146.32 for the run, 191 of 225 solved, No price listedNone
5o3 (high)81.3%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model o3 --reasoning-effort high
$0.116measured: $21.2 for the run, 183 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

5gpt-5 (low)81.3%
  • edit format: diff
  • reasoning effort: low
  • command: aider --model openai/gpt-5
$0.0567measured: $10.4 for the run, 183 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

7grok-4 (high)79.6%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model openrouter/x-ai/grok-4
$0.333measured: $59.6 for the run, 179 of 225 solved, No price listedNone
9o376.9%
  • edit format: diff
  • command: aider --model o3
$0.0795measured: $13.8 for the run, 173 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

11DeepSeek-V3.2-Exp (Reasoner)74.2%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.00781measured: $1.3 for the run, 167 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

13o4-mini (high)72%
  • edit format: diff
  • command: aider --model o4-mini
$0.121measured: $19.6 for the run, 162 of 225 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

13claude-opus-4-20250514 (32k thinking)72%
  • edit format: diff
  • thinking tokens: 32000
  • command: aider --model claude-opus-4-20250514
$0.406measured: $65.7 for the run, 162 of 225 solved, No price listedNone
15claude-opus-4-20250514 (no think)70.7%
  • edit format: diff
  • command: aider --model claude-opus-4-20250514
$0.432measured: $68.6 for the run, 159 of 225 solved, No price listedNone
16DeepSeek-V3.2-Exp (Chat)70.2%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.00554measured: $0.876 for the run, 158 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

17claude-3-7-sonnet-20250219 (32k thinking tokens)64.9%
  • edit format: diff
  • command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
$0.252measured: $36.8 for the run, 146 of 225 solved, No price listedNone
19claude-sonnet-4-20250514 (32k thinking)61.3%
  • edit format: diff
  • thinking tokens: 32000
  • command: aider --model claude-sonnet-4-20250514
$0.193measured: $26.6 for the run, 138 of 225 solved, No price listedNone
20claude-3-7-sonnet-20250219 (no thinking)60.4%
  • edit format: diff
  • command: aider --model sonnet
$0.13measured: $17.7 for the run, 136 of 225 solved, No price listedNone
21Qwen3 235B A22B diff, no think, Alibaba API59.6%
  • edit format: diff
  • command: aider --model openai/qwen3-235b-a22b
Not measuredNo price listedNone
22Kimi K259.1%
  • edit format: diff
  • command: aider --model openrouter/moonshotai/kimi-k2
$0.00929measured: $1.24 for the run, 133 of 225 solved, No price listedNone
23DeepSeek R156.9%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.0423measured: $5.42 for the run, 128 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

24claude-sonnet-4-20250514 (no thinking)56.4%
  • edit format: diff
  • command: aider --model claude-sonnet-4-20250514
$0.125measured: $15.8 for the run, 127 of 225 solved, No price listedNone
25DeepSeek V3 (0324)55.1%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.009measured: $1.12 for the run, 124 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

25gemini-2.5-flash-preview-05-20 (24k think)55.1%
  • edit format: diff
  • thinking tokens: 24576
  • command: aider --model gemini/gemini-2.5-flash-preview-05-20
$0.0691measured: $8.56 for the run, 124 of 225 solved, No price listedNone
27Quasar Alpha54.7%
  • edit format: diff
  • command: aider --model openrouter/openrouter/quasar-alpha
Not measuredNo price listedNone
28o3-mini (medium)53.8%
  • edit format: diff
  • command: aider --model o3-mini
$0.0732measured: $8.86 for the run, 121 of 225 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

29Grok 3 Beta53.3%
  • edit format: diff
  • command: aider --model openrouter/x-ai/grok-3-beta
$0.0919measured: $11 for the run, 120 of 225 solved, No price listedNone
30Optimus Alpha52.9%
  • edit format: diff
  • command: aider --model openrouter/openrouter/optimus-alpha
Not measuredNo price listedNone
31gpt-4.152.4%
  • edit format: diff
  • command: aider --model gpt-4.1
$0.0835measured: $9.86 for the run, 118 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

32claude-3-5-sonnet-2024102251.6%
  • edit format: diff
  • command: aider --model claude-3-5-sonnet-20241022
$0.124measured: $14.4 for the run, 116 of 225 solved, No price listedNone
34DeepSeek Chat V3 (prev)48.4%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.00309measured: $0.337 for the run, 109 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

35gemini-2.5-flash-preview-04-17 (default)47.1%
  • edit format: diff
  • command: aider --model gemini/gemini-2.5-flash-preview-04-17
$0.0174measured: $1.85 for the run, 106 of 225 solved, No price listedNone
36chatgpt-4o-latest (2025-03-29)45.3%
  • edit format: diff
  • command: aider --model chatgpt-4o-latest
$0.194measured: $19.7 for the run, 102 of 225 solved, No price listedNone
37gemini-2.5-flash-preview-05-20 (no think)44%
  • edit format: diff
  • thinking tokens: 0
  • command: aider --model gemini/gemini-2.5-flash-preview-05-20
$0.0115measured: $1.14 for the run, 99 of 225 solved, No price listedNone
38gpt-oss-120b (high)41.8%
  • edit format: diff
  • command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
$0.00788measured: $0.741 for the run, 94 of 225 solved, No price listedNone
39Qwen3 32B40%
  • edit format: diff
  • command: aider --model openrouter/qwen/qwen3-32b
$0.00845measured: $0.76 for the run, 90 of 225 solved, No price listedNone
44gpt-4.1-mini32.4%
  • edit format: diff
  • command: aider --model gpt-4.1-mini
$0.0273measured: $1.99 for the run, 73 of 225 solved,

$0.4 in · $1.6 outopenai, USD, observed

$0.0096 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0016 one chat reply at openaiestimate from the price observed ; assumptions

45claude-3-5-haiku-2024102228%
  • edit format: diff
  • command: aider --model claude-3-5-haiku-20241022
$0.0962measured: $6.06 for the run, 63 of 225 solved, No price listedNone
47gpt-4o-2024-08-0623.1%
  • edit format: diff
  • command: aider --model gpt-4o-2024-08-06
$0.135measured: $7.03 for the run, 52 of 225 solved,

$2.5 in · $10 outopenai, USD, observed

$0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.01 one chat reply at openaiestimate from the price observed ; assumptions

49qwen-max-2025-01-2521.8%
  • edit format: diff
  • command: OPENAI_API_BASE=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 aider --model openai/qwen-max-2025-01-25
Not measuredNo price listedNone
50QwQ-32B20.9%
  • edit format: diff
  • command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b
Not measured

$0.9 in · $0.9 outfireworks_ai, USD, observed

$0.0189 one coding-assistant request at fireworks_aiestimate from the price observed ; assumptions

$0.00225 one chat reply at fireworks_aiestimate from the price observed ; assumptions

51gpt-4o-2024-11-2018.2%
  • edit format: diff
  • command: aider --model gpt-4o-2024-11-20
$0.164measured: $6.74 for the run, 41 of 225 solved,

$2.5 in · $10 outopenai, USD, observed

$0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.01 one chat reply at openaiestimate from the price observed ; assumptions

51gemini-2.0-flash-thinking-exp-01-2118.2%
  • edit format: diff
  • command: aider --model gemini/gemini-2.0-flash-thinking-exp-01-21
Not measuredNo price listedNone
53DeepSeek Chat V2.517.8%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.0128measured: $0.51 for the run, 40 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

61Qwen2.5-Coder-32B-Instruct8%
  • edit format: diff
  • command: aider --model openai/Qwen/Qwen2.5-Coder-32B-Instruct # via hyperbolic
Not measuredNo price listedNone

  1. #1 gpt-5 (high)

    Pass rate, second attempt
    88%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model openai/gpt-5
    Cost per solved task
    $0.147 measured (198 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  2. #2 gpt-5 (medium)

    Pass rate, second attempt
    86.7%,
    Run settings
    edit format: diff; reasoning effort: medium; command: aider --model openai/gpt-5
    Cost per solved task
    $0.0907 measured (195 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  3. #3 o3-pro (high)

    Pass rate, second attempt
    84.9%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model o3-pro
    Cost per solved task
    $0.766 measured (191 of 225 solved),
    Price per 1M tokens
    No price listed

  4. #5 o3 (high)

    Pass rate, second attempt
    81.3%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model o3 --reasoning-effort high
    Cost per solved task
    $0.116 measured (183 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  5. #5 gpt-5 (low)

    Pass rate, second attempt
    81.3%,
    Run settings
    edit format: diff; reasoning effort: low; command: aider --model openai/gpt-5
    Cost per solved task
    $0.0567 measured (183 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  6. #7 grok-4 (high)

    Pass rate, second attempt
    79.6%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model openrouter/x-ai/grok-4
    Cost per solved task
    $0.333 measured (179 of 225 solved),
    Price per 1M tokens
    No price listed

  7. #9 o3

    Pass rate, second attempt
    76.9%,
    Run settings
    edit format: diff; command: aider --model o3
    Cost per solved task
    $0.0795 measured (173 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  8. #11 DeepSeek-V3.2-Exp (Reasoner)

    Pass rate, second attempt
    74.2%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.00781 measured (167 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  9. #13 o4-mini (high)

    Pass rate, second attempt
    72%,
    Run settings
    edit format: diff; command: aider --model o4-mini
    Cost per solved task
    $0.121 measured (162 of 225 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  10. #13 claude-opus-4-20250514 (32k thinking)

    Pass rate, second attempt
    72%,
    Run settings
    edit format: diff; thinking tokens: 32000; command: aider --model claude-opus-4-20250514
    Cost per solved task
    $0.406 measured (162 of 225 solved),
    Price per 1M tokens
    No price listed

  11. #15 claude-opus-4-20250514 (no think)

    Pass rate, second attempt
    70.7%,
    Run settings
    edit format: diff; command: aider --model claude-opus-4-20250514
    Cost per solved task
    $0.432 measured (159 of 225 solved),
    Price per 1M tokens
    No price listed

  12. #16 DeepSeek-V3.2-Exp (Chat)

    Pass rate, second attempt
    70.2%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.00554 measured (158 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  13. #17 claude-3-7-sonnet-20250219 (32k thinking tokens)

    Pass rate, second attempt
    64.9%,
    Run settings
    edit format: diff; command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
    Cost per solved task
    $0.252 measured (146 of 225 solved),
    Price per 1M tokens
    No price listed

  14. #19 claude-sonnet-4-20250514 (32k thinking)

    Pass rate, second attempt
    61.3%,
    Run settings
    edit format: diff; thinking tokens: 32000; command: aider --model claude-sonnet-4-20250514
    Cost per solved task
    $0.193 measured (138 of 225 solved),
    Price per 1M tokens
    No price listed

  15. #20 claude-3-7-sonnet-20250219 (no thinking)

    Pass rate, second attempt
    60.4%,
    Run settings
    edit format: diff; command: aider --model sonnet
    Cost per solved task
    $0.13 measured (136 of 225 solved),
    Price per 1M tokens
    No price listed

  16. #21 Qwen3 235B A22B diff, no think, Alibaba API

    Pass rate, second attempt
    59.6%,
    Run settings
    edit format: diff; command: aider --model openai/qwen3-235b-a22b
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  17. #22 Kimi K2

    Pass rate, second attempt
    59.1%,
    Run settings
    edit format: diff; command: aider --model openrouter/moonshotai/kimi-k2
    Cost per solved task
    $0.00929 measured (133 of 225 solved),
    Price per 1M tokens
    No price listed

  18. #23 DeepSeek R1

    Pass rate, second attempt
    56.9%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.0423 measured (128 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  19. #24 claude-sonnet-4-20250514 (no thinking)

    Pass rate, second attempt
    56.4%,
    Run settings
    edit format: diff; command: aider --model claude-sonnet-4-20250514
    Cost per solved task
    $0.125 measured (127 of 225 solved),
    Price per 1M tokens
    No price listed

  20. #25 DeepSeek V3 (0324)

    Pass rate, second attempt
    55.1%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.009 measured (124 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  21. #25 gemini-2.5-flash-preview-05-20 (24k think)

    Pass rate, second attempt
    55.1%,
    Run settings
    edit format: diff; thinking tokens: 24576; command: aider --model gemini/gemini-2.5-flash-preview-05-20
    Cost per solved task
    $0.0691 measured (124 of 225 solved),
    Price per 1M tokens
    No price listed

  22. #27 Quasar Alpha

    Pass rate, second attempt
    54.7%,
    Run settings
    edit format: diff; command: aider --model openrouter/openrouter/quasar-alpha
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  23. #28 o3-mini (medium)

    Pass rate, second attempt
    53.8%,
    Run settings
    edit format: diff; command: aider --model o3-mini
    Cost per solved task
    $0.0732 measured (121 of 225 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  24. #29 Grok 3 Beta

    Pass rate, second attempt
    53.3%,
    Run settings
    edit format: diff; command: aider --model openrouter/x-ai/grok-3-beta
    Cost per solved task
    $0.0919 measured (120 of 225 solved),
    Price per 1M tokens
    No price listed

  25. #30 Optimus Alpha

    Pass rate, second attempt
    52.9%,
    Run settings
    edit format: diff; command: aider --model openrouter/openrouter/optimus-alpha
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  26. #31 gpt-4.1

    Pass rate, second attempt
    52.4%,
    Run settings
    edit format: diff; command: aider --model gpt-4.1
    Cost per solved task
    $0.0835 measured (118 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  27. #32 claude-3-5-sonnet-20241022

    Pass rate, second attempt
    51.6%,
    Run settings
    edit format: diff; command: aider --model claude-3-5-sonnet-20241022
    Cost per solved task
    $0.124 measured (116 of 225 solved),
    Price per 1M tokens
    No price listed

  28. #34 DeepSeek Chat V3 (prev)

    Pass rate, second attempt
    48.4%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.00309 measured (109 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  29. #35 gemini-2.5-flash-preview-04-17 (default)

    Pass rate, second attempt
    47.1%,
    Run settings
    edit format: diff; command: aider --model gemini/gemini-2.5-flash-preview-04-17
    Cost per solved task
    $0.0174 measured (106 of 225 solved),
    Price per 1M tokens
    No price listed

  30. #36 chatgpt-4o-latest (2025-03-29)

    Pass rate, second attempt
    45.3%,
    Run settings
    edit format: diff; command: aider --model chatgpt-4o-latest
    Cost per solved task
    $0.194 measured (102 of 225 solved),
    Price per 1M tokens
    No price listed

  31. #37 gemini-2.5-flash-preview-05-20 (no think)

    Pass rate, second attempt
    44%,
    Run settings
    edit format: diff; thinking tokens: 0; command: aider --model gemini/gemini-2.5-flash-preview-05-20
    Cost per solved task
    $0.0115 measured (99 of 225 solved),
    Price per 1M tokens
    No price listed

  32. #38 gpt-oss-120b (high)

    Pass rate, second attempt
    41.8%,
    Run settings
    edit format: diff; command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
    Cost per solved task
    $0.00788 measured (94 of 225 solved),
    Price per 1M tokens
    No price listed

  33. #39 Qwen3 32B

    Pass rate, second attempt
    40%,
    Run settings
    edit format: diff; command: aider --model openrouter/qwen/qwen3-32b
    Cost per solved task
    $0.00845 measured (90 of 225 solved),
    Price per 1M tokens
    No price listed

  34. #44 gpt-4.1-mini

    Pass rate, second attempt
    32.4%,
    Run settings
    edit format: diff; command: aider --model gpt-4.1-mini
    Cost per solved task
    $0.0273 measured (73 of 225 solved),
    Price per 1M tokens

    $0.4 in · $1.6 out, openai, observed

    Workload estimate

    $0.0096 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0016 one chat reply at openai, from the price observed (assumptions)

  35. #45 claude-3-5-haiku-20241022

    Pass rate, second attempt
    28%,
    Run settings
    edit format: diff; command: aider --model claude-3-5-haiku-20241022
    Cost per solved task
    $0.0962 measured (63 of 225 solved),
    Price per 1M tokens
    No price listed

  36. #47 gpt-4o-2024-08-06

    Pass rate, second attempt
    23.1%,
    Run settings
    edit format: diff; command: aider --model gpt-4o-2024-08-06
    Cost per solved task
    $0.135 measured (52 of 225 solved),
    Price per 1M tokens

    $2.5 in · $10 out, openai, observed

    Workload estimate

    $0.06 one coding-assistant request at openai, from the price observed (assumptions)

    $0.01 one chat reply at openai, from the price observed (assumptions)

  37. #49 qwen-max-2025-01-25

    Pass rate, second attempt
    21.8%,
    Run settings
    edit format: diff; command: OPENAI_API_BASE=https://dashscope-intl.aliyuncs.com/compatible-mode/v1 aider --model openai/qwen-max-2025-01-25
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  38. #50 QwQ-32B

    Pass rate, second attempt
    20.9%,
    Run settings
    edit format: diff; command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b
    Cost per solved task
    Not measured
    Price per 1M tokens

    $0.9 in · $0.9 out, fireworks_ai, observed

    Workload estimate

    $0.0189 one coding-assistant request at fireworks_ai, from the price observed (assumptions)

    $0.00225 one chat reply at fireworks_ai, from the price observed (assumptions)

  39. #51 gpt-4o-2024-11-20

    Pass rate, second attempt
    18.2%,
    Run settings
    edit format: diff; command: aider --model gpt-4o-2024-11-20
    Cost per solved task
    $0.164 measured (41 of 225 solved),
    Price per 1M tokens

    $2.5 in · $10 out, openai, observed

    Workload estimate

    $0.06 one coding-assistant request at openai, from the price observed (assumptions)

    $0.01 one chat reply at openai, from the price observed (assumptions)

  40. #51 gemini-2.0-flash-thinking-exp-01-21

    Pass rate, second attempt
    18.2%,
    Run settings
    edit format: diff; command: aider --model gemini/gemini-2.0-flash-thinking-exp-01-21
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

  41. #53 DeepSeek Chat V2.5

    Pass rate, second attempt
    17.8%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.0128 measured (40 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  42. #61 Qwen2.5-Coder-32B-Instruct

    Pass rate, second attempt
    8%,
    Run settings
    edit format: diff; command: aider --model openai/Qwen/Qwen2.5-Coder-32B-Instruct # via hyperbolic
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

Source: Aider polyglot leaderboard. Each score links the published result it was read from.