Model comparisons

Aider polyglot (225 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind

45 of 63 results: cost — with a measured cost per solved task.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
1gpt-5 (high)88%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model openai/gpt-5
$0.147measured: $29.1 for the run, 198 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

2gpt-5 (medium)86.7%
  • edit format: diff
  • reasoning effort: medium
  • command: aider --model openai/gpt-5
$0.0907measured: $17.7 for the run, 195 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

3o3-pro (high)84.9%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model o3-pro
$0.766measured: $146.32 for the run, 191 of 225 solved, No price listedNone
4gemini-2.5-pro-preview-06-05 (32k think)83.1%
  • edit format: diff-fenced
  • thinking tokens: 32768
  • command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
$0.267measured: $49.9 for the run, 187 of 225 solved, No price listedNone
5o3 (high)81.3%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model o3 --reasoning-effort high
$0.116measured: $21.2 for the run, 183 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

5gpt-5 (low)81.3%
  • edit format: diff
  • reasoning effort: low
  • command: aider --model openai/gpt-5
$0.0567measured: $10.4 for the run, 183 of 225 solved,

$1.25 in · $10 outopenai, USD, observed

$0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0075 one chat reply at openaiestimate from the price observed ; assumptions

7grok-4 (high)79.6%
  • edit format: diff
  • reasoning effort: high
  • command: aider --model openrouter/x-ai/grok-4
$0.333measured: $59.6 for the run, 179 of 225 solved, No price listedNone
8gemini-2.5-pro-preview-06-05 (default think)79.1%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-06-05
$0.256measured: $45.6 for the run, 178 of 225 solved, No price listedNone
9Gemini 2.5 Pro Preview 05-0676.9%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-05-06
$0.216measured: $37.4 for the run, 173 of 225 solved, No price listedNone
9o376.9%
  • edit format: diff
  • command: aider --model o3
$0.0795measured: $13.8 for the run, 173 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

11DeepSeek-V3.2-Exp (Reasoner)74.2%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.00781measured: $1.3 for the run, 167 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

13o4-mini (high)72%
  • edit format: diff
  • command: aider --model o4-mini
$0.121measured: $19.6 for the run, 162 of 225 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

13claude-opus-4-20250514 (32k thinking)72%
  • edit format: diff
  • thinking tokens: 32000
  • command: aider --model claude-opus-4-20250514
$0.406measured: $65.7 for the run, 162 of 225 solved, No price listedNone
15claude-opus-4-20250514 (no think)70.7%
  • edit format: diff
  • command: aider --model claude-opus-4-20250514
$0.432measured: $68.6 for the run, 159 of 225 solved, No price listedNone
16DeepSeek-V3.2-Exp (Chat)70.2%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.00554measured: $0.876 for the run, 158 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

17claude-3-7-sonnet-20250219 (32k thinking tokens)64.9%
  • edit format: diff
  • command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
$0.252measured: $36.8 for the run, 146 of 225 solved, No price listedNone
18DeepSeek R1 + claude-3-5-sonnet-2024102264%
  • edit format: architect
  • editor model: claude-3-5-sonnet-20241022
  • editor edit format: editor-diff
  • command: aider --architect --model r1 --editor-model sonnet
$0.0923measured: $13.3 for the run, 144 of 225 solved, No price listedNone
19claude-sonnet-4-20250514 (32k thinking)61.3%
  • edit format: diff
  • thinking tokens: 32000
  • command: aider --model claude-sonnet-4-20250514
$0.193measured: $26.6 for the run, 138 of 225 solved, No price listedNone
20claude-3-7-sonnet-20250219 (no thinking)60.4%
  • edit format: diff
  • command: aider --model sonnet
$0.13measured: $17.7 for the run, 136 of 225 solved, No price listedNone
22Kimi K259.1%
  • edit format: diff
  • command: aider --model openrouter/moonshotai/kimi-k2
$0.00929measured: $1.24 for the run, 133 of 225 solved, No price listedNone
23DeepSeek R156.9%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.0423measured: $5.42 for the run, 128 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

24claude-sonnet-4-20250514 (no thinking)56.4%
  • edit format: diff
  • command: aider --model claude-sonnet-4-20250514
$0.125measured: $15.8 for the run, 127 of 225 solved, No price listedNone
25DeepSeek V3 (0324)55.1%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.009measured: $1.12 for the run, 124 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

25gemini-2.5-flash-preview-05-20 (24k think)55.1%
  • edit format: diff
  • thinking tokens: 24576
  • command: aider --model gemini/gemini-2.5-flash-preview-05-20
$0.0691measured: $8.56 for the run, 124 of 225 solved, No price listedNone
28o3-mini (medium)53.8%
  • edit format: diff
  • command: aider --model o3-mini
$0.0732measured: $8.86 for the run, 121 of 225 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

29Grok 3 Beta53.3%
  • edit format: diff
  • command: aider --model openrouter/x-ai/grok-3-beta
$0.0919measured: $11 for the run, 120 of 225 solved, No price listedNone
31gpt-4.152.4%
  • edit format: diff
  • command: aider --model gpt-4.1
$0.0835measured: $9.86 for the run, 118 of 225 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

32claude-3-5-sonnet-2024102251.6%
  • edit format: diff
  • command: aider --model claude-3-5-sonnet-20241022
$0.124measured: $14.4 for the run, 116 of 225 solved, No price listedNone
33Grok 3 Mini Beta (high)49.3%
  • edit format: whole
  • command: aider --model xai/grok-3-mini-beta --reasoning-effort high
$0.00662measured: $0.735 for the run, 111 of 225 solved, No price listedNone
34DeepSeek Chat V3 (prev)48.4%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.00309measured: $0.337 for the run, 109 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

35gemini-2.5-flash-preview-04-17 (default)47.1%
  • edit format: diff
  • command: aider --model gemini/gemini-2.5-flash-preview-04-17
$0.0174measured: $1.85 for the run, 106 of 225 solved, No price listedNone
36chatgpt-4o-latest (2025-03-29)45.3%
  • edit format: diff
  • command: aider --model chatgpt-4o-latest
$0.194measured: $19.7 for the run, 102 of 225 solved, No price listedNone
37gemini-2.5-flash-preview-05-20 (no think)44%
  • edit format: diff
  • thinking tokens: 0
  • command: aider --model gemini/gemini-2.5-flash-preview-05-20
$0.0115measured: $1.14 for the run, 99 of 225 solved, No price listedNone
38gpt-oss-120b (high)41.8%
  • edit format: diff
  • command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
$0.00788measured: $0.741 for the run, 94 of 225 solved, No price listedNone
39Qwen3 32B40%
  • edit format: diff
  • command: aider --model openrouter/qwen/qwen3-32b
$0.00845measured: $0.76 for the run, 90 of 225 solved, No price listedNone
42Grok 3 Mini Beta (low)34.7%
  • edit format: whole
  • command: aider --model openrouter/x-ai/grok-3-mini-beta
$0.0101measured: $0.786 for the run, 78 of 225 solved, No price listedNone
43o1-mini-2024-09-1232.9%
  • edit format: whole
  • command: aider --model o1-mini
$0.251measured: $18.6 for the run, 74 of 225 solved, No price listedNone
44gpt-4.1-mini32.4%
  • edit format: diff
  • command: aider --model gpt-4.1-mini
$0.0273measured: $1.99 for the run, 73 of 225 solved,

$0.4 in · $1.6 outopenai, USD, observed

$0.0096 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0016 one chat reply at openaiestimate from the price observed ; assumptions

45claude-3-5-haiku-2024102228%
  • edit format: diff
  • command: aider --model claude-3-5-haiku-20241022
$0.0962measured: $6.06 for the run, 63 of 225 solved, No price listedNone
47gpt-4o-2024-08-0623.1%
  • edit format: diff
  • command: aider --model gpt-4o-2024-08-06
$0.135measured: $7.03 for the run, 52 of 225 solved,

$2.5 in · $10 outopenai, USD, observed

$0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.01 one chat reply at openaiestimate from the price observed ; assumptions

51gpt-4o-2024-11-2018.2%
  • edit format: diff
  • command: aider --model gpt-4o-2024-11-20
$0.164measured: $6.74 for the run, 41 of 225 solved,

$2.5 in · $10 outopenai, USD, observed

$0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.01 one chat reply at openaiestimate from the price observed ; assumptions

53DeepSeek Chat V2.517.8%
  • edit format: diff
  • command: aider --model deepseek/deepseek-chat
$0.0128measured: $0.51 for the run, 40 of 225 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

58Codestral 25.0111.1%
  • edit format: whole
  • command: aider --model mistral/codestral-latest
$0.0793measured: $1.98 for the run, 25 of 225 solved,

$0.3 in · $0.9 outmistral, USD, observed

$0.0069 one coding-assistant request at mistralestimate from the price observed ; assumptions

$0.00105 one chat reply at mistralestimate from the price observed ; assumptions

60gpt-4.1-nano8.9%
  • edit format: whole
  • command: aider --model gpt-4.1-nano
$0.0214measured: $0.428 for the run, 20 of 225 solved,

$0.1 in · $0.4 outopenai, USD, observed

$0.0024 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0004 one chat reply at openaiestimate from the price observed ; assumptions

63gpt-4o-mini-2024-07-183.6%
  • edit format: whole
  • command: aider --model gpt-4o-mini-2024-07-18
$0.0404measured: $0.324 for the run, 8 of 225 solved,

$0.15 in · $0.6 outopenai, USD, observed

$0.0036 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0006 one chat reply at openaiestimate from the price observed ; assumptions

  1. #1 gpt-5 (high)

    Pass rate, second attempt
    88%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model openai/gpt-5
    Cost per solved task
    $0.147 measured (198 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  2. #2 gpt-5 (medium)

    Pass rate, second attempt
    86.7%,
    Run settings
    edit format: diff; reasoning effort: medium; command: aider --model openai/gpt-5
    Cost per solved task
    $0.0907 measured (195 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  3. #3 o3-pro (high)

    Pass rate, second attempt
    84.9%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model o3-pro
    Cost per solved task
    $0.766 measured (191 of 225 solved),
    Price per 1M tokens
    No price listed

  4. #4 gemini-2.5-pro-preview-06-05 (32k think)

    Pass rate, second attempt
    83.1%,
    Run settings
    edit format: diff-fenced; thinking tokens: 32768; command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
    Cost per solved task
    $0.267 measured (187 of 225 solved),
    Price per 1M tokens
    No price listed

  5. #5 o3 (high)

    Pass rate, second attempt
    81.3%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model o3 --reasoning-effort high
    Cost per solved task
    $0.116 measured (183 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  6. #5 gpt-5 (low)

    Pass rate, second attempt
    81.3%,
    Run settings
    edit format: diff; reasoning effort: low; command: aider --model openai/gpt-5
    Cost per solved task
    $0.0567 measured (183 of 225 solved),
    Price per 1M tokens

    $1.25 in · $10 out, openai, observed

    Workload estimate

    $0.035 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0075 one chat reply at openai, from the price observed (assumptions)

  7. #7 grok-4 (high)

    Pass rate, second attempt
    79.6%,
    Run settings
    edit format: diff; reasoning effort: high; command: aider --model openrouter/x-ai/grok-4
    Cost per solved task
    $0.333 measured (179 of 225 solved),
    Price per 1M tokens
    No price listed

  8. #8 gemini-2.5-pro-preview-06-05 (default think)

    Pass rate, second attempt
    79.1%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-06-05
    Cost per solved task
    $0.256 measured (178 of 225 solved),
    Price per 1M tokens
    No price listed

  9. #9 Gemini 2.5 Pro Preview 05-06

    Pass rate, second attempt
    76.9%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-05-06
    Cost per solved task
    $0.216 measured (173 of 225 solved),
    Price per 1M tokens
    No price listed

  10. #9 o3

    Pass rate, second attempt
    76.9%,
    Run settings
    edit format: diff; command: aider --model o3
    Cost per solved task
    $0.0795 measured (173 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  11. #11 DeepSeek-V3.2-Exp (Reasoner)

    Pass rate, second attempt
    74.2%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.00781 measured (167 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  12. #13 o4-mini (high)

    Pass rate, second attempt
    72%,
    Run settings
    edit format: diff; command: aider --model o4-mini
    Cost per solved task
    $0.121 measured (162 of 225 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  13. #13 claude-opus-4-20250514 (32k thinking)

    Pass rate, second attempt
    72%,
    Run settings
    edit format: diff; thinking tokens: 32000; command: aider --model claude-opus-4-20250514
    Cost per solved task
    $0.406 measured (162 of 225 solved),
    Price per 1M tokens
    No price listed

  14. #15 claude-opus-4-20250514 (no think)

    Pass rate, second attempt
    70.7%,
    Run settings
    edit format: diff; command: aider --model claude-opus-4-20250514
    Cost per solved task
    $0.432 measured (159 of 225 solved),
    Price per 1M tokens
    No price listed

  15. #16 DeepSeek-V3.2-Exp (Chat)

    Pass rate, second attempt
    70.2%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.00554 measured (158 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  16. #17 claude-3-7-sonnet-20250219 (32k thinking tokens)

    Pass rate, second attempt
    64.9%,
    Run settings
    edit format: diff; command: aider --model anthropic/claude-3-7-sonnet-20250219 --thinking-tokens 32k
    Cost per solved task
    $0.252 measured (146 of 225 solved),
    Price per 1M tokens
    No price listed

  17. #18 DeepSeek R1 + claude-3-5-sonnet-20241022

    Pass rate, second attempt
    64%,
    Run settings
    edit format: architect; editor model: claude-3-5-sonnet-20241022; editor edit format: editor-diff; command: aider --architect --model r1 --editor-model sonnet
    Cost per solved task
    $0.0923 measured (144 of 225 solved),
    Price per 1M tokens
    No price listed

  18. #19 claude-sonnet-4-20250514 (32k thinking)

    Pass rate, second attempt
    61.3%,
    Run settings
    edit format: diff; thinking tokens: 32000; command: aider --model claude-sonnet-4-20250514
    Cost per solved task
    $0.193 measured (138 of 225 solved),
    Price per 1M tokens
    No price listed

  19. #20 claude-3-7-sonnet-20250219 (no thinking)

    Pass rate, second attempt
    60.4%,
    Run settings
    edit format: diff; command: aider --model sonnet
    Cost per solved task
    $0.13 measured (136 of 225 solved),
    Price per 1M tokens
    No price listed

  20. #22 Kimi K2

    Pass rate, second attempt
    59.1%,
    Run settings
    edit format: diff; command: aider --model openrouter/moonshotai/kimi-k2
    Cost per solved task
    $0.00929 measured (133 of 225 solved),
    Price per 1M tokens
    No price listed

  21. #23 DeepSeek R1

    Pass rate, second attempt
    56.9%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.0423 measured (128 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  22. #24 claude-sonnet-4-20250514 (no thinking)

    Pass rate, second attempt
    56.4%,
    Run settings
    edit format: diff; command: aider --model claude-sonnet-4-20250514
    Cost per solved task
    $0.125 measured (127 of 225 solved),
    Price per 1M tokens
    No price listed

  23. #25 DeepSeek V3 (0324)

    Pass rate, second attempt
    55.1%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.009 measured (124 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  24. #25 gemini-2.5-flash-preview-05-20 (24k think)

    Pass rate, second attempt
    55.1%,
    Run settings
    edit format: diff; thinking tokens: 24576; command: aider --model gemini/gemini-2.5-flash-preview-05-20
    Cost per solved task
    $0.0691 measured (124 of 225 solved),
    Price per 1M tokens
    No price listed

  25. #28 o3-mini (medium)

    Pass rate, second attempt
    53.8%,
    Run settings
    edit format: diff; command: aider --model o3-mini
    Cost per solved task
    $0.0732 measured (121 of 225 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  26. #29 Grok 3 Beta

    Pass rate, second attempt
    53.3%,
    Run settings
    edit format: diff; command: aider --model openrouter/x-ai/grok-3-beta
    Cost per solved task
    $0.0919 measured (120 of 225 solved),
    Price per 1M tokens
    No price listed

  27. #31 gpt-4.1

    Pass rate, second attempt
    52.4%,
    Run settings
    edit format: diff; command: aider --model gpt-4.1
    Cost per solved task
    $0.0835 measured (118 of 225 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  28. #32 claude-3-5-sonnet-20241022

    Pass rate, second attempt
    51.6%,
    Run settings
    edit format: diff; command: aider --model claude-3-5-sonnet-20241022
    Cost per solved task
    $0.124 measured (116 of 225 solved),
    Price per 1M tokens
    No price listed

  29. #33 Grok 3 Mini Beta (high)

    Pass rate, second attempt
    49.3%,
    Run settings
    edit format: whole; command: aider --model xai/grok-3-mini-beta --reasoning-effort high
    Cost per solved task
    $0.00662 measured (111 of 225 solved),
    Price per 1M tokens
    No price listed

  30. #34 DeepSeek Chat V3 (prev)

    Pass rate, second attempt
    48.4%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.00309 measured (109 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  31. #35 gemini-2.5-flash-preview-04-17 (default)

    Pass rate, second attempt
    47.1%,
    Run settings
    edit format: diff; command: aider --model gemini/gemini-2.5-flash-preview-04-17
    Cost per solved task
    $0.0174 measured (106 of 225 solved),
    Price per 1M tokens
    No price listed

  32. #36 chatgpt-4o-latest (2025-03-29)

    Pass rate, second attempt
    45.3%,
    Run settings
    edit format: diff; command: aider --model chatgpt-4o-latest
    Cost per solved task
    $0.194 measured (102 of 225 solved),
    Price per 1M tokens
    No price listed

  33. #37 gemini-2.5-flash-preview-05-20 (no think)

    Pass rate, second attempt
    44%,
    Run settings
    edit format: diff; thinking tokens: 0; command: aider --model gemini/gemini-2.5-flash-preview-05-20
    Cost per solved task
    $0.0115 measured (99 of 225 solved),
    Price per 1M tokens
    No price listed

  34. #38 gpt-oss-120b (high)

    Pass rate, second attempt
    41.8%,
    Run settings
    edit format: diff; command: aider --model openrouter/openai/gpt-oss-120b --reasoning-effort high
    Cost per solved task
    $0.00788 measured (94 of 225 solved),
    Price per 1M tokens
    No price listed

  35. #39 Qwen3 32B

    Pass rate, second attempt
    40%,
    Run settings
    edit format: diff; command: aider --model openrouter/qwen/qwen3-32b
    Cost per solved task
    $0.00845 measured (90 of 225 solved),
    Price per 1M tokens
    No price listed

  36. #42 Grok 3 Mini Beta (low)

    Pass rate, second attempt
    34.7%,
    Run settings
    edit format: whole; command: aider --model openrouter/x-ai/grok-3-mini-beta
    Cost per solved task
    $0.0101 measured (78 of 225 solved),
    Price per 1M tokens
    No price listed

  37. #43 o1-mini-2024-09-12

    Pass rate, second attempt
    32.9%,
    Run settings
    edit format: whole; command: aider --model o1-mini
    Cost per solved task
    $0.251 measured (74 of 225 solved),
    Price per 1M tokens
    No price listed

  38. #44 gpt-4.1-mini

    Pass rate, second attempt
    32.4%,
    Run settings
    edit format: diff; command: aider --model gpt-4.1-mini
    Cost per solved task
    $0.0273 measured (73 of 225 solved),
    Price per 1M tokens

    $0.4 in · $1.6 out, openai, observed

    Workload estimate

    $0.0096 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0016 one chat reply at openai, from the price observed (assumptions)

  39. #45 claude-3-5-haiku-20241022

    Pass rate, second attempt
    28%,
    Run settings
    edit format: diff; command: aider --model claude-3-5-haiku-20241022
    Cost per solved task
    $0.0962 measured (63 of 225 solved),
    Price per 1M tokens
    No price listed

  40. #47 gpt-4o-2024-08-06

    Pass rate, second attempt
    23.1%,
    Run settings
    edit format: diff; command: aider --model gpt-4o-2024-08-06
    Cost per solved task
    $0.135 measured (52 of 225 solved),
    Price per 1M tokens

    $2.5 in · $10 out, openai, observed

    Workload estimate

    $0.06 one coding-assistant request at openai, from the price observed (assumptions)

    $0.01 one chat reply at openai, from the price observed (assumptions)

  41. #51 gpt-4o-2024-11-20

    Pass rate, second attempt
    18.2%,
    Run settings
    edit format: diff; command: aider --model gpt-4o-2024-11-20
    Cost per solved task
    $0.164 measured (41 of 225 solved),
    Price per 1M tokens

    $2.5 in · $10 out, openai, observed

    Workload estimate

    $0.06 one coding-assistant request at openai, from the price observed (assumptions)

    $0.01 one chat reply at openai, from the price observed (assumptions)

  42. #53 DeepSeek Chat V2.5

    Pass rate, second attempt
    17.8%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-chat
    Cost per solved task
    $0.0128 measured (40 of 225 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  43. #58 Codestral 25.01

    Pass rate, second attempt
    11.1%,
    Run settings
    edit format: whole; command: aider --model mistral/codestral-latest
    Cost per solved task
    $0.0793 measured (25 of 225 solved),
    Price per 1M tokens

    $0.3 in · $0.9 out, mistral, observed

    Workload estimate

    $0.0069 one coding-assistant request at mistral, from the price observed (assumptions)

    $0.00105 one chat reply at mistral, from the price observed (assumptions)

  44. #60 gpt-4.1-nano

    Pass rate, second attempt
    8.9%,
    Run settings
    edit format: whole; command: aider --model gpt-4.1-nano
    Cost per solved task
    $0.0214 measured (20 of 225 solved),
    Price per 1M tokens

    $0.1 in · $0.4 out, openai, observed

    Workload estimate

    $0.0024 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0004 one chat reply at openai, from the price observed (assumptions)

  45. #63 gpt-4o-mini-2024-07-18

    Pass rate, second attempt
    3.6%,
    Run settings
    edit format: whole; command: aider --model gpt-4o-mini-2024-07-18
    Cost per solved task
    $0.0404 measured (8 of 225 solved),
    Price per 1M tokens

    $0.15 in · $0.6 out, openai, observed

    Workload estimate

    $0.0036 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0006 one chat reply at openai, from the price observed (assumptions)

Source: Aider polyglot leaderboard. Each score links the published result it was read from.