This browser is not keeping saved items, so they will be forgotten when you leave this page. Everything else still works.
Model comparisons
Aider polyglot (225 exercises)
Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.
Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind
45 of 63 results: cost — cheapest per solved task first. Ordered by measured cost per solved task, cheapest first; the place shown is still the place by score.
| Place | Model | Pass rate, second attempt | Run settings | Cost per solved task | Price per 1M tokens | Workload estimate |
|---|---|---|---|---|---|---|
| 34 | DeepSeek Chat V3 (prev) | 48.4% |
| $0.00309measured: $0.337 for the run, 109 of 225 solved, | $0.28 in · $0.42 outdeepseek, USD, observed | $0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions $0.00077 one chat reply at deepseekestimate from the price observed ; assumptions |
| 16 | DeepSeek-V3.2-Exp (Chat) | 70.2% |
| $0.00554measured: $0.876 for the run, 158 of 225 solved, | $0.28 in · $0.42 outdeepseek, USD, observed | $0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions $0.00077 one chat reply at deepseekestimate from the price observed ; assumptions |
| 33 | Grok 3 Mini Beta (high) | 49.3% |
| $0.00662measured: $0.735 for the run, 111 of 225 solved, | No price listed | None |
| 11 | DeepSeek-V3.2-Exp (Reasoner) | 74.2% |
| $0.00781measured: $1.3 for the run, 167 of 225 solved, | $0.28 in · $0.42 outdeepseek, USD, observed | $0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions $0.00077 one chat reply at deepseekestimate from the price observed ; assumptions |
| 38 | gpt-oss-120b (high) | 41.8% |
| $0.00788measured: $0.741 for the run, 94 of 225 solved, | No price listed | None |
| 39 | Qwen3 32B | 40% |
| $0.00845measured: $0.76 for the run, 90 of 225 solved, | No price listed | None |
| 25 | DeepSeek V3 (0324) | 55.1% |
| $0.009measured: $1.12 for the run, 124 of 225 solved, | $0.28 in · $0.42 outdeepseek, USD, observed | $0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions $0.00077 one chat reply at deepseekestimate from the price observed ; assumptions |
| 22 | Kimi K2 | 59.1% |
| $0.00929measured: $1.24 for the run, 133 of 225 solved, | No price listed | None |
| 42 | Grok 3 Mini Beta (low) | 34.7% |
| $0.0101measured: $0.786 for the run, 78 of 225 solved, | No price listed | None |
| 37 | gemini-2.5-flash-preview-05-20 (no think) | 44% |
| $0.0115measured: $1.14 for the run, 99 of 225 solved, | No price listed | None |
| 53 | DeepSeek Chat V2.5 | 17.8% |
| $0.0128measured: $0.51 for the run, 40 of 225 solved, | $0.28 in · $0.42 outdeepseek, USD, observed | $0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions $0.00077 one chat reply at deepseekestimate from the price observed ; assumptions |
| 35 | gemini-2.5-flash-preview-04-17 (default) | 47.1% |
| $0.0174measured: $1.85 for the run, 106 of 225 solved, | No price listed | None |
| 60 | gpt-4.1-nano | 8.9% |
| $0.0214measured: $0.428 for the run, 20 of 225 solved, | $0.1 in · $0.4 outopenai, USD, observed | $0.0024 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0004 one chat reply at openaiestimate from the price observed ; assumptions |
| 44 | gpt-4.1-mini | 32.4% |
| $0.0273measured: $1.99 for the run, 73 of 225 solved, | $0.4 in · $1.6 outopenai, USD, observed | $0.0096 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0016 one chat reply at openaiestimate from the price observed ; assumptions |
| 63 | gpt-4o-mini-2024-07-18 | 3.6% |
| $0.0404measured: $0.324 for the run, 8 of 225 solved, | $0.15 in · $0.6 outopenai, USD, observed | $0.0036 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0006 one chat reply at openaiestimate from the price observed ; assumptions |
| 23 | DeepSeek R1 | 56.9% |
| $0.0423measured: $5.42 for the run, 128 of 225 solved, | $0.28 in · $0.42 outdeepseek, USD, observed | $0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions $0.00077 one chat reply at deepseekestimate from the price observed ; assumptions |
| 5 | gpt-5 (low) | 81.3% |
| $0.0567measured: $10.4 for the run, 183 of 225 solved, | $1.25 in · $10 outopenai, USD, observed | $0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0075 one chat reply at openaiestimate from the price observed ; assumptions |
| 25 | gemini-2.5-flash-preview-05-20 (24k think) | 55.1% |
| $0.0691measured: $8.56 for the run, 124 of 225 solved, | No price listed | None |
| 28 | o3-mini (medium) | 53.8% |
| $0.0732measured: $8.86 for the run, 121 of 225 solved, | $1.1 in · $4.4 outopenai, USD, observed | $0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0044 one chat reply at openaiestimate from the price observed ; assumptions |
| 58 | Codestral 25.01 | 11.1% |
| $0.0793measured: $1.98 for the run, 25 of 225 solved, | $0.3 in · $0.9 outmistral, USD, observed | $0.0069 one coding-assistant request at mistralestimate from the price observed ; assumptions $0.00105 one chat reply at mistralestimate from the price observed ; assumptions |
| 9 | o3 | 76.9% |
| $0.0795measured: $13.8 for the run, 173 of 225 solved, | $2 in · $8 outopenai, USD, observed | $0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.008 one chat reply at openaiestimate from the price observed ; assumptions |
| 31 | gpt-4.1 | 52.4% |
| $0.0835measured: $9.86 for the run, 118 of 225 solved, | $2 in · $8 outopenai, USD, observed | $0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.008 one chat reply at openaiestimate from the price observed ; assumptions |
| 2 | gpt-5 (medium) | 86.7% |
| $0.0907measured: $17.7 for the run, 195 of 225 solved, | $1.25 in · $10 outopenai, USD, observed | $0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0075 one chat reply at openaiestimate from the price observed ; assumptions |
| 29 | Grok 3 Beta | 53.3% |
| $0.0919measured: $11 for the run, 120 of 225 solved, | No price listed | None |
| 18 | DeepSeek R1 + claude-3-5-sonnet-20241022 | 64% |
| $0.0923measured: $13.3 for the run, 144 of 225 solved, | No price listed | None |
| 45 | claude-3-5-haiku-20241022 | 28% |
| $0.0962measured: $6.06 for the run, 63 of 225 solved, | No price listed | None |
| 5 | o3 (high) | 81.3% |
| $0.116measured: $21.2 for the run, 183 of 225 solved, | $2 in · $8 outopenai, USD, observed | $0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.008 one chat reply at openaiestimate from the price observed ; assumptions |
| 13 | o4-mini (high) | 72% |
| $0.121measured: $19.6 for the run, 162 of 225 solved, | $1.1 in · $4.4 outopenai, USD, observed | $0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0044 one chat reply at openaiestimate from the price observed ; assumptions |
| 32 | claude-3-5-sonnet-20241022 | 51.6% |
| $0.124measured: $14.4 for the run, 116 of 225 solved, | No price listed | None |
| 24 | claude-sonnet-4-20250514 (no thinking) | 56.4% |
| $0.125measured: $15.8 for the run, 127 of 225 solved, | No price listed | None |
| 20 | claude-3-7-sonnet-20250219 (no thinking) | 60.4% |
| $0.13measured: $17.7 for the run, 136 of 225 solved, | No price listed | None |
| 47 | gpt-4o-2024-08-06 | 23.1% |
| $0.135measured: $7.03 for the run, 52 of 225 solved, | $2.5 in · $10 outopenai, USD, observed | $0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.01 one chat reply at openaiestimate from the price observed ; assumptions |
| 1 | gpt-5 (high) | 88% |
| $0.147measured: $29.1 for the run, 198 of 225 solved, | $1.25 in · $10 outopenai, USD, observed | $0.035 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0075 one chat reply at openaiestimate from the price observed ; assumptions |
| 51 | gpt-4o-2024-11-20 | 18.2% |
| $0.164measured: $6.74 for the run, 41 of 225 solved, | $2.5 in · $10 outopenai, USD, observed | $0.06 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.01 one chat reply at openaiestimate from the price observed ; assumptions |
| 19 | claude-sonnet-4-20250514 (32k thinking) | 61.3% |
| $0.193measured: $26.6 for the run, 138 of 225 solved, | No price listed | None |
| 36 | chatgpt-4o-latest (2025-03-29) | 45.3% |
| $0.194measured: $19.7 for the run, 102 of 225 solved, | No price listed | None |
| 9 | Gemini 2.5 Pro Preview 05-06 | 76.9% |
| $0.216measured: $37.4 for the run, 173 of 225 solved, | No price listed | None |
| 43 | o1-mini-2024-09-12 | 32.9% |
| $0.251measured: $18.6 for the run, 74 of 225 solved, | No price listed | None |
| 17 | claude-3-7-sonnet-20250219 (32k thinking tokens) | 64.9% |
| $0.252measured: $36.8 for the run, 146 of 225 solved, | No price listed | None |
| 8 | gemini-2.5-pro-preview-06-05 (default think) | 79.1% |
| $0.256measured: $45.6 for the run, 178 of 225 solved, | No price listed | None |
| 4 | gemini-2.5-pro-preview-06-05 (32k think) | 83.1% |
| $0.267measured: $49.9 for the run, 187 of 225 solved, | No price listed | None |
| 7 | grok-4 (high) | 79.6% |
| $0.333measured: $59.6 for the run, 179 of 225 solved, | No price listed | None |
| 13 | claude-opus-4-20250514 (32k thinking) | 72% |
| $0.406measured: $65.7 for the run, 162 of 225 solved, | No price listed | None |
| 15 | claude-opus-4-20250514 (no think) | 70.7% |
| $0.432measured: $68.6 for the run, 159 of 225 solved, | No price listed | None |
| 3 | o3-pro (high) | 84.9% |
| $0.766measured: $146.32 for the run, 191 of 225 solved, | No price listed | None |
#34 DeepSeek Chat V3 (prev)
- Pass rate, second attempt
- 48.4%,
- Run settings
- edit format: diff; command: aider --model deepseek/deepseek-chat
- Cost per solved task
- $0.00309 measured (109 of 225 solved),
- Price per 1M tokens
$0.28 in · $0.42 out, deepseek, observed
- Workload estimate
$0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)
$0.00077 one chat reply at deepseek, from the price observed (assumptions)
#16 DeepSeek-V3.2-Exp (Chat)
- Pass rate, second attempt
- 70.2%,
- Run settings
- edit format: diff; command: aider --model deepseek/deepseek-chat
- Cost per solved task
- $0.00554 measured (158 of 225 solved),
- Price per 1M tokens
$0.28 in · $0.42 out, deepseek, observed
- Workload estimate
$0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)
$0.00077 one chat reply at deepseek, from the price observed (assumptions)
#33 Grok 3 Mini Beta (high)
#11 DeepSeek-V3.2-Exp (Reasoner)
- Pass rate, second attempt
- 74.2%,
- Run settings
- edit format: diff; command: aider --model deepseek/deepseek-reasoner
- Cost per solved task
- $0.00781 measured (167 of 225 solved),
- Price per 1M tokens
$0.28 in · $0.42 out, deepseek, observed
- Workload estimate
$0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)
$0.00077 one chat reply at deepseek, from the price observed (assumptions)
#38 gpt-oss-120b (high)
#39 Qwen3 32B
#25 DeepSeek V3 (0324)
- Pass rate, second attempt
- 55.1%,
- Run settings
- edit format: diff; command: aider --model deepseek/deepseek-chat
- Cost per solved task
- $0.009 measured (124 of 225 solved),
- Price per 1M tokens
$0.28 in · $0.42 out, deepseek, observed
- Workload estimate
$0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)
$0.00077 one chat reply at deepseek, from the price observed (assumptions)
#22 Kimi K2
#42 Grok 3 Mini Beta (low)
#37 gemini-2.5-flash-preview-05-20 (no think)
#53 DeepSeek Chat V2.5
- Pass rate, second attempt
- 17.8%,
- Run settings
- edit format: diff; command: aider --model deepseek/deepseek-chat
- Cost per solved task
- $0.0128 measured (40 of 225 solved),
- Price per 1M tokens
$0.28 in · $0.42 out, deepseek, observed
- Workload estimate
$0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)
$0.00077 one chat reply at deepseek, from the price observed (assumptions)
#35 gemini-2.5-flash-preview-04-17 (default)
#60 gpt-4.1-nano
- Pass rate, second attempt
- 8.9%,
- Run settings
- edit format: whole; command: aider --model gpt-4.1-nano
- Cost per solved task
- $0.0214 measured (20 of 225 solved),
- Price per 1M tokens
$0.1 in · $0.4 out, openai, observed
- Workload estimate
$0.0024 one coding-assistant request at openai, from the price observed (assumptions)
$0.0004 one chat reply at openai, from the price observed (assumptions)
#44 gpt-4.1-mini
- Pass rate, second attempt
- 32.4%,
- Run settings
- edit format: diff; command: aider --model gpt-4.1-mini
- Cost per solved task
- $0.0273 measured (73 of 225 solved),
- Price per 1M tokens
$0.4 in · $1.6 out, openai, observed
- Workload estimate
$0.0096 one coding-assistant request at openai, from the price observed (assumptions)
$0.0016 one chat reply at openai, from the price observed (assumptions)
#63 gpt-4o-mini-2024-07-18
- Pass rate, second attempt
- 3.6%,
- Run settings
- edit format: whole; command: aider --model gpt-4o-mini-2024-07-18
- Cost per solved task
- $0.0404 measured (8 of 225 solved),
- Price per 1M tokens
$0.15 in · $0.6 out, openai, observed
- Workload estimate
$0.0036 one coding-assistant request at openai, from the price observed (assumptions)
$0.0006 one chat reply at openai, from the price observed (assumptions)
#23 DeepSeek R1
- Pass rate, second attempt
- 56.9%,
- Run settings
- edit format: diff; command: aider --model deepseek/deepseek-reasoner
- Cost per solved task
- $0.0423 measured (128 of 225 solved),
- Price per 1M tokens
$0.28 in · $0.42 out, deepseek, observed
- Workload estimate
$0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)
$0.00077 one chat reply at deepseek, from the price observed (assumptions)
#5 gpt-5 (low)
- Pass rate, second attempt
- 81.3%,
- Run settings
- edit format: diff; reasoning effort: low; command: aider --model openai/gpt-5
- Cost per solved task
- $0.0567 measured (183 of 225 solved),
- Price per 1M tokens
$1.25 in · $10 out, openai, observed
- Workload estimate
$0.035 one coding-assistant request at openai, from the price observed (assumptions)
$0.0075 one chat reply at openai, from the price observed (assumptions)
#25 gemini-2.5-flash-preview-05-20 (24k think)
#28 o3-mini (medium)
- Pass rate, second attempt
- 53.8%,
- Run settings
- edit format: diff; command: aider --model o3-mini
- Cost per solved task
- $0.0732 measured (121 of 225 solved),
- Price per 1M tokens
$1.1 in · $4.4 out, openai, observed
- Workload estimate
$0.0264 one coding-assistant request at openai, from the price observed (assumptions)
$0.0044 one chat reply at openai, from the price observed (assumptions)
#58 Codestral 25.01
- Pass rate, second attempt
- 11.1%,
- Run settings
- edit format: whole; command: aider --model mistral/codestral-latest
- Cost per solved task
- $0.0793 measured (25 of 225 solved),
- Price per 1M tokens
$0.3 in · $0.9 out, mistral, observed
- Workload estimate
$0.0069 one coding-assistant request at mistral, from the price observed (assumptions)
$0.00105 one chat reply at mistral, from the price observed (assumptions)
#9 o3
- Pass rate, second attempt
- 76.9%,
- Run settings
- edit format: diff; command: aider --model o3
- Cost per solved task
- $0.0795 measured (173 of 225 solved),
- Price per 1M tokens
$2 in · $8 out, openai, observed
- Workload estimate
$0.048 one coding-assistant request at openai, from the price observed (assumptions)
$0.008 one chat reply at openai, from the price observed (assumptions)
#31 gpt-4.1
- Pass rate, second attempt
- 52.4%,
- Run settings
- edit format: diff; command: aider --model gpt-4.1
- Cost per solved task
- $0.0835 measured (118 of 225 solved),
- Price per 1M tokens
$2 in · $8 out, openai, observed
- Workload estimate
$0.048 one coding-assistant request at openai, from the price observed (assumptions)
$0.008 one chat reply at openai, from the price observed (assumptions)
#2 gpt-5 (medium)
- Pass rate, second attempt
- 86.7%,
- Run settings
- edit format: diff; reasoning effort: medium; command: aider --model openai/gpt-5
- Cost per solved task
- $0.0907 measured (195 of 225 solved),
- Price per 1M tokens
$1.25 in · $10 out, openai, observed
- Workload estimate
$0.035 one coding-assistant request at openai, from the price observed (assumptions)
$0.0075 one chat reply at openai, from the price observed (assumptions)
#29 Grok 3 Beta
#18 DeepSeek R1 + claude-3-5-sonnet-20241022
#45 claude-3-5-haiku-20241022
#5 o3 (high)
- Pass rate, second attempt
- 81.3%,
- Run settings
- edit format: diff; reasoning effort: high; command: aider --model o3 --reasoning-effort high
- Cost per solved task
- $0.116 measured (183 of 225 solved),
- Price per 1M tokens
$2 in · $8 out, openai, observed
- Workload estimate
$0.048 one coding-assistant request at openai, from the price observed (assumptions)
$0.008 one chat reply at openai, from the price observed (assumptions)
#13 o4-mini (high)
- Pass rate, second attempt
- 72%,
- Run settings
- edit format: diff; command: aider --model o4-mini
- Cost per solved task
- $0.121 measured (162 of 225 solved),
- Price per 1M tokens
$1.1 in · $4.4 out, openai, observed
- Workload estimate
$0.0264 one coding-assistant request at openai, from the price observed (assumptions)
$0.0044 one chat reply at openai, from the price observed (assumptions)
#32 claude-3-5-sonnet-20241022
#24 claude-sonnet-4-20250514 (no thinking)
#20 claude-3-7-sonnet-20250219 (no thinking)
#47 gpt-4o-2024-08-06
- Pass rate, second attempt
- 23.1%,
- Run settings
- edit format: diff; command: aider --model gpt-4o-2024-08-06
- Cost per solved task
- $0.135 measured (52 of 225 solved),
- Price per 1M tokens
$2.5 in · $10 out, openai, observed
- Workload estimate
$0.06 one coding-assistant request at openai, from the price observed (assumptions)
$0.01 one chat reply at openai, from the price observed (assumptions)
#1 gpt-5 (high)
- Pass rate, second attempt
- 88%,
- Run settings
- edit format: diff; reasoning effort: high; command: aider --model openai/gpt-5
- Cost per solved task
- $0.147 measured (198 of 225 solved),
- Price per 1M tokens
$1.25 in · $10 out, openai, observed
- Workload estimate
$0.035 one coding-assistant request at openai, from the price observed (assumptions)
$0.0075 one chat reply at openai, from the price observed (assumptions)
#51 gpt-4o-2024-11-20
- Pass rate, second attempt
- 18.2%,
- Run settings
- edit format: diff; command: aider --model gpt-4o-2024-11-20
- Cost per solved task
- $0.164 measured (41 of 225 solved),
- Price per 1M tokens
$2.5 in · $10 out, openai, observed
- Workload estimate
$0.06 one coding-assistant request at openai, from the price observed (assumptions)
$0.01 one chat reply at openai, from the price observed (assumptions)
#19 claude-sonnet-4-20250514 (32k thinking)
#36 chatgpt-4o-latest (2025-03-29)
#9 Gemini 2.5 Pro Preview 05-06
#43 o1-mini-2024-09-12
#17 claude-3-7-sonnet-20250219 (32k thinking tokens)
#8 gemini-2.5-pro-preview-06-05 (default think)
#4 gemini-2.5-pro-preview-06-05 (32k think)
#7 grok-4 (high)
#13 claude-opus-4-20250514 (32k thinking)
#15 claude-opus-4-20250514 (no think)
#3 o3-pro (high)
Source: Aider polyglot leaderboard. Each score links the published result it was read from.