This browser is not keeping saved items, so they will be forgotten when you leave this page. Everything else still works.
Model comparisons
Aider polyglot (225 exercises)
Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.
Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind
15 of 63 results: edit format — whole.
| Place | Model | Pass rate, second attempt | Run settings | Cost per solved task | Price per 1M tokens | Workload estimate |
|---|---|---|---|---|---|---|
| 33 | Grok 3 Mini Beta (high) | 49.3% |
| $0.00662measured: $0.735 for the run, 111 of 225 solved, | No price listed | None |
| 40 | gemini-exp-1206 | 38.2% |
| Not measured | $0 in · $0 outgemini, USD, observed | $0 one coding-assistant request at geminiestimate from the price observed ; assumptions $0 one chat reply at geminiestimate from the price observed ; assumptions |
| 41 | Gemini 2.0 Pro exp-02-05 | 35.6% |
| Not measured | No price listed | None |
| 42 | Grok 3 Mini Beta (low) | 34.7% |
| $0.0101measured: $0.786 for the run, 78 of 225 solved, | No price listed | None |
| 43 | o1-mini-2024-09-12 | 32.9% |
| $0.251measured: $18.6 for the run, 74 of 225 solved, | No price listed | None |
| 48 | gemini-2.0-flash-exp | 22.2% |
| Not measured | No price listed | None |
| 54 | Qwen2.5-Coder-32B-Instruct | 16.4% |
| Not measured | No price listed | None |
| 55 | Llama 4 Maverick | 15.6% |
| Not measured | No price listed | None |
| 56 | yi-lightning | 12.9% |
| Not measured | No price listed | None |
| 57 | command-a-03-2025-quality | 12% |
| Not measured | No price listed | None |
| 58 | Codestral 25.01 | 11.1% |
| $0.0793measured: $1.98 for the run, 25 of 225 solved, | $0.3 in · $0.9 outmistral, USD, observed | $0.0069 one coding-assistant request at mistralestimate from the price observed ; assumptions $0.00105 one chat reply at mistralestimate from the price observed ; assumptions |
| 59 | openhands-lm-32b-v0.1 | 10.2% |
| Not measured | No price listed | None |
| 60 | gpt-4.1-nano | 8.9% |
| $0.0214measured: $0.428 for the run, 20 of 225 solved, | $0.1 in · $0.4 outopenai, USD, observed | $0.0024 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0004 one chat reply at openaiestimate from the price observed ; assumptions |
| 62 | gemma-3-27b-it | 4.9% |
| Not measured | No price listed | None |
| 63 | gpt-4o-mini-2024-07-18 | 3.6% |
| $0.0404measured: $0.324 for the run, 8 of 225 solved, | $0.15 in · $0.6 outopenai, USD, observed | $0.0036 one coding-assistant request at openaiestimate from the price observed ; assumptions $0.0006 one chat reply at openaiestimate from the price observed ; assumptions |
#33 Grok 3 Mini Beta (high)
#40 gemini-exp-1206
- Pass rate, second attempt
- 38.2%,
- Run settings
- edit format: whole; command: aider --model gemini/gemini-exp-1206
- Cost per solved task
- Not measured
- Price per 1M tokens
$0 in · $0 out, gemini, observed
- Workload estimate
$0 one coding-assistant request at gemini, from the price observed (assumptions)
$0 one chat reply at gemini, from the price observed (assumptions)
#41 Gemini 2.0 Pro exp-02-05
- Pass rate, second attempt
- 35.6%,
- Run settings
- edit format: whole; command: aider --model gemini/gemini-2.0-pro-exp-02-05
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#42 Grok 3 Mini Beta (low)
#43 o1-mini-2024-09-12
#48 gemini-2.0-flash-exp
- Pass rate, second attempt
- 22.2%,
- Run settings
- edit format: whole; command: aider --model gemini/gemini-2.0-flash-exp
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#54 Qwen2.5-Coder-32B-Instruct
- Pass rate, second attempt
- 16.4%,
- Run settings
- edit format: whole; command: aider --model openai/Qwen2.5-Coder-32B-Instruct
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#55 Llama 4 Maverick
- Pass rate, second attempt
- 15.6%,
- Run settings
- edit format: whole; command: aider --model nvidia_nim/meta/llama-4-maverick-17b-128e-instruct
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#56 yi-lightning
- Pass rate, second attempt
- 12.9%,
- Run settings
- edit format: whole; command: aider --model openai/yi-lightning
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#57 command-a-03-2025-quality
- Pass rate, second attempt
- 12%,
- Run settings
- edit format: whole; command: OPENAI_API_BASE=https://api.cohere.ai/compatibility/v1 aider --model openai/command-a-03-2025-quality
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#58 Codestral 25.01
- Pass rate, second attempt
- 11.1%,
- Run settings
- edit format: whole; command: aider --model mistral/codestral-latest
- Cost per solved task
- $0.0793 measured (25 of 225 solved),
- Price per 1M tokens
$0.3 in · $0.9 out, mistral, observed
- Workload estimate
$0.0069 one coding-assistant request at mistral, from the price observed (assumptions)
$0.00105 one chat reply at mistral, from the price observed (assumptions)
#59 openhands-lm-32b-v0.1
- Pass rate, second attempt
- 10.2%,
- Run settings
- edit format: whole; command: aider --model openrouter/all-hands/openhands-lm-32b-v0.1
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#60 gpt-4.1-nano
- Pass rate, second attempt
- 8.9%,
- Run settings
- edit format: whole; command: aider --model gpt-4.1-nano
- Cost per solved task
- $0.0214 measured (20 of 225 solved),
- Price per 1M tokens
$0.1 in · $0.4 out, openai, observed
- Workload estimate
$0.0024 one coding-assistant request at openai, from the price observed (assumptions)
$0.0004 one chat reply at openai, from the price observed (assumptions)
#62 gemma-3-27b-it
- Pass rate, second attempt
- 4.9%,
- Run settings
- edit format: whole; command: aider --model openrouter/google/gemma-3-27b-it
- Cost per solved task
- Not measured
- Price per 1M tokens
- No price listed
#63 gpt-4o-mini-2024-07-18
- Pass rate, second attempt
- 3.6%,
- Run settings
- edit format: whole; command: aider --model gpt-4o-mini-2024-07-18
- Cost per solved task
- $0.0404 measured (8 of 225 solved),
- Price per 1M tokens
$0.15 in · $0.6 out, openai, observed
- Workload estimate
$0.0036 one coding-assistant request at openai, from the price observed (assumptions)
$0.0006 one chat reply at openai, from the price observed (assumptions)
Source: Aider polyglot leaderboard. Each score links the published result it was read from.