Model comparisons

Aider polyglot (224 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: DeepSeek, OpenAI

5 of 5 results: cost — with a measured cost per solved task.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
1o3 (high) + gpt-4.178.2%
  • edit format: architect
  • reasoning effort: high
  • editor model: gpt-4.1
  • editor edit format: editor-diff
  • command: aider --model o3
$0.1measured: $17.6 for the run, 175 of 224 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

2DeepSeek R1 (0528)71.4%
  • edit format: diff
  • command: aider --model deepseek/deepseek-reasoner
$0.03measured: $4.8 for the run, 160 of 224 solved,

$0.28 in · $0.42 outdeepseek, USD, observed

$0.00602 one coding-assistant request at deepseekestimate from the price observed ; assumptions

$0.00077 one chat reply at deepseekestimate from the price observed ; assumptions

3o1-2024-12-17 (high)61.7%
  • edit format: diff
  • command: aider --model openrouter/openai/o1
$1.35measured: $186.50 for the run, 138 of 224 solved, No price listedNone
4o3-mini (high)60.4%
  • edit format: diff
  • command: aider --model o3-mini --reasoning-effort high
$0.135measured: $18.2 for the run, 135 of 224 solved,

$1.1 in · $4.4 outopenai, USD, observed

$0.0264 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.0044 one chat reply at openaiestimate from the price observed ; assumptions

5gpt-4.5-preview44.9%
  • edit format: diff
  • command: aider --model openai/gpt-4.5-preview
$1.81measured: $183.18 for the run, 101 of 224 solved, No price listedNone

  1. #1 o3 (high) + gpt-4.1

    Pass rate, second attempt
    78.2%,
    Run settings
    edit format: architect; reasoning effort: high; editor model: gpt-4.1; editor edit format: editor-diff; command: aider --model o3
    Cost per solved task
    $0.1 measured (175 of 224 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

  2. #2 DeepSeek R1 (0528)

    Pass rate, second attempt
    71.4%,
    Run settings
    edit format: diff; command: aider --model deepseek/deepseek-reasoner
    Cost per solved task
    $0.03 measured (160 of 224 solved),
    Price per 1M tokens

    $0.28 in · $0.42 out, deepseek, observed

    Workload estimate

    $0.00602 one coding-assistant request at deepseek, from the price observed (assumptions)

    $0.00077 one chat reply at deepseek, from the price observed (assumptions)

  3. #3 o1-2024-12-17 (high)

    Pass rate, second attempt
    61.7%,
    Run settings
    edit format: diff; command: aider --model openrouter/openai/o1
    Cost per solved task
    $1.35 measured (138 of 224 solved),
    Price per 1M tokens
    No price listed

  4. #4 o3-mini (high)

    Pass rate, second attempt
    60.4%,
    Run settings
    edit format: diff; command: aider --model o3-mini --reasoning-effort high
    Cost per solved task
    $0.135 measured (135 of 224 solved),
    Price per 1M tokens

    $1.1 in · $4.4 out, openai, observed

    Workload estimate

    $0.0264 one coding-assistant request at openai, from the price observed (assumptions)

    $0.0044 one chat reply at openai, from the price observed (assumptions)

  5. #5 gpt-4.5-preview

    Pass rate, second attempt
    44.9%,
    Run settings
    edit format: diff; command: aider --model openai/gpt-4.5-preview
    Cost per solved task
    $1.81 measured (101 of 224 solved),
    Price per 1M tokens
    No price listed

Source: Aider polyglot leaderboard. Each score links the published result it was read from.