Model comparisons

Aider polyglot (224 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: DeepSeek, OpenAI

1 of 5 results: edit format — architect.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
1o3 (high) + gpt-4.178.2%
  • edit format: architect
  • reasoning effort: high
  • editor model: gpt-4.1
  • editor edit format: editor-diff
  • command: aider --model o3
$0.1measured: $17.6 for the run, 175 of 224 solved,

$2 in · $8 outopenai, USD, observed

$0.048 one coding-assistant request at openaiestimate from the price observed ; assumptions

$0.008 one chat reply at openaiestimate from the price observed ; assumptions

  1. #1 o3 (high) + gpt-4.1

    Pass rate, second attempt
    78.2%,
    Run settings
    edit format: architect; reasoning effort: high; editor model: gpt-4.1; editor edit format: editor-diff; command: aider --model o3
    Cost per solved task
    $0.1 measured (175 of 224 solved),
    Price per 1M tokens

    $2 in · $8 out, openai, observed

    Workload estimate

    $0.048 one coding-assistant request at openai, from the price observed (assumptions)

    $0.008 one chat reply at openai, from the price observed (assumptions)

Source: Aider polyglot leaderboard. Each score links the published result it was read from.