Model comparisons

Aider polyglot (225 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind

4 of 63 results: edit format — diff-fenced.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
4gemini-2.5-pro-preview-06-05 (32k think)83.1%
  • edit format: diff-fenced
  • thinking tokens: 32768
  • command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
$0.267measured: $49.9 for the run, 187 of 225 solved, No price listedNone
8gemini-2.5-pro-preview-06-05 (default think)79.1%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-06-05
$0.256measured: $45.6 for the run, 178 of 225 solved, No price listedNone
9Gemini 2.5 Pro Preview 05-0676.9%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-05-06
$0.216measured: $37.4 for the run, 173 of 225 solved, No price listedNone
12Gemini 2.5 Pro Preview 03-2572.9%
  • edit format: diff-fenced
  • command: aider --model gemini/gemini-2.5-pro-preview-03-25
Not measuredNo price listedNone

  1. #4 gemini-2.5-pro-preview-06-05 (32k think)

    Pass rate, second attempt
    83.1%,
    Run settings
    edit format: diff-fenced; thinking tokens: 32768; command: aider --model gemini/gemini-2.5-pro-preview-06-05 --thinking-tokens 32k
    Cost per solved task
    $0.267 measured (187 of 225 solved),
    Price per 1M tokens
    No price listed

  2. #8 gemini-2.5-pro-preview-06-05 (default think)

    Pass rate, second attempt
    79.1%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-06-05
    Cost per solved task
    $0.256 measured (178 of 225 solved),
    Price per 1M tokens
    No price listed

  3. #9 Gemini 2.5 Pro Preview 05-06

    Pass rate, second attempt
    76.9%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-05-06
    Cost per solved task
    $0.216 measured (173 of 225 solved),
    Price per 1M tokens
    No price listed

  4. #12 Gemini 2.5 Pro Preview 03-25

    Pass rate, second attempt
    72.9%,
    Run settings
    edit format: diff-fenced; command: aider --model gemini/gemini-2.5-pro-preview-03-25
    Cost per solved task
    Not measured
    Price per 1M tokens
    No price listed

Source: Aider polyglot leaderboard. Each score links the published result it was read from.