Model comparisons

Aider polyglot (223 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: OpenAI

1 of 1 results: cost — cheapest per solved task first. Ordered by measured cost per solved task, cheapest first; the place shown is still the place by score.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
1chatgpt-4o-latest (2025-02-15)27.1%
  • edit format: diff
  • command: aider --model chatgpt-4o-latest
$0.24measured: $14.4 for the run, 60 of 223 solved, No price listedNone

  1. #1 chatgpt-4o-latest (2025-02-15)

    Pass rate, second attempt
    27.1%,
    Run settings
    edit format: diff; command: aider --model chatgpt-4o-latest
    Cost per solved task
    $0.24 measured (60 of 223 solved),
    Price per 1M tokens
    No price listed

Source: Aider polyglot leaderboard. Each score links the published result it was read from.