This browser is not keeping saved items, so they will be forgotten when you leave this page. Everything else still works.

One ranking per benchmark

Benchmarks measure different things in different ways, so a score on one cannot be added to, or averaged with, a score on another. Each ranking orders results of a single benchmark, at a single version, by the one score that benchmark reports. Results that state no score are listed after the ranking as "Not ranked", never placed last with a zero. Equal scores share a place.

Models and agents are compared separately

A model evaluation runs different models through the same tool, so differences come mostly from the model. Anagent evaluation scores a complete system — its scaffold, its prompts and its model together — so a better score may come from any of them. The two are kept on separate pages and are never ranked together.

Aider polyglot (models)

Aider's polyglot benchmarkgives a model 225 Exercism coding exercises in C++, Go, Java, JavaScript, Python and Rust, through the aider coding tool. The score is the share of exercises whose tests pass after the model's second attempt, as aider reports it. Each run is listed with its own settings — edit format, reasoning settings, the exact command — because the same model can score differently with different settings; runs of one model are never merged. See the ranking.

What it cannot tell you: how a model does on your code, in another tool, or on tasks other than self-contained exercises; whether a model saw these public exercises in training; or how a run would score today, since each result is one run on the date shown.

SWE-bench Verified (agents and tools)

SWE-bench asks a system to resolve real issues from GitHub repositories; an issue counts as resolved when the tests SWE-bench runs pass. Only the Verified leaderboard is shown, and only the results SWE-bench marks as checked. Each result names the system, the models it used, and the number of attempts it was allowed. See the ranking.

What it cannot tell you: which part of a system — the scaffold or the model — earned its score; how it does on languages and kinds of work outside the benchmark's repositories; or what it costs, for the many submissions that state no cost.

Cost per solved task

A cost per solved task is shown only where one run published both what it cost and how many tasks it solved: the run's total cost divided by its solved tasks. Both numbers come from the same run, so they describe the same conditions. It reflects the prices when that run was made, not today's. Where a run states no cost it reads "Not measured" — aider records a cost of 0 when it could not price a run, and that is shown as not measured, never as free.

Prices

Prices are per million tokens, with input and output kept apart, and name the provider that charges them, the currency and the date we observed them. They come from theLiteLLM model price map, a list the LiteLLM project maintains — not each provider's own page — so check the provider before you rely on one. A price is shown only when the list's model id matches the id the run used exactly; otherwise the page says "No price listed".

Workload estimates

Without a measured cost, the only cost figure shown is an estimate: an observed price applied to a workload we assume. It is labelled as an estimate, and it says nothing about how many attempts a task needs or how many tokens a real task uses.

• One coding-assistant request: 20,000 input tokens and 1,000 output tokens (assumed, not measured).
• One chat reply: 2,000 input tokens and 500 output tokens (assumed, not measured).

Missing data

A figure a source does not state is shown as missing — "Not stated", "Not measured", "No price listed" — and is never replaced by a zero, an average or a guess.

Filters

Filters are plain links that narrow a ranking, for example to runs with a measured cost or one edit format. A filtered result keeps its place from the full ranking. "Cheapest per solved task first" orders by measured cost instead, among the runs that have one.

Sources and licences

Which sources are used, and why their terms allow it, is recorded in the project's benchmark source review. The SWE-bench leaderboard data is licensed CC BY-NC 4.0, the Aider leaderboard Apache-2.0 and the LiteLLM price map MIT.