Model comparisons

Aider polyglot (225 exercises)

Ranked by pass rate, second attempt on Aider polyglot alone. No score from another benchmark is mixed in, and nothing is averaged. How this ranking is made, and what it cannot tell you.

Reference pages for what is ranked here: Qwen, DeepSeek, Llama, Mistral, Gemma, OpenAI, Anthropic, Google DeepMind

2 of 63 results: edit format — architect.

PlaceModelPass rate, second attemptRun settingsCost per solved taskPrice per 1M tokensWorkload estimate
18DeepSeek R1 + claude-3-5-sonnet-2024102264%
  • edit format: architect
  • editor model: claude-3-5-sonnet-20241022
  • editor edit format: editor-diff
  • command: aider --architect --model r1 --editor-model sonnet
$0.0923measured: $13.3 for the run, 144 of 225 solved, No price listedNone
46QwQ-32B + Qwen 2.5 Coder Instruct26.2%
  • edit format: architect
  • editor model: fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct
  • editor edit format: editor-diff
  • command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b --architect
Not measured

$0.9 in · $0.9 outfireworks_ai, USD, observed

$0.0189 one coding-assistant request at fireworks_aiestimate from the price observed ; assumptions

$0.00225 one chat reply at fireworks_aiestimate from the price observed ; assumptions

  1. #18 DeepSeek R1 + claude-3-5-sonnet-20241022

    Pass rate, second attempt
    64%,
    Run settings
    edit format: architect; editor model: claude-3-5-sonnet-20241022; editor edit format: editor-diff; command: aider --architect --model r1 --editor-model sonnet
    Cost per solved task
    $0.0923 measured (144 of 225 solved),
    Price per 1M tokens
    No price listed

  2. #46 QwQ-32B + Qwen 2.5 Coder Instruct

    Pass rate, second attempt
    26.2%,
    Run settings
    edit format: architect; editor model: fireworks_ai/accounts/fireworks/models/qwen2p5-coder-32b-instruct; editor edit format: editor-diff; command: aider --model fireworks_ai/accounts/fireworks/models/qwq-32b --architect
    Cost per solved task
    Not measured
    Price per 1M tokens

    $0.9 in · $0.9 out, fireworks_ai, observed

    Workload estimate

    $0.0189 one coding-assistant request at fireworks_ai, from the price observed (assumptions)

    $0.00225 one chat reply at fireworks_ai, from the price observed (assumptions)

Source: Aider polyglot leaderboard. Each score links the published result it was read from.