AI & TECHNOLOGY / PRACTICAL GUIDE
How to Compare AI Model Performance: 5 Tasks
The best AI model depends on the task, your constraints and how much failure you can tolerate — no single leaderboard number settles that for you. Compare models on your own representative tasks, score the outputs against a rubric you wrote in advance, and only then look at which system produced which answer.
In this guide
Build five representative tasks, not one
A single clever prompt tells you almost nothing about how a model performs on your actual work. Build a small, fixed set of tasks that resembles what you'll really ask it to do, and reuse the exact same set every time you compare.
- A typical case: The kind of request you make most weeks — a summary, a draft reply, a rewrite.
- A difficult case: Something with ambiguity, missing context or a tricky edge case built in on purpose.
- A known failure case: A task type a model has struggled with before, so you can see whether that has changed.
- A task with a checkable answer: Something you can verify independently — a calculation, a fact, a piece of code that either runs or doesn't.
- A task with real source material: Give it a document to work from, so you can check whether it stayed inside the source or wandered.
Write the rubric before you run anything
Score criteria in advance, or you'll unconsciously grade toward whichever output you already like once you see the brand name attached to it.
| Criterion | What you're actually checking |
|---|---|
| Accuracy | Are the facts, numbers and claims correct and traceable to a source? |
| Completeness | Did it cover what was asked, or quietly skip the harder part of the task? |
| Style fit | Does it match the tone and format you actually need, not just "good writing" in general? |
| Latency | How long from request to a usable response, including any retries? |
| Cost per task | What did that specific call actually cost, at the volume you'd really run it? |
| Repair time | How many minutes of human editing did the output need before it was usable? |
Run the comparison and fill in the fields
For each task, run every model you're considering and record the same fields side by side. A blank template like this, filled in with your own numbers, is worth more than any public leaderboard, because it reflects your tasks and your acceptable failure rate, not a benchmark's.
| Field | What you record | Why it matters |
|---|---|---|
| Task and model | Which of your five tasks, run on which system | Keeps every score traceable back to a specific prompt |
| Accuracy score | Your rubric score, not a gut impression | Removes hindsight bias once you know which model wrote which answer |
| Latency | Seconds or minutes to a usable response | A fast wrong answer is not actually fast |
| Cost per call | Your actual billed cost for that request | Public pricing pages change often, so record what you were charged, not a rate you looked up once |
| Repair time | Minutes you personally spent editing before use | Often the real cost difference between two similarly priced models |
A worked scoring pass, with invented numbers
Here is what the template looks like once it is filled in. Everything below is made up to show the method. Model A and Model B are hypothetical systems, not real products, and the scores are illustrative numbers, not measurements of anything you can buy. The only real numbers are the ones you get by running your own five tasks.
Two of the five tasks, scored on three criteria. Accuracy and completeness are on a 1 to 5 scale against a rubric written in advance; repair time is minutes actually spent editing before the output was usable.
| Task | System | Accuracy (1-5) | Completeness (1-5) | Latency | Repair time |
|---|---|---|---|---|---|
| Summarize a 12-page report | Model A | 3 | 4 | 8 seconds | 14 minutes |
| Summarize a 12-page report | Model B | 5 | 4 | 41 seconds | 3 minutes |
| Fix a failing unit test | Model A | 4 | 3 | 6 seconds | 9 minutes |
| Fix a failing unit test | Model B | 4 | 5 | 35 seconds | 2 minutes |
On the summary task, Model A came back in eight seconds but got two figures wrong and attributed a recommendation to the wrong section, which took fourteen minutes to catch and correct against the source. Model B took five times longer to respond and matched the source on every figure, needing three minutes of tidying. On the test-fix task both were accurate, but Model A fixed the symptom and left a second failing case that the prompt had explicitly mentioned; Model B handled both, which is what the completeness gap records.
Side by side, the trade-off is plain. Model A wins on latency by a wide margin and loses on repair time by a wider one. If your work is a hundred quick, low-stakes rewrites an hour that you skim and move on from, a fast model is the right pick and the thirty-odd seconds of waiting per call would add up to real lost time. If it is a dozen summaries a day that other people will act on, the extra half-minute of waiting buys back eleven minutes of editing per task, and Model B wins on total time before you even count the errors that would have slipped through.
Cost per call would shift this further in one direction or the other, which is why it stays in the template even when the two prices look similar. The winner is whichever system scores best on the tasks you actually run, weighted by what each criterion costs you in practice. The method is the same whether your real numbers look anything like these or not.
Score blind, then run it twice
Hide which model produced which answer while you score, and run at least one of the five tasks more than once. One impressive answer can be luck; one bad answer can be an outlier — and either mistake is easy to make once you already know which system you're rooting for.
Record the real operating cost
Add up retries, human checking time and any workflow changes the output required, not just the per-call price. A model that's cheaper per request but needs heavy editing every time can cost more in practice than one with a higher sticker price and a lower repair-time score.
Worth remembering: Public pricing and benchmark leaderboards change often and can be shaped by the specific test set used to build them. Treat any number you didn't measure yourself as a starting estimate, not a fact to build a decision on.
What to do next
- Write your five tasks and your rubric today, before you open any model to test it.
- Run all five tasks on at least two systems, score them blind, then repeat one task to check for luck.
- Total the real cost — price, retries and repair time — before deciding which one actually won.
Sources and further reading
New research into practice: AI Productivity: Start the Clock Before the Prompt.