How to benchmark AI models on your own work, in six steps

Here is how to benchmark AI models for your own use: collect a few dozen real tasks with known good answers, run each model on them with identical settings, grade every output by the same rule, and log cost and time. Public leaderboards tell you which models deserve a test; your own set tells you which one to keep.

The six steps below need no research team.

1. Build a shortlist from public boards

Start where others have done the heavy lifting. Artificial Analysis reports an intelligence index alongside speed and cost per task. Arena ranks models from blind votes between two anonymous answers. For code, the SWE-bench and Terminal-Bench leaderboards score agents on real repositories and terminal work. Pick three or four models that rank well on the board closest to your job, and write down the date, because these standings move within weeks.

2. Write 20 to 50 tasks from your real work

Pull them from your own backlog: support tickets, SQL questions, contract clauses, bug reports, product copy. For each task, write the answer you would accept, or a short rubric if there is no single right answer. Mix easy and hard ones so the results spread out.

Keep the set private. Public test questions can end up in training data, which is why LiveCodeBench dates its problems and Terminal-Bench prints a canary line asking that its data never appear in training corpora. A set that only lives on your drive cannot leak that way.

3. Fix the settings

Use the same prompt, the same tools and the same effort level for every model, and record the exact model version. Run each task more than once, since outputs vary between runs. If one model gets web search and another does not, you are testing the setup, not the model.

4. Grade the same way every time

• For code, run the tests. That is how SWE-bench decides whether a patch worked.
• For facts and numbers, check against the answer key.
• For writing, score against your rubric with the model names hidden. Arena only counts votes cast while the names are hidden, and the same trick keeps your own bias out.

Keep a simple sheet: task, model, pass or fail, notes.

5. Move to a harness when the sheet gets big

Once you have more than a handful of tasks, a framework saves hours. Inspect, from the UK AI Security Institute and Meridian Labs, is open source, ships over 200 ready-made evaluations and has a log viewer for reading runs. EleutherAI's lm-evaluation-harness covers more than 60 academic benchmarks and suits open models you run yourself.

6. Count cost and speed, then rerun

A model that wins by a point but costs three times as much may not be the winner. Note tokens, price and time per task. ARC Prize plots its puzzle scores against cost per task for the same reason. Then rerun the whole set whenever a model updates.

Use the nerf rumors as your rerun alarm

There is a second trigger worth adding. Every few weeks the timeline decides a model "got dumber", and TechTwitter IO tracks exactly that. Its free weekly satire boards include Feel, which nets good and bad vibe posts on X and raises a Nerf Watch flag when the complaints come in a wave.

When that flag goes up on a model you rely on, rerun your fixed set. If the scores hold, you can ignore the noise. If they drop, you have evidence instead of a vibe. Treat TechTwitter IO as the alarm, not the measurement: it is satire, and its boards count posts, not test results.

If you lack the time to build a set at all, a broad published index is a better default than guessing. But even ten tasks of your own will tell you more about your work than any leaderboard.