Benchmarks

When a company releases a new AI model, the announcement almost always includes a chart. The chart has a few pairs of bars. In each pair, the bar for the new model is longer than the bar for the previous one. The message is that the new model is better.

The numbers behind those bars come from benchmarks. A benchmark is a standard test: a fixed set of questions or tasks that is given to different models so that their scores can be compared.

Benchmarks are useful, and the scores are usually real. The open question is how much a score says about the work you want a model to do. This lesson explains where benchmark numbers come from, who chooses which ones you see, how a score can be raised without the model getting better in general, and what a score can and cannot tell you.

After this lesson you can read a launch chart critically and explain why trying a model on your own tasks is the only reliable comparison.

1. What a benchmark is

Suppose you want to compare two students. You give both the same exam with the same 100 questions and count the correct answers. One gets 82 and the other gets 74. Because the exam was identical, the two numbers can be compared.

A benchmark does this for AI models. It has three parts:

• A fixed list of questions or tasks. The list does not change from model to model. It might hold maths problems, programming tasks, or questions about a text.
• A known correct answer for each one, or another fixed way of checking the result.
• A scoring rule, such as "the percentage of questions answered correctly".

There are many benchmarks, each for a different ability: reasoning, writing code, answering science questions, understanding images, and others.

2. The numbers arrive with the model

Benchmark results usually appear in the same week as the model launch, and they come from the company that is launching the model.

A launch chart tends to have the same shape every time. The names and bar lengths below are for illustration only and are not real results.

                 what came before     the new model

benchmark A      ##########           #################
benchmark B      ###########          #################
benchmark C      #########            ##################

On every line, the new model has the longer bar.

Two facts about this chart matter more than the bar lengths.

First, the company that built the model ran the tests and made the chart. The seller is reporting on its own product.

Second, the company chose which benchmarks appear on the chart. There are dozens it could have run. Three were printed. The chart does not say how the model did on the others, or whether they were run at all.

3. Published results are selected

A company announcing a model has little reason to print a chart on which its own model loses. So the charts that get published are, with few exceptions, the ones where the new model wins.

This is why launch charts from different companies all look alike. Each one shows the new model ahead. That uniformity is not evidence that every model is the best. It is the result of each company choosing what to show.

Showing the results that look good and leaving out the others is called selective reporting. It does not require any false number. Every bar on the chart can be accurate, and the overall picture can still mislead, because the reader does not see what was left out.

Selective reporting is a strong tendency, not an absolute rule. Some companies publish longer technical reports that include weaker results. A launch chart is the short, favourable version.

4. Why the benchmark lists never match

Take the launch posts of two competing models and compare which benchmarks each one shows. The letters stand for benchmarks and are used for illustration only.

The first post shows three benchmarks, A, C, and D, and its model wins all three. The second post also shows three, B, C, and F, and its model wins those. The two lists share only one benchmark, C.

Now read the table for what is missing:

• Benchmark B is missing from the first post.
• Benchmark A is missing from the second post.
• Benchmark E is missing from both.

The absences are the information here. A reader cannot compare the two models on A or on B, because each company printed only its own result. Nothing at all is known about E. The reader cannot tell whether a missing benchmark was left out because the model did poorly on it, because it was never run, or for some unrelated reason.

So two launch posts cannot be laid side by side to pick a winner. Each is a different selection.

One way around this is an independent comparison, where a third party runs the same benchmarks on many models under the same conditions. Such comparisons exist and remove the selection by the seller. They do not remove the other limits described below.

5. Why a score can be raised

Most benchmarks are published. The list of questions is available to anyone who wants to read it. That openness is useful, because other people can check the results. It also has a side effect.

flowchart LR
    M["The model,<br/>tuned for the benchmark"] -- "practises exactly these" --> B["The benchmark<br/>a fixed list of questions,<br/>published and readable by anyone"]
    E["Everything else you might ask<br/>no list, no launch chart, no tuning"]

The diagram has two areas. The first is the benchmark, a small fixed list, with an arrow from a model that has been adjusted to do well on it. The second is the box with no arrow: everything else a person might ask a model. That area has no list, appears on no chart, and nobody tuned the model for it. It is also the area where users spend all their time.

Because the questions are fixed and known, a model's score on them can be pushed up in ways that do not improve it elsewhere. There are two main routes:

• Tuning. The developers adjust the model or its training with a particular benchmark in mind, so that it gets good at that style of question.
• Contamination. The benchmark questions and their answers are on the public internet. Models are trained on very large amounts of internet text. So the questions can end up in the training data, and the model may have seen the answers before the test. This can happen without anyone intending it.

An everyday comparison: a student who has seen the exam paper in advance can score very well. The mark is real, since the answers are correct. It says less about how the student would do on a different paper covering the same subject. The comparison is not exact, because a model does not always reproduce what it saw in training, but the effect on the score points in the same direction.

Doing well on a benchmark is a real achievement. The model did answer those questions correctly. What the score cannot show is how much of that carries over to questions that are not on the list.

Some benchmark makers respond by keeping part of their questions secret or by replacing the questions regularly. This reduces the problem. Benchmarks also wear out in another way: once most strong models score near the top, the test no longer separates them, and a new, harder one is needed.

6. The test and the job

Even a benchmark that nobody tuned for has a built-in gap. The questions on the list are a different kind of thing from the questions people bring to a model at work.

A benchmark question has to be clean. It needs one correct answer so that it can be marked automatically by a script, which is a small program that compares each answer with the expected one. Everything needed to answer is contained in the question.

Real requests are not like this. A person pastes in three files and writes "can you sort this out". The wording is loose. Half of the relevant background exists only in the person's head. There may be several acceptable results and no single correct one. The outcome is judged by one standard: whether it helped.

Two examples show the gap. A model that scores highly on a programming benchmark has solved small, self-contained, clearly described tasks. A developer's actual task may be a vague bug report in a large, old codebase with its own unwritten rules. A model that scores highly on exam-style questions in some subject has picked correct answers from options. A professional's actual task may be a long conversation with incomplete information. The benchmark measured something real in both cases. It measured something narrower than the job.

7. What a benchmark result tells you

A launch chart shows somebody else's result: a particular model, on particular questions, selected and run by the company that sells it.

That result may hold up on your work, and it may not. Both outcomes remain possible after reading the chart. The model that leads on three benchmarks might also be the right one for your tasks. A model lower on the chart might suit your tasks better, cost less, or respond faster.

This suggests a practical way to use benchmarks:

1. Use benchmarks to make a shortlist. Scores are a reasonable first filter. A model that does poorly across many independent benchmarks is unlikely to be the right choice for demanding work.
2. Look at what was tested. A benchmark for writing code says little about summarising legal documents. Prefer benchmarks close to your kind of task.
3. Check who ran the test. A result from an independent party carries more weight than one from the seller.
4. Try the shortlisted models on your own tasks. Collect a handful of real examples from your work and run each model on them. This is the only run that settles the question for you, and it is the one that has not been done until you do it.

Step 4, done in an organised and repeatable way, is called an eval. That is the subject of the next lesson.

Interview points

• A benchmark is a fixed set of questions or tasks with a scoring rule, used to compare models on the same footing.
• Launch benchmark results are produced and selected by the company that built the model, so a launch chart shows results in which the new model does well.
• Selective reporting needs no false numbers. Accurate scores can still mislead when unfavourable results are left out.
• Different launch posts show different sets of benchmarks with little overlap, so they cannot be compared directly. Which benchmarks are absent is itself information.
• Most benchmarks are public, so a model can be tuned for them, and their questions can enter training data. Both raise the score without a matching gain on other tasks.
• A high benchmark score is real for the questions on the list. It does not show how the model performs on everything that is not on the list.
• Benchmark questions are clean, complete, single-answer, and marked automatically. Real requests are loosely worded, depend on unstated background, and are judged by usefulness.
• Benchmarks are suited to making a shortlist. The deciding comparison is running the candidate models on examples from one's own work.

Summary

• A benchmark is a standard test that gives every model the same questions.
• The scores shown at a launch are chosen by the model's maker.
• Published charts almost always show the new model winning.
• Two companies rarely report the same set of benchmarks.
• Public questions can be practised and can leak into training data.
• A benchmark covers a small fixed list, and real use covers everything else.
• Benchmark questions are clean and scored, and real requests are messy and unscored.
• A benchmark result may or may not carry over to your work.
• Testing a model on your own tasks is what settles the choice.