Evals

A company builds an AI assistant that answers customer emails. Before it goes live, someone has to check that its replies are good. The usual way to test software does not work here, because there is no single correct reply to compare against. Two replies can both be fine and have no words in common.

An eval, short for evaluation, is a test built for this situation. It runs the AI system on a prepared set of inputs and then judges each output against written requirements. The judging is usually automatic, and it is often done by a second model.

The previous lesson showed that a benchmark score describes a model in general and may not carry over to a particular job. An eval is the answer to that problem. It measures one specific system on one specific job.

After this lesson you can explain why ordinary tests fall short for generated text, list the parts of an eval, follow one run from start to finish, and estimate what a run costs.

1. Why the usual kind of test falls short

Ordinary software is tested by comparing a result with a known correct value. Here is such a test for a shopping cart:

assert total(cart) == 42

The word assert begins an assertion, a line that states what must be true. This one says: the total of the cart must equal 42. The program computes the total, and the test compares it with 42.

• There is one right answer.
• The test passes or it fails.
• Nobody has to judge anything.

Now try to write the same kind of test for a generated reply:

assert reply == ???

There is nothing to put in place of the question marks. Suppose a customer asks where a parcel is. "It ships Thursday." is a good reply. "Your order will be sent out on Thursday." is also a good reply. The two share almost no words. A test that demands one exact sentence would fail the other, even though both are correct.

The same input can also produce different wording on different runs, because a model picks its words with some randomness (see the lesson on temperature and sampling).

So a test for generated text cannot match a value. It has to measure how the system behaves: whether the reply answers the question, whether it stays within the rules, whether its tone is right. That kind of test is an eval.

2. Writing down what good behaviour means

Nothing can be measured until someone states what behaving well means for this system. This is the first step of building an eval, and it is done by people.

For the email assistant, the requirements might be written in a file like this:

support-replies.eval.yml

- check: answers the question asked
  kind:  capability
- check: no invented order numbers
  kind:  reliability
- check: never promises a refund
  kind:  safety
- check: polite, and kept short
  kind:  product

The file ends in .yml, which marks it as YAML, a plain-text format for lists and settings. Each entry has two lines. The check line is one requirement in ordinary words. The kind line says what sort of requirement it is.

Requirements commonly fall into five kinds:

The sample file shows four of the five kinds. A quality check would be one more entry of the same form.

Each of these checks is a line that somebody sat down and wrote. They come from knowing the business. "Never promises a refund" is in the file because someone at this company knows that only a human agent may approve refunds. "No invented order numbers" is there because models can produce made-up details that look real (see the lesson on hallucinations). A general model does not know these rules, and no benchmark tests for them.

3. A test of the whole system

Software developers distinguish two kinds of tests. A unit test checks one small piece of a program in isolation, such as the function that adds up a cart. An integration test runs many pieces together and checks that the combined result is right.

An eval is like an integration test. It puts prepared input in at one end, lets the whole system run, and makes a judgement about what comes out at the other end.

What is being tested is everything that was built: the system prompt, the documents the system retrieves, the tools it can call, the model that was chosen, and the settings. A fault in any of these shows up in the output, and the eval sees it there.

4. One run from start to finish

An eval run has four stages.

flowchart LR
    A["Dummy customer emails<br/>written once,<br/>reused every run"] --> B["Your system<br/>exactly as it ships"]
    B --> C["The answers<br/>it produced"]
    C --> D["A second model<br/>reads and scores<br/>each one"]
    D --> E["Results<br/>pass or flagged,<br/>per check"]

Stage 1: the test inputs. Someone writes a set of made-up customer emails:

refund for order 7781
where is my parcel?
cancel my plan please

This set is the test set. It is written once and reused in every run. Using the same inputs each time is what makes two runs comparable. A good test set covers the common requests and also the difficult ones, such as an angry customer or a request the assistant must refuse.

Stage 2: the system runs. The emails are fed to the system exactly as it will be shipped, with the real prompt, the real data, and the real settings. Testing a simplified version would measure something that customers never meet.

Stage 3: the answers. The system produces a reply for each email:

"Refund is on its way."
"It ships Thursday."
"Plan cancelled."

Stage 4: the judging. In an automated run, no person reads these answers. A second model does. This judge model receives each reply together with the checks from the requirements file and scores the reply against each check:

answers the question asked       pass
invented an order number         flagged

One reply answered what was asked, so it passes. One contained a made-up order number, so it is flagged for attention. The first reply above shows a third problem: "Refund is on its way." promises a refund, which the safety check forbids. The judge would flag that too.

A model can do this job because judging is usually a narrower task than writing. The judge is not asked to produce a good reply. It is asked a narrow question about a reply that already exists, such as "does this text promise a refund, yes or no".

5. Other ways of scoring

A judge model is the common choice when the requirement needs understanding of language. It is not the only way to score an output, and three methods are used side by side.

Code checks are the cheapest and are exact, so they are used wherever a requirement can be stated as a fixed rule. A judge model handles what a rule cannot express.

A judge model can be wrong. It is a model and makes the same kinds of mistakes as other models. For this reason, teams compare the judge's scores with a person's scores on a sample before trusting it, and repeat that comparison from time to time.

6. A score for your setup, not for the model

An eval answers a different question from a benchmark. The question is not how good the model is. It is how good this system is at this job.

Two companies can use the same model and get very different eval results. One has a carefully written prompt and accurate product data. The other has a vague prompt and outdated data. The benchmark score of the model is identical for both. Their eval scores are not, because an eval measures the model together with everything built around it.

This also makes evals the tool for comparing models for a particular use. Swap in a different model, run the same eval, and compare the two results on your own test set.

7. What one run costs

An eval run costs money, because every step that uses a model is a paid call. With a judge model, each test input is paid for twice: once to produce the answer and once to mark it.

Here is one run with illustrative numbers:

The exact counts do not matter. The pattern does. When the system makes one model call per email and the judge makes one call per answer, adding a judge doubles the number of model calls. The bill roughly doubles too, though not exactly, because the two calls can differ in length and in price.

A worked example with made-up prices shows the scale. Suppose answering one email costs 1 cent and judging one answer costs 1 cent.

one run:        50 answers x 1 cent  +  50 judgements x 1 cent  =  100 cents  =  $1
20 runs a day:  20 x $1  =  $20 per day

The total grows because the eval is run again after every change, and because real test sets often hold hundreds or thousands of inputs. If a system makes several model calls to answer one email, as an agent does, the cost per input rises further.

Common ways to control the cost are to use a smaller, cheaper model as the judge where it is accurate enough, to replace judge checks with code checks where a rule will do, and to run a small test set on every change and the full set less often.

8. What an eval gives you

An eval is an automated way of testing something that has no single right answer.

Because it is automated, it can be run on every change, in the same way as any other set of software tests. Someone rewrites a line of the prompt, switches to a newer model, or updates the product data, and the eval runs again.

This is how a regression is caught. A regression is something that used to work and stops working after a change. Suppose a prompt edit that was meant to make replies friendlier also makes the assistant start promising refunds. Without an eval, the first sign is a customer complaint. With an eval, the safety check is flagged on the next run, before any customer sees the reply.

An eval is only as good as its parts. If a requirement was never written down, it is never checked. If the test set lacks a kind of customer email, behaviour on that kind is unknown. So an eval grows over time: each new failure found in real use becomes a new input in the test set or a new check in the file.

Interview points

• An ordinary test compares an output with one expected value. Generated text has no single expected value, since two correct replies can differ in every word, so the test has to measure behaviour.
• An eval runs a system on a fixed test set and scores each output against written requirements.
• The requirements must be written by people who know the use case. Common kinds are capability, quality, safety, reliability, and product behaviour.
• An eval resembles an integration test, not a unit test. The thing under test is the whole system: prompts, data, tools, model, and settings, exactly as shipped.
• A run has four stages: reusable test inputs, the system under test, the outputs, and scoring. Scoring is often done by a second model that acts as judge.
• Scoring methods include code checks for fixed rules, a judge model for requirements that need language understanding, and human review to validate the judge.
• A benchmark gives one general score for a model. An eval gives a score for a specific setup on a specific use case, so the same model can score differently in two systems.
• With one call to answer each input and one call to judge each answer, a judge model doubles the number of model calls per run. Runs are repeated often, so cost has to be planned.
• Running evals on every change exposes regressions before customers meet them.

Summary

• An eval tests an AI system whose outputs have no single correct wording.
• It scores behaviour against written checks.
• Someone has to write down what good behaviour means first.
• The whole system is tested, not one part of it.
• The same test inputs are reused in every run.
• A second model often does the judging.
• The score describes your setup, not the model in general.
• Each input is paid for twice when a judge model is used.
• Evals run on every change and catch regressions early.