Lesson 65 of 73
Evaluating Models and Agents
You will often change something about a coding agent: a new model, a different quantization (a smaller way of storing the model's numbers, covered in Making, Quantizing, and Testing Your Own GGUF), a shorter system prompt, or a new edit tool. Each change might help, hurt, or do nothing. Trying the agent on one task and watching it succeed does not tell you which, because an agent is a program that asks a model what to do, and the model's answers contain randomness. One lucky run proves very little. Only a measurement over many tasks can tell.
This lesson builds that measurement in Java. It starts with two numbers that judge a model on plain text (perplexity and KL divergence). It then moves to the statistics that turn a few dozen pass or fail results into a number you can defend, the machinery that runs an agent on tasks whose tests it cannot see, and Project 28, the loop eval command. Along the way you will learn to read a public benchmark table critically, because every benchmark measures one narrow thing.
After this lesson you can read a benchmark score and say what it does and does not show, build a task suite for your own code base, and decide whether a difference between two configurations is real or noise.
| Idea | What it is used for |
|---|---|
| Perplexity, KL divergence | How well a model predicts text, with no task involved |
| pass@k and its unbiased estimator | The chance that at least one of k attempts solves a task |
| Public coding benchmarks | What each well-known test set measures, and what it leaves out |
| Contamination | Test problems that leaked into the model's training data |
| Task suite, runner, hidden tests | Measuring an agent on your own tasks |
| Pass rate, steps, tokens, time, cost | The numbers one evaluation run produces |
| Transcripts and replay | Debugging a run, and re-scoring it, without calling the model again |
| Wilson interval, paired comparison | Telling a real difference from luck |
| Spans | Where time and tokens go inside one run |
Every program here runs offline and prints the same output every time. Model behavior comes from fixed numbers, or from a scripted agent that uses a seeded random generator. A seeded generator always produces the same sequence of "random" numbers, so a demo that looks random is still repeatable. The real tools that produce such numbers are named where they appear.
Example 1: perplexity and KL divergence, a recap with five tokens
A language model reads text and, at every position, gives a probability to each token that might come next (a token is a small piece of text, such as int or ;). What a Language Model Computes introduced this, and Making, Quantizing, and Testing Your Own GGUF measured it with the llama-perplexity tool. This example repeats both measurements by hand on a tiny case, so every number in a llama-perplexity report has a small counterpart you can check with a calculator.
There are two measurements, and each answers a different question.
Perplexity asks how surprised a model is by real text. Suppose the text is int x = 0; and, after int, a model gives the token x a probability of 0.8. That is a small surprise. If it had given 0.01, that would be a large surprise. Surprise is measured as the negative natural logarithm of the probability, -ln(p) (ln is the natural logarithm, the Math.log method in Java). Perplexity averages the surprise over all positions and then applies exp to turn it back into a readable scale. A perplexity of 1.64 means the model was about as unsure as someone choosing among 1.64 equally likely tokens at each step. Lower is better.
KL divergence (Kullback-Leibler divergence) asks how different two probability distributions are. Here the two distributions are the full set of probabilities from the original model (the base model) and from its quantized copy, at one position. For distributions p (base) and q (quantized) over the same tokens, the formula is the sum over tokens of p * ln(p / q). Here is a two-token case worked by hand:
p (base) = 0.8, 0.2
q (quantized) = 0.7, 0.3
KL = 0.8 * ln(0.8 / 0.7) + 0.2 * ln(0.2 / 0.3)
= 0.8 * 0.1335 + 0.2 * (-0.4055)
= 0.1068 - 0.0811
= 0.0257If the two distributions are identical, every ln(p / q) is ln(1) = 0, so the KL divergence is 0. The more they differ, the larger the number. The program below uses a five-token vocabulary, int, x, =, 0, ;, and a four-position text. It prints, for every position, the probability each model gave to the correct next token, the KL divergence, and each model's favorite token. Look at position 3 in the output, where the two models disagree.
import java.util.Locale;
public class PerplexityAndKl {
static double[] softmax(double[] z) {
double max = Double.NEGATIVE_INFINITY, sum = 0;
for (double v : z) max = Math.max(max, v);
double[] p = new double[z.length];
for (int i = 0; i < z.length; i++) { p[i] = Math.exp(z[i] - max); sum += p[i]; }
for (int i = 0; i < z.length; i++) p[i] /= sum;
return p;
}
static int argmax(double[] p) {
int best = 0;
for (int i = 1; i < p.length; i++) if (p[i] > p[best]) best = i;
return best;
}
public static void main(String[] args) {
String[] vocab = {"int", "x", "=", "0", ";"};
int[] next = {1, 2, 3, 4}; // after "int" comes x, =, 0, ;
double[][] base = {{0.1, 3.0, 0.2, 0.5, 0.1}, {0.0, 0.3, 2.5, 0.1, 1.0}, {0.2, 0.4, 0.1, 1.8, 1.5}, {0.1, 0.2, 0.3, 0.6, 2.2}};
double[][] quant = {{0.1, 2.9, 0.3, 0.5, 0.1}, {0.1, 0.4, 2.3, 0.1, 1.1}, {0.2, 0.4, 0.1, 1.5, 1.7}, {0.1, 0.2, 0.3, 0.7, 2.1}};
double nllBase = 0, nllQuant = 0, klSum = 0;
int sameTop = 0;
for (int i = 0; i < next.length; i++) {
double[] p = softmax(base[i]), q = softmax(quant[i]);
nllBase -= Math.log(p[next[i]]);
nllQuant -= Math.log(q[next[i]]);
double kl = 0;
for (int t = 0; t < p.length; t++) kl += p[t] * Math.log(p[t] / q[t]);
klSum += kl;
if (argmax(p) == argmax(q)) sameTop++;
System.out.printf(Locale.ROOT, "position %d next=%-2s p_base=%.3f p_quant=%.3f KL=%.5f top base/quant = %s/%s%n",
i + 1, vocab[next[i]], p[next[i]], q[next[i]], kl, vocab[argmax(p)], vocab[argmax(q)]);
}
int n = next.length;
double pplBase = Math.exp(nllBase / n), pplQuant = Math.exp(nllQuant / n);
System.out.printf(Locale.ROOT, "PPL base = %.4f, PPL quant = %.4f, ln(PPL(Q)/PPL(base)) = %.4f%n", pplBase, pplQuant, Math.log(pplQuant / pplBase));
System.out.printf(Locale.ROOT, "mean KLD = %.5f, same top token = %.1f%%%n", klSum / n, 100.0 * sameTop / n);
}
}position 1 next=x p_base=0.798 p_quant=0.777 KL=0.00148 top base/quant = x/x
position 2 next== p_base=0.664 p_quant=0.598 KL=0.00942 top base/quant = =/=
position 3 next=0 p_base=0.422 p_quant=0.325 KL=0.02307 top base/quant = 0/;
position 4 next=; p_base=0.621 p_quant=0.589 KL=0.00257 top base/quant = ;/;
PPL base = 1.6384, PPL quant = 1.8302, ln(PPL(Q)/PPL(base)) = 0.1107
mean KLD = 0.00913, same top token = 75.0%Each position line compares the two models at one step of the text:
p_baseandp_quantare the probabilities each model gave to the token that actually came next. Perplexity uses only these numbers. At position 3 the base model gave0a probability of 0.422 and the quantized model only 0.325.KLis the KL divergence of the quantized distribution from the base distribution at that position: the sum over all five tokens ofp * ln(p / q). It uses the whole distribution, not only the actual next token, so it notices changes in tokens that did not occur. It is 0 when the two distributions are equal and grows as they differ.top base/quantis each model's most likely token. At position 3 they disagree: the base model's top token is0, the quantized model's is;. With greedy decoding (always picking the most likely token), the quantized model would have written a different program from this point on.
The perplexity arithmetic for the base model uses the four p_base values. The surprise at each position is -ln(p): about 0.226, 0.410, 0.863, and 0.476. Their mean is about 0.494, and exp(0.494) is about 1.64, which matches PPL base = 1.6384 (the printed probabilities are rounded, so a hand calculation is only close). The quantized model's four probabilities are lower on average, so its mean surprise is higher and its perplexity is 1.8302.
The summary lines are the numbers llama-perplexity prints:
- PPL is
expof the mean negative log-probability of the actual tokens. The quantized model's perplexity is higher, so it predicts this text slightly worse.ln(PPL(Q)/PPL(base))is the same comparison as a difference of mean log-losses (the average surprise of the quantized model minus that of the base model), the column thatllama-perplexity --kl-divergencereports next to each chunk. Here it is 0.1107, which isln(1.8302 / 1.6384). - Mean KLD is the average of the four KL values.
- Same top token is the share of positions where both models pick the same most likely token, the "Same top p" figure in the
llama-perplexitystatistics. 75% here means that the models disagree at one position out of four.
These measurements need no task and no tests, and they are cheap: one pass over a text file. Their limit for agents is that they measure text prediction, not task success. A quantization with a small KL divergence can still break tool calling if the tokens it changes are the ones that open a tool call or close a JSON string. A tiny change in probabilities can be harmless in prose and fatal in a format that must be exact. That is why the rest of this lesson measures agents on tasks.
Example 2: pass@k and its unbiased estimator
A coding task is either solved or not, which the tests decide. A model that samples (picks tokens with some randomness, see What a Language Model Computes) can fail on one attempt and succeed on the next, so a single attempt is a random event. pass@k is the probability that at least one of k independent attempts passes the tests. For k = 1 it is the plain chance that one attempt works.
The direct way to measure pass@k is to run k attempts, record whether any passed, and repeat many times. That wastes samples. The HumanEval paper (Chen et al., 2021) uses a cheaper idea: generate n attempts per task, where n is at least k, and count the c that pass. Then compute the probability that a random group of k attempts, chosen from the n that exist, contains at least one passing attempt.
Here is a small case. A task has n = 10 attempts and c = 3 passed. Choose k = 2 attempts at random. The group fails only if both chosen attempts come from the 7 failing ones:
C(a, b) = "a choose b" = the number of ways to pick b items out of a
C(7, 2) = 7 * 6 / 2 = 21 ways to pick 2 failing attempts
C(10, 2) = 10 * 9 / 2 = 45 ways to pick any 2 attempts
P(no pass in the group) = 21 / 45 = 0.4667
pass@2 = 1 - 0.4667 = 0.5333The general formula is 1 - C(n-c, k) / C(n, k). This example computes it in two ways and checks both against a simulation that really draws groups of attempts. One way is the exact formula with big integers. The other is the numerically stable product form from the HumanEval paper, 1 - prod(1 - k/i) over i from n-c+1 to n. For the case above that is 1 - (1 - 2/8) * (1 - 2/9) * (1 - 2/10) = 1 - 0.75 * 0.7778 * 0.8 = 0.5333, the same answer. The program also prints a naive formula, 1-(1-c/n)^k, so you can see why it is not used.
import java.math.BigDecimal;
import java.math.BigInteger;
import java.math.MathContext;
import java.util.Locale;
import java.util.Random;
public class PassAtK {
/** Numerically stable form from the HumanEval paper: 1 - prod_{i=n-c+1..n} (1 - k/i). */
static double passAtK(int n, int c, int k) {
if (n - c < k) return 1.0;
double prod = 1.0;
for (int i = n - c + 1; i <= n; i++) prod *= 1.0 - (double) k / i;
return 1.0 - prod;
}
static BigInteger choose(int a, int b) {
if (b < 0 || b > a) return BigInteger.ZERO;
BigInteger r = BigInteger.ONE;
for (int i = 0; i < b; i++) r = r.multiply(BigInteger.valueOf(a - i)).divide(BigInteger.valueOf(i + 1));
return r;
}
static double exact(int n, int c, int k) {
BigDecimal ratio = new BigDecimal(choose(n - c, k)).divide(new BigDecimal(choose(n, k)), MathContext.DECIMAL64);
return 1.0 - ratio.doubleValue();
}
public static void main(String[] args) {
int n = 10, c = 3;
Random random = new Random(42);
System.out.println("n = 10 attempts, c = 3 passed");
for (int k : new int[] {1, 2, 5, 8}) {
int hits = 0, trials = 100_000;
for (int t = 0; t < trials; t++) {
boolean[] chosen = new boolean[n];
boolean any = false;
for (int picked = 0; picked < k; ) {
int i = random.nextInt(n);
if (chosen[i]) continue;
chosen[i] = true;
picked++;
any |= i < c; // attempts 0..c-1 are the passing ones
}
if (any) hits++;
}
double naive = 1 - Math.pow(1 - (double) c / n, k);
System.out.printf(Locale.ROOT, "k=%d estimator=%.4f exact=%.4f simulated=%.4f 1-(1-c/n)^k=%.4f%n",
k, passAtK(n, c, k), exact(n, c, k), (double) hits / trials, naive);
}
int[] passedPerTask = {0, 3, 10};
for (int k : new int[] {1, 5}) {
double sum = 0;
for (int ci : passedPerTask) sum += passAtK(10, ci, k);
System.out.printf(Locale.ROOT, "suite of 3 tasks (c = 0, 3, 10): pass@%d = %.4f%n", k, sum / passedPerTask.length);
}
System.out.printf(Locale.ROOT, "n=200, c=13: pass@1=%.4f pass@10=%.4f pass@100=%.4f%n",
passAtK(200, 13, 1), passAtK(200, 13, 10), passAtK(200, 13, 100));
}
}n = 10 attempts, c = 3 passed
k=1 estimator=0.3000 exact=0.3000 simulated=0.3011 1-(1-c/n)^k=0.3000
k=2 estimator=0.5333 exact=0.5333 simulated=0.5328 1-(1-c/n)^k=0.5100
k=5 estimator=0.9167 exact=0.9167 simulated=0.9160 1-(1-c/n)^k=0.8319
k=8 estimator=1.0000 exact=1.0000 simulated=1.0000 1-(1-c/n)^k=0.9424
suite of 3 tasks (c = 0, 3, 10): pass@1 = 0.4333
suite of 3 tasks (c = 0, 3, 10): pass@5 = 0.6389
n=200, c=13: pass@1=0.0650 pass@10=0.4976 pass@100=0.9999The four k= lines answer the question for one task with 3 passing attempts out of 10:
estimatorandexactagree to four decimals. The product form avoids the huge binomial coefficients of the exact form (C(200, 100)has 59 digits, too large for adouble), which is why the HumanEval code uses it.simulateddrawskdistinct attempts from the ten 100,000 times and counts how often at least one of the three passing attempts is among them. It matches the estimator to within about 0.002, the sampling noise of 100,000 trials. The estimator is unbiased: on average over many tasks, it equals the true pass@k. It does not lean high or low.- For
k=1the estimator isc/n = 0.3, the plain pass rate. pass@1 is the number most leaderboards report. 1-(1-c/n)^ktreats the attempts as draws with replacement from a pass rate of 0.3. It is lower than the estimator for everyk > 1. The two disagree because the estimator samples without replacement from the 10 attempts that actually happened. Withk = 8, choosing 8 of 10 leaves out only 2, so at least one of the 3 passing attempts must be chosen; the estimator says exactly 1.0, while the naive formula says 0.94. In the code, this is the lineif (n - c < k) return 1.0: there are only 7 failing attempts, and a group of 8 cannot consist of failures alone.
The suite line averages per task, which is how a benchmark reports pass@k. A task that never passes contributes 0, a task that always passes contributes 1, and the 3-of-10 task contributes its own value. For pass@1 that is (0 + 0.3 + 1) / 3 = 0.4333. pass@5 is higher, 0.6389, because the middle task has five chances: (0 + 0.9167 + 1) / 3.
The last line uses n = 200, the number of samples per problem in the HumanEval paper's main results. With 13 passes, pass@1 is 0.065, while pass@100 is above 0.99. The gap is the reason a reported pass@k must name k. An agent that has one attempt per task (the normal case for a coding agent working on a user's request) is described by pass@1. pass@100 describes a system that can run 100 attempts and pick a good one using a perfect test oracle (a checker that always knows which attempt is right). A real user rarely has that checker.
Example 3: public coding benchmarks and what each measures
A benchmark is a fixed set of tasks with automatic grading, so that different models can be compared on the same work. Many public benchmarks exist for coding, and they differ in what one task looks like, which languages appear, and how an answer is graded. A high score on one tells you little about another. This example puts six well-known benchmarks in one table. It also computes one property that the table alone does not show: how precisely a benchmark of that size can measure a pass rate.
The precision comes from the margin of error. If a benchmark has n tasks and the true pass rate is p, the measured rate varies from run to run by roughly 1.96 * sqrt(p(1-p)/n) at 95% confidence. This is largest at p = 0.5, so the program uses that value. For HumanEval with 164 tasks:
sqrt(0.25 / 164) = sqrt(0.001524) = 0.0390
1.96 * 0.0390 = 0.0765 which is about 7.7 percentage pointsThe descriptions in the table come from each benchmark's paper or documentation, cited in the text below the output. Task counts are given only where the benchmark has a fixed size.
import java.util.List;
import java.util.Locale;
public class Benchmarks {
record Benchmark(String name, String unit, String languages, String graded, Integer tasks) {}
public static void main(String[] args) {
List<Benchmark> all = List.of(
new Benchmark("HumanEval", "function from signature and docstring", "Python", "hidden unit tests", 164),
new Benchmark("MBPP", "function from a one-line description", "Python", "3 assert statements", 974),
new Benchmark("LiveCodeBench", "contest problem with a release date", "Python", "hidden test cases", null),
new Benchmark("SWE-bench Verified", "real GitHub issue, whole repository", "Python", "repository tests fail before, pass after", 500),
new Benchmark("Aider polyglot", "Exercism exercise edited through the harness", "C++, Go, Java, JS, Python, Rust", "exercise tests, 2 attempts", 225),
new Benchmark("Terminal-Bench", "task in a terminal inside a container", "shell and any tool", "test script in the container", null));
for (Benchmark b : all) {
String precision = b.tasks() == null ? "size changes between releases"
: String.format(Locale.ROOT, "%d tasks, 95%% margin at 50%%: +/- %.1f points", b.tasks(),
100 * 1.96 * Math.sqrt(0.25 / b.tasks()));
System.out.printf(Locale.ROOT, "%-18s | %-44s | %s%n", b.name(), b.unit(), precision);
System.out.printf(Locale.ROOT, "%-18s | languages: %-33s | graded by: %s%n", "", b.languages(), b.graded());
}
}
}HumanEval | function from signature and docstring | 164 tasks, 95% margin at 50%: +/- 7.7 points
| languages: Python | graded by: hidden unit tests
MBPP | function from a one-line description | 974 tasks, 95% margin at 50%: +/- 3.1 points
| languages: Python | graded by: 3 assert statements
LiveCodeBench | contest problem with a release date | size changes between releases
| languages: Python | graded by: hidden test cases
SWE-bench Verified | real GitHub issue, whole repository | 500 tasks, 95% margin at 50%: +/- 4.4 points
| languages: Python | graded by: repository tests fail before, pass after
Aider polyglot | Exercism exercise edited through the harness | 225 tasks, 95% margin at 50%: +/- 6.5 points
| languages: C++, Go, Java, JS, Python, Rust | graded by: exercise tests, 2 attempts
Terminal-Bench | task in a terminal inside a container | size changes between releases
| languages: shell and any tool | graded by: test script in the containerThe margin column applies the normal approximation 1.96 * sqrt(p(1-p)/n) at a pass rate of 50%, where it is largest. On HumanEval's 164 tasks, a measured 50% means "somewhere between about 42% and 58%" with 95% confidence, so two models 5 points apart on HumanEval cannot be ranked by that number alone. Example 9 replaces this approximation with the Wilson interval, which stays correct near 0% and 100%.
What each benchmark measures, in order of the table:
- HumanEval (Chen et al., 2021, the Codex paper) has 164 hand-written Python problems. The model sees a function signature and docstring (the comment that describes the function) and writes the body. Hidden unit tests (small programs that call the function and check its results) decide. It measures writing one short function from a specification. It does not measure reading existing code, using tools, or working across files. The same paper introduced the pass@k estimator from Example 2.
- MBPP (Mostly Basic Python Problems, Austin et al., 2021) has 974 short problems written by crowd workers, each with a one-sentence description and three test assertions (statements that fail if a result is wrong). It measures entry-level programming from a short description; many problems are solvable in a few lines.
- LiveCodeBench collects problems from programming contests (LeetCode, AtCoder, and Codeforces) and records each problem's release date, so a model can be scored only on problems published after its training data was collected (Example 4). Besides code generation, it has scenarios for self-repair, code execution, and test output prediction. Its size grows with each release.
- SWE-bench (Jimenez et al., 2023) takes real issues and their fixing pull requests (a pull request is a proposed code change on GitHub) from 12 popular Python repositories. The agent gets the repository at the commit before the fix and the issue text, and must produce a patch (a description of file changes). Grading runs the repository's tests. The tests the original fix made pass (
FAIL_TO_PASS) must now pass, and the tests that passed before (PASS_TO_PASS) must still pass, so a fix that breaks other behavior does not count. SWE-bench Verified is a 500-task subset whose tasks human annotators checked to be solvable from the issue text, released by OpenAI in 2024. It measures repository-level bug fixing with an agent loop. mini-swe-agent 2.4.6 runs it from the datasetprinceton-nlp/SWE-Bench_Verified, one Docker image per task (a Docker image is a packaged, isolated environment), and writes apreds.jsonfile whose entries holdinstance_id,model_name_or_path, andmodel_patch. A separate harness (the program that does the grading) then applies each patch and runs the tests. - Aider polyglot is Aider's benchmark of 225 Exercism exercises (small practice problems) in C++, Go, Java, JavaScript, Python, and Rust. The model edits the exercise's files through Aider's edit format. If the tests fail, it sees the test output and gets a second attempt. It measures editing code in several languages and following an edit format, so the same model can score differently with a different edit format (Edit Formats).
- Terminal-Bench gives an agent a task in a terminal inside a container (build something, configure a service, fix an environment) and grades the final state of the container with a test script. It measures command-line work over many steps, closer to what
loop agentdoes with its shell tool.
The benchmarks also differ in what a score depends on. HumanEval and MBPP scores depend mostly on the model, because the model writes the code in one pass. SWE-bench, Aider polyglot, and Terminal-Bench scores depend on the model and the harness together. The same model in two agents can differ by many points, because tools, prompts, edit formats, and step limits change the result. A reported number is a property of the whole system, and comparing two models requires the same harness.
Example 4: contamination
Imagine a student who sees the exam questions and answers the night before. A high score then says little about understanding. Contamination is the same problem for models: a benchmark problem that appears in the model's training data can be answered from memory. The analogy fits in that both leak the answers in advance. It stops fitting in that a model cannot tell you what it memorized, so the leak is hard to see. Contamination inflates scores in a way no confidence interval shows, because the interval only measures random variation, not unfair advantage.
This example shows two detection methods. The first is n-gram overlap. An n-gram is a run of n consecutive words. For the sentence "the cat sat on the mat", the 3-grams are:
the cat sat
cat sat on
sat on the
on the matIf most n-grams of a benchmark problem also appear in a text the model may have trained on, the problem was probably seen. The second method is filtering by release date, which LiveCodeBench uses: keep only problems published after the model's training data ended. The program tests a problem copied word for word, the same problem reworded, and four problems with release dates against a training cutoff.
import java.time.LocalDate;
import java.util.Arrays;
import java.util.HashSet;
import java.util.List;
import java.util.Locale;
import java.util.Set;
public class Contamination {
static List<String> words(String text) {
return Arrays.stream(text.toLowerCase(Locale.ROOT).split("[^a-z0-9]+")).filter(w -> !w.isEmpty()).toList();
}
static Set<String> ngrams(String text, int n) {
List<String> w = words(text);
Set<String> out = new HashSet<>();
for (int i = 0; i + n <= w.size(); i++) out.add(String.join(" ", w.subList(i, i + n)));
return out;
}
static double overlap(String problem, Set<String> corpus, int n) {
Set<String> grams = ngrams(problem, n);
long found = grams.stream().filter(corpus::contains).count();
return grams.isEmpty() ? 0 : (double) found / grams.size();
}
record Problem(String id, String text, LocalDate released) {}
public static void main(String[] args) {
String crawled = "Forum post: Help with my homework. Write a function that returns the sum of all even numbers "
+ "in a list of integers, and return zero for an empty list. Here is my solution: def f(xs): return sum(x for x in xs if x % 2 == 0)";
Set<String> corpus = ngrams(crawled, 8);
String verbatim = "Write a function that returns the sum of all even numbers in a list of integers, and return zero for an empty list.";
String reworded = "Given a list of integers, add up the ones divisible by two. An empty list gives 0.";
System.out.printf(Locale.ROOT, "8-gram overlap, verbatim problem: %.0f%%%n", 100 * overlap(verbatim, corpus, 8));
System.out.printf(Locale.ROOT, "8-gram overlap, reworded problem: %.0f%%%n", 100 * overlap(reworded, corpus, 8));
LocalDate cutoff = LocalDate.of(2025, 6, 30); // the model's stated training cutoff
List<Problem> problems = List.of(
new Problem("lc-1001", "two pointers", LocalDate.of(2024, 11, 2)),
new Problem("ac-0420", "graph shortest path", LocalDate.of(2025, 6, 30)),
new Problem("cf-2210", "dynamic programming", LocalDate.of(2025, 8, 14)),
new Problem("lc-1377", "string parsing", LocalDate.of(2025, 9, 1)));
for (Problem p : problems)
System.out.println(p.id() + " released " + p.released() + ": " + (p.released().isAfter(cutoff) ? "kept" : "dropped (on or before the cutoff)"));
}
}8-gram overlap, verbatim problem: 100%
8-gram overlap, reworded problem: 0%
lc-1001 released 2024-11-02: dropped (on or before the cutoff)
ac-0420 released 2025-06-30: dropped (on or before the cutoff)
cf-2210 released 2025-08-14: kept
lc-1377 released 2025-09-01: keptThe first two lines are the overlap test. Every 8-word run of the verbatim problem appears in the crawled forum post, so the overlap is 100%: a model trained on that page has seen the problem and a solution. The reworded problem shares no 8-word run with the post, so the overlap is 0%, although it is the same task. That is the weakness of n-gram checks: they find copies, not paraphrases or translations. The GPT-3 paper used 13-gram overlap between benchmarks and its training data in the same way. Running such a check requires access to the training data, which most open-weight model releases do not publish.
The last four lines are the date filter. With a training cutoff of 2025-06-30, problems released on or before that date are dropped, including ac-0420 released on the cutoff day itself (the code keeps a problem only if isAfter(cutoff) is true, so the cutoff day is excluded to be safe). Problems released later are kept. The method needs no access to the training data, only trustworthy release dates, and it is why LiveCodeBench publishes dates for every problem. Its cost is that the evaluation set shrinks for newer models and changes over time, so scores from different windows are not comparable.
Other defenses follow from the same idea:
- Private task suites, written for the evaluation and never published, cannot have leaked. Example 5 builds one.
- Perturbed versions of public problems (renamed functions, changed constants) separate memorized answers from understanding: a large drop on the perturbed version suggests memorization.
- For agents, the repository's own recent commits make a natural private benchmark: a bug fixed last week, reverted in a copy, is a task no model has seen.
Example 5: a task suite format
Public benchmarks cannot tell you how an agent does on your code, so you need your own tasks. A task suite is the set of tasks an evaluation runs. Each task needs four things: a prompt for the agent, the starting files, the tests that decide success, and a time limit. It also needs a command that runs the tests.
One design rule matters more than the others: the tests stay hidden from the agent until it finishes. If the agent could see the tests, it could read them to guess the expected output, or edit them so they pass. Either way, a pass would mean nothing. For this reason the format keeps starting files and tests in two separate maps.
This example defines a JSON Lines format (one JSON object per line, so a suite is a text file that can be reviewed and extended line by line). It parses the file with Jackson, the JSON library used in earlier lessons, and validates the tasks. The third line of the sample file is deliberately broken. Read the validator's output to see which mistakes it catches.
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.PropertyNamingStrategies;
import java.util.ArrayList;
import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
public class SuiteFormat {
record Task(String id, String prompt, Map<String, String> files, Map<String, String> tests,
String testCommand, int timeoutSeconds, List<String> tags) {}
static List<String> validate(List<Task> tasks) {
List<String> problems = new ArrayList<>();
Set<String> ids = new HashSet<>();
for (Task t : tasks) {
if (!ids.add(t.id())) problems.add(t.id() + ": duplicate id");
if (t.prompt() == null || t.prompt().isBlank()) problems.add(t.id() + ": empty prompt");
if (t.tests() == null || t.tests().isEmpty()) problems.add(t.id() + ": no tests");
if (t.files() != null && t.tests() != null)
for (String f : t.tests().keySet()) if (t.files().containsKey(f)) problems.add(t.id() + ": test file " + f + " is also a starting file");
if (t.timeoutSeconds() <= 0) problems.add(t.id() + ": timeout must be positive");
}
return problems;
}
public static void main(String[] args) throws Exception {
String jsonl = """
{"id":"fix-add","prompt":"Calc.add returns the wrong value. Fix it.","files":{"Calc.java":"class Calc { static int add(int a, int b) { return a - b; } }\\n"},"tests":{"CalcTest.java":"..."},"test_command":"javac -d out *.java && java -cp out CalcTest","timeout_seconds":60,"tags":["java","bugfix"]}
{"id":"count-lines","prompt":"Write count.sh that prints the number of lines in the file given as $1.","files":{},"tests":{"test.sh":"..."},"test_command":"bash test.sh","timeout_seconds":20,"tags":["shell","new-file"]}
{"id":"fix-add","prompt":"","files":{"t.sh":"x"},"tests":{"t.sh":"y"},"test_command":"bash t.sh","timeout_seconds":0,"tags":[]}
""";
ObjectMapper json = new ObjectMapper().setPropertyNamingStrategy(PropertyNamingStrategies.SNAKE_CASE);
List<Task> tasks = new ArrayList<>();
for (String line : jsonl.strip().split("\n")) tasks.add(json.readValue(line, Task.class));
for (Task t : tasks)
System.out.println(t.id() + " files=" + t.files().keySet() + " hidden tests=" + t.tests().keySet()
+ " timeout=" + t.timeoutSeconds() + "s tags=" + t.tags());
System.out.println("problems:");
validate(tasks).forEach(p -> System.out.println(" " + p));
}
}fix-add files=[Calc.java] hidden tests=[CalcTest.java] timeout=60s tags=[java, bugfix]
count-lines files=[] hidden tests=[test.sh] timeout=20s tags=[shell, new-file]
fix-add files=[t.sh] hidden tests=[t.sh] timeout=0s tags=[]
problems:
fix-add: duplicate id
fix-add: empty prompt
fix-add: test file t.sh is also a starting file
fix-add: timeout must be positiveThe first three lines are the parsed tasks. PropertyNamingStrategies.SNAKE_CASE maps test_command and timeout_seconds in the JSON to the record components testCommand and timeoutSeconds, as in The Chat Completions Protocol and Streaming in Java. The tests contents are shortened to ... here; Example 6 has real ones.
fix-addstarts with a buggyCalc.java. Its test command compiles every Java file, including the hidden test that arrives after the agent finishes, and runs the test class.count-linesstarts with no files at all; the agent must createcount.sh. Its hidden test is a shell script.- The
tagsfield groups results later ("how does the agent do on shell tasks").
The validator found four problems in the third line, each of which would make a result meaningless:
- A duplicate id: the results of two tasks would merge.
- An empty prompt: the agent would have nothing to do.
- A test file with the same name as a starting file: the agent would see the test, or the test would overwrite the agent's work.
- A zero timeout: the task could never finish in time.
A suite is checked once when it loads, before any agent runs. A broken task found after a two-hour run wastes the run.
A JSONL file keeps a suite in one reviewable file and works well for small tasks. Larger suites use one directory per task (a task.json, a workspace/ directory, and a tests/ directory); SWE-bench goes further and stores a repository URL and commit instead of files. The fields stay the same.
Example 6: a runner that copies the workspace, runs the agent, and runs the tests
The runner is the program that executes one task from a suite and decides pass or fail. It follows the same steps every time:
1. create a fresh temporary directory (the workspace)
2. write the starting files into it
3. let the agent work in it
4. write the hidden tests into it <- only now
5. run the test command, with a time limit
6. record pass or fail, delete the directoryBy convention a command that succeeds exits with code 0 and a command that fails exits with a non-zero code, so the runner treats exit code 0 from the test command as a pass. This example runs four tasks with a scripted agent that edits files the way a model would, so no model is needed. Two tasks compile and run real Java tests with javac. One task has an agent that writes a script that never ends, to show the time limit working. Look at the four result lines in the output and match each to its task.
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
public class TaskRunner {
record Task(String id, String prompt, Map<String, String> files, Map<String, String> tests, String testCommand, int timeoutSeconds) {}
record Outcome(String taskId, boolean passed, String detail) {}
interface AgentRunner { void run(Path workspace, String prompt) throws Exception; }
static void write(Path dir, Map<String, String> files) throws Exception {
for (Map.Entry<String, String> f : files.entrySet()) Files.writeString(dir.resolve(f.getKey()), f.getValue());
}
static Outcome runOne(Task task, AgentRunner agent) throws Exception {
Path ws = Files.createTempDirectory("eval-" + task.id() + "-");
try {
write(ws, task.files()); // 1. a fresh copy of the starting files
agent.run(ws, task.prompt()); // 2. the agent works in the copy
write(ws, task.tests()); // 3. hidden tests arrive only now
Process p = new ProcessBuilder("bash", "-c", task.testCommand()).directory(ws.toFile()).redirectErrorStream(true).start();
CompletableFuture<String> output = CompletableFuture.supplyAsync(() -> {
try { return new String(p.getInputStream().readAllBytes()); } catch (Exception e) { return ""; }
});
if (!p.waitFor(task.timeoutSeconds(), TimeUnit.SECONDS)) { // 4. the time limit
p.descendants().forEach(ProcessHandle::destroyForcibly);
p.destroyForcibly().waitFor();
return new Outcome(task.id(), false, "timed out after " + task.timeoutSeconds() + " s");
}
List<String> lines = output.get().strip().lines().toList();
String last = lines.isEmpty() ? "" : lines.get(lines.size() - 1);
return new Outcome(task.id(), p.exitValue() == 0, "exit " + p.exitValue() + ", last line: " + last);
} finally {
try (Stream<Path> s = Files.walk(ws)) { s.sorted(Comparator.reverseOrder()).forEach(f -> f.toFile().delete()); }
}
}
public static void main(String[] args) throws Exception {
String calcTest = """
public class CalcTest {
public static void main(String[] a) {
int got = Calc.add(2, 3);
if (got != 5) { System.out.println("FAIL add(2, 3) = " + got + ", expected 5"); System.exit(1); }
System.out.println("PASS add(2, 3) = 5");
}
}
""";
String countTest = "printf 'a\\nb\\nc\\n' > data.txt\n[ \"$(bash count.sh data.txt)\" = 3 ] && echo PASS || { echo \"FAIL got '$(bash count.sh data.txt)'\"; exit 1; }\n";
List<Task> tasks = List.of(
new Task("fix-add", "Fix Calc.add.", Map.of("Calc.java", "class Calc { static int add(int a, int b) { return a - b; } }\n"),
Map.of("CalcTest.java", calcTest), "javac -d out *.java && java -cp out CalcTest", 60),
new Task("fix-add-wrong", "Fix Calc.add.", Map.of("Calc.java", "class Calc { static int add(int a, int b) { return a - b; } }\n"),
Map.of("CalcTest.java", calcTest), "javac -d out *.java && java -cp out CalcTest", 60),
new Task("count-lines", "Write count.sh.", Map.of(), Map.of("test.sh", countTest), "bash test.sh", 20),
new Task("count-lines-hangs", "Write count.sh.", Map.of(), Map.of("test.sh", countTest), "bash test.sh", 2));
Map<String, AgentRunner> agents = Map.of( // what a scripted agent does per task
"fix-add", (ws, p) -> Files.writeString(ws.resolve("Calc.java"), Files.readString(ws.resolve("Calc.java")).replace("a - b", "a + b")),
"fix-add-wrong", (ws, p) -> Files.writeString(ws.resolve("Calc.java"), Files.readString(ws.resolve("Calc.java")).replace("a - b", "a * b")),
"count-lines", (ws, p) -> Files.writeString(ws.resolve("count.sh"), "wc -l < \"$1\"\n"),
"count-lines-hangs", (ws, p) -> Files.writeString(ws.resolve("count.sh"), "while true; do :; done\n"));
for (Task t : tasks) {
Outcome o = runOne(t, agents.get(t.id()));
System.out.println((o.passed() ? "PASS " : "FAIL ") + o.taskId() + " (" + o.detail() + ")");
}
}
}PASS fix-add (exit 0, last line: PASS add(2, 3) = 5)
FAIL fix-add-wrong (exit 1, last line: FAIL add(2, 3) = 6, expected 5)
PASS count-lines (exit 0, last line: PASS)
FAIL count-lines-hangs (timed out after 2 s)Each line is one task's outcome:
fix-add: the agent changeda - btoa + b. After it finished, the runner addedCalcTest.java, compiled both files, and ran the test, which printedPASS add(2, 3) = 5and exited with 0. The exit code decides the outcome; the output is kept for the report.fix-add-wrong: the agent changeda - btoa * b, which compiles but computes 6. The test printed the actual and expected values and exited with 1. A test that prints what it got, not only "failed", makes the result useful to a person reading the report and, in a retry loop, to the model.count-lines: the agent createdcount.shfrom nothing, and the hiddentest.shcreated a three-line file, ran the script, and compared its output with3.count-lines-hangs: the agent wrote a script that loops forever. The test command did not finish within its 2-second limit, so the runner killed the process and its descendants (thebashrunningtest.shand thebashrunningcount.sh), as The Shell Tool does for agent commands, and recorded a timeout. Without the limit, one bad task would stop the whole evaluation.
The order of steps carries the design:
- A fresh temporary directory per run means no run sees files from another run, so runs are independent and can run in parallel.
- The tests arrive after the agent, so the agent cannot read or edit them.
- The
finallyblock deletes the directory even when a step throws. - The test output is read on a separate thread (
CompletableFuture) while the main thread waits for the process. This keeps a test that prints a lot from blocking on a full output buffer.
A production runner adds isolation: the agent's tools and the test command run inside a container or sandbox (the lesson Permissions, Sandboxing, and Prompt Injection), because an agent under evaluation runs model-written commands without a person approving them.
Example 7: metrics, pass rate, steps, tokens, time, and cost
A run produces more than pass or fail. The number of agent steps (each step is one call to the model), the tokens sent and received, and the wall-clock time (time as measured by a clock on the wall, not CPU time) show what a pass costs. A configuration that passes as often with half the tokens is better for a local machine, where tokens turn into waiting time and heat.
This example aggregates a fixed set of six run records into the summary table a report shows. The times are recorded values, not measured ones, so the output is fixed. Pay attention to the difference between the mean (the sum divided by the count) and the median (the middle value when the numbers are sorted). One extremely long run can drag the mean far away from what a typical run looks like, and the median does not move.
import java.util.Arrays;
import java.util.List;
import java.util.Locale;
public class Metrics {
record Run(String task, boolean passed, int steps, int promptTokens, int completionTokens, long millis) {}
static double median(double[] v) {
double[] s = v.clone();
Arrays.sort(s);
return s.length % 2 == 1 ? s[s.length / 2] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2;
}
public static void main(String[] args) {
List<Run> runs = List.of(
new Run("fix-add", true, 4, 9_800, 610, 21_400),
new Run("fix-off-by-one", true, 7, 21_300, 1_020, 48_900),
new Run("count-lines", true, 3, 6_100, 380, 12_700),
new Run("rename-method", false, 20, 118_000, 4_900, 301_000),
new Run("parse-csv", true, 9, 30_400, 1_650, 70_200),
new Run("add-test", false, 6, 17_900, 2_300, 55_800));
long passed = runs.stream().filter(Run::passed).count();
double[] steps = runs.stream().mapToDouble(Run::steps).toArray();
double[] tokens = runs.stream().mapToDouble(r -> r.promptTokens() + r.completionTokens()).toArray();
long totalPrompt = runs.stream().mapToLong(Run::promptTokens).sum();
long totalCompletion = runs.stream().mapToLong(Run::completionTokens).sum();
long totalMillis = runs.stream().mapToLong(Run::millis).sum();
System.out.printf(Locale.ROOT, "pass rate %d/%d = %.1f%%%n", passed, runs.size(), 100.0 * passed / runs.size());
System.out.printf(Locale.ROOT, "steps mean %.1f, median %.1f, max %.0f%n", Arrays.stream(steps).average().orElse(0), median(steps), Arrays.stream(steps).max().orElse(0));
System.out.printf(Locale.ROOT, "tokens per run mean %.0f, median %.0f%n", Arrays.stream(tokens).average().orElse(0), median(tokens));
System.out.printf(Locale.ROOT, "tokens total %d prompt + %d completion%n", totalPrompt, totalCompletion);
System.out.printf(Locale.ROOT, "time total %.1f min, per run median %.1f s%n", totalMillis / 60_000.0,
median(runs.stream().mapToDouble(r -> r.millis() / 1000.0).toArray()));
double failedShare = runs.stream().filter(r -> !r.passed()).mapToDouble(r -> r.promptTokens() + r.completionTokens()).sum() / Arrays.stream(tokens).sum();
System.out.printf(Locale.ROOT, "tokens spent on failed runs: %.0f%%%n", 100 * failedShare);
double inPrice = 0.50, outPrice = 1.50; // illustrative prices in dollars per million tokens
double cost = (totalPrompt * inPrice + totalCompletion * outPrice) / 1e6;
System.out.printf(Locale.ROOT, "cost at $%.2f/$%.2f per M tokens: $%.4f total, $%.4f per solved task%n", inPrice, outPrice, cost, cost / passed);
System.out.printf(Locale.ROOT, "local throughput needed for the median run in 60 s: %.0f prompt tok/s%n",
median(runs.stream().mapToDouble(Run::promptTokens).toArray()) / 60);
}
}pass rate 4/6 = 66.7%
steps mean 8.2, median 6.5, max 20
tokens per run mean 35727, median 21260
tokens total 203500 prompt + 10860 completion
time total 8.5 min, per run median 52.3 s
tokens spent on failed runs: 67%
cost at $0.50/$1.50 per M tokens: $0.1180 total, $0.0295 per solved task
local throughput needed for the median run in 60 s: 327 prompt tok/sThe lines read as follows:
- Pass rate is passed runs over all runs, 4 of 6. It is the headline number, and Examples 9 and 10 show how uncertain it is with this few runs.
- Steps are model calls per run. The values are 4, 7, 3, 20, 9, 6. The mean (49 / 6 = 8.2) is pulled up by
rename-method, which hit the 20-step limit. The median is the average of the two middle values in the sorted list 3, 4, 6, 7, 9, 20, which is (6 + 7) / 2 = 6.5, and it describes a typical run. A failed run that used every step is a different failure from one that stopped early with a wrong answer, so reports keep the maximum and the stop reason. - Tokens per run follow the same pattern: the median (21,260) is far below the mean (35,727) because one run used 122,900 tokens. Prompt tokens dominate because every step resends the history; Context Engineering for Coding Agents covers how compaction and prompt caching reduce this.
- Time totals the recorded wall-clock time. On a local model, time is the cost, and it depends on prompt processing speed as much as generation speed.
- Tokens spent on failed runs is 67%: most of the compute went to the two runs that failed (122,900 + 20,200 of 214,360 tokens). Stopping runs that are clearly stuck (the loop detection of the lesson Making Small Local Models Reliable) saves more than shortening successful ones.
- Cost applies per-token prices, here illustrative values of $0.50 per million prompt tokens and $1.50 per million completion tokens. For the totals above, 203,500 prompt tokens cost 203,500 * 0.50 / 1,000,000 = $0.1018 and 10,860 completion tokens cost $0.0163, which add up to $0.1180. Cost per solved task divides by passes (4), not runs (6), because failed runs cost the same and solve nothing.
- The last line takes the median of the prompt tokens per run (19,600) and divides by 60 seconds. The result, about 327 tokens per second, is the prompt-processing speed a local engine needs to finish that run's prompts in one minute. It connects the evaluation back to engine choice and quantization in earlier sections.
Example 8: transcripts and replay
When an agent fails on task 37 of a long evaluation, you want to know what it did. A transcript is the full record of one run: every model request and response, every tool call and result, and the test outcome. Saving it as JSON Lines, one event per line, makes three things possible:
- Reading what the agent did.
- Re-scoring a run with new tests without calling the model.
- Replay, which feeds the recorded model responses back to the agent so a failure can be reproduced exactly.
Replay works because the model's output is the only random part of a run. The tools (read a file, write a file) behave the same every time when given the same files. If you save what the model said, you can run the tools again and get the same run, without the model.
This example writes a transcript, reads it back, replays it against the same workspace and against a changed workspace, and shows where the replay stops matching. The workspace is a small in-memory map from file name to content, and there are two tools, read and write. Look in the output for the sentence "diverges at event 2".
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.TreeMap;
public class TranscriptReplay {
static final ObjectMapper M = new ObjectMapper();
/** A tool layer over an in-memory workspace: "read" returns a file, "write" replaces it. */
static String tool(Map<String, String> ws, String name, JsonNode args) {
String path = args.get("path").asText();
if (name.equals("read")) return ws.getOrDefault(path, "ERROR: no such file " + path);
ws.put(path, args.get("content").asText());
return "wrote " + path;
}
/** Runs the model's tool calls in order and records every event. */
static List<String> run(List<JsonNode> modelCalls, Map<String, String> ws) throws Exception {
List<String> events = new ArrayList<>();
for (JsonNode call : modelCalls) {
events.add(M.createObjectNode().put("type", "tool_call").put("name", call.get("name").asText()).set("args", call.get("args")).toString());
events.add(M.createObjectNode().put("type", "tool_result").put("content", tool(ws, call.get("name").asText(), call.get("args"))).toString());
}
events.add(M.createObjectNode().put("type", "test").put("passed", ws.getOrDefault("Calc.java", "").contains("a + b")).toString());
return events;
}
public static void main(String[] args) throws Exception {
List<JsonNode> model = List.of(
M.readTree("{\"name\":\"read\",\"args\":{\"path\":\"Calc.java\"}}"),
M.readTree("{\"name\":\"write\",\"args\":{\"path\":\"Calc.java\",\"content\":\"int add(int a, int b) { return a + b; }\"}}"));
Map<String, String> ws = new TreeMap<>(Map.of("Calc.java", "int add(int a, int b) { return a - b; }"));
Path file = Files.createTempFile("run-", ".jsonl");
Files.write(file, run(model, ws));
List<String> saved = Files.readAllLines(file);
System.out.println("transcript (" + saved.size() + " events):");
saved.forEach(l -> System.out.println(" " + l));
List<JsonNode> recordedCalls = new ArrayList<>(); // replay: the model's side comes from the file
for (String line : saved) {
JsonNode e = M.readTree(line);
if (e.get("type").asText().equals("tool_call")) recordedCalls.add(M.createObjectNode().put("name", e.get("name").asText()).set("args", e.get("args")));
}
Map<String, String> fresh = new TreeMap<>(Map.of("Calc.java", "int add(int a, int b) { return a - b; }"));
System.out.println("replay on the original workspace identical: " + run(recordedCalls, fresh).equals(saved));
Map<String, String> changed = new TreeMap<>(Map.of("Calc.java", "long add(long a, long b) { return a - b; }"));
List<String> replayed = run(recordedCalls, changed);
for (int i = 0; i < saved.size(); i++)
if (!saved.get(i).equals(replayed.get(i))) {
System.out.println("replay on a changed workspace diverges at event " + (i + 1) + ":");
System.out.println(" recorded " + saved.get(i));
System.out.println(" replayed " + replayed.get(i));
break;
}
long rescoredPasses = saved.stream().filter(l -> l.contains("\"type\":\"tool_call\"") && l.contains("return a + b")).count();
System.out.println("re-score with a new check (final write contains 'a + b'): " + (rescoredPasses > 0 ? "pass" : "fail"));
Files.delete(file);
}
}transcript (5 events):
{"type":"tool_call","name":"read","args":{"path":"Calc.java"}}
{"type":"tool_result","content":"int add(int a, int b) { return a - b; }"}
{"type":"tool_call","name":"write","args":{"path":"Calc.java","content":"int add(int a, int b) { return a + b; }"}}
{"type":"tool_result","content":"wrote Calc.java"}
{"type":"test","passed":true}
replay on the original workspace identical: true
replay on a changed workspace diverges at event 2:
recorded {"type":"tool_result","content":"int add(int a, int b) { return a - b; }"}
replayed {"type":"tool_result","content":"long add(long a, long b) { return a - b; }"}
re-score with a new check (final write contains 'a + b'): passThe transcript has five events: two tool calls, their two results, and the test outcome. Real transcripts also hold each model request (the full message list and sampling settings) and response (text, tool calls, token counts); this one keeps only what the replay needs. mini-swe-agent's SWE-bench runner writes the same kind of record, a trajectory file named <instance_id>.traj.json, for each task.
The replay lines show its two uses:
- Replay on the original workspace fed the recorded tool calls into the same tool layer, with no model involved. Every event matched, so the run is reproduced exactly. This is how a failure seen once in a long evaluation is debugged: replay it, add logging to a tool, replay again. The model's output is the only random part of a run, and replay removes it.
- Replay on a changed workspace started from a file whose signature uses
long. The first event (thereadcall) matched, but its result, event 2, differed. From that point a real model would have seen different input and might have acted differently, so the recorded calls after event 2 are no longer valid. Replay is exact only while the recorded results match; the first divergence marks where a change in tools or files affected the run.
The last line re-scores the saved transcript with a new check without running anything. Re-scoring is how a suite's grading can be fixed after a run: if a test turns out to be wrong, the saved transcripts and final workspaces are scored again, and the expensive model calls are not repeated.
Example 9: variance and Wilson confidence intervals
Run the same agent on the same suite twice and you will usually get two different pass rates. There are two reasons. The model samples, so each attempt can go differently. And a small suite holds few tasks, so a few lucky or unlucky tasks move the percentage a lot. With 10 tasks, one task is 10 points.
A confidence interval is a range computed from the results. If you repeated the whole evaluation many times and built the range each time, a stated share of those ranges (usually 95%) would contain the true pass rate. It is a way to report "the pass rate is probably between 40% and 89%" instead of a single number that looks more precise than it is.
The textbook formula p ± 1.96 * sqrt(p(1-p)/n) is the Wald interval, which Example 3 used. Here is the Wald interval for 7 passes in 10 runs, computed by hand:
p = 7 / 10 = 0.7
half = 1.96 * sqrt(0.7 * 0.3 / 10) = 1.96 * 0.1449 = 0.284
Wald = [0.7 - 0.284, 0.7 + 0.284] = [0.416, 0.984]Wald fails for small n and for pass rates near 0 or 1. The Wilson score interval fixes both. Its formula, with z = 1.96 and p the measured rate, is:
center = (p + z^2 / (2n)) / (1 + z^2 / n)
half = z * sqrt(p(1-p)/n + z^2 / (4 n^2)) / (1 + z^2 / n)
Wilson = [center - half, center + half]For 7 of 10: z^2 = 3.8415, so center = (0.7 + 0.192) / 1.384 = 0.644 and half = 0.248, which gives [0.397, 0.892]. The idea behind it is that it does not trust the measured p blindly: it pulls the center toward 0.5 when n is small and never produces a range outside 0 to 1.
The program prints both intervals for six results, then runs a seeded simulation. The simulation repeats a 20-task evaluation 10,000 times with a true pass rate of 60%, and counts how often each interval contained the true value. That count is called the interval's coverage.
import java.util.Arrays;
import java.util.Locale;
import java.util.Random;
public class WilsonInterval {
static final double Z = 1.959964; // the 97.5th percentile of the standard normal distribution
static double[] wilson(int passed, int n) {
double p = (double) passed / n, z2 = Z * Z;
double center = (p + z2 / (2 * n)) / (1 + z2 / n);
double half = Z * Math.sqrt(p * (1 - p) / n + z2 / (4.0 * n * n)) / (1 + z2 / n);
return new double[] {center - half, center + half};
}
static double[] wald(int passed, int n) {
double p = (double) passed / n, half = Z * Math.sqrt(p * (1 - p) / n);
return new double[] {p - half, p + half};
}
public static void main(String[] args) {
int[][] results = {{7, 10}, {14, 20}, {70, 100}, {0, 10}, {10, 10}, {1, 30}};
for (int[] r : results) {
double[] w = wilson(r[0], r[1]), a = wald(r[0], r[1]);
System.out.printf(Locale.ROOT, "%3d/%-3d Wilson [%.3f, %.3f] Wald [%.3f, %.3f]%n", r[0], r[1], w[0], w[1], a[0], a[1]);
}
Random random = new Random(7);
double trueRate = 0.6;
int tasks = 20, evaluations = 10_000, wilsonCovers = 0, waldCovers = 0;
int[] counts = new int[evaluations];
for (int e = 0; e < evaluations; e++) {
int passed = 0;
for (int t = 0; t < tasks; t++) if (random.nextDouble() < trueRate) passed++;
counts[e] = passed;
double[] w = wilson(passed, tasks), a = wald(passed, tasks);
if (w[0] <= trueRate && trueRate <= w[1]) wilsonCovers++;
if (a[0] <= trueRate && trueRate <= a[1]) waldCovers++;
}
Arrays.sort(counts);
System.out.printf(Locale.ROOT, "true rate 0.6, 20 tasks, %d evaluations: pass rate 5th..95th percentile %.2f..%.2f, min %.2f, max %.2f%n",
evaluations, counts[evaluations / 20] / 20.0, counts[evaluations * 19 / 20] / 20.0, counts[0] / 20.0, counts[evaluations - 1] / 20.0);
System.out.printf(Locale.ROOT, "coverage: Wilson %.1f%%, Wald %.1f%%%n", 100.0 * wilsonCovers / evaluations, 100.0 * waldCovers / evaluations);
}
} 7/10 Wilson [0.397, 0.892] Wald [0.416, 0.984]
14/20 Wilson [0.481, 0.855] Wald [0.499, 0.901]
70/100 Wilson [0.604, 0.781] Wald [0.610, 0.790]
0/10 Wilson [0.000, 0.278] Wald [0.000, 0.000]
10/10 Wilson [0.722, 1.000] Wald [1.000, 1.000]
1/30 Wilson [0.006, 0.167] Wald [-0.031, 0.098]
true rate 0.6, 20 tasks, 10000 evaluations: pass rate 5th..95th percentile 0.40..0.75, min 0.20, max 0.95
coverage: Wilson 96.5%, Wald 93.0%The six result lines compare the intervals:
7/10and70/100have the same pass rate, 70%, but the interval for 10 runs is several times wider than for 100. Width shrinks with the square root ofn: four times the runs halves the width.14/20is the size of many personal evaluation suites. The Wilson interval spans about 48% to 85%; a single 20-task run cannot tell 60% from 80%.0/10shows the Wald interval's failure: withp = 0its width is 0, claiming certainty that the true rate is exactly 0. Wilson gives[0, 0.278]: zero passes in 10 runs is consistent with a true rate up to about 28%.10/10is the mirror case. Wald says exactly 100%; Wilson says at least 72%.1/30shows Wald going below 0, which is impossible for a rate. Wilson stays inside[0, 1]for every input.
The two simulation lines repeat a 20-task evaluation 10,000 times with a true pass rate of 60%. The middle 90% of measured pass rates spans 0.40 to 0.75 (the 5th to the 95th percentile). So the same agent, on equally hard tasks, often scores 45% one time and 70% the next. The extremes are wider still: 0.20 at the lowest and 0.95 at the highest. The coverage line counts how often each interval contained the true 60%. Wilson's coverage is close to the promised 95%; Wald's is lower, so its intervals are too narrow too often. Coverage of an interval for a binomial count (the number of passes out of a fixed number of tries) is never exactly 95%, because the count is a whole number.
In an evaluation, n in the formula is the number of independent attempts. Several runs of the same task are not fully independent (a task that is hard stays hard), so with r runs per task the interval over all n * r attempts is narrower than the true uncertainty about the suite. Reports state both the number of tasks and the number of runs.
Example 10: comparing two configurations
The question an evaluation usually answers is whether configuration B is better than configuration A, for example Q4_K_M against Q8_0 of the same model (two quantization levels, where Q4_K_M uses fewer bits and is smaller). Running both on the same tasks gives paired results: each task has an outcome under A and under B. Each task falls into one of four groups:
| B passes | B fails | |
|---|---|---|
| A passes | both pass | only A passes |
| A fails | only B passes | both fail |
Tasks where both pass or both fail say nothing about which configuration is better. Only the discordant tasks, where one passes and the other fails, carry information. If A and B were equally good, a discordant task would be equally likely to favor either one, like a fair coin flip. McNemar's exact test asks how likely a split at least as lopsided as the observed one would be under fair coin flips. That probability is the p-value. A small p-value (the common threshold is 0.05) means the split is unlikely to be chance.
For example, with 4 discordant tasks that all favor A, the chance of 4 heads in 4 flips is (1/2)^4 = 1/16. The test is two-sided (a lopsided split in either direction counts), so the p-value is 2 * 1/16 = 0.125.
This example compares two configurations on 30 tasks, then the same pattern repeated to 120 tasks, then a third configuration that has the same total as A but different failures. Each task is one character in a string: 1 for passed, 0 for failed.
import java.math.BigInteger;
import java.util.Locale;
public class CompareConfigs {
static double binomTailTwoSided(int k, int n) { // P(X <= k) * 2 for X ~ Binomial(n, 0.5), capped at 1
BigInteger sum = BigInteger.ZERO, c = BigInteger.ONE;
for (int i = 0; i <= k; i++) {
sum = sum.add(c);
c = c.multiply(BigInteger.valueOf(n - i)).divide(BigInteger.valueOf(i + 1));
}
double p = sum.doubleValue() / Math.pow(2, n);
return Math.min(1.0, 2 * p);
}
static void compare(String label, String a, String b) {
int bothPass = 0, onlyA = 0, onlyB = 0, bothFail = 0;
for (int i = 0; i < a.length(); i++) {
boolean pa = a.charAt(i) == '1', pb = b.charAt(i) == '1';
if (pa && pb) bothPass++; else if (pa) onlyA++; else if (pb) onlyB++; else bothFail++;
}
int n = a.length(), discordant = onlyA + onlyB;
double p = binomTailTwoSided(Math.min(onlyA, onlyB), discordant);
System.out.printf(Locale.ROOT, "%s%n first passes %d/%d, second passes %d/%d%n", label, bothPass + onlyA, n, bothPass + onlyB, n);
System.out.printf(Locale.ROOT, " both pass %d, only first %d, only second %d, both fail %d%n", bothPass, onlyA, onlyB, bothFail);
System.out.printf(Locale.ROOT, " McNemar exact two-sided p = %.6f%n", p);
}
public static void main(String[] args) {
// one character per task: 1 = passed, 0 = failed; same 30 tasks, same order
compare("Q8_0 (A) against Q4_K_M (B), 30 tasks",
"111101111011110111110011101111",
"111101101011010111110011001101");
compare("A against B on 120 tasks (the same pattern four times)",
"111101111011110111110011101111".repeat(4),
"111101101011010111110011001101".repeat(4));
compare("A against C, 30 tasks, differences spread both ways",
"111101111011110111110011101111",
"011111110111110110111011101111");
}
}Q8_0 (A) against Q4_K_M (B), 30 tasks
first passes 24/30, second passes 20/30
both pass 20, only first 4, only second 0, both fail 6
McNemar exact two-sided p = 0.125000
A against B on 120 tasks (the same pattern four times)
first passes 96/120, second passes 80/120
both pass 80, only first 16, only second 0, both fail 24
McNemar exact two-sided p = 0.000031
A against C, 30 tasks, differences spread both ways
first passes 24/30, second passes 24/30
both pass 21, only first 3, only second 3, both fail 3
McNemar exact two-sided p = 1.000000The first comparison is the 30-task run:
- A passed 24 of 30 and B passed 20 of 30, a difference of 4 tasks or 13 percentage points.
- Only four tasks differ, and all four went A's way (
only first 4, only second 0). The other 26 tasks gave the same result under both configurations. - McNemar's p-value is the probability of a split at least this lopsided if the two configurations were equally good: like four coin flips all landing the same way, 2 × (1/16) = 0.125. A p-value of 0.125 does not show a real difference at the common 0.05 threshold. Four discordant tasks are too few.
The second comparison repeats the same pattern four times, as if the suite had 120 tasks. The pass rates are unchanged (80% against 67%), but now 16 discordant tasks all favor A, and the p-value drops to 0.000031 (2 divided by 2 to the 16th power). The difference is the same size; only the evidence grew.
The third comparison has C pass 24 of 30, the same total as A, but on a different set of tasks: three tasks only A passes and three only C passes. The p-value is 1.0. Equal totals with differences in both directions are what two equally good configurations produce, and the totals alone hide that each one fails tasks the other solves.
Paired comparison is stronger than comparing two separate pass rates with two Wilson intervals, because the tasks both configurations pass or fail cancel out. The intervals for 24/30 and 20/30 overlap widely, while the paired view shows that every difference favored A. In practice:
- Run both configurations on the same tasks, with the same seeds where the engine supports seeds.
- Report the discordant counts.
- Add tasks or runs until the discordant count is large enough to decide.
Example 11: tracing with spans per model call and tool call
Metrics say how long a run took. They do not say where the time went. If a run takes 20 seconds, is the model slow, or is a test command slow? A trace answers this. A trace is a tree of spans. A span records one piece of work: it has a name, a start time, an end time, a parent span (the larger piece of work it belongs to), and attributes (labels such as the model name or token counts).
In an agent run the structure looks like this:
agent.run (root span: the whole run)
step 1
chat (model call span)
tool read_file (tool call span)
step 2
chat
tool edit_file
...This example records spans with a fake clock, where each operation advances the clock by a fixed amount, so the numbers repeat exactly. It prints the tree with each span's duration and share of the run, and exports one span per line in JSON. The token attribute names follow vLLM 0.30.0's tracing module (vllm/tracing/utils.py), which uses the OpenTelemetry generative-AI names gen_ai.request.model, gen_ai.usage.prompt_tokens, and gen_ai.usage.completion_tokens. OpenTelemetry is an industry standard for traces. Look at which span takes the largest share inside step 3.
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.node.ObjectNode;
import java.util.ArrayDeque;
import java.util.ArrayList;
import java.util.Deque;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;
public class Tracing {
static final ObjectMapper M = new ObjectMapper();
static long clock; // milliseconds, advanced by the fake operations
static final class Span {
final int id, parent;
final String name;
final long start;
long end;
final Map<String, Object> attributes = new LinkedHashMap<>();
Span(int id, int parent, String name, long start) { this.id = id; this.parent = parent; this.name = name; this.start = start; }
}
static final class Tracer {
final List<Span> spans = new ArrayList<>();
final Deque<Span> open = new ArrayDeque<>();
Span start(String name) {
Span s = new Span(spans.size() + 1, open.isEmpty() ? 0 : open.peek().id, name, clock);
spans.add(s);
open.push(s);
return s;
}
void end() { open.pop().end = clock; }
}
static void modelCall(Tracer t, int prompt, int completion, long millis) {
Span s = t.start("chat qwen-coder-7b");
s.attributes.put("gen_ai.request.model", "qwen-coder-7b");
s.attributes.put("gen_ai.usage.prompt_tokens", prompt);
s.attributes.put("gen_ai.usage.completion_tokens", completion);
clock += millis;
t.end();
}
static void toolCall(Tracer t, String tool, long millis, int exitCode) {
Span s = t.start("tool " + tool);
s.attributes.put("tool.name", tool);
if (exitCode >= 0) s.attributes.put("process.exit_code", exitCode);
clock += millis;
t.end();
}
public static void main(String[] args) {
Tracer t = new Tracer();
t.start("agent.run fix-add").attributes.put("task.id", "fix-add");
t.start("step 1"); modelCall(t, 2_100, 45, 3_200); toolCall(t, "read_file", 2, -1); t.end();
t.start("step 2"); modelCall(t, 2_400, 80, 4_100); toolCall(t, "edit_file", 3, -1); t.end();
t.start("step 3"); modelCall(t, 2_600, 30, 3_000); toolCall(t, "bash", 6_500, 0); t.end();
t.start("step 4"); modelCall(t, 2_900, 60, 3_600); t.end();
t.end();
Map<Integer, Integer> depth = new LinkedHashMap<>();
long total = t.spans.get(0).end - t.spans.get(0).start;
for (Span s : t.spans) {
int d = s.parent == 0 ? 0 : depth.get(s.parent) + 1;
depth.put(s.id, d);
System.out.printf("%-30s %6d ms %5.1f%%%n", " ".repeat(d) + s.name, s.end - s.start, 100.0 * (s.end - s.start) / total);
}
long model = t.spans.stream().filter(s -> s.name.startsWith("chat")).mapToLong(s -> s.end - s.start).sum();
long tools = t.spans.stream().filter(s -> s.name.startsWith("tool")).mapToLong(s -> s.end - s.start).sum();
System.out.println("model calls " + model + " ms, tools " + tools + " ms");
Span first = t.spans.get(2);
ObjectNode line = M.createObjectNode().put("trace_id", "run-0001").put("span_id", first.id).put("parent_id", first.parent)
.put("name", first.name).put("start_ms", first.start).put("end_ms", first.end);
line.set("attributes", M.valueToTree(first.attributes));
System.out.println(line);
}
}agent.run fix-add 20405 ms 100.0%
step 1 3202 ms 15.7%
chat qwen-coder-7b 3200 ms 15.7%
tool read_file 2 ms 0.0%
step 2 4103 ms 20.1%
chat qwen-coder-7b 4100 ms 20.1%
tool edit_file 3 ms 0.0%
step 3 9500 ms 46.6%
chat qwen-coder-7b 3000 ms 14.7%
tool bash 6500 ms 31.9%
step 4 3600 ms 17.6%
chat qwen-coder-7b 3600 ms 17.6%
model calls 13900 ms, tools 6505 ms
{"trace_id":"run-0001","span_id":3,"parent_id":2,"name":"chat qwen-coder-7b","start_ms":0,"end_ms":3200,"attributes":{"gen_ai.request.model":"qwen-coder-7b","gen_ai.usage.prompt_tokens":2100,"gen_ai.usage.completion_tokens":45}}The tree shows the run, four steps, and their children, with each span's duration and its share of the whole run:
- The four model calls take 13,900 ms, about two thirds of the run. Their prompt sizes (2,100 to 2,900 tokens, in the attributes) grow with each step because the history grows.
tool bashtakes 6,500 ms: compiling and running the tests. One tool call is almost a third of the run. Without spans, a slow run looks like a slow model.read_fileandedit_filetake a few milliseconds each and do not matter for time.- Step 4 has no tool span: the model answered in text and the run ended.
A step's duration is slightly longer than its children added together in some cases: step 1 is 3,202 ms, which is the 3,200 ms model call plus the 2 ms tool call. The root span's 20,405 ms is the sum of all four steps.
The JSON line is one span in an export format: a trace id shared by every span of the run, the span's own id, its parent's id, the times, and the attributes. One line per span can be written to a file during the run and loaded into a trace viewer afterwards. OpenTelemetry is the common standard for this; its Java SDK replaces the hand-written Tracer with the same concepts (spans, parents, attributes, exporters). vLLM can send its own spans for each request when it is started with an OpenTelemetry endpoint, so a trace of an agent run can include the engine's view of the same model call (time in queue, time to first token).
Example 12: Project 28, the loop eval command
Project 28 combines the lesson into one tool: loop eval, a picocli command (picocli is a Java library for building command-line programs). It does the following:
- Reads a suite file.
- Runs an agent on each task several times, each time in a fresh workspace.
- Runs the hidden tests.
- Writes one JSON line per run.
- Prints pass@1, pass@k, a Wilson interval, and token totals.
Its exit code is 0 when the pass rate reaches a threshold and 1 otherwise, so a CI job (an automated check that runs on every code change) can fail when a change makes the agent worse.
The agent is an interface, AgentRunner, so the command does not care what is behind it. In loop, the agent is the Project 26 agent from Workshop: loop agent pointed at the run's workspace. In this demo, it is a scripted agent that writes a correct or a buggy solution, chosen by a seeded random generator, to stand in for a sampling model. Each task has a "skill" value, the chance that the scripted agent writes the correct solution: 0.9 for greet, 0.6 for sum, and 0.3 for max. Compare these chances with the pass counts in the output.
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.PropertyNamingStrategies;
import java.io.PrintWriter;
import java.nio.file.Files;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Random;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
import java.util.stream.Stream;
import picocli.CommandLine;
import picocli.CommandLine.Command;
import picocli.CommandLine.Option;
import picocli.CommandLine.Parameters;
@Command(name = "eval", mixinStandardHelpOptions = true, description = "Run the agent on a task suite and report pass rates.")
public class LoopEval implements Callable<Integer> {
static final ObjectMapper JSON = new ObjectMapper().setPropertyNamingStrategy(PropertyNamingStrategies.SNAKE_CASE);
record Task(String id, String prompt, Map<String, String> files, Map<String, String> tests, String testCommand, int timeoutSeconds) {}
record AgentOutcome(int steps, int tokens) {}
record RunRecord(String task, int run, boolean passed, int steps, int tokens) {}
interface AgentRunner { AgentOutcome run(Task task, Path workspace, int run) throws Exception; }
@Parameters(paramLabel = "SUITE", description = "JSONL file, one task per line") Path suite;
@Option(names = "--runs", defaultValue = "5", description = "Runs per task (default: ${DEFAULT-VALUE})") int runs;
@Option(names = "--k", defaultValue = "3", description = "k for pass@k (default: ${DEFAULT-VALUE})") int k;
@Option(names = "--results", paramLabel = "FILE", description = "Write one JSON line per run") Path results;
@Option(names = "--min-pass-rate", defaultValue = "0", description = "Exit with 1 below this pass@1") double minPassRate;
private final AgentRunner agent;
LoopEval(AgentRunner agent) { this.agent = agent; }
static double passAtK(int n, int c, int k) {
if (n - c < k) return 1.0;
double prod = 1.0;
for (int i = n - c + 1; i <= n; i++) prod *= 1.0 - (double) k / i;
return 1.0 - prod;
}
static double[] wilson(int passed, int n) {
double p = (double) passed / n, z = 1.959964, z2 = z * z;
double center = (p + z2 / (2 * n)) / (1 + z2 / n), half = z * Math.sqrt(p * (1 - p) / n + z2 / (4.0 * n * n)) / (1 + z2 / n);
return new double[] {center - half, center + half};
}
boolean runTests(Task t, Path ws) throws Exception {
for (Map.Entry<String, String> f : t.tests().entrySet()) Files.writeString(ws.resolve(f.getKey()), f.getValue());
Process p = new ProcessBuilder("bash", "-c", t.testCommand()).directory(ws.toFile()).redirectErrorStream(true)
.redirectOutput(ProcessBuilder.Redirect.DISCARD).start();
if (!p.waitFor(t.timeoutSeconds(), TimeUnit.SECONDS)) { p.descendants().forEach(ProcessHandle::destroyForcibly); p.destroyForcibly(); return false; }
return p.exitValue() == 0;
}
@Override public Integer call() throws Exception {
List<Task> tasks = new ArrayList<>();
for (String line : Files.readAllLines(suite)) if (!line.isBlank()) tasks.add(JSON.readValue(line, Task.class));
List<RunRecord> records = new ArrayList<>();
PrintWriter out = new PrintWriter(System.out, true);
out.printf(Locale.ROOT, "%-12s %7s %7s %7s %8s%n", "task", "passed", "pass@1", "pass@" + k, "tokens");
int passedTotal = 0, tokensTotal = 0;
double sum1 = 0, sumK = 0;
for (Task t : tasks) {
int c = 0, tokens = 0;
for (int r = 0; r < runs; r++) {
Path ws = Files.createTempDirectory("loop-eval-");
try {
for (Map.Entry<String, String> f : t.files().entrySet()) Files.writeString(ws.resolve(f.getKey()), f.getValue());
AgentOutcome o = agent.run(t, ws, r);
boolean passed = runTests(t, ws);
records.add(new RunRecord(t.id(), r, passed, o.steps(), o.tokens()));
if (passed) c++;
tokens += o.tokens();
} finally {
try (Stream<Path> s = Files.walk(ws)) { s.sorted(Comparator.reverseOrder()).forEach(f -> f.toFile().delete()); }
}
}
out.printf(Locale.ROOT, "%-12s %4d/%-2d %7.3f %7.3f %8d%n", t.id(), c, runs, passAtK(runs, c, 1), passAtK(runs, c, k), tokens);
sum1 += passAtK(runs, c, 1);
sumK += passAtK(runs, c, k);
passedTotal += c;
tokensTotal += tokens;
}
int n = tasks.size() * runs;
double[] ci = wilson(passedTotal, n);
double pass1 = sum1 / tasks.size();
out.printf(Locale.ROOT, "suite: pass@1 %.3f (95%% Wilson %.3f..%.3f over %d runs), pass@%d %.3f, %d tokens%n",
pass1, ci[0], ci[1], n, k, sumK / tasks.size(), tokensTotal);
if (results != null) {
List<String> lines = new ArrayList<>();
for (RunRecord rec : records) lines.add(JSON.writeValueAsString(rec));
Files.write(results, lines);
}
return pass1 >= minPassRate ? 0 : 1;
}
public static void main(String[] args) throws Exception {
Path dir = Files.createTempDirectory("suite-");
Path suite = dir.resolve("suite.jsonl"), results = dir.resolve("results.jsonl");
String sumTest = "[ \"$(bash sum.sh 2 3 4)\" = 9 ] && [ \"$(bash sum.sh)\" = 0 ]";
String maxTest = "[ \"$(bash max.sh 3 9 4)\" = 9 ] && [ \"$(bash max.sh -5 -2)\" = -2 ]";
String greetTest = "[ \"$(bash greet.sh Ada)\" = 'hello, Ada' ]";
List<Task> tasks = List.of(
new Task("greet", "Write greet.sh NAME that prints: hello, NAME", Map.of(), Map.of("t.sh", greetTest), "bash t.sh", 10),
new Task("sum", "Write sum.sh that prints the sum of its arguments.", Map.of(), Map.of("t.sh", sumTest), "bash t.sh", 10),
new Task("max", "Write max.sh that prints the largest argument.", Map.of(), Map.of("t.sh", maxTest), "bash t.sh", 10));
List<String> lines = new ArrayList<>();
for (Task t : tasks) lines.add(JSON.writeValueAsString(t));
Files.write(suite, lines);
Map<String, String[]> solutions = Map.of( // {correct, buggy} answers of the scripted "model"
"greet", new String[] {"echo \"hello, $1\"", "echo \"Hello $1\""},
"sum", new String[] {"s=0; for x in \"$@\"; do s=$((s + x)); done; echo $s", "echo $(($1 + $2))"},
"max", new String[] {"m=$1; for x in \"$@\"; do [ \"$x\" -gt \"$m\" ] && m=$x; done; echo $m", "m=0; for x in \"$@\"; do [ \"$x\" -gt \"$m\" ] && m=$x; done; echo $m"});
Map<String, Double> skill = Map.of("greet", 0.9, "sum", 0.6, "max", 0.3);
Random random = new Random(42); // one generator, called in a fixed order
AgentRunner scripted = (task, ws, run) -> {
boolean correct = random.nextDouble() < skill.get(task.id());
Files.writeString(ws.resolve(task.id() + ".sh"), solutions.get(task.id())[correct ? 0 : 1] + "\n");
return new AgentOutcome(2 + random.nextInt(4), 1_500 + random.nextInt(1_000));
};
CommandLine cli = new CommandLine(new LoopEval(scripted));
int code = cli.execute(suite.toString(), "--runs", "5", "--k", "3", "--results", results.toString(), "--min-pass-rate", "0.7");
System.out.println("exit code " + code);
List<String> saved = Files.readAllLines(results);
System.out.println("results.jsonl: " + saved.size() + " lines, first: " + saved.get(0));
try (Stream<Path> s = Files.walk(dir)) { s.sorted(Comparator.reverseOrder()).forEach(f -> f.toFile().delete()); }
}
}task passed pass@1 pass@3 tokens
greet 5/5 1.000 1.000 10045
sum 3/5 0.600 1.000 10518
max 1/5 0.200 0.600 10025
suite: pass@1 0.600 (95% Wilson 0.357..0.802 over 15 runs), pass@3 0.867, 30588 tokens
exit code 1
results.jsonl: 15 lines, first: {"task":"greet","run":0,"passed":true,"steps":4,"tokens":2384}The table has one line per task and one line for the suite:
- Each task ran 5 times in a fresh workspace, and the hidden test
t.shdecided each run.greetpassed 5 of 5,sum3 of 5, andmax1 of 5, close to the scripted success chances of 0.9, 0.6, and 0.3.pass@1equalsc/n, as Example 2 showed. pass@3is the unbiased estimator withn = 5. Forsumit is 1.000: with only 2 failed runs, every set of 3 runs out of 5 includes a passing one. Formaxit is1 - C(4,3)/C(5,3) = 1 - 4/10 = 0.600.tokenssums the scripted agent's reported tokens over the 5 runs. Inloop, these come from theusagefield of each chat response.- The suite line averages over tasks: pass@1 is (1.0 + 0.6 + 0.2) / 3 = 0.600 and pass@3 is (1.0 + 1.0 + 0.6) / 3 = 0.867. The Wilson interval over all 15 runs, 0.357 to 0.802, is wide, as Example 9 predicts for 15 attempts.
- The exit code is 1 because pass@1 (0.600) is below the
--min-pass-rateof 0.7 given on the command line. A CI job that runsloop eval suite.jsonl --min-pass-rate 0.7fails on this result.
The last line is the first record of results.jsonl: task, run index, outcome, steps, and tokens, in snake_case through the same Jackson naming strategy. Together with saved transcripts (Example 8), this file is what two configurations are compared from (Example 10): the paired view needs per-task, per-run outcomes, not only the suite averages.
The scripted agent shows the kinds of failure the tests catch:
greet's buggy answer printsHello Adainstead ofhello, Ada: a formatting mistake.sum's buggy answer adds only two arguments, so the test with no arguments fails.max's buggy answer starts from 0, which is correct for positive numbers and wrong for-5 -2. A test with only positive inputs would have passed it.
Hidden tests that cover edge cases are what make a suite's pass rate mean something.
The pieces that turn the demo into the real command are the earlier examples: the Project 26 agent as the AgentRunner, transcripts written per run, spans around each model and tool call, and parallel runs on separate workspaces (each run already has its own directory). A typical invocation against a running engine looks like this. It runs each task 5 times, computes pass@3, saves per-run results, and fails if pass@1 is below 0.6:
loop eval tasks/suite.jsonl --runs 5 --k 3 --results runs/q4km.jsonl --min-pass-rate 0.6Interview points
- Perplexity and KL divergence measure how a model predicts text and how far a quantized model drifts from its base model. They are cheap and need no tasks, but they do not measure task success.
- pass@k is the chance that at least one of k attempts passes. The unbiased estimator
1 - C(n-c,k)/C(n,k)uses n ≥ k samples per task and is computed as a product for numerical stability, because the binomial coefficients get very large. 1 - (1 - c/n)^kis not the same estimator. It samples with replacement and underestimates pass@k for k > 1.- HumanEval and MBPP measure single-function generation in Python. LiveCodeBench adds release dates against contamination. SWE-bench Verified measures repository-level issue fixing graded by FAIL_TO_PASS and PASS_TO_PASS tests. Aider polyglot measures multi-language editing through an edit format. Terminal-Bench measures multi-step terminal work graded on the container's final state.
- Agent benchmark scores belong to the model and the harness together, so comparing models needs the same harness, prompts, and limits.
- Contamination is detected by n-gram overlap (which needs the training data and misses paraphrases) or avoided by release-date filtering and private suites.
- A runner gives each run a fresh workspace, adds hidden tests after the agent finishes, enforces a timeout that kills the whole process tree, and records exit codes, steps, tokens, and time.
- Wilson intervals stay inside [0, 1] and keep near-95% coverage for small n and extreme rates, where the Wald interval collapses to zero width or goes outside the valid range.
- Configurations are compared on the same tasks by looking at the discordant tasks. McNemar's exact test asks whether their split could be chance.
- Transcripts allow replay without the model and re-scoring without re-running. A replay is exact until the first recorded tool result differs.
Summary
- Perplexity is
expof the mean negative log-probability of the actual tokens. KL divergence compares whole distributions. "Same top token" counts how often the most likely token agrees. - pass@1 equals
c/n. For n = 10 and c = 3, pass@5 is 0.917 and pass@8 is 1.0. - A reported pass@k must name k. pass@1 describes an agent with one attempt per task.
- A benchmark of 164 tasks has a 95% margin of about ±7.7 points at a 50% pass rate. 500 tasks give about ±4.4 points.
- SWE-bench Verified has 500 human-checked tasks from SWE-bench's real GitHub issues in 12 Python repositories. mini-swe-agent writes
preds.jsonwithinstance_id,model_name_or_path, andmodel_patch. - n-gram overlap finds verbatim copies. Release dates after the training cutoff avoid contamination without access to training data.
- A task has an id, a prompt, starting files, hidden tests, a test command, and a timeout. Suites are validated before any run.
- Medians describe typical runs. One stuck run can dominate mean tokens and time. Cost per solved task divides by passes.
- A transcript is one JSON event per line. Replay feeds recorded model output back through the tools.
- With 20 tasks and a true rate of 60%, the measured pass rate spans about 40% to 75% across repeated evaluations.
- Wilson's 95% interval for 0/10 is [0, 0.278]. For 14/20 it is about [0.48, 0.85].
- McNemar's exact test uses only discordant tasks. Four discordant tasks all favoring one side give p = 0.125.
- Spans form a tree per run. Model-call spans carry
gen_ai.request.model,gen_ai.usage.prompt_tokens, andgen_ai.usage.completion_tokensas in vLLM's tracing module. loop evalruns each task several times, writesresults.jsonl, prints pass@1, pass@k, and a Wilson interval, and exits with 1 below--min-pass-rate.