Benchmarking, Load Testing, and Profiling

Every optimization in this course claims to make inference faster or cheaper. A claim is worth only as much as the measurement behind it. If the test traffic does not look like real traffic, a configuration that wins the test can lose in production. If each test changes three things at once, nobody knows which change helped. Benchmarking is the work of measuring performance so that the numbers can be trusted. Load testing is benchmarking with many simultaneous requests. Profiling goes one step deeper and finds where the time inside a single request went.

This lesson explains how to build a realistic benchmark, which tools exist, how to run experiments that give clear answers, and when to profile. Afterwards you can compute percentile latencies by hand, design a test plan that changes one thing at a time, and read a profile to choose what to optimize. The latency terms (TTFT, TPS, and percentiles) come from Measuring latency and throughput.

Example 1: what makes a benchmark realistic

Without precise, accurate benchmarks there is no way to know whether an optimization works. A high-quality benchmark imitates real life as closely as possible. The best benchmark shadows production traffic. Shadowing means copying incoming requests to the system under test, so its performance can be measured without affecting the original request or the user who sent it.

flowchart LR
  U["User request"] --> P["Production system (answers the user)"]
  U -. "copy" .-> T["System under test (answer discarded, timings recorded)"]

When real traffic cannot be shadowed, it must be simulated. LLM performance depends on many factors, so a simulation has to match the expected production workload in several dimensions.

A worked case shows why the first row matters. Prefill work grows with the input length, and the KV cache grows with the total length.

Test A: every request has ISL 100 and OSL 100  -> 200 tokens of context per request
Production: ISL 4,000 and OSL 300             -> 4,300 tokens of context per request
Context ratio: 4,300 / 200 = 21.5 times more KV cache per request in production

A server that handles 64 concurrent requests in Test A would need about 21 times as much KV cache memory for the same count of production requests. Test A would suggest a batch size that does not fit. Optimization is about trade-offs and constraints, and maximizing results against bad inputs gives performance that production will not match.

Example 2: benchmarking tools

Every benchmark should reflect its own production traffic, so every setup looks a little different. A few tools are common.

Open-source evaluation datasets are another useful tool. They range from general ones such as MMLU and gsm8k to domain-specific ones such as SWE-bench. The purpose of a benchmark is to measure performance, not output quality, but these datasets still do two jobs. They provide a varied and realistic set of inputs, and they allow a spot check that a performance optimization did not damage the quality of the output (for example, an aggressive number format could). When possible, pick the dataset that matches the expected use of the production system. HumanEval is the example for reducing latency of a code completion system.

Example 3: percentiles by hand

A single benchmark number is often a percentile. The P90 latency is the value that 90 percent of requests are at or below. The P50 is the median. A worked case uses 20 made-up TTFT measurements in milliseconds, sorted from fastest to slowest.

rank:  1   2   3   4   5   6   7   8   9   10  11  12  13  14  15  16  17  18  19  20
ms:   120 125 130 130 135 140 140 145 150 150 155 160 165 170 180 200 240 350 420 900

nearest-rank method: percentile p is the value at rank = ceiling(p x count)
P50: ceiling(0.50 x 20) = 10 -> rank 10 = 150 ms
P90: ceiling(0.90 x 20) = 18 -> rank 18 = 350 ms
P99: ceiling(0.99 x 20) = 20 -> rank 20 = 900 ms
mean: sum = 4,305 ms, 4,305 / 20 = 215.25 ms

The mean (215 ms) hides the story. The median user waits 150 ms, about 1 request in 10 waits longer than 350 ms, and the worst waits 900 ms. This is why a benchmark reports a percentile such as "the P90 TTFT is 350 ms", and why a single slow outlier can distort an average.

Example 4: consistent benchmarks

Great benchmarks are realistic and also consistent. Send enough traffic to get a clear reading without being pushed around by outliers. When in doubt, run the benchmark several times and average the results.

five runs of the same configuration (tokens per second): 1,010  990  1,050  700  1,000
mean = (1,010 + 990 + 1,050 + 700 + 1,000) / 5 = 4,750 / 5 = 950
median = 1,000

One run (700) was disturbed by something else on the machine. A single run could have reported 700 or 1,050, a 50 percent difference for the same system. Several runs reveal the spread. The mean is pulled down by the outlier, and the median shows it more clearly. Both are useful to read together.

Example 5: one change at a time

Before any optimization work, build a solid baseline benchmark, the measurement of the system before any change. While testing optimizations, keep the benchmark configuration constant. Test each optimization alone and all of them together, to learn what drives the improvement. Optimizations can work against each other. A known case is speculative decoding with large batch sizes (Speculative decoding), because large batches already use the compute that speculation relies on.

A test plan with made-up numbers:

If the two gains multiplied, both would give 1.5 x 1.3 = 1.95 times. The measured 1.70 times shows that the two overlap and only testing each alone and together reveals it. The same rule applies to the benchmark itself. Different traffic patterns or sequence shapes need testing, and as with any experiment only one variable should change at a time, to keep results clear.

flowchart LR
  B["Baseline benchmark"] --> C["Change one thing"] --> M["Re-run the same benchmark"] --> D{"Better?"}
  D -- "yes" --> K["Keep it, new baseline"]
  D -- "no" --> R["Revert and note why"]
  K --> C
  R --> C

Example 6: profiling, one level deeper

A benchmark gives a single figure, such as "the P90 TTFT is 350 ms". A profiler shows where each of those milliseconds went inside the inference process. Benchmarking tells how the system performs, and profiling tells why.

A kernel profile of attention shows each operation inside a kernel and how long it takes, which reveals the bottleneck. The attention steps appear in Attention and Mixture of Experts. A worked case with made-up timings in microseconds:

The softmax step does little arithmetic and still takes more than half the time, which suggests excess memory reads. The fix from CUDA, kernels, and kernel fusion is a fused kernel that runs the activation work next to attention and avoids the extra reads. If fusion brings softmax to 30 microseconds, the total becomes 40 + 30 + 40 = 110, a speedup of 170 / 110 = 1.55 times for this kernel. The new kernel is then inserted into the PyTorch code, and the system-level benchmarks run again to check whether the latency targets are met.

Most inference engineers do not profile as part of daily work. With an already fast engine such as TensorRT-LLM, the workflow is a cycle of configuration and benchmarking, and profiling would be superfluous. Profiling belongs in the toolkit of people who:

• contribute to an inference framework such as vLLM or SGLang,
• write their own inference service in PyTorch, or
• work at the edge of a new modality such as video generation.

Example 7: profiling tools

TensorFlow and TensorRT also ship with their own built-in profilers. Profilers matter because their detail about compute and memory use directs optimization toward the most expensive steps of the pipeline. A typical case: PyTorch Profiler shows that activation functions take unusually long because of extra memory reads, so the engineer writes a fused kernel that runs the activations together with attention. Together, profiling and benchmarking supply the information needed to improve system performance and the confidence to deploy the optimizations in production.

Interview points

• A good benchmark imitates production. Shadowing copies live requests to the test system without affecting users.
• When simulating, match sequence lengths (ISL and OSL), traffic volume and pattern (with jitter), request contents (cache hits and draft acceptance), and input parameters (temperature, reasoning effort).
• Common tools: SGLang Genai-bench, NVIDIA GenAI-Perf, and Locust. Eval datasets (MMLU, gsm8k, SWE-bench, HumanEval) provide realistic inputs and a quality check.
• Be consistent: enough traffic, repeat runs, and a stable configuration. Start with a baseline, test each optimization alone and together, and change one variable at a time.
• Optimizations can conflict, for example speculative decoding with large batch sizes.
• Benchmarks say how the system performs. Profilers say why.
• Profilers: PyTorch Profiler (CPU time, GPU time, memory), Nsight Systems (system-wide, multi-GPU), Nsight Compute (single CUDA kernels). TensorFlow and TensorRT have built-in profilers.
• Engine users usually need only configuration and benchmarking. Framework contributors, custom PyTorch service authors, and new-modality engineers need profiling.

Summary

• Benchmark with shadowed or well-simulated traffic that matches sequence lengths, volume, contents, and parameters.
• Report percentiles, such as P50 and P90, instead of only the mean, and repeat runs.
• Establish a baseline and change one variable at a time. Test optimizations alone and together.
• Tools: Genai-bench, GenAI-Perf, Locust, and eval datasets.
• Profile with PyTorch Profiler, NSys, and NCU to find the expensive steps, then fix them (for example, by fusion) and re-run the benchmarks.