Measuring Latency and Throughput

A team cannot improve what it cannot measure, and "the model feels slow" is not a measurement. Serving a language model has its own vocabulary of metrics: time to first token, tokens per second, percentiles, and end-to-end time. Several of these terms are used loosely in everyday talk, and mixing them up leads to wrong decisions, for example tuning the model when the real delay is in the network.

This lesson defines each metric, shows how to compute the numbers by hand, and explains which metric to watch for which kind of request. Afterwards you can read a performance report, tell a latency number from a throughput number, explain why the average hides problems, and decide whether to optimize the model or the infrastructure.

Vocabulary used below: a token is a small piece of text. Streaming means the server sends tokens to the user as they are produced, not all at once at the end. Latency is how long something takes for one user. Throughput is how much work the whole service finishes per unit of time.

Example 1: the two basic LLM metrics

The two most common performance metrics for LLMs are TTFT (time to first token) and TPS (tokens per second). Metrics for other modalities, such as images and speech, appear in the modalities lessons.

Prefill and decode are the two phases of LLM inference: prefill reads the whole prompt, and decode produces the answer one token at a time. "Compute-bound" means limited by arithmetic speed, and "bandwidth-bound" means limited by how fast data moves from memory. The bottlenecks lesson explains both. For now it is enough to know that TTFT is dominated by the prefill phase and TPS by the decode phase.

A timeline of one streamed request:

flowchart LR
    A["Request sent"] -->|"prefill: TTFT = 400 ms"| B["Token 1"]
    B -->|"ITL = 25 ms"| C["Token 2"]
    C -->|"ITL = 25 ms"| D["Token 3"]
    D -->|"..."| E["Last token"]

The first arrow is TTFT (illustrative numbers). The later arrows are the gaps between tokens. If the gap between tokens is 25 ms, then the user receives 1000 / 25 = 40 tokens per second after the first token.

Example 2: three meanings of "tokens per second"

TTFT is a clear term. TPS is less precise, because it can mean two different things:

• a latency metric: tokens per second for one user, or
• a throughput metric: tokens per second for the whole inference service.

Most people use TPS for the per-user latency metric. When precision matters, use these more specific terms:

ITL and perceived TPS are two ways of writing the same fact: perceived TPS = 1000 / ITL in milliseconds. An ITL of 10 milliseconds equals 100 tokens per second per user, and an ITL of 25 ms equals 40.

Total TPS is a different quantity. With a batch of concurrent users, total TPS is roughly the number of users times each user's perceived TPS. If 50 users each see 40 tokens per second, the service generates 2,000 tokens per second in total. Batching is the technique that raises the number of users served at once, and it usually lowers perceived TPS a little while raising total TPS a lot. The two numbers can therefore move in opposite directions, which is why saying only "TPS" is ambiguous.

The calculation below puts these formulas together. It computes the total time of a streamed answer as TTFT plus the time of the remaining tokens.

perceived TPS = 1000 / ITL (ms)       = 1000 / 25            = 40 tokens/s per user
total time    = TTFT + (tokens - 1) x ITL
              = 400 + 199 x 25 = 400 + 4,975                  = 5,375 ms (5.38 s)
total TPS     = users x perceived TPS = 50 x 40               = 2,000 tokens/s
ITL of 10 ms  = 1000 / 10                                     = 100 tokens/s per user

The total time is 400 ms for the first token plus 199 further tokens at 25 ms each, which is 4,975 ms, so 5,375 ms. TTFT is only 7 percent of this answer, so for long answers decode speed (ITL) dominates the total time. For a short answer of 5 tokens the total would be 400 + 4 x 25 = 500 ms, and TTFT would be most of it. This is why chat applications watch TTFT, which decides how fast the interface feels, and long-output applications also watch TPS.

TTFT and TPS suit user-facing LLM systems such as chatbots, where output streams to a person. For other requests, such as a tool call by an agent, the individual tokens are not useful alone. For these, latency is measured as the total response time: the time until the complete answer is ready.

Example 3: why the average is not enough

Latency numbers vary from request to request. Total response time for an LLM is generally a right-skewed distribution: most times cluster around one common value (the mode), but a few outliers take much longer. "Right-skewed" means the long tail points toward larger values on a chart of times. A few very slow requests pull the mean (the average) above the typical request, so the mean is generally higher than the median (the P50 value).

The obvious but weak approach is to look at the mean TTFT or the mean TPS. It hides the outliers, and the outliers matter. They dramatically affect the user experience and trust in a product. It is not enough for most interactions to feel snappy if one in every ten takes several seconds.

Instead inference engineers measure percentiles. The Pxx value is the time below which xx percent of requests finish, so 100 minus xx percent of requests are slower.

Driving down the average still matters, and good performance work also targets P90 and P99 for a more reliable user experience.

The report below comes from 1,000 simulated total response times with a right-skewed shape. Sorted from fastest to slowest, the percentiles use the nearest-rank rule: Pxx is the value at position ceil(xx/100 x N). The numbers are simulated for illustration.

Example report for 1,000 requests (illustrative, right-skewed times)
Nearest-rank rule: Pxx is the value at position ceil(xx/100 x N) in the sorted list.

P50  = 500th value  =  839 ms
P90  = 900th value  = 1537 ms
P95  = 950th value  = 1801 ms
P99  = 990th value  = 2410 ms
max                 = 3393 ms
mean                =  924 ms

Read the report from the top. The mean is above the median (P50), the effect of the long right tail. P90 is about 1.8 times P50, and P99 is almost three times P50 (2,410 against 839 ms). For a user, this means that one request in 100 takes close to three times as long as the typical one. A report that says only "mean 900 ms" gives no sign of that tail. When reporting performance, state which percentile is meant. Comparing a P50 from one system with a P99 from another is a common mistake.

Example 4: inference time versus end-to-end time

flowchart LR
    U["User"] -->|"network"| Q["Queue"] -->|"wait"| G["GPU inference"] -->|"network"| U2["User sees the answer"]

Inference time covers only the GPU box. End-to-end time covers every box and arrow.

The second distinction concerns what a measurement includes:

• Inference time is the time on the GPU needed to generate the tokens.
• End-to-end time also includes the network latency and any time the request spent waiting in a queue.

Both numbers are useful. Inference time shows how effective the model performance work is. End-to-end time shows how fast the application feels to users. When inference time is fast and end-to-end time is slow, the delay is somewhere else, so the next step is to improve the infrastructure instead of the model.

The calculation breaks one request into parts and uses a simple rule: when the model's part is a small share of the total, look elsewhere. The time values are examples.

Deployment A: model is slow
  network in 20 + queue 10 + inference 900 + network out 10 + other 20 = 960 ms
  inference share = 900 / 960 = 94 percent

Deployment B: infrastructure is slow
  network in 80 + queue 600 + inference 450 + network out 80 + other 40 = 1,250 ms
  inference share = 450 / 1,250 = 36 percent

In deployment A, inference is 94 percent of the end-to-end time, so faster kernels, quantization, or speculation will help. In deployment B, inference takes only 450 ms but the user waits 1,250 ms, and the largest single delay is the 600 ms queue. Making the model 20 percent faster there would save 90 ms. Adding replicas or autoscaling would save most of the 600 ms. The production lessons cover queues and scaling.

Example 5: which metric for which application

The application categories from the previous lesson map to metrics.

Interview points

• TTFT is the time until the first token appears. It depends on prefill, which is compute-bound. TPS is the speed after the first token, based on decode, which is bandwidth-bound.
• Lower TTFT and higher TPS are both better latency.
• "TPS" is ambiguous. Perceived TPS is per user (latency), total TPS is for the whole service (throughput), and ITL is the time between tokens. ITL of 10 ms equals 100 tokens per second per user.
• For agent tool calls and other non-streamed requests, measure total response time.
• LLM response times are right-skewed, so the mean is above the median and hides outliers. Use percentiles: P50, P90, P95, and P99.
• P90 means 1 in 10 requests is slower, P99 means 1 in 100.
• Inference time shows model performance work. End-to-end time includes network and queue time and shows what users perceive. A fast inference time with a slow end-to-end time points to infrastructure.

Summary

• The common LLM metrics are TTFT and TPS. TPS needs a qualifier: perceived (per user), total (service-wide), or ITL (gap between tokens).
• Perceived TPS = 1000 / ITL in milliseconds. Total time = TTFT + (tokens minus one) x ITL.
• Response times are right-skewed, and the mean is higher than P50.
• Report percentiles and name which one is used, and work on P90 and P99 as well as the average.
• Measure inference time and end-to-end time separately to decide between model and infrastructure work.