Testing, Deployment, Cost, and Observability

A system that is fast on one replica, packaged in a container, and autoscaled across clusters still has three jobs left before it can be trusted in production. It must be tested as a whole. It must be updated without users noticing. And its cost and health must be understood in numbers.

This lesson covers those jobs. It starts with test strategies, then zero-downtime deployment with blue-green and canary releases, then a cost method for comparing per-token APIs with dedicated GPUs, and finally the metrics that observability for inference should track. After it, a reader can plan a test, roll out a new version with a canary, compare two costs on equal terms, and read latency percentiles.

1. Testing before deployment

Replica-level testing and benchmarking happen while the inference engine is configured (see benchmarking and profiling). A single replica that performs well does not prove that routing, queueing, autoscaling, and networking work together. The whole system needs end-to-end testing before deployment. Three strategies exist:

Testing inference is expensive. It costs engineering time to set up the tests and read the results, and it costs GPUs to run the test traffic. That is partly the cost of doing business, but it is worth minimizing. One way: start shadow traffic testing by copying a random sample of production traffic, and follow it with a shorter load test.

AI product usage generally changes on daily and weekly cycles. A test that runs only at a quiet hour misses the busy hours, so the test period or the sampling must cover the pattern.

flowchart LR
    M["Manual test: scripts, synthetic"] --> S["Shadow traffic: random sample of live requests"] --> L[Shorter load test] --> D[Deploy]

2. Zero-downtime deployment

Once a new version looks stable and fast, it must reach production without an outage. Inference engineers use high-availability deployment strategies.

Blue-green

A traditional design is the blue-green deployment. Two identical environments exist: the original blue deployment, and a new green deployment running the updated service. When green is ready, the full traffic load cuts over from blue to green. Blue stays ready for a rollback if something goes wrong.

Blue-green fits poorly with large inference. It has the same GPU capacity and cost problem that makes large-scale testing hard. If blue uses 100 GPUs, green needs another 100 GPUs before traffic can cut over.

Canary

flowchart LR
    subgraph BG[Blue-green]
        T1[Traffic] --> Bl["Blue: 100 GPUs"]
        Gr["Green: 100 GPUs, waiting"] -.cut over all at once.-> T1
    end
    subgraph CA[Canary]
        T2[Traffic] -->|95%| Old["Old: 95 GPUs"]
        T2 -->|5%| New["New: 5 GPUs"]
    end

Blue-green holds two full copies during the release, and a canary splits one copy's worth of GPUs between old and new.

A canary deployment gives similar benefits with less GPU overhead. The name comes from the canaries once used to detect gas in coal mines: a canary deployment catches errors before they reach many users. It has four steps:

1. Build a new deployment of the inference service and get it ready for incoming requests.
2. Direct a small percentage of live traffic to the new service.
3. Monitor the new service and make sure it handles traffic correctly. Revert if there are issues.
4. Raise traffic gradually, while watching for issues, until the new deployment handles 100 percent of traffic.

flowchart LR
    A[5% to new, 95% to old] --> B[25% to new] --> C[50% to new] --> D[100% to new, old removed]

Shifting traffic step by step prevents many problems during an update. A canary can roll out fast, with a ramp of only a few minutes, or slowly, to confirm stability at each stage.

With autoscaling, a canary does not add much cost at scale. Sending less traffic to the production system makes it scale down some replicas, so the GPUs move from the old version to the new one. A worked comparison for a service that needs 100 GPUs at full traffic follows. The replica counts are proportional to the traffic share, a simplification.

blue-green:  100 GPUs (blue) + 100 GPUs (green) before cut-over   = 200 GPUs at the peak of the release

canary, with autoscaling following the traffic split:
  new share   new GPUs   old GPUs   total
     5%           5          95      100
    25%          25          75      100
    50%          50          50      100
   100%         100           0      100

The canary table totals 100 GPUs at every step, so the release needs no spare capacity beyond a small rounding margin, while blue-green doubles the GPU count while both versions are up.

One caution applies. With autoscaling, a new deployment starts at its minimum number of replicas when it receives no traffic. Throughout the canary, the new deployment needs enough active replicas to handle its share. If it has too few, requests queue up until autoscaling catches up, and users see a latency spike (autoscaling covers queues and cold starts). In the table above, the 5 percent step needs 5 replicas, so a minimum of 1 would force a wait.

3. Cost estimation

A public API has simple pricing: a price per million tokens times the number of tokens used. There are a few variables, such as cache hits versus cache misses for input tokens and discounts for high-volume users. The cost stays a linear function of usage.

Taking control of unit economics, and escaping per-token pricing, is one reason to invest in inference engineering. The mental change is hard. The blessing and curse of dedicated inference is that cost depends on many variables. That gives control, and it makes estimating hard. The factors:

Because of this complexity, and because input and output tokens cost differently, it is generally more productive to turn the token price into a total cost and compare that with the dedicated total. The reverse, working out a per-token price from a GPU bill, is harder to do reliably.

Two equations with example numbers:

Per-token API (volumes in millions of tokens)
  total_input_tokens  = 1000   (millions)
  total_output_tokens =  500   (millions)
  price_per_million_in  = $1.25
  price_per_million_out = $10
  input_cost  = 1000 x 1.25 = $1,250
  output_cost =  500 x 10   = $5,000
  total_cost  = 1,250 + 5,000 = $6,250

Dedicated deployment
  total_gpu_hours    = 1600
  price_per_gpu_hour = $3.50
  total_cost = 1600 x 3.50 = $5,600

In this example, dedicated GPUs cost $650 less over the period. Output tokens made up 80 percent of the API bill ($5,000 of $6,250), which shows why the two token types must be priced separately.

Cost estimates should cover a long time, ideally at least a week, to smooth out changes in usage. A dedicated deployment has another cost: the engineering time spent building and maintaining the inference system. That investment is justified by higher reliability, security, and control, and it belongs in the comparison. Added to the GPU costs it gives the total cost of ownership (TCO) for inference. For example, if the engineering time in the example above were worth $1,000 over the same period, the dedicated total would be 5,600 + 1,000 = $6,600, which is $350 above the API. The $1,000 figure is invented to show the method. The break-even point moves with every input, so the comparison must be redone with real numbers.

4. Observability

Inference is mission-critical, so it is monitored like any mission-critical part of an application, with alerting, logs, and observability built at the right level of abstraction. The first question is what to monitor. Seven groups of measurements:

A percentile is the value below which a given share of observations fall. P50 is the median. P90 means 90 percent of requests were at least that fast. P99 means 99 percent of requests were at least that fast, so only the slowest 1 percent take longer. The definitions of the latency metrics are in measuring latency and throughput. A worked case with ten request latencies, using the nearest-rank rule (the value at position ceil(p x n) in the sorted list):

latencies (ms): 120, 130, 125, 140, 135, 128, 500, 132, 138, 1200
sorted:         120, 125, 128, 130, 132, 135, 138, 140, 500, 1200   (n = 10)

P50 = position ceil(0.50 x 10) = 5  -> 132 ms
P90 = position ceil(0.90 x 10) = 9  -> 500 ms
P99 = position ceil(0.99 x 10) = 10 -> 1200 ms
mean = (120+125+128+130+132+135+138+140+500+1200) / 10 = 2748 / 10 = 274.8 ms

The mean of 274.8 ms describes no actual request. The median says most users wait about 132 ms, and P90 and P99 show that a minority wait 4 to 9 times longer. Tracking only the average would hide that tail.

These metrics depend on each other. A latency spike could come from higher request volume, but it could also come from longer input sequences. Viewing the metrics together shows what is happening and why.

When something breaks, engineers need information to fix it. Logs deliver it in real time: both server logs, and audit logs that show changes to an inference service. Observability must not be isolated. It should integrate deeply with existing observability and alerting tools such as Grafana, Datadog, PagerDuty, and Sentry, so that inference data appears next to the rest of the application.

Interview points

• End-to-end tests of routing, queueing, autoscaling, and networking are needed on top of replica benchmarks. The strategies are manual testing, load testing, and shadow traffic. Tests cost engineering time and GPUs, and a random sample of production traffic followed by a short load test limits that cost. Usage changes on daily and weekly cycles.
• Blue-green keeps a second full environment and cuts all traffic over, with the old one kept for rollback. It needs double the GPUs (100 GPUs become 200), so it is a poor fit for large inference.
• A canary sends a small share of live traffic to the new deployment, watches it, reverts on problems, and ramps to 100 percent. With autoscaling, GPUs move from old to new, so cost stays near flat.
• A canary deployment needs enough replicas in the new version for its share, or requests queue and users see latency spikes.
• Per-token API cost is linear in usage. Dedicated cost depends on batch sizing, traffic patterns, and sequence lengths. Compare totals over at least a week, and include engineering time for total cost of ownership.
• Observability tracks volume, request and response sizes, 2XX, 4XX, and 5XX codes, latency percentiles (P50, P90, P99) for time to first token, tokens per second, and end-to-end time, replica counts, CPU, host memory, GPU, and GPU memory use, and queue depth. Server and audit logs and integration with Grafana, Datadog, PagerDuty, and Sentry complete it.

Summary

• Test the whole system with manual tests, load tests, and shadow traffic, and keep test cost low.
• Blue-green needs double capacity, and canary deployments shift traffic in steps with little extra cost.
• Make sure a canary has enough replicas for its traffic share.
• Compare cost by converting token pricing to a total over a long period, and add engineering time for TCO.
• Monitor seven groups of metrics together, with percentiles, logs, and alerting tied into existing tools.