Interview Questions: Techniques, Modalities, Production

Short answers for senior and lead interviews on lessons 19 to 31 of the course: runtime techniques, serving different modalities, and running inference in production. Each group links to the lessons that teach the topic. Numbers marked as illustrative are chosen for the exercise and are not measurements or prices. The questions on foundations, models, hardware, and software are in Interview Questions: Models, Hardware, Software.

Quantization

Lesson: Quantization.

1. What does quantization change, and why does it speed up both prefill and decode?

It lowers the number of bits per value. Prefill is compute-bound, and Tensor cores at lower precision have twice the FLOPS. Decode is memory-bound, and each value is half the size, so the effective memory bandwidth doubles. In practice one precision step gives about 30 to 50 percent, not 100 percent, because of overhead such as scale factors and parts of the model that stay in higher precision.

2. Calculation: how much memory do the weights of a 70B model need at each precision?

FP16 : 70 x 2   = 140 GB
FP8  : 70 x 1   =  70 GB
FP4  : 70 x 0.5 =  35 GB (plus a small amount for scale factors)

With the 50 percent headroom for the KV cache, FP8 needs about 105 GB, so it fits on two 80 GB GPUs or one 141 GB H200, while FP16 needs about 210 GB.

3. Why does floating point often beat integer at the same bit width?

Dynamic range, not only the bit count, decides quality. A floating-point format has an exponent, so it can represent outliers (rare, very large values) and small values together. An integer format of the same width spreads its values evenly and loses either the outliers or the small values.

4. What is quantization granularity, and what are MXFP8, MXFP4, and NVFP4?

Granularity is how many values share one scale factor: a whole tensor, a channel, or a block. Finer granularity protects outliers and costs extra storage and compute. MXFP8 and MXFP4 use blocks of 32 values. NVFP4 uses blocks of 16 plus a 32-bit global scale. These are the microscaling formats that Blackwell supports.

5. Which parts of a model are most sensitive to quantization?

From least to most sensitive: weights, activations, KV cache, attention. Softmax usually stays in the original precision. Errors in the KV cache and attention compound across tokens, so those parts need more care and more testing.

6. What can an open-weight user do, and what can they not do?

They can do post-training quantization (PTQ), for example with ModelOpt. Quantization-aware training needs the training run, which they do not have.

7. How do you decide that a quantized model is good enough?

Compare against the original model with perplexity, a public benchmark (MMLU, SWE-bench), and a custom eval built from the product's own data. A difference that cannot be told apart from noise passes. Look at individual cases as well as averages.

8. Spot the mistake: "FP4 halves the weights again, so we will get 2x the speed on top of FP8."

The gain for one precision step is about 30 to 50 percent, not 100 percent, because of scale factors, higher-precision components, and other overhead. Measure on the real workload before planning capacity.

9. Which format would you choose for a laptop and which for a production cluster?

Integer formats suit local deployments (GGUF, dynamic quantization such as the 1.58-bit Unsloth builds). FP8 or MXFP8 is the usual production choice because quality is close to the original and Hopper and Blackwell Tensor cores support it.

Speculative decoding

Lesson: Speculative Decoding.

10. How does speculative decoding work?

Decode is memory-bound, so the GPU has idle compute. A cheap method guesses several next tokens. The target model checks all guesses in one forward pass and keeps the guesses that match what it would have produced. A pass that accepts N draft tokens yields N + 1 tokens, because the target always produces one more token itself. The output is unchanged, and the pass is more productive.

11. Which metric improves, and which does not?

TPS and ITL improve. TTFT does not, because prefill is unchanged.

12. Calculation: how many tokens does one pass give?

Acceptance rate a per draft token (independent), k draft tokens. Expected tokens per pass is 1 + a + a^2 + ... + a^k.

a = 0.8, k = 3 : 1 + 0.8 + 0.64 + 0.512 = 2.952 tokens per pass
a = 0.5, k = 3 : 1 + 0.5 + 0.25 + 0.125 = 1.875 tokens per pass
a = 0.5, k = 7 : about 1.99 tokens per pass

At a = 0.5, drafting seven tokens barely beats drafting three. Guesses deep in the draft count only if all earlier guesses were right.

13. Calculation: what is the speedup?

Time is measured in target passes. A draft token costs c = 0.05 of a pass. One round costs 1 + k x c and yields the tokens from question 12.

a = 0.8, k = 3 : cost = 1 + 3 x 0.05 = 1.15, tokens 2.952, speedup = 2.952 / 1.15 = 2.57
a = 0.5, k = 3 : cost = 1.15, tokens 1.875, speedup = 1.63
a = 0.5, k = 7 : cost = 1 + 7 x 0.05 = 1.35, tokens 1.992, speedup = 1.48

The longer draft at low acceptance is slower than the shorter one, so short drafts with a high acceptance rate win. In real systems acceptance falls with depth, which makes the effect stronger.

14. What lowers the gain?

A higher temperature (flatter probabilities, lower acceptance) and larger batch sizes (the spare compute disappears). The engine should disable speculation dynamically at high batch sizes. Using speculation to cut latency lowers throughput and raises cost per token.

15. Compare draft-target, Medusa, EAGLE, and n-gram speculation.

Lookahead Decoding fills the n-gram dictionary during inference and needs extra compute.

16. Spot the mistake: "We enabled speculation and TTFT dropped by half."

Speculation does not change prefill, so TTFT should not change. A change in TTFT points to another cause, such as a different configuration, caching, or load during the test.

KV caching and long context

Lesson: KV Caching, Prefix Reuse, and Long Context.

17. What is prefix caching, and why do pay-per-token APIs price cache hits lower?

Every engine caches keys and values within a request. Prefix caching extends this across requests: if a new request starts with a prefix already in the cache, prefill for that part is skipped, so TTFT and compute shrink. A cache hit costs the provider less, so it is priced lower.

18. How should a prompt be organized to get cache hits?

A prefix ends at the first token that differs, and autoregression means one different token changes everything after it. Put stable content first (system prompt, instructions, documents) and novel tokens last. A different first token means no reuse at all. Timestamps or user ids at the start of a prompt are a common cause of misses.

19. Calculation: how big is the KV cache?

bytes per token = 2 (key and value) x layers x KV heads x head dimension x bytes per value
Illustrative 70B shape: 80 layers, 8 KV heads, head dimension 128, 16-bit values
                = 2 x 80 x 8 x 128 x 2 = 327,680 bytes (about 0.31 MB)

one request of 32,768 tokens : 327,680 x 32,768 = 10 GiB
50 such requests at once     : about 500 GiB
with an 8-bit cache          : half of each figure

The cache of many long requests can be larger than the weights, so the GPU memory left after weights limits concurrency.

20. What are the four cache levels?

G1 is VRAM, G2 is CPU RAM, G3 is local SSD, and G4 is networked SSD. Bandwidth falls and size rises from G1 to G4. NVIDIA Dynamo KVBM moves blocks between levels. GB200-class machines have a fast G2 because of the CPU-GPU link.

21. What happens when the cache is full?

The cache is configured as a share of free VRAM, for example 0.8 of what remains after the weights are loaded. When it fills, entries are evicted and misses rise, so TTFT rises and load on prefill grows.

22. What is cache-aware routing, and why add a global cache?

A router sends a request to the replica that already holds its prefix. A global cache on G4 prevents loss of cached prefixes when autoscaling adds or removes replicas, but a read from it is slower than a G1 hit.

23. How do CacheBlend and LMCache differ from prefix caching?

They handle reuse of text that is not a prefix, by correcting position information and selectively recomputing entries. Plain prefix reuse is the dominant case because of autoregression.

24. What techniques handle long contexts?

Long context means a cache big enough to cause problems, often past 32K, 64K, or 128K tokens. Use FlashAttention, PagedAttention, and chunked prefill first, and then parallelism if one GPU is not enough. KV cache quantization stores more of the cache and reads it faster, but its errors compound across tokens, so test quality.

25. Spot the mistake: "Our hit rate fell after adding a new replica, so the cache is broken."

A new replica has an empty cache and the router may send it requests whose prefixes live elsewhere. Cache-aware routing sends requests to the replica that holds the prefix, and a global cache on G4 keeps prefixes available when replicas change.

Model parallelism

Lesson: Model Parallelism.

26. Calculation: how many GPUs does a model need at minimum?

Rule: precision in bytes x parameters in billions x a KV-cache multiplier, rounded up to the next instance size. The worked case uses a multiplier of 1.8.

DeepSeek-V3.1, FP8: 1 x 671 x 1.8 = about 1,208 GB -> rounds up to 1,440 GB = eight B200 GPUs
Four B200 GPUs (720 GB) hold the 671 GB of weights but leave too little for the KV cache.

70B model, FP8: 1 x 70 x 1.8 = 126 GB -> next size up, for example two 80 GB GPUs (160 GB)

The weights alone are not the requirement. The cache and buffers need space too.

27. Why does communication limit scaling?

NVLink and InfiniBand are slower than VRAM, so every exchange between GPUs costs time that a single-GPU deployment does not pay. Topology-aware parallelism puts the chattiest traffic on the fastest links.

28. Compare Tensor, Pipeline, and Expert Parallelism.

29. Calculation: how are experts placed with EP?

128 experts, EP8 -> 128 / 8 = 16 experts per GPU

The router is small, so it is replicated on every GPU. Many deployments use TP for attention and EP for the MoE layers.

30. Design question: how do you serve a model that needs more than one node?

Over InfiniBand, use TP inside nodes with PP between them (TP8PP2) for dense models, or EP across nodes (EP16) for MoE models. TP8PP2 favors latency and EP16 favors throughput. Use multiple nodes only when the model needs them. Otherwise extra nodes are better as replicas or for disaggregation.

31. Spot the mistake: "We put Pipeline Parallelism inside the node to cut latency."

Pipeline Parallelism has poor latency and utilization because later stages wait for earlier ones. Inside a node, Tensor Parallelism over NVLink is the better latency choice. PP is for crossing nodes.

Prefill-decode disaggregation

Lesson: Prefill-Decode Disaggregation.

32. What problem does disaggregation solve?

Prefill is compute-bound and sets TTFT, and decode is memory-bound and sets TPS. On one node under heavy load, they compete: a long prefill stalls decode steps and heavy decode slows prefill. Separate engines for each phase remove the interference.

33. Describe the request flow and its added cost.

A prefill engine builds the KV cache and the first token. The cache moves over the interconnect to a decode engine, which produces the remaining tokens. The KV transfer is the cost added compared with one node.

34. What is conditional disaggregation?

The decode engine checks whether the prompt is already cached or short, and prefills locally if so, skipping the transfer. It suits real traffic better than always disaggregating. Dynamo uses a threshold on the uncached prompt length.

35. When is disaggregation worth it?

Only with large volume (about 100 million to 1 billion tokens per day), a large model (100B parameters or more), and prefill-heavy traffic. Otherwise add replicas, which is simpler and often cheaper.

36. What can be tuned separately, and what notation describes the layout?

Each engine has its own settings, for example prefill with a lower TP than decode. The xPyD notation gives engine counts, so 5P3D is five prefill and three decode engines, and the counts can change at run time. Dynamo supplies a prefill queue, threshold-based routing, and NIXL-based KV transfer with a kernel that transposes the layout when TP differs.

37. What new bottlenecks appear?

Prefill queue length (fix with the local-prefill threshold and xPyD changes) and decode KV cache space (fix with quantization and offloading).

Vision language and embedding models

Lesson: Vision Language Models and Embedding Models.

38. Why do LLM techniques carry over to other modalities?

Almost all generative models are autoregressive token generators or iterative denoisers. So KV quantization, EAGLE speculation, prefix caching for images, TP, and disaggregated prefill all apply to VLMs, embedding models, ASR, and TTS.

39. What is a VLM, and why does the small encoder matter?

A VLM is an LLM plus a vision encoder. In Mistral Large 3 the encoder has about 2B parameters against 673B for the language model. It is small, but encoder designs vary, so engine support (vLLM, SGLang) decides which models run well.

40. Calculation: how many tokens does an image or video cost?

One high-resolution image is about 1,000 tokens, and low resolution is about four times fewer.

4 seconds x 24 frames per second = 96 frames
96 frames x 1,000 tokens         = 96,000 tokens

A short video fills most of a long context, so downsampling frames is practically required.

41. Why do VLM pipelines often use several models?

Dedicated OCR and other small models often beat omni models in their domain. Production VLM serving is often a pipeline in which each stage is optimized and scaled separately.

42. Compare the two traffic profiles of embedding models.

Backfill (embedding a whole corpus) needs throughput. Lookups at query time need low latency. If both are needed at scale, serve them with separate systems.

43. What architectures do embedding models use, and what are Matryoshka vectors?

BERT-style models (under 1B parameters) and LLM-based models (up to 8B). Matryoshka vectors let a user trade vector dimension for quality by keeping the first dimensions. Vectors from different models are not comparable.

44. Calculation and design: embed 100 million documents in a batch pipeline.

Illustrative numbers: 300 tokens per document, 200,000 tokens per second per GPU.

total tokens  = 100,000,000 x 300 = 30 billion
GPU-seconds   = 30,000,000,000 / 200,000 = 150,000 s = 41.7 GPU-hours
with 10 GPUs  = about 4.2 hours

Design: TensorRT-LLM is the best runtime for supported embedding models, with large batches and a queue in front. Scale by replicas, since embedding requests are independent. Check FP8 quality by cosine similarity of at least 99 percent against the original vectors. Offline work can use spot GPUs and may run beyond peak hours.

45. Spot the mistake: "We upgraded the embedding model and kept the old index."

Vectors from different models are not comparable, so the whole corpus must be re-embedded, and queries must use the same model. Plan a backfill and a switch-over.

Speech models

Lesson: Speech: ASR, Diarization, TTS, Speech-to-Speech.

46. What does Whisper's architecture imply for serving?

Whisper is an encoder-decoder model of at most 1.55B parameters. Almost all time goes to the autoregressive decoder, so TensorRT-LLM (in-flight batching, C++ runtime, CUDA kernels) is the main tool, and MIG slices on H100, Hopper, or Blackwell fit the small model.

47. How is live transcription served?

The target is about 200 ms per chunk. Streaming is a feature of the API server: WebSocket in and out, and VAD (voice activity detection) to chunk the stream. The same GPU handles a given stream so the previous output can be the next prefix.

48. Calculation: what is RTF, and how do you transcribe a long file fast?

RTF (real-time factor) is audio length divided by processing time.

1 hour of audio = 3,600 s, processed in 3.6 s -> RTF = 3,600 / 3.6 = 1,000X
Whisper takes 30-second chunks: 3,600 / 30 = 120 chunks

Pipeline: VAD on its own hardware, chunks in parallel over GPUs or MIG slices with in-flight batching (RTF scales roughly linearly with GPUs), and stitching the pieces by timestamp.

49. How do you detect and fix Whisper hallucinations?

Detect with the compression ratio of the text and words per minute. Fix by re-running with a higher temperature or by re-chunking the audio.

50. How does diarization differ from transcription in serving?

Diarization (who spoke when) is a pipeline of classic models: segmentation, embedding, and clustering, for example pyannote audio. It takes at least twice as long as transcription even when optimized.

51. How are text-to-speech models built, and how are they served?

TTS models are fine-tuned LLMs. Orpheus comes from Llama 3.2 3B with audio tokens added to the vocabulary and a separate audio decoder (SNAC). Weights and KV cache can use FP8. The decoder is compiled PyTorch with dynamic batching and about a 15 ms timeout, with no in-flight batching.

52. Calculation: how many TTS streams can one GPU hold?

Real time needs only about 80 to 100 tokens per second per stream.

Illustrative GPU output: 2,000 tokens per second in total
2,000 / 100 = 20 concurrent real-time streams (25 at 80 tokens per second)

A stream faster than real time wastes the capacity. TTS metrics are TTFB (about 150 ms for Orpheus on one H100), time to first sentence, and TPS. Speech degrades past about 30 seconds of input.

53. Design question: a real-time voice agent.

A cascade of ASR, LLM, and TTS, plus VAD and embeddings. Illustrative latency budget: ASR chunk 200 ms, LLM time to first token 300 ms, TTS first byte 150 ms, total about 650 ms before network time. Rules: keep all stages in one cluster to avoid hops, stream between stages (first LLM sentence goes to TTS at once), use WebSockets for audio, and pin a stream to one GPU. Scale by concurrent streams. Cascading beats closed speech-to-speech models such as gpt-realtime in capability and cost today, and it lets each stage be swapped and tuned.

54. Spot the mistake: "TTS needs the fastest GPU because speech must be fast."

Real-time speech needs only about 80 to 100 tokens per second per stream. Above that, extra speed is wasted for one stream. Scale by the number of concurrent streams, and small models can share a GPU through MIG.

Image and video serving

Lesson: Serving Image and Video Generation Models.

55. Why are image and video models served differently from LLMs?

They are mostly iterative denoisers built as pipelines of small models in latent space, about ten to twenty times smaller than frontier LLMs, and compute-bound instead of bandwidth-bound. Tooling is newer: SGLang Diffusion and vLLM Omni are recent, and many deployments use PyTorch or TensorRT directly.

56. Why is a diffusers model-card example slow?

It is barely optimized. Memory-efficient kernels and fusion are needed to even become compute-bound. The key kernels are attention (FlashAttention 3 on Hopper, 4 on Blackwell), fused normalization such as RMSNorm, and GEMM kernels (CuTe, CUTLASS, DeepGEMM) in FP8 for twice the Tensor Core FLOPS.

57. What do you need to know about torch compilation for diffusion?

It fuses kernels, takes minutes, and is specific to the GPU type. Compile on the target GPU and cache the result so new replicas do not pay the cost at start.

58. Calculation: what does skipping guidance save?

Each guided step runs two forward passes. Skipping guidance for the last 20 of 50 steps:

30 guided steps x 2 passes + 20 unguided x 1 pass = 60 + 20 = 80 passes (down from 100)

Early steps set the outline of the image, so guidance matters there. Test quality because late steps affect fine detail.

59. How do you evaluate image quality?

Human preference is the reliable measure. VLM judges are only directional.

60. What changes for video?

Blackwell (or Rubin) is preferred, batch size is one on eight GPUs, and about 50 steps run over a latent with width, height, and time, with attention taking 70 to 80 percent of compute. Video caching (timestep-based or transformer-based) gives 30 to 40 percent speedups but can ruin quality, so test it. Quantize attention carefully: MXFP8, FP16 for early steps, and first and last layers kept in high precision. SageAttention is an 8-bit attention kernel. Context Parallelism replicates weights and splits attention across GPUs with ring attention, and the VAE decoding step (3 to 5 percent of time) can also be parallelized.

61. Design question: an image generation service.

Illustrative requirements: spiky consumer traffic, 8 second target per image. Choose the model and a few-step variant if quality allows (80 to 90 percent faster, with visible quality loss). Use a compiled, FP8-quantized pipeline built on the target GPU and cached in the image or a nearby store. Run text encoder, denoiser, and VAE in one replica to avoid network hops, or as separate scaled stages if sizes differ. Autoscale on concurrency, keep a queue in front, and use a warm minimum replica count because cold starts are long (compile and weight loading). Measure P50, P90, and P99 end-to-end time, and score quality with human preference samples.

62. Spot the mistake: "We cached video timesteps and got 40 percent faster, so we shipped it."

Caching can ruin quality, and the speedup range is 30 to 40 percent only when quality holds. Test with representative prompts and human review before shipping.

Containerization

Lesson: Containerization, Dependencies, and NIMs.

63. Define image, container, Dockerfile, and registry.

The image is the built package. The container is the running environment. The Dockerfile is the build recipe. The registry stores images (Docker Hub for images, comparable to Hugging Face for weights and PyPI for Python packages). An image has base layers, layers from Dockerfile steps, and a thin ephemeral container layer discarded when the container ends.

64. What belongs in an inference image?

The CUDA toolkit version (CUDA, cuDNN, drivers), Python packages such as torch, transformers, and diffusers, the inference engine version, and system packages such as ffmpeg. Pin every dependency to an exact version so a rebuild resolves identically, and keep the image small because images are often many gigabytes.

65. Why start from an official vLLM or SGLang base image?

Those images already hold a tested combination of CUDA, drivers, and engine. Starting from one is generally better than building from scratch. Day-zero support for a new model often needs pre-release builds, which are later rebuilt on stable releases.

66. Compare multi-LLM NIMs, LLM-specific NIMs, and a custom container.

A multi-LLM NIM serves a model family on a supported architecture. An LLM-specific NIM is tuned for one model on one GPU setup. Build a custom container when maximum control matters more than a ready configuration.

67. Spot the mistake: "pip install torch" without a version in the Dockerfile.

An unpinned dependency can resolve to a different version on the next build, and a new version can change behavior or break kernels. Pin exact versions.

Autoscaling, batching, cold starts, and routing

Lesson: Autoscaling, Batching, Cold Starts, and Routing.

68. What does autoscaling do, and what is the architecture?

It keeps enough replicas to meet latency SLAs without paying for idle GPUs. It runs on Kubernetes, where a control plane decides routing and scaling and a worker plane runs the containers.

69. Utilization or traffic: which signal should scale replicas?

Utilization is a lagging signal, since load must already be present. Traffic can be used proactively. They can disagree, because a few huge uncached prompts load a GPU more than many small cached ones. Use both.

70. Name the five traffic-based autoscaling settings.

Min replicas, max replicas, autoscaling window, scale down delay, and concurrency target. A longer scale down delay protects against spiky traffic and wastes money after traffic cools.

71. Calculation: how many replicas does a load need?

Illustrative: 40 requests per second, each lasting 6 seconds, replica concurrency target 32, 20 percent headroom.

requests in flight = rate x duration = 40 x 6 = 240
replicas           = 240 / 32 = 7.5 -> 8
with headroom      = 8 x 1.2 = 9.6 -> 10 replicas

A token-based check: 50,000 tokens per second of demand at 3,000 tokens per second per GPU needs 50,000 / 3,000 = 16.7, so 17 GPUs. The concurrency target and the replica batch size must match, or requests queue even though capacity exists.

72. Calculation: what does a token cost on a dedicated GPU?

Illustrative: 4 dollars per GPU-hour, 3,000 tokens per second.

tokens per hour at full use = 3,000 x 3,600 = 10,800,000
cost per million tokens     = 4 / 10.8 = 0.37 dollars
at 40 percent utilization   = 0.37 / 0.40 = 0.93 dollars

Utilization drives cost per token. Autoscaling that removes idle GPUs lowers cost directly.

73. Compare static, dynamic, and continuous batching.

Static batching waits for a full batch. Dynamic batching adds a timeout so a partial batch runs. Continuous (in-flight) batching swaps requests in and out at the token level, so a finished request frees its slot immediately. Bigger batches raise throughput and worsen per-user latency.

74. List the four steps of a cold start and how to shorten each.

GPU procurement, image loading, model loading, and engine startup. Use smaller images, quantized weights, weights cached near the GPU in the same datacenter, and cached compiled engines. A compiled engine is valid only for the same GPU type, CUDA version, and dependencies. Weights load separately from the image because they are far larger than the image for current models.

75. Router, load balancer, and queue: what does each do?

A load balancer answers where a request could go and a router answers where it should go. KV cache-aware and LoRA-aware routing send requests to replicas that already hold the prefix or the adapter. A queue holds excess traffic, can be a priority queue, and must hand queued requests to new replicas, up to their concurrency limit, as soon as they are active.

76. When is scale to zero a good idea?

It needs fast cold starts and reliable queueing and fits development and periodic workloads. For a latency-sensitive app with light traffic it is a warning sign, and a pay-per-token API is the better fit.

77. Why keep an inference pipeline in one cluster?

Network hops add latency per hop. At 40 ms extra per hop over 5 hops, that is 200 ms, which is 20 percent of a 1 second SLA. Pipelines scale per component but run in one cluster.

Multi-cloud capacity and reliability

Lesson: Multi-Cloud Capacity, Reliability, and Security.

78. What is the difference between siloed and true multi-cloud?

Siloed multi-cloud uses several clouds with no fluid use of compute across them. True multi-cloud has a multi-region, multi-provider bin-packing tool that treats pools of compute as fungible (interchangeable), takes a global view, and supports self-healing and global scheduling. The benefits are capacity, redundancy, latency, and compliance.

79. How is the control plane separated from the workload planes?

The control plane handles deployment, global scaling decisions, and real-time event streams. Workload planes handle inference traffic and in-cluster scaling and report utilization and demand. They serve independently, so one failure does not take down the rest.

80. Compare reserved, on-demand, and spot GPUs. Follow-up: how do large systems mix them?

Reserved GPUs are discounted for months or years. On-demand GPUs cost more per hour up to a quota. Spot GPUs are discounted but can be pre-empted after a notice that is often minutes. Large systems use reserved for the baseline and on-demand plus spot for peaks. Provider types are hyperscalers (AWS, GCP, at a premium), neoclouds (Coreweave, Nebius), and resellers (SF Compute Company), and each trades cost against uptime SLAs, support, regions, instance types, and cluster size.

81. Calculation: how far apart are regions in latency?

The rule of thumb is 5 ms per time zone. New York to San Francisco is three zones, so 3 x 5 = 15 ms one way, about 30 ms round trip. A global load balancer avoids both queues while capacity sits idle elsewhere and long trips for latency-sensitive requests.

82. Calculation: how many GPU failures should a fleet expect?

Llama 3 training had 419 unexpected interruptions over 54 days on 16,000 GPUs, about one failure per 50,000 GPU-hours, and GPUs were the largest cause.

one 8-GPU node for a year : 8 x 24 x 365 = 70,080 GPU-hours -> about 1.4 failures
a 100-GPU fleet for 30 days: 100 x 24 x 30 = 72,000 GPU-hours -> about 1.4 failures

Failures are routine, not rare. Keep node health by recording failures, cordoning nodes, and cycling pods, because a failed GPU is often followed by more failures on the same node.

83. Compare active-active and active-passive.

Active-active serves live traffic from several planes, so a loss shrinks capacity but not availability. Active-passive keeps an idle hot standby and cuts over on failure, which costs idle GPUs and takes time during the cutover.

84. Design question: multi-region failover for a chat service.

Illustrative: two regions on two providers. Run both active with a global load balancer and health checks, each sized so the surviving region can take the whole load at reduced quality of service, or hold reserved capacity for the baseline and bring on-demand or spot GPUs for peaks. Keep the control plane separate from the workload planes so a plane outage does not stop deployment. Pre-warm weights and compiled engines in both regions to shorten cold starts. Route by latency (about 5 ms per time zone) and by compliance rules, because some data cannot leave a country. Test failover with game days, and watch the queue depth and P99 during a cutover.

85. What does inference security cover?

User data, model weights, and infrastructure. Not storing inputs and outputs is one of the simplest improvements. Standard measures are encryption, container security, access controls, isolation, and penetration testing. SOC 2 Type II and HIPAA need compliant providers, and data residency rules need clusters in the right countries.

Testing, deployment, cost, and observability

Lesson: Testing, Deployment, Cost, and Observability.

86. Why test beyond replica benchmarks?

Routing, queueing, autoscaling, and networking affect behavior only when the whole system runs. The strategies are manual testing, load testing, and shadow traffic. Tests cost engineering time and GPUs, so use a random sample of production traffic and a short load test. Usage changes on daily and weekly cycles.

87. Compare blue-green and canary deployments for GPU services.

Blue-green keeps a second full environment and cuts all traffic over, with the old one kept for rollback. It needs double the GPUs (100 GPUs become 200), so it is a poor fit for large inference. A canary sends a small share of live traffic to the new deployment, watches it, reverts on problems, and ramps to 100 percent. With autoscaling, GPUs move from old to new, so cost stays near flat.

88. What is a common failure of canary releases?

The new version must have enough replicas for its share. With 10 replicas in total, 5 percent of traffic is half a replica, so at least one replica is needed, which is 10 percent of traffic. If the new version has fewer replicas than its share needs, requests queue and users see latency spikes.

89. How do you compare API cost with dedicated cost?

Per-token API cost is linear in usage: price per million tokens times tokens, with cache hits and volume discounts as variables. Dedicated cost depends on batch sizing, traffic patterns, and sequence lengths. Compare totals over at least a week, and include engineering time for total cost of ownership.

90. What should observability track?

Volume, request and response sizes, 2XX, 4XX, and 5XX status codes, latency percentiles (P50, P90, P99) for time to first token, tokens per second, and end-to-end time, replica counts, CPU, host memory, GPU, and GPU memory use, and queue depth. Server and audit logs and integration with Grafana, Datadog, PagerDuty, and Sentry complete the picture.

91. Spot the mistake: "We tested on a quiet Tuesday and the new deployment passed."

Usage changes on daily and weekly cycles, so a quiet period hides queueing and autoscaling behavior. Sample production traffic from different times and include a peak-like load test.

Client code

Lesson: Client Code: Latency, Async Inference, Streaming.

92. Define client and server here, and name common client libraries.

The client is the browser, agent, or application, and the server is the inference service. The OpenAI SDK is the industry standard client. LangChain, Vercel AI SDK, LiteLLM, and LlamaIndex can serve as clients too.

93. Calculation: how much of a latency budget does TLS use?

SLA: 300 ms at P95. A TLS handshake costs at least 30 ms.
30 / 300 = 10 percent of the budget before inference starts

Reusing sessions avoids repeating the handshake. Custom clients for non-standard modalities must do this deliberately.

94. When should a request be asynchronous?

Asynchronous inference is fire and forget: the server acknowledges at once and posts the result to a webhook. It suits bulk document processing and corpus embedding, and its limits are hours instead of the minutes allowed for synchronous requests.

95. Compare streaming over HTTP, WebSockets, and gRPC.

Live voice and video need bi-directional protocols.

96. Spot the mistake: "Server inference time is 120 ms, but users report 400 ms."

On-server inference time is only a fraction of end-to-end time. The rest is network time, TLS handshakes, and queue time. Measure end-to-end time at the client and compare it with server time to find where the gap is.

Summary

• Quantization lowers bits per value and gives about 30 to 50 percent per precision step. Granularity, dynamic range, and component sensitivity decide quality, and quality is checked with perplexity, benchmarks, and custom evals.
• Speculative decoding gives N + 1 tokens per pass and improves TPS and ITL. Short drafts with a high acceptance rate win, and speculation is switched off at large batch sizes.
• Prefix caching depends on stable content first. KV cache size is 2 x layers x KV heads x head dimension x bytes per value, per token.
• Minimum GPUs follow bytes x parameters x a cache multiplier. TP suits one node, PP crosses nodes, and EP raises MoE throughput.
• Disaggregation pays off at large volume, with large models and prefill-heavy traffic, and its added cost is the KV transfer.
• VLMs, embedding models, speech, image, and video models reuse the same techniques, with different bottlenecks, token counts, and metrics such as RTF and TTFB.
• Production work centers on containers with pinned dependencies, autoscaling with matched concurrency, short cold starts, and routing that knows about caches.
• Multi-cloud design mixes reserved, on-demand, and spot capacity, plans for routine GPU failures, and separates control and workload planes.
• Canary releases keep GPU cost flat. Observability tracks percentiles and queue depth, and clients must account for network, TLS, and protocol choices.