Interview Questions: Models, Hardware, Software

Short answers for senior and lead interviews on the first 18 lessons of the course: foundations, models, hardware, and software. Each group links to the lessons that teach the topic. Calculations are worked step by step, and the numbers marked as illustrative are chosen for the exercise and are not vendor prices or measurements. The questions on techniques, modalities, and production are in Interview Questions: Techniques, Modalities, Production.

Foundations: the field and the product

Lessons: What Is Inference Engineering, Scale, Specialization, and Your App.

1. What is inference, and how does it differ from training?

Training learns the weights (the numbers inside the model) from data. Inference is the phase where the finished model answers production requests. Training runs once or occasionally. Inference runs for every request, all day, so its cost, speed, and reliability are ongoing concerns.

2. Why is serving a generative model harder than downloading the weights and renting a GPU?

The models are large and need GPUs, and a plain setup gives slow, costly, and unreliable serving. The work is in batching requests, caching, lowering precision, speculating tokens, splitting the model across GPUs, scaling replicas up and down, routing traffic, surviving hardware failures, and watching quality and cost. The weights are only the starting point.

3. Name the three layers of a complete inference stack.

Problems in one layer often show up in another, so the layers must be designed together.

4. Which six runtime techniques does the course group together?

Batching, caching, quantization, speculation, parallelism, and disaggregation. They apply to every modality because almost all generative models are either token generators or iterative denoisers.

5. How do infrastructure problems change as the fleet grows?

Autoscaling is the first problem. Past a few hundred GPUs, capacity becomes the problem, because one provider may not have enough. Then separate silos of compute appear, and the goal becomes one global pool of compute that the scheduler treats as interchangeable.

6. Compare closed and open models for a production product. Follow-up: what must be checked before using an open model?

A closed model has no public weights and is reached through the owner's API. An open model has public weights, usually under MIT or a similar license. Open models allow dedicated deployments that can tune latency, reach four nines of availability or better, and often cost at least 80 percent less at scale. Follow-up answer: read the license terms, because licenses differ in what use and redistribution they allow.

7. Calculation: how much downtime per year does each availability level allow?

minutes per year = 365 x 24 x 60 = 525,600

two nines  (99%)     : 1% x 525,600     = 5,256 minutes (about 87.6 hours)
three nines (99.9%)  : 0.1% x 525,600   = 525.6 minutes (about 8.76 hours)
four nines (99.99%)  : 0.01% x 525,600  = 52.56 minutes
five nines (99.999%) : 0.001% x 525,600 = 5.256 minutes

Each extra nine divides the allowed downtime by ten. A shared API at two nines can be down for several days a year in total. A dedicated deployment at four nines is down under an hour.

8. When should a team use shared inference, and when should it move to dedicated deployments?

Shared inference means paying per token on a public endpoint. It has no cold starts and almost no engineering work, so it fits early products where requirements are still changing. Its cost grows linearly, the provider caps the uptime, and the team has no control over latency, quality, or rate limits. The three reasons to move are scale, specialization, and orchestration. Switch only for a clear and immediate business need, because dedicated deployment adds engineering effort and a higher monthly floor.

9. Calculation: at what volume does a dedicated GPU beat a per-token price?

Illustrative numbers: shared price 0.60 dollars per million tokens, one GPU costs 4 dollars per hour and serves 3,000 tokens per second.

tokens per GPU-hour = 3,000 x 3,600          = 10,800,000
cost per million at full use = 4 / 10.8      = 0.37 dollars
break-even utilization = 0.37 / 0.60         = about 62 percent

Below about 62 percent average utilization, the shared price is lower in this example. Above it, the dedicated GPU is cheaper. Real traffic is uneven, so the average utilization of a dedicated deployment is lower than its peak, and that gap decides the answer.

10. Contrast online and offline workloads, and consumer and B2B products.

Online workloads (chat, voice, agents) optimize latency because a person or a program waits. Offline workloads (bulk document processing, embedding a corpus) optimize throughput and cost. One model may need two deployments for the two uses. Consumer apps stress marginal cost and spiky traffic. B2B apps stress latency, uptime, and often compliance.

11. What are the three compliance questions that can restrict where inference runs?

Data sovereignty (which country the data may live in), user privacy (what is stored and who can see it), and regulatory compliance (for example SOC 2 or HIPAA). They can rule out a provider, a region, or a shared endpoint.

12. Spot the mistake: "We picked the fastest benchmark result, so our product is optimized."

Optimization balances latency, throughput, and quality for one specific product. The five facts to establish first are the model requirements, the application interface, the latency budget, the unit economics, and the usage patterns. A single maximized metric can break another, such as a higher throughput that raises per-user latency past the budget.

Foundations: model choice and latency metrics

Lessons: Choosing a Model, Measuring Latency and Throughput.

13. Why does model choice matter more than the runtime?

The parameter count sets memory use, data movement, and arithmetic for every request. With everything else equal, a smaller model is faster and cheaper. The choice between an 8B and a 70B model changes the memory and arithmetic per token by almost nine times, which is more than most runtime techniques can recover.

14. Calculation: how much memory do the weights need?

weight memory = parameters x bytes per parameter

8B model,  FP16 (2 bytes): 8 x 2  = 16 GB
70B model, FP16 (2 bytes): 70 x 2 = 140 GB
70B model, FP8  (1 byte) : 70 x 1 = 70 GB
70B model, FP4 (0.5 byte): 70 x 0.5 = 35 GB (plus a little for scale factors)

The KV cache and other buffers come on top of these figures, so the GPU needs more memory than the weights alone.

15. What are benchmarks and evals, and why are both used?

Benchmarks such as MMLU and SWE-bench measure common tasks. They help shortlist models, but many are saturated or gamed, which is Goodhart's Law (a measure that becomes a target stops measuring well). Elo rating from head-to-head wins is a better general measure. Evals are tests built for one product's real tasks. They show whether a model is worth speeding up and give the quality baseline against which every later optimization is checked.

16. When does fine-tuning make sense? Follow-up: how does it differ from distillation?

Fine-tuning adapts a pre-trained model with new data. It changes the weights and keeps the architecture. It needs a cleanly scoped domain, strong eval criteria, and high-quality labeled data. In the text-to-SQL example, a model of a few billion parameters matched a model of hundreds of billions on that one task. Follow-up answer: distillation trains a smaller student on the teacher's probability distributions, not only its answers, so it copies the teacher's behavior, good and bad. It is used less than fine-tuning. DeepSeek-R1 (671B) shipped with distills on Llama 3 and Qwen 2.5.

17. Define TTFT, TPS, and ITL, and say which phase sets each.

TTFT (time to first token) is the wait before the first token appears and depends on prefill, which is compute-bound. TPS (tokens per second) is the speed after the first token and depends on decode, which is memory-bandwidth-bound. ITL (inter-token latency) is the gap between tokens and equals 1000 divided by the per-user TPS in milliseconds.

18. Calculation: how long does a response take?

Illustrative: TTFT 400 ms, ITL 20 ms, 301 output tokens.

per-user TPS = 1000 / 20            = 50 tokens per second
total time   = TTFT + (tokens - 1) x ITL
             = 0.4 s + 300 x 0.02 s = 6.4 s

The first token comes from the prefill step, so only 300 more tokens come from decode.

19. Why does "TPS" need a qualifier?

It has three meanings. Perceived TPS is per user and is a latency measure. Total TPS is for the whole service and is a throughput measure. ITL is the time between tokens. A system can have a high total TPS and a poor per-user TPS if it batches heavily.

20. Why report percentiles instead of the mean? Follow-up: what does P99 mean?

Response times are right-skewed: most requests are quick and a few are very slow, so the mean is above the median and hides the slow ones. P50, P90, P95, and P99 describe the distribution. P90 means one request in ten is slower than that value, and P99 means one in a hundred. Follow-up answer: always name which percentile a figure is.

21. Spot the mistake: "Inference time is 80 ms, so users see 80 ms."

Inference time covers model work only. End-to-end time adds network time and queue time and is what users perceive. If inference time is fast and end-to-end time is slow, the fix is in infrastructure (routing, queueing, network), not in the model.

Models: neural networks and LLM generation

Lessons: Neural Networks for Inference Engineers, How an LLM Generates Text.

22. Why do deep networks need activation functions?

A matrix multiplication followed by another matrix multiplication is a single matrix multiplication, so a stack of linear layers collapses into one layer. A non-linear activation function between layers prevents that and lets depth add capacity. Activations are also mostly differentiable, which back-propagation needs. ReLU returns x for positive x and zero otherwise. SiLU, Swish, and SwiGLU share the pattern of squashing negatives and keeping positives.

23. Calculation: how many parameters does a linear layer have?

A layer with n inputs and m outputs computes y = xW + b, with an n x m weight matrix and m biases.

n = 4,096, m = 4,096
weights = 4,096 x 4,096 = 16,777,216
biases  = 4,096
total   = 16,781,312 parameters (about 16.8 million)

A model with billions of parameters is mostly many layers of this kind.

24. Compare encoder-only, decoder-only, and encoder-decoder models, with one example each.

An encoder builds an internal representation of an input and a decoder generates output from a representation. Modern LLMs are decoder-only. BERT-style embedding models are encoder-only. Whisper is encoder-decoder.

25. What are the two generation styles in transformer-based models?

Autoregressive token generation (LLMs, speech synthesis from tokens) and iterative denoising (image and video generation). They have different bottlenecks, which is why the optimization techniques differ in effect.

26. What happens in one LLM request from text to text?

The tokenizer maps the text to token ids. Prefill processes the whole input in one pass and builds the KV cache. Decode then runs one forward pass per output token: the output layer produces one logit per vocabulary token, normalization turns logits into probabilities, and sampling picks a token. The loop repeats until the stop token, max_tokens, or the context window limit.

27. What do temperature, top-k, and top-p do? Follow-up: how do you get deterministic output?

Temperature adjusts the logits before normalization (a lower value sharpens the distribution). Top-k keeps the k most likely tokens. Top-p keeps the smallest set of tokens whose probabilities add up to p. Follow-up answer: temperature 0 or top-k of 1 always takes the most likely token. Logit biasing can also force or ban tokens, which helps structured output and tool use.

28. What is a chat template and why does it cause bugs?

It flattens roles, tool signatures, and multimodal inputs into one token sequence. Each model has its own format, and a mismatch (wrong role markers, missing special tokens) gives worse answers without any error message. The engine must implement the model's template correctly.

29. How do you read an architecture name such as Qwen3MoeForCausalLM?

Qwen is the family, 3 is the major version, Moe means Mixture of Experts, and CausalLM means a causal language model. The config.json file holds these details. Sizes, base and instruct variants, and LoRA fine-tunes share an architecture, so engine support and optimizations carry across them.

30. Spot the mistake: "Our tokenizer needs a GPU, so tokenization is the bottleneck."

A tokenizer is a plain mapping between strings and token ids and needs no neural network. It runs on the CPU. Token count still matters: fewer tokens for the same output means fewer decode passes and faster responses.

Models: attention, mixture of experts, and attention optimization

Lessons: Attention and Mixture of Experts, Optimizing Attention.

31. Explain scaled dot-product attention in words. Follow-up: what are Q, K, and V?

Attention relates a token to other tokens: softmax(Q K^T / sqrt(d_k)) V. Q (queries) represents the token being updated, K (keys) represent prior tokens, and the match between Q and K gives weights that blend the V (values). Multi-head attention runs several of these in parallel so each head can learn a different relationship, such as subject-verb agreement.

32. Compare self-attention and cross-attention.

Self-attention takes Q, K, and V from one sequence and LLMs use it with a causal mask (a token sees only earlier tokens). Cross-attention takes Q from one sequence and K and V from another, which image generation and multimodal models use to condition one sequence on another.

33. Why is attention expensive, and how does the KV cache help?

Cost grows with the square of the sequence length, because every token is compared with every other token. During decode, the cache stores the keys and values of earlier tokens, so each new token only adds one query against stored keys. A decode step is then linear in length. The cache is built during prefill and lives in GPU memory.

34. How does a Mixture of Experts model save work, and where does the saving disappear?

It replaces one big matrix with many smaller experts and a router that activates a few per layer. Qwen3-235B-A22B has 128 experts, picks eight per layer across 94 layers, and uses 22B of 235B parameters per token, so about 9 percent are active. In batched serving, different requests pick different experts, so almost all parameters are active and all must sit in memory. Expert Parallelism restores the saving at scale.

35. When is a dense model the better choice than a MoE model?

Dense models are efficient below about 32B parameters, especially below 8B, and for single-purpose models such as tab completion. MoE shines for single-request local inference and for very large models served with Expert Parallelism.

36. Why are attention errors costly in quantization or approximation?

Errors in attention accumulate across tokens, because each generated token depends on all earlier ones. This makes attention the most sensitive part of the model and the reason its optimization is careful work.

37. What does FlashAttention change, and what does it not change?

The standard algorithm writes the intermediate matrices S and P to memory and reads them straight back, and most of its memory traffic is that. FlashAttention fuses the steps in hand-tuned, GPU-specific kernels so the intermediates stay on the chip. The result is the same math with far less memory traffic. It stays quadratic in length, and it helps most for compute-bound work such as prefill and video generation.

38. What is PagedAttention?

It divides the KV cache into pages that a lookup table finds, so the cache can sit in fragmented memory. This reduces wasted memory, so more requests fit at once.

39. Calculation: how much does sliding window attention save?

Standard attention costs about N squared. Sliding window attention uses only the previous w tokens, so the cost is about N x w.

N = 128,000 tokens, w = 32,000
standard : N x N = 16.4 billion token pairs
window   : N x w =  4.1 billion token pairs
saving   : N / w = 4 times fewer

The saving grows with N. The cost is some quality, because tokens beyond the window are not directly visible. Training can reduce that loss.

40. What are Mamba and hybrid models?

Mamba is a selective state-space model with a recurrent state update that scales linearly with length. Hybrid models mix Mamba-style blocks with transformer blocks, for example NVIDIA Nemotron 3 Nano. They avoid full attention in most layers. Other attention variants are gated, linear, compressed, and multi-latent attention.

Models: image and video generation

Lesson: How Image and Video Models Generate.

41. Describe the image generation pipeline.

A text encoder turns the prompt into vectors. A denoising model refines a latent (a compressed image representation) from noise over 30 to 50 steps. A VAE (variational autoencoder) decodes the final latent to pixels. LoRAs and ControlNets extend the pipeline and ComfyUI builds such pipelines.

42. Why work in latent space?

Attention over every pixel is not feasible. A 1024 x 1024 image becomes a 128 x 128 latent, which is far fewer values, so attention is affordable. Image models use diffusion transformers over patches of 2 x 2 or 4 x 4 pixels.

43. Calculation: how many forward passes does an image take?

Each step has two forward passes, one with and one without the text conditioning (guidance).

50 steps x 2 passes = 100 forward passes

Time is linear in the number of steps, so 25 steps take about half as long. Few-step models use eight or fewer steps, are 80 to 90 percent faster, and have visibly lower quality. Distillation is more common than latent consistency for this.

44. How do video models differ from image models?

They have three to five times more parameters and ten to one hundred times more latent information. They keep the whole video in an X, Y, T (width, height, time) latent to avoid errors that build up frame by frame. Generation runs at batch size one on an eight-GPU node at about 50 steps, taking several seconds of inference per second of video.

45. Spot the mistake: "Image models are small, so serving them is cheap and memory-bound."

Image models are about a tenth the size of frontier LLMs, but their attention is equally demanding and the work is compute-bound, not bandwidth-bound. A large share of the cost is arithmetic on a big latent, repeated for every step.

Models: finding the bottleneck

Lesson: Finding the Bottleneck: Compute or Memory.

46. Define ops:byte ratio, arithmetic intensity, and the roofline.

The ops:byte ratio of a GPU is its compute (FLOPS) divided by its memory bandwidth (bytes per second). Arithmetic intensity of an algorithm is its total operations divided by its total memory traffic. On a roofline chart, an algorithm whose intensity is above the GPU ratio is compute-bound (flat ceiling), and below it is memory-bound (diagonal ceiling). Improving the resource that is not the limit gains nothing.

47. Calculation: what is the ops:byte ratio of an H100 in FP16?

compute   = 989 teraFLOPS
bandwidth = 3.35 TB/s
ratio     = 989 / 3.35 = about 295 operations per byte

An algorithm needs more than about 295 operations per byte moved to keep the Tensor cores busy.

48. Calculation: is standard attention compute-bound or memory-bound?

Take N = 4,096 tokens, d = 128, FP16.

bytes moved = 8N^2 + 8Nd
            = 8 x 16,777,216 + 8 x 4,096 x 128
            = 134,217,728 + 4,194,304 = 138,412,032

operations  = 4N^2 d + 3N^2
            = 4 x 16,777,216 x 128 + 3 x 16,777,216
            = 8,589,934,592 + 50,331,648 = 8,640,266,240

intensity   = 8,640,266,240 / 138,412,032 = about 62

62 is below 295, so standard attention is memory-bound on an H100. This is the reason FlashAttention, which removes most of the traffic, helps.

49. Why is prefill compute-bound and decode memory-bound?

Prefill processes many tokens at once, so each weight loaded from memory is used for a large matrix-matrix product. Decode produces one token per pass, so the weights are loaded again for every token and each is used for a cheap vector-matrix product. Prefill sets TTFT and decode sets TPS.

50. Calculation: what is the decode speed limit of a model on one GPU at batch size 1?

Each token requires reading all active weights once. Using an H100 at 3.35 TB/s and FP16 weights:

8B  model : 16 GB  / 3,350 GB/s = 4.8 ms per token  -> about 209 tokens/s upper bound
70B model : 140 GB / 3,350 GB/s = 41.8 ms per token -> about 24 tokens/s upper bound (needs more than one 80 GB GPU in practice, so this shows the shape of the limit)

These ignore KV cache reads and overhead, so real speeds are lower. The point is that decode speed follows bytes read, so a smaller or lower-precision model is faster.

51. How does batching change the decode bottleneck?

Many requests share one read of the weights, so the arithmetic per byte rises. At FP16, each weight is 2 bytes and gives 2 operations per token in the batch, so intensity is about the batch size. Roughly 295 requests in a batch would bring an H100 near compute-bound, ignoring the KV cache. This is why batching raises throughput, and why it raises per-user latency once compute is saturated.

Hardware: GPU architecture and generations

Lessons: Inside a GPU, GPU Generations.

52. Why does a GPU suit inference better than a CPU?

A GPU applies one simple operation to thousands of independent values at once, and a CPU is built for complex sequential execution. Inference is vector and matrix multiplication, which fits the GPU pattern.

53. Describe the compute units inside a GPU.

A GPU has many Streaming Multiprocessors (SMs). Each holds CUDA cores (scalar operations), Tensor cores (vector and matrix operations), and Special Function Units (sin, cos, log). Inference compute is quoted in Tensor core FLOPS, because Tensor cores run Matrix Multiply and Accumulate instructions, D = A x B + C.

54. A spec sheet lists dense and sparse FLOPS. Which one applies to inference, and why?

Dense. Sparse FLOPS need 2:4 structured sparsity, where half the values are zero, and the sparse number is often double the dense one. Inference runs dense by default. FLOPS also double with each halving of precision, so compare GPUs only at the same precision.

55. Describe the GPU memory hierarchy.

VRAM is DRAM, built as HBM3, HBM3e, or HBM4. On-chip SRAM forms the L0 (instruction cache per Tensor core), L1 (shared memory per SM), and L2 (global) caches. An H100 has 256 KB of L1 per SM and 50 MB of L2. On-chip memory is much faster and much smaller than VRAM.

56. How much VRAM does a deployment need? Follow-up: what happens with too little?

The weights plus at least 50 percent headroom for the KV cache and buffers. With too little, loading fails with an out-of-memory error, or inference runs slowly or crashes when the cache fills.

57. Decode the name "H200" and list the architectures in order.

The letter names the architecture and the number names the model. H100 and H200 are both Hopper, and a bigger number means a larger and more expensive GPU. Order: Volta (2017), Turing (2018), Ampere (2020), Hopper (2022), Lovelace (2022), Blackwell (2024), Rubin (2026), Feynman (2028). Most deployments use Lovelace, Hopper, or Blackwell.

58. Compare H100, H200, L4, L40, and B200 for a decode-heavy LLM.

Decode speed follows bandwidth, so H200 decodes faster than H100 with the same compute. Capacity decides which model fits.

59. What does Blackwell add, and what does Rubin add?

Blackwell adds FP4 and the MXFP8, MXFP4, and NVFP4 microscaling formats, and runs FlashAttention 4. Hopper introduced FP8, fourth-generation Tensor cores, thread block clusters, and distributed shared memory, and runs FlashAttention 3. Rubin uses HBM4 and adds the CPX chip for compute-bound prefill.

60. What are Grace and Vera, and why do they matter for inference?

They are NVIDIA ARM CPUs (Vera pairs with Rubin). NVLink C2C gives up to 900 GB/s of bi-directional bandwidth between CPU and GPU, several times PCIe, so offloaded LoRA weights and KV caches come back faster.

61. Spot the mistake: "We moved to the newest GPU architecture on release day and it should beat the old one."

Full software support takes about a year after a new architecture, so early kernels and engine builds may not use the hardware well. Judge a new architecture with real benchmarks of the actual workload.

Hardware: instances, accelerators, and local inference

Lessons: Instances, Multi-GPU, MIG, and Other Accelerators, Local Inference.

62. What is in an instance, and why check every part?

An instance is a virtual machine with GPUs (the device), CPUs (the host), host memory, storage, networking, and interconnect. Any part can be a bottleneck or a failure point. The same GPU can also differ between instances: the A100 comes as PCIe and SXM, and SXM has about five percent more memory bandwidth.

63. Compare NVLink, NVSwitch, InfiniBand, and Ethernet.

A node has eight GPUs. NVLink is about an order of magnitude faster than the links between nodes, so topology decides which traffic goes where.

64. What is the NVL72 GB200?

A rack joining 72 Blackwell GPUs and 36 Grace CPUs on NVLink. It lets a large model be split across many GPUs without leaving the fast interconnect. The Vera Rubin NVL 144 CPX is the next generation.

65. Explain MIG and when it pays off. Follow-up: why do some SMs sit idle?

MIG (Multi-Instance GPU) splits an A100, H100, H200, or B200 into up to seven slices. An H100 has seven compute slices and eight memory slices, so a 3-slice instance has about 3/7 of the compute and up to half of the 80 GB of memory. It suits small models, for example Orpheus TTS (3B parameters), better than a whole large GPU. Follow-up answer: an SXM H100 has 132 SMs, which do not divide evenly into seven slices, so some SMs are unused.

66. When would a team consider a non-NVIDIA accelerator?

When memory bandwidth, power efficiency, or platform integration is better for the workload. The challenges are software support, manufacturing capacity, and distribution.

67. List the strengths and weaknesses of local inference.

Strengths: zero network latency, independence from connection and server load, better privacy, and no GPU cost for the developer. Weaknesses: limited hardware, thermal limits, a fragmented support matrix, and battery drain. The median user has an older and weaker device than an enthusiast.

68. Calculation: can a 100B-parameter model run locally?

Compare the memory for a 100B model at 4 bits (0.5 byte per parameter) with a high-end desktop.

weights = 100 x 0.5 = 50 GB (plus cache and overhead)

Apple M3 Ultra : 512 GB at 819 GB/s  -> fits, bandwidth sets decode speed
RTX 5090       : 32 GB at 1,792 GB/s -> does not fit
dense decode bound on M3 Ultra: 50 GB / 819 GB/s = 61 ms per token = about 16 tokens/s

Capacity decides what loads and bandwidth decides decode speed. A MoE model reads only the active experts per token, so it decodes faster than the dense figure.

69. What tools exist for local inference?

Ollama and llama.cpp run aggressively quantized models on personal hardware. ComfyUI runs image pipelines and WebLLM runs models in the browser. Phones use Android's AI Edge SDK and ML Kit GenAI (Gemini Nano, Gemma) and iOS's Foundation Models and Core ML. Models above one or two billion parameters are hard on phones.

70. Design question: where should an app run speech and text features, on the device or in the cloud?

A hybrid design. Small models and quick queries run on the device (transcription and speech synthesis are latency-sensitive, and translation suits small fine-tuned models). Demanding workloads run in the cloud. The criteria are model size against device memory, latency, privacy, battery, and the age of the devices the users own.

Software: CUDA, frameworks, and engines

Lessons: CUDA, Kernels, and Kernel Fusion, PyTorch, Model Formats, ONNX, TensorRT, Transformers, Inference Engines.

71. Order the software stack from low to high abstraction.

CUDA, deep learning frameworks (PyTorch), inference engines (vLLM, SGLang, TensorRT-LLM), and NVIDIA Dynamo. Most inference engineering happens at the higher levels.

72. What is CUDA, and what is a kernel?

CUDA is a proprietary platform and programming model, not a language. Programs are usually C++ compiled by nvcc into CPU and GPU code. Its parts are the kernel, the graph (a DAG for repeated workflows), the driver (low-level), and the runtime (the developer API). A kernel is a user-defined function that runs in parallel on the GPU. It is unrelated to the Linux kernel.

73. Name the main GPU libraries and what each provides.

BLAS (from the 1970s, in Fortran) is a specification of linear algebra routines. cuBLAS is its CUDA implementation. cuDNN supplies neural-network primitives. CUTLASS and CuTe are template libraries for writing matrix kernels. FlashInfer holds LLM inference kernels. GEMM (general matrix multiply) is the most used BLAS operation in inference.

74. Why do kernels need rewriting for each GPU generation?

Kernels hard-code hardware details such as tile sizes and memory behavior, so a kernel tuned for an H100 may not suit a B200. This is why FlashAttention is tens of thousands of lines for the same math as attention, and why porting each generation takes substantial work.

75. What is kernel fusion, and where does it matter most?

Fusion merges several kernels so intermediate results stay on the chip instead of making a round trip through VRAM. It matters most in bandwidth-bound decode. Compilers find simple fusions, and FlashAttention needs handwritten fusion.

76. What does torch.compile do, and what are its limits?

It selects and fuses kernels for a specific GPU. It cannot fuse plugin kernels (DeepGEMM, FlashAttention, custom kernels), so it helps LLM inference less. It helps rare architectures and long sequences of lightweight kernels, and it may force code rewrites that avoid Python-specific features.

77. Compare safetensors and ONNX. Follow-up: where does TensorRT fit?

Safetensors stores only tensor data and cannot execute code. Weights are split across dozens of files, and memory mapping makes loading faster and safer. ONNX stores weights together with the execution graph and is portable. ONNX Runtime is open source and supports many GPUs. TensorRT is NVIDIA-only and compiles an optimized engine, and it stays popular for image and video models. Follow-up answer: export can fail on unsupported operations, for example MLA in DeepSeek V3.

78. Why not use transformers or diffusers for production serving?

They are reference implementations with config.json files and download helpers. Model-card examples are barely optimized and are not designed for large-scale production.

79. Design question: pick an engine for three situations.

All three support continuous batching, quantization, speculative decoding, prefix caching, parallelism, and disaggregation. Dynamo orchestrates them with KV cache reuse and routing, disaggregation, multi-node parallelism, and an SLA-based planner. Without enough volume, Dynamo is overhead.

80. Spot the mistake: "A new model was released, so we pull the latest stable build of everything."

Hardware changes every year or two and software changes in weeks. Day-zero support for a new model often needs nightly builds of every dependency. Stable releases usually come later.

Software: benchmarking and profiling

Lesson: Benchmarking, Load Testing, and Profiling.

81. What makes a benchmark representative?

It imitates production. Match the input and output sequence lengths (ISL and OSL), the traffic volume and pattern (with jitter), the request contents (cache hits and draft acceptance), and the parameters (temperature, reasoning effort). Shadowing copies live requests to the test system without affecting users.

82. Which tools generate load?

SGLang Genai-bench, NVIDIA GenAI-Perf, and Locust. Eval datasets such as MMLU, gsm8k, SWE-bench, and HumanEval give realistic inputs and a quality check.

83. Explain the method for benchmarking optimizations. Follow-up: why test combinations?

Start with a baseline, test each optimization alone and together, change one variable at a time, and use enough traffic, repeated runs, and a stable configuration. Follow-up answer: optimizations can conflict. Speculative decoding wastes its gain at large batch sizes because the spare compute is gone.

84. Benchmark or profiler: which answers what?

Benchmarks say how the system performs. Profilers say why. PyTorch Profiler shows CPU time, GPU time, and memory. Nsight Systems is system-wide and covers multi-GPU. Nsight Compute analyzes single CUDA kernels. TensorFlow and TensorRT have built-in profilers.

85. Who needs profiling, and who does not?

Engine users usually need only configuration and benchmarking. Framework contributors, authors of custom PyTorch services, and teams serving a new modality need profiling.

86. Spot the mistake: "The load test used one short prompt repeated, and it showed 3,000 TPS."

The prompt cached after the first request, so prefill was almost free, and the single short length does not match production sequence lengths. Use a realistic mix of input and output lengths and contents, or the figure overstates throughput.

Summary

• Inference engineering covers runtime, infrastructure, and tooling. Availability levels differ by a factor of ten per nine.
• Model size sets memory: parameters times bytes per parameter, plus at least 50 percent headroom for the cache.
• Report TTFT, per-user TPS, total TPS, and percentiles. Total time is TTFT plus (tokens minus one) times ITL.
• Prefill is compute-bound, decode is memory-bound, and the H100 FP16 ops:byte ratio is about 295.
• Attention is quadratic and sensitive. FlashAttention cuts memory traffic and sliding window attention cuts the pairs compared.
• MoE lowers active parameters per token but not memory in batched serving.
• GPU choice follows memory capacity for fit and bandwidth for decode, and compare FLOPS at the same precision.
• Interconnect topology decides where traffic can run. MIG fits small models. Local inference trades hardware limits for latency and privacy.
• Pick the engine by model, hardware, and effort budget. Benchmark with production-like traffic, and profile to find why.