Choosing a Model: Evaluation, Fine-Tuning, Distillation

Tuning a model server takes weeks of work, and the gains are often modest. Picking a model with far fewer parameters changes the cost before any tuning starts. That is why model selection comes first in this course: of all the decisions in performance work, the choice of model matters most.

This lesson explains why a smaller model is faster and cheaper, how to measure whether a model is good enough (evaluation), and two ways to get a small model that is good enough: fine-tuning and distillation. Afterwards you can decide how to shortlist models, build a small eval, judge when fine-tuning is realistic, and explain how distillation differs from fine-tuning.

Two terms first. A parameter is one learned number inside a model, and the model's size is quoted as a parameter count, such as 8B for eight billion. The parameters are also called the weights. Training sets them, and a finished model fixes them.

Example 1: smaller models are faster and cheaper

If everything else is equal (hardware, runtime, optimizations, architecture), inference on a model with fewer parameters is faster and cheaper than inference on a model with more parameters. The reason is physical. Every generated token needs the weights to be read from memory and used in arithmetic, so fewer weights mean less memory to hold, less data to move, and less arithmetic. The bottlenecks lesson works this out in detail.

The calculation below shows the memory side. A value stored in FP16 (16-bit floating point, a common number format) takes 2 bytes, so the weights of an N-parameter model take 2N bytes. The model sizes are examples: 8B and 70B are Llama sizes mentioned later in this course, and 671B is DeepSeek-R1's size.

weight memory (FP16) = parameters x 2 bytes

3B   : 3 x 10^9   x 2 =     6 GB   ->  6 / 80    = 0.08 GPUs of 80 GB
8B   : 8 x 10^9   x 2 =    16 GB   -> 16 / 80    = 0.20
70B  : 70 x 10^9  x 2 =   140 GB   -> 140 / 80   = 1.75
671B : 671 x 10^9 x 2 = 1,342 GB   -> 1342 / 80  = 16.78

A 3B model fits on one GPU with room to spare. An 8B model needs about 16 GB. A 70B model needs 140 GB, so at least two 80 GB GPUs for the weights alone. A 671B model needs 1,342 GB, which is more than 16 such GPUs. These numbers ignore the memory that the model needs while running, so the real need is higher. They also assume FP16, and the quantization lesson shows how lower precision shrinks them. The point for model selection is that every step down in size removes whole machines from the bill.

The advice therefore changes with the stage of a product:

• Early stage: use prebuilt pay-per-token APIs for strong frontier models such as Kimi and DeepSeek, or even closed models such as GPT and Gemini. Before product-market fit, doing your own inference is not worth the time or money.
• At scale: the opposite advice applies. Find or create the smallest, easiest-to-run model that is smart enough for the task. Sometimes that is still a trillion-parameter frontier model. It is always worth checking whether a smaller, cheaper, faster model can do the job.

The model also decides which optimizations exist. Inference engines (the programs that serve models) differ in how well they support each model architecture. Choosing popular architectures leads to solid support across the performance tooling landscape.

Example 2: model evaluation

Model evaluation, or evals, is the practice of systematically measuring a model's intelligence. Strong evals are a prerequisite for inference engineering, for two reasons:

• Spend time wisely: before investing in making a model fast, evals confirm that the model is useful.
• Establish a baseline: some performance techniques can reduce model quality, so there must be a baseline to compare against.

Evals are different from standard intelligence benchmarks. Benchmarks such as MMLU or SWE-bench measure model capabilities on common tasks. Evals are tailored to a specific product, domain, and task.

Benchmarks help to shortlist models, but they have become saturated (many models score near the top) or even gamed. Goodhart's Law says: "when a measure becomes a target, it ceases to be a good measure." It applies here because frontier labs have a strong incentive to show a new record on benchmarks with every release. A better way to compare general intelligence is an Elo rating, which ranks models by their win rate in head-to-head comparisons against other models. Even so, nothing replaces measuring how a model performs for your own application.

Three tips for useful eval work:

The reference lesson at the end of the course (further reading) lists tools and reading about evals.

Here is a very small eval, to show the mechanics. It uses ten test cases for a text-to-SQL product and the answers of two systems: the baseline model and an optimized version of it. The method is to compute accuracy for each and list the cases that changed: a baseline number, a candidate number, and a look at every difference.

10 text-to-SQL cases. Baseline and optimized each answer all 10.
Baseline correct: 9 of 10 (fails "distinct cities": forgot DISTINCT)
Optimized correct: 9 of 10 (fails "emails of admins": dropped WHERE admin)

baseline accuracy  = 9 / 10 = 90 percent
optimized accuracy = 9 / 10 = 90 percent
changed cases:
  'emails of admins'  baseline pass, optimized fail
  'distinct cities'   baseline fail, optimized pass

Both systems score 9 of 10 on this set, so the headline accuracy is identical. The two changed cases show that they fail on different cases. The aggregate hides a loss on "emails of admins" and a gain on "distinct cities". This is the reason for the tip "look at your data": an optimization that keeps the average while breaking a case the product cares about is not safe. It is also the reason to focus the eval on the hardest problems the model must solve. Real evals use far more than ten cases, and an optimization is accepted only when its eval result matches the baseline within the quality loss the product can tolerate.

Example 3: fine-tuning for domain-specific quality

Fine-tuning takes a pre-trained foundation model (a large general model) and adapts it to a specific use by training it further on new data. Fine-tuning changes the values of the weights and keeps the overall architecture (the layout of the network) the same.

flowchart LR
    A["Pre-trained model: same layers and sizes, general weights"] -->|"train on domain data"| B["Fine-tuned model: same layers and sizes, adapted weights"]

The link to inference is direct. If a small model can be fine-tuned until it passes the evals, hitting latency and cost targets becomes much easier.

A good example is translating English into SQL, the language used to query databases. General-purpose coding models write SQL well, but they have hundreds of billions of parameters. SQL is a relatively constrained language. For an application that only turns natural-language prompts into SQL queries, a tiny fine-tuned model of only a few billion parameters can reach equal performance on that one task. Using the sizes from Example 1, a few billion parameters (say 3B) is about 6 GB in FP16, while a model with hundreds of billions of parameters needs hundreds of gigabytes, so the saving is a factor of 50 to 100.

Text-to-SQL is an extreme case, and many domains will not allow such a large reduction. It shows what is possible with three ingredients:

• a cleanly scoped domain,
• a strong set of evaluation criteria,
• high-quality labeled data (examples with the correct answers attached) for the fine-tuning.

Example 4: distillation

What if a model could keep most of a large model's intelligence at a fraction of the size? That is the idea of distillation: use a large "teacher" model to train a smaller "student" model so that it imitates the teacher's behavior.

flowchart LR
    T["Teacher: large model"] -->|"probability distributions"| S["Student: small model, trained to match"]

Distillation is different from fine-tuning on synthetic data. Fine-tuning on synthetic data trains the model on input-output pairs, which are only the final answers. Distillation shows the student the teacher's actual probability distributions. When a model picks the next token it computes a probability for every token in its vocabulary, and those probabilities carry more information than the single chosen answer.

Fine-tuning teaches a model to do better in one domain. Distillation teaches a model how to imitate a larger model's behavior, both the good and the bad.

The calculation below shows the extra information on one prediction. The teacher gives probabilities over four candidate next tokens. A training signal from the final answer alone says "token 0 is right". A signal from the full distribution says "token 0 is right, token 1 is a plausible second choice, tokens 2 and 3 are unlikely". The loss measures how far the student's distribution is from the target. A smaller loss means a better match. With a one-hot target (all weight on token 0), the loss is the cross-entropy -ln(student probability of token 0). With the teacher's distribution as target, it is the sum over tokens of -teacherProb x ln(studentProb).

Cross-entropy loss = sum over tokens of ( -target x ln(student) )

teacher (soft target): [0.70, 0.20, 0.07, 0.03]
hard target (answer only): [1, 0, 0, 0]
student A: [0.60, 0.25, 0.10, 0.05]   student B: [0.60, 0.05, 0.05, 0.30]

Hard-label loss:
  A: -ln(0.60) = 0.511      B: -ln(0.60) = 0.511

Teacher-distribution loss:
  A: 0.70 x 0.511 + 0.20 x 1.386 + 0.07 x 2.303 + 0.03 x 2.996 = 0.886
  B: 0.70 x 0.511 + 0.20 x 2.996 + 0.07 x 2.996 + 0.03 x 1.204 = 1.203

Both students give token 0 the same probability, 0.60, so the hard-label loss is identical (0.511). The hard label cannot tell them apart. The teacher-distribution loss can: student A, which also ranks token 1 second as the teacher does, scores lower than student B, which gives token 3 a large 0.30 that the teacher thinks is unlikely. That is how a student absorbs the teacher's behavior and not only its answers. Neither value is near zero, because this loss includes the teacher's own uncertainty, so only the difference between students is meaningful. The student can also copy a teacher's mistakes, which is the meaning of "good and bad" above.

Distillation is used much less in the real world than fine-tuning. Two facts explain how it appears in practice:

• When a frontier lab releases a family of models in several sizes, the smaller ones are generally not distilled from the largest. They are trained independently, so that the biases of the large model do not artificially limit the smaller ones. If the lab trains only a large model, distillation can make it more accessible.
• In January 2025 the open model lab DeepSeek released its flagship reasoning model at the time, DeepSeek-R1, with 671B parameters. Because it was so large, the lab also released distilled versions built on the most popular open architectures of the time, Llama 3 and Qwen 2.5.

Those distilled models showed reasoning behavior similar to the full DeepSeek-R1, though with worse scores on intelligence benchmarks. They were small, and they could use existing performance work for the Llama and Qwen architectures. This links back to Example 1: popular architectures come with mature tooling. At the time of writing, the DeepSeek-R1 distills were still among the most popular distilled models on Hugging Face, together with distills of models such as Whisper for audio transcription and some image generation models. The image generation lessons cover image distillation.

Example 5: a decision path for choosing a model

The three techniques fit one sequence of questions. The steps below combine them.

flowchart TD
    A["Define evals for the product"] --> B["Shortlist models with public benchmarks"]
    B --> C["Run your own evals on the shortlist"]
    C --> D{"Smallest model that passes?"}
    D -->|"yes"| G["Optimize and deploy it"]
    D -->|"no, but a small model is close"| E["Fine-tune a small model on labeled domain data"]
    E --> C
    D -->|"only a large model is available"| F["Consider distilling it into a smaller student"]
    F --> C

Each arrow back to the evals is deliberate. Every change to the model, whether a different size, a fine-tune, or a distill, is judged by the same evals, because the evals are the baseline for the whole project.

Interview points

• The most important performance decision is which model to run. A smaller model is faster and cheaper when everything else is equal.
• Early on, use pay-per-token APIs for frontier models. At scale, find or create the smallest model that is smart enough.
• Popular architectures get the deepest support in inference engines and performance tooling.
• Evals measure model intelligence for a specific product. Benchmarks such as MMLU and SWE-bench measure common tasks and are good for shortlisting, but they are saturated or gamed (Goodhart's Law). Elo rating from head-to-head wins is a better general measure.
• Evals ensure the model is worth speeding up and give a baseline for techniques that may reduce quality.
• Eval tips: look at your data, be precise about the hardest problems, use existing tools.
• Fine-tuning adapts a pre-trained model with new data. It changes weights and keeps the architecture. Text-to-SQL shows a model of a few billion parameters matching a model of hundreds of billions on one narrow task.
• Fine-tuning needs a cleanly scoped domain, strong eval criteria, and high-quality labeled data.
• Distillation trains a student on the teacher's probability distributions, not only its answers. It copies behavior, good and bad, and sees less real-world use than fine-tuning.
• Labs usually train smaller family members independently. DeepSeek-R1 (671B) was released with distills on Llama 3 and Qwen 2.5.

Summary

• Model choice outweighs runtime and algorithm choices because parameter count sets memory, data movement, and arithmetic.
• At weights of 2 bytes per parameter (FP16), an 8B model needs about 16 GB and a 70B model about 140 GB.
• Evals tailored to a product give the baseline for all later optimization. Check results by looking at individual cases, not only averages.
• Fine-tuning shrinks the model needed for a scoped domain such as text-to-SQL.
• Distillation shrinks a model by matching the teacher's probability distributions. It is rarer than fine-tuning.
• DeepSeek-R1 distills on Llama 3 and Qwen 2.5 stayed popular on Hugging Face, along with distills of Whisper and some image generation models.