How Models Are Trained, and the Variants You Download

Open a model repository and you see many files: a base version and an instruct version, a thinking version, a mixture-of-experts model, small adapter files, and quantized files with long names such as Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf. These are not random labels. Each part of a name records a training stage the model went through, or a design choice in its architecture (the way its layers are built).

This lesson walks through the stages in the order a model goes through them. Each idea that can be computed gets a small Java program, so the numbers are visible. The lesson ends with a method for reading file names and a checklist for choosing a model for local coding work. After it, you can look at any model page and say what kind of model it is, how it was made, and whether it fits your machine and your agent.

Two earlier lessons are used here. Next-token prediction explained loss (a number that says how wrong the model's guesses are) and gradient descent (the method that changes the weights to make the loss smaller). The transformer block explained how to count a model's parameters (the numbers that are learned, also called weights).

The table is a map of the whole lesson. Read it once now and come back to it at the end.

Example 1: the pretraining objective, loss falling over gradient steps

Pretraining is the first and by far the largest stage. The model reads a huge amount of text and, at every position, tries to guess the next token. At the start its guesses are close to random. Each time it is wrong, gradient descent nudges the weights so that the correct next token becomes a little more likely. After many such nudges the model has learned the patterns of the text.

The number that tracks progress is the loss. For one position the loss is -ln p, where p is the probability the model gave to the token that actually came next. The unit is the nat (the unit you get when you use the natural logarithm). If the model gives the right token probability 0.5, the loss is 0.693. If it gives 1.0, the loss is 0. If it gives 0.01, the loss is 4.6. The training loss is the average of this value over all positions. Perplexity is e raised to the loss. It can be read as "the model is about as unsure as if it were choosing between this many equally likely tokens".

The program trains a toy model on the small corpus from lesson 5. The task is to predict the next character from the previous character. The corpus has 15 different characters. The model is built with a deliberate bottleneck. Each character is first turned into a vector (a short list of numbers, called an embedding) of only 3 numbers. A 3 by 15 matrix then turns that vector into 15 logits, one raw score per possible next character. Softmax turns the logits into probabilities that add up to 1. Each step of full-batch gradient descent uses all positions at once and updates both the embeddings and the matrix, using the gradient p - onehot(target) at the logits. This gradient has a simple meaning: it pushes the probability of the correct character up and the probabilities of all other characters down, in proportion to how much probability they hold. The program prints the loss at five points in time.

import java.util.Random;

public class Pretraining {
    static final String CORPUS = "the cat sat on the mat. the dog sat on the log. "
        + "the cat saw the dog. the dog saw the cat. a cat and a dog sat on a mat.";

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String alphabet = CORPUS.chars().distinct().sorted().collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString();
        int n = alphabet.length(), dim = 3, positions = CORPUS.length() - 1;
        int[] ctx = new int[positions], target = new int[positions];
        for (int i = 0; i < positions; i++) { ctx[i] = alphabet.indexOf(CORPUS.charAt(i)); target[i] = alphabet.indexOf(CORPUS.charAt(i + 1)); }

        Random r = new Random(1);
        double[][] embed = new double[n][dim], out = new double[dim][n];      // logits = out^T * embed[context]
        for (double[] row : embed) for (int k = 0; k < dim; k++) row[k] = r.nextGaussian() * 0.1;
        for (double[] row : out) for (int j = 0; j < n; j++) row[j] = r.nextGaussian() * 0.1;

        double lr = 0.5;
        for (int step = 0; step <= 1000; step++) {
            double loss = 0;
            double[][] gEmbed = new double[n][dim], gOut = new double[dim][n];
            for (int i = 0; i < positions; i++) {
                double[] p = new double[n];
                double max = -1e9, sum = 0;
                for (int j = 0; j < n; j++) { for (int k = 0; k < dim; k++) p[j] += embed[ctx[i]][k] * out[k][j]; max = Math.max(max, p[j]); }
                for (int j = 0; j < n; j++) { p[j] = Math.exp(p[j] - max); sum += p[j]; }
                for (int j = 0; j < n; j++) p[j] /= sum;
                loss -= Math.log(p[target[i]]) / positions;
                for (int j = 0; j < n; j++) {
                    double d = (p[j] - (j == target[i] ? 1 : 0)) / positions;   // gradient of the loss with respect to logit j
                    for (int k = 0; k < dim; k++) { gOut[k][j] += d * embed[ctx[i]][k]; gEmbed[ctx[i]][k] += d * out[k][j]; }
                }
            }
            if (step == 0 || step == 10 || step == 50 || step == 200 || step == 1000)
                System.out.printf("step %4d: loss %.3f nats, perplexity %.2f%n", step, loss, Math.exp(loss));
            for (int c = 0; c < n; c++) for (int k = 0; k < dim; k++) embed[c][k] -= lr * gEmbed[c][k];
            for (int k = 0; k < dim; k++) for (int j = 0; j < n; j++) out[k][j] -= lr * gOut[k][j];
        }
        System.out.printf("loss of guessing uniformly: %.3f nats (perplexity %d)%n", Math.log(n), n);
    }
}

step    0: loss 2.701 nats, perplexity 14.89
step   10: loss 2.683 nats, perplexity 14.64
step   50: loss 2.207 nats, perplexity 9.09
step  200: loss 1.164 nats, perplexity 3.20
step 1000: loss 0.924 nats, perplexity 2.52
loss of guessing uniformly: 2.708 nats (perplexity 15)

Start with the last line. A model that guesses uniformly over 15 characters gives each one a probability of 1/15, so its loss is ln 15 = 2.708 nats and its perplexity is 15. The model starts with small random weights (the seed makes the run repeatable), so it begins at 2.701, which is almost the same as pure guessing.

The loss then falls in three phases. After 10 steps it is still 2.683, because the weights start near zero and the gradients are small, so progress is slow at first. After 200 steps it is 1.164, which is the fast phase. After 1000 steps it flattens at 0.924, which is a perplexity of 2.52: the model now behaves as if it were choosing between about 2.5 characters.

Why does it stop at 0.924? Lesson 5 showed that a table with a free row for each character reaches a perplexity of 2.458. That table can store anything. This toy model must squeeze each character through only 3 numbers, so it cannot store everything and ends above the lowest possible loss. This is the usual situation for real models too. A model of limited size ends above the lowest possible loss, and a larger model gets closer.

A real run has the same shape with a different scale. The text is trillions of tokens, the model has billions of parameters, each step uses a batch of many sequences instead of the whole corpus, and the loss is averaged over every position of every sequence.

Example 2: the training compute estimate

Training a model is expensive. Before a lab starts, it estimates how much arithmetic the run needs. This example shows the standard estimate and what it implies for model sizes.

The unit of arithmetic is the FLOP (floating-point operation, one addition or one multiplication on decimal numbers). The standard estimate for training is 6 * parameters * tokens FLOPs. The reason for the 6 is as follows. For each token, the forward pass (computing the prediction) costs about 2 FLOPs per parameter, one multiplication and one addition. The backward pass (computing the gradients) costs about twice the forward pass, which is 4 FLOPs per parameter. Lesson 8 counted the forward cost of 2 per parameter. Adding them gives 6.

A second number matters: how many tokens to train on. The Chinchilla study (Hoffmann et al., 2022) found that, for a fixed compute budget, the best final loss comes from training on about 20 tokens per parameter. Meta reports that Llama 3 was pretrained on over 15 trillion tokens. The program applies the estimate to three runs. The speed of an accelerator (a GPU or similar chip) is an assumption: 400 teraFLOP/s sustained, which is 4e14 FLOPs per second.

public class TrainingCompute {
    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        // FLOPs for training are about 6 * parameters * tokens: 2 for the forward pass, 4 for the backward pass
        double[][] runs = {{1.0e9, 20e9}, {8.03e9, 160e9}, {8.03e9, 15e12}};     // {parameters, tokens}
        String[] names = {"1B model, 20 tokens per parameter", "8B model, 20 tokens per parameter", "8B model, 15 trillion tokens"};
        double effective = 4e14;                               // assumed sustained FLOP/s of one accelerator
        for (int i = 0; i < runs.length; i++) {
            double flops = 6 * runs[i][0] * runs[i][1];
            double gpuHours = flops / effective / 3600;
            System.out.printf("%-36s %.2e FLOPs, %.2e accelerator-hours at %.0e FLOP/s, %.1f tokens per parameter%n",
                names[i], flops, gpuHours, effective, runs[i][1] / runs[i][0]);
        }
        // reading the same model: a forward pass costs about 2 * parameters FLOPs per token
        System.out.printf("one generated token of the 8B model: %.2e FLOPs%n", 2 * 8.03e9);
    }
}

1B model, 20 tokens per parameter    1.20e+20 FLOPs, 8.33e+01 accelerator-hours at 4e+14 FLOP/s, 20.0 tokens per parameter
8B model, 20 tokens per parameter    7.71e+21 FLOPs, 5.35e+03 accelerator-hours at 4e+14 FLOP/s, 19.9 tokens per parameter
8B model, 15 trillion tokens         7.23e+23 FLOPs, 5.02e+05 accelerator-hours at 4e+14 FLOP/s, 1868.0 tokens per parameter
one generated token of the 8B model: 1.61e+10 FLOPs

Check the first row by hand. The 1B model has 1.0e9 parameters and trains on 20e9 tokens. The estimate is 6 * 1.0e9 * 20e9 = 1.2e20 FLOPs. Dividing by 4e14 FLOPs per second gives 3e5 seconds, and dividing by 3600 seconds per hour gives about 83 accelerator-hours.

The second row, an 8B model at 20 tokens per parameter, needs 7.7e21 FLOPs, about 5,350 accelerator-hours. The third row is the 8B model trained on 15 trillion tokens. It needs 7.2e23 FLOPs, about 500,000 accelerator-hours. That is 94 times the compute of the second row, because it uses 1,868 tokens per parameter instead of 20.

If 20 tokens per parameter is the best use of a compute budget, why would anyone train far beyond it? Because the Chinchilla ratio answers a question about the cost of training only. Models meant for local use are trained on far more tokens than that ratio, for two reasons. Every extra training token still lowers the loss of a model of fixed size. And a smaller model is cheaper to run for every user afterwards. The lab pays more once, and every user saves on every request.

The last line compares with inference (running the model). One generated token of the 8B model costs about 2 * 8.03e9 = 1.6e10 FLOPs. A training token costs about three times that, because of the backward pass.

Example 3: base and instruct models, the same prompt

The result of pretraining is a base model. A base model does one thing: it continues text. It has never been taught that a prompt is a request from a person. If its training data contained a web page with a list of programming exercises, then after a line like "Write a function that reverses a string." the most likely next text is another exercise.

An instruct model has been trained further to follow requests. The name varies: Instruct, Chat, or it (short for "instruction tuned"). It is trained with a chat template, a fixed text format that marks who is speaking (the next lesson, Chat Templates, Roles, and Special Tokens, covers templates). The block below shows the pattern with one prompt. It is an illustration written for this lesson. It is not the output of a specific model.

Prompt: Write a function that reverses a string.

Base model (continues the document):
Write a function that checks whether a string is a palindrome.
Write a function that counts the vowels in a string.

Instruct model (answers the request, ends its turn):
Here is a Java method that reverses a string: ...

The base model treats the prompt as the first line of a document, and it continues the document with more exercises. The instruct model sees the same request wrapped in a template (<|im_start|>user ... <|im_end|>) and answers it, because its fine-tuning data pairs requests with answers.

Model cards describe base models as intended for further fine-tuning or for completion tasks. They describe instruct models as intended for assistants. A coding agent sends requests and tool results as chat messages, so a chat or instruct model is the starting point. There is one common exception. Fill-in-the-middle code completion (the editor feature that fills a gap between code before and after the cursor) is a plain continuation task, so base models are used for it.

Example 4: supervised fine-tuning, with the loss masked to the answer

The first step that turns a base model into an assistant is supervised fine-tuning (SFT). The model trains on example conversations written or collected by people, in the same chat template it will see later. The training objective is still next-token prediction. One detail changes: the loss is computed only on the tokens the assistant should produce.

The reason is simple. In a conversation, the system text and the user message are input. At inference time they come from the user, and the model never has to write them. If the loss included them, the model would spend effort learning to predict what a user might type. So those positions are masked: they are left out of the loss, and therefore they add nothing to the gradient either.

The toy below uses a six-token conversation. Tokens 0, 1, and 2 are the user message. Tokens 3, 4, and 5 are the answer. The model is a table with one row of logits for each previous token, which is the simplest model that can learn "after token a, predict token b". It trains twice for 300 steps: once with the loss on every position, and once on the answer positions only. Then it prints the final loss of each target. In the output, U marks a user target and A an assistant target.

public class MaskedLoss {
    // tokens: 0 1 2 = user message, 3 4 5 = assistant answer; the model is a table of logits per previous token
    static double[] softmax(double[] l) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double v : l) max = Math.max(max, v);
        double[] p = new double[l.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(l[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        int[] tokens = {0, 1, 2, 3, 4, 5};
        boolean[] assistantTarget = {false, false, true, true, true};      // is the token to predict (tokens[i+1]) part of the answer?
        for (boolean masked : new boolean[] {false, true}) {
            double[][] w = new double[6][6];                                // all logits start at 0: uniform predictions
            int counted = 0;
            for (boolean a : assistantTarget) if (!masked || a) counted++;
            for (int step = 0; step < 300; step++) {
                double[][] grad = new double[6][6];
                for (int i = 0; i < 5; i++) {
                    if (masked && !assistantTarget[i]) continue;            // masked targets add nothing to loss or gradient
                    double[] p = softmax(w[tokens[i]]);
                    for (int j = 0; j < 6; j++) grad[tokens[i]][j] += (p[j] - (j == tokens[i + 1] ? 1 : 0)) / counted;
                }
                for (int a = 0; a < 6; a++) for (int b = 0; b < 6; b++) w[a][b] -= 1.0 * grad[a][b];
            }
            System.out.print(masked ? "loss on assistant tokens only:" : "loss on every token:        ");
            for (int i = 0; i < 5; i++) System.out.printf("  %s%d %.3f", assistantTarget[i] ? "A" : "U", tokens[i + 1], -Math.log(softmax(w[tokens[i]])[tokens[i + 1]]));
            System.out.println();
        }
    }
}

loss on every token:          U1 0.015  U2 0.015  A3 0.015  A4 0.015  A5 0.015
loss on assistant tokens only:  U1 1.792  U2 1.792  A3 0.009  A4 0.009  A5 0.009

In the first line, with no mask, all five targets end at a loss of 0.015. The model has learned to predict the user tokens as well as the answer.

In the second line, the answer targets (A3, A4, A5) reach 0.009. The two user targets (U1, U2) stay at 1.792. That number is ln 6, the loss of a uniform guess over 6 tokens. The model never saw a training signal for those positions, so it is still guessing.

This is the intended result. The model learned what to say next and gained nothing from learning what the user would write. The masked run also reaches a slightly lower answer loss (0.009 against 0.015). In the code, the gradient is divided by the number of counted positions, which is 3 in the masked run and 5 in the unmasked run, so each counted position gets a larger share of every step.

Fine-tuning libraries offer this as an "assistant-only loss". The chat template marks which tokens belong to the assistant, and the library masks the rest.

Example 5: preference tuning and the DPO loss

After SFT the model follows requests, but some of its answers are better than others. Preference tuning teaches it which kind to prefer. The training data has a prompt, a chosen answer (the one people preferred), and a rejected answer.

There are two main methods. RLHF (reinforcement learning from human feedback) is the older route. It first trains a separate reward model on the preference pairs. The reward model gives a score to any answer. Then reinforcement learning (learning by trial and error from scores) adjusts the language model to get higher scores. DPO (Direct Preference Optimization) skips the reward model. It uses a single loss computed from the language model's own probabilities.

The DPO loss is -ln sigmoid(beta * ((log p(chosen) - log p_ref(chosen)) - (log p(rejected) - log p_ref(rejected)))). The pieces are as follows.

• p is the model being trained, and p_ref is a frozen copy of the model from before this training step (the reference).
• log p(answer) is the log-probability of a whole answer: the sum of the log-probabilities of its tokens. Numbers like -42 are normal, because probabilities are below 1 and so their logs are negative.
• log p(chosen) - log p_ref(chosen) says how much the model has moved toward the chosen answer compared with the reference. The same difference for the rejected answer says how much it has moved toward the rejected one.
• The difference between those two movements is the margin. A large positive margin means the model has moved toward the chosen answer and away from the rejected one.
• sigmoid(x) = 1 / (1 + e^-x) squeezes any number into the range 0 to 1, and beta (here 0.1) controls how sharply the margin counts.

The program uses made-up sequence log-probabilities. It tries six policies against one fixed reference and prints the margin and the loss for each.

public class Dpo {
    static double sigmoid(double x) { return 1 / (1 + Math.exp(-x)); }

    // loss = -ln sigmoid(beta * ((logp_chosen - ref_chosen) - (logp_rejected - ref_rejected)))
    static double loss(double beta, double chosen, double refChosen, double rejected, double refRejected) {
        return -Math.log(sigmoid(beta * ((chosen - refChosen) - (rejected - refRejected))));
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        double beta = 0.1, refChosen = -42.0, refRejected = -40.0;       // sequence log-probabilities under the frozen reference model
        System.out.println("policy chosen  policy rejected  margin  loss");
        double[][] policies = {{-42.0, -40.0}, {-41.0, -41.0}, {-38.0, -43.0}, {-34.0, -48.0}, {-30.0, -55.0}, {-45.0, -38.0}};
        for (double[] p : policies) {
            double margin = (p[0] - refChosen) - (p[1] - refRejected);
            System.out.printf("%13.1f  %15.1f  %6.1f  %.4f%n", p[0], p[1], margin, loss(beta, p[0], refChosen, p[1], refRejected));
        }
        System.out.printf("loss at margin 0: %.4f = ln 2%n", Math.log(2));
    }
}

policy chosen  policy rejected  margin  loss
        -42.0            -40.0     0.0  0.6931
        -41.0            -41.0     2.0  0.5981
        -38.0            -43.0     7.0  0.4032
        -34.0            -48.0    16.0  0.1839
        -30.0            -55.0    27.0  0.0650
        -45.0            -38.0    -5.0  0.9741
loss at margin 0: 0.6931 = ln 2

Take the third row by hand. The policy gives -38 to the chosen answer and -43 to the rejected one. The reference gave -42 and -40. The chosen answer moved by -38 - (-42) = +4. The rejected answer moved by -43 - (-40) = -3. The margin is 4 - (-3) = 7. Then beta * margin = 0.7, sigmoid(0.7) = 0.668, and -ln 0.668 = 0.403, which matches the table.

The first row is the starting point. The policy equals the reference, so the margin is 0, sigmoid(0) = 0.5, and the loss is ln 2 = 0.693. As the chosen answer becomes more likely and the rejected answer less likely, the margin grows and the loss falls: 0.403 at margin 7, 0.184 at margin 16, and 0.065 at margin 27. The last row moves the wrong way. The rejected answer became more likely, the margin is -5, and the loss rises to 0.974. Training pushes the model away from rows like that one.

The reference terms matter. They make the loss measure change, not absolute probability. This keeps the model near its starting point, so it does not drift far from the SFT model while chasing preferences.

Example 6: reasoning models, rewards that a program can check

Reasoning models write a long chain of thought before the final answer. These are the thinking tokens, often wrapped in <think> tags. The chain of thought works like scratch paper: the model tries steps, checks them, and then writes the answer.

These models are trained with reinforcement learning on tasks where a program can check the answer. A math question has a known result. A coding task has unit tests that pass or fail. The reward is then 1 for a correct answer and 0 for a wrong one, with no human judge needed.

One method for this is GRPO (group relative policy optimization), used in the DeepSeek-R1 work. For each prompt, the model samples a group of answers. Each sample is then scored against the group with the advantage, (reward - group mean) / group standard deviation. The advantage says how much better or worse a sample was than the group's average, measured in standard deviations. Samples with a positive advantage are made more likely, and samples with a negative advantage are made less likely.

The program computes this for 8 samples of one prompt. It then shows the price of thinking at inference time. It assumes 150 answer tokens and a decode speed of 20 tokens per second (decode is the step-by-step generation of new tokens, covered in lesson 11), and it varies the number of thinking tokens.

public class ReasoningRewards {
    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        // 8 sampled completions for one prompt; the reward is verifiable: 1 if the final answer is correct, else 0
        int[] thinkingTokens = {1800, 2400, 900, 3100, 1500, 4200, 2000, 2600};
        double[] reward = {1, 0, 1, 1, 0, 0, 1, 1};
        double mean = 0, var = 0;
        for (double r : reward) mean += r / reward.length;
        for (double r : reward) var += (r - mean) * (r - mean) / reward.length;
        double std = Math.sqrt(var);
        System.out.printf("group mean reward %.3f, standard deviation %.3f%n", mean, std);
        for (int i = 0; i < reward.length; i++)       // advantage: how much better than the group's average this sample was
            System.out.printf("sample %d: %4d thinking tokens, reward %.0f, advantage %+.3f%n", i, thinkingTokens[i], reward[i], (reward[i] - mean) / std);

        // the price at inference time
        int answerTokens = 150;
        double decodeSpeed = 20;                         // assumed tokens per second on a local machine
        for (int thinking : new int[] {0, 1000, 4000}) {
            int total = thinking + answerTokens;
            System.out.printf("thinking %4d tokens: %4d tokens to generate, %5.1f s at %.0f tokens per second%n", thinking, total, total / decodeSpeed, decodeSpeed);
        }
    }
}

group mean reward 0.625, standard deviation 0.484
sample 0: 1800 thinking tokens, reward 1, advantage +0.775
sample 1: 2400 thinking tokens, reward 0, advantage -1.291
sample 2:  900 thinking tokens, reward 1, advantage +0.775
sample 3: 3100 thinking tokens, reward 1, advantage +0.775
sample 4: 1500 thinking tokens, reward 0, advantage -1.291
sample 5: 4200 thinking tokens, reward 0, advantage -1.291
sample 6: 2000 thinking tokens, reward 1, advantage +0.775
sample 7: 2600 thinking tokens, reward 1, advantage +0.775
thinking    0 tokens:  150 tokens to generate,   7.5 s at 20 tokens per second
thinking 1000 tokens: 1150 tokens to generate,  57.5 s at 20 tokens per second
thinking 4000 tokens: 4150 tokens to generate, 207.5 s at 20 tokens per second

Five of the eight samples are correct, so the mean reward is 5/8 = 0.625. The standard deviation is 0.484. Each correct sample gets an advantage of (1 - 0.625) / 0.484 = +0.775. Each wrong sample gets (0 - 0.625) / 0.484 = -1.291. The wrong samples have a larger number in absolute value because they are rarer in the group, so each one stands out more.

If all samples had the same reward, the standard deviation would be 0, and the group would give no signal at all. A prompt that the model always solves, or never solves, teaches nothing in this scheme.

The last three lines show the cost of thinking. An answer of 150 tokens takes 150 / 20 = 7.5 seconds. With 1,000 thinking tokens the model generates 1,150 tokens, which takes 57.5 seconds. With 4,000 thinking tokens it takes 207.5 seconds. Thinking tokens are generated one at a time like every other token, so time grows in proportion to their number.

The thinking tokens also occupy space in the KV cache (the memory that holds the keys and values of earlier tokens, covered in lesson 11). The chat template decides whether earlier thinking stays in the history (Chat Templates, Roles, and Special Tokens). Many reasoning models offer a switch, or a separate non-thinking variant, and the model card names it.

Example 7: distillation, KL divergence between teacher and student

Distillation trains a small model (the student) to copy a large model (the teacher). The student does not learn from a single correct token. It learns from the teacher's whole probability distribution over the next token.

Here is why that carries more information. Suppose the next token after "the cat sat on the" could be "mat" with probability 0.66, "log" with 0.24, and a few others with small probabilities. A normal training label says only "the answer is mat" (a one-hot label, which puts 1 on one token and 0 on all others). The teacher's distribution also says that "log" is a reasonable second choice, and that the other tokens are unlikely. This extra knowledge is what the student gains.

The training measure is the KL divergence (Kullback-Leibler divergence): KL(teacher || student) = sum p_t * ln(p_t / p_s). It adds up, over all tokens, how far the student's probability p_s is from the teacher's probability p_t. It is 0 only when the two distributions are equal, and it grows as they differ.

The program has a teacher with fixed logits over 5 tokens. A student with 5 logits that start at 0 is trained by gradient descent to match it. The program prints the KL at three moments, prints both distributions at the end, and shows what a one-hot label would do instead. The last line uses temperature: dividing the logits by a number T greater than 1 before softmax flattens the distribution.

Some "Distill" models in repositories, for example DeepSeek-R1-Distill-Qwen-7B, were made in a related way. They were fine-tuned on text generated by a larger model, which is another form of the same idea.

public class Distillation {
    static double[] softmax(double[] l, double t) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double v : l) max = Math.max(max, v / t);
        double[] p = new double[l.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(l[i] / t - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    static double kl(double[] p, double[] q) {            // KL(p || q) in nats
        double s = 0;
        for (int i = 0; i < p.length; i++) if (p[i] > 0) s += p[i] * Math.log(p[i] / q[i]);
        return s;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        double[] teacherLogits = {3.0, 2.0, 0.5, 0.0, -1.0};
        double[] teacher = softmax(teacherLogits, 1.0);
        double[] student = new double[5];                // student logits start at 0 and are trained to match the teacher
        for (int step = 0; step <= 200; step++) {
            double[] q = softmax(student, 1.0);
            if (step == 0 || step == 20 || step == 200) System.out.printf("step %3d: KL(teacher || student) = %.4f%n", step, kl(teacher, q));
            for (int i = 0; i < 5; i++) student[i] -= 0.5 * (q[i] - teacher[i]);     // gradient of the KL with respect to the student logits
        }
        double[] hard = new double[5];
        hard[0] = 1;                                     // a one-hot label keeps only the teacher's top choice
        System.out.printf("teacher distribution: %.3f %.3f %.3f %.3f %.3f%n", teacher[0], teacher[1], teacher[2], teacher[3], teacher[4]);
        double[] q = softmax(student, 1.0);
        System.out.printf("student after 200 steps: %.3f %.3f %.3f %.3f %.3f%n", q[0], q[1], q[2], q[3], q[4]);
        System.out.printf("a student trained on the hard label would be pushed toward: 1.000 0.000 0.000 0.000 0.000 (KL to the teacher is %.4f at best)%n", kl(teacher, new double[] {1 - 4e-9, 1e-9, 1e-9, 1e-9, 1e-9}));
        System.out.printf("teacher at temperature 2: %s%n", java.util.Arrays.toString(java.util.Arrays.stream(softmax(teacherLogits, 2.0)).map(v -> Math.round(v * 1000) / 1000.0).toArray()));
    }
}

step   0: KL(teacher || student) = 0.6678
step  20: KL(teacher || student) = 0.0352
step 200: KL(teacher || student) = 0.0003
teacher distribution: 0.659 0.242 0.054 0.033 0.012
student after 200 steps: 0.658 0.242 0.053 0.032 0.015
a student trained on the hard label would be pushed toward: 1.000 0.000 0.000 0.000 0.000 (KL to the teacher is 6.1306 at best)
teacher at temperature 2: [0.444, 0.269, 0.127, 0.099, 0.06]

The KL starts at 0.668 (the student guesses uniformly, the teacher does not). It falls to 0.035 after 20 steps and to 0.0003 after 200. The student's final distribution, 0.658, 0.242, 0.053, 0.032, 0.015, is very close to the teacher's 0.659, 0.242, 0.054, 0.033, 0.012.

Now compare with a one-hot label. A student trained on the hard label would be pushed toward 1.000 on the top token and 0 on all others. That throws away the teacher's opinion about the alternatives. If the student matched the one-hot label exactly, with a floor of 1e-9 on the other tokens, its KL to the teacher would be 6.13 nats, which is large. The teacher's soft distribution is the richer teaching signal.

The last line shows the teacher at temperature 2: 0.444, 0.269, 0.127, 0.099, 0.06. The top token falls from 0.659 to 0.444 and the tail rises. Distillation often uses a higher temperature to soften the targets, so the unlikely tokens carry more signal.

Example 8: mixture of experts, routing a token

A transformer layer has an MLP block (the part that holds most of the layer's parameters). A mixture-of-experts (MoE) layer replaces that single MLP with several parallel MLPs called experts, plus a small router. The motivation is to have a model with a large total capacity while spending the compute of a smaller one on each token.

The routing works in four steps for each token.

1. The router gives every expert a score for this token.
2. The top few experts (the top-k) are chosen.
3. A softmax over only the chosen scores gives each chosen expert a weight, and the weights add up to 1.
4. The chosen experts run, and their outputs are added together using those weights.

The program has 4 experts. Each expert is a 4 by 4 matrix, so the example stays small. It routes three tokens to the top 2 experts and prints the router scores, the chosen experts with their weights, and the combined output.

import java.util.Arrays;
import java.util.Random;

public class MoeRouter {
    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        int dim = 4, experts = 4, topK = 2;
        Random r = new Random(5);
        double[][] router = new double[experts][dim];                       // one row per expert
        double[][][] expert = new double[experts][dim][dim];                // each expert: a small linear map
        for (int e = 0; e < experts; e++) for (int i = 0; i < dim; i++) {
            router[e][i] = r.nextGaussian();
            for (int j = 0; j < dim; j++) expert[e][i][j] = r.nextGaussian() * 0.5;
        }
        for (int token = 0; token < 3; token++) {
            double[] x = new double[dim];
            for (int i = 0; i < dim; i++) x[i] = r.nextGaussian();
            double[] score = new double[experts];
            for (int e = 0; e < experts; e++) for (int i = 0; i < dim; i++) score[e] += router[e][i] * x[i];
            Integer[] order = {0, 1, 2, 3};
            Arrays.sort(order, (a, b) -> Double.compare(score[b], score[a]));
            double z = 0;
            double[] w = new double[topK];
            for (int k = 0; k < topK; k++) { w[k] = Math.exp(score[order[k]]); z += w[k]; }   // softmax over the chosen experts only
            double[] y = new double[dim];
            for (int k = 0; k < topK; k++) {
                int e = order[k];
                w[k] /= z;
                for (int i = 0; i < dim; i++) for (int j = 0; j < dim; j++) y[i] += w[k] * expert[e][i][j] * x[j];
            }
            System.out.printf("token %d: router scores %s -> experts %d (%.3f) and %d (%.3f), output %.3f %.3f %.3f %.3f%n", token,
                Arrays.toString(Arrays.stream(score).map(v -> Math.round(v * 100) / 100.0).toArray()), order[0], w[0], order[1], w[1], y[0], y[1], y[2], y[3]);
        }
    }
}

token 0: router scores [-2.31, -3.47, -0.07, 0.84] -> experts 3 (0.713) and 2 (0.287), output 0.003 -0.423 -0.312 0.626
token 1: router scores [3.96, 1.22, -1.48, 1.94] -> experts 0 (0.883) and 3 (0.117), output -3.038 0.790 0.640 -0.757
token 2: router scores [0.9, 0.46, -0.65, 1.27] -> experts 3 (0.592) and 0 (0.408), output -0.177 -0.770 -0.847 -0.339

Each line shows one token. Token 0 has scores [-2.31, -3.47, -0.07, 0.84]. The two highest are expert 3 (0.84) and expert 2 (-0.07), so those two run. Softmax over the two chosen scores gives weights 0.713 and 0.287. The output is 0.713 times expert 3's result plus 0.287 times expert 2's result.

Token 1 favors expert 0 with weight 0.883. Its score of 3.96 is 2.02 above the second-best chosen score of 1.94, and the softmax turns a large gap into a large weight. Different tokens use different experts: token 0 uses experts 3 and 2, token 1 uses 0 and 3, and token 2 uses 3 and 0.

Only the chosen experts compute anything. So the work per token depends on the number of active experts, not on the total number of experts. All experts must still sit in memory, because any token may need any of them.

Example 9: MoE accounting, total versus active parameters

An MoE model has two sizes, and they matter for different reasons. The total parameters are all the weights that must be stored. The active parameters are the ones used for a single token. Memory follows the total. Per-token speed follows the active count.

The program counts both from configuration values. It counts these parts:

• Attention matrices: query and output matrices of size hidden * heads * headDim, and key and value matrices of size hidden * kvHeads * headDim.
• The router, which has one row per expert.
• The expert matrices, three per expert (gate, up, and down, as in the SwiGLU design used by these models).
• Two embedding matrices, one for input tokens and one for output scores.
• The small norm vectors.

It applies the counts to a Mixtral 8x7B shape (8 experts, 2 active per token) and a Qwen3-30B-A3B shape (128 experts, 8 active).

public class MoeParams {
    record Moe(String name, long hidden, long layers, long heads, long kvHeads, long headDim, long vocab, long experts, long active, long expertSize) {
        long attention() { return hidden * heads * headDim * 2 + hidden * kvHeads * headDim * 2; }
        long router() { return hidden * experts; }
        long oneExpert() { return 3 * hidden * expertSize; }                          // gate, up, down
        long embeddings() { return 2 * vocab * hidden; }                              // untied embedding and output matrix
        long total() { return embeddings() + layers * (attention() + router() + experts * oneExpert() + 2 * hidden) + hidden; }
        long activePerToken() { return embeddings() + layers * (attention() + router() + active * oneExpert() + 2 * hidden) + hidden; }
    }

    public static void main(String[] args) {
        Moe[] models = {
            new Moe("Mixtral 8x7B shape", 4096, 32, 32, 8, 128, 32000, 8, 2, 14336),
            new Moe("Qwen3-30B-A3B shape", 2048, 48, 32, 4, 128, 151936, 128, 8, 768)};
        for (Moe m : models)
            System.out.printf("%-20s total %,15d (%.1fB), active per token %,15d (%.1fB), %.1f%% of the weights%n", m.name(), m.total(), m.total() / 1e9,
                m.activePerToken(), m.activePerToken() / 1e9, 100.0 * m.activePerToken() / m.total());
    }
}

Mixtral 8x7B shape   total  46,702,792,704 (46.7B), active per token  12,879,925,248 (12.9B), 27.6% of the weights
Qwen3-30B-A3B shape  total  30,532,110,336 (30.5B), active per token   3,353,020,416 (3.4B), 11.0% of the weights

Check the Mixtral line by hand. One expert has 3 * 4096 * 14336 = 176,160,768 parameters. Eight experts in each of 32 layers give about 45.1 billion parameters, which is most of the 46.7 billion total. Only 2 of the 8 experts are active, so the experts contribute about 11.3 billion per token. Adding the shared attention and embeddings gives 12.9 billion active, which is 27.6% of the weights.

The Mixtral shape has 46.7B parameters in total and 12.9B active per token, which are the figures Mistral gave for the model. The 8x7B in its name does not mean 56B, because attention and embeddings are shared between the experts and are not repeated eight times.

The Qwen3 shape has 30.5B in total and 3.35B active, printed rounded as 3.4B. The Qwen3-30B-A3B model card lists 30.5B total and 3.3B activated parameters, and the name says the same: A3B means about 3 billion active. Two small details explain the tiny difference. The count here includes both embedding matrices. It also leaves out two small per-layer norm vectors for queries and keys (128 values each) that Qwen3 adds, which change nothing at this precision.

The shape values (hidden size 2048, 48 layers, 32 query heads, 4 key/value heads, head dimension 128, 128 experts with 8 active, expert size 768, vocabulary 151,936) are those of the published config.json. The query width, 32 * 128 = 4096, is larger than the hidden size of 2048. That is why the program computes attention from the head dimension and not from hidden / heads.

What this means for running a model locally: memory follows the total, and decode speed follows the active parameters, because a decode step reads only the chosen experts (Generation, Prefill, Decode, and the KV Cache covered bytes per token). A 30B MoE model needs the memory of a 30B model, and a decode step reads about as many weight bytes as a 3B to 4B model. FLOPs, Bandwidth, and the Roofline Model develops this. A later lesson on tuning llama.cpp shows how to keep the expert weights in system memory.

Example 10: LoRA, a low-rank update

Fine-tuning every weight of a model is costly. During training, each parameter also needs a gradient and extra optimizer state, so the memory needed is a multiple of the model size. LoRA (low-rank adaptation) avoids this by freezing the original weight matrix W and training two much smaller matrices next to it.

The idea is that a useful change to a big matrix can often be written as the product of two thin matrices. Matrix A has r rows, and matrix B has r columns, where the rank r is small, for example 8. The changed layer computes W x + (alpha / r) * B (A x). The first part is the original layer. The second part is the adapter. The constant alpha sets how strong the adapter is, and dividing by r keeps the scale steady when you change the rank.

After training, the adapter can be merged into the weights once: W' = W + (alpha / r) * B A. The merged model gives the same output and runs with no extra cost.

The program does two checks. First it computes the output both ways, with the adapter on the side and with the merged matrix, on a small random example, and prints the largest difference. Second it counts the adapter's parameters for a 4096 by 4096 matrix at three ranks.

import java.util.Random;

public class Lora {
    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        int d = 6, r = 2;
        double alpha = 4, scale = alpha / r;                       // LoRA scales the update by alpha / r
        Random rng = new Random(2);
        double[][] w = new double[d][d], a = new double[r][d], b = new double[d][r];
        for (double[] row : w) for (int j = 0; j < d; j++) row[j] = rng.nextGaussian();
        for (double[] row : a) for (int j = 0; j < d; j++) row[j] = rng.nextGaussian() * 0.3;
        for (double[] row : b) for (int j = 0; j < r; j++) row[j] = rng.nextGaussian() * 0.3;
        double[] x = new double[d];
        for (int j = 0; j < d; j++) x[j] = rng.nextGaussian();

        double[] ax = new double[r], y1 = new double[d], y2 = new double[d];
        for (int k = 0; k < r; k++) for (int j = 0; j < d; j++) ax[k] += a[k][j] * x[j];
        for (int i = 0; i < d; i++) {                              // adapter on the side: W x + scale * B (A x)
            for (int j = 0; j < d; j++) y1[i] += w[i][j] * x[j];
            for (int k = 0; k < r; k++) y1[i] += scale * b[i][k] * ax[k];
        }
        double[][] merged = new double[d][d];                      // merged: W' = W + scale * B A, computed once
        for (int i = 0; i < d; i++) for (int j = 0; j < d; j++) {
            merged[i][j] = w[i][j];
            for (int k = 0; k < r; k++) merged[i][j] += scale * b[i][k] * a[k][j];
        }
        for (int i = 0; i < d; i++) for (int j = 0; j < d; j++) y2[i] += merged[i][j] * x[j];
        double diff = 0;
        for (int i = 0; i < d; i++) diff = Math.max(diff, Math.abs(y1[i] - y2[i]));
        System.out.println("largest difference between adapter and merged output: " + (diff < 1e-12 ? "below 1e-12" : diff));

        long dim = 4096;
        System.out.println("full matrix " + dim + " x " + dim + ": " + dim * dim + " parameters");
        for (int rank : new int[] {8, 16, 64})
            System.out.printf("rank %2d: %,d parameters in A and B (%.2f%% of the matrix)%n", rank, 2 * rank * dim, 100.0 * 2 * rank * dim / (dim * dim));
    }
}

largest difference between adapter and merged output: below 1e-12
full matrix 4096 x 4096: 16777216 parameters
rank  8: 65,536 parameters in A and B (0.39% of the matrix)
rank 16: 131,072 parameters in A and B (0.78% of the matrix)
rank 64: 524,288 parameters in A and B (3.13% of the matrix)

The first line shows that the adapter form and the merged form agree to within floating-point rounding error (the difference is below 1e-12). So merging loses nothing.

The other lines show the saving. The full 4096 by 4096 matrix has 16,777,216 parameters. At rank 8, A has 8 * 4096 and B has 4096 * 8, so training touches 2 * 8 * 4096 = 65,536 parameters, which is 0.39% of the matrix. At rank 16 it is 131,072 (0.78%), and at rank 64 it is 524,288 (3.13%). A higher rank can express a richer change and costs proportionally more.

Adapters are stored as small files and can be loaded on top of a base model, so one base model can serve several tasks. llama.cpp loads LoRA adapters in GGUF form (--lora, with --lora-scaled for a custom scale). Servers can switch adapters per request. Multi-LoRA serving is covered later in this course.

Example 11: decoding a model file name

A model file name packs several facts into one string. Names on model hubs follow loose conventions. A typical name has the model family and version, the parameter count, the variant, and the quantization. A large model may also be split into numbered shards (parts of one model stored as several files).

Take Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf. Read it from left to right: Qwen2.5 is the family and version, Coder says it is trained on code, 7B means 7 billion parameters, Instruct says it follows requests, and Q4_K_M names the number format of the weights.

The program parses names in this style with regular expressions (patterns that match text). The quantization names come from the types listed by llama-quantize --help in the pinned build: Q4_0, Q5_1, Q2_K, Q4_K_M, Q6_K, Q8_0, IQ4_XS, F16, BF16, F32, and others. The file names are examples written in the common style. They are not checked against a hub listing. The program prints one row for each of seven names.

import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class ModelNames {
    static final Pattern QUANT = Pattern.compile("(?i)(?:IQ\\d_(?:XXS|XS|S|M|NL)|TQ\\d_0|MXFP4(?:_MOE)?|Q\\d_K(?:_[SML])?|Q\\d_[01]|BF16|F16|F32)");
    static final Pattern SIZE = Pattern.compile("(?i)(?<![A-Za-z0-9.])(\\d+x)?(\\d+(?:\\.\\d+)?)([BM])(?![A-Za-z0-9])");
    static final Pattern ACTIVE = Pattern.compile("(?i)-A(\\d+(?:\\.\\d+)?)B");
    static final Pattern SHARD = Pattern.compile("-(\\d{5})-of-(\\d{5})");
    static final String[] VARIANTS = {"Instruct", "Base", "Chat", "Coder", "Thinking", "Reasoning", "Distill", "it"};

    static void describe(String file) {
        String name = file.replaceAll("(?i)\\.gguf$", "");
        Matcher shard = SHARD.matcher(name);
        String shards = shard.find() ? shard.group(1) + " of " + shard.group(2) : "single file";
        name = shard.replaceAll("");
        Matcher q = QUANT.matcher(name);
        String quant = q.find() ? q.group().toUpperCase() : "none";
        name = q.replaceAll("");
        Matcher a = ACTIVE.matcher(name);
        String active = a.find() ? a.group(1) + "B active" : "";
        Matcher s = SIZE.matcher(name);
        String size = s.find() ? (s.group(1) != null ? s.group(1) + s.group(2) : s.group(2)) + s.group(3).toUpperCase() : "?";
        StringBuilder variant = new StringBuilder();
        for (String v : VARIANTS) if (Pattern.compile("(?i)(^|[-_.])" + v + "($|[-_.])").matcher(name).find()) variant.append(v).append(' ');
        String family = s.reset().find() ? name.substring(0, s.start()).replaceAll("[-_.]+$", "") : name;   // everything before the size
        System.out.printf("%-42s name+version %-24s size %-5s %-9s variant %-16s quant %-7s %s%n", file, family, size, active, variant.toString().trim(), quant, shards);
    }

    public static void main(String[] args) {
        for (String f : new String[] {"Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf", "Llama-3.1-8B-Instruct-Q8_0.gguf", "Qwen3-30B-A3B-Q4_K_M.gguf",
                "Mixtral-8x7B-Instruct-v0.1.Q5_K_M.gguf", "DeepSeek-R1-Distill-Qwen-7B-IQ4_XS.gguf", "gemma-2-9b-it-BF16-00001-of-00002.gguf", "SmolLM2-135M-Instruct-F16.gguf"})
            describe(f);
    }
}

Qwen2.5-Coder-7B-Instruct-Q4_K_M.gguf      name+version Qwen2.5-Coder            size 7B              variant Instruct Coder   quant Q4_K_M  single file
Llama-3.1-8B-Instruct-Q8_0.gguf            name+version Llama-3.1                size 8B              variant Instruct         quant Q8_0    single file
Qwen3-30B-A3B-Q4_K_M.gguf                  name+version Qwen3                    size 30B   3B active variant                  quant Q4_K_M  single file
Mixtral-8x7B-Instruct-v0.1.Q5_K_M.gguf     name+version Mixtral                  size 8x7B            variant Instruct         quant Q5_K_M  single file
DeepSeek-R1-Distill-Qwen-7B-IQ4_XS.gguf    name+version DeepSeek-R1-Distill-Qwen size 7B              variant Distill          quant IQ4_XS  single file
gemma-2-9b-it-BF16-00001-of-00002.gguf     name+version gemma-2                  size 9B              variant it               quant BF16    00001 of 00002
SmolLM2-135M-Instruct-F16.gguf             name+version SmolLM2                  size 135M            variant Instruct         quant F16     single file

Each row gives these parts, in order.

• The part before the size: the family and version, such as Qwen2.5-Coder or Llama-3.1.
• The size: 7B, 8x7B for an MoE model with 8 experts of 7B each, or 135M for 135 million parameters.
• The active-parameter tag where there is one: A3B in the Qwen3 row.
• The variants found in the name.
• The quantization.
• The shard position: 00001 of 00002 means the first of two files.

Some notes on the rows. Q4_K_M means 4-bit k-quant with the medium mix of tensor types, which k-quants, i-quants, and the Importance Matrix covers. A name like 00001-of-00002 marks a file split into shards that llama.cpp loads together when the first one is given. The first row lists two variants, Instruct and Coder, because both words appear in the name.

The variants carry meaning. Instruct or it marks chat models. Coder marks code-focused training. Distill marks a model trained on a larger model's output. Thinking or Reasoning marks models that write a chain of thought.

The parser has limits, and they are the point to remember. The name is only a hint. The model card and the GGUF metadata (general.architecture, parameter counts, chat template) are the source of truth.

Example 12: the model card checklist, licenses, and criteria for a local coding model

A model card is the README of a model repository. It takes a few minutes to read, and it answers most questions that would otherwise show up later as failures. The table lists what to check, where to find it, and why it matters.

The license row needs more detail. Licenses fall into three groups.

• Permissive licenses such as Apache 2.0 and MIT allow commercial use and modification with few conditions.
• Custom community or model licenses (the Llama licenses are an example) allow broad use but add terms, such as an acceptable-use policy or conditions for very large deployments.
• Licenses that restrict use to research or non-commercial work exclude production use.

The license text, not a summary of it, decides what is allowed.

For a local coding agent, the criteria combine into one decision.

• Memory: the weights plus the KV cache for the intended context must fit the machine's memory (Memory Math).
• Speed: decode speed follows the active parameters and the memory bandwidth. An agent makes many sequential calls, so the speed of each call is multiplied.
• Tools: the chat template must support tool calls, and the model must produce them reliably.
• Context: the context length must hold the files the agent reads.
• Variant: a coder-tuned variant fits code tasks, and a general instruct variant fits mixed tasks. A thinking variant costs more tokens per step (Example 6) and fits hard problems.
• License: it must allow the intended use.

No single ranking follows from these criteria, since each machine and task weighs them differently. A later lesson on evaluating models and agents covers testing candidates on the actual tasks.

Interview points

• Pretraining minimizes next-token cross-entropy (the loss -ln p of the correct token, averaged over positions) on a large corpus. The result is a base model that continues text.
• Training compute is about 6 * parameters * tokens, and inference is about 2 * parameters per token. Compute-optimal training uses about 20 tokens per parameter, and models meant for deployment are often trained on far more, because a smaller model is cheaper to serve.
• Instruct models are fine-tuned on conversations in a chat template. A base model given a request tends to continue the text and not answer it.
• SFT masks the loss to the assistant's tokens. Masked positions add nothing to the loss or the gradient, so the model learns to write answers and not user messages.
• DPO's loss is -ln sigmoid(beta * margin), where the margin compares chosen and rejected answers against a frozen reference model. It needs no separate reward model, and it equals ln 2 when the model has not moved from the reference.
• Reasoning models are trained with reinforcement learning on verifiable rewards, such as unit tests. GRPO normalizes rewards inside a group of samples. The price is many thinking tokens per answer, which cost time and KV cache.
• Distillation trains on the teacher's whole distribution (KL divergence), so it passes on more information per token than a one-hot label.
• MoE stores many experts and runs a few per token. Total parameters set memory, and active parameters set per-token compute and bytes read.
• Mixtral 8x7B has 46.7B total and 12.9B active parameters, because attention and embeddings are shared between the experts.
• LoRA trains B A with rank r next to the frozen W, and merging gives W + (alpha / r) B A with the same output.
• A file name suggests family, size, variant, and quantization, and the model card and GGUF metadata are the source of truth.
• Before choosing a model, check architecture support, context length, chat template and tool support, quantized files, and license.

Summary

• A model is made in stages: pretraining, supervised fine-tuning, preference tuning, and reasoning training. Distillation and quantization are additional routes to smaller or cheaper models.
• Pretraining loss starts near ln(vocabulary size) and falls with gradient descent. The floor depends on model capacity.
• Training FLOPs are about 6 * N * D, where N is the parameter count and D is the number of tokens. The Chinchilla ratio is about 20 tokens per parameter.
• Base models continue text, and instruct models answer requests.
• SFT computes loss only on assistant tokens, and the gradient of masked tokens is zero.
• DPO loss is ln 2 at zero margin and falls as the chosen answer gains over the rejected one relative to the reference.
• Reinforcement learning with verifiable rewards needs no human judge. The GRPO advantage is (reward - mean) / standard deviation within a sampled group.
• Thinking tokens multiply generation time and use KV cache.
• Distillation minimizes KL(teacher || student). A one-hot label discards the teacher's alternatives.
• A MoE layer routes each token to the top-k experts and sums their outputs with softmax weights.
• Mixtral 8x7B has 46.7B total and 12.9B active parameters. The Qwen3-30B-A3B shape has 30.5B total and about 3.35B active.
• LoRA adds (alpha / r) B A to a frozen matrix. Rank 8 on a 4096 by 4096 matrix trains 65,536 parameters (0.39%).
• Quantization suffixes such as Q4_K_M, Q8_0, and F16 describe the number format of the file.
• A model card check covers architecture, size, context, template and tools, sampling advice, benchmarks, license, and available quantizations.