codeloopback
Guide contents

Lesson 12 of 73

Sampling: Greedy, Temperature, Top-k, Top-p, and Min-p

A language model does not output a word. After it reads the text so far, it outputs one number for every entry in its vocabulary (the list of all tokens it knows). Each number is a score for how well that token fits as the next one. These scores are called logits. A sampler is the small piece of code that looks at the logits and picks one token id. Text generation is this step repeated: pick a token, append it to the text, run the model again, pick the next token.

The sampler matters because the same model produces very different text depending on how it picks. Always taking the top score gives repetitive text. Drawing completely at random from all tokens gives nonsense. The samplers in this lesson sit between those two extremes, and every chat or coding tool built on a local model exposes their settings.

This lesson implements the basic samplers in Java on a 6-token vocabulary, compares them on a confident and a flat distribution, and checks their rules against the llama.cpp source. Afterwards you can predict what each setting does to a distribution, and you can read the sampler chain line that llama.cpp prints. The lesson follows the transformer lesson, whose forward pass (one run of the model over the text) ends in logits, and next-token prediction, which built the bigram model (a model that predicts the next character from the previous one only) used in Example 14.

Sampler Effect, in plain words
Greedy Always take the token with the highest score
Sampling Draw a random token, where likely tokens are drawn more often
Temperature Make the distribution more peaked or more even
Top-k Keep only the k most probable tokens
Top-p Keep the smallest group of top tokens whose probabilities add up to p
Min-p Keep tokens that have at least min_p times the top token's probability

Penalties, logit bias, and other samplers follow in Penalties, Logit Bias, and Sampler Chains. Most programs in this lesson use one vocabulary and one logit vector, so results can be compared across examples. The tokens are the, cat, sat, mat, dog, ran, with logits 2.0, 1.0, 0.5, 0.1, -0.5, -1.0. Think of them as the candidates for the word after "the dog sat on the".

Example 1: from logits to a probability table

Logits are hard to reason about: they can be negative, they do not add up to anything, and they have no fixed range. To draw a random token we need probabilities instead, which are positive numbers that add up to 1. The function that converts one into the other is called softmax.

Softmax works in two steps. First it raises e (about 2.718) to the power of each logit, which makes every value positive and keeps the order. Then it divides each value by the sum of all of them, so the total is 1.

This example prints the logit, the exponential, and the final probability of each token. The program subtracts the maximum logit before exponentiating. That does not change the result, and it prevents overflow (a number too large for a double) when logits are big. Look at the last column and at the final line, which shows the probabilities add up to 1.

Java
public class LogitsToProbs {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        double[] p = softmax(logits);
        double sum = 0;
        for (int i = 0; i < words.length; i++) {
            System.out.printf("%-4s logit %5.1f  exp %6.3f  probability %.3f%n", words[i], logits[i], Math.exp(logits[i]), p[i]);
            sum += p[i];
        }
        System.out.printf("sum of probabilities: %.3f%n", sum);
    }
}
Output
the  logit   2.0  exp  7.389  probability 0.534
cat  logit   1.0  exp  2.718  probability 0.196
sat  logit   0.5  exp  1.649  probability 0.119
mat  logit   0.1  exp  1.105  probability 0.080
dog  logit  -0.5  exp  0.607  probability 0.044
ran  logit  -1.0  exp  0.368  probability 0.027
sum of probabilities: 1.000

Take the first row by hand. A logit of 2.0 gives exp(2.0) = 7.389. The six exponentials add up to 13.836, so the has probability 7.389 / 13.836 = 0.534.

The numbers show how logits behave. A difference of one logit unit is a factor of e = 2.718 in probability: cat has a logit one below the, and its probability is 2.718 times smaller (0.196 against 0.534). Logits have no scale of their own, and only differences matter. Adding 100 to every logit gives the same probabilities, which is the reason subtracting the maximum is safe.

Example 2: greedy choice

The simplest way to pick a token is to take the one with the highest logit and ignore the rest. This is called greedy decoding. The program finds the best index with a loop that replaces the current best only when a logit is strictly larger, so on a tie the first index wins. The llama.cpp greedy sampler is written the same way.

The program runs the choice three times to show it never changes, and then tests a tie between cat and sat.

Java
public class Greedy {
    // first index of the largest value, as llama.cpp's greedy sampler does (it replaces only on a strictly larger logit)
    static int argmax(double[] logits) {
        int best = 0;
        for (int i = 1; i < logits.length; i++) if (logits[i] > logits[best]) best = i;
        return best;
    }

    public static void main(String[] args) {
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        for (int run = 1; run <= 3; run++) System.out.println("run " + run + ": " + words[argmax(logits)]);
        double[] tie = {1.0, 3.0, 3.0, 0.0, 0.0, 0.0};
        System.out.println("tie between cat and sat: " + words[argmax(tie)]);
    }
}
Output
run 1: the
run 2: the
run 3: the
tie between cat and sat: cat

All three runs print the, because the same logits always give the same answer. This is what deterministic means. In the tie test both cat and sat have logit 3.0, and cat wins because it has the lower index.

Greedy needs no probabilities, because softmax never changes the order of the logits, so the largest logit is also the largest probability. The cost of greedy is repetition. Once a context makes one continuation the best, greedy chooses it again every time the same situation returns. This is why lesson 5 saw loops such as "the the the".

Example 3: sampling by inverse CDF

Sampling draws a token at random, but not evenly: each token is drawn with the probability the model gave it. A token with probability 0.534 should come out in about 53 of 100 draws.

A computer can only produce a uniform random number u between 0 and 1, where every value is equally likely. The trick is to line the tokens up and give each one a slice of the range from 0 to 1, as wide as its probability. A u that lands in a token's slice selects that token. Because u is uniform, the chance of landing in a slice equals the slice width.

In code, the program keeps a running total of the probabilities and stops at the first token whose running total passes u. This is called inverse-CDF sampling. CDF means cumulative distribution function, which is the running total of the probabilities in a fixed token order. The program uses a java.util.Random with a fixed seed (a starting number that makes the sequence of random numbers repeatable), so the five draws are the same on every run.

The program first prints each token's interval, then draws five values of u and shows which token each one selects.

Java
import java.util.Random;

public class InverseCdf {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    // walk along the probabilities until the running total passes u
    static int sample(double[] p, double u) {
        double run = 0;
        for (int i = 0; i < p.length; i++) { run += p[i]; if (u < run) return i; }
        return p.length - 1;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] p = softmax(new double[] {2.0, 1.0, 0.5, 0.1, -0.5, -1.0});
        double run = 0;
        for (int i = 0; i < p.length; i++) {
            System.out.printf("%-4s p %.3f  interval [%.3f, %.3f)%n", words[i], p[i], run, run + p[i]);
            run += p[i];
        }
        Random rng = new Random(42);
        for (int i = 0; i < 5; i++) {
            double u = rng.nextDouble();
            System.out.printf("u = %.3f -> %s%n", u, words[sample(p, u)]);
        }
    }
}
Output
the  p 0.534  interval [0.000, 0.534)
cat  p 0.196  interval [0.534, 0.731)
sat  p 0.119  interval [0.731, 0.850)
mat  p 0.080  interval [0.850, 0.930)
dog  p 0.044  interval [0.930, 0.973)
ran  p 0.027  interval [0.973, 1.000)
u = 0.728 -> cat
u = 0.683 -> cat
u = 0.309 -> the
u = 0.277 -> the
u = 0.666 -> cat

The intervals split the range from 0 to 1 into six pieces, one per token, each as wide as the token's probability. A u below 0.534 selects the, a u from 0.534 to 0.731 selects cat, and so on. The five draws show the mapping. The value 0.728 falls in the cat interval, and 0.309 falls in the the interval. Three of the five draws are cat or the, which is expected because together they hold 73 percent of the probability.

The llama.cpp dist sampler uses the same walk. It does not normalize first. It draws u, multiplies it by the sum of the unnormalized weights exp(logit - max), and walks until the running sum reaches that target. The result is the same token.

Example 4: 10,000 draws match the probabilities

A few random draws prove nothing, because any token can come up by luck. This example checks that the walk of Example 3 really produces each token with its probability. It draws 100, 1,000, 10,000, and 100,000 tokens and compares the share of each token with its expected probability. Look at the "largest share error" column: it should shrink as the number of draws grows.

Java
import java.util.Random;

public class ManyDraws {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    // walk along the probabilities until the running total passes u
    static int sample(double[] p, double u) {
        double run = 0;
        for (int i = 0; i < p.length; i++) { run += p[i]; if (u < run) return i; }
        return p.length - 1;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] p = softmax(new double[] {2.0, 1.0, 0.5, 0.1, -0.5, -1.0});
        Random rng = new Random(42);
        for (int n : new int[] {100, 1_000, 10_000, 100_000}) {
            int[] counts = new int[6];
            for (int i = 0; i < n; i++) counts[sample(p, rng.nextDouble())]++;
            double worst = 0;
            for (int i = 0; i < 6; i++) worst = Math.max(worst, Math.abs((double) counts[i] / n - p[i]));
            System.out.printf("%,7d draws: share of the %.3f (expected %.3f), largest share error %.4f%n", n, (double) counts[0] / n, p[0], worst);
            if (n == 10_000)
                for (int i = 0; i < 6; i++) System.out.printf("    %-4s expected %5.0f  got %5d%n", words[i], p[i] * n, counts[i]);
        }
    }
}
Output
    100 draws: share of the 0.530 (expected 0.534), largest share error 0.0308
  1,000 draws: share of the 0.503 (expected 0.534), largest share error 0.0325
 10,000 draws: share of the 0.530 (expected 0.534), largest share error 0.0043
    the  expected  5341  got  5298
    cat  expected  1965  got  1967
    sat  expected  1192  got  1217
    mat  expected   799  got   783
    dog  expected   438  got   466
    ran  expected   266  got   269
100,000 draws: share of the 0.534 (expected 0.534), largest share error 0.0013

The counts approach the probabilities as the number of draws grows. The largest error in any token's share is 0.031 after 100 draws, 0.004 after 10,000, and 0.001 after 100,000. The 1,000-draw line is slightly worse than the 100-draw line, which shows that the error shrinks on average, not at every step.

At 10,000 draws the counts are close to the expected counts but not equal: 5,298 against 5,341 for the, and 466 against 438 for dog. Random counts typically deviate by about sqrt(n * p * (1 - p)). For the at n = 10,000 this is sqrt(10000 * 0.534 * 0.466), which is about 50. The deviations here are of that size, so the sampler draws each token with its probability.

One consequence matters in practice: a model that samples many tokens produces the low-probability tokens too. At a probability of 0.027, ran appears about once every 37 tokens. Over a long answer, rare tokens will appear, and some of them will be wrong. The filters later in this lesson exist to cut them off.

Example 5: temperature

Temperature is a setting that controls how adventurous the sampling is. It divides every logit by a number T before softmax. The effect follows from the gap between two tokens: if two logits differ by 1, their probabilities differ by a factor of e^1. After dividing by T = 0.5, the gap becomes 2, so the factor is e^2, which is bigger.

  • A value below 1 widens the gaps between logits, so the top token gains probability and the output becomes more predictable.
  • A value above 1 narrows the gaps, so the distribution flattens and less likely tokens get chosen more often.
  • At T = 1 the model's own distribution is unchanged.

The name comes from physics, where a hot system moves around randomly and a cold one settles into its lowest state. The mapping is loose: here "hot" means an even distribution and "cold" means a peaked one. The analogy stops there, because this temperature has no units.

The program prints one column of probabilities per temperature. Compare the the row across columns, and notice that the order of the tokens never changes.

Java
public class Temperature {
    static double[] softmax(double[] logits, double temperature) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l / temperature);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] / temperature - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        double[] temps = {0.2, 0.7, 1.0, 1.5};
        double[][] columns = new double[temps.length][];
        System.out.print("token   ");
        for (int t = 0; t < temps.length; t++) { columns[t] = softmax(logits, temps[t]); System.out.printf("  T=%.1f", temps[t]); }
        System.out.println();
        for (int i = 0; i < words.length; i++) {
            System.out.printf("%-6s ", words[i]);
            for (double[] c : columns) System.out.printf("  %.3f", c[i]);
            System.out.println();
        }
    }
}
Output
token     T=0.2  T=0.7  T=1.0  T=1.5
the      0.993  0.683  0.534  0.402
cat      0.007  0.164  0.196  0.206
sat      0.001  0.080  0.119  0.148
mat      0.000  0.045  0.080  0.113
dog      0.000  0.019  0.044  0.076
ran      0.000  0.009  0.027  0.054

Check the T = 0.2 column by hand. The logits become 10, 5, 2.5, 0.5, -2.5, -5. The gap between the first two is now 5, a factor of e^5 = 148. So the takes e^10 = 22,026 of a total of about 22,188, which is 0.993. The table shows it: the has probability 0.993 and the output is nearly deterministic.

At T = 0.7 the top token has 0.683. At T = 1.5 it drops to 0.402, and ran rises from 0.027 to 0.054, twice as likely. The order of the tokens never changes. Temperature changes only how much probability the leaders keep. llama.cpp defaults to a temperature of 0.80.

Example 6: temperature near zero, and how engines treat 0

If lower temperature means more predictable output, what is the limit? As T approaches 0, the top probability approaches 1, and the sampler turns into greedy decoding. At exactly 0 there is a problem: dividing by zero is undefined.

This example shows both parts. The first four lines print the top token's probability as T shrinks. The next lines show what plain floating-point math does at T = 0, and the last line shows the rule llama.cpp uses instead. In the code, applyTemperature implements that rule: at temp <= 0 it keeps only the first largest logit and sets every other logit to negative infinity.

Java
public class TemperatureZero {
    // the rule of llama.cpp's temperature sampler: at temp <= 0 keep only the first largest logit
    static double[] applyTemperature(double[] logits, double temp) {
        double[] out = new double[logits.length];
        if (temp <= 0) {
            int best = 0;
            for (int i = 1; i < logits.length; i++) if (logits[i] > logits[best]) best = i;
            java.util.Arrays.fill(out, Double.NEGATIVE_INFINITY);
            out[best] = logits[best];
        } else {
            for (int i = 0; i < logits.length; i++) out[i] = logits[i] / temp;
        }
        return out;
    }

    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        for (double t : new double[] {1.0, 0.1, 0.01, 0.001})
            System.out.printf("T = %-5s probability of the top token: %.6f%n", t, softmax(applyTemperature(logits, t))[0]);
        System.out.println("logit / 0.0 in Java: " + logits[0] / 0.0 + ", and " + logits[1] / 0.0 + " for the next one");
        System.out.println("softmax of those values: " + java.util.Arrays.toString(softmax(new double[] {Double.POSITIVE_INFINITY, Double.POSITIVE_INFINITY})));
        System.out.println("rule for T <= 0: " + java.util.Arrays.toString(softmax(applyTemperature(logits, 0))));
    }
}
Output
T = 1.0   probability of the top token: 0.534060
T = 0.1   probability of the top token: 0.999954
T = 0.01  probability of the top token: 1.000000
T = 0.001 probability of the top token: 1.000000
logit / 0.0 in Java: Infinity, and Infinity for the next one
softmax of those values: [NaN, NaN]
rule for T <= 0: [1.0, 0.0, 0.0, 0.0, 0.0, 0.0]

At T = 0.1 the top token has probability 0.99995, and at T = 0.01 it rounds to 1.000000. At T = 0 Java gives Infinity for a positive logit divided by zero. Softmax of two infinities produces NaN ("not a number"), because it computes Infinity - Infinity. A NaN cannot be used to pick a token.

Engines therefore treat T <= 0 as a special case. In the pinned llama.cpp source (the exact version this course uses), the temperature sampler keeps the first token with the highest logit, sets the other logits to negative infinity, and returns. The file common/common.h documents temp <= 0.0 as "sample greedily". The last output line shows the result: a probability of 1.0 for the top token and 0.0 for the rest. Servers inherit this rule, so "temperature": 0 in a request means greedy.

Example 7: entropy as a measure of randomness

Temperature is a knob, but how do we measure how random a distribution is? One number for this is entropy. If one token has nearly all the probability, the next token is almost certain and entropy is low. If many tokens are equally likely, the next token is hard to guess and entropy is high.

Entropy is measured in bits with the formula H = -sum(p * log2(p)). It is 0 when one token has all the probability, and log2(n) when all n tokens are equally likely. Here log2(x) is the power you raise 2 to in order to get x, so log2(8) = 3. The program computes entropy for the logits of Example 1 at several temperatures. The last six lines show each token's term of the sum at T = 1.0.

Java
public class Entropy {
    static double[] softmax(double[] logits, double temperature) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l / temperature);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] / temperature - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    static double entropyBits(double[] p) {
        double h = 0;
        for (double v : p) if (v > 0) h -= v * Math.log(v) / Math.log(2);
        return h;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        for (double t : new double[] {0.05, 0.2, 0.7, 1.0, 1.5, 5.0, 100.0})
            System.out.printf("T = %-6s entropy %.3f bits%n", t, entropyBits(softmax(logits, t)));
        System.out.printf("maximum for 6 tokens: %.3f bits%n", Math.log(6) / Math.log(2));
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] p = softmax(logits, 1.0);
        for (int i = 0; i < p.length; i++)                     // the terms of the sum at T = 1.0
            System.out.printf("  %-4s p %.3f  -p*log2(p) = %.3f%n", words[i], p[i], -p[i] * Math.log(p[i]) / Math.log(2));
    }
}
Output
T = 0.05   entropy 0.000 bits
T = 0.2    entropy 0.066 bits
T = 0.7    entropy 1.470 bits
T = 1.0    entropy 1.938 bits
T = 1.5    entropy 2.273 bits
T = 5.0    entropy 2.556 bits
T = 100.0  entropy 2.585 bits
maximum for 6 tokens: 2.585 bits
  the  p 0.534  -p*log2(p) = 0.483
  cat  p 0.196  -p*log2(p) = 0.461
  sat  p 0.119  -p*log2(p) = 0.366
  mat  p 0.080  -p*log2(p) = 0.291
  dog  p 0.044  -p*log2(p) = 0.198
  ran  p 0.027  -p*log2(p) = 0.139

Start with the last six lines, which compute the formula by hand at T = 1.0. Each token contributes -p * log2(p). The token the with probability 0.534 contributes 0.483 bits, and ran with 0.027 contributes 0.139. The six terms add up to 1.938 bits, the value in the T = 1.0 row. A token with probability near 0 or near 1 contributes almost nothing, because either p or log2(p) is near 0.

Now read the table from top to bottom. At T = 0.05 the entropy is 0.000 bits: the choice is certain. At T = 1.0 it is 1.938 bits, about as unsure as a choice among 4 equally likely tokens (2^1.938 = 3.8). As T grows the entropy approaches the maximum of 2.585 bits for six tokens, which is log2(6). At T = 100 it equals that maximum to three decimals.

Entropy returns in the mirostat sampler, which steers it toward a target. It is also related to the perplexity of lesson 5, which describes how surprised a model is by real text.

Example 8: top-k

Temperature reshapes the whole distribution, but it never removes any token. On a real vocabulary of 128,000 tokens, most entries are nonsense in the current context, and each has a tiny probability. Added together, those tiny probabilities can still be large enough that a sampler picks one of them now and then.

Top-k is the first filter that fixes this. It keeps the k most probable tokens and sets the others to 0. Then it renormalizes, which means dividing each kept probability by the kept total so the remaining probabilities add up to 1 again.

The program prints the original probabilities next to the results for k = 3 and k = 1.

Java
public class TopK {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    // keep the k most probable tokens, set the others to 0, and renormalize
    static double[] topK(double[] p, int k) {
        Integer[] order = new Integer[p.length];
        for (int i = 0; i < p.length; i++) order[i] = i;
        java.util.Arrays.sort(order, (a, b) -> Double.compare(p[b], p[a]));
        double[] out = new double[p.length];
        double sum = 0;
        for (int r = 0; r < Math.min(k, p.length); r++) { out[order[r]] = p[order[r]]; sum += p[order[r]]; }
        for (int i = 0; i < out.length; i++) out[i] /= sum;
        return out;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] p = softmax(new double[] {2.0, 1.0, 0.5, 0.1, -0.5, -1.0});
        double[] k3 = topK(p, 3), k1 = topK(p, 1);
        System.out.println("token  before  top-k 3  top-k 1");
        for (int i = 0; i < words.length; i++)
            System.out.printf("%-5s  %.3f   %.3f    %.3f%n", words[i], p[i], k3[i], k1[i]);
        System.out.println("top-k 0 or less keeps everything in llama.cpp; top-k 1 is greedy");
    }
}
Output
token  before  top-k 3  top-k 1
the    0.534   0.629    1.000
cat    0.196   0.231    0.000
sat    0.119   0.140    0.000
mat    0.080   0.000    0.000
dog    0.044   0.000    0.000
ran    0.027   0.000    0.000
top-k 0 or less keeps everything in llama.cpp; top-k 1 is greedy

With k = 3, the survivors the, cat, and sat have 0.534, 0.196, and 0.119, which add up to 0.849. Dividing each by 0.849 gives 0.629, 0.231, and 0.140. The ratios between the survivors are unchanged. Only the total is repaired. With k = 1 only the remains, which is greedy.

In llama.cpp, --top-k defaults to 40 and a value of 0 disables the filter. The weakness of a fixed k is that it ignores the shape of the distribution. With k = 40, a confident step with one clear answer still keeps 39 other candidates. A flat step with 100 plausible words loses 60 of them. Top-p, in the next example, addresses this.

Example 9: top-p (nucleus sampling): sort, accumulate, cut

Top-p adapts the number of survivors to the distribution. It is also called nucleus sampling, because the kept set is the core of the distribution. It works in three steps:

  1. Sort the tokens by probability, largest first.
  2. Add the probabilities up from the top, keeping a running sum.
  3. Stop when the running sum reaches p, and keep every token added so far.

The token that crosses the limit is included. This is also the rule in llama.cpp (cum_sum >= p). The kept set is renormalized, as in top-k.

The program prints each step for p = 0.9 on the logits of Example 1. The tokens are deliberately listed out of order, so the sort is visible: look at which token is added first.

Java
public class TopPSteps {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"mat", "the", "ran", "sat", "dog", "cat"};
        double[] p = softmax(new double[] {0.1, 2.0, -1.0, 0.5, -0.5, 1.0});   // Example 1's logits, shuffled
        Integer[] order = new Integer[p.length];
        for (int i = 0; i < p.length; i++) order[i] = i;
        java.util.Arrays.sort(order, (a, b) -> Double.compare(p[b], p[a]));   // 1. sort by probability
        double run = 0, limit = 0.9;
        int kept = 0;
        for (int id : order) {                                                // 2. accumulate
            run += p[id];
            kept++;
            System.out.printf("add %-4s p %.3f  running sum %.3f%s%n", words[id], p[id], run, run >= limit ? "  >= 0.9: cut here" : "");
            if (run >= limit) break;                                          // 3. cut after the crossing token
        }
        System.out.printf("kept %d of %d tokens; renormalized:", kept, p.length);
        for (int r = 0; r < kept; r++) System.out.printf(" %s %.3f", words[order[r]], p[order[r]] / run);
        System.out.println();
    }
}
Output
add the  p 0.534  running sum 0.534
add cat  p 0.196  running sum 0.731
add sat  p 0.119  running sum 0.850
add mat  p 0.080  running sum 0.930  >= 0.9: cut here
kept 4 of 6 tokens; renormalized: the 0.575 cat 0.211 sat 0.128 mat 0.086

The sort puts the first even though it is second in the input. The running sums are 0.534, 0.731, 0.850, and 0.930. The sum first reaches 0.9 at mat, so mat is kept and the cut falls after it.

The kept probabilities add up to 0.930, so dividing each by 0.930 makes them sum to 1 again. For example, the rises from 0.534 to 0.575. The two dropped tokens, dog and ran, held 0.070 of the probability, which is the amount that was redistributed.

Sorting is the expensive part on a real vocabulary of 150,000 tokens. The llama.cpp implementation avoids a full sort when it can. For more than 1,024 candidates it first sorts only the top 256, and it sorts the rest only if the running sum has not reached p by then.

Example 10: top-p on a confident and a flat distribution

The point of top-p is that the number of kept tokens changes with the model's confidence. This example runs top-p with p = 0.9 on three distributions:

  • the one from Example 1, called medium;
  • a confident one, with logits 5, 2, 1, 0, -1, -2;
  • a flat one, with logits from 1.0 down to 0.5 in steps of 0.1.

For each, the program prints how many tokens survive and how much probability they cover.

Java
import java.util.ArrayList;
import java.util.List;

public class TopP {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    // sort by probability, accumulate, and cut after the token whose running sum first reaches p
    static List<Integer> nucleus(double[] probs, double p) {
        Integer[] order = new Integer[probs.length];
        for (int i = 0; i < probs.length; i++) order[i] = i;
        java.util.Arrays.sort(order, (a, b) -> Double.compare(probs[b], probs[a]));
        List<Integer> kept = new ArrayList<>();
        double run = 0;
        for (int id : order) {
            kept.add(id);
            run += probs[id];
            if (run >= p) break;
        }
        return kept;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[][] logits = {{2.0, 1.0, 0.5, 0.1, -0.5, -1.0}, {5.0, 2.0, 1.0, 0.0, -1.0, -2.0}, {1.0, 0.9, 0.8, 0.7, 0.6, 0.5}};
        String[] names = {"medium", "confident", "flat"};
        for (int d = 0; d < logits.length; d++) {
            double[] probs = softmax(logits[d]);
            List<Integer> kept = nucleus(probs, 0.9);
            double mass = 0;
            StringBuilder sb = new StringBuilder();
            for (int id : kept) { mass += probs[id]; sb.append(words[id]).append(String.format(" %.3f, ", probs[id])); }
            System.out.printf("%-9s top-p 0.9 keeps %d: %scovering %.3f%n", names[d], kept.size(), sb, mass);
        }
    }
}
Output
medium    top-p 0.9 keeps 4: the 0.534, cat 0.196, sat 0.119, mat 0.080, covering 0.930
confident top-p 0.9 keeps 1: the 0.927, covering 0.927
flat      top-p 0.9 keeps 6: the 0.211, cat 0.191, sat 0.173, mat 0.156, dog 0.141, ran 0.128, covering 1.000

The confident distribution keeps one token, because the alone has 0.927, which is above 0.9. The medium one keeps four tokens, covering 0.930, as in Example 9. The flat one keeps all six, since each token holds around one sixth and the running sum only passes 0.9 at the last token.

The size of the kept set follows the model's confidence, which is the advantage over top-k. With a fixed k = 3, the confident step would keep two tokens it almost never needs, and the flat step would lose three plausible ones.

A value of 1.0 disables the filter (llama.cpp returns at once when p >= 1). llama.cpp defaults to 0.95.

Example 11: min-p

Top-p has a weakness: to reach its target sum it may have to include a long tail of unlikely tokens, because many small probabilities add up. Min-p uses a different test. It keeps every token whose probability is at least min_p times the probability of the top token.

An example with real numbers: if the top token has probability 0.60 and min_p is 0.1, the threshold is 0.06. A token with 0.07 stays and a token with 0.05 is cut. The threshold scales with the model's confidence. A confident head sets a high bar, and a flat head sets a low one.

In logit terms the rule is logit >= max_logit + ln(min_p), which is how llama.cpp computes it. This is the same test, because taking the logarithm of "probability at least min_p times the top probability" gives that expression.

The program compares min-p with top-p on three distributions, including one with a strong head and a long tail of 10 tokens at 0.015 each. The output shows how many tokens each filter keeps.

Java
public class MinP {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    static int topP(double[] p, double limit) {          // how many tokens top-p keeps
        double[] sorted = p.clone();
        java.util.Arrays.sort(sorted);
        double run = 0;
        for (int i = sorted.length - 1, n = 1; i >= 0; i--, n++) { run += sorted[i]; if (run >= limit) return n; }
        return p.length;
    }

    static int minP(double[] p, double minP) {           // keep tokens with p >= minP * p_max
        double max = 0;
        for (double v : p) max = Math.max(max, v);
        int kept = 0;
        for (double v : p) if (v >= minP * max) kept++;
        return kept;
    }

    public static void main(String[] args) {
        double[] tail = {0.30, 0.25, 0.20, 0.10, 0.015, 0.015, 0.015, 0.015, 0.015, 0.015, 0.015, 0.015, 0.015, 0.015};
        double[][] dists = {softmax(new double[] {5.0, 2.0, 1.0, 0.0, -1.0, -2.0}), softmax(new double[] {1.0, 0.9, 0.8, 0.7, 0.6, 0.5}), tail};
        String[] names = {"confident (6 tokens)", "flat (6 tokens)", "head and tail (14)"};
        for (int d = 0; d < dists.length; d++)
            System.out.printf("%-22s top-p 0.95 keeps %2d, min-p 0.1 keeps %2d%n", names[d], topP(dists[d], 0.95), minP(dists[d], 0.1));
    }
}
Output
confident (6 tokens)   top-p 0.95 keeps  2, min-p 0.1 keeps  1
flat (6 tokens)        top-p 0.95 keeps  6, min-p 0.1 keeps  6
head and tail (14)     top-p 0.95 keeps 11, min-p 0.1 keeps  4

On the confident distribution the min-p threshold is 0.1 * 0.927 = 0.0927, and the second token has only 0.046, so one token stays. Top-p 0.95 keeps two tokens there, because the alone (0.927) is below 0.95 and the second token brings the sum to 0.973.

On the flat distribution every token is above 0.1 * 0.211, and top-p needs all six to reach 0.95, so both keep six. On these two distributions the filters agree to within one token.

On the head-and-tail distribution they differ. Top-p with p = 0.95 keeps 11 tokens, because the four head tokens only cover 0.85 and 7 tail tokens are needed to reach 0.95. Min-p with 0.1 has a threshold of 0.03 (0.1 times 0.30), so the tail tokens at 0.015 are cut and 4 remain.

The lesson is that a long tail of unlikely tokens can add up to a large share of the probability. Top-p must include part of the tail to reach its target. Min-p ignores it. llama.cpp defaults min_p to 0.05, and --min-p 0 disables it.

Example 12: the order of the filters matters

Real engines apply several of these steps one after another, and each step sees the output of the previous one. This raises a question: does the order change the result? It does. Temperature and top-p do not commute, which means swapping them gives a different outcome.

The program applies temperature 1.5 and top-p 0.9 in both orders to the logits of Example 1. Compare which tokens survive in each line.

Java
import java.util.ArrayList;
import java.util.List;

public class Order {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    static double[] temperature(double[] logits, double t) {
        double[] out = new double[logits.length];
        for (int i = 0; i < out.length; i++) out[i] = logits[i] / t;
        return out;
    }

    // top-p on logits: kept tokens keep their logits, others become -infinity
    static double[] topP(double[] logits, double limit) {
        double[] p = softmax(logits);
        Integer[] order = new Integer[p.length];
        for (int i = 0; i < p.length; i++) order[i] = i;
        java.util.Arrays.sort(order, (a, b) -> Double.compare(p[b], p[a]));
        double[] out = new double[p.length];
        java.util.Arrays.fill(out, Double.NEGATIVE_INFINITY);
        double run = 0;
        for (int id : order) { out[id] = logits[id]; run += p[id]; if (run >= limit) break; }
        return out;
    }

    static String show(double[] logits) {
        double[] p = softmax(logits);
        StringBuilder sb = new StringBuilder();
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        for (int i = 0; i < p.length; i++) if (p[i] > 0) sb.append(String.format("%s %.3f  ", words[i], p[i]));
        return sb.toString().trim();
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        System.out.println("temperature 1.5 first, then top-p 0.9: " + show(topP(temperature(logits, 1.5), 0.9)));
        System.out.println("top-p 0.9 first, then temperature 1.5: " + show(temperature(topP(logits, 0.9), 1.5)));
    }
}
Output
temperature 1.5 first, then top-p 0.9: the 0.425  cat 0.218  sat 0.156  mat 0.120  dog 0.080
top-p 0.9 first, then temperature 1.5: the 0.462  cat 0.237  sat 0.170  mat 0.130

With temperature first, the distribution is the T=1.5 column of Example 5: 0.402, 0.206, 0.148, 0.113, 0.076. The first four add up to 0.869, below 0.9, so dog is needed and the sum reaches 0.945. The flatter distribution needs five tokens.

With top-p first, the cut happens on the original distribution and leaves four tokens, and temperature then reshapes only those four. The same two settings give different token sets, because raising the temperature first spreads probability into the tail, and top-p then has to keep more tokens to reach 0.9.

llama.cpp applies the filters first and temperature near the end. Its default order, from common/common.h, is penalties, DRY, top-n-sigma, top-k, typical, top-p, min-p, XTC, temperature, followed by the final dist sampler that draws the token. Penalties, DRY, top-n-sigma, typical, and XTC are covered in the next lesson, and all of them are off by default.

The tool llama-completion prints the chain it built. The commands below show it. Two details help in reading the output:

  • A ? in front of a name means the sampler is disabled by its parameter value (for example top-k with 0), and llama.cpp puts an empty placeholder in its place.
  • temp-ext is the temperature sampler with an optional dynamic-temperature range. The range is 0 (off) by default, so it divides the logits by the temperature as in Example 5.

The commands use the tiny random-weight model, because the log lines do not depend on the weights. The second command uses --samplers to replace the whole chain with top-k and temperature only.

Terminal
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "Hi" -n 2 -no-cnv --seed 7 2>&1 | grep "sampler chain"
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "Hi" -n 2 -no-cnv --seed 7 --samplers "top_k;temperature" 2>&1 | grep "sampler chain"

Representative output (from the pinned llama.cpp build; timestamps removed):

Output
sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist
sampler chain: logits -> top-k -> temp-ext -> dist

With defaults, the enabled samplers are top-k (40), top-p (0.95), min-p (0.05), temp-ext (0.80), and dist. The --samplers flag replaces the whole list and its order, which is why the second line has only top-k and temp-ext. Because temperature comes after the truncation filters, a higher temperature spreads probability over the tokens that survived and does not let removed tokens return.

Example 13: seeds, and why identical settings can still differ

A seed fixes the sequence of random numbers a sampler uses. Two samplers with the same seed and the same probabilities pick the same tokens. This is useful for tests and for reproducing a result.

The program shows three things. First, the same seed gives the same twelve draws twice. Second, a different seed gives a different sequence. Third, it shows what happens when the probabilities change slightly, which can happen when a computation runs in a different order. It changes two logits by 0.003 and counts how many of 10,000 draws change.

Java
import java.util.Random;

public class Seeds {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    static int sample(double[] p, double u) {
        double run = 0;
        for (int i = 0; i < p.length; i++) { run += p[i]; if (u < run) return i; }
        return p.length - 1;
    }

    static int[] draw(double[] logits, long seed, int n) {
        Random rng = new Random(seed);
        double[] p = softmax(logits);
        int[] out = new int[n];
        for (int i = 0; i < n; i++) out[i] = sample(p, rng.nextDouble());
        return out;
    }

    public static void main(String[] args) {
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        System.out.println("seed 7, run 1: " + java.util.Arrays.toString(draw(logits, 7, 12)));
        System.out.println("seed 7, run 2: " + java.util.Arrays.toString(draw(logits, 7, 12)));
        System.out.println("seed 8:        " + java.util.Arrays.toString(draw(logits, 8, 12)));

        // the same seed, but the logits differ in the third decimal, as can happen between kernels or batch shapes
        double[] shifted = logits.clone();
        shifted[0] += 0.003;
        shifted[2] -= 0.003;
        int[] a = draw(logits, 7, 10_000), b = draw(shifted, 7, 10_000);
        int different = 0;
        for (int i = 0; i < a.length; i++) if (a[i] != b[i]) different++;
        System.out.println("of 10000 draws with a 0.003 change in two logits, " + different + " pick a different token");
    }
}
Output
seed 7, run 1: [2, 2, 0, 3, 1, 0, 0, 3, 0, 3, 1, 2]
seed 7, run 2: [2, 2, 0, 3, 1, 0, 0, 3, 0, 3, 1, 2]
seed 8:        [1, 1, 0, 2, 1, 1, 0, 1, 0, 3, 0, 3]
of 10000 draws with a 0.003 change in two logits, 24 pick a different token

The first two lines are identical, and a different seed gives a different sequence. In the last test two logits move by 0.003, a size that rounding differences between kernels (the GPU or CPU routines that do the math) or batch shapes can produce in half precision. Of 10,000 draws, 24 pick a different token. These are the draws whose random number falls close to an interval boundary, so the small change in the interval moves it to the neighbor.

One different token is enough to change everything after it. The context is now different, so the model produces different logits for every later step, and the rest of a generated text diverges.

A seed therefore gives repeatable output only when everything before the sampler is also identical: the same model file, build, hardware, thread count, and batch composition. Floating-point sums depend on the order of addition, which an earlier lesson on numeric precision covered, and servers change that order when requests share a batch. llama.cpp documents --seed as the RNG (random number generator) seed, with a default of -1, which means a random seed. The dist sampler draws from a std::mt19937 generator that the seed initializes.

Example 14: text from the bigram model under each setting

The earlier examples used six tokens. This example shows the settings on real generated text. The program uses the character bigram model of lesson 5: a table that counts, for each character in a small corpus, which characters follow it. The row for the current character is the token distribution for the next step.

The program generates 60 characters with seven settings, all with the same seed. The settings apply truncation first and temperature last, as in llama.cpp. After the text, the program prints the row of counts for the character after a space, and the characters that top-k 3, top-p 0.9, and min-p 0.2 would allow there. Compare the seven lines of text first, then read the last four lines.

Java
import java.util.Map;
import java.util.Random;
import java.util.TreeMap;

public class BigramText {
    static final String CORPUS = "the cat sat on the mat. the dog sat on the log. "
        + "the cat saw the dog. the dog saw the cat. a cat and a dog sat on a mat.";

    // the token at sorted rank r survives top-k, min-p, and top-p (run = probability of the tokens ranked above it)
    static boolean keep(int r, double prob, double run, double max, int topK, double topP, double minP) {
        return (topK <= 0 || r < topK) && prob >= minP * max && (r == 0 || run < topP);
    }

    // settings: temperature, top-k (0 = off), top-p (1 = off), min-p (0 = off); temperature 0 means greedy
    static char next(Map<Character, Integer> row, double temp, int topK, double topP, double minP, Random rng) {
        int n = row.size();
        char[] chars = new char[n];
        double[] p = new double[n];
        int i = 0;
        for (Map.Entry<Character, Integer> e : row.entrySet()) { chars[i] = e.getKey(); p[i++] = e.getValue(); }
        Integer[] order = new Integer[n];
        for (int j = 0; j < n; j++) order[j] = j;
        java.util.Arrays.sort(order, (a, b) -> Double.compare(p[b], p[a]));
        if (temp <= 0) return chars[order[0]];
        double total = 0;
        for (double v : p) total += v;
        double[] q = new double[n];                                  // truncation first, then temperature last (llama.cpp order)
        double run = 0, max = p[order[0]] / total;
        for (int r = 0; r < n; r++) {
            int id = order[r];
            double prob = p[id] / total;
            if (keep(r, prob, run, max, topK, topP, minP)) q[id] = Math.pow(prob, 1 / temp);   // dividing the log by T equals this power
            run += prob;
        }
        double sum = 0;
        for (double v : q) sum += v;
        double u = rng.nextDouble() * sum;
        for (int j = 0; j < n; j++) { u -= q[j]; if (q[j] > 0 && u < 0) return chars[j]; }
        return chars[order[0]];
    }

    public static void main(String[] args) {
        Map<Character, Map<Character, Integer>> table = new TreeMap<>();
        for (int i = 0; i + 1 < CORPUS.length(); i++)
            table.computeIfAbsent(CORPUS.charAt(i), k -> new TreeMap<>()).merge(CORPUS.charAt(i + 1), 1, Integer::sum);
        String[] names = {"greedy", "T=0.7", "T=1.0", "T=1.5", "top-k 3", "top-p 0.9", "min-p 0.2"};
        double[][] settings = {{0, 0, 1, 0}, {0.7, 0, 1, 0}, {1.0, 0, 1, 0}, {1.5, 0, 1, 0}, {1.0, 3, 1, 0}, {1.0, 0, 0.9, 0}, {1.0, 0, 1, 0.2}};
        for (int s = 0; s < settings.length; s++) {
            Random rng = new Random(5);
            StringBuilder out = new StringBuilder("t");
            char c = 't';
            for (int i = 0; i < 59; i++) {
                c = next(table.get(c), settings[s][0], (int) settings[s][1], settings[s][2], settings[s][3], rng);
                out.append(c);
            }
            System.out.printf("%-10s %s%n", names[s], out);
        }
        Map<Character, Integer> space = table.get(' ');           // the row after a space
        System.out.println("row after ' ': " + space);
        Character[] sorted = space.keySet().toArray(new Character[0]);
        java.util.Arrays.sort(sorted, (x, y) -> space.get(y) - space.get(x));
        for (int s = 4; s < settings.length; s++) {
            StringBuilder kept = new StringBuilder();
            double total = 0;
            for (int count : space.values()) total += count;
            int above = 0;                                         // count of the characters ranked above
            for (int r = 0; r < sorted.length; r++) {
                double prob = space.get(sorted[r]) / total, max = space.get(sorted[0]) / total;
                if (keep(r, prob, above / total, max, (int) settings[s][1], settings[s][2], settings[s][3])) kept.append(sorted[r]);
                above += space.get(sorted[r]);
            }
            System.out.printf("%-10s can pick: %s%n", names[s], kept);
        }
    }
}
Output
greedy     the the the the the the the the the the the the the the the 
T=0.7      the og. t the maw dog a sat athe ca sathe sat the cathe sat 
T=1.0      the mat saw cat. t cat at. d dog ca sathe sat the ca t the a
T=1.5      the mat saw cat. t cat at. d dog ca saw sathe t. t a t the a
top-k 3    the sat t the saw sat. at. sathe at the t the the a aw the a
top-p 0.9  the mat t the maw dog a sat athe ca sathe sat the ca t the a
min-p 0.2  the mat t the maw dog a sat athe ca sathe sat the ca t the a
row after ' ': {a=4, c=4, d=4, l=1, m=2, o=3, s=5, t=7}
top-k 3    can pick: tsa
top-p 0.9  can pick: tsacdo
min-p 0.2  can pick: tsacdom

Greedy loops on the because each step picks the most frequent continuation: after t comes h, after h comes e, after e comes a space, and after a space comes t again. The four characters form a cycle that greedy decoding never leaves.

The sampled lines start alike because every setting uses the same seed, so the same random numbers are drawn, and the distributions are similar. A line changes where a draw falls in a region that the setting changed. T=0.7 leaves the other lines at the fifth character (og. instead of mat): the sharper distribution moves the boundaries between intervals, so the same u selects a different character. T=1.0 and T=1.5 agree for the first 38 characters and then diverge for the same reason.

The last four lines show one row of the table, the 30 characters that follow a space in the corpus, and the characters each filter leaves available:

  • Top-k 3 keeps only t, s, and a. The tie between a, c, and d at 4 counts is broken by sort order.
  • Top-p 0.9 keeps six characters. The running sum reaches exactly 27/30 = 0.9 at o, so m and l are cut.
  • Min-p 0.2 has a threshold of 0.2 * 7/30 = 0.047, so it keeps m (2/30 = 0.067) and cuts only l (1/30 = 0.033).

Top-p and min-p remove nearly the same characters, which is why their lines are identical here.

The bigram model is too small to show quality differences, because each row has only a handful of options. A large model shows the effect of each setting more clearly. The criteria for choosing values are in Penalties, Logit Bias, and Sampler Chains.

Interview points

  • Logits are unnormalized scores, one per vocabulary entry. Softmax turns them into probabilities, and only the differences between logits matter.
  • Greedy decoding always takes the highest logit, so it is deterministic and often repeats itself. Sampling draws from the distribution with a random number and the inverse-CDF walk.
  • Temperature divides the logits. Below 1 it sharpens the distribution, above 1 it flattens it, and it never changes the order of the tokens. T <= 0 is handled as greedy because the division is undefined.
  • Entropy measures randomness in bits: 0 for a certain choice, and log2(n) for a uniform choice over n tokens.
  • Top-k keeps a fixed number of tokens. Top-p keeps the smallest set whose probability reaches p. Min-p keeps tokens above a fraction of the top probability.
  • Min-p scales its threshold with the model's confidence. It avoids including a long tail, which top-p can do when the head is flat.
  • Filters do not commute, so the order changes the result. llama.cpp truncates first and applies temperature last by default.
  • llama.cpp defaults are temperature 0.80, top-k 40, top-p 0.95, and min-p 0.05, with the other samplers disabled.
  • A seed makes the random numbers repeatable. It does not make results portable across hardware, builds, or batch compositions.
  • The sampler chain log line shows the active samplers, and a ? marks a disabled one.

Summary

  • A sampler maps a vector of logits to one token id.
  • Softmax is p_i = exp(l_i - max) / sum_j exp(l_j - max). Adding a constant to all logits changes nothing.
  • Greedy picks the first largest logit. Sampling walks the cumulative probabilities until the running total passes a random u in [0, 1).
  • Temperature divides logits by T. At T <= 0 llama.cpp keeps only the top logit.
  • Entropy -sum(p log2 p) is 0 for a certain choice and at most log2(n).
  • Top-k keeps k tokens and renormalizes. In llama.cpp, 0 disables it and 40 is the default.
  • Top-p keeps tokens until the cumulative probability reaches p, including the token that crosses the limit. The default is 0.95.
  • Min-p keeps tokens with p_i >= min_p * p_max, computed as logit >= max_logit + ln(min_p). The default is 0.05.
  • Applying temperature before or after truncation gives different token sets. llama.cpp orders the chain penalties, DRY, top-n-sigma, top-k, typical, top-p, min-p, XTC, temperature, dist.
  • A seeded sampler repeats exactly if the logits are bit-identical. Small numeric differences change a few draws, and then the rest of the text.
  • --samplers sets the sampler list and order for llama-completion.