Penalties, Logit Bias, and Sampler Chains

The previous lesson, Sampling: Greedy, Temperature, Top-k, Top-p, and Min-p, reshaped one probability distribution at a time. Each of those samplers looks only at the current scores. They do not know what the model has already written, and they cannot be told "never say this word".

This lesson covers the samplers that fill those gaps. Some look at the text generated so far and push the model away from repeating itself (penalties and DRY). Some force or forbid chosen tokens (logit bias). Some steer how random the whole text feels (typical sampling, mirostat, and XTC). These matter for local coding agents because a small model is much more likely to fall into loops, and because tool calls and code need different settings than chat.

The lesson implements each sampler from the rule in the pinned llama.cpp source (the exact version this course uses), assembles them into a SamplerChain class (Project 2), and ends with criteria for choosing settings by task. Afterwards you can read a llama.cpp sampler chain, map its settings to the OpenAI API parameter names, and pick settings for a given job.

Sampler	Idea in plain words
Repetition, presence, frequency penalty	Lower the scores of tokens that already appeared
Logit bias	Add a fixed number to chosen scores, or ban the token
Locally typical	Keep tokens whose surprise is close to the average surprise
Mirostat	A feedback loop that holds the average surprise near a target
DRY	Penalize tokens that would extend a sequence that already occurred
XTC	Sometimes remove the most likely tokens
Sampler chain	The fixed order in which an engine applies all of them

Most programs reuse the 6-token vocabulary the cat sat mat dog ran with logits 2.0, 1.0, 0.5, 0.1, -0.5, -1.0 from the previous lesson. A logit is the raw score a model gives each token, and softmax turns the scores into probabilities that add up to 1.

Example 1: the repetition penalty

Models tend to repeat themselves. A phrase that was just written makes the same phrase likely again, and the loop feeds itself. The simplest cure is a repetition penalty: lower the score of every token that appears in a recent window of the text (the last few tokens the model produced).

The original method divided the logit by a penalty factor above 1. This works for a positive logit: 2.0 divided by 1.5 is 1.33, which is lower. It fails for a negative logit: -1.0 divided by 1.5 is -0.67, which is higher. An already unlikely token becomes more likely, the opposite of what a penalty should do.

The llama.cpp rule fixes this. It divides a positive logit and multiplies a logit that is zero or negative. The program applies both rules (called fixed and naive in the code) to the tokens the, dog, and ran with a penalty of 1.5. Look at the rows for dog and ran, where the two rules differ.

Java

public class RepeatPenalty {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        boolean[] inWindow = {true, false, false, false, true, true};   // tokens seen in the last n generated tokens
        double penalty = 1.5;
        double[] fixed = logits.clone(), naive = logits.clone();
        for (int i = 0; i < logits.length; i++) {
            if (!inWindow[i]) continue;
            fixed[i] = logits[i] > 0 ? logits[i] / penalty : logits[i] * penalty;   // llama.cpp rule
            naive[i] = logits[i] / penalty;                                          // dividing every logit
        }
        double[] p0 = softmax(logits), p1 = softmax(fixed), p2 = softmax(naive);
        System.out.println("token  seen  logit  after  naive-after  p before  p after  p naive");
        for (int i = 0; i < words.length; i++)
            System.out.printf("%-5s  %-5s %5.2f  %5.2f  %11.2f  %8.3f  %7.3f  %7.3f%n", words[i], inWindow[i] ? "yes" : "no", logits[i], fixed[i], naive[i], p0[i], p1[i], p2[i]);
    }
}

Output

token  seen  logit  after  naive-after  p before  p after  p naive
the    yes    2.00   1.33         1.33     0.534    0.381    0.361
cat    no     1.00   1.00         1.00     0.196    0.273    0.259
sat    no     0.50   0.50         0.50     0.119    0.166    0.157
mat    no     0.10   0.10         0.10     0.080    0.111    0.105
dog    yes   -0.50  -0.75        -0.33     0.044    0.047    0.068
ran    yes   -1.00  -1.50        -0.67     0.027    0.022    0.049

With the llama.cpp rule, the falls from 0.534 to 0.381, and the unseen tokens gain probability because the total has to stay at 1. The negative logits of dog and ran move down (-0.50 to -0.75 and -1.00 to -1.50), so their probabilities fall slightly.

Plain division moves those two logits up (-0.33 and -0.67). The probability of ran then rises from 0.027 to 0.049, which is the wrong direction for a penalty.

A penalty of 1.0 changes nothing, and that is the default. The window is set by --repeat-last-n (default 64, and 0 disables it). Tokens older than the window are not penalized.

Example 2: the presence penalty

The repetition penalty scales a logit. A second kind of penalty subtracts a fixed amount instead. The presence penalty subtracts the same amount from the logit of every token that occurred at least once in the window. How often the token occurred does not matter: once or ten times costs the same.

The program applies a presence penalty of 0.7 to the history the the the cat dog. The count column shows how often each token occurred. Look at the, which occurred three times, and cat, which occurred once.

Java

public class PresencePenalty {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        int[] history = {0, 0, 0, 1, 4};                         // the the the cat dog
        double presence = 0.7;
        int[] count = new int[words.length];
        for (int id : history) count[id]++;
        double[] after = logits.clone();
        for (int i = 0; i < words.length; i++) if (count[i] > 0) after[i] -= presence;   // once, whatever the count
        double[] p0 = softmax(logits), p1 = softmax(after);
        System.out.println("token  count  logit  after   p before  p after");
        for (int i = 0; i < words.length; i++)
            System.out.printf("%-5s  %5d  %5.2f  %5.2f   %8.3f  %7.3f%n", words[i], count[i], logits[i], after[i], p0[i], p1[i]);
    }
}

Output

token  count  logit  after   p before  p after
the        3   2.00   1.30      0.534    0.435
cat        1   1.00   0.30      0.196    0.160
sat        0   0.50   0.50      0.119    0.195
mat        0   0.10   0.10      0.080    0.131
dog        1  -0.50  -1.20      0.044    0.036
ran        0  -1.00  -1.00      0.027    0.044

The tokens the, cat, and dog occurred, so each loses exactly 0.70. The token the falls from 2.00 to 1.30 although it occurred three times, and cat falls from 1.00 to 0.30 after one occurrence. The tokens that did not occur keep their logits. Their probabilities rise because softmax divides by a smaller total: sat rises from 0.119 to 0.195.

A presence penalty pushes the text toward tokens it has not used yet. It does not distinguish a word used once from a word used many times. The frequency penalty, in the next example, does. The OpenAI API documents presence_penalty with this subtraction, and llama.cpp uses the flag --presence-penalty (default 0, off).

Example 3: the frequency penalty

The frequency penalty subtracts count * frequency from a token's logit. A token used three times loses three times as much as a token used once, so the more a word is used, the harder it is to use again.

The program applies a frequency penalty of 0.5 to the same history the the the cat dog. The subtracted column shows the amount taken from each token.

Java

public class FrequencyPenalty {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        int[] history = {0, 0, 0, 1, 4};                         // the the the cat dog
        double frequency = 0.5;
        int[] count = new int[words.length];
        for (int id : history) count[id]++;
        double[] after = logits.clone();
        for (int i = 0; i < words.length; i++) after[i] -= count[i] * frequency;   // grows with the count
        double[] p0 = softmax(logits), p1 = softmax(after);
        System.out.println("token  count  subtracted  logit  after   p before  p after");
        for (int i = 0; i < words.length; i++)
            System.out.printf("%-5s  %5d  %10.2f  %5.2f  %5.2f   %8.3f  %7.3f%n", words[i], count[i], count[i] * frequency, logits[i], after[i], p0[i], p1[i]);
    }
}

Output

token  count  subtracted  logit  after   p before  p after
the        3        1.50   2.00   0.50      0.534    0.243
cat        1        0.50   1.00   0.50      0.196    0.243
sat        0        0.00   0.50   0.50      0.119    0.243
mat        0        0.00   0.10   0.10      0.080    0.163
dog        1        0.50  -0.50  -1.00      0.044    0.054
ran        0        0.00  -1.00  -1.00      0.027    0.054

The subtraction follows the count. The token the occurred three times and loses 3 * 0.5 = 1.50, while cat and dog occurred once and lose 0.50. After the penalty, the, cat, and sat all have a logit of 0.50, so they share the top probability equally (0.243 each).

The OpenAI API documents frequency_penalty with this subtraction, and llama.cpp uses --frequency-penalty (default 0, off).

Both penalties can be negative in the OpenAI API, which rewards repetition instead of punishing it. In llama.cpp all penalties apply only to tokens inside the window of the last repeat_last_n tokens.

Example 4: the three penalties on the same history

The three penalties can be used together, and llama.cpp applies them in one step. For each token in the window it first applies the repeat rule (multiply or divide), and then subtracts count * frequency + presence.

The program implements that step with a window length. It prints the probabilities under each penalty separately and then all three together, with the history the the the cat dog. Compare each row with the none row at the top.

Java

import java.util.ArrayList;
import java.util.List;

public class ThreePenalties {
    // the penalty step of llama.cpp: the last `lastN` tokens are counted, then repeat, frequency, and presence are applied
    static double[] apply(double[] logits, List<Integer> history, int lastN, double repeat, double freq, double present) {
        double[] out = logits.clone();
        int[] count = new int[logits.length];
        for (int i = Math.max(0, history.size() - lastN); i < history.size(); i++) count[history.get(i)]++;
        for (int i = 0; i < out.length; i++) {
            if (count[i] == 0) continue;
            out[i] = out[i] <= 0 ? out[i] * repeat : out[i] / repeat;
            out[i] -= count[i] * freq + present;
        }
        return out;
    }

    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        List<Integer> history = new ArrayList<>(List.of(0, 0, 0, 1, 4));          // the the the cat dog
        String[] names = {"none", "repeat 1.3", "frequency 0.5", "presence 0.7", "all three"};
        double[][] settings = {{1, 0, 0}, {1.3, 0, 0}, {1, 0.5, 0}, {1, 0, 0.7}, {1.3, 0.5, 0.7}};
        System.out.printf("%-14s", "probability of");
        for (String w : words) System.out.printf("%7s", w);
        System.out.println();
        for (int s = 0; s < settings.length; s++) {
            double[] p = softmax(apply(logits, history, 64, settings[s][0], settings[s][1], settings[s][2]));
            System.out.printf("%-14s", names[s]);
            for (double v : p) System.out.printf("%7.3f", v);
            System.out.println();
        }
    }
}

Output

probability of    the    cat    sat    mat    dog    ran
none            0.534  0.196  0.119  0.080  0.044  0.027
repeat 1.3      0.445  0.206  0.158  0.106  0.050  0.035
frequency 0.5   0.243  0.243  0.243  0.163  0.054  0.054
presence 0.7    0.435  0.160  0.195  0.131  0.036  0.044
all three       0.116  0.146  0.371  0.249  0.035  0.083

The repeat penalty of 1.3 lowers the logits of the three tokens in the history (the, cat, dog), and the probability of the falls from 0.534 to 0.445.

The frequency penalty of 0.5 hits the hardest. Its logit falls to 0.5, which equals the logit of cat after its own penalty and that of sat, so the first three tokens get the same probability, 0.243. The presence penalty treats the and cat alike, since both occurred.

With all three active the order of the tokens changes. The token sat becomes the most likely at 0.371, although it had the third-highest logit at the start. Penalties are strong, and even small values can change the ranking.

Example 5: the window, and why code legitimately repeats identifiers

Penalties were designed for prose. Natural text avoids repeating a word within a sentence. Code does not work that way. An identifier (a variable name) that appears in one statement must appear again in the next one, or the program breaks.

The program takes the generated tokens count = count + 1 ; count = count + 1 ; count = and a model that is sure the next token is count (logit 3.0). It applies a repeat penalty at several strengths and window sizes. Watch what happens to the probabilities of total and i, which would be wrong names here.

Java

import java.util.Arrays;
import java.util.List;

public class CodeRepeats {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        List<String> vocab = List.of("count", "total", "i", "=", "+", "1", ";", "(");
        // generated so far: count = count + 1 ; count = count + 1 ; count =
        String[] text = "count = count + 1 ; count = count + 1 ; count =".split(" ");
        // the model is sure that the identifier repeats: "count = count + 1" is a normal statement
        double[] logits = {3.0, 1.0, 0.8, -1.0, 0.0, 0.4, -1.0, 0.5};

        for (double[] setting : new double[][] {{1.0, 64}, {1.1, 64}, {1.3, 64}, {1.5, 64}, {1.5, 1}}) {
            double[] l = logits.clone();
            boolean[] seen = new boolean[vocab.size()];
            for (int i = (int) Math.max(0, text.length - setting[1]); i < text.length; i++) seen[vocab.indexOf(text[i])] = true;
            for (int i = 0; i < l.length; i++) if (seen[i]) l[i] = l[i] > 0 ? l[i] / setting[0] : l[i] * setting[0];
            double[] p = softmax(l);
            System.out.printf("repeat penalty %.1f, window %2d: P(count) %.3f  P(total) %.3f  P(i) %.3f%n", setting[0], (int) setting[1], p[0], p[1], p[2]);
        }
    }
}

Output

repeat penalty 1.0, window 64: P(count) 0.672  P(total) 0.091  P(i) 0.074
repeat penalty 1.1, window 64: P(count) 0.612  P(total) 0.109  P(i) 0.089
repeat penalty 1.3, window 64: P(count) 0.514  P(total) 0.139  P(i) 0.114
repeat penalty 1.5, window 64: P(count) 0.442  P(total) 0.162  P(i) 0.133
repeat penalty 1.5, window  1: P(count) 0.675  P(total) 0.091  P(i) 0.075

Without a penalty count has probability 0.672. A penalty of 1.1 lowers it to 0.612, 1.3 to 0.514, and 1.5 to 0.442. Meanwhile total and i rise from 0.091 and 0.074 to 0.162 and 0.133. The model is pushed toward inventing a new variable name, which breaks the code.

The last row shows that the window decides which tokens count as repeated. With a window of 1, only the last token (=) is penalized, and count keeps 0.675.

The llama.cpp default is a penalty of 1.0, which is off. A task whose correct output repeats tokens (code, JSON keys, quoted strings, tool-call syntax) gains nothing from a repetition penalty and can lose accuracy. For such tasks, DRY (Example 9) targets long repeated sequences without touching single tokens.

Example 6: logit bias

Sometimes the need is more direct: a token must never appear, or one answer must be favored. A logit bias adds a number to the logit of chosen token ids before anything else runs. A large negative number bans a token, and a positive number boosts it.

The program bans the with negative infinity, boosts mat by 2.0, and applies -100 to dog. Compare the probabilities before and after. Pay attention to dog, which is affected differently from the.

Java

public class LogitBias {
    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat", "dog", "ran"};
        double[] logits = {2.0, 1.0, 0.5, 0.1, -0.5, -1.0};
        double[] biased = logits.clone();
        biased[0] += Double.NEGATIVE_INFINITY;      // ban "the": llama-server accepts false as the bias for this
        biased[3] += 2.0;                           // boost "mat" by 2.0
        biased[4] += -100;                          // a bias of -100 makes "dog" almost impossible, but not exactly
        double[] before = softmax(logits), after = softmax(biased);
        for (int i = 0; i < words.length; i++)
            System.out.printf("%-4s before %.4f  after %.4f%n", words[i], before[i], after[i]);
    }
}

Output

the  before 0.5341  after 0.0000
cat  before 0.1965  after 0.2107
sat  before 0.1192  after 0.1278
mat  before 0.0799  after 0.6330
dog  before 0.0438  after 0.0000
ran  before 0.0266  after 0.0285

The banned token has probability 0, and the probabilities of the others rise to fill the gap. The token mat jumps from 0.080 to 0.633, because its logit gained 2.0 and the top competitor was removed.

The bias of -100 on dog gives 0.0000 after rounding, but it is not exactly zero. Only negative infinity removes a token completely.

The llama.cpp command-line flag is --logit-bias TOKEN_ID(+/-)BIAS, for example -l 15-100. The server accepts logit_bias as an array of [token, bias] pairs or as an object, and false as the bias bans the token. In the chain, the bias runs before every other sampler.

Typical uses are:

banning tokens that the format never allows;
boosting the first token of a required answer, such as yes or no;
with ignore_eos, suppressing end-of-generation tokens (the server implements ignore_eos as a bias on those tokens).

Example 7: locally typical sampling

Top-p and top-k keep the most probable tokens. Typical sampling starts from a different observation. Fluent human text carries about as much information per token as the average for that situation. A token that is far more likely than usual is dull, and one that is far less likely is odd. Both are less typical.

The information in one token is its surprise, -ln p (the negative natural logarithm of its probability). A likely token has low surprise and an unlikely one has high surprise. The average surprise of the whole distribution is its entropy, which the previous lesson introduced. Typical sampling keeps the tokens whose surprise is closest to the entropy.

The program sorts the tokens by the distance |surprise - entropy| and keeps them until their probability sum exceeds typical_p, the rule in the llama.cpp source. It compares the result with top-p for the distribution 0.4, 0.3, 0.2, 0.1. Entropy here is measured in nats, because the code uses the natural logarithm. A nat is the same idea as a bit with a different logarithm base.

Java

import java.util.ArrayList;
import java.util.List;

public class Typical {
    // keep the tokens whose surprise -ln p is closest to the entropy, until their probability exceeds typicalP
    static List<Integer> typical(double[] p, double typicalP) {
        double h = 0;
        for (double v : p) h -= v * Math.log(v);
        double entropy = h;
        Integer[] order = new Integer[p.length];
        for (int i = 0; i < p.length; i++) order[i] = i;
        java.util.Arrays.sort(order, (a, b) -> Double.compare(Math.abs(-Math.log(p[a]) - entropy), Math.abs(-Math.log(p[b]) - entropy)));
        List<Integer> kept = new ArrayList<>();
        double run = 0;
        for (int id : order) { kept.add(id); run += p[id]; if (run > typicalP) break; }
        return kept;
    }

    static List<Integer> topP(double[] p, double limit) {       // p is already sorted by decreasing probability
        List<Integer> kept = new ArrayList<>();
        double run = 0;
        for (int i = 0; i < p.length; i++) { kept.add(i); run += p[i]; if (run >= limit) break; }
        return kept;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String[] words = {"the", "cat", "sat", "mat"};
        double[] p = {0.4, 0.3, 0.2, 0.1};
        double entropy = 0;
        for (double v : p) entropy -= v * Math.log(v);
        System.out.printf("entropy %.3f nats%n", entropy);
        for (int i = 0; i < p.length; i++)
            System.out.printf("%-4s p %.1f  surprise %.3f  distance from entropy %.3f%n", words[i], p[i], -Math.log(p[i]), Math.abs(-Math.log(p[i]) - entropy));
        for (double limit : new double[] {0.45, 0.85}) {
            StringBuilder typ = new StringBuilder(), top = new StringBuilder();
            for (int id : typical(p, limit)) typ.append(words[id]).append(' ');
            for (int id : topP(p, limit)) top.append(words[id]).append(' ');
            System.out.printf("limit %.2f: typical keeps [%s], top-p keeps [%s]%n", limit, typ.toString().trim(), top.toString().trim());
        }
    }
}

Output

entropy 1.280 nats
the  p 0.4  surprise 0.916  distance from entropy 0.364
cat  p 0.3  surprise 1.204  distance from entropy 0.076
sat  p 0.2  surprise 1.609  distance from entropy 0.330
mat  p 0.1  surprise 2.303  distance from entropy 1.023
limit 0.45: typical keeps [cat sat], top-p keeps [the cat]
limit 0.85: typical keeps [cat sat the], top-p keeps [the cat sat]

The distribution (0.4, 0.3, 0.2, 0.1) has entropy 1.280 nats. The token cat has a surprise of 1.204, very close to that value, so it is first in the typical order. The token the has a surprise of 0.916, a distance of 0.364.

With a limit of 0.45, typical keeps cat and sat and drops the, the most likely token. Top-p keeps the and cat. With a limit of 0.85 both keep three tokens, but not the same three. The sampler is off at typical_p = 1.0, the default.

Example 8: mirostat, a feedback loop on surprise

Typical sampling works one step at a time. Mirostat looks at the whole text. It aims at a target average surprise (tau, in bits) over many tokens, and uses a feedback loop to hold it there. A thermostat is a good picture: it measures the room, compares it with the target, and adjusts the heating. The mapping is loose, because here the measured quantity is the surprise of each chosen token.

The loop works like this. At each step it removes tokens whose surprise exceeds a limit mu, samples from the rest, and measures the surprise of the chosen token.

If the observed surprise is above tau, it lowers mu, which makes later steps more conservative.
If the observed surprise is below tau, it raises mu, which allows more adventurous tokens.

The update is mu = mu - eta * (observed - tau), with eta the learning rate (0.1 by default), and mu starts at 2 * tau. A learning rate is the size of each correction. This is the mirostat 2 rule from the llama.cpp source.

The program runs it on made-up flat distributions over 50 tokens whose sharpness changes at every step, so that the average surprise of plain sampling is about 5 bits. It runs with tau = 2.0 and tau = 3.0, printing the first four steps of each, and then runs plain sampling for comparison.

Java

import java.util.Arrays;
import java.util.Random;

public class Mirostat {
    // a made-up next-token distribution that changes at every step: 50 tokens with random Zipf-like logits
    static double[] logitsFor(Random r) {
        double[] l = new double[50];
        double sharp = 0.3 + 0.9 * r.nextDouble();                 // fairly flat distributions with different sharpness
        for (int i = 0; i < l.length; i++) l[i] = -sharp * Math.log(i + 1) + 0.3 * r.nextGaussian();
        return l;
    }

    static double[] softmax(double[] logits) {
        double max = Double.NEGATIVE_INFINITY, sum = 0;
        for (double l : logits) max = Math.max(max, l);
        double[] p = new double[logits.length];
        for (int i = 0; i < p.length; i++) { p[i] = Math.exp(logits[i] - max); sum += p[i]; }
        for (int i = 0; i < p.length; i++) p[i] /= sum;
        return p;
    }

    static int draw(double[] p, Random r) {
        double u = r.nextDouble() * Arrays.stream(p).sum(), run = 0;
        for (int i = 0; i < p.length; i++) { run += p[i]; if (u < run) return i; }
        return p.length - 1;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        for (double tau : new double[] {2.0, 3.0}) {
            Random data = new Random(1), rng = new Random(2);
            double mu = 2 * tau, eta = 0.1, total = 0;               // mu starts at 2 * tau, as in llama.cpp
            int steps = 5000;
            for (int step = 0; step < steps; step++) {
                double[] p = softmax(logitsFor(data));               // already sorted by decreasing probability
                int keep = 0;
                while (keep < p.length && -Math.log(p[keep]) / Math.log(2) <= mu) keep++;   // drop tokens with surprise > mu
                keep = Math.max(keep, 1);
                double[] q = softmax(Arrays.stream(Arrays.copyOf(p, keep)).map(Math::log).toArray());
                int id = draw(q, rng);
                double surprise = Math.abs(Math.log(q[id]) / Math.log(2));
                mu -= eta * (surprise - tau);                        // feedback: too surprising lowers mu
                total += surprise;
                if (step < 4) System.out.printf("tau %.1f step %d: kept %2d tokens, surprise %.2f bits, mu now %.2f%n", tau, step, keep, surprise, mu);
            }
            System.out.printf("tau %.1f: average surprise over %d steps = %.2f bits%n", tau, steps, total / steps);
        }
        Random data = new Random(1), rng = new Random(2);
        double plain = 0;
        for (int step = 0; step < 5000; step++) {
            double[] p = softmax(logitsFor(data));
            plain += -Math.log(p[draw(p, rng)]) / Math.log(2);
        }
        System.out.printf("plain sampling: average surprise = %.2f bits%n", plain / 5000);
    }
}

Output

tau 2.0 step 0: kept  3 tokens, surprise 2.19 bits, mu now 3.98
tau 2.0 step 1: kept  4 tokens, surprise 2.50 bits, mu now 3.93
tau 2.0 step 2: kept  3 tokens, surprise 1.01 bits, mu now 4.03
tau 2.0 step 3: kept  1 tokens, surprise 0.00 bits, mu now 4.23
tau 2.0: average surprise over 5000 steps = 2.00 bits
tau 3.0 step 0: kept 12 tokens, surprise 4.10 bits, mu now 5.89
tau 3.0 step 1: kept  8 tokens, surprise 3.83 bits, mu now 5.81
tau 3.0 step 2: kept  8 tokens, surprise 2.63 bits, mu now 5.84
tau 3.0 step 3: kept 11 tokens, surprise 4.28 bits, mu now 5.72
tau 3.0: average surprise over 5000 steps = 3.00 bits
plain sampling: average surprise = 4.95 bits

Plain sampling averages 4.95 bits per token. With tau = 2.0 the loop settles at an average of 2.00 bits, and with tau = 3.0 at 3.00 bits. The feedback loop hits its target even though the distributions keep changing.

The first lines show the mechanism in the first run. A surprise of 2.19 bits is above tau = 2.0, so mu drops from 4.00 to 3.98. A surprise of 1.01 is below tau, so mu rises from 3.93 to 4.03. A surprise of 0 raises it to 4.23. A step with high surprise pulls mu down, and a step with low surprise pushes it up. The kept column shows the number of tokens that stay at each step, which varies with the shape of the distribution.

In llama.cpp, --mirostat 1 or 2 enables it, --mirostat-ent sets tau (default 5.0), and --mirostat-lr sets eta. Mirostat replaces the filters: the chain becomes temp -> mirostat-v2, and top-k, top-p, and the others are not used.

Example 9: DRY, penalizing repeated sequences

The penalties in Examples 1 to 3 punish single tokens, which hurts code. What actually goes wrong in a loop is that a long sequence repeats, such as the same sentence again and again. DRY (Don't Repeat Yourself) targets exactly that. It penalizes a token only when choosing it would extend a sequence that already occurred in the text.

For each candidate token, DRY checks the end of the history. It finds how long a suffix of the history already appeared earlier and was followed by this candidate. If that length reaches allowed_length, it subtracts multiplier * base^(length - allowed_length) from the logit.

Take a real case. The text so far ends with the cat sat on the. Earlier the same five tokens were followed by mat. Choosing mat again would extend a repeat of 5 tokens, so mat is penalized. Repeating a single word costs nothing, and the penalty grows exponentially with the length of the loop.

The program implements this rule in a reduced form and prints the penalty for three candidates, plus a table of how the penalty grows with the repeat length.

Java

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class Dry {
    // the length of the longest suffix of the history that already occurred earlier and was followed by `candidate`
    static int repeatLength(List<Integer> history, int candidate) {
        int n = history.size(), best = 0;
        for (int end = 0; end < n - 1; end++) {                  // an earlier occurrence ends at position `end`
            if (history.get(end + 1) != candidate) continue;
            int len = 0;
            while (len <= end && history.get(end - len).equals(history.get(n - 1 - len))) len++;
            best = Math.max(best, len);
        }
        return best;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        List<String> vocab = List.of("the", "cat", "sat", "on", "mat", "dog", ".");
        String[] text = "the cat sat on the mat . the cat sat on the".split(" ");
        List<Integer> history = new ArrayList<>();
        for (String t : text) history.add(vocab.indexOf(t));
        double multiplier = 0.8, base = 1.75;
        int allowed = 2;
        System.out.println("history: " + String.join(" ", text));
        for (String word : new String[] {"mat", "dog", "cat"}) {
            int len = repeatLength(history, vocab.indexOf(word));
            double penalty = len >= allowed ? multiplier * Math.pow(base, len - allowed) : 0;
            System.out.printf("candidate %-3s extends a repeat of %d tokens: logit penalty %.3f%n", word, len, penalty);
        }
        System.out.println("penalty by repeat length: " + Arrays.toString(new String[] {
            String.format("2 -> %.2f", multiplier), String.format("4 -> %.2f", multiplier * Math.pow(base, 2)), String.format("8 -> %.2f", multiplier * Math.pow(base, 6))}));
    }
}

Output

history: the cat sat on the mat . the cat sat on the
candidate mat extends a repeat of 5 tokens: logit penalty 4.288
candidate dog extends a repeat of 0 tokens: logit penalty 0.000
candidate cat extends a repeat of 1 tokens: logit penalty 0.000
penalty by repeat length: [2 -> 0.80, 4 -> 2.45, 8 -> 22.98]

After the cat sat on the, the candidate mat would extend a repeat of 5 tokens (the cat sat on the was followed by mat before). Its penalty is 0.8 * 1.75^(5 - 2) = 0.8 * 5.36 = 4.288. The candidate dog extends nothing, and cat extends a repeat of only 1 token, below the allowed length of 2, so neither is penalized.

The last line shows how the penalty grows with the length of the repeat: 0.80 at length 2, 2.45 at 4, and 22.98 at 8. A penalty of 22.98 on a logit is enough to break any loop.

This version scans the whole history and compares token ids directly. llama.cpp adds three things:

sequence breakers (by default the tokens for newline, :, a double quote, and *) that end a repeat;
a window dry_penalty_last_n (default 64);
a linear-time algorithm, so the scan stays fast on long histories.

The parameters are --dry-multiplier (default 0, which disables DRY), --dry-base (1.75), and --dry-allowed-length (2). The value 0.8 in the program is an example and not a default.

Example 10: XTC, excluding top choices

All the samplers so far make the likely tokens more likely or keep them. XTC ("exclude top choices") does the opposite on purpose. It works against the habit of always taking the likeliest words, which makes text predictable and generic.

With probability xtc_probability per step, XTC finds the tokens with probability at least xtc_threshold. It then removes all of them except the least likely one among them. The model still has the plausible tokens, but the very top choices are gone for that step.

The program applies the rule from the source to sorted probabilities. First it shows one application with probability 1 and threshold 0.1 on the distribution 0.50, 0.30, 0.10, 0.06, 0.04. Then it measures how often the top token is chosen over 10,000 draws with XTC off and with XTC on at 50 percent.

Java

import java.util.Random;

public class Xtc {
    // with probability `probability`: among tokens with p >= threshold, drop all but the least likely of them
    static double[] apply(double[] p, double probability, double threshold, Random rng) {
        if (rng.nextDouble() > probability) return p;           // p is sorted by decreasing probability
        int last = 0;
        while (last + 1 < p.length && p[last + 1] >= threshold) last++;
        if (p[0] < threshold || last == 0) return p;
        double[] out = new double[p.length];
        double sum = 0;
        for (int i = last; i < p.length; i++) { out[i] = p[i]; sum += p[i]; }
        for (int i = 0; i < out.length; i++) out[i] /= sum;
        return out;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        double[] p = {0.50, 0.30, 0.10, 0.06, 0.04};
        double[] once = apply(p, 1.0, 0.1, new Random(1));
        System.out.println("before:           " + show(p));
        System.out.println("XTC (p=1, t=0.1): " + show(once));

        Random rng = new Random(3);
        int[] topChosen = new int[2];
        for (int i = 0; i < 2; i++) {
            double prob = i == 0 ? 0.0 : 0.5;
            for (int draw = 0; draw < 10_000; draw++) {
                double[] q = apply(p, prob, 0.1, rng);
                double u = rng.nextDouble(), run = 0;
                int id = q.length - 1;
                for (int j = 0; j < q.length; j++) { run += q[j]; if (u < run) { id = j; break; } }
                if (id == 0) topChosen[i]++;
            }
        }
        System.out.println("share of draws that pick the top token, XTC off: " + topChosen[0] / 10_000.0);
        System.out.println("share of draws that pick the top token, XTC on at 50%: " + topChosen[1] / 10_000.0);
    }

    static String show(double[] v) {
        StringBuilder sb = new StringBuilder();
        for (double x : v) sb.append(String.format("%.3f ", x));
        return sb.toString().trim();
    }
}

Output

before:           0.500 0.300 0.100 0.060 0.040
XTC (p=1, t=0.1): 0.000 0.000 0.500 0.300 0.200
share of draws that pick the top token, XTC off: 0.4955
share of draws that pick the top token, XTC on at 50%: 0.2528

With probability 1 and threshold 0.1, the tokens with 0.50, 0.30, and 0.10 are all at or above the threshold. The first two are removed and the least likely of the three, with 0.10, stays. It and the two tokens below it are renormalized to 0.5, 0.3, and 0.2.

In the second test XTC is active in half of the steps. The top token is chosen in 49.6 percent of the draws without XTC, and in 25.3 percent with it. That is about 0.5 * 0.5: XTC has to be active (probability 0.5) and then the top token is gone, so only the other half of the steps can still pick it, at its 0.5 probability.

XTC does nothing when only one token passes the threshold, because the only candidate is the one it keeps. It is off by default (--xtc-probability 0). The source also skips it when the threshold is above 0.5, because at most one token can then reach the threshold.

Example 11: llama.cpp's sampler chain, `--samplers`, and the API mapping

An engine does not apply one sampler. It builds a chain of samplers from all the parameters, in a fixed order, and runs each over the logits in turn. The sampler chain log line prints the chain.

Without a mirostat, the order in common/common.h is:

logit bias (when set)
penalties
DRY
top-n-sigma
top-k
typical
top-p
min-p
XTC
temperature
a final dist sampler that draws the token

Top-n-sigma is one sampler this course has not covered separately. It keeps tokens whose logit is within n standard deviations of the top logit. A ? in front of a name marks a sampler that its parameters disable.

The commands below use the tiny random-weight model, because the chain does not depend on the weights. The first turns on every sampler. The second turns on mirostat. The third sets a logit bias.

Terminal

llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "Hi" -n 2 -no-cnv --seed 7 \
  --repeat-penalty 1.1 --presence-penalty 0.1 --dry-multiplier 0.8 --xtc-probability 0.5 \
  --typical 0.9 --top-n-sigma 2 2>&1 | grep "sampler chain"
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "Hi" -n 2 -no-cnv --seed 7 \
  --mirostat 2 --mirostat-ent 3 --temp 0.7 2>&1 | grep "sampler chain"
llama-completion -m tiny-random-qwen2vocab-f32.gguf -p "Hi" -n 2 -no-cnv --seed 7 \
  -l 15-100 2>&1 | grep "sampler chain"

Representative output (from the pinned llama.cpp build; timestamps removed):

Output

sampler chain: logits -> penalties -> dry -> top-n-sigma -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
sampler chain: logits -> temp -> mirostat-v2
sampler chain: logits -> logit-bias -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist

The first chain has every sampler enabled, so no name has a ?. The second shows that mirostat replaces the filters. The third shows the logit bias at the front, with the unused samplers marked.

The --samplers flag (a list separated by ;) replaces the default order for llama-completion and the server. The flag --sampler-seq takes a short letter sequence instead.

The server accepts the OpenAI parameter names and its own. The table lists the sampling fields in tools/server/server-schema.cpp and the matching OpenAI chat-completions parameters. The ranges in the OpenAI column come from OpenAI's API reference and are not checked here.

Concept	OpenAI parameter	llama-server field	Notes
Randomness	`temperature` (0 to 2)	`temperature`	0 means greedy in llama-server
Nucleus	`top_p` (0 to 1)	`top_p`	1.0 disables; default 0.95
Top-k	none	`top_k`	0 disables; default 40
Min-p	none	`min_p`	0 disables; default 0.05
Output limit	`max_tokens`, `max_completion_tokens`	`n_predict`, with both names as aliases	-1 means no limit
Stop strings	`stop`	`stop`	The stop string is not in the output
Seed	`seed`	`seed`	-1 means random
Frequency penalty	`frequency_penalty` (-2.0 to 2.0)	`frequency_penalty`	Same subtraction
Presence penalty	`presence_penalty` (-2.0 to 2.0)	`presence_penalty`	Same subtraction
Repetition penalty	none	`repeat_penalty`, `repeat_last_n`	Multiply or divide, then window
Logit bias	`logit_bias` (-100 to 100)	`logit_bias`	Also accepts pairs and `false`
Token probabilities	`logprobs`	`n_probs`, with alias `logprobs`
Others	none	`typical_p`, `mirostat`, `mirostat_tau`, `mirostat_eta`, `dry_`, `xtc_`, `top_n_sigma`, `dynatemp_*`, `samplers`

Example 12: Project 2, a SamplerChain with a builder

This project combines the pieces into one Java class. Each sampler becomes a stage: a function that takes the logits and the history and edits the logits. The chain runs the stages in order and ends with a seeded random draw. The builder is a helper object where you call setters in any order (topK(40).temperature(0.8)) and then call build().

The design has these rules:

The builder always builds the stages in the llama.cpp order: logit bias (only when set), penalties, top-k, top-p, min-p, temperature, and dist, whatever order the setters were called in.
Each stage edits a copy of the logits, and a removed token gets negative infinity.
Temperature at or below 0 keeps only the largest logit.
The final dist stage draws with a seeded Random.
The describe method prints the chain like llama.cpp, with ? for disabled stages.

The main method runs five chains on the character bigram model of lesson 5. The model's logits are the log of the counts in the row of the last character, and a zero count gives negative infinity (so that character can never be chosen). The program prints each text with the average number of candidates left per step, and ends with three describe lines.

Java

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.Map;
import java.util.Random;
import java.util.TreeMap;
import java.util.function.BiConsumer;

public class SamplerChain {
    record Stage(String name, BiConsumer<double[], List<Integer>> fn) {}

    private final List<Stage> stages;
    private final Random rng;
    int candidates;                                            // tokens left after the filters, for the last sample() call

    private SamplerChain(List<Stage> stages, long seed) { this.stages = stages; this.rng = new Random(seed); }

    String describe() {
        StringBuilder sb = new StringBuilder("logits");
        for (Stage s : stages) sb.append(" -> ").append(s.name());
        return sb.append(" -> dist").toString();
    }

    int sample(double[] logits, List<Integer> history) {
        double[] l = logits.clone();
        for (Stage s : stages) s.fn().accept(l, history);
        candidates = (int) Arrays.stream(l).filter(Double::isFinite).count();
        double max = Arrays.stream(l).max().getAsDouble(), sum = 0;
        double[] p = new double[l.length];
        for (int i = 0; i < l.length; i++) { p[i] = Math.exp(l[i] - max); sum += p[i]; }
        double u = rng.nextDouble() * sum, run = 0;                // dist: inverse CDF with the seeded generator
        for (int i = 0; i < p.length; i++) { run += p[i]; if (p[i] > 0 && u < run) return i; }
        return Arrays.stream(l).boxed().toList().indexOf(max);
    }

    static Builder builder() { return new Builder(); }

    static double[] softmax(double[] l) {
        double max = Arrays.stream(l).max().getAsDouble(), sum = 0;
        double[] p = new double[l.length];
        for (int i = 0; i < l.length; i++) { p[i] = Math.exp(l[i] - max); sum += p[i]; }
        for (int i = 0; i < l.length; i++) p[i] /= sum;
        return p;
    }

    static Integer[] byProbability(double[] p) {
        Integer[] order = new Integer[p.length];
        for (int i = 0; i < p.length; i++) order[i] = i;
        Arrays.sort(order, (a, b) -> Double.compare(p[b], p[a]));
        return order;
    }

    static class Builder {
        private final Map<Integer, Double> bias = new TreeMap<>();
        private int lastN = 64, topK = 0;
        private double repeat = 1, freq = 0, present = 0, topP = 1, minP = 0, temp = 1;
        private long seed = 0;

        Builder logitBias(int id, double b) { bias.put(id, b); return this; }
        Builder penalties(int lastN, double repeat, double freq, double present) { this.lastN = lastN; this.repeat = repeat; this.freq = freq; this.present = present; return this; }
        Builder topK(int k) { topK = k; return this; }
        Builder topP(double p) { topP = p; return this; }
        Builder minP(double p) { minP = p; return this; }
        Builder temperature(double t) { temp = t; return this; }
        Builder seed(long s) { seed = s; return this; }

        SamplerChain build() {                                 // always the llama.cpp order, whatever the call order was
            List<Stage> st = new ArrayList<>();
            if (!bias.isEmpty()) st.add(new Stage("logit-bias", (l, h) -> bias.forEach((id, b) -> l[id] += b)));
            st.add(new Stage((repeat != 1 || freq != 0 || present != 0 ? "" : "?") + "penalties", (l, h) -> {
                int[] count = new int[l.length];
                for (int i = Math.max(0, h.size() - lastN); i < h.size(); i++) count[h.get(i)]++;
                for (int i = 0; i < l.length; i++) {
                    if (count[i] == 0 || (repeat == 1 && freq == 0 && present == 0)) continue;
                    l[i] = l[i] <= 0 ? l[i] * repeat : l[i] / repeat;
                    l[i] -= count[i] * freq + present;
                }
            }));
            st.add(new Stage((topK > 0 ? "" : "?") + "top-k", (l, h) -> {
                if (topK <= 0 || topK >= l.length) return;
                double[] sorted = l.clone();
                Arrays.sort(sorted);
                double kth = sorted[l.length - topK];
                for (int i = 0; i < l.length; i++) if (l[i] < kth) l[i] = Double.NEGATIVE_INFINITY;
            }));
            st.add(new Stage((topP < 1 ? "" : "?") + "top-p", (l, h) -> {
                if (topP >= 1) return;
                double[] p = softmax(l);
                double run = 0;
                boolean cut = false;
                for (int id : byProbability(p)) {
                    if (cut) l[id] = Double.NEGATIVE_INFINITY;
                    run += p[id];
                    if (run >= topP) cut = true;
                }
            }));
            st.add(new Stage((minP > 0 ? "" : "?") + "min-p", (l, h) -> {
                if (minP <= 0) return;
                double min = Arrays.stream(l).max().getAsDouble() + Math.log(minP);
                for (int i = 0; i < l.length; i++) if (l[i] < min) l[i] = Double.NEGATIVE_INFINITY;
            }));
            st.add(new Stage((temp == 1 ? "?" : "") + "temp", (l, h) -> {
                if (temp > 0) { for (int i = 0; i < l.length; i++) l[i] /= temp; return; }
                int best = 0;                                  // temperature <= 0: keep only the first largest logit
                for (int i = 1; i < l.length; i++) if (l[i] > l[best]) best = i;
                for (int i = 0; i < l.length; i++) if (i != best) l[i] = Double.NEGATIVE_INFINITY;
            }));
            return new SamplerChain(st, seed);
        }
    }

    static final String CORPUS = "the cat sat on the mat. the dog sat on the log. the cat saw the dog. the dog saw the cat. a cat and a dog sat on a mat.";

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        String alphabet = CORPUS.chars().distinct().sorted().collect(StringBuilder::new, StringBuilder::appendCodePoint, StringBuilder::append).toString();
        int n = alphabet.length();
        double[][] counts = new double[n][n];                  // the bigram model of lesson 2
        for (int i = 0; i + 1 < CORPUS.length(); i++) counts[alphabet.indexOf(CORPUS.charAt(i))][alphabet.indexOf(CORPUS.charAt(i + 1))]++;

        String[] names = {"llama.cpp defaults", "exact (T=0)", "low T", "repeat 1.1", "min-p 0.1, T=1.2"};
        SamplerChain[] chains = {
            builder().topK(40).topP(0.95).minP(0.05).temperature(0.8).seed(11).build(),
            builder().temperature(0).seed(11).build(),
            builder().topP(0.95).temperature(0.2).seed(11).build(),
            builder().penalties(64, 1.1, 0, 0).topK(40).topP(0.95).minP(0.05).temperature(0.8).seed(11).build(),
            builder().minP(0.1).temperature(1.2).seed(11).build()};
        for (int c = 0; c < chains.length; c++) {
            List<Integer> history = new ArrayList<>(List.of(alphabet.indexOf('t')));
            long kept = 0;
            for (int step = 0; step < 59; step++) {
                double[] logits = new double[n];
                for (int j = 0; j < n; j++) logits[j] = Math.log(counts[history.get(history.size() - 1)][j]);   // log 0 = -infinity
                history.add(chains[c].sample(logits, history));
                kept += chains[c].candidates;
            }
            StringBuilder text = new StringBuilder();
            for (int id : history) text.append(alphabet.charAt(id));
            System.out.printf("%-20s %s  (%.1f candidates per step)%n", names[c], text, kept / 59.0);
        }
        System.out.println(chains[0].describe());
        System.out.println(chains[1].describe());
        System.out.println(builder().logitBias(3, 2.0).penalties(64, 1.1, 0, 0).build().describe());
    }
}

Output

llama.cpp defaults   the an mathe cathe caw sathe cathe on sat mandon cathe cat.   (3.0 candidates per step)
exact (T=0)          the the the the the the the the the the the the the the the   (1.0 candidates per step)
low T                the at the dog the sathe the sathe the the t. the sathe the   (3.0 candidates per step)
repeat 1.1           the an mathe cathe caw og t maw the sathe mandon cathe cat.   (3.1 candidates per step)
min-p 0.1, T=1.2     the a athe at londog the dog cathe mathe cat t. log on cat.   (3.4 candidates per step)
logits -> ?penalties -> top-k -> top-p -> min-p -> temp -> dist
logits -> ?penalties -> ?top-k -> ?top-p -> ?min-p -> temp -> dist
logits -> logit-bias -> penalties -> ?top-k -> ?top-p -> ?min-p -> ?temp -> dist

The five texts come from the same seed and the same model with different chains. Read them in turn:

At temperature 0 the text is the loop the the the, and the average number of candidates is 1.0.
The default-like chain keeps 3.0 candidates per step on average.
The low-temperature chain mostly takes the likeliest character and varies at some steps.
The repetition penalty changes the text where a character repeats within the last 64 characters.
Min-p 0.1 with temperature 1.2 keeps slightly more candidates (3.4).

The three describe lines at the end show the stage order and the ? marks. With a logit bias the chain starts with logit-bias, and the order stays the same however the builder was called.

The class leaves out typical, mirostat, DRY, and XTC. Each of them fits the same Stage shape: a function from the logits and the history to edited logits. Mirostat needs one more piece of state (mu) and replaces the final dist stage.

Example 13: choosing settings by task, on the same prompt

No setting is correct for every task. The useful question is what one unlikely token costs. A wrong token in a poem is a different word. A wrong token in a JSON tool call is a failed parse. The table lists tasks, the cost of a wrong token, and the settings that follow from it.

Task	Cost of one unlikely token	Settings direction	Reason
Tool-call arguments, JSON	Parse failure, wrong action	Temperature 0 or close to it; no repetition penalties	Keys, quotes, and braces must repeat; constrained decoding (later in this course) removes invalid tokens outright
Code	Compile error, wrong identifier	Low temperature; no repetition penalty; top-p or min-p to cut the tail	Identifiers repeat (Example 5)
Chat answers	Odd wording	Moderate temperature with min-p or top-p; a mild repeat penalty only if loops appear	Variety is acceptable, loops are not
Brainstorming, fiction	Dull text	Higher temperature with min-p to cap the tail; DRY or XTC if text repeats or turns generic	Variety is the goal
Reasoning models	Loops inside long thinking	Use the values the model card gives	Training and evaluation used them

The program applies one preset per row to the same prompt: the code position of Example 5, where the history is count = count + 1 ; count = count + 1 ; count = and the model gives count a logit of 3.0. For each preset it builds the final distribution with the llama.cpp order (penalties, top-k, top-p, min-p, temperature). It prints the probability of the correct token count and the number of candidates left, and draws eight tokens with the same seed.

The values in the presets are examples chosen to show the directions in the table, not recommendations from a model card.

Java

import java.util.Arrays;
import java.util.List;
import java.util.Random;

public class TaskSettings {
    record Preset(String task, double temp, int topK, double topP, double minP, double repeat) {}

    // the final distribution after penalties, top-k, top-p, min-p, and temperature (the llama.cpp order)
    static double[] distribution(double[] logits, List<String> vocab, String[] history, Preset s) {
        double[] l = logits.clone();
        for (int i = 0; i < l.length; i++)
            if (Arrays.asList(history).contains(vocab.get(i))) l[i] = l[i] <= 0 ? l[i] * s.repeat() : l[i] / s.repeat();
        Integer[] order = new Integer[l.length];
        for (int i = 0; i < l.length; i++) order[i] = i;
        Arrays.sort(order, (a, b) -> Double.compare(l[b], l[a]));
        double[] w = new double[l.length];
        if (s.temp() <= 0) { w[order[0]] = 1; return w; }                     // greedy
        double max = l[order[0]], sum = 0, run = 0;
        for (double v : l) sum += Math.exp(v - max);
        double pTop = 1 / sum;                                                 // exp(max - max) / sum
        for (int r = 0; r < l.length; r++) {
            int id = order[r];
            double p = Math.exp(l[id] - max) / sum;
            boolean keep = (s.topK() <= 0 || r < s.topK()) && (r == 0 || run < s.topP()) && p >= s.minP() * pTop;
            if (keep) w[id] = Math.exp((l[id] - max) / s.temp());
            run += p;
        }
        double total = Arrays.stream(w).sum();
        for (int i = 0; i < w.length; i++) w[i] /= total;
        return w;
    }

    public static void main(String[] args) {
        java.util.Locale.setDefault(java.util.Locale.ROOT);
        List<String> vocab = List.of("count", "total", "i", "=", "+", "1", ";", "(");
        String[] history = "count = count + 1 ; count = count + 1 ; count =".split(" ");
        double[] logits = {3.0, 1.0, 0.8, -1.0, 0.0, 0.4, -1.0, 0.5};        // the model of Example 5
        Preset[] presets = {
            new Preset("tool call, JSON", 0, 0, 1, 0, 1),
            new Preset("code", 0.2, 0, 0.95, 0.05, 1),
            new Preset("chat", 0.8, 40, 0.95, 0.05, 1),
            new Preset("chat, repeat 1.3", 0.8, 40, 0.95, 0.05, 1.3),
            new Preset("brainstorming", 1.3, 0, 1, 0.1, 1)};
        for (Preset s : presets) {
            double[] p = distribution(logits, vocab, history, s);
            Random rng = new Random(4);
            StringBuilder draws = new StringBuilder();
            for (int d = 0; d < 8; d++) {
                double u = rng.nextDouble(), run = 0;
                for (int i = 0; i < p.length; i++) { run += p[i]; if (p[i] > 0 && u < run) { draws.append(vocab.get(i)).append(' '); break; } }
            }
            System.out.printf("%-17s P(count) %.3f  candidates %d  draws: %s%n", s.task(), p[0], Arrays.stream(p).filter(v -> v > 0).count(), draws.toString().trim());
        }
    }
}

Output

tool call, JSON   P(count) 1.000  candidates 1  draws: count count count count count count count count
code              P(count) 1.000  candidates 5  draws: count count count count count count count count
chat              P(count) 0.814  candidates 5  draws: count i i count count count count count
chat, repeat 1.3  P(count) 0.629  candidates 6  draws: total 1 1 total count count total i
brainstorming     P(count) 0.715  candidates 3  draws: total i i count count count count total

The tool-call preset is greedy: one candidate, always count. The code preset keeps five candidates after top-p and min-p, but temperature 0.2 multiplies the logit gaps by 5, so count holds a probability of 1.000 after rounding and all eight draws are count.

The chat preset at temperature 0.8 gives count 0.814, and two of the eight draws pick i, a wrong identifier in this position. Adding a repetition penalty of 1.3 lowers count to 0.629 and lets total and 1 in, which shows the cost of Example 5 on a full chain.

The brainstorming preset has the highest temperature but keeps only three candidates, because min-p 0.1 removes the tokens below a tenth of the top probability. The token count still falls to 0.715.

For exact output, low randomness matters more than any filter. For open-ended output, a filter that cuts the tail (min-p or top-p) keeps a high temperature from reaching nonsense tokens. The values for each model family are usually listed in the model card, and How Models Are Trained, and the Variants You Download covers reading one. A coding agent applies the same logic per request: tool calls need a different sampler than a prose summary, and a later lesson on making small local models reliable returns to it.

Interview points

A repetition penalty divides positive logits and multiplies non-positive ones, because plain division would raise negative logits and make unlikely tokens more likely.
A frequency penalty subtracts count * value, and a presence penalty subtracts a fixed value once per token that occurred. llama.cpp applies repeat, frequency, and presence together over the last repeat_last_n tokens.
Repetition penalties hurt tasks whose correct output repeats tokens, such as code, JSON, and tool calls.
A logit bias adds a number to a logit before the other samplers. Negative infinity bans a token, and -100 only makes it very unlikely.
Locally typical sampling keeps tokens whose surprise is close to the entropy, and it can drop the most likely token.
Mirostat 2 keeps the average surprise near tau by adjusting a limit mu with mu -= eta * (observed - tau), and it replaces the other filters in llama.cpp.
DRY penalizes a token by multiplier * base^(length - allowed_length) when it extends a repeated sequence, so single words are free and loops cost exponentially more.
XTC removes all but the least likely of the tokens above a threshold, with a given probability per step, to reduce predictable output.
llama.cpp orders its chain as logit bias, penalties, DRY, top-n-sigma, top-k, typical, top-p, min-p, XTC, temperature, dist. Temperature comes after truncation.
The OpenAI parameters temperature, top_p, frequency_penalty, presence_penalty, logit_bias, seed, stop, and max_tokens exist in llama-server. Top-k, min-p, repeat penalty, mirostat, DRY, and XTC are llama.cpp additions.
Choose sampler settings by the cost of a wrong token: exact output needs low randomness, and creative output tolerates more.

Summary

Repetition penalty: logit <= 0 ? logit * r : logit / r for tokens in the last repeat_last_n tokens (default 64, and a penalty of 1.0 means off).
Frequency and presence penalties: logit -= count * freq + (count > 0 ? presence : 0), both 0 by default.
Logit bias: logit += bias before all other samplers. false or negative infinity bans a token in llama-server.
Typical sampling sorts by |-ln p - entropy| and keeps tokens until the cumulative probability exceeds typical_p (1.0 means off).
Mirostat 2 truncates tokens with surprise above mu, samples, and updates mu toward tau with rate eta. The value mu starts at 2 * tau.
DRY subtracts multiplier * base^(repeat_length - allowed_length). Defaults: multiplier 0 (off), base 1.75, allowed length 2, last 64 tokens, breakers \n, :, ", *.
XTC, with probability xtc_probability, removes tokens at or above xtc_threshold except the least likely of them (default off, threshold 0.1).
The chain order and ? markers appear in the sampler chain log line. With mirostat the chain is temp -> mirostat-v2.
--samplers and --sampler-seq change the order or the set. --logit-bias TOKEN_ID(+/-)BIAS sets a bias.
A SamplerChain is a list of stages over a logit array plus a seeded final draw. The builder fixes the stage order.
Settings follow from the cost of a wrong token, and model cards list the values a model was evaluated with.

Example 1: the repetition penalty

Example 2: the presence penalty

Example 3: the frequency penalty

Example 4: the three penalties on the same history

Example 5: the window, and why code legitimately repeats identifiers

Example 6: logit bias

Example 7: locally typical sampling

Example 8: mirostat, a feedback loop on surprise

Example 9: DRY, penalizing repeated sequences

Example 10: XTC, excluding top choices

Example 11: llama.cpp's sampler chain, --samplers, and the API mapping

Example 12: Project 2, a SamplerChain with a builder

Example 13: choosing settings by task, on the same prompt

Interview points

Summary

Example 11: llama.cpp's sampler chain, `--samplers`, and the API mapping