Temperature and Sampling

Ask a chat assistant the same question twice and the two answers will not match word for word. The facts may be the same, but the wording, the order, and sometimes the content differ. Nothing is broken. This behaviour is built into the way a model writes text.

This lesson explains where that variation comes from. A model does not produce one fixed next word. It produces a list of possible next tokens, each with a probability, and one of them is picked. The picking step is called sampling, and a setting called temperature controls how the pick is made.

Temperature comes up less often than terms such as token or context window. It becomes relevant when you build your own software on top of a model. After this lesson you can explain why answers vary, say what a low and a high temperature do, and choose a sensible value for a given job.

1. Where the setting appears

Most people never touch temperature. In a chat app such as ChatGPT, you type a message and an answer comes back. There is no temperature control on the screen, and at the time of writing the common chat apps do not usually offer one.

The setting shows up in other places. Tools made for developers often show it directly, under exactly that name: a field or slider labelled temperature. When a program sends a request to a model through an API, the interface that lets one program talk to another, temperature is one of the values the program can send along with the text.

So there are two situations:

2. What the setting controls

Temperature controls how predictable the model's answers are.

A predictable model gives nearly the same reply every time it gets the same input. An unpredictable model gives a noticeably different reply each time. Temperature moves the model between those two behaviours.

To see why such a setting can exist at all, look at what a model produces when it writes.

3. The same input does not give the same output

A calculator is deterministic. That word means the same input always gives the same output. Type 2 + 2 and the result is 4 every time.

A large language model used in the normal way is not deterministic. Send it the same text three times and, in most cases, three different texts come back.

flowchart LR
    A["The same input text<br/>sent again and again"] --> B["Output text 1"]
    A --> C["Output text 2"]
    A --> D["Output text 3"]

The diagram shows one input on the left and three separate outputs on the right. Each output is a complete, sensible answer. They differ from each other because of what happens each time the model writes a token.

4. The model produces a list of candidates, not one token

A model writes its answer one token at a time. For each position it looks at all the text so far and works out what could come next. An earlier lesson described this as predicting which token should come next.

The prediction is not a single, final token. It is a score for every token in the model's vocabulary. Each score is a probability: a number that says how likely that token is to be the next one. The probabilities of all tokens add up to 100%.

Take the unfinished sentence "It was a warm summer". The model's list for the next word could start like this:

These numbers are made up for the example. They do not come from any real model. The five rows add up to 94%. The remaining 6% is spread thinly over all the other tokens in the vocabulary, such as "breeze" or "holiday".

One token from this list gets chosen and is added to the text. The choice is random, but it is weighted by the probabilities. "Weighted" means that a token with a higher probability is picked more often. With the numbers above, "day" would be picked about 41 times out of 100, "evening" about 21 times, and "morning" about 7 times.

An everyday picture: put 100 slips of paper in a bag. Write "day" on 41 of them, "evening" on 21, "night" on 14, and so on. Draw one slip without looking. "day" is the most likely result, but it is far from certain. The picture stops being accurate in one way: after each draw the model builds a completely new bag, because the text has grown by one token and the probabilities for the next position are different.

This explains section 3. If the first draw gives "day" in one run and "evening" in another, the two texts differ from that point on, and every later token is predicted from a different beginning.

5. The choosing step is called sampling

The step that picks one token out of the list has its own name: sampling. The model computes the probabilities, and sampling turns them into one concrete token.

flowchart LR
    A["Text so far<br/>It was a warm summer"] --> B["The model<br/>computes a probability<br/>for every token"]
    B --> C["Sampling<br/>picks one token<br/>(temperature shapes this step)"]
    C --> D["Chosen token<br/>is added to the text"]
    D -- "repeat for the next token" --> A

The loop runs once per token. The model reads the text so far and produces probabilities. Sampling picks one token. The token is added to the text, and the longer text goes back in for the next round. An answer of 300 tokens means 300 trips around this loop.

Temperature controls how the choice in the sampling step is made. It does not change what the model knows, and it does not change the model's parameters. It only changes how strongly the pick follows the highest probabilities.

6. A low temperature: the top token nearly every time

Picture temperature as a slider with "lower" at the left end and "higher" at the right end. Different providers allow different ranges. A common range runs from 0 up to 1 or 2.

Move the slider to the low end. The sampling step now favours the most likely token very strongly. In the example, "day" is chosen almost every time, and "evening", "night", "afternoon", and "morning" are chosen rarely or never.

Technically, a low temperature reshapes the list before the draw. The largest probability grows and the small ones shrink. At a temperature of 0, the draw is replaced by a fixed rule: always take the token with the highest probability.

Because the same token wins at nearly every position, the whole answer comes out nearly the same on every run. That is the "predictable" end of the slider.

One caution about temperature 0. Answers become much more repeatable, but many providers do not promise identical output every time, because small differences in how the calculation runs on their hardware can still change a result.

7. A high temperature: less likely tokens get picked more often

A higher temperature works in the opposite direction. It flattens the list: the large probabilities shrink and the small ones grow. So the sampling step more often chooses a token that had a lower probability.

Two things follow from that:

• There is more variety in the answers that come back.
• You get the same reply less often when you ask the same thing again.

The table below shows the effect on the example. It keeps only the five listed words and rescales them so each column adds up to 100%. All values are illustrative and rounded.

Read the "day" row from left to right. At a low temperature "day" wins two times out of three. At a high temperature it wins less than one time in three, and "morning" goes from a rare pick to a regular one. The order of the words never changes. "day" is still the most likely token in every column. Only the gaps between the words change.

A very high temperature has a cost. Tokens that fit the sentence badly start to be picked, and one odd token affects everything written after it. The text can drift off topic or stop making sense.

8. What the setting is for: creative or predictable

Temperature lets you control how creative or how predictable the model is. One end of the slider gives predictable answers, and the other end gives varied ones.

"Creative" here has a narrow meaning. The model does not gain imagination. It picks less obvious tokens more often, so the wording and ideas are less expected.

The job decides which end fits:

9. Where you set it

In chat apps such as ChatGPT, temperature is typically not something a user can change. It is still there. A temperature is applied to every response you get back. The company that built the app picked a value, and you never set it yourself.

The setting becomes yours when you build your own AI-powered application. Your program sends requests to a model through an API, and each request can carry a temperature value. In that position you decide how predictable the model is for your users.

A request with the setting might contain these values:

model:        some-model-name
temperature:  0.2
input:        Extract the invoice number from the text below.

The first line names the model. The second line sets a low temperature, because the task has one correct answer. The third line is the text the model reads.

Temperature is the best-known sampling setting, but not the only one. Many APIs also offer settings that limit which tokens may be picked at all, for example only the few most likely ones. They work on the same sampling step and serve the same purpose.

Interview points

• A language model does not output one fixed next token. For each position it produces a probability for every token in its vocabulary.
• Sampling is the step that picks one token from that list. The pick is random but weighted, so more likely tokens are picked more often.
• Sampling is the reason a model is not deterministic: the same input can give different outputs on different runs.
• Temperature is a setting on the sampling step. It does not change the model's parameters or its knowledge.
• A low temperature sharpens the probabilities, so the most likely token is chosen almost every time and answers are predictable. At temperature 0 the most likely token is always chosen.
• A high temperature flattens the probabilities, so less likely tokens are chosen more often. Answers are more varied and repeat less.
• A very high temperature lets poorly fitting tokens through, which can make the text incoherent.
• Low temperatures fit tasks with one correct answer, such as extraction, classification, and code. Higher temperatures fit tasks where variety is wanted, such as brainstorming and creative writing.
• Chat apps typically do not show the setting, but a temperature chosen by the app's maker is applied to every response. Developers set it per request when they call a model through an API.

Summary

• A model writes one token at a time and produces a probability for every possible next token.
• Sampling picks one token from that list, weighted by the probabilities.
• Because of sampling, the same input gives different outputs.
• Temperature controls how the pick is made.
• Low temperature: the top token nearly every time, predictable answers.
• High temperature: lower-probability tokens more often, varied answers.
• You do not set temperature in typical chat apps. You set it in applications you build yourself.