How an LLM Generates Text

A language model does not read words and write sentences the way a person does. It reads numbers, produces one number at a time, and repeats. Every performance technique in this course speeds up some part of that loop, so the loop has to be clear first. This lesson walks through one request from the text a user types to the last token of the answer, and then opens the model to show its main parts.

Afterwards you can explain what a token is, name the two phases of inference (prefill and decode), compute how sampling settings such as temperature change the choice of the next token, read the architecture name in a model's config.json, and describe the layers inside a transformer block.

The lesson builds on Neural Networks for Inference Engineers, which defines layers, hidden states, and matrix multiplication. Calculations below are worked by hand in text blocks.

Example 1: tokens, tokenizers, and vocabularies

An LLM is an autoregressive token generation model: it generates new tokens one at a time, and each new token depends on every token before it. A token is the basic unit of a language model. It is a number that stands for a chunk of text. Modern LLMs use subword tokenization, which means that a token is a whole word or a fraction of a word. Common words and punctuation marks get one token each, and less common words are split into several tokens.

The split below is an illustration. Real tokenizers differ in where they cut, and the ids are made up.

text:    "The unbelievable cat sat."
tokens:  "The" | " unbel" | "iev" | "able" | " cat" | " sat" | "."
ids:      464     38172     1812    540      3797     3332     13

Seven tokens carry four words and a period. The common words each cost one token, and the rare word unbelievable costs three.

Converting text to tokens and tokens back to text needs no neural network. A tokenizer is a plain mapping between strings and their numeric token ids. The complete mapping between tokens and strings is the model's vocabulary. Vocabularies and tokenizers differ from model to model, and newer models use more efficient schemes. The reason this matters is speed: the fewer tokens an output needs, the faster the end-to-end inference, because each token costs one trip through the model.

Most models have more than 100,000 tokens in their vocabulary. In other modalities, such as speech synthesis, the vocabulary is extended so that tokens can stand for other information, such as audio waveforms.

Example 2: input, reasoning, and output sequences, and the context window

An inference request involves two or three sequences of tokens:

Together they must fit in the model's context window, which is the total number of tokens the model can process and generate per request. A request can also set max_tokens, a limit on the length of the output sequence.

A worked case: take a context window of 8,192 tokens, an input of 3,000 tokens, and a reasoning model that thinks for 1,500 tokens.

tokens left for the answer = context window - input - reasoning
                           = 8,192 - 3,000 - 1,500
                           = 3,692 tokens
with max_tokens = 1,000    -> generation stops at 1,000 output tokens
with max_tokens = 5,000    -> generation stops at 3,692 (the window is full first)

The reasoning tokens use window space even though the user may never see them.

The input sequence is one string, but models are trained on varied inputs: multi-turn chats with roles (system, user, assistant), function signatures for tool calls, and sometimes images or other modalities. All of that must be flattened into one sequence. The chat template does it. It differs subtly between models and has to be implemented correctly in the inference engine. A wrong template gives a model text in a layout it was not trained on, and quality drops.

chat messages:   system: "Be brief."   user: "Hi"
after template (illustrative layout):
  <|system|> Be brief. <|end|> <|user|> Hi <|end|> <|assistant|>

Tokenizing the input sequence with the chat template applied is step zero of inference.

Example 3: prefill and decode

After step zero there are two main phases:

A forward pass is one trip of data through all layers of the model. The KV cache holds results of the attention step for earlier tokens so that they are not recomputed. The next lesson explains attention and the cache.

flowchart LR
    A["Prompt: tokenize with chat template"] --> B["Prefill: one pass over all input tokens, fills KV cache"]
    B --> C["Decode pass 1: first output token"]
    C --> D["Decode pass 2: next token"]
    D --> E["... until stop token, max_tokens, or full context window"]

Each decode pass must end with a token, but a neural network outputs vectors, not tokens. That takes a few extra steps (Example 4). The prefill and the token-producing passes use large neural networks, so they take nearly all of the time and resources of inference. They are also the two phases whose speed the metrics lesson names: prefill sets time to first token, and decode sets tokens per second.

Example 4: from logits to a token

The output layer of the network produces a vector of logits, one raw score for every token in the vocabulary. The vector's length equals the vocabulary size. The scores are normalized into probabilities with the softmax function: each score is turned into e^score, and each result is divided by the sum of all of them, so the probabilities add up to 1.

A worked case with a vocabulary of three tokens. The logits are 2.0, 1.0, and 0.1.

e^2.0 = 7.389    e^1.0 = 2.718    e^0.1 = 1.105     sum = 11.212
probabilities:  7.389 / 11.212 = 0.659
                2.718 / 11.212 = 0.242
                1.105 / 11.212 = 0.099

The model does not always take the top token. The output token is picked by weighted random selection from the probabilities: here token A is chosen 65.9 percent of the time, B 24.2 percent, and C 9.9 percent. Three request settings change the selection:

Each setting applied to the example:

Temperature 0.5 (divide logits by 0.5 -> 4.0, 2.0, 0.2):
  e^4.0 = 54.598   e^2.0 = 7.389   e^0.2 = 1.221   sum = 63.208
  probabilities: 0.864, 0.117, 0.019        (sharper: A is now far ahead)

Temperature 2.0 (divide logits by 2 -> 1.0, 0.5, 0.05):
  e^1.0 = 2.718    e^0.5 = 1.649   e^0.05 = 1.051   sum = 5.418
  probabilities: 0.502, 0.304, 0.194        (flatter: more random)

Top-k with k = 2 on the original probabilities:
  keep A 0.659 and B 0.242, sum 0.901
  re-normalize: 0.659 / 0.901 = 0.731, 0.242 / 0.901 = 0.269, C is removed

Top-p with p = 0.8:
  A alone adds up to 0.659 (less than 0.8), A + B adds up to 0.901 (reaches 0.8)
  keep A and B, then re-normalize as for top-k

A lower temperature, a lower top-k, or a lower top-p makes the output more predictable, because the model is limited to highly likely tokens. Temperature 0 or top-k of 1 makes selection deterministic: the highest-probability token is always chosen. (Temperature 0 cannot be a literal division by zero, so engines treat it as "always take the maximum".)

For structured output, where the response must follow a schema such as JSON, there are more tools, such as logit biasing, which raises or lowers the scores of chosen tokens to steer the output. They apply after each forward pass. They support important abilities such as tool use, and implementing them correctly is part of high-quality inference.

The loop of producing logits and choosing a token continues until the model generates the stop token, a special value marking the end of the output sequence. It also ends if the context window or the max_tokens limit is reached first.

Example 5: reading a model's architecture

Every LLM on Hugging Face, the largest repository of open models, has a config.json file. It is a few dozen lines long and describes the model's architecture: the decisions made in training about the nature and shape of each component. One architecture can have:

Architecture matters because runtime and engine support depend on it. If one architecture has a highly optimized deployment, another variant of the same architecture gets the same performance gains.

The architecture name is one of the first lines of a config file. Here is how Qwen3MoeForCausalLM is parsed:

A causal language model predicts the next token from the previous tokens. The other kind is a masked language model, which fills in a blank using tokens on both the left and the right. All generative LLMs today are causal language models. The rest of config.json gives the nature and dimensions of the layers of the network and of the vectors passing through them during inference.

Example 6: the layout of a transformer block

The main body of an LLM is a series of dozens to hundreds of transformer blocks. The whole network has three kinds of layer:

Inside each transformer block there are sublayers for attention, a feed-forward neural network, and normalization (a step that keeps the numbers in a stable range). The original transformer paper, "Attention Is All You Need" (Vaswani et al., 2017), drew this block, and the layout is the same in principle today.

flowchart TB
    T["Tokens"] --> E["Embedding layer"]
    E --> B1["Transformer block 1: attention, feed-forward network, normalization"]
    B1 --> B2["Transformer block 2"]
    B2 --> BN["... dozens to hundreds of blocks ..."]
    BN --> L["Output layer (LMHead): logits for every vocabulary token"]
    L --> S["Softmax and sampling: next token"]

The feed-forward neural network is a multi-layer perceptron, a stack of the linear layers and activation functions from the previous lesson. These linear sublayers hold most of the trainable weights in an LLM. The attention sublayers are the second-largest part. Normalization and activation functions are a rounding error in the model's size. A rough picture, to build intuition and not as exact figures: if a model has 8 billion weights, most of them sit in linear layers, a smaller share in attention, and normalization adds almost nothing.

The size of the weights and the difficulty of inference are not the same thing. Linear sublayers are the biggest part of the weights, but attention is the more complex operation for inference. That is the topic of the next lesson.

Interview points

• An LLM is autoregressive: each token depends on all previous tokens. Tokens are numbers for chunks of text, produced by subword tokenization.
• A tokenizer is a plain mapping between strings and token ids and needs no neural network. The vocabulary is that mapping, with more than 100,000 tokens in most models. Fewer tokens per output means faster inference.
• Inference uses an input sequence, an optional reasoning sequence, and an output sequence. Together they fit in the context window. max_tokens limits the output.
• The chat template flattens roles, tool signatures, and multimodal inputs into one sequence. It differs per model and must be implemented correctly.
• Prefill processes the input and builds the KV cache. Decode generates tokens one forward pass at a time.
• The output layer produces logits, one per vocabulary token. Normalization turns them into probabilities, and a weighted random draw picks the token.
• Temperature adjusts logits before normalization, top-k keeps the k most likely tokens, and top-p keeps the smallest set whose probabilities add to p. Temperature 0 or top-k of 1 is deterministic.
• Logit biasing helps with structured output and tool use. Generation stops at the stop token, max_tokens, or the context window limit.
• config.json holds the architecture. Sizes, base and instruct variants, and LoRA fine-tunes share an architecture, so optimizations carry over.
• In Qwen3MoeForCausalLM, Qwen is the family, 3 the major version, MoE the Mixture of Experts design, and CausalLM the causal language model type.
• A transformer block has attention, a feed-forward network, and normalization. Linear sublayers hold most weights and attention holds the second most.

Summary

• Text becomes token ids, the model produces logits, sampling picks a token, and the loop repeats until a stop condition.
• Prefill and decode are the two phases, and the KV cache links them.
• Sampling settings change the probabilities of the next token. Deterministic selection takes the maximum.
• Architecture names in config.json identify the family, version, MoE, and causal LM type, and decide engine support.
• The network is an embedding layer, dozens to hundreds of transformer blocks, and an LMHead.