Tokenizers and Tokenization

The previous lesson showed that a model reads token IDs, which are numbers, and never reads text. Something has to stand between your text and the model and do the conversion in both directions. That tool is the tokenizer.

This lesson is short. It explains what a tokenizer does step by step, who builds it and at what point, and why the quality of its splits affects how much a model can learn. After this lesson you can describe the path your message takes before a model sees it and after the model replies.

1. From raw text to token IDs

A tokenizer works in steps. Take the text "Good morning":

The "·" marks the space, which is stored as part of the second token. The IDs are made-up examples.

Two words are often confused:

• Tokenizer is the tool. It is built together with each model.
• Tokenization is the process: splitting text into tokens and swapping each token for its ID.

Tokenization happens every time you send a message, before the model sees anything. When the model replies, the same tokenizer works in reverse. It takes the IDs the model produced and turns them back into readable text. The first direction is called encoding and the second is called decoding.

flowchart LR
    A["Your text"] -- "encode" --> B["Token IDs"]
    B --> M["The model"]
    M --> C["Token IDs"]
    C -- "decode" --> D["Text you read"]

2. Who builds a tokenizer, and when

A tokenizer belongs to the model it was built for.

Four consequences follow:

• Built by the model makers. Each lab creates its own tokenizer. There is no single shared standard that every model uses.
• Built first. Before training can start, the lab decides how text will be split and creates the vocabulary. All the training data is then tokenized with it, so the model learns using that exact tokenizer.
• Tied to its model. Once a model is trained, it only understands the IDs of its own tokenizer. The two always go together.
• Improved over time. When a lab builds a new model, it may also design a better tokenizer, for example one that handles more languages or programming code with fewer tokens.

3. A good tokenizer helps the model learn more

How well text is split into tokens affects how much the model can learn from it.

flowchart LR
    R["Raw text<br/>meaningful tokens"] --> T["The tokenizer<br/>splits the words"]
    T --> K["mean | ing | ful | to | ken | s"]
    K --> L["The model learns how<br/>the pieces relate"]

1. Raw text goes in, for example "meaningful tokens".
2. The tokenizer splits it into pieces, such as "mean" + "ing" + "ful" and "to" + "ken" + "s". (The split is an example.)
3. The model learns how these pieces connect. Pieces such as "ing", "ful", and "s" appear in thousands of words. Because the model meets them so often, it learns what they do: "-ful" means "full of", and "-s" usually means "more than one".

When the tokenizer splits words into meaningful, reusable parts, the model can pick up finer meaning, more detail, and more complex relations between a whole text and the parts it is made of.

A poor split has the opposite effect. If words are cut at places that do not repeat across other words, the model sees each piece less often and learns less from the same amount of text.

4. No tokenizer, no language model

A language model cannot work without a tokenizer.

• The model only works with numbers. It cannot read letters or words directly.
• The tokenizer is the translator. It turns your text into token IDs the model can process, and turns the model's output IDs back into readable text.
• It is needed at every stage. It prepares all the training data before the model learns anything. It is used again every time you send a message, and again for every reply.

Without a tokenizer, text could never get into the model or back out of it.

Interview points

• A tokenizer converts raw text into tokens and then into token IDs. Tokenization is the name of that process.
• The same tokenizer converts the model's output IDs back into text.
• Tokenization runs on every request before the model sees the input, and on every response after the model produces output.
• Each AI lab builds its own tokenizer. It is created before training, because the training text must be tokenized first.
• A trained model is tied to its tokenizer. IDs from a different tokenizer are meaningless to it.
• New models often ship with updated tokenizers that cover languages or code more efficiently.
• Splits that produce meaningful, reusable pieces let the model see each piece more often and learn finer relations.
• A language model depends on its tokenizer for training and for every use afterwards.

Summary

• The tokenizer is the tool. Tokenization is the process.
• The steps are: raw text, tokens, token IDs. The reverse turns IDs back into text.
• Labs build a tokenizer before training a model, and the two stay together.
• Better splits lead to a model that learns more from the same text.
• Without a tokenizer, a model can neither be trained nor used.