Tokens

As soon as you work with large language models, you meet the word token. Prices are quoted per token. Limits are stated in tokens. Speed is measured in tokens per second. The token is the unit almost everything is counted in.

An earlier lesson gave a first picture: a model cuts text into pieces before reading it. This lesson goes further. It shows that each piece is turned into a number, explains why the pieces are chosen the way they are, and shows how tokens end up on a bill.

After this lesson you can explain what happens to your text before a model sees it, estimate why one request costs more than another, and understand usage limits on a chat subscription.

1. What a token is

A token is a small chunk of text. It is the piece a model reads and writes. A token can be a whole word, part of a word, a number, or a punctuation mark.

• "cat" might be one token.
• "understanding" might be split into "under" + "stand" + "ing", which is three tokens.

For English text, one token is about three-quarters of a word on average. So 100 tokens is roughly 75 words, and a 750-word article is roughly 1,000 tokens.

Tokens matter because three things are counted in them:

• Cost. Many AI services charge per token, for what you send and for what comes back.
• Limits. A model can only handle a certain number of tokens at once. This limit is called its context window.
• Speed. Longer inputs and longer answers mean more tokens, and more tokens take more time to process.

2. Text in, text out, with tokens in between

You and the model do not exchange words directly. There is a conversion step on each side.

flowchart LR
    A["What you type<br/>Summarise this contract."] --> B["Tokens<br/>text split into pieces"]
    B --> C["The model"]
    C --> D["Tokens<br/>the model's result"]
    D --> E["What you read<br/>The lease runs five years..."]

1. You type text, such as "Summarise this contract."
2. The text is split into tokens before the model sees it.
3. The model works on those tokens and produces new tokens as its answer.
4. Those tokens are joined back into text that you can read.

The model only ever deals with tokens, going in and coming out. The text on either side is the human-readable form.

3. Every token is a number

A model cannot read letters. It works only with numbers. So after the text is split into tokens, each token is replaced by a number.

Here is one piece of text, split into tokens, with a number under each one:

The numbers in this lesson are made-up examples. Real models use different splits and different numbers.

Every model has a fixed list of all the tokens it knows. This list is its vocabulary. Each token in the vocabulary has its own number, called its token ID.

When you type something, your text is split into tokens, and each token is swapped for its ID. The row of IDs is what the model receives. When the model answers, it produces IDs, and those are turned back into text for you.

So underneath, a whole conversation is a stream of numbers going in and a stream of numbers coming out.

4. Each model has its own tokenizer

To train a model on a very large amount of text, that text first has to be cut into tokens. The tool that does the cutting is called a tokenizer. The team that builds the model also builds its tokenizer, in the same project.

Two different models can split the same word in different ways:

One model's tokenizer            Another model's tokenizer

"tokenization"                   "tokenization"

[ tokeni ][ zation ]             [ token ][ ization ]
   4812      9067                   2190      5534

The word is the same and the job is the same. The splits and the numbers are different. (Both are made-up examples.)

• Different splits. One tokenizer cuts "tokenization" into "tokeni" + "zation", another into "token" + "ization".
• Different numbers. The same piece of text gets completely different IDs in each system.
• Not interchangeable. A model only understands the IDs from its own tokenizer. IDs from another model's tokenizer would be meaningless to it.

This is also why the same text can count as a different number of tokens depending on which model you use. That affects both cost and limits. The next lesson covers tokenizers in more detail.

5. Why words are split into smaller pieces

A tokenizer does not keep every word whole. Longer or less common words are broken into smaller parts. Take a long word and a related one:

internationalisation   ->   inter | nation | al | isation
nationalised           ->   nation | al | ise | d

The piece "nation" appears in both. It also appears in "nation", "national", "international", and many more words. (The splits are examples.)

Splitting words into reusable pieces helps the model in three ways:

• Pieces repeat. The model meets "nation" again and again across many different words.
• More practice. Because it sees the same piece so often, it learns well what "nation" means, and how parts such as "inter-", "-al", or "-ise" change a word. This repetition is where the model picks up meaning and fine detail.
• New words work. A word the model has never seen whole can still be understood from familiar pieces, in the same way that you can guess what "renationalise" means from its parts.

6. Why a model's vocabulary is so large

Every unique token gets its own ID and keeps it. The training text covers many languages, so the list of tokens is long.

The "·" marks a space before the word. The space is stored as part of the token, so "·morning" (with a leading space) is a different token from "morning" at the start of a line. (All tokens and IDs shown are examples.)

Different words need different entries. "Good", "Guten", and "Buenos" each get their own ID. The vocabulary must cover every language in the training text, plus numbers, symbols, and programming code. A typical vocabulary has tens of thousands to a few hundred thousand tokens.

The vocabulary grows while the tokenizer is being built, as more distinct pieces are found in the text. Once the tokenizer is finished, the vocabulary is fixed. It does not keep growing while the model is in use.

7. What a model learns: how token IDs relate to each other

Take the sentence "understanding how text is formed". After tokenization, the model sees only the bottom row of this table:

Nothing in the bottom row is a word. The words are only what the numbers stand for. The model never sees the words.

During training, the model learns how these IDs relate to one another: which ones tend to appear together, and in what order. For example, across a very large amount of text, 4108 ("under") is often followed by 917 ("stand").

flowchart LR
    t1["4108<br/>under"] --- t2["917<br/>stand"] --- t3["2630<br/>ing"] --- t4["58<br/>how"] --- t5["7741<br/>text"] --- t6["1264<br/>is"] --- t7["3095<br/>formed"]
    t1 -. "learned relation" .- t3
    t5 -. "learned relation" .- t7

These learned relations are stored in the model's weights. They are what let the model work with text and predict which token should come next. A model's understanding is a very detailed map of how token IDs connect.

8. Two reasons to understand tokens

One reason is about how models behave. The other is about money.

9. Two meters: input tokens and output tokens

Every request to a model has two meters running. Suppose you attach a contract and ask for a summary.

The token counts are examples.

• Input tokens are everything you send: your question and any documents, files, or earlier messages that go with it. A long attachment adds many tokens even if your question is short.
• Output tokens are everything the model writes back. They are priced separately, and usually at a higher price per token than input.

The cost of one request is:

cost = (input tokens x input price) + (output tokens x output price)

A small worked case. Suppose a model costs 3 dollars per million input tokens and 15 dollars per million output tokens. These prices are made up for the example.

input:   1,284 tokens x $3  / 1,000,000  = $0.003852
output:    412 tokens x $15 / 1,000,000  = $0.006180
total:                                     $0.010032

The request costs about one cent. The output was a third of the size of the input but cost more, because each output token is five times the price. So there are two ways to lower cost: send less text, or ask for shorter answers.

10. Subscription or API: the meter is always running

There are two common ways to pay for a model.

Subscription. You pay a set amount each month to use an app such as ChatGPT or Claude. You never see a token count, but your usage is still measured in tokens. A subscription does not switch the meter off. It moves it out of sight. If you use a lot, you reach the plan's limit and have to wait or upgrade.

API. API stands for application programming interface. It is the way one program talks to another. Developers and businesses connect their own software directly to the model through the provider's API. There is no flat fee. Every input token and every output token is counted and billed, so you pay for what you use and you can see the numbers.

11. The token is the unit of everything

Tokens tie together three things from this lesson:

• What goes in. Your text, documents, and questions are split into tokens before the model sees them.
• What comes back. The model's answer is produced as tokens and then turned back into text.
• What it costs. You are billed by the number of input and output tokens, either directly through the API or indirectly through the usage limits of a subscription.

Interview points

• A token is a chunk of text: a word, part of a word, a number, or a punctuation mark. In English, one token is about three-quarters of a word.
• A model receives token IDs, which are numbers, not text. The vocabulary maps each token to its ID.
• Each model has its own tokenizer, built with the model. Splits and IDs differ between models and are not interchangeable.
• The same text can have a different token count in different models, which changes cost and how much fits in the context window.
• Words are split into reusable pieces so that common pieces are seen often in training and unseen words can be built from known pieces.
• A space is usually stored as part of the token that follows it.
• Vocabularies hold tens of thousands to a few hundred thousand tokens because they cover many languages, numbers, symbols, and code. The vocabulary is fixed once the tokenizer is built.
• Training teaches the model relations between token IDs. Those relations are stored in the weights.
• Billing has two counts: input tokens and output tokens. Output tokens usually cost more per token.
• A subscription hides token counts but still measures usage in tokens, which is why plans have usage limits. API usage is billed per token.

Summary

• Models read and write tokens, not words.
• Every token is represented by a number, its token ID.
• Tokenizers differ between models, so token counts differ too.
• Splitting words into pieces lets the model learn more from the same text and handle new words.
• A model learns how token IDs relate to each other.
• Cost is input tokens times the input price plus output tokens times the output price.
• Subscriptions and APIs both measure usage in tokens.