Pre-training vs Post-training

Type "Why is the sky blue?" into a chat assistant and you get a short, friendly explanation. That seems like the only reasonable thing a language model could do with a question. It is not. A model fresh from its main training run would more likely reply with further questions, or with a page of rambling text.

The difference comes from two stages of training. The previous lesson split a model's life into training and inference. This lesson looks inside the training part and splits it again, into pre-training and post-training.

After this lesson you can say what each stage does, explain why a model that has only been pre-trained is a poor assistant, and explain where an assistant's manners and refusals come from.

1. Two stages inside training

Both terms belong to the training part of a model's life. Neither is about inference.

flowchart LR
    D["Very large collection of text"] --> P["Pre-training"]
    P --> B["Base model<br/>predicts text"]
    B --> Q["Post-training"]
    E["Smaller set of<br/>chosen examples"] --> Q
    Q --> A["Assistant model<br/>answers, helps, refuses"]
    A --> I["Inference<br/>the model serves users"]

The diagram reads from left to right. A very large collection of text goes into pre-training, and a base model comes out. That model and a smaller, carefully chosen set of examples go into post-training, and an assistant model comes out. Only then is the model put to use.

"Pre" means before and "post" means after. Pre-training is the stage before the model is shaped for its job. Post-training is the shaping.

2. What pre-training is

Pre-training builds the broad underlying abilities of a model, and it does so from a very large set of data.

"Broad underlying abilities" means the general things every later use depends on: grammar, the meaning of words, facts about the world, how an argument is built, how program code is written, how many languages work. Pre-training does not aim at any one task.

Pre-training is also where almost all of the cost sits. It is the long run on many machines described in the previous lesson, and it sets the value of every one of the model's parameters starting from random numbers.

3. What goes into pre-training

For a large language model, the data set is text, in the largest amount the builders can collect. The main source is the public internet: websites, articles, forums, reference pages, and published code. Books and text licensed from publishers are often added.

A common short description is "all the text that can legally be collected from the internet". That is close, with two cautions. No model reads the whole internet, and builders filter out some material. Also, what may legally be used for training is still being argued in courts and differs between countries.

flowchart LR
    A["Text from the internet<br/>and other large collections"] --> B["Pre-training"]
    B --> C["A neural network<br/>that can predict text"]

The result of pre-training is a neural network that can predict text. It was trained on one exercise, repeated an enormous number of times: here is the beginning of a passage, predict the next token.

Text from the data:    The capital of France is Paris.

Shown to the model:    The capital of France is
The model predicts:    (a token)
Correct next token:    Paris

If the prediction is wrong, the parameters are adjusted a little, as in any training. No person has to write the correct answers, because the correct answer is already in the text: it is the word that really came next. This is why such huge amounts of text can be used.

To predict the next token well across billions of passages, the model has to pick up grammar, facts, and style along the way. That is how one plain exercise produces broad ability.

4. A pre-trained model is raw

The model that comes straight out of pre-training is called a base model. It is raw, and it is not a good assistant. All it can do is predict text, and the text it learned from came from the internet.

Two things follow from that:

The first column is about content. The training text includes careful explanations, and it also includes arguments, jokes, spam, mistakes, and rude comments. The base model learned to imitate all of it.

The second column matters more. Nothing in pre-training rewards the model for being useful to the person typing. Its only goal was to guess what text comes next.

5. Asking a raw model a question

Here is what can happen when a question is given to a base model. The exchange is an illustration, but it shows the kind of text that comes back.

You:    Why is the sky blue?

Model:  And why is the grass green?
        And why is the sea not blue?
        ?

The model did not answer. It added two more questions and a stray question mark.

From the model's side, nobody asked it anything. It received some text and continued it with text that is likely to follow. A list of children's questions is a believable continuation of "Why is the sky blue?", because lists like that exist on the internet.

The underlying reason is in the data. The training text never showed the model that every question gets an answer. On the internet, many questions get none. A question may be followed by another question, by a joke, by an unrelated comment, or by nothing. The model learned all of those patterns, and it has no reason to prefer the helpful one.

A base model also has no sense of where to stop. It may answer and then keep going, inventing the next question and the next answer, because the pages it learned from kept going too.

6. What post-training shapes

Post-training comes after pre-training and builds on the network that pre-training produced. It does not start again. It takes the base model, with everything it knows, and changes how it acts.

Three areas are shaped:

An everyday picture: pre-training is like years of reading everything in a huge library. Post-training is like job training afterwards, where a well-read person learns how to deal with customers. The picture stops being accurate in one way: a person in a library chooses what to read and what to believe, while a model in pre-training absorbs patterns from all of the text it is given.

7. What happens in post-training

Post-training is where specialists at the AI companies step in. Pre-training is mostly automatic once the data is collected. Post-training is designed by people who decide how the model should behave.

Their method, in three parts:

1. They prepare more training data. Much of it is created for the purpose instead of being collected from the internet. Some is written by people, and some is generated by other AI models.
2. This new data comes on top of what the model has already learned. It is tiny compared with the pre-training text.
3. They use it to fine-tune the model. Fine-tuning means training a model further on a smaller, chosen set of examples. The parameters that pre-training produced are adjusted a little more.

The new data looks different from internet text. It is made of example conversations that show the wanted behaviour:

User:       Why is the sky blue?
Assistant:  Sunlight is made of many colours. Air scatters blue
            light more than red light, so blue reaches your eyes
            from every direction. That is why the sky looks blue.

After seeing a large number of examples in this form, the model learns the pattern: when a user asks something, the assistant gives one direct answer and then stops.

Fine-tuning on examples is not the only technique used in this stage. Another is to have people rate the model's answers and train it towards the answers they prefer. A later lesson covers that method.

8. What the fine-tuning teaches

Three lessons are taught to the model at this stage:

• A question should get an answer. One answer, not an endless series of further questions and answers.
• The model should be friendly and helpful. Being correct is not enough. The tone and the usefulness of the reply are trained too.
• Some requests should be declined. The model is taught to say no.

The third point surprises many people, so take an example. Someone asks a model for instructions to build a bomb. The model should not provide them.

The base model had no such rule. It would continue that text like any other text, drawing on whatever it had read. The rule had to be put there by someone, and post-training is where that happens. The model is shown examples of harmful requests paired with refusals, until refusing becomes its normal response to such requests.

So an assistant's refusals are not something the model worked out for itself, and they are not a list of rules kept in a separate file. They are trained behaviour, stored in the same parameters as everything else. A product built around a model can add separate checks of its own on top, but the model's own habit of declining comes from this training. Because the behaviour is learned from examples and not written as fixed rules, it is not perfect: a model can sometimes refuse a harmless request or fail to refuse a harmful one.

9. The two stages side by side

All of the shaping described in sections 6 to 8 happens in post-training: after the big pass over the text, and on top of the network that came out of it. In everyday use, this shaping is commonly called fine-tuning, and the two words are often used as if they meant the same thing. Strictly, post-training is the stage, and fine-tuning is the main technique used in it. The next lesson looks at fine-tuning in detail.

Interview points

• Pre-training and post-training are both parts of training. Pre-training comes first and builds general ability. Post-training comes second and shapes behaviour.
• Pre-training uses a very large amount of text, mostly from the public internet, and trains the model on one task: predicting the next token. The correct answers come from the text itself, so no manual labelling is needed.
• The output of pre-training is a base model. It continues text and has no built-in idea of answering, helping, stopping, or refusing.
• A base model given a question may continue with more questions, because its training data contains many questions that are not followed by answers.
• Post-training builds on the pre-trained network. It does not train a new one.
• Post-training shapes behaviour, specialisation, reasoning, and safety. Its data is much smaller, is chosen or created on purpose, and often includes artificially generated examples.
• Fine-tuning, which means further training on a smaller chosen data set, is the main technique of post-training. The two terms are often used interchangeably.
• Refusals of harmful requests are learned in post-training. They are trained behaviour held in the model's parameters, which is why they are not perfectly reliable.

Summary

• Training has two stages: pre-training, then post-training.
• Pre-training builds broad ability from a very large amount of text.
• The result is a neural network that predicts text, called a base model.
• A base model is raw: it continues text and does not reliably answer questions.
• Post-training adds chosen and artificially generated examples and fine-tunes the model on them.
• Post-training teaches the model to give one answer, to be friendly and helpful, and to decline some requests.
• This shaping is commonly called fine-tuning.