Distillation

Building a capable model the usual way is slow and expensive. A lab has to collect enormous amounts of text, create extra examples, and pay people to review them. A second route exists that avoids most of that work: take a model that is already good, ask it a very large number of questions, and train your own model on its answers.

That route is called distillation. It is a core term around large language models and belongs to the post-training part of a model's life, next to fine-tuning and synthetic data. It is also a term that appears in the news, because AI companies have accused one another of doing it without permission.

After this lesson you can explain how distillation works, name the two models involved, say why it is so much cheaper than the usual route, and separate the accepted use of the technique from the disputed one.

1. Why the term is in the news

Many people first meet the word through complaints.

At the time of writing, several such accusations have been made in public. They are claims by the companies involved, and the accused companies do not necessarily accept them. This lesson does not judge any particular case. It explains what the technique is, so that such reports make sense.

2. The idea: use a model that already exists

The idea behind distillation is short: use a model that already exists. The model behind ChatGPT is one example of such a model. Any capable model would do.

A good model contains, in its parameters, the result of all the data and effort that went into it. You cannot see those parameters when the model is offered only as an online service. But you can see what the model writes. Every answer shows a little of what it knows and how it behaves. Collect enough answers and you have training material.

3. What you do, step by step

flowchart LR
    Q["You, asking<br/>a question<br/>another question<br/>a different problem<br/>and on, and on"] --> T["An existing model<br/>(ChatGPT, for instance)"]
    T -- "its output" --> S["Your own model<br/>trained on that output"]
    T -- "its output" --> I["Or a model you already have,<br/>improved with that output"]

The procedure has three steps.

1. Ask. Prepare a broad variety of problems: a question, another question, a different kind of problem, and so on, thousands or millions of times. The variety matters, because the new model can only learn what the questions cover.
2. Collect. Send each problem to the existing model and save its output.
3. Train. Use the saved pairs of problem and output as training examples for your own model.

A few collected pairs might look like this:

Asked:     Explain what a credit note is in two sentences.
Answered:  A credit note is a document a seller sends to reduce
           the amount a buyer owes. It is used after a return
           or a billing mistake.

Asked:     Rewrite this sentence politely: "Send the file now."
Answered:  Could you please send the file when you have a moment?

Each pair is one training example: the input is what was asked, and the wanted output is what the existing model answered. This is the same form that fine-tuning data has.

The collected output can be used in two ways. You can train a new model of your own on it. Or you can improve a model you already have, by fine-tuning it on the collected examples.

4. The two models have names

The names describe the relation well up to a point. A teacher demonstrates, and a student learns by imitating the demonstrations. The picture stops being accurate in one way: a human teacher explains and checks the student's understanding, while a teacher model only answers questions. It takes no part in the training and does not know it is being used as a teacher.

The word "distillation" comes from chemistry. Distilling a liquid means heating it and collecting the vapour, which separates the wanted part from the rest in a concentrated form. Here, the valuable part of a large model, its behaviour, is extracted into another model.

Distillation is closely related to synthetic data. The collected answers are training data written by a model, which is what synthetic data is. The term distillation is used when the purpose is to make the student behave like that particular teacher.

5. What the shortcut skips

Compare the two routes to the same destination: data to train your own model on.

flowchart LR
    subgraph LONG["The long way"]
        direction LR
        A["Collect text from<br/>the public internet"] --> B["Generate synthetic data<br/>on top of that"]
    end
    subgraph SHORT["The short way"]
        direction LR
        C["Ask a model<br/>that already exists"]
    end
    B --> D["Data to train on<br/>for your own model"]
    C -- "derived, not collected" --> D

The long way is the route the top labs took to improve their models. They gathered text from the public internet, which takes large crawling and cleaning efforts and a good deal of manual work. Then they generated synthetic data on top of that and had it reviewed. Earlier lessons described both steps.

The short way has a single source and a single step: ask a model that exists. The data is derived from that model's answers. It is not collected from the world.

The teacher's answers are valuable because the teacher's maker already paid for the long way. The answers are clean, well formed, and already shaped by post-training to be helpful. The student receives the result of that work without anyone repeating it.

The shortcut has limits. A student trained only on a teacher's answers generally does not become better than the teacher. It also copies the teacher's mistakes and habits along with its strengths. And it learns only the areas the questions covered.

6. Whether it is allowed

The technique is not the problem. Whose model it is pointed at might be.

Distillation is a training method like any other. Nothing about it is improper in itself. The dispute begins when the teacher is somebody else's model and its owner has not agreed.

Providers of commercial models publish terms of service, the rules for using their product. These typically forbid using the model's outputs to build a competing model. A company that distils such a model against those rules gains the benefit of the provider's investment without paying for it. Providers also look for the pattern, such as accounts sending unusually large numbers of systematic questions, and block it where they find it.

The legal questions around this are not settled and differ between countries. What can be said plainly is where the line is drawn in practice: the labs run distillation on models they own, and they object when others run it on their models without permission.

7. Distilling your own models

The accepted use looks like this:

1. Start with the expensive, complex model. It already exists, and the lab owns it.
2. Use it as the teacher, exactly as before: it answers a broad set of questions and so demonstrates how to respond.
3. Train a simpler model on those answers. The simpler model is cheaper to run than the one it came from.

The typical result is a smaller model derived from a larger model that the same company already has.

The reason to do this is cost and speed at inference. A large model needs a lot of memory and computing for every request. A smaller model needs less, answers faster, and can be offered at a lower price. Training a small model in the ordinary way gives a certain quality. Training the same small model on a strong teacher's answers often gives better quality, because the teacher's answers are cleaner and more consistent than raw text.

A worked comparison with made-up numbers:

                    Teacher model      Distilled student
Parameters          200 billion        8 billion
Memory needed       very large         fits on one server chip
Cost per request    high               a small fraction of the teacher's
Quality             the reference      somewhat lower in general,
                                       close on the covered tasks

Many providers offer a family of models in several sizes. Distillation from the largest model is one of the common ways the smaller, cheaper members of such a family are made.

In research writing, the word has a slightly narrower sense than in the news. There it usually means exactly this case: compressing a large model into a smaller one. A lab that owns the teacher can also use more than its written answers. It can use the teacher's full list of token probabilities at each step, which carries more information than the single token that was picked. Someone who can only chat with a model from outside sees only the text.

8. Where that leaves distillation

Both columns describe the same method. The difference is ownership and permission, not the mechanics.

Interview points

• Distillation means training one model on the outputs of another, already trained model.
• The model that supplies the answers is the teacher. The model trained on them is the student.
• The procedure is: send a broad variety of prompts to the teacher, save the outputs, and use the prompt and output pairs as training data for a new model or to improve an existing one.
• It is a shortcut because the data is derived from one source in one step, instead of being collected from the internet and extended with reviewed synthetic data.
• The student inherits the teacher's errors, covers only what the prompts covered, and generally does not exceed the teacher.
• Labs distil their own large models into smaller ones that are cheaper and faster at inference. This is a standard, accepted use.
• With full access to the teacher, distillation can use the teacher's token probabilities, not only its text. With outside access, only the text is available.
• The technique itself is neutral. The dispute concerns distilling another company's model without permission, which providers' terms of service typically forbid.
• Public accusations of unauthorised distillation between AI companies are claims by those companies, and the legal position is not settled.

Summary

• Distillation uses a model that already exists to produce training data.
• You ask the teacher model many varied questions and train the student model on its answers.
• The output can train a new model or improve one you already have.
• It skips the long route of collecting internet text and generating synthetic data.
• Labs distil their own expensive models into simpler, cheaper ones.
• The technique is widely used and accepted on models you own.
• Using it on someone else's model to save training cost is what the complaints are about.