Fine-tuning

A support team receives hundreds of messages a day, and each one has to be sent to the right department. A general chat model can do this if every request includes a long instruction that lists the departments and the rules. The team would prefer a model that already knows its departments and its rules, without being told each time.

Fine-tuning is the way to get that. It takes a finished model and trains it a little further on examples of one particular job. The previous lesson introduced fine-tuning as the main technique of post-training, the stage in which AI companies turn a raw model into an assistant. This lesson looks at the procedure itself, and shows that the same procedure is open to anyone who has the right starting material.

After this lesson you can describe what goes into and comes out of fine-tuning, list what is needed to do it yourself, explain what the new examples do to the model, and name the kinds of things fine-tuning can teach.

1. The same prompt, before and after

Pre-training produces a model. What is done to the model afterwards decides what it is good for.

A model straight out of pre-training is often called a raw foundation model: a large pre-trained model that other models and products are built on. It is the same thing the previous lesson called a base model. It continues text. It does not answer you.

Give the same request to the model before and after post-training:

A raw foundation model

> Write a short note to the team about Thursday.

Write a short note to the client about the delay. Write a short
note to your manager about...

The same model, post-trained

> Write a short note to the team about Thursday.

Quick one: the release moves to Thursday morning. Two fixes and
one new screen. Nothing else.

The raw model treats the request as the first line of a list of similar requests, and it writes more lines for the list. The post-trained model treats the request as something to carry out, and it writes the note.

The knowledge inside the model did not change between the two boxes. Both versions know what a note is and how a team talks about a release. What post-training added is the habit of answering.

2. No lab releases a model without this step

Because of that difference, this step is not optional for the companies and research groups that build models. No lab releases a general-purpose assistant without it. The version handed to the public is the version that post-training produced, not the raw one.

So every assistant you have used is already a fine-tuned model. The rest of this lesson is about running the same procedure again, on top of that finished model, for a purpose of your own.

3. A finished model goes in, a finished model comes out

Fine-tuning has two inputs and one output.

flowchart LR
    M["A finished model<br/>taken as it was released"] --> T["A fine-tuning tool"]
    E["Examples you wrote yourself"] --> T
    T --> U["An updated version<br/>of that same model"]

The first input is a finished model, taken exactly as it was released. The second input is a set of examples that you wrote. Both go into a fine-tuning tool, which is software that runs the extra training. Several such tools are available, some as free software and some as paid services. The output is an updated version of the same model.

Nothing here is built from scratch. The starting point is a model that already works.

For the support team, the examples could look like this:

"my card was declined"      ->  route: billing
"I cannot log in"           ->  route: access
"wrong month on invoice"    ->  route: finance

Each line is one training example: an input on the left, and on the right the output the team wants for that input. A real set would contain hundreds or thousands of lines like these, covering the many ways customers phrase their problems.

During fine-tuning, the tool does what any training does. It shows the model an input, compares the model's output with the wanted output, and adjusts the model's parameters a little. It repeats this over all the examples, usually several times.

4. What you need to run it yourself

Nothing about the procedure is reserved for a large lab. There are three conditions, and two of them are practical matters, not matters of skill.

Hardware that can hold the model while it trains. The first question is about the machine, not about AI. The whole model has to sit in memory during training, as it does during inference, and training needs additional memory on top of that for its working data. A small model can be fine-tuned on one well-equipped computer. A large one needs rented servers.

A model that is published in a form you can change. You need the model's weights, meaning the file of parameter values, and permission to download them, adjust them, and save the result. A model published this way is called an open-weight model. Not every model is offered that way. Many of the best-known models are kept on their makers' servers and cannot be downloaded. At the time of writing, some of those makers offer a fine-tuning service for some of their models instead. Which models are covered changes over time, so check the provider's current documentation. With such a service you upload your examples, the training runs on their servers, and you can use the resulting model but never receive its weights.

One job you want the model to be exceptionally good at. This is the condition that needs thought. Fine-tuning pays off for a narrow, repeated job. Examples are sorting your support queue, reading the format of your own log files, or drafting text in the wording your contracts use.

5. What the new data does to the model

A common worry is that new examples overwrite what the model already knows. With a sensible amount of training they do not. Your examples do not replace what is there. They outweigh it.

"Outweigh" has a concrete meaning here. The model's parameters already hold everything from its earlier training. Fine-tuning adjusts those same parameters a little more, and your examples are the most recent and most repeated thing the model has trained on. So in the situations your examples cover, their pattern counts for more than the older, general patterns.

Picture the model as an arrow pointing in some direction. The direction stands for everything the finished model can already do. Your new data pulls the arrow towards a new direction. The result is the same model, aimed a little differently. It is not a new arrow.

flowchart LR
    A["Finished model<br/>every ability it arrived with"] --> C["The same model,<br/>aimed a little differently"]
    B["Your new data<br/>counted more heavily<br/>than the rest"] --> C

In practice, the fine-tuned support model can still write an email, translate a sentence, and explain a concept. It has gained a strong tendency to answer a customer message with a route label.

The shift has limits in both directions. With very few examples the model barely changes. With heavy training on a very narrow set of examples, the model can lose some of its general ability. Practitioners test for both.

6. Three things fine-tuning can teach

A writing style.

You:    Draft the release note.

Model:  Thursday. Two fixes, one new screen. Nothing else moved.

The short, clipped style was learned from the team's own past release notes, which were used as training examples. It did not come from a style guide pasted into the request. The model writes this way without being asked.

A behaviour.

You:    Summarise this report.

Model:  Yes sir. Here is the summary.

Here the model has learned a habit: it starts every reply with "Yes sir." The habit is trivial on purpose. The point is that a fine-tuned model keeps a habit without being reminded, however small the habit is. Useful habits work the same way: always answering in a fixed format, always asking for an order number first, or always replying in a certain language.

Knowledge the model had no way to have.

You:    What does TR-114 mean here?

Model:  The retry cap on the nightly settlement job.

TR-114 is an internal code used inside one company. It appears nowhere on the internet, so no amount of pre-training could have taught it. The model knows it because the company put it into the training examples.

One caution about the third kind. Fine-tuning teaches patterns well, but it is less dependable for exact facts, and facts that change often would need a new round of training each time. For that reason, facts are frequently supplied in a different way: by placing the relevant document into the context of each request. Later lessons cover that approach.

7. The same procedure for a lab and for you

The procedure is the same no matter who runs it. What differs is the audience, and with it the meaning of "good".

A lab fine-tunes so that the model behaves well for millions of people with every kind of question. "Good" has to mean generally helpful, polite, and safe.

When you fine-tune, "good" has a much narrower meaning. It means good at your one job, with your formats, your wording, and your internal names. The model does not need to become better at anything else.

8. When fine-tuning fits

Fine-tuning costs effort: examples have to be collected and checked, training has to be run, and the result has to be tested. It is one of several ways to adapt a model, so it helps to know when it fits.

Interview points

• Fine-tuning is further training of an already trained model on a smaller, chosen set of examples. It starts from a working model and ends with an updated version of the same model.
• A raw foundation model continues text. Post-training by fine-tuning adds the habit of answering, without changing what the model knows. Every released assistant has been through this step.
• The inputs are a finished model and a set of input and output examples. A fine-tuning tool runs the training.
• Doing it yourself needs three things: hardware with enough memory to hold the model while it trains, a model whose weights you are allowed to download and modify (or a provider's fine-tuning service), and one narrow job to improve.
• New examples do not replace the model's existing knowledge. They shift the existing parameters, so the new patterns dominate in the cases they cover while general abilities remain.
• Too little data changes almost nothing. Heavy training on very narrow data can damage general abilities.
• Fine-tuning can teach a writing style, a behaviour or habit, and knowledge that was absent from pre-training, such as internal codes.
• A lab fine-tunes for a general audience. An individual team fine-tunes for one job, with a much narrower definition of a good result.
• Frequently changing facts are a poor fit for fine-tuning, because each change needs retraining.

Summary

• Fine-tuning trains a finished model further on your own examples.
• A finished model goes in, and an updated version of the same model comes out.
• Every public assistant has already been fine-tuned by its maker.
• You need suitable hardware, a model you are permitted to change, and one clear job.
• New data outweighs what is already in the model. It does not erase it.
• Fine-tuning can teach style, behaviour, and missing knowledge.
• Labs fine-tune for everybody. You fine-tune for one narrow purpose.