Training vs Inference

You open a chat app, type a question, and an answer appears a few seconds later. By the time you typed anything, the model that answers you had already been built. Building it took months and was finished long before your conversation started.

These two stages have names. Training is the stage where a model is made. Inference is the stage where the finished model is used. Both words come up as soon as people talk about AI, and about large language models in particular: in news about new models, in pricing pages, and in hardware requirements.

This lesson explains both terms and how they differ. After it you can say which stage you are dealing with when you use a chat app, explain why a model does not learn from your conversation while it answers, and explain why running a model on your own computer depends mostly on memory.

1. One model, two phases

Every model has two phases in its life.

flowchart LR
    T["Training<br/>the model is being made<br/>(done once per model)"] --> I["Inference<br/>the finished model serves users<br/>(for as long as it is running)"]
    R1["a request"] --> I
    R2["a request"] --> I
    R3["a request"] --> I
    I --> O["an output<br/>for each request"]

The left box is training. This is where the neural network is built. A neural network is a program made of layers with numbers on its connections. Those numbers are the model's parameters. Training is the process that works out their values: the network sees an example, makes a guess, the guess is compared with the known answer, and the numbers are adjusted a little. This repeats over a huge number of examples.

The right box is inference. Training has stopped, and the parameters are fixed. The model is now there to serve users. Requests arrive one after another, for as long as the model is kept running, and each one gets an output.

For a given model, training happens once and then ends. Inference goes on for months or years afterwards. Companies do train new models later, but each of those is a separate model with its own training phase. In practice a finished model is often trained further in later rounds, which the following lessons cover. The basic picture stays the same: first the model is made, then it is used.

2. The part you use is inference

Look at an ordinary chat:

new conversation

You:    Summarise this thread for me in three bullet points.

Model:  The vendor agreed to the new date, the price is unchanged,
        and they want the signed contract back by Friday.

Nothing in this exchange is training. The building of the model was over before this chat window existed. What answers you here is the finished model, running.

So when you use any AI product, you are on the inference side. You never see training happen. You only meet its result.

An everyday picture: training is the years a doctor spends studying. Inference is the doctor seeing patients afterwards. The patient meets only the trained doctor. The picture stops being accurate in one way: a doctor keeps learning from each patient, and a model does not, as section 4 explains.

3. Trained once, then put on a server

Follow one well-known example. At the time of writing, the models behind ChatGPT are trained by OpenAI.

flowchart LR
    A["Training<br/>done by OpenAI<br/>finished and over"] --> B["OpenAI's servers<br/>the model and every parameter<br/>loaded, running, and waiting"]
    C["You, in ChatGPT<br/>one request"] -- "what you asked" --> B
    B -- "some text back" --> C

The diagram has three boxes. On the left, OpenAI trains the model. That step is done and over with. In the middle, the finished model, with every parameter it ended up with, is placed on OpenAI's servers. A server is a computer in a data centre that runs software for many users over the internet. The model is loaded there, switched on, and waiting. On the right is you, in ChatGPT. Your message travels to the servers as one request, and some text comes back.

The middle box, together with the two arrows, is inference. The same pattern holds for other providers and their apps.

4. What the word "inference" refers to

"Inference" is a technical word that people use casually, and it can be hard to picture. It sounds like something happening inside a mind, a kind of thinking.

What it describes is plainer:

In everyday language, "to infer" means to reach a conclusion from what you know. A model at inference does something similar in a mechanical way. It takes the input it has been given and uses its fixed parameters to compute an output. For a language model, that output is the next token, then the next, until the answer is complete.

One fact follows from the two-phase picture and surprises many people. During inference the parameters do not change. The model does not learn from your conversation while it answers you. If it seems to remember what you said earlier in a chat, that is because the app sends the earlier messages again with each request, as the lesson on context explained. A provider may collect conversations and use them when training a later model, depending on its policy, but that is a new training phase, not something that happens while you chat.

5. What the serving side needs

Serving a model is demanding. Three points describe it:

• What runs. The model and every parameter it holds. Not a part of it. All of it.
• Where. On a server built for this job. That is why the serving side needs very capable hardware, usually machines with specialised chips called GPUs (graphics processing units), which do the model's arithmetic much faster than an ordinary processor.
• Training. It is at least as demanding as inference, and it uses a different kind of machine setup.

The two phases stress hardware in different ways:

Training a top model is a single enormous job. Inference for one request is small, but a popular product serves millions of requests, so the total computing spent on inference over a model's life can be as large as the training cost or larger.

6. For inference, memory matters most

Memory is the fast working storage of a computer. A program has to be in memory while it runs. This is different from the disk, the slower storage where files stay when they are not in use.

For inference, memory is the first requirement. Every parameter that was derived during training has to be loaded into the memory of the machine that does the serving.

flowchart LR
    A["An incoming request"] --> M["Memory of the serving machine<br/>every parameter from training,<br/>loaded in"]
    M --> O["The output"]

The diagram shows why. An incoming request is passed through the loaded parameters, and the output comes out on the other side. The request cannot skip the parameters, because computing the output means doing arithmetic with them. So they all have to be present.

A worked case with made-up numbers shows the scale. Suppose a model has 8 billion parameters and each one takes 2 bytes, a common size.

8,000,000,000 parameters x 2 bytes = 16,000,000,000 bytes = 16 GB

That model needs about 16 GB (gigabytes) of free memory for its parameters alone, plus some extra for the text it is working on. A model with 400 billion parameters would need about 800 GB by the same sum, far more than any personal computer has.

7. The same rule on your own computer

The memory rule does not only apply to data centres. Some models can be downloaded and run on a personal computer. The same demand applies there: nothing answers until the whole model is sitting in memory.

So the first question for running a model locally is whether your computer has enough memory for all of its parameters. If it does not, the result is usually not a slightly slower model. Either the model fails to load at all, or the computer has to keep moving parts of it between memory and the much slower disk, which makes answers too slow to be usable.

8. Why small models are attractive

Compare two models with the memory of an ordinary computer:

Both are built with the same technique. They are the same kind of neural network, produced by the same kind of training. The only difference in this comparison is the number of parameters to hold. The large model goes past the memory your computer has. The small model stays under that line.

This is what makes a small language model attractive. Because it has fewer parameters, it fits into the memory of a laptop or desktop, so inference can happen on a machine you own. The earlier lesson described the cost of that: fewer parameters mean less capacity, so a small model handles less detail than a large one.

Interview points

• Training is the phase in which a model's parameters are worked out from examples. Inference is the phase in which the finished model, with fixed parameters, computes outputs for new inputs.
• For a given model, training happens once and ends. Inference continues for as long as the model is in service, one request after another.
• Every use of an AI product is inference. Users never interact with a model while it is being trained.
• Parameters do not change during inference, so a model does not learn from a conversation while it answers. Earlier messages are available only because the app sends them again as context.
• In a hosted product, the provider trains the model, loads it onto its servers, and the app sends each user request to those servers and returns the output.
• Inference needs the whole model, with every parameter, loaded into memory, because computing an output uses the parameters.
• Training is at least as demanding as inference but in a different way: one very long job on a large group of GPU machines, compared with many short requests served by machines that each hold the model in memory.
• The memory needed for inference grows with the parameter count. This is why small models can run on personal computers and large ones cannot.

Summary

• A model has two phases: training, where it is made, and inference, where it is used.
• Training builds the neural network by working out its parameters.
• Inference is the finished model running on a machine and answering requests.
• Chatting with an AI app is inference. The model does not change while it answers.
• Inference needs every parameter loaded into memory.
• The same rule applies on a personal computer: no answer until the whole model is in memory.
• Small models have fewer parameters, so they fit in the memory of ordinary computers.