Parameters and Weights

Model descriptions are full of numbers: "a 7-billion-parameter model", "70B", "download the weights". These phrases all point at the same thing, which is the set of numbers a model is made of.

Earlier lessons introduced neural networks and used the word parameter to explain why some models are called large. This lesson looks at those numbers more closely. It explains what a parameter is, what a weight is, how training sets them, and what they do when the model runs.

After this lesson you can explain what "a model with 70 billion parameters" means, why people say "weights" and "parameters" for the same thing, and why a bigger count usually means a more capable model.

1. What a parameter is

An AI model is made of a very large collection of numbers. Each one of those numbers is a parameter. The value of each parameter is worked out during training, from the data the model was shown.

Three facts describe a parameter:

• Nobody sets it by hand. At the start of training, the parameters are random numbers.
• Training adjusts it. The model is shown many examples. After each one, every number is moved up or down by a tiny amount so that the model's answers get better.
• What is left is what the model learned. When training ends, the final values of all these numbers are, in effect, the model's knowledge.

One parameter on its own means almost nothing. The value 0.4407 does not stand for a fact or a word. The billions of parameters working together are what let a model read and write.

2. How weights relate to parameters

"Weights" is the word you will hear most often next to "parameters".

A weight is a number attached to one connection in the network. It sets how strongly one part of the network influences the next, like a volume dial on that connection. A high weight passes the signal on strongly. A weight near zero almost blocks it.

There are a few other kinds of parameter. The best known is the bias, a number added at each point of the network that shifts its result up or down. Weights make up most of a model, so the other kinds are rarely mentioned.

For this course, treat "parameters" and "weights" as the same thing. When you see either word, read it as: the numbers the model learned during training.

This matters because a large language model is a neural network. Parameters and weights are what an LLM is made of.

3. What training data is for: inputs and expected outputs

Training a model starts with a large amount of example data. Each example has two parts.

Training data works like a set of practice questions with an answer key. For a shop's recommendation system:

• The input is the question: a customer's past purchases.
• The expected output is the answer key: what that customer really bought next.

During training, the model sees the input and makes a guess. The guess is compared with the expected output. Then the parameters are adjusted slightly so that the next guess is closer. Training closes the gap between the two.

flowchart LR
    I["Input<br/>past purchases"] --> M["Model with its<br/>current parameters"]
    M --> G["The model's guess"]
    E["Expected output<br/>what was really bought next"] --> C{"How far apart?"}
    G --> C
    C -- "adjust the parameters slightly" --> M

This loop runs across a very large number of examples. At the end, the model's answers match the expected ones most of the time.

4. What a neural network is: a program with numbers on its connections

A neural network is a type of algorithm. An algorithm is a computer program that follows fixed steps. So a neural network is a program: an input goes in, the steps run, and an output comes out.

What makes it special is how it is organised. It is built as layers of connected points.

flowchart LR
    IN(["Input"]) --> a1(("a1")) & a2(("a2"))
    a1 -- "w1" --> b1(("b1"))
    a1 -- "w2" --> b2(("b2"))
    a2 -- "w3" --> b1
    a2 -- "w4" --> b2
    b1 -- "w5" --> c1(("c1"))
    b2 -- "w6" --> c1
    c1 --> OUT(["Output"])

The diagram is a very small network with three layers.

• The circles are the points of the network, often called neurons. They are arranged in layers: a, then b, then c.
• The lines between them are the connections. Each line has its own weight, labelled w1 to w6 here. These are the learned numbers.

Data moves through it in three steps:

1. The input goes in on the left. It has first been turned into numbers, for example a sentence or an image in number form.
2. It passes through the layers. At every connection, the signal is multiplied by that connection's weight, which makes it stronger or weaker. Each circle combines the signals that arrive at it and passes the result on.
3. The result comes out on the right as the model's answer.

Signals travel along every line at once, not along a single path. This small network has 6 weights. A real language model has billions, arranged in many more layers. Training adjusts all the weights until the network turns inputs into the right outputs.

5. What the layers decide

The layers, and the weights inside them, decide how data is changed as it travels through the network, and so what comes out at the end.

What comes out depends on the purpose the network was built and trained for.

Every neural network does the same basic thing: an input goes in, passes through the layers, and an output comes out. For a language model, the layers decide which output text comes back for which input text.

That is what a large language model is at its core: a very large neural network whose weights were trained to produce the right text for the text it is given.

6. Why more parameters usually give a better model

Read this as a chain where each step leads to the next:

1. More parameters. The model has more learned numbers.
2. More layers and more numbers per layer. The network is deeper, so data passes through more stages.
3. More connections. The model can capture finer detail and nuance, such as small differences in meaning, tone, or context.

The result is that larger models usually give more accurate and more careful answers. This is the same effect described for large and small language models: more parameters means more room to keep similar meanings apart.

Size is not the only factor. Two others matter a lot:

• The quality of the training data. A large model trained on poor data gives poor answers.
• How the model is trained. Better training methods get more out of the same number of parameters.

More parameters also have a cost. A bigger model needs more memory to hold its numbers and more computing hardware to run, so each answer costs more to produce.

Interview points

• A parameter is one of the numbers stored inside a model. Its value is set during training, not by a person.
• Parameters start as random values and are adjusted a little after each training example.
• A weight is the strength of one connection in the network. Weights are the most common kind of parameter, and the two words are used as synonyms in practice.
• Biases are another kind of parameter, far fewer in number than weights.
• Training data comes in pairs: an input and the expected output. Training reduces the gap between the model's output and the expected output.
• A neural network is an algorithm organised as layers of connected points, with a learned weight on each connection.
• When the network runs, each signal is multiplied by the weight of the connection it travels along.
• The same structure serves different purposes. What the output is depends on what the network was trained for. An LLM maps input text to output text.
• More parameters allow more layers and connections, which lets a model capture more nuance. Data quality and training method also matter, and larger models cost more to run.

Summary

• A model is a large set of numbers called parameters.
• Weights are the main kind of parameter, and the two words usually mean the same thing.
• Training sets the parameters by comparing the model's guesses with known answers.
• A neural network is a program made of layers, with a weight on every connection.
• An LLM is a neural network trained to turn input text into output text.
• More parameters usually mean a more capable model, at a higher hardware cost.