LoRA and Parameter-Efficient Fine-tuning

A company wants three specialised models: one that writes customer replies in its support tone, one that reviews contract clauses, and one that drafts release notes. The previous lesson showed how fine-tuning produces a specialised model from a finished one. Done in the most direct way, that would mean three long training runs and three complete copies of a model with billions of parameters.

There is a cheaper way. Fine-tuning can be done by two very different methods. Both end with a model shaped by your data. They differ in how much of the model has to be touched on the way: one method rewrites the model's parameters, and the other leaves them alone and adds a small set of new ones.

This lesson explains both methods and the names used for the cheaper one: parameter-efficient fine-tuning, PEFT, and LoRA. After it you can explain what each method changes, why the second is far cheaper, and how one model can serve several jobs at once.

1. What you start from

For both methods, the starting point is a finished model. It has been pre-trained, it is already useful as it is, and it is the thing you are about to shape.

The model can be large or small:

A small language model is the same kind of thing as a large one with fewer parameters. Fine-tuning works the same way on both. The method does not depend on which size you pick. The size only changes how much hardware and time the job needs.

2. What you bring: your own data

The second ingredient is a file of examples that show the behaviour you want. Here is a small file for the support-tone job:

support-tone.jsonl

{"prompt": "My order still hasn't arrived.", "reply": "I'm sorry about that. Could you share your order number?"}
{"prompt": "I want to cancel.", "reply": "Of course. I can do that today. Would you like the last invoice refunded as well?"}

The file name ends in .jsonl, which stands for JSON Lines. JSON is a common text format for structured data, written as named fields inside curly brackets. In a JSON Lines file, every line is one complete example.

Each example here has two fields. prompt holds what a customer might write. reply holds the answer the company wants its model to give. Notice what the replies have in common: they apologise or agree first, and then ask a helpful follow-up question. That shared pattern is the tone the model should learn.

A fine-tuning tool reads this file and uses the examples for training. Data flows in one direction only. Your examples go into the training. What is already inside the model does not come out into the file.

3. The first method: retrain every parameter

The direct method is called full fine-tuning. The model as it stands carries a full set of parameters and weights. They were derived during pre-training, plus any fine-tuning done since. Full fine-tuning retrains all of them. Not a selection: every single one.

flowchart LR
    A["The model as it stands<br/>all its parameters and weights"] -- "retrained on your data" --> B["The model afterwards<br/>new values for the<br/>parameters, all the way through"]

On the left is the model before, with the parameters it already carries. On the right is the model after, in which every parameter may hold a new value.

One point needs care. Full fine-tuning does not throw the old values away and start from random numbers. It starts from the existing values and moves each of them, usually by a small amount. That is why the model keeps most of what it knew. But in terms of work, there is no saving compared with ordinary training. Every parameter has to be loaded, a correction has to be computed for every parameter at every training step, and a complete new set of values has to be stored at the end.

So although the amount of data is small, the procedure is not a light adjustment. In effort and in output, it is like building the model again: the whole network goes through training, and what comes out is a whole new copy.

For a model with billions of parameters, this means a lot of memory, a long run, and a result file as large as the original model.

4. The efficient method and its three names

The other method has three names that often appear together. They sit at different levels:

PEFT is the family, and LoRA is its best-known member. Other PEFT methods exist, but LoRA is the one most tools and guides refer to, so the two words are often used as if they meant the same thing.

5. What LoRA does

LoRA makes two moves.

First, it freezes the original model. Frozen means not allowed to change during training. Every one of the original parameters and weights is left exactly as it was.

Second, it attaches a small amount of new material to the model. These are extra parameters and weights that did not exist before. They are attached to the model. They do not replace anything in it. Only these new parameters are trained on your data.

flowchart LR
    I["Input"] --> O["Original model<br/>frozen: every original<br/>parameter left as it was"]
    I --> X["Small add-on parts<br/>new parameters,<br/>the only ones trained"]
    O --> S["Results are added together"]
    X --> S
    S --> R["The outcome<br/>changed noticeably"]

The diagram shows the idea. The input goes through the frozen original model as before. It also goes through the small add-on parts. The two results are added together, so the add-on nudges the outcome. Although the new parts are small, they have a large effect on what comes out.

The new parts are often pictured as extra layers attached to the model. More exactly, they are small blocks of numbers placed alongside the model's existing layers, at many points inside the network. At each of those points, the block's result is added to the result of the layer it sits beside. The set of all these blocks is called an adapter, or a LoRA adapter.

An everyday picture: a printed textbook stays exactly as printed, and a teacher sticks notes on some pages that say "for our course, do it this way". A reader follows the book plus the notes. The book is the frozen model, the notes are the adapter, and the same book can be given a different set of notes for another course. The picture stops being accurate in one way: a reader sees book and note as separate things, while in the model the two are combined into one calculation and cannot be told apart in the output.

6. Why a small add-on is enough

The method works because it combines two properties. The add-on is small enough to be cheap to train. And it is placed at points in the network where it still changes what comes out.

The words "low rank" describe how the add-on is kept small. A layer of a model holds a large table of numbers. LoRA does not train a second table of the same size. It trains two much narrower tables whose combination stands in for a full-size correction.

A worked case with made-up numbers shows the saving:

One layer's table:      1,000 rows x 1,000 columns = 1,000,000 numbers

LoRA add-on for it:     1,000 rows x 8 columns     =     8,000 numbers
                      +     8 rows x 1,000 columns =     8,000 numbers
                                                     ----------------
                                                        16,000 numbers

Full fine-tuning would train all 1,000,000 numbers in this layer. LoRA trains 16,000, which is 1.6% of them. The "8" in the example is the rank. A lower rank gives a smaller and cheaper add-on, and a higher rank gives the add-on more room to learn.

Across a whole model, the share depends on the rank and on how many layers get an add-on. With common settings the trained part is often around 1% of the model's parameters or less. An adapter file is typically counted in megabytes, while the model it belongs to is counted in gigabytes.

The reason this is enough: fine-tuning for one job does not need to rework everything the model knows. It needs a fairly simple shift in behaviour, and a simple shift can be described with few numbers.

7. The two methods compared

The right-hand column is why companies usually choose LoRA. It gives a model fitted to one job for a fraction of the hardware, time, and storage.

Full fine-tuning still has a place. When a model has to change very deeply, for example to work in a field far from anything it was trained on, training every parameter gives it more room to change. That extra room is paid for with the costs in the left-hand column.

8. One base model, many adapters

Because the original model is never changed, it can be shared. In this setting the original is called the base model, and each job gets its own small adapter. The word is used a little differently here than in the lesson on pre-training, where it meant the raw model straight out of pre-training. Here it means whichever finished model the adapters are attached to, and that is usually a model that has already been post-trained.

the base model       untouched, shared

+ support-tone       customer replies
+ contract-clauses   legal review
+ release-notes      engineering

This answers the problem from the opening. The company keeps one copy of the base model, which is the expensive part: several gigabytes on disk and in memory. It trains three adapters, one per job. Each adapter is small and was cheap to train.

The result is three purpose-built models with only one copy of the expensive part. To serve a customer reply, the system runs the base model with the support-tone adapter attached. For a contract question, it attaches the contract-clauses adapter instead. Adding a fourth job means training one more small adapter, not another full model.

With full fine-tuning, the same three jobs would produce three complete models, each as large as the original.

Interview points

• Fine-tuning can be done in two ways: full fine-tuning, which updates every parameter, and parameter-efficient fine-tuning, which trains only a small number of parameters.
• Full fine-tuning starts from the existing parameter values and may change all of them. It needs memory and computing for the whole model and produces a complete new copy.
• PEFT is the general name for methods that train only a small part. LoRA, short for low-rank adaptation, is the most widely used PEFT method.
• LoRA freezes all original parameters and trains small added blocks placed alongside existing layers. Their result is added to the original layers' result.
• "Low rank" means each correction is expressed as two narrow tables of numbers instead of one full-size table, which cuts the number of trained parameters to a small fraction, often around 1% of the model or less.
• The trained result is an adapter: a small file that works only together with the base model it was trained for.
• LoRA is much faster and cheaper than full fine-tuning and still yields a model that is specific to one job.
• One base model can be shared by many adapters, one per job, so only one copy of the large model has to be stored and loaded.
• Both methods work on large and small language models. The starting point is always a finished model plus a file of examples.
• Full fine-tuning remains useful when the model must change more deeply than a small add-on allows.

Summary

• Both fine-tuning methods start from a finished model and your own examples.
• Full fine-tuning retrains all parameters and produces a whole new copy of the model.
• Parameter-efficient fine-tuning (PEFT) trains only a small number of parameters.
• LoRA is the common PEFT method: the original model is frozen, and small new parts are attached and trained.
• The add-on is small enough to be cheap and is placed where it still changes the output.
• LoRA is far faster and cheaper, and the original network stays untouched.
• One shared base model with a small adapter per job gives several purpose-built models.