Reinforcement Learning from Human Feedback (RLHF)

Some questions have one correct answer. "What is 12 times 12" can be checked against 144. Many of the tasks given to an AI model are not like that. "Which of two suppliers should get this job" or "write a polite reply to this complaint" have no single correct answer that could be stored in a file. There are only better and worse answers.

Fine-tuning, as described so far, needs a file of finished examples, each with the wanted output. That fits tasks where the wanted output can be written down. This lesson covers the method used when it cannot: reinforcement learning from human feedback, shortened to RLHF. A person grades what the model produces, and the grade is used to adjust the model.

The lesson starts one step earlier, with how a decision is worked out, because the first thing these methods teach is the habit of working a problem through. After this lesson you can explain why written-out reasoning is rare in training data, describe the RLHF loop, and say how it differs from ordinary fine-tuning.

1. A decision, written out

Start with how a person settles a question. Take an ordinary work decision: two suppliers, and one slot in the schedule. Put the working on paper, step by step.

Two suppliers, one slot in the schedule

Northwind is cheaper, but missed a deadline on the last job.

Harlow costs more, and has not.

That slot sits on the critical path, so a late delivery costs
more than the gap in price.

So: Harlow.

The "critical path" is the chain of tasks in a project where any delay pushes back the final date.

The last line is the decision. The lines above it are the working: the facts that were considered and how they were weighed against each other. The decision is only as good as that working. A different last line, such as "So: Northwind", would need different reasons above it.

2. In the head, or on the page

When you make a decision like this, most of the working happens silently. In many cases it can be put into words, like the note above. It is a kind of text that was never said aloud.

A language model can do something comparable, with one difference. A model's working is text that is written out, where it can be read.

This comparison is an analogy. It maps the steps of a person's reasoning to the lines of text a model writes before its final answer. It stops being accurate beyond that point: human thinking is not only inner sentences, and a model's written steps are not a complete record of the calculation inside the network. What matters for this lesson is narrower. A model that writes out intermediate steps before deciding tends to reach better decisions on multi-step problems, in the same way that a person who writes a problem down makes fewer slips.

So it is useful if a model has the habit of working a problem through. The next question is where it could learn that habit.

3. What the raw training data lacks

Pre-training uses text collected from the internet. That text is not full of worked-through problems.

The web has endless amounts of this:

Recommendation
--------------
Use Harlow for the March slot.

It has much less of this:

Working
-------
Northwind cheaper, missed a deadline.
Harlow dearer, has not.
Slot is on the critical path.
A miss costs more than the price gap.
-> Harlow.

Conclusions get published. The route to them usually stays in somebody's head. Reports, articles, and emails state what was decided. Few of them record each step of how.

So a model that has only been pre-trained has seen a huge number of conclusions and comparatively few examples of the reasoning behind them. It has learned to produce verdicts. It has had much less practice at producing the steps.

4. The working gets written on purpose

The first fix uses synthetic data, which is training data produced on purpose by a model and checked by people.

A capable model is asked to solve many problems and to write out every step. People review the results and keep the sound ones. Those worked examples become fine-tuning data.

The aim is not that the model memorises these particular problems. The aim is that it picks up the habit: when a problem arrives, lay out the facts, weigh them, and then conclude. A model trained on enough worked examples applies that pattern to problems it has never seen.

This is still ordinary fine-tuning. Finished examples go in, and the model is adjusted to imitate them.

5. One more step: RLHF

On top of post-training and ordinary fine-tuning, there is one more step. It is a special form of fine-tuning with a long name: reinforcement learning from human feedback.

The name has two halves:

• Reinforcement learning is training in which the learner tries something and gets back a score, and then changes so that it scores higher next time. What comes back is a score, not a correct answer.
• Human feedback means that a person supplies the score.

Three short statements sum it up: the model gets a score and not an answer key, the score comes from a person, and the whole thing is still fine-tuning. It is still fine-tuning because it starts from a finished model and makes small changes to its parameters.

An answer key is a stored list of correct answers, like the one a teacher uses to mark a multiple-choice test. Ordinary fine-tuning works with an answer key: each example includes the output the model should have given. RLHF works without one.

6. How the loop runs

flowchart LR
    P["Problems<br/>set for the model"] --> M["The model<br/>works through one"]
    M --> D["A decision"]
    D --> J["A human judge"]
    J --> G["A grade<br/>how good the decision was"]
    G --> T["The training loop"]
    T -- "weights adjusted a little" --> M

The steps, in order:

1. The trainers prepare a set of problems. They do not prepare solutions.
2. The model receives one problem and works through it by itself.
3. The model reaches a decision.
4. A human judge reads the result.
5. The judge gives a grade that expresses how good that decision was.
6. The grade goes into the training loop, which is the software that runs the training.
7. The training loop adjusts the model's weights a little. A good grade makes the kind of output that earned it more likely in future. A poor grade makes it less likely.
8. The next problem goes around the same circuit.

Two details matter.

What is graded is the quality of the decision. The judge does not check whether the output matched a stored answer, because there is no stored answer.

The weights are nudged, not rewritten. One grade causes a very small change. The effect comes from repeating the loop over a very large number of problems.

7. No answer key, only a verdict

In this kind of training, nothing is marked right or wrong. It is marked better or worse.

For the supplier problem, a judge might compare two outputs from the model:

Output A:  Northwind. It is cheaper.

Output B:  Harlow. Northwind is cheaper but missed its last
           deadline, and this slot is on the critical path,
           so a delay would cost more than the price difference.

Judge:     B is better.

Neither output is "the" answer from a file. The judge weighs them and gives a verdict: B considered the risk and A did not. That verdict is the feedback. In practice, asking a judge which of two outputs is better is a common way to collect it, because people tend to agree more when they compare two outputs than when each of them assigns a score out of ten.

An everyday picture: a driving instructor does not hand a learner a list of the correct steering angle for every moment. The learner drives, and the instructor says "that turn was too wide" or "that was good". The learner improves from the feedback. The picture stops being accurate in one way: the learner understands the instructor's words, while a model receives only a number, and a training algorithm turns that number into changes to the weights.

8. How RLHF differs from ordinary fine-tuning

In ordinary fine-tuning, the examples go straight into updating the parameters. In RLHF, there are extra stages in between: the model produces something, a judge grades it, and only the grade reaches the parameters.

9. Why the extra effort is worth it

RLHF is more work than ordinary fine-tuning. People have to read and judge outputs, which is slow and costly. The benefit is that you can grade work you could never have written down in advance.

For many tasks, nobody can produce a file of ideal outputs. No one can list the best reply to every possible complaint or the best explanation for every possible question. But a person who is shown two replies can usually say which one is better. Judging is far less demanding than authoring, and RLHF needs only the judging.

This is how qualities that resist exact definition get trained: helpfulness, a fitting tone, clear reasoning, and care with risky requests.

Three facts complete the picture.

First, scale. A person cannot grade the millions of outputs that training requires. In the usual setup, human judgements are collected on a sample and used to train a second model, often called a reward model, that learns to predict the grade a person would give. That reward model then grades most of the outputs automatically. The human judgement is still the source of the signal.

Second, the trained model reflects its judges. It learns what the graders preferred, including their blind spots. If graders favour answers that sound confident or agreeable, the model can learn to sound confident or agreeable instead of being correct.

Third, RLHF is not only about reasoning. It was first used widely to make assistants more helpful and safer in general, and reasoning is one of several behaviours shaped this way.

Interview points

• A decision is the end of a chain of working. Models produce better decisions on multi-step problems when they write out intermediate steps.
• Pre-training data contains many conclusions and comparatively little written-out reasoning, because people publish results more often than the route to them.
• Worked-through problems can be generated as synthetic data, reviewed, and used for ordinary fine-tuning. This teaches the habit of reasoning, not specific answers.
• RLHF stands for reinforcement learning from human feedback. It is a form of fine-tuning in which the training signal is a score given by a person, not a correct output.
• The RLHF loop is: a problem, the model's own attempt, a human grade for the quality of the attempt, and a small adjustment to the weights. Then the loop repeats.
• There is no answer key. Outputs are judged better or worse, often by comparing two outputs.
• Ordinary fine-tuning gives the model finished examples. RLHF gives it problems, and only the grade affects the parameters.
• RLHF makes it possible to train qualities for which ideal outputs cannot be written in advance, because judging an output is less demanding than writing the ideal one.
• At scale, a reward model trained on human judgements does most of the grading.
• The resulting model inherits the preferences and biases of its graders.

Summary

• Good decisions come from working a problem through, and that working can be written as text.
• The internet contains far more conclusions than worked reasoning.
• Reasoning examples are written on purpose as synthetic data and used in fine-tuning.
• RLHF is a special form of fine-tuning in which a person grades the model's output.
• The model gets a score, not a correct answer. Outputs are marked better or worse.
• The grade nudges the weights, and the loop repeats with the next problem.
• RLHF can train behaviour that nobody could have written down as examples.