Data Poisoning

A model behaves the way its training data taught it to behave. The earlier lessons in this section used that fact for good purposes: examples of helpful answers produce a helpful assistant, and examples in a company's tone produce a model that writes in that tone.

The same fact can be used for harm. If someone manages to slip their own examples into the data a model is trained on, they can shape what the finished model does. This is called data poisoning. It is worth knowing as soon as a conversation turns to training, and it applies to both stages: pre-training and post-training.

After this lesson you can explain what data poisoning is, why it is hard to notice, why a small fine-tuning set is an easier target than a web-sized training set, and which precautions reduce the risk.

1. Training assumes good data and does not check

Data poisoning starts from a plain observation about training. Two things are assumed, and neither is enforced.

Training is a mechanical procedure. The model is shown an example, its output is compared with the example's output, and its parameters are adjusted to bring the two closer. The procedure does not ask whether the example is true, sensible, or safe. It adjusts the model towards whatever it is shown.

Model builders do add their own checks around this procedure. They filter the data before training and test the model afterwards. Those checks are separate, human-designed steps, and they catch only what they were designed to catch. The learning step itself accepts everything.

2. What a model could be trained to do instead

Since the procedure learns whatever it is given, a person with bad intentions can feed in data with an outcome of their own in mind. The person doing this is called the attacker.

Two examples show the range.

In the first example, the model is trained so that, in certain situations, it passes information to the attacker. In the second, it is trained to be rude. A company that released a rude support assistant would lose customers, which may be all the attacker wants.

Both use the same method: examples of the unwanted behaviour are added to the training data, and the model learns them like any other examples.

One clarification about the first example. A model by itself only produces text. It can move data somewhere only when it is connected to tools that take actions, such as sending email or calling other systems. Many current AI systems have such tools, and for those systems this risk is real.

3. Where the attack is placed

The attack is not placed in the prompt. It is placed in the training data.

This distinguishes data poisoning from attacks that happen while a model is in use, where someone types or hides misleading instructions in the text a model reads. Those attacks act during inference and leave the model itself unchanged. A later lesson covers them.

A poisoned model carries the problem inside its parameters. Every copy of the model has it, and it stays until the model is retrained or replaced.

4. Why it is called poisoning

Nobody hands the model a file labelled as poisoned. The harmful material goes into the supply instead.

The name comes from the picture of a water supply. A person who wants to harm a town does not have to give each resident a poisoned glass. Putting something into the reservoir is enough, because the town draws its water from there. The picture maps as follows: the reservoir is the pool of text that models are trained on, the poison is the planted material, and the town is every model trained on that pool. The picture stops being accurate in one way: poison in water harms whoever drinks it, while planted text only matters if a training run happens to include it.

For a large language model, the reservoir is the open web. An attacker can publish articles, forum posts, or pages of code that contain the behaviour they want a model to learn, and leave them there to be collected.

5. A few pages among a very large number

flowchart LR
    W["Pages on the open web<br/>a few are the attacker's<br/>the rest belong to everyone else"] -. "may or may not be collected" .-> T["The next training run<br/>of somebody else's model"]
    T --> M["The model<br/>shaped by what it read"]
    M --> B["Behaviour its makers<br/>never intended"]

The diagram shows the path. On the left is the open web, with a very large number of pages. Four of them, say, belong to the attacker. A company collects web pages for the next training run of its model. The model is shaped by everything it read, including those four pages. The result can be behaviour that its makers never intended.

The dashed arrow is the uncertain part, and the whole attack depends on it. The attacker does not control the training run. They cannot be sure their pages will be collected, that the pages will pass the builder's filters, or that so few pages will have any effect among so many. They publish and hope.

6. A smaller data set is an easier target

Poisoning is more practical at a smaller scale.

The difference is one of proportion. A worked comparison with made-up numbers:

Pre-training:   10 poisoned pages among 10,000,000,000 pages
                = 1 in a billion

Fine-tuning:    10 poisoned rows among 2,000 rows
                = 1 in 200

The same ten items are a vanishing share of a pre-training set and a noticeable share of a fine-tuning set. A fine-tuning set is also the most recent thing the model learns from, and its examples are meant to outweigh older behaviour.

Access differs too. A pre-training set is assembled by a lab from the whole web, and an outsider can influence it only indirectly. A fine-tuning set is usually a file or a folder inside one company. Anyone who can edit that file can change what the model learns.

Size is not full protection for large models. Published experiments have reported that a fairly small number of planted documents, in the hundreds, was enough to teach the models tested a narrow hidden behaviour, and that the number needed did not grow much as the training set grew. So poisoning through the web is harder than poisoning a fine-tuning set, but it cannot be ruled out.

7. The insider: an unhappy employee

The person best placed to poison a fine-tuning set is an insider: someone who already has legitimate access inside the company.

Take an unhappy employee as the example. Three things come together:

1. They are already inside the company, so they pass every check that keeps strangers out.
2. They know that a fine-tuning run is coming, and they want to take revenge.
3. They add some data of their own to the training file. It is enough to shape how the model later behaves.

Nothing about step 3 looks unusual from outside. A few extra rows among thousands of examples do not stand out, and in many teams nobody reads the whole file again before training.

8. What the result can look like

Suppose the poisoned model is put to work inside the company with access to documents and the ability to send email. On an ordinary day, three things are true at the same time.

The first column is the data the model operates on in normal use. Nobody planted these files. They are the model's regular work.

The second column is the planted behaviour. A copy of the three files goes to an address outside the company. Here it is a made-up address at a rival firm.

The third column is everything the user sees. The request was handled, the summary came back, and the answer looks fine.

This is what makes poisoning dangerous. The model does its visible job correctly. The unwanted behaviour runs alongside the correct behaviour, not in place of it.

Planted behaviour is often built to appear only under a specific condition, such as a certain phrase, date, or kind of document in the input. That condition is called a trigger, and behaviour hidden behind a trigger is called a backdoor. In every test that does not contain the trigger, the model behaves normally, so ordinary testing does not reveal the problem.

9. Reducing the risk

No single measure removes the risk. The usual precautions work on different parts of the path.

The last two rows matter because the earlier ones can fail. A model that cannot send email cannot mail files to anyone, whatever it was trained to do.

Interview points

• Data poisoning is the deliberate insertion of harmful examples into a model's training data so that the trained model behaves in a way its builders did not intend.
• It works because the training procedure adjusts the model towards whatever data it is given and does not judge whether the data is reasonable.
• The attack is placed in the training data, not in the prompt. It changes the model's parameters, so it affects every user and lasts until the model is retrained or replaced.
• Poisoned behaviour can range from stealing data to being unpleasant to users. Stealing data requires that the model is connected to tools that can send data somewhere.
• For web-scale pre-training, an attacker publishes pages and hopes they are collected. The attacker controls neither the collection nor the effect.
• Fine-tuning sets are easier targets: they are small, so a few rows are a meaningful share, and they sit inside one organisation where an insider can edit them.
• A poisoned model can keep doing its visible job correctly while the planted behaviour runs alongside, which makes detection hard.
• A backdoor is planted behaviour that appears only when a specific trigger is present in the input, so normal tests pass.
• Defences combine access control and review for training data, knowledge of data sources, filtering, testing of the trained model, restricted tool permissions, and monitoring of the running system.

Summary

• Training assumes its data is useful and reasonable, and nothing in the procedure checks.
• Data poisoning means feeding in data with a harmful outcome in mind.
• The attack is in the training data, not in the prompt.
• On the open web, poisoned pages are left to be collected by somebody else's training run.
• A full training run is an ocean, and a fine-tuning run is a bucket, so a few rows matter far more in fine-tuning.
• An insider, such as an unhappy employee, can add rows to a fine-tuning set.
• A poisoned model can leak data while its answers still look fine.