Synthetic Data

Fine-tuning needs examples: hundreds or thousands of inputs paired with the outputs you want. For many jobs, nobody has such a collection. A team may know exactly what it wants a model to do and still have no file of examples to train it on.

Synthetic data is the usual way out. It is training data that was produced on purpose, most often written by an AI model, instead of being collected from text that people wrote for other reasons. The term comes up in many discussions of large language models, and most often around fine-tuning.

This lesson explains why data runs short, when you need synthetic data and when you do not, and how it is made in practice. After it you can say what synthetic data is, describe the steps for producing it, and explain why people check it before it is used.

1. Why new training data is hard to find

The large AI labs pre-trained their models on the public internet, as an earlier lesson described. That source has largely been used.

Reading the same text a second time teaches a model little that is new. New writing does appear online every day, but slowly compared with what was already collected. So for the next model, or the next improvement, fresh material has to come from somewhere else.

2. The exception: material only you have

There is one kind of data the labs never had: yours. Suppose you are fine-tuning a model to be very good with your organisation's internal legal documents.

A crawler is a program that visits web pages automatically and saves their text. It is how the labs collect public data. A crawler can only reach what is published on the open web.

The two files on the right are examples of private data: a services agreement with one customer and the fourth version of a confidentiality template. They sit on a shared drive inside the organisation. No crawler ever saw them, so no public model was trained on them. Their wording, clauses, and habits are new to any model, which is exactly what makes them useful for fine-tuning.

3. It depends on how much you have

Whether you need synthetic data comes down to one question: how large a collection of your own examples you can point to.

The first row needs no synthetic data. Real documents are already there.

The second and third rows often occur together, and that combination is the problem case. The job is clear, but there is nothing to train on.

4. The idea: a model writes the missing data

When there is nothing to hand to the fine-tuning tool, a model can write the data you do not have.

This sounds circular: a model produces the training material for a model. It works in practice, for a reason that the next sections make concrete. The writing model does not invent knowledge out of nothing. It is given real source material, such as documentation, and turns that material into many examples in the form that training needs.

5. A worked case: one programming library

Suppose the goal is a model that helps programmers. It does not need to get better at code in general. It should get better at one particular package of code that a company's software depends on.

Here is a small file from that company's software. It is written in Python, a common programming language. You do not need to read Python to follow the lesson. The explanation is below the block.

# app/checkout.py
from tideway import Ledger, CreditNote

ledger = Ledger(currency="EUR")

ledger.post(
    CreditNote(order="A-4471", amount=18.50)
)

The name tideway is made up. It stands for whatever package sits at the centre of a real codebase. Line by line:

• The first line, starting with #, is a label that gives the file's name and location. The computer ignores it.
• The line starting with from fetches two ready-made building blocks, Ledger and CreditNote, out of the tideway package.
• The next line creates a ledger (a book of accounts) that works in euros.
• The last lines record a credit note of 18.50 for order A-4471 in that ledger.

The programmer did not write the code that makes a ledger work. They used what tideway provides.

6. What a library is

A library, also called a package, is code you pull into your program instead of writing it yourself.

pip install tideway is the command a Python programmer types to download and install a package. A "call" is one use of a building block from the library, such as ledger.post(...) above.

The right-hand column is the important one for this lesson. A library saves work, but only for someone who knows how to use it. That is true for a human programmer, and it is true for a model that is asked to write code with the library.

7. When the library is newer than the model

A model knows only what was in its training data, and training ends on a certain date. Suppose tideway was published after the model finished training.

You:    How do I post a credit note with tideway?

Model:  I have nothing on a package by that name.

The model cannot help, because the library did not exist when it learned. A less careful model might do something worse and invent calls that look plausible but do not exist.

One solution is to attach the library's documentation to the request. The documentation becomes part of the context, and the model can answer from it. This works, but the model keeps nothing between conversations. So the documentation has to be attached again in the next conversation, and in the one after that. Every time, it uses up tokens, which costs money and takes space in the context window.

8. The other solution: change the model, not the prompt

The second solution moves the knowledge into the model's weights, where it does not have to be sent again.

• Fine-tune a specific model for this one package.
• So that it knows the library already, with nothing to attach.
• Built from examples of correct usage, produced on purpose.

The third point is where the gap appears. Fine-tuning needs many examples of questions about tideway with correct answers and correct code. For a new library, almost none exist. Nobody has written thousands of forum answers about it yet. All that exists is the documentation.

The two solutions compared:

9. How the examples are produced

The examples are made in a short pipeline.

flowchart LR
    A["tideway-docs.md<br/>the documentation you have"] --> B["An AI model<br/>asked to produce examples"]
    B --> C["Examples<br/>of correct usage"]
    C --> D["Humans<br/>review and adjust them"]
    D --> E["Fine-tuning data"]

1. Start with the documentation you have, here a file called tideway-docs.md.
2. Give it to a capable AI model with an instruction such as "write 500 realistic questions a programmer might ask about this library, each with a correct answer and working code".
3. The model produces examples of correct usage.
4. People who know the library review the examples. They fix mistakes, remove bad ones, and adjust the rest.
5. Only after that review do the examples become fine-tuning data.

One generated example might look like this:

Question:  How do I record a refund of 18.50 for order A-4471?

Answer:    Create a CreditNote and post it to the ledger:
           ledger.post(CreditNote(order="A-4471", amount=18.50))

The model that writes the examples can do so because the documentation is in its context while it writes. It is rephrasing and combining real information, in many variations, into the question and answer form that training needs.

The human step is there for a reason. A model can state wrong things with confidence. If a wrong example goes into the training data, the fine-tuned model learns the mistake and repeats it. Review is the point where such errors are caught.

10. What the term means

In one sentence, as the term is used around language models: synthetic data is training data written by a model and checked by people.

Two additions make the picture complete.

First, the general meaning of the term is wider. Synthetic data is any data that was created artificially instead of being recorded from real events or real writing. In other fields it can be made by simulations or by programs that follow fixed rules. Human checking is good practice, not part of the definition, and some synthetic data is used after only automatic checks.

Second, the AI labs use synthetic data too. The previous lessons mentioned that post-training adds artificially generated examples. That is the same technique at a larger scale.

Synthetic data has limits:

• Errors spread. Mistakes by the writing model end up in the trained model unless review catches them.
• Variety can be low. A model tends to phrase things in similar ways, so generated examples can be less varied than real ones. Asking for different styles, difficulty levels, and edge cases helps.
• The source sets the ceiling. Examples built from documentation cannot contain knowledge that the documentation lacks.

Interview points

• Synthetic data is training data that is generated on purpose, usually by an AI model, instead of being collected from existing human-written material.
• The need arises because the useful public text has largely been used for training already, and because many specific tasks have no existing collection of examples.
• Private documents were never crawled, so they still hold information that public models have not seen. With enough of them, fine-tuning can use them directly and no synthetic data is needed.
• Synthetic data fits the case of a narrow task with no examples: a model generates the examples from source material such as documentation.
• A typical pipeline is: source documents, a model generating examples of correct usage, human review and correction, then fine-tuning.
• Human review matters because errors in generated examples are learned by the fine-tuned model.
• For knowledge that is newer than a model's training, there are two options: attach the documentation to every request, which costs tokens each time, or fine-tune so that the knowledge sits in the weights.
• Synthetic data can lack variety and cannot exceed the quality of its source material.

Summary

• Fine-tuning needs examples, and often none exist.
• Public internet text has mostly been used already. Private documents have not.
• With many documents of your own, fine-tune on those directly.
• With a narrow task and no examples, have a model write the examples.
• A library that is newer than the model is a typical case: the model knows nothing about it.
• The examples are generated from documentation, then reviewed and adjusted by people.
• Synthetic data is training data written by a model and checked by people.