Retrieval-Augmented Generation (RAG)

A small online shop wants an assistant that answers staff questions such as "what is our refund window on damaged goods". A general chat model cannot answer that. The shop's rules are in the shop's own handbook, and the model was never trained on it. Asked anyway, the model either says it does not know or produces a typical-sounding policy, which is a hallucination.

The previous lesson gave one fix: attach the right document, so the facts are in the prompt. That works for one question. It does not work for an assistant that hundreds of people use all day, because nobody wants to find and attach the right file for every message.

Retrieval-augmented generation, shortened to RAG, automates that step. It has the same goal as grounding: get the relevant facts into the prompt. The difference is who puts them there. With RAG the software finds the right passages and adds them, on every message, without being asked. After this lesson you can describe each step of a RAG system, explain what the model receives, and say what is needed to build one.

1. The name, word by word

The three words of the name are the three steps, in order.

1. Retrieval. The software searches a collection of documents and pulls out the passages that match the question. To retrieve means to fetch.
2. Augmented. To augment means to add to something. The software adds the fetched passages to the prompt.
3. Generation. The model generates its answer, token by token, from the enlarged prompt.

So RAG is generation that has been augmented by retrieval: the model writes as usual, but with fetched facts placed in front of it first.

2. Start with documents the model never saw

RAG begins with material that was not in the model's training data. Take one shop's own handbook:

These files are private. No public model has read them. The collection is often called a knowledge base.

The handbook has been indexed. Indexing means the documents were processed and stored ahead of time in a form that can be searched quickly, in the way a book's index lets a reader find a topic without reading every page.

Something has to connect this handbook to a model. There are two common options:

• an AI agent that you built yourself, or
• an existing tool such as ChatGPT that has been connected to the document store.

In both cases the connection lives in the software around the model, which earlier lessons called the harness. The model does not reach out for anything. It has no way to open the handbook. It reads what the software puts in front of it.

3. What happens on every message

When someone sends a message to a RAG system, four things happen that the person never asked for.

flowchart LR
    U["Your prompt<br/>what you typed"] --> S["The software<br/>your agent, or a tool<br/>already connected to a store"]
    S -- "step 1: query on every message" --> D["The database<br/>your own documents"]
    D -- "step 2: matching passages" --> S
    S -- "step 3: rebuilds the prompt" --> P["One prompt, rebuilt<br/>the sources, pasted in for you<br/>+ what you typed"]
    P -- "step 4: sent" --> M["The model"]

1. The software queries the database. It takes your message and searches the document store for passages related to it. This happens on every message.
2. The database returns matches. The most relevant passages come back, for example the paragraph on damaged goods from returns-policy.md.
3. The software rebuilds the prompt. It pastes the returned passages into the prompt, next to what you typed.
4. The rebuilt prompt goes to the model. Only at this point does the model see any of it.

None of these steps happens because you requested it. You did not say "search the handbook". You did not choose which file to look in. You did not paste anything. The question you typed is the only part of the final prompt that you wrote.

That is the difference from the grounding of the previous lesson. There, a person attached a file by hand. Here, the software does the finding and the attaching.

4. What the model receives, with and without retrieval

Here is the same question, sent two ways.

Without retrieval, the model receives only the question:

What is our refund window
on damaged goods?

After retrieval, the model receives this:

-- from returns-policy.md --
Damaged goods may be returned
within 30 days of delivery.
Photographs are required; there
is no restocking fee.
-- end of source --

What is our refund window
on damaged goods?

In the first case the model has nothing to go on. It does not know who "our" refers to, so any specific answer would be a guess.

In the second case the answer is in the prompt. The model generates its tokens against the whole second block, not only the question at the bottom. It can reply that damaged goods can be returned within 30 days of delivery, that photographs are required, and that there is no restocking fee. It can also name the source, returns-policy.md, because the label is in the prompt too.

The model did nothing new. It did what it always does: read the context and write a likely continuation. RAG works by changing what is in the context, not by changing the model.

5. Why the whole handbook is not sent every time

A fair objection is that the software could paste all four files into every prompt and skip the search. For 18 pages that would work. For a real company it does not, for three reasons.

Size. A company may have thousands of documents. They do not fit in a context window.

Cost and speed. Every token in the prompt is paid for and takes time to process. A worked case with made-up numbers: if the full handbook is 12,000 tokens and the one relevant passage is 300 tokens, sending the passage uses 40 times fewer input tokens on every question.

Focus. A model tends to answer better when the prompt holds the relevant text and little else. Large amounts of unrelated text make it more likely that the model overlooks or mixes up details.

Retrieval solves all three by sending only the few passages that matter for this question.

6. RAG compared with other ways to give a model knowledge

RAG is the usual choice when the knowledge is private, large, or changes often. Updating a RAG system means updating documents, which is far cheaper than training a model.

7. What it takes to set RAG up

RAG is not a switch in a settings panel. It needs specific pieces.

Software with a retrieval step is required. Some program has to run the search on every message and paste the results into the prompt. Many chat tools do neither out of the box. A plain chat window sends your text and nothing more.

An agent of your own may be required. A ready-made tool cannot know about a document store that you set up yourself. So teams often build their own agent around their documents and connect it to their store. Some existing products can be connected to your documents instead, which avoids building an agent from scratch.

A place to keep the documents is required. The last piece is somewhere for all the sources to live, in a form that can be searched by meaning. The most common choice is a so-called vector database, which is the subject of the next lesson.

8. Limits of RAG

RAG reduces hallucinations because the model answers from supplied text. It does not make answers certain.

• Retrieval can miss. If the search returns the wrong passage, or no passage, the model answers from poor material or from training data. The quality of the answer is capped by the quality of the search.
• The model can misread. Even with the right passage in the prompt, the model can summarise it wrongly.
• The documents can be wrong or out of date. RAG repeats what the store contains.

So answers from a RAG system are still claims to check. Showing the source name with each answer, as in the returns-policy.md label above, makes that check quick.

Interview points

• RAG stands for retrieval-augmented generation: retrieve relevant passages, add them to the prompt, and let the model generate from the enlarged prompt.
• RAG is a form of grounding. The goal is the same, facts in the prompt. The difference is that software inserts the facts automatically on every message.
• The model is unchanged. It has no access to the document store and sees only the passages the software pasted into the prompt.
• The flow is: user prompt, query to the document store, matching passages returned, prompt rebuilt with sources plus the question, then one request to the model.
• RAG suits knowledge that is private, large, or frequently updated, because changing a document is cheaper than retraining or fine-tuning a model.
• Sending only relevant passages keeps prompts inside the context window, lowers token cost, and keeps the model focused.
• A RAG system needs software with a retrieval step, a searchable store for the documents, and often a custom agent connected to that store.
• Answer quality depends on retrieval quality. A wrong or missing passage leads to a wrong or ungrounded answer.
• Labelling passages with their source lets the model cite them and lets people verify the answer.

Summary

• RAG fetches relevant text from your documents and adds it to the prompt.
• It happens automatically, on every message, inside the software around the model.
• The only part of the final prompt the user writes is the question.
• The model answers from the fetched text, so it can use facts it was never trained on.
• Setting up RAG needs a retrieval step, a document store, and often a custom agent.
• The document store is usually a vector database.
• RAG lowers the risk of hallucination but depends on the search finding the right passage.