Vector Databases and Embeddings

The previous lesson described retrieval-augmented generation (RAG): software that fetches relevant passages from your documents and adds them to the prompt. It left one question open. The software has to find the passage that is related to the user's question, and "related" is hard for a computer.

Take a customer who asks, "Can I still send back the jacket I ordered in March?" The passage that answers this is titled "Returns policy" and talks about "items" and a "returns window". The question never uses the word "returns". A search that looks for matching words can miss the right passage completely.

A vector database solves this. An ordinary database finds the rows that match. A vector database finds the rows that are close in meaning. It does this with embeddings, which are lists of numbers that stand for the meaning of a piece of text. After this lesson you can explain what an embedding is, how documents get into a vector database, how a question finds its nearest passages, and where this fits in a RAG system.

1. What a vector database stores

A vector database still holds rows, like any database. The difference is one added column.

Here is an ordinary table of handbook entries:

id   title              body
--   ----------------   ----------------
14   Returns policy     "Items may be..."
15   Delivery windows   "Orders placed..."
16   Parking            "Staff spaces..."

Here are the same rows as a vector database sees them:

id   embedding
--   -----------------------------
14    0.41  -0.07   0.96   0.12  ...
15    0.38  -0.11   0.88   0.19  ...
16   -0.62   0.44  -0.05   0.71  ...

The values are illustrative. The text is still stored. What is added is a row of numbers that stands in for the text.

That row of numbers is a vector, which is the mathematical word for a list of numbers. When a vector represents the meaning of a piece of text, it is called an embedding. A real embedding is long. It commonly holds several hundred to a few thousand numbers.

Look at the three rows. Rows 14 and 15 have similar numbers: 0.41 and 0.38, then -0.07 and -0.11, then 0.96 and 0.88. Both entries are about customer orders. Row 16, about staff parking, has very different numbers. This is the property that makes embeddings useful: texts with related meanings get similar numbers.

2. Where embeddings come from

The numbers are produced by an embedding model. This is a kind of AI model that takes a piece of text as input and gives back one vector as output. It is different from a chat model, which gives back text.

An embedding model is trained on a very large amount of text so that passages with similar meaning end up with similar vectors. Nobody chooses the numbers by hand, and the individual numbers have no simple meaning that a person can read. Only the comparison between two vectors is informative.

Embeddings are different from the token IDs of the lesson on tokens. A token ID is a label, such as the number 8421 for the piece "large". Two tokens with nearby IDs have nothing in common. An embedding is a position, and nearby positions do mean related content.

3. How documents get into the database

Picture a shelf of documents that nobody can search properly: handbook.pdf, returns.pdf, and shipping.pdf. They are meant to be the company's knowledge base. Getting them into a vector database takes three steps.

flowchart TB
    A["Documents<br/>handbook.pdf, returns.pdf, shipping.pdf"] --> B["Extracted passages<br/>the text pulled out and cut into pieces,<br/>each about one thing"]
    B --> C["One embedding per passage<br/>stored beside the passage"]

1. Start with the documents. These are the files that make up the knowledge base.
2. Extract passages. The text is pulled out of each file and cut into short pieces, each about one thing. A piece is called a passage or a chunk, and the cutting is called chunking. A chunk is often a paragraph or a few paragraphs.
3. Create one embedding per passage. Each passage goes through the embedding model. The resulting vector is stored beside the passage. It records what the passage is near in meaning.

Documents are cut into pieces for a reason. One embedding for a whole 40-page handbook would blur returns, shipping, and parking into a single position. One embedding per passage gives each topic its own position, so a search can return the one paragraph that answers a question.

This preparation is done once, ahead of time, and repeated only when documents change. It is the indexing step from the previous lesson.

4. Closeness on a map

The mathematics matters only to people who build these systems. The picture is enough for everyone else.

Think of each embedding as a position on a map. Every stored passage gets a position. Suppose five short texts are placed:

The two delivery texts sit close together. The two returns texts sit close together. The parking text sits far away, because it is about something else.

The first pair shares almost no words. "When will my order arrive" and "how long does delivery take" have different vocabulary and the same meaning. The embedding model places them together anyway.

A map has two directions, left-right and up-down. Real embeddings have many more directions, one for each number in the vector, which is why they cannot be drawn. The idea of "near" and "far" works the same way.

Record such a position for every passage, and the result is a vector database.

5. Related becomes something measurable

Once every passage has a position, "related" stops being a vague word. It becomes a distance, and a distance can be calculated.

A small worked case with made-up positions on a flat map. Three passages are stored:

returns window:   (8, 2)
delivery times:   (2, 8)
staff parking:    (-6, -5)

A question arrives and is given the position (7, 3). The straight-line distance between two points is found by taking the difference in each direction, squaring both, adding them, and taking the square root.

to returns window:  differences 1 and 1    ->  square root of (1 + 1)     = 1.4
to delivery times:  differences 5 and 5    ->  square root of (25 + 25)   = 7.1
to staff parking:   differences 13 and 8   ->  square root of (169 + 64)  = 15.3

The returns passage is nearest, at a distance of 1.4, so it is the one handed back. No word matching was involved. Real systems do the same kind of calculation with hundreds of numbers per vector. Many use a related measure called cosine similarity, which compares the directions of two vectors.

This is the gain over ordinary search. Ordinary keyword search matches the words that were typed. Similarity search finds what sits near the prompt in meaning, whichever words it uses.

Keyword search remains better for exact strings such as an order number, so some systems combine both.

6. Why a model needs this

Go back to the customer's question:

Am I still inside the window to send back
the jacket I ordered in March?

Nothing in the model's training contains this shop's returns policy, and the model cannot look it up. So something has to fetch the passage that answers the question before the model is asked anything.

That raises the matter of which passage to fetch. The answer is: the one related to the question. And "related" is exactly what a vector database can measure.

7. How a question finds its passages

At question time, the same conversion is run once more, this time on the question.

flowchart LR
    Q["Your prompt<br/>Can I still send back<br/>the jacket I ordered in March?"] --> E["Embedding model<br/>the same one used<br/>for the stored passages"]
    E --> V["Position of the prompt"]
    V --> D["Vector database<br/>finds the stored positions<br/>closest to it"]
    D --> R["Nearest entries handed back<br/>how do I send it back?<br/>the returns window"]

1. The prompt goes through the embedding model and becomes a vector. It must be the same embedding model that was used for the stored passages, because positions from two different models cannot be compared.
2. The prompt now has a position in the same space where all the passages already sit.
3. The database finds the stored positions closest to it. In this example, the nearest are a passage about sending an item back and the passage on the returns window. Passages about parking are far away and are ignored.
4. The nearest entries are handed back. The software decides how many to take, often a handful.

The rule is that the nearest wins. A vector database is built to do this search quickly, even across millions of stored passages. It keeps an index of the positions so that it does not have to measure the distance to every stored passage one by one. Many vector databases accept a result that is very close to the nearest, instead of the exact nearest, in exchange for that speed.

8. From numbers back to text

The numbers were only ever for finding. A language model does not read embeddings from the database. What travels onward is the text of the passages that were found.

What came back from the database:

returns.pdf - passage 12
"Items may be returned within the
 window stated on your order
 confirmation..."

shipping.pdf - passage 4
"Orders placed before..."

The prompt that is sent to the model:

Use only the sources below.

[sources]
returns.pdf p12 - "Items may be..."
shipping.pdf p4 - "Orders placed..."

[question]
Can I still send back the jacket
I ordered in March?

The software builds one prompt with the sources folded in. It has three parts: an instruction to use only the sources, the retrieved passages with labels saying where each came from, and the original question. This is the rebuilt prompt from the RAG lesson.

The model never speaks to the database. The software queries the database, receives text, and writes that text into the prompt. The model reads the prompt.

9. The full stack

The pieces from the last three lessons sit on top of each other.

flowchart TB
    A["The documents you already keep<br/>the handbook, the policies, the wiki"] --> B["A vector database over them<br/>every passage, and where it sits<br/>relative to the rest"]
    B --> C["A RAG system on top<br/>fetches the nearest passages<br/>for whatever is asked"]
    C --> D["An answer with sources under it<br/>grounded in your own material"]

Read the diagram from the top down. Each layer is built on the one before it, and the documents are the base. The vector database stores every passage together with its position. The RAG system uses the database to fetch the nearest passages for each question. The result at the end is an answer grounded in the organisation's own material.

The key property is in the second layer. A vector database is indexed by how things relate, not by the words they contain. A plain table of text cannot do that, and it is the whole reason a vector database sits under a RAG system.

Closeness is not correctness. The nearest passage may be on the right topic and still not contain the answer, and a poorly cut passage may miss the key sentence. The quality of chunking and of the embedding model sets a limit on how good the final answers can be.

Interview points

• A vector is a list of numbers. An embedding is a vector that represents the meaning of a text, produced by an embedding model.
• Texts with similar meaning have embeddings that are close together, even when they share no words.
• A vector database stores each passage with its embedding and can quickly find the stored embeddings nearest to a query embedding.
• Ingestion has three steps: extract text from documents, cut it into chunks that each cover one topic, and create one embedding per chunk.
• Chunking matters because one embedding for a whole document blurs many topics into one position.
• At query time the question is embedded with the same embedding model, and the nearest chunks are returned. Embeddings from different models are not comparable.
• Closeness is measured with a distance or similarity calculation, commonly cosine similarity.
• Embeddings are used only for finding. The language model receives the text of the retrieved chunks inside the prompt and never talks to the database.
• Keyword search matches words, and similarity search matches meaning. Keyword search is still stronger for exact identifiers, so the two are sometimes combined.
• The stack is documents, then a vector database, then a RAG system, then a grounded answer with sources.

Summary

• An embedding is a list of numbers that stands for what a text means.
• A vector database stores passages together with their embeddings.
• Related texts sit close together, so relatedness can be measured as distance.
• Documents are cut into passages, and each passage gets one embedding.
• A question is converted the same way, and the nearest passages are returned.
• The passages go into the prompt as text. The model never queries the database.
• A vector database is the storage layer underneath most RAG systems.