Hallucinations

A language model can write an answer that reads well, sounds sure of itself, and is wrong. It can describe a product that was never made, quote a rule that is not in the document, or give a date that is off by a year. This kind of error is called a hallucination.

Hallucinations used to be a large problem with AI models. They are a smaller problem now, because newer models make them much less often. They have not disappeared, and the reason is built into how a model writes text.

This lesson shows what a hallucination looks like, explains where it comes from, and gives a method for checking answers. After this lesson you can explain why a model's confident tone tells you nothing about whether it is right, and you can check an answer in a way that catches the errors that are hardest to see.

1. What a hallucination looks like

Here is a question about a thing that does not exist. The name "Brindle valve" is invented for this example.

[you]    What does the Brindle valve do?

[model]  The Brindle valve regulates the flow between the two
         stages of a pump. It has three moving parts and needs
         no external power.

There is no Brindle valve, and there never was one. The model still produced two neat sentences with specific details: what the valve regulates, how many moving parts it has, and how it is powered.

The important observation is about the wording. Nothing in the answer marks it as invented. It has the same grammar, the same level of detail, and the same calm tone as a correct answer about a real valve. A reader who does not already know the truth cannot tell the two apart by reading.

That is the definition. A hallucination is model output that sounds plausible but is false, unsupported, or made up. The word is borrowed from human experience, where it means perceiving something that is not there. The comparison is loose. The model is not perceiving anything. It is producing text that fits the pattern of an answer.

2. How that answer was written

Earlier lessons showed that a model writes one token at a time. A token is a small piece of text, often a word or part of a word. To write each one, the model looks at everything so far and works out which tokens are likely to come next.

Take the answer above at the moment when four words have been written.

flowchart LR
    P["The prompt<br/>What does the Brindle valve do?"] --> M["The model<br/>fills one position at a time"]
    M --> S["Written so far:<br/>The Brindle valve regulates ..."]
    S --> C["Candidates for the next position<br/>flow 0.38<br/>pressure 0.29<br/>steam 0.11"]

The model has written "The Brindle valve regulates" and has to choose the next word. It produces a list of candidates, each with a probability. A probability is a number between 0 and 1 that says how likely something is, so 0.38 means 38 chances in 100. In this made-up example:

The three numbers add up to 0.78. The remaining 0.22 is spread over many other, less likely words.

The model picks one of the candidates, most often a likely one. Here it picked "flow". Then the same process runs again for the position after "flow", and again after that, until the answer is finished. The lesson on temperature and sampling covers how the pick is made.

Look at why "flow" scored well. In the text the model was trained on, valves regulate flow, pressure, and steam. So after "valve regulates", those words are likely. None of this depends on whether a Brindle valve exists. The model is completing a familiar pattern about valves in general. Each next word is a weighted guess, and nothing more.

3. The one question a model asks, and the one it does not

An answer is written by answering the same small question again and again.

At every step, the model works out a likely next token. It answers that from the patterns stored in its weights, which hold what it learned from its training text.

At no step is there a separate check for truth. The model has no built-in stage that takes a finished sentence and compares it with the real world. A model can only compare its text with something if that something is in front of it, for example a document in its context or the result of a search.

For most questions this causes no trouble, because likely and true point the same way. After "The capital of France is", the most likely token is "Paris", and Paris is also the correct answer. The model learned from a great deal of accurate text, so likely continuations are usually right.

A hallucination happens in the gap between the two. When the model has little or no real knowledge about the subject, there is still always a most likely next token. So the model keeps writing, and what it writes has the shape of an answer without the facts behind it.

An everyday comparison is a student who must fill every blank on an exam and is not allowed to leave one empty. Where the student knows the material, the answers are right. Where the student does not, the answers are fluent guesses in the right style. The comparison stops being accurate in one place: the student knows which answers were guesses. A model has no reliable record of that.

4. Better than before, but not gone

Hallucinations are far less common than they were, and it is worth being exact about what changed.

Newer models are trained on more and better data, and they are trained to say "I do not know" or "I cannot find that" more often. Many products also give the model tools to look things up. All of this lowers the rate.

None of it changes how a model writes. It still predicts one token after another and still has no built-in truth check. A lower rate is not a rate of zero. A modern model asked about the invented Brindle valve will often reply that it knows of no such valve. It will not do so every time, and no one can promise which time it will fail.

5. Treat every answer as a claim to check

The practical conclusion follows directly. An answer from a model is a claim to check. It is not a fact to copy.

Fluency and confidence are not evidence. In people, a sure tone often goes with knowledge, so readers learn to trust it. In a model, the tone is produced by the same token prediction as everything else. A wrong answer and a right answer come out in the same voice.

How much checking is needed depends on what is at stake. A suggestion for a birthday message needs none. A dose of medicine, a legal deadline, a figure in a financial report, or a quotation attributed to a real person needs a check against a source you trust: the original document, an official website, or a qualified person.

Details that are specific are the ones most worth checking, because a model can produce them in a convincing form without any basis. Typical cases are names, dates, numbers, quotations, web addresses, and references to books, articles, or court cases.

6. Check answers one claim at a time

A fully invented answer such as the Brindle valve is often the easier kind to catch, because a quick search can expose it. The harder case is an answer that is mostly right.

Suppose you give a model your contract and ask about renewal and cancellation. It replies:

Your contract renews yearly. The notice
period is 30 days. Cancellation has to be
posted by letter.

This reads as one answer. It is three separate claims in three short sentences. To check it, take the answer apart and test each claim against the contract.

Two claims hold, and one does not. The contract says nothing about letters. The model added a detail that often appears in contracts of this kind, because it was a likely continuation.

A largely correct answer is exactly where a wrong detail goes unnoticed. The first two claims check out, so the reader starts to trust the answer and stops checking before the third. The error can have real consequences. A reader who trusts the answer posts a letter, although the contract never asked for one, and may miss the 30-day notice period while the letter is in the post.

So the method is:

1. Split the answer into its separate claims.
2. Check each claim against the source, not the answer as a whole.
3. Treat a claim that cannot be found in the source as unsupported, however reasonable it sounds.

Asking the model to say where each claim comes from, such as a clause number or a page, makes this checking faster. The reference itself must be checked too, because a model can also invent a reference.

Interview points

• A hallucination is model output that is plausible in form but false or unsupported, such as a description of a product that does not exist.
• Hallucinated text carries no marker. It has the same fluency and confidence as correct text, so tone cannot be used to detect it.
• The cause is the generation mechanism. A model picks each next token by probability, based on patterns learned in training, and has no built-in step that checks a statement against the world.
• Likely and true usually agree, which is why models are right most of the time. Hallucinations appear where the model lacks knowledge but a likely continuation still exists.
• Newer models hallucinate much less often than the first generations. The mechanism has not changed, so the rate is lower but not zero.
• Model output should be treated as a claim that needs verification in proportion to the stakes, not as an established fact.
• Mostly correct answers are the highest risk, because correct claims build trust that carries a wrong detail through.
• The checking method is to split an answer into individual claims and test each one against a trusted source.
• Specific details such as names, numbers, dates, quotations, and citations deserve the most attention, and citations must themselves be verified.

Summary

• A hallucination is a confident, fluent answer that is wrong or invented.
• Models write by predicting likely tokens, not by checking facts.
• Nothing in the wording separates a hallucinated answer from a correct one.
• Hallucinations are rarer in modern models but still happen.
• Fluency and confidence are not evidence that an answer is true.
• Check answers claim by claim against a source you trust.
• The most dangerous error is one wrong detail inside a mostly correct answer.