Structured Outputs

A model generates tokens, which means it generates text. When you chat with a model, that text is written for you: sentences, paragraphs, a friendly closing line. You read it, understand it, and decide what to do next.

Now suppose the reader is not a person. A developer has written a program that sends each incoming email to a model and then acts on the model's answer. The program cannot understand a friendly paragraph. It needs the answer in a fixed form, with each piece of information in a known place.

An answer in such a fixed form is called a structured output. This lesson explains what that means, when it matters, what the most common format looks like, and why model providers now offer a special mode for it. After this lesson you can read a small piece of JSON, explain why a program needs it, and say what a structured output mode guarantees and what it does not.

1. The same model can write structured text

A model that writes ordinary sentences can also write structured output. Three points define the idea.

The middle row matters. Structured output is not a different kind of data coming out of a different part of the model. The model still produces one token after another. The tokens happen to spell out a strict format instead of flowing sentences.

2. When structure matters

Whether structure matters depends on who reads the answer.

In a chat app, a person reads the answer. People are flexible readers. Whether the model says "forward this to billing" or "this looks like one for the billing team", you understand both.

A program is not a flexible reader. It looks for specific things in specific places. If the thing is not there, the program fails or does the wrong thing.

3. An example: a program that sorts email

Suppose you are building software that handles incoming email. For every message, the program must do one of two things: store it, or forward it to a team.

flowchart LR
    A["An incoming email"] --> P["Your program"]
    P --> S["Store it"]
    P --> F["Forward it"]

The diagram shows an email arriving at the program, and two possible actions leaving it. The program takes one or the other, every time.

Deciding which action fits an email requires understanding the email, and that is a good job for a model. So the program sends each email to a model and asks what to do with it. The question for this lesson is what form the model's answer should take.

4. What the program needs

The program that receives the answer is ordinary software. It is not AI. It does not interpret or guess. It reads data, acts on it, and carries on with the next email.

Three facts follow from that.

"The same predefined shape" means the developer decided the form in advance, and every answer must match it. The program is written to look for a value called action, for example, and to check whether it says store or forward. If one answer puts the decision in the second sentence of a paragraph and the next puts it in the first, the program cannot find it.

5. Asking for a format

The simplest way to get structure is to ask for it. A prompt can tell the model to hand the data back in a certain format.

Here is the same answer written two ways. First, written for a person:

This one looks like a question about an invoice, so I would
forward it to the billing team. Let me know if you want me to
draft a reply as well.

Second, written for a program:

{
  "topic": "invoice",
  "action": "forward",
  "to": "billing",
  "reply_drafted": false
}

The second version is in a format called JSON, which stands for JavaScript Object Notation. JSON is one of the most common formats for passing data between programs. It is plain text with a few strict rules, and reading it does not require programming knowledge.

• The curly braces { and } mark the start and end of one group of data.
• Each line inside is a field. A field has a name on the left, a colon, and a value on the right.
• Names and text values are written in double quotes. Fields are separated by commas.
• A value can be text such as "billing", a number, or true or false.

Read line by line, the JSON says: the topic of the email is an invoice, the action to take is to forward it, the team to forward it to is billing, and no reply has been drafted.

Both versions carry the same answer. Only the second can be acted on by code. A program can read the action field, see forward, read the to field, see billing, and send the email there. In the first version, the same facts are spread through two sentences in wording that changes from one answer to the next.

A prompt that asks for this might say:

Classify the email below. Reply with JSON only, using exactly
these fields:
  topic          one word
  action         "store" or "forward"
  to             the team name, or null if the email is stored
  reply_drafted  true or false

The word null is JSON's way of writing "no value".

6. Why asking is not enough

Asking works most of the time. For software, most of the time is not good enough. A program that handles ten thousand emails a day will hit every rare failure.

Some ways a model can drift from the requested shape when it is only asked:

Each of these is a small slip for a human reader and a failure for a program.

This is why model providers no longer leave the matter to a polite request in the prompt.

With a structured output mode, the developer supplies a schema along with the request. A schema is a precise description of the required shape: which fields exist, what kind of value each one holds, and which values are allowed. The provider's system then makes the model's output follow that schema.

In broad terms, a common way to do this works as follows. A model writes one token at a time, and at each step it chooses from many possible next tokens. In structured output mode, the system removes from the choice any token that would break the schema. If the schema says the next thing must be the field name action, the model cannot write anything else there. So the result fits the schema by construction. This technique is called constrained decoding, because the step that turns the model's predictions into text (decoding) is restricted to what the schema allows.

At the time of writing, the major providers offer such a mode, often under the name "structured outputs". Some also offer a weaker option, often called "JSON mode", which makes sure the answer is correctly formed JSON but does not check it against a particular schema, so a field can still be missing or misnamed. The names and the exact strength of the guarantee differ between providers and change over time, so check the current documentation.

7. What the guarantee covers

A structured output mode guarantees the shape of a complete answer. It does not guarantee that the content is right.

{
  "topic": "invoice",
  "action": "forward",
  "to": "legal",
  "reply_drafted": false
}

This answer has a valid shape. Every field is present and every value is of the right kind. If the email was in fact a billing question, the answer is still wrong, because legal is the wrong team. The mode makes sure the program can read the answer. Whether the model judged the email correctly is a separate matter, and it has to be checked in other ways, such as testing the system on emails whose correct handling is known.

One more limit applies to the shape itself. The guarantee holds for an answer that the model finishes. If the answer is cut off, for example because it reached the maximum length the developer allowed, the text stops in the middle and is not complete JSON. So a careful program still checks that the answer arrived whole.

8. Structured output is still text

What comes back in structured output mode is still text. It is text that is not meant for a person to read.

The JSON above reaches the program as a string of characters, the same way a paragraph would. The program then runs a standard step called parsing, which reads the characters and turns them into data the program can use: a topic, an action, a team, and a true-or-false value. Parsing works only when the text follows the format exactly, which is the reason the shape matters so much.

So the term comes down to this: structured outputs are text written for machines, not for people.

Interview points

• A model always generates tokens. Structured output is text in a fixed, predefined shape, usually JSON, intended for a program to read.
• In a chat app a person reads the answer, so the form does not matter. When software consumes the answer, the form must be the same every time.
• Ordinary software does not interpret free text. It reads specific fields, so it needs a known set of field names and allowed values.
• JSON stores data as named fields with values inside curly braces. It is one of the most common formats for exchanging data between programs.
• Asking for JSON in the prompt works most of the time but can fail through extra prose, renamed or missing fields, unexpected values, or broken syntax.
• A structured output mode takes a schema and typically uses constrained decoding: at each step, only tokens that keep the output valid can be chosen, so a completed answer matches the schema. A plain JSON mode guarantees valid JSON but not a particular set of fields.
• The guarantee is about shape, not correctness. A valid answer can still contain a wrong value, so the content needs its own testing.
• The receiving program parses the text into data. Parsing fails on malformed text, which is why a guaranteed shape matters.

Summary

• Models generate text, and structured output is text in a fixed shape.
• People can read any phrasing, and programs need the same shape every time.
• JSON is the usual format: named fields with values.
• A prompt can ask for a format, but a request alone does not guarantee it.
• Providers offer a structured output mode that makes the answer follow a schema.
• The mode guarantees the shape of the answer, not the correctness of its content.