APIs for AI Models

Most people meet AI models through a chat app: a window with a text box, made by the company that built the model. That window is not the only way in. The same models can be reached from a program that you write yourself.

The way in for programs is called an API, which stands for application programming interface. The lesson on tokens mentioned the API as the pay-per-token way to use a model. This lesson shows what it is and what using it looks like: your code, the provider's model.

The lesson covers the difference between a chat app and your own program, what a request and a response look like, which settings a request can carry, how a request can offer the model a list of tools, and how chat apps themselves are built on the same API. After this lesson you can read a simple API request, name its parts, and explain what a developer gains by using the API instead of a chat app.

1. A chat app is somebody else's program

Whether you use a chat app or write your own program, the model underneath is the same. What differs is the software that sits on top of it, and who wrote that software.

In a chat app, another company made the decisions. They designed the window. They chose the hidden instructions and settings. They decided which tools the model has, such as web search or file reading.

In your own code, those decisions are yours. The program might have its own screen, or no screen at all. A program that sorts incoming email overnight has no window and no user typing. It sends text to the model and uses what comes back.

2. The model stays on the provider's machines

When you write a program that uses a hosted model, you do not receive a copy of the model. A large model needs specialised and expensive hardware, so the provider, the company that offers the model, keeps it running on its own machines. Your program runs somewhere else: on your laptop, your phone, or your company's servers.

Something has to connect the two.

flowchart LR
    A["Your software<br/>(runs where you run it)"] --> B["The API"]
    B --> C["The model<br/>(runs on the provider's machines)"]
    C --> B
    B --> A

The diagram shows your software on the left, the model on the right, and the API in the middle. A request travels from your software through the API to the model, and the answer travels back the same way. For a hosted model, the API is the only way across.

An application programming interface is a published way for one program to ask another for something and get an answer. "Published" means the provider documents it: what to send, where to send it, and what will come back. Any programmer who follows that description can use the model.

An everyday comparison: a restaurant kitchen is closed to customers. The menu and the waiter are the published way to get food out of it. You order in the form the menu allows, and a dish comes back. The kitchen is the model, and the menu and waiter are the API. The comparison stops being accurate at one point: a waiter can handle an unusual request by asking the chef, while an API accepts only requests in exactly the documented form.

Some models can be downloaded and run on your own machine, as the lesson on quantization described. Even then, programs usually talk to the local model through an API of the same style.

3. An endpoint is an address

To use the API, a program sends a message to a web address and reads the reply. That address is called an endpoint.

Your software sends:

POST https://api.provider.example/v1/messages

{
  "input": "Summarise this support ticket."
}

And gets back:

200 OK

{
  "output": "Customer cannot log in since the password reset."
}

The pieces, one at a time:

• POST is the standard word on the web for "here is some data, please process it". It is the same kind of message a browser sends when you submit a form.
• https://api.provider.example/v1/messages is the endpoint. This one is a made-up address. Each provider publishes its real one.
• The block in curly braces is the body of the request, written in JSON, the data format from the lesson on structured outputs. It has one field, input, which holds the prompt.
• 200 OK is the standard web status that means the request succeeded.
• The second block is the body of the response. Its output field holds the text the model wrote.

The field names in this lesson, such as input and output, are simplified for illustration. They are not the field names of any real provider. Real APIs use their own names, wrap the prompt in a slightly larger structure, and also require a field that says which model should answer. The pattern is the same: text goes in as a field, and text comes back as a field.

A real request also carries an API key. This is a secret code that the provider issues to each customer. It tells the provider who is sending the request, so that the usage can be billed to the right account, per token. Anyone who has the key can spend the account's money, so it must be kept private.

Whether a prompt is typed into a chat box or posted from a program, the thing that answers is the same model.

4. Settings are fields in the request

A chat window shows a text box and little else. The settings that control how the model answers are chosen by the app and kept out of view. In an API request, each of those settings is one more field.

{
  "input": ticket_text,
  "temperature": 0,
  "max_output": 200,
  "stop": ["---"],
  "format": "json"
}

Here ticket_text stands for the text of a support ticket that the program fills in. The other four fields are settings, usually called parameters. Their names are simplified too. Each provider has its own names for these settings, and not every model accepts every one of them.

Temperature was covered in the lesson on temperature and sampling. A low value makes the model pick the most likely next token almost every time, which suits a program that needs consistent results. A higher value allows more variety, which suits creative writing.

Maximum output puts a ceiling on the length of the answer. It protects against a runaway reply and caps the cost of the request, since output tokens are billed. If the ceiling is reached, the answer is cut off at that point, even in the middle of a sentence.

Stop gives one or more pieces of text, called stop sequences. When the model produces one of them, generation ends there. This is useful when the program wants only the first item of something the model might continue.

Format asks for structured output, so that the program can read the answer reliably.

A program can set these differently for every call. One call can summarise at temperature 0 with a 200-token limit, and the next can write a slogan at a higher temperature with a 50-token limit.

5. A request can carry a list of tools

A request can carry more than words and settings. It can carry a list of actions the model is allowed to ask for. Each entry is a tool definition: a name and a description of when the tool applies.

What the program puts in the request:

tools: [
  { name: "lookup_order",
    when: "the user names an order number" },
  { name: "issue_refund",
    when: "an order arrived faulty" }
]

What comes back when a user writes in:

user: "Order 4471 turned up broken."

use_tool: lookup_order
    with: { order: "4471" }

The request offered two tools. The user's message mentions an order number and a broken item. The model's reply is not a sentence for the user. It is a structured request to use the lookup_order tool with the order number 4471 filled in.

Nobody wrote a rule that says "if the message contains a four-digit number, call lookup_order". The model read the user's message, read the tool descriptions, and decided which tool fits. It also pulled the order number out of the sentence.

One point is often misunderstood. The model does not run the tool. It only writes out which tool it wants and with what values. Your program receives that reply, performs the lookup in your order system, and sends the result back to the model in a follow-up request. The model then continues, for example by asking for issue_refund or by writing a reply to the customer. A later lesson covers tools in detail.

In real APIs a tool definition also describes the inputs the tool takes, such as "order: a text value". The when line here is a simplified version of the description field.

6. Chat apps use the same API

A chat app is itself software built on top of a model. Behind its window, apps such as the ChatGPT app and the Claude app send requests to the provider's model in the same way your own code would.

flowchart TB
    A["A chat app<br/>comes with its own tools,<br/>such as reading your local files"] -- "behind the scenes" --> B["The same API<br/>the endpoint your own code would call"]
    B --> C["The model"]

The diagram shows the chat app at the top, the API in the middle, and the model at the bottom. The app adds its own layer above the API: the window, the hidden instructions, the conversation history, and a set of tools the app was shipped with. At the time of writing, a desktop chat app might come with a tool for reading files on your computer, along with whatever else that app includes.

What the chat window does not give you is a place to define an arbitrary tool of your own, the way the tools field in an API request does. At the time of writing, some chat apps let users add extra tools through connectors or plug-ins, but only in the forms the app supports. With the API, the list of tools is whatever your program sends.

This also explains a common surprise. The same model can behave a little differently in the chat app and through the API, because the app adds its own instructions and settings around your message, and a bare API request has only what your program put in it. In the same way, a basic API request does not remember earlier messages. A program that wants a conversation has to send the earlier turns again with each request, as the lesson on context described. At the time of writing, some providers offer optional features that keep the conversation on their side, but the model still receives the earlier turns with every request.

7. What the control is for

With the API, control means a list of fields that nobody else fills in for you.

The left column is every field on the request. The right column shows software that those fields make possible, and none of the three is a chat window. The support queue reads each new ticket and sends it to the right team. The contract reader checks documents against clauses that only your company uses. The digest runs on a schedule, summarises what arrived overnight, and delivers the result with no one typing a prompt.

The chat app is one program built on the API. A program you write is another, and it does what you built it to do.

The choice between the two follows from the task. A chat app fits a person doing varied, one-off tasks who wants to read the answers. The API fits a repeated task that should run without a person typing each request, a task that needs specific settings or tools, or a task whose output goes to another program.

Interview points

• An API is a documented interface through which one program requests something from another. For hosted models, it is the only path between a developer's software and the model on the provider's machines.
• An endpoint is the web address requests are sent to. A request is typically an HTTP POST with a JSON body, and the response is JSON containing the generated text.
• Requests are authenticated with an API key, and usage is billed per input and output token.
• Sampling and output settings are request parameters: temperature controls variation, a maximum output limit caps length and cost, stop sequences end generation at chosen text, and a format setting requests structured output.
• A request can include tool definitions. The model decides from the prompt and the descriptions which tool to request and with what arguments.
• The model does not execute tools. The calling program runs the tool and returns the result in a follow-up request.
• Chat apps are applications built on the same API. They add an interface, system instructions, conversation history, and a fixed set of tools.
• A basic API call is stateless: the program must resend the conversation history each time, unless it uses a provider feature that stores the history. Behaviour can differ from the chat app because the app's added instructions are absent.
• The API suits automated, repeated, or machine-consumed tasks. A chat app suits interactive, varied tasks where a person reads the result.

Summary

• The same models that run chat apps can be called from your own program.
• The model stays on the provider's machines, and the API is the way to reach it.
• A program posts a request to an endpoint and reads the response.
• Settings such as temperature, maximum output, stop sequences, and format are fields in the request.
• A request can list tools, and the model answers with the tool it wants to use.
• Chat apps are built on the same API, with their own interface, settings, and tools on top.
• Using the API gives control over the prompt, the settings, and the tools.