Computer Use

An agent can read a file, run a command, or search the web because someone built a tool for each of those jobs. Now suppose the job is to enter an invoice into an old accounting program that has only a window with fields and a Submit button. No tool exists for that program, and nobody is going to build one.

Computer use is the tool for this situation. It lets a model operate a computer the way a person does: it looks at a picture of the screen, then says where to move the mouse, where to click, and what to type.

Computer use is one tool among the others in an agent's list. It is the least precise of them and the slowest, and it is also the only one that works with any software that has a screen. After this lesson you can explain what is sent to the model and what comes back, why a small job takes many steps, and how to decide between computer use and a purpose-built tool.

1. Two kinds of software

Software falls into two groups, depending on who it was built for.

The first group has an API, an application programming interface. An API is a documented way for one program to ask another program to do something. A weather service is an example: a program sends the request "weather for Berlin" and receives the answer as text. To "call" software means to send it a request in this way. Ordinary agent tools are built on top of such entry points.

The second group has no such entry point. A great deal of software was only ever built to be looked at and operated by hand. A person sees the window, reads the labels, moves the mouse, and clicks. A program that wants to use this software has nothing to send a request to.

Computer use exists because of the second group. If the only way into a program is its screen, then the agent has to use the screen.

2. What is sent to the model and what comes back

Take a window on your screen with an invoice form:

Invoice number    A-4471
Supplier          Northwind
Amount            18.50
[ Submit ]

You see fields with labels and a button. The model does not receive fields, labels, or a button. It receives two things:

1. A picture of the window, as an image file (for example a PNG file, a common image format).
2. A question from the agent software, in effect "what should happen next".

Nothing more structured than that is sent. The model has to be able to read images, which is the ability described in the lesson on vision-language models. It looks at the picture, finds the Submit button by its appearance, and answers with two tool calls:

move_mouse  812, 466
click

The first line says where to put the mouse pointer. The second is an instruction to press the mouse button. The agent software then performs both actions on the real computer.

sequenceDiagram
    participant S as Your screen
    participant A as Agent software
    participant M as Model
    A->>S: Take a screenshot
    S-->>A: Picture of the window
    A->>M: Picture + what should happen next
    M-->>A: move_mouse 812, 466 then click
    A->>S: Moves the pointer and clicks

The diagram shows the direction of each piece. Pictures travel from the screen to the model. Mouse and keyboard instructions travel from the model back to the screen. As with every tool, the model only asks for the action. The agent software carries it out.

3. The model aims at a point on a picture

The reply move_mouse 812, 466 contains coordinates. Coordinates are two numbers that name one point on the screen. The first number, x, counts the distance from the left edge. The second, y, counts the distance from the top edge. The unit is the pixel, one of the small dots a screen image is made of. So "x 812, y 466" means 812 dots from the left and 466 dots down.

The model does not say "press the Submit button". It has no button name to use and no element it can address. All it has is a picture, so all it can give back is a point on that picture, followed by a click.

This has consequences:

• The model has to locate things by sight. If it misjudges the position by a small amount, the click lands next to the button and nothing happens, or a different control is pressed.
• The screen can change. If a window moves or a pop-up appears after the screenshot was taken, the coordinates point at the wrong thing.
• Nothing confirms the result. The click returns no answer such as "invoice saved". The only way to find out what happened is to take another screenshot and look.

A purpose-built tool has none of these problems, because it names what it wants directly and receives a direct answer.

4. Counting the steps

The cost of computer use is not in any single step. It is in how many steps a small job needs.

Here is the sequence for opening one file, starting from an empty screen:

1  screenshot        -> sent to the model
2  click(84, 120)       the Files icon
3  screenshot        -> sent to the model
4  click(310, 244)      the folder
5  screenshot        -> sent to the model
6  type("northwind")
7  screenshot        -> sent to the model
8  click(512, 380)      the file

Every odd-numbered line is a new picture of the whole screen. It has to be taken and sent before anything else can happen, because the model cannot know what the screen looks like after the last action. Every even-numbered line is the one action the model chose after looking.

So the pattern is: picture, look, move, click. Then picture, look, type, click. That is eight steps in total, and four of them send a full screenshot to the model. The file is only open at the end. It has not been read yet.

A round trip is one exchange with the model: something is sent to it and its reply comes back. In the simplest setup the agent software takes each screenshot by itself, so the model is asked four times. In many products the screenshot is an action too: the model has to ask for it, like any other tool call, and wait for the picture. Counted that way, each of the eight lines is its own round trip.

This is the agent loop with very small steps. Each pass through the loop achieves one click or one piece of typing.

Each screenshot also costs tokens. An image is turned into image tokens, and a full screen is a large image. Those tokens are paid for on every pass, and the earlier screenshots stay in the conversation history unless the agent software removes them.

5. Compared with a purpose-built tool

Reading a file is the clearest comparison, because a proper tool for it exists.

With the read-file tool, the model writes one line that names the file. The agent software runs ordinary program code that finds the file, opens it, and returns its contents as text. That is one request and one result.

The differences follow from this:

• Speed. One exchange finishes in a moment. A chain of exchanges with screenshots takes several times longer.
• Cost. One short tool call uses few tokens. Four full-screen images use many.
• Reliability. A file path is exact. A click on estimated coordinates can miss.

So the criterion is clear: when a purpose-built tool exists for a job, it is faster, cheaper, and more reliable than computer use for the same job.

6. The cases where computer use is the only option

The case for computer use is narrow, but it is real. It covers software that has no entry point for programs.

Typical examples are an old accounting program, a specialised program for a lab instrument, or a company's in-house booking system that was written years ago and only has a window. For these, the choice is not between computer use and a better tool. The choice is between computer use and doing the work by hand.

7. Where computer use sits in the tool list

Because the case is narrow but real, agent products that offer computer use include it next to the precise tools, not as a replacement for them. A tool list might look like this:

read_file       path
write_file      path, text
run_command     cmd
search_web      query

computer_use    screenshot
                move, click, type

The left column is the tool name. The right column is what the model has to supply or what the tool can do. The first four tools each name one job: read a file at a path, write text to a file, run a command, search the web for a query. The fifth is different. It does not name a job. It hands the model the screen and the means to operate it: take a screenshot, move the pointer, click, type.

The model chooses among these tools for each step. When a precise tool fits the step, that tool gets the job done in one call. Computer use is for the steps that no other tool can do.

8. The trade-off

Computer use is the slowest way for an agent to operate software, and it is the only way that works with everything.

Every other tool depends on the software having been built to accept requests from programs. Computer use depends only on the software having a screen, and nearly all software made for people has one.

At the time of writing, several AI providers offer a computer use tool, and its accuracy and speed are improving. The details differ by product, so check the current documentation before relying on it.

Computer use also raises a safety question. An agent that can click anywhere on a screen can do anything the logged-in person can do, including actions nobody intended. For this reason computer use is commonly run on a separate, restricted machine, and a person is asked to confirm actions that cannot be undone.

Interview points

• Computer use is a tool that lets a model operate a graphical interface. The model receives screenshots and returns mouse movements, clicks, and typed text.
• It exists because much software has no API. Such software offers only a window, menus, and buttons made for a person.
• The model is given a picture and a question about the next step. It does not receive the structure of the window, so it has no button names or elements to address.
• The model targets a point on the picture by x and y pixel coordinates and then asks for a click. The agent software performs the action.
• After every action a new screenshot is needed, because the model cannot know the state of the screen otherwise. A small job such as opening one file takes eight steps, four of which send a full screenshot to the model.
• A purpose-built tool such as read_file does the same job in one call, with no screenshots, so it is faster, uses fewer tokens, and is more reliable.
• Computer use fits desktop applications and internal tools that have no programmatic entry point and will not get one.
• Agents offer computer use next to precise tools. The precise tools are used when they fit, and computer use covers the remaining cases.
• The trade-off is generality against speed: computer use works with any software that has a screen and is the slowest option.

Summary

• Computer use lets a model work with software through its screen.
• A screenshot goes to the model, and a mouse or keyboard instruction comes back.
• The model names a point by coordinates. It does not name a button.
• Every action is followed by a new screenshot.
• Opening one file takes eight steps with computer use and one call with a read-file tool.
• Computer use is for software with no entry point for programs, such as desktop applications and internal tools.
• It is one tool in the list, next to tools for reading, writing, running commands, and searching.
• It is the slowest option and the one that works with the widest range of software.