Vision-Language Models

The language models in the previous lesson can only read and write text. If you show one a photo, it has no way to take it in.

A vision-language model (VLM) removes that limit on the input side. It can look at images and video as well as read text. You can show it a photo, a screenshot, a chart, or a scanned document and ask about it in plain words.

This lesson explains what goes into a VLM and what comes out, what jobs it is used for, and why it replaced a whole generation of single-purpose vision models. After this lesson you can explain the difference between an LLM and a VLM, and between a VLM and an older vision model.

1. Images and words go in together

A VLM accepts several kinds of input in the same request:

• An image, such as shelf-3b.jpg, a photo of a shelf.
• Your words, such as the question "Which shelf is empty?"
• A video, which the model looks at as a series of still frames, one after another.

flowchart LR
    I["An image<br/>shelf-3b.jpg"] --> M["One model<br/>reads all inputs together"]
    W["Your words<br/>Which shelf is empty?"] --> M
    V["A video<br/>frame after frame"] --> M
    M --> O["Text out<br/>The second shelf<br/>from the top is empty."]

The important point is that the model reads these inputs jointly. It does not look at the picture first and then read the question as a separate step. It connects them, so it knows which part of the image the question is about.

The model then reasons about what it sees and writes its answer as text.

2. What changes compared with an LLM, and what stays the same

Compare the two kinds of model side by side.

So the input gets wider. A VLM accepts more kinds of things. The output stays the same. Both models answer in text.

3. A file becomes something you can question

Because the image and the question arrive together, an attached file turns into something you can have a conversation about. Suppose you attach shelf-3b.jpg and walkthrough.mp4 and ask two things:

You:    What's visible on this image? And what is this video about?

Model:  Three shelves. The middle one is empty. In the clip someone
        restocks it, then scans a barcode.

The model does not return a label such as "shelf" or a score such as "0.93". It returns a sentence written for the question you asked. A different question about the same files would get a different sentence.

4. Common jobs for a vision-language model

The task is not built into the model. It arrives with your question. This is why one VLM can be used for many jobs. Five common ones:

In every case the answer is normal written language. It is never a fixed label chosen from a list that someone prepared in advance.

Interface understanding matters for later lessons. An AI agent that operates a computer needs to look at a screenshot and work out what the buttons do. A VLM makes that possible.

5. What older vision models gave back

Before VLMs, most vision models were built for one single job. A face detector is a typical example. One image goes in. This is what comes out:

in     one image

out    face   x=412  y=196  w=88  h=104
       face   x=734  y=180  w=76  h= 96
       face   x=903  y=244  w=61  h= 78

The output is a list of three rectangles. Each line gives the position (x, y) and the size (w for width, h for height) of one face. It says nothing about who the people are, where they are standing, or what any of them is doing.

The list of questions you may ask such a model is also short:

1.  Is there a face, and where?

That is the complete list. A second question, such as "Is anyone wearing a helmet?", needs a second model. That second model has to be trained from the beginning on a second large set of images collected for that question.

6. One job, or any question

The difference between the two generations comes down to how many questions the model can answer.

flowchart LR
    subgraph OLD["Older vision model: one job"]
        P1["A photo"] --> F["Is there a face?"] --> Y["found: yes"]
    end
    subgraph VLM["Vision-language model: any question"]
        P2["An image"] --> Q1["What is going on here?"]
        P2 --> Q2["What does the sign say?"]
        P2 --> Q3["Describe it for me."]
    end

What changed is where the question lives. In an older vision model, the question is built into the model during training. With a VLM, you bring the question. The same image can answer whatever you ask of it.

Interview points

• A vision-language model takes images, video, and text as input and produces text as output.
• The inputs are read together, so the model can relate the words of a question to the right part of an image.
• A video is processed as a sequence of frames.
• Compared with an LLM, a VLM widens the input. The output is still text.
• The task is not built into a VLM. It comes with the question, so one model covers visual question answering, document understanding, chart interpretation, interface understanding, and image description.
• An older single-purpose vision model, such as a face detector, returns fixed structured output such as bounding boxes and can answer only the one question it was trained for.
• With older models, each new question required training a new model on a new image set. With a VLM, a new question is a new prompt.

Summary

• A VLM is a language model that can also see.
• Images, video frames, and text go in together. Text comes out.
• The answer is a sentence written for the question, not a label or a score.
• Typical uses are answering questions about photos, reading documents, reading charts, understanding screens, and describing images.
• Older vision models did one job each and returned boxes or labels.
• A VLM answers any question about an image because the question is supplied by the user, not fixed in training.