Multimodality

Most AI assistants in use today already handle more than text. You can upload a photo and ask what is in it, share a screenshot or a PDF and ask questions about it, and in some apps talk to the assistant with your voice.

The technical word for this is multimodal. The vision-language models from the previous lesson are one example of it.

This lesson explains what a modality is, which modalities go in and come out of an assistant, and why two assistants that look alike can have different abilities. It also shows that "one assistant" is often several models working together. After this lesson you can read a product's feature list and tell which abilities come from which kind of model.

1. What "multimodal" means

A modality is a type of input or output, such as text, images, audio, or video. A multimodal model can handle more than one type.

A multimodal model does more than accept each type separately. It understands several kinds of input, combines them, and reasons about them together. If you send a photo and a question, the model uses both at once to work out the answer.

A vision-language model takes images and text, so it is multimodal. It is one of the most common kinds of multimodal model.

2. Modalities going in and coming out

An assistant can be multimodal on the input side, on the output side, or on both.

flowchart LR
    subgraph IN["What you give it (it understands)"]
        I1["Text"]
        I2["Images"]
        I3["Video"]
    end
    A["One AI assistant<br/>the thing you talk to"]
    subgraph OUT["What it gives back (it generates)"]
        O1["Text"]
        O2["Images"]
        O3["Audio"]
        O4["Video"]
    end
    IN --> A --> OUT

• What you can give it: text, images, and video. Many assistants also accept audio, such as a spoken question.
• What it can give back: text, and in some assistants also images, audio (such as a spoken reply), and video.

From your side, it feels like talking to one assistant, even though it can read, see, speak, and create pictures. Which inputs and outputs are available depends on the app you use.

3. Abilities depend on the model behind the assistant

The assistant is the app you chat with. The work is done by the AI model behind it, and that model decides what the assistant can do.

• If the model handles only text, the assistant can only read and write text.
• If the model is a vision-language model, the assistant can also look at images you upload.
• Some models also handle audio or video.

So two assistants with similar chat screens can have very different abilities, depending on which model runs behind each one.

4. Reading images and creating images are different abilities

Understanding a picture and producing a picture are two separate abilities. An assistant can have the first without the second.

With a read-only assistant, you can upload a photo and it describes it or answers questions about it. It replies only in words. It cannot draw anything.

With an assistant that reads and creates, you can also ask for a new image, for example "Make this room look modern" or "Draw a logo for my bakery". The image comes back alongside the text reply.

Examples at the time of writing:

These abilities change often as companies update their products. Check what your app can do today.

5. One assistant, several models behind it

When an assistant can both understand and create images, it is often not one single model doing everything.

flowchart TB
    U["What you see<br/>one assistant that understands text and images<br/>and replies with text and images"]
    S["The assistant software<br/>sends each request to the right model<br/>and combines the results"]
    U --> S
    S --> M1["Language model<br/>reads text and images,<br/>writes the reply"]
    S --> M2["Image model<br/>creates new pictures"]
    S --> M3["Other models<br/>for example speech or video"]

• What you see: one assistant. It understands your text and pictures together and can reply with words or new images.
• What happens behind it: the app connects several different models, and each has its own job. One model reads and writes text. Another creates images. Others may handle speech or video.
• Why it feels like one thing: the software passes your request to the right model and brings the results back into one conversation. You never see the switch.

6. Multimodality is about the combination

The point to keep from this lesson: multimodality is not about one single model. It is about the combination.

Multimodality depends on the software as much as on any single model. Several models are combined inside one piece of AI software. As a result, the software understands more than text, and it returns more than text.

This connects to a point from the first lesson: a model is not a product. The product is the software around one or more models. A multimodal assistant is a clear case of that.

Interview points

• A modality is a type of data such as text, images, audio, or video. A multimodal model or assistant handles more than one.
• Multimodal understanding means the inputs are combined and reasoned about together, not handled in separate steps.
• A vision-language model is one kind of multimodal model: images and text in, text out.
• Input modalities and output modalities are separate. An assistant can read images without being able to create them.
• What an assistant can do depends on the model or models behind it, not on how the chat screen looks.
• An assistant that reads and creates several modalities usually combines several models, such as a language model and an image model, inside one application.
• The application routes each request to the right model and merges the results into one conversation.

Summary

• Multimodal means more than one type of data.
• Assistants can accept text, images, audio, and video, and can return text, images, audio, and video, depending on the product.
• Reading an image and creating an image are different abilities.
• The model behind an assistant sets its abilities.
• One assistant often uses several models behind the scenes.
• Multimodality comes from the combination of models and the software that connects them.