Vision Language Models and Embedding Models

Earlier lessons treated a model as something that takes text in and gives text out. Many useful models take other things in or give other things out, such as images, audio, or lists of numbers. The modality of a model is the kind of input it accepts and the kind of output it creates. This lesson introduces the main modalities and then covers two that stay close to text models: vision language models, which read images and video, and embedding models, which turn content into vectors.

Afterwards you can explain what the extra parts of a vision language model do and why they make requests costly, count the tokens of an image or a video clip, choose how to deploy an embedding model for bulk jobs versus live search, and check an embedding model after quantization.

Example 1: the map of modalities

Generative AI offers many input and output combinations:

A vector is a list of numbers. A mask marks which pixels belong to an object. Despite the long table, there are only two broad archetypes of generative model (explained in How Image and Video Models Generate):

• Autoregressive token generation: start from a sequence of tokens and predict the most likely next token, repeatedly.
• Iterative denoising: start from random noise and refine it step by step toward the most likely output.

LLMs are the best-known autoregressive transformer models, but not the only ones. Vision language models, text and multimedia embedding models, automatic speech recognition (ASR) models, text-to-speech (TTS) models, and many others use similar architectures. This means many inference engines and techniques built for LLMs also apply to them.

Image and video generation models use iterative denoising. Increasingly, hybrid diffusion transformer models lead in quality. Some philosophies carry over, from kernel selection to parameter tuning, but the details differ a lot.

Each modality also changes what to measure. A single token of audio from a TTS model is not useful by itself, so instead of TTFT (time to first token) the measure becomes time to first word or time to first sentence. This chapter covers six common modalities beyond LLMs, with the special considerations of each.

Example 2: what a vision language model is made of

A vision language model (VLM) takes one or more images or videos plus a text prompt and generates a text response. It has two modules:

• LLM: a standard large language model.
• Vision encoder: a small model that takes raw images and videos and converts them into image tokens.

graph LR
    I["Image or video"] --> V["Vision encoder: converts pixels to image tokens"]
    T["Text prompt"] --> L["LLM"]
    V --> L
    L --> O["Text answer"]

The LLM is much larger than the encoder. In Mistral Large 3, the vision encoder has just two billion parameters, against 673 billion for the LLM, so the encoder is under 0.3 percent of the total (2 / 675 = 0.0030). Yet the encoder matters for inference. VLMs use many different encoder designs and implementations, so runtime support is more fragmented than for plain LLMs. That fragmentation makes vLLM and SGLang more important for serving VLMs (see Inference Engines).

As a rule of thumb, one high-resolution input image adds about a thousand visual tokens to the input. At a high level image tokens are like ordinary tokens, but they add up fast. The main challenge in VLM optimization is the longer input sequence and larger KV cache. It shows up in both phases:

• Prefill: images are cut into patches, embedded, tokenized, and fed into prefill as part of the input.
• Decode: same mechanics with a longer context, and some models use attention variants for the image tokens.

Worked cost of one image, using the illustrative cache size from KV Caching (327,680 bytes per token for a 70B-class shape at 16 bits):

1 image  = about 1,000 tokens
KV bytes = 1,000 x 327,680 = 327,680,000 bytes = about 0.33 GB
10 images in one conversation = about 3.3 GB of cache, before any text

Every technique from the techniques section helps with this load:

VLMs add a new quality-versus-speed choice: downsampling. An image can become visual tokens at different resolutions. A high-resolution version takes about four times more tokens than a low-resolution one but carries more detail. A single image rarely needs downsampling. Several images or video clips may.

high resolution: about 1,000 tokens per image
low resolution : about 1,000 / 4 = 250 tokens per image
8 images high  : 8,000 tokens      8 images low: 2,000 tokens

Example 3: video input

A video is more than a pile of frames. It may carry audio (many VLMs cannot process audio, so it must be transcribed separately and added to the prompt), and its frames show motion through space that single images lose. VLMs are trained on clips to learn that time dimension, and high-quality inference needs the whole clip in a single call.

The count: one second of cinematic video has 24 frames, and each frame is an image. If an HD image takes about 1,000 tokens, then:

tokens = frames per second x seconds x tokens per frame
       = 24 x 4 x 1,000
       = 96,000 tokens for a four-second clip   (nearly 100,000)

In practice videos do not reach this length, because downsampling is practically obligatory. Lowering the resolution and the frame rate allows a whole clip in one request, although video understanding models still accept only very short clips. Here is one downsampling choice:

reduce to 6 frames per second at 250 tokens per frame:
tokens = 6 x 4 x 250 = 6,000 tokens   (16 times fewer than 96,000)

After the video is tokenized and encoded, inference is like image inference with a much longer context. Prefix caching, KV cache offloading, and optimized attention are even more important for these inputs of tens of thousands of tokens (see Optimizing Attention).

Example 4: omni-modal models and pipelines

VLMs belong to a trend toward omni models that accept several input types and produce several output types. There are pros and cons. The mix of modalities offers unique abilities, but smaller specialized models are often faster and more accurate in their own domain. For example, many VLMs have text recognition trained into their image processing. It lags behind dedicated OCR (optical character recognition) models, which are generally a fraction of the size.

Production VLM inference therefore often coordinates a pipeline of several models and preprocessing steps: a preprocessor that extracts data from PDFs, OCR that reads text from images, or a transcriber for the audio of a video. Each part of the pipeline must be optimized for speed, and each should scale independently to avoid bottlenecks.

graph LR
    A["PDF or video"] --> P["Preprocessors: PDF extraction, OCR, audio transcription"]
    P --> V["VLM"]
    V --> R["Answer"]

Example 5: what an embedding model does

An embedding model converts a chunk of text of any length, or another input such as an image, into a fixed-length vector that captures its meaning. Content with similar meaning ends up with similar vectors. All content shares one semantic vector space, so the distance between two items can be compared with simple math. Embedding models, together with vector databases, power agent memory, RAG (retrieval-augmented generation: look up relevant documents and add them to the prompt), search, and recommendation systems.

The usual measure of closeness is cosine similarity: the dot product of two vectors divided by the product of their lengths. By hand with two small vectors, a = (1, 2, 2) and b = (2, 4, 4):

dot product = 1x2 + 2x4 + 2x4 = 18
length of a = sqrt(1 + 4 + 4) = 3
length of b = sqrt(4 + 16 + 16) = 6
cosine      = 18 / (3 x 6) = 1.0      (same direction: identical meaning)

c = (2, -1, 0):  dot with a = 2 - 2 + 0 = 0, so cosine = 0   (unrelated)

A value of 1 means the same direction, and 0 means unrelated. The same measure is used again to test a quantized embedding model below.

Embedding workloads have two traffic profiles:

1. High-throughput backfills: bulk jobs such as indexing millions of documents, updating product catalogs, or preparing data for LLM pre-training.
2. Low-latency lookups: single user-facing queries for search, retrieval, or recommendation, where each millisecond affects user experience.

The first step is to decide which profile to serve. If both are needed and the traffic justifies the cost, build a separate system for each.

Example 6: embedding model architecture and vector size

Tens of thousands of embedding models exist on Hugging Face, all built on one of two transformer architectures:

• BERT-style models: encoder-only networks, usually under 1B parameters, originally built to predict masked tokens.
• LLM-based models: modern language models, generally up to 8B parameters, reused to produce embeddings.

LLM-based models are substantially more capable now, but BERT-style models are still used for simple, latency-sensitive tasks such as classification.

Embedding models bring a speed-quality trade-off in embedding dimensionality, the length of the output vector. A vector has a few hundred to a few thousand values, and longer vectors hold more information. Modern models use Matryoshka representations, which are trained so that the first part of a vector is already a useful shorter vector. This allows a dynamic trade between dimensionality and quality while keeping more information in the shorter vectors.

Dimensionality does not materially change inference time. It changes storage, retrieval, and similarity computation time elsewhere in the system. A worked storage example with 4-byte numbers and 10 million documents:

1,024 values: 10,000,000 x 1,024 x 4 bytes = 40.96 GB
  256 values: 10,000,000 x   256 x 4 bytes = 10.24 GB   (4 times smaller)

Vectors from different embedding models usually cannot be compared, even with equal length, because each model encodes input into its own semantic space.

Example 7: serving embedding models

Embedding models with an LLM backbone, such as Qwen 3 Embed 8B, use the same tools and techniques as other high-volume, low-latency deployments of smaller LLMs. Several runtimes serve text embedding models: vLLM, SGLang, Infinity, and TEI (Text Embedding Inference, by Hugging Face). The best performance comes from adapting TensorRT-LLM to run them.

graph LR
    R["Requests"] --> T["Parallel tokenization"]
    T --> B["Batch management"]
    B --> E["Optimized inference engine"]
    E --> V["Vectors"]

TensorRT-LLM brings an optimized XQA kernel for fast attention and kernel fusion to reduce memory access (see CUDA, Kernels, and Kernel Fusion). For supported models it is the most performant engine for both latency and throughput.

Quantization adds more speed. Small models are more likely to lose quality, but FP8 for embedding model weights gives better performance with minimal quality loss. A direct check is to run the same inputs through the original and the quantized model and compute the cosine similarity of the output vectors. A similarity of 100 percent means identical vectors. At least 99 percent is the bar for confidence in the quantization.

original vector : (0.60, 0.80, 0.00)
quantized vector: (0.58, 0.81, 0.02)
dot = 0.348 + 0.648 + 0.000 = 0.996
lengths: 1.000 and sqrt(0.3364 + 0.6561 + 0.0004) = 0.9964
cosine = 0.996 / (1.000 x 0.9964) = 0.9996   -> 99.96 percent, passes the 99 percent bar

Embedding models process all tokens in parallel, so prefix caching and disaggregation do not apply. They are small, so spreading one model over several GPUs does not help. High-traffic deployments scale horizontally, with every GPU as its own replica.

Batching and queueing matter in busy deployments. Embedding models allow much higher batch sizes than other models: one request can carry dozens or hundreds of texts in a list, and many requests can share one GPU, since even the heaviest embedding models are small and fast. Traffic from a big backfill or a spike can still exceed those large batch sizes, so a well-built queueing system is required infrastructure (see Autoscaling, Batching, Cold Starts, and Routing).

Interview points

• Modality means input and output types. Almost all generative models are either autoregressive token generators or iterative denoisers, so LLM techniques transfer to VLM, embedding, ASR, and TTS models.
• A VLM is an LLM plus a small vision encoder (for example 2B against 673B parameters in Mistral Large 3). The encoder is small but matters because encoder designs vary, which is why vLLM and SGLang matter.
• One high-resolution image is about 1,000 tokens. Low resolution is about four times fewer. A four-second video at 24 frames per second with 1,000 tokens per frame would be about 96,000 tokens, so downsampling is practically required.
• Every technique of the previous section applies to VLMs: KV quantization, EAGLE speculation, prefix caching for images, TP, and disaggregated prefill.
• Dedicated OCR and other small models often beat omni models in their domain, so production VLM serving is often a pipeline, each stage optimized and scaled separately.
• Embedding models have two traffic profiles, backfill throughput and low-latency lookups. Serve them with separate systems if both are needed at scale.
• BERT-style (under 1B) and LLM-based (up to 8B) are the two architectures. Matryoshka vectors trade dimension for quality. Vectors from different models are not comparable.
• Best runtime for supported embedding models is TensorRT-LLM. Check FP8 quality with cosine similarity of at least 99 percent. Scale by replicas, with large batches and a queue.

Summary

• Modality is the input and output type of a model. Two archetypes cover most models: autoregressive generation and iterative denoising.
• A VLM combines an LLM with a small vision encoder that turns images and video into tokens, which makes inputs long and KV caches big.
• Downsampling trades quality for tokens. High resolution is about four times as many tokens as low.
• Omni models trade specialized accuracy for breadth, so VLM production serving often runs a multi-model pipeline.
• Embedding models map content to fixed-length vectors for search, RAG, memory, and recommendations, and serve backfill or lookup traffic.
• Embedding inference uses TensorRT-LLM or other runtimes, FP8 weights checked by cosine similarity, big batches, queues, and one replica per GPU.