Attention and Mixture of Experts

Two ideas explain most of what makes modern LLMs both smart and expensive to run. The first is attention, the step that lets each token look at the earlier tokens in the text. The second is Mixture of Experts (MoE), a design that lets a very large model use only a small part of itself for each token. Attention is the most expensive operation in inference, and MoE changes how a model's size relates to its cost, so both come back in later lessons on memory, GPUs, and parallelism.

Afterwards you can compute a small attention example by hand, explain what queries, keys, and values are, say why the KV cache turns a quadratic cost into a linear one, and calculate how many parameters and experts an MoE model uses per token.

The lesson builds on How an LLM Generates Text, which introduced tokens, prefill, decode, and transformer blocks.

Example 1: what attention does

Attention is the mechanism a transformer uses to relate one token to the other tokens of a sequence. People are good at seeing how words relate. Attention gives an LLM the same ability.

Take the sentence: "I decided to write a book because I thought it would be easy, but it was actually hard." To understand the word "it", a reader links it to "write a book". Attention is how the model learns that link: the representation of "it" is updated using the representations of the tokens that matter for it.

flowchart LR
    I["it"] -. "strong link" .-> W["write a book"]
    I -. "weak link" .-> D["decided"]
    I -. "weak link" .-> H["hard"]

The dotted arrows stand for weights that attention computes. The model gives high weights to relevant tokens and low weights to the rest. Example 2 shows how.

Example 2: queries, keys, values, and a worked calculation

The standard form is scaled dot-product attention:

Attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V

It has three inputs:

An analogy: a query is a question, a key is the label on each stored answer, and a value is the stored answer. The question is matched against all labels, and the answers are blended in proportion to how well their labels match. The analogy stops at one point: the match is not an exact lookup. Every stored answer contributes a little, and the blend weights come from softmax.

The formula in steps:

1. Q K^T: take the dot product (multiply matching numbers and add) of the query with every key. A higher number means a better match.
2. Divide by sqrt(d_k), the square root of the key dimension. This keeps the numbers from growing too large as the vectors get longer.
3. softmax: turn the scores into weights that are positive and add up to 1.
4. Multiply the weights by V and add: the output is a weighted blend of the values.

A worked case with vector size d = 2 and three earlier tokens:

query:  q  = [1, 0]
keys:   k1 = [1, 0]   k2 = [0, 1]   k3 = [1, 1]
values: v1 = [1, 0]   v2 = [0, 1]   v3 = [2, 2]

step 1, dot products:  q.k1 = 1   q.k2 = 0   q.k3 = 1
step 2, divide by sqrt(2) = 1.414:  0.707   0   0.707
step 3, softmax:  e^0.707 = 2.028   e^0 = 1   e^0.707 = 2.028   sum = 5.056
        weights:  2.028 / 5.056 = 0.401,  1 / 5.056 = 0.198,  0.401
step 4, blend:
        0.401 x [1, 0] + 0.198 x [0, 1] + 0.401 x [2, 2]
      = [0.401 + 0 + 0.802,  0 + 0.198 + 0.802]
      = [1.203, 1.000]

Tokens 1 and 3 match the query equally well, so they get the same weight, 0.401. Token 2 does not match, so its weight is smaller, 0.198. The output [1.203, 1.000] is the new representation for the query token.

Multi-head attention. Attention sublayers in a model are multi-head, and each head is one attention operation. In a diagram of the network, the heads sit in parallel on the same sublayer. Each head can specialize in a different kind of relationship between tokens, such as subject-verb agreement or co-reference resolution (finding which word "it" or "she" points to).

flowchart LR
    X["Hidden states"] --> H1["Head 1: e.g. subject-verb"]
    X --> H2["Head 2: e.g. co-reference"]
    X --> H3["Head 3 ... more heads"]
    H1 --> C["Combine heads"]
    H2 --> C
    H3 --> C

Two kinds of attention:

The causal mask matters during training and prefill. When a whole prompt is processed at once, the mask makes token 5 look only at tokens 1 to 5. Without it, a token could read the answer that comes after it.

Example 3: quadratic cost, and how the KV cache makes it linear

Attention compares the current token with every earlier token. So the work grows with the square of the sequence length: it is quadratic. Count the comparisons when a sequence of N tokens is processed from scratch: token 1 compares with 1 token, token 2 with 2, and so on.

comparisons = 1 + 2 + 3 + ... + N = N (N + 1) / 2

N = 1,000    ->        500,500
N = 10,000   ->     50,005,000
N = 100,000  ->  5,000,050,000

Making the sequence 10 times longer makes the work about 100 times larger. As the context grows, attention slows down.

In practice, decode attention is linear, not quadratic, thanks to the KV cache. The key and value vectors for earlier tokens never change once they are computed, so the cache stores a key-value pair for each previous token. At each decode step the model computes K and V only for the new token, appends them to the cache, and then compares the one new query with the cached keys. Counting per step for a sequence at length n:

without a cache, step n: recompute K and V for n tokens, then n comparisons
with a cache,    step n: compute K and V for 1 token,    then n comparisons (lookups)

generating 1,000 tokens, K/V computations:
  without a cache: 1 + 2 + ... + 1,000 = 500,500
  with a cache:    1,000

Each decode step then costs work proportional to n, the cache length, so the cost of one step is linear. The total over all steps is still the sum of those steps, but no step repeats an earlier step's work.

The KV cache is built during prefill, used and updated during decode, and lives in GPU memory by default. Because it grows with every token and every request, storing, reading, and reusing it is a major topic of inference engineering. The KV caching lesson covers it in detail.

flowchart LR
    P["Prefill: compute K and V for all prompt tokens"] --> C[("KV cache on GPU memory")]
    C --> D1["Decode step: new token's query reads the cache"]
    D1 -->|"append new K and V"| C

Example 4: Mixture of Experts

The density of a neural network is set by the number of connections between layers. Denser networks keep more information. Sparser networks need less compute and memory to run.

Mixture of Experts (MoE) is an architecture change that adds sparsity to the linear layers. A normal (dense) model has one giant matrix per linear layer. An MoE model has hundreds of smaller matrices, the experts, and sends each input to a small selection of them. Choosing experts for an input is called activating them.

Take the model Qwen3-235B-A22B. The name states its size: 235 billion total parameters, of which 22 billion are active per token (the "A22B"). In numbers:

active share of parameters = 22 B / 235 B = 0.094  (about 9.4 percent)

Routing is granular. One forward pass produces one token and goes through every layer. A router, a tiny model inside the LLM, picks which experts to activate at each layer. In the Qwen example there are 128 experts in each layer, and the router picks eight of them at each of the 94 layers for every generated token.

experts per layer used   = 8 of 128 = 6.25 percent
expert activations per token = 8 x 94 layers = 752

flowchart LR
    X["Token in this layer"] --> R["Router (tiny model)"]
    R --> E1["Expert 17"]
    R --> E2["Expert 52"]
    R --> E3["6 more experts of 128 total"]
    E1 --> M["Combine outputs"]
    E2 --> M
    E3 --> M

The diagram shows one layer. The expert numbers are examples, and the router chooses different experts for different tokens and different layers.

A catch in production. Because only a few parameters are active, MoE models are highly efficient for local inference with a single request. On a production server the picture changes. Different requests in a batch activate different experts, and almost all of the parameters are expected to be active at any given time. The exception is large-scale Expert Parallelism, which spreads experts across GPUs so that sparsity can be kept (the model parallelism lesson).

A simplified calculation shows why. Assume each request picks 8 of 128 experts at random and uniformly, which real routers do not do exactly, so this is only an illustration. The chance that a given expert is used by none of B requests in a layer is (1 - 8/128)^B = 0.9375^B.

B = 1   requests:  0.9375^1  = 0.938 unused   -> 6.3 percent of experts active
B = 8   requests:  0.9375^8  = 0.597 unused   -> 40.3 percent active
B = 16  requests:  0.9375^16 = 0.356 unused   -> 64.4 percent active
B = 32  requests:  0.9375^32 = 0.127 unused   -> 87.3 percent active
B = 64  requests:  0.9375^64 = 0.016 unused   -> 98.4 percent active

With 64 requests at once, nearly every expert is needed, so the server must hold and read nearly all 235 billion parameters even though each token uses 22 billion.

Where MoE is used. MoE architectures are especially popular for large models of 100B parameters and more, although some MoE models are as small as 20 to 30 billion parameters. MoE enables Expert Parallelism, which allows high-throughput inference for large models across multiple GPUs. A figure in the source shows an MoE architecture that combines sharding (splitting the experts across GPUs) and replicating (keeping copies of shared parts on each GPU) to use many GPUs well.

flowchart TB
    subgraph G1["GPU 1"]
        A1["Experts 1 to 64 (sharded)"]
        S1["Shared parts (replicated)"]
    end
    subgraph G2["GPU 2"]
        A2["Experts 65 to 128 (sharded)"]
        S2["Shared parts (replicated)"]
    end

The expert ranges are an example split.

Where MoE does not help. Models under 32B parameters, and especially under 8B, tend to use traditional dense architectures efficiently. Domain-specific models, for example for tab completion, also gain little from MoE, because the whole model is effectively one expert.

Interview points

• Attention relates a token to other tokens. Scaled dot-product attention is softmax(Q K^T / sqrt(d_k)) V.
• Q (queries) is the representation of the token being generated or updated, K (keys) are representations of prior tokens, and V (values) are the computed attention values for prior tokens.
• Multi-head attention runs several attention operations in parallel. Each head can handle a different relationship, such as subject-verb agreement or co-reference.
• Self-attention takes Q, K, and V from one sequence. LLMs use it with a causal mask. Cross-attention takes Q from another sequence, and image generation and multimodal models use it.
• Attention is quadratic in sequence length. The KV cache stores key-value pairs for previous tokens, so attention runs in linear time during decode. The cache is built during prefill and lives in GPU memory.
• MoE replaces one big matrix with many smaller experts and a router that activates a few per layer. Qwen3-235B-A22B has 22B active of 235B total, with 128 experts, eight picked per layer, over 94 layers.
• MoE is efficient for single-request local inference, but in batched production serving almost all parameters are active unless Expert Parallelism restores sparsity.
• Dense models are efficient under 32B parameters, especially under 8B, and for single-purpose models such as tab completion.

Summary

• Attention blends value vectors using weights from query-key matches, in many heads in parallel.
• Self-attention serves LLMs. Cross-attention conditions one sequence on another in image and multimodal models.
• Attention cost grows quadratically with sequence length, and the KV cache keeps each decode step linear.
• MoE keeps only a small part of the weights active for each token. Batching makes almost all experts active, so memory use stays at the full model size.
• Expert Parallelism is the technique that restores sparsity at scale.