Neural Networks for Inference Engineers

To make a model faster, an engineer must know what the model does with each request. Generative AI models look mysterious from outside, but they are built from a few simple parts repeated many times. This lesson explains those parts: the node, the layer, the hidden state, the encoder and decoder, the matrix multiplication, and the activation function. They are the base for every later lesson in the course, from attention to GPUs.

Afterwards you can describe a neural network in plain terms, compute a linear layer by hand, explain why a network with only linear layers collapses into one layer, and explain what an activation function such as ReLU fixes. The calculations are small enough to follow by hand.

No earlier lesson is required. The text defines every term when it first appears.

Example 1: a short history, from perceptrons to transformers

Generative AI models are compositions of big, complex neural networks. A neural network is a program made of many small calculations whose numbers (the weights) are learned from data instead of being written by hand. The history explains where the pieces come from.

A transformer is a neural network with an attention mechanism that can learn relationships between different parts of a sequence. The attention lesson explains the mechanism. Transformers are the foundation of generative AI. They are not only for LLMs. They power every modality of model, from embeddings to voice to image and video generation.

Across modalities there are two important styles of transformer-based models:

"Autoregressive" means that each new output depends on the outputs before it. Both styles are covered in this section of the course. The rest of this lesson prepares the general neural network vocabulary they share.

Example 2: nodes, layers, and hidden states

To be productive, an inference engineer needs an intuition for the basic ideas of neural networks. This lesson gives a high-level version, and the further reading lists deeper sources.

The basic unit of a neural network is a node, also called a neuron. A node is a short program: it takes an input, multiplies it by some weights, adds a bias, and returns the result. A bias is one more learned number that shifts the result. For example, a node with weight 3 and bias 1 turns the input 2 into 2 x 3 + 1 = 7.

A group of nodes forms a layer. Nodes within a layer are independent of each other, and each does its own calculation. The "network" in neural network refers to the connections between layers: the nodes in a layer receive the output of the previous layer.

The networks behind LLMs have dozens to hundreds of layers. There are three kinds:

flowchart LR
    I["Input layer"] --> H1["Hidden layer 1"] --> H2["Hidden layer 2"] --> H3["... many hidden layers ..."] --> O["Output layer"]

Each layer produces an output that the next layer reads as its input. For hidden layers, these outputs are called hidden states. A hidden state is one kind of internal representation, which is the form in which the network holds data inside. An important property of an internal representation is its dimensionality, the actual size of the vectors used. A vector is a list of numbers, and its dimensionality is its length. A vector [0.2, -1.5, 3.0] has dimensionality 3.

Internal representations change the dimensionality in different ways:

The worked calculation below follows a tiny network with three layers and shows the hidden state at each step, so the idea of "each layer's output is the next layer's input" is visible. The network has 3 inputs, a hidden layer with 4 nodes (using a ReLU, defined in Example 5), and an output layer with 2 nodes. The weights are fixed numbers chosen for illustration.

input x = [1, 2, -1]
layer 1 weights (3 inputs x 4 nodes), one column per node:
  node1: [0.5, 0.3, -0.2]  bias 0.1
  node2: [-1.0, 0.8, 0.1]  bias 0.0
  node3: [0.2, -0.5, 0.4]  bias -0.1
  node4: [0.0, 1.0, 0.6]   bias 0.2

node1 = 1x0.5 + 2x0.3 + (-1)x(-0.2) + 0.1 = 1.4
node2 = 1x(-1.0) + 2x0.8 + (-1)x0.1 + 0.0 = 0.5
node3 = 1x0.2 + 2x(-0.5) + (-1)x0.4 - 0.1 = -1.3  -> ReLU -> 0
node4 = 1x0.0 + 2x1.0 + (-1)x0.6 + 0.2    = 1.6
hidden state = [1.4, 0.5, 0, 1.6]

layer 2 weights (4 x 2): node A column [1.0, 0.5, -1.0, 0.2] bias 0.0
                         node B column [-1.0, 0.5, 0.0, 1.0] bias 0.5
A = 1.4x1.0 + 0.5x0.5 + 0 x(-1.0) + 1.6x0.2 + 0.0 = 1.97
B = 1.4x(-1.0) + 0.5x0.5 + 0 x 0.0 + 1.6x1.0 + 0.5 = 0.95
output = [1.97, 0.95]

parameters: layer 1 = 3x4 + 4 = 16, layer 2 = 4x2 + 2 = 10, total 26

The input vector has 3 numbers. The hidden state has 4: the layer widened the representation, like text embeddings do. The output has 2. The calculation also counts parameters: the first layer has 3 x 4 weights plus 4 biases, which is 16, and the second has 4 x 2 weights plus 2 biases, which is 10, for 26 in total. A real LLM does the same counting with billions of parameters, and that count is its size.

Example 3: encoders and decoders

Some networks create internal representations, and others use them:

• An encoder takes an input such as text or an image and builds an internal representation of it, with added information and semantic meaning.
• A decoder uses an internal representation to generate an output such as text or an image.

Neural networks are composable: several can be combined into one model, or used one after another to form a pipeline. The three arrangements in use are:

flowchart LR
    subgraph EO["Encoder-only (BERT-style embeddings)"]
        A1[Text] --> A2[Encoder] --> A3[Vector]
    end
    subgraph DO["Decoder-only (modern LLMs)"]
        B1[Tokens] --> B2[Decoder] --> B3[Next token]
    end
    subgraph ED["Encoder-decoder (Whisper)"]
        C1[Audio] --> C2[Encoder] --> C3[Decoder] --> C4[Text tokens]
    end

The speech lessons come back to Whisper, and the embedding lesson covers encoder-only models.

Example 4: matrix multiplication and the linear layer

The most important operation in a neural network is matrix multiplication, shortened to matmul. A matrix is a grid of numbers. A matmul takes an input vector and a matrix and multiplies the vector through the matrix to produce an output vector.

The simplest form is the linear layer. Given an input vector x, it multiplies by a weight matrix W and adds a bias vector b:

y = x W + b

Each output number is the sum of the input numbers multiplied by one column of W, plus one bias number. Here is a small case by hand. Let x = [1, 2], and let W have rows [3, 0, 1] and [1, 1, 2], so it maps 2 numbers to 3. Let b = [0, 1, -1].

y1 = 1*3 + 2*1 + 0  = 5
y2 = 1*0 + 2*1 + 1  = 3
y3 = 1*1 + 2*2 - 1  = 4
y = [5, 3, 4]

That is 2 x 3 = 6 multiplications and 6 additions in all, plus 3 biases. In words, every output node looks at every input number, which is why each linear layer has inputs x outputs weights. The weights of one linear layer are a small part of a generative model's total. Their values are set during training.

y = x W + b, with x = [1, 2], W rows [3, 0, 1] and [1, 1, 2], b = [0, 1, -1]
y = [5, 3, 4]    (6 multiplications, 6 additions, 3 biases)

Realistic size: one vector of 4,096 numbers through a 4,096 x 4,096 matrix
weights in the layer           = 4,096 x 4,096 = 16,777,216
multiplications for one vector = 16,777,216

The first block matches the hand calculation. The second shows why matmul dominates the work: a single 4096 x 4096 layer already holds about 16.8 million weights, and each vector pushed through it costs that many multiplications. A model with dozens of such layers per block and many blocks reaches billions of weights. This is the reason GPUs, which are very good at doing many multiplications in parallel, run these models.

Example 5: why activation functions are required

Matrix multiplication is composable: multiplying a vector by two matrices in turn gives the same result as multiplying it once by the product of those matrices. This creates a problem for deep networks. A stack of linear layers, each one a matmul, would collapse into a single layer whose matrix is all the matrices multiplied together. Ten layers would be worth no more than one.

Deep networks are valuable because more layers use more parameters effectively and encode more meaning in hidden states. To keep the layers from collapsing, a network separates them with an activation function, which breaks the linearity. Two properties matter:

• It is non-linear, so composing matmuls cannot collapse the layers.
• It is differentiable or mostly differentiable (it has a slope at almost every value), which supports back-propagation, the learning procedure that adjusts weights using those slopes.

The calculation below shows both facts with numbers. It multiplies a vector by two matrices in sequence, and then by the one combined matrix, and the results are identical. Then it puts a ReLU between the two matmuls, and the result no longer equals the combined matrix's result.

x = [1, -2]    W1 rows: [1, -1], [2, 1]    W2 rows: [1, 2], [-1, 1]

two linear layers:
  x W1 = [1x1 + (-2)x2, 1x(-1) + (-2)x1] = [-3, -3]
  [-3, -3] W2 = [-3x1 + (-3)x(-1), -3x2 + (-3)x1] = [0, -9]

one combined layer:
  W1 W2 = rows [2, 1] and [1, 5]
  x (W1 W2) = [1x2 + (-2)x1, 1x1 + (-2)x5] = [0, -9]      same result

two layers with a ReLU in between:
  ReLU([-3, -3]) = [0, 0]
  [0, 0] W2 = [0, 0]                                      different result

The first two results are equal: the single combined matrix gives the same [0, -9] as the two layers. The third result differs. The ReLU turned the middle vector [-3, -3] into [0, 0], so the final result is [0, 0]. The network is now doing something that no single matrix can imitate, and that is the property that lets deep networks learn complicated patterns.

The most basic activation function in inference is ReLU, the Rectified Linear Unit. Its rule is short: if x is greater than zero, return x, otherwise return zero. Dozens of activation functions exist, including one named Swish because its curve resembles the Nike logo. Most follow the same general pattern: negative values map to zero or close to zero, and positive values stay unchanged. Activation functions such as ReLU, SiLU, Swish, and SwiGLU are fast to run, easy to train because they have a gradient (a slope) at least for most values, and they break linearity, so multi-layer networks work.

SiLU(x) = x / (1 + e^(-x)). For x = 1: 1 / (1 + 0.368) = 0.731.

Both functions send large negative inputs to zero or near zero (for x = -4, ReLU gives 0 and SiLU gives about -0.072). Both keep large positive inputs almost unchanged (for x = 4, SiLU gives 3.928). The difference is the corner: ReLU has a sharp bend at 0, and SiLU curves smoothly through it. The pattern, negative values squashed and positive values kept, is the same, which is why this lesson treats them as one family. SiLU is the function that is also called Swish in the common form.

The next lesson uses these parts to describe a complete LLM: How an LLM Generates Text.

Interview points

• A neural network is made of nodes (multiply by weights, add a bias) grouped into layers. Nodes in a layer are independent, and the connections run between layers.
• LLM networks have dozens to hundreds of layers: an input layer, hidden layers, and an output layer. The outputs of hidden layers are hidden states.
• Dimensionality is the vector size. Text representations grow it to hundreds or thousands of numbers, and image representations shrink it from millions of pixels.
• An encoder builds an internal representation, and a decoder generates an output from one. Modern LLMs are decoder-only, BERT-style embedding models are encoder-only, and Whisper is encoder-decoder.
• A matmul multiplies an input vector through a matrix. A linear layer is y = xW + b. A linear layer with n inputs and m outputs has n x m weights plus m biases.
• Matmuls compose, so a stack of linear layers collapses into one layer. Activation functions are non-linear and mostly differentiable, which prevents the collapse and supports back-propagation.
• ReLU returns x for positive x and zero otherwise. SiLU, Swish, and SwiGLU are related functions that share the pattern of squashing negatives and keeping positives.
• The two styles of transformer-based generation are autoregressive token generation and iterative denoising.

Summary

• Neural networks grew from 1950s perceptrons through back-propagation, deep networks in the 2000s, AlexNet in 2012, and the transformer in 2017.
• Transformers power every modality, with two generation styles: autoregressive tokens and iterative denoising.
• Nodes form layers. Hidden layers pass hidden states, which are internal representations with a dimensionality.
• Encoders create representations, decoders use them. Networks are composable.
• Matmul and the linear layer are the core operation. Their weights are set in training.
• Activation functions such as ReLU break linearity, so deep stacks of layers do not collapse.