Backpropagation: How Neural Networks Actually Learn

Backpropagation gets described as the thing that makes deep learning possible, which is true, and as something mysterious, which is not. Strip away the branding and it is the chain rule from first-year calculus, applied mechanically and in a specific order to a graph of arithmetic operations. A neural network is a big composite function. Training it means nudging millions of internal numbers so the function’s output gets closer to what you wanted. To nudge a number sensibly you need to know which direction moves the error down and by how much — the derivative of the loss with respect to that number. Backpropagation is the bookkeeping trick that computes every one of those millions of derivatives in a single sweep, at roughly the same cost as evaluating the network once.

The reason it matters is efficiency, not novelty. The math was known for decades before anyone cared. What backpropagation buys you is the ability to compute the gradient of one scalar loss with respect to every parameter in the model in time proportional to the network itself, rather than re-running the network once per parameter. For a model with a billion parameters, the difference is between “trains overnight” and “would not finish before the heat death of the universe.” This post walks through what the algorithm is actually doing, in code, and where it breaks.

The problem: credit assignment in a pile of numbers

A feedforward network is a chain of layers. Each layer takes a vector, multiplies it by a weight matrix, adds a bias, and passes the result through a nonlinear activation function. Stack a few of these and you get a function with a lot of tunable knobs — the weights and biases, collectively the parameters. Training data gives you input-output pairs. You feed an input through, compare the network’s output to the desired output using a loss function, and get a single number that says how wrong you were.

The learning question is: which knobs do I turn, and in which direction, to make that number smaller? This is the credit-assignment problem. A wrong answer at the output is the accumulated fault of every weight in every layer, but not equally. Some weights barely influenced the result; some dominated it. You need a per-parameter answer, and you need it to be quantitative, because you are going to take a small step proportional to each parameter’s influence.

The quantity you want is the partial derivative of the loss L with respect to each parameter w: written dL/dw. It tells you the slope — if you increase w a tiny bit, does L go up or down, and how steeply. Collect all those partials into a vector and you have the gradient. Gradient descent then says: step every parameter a little bit in the direction opposite its gradient, because the negative gradient points downhill on the loss surface. Backpropagation is how you get the gradient.

The chain rule is the whole trick

Here is the entire mathematical content. If L depends on y, and y depends on w, then:

dL/dw = (dL/dy) * (dy/dw)

That is the chain rule. For a deep network the composition is longer — the loss depends on the last layer, which depends on the layer before, all the way back to the weights you care about — so the chain has more links:

dL/dw1 = (dL/dz3) * (dz3/dz2) * (dz2/dz1) * (dz1/dw1)

Every term on the right is a local derivative: the derivative of one operation with respect to its immediate input. Backpropagation’s insight is that you compute these local derivatives once each, then multiply them together in the right order to get the derivative for any parameter. Crucially, the leftmost factors are shared across many parameters. The dL/dz3 term shows up in the gradient for every weight that feeds into z3. If you compute it once and reuse it, you avoid an enormous amount of duplicated work. That reuse is the difference between a tractable algorithm and an intractable one.

The whole thing was independently discovered several times before it caught on. Seppo Linnainmaa described the reverse mode of automatic differentiation in 1970. Paul Werbos applied it to neural networks in his 1974 thesis. Earlier still, Henry Kelley (1960) and Arthur Bryson (1961) had the gradient computation in the language of optimal control. What made it famous was the 1986 Nature paper by David Rumelhart, Geoffrey Hinton, and Ronald Williams, “Learning representations by back-propagating errors,” which showed the algorithm learning useful internal representations and put it at the center of the field. The technique itself is old; the appreciation is what was new.

The two passes

Backpropagation runs in two phases over a computational graph — a directed graph where nodes are operations and edges carry values.

   FORWARD PASS  (compute values, left to right)
   ------------------------------------------------>
        x ──▶ [ * W1 ] ──▶ z1 ──▶ [ ReLU ] ──▶ a1 ──▶ [ * W2 ] ──▶ z2 ──▶ [ loss ] ──▶ L
                 │                    │                   │                    │
              cache x              cache z1            cache a1            cache z2
                 │                    │                   │                    │
   <------------------------------------------------
   BACKWARD PASS  (compute gradients, right to left)
        dL/dW1 ◀── dL/dz1 ◀── dL/da1 ◀── dL/dz2 ◀── dL/dL = 1

In the forward pass you push the input through the network and compute the loss, exactly as you would at inference time — but you also cache the intermediate values at each node, because you will need them to compute local derivatives on the way back.

In the backward pass you start at the output with the trivial fact that dL/dL = 1, then walk the graph in reverse. At each node you already know the gradient of the loss with respect to that node’s output (it was handed to you by the node downstream). You multiply it by the node’s local derivative to get the gradient with respect to the node’s input, and you pass that upstream. Each node also computes the gradient with respect to any parameters it owns and stashes it. By the time you reach the input, every parameter has its dL/dw.

The reason you cache during the forward pass is memory-for-time: the local derivative of z1 = W1 * x with respect to W1 is x, so you need the value of x that flowed through. This is why training a network uses far more memory than running it. Activations from the forward pass have to be held in memory until the backward pass consumes them, which is the single biggest reason that training batch sizes are limited by GPU memory. Techniques like activation checkpointing trade compute for memory by recomputing some forward values during the backward pass instead of storing them.

A worked example in NumPy

Nothing clarifies backprop like implementing it with no framework. Here is a two-layer network doing regression, with the forward and backward passes written out by hand.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

import numpy as np

np.random.seed(0)

# One hidden layer: x -> (W1) -> ReLU -> (W2) -> y_hat
# Shapes: x is (batch, 3), hidden is 4 units, output is 1
W1 = np.random.randn(3, 4) * 0.1
b1 = np.zeros((1, 4))
W2 = np.random.randn(4, 1) * 0.1
b2 = np.zeros((1, 1))

x = np.random.randn(16, 3)          # 16 samples, 3 features
y = np.random.randn(16, 1)          # regression targets
lr = 0.05

for step in range(2000):
    # ---- forward pass (cache everything we reuse) ----
    z1 = x @ W1 + b1                 # (16, 4)
    a1 = np.maximum(0, z1)           # ReLU
    z2 = a1 @ W2 + b2                # (16, 1)  == y_hat
    loss = np.mean((z2 - y) ** 2)    # mean squared error

    # ---- backward pass (chain rule, right to left) ----
    n = x.shape[0]
    dz2 = (2.0 / n) * (z2 - y)       # dL/dz2  (MSE derivative)
    dW2 = a1.T @ dz2                 # dL/dW2  = a1^T . dz2
    db2 = dz2.sum(axis=0, keepdims=True)

    da1 = dz2 @ W2.T                 # dL/da1  push gradient upstream
    dz1 = da1 * (z1 > 0)             # ReLU derivative: 1 where z1>0, else 0
    dW1 = x.T @ dz1                  # dL/dW1
    db1 = dz1.sum(axis=0, keepdims=True)

    # ---- gradient descent step ----
    W2 -= lr * dW2; b2 -= lr * db2
    W1 -= lr * dW1; b1 -= lr * db1

    if step % 500 == 0:
        print(f"step {step:4d}  loss {loss:.4f}")

Read the backward block carefully, because every line is one application of the chain rule. dz2 is the derivative of the loss with respect to the output — for mean squared error (z2 - y)^2, that is 2*(z2 - y)/n. To get dW2 we multiply the incoming gradient by the local derivative of z2 = a1 @ W2 with respect to W2, which is a1 — hence a1.T @ dz2. To move the gradient past the ReLU, we multiply by the ReLU’s derivative, which is 1 wherever the input was positive and 0 elsewhere: the (z1 > 0) mask. There is nothing else going on. The backward pass is a mechanical mirror of the forward pass, one local derivative at a time.

Reverse mode: why backward and not forward

There are two ways to walk a computational graph and accumulate derivatives, and the choice is the reason backprop is efficient. Automatic differentiation has a forward mode and a reverse mode, and they differ in which direction they propagate.

A neural network has an enormous number of inputs — every parameter is effectively an input to the loss function — and exactly one output, the scalar loss. Forward mode would require one pass per parameter to get the full gradient, which for a billion-parameter model means a billion forward passes. Reverse mode gets the derivative of that single output with respect to all inputs in one backward pass. That asymmetry is the entire reason deep learning is computationally feasible. The price you pay for reverse mode is memory: you must retain the forward-pass activations to compute local derivatives on the way back, whereas forward mode carries its derivatives alongside the values and needs no cache.

Frameworks do this for you

Nobody writes backward passes by hand in production. Modern frameworks build the computational graph as the forward pass executes and then differentiate it automatically. In PyTorch the system is called autograd, and it records every operation on a tensor that has requires_grad=True, assembling a graph of the operations. Calling .backward() on the loss walks that graph in reverse and populates the .grad attribute of every parameter.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

import torch

torch.manual_seed(0)
model = torch.nn.Sequential(
    torch.nn.Linear(3, 4),
    torch.nn.ReLU(),
    torch.nn.Linear(4, 1),
)
opt = torch.optim.Adam(model.parameters(), lr=0.05)
loss_fn = torch.nn.MSELoss()

x = torch.randn(16, 3)
y = torch.randn(16, 1)

for step in range(2000):
    opt.zero_grad()             # clear gradients from the previous step
    y_hat = model(x)            # forward pass builds the graph
    loss = loss_fn(y_hat, y)
    loss.backward()             # reverse pass fills every .grad
    opt.step()                  # optimizer applies the update

Three lines carry the whole algorithm. loss.backward() is the backward pass from the NumPy example, done automatically for an arbitrary graph. opt.step() is the gradient-descent update. And opt.zero_grad() matters more than it looks: PyTorch accumulates gradients into .grad by adding, so if you forget to zero them, each step’s gradient is contaminated by the previous step’s — a classic silent bug that manifests as a model that mysteriously refuses to converge. The framework handles the calculus; the failure modes it leaves entirely to you.

Where the gradients vanish and explode

The chain rule multiplies a long sequence of local derivatives, and long products of numbers are numerically treacherous. If each factor is a little less than 1, the product shrinks toward zero exponentially in the number of layers. If each factor is a little more than 1, the product blows up. This is the vanishing and exploding gradient problem, and it is the reason deep networks were hard to train for so long.

Sepp Hochreiter identified it formally in his 1991 diploma thesis, showing that back-propagated error signals in deep networks either shrink rapidly or grow out of bounds. The classic culprit is the sigmoid activation. Its derivative peaks at 0.25 and is much smaller across most of its range, so every layer you pass through multiplies the gradient by at most a quarter. Ten sigmoid layers and your gradient at the early layers is on the order of 0.25^10, effectively zero — the early layers stop learning because no signal reaches them. The tanh function is a little better but has the same disease.

The practical fix that unlocked deep networks was the Rectified Linear Unit. ReLU outputs zero for negative input and passes positive input through unchanged, so its derivative is exactly 1 for any positive value. It does not shrink the gradient at all on the active path. That single property let people train networks far deeper than sigmoid or tanh ever allowed. The trade-off is the “dying ReLU” problem — a unit stuck in the negative region has gradient zero and never recovers — which spawned variants like leaky ReLU and GELU.

Vanishing gradients are not the only defense that got built. Exploding gradients, common in recurrent networks, are usually handled with gradient clipping — capping the gradient norm before the update. Architectural tricks matter more than activation choice: residual connections (the skip connections in ResNets and transformers) give the gradient a direct path backward that bypasses the multiplicative chain, and normalization layers keep the scale of activations under control so the local derivatives stay near 1. Much of modern architecture design is, at bottom, gradient-flow engineering. The same numerical care shows up elsewhere in the stack; the reasons 0.1 + 0.2 is not 0.3 are the same reasons deep gradient products misbehave, a topic covered in the post on floating point.

What the optimizer does with the gradient

Backpropagation produces the gradient. It does not update anything. The update is the optimizer’s job, and the simplest optimizer is plain gradient descent: w = w - lr * dL/dw, where lr is the learning rate, a small positive number. Step every parameter a little bit downhill and repeat. In practice you compute the gradient on a small random batch of data rather than the whole dataset — stochastic gradient descent, or SGD — which is noisier but vastly cheaper and, helpfully, the noise helps escape shallow bad minima.

Vanilla SGD is rarely used alone anymore. Momentum accumulates a running average of past gradients, so the update has inertia — it powers through flat regions and dampens oscillation across narrow valleys. The dominant optimizer today is Adam, introduced by Diederik Kingma and Jimmy Ba at ICLR 2014 (arXiv 1412.6980). Adam — short for Adaptive Moment Estimation — keeps exponential moving averages of both the first moment (the mean of recent gradients, like momentum) and the second moment (the uncentered variance, a measure of how bumpy the gradient has been), applies a bias correction for the cold-start of those averages, and uses them to give each parameter its own effective learning rate. Parameters with consistently large, noisy gradients get smaller steps; parameters with small, steady gradients get larger ones.

None of this changes what backprop computes. The gradient is the gradient; the optimizer only decides how to spend it. That separation is clean and worth internalizing: backpropagation answers “which way is downhill and how steep,” and the optimizer answers “how big a step to take, given the recent history.” Choosing the learning rate remains the single most consequential knob in training. Too large and the loss diverges as steps overshoot; too small and training crawls or stalls in a bad region. Adam softened this sensitivity but did not remove it, which is why learning-rate schedules and warmup are still standard practice.

Honest trade-offs

Backpropagation is the correct algorithm for the job, but it is not free of costs and it is not the only conceivable approach.

• Memory is the real tax. Storing forward activations for the backward pass is why training needs multiples of the memory that inference does. On large models this dominates hardware planning, and it is the reason for techniques like activation checkpointing and the memory-conscious approaches in GPU infrastructure for ML.
• It needs differentiable operations. Backprop requires local derivatives at every node, so anything non-differentiable — a hard threshold, a discrete sampling step, an argmax — breaks the chain and needs a workaround (straight-through estimators, reparameterization, or reinforcement-learning-style estimators).
• Gradient pathologies are inherent, not incidental. Vanishing and exploding gradients fall out of the multiplicative structure. You can mitigate them with architecture and normalization, but they never fully go away, and debugging a model that will not learn often comes down to inspecting gradient magnitudes layer by layer.
• It is not how brains work. The backward pass requires symmetric weights and a separate error-propagation phase that has no clean biological analog. This is a live research complaint, not just philosophy — it motivates work on alternatives like feedback alignment and forward-forward learning, none of which has matched backprop’s efficiency at scale.
• The gradient is local, the loss surface is not. Backprop tells you the slope where you stand, nothing about the shape of the landscape. It can walk you into a poor local region, and it gives no global guarantee. That it works as well as it does on huge non-convex problems is an empirical fact, not a theorem.

The quantization and precision choices that make large-model training affordable interact directly with gradient magnitudes, which is why low-precision training is delicate; the tradeoffs there are laid out in the quantization deep dive and are a recurring concern when you fine-tune a model on your own hardware.

Verdict

Backpropagation is the chain rule, run in reverse over a computational graph, with intermediate results cached and reused so the whole gradient falls out in one backward sweep at the cost of one forward pass. That is the complete idea. The math predates deep learning by decades and was reinvented several times before the 1986 Nature paper made it central. Its power is not conceptual depth but computational leverage: it computes millions of derivatives for the price of one, which is the only reason training billion-parameter models is possible at all.

Understanding it changes how you debug. A model that will not learn is almost always a gradient-flow problem — vanishing gradients from a bad activation choice, exploding gradients from a missing clip, dead ReLUs, or the humble forgotten zero_grad(). Once you see training as “compute the gradient, then spend it,” those failures stop being mysterious and become the first things you check. The frameworks hide the calculus, but they hand you every one of its failure modes. Learn the algorithm and you own the failures too.

Sources

• Rumelhart, Hinton & Williams (1986), “Learning representations by back-propagating errors,” Nature
• Who Invented Backpropagation? — Jürgen Schmidhuber
• Baydin et al., “Automatic Differentiation in Machine Learning: a Survey,” arXiv:1502.05767
• Vanishing gradient problem — Wikipedia
• Kingma & Ba (2014), “Adam: A Method for Stochastic Optimization,” arXiv:1412.6980
• PyTorch autograd documentation
• CS231n: Backpropagation, Intuitions (Stanford)