We know that backpropagation is the key to deep neural networks. What enables this key to unlock the door to deep learning is the chain rule.
Mathematically, the chain rule for gradients guarantees that the direction of each local adjustment forms part of the overall direction of error reduction. Through chain rule differentiation, the algorithm allows each connection to compute only its own small portion of "responsibility." These local derivatives, when mathematically combined, precisely constitute the direction for minimizing the total error.
To truly understand backpropagation, one can view a neural network as a series of "relay functions." One layer transforms the input into an intermediate representation, the next layer then processes that into another representation, and so on, until the output. Mathematically, this is a composition of functions: the output is a function of the previous layer's output, which in turn is a function of the layer before it, link by link.
The chain rule is actually quite simple: when you observe the total error at the very end, if you slightly tweak a small weight somewhere upstream, that tiny change propagates through all the relevant downstream links. Each link amplifies or diminishes the change in its own way. By the time this change finally reaches the output, all these local effects are multiplied together, determining whether the weight's change improves or worsens the final result, and by how much. This is the so-called gradient: it quantitatively describes the degree of responsibility each parameter has for the final error.
What the backpropagation algorithm does is essentially reverse this reasoning process. Starting from the output error, it passes the "sensitivity to the result" backward through the network layer by layer, allowing each layer to compute its small share of "responsibility" based on its own local derivative.
This might still sound abstract. Let's paint a picture: imagine a river with many tributaries. The most downstream point is the "loss" deviation, and upstream are the weights. If you add a truckload of water to a small tributary, whether and how much the water level rises downstream depends on the width and slope of each river section along the way. Wide sections amplify less, narrow sections amplify more. The chain rule multiplies the "amplification factor" of each segment along the path and sums over all possible paths leading downstream to get the "final impact of this truckload of water on the total water level." In the network, the "amplification factors" are the local derivatives of each layer; "summing over all paths" corresponds to the weighted sum of connections through successive layers.
Applying this intuition to the simplest two-layer network makes it clearer. Let the output be y, the final loss be L(y), the intermediate hidden layer variable be h, and a certain weight be w. L is first affected by y, y is affected by h, and h is in turn affected by w. The chain rule breaks down "how much does changing w a little affect L?" into three parts multiplied together: first ask "how much does L change if y changes a little?" (∂L/∂y), then "how much does y change if h changes a little?" (∂y/∂h), and finally "how much does h change if w changes a little?" (∂h/∂w). Multiply these three answers, and you get the direction and magnitude for w's adjustment. That is:
∂L/∂w = (∂L/∂y) · (∂y/∂h) · (∂h/∂w)
If there are multiple pathways through which w's influence can reach y, the formula above sums the products from each pathway. You don't need to see the entire network's details. You just measure the "slope" at each small segment you pass through, finally multiply the slopes along this path, sum over all possible paths, and you get the true impact on the whole.
This is precisely the secret to backpropagation's efficiency. With hundreds of millions of weights w in the network, by remembering and reusing the local slopes during a single backward pass from "downstream to upstream," we can compute the respective "direction sense" for all weights. In engineering, this is called reverse-mode automatic differentiation: first, values flow forward (the forward pass), then "sensitivities" flow backward (the backward pass). The forward pass is like solving the problem; the backward pass is like grading it. Each node simply does two small things: it stores its local slope from the forward pass, and during the backward pass, it takes the sensitivity received from downstream, multiplies it by this stored slope, and distributes the result upstream along its connections. Thus, these local calculations aggregate into the correct global adjustment.
The question of "will local adjustments cancel each other out?" finds its answer here. The slopes everywhere are not guessed arbitrarily; they are genuine measurements of the gradient along the same "loss river." Each parameter takes a small step along its own "true downhill" direction. Mathematically, all these small steps collectively move toward descending the same error valley—although it doesn't guarantee finding the global minimum in one step, it does guarantee that this step is moving downhill. This is also why during training it's necessary to take small steps, many times; to use more robust descent strategies (like adaptive learning rate methods, e.g., Adam); and to use nonlinear activation functions less prone to causing extremely small or large slopes (e.g., ReLU instead of sigmoid). When the slopes in certain segments are close to zero, successive multiplications can "dampen" the sensitivity to nothing—this is the vanishing gradient problem. When slopes are consistently greater than one, multiplications can "explode" the sensitivity—that's the exploding gradient problem. These phenomena further confirm the real physical intuition of "chain multiplication and path summation": if the riverbed is too flat, the water flow dissipates; if the riverbed is too steep, the water flow amplifies. Generations of engineering improvements essentially involve "modifying the riverbed" so that sensitivity is neither excessively dissipated nor amplified, making the "downhill path" more navigable.
The success of backpropagation laid the algorithmic cornerstone for deep learning. It propelled connectionism from the ivory tower into practical application, providing the theoretical and technical prerequisites for the subsequent explosion of "deep learning" years later.
