By introducing "hidden layers" that perform nonlinear transformations, a network can map linearly inseparable low-dimensional problems (like the XOR gate) into higher-dimensional, separable spaces. From this point on, neural networks gained the ability to represent complex patterns for approximiating universal funtions. For connectionism, achieving a true renaissance became merely a question of algorithm, time, and computing power.
On the algorithmic front, one core problem long perplexed neural network researchers: how can we effectively train a complex multi-layer network? If the network makes a mistake, how do we determine which connection weight in which layer is at fault, and how should we correct it? This differs from symbolic logic, where rules can be directly encoded into transparent, debuggable computer programs—because the code itself is an interpretable symbolic language.
In 1986, in a landmark paper, Geoffrey Hinton, David Rumelhart, and Ronald Williams systematically elucidated the core method for cracking deep neural networks: the Backpropagation algorithm. It enabled neural networks to truly perform "deep learning" for the first time.
After the model produces an output, it first calculates the overall error—the difference between the prediction and the correct answer. The essence of backpropagation is that the model's error can propagate backward, layer by layer, like ripples. The connection weights in each layer can be adjusted to correct for their portion of the "error." In other words, each layer contemplates: "If I adjust myself slightly this way, I need to ensure it makes the final result a little better." This is how backpropagation teaches machines to learn from outcomes—through layer-by-layer adjustments, the entire network's output progressively converges toward the correct answer.
This algorithm solved the problem of training multi-layer networks.
You might ask: In a neural network with hundreds of millions of parameters, how is it possible to know which direction to adjust each one?
Indeed, there are countless "knobs" (parameters, weights) in a neural network, collectively determining the output. Improving the model requires coordinating all these knobs simultaneously amidst so many moving parts—it sounds like finding a needle in a haystack, blindfolded. But here's the elegance of mathematics: the "sense of direction" for each knob can be computed locally.
The key idea of backpropagation is this: although the network is vast, each connection is part of a clear chain of causality. If we can calculate "how much the final output error would change if this specific connection weight were tweaked slightly," then we know which direction to nudge it. This is precisely what the gradient represents. It's not based on vague intuition but is precisely computed using calculus—a derivative (slope). In a multi-dimensional space, the gradient points in the direction of the steepest ascent of the error; thus, moving against it decreases the error fastest.
In other words, even with hundreds of millions of parameters, each parameter can independently compute a tiny arrow—based on its "propagated relationship" with the output error—indicating which way to move to reduce the error slightly. These minuscule "better directions" aggregate into a coordinated adjustment across the entire network. Thus, without needing a global view, the machine finds direction locally. In each training cycle, the model "descends the hill" this way: from its current position, it follows the error-reducing gradient downward a bit, then recalculates the new gradient. Repeating this thousands or millions of times, the error shrinks, and the model grows smarter.
Perhaps you'd further ask: In a network with so many parameters, won't these local adjustments cancel each other out? Common sense suggests that in complex systems, local optimizations might throw the whole system off balance. However, the marvel of backpropagation lies in this: each local adjustment is not blind. They all share a common objective—to minimize the overall error (loss). The gradient, computed via calculus, indicates the steepest downhill direction in the vast "error landscape." The gradient's direction tells each connection which way to adjust, and its magnitude suggests how much. When all connections compute their own "little sense of direction" and update accordingly, the entire network moves toward minimizing error. Thus, even faced with an immensely complex system of billions of parameters, the model achieves local judgment leading to global improvement—this is the secret sauce enabling deep learning.
Think of it like a massive orchestra. Each musician reads only their own sheet music (local information), often paying little attention to the other parts. Yet, they all follow the same conductor (the loss function)—who, through gestures (gradient signals), instructs each player to be louder, softer, faster, or slower. Thus, although no single musician "sees the big picture," the whole orchestra plays in harmony.
To truly understand backpropagation, view the neural network as a chain of "relay functions." One layer transforms the input into an intermediate representation, the next layer compresses that into another, and so on, until the output. Mathematically, this is a composite function: the output is a function of the previous layer's output, which in turn is a function of the layer before it, and so forth, link by link.
With millions of weights w in the network, the trick is to compute and reuse local slopes during a single backward pass, moving from "downstream to upstream." This efficiently provides each weight with its required "direction sense." In engineering, this is called reverse-mode automatic differentiation: first, values flow forward (the forward pass), then "sensitivities" flow backward (the backward pass). The forward pass is like solving the problem; the backward pass is like grading it. Each node simply does two small tasks: it stores its local slope from the forward pass, and during the backward pass, it takes the sensitivity received from downstream, multiplies it by this stored slope, and distributes the result upstream along its connections. These local accounts, when summed, miraculously yield the correct global account.
The success of backpropagation laid the algorithmic foundation for deep learning. It propelled connectionism from academic ivory towers into practical application, providing the theoretical and technical prerequisites for the "deep learning" explosion years later.
