6.2 Learning by Backpropagating Errors

Learning by Backpropagating Errors: Rumelhart, Hinton & Williams (1986)

In 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published a seminal paper that would revolutionize the field of neural networks. Their work, titled "Learning representations by back-propagating errors", reintroduced and popularized the backpropagation algorithm, providing a practical method for training multi-layer neural networks.

Historical Context

By the mid-1980s, neural networks faced a significant challenge known as the "AI winter." The limitations of single-layer perceptrons, famously highlighted by Minsky and Papert in 1969, had dampened enthusiasm for neural network research. While the core ideas of backpropagation had been independently discovered earlier by several researchers (including Paul Werbos in 1974), it was this 1986 Nature paper that brought the algorithm to widespread attention and demonstrated its practical power.

The Backpropagation Algorithm

The algorithm introduced a systematic way to train multi-layer neural networks:

Forward pass: Input data flows through the network, generating predictions
Error calculation: The difference between predictions and actual values is computed
Backward pass: Errors propagate backward through the network using the chain rule of calculus
Weight updates: Each connection's weight is adjusted proportionally to its contribution to the error

This elegant solution to the credit assignment problem—determining how much each neuron contributed to the final error—enabled networks to learn complex, hierarchical representations of data.

Impact and Legacy

The publication of this paper marked a turning point in artificial intelligence. It demonstrated that neural networks could learn internal representations and solve problems that had previously seemed intractable. This work directly enabled:

The development of deep learning in the 2000s
Modern convolutional neural networks (CNNs)
Recurrent networks and transformers
Virtually all contemporary neural network architectures

Geoffrey Hinton would later be awarded the 2024 Nobel Prize in Physics (alongside John Hopfield) for foundational discoveries that enabled machine learning with artificial neural networks, with backpropagation being central to that contribution.

Original Article

The paper was published in Nature, one of the world's most prestigious scientific journals, making it accessible to a broad scientific audience and amplifying its impact.

References

Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986). "Learning representations by back-propagating errors". Nature, 323(6088), 533-536.
Original Article - DOI Link
PDF Version - MIT

For earlier work on backpropagation, see:

Werbos, P. J. (1974). "Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences". PhD Thesis, Harvard University.