7 - Neural networks as function approximators¶
Martha felt that the biological metaphors left something missing, and in her notes (section 7.6) she describes neural networks as what they are mathematically: function approximators that combine simple building blocks, basis functions. The universal approximation theorem says that a network with a single, wide enough hidden layer can approximate any continuous function on a bounded interval as closely as we want. In this notebook we see why, with ReLU neurons:
- Import the libraries.
- The building block: a ReLU as a hinged ruler.
- Adding hinges: piecewise-linear approximations.
- Letting a network choose where to put the hinges.
- Why depth helps.
Step 1: Import the libraries¶
import numpy as np
import matplotlib.pyplot as plt
np.set_printoptions(precision=3, suppress=True)
def relu(z):
return np.maximum(0, z)
Step 2: A ReLU as a hinged ruler¶
A shifted and scaled ReLU, $c \cdot \text{ReLU}(x - b)$, is what Martha calls a hinged ruler: it lies flat on the axis until the point $x = b$ (the kink), and from there on it is a straight line with slope $c$.
x = np.linspace(-3, 3, 400)
plt.figure(figsize=(8, 4))
for c, b in [(1.0, -2.0), (0.5, 0.0), (-1.5, 1.0), (2.0, 2.0)]:
plt.plot(x, c * relu(x - b), label=f'{c} * ReLU(x - ({b}))')
plt.axhline(0, color='gray', linewidth=0.5)
plt.title('Shifted and scaled ReLUs: each one starts to bend at its own kink')
plt.legend()
plt.grid(True)
plt.show()
Step 3: Adding hinges¶
Adding several hinges, $\hat{f}(x) = \sum_i c_i \,\text{ReLU}(x - b_i)$ (plus a straight line), gives a continuous piecewise-linear curve: between two consecutive kinks the slope is constant, and at each kink $b_i$ it changes by exactly $c_i$. With more kinks we get more, shorter segments, and we can trace a smooth curve as closely as we want.
We try it with a curvy target function, $f(x) = \sin(2x) + 0.3x^2$. We place the kinks evenly and compute the best coefficients $c_i$ by least squares (the same method as linear regression in chapter 2):
target = np.sin(2 * x) + 0.3 * x ** 2
def fit_hinges(n_kinks):
kinks = np.linspace(-3, 3, n_kinks + 2)[1:-1]
A = np.c_[np.ones_like(x), x, relu(x[:, None] - kinks[None, :])] # a straight line + one column per hinge
coef, *_ = np.linalg.lstsq(A, target, rcond=None)
return kinks, coef, A @ coef
fig, axes = plt.subplots(1, 4, figsize=(18, 4))
for ax, n in zip(axes, [3, 6, 12, 24]):
kinks, coef, approx = fit_hinges(n)
rmse = np.sqrt(np.mean((approx - target) ** 2))
print(f"{n:>2} hinges: root mean squared error {rmse:.4f}, largest error {np.max(np.abs(approx - target)):.4f}")
ax.plot(x, target, 'k', linewidth=2, label='target f(x)')
ax.plot(x, approx, 'r--', label='sum of hinges')
ax.plot(kinks, np.interp(kinks, x, approx), 'ro', markersize=4)
ax.set_title(f'{n} hinges: error {rmse:.3f}')
ax.legend(fontsize=8)
plt.show()
kinks, coef, approx = fit_hinges(6)
slope = np.diff(approx) / np.diff(x)
jumps = [slope[np.searchsorted(x, b) + 2] - slope[np.searchsorted(x, b) - 2] for b in kinks]
print(f"\nWith 6 hinges, slope change at each kink: {np.round(jumps, 3)}")
print(f"Coefficients c_i of the hinges: {np.round(coef[2:], 3)}")
3 hinges: root mean squared error 0.5228, largest error 1.1246 6 hinges: root mean squared error 0.1022, largest error 0.2371 12 hinges: root mean squared error 0.0254, largest error 0.0750 24 hinges: root mean squared error 0.0065, largest error 0.0210
With 6 hinges, slope change at each kink: [-2.3 2.259 3.069 -2.04 -1.231 3.333] Coefficients c_i of the hinges: [-2.3 2.259 3.069 -2.04 -1.231 3.333]
Each time we double the number of hinges, the error falls by about four: from 0.52 with 3 hinges to 0.0065 with 24, where the approximation is hard to tell apart from the curve. And the slope of the approximation changes at each kink by exactly the coefficient $c_i$ of its hinge, as Martha's formula says.
Step 4: Letting a network choose the hinges¶
A neural network with one hidden layer of ReLU neurons computes exactly this kind of sum: each hidden neuron $\text{ReLU}(w_i x + b_i)$ is a hinge with its kink at $x = -b_i / w_i$, and the output layer adds them up with weights $a_i$. The difference is that the network also learns where to put the kinks. We train a network with 12 hidden neurons, minimizing the mean squared error with the Adam optimizer (gradient descent with an adaptive step), from 6 different random starts:
def train_network(seed, hidden=12, learning_rate=0.01, steps=20000):
rng = np.random.default_rng(seed)
w = rng.choice([-1, 1], hidden) * rng.uniform(0.5, 1.5, hidden)
params = [w, -w * rng.uniform(-3, 3, hidden), rng.normal(0, 0.1, hidden), np.zeros(1)] # kinks start spread out
m, v = [np.zeros_like(p) for p in params], [np.zeros_like(p) for p in params]
for t in range(1, steps + 1):
z = x[:, None] * params[0] + params[1]
h = relu(z) # hidden layer: one hinge per neuron
error = h @ params[2] + params[3][0] - target
d_out = 2 * error / len(x)
d_hidden = d_out[:, None] * params[2] * (z > 0) # backpropagation through the ReLU
grads = [(d_hidden * x[:, None]).sum(axis=0), d_hidden.sum(axis=0), h.T @ d_out, np.array([d_out.sum()])]
for i, g in enumerate(grads): # Adam update
m[i] = 0.9 * m[i] + 0.1 * g
v[i] = 0.999 * v[i] + 0.001 * g ** 2
params[i] -= learning_rate * (m[i] / (1 - 0.9 ** t)) / (np.sqrt(v[i] / (1 - 0.999 ** t)) + 1e-8)
return np.sqrt(np.mean(error ** 2)), params
runs = [train_network(seed) for seed in range(6)]
for seed, (rmse, _) in enumerate(runs):
print(f"Start {seed}: root mean squared error {rmse:.4f}")
best_rmse, (w, b, a, c) = min(runs, key=lambda r: r[0])
learned_kinks = np.sort((-b / w)[(-b / w > -3) & (-b / w < 3)])
print(f"\nBest network: error {best_rmse:.4f}, kinks inside [-3, 3]: {learned_kinks}")
approx = relu(x[:, None] * w + b) @ a + c[0]
plt.figure(figsize=(9, 4))
plt.plot(x, target, 'k', linewidth=2, label='target f(x)')
plt.plot(x, approx, 'r--', label='trained network (12 neurons)')
for k in learned_kinks:
plt.axvline(k, color='red', alpha=0.25)
plt.title('The trained network places its kinks where the curve bends')
plt.legend()
plt.show()
Start 0: root mean squared error 0.0505 Start 1: root mean squared error 0.0358 Start 2: root mean squared error 0.0438 Start 3: root mean squared error 0.1602 Start 4: root mean squared error 0.0333 Start 5: root mean squared error 0.0425 Best network: error 0.0333, kinks inside [-3, 3]: [-2.276 -1.118 -0.826 -0.771 -0.758 -0.715 -0.417 0.567 0.995 2.028 2.495]
Five of the six starts reach an error between 0.033 and 0.051, and one gets stuck at 0.160. The best network does not spread its kinks evenly: it concentrates them near the points where the target bends the most, around ±0.8 and ±2.4. Even so, the best one (0.033) is not better than our 12 evenly spaced hinges fitted by least squares (0.025). As Martha warns, the theorem guarantees that good weights exist, not that training will find them.
Step 5: Why depth helps¶
Martha adds that, although one layer is enough in theory, deep networks are far more efficient in practice: each layer transforms the output of the previous one. A classic example makes this concrete. A layer of just 2 ReLU neurons can build a "tent" on the interval $[0, 1]$, $t(x) = 2\,\text{ReLU}(x) - 4\,\text{ReLU}(x - 0.5)$, which goes up and then down. Stacking the same layer $L$ times folds the interval again and again. We count the linear pieces of the result:
def tent(u):
return 2 * relu(u) - 4 * relu(u - 0.5) # a layer of 2 ReLU neurons
u = np.linspace(0, 1, 2 ** 12 + 1) # the kinks fall exactly on grid points
fig, axes = plt.subplots(1, 4, figsize=(18, 3.5))
y = u
for depth in range(1, 9):
y = tent(y)
slopes = np.round(np.diff(y) / np.diff(u), 3)
pieces = 1 + np.count_nonzero(np.diff(slopes))
print(f"{depth} layer(s) of 2 neurons = {2 * depth:>2} neurons: {pieces:>3} linear pieces")
if depth in (1, 2, 3, 5):
ax = axes[[1, 2, 3, 5].index(depth)]
ax.plot(u, y)
ax.set_title(f'{depth} layer(s): {pieces} pieces')
plt.show()
1 layer(s) of 2 neurons = 2 neurons: 2 linear pieces 2 layer(s) of 2 neurons = 4 neurons: 4 linear pieces 3 layer(s) of 2 neurons = 6 neurons: 8 linear pieces 4 layer(s) of 2 neurons = 8 neurons: 16 linear pieces 5 layer(s) of 2 neurons = 10 neurons: 32 linear pieces 6 layer(s) of 2 neurons = 12 neurons: 64 linear pieces 7 layer(s) of 2 neurons = 14 neurons: 128 linear pieces 8 layer(s) of 2 neurons = 16 neurons: 256 linear pieces
Each new layer of just 2 neurons doubles the number of linear pieces: 16 neurons in 8 layers produce 256 pieces. A network with a single hidden layer has at most one kink per neuron, so it would need about 255 neurons to build the same shape. That is Martha's point: composing simple transformations, one after another, is far more efficient than placing them side by side.
Conclusion¶
- A ReLU neuron is a hinged ruler, and a sum of them is a continuous piecewise-linear curve that can follow any continuous function as closely as we want, if we add enough hinges. That is the idea behind the universal approximation theorem.
- The theorem guarantees that a good approximation exists, but not that training will find it: one of our starts got stuck with a clearly worse result.
- A trained network learns both where to place its kinks and how much each one bends the curve, and it places them where the function changes most.
- Depth composes transformations: with a few neurons per layer, each new layer can multiply the number of linear pieces. This is one reason why deep networks capture complex patterns with fewer parameters than shallow ones.
- Language models are, at bottom, enormous function approximators of this kind: they map a sequence of tokens to the probabilities of the next one.