3.5 - Noah's Notes - Support vector machines (SVM) with scikit-learn

In the notebook SVM from scratch we trained a linear SVM with gradient descent. scikit-learn's SVC solves the same problem with a specialized optimizer, and above all it offers the kernel trick, which lets an SVM draw curved boundaries. In this notebook:

1. Import the libraries.
2. The book's example with SVC: the same line, and the effect of C.
3. The kernel trick: when a straight line is not enough.
4. Linear, polynomial and RBF kernels.
5. The parameter gamma and overfitting.
6. The Physiotherapy no-shows: linear or RBF kernel, chosen with cross-validation.
7. Final comparison with logistic regression.
8. Probabilities with an SVM.

Step 1: Import the libraries

---

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.svm import SVC
from sklearn.datasets import make_circles
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, confusion_matrix, roc_auc_score

np.set_printoptions(precision=3, suppress=True)

Step 2: The book's example with SVC

---

We use again the 14 points of the linear SVM figure in section 3.4. SVC minimizes exactly the book's soft-margin cost,

$$\frac{1}{2}\|w\|^2 + C \sum_{i=1}^{m} \xi_i$$

so scikit-learn's C is the book's $\lambda$. Careful: in LogisticRegression, C was the inverse of the regularization strength. In both cases a large C means a model that fits the training data more closely, but the formulas are not the same.

With a large C no violation of the margin is tolerated (a hard margin), and we should recover the book's line $x_2 = 0.5\,x_1 + 1.375$:

positive = np.array([[1, 5], [1.5, 4], [2, 5.5], [2.5, 4.5], [3, 5], [3.5, 3.75], [4, 4.8]])
negative = np.array([[3, 2], [3.5, 1.5], [4, 2.75], [4.5, 1.8], [5, 2.3], [5.5, 1.6], [6, 2]])
X_toy = np.vstack([positive, negative])
y_toy = np.array([1] * len(positive) + [-1] * len(negative))

svm = SVC(kernel='linear', C=1000).fit(X_toy, y_toy)
w, b = svm.coef_[0], svm.intercept_[0]
print(f"w = {w.round(3)},  b = {b:.3f}")
print(f"Boundary:        x2 = {-w[0] / w[1]:.3f} x1 + {-b / w[1]:.3f}")
print(f"Margin width:    {2 / np.linalg.norm(w):.3f}")
print(f"Support vectors: {svm.support_vectors_.tolist()}")

w = [-0.8  1.6],  b = -2.200
Boundary:        x2 = 0.500 x1 + 1.375
Margin width:    1.118
Support vectors: [[4.0, 2.75], [3.5, 3.75]]

The same line and the same two support vectors that we obtained from scratch, now with the exact values. Let's see what happens when we lower C, that is, when violations of the margin become cheaper. To draw the boundaries we write a small function that we will reuse for every model in this notebook:

def plot_decision(model, X, y, ax, title, margins=True):
    """Points, boundary (solid), margins (dashed) and support vectors (circled) of a fitted SVC."""
    pad = 0.5
    x1, x2 = np.meshgrid(np.linspace(X[:, 0].min() - pad, X[:, 0].max() + pad, 300),
                         np.linspace(X[:, 1].min() - pad, X[:, 1].max() + pad, 300))
    f = model.decision_function(np.c_[x1.ravel(), x2.ravel()]).reshape(x1.shape)
    ax.contourf(x1, x2, f > 0, alpha=0.15, cmap='coolwarm_r')
    levels, styles = ([-1, 0, 1], ['--', '-', '--']) if margins else ([0], ['-'])
    ax.contour(x1, x2, f, levels=levels, colors='k', linestyles=styles, linewidths=1)
    classes = np.unique(y)
    ax.scatter(*X[y == classes[-1]].T, color='tab:blue', s=25)
    ax.scatter(*X[y == classes[0]].T, color='tab:red', marker='s', s=25)
    ax.scatter(*model.support_vectors_.T, s=120, facecolors='none', edgecolors='k', linewidths=0.8)
    ax.set_title(title)

fig, axes = plt.subplots(1, 3, figsize=(15, 4.5))
for C, ax in zip([1000, 1, 0.1], axes):
    model = SVC(kernel='linear', C=C).fit(X_toy, y_toy)
    margin = 2 / np.linalg.norm(model.coef_[0])
    print(f"C = {C:<6} margin = {margin:.3f}   support vectors = {len(model.support_vectors_)}")
    plot_decision(model, X_toy, y_toy, ax, f'C = {C}: margin {margin:.2f}, {len(model.support_vectors_)} support vectors')
plt.show()

C = 1000   margin = 1.118   support vectors = 2
C = 1      margin = 1.789   support vectors = 2
C = 0.1    margin = 3.130   support vectors = 8

With C = 1000 we have the hard margin. With C = 1 the margin widens to 1.789: the two closest points are now inside it and each pays a small penalty. With C = 0.1 the margin is almost three times as wide as the hard one and 8 points lean on it. The smaller C, the more violations we tolerate and the more points take part in placing the boundary.

Step 3: The kernel trick

---

A linear SVM can only draw straight lines (hyperplanes, with more variables). Some problems need something else. In the following dataset, one class forms a ring around the other: no straight line can separate them.

X_ring, y_ring = make_circles(n_samples=300, noise=0.15, factor=0.4, random_state=0)
Xr_train, Xr_test, yr_train, yr_test = train_test_split(
    X_ring, y_ring, test_size=0.3, random_state=42, stratify=y_ring)

linear_ring = SVC(kernel='linear').fit(Xr_train, yr_train)
print(f"Linear SVM: training accuracy = {linear_ring.score(Xr_train, yr_train):.3f}, "
      f"test accuracy = {linear_ring.score(Xr_test, yr_test):.3f}")

fig, ax = plt.subplots(figsize=(5.5, 5))
plot_decision(linear_ring, X_ring, y_ring, ax, 'A straight line cannot separate a ring')
plt.show()

Linear SVM: training accuracy = 0.614, test accuracy = 0.478

Worse than tossing a coin. The idea of the book's figure (section 3.4) is to add a new dimension in which the classes do become separable. Here the natural choice is the squared distance to the center, $\phi(x_1, x_2) = (x_1,\; x_2,\; x_1^2 + x_2^2)$: the inner points stay low and the ring goes up, so a flat plane can separate them.

def phi(X):
    return np.c_[X, (X ** 2).sum(axis=1)]

linear_3d = SVC(kernel='linear').fit(phi(Xr_train), yr_train)
print(f"Linear SVM on (x1, x2, x1² + x2²): test accuracy = {linear_3d.score(phi(Xr_test), yr_test):.3f}")

Z = phi(X_ring)
fig = plt.figure(figsize=(12, 5))
ax1 = fig.add_subplot(1, 2, 1)
ax1.scatter(*X_ring[y_ring == 1].T, color='tab:blue', s=15, label='inner class')
ax1.scatter(*X_ring[y_ring == 0].T, color='tab:red', marker='s', s=15, label='ring')
ax1.set_title('Original space (2D)')
ax1.legend()
ax2 = fig.add_subplot(1, 2, 2, projection='3d')
ax2.scatter(*Z[y_ring == 1].T, color='tab:blue', s=15)
ax2.scatter(*Z[y_ring == 0].T, color='tab:red', marker='s', s=15)
g1, g2 = np.meshgrid(np.linspace(-1.5, 1.5, 10), np.linspace(-1.5, 1.5, 10))
w3, b3 = linear_3d.coef_[0], linear_3d.intercept_[0]
ax2.plot_surface(g1, g2, -(w3[0] * g1 + w3[1] * g2 + b3) / w3[2], alpha=0.3, color='gray')
ax2.set_zlabel('$x_1^2 + x_2^2$')
ax2.set_box_aspect(None, zoom=0.85)
ax2.set_title('Transformed space (3D): a plane separates them')
plt.show()

Linear SVM on (x1, x2, x1² + x2²): test accuracy = 0.967

A flat plane in 3D is a circle when we look at it from above, in the original 2D space. Choosing $\phi$ by hand, however, requires knowing the shape of the problem in advance, and with many variables the transformed space can be enormous.

The kernel trick avoids both problems. To train and to predict, an SVM only needs inner products between pairs of points, $\phi(x_i) \cdot \phi(x_j)$. A kernel function $K(x_i, x_j)$ computes that inner product directly from the original points, without ever building $\phi(x)$. The kernels in the book:

Kernel	$K(x_i, x_j)$	In SVC
Linear	$x_i \cdot x_j$	kernel='linear'
Polynomial	$(x_i \cdot x_j + 1)^d$	kernel='poly', degree=d, gamma=1, coef0=1
RBF (Gaussian)	$\exp(-\gamma \|x_i - x_j\|^2)$	kernel='rbf', gamma=γ

Step 4: Linear, polynomial and RBF kernels

---

kernels = {
    'linear': SVC(kernel='linear'),
    'polynomial, d = 2': SVC(kernel='poly', degree=2, gamma=1, coef0=1),
    'RBF, gamma = 1': SVC(kernel='rbf', gamma=1),
}
fig, axes = plt.subplots(1, 3, figsize=(15, 4.5))
for (name, model), ax in zip(kernels.items(), axes):
    model.fit(Xr_train, yr_train)
    train_acc, test_acc = model.score(Xr_train, yr_train), model.score(Xr_test, yr_test)
    print(f"{name:<18} training = {train_acc:.3f}   test = {test_acc:.3f}")
    plot_decision(model, X_ring, y_ring, ax, f'{name}: test {test_acc:.2f}')
plt.show()

linear             training = 0.614   test = 0.478

polynomial, d = 2  training = 0.967   test = 0.956

RBF, gamma = 1     training = 0.976   test = 0.967

The polynomial kernel of degree 2 contains the squares $x_1^2$ and $x_2^2$, so it can draw exactly the kind of boundary we built by hand. The RBF kernel has not been told anything about circles: it measures how similar two points are by their distance, and it can bend the boundary to follow almost any shape. That is why it is the most widely used kernel, and also why it has to be handled with care.

Step 5: The parameter gamma and overfitting

---

In the RBF kernel, $\gamma$ says how far the influence of each training point reaches. With a small $\gamma$ each point influences a wide area, and the boundary is smooth. With a large $\gamma$ each point only influences its immediate surroundings, and the boundary can wrap around individual points: that is overfitting again.

fig, axes = plt.subplots(1, 3, figsize=(15, 4.5))
for gamma, ax in zip([0.1, 1, 100], axes):
    model = SVC(kernel='rbf', gamma=gamma).fit(Xr_train, yr_train)
    train_acc, test_acc = model.score(Xr_train, yr_train), model.score(Xr_test, yr_test)
    print(f"gamma = {gamma:<4} training = {train_acc:.3f}   test = {test_acc:.3f}   "
          f"support vectors = {len(model.support_vectors_)} of {len(yr_train)}")
    plot_decision(model, Xr_train, yr_train, ax, f'gamma = {gamma}\ntraining {train_acc:.2f}, test {test_acc:.2f}',
                  margins=False)
plt.show()

gamma = 0.1  training = 0.862   test = 0.822   support vectors = 195 of 210

gamma = 1    training = 0.976   test = 0.967   support vectors = 56 of 210
gamma = 100  training = 1.000   test = 0.956   support vectors = 182 of 210

With $\gamma = 0.1$ the boundary is too smooth to follow the ring: it underfits (0.862 in training, 0.822 in test). With $\gamma = 1$ it follows the ring and generalizes well, with only 56 support vectors. With $\gamma = 100$ it reaches 100% on the training set by wrapping the boundary around individual points, it does slightly worse on new data, and it needs 182 of the 210 points as support vectors: a sign that it is memorizing the examples rather than summarizing them. With more noise in the data, the gap between training and test would be much larger.

C and gamma are chosen in the same way as $\lambda$ in the previous notebooks: with cross-validation.

Step 6: The Physiotherapy no-shows

---

Back to the chapter's problem, with the same synthetic appointments and the same split as in the other notebooks:

Column	Meaning
age	Age of the patient (18–85 years)
days_since_last_visit	Days since the patient's previous appointment
distance_km	Distance from home to the hospital, in km (Minermont serves a rural area)
hour	Time of the appointment (8 to 19 h)
first_visit	1 if it is a first visit, 0 if it is a follow-up
previous_no_shows	Number of appointments the patient has missed before
no_show	Target: 1 if the patient did not attend, 0 if they did

def generate_appointments(n, seed):
    """Synthetic Physiotherapy appointments at Minermont: one row per appointment."""
    rng = np.random.default_rng(seed)
    age = rng.integers(18, 86, n)                                  # years
    days = rng.integers(1, 181, n)                                 # days since the previous appointment
    distance = np.round(rng.gamma(2.0, 6.0, n).clip(0.5, 60), 1)   # km to the hospital (a rural area)
    hour = rng.integers(8, 20, n)                                  # appointment time, from 8:00 to 19:00
    first = rng.binomial(1, 0.35, n)                               # 1 = first visit, 0 = follow-up
    previous = rng.poisson(0.6, n).clip(0, 6)                      # no-shows in the patient's history
    late = np.where(hour >= 17, hour - 16, 0)                      # late-afternoon slots are missed more

    # The hidden rule behind the data. We only use it to generate the appointments:
    # the model never sees it and has to discover the pattern from the examples.
    z = (-1.55 - 0.069 * (age - 50) + 0.026 * (days - 90) + 0.11 * (distance - 12)
         + 1.5 * (previous - 0.6) + 0.56 * (late - 0.75) + 0.63 * (first - 0.35))
    no_show = rng.binomial(1, 1 / (1 + np.exp(-z)))                # 1 = did not attend

    return pd.DataFrame({
        'age': age, 'days_since_last_visit': days, 'distance_km': distance, 'hour': hour,
        'first_visit': first, 'previous_no_shows': previous, 'no_show': no_show,
    })

df = generate_appointments(1200, seed=42)
feature_names = ['age', 'days_since_last_visit', 'distance_km', 'hour', 'first_visit', 'previous_no_shows']
X = df[feature_names]
y = df['no_show']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)
print(f"Training set: {len(y_train)} appointments, test set: {len(y_test)}")

Training set: 900 appointments, test set: 300

GridSearchCV automates the procedure of Noah's notes (section 3.5): for each combination of values in the grid it runs a stratified 5-fold cross-validation on the training set, keeps the best combination and retrains with the whole training set. The scaling is part of the pipeline, so it is recomputed in each fold.

For the linear kernel we try the book's logarithmic grid of values for C (the book's $\lambda$). For the RBF kernel we also have to choose gamma, so the grid has two dimensions:

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

linear_search = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel='linear')),
                             {'svc__C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]}, cv=cv)
linear_search.fit(X_train, y_train)
print("Linear kernel, mean validation accuracy for each C:")
for C, score in zip([0.001, 0.01, 0.1, 1, 10, 100, 1000], linear_search.cv_results_['mean_test_score']):
    print(f"  C = {C:<6} {score:.3f}")
print(f"Best: C = {linear_search.best_params_['svc__C']}  ({linear_search.best_score_:.3f})\n")

rbf_search = GridSearchCV(make_pipeline(StandardScaler(), SVC(kernel='rbf')),
                          {'svc__C': [0.01, 0.1, 1, 10, 100, 1000], 'svc__gamma': [0.001, 0.01, 0.1, 1]}, cv=cv)
rbf_search.fit(X_train, y_train)
grid = pd.DataFrame(rbf_search.cv_results_).pivot_table(
    index='param_svc__C', columns='param_svc__gamma', values='mean_test_score')
grid.index.name, grid.columns.name = 'C', 'gamma'
print("RBF kernel, mean validation accuracy:")
print(grid.round(3).to_string())
print(f"Best: C = {rbf_search.best_params_['svc__C']}, gamma = {rbf_search.best_params_['svc__gamma']}  "
      f"({rbf_search.best_score_:.3f})")

Linear kernel, mean validation accuracy for each C:
  C = 0.001  0.738
  C = 0.01   0.808
  C = 0.1    0.839
  C = 1      0.832
  C = 10     0.832
  C = 100    0.831
  C = 1000   0.831
Best: C = 0.1  (0.839)

RBF kernel, mean validation accuracy:
gamma    0.001  0.010  0.100  1.000
C                                  
0.01     0.738  0.738  0.738  0.738
0.10     0.738  0.738  0.783  0.738
1.00     0.738  0.832  0.831  0.771
10.00    0.832  0.832  0.826  0.750
100.00   0.834  0.838  0.809  0.734
1000.00  0.833  0.826  0.791  0.734
Best: C = 100, gamma = 0.01  (0.838)

The table shows the two dangers we already know. In the top-left corner (small C and small gamma) the model is too rigid and predicts that everybody attends (0.738). In the right-hand column (large gamma) the validation accuracy drops again, because the boundary bends around individual appointments. The best RBF model (0.838) is no better than the best linear one (0.839). That is not a surprise: the hidden rule that generated the data is, apart from the late-afternoon slots, a linear combination of the variables, so the extra flexibility of the RBF kernel buys nothing.

Step 7: Final comparison on the test set

---

Now, and only now, we use the test set, once, to compare the chosen models with the logistic regression of the previous notebooks. We also add a linear SVM with class_weight='balanced', which gives more weight to the errors on the minority class. The AUC of an SVM is computed from its score, decision_function, since it does not produce probabilities.

best_C = linear_search.best_params_['svc__C']
models = {
    'Logistic regression': make_pipeline(StandardScaler(), LogisticRegression()).fit(X_train, y_train),
    f'Linear SVM (C = {best_C})': linear_search.best_estimator_,
    'RBF SVM (C = {svc__C}, gamma = {svc__gamma})'.format(**rbf_search.best_params_): rbf_search.best_estimator_,
    "Linear SVM, class_weight='balanced'": make_pipeline(
        StandardScaler(), SVC(kernel='linear', C=best_C, class_weight='balanced')).fit(X_train, y_train),
}

rows = []
for name, model in models.items():
    y_hat = model.predict(X_test)
    score = model.predict_proba(X_test)[:, 1] if hasattr(model, 'predict_proba') else model.decision_function(X_test)
    tn, fp, fn, tp = confusion_matrix(y_test, y_hat).ravel()
    rows.append({'model': name, 'accuracy': accuracy_score(y_test, y_hat),
                 'sensitivity': tp / (tp + fn), 'specificity': tn / (tn + fp), 'AUC': roc_auc_score(y_test, score)})
print(pd.DataFrame(rows).set_index('model').round(3).to_string())

                                     accuracy  sensitivity  specificity    AUC
model                                                                         
Logistic regression                     0.827        0.620        0.900  0.880
Linear SVM (C = 0.1)                    0.833        0.582        0.923  0.884
RBF SVM (C = 100, gamma = 0.01)         0.817        0.608        0.891  0.876
Linear SVM, class_weight='balanced'     0.770        0.810        0.756  0.874

On the test set the models are very close. The linear SVM has the highest accuracy and AUC (0.833 and 0.884), with logistic regression right behind, and the RBF SVM, as cross-validation anticipated, does not improve on them. With class_weight='balanced' the linear SVM detects 81% of the no-shows (sensitivity 0.810) in exchange for more false alarms, the same trade-off we obtained in the other notebooks by moving the threshold. Keep the scale in mind, though: one point of accuracy on 300 appointments is just three appointments.

Step 8: Probabilities with an SVM

---

An SVM gives a score, not a probability. If we need one, SVC(probability=True) calibrates the score: it fits a sigmoid that turns the score into a probability (Platt scaling), using an internal cross-validation. It is slower to train, and the probability is an add-on, not part of the model itself. Let's compare it with logistic regression for three new appointments:

calibrated = make_pipeline(StandardScaler(), SVC(kernel='linear', C=best_C, probability=True, random_state=42))
calibrated.fit(X_train, y_train)

new_appointments = pd.DataFrame([
    [35, 100, 18, 18, 1, 0],
    [33, 80, 14, 17, 1, 0],
    [62, 30, 6, 9, 0, 0],
], columns=feature_names)

result = new_appointments.copy()
result['SVM score'] = calibrated.decision_function(new_appointments).round(3)
result['P(no-show) SVM'] = calibrated.predict_proba(new_appointments)[:, 1].round(3)
result['P(no-show) logistic'] = models['Logistic regression'].predict_proba(new_appointments)[:, 1].round(3)
print(result.to_string(index=False))

 age  days_since_last_visit  distance_km  hour  first_visit  previous_no_shows  SVM score  P(no-show) SVM  P(no-show) logistic
  35                    100           18    18            1                  0      0.043           0.534                0.568
  33                     80           14    17            1                  0     -0.464           0.347                0.360
  62                     30            6     9            0                  0     -3.767           0.003                0.003

The calibrated probabilities of the SVM are very close to those of logistic regression. The first appointment is on the edge (score 0.043, probability 0.534); the second has a probability of about a third in both models, so it would only be flagged with a lower threshold; the third will almost certainly be attended. Note that predict uses the sign of the score, not the calibrated probability, so in borderline cases the two can disagree.

Conclusion

---

• SVC solves the book's soft-margin problem; its C is the book's $\lambda$ (and not the inverse, as in LogisticRegression).
• The kernel trick lets an SVM draw curved boundaries by computing inner products in a transformed space without building it. The RBF kernel can follow almost any shape, and its gamma controls how closely it follows the training data.
• C and gamma are chosen with cross-validation (GridSearchCV with stratified folds), and the test set is used only once, at the end.
• On the no-show data, whose pattern is essentially linear, the linear SVM and logistic regression perform alike, and the flexibility of the RBF kernel does not help. Choosing a model means trying the candidates honestly and keeping the simplest one that works, which is exactly what the team at Minermont has to do.