3.5 - Noah's Notes - Logistic regression with scikit-learn

In the notebook Logistic regression from scratch we predicted whether a Physiotherapy patient will miss their appointment, writing the sigmoid, the cost and gradient descent ourselves. Here we solve the same problem with scikit-learn, and we add the two tools of this chapter that a real project cannot do without:

• Regularization, which keeps the model from memorizing the training data (overfitting).
• k-fold cross-validation, which lets us choose the strength of the regularization without touching the test set.

Steps:

1. Import the libraries and create the data.
2. Train a logistic regression with scikit-learn.
3. Evaluate it: confusion matrix, classification report and ROC curve.
4. Overfitting: what happens with few patients and many variables.
5. L2 and L1 regularization.
6. Choosing $\lambda$ with k-fold cross-validation.
7. Elastic Net and correlated variables.
8. Imbalanced classes.

Step 1: Libraries and data

---

We use the same synthetic Physiotherapy appointments as in the from-scratch notebook (same generator and same seed, so the numbers can be compared):

Column	Meaning
age	Age of the patient (18–85 years)
days_since_last_visit	Days since the patient's previous appointment
distance_km	Distance from home to the hospital, in km (Minermont serves a rural area)
hour	Time of the appointment (8 to 19 h)
first_visit	1 if it is a first visit, 0 if it is a follow-up
previous_no_shows	Number of appointments the patient has missed before
no_show	Target: 1 if the patient did not attend, 0 if they did

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_validate
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (accuracy_score, confusion_matrix, ConfusionMatrixDisplay,
                             classification_report, roc_auc_score, roc_curve, RocCurveDisplay)

np.set_printoptions(precision=3, suppress=True)

def generate_appointments(n, seed):
    """Synthetic Physiotherapy appointments at Minermont: one row per appointment."""
    rng = np.random.default_rng(seed)
    age = rng.integers(18, 86, n)                                  # years
    days = rng.integers(1, 181, n)                                 # days since the previous appointment
    distance = np.round(rng.gamma(2.0, 6.0, n).clip(0.5, 60), 1)   # km to the hospital (a rural area)
    hour = rng.integers(8, 20, n)                                  # appointment time, from 8:00 to 19:00
    first = rng.binomial(1, 0.35, n)                               # 1 = first visit, 0 = follow-up
    previous = rng.poisson(0.6, n).clip(0, 6)                      # no-shows in the patient's history
    late = np.where(hour >= 17, hour - 16, 0)                      # late-afternoon slots are missed more

    # The hidden rule behind the data. We only use it to generate the appointments:
    # the model never sees it and has to discover the pattern from the examples.
    z = (-1.55 - 0.069 * (age - 50) + 0.026 * (days - 90) + 0.11 * (distance - 12)
         + 1.5 * (previous - 0.6) + 0.56 * (late - 0.75) + 0.63 * (first - 0.35))
    no_show = rng.binomial(1, 1 / (1 + np.exp(-z)))                # 1 = did not attend

    return pd.DataFrame({
        'age': age, 'days_since_last_visit': days, 'distance_km': distance, 'hour': hour,
        'first_visit': first, 'previous_no_shows': previous, 'no_show': no_show,
    })

df = generate_appointments(1200, seed=42)
feature_names = ['age', 'days_since_last_visit', 'distance_km', 'hour', 'first_visit', 'previous_no_shows']
X = df[feature_names]
y = df['no_show']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=42, stratify=y)

print(f"Training set: {len(y_train)} appointments, test set: {len(y_test)}")
print(f"No-show rate: {y.mean():.1%}")

Training set: 900 appointments, test set: 300
No-show rate: 26.2%

Step 2: Training with scikit-learn

---

A pipeline chains the scaling and the model into a single object. When we call fit, the scaler learns the mean and the standard deviation from the training data only, and when we call predict, it applies them to the new data. This avoids, by construction, the mistake of letting the test set leak into training.

LogisticRegression does not use plain gradient descent but a faster optimizer (L-BFGS), and it applies a light L2 regularization by default (C=1). With 900 appointments and six variables, its weights come out very close to the ones we obtained from scratch, and slightly smaller because of that light regularization.

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)

logreg = model[-1]
print("Weights:", pd.Series(logreg.coef_[0], index=feature_names).round(3).to_dict())
print(f"Bias: {logreg.intercept_[0]:.3f}")

Weights: {'age': -1.235, 'days_since_last_visit': 1.136, 'distance_km': 0.992, 'hour': 0.354, 'first_visit': 0.28, 'previous_no_shows': 1.0}
Bias: -1.776

Step 3: Evaluation

---

predict applies the 0.5 threshold; predict_proba returns the probabilities, which we need to try other thresholds. The classification report shows, for each class, the precision (what we called PPV for the no-show class), the recall (the sensitivity) and the F1-score, which combines both.

y_pred = model.predict(X_test)
p_test = model.predict_proba(X_test)[:, 1]

print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}  (always predicting 'attends': {1 - y_test.mean():.3f})\n")
print(classification_report(y_test, y_pred, target_names=['attends', 'no-show'], digits=3))

ConfusionMatrixDisplay.from_predictions(y_test, y_pred, display_labels=['attends', 'no-show'], cmap='Blues')
plt.xlabel('Predicted')
plt.ylabel('Actual')
plt.title('Confusion matrix (test set, threshold 0.5)')
plt.show()

Accuracy: 0.827  (always predicting 'attends': 0.737)

              precision    recall  f1-score   support

     attends      0.869     0.900     0.884       221
     no-show      0.690     0.620     0.653        79

    accuracy                          0.827       300
   macro avg      0.780     0.760     0.769       300
weighted avg      0.822     0.827     0.824       300

The ROC curve shows every threshold at once: for each one it plots the sensitivity (true positive rate) against 1 − specificity (false positive rate). A useless model follows the diagonal; the closer the curve gets to the top-left corner, the better. The area under the curve (AUC) summarizes it in a single number: 0.5 is chance and 1 is perfect. It can also be read as the probability that the model gives a higher score to a random no-show than to a random attendance.

print(f"AUC: {roc_auc_score(y_test, p_test):.3f}")

fpr, tpr, thresholds = roc_curve(y_test, p_test)
RocCurveDisplay(fpr=fpr, tpr=tpr, roc_auc=roc_auc_score(y_test, p_test)).plot()
for t, marker in [(0.5, 'o'), (0.3, 's')]:
    i = np.argmin(np.abs(thresholds - t))
    plt.plot(fpr[i], tpr[i], marker, markersize=10, label=f'threshold {t}')
plt.plot([0, 1], [0, 1], 'k--', label='chance')
plt.xlabel('False positive rate (1 - specificity)')
plt.ylabel('True positive rate (sensitivity)')
plt.title('ROC curve')
plt.legend()
plt.show()

AUC: 0.880

Moving from the 0.5 threshold to 0.3 climbs up the curve: more no-shows detected, at the price of more false alarms.

Step 4: Overfitting

---

In the book, the team's first hospital model was excellent on the training data and much worse on new patients: it had learned the particularities of the training examples, not the general pattern. It is easy to reproduce. We take a small sample of 300 appointments and add 50 extra columns that have nothing to do with attendance (think of all the other fields a hospital record contains). The complexity of the model grows with the number of variables, and with few examples it can memorize them.

To see pure overfitting we train with an almost negligible regularization (C=10000).

small = generate_appointments(300, seed=3)
X_small = small[feature_names].copy()
rng = np.random.default_rng(7)
for j in range(50):
    X_small[f'extra_{j}'] = rng.normal(size=len(small))   # irrelevant variables
y_small = small['no_show']

Xs_train, Xs_test, ys_train, ys_test = train_test_split(
    X_small, y_small, test_size=0.3, random_state=42, stratify=y_small)

overfit = make_pipeline(StandardScaler(), LogisticRegression(C=10000, max_iter=5000))
overfit.fit(Xs_train, ys_train)
print(f"Variables: {X_small.shape[1]}, training appointments: {len(ys_train)}")
print(f"Accuracy on the training set: {overfit.score(Xs_train, ys_train):.3f}")
print(f"Accuracy on the test set:     {overfit.score(Xs_test, ys_test):.3f}")
print(f"Always predicting 'attends':  {1 - ys_test.mean():.3f}")

Variables: 56, training appointments: 210
Accuracy on the training set: 1.000
Accuracy on the test set:     0.633
Always predicting 'attends':  0.722

Perfect on the training set, and worse on new appointments than simply assuming that everybody attends. That is overfitting.

Step 5: L2 and L1 regularization

---

Regularization adds to the cost a penalty that grows with the size of the weights, so the model can only use large weights when they really reduce the error:

$$J_{L2}(w, b) = \text{log loss} + \lambda \sum_j w_j^2 \qquad\qquad J_{L1}(w, b) = \text{log loss} + \lambda \sum_j |w_j|$$

• L2 (Ridge) shrinks all the weights towards zero, without cancelling them.
• L1 (Lasso) can set weights exactly to zero, so it also selects variables.
• The larger $\lambda$, the stronger the regularization. The bias $b$ is not penalized.

Careful with scikit-learn's convention. LogisticRegression does not take $\lambda$ but its inverse, C: a large C means weak regularization. For the formulas above, with $m$ training examples, the equivalence is $C = \frac{1}{2 \lambda m}$ for L2 and $C = \frac{1}{\lambda m}$ for L1.

m = len(ys_train)

def l2_model(lam):
    return make_pipeline(StandardScaler(), LogisticRegression(C=1 / (2 * lam * m), max_iter=5000))

def l1_model(lam):
    return make_pipeline(StandardScaler(),
                         LogisticRegression(C=1 / (lam * m), penalty='l1', solver='liblinear', max_iter=5000, random_state=0))

for lam in [0.001, 0.1]:
    mdl = l2_model(lam).fit(Xs_train, ys_train)
    w = mdl[-1].coef_[0]
    print(f"L2, lambda = {lam:<5}: size of the weights (sum of |w|) = {np.abs(w).sum():6.2f}, "
          f"training = {mdl.score(Xs_train, ys_train):.3f}, test = {mdl.score(Xs_test, ys_test):.3f}")

L2, lambda = 0.001: size of the weights (sum of |w|) =  26.14, training = 0.943, test = 0.733
L2, lambda = 0.1  : size of the weights (sum of |w|) =   3.81, training = 0.857, test = 0.744

A larger $\lambda$ shrinks the weights and closes the gap between training and test. But which $\lambda$ should we use? We cannot choose it by looking at the test set: then the test set would stop being "unseen data" and our final estimate would be optimistic.

Step 6: Choosing $\lambda$ with k-fold cross-validation (Noah's notes)

---

k-fold cross-validation answers the question using only the training data:

1. Split the training set into $k$ parts (folds) of similar size. With stratified folds each part keeps the proportion of no-shows.
2. For each candidate value of $\lambda$, train $k$ times: each time on $k-1$ folds, validating on the remaining one.
3. Average the $k$ validation scores. That average estimates how the model will behave on new data.
4. Choose the $\lambda$ with the best average and retrain with the whole training set.
5. Evaluate once on the test set, which has not been used for anything else.

As the chapter recommends, we use $k = 5$ and a logarithmic grid of values: $\lambda \in \{0.001, 0.01, 0.1, 1, 10, 100, 1000\}$.

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
lambdas = [0.001, 0.01, 0.1, 1, 10, 100, 1000]

rows = []
for lam in lambdas:
    scores = cross_validate(l2_model(lam), Xs_train, ys_train, cv=cv, return_train_score=True)
    rows.append({'lambda': lam, 'training': scores['train_score'].mean(), 'validation': scores['test_score'].mean()})
results = pd.DataFrame(rows)
print(results.round(3).to_string(index=False))

best_lambda = results.loc[results['validation'].idxmax(), 'lambda']
print(f"\nAlways predicting 'attends' on the training set: {1 - ys_train.mean():.3f}")
print(f"Best lambda (highest mean validation accuracy): {best_lambda:g}")

plt.figure(figsize=(8, 4))
plt.semilogx(results['lambda'], results['training'], 'o-', label='training (mean of the 5 folds)')
plt.semilogx(results['lambda'], results['validation'], 's-', label='validation (mean of the 5 folds)')
plt.axvline(best_lambda, color='gray', linestyle='--')
plt.xlabel('lambda (regularization strength)')
plt.ylabel('Accuracy')
plt.title('5-fold cross-validation, L2 regularization')
plt.legend()
plt.grid(True)
plt.show()

  lambda  training  validation
   0.001     0.977       0.752
   0.010     0.940       0.790
   0.100     0.845       0.752
   1.000     0.714       0.714
  10.000     0.714       0.714
 100.000     0.714       0.714
1000.000     0.714       0.714

Always predicting 'attends' on the training set: 0.714
Best lambda (highest mean validation accuracy): 0.01

The plot shows the two dangers. On the left (small $\lambda$) training accuracy is very high but validation is poor: overfitting. On the right (large $\lambda$) the weights are squashed towards zero and the model no longer learns anything: underfitting. From $\lambda = 1$ onwards both curves are stuck at exactly the accuracy of predicting that every patient attends. The best value lies in between.

The chapter also advises refining the search around the best value. We try a few values close to it, and then we evaluate the chosen model, once, on the test set:

fine = [best_lambda * f for f in [0.3, 0.5, 1, 2, 3]]
fine_scores = [cross_validate(l2_model(lam), Xs_train, ys_train, cv=cv)['test_score'].mean() for lam in fine]
best_lambda = fine[int(np.argmax(fine_scores))]
print("Refined search:", ", ".join(f"lambda {l:g} -> {s:.3f}" for l, s in zip(fine, fine_scores)))
print(f"Chosen lambda: {best_lambda:g}\n")

regularized = l2_model(best_lambda).fit(Xs_train, ys_train)
print(f"{'':<24}{'training':>10}{'test':>8}")
for name, mdl in [('Without regularization', overfit), (f'L2, lambda = {best_lambda:g}', regularized)]:
    print(f"{name:<24}{mdl.score(Xs_train, ys_train):>10.3f}{mdl.score(Xs_test, ys_test):>8.3f}")

Refined search: lambda 0.003 -> 0.767, lambda 0.005 -> 0.767, lambda 0.01 -> 0.790, lambda 0.02 -> 0.786, lambda 0.03 -> 0.786
Chosen lambda: 0.01

                          training    test
Without regularization       1.000   0.633
L2, lambda = 0.01            0.929   0.811

The regularized model gives up a little accuracy on the training set and gains a lot on new appointments.

L1 as a variable selector. With the same cross-validation we choose $\lambda$ for L1, and then look at which of the 56 weights survive:

l1_grid = [0.001, 0.003, 0.01, 0.03, 0.1, 0.3]
l1_scores = [cross_validate(l1_model(lam), Xs_train, ys_train, cv=cv)['test_score'].mean() for lam in l1_grid]
best_l1 = l1_grid[int(np.argmax(l1_scores))]

lasso = l1_model(best_l1).fit(Xs_train, ys_train)
weights = pd.Series(lasso[-1].coef_[0], index=X_small.columns)
kept = weights[weights != 0]
print(f"L1, lambda = {best_l1}: test accuracy = {lasso.score(Xs_test, ys_test):.3f}")
print(f"Variables with a non-zero weight: {len(kept)} of {len(weights)}")
print(kept.round(3).sort_values(key=np.abs, ascending=False).to_string())

L1, lambda = 0.03: test accuracy = 0.867
Variables with a non-zero weight: 10 of 56
days_since_last_visit    0.934
age                     -0.759
previous_no_shows        0.749
distance_km              0.500
first_visit              0.224
extra_5                  0.197
extra_3                 -0.057
extra_32                -0.057
extra_30                 0.046
extra_43                -0.028

L1 has removed most of the variables. The largest weights belong to real ones (days since the last visit, age, previous no-shows, distance), and the few noise columns that survive have small weights: with 50 random columns and only 210 appointments, some of them correlate with attendance by pure chance. The hour has been removed too, since its effect only appears in the last slots of the afternoon, and 210 appointments are too few to detect it. Regularization helps a great deal, but it does not replace good judgement about which variables make sense.

Step 7: Elastic Net and correlated variables

---

A hospital record often holds the same information more than once. Suppose that, besides the distance from home to the hospital, the record has the distance by road and the travel time by car. The three columns say almost the same thing. The book recommends Elastic Net for this situation, when there are correlated variables. It adds both penalties to the cost:

$$J_{\text{Elastic Net}}(w, b) = \text{log loss} + \lambda_1 \sum_j |w_j| + \lambda_2 \sum_j w_j^2$$

The L1 part can still remove variables, and the L2 part shares the weight among the variables that carry the same information. In scikit-learn it is penalty='elasticnet' with the saga solver, and the two lambdas become C and l1_ratio: $C = \frac{1}{m(\lambda_1 + 2\lambda_2)}$ and $\text{l1\_ratio} = \frac{\lambda_1}{\lambda_1 + 2\lambda_2}$.

We go back to the 900 training appointments of step 1 and add the two new columns and 20 irrelevant ones:

rng = np.random.default_rng(11)

def add_columns(X_part):
    X_part = X_part.copy()
    X_part['road_km'] = np.round(1.25 * X_part['distance_km'] + rng.normal(0, 1.5, len(X_part)), 1)   # distance by road
    X_part['travel_minutes'] = np.round(1.5 * X_part['distance_km'] + rng.normal(0, 2.5, len(X_part)))      # minutes by car
    for j in range(20):
        X_part[f'extra_{j}'] = rng.normal(size=len(X_part))                                          # irrelevant variables
    return X_part

Xg_train, Xg_test = add_columns(X_train), add_columns(X_test)
distance_group = ['distance_km', 'road_km', 'travel_minutes']
print(f"Variables: {Xg_train.shape[1]}, training appointments: {len(y_train)}")
print("Correlations between the three distance columns:")
print(Xg_train[distance_group].corr().round(3).to_string())

Variables: 28, training appointments: 900
Correlations between the three distance columns:
                distance_km  road_km  travel_minutes
distance_km           1.000    0.991           0.983
road_km               0.991    1.000           0.975
travel_minutes        0.983    0.975           1.000

We choose the lambdas of L1, L2 and Elastic Net with the same 5-fold cross-validation as in step 6. Elastic Net has two, so its grid has 12 combinations instead of 5 values: this is the extra tuning the book warns about. Then we train each model on the 900 appointments and look at the weights of the three distance columns, at how many irrelevant columns keep a weight, and at the accuracy on the test set.

m = len(y_train)   # 900 appointments now: l1_model, l2_model and the new function use it to convert lambda into C

def elastic_net_model(lam1, lam2):
    return make_pipeline(StandardScaler(),
                         LogisticRegression(C=1 / (m * (lam1 + 2 * lam2)), l1_ratio=lam1 / (lam1 + 2 * lam2),
                                            penalty='elasticnet', solver='saga', max_iter=20000, tol=1e-5, random_state=0))

def mean_validation(model):
    return cross_validate(model, Xg_train, y_train, cv=cv)['test_score'].mean()

grid = [0.001, 0.003, 0.01, 0.03, 0.1]
lam_l1 = max(grid, key=lambda lam: mean_validation(l1_model(lam)))
lam_l2 = max(grid, key=lambda lam: mean_validation(l2_model(lam)))
en_grid = [(lam1, lam2) for lam1 in [0.003, 0.01, 0.03] for lam2 in [0.003, 0.01, 0.03, 0.1]]
lam1, lam2 = max(en_grid, key=lambda lams: mean_validation(elastic_net_model(*lams)))

l1_name, en_name = f'L1, lambda = {lam_l1:g}', f'Elastic Net, lambdas = {lam1:g}, {lam2:g}'
models = {'Without regularization': make_pipeline(StandardScaler(), LogisticRegression(C=10000, max_iter=5000)),
          l1_name: l1_model(lam_l1),
          f'L2, lambda = {lam_l2:g}': l2_model(lam_l2),
          en_name: elastic_net_model(lam1, lam2)}
rows = []
for name, mdl in models.items():
    weights = pd.Series(mdl.fit(Xg_train, y_train)[-1].coef_[0], index=Xg_train.columns)
    rows.append({'model': name, **weights[distance_group].round(3).to_dict(),
                 'irrelevant kept': int((weights.filter(like='extra_') != 0).sum()),
                 'test accuracy': round(mdl.score(Xg_test, y_test), 3)})
print(pd.DataFrame(rows).to_string(index=False))

                             model  distance_km  road_km  travel_minutes  irrelevant kept  test accuracy
            Without regularization        0.997    0.520          -0.415               20          0.830
                 L1, lambda = 0.01        0.463    0.334           0.000                9          0.840
                L2, lambda = 0.001        0.527    0.553          -0.025               20          0.823
Elastic Net, lambdas = 0.01, 0.003        0.338    0.335           0.103                9          0.840

The three distance columns are correlated between 0.975 and 0.991. Without regularization, the model shares the distance among them in a way that makes no sense: 0.997 for the distance, 0.520 for the road and −0.415 for the travel time, a negative weight that on its own would mean that longer trips make patients more likely to attend. It also gives a weight to all 20 irrelevant columns. L2, with the lambda chosen by cross-validation, 0.001, is very light and barely changes that picture: the travel time still has a small negative weight, −0.025, and the 20 irrelevant columns keep theirs. L1 removes 11 of the 20 irrelevant columns, and it also removes the travel time, keeping 0.463 for the distance and 0.334 for the road. Elastic Net removes as many irrelevant columns, 9 remain, and keeps the three distance columns with positive weights: 0.338, 0.335 and 0.103. On the test set, the four models are between 0.823 and 0.840, a difference of 5 appointments out of 300: with 900 training appointments, the difference is in the weights, not in the accuracy.

How stable is the choice of L1? We train L1 and Elastic Net again on 30 bootstrap samples, each made of 900 appointments drawn at random, with replacement, from the training set, and count how many times each distance column keeps a weight:

boot_rng = np.random.default_rng(0)
samples = [boot_rng.integers(0, m, m) for _ in range(30)]
for name in [l1_name, en_name]:
    kept = pd.Series(0, index=distance_group)
    for idx in samples:
        mdl = models[name].fit(Xg_train.iloc[idx], y_train.iloc[idx])
        kept += (pd.Series(mdl[-1].coef_[0], index=Xg_train.columns)[distance_group] != 0).astype(int)
    print(f"{name:<32}", ", ".join(f"{col} {n} of 30" for col, n in kept.items()))

L1, lambda = 0.01                distance_km 23 of 30, road_km 22 of 30, travel_minutes 8 of 30

Elastic Net, lambdas = 0.01, 0.003 distance_km 30 of 30, road_km 30 of 30, travel_minutes 26 of 30

Which distance column L1 keeps depends on the sample: the distance in 23 of the 30 samples, the road in 22 and the travel time in only 8. Two models trained on almost the same appointments tell different stories about which variables matter. Elastic Net keeps the distance and the road in all 30 samples and the travel time in 26: its L2 part makes correlated columns share the weight, so they tend to stay in the model together. That is what the book means: Elastic Net can select variables, like L1, and reduce the weights of the others proportionally, like L2, and that is why it is especially useful when there are correlated variables.

Step 8: Imbalanced classes

---

Only about a quarter of the appointments are no-shows, so the model sees many more examples of attendance and leans towards predicting it. Besides moving the threshold, scikit-learn can weight the classes so that each error on the minority class counts more (class_weight='balanced'). Back to the full dataset:

def sensitivity_specificity(y_true, y_hat):
    tn, fp, fn, tp = confusion_matrix(y_true, y_hat).ravel()
    return tp / (tp + fn), tn / (tn + fp)

balanced = make_pipeline(StandardScaler(), LogisticRegression(class_weight='balanced'))
balanced.fit(X_train, y_train)

for name, y_hat in [('Default model, threshold 0.5', y_pred),
                    ('Default model, threshold 0.3', (p_test >= 0.3).astype(int)),
                    ("class_weight='balanced', threshold 0.5", balanced.predict(X_test))]:
    sens, spec = sensitivity_specificity(y_test, y_hat)
    print(f"{name:<40} accuracy {accuracy_score(y_test, y_hat):.3f} | sensitivity {sens:.3f} | specificity {spec:.3f}")

Default model, threshold 0.5             accuracy 0.827 | sensitivity 0.620 | specificity 0.900
Default model, threshold 0.3             accuracy 0.787 | sensitivity 0.772 | specificity 0.792
class_weight='balanced', threshold 0.5   accuracy 0.777 | sensitivity 0.797 | specificity 0.769

Weighting the classes has an effect similar to lowering the threshold: more no-shows detected, at the cost of some accuracy. Both are legitimate ways of telling the model that missing a no-show matters more to us than making a reminder call.

Conclusion

---

• With scikit-learn, a pipeline of StandardScaler and LogisticRegression reproduces in a few lines the model we wrote from scratch.
• A classifier is judged on the confusion matrix, the classification report and the ROC curve, not only on accuracy.
• With few examples and many variables the model overfits. Regularization penalizes large weights: L2 shrinks them and L1 can remove variables altogether.
• Elastic Net combines both penalties: it removes irrelevant variables like L1 and shares the weight among correlated variables like L2, at the price of a second $\lambda$ to tune.
• The regularization strength is chosen with k-fold cross-validation on the training data. The test set is used only once, at the end.
• With imbalanced classes, the threshold and the class weights let us put the balance between sensitivity and specificity where the hospital needs it.

The next notebooks tackle the same problem with the other model of this chapter, support vector machines.