4.2 - Random forest for emergency triage¶
A single decision tree is easy to read, but it overfits and it is unstable. A random forest trains many different trees and lets them vote: "It's like getting a second, third... a hundredth opinion!", says Claire in the book. It is the model the Minermont team finally chooses for triage. Its key ideas:
- Bagging: each tree is trained on a different bootstrap sample of the data.
- Feature randomness: at each node a tree only considers a random subset of $m$ features ($m \approx \sqrt{d}$).
- The forest predicts the most voted class (the mode of the trees' predictions).
- The patients a tree did not see give a free estimate of the error: the out-of-bag (OOB) error.
- The forest can estimate the importance of each feature.
In this notebook:
- Import the libraries.
- The triage data.
- Bootstrap samples, by hand.
- A forest by hand.
- The forest in scikit-learn: OOB error, number of trees and feature randomness.
- Stability.
- Evaluating a multiclass model: per-class metrics.
- Feature importance.
- The critical levels: class weights and decision thresholds.
- Comparison with the other models of the chapter.
Step 1: Import the libraries¶
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_predict
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.inspection import permutation_importance
from sklearn.metrics import (accuracy_score, recall_score, precision_score, classification_report,
ConfusionMatrixDisplay)
Step 2: The triage data¶
The emergency records of a real hospital are confidential, so we generate synthetic patients that behave like the ones in the book. The next cell contains the generator. You do not need to follow it line by line: what matters is that the urgency level comes from a hidden protocol (an early-warning score based on the vital signs, plus some clinical warning signs) and from the nurse's own judgement, which sometimes moves the level one step. The models never see that rule: they have to discover it from the examples.
| Column | Meaning |
|---|---|
age |
Age in years (1–97; about one patient in eight is a child) |
male |
1 if the patient is a man, 0 if a woman |
systolic_bp |
Systolic blood pressure (mmHg) |
heart_rate |
Heart rate (beats per minute) |
resp_rate |
Respiratory rate (breaths per minute) |
spo2 |
Oxygen saturation (%) |
temperature |
Temperature (°C) |
consciousness |
Level of consciousness: 0 alert, 1 responds to voice, 2 responds only to pain, 3 unresponsive |
pain |
Pain reported by the patient (0–10) |
chest_pain, dyspnea, trauma, vomiting |
Presenting symptoms: chest pain, difficulty breathing, injury, vomiting (1 = yes) |
level |
Target: urgency level, from 1 (red, immediate) to 5 (blue, non-urgent) |
def generate_patients(n, seed):
"""Synthetic emergency patients at Minermont: one row per patient, with the urgency level assigned at triage."""
rng = np.random.default_rng(seed)
age = np.clip(np.round(np.where(rng.random(n) < 0.12, rng.uniform(1, 17, n), rng.normal(52, 20, n))), 1, 97)
male = rng.binomial(1, 0.5, n)
s = rng.gamma(0.9, 0.55, n) # hidden severity of the episode (never shown to the model)
infection = rng.binomial(1, 0.12 + 0.1 * (s > 1), n)
trauma = rng.binomial(1, 0.2, n)
chest_pain = rng.binomial(1, np.clip(0.04 + 0.0025 * age, 0, 0.35), n)
dyspnea = rng.binomial(1, np.clip(0.06 + 0.25 * s, 0, 0.9), n)
vomiting = rng.binomial(1, 0.12, n)
shock = (rng.random(n) < 0.35) & (s > 1.2)
hr = 76 + 16 * s + 12 * infection + rng.normal(0, 10, n)
brady = (rng.random(n) < 0.08) & (s > 1.3) # a few severe patients have a very slow heart rate
hr = np.where(brady, rng.normal(42, 6, n), hr)
rr = 15 + 4.5 * s + 3 * dyspnea + rng.normal(0, 2.2, n)
spo2 = np.clip(98.2 - 2.2 * s * (0.5 + dyspnea) + rng.normal(0, 1.1, n), 70, 100)
sbp = np.clip(128 + 0.25 * (age - 50) + rng.normal(0, 16, n) - 28 * s * shock, 60, 230)
temp = 36.8 + rng.normal(0, 0.35, n) + infection * rng.normal(1.6, 0.6, n)
p_consc = np.clip(0.02 * s ** 2.2, 0, 0.9)
consc = np.where(rng.random(n) < p_consc, rng.choice([1, 2, 3], n, p=[0.55, 0.3, 0.15]), 0)
pain = np.clip(np.round(rng.normal(2.5 + 3.5 * trauma + 3 * chest_pain + 0.8 * s, 1.8, n)), 0, 10).astype(int)
hr, rr, sbp = np.round(hr).astype(int), np.round(rr).astype(int), np.round(sbp).astype(int)
temp, spo2 = np.round(temp, 1), np.round(spo2).astype(int)
# The hidden protocol behind the labels, used only to generate the data:
# an early-warning score in which a vital sign scores points when it is too low OR too high...
score = (np.select([rr <= 8, rr <= 11, rr <= 20, rr <= 24], [3, 1, 0, 2], 3)
+ np.select([spo2 <= 91, spo2 <= 93, spo2 <= 95], [3, 2, 1], 0)
+ np.select([sbp <= 90, sbp <= 100, sbp <= 110, sbp <= 219], [3, 2, 1, 0], 3)
+ np.select([hr <= 40, hr <= 50, hr <= 90, hr <= 110, hr <= 130], [3, 1, 0, 1, 2], 3)
+ np.where(consc > 0, 3, 0)
+ np.select([temp <= 35, temp <= 36, temp <= 38, temp <= 39], [3, 1, 0, 1], 2))
# ...plus clinical discriminators, from the least to the most urgent level
level = np.full(n, 5)
level = np.where((score >= 1) | (pain >= 4) | (trauma == 1), 4, level)
level = np.where((score >= 4) | (pain >= 7) | (temp >= 39) | ((chest_pain == 1) & (age < 45)), 3, level)
level = np.where((score >= 6) | ((chest_pain == 1) & (age >= 45)) | (spo2 <= 91), 2, level)
level = np.where((score >= 9) | (consc >= 2) | (sbp <= 80), 1, level)
# and the nurse's own judgement, which sometimes moves the level one step
level = np.clip(level + rng.choice([-1, 0, 1], n, p=[0.05, 0.9, 0.05]), 1, 5)
return pd.DataFrame({
'age': age.astype(int), 'male': male, 'systolic_bp': sbp, 'heart_rate': hr, 'resp_rate': rr,
'spo2': spo2, 'temperature': temp, 'consciousness': consc, 'pain': pain, 'chest_pain': chest_pain,
'dyspnea': dyspnea, 'trauma': trauma, 'vomiting': vomiting, 'level': level,
})
df = generate_patients(4000, seed=42)
feature_names = [c for c in df.columns if c != 'level']
level_names = ['1 Red', '2 Orange', '3 Yellow', '4 Green', '5 Blue']
X = df[feature_names]
y = df['level']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
print(f"Training set: {len(y_train)} patients, test set: {len(y_test)} patients\n")
print("Patients per urgency level:")
print(pd.DataFrame({'patients': y.value_counts().sort_index(),
'share': y.value_counts(normalize=True).sort_index().round(3)},).set_axis(level_names).to_string())
Training set: 3000 patients, test set: 1000 patients
Patients per urgency level:
patients share
1 Red 158 0.040
2 Orange 530 0.132
3 Yellow 780 0.195
4 Green 1671 0.418
5 Blue 861 0.215
Step 3: Bootstrap samples¶
A bootstrap sample has the same size as the training set, but it is drawn with replacement: some patients appear several times and others do not appear at all. The probability that a given patient is never chosen in $n$ draws is $(1 - 1/n)^n$, which for large $n$ is about $1/e \approx 0.368$. So each tree sees about 63% of the different patients, and the remaining 37% are its out-of-bag patients.
rng = np.random.default_rng(0)
n = len(X_train)
sample = rng.integers(0, n, n) # n draws with replacement
counts = np.bincount(sample, minlength=n)
print(f"Different patients in the sample: {np.mean(counts > 0):.1%} (theory: {1 - (1 - 1 / n) ** n:.1%})")
print(f"Patients chosen more than once: {np.mean(counts > 1):.1%}")
print(f"Out-of-bag patients: {np.mean(counts == 0):.1%}")
Different patients in the sample: 63.9% (theory: 63.2%) Patients chosen more than once: 25.8% Out-of-bag patients: 36.1%
Step 4: A forest by hand¶
With a bootstrap sample for each tree and max_features='sqrt' (at each node, the tree only looks at a random subset of $\sqrt{13} \approx 3$ features), building a forest takes a few lines. The prediction is the mode of the votes:
$$\hat{C}(x_q) = \text{mode}\{C_1(x_q), C_2(x_q), \dots, C_N(x_q)\}$$
def train_forest(X, y, n_trees, seed=0):
rng = np.random.default_rng(seed)
trees = []
for b in range(n_trees):
idx = rng.integers(0, len(X), len(X)) # bootstrap sample
tree = DecisionTreeClassifier(max_features='sqrt', random_state=b) # random features at each node
trees.append(tree.fit(X.iloc[idx], y.iloc[idx]))
return trees
def forest_predict(trees, X):
votes = np.array([tree.predict(X) for tree in trees]) # trees x patients
return np.array([np.bincount(v, minlength=6).argmax() for v in votes.T]) # most voted level
single_tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
print(f"A single tree (all the data, all the features): test accuracy {single_tree.score(X_test, y_test):.3f}")
trees = train_forest(X_train, y_train, n_trees=100)
for k in [1, 10, 100]:
print(f"Forest of {k:>3} tree(s): test accuracy {accuracy_score(y_test, forest_predict(trees[:k], X_test)):.3f}")
A single tree (all the data, all the features): test accuracy 0.730
Forest of 1 tree(s): test accuracy 0.692 Forest of 10 tree(s): test accuracy 0.848 Forest of 100 tree(s): test accuracy 0.875
One tree of the forest is worse than the single full tree (0.692 against 0.730): it has seen only part of the patients, and only a few features at each node. But combining them works: 10 trees reach 0.848 and 100 trees 0.875. Each tree makes different mistakes, and in the vote many of them cancel out.
Step 5: The forest in scikit-learn¶
RandomForestClassifier does the same, faster, and with oob_score=True it also computes the OOB accuracy: each training patient is predicted only by the trees that did not see it, so we get an estimate of the performance on new data without touching the test set.
rf = RandomForestClassifier(n_estimators=300, oob_score=True, random_state=0, n_jobs=-1)
rf.fit(X_train, y_train)
print(f"OOB accuracy (training data only): {rf.oob_score_:.3f}")
print(f"Test accuracy: {rf.score(X_test, y_test):.3f}")
OOB accuracy (training data only): 0.867 Test accuracy: 0.878
The OOB estimate (0.867) is very close to the test accuracy (0.878), and we got it without using the test set. It is the "free" validation that Martha highlights in the book.
The number of trees and the number of features per node are the key hyperparameters. The OOB accuracy lets us explore them cheaply:
n_trees_list = [25, 50, 100, 200, 300, 500]
oob_by_trees = [RandomForestClassifier(n_estimators=t, oob_score=True, random_state=0, n_jobs=-1)
.fit(X_train, y_train).oob_score_ for t in n_trees_list]
print("Number of trees:", " ".join(f"{t}: {s:.3f}" for t, s in zip(n_trees_list, oob_by_trees)))
max_features_list = [1, 2, 'sqrt', 6, 9, None]
oob_by_features = [RandomForestClassifier(n_estimators=300, max_features=m, oob_score=True, random_state=0, n_jobs=-1)
.fit(X_train, y_train).oob_score_ for m in max_features_list]
print("Features per node:", " ".join(f"{m if m else 'all 13'}: {s:.3f}" for m, s in zip(max_features_list, oob_by_features)))
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
axes[0].plot(n_trees_list, oob_by_trees, 'o-')
axes[0].set_xlabel('Number of trees')
axes[0].set_ylabel('OOB accuracy')
axes[0].set_title('More trees: better, until it stabilizes')
axes[1].plot([str(m) if m else 'all (13)' for m in max_features_list], oob_by_features, 's-')
axes[1].set_xlabel('Features considered at each node (max_features)')
axes[1].set_ylabel('OOB accuracy')
axes[1].set_title('Feature randomness')
for ax in axes:
ax.grid(True)
plt.show()
Number of trees: 25: 0.840 50: 0.857 100: 0.863 200: 0.868 300: 0.867 500: 0.869
Features per node: 1: 0.840 2: 0.860 sqrt: 0.867 6: 0.869 9: 0.864 all 13: 0.858
The OOB accuracy grows quickly up to 100–200 trees and then barely changes: more trees do not hurt, they only cost time. With a single feature per node the trees are too random (0.840). Considering all 13 features (0.858) removes the feature randomness and leaves only bagging, which is worse than the default 'sqrt' (3 features, 0.867). The best values (6 features, or 500 trees) are only a couple of thousandths better than the defaults, so we keep the defaults.
Step 6: Stability¶
In the decision tree notebook, two trees trained on two random samples of 90% of the patients gave different levels to many test patients. Let's repeat the experiment with forests:
samples = [np.random.default_rng(seed).choice(len(X_train), int(0.9 * len(X_train)), replace=False) for seed in (0, 1)]
for name, make in [('trees', lambda: DecisionTreeClassifier(random_state=0)),
('forests', lambda: RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1))]:
p1, p2 = [make().fit(X_train.iloc[s], y_train.iloc[s]).predict(X_test) for s in samples]
print(f"Two {name:<8} different level for {np.mean(p1 != p2):.1%} of the test patients")
Two trees different level for 18.1% of the test patients
Two forests different level for 0.9% of the test patients
The trees disagree on 18.1% of the test patients; the forests on only 0.9%. Averaging hundreds of votes makes the forest's decisions stable, which matters a great deal in triage: the level of a patient should not depend on which records happened to be used to train the model.
Step 7: Evaluating a multiclass model¶
Martha insists in the book that overall accuracy is not enough: we need to know how the model behaves with each level, especially the critical ones. In the multiclass confusion matrix the rows are the real classes and the columns the predicted ones. For each class $k$:
- $TP_k$: the cell $(k, k)$.
- $FP_k$: the rest of column $k$ (patients of other classes predicted as $k$).
- $FN_k$: the rest of row $k$ (patients of class $k$ predicted as something else).
- $TN_k$: all the cells outside row $k$ and column $k$.
$$\text{Precision}_k = \frac{TP_k}{TP_k + FP_k} \qquad \text{Recall}_k = \frac{TP_k}{TP_k + FN_k} \qquad F1_k = 2\,\frac{\text{Precision}_k \cdot \text{Recall}_k}{\text{Precision}_k + \text{Recall}_k}$$
Let's apply the formulas to the 3-level example table in the book:
book_matrix = np.array([[50, 5, 0], # real Level 1 (high)
[8, 200, 15], # real Level 2 (medium)
[1, 25, 500]]) # real Level 3 (low)
def per_class_metrics(cm, names):
rows = []
for k, name in enumerate(names):
tp = cm[k, k]
fp = cm[:, k].sum() - tp
fn = cm[k, :].sum() - tp
tn = cm.sum() - tp - fp - fn
precision, recall = tp / (tp + fp), tp / (tp + fn)
rows.append({'class': name, 'TP': tp, 'FP': fp, 'FN': fn, 'TN': tn, 'precision': precision,
'recall': recall, 'F1': 2 * precision * recall / (precision + recall)})
return pd.DataFrame(rows).set_index('class').round(3)
print(per_class_metrics(book_matrix, ['Level 1 (high)', 'Level 2 (medium)', 'Level 3 (low)']).to_string())
TP FP FN TN precision recall F1 class Level 1 (high) 50 9 5 740 0.847 0.909 0.877 Level 2 (medium) 200 30 23 551 0.870 0.897 0.883 Level 3 (low) 500 15 26 263 0.971 0.951 0.961
Level 1 has the highest recall (0.909) but the lowest precision (0.847): of the 59 patients classified as high urgency, 9 were not. Most errors are confusions between Levels 2 and 3 (15 and 25 patients), exactly as the book points out.
Now the same analysis for our forest on the test set:
y_pred = rf.predict(X_test)
ConfusionMatrixDisplay.from_predictions(y_test, y_pred, display_labels=level_names, cmap='Blues', xticks_rotation=30)
plt.xlabel('Predicted level')
plt.ylabel('Real level')
plt.title('Random forest on the test set')
plt.show()
print(classification_report(y_test, y_pred, target_names=level_names, digits=3))
precision recall f1-score support
1 Red 0.897 0.650 0.754 40
2 Orange 0.871 0.818 0.844 132
3 Yellow 0.846 0.815 0.830 195
4 Green 0.861 0.967 0.911 418
5 Blue 0.953 0.842 0.894 215
accuracy 0.878 1000
macro avg 0.885 0.818 0.846 1000
weighted avg 0.881 0.878 0.876 1000
The forest is the most accurate model so far (0.878), and very reliable when it says Level 1 (precision 0.897). But it still finds only 26 of the 40 critical patients (recall 0.650): 14 Level 1 patients are classified as less urgent.
Step 8: Feature importance¶
Ethan points out that the forest will not say exactly why it chose Level 2, but it can say which variables are the most decisive. There are two common measures:
- Impurity-based importance: how much each feature reduces the Gini impurity across all the splits of all the trees. It is free, but it tends to favor features with many different values.
- Permutation importance, the method described in the book: shuffle the values of one feature among the patients and measure how much the accuracy drops. If the model did not need that feature, the accuracy does not change. We compute it on the test set, repeating each shuffle 10 times.
perm = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=0, n_jobs=-1)
importance = pd.DataFrame({'impurity': rf.feature_importances_, 'permutation': perm.importances_mean},
index=feature_names).sort_values('permutation')
print(importance.sort_values('permutation', ascending=False).round(3).to_string())
importance.plot.barh(figsize=(9, 5), subplots=True, layout=(1, 2), sharey=True, legend=False,
title=['Impurity-based importance', 'Permutation importance (accuracy drop)'])
plt.tight_layout()
plt.show()
impurity permutation pain 0.213 0.251 chest_pain 0.124 0.187 systolic_bp 0.137 0.118 resp_rate 0.099 0.117 heart_rate 0.120 0.104 age 0.095 0.057 spo2 0.056 0.047 trauma 0.027 0.046 temperature 0.087 0.035 dyspnea 0.009 0.006 consciousness 0.011 0.006 male 0.013 -0.001 vomiting 0.009 -0.002
Both measures agree on the main variables: pain, chest pain, blood pressure, and the respiratory and heart rates. The differences are instructive. Temperature, with many different values, gets a much higher impurity importance (0.087) than permutation importance (0.035). Sex and vomiting, which play no role in the hidden protocol, have a permutation importance of practically zero (slightly negative, by chance). And the level of consciousness looks unimportant only because very few patients have it altered: for them it is decisive, but importance measures the effect over all the patients.
Step 9: The critical levels¶
"This is unacceptable!", exclaims Grace in the book when she sees the recall of the most urgent levels. As Martha explains, the cause is the class imbalance: the model is optimized to get the frequent levels right. The book mentions two remedies, and we try both without touching the test set.
Class weights. With class_weight='balanced' each error on a rare level counts more during training. We compare it with 5-fold cross-validation on the training set. Fully grown trees end in pure leaves, where weights hardly matter, so we also try leaves of at least 10 patients (min_samples_leaf=10):
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
options = {
'no weights': RandomForestClassifier(n_estimators=300, random_state=0, n_jobs=-1),
"class_weight='balanced'": RandomForestClassifier(n_estimators=300, class_weight='balanced', random_state=0, n_jobs=-1),
"'balanced' + min_samples_leaf=10": RandomForestClassifier(n_estimators=300, class_weight='balanced',
min_samples_leaf=10, random_state=0, n_jobs=-1),
}
rows = []
for name, model in options.items():
p = cross_val_predict(model, X_train, y_train, cv=cv)
recalls = recall_score(y_train, p, average=None)
rows.append({'option': name, 'accuracy': accuracy_score(y_train, p), 'recall Level 1': recalls[0], 'recall Level 2': recalls[1]})
print("5-fold cross-validation on the training set:")
print(pd.DataFrame(rows).set_index('option').round(3).to_string())
5-fold cross-validation on the training set:
accuracy recall Level 1 recall Level 2
option
no weights 0.863 0.602 0.804
class_weight='balanced' 0.861 0.661 0.812
'balanced' + min_samples_leaf=10 0.833 0.695 0.756
Balanced weights raise the Level 1 recall from 0.602 to 0.661 with almost no cost in accuracy. With larger leaves the weights have more effect (0.695), but the overall accuracy drops to 0.833 and Level 2 gets worse. It is the trade-off the book describes: better recall for the critical levels at the cost of some overall accuracy.
A decision threshold. The forest does not only vote: predict_proba gives the fraction of trees that vote for each level. By default the most voted level wins, but for the critical level we can be more cautious: if at least a fraction $t$ of the trees say Level 1, flag the patient as Level 1. We choose $t$ with the OOB probabilities, which, like the OOB accuracy, only use trees that did not see each patient:
def with_threshold(proba, t):
pred = rf.classes_[proba.argmax(axis=1)]
return np.where(proba[:, 0] >= t, 1, pred) # column 0 = probability of Level 1
oob_proba = rf.oob_decision_function_
rows = []
for t in [0.5, 0.4, 0.3, 0.2, 0.15, 0.1]:
p = with_threshold(oob_proba, t)
rows.append({'threshold t': t, 'OOB accuracy': accuracy_score(y_train, p),
'recall Level 1': recall_score(y_train, p, average=None)[0],
'precision Level 1': precision_score(y_train, p, average=None)[0]})
print(pd.DataFrame(rows).set_index('threshold t').round(3).to_string())
OOB accuracy recall Level 1 precision Level 1 threshold t 0.50 0.867 0.627 0.949 0.40 0.868 0.661 0.940 0.30 0.871 0.763 0.891 0.20 0.866 0.797 0.746 0.15 0.853 0.797 0.534 0.10 0.833 0.805 0.377
Lowering the threshold catches more Level 1 patients: the OOB recall goes from 0.627 with $t = 0.5$ to 0.797 with $t = 0.2$, while the accuracy barely moves. Below 0.2 the recall stops improving and the precision collapses (0.534 with 0.15): we would flag many patients who are not critical. We choose $t = 0.2$ and check it, once, on the test set:
t = 0.2
default_pred = rf.predict(X_test)
cautious_pred = with_threshold(rf.predict_proba(X_test), t)
real_1 = y_test.to_numpy() == 1
for name, p in [('most voted level', default_pred), (f'threshold t = {t}', cautious_pred)]:
print(f"{name:<18} accuracy {accuracy_score(y_test, p):.3f} | Level 1 patients found: {np.sum((p == 1) & real_1)} of {real_1.sum()} "
f"| flagged as Level 1 without being Level 1: {np.sum((p == 1) & ~real_1)}")
most voted level accuracy 0.878 | Level 1 patients found: 26 of 40 | flagged as Level 1 without being Level 1: 3 threshold t = 0.2 accuracy 0.876 | Level 1 patients found: 31 of 40 | flagged as Level 1 without being Level 1: 13
On the test set the cautious rule finds 31 of the 40 Level 1 patients instead of 26, and the accuracy only goes from 0.878 to 0.876. The price is 13 patients flagged as Level 1 who were not (3 with the default rule), an error in the safe direction. As Ethan says in the book, he would rather the system be slightly more cautious and flag high-urgency cases for the nurse to confirm.
Step 10: Comparison with the other models of the chapter¶
Finally, we put the forest next to the other options the team studied: KNN and the decision tree from the previous notebooks, and the two strategies of section 4.1 that turn a binary classifier into a multiclass one: One-vs-Rest with logistic regression (one classifier per level) and One-vs-One with SVM (one classifier per pair of levels).
ovr = make_pipeline(StandardScaler(), OneVsRestClassifier(LogisticRegression(max_iter=2000))).fit(X_train, y_train)
ovo = make_pipeline(StandardScaler(), OneVsOneClassifier(SVC())).fit(X_train, y_train)
print(f"One-vs-Rest trains {len(ovr[-1].estimators_)} binary classifiers; One-vs-One trains {len(ovo[-1].estimators_)}\n")
candidates = {
'KNN (K = 9, Manhattan)': make_pipeline(StandardScaler(), KNeighborsClassifier(n_neighbors=9, metric='manhattan')).fit(X_train, y_train).predict(X_test),
'Decision tree (depth 10)': DecisionTreeClassifier(max_depth=10, random_state=0).fit(X_train, y_train).predict(X_test),
'Logistic regression, One-vs-Rest': ovr.predict(X_test),
'SVM, One-vs-One': ovo.predict(X_test),
'Random forest': default_pred,
f'Random forest + threshold {t} for Level 1': cautious_pred,
}
rows = []
for name, p in candidates.items():
recalls = recall_score(y_test, p, average=None)
rows.append({'model': name, 'accuracy': accuracy_score(y_test, p), 'recall Level 1': recalls[0], 'recall Level 2': recalls[1]})
print(pd.DataFrame(rows).set_index('model').round(3).to_string())
One-vs-Rest trains 5 binary classifiers; One-vs-One trains 10
accuracy recall Level 1 recall Level 2 model KNN (K = 9, Manhattan) 0.739 0.575 0.750 Decision tree (depth 10) 0.827 0.550 0.773 Logistic regression, One-vs-Rest 0.625 0.700 0.735 SVM, One-vs-One 0.778 0.625 0.826 Random forest 0.878 0.650 0.818 Random forest + threshold 0.2 for Level 1 0.876 0.775 0.773
The forest is the most accurate model, and with the cautious threshold it also has the best Level 1 recall of the table. The decision tree comes second in accuracy and the SVM with One-vs-One (10 classifiers) third. Logistic regression with One-vs-Rest finds many critical patients (0.700) but gets only 62.5% of all the levels right: its linear boundaries cannot capture vital signs that are dangerous both when they are too high and too low. Naive Bayes, in its own notebook, reached 0.761.
Conclusion¶
- A random forest combines many decision trees trained on bootstrap samples and with random features at each node, and predicts the most voted class. Combining many imperfect, different trees gives a model that is more accurate and much more stable than a single tree.
- The OOB accuracy estimates the performance on new data during training, and it is a cheap way to choose the number of trees or of features per node.
- Feature importance, especially by permutation, gives clinicians an interpretable clue about which variables weigh most, and they can compare it with their own knowledge.
- For the critical levels, overall accuracy is not enough: we have to look at the per-class recall, and use class weights or a more cautious decision threshold, chosen without touching the test set.
- As in the book, the forest is the most pragmatic choice for triage: accurate, robust and reasonably interpretable, always as a support for the nurse's judgement and never as a replacement.