4.1 - One-vs-Rest, One-vs-One and softmax for emergency triage¶
In chapter 4 the Minermont team wants to suggest the urgency level of each patient arriving at the emergency department, from 1 (red, immediate) to 5 (blue, non-urgent). The tools they already have, logistic regression and SVMs, answer yes or no. Hazel explains the two usual ways of building a five-class classifier out of binary ones, One-vs-Rest (OvR) and One-vs-One (OvO), and the team raises their weak points: the imbalance of "Level 1 vs the rest", the ambiguity when no classifier is confident, and the ties in the vote. Grace concludes that they should look for an approach that considers all the classes at once. For linear models that approach is the softmax function, the generalization of the sigmoid to several classes.
In this notebook we build the three from scratch, with NumPy, and compare them on the same patients:
- Import the libraries.
- The triage data.
- The building block: binary logistic regression.
- One-vs-Rest: five classifiers.
- The ambiguity of One-vs-Rest.
- One-vs-One: ten classifiers and a vote.
- Softmax regression: one model for the five levels.
- Comparison: one patient, three answers.
Step 1: Import the libraries¶
The algorithms are written with NumPy. scikit-learn is only used to split the data and, at the end of step 7, to check our softmax regression against its LogisticRegression.
import itertools
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay
Step 2: The triage data¶
The emergency records of a real hospital are confidential, so we generate synthetic patients that behave like the ones in the book. The next cell contains the generator. You do not need to follow it line by line: what matters is that the urgency level comes from a hidden protocol (an early-warning score based on the vital signs, plus some clinical warning signs) and from the nurse's own judgement, which sometimes moves the level one step. The models never see that rule: they have to discover it from the examples.
def generate_patients(n, seed):
"""Synthetic emergency patients at Minermont: one row per patient, with the urgency level assigned at triage."""
rng = np.random.default_rng(seed)
age = np.clip(np.round(np.where(rng.random(n) < 0.12, rng.uniform(1, 17, n), rng.normal(52, 20, n))), 1, 97)
male = rng.binomial(1, 0.5, n)
s = rng.gamma(0.9, 0.55, n) # hidden severity of the episode (never shown to the model)
infection = rng.binomial(1, 0.12 + 0.1 * (s > 1), n)
trauma = rng.binomial(1, 0.2, n)
chest_pain = rng.binomial(1, np.clip(0.04 + 0.0025 * age, 0, 0.35), n)
dyspnea = rng.binomial(1, np.clip(0.06 + 0.25 * s, 0, 0.9), n)
vomiting = rng.binomial(1, 0.12, n)
shock = (rng.random(n) < 0.35) & (s > 1.2)
hr = 76 + 16 * s + 12 * infection + rng.normal(0, 10, n)
brady = (rng.random(n) < 0.08) & (s > 1.3) # a few severe patients have a very slow heart rate
hr = np.where(brady, rng.normal(42, 6, n), hr)
rr = 15 + 4.5 * s + 3 * dyspnea + rng.normal(0, 2.2, n)
spo2 = np.clip(98.2 - 2.2 * s * (0.5 + dyspnea) + rng.normal(0, 1.1, n), 70, 100)
sbp = np.clip(128 + 0.25 * (age - 50) + rng.normal(0, 16, n) - 28 * s * shock, 60, 230)
temp = 36.8 + rng.normal(0, 0.35, n) + infection * rng.normal(1.6, 0.6, n)
p_consc = np.clip(0.02 * s ** 2.2, 0, 0.9)
consc = np.where(rng.random(n) < p_consc, rng.choice([1, 2, 3], n, p=[0.55, 0.3, 0.15]), 0)
pain = np.clip(np.round(rng.normal(2.5 + 3.5 * trauma + 3 * chest_pain + 0.8 * s, 1.8, n)), 0, 10).astype(int)
hr, rr, sbp = np.round(hr).astype(int), np.round(rr).astype(int), np.round(sbp).astype(int)
temp, spo2 = np.round(temp, 1), np.round(spo2).astype(int)
# The hidden protocol behind the labels, used only to generate the data:
# an early-warning score in which a vital sign scores points when it is too low OR too high...
score = (np.select([rr <= 8, rr <= 11, rr <= 20, rr <= 24], [3, 1, 0, 2], 3)
+ np.select([spo2 <= 91, spo2 <= 93, spo2 <= 95], [3, 2, 1], 0)
+ np.select([sbp <= 90, sbp <= 100, sbp <= 110, sbp <= 219], [3, 2, 1, 0], 3)
+ np.select([hr <= 40, hr <= 50, hr <= 90, hr <= 110, hr <= 130], [3, 1, 0, 1, 2], 3)
+ np.where(consc > 0, 3, 0)
+ np.select([temp <= 35, temp <= 36, temp <= 38, temp <= 39], [3, 1, 0, 1], 2))
# ...plus clinical discriminators, from the least to the most urgent level
level = np.full(n, 5)
level = np.where((score >= 1) | (pain >= 4) | (trauma == 1), 4, level)
level = np.where((score >= 4) | (pain >= 7) | (temp >= 39) | ((chest_pain == 1) & (age < 45)), 3, level)
level = np.where((score >= 6) | ((chest_pain == 1) & (age >= 45)) | (spo2 <= 91), 2, level)
level = np.where((score >= 9) | (consc >= 2) | (sbp <= 80), 1, level)
# and the nurse's own judgement, which sometimes moves the level one step
level = np.clip(level + rng.choice([-1, 0, 1], n, p=[0.05, 0.9, 0.05]), 1, 5)
return pd.DataFrame({
'age': age.astype(int), 'male': male, 'systolic_bp': sbp, 'heart_rate': hr, 'resp_rate': rr,
'spo2': spo2, 'temperature': temp, 'consciousness': consc, 'pain': pain, 'chest_pain': chest_pain,
'dyspnea': dyspnea, 'trauma': trauma, 'vomiting': vomiting, 'level': level,
})
| Column | Meaning |
|---|---|
age |
Age in years (1–97; about one patient in eight is a child) |
male |
1 if the patient is a man, 0 if a woman |
systolic_bp |
Systolic blood pressure (mmHg) |
heart_rate |
Heart rate (beats per minute) |
resp_rate |
Respiratory rate (breaths per minute) |
spo2 |
Oxygen saturation (%) |
temperature |
Temperature (°C) |
consciousness |
Level of consciousness: 0 alert, 1 responds to voice, 2 responds only to pain, 3 unresponsive |
pain |
Pain reported by the patient (0–10) |
chest_pain, dyspnea, trauma, vomiting |
Presenting symptoms: chest pain, difficulty breathing, injury, vomiting (1 = yes) |
level |
Target: urgency level, from 1 (red, immediate) to 5 (blue, non-urgent) |
We use the same 4,000 patients and the same split as the other notebooks of this chapter. As in chapter 3, the features are standardized with the mean and the standard deviation of the training set.
df = generate_patients(4000, seed=42)
feature_names = [c for c in df.columns if c != 'level']
level_names = ['1 Red', '2 Orange', '3 Yellow', '4 Green', '5 Blue']
X = df[feature_names]
y = df['level']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
print(f"Training set: {len(y_train)} patients, test set: {len(y_test)} patients\n")
print("Patients per urgency level:")
print(pd.DataFrame({'patients': y.value_counts().sort_index(),
'share': y.value_counts(normalize=True).sort_index().round(3)},).set_axis(level_names).to_string())
Training set: 3000 patients, test set: 1000 patients
Patients per urgency level:
patients share
1 Red 158 0.040
2 Orange 530 0.132
3 Yellow 780 0.195
4 Green 1671 0.418
5 Blue 861 0.215
mean, std = X_train.mean(), X_train.std()
Z_train = ((X_train - mean) / std).to_numpy()
Z_test = ((X_test - mean) / std).to_numpy()
y_tr, y_te = y_train.to_numpy(), y_test.to_numpy()
levels = np.arange(1, 6)
print(f"Features: {Z_train.shape[1]}, levels: {len(levels)}")
Features: 13, levels: 5
Step 3: The building block: binary logistic regression¶
OvR and OvO reuse a binary classifier. We take the logistic regression of chapter 3: a linear score $z = \mathbf{w} \cdot \mathbf{x} + b$, the sigmoid $\sigma(z) = \frac{1}{1 + e^{-z}}$ to turn it into a probability, and gradient descent on the log loss. We also write a small function that measures, for each level, the share of its patients that the model assigns to the right level (the recall of each class).
def sigmoid(z):
return 1 / (1 + np.exp(-z))
def train_logistic(Z, t, learning_rate=1.0, iterations=5000):
"""Binary logistic regression by gradient descent. t is 1 for the positive class and 0 for the negative one."""
w, b = np.zeros(Z.shape[1]), 0.0
for _ in range(iterations):
error = sigmoid(Z @ w + b) - t
w -= learning_rate * Z.T @ error / len(t)
b -= learning_rate * error.mean()
return w, b
def recall_per_level(y_true, y_pred):
return pd.Series([np.mean(y_pred[y_true == k] == k) for k in levels], index=level_names)
Step 4: One-vs-Rest: five classifiers¶
As Martha sums up, with five levels OvR trains five models: Level 1 vs (2, 3, 4, 5), Level 2 vs (1, 3, 4, 5), and so on. Each one uses all the training patients, labelled 1 if they belong to its level and 0 otherwise. To predict, the five classifiers score the patient and the level whose classifier gives the highest score wins.
ovr_models = {}
for k in levels:
t = (y_tr == k).astype(float) # level k = 1, every other level = 0
ovr_models[k] = train_logistic(Z_train, t)
print(f"{level_names[k - 1]:<9} vs rest: {t.mean():5.1%} of the {len(t)} training patients are positive")
ovr_scores = np.column_stack([sigmoid(Z_test @ w + b) for w, b in ovr_models.values()])
ovr_pred = levels[ovr_scores.argmax(axis=1)] # the level whose classifier is the most confident
print(f"\nOvR accuracy on the test set: {np.mean(ovr_pred == y_te):.3f}")
print("Recall per level:")
print(recall_per_level(y_te, ovr_pred).round(3).to_string())
1 Red vs rest: 3.9% of the 3000 training patients are positive
2 Orange vs rest: 13.3% of the 3000 training patients are positive
3 Yellow vs rest: 19.5% of the 3000 training patients are positive
4 Green vs rest: 41.8% of the 3000 training patients are positive
5 Blue vs rest: 21.5% of the 3000 training patients are positive OvR accuracy on the test set: 0.625 Recall per level: 1 Red 0.700 2 Orange 0.735 3 Yellow 0.221 4 Green 0.813 5 Blue 0.544
Ethan's needle in a haystack is there: only 3.9% of the patients are positive for "Level 1 vs rest". Even so, OvR finds 0.700 of the red patients. The real victim is Level 3: only 0.221 of the yellow patients get their level. A linear score can separate "more urgent" from "less urgent", but "Level 3 vs rest" asks it to separate a band in the middle from patients on both sides of it, more urgent (levels 1 and 2) and less urgent (4 and 5). The Level 3 classifier rarely wins, and the confusion matrix in step 8 shows where its patients go: 108 of the 195 yellow patients in the test set are sent to green. Overall accuracy is 0.625.
Step 5: The ambiguity of One-vs-Rest¶
Owen's doubt: "What do we do if no classifier gives a high confidence, or if two give very similar ones?" The five classifiers are trained separately, so nothing makes their scores add up to 1 or be comparable with each other. We look at the scores of the first five test patients and count the doubtful cases.
first = pd.DataFrame(ovr_scores[:5], columns=level_names).round(3)
first['sum'] = ovr_scores[:5].sum(axis=1).round(3)
first['true level'] = y_te[:5]
print(first.to_string(), "\n")
best, second = np.sort(ovr_scores, axis=1)[:, ::-1][:, :2].T
print(f"Sum of the five scores: from {ovr_scores.sum(axis=1).min():.3f} to {ovr_scores.sum(axis=1).max():.3f}")
print(f"Patients for whom no classifier reaches 0.5: {np.sum(best < 0.5)} of {len(y_te)}")
print(f"Patients whose two highest scores differ by less than 0.1: {np.sum(best - second < 0.1)}")
print("Highest score that each classifier gives to any test patient:")
print(pd.Series(ovr_scores.max(axis=0), index=level_names).round(3).to_string())
1 Red 2 Orange 3 Yellow 4 Green 5 Blue sum true level 0 0.002 0.222 0.028 0.224 0.780 1.256 2 1 0.015 0.005 0.059 0.459 0.648 1.186 4 2 0.005 0.011 0.074 0.506 0.502 1.098 5 3 0.112 0.799 0.166 0.011 0.000 1.088 2 4 0.004 0.006 0.243 0.739 0.037 1.028 3 Sum of the five scores: from 0.352 to 2.417 Patients for whom no classifier reaches 0.5: 338 of 1000 Patients whose two highest scores differ by less than 0.1: 211 Highest score that each classifier gives to any test patient: 1 Red 1.000 2 Orange 0.996 3 Yellow 0.886 4 Green 0.801 5 Blue 0.963
The five scores of a patient add up to anything from 0.352 to 2.417: they are five separate opinions, not one probability distribution. For 338 of the 1,000 test patients no classifier reaches 0.5, and for 211 the two highest scores are less than 0.1 apart, like the third patient of the table, 0.506 for green and 0.502 for blue (the true level is blue). The middle levels are also the least confident: the highest score the Level 3 classifier gives to any patient is 0.886, and the Level 4 one, 0.801, while the red and orange classifiers reach 1.000 and 0.996. The score of the fifth patient, a yellow one, is 0.243 against 0.739 for green. That is what the book calls an inconsistent calibration between classifiers, and why Owen does not like it: in triage the answer has to be clear.
Step 6: One-vs-One: ten classifiers and a vote¶
OvO trains one classifier for each pair of levels, $\frac{K(K-1)}{2} = \frac{5 \times 4}{2} = 10$, each one only with the patients of its two levels. To predict, each classifier votes for one of its two levels and the level with the most votes wins. Noah asks what happens with a tie. The book's rule for logistic regression: choose, among the tied levels, the one with the highest average confidence in its victories.
ovo_models = {}
for i, j in itertools.combinations(levels, 2):
in_pair = (y_tr == i) | (y_tr == j) # only the patients of these two levels
ovo_models[(i, j)] = train_logistic(Z_train[in_pair], (y_tr[in_pair] == i).astype(float))
smaller = min(np.sum(y_tr == i), np.sum(y_tr == j)) / in_pair.sum()
print(f"Level {i} vs Level {j}: {in_pair.sum():>4} patients, the smaller level is {smaller:5.1%} of them")
votes = np.zeros((len(y_te), 5))
confidence = np.zeros((len(y_te), 5)) # summed confidence of each level's victories
for (i, j), (w, b) in ovo_models.items():
p = sigmoid(Z_test @ w + b) # probability of level i against level j
i_wins = p >= 0.5
votes[i_wins, i - 1] += 1
votes[~i_wins, j - 1] += 1
confidence[i_wins, i - 1] += p[i_wins]
confidence[~i_wins, j - 1] += 1 - p[~i_wins]
most_votes = votes == votes.max(axis=1, keepdims=True)
tied = most_votes.sum(axis=1) > 1
average_confidence = confidence / np.maximum(votes, 1)
ovo_pred = levels[np.where(most_votes, average_confidence, -1).argmax(axis=1)] # tie rule of the book
print(f"\nPatients with a tie for the most votes: {tied.sum()} of {len(y_te)}")
print(f"OvO accuracy on the test set: {np.mean(ovo_pred == y_te):.3f}")
print("Recall per level:")
print(recall_per_level(y_te, ovo_pred).round(3).to_string())
Level 1 vs Level 2: 516 patients, the smaller level is 22.9% of them Level 1 vs Level 3: 703 patients, the smaller level is 16.8% of them
Level 1 vs Level 4: 1371 patients, the smaller level is 8.6% of them
Level 1 vs Level 5: 764 patients, the smaller level is 15.4% of them Level 2 vs Level 3: 983 patients, the smaller level is 40.5% of them
Level 2 vs Level 4: 1651 patients, the smaller level is 24.1% of them Level 2 vs Level 5: 1044 patients, the smaller level is 38.1% of them
Level 3 vs Level 4: 1838 patients, the smaller level is 31.8% of them Level 3 vs Level 5: 1231 patients, the smaller level is 47.5% of them
Level 4 vs Level 5: 1899 patients, the smaller level is 34.0% of them Patients with a tie for the most votes: 5 of 1000 OvO accuracy on the test set: 0.660 Recall per level: 1 Red 0.725 2 Orange 0.720 3 Yellow 0.472 4 Green 0.763 5 Blue 0.581
Each pair uses fewer patients, from 516 to 1,899, and most pairs are more balanced than "Level 1 vs rest": in "Level 1 vs Level 2" the red patients are 22.9% instead of 3.9%. The most imbalanced pair, 1 against 4, still has 8.6%. Ties are rare, 5 of the 1,000 test patients, so Noah's question has a simple answer here. OvO raises the accuracy to 0.660 and, above all, the recall of Level 3 from 0.221 to 0.472: "Level 2 vs Level 3" and "Level 3 vs Level 4" are problems a linear score can solve, because each one separates two neighbouring levels.
Step 7: Softmax regression: one model for the five levels¶
Instead of five or ten separate classifiers, a single linear model gives each patient five scores at once, $z_k = \mathbf{w}_k \cdot \mathbf{x} + b_k$, one per level, and the softmax function turns them into a probability distribution:
$$\text{Softmax}(z_k) = \frac{e^{z_k}}{\sum_{j=1}^{5} e^{z_j}}$$
The five probabilities are positive and add up to 1, and the levels compete with each other: raising one score lowers the probability of all the others. It is trained, like the output layer of the neural networks in the book, by minimizing the categorical cross-entropy, the negative log of the probability given to the true level:
$$L = -\frac{1}{m} \sum_{i=1}^{m} \log \hat{y}_{i,\,\text{true level}}$$
Its gradient with respect to the scores has the same form as in the binary case: probability minus label, $\hat{\mathbf{y}} - \mathbf{y}$, where $\mathbf{y}$ is the true level written as a one-hot vector (level 3 is $[0, 0, 1, 0, 0]$). With two levels, softmax regression is exactly logistic regression.
def softmax(scores):
scores = scores - scores.max(axis=1, keepdims=True) # same probabilities, without overflowing exp
e = np.exp(scores)
return e / e.sum(axis=1, keepdims=True)
def cross_entropy(P, y):
return -np.mean(np.log(P[np.arange(len(y)), y - 1]))
Y_train = np.eye(5)[y_tr - 1] # one-hot: level 3 -> [0, 0, 1, 0, 0]
W, b = np.zeros((Z_train.shape[1], 5)), np.zeros(5) # one column of weights per level
history = []
for iteration in range(5000):
P = softmax(Z_train @ W + b)
error = P - Y_train # gradient of the cross-entropy with respect to the scores
W -= 1.0 * Z_train.T @ error / len(y_tr)
b -= 1.0 * error.mean(axis=0)
history.append(cross_entropy(P, y_tr))
softmax_probs = softmax(Z_test @ W + b)
softmax_pred = levels[softmax_probs.argmax(axis=1)]
print(f"Cross-entropy: {history[0]:.4f} at the start, {history[-1]:.5f} after {len(history)} iterations")
print(f"Sum of the five probabilities: from {softmax_probs.sum(axis=1).min():.6f} to {softmax_probs.sum(axis=1).max():.6f}")
print(f"Softmax accuracy on the test set: {np.mean(softmax_pred == y_te):.3f}")
print("Recall per level:")
print(recall_per_level(y_te, softmax_pred).round(3).to_string())
# Check: scikit-learn's LogisticRegression uses softmax for several classes (C very large = no regularization)
reference = LogisticRegression(C=1e6, max_iter=5000).fit(Z_train, y_tr)
print(f"\nscikit-learn: cross-entropy {cross_entropy(reference.predict_proba(Z_train), y_tr):.5f}, "
f"accuracy {reference.score(Z_test, y_te):.3f}, "
f"largest difference in a probability {np.abs(reference.predict_proba(Z_test) - softmax_probs).max():.4f}")
plt.figure(figsize=(8, 3.5))
plt.plot(np.arange(1, len(history) + 1), history)
plt.xscale('log'); plt.xlabel('iteration'); plt.ylabel('cross-entropy (training)'); plt.grid(True)
plt.title('Training the softmax regression'); plt.show()
Cross-entropy: 1.6094 at the start, 0.79455 after 5000 iterations Sum of the five probabilities: from 1.000000 to 1.000000 Softmax accuracy on the test set: 0.657 Recall per level: 1 Red 0.750 2 Orange 0.735 3 Yellow 0.446 4 Green 0.768 5 Blue 0.567 scikit-learn: cross-entropy 0.79452, accuracy 0.657, largest difference in a probability 0.0142
The cross-entropy starts at 1.6094, which is $\ln 5$: with all the weights at zero the model gives 0.2 to each level. It falls to 0.79455. The five probabilities of every test patient add up to exactly 1. With a single model, softmax reaches an accuracy of 0.657, close to OvO's 0.660, and a Level 3 recall of 0.446. scikit-learn's LogisticRegression, which also uses softmax when there are several classes, reaches a cross-entropy of 0.79452 and the same accuracy, 0.657. Its probabilities and ours differ by at most 0.0142, because our gradient descent stops just short of the minimum.
Step 8: Comparison: one patient, three answers¶
First the three strategies side by side, with their number of classifiers and of parameters (13 weights and a bias per linear score), and their confusion matrices on the test set:
n_params = Z_train.shape[1] + 1
summary = pd.DataFrame({
'classifiers': [len(ovr_models), len(ovo_models), 1],
'parameters': [len(ovr_models) * n_params, len(ovo_models) * n_params, W.size + b.size],
'accuracy': [np.mean(p == y_te) for p in (ovr_pred, ovo_pred, softmax_pred)],
'off by 2+ levels': [np.sum(np.abs(p - y_te) >= 2) for p in (ovr_pred, ovo_pred, softmax_pred)],
}, index=['One-vs-Rest', 'One-vs-One', 'Softmax'])
recalls = pd.DataFrame([recall_per_level(y_te, p) for p in (ovr_pred, ovo_pred, softmax_pred)], index=summary.index)
print(pd.concat([summary, recalls.add_prefix('recall ')], axis=1).round(3).to_string())
fig, axes = plt.subplots(1, 3, figsize=(16, 4.5))
for ax, (name, pred) in zip(axes, [('One-vs-Rest', ovr_pred), ('One-vs-One', ovo_pred), ('Softmax', softmax_pred)]):
ConfusionMatrixDisplay.from_predictions(y_te, pred, display_labels=level_names, ax=ax, colorbar=False, cmap='Blues')
ax.set_title(name); ax.set_xlabel('predicted level'); ax.set_ylabel('true level')
ax.tick_params(axis='x', labelrotation=45)
plt.tight_layout(); plt.show()
classifiers parameters accuracy off by 2+ levels recall 1 Red recall 2 Orange recall 3 Yellow recall 4 Green recall 5 Blue One-vs-Rest 5 70 0.625 30 0.700 0.735 0.221 0.813 0.544 One-vs-One 10 140 0.660 18 0.725 0.720 0.472 0.763 0.581 Softmax 1 70 0.657 18 0.750 0.735 0.446 0.768 0.567
OvO and softmax are practically tied, 0.660 and 0.657, and both are ahead of OvR, 0.625, which loses the yellow patients. OvO needs 10 classifiers and 140 parameters, twice as many as softmax, which does the same work with one model and 70. Most errors are one level away: OvO and softmax make 18 errors of two levels or more in 1,000 patients, and OvR 30. The diagonal is weakest between neighbouring levels, 3 and 4, and 4 and 5, where a linear score has the hardest time.
Finally, the first test patient for whom no OvR classifier reaches 0.5, and what each strategy answers for them:
patient = np.flatnonzero(best < 0.5)[0]
print(X_test.iloc[patient].to_string())
print(f"True level: {y_te[patient]}\n")
print(pd.DataFrame({'OvR score': ovr_scores[patient], 'OvO votes': votes[patient].astype(int),
'softmax probability': softmax_probs[patient]}, index=level_names).round(3).to_string())
print(f"\nPrediction: OvR {ovr_pred[patient]}, OvO {ovo_pred[patient]}, softmax {softmax_pred[patient]}")
age 52.0
male 0.0
systolic_bp 117.0
heart_rate 71.0
resp_rate 16.0
spo2 97.0
temperature 37.2
consciousness 0.0
pain 2.0
chest_pain 0.0
dyspnea 0.0
trauma 0.0
vomiting 0.0
True level: 5
OvR score OvO votes softmax probability
1 Red 0.010 0 0.002
2 Orange 0.018 1 0.006
3 Yellow 0.069 2 0.042
4 Green 0.429 3 0.477
5 Blue 0.457 4 0.474
Prediction: OvR 5, OvO 5, softmax 4
A 52-year-old woman with normal vital signs, pain 2 and no symptoms: her true level is 5. OvR gives her 0.429 for green and 0.457 for blue, neither above 0.5, and picks blue. OvO gives blue its 4 votes and also picks blue. Softmax gives 0.477 to green and 0.474 to blue and picks green, wrong this time. The three hesitate between the same two levels; the difference is how they show it. The softmax probabilities say it plainly: almost all the probability is split in two equal halves between green and blue. For a borderline patient that is the honest answer, and the useful one for a triage nurse: a case to review rather than a clear level.
Conclusion¶
- One-vs-Rest builds a $K$-class classifier out of $K$ binary ones. It is simple, but each classifier faces an imbalanced problem, their scores do not add up to 1, and a class in the middle of the others can be squeezed out.
- One-vs-One trains $K(K-1)/2$ classifiers on smaller, more balanced pairs and lets them vote. It needs twice as many parameters here, and a rule for ties.
- Softmax regression considers all the classes at once, as Grace wanted: one model, probabilities that add up to 1, and the cross-entropy as its cost. It is the multiclass version of logistic regression, and it is what the output layer of a neural network does.
- The three are linear, and the triage protocol is not: a vital sign scores points when it is too low or too high. That is why the team went on to algorithms that are multiclass by nature and not linear, such as decision trees and random forests, which the next notebooks of this chapter build.