The Confusion Matrix, and Choosing a Metric That Matches the Question

Context

A confusion matrix counts a classifier's yes-or-no answers on a set of listings in four cells, by what each listing really did and what the model said it would do. Every score this lesson computes comes from those four counts, and the question the model is used for decides which one to read.

A listings site that tells a seller the listing will probably be a fast sale (sold within 30 days of going up) reads one ratio of that matrix, and the same site filling its paid featured slots on its home page reads another. A team weighing a new classifier against the one in use sets both matrices side by side on the same held-out listings.

Without it, a model can miss nearly every listing that matters and still score well. On this market a rule that calls no listing a fast sale scores 0.79 accuracy (the share of listings labeled correctly), because 79 percent of listings do not sell within 30 days, and Lesson 3.4.1's logistic regression scores 0.788 while finding one of the 126 fast sales among the test listings. A matrix built with its two arguments swapped trades one kind of mistake for the other without an error, and the share of fast-sale calls that were right, computed for a rule that made none, comes back as 0.0 under a warning. Lesson 3.4.1 turned the model's probability into a yes or a no, and this lesson counts what those answers got right and wrong.

Goal

• Compute accuracy with accuracy_score and .score(), set it beside a DummyClassifier majority baseline, and explain why an accuracy near the majority label's share says little about the rarer label.
• Build a confusion matrix with confusion_matrix, name each cell as a true or false positive or negative, and identify the layouts that swapped arguments and labels= produce.
• Draw a ConfusionMatrixDisplay, save it, and prove what it holds by printing its counts and the text in its cells.
• Compute precision and recall by hand from the matrix and with precision_score and recall_score, and report a model that predicts no positive with zero_division.
• Compute F1 and read classification_report, and explain what the harmonic mean hides about the two kinds of mistake.
• Choose the metric a question names before any model is scored, for a seller alert and for a paid featured slot.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the four cells drawn, or every ratio built from them defined on one page, before you start.

• YouTube - "Machine Learning Fundamentals: The Confusion Matrix" by StatQuest with Josh Starmer. The four cells of a confusion matrix named, the matrices of several methods compared to choose between them, and a matrix for three classes. It puts the model's answers in the rows and the truth in the columns, yes before no, so its top-left cell counts the cases rightly called yes, where confusion_matrix by default counts the cases rightly called no.
• Google Machine Learning Crash Course - "Classification: Accuracy, recall, precision, and related metrics". Accuracy, recall, precision and the false positive rate defined from the four outcomes, why accuracy misleads when one class is rare, what a NaN in a metric means, and how the cost of each kind of mistake picks the metric.

Table of Contents

• Accuracy and the Majority Baseline
• The Confusion Matrix
• Precision and Recall
• F1, and What It Averages Away
• The Metric That Matches the Question

Theory

Nothing to install: scikit-learn has been in (bootcamp) since Lesson 3.1.1, and matplotlib since Module 2. Every Theory block below goes into one script, confusion.py, saved in ~/dev/bootcamp beside listings.csv and run with python3 confusion.py, each block added under the one before it.

Accuracy and the Majority Baseline

What. Accuracy is the share of rows whose predicted label (the class a row is given, True or False here) matches the true one, computed by accuracy_score(y_true, y_pred) from sklearn.metrics and returned by a fitted classifier's .score(X, y), unless the classifier is one of Lesson 3.6.3's hyperparameter searches told to score by another metric. The majority baseline is the rule that predicts the most common label for every row, and DummyClassifier, from sklearn.dummy, is that rule as an estimator, Lesson 3.1.4's DummyRegressor for labels: it ignores the features and predicts from the training labels alone. Why. Where one label is far more common than the other, a rule that never predicts the rare one is right exactly as often as the common one occurs. An accuracy says something about a model only beside that share. When. Every time a classifier is scored, with the majority baseline's accuracy printed beside it.

The first block rebuilds Lesson 3.4.1's model, the section's pipeline fitted on the split stratified on the label:

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
numeric_steps = make_pipeline(
    SimpleImputer(strategy="median", add_indicator=True), StandardScaler()
)
prep = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
model = make_pipeline(prep, LogisticRegression())
model.fit(X_train, y_train)
print(y_test.value_counts().to_dict())  # {False: 474, True: 126}

Of the 600 test listings, 474 did not sell within 30 days and 126 did. predict turns each test listing's probability into a label at Lesson 3.4.1's 0.5:

from sklearn.metrics import accuracy_score

pred = model.predict(X_test)
accuracy = accuracy_score(y_test, pred)
matches = (pred == y_test).mean()
score = model.score(X_test, y_test)
print(pred[:5])            # [False False False False False]
print(pred.sum())          # 3
print(round(accuracy, 3))  # 0.788
print(round(matches, 3))   # 0.788
print(round(score, 3))     # 0.788

The model called 3 of the 600 test listings fast sales, the Trues that sum counts. Its accuracy is 0.788, the same from accuracy_score, from the share of matches computed by hand, and from .score, which returns accuracy for this classifier where it returned R² for Lesson 3.1.4's regressor. The baseline, fitted on the same rows:

from sklearn.dummy import DummyClassifier

baseline = DummyClassifier().fit(X_train, y_train)
base_pred = baseline.predict(X_test)
base_score = baseline.score(X_test, y_test)
print(baseline.get_params()["strategy"])  # prior
print(baseline.class_prior_.round(3))     # [0.791 0.209]
print(base_pred.sum())                    # 0
print(round(base_score, 3))               # 0.79

DummyClassifier learned each label's share of the training rows, kept in class_prior_, and its default strategy, "prior", predicts the more common label, False, for every listing. It was fitted on the raw training table, text and gaps included, and nothing refused it, because it never reads a feature value. It called no listing a fast sale and scored 0.79, the 474 unsold test listings out of 600, while the model, which read ten columns, scored 0.788.

Both numbers sit next to 0.79 for one reason: on rows that are 79 percent unsold, calling nearly every listing unsold is right about 79 percent of the time, whatever the model learned. An accuracy close to the majority label's share says only that the model's right calls of the rarer label about equal its wrong ones, however many calls it made, and the next subtitle counts both.

DON'T:

• DON'T report a classifier's accuracy without the majority baseline's beside it. The model's 0.788 sits below the 0.79 that DummyClassifier reached by calling no listing a fast sale.
• DON'T read 0.788 as a model that is right about most fast sales. It called 3 of the 600 test listings fast sales, where 126 of them were.
• DON'T read a classifier's .score() as R². For the model it returns accuracy, 0.788.

The Confusion Matrix

What. A confusion matrix counts a classifier's predictions by true label and predicted label. confusion_matrix(y_true, y_pred), from sklearn.metrics, returns a square table with one row and one column for each label found in either argument, sorted, False then True, the order of Lesson 3.4.1's classes_: the rows are the true labels and the columns the predicted ones. With a fast sale as the positive class (the label the question is about), its four cells are the true negatives (unsold listings called unsold), the false positives (unsold listings called fast sales), the false negatives (fast sales called unsold) and the true positives (fast sales called fast sales). Why. Accuracy adds the two correct cells and divides by the total, so it cannot say which of the two mistakes a model makes, and the two cost different things. The matrix keeps all four counts, and every other metric in this lesson is computed from them. When. Every time a classifier is scored, before any single number is reported, and first of all when a score looks too good or too poor to be true.

from sklearn.metrics import confusion_matrix

matrix = confusion_matrix(y_test, pred)
print(matrix)
tn, fp, fn, tp = matrix.ravel()
print(tn, fp, fn, tp)  # 472 2 125 1
print(confusion_matrix(y_test, base_pred))

[[472   2]
 [125   1]]
[[474   0]
 [126   0]]

The first row is the 474 test listings that did not sell within 30 days: 472 called unsold, the true negatives, and 2 called fast sales, the false positives. The second row is the 126 that did: 125 called unsold, the false negatives, and 1 called a fast sale, the true positive. Down the columns, 597 listings were called unsold and 3 fast sales. Lesson 2.2.1's ravel lays the four cells out row by row, which is why the names unpack as tn, fp, fn, tp. Accuracy is the diagonal over the total, (472 + 1) / 600, and the baseline's second column is two zeros, because it called nothing a fast sale.

Nothing in the two arguments says which one is the truth:

print(confusion_matrix(pred, y_test))
print(confusion_matrix(y_test, pred, labels=[True, False]))

[[472 125]
 [  2   1]]
[[  1 125]
 [  2 472]]

Handed the prediction first, confusion_matrix reads it as the truth and returns the matrix turned about its diagonal: the 2 and the 125 trade places, so a reader who takes the rows as the truth finds 125 false positives where the model made 2. Both arguments are columns of True and False of the same length, so nothing refused the swap. labels= sets the order of the rows and the columns, and [True, False] puts the fast sales first, the layout many textbooks draw, with the true positive top left and the true negatives bottom right.

ConfusionMatrixDisplay, from sklearn.metrics, draws a matrix the way Lesson 2.7.2's heatmap drew a correlation matrix, a grid of colored cells with each count written in its cell. Its from_predictions computes the matrix in confusion_matrix's order and draws it on the Axes passed as ax=. The figure is saved, then checked by printing what it holds:

import matplotlib.pyplot as plt
from sklearn.metrics import ConfusionMatrixDisplay

fig, ax = plt.subplots(figsize=(5, 4))
shown = ConfusionMatrixDisplay.from_predictions(y_test, pred, ax=ax)
fig.savefig("confusion.png", dpi=150, bbox_inches="tight")
cells = [t.get_text() for t in shown.text_.ravel()]
print(shown.confusion_matrix.tolist())  # [[472, 2], [125, 1]]
print(cells)                            # ['472', '2', '125', '1']
print(ax.get_ylabel())                  # True label
print(ax.get_xlabel())                  # Predicted label
print(len(fig.axes))                    # 2
plt.close(fig)

The display keeps what it drew: confusion_matrix holds the counts, and text_ holds the text written in each cell, a 2 by 2 array that ravel lays out row by row like the matrix. The truth runs down the side and the prediction along the bottom, under the labels True label and Predicted label, and the color bar beside the grid is an Axes of its own, as in 2.7.2, which is why the figure holds two.

DON'T:

• DON'T hand confusion_matrix the prediction first. confusion_matrix(pred, y_test) turned the matrix about its diagonal without an error, and the 125 then sits where the false positives belong.
• DON'T unpack tn, fp, fn, tp from a matrix built with labels=[True, False]. Its cells run 1, 125, 2, 472, so tn receives the true positive.
• DON'T check a saved display by looking at the PNG alone. Print its confusion_matrix and its cells' text, which here read 472, 2, 125 and 1, in the order of the matrix confusion_matrix returned.

Precision and Recall

What. Precision (the share of the listings called fast sales that sold within 30 days) is the true positives over the predicted-positive column, tp / (tp + fp). Recall (the share of the listings that sold within 30 days that the model called fast sales) is the true positives over the positive row, tp / (tp + fn). precision_score(y_true, y_pred) and recall_score(y_true, y_pred), from sklearn.metrics, compute them, both counting True as the positive class through their default pos_label=1, which True equals. Why. The two mistakes sit in different cells, and each metric is lowered by one of them: every false positive lowers precision, and every false negative lowers recall. A question that fears one mistake more than the other is answered by the metric that mistake lowers. When. Whenever a question is about one label, the fast sales here, with both printed unless the question names one of them.

from sklearn.metrics import precision_score, recall_score

precision = precision_score(y_test, pred)
recall = recall_score(y_test, pred)
swapped = precision_score(pred, y_test)
print(round(precision, 3))       # 0.333
print(round(tp / (tp + fp), 3))  # 0.333
print(round(recall, 3))          # 0.008
print(round(tp / (tp + fn), 3))  # 0.008
print(round(swapped, 3))         # 0.008

Of the 3 listings the model called fast sales, 1 sold within 30 days, a precision of 1 / 3. Of the 126 that sold within 30 days, it called 1, a recall of 1 / 126. The two share a numerator and not a denominator: precision reads the matrix down its second column, and recall across its second row. Swap the arguments and precision_score reads the prediction as the truth, so the column it divides by is the old second row, and it returns the recall, 0.008, without an error.

The baseline called no listing a fast sale, so its predicted-positive column holds no listing, and 0 out of 0 has no value:

base_precision = precision_score(y_test, base_pred)
print(base_precision)

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/sklearn/metrics/_classification.py:1879: UndefinedMetricWarning: Precision is ill-defined and being set to 0.0 due to no predicted samples. Use `zero_division` parameter to control this behavior.
  _warn_prf(average, modifier, f"{metric.capitalize()} is", result.shape[0])
0.0

scikit-learn warns from its own _classification.py, names the cause, no predicted samples, and returns 0.0. The value comes from zero_division, the setting that decides what a metric returns when its denominator is 0, at its default, "warn", which returns 0.0 and warns. A value set by hand replaces the 0.0 and stops the warning:

for setting in [0.0, 1.0, np.nan]:
    print(precision_score(y_test, base_pred, zero_division=setting))
print(recall_score(y_test, base_pred))  # 0.0

0.0
1.0
nan

zero_division=1.0 hands a rule that called nothing a fast sale a perfect precision, and np.nan keeps the missing value visible as nan. recall_score, f1_score and classification_report take the same setting. The baseline's recall needs none: its denominator is the 126 fast sales, and none of them called is a measured 0.0, with no warning.

DON'T:

• DON'T swap the arguments of precision_score. precision_score(pred, y_test) returned the recall, 0.008, without an error.
• DON'T pass zero_division=1.0 to quiet the warning. The rule that called no listing a fast sale then scores a precision of 1.0.
• DON'T read the warned 0.0 as a measurement. No listing was called a fast sale, so there was nothing to measure, and zero_division=np.nan reports that as nan.

F1, and What It Averages Away

What. F1 is the harmonic mean (one over the average of one over each number, a mean the smaller number pulls toward itself) of precision and recall, f1_score(y_true, y_pred). classification_report(y_true, y_pred) returns a text table, ready to print, of precision, recall and F1 for every label, each label's support (the number of rows that truly hold it), the accuracy, and two averages over the labels. Why. One number per model is easier to compare than two, and the harmonic mean stays low when either rate is low, where a plain average does not. What it gives up is which of the two rates is low, which is exactly which mistake the model makes. When. When a question weighs a missed positive and a wrong one alike, and as a summary printed beside precision and recall, never in their place.

from sklearn.metrics import classification_report, f1_score

f1 = f1_score(y_test, pred)
harmonic = 2 / (1 / precision + 1 / recall)
from_cells = 2 * tp / (2 * tp + fp + fn)
plain = (precision + recall) / 2
print(round(f1, 3))          # 0.016
print(round(harmonic, 3))    # 0.016
print(round(from_cells, 3))  # 0.016
print(round(plain, 3))       # 0.171
print(classification_report(y_test, pred, digits=3))

              precision    recall  f1-score   support

       False      0.791     0.996     0.881       474
        True      0.333     0.008     0.016       126

    accuracy                          0.788       600
   macro avg      0.562     0.502     0.448       600
weighted avg      0.695     0.788     0.700       600

The reciprocals of a precision of 1 / 3 and a recall of 1 / 126 are 3 and 126, so the harmonic mean is 2 / 129, 0.016, where the plain average sits at 0.171. Written in counts, F1 is twice the true positives over twice the true positives plus both mistakes, and the 472 true negatives appear nowhere in it.

In the report, the True row holds this lesson's three numbers for the fast sales. The False row computes the same three with an unsold listing as the positive class: 472 of the 597 listings called unsold were unsold, 0.791, and 472 of the 474 unsold listings were called so, 0.996. support counts the test listings that truly hold each label. macro avg is the plain mean of the two rows, and weighted avg weights each row by its support, so the 474 unsold listings carry it, a weighted F1 of 0.700 for a model that found 1 fast sale in 126. digits=3 gives three decimals where the default gives two.

What the single number averages away shows beside a rule that makes the opposite mistake. DummyClassifier(strategy="constant", constant=True) predicts True for every row:

always = DummyClassifier(strategy="constant", constant=True)
always_pred = always.fit(X_train, y_train).predict(X_test)
for name, guess in [("model", pred), ("always sold", always_pred)]:
    a = accuracy_score(y_test, guess)
    p = precision_score(y_test, guess)
    r = recall_score(y_test, guess)
    f = f1_score(y_test, guess)
    print(name, round(a, 3), round(p, 3), round(r, 3), round(f, 3))

model 0.788 0.333 0.008 0.016
always sold 0.21 0.21 1.0 0.347

The always-sold rule calls all 600 test listings fast sales, so it is right about the 126 that were, a precision and an accuracy of 0.21, and misses none of them, a recall of 1.0. Its F1, 0.347, is more than twenty times the model's. F1 charges a wrong call and a missed fast sale alike, so 474 wrong calls and no miss outscore 2 wrong calls and 125 misses, and which mistake each one made, the thing the next subtitle's questions turn on, is what the one number averaged away.

DON'T:

• DON'T choose F1 by habit. It weighs a wrong call and a missed fast sale alike, and it scored the rule that calls every listing a fast sale at 0.347 against the model's 0.016.
• DON'T quote the report's weighted avg as the model's score on fast sales. Its F1 of 0.700 is carried by the 474 unsold listings, and the fast sales' own F1 is 0.016.

The Metric That Matches the Question

What. Lesson 3.1.4's rule, that the question decides the metric, holds for a yes or no, and the question decides it by naming the mistake it fears more. The seller alert tells a seller the listing will probably be a fast sale, and a fast sale the model misses costs that seller the alert, so the alert's metric is recall. The site's home page also has paid featured slots, filled with listings the model calls fast sales, and a pick that does not sell within 30 days costs a slot, so the slots' metric is precision. Why. The two mistakes cost different people different things, a seller's missed alert or a paid slot spent on a listing that sits unsold, and nothing in the matrix says which cost is larger. Only the question can, and it has to say so before the scores can argue for one metric over another. When. Before the first model is scored on validation rows or test rows, written down beside the question it serves.

In code the questions come first, each with its metric, and the scoring loop reads only what they name, with zero_division=np.nan so a rule that picked nothing scores nothing:

# each question names its metric before any model is scored
questions = {
    "seller alert": recall_score,      # lowered by missed fast sales
    "featured slot": precision_score,  # lowered by wrong picks
}
candidates = {
    "never sold": base_pred,
    "always sold": always_pred,
    "model": pred,
}
for question, metric in questions.items():
    scores = {}
    for name, guess in candidates.items():
        value = metric(y_test, guess, zero_division=np.nan)
        scores[name] = round(value, 3)
    print(question, scores)
print(confusion_matrix(y_test, always_pred))

seller alert {'never sold': 0.0, 'always sold': 1.0, 'model': 0.008}
featured slot {'never sold': nan, 'always sold': 0.21, 'model': 0.333}
[[  0 474]
 [  0 126]]

The alert's metric ranks the always-sold rule first, 1.0, and its matrix shows the price: 474 wasted alerts, every unsold listing's seller told to expect a fast sale. Recall is lowered by missed fast sales alone, so the alert's question is not finished until it says how many wasted alerts it will accept. The slots' question is not finished either until it says how many slots there are, because the model made only 3 picks, one of them a fast sale, a precision of 0.333, and the never-sold rule made none, so it has no precision at all, nan.

Every count in this lesson came from predict, which calls a listing a fast sale when its probability of True is above 0.5, Lesson 3.4.1's rule:

proba = model.predict_proba(X_test)[:, 1]
print((proba > 0.5).sum())    # 3
print(round(proba.max(), 3))  # 0.568

Three test listings are above 0.5, the highest at 0.568, so the model made 3 calls. Where that threshold sits sets how many listings are called fast sales, and with them every cell of the matrix and both questions' metrics. This lesson scored the model at 0.5, as predict comes, and Lesson 3.4.3 moves the threshold and chooses where it sits on validation rows, by the metric the question named.

DON'T:

• DON'T choose the metric after the scores are printed. Accuracy ranks the never-sold rule first, recall the always-sold rule, and precision the model, so a metric chosen afterwards can crown any of the three.
• DON'T read the always-sold rule's recall of 1.0 as an answer to the alert's question. It reaches every fast sale by sending 474 wasted alerts.
• DON'T judge the featured slots on the model's 0.333. It is one fast sale among three picks, at a threshold no one chose for the slots.

Lab Exercises

Lab Exercise 1: Accuracy by Neighborhood

Work in hood_accuracy.py in ~/dev/bootcamp. Rebuild the Theory's split, model and baseline, then build one frame of the test listings with each one's neighborhood, its true label, the model's prediction and the baseline's, and a column for each of the two predictions saying whether it was right.

Print, per neighborhood, the number of test listings, the share that sold within 30 days, how many the model called fast sales, and the model's and the baseline's accuracy, the shares to three decimals. Say in a comment what sets each neighborhood's baseline accuracy, and why the model's lands beside it.

Then take Downtown's test listings alone. Print their confusion matrix, unpack its four cells, and check that the diagonal over the number of listings gives the model's Downtown accuracy. Save the matrix as downtown_confusion.png with ConfusionMatrixDisplay, with a title, and print the title, the counts the display holds and the text in its cells.

Lab Exercise 2: The Price, Kept or Dropped

A teammate proposes dropping price from the model. Keeping a column or dropping it is a choice, so it is made on validation rows. Work in no_price.py: split as the Theory did, then carve the validation rows out of the training rows as Lesson 3.1.2's Assignment does, a quarter of them, with stratify=y_train.

Fit two pipelines on the remaining fit rows, the Theory's and the same pipeline with price left out of its numeric group. For each, print on the validation rows how many listings it calls fast sales, its precision, recall and F1 to three decimals, and its confusion matrix, computing precision so that a pipeline that calls nothing a fast sale reports nan. Print the classification report of the pipeline without the price, with three digits, reporting nan the same way.

Then write down which metric each team's question reads, before printing anything more, and print what the two pipelines score on it. End with a comment on what the validation rows can and cannot say about dropping the price.

import matplotlib.pyplot as plt
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import ConfusionMatrixDisplay, confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
numeric_steps = make_pipeline(
    SimpleImputer(strategy="median", add_indicator=True), StandardScaler()
)
prep = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
model = make_pipeline(prep, LogisticRegression()).fit(X_train, y_train)
baseline = DummyClassifier().fit(X_train, y_train)

# one row per test listing: its neighborhood, the truth, both guesses
check = X_test[["neighborhood"]].assign(
    sold=y_test,
    called=model.predict(X_test),
    base=baseline.predict(X_test),
)
check["model_right"] = check["called"] == check["sold"]
check["base_right"] = check["base"] == check["sold"]
table = check.groupby("neighborhood").agg(
    listings=("sold", "size"),
    sold_share=("sold", "mean"),
    called_fast=("called", "sum"),
    model_acc=("model_right", "mean"),
    base_acc=("base_right", "mean"),
)
print(table.round(3))
# Each baseline accuracy is the share of the neighborhood's listings that
# did not sell within 30 days, from 0.752 in Riverside to 0.868 in
# Southgate. The model calls at most one listing per neighborhood a fast
# sale, so its accuracy lands within one listing of the baseline's.

              listings  sold_share  called_fast  model_acc  base_acc
neighborhood                                                        
Downtown           121       0.240            1      0.752     0.760
Eastgate            75       0.173            1      0.813     0.827
Hillcrest           59       0.169            0      0.831     0.831
Lakeview            66       0.227            0      0.773     0.773
Northend            86       0.233            0      0.767     0.767
Riverside          117       0.248            1      0.761     0.752
Southgate           76       0.132            0      0.868     0.868

down = check[check["neighborhood"] == "Downtown"]
down_matrix = confusion_matrix(down["sold"], down["called"])
print(down_matrix)
tn, fp, fn, tp = down_matrix.ravel()
print(round((tn + tp) / len(down), 3))  # 0.752, as in the table

fig, ax = plt.subplots(figsize=(5, 4))
shown = ConfusionMatrixDisplay.from_predictions(
    down["sold"], down["called"], ax=ax
)
ax.set_title("Downtown, test listings")
fig.savefig("downtown_confusion.png", dpi=150, bbox_inches="tight")
print(ax.get_title())
print(shown.confusion_matrix.tolist())
print([t.get_text() for t in shown.text_.ravel()])
plt.close(fig)

[[91  1]
 [29  0]]
Downtown, test listings
[[91, 1], [29, 0]]
['91', '1', '29', '0']

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    f1_score,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
# a quarter of the training rows, 20 percent of the table, to choose on
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)


def make_model(num_cols):
    numeric_steps = make_pipeline(
        SimpleImputer(strategy="median", add_indicator=True), StandardScaler()
    )
    prep = ColumnTransformer([
        ("num", numeric_steps, num_cols),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])
    return make_pipeline(prep, LogisticRegression())


no_price = [col for col in numeric if col != "price"]
pipes = {"with price": make_model(numeric), "without price": make_model(no_price)}
val_pred = {}
for name, pipe in pipes.items():
    guess = pipe.fit(X_fit, y_fit).predict(X_val)
    val_pred[name] = guess
    p = precision_score(y_val, guess, zero_division=np.nan)
    r = recall_score(y_val, guess)
    f = f1_score(y_val, guess)
    print(name, guess.sum(), round(p, 3), round(r, 3), round(f, 3))
    print(confusion_matrix(y_val, guess))

with price 1 0.0 0.0 0.0
[[474   1]
 [125   0]]
without price 0 nan 0.0 0.0
[[475   0]
 [125   0]]

report = classification_report(
    y_val, val_pred["without price"], digits=3, zero_division=np.nan
)
print(report)

              precision    recall  f1-score   support

       False      0.792     1.000     0.884       475
        True        nan     0.000     0.000       125

    accuracy                          0.792       600
   macro avg      0.792     0.500     0.442       600
weighted avg      0.792     0.792     0.700       600

# written before the scores below are printed
questions = {"seller alert": recall_score, "featured slot": precision_score}
for question, metric in questions.items():
    for name, guess in val_pred.items():
        value = metric(y_val, guess, zero_division=np.nan)
        print(question, name, round(value, 3))
# Neither pipeline found one of the 125 fast sales on these rows, so
# the alert's recall is 0.0 for both and cannot rank them. The slots'
# precision sets one wrong pick, 0.0, against no pick at all, nan, and
# cannot rank them either. At a threshold of 0.5, these rows cannot say
# whether the price should stay.

seller alert with price 0.0
seller alert without price 0.0
featured slot with price 0.0
featured slot without price nan

Theory Assessment

Q1: Two Accuracies and Four Cells

The model scored 0.788 on the test rows and DummyClassifier() scored 0.79. Using both confusion matrices, explain how a rule that reads no feature comes out ahead of a model that reads ten columns, and what decides the baseline's number. Then name each cell of the model's matrix, [[472 2] [125 1]], as a true or false positive or negative, say what a reader who takes the rows as the truth would conclude from confusion_matrix(pred, y_test), and say why nothing raised.

Hint: Count the listings each rule labels correctly from its diagonal. For the swap, ask what confusion_matrix can know about which of its two arguments is the truth.

Q2: Two Denominators and an Empty Column

From the matrix [[472 2] [125 1]], explain why precision is 0.333 and recall 0.008, naming the count each one divides by, and why precision_score(pred, y_test) returned 0.008. Then explain why the baseline's precision drew an UndefinedMetricWarning and a 0.0 while its recall came back 0.0 with no warning, and what zero_division=1.0 and zero_division=np.nan each report for it.

Hint: Find the column and the row each metric reads. For the baseline, write each metric as a fraction of its counts and look at the denominator.

Q3: One Number, Two Teams

The model's F1 is 0.016 while the plain average of its precision and recall is 0.171, and the rule that calls every listing a fast sale scores an F1 of 0.347. Explain all three numbers and what F1 hides. Then say which metric the seller alert and the featured slots each name and why, why the always-sold rule's recall of 1.0 does not answer the alert's question, and why the model's precision of 0.333 does not answer the slots'.

Hint: Write the harmonic mean with reciprocals and find the reciprocal of 1 / 126. For the teams, ask which cell each feared mistake lands in, and what the always-sold rule's matrix holds in the cell recall never reads.

Q4: A Matrix With the Fast Sales First

A teammate wants the matrix in a textbook's layout, fast sales first, and adds this to the end of confusion.py:

layout = confusion_matrix(y_test, pred, labels=[True, False])
tn, fp, fn, tp = layout.ravel()
precision = tp / (tp + fp)
recall = tp / (tp + fn)
accuracy = (tn + tp) / layout.sum()
print(round(precision, 3), round(recall, 3), round(accuracy, 3))

The teammate then tells the slot team that 79.1 percent of the listings the model calls fast sales sell within 30 days, that the model finds 99.6 percent of the fast sales, and that it labels 78.8 percent of listings correctly. Read the script without running it, say what it prints, and say which of the three claims hold.

Hint: Write out the matrix labels=[True, False] returns, then hand its four cells to the four names in ravel's order. Compare what the script computes with the rows of the classification report.

0.791 0.996 0.788

tn, fp, fn, tp = confusion_matrix(y_test, pred).ravel()
# 0.333 0.008
print(round(tp / (tp + fp), 3), round(tp / (tp + fn), 3))

Assignment

Assignment: Shares Each Team Can Read

The alert team and the slot team each want the test rows' confusion matrix in their weekly report, and neither wants counts. Each wants the shares that answer its own question, drawn as a figure, with the cell that answers the question checked against the metric the question names.

confusion_matrix and ConfusionMatrixDisplay.from_predictions both take normalize=, which divides every count by a total before returning or drawing it: "true" divides each count by the total of its row, "pred" by the total of its column, and "all" by the number of rows. The display writes each share in its cell in a short format of its own unless values_format=, the display's counterpart of 2.7.2's fmt=, gives a format string such as ".3f".

Write team_shares.py on the Theory's split, model and baseline, in this order:

• Before any matrix is computed, write in a comment which normalize= each team needs, and why.
• For each team, print the model's normalized matrix rounded to three decimals, and show that the cell answering the team's question equals the metric its question names, from recall_score or precision_score.
• Save one figure, team_shares.png, 10 by 4 inches, holding the two teams' normalized matrices side by side, each titled with its team, and print each panel's title and the text in its cells. Every cell must read to three decimals.
• Normalize the baseline's matrix the slot team's way, and print it beside the baseline's precision with zero_division=np.nan.

End with comments saying which cell answers each team, why the two normalizations divide by different counts, and what the slot team's matrix says about the baseline that precision_score does not.

Hints:

• A share of a row divides by the number of listings that truly hold that row's label, and a share of a column by the number the model gave that column's label. Ask which of precision and recall divides by each.
• Print each panel's cell text before you trust the figure, the way the Theory printed text_. A short format can show a share as a whole number.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.dummy import DummyClassifier
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    ConfusionMatrixDisplay,
    confusion_matrix,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
numeric_steps = make_pipeline(
    SimpleImputer(strategy="median", add_indicator=True), StandardScaler()
)
prep = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
model = make_pipeline(prep, LogisticRegression()).fit(X_train, y_train)
pred = model.predict(X_test)
base_pred = DummyClassifier().fit(X_train, y_train).predict(X_test)

# The alert team asks what share of the fast sales the model called,
# a share of a row: normalize="true", recall's denominator.
# The slot team asks what share of the model's calls sold fast,
# a share of a column: normalize="pred", precision's denominator.
teams = {
    "seller alert": ("true", recall_score),
    "featured slot": ("pred", precision_score),
}
for team, (norm, metric) in teams.items():
    shares = confusion_matrix(y_test, pred, normalize=norm)
    print(team, norm)
    print(shares.round(3))
    print(round(shares[1, 1], 3), round(metric(y_test, pred), 3))

seller alert true
[[0.996 0.004]
 [0.992 0.008]]
0.008 0.008
featured slot pred
[[0.791 0.667]
 [0.209 0.333]]
0.333 0.333

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, (team, (norm, metric)) in zip(axes, teams.items()):
    shown = ConfusionMatrixDisplay.from_predictions(
        y_test, pred, normalize=norm, ax=ax
    )
    ax.set_title(team)
    print(ax.get_title(), [t.get_text() for t in shown.text_.ravel()])
plt.close(fig)

seller alert ['1', '0.0042', '0.99', '0.0079']
featured slot ['0.79', '0.67', '0.21', '0.33']

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
for ax, (team, (norm, metric)) in zip(axes, teams.items()):
    shown = ConfusionMatrixDisplay.from_predictions(
        y_test, pred, normalize=norm, values_format=".3f", ax=ax
    )
    ax.set_title(team)
    print(ax.get_title(), [t.get_text() for t in shown.text_.ravel()])
fig.savefig("team_shares.png", dpi=150, bbox_inches="tight")
plt.close(fig)

seller alert ['0.996', '0.004', '0.992', '0.008']
featured slot ['0.791', '0.667', '0.209', '0.333']

base_shares = confusion_matrix(y_test, base_pred, normalize="pred")
print(base_shares.round(3))
print(precision_score(y_test, base_pred, zero_division=np.nan))
# The alert team reads the bottom-right cell of the "true" matrix, 1 of
# the 126 fast sales called, its recall, 0.008. The slot team reads the
# bottom-right cell of the "pred" matrix, 1 of the 3 calls sold, its
# precision, 0.333. "true" divides each count by its row's total, the
# listings that truly hold that label, and "pred" by its column's, the
# listings the model gave that label. The baseline's second column held
# no listing, and normalize="pred" wrote 0.0 into both its cells without
# a warning, a share of nothing, where precision_score reports nan.

[[0.79 0.  ]
 [0.21 0.  ]]
nan

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want another pair of ratios read from the same four cells, or a single number that does not reward the majority label.

• YouTube - "Machine Learning Fundamentals: Sensitivity and Specificity" by StatQuest with Josh Starmer. Sensitivity, another name for recall, and specificity, the share of the negatives a model labels negative, both computed from confusion matrices and used to choose between methods.
• Official docs - scikit-learn 1.9 User Guide, "Metrics and scoring", the section "Balanced accuracy score". balanced_accuracy_score, the average of each label's recall, which avoids the inflated scores that imbalanced rows give accuracy, and its adjusted=True form, on which a rule that gives every row the same label scores 0 and random guessing scores 0 on average.

Summary

Accuracy turned out to be the number that said least about this model, and not because it is computed wrongly. It answers a question nobody asked, how often the model agrees with the truth across every listing, and on a market where four listings in five sit unsold past 30 days, agreeing with the truth is mostly agreeing with the majority. The rule that read no feature made the point plainly: a number that such a rule can match says nothing about what the features added.

The matrix is where an honest reading starts, because it refuses to add the two kinds of mistake together. A false positive and a false negative are different events with different costs, a seller told to expect a fast sale that never comes against a seller never told about one that did, and every metric after the matrix is built from a choice of which cells to divide by which. Precision and recall share a numerator and not a denominator, and most of what goes wrong with them goes wrong without a sound: a swapped argument divides by the wrong count, a reordered layout hands the names to the wrong cells, and a default turns a count of nothing into a zero.

Choosing among those ratios is not a technical step. It is the question the model serves, written down before the numbers can argue for themselves. The alert and the slots asked the same model different things, and neither got an answer at 0.5, which says less about the model than about a threshold nobody chose. Both questions still lack one number, how much of the other mistake they will tolerate, and the threshold is where that number takes hold.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

from sklearn.dummy import DummyClassifier
from sklearn.metrics import (
    classification_report, confusion_matrix, f1_score,
    precision_score, recall_score,
)

# the rule every accuracy is read beside
baseline = DummyClassifier().fit(X_train, y_train)
matrix = confusion_matrix(y_test, pred)  # truth first
tn, fp, fn, tp = matrix.ravel()          # [False, True] order
precision_score(y_test, pred)            # tp / (tp + fp)
recall_score(y_test, pred)               # tp / (tp + fn)
f1_score(y_test, pred)                   # harmonic mean of both
# nan, not 0.0, when nothing is called positive
precision_score(y_test, pred, zero_division=np.nan)
print(classification_report(y_test, pred, digits=3))
# shares of each true row: bottom right is recall
confusion_matrix(y_test, pred, normalize="true")