Thresholds, ROC, Class Weights and Imbalanced Classes

Context

A threshold is the probability at or above which a classifier calls a listing sold, and where it sits between 0 and 1 decides the confusion matrix the same fitted model gets. A precision-recall curve and a ROC curve (two charts of how the calls change as the threshold moves) show every threshold at once, and a class weight changes how much each class counts while the model is fitted.

A listings site meets all three whenever a probability becomes an action: the seller alert goes to the listings above a threshold someone chose, the featured slots to those above a higher one, a reviewer compares two classifiers' rankings before either has a threshold, and an alert team that prices a missed fast sale at five wasted alerts turns those prices into a threshold.

Left at 0.5, the threshold predict applies, Lesson 3.4.1's model calls 3 of 600 test listings sold and finds 1 of the 126 that sold. An area under the ROC curve of 0.642 gets reported as if it said how many alerts are wasted, a weighted model's probability of 0.55 reaches a seller's email as a better-than-even chance, and a threshold tuned on the test rows reports the recall it was tuned to reach. Lesson 3.4.2 matched each question to its metric, and this lesson moves the threshold until the metric answers it.

Goal

• Move a classifier's threshold over predict_proba, and trace what each move does to the confusion matrix, precision and recall.
• Compute a precision-recall curve and average precision, check both by hand, and read them against the share of listings that sold.
• Compute a ROC curve and its AUC, verify the AUC by counting pairs of listings, and explain why it cannot say what a threshold costs.
• Fit a pipeline with class_weight="balanced", and distinguish what the weights change in the calls from what they do to the probabilities.
• Choose a threshold by a stated rule on validation rows carved from the training rows, and score the test rows once at it.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want a threshold moved across a model's probabilities, or a ROC curve built point by point, before you start.

• YouTube - "ROC and AUC, Clearly Explained!" by StatQuest with Josh Starmer. A logistic regression's probabilities turned into calls at one threshold after another, the confusion matrix at each, the ROC curve plotted from them point by point, and the area under it used to compare two models.
• Google Machine Learning Crash Course - "Thresholds and the confusion matrix", the section "Effect of threshold on true and false positives and negatives". A slider that moves the threshold over three toy datasets, well separated, overlapping and imbalanced, to try what each move does to the four kinds of call.

Table of Contents

• The Threshold Is a Choice
• Precision-Recall Curves
• ROC Curves and AUC
• Class Weights
• Choosing a Threshold on Validation Rows

Theory

Nothing to install: every scikit-learn class and function this lesson uses arrived with Lesson 3.1.1's install. The first block is a module, sold_split.py, saved in ~/dev/bootcamp beside listings.csv, and every script of the lesson but Lab Exercise 1's imports from it, as Lesson 3.1.2's scripts imported memorizer.py. It loads the listings, makes the section's split, stratified on the label, and defines make_prep, which returns a new, unfitted copy of Lesson 3.4.1's preparation at every call, so each pipeline gets steps of its own, and the last subtitle shows what goes wrong when two pipelines share one.

Every Theory block after the module goes into one script, thresholds.py, run with python3 thresholds.py, each block added under the one before it.

The Threshold Is a Choice

What. A threshold (the probability at or above which a listing is called sold) turns predict_proba's probability into a yes or a no. Lesson 3.4.1's predict calls a listing sold above 0.5, and any number from 0 to 1 is a threshold just as valid, each with the confusion matrix its calls make. Why. The probabilities rank the listings, and the threshold decides how many of them are called sold. Every listing a lower threshold adds is either a sale found or an alert wasted, and which mix is worth having depends on what each mistake costs, Lesson 3.4.2's question, never on 0.5. When. Every time a probability becomes an action, an alert sent or a featured slot sold, which is every time predict would otherwise decide.

sold_split.py:

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)


def make_prep():
    numeric_steps = make_pipeline(
        SimpleImputer(strategy="median", add_indicator=True), StandardScaler()
    )
    return ColumnTransformer([
        ("num", numeric_steps, numeric),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])

thresholds.py fits Lesson 3.4.1's pipeline on the training rows:

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix, precision_score, recall_score
from sklearn.pipeline import make_pipeline
from sold_split import X_test, X_train, make_prep, y_test, y_train

prep = make_prep()
model = make_pipeline(prep, LogisticRegression()).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]  # the probability of True
sold = y_test.to_numpy()                   # True for each listing that sold
print(len(proba), sold.sum())              # 600 126

Before any threshold, how each class's probabilities spread:

by_class = pd.Series(proba).groupby(sold).describe()
print(by_class[["count", "25%", "50%", "75%", "max"]].round(3))

       count    25%    50%    75%    max
False  474.0  0.125  0.181  0.265  0.568
True   126.0  0.173  0.225  0.305  0.512

The listings that sold get higher probabilities on the whole, a median of 0.225 against 0.181, and the two ranges overlap almost end to end. None reaches 0.6, and the highest, 0.568, went to a listing that did not sell. A threshold is a vertical cut through both groups at once.

# True, no test listing sits exactly on predict's 0.5
print(np.array_equal(model.predict(X_test), proba >= 0.5))

for threshold in [0.5, 0.4, 0.3, 0.2, 0.1]:
    called = proba >= threshold  # True where the listing is called sold
    print(
        threshold,
        called.sum(),
        round(precision_score(y_test, called), 3),
        round(recall_score(y_test, called), 3),
    )

0.5 3 0.333 0.008
0.4 25 0.36 0.071
0.3 108 0.324 0.278
0.2 274 0.27 0.587
0.1 529 0.231 0.968

Each line is a threshold, the listings it calls sold, its precision and its recall. As the threshold falls, recall climbs from 0.008 to 0.968, while precision rises to 0.36 and then falls to 0.231. The confusion matrices at 0.5 and 0.2 show where the listings went:

print(confusion_matrix(y_test, proba >= 0.5))
print(confusion_matrix(y_test, proba >= 0.2))

[[472   2]
 [125   1]]
[[274 200]
 [ 52  74]]

Rows are the truth and columns the call, Lesson 3.4.2's layout. From 0.5 to 0.2, 271 more listings moved into the right-hand column, 73 sales the model had missed and 198 listings that did not sell. Recall divides the sales found by the 126 listings that sold, a count no threshold changes, so lowering the threshold never lowers it. Precision divides them by the listings called, and each listing a lower threshold adds pulls precision toward 1 if it sold and toward 0 if it did not.

These test-row numbers show what a threshold does. Choosing one of them would be choosing by the test score, so the threshold this lesson keeps is chosen on validation rows, in the last subtitle.

DON'T:

• DON'T take 0.5 as given. At 0.5 this model calls 3 of the 600 test listings sold and finds 1 of the 126 sales, a recall of 0.008.
• DON'T expect precision to rise with the threshold. It went from 0.36 at 0.4 down to 0.333 at 0.5, where 1 of the 3 listings called had sold.

Precision-Recall Curves

What. A precision-recall curve (precision plotted against recall at every threshold, one point for each distinct probability the model gave) comes from precision_recall_curve, which returns the precisions, the recalls and the thresholds, the thresholds rising. Average precision (the curve condensed to one number, each threshold's precision weighted by the recall it adds) is average_precision_score. Why. Five thresholds printed five points of one trade, and the curve holds all of them, so a threshold is chosen with every option in view. Its baseline is the share of listings that sold, not 0: calling every listing sold gives that precision, and a model that gives every listing the same probability scores that average precision. When. Whenever the question is about the listings that sold, the alert and the featured slot of Lesson 3.4.2, and before a threshold is chosen.

from sklearn.metrics import average_precision_score, precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, proba)
print(len(precision), len(recall), len(thresholds))
# the lowest threshold, which calls every listing sold
print(precision[0], recall[0], round(thresholds[0], 4))
# the highest threshold, which calls one listing
print(precision[-2], recall[-2], round(thresholds[-1], 3))
# the last point, which has no threshold
print(precision[-1], recall[-1])

601 601 600
0.21 1.0 0.0012
0.0 0.0 0.568
1.0 0.0

One threshold for each distinct probability, 600 here because every test listing has its own. The lowest calls every listing, so its recall is 1.0 and its precision the share sold, 0.21, and the highest calls one listing, which did not sell. The 601st point has no threshold behind it: precision_recall_curve appends it to every curve. The point at 0.2 is found by position:

# thresholds rise along the array, so the count of those
# below 0.2 is the position of the first one at 0.2 or above
at = (thresholds < 0.2).sum()
print(round(thresholds[at], 4), round(precision[at], 3), round(recall[at], 3))

0.2006 0.27 0.587

No probability lies between 0.2 and 0.2006, so that threshold calls the same 274 listings, and the curve's point is the loop's. Average precision, from the library and by hand:

ap = average_precision_score(y_test, proba)
gained = recall[:-1] - recall[1:]       # recall added over the next one up
by_hand = (gained * precision[:-1]).sum()
print(round(ap, 3), round(by_hand, 3))  # 0.306 0.306
found_at = precision[:-1][gained > 0]   # precision as each sale is found
# 126 0.306
print(len(found_at), round(found_at.mean(), 3))

A step down to a threshold that adds a sale adds 1/126 of recall, and a step that adds a listing that did not sell adds none, so the weights fall on 126 thresholds, all equal. Here average precision is the precision at the moment each sale is found, averaged over the 126 sales.

Lesson 3.4.2's DummyClassifier gives every listing one probability of sold, the share of the training rows that sold:

from sklearn.dummy import DummyClassifier

baseline = DummyClassifier().fit(X_train, y_train)
flat = baseline.predict_proba(X_test)[:, 1]
flat_ap = average_precision_score(y_test, flat)
print(flat[:3].round(3))  # [0.209 0.209 0.209]
print(round(flat_ap, 3))  # 0.21

One probability makes one threshold, which calls every listing, so the baseline's curve holds one point besides the appended one, precision 0.21 at recall 1.0, the level the model's 0.306 is measured from. The display draws both:

import matplotlib.pyplot as plt
from sklearn.metrics import PrecisionRecallDisplay

fig, ax = plt.subplots(figsize=(6, 5))
pr = PrecisionRecallDisplay.from_predictions(
    y_test, proba, name="logistic", ax=ax, plot_chance_level=True
)
fig.savefig("pr_curve.png", dpi=150, bbox_inches="tight")
print(len(ax.lines))                 # 2
print(len(pr.line_.get_xdata()))     # 601
print(pr.line_.get_label())          # logistic (AP = 0.31)
print(pr.chance_level_.get_label())  # Chance level (AP = 0.21)
plt.close(fig)

The Axes holds the curve, pr.line_, all 601 points, and the horizontal chance line plot_chance_level=True adds at the share sold, and the legend rounds both averages to two decimals.

DON'T:

• DON'T read average precision without the share sold beside it. The model's 0.306 is about one and a half times the 0.21 of one probability for every listing, not a third of the way to a perfect 1.
• DON'T pass predict's calls to average_precision_score. Three listings called sold leave almost no ranking to score, and average_precision_score(y_test, model.predict(X_test)) returns 0.211.
• DON'T read the curve's last point as a threshold the model reaches. Precision 1.0 at recall 0.0 is appended to every curve, and this model's highest threshold, 0.568, has a precision of 0.0.

ROC Curves and AUC

What. A ROC curve (receiver operating characteristic curve) plots two rates at every threshold: the true positive rate, which is recall, the share of the listings that sold that are called sold, against the false positive rate, the share of the listings that did not sell that are called sold. roc_curve returns both with their thresholds, and roc_auc_score the AUC (the area under the ROC curve), which equals the chance that a sold listing drawn at random gets a higher probability than an unsold one drawn at random, a tie counting half. Why. The AUC scores the order the probabilities put the listings in and nothing else, so two models can be compared before either has a threshold. For the same reason it knows nothing about any one threshold: how many alerts a threshold wastes, the question Lesson 3.4.2 asked, is not in it. When. To compare how two models rank the same listings, and never as the number a team plans with.

# the four cells, row by row: tn, fp, then fn, tp
tn, fp, fn, tp = confusion_matrix(y_test, proba >= 0.2).ravel()
tpr_02 = tp / (tp + fn)   # out of the 126 that sold
fpr_02 = fp / (fp + tn)   # out of the 474 that did not
prec_02 = tp / (tp + fp)  # out of the 274 called sold
print(tp, fn, fp, tn)     # 74 52 200 274
# 0.587 0.422 0.27
print(round(tpr_02, 3), round(fpr_02, 3), round(prec_02, 3))

roc_curve computes the first two rates at every threshold:

from sklearn.metrics import roc_auc_score, roc_curve

fpr, tpr, roc_thresholds = roc_curve(y_test, proba)
print(len(fpr), roc_thresholds[0], fpr[0], tpr[0])
print(fpr[-1], tpr[-1], round(roc_thresholds[-1], 4))
every_point = roc_curve(y_test, proba, drop_intermediate=False)
print(len(every_point[0]))

188 inf 0.0 0.0
1.0 1.0 0.0012
601

The thresholds fall along their array, from inf, which no probability reaches, so the curve runs from (0, 0), no listing called, to (1, 1), every listing called. roc_curve leaves out by default points that sit inside straight runs of the curve, which change nothing about how it is drawn, so it holds 188 of the 601. The AUC, from the library and by its definition, every pair of one sold and one unsold listing checked:

auc = roc_auc_score(y_test, proba)
sold_p = proba[sold]     # the 126 that sold
unsold_p = proba[~sold]  # the 474 that did not
# 2.2.4's broadcast: every sold listing against every unsold one
higher = sold_p[:, np.newaxis] > unsold_p[np.newaxis, :]
tied = sold_p[:, np.newaxis] == unsold_p[np.newaxis, :]
by_pairs = (higher.sum() + 0.5 * tied.sum()) / higher.size
print(higher.shape, higher.sum(), tied.sum())
print(round(auc, 3), round(by_pairs, 3))
print(round(roc_auc_score(y_test, flat), 3))

(126, 474) 38339 0
0.642 0.642
0.5

In 38,339 of the 59,724 pairs the listing that sold has the higher probability, none tied, and the baseline's one probability ties every pair, each counting half, so its AUC is 0.5. The count never compares a probability with a threshold, which is why no threshold chosen later can move it. Drawn beside the precision-recall curve, threshold 0.2 marked on both:

from sklearn.metrics import RocCurveDisplay

fig, (left, right) = plt.subplots(1, 2, figsize=(12, 5))
roc = RocCurveDisplay.from_predictions(
    y_test, proba, name="logistic", ax=left, plot_chance_level=True
)
PrecisionRecallDisplay.from_predictions(
    y_test, proba, name="logistic", ax=right, plot_chance_level=True
)
# threshold 0.2 on each: (fpr, tpr) left, (recall, precision) right
left.plot(fpr_02, tpr_02, "o", color="black")
right.plot(tpr_02, prec_02, "o", color="black")
fig.savefig("roc_beside_pr.png", dpi=150, bbox_inches="tight")
print(len(left.lines), len(right.lines))
print(roc.line_.get_label())
# each Axes' last line is its marker, get_xydata() its (x, y)
print(left.lines[-1].get_xydata().round(3))
print(right.lines[-1].get_xydata().round(3))
plt.close(fig)

3 3
logistic (AUC = 0.64)
[[0.422 0.587]]
[[0.587 0.27 ]]

On the left, threshold 0.2 sits well above the chance diagonal (the line a model that ranks listings at random follows), 59 percent of the sales found for 42 percent of the unsold listings flagged. On the right the same 274 alerts have a precision of 0.27, so 73 percent of them went to listings that did not sell: the false positive rate divides the 200 wasted alerts by the 474 unsold listings, and precision sets them against the 74 good ones. Resampling the test rows (drawing rows again so the classes come out in other proportions) shows which numbers depend on how many listings did not sell:

scored = pd.DataFrame({"sold": sold, "proba": proba})
# every sale beside 126 of the 474 unsold listings, drawn at random
balanced = pd.concat([
    scored[scored["sold"]],
    scored[~scored["sold"]].sample(n=126, random_state=42),
])
called = balanced["proba"] >= 0.2
truth = balanced["sold"]
print(confusion_matrix(truth, called))
print(round(precision_score(truth, called), 3))
print(round(roc_auc_score(truth, balanced["proba"]), 3))
print(round(average_precision_score(truth, balanced["proba"]), 3))

[[74 52]
 [52 74]]
0.587
0.647
0.607

The balanced rows still find 74 of the 126 sales, a recall of 0.587, but the false alarms fell from 200 to 52 with the unsold listings, so precision went from 0.27 to 0.587 and average precision from 0.306 to 0.607, while the AUC moved from 0.642 to 0.647. The ROC curve's two rates each divide by one class, so the share sold never enters them, and precision divides by the listings called, which mix both: where unsold listings outnumber sold ones, a ROC curve can look better than the precision the alerts deliver.

DON'T:

• DON'T report the AUC as the alert's quality. 0.642 holds at every threshold, and at 0.2 the model wastes 200 of its 274 alerts.
• DON'T read the false positive rate as the share of alerts wasted. At 0.2 it is 200 of the 474 unsold listings, 0.422, while 0.73 of the 274 alerts were wasted.
• DON'T compute the AUC from predict's calls. roc_auc_score(y_test, model.predict(X_test)) returns 0.502, barely above a constant's 0.5.
• DON'T resample the test rows. Balanced, they printed a precision of 0.587 for the 0.27 of the real balance, and the alerts go out into a market where about 1 listing in 5 sells within 30 days.

Class Weights

What. A class weight (a multiplier on how much each training row of a class counts in the loss the fit minimizes) is set with LogisticRegression's class_weight. "balanced" gives every row of a class the number of training rows over twice that class's count, so the two classes carry the same total, and a dict sets the two multipliers by hand. Why. In the plain fit every training row counts once, and 1,898 of the 2,400 did not sell. Weighting is a second lever on the trade the threshold controls: it changes the fit instead of the cut, and lifts the probabilities so that predict's fixed 0.5 calls far more listings sold. When. When the calls must come from the model's own predict, whose cut no setting of LogisticRegression moves from 0.5, and only where no one will read the probabilities as chances.

counts = y_train.value_counts()
weights = len(y_train) / (2 * counts)  # what "balanced" computes
print(counts.to_dict())                # {False: 1898, True: 502}
print(weights.round(3).to_dict())      # {False: 0.632, True: 2.39}
print((counts * weights).to_dict())    # {False: 1200.0, True: 1200.0}

weighted = make_pipeline(
    make_prep(), LogisticRegression(class_weight="balanced")
).fit(X_train, y_train)
by_dict = make_pipeline(
    make_prep(), LogisticRegression(class_weight=weights.to_dict())
).fit(X_train, y_train)
# True, "balanced" is these two multipliers
print(np.allclose(weighted[-1].coef_, by_dict[-1].coef_))

Each class now carries 1,200 of the 2,400 rows' weight, and a dict of the two multipliers fits the same weights as "balanced". The weighted model at its own 0.5:

w_proba = weighted.predict_proba(X_test)[:, 1]
w_called = weighted.predict(X_test)  # at its own 0.5
print(w_called.sum(), (w_called & sold).sum())
print(round(precision_score(y_test, w_called), 3))
print(round(recall_score(y_test, w_called), 3))
# the average probability of each model, then the share sold
print(round(proba.mean(), 3), round(w_proba.mean(), 3), round(sold.mean(), 3))
print(round(roc_auc_score(y_test, w_proba), 3))

261 70
0.268
0.556
0.208 0.471 0.21
0.64

It calls 261 listings sold and finds 70 of the 126 sales, where the plain model at 0.5 called 3. Its AUC, 0.64 against 0.642, says it ranks the listings almost as the plain model does, and what moved is the probabilities, an average of 0.471 against 0.208, beside a share sold of 0.21. The plain model reaches the same calls with a threshold:

# the plain model's threshold that calls as many listings,
# the 261st highest of its probabilities
cut = np.sort(proba)[::-1][w_called.sum() - 1]
same_count = proba >= cut
print(round(cut, 3), same_count.sum())  # 0.209 261
print((same_count & sold).sum())        # 70
print((same_count & w_called).sum())    # 258

At 0.209 the plain model calls 261 listings and finds 70 sales too, and 258 of its 261 listings are the weighted model's. On this model the weights reached almost exactly the calls a lower threshold reaches, and paid for them in the probabilities, band by band with 2.6.1's pd.cut:

bands = pd.cut(w_proba, [0, 0.3, 0.4, 0.5, 0.6, 0.7, 1])
table = pd.DataFrame({"proba": w_proba, "sold": sold})
print(table.groupby(bands, observed=True).mean().round(3))

            proba   sold
(0.0, 0.3]  0.222  0.053
(0.3, 0.4]  0.353  0.134
(0.4, 0.5]  0.447  0.248
(0.5, 0.6]  0.549  0.222
(0.6, 0.7]  0.642  0.282
(0.7, 1.0]  0.738  0.406

In every band the weighted model's average probability sits far above the share of its listings that sold: those between 0.5 and 0.6 average 0.549, and 0.222 of them sold. A weighted model's probability still ranks the listings, and no longer says how often such a listing sells.

DON'T:

• DON'T read a weighted model's probability as a chance. Its test probabilities average 0.471 where 0.21 of the listings sold, and the listings it scores between 0.5 and 0.6 sold 0.222 of the time.
• DON'T credit the weights with a better model. The plain model at 0.209 calls as many listings, finds as many sales and shares 258 of the 261 listings, and the two AUCs are 0.642 and 0.64.
• DON'T carry a threshold from the plain model to the weighted one. At 0.2, where the plain model calls 274 listings sold, the weighted model calls 579 of the 600.

Choosing a Threshold on Validation Rows

What. A threshold is a setting chosen among candidates, so it is chosen the way Lesson 3.3.2 chose k: a rule is written down first, the candidates are scored on validation rows carved from the training rows as in Lesson 3.1.2's Assignment, and the test rows are scored once, at the threshold kept. Why. Every threshold printed so far was measured on the test rows, and picking one because its test numbers suit the question is choosing by the test score, Lesson 3.1.2's second break: the test rows would report the very numbers they were used to pick. When. Once the question has named its metric, Lesson 3.4.2's step, and before any number is reported. Lesson 3.6.3 makes the same choice over the many splits of Lesson 3.6.2's cross-validation.

The validation rows come out of the training rows, stratified on the label like the first split:

from sklearn.model_selection import train_test_split

X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)
print(len(X_fit), len(X_val), y_val.sum())  # 1800 600 125
fit_model = make_pipeline(make_prep(), LogisticRegression())
fit_model.fit(X_fit, y_fit)
val_proba = fit_model.predict_proba(X_val)[:, 1]

The alert team's question is Lesson 3.4.2's, where a missed fast sale costs the seller the alert. Its rule, fixed before any threshold is tried: find at least 80 percent of the listings that sell within 30 days, with as few alerts as possible. Recall never rises as the threshold rises, so the rule's threshold is the highest one whose validation recall is still 0.8 or more:

v_precision, v_recall, v_thresholds = precision_recall_curve(y_val, val_proba)
reaches = v_recall[:-1] >= 0.8        # last point dropped, one per threshold
chosen = v_thresholds[reaches].max()  # the highest that reaches it
val_called = val_proba >= chosen
print(round(chosen, 4))
print(val_called.sum())
print(round(precision_score(y_val, val_called), 3))
print(round(recall_score(y_val, val_called), 3))

0.1546
388
0.258
0.8

On the validation rows 0.1546 calls 388 of 600 listings and finds 80 percent of their 125 sales, a target the rule chose it to meet on these very rows, not an estimate. The validation rows have done their job and are training rows again, as in Lesson 3.3.2: the threshold goes to model, fitted on all 2,400, and the test rows are scored once.

test_called = proba >= chosen  # model, fitted on all 2,400
print(test_called.sum())
print(round(precision_score(y_test, test_called), 3))
print(round(recall_score(y_test, test_called), 3))

396
0.263
0.825

These are the numbers the alert team plans with: 396 alerts for 600 listings posted, about two for every three, 0.263 of them to listings that sell within 30 days, and 0.825 of the fast sales found, one split's draw.

DON'T:

• DON'T choose the threshold on the test rows. The same rule run on them picks 0.1599 and prints a recall of 0.802, a target met on the rows that were meant to check it.
• DON'T report the validation recall. 0.8 is the target the rule was built to reach on those rows, and the test rows' 0.825 is the estimate.
• DON'T hand one prep object to two pipelines. make_pipeline keeps the very object it is given, so fitting make_pipeline(prep, LogisticRegression()) on the 1,800 fit rows refits model's own first step: all 600 of model's test probabilities moved, and at 0.1546 it called 404 listings instead of 396.

Lab Exercises

Lab Exercise 1: Price and Distance Alone

In two_column_thresholds.py, split the listings as sold_split.py does, and fit a pipeline of a StandardScaler and a LogisticRegression on two columns alone, price and distance_km, neither of which has a gap.

Print the median probability of the test listings that sold and of those that did not, then the lowest and the highest probability any test listing gets. For the thresholds 0.1, 0.15, 0.2, 0.25 and 0.3, print how many test listings each calls sold, its precision and its recall.

At 0.2, print the four cells of the confusion matrix, then precision and recall computed from the cells by hand, beside what precision_score and recall_score return.

Finally, print how many thresholds the precision-recall curve holds and its first point, then the average precision beside the share of test listings that sold. Draw the curve with PrecisionRecallDisplay and its chance level, save it as two_columns_pr.png, and print how many lines the Axes holds and the curve's label, with a comment on why the curve holds fewer thresholds than there are test listings.

Lab Exercise 2: The Featured Slot, Chosen on Validation Rows

The featured-slot team of Lesson 3.4.2 fills its paid featured slots on the home page, and a wrong pick costs the slot. Its rule, fixed before any threshold is tried: feature as many listings as possible while at least 35 percent of them sell within 30 days.

In featured_slot.py, import the split and make_prep from sold_split.py, and carve the validation rows as the Theory did. Fit the plain pipeline and one with class_weight="balanced" on the fit rows, and for each print its validation AUC from roc_auc_score and from a function of your own that counts pairs, and its mean validation probability, then the share of validation listings that sold.

Apply the team's rule to the plain model: among the thresholds precision_recall_curve returns for its validation probabilities, keep the lowest one whose precision is at least 0.35, and print it with the number of validation listings it features and their precision. Then fit the plain pipeline on all the training rows, score the test rows once at that threshold, and print how many listings it features, the precision and the recall. Finally print how many validation listings the weighted model would feature at the same threshold, with a comment saying why.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    PrecisionRecallDisplay,
    average_precision_score,
    confusion_matrix,
    precision_recall_curve,
    precision_score,
    recall_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

listings = pd.read_csv("listings.csv")
X = listings[["price", "distance_km"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
two = make_pipeline(StandardScaler(), LogisticRegression())
two.fit(X_train, y_train)
proba = two.predict_proba(X_test)[:, 1]
sold = y_test.to_numpy()
print(round(np.median(proba[sold]), 3), round(np.median(proba[~sold]), 3))
print(round(proba.min(), 3), round(proba.max(), 3))

for threshold in [0.1, 0.15, 0.2, 0.25, 0.3]:
    called = proba >= threshold
    print(
        threshold,
        called.sum(),
        round(precision_score(y_test, called), 3),
        round(recall_score(y_test, called), 3),
    )

0.216 0.193
0.043 0.386
0.1 582 0.211 0.976
0.15 478 0.226 0.857
0.2 287 0.24 0.548
0.25 164 0.28 0.365
0.3 75 0.24 0.143

called = proba >= 0.2
tn, fp, fn, tp = confusion_matrix(y_test, called).ravel()
precision_02 = tp / (tp + fp)  # out of the 287 called
recall_02 = tp / (tp + fn)     # out of the 126 that sold
print(tn, fp, fn, tp)
print(round(precision_02, 3), round(recall_02, 3))
print(
    round(precision_score(y_test, called), 3),
    round(recall_score(y_test, called), 3),
)

256 218 57 69
0.24 0.548
0.24 0.548

precision, recall, thresholds = precision_recall_curve(y_test, proba)
print(len(thresholds), precision[0], recall[0])
ap = average_precision_score(y_test, proba)
print(round(ap, 3), round(y_test.mean(), 3))

fig, ax = plt.subplots(figsize=(6, 5))
pr = PrecisionRecallDisplay.from_predictions(
    y_test, proba, name="price and distance", ax=ax, plot_chance_level=True
)
fig.savefig("two_columns_pr.png", dpi=150, bbox_inches="tight")
print(len(ax.lines), pr.line_.get_label())
plt.close(fig)
# 594 thresholds for 600 listings: six test listings repeat another's
# price and distance, so they share its probability and its threshold.

594 0.21 1.0
0.251 0.21
2 price and distance (AP = 0.25)

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    precision_recall_curve,
    precision_score,
    recall_score,
    roc_auc_score,
)
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sold_split import X_test, X_train, make_prep, y_test, y_train

X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)


def pairs_auc(truth, proba):
    sold_p, unsold_p = proba[truth], proba[~truth]
    higher = sold_p[:, np.newaxis] > unsold_p[np.newaxis, :]
    tied = sold_p[:, np.newaxis] == unsold_p[np.newaxis, :]
    return (higher.sum() + 0.5 * tied.sum()) / higher.size


truth = y_val.to_numpy()
val_probas = {}
for name, weight in [("plain", None), ("weighted", "balanced")]:
    pipe = make_pipeline(make_prep(), LogisticRegression(class_weight=weight))
    pipe.fit(X_fit, y_fit)
    val_probas[name] = pipe.predict_proba(X_val)[:, 1]
    auc = roc_auc_score(y_val, val_probas[name])
    by_pairs = pairs_auc(truth, val_probas[name])
    mean = val_probas[name].mean()
    print(name, round(auc, 3), round(by_pairs, 3), round(mean, 3))
print(round(y_val.mean(), 3))

plain 0.647 0.647 0.207
weighted 0.641 0.641 0.468
0.208

plain_val = val_probas["plain"]
precision, recall, thresholds = precision_recall_curve(y_val, plain_val)
meets = precision[:-1] >= 0.35    # one entry per threshold
chosen = thresholds[meets].min()  # the lowest features the most
featured = plain_val >= chosen
print(round(chosen, 4), featured.sum())
print(round(precision_score(y_val, featured), 3))

model = make_pipeline(make_prep(), LogisticRegression())
model.fit(X_train, y_train)
on_test = model.predict_proba(X_test)[:, 1] >= chosen
print(on_test.sum())
print(round(precision_score(y_test, on_test), 3))
print(round(recall_score(y_test, on_test), 3))
print((val_probas["weighted"] >= chosen).sum())
# The weighted model's validation probabilities average 0.468, not
# 0.207, so 0.3821 sits low among them: 425 of 600 listings clear it.

0.3821 42
0.357
31
0.387
0.095
425

Theory Assessment

Q1: One Step Down the Thresholds

precision_recall_curve returned 600 thresholds, one per test listing, so each step down to the next lower threshold calls one more listing sold. Say what the step does to recall and to precision when that listing sold and when it did not, and use it to explain why recall never fell from 0.5 to 0.1 while precision rose between 0.5 and 0.4 and fell below 0.4. Then say which threshold produced the curve's first point, precision 0.21 at recall 1.0, and which its last, precision 1.0 at recall 0.0.

Hint: Write recall and precision as fractions of the four cells, and ask which denominator a single step can change. For the two points, ask how many listings the lowest threshold calls.

Q2: What 0.642 Counts

A colleague reads the AUC of 0.642 as "the model is right about 64 percent of the listings". Explain what 0.642 counts instead, with this lesson's numbers, why it stays the same whatever threshold the alert team later picks, why the DummyClassifier scores exactly 0.5, and why roc_auc_score(y_test, model.predict(X_test)) returns 0.502.

Hint: Count the pairs one sold and one unsold listing make, and ask what a tie scores. Then ask what is left of the ranking once predict has turned 600 probabilities into 3 calls of True and 597 of False.

Q3: The Same Alerts, Two Denominators

At threshold 0.2 the model's false positive rate is 0.422, its true positive rate 0.587 and its precision 0.27. A teammate balances the test rows to 126 sold and 126 unsold listings and reports a precision of 0.587 and an AUC of 0.647 as "the model's real quality, free of the imbalance". Explain from each number's denominator why the true positive rate stayed, why the AUC barely moved and why precision more than doubled, and say which test rows the alert team should plan with.

Hint: Write the three rates as fractions of the cells, and ask which of their denominators mixes listings that sold with listings that did not.

Q4: A Weighted Model at 0.5

A teammate writes: "Class weights fixed the model. At 0.5 the plain model found 1 of the 126 sales and the weighted one finds 70, and its AUC barely moved, 0.642 to 0.64, so we keep the ranking and gain the recall. Its probabilities are better too: a listing it scores 0.55 can be told it is more likely than not to sell within 30 days." Say which claims hold, and correct the others with this lesson's numbers.

Hint: Set the weighted model at 0.5 beside the plain model at the threshold that calls the same number of listings. Then ask what share of the listings the weighted model scores between 0.5 and 0.6 sold.

# at the end of thresholds.py: np.diff runs up the thresholds,
# so a fall it counts is a rise on the step down, and the reverse
steps = np.diff(precision[:-1])
print((steps < 0).sum(), (steps > 0).sum())  # 126 473

# at the end of thresholds.py: the pairs once predict has decided
calls = model.predict(X_test).astype(int)  # 1 for True, 0 for False
ahead = calls[sold][:, np.newaxis] - calls[~sold][np.newaxis, :]
print((ahead > 0).sum(), (ahead < 0).sum(), (ahead == 0).sum())

472 250 59002

Assignment

Assignment: The Alert Threshold That Costs Least

The alert team has priced both of its mistakes: a fast sale the alert misses costs 50, the follow-up business the alert would have opened, and an alert to a listing that does not sell within 30 days costs 10, in support time and unsubscribes. The team now wants the threshold with the lowest total cost.

The prices alone point at a threshold, if the probabilities are chances. A listing that sells fast with probability p costs 10 * (1 - p) on average when it is alerted, since that is how often the alert is wasted, and 50 * p when it is not, since that is how often a fast sale is missed. Alerting pays wherever 10 * (1 - p) is below 50 * p, which is wherever p is above 10 / (10 + 50), about 0.167.

Write alert_cost.py, importing the split and make_prep from sold_split.py. Carve the validation rows as the Theory did, fit the plain pipeline and a class_weight="balanced" one on the fit rows, and write alert_cost(truth, called), which returns the total cost of a set of calls. For each model, scan the thresholds 0.01 to 0.99 in steps of 0.01 on the validation rows, and print the cheapest threshold, its cost, and the cost and the number of sellers alerted at the formula's threshold, then the validation cost of alerting every seller and of alerting none. The team's rule, fixed before the scan, keeps the model and threshold with the lowest validation cost. Refit that model on all the training rows, score the test rows once at its threshold, and print the cost, the alerts sent, the alerts wasted and the fast sales missed, beside the test cost of alerting everyone and no one. End with a comment saying why the formula lands beside one model's cheapest threshold and far from the other's.

Hints:

• confusion_matrix(truth, called).ravel() returns the cells as tn, fp, fn, tp, so a wasted alert is fp and a missed fast sale is fn. np.argmin over the grid's costs returns the position of the cheapest threshold, the lowest one when several tie.
• Keep each model's cheapest validation cost in a dict. min(costs, key=costs.get) returns the key with the smallest value, as Lesson 3.3.2's max returned the largest, and the test rows enter the script only after that line.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sold_split import X_test, X_train, make_prep, y_test, y_train

MISSED, WASTED = 50, 10   # a fast sale missed, an alert wasted

X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)


def alert_cost(truth, called):
    tn, fp, fn, tp = confusion_matrix(truth, called).ravel()
    return MISSED * fn + WASTED * fp


# alerting pays where WASTED * (1 - p) is below MISSED * p
formula = WASTED / (WASTED + MISSED)
print(round(formula, 3))  # 0.167

# 0.01 to 0.99, rounded so that 0.35 prints as 0.35
grid = np.round(np.arange(0.01, 1.0, 0.01), 2)
weights = {"plain": None, "weighted": "balanced"}
val_costs, cheapest = {}, {}
for name, weight in weights.items():
    pipe = make_pipeline(make_prep(), LogisticRegression(class_weight=weight))
    val_proba = pipe.fit(X_fit, y_fit).predict_proba(X_val)[:, 1]
    costs = np.array([alert_cost(y_val, val_proba >= t) for t in grid])
    cheapest[name] = grid[np.argmin(costs)]
    val_costs[name] = costs.min()
    at_formula = val_proba >= formula
    print(
        name,
        cheapest[name],
        costs.min(),
        alert_cost(y_val, at_formula),
        at_formula.sum(),
    )

everyone = np.ones(len(y_val), dtype=bool)
print(alert_cost(y_val, everyone), alert_cost(y_val, ~everyone))

plain 0.17 4040 4100 355
weighted 0.35 4100 4620 587
4750 6250

# the team's rule, fixed before the scan: the lowest validation cost
chosen_model = min(val_costs, key=val_costs.get)
threshold = cheapest[chosen_model]
# plain 0.17 4040
print(chosen_model, threshold, val_costs[chosen_model])

final = make_pipeline(
    make_prep(), LogisticRegression(class_weight=weights[chosen_model])
).fit(X_train, y_train)
called = final.predict_proba(X_test)[:, 1] >= threshold
tn, fp, fn, tp = confusion_matrix(y_test, called).ravel()
print(alert_cost(y_test, called), called.sum(), fp, fn)
everyone = np.ones(len(y_test), dtype=bool)
print(alert_cost(y_test, everyone), alert_cost(y_test, ~everyone))
# The formula assumes the probabilities are chances. The plain model's
# cheapest validation threshold, 0.17, sits beside the formula's 0.167.
# The weights lifted the probabilities, so the weighted model's cheapest
# is 0.35, and at 0.167 it alerts 587 of the 600 validation sellers.

4200 360 265 31
4740 6300

Additional Resources

Optional, and nothing later in the module depends on either. Worth reading for costs that change from row to row, or for another way to train on a rare class.

• Official docs - scikit-learn 1.9 example, "Post-tuning the decision threshold for cost-sensitive learning". A credit model whose two mistakes carry a cost table, one five times the other, its threshold tuned to the lowest total cost, then a fraud model whose cost depends on each transaction's amount.
• Google Machine Learning Crash Course - "Datasets: Class-imbalanced datasets", the section "Training a class-imbalanced dataset". Dropping most of the common class's rows from training, 24 of every 25 in its example, then multiplying the loss on each of that class's rows kept by 25, so the model learns what each class looks like and still learns how common each class is.

Summary

A classifier's probability never says sold. It says how likely, and someone decides where likely enough begins. The probabilities rank the listings and the threshold turns the ranking into calls, so one fitted model sent 3 alerts or 529 depending on a number nobody fitted, and the default of 0.5 sat beyond almost every probability the model gave.

The two curves split that labor. The ROC curve and its AUC judge the ranking alone, which is why they survive any threshold and any balance of the classes, and why they cannot say what one alert is worth. The precision-recall curve keeps the share sold inside every point, which is why it tells the alert team what its alerts deliver, and why balancing the test rows flattered it.

Class weights turned out to be a threshold in disguise on this model, the same calls reached by bending the fit instead of moving the cut, at the price of probabilities that stopped meaning chances. And the threshold is one more setting under the module's rule, chosen on rows the test never sees and reported at the recall the test rows returned, not the one the rule was built to reach.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

proba = model.predict_proba(X_test)[:, 1]    # probability of True
called = proba >= 0.2                        # a threshold you chose
tn, fp, fn, tp = confusion_matrix(y_test, called).ravel()

p, r, t = precision_recall_curve(y_test, proba)
average_precision_score(y_test, proba)       # baseline: the share sold
fpr, tpr, roc_t = roc_curve(y_test, proba)
roc_auc_score(y_test, proba)                 # baseline 0.5, no threshold

LogisticRegression(class_weight="balanced")  # probabilities stop being chances

# choose on validation rows, score the test rows once
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)
vp, vr, vt = precision_recall_curve(y_val, val_proba)
chosen = vt[vr[:-1] >= 0.8].max()            # recall 0.8, fewest calls