Cross-Validation

Context

Cross-validation cuts the training rows into k folds (parts of nearly equal size), fits the model k times, each time on every fold but one, and scores each fit on the fold it left out. Every training row validates the model exactly once, and the score arrives as k numbers with a spread.

A pricing team cross-validates before it touches the test rows: to set the linear pipeline beside a forest on the same listings, to report how far the average miss moves from one set of listings to the next, to check that a new feature's gain holds beyond one carve, and to score the 30-day sale classifier on folds that each hold their share of the sales.

Without it, a choice rides on one carve: the leaf size Lesson 3.5.1's validation rows chose won on none of ten other carves of the same training rows. With it, the mistakes are just as quiet. An integer cv on rows sorted by price scores the linear pipeline at -6.787 on one fold, the mean absolute error comes back negative, and two models scored on different folds trade places. Lesson 3.6.1 measured its models on one carve of 600 validation rows, and this lesson replaces that carve with five folds that cover all 2,400 training rows.

Goal

• Build five-fold cross-validation with KFold(n_splits=5, shuffle=True, random_state=42) on the training rows, prove that every training row validates exactly once, and report cross_val_score's five scores with their mean and spread.
• Diagnose what an integer cv builds with check_cv, and identify folds that are not as new to the model as the rows it will meet in use.
• Apply cross_validate with two scorers, flip the negative mean absolute error, and trace each fold's fitted imputer to the rows it was fitted on.
• Construct stratified folds for the 30-day sale label, score average precision on them, and explain what StratifiedKFold does with a whole-number price and with a band of it.
• Compare two models on the same folds by their per-fold differences, and identify a comparison made on different folds.

Prerequisites

Neither is required, and the lesson stands on its own. Both are here if you want the folds drawn, or scikit-learn's own account of them, before you start.

• YouTube - "Machine Learning Tutorial Python 12 - K Fold Cross Validation" by codebasics. Training and testing on the same rows, one train-test split and k-fold cross-validation set side by side, then KFold and its split method shown on the numbers 1 to 9, and StratifiedKFold and cross_val_score run on scikit-learn's handwritten digits.
• Official docs - scikit-learn 1.9 User Guide, "Cross-validation: evaluating estimator performance", the section "Computing cross-validated metrics". cross_val_score on five folds with the mean and standard deviation of its scores, the scoring parameter, the splitter an integer cv stands for, a pipeline whose steps learn from each fold's training part, and cross_validate's dictionary of scores, times and fitted models.

Table of Contents

• k-Fold Cross-Validation
• Folds as New as the Rows the Model Will Meet
• cross_validate, and Every Step Refitted Inside Each Fold
• Stratified Folds for a Label
• Comparing Two Models on the Same Folds

Theory

Nothing to install: every splitter and function here arrived with Lesson 3.1.1's scikit-learn. Every Theory block goes into one script, folds.py, saved in ~/dev/bootcamp beside listings.csv and run with python3 folds.py, each block under the one before, except the blocks of Stratified Folds for a Label, which start a script of their own, and the last subtitle adds its blocks to folds.py again. The regression blocks import from price_models.py, the module saved in Lesson 3.5.1: the split of all 3,000 listings into 2,400 training rows and 600 test rows, and the builders make_prep, fill and make_tree. Every fold the Theory, the labs and the Assignment cut comes from a split's 2,400 training rows, and that split's 600 test rows stay out of all of them.

k-Fold Cross-Validation

What. k-fold cross-validation cuts the training rows into k folds, fits the model k times, each time on k - 1 folds, and scores each fit on the one fold it left out, so one validation score becomes k of them and every training row validates the model exactly once. KFold, from sklearn.model_selection, cuts the folds, and cross_val_score, from the same module, runs the k fits and returns the k scores. Why. A validation carve is one draw of the training rows, Lesson 3.1.2's point about seeds, and a setting chosen on one carve can be chosen by that carve's luck. Five folds score the model on five validation sets that never overlap, and the spread of the five says how far the score moves when the listings change. When. From this lesson on, for every score that compares models, settings or feature sets on the training rows, and for every score reported about a model before the test rows score the final choice once. Lesson 3.6.3 chooses settings on such folds.

Lesson 3.5.1 chose its tree's leaf size on one carve of the training rows, 600 held back and 1,800 to fit. The same table on that carve and on ten others, each carve made by train_test_split with its own seed:

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score, train_test_split
from sklearn.pipeline import make_pipeline
from price_models import X_train, y_train, fill, make_prep, make_tree

leaves = [1, 2, 5, 10, 20, 50, 100]
winners = []
# seed 42 is Lesson 3.5.1's carve, then ten others
for seed in [42] + list(range(10)):
    X_a, X_b, y_a, y_b = train_test_split(
        X_train, y_train, test_size=0.25, random_state=seed
    )
    val = {}
    for leaf in leaves:
        tree = make_tree(min_samples_leaf=leaf).fit(X_a, y_a)
        val[leaf] = round(tree.score(X_b, y_b), 3)
    winners.append(max(val, key=val.get))
print(winners)  # [5, 10, 20, 10, 10, 10, 10, 10, 20, 10, 10]

The first carve is Lesson 3.5.1's, and it picks that lesson's leaf size, 5. On the ten other carves of the same 2,400 rows, 5 never wins: 10 wins eight times and 20 twice. Nothing changed between the eleven runs but which 600 listings were held back, so the win was a fact about one carve. KFold cuts five folds instead:

kfold = KFold(n_splits=5, shuffle=True, random_state=42)
times = np.zeros(len(X_train), dtype=int)  # validations per row
for fit_pos, val_pos in kfold.split(X_train):
    print(len(fit_pos), len(val_pos), val_pos[:4])
    times[val_pos] += 1                    # each fold's rows, once
print(np.bincount(times))                  # [   0 2400]

1920 480 [18 25 29 43]
1920 480 [ 2  8 13 20]
1920 480 [ 3  5  6 10]
1920 480 [ 0  7  9 24]
1920 480 [ 1  4 11 16]

shuffle=True shuffles the 2,400 positions once, random_state=42 fixes that shuffle, and n_splits=5 cuts the shuffled positions into five folds of 480. split yields each fold's two arrays of positions, the 1,920 rows to fit on and the 480 to validate on, and X_train is the only table it was handed, so no test listing is in any fold. times counts how often each position validated, and Lesson 3.5.2's np.bincount says that no position validated zero times and all 2,400 validated once. Five carves of a fifth each, as many fits as five folds, spread the validation unevenly:

positions = np.arange(len(X_train))
carved = np.zeros(len(X_train), dtype=int)
for seed in range(5):
    fit_part, val_part = train_test_split(
        positions, test_size=0.2, random_state=seed
    )
    carved[val_part] += 1
print(np.bincount(carved))  # [ 773 1010  478  122   17]

Each carve draws its fifth on its own, so the five never validated 773 of the 2,400 rows, validated 1,010 once and 17 four times, and their five scores weigh some listings four times and others not at all. The five folds' scores, first by hand and then by cross_val_score:

by_hand = []
for fit_pos, val_pos in kfold.split(X_train):
    model = make_pipeline(make_prep(fill()), LinearRegression())
    model.fit(X_train.iloc[fit_pos], y_train.iloc[fit_pos])
    val_r2 = model.score(X_train.iloc[val_pos], y_train.iloc[val_pos])
    by_hand.append(val_r2)

linear = make_pipeline(make_prep(fill()), LinearRegression())
scores = cross_val_score(linear, X_train, y_train, cv=kfold)
print(np.round(by_hand, 3))
print(scores.round(3))
print(np.array_equal(scores, by_hand))
print(round(scores.mean(), 3), round(scores.std(), 3))

[0.838 0.819 0.694 0.794 0.825]
[0.838 0.819 0.694 0.794 0.825]
True
0.794 0.052

The loop fits a new linear pipeline, Lesson 3.2.3's, on each fold's 1,920 fit rows and scores it on the fold's 480, and cross_val_score is that loop: handed the pipeline, the rows and the splitter as cv, it returns the same five scores, R² because a regression pipeline's score is R², Lesson 3.2.3's point. On five sets of listings it never saw, the linear pipeline scores from 0.694 to 0.838, a mean of 0.794 and a standard deviation of 0.052, NumPy's std with Module 2's ddof of 0. The fold that scored 0.694 holds 480 listings like the ones the model will price, and it is as much the model's score as the fold that scored 0.838, so the report carries the mean and the spread together.

DON'T:

• DON'T report the mean of the folds without their spread. The linear pipeline's five scores ran from 0.694 to 0.838 around a mean of 0.794.
• DON'T trust a setting because one validation carve chose it. Leaf size 5 won Lesson 3.5.1's carve and none of ten others.
• DON'T cut the folds from rows that hold the test listings. Every row handed to split validates once, so folds of all 3,000 listings score the model on the 600 test listings, and a choice made on those scores is made by the test score, Lesson 3.1.2's second break.

Folds as New as the Rows the Model Will Meet

What. Lesson 3.1.2's held-out rule, applied to each fold: every fold must be as new to the model as the rows it will meet in use. Nothing about a validation fold may reach the fit it scores, or any decision about the model made before that fit, and the fold must be new to its fit in the way the listings the model will price are new to it. Handed an integer, cv=5, cross_val_score builds unshuffled folds, so the order the rows arrive in decides what each fold holds. Why. A fold scores the model in place of the listings it will price. A fold that differs from them in a way the order of the rows decided measures that order, and its score prints exactly like an honest one. When. Every time cv is set. Pass a splitter, never an integer: on rows like these, a shuffled, seeded one.

check_cv, from sklearn.model_selection, returns the splitter cross_val_score builds from whatever it is handed as cv:

from sklearn.model_selection import check_cv
from price_models import listings, y_test

print(check_cv(5))
sold = listings["sold_in_30_days"]
print(check_cv(5, sold, classifier=True))

KFold(n_splits=5, random_state=None, shuffle=False)
StratifiedKFold(n_splits=5, random_state=None, shuffle=False)

An integer becomes KFold with shuffle=False, or, when the model is a classifier and the target holds classes, StratifiedKFold with shuffle=False, the splitter Stratified Folds for a Label takes apart. Unshuffled, the first fold is the first fifth of the rows in the order they sit, the second fold the next fifth, and so on. On the training rows sorted by price, the integer against the shuffled kfold:

order = y_train.sort_values().index  # cheapest listing first
X_sorted, y_sorted = X_train.loc[order], y_train.loc[order]
by_int = cross_val_score(linear, X_sorted, y_sorted, cv=5)
by_kfold = cross_val_score(linear, X_sorted, y_sorted, cv=kfold)
print(by_int.round(3))
print(by_kfold.round(3))

[-1.419 -6.787 -6.581 -3.071 -0.671]
[0.83  0.791 0.784 0.812 0.754]

The same 2,400 rows, the same pipeline, five folds both times. The integer's folds score from -6.787 to -0.671, worse on every fold than predicting that fold's own mean price, and the shuffled folds from 0.754 to 0.83. What each fold held, its lowest price on the first line and its highest on the second, then the test rows' range:

for splitter in (check_cv(5), kfold):
    lows, highs = [], []
    for fit_pos, val_pos in splitter.split(X_sorted):
        lows.append(int(y_sorted.iloc[val_pos].min()))
        highs.append(int(y_sorted.iloc[val_pos].max()))
    print(lows)
    print(highs)
print(y_test.min(), y_test.max())

[60000, 227000, 300000, 383000, 518000]
[227000, 300000, 382000, 517000, 1930000]
[90000, 64000, 65000, 60000, 60000]
[1619000, 1233000, 1083000, 1377000, 1930000]
66000 1410000

Each integer fold is one fifth of the listings by price, the cheapest 480 from 60,000 to 227,000 up to the dearest 480 from 518,000 to 1,930,000, and each is scored by a model fitted on the other four fifths: the dearest fold was priced by a model whose fit rows cost at most 517,000. R² measures a fold's miss against the spread of that fold's own prices, Lesson 3.1.4's point, and a fold that is one stretch of prices has little spread to measure against, the second running only from 227,000 to 300,000. Every shuffled fold reaches from the cheap listings to the dear ones, its lowest price between 60,000 and 90,000 and its highest between 1,083,000 and 1,930,000. The test rows, Lesson 3.1.2's stand-in for the listings the model will meet, reach the same way, from 66,000 to 1,410,000 inside the training rows' 60,000 to 1,930,000. Each sorted fold is new to its fit in a way the test rows never were, a stretch of prices its four fit folds hold none of, apart from any price two neighboring folds share where they meet, and the scores measure that order, not the model. On the training rows as the split left them:

as_split = cross_val_score(linear, X_train, y_train, cv=5)
print(as_split.round(3), round(as_split.mean(), 3))

[0.857 0.713 0.799 0.792 0.798] 0.792

train_test_split shuffled these rows, so the integer's folds come out mixed, and their mean, 0.792, sits beside the shuffled splitter's 0.794. X_train and X_sorted hold the same 2,400 listings, and only their order differs. Nothing in cross_val_score looks at the order, so an integer's folds are as mixed as the rows that arrive, and cv=kfold shuffles them whatever order they arrive in.

DON'T:

• DON'T pass an integer as cv. cv=5 built KFold(n_splits=5, random_state=None, shuffle=False), and on the training rows sorted by price its folds scored from -6.787 to -0.671.
• DON'T take an integer's sensible scores on one ordering as proof that it is safe. The same cv=5 scored a mean of 0.792 on the rows as the split left them and collapsed on the same rows sorted.

cross_validate, and Every Step Refitted Inside Each Fold

What. cross_validate, from sklearn.model_selection, runs the same fits as cross_val_score and returns a dictionary. Each scorer (an object scikit-learn calls with a fitted model, rows and their targets, which returns one number for which larger always means better) named in scoring adds a key, test_ and the scorer's name, holding one score per fold. return_train_score=True adds each fit's score on its own fit rows under train_, and return_estimator=True keeps the fitted pipelines under estimator. Why. One run gives every number a report needs on the same five folds, and the fitted pipelines show what each fold's fit learned, and from which rows. When. Whenever a model is reported with more than one metric, and whenever what a fitted step learned inside the folds has to be checked.

from sklearn.model_selection import cross_validate

results = cross_validate(
    linear, X_train, y_train, cv=kfold,
    scoring=["r2", "neg_mean_absolute_error"],
    return_train_score=True, return_estimator=True,
)
print(sorted(results))
print(results["test_r2"].round(3))
print(results["test_neg_mean_absolute_error"].round())

['estimator', 'fit_time', 'score_time', 'test_neg_mean_absolute_error', 'test_r2', 'train_neg_mean_absolute_error', 'train_r2']
[0.838 0.819 0.694 0.794 0.825]
[-56682. -54123. -59460. -58588. -54796.]

fit_time and score_time hold each fold's seconds, which change from machine to machine. test_r2 holds cross_val_score's five scores again. The mean absolute error, Lesson 3.1.4's average miss in currency, comes back negative: every scorer is built so that larger is better, and a smaller miss is better, so "neg_mean_absolute_error" returns the miss with its sign flipped. Flipped back:

# the sign flipped back, a miss in currency
mae = -results["test_neg_mean_absolute_error"]
print(mae.round().astype(int))
print(f"{mae.mean():,.0f} {mae.std():,.0f}")
print(results["train_r2"].round(3))

[56682 54123 59460 58588 54796]
56,730 2,071
[0.788 0.792 0.825 0.799 0.791]

The linear pipeline misses a validation listing by 56,730 on average across the folds, with a spread of 2,071 between them, from 54,123 to 59,460. Its scores on its own fit rows run from 0.788 to 0.825, below its validation scores on three folds and above them on two. On four folds the two sit close together, as Lesson 3.1.2 found the linear model's two errors, and on the third the fit rows scored 0.825 while the 480 validation listings scored 0.694. The fitted pipelines:

fitted = results["estimator"]
print(len(fitted), hasattr(linear[-1], "coef_"))
area = X_train["area_sqm"]
# a seeded splitter cuts the same folds at every call
for model, (fit_pos, val_pos) in zip(fitted, kfold.split(X_train)):
    imputer = model[0].named_transformers_["num"]
    print(
        imputer.statistics_[0],       # the area median it learned
        area.iloc[fit_pos].median(),  # its 1,920 fit rows' median
        area.iloc[val_pos].median(),  # its 480 validation rows'
    )
print(area.median())

5 False
89.0 89.0 90.3
89.3 89.3 91.1
90.1 90.1 88.35
89.9 89.9 89.3
90.1 90.1 86.6
89.7

results["estimator"] holds five fitted pipelines, one per fold, while linear itself has no coef_ and was never fitted: cross_validate fits a fresh copy of the pipeline in each fold, as Lesson 3.2.3's ColumnTransformer fits a copy of each group. Each copy's imputer learned the median area of exactly its own 1,920 fit rows, from 89.0 to 90.1, never the 89.7 of all 2,400 and never the median of the fold it was scored on, from 86.6 to 91.1. Cross-validating a pipeline refits every fitted step inside each fold, Lesson 3.2.3's rule run once per fold, so no validation fold reaches a fill, a scaler or an encoder: every statistic a fold's listings are transformed with was learned without them.

DON'T:

• DON'T read test_neg_mean_absolute_error as a signed error. -56,682 is a miss of 56,682 with its sign flipped for the scorer, so flip it back before it goes into a report.
• DON'T read a fitted step from the pipeline you handed cross_validate. linear was never fitted, and each fold's fitted copy, with its own medians, sits in results["estimator"].

Stratified Folds for a Label

What. A stratified fold (a fold that holds each class in nearly the share the whole set of rows holds it) is Lesson 3.1.2's stratify= applied to folds. StratifiedKFold, from sklearn.model_selection, cuts them, takes the same n_splits, shuffle and random_state as KFold, and its split needs the target beside the rows, because the classes decide which rows go to which fold. Why. About one listing in five sold within 30 days, and Lesson 3.4.3's average precision is read against the share that sold, so a fold that drew fewer sales scores against a lower baseline. Stratified folds hold that share steady, so every fold's average precision is read against nearly the same baseline. When. For every cross-validated classifier. A target that is not a set of classes, such as a price, is stratified through a band of it.

This subtitle's blocks start sold_folds.py, in the same folder, on Section 3.4's split stratified on the label, imported from Lesson 3.4.3's sold_split.py:

import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import KFold, StratifiedKFold, cross_val_score
from sklearn.pipeline import make_pipeline
from sold_split import X_train, make_prep, y_train

strat = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
for splitter in (strat, kfold):
    shares = [
        y_train.iloc[val_pos].mean()  # the share that sold
        for fit_pos, val_pos in splitter.split(X_train, y_train)
    ]
    print(np.round(shares, 3))

[0.208 0.208 0.208 0.21  0.21 ]
[0.212 0.192 0.208 0.217 0.217]

StratifiedKFold puts the sales at 0.208 or 0.21 of every fold, beside the training rows' 0.209 that Lesson 3.4.1 printed. The plain shuffled KFold, handed the same rows and the same label, gives one fold 0.192 and two 0.217. Section 3.4's logistic pipeline on the stratified folds, scored by Lesson 3.4.3's average precision:

logistic = make_pipeline(make_prep(), LogisticRegression())
ap = cross_val_score(
    logistic, X_train, y_train, cv=strat, scoring="average_precision"
)
print(ap.round(3))
print(round(ap.mean(), 3), round(ap.std(), 3))

[0.326 0.284 0.414 0.305 0.35 ]
0.336 0.045

The folds' average precision runs from 0.284 to 0.414, a mean of 0.336 and a spread of 0.045, each against a share sold of about 0.21, the baseline a model that ranks nothing scores. Lesson 3.1.2's stratify= refused the raw price, because a category needs a row for each side. The price column of these same training rows, handed to StratifiedKFold as the target, beside a quartile band of it, with the plain KFold last:

price = X_train["price"]
print(price.nunique(), (price.value_counts() == 1).sum())
band = pd.qcut(price, q=4, labels=["q1", "q2", "q3", "q4"])
for splitter, labels in [(strat, price), (strat, band), (kfold, band)]:
    per_fold = [
        band.iloc[val_pos].value_counts(sort=False).tolist()
        for fit_pos, val_pos in splitter.split(X_train, labels)
    ]
    print(per_fold)

703 202
/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/sklearn/model_selection/_split.py:812: UserWarning: The least populated class in y has only 1 members, which is less than n_splits=5.
  warnings.warn(
[[126, 119, 121, 114], [122, 118, 117, 123], [115, 116, 123, 126], [127, 117, 117, 119], [115, 125, 123, 117]]
[[121, 119, 121, 119], [121, 119, 120, 120], [121, 119, 120, 120], [121, 119, 120, 120], [121, 119, 120, 120]]
[[119, 127, 122, 112], [127, 107, 125, 121], [108, 123, 117, 132], [133, 98, 130, 119], [118, 140, 107, 115]]

Each inner list counts a fold's listings in the four quartile bands. The prices are whole numbers, and StratifiedKFold read every distinct one as a class, 703 classes among 2,400 listings, 202 of them a single listing. It refused nothing: it warned that its smallest class has 1 member, fewer than the 5 folds, and cut five folds anyway, holding 114 to 127 listings of a band each. Handed the band, it holds 119 to 121 of every band in every fold, where the plain KFold ranges from 98 to 140. The log of the price is not a whole number:

try:
    # list() asks split for the folds, which runs the check
    list(strat.split(X_train, np.log(price)))
except ValueError as err:
    print(err)

Supported target types are: ('binary', 'multiclass'). Got 'continuous' instead.

StratifiedKFold takes a target of classes, and a column of whole numbers passes as classes, a column of fractions does not. So a price is stratified through a band of it, and the band, like Lesson 3.1.2's, decides which listings each fold holds and is never a column the model is fitted on.

DON'T:

• DON'T cross-validate a classifier on folds that ignore the label. The plain shuffled KFold gave one fold a share sold of 0.192 and two 0.217, and average precision is read against that share.
• DON'T hand StratifiedKFold a whole-number price and read the run that continues after its warning as stratified. It made 703 classes of the prices and put 114 to 127 listings of a quartile band in a fold, where the band itself gave 119 to 121.

Comparing Two Models on the Same Folds

What. Two models cross-validated with the same seeded splitter are scored on the same five validation folds, so their scores pair fold by fold, and the per-fold difference, one model's score minus the other's on the same fold, compares them on the same listings. It is Lesson 3.1.2's gap read seed by seed, with folds in place of seeds. Why. Folds differ in how hard their listings are to price, so a difference read across different folds mixes the models with the folds. On one fold both models face the same listings, and R² values computed on the same rows are comparable, the condition Lesson 3.1.4 set. When. Every time two models, two settings or two feature sets are compared on the training rows.

Back in folds.py, Lesson 3.5.2's chosen forest, 100 trees each offered half the columns at a split, and a fully grown tree, on the linear pipeline's folds:

from sklearn.ensemble import RandomForestRegressor

forest = make_pipeline(
    make_prep("passthrough"),
    RandomForestRegressor(max_features=0.5, random_state=42),
)
forest_scores = cross_val_score(forest, X_train, y_train, cv=kfold)
tree_scores = cross_val_score(make_tree(), X_train, y_train, cv=kfold)
named = [("linear", scores), ("forest", forest_scores), ("tree", tree_scores)]
for name, s in named:
    print(name, s.round(3), round(s.mean(), 3), round(s.std(), 3))

linear [0.838 0.819 0.694 0.794 0.825] 0.794 0.052
forest [0.854 0.843 0.728 0.814 0.822] 0.812 0.045
tree [0.612 0.75  0.481 0.481 0.686] 0.602 0.108

The forest means 0.812 against the linear pipeline's 0.794, and the fully grown tree means 0.602 with the widest spread of the three, 0.108. All three score lowest on the third fold, though the tree's fourth matches it to three decimals. The forest against the linear pipeline, fold by fold:

gain = forest_scores - scores  # the same five folds
print(gain.round(3), round(gain.mean(), 3))
print((gain > 0).sum(), round(gain.std(), 3))

[ 0.017  0.023  0.034  0.021 -0.002] 0.019
4 0.012

The forest leads on four folds, by 0.017 to 0.034, and trails on the fifth by 0.002, a mean gain of 0.019. Its gains spread by 0.012, against 0.045 and 0.052 for the two models' own scores. The same two models on folds cut with another seed, and the forest's new scores set against the linear pipeline's old ones:

other = KFold(n_splits=5, shuffle=True, random_state=0)
forest_other = cross_val_score(forest, X_train, y_train, cv=other)
linear_other = cross_val_score(linear, X_train, y_train, cv=other)
mixed = forest_other - scores         # different folds
paired = forest_other - linear_other  # both on seed 0's
print(mixed.round(3), (mixed > 0).sum(), round(mixed.std(), 3))
print(paired.round(3), (paired > 0).sum(), round(paired.mean(), 3))

[-0.054 -0.031  0.162 -0.004 -0.063] 1 0.083
[ 0.029  0.014 -0.001 -0.012 -0.014] 2 0.003

Set against the linear pipeline's seed-42 folds, the forest's seed-0 folds trail on four and lead on one by 0.162, gains that spread by 0.083 where the paired gains spread by 0.012: unpaired, each gain mixes the difference between two folds with the difference between the models, and pairing takes the folds' difference out. Paired on seed 0's folds, the forest leads on two of five by 0.003 on average, and on seed 42's on four of five by 0.019. Both cuts are honest, and a lead this size moves with the cut. Whether a difference like this is more than luck is the question Lesson 3.7.1 asks, of two models scored on the same test listings.

DON'T:

• DON'T set one model's fold scores against another's from a different cut. Against the linear pipeline's seed-42 folds, the forest's seed-0 folds trailed on four and led on one by 0.162, gains that spread by 0.083 where the paired ones spread by 0.012.
• DON'T re-cut the folds until a model wins. The forest led on four of five folds cut with seed 42 and on two of five cut with seed 0, and keeping the cut that shows the lead is choosing by the score, Lesson 3.1.2's DON'T on seeds.

Lab Exercises

Lab Exercise 1: Two Trees on Five Folds, and Folds Sorted by Area

Work in tree_folds.py in ~/dev/bootcamp, importing from price_models.py. On its one validation carve, Lesson 3.5.1 scored a tree with at least 5 listings per leaf at 0.794 and a tree held to depth 5 at 0.731.

Cross-validate both trees on the 2,400 training rows with the Theory's five shuffled folds, and print each tree's five scores, their mean and their standard deviation.

Then build a ten-fold splitter with the same shuffle and seed, print the number of validation rows in each of its folds, and prove with np.bincount that every training row validates exactly once.

Finally sort the training rows by area_sqm, cross-validate the leaf-size tree on the sorted rows with cv=5 and with the five shuffled folds, and print each integer fold's smallest area, largest area and number of listings with no area. End with a comment saying what the integer folds measured.

Lab Exercise 2: The Label's Two Models on One Set of Folds

Work in sold_compare.py, importing from Lesson 3.4.3's sold_split.py. Lesson 3.5.2 set the logistic pipeline beside a random forest classifier, RandomForestClassifier(random_state=42), behind a preparation that passes the six numeric columns through, gaps included, and encodes the rest as Section 3.4 does.

Cross-validate both models with cross_validate on the Theory's stratified five folds, scoring average precision and ROC AUC, and keep the fitted copies. For each of the two metrics, print the logistic pipeline's five scores, the forest's five scores, the logistic pipeline's lead on each fold and the number of folds it leads on.

Then, fold by fold, print the mean price the logistic copy's scaler learned beside the mean price of that fold's fit rows and of its validation rows, and say in a comment what the three columns show.

import numpy as np
from sklearn.model_selection import KFold, cross_val_score
from price_models import X_train, y_train, make_tree

kfold = KFold(n_splits=5, shuffle=True, random_state=42)
trees = {
    "leaf 5": make_tree(min_samples_leaf=5),
    "depth 5": make_tree(max_depth=5),
}
for name, tree in trees.items():
    s = cross_val_score(tree, X_train, y_train, cv=kfold)
    print(name, s.round(3), round(s.mean(), 3), round(s.std(), 3))

ten = KFold(n_splits=10, shuffle=True, random_state=42)
times = np.zeros(len(X_train), dtype=int)
sizes = []
for fit_pos, val_pos in ten.split(X_train):
    sizes.append(len(val_pos))
    times[val_pos] += 1
print(sizes)
print(np.bincount(times))  # [   0 2400]

leaf 5 [0.781 0.748 0.603 0.654 0.773] 0.712 0.071
depth 5 [0.606 0.71  0.599 0.677 0.757] 0.67 0.061
[240, 240, 240, 240, 240, 240, 240, 240, 240, 240]

# the listings with no area sort last
order = X_train["area_sqm"].sort_values().index
X_area, y_area = X_train.loc[order], y_train.loc[order]
leaf = trees["leaf 5"]
print(cross_val_score(leaf, X_area, y_area, cv=5).round(3))
print(cross_val_score(leaf, X_area, y_area, cv=kfold).round(3))
for fit_pos, val_pos in KFold(n_splits=5).split(X_area):
    fold_area = X_area["area_sqm"].iloc[val_pos]
    print(fold_area.min(), fold_area.max(), fold_area.isna().sum())
# Each integer fold is one fifth of the listings by area, the last
# also holding the 90 with no area, and its tree was fitted on the
# other fifths, so each fold tested the tree on a stretch of areas
# its fit rows held none of, apart from any area two folds share,
# not on listings drawn the way new ones arrive.

[-0.486  0.218  0.232  0.358  0.178]
[0.771 0.736 0.689 0.627 0.738]
24.0 63.0 0
63.0 81.0 0
81.0 105.1 0
105.1 143.9 0
144.0 278.5 90

from sklearn.compose import ColumnTransformer
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from sold_split import (
    X_train, levels, make_prep, nominal, numeric, y_train,
)

strat = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Lesson 3.5.2's forest: the numbers passed through, gaps included
forest_prep = ColumnTransformer([
    ("num", "passthrough", numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
models = {
    "logistic": make_pipeline(make_prep(), LogisticRegression()),
    "forest": make_pipeline(
        forest_prep, RandomForestClassifier(random_state=42)
    ),
}
runs = {}
for name, model in models.items():
    runs[name] = cross_validate(
        model, X_train, y_train, cv=strat,
        scoring=["average_precision", "roc_auc"],
        return_estimator=True,
    )
for metric in ["test_average_precision", "test_roc_auc"]:
    lead = runs["logistic"][metric] - runs["forest"][metric]
    print(runs["logistic"][metric].round(3))
    print(runs["forest"][metric].round(3))
    print(lead.round(3), (lead > 0).sum())

[0.326 0.284 0.414 0.305 0.35 ]
[0.253 0.252 0.302 0.264 0.272]
[0.073 0.032 0.113 0.04  0.079] 5
[0.64  0.634 0.697 0.633 0.657]
[0.573 0.585 0.645 0.569 0.571]
[0.067 0.049 0.052 0.065 0.085] 5

price = X_train["price"]
pairs = zip(runs["logistic"]["estimator"], strat.split(X_train, y_train))
for model, (fit_pos, val_pos) in pairs:
    # the scaler after the imputer, the price its sixth column
    scaler = model[0].named_transformers_["num"][-1]
    print(
        round(scaler.mean_[5]),
        round(price.iloc[fit_pos].mean()),
        round(price.iloc[val_pos].mean()),
    )
# Each fold's scaler learned the mean price of exactly its own 1,920
# fit rows, and none learned its validation rows' mean.

377793 377793 389419
380890 380890 377033
381465 381465 374733
379809 379809 381354
380635 380635 378052

Theory Assessment

Q1: Eleven Carves and Five Folds

On ten carves of the training rows other than Lesson 3.5.1's, the leaf size that lesson chose never won. Explain what that says about a setting chosen on one carve. Then explain what each position of np.bincount's output means for the five folds, [ 0 2400], and for five carves of a fifth each, [ 773 1010 478 122 17], and why the linear pipeline's cross-validated score is reported as 0.794 with 0.052 beside it rather than as 0.794 alone.

Hint: Ask what changed between the eleven runs of the leaf table. For the counts, read the value at position 0 as the number of rows that validated zero times.

Q2: Same Rows, Two Orders

cv=5 scored the linear pipeline from -6.787 to -0.671 on the training rows sorted by price and a mean of 0.792 on the same rows as the split left them, while cv=kfold scored the sorted rows from 0.754 to 0.83. Explain what check_cv(5) showed that the integer builds, what each integer fold held on the sorted rows, why the second fold's R² could fall to -6.787, and how the rule that every fold must be as new to the model as the rows it will meet in use decides between the two splitters.

Hint: Read the folds' lowest and highest prices from the printed lists. For R², recall what Lesson 3.1.4 measured it against, and for the rule, set each kind of fold beside the test rows' range.

Q3: A Negative Miss, Five Medians and a Warning

Explain why cross_validate returned -56,682 for the first fold's mean absolute error and how a report states it. Explain why the five fitted imputers learned five area medians from 89.0 to 90.1 and none of them 89.7, and what that shows about the validation folds. Then explain why StratifiedKFold warned and went on when handed the price, why it refused the log of the price, and what a price model's folds are stratified on instead.

Hint: A scorer has one direction for better. For the medians, ask which rows each copy was fitted on. For the warning, ask what makes a column a set of classes.

Q4: One Splitter, Two Calls

A teammate compares the forest with the linear pipeline in compare.py:

from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from price_models import X, y, fill, make_prep

splitter = KFold(n_splits=5, shuffle=True)
linear = make_pipeline(make_prep(fill()), LinearRegression())
forest = make_pipeline(
    make_prep("passthrough"),
    RandomForestRegressor(max_features=0.5, random_state=42),
)
linear_scores = cross_val_score(linear, X, y, cv=splitter)
forest_scores = cross_val_score(forest, X, y, cv=splitter)
print((forest_scores > linear_scores).sum())
print(round(forest_scores.mean() - linear_scores.mean(), 3))

The review says: "One splitter served both calls, so the two models met the same five folds. It shuffles, so the order of the rows cannot shape the folds. And the folds cover all 3,000 listings, so no listing is wasted." Two runs of the script need not print the same two numbers, while the Theory's comparison printed a lead on 4 folds every time it ran. Say which of the three claims hold, correct the ones that do not, explain why the runs can differ, and rewrite the lines that need it.

Hint: Ask what random_state=None means for a splitter's second split call. Then ask which rows X holds that X_train does not.

import numpy as np
from sklearn.model_selection import KFold
from price_models import X, X_test

loose = KFold(n_splits=5, shuffle=True)
fixed = KFold(n_splits=5, shuffle=True, random_state=42)
for splitter in (loose, fixed):
    first = [val_pos for fit_pos, val_pos in splitter.split(X)]
    again = [val_pos for fit_pos, val_pos in splitter.split(X)]
    print(np.array_equal(first[0], again[0]))
print(X.index.isin(X_test.index).sum())

False
True
600

from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from price_models import X_train, y_train, fill, make_prep

splitter = KFold(n_splits=5, shuffle=True, random_state=42)
linear = make_pipeline(make_prep(fill()), LinearRegression())
forest = make_pipeline(
    make_prep("passthrough"),
    RandomForestRegressor(max_features=0.5, random_state=42),
)
linear_scores = cross_val_score(linear, X_train, y_train, cv=splitter)
forest_scores = cross_val_score(forest, X_train, y_train, cv=splitter)
print((forest_scores > linear_scores).sum())
print(round(forest_scores.mean() - linear_scores.mean(), 3))

4
0.019

Assignment

Assignment: Out-of-Fold Errors for the Pricing Review

Before the team chooses between the linear pipeline and Lesson 3.5.2's forest, the pricing review asks where in the range of prices each model misses most, and the review may not touch the test rows. Two tools this lesson has not used answer it. cross_val_predict, from sklearn.model_selection, takes the same model, rows, targets and cv as cross_val_score and returns one prediction for every row, each an out-of-fold prediction (made by the fit whose rows did not include that row). And cv accepts, in place of a splitter, a list of the pairs of fit positions and validation positions a splitter's split yields, so folds cut by stratifying on one column can score a model of another.

Write price_review.py. Cut five folds of the 2,400 training rows with StratifiedKFold(n_splits=5, shuffle=True, random_state=42) handed the q1 to q4 quartile band of the training prices, keep the pairs in a list, and print how many listings of each band every fold holds, then the same counts for the Theory's shuffled KFold. On the band's folds, cross-validate the linear pipeline and the forest with cross_validate, keeping each fold's fitted copy, and print both models' five R² values, the forest's gain on each fold and the number of folds it leads on. Collect each model's out-of-fold predictions with cross_val_predict on the same folds, prove for the first fold that they are exactly what that fold's fitted copy predicts, and print each model's mean absolute out-of-fold error in each band and over all 2,400 listings. End with a comment saying in which bands the forest misses less, by how much, and what the review may and may not conclude.

Hints:

• Hand cross_validate and cross_val_predict the list, never strat.split(...) itself: split returns a generator (an object that hands out its folds one at a time, once), and one call uses it up.
• pd.DataFrame({"band": band}) lines the band up with the training rows by label, a column of absolute errors joins it the same way, and Lesson 2.6.2's groupby("band") then gives each band's mean.

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import (
    KFold, StratifiedKFold, cross_val_predict, cross_validate,
)
from sklearn.pipeline import make_pipeline
from price_models import X_train, y_train, fill, make_prep

band = pd.qcut(y_train, q=4, labels=["q1", "q2", "q3", "q4"])
strat = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
folds = list(strat.split(X_train, band))  # reusable pairs
for splitter in (folds, kfold.split(X_train)):
    print([
        band.iloc[val_pos].value_counts(sort=False).tolist()
        for fit_pos, val_pos in splitter
    ])

[[120, 121, 119, 120], [120, 120, 120, 120], [120, 120, 120, 120], [121, 121, 119, 119], [121, 121, 119, 119]]
[[125, 111, 125, 119], [110, 131, 119, 120], [121, 130, 111, 118], [126, 108, 125, 121], [120, 123, 117, 120]]

models = {
    "linear": make_pipeline(make_prep(fill()), LinearRegression()),
    "forest": make_pipeline(
        make_prep("passthrough"),
        RandomForestRegressor(max_features=0.5, random_state=42),
    ),
}
r2 = {}
errors = pd.DataFrame({"band": band})
for name, model in models.items():
    run = cross_validate(
        model, X_train, y_train, cv=folds, return_estimator=True
    )
    r2[name] = run["test_score"]  # one scorer, so one key
    oof = cross_val_predict(model, X_train, y_train, cv=folds)
    fit_pos, val_pos = folds[0]
    first = run["estimator"][0].predict(X_train.iloc[val_pos])
    print(name, np.array_equal(oof[val_pos], first))
    errors[name] = np.abs(y_train - oof)

gain = r2["forest"] - r2["linear"]
print(r2["linear"].round(3), r2["forest"].round(3))
print(gain.round(3), (gain > 0).sum())
print(errors.groupby("band").mean().round().astype(int))
print(errors[["linear", "forest"]].mean().round().astype(int).tolist())
# The forest misses less in the three cheaper bands, by about 8,300,
# 6,850 and 4,550, and more in the dearest, by about 3,860. It led on
# all five folds. The review may say where each model misses on
# listings its fits never held. It may not call the lead more than
# luck, Lesson 3.7.1's question, or report any of these as the score
# on new listings, which the test rows give once, after the choice.

linear True
forest True
[0.779 0.835 0.789 0.812 0.765] [0.806 0.837 0.815 0.815 0.783]
[0.027 0.002 0.026 0.003 0.018] 5
      linear  forest
band                
q1     42729   34423
q2     43584   36732
q3     51808   47259
q4     88395   92258
[56581, 52607]

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want to know how many folds to choose, or what out-of-fold predictions are good for.

• YouTube - "10.3 K-fold CV for Model Evaluation (L10: Model Evaluation 3)" by Sebastian Raschka. k-fold cross-validation as a way to evaluate a model, and why 10 folds is a good choice set against 2 and 5 folds and against leave-one-out cross-validation (as many folds as rows), a lecture from his introductory machine learning course.
• Official docs - scikit-learn 1.9 example, "Plotting Cross-Validated Predictions". cross_val_predict with a linear regression on the diabetes data, its out-of-fold predictions plotted against the true values and as residuals, and the caution against reading one metric off the pooled predictions where per-fold scores from cross_val_score or cross_validate belong.

Summary

One validation carve answers for one set of listings, and the runs here kept showing how much rides on which set: the leaf size Lesson 3.5.1 chose won on none of ten other carves, one fold of the linear pipeline scored 0.144 below another, and the forest's lead held on four folds of one cut and two of another. Folds do not remove that luck. They spread it over every training row, once each, and put its size beside the score, so a number arrives with the margin it carries.

The rule under every fold is the rule under every held-out set: a fold stands in for listings that do not exist yet, so it must be as new to the model as they will be. Nothing announces a fold that breaks it. An integer cv followed the order the rows arrived in, and a sorted table turned five folds into five price ranges that scored like a broken model. The fills were the opposite case, kept honest by the structure rather than by care: a pipeline cross-validated whole learned every fill from four folds and never from the fifth.

Folds measure and stop there. They say how far a score moves when the listings change, and they set two models against the same listings so that a difference between them is not a difference between folds. A lead of 0.019 on four folds of one cut is still a measurement, and the test rows wait, untouched, for the one score the final choice is reported with.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

kfold = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X_train, y_train, cv=kfold)
scores.mean(), scores.std()  # report both
check_cv(5)                  # unshuffled KFold
results = cross_validate(
    model, X_train, y_train, cv=kfold,
    scoring=["r2", "neg_mean_absolute_error"],
    return_estimator=True,
)
mae = -results["test_neg_mean_absolute_error"]
results["estimator"][0]      # fold 0's fitted copy

strat = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
cross_val_score(
    clf, X_train, y_label, cv=strat, scoring="average_precision"
)
band = pd.qcut(price, q=4)   # stratify a price on this
folds = list(strat.split(X_train, band))
gain = cross_val_score(b, X_train, y_train, cv=kfold) - scores
(gain > 0).sum()             # folds where b leads