The Split, and What a Held-Out Set Is For

Context

A split divides a table's rows in two before any model sees them: the training rows, which the model is fitted on, and a held-out set, which the model is scored on and never fitted to. The held-out score stands in for the error on listings the model will price after it ships.

Every number reported about a model comes out of this step. The average miss of a pricing model in a weekly report, the comparison of two feature sets, the check that a retrained model beats the one in use, each is computed on listings the fit never saw, and Section 3.2's leakage-safe pipelines and Section 3.6's cross-validation are both ways of keeping those listings unseen.

Scored on its own training rows, a model that only remembers its listings scores nearly perfectly. A split unpacked in the wrong order fits on the held-out rows and reports a score that looks like any other, a mean computed from every row before the split carries the held-out rows into the fit, and one score from a small table moves by more than 20,000 across ten random splits. Lesson 3.1.1 fitted LinearRegression and priced listings it never saw, and this lesson measures how far off those prices are.

Goal

• Build a memorizer in the estimator shape and measure its error on its training rows and on held-out rows beside LinearRegression's.
• Split features and target with train_test_split, unpack the four results in the right order, and check the split by its shapes.
• Distinguish .loc[0] from .iloc[0] on a split frame, and pair predictions with prices by position.
• Apply the held-out rule: fit on the training rows alone, choose nothing by the test score, and identify a number that has already seen the held-out rows.
• Stratify a split on a pd.qcut price band and verify each band's share on both sides.
• Measure how a held-out score moves across random_state values, and explain why a small table moves more.

Prerequisites

Neither is required, the lesson stands on its own. They are here if you want the picture before the detail.

• YouTube - "Machine Learning Fundamentals: Bias and Variance" by StatQuest with Josh Starmer. A straight line and a squiggly line fitted to the same training points, then measured on points neither was fitted to, where the line that passed through every training point does worse.
• Official docs - scikit-learn 1.9 API reference, train_test_split. The parameters the function takes, test_size, random_state, shuffle and stratify among them, and the list of arrays it returns.

Table of Contents

• Why the Training Score Lies
• train_test_split
• What a Held-Out Set Must Be
• Stratifying the Split
• One Split Is One Draw

Theory

The blocks run in ~/dev/bootcamp on listings.csv, with (bootcamp) active. A block that opens with its imports is a script of its own, a block that does not continues the script above it, and a # raises comment marks a line that stops the script, so delete that line once you have seen its error.

Every model in the Theory works on two columns, area_sqm and distance_km (the listing's distance from the city center, in kilometers), with price as the target. The 125 listings with no area are dropped with Lesson 3.1.1's dropna, because LinearRegression refuses a missing value, which leaves 2,875. Every split takes test_size=0.2 and random_state=42 unless the text says why not.

Why the Training Score Lies

What. A training score is a model's error measured on the rows it was fitted to. It says how well the model reproduces listings it has already seen, and a model can reproduce nearly every one of them while knowing nothing that carries over to a listing it has not. Why. Every listing a pricing model prices in use is one it was not fitted on, so the error worth knowing is the error on such listings. A training score cannot tell a model that found a pattern from one that remembered its rows, and the one that remembered scores better. When. Every time a model gets a number, before anyone reads it: ask which rows the number came from.

Lesson 2.3.4's Assignment left the sentence this lesson starts from: a smaller sum of squared residuals on the listings a model was fitted to says nothing yet about new ones. The score here is the mean absolute error (the average size of a miss, in currency), np.abs(y - predicted).mean(), and Lesson 3.1.4 takes it apart with the other ways of scoring a model.

The model that shows the problem at full size is a memorizer in the shape of Lesson 3.1.1's estimators. Its fit stores every training listing's price in a dictionary keyed by that listing's feature values, and its predict looks each listing up and falls back to the training mean for a listing it never stored. Save it as memorizer.py in ~/dev/bootcamp, so every script of the lesson can import it:

import numpy as np


class Memorizer:
    def fit(self, X, y):
        self.prices_ = {}
        for row, price in zip(X.to_numpy(), y.to_numpy()):
            # a NumPy row cannot be a dictionary key, a tuple can
            self.prices_[tuple(row)] = price
        self.mean_ = y.mean()
        return self

    def predict(self, X):
        # a key it stored returns the price stored under it,
        # any other row returns the training mean
        return np.array([
            self.prices_.get(tuple(row), self.mean_)
            for row in X.to_numpy()
        ])

train_test_split sets a random fifth of the listings aside before the fit, and the next subtitle takes it apart. The memorizer is fitted on the other four fifths and scored on both parts:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from memorizer import Memorizer

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
X = listings[["area_sqm", "distance_km"]]
y = listings["price"]
# a random fifth held out, the next subtitle's subject
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

memorizer = Memorizer().fit(X_train, y_train)
train_mae = np.abs(y_train - memorizer.predict(X_train)).mean()
test_mae = np.abs(y_test - memorizer.predict(X_test)).mean()
print(round(train_mae), round(test_mae))  # 671 138260

On its own training rows the memorizer misses by 671 on average, and not by exactly 0 only because 25 training listings share both their area and their distance with a training listing stored after them, at a different price, and a dictionary keeps one price per key, the last one stored. On the 575 held-out listings it misses by 138,260. What it found there, and what no model at all would have scored:

found = [tuple(row) in memorizer.prices_ for row in X_test.to_numpy()]
mean_mae = np.abs(y_test - y_train.mean()).mean()
print(len(memorizer.prices_), sum(found))  # 2275 18
print(round(mean_mae))                     # 139562

The dictionary holds 2,275 prices. Of the held-out listings it found 18, each one a different listing that happens to share an area and a distance with a stored one, so each got a stranger's price, and the other 557 got the training mean. Predicting the training mean for every held-out listing, with no model at all, misses by 139,562, so a training error of 671 belongs to a model barely better than nothing on listings it has not seen. LinearRegression on the same rows:

model = LinearRegression().fit(X_train, y_train)
train_mae = np.abs(y_train - model.predict(X_train)).mean()
test_mae = np.abs(y_test - model.predict(X_test)).mean()
print(round(train_mae), round(test_mae))  # 88648 83663

LinearRegression keeps three numbers from the fit, a weight for each column and an intercept, and three numbers cannot store 2,300 prices, so whatever it learned is a pattern the listings share, and its two errors sit close together. On this split the held-out error even comes out below the training error, because of which listings landed in the held-out fifth, and the last subtitle measures how much that moves. A held-out error close to the training error is evidence that the training score was not bought by remembering. A held-out error far above it, 138,260 against 671, is the mark of a model that learned its rows instead of a pattern.

DON'T:

• DON'T report a score computed on the training rows. The memorizer's 671 came from the model that priced new listings barely better than the training mean did.
• DON'T expect the held-out error to come out above the training error every time. LinearRegression missed the held-out listings by 83,663 and its own training rows by 88,648, and which listings were held out decided the order.

train_test_split

What. train_test_split, from sklearn.model_selection, shuffles the rows and cuts every array it is given at the same place, returning each array's training part and then its held-out part, array by array. Given X and y, that is four objects in the fixed order X_train, X_test, y_train, y_test. Why. A listing's features and its price must land on the same side. One call cuts X and y with one shuffle, so the nth row of X_train and the nth value of y_train are the same listing. When. Before the first fit of any model you will score, and before anything is computed from the rows, which the next subtitle explains.

test_size=0.2 holds out a fifth of the rows, 575 of 2,875, and leaving it out holds out a quarter, 719. random_state=42 fixes the shuffle, so every run holds out the same listings. The shuffle is on by default, shuffle=True, so both parts come out in a new order, and the two parts of X and y keep the same labels in the same order:

print(X_train.shape, X_test.shape)          # (2300, 2) (575, 2)
print(y_train.shape, y_test.shape)          # (2300,) (575,)
print(X_train.index.equals(y_train.index))  # True
# Index([1653, 1240, 632, 2674, 639], dtype='int64')
print(X_train.index[:5])

The shapes are the check worth printing after every split: 2,300 and 575 rows, X two-dimensional and y one-dimensional. Given one table, the call returns two. Without stratify=, the shuffle depends on the number of rows and the seed alone, so the same seed on a table of the same length holds out the same listings whatever columns are passed:

train_rows, test_rows = train_test_split(
    listings, test_size=0.2, random_state=42
)
print(train_rows.shape, test_rows.shape)  # (2300, 13) (575, 13)
# True
print(train_rows.index.equals(X_train.index))

Without random_state, every call shuffles differently. Each call returns a list, and [1] is that call's X_test:

one = train_test_split(X, y, test_size=0.2)
two = train_test_split(X, y, test_size=0.2)
print(one[1].index.equals(two[1].index))  # False

A score from a split with no seed cannot be reproduced, by you tomorrow or by a reviewer.

The returned frames keep their original index labels, because the shuffle moved the rows and not the labels. .iloc[0] is the first training row, and .loc[0] is the listing labeled 0, wherever it landed:

print(X_train.iloc[0])
print(X_train.loc[0])
print(X_test.loc[0])  # raises KeyError: 0

area_sqm       71.7
distance_km     2.0
Name: 1653, dtype: float64
area_sqm       175.7
distance_km      9.6
Name: 0, dtype: float64

Listing 0 landed among the training rows, so X_train.loc[0] returned it without a word, the 175.7 square meter listing at 9.6 kilometers, and not the first training row, listing 1653. On X_test, which listing 0 is not in, the same line raises. That is Lesson 2.4.2's labels against positions, met where Lesson 2.4.2 said it would be. A prediction array carries positions and no labels, so y_test - model.predict(X_test) pairs each price with the prediction at the same position, which is the order X_test holds:

predicted = model.predict(X_test)
print(y_test.iloc[0], round(predicted[0]))  # 342000 445889

Python hands the four objects to the four names by position and never reads the names. Of the 24 orders four names can take, two run to the end, the right one and the one that swaps the two sides:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
X = listings[["area_sqm", "distance_km"]]
y = listings["price"]

# the sides swapped, X still with X and y with y
X_test, X_train, y_test, y_train = train_test_split(
    X, y, test_size=0.2, random_state=42
)
model = LinearRegression().fit(X_train, y_train)
test_mae = np.abs(y_test - model.predict(X_test)).mean()
print(X_train.shape, round(test_mae))  # (575, 2) 87812

The model was fitted on the 575 listings meant for testing and scored on the 2,300 meant for training, and 87,812 looks like any other score. Every other wrong order stops with an error at the fit or the predict. Grouped by side, the order gives the name y_train the held-out part of X:

# grouped by side
X_train, y_train, X_test, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
try:
    LinearRegression().fit(X_train, y_train)
except ValueError as err:
    print(err)

Found input variables with inconsistent numbers of samples: [2300, 575]

Printing the four shapes right after the split catches both.

DON'T:

• DON'T unpack the four results in any order but X_train, X_test, y_train, y_test. The swapped sides ran to the end on a model fitted to 575 listings, and only the shapes said so.
• DON'T use .loc with a number to mean a position on a split frame. X_train.loc[0] returned listing 0 from the middle of the training rows, and X_test.loc[0] raised KeyError: 0.
• DON'T leave out random_state. Two calls without it held out different listings, so a change in the score between two runs mixes the change you made with a change of held-out listings.
• DON'T leave out test_size and expect a fifth. The default holds out a quarter, 719 of 2,875 listings.

What a Held-Out Set Must Be

What. A held-out set is the rows the split sets aside, and one rule binds it: a held-out row must be as new to the model as the rows it will meet in use, so nothing about it may reach the fit, or any decision about the model made before the fit. Why. The held-out score is worth something only as a stand-in for the error on listings that do not exist yet while the model is built, and those reach no step of the work. A held-out row that reached the fit, a choice or a number the model uses stands in for nothing, and its score prints exactly like an honest one. When. At every step from loading the table to reporting the score. The order that keeps the rule is split first, fit on the training rows, and score the held-out rows once, at the end.

The plainest break is fitting before splitting. Here both models are fitted on all 2,875 listings and scored on the same 575 the first subtitle held out:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from memorizer import Memorizer

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
X = listings[["area_sqm", "distance_km"]]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# fitted on every row, the 575 held-out ones included
memorizer = Memorizer().fit(X, y)
model = LinearRegression().fit(X, y)
for fitted in (memorizer, model):
    test_mae = np.abs(y_test - fitted.predict(X_test)).mean()
    print(round(test_mae))

1689
83388

The memorizer's held-out error fell from 138,260 to 1,689, a training score under a held-out name. LinearRegression moved from 83,663 to 83,388, because a model that cannot store rows gains little from seeing them. Nothing in either number says the rule was broken. The size of a break depends on the model, and only the order of the work rules it out.

The second break is choosing by the test score. Every decision about the model, which features, which model, which setting, changes the number the test set will print, and a decision made by looking at that number turns the test set into training data: the model was chosen to fit it. The choosing is what a validation set is for, and the test set is scored once, after the last choice. Section 3.6 makes those choices over many validation splits.

The third break is a number computed from every row before the split. Filling the 125 missing areas with the mean area of the whole table puts the held-out listings' areas into every gap the fill closes, the training rows' gaps included:

import pandas as pd
from sklearn.model_selection import train_test_split

raw = pd.read_csv("listings.csv")
train_rows, test_rows = train_test_split(
    raw, test_size=0.2, random_state=42
)
print(round(raw["area_sqm"].mean(), 1))  # 99.0
# 99.6
print(round(train_rows["area_sqm"].mean(), 1))

The mean of the whole table is 99.0 square meters and the training rows' alone is 99.6, and the difference is the held-out listings at work inside a number the model would be fitted on. Lesson 3.2.3 fills gaps from the training rows alone, inside a pipeline, and this lesson only names the hazard.

DON'T:

• DON'T fit before you split. The memorizer fitted on every row scored 1,689 on listings it had stored, a training score under a held-out name.
• DON'T choose anything by the test score, a feature set, a model, a setting or a seed. The choosing happens on a validation set, and the test set is scored once, after the last choice.
• DON'T compute a fill value, or any number a step of the model will use, from every row before the split. The mean area of the whole table, 99.0, already holds the held-out listings' areas.
• DON'T read a small change as proof that nothing leaked. LinearRegression moved by 275 when fitted on every row, and the memorizer moved by 136,571 under the same break.

Stratifying the Split

What. stratify= takes one category per row and keeps each category's share of the rows nearly the same in the training part and the held-out part. A price band from 2.6.1's pd.qcut turns the price into such a category. Why. A random fifth of a small table can hold the price bands in shares far from the training rows' shares, and the model is then scored on a different mix of prices from the one it was fitted on. When. On a small table, or when a category that matters is rare, so that a random split can tilt it. A large table's random split lands close without it.

The effect shows on a small table, so the blocks draw 200 listings with sample(n=200, random_state=42), which returns that many rows chosen at random, the same 200 every run. The band is 2.6.1's quartiles of the price, four bands of 50 listings each, and value_counts(normalize=True) returns each band's share of the rows instead of its count:

import pandas as pd
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
sample = listings.sample(n=200, random_state=42)
X = sample[["area_sqm", "distance_km"]]
y = sample["price"]
band = pd.qcut(y, q=4, labels=["q1", "q2", "q3", "q4"])
# {'q1': 50, 'q2': 50, 'q3': 50, 'q4': 50}
print(band.value_counts(sort=False).to_dict())


def band_shares(band, y_train, y_test):
    # each side's listings looked up in the one band
    return pd.DataFrame({
        "train": band.loc[y_train.index].value_counts(normalize=True),
        "test": band.loc[y_test.index].value_counts(normalize=True),
    }).round(3)


X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(band_shares(band, y_train, y_test))

       train   test
price              
q1     0.256  0.225
q2     0.275  0.150
q3     0.231  0.325
q4     0.238  0.300

Without stratify=, the second band is 15 percent of the 40 held-out listings and the third 32.5 percent, against 27.5 and 23.1 percent of the training rows. The model is fitted on one mix of prices and scored on another. With stratify=band:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=band
)
print(band_shares(band, y_train, y_test))

       train  test
price             
q1      0.25  0.25
q2      0.25  0.25
q3      0.25  0.25
q4      0.25  0.25

Every band holds exactly a quarter of both sides, 40 training listings and 10 held-out ones each. On all 2,875 listings the same unstratified split lands closer. One array in gives two out, so the price alone is split:

full_band = pd.qcut(listings["price"], q=4, labels=["q1", "q2", "q3", "q4"])
y_train, y_test = train_test_split(
    listings["price"], test_size=0.2, random_state=42
)
print(band_shares(full_band, y_train, y_test))

       train   test
price              
q1     0.257  0.223
q2     0.246  0.277
q3     0.252  0.237
q4     0.245  0.264

The held-out shares of the full table sit between 22.3 and 27.7 percent, against 15 to 32.5 in the sample. The band reads every price, the held-out listings' included, and the held-out rule still holds, because the band is a decision about the split and not about the model: it decides which listings are held out, it is never a column the model is fitted on, and the fit sees the training rows as they are.

stratify= reads every distinct value as a category of its own, and a category needs at least two rows, one for each side. The raw price is refused, and the message goes on to list every price that occurs once, so the block prints its first two sentences:

try:
    train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)
except ValueError as err:
    print(str(err).split(" Classes")[0])
print((y.value_counts() == 1).sum())  # 136

The least populated classes in y have only 1 member, which is too few. The minimum number of groups for any class cannot be less than 2.
136

136 of the sample's 200 prices occur once. A continuous column makes most of its values a category of one, and a band from pd.qcut gives each category enough rows.

DON'T:

• DON'T pass a continuous column as stratify=. Cut it into bands with pd.qcut first, so every category holds rows enough for both sides.
• DON'T check the shares by cutting each side into quartiles of its own. pd.qcut on the training prices alone puts about a quarter of them in every band whatever the split did, so look each side's listings up in the one band, with .loc and their labels.

One Split Is One Draw

What. A held-out score depends on which listings the split held out, so the same model on the same table prints a different score for every random_state. One score is one draw from that spread. Why. The smaller the table, the wider that spread. Two models scored on the same split move together with the seed, so they are compared by the gap between them on each split, and a gap that keeps its sign from seed to seed was not decided by which listings a split held out, whether or not it beats one model's own spread. When. Before comparing two scores, and before reporting one from a small table.

Here the seed changes on purpose, 0 to 9, on the 200-listing sample and on the full table, with everything else held fixed:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
sample = listings.sample(n=200, random_state=42)
features = ["area_sqm", "distance_km"]


def held_out_mae(table, seed):
    X_train, X_test, y_train, y_test = train_test_split(
        table[features], table["price"], test_size=0.2, random_state=seed
    )
    model = LinearRegression().fit(X_train, y_train)
    return round(np.abs(y_test - model.predict(X_test)).mean())


for table in (sample, listings):
    maes = [held_out_mae(table, seed) for seed in range(10)]
    print(maes, min(maes), max(maes))

[92159, 74669, 71837, 80967, 80844, 80403, 77224, 87727, 83845, 68983] 68983 92159
[87261, 87305, 89361, 84862, 87848, 82515, 86980, 86967, 88859, 86040] 82515 89361

The sample's held-out set is 40 listings, and ten seeds moved its score from 68,983 to 92,159, a spread of 23,176. The full table's held-out set is 575 listings, and the same ten seeds moved its score by 6,846. Reporting 68,983 because it was the smallest of the ten is a choice made by the test score, the second break of What a Held-Out Set Must Be, with the seed as the setting. Averaging the score over many splits is Lesson 3.6.2's cross-validation.

Comparing two models reads their gap on each split. Area alone against area and distance, on 20 seeds of each table, the gap being area alone's error minus the other's:

def mae_gap(table, seed):
    X_train, X_test, y_train, y_test = train_test_split(
        table[features], table["price"], test_size=0.2, random_state=seed
    )
    errors = []
    for columns in (["area_sqm"], features):
        model = LinearRegression().fit(X_train[columns], y_train)
        predicted = model.predict(X_test[columns])
        errors.append(np.abs(y_test - predicted).mean())
    # above zero when area and distance missed by less
    return round(errors[0] - errors[1])


for table in (sample, listings):
    gaps = [mae_gap(table, seed) for seed in range(20)]
    print(min(gaps), max(gaps), sum([gap > 0 for gap in gaps]))

-8030 13024 15
2997 7691 20

On the sample the gap ran from -8,030 to 13,024, and area and distance missed by less on 15 of the 20 splits, so one split of the sample could name either model the better. On the full table the smallest gap, 2,997, is less than half the 6,846 the seed moved one model by, and area and distance won all 20 splits: the gap kept its sign, so which listings a split held out did not decide the winner.

DON'T:

• DON'T read one held-out score as the model's error. It is one draw, and on the sample ten draws spread over 23,176.
• DON'T try seeds until one scores well. The smallest of ten, 68,983, is the luckiest draw and not the model.
• DON'T compare two models scored on different splits. With the same random_state, test_size and stratify= on the same table, both are scored on the same held-out listings, so their gap is not partly a difference between two sets of listings.

Lab Exercises

Lab Exercise 1: A Memorizer on One Column

In one_column.py, load listings.csv, drop the listings with no area, and split with area_sqm alone as the features and price as the target.

Print the four shapes. Then prove in code that no listing sits on both sides and that the two sides together hold every listing.

Fit the Memorizer from memorizer.py and a LinearRegression on the training rows, and print each one's error on its training rows and on the held-out rows. Print how many prices the memorizer's dictionary holds and how many held-out listings it found in it, and say in a comment why its training error is far from 0 this time.

Finally, print the first held-out listing's area, its price and LinearRegression's prediction for it, each one taken by position, and say in a comment what y_test.loc[0] does and why.

Lab Exercise 2: Shares by Neighborhood, Spread by Seed

In neighborhood_split.py, draw the Theory's 200-listing sample from the listings that have an area.

Split the whole sample twice, once without stratify= and once with stratify=sample["neighborhood"], and for each split print every neighborhood's share of the training rows and of the held-out rows side by side, rounded to three decimals.

Then build the Theory's q1 to q4 price band on the sample. For the seeds 0 to 19, compute LinearRegression's held-out error on area_sqm and distance_km, once without stratify= and once with the band, and print the smallest and the largest error of each. Say in a comment whether stratifying on the band narrowed the spread, and why it could not remove it.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from memorizer import Memorizer

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
X = listings[["area_sqm"]]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(X_train.shape, X_test.shape)  # (2300, 1) (575, 1)
print(y_train.shape, y_test.shape)  # (2300,) (575,)
# 0, no label on both sides
print(X_train.index.isin(X_test.index).sum())
# True, the two sides hold every listing
print(len(X_train) + len(X_test) == len(listings))

memorizer = Memorizer().fit(X_train, y_train)
model = LinearRegression().fit(X_train, y_train)
for name, fitted in {"memorizer": memorizer, "linear": model}.items():
    train_mae = np.abs(y_train - fitted.predict(X_train)).mean()
    test_mae = np.abs(y_test - fitted.predict(X_test)).mean()
    print(name, round(train_mae), round(test_mae))

found = [tuple(row) in memorizer.prices_ for row in X_test.to_numpy()]
print(len(memorizer.prices_), sum(found))
# 2,300 training listings share 1,182 areas, and a key keeps the
# last price stored under it, so 1,117 training rows get back a
# price other than their own.

memorizer 57481 133105
linear 93898 87952
1182 427

predicted = model.predict(X_test)
print(X_test.iloc[0, 0], y_test.iloc[0])  # 123.2 342000
print(round(predicted[0]))                # 461717
# y_test.loc[0] raises KeyError: 0, listing 0 is a training row

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
sample = listings.sample(n=200, random_state=42)

for stratify in (None, sample["neighborhood"]):
    train_rows, test_rows = train_test_split(
        sample, test_size=0.2, random_state=42, stratify=stratify
    )
    shares = pd.DataFrame({
        "train": train_rows["neighborhood"].value_counts(normalize=True),
        "test": test_rows["neighborhood"].value_counts(normalize=True),
    })
    print(shares.round(3))

              train   test
neighborhood              
Downtown      0.219  0.150
Eastgate      0.119  0.125
Hillcrest     0.112  0.150
Lakeview      0.100  0.125
Northend      0.106  0.150
Riverside     0.169  0.150
Southgate     0.175  0.150
              train   test
neighborhood              
Downtown      0.206  0.200
Eastgate      0.119  0.125
Hillcrest     0.119  0.125
Lakeview      0.106  0.100
Northend      0.112  0.125
Riverside     0.169  0.150
Southgate     0.169  0.175

band = pd.qcut(sample["price"], q=4, labels=["q1", "q2", "q3", "q4"])
features = ["area_sqm", "distance_km"]
for stratify in (None, band):
    maes = []
    for seed in range(20):
        train_rows, test_rows = train_test_split(
            sample, test_size=0.2, random_state=seed, stratify=stratify
        )
        model = LinearRegression().fit(
            train_rows[features], train_rows["price"]
        )
        predicted = model.predict(test_rows[features])
        maes.append(round(np.abs(test_rows["price"] - predicted).mean()))
    print(min(maes), max(maes))
# Stratifying did not narrow the spread. The band decides how many
# listings of each quarter are held out, never which ones, and
# which ones moves the error.

68983 92159
64521 101136

Theory Assessment

Q1: Two Pairs of Numbers

The memorizer scored 671 on its training rows and 138,260 on the held-out rows, and LinearRegression scored 88,648 and 83,663 on the same rows. Explain what each pair says about the model it came from, why the memorizer's training error is not exactly 0, and why LinearRegression cannot remember its training rows the way the memorizer does.

Hint: Count what each model keeps from the fit. Then ask what a dictionary does when a key it already holds is stored again.

Q2: Four Names, One Order

Explain why X_test, X_train, y_test, y_train = train_test_split(X, y, test_size=0.2, random_state=42) runs to the end while X_train, y_train, X_test, y_test = train_test_split(X, y, test_size=0.2, random_state=42) stops at the fit, and say what the first one fitted the model on. Then explain why X_train.loc[0] returned a row while X_test.loc[0] raised, and name the accessor that returns the first training row.

Hint: Python matches values to names by position. Look at the lengths of the two objects the fit receives in each case, and at which labels each side kept.

Q3: A Band, a Raw Price and Ten Seeds

Explain why stratify=y with the raw price is refused while stratify=band with a pd.qcut band works, and why the band, computed from every price in the sample, the held-out listings' included, does not break the held-out rule. Then say what the ten seeds' scores on the sample, 68,983 to 92,159, mean for a report that quotes one of them.

Hint: Ask how many rows a category needs for one to go to each side. Then ask what the band decides and what it never becomes.

Q4: A Score That Pairs Strangers

A teammate scores the model with this script:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
X = listings[["area_sqm", "distance_km"]]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
model = LinearRegression().fit(X_train, y_train)
predicted = pd.Series(model.predict(X_test))
errors = (y_test - predicted).abs()
print(round(errors.mean()))
print(len(errors), errors.notna().sum())

199642
1027 123

The same model misses these listings by 83,663 on average. Explain each printed number, why no line raised, and fix the line that produced them.

Hint: pd.Series gives its values labels of its own. Ask which labels y_test kept from the split, and what subtraction between two Series does with labels.

errors = (y_test - model.predict(X_test)).abs()
print(round(errors.mean()))               # 83663
print(len(errors), errors.notna().sum())  # 575 575

Assignment

Assignment: A Validation Set, Then One Test Score

The pricing team publishes one number for its next model, and four candidates are on the table: the Memorizer on area_sqm and distance_km, LinearRegression on area_sqm alone, LinearRegression on both columns, and LinearRegression(fit_intercept=False) on both columns, Lesson 3.1.1's setting that forces the fit through zero. Choosing among them may not touch the test set, so the choice is made on a validation set, and the validation set comes from a second train_test_split call on the rows the first call kept: the first call holds out 20 percent of the table as the test set, and the second holds out a quarter of the remaining 80 percent, another 20 percent of the table, as the validation set, which leaves 60 percent to train on.

Write choose.py. Split the listings that have an area into training, validation and test rows that way, both calls with random_state=42 and both stratified on the q1 to q4 quartile band of all 2,875 prices. Print the three sizes, prove in one check that no listing sits in two of the sets and that together they hold every listing, and print each band's share of all three sets, rounded to three decimals. Fit every candidate on the training rows, print each one's validation error, and take the smallest. Then score the chosen candidate on the test set, once, and print its validation error beside its test error, with a comment saying which of the two the team publishes and why.

Hints:

• Split whole tables. train_test_split(listings, ...) returns two tables, every candidate takes its own columns from them, and the second call's stratify= takes the band of the rows it splits, in their order, which .loc and the first call's labels give.
• Stack the three sets with 2.6.3's pd.concat. A stack whose labels are unique and whose length is the table's proves both things at once.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from memorizer import Memorizer

listings = pd.read_csv("listings.csv").dropna(subset=["area_sqm"])
band = pd.qcut(listings["price"], q=4, labels=["q1", "q2", "q3", "q4"])

# the test set first, 20 percent of every band
rest, test = train_test_split(
    listings, test_size=0.2, random_state=42, stratify=band
)
# a quarter of the remaining 80 percent is 20 percent of the table
train, val = train_test_split(
    rest, test_size=0.25, random_state=42, stratify=band.loc[rest.index]
)
print(len(train), len(val), len(test))

together = pd.concat([train, val, test])
print(together.index.is_unique, len(together) == len(listings))
shares = pd.DataFrame({
    "train": band.loc[train.index].value_counts(normalize=True),
    "val": band.loc[val.index].value_counts(normalize=True),
    "test": band.loc[test.index].value_counts(normalize=True),
})
print(shares.round(3))

1725 575 575
True True
       train    val   test
price                     
q1     0.250  0.250  0.250
q2     0.252  0.252  0.252
q3     0.249  0.249  0.249
q4     0.249  0.249  0.249

both = ["area_sqm", "distance_km"]
candidates = {
    "memorizer": (Memorizer(), both),
    "area": (LinearRegression(), ["area_sqm"]),
    "area_distance": (LinearRegression(), both),
    "no_intercept": (LinearRegression(fit_intercept=False), both),
}
val_maes = {}
for name, (model, columns) in candidates.items():
    model.fit(train[columns], train["price"])
    predicted = model.predict(val[columns])
    val_maes[name] = round(np.abs(val["price"] - predicted).mean())
print(val_maes)

# tuples compare item by item, so the smallest error wins
best_mae, chosen = min([(mae, name) for name, mae in val_maes.items()])
model, columns = candidates[chosen]
test_mae = np.abs(test["price"] - model.predict(test[columns])).mean()
print(chosen, best_mae, round(test_mae))
# Publish the test error. The validation error was picked for being
# the smallest of four, and the test set reached no step before this.

{'memorizer': 147286, 'area': 89488, 'area_distance': 86394, 'no_intercept': 87539}
area_distance 86394 87062

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want the step after one split.

• YouTube - "Machine Learning Fundamentals: Cross Validation" by StatQuest with Josh Starmer. The data cut into blocks, each block held out in turn while the rest train, and the results combined, instead of one split's one draw.
• Official docs - scikit-learn 1.9 Glossary, random_state. What None, an integer and a RandomState instance each do across scikit-learn, and the advice to check that a result holds over several seeds.

Summary

A model's score is a claim about listings that do not exist yet, and every block in this lesson either kept that claim honest or broke it, most of them without a sound. The memorizer is the extreme that makes the rest readable: nearly perfect on what it has seen, barely better than the training mean on what it has not, and its training score could not tell the two apart. Every real model sits somewhere between it and a model that keeps only a pattern, and the distance between its training error and its held-out error is where that position shows.

The rule under every break was one rule. A held-out row has to be as new to the model as the listings it will meet, and fitting before the split, choosing by the test score and computing a fill value from every row all break it the same way, by letting the held-out rows into a step that comes before the score. None of them leaves a trace in the number. A score printed by a script that broke the rule looks exactly like an honest one, which is why the protocol, and not the output, is what a reviewer reads.

An honest split is still one draw. Stratifying fixes the mix of a column you name and nothing else, the seed decides which listings answer for the model, and the spread from one seed to the next is the margin a single score carries, wide on 200 listings and narrower on 2,875. A score means what its rows and its protocol let it mean, and it is reported with both.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(X_train.shape, X_test.shape)  # check before anything
# fit on the training rows only
model = LinearRegression().fit(X_train, y_train)
# score the held-out rows once, by position
mae = np.abs(y_test - model.predict(X_test)).mean()
X_train.iloc[0]                     # first training row
X_train.loc[0]                      # listing 0, or KeyError

band = pd.qcut(y, q=4, labels=["q1", "q2", "q3", "q4"])
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=band
)
# each band's share of the held-out rows
band.loc[y_test.index].value_counts(normalize=True)
# never stratify=y on a continuous price