Pipelines, ColumnTransformer and Leakage-Safe Imputation

Context

An imputer fills each gap in a column with a value it learned from the training rows, and a pipeline holds every step from the raw table to the price, the fills, the encodings and the model, as one object that a single call fits, each step on exactly the rows that call receives.

A pricing model meets the listings table as it arrives, with areas, construction years and energy ratings missing and neighborhoods written as text. The pipeline is fitted on that table as it is and is the one object the team scores, saves and hands to the service that quotes prices, so a seller who leaves the area blank on the form still gets a price, filled from the same training medians the reported score was computed with.

Without it, the gaps are handled by hand, and by hand the work goes wrong, mostly without a sound. Lesson 3.1.1's dropna cost the fit listings and leaves a model that cannot price a listing posted with a gap, a fill computed from every row before the split carries the held-out listings into the training data, Lesson 3.1.2's third break, and a column no preparation step was told about vanishes without a warning. Lessons 3.2.1 and 3.2.2 fitted one transformer at a time on the training rows, and this lesson fills the gaps the same way and puts every step inside one object whose fit never reaches past the rows it is given.

Goal

• Fill numeric gaps with SimpleImputer(strategy="median") fitted on the training rows, read what it learned in statistics_, and fill a text column with strategy="most_frequent".
• Keep a was-missing column for the model with add_indicator=True or MissingIndicator, and identify which columns get one and which new gaps go unrecorded.
• Build a ColumnTransformer that fills, one-hot encodes and ordinal encodes different groups of columns, and find a column that remainder dropped.
• Construct a Pipeline of named steps from the raw table to a price, reach its fitted steps by name and position, and score it on the test rows.
• Explain precisely why fitting a pipeline fits every step again on exactly the rows fit received, and trace the held-out rows into an imputer fitted before the split.

Prerequisites

Neither is required, the lesson stands on its own. They are here if you want to see a pipeline run once, or read scikit-learn's own account of a transformer per column, before you build one.

• YouTube - "Use Pipeline to chain together multiple steps" by Data School. An imputer and a model chained into one pipeline, fitted on a four-row table and asked to predict three new rows, one of them with a gap, which the pipeline fills with the value the imputer learned during the fit. Its imputer fills with the mean, and its model is a classifier, the kind Section 3.4 fits.
• Official docs - scikit-learn 1.9 User Guide, "Pipelines and composite estimators", the section "ColumnTransformer for heterogeneous data". A different transformer for each column of a four-row table, the columns remainder drops unless told otherwise, and the feature names the result carries.

Table of Contents

• SimpleImputer, Fitted on the Training Rows
• The Missing-Value Indicator
• ColumnTransformer
• Pipeline
• Why the Pipeline Is Leakage-Safe

Theory

Nothing to install: every scikit-learn class this lesson uses arrived with Lesson 3.1.1's install. Each Theory block below goes into one script, pipeline.py, saved in ~/dev/bootcamp beside listings.csv and run with python3 pipeline.py, each block added under the one before it.

SimpleImputer, Fitted on the Training Rows

What. SimpleImputer, from sklearn.impute, is a transformer that fills the gaps in each column with one value for that column, learned in fit and kept in statistics_. strategy="median" learns each column's median, and strategy="most_frequent" its most common value, the one of the two that also takes text. Why. Lesson 3.1.1's dropna threw away listings the fit could have learned from, and it has no answer for a listing that arrives with a gap to be priced. A fill value goes into every training row with a gap in its column, so Lesson 3.1.2's held-out rule decides where it may come from, which settles the question Lesson 2.5.1 left open: the training rows, and only them. When. Whenever a feature has gaps and the model refuses them, as LinearRegression does, and before any step that needs the column complete.

The model in this lesson takes nine features, every column of listings.csv but four: price, the target, listing_id and listing_date, which this section never uses, and sold_in_30_days, Section 3.4's target. The split is Lesson 3.1.2's, on all 3,000 listings this time, because nothing is dropped:

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
nominal = ["neighborhood", "property_type", "energy_rating"]
features = numeric + nominal + ["condition"]
X = listings[features]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(X.shape, X_train.shape, X_test.shape)  # (3000, 9) (2400, 9) (600, 9)
gappy = ["area_sqm", "year_built", "energy_rating"]
print(X_train[gappy].isna().sum().tolist())  # [90, 110, 215]
print(X_test[gappy].isna().sum().tolist())   # [35, 30, 52]
kept_train = len(X_train.dropna())
kept_test = len(X_test.dropna())
print(kept_train, kept_test)                 # 2005 493

Three of the nine have gaps on both sides of the split, the area, the construction year and the energy rating. A bare dropna keeps 2,005 of the 2,400 training listings, and a model fitted on them still cannot price the 107 test listings that hold a gap: LinearRegression refuses a missing area or year in predict as it does in fit, and an encoder at its default, fitted on rows with every rating present, refuses a missing rating as an unknown category, as Lesson 3.2.2 showed. The imputer, fitted on the training rows:

pair = ["area_sqm", "year_built"]
imputer = SimpleImputer(strategy="median")
imputer.fit(X_train[pair])
print(imputer.statistics_.tolist())     # [89.7, 1992.0]
print(X_train[pair].median().tolist())  # [89.7, 1992.0]
print(X[pair].median().tolist())        # [89.2, 1992.0]
print(X_test[pair].median().tolist())   # [87.1, 1991.0]

statistics_ holds the training rows' medians, one per column, in the order the columns were fitted. Every row's median puts the area at 89.2, because the held-out listings' areas went into it. On the year, the training rows and every row print the same 1992.0, which makes the two numbers equal without making them the same number: one was computed without the held-out rows and one with them.

imputer.set_output(transform="pandas")
filled = imputer.transform(X_test[pair])
was_gap = X_test["area_sqm"].isna()
print(was_gap.sum())                         # 35
print(filled["area_sqm"][was_gap].unique())  # [89.7]
print(filled.isna().sum().sum())             # 0

set_output, Lesson 3.2.1's switch to a DataFrame, keeps the column names. transform filled all 35 test listings that have no area with the training median, 89.7, and not with their own rows' 87.1.

The energy rating is text, and a median has to order its values as numbers:

rating = SimpleImputer(strategy="most_frequent")
rating.fit(X_train[["energy_rating"]])
print(rating.statistics_.tolist())   # ['C']
default = SimpleImputer().fit(X_train[["area_sqm"]])
print(default.strategy)              # mean
print(default.statistics_.round(2))  # [99.57]

try:
    SimpleImputer(strategy="median").fit(X_train[["energy_rating"]])
except ValueError as err:
    print(err)

Cannot use median strategy with non-numeric data:
could not convert string to float: 'B'

The median strategy stopped on a rating it could not read as a number, B. most_frequent counts values instead of ordering them, so it fills a rating as readily as a year, here with C, the most common rating among the training listings. Left at its default, the imputer fills with the mean.

DON'T:

• DON'T leave strategy at its default without choosing it. SimpleImputer() fills with the mean, 99.57 square meters on the training rows, against a median of 89.7.
• DON'T ask for the median of a text column. The energy rating stops strategy="median" with Cannot use median strategy with non-numeric data:, and most_frequent fills it.
• DON'T drop the rows with a gap to get past the model's refusal. The model fitted on the 2,005 complete training listings still refuses the 107 test listings that hold one.

The Missing-Value Indicator

What. add_indicator=True makes SimpleImputer add one column for every column that had a gap in the rows it was fitted on, named missingindicator_ and that column's name, holding 1 where the value was missing and 0 where it was measured (True and False beside a text column). It is Lesson 2.5.1's was-missing column, built by the imputer as it fills. MissingIndicator, from the same module, builds those columns alone, as a step of its own. Why. After the fill, a filled 89.7 and a measured 89.7 are one number. When the listings with a gap differ from the rest, the indicator is the only column left that tells the model which ones they were. When. Whenever a gap may say something about the listing, and above all when the listings with a gap are not like the others.

flagged = SimpleImputer(strategy="median", add_indicator=True)
flagged.set_output(transform="pandas")
train_flagged = flagged.fit_transform(X_train[numeric])
print(train_flagged.columns.tolist()[5:])
gap_rows = X_train["area_sqm"].isna()
shown = ["area_sqm", "missingindicator_area_sqm"]
print(train_flagged.loc[gap_rows, shown].head(2))
print(train_flagged[shown[1]].sum())  # 90.0

['missingindicator_area_sqm', 'missingindicator_year_built']
      area_sqm  missingindicator_area_sqm
2794      89.7                        1.0
1784      89.7                        1.0

Five columns went in and seven came out. distance_km, rooms and bathrooms had no gap in the training rows and got no indicator, and each of the 90 training listings with no area now carries the median, 89.7, beside a 1 that says it was filled.

Lesson 3.2.2 found, on its training rows, a rating missing about three times as often on a building from before 1960 as on a newer one, so that gap says something about the listing. What the most frequent fill does to those older listings here:

built = X_train["year_built"]
old_ratings = X_train.loc[built < 1960, "energy_rating"]
missing_old = old_ratings.isna().sum()
rated_old = old_ratings.notna().sum()
print(missing_old, rated_old)      # 58 175
print((old_ratings == "C").sum())  # 15
print(old_ratings.value_counts().head(3).to_dict())

{'E': 55, 'D': 51, 'F': 37}

The training rows hold 58 listings from before 1960 with no rating and 175 with one. The fill writes C into all 58, while only 15 of the 175 hold a C and their most common ratings are E, D and F, so once the fill has run, only an indicator still says which of the 58 had no rating at all.

MissingIndicator marks the gaps without filling anything:

from sklearn.impute import MissingIndicator

marker = MissingIndicator()
marks = marker.fit_transform(X_train[gappy])
print(marks.dtype, marks.shape)  # bool (2400, 3)
print(marks.sum(axis=0))         # [ 90 110 215]
print(marker.get_feature_names_out())

['missingindicator_area_sqm' 'missingindicator_year_built'
 'missingindicator_energy_rating']

It returns True and False, one column for each of the three columns with a gap at fit time, under the names add_indicator=True uses. The two part ways on a new gap, in a column that had none in the training rows. Both fitted on two columns with no gap, then handed a test listing whose distance is missing:

complete = ["distance_km", "rooms"]
silent = SimpleImputer(strategy="median", add_indicator=True)
silent.fit(X_train[complete])
strict = MissingIndicator().fit(X_train[complete])
new = X_test[complete].head(1).copy()
new["distance_km"] = np.nan   # a listing with no distance
print(silent.transform(new))  # [[5.75 1.  ]]
print(silent.indicator_)      # MissingIndicator(error_on_new=False)
try:
    strict.transform(new)
except ValueError as err:
    print(err)

The features [0] have missing values in transform but have no missing values in fit.

The imputer filled the distance with its training median, 5.75, and recorded nothing, because its indicator has a column only for a column that had a gap at fit, and raised nothing, because the MissingIndicator it builds inside itself is set to error_on_new=False. The MissingIndicator of its own, at its default, refused the listing and named the column by its position, 0.

DON'T:

• DON'T expect an indicator for a column that had no gap in the rows the imputer was fitted on. The test listing with no distance was filled with 5.75 and marked nowhere.
• DON'T let most_frequent stand alone where the gaps sit on one kind of listing. It writes C into 58 older listings, where the older listings that have a rating mostly hold E, D or F, and only the indicator keeps the 58 apart.

ColumnTransformer

What. ColumnTransformer, from sklearn.compose, takes a list of groups, each a name, a transformer and the columns it gets. fit fits each transformer on its own columns, and transform joins their outputs side by side in the order of the list. A column named in no group goes to remainder, which drops it unless told otherwise. Why. The listings table holds measured numbers with gaps, text categories and one column of ordered levels. Each needs a different step, and every one of those steps has to learn from the same training rows, in one call. When. For any table whose columns need different steps, as every table that mixes numbers and text does.

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

levels = ["poor", "fair", "good", "excellent"]


def make_prep(num_cols, remainder="drop"):
    return ColumnTransformer([
        ("num", SimpleImputer(strategy="median", add_indicator=True), num_cols),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ], remainder=remainder)


prep = make_prep(numeric)
table = prep.fit_transform(X_train)
print(type(table).__name__, table.shape)  # ndarray (2400, 26)
names = prep.get_feature_names_out()
print(names[[0, 5, 7, 24, 25]])

['num__area_sqm' 'num__missingindicator_area_sqm'
 'cat__neighborhood_Downtown' 'cat__energy_rating_nan' 'cond__condition']

Nine columns went in and 26 came out: the five numbers and their two indicators, then Lesson 3.2.2's one-hot columns, seven neighborhoods, three property types and the energy rating's A to G and its gap, then one condition code from that lesson's OrdinalEncoder, the levels in their real order. Each name starts with its group's name and two underscores, so a column says which step made it. The energy rating needs no imputer: Lesson 3.2.2's OneHotEncoder keeps its gap as a category of its own, energy_rating_nan, the indicator in one-hot form.

A column left out of every group is the quiet failure:

no_distance = ["area_sqm", "year_built", "rooms", "bathrooms"]
forgot = make_prep(no_distance)
print(forgot.fit_transform(X_train).shape)  # (2400, 25)
print(forgot.transformers_[-1])             # ('remainder', 'drop', ['distance_km'])
kept = make_prep(no_distance, remainder="passthrough")
print(kept.fit_transform(X_train).shape)    # (2400, 26)
print(kept.get_feature_names_out()[-1])     # remainder__distance_km

Leaving distance_km out of the numeric group raised nothing. The transformer returned 25 columns instead of 26, and only transformers_, set by the fit like every name with a trailing underscore, shows where the column went: to remainder, which dropped it. remainder="passthrough" keeps such a column, unchanged, at the end of the output, under the prefix remainder__.

With pandas output, one transformed listing can be read entry by entry. Listing 2794 is a training row with no area:

prep.set_output(transform="pandas")
frame = prep.fit_transform(X_train)
print(X_train.loc[2794])
listing = frame.loc[2794]
# only the entries that are not 0.0
print(listing[listing != 0])

area_sqm               NaN
year_built          1993.0
distance_km            8.5
rooms                    3
bathrooms                2
neighborhood     Hillcrest
property_type    townhouse
energy_rating            C
condition             fair
Name: 2794, dtype: object
num__area_sqm                       89.7
num__year_built                   1993.0
num__distance_km                     8.5
num__rooms                           3.0
num__bathrooms                       2.0
num__missingindicator_area_sqm       1.0
cat__neighborhood_Hillcrest          1.0
cat__property_type_townhouse         1.0
cat__energy_rating_C                 1.0
cond__condition                      1.0
Name: 2794, dtype: float64

Its area is the training median, 89.7, beside a 1 in num__missingindicator_area_sqm. Its neighborhood, type and rating are single 1s among their groups' columns, and its condition, fair, is the code 1.0, its position in the levels list. listing[listing != 0] keeps only the entries that are not 0.0, which hides the other one-hot columns and the year's indicator.

DON'T:

• DON'T trust a ColumnTransformer to report a column you never gave a group. remainder="drop" removed distance_km without a warning, so read get_feature_names_out() after every change to the groups.
• DON'T reach for remainder="passthrough" without checking what else it passes. It hands over every column named in no group exactly as it is, gaps included: with year_built left out of the numeric group, LinearRegression fitted on the output stops with Input X contains NaN.

Pipeline

What. Pipeline, from sklearn.pipeline, takes a list of named steps, every one a transformer except the last, which here is the model. fit calls fit_transform on each transformer in turn, hands its output to the next, and fits the last step on what comes out. predict sends new rows through each fitted step's transform and calls the last step's predict. Why. Lesson 1.8.3's Pipeline kept a plain list of step objects and threaded a value through their apply methods. scikit-learn's pairs each step with a name, so a fitted step can be reached by the name you chose, and it carries fit and predict for the whole chain, so the same steps run in the same order on the training rows and on every listing priced afterward. When. For every model that needs any step before it, which in this section is every model fitted on the table as it arrives.

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.pipeline import Pipeline, make_pipeline

pipe = Pipeline([
    ("prep", make_prep(numeric)),
    ("model", LinearRegression()),
])
pipe.fit(X_train, y_train)
predicted = pipe.predict(X_test)
print(len(predicted), predicted[:3].round())
mae = mean_absolute_error(y_test, predicted)
print(f"{mae:,.0f}", round(r2_score(y_test, predicted), 3))

600 [159471. 494584. 633814.]
56,562 0.766

The whole table went in, gaps and text included, and the pipeline priced all 600 test listings. Against Section 3.1's model, price on area alone, with an imputer in front of it so that it can price the 35 test listings with no area, on the same 600 listings:

area_only = make_pipeline(SimpleImputer(strategy="median"), LinearRegression())
area_only.fit(X_train[["area_sqm"]], y_train)
area_pred = area_only.predict(X_test[["area_sqm"]])
area_mae = mean_absolute_error(y_test, area_pred)
print(f"{area_mae:,.0f}", round(r2_score(y_test, area_pred), 3))
print([name for name, step in area_only.steps])

94,553 0.489
['simpleimputer', 'linearregression']

On the same 600 listings, the full table misses the average test listing by 56,562 against 94,553 for the area alone, with an R² of 0.766 against 0.489. make_pipeline builds the same kind of object and names each step after its class, in lowercase. A fitted pipeline answers to a step's name or its position:

steps = pipe.named_steps
print(list(steps))                           # ['prep', 'model']
print(steps["prep"] is pipe["prep"])         # True
print(pipe[0] is pipe["prep"])               # True
print(pipe[-1] is pipe["model"])             # True
print(pipe["model"].coef_.shape)             # (26,)
seen = pipe[:-1].transform(X_test)
print(seen.shape)                            # (600, 26)
by_hand = pipe["model"].predict(seen)
print(np.allclose(by_hand, predicted))       # True
print(round(pipe.score(X_test, y_test), 3))  # 0.766

pipe[0] is the first step and pipe[-1] the model, whose 26 weights match the 26 columns the ColumnTransformer returns. A slice, pipe[:-1], is a pipeline of every step but the last, and its transform returns the table the model sees: the model's own predict on it gives the pipeline's predictions, which is all the pipeline's predict does. score is the last step's, so a regression pipeline returns R².

The groups inside the ColumnTransformer are reached differently:

given = pipe["prep"].transformers[0][1]
fitted = pipe["prep"].named_transformers_["num"]
print(given is fitted)              # False
print(fitted.statistics_.tolist())  # [89.7, 1992.0, 5.75, 3.0, 2.0]
try:
    print(given.statistics_)
except AttributeError as err:
    print(err)

'SimpleImputer' object has no attribute 'statistics_'

transformers, with no underscore, holds the imputer handed to the ColumnTransformer, and that imputer was never fitted. The ColumnTransformer fits a copy of each transformer it is given and keeps the fitted copies in transformers_ and named_transformers_, which exist only after the fit, Lesson 3.1.1's trailing underscore at work.

The ColumnTransformer also decides what a table must hold to be priced:

shuffled = X_test[features[::-1]].assign(listing_id=0)
again = pipe.predict(shuffled)
print(np.allclose(again, predicted))  # True
try:
    pipe.predict(X_test.drop(columns=["year_built"]))
except ValueError as err:
    print(err)
try:
    pipe.predict(X_test.to_numpy())
except ValueError as err:
    print(err)

columns are missing: {'year_built'}
Specifying the columns using strings is only supported for dataframes.

It picks its columns by name, so a table with its columns in reverse order and an extra listing_id is priced exactly as before. A table without a column it uses is refused with the column named, and a NumPy array, which has no names to pick by, is refused outright, where Lesson 3.1.1's LinearRegression priced one under a warning.

A pipeline whose last step is a transformer is a transformer itself, so it can stand as one group of a ColumnTransformer. The numeric group below fills and then scales, with Lesson 3.2.1's StandardScaler:

from sklearn.preprocessing import StandardScaler

numeric_steps = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", StandardScaler()),
])
scaled_prep = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
pipe = Pipeline([("prep", scaled_prep), ("model", LinearRegression())])
pipe.fit(X_train, y_train)
scaled_pred = pipe.predict(X_test)
print(np.allclose(scaled_pred, predicted))   # True
num = pipe["prep"].named_transformers_["num"]
print(round(num["scale"].mean_[0], 2))       # 99.2
filled_area = X_train["area_sqm"].fillna(89.7)
print(round(filled_area.mean(), 2))          # 99.2
print(round(X_train["area_sqm"].mean(), 2))  # 99.57

The predictions did not move, which is what Lesson 3.2.1 found scaling does to a LinearRegression's predictions. The scaler's mean shows the order of the work: 99.2 is the mean area after the fill, not the raw 99.57, because the scaler was fitted on what the imputer handed it. This is the pipeline the rest of the lesson uses.

DON'T:

• DON'T read a learned value from the transformer you handed a ColumnTransformer. It is never fitted, and the fitted copy is pipe["prep"].named_transformers_["num"].
• DON'T price a table that lacks a column the pipeline uses. It stops with columns are missing: {'year_built'}, naming the column it did not find.
• DON'T hand a pipeline a NumPy array when its ColumnTransformer picks columns by name. It refuses with Specifying the columns using strings is only supported for dataframes.

Why the Pipeline Is Leakage-Safe

What. Lesson 3.1.2's held-out rule, enforced by the pipeline's structure. Whenever a pipeline is fitted, every step inside it is fitted again, on exactly the rows fit received, and predict only applies what that fit learned. When the pipeline is handed the training rows as the split left them, no step can learn a statistic of a held-out row, so none reaches the model. Why. Every statistic a step learns, a median, a scaler's mean, an encoder's categories, can change the numbers the model is fitted on or predicts from, and each step fitted by hand is one more place to hand it the wrong rows. The pipeline leaves one such place, the rows passed to fit, and Lesson 3.6.2 fits one pipeline again and again on different rows, relying on exactly this. When. Every time any step learns from the rows, which is every imputer, encoder and scaler in this section.

The fitted pipeline's imputer, read through the pipeline before and after a prediction:

print(num["impute"].statistics_.tolist())
print(X_train[numeric].median().tolist())
pipe.predict(X_test)
print(num["impute"].statistics_.tolist())
print(X_test[numeric].median().tolist())

[89.7, 1992.0, 5.75, 3.0, 2.0]
[89.7, 1992.0, 5.75, 3.0, 2.0]
[89.7, 1992.0, 5.75, 3.0, 2.0]
[87.1, 1991.0, 5.4, 3.0, 1.0]

Fitted on the training rows, it holds their medians, and predict on the test rows used them and changed nothing, though the test rows' own medians differ in four of the five columns. Fitted again on other rows, the same pipeline keeps nothing of the first fit:

first = X_train.iloc[:1000]
pipe.fit(first, y_train.iloc[:1000])
now = pipe["prep"].named_transformers_["num"]
print(now["impute"].statistics_.tolist())
print(first[numeric].median().tolist())
print(num["impute"].statistics_.tolist())
pipe.fit(X_train, y_train)  # the training rows again

[90.8, 1992.0, 5.8, 3.0, 2.0]
[90.8, 1992.0, 5.8, 3.0, 2.0]
[89.7, 1992.0, 5.75, 3.0, 2.0]

On the first 1,000 training rows, the imputer holds those rows' medians, 90.8 for the area and 5.8 for the distance. num, read before the refit, still prints the 2,400 rows' medians: the ColumnTransformer fitted a fresh copy of each group's steps, so a step taken out of named_transformers_ belongs to the fit it was taken from. The last line fits the pipeline on the training rows again.

Beside it, the broken order, an imputer fitted on every row before the split and its output handed to the model:

# the broken order: fill from every row, then split
leaky = SimpleImputer(strategy="median").set_output(transform="pandas")
filled_all = leaky.fit_transform(X[numeric])
print(leaky.statistics_.tolist())            # [89.2, 1992.0, 5.7, 3.0, 2.0]
f_train, f_test, fy_train, fy_test = train_test_split(
    filled_all, y, test_size=0.2, random_state=42
)
broken = LinearRegression().fit(f_train, fy_train)
print(f_test["area_sqm"][was_gap].unique())  # [89.2]
print(f_train.isna().sum().sum())            # 0
print(X_test["area_sqm"].notna().sum())      # 565

This is Lesson 3.1.2's third break, written out with an imputer. Fitted on every row, the imputer learned 89.2 for the area and 5.7 for the distance where the training rows give 89.7 and 5.75, because 565 held-out areas and 600 held-out distances went into those medians, and it wrote 89.2 into every gap, the 35 held-out ones included. The filled table has no gap left, so the model was fitted without a word, and nothing in the table shows that the held-out rows shaped it. This lesson prints no score for the broken order: what rules it out is the order of the work, as Lesson 3.1.2 showed, not the size of a number.

DON'T:

• DON'T fill before you split. The every-row median, 89.2, carries 565 held-out areas into every gap it fills.
• DON'T fit a step outside the pipeline and hand the pipeline its output. The pipeline refits only the steps it holds, and a table filled beforehand reaches them with no gap left to fill.
• DON'T keep a step taken from named_transformers_ in a variable across a refit. num still printed the 2,400 rows' medians after the pipeline was fitted on 1,000.

Lab Exercises

Lab Exercise 1: The Year's Gaps and the Rating's Gaps

In fill_check.py, load listings.csv, keep the Theory's nine feature columns, and split them with price using test_size=0.2 and random_state=42.

Fit a SimpleImputer with the median strategy and pandas output on the training rows' year_built alone. Print what it learned, the median year of every row and the median year of the test rows. Fill the test rows' years, print how many gaps the column holds before and after the fill, and add a comment saying which of the three medians filled the test rows and why two of them printing the same value does not make them interchangeable.

Then fit a SimpleImputer with the most-frequent strategy, an indicator and pandas output on the training rows' energy_rating. Print the output's column names, the rating it learned and how many training rows its indicator marks. Finally print the median construction year of the training listings with a rating and of those without one, and say in a comment what the indicator keeps that the fill erases.

Lab Exercise 2: A Smaller Pipeline, Refitted

In small_pipeline.py, split six columns, area_sqm, year_built, neighborhood, property_type, condition and distance_km, with price, using test_size=0.2 and random_state=42.

Build a pipeline with make_pipeline from a ColumnTransformer and a LinearRegression. The ColumnTransformer has three groups: gaps fills area_sqm and year_built with their medians and adds their indicators, cat one-hot encodes neighborhood and property_type into dense columns, and cond encodes condition from poor to excellent, while remainder passes distance_km through unchanged. Print the names make_pipeline gave the two steps.

Fit on the training rows, and print how many columns the model receives with the first and the last of their names, then the test MAE in whole units with R² to three decimals. Print the medians the fitted gaps group learned, refit the pipeline on the second half of the training rows and print what the group learned beside those rows' own medians, then refit on all the training rows and print the group's medians once more. Finally try to price the test rows from a table without distance_km, printing the message.

import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
features = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms",
            "neighborhood", "property_type", "energy_rating", "condition"]
X_train, X_test, y_train, y_test = train_test_split(
    listings[features], listings["price"], test_size=0.2, random_state=42
)

year = SimpleImputer(strategy="median").set_output(transform="pandas")
year.fit(X_train[["year_built"]])
print(year.statistics_.tolist())        # [1992.0]
print(listings["year_built"].median())  # 1992.0
print(X_test["year_built"].median())    # 1991.0
filled = year.transform(X_test[["year_built"]])
before = X_test["year_built"].isna().sum()
after = filled["year_built"].isna().sum()
print(before, after)                    # 30 0
# The training rows' 1992.0 filled the 30 test gaps. Every row's median
# prints the same value, but the held-out years went into it, so it is
# computed partly from the held-out rows and only lands on 1992.0.

rating = SimpleImputer(strategy="most_frequent", add_indicator=True)
rating.set_output(transform="pandas")
marked = rating.fit_transform(X_train[["energy_rating"]])
print(marked.columns.tolist())
print(rating.statistics_.tolist())  # ['C']
flag = marked["missingindicator_energy_rating"]
print(flag.sum())                   # 215
no_rating = X_train["energy_rating"].isna()
print(X_train.groupby(no_rating)["year_built"].median())
# The listings with no rating were built in 1985 at the median, against
# 1993 for the rest. The fill writes C into all 215, and only the
# indicator still tells the model which listings had no rating.

['energy_rating', 'missingindicator_energy_rating']
energy_rating
False    1993.0
True     1985.0
Name: year_built, dtype: float64

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

listings = pd.read_csv("listings.csv")
# only the columns the groups and the remainder need
columns = ["area_sqm", "year_built", "neighborhood", "property_type",
           "condition", "distance_km"]
X_train, X_test, y_train, y_test = train_test_split(
    listings[columns], listings["price"], test_size=0.2, random_state=42
)
levels = ["poor", "fair", "good", "excellent"]
prep = ColumnTransformer([
    ("gaps", SimpleImputer(strategy="median", add_indicator=True),
     ["area_sqm", "year_built"]),
    ("cat", OneHotEncoder(sparse_output=False),
     ["neighborhood", "property_type"]),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
], remainder="passthrough")
pipe = make_pipeline(prep, LinearRegression())
print([name for name, step in pipe.steps])

pipe.fit(X_train, y_train)
names = pipe["columntransformer"].get_feature_names_out()
print(len(names), names[0], names[-1])
predicted = pipe.predict(X_test)
mae = mean_absolute_error(y_test, predicted)
print(f"{mae:,.0f}", round(r2_score(y_test, predicted), 3))

['columntransformer', 'linearregression']
16 gaps__area_sqm remainder__distance_km
58,283 0.752

gaps = pipe["columntransformer"].named_transformers_["gaps"]
print(gaps.statistics_.tolist())   # [89.7, 1992.0]
half = X_train.iloc[1200:]
pipe.fit(half, y_train.iloc[1200:])
refit = pipe["columntransformer"].named_transformers_["gaps"]
print(refit.statistics_.tolist())  # [88.25, 1992.0]
print(half[["area_sqm", "year_built"]].median().tolist())
pipe.fit(X_train, y_train)
again = pipe["columntransformer"].named_transformers_["gaps"]
print(again.statistics_.tolist())  # [89.7, 1992.0]

try:
    pipe.predict(X_test.drop(columns=["distance_km"]))
except ValueError as err:
    print(err)

[88.25, 1992.0]
columns are missing: {'distance_km'}

Theory Assessment

Q1: Four Medians of One Column

The Theory printed four medians of area_sqm: 89.7 in the imputer's statistics_, 89.7 on the training rows, 89.2 on every row and 87.1 on the test rows. Say which value filled the 35 test listings with no area and why, which of the four values Lesson 3.1.2's held-out rule forbids as a fill, and why the year's 1992.0, printed on both the training line and the every-row line, does not make the every-row median safe. Then say what SimpleImputer() with no arguments fills with, and why strategy="median" refuses the energy rating while most_frequent fills it.

Hint: Ask which rows each number was computed from, and which of fit and transform learns. For the last part, ask what a median has to do with the values before it can pick one.

Q2: Seven Columns From Five, and 25 From 26

Explain why the five numeric columns came out of the flagged imputer as seven, why distance_km got no indicator, and what the imputer and a MissingIndicator each did with a test listing whose distance was missing. Then explain why the ColumnTransformer without distance_km in its numeric group returned 25 columns and no error, where to look to find the column, and what remainder="passthrough" does with a left-out column that has gaps.

Hint: An indicator is made at fit time, for the columns that had a gap then. For the second part, look for an attribute that exists only after the fit and lists every group with its columns, the remainder included.

Q3: What fit Does, Step by Step

Walk through what pipe.fit(X_train, y_train) does to each step of the final pipeline, the one whose numeric group fills and then scales, in the order it happens, and what pipe.predict(X_test) does. Use the scaler's mean of 99.2 against the raw training mean of 99.57 to show the order. Then explain why, in the first pipeline, pipe["prep"].transformers[0][1] had no statistics_ while named_transformers_["num"] held a fitted imputer, and why num, read before the refit on 1,000 rows, still printed the 2,400 rows' medians.

Hint: Follow the training rows through the ColumnTransformer group by group, and inside the numeric group step by step. For the last part, ask what the ColumnTransformer fits, the object you handed it or something else, and how often.

Q4: Two Teammates, One Imputer Each

Read this script without running it.

import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline, make_pipeline

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
X = listings[numeric]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# teammate A: an imputer fitted on every row, placed in a pipeline
outside = SimpleImputer(strategy="median").fit(X)
print(outside.statistics_[0])
a_pipe = Pipeline([("impute", outside), ("model", LinearRegression())])
a_pipe.fit(X_train, y_train)
print(outside.statistics_[0])

# teammate B: every row filled first, then a pipeline of its own
filler = SimpleImputer(strategy="median").set_output(transform="pandas")
filled = filler.fit_transform(X)
b_train = filled.loc[X_train.index]
b_pipe = make_pipeline(SimpleImputer(strategy="median"), LinearRegression())
b_pipe.fit(b_train, y_train)
print(b_pipe[0].statistics_[0], b_train.isna().sum().sum())

A teammate reads it and says A leaks, because its imputer was fitted on all 3,000 listings, and B is safe, because its pipeline fits its own imputer on the training rows alone. Say what each print shows, and answer both claims.

Hint: Ask whether a pipeline fits a step again when the step arrives already fitted, and whether an imputer can tell a filled value from a measured one. For B's last line, ask where the 90 values the fill wrote land among the sorted training areas.

89.2
89.7
89.2 0

print(a_pipe["impute"] is outside)          # True
area = b_train["area_sqm"].sort_values()
print((area == 89.2).sum())                 # 90
print(area.iloc[1199:1201].tolist())        # [89.2, 89.2]
print((X_train["area_sqm"] == 89.2).sum())  # 0

Assignment

Assignment: A Saved Pipeline Prices a Half-Filled Form

The listings site lets a seller leave the area, the construction year and the energy rating blank, and the pricing service still quotes a price, with the same fitted steps that produced the team's reported test score. The service is a separate program, so the fitted pipeline reaches it as a file. joblib, which arrived with scikit-learn in Lesson 3.1.1, does it in two calls: joblib.dump(obj, "name.joblib") writes a Python object, a fitted pipeline included, to a file and returns the list of files it wrote, and joblib.load("name.joblib") rebuilds the object in another script. Loading a file can run any code its maker put in it, so a file from a source you cannot vouch for is never loaded.

Write train_price.py: split the nine feature columns with price as the Theory did, fit the Theory's final pipeline, the one whose numeric group fills and then scales, on the training rows, print the test MAE in whole units with R² to three decimals, and save the pipeline as price_pipeline.joblib.

Then write price_form.py, which takes nothing from the first script but that file. Prove the file holds the saved pipeline by scoring it on the same test rows. Price three Riverside apartments in good condition, 3.0 kilometers out, with 3 rooms and 1 bathroom: one complete, at 85.0 square meters, built in 1998 and rated C, one with the area blank, and one with the area, the year and the rating blank. Print what the pipeline wrote into each numeric blank and each indicator, and each listing's value in the rating's gap column. Show what a form that leaves out the year_built column altogether draws, and price it with the column put back, blank. Finally price the third listing alone and as one row among the 600 test listings, and end with a comment saying why the two prices must match and where every fill value came from.

Hints:

• Both scripts must score the same test rows, so both split the same nine columns with test_size=0.2 and random_state=42. The loaded pipeline's feature_names_in_ lists the nine in the order it was fitted with, the attribute Lesson 3.1.1 met on a model.
• The pipeline's own output is scaled, which hides the fill values, so read them from the imputer inside the numeric group, pipe["prep"].named_transformers_["num"]["impute"], whose transform and get_feature_names_out() give the filled values with their names. A blank on the form is np.nan.

import joblib
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
nominal = ["neighborhood", "property_type", "energy_rating"]
features = numeric + nominal + ["condition"]
X_train, X_test, y_train, y_test = train_test_split(
    listings[features], listings["price"], test_size=0.2, random_state=42
)

numeric_steps = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", StandardScaler()),
])
levels = ["poor", "fair", "good", "excellent"]
prep = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
pipe = Pipeline([("prep", prep), ("model", LinearRegression())])
pipe.fit(X_train, y_train)

predicted = pipe.predict(X_test)
mae = mean_absolute_error(y_test, predicted)
print(f"{mae:,.0f}", round(r2_score(y_test, predicted), 3))
print(joblib.dump(pipe, "price_pipeline.joblib"))

56,562 0.766
['price_pipeline.joblib']

import joblib
import numpy as np
import pandas as pd
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split

pipe = joblib.load("price_pipeline.joblib")
features = list(pipe.feature_names_in_)  # the nine, in fitted order
listings = pd.read_csv("listings.csv")
X_train, X_test, y_train, y_test = train_test_split(
    listings[features], listings["price"], test_size=0.2, random_state=42
)
predicted = pipe.predict(X_test)
mae = mean_absolute_error(y_test, predicted)
print(f"{mae:,.0f}", round(r2_score(y_test, predicted), 3))

form = pd.DataFrame({
    "area_sqm": [85.0, np.nan, np.nan],
    "year_built": [1998.0, 1998.0, np.nan],
    "distance_km": [3.0, 3.0, 3.0],
    "rooms": [3, 3, 3],
    "bathrooms": [1, 1, 1],
    "neighborhood": ["Riverside", "Riverside", "Riverside"],
    "property_type": ["apartment", "apartment", "apartment"],
    "energy_rating": ["C", "C", np.nan],
    "condition": ["good", "good", "good"],
})
print(pipe.predict(form).round())

impute = pipe["prep"].named_transformers_["num"]["impute"]
written = pd.DataFrame(
    impute.transform(form[impute.feature_names_in_]),
    columns=impute.get_feature_names_out(),
)
print(written.iloc[:, [0, 1, 5, 6]])
cat = pipe["prep"].named_transformers_["cat"]
rating_cols = cat.transform(form[cat.feature_names_in_])
print(cat.get_feature_names_out()[-1], rating_cols[:, -1])

56,562 0.766
[384235. 426163. 434348.]
   area_sqm  year_built  missingindicator_area_sqm  missingindicator_year_built
0      85.0      1998.0                        0.0                          0.0
1      89.7      1998.0                        1.0                          0.0
2      89.7      1992.0                        1.0                          1.0
energy_rating_nan [0. 0. 1.]

no_year = form.drop(columns=["year_built"])
try:
    pipe.predict(no_year)
except ValueError as err:
    print(err)
no_year["year_built"] = np.nan  # the column back, every value blank
print(pipe.predict(no_year).round())

alone = pipe.predict(form.iloc[[2]])
crowd = pipe.predict(pd.concat([form, X_test]))
print(alone.round(), crowd[2].round(), len(crowd))
# Every fill value was learned from the 2,400 training rows when
# train_price.py fitted the pipeline, and predict only applies it, so
# the third listing gets 434,348 alone or among 603 rows, and nothing
# the service prices can move a fill value.

columns are missing: {'year_built'}
[395854. 437782. 434348.]
[434348.] 434348.0 603

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want a fill that looks at a listing's other columns, or the limits of a model saved to a file.

• YouTube - "Impute missing values using KNNImputer or IterativeImputer" by Data School. Two imputers that fill a gap by looking at the other columns of the same row instead of at one statistic of the whole column, on a six-row table, with the import line that switches on IterativeImputer, which scikit-learn 1.9.1 still marks experimental.
• Official docs - scikit-learn 1.9 User Guide, "Model persistence", the section "Security & Maintainability Limitations". Why a saved model is loaded only from a source you trust, and why loading it under another scikit-learn version is unsupported.

Summary

Every step in this lesson learned something, and what decided whether it was safe was never what it learned but which rows it learned from. A median, a most common rating, a scaler's mean and an encoder's categories are each a small fit, and each is a way for a held-out listing to reach the model. The pipeline replaces all of those chances with one: whatever rows fit receives, every step inside learns from those rows and nothing else.

The indicator and the remainder are the same lesson told from two sides. A fill erases a fact the table held, that a value was never measured, and the indicator is the one place that fact survives. A column left out of every group vanishes, and nothing reports it. In both, the failure is silence, and the defense is to read what came out, its names and its count, before trusting what went in.

What the pipeline cannot guard is the work done before it. A table filled from every row arrives with its gaps closed, and the pipeline fits its steps faithfully on numbers that already carry the held-out rows. The structure protects the steps it holds, and the order of the work protects the rest: split first, hand the pipeline the training rows, and let its fit be the only place anything is learned.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

numeric_steps = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", StandardScaler()),
])
prep = ColumnTransformer([
    ("num", numeric_steps, numeric),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
pipe = Pipeline([("prep", prep), ("model", LinearRegression())])
pipe.fit(X_train, y_train)    # every step, training rows only
pipe.predict(X_test)          # applies what fit learned
num = pipe["prep"].named_transformers_["num"]
num["impute"].statistics_     # the training medians
pipe[:-1].transform(X_test)   # the table the model sees
prep.get_feature_names_out()  # catch a dropped column