What Makes a Feature Good, and Building Features From the Columns You Have

Context

A feature is a column a model is fitted on, and building one means computing a new column from the columns a table already holds: a listing's age from two years, square meters per room from an area and a room count, the log of the area a price grows with. A feature earns its place by carrying information about the price in a form the model can use, existing at the moment the model is asked for a price, and meaning the same thing on every listing the model meets.

Once a pricing model has a pipeline, this is most of the work on it. The team turns the date a listing was posted into a number, gives the model each building's age, tries the price and the area on a log scale, and questions every column a teammate proposes, a price per square meter from a market report or the flag that says whether a listing sold within 30 days.

The failures look like progress. A column computed from the price lifts a validation R² from 0.84 to 0.922 and drops it to 0.686 once the same listings arrive without a price, a column that is settled only after the price is set lowers the average miss about as much as an honest feature, and a feature built outside the pipeline makes the model refuse the raw rows the pricing service sends. Lesson 3.2.3 put every preparation step inside one pipeline, and this lesson decides which columns go into it.

Goal

• Build features from dates, pairs of columns, bands and a neighborhood statistic, and judge each on a validation set and across 20 splits of the training rows.
• Apply the moment-of-prediction rule to every column, and identify a feature computed from the target or one that arrives after the price is set.
• Fit a model on the log of the price and the log of the area with TransformedTargetRegressor and FunctionTransformer, and diagnose from the residuals when a log transform matches the data.
• Build features inside a pipeline with FunctionTransformer and a transformer class whose fit learns from the rows it receives.
• Debug a log of zero and a log of a negative value inside a fitted pipeline.

Prerequisites

Both are optional, and the lesson stands without them. They are here if you want the arithmetic of logs, or one clean case of a column that will not exist in use, before you start.

• YouTube - "Logs (logarithms), Clearly Explained!!!" by StatQuest with Josh Starmer. A log isolates an exponent, multiplying becomes adding, eight times up and eight times down sit the same distance from where they started, and the log of zero has no value.
• Kaggle Learn - "Data Leakage" by Alexis Cook, the section "Target leakage". Leakage defined by the order in which data becomes available, with a patient's antibiotic record that is updated after the diagnosis it would help predict.

Table of Contents

• What Makes a Feature Good
• Features That Exist at the Moment of Prediction
• Building Features From Dates and Pairs of Columns
• Transforms That Match the Data's Structure
• Features Inside the Pipeline

Theory

Nothing to install: every scikit-learn class this lesson uses arrived with Lesson 3.1.1's install. Every Theory block below goes into one script, features.py, saved in ~/dev/bootcamp beside listings.csv and run with python3 features.py, each block added under the one before it.

What Makes a Feature Good

What. A feature is a column the model is fitted on. A good one carries information about the target in a form the model can use, exists when the prediction is made, and means the same thing on every row the model meets, the training rows and every listing priced after the model ships. Why. LinearRegression adds a weight times each value it is handed, so information a column holds as text never reaches the price, and information along a curve reaches it only as far as a straight line follows the curve. A column missing when a listing is priced cannot be used at all, and one computed differently on new rows hands the model a number it never learned. When. Before any column joins the model, and again whenever the model's form changes, because a column that earns its place in one form can lose it in another.

Every candidate in this lesson is judged on a validation set carved from the training rows, as Lesson 3.1.2's Assignment carved one, and the test rows are scored once, at the end. Keeping a feature is a choice about the model, and a choice made by the test score is Lesson 3.1.2's second break.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

listings = pd.read_csv("listings.csv")
X = listings.drop(columns=["price"])
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
# a quarter of the training rows, 600 listings
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42
)
print(len(X_fit), len(X_val), len(X_test))  # 1800 600 600

The first split is Lesson 3.2.3's, so the 600 test listings are the ones its pipeline was scored on, and the second leaves 1,800 fit rows, the rows every candidate is fitted on before the validation rows score it. X holds every column but the price, and a ColumnTransformer takes only the columns it names.

numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]


def make_prep(num_cols, cat_cols=nominal):
    return ColumnTransformer([
        ("num", SimpleImputer(strategy="median", add_indicator=True), num_cols),
        ("cat", OneHotEncoder(sparse_output=False), cat_cols),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])


def raw_pipe(num_cols, cat_cols=nominal):
    prep = make_prep(num_cols, cat_cols)
    return Pipeline([("prep", prep), ("model", LinearRegression())])


def validate(pipe, fit_rows=X_fit, val_rows=X_val):
    pipe.fit(fit_rows, y_fit)
    predicted = pipe.predict(val_rows)
    mae = mean_absolute_error(y_val, predicted)
    print(f"{mae:,.0f}", round(r2_score(y_val, predicted), 3))
    return pipe


baseline = validate(raw_pipe(numeric))  # 55,973 0.84

make_prep is Lesson 3.2.3's, with the category columns as a parameter in place of remainder, and validate fits a pipeline on the fit rows and prints its average miss and its R² on the 600 validation rows. Lesson 3.2.3's pipeline misses a validation listing by 55,973 on average, with an R² of 0.84 where it printed 0.766 on the test rows, and an R² is measured against the spread of the rows it is computed on, Lesson 3.1.4's point, so every comparison from here is made on the validation rows. The baseline's residuals, by the year each listing was posted:

residual = y_val - baseline.predict(X_val)
year = pd.to_datetime(X_val["listing_date"]).dt.year.rename("year")
print(residual.groupby(year).agg(["count", "mean"]).round(0))

      count     mean
year                
2022    130 -15301.0
2023    171 -16764.0
2024    153   2811.0
2025    146  16262.0

The baseline prices the listings posted in 2022 and 2023 about 15,000 to 17,000 too high on average and those posted in 2025 16,262 too low, Lesson 3.1.4's residual means across a group the model never saw. The date carries information about the price, and the model has never had it: it sits in X as text, where each date is a category of its own:

dates = OneHotEncoder(sparse_output=False)
dates.fit(X_fit[["listing_date"]])
print(len(dates.categories_[0]))  # 1030
known = X_val["listing_date"].isin(dates.categories_[0])
print((~known).sum())             # 151
try:
    dates.transform(X_val[["listing_date"]])
except ValueError as err:
    print(str(err)[:24])          # Found unknown categories

One-hot encoding would give each of the 1,030 dates in the fit rows a column, and 151 of the 600 validation listings were posted on a date no fit row holds, so the encoder refuses the batch, Lesson 3.2.2's unknown category, as it would every listing posted after the model ships. The date carries information and exists when a listing is posted, and in this form it fails the third test: one column per date means nothing on a new row. The third subtitle turns it into a number that does.

DON'T:

• DON'T set a validation score beside a test score. The baseline's 0.84 on the validation rows and 0.766 on the test rows score different listings with fits on different rows, so their gap says nothing about a feature.
• DON'T choose a feature by the test score. Every comparison in this lesson runs on the validation rows, and the test rows are scored once, after the last choice.
• DON'T one-hot encode a column whose new rows bring values no training row holds. 151 of the 600 validation listings carried a date the fit rows never had, and every listing posted after the model ships carries one.

Features That Exist at the Moment of Prediction

What. The moment of prediction is the moment the model is asked for a price: the seller has filled in the listing form, and the pricing service quotes a price for it. A feature must exist at that moment, and it is never computed from the target. Why. The validation and test scores stand in for the error on listings priced in use, Lesson 3.1.2's rule. A column that exists only once a price does, or only some time after the listing is priced, is present on the held-out rows and on no listing the service prices, so its score describes a model that can never run. When. For every column, before it joins the model, by two questions: when does it become known, set against the moment of prediction, and what is it computed from.

A teammate's market report divides each listing's price by its area. As a column beside the baseline's:

leak_fit = X_fit.assign(price_per_sqm=y_fit / X_fit["area_sqm"])
leak_val = X_val.assign(price_per_sqm=y_val / X_val["area_sqm"])
leaky = raw_pipe(numeric + ["price_per_sqm"])
validate(leaky, leak_fit, leak_val)  # 37,725 0.922

One column did more than any honest change in this lesson does. What it holds:

measured = leak_val["area_sqm"].notna()
rebuilt = leak_val["price_per_sqm"] * leak_val["area_sqm"]
print(measured.sum())  # 581
# True: the column times the area is the price
print(np.allclose(rebuilt[measured], y_val[measured]))

On the 581 validation listings with an area, the column times the area is the listing's own price, so each of those rows handed the model its answer in two pieces. The service prices a listing that has no price yet, so it has no price per square meter either, and the column reaches the pipeline blank:

posted = X_val.assign(price_per_sqm=np.nan)  # no price yet
in_use = leaky.predict(posted)
in_use_mae = mean_absolute_error(y_val, in_use)
# 79,876 0.686
print(f"{in_use_mae:,.0f}", round(r2_score(y_val, in_use), 3))

The imputer fills every blank with the fit rows' median price per square meter, and the same fitted pipeline does worse on the same 600 listings than the baseline that never had the column. The 0.922 belonged to a model that cannot run. A price per square meter is a fair number in a market report or a query, and Lesson 3.1.1's PricePerSqm learned one as a parameter, the median over its training listings, inside fit. As a column, it is the price.

The second leak is a later event. sold_in_30_days records whether a listing sold within 30 days of being posted:

sold = raw_pipe(numeric + ["sold_in_30_days"])
validate(sold)  # 54,504 0.845
by_sale = residual.groupby(X_val["sold_in_30_days"])
print(by_sale.agg(["count", "mean"]).round(0))

                 count     mean
sold_in_30_days                
False              476   3790.0
True               124 -31094.0

The 124 validation listings that went on to sell within 30 days sit 31,094 below the baseline's predictions on average, so the column carries information about the price, and with it the model cut the average miss by 1,469. Its value is settled after the moment of prediction, when the listing sells or its 30 days run out, so no listing the service prices has it. Its gain is about the size of the honest features of the next subtitle, so the size of a gain clears nothing: a jump like price per square meter's is what sends you looking, and the two questions decide every column, large gain or small.

DON'T:

• DON'T hand a price model a column computed from the price. price_per_sqm lifted the validation R² to 0.922, and the same model scored 0.686 on the same listings with the column blank, as the service hands every listing.
• DON'T keep a column written after the moment of prediction because its gain is modest. sold_in_30_days cut the average miss by 1,469, and not one listing the service prices has it.

Building Features From Dates and Pairs of Columns

What. A built feature is a column computed from columns the table already holds: a listing's age from its listing year and year_built, the days from a fixed date to listing_date, square meters per room from area_sqm and rooms, a band of one column, or a column set against a statistic of its group. Why. A built column hands the model information the raw columns hold in a form it cannot use, the date above all, and every one is a candidate: it stays only when its gain holds on the validation rows and across splits. When. When a residual pattern or a fact about the market points at information sitting in an unusable form, and only from columns that pass the two questions.

def add_features(table):
    listed = pd.to_datetime(table["listing_date"])
    return table.assign(
        age=listed.dt.year - table["year_built"],
        days_in=(listed - pd.Timestamp("2022-01-01")).dt.days,
        sqm_per_room=table["area_sqm"] / table["rooms"],
    )


F_fit, F_val = add_features(X_fit), add_features(X_val)
built = ["age", "days_in", "sqm_per_room"]
shown = ["listing_date", "year_built", "area_sqm", "rooms"] + built
print(F_val[shown].head(3))

     listing_date  year_built  area_sqm  rooms   age  days_in  sqm_per_room
1705   2024-04-10      1981.0      69.9      2  43.0      830         34.95
1150   2023-07-02      1979.0     144.7      5  44.0      547         28.94
453    2022-07-25      2005.0     157.2      5  17.0      205         31.44

.dt.year and .dt.days are Lesson 2.5.3's, and assign returns a new table with the three columns added. Each value comes from the listing's own row, and the days are counted from a date written into the code, so a day means the same thing on every row:

one = X_val.head(1)                          # one listing, priced alone
own = pd.to_datetime(one["listing_date"])
print((own - own.min()).dt.days.iloc[0])     # 0
print(add_features(one)["days_in"].iloc[0])  # 830

Counted from the first date of the rows at hand, a listing priced alone is always day 0, and this 2024 listing is day 830 from the fixed date. The model learned from the fit rows what day 830 means, and to it day 0 is a listing from the first day of 2022.

negative = F_fit["age"] < 0
print(negative.sum(), F_fit["age"].min())  # 41 -3.0

41 fit listings have a negative age, down to -3, each posted in a year before the one its building is dated to. Each built column, added alone to the baseline:

for column in built:
    print(column, end=" ")
    validate(raw_pipe(numeric + [column]), F_fit, F_val)

age 54,832 0.845
days_in 54,445 0.846
sqm_per_room 54,459 0.844

Each lowers the validation miss from the baseline's 55,973 to between 54,445 and 54,832, about what sold_in_30_days gained, and one split is one draw, Lesson 3.1.2's point. Whether the listings a split held out decided a gain is measured the way Lesson 3.1.2 compared two models, across many splits, here 20 splits of the 2,400 training rows:

def wins(make, extra, rows):
    # the MAE without extra minus the MAE with it, on 20 splits
    # of the training rows, the test rows untouched
    gaps = []
    for seed in range(20):
        X_a, X_b, y_a, y_b = train_test_split(
            rows, y_train, test_size=0.25, random_state=seed
        )
        without = make([]).fit(X_a, y_a).predict(X_b)
        with_it = make(extra).fit(X_a, y_a).predict(X_b)
        miss_without = mean_absolute_error(y_b, without)
        miss_with = mean_absolute_error(y_b, with_it)
        gaps.append(miss_without - miss_with)
    return round(min(gaps)), round(max(gaps)), sum([gap > 0 for gap in gaps])


def raw_with(cols):
    return raw_pipe(numeric + cols)


F_train = add_features(X_train)
for column in built + ["sold_in_30_days"]:
    print(column, *wins(raw_with, [column], F_train))

age 219 2660 20
days_in 803 3177 20
sqm_per_room 142 1941 20
sold_in_30_days 55 2095 20

wins returns the smallest gap, the largest and the number of splits on which the column lowered the miss. All three built columns won 20 of 20, so their gains are not one split's luck. sold_in_30_days won 20 of 20 as well: the splits measure whether the listings a split held out decided a gain, and never when a column exists.

A band turns a number into a few categories, so a linear model can give each range its own amount:

def add_band(table):
    age = table["age"]
    conditions = [age.isna(), age < 0, age < 10, age < 40]
    choices = ["unknown", "negative", "new", "middle"]
    band = np.select(conditions, choices, default="old")
    return table.assign(age_band=band)


B_fit, B_val = add_band(F_fit), add_band(F_val)
print(B_fit["age_band"].value_counts().to_dict())
validate(raw_pipe(numeric, nominal + ["age_band"]), B_fit, B_val)

{'middle': 798, 'old': 622, 'new': 268, 'unknown': 71, 'negative': 41}
56,036 0.84

Lesson 2.6.1's np.select checks the gap first, so the 71 fit listings with no age keep a band of their own. A band gives every age inside it one value, and the five age bands raised the validation miss by 63, where age as a number lowered it by 1,141.

The last kind sets a listing against its group, here its area against its neighborhood's median area. The medians are a statistic, so they are learned from the fit rows and carried to every other row:

medians = X_fit.groupby("neighborhood")["area_sqm"].median()
every_row = X.groupby("neighborhood")["area_sqm"].median()
print((medians - every_row).round(1).to_dict())


def area_vs_hood(table):
    hood_median = table["neighborhood"].map(medians)
    return table.assign(area_vs_hood=table["area_sqm"] / hood_median)


G_fit, G_val = area_vs_hood(F_fit), area_vs_hood(F_val)
validate(raw_pipe(numeric + ["area_vs_hood"]), G_fit, G_val)
alone = one.groupby("neighborhood")["area_sqm"].transform("median")
print((one["area_sqm"] / alone).iloc[0])  # 1.0
learned = G_val["area_vs_hood"].iloc[0]
print(round(learned, 3))                  # 0.668

{'Downtown': 0.3, 'Eastgate': 0.5, 'Hillcrest': -4.0, 'Lakeview': 2.2, 'Northend': -1.0, 'Riverside': -2.1, 'Southgate': 0.8}
55,289 0.842

Lesson 2.6.1's map looks each neighborhood up in the Series' labels as it looked keys up in a dictionary. Every row's medians, which the validation and test listings' areas went into, differ from the fit rows' by up to 4.0 square meters, in Hillcrest, the hazard Lesson 3.1.2 named. Lesson 2.6.2's transform computes a median from the rows it is handed, so a listing priced alone is a neighborhood of one, at 1.0 against itself, where the learned medians put it at 0.668. The column cuts the validation miss by 684, and the last subtitle measures across 20 splits whether the listings a split held out decided it, once the medians are learned inside the pipeline. A statistic of the price is computed from the target, so in this module no group statistic of the price is ever a feature of a price model.

DON'T:

• DON'T count days from the first date of the rows at hand. A listing priced alone is day 0 that way, and day 830 from 2022-01-01.
• DON'T compute a group statistic from every row or from the rows being priced. Every row's Hillcrest median sits 4.0 square meters from the fit rows', and a listing priced alone is 1.0 against its own median.
• DON'T keep a column for one split's gain, or trust the splits to catch a leak. The three built columns and sold_in_30_days all won 20 of 20.
• DON'T band a number without measuring the bands against the number itself. Five age bands raised the miss by 63 where age as a number lowered it by 1,141.

Transforms That Match the Data's Structure

What. A transform replaces every value of a column with a function of it, here the natural log. TransformedTargetRegressor, from sklearn.compose (a wrapper that fits its regressor on a transformed target and turns every prediction back with the inverse function), fits on the log of the price and predicts in currency. FunctionTransformer, from sklearn.preprocessing (a transformer that applies a function you hand it and learns nothing in fit), logs a feature inside the pipeline. Why. A linear model adds weighted columns. When a price is a product, an area times a rate the neighborhood, the condition and the rest multiply together, its log is a sum, the log of the area plus the logs of the rest, which is the form a linear model takes. When. When the residuals fan out as the prediction grows, Lesson 3.1.4's mark of misses that grow with the price, and only on a column whose every value, in training and in use, is above zero.

from sklearn.compose import TransformedTargetRegressor
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import FunctionTransformer


def log_pipe(extra=(), log_price=True, log_area=True):
    area = SimpleImputer(strategy="median", add_indicator=True)
    if log_area:
        area = make_pipeline(FunctionTransformer(np.log), area)
    rest = ["year_built", "distance_km", "rooms", "bathrooms"] + list(extra)
    prep = ColumnTransformer([
        ("area", area, ["area_sqm"]),
        ("num", SimpleImputer(strategy="median", add_indicator=True), rest),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])
    model = LinearRegression()
    if log_price:
        model = TransformedTargetRegressor(
            regressor=model, func=np.log, inverse_func=np.exp
        )
    return Pipeline([("prep", prep), ("model", model)])

log_pipe is Lesson 3.2.3's pipeline with two switches: log_area sends the area through FunctionTransformer(np.log) before its imputer, so a gap is filled on the log scale, and log_price wraps the model in TransformedTargetRegressor. The four combinations:

forms = {"neither": (False, False), "area": (False, True),
         "price": (True, False), "both": (True, True)}
fitted = {}
for name, (log_price, log_area) in forms.items():
    print(name, end=" ")
    pipe = log_pipe([], log_price, log_area)
    fitted[name] = validate(pipe, F_fit, F_val)
loglog = fitted["both"]

neither 55,973 0.84
area 59,826 0.818
price 52,275 0.837
both 44,213 0.881

With neither logged, the pipeline is the baseline, 55,973 and 0.84. The area alone logged made both numbers worse, the price alone logged cut the miss and lowered R², and both logged cut the average miss to 44,213 and lifted R² to 0.881. The residuals show the structure behind those numbers:

labels = ["q1", "q2", "q3", "q4"]
raw_pred = baseline.predict(X_val)
log_pred = loglog.predict(F_val)
raw_miss = y_val - raw_pred
log_miss = np.log(y_val) - np.log(log_pred)
raw_q = pd.qcut(raw_pred, q=4, labels=labels)
log_q = pd.qcut(log_pred, q=4, labels=labels)
print(raw_miss.groupby(raw_q).std().round(0).tolist())
print(log_miss.groupby(log_q).std().round(3).tolist())

[45871.0, 59217.0, 86481.0, 101712.0]
[0.149, 0.169, 0.162, 0.139]

The baseline's misses more than double from the cheapest quarter of its predictions to the dearest, Lesson 3.1.4's fan. The log model's misses, measured on the log scale, where a miss of the same proportion has the same size at every price, hold between 0.139 and 0.169 in every quarter. Only with both sides logged is the product a sum the model can fit: with the price alone logged, the log of the price meets a raw area, and with the area alone logged, a raw price meets the log of an area.

fit_pred = loglog.predict(F_fit)
print(round((y_fit - fit_pred).mean()))    # 5193
log_fit_miss = np.log(y_fit) - np.log(fit_pred)
print(np.isclose(log_fit_miss.mean(), 0))  # True

Lesson 3.1.4's zero training mean holds where least squares did the fitting, on the log scale. In currency the fit rows' residuals average 5,193, so on the rows it was fitted to, the log model's prices sit below the listings' prices on average. A log also has a domain, Lesson 2.2.3's, and one listing sent by a form can leave it:

form = F_val.head(1).copy()  # one listing as a form sends it
form["area_sqm"] = 0.0
try:
    loglog.predict(form)
except ValueError as err:
    print(err)
form["area_sqm"] = -85.0
print(loglog.predict(form).round())
form["area_sqm"] = np.nan
print(loglog.predict(form).round())

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py:347: RuntimeWarning: divide by zero encountered in log
  result = func(self.values, **kwargs)
Input X contains infinity or a value too large for dtype('float64').
/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py:347: RuntimeWarning: invalid value encountered in log
  result = func(self.values, **kwargs)
[362832.]
[362832.]

An area keyed as 0 becomes -inf, and the pipeline refuses the listing. A negative area becomes nan under a warning that names a pandas file rather than your script, and the imputer fills it like any gap, so -85 square meters is priced exactly as a blank area, 362,832, and nothing stops it. The built columns, measured again in the log form:

def log_with(cols):
    return log_pipe(cols)


for column in built:
    print(column, *wins(log_with, [column], F_train))

age 108 2707 20
days_in 721 3122 20
sqm_per_room -877 696 11

In the log form the age and the date still win every split, and sqm_per_room, which won 20 of 20 beside the raw price, wins 11, a coin toss. The transform changed the model's form, and a feature is good for a form of model, not in general.

DON'T:

• DON'T log the price alone, or the area alone, and call it the log transform. Only both together beat the baseline on both numbers.
• DON'T take the log of a column that can hold zero or a negative value, in training or in a form. An area of 0 stops the pipeline at Input X contains infinity, -85 is priced as a blank area, and age holds 41 negative values in the fit rows.
• DON'T expect a log model's residuals to average zero in currency. On its fit rows they average 5,193.

Features Inside the Pipeline

What. FunctionTransformer(add_features) makes the lesson's feature function a step, so fit and predict each run it on the rows they receive. A feature that needs something learned, a neighborhood's median area, is a transformer class in the shape of Lesson 3.2.1's Assignment, whose fit learns from the rows it receives and whose transform applies what it learned. Why. A feature built outside the pipeline is a step the pricing service has to repeat by hand, the same way, on every form. Inside, it travels with the fitted pipeline, and a learned feature is refitted on exactly the rows fit receives, Lesson 3.2.3's rule. When. For every feature of a model that will price listings it has not seen.

outside = raw_pipe(numeric + ["days_in"]).fit(F_fit, y_fit)
try:
    outside.predict(X_val)  # the raw rows, as the service gets them
except ValueError as err:
    print(err)

columns are missing: {'days_in'}

A pipeline fitted on rows with days_in built beforehand needs that column in every table it prices, and the raw rows the service receives have none. As a step:

builder = FunctionTransformer(add_features)
print(builder.fit_transform(X_val).shape)  # (600, 15)

fit learns nothing and transform calls add_features, so the 12 raw columns come back as 15. The medians need a fit that learns:

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.utils.validation import check_is_fitted


class AreaVsHood(TransformerMixin, BaseEstimator):
    def fit(self, X, y=None):
        # each neighborhood's median area, from the rows fit receives
        self.medians_ = X.groupby("neighborhood")["area_sqm"].median()
        return self

    def transform(self, X):
        check_is_fitted(self)
        hood_median = X["neighborhood"].map(self.medians_)
        return X.assign(area_vs_hood=X["area_sqm"] / hood_median)

TransformerMixin supplies fit_transform and BaseEstimator the printed form, as in Lesson 3.2.1's Assignment, and check_is_fitted refuses a transform before any fit. In a pipeline that starts from the raw rows:

hood_pipe = Pipeline([
    ("build", FunctionTransformer(add_features)),
    ("hood", AreaVsHood()),
    ("prep", make_prep(numeric + ["area_vs_hood"])),
    ("model", LinearRegression()),
])
validate(hood_pipe)            # 55,289 0.842
inside = hood_pipe["hood"].medians_
print(inside.equals(medians))  # True

The pipeline built the date columns and the ratio itself and matched the hand-built ratio exactly, 55,289 and 0.842, because its step learned exactly the fit rows' medians. Measured across 20 splits, with the step refitted inside every one:

def raw_hood(cols):
    return make_pipeline(AreaVsHood(), raw_pipe(numeric + cols))


def log_hood(cols):
    return make_pipeline(AreaVsHood(), log_pipe(cols))


print("raw", *wins(raw_hood, ["area_vs_hood"], F_train))
print("log", *wins(log_hood, ["area_vs_hood"], F_train))

raw 469 1546 20
log -695 498 7

A pipeline can hold another pipeline as its last step, so each maker puts the class in front of the model it tests, and each split's medians come from that split's own rows. Beside the raw price the ratio wins 20 of 20, and in the log form 7 of 20. A class like this one is how a learned feature gets into a pipeline, and the runs decide whether it stays: the model that ships leaves the ratio and sqm_per_room out and keeps the two columns that won in its own form. Every choice is made, so it is fitted on all 2,400 training rows and scored on the test rows once:

final = Pipeline([
    ("build", FunctionTransformer(add_features)),
    ("price", log_pipe(["age", "days_in"])),
])
final.fit(X_train, y_train)
test_pred = final.predict(X_test)
test_mae = mean_absolute_error(y_test, test_pred)
print(f"{test_mae:,.0f}", round(r2_score(y_test, test_pred), 3))

47,060 0.789

Lesson 3.2.3's pipeline missed the same 600 test listings by 56,562 on average with an R² of 0.766. This one misses them by 47,060 with an R² of 0.789, from the raw table, with every feature built inside it.

DON'T:

• DON'T build features before the pipeline and ship the pipeline. Fitted on built rows, it refused the service's raw rows with columns are missing: {'days_in'}.
• DON'T compute a learned feature inside a FunctionTransformer. Its fit learns nothing, so a neighborhood median inside it is recomputed from the rows being priced, the 1.0 of a listing priced alone, and a class whose fit learns the medians is the step.
• DON'T carry a feature's verdict from one form of model to another. The ratio won 20 of 20 splits beside the raw price and 7 of 20 in the log form.

Lab Exercises

Lab Exercise 1: Three Columns Before They Join

Start audit.py with the Theory's first two blocks, which make both splits, define make_prep, raw_pipe and validate, and print the baseline's validation score.

A teammate proposes three columns: month, the month the listing was posted, bath_per_room, bathrooms divided by rooms, and price_band, the price cut by Lesson 2.6.1's pd.cut at 250,000, 400,000 and 600,000 into low, mid, high and top. Write a function that adds all three to a table given its prices, apply it to the fit rows and the validation rows, and print the three columns for the first three fit rows.

Score month and bath_per_room each added alone to the baseline, and price_band added as a category. Hand the fitted band pipeline the validation rows as the service receives them and print what it says, and end with a comment that answers the two questions of the moment of prediction for each column and says which may join.

Lab Exercise 2: Distance, Plain and Logged

Start log_distance.py with the Theory's first two blocks, add_features, and the imports and log_pipe of its fourth subtitle.

Write log_dist_pipe(extra), log_pipe with both logs on and distance_km moved out of the numeric group into a group of its own that applies FunctionTransformer(np.log). Write from_raw(make_price), a pipeline of a FunctionTransformer(add_features) step named build and a step named price built by make_price(["age", "days_in"]), and score from_raw(log_pipe) and from_raw(log_dist_pipe) on the validation rows.

Compare the two across the 20 splits of the training rows that wins uses, printing the smallest gap, the largest and the number of splits the logged distance wins, and print the smallest distance in the table. Then set the first validation listing's distance to 0.0, a listing at the city center, price it with both pipelines, catching what the logged one raises, and end with a comment saying which distance the model takes.

def propose(table, prices):
    listed = pd.to_datetime(table["listing_date"])
    edges = [0, 250_000, 400_000, 600_000, np.inf]
    labels = ["low", "mid", "high", "top"]
    return table.assign(
        month=listed.dt.month,
        bath_per_room=table["bathrooms"] / table["rooms"],
        price_band=pd.cut(prices, bins=edges, labels=labels),
    )


C_fit, C_val = propose(X_fit, y_fit), propose(X_val, y_val)
print(C_fit[["month", "bath_per_room", "price_band"]].head(3))
for column in ["month", "bath_per_room"]:
    print(column, end=" ")
    validate(raw_pipe(numeric + [column]), C_fit, C_val)
print("price_band", end=" ")
banded = raw_pipe(numeric, nominal + ["price_band"])
validate(banded, C_fit, C_val)
try:
    banded.predict(X_val)
except ValueError as err:
    print(err)
# month and bath_per_room are known when the listing is posted and come
# from its own columns, so both may join, and only month lowered the
# miss on this split. price_band is cut from the price, so its 0.913
# belongs to a model the service cannot run, and it never joins.

1800 600 600
55,973 0.84
      month  bath_per_room price_band
2524      4            0.4        mid
2754      8            0.5        low
2843      9            0.5        mid
month 55,615 0.842
bath_per_room 56,232 0.839
price_band 38,287 0.913
columns are missing: {'price_band'}

def log_dist_pipe(extra=()):
    area = make_pipeline(
        FunctionTransformer(np.log),
        SimpleImputer(strategy="median", add_indicator=True),
    )
    rest = ["year_built", "rooms", "bathrooms"] + list(extra)
    prep = ColumnTransformer([
        ("area", area, ["area_sqm"]),
        ("dist", FunctionTransformer(np.log), ["distance_km"]),
        ("num", SimpleImputer(strategy="median", add_indicator=True), rest),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])
    model = TransformedTargetRegressor(
        regressor=LinearRegression(), func=np.log, inverse_func=np.exp
    )
    return Pipeline([("prep", prep), ("model", model)])


def from_raw(make_price):
    build = FunctionTransformer(add_features)
    price = make_price(["age", "days_in"])
    return Pipeline([("build", build), ("price", price)])


plain = validate(from_raw(log_pipe))        # 42,416 0.886
logged = validate(from_raw(log_dist_pipe))  # 42,664 0.885

gaps = []
for seed in range(20):
    X_a, X_b, y_a, y_b = train_test_split(
        X_train, y_train, test_size=0.25, random_state=seed
    )
    a = from_raw(log_pipe).fit(X_a, y_a).predict(X_b)
    b = from_raw(log_dist_pipe).fit(X_a, y_a).predict(X_b)
    gaps.append(mean_absolute_error(y_b, a) - mean_absolute_error(y_b, b))
# -537 74 1
print(round(min(gaps)), round(max(gaps)), sum([gap > 0 for gap in gaps]))
print(X["distance_km"].min())         # 0.3

center = X_val.head(1).copy()
center["distance_km"] = 0.0           # a listing at the city center
print(plain.predict(center).round())  # [307902.]
try:
    logged.predict(center)
except ValueError as err:
    print(err)
# The logged distance lost 19 of 20 splits and cannot price a listing
# at the center, so the model takes the distance as it is.

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/pandas/core/internals/blocks.py:347: RuntimeWarning: divide by zero encountered in log
  result = func(self.values, **kwargs)
Input X contains infinity or a value too large for dtype('float64').

Theory Assessment

Q1: Two Columns the Score Loved

Adding price_per_sqm lifted the validation R² from 0.84 to 0.922, and the same fitted pipeline scored 0.686 on the same listings with the column blank. Adding sold_in_30_days cut the average miss by 1,469, and the listings that sold within 30 days sit 31,094 below the baseline's predictions. Say what each of the three scores measures, why the blank column is the listings as the service will hand them, which of the two questions catches each column, and why a gain of 1,469 does not clear sold_in_30_days.

Hint: Multiply the first column by the area. For the second, ask when a listing's row receives its value, set against the moment the service quotes a price.

Q2: A Day, a Median and a Band

Explain why days_in is counted from pd.Timestamp("2022-01-01") rather than from the earliest date of the rows at hand, using the listing that came out as day 0 and as day 830. Then explain why the neighborhood medians are learned from the fit rows and mapped onto every other row, what every row's medians and a listing priced alone would have given instead, and what the five age bands did to the age that the number did not.

Hint: A count means the same thing on a new row only if the point it starts from does not depend on which rows are at hand. For the band, count the values the model can tell apart.

Q3: Logs, Fans and a Form

The validation miss was 55,973 with nothing logged, 59,826 with the area logged, 52,275 with the price logged and 44,213 with both. Explain why only both matched the data, using the residual spreads by quarter, [45871.0, 59217.0, 86481.0, 101712.0] for the baseline and [0.149, 0.169, 0.162, 0.139] for the log model. Then explain what the pipeline did with an area of 0 and with -85, why the log model's fit rows average a residual of 5,193 where Lesson 3.1.4's averaged zero, and what sqm_per_room's 20 of 20 beside the raw price and 11 in the log form say about judging a feature.

Hint: Write the price as an area times a rate, take the log of both sides, and ask which scale each side needs. For the residual mean, ask on which scale least squares held the residuals at zero.

Q4: Four Columns Built Before the Split

A teammate builds four columns on the whole table, then splits and scores them as the Theory does. Read this without running it:

listings = pd.read_csv("listings.csv")
listed = pd.to_datetime(listings["listing_date"])
listings["age"] = listed.dt.year - listings["year_built"]
hoods = listings.groupby("neighborhood")["area_sqm"]
listings["hood_area"] = hoods.transform("median")
listings["price_rank"] = listings["price"].rank(pct=True)
listings["quick_sale"] = listings["sold_in_30_days"].astype(int)

Added to the baseline, the four score 43,432 and 0.885 on the validation rows, against the baseline's 55,973 and 0.84. Asked about leaks, the teammate answers that every leak comes from computing on the whole table before the split, so moving the four into a FunctionTransformer at the head of the pipeline makes all four safe, quick_sale included, the same values the lesson's 20 splits saw win 20 of 20 as sold_in_30_days. Say which of the four the price model may take, which ones the move repairs, and where the answer goes wrong.

Hint: Sort the four by what each is computed from: the listing's own row, every row, the price, or a later event. Then ask what a FunctionTransformer learns in fit, and what it computes from when the service hands it one listing.

def hood_area(table):
    hoods = table.groupby("neighborhood")["area_sqm"]
    return table.assign(hood_area=hoods.transform("median"))


X = listings.drop(columns=["price"])
step = FunctionTransformer(hood_area).fit(X)
alone = step.transform(X.head(1))
print(alone[["area_sqm", "hood_area"]].iloc[0].tolist())

[175.7, 175.7]

Assignment

Assignment: What One More Square Meter Adds, Neighborhood by Neighborhood

In the raw form one more square meter adds the same amount to a price in every neighborhood, and the log form lets each neighborhood multiply the price. A teammate proposes a third route that stays in currency: interaction columns. An interaction column multiplies a numeric column by one category's 0 or 1 column, so area_x_Eastgate holds the area on an Eastgate listing and 0.0 on every other one, and a linear model fitted on one such column per neighborhood, with no plain area column, takes a weight for each neighborhood: what one more square meter adds to a price there.

Write interactions.py, starting from the Theory's first two blocks, add_features with F_fit, F_val and F_train, and the imports and log_pipe of its fourth subtitle. Write a transformer class AreaByHood whose fit learns the median area of the rows it receives and their sorted neighborhoods, and whose transform fills a missing area with that median and adds one interaction column per learned neighborhood. Compare three forms, each taking age and days_in: the raw form, the log form, and the interaction form, a pipeline of AreaByHood and a ColumnTransformer whose numeric group takes the seven interaction columns in place of the area while Lesson 3.2.3's MissingIndicator marks the area's gaps.

Score the three on the validation rows, then count across 20 splits of F_train, made as wins makes them, how often the interaction form beats the raw form and how often the log form beats the interaction form. Show what the imputer would fill a gap in an interaction column with, print the seven interaction weights in whole units, and end with a comment on which form ships and what the seven weights tell a seller that the log form's do not.

Hints:

• Lesson 2.2.2's np.where(table["neighborhood"] == hood, area, 0.0) builds one interaction column. Fill the area inside AreaByHood before multiplying, because a gap left in the area becomes a gap in one interaction column, which the imputer after it fills with that column's median over the fit rows.
• Each weight is read by name from get_feature_names_out() on the fitted ColumnTransformer, where the numeric group's columns start with num__, and one loop over the splits can fit all three forms on each split.

from sklearn.base import BaseEstimator, TransformerMixin
from sklearn.impute import MissingIndicator
from sklearn.utils.validation import check_is_fitted


class AreaByHood(TransformerMixin, BaseEstimator):
    def fit(self, X, y=None):
        self.fill_ = X["area_sqm"].median()
        self.hoods_ = sorted(X["neighborhood"].unique())
        return self

    def transform(self, X):
        check_is_fitted(self)
        # filled before any product is taken
        area = X["area_sqm"].fillna(self.fill_)
        crossed = {
            f"area_x_{hood}": np.where(X["neighborhood"] == hood, area, 0.0)
            for hood in self.hoods_
        }
        return X.assign(**crossed)


hoods = sorted(X["neighborhood"].unique())
crossed = [f"area_x_{hood}" for hood in hoods]
both = ["age", "days_in"]
rest = ["year_built", "distance_km", "rooms", "bathrooms"] + both


def cross_form():
    prep = ColumnTransformer([
        ("num", SimpleImputer(strategy="median", add_indicator=True),
         rest + crossed),
        ("gap", MissingIndicator(), ["area_sqm"]),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])
    steps = [("cross", AreaByHood()), ("prep", prep)]
    return Pipeline(steps + [("model", LinearRegression())])


forms = {"raw": lambda: raw_pipe(numeric + both), "cross": cross_form,
         "log": lambda: log_pipe(both)}
models = {}
for name, make in forms.items():
    print(name, end=" ")
    models[name] = validate(make(), F_fit, F_val)

raw 54,489 0.846
cross 46,748 0.879
log 42,416 0.886

gaps = {"cross over raw": [], "log over cross": []}
for seed in range(20):
    X_a, X_b, y_a, y_b = train_test_split(
        F_train, y_train, test_size=0.25, random_state=seed
    )
    miss = {}
    for name, make in forms.items():
        predicted = make().fit(X_a, y_a).predict(X_b)
        miss[name] = mean_absolute_error(y_b, predicted)
    gaps["cross over raw"].append(miss["raw"] - miss["cross"])
    gaps["log over cross"].append(miss["cross"] - miss["log"])
for name, g in gaps.items():
    print(name, round(min(g)), round(max(g)), sum([gap > 0 for gap in g]))

cross_model = models["cross"]
crossed_fit = cross_model["cross"].transform(F_fit)
cross_medians = crossed_fit[crossed].median()
print(cross_medians.unique())
names = cross_model["prep"].get_feature_names_out()
weights = pd.Series(cross_model["model"].coef_, index=names)
per_sqm = weights[[f"num__area_x_{hood}" for hood in hoods]]
per_sqm.index = hoods
print(per_sqm.round(0).to_dict())
# The log form beat the interactions on 20 of 20 splits and ships.
# The interactions beat the raw form on 20 of 20, so one more square
# meter does not add the same amount everywhere: it adds 4,303 in
# Eastgate and 1,913 in Hillcrest, in currency, where the log form's
# neighborhood weights say only how much a neighborhood multiplies a
# price.

cross over raw 4545 8201 20
log over cross 1147 4389 20
[0.]
{'Downtown': 2973.0, 'Eastgate': 4303.0, 'Hillcrest': 1913.0, 'Lakeview': 3365.0, 'Northend': 1958.0, 'Riverside': 2958.0, 'Southgate': 2194.0}

Additional Resources

Optional, and nothing later depends on either. Worth opening for leaks outside a table of listings, or for bands whose edges follow the data.

• YouTube - "Examples of Data or Target Leakage in Machine Learning" by Rajistics. Two leaks in under two minutes: a salary model for new hires handed each employee's weekly income, and a skin-tumor image model that learned to look for the ruler doctors place beside the worst tumors.
• Google Machine Learning Crash Course - "Numerical data: Binning", the "Quantile Bucketing" section. Bins placed so each holds about the same number of examples, set against bins of equal width on a skewed column such as a price.

Summary

The best score in this lesson came from the one column that could never run. Price per square meter lifted a validation R² to 0.922 because it was the price in two pieces, and a model that needs the answer before it can give one is a model nobody can call. The second leak was quieter, a gain the size of an honest feature's that passed the same twenty splits. What separated the leaks from the features was never in a score but in when each column becomes known and what it is computed from, two questions a number cannot ask for you.

The largest honest gain came from no new column. The log of the price and the log of the area together turned a product into the sum a linear model fits, cut the average miss by about a fifth and left misses of one size in proportion to every price, and the same change of form took the value out of two features that had won every split beside the raw price. A feature is good for a model, not in general, and its worth is measured again whenever the model changes.

Where a feature is computed mattered as much as what it is. A count of days means the same thing on every row only from a fixed date, a neighborhood median only when it is learned once and carried to every other row, and a feature built outside the pipeline is a step the service has to remember. Built inside the pipeline from the raw table, the two logs and the two columns built from the listing date took the test miss from 56,562 to 47,060, with the test rows touched once.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

def add_features(table):
    listed = pd.to_datetime(table["listing_date"])
    return table.assign(
        age=listed.dt.year - table["year_built"],
        days_in=(listed - pd.Timestamp("2022-01-01")).dt.days,
    )


class AreaVsHood(TransformerMixin, BaseEstimator):
    def fit(self, X, y=None):
        groups = X.groupby("neighborhood")["area_sqm"]
        self.medians_ = groups.median()  # fit rows only
        return self

    def transform(self, X):
        check_is_fitted(self)
        hood = X["neighborhood"].map(self.medians_)
        return X.assign(area_vs_hood=X["area_sqm"] / hood)


model = TransformedTargetRegressor(
    regressor=LinearRegression(), func=np.log, inverse_func=np.exp
)
final = Pipeline([("build", FunctionTransformer(add_features)),
                  ("price", log_pipe(["age", "days_in"]))])
final.fit(X_train, y_train)              # every step, training rows only