Scaling and the Transformer API

Context

Scaling puts features measured in different units on one footing, by subtracting one number from each column and dividing by another, both learned from the training rows. scikit-learn does it with a transformer (an object whose fit learns from the training rows and whose transform changes any rows with what it learned).

Every preprocessing step this section fits is a transformer: the scaler in front of a model fitted by small steps, the encoder that turns a neighborhood into numbers in Lesson 3.2.2, and the step that fills a missing area in Lesson 3.2.3. A pricing team reaches for a scaler to compare what a square meter and a kilometer do to a price in one unit, and the same fitted scaler meets every listing the site prices after the model ships.

Most of the failures print numbers, not errors. On unscaled areas and distances, a model fitted by small steps scores an R² below minus eleven trillion without a warning, a scaler refitted on each new listing turns every listing into zeros and the model prices them all alike, one area keyed with its decimal point misplaced stretches the scale every other listing is measured on, and a missing area passes through the scaler for the model to refuse. Lessons 2.2.5 and 2.3.3 standardized by hand, Lesson 3.1.2 ruled that a number computed from every row has already seen the held-out rows, and this lesson fits that arithmetic on the training rows alone.

Goal

• Fit a transformer on the training rows with fit or fit_transform, apply it to the test rows and to new listings with transform, and trace what refitting it on them does.
• Reproduce StandardScaler's mean_ and scale_ with NumPy at ddof=0, undo it with inverse_transform, and predict what it returns for a constant column and a missing value.
• Distinguish StandardScaler, MinMaxScaler and RobustScaler by what they learn, and identify which of their statistics one mistyped value moves.
• Identify which models a scaler changes: show LinearRegression's predictions unchanged, read its scaled weights per standard deviation, and fit SGDRegressor on scaled features.
• Keep column names and listing labels through a transformer with get_feature_names_out() and set_output(transform="pandas").

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want scaling pictured, or scikit-learn's own first scaler, before you start.

• YouTube - "Standardization vs Normalization Clearly Explained!" by Normalized Nerd. Feature scaling, and how standardization differs from normalization, the video's word for min-max scaling (mapping a feature's smallest value to 0 and its largest to 1), in under six minutes.
• Official docs - scikit-learn 1.9 User Guide, "Preprocessing data", the section "Standardization, or mean removal and variance scaling". StandardScaler fitted on a three-row array, the mean_ and scale_ it learns, and why it keeps them: to apply the training set's transformation again to the test set.

Table of Contents

• The Transformer Interface
• StandardScaler Is 2.2.5's Standardization
• MinMaxScaler and RobustScaler
• Which Models Care About Scale
• Feature Names and Pandas Output

Theory

Nothing to install, scikit-learn has been in (bootcamp) since Lesson 3.1.1. The blocks run in ~/dev/bootcamp beside listings.csv, with (bootcamp) active. Every block adds to one script, scaling.py, under the block before it, unless the sentence above it names a script of its own, and python3 scaling.py runs what you have so far.

Unless a block says otherwise, every scaler and model here works on the 2,875 listings that have an area, Lesson 3.1.1's has_area, with area_sqm and distance_km as the features and price as the target, split as Lesson 3.1.2 split them into 2,300 training rows and 575 test rows.

The Transformer Interface

What. A transformer is created with its hyperparameters and fitted with fit, which sets its learned attributes, trailing underscore and all, and returns it, exactly as Lesson 3.1.1's estimators are. Where a model has predict, a transformer has transform, which returns the rows it is handed, changed by what fit learned, and fit_transform fits on some rows and transforms the same rows in one call. StandardScaler, from sklearn.preprocessing, is the transformer for Lesson 2.2.5's standardization. Why. A scaler learns numbers from rows, two for each column, and Lesson 3.1.2's held-out rule decides which rows: the training rows alone. Keeping the learning in fit and the applying in transform is what lets numbers learned once be applied unchanged to the held-out rows and to every listing priced after the model ships. When. For every preprocessing step that learns from data, the scalers of this lesson, the encoders of Lesson 3.2.2 and the gap fillers of Lesson 3.2.3: fit on the training rows, transform on every set of rows the model will see.

The first block opens scaling.py and creates the scaler:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
features = ["area_sqm", "distance_km"]
X_train, X_test, y_train, y_test = train_test_split(
    has_area[features], has_area["price"], test_size=0.2, random_state=42
)

scaler = StandardScaler()
print(scaler.get_params())

{'copy': True, 'with_mean': True, 'with_std': True}

All three hyperparameters stay at their defaults in this lesson: with_mean and with_std switch the subtraction and the division on or off, and copy makes the scaler work on a copy of the data it is handed. The fit:

print(scaler.fit(X_train) is scaler)  # True
print(np.round(scaler.mean_, 2))      # [98.59  5.47]
print(np.round(scaler.scale_, 2))     # [44.54  3.04]

fit learned one mean and one standard deviation per column from the 2,300 training rows: the average training listing has 98.59 square meters and sits 5.47 kilometers from the center. transform applies them to any rows:

train_z = scaler.transform(X_train)       # z for standardized, 2.2.5's Z
test_z = scaler.transform(X_test)
print(type(test_z), test_z.shape)         # <class 'numpy.ndarray'> (575, 2)
print(np.round(train_z.mean(axis=0), 3))  # [ 0. -0.]
print(np.round(train_z.std(axis=0), 3))   # [1. 1.]
print(np.round(test_z.mean(axis=0), 3))   # [0.048 0.037]
print(np.round(test_z.std(axis=0), 3))    # [0.944 1.002]

The -0. is Lesson 2.2.5's float noise. The test rows land near 0 and 1 and not on them, because they were measured with the training rows' mean and deviation, and that is the held-out rule working: a test listing is scaled exactly as a listing posted tomorrow will be. fit_transform is the two calls in one, for the rows fit learns from:

in_one_call = StandardScaler().fit_transform(X_train)
print(np.array_equal(in_one_call, train_z))  # True

Every listing the model meets later goes through transform alone. The model is Lesson 3.1.1's LinearRegression, fitted on the scaled training rows, and two listings posted this morning arrive, 85 square meters at 3 kilometers and 200 square meters at 1:

from sklearn.linear_model import LinearRegression

model = LinearRegression().fit(train_z, y_train)
new = pd.DataFrame({"area_sqm": [85.0, 200.0], "distance_km": [3.0, 1.0]})
print(np.round(scaler.transform(new), 3))
print(np.round(model.predict(scaler.transform(new))))

[[-0.305 -0.812]
 [ 2.277 -1.47 ]]
[365084. 802177.]

Fitting a scaler on each new listing instead, the habit fit_transform invites:

for i in range(len(new)):
    one = new.iloc[[i]]  # one listing, a one-row table
    alone = StandardScaler().fit_transform(one)
    print(alone, np.round(model.predict(alone)))

[[0. 0.]] [380125.]
[[0. 0.]] [380125.]

Fitted on one listing, a scaler learns that listing's own values as the means and finds no spread, so the listing comes out as zeros, and so does every listing scaled that way: 85 and 200 square meters got the same price, 380,125. Why a column with no spread becomes 0 and not nan is the next subtitle's, and why zeros are priced at 380,125 is the fourth subtitle's.

A transformer takes a two-dimensional X like any estimator: scaler.fit(X_train["area_sqm"]) is refused with the ValueError Lesson 3.1.1's LinearRegression raised for a Series.

DON'T:

• DON'T call fit or fit_transform on the test rows or on a new listing. Refitted on the 575 test rows, the scaler learns a mean area of 100.74 and a deviation of 42.04 in place of the training rows' 98.59 and 44.54, and refitted on one listing at a time, it turned 85 and 200 square meters into the same zeros, priced alike at 380,125.
• DON'T fit a scaler on the whole table before the split. Its mean area, 99.02 against the training rows' 98.59, already holds the held-out listings, the number computed from every row that Lesson 3.1.2 named.
• DON'T hand a transformer a Series. X_train[["area_sqm"]] is the one-column table it takes.

StandardScaler Is 2.2.5's Standardization

What. StandardScaler computes Lesson 2.2.5's (X - X.mean(axis=0)) / X.std(axis=0), with the mean and the standard deviation taken from the rows it was fitted on and kept in mean_ and scale_, the deviation at NumPy's default ddof=0. inverse_transform runs the arithmetic backwards, times scale_ plus mean_, and returns values in their own units. Why. Nothing in the scaler is new arithmetic, so every number it returns can be checked by hand, and the two places where it departs from Lesson 2.2.5's arithmetic, a column that never varies and a missing value, are choices the library made that your data can meet. When. As the default scaler for features with no fixed range, areas, distances and years, and whenever a scaled value has to be checked by hand or read back in square meters and kilometers.

Lesson 2.2.5's NumPy arithmetic reproduces the fitted scaler:

train_values = X_train.to_numpy()
mu = train_values.mean(axis=0)         # 2.2.5's mean per feature
sd = train_values.std(axis=0)          # NumPy's default, ddof=0
print(np.allclose(scaler.mean_, mu))   # True
print(np.allclose(scaler.scale_, sd))  # True
by_hand = (X_test.to_numpy() - mu) / sd
print(np.allclose(by_hand, test_z))    # True

The divisor is the one thing to watch, because pandas divides by n minus 1, Lesson 2.2.5's ddof=1:

sd_ddof1 = X_train.std()                     # pandas divides by n - 1
print(np.round(sd_ddof1.to_numpy(), 3))      # [44.549  3.044]
print(np.allclose(scaler.scale_, sd_ddof1))  # False
# True, the same call with NumPy's divisor
print(np.allclose(scaler.scale_, X_train.std(ddof=0)))

pandas misses scale_ by about 0.01 square meters, enough for np.allclose to say False, and the same call with ddof=0 matches. Back to square meters and kilometers:

back = scaler.inverse_transform(test_z)
print(np.allclose(back, X_test))  # True
print(back[0])                    # [123.2   7.1]

Lesson 3.1.4's first test listing, 123.2 square meters in Southgate at 7.1 kilometers, comes back as it went in. The check is np.allclose, because multiplying and adding back can differ from the original in the last digits.

The first of the two departures is a column that never varies: among the one-room listings, the studios, rooms is 1 in every row. Save this as studios.py:

import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
studios = has_area[has_area["rooms"] == 1]
X = studios[["area_sqm", "rooms"]]
print(len(X), X["rooms"].unique())

scaler = StandardScaler().fit(X)
print(scaler.mean_, scaler.scale_)
print(scaler.transform(X)[:2])
values = X.to_numpy()
z = (values - values.mean(axis=0)) / values.std(axis=0)
print(z[:2])

243 [1]
[45.64279835  1.        ] [13.96735684  1.        ]
[[-1.08415633  0.        ]
 [ 0.26899876  0.        ]]
/home/goku/dev/bootcamp/studios.py:15: RuntimeWarning: invalid value encountered in divide
  z = (values - values.mean(axis=0)) / values.std(axis=0)
[[-1.08415633         nan]
 [ 0.26899876         nan]]

The standard deviation of rooms is 0. NumPy divides by that 0, warns, and returns nan for every studio. StandardScaler makes a choice written into the library instead: a feature with no spread gets a scale_ of 1.0, so its centered values, all zeros, stay zeros. Neither result has a standard deviation of 1, because no division gives a column with no spread one, and the zeros say only that the column never varied in the rows the scaler was fitted on. That is what happened to each new listing in the first subtitle: a scaler fitted on one row finds no spread in any column.

The second is a missing value. Back in scaling.py, split the whole table, gaps included, as Lesson 3.1.2's third break did:

train_rows, test_rows = train_test_split(
    listings, test_size=0.2, random_state=42
)
gappy = train_rows[["area_sqm"]]           # the 2,400 training rows
print(gappy["area_sqm"].isna().sum())      # 90
gap_scaler = StandardScaler().fit(gappy)
print(gap_scaler.n_samples_seen_)          # 2310.0
print(round(gap_scaler.mean_[0], 2))       # 99.57
print(round(gappy["area_sqm"].mean(), 2))  # 99.57
gaps_z = gap_scaler.transform(gappy)
print(np.isnan(gaps_z).sum())              # 90
try:
    LinearRegression().fit(gaps_z, train_rows["price"])
except ValueError as err:
    print(str(err).splitlines()[0])        # Input X contains NaN.

The scaler skipped the 90 gaps when it learned, as pandas' mean does, and handed them back untouched for the model to refuse with Lesson 3.1.1's message. Lesson 3.2.3 fills them.

DON'T:

• DON'T check scale_ against pandas' std() with no argument. It divides by n minus 1 and gives 44.549 for area against scale_'s 44.539, and X_train.std(ddof=0) is the match.
• DON'T expect a scaler to fill a gap. It learned its mean from the 2,310 areas it had and returned the 90 gaps as nan, which the model refused.
• DON'T read a constant column's zeros as information. Every studio's rooms became 0 because the column never varies among studios, and NumPy's arithmetic on the same column returns nan.

MinMaxScaler and RobustScaler

What. Two more transformers with the same interface, each learning a different pair of numbers per feature. MinMaxScaler learns the smallest and the largest training value, data_min_ and data_max_, and maps that range onto 0 to 1. RobustScaler learns the median, center_, and the interquartile range (the distance from the 25th percentile to the 75th, Lesson 2.2.5's cut points), scale_, subtracts the first and divides by the second. Why. The three scalers answer to extreme values differently. A minimum and a maximum are set by the two most extreme training listings alone, a mean and a standard deviation by every listing, the far ones most, and a median and quartiles only by the listings near the middle and the quarter marks, so a far value counts no more than a near one on the same side, the contrast Lesson 3.1.3 drew between the mean and the median. When. MinMaxScaler for a feature whose limits are fixed by what it measures and already reached in the training rows, a share that runs from 0 to 100 percent, so a new value cannot land outside 0 to 1. RobustScaler for a feature that can hold a value far from the rest, a mistyped one included, which should not set the scale every other listing is measured on.

from sklearn.preprocessing import MinMaxScaler, RobustScaler

minmax = MinMaxScaler().fit(X_train)
print(minmax.data_min_, minmax.data_max_)  # [24.   0.3] [278.5  12. ]
test_mm = minmax.transform(X_test)
print(np.round(test_mm.min(axis=0), 3))    # [0. 0.]
print(np.round(test_mm.max(axis=0), 3))    # [0.817 1.017]
print(X_test["distance_km"].max())         # 12.2

The farthest training listing sits 12.0 kilometers out, so 12.0 became 1. One held-out listing sits at 12.2 and lands at 1.017: the range came from the training rows, and nothing stops a new listing from falling outside it. RobustScaler on the same rows, checked against Lesson 2.2.5's np.median and np.percentile:

robust = RobustScaler().fit(X_train)
q25, q75 = np.percentile(train_values, [25, 75], axis=0)
print(robust.center_, robust.scale_)
print(np.median(train_values, axis=0), q75 - q25)

[88.4  5.7] [61.225  5.1  ]
[88.4  5.7] [61.225  5.1  ]

The median training listing has 88.4 square meters, and the middle half of the areas spans 61.225 square meters. A decimal point keyed in the wrong place shows what separates RobustScaler from StandardScaler: the largest training listing, 278.5 square meters, entered as 2,785.

slip = X_train.copy()
largest = slip["area_sqm"].idxmax()     # the label of the 278.5
slip.loc[largest, "area_sqm"] = 2785.0  # its decimal point lost
first = X_test.iloc[[0]]                # 3.1.4's first test listing


def report(candidate, rows):
    candidate.fit(rows)
    area_scale = candidate.scale_[0]
    at_first = candidate.transform(first)[0, 0]
    print(round(area_scale, 3), round(at_first, 3))


report(StandardScaler(), X_train)       # 44.539 0.553
report(StandardScaler(), slip)          # 71.458 0.329
report(RobustScaler(), X_train)         # 61.225 0.568
report(RobustScaler(), slip)            # 61.225 0.568

One slipped listing out of 2,300 stretched StandardScaler's scale, and every other listing's standardized area moved with it. The median and the quartiles did not move at all, because the largest listing sat above the 75th percentile before the slip and sits above it after. Ignoring the slip is not removing it, though:

row = slip.loc[[largest]]  # the slipped listing, as a table
for fitted in [StandardScaler().fit(slip), RobustScaler().fit(slip)]:
    print(round(fitted.transform(row)[0, 0], 2))

37.58
44.04

DON'T:

• DON'T expect MinMaxScaler to keep new listings inside 0 to 1. A held-out listing 12.2 kilometers out landed at 1.017, past the farthest training listing.
• DON'T standardize a column that can hold a slip and trust its scale. One area keyed as 2,785 moved scale_ from 44.539 to 71.458, and every other listing's standardized area with it.
• DON'T read RobustScaler as removing a slip. Its statistics ignored the 2,785, and the slipped listing itself still scales to 44.04, far beyond every real listing.

Which Models Care About Scale

What. A model cares about scale when rescaling a feature changes what it predicts. LinearRegression with its intercept does not: subtracting a number from a feature and dividing by another changes the units of its weight and leaves every prediction where it was. SGDRegressor (a linear model fitted by many small corrective steps instead of one least-squares solve, the loop Module 4 builds by hand) does, and so does a model that compares listings by distance. Why. A scaler is one more fitted object to carry from training to every prediction, so it belongs in front of the models whose answers depend on it, and wherever it sits, the weights it rescales are read in their new units. When. Before a model fitted by steps or one that measures distance, and whenever the weights of features measured in different units are compared.

The same LinearRegression on the unscaled training rows, beside model, which was fitted on train_z:

raw_model = LinearRegression().fit(X_train, y_train)
raw_pred = raw_model.predict(X_test)
z_pred = model.predict(test_z)
print(np.allclose(raw_pred, z_pred))     # True
print(np.round(raw_model.coef_))         # [  3566. -13514.]
print(np.round(model.coef_))             # [158817. -41130.]
print(round(model.intercept_))           # 380125
print(round(y_train.mean()))             # 380125
per_sd = raw_model.coef_ * scaler.scale_
print(np.allclose(per_sd, model.coef_))  # True

The raw weights are per square meter and per kilometer, two units no one can set side by side. The scaled weights, each the raw weight times its feature's scale_, are per standard deviation, 44.54 square meters of area and 3.04 kilometers of distance, so they compare: with the other feature held where it is, a typical difference in area moves a price by 158,817 and a typical difference in distance by 41,130, about a quarter as much. The intercept is the training mean price because a least squares line with an intercept passes through the averages, and the average training listing scales to 0 in every column. That is why the first subtitle's one-listing scalers, which turned every listing into zeros, priced every listing at 380,125.

Scaling the price itself changes nothing either. Fitted on the standardized price and converted back, the predictions are the raw model's:

y_mean, y_sd = y_train.mean(), y_train.std(ddof=0)
price_z = (y_train - y_mean) / y_sd      # the price, standardized
z_target = LinearRegression().fit(X_train, price_z)
back_pred = z_target.predict(X_test) * y_sd + y_mean
print(np.allclose(back_pred, raw_pred))  # True

For LinearRegression, scaling the target is never needed, and its intercept is what makes scaling the features harmless. Without the intercept the same step does damage:

from sklearn.metrics import r2_score

pinned_raw = LinearRegression(fit_intercept=False).fit(X_train, y_train)
pinned_z = LinearRegression(fit_intercept=False).fit(train_z, y_train)
raw_score = r2_score(y_test, pinned_raw.predict(X_test))
z_score = r2_score(y_test, pinned_z.predict(test_z))
print(round(r2_score(y_test, raw_pred), 3))  # 0.605
# 0.565 -3.879
print(round(raw_score, 3), round(z_score, 3))
pinned_mean = pinned_z.predict(train_z).mean()
print(round(pinned_mean))                    # 0

Lesson 3.1.1's fit_intercept=False pins the line where every feature is 0. Unscaled, that point is a listing of no area at the city center, and the pinned line scores 0.565 against the full model's 0.605. Scaled, 0 is the average training listing, so the pinned line prices the average listing at 0, its predictions average 0 over the training rows, and its test R² is -3.879.

SGDRegressor cares for another reason. Each of its steps moves every weight by an amount that grows with the feature's value and with the size of the miss, so the scale of the features sets the size of the steps:

from sklearn.linear_model import SGDRegressor

sgd_raw = SGDRegressor(random_state=42).fit(X_train, y_train)
sgd_raw_r2 = r2_score(y_test, sgd_raw.predict(X_test))
print(f"{sgd_raw_r2:.3e}")        # -1.121e+13
print(f"{sgd_raw.coef_[0]:.2e}")  # 6.49e+09
sgd_z = SGDRegressor(random_state=42).fit(train_z, y_train)
sgd_z_r2 = r2_score(y_test, sgd_z.predict(test_z))
print(round(sgd_z_r2, 3))         # 0.605
print(np.round(sgd_z.coef_))      # [157081. -41285.]

On areas in the tens to hundreds and prices in the hundreds of thousands the steps were far too large, the weights ran off to billions, and nothing warned or raised. Scaled, the same model with the same seed reaches LinearRegression's 0.605, with weights close to the least-squares ones.

Two kinds of model are left to later lessons. kNN (a model that predicts a listing's price from the listings nearest to it), in Lesson 3.5.1, and k-means (a method that groups listings around a few centers), in Lesson 3.8.1, both measure distance, so a feature's scale decides how much it counts in them, Lesson 2.3.3's point. A tree (a model that predicts through a sequence of yes-or-no questions, each comparing one feature with a threshold), also in Lesson 3.5.1, rescales its thresholds with the feature, so scaling offers it the same splits.

DON'T:

• DON'T compare raw weights across features in different units. Per unit, distance's -13,514 per kilometer looks larger than area's 3,566 per square meter, and per standard deviation area's 158,817 is nearly four times distance's -41,130.
• DON'T fit SGDRegressor on unscaled features and trust that it finished. It returned an R² of -1.121e+13 without a warning.
• DON'T scale the features of a LinearRegression(fit_intercept=False) and expect the same model. Its test R² went from 0.565 to -3.879, because the line stayed pinned at 0 and 0 became the average listing.

Feature Names and Pandas Output

What. get_feature_names_out() returns the names of the columns a transformer outputs, for a scaler the names it was fitted on, or x0, x1 and so on when it was fitted on an array. set_output(transform="pandas") makes the transformer's transform and fit_transform return a DataFrame instead of a NumPy array, with those names as its columns and, for a DataFrame handed in, its index labels as the index, and returns the transformer itself. Why. A NumPy array carries no names and no labels. A model fitted on one records no feature_names_in_, so Lesson 3.1.1's checks on names and order are gone, and a scaled row can no longer be found by its listing's label. When. Whenever a model fitted on a transformer's output will later be handed DataFrames, and whenever a scaled value has to be read against its listing.

print(scaler.get_feature_names_out())       # ['area_sqm' 'distance_km']
print(hasattr(model, "feature_names_in_"))  # False

model was fitted on the scaler's arrays and kept no names. pandas output carries them through:

scaler.set_output(transform="pandas")
test_frame = scaler.transform(X_test)
print(type(test_frame))  # <class 'pandas.DataFrame'>
print(test_frame.head(3).round(3))

      area_sqm  distance_km
765      0.553        0.535
2542     0.225        1.750
1346     0.200       -0.517

The scaled rows keep their listings' labels, 765, 2542 and 1346, Lesson 3.1.4's first three test listings, and their columns keep their names. A model fitted on pandas output records the names:

train_frame = scaler.transform(X_train)
named = LinearRegression().fit(train_frame, y_train)
print(named.feature_names_in_)            # ['area_sqm' 'distance_km']
named_pred = named.predict(test_frame)
print(np.allclose(named_pred, raw_pred))  # True

The scaler checks names too, because it was fitted on a DataFrame, so a table with its columns swapped is stopped at the first step, before a number reaches the model:

swapped = X_test[["distance_km", "area_sqm"]]
try:
    scaler.transform(swapped)
except ValueError as err:
    print(str(err).splitlines()[1])

Feature names must be in the same order as they were in fit.

That is the second line of the message Lesson 3.1.1's LinearRegression gave for the same swap.

DON'T:

• DON'T turn pandas output on after fitting a model on the scaler's arrays. model was fitted on train_z, and model.predict(test_frame) warns X has feature names, but LinearRegression was fitted without feature names.
• DON'T fit a transformer on X_train.to_numpy() and expect names back. Its get_feature_names_out() returns ['x0' 'x1'].

Lab Exercises

Lab Exercise 1: Year and Distance on the Training Rows' Scale

In year_scale.py in ~/dev/bootcamp, load listings.csv, keep the listings that have a year_built, and split year_built and distance_km against price with test_size=0.2 and random_state=42.

Create a StandardScaler, fit it on the training rows and scale them in one call, then scale the test rows. Print mean_ and scale_ rounded to two decimals, and check both against NumPy's mean and std of the training rows. For each side, print the scaled columns' means and standard deviations rounded to three decimals.

Undo the scaling of the test rows and check that the years and distances come back. Then scale a new listing, built in 2024 and 6.0 kilometers out, once with the fitted scaler and once with a scaler fitted on that listing alone, and say in a comment which of the two a model should receive.

Finally, split the whole table, gaps included, with the same seed, fit a scaler on the training rows' year_built alone, and print how many years it learned from, how many training rows have no year, and how many nan values its output holds.

Lab Exercise 2: Three Scalers in Front of Two Models

In three_scalers.py, keep the listings that have both an area and a year, and split area_sqm, distance_km and year_built against price with the lesson's seed and test size.

Fit a LinearRegression on the unscaled training rows, and print its test R² and its weights in whole units. Then, for each of StandardScaler, MinMaxScaler and RobustScaler, set pandas output, scale the training and test rows, fit a LinearRegression and an SGDRegressor(random_state=42) on the scaled training rows, and print on one line the scaler, the linear model's weights in whole units, whether its test predictions match the unscaled model's, and the SGDRegressor's test R² to three decimals.

Fit an SGDRegressor(random_state=42) on the unscaled training rows and print its test R² in scientific notation with three decimals. Print the feature names the last linear model recorded, and end with a comment saying which of the four sets of weights you would show a pricing team asking which feature moves a price most, and why.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

listings = pd.read_csv("listings.csv")
has_year = listings.dropna(subset=["year_built"])
columns = ["year_built", "distance_km"]
X_train, X_test, y_train, y_test = train_test_split(
    has_year[columns], has_year["price"], test_size=0.2, random_state=42
)

scaler = StandardScaler()
train_z = scaler.fit_transform(X_train)  # fit and scale, one call
test_z = scaler.transform(X_test)        # the training rows' numbers
print(np.round(scaler.mean_, 2))         # [1990.33    5.53]
print(np.round(scaler.scale_, 2))        # [22.61  3.04]
values = X_train.to_numpy()
print(np.allclose(scaler.mean_, values.mean(axis=0)))
print(np.allclose(scaler.scale_, values.std(axis=0)))
for side in [train_z, test_z]:
    print(np.round(side.mean(axis=0), 3), np.round(side.std(axis=0), 3))

True
True
[0. 0.] [1. 1.]
[-0.025 -0.02 ] [1.008 1.014]

back = scaler.inverse_transform(test_z)
print(np.allclose(back, X_test))            # True
new = pd.DataFrame({"year_built": [2024.0], "distance_km": [6.0]})
print(np.round(scaler.transform(new), 3))   # [[1.489 0.153]]
print(StandardScaler().fit_transform(new))  # [[0. 0.]]
# The model gets [[1.489 0.153]], the listing measured on the
# training rows' scale. The zeros describe no listing at all.

train_rows, test_rows = train_test_split(
    listings, test_size=0.2, random_state=42
)
years = train_rows[["year_built"]]
gap_scaler = StandardScaler().fit(years)
print(gap_scaler.n_samples_seen_)           # 2290.0
print(years["year_built"].isna().sum())     # 110
# 110
print(np.isnan(gap_scaler.transform(years)).sum())

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, SGDRegressor
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler

listings = pd.read_csv("listings.csv")
both = listings.dropna(subset=["area_sqm", "year_built"])
features = ["area_sqm", "distance_km", "year_built"]
X_train, X_test, y_train, y_test = train_test_split(
    both[features], both["price"], test_size=0.2, random_state=42
)

plain = LinearRegression().fit(X_train, y_train)
plain_pred = plain.predict(X_test)
# 0.568
print(round(r2_score(y_test, plain_pred), 3))
print(np.round(plain.coef_))  # [  3563. -17362.   1615.]

for scaler in [StandardScaler(), MinMaxScaler(), RobustScaler()]:
    scaler.set_output(transform="pandas")
    train_frame = scaler.fit_transform(X_train)
    test_frame = scaler.transform(X_test)
    linear = LinearRegression().fit(train_frame, y_train)
    same = np.allclose(linear.predict(test_frame), plain_pred)
    sgd = SGDRegressor(random_state=42).fit(train_frame, y_train)
    sgd_r2 = round(r2_score(y_test, sgd.predict(test_frame)), 3)
    print(scaler, np.round(linear.coef_), same, sgd_r2)

StandardScaler() [159848. -53449.  36774.] True 0.568
MinMaxScaler() [ 906717. -206611.  161458.] True 0.569
RobustScaler() [223027. -88548.  51667.] True 0.568

raw_sgd = SGDRegressor(random_state=42).fit(X_train, y_train)
raw_sgd_r2 = r2_score(y_test, raw_sgd.predict(X_test))
print(f"{raw_sgd_r2:.3e}")  # -1.392e+19
print(linear.feature_names_in_)
# The StandardScaler weights: each is the change in price for one
# standard deviation of its feature, a typical difference between
# listings, so area, 159,848, moves a price most, then distance and
# year. The raw weights rank distance first only because they are per
# unit, and a kilometer is a large part of distance's spread where a
# square meter is a small part of area's. MinMaxScaler's span the
# whole training range, which its two most extreme listings set.

['area_sqm' 'distance_km' 'year_built']

Theory Assessment

Q1: The Training Rows' Ruler

The scaled training rows came out with means of 0 and standard deviations of 1, and the scaled test rows with means of 0.048 and 0.037 and standard deviations of 0.944 and 1.002. Explain both results, and say why the second is the scaler working as it should. Then explain why scaling each new listing with a scaler fitted on that listing alone priced 85 square meters and 200 square meters at the same 380,125.

Hint: Ask which rows each set of statistics was computed from, and which rows a listing posted tomorrow will be measured against. For the second part, ask what a scaler fitted on one row learns as its mean and as its spread.

Q2: What Each Scaler Learned

For area, StandardScaler's scale_ is 44.539 and pandas' X_train["area_sqm"].std() is 44.549, and in the studios table rooms gets a scale_ of 1.0 where NumPy's arithmetic gives nan. Explain each. Then explain why one area keyed as 2,785 raised StandardScaler's scale_ to 71.458 and left RobustScaler's at 61.225, and why MinMaxScaler put a held-out listing at 1.017.

Hint: For the first pair, ask what each divides the summed squared distances by. For the slip, ask how much one far value adds to a sum of squares, and whether it changes which listings sit near the 25th and 75th percentiles.

Q3: Same Prices, Different Weights

LinearRegression fitted on the scaled training rows and on the unscaled ones prices every test listing the same, its weights reading [158817. -41130.] scaled against [ 3566. -13514.] unscaled, with an intercept of 380,125 scaled. Explain why the predictions agree, what each pair of weights means, and why the intercept is the training mean price. Then explain why SGDRegressor scored -1.121e+13 unscaled and 0.605 scaled, and why LinearRegression(fit_intercept=False) fell from 0.565 to -3.879 when its features were scaled.

Hint: Write a scaled area as the area minus a mean, over a deviation, and ask what the fit can do with the weight and the intercept to absorb both. For the pinned line, ask where it is pinned before and after scaling.

Q4: A Robust Scaler for a Slipped Area

A teammate finds the 2,785 square meter slip in the training rows and, instead of correcting it, puts RobustScaler in front of the price model, reasoning that a robust scaler ignores outliers. Read this script without running it.

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import RobustScaler, StandardScaler

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
features = ["area_sqm", "distance_km"]
X_train, X_test, y_train, y_test = train_test_split(
    has_area[features], has_area["price"], test_size=0.2, random_state=42
)
slip = X_train.copy()
slip.loc[slip["area_sqm"].idxmax(), "area_sqm"] = 2785.0

scalers = {"standard": StandardScaler(), "robust": RobustScaler()}
for name, scaler in scalers.items():
    model = LinearRegression().fit(scaler.fit_transform(slip), y_train)
    score = r2_score(y_test, model.predict(scaler.transform(X_test)))
    print(name, round(score, 3))

On the clean training rows the same model scored 0.605. Say what the two lines print, and where the teammate's reasoning goes wrong.

Hint: Ask first whether either scaler can change LinearRegression's predictions at all. Then ask what value the slipped listing reaches the fit with under RobustScaler, and what Lesson 3.1.3's squared error does with a listing that far from the line.

sq_dist = (slip["area_sqm"] - slip["area_sqm"].mean()) ** 2
print(round(sq_dist[largest]))             # 7210951
print(round(sq_dist.drop(largest).sum()))  # 4533342

standard 0.398
robust 0.398

raw_slip = LinearRegression().fit(slip, y_train)
raw_score = r2_score(y_test, raw_slip.predict(X_test))
print(round(raw_score, 3))     # 0.398
print(raw_slip.coef_.round())  # [ 1440. -5622.]
fixed = slip.copy()
fixed.loc[fixed["area_sqm"].idxmax(), "area_sqm"] = 278.5
fixed_model = LinearRegression().fit(fixed, y_train)
fixed_score = r2_score(y_test, fixed_model.predict(X_test))
print(round(fixed_score, 3))   # 0.605

Assignment

Assignment: A Scaler That Caps the Slip

Your team's preprocessing code takes any scikit-learn scaler: it sets pandas output, calls fit_transform on the training rows and transform on every other set of rows, and reads get_feature_names_out(). After the 2,785 square meter slip, a colleague proposes a scaler that standardizes exactly as StandardScaler does and then caps every scaled value at limit standard deviations either side of the training mean, 3.0 unless set, so a slip reaches the model as 3.0 rather than 37.58.

Write capped.py with a class CappedScaler, a transformer in the shape of Lesson 3.1.1's Assignment. Its one hyperparameter is limit. fit(X, y=None) takes a DataFrame with no gaps, learns mean_ and scale_ per column with NumPy at ddof=0, gives a column with no spread a scale_ of 1.0 as StandardScaler does, records n_features_in_ and feature_names_in_, and returns the transformer. transform(X) refuses a transformer that was never fitted, standardizes with the learned numbers and caps the result.

Three pieces do part of the work. TransformerMixin, from sklearn.base, is a class to inherit that supplies fit_transform, as fit followed by transform, and the set_output method, once the class has a get_feature_names_out. OneToOneFeatureMixin, also from sklearn.base, supplies get_feature_names_out() for a transformer whose output columns are its input columns, built from n_features_in_ and feature_names_in_. np.clip(a, low, high) returns a with every value below low raised to low and every value above high lowered to high.

On the lesson's split, show, in this order:

• The printed form of CappedScaler() and of CappedScaler(limit=2.5), and the message transform raises before the fit.
• That CappedScaler(limit=100.0) scales the test rows as StandardScaler does, and how many training values the default cap changes.
• Fitted on the slipped training rows, the slipped listing's scaled area and the first test listing's under StandardScaler, RobustScaler and CappedScaler, and the area the cap stands for on the slipped rows' scale, in whole square meters.
• With pandas output, the first three scaled test rows rounded to three decimals, and the feature names.
• The test R² of LinearRegression fitted after each of the three scalers on the slipped rows, and after CappedScaler on the clean rows.

End with a comment saying which listings the cap protects and which it does not, and why it is the only one of the three that changes LinearRegression's score.

Hints:

• The class line names all three classes to inherit, the two this assignment introduces and BaseEstimator, and fit takes y=None because fit_transform calls it with the rows alone.
• np.where(spread == 0, 1.0, spread) gives a column with no spread its 1.0, and comparing the capped training rows with StandardScaler's, element by element, gives a mask whose sum counts the values the cap changed.

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator, OneToOneFeatureMixin, TransformerMixin
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import RobustScaler, StandardScaler
from sklearn.utils.validation import check_is_fitted


class CappedScaler(OneToOneFeatureMixin, TransformerMixin, BaseEstimator):
    def __init__(self, limit=3.0):
        self.limit = limit           # stored as given, nothing else

    def fit(self, X, y=None):
        values = X.to_numpy()
        self.mean_ = values.mean(axis=0)
        spread = values.std(axis=0)  # ddof=0, as StandardScaler
        # no spread: divide by 1.0, StandardScaler's rule
        self.scale_ = np.where(spread == 0, 1.0, spread)
        self.n_features_in_ = X.shape[1]
        self.feature_names_in_ = np.array(X.columns)
        return self

    def transform(self, X):
        check_is_fitted(self)        # NotFittedError before fit
        z = (X.to_numpy() - self.mean_) / self.scale_
        return np.clip(z, -self.limit, self.limit)

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
features = ["area_sqm", "distance_km"]
X_train, X_test, y_train, y_test = train_test_split(
    has_area[features], has_area["price"], test_size=0.2, random_state=42
)
slip = X_train.copy()
largest = slip["area_sqm"].idxmax()
slip.loc[largest, "area_sqm"] = 2785.0

print(CappedScaler(), CappedScaler(limit=2.5))
try:
    CappedScaler().transform(X_test)
except ValueError as err:
    print(err)

standard = StandardScaler().fit(X_train)
wide = CappedScaler(limit=100.0).fit(X_train)
print(np.allclose(wide.transform(X_test), standard.transform(X_test)))
capped = CappedScaler().fit(X_train)
changed = capped.transform(X_train) != standard.transform(X_train)
print(changed.sum())

CappedScaler() CappedScaler(limit=2.5)
This CappedScaler instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
True
11

row = slip.loc[[largest]]
first = X_test.iloc[[0]]
for scaler in [StandardScaler(), RobustScaler(), CappedScaler()]:
    scaler.fit(slip)
    at_slip = round(scaler.transform(row)[0, 0], 2)
    at_first = round(scaler.transform(first)[0, 0], 3)
    print(scaler, at_slip, at_first)
capped_slip = CappedScaler().fit(slip)
cap_area = capped_slip.mean_[0] + 3.0 * capped_slip.scale_[0]
print(round(cap_area))

framed = CappedScaler().set_output(transform="pandas").fit(slip)
print(framed.transform(X_test).head(3).round(3))
print(framed.get_feature_names_out())

StandardScaler() 37.58 0.329
RobustScaler() 44.04 0.568
CappedScaler() 3.0 0.329
314
      area_sqm  distance_km
765      0.329        0.535
2542     0.125        1.750
1346     0.109       -0.517
['area_sqm' 'distance_km']

for scaler in [StandardScaler(), RobustScaler(), CappedScaler()]:
    model = LinearRegression().fit(scaler.fit_transform(slip), y_train)
    score = r2_score(y_test, model.predict(scaler.transform(X_test)))
    print(scaler, round(score, 3))
clean = CappedScaler()
model = LinearRegression().fit(clean.fit_transform(X_train), y_train)
score = r2_score(y_test, model.predict(clean.transform(X_test)))
print("clean rows", round(score, 3))

# The cap protects the fit from the slipped listing: it reaches the
# model at 3.0, about 314 square meters on the slipped rows' scale,
# and the score returns to 0.605. It does not protect the other
# listings: scale_ was learned with the slip in it, so the first test
# listing still shrinks to 0.329, as under StandardScaler, and on the
# clean rows the cap changed 11 real areas, 0.604 against 0.605. It is
# the only one that moves LinearRegression's score, because clipping
# is not a subtract-and-divide: it changes the value a listing brings
# to the fit, where the other two only change its units.

StandardScaler() 0.398
RobustScaler() 0.398
CappedScaler() 0.605
clean rows 0.604

Additional Resources

Optional, and nothing later in the course depends on either. Worth opening if you want to see how the steps of SGDRegressor work, or the scalers this lesson never met.

• YouTube - "Stochastic Gradient Descent, Clearly Explained!!!" by StatQuest with Josh Starmer. The problem stochastic gradient descent solves, how it solves it, and the situations where it is most useful.
• Official docs - scikit-learn 1.9 example, "Compare the effect of different scalers on data with outliers". Two features of the California housing data with a few very large outliers, put through StandardScaler, MinMaxScaler and RobustScaler beside MaxAbsScaler, PowerTransformer, QuantileTransformer and Normalizer, each plotted with the outliers in view and zoomed in without them.

Summary

A scaler turned out to be Lesson 2.2.5's arithmetic with a memory. Nothing in the subtraction or the division changed. What changed is where the two numbers come from and when they are used: learned once, from the training rows, then carried unchanged to the held-out rows and to every listing priced afterwards. Lesson 3.1.2's held-out rule stopped being a discipline to remember and became the shape of an object, fit on one set of rows and transform on all of them, and the listing scaled by itself showed what ignoring that shape costs.

The three scalers differ only in whom they listen to. MinMaxScaler listens to the two most extreme training listings, StandardScaler to every listing, the far ones most, and RobustScaler only to the listings near the middle and the quarter marks, which is why one mistyped area moved StandardScaler's statistics and left RobustScaler's untouched. None of them is cleaning. A gap went through StandardScaler untouched, and a slipped value stayed slipped under the one that ignored it.

Whether to scale at all turned out to be a property of the model rather than the data. LinearRegression with its intercept absorbed every subtract-and-divide into its weights and changed only how they read, while a model fitted by steps returned a nonsense score without a word until its features shared a scale. Every transformer in the rest of this section, the encoders and the gap fillers, has the shape this lesson met, and the rule for every one of them is the same: fitted on the training rows alone.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

from sklearn.preprocessing import MinMaxScaler, RobustScaler, StandardScaler

scaler = StandardScaler().set_output(transform="pandas")
train_z = scaler.fit_transform(X_train)  # learn on training rows
test_z = scaler.transform(X_test)        # their numbers, no refit
scaler.mean_, scaler.scale_              # ddof=0 mean and deviation
scaler.inverse_transform(test_z)         # back to the original units
scaler.get_feature_names_out()           # the fitted column names
MinMaxScaler().fit(X_train)              # training range onto 0 to 1
RobustScaler().fit(X_train)              # median, interquartile range