Encoding Categorical Features

Context

Encoding turns a column of categories, a neighborhood, a property type, a condition grade, into numbers a model can multiply by a weight. One-hot encoding gives each category a column of 0s and 1s of its own, and ordinal encoding gives each category a single number, its place in a list.

Every model in this module that reads the listings' text columns reads them through an encoder. A price model adds one amount for Eastgate and another for Hillcrest, a condition grade goes in ranked from poor to excellent, a listing posted this morning is encoded into exactly the columns the model was fitted on, and a missing energy rating stays a fact the model can learn from.

Most of what goes wrong is quiet. Category codes pass the fit and hand the model an order the alphabet chose, pd.get_dummies (pandas' own one-hot function) gives a single new listing different columns from the training rows, and once the error that names an unknown neighborhood is switched off, its listings are priced anyway, as some other neighborhood or as none at all. Lesson 3.1.4's residuals split by neighborhood because the model had never seen one, and Lesson 3.2.1's transformer interface is how it gets to.

Goal

• Diagnose LinearRegression's refusal of a text column, and encode the column with a transformer fitted on the training rows.
• Build one-hot columns with OneHotEncoder, read categories_ and the feature names, and get an array or a DataFrame out of its sparse default.
• Explain why one-hot columns beside an intercept cost one unit of rank, and why drop="first" changes the weights of LinearRegression and not its predictions for the categories it knows.
• Distinguish handle_unknown="error" from "ignore" on a category the training rows never had, and state what "ignore" costs.
• Encode an ordered category with OrdinalEncoder in its real order, and identify the assumption its single weight per step carries.
• Keep a missing category as a column of its own, and measure what the gap says about a listing.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see categories turned into columns before you do it in code.

• YouTube - "Design Matrices For Linear Models, Clearly Explained!!!" by StatQuest with Josh Starmer. Design matrices for two groups, for a continuous measurement, and for both at once, built column by column out of 1s, 0s and measured values.
• Google Machine Learning Crash Course - "Categorical data: Vocabulary and one-hot encoding". Why a model cannot take a category as a string or as an index number, one-hot vectors, and the sparse representation that stores only where the 1 sits.

Table of Contents

• Why a Model Needs Numbers
• One-Hot Encoding
• Categories the Training Rows Never Had
• Ordinal Encoding and Its Order
• Missing Values in a Category

Theory

Nothing to install, scikit-learn has been in (bootcamp) since Lesson 3.1.1. Every Theory block below goes into one script, encoding.py, saved in ~/dev/bootcamp beside listings.csv and run with python3 encoding.py, each block added under the one before it. The rows are Section 3.1's 2,875 listings that have an area, split as one table with test_size=0.2 and random_state=42, which holds out the same 575 listings Lessons 3.1.2 to 3.1.4 held out.

Why a Model Needs Numbers

What. LinearRegression prices a listing as intercept_ plus each feature value times its weight, so every value it is handed has to be a number. Four columns of listings.csv hold categories (values from a short fixed list, such as the seven neighborhoods) stored as text, and the fit refuses them. Why. A model of area alone prices two listings of the same area alike whatever their neighborhoods, which is why Lesson 3.1.4's residuals averaged 145,253 above zero in Eastgate and 120,177 below it in Hillcrest. Encoding (turning each category into numbers a model can multiply) is how the neighborhood reaches the model. When. Before any text column goes into a model, for every model in this module.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
data = listings.dropna(subset=["area_sqm"])
train, test = train_test_split(data, test_size=0.2, random_state=42)
y_train, y_test = train["price"], test["price"]

print(train[["area_sqm", "neighborhood"]].dtypes)
try:
    LinearRegression().fit(train[["area_sqm", "neighborhood"]], y_train)
except ValueError as err:
    print(err)

area_sqm        float64
neighborhood        str
dtype: object
could not convert string to float: 'Downtown'

Before it computes anything, the fit converts every value of X to a float, and it stops at the first value with no float form, 'Downtown', the neighborhood of listing 1653, the first training row. The area column was never the problem, and the four category columns are short lists:

categorical = ["neighborhood", "property_type", "condition", "energy_rating"]
print(train[categorical].nunique())

neighborhood     7
property_type    3
condition        4
energy_rating    7
dtype: int64

2.5.3's nunique counts each column's distinct values with the gaps left out, so the four columns hold 21 categories between them across 2,300 training listings. An encoder is a transformer, 3.2.1's interface: fit learns a column's categories from the training rows, and transform turns any rows into numbers by what it learned.

DON'T:

• DON'T read could not convert string to float as a fault in the price or the area. It quotes the first text value the fit met, 'Downtown', and that value names the column to encode.
• DON'T get past the refusal with 2.5.3's astype("category").cat.codes. The codes are numbers, so the fit takes them without a word, and Ordinal Encoding and Its Order below measures what a model reads into them.

One-Hot Encoding

What. One-hot encoding replaces a column of categories with one column per category, holding 1 in the column of the row's own category and 0 in the others. OneHotEncoder, from sklearn.preprocessing, learns the categories from the training rows in fit, stores them in categories_, and writes the columns for any rows in transform. Why. Each column takes a weight of its own, so a linear model adds a separate amount for every neighborhood, and no column puts one neighborhood above or below another. When. For a category with no order among its values, a neighborhood or a property type, going into a linear model.

from sklearn.preprocessing import OneHotEncoder

encoder = OneHotEncoder()
hood = encoder.fit_transform(train[["neighborhood"]])
print(encoder.categories_)
print(encoder.get_feature_names_out())
print(type(hood))
print(hood.shape)  # (2300, 7)

[array(['Downtown', 'Eastgate', 'Hillcrest', 'Lakeview', 'Northend',
       'Riverside', 'Southgate'], dtype=object)]
['neighborhood_Downtown' 'neighborhood_Eastgate' 'neighborhood_Hillcrest'
 'neighborhood_Lakeview' 'neighborhood_Northend' 'neighborhood_Riverside'
 'neighborhood_Southgate']
<class 'scipy.sparse._csr.csr_matrix'>

categories_ holds one array per column the encoder was fitted on, its categories sorted, alphabetically for text, and get_feature_names_out() names each new column after its input column and its category. What fit_transform returned is not a NumPy array but a SciPy sparse matrix (a matrix that stores only its nonzero entries and where they sit): one 1 per row, 2,300 stored values standing for 16,100 cells.

print(np.array(hood).shape)  # ()
try:
    np.column_stack([train["area_sqm"], hood])
except ValueError as err:
    print(err)

all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 2300 and the array at index 1 has size 1

NumPy does not read a sparse matrix as a table. np.array(hood) wraps the whole matrix as one object, of shape (), so np.column_stack, which built 3.1.1's design matrix, finds 2,300 areas beside a single object. hood.toarray() returns an ordinary array, and sparse_output=False makes the encoder return one from the start, which 3.2.1's pandas output requires: on the default, set_output(transform="pandas") makes fit_transform raise Pandas output does not support sparse data., the first sentence of its message.

encoder = OneHotEncoder(sparse_output=False).set_output(transform="pandas")
hood_train = encoder.fit_transform(train[["neighborhood"]])
print(train["neighborhood"].head(3).tolist())
print(hood_train.head(3).to_numpy())
print(hood_train.sum(axis=1).unique())  # [1.]

['Downtown', 'Northend', 'Hillcrest']
[[1. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0. 0.]
 [0. 0. 1. 0. 0. 0. 0.]]

Downtown is first among the seven categories, so the Downtown listing has its 1 in the first column, the Northend listing in the fifth and the Hillcrest listing in the third. Each row holds exactly one 1, so every row's seven columns add up to 1, and side by side the seven columns add up to 2.3.4's column of ones. That costs a unit of rank:

ones = np.ones(len(train))
design = np.column_stack([ones, train["area_sqm"], hood_train])
# 9 8
print(design.shape[1], np.linalg.matrix_rank(design))

X_train = pd.concat([train[["area_sqm"]], hood_train], axis=1)
model = LinearRegression().fit(X_train, y_train)
print(round(model.intercept_))  # 40133
print(pd.Series(model.coef_, index=model.feature_names_in_).round(0))

area_sqm                    3467.0
neighborhood_Downtown      -4805.0
neighborhood_Eastgate     175179.0
neighborhood_Hillcrest   -126232.0
neighborhood_Lakeview     104153.0
neighborhood_Northend    -116838.0
neighborhood_Riverside     23307.0
neighborhood_Southgate    -54764.0
dtype: float64

The design matrix has 9 columns and rank 8, 2.3.5's collinear columns. LinearRegression raises nothing and fits, and 2.6.3's pd.concat with axis=1 set the area beside the seven columns by label, the labels the encoder's pandas output kept from the training rows. What the missing unit of rank does to the weights shows in one shift:

shifted = model.coef_.copy()
shifted[1:] = shifted[1:] + 1_000            # each neighborhood weight
moved = model.intercept_ - 1_000 + X_train.to_numpy() @ shifted
# True, every training prediction unchanged
print(np.allclose(moved, model.predict(X_train)))
print(np.isclose(model.coef_[1:].sum(), 0))  # True

Since each row has exactly one neighborhood column at 1, 1,000 more on every neighborhood weight adds 1,000 to every prediction, and 1,000 less on the intercept takes it back. The two sets of weights predict alike, and so would any amount in place of 1,000, so the data cannot choose among them. LinearRegression returns the set whose coef_ is shortest, 2.3.5's rule with the intercept left out of the length, and here that is the set whose seven neighborhood weights sum to zero. Each weight therefore measures its neighborhood against the plain average of the seven, Eastgate 175,179 above it and Hillcrest 126,232 below, and none is the effect of one neighborhood on its own. drop="first" takes the redundancy out:

first_enc = OneHotEncoder(sparse_output=False, drop="first")
first_enc.set_output(transform="pandas")
hood_first = first_enc.fit_transform(train[["neighborhood"]])
design = np.column_stack([ones, train["area_sqm"], hood_first])
# 8 8
print(design.shape[1], np.linalg.matrix_rank(design))

X_first = pd.concat([train[["area_sqm"]], hood_first], axis=1)
model_first = LinearRegression().fit(X_first, y_train)
print(round(model_first.intercept_))  # 35328
names = model_first.feature_names_in_
print(pd.Series(model_first.coef_, index=names).round(0))
same = np.allclose(model_first.predict(X_first), model.predict(X_train))
print(same)                           # True

area_sqm                    3467.0
neighborhood_Eastgate     179985.0
neighborhood_Hillcrest   -121427.0
neighborhood_Lakeview     108959.0
neighborhood_Northend    -112033.0
neighborhood_Riverside     28112.0
neighborhood_Southgate    -49959.0
dtype: float64

drop="first" drops the column of the first category, Downtown, so a Downtown listing is a row of six zeros. The six columns no longer add up to the column of ones, the rank is 8 of 8, and for LinearRegression not one prediction on the training rows moved. The weights now have one reading each. Downtown is the baseline category (the category whose column is dropped, so the intercept carries it): a Downtown listing is priced at 35,328 plus 3,467 per square meter, and every other weight is what its neighborhood adds over a Downtown listing of the same area, 179,985 for Eastgate and minus 121,427 for Hillcrest. That reading has a cost. Downtown has no weight of its own, every comparison runs through it, and "first" chose it by the alphabet. On the test rows the encoder runs transform and never fit, as 3.2.1 set out:

test_hood = first_enc.transform(test[["neighborhood"]])
X_test = pd.concat([test[["area_sqm"]], test_hood], axis=1)
test_pred = model_first.predict(X_test)
mae = np.abs(y_test - test_pred).mean()
print(round(mae))  # 54521
by_hood = (y_test - test_pred).groupby(test["neighborhood"]).mean()
print(by_hood.sort_values().round(0))

neighborhood
Eastgate    -32712.0
Downtown     -6976.0
Southgate    -2694.0
Hillcrest     1417.0
Lakeview      1788.0
Northend      4452.0
Riverside     4505.0
Name: price, dtype: float64

On the same 575 held-out listings, area alone missed by 87,952 on average in Lesson 3.1.4, and area with the neighborhood misses by 54,521. The neighborhood means of the residuals, which ran from -120,177 to 145,253 there, now run from -32,712 to 4,505. pd.get_dummies (a pandas function that replaces each text column of a frame with one True or False column per category it finds) looks like a shortcut to the same columns:

dummies_train = pd.get_dummies(train[["area_sqm", "neighborhood"]])
dummies_test = pd.get_dummies(test[["area_sqm", "neighborhood"]])
# True
print(dummies_train.columns.equals(dummies_test.columns))
by_dummies = LinearRegression().fit(dummies_train, y_train)

posted = pd.DataFrame({"area_sqm": [85.0], "neighborhood": ["Lakeview"]})
print(pd.get_dummies(posted).columns.tolist())
try:
    by_dummies.predict(pd.get_dummies(posted))
except ValueError as err:
    print(err)

['area_sqm', 'neighborhood_Lakeview']
The feature names should match those that were passed during fit.
Feature names seen at fit time, yet now missing:
- neighborhood_Downtown
- neighborhood_Eastgate
- neighborhood_Hillcrest
- neighborhood_Northend
- neighborhood_Riverside
- ...

get_dummies reads only the rows it is handed. Every neighborhood landed on both sides of the split, so the training and test frames came out with the same eight columns. A listing posted this morning holds one neighborhood and gets one neighborhood column, and predict refuses it with 3.1.1's message, which lists five of the six missing names and cuts the rest to - .... The encoder learned its columns from the training rows once and writes all of them for any rows:

posted_hood = first_enc.transform(posted[["neighborhood"]])
X_posted = pd.concat([posted[["area_sqm"]], posted_hood], axis=1)
posted_price = model_first.predict(X_posted)
print(X_posted.shape)          # (1, 7)
print(np.round(posted_price))  # [438979.]

DON'T:

• DON'T run pd.get_dummies on each set of rows separately. The training and test frames matched only because every neighborhood landed on both sides, and one new listing got two columns where the model expected eight.
• DON'T handle the encoder's default output as an array. It is a sparse matrix, np.array wraps it as one object of shape (), and np.column_stack refuses it, so pass sparse_output=False or call .toarray().
• DON'T read the weights of all seven columns as seven separate effects. Beside an intercept they are one choice among endless sets that predict alike, and the choice LinearRegression returns measures each neighborhood against the average of the seven.

Categories the Training Rows Never Had

What. A category the training rows never had has no place in the encoder's categories_. OneHotEncoder's default, handle_unknown="error", refuses a row holding one at transform and names the category, and handle_unknown="ignore" encodes it as a row of zeros. Why. A fitted encoder knows only the categories its training rows held. A neighborhood that opened after the fit, a typo and a new spelling all reach it as unknown categories, and the setting decides whether the prediction stops or a price comes out anyway. When. Whenever a fitted encoder meets rows that were not its training rows: the test rows, a batch from the listings form, one listing posted this morning. The test rows passed above because every neighborhood in them is among the training rows' seven.

batch = pd.DataFrame({
    "area_sqm": [85.0, 85.0],
    "neighborhood": ["Westbrook", "Eastgate"],
})
try:
    first_enc.transform(batch[["neighborhood"]])
except ValueError as err:
    print(err)

typo = pd.DataFrame({"area_sqm": [85.0], "neighborhood": ["eastgate"]})
try:
    first_enc.transform(typo[["neighborhood"]])
except ValueError as err:
    print(err)

Found unknown categories ['Westbrook'] in column 0 during transform
Found unknown categories ['eastgate'] in column 0 during transform

Westbrook is a neighborhood no training listing is in, and eastgate is Eastgate typed in lowercase, a different string as far as the encoder is concerned. The message names the category and its column, column 0, the position of neighborhood among the columns the encoder was fitted on, its only one. Nothing comes back for any row of the batch, the Eastgate listing included. This is the loud failure: the prediction stops and says where to look. The other setting:

new = pd.DataFrame({
    "area_sqm": [85.0, 85.0, 85.0],
    "neighborhood": ["Westbrook", "eastgate", "Eastgate"],
})
lenient = OneHotEncoder(sparse_output=False, handle_unknown="ignore")
lenient.set_output(transform="pandas")
lenient.fit(train[["neighborhood"]])
zeros = lenient.transform(new[["neighborhood"]])
print(zeros.to_numpy())
X_new = pd.concat([new[["area_sqm"]], zeros], axis=1)
# known neighborhoods encode as before, so model applies
print(np.round(model.predict(X_new)))
# 334826, the intercept and the area weight alone
print(round(model.intercept_ + model.coef_[0] * 85))

[[0. 0. 0. 0. 0. 0. 0.]
 [0. 0. 0. 0. 0. 0. 0.]
 [0. 1. 0. 0. 0. 0. 0.]]
[334826. 334826. 510005.]

Neither unknown row raised. Each became seven zeros, which switch every neighborhood weight off and leave the intercept and the area weight alone. With the seven weights summing to zero, that is the plain average of the seven neighborhood prices for 85 square meters, a price for no neighborhood in particular, and Westbrook and the mistyped Eastgate get it alike. The real Eastgate listing beside them is priced at 510,005, so the typo cost its listing 175,179, and no warning was printed.

That is the cost of "ignore": an unknown category is priced the same way whatever it is, a new neighborhood and a typo alike, and nothing in the prices marks which listings it happened to. Keep the default wherever nothing says a new category is expected. Its error is the encoder refusing to guess, and the fix belongs where the category came from: a typo is corrected in the input, and a new neighborhood waits for listings to fit on, or is priced by a rule someone chose and wrote down. "ignore" is one such rule, chosen when new categories are expected and a price without their effect is acceptable, and never chosen because the default raised.

DON'T:

• DON'T switch to handle_unknown="ignore" to make the error go away. The error named Westbrook and eastgate, and "ignore" priced both at 334,826 without a word.
• DON'T expect the known rows of a batch to come back when one row is unknown. transform raised for the whole batch, and the Eastgate listing beside Westbrook got no columns either.
• DON'T fit the encoder on every row so that no category can be unknown. The held-out rows' categories then reach a step before the fit, which 3.1.2's held-out rule forbids, and a listing posted after the fit can still bring a category no row had.

Ordinal Encoding and Its Order

What. Ordinal encoding replaces each category with one number, its position in a list of the categories. OrdinalEncoder, from sklearn.preprocessing, learns the list in fit, alphabetical unless categories= gives one, and transform writes one column of positions for each column it was fitted on. Why. A category whose levels rank, condition from poor to excellent, can reach a model as one column whose numbers keep that rank, where one-hot spends a column per level and keeps no rank at all. When. For a category whose levels have a real order, with the order written out in categories=, and, for a linear model, never for one without, such as a neighborhood.

from sklearn.preprocessing import OrdinalEncoder

alphabetical = OrdinalEncoder().fit(train[["condition"]])
print(alphabetical.categories_)
levels = pd.DataFrame({"condition": ["poor", "fair", "good", "excellent"]})
print(alphabetical.transform(levels)[:, 0])  # [3. 1. 2. 0.]

ranked = OrdinalEncoder(categories=[["poor", "fair", "good", "excellent"]])
ranked.fit(train[["condition"]])
print(ranked.transform(levels)[:, 0])        # [0. 1. 2. 3.]

[array(['excellent', 'fair', 'good', 'poor'], dtype=object)]

With no list, the positions follow the alphabet, as 2.5.3's codes did, so excellent is 0, poor is 3, and fair sits above excellent. categories= takes one list per column, in the order the positions should run, and 2.2.2's [:, 0] takes the one column transform returned. Each encoding gives a linear model one weight per step:

def fit_and_score(enc, column):
    # area beside one encoded column, scored by test MAE
    X_tr = np.column_stack([train["area_sqm"], enc.transform(train[[column]])])
    X_te = np.column_stack([test["area_sqm"], enc.transform(test[[column]])])
    fitted = LinearRegression().fit(X_tr, y_train)
    mae = np.abs(y_test - fitted.predict(X_te)).mean()
    return fitted, round(mae)


for enc in [alphabetical, ranked]:
    fitted, mae = fit_and_score(enc, "condition")
    print(round(fitted.coef_[1]), mae)

-23538 87885
37065 86846

The alphabetical codes run excellent, fair, good, poor, a sequence that ranks nothing, and the fit gives it 23,538 off per step, a weight on an order that does not exist. The ranked codes take 37,065 per step and the lower test error. That single weight is what ordinal codes assume in a linear model: every step between neighboring levels is worth the same amount. One-hot with the same categories= list, which OneHotEncoder takes too, and poor as the baseline, lets each level take its own amount:

hot = OneHotEncoder(
    categories=[["poor", "fair", "good", "excellent"]],
    drop="first",
    sparse_output=False,
)
hot.fit(train[["condition"]])
fitted, mae = fit_and_score(hot, "condition")
w = fitted.coef_[1:]
print(hot.get_feature_names_out())
print(np.round(w))  # [ 57628.  85136. 123014.]
# [57628. 27508. 37878.], each level over the one below
print(np.round([w[0], w[1] - w[0], w[2] - w[1]]))
print(mae)          # 86722

['condition_fair' 'condition_good' 'condition_excellent']

The steps come out at 57,628 from poor to fair, 27,508 from fair to good and 37,878 from good to excellent, nowhere near equal, and the test error falls a little further, to 86,722. Equal steps is a modeling choice, one weight in place of three, to be made knowingly. A neighborhood has no order to keep, and its codes show what a model does with positions:

codes = train["neighborhood"].astype("category").cat.codes
positions = OrdinalEncoder().fit_transform(train[["neighborhood"]])
# True, the same numbers as 2.5.3's codes
print(np.array_equal(positions[:, 0], codes))

appearance = [train["neighborhood"].unique().tolist()]
for order in ["auto", appearance]:
    enc = OrdinalEncoder(categories=order).fit(train[["neighborhood"]])
    fitted, mae = fit_and_score(enc, "neighborhood")
    print(round(fitted.coef_[1]), mae)

-10978 87160
26778 83636

OrdinalEncoder with no list writes 2.5.3's category codes, and a model fitted on them takes one weight per position, minus 10,978, as if Eastgate sat between Downtown and Hillcrest on some scale. In the order the neighborhoods first appear in the training rows, the order unique() returns, the same model takes 26,778 per position and a different test error, 83,636 against 87,160. Two orders nobody chose for a reason gave two models, because a code is a position in a list, 2.5.3's point, and one-hot, which imposes no order, reached 54,521 on the same rows.

DON'T:

• DON'T leave out categories= for an ordered scale. With no list, condition came out excellent 0, fair 1, good 2, poor 3.
• DON'T ordinal-encode a category with no order for a linear model. Two arbitrary orders of the neighborhoods gave test errors of 87,160 and 83,636, where one-hot gave 54,521.
• DON'T expect drop="first" to drop the lowest level. It drops the first category in categories_, alphabetical unless categories= sets the order, so on condition it drops excellent and keeps condition_fair, condition_good and condition_poor.

Missing Values in a Category

What. A gap in a category column is a missing value, shown as NaN in a pandas str column. OneHotEncoder treats it as one more category, listed last in categories_ with a column of its own, and OrdinalEncoder passes it through as nan, a gap LinearRegression then refuses. Why. Whether a listing has a rating at all can say something about the listing, and the encoder decides whether the model gets to see it. When. For any category column with gaps, energy_rating here, and before dropping a single row for having one.

print(train["energy_rating"].head(3))
print(train["energy_rating"].isna().sum())  # 198

rating = OneHotEncoder(sparse_output=False).fit(train[["energy_rating"]])
print(rating.categories_)
print(rating.get_feature_names_out()[-1])   # energy_rating_nan

1653      A
1240    NaN
632       A
Name: energy_rating, dtype: str
[array(['A', 'B', 'C', 'D', 'E', 'F', 'G', nan], dtype=object)]

198 of the 2,300 training listings have no rating, listing 1240 among them. pandas prints the gap as NaN, the encoder lists it as nan after the seven ratings, and it gets a column of its own, energy_rating_nan. pd.get_dummies does the opposite by default:

gaps = pd.get_dummies(train[["energy_rating"]])
print(gaps.shape[1])                  # 7
print((gaps.sum(axis=1) == 0).sum())  # 198

Seven columns, one per rating, and 198 rows of all False: the gap has no column, and a listing with no rating looks like a listing with no category at all. pd.get_dummies(..., dummy_na=True) gives the gap a column. An ordinal list meets the gap differently:

grades = ["G", "F", "E", "D", "C", "B", "A"]
try:
    OrdinalEncoder(categories=[grades]).fit(train[["energy_rating"]])
except ValueError as err:
    print(err)

graded = OrdinalEncoder(categories=[grades + [np.nan]])
grade_codes = graded.fit_transform(train[["energy_rating"]])
print(np.isnan(grade_codes).sum())  # 198
X_grade = np.column_stack([train["area_sqm"], grade_codes])
try:
    LinearRegression().fit(X_grade, y_train)
except ValueError as err:
    print(str(err).splitlines()[0])

Found unknown categories [nan] in column 0 during fit
Input X contains NaN.

A list given to categories= has to hold every value the training rows hold, and the gap is one of them, so the seven ratings alone are refused with the unknown-category message, during fit this time. scikit-learn accepts np.nan only as the last entry of the list. With it there, the codes run from G at 0 to A at 6, and a gap stays a gap: 198 codes are nan, and LinearRegression refuses them as it refused the missing areas in 3.1.1. Lesson 3.2.3 fills such gaps from the training rows. Whether the gap is worth keeping is a question for the data:

built = train.dropna(subset=["year_built"])
old = built["year_built"] < 1960
missing = built["energy_rating"].isna()
print(old.sum(), (~old).sum())         # 231 1957
print(round(missing[old].mean(), 3))   # 0.216
print(round(missing[~old].mean(), 3))  # 0.068

Among the training listings with a known construction year, 21.6 percent of the 231 built before 1960 have no rating, against 6.8 percent of the 1,957 built from 1960 on. A rating is missing about three times as often on an older building as on a newer one, so the gap carries information, and energy_rating_nan hands it to the model as a column of its own. Lesson 3.2.3's missing-value indicator keeps the same kind of fact for a numeric column.

DON'T:

• DON'T drop the listings with no rating as noise. 21.6 percent of the training listings built before 1960 have none, against 6.8 percent of the newer ones, so the drop removes an older building about three times as often as a newer one.
• DON'T fit the encoder on the rated listings alone. It then has no nan category, and the first listing without a rating stops transform with Found unknown categories [nan] in column 0 during transform.
• DON'T use pd.get_dummies on a column with gaps at its default. It turned 198 missing ratings into rows of all False, with no column to say so.

Lab Exercises

Lab Exercise 1: Property Type, One Column per Type

In property_type.py, load listings.csv, keep the listings that have an area, and split the table with test_size=0.2 and random_state=42, as the Theory did.

Fit LinearRegression on area_sqm and property_type inside a try, and print the message. Fit a default OneHotEncoder on the training rows' property_type, and print its categories, its feature names and the type of what its transform returns.

Then use a dense encoder with pandas output. Show that every training row's type columns add up to 1, and print the column count and the rank of a design matrix that holds a column of ones, the areas and the type columns.

Finally, fit price on area and the type columns twice, with all three columns and with drop="first". Print both intercepts and the second model's weights, check that the two models give the same test predictions, print the test MAE to set beside area alone's 87,952, and say in a comment what the second intercept is the price of.

Lab Exercise 2: Energy Ratings in Order, as Columns, and Unknown

In ratings.py, with the rows and split of Lab Exercise 1, encode energy_rating three ways.

First, as ordinal codes in the real order from G, the lowest, to A, the highest, with room for the gap. Print the codes for G, D, A and a missing rating, then fit LinearRegression on area and the codes inside a try, print the first line of the message, and say in a comment what has to happen before the fit.

Second, one-hot, dense, with pandas output and drop="first". Print the feature names, fit price on area and the rating columns, and print the weights and the test MAE. Say in a comment which rating the weights are measured from and where the gap's weight falls among the ratings'.

Third, a teammate fits the same one-hot encoder on the training listings that have a rating, and on nothing else. Transform the test rows with it inside a try and print the message. Then send a listing of 85.0 square meters rated "A+", a grade no listing has, through your own one-hot encoder, and print that message too.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

listings = pd.read_csv("listings.csv")
data = listings.dropna(subset=["area_sqm"])
train, test = train_test_split(data, test_size=0.2, random_state=42)
y_train, y_test = train["price"], test["price"]

try:
    LinearRegression().fit(train[["area_sqm", "property_type"]], y_train)
except ValueError as err:
    print(err)  # could not convert string to float: 'apartment'

default = OneHotEncoder().fit(train[["property_type"]])
print(default.categories_)
print(default.get_feature_names_out())
print(type(default.transform(train[["property_type"]])))

[array(['apartment', 'house', 'townhouse'], dtype=object)]
['property_type_apartment' 'property_type_house' 'property_type_townhouse']
<class 'scipy.sparse._csr.csr_matrix'>

full = OneHotEncoder(sparse_output=False).set_output(transform="pandas")
types_train = full.fit_transform(train[["property_type"]])
print(types_train.sum(axis=1).unique())  # [1.]
ones = np.ones(len(train))
design = np.column_stack([ones, train["area_sqm"], types_train])
# 5 4, the three type columns add up to the column of ones
print(design.shape[1], np.linalg.matrix_rank(design))

first = OneHotEncoder(sparse_output=False, drop="first")
first.set_output(transform="pandas")
first.fit(train[["property_type"]])


def with_area(encoder, rows):
    encoded = encoder.transform(rows[["property_type"]])
    return pd.concat([rows[["area_sqm"]], encoded], axis=1)


full_model = LinearRegression().fit(with_area(full, train), y_train)
first_model = LinearRegression().fit(with_area(first, train), y_train)
# -1647 24312
print(round(full_model.intercept_), round(first_model.intercept_))
weights = pd.Series(first_model.coef_, index=first_model.feature_names_in_)
print(weights.round(0))

area_sqm                    3812.0
property_type_house       -69490.0
property_type_townhouse    -8385.0
dtype: float64

full_pred = full_model.predict(with_area(full, test))
first_pred = first_model.predict(with_area(first, test))
mae = np.abs(y_test - first_pred).mean()
print(np.allclose(full_pred, first_pred))  # True
print(round(mae))                          # 87822
# 24312 is the apartment line at zero square meters, a listing that
# does not exist: an apartment is priced 24312 plus 3812 per square
# meter, a house 69490 less and a townhouse 8385 less at that area.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

listings = pd.read_csv("listings.csv")
data = listings.dropna(subset=["area_sqm"])
train, test = train_test_split(data, test_size=0.2, random_state=42)
y_train, y_test = train["price"], test["price"]

grades = ["G", "F", "E", "D", "C", "B", "A"]
ordinal = OrdinalEncoder(categories=[grades + [np.nan]])
ordinal.fit(train[["energy_rating"]])
probe = pd.DataFrame({"energy_rating": ["G", "D", "A", np.nan]})
print(ordinal.transform(probe)[:, 0])  # [ 0.  3.  6. nan]
codes = ordinal.transform(train[["energy_rating"]])
X_codes = np.column_stack([train["area_sqm"], codes])
try:
    LinearRegression().fit(X_codes, y_train)
except ValueError as err:
    print(str(err).splitlines()[0])    # Input X contains NaN.
# the 198 gaps stay nan, and LinearRegression needs them filled
# first, from the training rows, which Lesson 3.2.3 does

rating = OneHotEncoder(sparse_output=False, drop="first")
rating.set_output(transform="pandas")
rating.fit(train[["energy_rating"]])
print(rating.get_feature_names_out())


def with_area(encoder, rows):
    encoded = encoder.transform(rows[["energy_rating"]])
    return pd.concat([rows[["area_sqm"]], encoded], axis=1)


model = LinearRegression().fit(with_area(rating, train), y_train)
print(pd.Series(model.coef_, index=model.feature_names_in_).round(0))
test_pred = model.predict(with_area(rating, test))
mae = np.abs(y_test - test_pred).mean()
print(round(mae))  # 87496
# Measured from A, the first rating and the baseline. The gap's
# -41388 falls between C at -33272 and D at -48837.

['energy_rating_B' 'energy_rating_C' 'energy_rating_D' 'energy_rating_E'
 'energy_rating_F' 'energy_rating_G' 'energy_rating_nan']
area_sqm              3295.0
energy_rating_B      -7787.0
energy_rating_C     -33272.0
energy_rating_D     -48837.0
energy_rating_E     -67482.0
energy_rating_F     -77224.0
energy_rating_G     -87856.0
energy_rating_nan   -41388.0
dtype: float64

rated = train.dropna(subset=["energy_rating"])
strict = OneHotEncoder(sparse_output=False, drop="first")
strict.fit(rated[["energy_rating"]])
try:
    strict.transform(test[["energy_rating"]])
except ValueError as err:
    print(err)

a_plus = pd.DataFrame({"area_sqm": [85.0], "energy_rating": ["A+"]})
try:
    with_area(rating, a_plus)
except ValueError as err:
    print(err)

Found unknown categories [nan] in column 0 during transform
Found unknown categories ['A+'] in column 0 during transform

Theory Assessment

Q1: Seven Columns Beside an Intercept

LinearRegression refused area and neighborhood with could not convert string to float: 'Downtown'. Explain why it refuses text and why the message names Downtown. Then explain why a design matrix of a column of ones, the areas and the seven one-hot neighborhood columns has 9 columns and rank 8, why 1,000 added to every neighborhood weight and 1,000 taken from the intercept leave every prediction where it was, and what drop="first" changes in the rank, in the weights and in the predictions.

Hint: Ask what predict does with each value it is handed, and which training row the fit reads first. For the rank, add up the seven columns of any one row.

Q2: Rows the Encoder Was Not Fitted On

A listing posted this morning, 85.0 square meters in Lakeview, is refused by the model fitted on pd.get_dummies columns and priced at 438,979 through the encoder. Explain the difference. Then say what the default encoder did with a batch holding a Westbrook listing and an Eastgate listing, and what the seven-column encoder with handle_unknown="ignore" did with Westbrook and with "eastgate", using the numbers the lesson printed. Finish with why the lesson keeps the default.

Hint: Ask what each tool knows about the categories when it meets one listing. For "ignore", ask which weights a row of zeros switches off and what is left of the prediction.

Q3: Codes, Steps and a Gap

Explain why OrdinalEncoder() coded condition as excellent 0, fair 1, good 2 and poor 3, what categories= changed, and what the single weight of 37,065 per step assumes that the one-hot weights of 57,628, 85,136 and 123,014 against poor do not. Then explain why OrdinalEncoder(categories=[grades]) with the seven ratings was refused at fit time, what OneHotEncoder and pd.get_dummies each did with the 198 missing ratings, and why the gap is worth a column.

Hint: For the steps, subtract each one-hot weight from the next. For the refusal, compare the values the training rows hold with the values the list names.

Q4: A Baseline Meets an Unknown

A teammate wants the readable weights of drop="first" and a model that never stops on a new neighborhood, and saves this as baseline_unknown.py:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

listings = pd.read_csv("listings.csv")
data = listings.dropna(subset=["area_sqm"])
train, test = train_test_split(data, test_size=0.2, random_state=42)

encoder = OneHotEncoder(
    drop="first", handle_unknown="ignore", sparse_output=False
)
encoder.set_output(transform="pandas")
hood = encoder.fit_transform(train[["neighborhood"]])
X_train = pd.concat([train[["area_sqm"]], hood], axis=1)
model = LinearRegression().fit(X_train, train["price"])

new = pd.DataFrame({
    "area_sqm": [85.0, 85.0],
    "neighborhood": ["Westbrook", "Downtown"],
})
hood_new = encoder.transform(new[["neighborhood"]])
X_new = pd.concat([new[["area_sqm"]], hood_new], axis=1)
print(np.round(model.predict(X_new)))

The teammate expects Westbrook to come out at 334,826, the price for no neighborhood in particular that the lesson's "ignore" gave, and reads a run that raises nothing as a clean one. Read the script without running it, say what it prints, and say where each expectation goes wrong.

Hint: Write out the row the encoder gives a Downtown listing, then the row it gives Westbrook. For 334,826, ask which encoder the lesson's "ignore" was set on and what a row of zeros meant there.

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/sklearn/preprocessing/_encoders.py:262: UserWarning: Found unknown categories in columns [0] during transform. These unknown categories will be encoded as all zeros
  warnings.warn(msg, UserWarning)
[330020. 330020.]

Assignment

Assignment: Pricing a Grade No Listing Has

The rating agency is adding a grade above A, "A+", and the listings form will offer it before any listing in listings.csv carries one. Until such listings exist, the pricing team has to decide what its model does with an A+ listing, and it wants that decision made on numbers, not on the first setting that stops an error.

Write rating_policy.py. On the Theory's rows and split, model price from area_sqm and a dense one-hot energy_rating that keeps the gap as a column and drops no category, under three encoders: the default, handle_unknown="ignore", and one the lesson did not show. OneHotEncoder(min_frequency=100, handle_unknown="infrequent_if_exist") merges every category the training rows hold fewer than 100 times into one shared column, named energy_rating_infrequent_sklearn, records the merged categories in its learned attribute infrequent_categories_, and sends a category it never saw into that shared column.

For each encoder, print the test MAE and the prices of four listings of 85.0 square meters rated "A+", "A", "G" and not rated, or, where a listing is refused, the message and the other three prices. Print the categories the third encoder merged, and check the "ignore" price of A+ against the prices its model gives the eight categories it knows. Then write the rule you would ship instead, as a change made to the rating before the encoder sees it, price the four listings under it, and end with a comment saying what each encoder priced A+ as, what each choice costs, and which you would ship.

Hints:

• value_counts(dropna=False) on the training ratings shows which categories fall under 100, the gap included, before the third encoder decides it.
• One function that sets the area beside an encoder's columns for any rows keeps the three models identical in everything but the encoder. predict cannot run on a frame the encoder refused to build, so price the other three listings on their own.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

listings = pd.read_csv("listings.csv")
data = listings.dropna(subset=["area_sqm"])
train, test = train_test_split(data, test_size=0.2, random_state=42)
y_train, y_test = train["price"], test["price"]
print(train["energy_rating"].value_counts(dropna=False).to_dict())

policies = {
    "error": OneHotEncoder(sparse_output=False),
    "ignore": OneHotEncoder(sparse_output=False, handle_unknown="ignore"),
    "infrequent": OneHotEncoder(
        sparse_output=False,
        min_frequency=100,
        handle_unknown="infrequent_if_exist",
    ),
}
four = pd.DataFrame({
    "area_sqm": [85.0, 85.0, 85.0, 85.0],
    "energy_rating": ["A+", "A", "G", np.nan],
})


def with_area(encoder, rows):
    encoded = encoder.transform(rows[["energy_rating"]])
    return pd.concat([rows[["area_sqm"]], encoded], axis=1)


models = {}
for name, encoder in policies.items():
    encoder.set_output(transform="pandas")
    encoder.fit(train[["energy_rating"]])
    models[name] = LinearRegression().fit(with_area(encoder, train), y_train)
    test_pred = models[name].predict(with_area(encoder, test))
    print(name, round(np.abs(y_test - test_pred).mean()))
    try:
        print(np.round(models[name].predict(with_area(encoder, four))))
    except ValueError as err:
        print(err)
        rest = with_area(encoder, four.iloc[1:])
        print(np.round(models[name].predict(rest)))

{'C': 510, 'B': 489, 'A': 460, 'D': 384, nan: 198, 'E': 179, 'F': 62, 'G': 18}
error 87496
Found unknown categories ['A+'] in column 0 during transform
[364114. 276258. 322726.]
ignore 87496
[318633. 364114. 276258. 322726.]
infrequent 87538
[284498. 364108. 284498. 322722.]

merged = policies["infrequent"]
print(merged.infrequent_categories_)
print(merged.get_feature_names_out()[-1])

eight = pd.DataFrame({
    "area_sqm": [85.0] * 8,
    "energy_rating": ["A", "B", "C", "D", "E", "F", "G", np.nan],
})
eight_prices = models["ignore"].predict(with_area(policies["ignore"], eight))
print(np.round(eight_prices))
print(round(eight_prices.mean()))  # 318633

mapped = four.copy()
mapped.loc[mapped["energy_rating"] == "A+", "energy_rating"] = "A"
print(np.round(models["error"].predict(with_area(policies["error"], mapped))))
# error: A+ refused by name, and nothing priced in its place.
# ignore: 318633, the plain average of the eight category prices,
#   between C and D and 45481 below A, with no warning.
# infrequent: 284498, the merged F and G column, the two lowest
#   grades, 79610 below A, with no warning.
# Ship the default with the mapping above: A+ priced as A, 364114,
# and the cost written beside the model, whatever A+ adds over A
# left unpriced until A+ listings exist to fit on.

[array(['F', 'G'], dtype=object)]
energy_rating_infrequent_sklearn
[364114. 356327. 330842. 315277. 296632. 286890. 276258. 322726.]
[364114. 364114. 276258. 322726.]

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want to go past one column per category.

• Official docs - scikit-learn 1.9 User Guide, "Preprocessing data", the section "Infrequent categories". min_frequency and max_categories, which merge the rarest categories into one output, in OneHotEncoder and OrdinalEncoder alike, and what each does with a category it never saw.
• Google Machine Learning Crash Course - "Categorical data: Feature crosses". One column for every pair of categories from two features, which lets a linear model give each pair its own weight, and how fast the column count grows.

Summary

A category reaches a model only as numbers, and the encoder decides which numbers, and with them what the model is allowed to believe. One-hot encoding gave each neighborhood a column and let the data set its amount, and the average miss on the same 575 listings fell from 87,952 to 54,521. Ordinal codes gave each category a single number and with it an order, the alphabet's unless someone wrote the real one down, and one weight per step that treats every step as the same size.

The columns carried a cost the predictions never showed. Seven columns that add up to the column of ones left the weights without a single meaning, and dropping one bought readable weights by making one category a baseline that every other is measured from. For every neighborhood the encoder knew, the predictions of LinearRegression did not move either way, so the choice decides how the weights read, and, once "ignore" lets an unknown category through, whose price it gets.

An encoder is fitted, and fitted on the training rows, so its list of categories is all it knows. The default's error at an unknown category is that limit stated plainly, and every setting that silences it prices the listing as something else: the plain average of the neighborhoods, the baseline, or a merged group of the rarest grades. The gap in a rating told the same story from the other side, a missing value that looked like noise and turned up about three times as often on an older building as on a newer one, and an encoder that gave it a column let the model use it. The next lesson assembles filling, scaling and encoding into one object, fitted in one call on the training rows.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

enc = OneHotEncoder(sparse_output=False, drop="first")
enc.set_output(transform="pandas")      # a DataFrame out
# fit on the training rows, once
hood_train = enc.fit_transform(train[["neighborhood"]])
# transform, never fit, every other row
hood_test = enc.transform(test[["neighborhood"]])
enc.categories_                         # learned, sorted
enc.get_feature_names_out()             # neighborhood_Eastgate, ...
X = pd.concat([train[["area_sqm"]], hood_train], axis=1)

OneHotEncoder(handle_unknown="ignore")  # unknowns become a row of zeros
levels = [["poor", "fair", "good", "excellent"]]
OrdinalEncoder(categories=levels)       # poor 0 up to excellent 3
grades = [["G", "F", "E", "D", "C", "B", "A", np.nan]]
OrdinalEncoder(categories=grades)       # nan last, stays nan