Apply and Element-wise Transformation

Context

A transformation builds a new column out of the columns a table already has. It can be one expression over whole columns, a lookup that swaps each value for another, a Python function of your own called once per value or once per row, or a rule that drops each number into a band between two edges.

Most features a model learns from are made this way, and the model this section feeds predicts how long a listing stays on the market, where the listing's own price is a fair input. Price per square meter is two columns divided, the condition word becomes an ordered score a model can compare, a build year becomes an age, and a price becomes a budget, mid or premium band you can count listings in, or the quartile band Module 3 splits the data on.

Most failures here produce a full, plausible column rather than an error. A lookup missing one label turns every listing with that label into a missing value, a band whose edges stop short loses the listings beyond them, and a missing value handed to your own function either crashes it or quietly takes a branch written for real values. A per-row function written where one column expression would do costs a Python call for every row and buys nothing. Section 2.5 cleaned the table, and this lesson starts building on it.

Goal

• Build features with column arithmetic, np.where and np.select before reaching for a Python function.
• Map a column through a dictionary or a function with Series.map, and count the values the lookup failed to match.
• Apply a named function or a lambda to every value with Series.apply, guarded against missing values.
• Apply a function across rows with DataFrame.apply(axis=1), and predict what the default axis=0 hands it instead.
• Bin a numeric column with pd.cut and pd.qcut, and account for every value that falls outside the edges.

Prerequisites

Optional, and the lesson stands without them. They are here if you want to watch a function reach every value of a table before you write one.

• YouTube - "How do I apply a function to a pandas Series or DataFrame?" by Data School. It was recorded on an older pandas, so its applymap is the method pandas 3.0.5 calls DataFrame.map.
• Official docs - pandas User Guide, "Essential basic functionality", the "Function application" section. The split between apply, map and the per-cell form, with the axis argument.

Table of Contents

• Vectorized First
• Series.map
• Series.apply and lambda
• DataFrame.apply Across Rows
• Binning with cut and qcut

Theory

Nothing to install, (bootcamp) has pandas from Lesson 2.4.1 and NumPy from Lesson 2.2.1. Every Theory block below goes into one script, features.py, saved in ~/dev/bootcamp beside housing.csv and run with python3 features.py, each block added under the one before it, so a name one block creates is there for the next.

Vectorized First

What. A vectorized expression computes a new column in one operation over whole columns: arithmetic, a comparison, 2.2.2's np.where, or np.select, which takes a list of conditions and a list of choices, gives each row the choice belonging to the first condition it meets, and gives a row that meets none the default. Why. The loop runs inside NumPy's compiled code, as 2.2.3 showed, a missing value passes through arithmetic, as NaN, as <NA> in a nullable column or as NaT in a date column, with no guard written, and the line reads as the formula it computes. When. First, for every feature that is arithmetic, a comparison or a choice among a few conditions. A Python function called per value, the subject of most of the rest of this lesson, is for logic none of these can express.

The first block loads the course file and repeats the cleaning this lesson needs: Lesson 2.5.4's duplicate rows go, then the four listings carrying the injected extremes. A > comparison with NaN is False, 2.4.2's trap, so ~extreme keeps every listing with a gap, and reset_index(drop=True) renumbers what is left. The gaps stay on purpose. Lesson 2.5.1 drops the unpriced listings before a model trains and labels the missing condition unknown, and this lesson keeps both kinds of gap so that every tool shows what it does with one.

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
extreme = (
    (df["price"] > 2_000_000)
    | (df["area_sqm"] > 1_000)
    | (df["rooms"] > 20)
)
df = df[~extreme].reset_index(drop=True)
print(df.shape)  # (200, 7)

Arithmetic between two columns pairs them row by row, and a row missing either value gets NaN, or <NA> in a nullable column, so the count of finished values is the first thing to print.

df["price_per_sqm"] = df["price"] / df["area_sqm"]
# 166, the other 34 rows lack a price or an area
print(df["price_per_sqm"].notna().sum())

A choice between two labels is 2.2.2's np.where, used here on a pandas column exactly as it was on an array. to_dict() turns a Series into a dictionary keyed by its labels, so the counts print on one line.

df["layout"] = np.where(df["rooms"] >= 4, "family", "compact")
# {'compact': 102, 'family': 98}
print(df["layout"].value_counts().to_dict())

np.select checks the conditions in the order they are listed and gives each row the choice of the first one that holds, the way an if and elif chain stops at its first true branch, and a row that meets none takes default. A missing area fails every < test, so it gets a condition of its own, and that condition goes first.

conditions = [
    df["area_sqm"].isna(),
    df["area_sqm"] < 80,
    df["area_sqm"] < 120,
]
choices = ["unknown", "small", "medium"]
df["size_band"] = np.select(conditions, choices, default="large")
# {'medium': 92, 'large': 52, 'small': 41, 'unknown': 15}
print(df["size_band"].value_counts().to_dict())

A 70 square meter listing meets both < 80 and < 120 and comes out small, because that condition is checked first. np.select returns a NumPy array, and the column assignment takes it in row order. Leave the isna() condition out and the 15 listings with no area fail every test and fall to the default, filed as large with nothing to say so.

unguarded = np.select(conditions[1:], choices[1:], default="large")
print((unguarded == "large").sum())  # 67, the 52 large and the 15 with no area

DON'T:

• DON'T write a Python function for a feature that is arithmetic between columns, df["price"] / df["area_sqm"] already handles every row and every gap.
• DON'T let a missing value fall to np.select's default. Give isna() a condition of its own, or a NaN that fails every comparison is filed under whatever the default says.
• DON'T leave out default= when the choices are text. The default is then the number 0, and NumPy refuses to mix it with text, raising a TypeError that begins Choicelist and default value do not have a common dtype.

Series.map

What. Series.map(lookup) replaces every value of a column by looking it up. Given a dictionary, it returns the value stored under each key, and given a function, it returns what the function returns for each value. Why. A condition word is not something a model can compare, and a lookup turns condition into an ordered score in one line, the same rule for every listing. When. When the lookup covers every value the column holds. To change a few values and keep the rest, replace is the tool.

Given a dictionary, map looks every value up as a key. The one listing with no condition finds no key and comes back missing, which is also why the scores arrive as float64, 2.5.2's reason a gap turns integers into floats.

scores = {"poor": 1, "fair": 2, "good": 3, "excellent": 4}
df["condition_score"] = df["condition"].map(scores)
print(df["condition_score"].isna().sum())  # 1
print(df["condition_score"].dtype)         # float64

A value the dictionary lacks becomes missing the same way, and nothing warns. Leave one key out and every listing holding that label loses its value, and the isna().sum() after the map is the only line that tells you.

partial = {"poor": 1, "fair": 2, "good": 3}
# 34, the 33 excellent listings and the 1 gap
print(df["condition"].map(partial).isna().sum())

Given a function, map calls it on every value, the missing one included. str.title is the string method title named through its type, so it can be passed like any function, and it raises on the missing condition, which arrives as NaN, a float. na_action="ignore" skips missing values and leaves them missing.

try:
    df["condition"].map(str.title)
except TypeError as err:
    # TypeError: descriptor 'title' for 'str' objects doesn't apply to a 'float' object
    print(f"TypeError: {err}")

titled = df["condition"].map(str.title, na_action="ignore")
# {'Good': 90, 'Fair': 53, 'Excellent': 33, 'Poor': 23, nan: 1}
print(titled.value_counts(dropna=False).to_dict())

replace is the contrast. It changes the values it has a rule for and leaves every other value as it was, so folding the rare poor label into fair is a replace, while the same dictionary handed to map keeps only the one label it names.

folded = df["condition"].replace({"poor": "fair"})
# {'good': 90, 'fair': 76, 'excellent': 33, nan: 1}
print(folded.value_counts(dropna=False).to_dict())
# 177, every listing that was not poor
print(df["condition"].map({"poor": "fair"}).isna().sum())

DON'T:

• DON'T map through a dictionary without counting isna() before and after. A label the dictionary lacks becomes missing, and the count is the only witness.
• DON'T use map to change a few values, it keeps only the values its dictionary names, and replace is the edit that leaves the rest alone.
• DON'T hand a string method to map on a column with gaps without na_action="ignore", the missing value arrives as a float and raises.

Series.apply and lambda

What. Series.apply(func) calls a Python function once per value and collects what each call returns into a new Series. A lambda writes a one-expression function inline, lambda area: area * 10.7639, the form Module 1's Assignments passed to max and sorted as key=. Why. Logic with no vectorized form, a guard, a branch, a call into code that takes one value at a time, still has to reach every value, and apply is how it reaches them. When. After arithmetic, np.select, map and the accessors have all failed to say it. A lambda for one short expression, a named def for a guard or a branch.

apply calls the function once per value, and the list calls records each call to prove it.

calls = []

def to_sqft(area):
    calls.append(area)      # one entry per call
    return area * 10.7639

sqft = df["area_sqm"].apply(to_sqft)
print(len(calls))           # 200
vector = df["area_sqm"] * 10.7639
print(sqft.equals(vector))  # True

Two hundred calls for two hundred values, the 15 missing areas included: each arrived as NaN, and NaN * 10.7639 is NaN, so nothing broke. The vectorized line returned the same Series without calling a Python function once per value, so this conversion never needed apply.

A lambda writes the same function inline. The name before the colon is the parameter, and the one expression after it is the value returned.

sqft = df["area_sqm"].apply(lambda area: area * 10.7639)
print(sqft.equals(vector))  # True

A function that calls a string method is where a missing value bites. The listing with no condition arrives as NaN, a float, and the method does not exist on it.

try:
    df["condition"].apply(lambda c: c.upper())
except AttributeError as err:
    print(f"AttributeError: {err}")  # AttributeError: 'float' object has no attribute 'upper'

A lambda holds a single expression, so a guard there has to be squeezed into a conditional expression. A named function says it top to bottom and has room for the next rule. pd.isna(value) is the one-value form of 2.5.1's isna().

def condition_label(c):
    if pd.isna(c):
        return "UNKNOWN"
    return c.upper()

labels = df["condition"].apply(condition_label)
# {'GOOD': 90, 'FAIR': 53, 'EXCELLENT': 33, 'POOR': 23, 'UNKNOWN': 1}
print(labels.value_counts().to_dict())
# 1, the accessor passes the gap through
print(df["condition"].str.upper().isna().sum())

The last line is the vectorized form, 2.5.3's str accessor, which carries the missing value through untouched with no guard written. apply earns its place only for logic that arithmetic, np.select, map and the accessors cannot say.

DON'T:

• DON'T call a string method inside an applied function without a guard, the missing value is a float and raises AttributeError.
• DON'T stretch a lambda into nested conditional expressions to avoid a def, a rule with branches belongs in a named function.
• DON'T write a guarded function for what an accessor already does, .str.upper() carries the gap through with no function at all.

DataFrame.apply Across Rows

What. df.apply(func, axis=1) calls the function once per row and hands it that row as a Series whose index is the column names, so row["price"] reads one field. The default, axis=0, hands it one whole column per call instead. Why. A rule that reads several fields of the same listing gets all of them in one call, written as ordinary Python with if and and. When. For a rule over several columns that no mask, lookup or arithmetic expresses, which is rarer than it first looks.

With axis=1, each call receives one listing, and the function reads its fields by name.

def needs_renovation(row):
    old = row["year_built"] < 1970
    worn = row["condition"] in ("poor", "fair")
    return old and worn

flag = df.apply(needs_renovation, axis=1)
print(flag.sum())  # 23

This rule is two comparisons joined by and, so it also has a vectorized form, two masks joined by 2.2.2's &. The two agree on every row, the listing with no condition included, because neither in nor isin matches a NaN to a label.

masks = (df["year_built"] < 1970) & df["condition"].isin(["poor", "fair"])
print(flag.equals(masks))  # True

2.2.5's rule holds here unchanged: the axis you name is the one that disappears, so axis=0 returns one value per column and axis=1 one value per row.

numeric = df[["price", "area_sqm", "rooms"]]
spread = numeric.apply(lambda col: col.max() - col.min())
# {'price': 371000.0, 'area_sqm': 157.8, 'rooms': 5.0}
print(spread.round(1).to_dict())
present = numeric.apply(lambda row: row.notna().sum(), axis=1)
print(len(spread), len(present))  # 3 200

DataFrame.map is the per-cell form. It calls the function on every cell and returns a frame of the same shape, here formatting price and area for a report with Module 1's format spec, , grouping thousands and .0f rounding to no decimals. pandas 3.0.5 has no applymap, the name older tutorials use for the per-cell method.

shown = df[["price", "area_sqm"]].head(4).map(lambda v: f"{v:,.0f}")
print(shown)
try:
    df.applymap(lambda v: v)
except AttributeError as err:
    print(f"AttributeError: {err}")

     price area_sqm
0  437,000      132
1  268,000      112
2  268,000      102
3      nan       67
AttributeError: 'DataFrame' object has no attribute 'applymap'

Row 3 has no price, and the format spec turned its NaN into the text nan without an error, a report cell that looks like data.

DON'T:

• DON'T read axis backwards. axis=1 hands the function rows and axis=0 hands it columns.
• DON'T write applymap, pandas 3.0.5 raises AttributeError, and DataFrame.map is the per-cell method.
• DON'T format numbers with DataFrame.map and go on computing with the result, every cell is now text and a missing value reads nan.

Binning with cut and qcut

What. pd.cut(values, bins=edges) puts each number into the interval between two neighboring edges and returns 2.5.3's ordered category. pd.qcut(values, q=4) takes a number of bins instead and places the edges at the data's quantiles, so each bin holds about the same number of values. Why. A band is a category you can count, group and split on. Lesson 3.1.2 stratifies a train and test split on a band like this, dividing the rows so each band keeps nearly the same share of listings in the training rows and the test rows. When. cut when the edges mean something in the domain, a budget bracket or a size class. qcut when you want groups of equal size and the edges may fall wherever the data puts them.

pd.cut with a list of edges builds one interval between each neighboring pair. An interval includes its right edge and excludes its left, which is what (100000, 200000] says.

edges = [100_000, 200_000, 300_000, 500_000]
band = pd.cut(df["price"], bins=edges)
print(band.value_counts(sort=False))

price
(100000, 200000]    47
(200000, 300000]    69
(300000, 500000]    62
Name: count, dtype: int64

The one listing priced exactly 200,000 lands in the lower band, because that interval owns its right edge. The three bands hold 178 listings, three short of the 181 that have a price, and a value outside every interval becomes missing without a warning.

# [Interval(100000, 200000, closed='right')]
print(band[df["price"] == 200_000].tolist())
print(band.isna().sum())  # 22
lost = df.loc[band.isna() & df["price"].notna(), "price"]
print(lost.tolist())      # [506000.0, 510000.0, 511000.0]

Twenty-two missing: the 19 listings with no price, and the three priced above the last edge, gone from every band. labels= names the bands, one name per interval.

df["price_band"] = pd.cut(
    df["price"],
    bins=[100_000, 200_000, 300_000, 600_000],
    labels=["budget", "mid", "premium"],
)
# {'budget': 47, 'mid': 69, 'premium': 65}
print(df["price_band"].value_counts(sort=False).to_dict())
print(df["price_band"].isna().sum())  # 19

pd.qcut puts its edges at the quantiles, so q=4 cuts at the quartiles of 2.5.4.

df["price_quartile"] = pd.qcut(df["price"], q=4, labels=["q1", "q2", "q3", "q4"])
# {'q1': 46, 'q2': 45, 'q3': 46, 'q4': 44}
print(df["price_quartile"].value_counts(sort=False).to_dict())
# [197000.0, 245000.0, 377000.0]
print(df["price"].quantile([0.25, 0.5, 0.75]).tolist())

The counts come close without matching. 181 prices do not split into four equal groups, and two listings share each edge price, so both go into the bin below that edge.

DON'T:

• DON'T trust a cut to cover every value. A value beyond the last edge becomes missing without a warning, so compare isna() after the cut with the gaps you had before it.
• DON'T forget which edge an interval owns, the right one, so a price equal to an inner edge belongs to the band below it, and a price equal to the first edge belongs to no band.
• DON'T expect qcut bins to hold exactly equal counts, the number of values and the ties at each edge decide how close they come.

Lab Exercises

Lab Exercise 1: Four Features, One Guard

Start a new script in ~/dev/bootcamp with the loading and cleaning block that opens the Theory, then build four columns and print what each one cost.

1. price_per_room, the price divided by the number of rooms, as one vectorized expression. Print how many values are missing and the first three, rounded to one decimal.
2. rooms_band with np.select: "compact" for 1 or 2 rooms, "standard" for 3 or 4, "large" otherwise. Print the counts.
3. zone, by mapping neighborhood through {"Downtown": "central", "Riverside": "central", "Hillcrest": "north", "Northend": "north", "Eastgate": "east", "Lakeview": "west"}. Print how many listings got no zone and which neighborhood they belong to.
4. listing_year, the year of listing_date as a number, through apply and a named function that returns None for a missing date. Print the column's dtype, then convert it with 2.5.2's astype("Int64") and print its counts with the gaps included.

Lab Exercise 2: Rows, Columns and Bands

Start another script with the same opening block.

1. Write price_flag(row), returning "incomplete" when the price or the area is missing, "expensive" when the price per square meter is above 3,000, and "normal" otherwise. Apply it across the rows and print the counts. Then build the same labels with np.select and print whether every label agrees.
2. With apply and the default axis, print the median of price, area_sqm and rooms.
3. Bin area_sqm with pd.cut at the edges 40, 80, 120 and 160, labeled small, medium and large. Print the counts, the number of missing values, and the sorted areas that fell outside every band.
4. Split area_sqm into three equal-count bands with pd.qcut, labeled low, mid and high, and print the counts.

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
extreme = (
    (df["price"] > 2_000_000)
    | (df["area_sqm"] > 1_000)
    | (df["rooms"] > 20)
)
df = df[~extreme].reset_index(drop=True)

# rooms is never missing, so only the 19 unpriced rows are NaN
df["price_per_room"] = df["price"] / df["rooms"]
print(df["price_per_room"].isna().sum())  # 19
# [218500.0, 53600.0, 44666.7]
print(df["price_per_room"].head(3).round(1).tolist())

# narrowest condition first, the first match wins
conditions = [df["rooms"] <= 2, df["rooms"] <= 4]
df["rooms_band"] = np.select(conditions, ["compact", "standard"], default="large")
# {'large': 70, 'compact': 69, 'standard': 61}
print(df["rooms_band"].value_counts().to_dict())

zone = {
    "Downtown": "central",
    "Riverside": "central",
    "Hillcrest": "north",
    "Northend": "north",
    "Eastgate": "east",
    "Lakeview": "west",
}
df["zone"] = df["neighborhood"].map(zone)
print(df["zone"].isna().sum())  # 36
# ['Southgate'], unique() lists each distinct value once
print(df.loc[df["zone"].isna(), "neighborhood"].unique().tolist())

def listing_year(d):
    if pd.isna(d):     # three listings have no date
        return None
    return int(d[:4])  # '2024-10-27' gives 2024

df["listing_year"] = df["listing_date"].apply(listing_year)
print(df["listing_year"].dtype)
df["listing_year"] = df["listing_year"].astype("Int64")
print(df["listing_year"].value_counts(dropna=False))

float64
listing_year
2024    107
2023     90
<NA>      3
Name: count, dtype: Int64

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
extreme = (
    (df["price"] > 2_000_000)
    | (df["area_sqm"] > 1_000)
    | (df["rooms"] > 20)
)
df = df[~extreme].reset_index(drop=True)

def price_flag(row):
    if pd.isna(row["price"]) or pd.isna(row["area_sqm"]):
        return "incomplete"
    if row["price"] / row["area_sqm"] > 3_000:
        return "expensive"
    return "normal"

flag = df.apply(price_flag, axis=1)  # one call per listing
# {'normal': 99, 'expensive': 67, 'incomplete': 34}
print(flag.value_counts().to_dict())

ppsqm = df["price"] / df["area_sqm"]
vector = np.select(
    [ppsqm.isna(), ppsqm > 3_000],
    ["incomplete", "expensive"],
    default="normal",
)
print((flag == vector).all())        # True

numeric = df[["price", "area_sqm", "rooms"]]
# {'price': 245000.0, 'area_sqm': 102.4, 'rooms': 3.0}
print(numeric.apply(lambda col: col.median()).to_dict())

area_band = pd.cut(
    df["area_sqm"],
    bins=[40, 80, 120, 160],
    labels=["small", "medium", "large"],
)
# {'small': 39, 'medium': 92, 'large': 47}
print(area_band.value_counts(sort=False).to_dict())
print(area_band.isna().sum())  # 22
outside = df.loc[area_band.isna() & df["area_sqm"].notna(), "area_sqm"]
# [26.4, 31.1, 164.1, 165.2, 172.0, 182.9, 184.2]
print(sorted(outside.tolist()))

thirds = pd.qcut(df["area_sqm"], q=3, labels=["low", "mid", "high"])
# {'low': 62, 'mid': 61, 'high': 62}
print(thirds.value_counts(sort=False).to_dict())

Theory Assessment

Q1: The Band Nobody Got

A teammate builds a size band with np.select([df["area_sqm"] < 120, df["area_sqm"] < 80], ["medium", "small"], default="large"). Say how many listings come out small, where the 15 listings with no area end up, and what fixes each problem.

Hint: Ask which condition a 60 square meter listing meets first. Then ask what NaN < 120 evaluates to.

Q2: The Same Dictionary, Two Methods

df["condition"].map({"poor": "fair"}) and df["condition"].replace({"poor": "fair"}) both run without a warning. Predict how many missing values each returns on the cleaned table, say which one folds the 23 poor listings into fair, and explain why the other one cannot.

Hint: One method treats its dictionary as the complete list of answers, and the other treats it as a list of edits.

Q3: What the Function Is Handed

For numeric = df[["price", "area_sqm", "rooms"]], state for numeric.apply(f) and for numeric.apply(f, axis=1) how many times f is called, what it receives on each call, and how long the result is when f returns one number.

Hint: 2.2.5's rule applies unchanged, the axis you name is the one that disappears from the shape.

Q4: Three Columns, One Report

The cleaned table has 19 listings with no price, 1 with no condition and 15 with no area. Read this without running it.

features = pd.DataFrame({
    "band": pd.cut(
        df["price"],
        bins=[df["price"].min(), 250_000, df["price"].max()],
        labels=["lower", "upper"],
    ),
    "score": df["condition"].map(
        {"poor": 1, "fair": 2, "good": 3, "Excellent": 4}
    ),
    "size": np.select(
        [df["area_sqm"] < 80, df["area_sqm"] < 120],
        ["small", "medium"],
        default="large",
    ),
})
print(features.isna().sum())

Predict the three numbers it prints, explain each one, and say which of the three is the most dangerous number in the report.

Hint: The first edge of the cut is a price some listing actually has. One key of the dictionary is spelled differently from the data. And np.select never produces a missing value of its own.

wrong = np.select(
    [df["area_sqm"] < 120, df["area_sqm"] < 80],
    ["medium", "small"],
    default="large",
)
# {'medium': 133, 'large': 67}
print(pd.Series(wrong).value_counts().to_dict())

handed = []
numeric.apply(lambda col: handed.append(col.name))
print(handed)                  # ['price', 'area_sqm', 'rooms']
sizes = []
numeric.apply(lambda row: sizes.append(len(row)), axis=1)
print(len(sizes), set(sizes))  # 200 {3}

band     20
score    34
size      0
dtype: int64

Assignment

Assignment: A Feature Builder That Refuses Bad Labels

The training script for the model of how long a listing stays on the market calls one function, build_features(listings), and trusts what comes back. Write it so that it returns a new frame, leaves the frame it was given untouched, and adds six columns: price_per_sqm, age measured from the fixed reference year 2025, condition_score from the poor-to-excellent lookup, size_band with an unknown band for a missing area, price_quartile from qcut, and worn_and_old, true for a listing at least 55 years old whose condition score is 2 or less. Before it builds anything, the function must stop with a ValueError that names every condition label its lookup cannot score, because a label the lookup lacks would otherwise become a silent gap in condition_score.

Build all six columns in a single call to DataFrame.assign. df.assign(name=values, ...) returns a new DataFrame with each keyword added as a column and leaves df as it was. A value may also be a function, and then assign calls it with the frame being built, so a column can read columns created earlier in the same call, which worn_and_old needs and which an ordinary expression written against listings cannot do.

Run it on the cleaned table and print the shapes of the input and the result and the missing count of every added column. Then copy the table, set one listing's condition to "Good ", with the capital and the trailing space 2.5.3 cleaned away, and show that the function refuses the copy while the original keeps its label.

Hints:

• set(listings["condition"].dropna()) - set(SCORES) is every label the lookup lacks, and an empty set is falsy, so a raise under if unknown: runs only when a label is missing.
• Write each assign column as lambda d: ... and read the columns from d, the frame being built, never from listings.

import numpy as np
import pandas as pd

REFERENCE_YEAR = 2025
SCORES = {"poor": 1, "fair": 2, "good": 3, "excellent": 4}

def build_features(listings):
    unknown = set(listings["condition"].dropna()) - set(SCORES)
    if unknown:
        raise ValueError(f"no score for {sorted(unknown)}")
    return listings.assign(
        price_per_sqm=lambda d: d["price"] / d["area_sqm"],
        age=lambda d: REFERENCE_YEAR - d["year_built"],
        condition_score=lambda d: d["condition"].map(SCORES),
        size_band=lambda d: np.select(
            [d["area_sqm"].isna(), d["area_sqm"] < 80, d["area_sqm"] < 120],
            ["unknown", "small", "medium"],
            default="large",
        ),
        price_quartile=lambda d: pd.qcut(
            d["price"], q=4, labels=["q1", "q2", "q3", "q4"]
        ),
        # reads two columns created earlier in this same call
        worn_and_old=lambda d: (d["age"] >= 55) & (d["condition_score"] <= 2),
    )

raw = pd.read_csv("housing.csv").drop_duplicates()
extreme = (
    (raw["price"] > 2_000_000)
    | (raw["area_sqm"] > 1_000)
    | (raw["rooms"] > 20)
)
listings = raw[~extreme].reset_index(drop=True)

features = build_features(listings)
print(listings.shape, features.shape)
added = [c for c in features.columns if c not in listings.columns]
print(features[added].isna().sum())

(200, 7) (200, 13)
price_per_sqm      34
age                 0
condition_score     1
size_band           0
price_quartile     19
worn_and_old        0
dtype: int64

bad = listings.copy()                # a second frame, so the edit cannot reach 'listings'
bad.loc[3, "condition"] = "Good "
try:
    build_features(bad)
except ValueError as err:
    print(f"ValueError: {err}")      # ValueError: no score for ['Good ']
print(listings.loc[3, "condition"])  # good

try:
    listings.assign(
        age=REFERENCE_YEAR - listings["year_built"],
        old=listings["age"] >= 55,
    )
except KeyError as err:
    print(f"KeyError: {err}")        # KeyError: 'age'

Additional Resources

Optional, and nothing later depends on them. Worth opening if you want to see more of what pandas offers.

• Official docs - pandas API reference, Series.case_when. A pandas method for the same first-condition-wins choice np.select makes, returning a Series that keeps its index.
• Official docs - pandas User Guide, "Enhancing performance". What remains when a per-row function genuinely has no vectorized form and is still too slow, starting from a plain apply.

Summary

A feature is a decision about how a recorded fact becomes something a model can compare, and this lesson was mostly about who handles the values that do not fit the decision. Arithmetic carries a gap through and says so. A lookup turns an unlisted label into a gap and says nothing. A band drops whatever lies beyond its edges, and np.select files a NaN that fails every test under its default as if it were data. Each tool is honest about some cases and silent about others, and knowing which is which is most of the skill.

The other thread was restraint. apply reaches every value with any logic you can write, and that generality is exactly why it is the last tool to pick rather than the first. The vectorized line and the per-row function returned identical columns every time they were compared here, so the output never argues for the faster one. What argues for it is the question asked before writing the function: could whole columns say this? Most of the time they can, and the per-row function survives only where the rule genuinely branches in ways no mask or lookup expresses.

What ties the lesson together is counting. Most features here were followed by a count of what they lost, the 34 rows without a price per square meter, the 36 listings without a zone, the three prices beyond the last edge, and those counts are what separate a feature table you can explain from one that merely has no errors. The columns built here describe each listing on its own. The next lesson asks how a listing compares with the others in its neighborhood, which needs the table split into groups first.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np
import pandas as pd

df["price_per_sqm"] = df["price"] / df["area_sqm"]
df["size_band"] = np.select(
    # the gap gets the first condition
    [df["area_sqm"].isna(), df["area_sqm"] < 80],
    ["unknown", "small"],
    default="large",
)
scores = {"poor": 1, "fair": 2, "good": 3, "excellent": 4}
df["score"] = df["condition"].map(scores)  # count isna() after
folded = df["condition"].replace({"poor": "fair"})

def label(c):                              # named: it needs a guard
    if pd.isna(c):
        return "UNKNOWN"
    return c.upper()

df["condition"].apply(label)               # one call per value
df.apply(needs_renovation, axis=1)         # one row per call
df[["price"]].map(lambda v: f"{v:,.0f}")   # one cell per call
# each interval owns its right edge, (a, b]
pd.cut(df["price"], bins=[100_000, 300_000, 600_000])
pd.qcut(df["price"], q=4, labels=["q1", "q2", "q3", "q4"])