Testing ML Systems

Context

A behavior test trains the model once and then asserts what its predictions must do: score above a floor on the test rows, give the same price whatever the batch or the column order, move the right way when one column of a listing changes, rank the four conditions in order, return one finite number per listing, and refuse a listing it cannot price. It treats the fitted pipeline as a black box and checks its answers, never its code.

A pricing team runs these tests on every change that can move a price: a new preprocessing step, another imputer setting, a retrain on fresh listings, a dependency upgrade. They are the checks that stand between a training script and the service Section 6.2 builds around the same pipeline, and the expectations they encode, a bigger home costs more, a home farther out costs less, are the ones a team states before it has seen any model.

Lesson 6.1.2's eight tests check the code: the split's sizes, the pipeline's step names, its column groups, a saved file. None of them asks what the fitted pipeline predicts, so a pipeline that has quietly lost the order of condition passes all eight, and passes a score floor too, at an average miss of 59,623 against the correct pipeline's 56,562, while it prices a home in good condition below the same home in fair condition. Only a test of what the predictions do can see that.

Goal

• Build a session-scoped trained fixture and seven behavior tests, nine test cases in tests/test_model.py, taking the suite from 8 to 17.
• Set a score floor from the measured 56,562 and 0.766, and explain from ten split seeds what a floor can catch and what it cannot.
• Write invariance and direction tests that hold on every test row, and measure the per-row effect of each numeric column, the bathroom that lowers every price among them.
• Test the order of condition on every row, the shape and type of the output, and the refusal of a missing column and an unknown neighborhood.
• Trace a silent failure, OrdinalEncoder without its categories, that passes the floor at 59,623 and fails exactly one test.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the idea of behavioral testing from the person who named its test types, or a second project's model tests written with pytest.

• YouTube - "Beyond Accuracy: Behavioral Testing of NLP Models with CheckList" by Marco Tulio Ribeiro, hosted by AISC. The paper's first author presents CheckList, a behavioral testing method borrowed from software engineering, and the failures it found in commercial and state-of-the-art models.
• Made With ML - "Testing", the Behavioral testing section. Invariance, directional and minimum functionality tests for a text classifier, written as pytest functions with parametrize and a fixture that loads the trained model.

Table of Contents

• Behavior Tests and the Trained Fixture
• The Score Floor
• Invariance Tests
• Direction Tests
• The Condition Order
• The Output Schema and Refusals
• The Failure Only One Test Sees

Theory

Nothing new is installed: pytest arrived in Lesson 6.1.2 and NumPy in Lesson 6.1.1, and both are in the project's lock. Every command runs in ~/dev/listings-service at Lesson 6.1.2's commit, with (bootcamp) deactivated as Module 6's setup page says. The lesson writes two files of the project, tests/conftest.py and tests/test_model.py, and commits them. Everything else it runs is a probe, a script that measures something and ships nowhere, so the probes live in a folder of their own beside the project:

mkdir ~/dev/listings-probes

A probe is run from the project folder as uv run python ../listings-probes/<name>.py, so it gets the project's environment, where listings is installed, and the project's working directory, where data/listings.csv is. A break the lesson makes on purpose is made in a project file and undone with Lesson 0.3.2's git restore, so the commit comes before the first break that needs it.

Behavior Tests and the Trained Fixture

What. A behavior test calls the fitted pipeline's predict on listings and asserts something about the prices that come back. trained is the fixture every one of them takes: Lesson 6.1.1's build_pipeline() fitted on the 2,400 training rows of Lesson 6.1.2's split_data, at session scope like listings and split_data, so the session fits once and every test reads the same fitted object. Why. A pipeline can be wired exactly as Lesson 6.1.2's tests demand, the right steps, the right column groups, the right split, and still predict nonsense, because what it predicts depends on what it learned and nothing in those tests looks at that. Fitting inside each test would repeat the same fit once per test, and a session fixture states that every test judges one model. When. For every test whose assertion is about predictions, as opposed to the code that produces them.

tests/conftest.py gains an import and one fixture, and in full reads:

import pytest

from listings.data import load, split
from listings.features import build_pipeline


@pytest.fixture(scope="session")
def listings():
    return load()


@pytest.fixture(scope="session")
def split_data(listings):
    return split(listings)


@pytest.fixture(scope="session")
def trained(split_data):
    X_train, _, y_train, _ = split_data
    return build_pipeline().fit(X_train, y_train)

trained asks for split_data, which asks for listings, so pytest builds the chain in that order the first time a test needs it. The tests that take trained read their listings from split_data[1], the 600 test rows, which no fit has seen. The first of them goes in the next subtitle, and Lesson 6.1.2's --setup-show shows the chain built once. Every test here changes listings with assign, which returns a new frame, so none of them touches the session's shared split_data.

DON'T:

• DON'T run a behavior test on split_data[0]. Those are the rows the fixture was fitted on, and a test there asks how well the fit remembers them, not how it prices a listing it has never seen.
• DON'T fit the model inside a behavior test. Each test would judge its own fit, and the suite would no longer say anything about one model.

The Score Floor

What. A score floor asserts that the trained pipeline's error on the test rows stays under a fixed bound: here a mean absolute error under 60,000 and an R² over 0.74, computed with Lesson 6.1.1's evaluate, where the pipeline scores 56,562 and 0.766. Why. It is the one test that sees a model broken across the board, whatever broke it, because every broken thing shows up in the average miss. It is set below today's score and never equal to it, because its job is to refuse a model that got materially worse, not to freeze today's digits. When. In every model's suite, as the first behavior test, with its bounds chosen from measured scores.

tests/test_model.py opens with every import its tests use and the floor:

import numpy as np
import pytest

from listings.data import LEVELS
from listings.train import evaluate


def test_score_floor(trained, split_data):
    _, X_test, _, y_test = split_data
    scores = evaluate(trained, X_test, y_test)
    assert scores["mae"] < 60_000
    assert scores["r2"] > 0.74

goku@machine:~/dev/listings-service$ uv run pytest --setup-show tests/test_model.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 1 item

tests/test_model.py 
SETUP    S listings
SETUP    S split_data (fixtures used: listings)
SETUP    S trained (fixtures used: split_data)
        tests/test_model.py::test_score_floor (fixtures used: listings, split_data, trained) .
TEARDOWN S trained
TEARDOWN S split_data
TEARDOWN S listings

============================== 1 passed in 0.07s ===============================

To see what the floor catches, break the fit for one run. In tests/conftest.py, change the fixture's last line to return build_pipeline().fit(X_train, y_train.sort_values()), run uv run pytest tests/test_model.py, and put y_train back afterwards. The report, cut to the lines that name the failure:

goku@machine:~/dev/listings-service$ uv run pytest tests/test_model.py
>       assert scores["mae"] < 60_000
E       assert 154453.92855066748 < 60000

tests/test_model.py:11: AssertionError
=========================== short test summary info ============================
FAILED tests/test_model.py::test_score_floor - assert 154453.92855066748 < 60000
============================== 1 failed in 0.30s ===============================

sort_values() returns the same 2,400 prices with their index labels, in a new order, and scikit-learn pairs X and y by position and never by index, so the listings were trained against prices that belong to other listings. Nothing raised, and the average miss came out at 154,454, which the floor's first line refused.

How far below today's score a floor sits is a measurement too. The same pipeline on ten other splits of the same listings, seeds 0 to 9, in ../listings-probes/spread.py:

from sklearn.model_selection import train_test_split
from listings.data import load, FEATURES, TARGET
from listings.features import build_pipeline
from listings.train import evaluate

listings = load()
maes, r2s = [], []
for seed in range(10):
    X_train, X_test, y_train, y_test = train_test_split(
        listings[FEATURES], listings[TARGET], test_size=0.2, random_state=seed
    )
    scores = evaluate(build_pipeline().fit(X_train, y_train), X_test, y_test)
    maes.append(scores["mae"])
    r2s.append(scores["r2"])
print(f"{min(maes):,.0f}", f"{max(maes):,.0f}", round(min(r2s), 3), round(max(r2s), 3))
print(sum(m >= 60_000 for m in maes), sum(r <= 0.74 for r in r2s))

53,738 61,272 0.711 0.804
1 2

The same pipeline's average miss runs from 53,738 to 61,272 depending on which 600 listings are held out, a spread of 7,534 that no change to the model made. On one of the ten splits the floor's MAE bound would fail, and on two its R² bound. The floor is a statement about the 600 rows of random_state=42, where the model sits 3,438 under it. A model change worth less than that margin passes, and the spread says why the margin cannot shrink much: a floor pinned near 56,562 would turn red on the next retrain over fresh rows with nothing wrong.

DON'T:

• DON'T set a floor equal to today's score. The same pipeline moved by 7,534 across ten splits, and a floor with no margin fails on the rows drawn, not on the model.
• DON'T read a passing floor as a correct model. A floor sees only what moves the average, and the last subtitle's broken encoder passes it.
• DON'T tune a setting until the floor passes. The suite reads the test rows on every run because it chooses nothing, and a setting chosen to clear it turns them into the validation rows Lesson 3.1.2 kept apart.

Invariance Tests

What. An invariance test changes something the price must not depend on and asserts that the prices stay the same. Here there are two: a listing priced alone against the same listing priced inside the 600-row batch, for the first five rows, and the 600 listings with their columns in reverse order against the original order. Why. The service Section 6.2 builds prices one listing per request, while every score so far was computed on a batch, so a price that depended on its neighbors in a batch would make every reported score a score of some other model. Column order is what a form, a database row or a JSON body is most likely to scramble. When. For every property that must hold by design, batch size, column order, row order, and above all for the ones the serving path changes.

Whether "the same" means equal to the last bit is measured first, in ../listings-probes/invariance.py:

import numpy as np
from listings.data import load, split
from listings.features import build_pipeline

X_train, X_test, y_train, y_test = split(load())
pipe = build_pipeline().fit(X_train, y_train)
batch = pipe.predict(X_test)
alone = np.array([pipe.predict(X_test.iloc[[i]])[0] for i in range(600)])
reordered = pipe.predict(X_test[X_test.columns[::-1]])
print((batch == alone).sum(), np.abs(batch - alone).max())
print((batch == reordered).sum(), np.allclose(batch, alone))

518 1.1641532182693481e-10
600 True

On this machine 518 of the 600 listings priced alone match their batch price to the last bit, and the other 82 differ by at most 1.2e-10, the same arithmetic run over one row instead of 600 rounding differently in its last digits, which is why the test compares with np.allclose (NumPy's check that two arrays agree within a small tolerance) and not with ==. Reversed columns give all 600 prices exactly, because the ColumnTransformer picks its columns by name and runs the same arithmetic whatever order they arrive in. Append to tests/test_model.py:

def test_batch_and_column_order_do_not_matter(trained, split_data):
    X_test = split_data[1]
    batch = trained.predict(X_test)
    alone = [trained.predict(X_test.iloc[[i]])[0] for i in range(5)]
    reordered = trained.predict(X_test[X_test.columns[::-1]])
    assert np.allclose(batch[:5], alone)
    assert np.allclose(batch, reordered)

X_test.iloc[[i]], with a list inside, keeps row i as a one-row frame with its column names, the shape a single request becomes, where iloc[i] returns a Series, which predict refuses with X does not contain any features, but ColumnTransformer is expecting 9 features. Five rows keep the test quick, since each single-row call runs the whole pipeline for one listing, and the probe above checked all 600 once. uv run pytest tests/test_model.py prints 2 passed.

DON'T:

• DON'T compare a single-row price with a batch price by ==. 82 of the 600 differed by up to 1.2e-10, and a test written that way fails on rounding, not on a model.
• DON'T loosen np.allclose past its defaults. Its relative tolerance is a hundred-thousandth, about 4 on a price of 400,000, already wide beside a rounding gap of 1.2e-10.

Direction Tests

What. A direction test moves one column of every test listing by a fixed step and asserts the sign of the change in price: 10 more square meters raises it, a build year 10 years later raises it, 1 more kilometer from the center lowers it. One parametrize, Lesson 6.1.2's, with ids that name each case, runs the three. Why. These are the market's expectations about the same home changed in one respect, so a model that breaks one has learned something the team would never sign off on, however good its average. The test asserts the sign on every row, not on the average, because a model can be right on average and wrong for the listings a seller actually sends. When. For every column whose effect on the target the domain fixes in advance, and only for those.

The effect each numeric column has on a price is measured before any assertion is written, in ../listings-probes/effects.py, with the column moved on the test rows that have a value in it:

from listings.data import load, split
from listings.features import build_pipeline

X_train, X_test, y_train, y_test = split(load())
pipe = build_pipeline().fit(X_train, y_train)
steps = {"area_sqm": 10, "year_built": 10, "distance_km": 1, "rooms": 1, "bathrooms": 1}
for column, step in steps.items():
    rows = X_test.dropna(subset=[column])
    moved = rows.assign(**{column: rows[column] + step})
    change = pipe.predict(moved) - pipe.predict(rows)
    print(column, step, f"{change.min():,.2f}", f"{change.max():,.2f}", len(X_test) - len(rows))
blank = X_test[X_test["area_sqm"].isna()]
moved = blank.assign(area_sqm=blank["area_sqm"] + 10)
print((pipe.predict(moved) - pipe.predict(blank))[:3])

area_sqm 10 28,217.45 28,217.45 35
year_built 10 5,446.69 5,446.69 30
distance_km 1 -6,170.74 -6,170.74 0
rooms 1 20,811.57 20,811.57 0
bathrooms 1 -1,360.29 -1,360.29 0
[0. 0. 0.]

assign(**{column: ...}) unpacks a one-key dictionary into assign's keyword, so one line moves whichever column the loop holds. Each column's smallest and largest change are the same number, because a linear model adds a fixed amount per unit of a column to every listing: 10 square meters add 28,217.45 to every price, 10 years 5,446.69, and a kilometer takes 6,170.74 off. The last column counts the test rows the probe left out, 35 with no area and 30 with no year, and the last line shows why they are left out: a blank plus 10 is still a blank, the imputer fills it with the same median either way, and the change is exactly 0, whose sign is neither 1 nor -1.

Two more rows are worth reading. An extra room adds 20,811.57 at the same area. An extra bathroom takes 1,360.29 off every price. A test that a bathroom raises the price would fail on all 600 rows of this model, and the failure would be a finding about the data rather than a bug: bathrooms rise with rooms and area, and beside them the 2,400 training rows say almost nothing about what a bathroom adds on its own, Lesson 3.3.2's near twins, whose weight can be read for neither its size nor its sign. A direction test belongs to a column whose effect the data can pin, so the bathroom stays out of the suite. Append:

@pytest.mark.parametrize("column, step, sign", [
    ("area_sqm", 10, 1),
    ("year_built", 10, 1),
    ("distance_km", 1, -1),
], ids=["bigger", "newer", "farther"])
def test_direction(trained, split_data, column, step, sign):
    rows = split_data[1].dropna(subset=[column])
    moved = rows.assign(**{column: rows[column] + step})
    change = trained.predict(moved) - trained.predict(rows)
    assert (np.sign(change) == sign).all()

np.sign turns each change into 1, -1 or 0, and .all() asks that every row's sign equal the expected one. uv run pytest tests/test_model.py prints 5 passed, the floor, the invariance test and test_direction[bigger], [newer] and [farther].

DON'T:

• DON'T move a column on rows where it is blank. A blank plus 10 stays blank, the change is 0, and without dropna the bigger and newer cases fail on the 35 and 30 blank rows while the model is fine.
• DON'T write a direction test for a near twin. The bathroom lowers every price by 1,360.29 here, and Lab Exercise 1 measures how far that sign holds.
• DON'T assert the average change. A model can move the average up and still lower the price of the listings in front of it, and the test asserts every row.

The Condition Order

What. The ordering test prices every test listing four times, once as each level of condition in LEVELS, poor, fair, good, excellent, and asserts that each level prices every listing above the level before it. Why. condition is the one column whose order the pipeline is told rather than shown: OrdinalEncoder(categories=[LEVELS]) hands the model the order, and Lesson 3.2.2 showed that without that list the encoder sorts the levels alphabetically. Nothing else in the suite checks that the order arrived. When. For every ordinal column, and for any column whose encoding carries a meaning the model cannot recover from the data.

The prices by level, measured in ../listings-probes/levels.py, with the test score and the order the fitted encoder holds:

from listings.data import load, split, LEVELS
from listings.features import build_pipeline
from listings.train import evaluate

X_train, X_test, y_train, y_test = split(load())
pipe = build_pipeline().fit(X_train, y_train)
scores = evaluate(pipe, X_test, y_test)
print(f"{scores['mae']:,.0f}", round(scores["r2"], 3))
print(pipe["prep"].named_transformers_["cond"].categories_[0])
prices = [pipe.predict(X_test.assign(condition=c)) for c in LEVELS]
for level, p in zip(LEVELS, prices):
    print(level, f"{p.mean():,.0f}")
for worse, better in zip(prices, prices[1:]):
    print(f"{(better - worse).min():,.2f}", f"{(better - worse).max():,.2f}")

56,562 0.766
['poor' 'fair' 'good' 'excellent']
poor 321,372
fair 355,132
good 388,892
excellent 422,652
33,760.15 33,760.15
33,760.15 33,760.15
33,760.15 33,760.15

The encoder holds the order LEVELS gave it, and each step up adds 33,760.15 to every listing, one weight per step, the equal steps Lesson 3.2.2 said ordinal codes assume in a linear model. Append:

def test_better_condition_raises_price(trained, split_data):
    X_test = split_data[1]
    prices = [trained.predict(X_test.assign(condition=c)) for c in LEVELS]
    for worse, better in zip(prices, prices[1:]):
        assert (better > worse).all()

zip(prices, prices[1:]) pairs each level's 600 prices with the next level's, poor with fair, fair with good, good with excellent, and the assertion compares them row by row. uv run pytest tests/test_model.py prints 6 passed.

DON'T:

• DON'T compare the four levels' average prices only. The test compares every listing, so a model that ranks the levels in order on average and out of order for some homes still fails it.
• DON'T write the levels into the test by hand. The test and the encoder read the one LEVELS in listings.data, so a level added there is tested in the order the encoder was given.

The Output Schema and Refusals

What. The schema test asserts what comes back from predict on the 600 test rows: an array of shape (600,), one price per listing, of type float64, every value finite. The two refusal tests assert that a listing the pipeline cannot price raises ValueError instead of a price: one with the year_built column missing, and one from Uptown, a neighborhood no training row holds. pytest.raises(ValueError, match=...) passes only when the block raises that error with a message that contains the pattern. Why. Every later layer reads this output: in Section 6.2 the service turns a price into a number in a JSON body, and from Section 6.3 a database stores it, so a shape or a type that changes reaches them before any score does. A refusal is behavior too: a listing the model knows nothing about must stop with an error rather than come back with a confident price. When. For the output of every model that something else consumes, and for every input the model is meant to refuse.

The messages the two refusals carry, and the cheapest price the model gives a test listing, in ../listings-probes/refusals.py:

from listings.data import load, split
from listings.features import build_pipeline

X_train, X_test, y_train, y_test = split(load())
pipe = build_pipeline().fit(X_train, y_train)
missing = X_test.drop(columns=["year_built"])
uptown = X_test.head(1).assign(neighborhood="Uptown")
for rows in (missing, uptown):
    try:
        pipe.predict(rows)
    except ValueError as error:
        print(error)
predicted = pipe.predict(X_test)
cheapest = predicted.argmin()
print(f"{predicted[cheapest]:,.0f}", y_test.iloc[cheapest], y_test.min(), (predicted < y_test.min()).sum())

columns are missing: {'year_built'}
Found unknown categories ['Uptown'] in column 0 during transform
1,591 103000 66000 5

The ColumnTransformer names the missing column, and the OneHotEncoder names the unknown neighborhood, each in a ValueError. Each test's match takes a fragment that names the reason and leaves out the details that vary with the listing. The last line is a limit of the schema test: the model prices one test listing at 1,591, a home that sold for 103,000, and five of the 600 under 66,000, the cheapest real price among them, and every one of those is a finite float64. Append:

def test_output_schema(trained, split_data):
    predicted = trained.predict(split_data[1])
    assert predicted.shape == (600,)
    assert predicted.dtype == np.float64
    assert np.isfinite(predicted).all()


def test_missing_column_is_refused(trained, split_data):
    rows = split_data[1].drop(columns=["year_built"])
    with pytest.raises(ValueError, match="columns are missing"):
        trained.predict(rows)


def test_unknown_neighborhood_is_refused(trained, split_data):
    rows = split_data[1].head(1).assign(neighborhood="Uptown")
    with pytest.raises(ValueError, match="unknown categories"):
        trained.predict(rows)

The whole suite:

goku@machine:~/dev/listings-service$ uv run pytest
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
testpaths: tests
collected 17 items

tests/test_data.py ...                                                   [ 17%]
tests/test_features.py .....                                             [ 47%]
tests/test_model.py .........                                            [100%]

============================== 17 passed in 0.37s ==============================

Seven test functions, nine cases with the three directions, beside Lesson 6.1.2's eight. Commit them before anything is broken on purpose:

git add tests
git commit -m "test: behavior tests for the trained model"

DON'T:

• DON'T read a finite price as a plausible one. The schema test passes a price of 1,591 for a home that sold at 103,000, and judging prices is the floor's job and the direction tests'.
• DON'T drop the shape assertion as obvious. Fitted with y_train.to_frame(), a one-column frame, the pipeline returns (600, 1), and in a probe run test_output_schema was the only one of the 17 that failed.
• DON'T write a match that copies the whole message. The message lists the listing's own values, ['Uptown'], {'year_built'}, and the test should pass for any unknown neighborhood and any missing column.

The Failure Only One Test Sees

What. A one-word change to src/listings/features.py: OrdinalEncoder() in place of OrdinalEncoder(categories=[LEVELS]). The encoder then learns the alphabetical order Lesson 3.2.2 printed, excellent, fair, good, poor, and the model is fitted on codes that rank nothing. Why. It is the failure this section exists for: no exception, no warning, every unit test green, a score that clears the floor, and prices that tell sellers a home in good condition is worth less than the same home in fair condition. A suite proves its worth on exactly this kind of change. When. Every time a suite is written, break the property each test guards once, on purpose, and watch which test turns red. A test that never failed has not yet shown it can.

Make the change and run the suite. The report, cut to the lines that name the failure:

goku@machine:~/dev/listings-service$ uv run pytest
tests/test_data.py ...                                                   [ 17%]
tests/test_features.py .....                                             [ 47%]
tests/test_model.py .....F...                                            [100%]

=================================== FAILURES ===================================
______________________ test_better_condition_raises_price ______________________
>           assert (better > worse).all()
E           assert np.False_

tests/test_model.py:40: AssertionError
=========================== short test summary info ============================
FAILED tests/test_model.py::test_better_condition_raises_price - assert np.Fa...
========================= 1 failed, 16 passed in 0.46s =========================

Sixteen pass. Lesson 6.1.2's test_column_groups[cond] passes, because the cond group still holds condition, the floor passes, and the three direction tests pass. The probe from the condition subtitle, run on the broken pipeline, says what the one failure means:

goku@machine:~/dev/listings-service$ uv run python ../listings-probes/levels.py
59,623 0.748
['excellent' 'fair' 'good' 'poor']
poor 342,825
fair 388,020
good 365,423
excellent 410,618
45,194.88 45,194.88
-22,597.44 -22,597.44
45,194.88 45,194.88

The average miss rose by 3,061 to 59,623, and R² fell to 0.748, both still inside the floor. The alphabetical codes put excellent at 0, fair at 1, good at 2 and poor at 3, and the fit gave them one weight of -22,597.44 per code, so poor to fair adds two steps, 45,194.88, fair to good takes one away, and on every one of the 600 listings good is priced 22,597.44 below fair. The ordering test failed at its second pair and named the line. Undo the break with git restore src/listings/features.py, and uv run pytest ends 17 passed again.

DON'T:

• DON'T trust a floor to see a broken encoding. This one cost 3,061 of average miss, less than half the 7,534 the same pipeline moves between splits.
• DON'T stop at the failing test's name. assert np.False_ says only that some listing broke the order, and the probe run on the broken pipeline says which pair and by how much.

Lab Exercises

Lab Exercise 1: Which Columns Earn a Direction Test

The bathroom's -1,360.29 is one fit on one split. In ../listings-probes/seeds.py, for each of the split seeds 0 to 9, split the listings with train_test_split as split does but with that seed, fit build_pipeline() on the training rows, and measure the change in price that each step of the effects probe makes on the test rows that have a value in that column: 10 square meters, 10 years, 1 kilometer, 1 room and 1 bathroom. Print each column's smallest and largest change over the ten seeds and on how many seeds it was negative, then the bathroom's ten changes in whole units.

Then add a fourth case to test_direction, ("rooms", 1, 1) with the id more-rooms, run uv run pytest -v tests/test_model.py, and undo the edit with git restore. Say in a comment which columns' signs the ten fits agree on, and why the bathroom gets no direction test in either direction.

Lab Exercise 2: A Fix That Stops the Crash

A teammate is tired of the service refusing listings from new neighborhoods and changes the cat group in src/listings/features.py to OneHotEncoder(sparse_output=False, handle_unknown="ignore"), which encodes a category the encoder never saw as a row of zeros instead of raising. Make the change and run the whole suite. Then, with the change still in place, write ../listings-probes/unknown.py: print the test score, then price the first test listing as each of the seven neighborhoods the encoder knows, as "Uptown", and as "Riverside " with a trailing space, a typo a web form can send. Undo the change with git restore and say in a comment which tests could have caught it and what "Riverside " was priced as.

from sklearn.model_selection import train_test_split
from listings.data import load, FEATURES, TARGET
from listings.features import build_pipeline

listings = load()
steps = {"area_sqm": 10, "year_built": 10, "distance_km": 1, "rooms": 1, "bathrooms": 1}
changes = {column: [] for column in steps}
for seed in range(10):
    X_train, X_test, y_train, y_test = train_test_split(
        listings[FEATURES], listings[TARGET], test_size=0.2, random_state=seed
    )
    pipe = build_pipeline().fit(X_train, y_train)
    for column, step in steps.items():
        rows = X_test.dropna(subset=[column])
        moved = rows.assign(**{column: rows[column] + step})
        changes[column].append((pipe.predict(moved) - pipe.predict(rows)).mean())
for column, seen in changes.items():
    print(column, f"{min(seen):,.0f}", f"{max(seen):,.0f}", sum(c < 0 for c in seen))
print([round(c) for c in changes["bathrooms"]])
# Area, year, distance and rooms keep one sign on all ten fits. The
# bathroom is negative on nine and positive on one, a near twin whose
# sign the rows decide, so a test of either sign fails on some retrain.

area_sqm 26,428 28,619 0
year_built 3,822 6,277 0
distance_km -6,584 -5,088 10
rooms 21,057 25,054 0
bathrooms -5,721 710 9
[-5020, -4219, -2061, 710, -5225, -5721, -8, -1164, -5269, -1848]

goku@machine:~/dev/listings-service$ uv run pytest -v tests/test_model.py
tests/test_model.py::test_score_floor PASSED                             [ 10%]
tests/test_model.py::test_batch_and_column_order_do_not_matter PASSED    [ 20%]
tests/test_model.py::test_direction[bigger] PASSED                       [ 30%]
tests/test_model.py::test_direction[newer] PASSED                        [ 40%]
tests/test_model.py::test_direction[farther] PASSED                      [ 50%]
tests/test_model.py::test_direction[more-rooms] PASSED                   [ 60%]
tests/test_model.py::test_better_condition_raises_price PASSED           [ 70%]
tests/test_model.py::test_output_schema PASSED                           [ 80%]
tests/test_model.py::test_missing_column_is_refused PASSED               [ 90%]
tests/test_model.py::test_unknown_neighborhood_is_refused PASSED         [100%]

git restore tests/test_model.py

goku@machine:~/dev/listings-service$ uv run pytest
tests/test_data.py ...                                                   [ 17%]
tests/test_features.py .....                                             [ 47%]
tests/test_model.py ........F                                            [100%]
>       with pytest.raises(ValueError, match="unknown categories"):
E       Failed: DID NOT RAISE ValueError
tests/test_model.py:58: Failed
FAILED tests/test_model.py::test_unknown_neighborhood_is_refused - Failed: DI...
========================= 1 failed, 16 passed in 0.51s =========================

from listings.data import load, split
from listings.features import build_pipeline
from listings.train import evaluate

X_train, X_test, y_train, y_test = split(load())
pipe = build_pipeline().fit(X_train, y_train)
scores = evaluate(pipe, X_test, y_test)
print(f"{scores['mae']:,.0f}", round(scores["r2"], 3))
listing = X_test.head(1)
known = list(pipe["prep"].named_transformers_["cat"].categories_[0])
for name in known + ["Uptown", "Riverside "]:
    price = pipe.predict(listing.assign(neighborhood=name))[0]
    print(repr(name), f"{price:,.0f}")
# Only the refusal test can see this change: the 600 test listings hold
# no unknown neighborhood, so every other test reads the same prices.
# "Riverside " lost its neighborhood and was priced as Uptown, 141,612.

56,562 0.766
'Downtown' 117,655
'Eastgate' 297,545
'Hillcrest' 27,414
'Lakeview' 243,864
'Northend' 48,824
'Riverside' 159,471
'Southgate' 96,513
'Uptown' 141,612
'Riverside ' 141,612

git restore src/listings/features.py

Theory Assessment

Q1: Where the Floor Sits

The floor asserts a mean absolute error under 60,000 and an R² over 0.74, where the trained pipeline scores 56,562 and 0.766. A teammate proposes 57,000, which would have caught the alphabetical encoder at 59,623. Say what the floor exists to catch, with the sorted prices' run as the example, what the ten split seeds say about a floor 438 above today's score, and which test should catch the encoder instead.

Hint: Set the 7,534 the same pipeline moved across ten splits beside the 3,061 the encoder cost.

Q2: Equal, Close, and the Shape

The invariance test compares five single-row prices with their batch prices and the reversed-column prices with the original ones, both with np.allclose. Say why == is wrong for the first comparison and could have stood for the second, why the test checks five rows and not 600, and what a model trained on y_train.to_frame() passes and fails in this suite.

Hint: Read the invariance probe's two counts, and ask which assertion in the suite looks at the shape of what predict returns.

Q3: Blank Rows and a Bathroom

Without dropna(subset=[column]), test_direction[bigger] and test_direction[newer] fail and test_direction[farther] passes. Separately, an extra bathroom takes 1,360.29 off every price, and the suite has no bathroom case. Say why the two cases fail without dropna and the third does not, and why the bathroom is left out rather than tested for a lower price.

Hint: Add 10 to a blank and ask what the imputer does with the result. For the bathroom, recall what Lesson 3.3.2 found when it fitted bathrooms beside area and rooms.

Q4: A Refactor All 17 Pass

A teammate changes the imputer in build_pipeline() from strategy="median" to strategy="mean", runs the suite, sees 17 passed, and a reviewer concludes the suite cannot see the imputer at all. With strategy="most_frequent" the suite passes too, and every prediction stays within 1e-9 of today's. With add_indicator=False and "most_frequent", the floor fails with assert 61504.26811193809 < 60000. Say why the strategy moves no prediction while the indicator is on, why the reviewer's conclusion is wrong, and what the third run shows.

Hint: A filled value is a constant written into every blank row of its column. Ask what the indicator column, 1 on exactly those rows, lets a linear fit do with any constant.

Assignment

Assignment: Recording the Bathroom

The bathroom finding has no test, and a teammate argues that a finding with no test is forgotten. pytest can record a test that is expected to fail: pytest.param(..., marks=pytest.mark.xfail(reason="...")) is one case of a parametrize list that carries a mark, and an xfail mark makes a failing case count as an expected failure, reported as x and never failing the suite. When such a case passes instead, pytest calls it XPASS, and a strict xfail turns an XPASS into a failure of the suite. -rx (a report option, one line for each expected failure) prints each one with its reason.

Add the bathroom to test_direction as pytest.param("bathrooms", 1, 1, marks=...) with the reason a near twin of area and rooms, writing no strict= argument, and give it the id more-baths. Run the suite and uv run pytest -rx tests/test_model.py. Then retrain on other rows by changing random_state=42 to random_state=3 in split in src/listings/data.py, and run the suite again. Say what the second run reports, why a case marked without strict=True failed the suite, how the model at seed 3 scores, and whether this test belongs in the project. Undo both edits with git restore and finish with a whole suite.

Hints:

• ids= must name every case in the list, the new one included, or pytest stops before running anything.
• Read the [tool.pytest] table Lesson 6.1.2 wrote in pyproject.toml, and run the effects probe and the levels probe with the seed at 3 to see how the model at seed 3 prices and scores.

@pytest.mark.parametrize("column, step, sign", [
    ("area_sqm", 10, 1),
    ("year_built", 10, 1),
    ("distance_km", 1, -1),
    pytest.param(
        "bathrooms", 1, 1,
        marks=pytest.mark.xfail(reason="a near twin of area and rooms"),
    ),
], ids=["bigger", "newer", "farther", "more-baths"])

goku@machine:~/dev/listings-service$ uv run pytest
======================== 17 passed, 1 xfailed in 0.48s =========================
goku@machine:~/dev/listings-service$ uv run pytest -rx tests/test_model.py
=========================== short test summary info ============================
XFAIL tests/test_model.py::test_direction[more-baths] - a near twin of area and rooms
========================= 9 passed, 1 xfailed in 0.43s =========================

goku@machine:~/dev/listings-service$ uv run pytest
=================================== FAILURES ===================================
__________________________ test_direction[more-baths] __________________________
[XPASS(strict)] a near twin of area and rooms
=========================== short test summary info ============================
FAILED tests/test_model.py::test_direction[more-baths] - [XPASS(strict)] a ne...
========================= 1 failed, 17 passed in 0.27s =========================

goku@machine:~/dev/listings-service$ uv run python ../listings-probes/effects.py
area_sqm 10 27,480.61 27,480.61 24
year_built 10 5,595.96 5,595.96 32
distance_km 1 -6,198.08 -6,198.08 0
rooms 1 21,056.83 21,056.83 0
bathrooms 1 709.68 709.68 0
[0. 0. 0.]
goku@machine:~/dev/listings-service$ uv run python ../listings-probes/levels.py
59,053 0.777
['poor' 'fair' 'good' 'excellent']
poor 328,827
fair 363,279
good 397,730
excellent 432,182
34,451.97 34,451.97
34,451.97 34,451.97
34,451.97 34,451.97

Additional Resources

Optional, and nothing later depends on either. Worth opening for the rest of what pytest's expected failures can do, or for the exact rule behind np.allclose.

• pytest documentation - "How to use skip and xfail to deal with tests that cannot succeed", the strict parameter. The xfail mark's condition, reason, raises and run arguments, how strict=True turns an unexpected pass into a failure, and marks on single parametrize cases with pytest.param.
• NumPy documentation - numpy.allclose. The defaults rtol=1e-05 and atol=1e-08, the inequality absolute(a - b) <= (atol + rtol * absolute(b)), and why that test is not symmetric in a and b.

Summary

Lesson 6.1.2's tests asked whether the code was built as intended, and every one of them could pass while the model priced homes wrongly. The tests here ask a different question, what the fitted pipeline does to a listing, and they answer it with the properties a pricing team fixes before it sees any model: an average miss that stays under a bound, prices that ignore the batch and the column order, prices that rise with area and a later build year and fall with distance, an order on condition that arrives intact, one finite number per listing, and an error for a listing the model knows nothing about.

The measurements decided each test before it was written. The batch probe decided np.allclose over ==, the blank rows decided dropna, ten splits decided how far below today's score the floor can sit, and the bathroom's per-row effect, -1,360 on every listing and a different sign on another split, decided that a direction test belongs only to a column whose effect the data can pin. A test is a claim about the model, and a claim needs a measurement behind it as much as a sentence in a lesson does.

Then the suite was broken on purpose, and the most useful result was the narrowest one. An encoder that lost its order passed sixteen tests, the score floor among them, and one test, the one that guards that order, failed and named the line. That is the shape of most failures that matter in ML: a model that runs, scores almost as well, and is wrong about one thing for every listing. A suite earns its place by having one test for each such thing, each seen to fail once.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

@pytest.fixture(scope="session")
def trained(split_data):
    X_train, _, y_train, _ = split_data
    return build_pipeline().fit(X_train, y_train)

# blanks never move, so leave them out
rows = split_data[1].dropna(subset=[column])
moved = rows.assign(**{column: rows[column] + step})
change = trained.predict(moved) - trained.predict(rows)
assert (np.sign(change) == sign).all()  # every row, not the mean

assert np.allclose(batch[:5], alone)    # rounding, not ==
with pytest.raises(ValueError, match="unknown categories"):
    trained.predict(rows)

# fixtures as they build
uv run pytest --setup-show tests/test_model.py
# measure before asserting
uv run python ../listings-probes/levels.py
# undo a deliberate break
git restore src/listings/features.py