pytest

Context

pytest is a program that finds the functions in your project whose names start with test, runs each one, and reports which of them ended with a false assert. A test is ordinary Python: it calls your code with known inputs and states what must come back, and pytest turns every such statement into a line of a report and a single exit code.

Every change to a price model's project goes through it before the change is trusted: a new split ratio, an extra column, a different imputer, a library upgrade, a refactor of the training script. The same command runs on your machine before a commit and, later in this module, inside the automated pipeline that checks every push, which reads nothing but pytest's exit code.

Without it, the only check on listings is running the training script and recognizing its output. Lesson 6.1.1 turned the script into a package whose split() gives 2,400 training rows and 600 test rows, and nothing yet says that it must. A one-character edit to test_size moves 150 listings out of the training rows, the model still fits, the script still prints a score, and the score is no longer the one every later lesson compares against. A test catches that edit the moment it is made.

Goal

• Install pytest 9.1.1 as a development dependency of the project with uv add --dev, and explain what a dependency group changes in pyproject.toml, uv.lock and .venv.
• Write test functions with plain assert statements and share the data they need through fixtures in tests/conftest.py, scoped to the session.
• Parametrize one test over the pipeline's three column groups with readable ids, and test a joblib round trip inside tmp_path.
• Configure pytest in pyproject.toml's [tool.pytest] table with testpaths and strict, and show what each one refuses.
• Break the split on purpose, read every part of the failure report, rerun only what failed, and commit the eight passing tests.

Prerequisites

None of this is needed, and the lesson stands on its own. Both are here if you want to watch pytest run and fail before you write a test yourself.

• YouTube - "getting started with pytest" by anthonywritescode, a walk through starting with pytest by one of its core developers.
• pytest documentation - Get Started. Installing pytest, a first failing test and its report, how pytest finds test files by their names, and the tmp_path directory a test can ask for.

Table of Contents

• pytest as a Development Dependency
• Test Files, Test Functions and assert
• Fixtures and conftest.py
• Fixture Scope
• parametrize and ids
• tmp_path
• The [tool.pytest] Table
• Reading a Failure

Theory

Every command runs in ~/dev/listings-service, Lesson 6.1.1's project, with (bootcamp) deactivated as Module 6's setup page says, so uv uses the project's own .venv. Every Python block is a file of the project, saved at the path named above it. pytest draws its rules of = as wide as your terminal and prints its own seconds on every run: the transcripts below come from an 80-column terminal on a machine that was busy with other work, so your lines can be longer and your seconds will differ.

pytest as a Development Dependency

What. pytest is installed into the project with uv add --dev pytest==9.1.1, which writes it into a dependency group named dev in pyproject.toml, locks it in uv.lock beside every other package, and installs it into .venv. A dependency group is a named list of packages the project needs to be worked on, kept apart from the dependencies it needs to run. Why. Training and serving the price model need pandas, NumPy and scikit-learn. Nobody who only prices a listing needs a test runner, so pytest belongs to the people who change the code. Kept in its own group, it is pinned and reproduced like everything else and still stays out of anything built with --no-dev. When. For every tool that checks or builds the code and never runs inside it: the test runner here, and the client a later lesson's tests use to call the service.

Before the install, the project has no pytest to run:

goku@machine:~/dev/listings-service$ uv run pytest
error: Failed to spawn: pytest
  cause: No such file or directory (os error 2)

In ~/dev/listings-service:

uv add --dev pytest==9.1.1

Resolved 18 packages in 292ms
   Building listings @ file:///home/goku/dev/listings-service
      Built listings @ file:///home/goku/dev/listings-service
Prepared 1 package in 9ms
Uninstalled 1 package in 0.29ms
Installed 6 packages in 3ms
 + iniconfig==2.3.1
 ~ listings==0.1.0 (from file:///home/goku/dev/listings-service)
 + packaging==26.3
 + pluggy==1.7.0
 + pygments==2.21.0
 + pytest==9.1.1

pytest brought four libraries of its own: iniconfig (reads configuration files), packaging (compares version numbers), pluggy (the plugin system pytest is built on) and pygments (colors the code in a report). The lock resolved 18 packages, six more than Lesson 6.1.1's twelve, where five were installed: the sixth is colorama, which pytest needs only on Windows, so uv.lock records it with the marker sys_platform == 'win32', and Linux, WSL included, and macOS never install it. The ~ line is uv rebuilding the project's own package, which it does whenever pyproject.toml changes. The file now ends with the new group:

[dependency-groups]
dev = [
    "pytest==9.1.1",
]

uv run pytest now starts pytest, and with no tests written it says so and exits with 5:

goku@machine:~/dev/listings-service$ uv run pytest; echo $?
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 0 items

============================ no tests ran in 0.01s =============================
5

The header names the interpreter and the versions, then the rootdir, the folder pytest found by walking up from where it was started to the first pyproject.toml, and the configfile it read. Exit code 5 means nothing was collected, which is not a pass: a pipeline that runs uv run pytest treats any code but 0 as a failure.

The group is part of a plain uv sync and a plain uv run, and --no-dev leaves it out:

goku@machine:~/dev/listings-service$ uv sync --no-dev
Resolved 18 packages in 1ms
Uninstalled 5 packages in 7ms
 - iniconfig==2.3.1
 - packaging==26.3
 - pluggy==1.7.0
 - pygments==2.21.0
 - pytest==9.1.1
goku@machine:~/dev/listings-service$ uv run pytest --version
Installed 5 packages in 10ms
pytest 9.1.1

uv run syncs the environment to the lock, development group included, before it runs anything, so the five came back without being asked for. The image Section 6.3 builds installs the project with --no-dev, and pytest never enters it.

DON'T:

• DON'T uv add pytest without --dev. It lands in dependencies, and everything that installs the project to run it installs a test runner it never calls.
• DON'T read exit code 5 as a pass. no tests ran means pytest found nothing, and a pipeline that reads the code fails on it.

Test Files, Test Functions and assert

What. pytest collects every file named test_*.py or *_test.py under the folders it searches, and in each one every function whose name starts with test. A test function calls the code under test and checks the result with Lesson 1.6.3's assert statement, and the test passes when every assert holds and nothing raises. Why. A rule that needs no registration means a test is a function you write and nothing else: no class to inherit from, no list of tests to keep up to date, no special assertion methods to learn. And because pytest rewrites every assert in a test file before running it, a failing one reports the values on both sides, where a bare assert in a script gives only AssertionError. When. For every fact about the project that must stay true from one commit to the next: the split's sizes here, the pipeline's structure later in this lesson, the model's quality in the next.

The tests live in a tests/ folder beside src/, outside the package they test. tests/test_data.py:

from listings.data import load, split


def test_split_sizes():
    X_train, X_test, y_train, y_test = split(load())
    assert len(X_train) == 2400
    assert len(X_test) == 600

The import works because uv installed the listings package into .venv in Lesson 6.1.1 as an editable install, a link to src/ rather than a copy, so a test imports the package exactly as the training script does and sees every edit to it at once. Run it:

goku@machine:~/dev/listings-service$ uv run pytest
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 1 item

tests/test_data.py .                                                     [100%]

============================== 1 passed in 4.88s ===============================

Each file gets one line, each test one character on it, . for a pass, and the percentage counts the tests run so far. The seconds are mostly the import of pandas and scikit-learn, and they move from run to run. -v gives each test a line of its own, named by its node id, the file and the function joined by :::

goku@machine:~/dev/listings-service$ uv run pytest -v
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 1 item

tests/test_data.py::test_split_sizes PASSED                              [100%]

============================== 1 passed in 1.42s ===============================

The header now also names the interpreter, the project's own .venv/bin/python, and .pytest_cache, the folder pytest keeps between runs to remember which tests failed last. pytest writes a .gitignore holding * inside that folder, so git never shows it. The test holds two numbers, 2,400 and 600, that the project's every later score rests on: Lesson 3.2.3 split the 3,000 listings 80/20 with random_state=42, Lesson 6.1.1's split() kept that split, and from now on a change to it fails here first.

DON'T:

• DON'T name a test file price_checks.py or a test function check_split. Neither matches pytest's patterns, nothing collects them, and the run reports only the tests it did find.
• DON'T put the tests inside src/listings/. They would ship with the package, and they import it the way any other user of it does only from outside it.

Fixtures and conftest.py

What. A fixture is a function decorated with @pytest.fixture whose return value pytest hands to any test that names it as a parameter. A fixture can name other fixtures as its own parameters, and fixtures defined in a file called conftest.py are available to every test in that folder and below it without an import. Why. Every test of the split needs the listings loaded and split. Written into each test, those two lines repeat, and a change to how the data is loaded means editing every test. A fixture says it once, and a test's parameters list exactly what it depends on. When. For any input more than one test needs: the loaded table and its split here, a fitted pipeline in the next lesson, a running service and a database connection later in the module.

tests/conftest.py, first with the default scope:

import pytest

from listings.data import load, split


@pytest.fixture
def listings():
    return load()


@pytest.fixture
def split_data(listings):
    return split(listings)

split_data asks for listings, so pytest builds listings first and passes it in. tests/test_data.py, rewritten on the two fixtures, with two more tests:

from listings.data import FEATURES, TARGET, split


def test_split_sizes(split_data):
    X_train, X_test, y_train, y_test = split_data
    assert len(X_train) == 2400
    assert len(X_test) == 600


def test_split_is_deterministic(listings, split_data):
    X_test = split_data[1]
    again = split(listings)[1]
    assert X_test.index.equals(again.index)


def test_features_exclude_target(split_data):
    X_train = split_data[0]
    assert list(X_train.columns) == FEATURES
    assert TARGET not in X_train.columns

split_data is the four-item list train_test_split returns, in Lesson 3.1.2's order, training features, test features, training prices, test prices. test_split_is_deterministic splits the same listings a second time and compares the test rows' index, the row labels pandas carried over from the CSV, so it fails if the split ever stops being the same rows. test_features_exclude_target checks that the features are exactly the nine FEATURES in their order, and that price is not among them, so the model is never handed the price it predicts.

goku@machine:~/dev/listings-service$ uv run pytest
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 3 items

tests/test_data.py ...                                                   [100%]

============================== 3 passed in 0.05s ===============================

The session's 0.05 seconds no longer hold the imports: pytest imports conftest.py, and with it pandas and scikit-learn, while it configures itself, before the clock the summary reads starts.

DON'T:

• DON'T define a fixture in one test file for tests in another. A test in a second file that names it errors with fixture 'rows' not found, and conftest.py is where shared fixtures live.
• DON'T call a fixture like a function, listings(), inside a test. The parameter already holds its value, a DataFrame, so the call fails the test with TypeError: 'DataFrame' object is not callable.

Fixture Scope

What. A fixture's scope is how long pytest keeps its value. The default, "function", builds it again for every test that asks for it, and scope="session" builds it once, the first time a test asks, and hands the same object to every test of the run. --setup-show prints every fixture's setup and teardown with its scope's letter, F or S. Why. Every test of the data asks for the same 3,000 rows, read from the same file and split the same way. Built per test, the CSV is read and split once for every test, and the next lesson's fixture that fits the pipeline would fit it once for every test too. When. For a value that is expensive to build and that no test changes. A value a test changes needs a fresh one per test, the default.

With the default scope:

goku@machine:~/dev/listings-service$ uv run pytest --setup-show
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 3 items

tests/test_data.py 
        SETUP    F listings
        SETUP    F split_data (fixtures used: listings)
        tests/test_data.py::test_split_sizes (fixtures used: listings, split_data) .
        TEARDOWN F split_data
        TEARDOWN F listings
        SETUP    F listings
        SETUP    F split_data (fixtures used: listings)
        tests/test_data.py::test_split_is_deterministic (fixtures used: listings, split_data) .
        TEARDOWN F split_data
        TEARDOWN F listings
        SETUP    F listings
        SETUP    F split_data (fixtures used: listings)
        tests/test_data.py::test_features_exclude_target (fixtures used: listings, split_data) .
        TEARDOWN F split_data
        TEARDOWN F listings

============================== 3 passed in 0.05s ===============================

Three tests, three loads, three splits. Give both fixtures scope="session", and tests/conftest.py takes its final form:

import pytest

from listings.data import load, split


@pytest.fixture(scope="session")
def listings():
    return load()


@pytest.fixture(scope="session")
def split_data(listings):
    return split(listings)

goku@machine:~/dev/listings-service$ uv run pytest --setup-show
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 3 items

tests/test_data.py 
SETUP    S listings
SETUP    S split_data (fixtures used: listings)
        tests/test_data.py::test_split_sizes (fixtures used: listings, split_data) .
        tests/test_data.py::test_split_is_deterministic (fixtures used: listings, split_data) .
        tests/test_data.py::test_features_exclude_target (fixtures used: listings, split_data) .
TEARDOWN S split_data
TEARDOWN S listings

============================== 3 passed in 0.03s ===============================

One load and one split for the whole run, set up before the first test that asked and torn down after the last. split_data asks for listings, so listings has to be a session fixture too: a session fixture that asks for a function-scoped one errors at setup with ScopeMismatch: You tried to access the function scoped fixture, naming the fixture.

The price of sharing is that every test receives the same DataFrame object, and a test that changes it in place, listings.dropna(inplace=True), changes it for every test that runs after it in the session, while the same tests run without it still pass.

DON'T:

• DON'T change a session fixture's value inside a test. Work on a copy, listings.dropna() returns a new frame, or give that test a function-scoped fixture of its own.
• DON'T make a fixture session-scoped to save time on a value tests modify. The time saved is paid back in a test whose result depends on which tests ran before it.

parametrize and ids

What. @pytest.mark.parametrize(names, values) runs one test function once per entry of values, binding each entry to the parameters names lists. ids= gives each run its name, which pytest puts in square brackets after the function's name in the node id. Why. The pipeline's ColumnTransformer routes three groups of columns to three transformers, and one test per group would be three copies of the same three lines. A parametrized test writes the check once, reports each group as its own pass or failure, and a fourth group is one more line in the list. When. Whenever the same check applies to several inputs that should each pass or fail on their own: column groups, categories, file formats, the conditions of a listing.

tests/test_features.py:

import pytest

from listings.data import NOMINAL, NUMERIC
from listings.features import build_pipeline


def test_pipeline_steps():
    pipe = build_pipeline()
    assert list(pipe.named_steps) == ["prep", "model"]


@pytest.mark.parametrize("group, columns", [
    ("num", NUMERIC),
    ("cat", NOMINAL),
    ("cond", ["condition"]),
], ids=["num", "cat", "cond"])
def test_column_groups(group, columns):
    prep = build_pipeline()["prep"]
    groups = {name: cols for name, _, cols in prep.transformers}
    assert groups[group] == columns

Both tests read the pipeline before it is fitted, so neither touches the data. named_steps holds the pipeline's steps by name, and prep.transformers is the list Lesson 6.1.1's build_pipeline() gave the ColumnTransformer, one (name, transformer, columns) triple per group, which the dictionary comprehension turns into a lookup from group name to columns. A pipeline whose step is renamed, or whose numeric group loses bathrooms, fails here before anything trains on it.

goku@machine:~/dev/listings-service$ uv run pytest -v tests/test_features.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 4 items

tests/test_features.py::test_pipeline_steps PASSED                       [ 25%]
tests/test_features.py::test_column_groups[num] PASSED                   [ 50%]
tests/test_features.py::test_column_groups[cat] PASSED                   [ 75%]
tests/test_features.py::test_column_groups[cond] PASSED                  [100%]

============================== 4 passed in 0.32s ===============================

A path after pytest runs that file alone. Without ids=, pytest builds each name from the values, keeping a string as it is and naming a list by its parameter and its position, so the three runs read test_column_groups[num-columns0], [cat-columns1] and [cond-columns2]. -k keeps only the tests whose node id contains a word, across every file:

goku@machine:~/dev/listings-service$ uv run pytest -v -k cat
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 7 items / 6 deselected / 1 selected

tests/test_features.py::test_column_groups[cat] PASSED                   [100%]

======================= 1 passed, 6 deselected in 0.08s ========================

DON'T:

• DON'T loop over the groups inside one test. The first group that fails stops the loop, and the report names one test where three checks ran or never ran.
• DON'T reuse an id within one parametrize. Left to itself pytest renames the two runs num0 and num1, and the strict setting of this lesson's configuration turns the repeat into a collection error.

tmp_path

What. tmp_path is a fixture pytest provides: a pathlib.Path to an empty folder created for the one test that asks for it, under pytest-of-goku/pytest-<N>/ in the system's temporary folder, /tmp on Linux and WSL, where <N> counts your pytest runs. pytest keeps the folders of the last three runs and deletes older ones. Why. A test of saving and loading has to write a file somewhere. Written into the project, it would overwrite models/price_pipeline.joblib, the file the training script produced, and two tests writing the same name would collide. tmp_path gives every test a folder no other test and no other run shares. When. For every test that writes: a saved model, an exported CSV, a log file, a database file.

Add import joblib as the first line of tests/test_features.py, above import pytest, and this test at the end of the file:

def test_saved_pipeline_predicts_the_same(split_data, tmp_path):
    X_train, X_test, y_train, _ = split_data
    pipe = build_pipeline().fit(X_train, y_train)
    path = tmp_path / "pipe.joblib"
    joblib.dump(pipe, path)
    loaded = joblib.load(path)
    assert (loaded.predict(X_test) == pipe.predict(X_test)).all()

It fits the pipeline on the 2,400 training rows, saves it with Lesson 3.2.3's joblib.dump, loads the file back, and requires every one of the 600 test predictions to be identical, not close. A round trip that loses a fitted step, the imputer's medians or the encoder's categories, changes the predictions and fails here. _ takes the test prices, which this test does not need.

goku@machine:~/dev/listings-service$ uv run pytest --setup-show tests/test_features.py::test_saved_pipeline_predicts_the_same
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 1 item

tests/test_features.py 
SETUP    S listings
SETUP    S split_data (fixtures used: listings)
SETUP    S tmp_path_factory
        SETUP    F tmp_path (fixtures used: tmp_path_factory)
        tests/test_features.py::test_saved_pipeline_predicts_the_same (fixtures used: listings, request, split_data, tmp_path, tmp_path_factory) .
        TEARDOWN F tmp_path
TEARDOWN S tmp_path_factory
TEARDOWN S split_data
TEARDOWN S listings

============================== 1 passed in 0.21s ===============================

A node id after pytest runs that one test. tmp_path is function-scoped, built from tmp_path_factory, the session fixture that owns the run's folder, and request is pytest's own fixture that tells a fixture which test asked for it. The file stays after the test, and pytest keeps a -current link to the newest run and to the newest folder of each test:

goku@machine:~/dev/listings-service$ wc -c /tmp/pytest-of-goku/pytest-current/test_saved_pipeline_predicts_tcurrent/pipe.joblib
5554 /tmp/pytest-of-goku/pytest-current/test_saved_pipeline_predicts_tcurrent/pipe.joblib

5,554 bytes, the size of Lesson 6.1.1's models/price_pipeline.joblib, which this run never touched. pytest cut the test's name to 30 characters for the folder, test_saved_pipeline_predicts_t, and numbered it 0 within its run. The whole suite, eight tests:

goku@machine:~/dev/listings-service$ uv run pytest
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 8 items

tests/test_data.py ...                                                   [ 37%]
tests/test_features.py .....                                             [100%]

============================== 8 passed in 0.17s ===============================

DON'T:

• DON'T save into models/ from a test. The test overwrites the artifact the training script produced, and the next test run decides what that file holds.
• DON'T compare the loaded pipeline's predictions with a tolerance. The same fitted numbers give the same predictions to the last bit, and a tolerance would let a round trip that changed them pass.

The [tool.pytest] Table

What. pytest 9 reads its settings from a [tool.pytest] table in pyproject.toml, in TOML's own types, a list for a list and true for a boolean. testpaths = ["tests"] names the folders pytest searches when it is started in the rootdir with no path, and strict = true turns on all four of pytest's strictness options: strict_config, strict_markers, strict_xfail and strict_parametrization_ids. Why. Without testpaths, pytest searches every folder under the rootdir, and any file that matches its patterns runs as part of the suite. Without strict, a misspelt setting or a repeated id is a warning at the bottom of a passing run, and a passing run is the one nobody reads. When. Once, as soon as a project has tests, so every run, yours and the pipeline's, reads the same settings from the same file. Before 9.0, pyproject.toml held pytest's settings only in the older [tool.pytest.ini_options] table, whose values are INI-style strings.

A one-off check saved at the project's root, test_quick.py, runs with the suite:

import pandas as pd


def test_price_column():
    assert "price" in pd.read_csv("data/listings.csv").columns

A bare run collects it with the suite:

goku@machine:~/dev/listings-service$ uv run pytest
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 9 items

test_quick.py .                                                          [ 11%]
tests/test_data.py ...                                                   [ 44%]
tests/test_features.py .....                                             [100%]

============================== 9 passed in 0.20s ===============================

Add the table at the end of pyproject.toml:

[tool.pytest]
testpaths = ["tests"]
strict = true

goku@machine:~/dev/listings-service$ uv run pytest
   Building listings @ file:///home/goku/dev/listings-service
      Built listings @ file:///home/goku/dev/listings-service
Uninstalled 1 package in 0.39ms
Installed 1 package in 0.69ms
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
testpaths: tests
collected 8 items

tests/test_data.py ...                                                   [ 37%]
tests/test_features.py .....                                             [100%]

============================== 8 passed in 0.21s ===============================

The four lines on top are uv rebuilding the package after the edit to pyproject.toml, the ~ line of the install. The header gains testpaths: tests, and test_quick.py is no longer collected, though uv run pytest test_quick.py still runs it, because a path given on the command line always wins. Delete it, rm test_quick.py, since the suite is tests/.

strict_config refuses a setting pytest does not know. With testpaths misspelt as testpath:

goku@machine:~/dev/listings-service$ uv run pytest; echo $?
   Building listings @ file:///home/goku/dev/listings-service
      Built listings @ file:///home/goku/dev/listings-service
Uninstalled 1 package in 0.28ms
Installed 1 package in 0.48ms
ERROR: Unknown config option: testpath


============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 8 items

============================ no tests ran in 0.08s =============================
4

Exit code 4 is pytest's usage error, and no test ran. The same typo with strict = false ran all eight tests, ended 8 passed, 1 warning, printed PytestConfigWarning: Unknown config option: testpath in the warnings summary, and exited 0, with testpaths never set. strict_parametrization_ids refuses the repeated id the previous subtitle's DON'T named. With ids=["num", "cat", "num"]:

goku@machine:~/dev/listings-service$ uv run pytest; echo $?
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
testpaths: tests
collected 3 items / 1 error

==================================== ERRORS ====================================
___________________ ERROR collecting tests/test_features.py ____________________
Duplicate parametrization IDs detected, but strict_parametrization_ids is set.

Test name:      tests/test_features.py::test_column_groups
Parameters:     group, columns
Parameter sets: ['num', ['area_sqm', 'year_built', 'distance_km', 'rooms', 'bathrooms']], ['cat', ['neighborhood', 'property_type', 'energy_rating']], ['cond', ['condition']]
IDs:            num, cat, num
Duplicates:     num

You can fix this problem using `@pytest.mark.parametrize(..., ids=...)` or `pytest.param(..., id=...)`.
=========================== short test summary info ============================
ERROR tests/test_features.py
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.29s ===============================
2

A file that cannot be collected stops the whole run, exit code 2, and the three tests of test_data.py that were collected never ran. Without strict, the two runs were renamed and all eight passed. Put the ids back to ["num", "cat", "cond"]. strict_markers and strict_xfail concern markers, labels a test can carry, which this project does not use yet.

DON'T:

• DON'T configure pytest in two places. Even an empty pytest.ini beside pyproject.toml wins, and the header reads configfile: pytest.ini (WARNING: ignoring pytest config in pyproject.toml!).
• DON'T read testpaths as a limit on what can run. It decides only what a bare uv run pytest started in the rootdir collects, and a path on the command line runs anything.

Reading a Failure

What. A failing run prints a FAILURES section with one block per failed test, then a short summary with one line per failure, and exits with 1. Each block shows the test's arguments, its source up to the failing line, marked >, and pytest's explanation of the assert on lines marked E. An exception raised while a fixture is being built is reported as an error, E in the progress line and ERROR at setup of in its block, not as a failure. Why. A failure is only worth what the report lets you learn without opening a debugger. pytest's assertion rewriting puts the value of every expression in the failing assert on the screen, so the report itself says which number moved and by how much. When. Every time a run is not all ., read from the short summary up: it names what failed and the first value that was wrong, and the block above it says where.

Break the split on purpose: in src/listings/data.py, change test_size=0.2 to test_size=0.25, the kind of edit made while trying another ratio.

goku@machine:~/dev/listings-service$ uv run pytest; echo $?
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
testpaths: tests
collected 8 items

tests/test_data.py F..                                                   [ 37%]
tests/test_features.py .....                                             [100%]

=================================== FAILURES ===================================
_______________________________ test_split_sizes _______________________________

split_data = [      area_sqm  year_built  distance_km  ...  property_type  energy_rating  condition
2922      65.9      2016.0     .....  
195     673000
1775    267000
2841    474000
1735    327000
179     264000
Name: price, Length: 750, dtype: int64]

    def test_split_sizes(split_data):
        X_train, X_test, y_train, y_test = split_data
>       assert len(X_train) == 2400
E       assert 2250 == 2400
E        +  where 2250 = len(      area_sqm  year_built  distance_km  ...  property_type  energy_rating  condition\n2922      65.9      2016.0      ...    good\n860      150.3      2008.0          8.1  ...          house              B       fair\n\n[2250 rows x 9 columns])

tests/test_data.py:6: AssertionError
=========================== short test summary info ============================
FAILED tests/test_data.py::test_split_sizes - assert 2250 == 2400
========================= 1 failed, 7 passed in 0.35s ==========================
1

The progress line's F is the first test of test_data.py. The block opens with the test's one argument, split_data, the whole four-item list printed and cut short by pytest, from the first DataFrame's columns to the end of the test prices, Length: 750, already 150 more test prices than the 600 the test expects. The E lines are the rewritten assert: the comparison as it ran, 2250 == 2400, and under it where the 2250 came from, len() of a DataFrame of [2250 rows x 9 columns]. tests/test_data.py:6 is the line. The other seven tests passed: the split is still the same rows on every run, still has the nine features and no price, and the pipeline does not care how many rows it is fitted on, so only the test that pinned the sizes saw the edit.

Three options shorten the loop of fixing and rerunning. --lf, last failed, reruns only the tests that failed in the previous run, read from .pytest_cache, and its header says what it skipped:

goku@machine:~/dev/listings-service$ uv run pytest --lf
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
testpaths: tests
collected 1 item
run-last-failure: rerun previous 1 failure (skipped 1 file)

tests/test_data.py F                                                     [100%]

The rest of that run is the same failure block and ends 1 failed. -x stops the run at the first failure and prints stopping after 1 failures, and -q drops the header and the per-file lines, so uv run pytest -x -q prints one failure block and nothing after it but the summary. -k split ran the two tests whose names hold split and ended 1 failed, 1 passed, 6 deselected. Change test_size back to 0.2:

goku@machine:~/dev/listings-service$ uv run pytest --lf -q
.                                                                        [100%]
1 passed in 0.02s

--lf after a clean pass has no failures to rerun and runs every test. The full suite is back to 8 passed, and the work is ready to commit:

goku@machine:~/dev/listings-service$ git status --short
 M pyproject.toml
 M uv.lock
?? tests/
goku@machine:~/dev/listings-service$ git add pyproject.toml uv.lock tests
goku@machine:~/dev/listings-service$ git commit -m "test: unit tests for the split and the pipeline"
[main 7ae32d3] test: unit tests for the split and the pipeline
 5 files changed, 140 insertions(+)
 create mode 100644 tests/conftest.py
 create mode 100644 tests/test_data.py
 create mode 100644 tests/test_features.py

Your commit's hash differs from 7ae32d3. tests/__pycache__/, the compiled copies Python wrote while importing the tests, stays out of the commit through Lesson 6.1.1's .gitignore line __pycache__/, and .pytest_cache/ through its own .gitignore.

DON'T:

• DON'T change the expected number to make a failing test pass. 2400 is the fact the scores rest on, and the edit that broke it was in data.py.
• DON'T read an ERROR as a failed check. It says the test never ran, because a fixture it needed raised first.

Lab Exercises

Each lab writes one file in tests/, so the session fixtures of tests/conftest.py reach it, and ends by deleting it, so the committed suite stays the eight tests above.

Lab Exercise 1: Three Facts About the Rows

Write tests/test_lab_rows.py with three tests built on the two session fixtures. test_raw_table_shape checks that the loaded table has 3,000 rows and 13 columns and that no price is missing. test_no_listing_in_both_splits checks that no row label appears in both the training and the test features. test_prices_line_up_with_rows checks that the training prices carry exactly the training features' row labels in the same order, that the test prices do the same for the test features, and that every training price is above zero. Run tests/test_data.py and your file together with --setup-show and say how many times each fixture was set up. Then remove the test_ prefix from test_raw_table_shape, run your file with -v, and say what the report shows and what it does not. Delete the file.

Lab Exercise 2: Inputs the Pipeline Will Meet

Write tests/test_lab_inputs.py with two tests. The first is parametrized over the four text columns, NOMINAL and condition, with each column's name as its id, and checks that every value found in that column of the test features also appears in the training features, ignoring missing values. The second fits the pipeline on the training rows, writes the 600 test features to test_rows.csv inside tmp_path without the index, reads the file back with pandas, and checks that the reloaded frame has the same columns in the same order and that the pipeline prices every reloaded row exactly as it prices the row in memory. Run the file with -v, then run only the energy rating's case, and list what the round trip left in its folder. Delete the file.

from listings.data import TARGET


def test_raw_table_shape(listings):
    assert listings.shape == (3000, 13)
    assert listings[TARGET].notna().all()


def test_no_listing_in_both_splits(split_data):
    X_train, X_test = split_data[0], split_data[1]
    assert X_train.index.intersection(X_test.index).empty


def test_prices_line_up_with_rows(split_data):
    X_train, X_test, y_train, y_test = split_data
    assert y_train.index.equals(X_train.index)
    assert y_test.index.equals(X_test.index)
    assert (y_train > 0).all()

goku@machine:~/dev/listings-service$ uv run pytest --setup-show tests/test_data.py tests/test_lab_rows.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collected 6 items

tests/test_data.py 
SETUP    S listings
SETUP    S split_data (fixtures used: listings)
        tests/test_data.py::test_split_sizes (fixtures used: listings, split_data) .
        tests/test_data.py::test_split_is_deterministic (fixtures used: listings, split_data) .
        tests/test_data.py::test_features_exclude_target (fixtures used: listings, split_data) .
tests/test_lab_rows.py 
        tests/test_lab_rows.py::test_raw_table_shape (fixtures used: listings) .
        tests/test_lab_rows.py::test_no_listing_in_both_splits (fixtures used: listings, split_data) .
        tests/test_lab_rows.py::test_prices_line_up_with_rows (fixtures used: listings, split_data) .
TEARDOWN S split_data
TEARDOWN S listings

============================== 6 passed in 0.04s ===============================

goku@machine:~/dev/listings-service$ uv run pytest -v tests/test_lab_rows.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 2 items

tests/test_lab_rows.py::test_no_listing_in_both_splits PASSED            [ 50%]
tests/test_lab_rows.py::test_prices_line_up_with_rows PASSED             [100%]

============================== 2 passed in 0.03s ===============================

import pandas as pd
import pytest

from listings.data import NOMINAL
from listings.features import build_pipeline

TEXT_COLUMNS = NOMINAL + ["condition"]


@pytest.mark.parametrize("column", TEXT_COLUMNS, ids=TEXT_COLUMNS)
def test_test_values_seen_in_training(split_data, column):
    X_train, X_test = split_data[0], split_data[1]
    assert set(X_test[column].dropna()) <= set(X_train[column].dropna())


def test_csv_round_trip_prices_the_same(split_data, tmp_path):
    X_train, X_test, y_train, _ = split_data
    pipe = build_pipeline().fit(X_train, y_train)
    path = tmp_path / "test_rows.csv"
    X_test.to_csv(path, index=False)
    reloaded = pd.read_csv(path)
    assert list(reloaded.columns) == list(X_test.columns)
    assert (pipe.predict(reloaded) == pipe.predict(X_test)).all()

goku@machine:~/dev/listings-service$ uv run pytest -v tests/test_lab_inputs.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 5 items

tests/test_lab_inputs.py::test_test_values_seen_in_training[neighborhood] PASSED [ 20%]
tests/test_lab_inputs.py::test_test_values_seen_in_training[property_type] PASSED [ 40%]
tests/test_lab_inputs.py::test_test_values_seen_in_training[energy_rating] PASSED [ 60%]
tests/test_lab_inputs.py::test_test_values_seen_in_training[condition] PASSED [ 80%]
tests/test_lab_inputs.py::test_csv_round_trip_prices_the_same PASSED     [100%]

============================== 5 passed in 0.19s ===============================
goku@machine:~/dev/listings-service$ uv run pytest -v -k energy tests/test_lab_inputs.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 5 items / 4 deselected / 1 selected

tests/test_lab_inputs.py::test_test_values_seen_in_training[energy_rating] PASSED [100%]

======================= 1 passed, 4 deselected in 0.09s ========================
goku@machine:~/dev/listings-service$ ls /tmp/pytest-of-goku/pytest-current/*current/
test_rows.csv

Theory Assessment

Q1: The Test That Never Ran

A teammate adds tests/price_checks.py holding one function, test_prices_are_positive(listings), which asserts that every price is above zero. uv run pytest -q prints 8 passed. uv run pytest -q tests/price_checks.py prints 1 passed. Say why the first run left the test out, why the second ran it, what the first run's output would have had to say for anyone to notice, and the smallest fix.

Hint: pytest decides what a file is from its name before it reads a line of it. Ask what changes when the file is named on the command line.

Q2: A Determinism Test That Cannot Fail

A teammate shortens test_split_is_deterministic to assert split_data[1].index.equals(split_data[1].index). To check both versions, they delete random_state=42 from split() and run the suite with the original test and theirs side by side: 1 failed, 8 passed, the original failing. Say why the shortened test passed on a split that changes every run, what the original does that it does not, and what session scope has to do with it.

Hint: Count how many times split() runs in each version during one session, and ask what split_data is the second time a test asks for it.

Q3: Started From the Wrong Folder

From inside tests/, uv run pytest ends 4 passed, 4 errors. The progress lines read test_data.py EEE and test_features.py ....E, the header still names /home/goku/dev/listings-service as the rootdir, and every error ends FileNotFoundError: [Errno 2] No such file or directory: 'data/listings.csv'. Say which four tests passed and why, why the other four are errors rather than failures, why testpaths did not stop this, and what the run says about the project's code.

Hint: The rootdir is where pytest found pyproject.toml. The folder relative paths start from is another thing, and DATA_PATH is one.

Q4: Green Alone, Red Together

A teammate adds tests/test_clean.py, one test that calls listings.dropna(inplace=True) on the session fixture and checks that more than 2,000 complete rows remain. uv run pytest -q tests/test_data.py prints 3 passed, and uv run pytest -q tests/test_data.py tests/test_clean.py prints 4 passed, but uv run pytest -q fails with FAILED tests/test_data.py::test_split_sizes - assert 1998 == 2400. Say why the full run fails, why each of the other two passes, where the 1,998 comes from, which test is at fault although another one failed, and how to fix it without giving up session scope for the others.

Hint: In the full run pytest collects tests/ file by file, in order of name. Ask which test asks for listings first, and when split_data is built.

Assignment

Assignment: The One Function No Test Calls

train.main() is the entry point Lesson 6.1.1 runs with uv run python -m listings.train, and none of the eight tests calls it, because it writes models/price_pipeline.joblib, the very artifact a test must never overwrite. Test it anyway, in tests/test_lab_train.py, so that the file it saves lands in the test's own folder, what it prints is checked line for line, and the project's models/ is left as it was.

Two fixtures pytest provides do what this lesson did not show. monkeypatch changes an attribute for the length of one test and puts the original back when the test ends: monkeypatch.setattr(module, "NAME", value) replaces module.NAME. capsys captures what the code under test prints: capsys.readouterr().out returns everything written to the screen since the test began, as one string.

Write two tests. test_main_saves_where_told points listings.train's MODEL_PATH at models/price_pipeline.joblib inside tmp_path, calls train.main(), and checks that the printed lines are exactly the score line Lesson 6.1.1 printed, 56,562 0.766, and the list joblib.dump returned, holding the new path as a string, and that the saved file is 5,554 bytes. test_patch_is_undone, which asks for no fixture, checks that train.MODEL_PATH is back to Path("models/price_pipeline.joblib"). Run the file with -v, then the whole suite with it, then the two tests in reverse order. End with one sentence on why the test patches train.MODEL_PATH and does not import MODEL_PATH into the test file and reassign it there. Delete the file.

Hints:

• main() reads MODEL_PATH from its own module's global names every time it runs, so the attribute to replace is the one on the module object from listings import train gives you.
• main() creates the model's folder with mkdir(exist_ok=True), which makes one level, so tmp_path / "models" works because tmp_path already exists.

from pathlib import Path

from listings import train


def test_main_saves_where_told(tmp_path, monkeypatch, capsys):
    target = tmp_path / "models" / "price_pipeline.joblib"
    monkeypatch.setattr(train, "MODEL_PATH", target)
    train.main()
    printed = capsys.readouterr().out.splitlines()
    assert printed == ["56,562 0.766", str([str(target)])]
    assert target.stat().st_size == 5554


def test_patch_is_undone():
    assert train.MODEL_PATH == Path("models/price_pipeline.joblib")

goku@machine:~/dev/listings-service$ uv run pytest -v tests/test_lab_train.py
============================= test session starts ==============================
platform linux -- Python 3.12.8, pytest-9.1.1, pluggy-1.7.0 -- /home/goku/dev/listings-service/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/goku/dev/listings-service
configfile: pyproject.toml
collecting ... collected 2 items

tests/test_lab_train.py::test_main_saves_where_told PASSED               [ 50%]
tests/test_lab_train.py::test_patch_is_undone PASSED                     [100%]

============================== 2 passed in 0.16s ===============================
goku@machine:~/dev/listings-service$ uv run pytest -q
..........                                                               [100%]
10 passed in 0.23s
goku@machine:~/dev/listings-service$ uv run pytest -q tests/test_lab_train.py::test_patch_is_undone tests/test_lab_train.py::test_main_saves_where_told
..                                                                       [100%]
2 passed in 0.16s

Additional Resources

Optional, and nothing later depends on either. Worth opening for fixtures beyond the ones this project needs, or for why the tests sit outside src/.

• YouTube - "pytest: everything you need to know about fixtures" by anthonywritescode, a longer look at fixtures by one of pytest's core developers.
• pytest documentation - Good Integration Practices, "Choosing a test layout". Tests kept in a folder outside the application code, run against an installed or editable copy of it, and why the documentation recommends the src layout.

Summary

The suite this lesson committed is eight small statements about a project that already worked: 2,400 and 600, the same rows every run, nine features and no price, two steps named prep and model, three column groups, and a saved file that predicts exactly what the fitted pipeline does. None of them is about whether the model is good. They pin down the shape of the work, so that the next lesson's tests of quality, and every score quoted after them, stand on a split and a pipeline that cannot change without someone noticing.

What made each statement useful was less the assert than what surrounds it. Fixtures put the data in one place and session scope built it once, at the cost of a shared object that one careless test can change for every test after it. Parametrize gave each column group its own verdict, tmp_path gave the round trip somewhere harmless to write, and the configuration table decided what a bare uv run pytest means in this project and refused a misspelt setting instead of warning about it at the bottom of a green run.

The failures were the lesson's real content. A changed ratio failed one test with the 2,250 and the 2,400 side by side, a misnamed file and a renamed function passed by not running, a run from the wrong folder errored before testing anything, and a suite passed or failed according to the order of its files. pytest's report is precise about what it ran, and only a reader who checks the count, the letters and the order knows what it did not.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import pytest

from listings.data import load, split


@pytest.fixture(scope="session")
def listings():
    return load()                    # once per run, never mutate


@pytest.mark.parametrize("group, columns", [
    ("num", NUMERIC),
    ("cat", NOMINAL),
], ids=["num", "cat"])               # names in the node id
def test_column_groups(group, columns):
    ...


def test_round_trip(split_data, tmp_path):
    path = tmp_path / "pipe.joblib"  # a folder per test

[dependency-groups]
dev = ["pytest==9.1.1"]

[tool.pytest]
testpaths = ["tests"]
strict = true