Data Drift and the Retraining Loop

Context

Drift is the gap between the listings a model was trained on and the listings it is asked to price now. It is measured one numeric feature at a time, the training rows against the inputs the service logged, with a two-sample Kolmogorov-Smirnov test (a test of whether two samples come from one distribution) and the population stability index (a number for how far a feature's share of rows in each bin has moved), and when both say a feature moved, a challenger model is fitted on fresh labeled rows and replaces the champion only if it prices those rows better.

A pricing team runs this check every night over the day's logged requests: after a new listing site starts sending the service larger homes, after a new train line moves which neighborhoods sell, after a year in which every square meter got dearer. The sale prices arrive weeks after the quotes, so the inputs are the first thing anyone can watch, and the retrain waits for the labels the market supplies later.

Lesson 6.4.1's /health counts requests, refusals and errors, and every one of those numbers stays perfect while the market moves: a model priced for last year's homes answers every request with a 200 in a few milliseconds. Without a drift check the first signal is a sales team noticing that every quote comes in low, and without a fair contest a retrain can replace a model with a worse one.

Goal

• Send a deterministic moved market through the running service with simulate.py, and show from PostgreSQL and /health that the service stays healthy while it prices three listings in four under their sale price.
• Run a two-sample KS test with scipy.stats.ks_2samp and read its statistic as the largest gap between two cumulative shares, at the area where that gap sits.
• Write the population stability index in NumPy over the training deciles, and trace a PSI back to the bins that produce it.
• Build drift.py over the listings logged in PostgreSQL, apply the trigger PSI > 0.25 and KS p < 0.01, and measure why neither number alone can be the trigger.
• Fit a challenger on fresh labeled rows with retrain.py, move the alias only when it beats the champion on fresh rows neither has seen, and redeploy until /health names the new version.

Prerequisites

Neither is required, and the lesson explains everything it uses. They picture the KS test before the code does.

• YouTube - "Kolmogorov-Smirnov Test Explained | Data Science Fundamentals" by NannyML. Three minutes on the intuition behind the KS test and what its results look like, from a team that builds drift detection.
• SciPy documentation - "scipy.stats.ks_2samp". The test this lesson runs: the statistic as the largest gap between the two empirical distribution functions, the p-value, and the statistic_location and statistic_sign the result carries.

Table of Contents

• A Moved Market Through the Service
• A Healthy Service That Prices Wrong
• The Two-Sample KS Test
• The Population Stability Index
• drift.py and the Retraining Trigger
• A Challenger Against the Champion, and the Redeploy

Theory

Everything runs in ~/dev/listings-service at the state Lesson 6.4.1 left it, with (bootcamp) deactivated as Module 6's setup page says. Section 6.3's compose stack runs the service and PostgreSQL, the service writes every price it returns to the predictions table with the listing as jsonb, the model version and the model's latency, and /health reports the service's own counts. champion points at version 3 of listings-price, which Lesson 6.3.3 trained, promoted and exported into model/, the folder the image serves. The lesson declares SciPy and adds src/listings/simulate.py, src/listings/drift.py, src/listings/retrain.py, tests/test_drift.py and one line of .gitignore, and every probe lives in Lesson 6.1.3's ~/dev/listings-probes, run as uv run python ../listings-probes/<name>.py. The stack runs in the background, so one terminal in the project folder does everything. Timings and the commit hash differ on your machine, and every other line matches.

SciPy is already installed: Lesson 6.1.1's uv tree showed it under scikit-learn, which needs it. drift.py imports scipy.stats itself, though, so the project declares the version it runs, the case where Lesson 6.2.4 declared pydantic:

uv add scipy==1.18.1

Resolved 101 packages in 28ms
   Building listings @ file:///home/goku/dev/listings-service
      Built listings @ file:///home/goku/dev/listings-service
Prepared 1 package in 4ms
Uninstalled 1 package in 0.17ms
Installed 1 package in 0.28ms
 ~ listings==0.1.0 (from file:///home/goku/dev/listings-service)

The same 101 packages resolve and only the project reinstalls. pyproject.toml gains "scipy==1.18.1" among its dependencies, and uv.lock records it as one of the project's own.

A Moved Market Through the Service

What. simulate.py builds a moved market from the 600 test listings of Lesson 6.1.1's split: every area is 30% larger, and every sale price is 1.3 times higher for the extra space and 8% higher again because each square meter now sells dearer. It sends each listing through the running service, the way the listing site would, and writes the batch, with the prices those listings later sold for, to data/fresh.csv. Why. A monitor is trusted only after it has caught a shift whose answer you already know. Here one feature moved, by a known factor, and the moved listings come with labels, so every number the lesson computes can be checked against the truth. When. Before trusting any drift check or retraining step, and every time one of them changes.

Save src/listings/simulate.py:

"""A seeded market that has moved: the test listings, 30% larger, priced higher.

Sends every listing through the running service, and writes the batch with the
prices those listings later sold for to data/fresh.csv.
"""

import math

import httpx

from listings.data import load, split

FRESH_PATH = "data/fresh.csv"
GROWTH = 1.3        # listed homes are 30% larger
MARKET_RISE = 1.08  # and each square meter sells 8% dearer


def moved_market():
    _, X_test, _, y_test = split(load())
    batch = X_test.copy()
    batch["area_sqm"] = (batch["area_sqm"] * GROWTH).round(1)
    batch["price"] = (y_test * GROWTH * MARKET_RISE).round(-3)
    return batch


def as_request(row):
    body = {k: None if isinstance(v, float) and math.isnan(v) else v
            for k, v in row.items() if k != "price"}
    if body["year_built"] is not None:
        body["year_built"] = int(body["year_built"])
    return body


def main(url="http://localhost:8000/predict"):
    batch = moved_market()
    with httpx.Client() as client:
        codes = [client.post(url, json=as_request(row)).status_code
                 for row in batch.to_dict(orient="records")]
    batch.to_csv(FRESH_PATH, index=False)
    print(f"sent {len(codes)} listings, {codes.count(200)} priced")


if __name__ == "__main__":
    main()

The file draws nothing at random: it moves the rows the split's fixed random_state=42 picked, so every run builds the same 600 listings. moved_market scales area_sqm by GROWTH, rounded to one decimal, and adds a price column, the test prices scaled by both factors and rounded to the thousand by round(-3). A blank area stays blank. as_request turns one row of to_dict(orient="records"), Lesson 6.2.2's list of row dictionaries, into the body Lesson 6.2.4's Listing expects: the comprehension replaces pandas' NaN with None, the blank the schema accepts, and leaves out price, which extra="forbid" would refuse. year_built comes out of pandas as a float, because a column with blanks is float64, and int makes it the whole number the schema declares, though lax mode would convert 1991.0 itself, as Lesson 6.2.4 showed for 3.0 rooms. main opens one httpx.Client, the library Lesson 6.2.3's TestClient is built on, which keeps one connection open for every request, and writes the CSV after the last one. httpx is Lesson 6.2.3's dev dependency, so the script runs on your machine and never inside the image, which installs no dev group: docker compose exec api python -c 'import listings.simulate' ends with ModuleNotFoundError: No module named 'httpx'.

Start the stack as Lesson 6.4.1 does. The image rebuilds, because pyproject.toml, uv.lock and src/ changed:

docker compose up -d --build --wait

The last two lines:

 Container listings-service-db-1 Healthy 
 Container listings-service-api-1 Healthy 

Then send the market:

uv run python -m listings.simulate

sent 600 listings, 600 priced

The whole command takes about eight seconds, and the 600 requests are about six and a half of them. The batch's first rows:

head -3 data/fresh.csv

area_sqm,year_built,distance_km,rooms,bathrooms,neighborhood,property_type,energy_rating,condition,price
61.1,1991.0,4.9,1,1,Riverside,apartment,B,poor,243000.0
174.3,2025.0,8.6,5,2,Southgate,townhouse,B,good,590000.0

data/fresh.csv is output that running the code rebuilds, so it joins .gitignore's block for exactly that, which now ends:

# What the code produces, rebuilt by running it
mlflow.db
mlruns/
model/
data/fresh.csv

DON'T:

• DON'T send pandas' NaN in a request body. A variant without the comprehension stopped at the first listing with a blank area, inside httpx, with ValueError: Out of range float values are not JSON compliant: nan, before the request left the machine.
• DON'T call httpx.post once per listing. Each call opens and closes its own connection, and a timing probe of the sends alone put the same 600 listings through in about 13 seconds that way and in about 6.5 through one Client.
• DON'T commit data/fresh.csv. It is rebuilt by running simulate.py, exactly like mlflow.db.

A Healthy Service That Prices Wrong

What. After the moved market, everything the service measures about itself is perfect, while its prices are further off than they have ever been. Health is whether the service answers, quality is whether its answers are right, and only quality needs the truth, the price each listing later sold for. Why. Every signal Section 6.3 and Lesson 6.4.1 built, a healthcheck, status counts and latencies, watches the service. None of them knows what a correct price is, so none can move when the model is wrong. When. Every time a dashboard of service health is read as evidence that the model works.

curl -s http://127.0.0.1:8000/health; echo

{"status":"ok","model_version":"3","uptime_s":10,"requests":602,"rejected":0,"errors":0}

The 602 requests are the 600 listings and two of the healthchecks Docker sends every five seconds by the compose file's interval: 5s. Not one listing was refused and not one request failed. The model's own latency, which Lesson 6.4.1 records with every row:

docker compose exec db psql -U listings -d listings -c 'SELECT count(*), round(avg(latency_ms)::numeric, 2) AS avg_ms, round(max(latency_ms)::numeric, 2) AS max_ms FROM (SELECT latency_ms FROM predictions ORDER BY id DESC LIMIT 600) AS recent'

 count | avg_ms | max_ms 
-------+--------+--------
   600 |   6.56 |  21.56
(1 row)

The inner SELECT takes the newest 600 rows, the batch, and ::numeric converts the average to the type PostgreSQL's two-argument round accepts. Under 7 milliseconds on average, 22 at worst. The quality needs the sale prices in data/fresh.csv and the prices the service returned, read from PostgreSQL on your machine through the published port, so the address names localhost where the service's own says db. Export it once in this terminal, with the password your .env holds:

export DATABASE_URL=postgresql://listings:listings@localhost:5432/listings

~/dev/listings-probes/quality.py:

import os

import mlflow
import numpy as np
import pandas as pd
import psycopg

from listings.data import load, split

champion = mlflow.pyfunc.load_model("models:/listings-price@champion")
_, X_test, _, y_test = split(load())
fresh = pd.read_csv("data/fresh.csv")
with psycopg.connect(os.environ["DATABASE_URL"]) as conn:
    rows = conn.execute(
        "SELECT price FROM predictions ORDER BY id DESC LIMIT %s", (len(fresh),)
    ).fetchall()
returned = [row[0] for row in reversed(rows)]

for name, priced, sold in (("test rows", champion.predict(X_test), y_test),
                           ("moved market", returned, fresh["price"])):
    miss = np.asarray(priced) - np.asarray(sold)
    print(f"{name}: MAE {np.abs(miss).mean():,.0f}, "
          f"mean miss {miss.mean():,.0f}, under {(miss < 0).mean():.0%}")

uv run python ../listings-probes/quality.py

test rows: MAE 56,562, mean miss 3,909, under 45%
moved market: MAE 92,296, mean miss -70,335, under 76%

ORDER BY id DESC reads the newest rows first, so reversed puts the prices back in the order fresh.csv holds. The champion is version 3, the model the service runs, and on Lesson 6.1.1's test rows it scores that lesson's 56,562 and misses both ways, under the sale 45% of the time. On the moved market its error rises to 92,296, and part of that rise is only size: every moved price is 1.3 times 1.08, 1.404 times its old value, and errors grow with prices. The mean miss is the part that is not size. It falls from 3,909 to -70,335, and three listings in four are priced under the sale, because the model learned what a square meter added in the old market and the new one pays more for each.

DON'T:

• DON'T read "errors":0 as a statement about prices. All 600 requests came back 200, and three in four were priced under the sale.
• DON'T compare an MAE across markets whose prices differ in scale. Part of the jump from 56,562 to 92,296 is prices 1.404 times larger, and the bias is the mean miss, 3,909 against -70,335.
• DON'T wait for the labels before looking. The sale prices arrive weeks after the quotes, and the listings were in PostgreSQL the moment they were priced.

The Two-Sample KS Test

What. The two-sample Kolmogorov-Smirnov test compares two samples of one numeric feature. For each sample it builds the empirical cumulative distribution, the share of the sample at or below each value, and its statistic, D, is the largest vertical gap between the two curves, 0 for identical samples and 1 for samples that do not overlap at all. Its p-value is Lesson 3.7.1's: the chance of a gap at least that large if both samples came from one distribution. scipy.stats.ks_2samp(a, b) returns both, and the value where the gap sits. Why. It assumes nothing about the shape of the feature, works on any numeric column, and turns "these look different" into a number with a known rate of false alarms. When. For every numeric feature, comparing the rows the model was trained on with the inputs it is seeing now.

Save src/listings/drift.py whole. Its KS lines belong to this subtitle, psi to the next, and the rest to the one after:

"""How far the listings the service saw have moved from the training rows."""

import os
import sys

import numpy as np
import pandas as pd
import psycopg
from scipy.stats import ks_2samp

from listings.data import NUMERIC, load, split

PSI_LIMIT = 0.25
P_LIMIT = 0.01


def psi(expected, actual, bins=10):
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None)
    return float(np.sum((a - e) * np.log(a / e)))


def drift_report(reference, live):
    rows = []
    for col in NUMERIC:
        ref, cur = reference[col].dropna(), live[col].dropna()
        ks = ks_2samp(ref, cur)
        rows.append({"feature": col, "ks": ks.statistic,
                     "p_value": ks.pvalue, "psi": psi(ref, cur)})
    return pd.DataFrame(rows).set_index("feature")


def live_listings(url, last):
    with psycopg.connect(url) as conn:
        rows = conn.execute(
            "SELECT listing FROM predictions ORDER BY id DESC LIMIT %s", (last,)
        ).fetchall()
    return pd.DataFrame([row[0] for row in rows])


def main():
    last = int(sys.argv[1]) if len(sys.argv) > 1 else 600
    live = live_listings(os.environ["DATABASE_URL"], last)
    report = drift_report(split(load())[0], live)
    print(f"{len(live)} live listings against the training rows")
    print(report.round(3))
    moved = (report["psi"] > PSI_LIMIT) & (report["p_value"] < P_LIMIT)
    drifted = report.index[moved].tolist()
    print(f"retrain: drift in {drifted}" if drifted else "no drift")


if __name__ == "__main__":
    main()

ks_2samp(ref, cur) is the whole test, and its result carries statistic and pvalue. A probe that runs it on the area alone, the training rows against the 600 logged listings, ~/dev/listings-probes/ks_area.py:

import os

from scipy.stats import ks_2samp

from listings.data import load, split
from listings.drift import live_listings

train = split(load())[0]["area_sqm"].dropna()
live = live_listings(os.environ["DATABASE_URL"], 600)["area_sqm"].dropna()

result = ks_2samp(train, live)
print(f"D {result.statistic:.3f}, p {result.pvalue:.1e}, "
      f"at {result.statistic_location}")
at = result.statistic_location
print(f"at or under {at}: {(train <= at).mean():.3f} of training areas, "
      f"{(live <= at).mean():.3f} of live areas")

uv run python ../listings-probes/ks_area.py

D 0.215, p 7.9e-19, at 84.8
at or under 84.8: 0.455 of training areas, 0.241 of live areas

dropna() keeps the 2,310 training areas and the 565 live ones that are not blank. At 84.8 square meters, 45.5% of the training homes are that size or smaller and only 24.1% of the live homes are, and that gap, 0.215, is the widest anywhere along the two curves: the live curve rises later because every home got bigger. statistic_sign, the result's last field, is 1 here, the first sample's curve the higher at that point. The p-value, 7.9e-19, is so small that drift.py's rounding prints it as 0.000.

DON'T:

• DON'T hand ks_2samp a column with its blanks in. With the 90 blank training areas and 35 blank live ones left in, every field of the result came back nan, and nan < 0.01 is False, so the trigger would never fire.
• DON'T read D as the share of homes that changed. Every live area moved by 30%, and D is 0.215, the widest gap between two cumulative shares at a single area.

The Population Stability Index

What. The PSI cuts a feature into bins at the training sample's deciles, the nine values that split it into ten groups of equal size, and compares each bin's share of the training rows, e, with its share of the live rows, a. Each bin adds (a - e) * ln(a / e), which is zero when the two shares match and positive whenever they differ, and the PSI is the sum over the bins. The usual reading comes from credit scoring, where the index began: under 0.1 the feature is stable, 0.1 to 0.25 it has moved moderately, above 0.25 it has moved enough to act on. Why. The KS test answers whether two samples differ. The PSI answers by how much, and its bins say where the rows went, so a reader sees which homes the market gained and lost. When. Beside the KS test on every numeric feature, and whenever someone asks which part of a feature moved.

psi in drift.py builds it in six lines. np.linspace(0, 1, bins + 1) gives the eleven fractions 0, 0.1, up to 1, and np.quantile, Lesson 2.2.5's, returns the training values at them, the edges of ten bins of a tenth each. np.unique drops repeated edges, which a whole-number feature produces: rooms keeps 6 bins and bathrooms 3, because no cut falls between two homes with the same number of bathrooms. Minus and plus infinity as the outer edges make the end bins open, so every live value lands in a bin, even one past every training value. np.histogram(x, edges), NumPy's counter of values per bin, returns the counts and the edges, [0] keeps the counts, and dividing by the length makes them shares. np.clip(..., 1e-4, None) raises every share below 0.0001 to 0.0001, because an empty bin would put the logarithm of zero in the sum.

A probe that prints the ten bins of the area, ~/dev/listings-probes/psi_bins.py:

import os

import numpy as np
import pandas as pd

from listings.data import load, split
from listings.drift import live_listings

train = split(load())[0]["area_sqm"].dropna()
live = live_listings(os.environ["DATABASE_URL"], 600)["area_sqm"].dropna()

edges = np.quantile(train, np.linspace(0, 1, 11))
edges[0], edges[-1] = -np.inf, np.inf
e = np.histogram(train, edges)[0] / len(train)
a = np.histogram(live, edges)[0] / len(live)
bins = pd.DataFrame({
    "upper": edges[1:], "train": e, "live": a,
    "term": (a - e) * np.log(a / e),
})
print(len(train), len(live))
print(bins.round(3))
print(f"PSI {bins['term'].sum():.3f}")

uv run python ../listings-probes/psi_bins.py

2310 565
    upper  train   live   term
0   50.80  0.100  0.028  0.090
1   61.88  0.100  0.053  0.030
2   71.30  0.100  0.051  0.032
3   79.80  0.100  0.067  0.013
4   89.70  0.100  0.101  0.000
5  101.80  0.100  0.115  0.002
6  117.60  0.100  0.113  0.002
7  137.40  0.100  0.122  0.005
8  164.22  0.101  0.126  0.005
9     inf  0.100  0.223  0.099
PSI 0.277

The probe repeats psi's lines without np.unique and np.clip, which change nothing for the area, so that each term can be printed. Every training share is a tenth, by construction. The live shares fall under a tenth in the four bins below 79.8 square meters and rise above it from 89.7 up, until the open top bin, homes over 164.22, holds 22.3% of the live listings. The two outer bins carry 0.189 of the 0.277, so the PSI says the market lost its smallest homes and gained large ones, where the KS test said only that the samples differ.

DON'T:

• DON'T compute the PSI with blanks in the column. With the 90 blank training areas and 35 blank live ones left in, np.quantile returned NaN edges and psi returned 0.0, a perfect score for the one feature that moved.
• DON'T leave the outer edges at the training minimum and maximum. Five live areas lie above the largest training area, 278.5, np.histogram dropped them uncounted, and the PSI fell to 0.265.
• DON'T cut the bins at the live rows' deciles. With the two arguments swapped, the same samples gave 0.336 instead of 0.277, because the moved market's deciles cut other bins.
• DON'T drop the clip. Without it the 30 listings of the next subtitle returned inf, after RuntimeWarning: divide by zero encountered in log.

drift.py and the Retraining Trigger

What. drift_report runs both measures on every numeric feature and returns one row per feature, live_listings reads the newest listings the service logged out of PostgreSQL, and main prints the report and the trigger: a feature has drifted when its PSI is above 0.25 AND its KS p-value is below 0.01. Why. Each number fails alone, in opposite directions. On many rows the KS test calls a modest move certain, on few rows the PSI reads noise as a large move, and requiring both asks for a move that is large and real at once. When. On a schedule, over the newest window of logged listings, before any retrain is considered.

drift_report loops over NUMERIC, Lesson 6.1.1's five numeric columns, drops each column's blanks once for both measures, and collects one dictionary per feature into a frame indexed by the feature's name. The categorical columns are not measured. live_listings reads the newest last rows, and row[0] is the listing column, which psycopg returns as a dictionary because Lesson 6.3.2 stored it as jsonb, so the rows become a frame with a blank as NaN. main takes the window size from the command line, 600 by default, compares it with the 2,400 training rows of split, and turns the two limits into one boolean mask, the & of two comparisons.

uv run python -m listings.drift

600 live listings against the training rows
                ks  p_value    psi
feature                           
area_sqm     0.215    0.000  0.277
year_built   0.037    0.555  0.011
distance_km  0.046    0.261  0.018
rooms        0.032    0.684  0.006
bathrooms    0.032    0.714  0.004
retrain: drift in ['area_sqm']

The area alone passes both limits, and the other four read a PSI under 0.02 and a p-value over 0.26.

Now each number alone. On 600 rows, the test listings with every area 20% larger, not 30%, gave the area a p-value of 0.000 and a PSI of 0.127: the KS test is certain that the area moved, and the PSI calls the move moderate, so the trigger waits for it to grow. The opposite case needs few rows. The first 30 listings of the unmoved test split, which come from the same market as the training rows by construction, in ~/dev/listings-probes/small.py:

import numpy as np

from listings.data import load, split
from listings.drift import drift_report

X_train, X_test, _, _ = split(load())
few = X_test.head(30)
print(drift_report(X_train, few).round(3))

train = X_train["year_built"].dropna()
edges = np.quantile(train, np.linspace(0, 1, 11))
edges[0], edges[-1] = -np.inf, np.inf
print(np.histogram(few["year_built"].dropna(), edges)[0])

uv run python ../listings-probes/small.py

                ks  p_value    psi
feature                           
area_sqm     0.111    0.832  0.162
year_built   0.185    0.246  1.122
distance_km  0.096    0.923  0.144
rooms        0.078    0.988  0.074
bathrooms    0.098    0.913  0.105
[2 1 4 2 3 5 0 1 9 2]

Nothing moved, and the PSI of year_built reads 1.122, four times the limit, while its p-value, 0.246, says a gap of that size is ordinary for 30 rows. The last line counts the 29 years that are not blank into the ten training bins: the seventh, homes built from 1999 to 2004, holds 9.8% of the training years and none of these, and with its share clipped to 0.0001 that one bin adds 0.673 on its own. The ninth, 2012 to 2018, holds 9 of the 29 and adds 0.234. Ten bins over 30 rows expect three each, and at three per bin an empty bin is chance, not drift. A PSI-only trigger would have retrained here, and the KS half refuses.

tests/test_drift.py checks the trigger end to end on the moved market built in memory, without the service or the database:

from listings.drift import P_LIMIT, PSI_LIMIT, drift_report


def test_only_the_moved_feature_drifts(split_data):
    X_train, X_test = split_data[0], split_data[1]
    moved = X_test.assign(area_sqm=X_test["area_sqm"] * 1.3)
    report = drift_report(X_train, moved)
    drifted = (report["psi"] > PSI_LIMIT) & (report["p_value"] < P_LIMIT)
    assert report.index[drifted].tolist() == ["area_sqm"]

split_data is Lesson 6.1.2's session fixture, and assign returns a copy with one column replaced. The suite, with DATABASE_URL removed for this one command so the database test skips:

env -u DATABASE_URL uv run pytest

The collection, the per-file lines and the last line:

collected 29 items

tests/test_app.py .........                                              [ 31%]
tests/test_data.py ....                                                  [ 44%]
tests/test_db.py s                                                       [ 48%]
tests/test_drift.py .                                                    [ 51%]
tests/test_features.py .....                                             [ 68%]
tests/test_model.py .........                                            [100%]

================== 28 passed, 1 skipped, 2 warnings in 1.49s ===================

env -u DATABASE_URL runs the command without that variable. Lesson 6.4.1's 27 passing tests gained one. With the variable set, test_db.py also runs against the stack's database, writing one row and deleting it, and the suite reads 29 passed, 2 warnings.

DON'T:

• DON'T trigger on the PSI alone. Thirty listings of the unmoved market read year_built at 1.122, four times the limit, from one empty bin.
• DON'T trigger on the KS test alone. A 20% larger area over 600 rows was already p 0.000 while its PSI was 0.127, and a window larger than 600 makes ever smaller moves certain.
• DON'T measure a window the bins cannot fill. Ten bins over 30 rows expect three per bin, and one empty bin added 0.673 by itself.

A Challenger Against the Champion, and the Redeploy

What. retrain.py splits the fresh labeled rows in half, fits a challenger on the 2,400 training rows plus one half, scores the champion and the challenger on the other half, registers the challenger as a new version, and moves champion only when the challenger's MAE is lower. The redeploy is the path Section 6.3 built: export the champion into model/, rebuild the image, restart the service, and read the version /health names. Why. A retrain makes a new model, and new is not the same as better. Scoring both on fresh rows that neither was fitted on is the one fair contest, and a champion that wins it stays. When. Every time the trigger fires and labeled fresh rows exist.

Save src/listings/retrain.py:

"""Retrain with the fresh labeled rows; move the alias only if the challenger wins."""

import mlflow
import pandas as pd
from sklearn.model_selection import train_test_split

from listings.data import FEATURES, TARGET, load, split, to_frame
from listings.features import build_pipeline
from listings.promote import promote
from listings.simulate import FRESH_PATH
from listings.train import MODEL_NAME, evaluate, log_pipeline


def main():
    fresh = pd.read_csv(FRESH_PATH)
    seen, holdout = train_test_split(fresh, test_size=0.5, random_state=42)
    X_train, _, y_train, _ = split(load())
    X = pd.concat([X_train, seen[FEATURES]])
    y = pd.concat([y_train, seen[TARGET]])
    X_holdout = to_frame(holdout[FEATURES])

    champion = mlflow.pyfunc.load_model(f"models:/{MODEL_NAME}@champion")
    old = evaluate(champion, X_holdout, holdout[TARGET])
    mlflow.set_experiment("listings-price")
    with mlflow.start_run(run_name="retrain"):
        challenger = build_pipeline().fit(X, y)
        new = evaluate(challenger, X_holdout, holdout[TARGET])
        mlflow.log_params({"strategy": "median", "indicator": True,
                           "fresh_rows": len(seen)})
        mlflow.log_metrics({"mae": new["mae"], "r2": new["r2"],
                            "champion_mae": old["mae"]})
        version = log_pipeline(challenger, X).registered_model_version

    print(f"champion MAE {old['mae']:,.0f}, challenger version {version} "
          f"MAE {new['mae']:,.0f} on {len(holdout)} fresh rows")
    if new["mae"] < old["mae"]:
        promote(version)
        print(f"version {version} is now champion")
    else:
        print("champion kept")


if __name__ == "__main__":
    main()

train_test_split with test_size=0.5 and a fixed seed cuts the 600 fresh rows into seen, 300 the challenger learns from, and holdout, 300 that score both models. pd.concat stacks the old training rows and the seen fresh ones, features and prices in the same order. to_frame, Lesson 6.2.2's, gives the holdout the dtypes the pyfunc champion enforces. evaluate and log_pipeline are train.py's own, from Lessons 6.1.1 and 6.2.2, so the challenger is scored and registered exactly as every earlier version was, and the run, named retrain, logs the champion's MAE beside the challenger's, so the decision can be read back from MLflow. promote is Lesson 6.2.2's, called only when the challenger's MAE is strictly lower.

uv run python -m listings.retrain

The last four lines:

Registered model 'listings-price' already exists. Creating a new version of this model...
Created version '4' of model 'listings-price'.
champion MAE 94,402, challenger version 4 MAE 87,081 on 300 fresh rows
version 4 is now champion

The lines above them are train.py's, the infer_signature warning Lesson 6.2.2 explained and the requirements export Lesson 6.2.1 read. On the 300 held-out fresh rows the champion misses by 94,402 on average, not the 92,296 of all 600 because these are other rows, and the challenger by 87,081, so version 4 wins by 7,321 and the alias moves. It has not solved the new market: 2,400 of its 2,700 rows are the old one, and on Lesson 6.1.1's old test rows it scores 58,897 where the champion scored 56,562. It is better on the market that exists now, the only one it will be asked about.

The service still runs version 3, because the image carries the folder its build copied. Export the new champion into model/, the command Lesson 6.3.1 introduced and Lesson 6.3.3 ran for version 3:

uv run mlflow artifacts download -u models:/listings-price@champion -d model
cat model/registered_model_meta

The download prints the folder's path, then:

model_name: listings-price
model_version: 4

Rebuild and restart:

docker compose up -d --build --wait
curl -s http://127.0.0.1:8000/health; echo

The last two lines of up, then /health:

 Container listings-service-db-1 Healthy 
 Container listings-service-api-1 Healthy 
{"status":"ok","model_version":"4","uptime_s":2,"requests":1,"rejected":0,"errors":0}

The redeploy took about nine seconds, and the new container's one request is the healthcheck --wait waited for. The reference listing Lesson 6.2.4 priced at 384235:

curl -s -X POST http://127.0.0.1:8000/predict -H "Content-Type: application/json" -d '{"area_sqm": 85.0, "year_built": 1998, "distance_km": 3.0, "rooms": 3, "bathrooms": 1, "neighborhood": "Riverside", "property_type": "apartment", "energy_rating": "C", "condition": "good"}'; echo

{"price":394357}

Version 4 prices the same home 10,122 higher, the fresh market's dearer square meters in its coefficients, and every row logged from here on carries model_version 4. The loop is closed, so commit it:

git add .gitignore pyproject.toml uv.lock src/listings/simulate.py src/listings/drift.py src/listings/retrain.py tests/test_drift.py
git commit -m "feat: drift on the logged listings and a retraining loop that guards the champion"

[main 0fb2412] feat: drift on the logged listings and a retraining loop that guards the champion
 7 files changed, 157 insertions(+)
 create mode 100644 src/listings/drift.py
 create mode 100644 src/listings/retrain.py
 create mode 100644 src/listings/simulate.py
 create mode 100644 tests/test_drift.py

DON'T:

• DON'T score the challenger on the fresh rows it was fitted on. On its own 300 rows version 4 scored 81,462, against 87,081 on the 300 it never saw.
• DON'T compare the challenger's fresh MAE with the champion's old test MAE. 87,081 against 56,562 says the challenger lost, and on the same fresh rows it won by 7,321.
• DON'T expect a running service to follow the alias. With champion moved to another version, /health still answered "model_version":"4", because the container serves the model/ its image was built with.

Lab Exercises

Lab Exercise 1: How Big a Move the KS Test Calls Real

In ~/dev/listings-probes/ks_shift.py, compare the training areas with the test areas multiplied by 1.0, 1.05, 1.1, 1.2 and 1.3, each once on the first 60 test listings and once on all 600, with ks_2samp, dropping blanks as drift_report does. Print the factor, the number of areas, D and the p-value with four decimals for each of the ten pairs. Say in a comment how D and the p-value each respond to the size of the move and to the number of rows, which pairs pass P_LIMIT, and why 1.05 on all 600 reads a smaller D than 1.0.

Lab Exercise 2: How Many Logged Listings the Trigger Needs

The moved market is in data/fresh.csv. In ~/dev/listings-probes/rows_needed.py, run drift_report against the training rows on the first 25, 50, 100, 200 and 600 of its listings, and for each window print the area's PSI and p-value, the features whose PSI alone passes PSI_LIMIT, and the features the full trigger names. Say in a comment the smallest window that fires, what a PSI-only trigger would have reported below it, and why the area's PSI is larger on 100 rows than on 600.

from scipy.stats import ks_2samp

from listings.data import load, split

X_train, X_test, _, _ = split(load())
train = X_train["area_sqm"].dropna()
for factor in (1.0, 1.05, 1.1, 1.2, 1.3):
    for n in (60, 600):
        live = (X_test["area_sqm"].head(n) * factor).dropna()
        result = ks_2samp(train, live)
        print(f"x{factor:<4} {len(live):>3} areas: "
              f"D {result.statistic:.3f}, p {result.pvalue:.4f}")
# D measures the move: about 0.22 at x1.3 on both sizes. The p-value
# also depends on the rows: x1.2 is p 0.0916 on 57 areas and 0.0000 on
# 565. P_LIMIT passes x1.3 on 57, and x1.2 and x1.3 on 565. The test
# areas sit a little under the training ones by chance, and x1.05
# closes part of that gap, so its D is smaller than x1.0's.

uv run python ../listings-probes/ks_shift.py

x1.0   57 areas: D 0.083, p 0.8019
x1.0  565 areas: D 0.043, p 0.3548
x1.05  57 areas: D 0.070, p 0.9285
x1.05 565 areas: D 0.037, p 0.5538
x1.1   57 areas: D 0.108, p 0.5021
x1.1  565 areas: D 0.071, p 0.0190
x1.2   57 areas: D 0.163, p 0.0916
x1.2  565 areas: D 0.148, p 0.0000
x1.3   57 areas: D 0.218, p 0.0083
x1.3  565 areas: D 0.215, p 0.0000

import pandas as pd

from listings.data import FEATURES, load, split
from listings.drift import P_LIMIT, PSI_LIMIT, drift_report

X_train = split(load())[0]
fresh = pd.read_csv("data/fresh.csv")[FEATURES]
for n in (25, 50, 100, 200, 600):
    report = drift_report(X_train, fresh.head(n))
    area = report.loc["area_sqm"]
    high = report.index[report["psi"] > PSI_LIMIT].tolist()
    moved = report.index[(report["psi"] > PSI_LIMIT)
                         & (report["p_value"] < P_LIMIT)].tolist()
    print(f"{n:>3} rows: area PSI {area['psi']:.3f}, p {area['p_value']:.4f}, "
          f"PSI over the limit {high}, trigger {moved}")
# The trigger first fires at 100 rows, on the area alone. Below it a
# PSI-only trigger would have named year_built beside the area, a
# feature that never moved. Every term of the PSI is zero or positive,
# so noise in a small window's bin shares adds to it on average.

uv run python ../listings-probes/rows_needed.py

 25 rows: area PSI 0.458, p 0.0394, PSI over the limit ['area_sqm', 'year_built'], trigger []
 50 rows: area PSI 0.311, p 0.0144, PSI over the limit ['area_sqm', 'year_built'], trigger []
100 rows: area PSI 0.561, p 0.0000, PSI over the limit ['area_sqm'], trigger ['area_sqm']
200 rows: area PSI 0.443, p 0.0000, PSI over the limit ['area_sqm'], trigger ['area_sqm']
600 rows: area PSI 0.277, p 0.0000, PSI over the limit ['area_sqm'], trigger ['area_sqm']

Theory Assessment

Q1: The Status Report

After the moved market, a teammate's status report reads: /health ok, 602 requests, 0 rejected, 0 errors, model latency 6.56 milliseconds on average, so the model is fine. Say what each of those numbers measures, which numbers in the lesson contradict the conclusion, and which of those could have been computed the day the listings arrived.

Hint: Ask whether any of the five numbers depends on the price the model returned.

Q2: KS Alone

A teammate proposes to drop the PSI and trigger on the KS p-value alone, because "the KS test is the real statistical test, and the PSI is a rule of thumb". Say what the KS test answers and what it does not, what happened to the test listings with every area 20% larger, and what a nightly window of several thousand listings would do to such a trigger.

Hint: Separate what D measures from what the p-value measures, and ask which of the two depends on the number of rows.

Q3: Twenty Bins

The same teammate wants psi to use 20 bins "for more resolution", and keeps the 0.25 limit. On the 565 moved areas, psi gives 0.237 with 5 bins, 0.277 with 10 and 0.306 with 20, and on the first 30 unmoved test listings year_built gives 0.481, 1.122 and 2.63. Say why the PSI rises with the bins, what the 0.25 limit is tied to, and what each of the two changes, 5 or 20 bins, would do to the trigger on these samples.

Hint: Every term of the sum is zero or positive. Ask what more bins do to the number of terms and to the rows each bin holds.

Q4: The Loop That Never Stops

After the redeploy, drift.py prints retrain: drift in ['area_sqm'] again, the area at PSI 0.280 and p 0.000, so the teammate runs retrain.py again and plans a nightly job that retrains whenever drift.py names a feature. The second retrain prints champion MAE 87,081, challenger version 5 MAE 87,081 on 300 fresh rows and champion kept. Say why drift.py still fires, why the second challenger scored exactly the champion's MAE, what the nightly job would leave in the registry, and what drift.py should compare the live listings with after a retrain. Against the 2,700 rows version 4 was fitted on, the same 600 logged listings read the area at PSI 0.233 and p 0.000.

Hint: Read which rows drift_report receives as its reference in main, and which rows retrain.py trains on each time it runs.

Assignment

Assignment: How Much a Fresh Row Should Count

retrain.py gives each fresh row the same say in the fit as each old one, so the challenger learns mostly a market that no longer exists. Write ~/dev/listings-probes/weights.py, which cuts data/fresh.csv exactly as retrain.py does and fits build_pipeline() on the same 2,700 rows with each fresh row counting 1, 3, 10 and 30 times as much as an old one, and once on the 300 fresh rows alone. For each of the five, print the MAE on the 300 held-out fresh rows and on Lesson 6.1.1's 600 old test rows. Register nothing: the probe compares, and only retrain.py writes to the registry.

A fit can weigh its rows. LinearRegression.fit(X, y, sample_weight=w) takes one weight per row and minimizes the weighted sum of squared errors, so a row of weight 10 pulls on the coefficients as hard as ten copies of itself. A Pipeline hands a fit argument to one of its steps by the same double underscore Lesson 3.6.3 used to reach a step's setting: build_pipeline().fit(X, y, model__sample_weight=w) passes w to the step named model, the regression, and nothing to the preprocessing. Say in a comment which weight you would put into retrain.py, what the old test rows show about that choice, and why choosing the weight by its score on these 300 held-out rows makes that score optimistic.

Hints:

• np.r_[np.ones(len(X_train)), np.full(len(seen), weight)] joins two arrays into one, the weights in the same order as pd.concat([X_train, seen[FEATURES]]) stacks the rows.
• evaluate from train.py scores any fitted pipeline, and with weight 1 the probe must reproduce retrain.py's challenger, 87,081 on the fresh rows.

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

from listings.data import FEATURES, TARGET, load, split, to_frame
from listings.features import build_pipeline
from listings.simulate import FRESH_PATH
from listings.train import evaluate

fresh = pd.read_csv(FRESH_PATH)
seen, holdout = train_test_split(fresh, test_size=0.5, random_state=42)
X_train, X_test, y_train, y_test = split(load())
X = pd.concat([X_train, seen[FEATURES]])
y = pd.concat([y_train, seen[TARGET]])
X_holdout = to_frame(holdout[FEATURES])


def report(name, pipe):
    new = evaluate(pipe, X_holdout, holdout[TARGET])["mae"]
    old = evaluate(pipe, X_test, y_test)["mae"]
    print(f"{name:<10} fresh MAE {new:,.0f}, old test MAE {old:,.0f}")


for weight in (1, 3, 10, 30):
    w = np.r_[np.ones(len(X_train)), np.full(len(seen), weight)]
    report(f"weight {weight}",
           build_pipeline().fit(X, y, model__sample_weight=w))
report("fresh only", build_pipeline().fit(seen[FEATURES], seen[TARGET]))
# Weight 10 is the choice: 81,807 on the fresh rows against 87,081 at
# weight 1. The old rows show its price, 56,562 for the champion and
# 70,126 here, the model has moved to the new market. The weight was
# picked on these 300 rows, so 81,807 is the best of five tries on
# them, and a fair score needs rows the choice never saw.

uv run python ../listings-probes/weights.py

weight 1   fresh MAE 87,081, old test MAE 58,897
weight 3   fresh MAE 83,472, old test MAE 62,529
weight 10  fresh MAE 81,807, old test MAE 70,126
weight 30  fresh MAE 82,638, old test MAE 80,238
fresh only fresh MAE 86,276, old test MAE 100,488

Additional Resources

Optional, and nothing depends on them. They name the kinds of drift, and the rules teams follow to keep training and serving in step.

• Chip Huyen - "Data Distribution Shifts and Monitoring". Covariate shift, label shift and concept drift told apart, statistical methods and time windows for detecting a shift, and what to monitor beyond the features: accuracy, predictions and raw inputs.
• Google for Developers - "Rules of Machine Learning", Training-Serving Skew. Rules 29 to 37, among them saving the features used at serving time to train on later, testing a model on data from after its training data, and measuring training and serving skew.

Summary

The last lesson of the bootcamp adds almost no machinery: a batch of requests, two statistics in a dozen lines, and a script that fits Lesson 3.2.3's pipeline again and asks whether it beats the one in service. What it adds is a question no earlier lesson could ask, because each ended when the model was right: what happens when it stops being right and nothing breaks. The answer was a service with perfect health, three quotes in four too low, and nothing in its own logs able to tell.

Each measurement in the loop turned out to need a partner. The KS test was certain about moves too small to act on once the rows were many, the PSI saw drift in thirty listings of an unmoved market, the champion's error on the new market meant little until the challenger was scored on the same rows, and an alias that moved changed nothing until the image was rebuilt. Every one of those failures was a single number read alone, and every fix paired it with the number that could contradict it.

That is the shape of everything before this lesson too. A container reported healthy while its database connection was dead, a 200 for a boolean sent as a room count, a winner's score chosen from many tries: across seven modules the defects that cost the most looked like success, and the habit that caught them was the same each time, to ask what a number could not have seen and then measure that.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

def psi(expected, actual, bins=10):
    edges = np.unique(np.quantile(expected, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf  # every live value lands
    e = np.histogram(expected, edges)[0] / len(expected)
    a = np.histogram(actual, edges)[0] / len(actual)
    e, a = np.clip(e, 1e-4, None), np.clip(a, 1e-4, None)
    return float(np.sum((a - e) * np.log(a / e)))


ref, cur = train[col].dropna(), live[col].dropna()
ks = ks_2samp(ref, cur)                    # .statistic, .pvalue
moved = psi(ref, cur) > 0.25 and ks.pvalue < 0.01

uv run python -m listings.simulate
uv run python -m listings.drift
uv run python -m listings.retrain
uv run mlflow artifacts download -u models:/listings-price@champion -d model
docker compose up -d --build --wait