Offline Gains and Online Reality

Context

An offline evaluation (a score computed on listings already stored, the test rows here) runs every model on the same stored listings and measures each miss against a recorded price, one that on the test rows no model suggested. Online, a model prices listings that arrive after it goes live, each listing meets one model, and the suggestions it makes can change the prices and the sales it is later scored on.

A pricing team crosses from one to the other every time a model leaves the test set: the review that weighs boosting's lower MAE against the linear pipeline already in use, the product manager who asks how many sellers will get a price they can list at, the plan for an A/B test (live listings split at random between two models), and the launch report written a month later in a market where half the listings are houses.

Cross it carelessly and an offline gain of 3,788 is announced as what every seller will see, a test on a few hundred live listings has boosting looking worse about one time in four, and a live MAE measured on a month in which half the listings are houses is read as a model that got worse when only the mix of listings moved. Lesson 3.7.1 asked whether a gain on the test rows could be the luck of those 600 listings, and this lesson asks what that gain says about the listings the site prices next.

Goal

• Compute an offline gain as the mean of per-listing differences on the test rows, and name what it takes for granted about the listings a model meets online.
• Measure the share of listings priced within 10 percent, and trace where a lower MAE does and does not move it.
• Simulate A/B tests on the test listings, by random assignment and by draws with replacement, set an A/A test beside them, and read how often the model with the lower offline MAE looks worse at each size.
• Reweight per-type scores to a new mix of listings with np.average, and identify a ranking that reverses on one type without claiming why.
• Explain how a suggested price that sellers adopt, or an alert that changes what they do, can change what a model is scored on.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the idea of a test too small to see a real difference, or the shape of an A/B test, before you start.

• YouTube - "Statistical Power, Clearly Explained!!!" by StatQuest with Josh Starmer. Statistical power defined, then how the overlap between two groups' values and the size of the samples raise or lower it. It assumes p-values and the normal distribution are familiar.
• Wikipedia - "A/B testing", the section "Definition". An A/B test as a simple randomized controlled experiment between two versions, an email campaign whose two versions are measured by the sales each brings in, and a test's need for a defined, measurable outcome.

Table of Contents

• An Offline Gain Is Measured on Stored Rows
• The Metric the Business Feels
• Online, Each Listing Sees One Model
• The Mix of Listings Moves the Score
• When the Model Changes What It Is Scored On

Theory

Nothing to install: the models arrived with Lesson 3.1.1's scikit-learn, and NumPy and pandas with Module 2. Every Theory block goes into one script, online.py, saved in ~/dev/bootcamp beside listings.csv and price_models.py, the module saved in Lesson 3.5.1, and run with python3 online.py, each block added under the one before it.

The three models are Section 3.7's: Lesson 3.2.3's linear pipeline, Lesson 3.5.2's chosen forest, max_features=0.5 with 100 trees, and Lesson 3.5.3's HistGradientBoosting at its defaults. Each had its settings fixed before the test rows scored it, each is fitted once here on the 2,400 training rows and never again, and every block after the first works on the 600 test listings' misses. No comparison in this lesson chooses a model, Lesson 3.1.2's rule: the test rows only measure what each one does.

An Offline Gain Is Measured on Stored Rows

What. An offline evaluation scores every model on the same stored listings, each miss measured against the price recorded for that listing, and on these test rows every price was set before any model suggested one. An online evaluation (what a model achieves once its predictions are used, on listings that arrive after it goes live) scores it on listings that do not exist yet when the decision to use it is made. A gain between two models, measured offline, is the mean of their per-listing differences on stored rows. Why. Every score in this module is offline, and a team decides whether to put a model in front of sellers on such a number. What the number promises about use depends on how much of the offline setting carries over: the measure, the listings, which model prices which listing, and what each prediction is scored against. When. Whenever an offline gain becomes a claim about the site: a model review, a launch plan, a report to the people who run the business.

The first block fits the three models and keeps each one's absolute miss on every test listing:

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from price_models import X_test, X_train, y_test, y_train, fill, make_prep

models = {
    "linear": make_pipeline(make_prep(fill()), LinearRegression()),
    "forest": make_pipeline(
        make_prep("passthrough"),
        RandomForestRegressor(max_features=0.5, random_state=42),
    ),
    "boosting": make_pipeline(
        make_prep("passthrough"),
        HistGradientBoostingRegressor(random_state=42),
    ),
}
prices = y_test.to_numpy()
errors = {}
for name, model in models.items():
    model.fit(X_train, y_train)  # once, and never again
    errors[name] = np.abs(prices - model.predict(X_test))
    print(name, f"{errors[name].mean():,.0f}")

linear 56,562
forest 53,802
boosting 52,773

The linear pipeline's 56,562 and boosting's 52,773 are the test MAEs Lessons 3.2.3 and 3.5.3 printed, and the forest Lesson 3.5.2 scored at an R² of 0.775 misses the average test listing by 53,802. Boosting's gain over the linear pipeline, listing by listing:

# one difference per listing, the linear miss less boosting's
gain = errors["linear"] - errors["boosting"]
print(f"{gain.mean():,.0f}")
paired = pd.DataFrame(errors, index=X_test.index)
print(paired.assign(price=prices).head(4).round().astype(int))

3,788
      linear  forest  boosting   price
1801   13529   23600      6784  173000
1190   74584   37350     40308  420000
1817   34186  116850    124107  668000
251    36377   27220     30459  520000

The gain is 3,788, the difference between the two MAEs before rounding, which is why subtracting the printed 52,773 from 56,562 gives 3,789. It is a mean of 600 differences, and the four listings show what each difference is: one stored listing, shown by its row label, priced by both models and measured against one recorded price. Boosting misses listings 1801, 1190 and 251 by less, and misses listing 1817, listed at 668,000, by about 90,000 more than the linear pipeline does. Lesson 3.7.1 asked whether a mean of such differences could be the luck of these 600 listings, and its paired interval for this gain stays above 0.

That settles what the gain is on these rows. What it promises about the listings the site prices next rests on four things the offline number takes for granted: that MAE is what the business counts, that both models price the same listings, that the next listings arrive in the same mix as these 600, and that the price each listing is scored against does not depend on which model priced it. The four subtitles that follow take them away one at a time.

DON'T:

• DON'T announce an offline gain as an online one. 3,788 is a mean over 600 stored listings, each priced by both models against a price neither suggested.
• DON'T read 3,788 as what each listing gains. Boosting missed listing 1817 by about 90,000 more than the linear pipeline.

The Metric the Business Feels

What. A business metric (a number the business acts on, stated in its own terms) is chosen by the question the business asks, Lesson 3.1.4's rule, and it need not be the metric the models were compared on. A listings site judges a suggested price by whether it lands close to the listing's price in proportion to that price, so its metric here is the share of listings priced within 10 percent of their price: every listing counts once, and a miss beyond the line counts the same however far beyond it falls. Why. MAE averages misses in currency, so a model can lower it by shrinking misses that stay beyond any line a seller would accept, and a count of listings priced close never sees that. A gain in one number can be small, missing or reversed in the other. When. Before any model is scored, the business names its line, as Lesson 3.4.2's seller alert named its metric before a model was scored. Every offline gain is then read in that metric as well as in the one the models were compared on.

for name, err in errors.items():
    close = err / prices <= 0.10  # within 10 percent
    print(name, close.sum(), round(close.mean(), 3))

linear 269 0.448
forest 268 0.447
boosting 287 0.478

The forest's MAE is 2,760 below the linear pipeline's, and it prices 268 of the 600 test listings within 10 percent, one fewer than the linear pipeline's 269. Boosting, 3,788 below the linear pipeline, prices 18 more, 287, a share of 0.478 against 0.448. One lower MAE came with a listing fewer and the other with 18 more, so the size of an MAE gain fixes nothing about the count. Where the forest's gain went, by whether each model prices each listing within 10 percent:

table = pd.DataFrame({
    "linear_close": errors["linear"] / prices <= 0.10,
    "forest_close": errors["forest"] / prices <= 0.10,
    "gain": errors["linear"] - errors["forest"],
})
groups = table.groupby(["linear_close", "forest_close"])["gain"]
print(groups.agg(["count", "sum"]).round())
print(round(table["gain"].sum()))

                           count        sum
linear_close forest_close                  
False        False           235  1907169.0
             True             96  4142479.0
True         False            97 -3975468.0
             True            172  -418242.0
1655938

Each row is one group of listings, by whether the linear pipeline and the forest price it within 10 percent, with its count and the forest's gain summed over it. The four sums add up to 1,655,938, which is 600 times the forest's gain before it is rounded to 2,760. The 235 listings neither model prices within 10 percent carry 1,907,169, more than the whole gain: on the 172 that both price within 10 percent the forest misses by 418,242 more in all, and the 96 listings it brings inside the line and the 97 it pushes out nearly cancel. The forest's lower MAE comes from listings that stay beyond 10 percent under both models, where its misses are smaller in total, and between the two models the share within 10 percent moves only when a listing crosses the line. Where the business draws the line matters as much:

for line in [0.05, 0.10, 0.15, 0.20]:
    shares = [(err / prices <= line).mean() for err in errors.values()]
    print(line, np.round(shares, 3))

0.05 [0.268 0.217 0.227]
0.1 [0.448 0.447 0.478]
0.15 [0.567 0.617 0.635]
0.2 [0.698 0.752 0.753]

Each line holds the share within that line for the linear pipeline, the forest and boosting, in that order. At 5 percent the linear pipeline prices the most listings close, 0.268 against boosting's 0.227 and the forest's 0.217. At 10 percent boosting leads and the other two sit one listing apart, and at 15 and 20 percent the forest passes the linear pipeline while boosting stays ahead of both, a single listing ahead of the forest at 20 percent. A business that named its line after reading this table could have crowned the linear pipeline or boosting, which is why the question names the line first, Lesson 3.4.2's order.

DON'T:

• DON'T read a lower MAE as more listings priced well. The forest's MAE is 2,760 lower than the linear pipeline's, and it prices 268 listings within 10 percent against 269.
• DON'T set the line after seeing the shares. At 5 percent the linear pipeline led with 0.268 against boosting's 0.227, and at 10 percent boosting led with 0.478 against 0.448.

Online, Each Listing Sees One Model

What. An A/B test splits live listings at random between two models and prices each listing with one of them, so it compares two groups of different listings: each model's MAE is a mean over its own half, and each listing gets one miss, from the one model that priced it, so no listing's two misses can be paired. An A/A test (the same random split with one model pricing both halves) measures what the split does by itself. Why. Offline, every listing is priced by both models, so the difficulty its two misses share cancels in its own difference, the pairing of Lesson 3.7.1. Online, the hard listings land in one half or the other by chance, that luck enters the difference between the halves, and the same gain needs far more listings to show. When. Before an A/B test runs, to size it for the gain it looks for, and again before its result is read.

The 600 test listings stand in for live ones. Each test shuffles their positions with rng.permutation, which Lesson 3.6.1's Assignment used to reshuffle the fit rows, and splits them into two halves of 300, each priced by one model, and the difference is the first half's MAE minus the second's, so with the linear pipeline first, a negative difference is a test in which boosting looks worse. The simulation assumes that live listings arrive like the test rows and that each model misses each of them exactly as it missed it on the test rows. 2,000 A/B tests, then 2,000 A/A tests with the linear pipeline on both halves:

lin, boost = errors["linear"], errors["boosting"]
rng = np.random.default_rng(42)


def split_test(errors_a, errors_b):
    order = rng.permutation(600)  # 0 to 599, shuffled
    # the first 300 go to one model, the last 300 to the other
    return errors_a[order[:300]].mean() - errors_b[order[300:]].mean()


ab = np.array([split_test(lin, boost) for _ in range(2000)])
aa = np.array([split_test(lin, lin) for _ in range(2000)])
for diffs in [ab, aa]:
    low, high = np.percentile(diffs, [2.5, 97.5])
    print(round(low), round(high), round((diffs < 0).mean(), 3))
print(round((np.abs(aa) >= gain.mean()).mean(), 3))

-7109 15447 0.262
-11647 11543 0.506
0.548

95 percent of the A/B tests land between -7,109 and 15,447, and in 26.2 percent of them boosting looks worse. The A/A tests spread about as wide, from -11,647 to 11,543 around 0: with one model on both halves, that whole width is which listings landed in which half, and in 54.8 percent of A/A tests it opened a gap at least as large as the offline gain, one way or the other. The A/B range is the same kind of luck, centered on the gain instead of on 0. Larger tests are simulated by drawing positions from the 600 with replacement, as Lesson 3.5.2's bootstrap drew rows, the first half of each draw priced by the linear pipeline and the second by boosting, with the same generator carrying on:

def ab_test(n_listings):
    # positions drawn with replacement, half per model
    drawn = rng.integers(0, 600, size=n_listings)
    half = n_listings // 2
    return lin[drawn[:half]].mean() - boost[drawn[half:]].mean()


for n_listings in [600, 1200, 2400, 4800, 9600]:
    diffs = np.array([ab_test(n_listings) for _ in range(2000)])
    low, high = np.percentile(diffs, [2.5, 97.5])
    print(n_listings, round(low), round(high), round((diffs < 0).mean(), 3))

600 -8076 16017 0.25
1200 -4405 12194 0.191
2400 -2289 9458 0.106
4800 -321 7911 0.034
9600 885 6564 0.004

Each line is a test size, the range holding 95 percent of its differences, and the share of tests in which boosting looks worse. Drawn with replacement, 600 listings put boosting behind in a quarter of tests, as the random split did. With 600 listings on each side, 1,200 in all, boosting is still behind in 19.1 percent of tests, though Lesson 3.7.1's paired comparison of 600 listings already told this gain from 0. The share falls to 3.4 percent at 4,800 listings and 0.4 percent at 9,600, and even there the difference runs from 885 to 6,564 around a gain of 3,788. A gain that 600 paired listings show offline needs an online test of thousands to show.

DON'T:

• DON'T read which model a small A/B test puts ahead as the better one. Split at random into halves of 300, the test listings put boosting behind in 26.2 percent of tests, though it misses them by 3,788 less on average.
• DON'T size an online test from the offline comparison. With 600 listings on each side boosting was still behind in 19.1 percent of tests, and it took 9,600 listings to bring that to 0.4 percent.

The Mix of Listings Moves the Score

What. The mix of a set of listings is each group's share of it. A score that is a mean over listings, MAE or the share within 10 percent, equals each group's score weighted by that group's share, so a new mix moves the score with no change to the model. Lesson 2.2.5's np.average with weights= computes the score at any mix from the per-group scores. Why. The listings a site prices are not its test rows: another city, another season or a campaign aimed at houses brings its own share of each type. A live score set beside an offline one, or this month's beside last month's, compares two mixes as much as two models. When. Whenever two scores come from different sets of listings, before the difference between them is read as a change in a model.

Each model's test MAE on each property type, with Lesson 2.6.2's groupby, and each type's share of the test rows:

types = X_test["property_type"]
by_type = paired.groupby(types).mean()
print(by_type.round().astype(int))
shares = types.value_counts(normalize=True)
print(shares.round(3))

               linear  forest  boosting
property_type                          
apartment       49567   45061     42633
house           75451   74549     74288
townhouse       53384   53175     55038
property_type
apartment    0.555
house        0.240
townhouse    0.205
Name: proportion, dtype: float64

Every model's MAE is highest on houses. The table shows that, nothing in this lesson removed a cause for it, and none is claimed. Apartments are 55.5 percent of the test rows. Weighted by those shares, the per-type MAEs return each model's test MAE, and weighted by a market of half houses and a quarter each of apartments and townhouses, they give the MAE each model would score on such a market, if each type's listings there were missed as the test rows' listings of that type were:

# each type's MAE weighted by its share of the test rows
same_mix = np.average(by_type, axis=0, weights=shares[by_type.index])
print(np.round(same_mix))
market = pd.Series({"apartment": 0.25, "house": 0.5, "townhouse": 0.25})
houses_half = np.average(by_type, axis=0, weights=market[by_type.index])
print(np.round(houses_half))
print(round(houses_half[0] - houses_half[2]))

[56562. 53802. 52773.]
[63463. 61834. 61562.]
1901

np.average weights the rows of by_type, the three types, so weights= lists the shares in that same order, which shares[by_type.index] and market[by_type.index] make sure of. The test rows' own shares give back 56,562, 53,802 and 52,773. The house-heavy market moves the linear pipeline to 63,463 and boosting to 61,562 with no change to either model, so the linear pipeline's MAE moves by 6,901 with the mix alone, more than the 3,788 between the two models on the test rows. Boosting's gain shrinks to 1,901. It is a weighted mean of the gain on each type, 6,934 on the average apartment, 1,163 on the average house and a loss on townhouses, 55,038 against the linear pipeline's 53,384, and the house-heavy market moves weight from the type where boosting gains most to one where it gains little. The reversal on townhouses, on the townhouses alone:

town = (types == "townhouse").to_numpy()
town_gain = gain[town]            # the 123 townhouses' gains
print(town.sum(), round(town_gain.mean()), (town_gain > 0).sum())
boot = np.random.default_rng(42)  # a generator of its own
picks = boot.integers(0, 123, size=(2000, 123))
means = town_gain[picks].mean(axis=1)
print(np.percentile(means, [2.5, 97.5]).round())

123 -1654 62
[-7904.  5101.]

Boosting prices 62 of the 123 townhouses closer than the linear pipeline and still misses them by 1,654 more on average. Lesson 3.7.1's paired interval for the linear miss less boosting's, built here from 2,000 resamples of the 123 townhouses, runs from -7,904 to 5,101. These listings cannot say which model prices townhouses better. A per-type ranking rests on a fraction of the rows, and three types are three comparisons, Lesson 3.7.1's warning about many of them, so this reversal is a reason to measure that type online, not a fact about it.

DON'T:

• DON'T compare a live score with an offline one across different mixes. Reweighted to half houses, boosting's MAE moved from 52,773 to 61,562 with no change to the model.
• DON'T read one type's reversal as a fact about that type from its mean alone. Boosting's MAE on the 123 townhouses is 1,654 above the linear pipeline's, and the paired interval for the linear MAE less boosting's runs from -7,904 to 5,101.

When the Model Changes What It Is Scored On

What. Every score Sections 3.1 to 3.6 computed for a model compares its predictions with a price or a label recorded without the model. Once a model is used, its output reaches the people whose choices produce those prices and labels, which can make a feedback loop (the model's output changing the data it is later fitted on or scored against), and an online score can then measure the model partly against its own effect. Why. A seller who lists at the suggested price makes the suggestion the listing's price, and an alert that changes what a seller does can change whether the listing sells within 30 days. The stored rows were recorded before any model existed, so no run on them can measure either effect, and this subtitle names both without one. When. Whenever a model's output is shown to the people who set the prices or make the sales it is scored on: every suggested price a seller can adopt, every alert a seller reads.

The suggested price first. The test rows' prices were set by sellers who never saw a suggestion. After launch, a seller who takes the suggestion lists at it, and that listing's miss, measured against its listed price, is 0 for whichever model made the suggestion. A live MAE against listed prices then mixes two things the offline MAE kept apart: how close the model comes to the price a seller would have chosen alone, and how many sellers took its number. A later model fitted on those listings learns the earlier model's suggestions as prices, its misses included, and is scored against them in turn.

The alert works the same way on the label. Lesson 3.4.3's seller alert, at the threshold its validation rows chose, 0.1546, went to 396 of the 600 test listings, and 0.263 of those sold within 30 days. Those listings sold or did not with no alert sent. Once alerts go out, a seller told to expect a fast sale can hold the price, cut it, or change nothing, and what the seller does can change whether the listing sells within 30 days. The share of alerted listings that sell is then measured on outcomes the alert helped decide, so 0.263 is an offline number until a test that sends real alerts measures what they achieve.

Neither effect has a size or a direction in this lesson: no run here can give one, and none is claimed. Beside them sits drift (the listings a model meets changing over time), the mix of the last subtitle moving while the model is live, which Module 6 watches on a running service.

DON'T:

• DON'T read Lesson 3.4.3's alert precision as what live alerts will achieve. Its 0.263 was measured on listings that sold or did not with no alert sent.
• DON'T score a suggested price against listed prices that sellers may have taken from it. Every seller who lists at the suggestion gives it a miss of 0, whichever model made it.

Lab Exercises

Lab Exercise 1: Too Low, Too High

Work in price_sides.py in ~/dev/bootcamp, importing from price_models.py. The sellers' team watches the listings priced more than 10 percent below their price, which would cost a seller who took the suggestion money, and the listings team watches those priced more than 10 percent above it, which can sit unsold, so each team reads one side of the line. Fit the Theory's three models once on the training rows, and for each one print its test MAE in whole units beside an array of three counts: the test listings it prices more than 10 percent too low, within 10 percent, and more than 10 percent too high. Then print how the three counts move from the linear pipeline to the forest and from the linear pipeline to boosting, and say in a comment what the forest's lower MAE did to each side.

Lab Exercise 2: The Forest Online, and a Launch in Two Neighborhoods

Work in forest_online.py, importing from price_models.py. Fit the linear pipeline and the forest once, keep each one's absolute miss on every test listing, and print the forest's offline gain, the linear pipeline's MAE minus the forest's. With one generator, np.random.default_rng(42), split the 600 test listings at random into two halves of 300, one per model, 2,000 times, and print the share of tests in which the forest looks worse. With the same generator, simulate 2,000 tests each of 2,400, 9,600 and 38,400 listings drawn with replacement, half per model, and print each size beside its share. Last, print both models' MAE in each neighborhood beside the number of test listings there, the neighborhoods where the forest's MAE is above the linear pipeline's, and each model's MAE in a launch market of Downtown and Riverside listings, half each, with the forest's gain in it and that gain's paired 95 percent interval from 2,000 resamples of each neighborhood's test listings, drawn Downtown first with a fresh np.random.default_rng(42).

import numpy as np
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from price_models import X_test, X_train, y_test, y_train, fill, make_prep

models = {
    "linear": make_pipeline(make_prep(fill()), LinearRegression()),
    "forest": make_pipeline(
        make_prep("passthrough"),
        RandomForestRegressor(max_features=0.5, random_state=42),
    ),
    "boosting": make_pipeline(
        make_prep("passthrough"),
        HistGradientBoostingRegressor(random_state=42),
    ),
}
prices = y_test.to_numpy()
counts = {}
for name, model in models.items():
    predicted = model.fit(X_train, y_train).predict(X_test)
    off = (predicted - prices) / prices  # signed, below 0 too low
    counts[name] = np.array([
        (off < -0.10).sum(),             # more than 10 percent low
        (np.abs(off) <= 0.10).sum(),     # within 10 percent
        (off > 0.10).sum(),              # more than 10 percent high
    ])
    mae = np.abs(predicted - prices).mean()
    print(name, f"{mae:,.0f}", counts[name])
print(counts["forest"] - counts["linear"])
print(counts["boosting"] - counts["linear"])

# The forest's lower MAE took 14 off the too-low count and put 15 on
# the too-high count, with one listing fewer within 10 percent.

linear 56,562 [126 269 205]
forest 53,802 [112 268 220]
boosting 52,773 [101 287 212]
[-14  -1  15]
[-25  18   7]

import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from price_models import X_test, X_train, y_test, y_train, fill, make_prep

linear = make_pipeline(make_prep(fill()), LinearRegression())
forest = make_pipeline(
    make_prep("passthrough"),
    RandomForestRegressor(max_features=0.5, random_state=42),
)
prices = y_test.to_numpy()
lin = np.abs(prices - linear.fit(X_train, y_train).predict(X_test))
fst = np.abs(prices - forest.fit(X_train, y_train).predict(X_test))
print(f"{lin.mean() - fst.mean():,.0f}")

rng = np.random.default_rng(42)
assigned = []
for _ in range(2000):
    order = rng.permutation(600)  # 300 listings a side
    assigned.append(lin[order[:300]].mean() - fst[order[300:]].mean())
print(600, round((np.array(assigned) < 0).mean(), 3))
for n_listings in [2400, 9600, 38400]:
    diffs = []
    for _ in range(2000):
        # positions drawn with replacement, half per model
        drawn = rng.integers(0, 600, size=n_listings)
        half = n_listings // 2
        diffs.append(lin[drawn[:half]].mean() - fst[drawn[half:]].mean())
    print(n_listings, round((np.array(diffs) < 0).mean(), 3))

hoods = X_test["neighborhood"]
table = pd.DataFrame({"linear": lin, "forest": fst}, index=X_test.index)
by_hood = table.groupby(hoods).mean()
# value_counts lines up with by_hood by neighborhood name
by_hood["listings"] = hoods.value_counts()
print(by_hood.round().astype(int))
print(list(by_hood.index[by_hood["forest"] > by_hood["linear"]]))
# half Downtown and half Riverside, so a plain mean of the two rows
launch = by_hood.loc[["Downtown", "Riverside"], ["linear", "forest"]].mean()
print(launch.round().astype(int).to_dict())
print(round(launch["linear"] - launch["forest"]))

# each neighborhood's gains resampled on their own, then half each
gain = lin - fst
boot = np.random.default_rng(42)  # a generator of its own
halves = []
for hood in ["Downtown", "Riverside"]:
    hood_gain = gain[(hoods == hood).to_numpy()]
    picks = boot.integers(0, len(hood_gain), size=(2000, len(hood_gain)))
    halves.append(hood_gain[picks].mean(axis=1))
print(np.percentile((halves[0] + halves[1]) / 2, [2.5, 97.5]).round())

2,760
600 0.33
2400 0.182
9600 0.037
38400 0.0
              linear  forest  listings
neighborhood                          
Downtown       54460   56896       133
Eastgate       97656   88823        60
Hillcrest      52967   44223        65
Lakeview       71770   69350        84
Northend       48645   43070        75
Riverside      42799   41528       112
Southgate      41140   39483        71
['Downtown']
{'linear': 48629, 'forest': 49212}
-582
[-4730.  3253.]

Theory Assessment

Q1: What 3,788 Is an Average Of

A model review states: "HistGradientBoosting cuts the miss on a listing by 3,788." Using the four test listings the first subtitle printed and the gain itself, say what 3,788 is a mean of, over which listings, against which prices, and why the review's sentence is wrong about listing 1817. Then name the four things the number takes for granted once it is carried to the listings the site prices next.

Hint: Read listing 1817's two misses. Then ask who set the 668,000 that every miss on it is measured against, and whether that would still hold after launch.

Q2: One Listing Fewer for 2,760

The forest's test MAE is 2,760 below the linear pipeline's, and it prices 268 of the 600 test listings within 10 percent against the linear pipeline's 269. Use the four groups of the second subtitle's table to explain how both hold: what each group's sum is, which groups the share within 10 percent can see, and where the forest's lower MAE was earned. Then say what the same three models would have told a business that drew its line at 5 percent.

Hint: The four sums add up to 600 times the forest's gain before rounding. A listing moves the share within 10 percent only by crossing the line.

Q3: A Test That Beats Itself

In the third subtitle's simulation, splitting the 600 test listings at random between the linear pipeline and boosting gave differences from -7,109 to 15,447, and splitting them between two copies of the linear pipeline gave -11,647 to 11,543. Explain why the second range is about as wide as the first, what the 0.548 printed after them measures, and why 1,200 listings, 600 per side, still put boosting behind in 19.1 percent of tests while Lesson 3.7.1's paired comparison of 600 listings told the same gain from 0.

Hint: Ask what differs between the two halves of an A/A test. Then ask what a listing's own difference removes that two halves of different listings keep.

Q4: The Report After Launch

A month after boosting went live on every listing, a report reads: "Live MAE is 61,000 on this month's 2,000 listings, against 52,773 offline, so the model has degraded online. The linear pipeline scored 56,562 offline, so switching back would save 4,438 per listing." This month's listings were half houses and a quarter each apartments and townhouses, and every seller was shown the suggested price before listing. Say what each comparison in the report sets side by side, which of its numbers the lesson's runs show could come out this way with no change to either model, what the suggested price does to a live MAE, and what evidence would answer whether to switch back.

Hint: The fourth subtitle priced exactly this month's mix. Then ask on which listings, in which mix, the 56,562 was measured, and what a seller who lists at the suggestion does to a miss.

Assignment

Assignment: The Launch Test, Sized in Two Cities

The site will A/B test boosting against the linear pipeline before switching, and judges the test on the share of listings priced within 10 percent, with MAE reported beside it. Next quarter it opens in a second city, where half the listings are houses and a quarter each are apartments and townhouses, each type assumed to be priced there as the test rows' listings of that type are. Work in launch_test.py, importing from price_models.py, with the linear pipeline and boosting fitted once on the training rows, and give each test listing two outcomes per model: its absolute miss, and a 1 or a 0 for whether that model prices it within 10 percent.

One A/B test ends in one comparison of two groups of different listings. ttest_ind, the unpaired t-test Lesson 3.7.1's Q4 set beside ttest_rel, compares two such groups, and ttest_ind(b, a, equal_var=False) runs it as Welch's t-test, which does not assume the two groups spread equally. Its result's .pvalue is the test's p-value for the difference between the two means. The share of simulated tests that find, with a p-value below 0.05, a lead the listings really hold is the test's power (the chance that a test of that size detects the gain). rng.choice(600, size=n, p=chances) draws n positions from 0 to 599 with replacement, position i with chance chances[i], the chances summing to 1, and with p left out every position is equally likely.

1. In the first city, drawn like the test rows: for tests of 2,400, 9,600 and 38,400 listings, 2,000 tests per size and half the listings per model, print the power and the share of tests in which boosting looks behind, on the business metric and on MAE.
2. In the second city: reweight each model's per-type MAE and per-type share within 10 percent to the city's mix, and print all four.
3. Give each test listing the chance that makes each property type's share of the draws the city's share, and simulate 2,000 tests of 38,400 listings there on each metric. Print, for each metric, the share of tests in which boosting is ahead, and the shares that reach a p-value below 0.05 with boosting ahead and with the linear pipeline ahead.

Use one generator, np.random.default_rng(42), for every draw, in the order above, with the business metric before MAE at each size, and end with comments saying what each city's test can find.

Hints:

• A listing's chance in the second city is its type's share of the city divided by the number of test listings of that type, so the 144 test houses split 0.5 between them, and the chances of all 600 sum to 1.
• Track boosting's lead as a number that is positive when boosting is ahead: on the share within 10 percent that is its share minus the linear pipeline's, and on MAE the linear pipeline's MAE minus its own, because the lower miss leads.

import numpy as np
import pandas as pd
from scipy.stats import ttest_ind
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from price_models import X_test, X_train, y_test, y_train, fill, make_prep

linear = make_pipeline(make_prep(fill()), LinearRegression())
boosting = make_pipeline(
    make_prep("passthrough"), HistGradientBoostingRegressor(random_state=42)
)
prices = y_test.to_numpy()
outcomes = {}
for name, model in [("linear", linear), ("boosting", boosting)]:
    miss = np.abs(prices - model.fit(X_train, y_train).predict(X_test))
    outcomes[f"mae_{name}"] = miss
    outcomes[f"close_{name}"] = (miss / prices <= 0.10).astype(float)
outcomes = pd.DataFrame(outcomes, index=X_test.index)
rng = np.random.default_rng(42)


def launch_tests(metric, n_listings, chances=None):
    # boosting's lead and Welch's p-value in each of 2,000 tests
    a_all = outcomes[f"{metric}_linear"].to_numpy()
    b_all = outcomes[f"{metric}_boosting"].to_numpy()
    sign = 1 if metric == "close" else -1  # a lower miss leads
    half = n_listings // 2
    leads, pvalues = [], []
    for _ in range(2000):
        drawn = rng.choice(600, size=n_listings, p=chances)
        a, b = a_all[drawn[:half]], b_all[drawn[half:]]
        leads.append(sign * (b.mean() - a.mean()))
        pvalues.append(ttest_ind(b, a, equal_var=False).pvalue)
    return np.array(leads), np.array(pvalues)


for n_listings in [2400, 9600, 38400]:
    for metric in ["close", "mae"]:
        leads, pvalues = launch_tests(metric, n_listings)
        power = ((leads > 0) & (pvalues < 0.05)).mean()
        behind = (leads < 0).mean()
        print(metric, n_listings, round(power, 3), round(behind, 3))

close 2400 0.309 0.062
mae 2400 0.25 0.107
close 9600 0.842 0.001
mae 9600 0.716 0.006
close 38400 1.0 0.0
mae 38400 0.999 0.0

types = X_test["property_type"]
city = pd.Series({"apartment": 0.25, "house": 0.5, "townhouse": 0.25})
by_type = outcomes.groupby(types).mean()
in_city = pd.Series(
    np.average(by_type, axis=0, weights=city[by_type.index]),
    index=by_type.columns,
)
print(in_city[["mae_linear", "mae_boosting"]].round().astype(int).tolist())
print(in_city[["close_linear", "close_boosting"]].round(3).tolist())

[63463, 61562]
[0.466, 0.462]

# each listing's chance: its type's share in the city, split evenly
chances = (city / types.value_counts())[types].to_numpy()
print(round(chances.sum(), 6))
for metric in ["close", "mae"]:
    leads, pvalues = launch_tests(metric, 38400, chances)
    found = pvalues < 0.05
    ahead = (leads > 0).mean()
    ahead_found = ((leads > 0) & found).mean()
    behind_found = ((leads < 0) & found).mean()
    print(metric, round(ahead, 3), round(ahead_found, 3), round(behind_found, 3))

# First city: 9,600 listings find boosting's lead on the share within
# 10 percent in 0.842 of tests. Second city: that share puts the linear
# pipeline 0.004 ahead, and 38,400 listings find the linear pipeline
# ahead at p below 0.05 in 0.15 of tests and boosting in 0.002, while
# MAE still finds boosting ahead. The metric named first decides.

1.0
close 0.197 0.002 0.15
mae 0.993 0.726 0.0

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want to size a test before it runs, or to run the unpaired test on two groups of your own.

• YouTube - "Power Analysis, Clearly Explained!!!" by StatQuest with Josh Starmer. Why an experiment gets a power analysis before it runs, the two main things that affect power, how the sample size moves it, and how to work out the sample size a chosen power needs.
• Official docs - SciPy v1.18.0 Manual, scipy.stats.ttest_ind. The t-test for the means of two independent samples, its equal_var parameter, which with False runs Welch's t-test and drops the assumption of equal spreads, its alternative parameter for a one-sided test, and the result's confidence_interval() for the difference between the means.

Summary

Every number before this lesson was a fact about stored rows. The test set was held out, the settings were fixed first, and the gain, tested alone, held up against the luck of the 600 listings. All of it stayed a statement about listings already priced by sellers who never saw a model. The offline number is still where the work starts, because it is the one place where both models face the same listings and nothing either model does can reach the prices it is scored against.

What changed online was not the models but everything around them. The business counted listings priced close, and a gain in currency turned out to sit in misses that shrank without crossing the line a seller would accept. Each listing met one model, and the luck of which listings landed in which half swamped a gain that pairing had told from 0. The listings came in another mix, and on the test rows reweighted to it the same models scored thousands worse without changing at all. And the prices themselves could start to follow the suggestions.

None of this makes the offline gain wrong. It makes it one claim among the several a launch has to test: on the metric the business named before it looked, with a test large enough for the gain, at the mix the site will actually see, and with an eye on what the model's own output does to the data it is scored on next. Module 6 watches the third of these, the mix moving as drift, on a running service.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

close = errors["boosting"] / prices <= 0.10  # the business line
share = close.mean()                         # share within 10 percent

order = rng.permutation(600)                 # one random split
ab = lin[order[:300]].mean() - boost[order[300:]].mean()
drawn = rng.integers(0, 600, size=9600)      # a larger test

# a mean score at another mix of types
mixed = np.average(by_type, axis=0, weights=market[by_type.index])

p = ttest_ind(b, a, equal_var=False).pvalue  # Welch, unpaired