Bias, Variance, Capacity and Learning Curves

Context

A model misses new listings for two reasons that one validation score cannot tell apart. Its form may be too coarse to follow the prices whatever rows it is fitted on, which is bias (the error a model makes even when its predictions are averaged over many training sets), or its prices may move with the particular rows it happened to get, which is variance (how far its predictions move when its training rows change). A learning curve, Lesson 2.7.2's score plotted against how much data a model trained on, shows what more rows do to its miss.

A pricing team meets the question whenever a model disappoints and the budget pays for one fix: a partner site's offer of more listings, a deeper tree or a bigger forest, or a feature someone wants to build. Each fix answers one of the two errors, and some make the other worse, so the diagnosis decides where the money goes before anyone spends it.

Skip the diagnosis and the money goes to the wrong fix. Rows are bought for a linear model that more rows cannot help, a tree whose error is already its movement is given more depth, one tree's lucky validation score is read as the model's, and a learning curve is read from a single point that the next slice of rows overturns. Lesson 3.5.2 accounted for a forest's gain as the spread of its trees, and this lesson finds that spread in every model and sets it beside the error that no averaging removes.

Goal

• Distinguish bias from variance on the linear pipeline, a depth-2 tree, a fully grown tree and the forest, and identify which of the two makes up most of each one's validation miss.
• Measure both by refitting a model on 30 bootstrap samples of the fit rows, as the spread of each listing's predictions and the miss of their average, and tie the two to Lesson 3.5.2's accounting.
• Trace how a tree's depth and leaf size trade the averaged miss against the spread, and apply less capacity as the first answer to variance.
• Build learning curves on growing slices of the fit rows, draw four on one 2-by-2 figure, and check the figure by printing every line it holds.
• Read a learning curve's gap and slope beside the refits, confirm each reading on a second order of the rows, and name the step it points to: more rows, more capacity, less capacity, averaging or a better form.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the two errors drawn on a small example before you meet them on listings.

• YouTube - "Lecture 08 - Bias-Variance Tradeoff" by Yaser Abu-Mostafa, Caltech's CS 156, from 4:51. Bias and variance defined in general and worked out on a sine curve learned from two points at a time, where a constant beats a straight line because the line moves so much more from one pair of points to the next, then from 46:18 the learning curves of a simple and a complex model, the error on the training examples rising and the error on new ones falling as examples are added.
• Official docs - scikit-learn 1.9 example, "Underfitting vs. Overfitting". Polynomials of degree 1, 4 and 15 fitted to 30 noisy points of part of a cosine curve, the first too simple to follow it and the last following the noise, each scored by cross-validated mean squared error.

Table of Contents

• Two Ways to Be Wrong
• Measuring Both by Refitting
• Capacity
• Learning Curves
• Reading a Learning Curve

Theory

Nothing to install: scikit-learn arrived in Lesson 3.1.1 and matplotlib in Lesson 2.7.2. Every script here imports from price_models.py, the module Lesson 3.5.1 saved in ~/dev/bootcamp beside listings.csv, so the rows are that lesson's: 1,800 fit rows and 600 validation rows carved from the 2,400 training rows. The blocks of the first three subtitles go into refits.py and those of the last two into curves.py, each block added under the one before it, run with python3 refits.py and python3 curves.py. Every comparison is made on the validation rows and the test rows take no part, because a diagnosis is a choice about what to do next, and Lesson 3.1.2 keeps every choice off the test rows.

Two Ways to Be Wrong

What. A model's miss on a new listing has two sources. Bias (the error a model makes even when its predictions are averaged over many training sets, because its form cannot follow the prices) is there whichever rows the model is fitted on. Variance (how far a model's predictions move when its training rows change) comes from the particular rows it got. This bias is not Lesson 2.3.1's bias, the b of w . x + b that scikit-learn keeps in intercept_: that one is a parameter, and this one is an error. Why. The two call for opposite steps. A model held back by bias needs room to follow the prices, a model held back by variance needs to move less, and one validation score mixes the two, so on its own it cannot say which step to take. When. Every time a validation score disappoints, before deciding what to change, and before more rows are bought.

Lesson 3.5.1 grew fully grown trees on the two halves of the fit rows and found them pricing the validation listings 96,927 apart on average. refits.py opens by growing the same two halves to depth 2 and fully:

import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.pipeline import make_pipeline
from price_models import (
    X_fit, X_val, y_fit, y_val, fill, make_prep, make_tree,
)

print(y_val.min(), y_val.max())  # 71000 1358000
for depth in [2, None]:
    halves = [
        make_tree(max_depth=depth).fit(X_fit.iloc[:900], y_fit.iloc[:900]),
        make_tree(max_depth=depth).fit(X_fit.iloc[900:], y_fit.iloc[900:]),
    ]
    pred_a, pred_b = [half.predict(X_val) for half in halves]
    gap = np.abs(pred_a - pred_b).mean()
    miss_a = mean_absolute_error(y_val, pred_a)
    miss_b = mean_absolute_error(y_val, pred_b)
    distinct = len(np.unique(pred_a))
    print(depth, distinct, round(gap), round(miss_a), round(miss_b))

2 4 45900 94842 92559
None 262 96927 78488 82308

Each line holds the depth, how many different prices the first half's tree gives the 600 validation listings, how far the two trees' prices for one listing sit apart on average, and each tree's MAE on the validation rows. Held to depth 2, a tree answers at most two questions per listing and has at most four leaves, so the first tree hands listings priced from 71,000 to 1,358,000 one of 4 prices, and both trees miss by more than 92,000 whichever half they learned from. That is bias, a form too coarse to follow the prices. A model short of the pattern this way underfits (misses the pattern on its own training rows about as badly as on new ones), the opposite failure to Lesson 3.5.1's overfitting, and that lesson's depth table scored the depth-2 tree 0.489 on its fit rows and 0.538 on the validation rows.

Fully grown, the first tree gives 262 different prices and each tree misses by less, 78,488 and 82,308, while the two disagree by 96,927 on an average listing, more than either one misses by. The rows each tree happened to get decided much of its price, which is variance. The depth-2 trees disagree too, by 45,900 on an average listing, so one pair of fits does not say how much of a model's miss is which, and the next subtitle measures both.

DON'T:

• DON'T read this bias as Lesson 2.3.1's b. The b is a number the fit chooses, and the bias here is a miss the model keeps whatever rows it is fitted on.
• DON'T take two fits that disagree for a model whose error is variance alone. The depth-2 trees disagreed by 45,900 and each still missed by more than 92,000.

Measuring Both by Refitting

What. The model is refitted on many bootstrap samples of the fit rows, Lesson 3.5.2's draws with replacement, and predicts the same 600 validation listings after each fit. A listing's predictions across the refits have a standard deviation, Lesson 2.2.5's, in currency, and its average over the 600 listings is the spread, this lesson's measure of variance. The mean of a listing's predictions is the averaged prediction, the model with its movement averaged away, and its MAE stands for bias. Why. One fit is one draw from the rows, and its miss holds both errors at once. Thirty refits lay the draws side by side, so how far they scatter and where their center lands are read apart. When. Before deciding whether a model needs room to bend or a way to move less, always on samples of the fit rows, so the validation rows stay out of every refit.

rng = np.random.default_rng(42)
# drawn once, so every model is refitted on the same 30 samples,
# each 1,800 positions in the fit rows drawn with replacement
draws = [rng.integers(0, 1800, size=1800) for _ in range(30)]


def refit(model):
    # fit refits every step of a pipeline, so one object serves all 30
    rows_of_preds = []
    for rows in draws:
        model.fit(X_fit.iloc[rows], y_fit.iloc[rows])
        rows_of_preds.append(model.predict(X_val))
    # one row of 600 validation predictions per sample
    return np.array(rows_of_preds)

The four models are Lesson 3.2.3's linear pipeline, the depth-2 tree, the fully grown tree and Lesson 3.5.2's chosen forest, half the columns per split and 100 trees. The forest's 30 refits grow 3,000 trees, so this block takes longer than the ones around it:

models = {
    "linear": make_pipeline(make_prep(fill()), LinearRegression()),
    "depth-2 tree": make_tree(max_depth=2),
    "fully grown tree": make_tree(),
    "forest": make_pipeline(
        make_prep("passthrough"),
        RandomForestRegressor(max_features=0.5, random_state=42),
    ),
}
prices = y_val.to_numpy()


def two_errors(preds):
    # axis=0 runs down the 30 refits, giving one value per listing
    spread = preds.std(axis=0).mean()
    avg_miss = np.abs(prices - preds.mean(axis=0)).mean()
    return round(spread), round(avg_miss)


refits = {name: refit(model) for name, model in models.items()}
for name, preds in refits.items():
    # the 600 prices against all 30 rows, Lesson 2.2.4's broadcasting
    one_miss = np.abs(prices - preds).mean()
    print(name, *two_errors(preds), round(one_miss))

linear 11232 56269 57059
depth-2 tree 46376 87448 96187
fully grown tree 77407 52824 78743
forest 21027 49826 53208

Each line holds the spread, the averaged prediction's MAE and one refit's MAE, averaged over the 30. The linear pipeline moves least, 11,232, and its averaged prediction misses by 56,269, close to one refit's 57,059: averaging its refits removes almost nothing, so nearly all of its miss is bias. The fully grown tree moves by 77,407, and its averaged prediction misses by 52,824, less than the linear pipeline's, while one refit misses by 78,743. Its form can follow the prices once its movement is averaged away, and a single tree carries that movement whole, so most of its miss is variance. The depth-2 tree carries both, the largest averaged miss, 87,448, and a spread four times the linear pipeline's, 46,376. The forest moves by 21,027, about a quarter of the fully grown tree's spread, and its averaged prediction misses least, 49,826: it keeps a fully grown tree's low bias and averages most of the variance away, Lesson 3.5.2's averaging, measured.

The averaged miss is not bias alone. A price also carries noise, Lesson 3.5.1's word for what a fully grown tree learns along with the pattern, and no model fitted to these columns predicts the noise in a validation listing's price, so it sits inside every model's averaged miss. The refits also stand in for fresh training sets: all 30 are drawn from the same 1,800 fit rows, each holding nearly two thirds of them, so their average still carries what those particular listings put into a price. The averaged misses are compared with each other, and none is read as a model's bias in currency. One listing shows what the spread means, the first validation listing, Lesson 3.5.1's 69.9 square meter Lakeview house:

print(prices[0])  # 346000
for name, preds in refits.items():
    first = preds[:, 0]
    print(name, round(first.min()), round(first.max()))

linear 301800 354601
depth-2 tree 222309 352200
fully grown tree 201000 523000
forest 318430 375860

The fully grown tree's 30 refits price a house of 346,000 anywhere from 201,000 to 523,000, the forest's from 318,430 to 375,860 and the linear pipeline's from 301,800 to 354,601. Variance is a fact about every price a model quotes, not only about its score.

Lesson 3.5.2 accounted for a forest's R² as its trees' average R² plus their spread, each listing's variance across the trees put on R²'s scale. The same accounting, with the 30 refits in place of the trees:

var_val = np.var(prices)
for name, preds in refits.items():
    avg_sq = ((prices - preds.mean(axis=0)) ** 2).mean()
    one_sq = ((prices - preds) ** 2).mean()
    spread_sq = np.var(preds, axis=0).mean()
    three = [1 - avg_sq / var_val, 1 - one_sq / var_val, spread_sq / var_val]
    print(name, np.round(three, 3))

linear [0.839 0.835 0.004]
depth-2 tree [0.619 0.539 0.08 ]
fully grown tree [0.84  0.585 0.255]
forest [0.861 0.841 0.02 ]

Each list holds the averaged prediction's R², one refit's average R² and the spread, Lesson 3.5.2's three numbers, and the first is the other two added, to rounding. On this scale the two errors are the two terms: what the averaged prediction leaves below 1 stands for the bias, noise included, and the spread for the variance. The fully grown tree gives up 0.16 to bias and 0.255 to variance, and the linear pipeline 0.161 and 0.004, the same split the currency numbers showed.

DON'T:

• DON'T read a model's variance off one fit. One fully grown tree's MAE says nothing about how far its prices move, and its 30 refits priced the first validation listing anywhere from 201,000 to 523,000.
• DON'T read an averaged miss as a model's bias in currency. It holds the noise in the validation prices, which no model fitted to these columns predicts, so it compares models with each other and states no model's bias on its own.

Capacity

What. Capacity (how far a model can bend to fit its training rows) is set by a model's settings: a tree's depth and leaf size, kNN's k, and for a least squares fit the number of its columns. More capacity lets a model follow its rows more closely, their pattern and their noise alike. Why. Capacity is the lever between the two errors. On the refits below, every step up in depth lowered the averaged miss and raised the spread, so a setting decides where a model sits between them, and only rows the fit never saw can say where it sits best. When. Whenever a setting is chosen, and before more rows or a bigger model are bought for a model whose error is mostly variance.

Lesson 3.5.1's depth table read capacity off two scores: the R² on the fit rows climbed at every depth, from 0.348 with one question to 1.0 with no limit, while the validation R² peaked at 0.731 at depth 5 and fell back to 0.614. Its kNN table ran the other way, the fit rows scoring 1.0 at k=1 and 0.639 at k=100, since a larger k averages more listings into every price. Lesson 3.3.2's noise columns raised a least squares fit's training R² as each block of them was added, from 0.696 to 0.854 on 200 rows. The refits split the depth table's two scores into the two errors:

for depth in [1, 2, 3, 5, 8, None]:
    print(depth, *two_errors(refit(make_tree(max_depth=depth))))

1 30296 108970
2 46376 87448
3 49520 73523
5 51152 59532
8 60629 54033
None 77407 52824

Each line holds the depth, the spread and the averaged prediction's MAE. From one question to no limit the averaged miss fell at every step, from 108,970 to 52,824, and the spread rose at every step, from 30,296 to 77,407. The shallow end is bias, a tree with too few questions to follow the prices whatever rows it draws, and the deep end is variance, a tree whose questions follow the rows it drew. The last step, from depth 8 to no limit, took 1,209 off the averaged miss and raised the spread from 60,629 to 77,407.

Less capacity is the first answer to variance, and the leaf size gives it from the other side. With one refit's MAE beside the two errors:

for leaf in [1, 5, 20, 100]:
    preds = refit(make_tree(min_samples_leaf=leaf))
    one_miss = np.abs(prices - preds).mean()
    print(leaf, *two_errors(preds), round(one_miss))

1 77407 52824 78743
5 61729 52010 70787
20 49843 56373 69124
100 41709 76915 84732

One listing per leaf is the fully grown tree. Five per leaf cut the spread from 77,407 to 61,729 while the averaged miss moved by 814, so one refit missed by 70,787 instead of 78,743, less variance for no bias at all. Twenty per leaf cut the spread to 49,843 for 3,549 more averaged miss than one per leaf, and one refit missed least, 69,124. A hundred per leaf cut the spread again, to 41,709, and raised the averaged miss to 76,915, so one refit missed by 84,732, more than the fully grown tree: the cut went past the variance and into bias. Lesson 3.5.1's single fit on the 1,800 fit rows put five per leaf ahead of twenty, 0.794 against 0.76. That was one draw of the rows, and over these 30, one refit's miss puts twenty ahead, 69,124 against 70,787, because refit fits on every repeat in a draw as a row of its own, and min_samples_leaf counts each one, so a refit's leaf of five can hold fewer than five different listings.

DON'T:

• DON'T add capacity to a model whose error is mostly variance. From depth 8 to no limit the averaged miss fell by 1,209 and the spread rose from 60,629 to 77,407.
• DON'T read a rising training score as a falling error. Lesson 3.5.1's fit rows scored higher at every depth, and on the refits the spread rose at every depth with it.
• DON'T cut capacity unmeasured. At a hundred listings per leaf one refit missed by 84,732, more than the fully grown tree's 78,743.

Learning Curves

What. A learning curve, Lesson 2.7.2's score plotted against how much data a model trained on, is drawn here as two lines per model: the model is fitted on the first 500, 600, 800, 1,000, 1,200, 1,500 and 1,800 fit rows, and each fit is scored on the slice it was fitted on and on the 600 validation rows. Why. The gap between the two lines and the direction of the validation line come from a handful of fits, and they say whether more rows would help before anyone pays for them. When. Before buying or labeling more listings, and beside every diagnosis made by refitting.

Both lines are scored by MAE, so the gap between them is an amount of money, and no R² measured against one set of rows' spread is set beside one measured against another's, Lesson 3.1.4's warning. The fit rows came out of train_test_split shuffled, so their first 500 are a random 500. curves.py builds the four models of refits.py and tries the smallest slices first:

import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.pipeline import make_pipeline
from price_models import (
    X_fit, X_val, y_fit, y_val, fill, make_prep, make_tree,
)

models = {
    "linear": make_pipeline(make_prep(fill()), LinearRegression()),
    "depth-2 tree": make_tree(max_depth=2),
    "fully grown tree": make_tree(),
    "forest": make_pipeline(
        make_prep("passthrough"),
        RandomForestRegressor(max_features=0.5, random_state=42),
    ),
}
linear = models["linear"]
for n in [100, 200]:
    linear.fit(X_fit.iloc[:n], y_fit.iloc[:n])
    try:
        linear.predict(X_val)
    except ValueError as err:
        print(n, err)
fit_ratings = list(X_fit["energy_rating"])
val_ratings = list(X_val["energy_rating"])
print(fit_ratings.index("G"))  # 474
print(val_ratings.count("G"))  # 4

100 Found unknown categories ['G'] in column 2 during transform
200 Found unknown categories ['G'] in column 2 during transform

Both fits ran, and both predictions stopped at the encoder with Lesson 3.2.2's refusal of a category the fit never saw, column 2 of the one-hot group being energy_rating. The first fit row rated G sits at position 474 and the validation rows hold 4 such listings, so no slice shorter than 475 rows can be scored on them, and the curves start at 500. With the encoder's default, a first slice must hold every category the validation rows hold, so the last of them to appear in the fit rows sets how small it can be. curve_of takes the rows as X and y, so the last subtitle can hand it the fit rows in another order:

sizes = [500, 600, 800, 1000, 1200, 1500, 1800]


def curve_of(model, X, y):
    # MAE on the slice it was fitted on, and on the validation rows
    on_slice, on_val = [], []
    for n in sizes:
        X_n, y_n = X.iloc[:n], y.iloc[:n]
        model.fit(X_n, y_n)
        on_slice.append(round(mean_absolute_error(y_n, model.predict(X_n))))
        on_val.append(round(mean_absolute_error(y_val, model.predict(X_val))))
    return on_slice, on_val


curves = {
    name: curve_of(model, X_fit, y_fit) for name, model in models.items()
}

Seven fits per model, and the four models on one figure, checked by printing what each line holds:

# one MAE scale for all four, and room for every label
fig, axes = plt.subplots(
    2, 2, figsize=(12, 8), sharey=True, layout="constrained"
)
for ax, name in zip(axes.ravel(), curves):
    on_slice, on_val = curves[name]
    ax.plot(sizes, on_slice, marker="o", label="rows fitted on")
    ax.plot(sizes, on_val, marker="o", label="validation rows")
    ax.set_title(name)
    ax.set_xlabel("fit rows used")
    ax.set_ylabel("MAE")
    ax.legend()
fig.savefig("learning_curves.png", dpi=150, bbox_inches="tight")
for ax in axes.ravel():
    print(ax.get_title(), [line.get_label() for line in ax.lines])
    for line in ax.lines:
        print(line.get_ydata())
plt.close(fig)

linear ['rows fitted on', 'validation rows']
[56933 57743 57328 57506 58455 57559 56332]
[59042 59848 59190 57654 57535 56814 55973]
depth-2 tree ['rows fitted on', 'validation rows']
[100017  99346  96992 101401 101753 103033 100535]
[98065 98885 96370 96160 96799 98202 96959]
fully grown tree ['rows fitted on', 'validation rows']
[0 0 0 0 0 0 0]
[96645 77453 80465 74532 76595 78440 76358]
forest ['rows fitted on', 'validation rows']
[23397 23042 22119 21481 21194 20825 20423]
[57946 56712 56440 53674 53470 50858 50340]

sharey=True gives the four panels one y axis, so a height means the same MAE in every panel, and layout="constrained" (matplotlib's own spacing of the panels) keeps each title clear of the axis label above it. Every value printed came back out of the figure, the check Lesson 2.7.2 ran on its charts, and the four panels hold four shapes. The linear pipeline's lines start 2,109 apart at 500 rows, and from 1,000 rows on they run within 920 of each other: they have met. The depth-2 tree's lines meet too, between 96,160 and 103,033, near 100,000. The fully grown tree's slice line is 0 at every size, every fit row priced exactly, while its validation line runs between 74,532 and 80,465 after one point at 96,645. The forest's lines stay apart, 29,917 at 1,800 rows, and its validation line falls at every slice, from 57,946 to 50,340.

DON'T:

• DON'T start a learning curve on a slice smaller than the first that holds every category the validation rows hold. At 100 and 200 fit rows no listing rated G had been fitted, and the validation rows stopped with Found unknown categories ['G'] in column 2 during transform.
• DON'T read the slice line as the model's error. The fully grown tree scored 0 on every slice it was fitted on and missed the validation listings by 76,358 at 1,800 rows.

Reading a Learning Curve

What. A curve is read twice. The gap between the lines at the largest slice says how much better the model prices its own rows than new ones, and a gap that stays open is one the refits then trace to variance or not. The validation line over the last slices says whether more rows of the same kind still lower the miss: still falling, they do, and flat, they no longer do. Lines that have met and flattened belong to a model more rows no longer change, and the step for it is more capacity or a better form. Why. Each reading names a different next step, more rows, more capacity, less capacity, averaging or a better form, and money spent on the wrong one goes where the miss does not come from. When. After the curves are drawn, and only once each reading has held on a second order of the rows.

One order of the fit rows is one draw of which listings come first. The same slices, taken from the other end:

for name, model in models.items():
    print(name, curve_of(model, X_fit.iloc[::-1], y_fit.iloc[::-1])[1])

linear [55721, 54639, 54606, 54911, 55130, 55805, 55973]
depth-2 tree [96873, 96410, 92782, 94677, 93362, 93241, 96959]
fully grown tree [78495, 77645, 82483, 72338, 75058, 76707, 76358]
forest [57133, 55863, 54139, 52995, 52020, 50431, 49799]

X_fit.iloc[::-1] reverses the fit rows, so the first 500 of this order share no listing with the first 500 of the other, and each line is the validation MAE at the seven slices. At 1,800 rows both orders hold the same listings, and the linear pipeline and the two trees land on the numbers the figure printed, while the forest lands on 49,799 instead of 50,340: its trees draw their bootstrap samples by position, Lesson 3.5.2's estimators_samples_, and the same listings in another order sit at other positions.

The linear pipeline's lines met, and its validation line fell by 3,069 in the first order and rose by 252 in this one, so the 1,300 rows added moved it by less than the choice of which 500 came first did, 59,042 against 55,721. Its line has flattened and its refits put nearly all its miss in bias, so more rows of the same kind are the wrong purchase. A better form is the step, and Lesson 3.3.1 took it: its log form, fitted on the same 1,800 rows, missed these validation listings by 44,213 against the linear pipeline's 55,973.

The depth-2 tree's lines met near 100,000, and its validation line stayed between 92,782 and 98,885 in both orders. More rows leave it there, and capacity moves it: the depth refits put depth 5's averaged miss at 59,532 against depth 2's 87,448.

The fully grown tree's slice line is 0, so its gap is the whole validation miss, 76,358 at 1,800 rows, a gap its refits trace to variance. Its 500-row point is 96,645 in one order and 78,495 in the other, so the fall of 19,192 to the next slice in the first order measures a tree grown on an unlucky 500 listings, not what 100 more rows are worth: from the other end, 100 more rows moved the line by 850. Less capacity and averaging both narrow the gap. Five listings per leaf, on the same slices:

leaf_5 = curve_of(make_tree(min_samples_leaf=5), X_fit, y_fit)
print(leaf_5[0])
print(leaf_5[1])

[47814, 46249, 45416, 44071, 43218, 40863, 40077]
[81594, 77380, 73215, 71371, 70489, 65553, 62441]

At 1,800 rows the leaf limit leaves a gap of 22,364 and misses the validation listings by 62,441, and the forest leaves a gap of 29,917 and misses by 50,340, both against the fully grown tree's 76,358. A narrower gap is not a better model: the forest's gap is the wider of the two and its miss the lower.

The forest's gap stayed open, 34,549 at 500 rows and 29,917 at 1,800, and its validation line fell in both orders, by 7,606 and 7,334 from 500 rows to 1,800 and by 518 and 632 over the last 300. More rows still lower its miss, more slowly over the last 300 than across the whole run, and of the four it is the one model these runs say rows are worth buying for.

A closed gap is not a model without variance. The depth-2 tree's lines met, and its refits spread by 46,376, more than twice the forest's 21,027: a curve shows how much better a model prices its own rows than new ones, and the refits show how far its prices move. On these four runs the lines that met and flattened belonged to the two models whose refits put most of their miss in bias, and the reading goes no further than that.

DON'T:

• DON'T expect more rows to cure bias. The depth-2 tree's validation miss stayed between 92,782 and 98,885 from 500 rows to 1,800 in both orders, while depth 5 cut the averaged miss from 87,448 to 59,532.
• DON'T read a learning curve from one point. The fully grown tree's 500-row point was 96,645 in one order and 78,495 in the other.
• DON'T read met lines as a model without variance. The depth-2 tree's lines met and its refits spread by 46,376.
• DON'T read a narrower gap as the better model. Five listings per leaf left a gap of 22,364 and a miss of 62,441, the forest a gap of 29,917 and a miss of 50,340.

Lab Exercises

Lab Exercise 1: Thirty Refits at Five Values of k

Work in knn_refits.py in ~/dev/bootcamp, importing from price_models.py. Draw the Theory's 30 bootstrap samples with the same generator and seed, and refit Lesson 3.5.1's scaled kNN pipeline, make_pipeline(make_prep(fill_scale()), KNeighborsRegressor(n_neighbors=k)), on every sample at k of 5, 10, 20, 50 and 100, predicting the validation rows after each fit.

For each k, print in one list the spread, the averaged prediction's MAE, one refit's MAE, and the lowest and highest price the refits give the first validation listing, all in whole units. Say in comments which way capacity runs with k, which value carries the most variance and which the most bias, and at which k one refit misses least.

Lab Exercise 2: Three Values of k, Read From Their Curves

Work in knn_curves.py. Draw learning curves for the scaled kNN at k of 1, 10 and 100 on the Theory's slices of the fit rows, one panel per k in a figure of one row of three panels sharing the y axis, saved as knn_curves.png, and print each panel's title and both lines' values from the figure.

Then build the validation line of k=100 again on the fit rows in reverse order. Say in comments, for each k, whether its lines met, whether its validation line is still falling, and what each says about buying more rows.

import numpy as np
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from price_models import X_fit, X_val, y_fit, y_val, fill_scale, make_prep

rng = np.random.default_rng(42)
draws = [rng.integers(0, 1800, size=1800) for _ in range(30)]
prices = y_val.to_numpy()

for k in [5, 10, 20, 50, 100]:
    knn = make_pipeline(
        make_prep(fill_scale()), KNeighborsRegressor(n_neighbors=k)
    )
    rows_of_preds = []
    for rows in draws:
        knn.fit(X_fit.iloc[rows], y_fit.iloc[rows])
        rows_of_preds.append(knn.predict(X_val))
    preds = np.array(rows_of_preds)
    first = preds[:, 0]
    errors = [
        preds.std(axis=0).mean(),
        np.abs(prices - preds.mean(axis=0)).mean(),
        np.abs(prices - preds).mean(),
        first.min(),
        first.max(),
    ]
    print(k, [round(e) for e in errors])
# A larger k averages more listings into every price, so capacity
# falls as k grows: the spread fell from 46,842 to 14,703 and the
# averaged miss rose from 66,604 to 80,546. k=5 carries the most
# variance of the five and k=100 the most bias, and one refit misses
# least at k=20, 73,515, with k=10 at 74,319.

5 [46842, 66604, 77148, 252600, 431400]
10 [37286, 68394, 74319, 293000, 386500]
20 [28092, 70213, 73515, 267000, 385950]
50 [19878, 75730, 77443, 261460, 321520]
100 [14703, 80546, 81534, 257040, 303570]

import matplotlib.pyplot as plt
from sklearn.metrics import mean_absolute_error
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from price_models import X_fit, X_val, y_fit, y_val, fill_scale, make_prep

sizes = [500, 600, 800, 1000, 1200, 1500, 1800]


def curve_of(model, X, y):
    on_slice, on_val = [], []
    for n in sizes:
        X_n, y_n = X.iloc[:n], y.iloc[:n]
        model.fit(X_n, y_n)
        on_slice.append(round(mean_absolute_error(y_n, model.predict(X_n))))
        on_val.append(round(mean_absolute_error(y_val, model.predict(X_val))))
    return on_slice, on_val


def knn(k):
    neighbors = KNeighborsRegressor(n_neighbors=k)
    return make_pipeline(make_prep(fill_scale()), neighbors)


fig, axes = plt.subplots(
    1, 3, figsize=(15, 4), sharey=True, layout="constrained"
)
for ax, k in zip(axes, [1, 10, 100]):
    on_slice, on_val = curve_of(knn(k), X_fit, y_fit)
    ax.plot(sizes, on_slice, marker="o", label="rows fitted on")
    ax.plot(sizes, on_val, marker="o", label="validation rows")
    ax.set_title(f"k={k}")
    ax.set_xlabel("fit rows used")
    ax.legend()
axes[0].set_ylabel("MAE")
fig.savefig("knn_curves.png", dpi=150, bbox_inches="tight")
for ax in axes:
    print(ax.get_title())
    for line in ax.lines:
        print(line.get_ydata())
plt.close(fig)
print(curve_of(knn(100), X_fit.iloc[::-1], y_fit.iloc[::-1])[1])
# k=1: the slice line is 0 and the gap is the whole validation miss,
# 88,975 at 1,800 rows. The validation line fell by 7,130, so rows
# help it a little, and less capacity helps more.
# k=10: a gap of 3,031 at 1,800 rows, and both lines still falling.
# k=100: the lines met at every slice, within 2,070, and both kept
# falling, the validation line by 14,177 here and 16,909 from the
# other end. Met is not flat: more rows still lower this model's miss.

k=1
[0 0 0 0 0 0 0]
[96105 95360 92733 87715 88200 89112 88975]
k=10
[73272 69519 67929 66429 66965 66632 64800]
[83094 80105 73895 71124 71364 69440 67831]
k=100
[96564 91851 87456 84515 83668 81380 79258]
[94494 92265 89079 85623 83701 81648 80317]
[97226, 94121, 90503, 87346, 84396, 82460, 80317]

Theory Assessment

Q1: Thirty Refits, Three Numbers

The fully grown tree's 30 refits spread by 77,407, their averaged prediction misses by 52,824 and one refit by 78,743, and the linear pipeline's give 11,232, 56,269 and 57,059. Explain what each triple says about the model's bias and variance, and why the fully grown tree's averaged prediction beats the linear pipeline's while one fully grown tree loses to it. Then say what the forest's 21,027 and 49,826 say about Lesson 3.5.2's averaging, and why no averaged miss is read as a model's bias in currency.

Hint: Ask what averaging 30 refits removes and what it cannot. For the last part, ask what part of a validation price no column records.

Q2: The Lever Between the Two

From depth 1 to no limit the spread rose from 30,296 to 77,407 and the averaged miss fell from 108,970 to 52,824. Explain both columns from what each extra question lets a tree do, and why one refit's miss fell from 78,743 to 69,124 between one and twenty listings per leaf and rose to 84,732 at a hundred. Then explain why Lesson 3.5.1's one fit put five listings per leaf first while the refits put twenty first.

Hint: One refit's miss holds both errors. For the last part, recall what one split of the rows is, and what a bootstrap sample repeats.

Q3: Four Panels and a Second Order

For each of the Theory's four panels, say whether its lines met, whether its validation line is still falling in both orders of the rows, and what step the panel points to. Then explain why the fully grown tree's fall from 96,645 at 500 rows to 77,453 at 600 is not read as what 100 more rows buy.

Hint: The gap and the slope are separate readings. Set the two orders' 500-row points side by side.

Q4: Four Claims Before the Budget Meeting

The team meets to spend next quarter's budget, and a teammate brings four claims drawn from this lesson's runs. Say which hold, and correct the ones that do not, with the lesson's numbers.

1. The depth-2 tree's two lines met, so its prices barely move when its rows change.
2. The fully grown tree's averaged prediction missed by 52,824, less than the linear pipeline's 56,269, so a fully grown tree is the better model to ship.
3. The linear pipeline's validation line has flattened, so 55,973 is the least any model can miss these listings by with these columns.
4. The forest's validation line is still falling, so buying enough rows will close its gap of 29,917.

Hint: Set each claim against the refits as well as the curves. For the last, compare how far each of the forest's two lines moved over the last 300 rows.

Assignment

Assignment: Buy 1,200 More Listings, or Not

A partner site offers 1,200 more listings like these, and the team will pay only for a model whose validation miss more rows would still lower. The Theory read each curve from two orders of the fit rows, and a purchase needs more than two. Five orders take two more tools. On a generator from np.random.default_rng(42), rng.permutation(1800) returns the whole numbers 0 to 1,799 in a random order, so X_fit.iloc[order] is the fit rows reshuffled. On an Axes, ax.fill_between(x, low, high, alpha=0.3, label=...) shades the area between two lines drawn over the same x values, alpha setting how much of what lies behind the shade shows through, and the shaded area lands in ax.collections rather than ax.lines.

Write buy_rows.py for the linear pipeline and the forest. Draw five orders from one generator and print where the first listing rated G sits in each, the category that held the Theory's first slice back. For each model, build the validation line on every order at the Theory's slices, the rows reshuffled first and then sliced from the front. Draw one panel per model, the two sharing the y axis, each holding a band from the lowest to the highest of its five lines at every slice and the mean of the five with markers, save it as buy_rows.png, and print how many collections and lines each panel holds. Print, per model, how far the mean fell from 500 rows to 1,800 and over the last 300 rows, and the band's width at every slice, in whole units. End with comments saying which model the purchase is for, what the band shows that the Theory's two orders could not, and why the linear pipeline's band closes at 1,800 rows while the forest's does not.

Hints:

• runs.min(axis=0), runs.mean(axis=0) and runs.max(axis=0) on a (5, 7) array of validation lines give one value per slice, Lesson 2.2.5's axis.
• A mean that falls by less than the band is wide says little on its own. For the last part, ask what every order holds at 1,800 rows, and what Lesson 3.5.2's estimators_samples_ returns.

import matplotlib.pyplot as plt
import numpy as np
from sklearn.ensemble import RandomForestRegressor
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error
from sklearn.pipeline import make_pipeline
from price_models import X_fit, X_val, y_fit, y_val, fill, make_prep

sizes = [500, 600, 800, 1000, 1200, 1500, 1800]
rng = np.random.default_rng(42)
# five orders of the positions 0 to 1,799
orders = [rng.permutation(1800) for _ in range(5)]
ratings = [list(X_fit["energy_rating"].iloc[order]) for order in orders]
print([rating.index("G") for rating in ratings])
models = {
    "linear": make_pipeline(make_prep(fill()), LinearRegression()),
    "forest": make_pipeline(
        make_prep("passthrough"),
        RandomForestRegressor(max_features=0.5, random_state=42),
    ),
}


def val_curve(model, order):
    X, y = X_fit.iloc[order], y_fit.iloc[order]
    on_val = []
    for n in sizes:
        model.fit(X.iloc[:n], y.iloc[:n])
        on_val.append(mean_absolute_error(y_val, model.predict(X_val)))
    return on_val

[273, 2, 30, 45, 16]

fig, axes = plt.subplots(
    1, 2, figsize=(12, 4), sharey=True, layout="constrained"
)
for ax, name in zip(axes, models):
    runs = np.array([val_curve(models[name], order) for order in orders])
    low, high = runs.min(axis=0), runs.max(axis=0)
    mean = runs.mean(axis=0)
    ax.fill_between(sizes, low, high, alpha=0.3, label="lowest to highest")
    ax.plot(sizes, mean, marker="o", label="mean of five orders")
    ax.set_title(name)
    ax.set_xlabel("fit rows used")
    ax.legend()
    fall, last = mean[0] - mean[-1], mean[-2] - mean[-1]
    print(name, round(fall), round(last), [round(w) for w in high - low])
axes[0].set_ylabel("validation MAE")
fig.savefig("buy_rows.png", dpi=150, bbox_inches="tight")
print([(len(ax.collections), len(ax.lines)) for ax in axes])
plt.close(fig)
# Buy for the forest. Its mean fell by 8,895 from 500 rows to 1,800,
# more than its band was wide at any slice, 5,173 at most, and still
# by 1,021 over the last 300. The linear pipeline's mean fell by 2,339
# in all and by 112 over the last 300, while its band was 4,731 wide
# at 500 rows: one order alone could show it falling or flat, as the
# Theory's two did. At 1,800 rows every order holds the same listings,
# least squares fits them the same in any order and its band closes,
# while the forest's trees draw by position, so its band stays 1,097
# wide.

linear 2339 112 [4731, 3018, 3390, 2121, 1272, 893, 0]
forest 8895 1021 [5173, 1614, 2748, 2885, 1109, 2058, 1097]
[(1, 1), (1, 1)]

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the two errors split by a formula, or scikit-learn's own tools for drawing these curves.

• Official docs - scikit-learn 1.9 example, "Single estimator versus bagging: bias-variance decomposition". The expected squared error of one decision tree and of bagged trees split into bias squared, variance and noise, measured over 50 randomly drawn training sets of a toy problem whose noise is known: bagging raises the bias slightly and cuts the variance.
• Official docs - scikit-learn 1.9 example, "Plotting Learning Curves and Checking Models' Scalability". LearningCurveDisplay drawing training and test scores against the training set's size over 50 shuffled splits, read for two classifiers of handwritten digits, then the time each model takes to fit and to score as the rows grow.

Summary

A validation score had been one number since Section 3.1, and this lesson took it apart. The same miss can come from a form that cannot follow the prices or from prices that follow the rows too closely, and the two looked identical until the model was refitted and its refits were laid side by side. A house priced anywhere from 201,000 to 523,000 by trees that differed only in the rows they drew is variance made concrete, and a tree that gave 600 listings four prices is bias made concrete.

Four settings the module had met turned out to be one lever. Depth, leaf size, k and the columns of a least squares fit each set how far a model bends to its rows, and every step of the depth refits traded one error for the other. The training score only ever reported how far the lever had been pushed, which is why it kept rising after the validation score had turned down.

The learning curves asked the same question with rows instead of refits, and a curve's reading counted only once a second order of the rows agreed with it. Even then a curve could not show everything, since the depth-2 tree's lines met while its prices still moved widely. A diagnosis is two instruments read together, each stated only as far as its runs go, and a decision that spends money waits for both.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

rng = np.random.default_rng(42)
draws = [rng.integers(0, 1800, size=1800) for _ in range(30)]
preds = refit(model)                         # (30, 600)
spread = preds.std(axis=0).mean()            # variance
averaged = preds.mean(axis=0)                # one price per listing
avg_miss = np.abs(prices - averaged).mean()  # bias, noise inside
one_miss = np.abs(prices - preds).mean()     # both errors

for n in [500, 600, 800, 1000, 1200, 1500, 1800]:
    X_n, y_n = X_fit.iloc[:n], y_fit.iloc[:n]
    model.fit(X_n, y_n)
    # the slice line, then the validation line
    mean_absolute_error(y_n, model.predict(X_n))
    mean_absolute_error(y_val, model.predict(X_val))

fig, axes = plt.subplots(2, 2, sharey=True, layout="constrained")
[line.get_ydata() for ax in axes.ravel() for line in ax.lines]
# the same slices from the other end of the fit rows
curve_of(model, X_fit.iloc[::-1], y_fit.iloc[::-1])
# lowest to highest over several orders of the rows
ax.fill_between(sizes, low, high, alpha=0.3)