Gradient Boosting, and Why It Is Not Averaging

Context

Gradient boosting builds a price model as a sum. Every listing starts at the mean of the training prices, and small trees are added one after another, each fitted to what the sum before it still gets wrong and added at a fraction of its size. XGBoost is a library built for this one kind of model, and it is the second library Module 3 installs.

A pricing team reaches for it once a linear pipeline and a forest have set the bar. It chooses on validation rows how far each tree moves the sum and how many trees to add, lets those rows end the fit when more trees stop helping, and ships the model at the end of the pipeline that encodes the neighborhoods.

Read as averaging, boosting fails without a sound. More trees are expected to settle it the way they settle a forest, and a boosted model whose validation R² peaked after 34 trees is run to 1,000, where it scores 0.785 while its training R² climbs to 0.997. Stopping on the test rows reports the best tree count on the rows meant to judge the model, and text columns stop XGBoost before its first tree. Lesson 3.5.2 averaged trees grown apart, and this lesson adds trees that each depend on every tree before them.

Goal

• Install XGBoost 3.4.1 with the command for your system, and confirm the version your scripts import.
• Build gradient boosting by hand from the training mean and depth-2 trees fitted to residuals, and show that GradientBoostingRegressor runs the same loop.
• Distinguish boosting from averaging by what each tree predicts and by what more trees do to a validation score.
• Choose a learning rate and a number of rounds on validation rows, with staged_predict and with XGBoost's early stopping, and score the test rows once.
• Build an XGBoost pipeline for a price and for sold_in_30_days, diagnose its refusal of a str column, and print its float32 output rounded.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the loop drawn before you write it.

• YouTube - "Gradient Boost Part 1 (of 4): Regression Main Ideas" by StatQuest with Josh Starmer. The average as the first prediction, a small tree fitted to the residuals, and a learning rate scaling each tree before it joins the sum, the loop this lesson builds by hand.
• Official docs - scikit-learn 1.9 User Guide, "Ensembles", the section "Shrinkage via learning rate". The update that adds each new tree at the learning rate, and the note that a smaller rate needs more trees to reach the same training error.

Table of Contents

• Installing XGBoost
• Boosting by Hand
• Learning Rate and Rounds
• Early Stopping on Validation Rows
• XGBoost in a Pipeline

Theory

Installing XGBoost

What. XGBoost is a library built for boosted trees, with estimators shaped like scikit-learn's, XGBRegressor for a price and XGBClassifier for a label. It is installed as xgboost-cpu on WSL and Linux and as xgboost on macOS, imported as xgboost either way, at 3.4.1, the release every result in this lesson was printed by. Why. scikit-learn's boosting comes first, because GradientBoostingRegressor computes the loop you write by hand, and XGBoost follows, with early stopping on the validation rows built into its fit. The install takes three forms, because xgboost-cpu is built for Linux and Windows only, and the xgboost package a Mac installs needs libomp installed before it. When. Once, now, with (bootcamp) active, in the form for your system, before the first block of Boosting by Hand.

Before the install, (bootcamp) has no xgboost to import:

(bootcamp) goku@machine:~/dev/bootcamp$ python3 -c "import xgboost"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'xgboost'

On WSL and Linux, in ~/dev/bootcamp with (bootcamp) active:

uv pip install xgboost-cpu==3.4.1

Among the lines uv prints, the ones that start with + name what it added:

 + xgboost-cpu==3.4.1

One line: XGBoost requires NumPy and SciPy, which (bootcamp) already holds, so the install added XGBoost alone. Only xgboost-cpu is pinned, so on your machine other + lines can appear, at versions of their own, where this run printed none. xgboost-cpu is XGBoost built for the processor alone, without the code that runs it on graphics cards.

On an Apple Silicon Mac, with the Homebrew of Module 0's setup page:

brew install libomp
uv pip install xgboost==3.4.1

On an Intel Mac, which Module 0's setup page gave MacPorts in place of Homebrew:

sudo port install libomp
sudo mkdir -p /usr/local/opt/libomp/lib
sudo ln -s /opt/local/lib/libomp/libomp.dylib /usr/local/opt/libomp/lib/libomp.dylib
uv pip install xgboost==3.4.1

XGBoost's package for an Intel Mac looks for libomp in one folder only, /usr/local/opt/libomp/lib, where Homebrew would have put it, and MacPorts installs it in /opt/local/lib/libomp/ instead. The two middle lines create that folder and put a symbolic link in it (a small file that stands for another file) pointing at MacPorts' libomp.dylib.

On any system, confirm the release from a script, check_xgboost.py:

import xgboost

print(xgboost.__version__)  # 3.4.1

DON'T:

• DON'T install the package xgboost on WSL or Linux. There it requires nvidia-nccl-cu13, NVIDIA's library for running work across graphics cards, which no lesson uses, and in this run uv pip install --dry-run xgboost==3.4.1, which lists what an install would add without installing it, listed + nvidia-nccl-cu13==2.32.3, a 305 MB download. xgboost does not pin that library, so its version, and its size, can differ on your machine.
• DON'T run the WSL line on a Mac. xgboost-cpu has no build for macOS, and the Mac lines install xgboost after libomp.
• DON'T import the package's name. import xgboost_cpu raises ModuleNotFoundError: No module named 'xgboost_cpu', and import xgboost works whichever package installed it.

Boosting by Hand

What. Boosting (building a model as a sum of small models fitted one after another, each to what the sum before it still gets wrong) starts every listing at the mean of the training prices, fits a small tree to the residuals of that start, adds a fraction of the tree's prediction to every listing's sum, and fits the next tree to the new residuals. Each fit and add is a round, and the fraction is the learning rate (the share of each new tree's prediction that joins the sum). Fitted to residuals under Lesson 3.1.3's squared error, this loop is gradient boosting, the "gradient" being a derivative, Module 4's subject. Why. A depth-2 tree has four leaves, so alone it can return four numbers. Fitted to residuals, each new tree spends its leaves on the part of each price the sum before it still misses, and the learning rate keeps any one tree from moving the sum far, so trees too coarse to price a listing well add up to a model that does. When. Whenever a boosted model is fitted, read or tuned. Its learning rate and its number of rounds are the two settings the loop adds to its trees' own, and every boosting class in this lesson takes both.

Every Python block from here on goes into one script, boosting.py, in ~/dev/bootcamp, each block added under the one before it. The split is Section 3.5's: Lesson 3.2.3's nine columns, a fifth of the rows held out for testing, and a quarter of the 2,400 training rows held out again as validation rows, the carve of Lesson 3.1.2's Assignment, which leaves 1,800 to fit. Settings are compared on the 600 validation rows, and the test rows score a model once, after its settings are fixed.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from sklearn.tree import DecisionTreeRegressor

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42
)
print(len(X_fit), len(X_val), len(X_test))  # 1800 600 600


def tree_prep(num_cols):
    return ColumnTransformer([
        ("num", "passthrough", num_cols),   # gaps left in
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])

tree_prep is Lesson 3.5.2's: Lesson 3.2.3's groups with the numeric columns passed through, gaps and all, because a scikit-learn tree takes a gap, Lesson 3.5.1's point. Its argument lets the classifier at the end of the lesson add price to the numbers. Five rounds of depth-2 trees at a learning rate of 0.5:

prep = tree_prep(numeric)
fit_table = prep.fit_transform(X_fit)  # learned on the 1,800 alone
val_table = prep.transform(X_val)
start = y_fit.mean()
fit_sum = np.full(len(y_fit), start)   # every listing starts here
val_sum = np.full(len(y_val), start)
trees = []
print(round(start), round(mean_absolute_error(y_fit, fit_sum)))

for stage in range(1, 6):
    residual = y_fit - fit_sum         # what the sum still misses
    tree = DecisionTreeRegressor(max_depth=2, random_state=42)
    tree.fit(fit_table, residual)
    trees.append(tree)
    fit_sum = fit_sum + 0.5 * tree.predict(fit_table)
    val_sum = val_sum + 0.5 * tree.predict(val_table)
    fit_mae = mean_absolute_error(y_fit, fit_sum)
    print(stage, round(fit_mae), round(r2_score(y_val, val_sum), 3))

381243 147724
1 112926 0.403
2 95067 0.574
3 83867 0.661
4 77010 0.725
5 71805 0.759

The 1,800 fit prices average 381,243, and that one price misses them by 147,724 on average. Five rounds of four-leaf trees, each fitted to residual and never to the price, cut the training miss to 71,805 and lift the validation R² to 0.759. What the first two trees return:

# each tree's four leaves, the only numbers it returns
for tree in trees[:2]:
    print(np.round(np.unique(tree.predict(fit_table))))

[-169243.  -40381.  102234.  317832.]
[-62292.   7252.  95031. 187850.]

None of the eight is a price. The first tree's leaves run from -169,243 to 317,832, amounts to take off the 381,243 or add to it, and the second's, fitted to what the first left, from -62,292 to 187,850. One validation listing through all five rounds:

first = val_table[:1]  # the first validation listing
steps = [0.5 * tree.predict(first)[0] for tree in trees]
print(y_val.iloc[0], np.round(steps))
print(round(start + sum(steps)), round(val_sum[0]))

346000 [-20190. -31146.   5157.  -2570. -11607.]
320887 320887

Listed at 346,000, the listing starts 35,243 too high, and five half-steps of its trees' corrections land it at 320,887, the number val_sum holds. The second step carries it past its price, because each leaf was learned on training listings, and a listing the fit never saw gets their correction. Every step corrects the sum before it, which is why averaging the trees means nothing:

# the five trees averaged, the way a forest averages
averaged = np.mean([t.predict(val_table) for t in trees], axis=0)
print(np.round(averaged[:3]))               # [-24142.  34280.  98758.]
print(round(r2_score(y_val, averaged), 3))  # -3.473

Averaged the way Lesson 3.5.2's forest averages its trees, the five give -24,142 for a listing priced at 346,000, and an R² of -3.473: the mean of five corrections is a correction. A forest's trees each predict a price from their own resampled rows, and their mean is a price. A boosted model's trees are fitted in order, each to what the trees before it left, so its prediction is the start plus the learning rate times the sum of every tree.

DON'T:

• DON'T read a boosted tree's leaves as prices. The first tree's run from -169,243 to 317,832, amounts that move a price only once they are added to the start of 381,243.
• DON'T average a boosted model's trees. The mean of the five came out at -24,142 for a listing priced at 346,000, and at an R² of -3.473.
• DON'T fit each round's tree to the prices. Every round then adds half a price to a sum that already holds one: after five rounds the validation listings average a prediction of 1,346,666 against an average price of 385,398, and the R² is -25.223.

Learning Rate and Rounds

What. The learning rate scales each tree before it joins the sum, and the number of rounds counts the trees. scikit-learn's GradientBoostingRegressor computes the hand loop, and HistGradientBoostingRegressor (scikit-learn's histogram-based boosting, which sorts each feature's values into at most 255 bins and grows its trees on the bins, and which takes a gap) runs the same rounds, both from sklearn.ensemble. The rate is learning_rate in both, and the rounds are n_estimators in the first and max_iter in the second. Why. The two settings trade against each other: each round adds the learning rate's share of its tree, so a smaller rate needs more rounds to move the sum as far. And unlike a forest's trees, every round keeps fitting what the training rows still miss, so past some round more rounds lower the validation score. When. For every boosted model, with both settings compared on the validation rows, before the test rows score the one you keep.

DecisionTreeRegressor took the tree table's gaps, and scikit-learn's class for the hand loop does not:

from sklearn.ensemble import GradientBoostingRegressor

try:
    GradientBoostingRegressor(random_state=42).fit(fit_table, y_fit)
except ValueError as err:
    lines = str(err).splitlines()
    print(lines[0])
    print(lines[1].split(". ")[0])

Input X contains NaN.
GradientBoostingRegressor does not accept missing values encoded as NaN natively

The message goes on to suggest HistGradientBoosting, which takes gaps, or an imputer in a pipeline. Behind Lesson 3.2.3's own make_prep, which takes the numeric columns where Lesson 3.5.1's took their step, imputer and indicators included, the class runs the hand loop on the filled table:

from sklearn.impute import SimpleImputer


def make_prep(num_cols, remainder="drop"):
    return ColumnTransformer([
        ("num", SimpleImputer(strategy="median", add_indicator=True), num_cols),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ], remainder=remainder)


filler = make_prep(numeric)         # 3.2.3's, imputer and all
fit_filled = filler.fit_transform(X_fit)
val_filled = filler.transform(X_val)
gbr = GradientBoostingRegressor(
    n_estimators=5, learning_rate=0.5, max_depth=2, random_state=42
)
gbr.fit(fit_filled, y_fit)
print(gbr.init_)                    # DummyRegressor()
first_tree = gbr.estimators_[0, 0]  # round 1's tree
print(np.round(np.unique(first_tree.predict(fit_filled))))
rebuilt = np.full(len(y_val), y_fit.mean())
for tree in gbr.estimators_[:, 0]:  # one tree per round
    rebuilt = rebuilt + 0.5 * tree.predict(val_filled)
# True
print(np.allclose(rebuilt, gbr.predict(val_filled)))

[-169243.  -40381.  102234.  317832.]

init_, the start, is Lesson 3.1.4's DummyRegressor, the training mean. estimators_ holds one tree per round, the first with the hand loop's four leaves, and the start plus half of every tree's prediction matches predict. staged_predict returns the predictions after the first round, then after the second, and so on, so one fit of 1,000 rounds scores every shorter run. At three learning rates, with depth-3 trees:

for rate in [0.1, 0.3, 1.0]:
    deep = GradientBoostingRegressor(
        n_estimators=1000, learning_rate=rate, max_depth=3, random_state=42
    )
    deep.fit(fit_filled, y_fit)
    fit_r2 = [r2_score(y_fit, p) for p in deep.staged_predict(fit_filled)]
    val_r2 = [r2_score(y_val, p) for p in deep.staged_predict(val_filled)]
    best = int(np.argmax(val_r2))
    at_best = round(val_r2[best], 3)
    print(rate, best + 1, at_best, round(val_r2[-1], 3), round(fit_r2[-1], 3))

0.1 128 0.867 0.831 0.974
0.3 34 0.863 0.785 0.997
1.0 8 0.777 0.652 1.0

Each line is a rate, its best round, the validation R² there, and the validation and training R² after round 1,000. The best round falls as the rate rises, 128, 34 and 8, because each round adds three times as much of its tree at 0.3 as at 0.1. Past it, every rate kept raising the training R², to 0.974, 0.997 and 1.0, while the validation R² fell, to 0.831, 0.785 and 0.652: the later rounds fit what is particular to the 1,800 fit listings, Lesson 3.5.1's overfitting.

HistGradientBoosting fits the tree pipeline as it is, gaps and all. Its defaults, a learning rate of 0.1 and 100 rounds, against five times the rounds, and against a rate of 1.0 for those 500 rounds:

from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.pipeline import Pipeline

for rate, rounds in [(0.1, 100), (0.1, 500), (1.0, 500)]:
    hgb = HistGradientBoostingRegressor(
        learning_rate=rate, max_iter=rounds, random_state=42
    )
    model = Pipeline([("prep", tree_prep(numeric)), ("model", hgb)])
    model.fit(X_fit, y_fit)
    fit_r2 = round(model.score(X_fit, y_fit), 3)
    val_r2 = round(model.score(X_val, y_val), 3)
    print(rate, rounds, fit_r2, val_r2, hgb.n_iter_)
# auto False
print(hgb.early_stopping, hgb.do_early_stopping_)

0.1 100 0.928 0.863 100
0.1 500 0.99 0.84 500
1.0 500 1.0 0.715 500

The defaults validate best of the three, 0.863, against 0.84 for five times the rounds and 0.715 for a rate of 1.0, which trains to 1.0. n_iter_ shows every round ran. HistGradientBoosting can stop early on its own, but early_stopping="auto", its default, turns that on only when fit receives more than 10,000 rows, so here do_early_stopping_ is False.

The test rows score the setting you keep, once. Lesson 3.2.3's linear pipeline, fitted on the same 1,800 rows, shows why a validation score is never set beside a test score:

from sklearn.linear_model import LinearRegression

linear = Pipeline([("prep", make_prep(numeric)), ("model", LinearRegression())])
linear.fit(X_fit, y_fit)                     # the 1,800 rows
print(round(linear.score(X_val, y_val), 3))  # 0.84
# 0.766
print(round(linear.score(X_test, y_test), 3))

One fitted pipeline scores 0.84 on the validation rows and 0.766 on the test rows. Two sets of 600 listings are two draws, Lesson 3.1.2's point, so validation scores are compared with validation scores. The defaults, refitted on all 2,400 training rows and scored once:

final = Pipeline([
    ("prep", tree_prep(numeric)),
    ("model", HistGradientBoostingRegressor(random_state=42)),
])
final.fit(X_train, y_train)                  # every training row
final_pred = final.predict(X_test)
final_mae = mean_absolute_error(y_test, final_pred)
final_r2 = r2_score(y_test, final_pred)
print(round(final_r2, 3), round(final_mae))  # 0.777 52773

On the same 600 test listings, Lesson 3.2.3's linear pipeline missed by 56,562 with an R² of 0.766, and HistGradientBoosting misses by 52,773 with 0.777. One split cannot say whether a gap that size would hold on other rows, and Lesson 3.7.1 measures it.

DON'T:

• DON'T expect more rounds to help the way more trees help a forest. At a learning rate of 0.3 the validation R² peaked at round 34, 0.863, and fell to 0.785 by round 1,000 while the training R² reached 0.997.
• DON'T pair a high learning rate with many rounds. HistGradientBoosting at 1.0 for 500 rounds trained to 1.0 and validated at 0.715, against 0.863 for its defaults.
• DON'T hand GradientBoostingRegressor a table with gaps. It refuses with Input X contains NaN., and HistGradientBoosting, or an imputer in front of it, answers that.

Early Stopping on Validation Rows

What. Early stopping (ending a fit once the score on rows held out from it has not improved for a set number of rounds, and keeping the round that scored best) is built into XGBoost. XGBRegressor, its estimator for a price, takes early_stopping_rounds, the number of rounds to wait for a new best, and its fit takes eval_set, a list of the rows to score after every round. Why. The staged scores above needed 1,000 rounds fitted to find round 34. Early stopping finds the best round during the fit and stops soon after, and the rows it watches choose the number of rounds, so they must be validation rows: a setting chosen on the test rows is chosen by the test score, Lesson 3.1.2's second break. When. Whenever a booster's number of rounds is not fixed in advance: the validation rows as eval_set, a ceiling of rounds it should never reach, and the test rows scored once afterward.

fit_table and val_table are Boosting by Hand's tables, the preparation fitted on the 1,800 fit rows alone, because XGBoost scores the eval_set rows exactly as it is handed them:

from xgboost import XGBRegressor

test_table = prep.transform(X_test)
booster = XGBRegressor(
    n_estimators=1000,
    learning_rate=0.1,
    early_stopping_rounds=50,
    random_state=42,
)
booster.fit(
    fit_table, y_fit, eval_set=[(val_table, y_val)], verbose=False
)
kept = booster.get_booster().num_boosted_rounds()
print(booster.best_iteration, kept)  # 34 85
print(round(booster.best_score))     # 72540

best_iteration counts rounds from 0, so 34 is the 35th round, the one whose validation RMSE, Lesson 3.1.4's root mean squared error and XGBoost's measure for a price unless told otherwise, was lowest, 72,540. Fifty rounds followed without a new best, the patience early_stopping_rounds set, so the fit stopped at 85. get_booster() returns the fitted XGBoost model inside the estimator, and verbose=False keeps fit from printing every round's score. Which rounds predict uses:

val_pred = booster.predict(val_table)
best_35 = booster.predict(val_table, iteration_range=(0, 35))
every_85 = booster.predict(val_table, iteration_range=(0, 85))
print(np.array_equal(val_pred, best_35))    # True
print(round(r2_score(y_val, val_pred), 3))  # 0.86
# 0.852
print(round(r2_score(y_val, every_85), 3))

iteration_range=(0, 35) asks for the first 35 rounds, and plain predict returned exactly that: the estimator predicts with its best round and the ones before it, not with all 85, which score 0.852. The test rows, once:

test_pred = booster.predict(test_table)    # the test rows, once
test_mae = mean_absolute_error(y_test, test_pred)
test_r2 = r2_score(y_test, test_pred)
print(round(test_r2, 3), round(test_mae))  # 0.76 54986

The booster learned from the 1,800 fit rows, while HistGradientBoosting's 0.777 came from a refit on all 2,400, so the two test scores come from fits on different rows and rank nothing. The broken order hands early stopping the test rows:

# the broken order: stopping on the test rows
peeking = XGBRegressor(
    n_estimators=1000,
    learning_rate=0.1,
    early_stopping_rounds=50,
    random_state=42,
)
peeking.fit(
    fit_table, y_fit, eval_set=[(test_table, y_test)], verbose=False
)
peek_r2 = r2_score(y_test, peeking.predict(test_table))
print(peeking.best_iteration)  # 52
print(round(peek_r2, 3))       # 0.764

Watching the test rows, early stopping kept round 53, the round with the lowest RMSE on them, so 0.764 is the best of every round it tried, measured on the rows that report it. It sits above the honest 0.76 because it was chosen by the test score, Lesson 3.1.2's second break with the number of rounds as the setting, and nothing in the number says so.

DON'T:

• DON'T hand eval_set the test rows. Early stopping on them kept round 53 and reported 0.764, the best of every round on the rows meant to judge the model.
• DON'T read best_iteration as the number of rounds. It counts from 0, so 34 means 35 rounds, and the 85 the booster holds include the 50 that found nothing better.

XGBoost in a Pipeline

What. XGBRegressor and XGBClassifier refuse a str column, so each goes at the end of a Pipeline, after the tree preparation, the shape Lesson 3.2.3 gave every model fitted on the table as it arrives. Why. XGBoost checks a DataFrame's column types before its first tree and refuses text, and in a pipeline the preparation writes the same numeric columns for any rows, a listing posted this morning included. When. Every time XGBoost meets the listings table as it arrives, for the price or for sold_in_30_days.

try:
    XGBRegressor(random_state=42).fit(X_fit, y_fit)
except ValueError as err:
    print(err)

DataFrame.dtypes for data must be int, float, bool or category. When categorical type is supplied, the experimental DMatrix parameter`enable_categorical` must be set to `True`.  Invalid columns:neighborhood: str, property_type: str, energy_rating: str, condition: str

XGBoost names every column it cannot take, the four text columns, str in pandas 3.0.5 as Lesson 3.2.2 printed. The message's other route, pandas' category type, is XGBoost's own way of reading categories, and the four columns converted to it fit without an error, but this module hands every model the encoders' numbers, Lesson 3.2.2's way, every feature a model receives stored as a number:

pipe = Pipeline([
    ("prep", tree_prep(numeric)),
    ("model", XGBRegressor(random_state=42)),
])
pipe.fit(X_fit, y_fit)
print(round(pipe.score(X_fit, y_fit), 3))  # 0.995
print(round(pipe.score(X_val, y_val), 3))  # 0.805

spelled = XGBRegressor(
    n_estimators=100, learning_rate=0.3, max_depth=6, random_state=42
)
same = Pipeline([("prep", tree_prep(numeric)), ("model", spelled)])
same.fit(X_fit, y_fit)
# True
print(np.array_equal(same.predict(X_val), pipe.predict(X_val)))

Spelled out, 100 rounds at a learning rate of 0.3 with trees six levels deep predict exactly as XGBoost's defaults do, three times HistGradientBoosting's default rate, and the defaults train to 0.995 and validate at 0.805, below the early-stopped booster's 0.86 on the same rows. What the pipeline returns:

pipe_pred = pipe.predict(X_val)
print(pipe_pred.dtype)  # float32
print(pipe_pred[:3])
print(list(pipe_pred[:3]))
print(np.round(pipe_pred[:3]))

[306433.34 388788.7  766176.  ]
[np.float32(306433.34), np.float32(388788.7), np.float32(766176.0)]
[306433. 388789. 766176.]

XGBoost predicts in Lesson 2.5.2's float32, where scikit-learn's regressors here return float64, and a list of float32 values prints each one with its type. np.round prints the whole numbers a price is reported in. The same shape classifies: Section 3.4's stratified split and logistic pipeline, rebuilt here, against XGBoost at its defaults behind the tree preparation, with price among the numbers as in Section 3.4:

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.preprocessing import StandardScaler
from xgboost import XGBClassifier

sold = listings["sold_in_30_days"]
numeric_c = numeric + ["price"]             # Section 3.4's numeric group
Xc = listings[numeric_c + nominal + ["condition"]]
Xc_train, Xc_test, yc_train, yc_test = train_test_split(
    Xc, sold, test_size=0.2, random_state=42, stratify=sold
)
scaled_num = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", StandardScaler()),
])
logistic = Pipeline([
    ("prep", ColumnTransformer([
        ("num", scaled_num, numeric_c),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])),
    ("model", LogisticRegression()),
])
boosted = Pipeline([
    ("prep", tree_prep(numeric_c)),
    ("model", XGBClassifier(random_state=42)),
])
for name, clf in [("logistic", logistic), ("xgboost", boosted)]:
    clf.fit(Xc_train, yc_train)
    proba = clf.predict_proba(Xc_test)[:, 1]
    auc = round(roc_auc_score(yc_test, proba), 3)
    ap = round(average_precision_score(yc_test, proba), 3)
    print(name, proba.dtype, auc, ap)
print(logistic.classes_, boosted.classes_)  # [False  True] [0 1]
print(np.round(proba[:3], 3))               # [0.105 0.264 0.018]
train_proba = boosted.predict_proba(Xc_train)[:, 1]
train_auc = roc_auc_score(yc_train, train_proba)
print(round(train_auc, 3))                  # 1.0

logistic float64 0.642 0.306
xgboost float32 0.606 0.287

The logistic pipeline scores the 0.642 ROC AUC and 0.306 average precision Section 3.4 read on these test rows, and XGBoost at its defaults scores 0.606 and 0.287. XGBoost labels the classes 0 and 1, where the logistic pipeline keeps False and True, so column 1 of predict_proba is still the probability of a sale, in float32 again. On its own training rows its ROC AUC is 1.0, the 2,400 listings it was fitted on learned by heart.

DON'T:

• DON'T hand XGBoost the raw table. It refuses every str column by name, and the tree preparation in front of it in a pipeline turns them into numbers.
• DON'T give an XGBoost estimator early_stopping_rounds and then fit it in a pipeline with no eval_set. The fit stops with Must have at least 1 validation dataset for early stopping.
• DON'T print XGBoost's predictions raw. They are float32, a list of them prints np.float32(306433.34), and np.round gives the whole numbers a price is reported in.
• DON'T expect boosting to win every question. On sold_in_30_days, XGBoost at its defaults scored 0.606 and 0.287 against the logistic pipeline's 0.642 and 0.306, after reaching a training ROC AUC of 1.0.

Lab Exercises

Lab Exercise 1: Two Learning Rates by Hand

With (bootcamp) active, confirm with a one-line check that XGBoost imports at 3.4.1.

Then, in two_rates.py, build the Theory's split, validation carve and tree preparation, and write a function boost_by_hand(rate, rounds) that runs the hand loop with depth-2 trees and prints the rate, the round, the training MAE and the validation R² after rounds 1, 5, 10, 20 and 40. Run it for 40 rounds at learning rates 0.1 and 1.0, and say in a comment what the higher rate bought and what it cost.

Lab Exercise 2: XGBoost for sold_in_30_days, Stopped on Validation Rows

In sold_boost.py, split the listings for sold_in_30_days as Section 3.4 does, with price among the numeric columns, and carve a validation set from the training rows the way Lesson 3.1.2's Assignment does, stratified on the label like the split. Print the three sizes.

Hand XGBClassifier the raw fit rows inside a try, printing only the part of its message after Invalid columns:. Then fit the tree preparation on the fit rows, and fit XGBClassifier with at most 1,000 rounds, a learning rate of 0.1 and a patience of 50, watching the validation rows. Print the best round counted from 1, the rounds it kept, its classes_, the probabilities' dtype and the first three test probabilities to three decimals. Score the test rows once, ROC AUC and average precision, and say in a comment what the comparison with the logistic pipeline's 0.642 and 0.306 can and cannot show.

# 3.4.1
python3 -c "import xgboost; print(xgboost.__version__)"

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from sklearn.tree import DecisionTreeRegressor

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42
)
prep = ColumnTransformer([
    ("num", "passthrough", numeric),  # gaps left in
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
fit_table = prep.fit_transform(X_fit)
val_table = prep.transform(X_val)


def boost_by_hand(rate, rounds):
    fit_sum = np.full(len(y_fit), y_fit.mean())
    val_sum = np.full(len(y_val), y_fit.mean())
    for r in range(1, rounds + 1):
        tree = DecisionTreeRegressor(max_depth=2, random_state=42)
        # this round's residuals, never the prices
        tree.fit(fit_table, y_fit - fit_sum)
        fit_sum = fit_sum + rate * tree.predict(fit_table)
        val_sum = val_sum + rate * tree.predict(val_table)
        if r in [1, 5, 10, 20, 40]:
            fit_mae = round(mean_absolute_error(y_fit, fit_sum))
            print(rate, r, fit_mae, round(r2_score(y_val, val_sum), 3))


for rate in [0.1, 1.0]:
    boost_by_hand(rate, 40)

# 1.0 bought speed, a training miss of 45,039 against 61,697 after 40
# rounds, and cost the validation rows: it reached 0.776 by round 5 and
# only 0.796 by round 40, while 0.1 kept climbing, to 0.808.

0.1 1 139537 0.102
0.1 5 115654 0.38
0.1 10 98017 0.546
0.1 20 78438 0.708
0.1 40 61697 0.808
1.0 1 100535 0.538
1.0 5 66487 0.776
1.0 10 57412 0.775
1.0 20 50865 0.796
1.0 40 45039 0.796

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.metrics import average_precision_score, roc_auc_score
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from xgboost import XGBClassifier

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
numeric_c = numeric + ["price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
Xc = listings[numeric_c + nominal + ["condition"]]
sold = listings["sold_in_30_days"]
Xc_train, Xc_test, yc_train, yc_test = train_test_split(
    Xc, sold, test_size=0.2, random_state=42, stratify=sold
)
Xc_fit, Xc_val, yc_fit, yc_val = train_test_split(
    Xc_train, yc_train, test_size=0.25, random_state=42, stratify=yc_train
)
# 1800 600 600
print(len(Xc_fit), len(Xc_val), len(Xc_test))

try:
    XGBClassifier(random_state=42).fit(Xc_fit, yc_fit)
except ValueError as err:
    print(str(err).split("Invalid columns:")[1])

prep = ColumnTransformer([
    ("num", "passthrough", numeric_c),
    ("cat", OneHotEncoder(sparse_output=False), nominal),
    ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
])
fit_table = prep.fit_transform(Xc_fit)       # the fit rows alone
val_table = prep.transform(Xc_val)
test_table = prep.transform(Xc_test)
clf = XGBClassifier(
    n_estimators=1000,
    learning_rate=0.1,
    early_stopping_rounds=50,
    random_state=42,
)
clf.fit(
    fit_table, yc_fit, eval_set=[(val_table, yc_val)], verbose=False
)
kept = clf.get_booster().num_boosted_rounds()
print(clf.best_iteration + 1, kept, clf.classes_)
proba = clf.predict_proba(test_table)[:, 1]  # the test rows, once
print(proba.dtype, np.round(proba[:3], 3))
auc = roc_auc_score(yc_test, proba)
ap = average_precision_score(yc_test, proba)
print(round(auc, 3), round(ap, 3))           # 0.612 0.299

# Below the logistic pipeline on both, 0.612 against 0.642 and 0.299
# against 0.306. One split shows this booster lost here, never that
# boosting always would.

neighborhood: str, property_type: str, energy_rating: str, condition: str
23 73 [0 1]
float32 [0.132 0.205 0.137]

Theory Assessment

Q1: Corrections, Not Prices

The first validation listing, listed at 346,000, started at 381,243 and was moved by -20,190, -31,146, 5,157, -2,570 and -11,607. Explain what the start is, what each of the five numbers is and why four of them are negative, why the second carried the listing past its price, and why the five trees averaged scored an R² of -3.473 when averaging is exactly how a forest predicts.

Hint: Ask what each tree was fitted to and what the 0.5 does to it. Then ask what a forest's trees are each fitted to, and what the mean of five corrections is.

Q2: Three Rates, One Thousand Rounds

At 1,000 rounds of depth-3 trees, GradientBoostingRegressor peaked on the validation rows at round 128 at a learning rate of 0.1, 34 at 0.3 and 8 at 1.0, and after round 1,000 every rate's validation R² sat below its best while its training R² sat between 0.974 and 1.0. Explain why the best round falls as the rate rises, why the validation R² falls past it while the training R² keeps climbing, and why HistGradientBoosting at its defaults, and not at 500 rounds, went to the test rows.

Hint: Ask how much of each tree joins the sum at each rate. Then ask what each later round is fitted to, and on which rows the three HistGradientBoosting settings were compared.

Q3: Round 34, 85 Rounds, Two Test Scores

XGBoost with early_stopping_rounds=50 watched the validation rows, reported a best_iteration of 34, held 85 rounds, and scored 0.86 on the validation rows and 0.76 on the test rows. Watching the test rows instead, it kept round 53 and scored 0.764. Explain what 34 and 85 count, which rounds predict uses and how the lesson showed it, why the validation rows are the eval_set, and why 0.764 is not the better result.

Hint: Ask where the counting starts and how the patience adds rounds. Then ask what early stopping does with the rows it watches, and what that makes of a score measured on those same rows.

Q4: Every Round It Kept

A teammate adds these lines under the Early Stopping blocks of boosting.py:

kept = booster.get_booster().num_boosted_rounds()
print(kept, round(booster.score(val_table, y_val), 3))
for rounds in [booster.best_iteration + 1, kept]:
    refit = Pipeline([
        ("prep", tree_prep(numeric)),
        ("model", XGBRegressor(
            n_estimators=rounds, learning_rate=0.1, random_state=42
        )),
    ])
    refit.fit(X_train, y_train)
    print(rounds, round(refit.score(X_test, y_test), 3))

85 0.86
35 0.769
85 0.778

The teammate concludes that early stopping chose 85 rounds, that 0.86 on the validation rows against 0.778 on the test rows shows the booster overfit its validation rows, and that since 85 rounds beat 35 on the test rows, every later refit should take every round early stopping kept. Explain what each printed number is and where each conclusion goes wrong.

Hint: Ask which rounds booster.score predicted with, and which rows and which fit each score came from. Then ask what deciding between 35 and 85 by the last two lines would make of the test rows.

Assignment

Assignment: Rounds Chosen by the Number the Team Reports

The pricing team reports Lesson 3.1.4's MAE, the average miss in currency, and wants an XGBoost pipeline whose rounds were chosen by that number, where early stopping watches RMSE unless told otherwise. Two pieces of XGBoost this lesson did not use do it. eval_metric="mae", passed to XGBRegressor, makes it score the eval_set rows by MAE after every round and stop on that score, while its trees are still fitted to squared error. booster.evals_result(), after a fit with an eval_set, returns every score the fit recorded as a dict, and ["validation_0"]["mae"] is the list of validation MAEs, one per round it ran.

Write team_rounds.py on the Theory's tables. Fit three boosters with early stopping on MAE, at learning rates 0.3, 0.1 and 0.03, with at most 3,000 rounds and a patience of 50, and print for each the rate, the best round counted from 1, the rounds it ran and the best round's validation MAE from evals_result(). Show for each that the metric changed no tree, by predicting the validation rows with a booster of the same rate and rounds, no eval_metric and no early stopping. Draw the three validation MAE curves on one fig, ax with a dashed vertical line at each best round, save team_rounds.png, and print how many lines the Axes holds and how many points each one has. Then refit a pipeline of the tree preparation and a booster at the rate with the lowest validation MAE and its best number of rounds, on all 2,400 training rows, and score the test rows once, MAE and R². End with a comment saying which number the team publishes and how the rounds MAE chose at 0.1 compare with the 35 RMSE chose.

Hints:

• booster.best_iteration indexes the list evals_result() returns, so curve[booster.best_iteration] is the best round's validation MAE, and len(curve) counts the rounds the fit ran.
• Collect (validation MAE, rate, rounds) tuples while you fit, and min of the list picks the rate, Lesson 3.1.2's Assignment's tuple trick. Plot each curve against range(1, len(curve) + 1), so the x axis counts rounds from 1 as best_iteration + 1 does.

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder
from xgboost import XGBRegressor

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms", "bathrooms"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["price"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42
)


def tree_prep(num_cols):
    return ColumnTransformer([
        ("num", "passthrough", num_cols),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])


prep = tree_prep(numeric)
fit_table = prep.fit_transform(X_fit)
val_table = prep.transform(X_val)

candidates = []
fig, ax = plt.subplots(figsize=(8, 5))
for rate in [0.3, 0.1, 0.03]:
    booster = XGBRegressor(
        n_estimators=3000,
        learning_rate=rate,
        early_stopping_rounds=50,
        eval_metric="mae",
        random_state=42,
    )
    booster.fit(
        fit_table, y_fit, eval_set=[(val_table, y_val)], verbose=False
    )
    curve = booster.evals_result()["validation_0"]["mae"]
    rounds = booster.best_iteration + 1
    best_mae = curve[booster.best_iteration]
    candidates.append((best_mae, rate, rounds))
    print(rate, rounds, len(curve), round(best_mae))
    ax.plot(range(1, len(curve) + 1), curve, label=f"learning rate {rate}")
    ax.axvline(rounds, linestyle="--", color="black")

    # the same rate and rounds, no eval_metric, no stopping
    plain = XGBRegressor(
        n_estimators=rounds, learning_rate=rate, random_state=42
    )
    plain.fit(fit_table, y_fit)
    same = plain.predict(val_table)
    print(np.array_equal(same, booster.predict(val_table)))

ax.set_xlabel("round")
ax.set_ylabel("validation MAE")
ax.legend()
fig.savefig("team_rounds.png", dpi=150, bbox_inches="tight")
print(len(ax.lines), [len(line.get_xdata()) for line in ax.lines])
plt.close(fig)

0.3 15 65 53215
True
0.1 42 92 49689
True
0.03 160 210 49785
True
6 [65, 2, 92, 2, 210, 2]

# tuples compare item by item, so the smallest MAE wins
best_mae, rate, rounds = min(candidates)
final = Pipeline([
    ("prep", tree_prep(numeric)),
    ("model", XGBRegressor(
        n_estimators=rounds, learning_rate=rate, random_state=42
    )),
])
final.fit(X_train, y_train)  # all 2,400 training rows
test_pred = final.predict(X_test)
test_mae = mean_absolute_error(y_test, test_pred)
test_r2 = r2_score(y_test, test_pred)
print(rate, rounds, round(test_mae), round(test_r2, 3))

# Publish the test MAE, 52,174: the rate and the rounds were chosen
# on the validation rows, and the test rows reached no step before it.
# At 0.1, MAE kept 42 rounds where RMSE kept 35.

0.1 42 52174 0.773

Additional Resources

Optional, and nothing later depends on either. Both open the trees XGBoost grows.

• YouTube - "XGBoost Part 1 (of 4): Regression" by StatQuest with Josh Starmer. How XGBoost grows each tree, scoring every split by the gain in similarity score it brings and shrinking every leaf with a penalty, where the hand loop's trees take the plain mean of their residuals. Two defaults it names are not XGBoost 3.4.1's: a regressor there starts from the mean of the training target rather than 0.5, and lambda, the weight of that penalty, is 1 rather than 0.
• Official docs - XGBoost documentation, "Introduction to Boosted Trees", the section "Tree Boosting". XGBoost's objective, a training loss plus a charge for every leaf and for the size of each leaf's score, and how each round's tree is chosen to lower it one split at a time.

Summary

Boosting turned out to be a forest's opposite in the one place that matters. A forest asks many trees the same question on different rows and trusts their average. Boosting asks each tree a different question, what the sum so far still misses, and trusts the sum. That one change is why a boosted tree's leaves are corrections rather than prices, why averaging its trees produced nonsense, and why a library's boosting class could be rebuilt from a mean and a list of trees.

It is also why the settings behave as they do. More trees steady a forest, because each new tree is one more answer to the whole price, and more rounds eventually wear a boosted model down, because late in the fit what the training rows still miss is what is particular to them. The best number of rounds is a fact about the rows a model is judged on, which is why stopping on the test rows turns the test score into one more setting.

XGBoost changed the tools and none of the rules. Its defaults, its float32 output, its refusal of text and its early stopping inside fit were each handled with what the module already had: a pipeline, validation rows, and test rows scored once. On the price, boosting at the settings the validation rows chose scored above the linear pipeline on those rows, and on sold_in_30_days XGBoost at its defaults did not beat the logistic pipeline, so the question still chooses the model, never the other way round.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

# the hand loop GradientBoostingRegressor runs
fit_sum = np.full(len(y_fit), y_fit.mean())
for _ in range(5):
    tree = DecisionTreeRegressor(max_depth=2, random_state=42)
    tree.fit(fit_table, y_fit - fit_sum)  # residuals, not prices
    fit_sum = fit_sum + 0.5 * tree.predict(fit_table)

hgb = HistGradientBoostingRegressor(random_state=42)
model = Pipeline([("prep", tree_prep(numeric)), ("model", hgb)])
model.fit(X_fit, y_fit)                   # takes gaps as they are

booster = XGBRegressor(n_estimators=1000, learning_rate=0.1,
                       early_stopping_rounds=50, random_state=42)
booster.fit(fit_table, y_fit, eval_set=[(val_table, y_val)],
            verbose=False)                # validation rows, never test
booster.best_iteration + 1                # rounds chosen
np.round(booster.predict(test_table))     # float32, rounded