Hyperparameter Search

Context

A hyperparameter search scores one pipeline at many settings by cross-validation on the training rows, keeps the setting whose folds scored best, refits it on every training row and leaves the test rows to score it once. A grid search tries every combination of the values you list, and a random search draws each candidate's settings from ranges you describe.

A pricing team runs one before any model ships and again each time the listings are refreshed: the depth and leaf size of a tree, the learning rate and rounds of a boosted model, the number of columns a selector keeps, and, for the alert on listings likely to sell within 30 days, the cost on the weights and the threshold that turns a probability into an alert.

Without one, settings are picked on a single validation set of 600 listings, the draw on which Lesson 3.5.1 chose a leaf size by 0.028. Run carelessly, a search fails as quietly: the winner's cross-validated 0.734 gets reported where the test rows give 0.682, a setting named without its pipeline step stops the run with Invalid parameter 'max_depth', and a threshold wrapper's score of 0.533 is read as the cost it was tuned for. Lesson 3.6.2 replaced one validation set with five folds, and this lesson chooses settings on them.

Goal

• Choose a pipeline's settings with GridSearchCV on folds of the training rows, naming each through its step as step__setting, and score the refitted winner once on the test rows.
• Read cv_results_ as a DataFrame and identify which gaps between candidates the folds' spread can carry.
• Draw continuous and whole-number settings from SciPy's loguniform and randint with RandomizedSearchCV, and explain what a random search spends its fits on.
• Choose a classifier's C by average precision and its threshold for a cost with TunedThresholdClassifierCV and make_scorer, and distinguish the scorer each score call uses.
• Measure over repeated splits how far best_score_ overstates its winner, and report the test score instead.

Prerequisites

Neither is required, and the lesson stands without them. They are here if you want the case for random search, or scikit-learn's own outline of a search, before the code.

• YouTube - "Tuning Process (C2W3L01)" by DeepLearningAI, Andrew Ng. Which settings of a neural network tend to matter most, why points drawn at random beat a grid when one setting matters far more than another, and a coarse-to-fine search that zooms in on the region where the best points landed, in about seven minutes.
• Official docs - scikit-learn 1.9 User Guide, "Tuning the hyper-parameters of an estimator". The five parts every search is made of, an estimator, a space of settings, a way of picking candidates from it, a cross-validation scheme and a score, then, under "Exhaustive Grid Search", a grid written as lists of values for each setting.

Table of Contents

• Settings Chosen on Folds
• GridSearchCV
• RandomizedSearchCV
• Searching a Classifier's C and Its Threshold
• The Winner's Score Has Been Spent

Theory

Nothing to install: GridSearchCV, RandomizedSearchCV and TunedThresholdClassifierCV came with Lesson 3.1.1's scikit-learn, and the distributions a random search draws from come from SciPy, which the same install brought. The regression scripts import from price_models.py, the module saved in Lesson 3.5.1, and the classification script from sold_split.py, the module saved in Lesson 3.4.3, both in ~/dev/bootcamp beside listings.csv.

A search fits its pipeline dozens or hundreds of times, so each subtitle's blocks go into a script of their own, named at the subtitle's first block, each block added under the one before it and the script run with python3 and its name. The first two subtitles share tree_search.py, and no script reruns another subtitle's searches.

Settings Chosen on Folds

What. A hyperparameter is chosen by its cross-validated score on the training rows. Every candidate setting is scored on the same folds of the 2,400 training rows, the setting whose fold scores have the highest mean wins, the winner is refitted on all 2,400 rows, and the test rows score it once. The folds are Lesson 3.6.2's, KFold(n_splits=5, shuffle=True, random_state=42) for the price and StratifiedKFold with the same arguments for the label, always passed explicitly. Why. One validation set is one draw, so a setting chosen on it is chosen partly by which 600 listings the draw held. Five folds give every candidate five draws, the same five for each, so a setting has to win on average, and the test rows stay out of every choice. When. From this lesson on, for every setting a model that predicts a target fixes before its final fit: a depth, a leaf size, a k, a C, a learning rate, a threshold.

The hand version first, for the leaf sizes of Lesson 3.5.1's tree. Save it as tree_search.py:

from sklearn.model_selection import KFold, cross_val_score
from price_models import X_test, X_train, y_test, y_train, make_tree

folds = KFold(n_splits=5, shuffle=True, random_state=42)
by_leaf = {}
for leaf in [1, 5, 10, 20]:
    tree = make_tree(min_samples_leaf=leaf)
    scores = cross_val_score(tree, X_train, y_train, cv=folds)
    by_leaf[leaf] = scores.mean()
    print(leaf, scores.round(3), round(scores.mean(), 3))

1 [0.612 0.75  0.481 0.481 0.686] 0.602
5 [0.781 0.748 0.603 0.654 0.773] 0.712
10 [0.776 0.764 0.644 0.709 0.774] 0.734
20 [0.765 0.758 0.643 0.714 0.731] 0.722

Each line holds a leaf size, its five fold scores and their mean. No leaf size scores lower anywhere than on the third fold, and the folds of one leaf size spread far wider than the leaf sizes differ: 10 per leaf runs from 0.644 to 0.776 across its folds, while the best mean and the third differ by 0.022. Every candidate met the same five folds, so the means rank them: 10 per leaf first at 0.734, then 20 at 0.722 and 5 at 0.712, and one listing per leaf, the fully grown tree, last at 0.602. Lesson 3.5.1's one validation set ranked 5 per leaf first, 0.794 against 0.766 for 10. The choice made, the folds have done their job:

best_leaf = max(by_leaf, key=by_leaf.get)
chosen = make_tree(min_samples_leaf=best_leaf)
chosen.fit(X_train, y_train)  # all 2,400 rows
# 10 0.682
print(best_leaf, round(chosen.score(X_test, y_test), 3))

max returns the leaf size with the highest mean, as it returned Lesson 3.3.2's k, and the chosen tree is refitted on all 2,400 training rows, more than any fold model saw, each of which fitted on 1,920. The test rows score it once: 0.682.

DON'T:

• DON'T choose by one fold. The first fold alone ranks 5 per leaf first, 0.781 against 0.776, and the mean of all five ranks 10 first.
• DON'T set the chosen tree's 0.682 beside Lesson 3.5.1's 0.673 to decide which way of choosing works. Both are test scores, and picking between two choices by them is choosing by the test score, Lesson 3.1.2's second break.
• DON'T ship a fold model. Each was fitted on 1,920 rows to be scored, and the model that prices listings is the winner refitted on all 2,400.

GridSearchCV

What. A grid search (every combination of the values listed for each setting, each one cross-validated) is GridSearchCV, from sklearn.model_selection. It takes a pipeline, a dict mapping each setting's name to its list of values, and cv=. Its fit scores every combination on the folds, keeps the winner's settings in best_params_ and its mean fold score in best_score_, keeps every candidate's scores in cv_results_, and with refit=True, its default, fits the winner on every row fit received as best_estimator_, which the search's own predict and score use. Inside a pipeline a setting is named by its step's name, two underscores and the setting's name, decisiontreeregressor__max_depth, where decisiontreeregressor is the name make_pipeline gave the step, its class in lowercase, Lesson 3.2.3. Why. The hand loop grows with every setting: five depths beside four leaf sizes are 20 candidates and 100 fits, and the search runs them, keeps the books and refits the winner with no loop of yours. The double underscore is how a setting reaches a step, because a pipeline's own settings are its steps and three others, so a setting's name without its step names nothing the pipeline has. When. When each setting has a short list of values worth trying and their combinations are few enough to fit them all.

Added to tree_search.py:

import numpy as np
import pandas as pd
from sklearn.model_selection import GridSearchCV

params = make_tree().get_params()
print(len(params))  # 43
print([n for n in params if "max_depth" in n or "min_samples_leaf" in n])

['decisiontreeregressor__max_depth', 'decisiontreeregressor__min_samples_leaf']

get_params(), Lesson 3.1.1's, lists the pipeline's own settings and every setting of every step, 43 in all, each step's under its name and two underscores. The tree's two names are the ones a grid takes. Without the step:

wrong = GridSearchCV(make_tree(), {"max_depth": [4, 8]}, cv=folds)
try:
    wrong.fit(X_train, y_train)
except ValueError as err:
    message = str(err)
    print(message[:message.index(" for")])
    print(message[message.index("Valid"):])

Invalid parameter 'max_depth'
Valid parameters are: ['memory', 'steps', 'transform_input', 'verbose'].

The search sets each candidate's values on a fresh copy of the pipeline, and the pipeline has no max_depth. The message, which also prints the whole pipeline between those two lines, names the four settings the pipeline holds of its own, and the search stops. The grid itself, five depths, None among them for no limit, and four leaf sizes:

grid = {
    "decisiontreeregressor__max_depth": [4, 6, 8, 10, None],
    "decisiontreeregressor__min_samples_leaf": [1, 5, 10, 20],
}
search = GridSearchCV(make_tree(), grid, cv=folds)
search.fit(X_train, y_train)
print(search.best_params_)
print(round(search.best_score_, 3), search.best_score_ == by_leaf[best_leaf])

{'decisiontreeregressor__max_depth': None, 'decisiontreeregressor__min_samples_leaf': 10}
0.734 True

The search agrees with the hand loop, no depth limit at 10 per leaf, and its best_score_ is the loop's mean for 10 per leaf to the last digit, because it ran the same pipeline on the same folds. Every candidate's scores:

results = pd.DataFrame(search.cv_results_)
print(results.shape)  # (20, 15)
short = {
    "param_decisiontreeregressor__max_depth": "depth",
    "param_decisiontreeregressor__min_samples_leaf": "leaf",
    "mean_test_score": "mean",
    "std_test_score": "std",
    "rank_test_score": "rank",
}
table = results[list(short)].rename(columns=short)
print(table.sort_values("rank").head(6).round(3))

   depth  leaf   mean    std  rank
18  None    10  0.734  0.051     1
14    10    10  0.733  0.050     2
10     8    10  0.732  0.050     3
6      6    10  0.723  0.046     4
15    10    20  0.722  0.044     5
19  None    20  0.722  0.044     6

cv_results_ is a dict holding one entry per candidate under each key, which pd.DataFrame turns into a table of 20 rows and 15 columns: each setting under param_ and its name, all of a candidate's settings again as one dict under params, the five fold scores, their mean and standard deviation, the rank, and how long the fits and the scoring took. The three leaders share 10 per leaf and differ only in depth, and their means lie within 0.002 of each other while each one's folds spread with a standard deviation of 0.05. At every depth from 6 up, 10 per leaf ranks first. The grid has settled the leaf size, and among the depths of 8, 10 and no limit it holds no difference the folds can carry, so best_params_ names one of three settings that scored alike. The refit, against the first subtitle's chosen:

best = search.best_estimator_
print(search.refit, best[-1].get_depth(), best[-1].get_n_leaves())
same = np.array_equal(best.predict(X_test), chosen.predict(X_test))
print(same, round(search.score(X_test, y_test), 3))

True 15 188
True 0.682

best_estimator_ is the winning pipeline refitted on all 2,400 rows, a tree of depth 15 with 188 leaves, and it prices every test listing exactly as chosen does: the refit is the step the hand loop took by hand. A search's score uses the scorer it was handed, and with none, as here, its estimator's own score, R² for this pipeline, 0.682, once.

Lesson 3.3.2 chose SelectKBest's k on one validation set and left this search to this lesson. The selector reads the price in its fit, so it stays where that lesson put it, inside the pipeline, here between Lesson 3.2.3's preparation and its LinearRegression, and the search refits it with the rest of the pipeline on the four folds that fit each candidate. Its k is one more name in the grid, and "all", a value SelectKBest accepts, keeps every column:

from sklearn.feature_selection import SelectKBest, f_regression
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from price_models import fill, make_prep

selected = make_pipeline(
    make_prep(fill()), SelectKBest(f_regression), LinearRegression()
)
k_grid = {"selectkbest__k": [3, 5, 8, 12, 16, "all"]}
k_search = GridSearchCV(selected, k_grid, cv=folds).fit(X_train, y_train)
k_columns = ["param_selectkbest__k", "mean_test_score", "std_test_score"]
print(pd.DataFrame(k_search.cv_results_)[k_columns].round(3))
print(k_search.best_params_)
# 0.766
print(round(k_search.score(X_test, y_test), 3))

  param_selectkbest__k  mean_test_score  std_test_score
0                    3            0.534           0.049
1                    5            0.539           0.051
2                    8            0.708           0.053
3                   12            0.751           0.053
4                   16            0.783           0.050
5                  all            0.794           0.052
{'selectkbest__k': 'all'}

Three and five columns stay near 0.53, and the score climbs at each larger k the grid tried, to 0.794 with every column, the shape Lesson 3.3.2's validation rows traced. "all" wins and hands every column to the model, so the refitted pipeline predicts as Lesson 3.2.3's does, and the test rows give the 0.766 they gave that lesson.

DON'T:

• DON'T name a step's setting without its step. {"max_depth": [4, 8]} stopped the search with Invalid parameter 'max_depth', and the names get_params() lists are the ones a grid takes.
• DON'T read the order of the grid's top three as a finding. No depth limit, depth 10 and depth 8, each at 10 per leaf, scored 0.734, 0.733 and 0.732 with fold spreads of 0.05.
• DON'T fit a selector on the training rows and hand the search the columns it kept. Its fit reads the price, Lesson 3.3.2, so it belongs inside the pipeline the search refits for every fold, and a selector fitted outside has read the prices of every fold it is then scored on.

RandomizedSearchCV

What. A random search (each candidate's settings drawn at random, instead of every combination listed) is RandomizedSearchCV, from sklearn.model_selection. Where GridSearchCV takes lists, it takes a distribution (a rule for drawing values at random, with a chance for each) for any setting, n_iter, the number of candidates to draw, and random_state, which fixes the draws, and it cross-validates, keeps, refits and scores as a grid search does. SciPy's scipy.stats holds the distributions: loguniform(0.01, 0.5) draws a number from 0.01 to 0.5 spread evenly on a logarithmic scale, so every tenfold stretch inside the range gets the same share of the draws, and randint(50, 500) draws a whole number from 50 up to 499, every one equally likely. Why. A grid spends its fits on combinations. Two values of each of four settings already make 16 candidates and three of each make 81, so a grid of at most 30 candidates that varies all four settings holds at most three values of any one, while 30 draws try up to 30 values of every setting, which pays whenever one setting matters more than the others and nobody knows beforehand which. A learning rate scales each round's tree before it joins the sum, Lesson 3.5.3, so 0.01 to 0.02 changes the step in the same proportion as 0.25 to 0.5, and a log-uniform draw gives each such doubling the same share. When. For several settings at once, some of them continuous, when the fits are too slow to try every combination: a boosted model's learning rate, rounds and leaf limits.

The distributions first. Save as boost_search.py:

import numpy as np
from scipy.stats import loguniform, randint

rates = loguniform(0.01, 0.5)
print(rates.rvs(size=4, random_state=42).round(3))
draws = rates.rvs(size=10_000, random_state=42)
# share of draws from 0.01 up to 0.1, then from 0.05 up to 0.5
below = (draws < 0.1).mean()
upper = ((draws >= 0.05) & (draws < 0.5)).mean()
print(round(below, 3), round(upper, 3), round(np.log(10) / np.log(50), 3))
rounds = randint(50, 500)
print(rounds.rvs(size=4, random_state=42))
counts = rounds.rvs(size=10_000, random_state=42)
print(counts.min(), counts.max())

[0.043 0.412 0.175 0.104]
0.599 0.581 0.589
[152 485 398 320]
50 499

rvs draws from a distribution, size values at once, with random_state fixing them. Of 10,000 learning rates, 0.599 fell from 0.01 up to 0.1, under a fifth of the range's length, and 0.581 from 0.05 up to 0.5, both near 0.589, the logarithm of 10 over the logarithm of 50, which is the share a log-uniform draw gives any tenfold stretch of 0.01 to 0.5. The round counts ran from 50 to 499, because randint leaves its upper end out, as Python's range does.

The model is Lesson 3.5.3's HistGradientBoosting on the tree preparation, its gaps passed through. Two of its settings are new here: max_leaf_nodes (the most leaves any one round's tree may grow, 31 unless set) and its min_samples_leaf (the fewest training rows a leaf may hold, Lesson 3.5.1's setting for a tree, 20 unless set here). A prefix kept in a variable keeps the four long names readable, and time.perf_counter() (Python's clock for timing code) measures the search:

import time
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.model_selection import KFold, RandomizedSearchCV
from sklearn.pipeline import make_pipeline
from price_models import X_test, X_train, y_test, y_train, make_prep

folds = KFold(n_splits=5, shuffle=True, random_state=42)
boost = make_pipeline(
    make_prep("passthrough"), HistGradientBoostingRegressor(random_state=42)
)
step = "histgradientboostingregressor__"
space = {
    step + "learning_rate": loguniform(0.01, 0.5),
    step + "max_iter": randint(50, 500),
    step + "max_leaf_nodes": randint(8, 64),
    step + "min_samples_leaf": randint(5, 50),
}
boost_search = RandomizedSearchCV(
    boost, space, n_iter=30, cv=folds, random_state=42
)
start = time.perf_counter()
boost_search.fit(X_train, y_train)
print(f"{time.perf_counter() - start:.0f} seconds")

Thirty candidates on five folds are 150 boosted fits, then the refit. Across runs on the machine these lessons were checked on, the search took between about one and two minutes where the tree grid's 100 fits took a second or two, and your machine prints its own count of seconds. The winner and its two scores:

for name, value in boost_search.best_params_.items():
    print(name.split("__")[1], round(value, 3))
print(round(boost_search.best_score_, 3))
print(round(boost_search.score(X_test, y_test), 3))

learning_rate 0.032
max_iter 350
max_leaf_nodes 8
min_samples_leaf 29
0.823
0.792

best_params_ holds the values exactly as drawn, so the loop rounds them to three decimals. The winner pairs a slow learning rate, 0.032, with 350 rounds, trees of at most 8 leaves and at least 29 listings per leaf, and its folds averaged 0.823. The test rows score it once, 0.792. What the 30 candidates tried:

from sklearn.model_selection import cross_val_score

tried = pd.DataFrame(boost_search.cv_results_)
short = {
    "param_" + step + "learning_rate": "rate",
    "param_" + step + "max_iter": "rounds",
    "param_" + step + "max_leaf_nodes": "leaves",
    "param_" + step + "min_samples_leaf": "per_leaf",
    "mean_test_score": "mean",
    "std_test_score": "std",
}
table = tried[list(short)].rename(columns=short)
print(len(table), boost_search.n_splits_)  # 30 5
print(table.iloc[:, :4].nunique().tolist())
print(table.sort_values("mean", ascending=False).head(5).round(3))
defaults = cross_val_score(boost, X_train, y_train, cv=folds)
print(round(defaults.mean(), 3))           # 0.813

[30, 28, 25, 24]
     rate  rounds  leaves  per_leaf   mean    std
22  0.032     350       8        29  0.823  0.053
9   0.075     369      10        41  0.818  0.049
10  0.108      70      16        43  0.816  0.050
13  0.056     477      15        39  0.814  0.048
0   0.043     398      22        47  0.813  0.049

n_splits_ is the number of folds each candidate ran on. Thirty draws tried 30 learning rates, 28 round counts, 25 leaf counts and 24 leaf sizes. The five leaders run from a rate of 0.032 with 350 rounds to 0.108 with 70, within 0.01 of each other while each one's folds spread with a standard deviation near 0.05, so the search found a region of settings that score alike rather than one setting that beats its neighbors. At its defaults, HistGradientBoosting averaged 0.813 on the same folds, and the winner's 0.823 is the highest of 30 means, which the last subtitle takes apart.

DON'T:

• DON'T read loguniform(0.01, 0.5) as spread evenly over 0.01 to 0.5. 0.599 of its 10,000 draws fell below 0.1, under a fifth of the range.
• DON'T expect randint(50, 500) to draw 500. 10,000 draws ran from 50 to 499.
• DON'T count n_iter as the fits. Thirty candidates on five folds fitted 150 pipelines, and the refit made 151.
• DON'T read the winner's 0.823 against the defaults' 0.813 as what the search bought. One is the highest of 30 means on these folds, and the other is a single mean.

Searching a Classifier's C and Its Threshold

What. GridSearchCV searches a classifier as it searches a regressor, with scoring= naming the metric, "average_precision" for Lesson 3.4.3's average precision, and StratifiedKFold folds. A threshold is not a setting of the pipeline, so TunedThresholdClassifierCV (a wrapper that chooses the threshold by cross-validation), from sklearn.model_selection, chooses it. On each fold it fits the pipeline on the other four, scores the held-out fold's calls at 100 thresholds spread evenly between that fold's lowest and highest probability, averages the five folds' scores on one shared set of 100 thresholds, reading each fold's score at a shared threshold off the straight line between the two of its own thresholds around it, or at its nearest own threshold where the shared one lies outside its range, and keeps the threshold with the highest average in best_threshold_ and that average in best_score_. With refit=True, its default, it then fits the pipeline on every row fit received, and its predict calls a listing sold at or above best_threshold_. make_scorer (a function that turns a metric function into a scorer, the kind of object scoring= takes), from sklearn.metrics, hands it a question of your own, and greater_is_better=False marks the metric as a cost, which the scorer returns negated so that the highest score is the lowest cost. Why. C shapes the probabilities and the threshold turns them into calls, so the two answer different questions: average precision scores how the probabilities rank the listings at every threshold at once, and the alert team's cost scores the calls made at one. Lesson 3.4.3 chose its thresholds on one validation set of 600 listings, by a recall target and, in its Assignment, by this cost, and the wrapper makes the same choice on five folds for whatever scorer it is handed. When. Whenever a classifier's calls trigger an action: its settings searched by a metric of the ranking, then the threshold chosen by the question's own metric or cost, then the test rows scored once.

Save as sold_search.py, with Section 3.4's split and preparation from sold_split.py:

import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, StratifiedKFold
from sklearn.pipeline import make_pipeline
from sold_split import X_test, X_train, make_prep, y_test, y_train

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
logit = make_pipeline(make_prep(), LogisticRegression())
c_search = GridSearchCV(
    logit,
    {"logisticregression__C": [0.001, 0.01, 0.1, 1.0, 10.0]},
    scoring="average_precision",
    cv=folds,
)
c_search.fit(X_train, y_train)
c_columns = ["param_logisticregression__C", "mean_test_score", "std_test_score"]
print(pd.DataFrame(c_search.cv_results_)[c_columns].round(4))
print(c_search.best_params_)

   param_logisticregression__C  mean_test_score  std_test_score
0                        0.001           0.2917          0.0253
1                        0.010           0.3068          0.0353
2                        0.100           0.3280          0.0411
3                        1.000           0.3359          0.0450
4                       10.000           0.3361          0.0453
{'logisticregression__C': 10.0}

C=1.0, the default, is the pipeline Lesson 3.6.2 cross-validated on these folds. Average precision climbs from 0.2917 at C=0.001 to 0.3359 at 1.0, and C=10.0 passes it by 0.0002, while each one's folds spread with a standard deviation of 0.045. The table prints four decimals because at three the last two print alike. The search has to name a winner, and here it names one by a margin no fold could notice. What its score returns:

from sklearn.metrics import accuracy_score, average_precision_score

proba = c_search.predict_proba(X_test)[:, 1]
ap = average_precision_score(y_test, proba)
same = c_search.score(X_test, y_test) == ap
accuracy = accuracy_score(y_test, c_search.predict(X_test))
# True 0.308 0.788
print(same, round(ap, 3), round(accuracy, 3))

A search's score uses the scorer it was given, so on the test rows c_search.score is the average precision, 0.308, once. Lesson 3.4.2's accuracy, the share of calls that match the truth, reads 0.788 for the same model's calls and says nothing about how it ranks the listings.

The alert team's cost is Lesson 3.4.3's: a fast sale the alert misses costs 50, and an alert to a listing that does not sell within 30 days costs 10. alert_cost is that lesson's function, make_scorer turns it into a scorer, and the wrapper gets the pipeline at the C the search chose:

import numpy as np
from sklearn.metrics import confusion_matrix, make_scorer
from sklearn.model_selection import TunedThresholdClassifierCV

MISSED, WASTED = 50, 10  # a fast sale missed, an alert wasted


def alert_cost(truth, called):
    tn, fp, fn, tp = confusion_matrix(truth, called).ravel()
    return MISSED * fn + WASTED * fp


cost = make_scorer(alert_cost, greater_is_better=False)
best_C = c_search.best_params_["logisticregression__C"]
tuned = TunedThresholdClassifierCV(
    make_pipeline(make_prep(), LogisticRegression(C=best_C)),
    scoring=cost,
    cv=folds,
    store_cv_results=True,
)
tuned.fit(X_train, y_train)
print(round(tuned.best_threshold_, 4), round(tuned.best_score_))

0.1756 -3301

The folds put the threshold at 0.1756. best_score_ is negative because the scorer returns the cost negated: read off their straight lines at that threshold, the five validation folds of 480 listings average a cost of 3,301. The averaged curve around it:

curve = pd.DataFrame(tuned.cv_results_)
curve["cost"] = (-curve["scores"]).round().astype(int)
at = curve["scores"].idxmax()
print(len(curve), at)  # 100 25
around = [at - 15, at - 5, at, at + 5, at + 15]
print(curve.loc[around, ["thresholds", "cost"]].round(4))

    thresholds  cost
10      0.0702  3597
20      0.1405  3398
25      0.1756  3301
30      0.2107  3463
40      0.2810  3773

store_cv_results=True keeps the averaged curve in cv_results_, the 100 shared thresholds and their scores, and the cheapest sits at position 25. Five positions either side the cost has risen only to 3,398 and 3,463, and fifteen positions out it is 3,597 at 0.0702 and 3,773 at 0.281. Lesson 3.4.3's Assignment found 0.17 on one validation set for the default C, beside the 0.167 its cost formula gives when the probabilities are read as chances, and the five folds put the bottom of the averaged cost close to both. The test rows, once:

called = tuned.predict(X_test)
tuned_proba = tuned.predict_proba(X_test)[:, 1]
print(np.array_equal(called, tuned_proba >= tuned.best_threshold_))
tn, fp, fn, tp = confusion_matrix(y_test, called).ravel()
print(alert_cost(y_test, called), called.sum(), fp, fn)
print(cost(tuned, X_test, y_test))
print(round(tuned.score(X_test, y_test), 3))
print(round(accuracy_score(y_test, called), 3))

True
4320 330 242 38
-4320
0.533
0.533

The wrapper calls a listing sold at or above its threshold, Lesson 3.4.3's >=. On the test rows the alert costs 4,320: 330 alerts, 242 of them to listings that did not sell within 30 days, and 38 fast sales missed, where Lesson 3.4.3 measured 4,740 for alerting every seller on these rows and 6,300 for alerting none. A scorer is called with a fitted model and rows, cost(tuned, X_test, y_test), and returns the cost negated. The wrapper's own score does not use its scorer: it returns 0.533, the accuracy of its calls, whatever it was tuned for.

DON'T:

• DON'T read C=10.0's lead as a finding. It scored 0.3361 against 0.3359 for C=1.0, with folds that spread by 0.045.
• DON'T read tuned.score as the cost. It returned 0.533, the accuracy of the wrapper's calls, while cost(tuned, X_test, y_test) returned -4320.
• DON'T read best_score_ as the test cost. -3,301 averages the five folds' negated cost curves read at 0.1756, each fold of 480 listings, and the 600 test listings cost 4,320.
• DON'T hand the wrapper the test rows. It chooses on whatever rows fit receives, and rows that chose a threshold can no longer score it.

The Winner's Score Has Been Spent

What. best_score_ is the highest of the candidates' cross-validated means, the number that chose the winner. Each mean carries its folds' luck along with its setting's quality, and the highest of several is the one most likely to have been lifted by its luck, so once it has chosen it is no longer an honest estimate of the winner. The test rows, which no choice touched, give the number to report. Why. Every search here printed a best_score_ above its winner's test score: 0.734 against 0.682 for the tree, 0.794 against 0.766 for the selector's k, 0.823 against 0.792 for the boosted model, and 0.3361 against 0.308 for the classifier's average precision. One split cannot say how much of each gap the choosing made, because on this split the test rows score below the folds even when nothing is chosen, as the first block below shows. When. Every time a search's result is reported, logged, or set beside another model's.

Save as spent.py:

import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import KFold, cross_val_score
from sklearn.pipeline import make_pipeline
from price_models import X_test, X_train, y_test, y_train, fill, make_prep

folds = KFold(n_splits=5, shuffle=True, random_state=42)
linear = make_pipeline(make_prep(fill()), LinearRegression())
cv_mean = cross_val_score(linear, X_train, y_train, cv=folds).mean()
test_r2 = linear.fit(X_train, y_train).score(X_test, y_test)
print(round(cv_mean, 3), round(test_r2, 3))  # 0.794 0.766

Lesson 3.2.3's linear pipeline, with no setting chosen, scores 0.794 on the folds and 0.766 on the test rows: on this split a gap of almost 0.03 needs no search at all. Ten other splits of the whole table, each holding out its own 600 test rows, set the choosing's share apart from the draw's. random_tree grows a tree that draws half the columns at each split, Lesson 3.5.2's max_features=0.5 on a single tree, so on the same rows two copies differ only by their seed, and twenty such copies share one recipe. The fixed recipe is one copy, with nothing to choose, and the grid is this lesson's:

from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.tree import DecisionTreeRegressor
from price_models import X, make_tree, y


def random_tree(seed=42):
    tree = DecisionTreeRegressor(
        max_features=0.5, min_samples_leaf=10, random_state=seed
    )
    return make_pipeline(make_prep("passthrough"), tree)


seed_setting = "decisiontreeregressor__random_state"
grid = {
    "decisiontreeregressor__max_depth": [4, 6, 8, 10, None],
    "decisiontreeregressor__min_samples_leaf": [1, 5, 10, 20],
}
searches = {
    "fixed": (random_tree(), {seed_setting: [42]}),
    "seeds": (random_tree(), {seed_setting: list(range(20))}),
    "grid": (make_tree(), grid),
}
rows = []
for split in range(10):
    X_in, X_out, y_in, y_out = train_test_split(
        X, y, test_size=0.2, random_state=split
    )
    for name, (pipe, settings) in searches.items():
        found = GridSearchCV(pipe, settings, cv=folds).fit(X_in, y_in)
        test_r2 = found.score(X_out, y_out)
        rows.append([split, name, found.best_score_, test_r2])
runs = pd.DataFrame(rows, columns=["split", "search", "best", "test"])
runs["gap"] = runs["best"] - runs["test"]
print(runs.groupby("search")[["best", "test", "gap"]].mean().round(3))
gaps = runs.pivot_table(index="split", columns="search", values="gap")
seeds_wider = (gaps["seeds"] > gaps["fixed"]).sum()
grid_wider = (gaps["grid"] > gaps["fixed"]).sum()
print(seeds_wider, grid_wider)  # 10 7

         best   test    gap
search                     
fixed   0.715  0.700  0.016
grid    0.734  0.703  0.031
seeds   0.741  0.703  0.038

Each row averages ten splits: the reported best_score_, the test score of the refitted winner, and the gap between them. The fixed copy's cross-validated score came out 0.016 above its test score on average, a gap with no choosing in it. Choosing among the twenty seeds raised the reported score from 0.715 to 0.741 and the test score from 0.700 to 0.703, so of the 0.026 the choosing added to best_score_, less than 0.004 reached the test rows, and the seed search's gap was the wider in all ten splits. The grid of real settings reported 0.734, tested at 0.703 and had the wider gap in 7 of the 10. A winner's best_score_ carries the luck of the choice, and among candidates that differ only by luck nearly all of its lead is luck. How large the overstatement runs on another table or another grid, ten splits of this one do not say. The run fits 2,080 pipelines and took under a minute on the machine these lessons were checked on.

DON'T:

• DON'T report best_score_ as the model's score. The tree search's 0.734 chose its winner, and the test rows gave 0.682.
• DON'T search a seed. Over ten splits, twenty seeds raised the reported score by 0.026 over one fixed seed and the test score by less than 0.004.
• DON'T credit a search with the whole gap on one split. The linear pipeline chose nothing and still scored 0.794 on the folds and 0.766 on the test rows.

Lab Exercises

Lab Exercise 1: Comparables, Chosen on Folds

Work in knn_search.py in ~/dev/bootcamp, importing from price_models.py. Lesson 3.5.1 chose kNN's k on one validation set, and this time the folds choose it.

Build that lesson's scaled kNN pipeline, make_prep(fill_scale()) before a KNeighborsRegressor, and print the name of its n_neighbors setting as the pipeline lists it. Search k over 1, 2, 5, 10, 20, 50 and 100 with the lesson's folds on the 2,400 training rows. Print the candidates as a table of k, mean, standard deviation and rank, sorted by rank, then the winner and its best_score_.

Prove that best_score_ is the mean cross_val_score gives the winning pipeline on the same folds, show that the refitted model kept all 2,400 training rows, and score the test rows once. End with a comment comparing the folds' k with Lesson 3.5.1's and saying what the margins between the first three mean.

Lab Exercise 2: A Random Search for the Tree, and a Threshold for F1

In tree_random.py, search the Theory's tree pipeline at random: max_depth drawn from randint(3, 26) and min_samples_leaf from randint(1, 51), 20 candidates, as many as the grid had, with the lesson's folds and random_state=42. Print how many distinct depths and leaf sizes it tried, the four best candidates with their means and standard deviations, the winner's settings, and its best_score_, test score and the gap between the two, with a comment setting them beside the grid's.

Then, in alert_f1.py, importing from sold_split.py, hand TunedThresholdClassifierCV the Theory's logistic pipeline at C=10.0 with scoring="f1" and the stratified folds. Print its threshold and best_score_, score the test rows once by the number of alerts, precision, recall and F1, and say in a comment why F1's threshold sits above the cost's.

import pandas as pd
from sklearn.model_selection import GridSearchCV, KFold, cross_val_score
from sklearn.neighbors import KNeighborsRegressor
from sklearn.pipeline import make_pipeline
from price_models import (
    X_test, X_train, y_test, y_train, fill_scale, make_prep,
)

folds = KFold(n_splits=5, shuffle=True, random_state=42)
knn = make_pipeline(make_prep(fill_scale()), KNeighborsRegressor())
print([name for name in knn.get_params() if "n_neighbors" in name])

grid = {"kneighborsregressor__n_neighbors": [1, 2, 5, 10, 20, 50, 100]}
search = GridSearchCV(knn, grid, cv=folds).fit(X_train, y_train)
short = {
    "param_kneighborsregressor__n_neighbors": "k",
    "mean_test_score": "mean",
    "std_test_score": "std",
    "rank_test_score": "rank",
}
table = pd.DataFrame(search.cv_results_)[list(short)].rename(columns=short)
print(table.sort_values("rank").round(3))
print(search.best_params_, round(search.best_score_, 3))

['kneighborsregressor__n_neighbors']
     k   mean    std  rank
3   10  0.727  0.041     1
4   20  0.722  0.038     2
2    5  0.718  0.039     3
5   50  0.682  0.041     4
1    2  0.655  0.028     5
6  100  0.646  0.035     6
0    1  0.531  0.058     7
{'kneighborsregressor__n_neighbors': 10} 0.727

best_k = search.best_params_["kneighborsregressor__n_neighbors"]
winner = make_pipeline(
    make_prep(fill_scale()), KNeighborsRegressor(n_neighbors=best_k)
)
by_hand = cross_val_score(winner, X_train, y_train, cv=folds)
print(by_hand.mean() == search.best_score_)  # True
# 2400, every training row, where each fold model kept 1,920
print(search.best_estimator_[-1].n_samples_fit_)
# 0.702
print(round(search.score(X_test, y_test), 3))
# The folds chose k=10, as Lesson 3.5.1's validation rows did. 10, 20
# and 5 lie within 0.009 of each other while each spreads by about
# 0.04 across the folds: the folds rank them and cannot tell them apart.

import pandas as pd
from scipy.stats import randint
from sklearn.model_selection import KFold, RandomizedSearchCV
from price_models import X_test, X_train, y_test, y_train, make_tree

folds = KFold(n_splits=5, shuffle=True, random_state=42)
space = {
    "decisiontreeregressor__max_depth": randint(3, 26),
    "decisiontreeregressor__min_samples_leaf": randint(1, 51),
}
search = RandomizedSearchCV(
    make_tree(), space, n_iter=20, cv=folds, random_state=42
).fit(X_train, y_train)
short = {
    "param_decisiontreeregressor__max_depth": "depth",
    "param_decisiontreeregressor__min_samples_leaf": "leaf",
    "mean_test_score": "mean",
    "std_test_score": "std",
}
table = pd.DataFrame(search.cv_results_)[list(short)].rename(columns=short)
# [12, 17]
print(table[["depth", "leaf"]].nunique().tolist())
print(table.sort_values("mean", ascending=False).head(4).round(3))
print(search.best_params_)
best_cv = search.best_score_
test_r2 = search.score(X_test, y_test)
print(round(best_cv, 3), round(test_r2, 3), round(best_cv - test_r2, 3))
# 20 draws tried 12 depths and 17 leaf sizes, against the grid's 5 and
# 4, and chose depth 25 at 11 per leaf: 0.733 on the folds and 0.681 on
# the test rows. The grid's winner scored 0.734 and 0.682, so both
# searches reported about 0.05 more than their test rows gave.

    depth  leaf   mean    std
4      25    11  0.733  0.051
17     18    15  0.730  0.051
3       9    19  0.723  0.047
2      10    21  0.723  0.043
{'decisiontreeregressor__max_depth': 25, 'decisiontreeregressor__min_samples_leaf': 11}
0.733 0.681 0.052

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score, precision_score, recall_score
from sklearn.model_selection import StratifiedKFold, TunedThresholdClassifierCV
from sklearn.pipeline import make_pipeline
from sold_split import X_test, X_train, make_prep, y_test, y_train

folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
tuned = TunedThresholdClassifierCV(
    make_pipeline(make_prep(), LogisticRegression(C=10.0)),
    scoring="f1",
    cv=folds,
)
tuned.fit(X_train, y_train)
print(round(tuned.best_threshold_, 4), round(tuned.best_score_, 3))
called = tuned.predict(X_test)
print(
    called.sum(),
    round(precision_score(y_test, called), 3),
    round(recall_score(y_test, called), 3),
    round(f1_score(y_test, called), 3),
)
# F1 charges a missed fast sale and a wasted alert alike, so it stops
# calling at 0.2599, where the cost, which charges a miss five times an
# alert, kept calling down to 0.1756: 179 test alerts against 330.

0.2599 0.4
179 0.302 0.429 0.354

Theory Assessment

Q1: One Leaf Size, Twice

The hand loop scored four leaf sizes with cross_val_score, the grid search scored 20 settings, and the search's best_score_ equaled the loop's mean for 10 per leaf exactly. Explain why the two agree, how many fits the search made and on which rows, what refit=True did after them and how the lesson showed it, and why {"max_depth": [4, 8]} stopped the search with a message listing only memory, steps, transform_input and verbose.

Hint: Ask what the search runs for each candidate, and on which folds. For the message, ask which settings a pipeline holds of its own, and what get_params() printed for the tree's.

Q2: Thirty Draws and a Tenfold Stretch

Explain what loguniform(0.01, 0.5) and randint(50, 500) draw, why 0.599 of 10,000 learning rates fell below 0.1 while none of 10,000 round counts reached 500, and why a learning rate is drawn on that scale. Then explain why the random search could try 30 learning rates where a grid of at most 30 candidates varying the same four settings could not, how many pipelines it fitted, and what its five leaders, within 0.01 of each other, say about its winner.

Hint: Write a tenfold stretch of 0.01 to 0.5 as a share of the range's logarithm. For the grid, multiply the numbers of values of the four settings.

Q3: Two Questions, a Search and a Wrapper

The C search chose by average precision and the threshold wrapper by the alert's cost. Explain why the two were not chosen by one metric, what the wrapper computes on each fold and how it picks best_threshold_, why its best_score_ is -3,301, why c_search.score returned 0.308 and tuned.score 0.533, and how the wrapper's 0.1756 relates to Lesson 3.4.3's 0.17.

Hint: Ask which of the two settings changes the probabilities and which changes only the calls made from them. For the two score calls, ask which object uses its scorer in score, and what the other returns instead.

Q4: Four Lines for the Report

A teammate writes the lesson's results up for the pricing and alert teams in four lines:

1. The boosted model's cross-validated R² is 0.823, and no test row touched it, so 0.823 is what it will score on new listings.
2. The random search fitted 30 models, one per candidate.
3. The alert's threshold, 0.1756, was chosen on folds of the training rows, so 4,320 on the 600 test listings is a cost the team can plan with.
4. Lesson 3.4.3's single validation set chose 0.17, and its threshold cost 4,200 on the same test rows against the folds' 4,320, so for this alert one validation set chooses better than five folds.

Say which lines hold, and correct the others with the lesson's numbers.

Hint: For each line, ask what chose its number, which rows scored it, and whether the comparison it makes is itself a choice.

Assignment

Assignment: What the Choosing Added, Without the Test Rows

The pricing team logs every search's best_score_ and wants to know how much of it is the choosing before any test row is spent. Nested cross-validation (a search cross-validated as a whole) answers that. Outer folds hold rows out of the entire search, the search runs its own inner folds on the rest, chooses and refits its winner there, and the held-out outer fold, which nothing in the search saw, scores that winner. In scikit-learn it needs no new class: a GridSearchCV is an estimator, so cross_validate takes it as it takes a pipeline, and with return_estimator=True it returns the five fitted searches, each holding the best_params_ its own inner folds chose.

Write nested.py, importing from price_models.py. For the Theory's seed search, twenty copies of random_tree that differ only in random_state, and for its grid of depths and leaf sizes, fit the search on the 2,400 training rows and print its best_score_. Then cross-validate each search with the lesson's folds as the outer folds, and print the five outer scores, their mean, and each outer fold's winning settings. Touch no test row. End with comments saying how much of each best_score_ the outer folds took back, what the nested mean estimates, and what the five winners say about the setting the full search chose.

Hints:

• cross_validate(search, X_train, y_train, cv=folds, return_estimator=True) uses folds as the outer folds, and each of the five searches cuts its own 1,920 rows into five again with the folds it was built with.
• nested["estimator"] is a list of five fitted searches, and list(inner.best_params_.values()) gives one winner's settings on one line.

from sklearn.model_selection import GridSearchCV, KFold, cross_validate
from sklearn.pipeline import make_pipeline
from sklearn.tree import DecisionTreeRegressor
from price_models import X_train, y_train, make_prep, make_tree

folds = KFold(n_splits=5, shuffle=True, random_state=42)


def random_tree(seed=42):
    tree = DecisionTreeRegressor(
        max_features=0.5, min_samples_leaf=10, random_state=seed
    )
    return make_pipeline(make_prep("passthrough"), tree)


seed_setting = "decisiontreeregressor__random_state"
grid = {
    "decisiontreeregressor__max_depth": [4, 6, 8, 10, None],
    "decisiontreeregressor__min_samples_leaf": [1, 5, 10, 20],
}
searches = {
    "seeds": GridSearchCV(
        random_tree(), {seed_setting: list(range(20))}, cv=folds
    ),
    "grid": GridSearchCV(make_tree(), grid, cv=folds),
}
for name, search in searches.items():
    search.fit(X_train, y_train)
    nested = cross_validate(
        search, X_train, y_train, cv=folds, return_estimator=True
    )
    outer = nested["test_score"]
    print(name, round(search.best_score_, 3), round(outer.mean(), 3))
    print(outer.round(3))
    print([list(inner.best_params_.values()) for inner in nested["estimator"]])
# The seed search reported 0.749 and the outer folds scored its
# procedure at 0.732, so 0.017 of its best_score_ was the choosing.
# The grid reported 0.734 and scored 0.732, so on these rows its
# choosing added 0.002. A nested mean scores the procedure, search
# then refit, whatever setting wins, and the winners changed between
# outer folds: seeds 5 and 6, and depths 6, 8, 10 and unlimited, with
# 10 per leaf in four of the five.

seeds 0.749 0.732
[0.779 0.782 0.652 0.706 0.74 ]
[[6], [5], [5], [6], [5]]
grid 0.734 0.732
[0.773 0.761 0.645 0.709 0.772]
[[8, 10], [8, 10], [10, 10], [None, 10], [6, 5]]

Additional Resources

Optional, and nothing later in the module depends on either. Worth opening if you want the choosing's optimism measured on another dataset, or a search that spends fewer fits on candidates that fall behind.

• Official docs - scikit-learn 1.9 example, "Nested versus non-nested cross-validation". An SVM's C and gamma searched on the iris flowers in 30 trials with fresh folds each time, every search's best score set beside a nested estimate of the same search, and the difference between the two plotted trial by trial.
• Official docs - scikit-learn 1.9 example, "Successive Halving Iterations". HalvingRandomSearchCV run as a tournament on a forest's settings: every candidate is scored on a few rows, the better half goes on with twice the rows, and the rounds repeat until about two remain, with the one extra import the class needs while it is experimental.

Summary

A search turned out to be the hand loop made patient. Every candidate met the same folds, the mean chose, the winner was refitted on every training row, and the test rows were spent once, at the end. The one piece of syntax it added was the name that reaches inside a pipeline, step, two underscores and setting, and the grid, the random draws and the threshold wrapper were three ways of deciding which candidates get those folds and how.

What the searches returned was rarely a single right answer. The tree's top three shared a leaf size and could not tell their depths apart, the boosted model's leaders formed a region rather than a point, and C=10.0 beat C=1.0 by a margin no fold could notice. A search always names a winner, and reading cv_results_ is how you learn whether the name is a finding or a pick among near-ties.

Its own number is the last thing to trust. best_score_ did the choosing, so it carries the choice's luck, and twenty seeds of one tree showed how much: nearly all of what the choosing added to the reported score stayed behind when the test rows scored the winner. The test score is the one reported, a best_score_ is logged as the choosing's own number, and every comparison between choices is made before the test rows are opened, never after.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

folds = KFold(n_splits=5, shuffle=True, random_state=42)
grid = {"decisiontreeregressor__max_depth": [4, 8, None]}
search = GridSearchCV(make_tree(), grid, cv=folds)
search.fit(X_train, y_train)      # every candidate, every fold, refit
search.best_params_               # the winner's settings
pd.DataFrame(search.cv_results_)  # every candidate's fold scores
search.score(X_test, y_test)      # once, the number to report

step = "histgradientboostingregressor__"
space = {step + "learning_rate": loguniform(0.01, 0.5)}
RandomizedSearchCV(boost, space, n_iter=30, cv=folds, random_state=42)

cost = make_scorer(alert_cost, greater_is_better=False)
tuned = TunedThresholdClassifierCV(pipe, scoring=cost, cv=strat_folds)
tuned.fit(X_train, y_train)
tuned.best_threshold_             # predict calls at or above it
cost(tuned, X_test, y_test)       # the cost, negated