Logistic Regression and the Probability It Returns

Context

Logistic regression computes 2.3.1's weighted sum of a listing's features and passes it through a function that turns any number into one between 0 and 1, so for a question whose answer is a yes or a no it returns a probability, the model's chance of a yes.

The alert team of a listings site wants each new listing's chance of selling within 30 days on the morning it goes up: sellers whose listing will probably go fast hear it first, agents get the week's listings ranked by that chance, and the fitted weights say which features push the chance up and which pull it down.

Without it, the numbers go wrong, most of them without an error. A line fitted to sales coded as 1 and 0 returns values below 0, predict hands back True or False and drops the chance it was computed from, a pipeline left unscaled stops its solver (the routine that searches for the weights) before it finishes, and a line copied from an older tutorial prints a warning or stops with an error. Sections 3.1 to 3.3 fitted models that return a price, and this lesson gives the same preparation a target whose answer is a yes or a no.

Goal

• Diagnose what LinearRegression returns when it is fitted on a yes-or-no target coded as 1 and 0, and count the values no probability can take.
• Compute a probability from a weighted sum with the logistic function, convert it to odds and log-odds and back, and check the result against predict_proba and decision_function.
• Build the section's pipeline with LogisticRegression at its end, read classes_, predict_proba and predict, and diagnose the ConvergenceWarning that unscaled columns draw.
• Translate a weight into a change in log-odds and a factor on the odds, and verify the factor by changing one column of one listing.
• Apply C and l1_ratio to strengthen or weaken regularization, measure what each does to the weights, and replace the deprecated penalty= of an older tutorial.

Prerequisites

Neither is required, and the lesson stands on its own. They are here if you want the S-shaped curve drawn before you compute it, or scikit-learn's own statement of what its fit minimizes.

• YouTube - "StatQuest: Logistic Regression" by StatQuest with Josh Starmer. A review of linear regression, then a curve shaped like an S fitted to a yes-or-no outcome, which turns a measurement into the probability of a yes, and how the two methods differ, in under nine minutes.
• Official docs - scikit-learn 1.9 User Guide, "Linear Models", the section "Logistic regression". Why scikit-learn counts logistic regression as a classifier and regularizes it by default, then, under "Binary Case", the formula predict_proba computes, the cost the fit minimizes, and the table of what C and l1_ratio select.

Table of Contents

• Why Not Linear Regression on a Yes or No
• From a Score to a Probability
• LogisticRegression in a Pipeline
• Reading the Weights as Odds
• C, and What Regularization Holds Back

Theory

Nothing to install: every scikit-learn class this lesson uses arrived with Lesson 3.1.1's install, and Python's own warnings module needs none. Every Theory block below goes into one script, sold.py, saved in ~/dev/bootcamp beside listings.csv and run with python3 sold.py, each block added under the one before it.

The target is sold_in_30_days, True for a listing that sold within 30 days of going up and False for one that did not. The features are the nine columns Lesson 3.2.3's pipeline took, with the listing's own price added as a tenth. The price was the target of Sections 3.1 to 3.3, and here it is an input: it is set the day the listing goes up, so it exists at the moment the prediction is made, Lesson 3.3.1's test for a feature, and nothing in it is computed from whether the listing sold.

Why Not Linear Regression on a Yes or No

What. A target whose every value is a yes or a no is a category with two values, and a model that predicts a category is a classifier (an estimator whose predict returns a category instead of a number), each category it chooses between being a class. LinearRegression fitted on the yes as 1 and the no as 0 returns what it always returns, 2.3.1's weighted sum, a point on a line with no floor and no ceiling. Why. The alert team asks for a chance, a number from 0 to 1, and Lesson 3.1.1 found that a model returns a number for any area it is handed: a number below 0 is no one's chance of anything. When. Whenever the target is a yes or a no: sold within 30 days or not, a seller who answered an alert or not, a listing a reviewer flagged or not.

The split is Lesson 3.1.2's, with the target itself as stratify=:

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms",
           "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
features = numeric + nominal + ["condition"]
X = listings[features]
y = listings["sold_in_30_days"]
print(y.dtype, y.value_counts().to_dict())
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
print(X_train.shape, X_test.shape)  # (2400, 10) (600, 10)
print(y_train.sum(), y_test.sum())  # 502 126
print(round(y_train.mean(), 3), round(y_test.mean(), 3))

bool {False: 2372, True: 628}
0.209 0.21

On a bool column, 2.2.2's sum counts the True values and mean is their share: 628 of the 3,000 listings sold within 30 days, and stratifying held the share at 20.9 percent of the training rows and 21.0 percent of the test rows. The raw price failed as stratify= in Lesson 3.1.2 because most prices were a category of one, and this target is two categories of hundreds of listings.

The preparation is Lesson 3.2.3's, its numeric group filled and then scaled. make_prep builds it fresh on every call, so every model below gets steps of its own, and scale=False leaves the scaler out:

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

levels = ["poor", "fair", "good", "excellent"]


def make_prep(scale=True):
    impute = SimpleImputer(strategy="median", add_indicator=True)
    num = make_pipeline(impute, StandardScaler()) if scale else impute
    return ColumnTransformer([
        ("num", num, numeric),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])

LinearRegression on the target as 1 and 0:

line = make_pipeline(make_prep(), LinearRegression())
line.fit(X_train, y_train.astype(int))  # each True as 1, False as 0
for rows in (X_test, X_train):
    out = line.predict(rows)
    print((out < 0).sum(), round(out.min(), 3), round(out.max(), 3))
print(round(line[-1].coef_[5], 3))      # the price column's weight
print(X_test.loc[line.predict(X_test) < 0, "price"].tolist())

6 -0.336 0.438
35 -0.775 0.506
-0.142
[366000, 952000, 995000, 1544000, 941000, 845000]

The first two lines count the values below 0 and give the lowest and the highest, on the 600 test listings and then on the 2,400 the line was fitted to: six test values below 0, as low as -0.336, and 35 training values, as low as -0.775. The weight on the scaled price, -0.142, lowers the line's value by 0.142 for each standard deviation more on the price, every other column held, with no floor to stop it, and five of the six test listings below 0 are listed above 840,000.

DON'T:

• DON'T read a LinearRegression fitted on 1 and 0 as a chance. It returned six test values below 0, as low as -0.336, and 35 below 0 on its own training rows.
• DON'T split a yes-or-no target without stratify=y. Unstratified, the same seed puts 130 sales among the 600 test listings, 21.7 percent, and 498 among the 2,400 training listings, 20.8 percent.

From a Score to a Probability

What. The logistic function (it maps any number to one between 0 and 1), 1 / (1 + np.exp(-z)), turns a score z into a probability, and logistic regression computes the score as 2.3.1's weighted sum, w . x + b, before passing it through. Read the other way, the score is the log-odds of a yes: odds (the chance of a yes divided by the chance of a no, p / (1 - p)) and log-odds (the natural logarithm of the odds, 2.2.3's np.log). Why. Every score, however far from 0, comes out between 0 and 1, a higher score always gives a higher probability, and a score of 0 gives exactly one half. The weights keep the job they had in 2.3.1, adding up a weighted sum, and the sum they build is now the log-odds. When. Whenever a logistic model's probability is checked by hand, and whenever a probability has to be turned back into the score behind it.

def logistic(z):
    return 1 / (1 + np.exp(-z))  # 2.2.3's np.exp, entry by entry


def to_odds(p):
    return p / (1 - p)           # a yes against a no


z = np.array([-3.0, -2.0, -1.0, 0.0, 1.0, 2.0, 3.0])
p = logistic(z)
print(p.round(3))
print(to_odds(p).round(3))
print(np.log(to_odds(p)).round(3))

[0.047 0.119 0.269 0.5   0.731 0.881 0.953]
[ 0.05   0.135  0.368  1.     2.718  7.389 20.086]
[-3. -2. -1.  0.  1.  2.  3.]

The probabilities stay between 0 and 1 and climb with the score, steeply around a score of 0 and slowly toward the ends: from -3 to -1 the probability rises by 0.222, and from -1 to 1 by 0.462. The odds are np.exp(z). A score of 0 is even odds, 1 to 1, and each step of 1 multiplies the odds by e, about 2.718, wherever the step starts, so the logarithm of the odds gives the scores back.

The section's model, fitted here and taken apart in the next subtitle:

from sklearn.linear_model import LogisticRegression

prep = make_prep()
model = make_pipeline(prep, LogisticRegression())
model.fit(X_train, y_train)

The weighted sum for the first three test listings, by hand:

three = X_test.head(3)
seen = prep.transform(three)  # the 27 columns the model sees
w = model[-1].coef_[0]        # the one row of weights
b = model[-1].intercept_[0]
z_three = seen @ w + b        # 2.3.2's X @ w, plus the bias
print(model[-1].coef_.shape)  # (1, 27)
print(z_three.round(3))       # [-2.096 -2.172 -0.37 ]
print(logistic(z_three).round(3))
print(model.predict_proba(three)[:, 1].round(3))
print(np.allclose(z_three, model.decision_function(three)))

[0.109 0.102 0.409]
[0.109 0.102 0.409]
True

coef_ is a table of one row, 27 weights for the 27 columns the preparation hands the model, so [0] takes the row out, and intercept_ holds b in a one-entry array. The logistic function of the three hand-made scores matches the second column of predict_proba, and decision_function (the method that returns each row's score) matches the hand computation. The third listing scores -0.37 and gets 0.409: a score below 0 is a probability below one half.

DON'T:

• DON'T read a score as a probability. The third test listing scores -0.37, and its probability of a sale is 0.409.
• DON'T read odds as a probability. A probability of 0.731 is odds of 2.718, a sale 2.718 times as likely as none.
• DON'T multiply the columns by coef_ itself. It is a table of one row, and seen @ model[-1].coef_ stops with a ValueError that opens matmul: Input operand 1 has a mismatch in its core dimension 0.

LogisticRegression in a Pipeline

What. LogisticRegression, from sklearn.linear_model, is a classifier despite its name. Its fit is not 2.3.4's least squares, and no one-step formula reaches it: the fit minimizes log loss (the average, over the training rows, of minus the logarithm of the probability the model gave each row's actual class), plus the cost this lesson's final subtitle sets, with a solver, lbfgs by default, that improves the weights in steps and stops once they stop improving or after max_iter steps, 100 unless set. After the fit, classes_ lists the classes in sorted order, predict_proba returns one row per listing and one column per class in that order, and predict returns, of two classes, the second where the score is above 0, which is where its probability is above one half, and the first everywhere else. Why. Section 3.2's preparation carries over unchanged, every step fitted on the training rows alone, and the step at the end returns the chance the alert team asked for. predict keeps only which side of one half that chance fell on. When. For every model in this section, and every time a chance, rather than a yes or a no, is what someone will act on.

print(model.classes_)                    # [False  True]
both = model.predict_proba(X_test)
print(both.shape)                        # (600, 2)
print(both[:3].round(3))
print(np.allclose(both.sum(axis=1), 1))  # True
proba = both[:, 1]                       # the chance of True

[[0.891 0.109]
 [0.898 0.102]
 [0.591 0.409]]

classes_ sorts the target's two values, False before True, so column 0 holds each listing's chance of not selling and column 1 its chance of selling, and every row adds up to 1. proba keeps column 1, the section's name for it. predict against it:

pred = model.predict(X_test)
print(pred.dtype, pred.sum())             # bool 3
print(np.array_equal(pred, proba > 0.5))  # True
print(np.sort(proba[pred]).round(5))
print(np.sort(proba)[-5:-3].round(3))

[0.50001 0.51164 0.56797]
[0.461 0.477]

predict returned True for three of the 600 test listings, the three whose probability is above 0.5, one of them by less than 0.00001, while 126 of the 600 sold. The listings at 0.477 and 0.461 came out False, the same answer as every listing below them. The probabilities keep what predict drops:

train_proba = model.predict_proba(X_train)[:, 1]
print(round(proba.mean(), 3), round(y_test.mean(), 3))
print(round(train_proba.mean(), 3), round(y_train.mean(), 3))
# the chance each training listing got for what happened
given = np.where(y_train, train_proba, 1 - train_proba)
print(round(-np.mean(np.log(given)), 4))
share = y_train.mean()
flat = np.where(y_train, share, 1 - share)
print(round(-np.mean(np.log(flat)), 4))

0.208 0.21
0.209 0.209
0.4783
0.5129

Over the 600 test listings the probabilities average 0.208, against the 0.21 that sold, and over the training rows they average the training share to three decimals. The last two lines are the fit's own loss, by hand: 0.4783 on the training rows, against 0.5129 for one chance, the training share, given to every listing alike, the lower belonging to the model that gave more of its chance to what happened.

Unscaled, the same pipeline runs out of steps. Its warning runs to nine lines, so the block catches it with Python's warnings module: warnings.catch_warnings(record=True) collects every warning raised inside its with block into a list instead of printing it, and simplefilter("always") keeps a warning even when the same one was shown before.

import warnings

raw = make_pipeline(make_prep(scale=False), LogisticRegression())
with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")        # record every warning
    raw.fit(X_train, y_train)
print(len(caught), caught[0].category.__name__)
print(str(caught[0].message).splitlines()[0])
print(raw[-1].n_iter_, model[-1].n_iter_)  # [100] [69]
raw_proba = raw.predict_proba(X_test)[:, 1]
print(raw.predict(X_test).sum())           # 0
print(round(np.abs(raw_proba - proba).max(), 3))

1 ConvergenceWarning
lbfgs failed to converge after 100 iteration(s) (status=1):
0.24

The unscaled fit stopped at its 100th step with a ConvergenceWarning (scikit-learn's warning that a solver stopped before it finished), whose first line names the solver and the limit it hit, and the rest of the message suggests raising max_iter or scaling the data. n_iter_ counts the steps a solver took, and the scaled pipeline's solver stopped on its own after 69. The scaler is the one difference between the two pipelines. Unscaled, no test listing came out True, and the probabilities sit as far as 0.24 from the scaled model's.

DON'T:

• DON'T read predict as a probability. It returned True for three listings, one at 0.50001, and False for 597, among them listings at 0.477 and 0.461.
• DON'T take column 0 of predict_proba as the chance of a sale. classes_ puts False first, so column 0 is the chance of not selling, 0.891 for the first test listing.
• DON'T answer the convergence warning by raising max_iter alone. At max_iter=1000 the unscaled fit stops with the same warning after 1,000 steps.

Reading the Weights as Odds

What. A weight is the change in the log-odds for one unit of its column, with every other column held where it is, and np.exp of the weight is the factor the odds are multiplied by. One unit is one standard deviation of the training rows on the six scaled numeric columns and on the two was-missing indicators the scaler standardized with them, the switch from 0 to 1 on a one-hot column, and one level on the condition code. Why. The log-odds are the weighted sum, so a weight adds the same amount to them, and multiplies the odds by the same factor, for every listing. The probability is the logistic function of that sum, so the same weight moves a probability near one half far more than one near 0, and a weight is never a change in probability. When. Whenever a weight is reported, compared or explained to someone who will act on it.

names = prep.get_feature_names_out()
weights = pd.Series(model[-1].coef_[0], index=names)
reading = pd.DataFrame({"weight": weights, "odds factor": np.exp(weights)})
ends = reading.sort_values("weight").iloc[[0, 1, 2, -3, -2, -1]]
print(ends.round(3))

                             weight  odds factor
num__price                   -1.405        0.245
cat__neighborhood_Hillcrest  -1.038        0.354
cat__neighborhood_Northend   -0.727        0.483
cat__neighborhood_Lakeview    0.620        1.859
num__area_sqm                 0.810        2.249
cat__neighborhood_Eastgate    1.030        2.800

The three lowest weights and the three highest, each beside its odds factor. The price weight, -1.405, says that one standard deviation more on the price, every other column held, lowers the log-odds of a sale by 1.405, which multiplies the odds by 0.245, about a quarter. The standard deviation is the one the scaler learned from the training prices, and two listings check the reading:

scaler = prep.named_transformers_["num"]["standardscaler"]
price_sd = scaler.scale_[5]  # price, the sixth numeric column
print(round(price_sd))       # 196344
first, top = X_test.index[0], X_test.index[proba.argmax()]
for label in (first, top):
    listing = X_test.loc[[label]]
    pricier = listing.assign(price=listing["price"] + price_sd)
    before = model.predict_proba(listing)[0, 1]
    after = model.predict_proba(pricier)[0, 1]
    factor = to_odds(after) / to_odds(before)
    print(label, round(before, 3), round(after, 3), round(factor, 3))

566 0.109 0.029 0.245
170 0.568 0.244 0.245

The scaler inside the numeric group is reached by the name make_pipeline gave it, standardscaler, and scale_[5] is the training prices' standard deviation, 196,344. Listing 566 is the first test listing, and 170 the one the model gives its highest probability. One standard deviation added to each price multiplied both listings' odds by 0.245, the exponential of the weight, while the probability fell by 0.08 for one and by 0.324 for the other.

A one-hot weight reads in pairs. The seven neighborhood columns add up to the column of ones, Lesson 3.2.2's lost unit of rank, so a listing that switches one neighborhood on switches another off, and its log-odds move by the difference between the two weights:

hill = X_test[X_test["neighborhood"] == "Hillcrest"].head(1)
moved = hill.assign(neighborhood="Eastgate")
before = model.predict_proba(hill)[0, 1]
after = model.predict_proba(moved)[0, 1]
gap = (weights["cat__neighborhood_Eastgate"]
       - weights["cat__neighborhood_Hillcrest"])
print(hill.index[0], round(before, 3), round(after, 3))
print(round(to_odds(after) / to_odds(before), 3), round(np.exp(gap), 3))

2590 0.126 0.533
7.903 7.903

Moved from Hillcrest to Eastgate with nothing else changed, listing 2590 went from 0.126 to 0.533, and its odds were multiplied by 7.903, the exponential of the Eastgate weight minus the Hillcrest weight, which print rounded as 1.030 and -1.038. Neither weight alone is the effect of its neighborhood, as Lesson 3.2.2 found for a price model, and their difference is the effect of the move.

A weight is worth reading only where the rows pin it down. Lesson 3.3.2 found area's twins, rooms and bathrooms, changing sign between halves of the same listings, and here twenty halves of the training rows, each fitted with the section's model, count how often each numeric weight came out negative:

negative = []
for seed in range(10):
    X_a, X_b, y_a, y_b = train_test_split(
        X_train, y_train, test_size=0.5, random_state=seed, stratify=y_train
    )
    for X_half, y_half in [(X_a, y_a), (X_b, y_b)]:
        half = make_pipeline(make_prep(), LogisticRegression())
        half.fit(X_half, y_half)
        negative.append(half[-1].coef_[0][:6] < 0)
print(pd.DataFrame(negative, columns=numeric).sum().to_dict())

{'area_sqm': 0, 'year_built': 2, 'distance_km': 19, 'rooms': 5, 'bathrooms': 12, 'price': 20}

The price weight came out negative in all twenty fits and the area weight in none, while rooms' was negative in 5 and bathrooms' in 12. So the model gives a listing priced higher, every other column the same, a lower chance of selling within 30 days, and twenty halves agree. That describes the fitted model's predictions: why listings sell as they do is not something a weight shows, and no run here took a cause away.

DON'T:

• DON'T read a weight as a change in probability. One standard deviation of price took listing 566 from 0.109 to 0.029 and listing 170 from 0.568 to 0.244, under one weight, -1.405.
• DON'T read one neighborhood's weight as the effect of its neighborhood. The move from Hillcrest to Eastgate multiplied the odds by 7.903, the exponential of the difference between two weights.
• DON'T quote a twin's weight before halves of the rows have pinned it down. Over twenty halves, rooms' weight came out negative in 5 and bathrooms' in 12.

C, and What Regularization Holds Back

What. Regularization (a cost on large weights that the fit pays beside its loss) is the general form of the penalty Lesson 3.1.3 switched off in QuantileRegressor with alpha=0.0. In LogisticRegression the cost enters the fit divided by C, 1.0 unless set, and by the number of training rows, so a larger C is a weaker cost, and C=np.inf switches it off. l1_ratio sets its shape: at 0, the default, it charges half the sum of the squared weights, at 1 the sum of their sizes, and between the two a mix. Why. The cost keeps a weight small unless the log loss on the training rows pays for it, which holds back the weights the rows cannot pin down. Every LogisticRegression pays it unless told otherwise, so every weight the last subtitle read was already held back by C=1.0. When. Whenever weights are read, because the cost shaped them, and whenever C or l1_ratio is set, a choice like any other setting, made on validation rows carved from the training rows as in Lesson 3.1.2's Assignment. Lesson 3.6.3 searches such settings systematically.

eastgate = "cat__neighborhood_Eastgate"
for C in [0.01, 0.1, 1.0, 10.0]:
    held = make_pipeline(make_prep(), LogisticRegression(C=C))
    held.fit(X_train, y_train)
    w_c = pd.Series(held[-1].coef_[0], index=names)
    p_c = held.predict_proba(X_test)[:, 1]
    print(C, (w_c == 0).sum(), round(np.sqrt(w_c @ w_c), 3),
          round(w_c["num__price"], 3), round(w_c[eastgate], 3),
          round(p_c.min(), 3), round(p_c.max(), 3))

0.01 0 0.458 -0.279 0.072 0.05 0.369
0.1 0 1.496 -0.888 0.546 0.006 0.459
1.0 0 2.648 -1.405 1.03 0.001 0.568
10.0 0 3.178 -1.521 0.95 0.001 0.599

Each line holds C, how many weights came out exactly 0, the length of the 27 weights, 2.3.1's np.sqrt(w @ w), the price and Eastgate weights, and the smallest and largest test probability. As C falls from 10 to 0.01 the length shrinks from 3.178 to 0.458 and the price weight from -1.521 to -0.279, while the probabilities close in on each other: at C=0.01 none is below 0.05 or above 0.369. The squared cost holds no weight at exactly 0, and it shrinks the weights as a whole, not each one.

The absolute cost, l1_ratio=1.0, is refused by the default solver:

try:
    l1_fit = make_pipeline(make_prep(), LogisticRegression(l1_ratio=1.0))
    l1_fit.fit(X_train, y_train)
except ValueError as err:
    print(err)

Solver lbfgs supports only 'l2' or None penalties, got l1 penalty.

The message still speaks in penalties, the old name. solver="saga" (a solver that steps through the training rows in a shuffled order and accepts every l1_ratio from 0 to 1) takes it, with random_state fixing its shuffle:

for C in [0.01, 0.1]:
    sparse = make_pipeline(
        make_prep(),
        LogisticRegression(solver="saga", l1_ratio=1.0, C=C, random_state=42),
    )
    sparse.fit(X_train, y_train)
    kept = pd.Series(sparse[-1].coef_[0], index=names)
    print(C, (kept == 0).sum(), sparse[-1].n_iter_)
print(kept[kept != 0].round(3))  # the C=0.1 fit's

0.01 25 [18]
0.1 17 [19]
num__area_sqm                       0.158
num__year_built                     0.008
num__distance_km                   -0.300
num__price                         -0.443
num__missingindicator_year_built    0.012
cat__neighborhood_Eastgate          0.100
cat__neighborhood_Hillcrest        -0.246
cat__neighborhood_Riverside         0.013
cat__energy_rating_C               -0.157
cond__condition                     0.203
dtype: float64

At C=0.01 the absolute cost held 25 of the 27 weights at exactly 0, and at C=0.1 it held 17 and kept the ten printed. Rooms and bathrooms are among the seventeen at 0, and so is every energy rating but C, and a weight at 0 takes its column out of the model.

Most tutorials set the cost with penalty=:

old = make_pipeline(make_prep(), LogisticRegression(penalty="l2"))
old.fit(X_train, y_train)
print(np.allclose(old.predict_proba(X_test)[:, 1], proba))

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/sklearn/linear_model/_logistic.py:1381: FutureWarning: 'penalty' was deprecated in version 1.8 and will be removed in 1.10. To avoid this warning, leave 'penalty' set to its default value and use 'l1_ratio' or 'C' instead. Use l1_ratio=0 instead of penalty='l2', l1_ratio=1 instead of penalty='l1', l1_ratio set to a float between 0 and 1 instead of penalty='elasticnet', and C=np.inf instead of penalty=None.
  warnings.warn(
True

penalty="l2" fits the default model, the same probabilities as model, under a FutureWarning (Python's warning that a behavior will change in a later release), whose message gives the replacement for every old value: l1_ratio=0 for "l2", l1_ratio=1 for "l1", a float between 0 and 1 for "elasticnet" and C=np.inf for None.

DON'T:

• DON'T copy penalty= from a tutorial. penalty="l2" fits the default model under a FutureWarning, and penalty="l1" prints the same warning and then stops with Solver lbfgs supports only 'l2' or None penalties, got l1 penalty.
• DON'T read a larger C as a stronger cost. The weights' length is 0.458 at C=0.01 and 3.178 at C=10.
• DON'T expect every weight to shrink as C falls. Eastgate's is 0.95 at C=10 and 1.03 at C=1.0, while the weights' length fell.
• DON'T read the columns the absolute cost keeps as the columns that matter. It kept 2 at C=0.01 and 10 at C=0.1.
• DON'T choose C by the test rows. The probabilities printed for four values of C show what it does, and the choice is made on validation rows.

Lab Exercises

Lab Exercise 1: Price and Distance, as a Line and as a Chance

In two_columns.py in ~/dev/bootcamp, load listings.csv and split price and distance_km against sold_in_30_days with the Theory's test size, seed and stratifying. Neither column has a gap, so each model below is a pipeline of a StandardScaler and the model.

Fit LinearRegression on the target as 1 and 0, and print how many test listings it puts below 0 and the lowest value, rounded to three decimals. Fit LogisticRegression on the target and print its two weights and its intercept, rounded the same way.

Then, for the first five test listings, scale the two columns with the pipeline's fitted scaler, compute each listing's score from the weights and the intercept and its probability from the score, and print both, rounded. Check the probabilities against predict_proba and the scores against decision_function. Finally print the first listing's odds and log-odds beside its score.

Lab Exercise 2: Condition Under Three Costs

In condition_costs.py, rebuild the Theory's split and its preparation, scaler included, and fit the section's pipeline. Print classes_, then the mean probability of a sale among the test listings that sold and among the ones that did not.

Print the condition weight and its odds factor. Take the first test listing in poor condition, raise it one level to fair, and print its index label, its probability before and after, and the factor its odds were multiplied by. Do the same for the first test listing in excellent condition moved down to poor, print the exponential of minus three times the condition weight, and say in a comment why the two numbers match.

Then fit the pipeline at C of 0.05, 0.5 and 5, and print for each the column with the largest weight in size, that size and the condition weight. Finally fit the absolute cost with saga at C=0.05 and random_state=42, and print how many weights it holds at 0 and the ones it keeps, by name. End with a comment saying what the condition weight claims about a listing, and what C did to it.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

listings = pd.read_csv("listings.csv")
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    listings[["price", "distance_km"]], y,
    test_size=0.2, random_state=42, stratify=y
)
line = make_pipeline(StandardScaler(), LinearRegression())
line.fit(X_train, y_train.astype(int))
out = line.predict(X_test)
print((out < 0).sum(), out.min().round(3))  # 1 -0.044

model = make_pipeline(StandardScaler(), LogisticRegression())
model.fit(X_train, y_train)
w = model[-1].coef_[0]
b = model[-1].intercept_[0]
print(w.round(3), round(b, 3))              # [-0.29  -0.315] -1.383

five = X_test.head(5)
scaled = model[0].transform(five)  # the training rows' scaler
z = scaled @ w + b
p = 1 / (1 + np.exp(-z))
print(z.round(3))
print(p.round(3))
print(np.allclose(p, model.predict_proba(five)[:, 1]))
print(np.allclose(z, model.decision_function(five)))
odds = p[0] / (1 - p[0])
print(round(odds, 3), round(np.log(odds), 3), round(z[0], 3))

[-1.806 -2.107 -1.517 -0.732 -1.024]
[0.141 0.108 0.18  0.325 0.264]
True
True
0.164 -1.806 -1.806

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms",
           "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)


def make_prep():
    impute = SimpleImputer(strategy="median", add_indicator=True)
    return ColumnTransformer([
        ("num", make_pipeline(impute, StandardScaler()), numeric),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])


def to_odds(p):
    return p / (1 - p)


model = make_pipeline(make_prep(), LogisticRegression())
model.fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
sold = y_test.to_numpy()
print(model.classes_)  # [False  True]
print(round(proba[sold].mean(), 3), round(proba[~sold].mean(), 3))

0.245 0.198

names = model[0].get_feature_names_out()
weights = pd.Series(model[-1].coef_[0], index=names)
step = weights["cond__condition"]
print(round(step, 3), round(np.exp(step), 3))

moves = [("poor", "fair"), ("excellent", "poor")]
for start, end in moves:
    listing = X_test[X_test["condition"] == start].head(1)
    changed = listing.assign(condition=end)
    before = model.predict_proba(listing)[0, 1]
    after = model.predict_proba(changed)[0, 1]
    factor = to_odds(after) / to_odds(before)
    print(listing.index[0], round(before, 3), round(after, 3), round(factor, 3))
print(round(np.exp(-3 * step), 3))
# Excellent to poor is three levels down: the log-odds lose 3 * 0.401,
# so the odds are divided by 1.493 three times, a factor of 0.301 from the unrounded weight.

0.401 1.493
794 0.257 0.341 1.493
24 0.415 0.176 0.301
0.301

for C in [0.05, 0.5, 5.0]:
    held = make_pipeline(make_prep(), LogisticRegression(C=C))
    held.fit(X_train, y_train)
    w = pd.Series(held[-1].coef_[0], index=names)
    print(C, w.abs().idxmax(), round(w.abs().max(), 3),
          round(w["cond__condition"], 3))

sparse = make_pipeline(
    make_prep(),
    LogisticRegression(solver="saga", l1_ratio=1.0, C=0.05, random_state=42),
)
sparse.fit(X_train, y_train)
kept = pd.Series(sparse[-1].coef_[0], index=names)
print((kept == 0).sum())  # 23
print(kept[kept != 0].round(3))
# The condition weight claims that one level better, all else held,
# multiplies a listing's odds of a sale by 1.493. A stronger cost
# shrinks it to 0.256 at C=0.05, a weaker one lets it grow to 0.422
# at C=5, and the absolute cost at C=0.05 keeps it as one of four.

0.05 num__price 0.668 0.256
0.5 num__price 1.312 0.384
5.0 num__price 1.512 0.422
num__distance_km       -0.267
num__price             -0.249
cat__energy_rating_C   -0.006
cond__condition         0.131
dtype: float64

Theory Assessment

Q1: A Line Below Zero, and a Score Below One Half

The line fitted to sold_in_30_days as 1 and 0 returned six test values below 0, as low as -0.336, and no probability from LogisticRegression can fall outside 0 to 1. Explain both facts. Then take the third test listing, which scored -0.37: give its probability, its odds and its log-odds, and explain why each step of 1 in the score multiplied the odds by the same 2.718 while the probability rose by 0.222 from -3 to -1 and by 0.462 from -1 to 1.

Hint: Ask what a weighted sum can return, and what the logistic function does with any number it is handed. For the steps, write the odds as np.exp of the score, and the probability as the odds over 1 plus the odds.

Q2: Three Trues, and a Fit That Ran Out of Steps

predict said True for three of the 600 test listings, while 126 of them sold and the probabilities averaged 0.208. Explain what predict computes, why the listing at 0.50001 came out True and the ones at 0.477 and 0.461 False, and what classes_ decides about the columns of predict_proba. Then say what the ConvergenceWarning and n_iter_ report about the unscaled and the scaled pipelines, which difference between the two decided it, and what raising max_iter to 1,000 changed.

Hint: Ask what predict keeps of a probability and what it throws away. For the warning, set [100] and [69] beside max_iter.

Q3: One Factor, Two Drops, and a Cost on the Weights

Explain what the price weight, -1.405, says about the log-odds and the odds, and why one standard deviation added to two listings' prices multiplied both their odds by 0.245 while one probability fell by 0.08 and the other by 0.324. Explain why the move from Hillcrest to Eastgate is read from two weights, and why the weights of rooms and bathrooms are not read at all. Then say what the four fits from C=0.01 to C=10, and the absolute cost's 25 and 17 weights at 0, show about any weight a LogisticRegression returns.

Hint: A weight lives in the log-odds, and the probability is the logistic function of their sum. For the costs, ask what the fit pays for a large weight, and what a larger C does to that price.

Q4: Four Claims About One Refit

A teammate adds these lines at the end of sold.py.

tight = make_pipeline(make_prep(), LogisticRegression(C=100.0))
tight.fit(X_train, y_train)
print(tight.predict(X_test).mean())
print(round(tight.predict_proba(X_test)[:, 0].mean(), 3))
print(round(tight[-1].coef_[0][5], 3))

Reading the run, the teammate sends four claims to the alert team:

1. The default fits with no regularization, and C=100.0 switches it on as strong as it goes, so these are the safest weights to read.
2. The model expects half a percent of these 600 listings to sell within 30 days.
3. A listing's chance of selling within 30 days averages 0.793.
4. One standard deviation more on the price multiplies a listing's odds of selling by about 0.21, whatever the rest of the listing holds.

Without running the lines, say what they print, which claims hold, and for each that does not, the number or the check that refutes it.

Hint: For each claim, ask which printed number it reads and what that number measures: a share of yes-or-no answers or a mean of probabilities, which column of predict_proba, and which way C turns the cost.

0.005
0.793
-1.541

chance = tight.predict_proba(X_test)[:, 1]
price_w = tight[-1].coef_[0][5]
print(round(chance.mean(), 3))    # 0.207
print(round(np.exp(price_w), 3))  # 0.214

Assignment

Assignment: A Chance the Alert Team Can Quote

The alert team wants to tell each seller, on the morning a listing goes up, its chance of selling within 30 days, as a number. A quoted chance is a promise about a group: among the listings quoted about 30 percent, about 30 percent should sell. Before anything is quoted, the team wants C chosen for these probabilities, and the probabilities held up against what happened.

Two pieces do part of the work. log_loss(y_true, p), from sklearn.metrics, computes the lesson's log loss for any rows, so it scores probabilities on rows the fit never saw, smaller being better. A reliability table checks the chances themselves: cut the listings into five bands of equal size by their probability with 2.6.1's pd.qcut, and set each band's mean probability beside the share of its listings that sold.

Write quote_check.py. On the section's split, carve validation rows out of the training rows as Lesson 3.1.2's Assignment did, stratified on the target. For C of 0.001, 0.01, 0.1, 1, 10 and 100, fit the section's pipeline on the rows left and print each validation log loss to four decimals, beside the log loss of quoting every validation listing one number, the share of sales among the rows the models were fitted on. Choose C by a rule written before the loop: the smallest validation log loss to four decimals, and of a tie the smaller C, the stronger cost. Refit at that C on all the training rows and score the test rows once: their log loss beside the training share quoted to every test listing, then the reliability table, each band's count, mean probability and share sold, to three decimals. End with a comment saying whether the team can quote these probabilities to sellers as chances, and in what form.

Hints:

• The validation rows come from a second train_test_split of X_train and y_train with stratify=y_train, and 2.2.1's np.full(n, value) builds the one-number quote.
• pd.qcut(proba, 5, labels=[...]) returns one band per test listing, and a DataFrame of band, probability and outcome grouped by band with 2.6.2's named aggregation gives the whole table in one call.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

listings = pd.read_csv("listings.csv")
numeric = ["area_sqm", "year_built", "distance_km", "rooms",
           "bathrooms", "price"]
nominal = ["neighborhood", "property_type", "energy_rating"]
levels = ["poor", "fair", "good", "excellent"]
X = listings[numeric + nominal + ["condition"]]
y = listings["sold_in_30_days"]
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)
# validation rows carved from the training rows, 3.1.2's Assignment
X_fit, X_val, y_fit, y_val = train_test_split(
    X_train, y_train, test_size=0.25, random_state=42, stratify=y_train
)
print(len(X_fit), len(X_val), len(X_test))  # 1800 600 600


def make_model(C):
    impute = SimpleImputer(strategy="median", add_indicator=True)
    prep = ColumnTransformer([
        ("num", make_pipeline(impute, StandardScaler()), numeric),
        ("cat", OneHotEncoder(sparse_output=False), nominal),
        ("cond", OrdinalEncoder(categories=[levels]), ["condition"]),
    ])
    return make_pipeline(prep, LogisticRegression(C=C))


# one quote, the fit rows' share, for every validation listing
flat_val = np.full(len(y_val), y_fit.mean())
print(round(log_loss(y_val, flat_val), 4))  # 0.5117
val_loss = {}
for C in [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]:
    fitted = make_model(C).fit(X_fit, y_fit)
    p_val = fitted.predict_proba(X_val)[:, 1]
    val_loss[C] = round(log_loss(y_val, p_val), 4)
print(val_loss)
# smallest loss to four decimals, min keeps the first C of a tie
best_C = min(val_loss, key=val_loss.get)
print(best_C)

{0.001: 0.506, 0.01: 0.5004, 0.1: 0.4939, 1.0: 0.4909, 10.0: 0.4903, 100.0: 0.4903}
10.0

model = make_model(best_C).fit(X_train, y_train)
proba = model.predict_proba(X_test)[:, 1]
flat_test = np.full(len(y_test), y_train.mean())
print(round(log_loss(y_test, proba), 4), round(log_loss(y_test, flat_test), 4))

bands = pd.qcut(proba, 5, labels=["q1", "q2", "q3", "q4", "q5"])
table = pd.DataFrame({"band": bands, "quoted": proba, "sold": y_test.to_numpy()})
check = table.groupby("band").agg(
    listings=("quoted", "size"),
    quoted=("quoted", "mean"),
    sold=("sold", "mean"),
)
print(check.round(3))
# The ends hold: the lowest fifth was quoted 8.2 percent and 5.0
# percent sold, the highest 36.4 and 32.5. The middle three fifths
# were quoted 14.2 to 25.8 percent and sold at 21.7 to 23.3, nearly
# alike. On these 600 listings, one draw, the team can quote three
# groups, less likely, typical and more likely, not a percent.

0.4926 0.514
      listings  quoted   sold
band                         
q1         120   0.082  0.050
q2         120   0.142  0.217
q3         120   0.190  0.225
q4         120   0.258  0.233
q5         120   0.364  0.325

default = make_model(1.0).fit(X_train, y_train)
p_one = default.predict_proba(X_test)[:, 1]
print(round(log_loss(y_test, p_one), 4))  # 0.4919

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want to see how the curve's weights are chosen, or how a classifier's probabilities are checked against what happened.

• YouTube - "Logistic Regression Details Pt 2: Maximum Likelihood" by StatQuest with Josh Starmer. How the S-shaped curve is fitted to the data by maximum likelihood, keeping the candidate curve that gives the outcomes that happened the highest chance. It builds on the channel's Part 1, on the coefficients.
• Official docs - scikit-learn 1.9 User Guide, "Probability calibration", the section "Calibration curves". How a calibration curve sets a classifier's probabilities against how often a yes turns up, bin by bin, why LogisticRegression tends to pass that check, and how three other classifiers lean away from it, one pushing its probabilities toward 0 and 1, the other two keeping them away from both.

Summary

A classifier's real output turned out to be a number, not a class. Logistic regression computes the weighted sum every linear model in this module computed, and what changed is what the sum means: the log-odds of a yes, turned by the logistic function into a chance that stays between 0 and 1 however far the sum runs. The yes or no from predict is that chance cut at one half, and here the cut let three of 600 through, which says more about where one half sits than about the listings.

Reading the model came down to keeping three scales apart. A weight adds the same amount to every listing's log-odds, its exponential multiplies every listing's odds by the same factor, and the probability, the one scale a seller understands, moves by a different amount for each listing under one weight. The price weight kept its sign across every half of the training rows and the twins' did not, so even a correct reading waits on whether the rows pinned the weight down.

Every weight was also a compromise struck with a cost. C sets what large weights are worth paying for, the squared cost shrinks them as a whole while the absolute cost removes some, and the default is a cost, not its absence. The probability is the product the rest of this section works with: how good a yes or a no is depends on which mistakes cost what, and where to cut the chance is a choice of its own.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

p = 1 / (1 + np.exp(-z))                   # score to probability
z = np.log(p / (1 - p))                    # probability to score
model = make_pipeline(prep, LogisticRegression())
model.fit(X_train, y_train)                # log loss plus the cost
model.classes_                             # [False  True], sorted
proba = model.predict_proba(X_test)[:, 1]  # the chance of True
model.predict(X_test)                      # True where proba > 0.5
model.decision_function(X_test)            # the scores, the log-odds
np.exp(model[-1].coef_[0])                 # each weight's odds factor
LogisticRegression(C=0.1)                  # stronger cost, shorter weights
# the absolute cost, some weights held at 0
LogisticRegression(solver="saga", l1_ratio=1.0, C=0.1, random_state=42)