Installing scikit-learn, and What a Model Actually Is

Context

A model is a function that takes a listing's feature values and returns a number, and its parameters (the numbers inside it that the fit chooses) decide which number. Fitting chooses them from listings whose prices are already known, and scikit-learn is the library that holds models built this way, each one an object you create with its settings, fit, and then ask for predictions.

Every library model in Module 3 but Lesson 3.5.3's XGBoost comes from it. A listings site fits a price model on the listings it already holds, asks the fitted model for the price of a listing posted this morning, reads the weight the fit chose to see what one more square meter adds to a price, and in Section 3.5 swaps the linear model for a tree by changing the one line that creates it.

What goes wrong is rarely loud. A model returns a price for any area it is handed, an area seven times the largest in the file included, and a NumPy array with its two columns swapped is priced at minus a million under nothing more than a warning about missing column names. What does stop the work is a single column passed as a Series, a missing area, or a learned weight read before the fit. Lesson 2.3.4 computed a least squares fit by hand with np.linalg.lstsq, and this lesson installs the library that runs the same computation behind one call, .fit().

Goal

• Install scikit-learn 1.9.1 and SciPy 1.18.1 into (bootcamp) with uv pip install, and confirm the versions your scripts import.
• Distinguish a model's parameters, which the fit chooses, from its hyperparameters, which you choose before the fit.
• Fit a LinearRegression with .fit(X, y), read the attributes the fit set, and diagnose the errors for a Series as X, a missing value and an attribute read before the fit.
• Reproduce LinearRegression's coef_ and intercept_ with np.linalg.lstsq and a column of ones, and explain what fit_intercept=False changes.
• Predict prices for new listings with .predict(), and identify which column mismatches raise, which only warn, and which inputs come back priced without a word.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want the picture of a line being fitted, or scikit-learn's own first example, before you start.

• YouTube - "The Main Ideas of Fitting a Line to Data (The Main Ideas of Least Squares and Linear Regression.)" by StatQuest with Josh Starmer. How far a line sits from a set of points is measured as one number, and fitting is finding the line where that number is smallest.
• Official docs - scikit-learn 1.9, "Getting Started", the section "Fitting and predicting: estimator basics". fit and predict on a two-row example, and the shapes X and y take. Its model is a classifier (a model that predicts a category rather than a number), the kind Section 3.4 fits.

Table of Contents

• Installing scikit-learn
• A Model Is a Function With Learned Parameters
• The Estimator Interface
• LinearRegression Is Least Squares
• Predicting on New Rows

Theory

This lesson is the first to open listings.csv, Module 3's dataset of 3,000 listings, one row per property, which Module 3's setup page had you save into ~/dev/bootcamp. The first subtitle opens it once the install is done. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Installing scikit-learn

What. scikit-learn is the third-party library of models for tables, with the tools that prepare their data and score their predictions. It is installed under the name scikit-learn and imported under the name sklearn, and scikit-learn==1.9.1, with scipy==1.18.1 beside it, is the exact pair every result in this module was printed by. Why. (bootcamp) holds Module 2's five libraries, and none of them is a library of models. SciPy (a library of scientific routines) is one scikit-learn requires and calls for its numerical work, LinearRegression's least squares included, and its release decides numbers that later lessons in this module print, so it is pinned beside scikit-learn. When. Once, now, with (bootcamp) active. The rest of the module fits its models with what this line installs, and Lesson 3.5.3 adds XGBoost beside it.

Before the install, (bootcamp) has no sklearn to import:

(bootcamp) goku@machine:~/dev/bootcamp$ python3 -c "import sklearn"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'sklearn'

In ~/dev/bootcamp, with (bootcamp) active, install the pinned pair:

uv pip install scikit-learn==1.9.1 scipy==1.18.1

Among the lines uv prints, the ones that start with + name what it added:

 + cloudpickle==3.1.2
 + joblib==1.6.0
 + narwhals==2.27.1
 + scikit-learn==1.9.1
 + scipy==1.18.1
 + threadpoolctl==3.7.0

Two names went in and six libraries came out. scikit-learn lists joblib (runs work in parallel on several processor cores), narwhals (a common layer over several table libraries, pandas among them) and threadpoolctl (limits how many threads the numeric code underneath may start) as libraries it needs, and cloudpickle arrives because joblib needs it in turn. Only scikit-learn and SciPy are pinned, so the other four lines can read differently on your machine, in their versions and in which of them appear. NumPy has no + line, because numpy 2.5.2 is already in (bootcamp), from Lesson 2.2.1 or from Module 3's setup page, and both new libraries run on it as it is. Confirm the pair from a script, check_sklearn.py:

import scipy
import sklearn

print(sklearn.__version__)  # 1.9.1
print(scipy.__version__)    # 1.18.1

SciPy's arrival also finishes a call Lesson 2.7.1 left stopped. Spearman's correlation between two Series hands its work to SciPy, and on the module's listings it now runs:

import pandas as pd

listings = pd.read_csv("listings.csv")
rho = listings["price"].corr(listings["area_sqm"], method="spearman")
print(round(rho, 3))  # 0.799

The import name installs nothing. uv pip install sklearn asks the package index for a package called sklearn, and that name belongs to a deprecated package whose build stops with an error:

(bootcamp) goku@machine:~/dev/bootcamp$ uv pip install sklearn
error: Failed to build `sklearn==0.0.post12`
  cause: The build backend returned an error

Further down, uv quotes the package's own message, which opens The 'sklearn' PyPI package is deprecated, use 'scikit-learn', and one of its hints asks whether you meant to install scikit-learn instead. Nothing was installed, and import sklearn raises the same ModuleNotFoundError as before.

DON'T:

• DON'T install sklearn. It is the name you import, not the name of the library, and uv pip install sklearn fails to build and installs nothing.
• DON'T leave scipy==1.18.1 off the line. The install still succeeds, because scikit-learn brings SciPy with it, but at the newest release its requirements allow on the day you run it, and every SciPy release after 1.18.1 is one this module was never printed with.

A Model Is a Function With Learned Parameters

What. A model is a function that takes a listing's feature values and returns a prediction, here a price. Its parameters decide which price comes out, and fitting (choosing the parameters from rows whose target is known) is how they get their values. Why. The function's form is chosen once and the parameters carry everything the model knows, so two models of the same form with different parameters price the same listing differently, and only the data can say which parameters to keep. When. Every time you read what a model predicted or what it learned: a prediction is the function applied to one row, and what it learned is its parameters.

Lesson 2.3.1's w . x + b is a model. With one feature, area, it prices a listing at w times its area plus b, 2.3.1's bias, and w and b are its two parameters. Set by hand, two choices of them price the first four listings of the file, all four of which have an area:

import pandas as pd

listings = pd.read_csv("listings.csv")
first = listings.head(4)
area = first["area_sqm"].to_numpy()
price = first["price"].to_numpy()

def predict_price(area, w, b):
    return w * area + b  # 2.3.1's w . x + b, one feature

print(area)              # [175.7  30.5 143.6  42. ]
print(price)             # [509000 169000 582000  93000]
print(predict_price(area, 3000.0, 50000.0))
print(predict_price(area, 1000.0, 300000.0))
# -10000.0, a price for an area no listing has
print(predict_price(-20.0, 3000.0, 50000.0))

[577100. 141500. 480800. 176000.]
[475700. 330500. 443600. 342000.]

Both are models. Each returned a price for all four listings, and nothing in either says how far off it is: the first misses the fourth listing by 83,000 and the second by 249,000, while on the first listing the second comes closer. The function also returns a number for any area it is handed, so an area of minus 20 square meters, which no listing can have, came back as a price of minus 10,000.

Choosing between the two needs a loss (one number that scores a whole set of predictions against the known prices, smaller being better), which Lesson 3.1.3 takes apart. Fitting is the search for the parameters that make the loss as small as the function's form allows.

Not every number a model depends on is a parameter. A hyperparameter (a setting you choose before the fit, which the fit never changes) can decide what the fit is allowed to choose. Whether the formula has a b at all is one: drop it, and the fit chooses w alone, so every line it can return passes through a price of zero at an area of zero. The fit sets the parameters, and you set the hyperparameters.

A model also knows columns, not listings. All it sees of a listing is the feature values it is handed, so a price-on-area model gives every listing of the same area the same price, whatever its neighborhood, type or condition:

same_area = listings[listings["area_sqm"] == 63.2]
print(same_area[["neighborhood", "property_type", "price"]])
print(predict_price(same_area["area_sqm"].to_numpy(), 3000.0, 50000.0))

     neighborhood property_type    price
447     Riverside     apartment   235000
935      Downtown     apartment   295000
1745     Eastgate     townhouse  1255000
1834     Lakeview     apartment   312000
2129     Downtown     apartment   297000
2859     Downtown     apartment   309000
[239600. 239600. 239600. 239600. 239600. 239600.]

Six listings of 63.2 square meters are listed at between 235,000 and 1,255,000, and the model priced all six at 239,600. No choice of w and b separates them, because the only thing the model sees of each one is 63.2.

DON'T:

• DON'T take a returned price as evidence that the parameters are good. Both hand-set pairs priced all four listings, and only the known prices, through a loss, say which pair to keep.
• DON'T expect a model to separate listings its features cannot. Six listings of 63.2 square meters run from 235,000 to 1,255,000, and a model that sees only area gives all six one price.

The Estimator Interface

What. An estimator (scikit-learn's word for an object that learns from data in a fit method) is created with its hyperparameters, learns its parameters in .fit(X, y), and is then asked for predictions with .predict(X). LinearRegression, from sklearn.linear_model, is the estimator for w . x + b, with one weight per feature. Why. It is the shape Section 1.8 built by hand, settings at construction and learned state written by .fit() for .predict() to read. Every scikit-learn estimator takes its hyperparameters at construction and learns in fit, and every one that predicts does it with predict, so code written around one regression model fits another with only the line that creates it changed. When. For every model in this module: LinearRegression in this section, logistic regression in Section 3.4, trees and forests in Section 3.5.

The hyperparameters go in when the model is created, and printing the model shows them:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
model = LinearRegression()
through_zero = LinearRegression(fit_intercept=False)
explicit = LinearRegression(fit_intercept=True)

print(model)         # LinearRegression()
print(through_zero)  # LinearRegression(fit_intercept=False)
print(explicit)      # LinearRegression()
print(model.get_params())

{'copy_X': True, 'fit_intercept': True, 'n_jobs': None, 'positive': False, 'tol': 1e-06}

get_params() returns every hyperparameter with its value, as a dict, five for LinearRegression, and the one this lesson changes is fit_intercept, the choice between a line with a b and one without. The printed form names only the hyperparameters that differ from their defaults, so fit_intercept=True, the default, prints as nothing even when it is passed out loud.

X holds the features, one row per listing and one column per feature, and y holds the target, one price per listing. Before the fit, the model has nothing learned to show:

X = listings[["rooms"]]  # a DataFrame, shape (3000, 1)
y = listings["price"]    # a Series, shape (3000,)

try:
    print(model.coef_)
except AttributeError as err:
    print(err)
try:
    model.predict(X)
except ValueError as err:
    print(err)

'LinearRegression' object has no attribute 'coef_'
This LinearRegression instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.

coef_ does not exist yet, so reading it raises Python's own AttributeError. predict checks for a fit before it computes anything and raises NotFittedError, scikit-learn's error for an estimator used too early, which except ValueError catches because NotFittedError is a subclass of ValueError. The fit itself:

fitted = model.fit(X, y)
print(fitted is model)          # True
print(np.round(model.coef_))    # [77791.]
print(round(model.intercept_))  # 103178
print(model.n_features_in_)     # 1
print(model.feature_names_in_)  # ['rooms']
print(model)                    # LinearRegression()

fit changes the model it was called on and returns that same model, which is why LinearRegression().fit(X, y) on one line hands back a fitted model ready to assign. Every attribute the fit set ends in an underscore. coef_ holds one weight per feature column, the w, here 77,791 per room, intercept_ is the b, n_features_in_ counts the columns the fit saw, and feature_names_in_ records their names, because X was a DataFrame. The printed model reads the same after the fit as before it, because printing shows hyperparameters and never what was learned.

The trailing underscore is scikit-learn's public mark for "set by fit": an attribute meant to be read, which exists only once a fit has run. It is not Lesson 1.8.2's leading underscore, which scikit-learn keeps, as Section 1.8 did, for its own internals, the ones you are not meant to touch.

That makes X two-dimensional even with a single feature, a table with one column, and y one-dimensional. Lesson 2.4.1's two shapes decide it: listings[["rooms"]] is a DataFrame of shape (3000, 1), and listings["rooms"] is a Series of shape (3000,), which fit refuses as X:

try:
    LinearRegression().fit(listings["rooms"], y)
except ValueError as err:
    print(err)

Expected a 2-dimensional container but got <class 'pandas.Series'> instead. Pass a DataFrame containing a single row (i.e. single sample) or a single column (i.e. single feature) instead.

DON'T:

• DON'T pass a single column as a Series for X. fit refuses listings["rooms"], and listings[["rooms"]] is the one-column table it expects.
• DON'T read a learned attribute before the fit. coef_ raises AttributeError until fit has run, and predict raises NotFittedError.
• DON'T pass listings[["price"]] as y. Nothing raises, and the model treats it as a table of targets with one column: coef_ comes back as [[77791.19840795]], intercept_ as the array [103178.07368097], and each prediction as a row of a (n, 1) table where a Series target gives an array of shape (n,).

LinearRegression Is Least Squares

What. LinearRegression fits by least squares, Lesson 2.3.4's computation: it chooses the coef_ and intercept_ whose predictions leave the smallest sum of squared residuals on the training rows (the rows a model is fitted on). Its default, fit_intercept=True, plays the part of 2.3.4's column of ones. Why. A fit you can reproduce by hand has nothing hidden in its numbers. Every weight LinearRegression reports is the answer np.linalg.lstsq gives for the same columns, unless the column of ones is one of Lesson 2.3.5's collinear columns, a combination of the others, as it is beside the seven neighborhood columns Lesson 3.2.2 builds. What fit_intercept does is answered by adding or removing that column of ones. When. Every time you read a LinearRegression's coef_ and intercept_, or decide whether its line should pass through zero.

Area is the feature 2.3.4 fitted price on, and in listings.csv 125 listings have none:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
print(listings["area_sqm"].isna().sum())  # 125

try:
    LinearRegression().fit(listings[["area_sqm"]], listings["price"])
except ValueError as err:
    print(str(err).splitlines()[0])       # Input X contains NaN.

The message runs on to a second line, cut off here with splitlines, which names models that accept a missing value and suggests filling the gaps or dropping the rows. Lesson 3.2.3 fills such gaps, and this lesson drops the 125 listings with Lesson 2.5.1's dropna, which leaves 2,875:

has_area = listings.dropna(subset=["area_sqm"])
print(len(has_area))            # 2875

model = LinearRegression().fit(has_area[["area_sqm"]], has_area["price"])
print(np.round(model.coef_))    # [3288.]
print(round(model.intercept_))  # 54717

The fitted line prices a listing at 54,717 plus 3,288 per square meter. Lesson 2.3.4 found such weights with a column of ones beside the feature and np.linalg.lstsq, and on the same 2,875 listings that route finds these:

ones = np.ones(len(has_area))
design = np.column_stack([ones, has_area["area_sqm"]])
w, ssr, rank, sv = np.linalg.lstsq(design, has_area["price"])

print(np.round(w))             # [54717.  3288.]
fitted = np.array([model.intercept_, model.coef_[0]])
print(np.allclose(w, fitted))  # True

The weight on the column of ones is intercept_ and the weight on the area column is coef_, equal to within rounding, so the comparison is np.allclose. Fitting a LinearRegression is 2.3.4's projection, with fit_intercept=True standing in for the column of ones. Set it to False and the column is gone:

through_zero = LinearRegression(fit_intercept=False)
through_zero.fit(has_area[["area_sqm"]], has_area["price"])
print(np.round(through_zero.coef_))         # [3750.]
print(through_zero.intercept_)              # 0.0

area_col = has_area[["area_sqm"]]           # no column of ones
w0, ssr0, rank0, sv0 = np.linalg.lstsq(area_col, has_area["price"])
print(np.allclose(w0, through_zero.coef_))  # True

Without the column of ones, intercept_ is fixed at 0.0 and the line is pinned to a price of zero at an area of zero, 2.3.4's line through the origin, which lstsq on the area column alone reproduces. Pinned there, the line has to tilt more steeply to reach the prices, so the weight per square meter climbs from 3,288 to 3,750.

DON'T:

• DON'T fill the missing areas with 0 to get past the refusal. listings[["area_sqm"]].fillna(0) fits without a word and hands the fit 125 listings of zero square meters at their real prices, which drops the weight from 3,288 to 2,698 per square meter and more than doubles the intercept, to 124,723.
• DON'T set fit_intercept=False because a listing with no area should cost nothing. The smallest area in the file is 24.0 square meters, so no listing sits near the point the line is pinned to, and pinning it there tilts the line through every listing that exists, 3,750 per square meter against 3,288.

Predicting on New Rows

What. .predict(X) applies the fitted parameters to each row of X and returns one prediction per row, as a NumPy array: intercept_ plus each feature value times its weight in coef_. Why. A model exists to price listings whose price nobody knows yet, and predict is the call that does it, for one listing from a form or for a whole batch at once. When. After the fit, whenever a listing the fit never saw needs a price.

A new listing is a DataFrame with the column the fit saw, one row per listing:

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
model = LinearRegression().fit(has_area[["area_sqm"]], has_area["price"])

one = pd.DataFrame({"area_sqm": [85.0]})
batch = pd.DataFrame({"area_sqm": [45.0, 85.0, 140.0]})

print(np.round(model.predict(one)))    # [334224.]
print(np.round(model.predict(batch)))  # [202691. 334224. 515081.]
by_hand = model.intercept_ + model.coef_[0] * 85.0
print(round(by_hand))                  # 334224

One listing or three, the answer is an array. Computed by hand from intercept_ and coef_, the listing of 85 square meters gets the same 334,224, which is all predict does for this model.

The column must be the one the fit saw. A model fitted on a DataFrame recorded feature_names_in_, and predict compares the new rows against it. A NumPy array has no column names to compare, so it is priced with a warning:

as_array = np.array([[85.0]])  # the same listing, with no column name
print(np.round(model.predict(as_array)))

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/sklearn/utils/validation.py:2830: UserWarning: X does not have valid feature names, but LinearRegression was fitted with feature names
  warnings.warn(
[334224.]

The warning's first line names the file it came from, scikit-learn's own validation.py rather than your script, and its second line shows the line of that file, warnings.warn(. The price is right this time, because the array's one column is the area. A column renamed between the fit and the prediction is refused, with both names in the message:

renamed = pd.DataFrame({"area": [85.0]})
try:
    model.predict(renamed)
except ValueError as err:
    print(err)

The feature names should match those that were passed during fit.
Feature names unseen at fit time:
- area
Feature names seen at fit time, yet now missing:
- area_sqm

The message ends with a line break of its own, so print leaves an empty line after it. Order counts as much as names. A model on two features, area and the distance to the city center, refuses its two columns in the other order, and selecting the columns by the names it recorded, feature_names_in_, puts them back:

features = ["area_sqm", "distance_km"]
two = LinearRegression().fit(has_area[features], has_area["price"])
print(np.round(two.coef_))  # [  3549. -13267.]

swapped = pd.DataFrame({"distance_km": [3.0], "area_sqm": [85.0]})
try:
    two.predict(swapped)
except ValueError as err:
    print(err)
print(np.round(two.predict(swapped[two.feature_names_in_])))

The feature names should match those that were passed during fit.
Feature names must be in the same order as they were in fit.

[363672.]

The same swap in a NumPy array leaves predict no names to check. It multiplies the weights by the values in the order they arrive, Lesson 2.3.1's two vectors that disagree about the order of the features:

# distance read as area, area read as distance
print(np.round(two.predict(swapped.to_numpy())))

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/sklearn/utils/validation.py:2830: UserWarning: X does not have valid feature names, but LinearRegression was fitted with feature names
  warnings.warn(
[-1015259.]

A listing of 85 square meters, 3 kilometers out, is priced at minus 1,015,259 under the very warning the correctly ordered array drew. The warning reports the missing names and says nothing about the order, so it reads the same over a right price and a wrong one.

predict checks names, order and the number of columns, and refuses a missing value, but it never asks whether a value is like anything the fit saw. Like predict_price, it returns a number for any area it is handed:

extremes = pd.DataFrame({"area_sqm": [0.0, 2000.0]})
print(np.round(model.predict(extremes)))  # [  54717. 6631346.]
print(has_area["area_sqm"].max())         # 278.5
print(listings["price"].max())            # 1930000

An area of 0 returns intercept_ itself, 54,717, a price for a listing that cannot exist. An area of 2,000 square meters, seven times the largest in the file, returns more than 6.6 million, over three times the most expensive listing, and nothing warns: the line runs on past the data at the same slope, and nothing in the model records where the data stopped.

DON'T:

• DON'T hand predict a NumPy array for a model fitted on a DataFrame. The array draws a warning about missing names and nothing more, the same warning over the right price and over the minus 1,015,259 the swapped columns produced.
• DON'T silence a refused DataFrame by converting it to an array. The refusal caught a renamed or reordered column: new[model.feature_names_in_] puts reordered columns back in the fitted order, and a renamed column needs its fitted name back.
• DON'T trust a prediction for a listing unlike any the fit saw. The model priced 2,000 square meters at more than 6.6 million without a warning, from a line fitted on areas of 24.0 to 278.5.

Lab Exercises

Lab Exercise 1: Bathrooms Through the Estimator Interface

With (bootcamp) active, confirm with a one-line check that scikit-learn imports at 1.9.1 and SciPy at 1.18.1.

Then, in bath_model.py, load listings.csv and create a LinearRegression. Print the value of its fit_intercept hyperparameter from get_params(), then read intercept_ inside a try that catches the error and prints its message. Fit price on bathrooms with the column passed as a Series, inside a try as well, and print that message.

Then fit price on bathrooms the right way, creating and fitting the model on one line. Print n_features_in_ and feature_names_in_, the weight in whole currency units and the intercept rounded to a whole number. Finally, write a function price_for(bathrooms) that computes a prediction from those two learned numbers alone, without calling predict, print its price for 1, 2 and 3 bathrooms, and say in a comment which of the numbers you printed are parameters and which is a hyperparameter.

Lab Exercise 2: Price on Area and Construction Year

In area_year.py, fit price on two features, area_sqm and year_built. First try it on every listing inside a try, printing only the first line of the message. Then drop the listings missing either column, print how many remain, fit, and print the two weights and the intercept in whole currency units.

Check the fit against np.linalg.lstsq on a design matrix of a column of ones, the areas and the years, and print the result of the comparison. Add a comment saying what listing the intercept would be the price of. Then price two new listings from one DataFrame, 95.0 square meters built in 1985 and 140.0 square meters built in 2018. Build the same two listings with their columns in the order year_built, area_sqm, print the message predict raises for them, and price them again by selecting the fitted columns.

# 1.9.1 1.18.1
python3 -c "import sklearn, scipy; print(sklearn.__version__, scipy.__version__)"

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")

model = LinearRegression()
print(model.get_params()["fit_intercept"])  # True
try:
    print(model.intercept_)
except AttributeError as err:
    print(err)
try:
    model.fit(listings["bathrooms"], listings["price"])
except ValueError as err:
    print(err)

'LinearRegression' object has no attribute 'intercept_'
Expected a 2-dimensional container but got <class 'pandas.Series'> instead. Pass a DataFrame containing a single row (i.e. single sample) or a single column (i.e. single feature) instead.

model = LinearRegression().fit(listings[["bathrooms"]], listings["price"])
print(model.n_features_in_)     # 1
print(model.feature_names_in_)  # ['bathrooms']
print(np.round(model.coef_))    # [136482.]
print(round(model.intercept_))  # 147970

def price_for(bathrooms):
    return model.intercept_ + model.coef_[0] * bathrooms

for n in [1, 2, 3]:
    print(n, round(price_for(n)))

# fit_intercept is a hyperparameter, chosen before the fit.
# coef_ and intercept_ are the parameters, the w and b it chose.

1 284452
2 420934
3 557416

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
features = ["area_sqm", "year_built"]

try:
    LinearRegression().fit(listings[features], listings["price"])
except ValueError as err:
    print(str(err).splitlines()[0])  # Input X contains NaN.

both = listings.dropna(subset=features)
print(len(both))                     # 2739

model = LinearRegression().fit(both[features], both["price"])
print(np.round(model.coef_))         # [3240.  936.]
print(round(model.intercept_))       # -1803629

ones = np.ones(len(both))
design = np.column_stack([ones, both["area_sqm"], both["year_built"]])
w, ssr, rank, sv = np.linalg.lstsq(design, both["price"])
fitted = np.array([model.intercept_, model.coef_[0], model.coef_[1]])
print(np.allclose(w, fitted))        # True

# -1803629 prices a listing of zero area built in year 0, which no
# listing is. A real year, 1990 for one, adds 936 times 1990, about
# 1.86 million, back through the year weight.

new = pd.DataFrame({"area_sqm": [95.0, 140.0], "year_built": [1985.0, 2018.0]})
print(np.round(model.predict(new)))  # [361856. 538525.]

reordered = new[["year_built", "area_sqm"]]
try:
    model.predict(reordered)
except ValueError as err:
    print(err)
print(np.round(model.predict(reordered[model.feature_names_in_])))

The feature names should match those that were passed during fit.
Feature names must be in the same order as they were in fit.

[361856. 538525.]

Theory Assessment

Q1: Parameters, Hyperparameters, and Six Listings at One Price

In the price-on-area model w * area + b, name the parameters, name a hyperparameter of the same model, and say which of the two the fit sets. Then explain why six listings of 63.2 square meters, listed at between 235,000 and 1,255,000, get one price from that model whatever w and b are.

Hint: For the first part, ask which numbers can only be known once the listings have been seen, and which choice has to be made before any listing is. For the second, ask what the model is handed for each of the six listings.

Q2: What the Fit Returns, and What the Underscore Promises

Explain why model = LinearRegression().fit(X, y) leaves a fitted model in model, what the trailing underscore on coef_, intercept_, n_features_in_ and feature_names_in_ tells you, and what reading coef_ and calling predict each raise before the fit. Then say why X must be listings[["rooms"]] and not listings["rooms"].

Hint: Look at what fit returns, and at when each attribute comes to exist. For the shapes, ask how many dimensions a table of features has when it holds a single feature.

Q3: The Column of Ones, Kept and Removed

Explain how np.linalg.lstsq on a column of ones beside the area column reproduces the intercept_ and coef_ of LinearRegression() on the same 2,875 listings, and why the check is np.allclose. Then explain what fit_intercept=False removes, what intercept_ becomes, and why the weight per square meter rose from 3,288 to 3,750.

Hint: Match each weight lstsq returned to the column it multiplies. For the second part, ask where the line has to pass once the column of ones is gone, and what it must then do to reach listings of a hundred square meters at their prices.

Q4: Two Models, One Object

Read this script without running it, then say what each print shows, using the numbers this lesson printed.

import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
X = has_area[["area_sqm"]]
y = has_area["price"]

a = LinearRegression().fit(X, y)
b = LinearRegression(fit_intercept=False)
c = b.fit(X, y)
zero = pd.DataFrame({"area_sqm": [0.0]})

print(a)
print(c is b, c)
print(round(a.predict(zero)[0]), round(c.predict(zero)[0]))
print(a.coef_[0] < c.coef_[0])

A teammate concludes from the second line that c is a fitted copy of b and that b is still unfitted, and from the last line that the model through zero is the better of the two, since it values a square meter more. Explain where each conclusion goes wrong.

Hint: Ask what fit returns and whether it builds anything new. For the last line, ask where c's line is pinned and what pinning does to its slope, then ask which of the two fits chose from more lines.

LinearRegression()
True LinearRegression(fit_intercept=False)
54717 0
True

ssr_a = ((y - a.predict(X)) ** 2).sum()
ssr_c = ((y - c.predict(X)) ** 2).sum()
print(f"{ssr_a:.4e} {ssr_c:.4e}")  # 4.9523e+13 5.0946e+13

Assignment

Assignment: A Price-per-Square-Meter Estimator That scikit-learn Recognizes

Your team's pricing code takes any scikit-learn regression model, prints it, reads its hyperparameters with get_params(), and calls fit and predict on it. A colleague prices listings by one number instead, the typical price per square meter of past listings times the new listing's area, and wants that rule to run through the same code.

Write per_sqm.py with a class PricePerSqm, an estimator in Section 1.8's shape that keeps every convention this lesson met. Its one hyperparameter, statistic, is "median" unless set to "mean". fit(X, y) takes a one-column DataFrame of areas and the matching prices, learns price_per_sqm_, that statistic of each training listing's price divided by its area, records n_features_in_ and feature_names_in_ as LinearRegression does, and returns the estimator. predict(X) returns each row's area times price_per_sqm_ as a NumPy array, refuses an estimator that was never fitted, and refuses a DataFrame whose column names differ from the fitted ones with a ValueError naming both.

Two pieces of scikit-learn do part of the work. BaseEstimator, from sklearn.base, is a class to inherit from: it reads the parameter names of your __init__ and gives the class the get_params() and the printed form LinearRegression has, provided __init__ stores each argument, unchanged, as an attribute of the same name. check_is_fitted(estimator), from sklearn.utils.validation, raises the NotFittedError that LinearRegression.predict raised before its fit, unless the estimator holds an attribute whose name ends in an underscore.

On the listings that have an area, show, in this order:

• The printed form of PricePerSqm() and of PricePerSqm(statistic="mean"), and the first one's hyperparameters.
• The message a prediction raises before the fit, and that fit returns the estimator itself.
• The price per square meter each statistic learns, beside the weight of LinearRegression(fit_intercept=False) on the same rows.
• The prices your two models and LinearRegression, with and without its intercept, give listings of 45, 85 and 140 square meters, each line led by the printed model.
• The message a column named area draws.

End with a comment saying which of the four models share one form, and what separates them.

Hints:

• BaseEstimator looks each hyperparameter up by its argument's name, so __init__ holds self.statistic = statistic and nothing more, and everything the data decides is set in fit, where check_is_fitted will find it.
• check_is_fitted(self) on the first line of predict does the refusing, and comparing list(X.columns) with list(self.feature_names_in_) is the column test. A loop over the four fitted models prints each one beside its prices.

import numpy as np
import pandas as pd
from sklearn.base import BaseEstimator
from sklearn.linear_model import LinearRegression
from sklearn.utils.validation import check_is_fitted

class PricePerSqm(BaseEstimator):
    def __init__(self, statistic="median"):
        self.statistic = statistic   # stored as given, nothing else

    def fit(self, X, y):
        per_sqm = y / X["area_sqm"]  # one value per training listing
        if self.statistic == "median":
            self.price_per_sqm_ = per_sqm.median()
        else:
            self.price_per_sqm_ = per_sqm.mean()
        self.n_features_in_ = X.shape[1]
        self.feature_names_in_ = np.array(X.columns)
        return self

    def predict(self, X):
        check_is_fitted(self)        # NotFittedError until fit has run
        fitted = list(self.feature_names_in_)
        given = list(X.columns)
        if given != fitted:
            raise ValueError(f"fitted on {fitted}, got {given}")
        return X["area_sqm"].to_numpy() * self.price_per_sqm_

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
X = has_area[["area_sqm"]]
y = has_area["price"]

print(PricePerSqm())                       # PricePerSqm()
print(PricePerSqm(statistic="mean"))       # PricePerSqm(statistic='mean')
print(PricePerSqm().get_params())          # {'statistic': 'median'}

median_model = PricePerSqm()
try:
    median_model.predict(X)
except ValueError as err:
    print(err)

# True, fit returned the estimator it changed
print(median_model.fit(X, y) is median_model)
mean_model = PricePerSqm(statistic="mean").fit(X, y)
through_zero = LinearRegression(fit_intercept=False).fit(X, y)
with_b = LinearRegression().fit(X, y)

print(round(median_model.price_per_sqm_))  # 3746
print(round(mean_model.price_per_sqm_))    # 3935
print(round(through_zero.coef_[0]))        # 3750

new = pd.DataFrame({"area_sqm": [45.0, 85.0, 140.0]})
for model in [median_model, mean_model, through_zero, with_b]:
    print(model, np.round(model.predict(new)))

try:
    median_model.predict(pd.DataFrame({"area": [85.0]}))
except ValueError as err:
    print(err)

# PricePerSqm(), PricePerSqm(statistic='mean') and
# LinearRegression(fit_intercept=False) share one form, a line through
# zero, price = k * area, and differ only in the rule that chose k:
# a median, a mean, least squares. LinearRegression() adds a b.

This PricePerSqm instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
PricePerSqm() [168562. 318395. 524415.]
PricePerSqm(statistic='mean') [177089. 334502. 550944.]
LinearRegression(fit_intercept=False) [168731. 318714. 524941.]
LinearRegression() [202691. 334224. 515081.]
fitted on ['area_sqm'], got ['area']

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want to write estimators of your own, or to meet a LinearRegression hyperparameter this lesson printed and never set.

• Official docs - scikit-learn 1.9, "Developing scikit-learn estimators", the section "Rolling your own estimator". The rules a home-made estimator follows to work with the rest of the library, and check_estimator, which tests an estimator against them.
• Official docs - scikit-learn 1.9 User Guide, "Linear Models", the section "Non-Negative Least Squares". LinearRegression(positive=True), which holds every weight in coef_ at zero or above, for quantities that cannot be negative.

Summary

A model turned out to be a small thing: a function whose form you choose and whose numbers the data chooses. Everything that looked like learning here was a fit choosing a few numbers, and everything that looked like prediction was those numbers applied to a new row. The numbers were chosen by least squares, the computation Module 2 carried out by hand, now behind one call, and the column of ones that once had to be built became a hyperparameter you switch on or off.

The interface is Section 1.8's shape made strict, and its strictness is where its protection lives. It refuses a Series where a table belongs, a missing value, an attribute read too early and a column renamed or moved, and each refusal says what it caught. What it lets through is the more dangerous half: an array whose columns are in the wrong order, and a listing far outside anything the fit saw, priced under a warning that misses the point or under no warning at all.

What this lesson never asked is whether a fitted line is any good. Every number it printed came from the rows the fit was chosen on, and a model judged on those rows is judged on questions it was shown the answers to. Holding rows back is the next lesson's work, and what a fit is really minimizing, and how to score what it predicts, are the two after it.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
rows = listings.dropna(subset=["area_sqm"])  # LinearRegression refuses NaN
model = LinearRegression().fit(rows[["area_sqm"]], rows["price"])
model.coef_, model.intercept_                # set by fit, trailing _
model.get_params()                           # hyperparameters, a dict
print(model)                                 # non-default ones only

new = pd.DataFrame({"area_sqm": [85.0]})
model.predict(new)                           # array, one price per row
model.predict(new[model.feature_names_in_])  # fitted columns, fitted order

ones = np.ones(len(rows))
design = np.column_stack([ones, rows["area_sqm"]])
# w is [intercept_, coef_[0]], up to rounding
w, ssr, rank, sv = np.linalg.lstsq(design, rows["price"])