From Neuron to Network - the MLP

Context

A multilayer perceptron is Lesson 4.2.1's neuron repeated in layers: every one of a listing's 18 columns feeds every unit of a hidden layer of 32, a function between the layers turns each negative sum into a zero, and one last unit adds the 32 results into a price. nn.Sequential holds the three in order, and Lesson 4.2.2's loop trains it with the model line changed and nothing else.

Every network this module trains is this shape or ends in it. Lesson 4.3.2's classifier is the same three lines on 19 columns with the loss swapped, Section 4.4's regularized models are this network with a tool added, and Section 4.5's image model finishes in layers like these. The table it reads is the one Module 3 built, standardized numeric columns beside one-hot categories, so a neighborhood reaches the first layer as six zeros and a one.

Without the function between the layers, two layers are one, 641 parameters doing the work of 19, and a run shows the collapse. Without the target scaled, the loss is inf by the third epoch at 0.1, Section 4.2's rate. And a hidden weight read as a price per square meter means nothing. Lesson 4.2.3 scored one neuron's line at LinearRegression's number. This lesson hands the same loop a network on 18 columns and watches it beat the line fitted on those columns, by 7,140 of MAE on the validation rows.

Goal

• Build the table a network reads as one prep function in prep_net.py, the fit rows' statistics and a one-hot encoder fitted on them, returning an 18-column float32 tensor and a scaled (n, 1) target, and read one row back.
• Show from a run what the raw price does to a network's loss at 0.1 and to one layer, and multiply every prediction back before scoring it.
• Build nn.Sequential(nn.Linear(18, 32), nn.ReLU(), nn.Linear(32, 1)), trace a batch's shape through its three modules, and collapse two linear layers with nothing between them into one with W2 @ W1.
• Train the network with Section 4.2's loop, saved as train_loop.py with the network's constructor and the score for the rest of the section to import, score it against nn.Linear(18, 1) and LinearRegression on the validation rows, and state what the hidden layer bought.
• Read a network's parameters by name, shape and count, distinguish a linear weight that reads back in currency from a hidden weight that does not, and inspect the hidden layer's representation of a listing.
• Fit four widths and read what more width does to the training loss, the validation loss and the seconds.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want the picture of a network before you build one in code.

• YouTube - "But what is a neural network? | Deep learning chapter 1" by 3Blue1Brown. Eighteen and a half minutes on the shape this lesson writes in three lines: layers of units, each a weighted sum plus a bias, a whole layer's sums as one matrix product, a function squashing each sum, and two hidden layers between an image's pixels and ten outputs, on handwritten digits. Its squashing function is the sigmoid, and it names ReLU as the one in use today.
• Official docs - PyTorch 2.14, torch.nn.Sequential. The container this lesson's network lives in: modules run in the order they are given, each output handed to the next module as its input, the whole treated as one module, with append, extend, insert and pop for changing the chain after it is built.

Table of Contents

• The Table a Network Reads
• Why the Target Is Scaled Too
• Stacking Layers, and Why the Nonlinearity Is Not Optional
• The MLP, Fitted and Compared
• Reading a Network's Parameters
• Width, Previewed

Theory

Every block in this lesson runs in ~/dev/bootcamp on the 2,875 listings with an area, split as Lesson 4.2.3 split them, 1,725 fit rows, 575 validation rows and 575 test rows. The loop is Lesson 4.2.2's five steps with Lesson 4.2.3's validation pass under both switches, nn.MSELoss() and optim.SGD at 0.1. The first subtitle writes prep_net.py and the fourth writes train_loop.py, which the rest of the section imports, as Lesson 4.2.3's scripts imported prep.py. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

The Table a Network Reads

What. The table is Module 3's: the four numeric columns area_sqm, rooms, bathrooms and distance_km standardized with the fit rows' mean and std(ddof=0), beside the 14 columns OneHotEncoder writes for neighborhood, property_type and condition, fitted on the fit rows, stacked into one float32 tensor of 18 columns. prep(frame) builds it for any set of rows, with the target beside it. Why. A layer computes X @ W.T + b, and a matrix product reads numbers, Lesson 3.2.2's rule for LinearRegression unchanged for a network: a listing's neighborhood reaches the first layer as six zeros and a one, and its area reaches it on the scale the other three numeric columns share, Lesson 3.2.1's rule in tensor form. When. Before any model is built, once, as a function, because every set of rows the network ever sees goes through the same four means, the same four standard deviations and the same 14 categories.

Save it as prep_net.py in ~/dev/bootcamp:

import numpy as np
import pandas as pd
import torch
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import OneHotEncoder

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
train_rows, test_rows = train_test_split(has_area, test_size=0.2, random_state=42)
fit_rows, val_rows = train_test_split(train_rows, test_size=0.25, random_state=42)

numeric = ["area_sqm", "rooms", "bathrooms", "distance_km"]
text = ["neighborhood", "property_type", "condition"]
# the fit rows' statistics and categories, applied to every set
mu, sd = fit_rows[numeric].mean(), fit_rows[numeric].std(ddof=0)
enc = OneHotEncoder(sparse_output=False, handle_unknown="ignore").fit(fit_rows[text])

def prep(frame):
    scaled = ((frame[numeric] - mu) / sd).to_numpy()
    hot = enc.transform(frame[text])
    X = torch.tensor(np.hstack([scaled, hot]), dtype=torch.float32)
    y = torch.tensor(frame["price"].to_numpy() / 100_000, dtype=torch.float32)
    return X, y.unsqueeze(1)

X_fit, y_fit = prep(fit_rows)
X_val, y_val = prep(val_rows)
X_test, y_test = prep(test_rows)

if __name__ == "__main__":
    print(len(fit_rows), len(val_rows), len(test_rows))
    print(X_fit.shape, X_fit.dtype, y_fit.shape, y_fit.dtype)
    print(enc.get_feature_names_out())

python3 prep_net.py prints the three lines under its main guard, and an import prep_net runs everything above the guard in silence, as prep.py did:

1725 575 575
torch.Size([1725, 18]) torch.float32 torch.Size([1725, 1]) torch.float32
['neighborhood_Downtown' 'neighborhood_Eastgate' 'neighborhood_Hillcrest'
 'neighborhood_Lakeview' 'neighborhood_Northend' 'neighborhood_Riverside'
 'neighborhood_Southgate' 'property_type_apartment' 'property_type_house'
 'property_type_townhouse' 'condition_excellent' 'condition_fair'
 'condition_good' 'condition_poor']

Splitting the frame instead of two arrays gives Lesson 4.2.3's rows, because train_test_split shuffles by the row count and the seed, so the fit rows' mean area is still 99.44. mu and sd are four numbers each, pandas Series indexed by column name, with ddof=0 as Lesson 4.2.1's correction=0. enc learned 14 categories from the fit rows alone, Lesson 3.2.1's rule for every transformer, with handle_unknown="ignore", Lesson 3.2.2's setting that turns a category the fit rows never had into a row of zeros. Inside prep, pandas aligns mu and sd with the four columns by name, enc.transform writes the 14 columns in the order printed above, and np.hstack (NumPy's horizontal stack, which puts arrays side by side as the columns of one array) joins the two blocks. The target is a (n, 1) float32 tensor, Lesson 4.2.2's shape. One row, read back, from a script that imports the file:

from prep_net import has_area, fit_rows, numeric, text, X_fit, y_fit, X_val

print(fit_rows[numeric + text + ["price"]].iloc[0].to_dict())
print(X_fit[0])
print(y_fit[:3].squeeze(1), round(y_fit.mean().item(), 3))

{'area_sqm': 59.5, 'rooms': 3, 'bathrooms': 1, 'distance_km': 8.6, 'neighborhood': 'Southgate', 'property_type': 'apartment', 'condition': 'excellent', 'price': 272000}
tensor([-0.8938, -0.3373, -0.8639,  1.0243,  0.0000,  0.0000,  0.0000,  0.0000,
         0.0000,  0.0000,  1.0000,  1.0000,  0.0000,  0.0000,  1.0000,  0.0000,
         0.0000,  0.0000])
tensor([2.7200, 2.2600, 3.1500]) 3.848

In X_fit[0] the first four numbers are the four columns standardized, -0.8938 for 59.5 against a mean of 99.44 and a standard deviation of 44.69, and 1.0243 for 8.6 kilometers against 5.52 and 3.01, then seven neighborhood columns with the 1 in the seventh, Southgate, last alphabetically, three property-type columns with the 1 in the first, apartment, and four condition columns with the 1 in the first, excellent. The target's first three values are 272,000, 226,000 and 315,000 in hundreds of thousands, and its mean is 3.848, the fit rows' mean price of 384,795 on the same scale. What the statistics did to each set, and what the two columns not in the table carry:

print(X_fit[:, :4].std(correction=0, dim=0))
print(X_val[:, :4].mean(dim=0))
print(X_val[:, 4:11].sum(dim=1).unique(), X_val[:, 14:].sum(dim=1).unique())
print(has_area[["year_built", "energy_rating"]].isna().sum().to_dict())

tensor([1., 1., 1., 1.])
tensor([-0.0762, -0.0730, -0.0819, -0.0625])
tensor([1.]) tensor([1.])
{'year_built': 136, 'energy_rating': 257}

The fit rows' four columns have a standard deviation of exactly one, because the numbers came from them, and the validation rows' means sit near zero and not at it, -0.0762 for the area exactly as Lesson 4.2.3 printed it, because those rows were transformed with numbers that know nothing about them. Every validation row's seven neighborhood columns sum to one, and so do its four condition columns, one category per listing and each already among the fit rows', so handle_unknown changed nothing here. year_built and energy_rating stay out because 136 and 257 of the 2,875 listings have no value in them, and filling a gap is Lesson 3.2.3's imputation, not this section's subject.

DON'T:

• DON'T hand a network a text column. torch.tensor refused numpy.object_ in Lesson 4.1.1, and a neighborhood reaches X @ W.T + b only as the seven columns enc writes.
• DON'T fit mu, sd or enc on has_area. The 1,150 held-out listings' areas and categories then enter numbers the fit uses, Lesson 3.1.2's third break, with nothing raised.

Why the Target Is Scaled Too

What. The target enters the network as price / 100_000, and a prediction is multiplied back by 100,000 before mean_absolute_error or r2_score reads it. A loss on this target prints with four decimals, f"{loss.item():.4f}", where Section 4.2's printed in .4e. Why. On the raw price the network's loss is inf by the third epoch at 0.1 and nan from the fifth, while nn.Linear(18, 1) fits the raw price at 0.1 as one neuron did throughout Section 4.2, both from the runs below. Lesson 3.2.1's sentence, that a fit by small steps cares about scale, reaches the output side the moment a hidden layer stands between the input and the answer. When. Every regression target this module hands a network. A classifier's target, Lesson 4.3.2's zeros and ones, needs no scaling.

import torch
from torch import nn, optim
from prep_net import X_fit, y_fit

y_fit_raw = y_fit * 100_000
loss_fn = nn.MSELoss()
torch.manual_seed(42)
net = nn.Sequential(nn.Linear(18, 32), nn.ReLU(), nn.Linear(32, 1))
optimizer = optim.SGD(net.parameters(), lr=0.1)
for epoch in range(1, 6):
    loss = loss_fn(net(X_fit), y_fit_raw)
    loss.backward()
    # the loss, the largest output weight, the largest output gradient
    print(epoch, f"{loss.item():.4f}", f"{net[2].weight.abs().max().item():.4e}", f"{net[2].weight.grad.abs().max().item():.4e}")
    optimizer.step()
    optimizer.zero_grad()

1 190757453824.0000 1.7292e-01 3.0150e+05
2 1415914378049355776.0000 3.0150e+04 4.3301e+13
3 inf 4.3301e+12 9.7465e+37
4 inf 9.7465e+36 nan
5 nan nan nan

net is the next subtitle's network, three modules in nn.Sequential (a module that runs the modules it holds in order, each output the next one's input), taken as a box here and opened there, and net[2] is its last layer. The first loss is the mean squared price, 1.9076e+11 in Section 4.2's notation, and the gradient on the output weights is in the hundreds of thousands, because the derivative of a squared error carries the residual, Lesson 4.1.3's -2 * r, and every residual is a price. One step at 0.1 takes the output weights from at most 0.17 to as much as 30,150. At the second epoch every hidden gradient is multiplied by those weights on its way back through the layer above, Lesson 4.1.3's chain rule, the output gradient is 4.3301e+13, the step puts the weights at 4.3301e+12, and the third loss is past what float32 can hold, inf. A gradient computed from inf is nan, and from epoch 5 the loss and every output weight are nan for as long as the loop runs, with nothing raised. The same raw price under one layer:

torch.manual_seed(42)
one = nn.Linear(18, 1)
optimizer = optim.SGD(one.parameters(), lr=0.1)
for epoch in range(1, 501):
    loss = loss_fn(one(X_fit), y_fit_raw)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if epoch in (1, 10, 100, 500):
        print(epoch, f"{loss.item():.4e}")
print(f"{one.weight[0, 0].item():,.0f}")

1 1.9076e+11
10 1.6728e+10
100 8.9699e+09
500 8.8441e+09
168,308

One layer fits the raw price at 0.1 as Lesson 4.2.3's one neuron did, settling at 8.8441e+09 with its area weight at 168,308 per standard deviation: a linear model's weights are prices, up to the hundreds of thousands themselves, and steps of tens of thousands are the road to them. The network's output layer took a step of the same size, tens of thousands, and then handed it to 32 hidden units as a multiplier on their gradients. A rate the network survives on the raw price exists:

torch.manual_seed(42)
net = nn.Sequential(nn.Linear(18, 32), nn.ReLU(), nn.Linear(32, 1))
optimizer = optim.SGD(net.parameters(), lr=1e-7)
for epoch in range(1, 501):
    loss = loss_fn(net(X_fit), y_fit_raw)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if epoch in (1, 10, 100, 500):
        print(epoch, f"{loss.item():.4e}")

1 1.9076e+11
10 1.9076e+11
100 1.6130e+10
500 8.7165e+09

At 1e-7, a million times smaller than the section's rate, the network survives, sits at its first loss for ten epochs, and reaches 8.7165e+09 by epoch 500 with the loss still falling, Lesson 4.2.1's crawl on the raw input met again on the raw output. From this start 1e-6 survives too, and every rate from 0.01 down to 1e-5 prints nan by epoch 10. The same network on the scaled target, at the section's rate:

torch.manual_seed(42)
net = nn.Sequential(nn.Linear(18, 32), nn.ReLU(), nn.Linear(32, 1))
optimizer = optim.SGD(net.parameters(), lr=0.1)
for epoch in range(1, 6):
    loss = loss_fn(net(X_fit), y_fit)
    loss.backward()
    print(epoch, f"{loss.item():.4f}", f"{net[2].weight.abs().max().item():.4f}", f"{net[2].weight.grad.abs().max().item():.4f}")
    optimizer.step()
    optimizer.zero_grad()

1 19.5549 0.1729 3.0288
2 9.7497 0.4370 2.8790
3 4.2237 0.7249 2.0846
4 2.7399 0.9334 0.5853
5 2.3499 0.8748 0.4920

The first loss is 19.5549 in hundreds of thousands squared, the gradients are single digits and the weights move by tenths, so the loss falls by a factor of eight in five epochs at the section's rate, and the fourth subtitle runs it out. Dividing the target by a constant changes nothing the model has to learn.

DON'T:

• DON'T hand a network the price in currency. At 0.1 its loss is inf by epoch 3, and a rate that survives, 1e-7, crawls.
• DON'T read a prediction without multiplying it back. The fitted network's validation predictions score 366,109 of MAE and an R² of -4.27 before the factor, and 45,585 and 0.859 after it.

Stacking Layers, and Why the Nonlinearity Is Not Optional

What. A layer (one nn.Linear, every input feeding every unit, X @ W.T + b with one row of W per unit) of 32 units, then ReLU (the rectified linear unit, max(0, x), zero for a negative input and the input itself otherwise), then a layer of one unit. The first layer is a hidden layer (a layer whose outputs are not the answer but the next layer's input), each of its units a hidden unit (one neuron of a hidden layer, Lesson 4.2.1's weighted sum plus a bias, on all 18 columns), and the whole is an MLP (a multilayer perceptron, linear layers with a nonlinearity between each pair). The nonlinearity (a function between two layers that is not a matrix product, so the two cannot merge into one), also called the activation function, is what makes the stack more than one layer. Why. Two linear layers with nothing between them are one linear layer, W2 @ (W1 @ x + b1) + b2 is (W2 @ W1) @ x + (W2 @ b1 + b2), and a run shows the collapse. ReLU between them is one line of code and the whole difference between a 641-parameter line and a model that can bend. When. From here to the end of the module, every network is layers with a nonlinearity between each pair, and the last layer alone has nothing after it, because its output is the answer: a price here, a logit in Lesson 4.3.2.

import torch
from torch import nn
from prep_net import X_fit

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(18, 32), nn.ReLU(), nn.Linear(32, 1))
print(model)

Sequential(
  (0): Linear(in_features=18, out_features=32, bias=True)
  (1): ReLU()
  (2): Linear(in_features=32, out_features=1, bias=True)
)

print(len(model), model[1])  # 3 ReLU()
# [0.1802, 0.1956, -0.0552]
print([round(v, 4) for v in model[0].weight[0, :3].tolist()])
# [-0.0686, 0.1259, -0.1163]
print([round(v, 4) for v in model[0].bias[:3].tolist()])

nn.Sequential numbers its modules from 0, so model[0] is the first layer, model[1] the ReLU and model[2] the last layer. The seed immediately before the constructor fixes every draw, the same on every run of this release on a processor, drawn between -1 / sqrt(18) and 1 / sqrt(18) as Lesson 4.2.2's layer drew between -1 and 1. ReLU() holds no parameter. A batch of four fit rows through the three, one module at a time:

# tensor([0., 0., 3.])
print(torch.relu(torch.tensor([-2.0, 0.0, 3.0])))
four = X_fit[:4]
hidden = model[0](four)
active = model[1](hidden)
out = model[2](active)
# torch.Size([4, 18]) torch.Size([4, 32]) torch.Size([4, 32]) torch.Size([4, 1])
print(four.shape, hidden.shape, active.shape, out.shape)
# 74 74
print((hidden < 0).sum().item(), (active == 0).sum().item())
print(torch.equal(model(four), out))  # True

torch.relu is the function and nn.ReLU() the module that calls it. The first layer writes one number per hidden unit, Lesson 4.1.2's X @ w + b with 32 weight vectors as the rows of W, ReLU turns exactly the negative ones into zeros and keeps the shape, the last layer turns each row's 32 numbers into one, the price in hundreds of thousands, and model(four) is the three calls in order. What the ReLU is for, shown by taking it out:

torch.manual_seed(42)
stack = nn.Sequential(nn.Linear(18, 32), nn.Linear(32, 1))
W1, b1 = stack[0].weight, stack[0].bias
W2, b2 = stack[1].weight, stack[1].bias
merged = nn.Linear(18, 1)
with torch.no_grad():
    merged.weight.copy_(W2 @ W1)
    merged.bias.copy_(W2 @ b1 + b2)
print((W2 @ W1).shape, (W2 @ b1 + b2).shape)
print(torch.allclose(stack(X_fit), merged(X_fit), atol=1e-5))
print(sum(p.numel() for p in stack.parameters()), sum(p.numel() for p in merged.parameters()))
print(torch.allclose(model(X_fit), merged(X_fit), atol=1e-5))

torch.Size([1, 18]) torch.Size([1])
True
641 19
False

stack is the network with the ReLU removed, seeded to the same draws, since nn.ReLU() draws nothing. W2 @ W1 is (1, 32) times (32, 18), a (1, 18) weight, W2 @ b1 + b2 is a (1,) bias, and merged, the one layer holding them, prices all 1,725 fit rows as stack does within 1e-5. Two layers of 641 parameters compute what one layer of 19 computes, and ten layers with nothing between them would too, because a product of matrices is a matrix. model, with the ReLU in place, is not within 1e-5 of merged: the 74 zeros on four rows are 74 places where the function bent, and no (1, 18) weight bends. The next subtitle measures what that is worth.

DON'T:

• DON'T stack linear layers and call the model deeper. nn.Linear(18, 32) into nn.Linear(32, 1) is one nn.Linear(18, 1), 641 parameters doing 19's work.
• DON'T compare the collapsed layer with torch.allclose at its defaults. stack and merged differ by about 1e-7 at most, float32's rounding on two products against one, and the default tolerances fail on an output near zero, so the comparison carries atol=1e-5.

The MLP, Fitted and Compared

What. The lesson's loop as a function, run(model, optimizer, epochs), at 0.1 for 500 epochs, with both losses kept per epoch. The score is Lesson 3.1.4's MAE and R² on the validation rows, on predictions multiplied back, returned by score(model, X, frame) as two floats. Why. The network is judged on held-out rows against the model it claims to beat, nn.Linear(18, 1) in the same loop and LinearRegression on the same 18 columns, so what the hidden layer bought is one subtraction. When. Before any test score, which this lesson never takes: the validation rows compare the models, and Lesson 4.3.4 scores the test rows once, after the rate is chosen.

The loop, the validation pass, the score and the network's constructor go into one file, train_loop.py, saved beside prep_net.py:

import torch
from torch import nn, optim
from sklearn.metrics import mean_absolute_error, r2_score
from prep_net import X_fit, y_fit, X_val, y_val

loss_fn = nn.MSELoss()

def network(width=32):
    torch.manual_seed(42)
    return nn.Sequential(nn.Linear(18, width), nn.ReLU(), nn.Linear(width, 1))

def validate(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = loss_fn(model(X), y)
    model.train()
    return loss.item()

def run(model, optimizer, epochs, report=()):
    train_losses, val_losses = [], []
    for epoch in range(1, epochs + 1):
        loss = loss_fn(model(X_fit), y_fit)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        train_losses.append(loss.item())
        val_losses.append(validate(model, X_val, y_val))
        if epoch in report:
            print(epoch, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}")
    return train_losses, val_losses

def score(model, X, frame):
    model.eval()
    with torch.no_grad():
        pred = model(X).squeeze(1).numpy() * 100_000
    model.train()
    return mean_absolute_error(frame["price"], pred), r2_score(frame["price"], pred)

if __name__ == "__main__":
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    run(model, optimizer, 500, (1, 10, 50, 100, 200, 500))

python3 train_loop.py fits the section's network under its main guard:

1 19.5549 7.9833
10 1.4752 0.9882
50 0.8309 0.4995
100 0.8009 0.4683
200 0.7655 0.4509
500 0.7188 0.4442

network(width) is the subtitle above's three lines with the seed inside and the width as an argument, 32 unless one is given, so every script that calls network() starts from the draws printed there. run is Lesson 4.2.3's loop with the model and the optimizer as arguments, the optimizer built outside it so that a later lesson can hand it a different one, and score multiplies the predictions back to currency and scores them against the frame's own price column, never against y_val, which is in hundreds of thousands. The validation loss sits below the training loss as Lesson 4.2.3's did, the validation rows easier to price by the split's draw. The lowest validation loss and the score in currency, from a script that imports both files:

import numpy as np
import torch
from torch import nn, optim
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from prep_net import fit_rows, val_rows, sd, X_fit, X_val
from train_loop import network, run, score

model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses = run(model, optimizer, 500)
lowest = int(np.argmin(val_losses))
# 366 0.4424
print(lowest + 1, f"{val_losses[lowest]:.4f}")
mae, r2 = score(model, X_val, val_rows)
# 45,585 0.859
print(f"{mae:,.0f}", round(r2, 3))

The import ran the file's definitions and nothing under its main guard, so model is the same fit again, in silence. Its lowest validation loss sits 0.4 percent under its end, and the network misses a validation listing by 45,585 on average and explains 85.9 percent of the validation prices' variance, score's two floats printed as Lesson 3.1.4 printed them. The two linear fits:

torch.manual_seed(42)
linear = nn.Linear(18, 1)
optimizer = optim.SGD(linear.parameters(), lr=0.1)
run(linear, optimizer, 500, (1, 10, 100, 500))
mae, r2 = score(linear, X_val, val_rows)
print(f"{mae:,.0f}", round(r2, 3))
solve = LinearRegression().fit(X_fit.numpy(), fit_rows["price"])
solve_pred = solve.predict(X_val.numpy())
print(f"{mean_absolute_error(val_rows['price'], solve_pred):,.0f}", round(r2_score(val_rows["price"], solve_pred), 3))

1 17.6329 6.9731
10 1.6255 1.0760
100 0.8981 0.5546
500 0.8845 0.5527
52,722 0.824
52,725 0.824

One layer on the same 18 columns, in the same loop, settles at 0.8845 and 0.5527 and scores 52,722 and 0.824, and LinearRegression on the same 18 columns and the raw price, Lesson 3.1.1's one solve, scores 52,725 and 0.824: the loop found least squares' line to within four units of MAE, because 500 steps stop just short of the exact bottom, which 5,000 reach. The hidden layer bought 7,140 of MAE on the validation rows, 52,725 to 45,585, and a 32-unit layer with a ReLU is the only difference between the two models. The second subtitle's DON'T quoted what the same predictions score before the factor:

model.eval()
with torch.no_grad():
    forgot = model(X_val).squeeze(1).numpy()
model.train()
# 366,109 -4.27
print(f"{mean_absolute_error(val_rows['price'], forgot):,.0f}", round(r2_score(val_rows["price"], forgot), 3))
# 3.65 366,113
print(round(forgot.mean(), 2), f"{val_rows['price'].mean():,.0f}")

Why the network won, as far as the data and the runs say. The listings' prices were generated as a product, an area times a rate per square meter that depends on the neighborhood, times factors for the property type, the condition and the distance, the form behind Lesson 3.1.4's fan. A line adds a fixed amount per square meter whatever the neighborhood and a fixed amount per neighborhood whatever the area, so it misses a large Eastgate house and a small Hillcrest apartment on one set of weights. A hidden unit can be positive only when the area is large and the listing is in Eastgate, zero otherwise, a product the output layer can add up, and 32 such units with their ReLUs can bend the line where the data bends. The runs say that much and no more: without the ReLU the same 641 parameters are a line, with it they are 7,140 closer on rows the fit never saw.

DON'T:

• DON'T report the network's win without the linear model beside it in the same loop. 45,585 means nothing until 52,722 sits next to it.
• DON'T score the test rows here. The validation rows have chosen nothing yet, the width and the rate are still open, and Lesson 4.3.4 takes the one test score after the rate is chosen.

Reading a Network's Parameters

What. named_parameters() lists the model's four tensors by name and shape, 641 numbers, and state_dict() carries the same four names, Lesson 4.2.2's dictionary with the module's position in front of each. A hidden weight is read by its shape and its role, never as a coefficient. Why. Lesson 4.2.3 read a weight back as 3,408 per square meter, and the same reading of 0.weight[0, 0] gives a number that means nothing in square meters. What the hidden layer computes for a listing is a learned representation (an intermediate set of 32 numbers the network computes for each listing, chosen by the fit and read by nothing but the next layer). When. Every time a network is saved, Lesson 4.4.3, counted, the cost Lesson 4.3.4 sets beside a fit's seconds, or debugged, and every time someone asks what a weight means.

for name, p in model.named_parameters():
    print(name, tuple(p.shape), p.numel())
print(sum(p.numel() for p in model.parameters()))
print(list(model.state_dict().keys()))

0.weight (32, 18) 576
0.bias (32,) 32
2.weight (1, 32) 32
2.bias (1,) 1
641
['0.weight', '0.bias', '2.weight', '2.bias']

The names are the module's position in the Sequential and the parameter's name inside it, so 0.weight is model[0].weight, one row of 18 weights per hidden unit, 2.weight holds the output layer's one weight per hidden unit, and the ReLU at position 1 has nothing to list. Reading them:

w_area = linear.weight[0, 0].item() * 100_000 / sd["area_sqm"]
# 3768 3764
print(round(w_area), round(solve.coef_[0] / sd["area_sqm"]))
# 0.7364 1.0780
print(f"{model[0].weight[0, 0].item():.4f} {model[2].weight[0, 0].item():.4f}")

The linear model's first weight reads back as Lesson 4.2.3 read its one weight, times 100,000 for the target's scale and divided by the area's standard deviation, a price per square meter within four of LinearRegression's, the 500 steps' shortfall. The network's 0.weight[0, 0] is the first hidden unit's weight on the standardized area, and no factor turns it into a price per square meter: the unit adds its 18 weighted inputs and a bias, ReLU zeroes the sum where it is negative, and the output layer multiplies what is left by 2.weight[0, 0] before adding 31 other units' contributions. What the hidden layer holds for one listing, and across the fit rows:

with torch.no_grad():
    rep = model[1](model[0](X_val[:1]))
    share = (model[1](model[0](X_fit)) > 0).float().mean(dim=0)
# torch.Size([1, 32]) 18
print(rep.shape, (rep == 0).sum().item())
# 0.037 0.987 0
print(round(share.min().item(), 3), round(share.max().item(), 3), (share == 0).sum().item())

rep is the first validation listing's learned representation, 32 numbers, zero for the units whose sum came out negative for that listing, and share is, for each unit, the share of the 1,725 fit rows on which it is positive, and none is a dead unit (a hidden unit that is zero on every row of the table and contributes nothing). The 32 numbers are what the output layer prices, and the fit chose them with no instruction beyond the loss, which is why none of the 32 has a name a listing site could print.

DON'T:

• DON'T read a hidden weight as a coefficient. 0.7364 on the standardized area is not 73,640 per standard deviation or anything per square meter.
• DON'T count parameters by the shapes you expect. sum(p.numel() for p in model.parameters()) counts what the model holds, 641 here, and a Sequential with a layer added or widened prints its own count.

Width, Previewed

What. Width (the number of hidden units in a layer) is the setting nn.Linear(18, 32) fixes at 32. Four networks, 8, 32, 128 and 512 wide, each from network(width) and run for 1,000 epochs at 0.1, timed with time.perf_counter(), Lesson 3.6.3's clock. Why. More width is more capacity, Lesson 3.6.1's word, and the loss curves and the clock say what capacity buys on 1,725 rows. When. Here as a preview, so that 32 reads as a choice and not a law. Lesson 4.4.1 widens the network until it memorizes a slice of the listings and gives width its proper treatment.

import time
import numpy as np
from torch import optim
from train_loop import network, run

seconds = []
for width in (8, 32, 128, 512):
    net = network(width)
    count = sum(p.numel() for p in net.parameters())
    optimizer = optim.SGD(net.parameters(), lr=0.1)
    start = time.perf_counter()
    train_losses, val_losses = run(net, optimizer, 1000)
    seconds.append(time.perf_counter() - start)
    lowest = int(np.argmin(val_losses))
    # width, parameters, final training loss, final validation loss, its lowest epoch and value
    print(width, count, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", lowest + 1, f"{val_losses[lowest]:.4f}")

8 161 0.7230 0.4612 268 0.4382
32 641 0.7063 0.4541 366 0.4424
128 2561 0.6902 0.4743 248 0.4412
512 10241 0.6268 0.4827 384 0.4501

print([round(s, 1) for s in seconds])

The training loss falls with every step up in width, 0.7230 to 0.6268: more units fit the fit rows better, as 101 weights fitted 150 rows better than one in Lesson 4.2.3. The validation loss does not. Its final value is lowest at width 32 and rises from there, 0.4541 to 0.4827 at 512, and its lowest point along the way lands between epoch 248 and epoch 384 for all four and rises after it, Lesson 4.2.3's gap, widest at 128 and 512. The last line printed [0.8, 0.9, 1.5, 8.5] on the machine these runs were made on, seconds for 1,000 epochs, and between ten and fifteen times the narrowest network's seconds for the widest across three runs of it there, for a final validation loss 0.02 worse, and your machine prints its own seconds. A width is a choice made on the validation rows with its seconds beside it, and 32 holds until Lesson 4.4.1 asks the question properly.

DON'T:

• DON'T widen a network for a lower training loss. 0.6268 at width 512 is the lowest training loss of the four and the highest final validation loss, 0.4827.
• DON'T read the four lowest validation losses as a ranking you can ship. They sit within 0.012 of each other on one split and one seed, which Lesson 3.1.2 called one draw, and the seconds are the one difference that is not a draw: ten to one and more.

Lab Exercises

Lab Exercise 1: The Table Read Back, and a Collapse at Width 8

Start width8.py by importing what it needs from prep_net and train_loop. Print the first validation row's seven columns and price from the frame beside its 18 numbers in X_val. Print whether the fit rows' four standardized columns have a mean of zero within 1e-5 and what their standard deviations are, then the validation rows' four means and four standard deviations, and the unique row sums of each of the three one-hot blocks on the validation rows. Print the validation target's shape, its first three values and its mean to three decimals, then the validation rows' mean price in whole units.

Seed with 42 and build nn.Sequential(nn.Linear(18, 8), nn.Linear(8, 1)), collapse it into one nn.Linear(18, 1), and print whether the two agree on the validation rows within atol=1e-5 and their two parameter counts. Build network(8), print whether its two layers equal the stack's and whether it agrees with the collapsed layer, and, on the first four validation rows, the hidden tensor's shape, how many of its values are negative, how many come out of the ReLU as zero, and the two models' four predictions. Finish with a fresh network(8) on the raw price and another on the scaled target, five epochs each at 0.1, printing the loss at every epoch.

Lab Exercise 2: Width 16 Against Width 64

Start widths.py by importing what it needs from prep_net and train_loop. For widths 16 and 64 in turn, build network(width), print its parameters by name, shape and count and their total, run 500 epochs at 0.1 printing both losses at epochs 1, 100 and 500, and print the epoch of the lowest validation loss, that loss, and the ratio of the final validation loss to it to three decimals. Score each on the validation rows in currency, then print, for the first validation listing, the representation's shape and how many of its values are zero, and across the fit rows the smallest and largest share of rows on which a unit is positive and the number of units never positive. Say in a comment which width you would carry forward against the Theory's 32, and on which number.

import torch
from torch import nn, optim
from prep_net import val_rows, numeric, text, X_fit, y_fit, X_val, y_val
from train_loop import loss_fn, network

print(val_rows[numeric + text + ["price"]].iloc[0].to_dict())
print(X_val[0])
print(torch.allclose(X_fit[:, :4].mean(dim=0), torch.zeros(4), atol=1e-5))
print(X_fit[:, :4].std(correction=0, dim=0))
print(X_val[:, :4].mean(dim=0))
print(X_val[:, :4].std(correction=0, dim=0))
print(X_val[:, 4:11].sum(dim=1).unique(), X_val[:, 11:14].sum(dim=1).unique(), X_val[:, 14:].sum(dim=1).unique())
print(y_val.shape, y_val[:3].squeeze(1), round(y_val.mean().item(), 3))
print(f"{val_rows['price'].mean():,.0f}")

{'area_sqm': 131.4, 'rooms': 5, 'bathrooms': 2, 'distance_km': 7.0, 'neighborhood': 'Southgate', 'property_type': 'house', 'condition': 'good', 'price': 570000}
tensor([0.7152, 0.8183, 0.3546, 0.4923, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,
        0.0000, 1.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000, 1.0000, 0.0000])
True
tensor([1., 1., 1., 1.])
tensor([-0.0762, -0.0730, -0.0819, -0.0625])
tensor([0.9845, 0.9646, 0.9233, 1.0457])
tensor([1.]) tensor([1.]) tensor([1.])
torch.Size([575, 1]) tensor([5.7000, 2.5000, 2.7900]) 3.661
366,113

torch.manual_seed(42)
stack = nn.Sequential(nn.Linear(18, 8), nn.Linear(8, 1))
W1, b1 = stack[0].weight, stack[0].bias
W2, b2 = stack[1].weight, stack[1].bias
merged = nn.Linear(18, 1)
with torch.no_grad():
    merged.weight.copy_(W2 @ W1)
    merged.bias.copy_(W2 @ b1 + b2)
print(torch.allclose(stack(X_val), merged(X_val), atol=1e-5))
print(sum(p.numel() for p in stack.parameters()), sum(p.numel() for p in merged.parameters()))

bent = network(8)
print(torch.equal(bent[0].weight, stack[0].weight), torch.equal(bent[2].weight, stack[1].weight))
print(torch.allclose(bent(X_val), merged(X_val), atol=1e-5))
with torch.no_grad():
    hidden = bent[0](X_val[:4])
    print(hidden.shape, (hidden < 0).sum().item(), (bent[1](hidden) == 0).sum().item())
    print(bent(X_val[:4]).squeeze(1))
    print(merged(X_val[:4]).squeeze(1))

True
161 19
True True
False
torch.Size([4, 8]) 14 14
tensor([0.1894, 0.3422, 0.3446, 0.3419])
tensor([0.1343, 0.3241, 0.4663, 0.3136])

y_fit_raw = y_fit * 100_000
for target in (y_fit_raw, y_fit):
    bent = network(8)
    optimizer = optim.SGD(bent.parameters(), lr=0.1)
    for epoch in range(1, 6):
        loss = loss_fn(bent(X_fit), target)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        print(epoch, f"{loss.item():.4f}")

1 190757175296.0000
2 940144976472309760.0000
3 inf
4 inf
5 nan
1 16.8602
2 9.5392
3 4.1989
4 2.3160
5 1.9881

import numpy as np
import torch
from torch import optim
from prep_net import val_rows, X_fit, X_val
from train_loop import network, run, score

for width in (16, 64):
    net = network(width)
    for name, p in net.named_parameters():
        print(name, tuple(p.shape), p.numel())
    print(sum(p.numel() for p in net.parameters()))
    optimizer = optim.SGD(net.parameters(), lr=0.1)
    train_losses, val_losses = run(net, optimizer, 500, (1, 100, 500))
    lowest = int(np.argmin(val_losses))
    print(lowest + 1, f"{val_losses[lowest]:.4f}", round(val_losses[-1] / val_losses[lowest], 3))
    mae, r2 = score(net, X_val, val_rows)
    print(f"{mae:,.0f}", round(r2, 3))
    with torch.no_grad():
        rep = net[1](net[0](X_val[:1]))
        share = (net[1](net[0](X_fit)) > 0).float().mean(dim=0)
    print(rep.shape, (rep == 0).sum().item(), round(share.min().item(), 3), round(share.max().item(), 3), (share == 0).sum().item())
# Width 16 at 46,227 and width 64 at 49,183 both trail the Theory's 32 at 45,585 on
# the validation rows, and 64's validation loss turned at epoch 285 and ended 12.4
# percent above its lowest: 32 goes forward, on the validation MAE, seconds beside it.

0.weight (16, 18) 288
0.bias (16,) 16
2.weight (1, 16) 16
2.bias (1,) 1
321
1 20.9582 9.7979
100 0.8184 0.4827
500 0.7658 0.4520
500 0.4520 1.0
46,227 0.856
torch.Size([1, 16]) 9 0.064 0.99 0
0.weight (64, 18) 1152
0.bias (64,) 64
2.weight (1, 64) 64
2.bias (1,) 1
1281
1 18.5354 3.1552
100 0.8305 0.5584
500 0.7320 0.4990
285 0.4440 1.124
49,183 0.841
torch.Size([1, 64]) 33 0.006 0.954 0

Theory Assessment

Q1: Eighteen Numbers for One Listing

The first fit row prints as -0.8938, -0.3373, -0.8639, 1.0243, then 0, 0, 0, 0, 0, 0, 1, then 1, 0, 0, then 1, 0, 0, 0. Say what each of the four blocks is and why a network has to receive a listing in this form, why the validation rows' standardized means print -0.0762, -0.0730, -0.0819, -0.0625 rather than zero and what that says about the numbers used, and what fitting enc on has_area would change here in the numbers and in the rule.

Hint: The first layer computes X @ W.T + b. Ask what a matrix product can multiply, what Lesson 3.2.2 did about it, and which rows Lesson 3.1.2 lets a fit learn anything from. For the encoder, ask which categories the validation rows hold that the fit rows do not.

Q2: One Layer or Two

nn.Sequential(nn.Linear(18, 32), nn.Linear(32, 1)) priced the fit rows within 1e-5 of one nn.Linear(18, 1), and the same two layers with nn.ReLU() between them did not. Say what the collapsed layer's weight and bias are and why the collapse holds for any number of linear layers, what ReLU does to the 74 negative values of a four-row batch and why that stops the collapse, and what the shapes (4, 18), (4, 32), (4, 32) and (4, 1) say about where the ReLU acts.

Hint: Write the two layers as one expression in W1, b1, W2 and b2 and multiply out. Then ask whether max(0, x) can be written as a matrix times x, and which of the four shapes the ReLU leaves unchanged.

Q3: The Scale of the Answer

At 0.1 on the raw price the network printed inf at epoch 3, and nn.Linear(18, 1) on the same raw price at the same rate settled at 8.8441e+09. Say what the output weight of 30,150 after one step and the output gradient of 4.3301e+13 at the second epoch show about where the raw price enters the step, why the one-layer model survived the same target, why the scaled target at 0.1 falls by a factor of eight in five epochs instead, and what a prediction of 2.72 is worth before and after the factor.

Hint: The derivative of a squared error carries the residual, and a residual on the raw price is a price. Ask what a weight of 30,150 on the output layer does to the hidden layer's gradients at the next epoch, and what size a one-layer model's weights have to reach anyway.

Q4: The Network That Was a Line

Read this script without running it:

import torch
from torch import nn, optim
from prep_net import X_fit, y_fit, X_val, val_rows
from train_loop import loss_fn, score

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(18, 32), nn.Linear(32, 1))
print(sum(p.numel() for p in model.parameters()))
optimizer = optim.SGD(model.parameters(), lr=0.1)
for epoch in range(1, 501):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if epoch in (1, 10, 100, 500):
        print(epoch, f"{loss.item():.4f}")
mae, r2 = score(model, X_val, val_rows)
print(f"{mae:,.0f}", round(r2, 3))

A teammate ran it, read 641 on the first line and 52,726 0.824 on the last, and reports that a 641-parameter network trained with the Theory's loop, seed, table and scaled target prices a listing no better than LinearRegression's 52,725, so the hidden layer buys nothing and Module 3's model should ship. Say what the four loss lines print and why the last line is the number it is, what the model learned, and what in the report is right.

Hint: Count the modules in this Sequential, then count the modules in the Theory's. Ask what the third subtitle proved about two linear layers with nothing between them, and what the fourth subtitle's network scored with the one module this script lacks.

641
1 19.2628
10 1.8569
100 0.8943
500 0.8843
52,726 0.824

Assignment

Assignment: A Second Hidden Layer, and a Smoother Function

A teammate reviewing the pricing network proposes two changes, each sold as more capacity, and wants both measured before either ships. The first is a second hidden layer of 32 units with its own ReLU, nn.Linear(32, 32) after the first ReLU, so the network reads 18 columns into 32 numbers, those into 32 more, and those into a price. The second is nn.Tanh in place of nn.ReLU, which this lesson did not use: the hyperbolic tangent, a smooth function that maps any input into the open interval between -1 and 1, approaching both ends and reaching neither, so a large input moves its output almost not at all.

Write deeper.py, importing what it needs from prep_net and train_loop. Print torch.tanh of [-2.0, 0.0, 3.0] and nn.Tanh() of [-20.0, 20.0]. Write build(activation, hidden_layers), which returns a Sequential of nn.Linear(18, 32), the activation, hidden_layers - 1 further pairs of nn.Linear(32, 32) and the activation, and nn.Linear(32, 1), built with nn.Sequential(*layers), which hands the items of a list to the constructor as separate arguments. For the four combinations of ReLU or Tanh and one or two hidden layers, seed with 42, build, print the activation's name, the number of hidden layers, the parameter count and len(net), run 500 epochs at 0.1 printing both losses at epochs 1, 50, 100, 200 and 500, and print the epoch of the lowest validation loss with its value and the validation score in currency. Print the two-hidden-layer Tanh network. Then say which of the four you would carry forward at epoch 500, which would have won read at its own lowest epoch, and what that difference is a question about.

Hints:

• A module class is a value: activation() builds a fresh nn.ReLU() or nn.Tanh() each time it is called, and activation.__name__ prints its name. The ReLU network with one hidden layer is network()'s three modules from the same seed, and its numbers must come out the same, which is the check that build is right.
• Print each network's lowest validation loss beside the Theory's 0.4424, not only its end.

import numpy as np
import torch
from torch import nn, optim
from prep_net import val_rows, X_val
from train_loop import run, score

# tensor([-0.9640,  0.0000,  0.9951])
print(torch.tanh(torch.tensor([-2.0, 0.0, 3.0])))
# tensor([-1.,  1.])
print(nn.Tanh()(torch.tensor([-20.0, 20.0])))

def build(activation, hidden_layers):
    layers = [nn.Linear(18, 32), activation()]
    for _ in range(hidden_layers - 1):
        layers += [nn.Linear(32, 32), activation()]
    layers.append(nn.Linear(32, 1))
    return nn.Sequential(*layers)

for activation in (nn.ReLU, nn.Tanh):
    for hidden_layers in (1, 2):
        torch.manual_seed(42)
        net = build(activation, hidden_layers)
        count = sum(p.numel() for p in net.parameters())
        print(activation.__name__, hidden_layers, count, len(net))
        optimizer = optim.SGD(net.parameters(), lr=0.1)
        train_losses, val_losses = run(net, optimizer, 500, (1, 50, 100, 200, 500))
        lowest = int(np.argmin(val_losses))
        mae, r2 = score(net, X_val, val_rows)
        print(lowest + 1, f"{val_losses[lowest]:.4f}", f"{mae:,.0f}", round(r2, 3))
print(build(nn.Tanh, 2))

ReLU 1 641 3
1 19.5549 7.9833
50 0.8309 0.4995
100 0.8009 0.4683
200 0.7655 0.4509
500 0.7188 0.4442
366 0.4424 45,585 0.859
ReLU 2 1697 5
1 19.0860 9.4360
50 1.2664 1.1250
100 1.0581 0.7831
200 0.8570 0.5464
500 0.7429 0.4735
468 0.4708 46,482 0.849
Tanh 1 641 3
1 19.2514 6.8063
50 2.0049 2.7254
100 0.8600 0.4810
200 1.0081 0.7461
500 0.7786 0.5033
241 0.4334 50,560 0.84
Tanh 2 1697 5
1 20.1557 8.7290
50 1.2190 0.6919
100 1.0059 0.6347
200 0.8740 0.5388
500 0.7874 0.4829
448 0.4333 48,042 0.846
Sequential(
  (0): Linear(in_features=18, out_features=32, bias=True)
  (1): Tanh()
  (2): Linear(in_features=32, out_features=32, bias=True)
  (3): Tanh()
  (4): Linear(in_features=32, out_features=1, bias=True)
)

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the other activation functions and the vocabulary of depth before Sections 4.4 and 4.5 use them.

• Course notes - Stanford CS231n, "Neural Networks Part 1: Setting up the Architecture", the sections "Commonly used activation functions" and "Layer-wise organization". The sigmoid, tanh, ReLU, leaky ReLU and maxout written out with what each costs in training, saturation and dead units among the costs, then how layers are counted and named, why the output layer carries no activation, and the parameter count of two small networks worked by hand, the arithmetic this lesson did for 641.
• Official docs - PyTorch 2.14, torch.nn.ReLU. The module's one definition, max(0, x) applied element by element, its inplace argument, which this lesson leaves at its default, and the shape contract, any shape in and the same shape out, which is why the hidden tensor stayed (4, 32) through it.

Summary

The step from a neuron to a network was three lines and one idea. The three lines are a layer, a function and a layer, held in order by nn.Sequential and trained by the loop of Section 4.2 without a change to the loop. The idea is that the function between the layers is not decoration: a run collapsed two linear layers into one, and a trained stack confirmed that 641 parameters with nothing between them price a listing as 19 do, to within a few units of MAE. Everything a network can do that a line cannot lives where ReLU bent the sum, and on these listings that was worth 7,140 of MAE on rows the fit never saw.

The table and the target were Module 3's discipline carried into tensors, with one addition the hidden layer forced: the price itself had to be scaled, because a residual in the hundreds of thousands drives a two-layer step past what float32 holds while a one-layer step survives it. What the network learned was readable in count and in shape and in nothing else, 32 numbers per listing that the fit chose, about half of them zero for a typical listing, with no column a listing site could print.

Width turned out to be a choice, and this lesson left it at 32 with its seconds and its validation loss on record. The table and the loop went into two files, prep_net.py and train_loop.py, and the rest of the section imports them instead of writing them again. Lesson 4.3.2 imports the table, adds the price as a column, and changes the last layer's meaning and the loss. Lesson 4.3.3 changes how many rows a step sees, Lesson 4.3.4 changes the step itself, and Section 4.4 takes up what the width table showed, a training loss that falls with every parameter and a validation loss that does not.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

mu, sd = fit_rows[numeric].mean(), fit_rows[numeric].std(ddof=0)
enc = OneHotEncoder(sparse_output=False, handle_unknown="ignore").fit(fit_rows[text])

def prep(frame):
    # fit rows' statistics, then the fit rows' categories
    scaled = ((frame[numeric] - mu) / sd).to_numpy()
    hot = enc.transform(frame[text])
    X = torch.tensor(np.hstack([scaled, hot]), dtype=torch.float32)
    # the price in hundreds of thousands, (n, 1)
    y = torch.tensor(frame["price"].to_numpy() / 100_000, dtype=torch.float32)
    return X, y.unsqueeze(1)

torch.manual_seed(42)                       # before the model, never after
model = nn.Sequential(
    nn.Linear(18, 32),                      # X @ W.T + b, 32 hidden units
    nn.ReLU(),                              # max(0, x), the nonlinearity
    nn.Linear(32, 1),                       # the price, in hundreds of thousands
)
print(model[0].weight.shape)                # torch.Size([32, 18])
sum(p.numel() for p in model.parameters())  # 641
optimizer = optim.SGD(model.parameters(), lr=0.1)
# both losses per epoch, the optimizer handed in
train_losses, val_losses = run(model, optimizer, 500)
mae, r2 = score(model, X_val, val_rows)     # two floats, formatted at the print
pred = model(X_val) * 100_000               # back to currency before MAE