Train and Validation Discipline, and the Loss Curve

Context

A disciplined training loop fits its parameters on one set of rows, scores a second set it never fits at every epoch's end, and scores a third set once, after every choice. The two numbers it records every epoch, the training and validation losses, are the loss curve, and the loop of Lesson 4.2.2 gains two lists and the two lines that fill them to produce it.

Every model from here to the end of Module 6 is trained this way: the network and classifiers of Section 4.3, the regularized models of Section 4.4, whose every regularization tool is judged on nothing but the curve this lesson draws, and the image network of Section 4.5. A loss curve is also the first thing a reviewer asks for when a number looks wrong.

Without the discipline the loop still runs, exits 0 and prints a loss that falls, and nothing says whether the number means anything. A loop that standardizes each set by its own mean, scores the validation rows with a graph attached, or trusts a falling loss that never beat predicting the mean price reports a model that does not exist. Lesson 4.2.2 proved that nn.Linear, nn.MSELoss and optim.SGD compute what Lesson 4.2.1's hand loop computed on all 2,875 listings, the same printed loss at every epoch from one seeded start and the same parameters within allclose. This lesson holds rows back from that loop as Lesson 3.1.2 did from LinearRegression, and reads the two curves on a model that cannot overfit and then on one built to.

Goal

• Split the listings into fit, validation and test rows with two train_test_split calls, and standardize all three with the fit rows' mean and standard deviation alone.
• Write a validation pass under torch.no_grad() and model.eval(), restore model.train() after it, and show from a run that neither switch changes a number on a model without dropout.
• Compute the mean-price baseline and fit a two-listing batch, and reject a loop whose validation loss ends above the first or that cannot reach the floor of the second.
• Collect the training and validation losses per epoch as Python floats, draw them as one fig, ax figure, and check the chart by printing what it holds.
• Read two curves that fall together and two that part, and name the epoch where a validation curve turns as the point a later lesson's early stopping acts on.
• Score the test rows once with mean_absolute_error and r2_score, and show the neuron's weight and intercept match LinearRegression on the same rows to the unit.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see the loop and its validation pass written by the people who wrote the library, before you build yours.

• YouTube - "Training with PyTorch" by PyTorch. Fifteen minutes on the pieces a training loop is built from: the Dataset and DataLoader classes that hand a model its rows in batches, which Lesson 4.3.3 introduces, the loss functions, the optimizers, and the loop itself, whose validation pass once per epoch runs with training mode switched off by model.train(False) and without torch.no_grad(), and keeps the model of the lowest validation loss.
• Official docs - PyTorch Tutorials, "Optimizing Model Parameters", the section "Full Implementation". A train_loop and a test_loop on an image classifier, the first under model.train() with the five steps, the second under model.eval() and torch.no_grad(), with comments beside both saying that neither call changes anything for that model and that both are there as a habit.

Table of Contents

• Fit Rows, Validation Rows, Test Rows
• Two Switches: no_grad and eval
• Two Checks Before Trusting a Loop
• The Loss Curve, Drawn and Checked
• Reading the Two Curves Together
• The Test Rows, Scored Once

Theory

Every Theory block runs in ~/dev/bootcamp on the 2,875 listings with an area, split as the first subtitle splits them, with Lesson 4.2.2's loop, the fifth subtitle's nn.Linear(101, 1) on its 101 columns aside: nn.Linear(1, 1), nn.MSELoss() and optim.SGD at 0.1 on a standardized area held as a (n, 1) float32 tensor against a (n, 1) price, the five steps in 4.2.2's order with optimizer.zero_grad() last. The first subtitle writes prep.py, which every later Theory script imports, as Lesson 3.1.2's scripts imported memorizer.py. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Fit Rows, Validation Rows, Test Rows

What. Fit rows (the rows the loop's gradient is computed from, Lesson 3.3.1's word for the training rows left after a validation set is carved out of them) are one of three sets. The validation rows are scored at the end of every epoch and never fitted, and the test rows are scored once, after every choice. The split is Lesson 3.1.2's held-out rule applied to a loop. Why. A loop drives its fit rows' loss down for as long as it runs, so the training loss says nothing about new rows. The validation loss is the number the loop is judged by, and every decision made on it, how many epochs, which learning rate, which feature, changes the number the test rows will print, so the test rows stay out of every decision. When. Before the first tensor is built, with the two train_test_split calls ahead of every statistic and every tensor.

Lesson 3.1.2's split holds out 20 percent of the table as the test rows, the same 575 listings that lesson held out, and a second call carves a quarter of the rest as the validation rows, the sizes of its Assignment's carve without that carve's stratification, both with random_state=42. The two numbers are Lesson 4.2.1's mean and std(correction=0) kept as tensors, computed on the 1,725 fit rows and applied to all three sets, Lesson 3.2.1's rule for a scaler written on tensors. Save it as prep.py:

import pandas as pd
import torch
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = has_area["area_sqm"].to_numpy()
price = has_area["price"].to_numpy()

# the test rows first, then the validation rows out of the training rows
area_train, area_test, price_train, price_test = train_test_split(
    area, price, test_size=0.2, random_state=42
)
area_fit, area_val, price_fit, price_val = train_test_split(
    area_train, price_train, test_size=0.25, random_state=42
)

# the fit rows' two numbers, kept as tensors, standardize every set
fit_tensor = torch.tensor(area_fit, dtype=torch.float32)
mean, sd = fit_tensor.mean(), fit_tensor.std(correction=0)

def to_tensors(area_rows, price_rows):
    z = (torch.tensor(area_rows, dtype=torch.float32) - mean) / sd
    y = torch.tensor(price_rows, dtype=torch.float32)
    return z.unsqueeze(1), y.unsqueeze(1)

X_fit, y_fit = to_tensors(area_fit, price_fit)
X_val, y_val = to_tensors(area_val, price_val)
X_test, y_test = to_tensors(area_test, price_test)

if __name__ == "__main__":
    print(len(area_fit), len(area_val), len(area_test))
    print(round(mean.item(), 2), round(sd.item(), 4))
    print(X_fit.shape, y_fit.shape)
    print(X_val.shape, X_test.shape)
    print(torch.allclose(X_fit.mean(), torch.tensor(0.0), atol=1e-5))
    print(round(X_fit.std(correction=0).item(), 4))
    print(round(X_val.mean().item(), 4), round(X_val.std(correction=0).item(), 4))

python3 prep.py runs the block under the main guard, Lesson 1.7.2's, and an import prep in another script runs everything above it in silence:

1725 575 575
99.44 44.6871
torch.Size([1725, 1]) torch.Size([1725, 1])
torch.Size([575, 1]) torch.Size([575, 1])
True
1.0
-0.0762 0.9845

The validation rows' standardized area lands near zero and one and at neither, the mark of a set transformed with numbers that know nothing about it, and every tensor is (n, 1), Lesson 4.2.2's shape for both sides of nn.MSELoss.

The wrong version computes each set's own numbers:

import torch
from prep import area_val, mean, sd

own = torch.tensor(area_val, dtype=torch.float32)
# 96.03 43.9933
print(round(own.mean().item(), 2), round(own.std(correction=0).item(), 4))
listing = torch.tensor(140.0)
# 0.9076
print(round(((listing - mean) / sd).item(), 4))
# 0.9994
print(round(((listing - own.mean()) / own.std(correction=0)).item(), 4))

The model learned its weight on the fit rows' scale, and the second number feeds it an input from a scale it never saw.

DON'T:

• DON'T standardize each set by its own numbers. The validation rows' 96.03 and 43.9933 put a 140 square meter listing at 0.9994 where the model's scale puts it at 0.9076, and the loss then measures the change of scale as much as the model.
• DON'T compute the two numbers on all 2,875 listings. Lesson 4.2.1's 99.02 and 44.0589 hold the held-out listings' areas, Lesson 3.1.2's third break.
• DON'T touch the test rows inside a loop. Nothing in this lesson reads X_test before the last subtitle.

Two Switches: no_grad and eval

What. torch.no_grad() is Lesson 4.1.4's context in which nothing is recorded, and a validation pass runs inside it. model.eval() sets model.training to False, which changes what dropout (a layer that switches off a random share of its inputs on every training pass, Lesson 4.4.2's) and batch normalization (a layer that rescales its input by the batch's own mean and spread while training) compute, and changes nothing else. model.train() sets it back to True. Why. A validation loss with a graph attached is one backward() from fitting the validation rows, and the graph holds every tensor of the pass in memory for a number that only gets printed. eval() changes no number in this section, which is the reason to write it now: Lesson 4.4.2's dropout reads the flag, and a validation pass without eval() there looks like overfitting and is dropout left running. When. Both switches on every validation pass and every test pass, every time, and model.train() before the next fit step.

The seeded model of Lesson 4.2.2, before any step, scored on the validation rows twice:

import torch
from torch import nn
from prep import X_val, y_val

loss_fn = nn.MSELoss()
torch.manual_seed(42)
model = nn.Linear(1, 1)

recorded = loss_fn(model(X_val), y_val)
# True MseLossBackward0
print(recorded.requires_grad, recorded.grad_fn.name())
with torch.no_grad():
    unrecorded = loss_fn(model(X_val), y_val)
# False None
print(unrecorded.requires_grad, unrecorded.grad_fn)
print(torch.equal(recorded, unrecorded))  # True
print(f"{unrecorded.item():.4e}")         # 1.6543e+11

1.6543e+11 is the seeded start, weight 0.7645 and bias 0.8300 pricing every listing near zero, and the next subtitle sets it against a baseline. What the graph allows:

print(model.weight.grad)        # None
recorded.backward()
print(model.weight.grad.shape)  # torch.Size([1, 1])

One backward(), and the validation rows have written a gradient into the model's weight for the next step() to apply. The second switch:

print(model.training)                    # True
model.eval()
print(model.training)                    # False
with torch.no_grad():
    in_eval = loss_fn(model(X_val), y_val)
model.train()
print(model.training)                    # True
print(torch.equal(in_eval, unrecorded))  # True

From here on, every validation pass in this module follows this function's pattern:

def validate(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = loss_fn(model(X), y)
    model.train()
    return loss.item()

# 1.6543e+11
print(f"{validate(model, X_val, y_val):.4e}")
print(model.training)  # True

DON'T:

• DON'T score the validation rows outside torch.no_grad(). A backward() on that loss, yours or a helper's, is then one line from fitting them.
• DON'T drop eval() because the number does not change on nn.Linear. The habit exists for Lesson 4.4.2's layer, on which it does.
• DON'T forget train() after the pass. Nothing raises, and every fit step from then on runs in evaluation mode, a different model once dropout is in it.

Two Checks Before Trusting a Loop

What. A baseline (the loss of predicting one number, the fit rows' mean price, for every validation row) is the number a loop has to beat, and the tiny-batch fit (the loop run on two listings until its loss settles a few hundredths from zero, as close as its float32 steps bring it) is the proof that the loop can fit anything at all. A loop passes both before its curves are read. Why. A falling loss is not a working loop: a loop with a wrong shape or a forgotten zeroing can print a loss that falls for a while, and a loop whose validation loss ends above the baseline has learned nothing worth keeping. Two listings have an exact answer, and a loop that cannot reach it on two rows will not reach anything on 1,725. When. Once per loop you write, before its first full run, and again whenever the loop changes.

The baseline, Lesson 3.1.4's DummyRegressor as one tensor:

import numpy as np
import torch
from torch import nn, optim
from prep import X_fit, y_fit, X_val, y_val, area_fit

loss_fn = nn.MSELoss()

def validate(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = loss_fn(model(X), y)
    model.train()
    return loss.item()

mean_price = y_fit.mean()
print(f"{mean_price.item():,.2f}")  # 384,795.38
baseline = loss_fn(torch.full_like(y_val, mean_price.item()), y_val)
print(f"{baseline.item():.4e}")     # 3.1740e+10
# 3.1391e+10
print(f"{y_val.var(correction=0).item():.4e}")

torch.manual_seed(42)
model = nn.Linear(1, 1)
start = validate(model, X_val, y_val)
# 1.6543e+11 5.21
print(f"{start:.4e}", round(start / baseline.item(), 2))

torch.full_like, Lesson 4.1.1's torch.full in y_val's shape, prices every validation row at the fit rows' mean. The validation prices' own variance is the loss their own mean would score, and the baseline sits above it by the square of the distance between the two means, Lesson 3.1.4's reason a constant from the training rows scores below zero.

The tiny batch is the two fit listings whose areas sit farthest apart:

lowest, highest = int(area_fit.argmin()), int(area_fit.argmax())
print(area_fit[[lowest, highest]])  # [ 24.  278.5]
pair_X, pair_y = X_fit[[lowest, highest]], y_fit[[lowest, highest]]
print(pair_y.squeeze(1))            # tensor([ 74000., 885000.])
# 9.45
print(round((pair_X ** 2).mean().item(), 2))

torch.manual_seed(42)
tiny = nn.Linear(1, 1)
optimizer = optim.SGD(tiny.parameters(), lr=0.1)
for epoch in range(1, 501):
    loss = loss_fn(tiny(pair_X), pair_y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if epoch % 100 == 0:
        print(epoch, f"{loss.item():.4e}")
with torch.no_grad():
    print(tiny(pair_X).squeeze(1))
print(torch.tensor(885000.03))

100 3.2069e+04
200 3.5645e-02
300 3.5645e-02
400 3.5645e-02
500 3.5645e-02
tensor([ 74000.0625, 884999.7500])
tensor(885000.)

Two listings fix a line exactly, and the loop finds it: the loss sits at a few hundredths from epoch 200, 3.5645e-02 at every hundredth epoch printed, and the predictions sit within a quarter of a unit of the prices. The floor is not the prices, which float32 holds exactly, but the loop's own float32 numbers: a float32 near 885,000 moves in steps of 0.0625, which is why torch.tensor(885000.03) prints tensor(885000.), and the weight and bias, both in the hundreds of thousands, move in steps of hundredths at their size, so the loop can only land on a grid of lines and settles into two of them, each a few hundredths in loss and at most a quarter of a unit from the prices, without landing on the exact line the same grid holds. The same loop in float64 reaches 2.8799e-20. Which two listings matters:

torch.manual_seed(42)
close = nn.Linear(1, 1)
optimizer = optim.SGD(close.parameters(), lr=0.1)
for epoch in range(20000):
    loss = loss_fn(close(X_fit[:2]), y_fit[:2])
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
print(area_fit[:2], f"{loss.item():.4e}")  # [59.5 49.8] 9.8938e-01

Two inputs close together leave the loss nearly flat along one direction of the two parameters, because many lines pass close to both points, and the steps along a flat direction are small. The learning rate is the pair's too. mean(z ** 2) on these two rows is 9.45 against 1 on all the fit rows, the rate a loss tolerates shrinks as that number grows, and 0.5, the one-step rate of Lesson 4.2.1 on all 2,875 rows, on the pair:

torch.manual_seed(42)
fast = nn.Linear(1, 1)
optimizer = optim.SGD(fast.parameters(), lr=0.5)
for epoch in range(1, 101):
    loss = loss_fn(fast(pair_X), pair_y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    if epoch in (1, 10, 20, 100):
        print(epoch, f"{loss.item():.4e}")

1 3.9435e+11
10 2.1734e+28
20 inf
100 nan

Eight rows have no exact line through them, and their floor is least squares', Lesson 2.3.4's lstsq on a column of ones beside the eight standardized areas, whose sum of squared residuals over eight is the lowest mean squared error any line reaches on them:

torch.manual_seed(42)
eight = nn.Linear(1, 1)
optimizer = optim.SGD(eight.parameters(), lr=0.1)
for epoch in range(500):
    loss = loss_fn(eight(X_fit[:8]), y_fit[:8])
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
z8 = X_fit[:8].squeeze(1).double().numpy()
prices8 = y_fit[:8].squeeze(1).double().numpy()
design = np.column_stack([np.ones(8), z8])
w8, ssr, rank, sv = np.linalg.lstsq(design, prices8)
# 1.3714e+10 1.3714e+10
print(f"{loss.item():.4e}", f"{ssr[0] / 8:.4e}")

A loop that reaches the floor on two rows and on eight computes the right loss, moves the right way and stops where least squares stops.

DON'T:

• DON'T read a loss that falls as a loop that works. The seeded model's 1.6543e+11 falls at the first step of a loop whose zeroing is missing or whose target has the wrong shape, and the baseline is the first line it has to cross.
• DON'T pick the tiny batch from neighbors. The first two fit rows took 20,000 epochs to reach 0.9894, and the check would report a broken loop that was only slow.
• DON'T carry a learning rate from one batch to another. The rate that fitted all 2,875 rows in one step sent two of them to nan.

The Loss Curve, Drawn and Checked

What. The loss curve is two lists with one number per epoch, the training loss (the loss on the fit rows that the epoch's step was taken from) and the validation loss (the loss on the validation rows after that step), collected as Python floats through .item() and drawn as two lines on one Axes. Why. A loss printed every tenth epoch is a sample, and the two curves are the record: when the validation loss stopped falling, whether it turned up, how far the two sit apart, and whether either moved after the first fifty epochs. Every regularization tool of Section 4.4 reads this chart. When. Every loop, every run. The lists cost one float per epoch, and the chart is checked by printing what it holds, Lesson 2.7.2's rule, before anyone looks at it.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn, optim
from prep import X_fit, y_fit, X_val, y_val

loss_fn = nn.MSELoss()

def validate(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = loss_fn(model(X), y)
    model.train()
    return loss.item()

torch.manual_seed(42)
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses = [], []
for epoch in range(1, 201):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    train_losses.append(loss.item())
    val_losses.append(validate(model, X_val, y_val))
    if epoch in (1, 10, 15, 50, 200):
        print(epoch, f"{train_losses[-1]:.4e}", f"{val_losses[-1]:.4e}")

1 1.9076e+11 1.0929e+11
10 2.2577e+10 1.4862e+10
15 1.9823e+10 1.3951e+10
50 1.9491e+10 1.4118e+10
200 1.9491e+10 1.4118e+10

Two lines are new, train_losses.append(loss.item()) and val_losses.append(validate(model, X_val, y_val)), so an epoch's two numbers are one step apart, and both losses fall by a factor of seven or eight in ten epochs.

# <class 'float'> 200
print(type(train_losses[0]), len(val_losses))
lowest = int(np.argmin(val_losses))
# 15 1.3951e+10
print(lowest + 1, f"{val_losses[lowest]:.4e}")

np.argmin returns an index, and the epoch is one more. The chart is Lesson 2.7.2's figure:

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 201), train_losses, label="fit rows")
ax.plot(range(1, 201), val_losses, label="validation rows")
ax.set_yscale("log")
ax.set_xlabel("Epoch")
ax.set_ylabel("Mean squared error")
ax.set_title("Price on standardized area, one neuron")
ax.legend()
fig.savefig("loss_curve.png", dpi=150, bbox_inches="tight")
print(len(ax.lines))                 # 2
print(len(ax.lines[1].get_ydata()))  # 200
# 15
print(int(np.argmin(ax.lines[1].get_ydata())) + 1)
# ['fit rows', 'validation rows']
print([t.get_text() for t in ax.get_legend().get_texts()])
plt.close(fig)

ax.set_yscale("log") is 2.7.2's log axis, and it is what makes the chart readable: the first epoch's losses are eight to ten times the last ones, and on a linear axis the 190 epochs that matter would be one flat line along the bottom. What the lists must never hold:

kept = []
for epoch in range(3):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    kept.append(loss)
print(kept[0])
fig, ax = plt.subplots()
try:
    ax.plot(kept)
except RuntimeError as err:
    print(err)
plt.close(fig)

tensor(1.9491e+10, grad_fn=<MseLossBackward0>)
Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.

Each tensor in kept holds the whole forward pass over 1,725 rows that produced it for as long as the list lives, and .item() at the moment of collection is what keeps a 1,000-epoch loop's memory flat.

DON'T:

• DON'T collect the losses as tensors. Each keeps its graph alive, and ax.plot refuses the list.
• DON'T read the curve from the printed epochs. None of the five showed the lowest validation loss.
• DON'T trust a chart you have not printed. The picture proves nothing, and the prints under it prove the figure holds the two lists.

Reading the Two Curves Together

What. Two curves are read for three things: whether both fall and settle, where the validation curve is lowest, and whether it rises again while the training curve keeps falling. The last is Lesson 3.6.1's gap, the model fitting its own rows better than new ones, drawn epoch by epoch instead of row by row. Why. The training loss cannot rise for long under a working loop, because every step lowers it on the rows it is computed from. Overfitting shows on the validation curve, as a turn. When. After every run, on the chart, before any number from the run is reported.

The one-feature curves first, continuing the script above:

# True
print(all(v < t for v, t in zip(val_losses, train_losses)))
# 1.9491e+10 1.4118e+10
print(f"{train_losses[-1]:.4e}", f"{val_losses[-1]:.4e}")

The validation loss sits below the training loss at all 200 epochs because that is what these rows happen to be: by the split's draw the 575 validation listings are easier to price from their area than the 1,725 fit rows, and a validation loss below a training loss is not a defect and not a sign of a leak, any more than Lesson 3.1.4's test R² above its training R² was. No gap opens, because two parameters cannot store 1,725 listings: the model is a line, as good on new listings as on old ones, with nothing for 185 more epochs to memorize.

A gap needs capacity, which Section 4.4 gives the model for real. Here it is made up for the purpose, Lesson 3.3.2's device: 150 of the fit rows, their standardized area beside 100 columns of seeded noise, the same 100 columns drawn for the validation rows, and a neuron with 101 weights.

torch.manual_seed(42)
noise_fit = torch.randn(150, 100)
noise_val = torch.randn(575, 100)
X_fit_noise = torch.cat([X_fit[:150], noise_fit], dim=1)
X_val_noise = torch.cat([X_val, noise_val], dim=1)
y_fit_150 = y_fit[:150]
# torch.Size([150, 101]) torch.Size([575, 101])
print(X_fit_noise.shape, X_val_noise.shape)

torch.cat is Lesson 4.1.2's, along dimension 1 here, so the noise columns sit beside the area column. The noise is random by construction, no weight on it can help on the validation rows, whose noise was drawn separately, and every weight the loop gives it is memorization.

torch.manual_seed(42)
noisy = nn.Linear(101, 1)
optimizer = optim.SGD(noisy.parameters(), lr=0.1)
noisy_train, noisy_val = [], []
for epoch in range(1, 1001):
    loss = loss_fn(noisy(X_fit_noise), y_fit_150)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    noisy_train.append(loss.item())
    noisy_val.append(validate(noisy, X_val_noise, y_val))
for epoch in (1, 14, 50, 200, 1000):
    print(epoch, f"{noisy_train[epoch - 1]:.4e}", f"{noisy_val[epoch - 1]:.4e}")
turn = int(np.argmin(noisy_val))
print(turn + 1, f"{noisy_val[turn]:.4e}", round(noisy_val[-1] / noisy_val[turn], 2))

1 2.3087e+11 1.1089e+11
14 2.4966e+10 4.6389e+10
50 1.6229e+10 6.3779e+10
200 1.4929e+10 9.6789e+10
1000 1.4889e+10 1.0540e+11
14 4.6389e+10 2.27

The training loss falls and settles at 1.4889e+10, below the 1.9491e+10 the area reached on all 1,725 rows, because 101 weights on 150 rows can fit the rows' noise as well as their areas, and the validation loss turns at epoch 14 and climbs for the 986 epochs after it: every one of them made the model worse at the 575 rows it was never fitted on, while the training loss fell through epoch 543 and then sat at four digits. The same 150 rows with the area alone:

torch.manual_seed(42)
small = nn.Linear(1, 1)
optimizer = optim.SGD(small.parameters(), lr=0.1)
for epoch in range(1000):
    loss = loss_fn(small(X_fit[:150]), y_fit_150)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
# 4.0733e+10 1.4946e+10
print(f"{loss.item():.4e}", f"{validate(small, X_val, y_val):.4e}")

The noise columns lowered the training loss and raised the validation loss, Lesson 3.3.2's finding, and the curve shows when. The chart, with the turn marked:

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 1001), noisy_train, label="fit rows")
ax.plot(range(1, 1001), noisy_val, label="validation rows")
ax.axvline(turn + 1, color="black", linestyle="--", label="lowest validation loss")
ax.set_yscale("log")
ax.set_xlabel("Epoch")
ax.set_ylabel("Mean squared error")
ax.set_title("Area and 100 noise columns, 150 fit rows")
ax.legend()
fig.savefig("gap_curve.png", dpi=150, bbox_inches="tight")
# 3 14
print(len(ax.lines), ax.lines[2].get_xdata()[0])
plt.close(fig)

ax.axvline is 2.7.2's vertical line, drawn at epoch 14 and counted among the lines, so the Axes holds three. In gap_curve.png the two curves fall together for fourteen epochs, part at the dashed line, and run in opposite directions from there, and the dashed line is where the loop should have stopped. Stopping there, by watching the validation loss and keeping the parameters of its lowest epoch, is early stopping, one of Lesson 4.4.2's two tools, named here and built there. Lesson 3.6.1's learning curve had rows on its axis and asked whether more data would help, and this one has epochs and asks when more training stops helping.

DON'T:

• DON'T read a validation loss below the training loss as a bug. These 575 listings happen to be easier to price than the 1,725, and nothing leaked.
• DON'T read a rising validation curve as a learning rate too large. A rate too large sends both curves up, as 0.5 sent the pair's loss to nan, and the noisy model's training curve never turned up, falling at every epoch through 543 and then sitting at 1.4889e+10.
• DON'T report the epoch-1000 model because its training loss is the lowest. The validation rows priced its parameters worse at every epoch after the fourteenth.

The Test Rows, Scored Once

What. The test score is one number per metric, taken on the test rows by the model the validation rows chose, after every choice, with Lesson 3.1.4's mean_absolute_error and r2_score on predictions computed under both switches and handed to NumPy through .numpy(). Why. The validation loss was read 200 times and a choice could rest on any reading, so the validation rows are no longer new to the model. The test rows reached no step, no number and no choice, and their one score is the only number here that stands in for a listing the model has never seen. When. Once, at the end, and never again for this model. A second score after a change is a validation score under the wrong name.

import numpy as np
import torch
from torch import nn, optim
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from prep import X_fit, y_fit, X_test, area_fit, price_fit, area_test, price_test, mean, sd

loss_fn = nn.MSELoss()
torch.manual_seed(42)
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
for epoch in range(200):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

model.eval()
with torch.no_grad():
    test_pred = model(X_test).squeeze(1).numpy()
# 88,360
print(f"{mean_absolute_error(price_test, test_pred):,.0f}")
# 0.562
print(round(r2_score(price_test, test_pred), 3))

The loop is the fourth subtitle's without the lists, and .numpy() is allowed because nothing computed under no_grad requires a gradient. The weights read back in square meters, Lesson 4.2.1's reading, against the one-solve fit on the same rows:

w_sqm = model.weight.item() / sd.item()
b_sqm = model.bias.item() - w_sqm * mean.item()
print(round(w_sqm), round(b_sqm))  # 3408 45865

solve = LinearRegression().fit(area_fit.reshape(-1, 1), price_fit)
# 3408 45865
print(round(solve.coef_[0]), round(solve.intercept_))
# True
print(np.allclose([w_sqm, b_sqm], [solve.coef_[0], solve.intercept_], rtol=1e-4))
solve_pred = solve.predict(area_test.reshape(-1, 1))
# 88,360
print(f"{mean_absolute_error(price_test, solve_pred):,.0f}")
# 0.562
print(round(r2_score(price_test, solve_pred), 3))

The same two numbers to the unit and the same test score to the last digit: the test score is the line's and not the route's, and the two test scores are now spent.

DON'T:

• DON'T score the test rows more than once for one model. The second score is a choice waiting to be made by it.
• DON'T choose the epoch, the learning rate or the feature by the test score. The epoch came from the validation curve, and the test rows reached no decision, which is what their 88,360 is worth.
• DON'T set the test MAE beside the loss curve. The curve is a mean of squared misses and 88,360 is the mean miss in currency, Lesson 3.1.4's distinction.

Lab Exercises

Lab Exercise 1: Checks for a Rooms Model

In rooms_checks.py, price the listings by their room count, all 3,000 of them, since no listing lacks one. Split them as prep.py splits the listings with an area, both calls with random_state=42, and print the three sizes. Standardize the room counts with the fit rows' mean and standard deviation, as (n, 1) float32 tensors against (n, 1) prices, and print the two numbers, the fit and validation shapes, and the validation rows' standardized mean and standard deviation.

Print the fit rows' mean price beside the baseline it gives, then the validation prices' variance. Seed and create the neuron, score the validation rows once with a graph and once under both switches, print whether each result requires a gradient and whether the two are equal, and print the switched loss beside its ratio to the baseline. Fit the first fit row with the fewest rooms and the first with the most for 500 epochs at 0.1, printing the two room counts and prices, mean(z ** 2) of the pair, the loss at the end and the two predictions. Finish by fitting the first eight fit rows for 500 epochs and printing the loop's loss beside least squares' floor for those rows.

Lab Exercise 2: The Distance Model's Curve, Read Against the Baseline

In distance_curve.py, price the listings by their distance from the center, all 3,000, with Lab 1's split and standardization, printing the fit rows' two numbers and the baseline. Run the module loop for 200 epochs, collecting both losses as floats, and print them at epochs 1, 10, 20, 50 and 200. Print the epoch of the lowest validation loss with its value, the first epoch at which the validation loss fell below the baseline with how many epochs sit below it, and whether the validation loss is below the training loss at every epoch.

Draw both curves on one log-scaled Axes with a dashed horizontal line at the baseline, all three labelled, save it as distance_curve.png, and print how many lines the Axes holds, how many points the validation line has and the epoch of its lowest point. Score the test rows once with MAE and R², print the weight per kilometer and the intercept beside LinearRegression's on the same fit rows, to the unit, then the library fit's test MAE and R², and the test rows' mean price beside the fit rows'. Say in a comment what the baseline, the R² and the two curves agree on.

import numpy as np
import pandas as pd
import torch
from torch import nn, optim
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
rooms = listings["rooms"].to_numpy()
price = listings["price"].to_numpy()
rooms_train, rooms_test, price_train, price_test = train_test_split(
    rooms, price, test_size=0.2, random_state=42
)
rooms_fit, rooms_val, price_fit, price_val = train_test_split(
    rooms_train, price_train, test_size=0.25, random_state=42
)
print(len(rooms_fit), len(rooms_val), len(rooms_test))

fit_tensor = torch.tensor(rooms_fit, dtype=torch.float32)
mean, sd = fit_tensor.mean(), fit_tensor.std(correction=0)
print(round(mean.item(), 2), round(sd.item(), 4))

def to_tensors(room_rows, price_rows):
    z = (torch.tensor(room_rows, dtype=torch.float32) - mean) / sd
    y = torch.tensor(price_rows, dtype=torch.float32)
    return z.unsqueeze(1), y.unsqueeze(1)

X_fit, y_fit = to_tensors(rooms_fit, price_fit)
X_val, y_val = to_tensors(rooms_val, price_val)
print(X_fit.shape, X_val.shape)
print(round(X_val.mean().item(), 4), round(X_val.std(correction=0).item(), 4))

loss_fn = nn.MSELoss()
mean_price = y_fit.mean()
baseline = loss_fn(torch.full_like(y_val, mean_price.item()), y_val)
print(f"{mean_price.item():,.2f}", f"{baseline.item():.4e}")
print(f"{y_val.var(correction=0).item():.4e}")

torch.manual_seed(42)
model = nn.Linear(1, 1)
recorded = loss_fn(model(X_val), y_val)
model.eval()
with torch.no_grad():
    start = loss_fn(model(X_val), y_val)
model.train()
print(recorded.requires_grad, start.requires_grad, torch.equal(recorded, start))
print(f"{start.item():.4e}", round(start.item() / baseline.item(), 2))

1800 600 600
3.59 1.7239
torch.Size([1800, 1]) torch.Size([600, 1])
-0.0019 0.9856
381,243.34 3.7652e+10
3.7635e+10
True False True
1.8617e+11 4.94

fewest, most = int(rooms_fit.argmin()), int(rooms_fit.argmax())
print(rooms_fit[[fewest, most]], price_fit[[fewest, most]])
pair_X, pair_y = X_fit[[fewest, most]], y_fit[[fewest, most]]
print(round((pair_X ** 2).mean().item(), 2))
torch.manual_seed(42)
tiny = nn.Linear(1, 1)
optimizer = optim.SGD(tiny.parameters(), lr=0.1)
for epoch in range(500):
    loss = loss_fn(tiny(pair_X), pair_y)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
print(f"{loss.item():.4e}")
with torch.no_grad():
    print(tiny(pair_X).squeeze(1))

torch.manual_seed(42)
eight = nn.Linear(1, 1)
optimizer = optim.SGD(eight.parameters(), lr=0.1)
for epoch in range(500):
    loss = loss_fn(eight(X_fit[:8]), y_fit[:8])
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
z8 = X_fit[:8].squeeze(1).double().numpy()
prices8 = y_fit[:8].squeeze(1).double().numpy()
design = np.column_stack([np.ones(8), z8])
w8, ssr, rank, sv = np.linalg.lstsq(design, prices8)
print(f"{loss.item():.4e}", f"{ssr[0] / 8:.4e}")

[ 1 10] [102000 695000]
8.05
1.9531e-03
tensor([101999.9375, 694999.9375])
2.6614e+09 2.6614e+09

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch
from torch import nn, optim
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error, r2_score
from sklearn.model_selection import train_test_split

listings = pd.read_csv("listings.csv")
distance = listings["distance_km"].to_numpy()
price = listings["price"].to_numpy()
distance_train, distance_test, price_train, price_test = train_test_split(
    distance, price, test_size=0.2, random_state=42
)
distance_fit, distance_val, price_fit, price_val = train_test_split(
    distance_train, price_train, test_size=0.25, random_state=42
)
fit_tensor = torch.tensor(distance_fit, dtype=torch.float32)
mean, sd = fit_tensor.mean(), fit_tensor.std(correction=0)
print(round(mean.item(), 2), round(sd.item(), 4))

def to_tensors(distance_rows, price_rows):
    z = (torch.tensor(distance_rows, dtype=torch.float32) - mean) / sd
    y = torch.tensor(price_rows, dtype=torch.float32)
    return z.unsqueeze(1), y.unsqueeze(1)

X_fit, y_fit = to_tensors(distance_fit, price_fit)
X_val, y_val = to_tensors(distance_val, price_val)
X_test, y_test = to_tensors(distance_test, price_test)

loss_fn = nn.MSELoss()
baseline = loss_fn(torch.full_like(y_val, y_fit.mean().item()), y_val)
print(f"{baseline.item():.4e}")

def validate(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = loss_fn(model(X), y)
    model.train()
    return loss.item()

torch.manual_seed(42)
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses = [], []
for epoch in range(1, 201):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    train_losses.append(loss.item())
    val_losses.append(validate(model, X_val, y_val))
    if epoch in (1, 10, 20, 50, 200):
        print(epoch, f"{train_losses[-1]:.4e}", f"{val_losses[-1]:.4e}")

lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4e}")
below = [epoch + 1 for epoch, v in enumerate(val_losses) if v < baseline.item()]
print(below[0], len(below))
print(all(v < t for v, t in zip(val_losses, train_losses)))

5.55 3.0727
3.7652e+10
1 1.8392e+11 1.3320e+11
10 4.1174e+10 3.9645e+10
20 3.8585e+10 3.7683e+10
50 3.8555e+10 3.7627e+10
200 3.8555e+10 3.7627e+10
68 3.7627e+10
23 178
True

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 201), train_losses, label="fit rows")
ax.plot(range(1, 201), val_losses, label="validation rows")
ax.axhline(baseline.item(), color="black", linestyle="--", label="baseline")
ax.set_yscale("log")
ax.set_xlabel("Epoch")
ax.set_ylabel("Mean squared error")
ax.set_title("Price on standardized distance, one neuron")
ax.legend()
fig.savefig("distance_curve.png", dpi=150, bbox_inches="tight")
print(len(ax.lines), len(ax.lines[1].get_ydata()))
print(int(np.argmin(ax.lines[1].get_ydata())) + 1)
plt.close(fig)

model.eval()
with torch.no_grad():
    test_pred = model(X_test).squeeze(1).numpy()
print(f"{mean_absolute_error(price_test, test_pred):,.0f}")
print(round(r2_score(price_test, test_pred), 3))
w_km = model.weight.item() / sd.item()
b_km = model.bias.item() - w_km * mean.item()
solve = LinearRegression().fit(distance_fit.reshape(-1, 1), price_fit)
print(round(w_km), round(solve.coef_[0]))
print(round(b_km), round(solve.intercept_))
solve_pred = solve.predict(distance_test.reshape(-1, 1))
print(f"{mean_absolute_error(price_test, solve_pred):,.0f}")
print(round(r2_score(price_test, solve_pred), 3))
print(f"{price_test.mean():,.0f}", f"{price_fit.mean():,.0f}")
# The validation loss crossed below the baseline at epoch 23 and
# settled 0.07 percent under it, the test R² is -0.001, and both
# curves flattened by epoch 50: the loop works and the distance
# alone prices a listing no better than the fit rows' mean does.

3 200
68
147,587
-0.001
1362 1362
373686 373686
147,587
-0.001
374,687 381,243

Theory Assessment

Q1: One Pair of Numbers for Three Sets

The fit rows' mean area is 99.44 with a standard deviation of 44.6871, the validation rows' own numbers are 96.03 and 43.9933, and all 2,875 listings' are 99.02 and 44.0589. Say which pair standardizes each of the three sets and why, what the validation rows' standardized mean of -0.0762 says about the pair that was used, and which of Lesson 3.1.2's breaks each of the other two choices commits.

Hint: Ask which rows the model is allowed to know anything about, and which listing the validation rows are standing in for. A standardized set whose mean prints exactly zero was standardized by its own numbers.

Q2: Two Switches on a Model That Ignores One

torch.equal said the validation loss under no_grad equals the one computed with a graph, and the loss in evaluation mode equals the loss out of it. A teammate concludes that no_grad is a speed trick and eval() does nothing, and removes both from the validation pass along with the train() after it, until the team adds dropout. Say what each switch changes on this model and what each will change in Lesson 4.4.2, what the graph on a validation loss makes possible, and what the loop does once train() is gone.

Hint: Two switches, two different things they govern: one is about the record, the other about a flag every layer reads. Then ask what one backward() on the recorded validation loss leaves behind, and what a flag that is set once and never reset does to the fit steps.

Q3: Two Checks, Then Two Curves

On the same 150 fit rows, the noisy model ends with a training loss of 1.4889e+10 and a validation loss of 1.0540e+11, lowest at epoch 14 at 4.6389e+10, and the area-only model ends at 4.0733e+10 and 1.4946e+10. Both loops passed the baseline and the tiny-batch checks. Say which model you would report and at which epoch, why the two training losses read alone point the other way, and what each of the two checks catches that this comparison cannot.

Hint: The training loss is the number the loop drives down on the rows it was computed from. The validation loss is the number the model is judged by. Then ask what a loop with a wrong zeroing or a wrong shape prints on two rows, and what a loop that learned nothing prints beside the mean price.

Q4: A Lower Number From the Same Model

Read this script without running it, then say what the teammate who wrote it got right and wrong about its one printed line.

import numpy as np
import torch
from torch import nn, optim
from prep import X_fit, y_fit, y_val, area_val

loss_fn = nn.MSELoss()
own = torch.tensor(area_val, dtype=torch.float32)
X_val_own = ((own - own.mean()) / own.std(correction=0)).unsqueeze(1)

torch.manual_seed(42)
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
val_losses = []
for epoch in range(1, 201):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    val_losses.append(loss_fn(model(X_val_own), y_val).item())
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4e}")

It prints 13 1.3903e+10, and the teammate reports that this loop trains a better model than the fourth subtitle's, whose lowest validation loss was 1.3951e+10 at epoch 15, and converges two epochs sooner. Say what in this script changed the model, what changed only the number, what the validation pass is missing and what that costs here, and what the report should have said.

Hint: Ask what the parameters depend on, line by line, and whether anything the validation rows went through reaches them. A lower number on a different input is a different curve, not a different model.

Assignment

Assignment: The Checkpoint at the Turn

The noisy model's validation loss was lowest at epoch 14 and the loop ran on to 1,000, reporting a model worse than one it had passed through. Write early_stop.py so that the loop keeps the parameters of the epoch with the lowest validation loss, stops on its own once that loss has gone 50 epochs without improving, and restores the kept parameters at the end. Build the 150-row table with its 100 noise columns and the validation table exactly as the Theory built them, with the same seeds, and seed the nn.Linear(101, 1) the same way.

Three things this lesson did not use. model.state_dict(), Lesson 4.2.2's dictionary of a model's parameters, hands back tensors over the parameters' own memory, so a change to the parameters shows in a dictionary taken earlier, and a copy taken for keeping is made with copy.deepcopy from Python's standard copy module, which copies the dictionary and every tensor in it. model.load_state_dict(state) copies the values of state back into the model's parameters and returns a report naming any key that did not match. A patience rule (stop once the validation loss has gone a set number of epochs without a new low) is the stopping rule of Lesson 4.4.2's early stopping, built once here by hand.

Before the loop, take the state dict once as live and a deep copy once as frozen, and print the keys, the weight's shape and the two biases to four decimals. In the loop keep best_loss, best_epoch and best_state, a deep copy taken at every new low, with a counter that resets at a new low and breaks the loop when it reaches 50. After it, print the epoch the loop stopped at, the best epoch, the best validation loss and the last one, then their ratio to two decimals. Print live's bias and the model's to the unit, frozen's to four decimals and best_state's to the unit, and whether live["weight"] equals model.weight. Print the mean size of the 100 noise weights now and in best_state, to the unit. Restore best_state, print what load_state_dict returns, the restored model's validation loss, and whether its weight equals the kept one.

Hints:

• copy.deepcopy(model.state_dict()) is the line that keeps a checkpoint, and it runs only at a new low, so an epoch without one costs nothing. break leaves epoch at the epoch that ran last.
• The noise weights are model.weight[0, 1:], and .abs().mean() is their mean size.

import copy
import torch
from torch import nn, optim
from prep import X_fit, y_fit, X_val, y_val

loss_fn = nn.MSELoss()

def validate(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = loss_fn(model(X), y)
    model.train()
    return loss.item()

torch.manual_seed(42)
noise_fit = torch.randn(150, 100)
noise_val = torch.randn(575, 100)
X_fit_noise = torch.cat([X_fit[:150], noise_fit], dim=1)
X_val_noise = torch.cat([X_val, noise_val], dim=1)
y_fit_150 = y_fit[:150]

torch.manual_seed(42)
model = nn.Linear(101, 1)
optimizer = optim.SGD(model.parameters(), lr=0.1)
live = model.state_dict()
frozen = copy.deepcopy(model.state_dict())
print(list(live.keys()), live["weight"].shape)
print(round(live["bias"].item(), 4), round(frozen["bias"].item(), 4))

best_loss, best_epoch, best_state = float("inf"), 0, None
patience, since_best = 50, 0
for epoch in range(1, 1001):
    loss = loss_fn(model(X_fit_noise), y_fit_150)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    val_loss = validate(model, X_val_noise, y_val)
    if val_loss < best_loss:
        best_loss, best_epoch, since_best = val_loss, epoch, 0
        best_state = copy.deepcopy(model.state_dict())
    else:
        since_best += 1
        if since_best == patience:
            break
print(epoch, best_epoch, f"{best_loss:.4e}", f"{val_loss:.4e}")
print(round(val_loss / best_loss, 2))

['weight', 'bias'] torch.Size([1, 101])
0.0249 0.0249
64 14 4.6389e+10 7.0338e+10
1.52

print(round(live["bias"].item()), round(model.bias.item()))
print(round(frozen["bias"].item(), 4), round(best_state["bias"].item()))
print(torch.equal(live["weight"], model.weight))
print(round(model.weight[0, 1:].abs().mean().item()), round(best_state["weight"][0, 1:].abs().mean().item()))

report = model.load_state_dict(best_state)
print(report)
print(f"{validate(model, X_val_noise, y_val):.4e}")
print(torch.equal(model.weight, best_state["weight"]))

411845 411845
0.0249 324667
True
18537 12841
<All keys matched successfully>
4.6389e+10
True

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the full list of checks a loop passes before its curves are trusted, or the exact contract of the context the validation pass runs in.

• Blog post - "A Recipe for Training Neural Networks", step 2, "Set up the end-to-end training/evaluation skeleton + get dumb baselines" by Andrej Karpathy. Fix the seed, read the loss at initialization, beat an input-independent baseline and overfit one batch of as few as two examples are the checks this lesson's third subtitle ran, and the step goes on to visualizing what enters the model and how its predictions move, with four further steps after it.
• Official docs - PyTorch 2.14, torch.no_grad. The contract of the context manager: every result computed inside it has requires_grad=False even when an input requires a gradient, memory is saved on computations that would otherwise be recorded, the same object works as a decorator, and the one exception is a tensor created inside it with requires_grad=True passed as an argument.

Summary

The loop did not change. The five steps of Lesson 4.2.2 ran exactly as before, on fewer rows, and everything this lesson added sat around them: a split made before the first tensor, two numbers computed on the fit rows and applied to every set, a function that scores the validation rows under two switches and hands back a float, two lists, and a chart printed before it was looked at. Discipline turned out to be the order of the work, as it was for LinearRegression in Lesson 3.1.2, with one new fact: a loop that is judged every epoch has to be judged on rows it never fits, by a pass that cannot reach its parameters.

The two checks and the two curves are the same idea read at different times. Before the first full run, the baseline says what a loop has to beat and the tiny batch says whether the loop can fit anything at all. After the run, the curves say when the model stopped improving on rows it never saw and whether it then got worse on them: the area model's two curves fell together because a line has nothing to memorize, and 100 noise columns on 150 rows drew the gap Lesson 3.6.1 drew with rows, now with epochs along the axis and a turn at the fourteenth.

What the turn asks for is a tool, and this lesson stopped at naming it. Lesson 4.4.2 watches the validation loss and keeps the parameters of its lowest epoch, as the Assignment did once by hand, and the rest of Section 4.4 gives the model the real capacity the noise columns stood in for. Every one of those tools is read off the chart this lesson drew.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import torch
from torch import nn, optim

def validate(model, X, y):
    model.eval()           # the flag every layer reads
    with torch.no_grad():  # no record, no memory, no backward()
        loss = loss_fn(model(X), y)
    model.train()          # back before the next fit step
    return loss.item()     # a float, never a tensor

train_losses, val_losses = [], []
for epoch in range(1, 201):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    train_losses.append(loss.item())
    val_losses.append(validate(model, X_val, y_val))
lowest = int(np.argmin(val_losses)) + 1

baseline = loss_fn(torch.full_like(y_val, y_fit.mean().item()), y_val)
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 201), train_losses, label="fit rows")
ax.plot(range(1, 201), val_losses, label="validation rows")
ax.set_yscale("log")
ax.legend()
fig.savefig("loss_curve.png", dpi=150, bbox_inches="tight")
# 2 200
print(len(ax.lines), len(ax.lines[1].get_ydata()))
plt.close(fig)

model.eval()
with torch.no_grad():
    # once, at the end
    test_pred = model(X_test).squeeze(1).numpy()
mean_absolute_error(price_test, test_pred)  # 88,360 on the area model