Overfitting on Purpose - Capacity and Weight Decay

Context

Overfitting on purpose is a network handed far more parameters than its rows can pin down, 10,241 for 200 listings, and trained until it prices those 200 almost exactly while its validation loss climbs. Weight decay is one setting of the optimizer, weight_decay, that takes the same small share off every parameter at every step, on top of the gradient's move, so a parameter stays large only where the loss keeps pushing it back.

You make the gap before you fight it whenever a network has to be held back: choosing a width for a few hundred labeled rows, setting a decay copied from another team's training script, or deciding whether a smaller network or a penalty was the change that helped. Lesson 4.4.2 measures dropout and early stopping against the network this lesson overfits.

Without a gap there is nothing to trade: Lesson 4.3.1's four widths on all 1,725 fit rows ended every validation loss within 8 percent of its lowest. A tool measured together with another change is credited with that change too, which is how a smaller network gets praised for what the decay did, and a difference read at the end of a run can be smaller than what the run moves on its own between two epochs. Lesson 4.3.1 showed the training loss falling with every step up in width while the validation loss did not, and this lesson takes width to where that gap is wide open.

Goal

• Make the 512-unit network memorize the first 200 fit rows on purpose with the fit of slice_loop.py, and state the gap in three numbers: the final training loss, the final validation loss, and the lowest validation loss with its epoch.
• Build the capacity ladder from 1 to 512 hidden units on those 200 rows and identify the rung at which the validation loss stops following the training loss.
• Run the same network on all 1,725 fit rows and explain, from the parameters per row and the two runs, why a tool is measured on the slice and checked on the full rows before its effect is quoted.
• Apply weight decay with optim.SGD(weight_decay=...), verify its step against p - lr * (grad + wd * p) and against Module 3's squared cost with allclose, and track the weight norm it moves.
• Measure each change alone against the baseline in one table, read the end of every run beside the epoch before it, and reject a comparison in which two changes, or two thread counts, moved at once.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want the penalty on a network's weights, and the reason it is called a decay, before you put one in a training script.

• YouTube - "Regularization (C2W1L04)" by DeepLearningAI. Just under ten minutes, in the chapters its page lists: logistic regression from 0:26 with regularization added from 0:54, the regularization parameter from 1:00 and L2 regularization from 1:28, then a neural network from 4:33, its squared norm from 5:12 and the Frobenius norm of a matrix from 5:54.
• YouTube - "Lecture 12 - Regularization" by Yaser Abu-Mostafa, Caltech's CS 156, from 40:43. Its chapter on weight decay, then "Variations of weight decay" from 44:22 and "Even weight growth!" from 47:24, in a lecture its page sums up as putting the brakes on fitting the noise: hard and soft constraints, the augmented error and weight decay.

Table of Contents

• Memorizing a Small Slice on Purpose
• The Capacity Ladder, Read on the Slice
• The Gap on the Full Rows
• Weight Decay, the Shrink in the Step
• Measuring One Tool Alone

Theory

Every block in this lesson runs in ~/dev/bootcamp on the 2,875 listings with an area, split as Lesson 4.2.3 split them into 1,725 fit, 575 validation and 575 test rows and prepared as Lesson 4.3.1 prepared them, 18 columns against the price in hundreds of thousands. Every script imports that lesson's train_loop.py, and with it prep_net.py: network(width) seeds 42 inside and builds nn.Linear(18, width), nn.ReLU() and nn.Linear(width, 1), so two runs at one width start from the same parameters, and validate and score are that lesson's validation pass and its MAE and R² in currency. The loop is Lesson 4.2.3's, full batch, one step per epoch at optim.SGD's rate of 0.1. The first subtitle writes slice_loop.py, and no block scores the test rows. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Memorizing a Small Slice on Purpose

What. Memorizing (a model fitting each of its training rows' prices, noise included, until its training loss is near zero) is Lesson 3.5.1's overfitting carried to its end, done here on purpose: the first 200 of the 1,725 fit rows, network(512) with its 10,241 parameters, optim.SGD at 0.1 for 3,000 epochs, and the 575 validation rows unchanged as the judge. Regularization, which Lesson 3.4.1 met as a cost on large weights, means more in this section (every tool that trades training loss for validation loss). Why. A tool that trades training loss for validation loss has nothing to trade on a model whose two losses agree, so the gap comes first, wide enough that a tool's effect stands out of it. A loop that cannot make a network this wide memorize 200 rows has a defect: Lesson 4.2.3's tiny batch asked one neuron to fit two listings, and this asks a network with 51 parameters per row to fit 200. When. Once per network and set of rows, before the first tool is tried, and again whenever either changes.

The slice and its loop go into one file. Save it as slice_loop.py in ~/dev/bootcamp, because the rest of the section imports it, as Lesson 4.3.1 saved prep_net.py and train_loop.py for the rest of Section 4.3:

from prep_net import X_fit, y_fit, X_val, y_val
from train_loop import loss_fn, validate

# the first 200 fit rows, memorized on purpose
X_slice, y_slice = X_fit[:200], y_fit[:200]

def fit(model, optimizer, X, y, epochs, report=()):
    train_losses, val_losses = [], []
    for epoch in range(1, epochs + 1):
        loss = loss_fn(model(X), y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        train_losses.append(loss.item())
        val_losses.append(validate(model, X_val, y_val))
        if epoch in report:
            print(epoch, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}")
    return train_losses, val_losses

X_slice and y_slice are the first 200 rows of Lesson 4.3.1's fit tensors, standardized and scaled with the fit rows' numbers like every other set, so the validation rows need nothing new. fit is train_loop.run with the rows it fits as two arguments, because run fits all 1,725 every time: it records the loss each step was taken from and the validation loss after that step, so an epoch's two numbers are one step apart as in Lesson 4.2.3, prints the epochs named in report, and returns the two lists. The memorizing run, from a script that imports it:

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import optim
from prep_net import fit_rows, val_rows, X_val, y_val
from train_loop import network, validate, loss_fn, score
from slice_loop import X_slice, y_slice, fit

print(X_slice.shape, round(y_slice.mean().item(), 3))
model = network(512)
print(sum(p.numel() for p in model.parameters()))
print(f"{validate(model, X_val, y_val):.4f}")
optimizer = optim.SGD(model.parameters(), lr=0.1)
report = (1, 2, 10, 100, 300, 1000, 3000)
train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000, report)
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4f}", round(val_losses[-1] / val_losses[lowest], 1))

torch.Size([200, 18]) 4.009
10241
17.4358
1 22.9293 158.0769
2 152.9964 49.7290
10 6.8777 3.2553
100 1.0564 0.9002
300 0.4478 1.7195
1000 0.0839 3.0258
3000 0.0240 3.3360
38 0.7764 4.3

The slice's mean price is 4.009 in hundreds of thousands, and the seeded network scores 17.4358 on the validation rows before any step. The first step at 0.1 raised the slice's loss from 22.9293, the loss it was taken from, to 152.9964, the second line's training loss, and the validation loss to 158.0769, and by epoch 10 both are single digits. The validation loss is lowest at epoch 38, 0.7764, and climbs from there while the training loss keeps falling across the printed epochs, 0.4478 against 1.7195 at epoch 300 and 0.0240 against 3.3360 at epoch 3,000, where the validation loss is 4.3 times its lowest. In currency, and against one constant price:

slice_rows = fit_rows.iloc[:200]
mae, r2 = score(model, X_slice, slice_rows)
print(f"{mae:,.0f}", round(r2, 3))  # 11,749 0.996
mae, r2 = score(model, X_val, val_rows)
print(f"{mae:,.0f}", round(r2, 3))  # 96,693 -0.063
constant = torch.full_like(y_val, y_slice.mean().item())
# 3.2604
print(f"{loss_fn(constant, y_val).item():.4f}")

The network misses a slice listing by 11,749 on average with an R² of 0.996, and a validation listing by 96,693 with an R² of -0.063, below the zero that the validation rows' own mean would score. constant prices every validation listing at the slice's mean, Lesson 4.2.3's baseline built from the rows this loop fits, and scores 3.2604, under the network's 3.3360: at epoch 3,000 the memorizing network prices new listings worse than one number does, the loop that lesson said "has learned nothing worth keeping", made on purpose. What the end of the run does from one epoch to the next:

# [3.5514, 3.3364, 3.5527, 3.3374, 3.5526, 3.336]
print([round(v, 4) for v in val_losses[-6:]])
# [0.024, 0.024, 0.0241, 0.0241, 0.024, 0.024]
print([round(t, 4) for t in train_losses[-6:]])
moves = np.sign(np.diff(val_losses))  # 1.0 up, -1.0 down
# the epochs whose move repeats the one before
repeats = np.nonzero(moves[1:] == moves[:-1])[0] + 3
print(repeats[-1])                    # 67

From epoch 68 to the end every epoch moves the validation loss the other way from the epoch before, by about 0.2 at the end, while the training loss moves in its fourth decimal. One step after another carries the network between parameters that fit the 200 rows equally well and price the validation listings differently, memorizing seen one step at a time. A final number therefore depends on which epoch reads it, 3.5526 at epoch 2,999 and 3.3360 at 3,000, so every run in this lesson is read at epoch 3,000, and the fifth subtitle sets that end beside the epoch before it. Further into the run the printed digits also depend on how many processor threads PyTorch splits its arithmetic across, which the fifth subtitle measures, while the lowest point, 0.7764 at epoch 38, does not. The chart, Lesson 2.7.2's figure on a log axis, as Lesson 4.2.3 drew its curves:

above = [e + 1 for e, (t, v) in enumerate(zip(train_losses, val_losses)) if t > v]
print(above[-1])  # 128
fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 3001), train_losses, label="slice rows")
ax.plot(range(1, 3001), val_losses, label="validation rows")
ax.axvline(lowest + 1, color="black", linestyle="--", label="lowest validation loss")
ax.set_yscale("log")
ax.set_xlabel("Epoch")
ax.set_ylabel("Mean squared error")
ax.set_title("512 hidden units on 200 fit rows")
ax.legend()
fig.savefig("slice_curve.png", dpi=150, bbox_inches="tight")
# 3 3000
print(len(ax.lines), len(ax.lines[1].get_ydata()))
# 38
print(int(np.argmin(ax.lines[1].get_ydata())) + 1)
plt.close(fig)

above lists the epochs at which the slice's loss sat above the validation loss, and the last is 128, so from epoch 129 on the training loss is the lower of the two at every epoch. In slice_curve.png both lines drop from the first step's spike, the validation line turns at the dashed line and climbs most of the way by epoch 1,000, 3.0258 there against 3.3360 at the end, drawn as a thick band because it alternates, and the training line falls through three decades with small bumps on the way. It is the gap Lesson 4.2.3 drew with noise columns, made here by width alone.

DON'T:

• DON'T hunt for a bug in a loop that memorizes. Its five steps drove the slice's loss to 0.0240, which is the loop doing its job on rows it has the parameters to store.
• DON'T judge the memorizing network on its own rows. Its R² of 0.996 there sits beside a validation loss above the 3.2604 of one constant price.
• DON'T compare the ends of two runs read at different epochs. One run printed 3.5526 at epoch 2,999 and 3.3360 at epoch 3,000.
• DON'T take the first line's 158.0769 for a broken start. One step at 0.1 put the 512-unit network there, and by epoch 10 its losses were 6.8777 and 3.2553.

The Capacity Ladder, Read on the Slice

What. The capacity ladder is the slice's run at five widths, 1, 8, 32, 128 and 512 hidden units, with Lesson 3.6.1's capacity counted as parameters, 21 to 10,241, and every rung read by the three numbers and the end's ratio to the lowest. Why. Width is what opened the gap, and the ladder says at which rung the validation loss stops following the training loss, which is the capacity a regularization tool has to work against. Lesson 4.3.1's preview ran four of these widths on all 1,725 rows for 1,000 epochs, and on 200 rows the same widths say where the gap opens. When. Before a width is chosen for a set of rows, and again whenever the rows change, because a width is wide or narrow only against the rows it has to fit.

import numpy as np
import torch
from torch import optim
from sklearn.linear_model import LinearRegression
from prep_net import X_val, y_val
from train_loop import network, loss_fn
from slice_loop import X_slice, y_slice, fit

for width in (1, 8, 32, 128, 512):
    model = network(width)
    count = sum(p.numel() for p in model.parameters())
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
    lowest = int(np.argmin(val_losses))
    # width, parameters, the three numbers, the end over the lowest
    print(width, count, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", lowest + 1, f"{val_losses[lowest]:.4f}", round(val_losses[-1] / val_losses[lowest], 2))

1 21 1.6054 0.8570 18 0.6839 1.25
8 161 0.5990 3.4673 18 0.7484 4.63
32 641 0.0803 3.9878 18 0.7789 5.12
128 2561 0.0370 3.5977 20 0.7514 4.79
512 10241 0.0240 3.3360 38 0.7764 4.3

The training loss falls at every rung, from 1.6054 at one unit to 0.0240 at 512, and from 32 units on it is under a tenth. The validation loss splits from it one rung earlier: from eight units up every run ends between 4.3 and 5.12 times its lowest, while one unit ends 1.25 times its lowest. The lowest points barely move, 0.6839 to 0.7789, every one reached by epoch 38, so on this slice capacity decides how far the validation loss climbs after its minimum far more than the minimum itself. A 1,024-unit network at the same rate prints nan on the slice by epoch 10, so the ladder stops at 512. What one unit can and cannot do, beside least squares on the same 200 rows:

solve = LinearRegression().fit(X_slice.numpy(), y_slice.numpy())
slice_pred = torch.tensor(solve.predict(X_slice.numpy()), dtype=torch.float32)
val_pred = torch.tensor(solve.predict(X_val.numpy()), dtype=torch.float32)
# 1.6353 0.9055
print(f"{loss_fn(slice_pred, y_slice).item():.4f}", f"{loss_fn(val_pred, y_val).item():.4f}")

one = network(1)
optimizer = optim.SGD(one.parameters(), lr=0.1)
fit(one, optimizer, X_slice, y_slice, 3000)
with torch.no_grad():
    unit = one[1](one[0](X_slice))
# 171 1.8166 1.6993
print(int((unit > 0).sum()), round(one[2].weight.item(), 4), round(one[2].bias.item(), 4))

Least squares on the slice's 18 columns, the one solve Lesson 4.3.1 set beside its network, reaches 1.6353 on the slice and 0.9055 on the validation rows, and the one-unit network sits just under it at 1.6054 and 0.8570. Its unit is positive on 171 of the 200 rows, where the output is that unit's weighted sum times 1.8166 plus 1.6993, a line through the 18 columns, and on the other 29 the output is the constant 1.6993. One bend in a line is all one unit has, and a line cannot store 200 prices. Eight units have eight places to bend, and that is already enough to fit the slice's own noise: their validation loss ends 4.63 times its lowest while their training loss is still 0.5990.

DON'T:

• DON'T call one unit broken because its training loss stops at 1.6054. Least squares on the same 200 rows stops at 1.6353, and one unit is a line with one bend.
• DON'T wait for a training loss near zero before calling a gap open. Eight units ended their validation loss 4.63 times its lowest at a training loss of 0.5990.
• DON'T read the five lowest points as a ranking of widths. They sit within 0.10 of each other on one seed and one slice.

The Gap on the Full Rows

What. The same network, rate and 3,000 epochs on all 1,725 fit rows, read by the three numbers beside the slice's, with the parameters per row as the measure the rows set: 51.2 on the slice and 5.9 on the fit rows. Why. The slice was chosen to make the gap wide, and the full rows are what a model is trained on, so a tool is measured on the slice, where its effect stands out, and checked on the full rows before its effect is quoted for them. Only the rows differ between the two runs. When. Every time a number measured on the slice is about to be said about the listings.

import numpy as np
from torch import optim
from prep_net import fit_rows, val_rows, X_fit, y_fit, X_val
from train_loop import network, score
from slice_loop import fit

print(X_fit.shape, round(10241 / 200, 1), round(10241 / 1725, 1))
model = network(512)
optimizer = optim.SGD(model.parameters(), lr=0.1)
report = (1, 10, 100, 384, 1000, 2000, 3000)
train_losses, val_losses = fit(model, optimizer, X_fit, y_fit, 3000, report)
lowest = int(np.argmin(val_losses))
at_1000 = val_losses[999] / val_losses[lowest]
print(lowest + 1, f"{val_losses[lowest]:.4f}", round(at_1000, 2), round(val_losses[-1] / val_losses[lowest], 2))
above = [e + 1 for e, (t, v) in enumerate(zip(train_losses, val_losses)) if t > v]
print(above[-1])
mae, r2 = score(model, X_fit, fit_rows)
print(f"{mae:,.0f}", round(r2, 3))
mae, r2 = score(model, X_val, val_rows)
print(f"{mae:,.0f}", round(r2, 3))
print([round(v, 4) for v in val_losses[-4:]])

torch.Size([1725, 18]) 51.2 5.9
1 19.9893 147.9466
10 5.1633 3.4116
100 0.7520 0.4724
384 0.6812 0.4501
1000 0.6268 0.4827
2000 0.5483 0.5330
3000 0.5089 0.5977
384 0.4501 1.07 1.33
2130
41,494 0.881
54,475 0.81
[0.6297, 0.5966, 0.6308, 0.5977]

Through epoch 1,000 this is Lesson 4.3.1's width-512 run, 0.6268 and 0.4827 there and the lowest validation loss 0.4501 at epoch 384, because fit on all the fit rows computes what run computed. Past it the gap keeps opening: the validation loss stands 1.07 times its lowest at epoch 1,000 and ends 1.33 times it at 3,000. It sits below the training loss for most of the run, the validation rows easier to price by the split's draw as Lesson 4.3.1 found, the last time at epoch 2,130, and above it at every epoch after. The training loss reaches 0.5089, about 21 times the slice's 0.0240: with 5.9 parameters per row the network does not memorize 1,725 listings in 3,000 epochs, and in currency it misses a fit listing by 41,494 and a validation listing by 54,475, against 11,749 and 96,693 on the slice. Its last four validation losses alternate too, by about 0.03. The same mechanism runs here, slower and smaller.

DON'T:

• DON'T quote the slice's numbers for the full rows. The same network, loop and epochs ended 4.3 times its lowest on the slice and 1.33 times on all 1,725 rows.
• DON'T read a validation loss below the training loss as a model without a gap. It was still below at epoch 2,130, long after it began climbing from its lowest at epoch 384.
• DON'T stop a full-rows run at 1,000 epochs and call its gap small. 1.07 times the lowest at epoch 1,000 was 1.33 times at 3,000.

Weight Decay, the Shrink in the Step

What. Weight decay (a penalty on the size of the parameters, applied as a shrink of every parameter at each step) is optim.SGD(model.parameters(), lr=0.1, weight_decay=wd), the setting Lesson 4.2.2's print(optimizer) showed at 0. Its step is p - lr * (grad + wd * p), which is (1 - lr * wd) * p - lr * grad: the parameter shrunk by a fixed factor, then the plain step. The weight norm (the square root of the sum of every parameter squared, Lesson 3.4.1's length of the weights taken over all of a network's parameters) is the number it moves. Why. wd * p is the gradient of wd / 2 times the sum of the squared parameters, the squared cost Lesson 3.4.1 described as half the sum of the squared weights, so SGD with weight_decay fits the loss plus that cost. Every parameter pays for its own size at every step, so it stays large only where the loss's gradient pushes back by as much as the shrink takes, and a memorizing network is one whose parameters grew to fit 200 rows' noise. When. When a network has more parameters than its rows pin down and the gap is open, with the setting tried a factor of ten apart, Lesson 4.3.4's log grid, and chosen on the validation rows.

One step of the optimizer against the same step written by hand, both ways:

import torch
from torch import optim
from train_loop import network, loss_fn
from slice_loop import X_slice, y_slice

model = network(512)
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)
before = [p.detach().clone() for p in model.parameters()]
loss = loss_fn(model(X_slice), y_slice)
loss.backward()
grads = [p.grad.clone() for p in model.parameters()]
optimizer.step()
optimizer.zero_grad()
after = list(model.parameters())
by_hand = [b - 0.1 * (g + 0.1 * b) for b, g in zip(before, grads)]
shrunk = [0.99 * b - 0.1 * g for b, g in zip(before, grads)]
# True
print(all(torch.allclose(a, h) for a, h in zip(after, by_hand)))
# True
print(all(torch.allclose(a, s) for a, s in zip(after, shrunk)))
# False
print(all(torch.equal(a, h) for a, h in zip(after, by_hand)))

The hand step and the shrink form land within allclose of the optimizer's step and not on it bit for bit, as Lesson 4.2.2's hand step and step() did, which is why the comparison is allclose. The same step, reached as a cost on the loss:

penalized = network(512)
plain = optim.SGD(penalized.parameters(), lr=0.1)
penalty = 0.1 / 2 * sum(p.pow(2).sum() for p in penalized.parameters())
print(f"{penalty.item():.2f}")  # 9.02
loss = loss_fn(penalized(X_slice), y_slice) + penalty
loss.backward()
plain.step()
plain.zero_grad()
# True
print(all(torch.allclose(q, a) for q, a in zip(penalized.parameters(), after)))

penalized starts from the same seeded parameters and takes one plain step on the loss plus 0.1 / 2 times the sum of its squared parameters, 9.02 at the seed, and it lands where weight_decay=0.1 landed. For SGD the cost on the loss and the shrink in the step are one thing, and the setting saves writing the cost. No decay and four settings a factor of ten apart, each from the seed for 3,000 epochs:

import numpy as np
from torch import optim
from train_loop import network
from slice_loop import X_slice, y_slice, fit

def norm(model):
    return sum(p.pow(2).sum() for p in model.parameters()).sqrt().item()

print(f"{norm(network(512)):.2f}")  # 13.43
for wd in (0, 0.0001, 0.001, 0.01, 0.1):
    model = network(512)
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=wd)
    train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
    lowest = int(np.argmin(val_losses))
    # decay, the three numbers, the weight norm, the output bias
    print(wd, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", lowest + 1, f"{val_losses[lowest]:.4f}", f"{norm(model):.2f}", round(model[2].bias.item(), 2))

0 0.0240 3.3360 38 0.7764 19.90 3.92
0.0001 0.0175 3.3185 38 0.7763 19.51 3.91
0.001 0.0514 3.4542 38 0.7758 16.62 3.61
0.01 0.2484 3.0090 37 0.7697 9.78 2.4
0.1 1.6134 0.8670 98 0.6933 3.71 1.08

The seeded network's norm is 13.43, the same draw on every run of torch 2.14.1 on a processor, and memorizing grew it to 19.90. What a shrink per step adds up to over the run:

for wd in (0.0001, 0.001, 0.01, 0.1):
    # the shrink per step, and what 3,000 of them leave
    print(wd, round(1 - 0.1 * wd, 5), f"{(1 - 0.1 * wd) ** 3000:.2e}")

0.0001 0.99999 9.70e-01
0.001 0.9999 7.41e-01
0.01 0.999 4.97e-02
0.1 0.99 8.05e-14

The validation readings move by less than a hundredth until 0.01. At 0.0001 and 0.001 the lowest point stays at 0.7763 and 0.7758 at epoch 38, the end stays within 0.12 of the baseline's 3.3360, and the norm ends at 19.51 and 16.62, from shrinks per step that would leave 97 and 74 percent of a parameter no gradient pushed back after the whole run. At 0.01 the norm halves, to 9.78, the training loss rises to 0.2484 and the end falls to 3.0090, while the lowest point barely moves. At 0.1 the norm ends at 3.71, below the seeded 13.43, the training loss rises to 1.6134, the end falls to 0.8670, and the lowest point moves to 0.6933 at epoch 98, the first setting that lowers the minimum by more than a hundredth and not only the end. The last column is the output layer's bias, which carries the slice's mean price of 4.009: the decay shrinks it with everything else, from 3.92 to 1.08.

DON'T:

• DON'T set weight decay by its training loss. The training loss rose at every setting from 0.001 up, to 1.6134 at 0.1, and that rise is the trade the validation loss was bought with.
• DON'T read a decay that moved no validation reading by more than a hundredth as a tool that failed. 0.0001 takes one part in 100,000 off a parameter per step, 97 percent of it left after 3,000 steps, too small for this run rather than wrong.
• DON'T assume the decay spares the biases. weight_decay shrinks every parameter handed to the optimizer, and the output bias that carries the mean price fell from 3.92 to 1.08.

Measuring One Tool Alone

What. The protocol this section reads every tool by: the baseline's three numbers, then each change's three numbers from the same seed, rows, rate, epoch count and thread count, in one table, with the end read beside its swing and the decision taken on the validation columns, never on the training one. Why. A change measured alone is credited with its own effect and nothing else. Two changes made together hand their joint effect to whichever one the report names, and a difference smaller than what the run moves between two epochs, or what a second thread count moves it, is not an effect at all. When. Every time a tool or a setting is tried, and before two tools are combined, which Lesson 4.4.2 does last, against these single measurements.

import numpy as np
import torch
from torch import optim
from train_loop import network
from slice_loop import X_slice, y_slice, fit

def measure(width, weight_decay):
    model = network(width)
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=weight_decay)
    train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
    lowest = int(np.argmin(val_losses))
    return train_losses[-1], val_losses[-2], val_losses[-1], val_losses[lowest], lowest + 1

for width, weight_decay in ((512, 0), (32, 0), (512, 0.1), (32, 0.1)):
    final_train, before_last, final_val, best, epoch = measure(width, weight_decay)
    print(width, weight_decay, f"{final_train:.4f}", f"{before_last:.4f}", f"{final_val:.4f}", f"{best:.4f}", epoch)

512 0 0.0240 3.5526 3.3360 0.7764 38
32 0 0.0803 4.5207 3.9878 0.7789 18
512 0.1 1.6134 1.3842 0.8670 0.6933 98
32 0.1 1.5511 1.3825 0.8335 0.7137 30

The fourth row is the run a teammate makes when the width and the decay change in one commit, the end falling from 3.3360 to 0.8335, and its report credits the narrower network. The rows above it say otherwise. Alone, the narrower network ended above the baseline, 3.9878, its lowest point about where the baseline's was, 0.7789, and the decay alone ended at 0.8670 and moved the lowest point to 0.6933. On top of the decay the narrower network moves the end by 0.0335, well inside the swings of both decayed runs, 1.3842 to 0.8670 and 1.3825 to 0.8335, and raises the lowest point to 0.7137: the drop is the decay's.

torch.set_num_threads(1) (the thread count set to one, left at its default in every other block of this lesson) is the one change in the next run of the baseline and of the decay:

torch.set_num_threads(1)
model = network(512)
optimizer = optim.SGD(model.parameters(), lr=0.1)
report = (38, 100, 300, 3000)
train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000, report)
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4f}", f"{val_losses[-2]:.4f}")
final_train, before_last, final_val, best, epoch = measure(512, 0.1)
print(f"{final_train:.4f}", f"{before_last:.4f}", f"{final_val:.4f}", f"{best:.4f}", epoch)

38 1.6458 0.7764
100 1.0564 0.9002
300 0.4488 1.7190
3000 0.0212 3.3174
38 0.7764 3.4977
1.6082 1.3828 0.8649 0.6933 98

On one thread the baseline prints the same numbers at epochs 38 and 100 and other digits from there, 0.4488 and 1.7190 at epoch 300 against the default's 0.4478 and 1.7195, and it ends at 0.0212 and 3.3174, with 3.4977 the epoch before. Its lowest point is the same 0.7764 at epoch 38, and the decay's is the same 0.6933 at epoch 98, while the decay's end moves from 0.8670 to 0.8649. The end of a run on this slice therefore carries a margin, its swing between the last two epochs and what a second thread count moves it, and the lowest point is the steady reading. The decay's effect stands far outside that margin, and 0.0001's move of 0.0175 at the end, with its lowest point at 0.7763, stands inside it, the nothing the fourth subtitle found.

DON'T:

• DON'T change two settings in one run and credit either. Width 32 with the decay ended at 0.8335, and the width alone ended at 3.9878, above the baseline.
• DON'T compare final digits across thread counts. One thread moved the baseline's end from 3.3360 to 3.3174 with nothing else changed.
• DON'T credit a tool with a difference smaller than the run's own swing. 0.0001 moved the end by 0.0175 where the baseline swings by about 0.2 between its last two epochs.

Lab Exercises

Lab Exercise 1: The Ladder on 100 Listings

In ladder_100.py, take the first 100 rows of prep_net's fit tensors as the rows to memorize, print their shape and their mean scaled price to three decimals, and print the validation loss of pricing every validation listing at that mean. Build the ladder on them: for widths 1, 8, 32, 128 and 512, fit network(width) for 3,000 epochs at 0.1 with the fit of slice_loop, and print the width, the parameter count, the parameters per row to one decimal, the three numbers and the end over the lowest to two decimals. Score the 512-unit network with score on its 100 rows and on the validation rows, and print its last four validation losses. Say in a comment where the gap opens on 100 rows and how this ladder differs from the Theory's on 200.

Lab Exercise 2: Weight Decay on the 128-Unit Network

In decay_128.py, check the decayed step on network(128): build optim.SGD at 0.1 with a decay of 0.01, take one step on the slice, and print whether every parameter is within allclose of (1 - lr * wd) * p - lr * grad computed from copies taken before the step. Print the seeded network's weight norm, then for decays of 0, 0.001, 0.01, 0.1 and 1.0 fit network(128) on the slice for 3,000 epochs at 0.1 and print the decay, the three numbers, the weight norm and the last two validation losses. Say in a comment which decay you would carry forward for this width and on which column, and how it compares with the Theory's 512-unit network at the same decay.

import numpy as np
import torch
from torch import optim
from prep_net import fit_rows, val_rows, X_fit, y_fit, X_val, y_val
from train_loop import network, loss_fn, score
from slice_loop import fit

X_100, y_100 = X_fit[:100], y_fit[:100]
print(X_100.shape, round(y_100.mean().item(), 3))
constant = torch.full_like(y_val, y_100.mean().item())
print(f"{loss_fn(constant, y_val).item():.4f}")
for width in (1, 8, 32, 128, 512):
    model = network(width)
    count = sum(p.numel() for p in model.parameters())
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    train_losses, val_losses = fit(model, optimizer, X_100, y_100, 3000)
    lowest = int(np.argmin(val_losses))
    print(width, count, round(count / 100, 1), f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", lowest + 1, f"{val_losses[lowest]:.4f}", round(val_losses[-1] / val_losses[lowest], 2))
mae, r2 = score(model, X_100, fit_rows.iloc[:100])
print(f"{mae:,.0f}", round(r2, 3))
mae, r2 = score(model, X_val, val_rows)
print(f"{mae:,.0f}", round(r2, 3))
print([round(v, 4) for v in val_losses[-4:]])
# The gap opens at eight units, 3.48 times the lowest, as on 200 rows.
# From 32 units up the end sits nearer the lowest than on 200 rows,
# 1.72 against 4.3 at 512 units.

torch.Size([100, 18]) 3.938
3.2156
1 21 0.2 1.1743 0.8862 16 0.7009 1.26
8 161 1.6 0.2482 2.6409 16 0.7583 3.48
32 641 6.4 0.0250 1.7802 145 0.8291 2.15
128 2561 25.6 0.0143 1.4188 119 0.7832 1.81
512 10241 102.4 0.0037 1.3625 35 0.7939 1.72
4,066 0.999
82,916 0.566
[1.3806, 1.3626, 1.3811, 1.3625]

import numpy as np
import torch
from torch import optim
from train_loop import network, loss_fn
from slice_loop import X_slice, y_slice, fit

def norm(model):
    return sum(p.pow(2).sum() for p in model.parameters()).sqrt().item()

model = network(128)
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.01)
before = [p.detach().clone() for p in model.parameters()]
loss = loss_fn(model(X_slice), y_slice)
loss.backward()
grads = [p.grad.clone() for p in model.parameters()]
optimizer.step()
optimizer.zero_grad()
by_hand = [(1 - 0.1 * 0.01) * b - 0.1 * g for b, g in zip(before, grads)]
print(all(torch.allclose(p, h) for p, h in zip(model.parameters(), by_hand)))

print(f"{norm(network(128)):.2f}")
for wd in (0, 0.001, 0.01, 0.1, 1.0):
    model = network(128)
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=wd)
    train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
    lowest = int(np.argmin(val_losses))
    print(wd, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", lowest + 1, f"{val_losses[lowest]:.4f}", f"{norm(model):.2f}", [round(v, 4) for v in val_losses[-2:]])
# 0.1 for 128 units, on the lowest validation loss, 0.7109, the
# lowest of the five, with the end down from 3.5977 to 0.8329, far
# past the swing. Beside the Theory's 512 units at 0.1, the ends differ
# by less than either swing and 512 units sit lower at the lowest
# point, 0.6933.

True
6.67
0 0.0370 3.5977 20 0.7514 13.67 [3.9303, 3.5977]
0.001 0.0697 3.2736 20 0.7509 12.48 [3.8335, 3.2736]
0.01 0.2086 2.5680 20 0.7468 9.42 [3.1942, 2.568]
0.1 1.5441 0.8329 22 0.7109 3.80 [1.3833, 0.8329]
1.0 2.3650 1.3330 29 0.7418 2.31 [1.1108, 1.333]

Theory Assessment

Q1: The Run That Prices New Listings Worse Than One Number

At epoch 3,000 the memorizing run printed a training loss of 0.0240 and a validation loss of 3.3360, its lowest validation loss was 0.7764 at epoch 38, and its validation R² is -0.063 while one constant price scores 3.2604 on the same rows. Say what the four numbers say about the network, why this run is the loop working and not a defect in it, and why the section makes this gap before it tries a single tool.

Hint: Ask what the loop drives down, on which rows, and what Lesson 4.2.3's tiny batch would make of it. Then ask what a tool that gives up training loss for validation loss could give up on a model whose two losses already agree.

Q2: Fifty-One Parameters per Row, and Six

One hidden unit on the slice ends at a training loss of 1.6054 against least squares' 1.6353, eight units end with the validation loss 4.63 times its lowest at a training loss of 0.5990, and 512 units on all 1,725 fit rows end at 0.5089 and 1.33 times their lowest after the same 3,000 epochs. Say why one unit cannot memorize 200 listings, what changes at eight, why the network that memorized 200 rows did not memorize 1,725, and what the two runs together say about where a tool is measured.

Hint: Write the one-unit network's output on a row where its unit is positive and on one where it is zero. Then count the parameters each listing has to itself on the slice and on the fit rows.

Q3: Shrink, Then Step

A teammate writes the decayed step as p = 0.99 * p - 0.1 * grad and asks why anyone calls it a penalty. Say which lr and wd make the factor 0.99, which cost added to the loss has wd * p as its gradient and where Module 3 charged it, why 0.0001 changed almost nothing over 3,000 epochs while 0.1 took the norm below its seeded value, and what the decay did to the output layer's bias.

Hint: Factor p out of p - lr * (grad + wd * p), and differentiate wd / 2 * p ** 2. Then raise each setting's factor per step to the 3,000th power, and remember what the output bias carries.

Q4: The Report That Credits the Wrong Change

A teammate's report on the slice reads: "Narrowing the network to 32 units and adding a decay of 0.1 cut the validation loss from 3.3360 to 0.8335, so the narrow network generalizes better. A decay of 0.0001 alone helps too, 3.3185 against 3.3360. Run on one thread the baseline ends at 3.3174, so one thread trains a better model. And the narrow decayed network has the lowest training loss of the decayed runs, 1.5511 against 1.6134." Every number is from a run of this lesson. Say what each claim gets right and what it gets wrong, and what the report should have said.

Hint: Find the rows of the fifth subtitle's table that isolate each change. For every difference the report reads, set it beside the swing between that run's last two epochs and beside what a second thread count moves the same run, and ask which column a decision is read from.

Assignment

Assignment: Decay the Weights, Not the Biases

The decay of 0.1 shrank every parameter of the 512-unit network, the output layer's bias too, from 3.92 to 1.08, although that one number carries the slice's mean price of 4.009. A teammate proposes decaying the two weight matrices and leaving the two bias vectors alone, measured against the decay on everything and against no decay, each alone, with the weight norm followed through the run rather than read once at its end. The element this lesson did not use: an optimizer can be built from a list of dictionaries instead of model.parameters(), one dictionary per parameter group, Lesson 4.2.2's set of parameters that share one set of settings, each with a "params" key holding its tensors and, beside it, any setting the group overrides, such as "weight_decay", while a setting passed as a keyword argument, lr=0.1 here, applies to every group that does not override it.

Write bias_decay.py. Give it the Theory's norm and a make_optimizer(model, mode) that returns plain SGD at 0.1 for "none", SGD at 0.1 with a decay of 0.1 for "all", and for "weights" SGD at 0.1 built from two groups, the parameters whose name ends in weight at a decay of 0.1 and those whose name ends in bias at 0, picked from named_parameters(). For each mode build network(512) and fit the slice in six calls of 500 epochs, appending each call's two lists to one list per run and the weight norm, to two decimals, to a third list after each call. Then print the mode, the final training loss, the validation loss at epoch 2,999 and at 3,000, and the lowest validation loss with its epoch, and on a second line the six norms, the output bias to two decimals and the validation MAE and R² from score. After the three runs print each group's tensor count and decay, then the grouped optimizer itself. Say in a comment which mode you would carry forward, and on which reading.

Hints:

• fit keeps nothing between calls, the model and the optimizer do, so six calls of 500 epochs continue one run of 3,000, and the lowest epoch is read from the joined list.
• name.endswith("weight") picks 0.weight and 2.weight out of named_parameters(), and print(optimizer) lists one block of settings per group, the decay the only line that differs.

import numpy as np
from torch import optim
from prep_net import val_rows, X_val
from train_loop import network, score
from slice_loop import X_slice, y_slice, fit

def norm(model):
    return sum(p.pow(2).sum() for p in model.parameters()).sqrt().item()

def make_optimizer(model, mode):
    if mode == "none":
        return optim.SGD(model.parameters(), lr=0.1)
    if mode == "all":
        return optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)
    weights = [p for name, p in model.named_parameters() if name.endswith("weight")]
    biases = [p for name, p in model.named_parameters() if name.endswith("bias")]
    groups = [{"params": weights, "weight_decay": 0.1}, {"params": biases, "weight_decay": 0.0}]
    return optim.SGD(groups, lr=0.1)

for mode in ("none", "all", "weights"):
    model = network(512)
    optimizer = make_optimizer(model, mode)
    train_losses, val_losses, norms = [], [], []
    for call in range(6):
        t, v = fit(model, optimizer, X_slice, y_slice, 500)
        train_losses += t
        val_losses += v
        norms.append(round(norm(model), 2))
    lowest = int(np.argmin(val_losses))
    mae, r2 = score(model, X_val, val_rows)
    print(mode, f"{train_losses[-1]:.4f}", f"{val_losses[-2]:.4f}", f"{val_losses[-1]:.4f}", lowest + 1, f"{val_losses[lowest]:.4f}")
    print(norms, round(model[2].bias.item(), 2), f"{mae:,.0f}", round(r2, 3))
print([len(g["params"]) for g in optimizer.param_groups], [g["weight_decay"] for g in optimizer.param_groups])
print(optimizer)
# "weights", on the lowest validation loss, 0.6827 against 0.6933 for
# "all", with the output bias kept at 3.03 where "all" pulled it to 1.08.

none 0.0240 3.5526 3.3360 38 0.7764
[18.28, 19.16, 19.47, 19.66, 19.79, 19.9] 3.92 96,693 -0.063
all 1.6134 1.3842 0.8670 98 0.6933
[3.6, 3.69, 3.7, 3.71, 3.71, 3.71] 1.08 62,956 0.724
weights 1.3233 1.0598 0.7170 124 0.6827
[7.51, 7.55, 7.6, 7.64, 7.67, 7.7] 3.03 56,706 0.772
[2, 2] [0.1, 0.0]
SGD (
Parameter Group 0
    dampening: 0
    differentiable: False
    foreach: None
    fused: None
    lr: 0.1
    maximize: False
    momentum: 0
    nesterov: False
    weight_decay: 0.1

Parameter Group 1
    dampening: 0
    differentiable: False
    foreach: None
    fused: None
    lr: 0.1
    maximize: False
    momentum: 0
    nesterov: False
    weight_decay: 0.0
)

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want weight decay in an Adam-style optimizer, or the decay set differently for different parameters.

• Official docs - PyTorch 2.14, torch.optim.AdamW. The class whose page says its weight decay "does not accumulate in the momentum nor variance": its algorithm box shrinks every parameter by the learning rate times the decay before the step built from the two running averages, and its weight_decay defaults to 0.01.
• Official docs - PyTorch 2.14, torch.optim, "Per-parameter options". An optimizer built from a list of dictionaries, each one parameter group with its own settings, shown first with a learning rate per part of a model and then with the biases picked out of named_parameters() by name and given a weight_decay of 0 while every other parameter keeps the decay passed for all.

Summary

The lesson broke a model on purpose and called the break a success. A network that prices 200 listings to within 11,749 and misses new ones by 96,693 is the loop doing what it was built to do on a model with room to do it, and only a model in that state can show what a regularization tool is worth. Width made the room: one unit drew a bent line through the slice, eight let the validation loss run away, and from 32 the training loss was all but gone.

The full rows put the slice in its place. The same network on 1,725 listings had six parameters per row instead of fifty-one, did not memorize in 3,000 epochs, and opened its gap slowly, ending a third above its lowest where the slice ended at 4.3 times it. The slice is a magnifying glass: a tool is measured there and confirmed on the full rows before anyone quotes it.

Weight decay turned out to be Module 3's squared cost in a new place, charged through the step as a shrink, and the validation loss did not move until the shrink per step reached a thousandth. The measuring mattered as much as the tool. The end of a memorizing run swings from one epoch to the next and moves with the thread count while its minimum holds still, and a change measured beside another is credited with both. Lesson 4.4.2 brings two more tools to the same table.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

X_slice, y_slice = X_fit[:200], y_fit[:200]  # memorized on purpose
model = network(512)                         # 10,241 parameters
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)
train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
lowest = int(np.argmin(val_losses))          # the steady reading
end = (val_losses[-2], val_losses[-1])       # the end, beside its swing
# the weight norm, the number the decay moves
norm = sum(p.pow(2).sum() for p in model.parameters()).sqrt()