Dropout and Early Stopping

Context

Dropout is a layer that zeros a random share of the hidden units' outputs at every training step and passes all of them once the model is switched to evaluation. Early stopping watches the validation loss every epoch, ends the loop once it has stopped improving, and hands back the parameters of its lowest epoch.

Early stopping ends a run once its validation curve has turned, on any network trained epoch by epoch, and the copy of the best parameters it keeps is what Lesson 4.4.3 saves to a file and loads in a second process. Dropout is one line in a network's definition and one switch in every pass that scores it, the reason the module's validation passes have called model.eval() since Lesson 4.2.3.

Without them the slice's 512-unit network, the baseline Lesson 4.4.1 memorized on purpose, runs all 3,000 epochs, ending that lesson's run at a validation loss of 3.3360 after passing through 0.7764 at epoch 38, and nothing in the loop says the better model existed. The failures in this lesson raise nothing: a validation pass left in training mode prices the rows through a random half of the network, which put one dropout model's loss a quarter too high on average, and a best model kept without a copy turns out, at the restore, to be the last one.

Goal

• Place nn.Dropout(p) after the ReLU of the 512-unit network, trace one listing's hidden units through it in training and evaluation mode, and explain the 1 / (1 - p) scale from a run.
• Measure dropout at p 0.2 and 0.5 against the baseline and weight decay, read each end inside its band, and read a dropout network's fit from its evaluation-mode loss, not the loss fit records.
• Distinguish a validation loss taken in training mode from the model's own, and show from runs that it moves from pass to pass and climbs with p.
• Write early_stop in stopping.py, stop three networks with it, and set a patience that survives a curve that wobbles.
• Keep the best parameters with copy.deepcopy, restore them with load_state_dict, and prove from a run that a state dict kept without the copy restores nothing.
• Combine weight decay and dropout under early stopping, and credit the result against each tool alone.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want the two tools described once before you measure them.

• YouTube - "MIT Introduction to Deep Learning (2024) | 6.S191", the chapter "Regularization: dropout and early stopping" by Alexander Amini. Lecture 1, Foundations of Deep Learning, whose outline puts this chapter at 1:02:28, the last one before the summary at 1:08:47, after its chapters on loss functions, training and gradient descent, backpropagation, setting the learning rate and batched gradient descent.
• Official docs - PyTorch 2.14, torch.nn.Dropout. The layer's whole contract: during training each element is zeroed with probability p, drawn independently on every forward call, the outputs are scaled by 1 / (1 - p), and during evaluation the module computes the identity, with p defaulting to 0.5 and an inplace flag this lesson leaves off.

Table of Contents

• Dropout, Two Modes of One Layer
• Dropout on the Slice, Measured
• Validation in Training Mode Looks Like Overfitting
• Early Stopping, the Epoch of the Minimum
• Keeping the Best Parameters
• The Tools Together

Theory

Every block in this lesson runs in ~/dev/bootcamp, and every script after the first runs on Lesson 4.3.1's table and imports what earlier lessons saved: prep_net for the six tensors, train_loop for loss_fn, validate and network(width), which seeds 42 inside, and slice_loop, Lesson 4.4.1's file, for the slice, X_slice and y_slice, and its loop, fit. Every fit in the Theory is optim.SGD on the full batch at 0.1, and every run is read as Lesson 4.4.1 read its own: the lowest validation loss is the steady reading, and the end of a run on the slice swings, so it counts only against the margin it swings across. The fourth subtitle saves stopping.py, which later scripts and Lesson 4.4.3 import. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Dropout, Two Modes of One Layer

What. Dropout (a layer that, in training mode, zeros each of its inputs with probability p and multiplies the inputs it keeps by 1 / (1 - p), and in evaluation mode passes every input through unchanged) is nn.Dropout(p). In this lesson's network it sits after the ReLU, so its inputs are the 512 hidden units' outputs, and model.train() and model.eval(), the switch Lesson 4.2.3 wrote into every validation pass, choose its mode. Why. In training mode every step is taken through a different random selection of the hidden units, about half of them at p 0.5, so no step can count on any one unit being there, and the second subtitle measures what that does to the gap. The scale keeps each output's expected value equal to its input, so evaluation mode, which keeps every unit, needs no correction. When. In a network whose validation loss turns while its training loss keeps falling, after a hidden layer's activation and before the next layer, with p chosen on the validation rows.

The layer on its own, on eight inputs of one:

import torch
from torch import nn

torch.manual_seed(42)
drop = nn.Dropout(0.5)
ones = torch.ones(8)
print(drop, drop.training)  # Dropout(p=0.5, inplace=False) True
print(drop(ones))           # tensor([2., 2., 2., 2., 0., 2., 0., 0.])
print(drop(ones))           # tensor([2., 2., 2., 2., 0., 0., 2., 0.])
drop.eval()
print(drop(ones))           # tensor([1., 1., 1., 1., 1., 1., 1., 1.])

The module holds no parameter and starts in training mode. Each call draws a fresh mask from the seeded stream, so the first call zeroed three of the eight inputs and the second a different three, p being a probability for each input and not a count. Every input a mask kept came out as 2, 1 / (1 - 0.5). What the 2 is for, over many inputs, and what p 0.2 does:

drop.train()
many = drop(torch.ones(100_000))
zeroed = (many == 0).float().mean().item()
# 0.4998 1.0004
print(round(zeroed, 4), round(many.mean().item(), 4))
torch.manual_seed(42)
# tensor([1.2500, 1.2500, 1.2500, 1.2500, 1.2500, 1.2500, 0.0000, 0.0000])
print(nn.Dropout(0.2)(ones))

Over 100,000 inputs the layer zeroed 0.4998 of them and the mean output is 1.0004: an input survives half the time and comes out doubled when it does, so its expected value is 0.5 * 0 + 0.5 * 2, the input itself. At p 0.2 the draw zeroed two of eight and multiplied the rest by 1.25, 1 / (1 - 0.2). The layer in the slice's network, built by hand:

import torch
from torch import nn
from prep_net import X_val, y_val
from train_loop import loss_fn, network, validate

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(18, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, 1))
print(model)

Sequential(
  (0): Linear(in_features=18, out_features=512, bias=True)
  (1): ReLU()
  (2): Dropout(p=0.5, inplace=False)
  (3): Linear(in_features=512, out_features=1, bias=True)
)

plain = network(512)
pairs = zip(model.parameters(), plain.parameters())
# True
print(all(torch.equal(a, b) for a, b in pairs))
# ['0.weight', '0.bias', '3.weight', '3.bias']
print(list(model.state_dict().keys()))

nn.Dropout(0.5) takes position 2, so the output layer's parameters are named 3.weight and 3.bias where network(512)'s are 2.weight and 2.bias, and the layer adds no key of its own. It draws nothing at construction, so the seed gives this network network(512)'s 10,241 parameters to the bit, and every difference this lesson measures between the two networks is the layer at work, never a different start. One validation listing through the hidden layer, then through the dropout layer:

with torch.no_grad():
    hidden = model[1](model[0](X_val[:1]))
    kept = model[2](hidden)
positive = hidden[0] > 0
dropped = (kept[0] == 0) & positive
# 273 134
print(positive.sum().item(), dropped.sum().item())
# tensor([1.0832, 0.1962, 0.6542, 0.1414, 0.6843, 0.0293, 0.4172, 0.2346])
print(hidden[0][positive][:8])
# tensor([0.0000, 0.3925, 1.3084, 0.0000, 1.3685, 0.0586, 0.0000, 0.0000])
print(kept[0][positive][:8])
# True
print(torch.equal(kept[kept != 0], 2 * hidden[kept != 0]))

The ReLU left 273 of the listing's 512 hidden units positive, and the mask zeroed 134 of those, close to the half p asks for. torch.equal confirms that every kept value is exactly doubled, so the printed 0.3925 reads one digit off twice 0.1962 only because both are rounded to four places. Evaluation mode, and the loss each mode gives the 575 validation rows:

model.eval()
with torch.no_grad():
    # True
    print(torch.equal(model[2](hidden), hidden))
model.train()
with torch.no_grad():
    in_training = loss_fn(model(X_val), y_val).item()
# 17.3740 17.4358
print(f"{in_training:.4f}", f"{validate(model, X_val, y_val):.4f}")

The same untrained parameters score the validation rows at 17.3740 in training mode and 17.4358 in evaluation mode, two numbers from one model, and the third subtitle measures the pair on fitted networks.

DON'T:

• DON'T read p as the share of units kept. nn.Dropout(0.2) zeroed two of eight ones in its draw and kept the rest at 1.25, and nn.Dropout(0.5) zeroed 134 of one listing's 273 positive units.
• DON'T multiply by 1 - p when the model is evaluated. The layer already scaled the kept values by 2 in training mode, and evaluation mode returned the hidden tensor unchanged.
• DON'T expect a seeded layer to repeat its mask. Two calls after one torch.manual_seed(42) printed two different masks.

Dropout on the Slice, Measured

What. The dropout network at p 0.2 and 0.5, fitted on the slice exactly as the baseline, network(512), for 3,000 epochs, beside weight decay at 0.1, Lesson 4.4.1's tool. Each run is read by the section's three numbers, the final training loss fit records, the final validation loss and the lowest validation loss with its epoch, beside the band the end wanders across, the lowest and highest validation loss of its last 100 epochs. Why. Every row differs from the baseline by one change from the same seeded start at the same rate for the same epochs, so a difference between two rows is the tool's once it stands outside the margin. Lesson 4.4.1 read the end beside the epoch before it, and a dropout network's end wanders further than one epoch shows, so the band takes the last 100. When. Every time a tool is tried, and before it is combined with another.

Each line is the setting, the final training and validation losses, the band, and the lowest validation loss with its epoch:

import numpy as np
import torch
from torch import nn, optim
from prep_net import X_fit, y_fit, X_val, y_val
from train_loop import loss_fn, network, validate
from slice_loop import X_slice, y_slice, fit

def measure(name, model, decay=0.0):
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=decay)
    train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
    end, lowest = val_losses[-100:], int(np.argmin(val_losses))
    print(name, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{min(end):.4f}", f"{max(end):.4f}", f"{val_losses[lowest]:.4f}", lowest + 1)
    return model

models = {"baseline": measure("baseline", network(512))}
for p in (0.2, 0.5):
    torch.manual_seed(42)
    dropped = nn.Sequential(nn.Linear(18, 512), nn.ReLU(), nn.Dropout(p), nn.Linear(512, 1))
    models[f"p {p}"] = measure(f"p {p}", dropped)
models["decay 0.1"] = measure("weight decay 0.1", network(512), decay=0.1)

baseline 0.0240 3.3360 3.3323 3.5527 0.7764 38
p 0.2 0.1520 3.5951 3.0896 3.9996 0.7113 36
p 0.5 0.2917 2.9550 2.7659 4.2084 0.7583 49
weight decay 0.1 1.6134 0.8670 0.8559 1.3931 0.6933 98

The baseline's end moves between 3.3323 and 3.5527, Lesson 4.4.1's alternation. The dropout networks' ends wander much further, from 3.0896 to 3.9996 at p 0.2 and from 2.7659 to 4.2084 at 0.5, and both bands cover the baseline's, so nothing at the end tells either rate from the baseline. The steady readings do: dropout lowered the lowest validation loss to 0.7113 at epoch 36 and to 0.7583 at epoch 49, against 0.7764 at 38. Weight decay at 0.1 is the one row whose whole band, 0.8559 to 1.3931, lies below the other rows', with its lowest at 0.6933: on these 200 rows, from this seed, it closed more of the gap than either dropout rate, a measurement of one slice and not a rule about the two tools. The fit, read in evaluation mode:

for name in ("baseline", "p 0.2", "p 0.5"):
    print(name, f"{validate(models[name], X_slice, y_slice):.4f}")

baseline 0.0241
p 0.2 0.0316
p 0.5 0.0899

fit recorded 0.1520 and 0.2917 for the two dropout networks against the baseline's 0.0240, the losses of thinned networks, each on the one mask its last step drew. In evaluation mode the same networks score the slice at 0.0316 and 0.0899 against 0.0241: dropout held the fit back, by far less than the recorded losses say, and more of it held the fit back further.

DON'T:

• DON'T read a dropout network's fit from the loss fit records. That pass runs in training mode on a fresh mask, 0.2917 for p 0.5, where the same network scores the slice at 0.0899 in evaluation mode.
• DON'T credit dropout with its last epoch. p 0.5 ended at 2.9550 inside a band from 2.7659 to 4.2084 over its last 100 epochs, a band that covers the baseline's whole end.
• DON'T change the rate or the epochs along with the tool. Every row here is SGD at 0.1 for 3,000 epochs from one seeded start.

Validation in Training Mode Looks Like Overfitting

What. A validation pass that skips model.eval() runs the dropout layer in training mode, so it prices every validation row through a thinned network of its own, a fresh mask per row, and prints a different number on every pass. The model's validation loss is the evaluation-mode one, the number validate returns, which is why Lesson 4.2.3 put model.eval() inside it. Why. On the full rows the training-mode number sits above the model's, further above at a larger p, which is what a model that fits its rows and fails on new ones looks like from the validation column, and the remedy that reading suggests, more dropout, raises it again. Nothing raises an error. When. Every pass that scores a model with dropout in it, the validation rows, the test rows and any listing priced for someone, runs in evaluation mode.

The slice's p 0.5 network, continuing the script, scored once by validate and three times with the layer left in training mode:

heavy = models["p 0.5"]
print(f"{validate(heavy, X_val, y_val):.4f}")
with torch.no_grad():
    for _ in range(3):
        print(f"{loss_fn(heavy(X_val), y_val).item():.4f}")

2.9550
3.2493
3.2345
3.6334

Three passes, three masks, three numbers, none of them 2.9550. A hundred more, as the lowest, the mean, the highest, and how many land above the model's number:

with torch.no_grad():
    passes = torch.tensor([loss_fn(heavy(X_val), y_val).item() for _ in range(100)])
evaluated = validate(heavy, X_val, y_val)
print(f"{passes.min().item():.4f}", f"{passes.mean().item():.4f}", f"{passes.max().item():.4f}")
print((passes > evaluated).sum().item())

2.2558 3.3225 4.2629
83

On the slice a training-mode pass usually lands above the model's number and not always: 83 of 100 did, and the lowest sat at 2.2558, below 2.9550. On all 1,725 fit rows the picture is cleaner. Four networks fitted for 300 epochs, p from 0 to 0.8, each scored once by validate and 20 times in training mode, printed as p, the model's validation loss, the lowest and the mean training-mode loss, and the mean's ratio to the model's:

for p in (0.0, 0.2, 0.5, 0.8):
    torch.manual_seed(42)
    model = nn.Sequential(nn.Linear(18, 512), nn.ReLU(), nn.Dropout(p), nn.Linear(512, 1))
    fit(model, optim.SGD(model.parameters(), lr=0.1), X_fit, y_fit, 300)
    with torch.no_grad():
        passes = torch.tensor([loss_fn(model(X_val), y_val).item() for _ in range(20)])
    evaluated = validate(model, X_val, y_val)
    print(p, f"{evaluated:.4f}", f"{passes.min().item():.4f}", f"{passes.mean().item():.4f}", round(passes.mean().item() / evaluated, 2))

0.0 0.4541 0.4541 0.4541 1.0
0.2 0.4579 0.4832 0.5177 1.13
0.5 0.4609 0.5247 0.5797 1.26
0.8 0.4780 0.6608 0.7613 1.59

p 0 is the plain network, whose two modes agree because a layer that zeros nothing has nothing to switch. With dropout in it, even the lowest of the 20 training-mode passes lands above the model's number, and the mean climbs with p, 13, 26 and 59 percent above it. Through validate the three dropout networks score within 0.024 of the plain network's 0.4541, and the mean of their passes in training mode climbs to 0.7613: a loop that forgot eval() reports that dropout raised the validation loss and that more of it raised the loss further, while the models themselves barely moved.

DON'T:

• DON'T score a dropout network without model.eval(). A pass in training mode put the full rows' p 0.5 network at 0.5797 on average, where the model's validation loss is 0.4609.
• DON'T forget model.train() after a validation pass written by hand. A loop that called eval() before each validation and never train() after it fitted the p 0.5 network without dropout from its second epoch on, its training loss ending at 0.0264 beside the baseline's 0.0240 rather than the 0.2917 of the network trained with its masks, and nothing raised.
• DON'T answer a training-mode gap with a larger p. At 0.8 the training-mode mean rose to 0.7613 while the model's own number moved to 0.4780.

Early Stopping, the Epoch of the Minimum

What. Early stopping (ending the loop once the validation loss has gone a set number of epochs without a new low, and keeping the parameters of that low) is the tool Lesson 4.2.3 named at the turn of its noisy model's curve, and patience (the number of epochs the loop waits after its lowest validation loss for a lower one before it stops) is its one setting. Why. Every run of the second subtitle reached its lowest validation loss before epoch 100 and spent the rest of its 3,000 epochs away from it. A loop that stops at the minimum hands back the minimum's model and costs patience epochs past it instead of thousands. When. Any loop whose validation curve turns, with the patience set by how much the curve wobbles.

Save it as stopping.py:

import copy
from prep_net import X_val, y_val
from train_loop import loss_fn, validate

def early_stop(model, optimizer, X, y, patience, max_epochs):
    best_loss, best_epoch, best_state = float("inf"), 0, None
    for epoch in range(1, max_epochs + 1):
        loss = loss_fn(model(X), y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        val_loss = validate(model, X_val, y_val)
        if val_loss < best_loss:
            best_loss, best_epoch = val_loss, epoch
            best_state = copy.deepcopy(model.state_dict())
        elif epoch - best_epoch >= patience:
            break
    model.load_state_dict(best_state)
    return epoch, best_epoch, best_loss

early_stop is fit's loop with three things kept beside it, the lowest validation loss so far, its epoch and a copy of the parameters at that epoch, the loop Lesson 4.2.3's Assignment wrote once by hand for its noise columns. The elif breaks out once patience epochs have passed since the best one without a lower loss, so a stopped loop always ran exactly patience epochs past the epoch it keeps, and max_epochs ends a curve that never stops improving. After the loop, load_state_dict(best_state) puts the kept parameters back into the model, the next subtitle's subject. Like fit, it takes the rows it fits and validates on prep_net's. Three networks, each printed as the stopping epoch, the best epoch and the best loss:

import copy
import numpy as np
import torch
from torch import nn, optim
from prep_net import X_fit, y_fit, X_val, y_val
from train_loop import network, validate
from slice_loop import X_slice, y_slice, fit
from stopping import early_stop

def dropout_net(p):
    torch.manual_seed(42)
    return nn.Sequential(nn.Linear(18, 512), nn.ReLU(), nn.Dropout(p), nn.Linear(512, 1))

def stop(model, X, y, patience, decay=0.0):
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=decay)
    stopped, best_epoch, best_loss = early_stop(model, optimizer, X, y, patience, 3000)
    print(stopped, best_epoch, f"{best_loss:.4f}")
    return model

baseline = stop(network(512), X_slice, y_slice, 50)
stop(dropout_net(0.5), X_slice, y_slice, 50)
stop(network(128), X_fit, y_fit, 100)

88 38 0.7764
99 49 0.7583
348 248 0.4412

The baseline stopped at epoch 88 of its 3,000 and kept epoch 38 and 0.7764, the lowest point of its full run, and the dropout network kept its full run's lowest point too, epoch 49 and 0.7583, because the stopped loop takes the same steps as the full one until it stops. On all 1,725 fit rows the width-128 network stopped at 348 with a patience of 100 and kept epoch 248 and 0.4412, the lowest point Lesson 4.3.1's width table printed for it. What the patience does on the baseline's curve and on the dropout network's:

for patience in (1, 10):
    stop(network(512), X_slice, y_slice, patience)
    stop(dropout_net(0.5), X_slice, y_slice, patience)

37 36 0.7766
25 24 1.0398
48 38 0.7764
59 49 0.7583

A patience of 1 stops at the first epoch without a new low. On the baseline's curve that is epoch 37, a small rise one epoch before its true lowest at 38, so it keeps epoch 36 at 0.7766 against 0.7764. The dropout network's curve wobbles far more, every step taken through a fresh mask, and a patience of 1 stopped it at epoch 25 with 1.0398, while the curve still had 24 epochs of falling to do. A patience of 10 waited through the wobbles to the same epochs 38 and 49 that 50 found.

DON'T:

• DON'T set a patience of 1 on a curve that wobbles. The dropout network stopped at epoch 25 with 1.0398, where 10 epochs of patience reached 0.7583 at epoch 49.
• DON'T report the stopping epoch as the model's. The baseline stopped at 88 and handed back epoch 38's parameters, and the 50 epochs between are what it cost to know that 38 was the lowest.
• DON'T read a stop as the lowest point the curve will ever reach. The loop sees only the epochs before it stops, so a curve that falls again after a longer wait than the patience is stopped on its first low.

Keeping the Best Parameters

What. The copy early_stop keeps, copy.deepcopy(model.state_dict()) at every new low, is a dictionary of new tensors holding that epoch's numbers, and model.load_state_dict(best_state) after the loop copies them back into the model's own parameters. Why. When the loop breaks, the model holds the stop epoch's parameters, patience epochs past the best. And model.state_dict() without the copy is not a copy at all, as Lesson 4.2.3's Assignment found: its tensors are the parameters' own memory, so a dictionary kept at the best epoch holds the stop epoch's numbers by the time it is restored. When. At every new low of every early-stopped run, and whenever parameters are kept for later.

The baseline early_stop handed back, continuing the script:

# 0.7764
print(f"{validate(baseline, X_val, y_val):.4f}")

The restored model scores exactly the best loss the loop recorded, the epoch-38 model, 50 epochs after the loop left it behind. The two ways of keeping it, on the baseline's own timeline, 38 epochs, a keep, then 50 more:

model = network(512)
optimizer = optim.SGD(model.parameters(), lr=0.1)
fit(model, optimizer, X_slice, y_slice, 38)
weight = model[0].weight
by_reference = model.state_dict()
by_copy = copy.deepcopy(model.state_dict())
# True
print(by_reference["0.weight"].data_ptr() == weight.data_ptr())
# False
print(by_copy["0.weight"].data_ptr() == weight.data_ptr())

data_ptr() (the memory address of a tensor's first number) settles what two equal-looking tensors cannot: the reference's weight starts at the parameter's own address, and the deep copy's starts somewhere else. Fifty more epochs, then each kept dictionary loaded and scored:

fit(model, optimizer, X_slice, y_slice, 50)
print(f"{validate(model, X_val, y_val):.4f}")
print(model.load_state_dict(by_reference))
print(f"{validate(model, X_val, y_val):.4f}")
print(model.load_state_dict(by_copy))
print(f"{validate(model, X_val, y_val):.4f}")
print(model[0].weight is weight)

0.9019
<All keys matched successfully>
0.9019
<All keys matched successfully>
0.7764
True

After 50 more epochs the model validates at 0.9019, the epoch-88 model the loop stopped with. Loading the reference hands the model its own current numbers and changes nothing, and load_state_dict reports <All keys matched successfully> all the same, because it checks names and shapes and never which epoch the numbers come from. Loading the copy brings back 0.7764. model[0].weight is still the object it was before both loads: load_state_dict writes the values into the tensors the model already holds, so the optimizer built on model.parameters() goes on stepping the same tensors.

DON'T:

• DON'T keep model.state_dict() without copy.deepcopy. Its weight starts at the parameter's own address, and the dictionary kept at epoch 38 restored the epoch-88 model's 0.9019.
• DON'T read <All keys matched successfully> as the right model restored. load_state_dict printed it for the reference too.

The Tools Together

What. Weight decay at 0.1 and dropout at 0.5 in one network, fitted for 3,000 epochs and early-stopped at patience 50, read against each tool alone by the same readings, the end's band, the lowest point and the early stop. Why. A combination measured only against the baseline is credited with all of its gain. The singles say how much of that gain each tool carries, and whether the two together beat the better one alone at all. When. Only after each tool has been measured alone, the order this section has followed, never as the first run.

The combination's full run, printed as the second subtitle printed its rows, then weight decay alone and the combination, each early-stopped:

model = dropout_net(0.5)
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)
train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
end, lowest = val_losses[-100:], int(np.argmin(val_losses))
print(f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{min(end):.4f}", f"{max(end):.4f}", f"{val_losses[lowest]:.4f}", lowest + 1)
stop(network(512), X_slice, y_slice, 50, decay=0.1)
stop(dropout_net(0.5), X_slice, y_slice, 50, decay=0.1)

1.5734 1.2601 0.6904 1.3898 0.6568 109
148 98 0.6933
159 109 0.6568

The four settings side by side, every number printed by this lesson's runs:

At the end the combination and weight decay alone share most of a band, so the last epoch cannot rank them, and epoch 3,000 happens to put the combination above. The steady readings rank them, 0.6568 against 0.6933, the lowest validation loss of any setting this lesson ran on the slice, and early stopping keeps exactly those. The singles divide the combination's fall from the baseline's 0.7764: weight decay alone reaches 0.6933, and dropout on top of it takes the rest, a larger fall than the 0.7764 to 0.7583 it bought alone, so the two tools do not add and a combination is a measurement of its own.

DON'T:

• DON'T credit a combination against the baseline alone. 0.7764 to 0.6568 reads as the work of two tools, and weight decay alone reached 0.6933.
• DON'T rank a combination by its last epoch. The two tools together ended at 1.2601 and weight decay alone at 0.8670, inside bands of 0.6904 to 1.3898 and 0.8559 to 1.3931 that overlap.
• DON'T expect two tools to add. Dropout lowered the early-stopped loss from 0.7764 to 0.7583 on the baseline and from 0.6933 to 0.6568 on top of weight decay.

Lab Exercises

Lab Exercise 1: Dropout at 0.3 and 0.7

In dropout_rates.py, import what you need from prep_net, train_loop and slice_loop. Seed with 42, build nn.Dropout(0.3), and print its output on torch.ones(10) twice in training mode and once in evaluation mode, then 1 / (1 - p) for 0.3 and for 0.7, rounded to four places.

For p 0.3 and then 0.7, seed with 42, build the lesson's dropout network with that p, and print whether its parameters equal network(512)'s. Fit it on the slice for 3,000 epochs with fit, print the numbers the Theory's measure prints, then the slice's loss and the validation loss in evaluation mode, and three validation losses in training mode under torch.no_grad(). Say in a comment how the four rates, 0.3 and 0.7 beside the Theory's 0.2 and 0.5, rank on the lowest validation loss and on the end.

Lab Exercise 2: Patience for a Narrower Network

In patience.py, import what you need from prep_net, train_loop, slice_loop and stopping, and early-stop four width-128 networks on the slice at patience 50: network(128) alone, the same layers with nn.Dropout(0.5) after the ReLU built after torch.manual_seed(42), the plain one with weight decay 0.1, and the dropout one with weight decay 0.1. Print each one's dropout rate and weight decay, its stopping epoch, best epoch and best loss, and its validation loss after the restore.

Fit the setting with both tools for 3,000 epochs with fit and print the numbers the Theory's measure prints. Early-stop the same setting again at patience 100, 200 and 500, printing the patience and the three returned numbers. Say in a comment which setting and which patience you would carry forward, on which numbers, and what that patience cost in epochs.

import numpy as np
import torch
from torch import nn, optim
from prep_net import X_val, y_val
from train_loop import loss_fn, network, validate
from slice_loop import X_slice, y_slice, fit

torch.manual_seed(42)
drop = nn.Dropout(0.3)
ones = torch.ones(10)
print(drop(ones))
print(drop(ones))
drop.eval()
print(drop(ones))
print(round(1 / (1 - 0.3), 4), round(1 / (1 - 0.7), 4))

tensor([1.4286, 1.4286, 1.4286, 1.4286, 1.4286, 1.4286, 0.0000, 0.0000, 1.4286,
        1.4286])
tensor([1.4286, 1.4286, 0.0000, 0.0000, 1.4286, 1.4286, 1.4286, 0.0000, 1.4286,
        1.4286])
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
1.4286 3.3333

for p in (0.3, 0.7):
    torch.manual_seed(42)
    model = nn.Sequential(nn.Linear(18, 512), nn.ReLU(), nn.Dropout(p), nn.Linear(512, 1))
    plain = network(512)
    print(p, all(torch.equal(a, b) for a, b in zip(model.parameters(), plain.parameters())))
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
    end, lowest = val_losses[-100:], int(np.argmin(val_losses))
    print(f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{min(end):.4f}", f"{max(end):.4f}", f"{val_losses[lowest]:.4f}", lowest + 1)
    print(f"{validate(model, X_slice, y_slice):.4f}", f"{validate(model, X_val, y_val):.4f}")
    with torch.no_grad():
        print([f"{loss_fn(model(X_val), y_val).item():.4f}" for _ in range(3)])
# On the lowest validation loss the rates rank 0.3, 0.2, 0.7, 0.5: 0.6818, 0.7113,
# 0.7510, 0.7583. On the end they do not rank at all, every band overlapping the others.

0.3 True
0.1940 3.2857 2.9847 4.0861 0.6818 37
0.0519 3.2857
['3.5927', '3.2119', '4.1134']
0.7 True
0.4021 3.1319 2.2066 3.5986 0.7510 58
0.1175 3.1319
['3.4739', '3.9317', '4.0688']

import numpy as np
import torch
from torch import nn, optim
from prep_net import X_val, y_val
from train_loop import network, validate
from slice_loop import X_slice, y_slice, fit
from stopping import early_stop

def narrow(p):
    if p == 0:
        return network(128)
    torch.manual_seed(42)
    return nn.Sequential(nn.Linear(18, 128), nn.ReLU(), nn.Dropout(p), nn.Linear(128, 1))

for p, decay in ((0, 0.0), (0.5, 0.0), (0, 0.1), (0.5, 0.1)):
    model = narrow(p)
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=decay)
    stopped, best_epoch, best_loss = early_stop(model, optimizer, X_slice, y_slice, 50, 3000)
    print(p, decay, stopped, best_epoch, f"{best_loss:.4f}", f"{validate(model, X_val, y_val):.4f}")

0 0.0 70 20 0.7514 0.7514
0.5 0.0 75 25 0.7412 0.7412
0 0.1 72 22 0.7109 0.7109
0.5 0.1 75 25 0.7055 0.7055

model = narrow(0.5)
optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)
train_losses, val_losses = fit(model, optimizer, X_slice, y_slice, 3000)
end, lowest = val_losses[-100:], int(np.argmin(val_losses))
print(f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{min(end):.4f}", f"{max(end):.4f}", f"{val_losses[lowest]:.4f}", lowest + 1)
for patience in (100, 200, 500):
    model = narrow(0.5)
    optimizer = optim.SGD(model.parameters(), lr=0.1, weight_decay=0.1)
    stopped, best_epoch, best_loss = early_stop(model, optimizer, X_slice, y_slice, patience, 3000)
    print(patience, stopped, best_epoch, f"{best_loss:.4f}")
# Both tools at patience 500, 0.6641 at epoch 1101, when 1,601 epochs are affordable,
# and both tools at patience 50, 0.7055 after 75 epochs, when they are not.

1.8597 0.8346 0.6733 1.5235 0.6641 1101
100 186 86 0.6853
200 615 415 0.6775
500 1601 1101 0.6641

Theory Assessment

Q1: Eight Ones Through the Layer

After torch.manual_seed(42), nn.Dropout(0.5) on torch.ones(8) printed tensor([2., 2., 2., 2., 0., 2., 0., 0.]), then tensor([2., 2., 2., 2., 0., 0., 2., 0.]), and after drop.eval() eight ones. Say what decides which inputs come out as 0, why the survivors are 2 and not 1, why the two calls differ after one seed, and why evaluation mode returns the ones untouched rather than halved. Then say what nn.Dropout(0.2) prints in place of the 2s and how many of eight inputs it zeros on average.

Hint: Ask what one output's expected value is when it survives with probability 1 - p and comes out multiplied by 1 / (1 - p), and which values the next layer's weights were trained on.

Q2: Six Numbers for One Network

The slice's p 0.5 network printed 0.2917 as the final training loss fit recorded, 0.0899 as its loss on the slice in evaluation mode, 2.9550 as its validation loss through validate, and 3.2493, 3.2345 and 3.6334 as three validation losses in training mode under torch.no_grad(). Say which of these measure the model, what each of the others measures, and why the training-mode losses differ from pass to pass. Then say what a teammate who compares dropout rates on the full rows with the layer left in training mode concludes about raising p, and what the evaluation-mode numbers say instead.

Hint: Ask which passes run through a mask and which do not, and how many masks one pass draws. For the teammate, set the full-rows table's rows side by side twice, once per mode.

Q3: Epoch 88, Epoch 38

early_stop returned 88 38 0.7764 for the baseline at patience 50, and the restored model validates at 0.7764. A teammate changes the line that keeps the best state to best_state = model.state_dict(), prints what the restore returns, sees <All keys matched successfully> and gets the same 88 38 0.7764 back, and ships the model. Say what the model held when the loop broke, what the teammate's restore loaded and what the shipped model validates at, why neither load_state_dict nor the returned numbers gave the change away, and what data_ptr() showed that settles it.

Hint: The patience fixes the distance between the two epochs. Then ask where the tensors of model.state_dict() live, and what a dictionary of references to them holds 50 epochs later.

Q4: A Stop Taken in Training Mode

A teammate wrote this early stop by hand for the slice's p 0.5 network and pasted its output into the comment above the last line:

import copy
import torch
from torch import nn, optim
from prep_net import X_val, y_val
from train_loop import loss_fn, validate
from slice_loop import X_slice, y_slice

torch.manual_seed(42)
model = nn.Sequential(nn.Linear(18, 512), nn.ReLU(), nn.Dropout(0.5), nn.Linear(512, 1))
optimizer = optim.SGD(model.parameters(), lr=0.1)
best_loss, best_epoch = float("inf"), 0
for epoch in range(1, 3001):
    loss = loss_fn(model(X_slice), y_slice)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    with torch.no_grad():
        val_loss = loss_fn(model(X_val), y_val).item()
    if val_loss < best_loss:
        best_loss, best_epoch = val_loss, epoch
        best_state = copy.deepcopy(model.state_dict())
    elif epoch - best_epoch >= 50:
        break
model.load_state_dict(best_state)
# 85 35 0.9427 0.7253
print(epoch, best_epoch, f"{best_loss:.4f}", f"{validate(model, X_val, y_val):.4f}")

The teammate reports that early stopping kept epoch 35 with a validation loss of 0.9427, and that the restored model's 0.7253 beats the Theory's 0.7583 at epoch 49, so this hand-written loop is the better tool. Say what each of the four printed numbers is, which of them measures the kept model, why 0.7253 cannot be set beside 0.7583, and what one change turns the loop into the Theory's.

Hint: Find the line that scores the validation rows and ask which mode the model is in when it runs. Then ask where that pass's masks come from, and what drawing them does to the masks of every later training step.

Assignment

Assignment: The Other Layer the Switches Change

Lesson 4.2.3 named batch normalization beside dropout as the two layers model.eval() changes, and this lesson used only dropout. Batch normalization is nn.BatchNorm1d(512). In training mode it standardizes each of its 512 inputs with the mean and the variance of the batch it is handed, then multiplies each by a learned scale and adds a learned shift, two parameters per input. It also updates two buffers (tensors a module keeps in its state dict that no optimizer steps), running_mean and running_var, as moving averages of the batch statistics, each new value 0.9 of the old one plus 0.1 of the batch's, and a third buffer, num_batches_tracked, counts the batches. In evaluation mode it standardizes with the two running buffers instead of the batch. model.buffers() yields a model's buffers as model.parameters() yields its parameters.

Write batch_norm.py, importing what you need from prep_net, train_loop, slice_loop and stopping. Build nn.Sequential(nn.Linear(18, width), nn.BatchNorm1d(width), nn.ReLU(), nn.Linear(width, 1)) after torch.manual_seed(42) in a function of width, and at width 512 print its state dict's keys, its parameter count and its buffer count, and whether its two linear layers start from network(512)'s draws. Pass the validation rows through it once in training mode under torch.no_grad() and print whether running_mean stayed the same and the batch count, then once in evaluation mode with the same two prints. Print the first validation listing's prediction from batches of the first 2, 10 and 575 validation rows, in training mode and then in evaluation mode, and pass that listing alone in training mode, printing the error, and in evaluation mode, printing the prediction.

Fit a fresh one on the slice at 0.1 for five epochs with fit, reporting epochs 1 and 5. Then, at 0.01, fit it and network(512) for 3,000 epochs each, printing the line the Theory's measure prints and the slice's loss in evaluation mode, and early-stop each at patience 50, printing the three returned numbers and the restored validation loss. On the early-stopped network with batch normalization, print validate's loss, a validation loss taken in training mode under torch.no_grad(), and validate's loss again. Say in a comment whether batch normalization regularized this network on these rows, and on which numbers.

Hints:

• nn.BatchNorm1d draws nothing at construction, its scale starts at ones and its shift at zeros, so the seed reaches the two linear layers exactly as in network(512). Training mode refuses a batch of one row, and the error names the size it got.
• The network with batch normalization does not survive 0.1 on the slice, so the comparison runs both networks at 0.01, one change between them. A tool that regularizes raises the slice's evaluation-mode loss and lowers the validation loss, so read the two side by side.

import numpy as np
import torch
from torch import nn, optim
from prep_net import X_val, y_val
from train_loop import loss_fn, network, validate
from slice_loop import X_slice, y_slice, fit
from stopping import early_stop

def normed(width):
    torch.manual_seed(42)
    return nn.Sequential(nn.Linear(18, width), nn.BatchNorm1d(width), nn.ReLU(), nn.Linear(width, 1))

model = normed(512)
plain = network(512)
print(list(model.state_dict().keys()))
print(sum(p.numel() for p in model.parameters()), sum(b.numel() for b in model.buffers()))
print(torch.equal(model[0].weight, plain[0].weight), torch.equal(model[3].weight, plain[2].weight))

norm = model[1]
before = norm.running_mean.clone()
with torch.no_grad():
    model(X_val)
print(torch.equal(before, norm.running_mean), norm.num_batches_tracked.item())
model.eval()
before = norm.running_mean.clone()
with torch.no_grad():
    model(X_val)
print(torch.equal(before, norm.running_mean), norm.num_batches_tracked.item())

with torch.no_grad():
    model.train()
    print([f"{model(X_val[:n])[0].item():.4f}" for n in (2, 10, 575)])
    model.eval()
    print([f"{model(X_val[:n])[0].item():.4f}" for n in (2, 10, 575)])
model.train()
try:
    model(X_val[:1])
except ValueError as err:
    print(err)
model.eval()
with torch.no_grad():
    print(f"{model(X_val[:1]).item():.4f}")

['0.weight', '0.bias', '1.weight', '1.bias', '1.running_mean', '1.running_var', '1.num_batches_tracked', '3.weight', '3.bias']
11265 1025
True True
False 1
True 1
['-0.1543', '-0.0692', '-0.1687']
['-0.0571', '-0.0571', '-0.0571']
Expected more than 1 value per channel when training, got input size torch.Size([1, 512])
-0.0571

model = normed(512)
fit(model, optim.SGD(model.parameters(), lr=0.1), X_slice, y_slice, 5, (1, 5))
for name, build in (("plain", network), ("batch norm", normed)):
    model = build(512)
    train_losses, val_losses = fit(model, optim.SGD(model.parameters(), lr=0.01), X_slice, y_slice, 3000)
    end, lowest = val_losses[-100:], int(np.argmin(val_losses))
    print(name, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{min(end):.4f}", f"{max(end):.4f}", f"{val_losses[lowest]:.4f}", lowest + 1)
    print(f"{validate(model, X_slice, y_slice):.4f}")
    model = build(512)
    stopped, best_epoch, best_loss = early_stop(model, optim.SGD(model.parameters(), lr=0.01), X_slice, y_slice, 50, 3000)
    print(stopped, best_epoch, f"{best_loss:.4f}", f"{validate(model, X_val, y_val):.4f}")
first = validate(model, X_val, y_val)
with torch.no_grad():
    in_training = loss_fn(model(X_val), y_val).item()
print(f"{first:.4f}", f"{in_training:.4f}", f"{validate(model, X_val, y_val):.4f}")
# Batch normalization did not regularize this network on these rows: at 0.01 it fitted
# the slice tighter, 0.0177 against 0.2330, and validated worse on both readings, its
# end's band above the plain network's and its lowest point 0.9131 against 0.7531.
# One pass in training mode also rewrote its running statistics.

1 23.5133 781.7670
5 inf nan
plain 0.2332 2.0078 1.9697 2.0079 0.7531 115
0.2330
165 115 0.7531 0.7531
batch norm 0.0174 2.8810 2.7428 3.0634 0.9131 41
0.0177
91 41 0.9131 0.9131
0.9131 1.1831 0.9144

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the Assignment's layer worked through by hand, or the same patience used to lower a learning rate instead of stopping.

• YouTube - "Building makemore Part 3: Activations & Gradients, BatchNorm", the chapter "batch normalization" by Andrej Karpathy. From 40:40 to the chapter "batch normalization: summary" at 1:03:07, in a lecture whose description says it introduces batch normalization as "the first modern innovation" that made training deep networks much easier, built in a notebook the description links before a later chapter rewrites the code in PyTorch's own style.
• Official docs - PyTorch 2.14, torch.optim.lr_scheduler.ReduceLROnPlateau. A scheduler that reads a metric after each epoch, the validation loss in its example, and, once it has gone patience epochs without improving, multiplies the learning rate by factor, 0.1 by default, with a threshold under which a lower loss does not count as an improvement, where early_stop counts any lower loss.

Summary

Both tools turned out to be about which model gets reported. Dropout changed the network at every training step and not at all at evaluation, and its whole effect showed only in numbers read in evaluation mode: the training loss fit records and a validation pass that skips eval() both measured thinned networks that never ship. Early stopping changed nothing about the steps. It changed which epoch's parameters leave the loop, and that choice, held in a deep copy, moved the slice's reported validation loss from 3.3360 to 0.7764 with no other change to the run.

The measurements kept refusing simple stories. A dropout network's end wandered so widely that no end could tell either rate from the baseline, its lowest point moved a few hundredths, and weight decay alone beat both on every reading. The two tools together could not be ranked against weight decay at the end and beat it at the minimum, the reading early stopping keeps. Each reading came from one change against one baseline at one rate from one seed, the only arrangement in which a number can be credited to a tool at all, and the same arrangement caught a patience that stopped a narrower network at epoch 75 of a run whose lowest point came at 1101.

What travels from here is the habit more than the tools: score in evaluation mode, keep a copy and never a reference, and judge a combination by its singles at the steady reading. Lesson 4.4.3 takes the copy early_stop keeps in memory and writes it to a file, where a second process can load it, and where a save that never asks whether the model improved quietly keeps the wrong one.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

torch.manual_seed(42)                # network(512)'s draws
model = nn.Sequential(
    nn.Linear(18, 512),
    nn.ReLU(),
    nn.Dropout(0.5),                 # zero half, double the rest
    nn.Linear(512, 1),
)
validate(model, X_val, y_val)        # eval() inside, train() after
lowest = int(np.argmin(val_losses))  # the steady reading
end = val_losses[-100:]              # the band the end wanders

from stopping import early_stop

optimizer = optim.SGD(model.parameters(), lr=0.1)
stopped, best_epoch, best_loss = early_stop(
    model, optimizer, X_slice, y_slice, 50, 3000
)                                  # 99 49 0.7583 for p 0.5
# a copy, never model.state_dict() alone
best_state = copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)  # values into the same tensors