SGD, Adam and the Learning-Rate Sweep

Context

An optimizer is the rule that turns a gradient into a step, and its learning rate is the setting that decides whether a network fits, crawls or blows up. Every loop so far has used SGD at a rate Section 4.2 settled on one neuron, and print(optimizer) has listed momentum at 0 since Lesson 4.2.2.

Every training script sets these two lines, and each of these failures starts there: a rate copied from a script for another optimizer, a default taken on faith, a sweep whose runs share one network. Section 4.5 trains its image model with Adam, Section 4.4 compares its regularized models at the rate this lesson chooses, and Module 4's capstone reports a model's cost in the seconds this lesson's clock measures.

Without the sweep there is no telling whether a validation loss of 0.5876 at a rate of 0.01 is the network's limit or the rate's: at 0.1 the same network reaches 0.4509, at 1.0 it prints nan by epoch 5, and Adam at the rate SGD liked turns up after seventy epochs while its training loss falls from 0.7517 to 0.7100. Lesson 4.3.1 scaled the target because the raw price sent the loss to inf by epoch 3, and this lesson shows the opposite failure on that target, a step so bounded that 500 of them move the mean prediction from near zero to 51 against prices near 385,000.

Goal

• Read what optim.SGD decides at each step and its lr, momentum and weight_decay off its parameter group, and say why a network's loss has no one-step rate.
• Add momentum to SGD, build torch.optim.Adam, read the state each one keeps, and show from a run that Adam's first step is its rate whatever the gradient, where SGD's scales with the gradient.
• Sweep the learning rate of both optimizers across a log grid with a fresh seeded network per run, and print the ten validation losses as one table with each run's lowest value and its epoch.
• Name divergence, a crawl, a plateau and the gap from a sweep's curves, drawn as one figure and checked by printing what it holds.
• Choose the rate by the final validation loss, score the test rows once, and time a fit with time.perf_counter() to report its cost in seconds and parameters beside its loss.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want Adam's two running averages drawn out before you print them.

• YouTube - "Adam Optimization Algorithm (C2W2L08)" by DeepLearningAI. Seven minutes: Adam as momentum and a running average of the squared gradient combined, the correction both averages get on the first steps, and the defaults 0.9, 0.999 and 1e-8 that this lesson prints as betas and eps.
• Official docs - PyTorch 2.14, torch.optim.Adam. The algorithm box that writes the two averages, their correction and the step out line by line, and every parameter with its default, lr at 1e-3, betas at (0.9, 0.999), eps at 1e-8, weight_decay at 0, and decoupled_weight_decay, the switch that turns the class into AdamW, which no lesson here turns on.

Table of Contents

• What an Optimizer Decides
• Momentum, and Adam
• The Sweep: a Log Grid of Rates
• Reading the Curves of a Sweep
• Choosing on Validation, Scoring the Test Once
• The Cost of a Fit

Theory

Every Theory block runs in ~/dev/bootcamp on the 2,875 listings with an area, split as Lesson 4.2.3 split them into 1,725 fit, 575 validation and 575 test rows, prepared as Lesson 4.3.1 prepared them, 18 columns against the price in hundreds of thousands, and fitted with that lesson's regression network, nn.Linear(18, 32), nn.ReLU() and nn.Linear(32, 1), 641 parameters created after torch.manual_seed(42), unless the block names the raw price or another width. The loop is Lesson 4.2.3's, full batch, one step per epoch, with its validation pass under both switches, and every loss prints to four decimals. Lesson 4.3.3's mini-batches stay out, because a sweep of ten runs reads cleaner when an epoch is one step. Every Theory script imports Lesson 4.3.1's prep_net.py and train_loop.py. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

What an Optimizer Decides

What. An optimizer decides two things at every step: which way each parameter moves, and how far. optim.SGD as Lesson 4.2.2 left it moves every parameter against its own gradient by the learning rate times that gradient, p -= lr * p.grad, and nothing else, because momentum and weight_decay, the two settings this module turns on, sit at 0 in the block print(optimizer) showed there. Why. The rate is the setting that matters most because it is the only one live in the loop so far, and because on a network it has no formula. Lesson 4.2.1's six rates on one neuron had one, 1 / (2 * mean(z ** 2)) landing on the bottom in a step, exact, as that lesson said, only for one standardized feature, whose mean(z ** 2) is 1. A ReLU between two layers makes the loss a different function on each side of every hidden unit's zero, and no formula gives its rate, so the rate is found the way Lesson 3.6.3 found a setting, by trying candidates and reading the validation rows. When. Every loop. The optimizer and its rate are the two lines a training script sets before anything about the model is tuned.

train_loop.py holds loss_fn, network, validate, run and score: network() seeds before it builds, so every run below at a given width starts from the same parameters, and run takes an optimizer built outside it, which is what lets this lesson hand it Adam. The reference fit, 200 epochs of SGD at 0.1:

from torch import optim
from train_loop import network, run

model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
run(model, optimizer, 200, report=(1, 10, 50, 100, 200))
print(sum(p.numel() for p in model.parameters()))

1 19.5549 7.9833
10 1.4752 0.9882
50 0.8309 0.4995
100 0.8009 0.4683
200 0.7655 0.4509
641

Lesson 4.3.1's numbers at those epochs. What the optimizer holds:

group = optimizer.param_groups[0]
print(len(optimizer.param_groups), len(group["params"]))
print(group["lr"], group["momentum"], group["weight_decay"])

1 4
0.1 0 0

One group, Lesson 4.2.2's list with one entry, whose dictionary holds the four parameter tensors and the nine settings print(optimizer) listed. The step reads group["lr"] at every call, which is how the Assignment changes the rate while the loop runs, and the next subtitle sets momentum for the first time.

DON'T:

• DON'T carry a rate from one loss, one model or one set of rows to another. 0.1 fits this network and fitted one neuron, and Lesson 4.2.3's two-listing batch printed nan at 0.5 where all the rows took that rate in one step.
• DON'T look for the one-step rate on a network.
• DON'T read momentum: 0 and weight_decay: 0 as settings that do nothing. They are the two this module turns on, and an optimizer line copied from another script may carry either one set.

Momentum, and Adam

What. Momentum (a running sum of past gradients, each older one weighted down, that the step follows instead of the latest gradient alone) is optim.SGD(..., momentum=0.9): at every step the optimizer sets a buffer to 0.9 * buffer + p.grad and moves by lr * buffer. Adam (an optimizer that keeps a running average of each parameter's gradient and of its square, and scales each parameter's step by the two) is torch.optim.Adam(model.parameters(), lr=1e-3), whose step is the rate times the first average divided by the square root of the second, both first corrected for starting at zero, a ratio that does not grow with the gradient. Why. A gradient summed over steps keeps the direction the steps agree on and cancels the part that flips sign, so a small rate moves further with momentum than without. Adam's division by the root of the squared average takes the gradient's size out of the step and leaves the rate to set it, which is why it survives rates SGD cannot and why a raw price target sits still under it. When. Momentum when SGD crawls at the rate the loss tolerates, Adam when the loss's scale is unknown or the gradients differ by orders of magnitude between parameters. Neither replaces the sweep.

Momentum first, at three rates, each run on a fresh network beside the plain run:

import torch
from torch import optim
from prep_net import X_fit, y_fit, fit_rows
from train_loop import network, run, loss_fn

for lr in (0.001, 0.01, 0.1):
    model = network()
    train_plain, val_plain = run(model, optim.SGD(model.parameters(), lr=lr), 200)
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=lr, momentum=0.9)
    train_heavy, val_heavy = run(model, optimizer, 200)
    print(lr, f"{val_plain[-1]:.4f}", f"{val_heavy[-1]:.4f}")
print([f"{v:.4f}" for v in val_heavy[:4]])
print(list(optimizer.state[model[0].weight].keys()))

0.001 3.2759 0.6367
0.01 0.5876 0.4639
0.1 0.4509 0.4720
['7.9833', '2.8067', '12.8068', '3.5753']
['momentum_buffer']

Momentum lifts the two smaller rates and at 0.1 ends above plain SGD. When successive gradients agree the buffer grows toward ten times one gradient, since 1 + 0.9 + 0.81 + ... sums to 10, so momentum at 0.01 moves about as far per step as plain SGD near 0.1, and at 0.1 it reaches past what the loss tolerates: the first validation loss is plain SGD's 7.9833, because the buffer starts as the first gradient, the third epoch jumps to 12.8068, and the curve recovers as the gradient turns against the buffer. The buffer lives in optimizer.state under momentum_buffer, one tensor per parameter, which momentum=0 never allocates.

model = network()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
print(optimizer)

Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    capturable: False
    decoupled_weight_decay: False
    differentiable: False
    eps: 1e-08
    foreach: None
    fused: None
    lr: 0.001
    maximize: False
    weight_decay: 0
)

Eleven settings. lr defaults to 0.001, a hundredth of SGD's 0.1 here. betas weight the two running averages, 0.9 for the gradient's and 0.999 for the squared gradient's, so the first averages over about the last ten steps and the second over about the last thousand, and eps, 1e-8, joins the square root so a parameter whose gradient has been 0 does not divide by 0. weight_decay is Lesson 4.4.1's, and the other seven are switches no lesson of this module touches. One step, and what the optimizer then holds:

loss = loss_fn(model(X_fit), y_fit)
loss.backward()
optimizer.step()
optimizer.zero_grad()
state = optimizer.state[model[0].weight]
print(list(state.keys()))
print(state["step"], state["exp_avg"].shape, state["exp_avg_sq"].shape)
print(len(optimizer.state), 2 * sum(p.numel() for p in model.parameters()))

['step', 'exp_avg', 'exp_avg_sq']
tensor(1.) torch.Size([32, 18]) torch.Size([32, 18])
4 1282

step, a count, and the two averages, each the shape of its parameter: the averages hold 1,282 numbers beside the network's 641, where momentum's buffer holds 641 and plain SGD keeps nothing. Why the step is bounded, on the raw price, Lesson 4.3.1's target before it was scaled: each optimizer's first step from the seeded start, the largest gradient beside the largest move.

y_raw = torch.tensor(fit_rows["price"].to_numpy(), dtype=torch.float32).unsqueeze(1)

def first_step(optimizer_class, lr):
    model = network()
    optimizer = optimizer_class(model.parameters(), lr=lr)
    before = [p.detach().clone() for p in model.parameters()]
    loss = loss_fn(model(X_fit), y_raw)
    loss.backward()
    largest_grad = max(p.grad.abs().max().item() for p in model.parameters())
    optimizer.step()
    moves = [(p.detach() - b).abs().max().item() for p, b in zip(model.parameters(), before)]
    print(f"{largest_grad:.4e} {max(moves):.4e}")

first_step(optim.SGD, 0.1)
first_step(optim.Adam, 0.001)

7.6959e+05 7.6959e+04
7.6959e+05 1.0000e-03

SGD at 0.1 moves the parameter with the largest gradient by a tenth of it, and Adam at 0.001 by its rate: on the first step the two corrected averages are the gradient and its square, their ratio is the gradient's sign, and every parameter whose gradient is not zero moves by the rate. SGD's step is as large as the gradient makes it, which sent the raw price's loss to inf by epoch 3 in Lesson 4.3.1, and Adam's is set by the rate, which is why 500 of them go nowhere on a target near 385,000:

def adam_on_raw(lr):
    model = network()
    optimizer = optim.Adam(model.parameters(), lr=lr)
    for epoch in range(500):
        loss = loss_fn(model(X_fit), y_raw)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    with torch.no_grad():
        mean_pred = model(X_fit).mean().item()
    print(f"{loss.item():.4f} {model[2].bias.item():.2f} {mean_pred:,.1f}")

print(f"{network()[2].bias.item():.4f} {y_raw.mean().item():,.1f}")
adam_on_raw(0.001)
adam_on_raw(10.0)

0.0361 384,795.4
190712250368.0000 0.54 50.9
7253249024.0000 142.53 384,829.6

The output bias starts at 0.0361 against a mean price of 384,795.4, and 500 epochs at the default rate move it to 0.54, the mean prediction to 50.9 and the loss almost nowhere. The same loop at a rate of 10 moves the mean prediction to 384,829.6 in the same 500 epochs, so it was the rate and not the loss that held the network still. The scaled target is still the fix, not a rate of 10, because on it 0.1 trains under both optimizers.

DON'T:

• DON'T carry a rate between optimizers. 0.1 is SGD's best rate on the sweep's grid and the rate at which Adam's validation loss turns up, and Adam's default 0.001 is a rate at which SGD reaches only 3.2759.
• DON'T take Adam's default rate as right for every loss. Its step does not grow with the gradient, and on the raw price it moved the bias from 0.04 to 0.54 in 500 epochs.
• DON'T set momentum and keep the rate that worked without it. Momentum's best rate on the grid is 0.01, a tenth of plain SGD's.

The Sweep: a Log Grid of Rates

What. A learning-rate sweep (one fit per candidate rate, each from a fresh seeded network, compared by the validation loss) on a log grid (candidates spaced by a constant factor, ten here, rather than a constant difference): 0.0001, 0.001, 0.01, 0.1 and 1.0, for SGD and for Adam, 200 epochs each, the ten final validation losses in one table. Why. The rate multiplies the step under both optimizers, so 0.001 against 0.01 is the same kind of change as 0.1 against 1.0, and a grid spaced by a constant difference puts four of its five candidates in the range's last decade, 0.1 to 1.0. A fresh network per run is what makes the table a comparison of rates, because a model carried from one rate to the next arrives already fitted, and this is Lesson 3.6.3's search over one setting with the loss curve as the readout. When. For every new model, loss or dataset, before any other setting is tuned, and again whenever one of the three changes.

import numpy as np
import matplotlib.pyplot as plt
from torch import optim
from prep_net import X_val, X_test, val_rows, test_rows
from train_loop import network, run, score

rates = [0.0001, 0.001, 0.01, 0.1, 1.0]
results = {}
for name, optimizer_class in (("SGD", optim.SGD), ("Adam", optim.Adam)):
    for lr in rates:
        model = network()
        optimizer = optimizer_class(model.parameters(), lr=lr)
        train_losses, val_losses = run(model, optimizer, 200)
        results[(name, lr)] = (model, train_losses, val_losses)

print(f"{'rate':>8} {'SGD':>9} {'Adam':>9}")
for lr in rates:
    sgd = results[("SGD", lr)][2][-1]
    adam = results[("Adam", lr)][2][-1]
    print(f"{lr:>8} {sgd:>9.4f} {adam:>9.4f}")

    rate       SGD      Adam
  0.0001   14.8656   15.1968
   0.001    3.2759    1.5586
    0.01    0.5876    0.4669
     0.1    0.4509    0.4585
     1.0       nan    1.1401

SGD's column improves at every tenfold step up to 0.1 and prints nan at 1.0. Adam's improves up to 0.1 as well, to 0.4585, and prints a number at 1.0 where SGD printed none. An ending never says whether a run was still falling, so the table carries each run's lowest validation loss and its epoch as well, through np.nanargmin (argmin that skips nan):

for (name, lr), (model, train_losses, val_losses) in results.items():
    lowest = int(np.nanargmin(val_losses))
    row = [train_losses[-1], val_losses[-1], val_losses[lowest]]
    print(f"{name:>4} {lr:>6}", " ".join(f"{v:8.4f}" for v in row), lowest + 1)

 SGD 0.0001  17.2746  14.8656  14.8656 200
 SGD  0.001   4.1572   3.2759   3.2759 200
 SGD   0.01   0.9320   0.5876   0.5876 200
 SGD    0.1   0.7655   0.4509   0.4509 200
 SGD    1.0      nan      nan 302.0385 1
Adam 0.0001  17.6288  15.1968  15.1968 200
Adam  0.001   2.0742   1.5586   1.5586 200
Adam   0.01   0.7441   0.4669   0.4669 200
Adam    0.1   0.7100   0.4585   0.4324 70
Adam    1.0   0.8724   1.1401   1.1401 200

Final training loss, final validation loss, lowest validation loss and its epoch. Eight of the nine finite runs are lowest at epoch 200, still falling when the budget ran out, and Adam at 0.1 is the one whose lowest value, 0.4324, sits at epoch 70, below every other number in the table. Why the grid is logarithmic, from a run:

print(np.linspace(0.0001, 1.0, 5))
for lr in (0.250075, 0.50005, 0.750025):
    model = network()
    train_losses, val_losses = run(model, optim.SGD(model.parameters(), lr=lr), 200)
    print(lr, f"{val_losses[-1]:.4f}")

[1.00000e-04 2.50075e-01 5.00050e-01 7.50025e-01 1.00000e+00]
0.250075 0.7037
0.50005 nan
0.750025 nan

A constant difference spends one candidate below 0.25 and three at or past the edge where SGD diverges, 0.7037 at 0.250075 and nan at 0.50005 and 0.750025, and never tries 0.1. A log grid spends one candidate per decade, four of its five below 0.25.

DON'T:

• DON'T share one network across the sweep. A model built once above the loop over rates arrives at each rate already fitted by every rate before it, and Lab Exercise 1 prints that table: 0.4468 at 0.1, below the fresh run's 0.4509, from 600 extra epochs that no rate owns.
• DON'T sweep on a linear grid.
• DON'T read the final column alone. Adam at 0.1's ending cannot show the 0.4324 it passed through.

Reading the Curves of a Sweep

What. Divergence (a loss that grows without bound, printing inf and then nan), a crawl (a curve still falling at the budget's end, far from where the faster rates settle), a plateau (a curve that flattens far above where the other rates settle) and the gap (a validation curve that turns up while its training curve keeps falling, Lesson 4.2.3's), each named from the run's numbers and drawn as Lesson 4.2.3 drew its curves. Why. The table says where each run ended, and the curve says how: whether the number is a floor, a point on a descent or the far side of a turn. A rate is chosen from the table and understood from the curve. When. After every sweep, on the figure, before the table's winner is reported.

Divergence first, SGD at 1.0, both lists for the first five epochs, continuing the sweep script:

model, train_losses, val_losses = results[("SGD", 1.0)]
print([f"{v:.4f}" for v in train_losses[:5]])
print([f"{v:.4f}" for v in val_losses[:5]])

['19.5549', '311.6738', '10692084.0000', '11397270835985582653440.0000', 'inf']
['302.0385', '13333239.0000', '9103545324519282442240.0000', 'inf', 'nan']

The training list records the loss each step was taken from, so its first entry is the seeded start, 19.5549, while the validation loss after that one step is already 302.0385 where 0.1's landed at 7.9833. Epoch 3 prints a number with 22 digits, epoch 4 inf, and from epoch 5 nan: the loss overflows first, while every parameter is still a number, the gradient computed from the overflow holds inf and nan, the next step writes both into the parameters, and no later step removes a nan. The crawl, SGD at 0.0001:

val_losses = results[("SGD", 0.0001)][2]
print([f"{val_losses[epoch - 1]:.4f}" for epoch in (1, 50, 100, 150, 200)])
model = network()
train_losses, slow = run(model, optim.SGD(model.parameters(), lr=0.0001), epochs=2000)
print(f"{slow[-1]:.4f}", f"{results[('SGD', 0.001)][2][-1]:.4f}")

['16.9791', '16.4268', '15.8865', '15.3669', '14.8656']
3.2760 3.2759

It falls at every printed epoch and keeps falling past the budget: run for 2,000 epochs it ends at 3.2760, where SGD at 0.001 ended after 200. A crawl is the same descent at a tenth of the speed, and what a rate too small costs is the budget, not the fit. The plateau, Adam at 1.0:

val_losses = results[("Adam", 1.0)][2]
print([f"{val_losses[epoch - 1]:.4f}" for epoch in (1, 2, 5, 10, 50, 100, 150, 200)])
print(round(val_losses[-1] / results[("SGD", 0.1)][2][-1], 2))

['2755.3843', '11.2803', '73.8227', '22.0252', '2.0727', '1.5610', '1.2193', '1.1401']
2.53

It does not diverge, because the step is bounded: 2755.3843 is followed by 11.2803, not by inf, and the curve climbs back to 73.8227 at epoch 5 and stays in the tens through epoch 18. It does not settle either: from 2.0727 at epoch 50 it slides to 1.1401 at 200, two and a half times SGD 0.1's ending. The same optimizer at 0.01 ended at 0.4669, so the rate is what holds the curve up. The gap, Adam at 0.1:

model, train_losses, val_losses = results[("Adam", 0.1)]
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4f}", f"{train_losses[lowest]:.4f}")
print(f"{val_losses[-1]:.4f}", f"{train_losses[-1]:.4f}", round(val_losses[-1] / val_losses[lowest], 3))
print(all(v > val_losses[lowest] for v in val_losses[100:]))

70 0.4324 0.7517
0.4585 0.7100 1.061
True

Lowest at epoch 70, 0.4324, then 0.4585 at 200, 6.1 percent above and above the lowest at every epoch after the hundredth, while the training loss fell from 0.7517 to 0.7100. That is Lesson 4.2.3's gap, drawn by a rate instead of noise columns, and the epoch of the turn is the one whose parameters Lesson 4.4.2's early stopping keeps. The figure, all ten curves on two Axes, Lesson 2.7.2's plt.subplots(1, 2):

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
for ax, name in zip(axes, ("SGD", "Adam")):
    for lr in rates:
        ax.plot(range(1, 201), results[(name, lr)][2], label=f"lr={lr}")
    ax.set_yscale("log")
    ax.set_xlabel("Epoch")
    ax.set_ylabel("Validation mean squared error")
    ax.set_title(name)
    ax.legend()
fig.savefig("sweep_curves.png", dpi=150, bbox_inches="tight")
print(len(axes[0].lines), len(axes[1].lines))
print(int(np.isfinite(axes[0].lines[4].get_ydata()).sum()))
print(int(np.argmin(axes[1].lines[3].get_ydata())) + 1)
print([t.get_text() for t in axes[1].get_legend().get_texts()])
plt.close(fig)

5 5
3
70
['lr=0.0001', 'lr=0.001', 'lr=0.01', 'lr=0.1', 'lr=1.0']

Five lines per Axes on a log scale. The SGD 1.0 line holds three finite points of its 200, counted by np.isfinite (true for a number that is neither inf nor nan), and matplotlib draws nothing at the other 197, the Adam 0.1 line's lowest point is at epoch 70, and the legend names the five rates in the order drawn. In sweep_curves.png, the fifth SGD line's climb to 9.1e21 by its third epoch stretches that panel's axis past 1e23 and presses the four descents, stacked by rate, against its bottom, and Adam's panel has the 0.1 curve lowest at epoch 70 and rising after it.

DON'T:

• DON'T read a rising validation curve as a rate too large. A rate too large sends both curves up, as 1.0 did by epoch 2, and Adam at 0.1's training curve fell from 0.7517 to 0.7100 while its validation curve rose.
• DON'T stop a crawl early and call the rate a failure.
• DON'T trust a curve you have not printed. Three finite points on the SGD 1.0 line is what the figure holds, and the picture shows a line that stops.

Choosing on Validation, Scoring the Test Once

What. The pick is the optimizer and rate whose final validation loss at 200 epochs is lowest, Lesson 3.1.2's rule applied to a sweep. The test rows are then scored once with that model through train_loop's score, Lesson 3.1.4's mean_absolute_error and r2_score on predictions multiplied back by 100,000. Why. The sweep compared its ten runs at one fixed budget, and the rule that reads it is fixed before the table is. Adam at 0.1's 0.4324 at epoch 70 is a choice of the epoch as well, a setting the sweep held fixed, which Lesson 4.4.2 makes with a tool built for it. The test rows reached no decision, so their one score stands in for a listing the model has never seen. When. Once, after the sweep, and the test score once per model, never again after a later choice.

finals = {key: val_losses[-1] for key, (model, train_losses, val_losses) in results.items()}
finals = {key: loss for key, loss in finals.items() if np.isfinite(loss)}
best = min(finals, key=finals.get)
print(best, f"{finals[best]:.4f}", len(finals))

('SGD', 0.1) 0.4509 9

finals keeps the nine finite endings, because nan compares as smaller than nothing and as larger than nothing, and a min that meets it first keeps it. min(finals, key=finals.get) is Lesson 3.4.1's pick of its best C: SGD at 0.1. The chosen model's validation score in currency, then the test rows:

chosen = results[best][0]
mae, r2 = score(chosen, X_val, val_rows)
print(f"{mae:,.0f}", round(r2, 3))
mae, r2 = score(chosen, X_test, test_rows)
print(f"{mae:,.0f}", round(r2, 3))

45,949 0.856
45,738 0.843

The pick misses a validation listing by 45,949 on average and explains 85.6 percent of the prices' variance, 364 above Lesson 4.3.1's 500-epoch fit at the same rate and 6,776 below Module 3's linear model. On the test rows, once: MAE 45,738 and R² 0.843, the section's one test score on the price, compared with no other rate, optimizer or epoch count, because a choice made by the test score makes the test rows the validation rows under another name, Lesson 3.1.2's second break.

DON'T:

• DON'T pick by the training loss. Adam at 0.1 has the lowest training loss of the ten, 0.7100, and a validation loss above SGD 0.1's, because the training loss is the number every step was computed to lower.
• DON'T report Adam 0.1's 0.4324 as the sweep's best. It is epoch 70's number for parameters the loop stepped away from 130 times, and the model in results scores 0.4585.
• DON'T score the test rows for a second candidate. 45,738 is this section's one test score on the price, and the Assignment's better validation loss is left without one for that reason.

The Cost of a Fit

What. time.perf_counter(), Lesson 3.6.3's clock, around a loop, so that a fit reports three numbers: its validation loss, its parameter count and its seconds. Why. A sweep is ten fits and a wider network is a dearer fit, and the seconds decide how many candidates a budget allows, which the loss never says. Module 4's capstone reports a model's cost beside its score, and this is where the cost is measured. When. Once per model shape, before a sweep is sized, and in every report that compares models of different sizes.

import time
import torch
from torch import optim
from train_loop import network, run

def timed(width, epochs):
    model = network(width)
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    start = time.perf_counter()
    train_losses, val_losses = run(model, optimizer, epochs)
    seconds = time.perf_counter() - start
    params = sum(p.numel() for p in model.parameters())
    return params, seconds, val_losses[-1]

params, seconds, final = timed(32, 500)
print(params, f"{seconds:.2f}", f"{seconds / 500 * 1000:.2f}", f"{final:.4f}")

641 0.47 0.94 0.4442

The seconds are this machine's, and yours will differ: another processor prints another number, and so does this machine on another run. What holds from run to run is the order, about a millisecond an epoch and about half a second for 500, while the loss, 0.4442, Lesson 4.3.1's 500-epoch number, does not move at all. The widths of that lesson's preview, timed for 1,000 epochs each:

for width in (8, 32, 128, 512):
    params, seconds, final = timed(width, 1000)
    print(width, params, f"{seconds:.2f}", f"{final:.4f}")

8 161 0.71 0.4612
32 641 0.83 0.4541
128 2561 1.38 0.4743
512 10241 10.42 0.4827

The widest network cost more than ten times the narrowest's seconds for a validation loss 0.0215 higher: width is paid for in seconds and parameters and, on these rows, past 32 units it does not pay back, which is what Lesson 4.3.1's preview said. The optimizers, per epoch on the width-32 network, with __name__ (the class's own name as a string) labelling each:

for optimizer_class, lr in ((optim.SGD, 0.1), (optim.Adam, 0.01)):
    model = network()
    start = time.perf_counter()
    run(model, optimizer_class(model.parameters(), lr=lr), epochs=500)
    print(optimizer_class.__name__, f"{(time.perf_counter() - start) / 500 * 1000:.2f}")

SGD 0.96
Adam 1.13

Adam's epoch costs more than SGD's, two averages updated per parameter on top of the step itself. The sweep's price, ten fits of 200 epochs:

start = time.perf_counter()
for optimizer_class in (optim.SGD, optim.Adam):
    for lr in (0.0001, 0.001, 0.01, 0.1, 1.0):
        model = network()
        run(model, optimizer_class(model.parameters(), lr=lr), 200)
print(f"{time.perf_counter() - start:.2f}")

2.13

About two seconds on this network, which is why the rate is swept and never guessed. The same ten fits at width 512 would take about ten times as long by the table above, and Section 4.5's 60,000 images take seconds per epoch, which is when the seconds decide how many candidates a sweep gets.

DON'T:

• DON'T draw a conclusion from a timing's digits, or compare them across machines. 0.71 and 10.42 are this run's.
• DON'T choose a width without its seconds.
• DON'T report seconds without the epochs and the parameters beside them. 10.42 seconds says nothing until 1,000 epochs and 10,241 parameters stand next to it.

Lab Exercises

Lab Exercise 1: The Sweep Done Wrong, Then With Momentum

In shared_sweep.py, import network and run from train_loop.py and sweep the Theory's five rates for SGD and for Adam twice: first with a fresh network() for every run, keeping each final validation loss, then with one network() built before the loop over rates for each optimizer and a new optimizer on that same model at every rate, the rates in increasing order. Print the ten runs as one table, optimizer, rate, the fresh number and the shared number, and say in a comment which shared numbers sit below the fresh ones and what a shared number measures.

Then sweep the five rates for SGD with momentum=0.9, a fresh network per run, and print for each rate the plain SGD number, the momentum run's final validation loss, its lowest and that value's epoch, through np.nanargmin. Pick momentum's best rate by its final validation loss among the finite endings with min and a dictionary's get, print it with its loss, and compare it with plain SGD's best in a comment.

Lab Exercise 2: Sweeping the Classifier's Rate

In classifier_sweep.py, import Lesson 4.3.2's table and label from sold_prep.py, the 18 columns of prep_net.py with the fit rows' standardized price as the 19th and sold_in_30_days as a float32 (n, 1) tensor, for all three sets. Print the fit table's shape, the label's, and the validation and test rows' share of sales to three decimals. Write the classifier as Lesson 4.3.2 built it, nn.Linear(19, 32), nn.ReLU() and nn.Linear(32, 1) after the seed, a validation pass under both switches for nn.BCEWithLogitsLoss, and the loop that records both losses per epoch. Sweep the five rates for SGD and Adam, 200 epochs each from a fresh classifier, timing the ten fits with time.perf_counter(), and print one line per run: optimizer, rate, final training loss, final validation loss, lowest validation loss and its epoch.

Draw the ten validation curves on two log-scaled Axes, save the figure as classifier_sweep.png, and print how many lines each Axes holds and the epoch of the lowest point of Adam's 0.1 line. Pick the run with the lowest final validation loss, print it, then its validation AUC, its largest validation probability and the number of validation listings it calls a sale at 0.5. Score the test rows once with accuracy at 0.5 and AUC, print the test rows' majority share and how many test listings the model calls a sale, and say in a comment which runs showed the gap and which rate won, against the regression sweep's.

import numpy as np
import torch
from torch import optim
from train_loop import network, run

rates = [0.0001, 0.001, 0.01, 0.1, 1.0]
optimizers = (("SGD", optim.SGD), ("Adam", optim.Adam))
fresh = {}
for name, optimizer_class in optimizers:
    for lr in rates:
        model = network()
        train_losses, val_losses = run(model, optimizer_class(model.parameters(), lr=lr), 200)
        fresh[(name, lr)] = val_losses[-1]

shared = {}
for name, optimizer_class in optimizers:
    model = network()
    for lr in rates:
        train_losses, val_losses = run(model, optimizer_class(model.parameters(), lr=lr), 200)
        shared[(name, lr)] = val_losses[-1]

for key in fresh:
    print(f"{key[0]:>4} {key[1]:>6} {fresh[key]:>9.4f} {shared[key]:>9.4f}")
# Below the fresh number: SGD at 0.001, 0.01 and 0.1, Adam at 0.001 and 0.1.
# A shared number is the network after every earlier rate's 200 epochs plus
# 200 at this one, so SGD's 0.4468 at 0.1 is an 800-epoch fit, not 0.1's.

 SGD 0.0001   14.8656   14.8656
 SGD  0.001    3.2759    2.8405
 SGD   0.01    0.5876    0.5649
 SGD    0.1    0.4509    0.4468
 SGD    1.0       nan       nan
Adam 0.0001   15.1968   15.1968
Adam  0.001    1.5586    1.4895
Adam   0.01    0.4669    0.4984
Adam    0.1    0.4585    0.4568
Adam    1.0    1.1401    1.5169

heavy = {}
for lr in rates:
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=lr, momentum=0.9)
    train_losses, val_losses = run(model, optimizer, 200)
    heavy[lr] = val_losses
for lr in rates:
    lowest = int(np.nanargmin(heavy[lr]))
    plain = fresh[("SGD", lr)]
    print(f"{lr:>6} {plain:>9.4f} {heavy[lr][-1]:>9.4f} {heavy[lr][lowest]:>9.4f} {lowest + 1:>4}")
finite = {lr: heavy[lr][-1] for lr in rates if np.isfinite(heavy[lr][-1])}
best = min(finite, key=finite.get)
print(best, f"{finite[best]:.4f}")
# Momentum's best on the grid, 0.01 at 0.4639, sits above plain SGD's best,
# 0.1 at 0.4509: the buffer buys a smaller rate the distance a larger one
# covers alone.

0.0001   14.8656    3.5168    3.5168  200
 0.001    3.2759    0.6367    0.6367  200
  0.01    0.5876    0.4639    0.4639  200
   0.1    0.4509    0.4720    0.4705  172
   1.0       nan       nan  302.0385    1
0.01 0.4639

import time
import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import nn, optim
from sklearn.metrics import accuracy_score, roc_auc_score
from sold_prep import X_fit, y_fit, X_val, y_val, X_test, y_test

print(X_fit.shape, y_fit.shape, round(y_val.mean().item(), 3), round(y_test.mean().item(), 3))

bce = nn.BCEWithLogitsLoss()

def classifier():
    torch.manual_seed(42)
    return nn.Sequential(nn.Linear(19, 32), nn.ReLU(), nn.Linear(32, 1))

def validate_bce(model, X, y):
    model.eval()
    with torch.no_grad():
        loss = bce(model(X), y)
    model.train()
    return loss.item()

def run_classifier(model, optimizer, epochs=200):
    train_losses, val_losses = [], []
    for epoch in range(epochs):
        loss = bce(model(X_fit), y_fit)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        train_losses.append(loss.item())
        val_losses.append(validate_bce(model, X_val, y_val))
    return train_losses, val_losses

rates = [0.0001, 0.001, 0.01, 0.1, 1.0]
results = {}
start = time.perf_counter()
for name, optimizer_class in (("SGD", optim.SGD), ("Adam", optim.Adam)):
    for lr in rates:
        model = classifier()
        optimizer = optimizer_class(model.parameters(), lr=lr)
        train_losses, val_losses = run_classifier(model, optimizer)
        results[(name, lr)] = (model, train_losses, val_losses)
print(f"{time.perf_counter() - start:.2f}")
for (name, lr), (model, train_losses, val_losses) in results.items():
    lowest = int(np.nanargmin(val_losses))
    row = [train_losses[-1], val_losses[-1], val_losses[lowest]]
    print(f"{name:>4} {lr:>6}", " ".join(f"{v:8.4f}" for v in row), lowest + 1)

torch.Size([1725, 19]) torch.Size([1725, 1]) 0.202 0.219
3.62
 SGD 0.0001   0.7476   0.7479   0.7479 200
 SGD  0.001   0.7073   0.7068   0.7068 200
 SGD   0.01   0.5414   0.5370   0.5370 200
 SGD    0.1   0.4986   0.4934   0.4934 200
 SGD    1.0   0.4587   0.4729   0.4729 200
Adam 0.0001   0.6788   0.6781   0.6781 200
Adam  0.001   0.4970   0.4938   0.4938 200
Adam   0.01   0.4496   0.4812   0.4754 133
Adam    0.1   0.3685   0.5928   0.4764 33
Adam    1.0   0.4783   0.7148   0.5027 22

fig, axes = plt.subplots(1, 2, figsize=(14, 5))
for ax, name in zip(axes, ("SGD", "Adam")):
    for lr in rates:
        ax.plot(range(1, 201), results[(name, lr)][2], label=f"lr={lr}")
    ax.set_yscale("log")
    ax.set_xlabel("Epoch")
    ax.set_ylabel("Validation binary cross-entropy")
    ax.set_title(name)
    ax.legend()
fig.savefig("classifier_sweep.png", dpi=150, bbox_inches="tight")
print(len(axes[0].lines), len(axes[1].lines))
print(int(np.argmin(axes[1].lines[3].get_ydata())) + 1)
plt.close(fig)

finals = {key: val_losses[-1] for key, (model, train_losses, val_losses) in results.items()}
best = min(finals, key=finals.get)
print(best, f"{finals[best]:.4f}")

def probabilities(model, X):
    model.eval()
    with torch.no_grad():
        prob = torch.sigmoid(model(X)).squeeze(1).numpy()
    model.train()
    return prob

chosen = results[best][0]
prob_val = probabilities(chosen, X_val)
truth_val = y_val.squeeze(1).numpy()
print(round(roc_auc_score(truth_val, prob_val), 3), round(prob_val.max(), 3), int((prob_val >= 0.5).sum()))
prob_test = probabilities(chosen, X_test)
truth_test = y_test.squeeze(1).numpy()
print(round(accuracy_score(truth_test, prob_test >= 0.5), 3), round(roc_auc_score(truth_test, prob_test), 3))
print(round(1 - truth_test.mean(), 3), int((prob_test >= 0.5).sum()))
# Adam at 0.01, 0.1 and 1.0 turned up while their training losses ended
# lower, 0.1 lowest at epoch 33 and 0.5928 at 200 with the training loss at
# 0.3685. SGD at 1.0 won, the rate the regression sweep could not survive:
# a rate belongs to the loss it was swept on.

5 5
33
('SGD', 1.0) 0.4729
0.668 0.487 0
0.781 0.622
0.781 0

Theory Assessment

Q1: One Rate, Two Steps

SGD at 1.0 printed a validation loss of 302.0385 after its first epoch and nan from its fifth, while Adam at 1.0 printed 2755.3843 after its first and 1.1401 at its two-hundredth. On the raw price, the same seeded network's largest gradient was 7.6959e+05, and the first step moved that parameter by 7.6959e+04 under SGD at 0.1 and by 1.0000e-03 under Adam at 0.001. Say what each optimizer's step is made of, why neither of Adam's runs could print inf, why Adam at 0.001 could not reach a target near 385,000 in 500 steps, and what a rate that worked for one optimizer says about the other.

Hint: Write both steps as a product, a rate times something. For Adam, ask what the ratio of a gradient's running average to the root of its squared average can be at most, and multiply that by 500 steps.

Q2: The Grid and the Seed

Say why the sweep's five rates are spaced by a factor of ten and what np.linspace(0.0001, 1.0, 5) would have spent its five candidates on, from the numbers the Theory printed. Then a teammate moves model = network() above the loop over rates to save the seeding, runs SGD's five rates in increasing order, gets 0.4468 at 0.1, below the sweep's 0.4509, and reports 0.1 as even better than the sweep said. Say what the 0.4468 measures, how many epochs the network behind it has run, and why the number is not the rate's.

Hint: Count the epochs the shared network has seen by the time its 0.1 run starts, then ask where each of the sweep's runs started.

Q3: Four Shapes, One Rule

From the sweep's table, name the shape of SGD at 1.0, SGD at 0.0001, Adam at 1.0 and Adam at 0.1, with the numbers that show each, and say for each what it says about the rate. Then say which single number the pick reads, why Adam 0.1's 0.4324 is not that number, and what the test score of 45,738 may be used for after it is printed.

Hint: The four shapes are the fourth subtitle's four names. For the pick, ask which settings the sweep varied and which one a lowest-ever value would be choosing.

Q4: The Sweep That Picked by the Wrong Number

Read this script without running it.

from torch import optim
from prep_net import X_test, test_rows
from train_loop import network, run, score

results = {}
for name, optimizer_class in (("SGD", optim.SGD), ("Adam", optim.Adam)):
    for lr in (0.0001, 0.001, 0.01, 0.1, 1.0):
        model = network()
        train_losses, val_losses = run(model, optimizer_class(model.parameters(), lr=lr), 200)
        results[(name, lr)] = (model, train_losses, val_losses)

best = min(results, key=lambda key: results[key][1][-1])
print(best, f"{results[best][1][-1]:.4f}")
print(f"{min(results[best][2]):.4f}")
chosen = results[best][0]
mae, r2 = score(chosen, X_test, test_rows)
print(f"{mae:,.0f}")

A teammate reports that the script found a better model than the lesson's, with a validation loss of 0.4324 against 0.4509, and then reports its test MAE. Say which run the script picks and by what number, what its second line reports and which parameters that number belongs to, why the nan run was never picked, and what the third line scores that it should not.

Hint: Ask what the training loss is the number of, which epoch a list's smallest value belongs to, and which model results[best][0] holds after 200 epochs. For the nan, ask what nan < 17.2746 returns.

Assignment

Assignment: The Rate Halved on a Timetable

Adam at 0.1 passed through the sweep's lowest validation loss at epoch 70 and ended above SGD's pick, and SGD at 0.1 was still descending when the 200 epochs ran out. A schedule changes the rate while the loop runs, and this assignment measures what one does to each optimizer's curve. Write schedule.py on Lesson 4.3.1's modules.

What this lesson did not use: torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.5) is a scheduler (an object that rewrites the optimizer's lr on a timetable): every call of scheduler.step() counts one epoch, and every step_size epochs it multiplies every parameter group's lr by gamma, so the call goes after optimizer.step(), once per epoch, and optimizer.param_groups[0]["lr"] is where the change lands. scheduler.get_last_lr() returns the current rates as a list, one per group, and scheduler.last_epoch counts the scheduler's own steps.

Write a loop that takes a model, an optimizer and a scheduler, which may be None, runs 200 epochs and records three lists, the training loss, the validation loss and the group's lr after each epoch. Run four fits from fresh networks, SGD and Adam at 0.1, each once with a fixed rate and once halved every 50 epochs, printing get_last_lr() and last_epoch after each halved run. Print the halved SGD run's rate at epochs 1, 50, 51, 100, 101, 150, 151 and 200, the fixed SGD run's rate at its first and last epoch, and one line per run with the final training loss, the final validation loss, the lowest validation loss and its epoch. Draw the four validation curves on one log-scaled Axes with a dashed vertical line at epochs 50, 100 and 150, save it as schedule_curves.png, and print how many lines the Axes holds, the legend's entries, and the epoch of the halved Adam curve's lowest point with the curve's last value. Print the fixed and the halved Adam curves at epochs 70, 100, 150 and 200, and the fixed and the halved SGD curves at 50, 100, 150 and 200. Then say which run the lesson's rule reports, what the schedule did to each optimizer's curve, and what happens to the test rows.

Hints:

• The rate scheduler.step() leaves in param_groups[0]["lr"] is the one the next epoch steps with, so the value recorded at epoch 50 is already 0.05 and epoch 51 is the first to move at it.
• Keep the four runs in one dictionary keyed by optimizer and kind, so one loop draws the chart and one prints the table, and remember that ax.lines holds the dashed lines after the four curves.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import optim
from torch.optim.lr_scheduler import StepLR
from prep_net import X_fit, y_fit, X_val, y_val
from train_loop import network, validate, loss_fn

def run_scheduled(model, optimizer, scheduler=None, epochs=200):
    train_losses, val_losses, rates = [], [], []
    for epoch in range(epochs):
        loss = loss_fn(model(X_fit), y_fit)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        if scheduler is not None:
            scheduler.step()
        train_losses.append(loss.item())
        val_losses.append(validate(model, X_val, y_val))
        rates.append(optimizer.param_groups[0]["lr"])
    return train_losses, val_losses, rates

curves = {}
for name, optimizer_class in (("SGD", optim.SGD), ("Adam", optim.Adam)):
    model = network()
    optimizer = optimizer_class(model.parameters(), lr=0.1)
    curves[(name, "fixed")] = run_scheduled(model, optimizer)
    model = network()
    optimizer = optimizer_class(model.parameters(), lr=0.1)
    scheduler = StepLR(optimizer, step_size=50, gamma=0.5)
    curves[(name, "halved")] = run_scheduled(model, optimizer, scheduler)
    print(scheduler.get_last_lr(), scheduler.last_epoch)

rates = curves[("SGD", "halved")][2]
print([rates[epoch - 1] for epoch in (1, 50, 51, 100, 101, 150, 151, 200)])
print(curves[("SGD", "fixed")][2][:1], curves[("SGD", "fixed")][2][-1:])
for key, (train_losses, val_losses, rates) in curves.items():
    lowest = int(np.argmin(val_losses))
    print(key, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{val_losses[lowest]:.4f}", lowest + 1)

[0.00625] 200
[0.00625] 200
[0.1, 0.05, 0.05, 0.025, 0.025, 0.0125, 0.0125, 0.00625]
[0.1] [0.1]
('SGD', 'fixed') 0.7655 0.4509 0.4509 200
('SGD', 'halved') 0.7821 0.4808 0.4808 200
('Adam', 'fixed') 0.7100 0.4585 0.4324 70
('Adam', 'halved') 0.7344 0.4444 0.4357 78

fig, ax = plt.subplots(figsize=(8, 5))
for (name, kind), (train_losses, val_losses, rates) in curves.items():
    ax.plot(range(1, 201), val_losses, label=f"{name}, {kind}")
for epoch in (50, 100, 150):
    ax.axvline(epoch, color="black", linestyle="--")
ax.set_yscale("log")
ax.set_xlabel("Epoch")
ax.set_ylabel("Validation mean squared error")
ax.set_title("A fixed rate of 0.1 against the same rate halved every 50 epochs")
ax.legend()
fig.savefig("schedule_curves.png", dpi=150, bbox_inches="tight")
print(len(ax.lines), [t.get_text() for t in ax.get_legend().get_texts()])
print(int(np.argmin(ax.lines[3].get_ydata())) + 1, f"{ax.lines[3].get_ydata()[-1]:.4f}")
plt.close(fig)
for kind in ("fixed", "halved"):
    print(kind, [f"{curves[('Adam', kind)][1][epoch - 1]:.4f}" for epoch in (70, 100, 150, 200)])
for kind in ("fixed", "halved"):
    print(kind, [f"{curves[('SGD', kind)][1][epoch - 1]:.4f}" for epoch in (50, 100, 150, 200)])

7 ['SGD, fixed', 'SGD, halved', 'Adam, fixed', 'Adam, halved']
78 0.4444
fixed ['0.4324', '0.4454', '0.4486', '0.4585']
halved ['0.4473', '0.4397', '0.4428', '0.4444']
fixed ['0.4995', '0.4683', '0.4562', '0.4509']
halved ['0.4995', '0.4912', '0.4840', '0.4808']

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the tuning advice of a team that has worked in deep learning for many years, or the full list of schedules the Assignment's StepLR is one of.

• GitHub - Google Research, "Deep Learning Tuning Playbook", "Choosing the optimizer" by Google Research. Start with the most popular optimizer for the problem and give attention to every setting it has, SGD with momentum and Adam among the ones named. The section "Choosing the initial configuration", two sections down, argues for a constant learning rate and a small model before any schedule, "Examining the training curves" reads the curves of the best trials for a validation error that starts increasing, and the sections on the batch size and the FAQ on which learning-rate decay to use as a default reach into batch sizes and schedules this lesson did not use.
• Official docs - PyTorch 2.14, torch.optim, "How to adjust learning rate". The rule that a scheduler's step() follows the optimizer's, a table of every scheduler with one line each, StepLR decaying every group's rate by gamma every step_size epochs beside MultiStepLR, ExponentialLR, CosineAnnealingLR and ReduceLROnPlateau, which lowers the rate when a validation measurement stops improving, and chaining, where schedulers called back to back act one after the other on the rate.

Summary

The optimizer turned out to be two decisions, a direction and a distance, and the learning rate turned out to be the distance in two different currencies. SGD's rate buys a share of the gradient, so its step is as large or as small as the loss makes it, and the raw price blew it up. Adam's rate buys a distance, so the gradient's size, a thousand or a million, does not reach its step, and the raw price left it standing still. Momentum sat between them, a buffer that lets a small rate move like a larger one and makes a large one overshoot. None of the three has a rate that can be computed on a network, and the sweep is what replaced the formula: five candidates a decade apart, a fresh seeded network for each, and a table read beside its curves.

The curves carried more than the table. A number at the end of 200 epochs could be a floor, a point on a crawl, a plateau above the valley or the far side of a turn, and each said something different about the rate behind it. The pick stayed one rule, the final validation loss at a fixed budget, which is why Adam's deepest point at epoch 70 was reported as a shape and not as a score. The test rows were scored once for the price, by the rule's pick, and that score was spent the moment it printed, which the Assignment's better validation loss then had to respect.

The clock closed the lesson because the sweep and the width both have a price in seconds, and a report of a model carries three numbers, not one. From here the module takes the rate as settled and turns to what the gap asks for, a loop that stops at the turn and a model held back from memorizing, which Section 4.4 builds on the curves this lesson drew.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np
import torch
from torch import optim
from torch.optim.lr_scheduler import StepLR

optimizer = optim.SGD(params, lr=0.1)
# the step follows a weighted sum of gradients
optimizer = optim.SGD(params, lr=0.01, momentum=0.9)
# a step set by lr, not by the gradient's size
optimizer = optim.Adam(params, lr=1e-3)
optimizer.param_groups[0]["lr"]  # where the rate lives
optimizer.state[p].keys()        # momentum_buffer, or step and two averages

rates = [0.0001, 0.001, 0.01, 0.1, 1.0]  # by factors of ten
results = {}
for name, optimizer_class in (("SGD", optim.SGD), ("Adam", optim.Adam)):
    for lr in rates:
        model = network()                # fresh seeded network per run
        optimizer = optimizer_class(model.parameters(), lr=lr)
        results[(name, lr)] = run(model, optimizer, 200)
finals = {key: v[-1] for key, (t, v) in results.items() if np.isfinite(v[-1])}
best = min(finals, key=finals.get)       # ('SGD', 0.1)
scheduler = StepLR(optimizer, step_size=50, gamma=0.5)
scheduler.step()                         # after optimizer.step(), once per epoch