Unfreezing, Careful Learning Rates and Error Analysis

Context

Unfreezing is the ground between a frozen backbone and a full fine-tune: some layers train and some do not, or every layer trains at a rate of its own, so the layers ImageNet shaped most generally move least. An optimizer can hold several parameter groups, each with its own learning rate, and one network can climb from a trained head to every weight in stages. Error analysis then reads the chosen network's mistakes, class by class and tile by tile, to say what kind of error is left.

Every fine-tune of a pretrained model on a new kind of image runs into these choices: a land-use classifier, a defect detector, a scan triage model. The learning rate is the setting that decides whether a fine-tune adapts the pretrained weights or overwrites them, and the error analysis is what a report shows before it quotes a single accuracy, because the next hour of work goes where the errors are.

Lesson 5.2.2 fine-tuned every weight at 1e-4 and reached 0.9567 on the validation tiles. At 1e-3, the rate its head trained at, the same network ends at 0.8974 with layer4's last weight moved by two thirds of itself, and the pretrained values it overwrote are stored nowhere in it any more. The careful schemes that protect the early layers are not automatically better, and a single test score hides which tiles fail and how sure the network was when it failed them.

Goal

• Fine-tune ResNet-18 at 1e-3 from a fresh head and from a trained one, measure how far each moved its weights, and say what the damage costs and what undoes it.
• Build one optimizer with three parameter groups at 1e-5, 1e-4 and 1e-3, and read from a run what protecting the early layers buys and costs.
• Unfreeze in two stages, layer4 and the head first, then every weight, and compare each stage with the strategies beside it.
• Train only the BatchNorms and the head, 14,730 parameters, and set the result beside the frozen head's.
• Choose among every strategy of the section on the validation tiles, score the test tiles once, and read the errors by class, by pair of classes and by tile.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the error-analysis habit argued from the start, or the exact form PyTorch takes several learning rates in.

• YouTube - "Carrying Out Error Analysis (C3W2L01)" by DeepLearningAI, Andrew Ng's course lecture. Looking at a sample of misclassified examples by hand, counting them by category in a table, and using the counts to decide which fix is worth the time.
• Official docs - PyTorch 2.14, torch.optim, Per-parameter options. Passing an optimizer a list of dictionaries instead of a list of parameters, each dictionary a parameter group with its own options, and the defaults that fill in what a group leaves out.

Table of Contents

• The Rate That Undoes Pretraining
• A Learning Rate per Layer Group
• Unfreezing in Stages
• Training the BatchNorms Alone
• Choosing, Testing Once and Reading the Errors

Theory

Every block runs in ~/dev/bootcamp, beside sat.py, Lesson 5.2.1's, image_loop.py, Lesson 4.5.2's, and the files Lesson 5.2.2 saved, head.pt and finetuned.pt. Nothing here downloads. The first subtitle saves ladder.py, the one module every strategy in this lesson imports, and each strategy runs as a script of its own, so each imports sat.py afresh and its train_loader shuffles the 2,700 training tiles in the same order as every other strategy's, Lesson 5.2.2's fine-tune included. The trained figures are this machine's on its default eight threads, and a seeded draw prints what it prints on torch 2.14.1 on a processor. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

The Rate That Undoes Pretraining

What. Fine-tuning every weight at 1e-3, the rate Lesson 5.2.2's head trained at, with Adam, Lesson 4.3.4's optimizer. Run twice, once from a fresh seeded head and once from the head Lesson 5.2.2 trained on the frozen features, a warm head, and measured the way Lesson 5.2.2 measured its fine-tune: validation accuracy per epoch, and how far three weight tensors moved from their pretrained values. Why. A pretrained weight is worth something because of where it sits, and a step large enough to move it far from there spends what the pretraining paid for. A fresh head's gradients are large and point nowhere useful at first, so a large rate lets them push every layer below by a large amount in the first steps. The warm head separates the two causes, the head's noise and the rate itself. When. Every time a pretrained network's rate is set, and above all when the rate that worked for a fresh layer is about to be reused for the whole network.

Save the section's shared pieces as ladder.py:

import torch
from torch import nn
from torchvision import models
from sat import weights, train_loader, val_loader
from image_loop import accuracy, train_epoch

pretrained = models.resnet18(weights=weights).state_dict()
WATCHED = ("conv1.weight", "layer1.0.conv1.weight", "layer4.1.conv2.weight")

def build(warm=False):
    model = models.resnet18(weights=weights)
    torch.manual_seed(42)
    model.fc = nn.Linear(512, 10)
    if warm:
        model.fc.load_state_dict(torch.load("head.pt", weights_only=True))
    return model

def moved(model):
    now = model.state_dict()
    return [round(((now[n] - pretrained[n]).norm() / pretrained[n].norm()).item(), 4) for n in WATCHED]

def run(model, optimizer, epochs=3):
    for epoch in range(1, epochs + 1):
        loss = train_epoch(model, optimizer, train_loader)
        print(epoch, f"{loss:.4f}", f"{accuracy(model, val_loader):.4f}")
    print(moved(model))

build is Lesson 5.2.2's network, a seeded nn.Linear(512, 10) head on the pretrained weights, and with warm=True it loads head.pt into that head. moved is Lesson 5.2.2's measure of change, the size of each watched tensor's change over its pretrained size, and run trains for a number of epochs and prints each epoch's training loss and validation accuracy, then the three changes. Every weight at 1e-3, from a fresh head:

from torch import optim
from ladder import build, run

model = build()
run(model, optim.Adam(model.parameters(), lr=1e-3))

1 0.7999 0.8244
2 0.4803 0.8444
3 0.4168 0.8974
[0.0554, 0.1784, 0.6583]

Lesson 5.2.2's same network at 1e-4, the same shuffles and the same seeded head, read 0.9263, 0.9541 and 0.9567 and moved its three tensors by 0.0077, 0.0196 and 0.1257. At ten times the rate it reads 0.8244, 0.8444 and 0.8974, ending below even the frozen head's 0.9226, and its watched tensors moved by 5.5 percent, 17.8 percent and 65.8 percent of themselves: layer4's last weight is two thirds a different tensor. The same rate from the head that already scores 0.9226:

from torch import optim
from ladder import build, run, accuracy, val_loader

model = build(warm=True)
print(f"{accuracy(model, val_loader):.4f}")
run(model, optim.Adam(model.parameters(), lr=1e-3))

0.9226
1 1.3304 0.8470
2 0.4742 0.8811
3 0.3666 0.9322
[0.0421, 0.1198, 0.5149]

The warm network starts at 0.9226, the frozen head's own score, since its backbone is untouched and its head is the trained one. One epoch at 1e-3 takes it to 0.8470, below where it started, with the first epoch's training loss at 1.3304, the highest of any epoch in this section: the step size threw the backbone off its pretrained values faster than the head could follow. It recovers to 0.9322 in three epochs and still moves layer4's last weight by half of itself. The warm head softened the first epoch's fall, 0.8470 against the fresh head's 0.8244, and did not prevent it, and the rate is what does the damage. A network that has been through it no longer holds the pretrained values anywhere, so the one sure way back to them is to reload the weights, build again, and restart at the lower rate. How early the damage happens, the warm network's first 20 steps at each rate:

from torch import optim
from ladder import build, moved, train_loader
from image_loop import loss_fn

for lr in (1e-4, 1e-3):
    model = build(warm=True)
    model.train()
    optimizer = optim.Adam(model.parameters(), lr=lr)
    for i, (x, y) in enumerate(train_loader):
        loss = loss_fn(model(x), y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        if i in (0, 4, 19):
            print(lr, i + 1, moved(model))
        if i == 19:
            break

0.0001 1 [0.0007, 0.0017, 0.0076]
0.0001 5 [0.0019, 0.0048, 0.0213]
0.0001 20 [0.0036, 0.0091, 0.0403]
0.001 1 [0.0072, 0.0175, 0.0758]
0.001 5 [0.0173, 0.0453, 0.1894]
0.001 20 [0.0285, 0.0845, 0.3636]

break ends the loop after the 20th batch of 32, 640 tiles of the 2,700. One step at 1e-3 moves every watched tensor ten times as far as one step at 1e-4, the ratio of the two rates, since Adam's first step is the rate in size whatever the gradient, Lesson 4.3.4's. After 20 steps at 1e-3, layer4's last weight has moved 36.4 percent of itself, more than two thirds of the 51.5 percent the warm network's three epochs reached, where 1e-4 has moved it 4 percent. Most of the damage is done in the first quarter of the first epoch, before a single validation score would show it.

DON'T:

• DON'T reuse a fresh head's learning rate for every weight. At 1e-3 the fine-tune ended at 0.8974, below the frozen head, with layer4's last weight moved by 65.8 percent, where 1e-4 reached 0.9567.
• DON'T expect a trained head to make a large rate safe. The warm network fell from 0.9226 to 0.8470 in its first epoch at 1e-3.
• DON'T keep training a network a large rate has damaged and call the result a fine-tune of ImageNet's weights. Its pretrained values are stored nowhere in it, and build is the one way they come back.
• DON'T wait for the first epoch's validation score to judge a rate. After 20 steps at 1e-3, layer4's last weight had already moved 36.4 percent, and the moved figures are worth printing that early.

A Learning Rate per Layer Group

What. optim.Adam takes a list of dictionaries in place of a list of parameters, each dictionary a parameter group with its own "params" and its own "lr", so one optimizer steps every group at its own rate. Here three groups: the stem, layer1 and layer2 at 1e-5, layer3 and layer4 at 1e-4, and the warm head at 1e-3, so the earliest layers move least. Why. One rate for the whole network is a compromise between the head, which starts far from where it must be, and the early layers, which start close. Groups remove the compromise: each part moves at a rate fitted to how far it should go. When. When a network's parts need different amounts of change, a new head above pretrained layers above all, and whenever a fine-tune should be able to say how much each part was allowed to move.

import torch
from torch import nn, optim
from ladder import build, run

model = build(warm=True)
early = [p for n, p in model.named_parameters() if n.startswith(("conv1", "bn1", "layer1", "layer2"))]
late = [p for n, p in model.named_parameters() if n.startswith(("layer3", "layer4"))]
groups = [{"params": early, "lr": 1e-5}, {"params": late, "lr": 1e-4}, {"params": model.fc.parameters(), "lr": 1e-3}]
optimizer = optim.Adam(groups)
print([(len(g["params"]), g["lr"]) for g in optimizer.param_groups])
run(model, optimizer)
torch.save(model.state_dict(), "groups.pt")

[(30, 1e-05), (30, 0.0001), (2, 0.001)]
1 0.4831 0.9215
2 0.1016 0.9293
3 0.0477 0.9333
[0.0008, 0.0021, 0.0791]

n.startswith((...)) takes a tuple and is true for any of its prefixes, so the first group holds the 30 parameter tensors of the stem, layer1 and layer2, the second the 30 of layer3 and layer4, and the third the head's weight and bias, 62 tensors in three groups. The first epoch holds the warm network near its start, 0.9215 against 0.9226, where 1e-3 everywhere had thrown it to 0.8470, and three epochs reach 0.9333. The watched tensors moved by 0.08 percent, 0.21 percent and 7.9 percent: the early layers were held almost exactly where ImageNet left them, a tenth of what Lesson 5.2.2's uniform 1e-4 moved them, and layer4 moved less than there too. And the score is lower than that uniform 1e-4's 0.9567. On these tiles and in three epochs the three groups ended 2.3 points under the uniform 1e-4, and with three settings changed at once, the early group's rate, the warm head and the head's rate of 1e-3, the run cannot say which of them the gap belongs to. Two details of the groups, on two small layers so nothing trains:

a, b = nn.Linear(4, 4), nn.Linear(4, 2)
two = optim.Adam([{"params": a.parameters()}, {"params": b.parameters(), "lr": 1e-3}], lr=1e-5)
print([g["lr"] for g in two.param_groups])
two.param_groups[0]["lr"] = 0.0
print([g["lr"] for g in two.param_groups])

[1e-05, 0.001]
[0.0, 0.001]

A group that names no rate takes the optimizer's own lr=, here 1e-5, and a group that names one keeps it. Each group is a plain dictionary inside param_groups, so a rate can be changed in place between epochs, to quiet a group or wake it, which is how a schedule over groups is written by hand. The figure below draws the two full runs side by side.

DON'T:

• DON'T assume a smaller rate on the early layers is the safer choice. The three-group run held conv1 within 0.08 percent of ImageNet's values and still ended at 0.9333, 2.3 points under the uniform 1e-4.
• DON'T pass model.parameters() and a list of groups to the same optimizer. Each tensor belongs to exactly one group, and the groups here split the 62 tensors 30, 30 and 2 with none left over.
• DON'T leave a group's rate to chance. A group without its own "lr" silently takes the optimizer's default, and printing [g["lr"] for g in optimizer.param_groups] before training says what each part will do.

Unfreezing in Stages

What. Unfreezing in stages trains the network top down in steps: first the warm head and layer4 alone, with every other flag off, for one epoch at 1e-4, then every weight at 1e-4 for two more. Each stage builds a new optimizer over the parameters that train in it, and the flags are switched on between the stages. Why. The stage-one network can only change its last stage and its head, so the features of layer3 and below stay exactly ImageNet's while the top adapts, and only then does the rest of the network move, from a top that already fits the task. It is the frozen head of Lesson 5.2.2 extended one stage at a time, with the gradient stopping at layer4 in the first stage, Lesson 5.2.2's Lab 1. When. When the early layers should be held fixed until the top has settled, when a budget allows only the top to train first, and when a fine-tune must stop at any stage and still leave a working network behind.

from torch import optim
from ladder import build, run

model = build(warm=True)
for name, p in model.named_parameters():
    p.requires_grad_(name.startswith(("layer4", "fc")))
print(sum(p.numel() for p in model.parameters() if p.requires_grad))
run(model, optim.Adam([p for p in model.parameters() if p.requires_grad], lr=1e-4), epochs=1)
for p in model.parameters():
    p.requires_grad_(True)
run(model, optim.Adam(model.parameters(), lr=1e-4), epochs=2)

8398858
1 0.4756 0.9119
[0.0, 0.0, 0.0655]
1 0.2109 0.9233
2 0.1035 0.9296
[0.006, 0.0175, 0.1029]

Stage one trains 8,398,858 parameters, Lesson 5.2.2's Lab 1 count, layer4's and the warm head's. The first run is that stage, one epoch: conv1 and layer1 moved by exactly 0.0, since no gradient reached them, layer4 by 6.55 percent, and the network dipped to 0.9119, under the 0.9226 it started from. The second run, every weight for two epochs at 1e-4, its epochs counted from 1 again, reached 0.9233 and 0.9296, and by the end the early layers had moved by 0.6 and 1.75 percent. Three epochs in all, and the staged network ended under both the uniform 1e-4's 0.9567 and the three groups' 0.9333.

DON'T:

• DON'T switch the flags on for the second stage and keep the first stage's optimizer. It was built over layer4 and the head, so the stem and the first three stages would gain gradients and never be stepped.
• DON'T read the staged network's dip as the stage failing. Unfreezing layer4 moved it 6.55 percent in one epoch and cost a point on the validation tiles, and the second stage climbed past the starting 0.9226 within one more epoch.

Training the BatchNorms Alone

What. Every convolution frozen, and only the 20 BatchNorms' learned weights and biases and the warm head training, 14,730 parameters, 0.13 percent of the network, at 1e-3. Each BatchNorm's weight and bias scale and shift one channel, Lesson 5.1.2's two learned numbers per channel, so this rescales every feature map the pretrained filters write without touching a filter. Why. A method that trains a small fraction of the weights is cheap to store, one small set of numbers per task beside one shared backbone, and its claim is that the pretrained filters are fine and only their scale needs fitting. This run measures that claim against the frozen head, which trains a third of these parameters, and the full fine-tune, which trains all of them. When. When many tasks must share one backbone and each task may store only a little, or when compute rules out a full fine-tune and a frozen head falls short, and in both cases only after a run like this one on the task's own validation tiles.

from torch import nn, optim
from ladder import build, run

model = build(warm=True)
for name, p in model.named_parameters():
    p.requires_grad_("bn" in name or "downsample.1" in name or name.startswith("fc"))
trainable = [p for p in model.parameters() if p.requires_grad]
print(sum(p.numel() for p in trainable))
run(model, optim.Adam(trainable, lr=1e-3))

14730
1 0.5658 0.8930
2 0.2405 0.9067
3 0.1593 0.9141
[0.0, 0.0, 0.0]

downsample.1 is the BatchNorm inside each of the three shortcuts, Lesson 5.2.1's, named by position rather than by bn. The 9,600 BatchNorm parameters and the head's 5,130 make the 14,730. Every watched convolution moved by exactly 0.0, and the network went from 0.9226 to 0.8930 after one epoch and recovered to 0.9141, under the frozen head it started from. Training the BatchNorms in training mode also moves their running statistics, Lesson 5.2.2's third subtitle, so this network's features change two ways at once. On these tiles a small fraction of the weights bought nothing over the frozen head and fell 4.3 points short of the full fine-tune. Why the filter names downsample.1 as well as bn:

norms = [n for n, m in model.named_modules() if isinstance(m, nn.BatchNorm2d)]
print(len(norms), sum("bn" in n for n in norms), [n for n in norms if "bn" not in n])

20 17 ['layer2.0.downsample.1', 'layer3.0.downsample.1', 'layer4.0.downsample.1']

isinstance(m, nn.BatchNorm2d) finds every BatchNorm by its type, 20, and 17 of them carry bn in their names. The three in the shortcuts are position 1 of each downsample, so a filter by name alone would have left them frozen, and a filter by type is the one that cannot miss.

DON'T:

• DON'T read a method that trains a small fraction of the weights as a free full fine-tune. Here 14,730 parameters ended at 0.9141, under the frozen head's 0.9226 and the full fine-tune's 0.9567.
• DON'T select a network's BatchNorms by the substring bn alone. Selected by type there are 20, and 3 of them, the shortcuts' downsample.1, carry no bn in their names.

Choosing, Testing Once and Reading the Errors

What. The validation tiles chose among every strategy of the section, each scored at the last epoch it trained, the best of the epochs it printed:

The everything-at-1e-4 network, finetuned.pt, is the choice, and the 2,700 test tiles score it once. Its errors are then read three ways: accuracy per class, the confusion matrix's largest cells, Lesson 3.4.2's, and the misclassified tiles themselves, drawn. Why. The validation tiles made the choice, so only the test tiles can say what the choice does on tiles no decision touched, Lesson 3.1.2's rule. One accuracy says how many tiles fail and nothing about which, and the next hour of work, more labels for one class, a transform, a cleaner label, depends on which. When. Once, at the end of the choosing, and the error analysis every time before a score is reported.

import matplotlib.pyplot as plt
import torch
from torch import nn
from sklearn.metrics import confusion_matrix
from torchvision import datasets, models
from sat import weights, full, test_set, test_loader

model = models.resnet18(weights=weights)
model.fc = nn.Linear(512, 10)
print(model.load_state_dict(torch.load("finetuned.pt", weights_only=True)))
model.eval()
with torch.no_grad():
    logits = torch.cat([model(x) for x, y in test_loader])
labels = torch.cat([y for x, y in test_loader])
pred = logits.argmax(dim=1)
print((pred == labels).sum().item(), f"{(pred == labels).float().mean().item():.4f}")
names = full.classes
for k in range(10):
    mine = labels == k
    print(names[k], mine.sum().item(), f"{(pred[mine] == k).float().mean().item():.3f}")

<All keys matched successfully>
2577 0.9544
AnnualCrop 301 0.927
Forest 318 1.000
HerbaceousVegetation 281 0.975
Highway 243 0.914
Industrial 250 0.996
Pasture 182 0.907
PermanentCrop 246 0.886
Residential 319 0.981
River 248 0.915
SeaLake 312 1.000

The chosen network calls 2,577 of the 2,700 test tiles right, 0.9544, a little under its validation 0.9567, as every chosen model of Module 5 has been under its own. Two classes are perfect, forest and sea or lake, the two near-uniform dark squares of Lesson 5.2.1's grid, and the four weakest are permanent crops at 0.886, pasture at 0.907, highway at 0.914 and river at 0.915. Where the 123 mistakes go:

matrix = confusion_matrix(labels, pred)
print(matrix)
off = [(matrix[i, j], names[i], names[j]) for i in range(10) for j in range(10) if i != j]
for count, truth, called in sorted(off, reverse=True)[:5]:
    print(count, truth, "called", called)
print(sum(1 for count, truth, called in off if count > 0), (pred != labels).sum().item())

[[279   0   1   2   0   2  12   0   1   4]
 [  0 318   0   0   0   0   0   0   0   0]
 [  0   3 274   0   0   0   4   0   0   0]
 [  2   2   0 222   2   0   2   4   9   0]
 [  1   0   0   0 249   0   0   0   0   0]
 [  7   3   6   0   0 165   0   1   0   0]
 [  9   0  19   0   0   0 218   0   0   0]
 [  0   1   0   1   3   0   1 313   0   0]
 [  1   4   5   7   0   4   0   0 227   0]
 [  0   0   0   0   0   0   0   0   0 312]]
19 PermanentCrop called HerbaceousVegetation
12 AnnualCrop called PermanentCrop
9 PermanentCrop called AnnualCrop
9 Highway called River
7 River called Highway
30 123

Rows are the true classes and columns the calls, both in label order, Lesson 3.4.2's layout on ten classes. The forest row and the sea and lake row are bare off the diagonal, and the annual crop and highway rows are the most scattered, their 22 and 21 errors each spread over six classes. The 123 errors spread over 30 of the 90 cells off the diagonal, and the five largest are two families: the three crop and vegetation classes, which trade 40 tiles among them in the top three cells alone, and highway and river, which trade 16 in both directions. The tiles themselves:

raw = datasets.EuroSAT("data")
wrong = (pred != labels).nonzero().flatten()[:8]
fig, axes = plt.subplots(2, 4, figsize=(10, 5))
for ax, i in zip(axes.flat, wrong.tolist()):
    img, label = raw[test_set.indices[i]]
    ax.imshow(img)
    ax.set_title(f"{names[label]}\ncalled {names[pred[i]]}", fontsize=8)
    ax.axis("off")
fig.savefig("errors.png", dpi=150, bbox_inches="tight")
print(len(fig.axes))
plt.close(fig)
confidence = logits.softmax(dim=1).max(dim=1).values
print(f"{confidence[pred == labels].mean().item():.3f}", f"{confidence[pred != labels].mean().item():.3f}")

8
0.965 0.702

nonzero().flatten() lists the positions of the misclassified test tiles, and raw[test_set.indices[i]] reads each one's original 64 by 64 picture, without the preprocessing. In errors.png the first eight errors are mostly tiles that hold two land uses at once or sit on a border between classes: a narrow road among fields called a river, a river's straight bank along a dike called a highway, permanent crops on dry, patchy ground called herbaceous vegetation, a dense town with large pale roofs called industrial. The network was 0.965 sure on average of what it got right and 0.702 of what it got wrong, so its own confidence separates the two, on average. The eight drawn are border and mixed tiles, and the largest cells of the matrix sit between classes that share such borders, which points the next hour at those tiles, their labels and more of them.

DON'T:

• DON'T report 0.9544 without the classes behind it. The same network called every forest tile right and 0.886 of the permanent crops.
• DON'T read the test errors and then change the model to fix them. Every decision is made on the validation tiles' errors, and a fix chosen from the test tiles turns them into validation tiles, Lesson 3.1.2's rule.
• DON'T read the matrix's bare rows as finished classes. Forest and sea or lake were perfect on these 2,700 tiles, and the 0.961 precision of forest, Lab 2's, says other classes still land on it.

Lab Exercises

Lab Exercise 1: Four Groups, the Stem Frozen

In four_groups.py, build the warm network from ladder.py, turn off the flag on the stem's and layer1's parameters, and build one optimizer with four groups: layer2 at 1e-5, layer3 at 3e-5, layer4 at 1e-4 and the head at 1e-3. Print each group's number of tensors and its rate, train for one epoch with run, and say in a comment how the result and the watched tensors compare with the three-group run's first epoch.

Lab Exercise 2: Recall, Precision and Confident Mistakes

In precision.py, load finetuned.pt, compute its predictions on the test tiles, and from the confusion matrix print, for each class, its recall (the share of its tiles called right, the diagonal over the row) and its precision (the share of the tiles called that class that are that class, the diagonal over the column), Lesson 3.4.2's two readings. Then print how many test tiles the network gets wrong, how many of those it called with a probability above 0.9, and how many tiles it got right with a probability under 0.5. Say in a comment which class's precision is the lowest and what that means for a map built from the network's calls.

from torch import optim
from ladder import build, run

model = build(warm=True)
for name, p in model.named_parameters():
    p.requires_grad_(not name.startswith(("conv1", "bn1", "layer1")))
rates = [("layer2", 1e-5), ("layer3", 3e-5), ("layer4", 1e-4), ("fc", 1e-3)]
groups = [{"params": [p for n, p in model.named_parameters() if n.startswith(prefix)], "lr": lr} for prefix, lr in rates]
optimizer = optim.Adam(groups)
print([(len(g["params"]), g["lr"]) for g in optimizer.param_groups])
run(model, optimizer, epochs=1)
# 0.9211 after one epoch against the three groups' 0.9215, with conv1
# and layer1 at exactly 0.0, frozen, and layer4 moved 6.39 percent.

[(15, 1e-05), (15, 3e-05), (15, 0.0001), (2, 0.001)]
1 0.4753 0.9211
[0.0, 0.0, 0.0639]

import torch
from torch import nn
from sklearn.metrics import confusion_matrix
from torchvision import models
from sat import weights, full, test_loader

model = models.resnet18(weights=weights)
model.fc = nn.Linear(512, 10)
model.load_state_dict(torch.load("finetuned.pt", weights_only=True))
model.eval()
with torch.no_grad():
    logits = torch.cat([model(x) for x, y in test_loader])
labels = torch.cat([y for x, y in test_loader])
pred = logits.argmax(dim=1)
matrix = confusion_matrix(labels, pred)
names = full.classes
for k in range(10):
    print(names[k], f"{matrix[k, k] / matrix[k].sum():.3f}", f"{matrix[k, k] / matrix[:, k].sum():.3f}")
confidence = logits.softmax(dim=1).max(dim=1).values
wrong = pred != labels
print(wrong.sum().item(), (confidence[wrong] > 0.9).sum().item(), (confidence[~wrong] < 0.5).sum().item())
# HerbaceousVegetation has the lowest precision, 0.898: about one tile
# in ten a map paints as herbaceous vegetation is something else, most
# often the permanent crops it was confused with.

AnnualCrop 0.927 0.933
Forest 1.000 0.961
HerbaceousVegetation 0.975 0.898
Highway 0.914 0.957
Industrial 0.996 0.980
Pasture 0.907 0.965
PermanentCrop 0.886 0.920
Residential 0.981 0.984
River 0.915 0.958
SeaLake 1.000 0.987
123 25 14

Theory Assessment

Q1: The Same Rate, Two Heads

At 1e-3, the fresh-head network ended at 0.8974 and the warm-head network at 0.9322, while the warm one had fallen from 0.9226 to 0.8470 in its first epoch. A teammate concludes that the fresh head was the problem and that warm heads make any rate safe. Say what each run shows about the head and about the rate, what both did to layer4's last weight, and what to do with a network that has already been through the first epoch at 1e-3.

Hint: Compare the first epoch of the warm run with its starting score, and the two runs' moved figures with Lesson 5.2.2's at 1e-4.

Q2: Held Still, and Lower

The three-group optimizer held conv1 within 0.08 percent of ImageNet's values and reached 0.9333, where one rate of 1e-4 for everything moved conv1 by 0.77 percent and reached 0.9567. Say how an optimizer with parameter groups is built and what each group's rate controls, what the run shows the three groups bought and what it cannot attribute to the early layers' rate, and whether it shows that early layers should always move.

Hint: Separate what the run measured, three epochs on 2,700 tiles at these rates, from a rule about every dataset.

Q3: 0.13 Percent of the Weights

The BatchNorms and the head, 14,730 parameters, ended at 0.9141, under the frozen head's 0.9226. Say what those parameters can change about the network's features and what they cannot, the second way this network's features changed that the frozen head's never did, and what the run says about the claim that a small fraction of the weights matches a full fine-tune.

Hint: Recall the two kinds of numbers a BatchNorm holds, the learned ones and the running ones, and which of them training mode moves.

Q4: Fixing What the Test Set Showed

A teammate reads the test errors, sees permanent crops at 0.886, adds a transform that targets crop textures, retrains, scores the test tiles again at 0.9611, and reports a gain of 0.67 points. Say what the test tiles were for, what reading their errors and acting on them turned them into, why 0.9611 is not an estimate of the new network on unseen tiles, and how the same idea should have been tested.

Hint: Lesson 3.1.2's rule concerns any decision made by looking at a set's results, and a fix chosen from the errors is such a decision.

Assignment

Assignment: Flips a Satellite Never Fixes

A satellite has no up: a field photographed from orbit is the same field mirrored or turned upside down, unlike Lesson 5.1.4's shoes, which always faced one way. Write flips.py, which fine-tunes with random flips in both directions and tries a second use of the same idea, flipping the tiles at prediction time.

Two things this lesson did not use. transforms.RandomVerticalFlip() flips an image top to bottom with a probability of 0.5, the vertical twin of Lesson 5.1.4's RandomHorizontalFlip. And test-time augmentation (scoring an image as the average of the network's softmax over several transformed copies of it) predicts each tile from its four flips, the tile, mirrored, upside down, and both, with x.flip(3) and x.flip(2).

Build a training set whose transform flips both ways and then applies weights.transforms(crop_size=160, resize_size=160), over train_set.indices, with a loader of 32 shuffled by its own generator seeded 42, and print the transform. Fine-tune build() from ladder.py on it for three epochs at 1e-4, printing each epoch's training loss and validation accuracy, and keep the best epoch. Then load finetuned.pt into a second build(), and print, for both networks, the validation accuracy with a single view and with the four flipped views averaged. Say in a comment which of the four combinations the validation tiles choose, and why none of this touches the test tiles.

Hints:

• transforms.Compose takes the weights' own ImageClassification transform as a step, so the flips go before it and run on the PIL image.
• Average the four softmax outputs, not the four argmaxes: sum(model(v).softmax(dim=1) for v in views) / 4, then take the argmax of the average.

import copy
import torch
from torch import optim
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
from sat import weights, train_set, val_loader
from image_loop import accuracy, train_epoch
from ladder import build

flips = transforms.Compose([transforms.RandomHorizontalFlip(), transforms.RandomVerticalFlip(), weights.transforms(crop_size=160, resize_size=160)])
print(flips)
flipping = datasets.EuroSAT("data", transform=flips)
flip_loader = DataLoader(Subset(flipping, train_set.indices), batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(42))
model = build()
optimizer = optim.Adam(model.parameters(), lr=1e-4)
best_acc, best_state = 0.0, None
for epoch in range(1, 4):
    loss = train_epoch(model, optimizer, flip_loader)
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{loss:.4f}", f"{val_acc:.4f}")
    if val_acc > best_acc:
        best_acc, best_state = val_acc, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)

Compose(
    RandomHorizontalFlip(p=0.5)
    RandomVerticalFlip(p=0.5)
    ImageClassification(
    crop_size=[160]
    resize_size=[160]
    mean=[0.485, 0.456, 0.406]
    std=[0.229, 0.224, 0.225]
    interpolation=InterpolationMode.BILINEAR
)
)
1 0.5788 0.9289
2 0.2036 0.9437
3 0.1357 0.9500

plain = build()
plain.load_state_dict(torch.load("finetuned.pt", weights_only=True))

def four_views(m):
    m.eval()
    correct = total = 0
    with torch.no_grad():
        for x, y in val_loader:
            views = [x, x.flip(3), x.flip(2), x.flip(2).flip(3)]
            probs = sum(m(v).softmax(dim=1) for v in views) / 4
            correct += (probs.argmax(dim=1) == y).sum().item()
            total += len(y)
    return correct / total

for name, m in (("plain", plain), ("flips", model)):
    print(name, f"{accuracy(m, val_loader):.4f}", f"{four_views(m):.4f}")
# The validation tiles choose the network trained without flips, scored
# on four flipped views: 0.9630. Every comparison ran on the validation
# tiles, so the test tiles, scored once in the Theory, stay unspent.

plain 0.9567 0.9630
flips 0.9500 0.9593

Additional Resources

Optional, and nothing later depends on it. Worth opening for the same recipe on a different dataset, with the head's rate set ten times the backbone's.

• Book chapter - Dive into Deep Learning, "Fine-Tuning" by Zhang, Lipton, Li and Smola, under CC BY-SA 4.0. A pretrained ResNet-18 fine-tuned to tell hot dogs from other food, the new output layer trained at ten times the base learning rate, against the same network trained from random weights.

Summary

The learning rate was the one setting in this section that decided whether pretraining survived the fine-tune. At 1e-3 the network moved its last stage's last weight by half to two thirds of itself, ended under the frozen head it was meant to beat from a fresh head and a point above it from a trained one, and the one sure way back was to reload what had been overwritten. At 1e-4 the same network kept its early layers within a few percent of ImageNet and won. The damage was early and fast: that weight had moved by a third of itself within 20 steps, which is why the moved figure is worth watching from the first batch and the validation score alone arrives too late to stop it.

The careful schemes were measured rather than assumed, and on these tiles none of them won. Per-group rates held the early layers almost exactly still and ended two points under the uniform rate, staging the unfreezing dipped and recovered to below the uniform rate, and training only the BatchNorms landed under the frozen head. Each is a real tool with a real use, and each answered here the way the data, the budget and the distance from ImageNet made it answer, which is why the validation tiles, not a habit, chose.

And the errors that remain have a shape. They sit at the borders between land uses that look alike from orbit, crops against vegetation, a road against a river, and on tiles that hold two classes at once, with the network less sure of them than of what it gets right. That shape says where the next hour goes, labels and tiles near those borders, and the rule that kept every comparison on the validation tiles is what keeps the 0.9544 an honest number to report.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

early = [p for n, p in model.named_parameters() if n.startswith(("conv1", "bn1", "layer1", "layer2"))]
late = [p for n, p in model.named_parameters() if n.startswith(("layer3", "layer4"))]
optimizer = optim.Adam([
    {"params": early, "lr": 1e-5},
    {"params": late, "lr": 1e-4},
    {"params": model.fc.parameters(), "lr": 1e-3},
])

views = [x, x.flip(3), x.flip(2), x.flip(2).flip(3)]
probs = sum(model(v).softmax(dim=1) for v in views) / 4
pred = probs.argmax(dim=1)  # test-time augmentation