Freeze and Fine-Tune on EuroSAT

Context

Freezing keeps a pretrained network's weights fixed and trains only a new last layer on top of what the network already computes. Fine-tuning trains every weight, the new layer and the eleven million below it, starting from the pretrained values instead of a seeded draw. The two are the ends of one dial, and every adaptation of a pretrained model sits somewhere on it.

A team with a few thousand labeled satellite tiles, product photos or scans decides where on that dial to sit before anything else. Freezing trains in seconds on features computed once and cannot damage what the network learned. Fine-tuning costs a full pass forward and backward through the whole network on every batch, and it can adapt every layer to the new images.

Lesson 5.2.1 loaded ResNet-18, found its last layer answering sharks and parachutes for fields and lakes, and found the layers under it classifying land use by nearest mean with nothing trained. Freezing looks like the safe end of the dial and fails three ways without an error: a frozen network left in training mode rewrites its own BatchNorm statistics, tiles served without the normalization the head was trained with score less than half as well, and features cached once go stale the moment the inputs change.

Goal

• Freeze ResNet-18 with requires_grad_(False), replace its head, and confirm from the gradients and the optimizer that only 5,130 parameters train.
• Compute the backbone's 512 features once in evaluation mode, train the head on them in under a second, and prove it equals the frozen network run end to end.
• Show that a frozen backbone left in training mode moves its BatchNorm statistics and takes the head's accuracy from 0.9226 to 0.5563.
• Measure the normalization contract twice: a head trained without it, and a head served without it.
• Fine-tune every weight at 1e-4, compare its validation accuracy and its seconds with the frozen head's, and save it for Lesson 5.2.3.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the two ends of the dial argued first, or PyTorch's own walk-through of the same two strategies on another dataset.

• YouTube - "C4W2L09 Transfer Learning" by DeepLearningAI, Andrew Ng's course lecture. Freezing a pretrained network and training a new softmax layer, precomputing the frozen layers' activations and saving them to disk, and training more of the layers as the labeled set grows.
• Official tutorial - PyTorch, "Transfer Learning for Computer Vision Tutorial". A pretrained ResNet-18 adapted to classify ants and bees two ways, every weight fine-tuned and the network frozen with a new final layer, with the training loop and the predictions drawn.

Table of Contents

• Freezing the Backbone
• Features Computed Once
• The Frozen Backbone in Training Mode
• The Normalization Contract
• Fine-Tuning Everything

Theory

Every block runs in ~/dev/bootcamp, beside sat.py, Lesson 5.2.1's, and image_loop.py, Lesson 4.5.2's, whose loss_fn, accuracy and train_epoch take any model. Nothing here downloads, since Lesson 5.2.1 cached the weights and EuroSAT. The second subtitle saves features.pt and head.pt, and the last saves finetuned.pt, and Lesson 5.2.3 loads head.pt and finetuned.pt. The trained figures are this machine's on its default eight threads, as Section 5.1's were, a seeded draw prints what it prints on torch 2.14.1 on a processor, and seconds are compared only inside the script that timed them. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Freezing the Backbone

What. Freezing calls requires_grad_(False) on every pretrained parameter, so backward() computes no gradient for it, then replaces fc with a new nn.Linear(512, 10), whose parameters are created with requires_grad on, Lesson 4.1.4's flag. The optimizer is handed only the parameters that still require a gradient. Why. The backbone's eleven million weights already compute features that separate land uses, Lesson 5.2.1's nearest-mean probe, and 2,700 tiles are few enough that retraining them risks undoing that. Freezing trains 5,130 parameters on those features and leaves the rest exactly as ImageNet left them. When. When labels are few, when compute is short, and as the first rung of every adaptation, because a frozen head is the baseline every other strategy is measured against.

import torch
from torch import nn, optim
from torchvision import models
from sat import weights, train_loader
from image_loop import loss_fn

model = models.resnet18(weights=weights)
for p in model.parameters():
    p.requires_grad_(False)
torch.manual_seed(42)
model.fc = nn.Linear(512, 10)
trainable = [p for p in model.parameters() if p.requires_grad]
print(sum(p.numel() for p in trainable), sum(p.numel() for p in model.parameters()))
optimizer = optim.Adam(trainable, lr=0.001)
print(sum(p.numel() for group in optimizer.param_groups for p in group["params"]))

5130 11181642
5130

The loop sets the flag on every parameter the pretrained network holds, and the new fc, created after it, keeps its own flag on, so 5,130 of the 11,181,642 parameters train. optimizer.param_groups is the list of parameter groups an optimizer keeps, here one, and it holds the same 5,130. By name:

print([name for name, p in model.named_parameters() if p.requires_grad])
print(sum(1 for p in model.parameters()), sum(1 for p in model.parameters() if not p.requires_grad))

['fc.weight', 'fc.bias']
62 60

The two trainable tensors are the head's weight and bias, and the other 60 of the network's 62 parameter tensors are frozen. A list of names is the check worth printing before any training run that freezes part of a network, because it says what will move in words a mistake cannot hide behind. One batch through the frozen network, in evaluation mode:

model.eval()
x, y = next(iter(train_loader))
loss = loss_fn(model(x), y)
loss.backward()
print(model.conv1.weight.grad, model.layer4[1].conv2.weight.grad)
print(model.fc.weight.grad.shape, model.bn1.num_batches_tracked)

None None
torch.Size([10, 512]) tensor(0)

backward() filled the head's gradient, (10, 512) like its weight, and left every frozen parameter's .grad at None, not zero: no gradient was computed for them at all, so the backward pass stopped at the head. The first BatchNorm's batch count still reads 0, the count a newly built network starts at, because evaluation mode read the running statistics and changed none of them. One step of the optimizer:

conv_before, fc_before = model.conv1.weight.clone(), model.fc.weight.clone()
optimizer.step()
print(torch.equal(conv_before, model.conv1.weight), torch.equal(fc_before, model.fc.weight))
print(round((model.fc.weight - fc_before).abs().max().item(), 4))

True False
0.001

clone() keeps a copy of each weight as it stood before the step, since the step changes the tensors in place. The first convolution is bit for bit what it was and the head is not: Adam's first step moved each head weight by up to 0.001, the learning rate, the size Lesson 4.3.4's Adam takes on its first step whatever the gradient's size. The order of the two steps matters:

late = models.resnet18(weights=weights)
torch.manual_seed(42)
late.fc = nn.Linear(512, 10)
late.requires_grad_(False)
left = [p for p in late.parameters() if p.requires_grad]
print(len(left))
try:
    optim.Adam(left, lr=0.001)
except ValueError as err:
    print(err)

0
optimizer got an empty parameter list

late.requires_grad_(False) is the module's own method, the loop over every parameter in one call, and here it runs after the new head is attached, so it freezes the head as well. No parameter is left to train, and Adam refuses the empty list when it is built, which is the lucky case: an optimizer built over model.parameters() instead would have been accepted and would never have moved anything.

DON'T:

• DON'T hand the optimizer model.parameters() after freezing and trust the flag to sort it out. Adam skips a parameter whose .grad is None, so nothing frozen moves here, but its count of what it holds is 11,181,642, and every report built on it is wrong.
• DON'T check freezing by the loss falling. The head alone makes the loss fall, and only the gradients, None on every frozen parameter, show what trains.
• DON'T freeze after attaching the new head. requires_grad_(False) on the whole model froze the head too, and Adam refused the empty list it was left with.

Features Computed Once

What. A frozen backbone in evaluation mode is a fixed function: the same tile gives the same 512 values on every pass. So the 512 values avgpool writes for every training and validation tile are computed once, with fc replaced by nn.Identity(), kept as two tensors, and the head trains on them directly, twenty epochs in under a second instead of twenty full passes through eleven million parameters. Why. Training the head end to end would run the whole frozen network forward on every tile at every epoch, to recompute numbers that never change. Computing them once moves all of that cost to one pass, and the head's training becomes Lesson 4.3.3's loop on a (2700, 512) tensor. When. Whenever the backbone is frozen and the inputs do not change between epochs, which rules it out the moment a random transform, Lesson 5.1.4's, enters the training images.

import time
import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import models
from sat import weights, train_set, train_loader, val_loader
from image_loop import loss_fn, accuracy

backbone = models.resnet18(weights=weights)
backbone.fc = nn.Identity()
backbone.eval()

def features(loader):
    out, labels = [], []
    with torch.no_grad():
        for x, y in loader:
            out.append(backbone(x))
            labels.append(y)
    return torch.cat(out), torch.cat(labels)

start = time.perf_counter()
F, Y = features(DataLoader(train_set, batch_size=100))
G, Z = features(val_loader)
print(F.shape, G.shape, f"{time.perf_counter() - start:.1f}")
torch.save({"F": F, "Y": Y, "G": G, "Z": Z}, "features.pt")

torch.Size([2700, 512]) torch.Size([2700, 512]) 40.7

DataLoader(train_set, batch_size=100) reads the training tiles in a fixed order, so F[i] and Y[i] belong to the same tile, and the 5,400 tiles go through the network once. features.pt keeps the four tensors for the last subtitle and the Assignment. The head, trained on them:

torch.manual_seed(42)
head = nn.Linear(512, 10)
optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
order = torch.Generator().manual_seed(42)
start = time.perf_counter()
for epoch in range(1, 21):
    perm = torch.randperm(len(F), generator=order)
    for i in range(0, len(F), 32):
        idx = perm[i:i + 32]
        loss = loss_fn(head(F[idx]), Y[idx])
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    if epoch in (1, 5, 10, 20):
        with torch.no_grad():
            print(epoch, f"{(head(G).argmax(dim=1) == Z).float().mean().item():.4f}")
print(f"{time.perf_counter() - start:.1f}")
torch.save(head.state_dict(), "head.pt")

1 0.8663
5 0.9070
10 0.9156
20 0.9226
0.5

torch.randperm draws a new order of the 2,700 rows every epoch from its own seeded generator, and the slices of 32 are Lesson 4.3.3's mini-batches taken by hand, since the rows are a tensor already in memory. Twenty epochs took half a second in the script whose feature pass took 40.7, and the validation accuracy rises from 0.8663 after one to 0.9226 after twenty, against the 0.8348 the nearest-mean probe reached with nothing trained. The same head inside the frozen network, run end to end:

backbone.fc = head
print(f"{accuracy(backbone, val_loader):.4f}")
x, y = next(iter(val_loader))
backbone.eval()
with torch.no_grad():
    print(torch.allclose(backbone(x), head(G[:100]), atol=1e-5))

0.9226
True

Put back as fc, the trained head makes the frozen network score the same 0.9226 through accuracy, Lesson 4.5.2's, and its logits for the first validation batch match the head's on the cached features within 1e-5. The cached route is the frozen network, computed once.

DON'T:

• DON'T cache features from a backbone in training mode. The cached tensors are only the frozen network's output if every BatchNorm reads its running statistics, which the next subtitle measures.
• DON'T reuse cached features once the training images get a random transform. Each epoch would then see new pixels, and features computed once are features of pixels no longer used.

The Frozen Backbone in Training Mode

What. requires_grad_(False) stops a parameter from receiving a gradient and does nothing to a buffer. A BatchNorm's running mean and running variance are buffers, Lesson 5.1.2's, updated on every forward pass in training mode whether or not any parameter requires a gradient. A frozen backbone left in training mode for one pass over the training tiles keeps every weight and replaces its normalization. Why. The head was trained on the features of the backbone ImageNet left, running statistics included. A backbone whose statistics moved computes other features from the same tiles, and the head reads them with weights fitted to the old ones, with no error and no change in a single parameter. When. Every time a frozen network is trained end to end through a loop that calls model.train(), which accuracy in image_loop.py does on its way out.

drifting = models.resnet18(weights=weights)
drifting.fc = nn.Identity()
for p in drifting.parameters():
    p.requires_grad_(False)
before = drifting.bn1.running_mean.clone()
drifting.train()
with torch.no_grad():
    for x, y in train_loader:
        drifting(x)
drifting.eval()
print(drifting.bn1.num_batches_tracked, round((drifting.bn1.running_mean - before).abs().max().item(), 4))
G2 = torch.cat([drifting(x).detach() for x, y in val_loader])
print(round((G2 - G).abs().max().item(), 4))
with torch.no_grad():
    print(f"{(head(G2).argmax(dim=1) == Z).float().mean().item():.4f}")

tensor(85) 1.7187
9.9504
0.5563

drifting is a second frozen backbone, every parameter's flag off, passed once over the 85 training batches in training mode under no_grad, so not one weight could move. Its first BatchNorm counted all 85 batches, and its running mean moved by up to 1.7187. Back in evaluation mode, the validation tiles' 512 features differ from the cached ones by up to 9.9504, and the head trained on the cached features scores 0.5563 on them, where it scored 0.9226. Freezing the weights did not freeze the network. The same mistake made the way a training loop makes it, a frozen network with a new head trained end to end for two epochs, once with the network in training mode and once in evaluation mode:

import torch
from torch import nn, optim
from torchvision import models
from sat import weights, train_loader, val_loader
from image_loop import loss_fn, accuracy

def frozen_model():
    m = models.resnet18(weights=weights)
    for p in m.parameters():
        p.requires_grad_(False)
    torch.manual_seed(42)
    m.fc = nn.Linear(512, 10)
    return m

for mode in ("train", "eval"):
    m = frozen_model()
    kept = m.layer4[1].conv2.weight.clone()
    optimizer = optim.Adam(m.fc.parameters(), lr=0.001)
    for epoch in range(1, 3):
        m.train() if mode == "train" else m.eval()
        for x, y in train_loader:
            loss = loss_fn(m(x), y)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        same = torch.equal(kept, m.layer4[1].conv2.weight)
        print(mode, epoch, f"{accuracy(m, val_loader):.4f}", m.bn1.num_batches_tracked.item(), same)

train 1 0.8289 85 True
train 2 0.8633 170 True
eval 1 0.8648 0 True
eval 2 0.8930 0 True

Both networks keep layer4's last weight bit for bit through both epochs, so no frozen weight moved in either. The one in training mode counted 170 batches into its first BatchNorm and its head scored 0.8633 after two epochs, the one in evaluation mode counted none and scored 0.8930. Each epoch costs a full forward pass through the frozen network here, where the cached features of the second subtitle paid that pass once, and m.eval() is called at the top of every epoch because accuracy leaves the model in training mode. A loop that must keep the whole model in training mode, because some later part of it trains and has its own BatchNorms, can switch the BatchNorms alone back:

m = models.resnet18(weights=weights)
m.fc = nn.Identity()
m.requires_grad_(False)
before = m.bn1.running_mean.clone()
m.train()
for module in m.modules():
    if isinstance(module, nn.BatchNorm2d):
        module.eval()
with torch.no_grad():
    for x, y in train_loader:
        m(x)
print(m.training, m.bn1.training, m.bn1.num_batches_tracked.item(), torch.equal(before, m.bn1.running_mean))

True False 0 True

m.modules() walks every module at every depth, and module.eval() on each nn.BatchNorm2d sets its own flag back, so the model reports training mode while its 20 BatchNorms read their running statistics. After a full pass over the training tiles the first one counted nothing and its running mean is bit for bit what it was. The flag is set again after every model.train(), which resets every module's.

DON'T:

• DON'T read requires_grad_(False) as freezing a network that holds BatchNorm. The 20 BatchNorms' running statistics are buffers, and one pass in training mode moved them enough to take the head from 0.9226 to 0.5563.
• DON'T leave a frozen backbone to whatever mode a training function leaves it in. accuracy hands its model back in training mode, so a frozen backbone is set to eval() again before every pass that should read it as it is.

The Normalization Contract

What. The preprocessing weights.transforms builds, ImageNet's mean and standard deviation subtracted and divided out per channel, Lesson 5.2.1's, puts every tile on the scale the first layer's filters were trained on. Three variations measure what it is worth: no normalization at all, normalization with EuroSAT's own statistics in place of ImageNet's, and a head trained with ImageNet's normalization served tiles without it. Why. None of the three raises anything: every version of a tile is a valid (3, 160, 160) float tensor, the network computes 512 features from it, and the head answers. The only signal is the score, and only where labels exist to compute one, so the contract is measured, not assumed. When. Every time a pretrained network's input pipeline is written, and again every time the same model is served by code other than the code that trained it.

import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, models, transforms
from sat import weights, train_set, val_set, val_loader
from image_loop import loss_fn

backbone = models.resnet18(weights=weights)
backbone.fc = nn.Identity()
backbone.eval()
plain = transforms.Compose([transforms.Resize(160), transforms.ToTensor()])
raw = datasets.EuroSAT("data", transform=plain)

def features(dataset, indices):
    out, labels = [], []
    with torch.no_grad():
        for x, y in DataLoader(Subset(dataset, indices), batch_size=100):
            out.append(backbone(x))
            labels.append(y)
    return torch.cat(out), torch.cat(labels)

F, Y = features(raw, train_set.indices)
G, Z = features(raw, val_set.indices)
x, y = next(iter(DataLoader(Subset(raw, val_set.indices), batch_size=100)))
print(round(x.min().item(), 4), round(x.max().item(), 4))

0.0784 1.0

plain is Resize and ToTensor with no Normalize, so every pixel sits between 0 and 1, the first validation batch's from 0.0784 to 1.0, and Subset(raw, train_set.indices) reads the same 2,700 and 2,700 tiles sat.py split, through the other transform. The first validation batch both ways, per channel:

xn, yn = next(iter(val_loader))
print(x.mean(dim=(0, 2, 3)).round(decimals=3), xn.mean(dim=(0, 2, 3)).round(decimals=3))
print(x.std(dim=(0, 2, 3)).round(decimals=3), xn.std(dim=(0, 2, 3)).round(decimals=3))

tensor([0.3200, 0.3670, 0.3970]) tensor([-0.7220, -0.3970, -0.0400])
tensor([0.1960, 0.1310, 0.1120]) tensor([0.8580, 0.5830, 0.4990])

Unnormalized, the three channels average 0.32 to 0.40 and spread by 0.11 to 0.20. Normalized with ImageNet's statistics they average -0.72 to -0.04 and spread by 0.50 to 0.86: not zero and one, because satellite tiles are darker and plainer than ImageNet's photos, but on the scale the first layer's filters were trained on, about four times wider than the raw pixels. The second subtitle's head recipe as a function, a head trained on the unnormalized features, and the second subtitle's own head served those same unnormalized validation features:

def fit(F, Y):
    torch.manual_seed(42)
    head = nn.Linear(512, 10)
    optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
    order = torch.Generator().manual_seed(42)
    for epoch in range(20):
        perm = torch.randperm(len(F), generator=order)
        for i in range(0, len(F), 32):
            idx = perm[i:i + 32]
            loss = loss_fn(head(F[idx]), Y[idx])
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return head

def score(head, G, Z):
    with torch.no_grad():
        return f"{(head(G).argmax(dim=1) == Z).float().mean().item():.4f}"

trained_with = nn.Linear(512, 10)
trained_with.load_state_dict(torch.load("head.pt", weights_only=True))
print(score(fit(F, Y), G, Z), score(trained_with, G, Z))

0.8730 0.4281

Trained and validated without normalization, consistently, the frozen head reaches 0.8730, five points under the 0.9226 it reaches with ImageNet's statistics on the same tiles. Trained with normalization and served without it, the same 2,700 validation tiles score 0.4281: the training script and the serving script disagree about what a tile is, every tile gets an answer, and fewer than half of them are right. EuroSAT's own statistics in place of ImageNet's, computed from the 2,700 training tiles alone, Lesson 4.5.1's rule:

pixels = torch.cat([x for x, y in DataLoader(Subset(raw, train_set.indices), batch_size=100)])
mean, std = pixels.mean(dim=(0, 2, 3)), pixels.std(dim=(0, 2, 3))
print(mean.round(decimals=4), std.round(decimals=4))
own = transforms.Compose([plain, transforms.Normalize(mean.tolist(), std.tolist())])
tiles = datasets.EuroSAT("data", transform=own)
F3, Y3 = features(tiles, train_set.indices)
G3, Z3 = features(tiles, val_set.indices)
print(score(fit(F3, Y3), G3, Z3))

tensor([0.3385, 0.3765, 0.4046]) tensor([0.1963, 0.1320, 0.1109])
0.9296

transforms.Compose takes another Compose as a step, so own is plain followed by a Normalize with EuroSAT's own three means and three standard deviations, which put each channel at a mean of zero and a spread of one over the training tiles. The frozen head reaches 0.9296, within a point of ImageNet's 0.9226, about 19 tiles of 2,700: what the filters need is inputs on roughly the scale they were trained on, and the dataset's own statistics give that as well as ImageNet's do. The number that matters is the 0.4281, a pipeline that trains one way and serves another.

DON'T:

• DON'T serve a model through preprocessing written apart from the code that trained it. The head trained on normalized features scored 0.4281 on the same tiles served without normalization.
• DON'T read a frozen model's lower score as the features' ceiling before checking its input. The same backbone and the same head recipe gave 0.8730 without normalization and 0.9226 with it.
• DON'T treat ImageNet's six numbers as the only correct normalization. EuroSAT's own, computed on its training tiles, gave 0.9296, and the rule the runs support is a sensible scale and the same preprocessing in training and in serving.

Fine-Tuning Everything

What. Fine-tuning builds the pretrained network with a new seeded fc, leaves every flag on, and trains all 11,181,642 parameters with train_epoch, Lesson 4.5.2's loop, in training mode, so every BatchNorm's statistics follow the new images as the weights do. The rate is 1e-4, a tenth of the head's, with the best epoch kept by validation accuracy, Lesson 4.4.2's copy.deepcopy. Why. A frozen backbone computes the features ImageNet's task needed, and some of what separates land uses was never part of that task. Fine-tuning lets every layer move toward EuroSAT, and the small rate keeps each step a small change to weights that already work. When. When the frozen head's score leaves room, when an epoch's cost is affordable, and once the head is no longer the only thing the data can improve.

import copy
import time
import torch
from torch import nn, optim
from torch.utils.data import DataLoader
from torchvision import models
from sat import weights, full, train_set, train_loader, val_loader
from image_loop import accuracy, train_epoch

reference = models.resnet18(weights=weights)
start = time.perf_counter()
reference.eval()
with torch.no_grad():
    for x, y in DataLoader(train_set, batch_size=32):
        reference(x)
print(f"{time.perf_counter() - start:.1f}")
model = models.resnet18(weights=weights)
torch.manual_seed(42)
model.fc = nn.Linear(512, 10)
optimizer = optim.Adam(model.parameters(), lr=1e-4)
print(f"{accuracy(model, val_loader):.4f}")
best_acc, best_state = 0.0, None
for epoch in range(1, 4):
    start = time.perf_counter()
    train_loss = train_epoch(model, optimizer, train_loader)
    seconds = time.perf_counter() - start
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}", f"{seconds:.1f}")
    if val_acc > best_acc:
        best_acc, best_state = val_acc, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
torch.save(model.state_dict(), "finetuned.pt")

18.5
0.0474
1 0.5745 0.9263 50.3
2 0.1301 0.9541 51.3
3 0.0508 0.9567 50.6

The timed pass runs the network forward over the 2,700 training tiles once, in batches of 32 and in order, 18.5 seconds, and each fine-tuning epoch, forward and backward through every layer with Adam's step on 11,181,642 parameters, took nearly three times that in the same script. Before any step, with its seeded head, the network calls 0.0474 of the validation tiles right, under a guess's one in ten. After one epoch at 1e-4 it reaches 0.9263, past the frozen head's 0.9226 after twenty, and after three 0.9567. How far each part of the network moved:

for name in ("conv1.weight", "layer1.0.conv1.weight", "layer4.1.conv2.weight"):
    moved = (model.state_dict()[name] - reference.state_dict()[name]).norm() / reference.state_dict()[name].norm()
    print(name, f"{moved.item():.4f}")

conv1.weight 0.0077
layer1.0.conv1.weight 0.0196
layer4.1.conv2.weight 0.1257

Each line is the size of the change in one weight tensor divided by the tensor's own size, norm being Lesson 2.3.3's straight-line length, np.linalg.norm, taken over every entry of a tensor of any shape. The first convolution moved by 0.77 percent of itself, layer1's first by 2 percent, and layer4's last by 12.6 percent: three epochs at 1e-4 changed layer4's last weight far more than the two early ones. finetuned.pt holds the network of epoch 3, the best of the three, for Lesson 5.2.3. How each strategy fits its own training tiles, and where fine-tuning's gain on the validation tiles comes from, class by class:

print(f"{accuracy(model, DataLoader(train_set, batch_size=100)):.4f}", f"{accuracy(model, val_loader):.4f}")
cached = torch.load("features.pt", weights_only=True)
head = nn.Linear(512, 10)
head.load_state_dict(torch.load("head.pt", weights_only=True))
model.eval()
with torch.no_grad():
    print(f"{(head(cached['F']).argmax(dim=1) == cached['Y']).float().mean().item():.4f}")
    tuned = torch.cat([model(x) for x, y in val_loader]).argmax(dim=1)
    frozen = head(cached["G"]).argmax(dim=1)
for k, name in enumerate(full.classes):
    mine = cached["Z"] == k
    print(name, f"{(frozen[mine] == k).float().mean().item():.3f}", f"{(tuned[mine] == k).float().mean().item():.3f}")

0.9985 0.9567
0.9844
AnnualCrop 0.905 0.915
Forest 0.976 0.993
HerbaceousVegetation 0.928 0.974
Highway 0.852 0.934
Industrial 0.943 0.967
Pasture 0.936 0.911
PermanentCrop 0.879 0.933
Residential 0.976 0.986
River 0.835 0.934
SeaLake 0.964 0.991

The fine-tuned network calls 0.9985 of its own 2,700 training tiles right and 0.9567 of the validation tiles, a gap of four points, and the frozen head 0.9844 against 0.9226, a gap of six: both fit what they trained on more closely than what they did not, Lesson 3.5.1's overfitting, and fine-tuning's eleven million parameters did not widen the gap. Class by class, the first column the frozen head's and the second the fine-tuned network's, rivers rise by about ten points and highways by eight, the two classes whose tiles Lesson 5.2.1 drew as a dark band and a pale strip, and the one class that falls is pasture, from 0.936 to 0.911, the class with the fewest tiles.

DON'T:

• DON'T fine-tune at the head's rate because the head trained well at it. Lesson 5.2.3 runs every weight at 1e-3 and measures what that does to a pretrained network.
• DON'T read the frozen head as the safe choice because it changes nothing. On these 2,700 tiles it stopped at 0.9226 after twenty epochs where fine-tuning passed it after one.
• DON'T read the fine-tuned network's 0.9985 on its training tiles as its skill. Those are the tiles it was fitted to, and the 2,700 validation tiles, 0.9567, are the ones that say what it does on tiles it has not seen.

Lab Exercises

Lab Exercise 1: Unfreezing One Stage

In one_stage.py, load ResNet-18 with its ImageNet weights, turn off the flag on every parameter except those of layer4, then give it a seeded nn.Linear(512, 10) head. Print how many parameters train, and say in a comment what that number is made of.

Put the model in evaluation mode, run one batch from train_loader through it and call backward() on the loss. Print whether layer3's last convolution received a gradient, the shape of the gradient layer4's first convolution received, and the first BatchNorm's batch count, and say in a comment why the count reads what it reads.

Lab Exercise 2: Three Image Sizes, Frozen

In sizes.py, for tiles resized to 64, 160 and 224 pixels through weights.transforms, compute the frozen backbone's training and validation features, through Subset on sat.py's indices, timing each size's pair of passes in the same script. Train a head on each size's features exactly as the second subtitle did, and print the size, the seconds and the validation accuracy after twenty epochs. Say in a comment what the larger size buys, at what cost, and why 64 is not the free choice it looks like.

import torch
from torch import nn
from torchvision import models
from sat import weights, train_loader
from image_loop import loss_fn

model = models.resnet18(weights=weights)
for name, p in model.named_parameters():
    p.requires_grad_(name.startswith("layer4"))
torch.manual_seed(42)
model.fc = nn.Linear(512, 10)
print(sum(p.numel() for p in model.parameters() if p.requires_grad))
# layer4's 8,393,728 parameters and the new head's 5,130.
model.eval()
x, y = next(iter(train_loader))
loss_fn(model(x), y).backward()
print(model.layer3[1].conv2.weight.grad is None, model.layer4[0].conv1.weight.grad.shape, model.bn1.num_batches_tracked)
# Evaluation mode reads the running statistics and updates none,
# so the count stays at the 0 a newly built network starts at.

8398858
True torch.Size([512, 256, 3, 3]) tensor(0)

import time
import torch
from torch import nn
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, models
from sat import weights, train_set, val_set
from image_loop import loss_fn

backbone = models.resnet18(weights=weights)
backbone.fc = nn.Identity()
backbone.eval()

def features(dataset, indices):
    out, labels = [], []
    with torch.no_grad():
        for x, y in DataLoader(Subset(dataset, indices), batch_size=100):
            out.append(backbone(x))
            labels.append(y)
    return torch.cat(out), torch.cat(labels)

def fit(F, Y):
    torch.manual_seed(42)
    head = nn.Linear(512, 10)
    optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
    order = torch.Generator().manual_seed(42)
    for epoch in range(20):
        perm = torch.randperm(len(F), generator=order)
        for i in range(0, len(F), 32):
            idx = perm[i:i + 32]
            loss = loss_fn(head(F[idx]), Y[idx])
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return head

for side in (64, 160, 224):
    ds = datasets.EuroSAT("data", transform=weights.transforms(crop_size=side, resize_size=side))
    start = time.perf_counter()
    F, Y = features(ds, train_set.indices)
    G, Z = features(ds, val_set.indices)
    seconds = time.perf_counter() - start
    head = fit(F, Y)
    with torch.no_grad():
        print(side, f"{seconds:.1f}", f"{(head(G).argmax(dim=1) == Z).float().mean().item():.4f}")
# 224 adds half a point over 160 for about twice the seconds, and 64
# is fastest and loses over two and a half points: at 64 layer4 writes
# 2 by 2 maps, so avgpool averages 4 positions where 160 averages 25.

64 6.6 0.8959
160 40.9 0.9226
224 86.4 0.9274

Theory Assessment

Q1: A Gradient of None

After freezing, model.conv1.weight.grad printed None while model.fc.weight.grad printed a (10, 512) tensor. Say what requires_grad_(False) changes about the backward pass, why None and not a tensor of zeros, what Adam does with a parameter whose gradient is None, and why the optimizer should still be handed only the trainable parameters.

Hint: Ask where autograd stops building the graph once no input to a layer requires a gradient, and what a count of the optimizer's parameters is used for.

Q2: The Same Score Twice

The head trained on cached features scored 0.9226, and the same head put back as fc scored 0.9226 through accuracy, its logits matching within 1e-5. Say why the two routes must agree, what the cached route saves and on which part of the work, and the one change to the training images that would make the cache wrong.

Hint: Write what the frozen network computes for one tile in evaluation mode, and ask whether anything in it depends on the epoch.

Q3: Two Ways to Skip a Step

Without normalization, a head trained and validated that way scored 0.8730, and the head trained with normalization scored 0.4281 on tiles served without it. Say why the second number is so much lower than the first, what the first one measures, and which one a team that writes its serving code separately from its training code should fear.

Hint: In one case the head was fitted to the features it is scored on, and in the other it was fitted to features it never sees again.

Q4: Frozen, and Still Moving

A teammate freezes ResNet-18 with requires_grad_(False), trains a new head end to end through a loop that calls model.train() before every epoch, sees the head's validation accuracy wobble from epoch to epoch, and blames the learning rate. Say what moves in that network although no weight does, how much it moved in the third subtitle and what it cost the head, and the two ways to train a frozen head without it.

Hint: Separate a module's parameters from its buffers, and ask which of the two requires_grad_ reaches and which of the two training mode updates.

Assignment

Assignment: What Eight Times the Labels Buy

The frozen head trained on 2,700 tiles reached 0.9226, and fine-tuning on the same 2,700 reached 0.9567. Write more_labels.py, which asks the other question: what the frozen head reaches with more labeled tiles, up to the 21,600 that train_set and held_back hold together.

One class this lesson did not use. torch.utils.data.ConcatDataset([a, b]) joins datasets end to end into one, of length len(a) + len(b), whose first len(a) items are a's in order and the rest b's, so one loader reads both.

Compute the frozen backbone's features, in evaluation mode, for ConcatDataset([train_set, held_back]) in batches of 100 without shuffling, print its length, the features' shape and the seconds the pass took, and load the validation features from features.pt. Train a head the second subtitle's way on the first 270, 1,350, 2,700, 8,100 and 21,600 rows, and print each count with its validation accuracy. Say in a comment what the labels buy up to 2,700 and what eight times as many buy after, how the frozen head on 21,600 tiles compares with fine-tuning on 2,700, and what the two numbers say about where a team's next hour goes.

Hints:

• Because ConcatDataset keeps train_set first and in order, its first 2,700 rows are the 2,700 the second subtitle trained on, and the 2,700 row should come back as 0.9226.
• The 18,900 held-back tiles are seven times the 2,700 training tiles, so the pass is long: run it once and keep its features in memory for all five heads.

import time
import torch
from torch import nn
from torch.utils.data import ConcatDataset, DataLoader
from torchvision import models
from sat import weights, train_set, held_back
from image_loop import loss_fn

backbone = models.resnet18(weights=weights)
backbone.fc = nn.Identity()
backbone.eval()
both = ConcatDataset([train_set, held_back])
start = time.perf_counter()
out, labels = [], []
with torch.no_grad():
    for x, y in DataLoader(both, batch_size=100):
        out.append(backbone(x))
        labels.append(y)
F, Y = torch.cat(out), torch.cat(labels)
print(len(both), F.shape, f"{time.perf_counter() - start:.1f}")
cached = torch.load("features.pt", weights_only=True)
G, Z = cached["G"], cached["Z"]

21600 torch.Size([21600, 512]) 166.5

def fit(F, Y):
    torch.manual_seed(42)
    head = nn.Linear(512, 10)
    optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
    order = torch.Generator().manual_seed(42)
    for epoch in range(20):
        perm = torch.randperm(len(F), generator=order)
        for i in range(0, len(F), 32):
            idx = perm[i:i + 32]
            loss = loss_fn(head(F[idx]), Y[idx])
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return head

for n in (270, 1350, 2700, 8100, 21600):
    head = fit(F[:n], Y[:n])
    with torch.no_grad():
        print(n, f"{(head(G).argmax(dim=1) == Z).float().mean().item():.4f}")
# From 270 to 2,700 labels the frozen head gains 6.5 points, and from
# 2,700 to 21,600 it gains 2.3. On 21,600 it reaches 0.9456, still under
# the 0.9567 fine-tuning reached on 2,700: here an hour spent fine-tuning
# beats an hour spent labeling, until labels are cheap.

270 0.8581
1350 0.9022
2700 0.9226
8100 0.9281
21600 0.9456

Additional Resources

Optional, and nothing later depends on either. Worth opening for when transfer pays off at all, or for evaluation mode's one-line contract.

• YouTube - "Transfer Learning (C3W2L07)" by DeepLearningAI, Andrew Ng's course lecture. When transfer from one task to another makes sense, the two tasks sharing their inputs, the first having far more data, and its low-level features likely to help the second, with pretraining and fine-tuning named as the two phases.
• Official docs - PyTorch 2.14, torch.nn.Module.eval. The contract of evaluation mode, which modules it affects, BatchNorm and Dropout among them, and the note that it is not the same as torch.no_grad().

Summary

Freezing and fine-tuning turned out to be less a choice of safety against risk than a choice of what to trust. Freezing trusts ImageNet's features and trains 5,130 numbers on top of them, and it was fast, cheap and good, 0.9226 from features computed once. Fine-tuning trusts the data to improve every layer, and it passed the frozen head in its first epoch, at the cost of a full pass forward and backward through eleven million parameters on every batch.

The frozen end held the quieter dangers. A network with every flag off was still a network with buffers, and one pass in training mode rewrote them and took the head from 0.9226 to 0.5563 without moving a weight. A head trained through one pipeline and served through another fell to 0.4281 with every input valid, while a head trained and served on EuroSAT's own statistics did as well as one on ImageNet's: the contract was never about the six numbers, only about a sensible scale and the same pipeline on both sides. Neither failure appeared anywhere a loss is printed, which is why each was measured here against the score of the version done right.

And the labels had their own lesson. Eight times as many of them lifted the frozen head by two points, which fine-tuning beat on the original 2,700. The next question is how far fine-tuning can be pushed before it starts to damage what it was given, which is a question about learning rates, Lesson 5.2.3's.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

for p in model.parameters():
    p.requires_grad_(False)
model.fc = nn.Linear(512, 10)  # trains
model.eval()                   # buffers stay put
trainable = [p for p in model.parameters() if p.requires_grad]
optimizer = optim.Adam(trainable, lr=0.001)

backbone.fc = nn.Identity()
backbone.eval()
with torch.no_grad():
    F = torch.cat([backbone(x) for x, y in loader])
head = nn.Linear(512, 10)  # trained on F, then
backbone.fc = head         # the same frozen network