Pretrained Models and the Weights API

Context

A pretrained model is a network whose weights someone else already trained, on far more images than you have, shipped as a file you load into the same architecture. torchvision's weights API names each such file, downloads it once, and hands back with it the facts that come with the weights: the classes they were trained to name, the scores they reached, and the exact preprocessing their first layer expects.

Most real vision work starts here, not from seeded weights. A team classifying satellite tiles, product photos or medical scans loads a network trained on ImageNet (a public set of about 1.3 million photos in 1,000 everyday classes), reads what it contains, and adapts it, because training the same network from nothing would need data and processor time the team does not have.

Section 5.1 trained a 20,538-parameter network from a seeded start on 50,000 grayscale garments. Loading eleven million trained parameters is one line, and almost every mistake around it is silent: a model built without its weights trains from random filters and is never told, an image fed without the preprocessing the weights expect gives worse answers with no error, and a pretrained network asked about a satellite tile answers with an ImageNet class, a nematode for a highway, at 0.655 confidence.

Goal

• Load ResNet-18 with ResNet18_Weights.IMAGENET1K_V1, read its metadata, and show that weights=None builds the same shape with none of the trained numbers.
• Trace a 160 by 160 image through ResNet-18's stages and say where its 11,689,512 parameters sit.
• Compute a residual block's output by hand, with and without its downsample shortcut, and read the refusal a shortcut of the wrong shape raises.
• Build the weights' own preprocessing at 160 pixels with weights.transforms, and name what it resizes, converts and normalizes.
• Save sat.py, EuroSAT split into four seeded parts, and read what the ImageNet classifier answers on its tiles.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the residual block drawn before you meet it in code, or torchvision's own page on the weights this lesson loads.

• YouTube - "C4W2L03 Resnets" by DeepLearningAI, Andrew Ng's course lecture. The residual block, a layer's input added back to its output two layers later, and why a very deep plain network trains worse than a residual one of the same depth.
• Official docs - torchvision 0.29, "Models and pre-trained weights", General information on pre-trained weights. Building a model with weights=, the DEFAULT alias, the preprocessing bundled with each set of weights, evaluation mode before inference, the cache folder TORCH_HOME sets, and the table of every classification model's weights with its accuracy.

Table of Contents

• Loading ResNet-18 and Its Weights
• Reading the Network
• The Residual Connection
• The Weights' Own Preprocessing
• EuroSAT, Tiles ImageNet Never Held

Theory

This lesson installs nothing, since torchvision came in Lesson 4.5.1, and it downloads two files. ResNet-18's weights, 46,830,571 bytes, arrive on the first models.resnet18(weights=...) call, which prints a Downloading: line naming the file and the folder it goes to, ~/.cache/torch/hub/checkpoints/ under your home folder, then a progress counter, and every later call reads the file from there in silence. EuroSAT, 94 MB, arrives on the first datasets.EuroSAT("data", download=True), which prints its progress while it downloads and extracts into data/eurosat beside Section 4.5's FashionMNIST, about 223 MB on disk once extracted, the 94 MB archive included. Every block runs in ~/dev/bootcamp. The last subtitle saves sat.py, which Lessons 5.2.2 and 5.2.3 import. A seeded draw prints what it prints on torch 2.14.1 on a processor. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Loading ResNet-18 and Its Weights

What. models.resnet18(weights=ResNet18_Weights.IMAGENET1K_V1) builds ResNet-18 (an 18-layer convolutional network from 2015, built from the blocks of Section 5.1 and one addition, the third subtitle's) and fills it with the weights trained on ImageNet. The weights object is an enum (a fixed set of named values), and each value carries the file's address and a meta dictionary of facts about the weights: their parameter count, the 1,000 class names in output order, and the accuracy they reached. weights=None builds the same architecture with seeded random weights. Why. The architecture and the numbers are two things, Lesson 4.4.3's split between the script that builds a shape and the file that fills it, and the weights API keeps them apart in the same way. A model built without its weights has every parameter in place and none of the training, and it raises no error when used. When. Every time a pretrained network is the starting point, which in vision is nearly every time a dataset is small.

import os
import torch
from torch import nn
from torchvision import models
from torchvision.models import ResNet18_Weights

weights = ResNet18_Weights.IMAGENET1K_V1
model = models.resnet18(weights=weights)
print(weights, ResNet18_Weights.DEFAULT is weights)
print(weights.meta["num_params"], sum(p.numel() for p in model.parameters()))
print(weights.meta["_metrics"]["ImageNet-1K"])
print(len(weights.meta["categories"]), weights.meta["categories"][:3])
path = os.path.join(torch.hub.get_dir(), "checkpoints", "resnet18-f37072fd.pth")
print(os.path.getsize(path))

ResNet18_Weights.IMAGENET1K_V1 True
11689512 11689512
{'acc@1': 69.758, 'acc@5': 89.078}
1000 ['tench', 'goldfish', 'great white shark']
46830571

IMAGENET1K_V1 is the first set of ImageNet-1K weights published for ResNet-18, and DEFAULT is an alias for the best set torchvision ships for this architecture, here the same one, so is prints True. The metadata's parameter count is the model's own, 11,689,512, and the weights reached 69.758 percent top-1 accuracy (the share of ImageNet's validation photos whose most probable class was right) and 89.078 percent top-5 (the right class among the five most probable), measured at 224 by 224. The 1,000 categories start with three fish, and torch.hub.get_dir() is the cache folder, so the weights file sits on disk at 46,830,571 bytes. The rest of the metadata prices the network:

print(sorted(weights.meta))
print(weights.meta["_ops"], weights.meta["_file_size"], weights.meta["min_size"])

['_docs', '_file_size', '_metrics', '_ops', 'categories', 'min_size', 'num_params', 'recipe']
1.814 44.661 (1, 1)

_ops is torchvision's count of the billions of floating-point operations one 224 by 224 image costs, 1.814. _file_size is the file in megabytes counted in units of 1,048,576 bytes, the 46,830,571 again. min_size says the network accepts an image as small as one pixel a side, and recipe links the training script that produced the weights. At 160 by 160 the convolutions slide over about half the positions they do at 224, (160 / 224) ** 2, the reason Section 5.2 runs at 160 on a processor. The same shape without the weights:

torch.manual_seed(42)
blank = models.resnet18(weights=None)
print(torch.equal(blank.conv1.weight, model.conv1.weight))
print(round(model.conv1.weight.std().item(), 4), round(blank.conv1.weight.std().item(), 4))

False
0.1297 0.025

blank holds 11,689,512 parameters of its own, seeded, and its first layer's weights spread about a fifth as widely as the trained ones: training moved them far from where torch.manual_seed(42) puts them. Nothing about blank says it is untrained except its predictions. The file the weights came in, read the way Lesson 4.4.3 read its own:

state = torch.load(path, weights_only=True)
print(type(state).__name__, len(state), list(state)[:3])
print(sum(v.numel() for k, v in state.items() if "running" not in k))
print(blank.load_state_dict(state))
same = zip(blank.state_dict().values(), model.state_dict().values())
print(all(torch.equal(a, b) for a, b in same))

OrderedDict 102 ['conv1.weight', 'bn1.running_mean', 'bn1.running_var']
11689512
<All keys matched successfully>
True

The weights file is a state dict, an ordered dictionary of 102 named tensors: every parameter, 11,689,512 numbers once the BatchNorms' running statistics are left out, and those running statistics, the buffers of Lesson 5.1.2. Loaded into the seeded blank, it makes blank the pretrained network tensor for tensor. weights=IMAGENET1K_V1 is this, done for you: build the shape, download the file once, load it. Older tutorials build the pretrained model another way:

old = models.resnet18(pretrained=True)
print(torch.equal(old.conv1.weight, model.conv1.weight))

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/torchvision/models/_utils.py:207: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
  warnings.warn(
/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/torchvision/models/_utils.py:222: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
  warnings.warn(msg)
True

pretrained=True still loads the same weights in torchvision 0.29.1, under two warnings raised from torchvision's own _utils.py, not from your script, which say the argument may be removed and name the enum that replaces it.

DON'T:

• DON'T read a model's parameter count as proof that it is pretrained. weights=None built the same 11,689,512 parameters, and its first layer's weights matched none of the trained ones.
• DON'T copy pretrained=True from an older tutorial. It loads the same weights today under two deprecation warnings, and the enum says which weights you load.
• DON'T treat the weights file as something other than a state dict. It is 102 named tensors, and load_state_dict put them into a seeded network tensor for tensor, the mechanism Lesson 4.4.3 taught.

Reading the Network

What. ResNet-18 is a stem, conv1 (a 7 by 7 convolution at stride 2), bn1, a ReLU and a max pooling, then four stages, layer1 to layer4, each two residual blocks, then avgpool, Lesson 5.1.3's nn.AdaptiveAvgPool2d(1), and fc, a linear layer from 512 values to 1,000 logits. Each stage after the first halves the map and doubles the channels, Lesson 5.1.1's arithmetic. Why. Every later decision about a pretrained network is made by stage: which layers to freeze, which to train, at what rate, and what to replace. That needs the names, the shapes and where the parameters sit, read from the model itself. When. Before freezing, replacing or retraining any part of a pretrained model, and before a learning rate is set per part.

for name, child in model.named_children():
    print(name, type(child).__name__, sum(p.numel() for p in child.parameters()))
print(model.layer1[0])

conv1 Conv2d 9408
bn1 BatchNorm2d 128
relu ReLU 0
maxpool MaxPool2d 0
layer1 Sequential 147968
layer2 Sequential 525568
layer3 Sequential 2099712
layer4 Sequential 8393728
avgpool AdaptiveAvgPool2d 0
fc Linear 513000
BasicBlock(
  (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
)

named_children walks the top level in order. layer4 holds 8,393,728 of the 11,689,512 parameters, and fc holds 513,000, 512 weights for each of 1,000 classes and their biases. A block is Lesson 5.1.2's conv block twice over, two 3 by 3 convolutions without a bias, each followed by BatchNorm, with no pooling inside. inplace=True lets the ReLU overwrite its input instead of allocating a new tensor, the same result in less memory. A 160 by 160 image through every top-level layer, in evaluation mode:

model.eval()
h = torch.zeros(1, 3, 160, 160)
with torch.no_grad():
    for name, child in model.named_children():
        if name == "fc":
            h = torch.flatten(h, 1)
        h = child(h)
        print(name, tuple(h.shape))
print([n for n, m in model.named_modules() if n.endswith("downsample")])

conv1 (1, 64, 80, 80)
bn1 (1, 64, 80, 80)
relu (1, 64, 80, 80)
maxpool (1, 64, 40, 40)
layer1 (1, 64, 40, 40)
layer2 (1, 128, 20, 20)
layer3 (1, 256, 10, 10)
layer4 (1, 512, 5, 5)
avgpool (1, 512, 1, 1)
fc (1, 1000)
['layer2.0.downsample', 'layer3.0.downsample', 'layer4.0.downsample']

The stem takes 160 to 80 with its stride and to 40 with its pooling, and layer2 to layer4 halve it to 5, while the channels go from 64 to 512. The model's own forward flattens between avgpool and fc, which is why the loop flattens there too. named_modules walks every module at every depth, and three of them are named downsample, one in the first block of each stage that halves. The weights were trained at 224 by 224, and the same network reads 160 and 64 too:

with torch.no_grad():
    for side in (64, 160, 224):
        h = torch.zeros(1, 3, side, side)
        for name in ("conv1", "bn1", "relu", "maxpool", "layer1", "layer2", "layer3", "layer4"):
            h = getattr(model, name)(h)
        print(side, tuple(h.shape), tuple(model.avgpool(h).shape))

64 (1, 512, 2, 2) (1, 512, 1, 1)
160 (1, 512, 5, 5) (1, 512, 1, 1)
224 (1, 512, 7, 7) (1, 512, 1, 1)

layer4 writes 512 maps of 2 by 2, 5 by 5 or 7 by 7, a 32nd of the side, and avgpool reduces every one of them to its mean, so fc always reads 512 values. Every layer before avgpool is a convolution, a BatchNorm, a ReLU or a pooling, none built for a size, Lesson 5.1.3's Answer 1, which is why weights trained at 224 run at 160 with no change. Whether they work as well there is a measurement, Lesson 5.2.2's.

DON'T:

• DON'T read a stage's position as its share of the network. layer4, the last stage, holds 8,393,728 parameters and layer1 147,968, and a decision made by stage weighs that.
• DON'T call the children in order and skip the flatten. fc is nn.Linear(512, 1000), and the (1, 512, 1, 1) that avgpool writes needs torch.flatten(h, 1) first, which the model's own forward does.

The Residual Connection

What. A residual block computes two convolutions with their BatchNorms, its path, and adds the block's input back before the last ReLU: relu(path(x) + x), the addition called a residual connection, or shortcut. Where the block halves the map and doubles the channels, the input no longer has the path's shape, so the shortcut carries it through downsample, a 1 by 1 convolution at stride 2 and a BatchNorm, before the addition. Why. With the input added back, a block that has nothing to add can leave its input almost as it came, and the gradient of the loss reaches the early layers through the additions as well as through every convolution. That is what let networks of 18 layers and far more train well where plain stacks of the same depth trained worse, the argument of the paper that introduced it. When. Every time a ResNet's block is read, retrained or cut, and every time a shape error names a block's output.

import torch
from torchvision import models
from torchvision.models import ResNet18_Weights

model = models.resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
model.eval()
torch.manual_seed(42)
x = torch.randn(1, 64, 40, 40)
block = model.layer1[0]
with torch.no_grad():
    path = block.bn2(block.conv2(block.relu(block.bn1(block.conv1(x)))))
    print(torch.allclose(block(x), torch.relu(path + x), atol=1e-5))
    print(round((block(x) - torch.relu(path)).abs().max().item(), 4))
    print(round(path.abs().mean().item(), 4), round(x.abs().mean().item(), 4))

True
4.6291
1.2659 0.8022

A seeded (1, 64, 40, 40) tensor stands in for layer1's input, the shape the trace printed. The block's output is the path plus its input through the ReLU, allclose within 1e-5, and the same path without the input added differs from the block's output by up to 4.6291, so dropping the shortcut changes what the block writes. Here the path's values average 1.2659 in size against the input's 0.8022. A block that halves:

print(model.layer2[0].downsample)
b = model.layer2[0]
with torch.no_grad():
    path = b.bn2(b.conv2(b.relu(b.bn1(b.conv1(x)))))
    short = b.downsample(x)
    print(b(x).shape, short.shape, torch.allclose(b(x), torch.relu(path + short), atol=1e-5))
    try:
        path + x
    except RuntimeError as err:
        print(err)

Sequential(
  (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
  (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, bias=True, track_running_stats=True)
)
torch.Size([1, 128, 20, 20]) torch.Size([1, 128, 20, 20]) True
The size of tensor a (20) must match the size of tensor b (40) at non-singleton dimension 3

layer2[0]'s first convolution has stride 2 and 128 filters, so its path writes (1, 128, 20, 20) from a (1, 64, 40, 40) input. The downsample is Lesson 5.1.1's 1 by 1 convolution, here at stride 2, mixing 64 channels into 128 at every second position, so the shortcut arrives in the path's shape and the sum is the block's output. Without it the addition is refused, the message naming the widths, 20 against 40, before it ever reached the channels.

DON'T:

• DON'T rebuild a ResNet block as two convolutions and a ReLU. The path alone differed from the block's output by up to 4.6291, because the block adds its input back.
• DON'T add a halving block's input to its path directly. The shapes (1, 128, 20, 20) and (1, 64, 40, 40) were refused, and the block's own downsample is what makes the sum possible.
• DON'T read the downsample as a pooling layer. It is a 1 by 1 convolution with 8,192 trained weights and its own BatchNorm, so it learns how the 64 channels become 128, and it trains with the rest of the block.

The Weights' Own Preprocessing

What. weights.transforms() returns the preprocessing the weights were evaluated with: resize a photo so its shorter side is 256 pixels, crop the middle 224 by 224, convert it to a float tensor from 0 to 1, and normalize each channel with ImageNet's mean and standard deviation. weights.transforms(crop_size=160, resize_size=160) keeps the normalization and changes the two sizes, the 160 by 160 Section 5.2 runs at. Why. The first layer's filters were trained on inputs normalized by those six numbers, so an input on another scale reaches filters tuned for values it no longer has, and the network answers anyway. Lesson 5.2.2 measures what skipping the normalization costs, and what normalizing with the dataset's own statistics instead gives. When. Every time a pretrained network reads an image, in training and in evaluation alike.

import torch
from torchvision import datasets, transforms
from torchvision.models import ResNet18_Weights

print(ResNet18_Weights.IMAGENET1K_V1.transforms())
prep = ResNet18_Weights.IMAGENET1K_V1.transforms(crop_size=160, resize_size=160)
print(prep)
raw = datasets.EuroSAT("data", download=True)
img, label = raw[0]
print(type(img).__name__, img.size, img.mode, raw.classes[label])
t = prep(img)
print(t.shape, t.dtype, round(t.min().item(), 4), round(t.max().item(), 4))
print(t.mean(dim=(1, 2)).round(decimals=4))

ImageClassification(
    crop_size=[224]
    resize_size=[256]
    mean=[0.485, 0.456, 0.406]
    std=[0.229, 0.224, 0.225]
    interpolation=InterpolationMode.BILINEAR
)
ImageClassification(
    crop_size=[160]
    resize_size=[160]
    mean=[0.485, 0.456, 0.406]
    std=[0.229, 0.224, 0.225]
    interpolation=InterpolationMode.BILINEAR
)
Image (64, 64) RGB AnnualCrop
torch.Size([3, 160, 160]) torch.float32 -0.7137 1.3755
tensor([-0.2477, -0.3281,  0.0248])

ImageClassification is torchvision's class for this preprocessing, and both printouts carry ImageNet's three channel means, red, green and blue, and three standard deviations. Bilinear interpolation (each new pixel a weighted mean of the four nearest old ones) does the resizing. EuroSAT's first image is a 64 by 64 color picture, a PIL image (the image type the Pillow library defines), of annual crops, and after prep it is a (3, 160, 160) float tensor from -0.7137 to 1.3755, each channel's mean near zero but not at it, because the tile is not an average ImageNet photo. A 64 by 64 tile resized to 160 is enlarged two and a half times, so its 160 by 160 pixels hold no more detail than its 4,096 had. The same preprocessing from four transforms, two of them Lesson 4.5.1's:

steps = [transforms.Resize(160), transforms.CenterCrop(160), transforms.ToTensor()]
steps.append(transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]))
by_hand = transforms.Compose(steps)
b = by_hand(img)
print(b.shape, torch.allclose(t, b, atol=1e-6), round((t - b).abs().max().item(), 7))

torch.Size([3, 160, 160]) True 0.0

transforms.Resize(160) sets the shorter side to 160, CenterCrop(160) cuts the middle square, a no-op on a square tile, and ToTensor and Normalize are Lesson 4.5.1's, here with ImageNet's six numbers. The two pipelines agree to the last bit. weights.transforms is the safer one to write, because it carries the numbers with the weights and cannot be copied wrong.

DON'T:

• DON'T leave a pretrained network's input unnormalized because the tile already sits between 0 and 1. The filters were trained on inputs centered near zero and spread far wider than raw pixels, and Lesson 5.2.2 measures what the raw scale costs.
• DON'T read the 160 by 160 tensor as a sharper tile. It is the 64 by 64 picture enlarged by interpolation, every pixel computed from the 4,096 that exist.
• DON'T copy ImageNet's six numbers into a hand-written Normalize when the weights carry them. The hand-built pipeline matched to the bit here, and weights.transforms is the version that cannot drift from the weights it belongs to.

EuroSAT, Tiles ImageNet Never Held

What. EuroSAT is 27,000 color tiles of 64 by 64 pixels cut from Sentinel-2 satellite photographs of Europe, each labeled with one of ten land uses, from annual crops to sea and lake, released by its authors under the MIT license. torchvision downloads it as one set with no split, and sat.py splits it once, seeded, into 2,700 tiles to train on, 2,700 to choose with, 2,700 to test once, and 18,900 held back. Why. ImageNet's 1,000 classes are animals, objects and scenes photographed from the ground, and none of them is a land use seen from orbit, so the pretrained network has to transfer, not recognize. Training on 2,700 tiles is the case transfer learning exists for, few labels, and 2,700 for each of validation and test leaves about 270 tiles a class in each, so one tile moves a class's accuracy by under half a point in the validation part and by at most 0.55 of a point in the test part, whose smallest class holds 182. When. Every lesson of this section reads EuroSAT through sat.py.

import collections
import matplotlib.pyplot as plt
from torchvision import datasets

raw = datasets.EuroSAT("data", download=True)
print(len(raw), raw.classes)
print(sorted(collections.Counter(raw.targets).items()))

27000 ['AnnualCrop', 'Forest', 'HerbaceousVegetation', 'Highway', 'Industrial', 'Pasture', 'PermanentCrop', 'Residential', 'River', 'SeaLake']
[(0, 3000), (1, 3000), (2, 3000), (3, 2500), (4, 2500), (5, 2000), (6, 2500), (7, 3000), (8, 2500), (9, 3000)]

raw.targets is the list of 27,000 labels, and the classes are not the same size: 3,000 tiles of annual crops, forest, herbaceous vegetation, residential land and sea or lake, 2,500 of highway, industrial land, permanent crops and river, and 2,000 of pasture. The first tile of each class, drawn:

first = {}
for i, (file, k) in enumerate(raw.samples):
    first.setdefault(k, i)
print(first)
fig, axes = plt.subplots(2, 5, figsize=(12, 5))
for k, ax in enumerate(axes.flat):
    img, label = raw[first[k]]
    ax.imshow(img)
    ax.set_title(raw.classes[label], fontsize=9)
    ax.axis("off")
fig.savefig("tiles.png", dpi=150, bbox_inches="tight")
print(len(fig.axes), axes[0, 0].images[0].get_array().shape)
plt.close(fig)

{0: 0, 1: 3000, 2: 6000, 3: 9000, 4: 11500, 5: 14000, 6: 16000, 7: 18500, 8: 21500, 9: 24000}
10 (64, 64, 3)

raw.samples is the list of 27,000 file paths and labels in folder order, one class after another, and setdefault keeps the first index each label meets. imshow draws a PIL image directly, held as a (64, 64, 3) array, height, width and the three colors, the channels last where a tensor holds them first. In tiles.png the highway is a pale diagonal strip across fields and the river a dark diagonal band, two lines across land that differ mostly in brightness, and the forest and the sea are near-uniform dark squares. Save the split as sat.py:

import torch
from torch.utils.data import DataLoader, random_split
from torchvision import datasets
from torchvision.models import ResNet18_Weights

weights = ResNet18_Weights.IMAGENET1K_V1
prep = weights.transforms(crop_size=160, resize_size=160)
full = datasets.EuroSAT("data", download=True, transform=prep)
# 2,700 to fit, 2,700 to choose on, 2,700 to test once, 18,900 held back
train_set, val_set, test_set, held_back = random_split(full, [2_700, 2_700, 2_700, 18_900], generator=torch.Generator().manual_seed(42))

train_loader = DataLoader(train_set, batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(42))
val_loader = DataLoader(val_set, batch_size=100)
test_loader = DataLoader(test_set, batch_size=100)

if __name__ == "__main__":
    print(len(train_set), len(val_set), len(test_set), len(held_back))
    print(len(train_loader), len(val_loader), len(test_loader))

2700 2700 2700 18900
85 27 27

fashion.py's pattern, Lesson 4.5.1's: one dataset with the weights' preprocessing at 160, one random_split with a generator seeded 42, here into four parts, and three loaders, the training one shuffled by its own seeded generator. 2,700 tiles in batches of 32 make 85 batches, the last of 12. What ImageNet's classifier says about the validation tiles, from a second script:

import collections
import copy
import torch
from torchvision import models
from sat import weights, full, train_set, val_set, test_set, val_loader

for part in (train_set, val_set, test_set):
    counts = collections.Counter(full.targets[i] for i in part.indices)
    print(min(counts.values()), max(counts.values()), counts[5])
model = models.resnet18(weights=weights)
model.eval()
names = weights.meta["categories"]
x, y = next(iter(val_loader))
with torch.no_grad():
    probs = model(x).softmax(dim=1)
top = probs.max(dim=1)
for i in range(6):
    print(full.classes[y[i]], names[top.indices[i]], round(top.values[i].item(), 3))

206 308 206
202 335 202
182 319 182
Residential overskirt 0.09
Highway nematode 0.655
SeaLake kite 0.047
Residential armadillo 0.044
SeaLake great white shark 0.054
Pasture dugong 0.133

full.targets[i] for i in part.indices reads each part's labels without loading a tile. The seeded split is not stratified, Lesson 3.1.2's word, so each part holds a class in rough proportion to its size, from 182 to 335 tiles, with pasture, the smallest class, the least in every part. The ImageNet classifier answers every tile with one of its own classes: a residential tile is an overskirt at 0.09, a highway a nematode at 0.655, a pasture a dugong. Over all 2,700:

with torch.no_grad():
    logits = torch.cat([model(xb) for xb, yb in val_loader])
labels = torch.cat([yb for xb, yb in val_loader])
pred = logits.argmax(dim=1)
for idx, n in collections.Counter(pred.tolist()).most_common(5):
    print(names[idx], n)
for k in range(10):
    label, n = collections.Counter(pred[labels == k].tolist()).most_common(1)[0]
    print(full.classes[k], names[label], n, (labels == k).sum().item())

great white shark 140
dugong 136
tiger shark 125
nematode 108
parachute 108
AnnualCrop tile roof 26 305
Forest tiger shark 104 286
HerbaceousVegetation volcano 31 306
Highway nematode 26 243
Industrial container ship 16 245
Pasture dugong 28 202
PermanentCrop solar dish 24 240
Residential prayer rug 30 295
River nematode 38 243
SeaLake parachute 85 335

The five ImageNet classes the network gives most are two sharks, a sea mammal, a worm and a parachute. Within each EuroSAT class its answers are scattered, the most common one covering from 16 of 245 industrial tiles to 104 of 286 forests, called tiger sharks. The fc layer answers a question about ImageNet's 1,000 classes, and no question about land use reaches it. The first batch of 100 again, through a copy of the network left in training mode:

trained = copy.deepcopy(model)
trained.train()
with torch.no_grad():
    judged = model(x).argmax(dim=1)
    batch = trained(x).argmax(dim=1)
print((judged != batch).sum().item(), len(x), trained.bn1.num_batches_tracked.item(), model.bn1.num_batches_tracked.item())

99 100 1 0

In training mode every one of the 20 BatchNorms normalizes the batch by its own 100 tiles, Lesson 5.1.2's trap, and 99 of the 100 answers change. copy.deepcopy kept the trap off the real model: the copy's first BatchNorm counted the batch, and the model's still reads 0, the count a newly built network starts at, since the weights file carries no batch counter.

DON'T:

• DON'T expect a seeded split to give each class the same share. The validation part holds from 202 to 335 tiles a class, and only a stratified split would match the 27,000's proportions exactly.
• DON'T read the ImageNet answers as what the network sees. They are its last layer's best match among 1,000 classes that hold no land use, and Lesson 5.2.2 replaces that layer.
• DON'T dip into the 18,900 held-back tiles to settle a choice or to score one. They are neither validation nor test, and the one question they answer is what more labels would buy, which an Assignment of Lesson 5.2.2 asks.
• DON'T read a pretrained network in training mode, even for a moment. On the first batch of 100, training mode changed 99 answers and counted the batch into the copy's running statistics, which a real model would have kept.

Lab Exercises

Lab Exercise 1: Where the Parameters Sit, and a Third Shortcut

In stages.py, load ResNet-18 with its ImageNet weights and print, for conv1, each of the four stages and fc, the name, the parameter count and its share of the 11,689,512 as a percentage to one decimal. Count the network's BasicBlock modules and its nn.Conv2d modules, importing BasicBlock from torchvision.models.resnet, and say in a comment why the two counts give 20 convolutions for 18 layers. Print the parameter count of layer2's downsample.

Then, in evaluation mode and seeded 42, compute layer3[0]'s output by hand for a (1, 128, 20, 20) input: its path, its shortcut through downsample, the two shapes, and whether the ReLU of their sum matches the block's output within 1e-5.

Lab Exercise 2: Three Sizes and One Lake

In tiles.py, take EuroSAT's first image and run it through weights.transforms at crop and resize sizes of 64, 160 and 224, printing each side, the tensor's shape and its mean to four decimals, and say in a comment why the means barely move.

Then import from sat, run the ImageNet classifier in evaluation mode over all 2,700 validation tiles, and print the three ImageNet classes it gives most often to the sea and lake tiles, with their counts and the number of such tiles, and how many different ImageNet classes it uses across the 2,700.

import torch
from torch import nn
from torchvision import models
from torchvision.models import ResNet18_Weights
from torchvision.models.resnet import BasicBlock

model = models.resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
model.eval()
total = sum(p.numel() for p in model.parameters())
for name in ("conv1", "layer1", "layer2", "layer3", "layer4", "fc"):
    n = sum(p.numel() for p in getattr(model, name).parameters())
    print(name, n, f"{100 * n / total:.1f}")
print(sum(isinstance(m, BasicBlock) for m in model.modules()), sum(isinstance(m, nn.Conv2d) for m in model.modules()))
print(sum(p.numel() for p in model.layer2[0].downsample.parameters()))
# 8 blocks of 2 convolutions, the stem's conv1 and fc make the 18
# layers, and the 3 downsample convolutions are the other 3 of 20.

conv1 9408 0.1
layer1 147968 1.3
layer2 525568 4.5
layer3 2099712 18.0
layer4 8393728 71.8
fc 513000 4.4
8 20
8448

torch.manual_seed(42)
x = torch.randn(1, 128, 20, 20)
b = model.layer3[0]
with torch.no_grad():
    path = b.bn2(b.conv2(b.relu(b.bn1(b.conv1(x)))))
    short = b.downsample(x)
    print(path.shape, short.shape, torch.allclose(b(x), torch.relu(path + short), atol=1e-5))

torch.Size([1, 256, 10, 10]) torch.Size([1, 256, 10, 10]) True

import collections
import torch
from torchvision import datasets, models
from sat import weights, val_loader

raw = datasets.EuroSAT("data", download=True)
img, label = raw[0]
for side in (64, 160, 224):
    t = weights.transforms(crop_size=side, resize_size=side)(img)
    print(side, tuple(t.shape), round(t.mean().item(), 4))
# Resizing interpolates between the same 4,096 pixels, and the
# normalization is the same six numbers, so the mean hardly moves.

64 (3, 64, 64) -0.1854
160 (3, 160, 160) -0.1837
224 (3, 224, 224) -0.1842

model = models.resnet18(weights=weights)
model.eval()
names = weights.meta["categories"]
with torch.no_grad():
    logits = torch.cat([model(xb) for xb, yb in val_loader])
labels = torch.cat([yb for xb, yb in val_loader])
pred = logits.argmax(dim=1)
sea = pred[labels == 9]
print([(names[i], n) for i, n in collections.Counter(sea.tolist()).most_common(3)], len(sea))
print(len(set(pred.tolist())))

[('parachute', 85), ('flagpole', 67), ('great white shark', 50)] 335
311

Theory Assessment

Q1: The Same Shape Twice

A teammate builds models.resnet18(weights=None) to save the download, checks that it has 11,689,512 parameters and the same printout as the pretrained one, and starts training. Say what the two models share, what they do not, how the run in the first subtitle told them apart, and what ResNet18_Weights.DEFAULT would have loaded.

Hint: Separate the architecture, which the function builds, from the numbers, which the weights file fills in, Lesson 4.4.3's split.

Q2: A Thousand Outputs for Ten Classes

ResNet-18's fc maps 512 values to 1,000 logits. Say how many parameters it holds and what share of the network that is, why it cannot classify EuroSAT as it stands, what a replacement for ten classes holds, and which other parameters a new head leaves untouched.

Hint: Count 512 weights and one bias per output, and compare the share with layer4's.

Q3: Two Blocks, One Shortcut

layer1[0] has no downsample and layer2[0] has one. Say what each block adds to its path, why the second needs a convolution on its shortcut and the first does not, what that convolution's weight shape is and how many parameters it and its BatchNorm hold, and what the refusal said when the input went in unchanged.

Hint: Write the input's and the path's shapes for each block, from the trace of the second subtitle.

Q4: Sharks Over the Sea

The ImageNet classifier called 85 of 335 sea and lake tiles parachutes and 104 of 286 forests tiger sharks. A teammate concludes that ImageNet's weights know nothing useful about land and that EuroSAT should be trained from seeded weights. Say what the 1,000 outputs measure and what they do not, then test the claim without training anything: replace fc with nn.Identity(), compute the 512 values avgpool writes for every training and validation tile, and classify each validation tile by the nearest of the ten training classes' mean vectors.

Hint: nn.Identity() returns its input, so the model then returns the flattened avgpool output. torch.cdist(a, b) computes the distance between every row of a and every row of b, and argmin(dim=1) picks the nearest.

import torch
from torch import nn
from torchvision import models
from torchvision.models import ResNet18_Weights

model = models.resnet18(weights=ResNet18_Weights.IMAGENET1K_V1)
kept = model.layer4[1].conv2.weight.clone()
torch.manual_seed(42)
model.fc = nn.Linear(512, 10)
model.eval()
with torch.no_grad():
    out = model(torch.zeros(1, 3, 160, 160))
print(out.shape, sum(p.numel() for p in model.fc.parameters()), sum(p.numel() for p in model.parameters()))
print(torch.equal(kept, model.layer4[1].conv2.weight))

torch.Size([1, 10]) 5130 11181642
True

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import models
from sat import weights, train_set, val_loader

model = models.resnet18(weights=weights)
model.fc = nn.Identity()
model.eval()

def feats(loader):
    a, b = [], []
    with torch.no_grad():
        for xb, yb in loader:
            a.append(model(xb))
            b.append(yb)
    return torch.cat(a), torch.cat(b)

F, Y = feats(DataLoader(train_set, batch_size=100))
G, Z = feats(val_loader)
print(F.shape)
cent = torch.stack([F[Y == k].mean(dim=0) for k in range(10)])
near = torch.cdist(G, cent).argmin(dim=1)
print(f"{(near == Z).float().mean().item():.4f}")

torch.Size([2700, 512])
0.8348

Assignment

Assignment: Which Stage Knows Land

Answer 4 classified validation tiles by the nearest class mean of the last stage's pooled features. Write stage_probe.py, which does the same with the output of every stage at once and says which stage's features separate land uses best.

One method this lesson did not use. module.register_forward_hook(fn) asks PyTorch to call fn(module, inputs, output) every time the module runs forward, after it computes its output, so a function can keep what a layer inside the network wrote without changing the network or its forward. It returns a handle whose remove() takes the hook off.

Load ResNet-18 with its ImageNet weights in evaluation mode. Register a hook on each of layer1 to layer4 that stores the stage's output averaged over its height and width, one vector per tile of the stage's channel count, in a dictionary under the stage's name. Run the training tiles, in batches of 100 without shuffling, and the validation tiles through the network once each, collecting every stage's vectors and the labels. For each stage, compute the ten training class means, classify every validation tile by the nearest mean with torch.cdist, and print the stage, its channel count and the accuracy to four decimals. Say in a comment which stage wins, how layer4's figure relates to Answer 4's, and what the ranking suggests about which layers carry ImageNet's own concerns.

Hints:

• The hook receives the output of the whole stage, (batch, channels, height, width), and output.mean(dim=(2, 3)) reduces it to (batch, channels), Lesson 5.1.3's global average pooling by hand.
• One forward pass fills all four entries of the dictionary, so collect them after every batch, before the next one overwrites them.

import collections
import torch
from torch.utils.data import DataLoader
from torchvision import models
from sat import weights, train_set, val_loader

model = models.resnet18(weights=weights)
model.eval()
grabbed = {}

def keep(name):
    def hook(module, inputs, output):
        grabbed[name] = output.mean(dim=(2, 3))
    return hook

for name in ("layer1", "layer2", "layer3", "layer4"):
    getattr(model, name).register_forward_hook(keep(name))

def run(loader):
    out, labels = collections.defaultdict(list), []
    with torch.no_grad():
        for xb, yb in loader:
            model(xb)
            labels.append(yb)
            for k, v in grabbed.items():
                out[k].append(v)
    return {k: torch.cat(v) for k, v in out.items()}, torch.cat(labels)

tr, ytr = run(DataLoader(train_set, batch_size=100))
va, yva = run(val_loader)
for stage in ("layer1", "layer2", "layer3", "layer4"):
    F, G = tr[stage], va[stage]
    cent = torch.stack([F[ytr == k].mean(dim=0) for k in range(10)])
    near = torch.cdist(G, cent).argmin(dim=1)
    print(stage, F.shape[1], f"{(near == yva).float().mean().item():.4f}")
# layer3 wins, 0.8619 to layer4's 0.8348, which is Answer 4's figure,
# since averaging layer4's maps is exactly what avgpool does. The last
# stage, closest to ImageNet's classes, transfers less than the one below.

layer1 64 0.6900
layer2 128 0.7933
layer3 256 0.8619
layer4 512 0.8348

Additional Resources

Optional, and nothing later depends on either. Worth opening for why the residual connection makes deep networks trainable, or for the dataset's own account of where its tiles come from.

• YouTube - "C4W2L04 Why ResNets Work" by DeepLearningAI, Andrew Ng's course lecture. Why a residual block can learn the identity easily, so adding blocks rarely hurts, and how the shortcut's shapes are matched when a block changes them.
• GitHub - phelber/EuroSAT, README by Patrick Helber and colleagues, MIT license. The dataset's origin in Sentinel-2 imagery, its 27,000 labeled tiles in ten classes, the RGB version this lesson loads beside the full set of 13 spectral bands, and the paper it accompanies.

Summary

A pretrained model is two things that look like one. The architecture is code, eleven million slots in a fixed arrangement, and the weights are a 46.8 MB file of numbers that fill them, with a set of facts attached: what the outputs mean, what the numbers scored, and what an input has to look like before the first filter reads it. Everything that goes wrong silently around a pretrained network goes wrong in the gap between those two, a shape built without its numbers, or numbers handed an input on the wrong scale.

Reading the network turned out to be reading where the decisions sit. Most of the parameters are in the last stage, every block adds its input back, and the final layer is the one place where ImageNet is named outright, a thousand questions about fish and parachutes asked of every input. On satellite tiles that layer is useless, answering sharks and parachutes for fields and lakes, and switching the network into training mode for a single batch rewrote 99 of 100 of those answers, a reminder that a pretrained model's behavior lives in its running statistics as well as its weights. The layers under that last one are not useless: a classifier with nothing trained called most tiles right from their averaged features, and the stage below the last did better than the last.

So the pretrained network is not a model of the new task. It is a source of features of varying generality, and the work of the next two lessons is choosing how much of it to keep fixed, how much to retrain, and at what rate, with each choice measured on the 2,700 tiles held for choosing.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

weights = ResNet18_Weights.IMAGENET1K_V1
model = models.resnet18(weights=weights)
prep = weights.transforms(crop_size=160, resize_size=160)
model.eval()  # BatchNorm on running stats
with torch.no_grad():
    probs = model(x).softmax(dim=1)

block = model.layer2[0]
path = block.bn2(block.conv2(block.relu(block.bn1(block.conv1(x)))))
# equal to block(x)
out = torch.relu(path + block.downsample(x))
model.fc = nn.Identity()  # 512 values per image