Building the CNN - Beat Your MLP

Context

A convolutional network (a CNN, a network whose first layers are convolutions) is a stack of conv blocks that turns an image into a few small maps, and a head (the layers after the last convolution that turn its maps into the answer) that reads those maps into ten logits. It trains through the same loop, loss and optimizer as the fully connected network did, one batch at a time.

Image classifiers in real work are built this way: the model that sorts product photos into a catalog's categories, the one that flags a defect on a production line, the one that sorts scans before a person reads them. Many pretrained vision models are built from the same parts, many layers deeper, and reading one of them, choosing where to cut it, and comparing it with something simpler all start from a small one trained and measured by hand.

Lesson 4.5.3 ended on a question it did not answer: whether a network built from filters keeps its score when the test images move, against the fully connected network's 0.8677 in place and 0.4554 two pixels over. Answered by assertion, a CNN wins because convolution is the better idea. Answered by a run on the same images, loop and epochs, it wins by a measured amount, and the same run shows where its tolerance for movement stops, which no assertion would.

Goal

• Build cnn() in cnn.py from two of Lesson 5.1.2's conv blocks and a linear head, trace a batch through every layer, and compute the head's 1,568 inputs from the arithmetic before the layer refuses a wrong one.
• Train it for five epochs with image_loop.py and Adam at 0.001, keep the best epoch with Lesson 4.4.2's copy.deepcopy, and set its validation accuracy, seconds and parameters beside the fully connected network's.
• Score the test set once and read the per-class accuracies and the largest confusions beside the fully connected network's.
• Run Lesson 4.5.3's shift table on both networks, and locate where the CNN's tolerance ends, in its body or its head, from a four-pixel move.
• Draw the 16 learned first-layer filters and measure what each is worth by silencing it.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want a small network's shapes and parameter counts laid out in a table first, or the same kind of network trained on the same images in another book's code.

• YouTube - "C4W1L10 CNN Example" by DeepLearningAI, Andrew Ng's course lecture. A network in the style of LeNet-5 on a 32 by 32 by 3 image, two convolution and pooling layers then fully connected layers, with every layer's output shape and parameter count tabulated.
• Book chapter - Dive into Deep Learning, "Convolutional Neural Networks (LeNet)" by Zhang, Lipton, Li and Smola, under CC BY-SA 4.0. LeNet built in PyTorch layer by layer, its shape printed through each layer, and trained on Fashion-MNIST.

Table of Contents

• The Network, Shape by Shape
• Five Epochs Through the Same Loop
• The Test Set, Scored Once
• The Shift Test, Again
• What the First Filters Learned

Theory

Every Theory block runs in ~/dev/bootcamp, beside fashion.py, Lesson 4.5.1's, image_loop.py, Lesson 4.5.2's, trained.py and the two models it loads, Lesson 4.5.3's, and blocks.py, Lesson 5.1.2's. The first subtitle saves cnn.py, and the second trains the network and saves it as fashion_cnn.pt, which every later subtitle and Lesson 5.1.4 load instead of training again. The trained figures are this machine's on the eight threads torch 2.14.1 uses here by default, as Lesson 4.5.2's were: the same five epochs on one thread printed 0.8652 after epoch 1 where eight printed 0.8677, and kept the same best epoch at 0.9076 against 0.9077, so your last digits can differ. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

The Network, Shape by Shape

What. cnn() returns nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.Flatten(), nn.Linear(32 * 7 * 7, 10)), seeding 42 inside as Lesson 4.5.2's mlp() does. The two blocks take a (N, 1, 28, 28) batch to (N, 16, 14, 14) and then to (N, 32, 7, 7), nn.Flatten() lays each image's 32 maps of 7 by 7 out as 1,568 values, and the head's one linear layer turns them into ten logits. Why. The blocks find patterns at every position and halve the maps twice, so the head reads 1,568 values about what was found where instead of 784 pixels, and Lesson 5.1.1's arithmetic says exactly how many: 28 halves to 14 and 14 to 7, times 32 channels. The size is a number the head must be built with, so it is computed before the layer is written. When. Every time the network is built, and every time a block, a channel count or the image size changes, since each changes the number the head reads.

Save it as cnn.py in ~/dev/bootcamp:

import torch
from torch import nn
from blocks import conv_block

def cnn():
    torch.manual_seed(42)
    return nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.Flatten(), nn.Linear(32 * 7 * 7, 10))

The network, layer by layer, on one batch:

import torch
from torch import nn
from fashion import train_loader
from blocks import conv_block
from trained import network
from cnn import cnn

model = cnn()
print(model[2:])
x, y = next(iter(train_loader))
h = x
for layer in model:
    h = layer(h)
    print(type(layer).__name__, tuple(h.shape), sum(p.numel() for p in layer.parameters()))
print(sum(p.numel() for p in model.parameters()), sum(p.numel() for p in network.parameters()))

Sequential(
  (2): Flatten(start_dim=1, end_dim=-1)
  (3): Linear(in_features=1568, out_features=10, bias=True)
)
Sequential (64, 16, 14, 14) 176
Sequential (64, 32, 7, 7) 4672
Flatten (64, 1568) 0
Linear (64, 10) 15690
20538 101770

model[2:] is the head as Lesson 4.5.2's network printed its layers, and the two blocks above it print as Lesson 5.1.2's did. type(layer).__name__ is each layer's class name, Sequential for a block. The CNN holds 20,538 parameters, about a fifth of the fully connected network's 101,770, and 15,690 of them, 76 percent, sit in the head: 1,568 weights for each of ten classes and ten biases. The blocks hold 4,848. A head built for the wrong size:

wrong = nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.Flatten(), nn.Linear(32 * 14 * 14, 10))
try:
    wrong(x)
except RuntimeError as err:
    print(err)

mat1 and mat2 shapes cannot be multiplied (64x1568 and 6272x10)

32 * 14 * 14 counts the maps after one halving instead of two. The layer is built without complaint, and the first batch through it is refused with the message Lesson 4.5.2's unflattened image raised, naming the 1,568 values per image that arrived and the 6,272 the weight was built for. Nothing in the message names a block or a pooling layer, so the number is checked against the arithmetic, 28 // 2 // 2 per side times the last block's channels.

DON'T:

• DON'T read a CNN's parameter count as living in its convolutions. 15,690 of this network's 20,538 sit in the linear head.
• DON'T size the head by guessing. A head built for 6,272 inputs was refused on the first batch with a message about matrices, and 1,568 is 32 * (28 // 2 // 2) ** 2.

Five Epochs Through the Same Loop

What. train_epoch and accuracy from image_loop.py, Lesson 4.5.2's, train and score the CNN with no change, because both take any model: train_epoch takes one Adam step per batch of train_loader, 782 an epoch, and accuracy switches to evaluation mode, scores the validation images and switches back. Lesson 4.4.2's copy.deepcopy keeps the parameters and the BatchNorm buffers of the epoch with the highest validation accuracy, and the network that is saved is that one. Why. Every condition but the network is the fully connected network's: the same 50,000 images in the same shuffles, the same loss, the same optimizer at the same rate and the same five epochs. So whatever separates the two validation accuracies is the network, and the comparison is a measurement, Lesson 4.5.2's linear model against its network again. When. Every time two networks are compared on the validation images, before either is scored on the test set.

import copy
import time
import torch
from torch import optim
from fashion import train_loader, val_loader
from image_loop import accuracy, train_epoch
from cnn import cnn

model = cnn()
optimizer = optim.Adam(model.parameters(), lr=0.001)
print(f"{accuracy(model, val_loader):.4f}")
best_acc, best_epoch, best_state = 0.0, 0, None
for epoch in range(1, 6):
    start = time.perf_counter()
    train_loss = train_epoch(model, optimizer, train_loader)
    seconds = time.perf_counter() - start
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}", f"{seconds:.1f}")
    if val_acc > best_acc:
        best_acc, best_epoch, best_state = val_acc, epoch, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
print(best_epoch, f"{accuracy(model, val_loader):.4f}")
torch.save(model.state_dict(), "fashion_cnn.pt")

0.0090
1 0.4118 0.8677 5.6
2 0.2880 0.8968 5.5
3 0.2548 0.9077 5.4
4 0.2317 0.9028 5.7
5 0.2155 0.8909 6.1
3 0.9077

Before any step the network calls 90 of the 10,000 validation images right, under the one in ten of a guess, a seeded start that predicts nothing. After the first epoch it calls 8,677 right, and its validation accuracy rises to 0.9077 at epoch 3, then falls to 0.9028 and 0.8909 while the training loss keeps falling, from 0.2548 to 0.2155. The fully connected network of Lesson 4.5.2 rose at every epoch to 0.8725 at epoch 5, so its best epoch was its last, and the CNN's is its third: the loop keeps epoch 3's state, load_state_dict restores it, the reloaded network scores the same 0.9077, and fashion_cnn.pt holds that network, its BatchNorm statistics included. The CNN leads by 352 of the 10,000 validation images.

The seconds are this machine's at the moment of the run, and they move with whatever else the machine is doing: the same five epochs printed from about 5 to about 11 seconds an epoch across runs here. Two networks' seconds are compared only when one script times both, which Answer 2 does.

DON'T:

• DON'T save the network that ended the loop. Its validation accuracy fell from 0.9077 at epoch 3 to 0.8909 at epoch 5, and the copy kept at epoch 3 is the one the validation images chose.
• DON'T keep the best state as model.state_dict() without copy.deepcopy. The dict holds the live tensors, Lesson 4.4.2's trap, and the next epoch's steps write over them.

The Test Set, Scored Once

What. The validation images chose the CNN over the fully connected network, 0.9077 against 0.8725, and the test set now scores that choice once: the share of the 10,000 test images fashion_cnn.pt calls right, then the same predictions class by class beside the fully connected network's, Lesson 4.5.2's reading, and the confusion matrix's largest cells. Why. The test images are the only ones no choice has touched, so their score is the estimate of what the chosen network does on images it has never met, Lesson 3.1.2's rule. The per-class view says where the gain came from, which one accuracy cannot. When. Once, after the choice is made, and never as a step in making it.

import torch
from sklearn.metrics import confusion_matrix
from fashion import test_set, test_loader
from trained import network
from cnn import cnn

model = cnn()
print(model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True)))
model.eval()
with torch.no_grad():
    logits = torch.cat([model(x) for x, y in test_loader])
    mlp_logits = torch.cat([network(x) for x, y in test_loader])
labels = test_set.targets
pred, mlp_pred = logits.argmax(dim=1), mlp_logits.argmax(dim=1)
print((pred == labels).sum().item(), f"{(pred == labels).float().mean().item():.4f}")
names = test_set.classes
for k, name in enumerate(names):
    mine = labels == k
    print(name, f"{(pred[mine] == k).float().mean().item():.3f}", f"{(mlp_pred[mine] == k).float().mean().item():.3f}")

<All keys matched successfully>
9008 0.9008
T-shirt/top 0.865 0.814
Trouser 0.981 0.969
Pullover 0.839 0.834
Dress 0.903 0.849
Coat 0.849 0.776
Sandal 0.967 0.956
Shirt 0.699 0.623
Sneaker 0.969 0.927
Bag 0.969 0.970
Ankle boot 0.967 0.959

model.eval() comes before the logits because the network holds BatchNorm, Lesson 5.1.2's reason, and network from trained.py is already in evaluation mode. The CNN calls 9,008 of the 10,000 test images right, 0.9008, against the fully connected network's 0.8677, a little under its own validation 0.9077 as that network's was under its 0.8725. Nine of the ten classes rise. Shirt rises most, from 0.623 to 0.699, then Coat, from 0.776 to 0.849, and Bag stays where it was, 0.969 against 0.970. Shirt is still the worst class by far. Where the mistakes go now, and how the two networks' mistakes overlap:

matrix = confusion_matrix(labels, pred)
off = [(matrix[i, j], names[i], names[j]) for i in range(10) for j in range(10) if i != j]
for count, truth, called in sorted(off, reverse=True)[:4]:
    print(count, truth, "called", called)
gained = ((pred == labels) & (mlp_pred != labels)).sum().item()
lost = ((pred != labels) & (mlp_pred == labels)).sum().item()
print(gained, lost, gained - lost)

123 Shirt called T-shirt/top
99 T-shirt/top called Shirt
77 Shirt called Coat
71 Shirt called Pullover
584 253 331

The largest confusions are still among the tops, Lesson 4.5.2's finding, and shirts are in all four, but the 151 shirts called pullovers are down to 71, and the 133 coats called pullovers are no longer among the four largest. The CNN gets 584 test images right that the fully connected network gets wrong, and gets 253 wrong that it gets right: the 331 between them are the gain, made of two sets of mistakes that differ in hundreds of images, not of one network's mistakes with some removed.

DON'T:

• DON'T score the test set for each network and pick the higher. The validation images chose the CNN at 0.9077 against 0.8725, and the test set's 0.9008 is the estimate of that choice.
• DON'T read the gain as the CNN fixing the fully connected network's mistakes. It fixed 584 and made 253 of its own.

The Shift Test, Again

What. Lesson 4.5.3's table, the test images rolled right by 0 to 8 pixels and down by 1, 2 and 5, and its black-filled slide, run on the CNN beside the fully connected network. Then the CNN cut in two at the head: its body (the two blocks, everything before the flatten), model[:2], read on the test images and on the test images moved four pixels, to ask whether its maps moved with the garments. Why. A filter's maps move with the shirt, Lesson 4.5.3 showed, and a CNN is filters, so it should lose less than the fully connected network when the images move. How much less, and where the loss that remains comes from, is the answer Module 4 left open, and it decides what Lesson 5.1.4 has to fix. When. Before a CNN's test score is read as what it knows, exactly as before a fully connected network's.

import torch
from fashion import test_loader
from image_loop import accuracy
from trained import network
from cnn import cnn

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
batches = list(test_loader)
for pixels in (0, 1, 2, 3, 4, 5, 8):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print(pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(network, moved):.4f}")

0 0.9008 0.8677
1 0.8752 0.7896
2 0.7795 0.4554
3 0.5196 0.2242
4 0.2861 0.1393
5 0.2303 0.1293
8 0.1385 0.1086

accuracy puts the CNN in evaluation mode on every call, so loading it is the only setup. The first column is the CNN, the second the fully connected network's row of Lesson 4.5.3's table, with 4 pixels added. One pixel costs the CNN 0.0256 where it cost the fully connected network 0.0781, and two pixels leave it at 0.7795 where the other fell to 0.4554. From three pixels it falls fast too: 0.5196 at three, 0.2861 at four, and 0.1385 at eight, three points above the other's 0.1086. Without the wrap-around, and down:

def slide(x, pixels):
    out = torch.zeros_like(x)
    out[..., pixels:] = x[..., :-pixels]
    return out

for pixels in (2, 4):
    moved = [(slide(x, pixels), y) for x, y in batches]
    print("slide", pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(network, moved):.4f}")
for pixels in (1, 2, 5):
    moved = [(torch.roll(x, shifts=pixels, dims=2), y) for x, y in batches]
    print("down", pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(network, moved):.4f}")

slide 2 0.8058 0.5456
slide 4 0.3505 0.2284
down 1 0.8742 0.8020
down 2 0.8275 0.6438
down 5 0.5690 0.3988

slide is Lesson 4.5.3's shift with black brought in, and with it two pixels right cost the CNN 0.0950, from 0.9008 to 0.8058, where the fully connected network lost 0.3221. Four pixels still take the CNN to 0.3505. Down costs both networks less than right at every distance, as it did in Lesson 4.5.3. Module 4's question has its answer: a network built from filters keeps far more of its score than one built from a weight per pixel, and it does not keep its score. Where the loss comes from, from the body's maps for the images in place and moved four pixels:

model.eval()
body = model[:2]
images = torch.cat([x for x, y in batches])
with torch.no_grad():
    maps = body(images)
    for pixels in (4, 2):
        maps_moved = body(slide(images, pixels))
        # the moved maps one column over, and in place
        over = (maps_moved[..., 1:] - maps[..., :-1]).abs().mean().item()
        print(pixels, round(over, 4), round((maps_moved - maps).abs().mean().item(), 4))
print(maps.shape)

4 0.0419 0.5063
2 0.2952 0.291
torch.Size([10000, 32, 7, 7])

The body's output is 32 maps of 7 by 7 per image, and each column of those maps stands for 4 columns of pixels, two halvings of 2. Moved four pixels, the images' maps differ from the maps in place by 0.5063 on average, and compared one column over, the column the move should carry each value to, they differ by 0.0419, under a tenth as much: the body's maps moved with the garments, one column for four pixels, very nearly unchanged. Moved two pixels, half a column, the maps neither stay nor move by a column, and both comparisons differ by about 0.29. The head is an nn.Linear on the 1,568 flattened values, one weight per position for each class, so maps that moved one column meet other weights, Lesson 4.5.3's first finding one layer later: the body carried the garments over and the head scored them 0.3505. Pooling forgave part of a move of under a column, Lesson 5.1.2's share, and the head forgives none.

DON'T:

• DON'T read a CNN's test score as tolerance for moved images. It scored 0.9008 in place and 0.2861 four pixels right.
• DON'T blame the convolutions for the CNN's fall. Its body's maps moved one column for four pixels and differed by 0.0419 there, and the linear head reading them by position took the score to 0.3505.

What the First Filters Learned

What. The first block's convolution holds 16 filters of 3 by 3, model[0][0].weight of shape (16, 1, 3, 3), set by five epochs of the loop from seeded starts. Drawn as sixteen small grids, compared with Lesson 4.5.3's vertical edge filter, and silenced one at a time, their channels zeroed after the first block, they show what the loss made of them and which ones the network leans on. Why. Nobody wrote these filters, so what they hold is found by looking and measured by removing. A drawing says what a filter looks like, and silencing says what it is worth to the score, which a drawing cannot. When. Whenever a trained network has to be explained, checked for a dead or a dominant filter, or compared with an intuition about edges.

import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F
from fashion import test_set, test_loader
from cnn import cnn

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
model.eval()
w = model[0][0].weight.detach()
print(w.shape, round(w.min().item(), 4), round(w.max().item(), 4))
fig, axes = plt.subplots(2, 8, figsize=(12, 3))
limit = w.abs().max().item()
for k, ax in enumerate(axes.flat):
    ax.imshow(w[k, 0], cmap="coolwarm", vmin=-limit, vmax=limit)
    ax.set_title(str(k))
    ax.axis("off")
fig.savefig("filters.png", dpi=150, bbox_inches="tight")
print(len(fig.axes), axes[0, 0].images[0].get_array().shape)
plt.close(fig)

torch.Size([16, 1, 3, 3]) -0.5511 0.3864
16 (3, 3)

axes.flat walks the 2 by 8 grid of Axes in reading order, so filter k lands in panel k, and ax.axis("off") hides the ticks a 3 by 3 image does not need. Every panel shares one color scale centered on zero, its ends set by the largest weight in size, so a weak filter looks weak, and the weights run from -0.5511 to 0.3864. How close each filter is to a hand-written edge:

vertical = torch.tensor([[-1.0, 0.0, 1.0]] * 3).flatten()
similar = F.cosine_similarity(w.view(16, 9), vertical.expand(16, 9), dim=1)
print(similar.round(decimals=2))
print(similar.abs().argmax().item())

tensor([-0.0400,  0.1100,  0.4600, -0.0200,  0.1200, -0.0600, -0.1400, -0.0800,
        -0.3500,  0.3700, -0.5300,  0.1400,  0.3000,  0.0700,  0.4300, -0.2600])
10

F.cosine_similarity (the dot product of two vectors divided by both their lengths, 1 for the same direction, -1 for opposite, 0 for unrelated) compares each filter's nine weights with the vertical edge filter's nine, expand repeating the edge filter's row sixteen times without copying it. The closest is filter 10 at -0.53, nearer a left-to-right fall in brightness than a rise, and no filter's similarity reaches 0.6 in either direction: the loss asked for garment classes, and it did not set an edge filter in Lesson 5.1.1's sense. What each filter is worth:

images = torch.cat([x for x, y in test_loader])
labels = test_set.targets
with torch.no_grad():
    first = model[0](images)
    base = (model[1:](first).argmax(dim=1) == labels).float().mean().item()
    drops = []
    for k in range(16):
        cut = first.clone()
        cut[:, k] = 0.0
        acc = (model[1:](cut).argmax(dim=1) == labels).float().mean().item()
        drops.append(round(base - acc, 4))
print(f"{base:.4f}")
print(drops)
print(w[7, 0].round(decimals=2))

0.9008
[0.0151, 0.0014, 0.0044, 0.0047, 0.0106, 0.0036, 0.0161, 0.0909, 0.0242, 0.0072, 0.0368, 0.0085, 0.002, 0.0118, 0.0326, 0.0056]
tensor([[-0.0900,  0.0700, -0.0300],
        [ 0.3400, -0.1900, -0.1800],
        [-0.2500,  0.2800,  0.0800]])

model[0] is the first block and model[1:] everything after it, so zeroing channel k of the first block's output is the network with filter k silenced and every other weight untouched. Every filter is worth something, and the worth is uneven: silencing filter 7 costs 0.0909 of the test accuracy, more than twice the next, filter 10's 0.0368, and silencing filter 1 costs 0.0014. Filter 7's grid, the one the network leans on most, holds weights from -0.25 to 0.34 in no pattern a person would name. One shirt through the first block, before its pooling:

shirt = test_set[7][0].unsqueeze(0)
with torch.no_grad():
    maps = model[0][:3](shirt)
    all_maps = model[0][:3](images)
print(test_set.classes[test_set[7][1]], maps.shape)
print((all_maps > 0).float().mean(dim=(0, 2, 3)).round(decimals=2))
fig, axes = plt.subplots(2, 8, figsize=(12, 3))
for k, ax in enumerate(axes.flat):
    ax.imshow(maps[0, k], cmap="gray")
    ax.set_title(str(k))
    ax.axis("off")
fig.savefig("shirt_maps.png", dpi=150, bbox_inches="tight")
print(len(fig.axes), axes[0, 0].images[0].get_array().shape)
plt.close(fig)

Shirt torch.Size([1, 16, 28, 28])
tensor([0.3900, 0.3800, 0.5500, 0.6200, 0.3700, 0.5600, 0.3000, 0.3300, 0.6200,
        0.5700, 0.3000, 0.5600, 0.5700, 0.5600, 0.3500, 0.5500])
16 (28, 28)

model[0][:3] is the first block's convolution, BatchNorm and ReLU, its pooling left off, so the shirt's 16 maps are 28 by 28 and shirt_maps.png shows each filter's response at every pixel of test image 7. Over all 10,000 test images every channel is above zero at between 0.30 and 0.62 of its positions: no filter is dead, where two of Lesson 5.1.1's seeded filters wrote zero over nearly all of the boot.

DON'T:

• DON'T read a filter's worth off its drawing. Filter 7 and filter 1 are both small grids of weights, and silencing them cost 0.0909 and 0.0014.
• DON'T expect a trained filter to be a hand-written one. The closest of the sixteen to the vertical edge filter stood at -0.53, and none reached 0.6 in either direction.

Lab Exercises

Lab Exercise 1: A Narrower Network, and a Third Block

In small_cnn.py, write small(), which seeds 42 and returns nn.Sequential(conv_block(1, 8), conv_block(8, 16), nn.Flatten(), nn.Linear(16 * 7 * 7, 10)). Trace one batch from train_loader through it, printing each layer's class name, output shape and parameter count, then the total. Add a third block, conv_block(16, 32), after the second while keeping the head at nn.Linear(16 * 7 * 7, 10), and print the error the batch raises. Say in a comment what the head should be built with and why.

Then train small() for three epochs with train_epoch, Adam at 0.001 and the subtitle's best-epoch copy, printing each epoch's training loss, validation accuracy and seconds, then the best epoch and its accuracy. Say in a comment how it compares with the CNN's 0.9077 and the fully connected network's 0.8725, and with its own parameter count beside theirs.

Lab Exercise 2: Down, by Class, and Two Columns Over

In shift_more.py, load fashion_cnn.pt into cnn() and import network from trained. Print the two networks' accuracy on the test images rolled down by 1, 2, 3 and 4 pixels. Then compute the CNN's predictions on the test images in place and rolled two pixels right, through the ten batches of test_loader as the Theory did, and print the three classes whose accuracy falls most, each with its accuracy in place and moved.

Finally, cut the CNN at its body, move the test images eight pixels right with slide, and print the mean difference between the moved images' maps and the maps in place compared two columns over, beside the mean difference compared in place, then the CNN's accuracy on those moved images. Say in a comment what eight pixels are in the body's columns and what the three numbers show.

import copy
import time
import torch
from torch import nn, optim
from fashion import train_loader, val_loader
from image_loop import accuracy, train_epoch
from blocks import conv_block

def small():
    torch.manual_seed(42)
    return nn.Sequential(conv_block(1, 8), conv_block(8, 16), nn.Flatten(), nn.Linear(16 * 7 * 7, 10))

x, y = next(iter(train_loader))
model = small()
h = x
for layer in model:
    h = layer(h)
    print(type(layer).__name__, tuple(h.shape), sum(p.numel() for p in layer.parameters()))
print(sum(p.numel() for p in model.parameters()))
deeper = nn.Sequential(conv_block(1, 8), conv_block(8, 16), conv_block(16, 32), nn.Flatten(), nn.Linear(16 * 7 * 7, 10))
try:
    deeper(x)
except RuntimeError as err:
    print(err)
# A third block pools 7 to 3 and writes 32 channels, so the head
# reads 32 * 3 * 3 = 288 values and is built as nn.Linear(288, 10).

Sequential (64, 8, 14, 14) 88
Sequential (64, 16, 7, 7) 1184
Flatten (64, 784) 0
Linear (64, 10) 7850
9122
mat1 and mat2 shapes cannot be multiplied (64x288 and 784x10)

model = small()
optimizer = optim.Adam(model.parameters(), lr=0.001)
best_acc, best_epoch, best_state = 0.0, 0, None
for epoch in range(1, 4):
    start = time.perf_counter()
    train_loss = train_epoch(model, optimizer, train_loader)
    seconds = time.perf_counter() - start
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}", f"{seconds:.1f}")
    if val_acc > best_acc:
        best_acc, best_epoch, best_state = val_acc, epoch, copy.deepcopy(model.state_dict())
print(best_epoch, f"{best_acc:.4f}")
# 0.8951 from 9,122 parameters in three epochs: below the CNN's 0.9077
# from 20,538 in five, above the fully connected network's 0.8725
# from 101,770 in five.

1 0.4533 0.8792 5.1
2 0.3215 0.8792 3.9
3 0.2914 0.8951 4.0
3 0.8951

import torch
from fashion import test_set, test_loader
from image_loop import accuracy
from trained import network
from cnn import cnn

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
model.eval()
batches = list(test_loader)
for pixels in (1, 2, 3, 4):
    moved = [(torch.roll(x, shifts=pixels, dims=2), y) for x, y in batches]
    print(pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(network, moved):.4f}")
labels = test_set.targets
model.eval()
with torch.no_grad():
    pred = torch.cat([model(x) for x, y in batches]).argmax(dim=1)
    pred2 = torch.cat([model(torch.roll(x, shifts=2, dims=3)) for x, y in batches]).argmax(dim=1)
rows = []
for k, name in enumerate(test_set.classes):
    mine = labels == k
    before, after = (pred[mine] == k).float().mean().item(), (pred2[mine] == k).float().mean().item()
    rows.append((after - before, name, before, after))
for change, name, before, after in sorted(rows)[:3]:
    print(name, f"{before:.3f}", f"{after:.3f}")

1 0.8742 0.8020
2 0.8275 0.6438
3 0.7730 0.5212
4 0.6700 0.4454
Ankle boot 0.967 0.666
Coat 0.849 0.558
Dress 0.903 0.665

def slide(x, pixels):
    out = torch.zeros_like(x)
    out[..., pixels:] = x[..., :-pixels]
    return out

images = torch.cat([x for x, y in batches])
model.eval()
with torch.no_grad():
    maps = model[:2](images)
    moved = model[:2](slide(images, 8))
print(round((moved[..., 2:] - maps[..., :-2]).abs().mean().item(), 4), round((moved - maps).abs().mean().item(), 4))
print(f"{accuracy(model, [(slide(x, 8), y) for x, y in batches]):.4f}")
# Eight pixels are two of the body's columns. Two columns over, the
# maps differ by 0.0677 against 0.5774 in place: they moved with the
# garments, and the head scored the moved images 0.1804.

0.0677 0.5774
0.1804

Theory Assessment

Q1: A Bigger Photo

A teammate feeds cnn() 32 by 32 images, a new camera's crop, and the first batch is refused. Say what the two blocks hand the flatten for one 32 by 32 image, the size the head must be built with, why the blocks themselves needed no change, and how many parameters the rebuilt network holds against the 20,538 of the original.

Hint: Halve 32 twice, multiply by the last block's channels, and ask which layer of the network was built with a number that depends on the image's size.

Q2: A Fifth of the Parameters, Longer Epochs

The CNN holds 20,538 parameters to the fully connected network's 101,770, and its epochs take longer. A teammate calls that a mistake, since fewer parameters should mean less work. Say what the parameter count measures and what the seconds measure, with the counts Lesson 5.1.2 made, and time one epoch and 100 steps on batches already in memory for each network, in one script, to show it.

Hint: A parameter is stored once and a filter's weights are used at every position. Time both networks in the same run, because seconds from two runs measure the machine at two different moments.

Q3: Two Pixels, Two Answers

Two pixels to the right, the CNN scores 0.7795 rolled and 0.8058 slid, and the fully connected network 0.4554 and 0.5456. A teammate concludes that the CNN is two pixels from being shift-proof and that more pooling layers will finish the job. Say what the four-pixel comparison of the body's maps showed, what the head does with maps that moved, what one more pooling layer would change, and what a head that ignores position would need, without building it.

Hint: Compare 0.0419 with 0.5063 and say what the head's weights are tied to. Then ask what a fourth halving does to the position a feature has in the head's input.

Q4: The Better Epoch on the Wrong Images

A teammate reruns the training, scores all five epochs on the test set, sees the test accuracy is highest at epoch 3 too, and says this confirms the copy kept at epoch 3 and that next time the test set can do the choosing. Say what the validation images and the test images are each for in this lesson, what the five test scores would cost the 0.9008 if they chose the epoch, and why agreeing with the validation choice this once proves nothing about the next.

Hint: Lesson 3.1.2's rule is about what a choice made on a set does to that set's score, and the copy kept at epoch 3 was chosen before the test set was read.

import torch
from torch import nn
from blocks import conv_block

torch.manual_seed(42)
body = nn.Sequential(conv_block(1, 16), conv_block(16, 32))
photo = torch.zeros(1, 1, 32, 32)
print(body(photo).shape, 32 * 8 * 8)
torch.manual_seed(42)
rebuilt = nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.Flatten(), nn.Linear(32 * 8 * 8, 10))
print(rebuilt(photo).shape, sum(p.numel() for p in rebuilt.parameters()))

torch.Size([1, 32, 8, 8]) 2048
torch.Size([1, 10]) 25338

import time
import torch
from torch import optim
from fashion import train_loader
from image_loop import loss_fn, mlp, train_epoch
from cnn import cnn

batches = []
for i, (x, y) in enumerate(train_loader):
    batches.append((x, y))
    if i == 99:
        break
for build in (mlp, cnn):
    model = build()
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    start = time.perf_counter()
    train_epoch(model, optimizer, train_loader)
    epoch = time.perf_counter() - start
    start = time.perf_counter()
    for x, y in batches:
        loss = loss_fn(model(x), y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    print(build.__name__, f"{epoch:.1f}", f"{time.perf_counter() - start:.2f}")

mlp 2.2 0.06
cnn 5.7 0.46

Assignment

Assignment: A Head That Reads No Positions

The shift test placed the CNN's loss in its head, a linear layer with a weight for every position of the body's maps. Write pooled_head.py, which replaces that head with one that reads no positions at all, trains it, and runs the shift table again.

One layer this lesson did not use. nn.AdaptiveAvgPool2d(1) reduces every map of a batch to its mean, whatever the map's size: it takes (N, C, H, W) to (N, C, 1, 1), a global average pooling (each map replaced by the mean of all its positions), so a feature contributes the same to the mean wherever it sits. The 1 is the output size it is asked for, and it chooses its own window to reach it.

Seed 42 and build nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10)). Print the pooling layer and the network's parameter count, and trace one batch through every layer, printing each layer's class name and output shape. Train it for five epochs exactly as the Theory trained the CNN, printing each epoch's training loss, validation accuracy and seconds, keep the best epoch with copy.deepcopy, and print the best epoch and its accuracy. Then load fashion_cnn.pt into cnn() and print both networks' accuracy on the test images rolled right by 0, 1, 2, 3, 4, 5 and 8 pixels, and say in a comment what each network buys and what it pays.

Hints:

• The head after the pooling reads 32 values, one mean per map, so its weight is (10, 32), and the network's parameters are the blocks' 4,848 plus 330.
• The pooled network's validation accuracy is the choice between the two heads on these images, and the shift table on the test images is the subject of a different question, as it was in the Theory.

import copy
import time
import torch
from torch import nn, optim
from fashion import train_loader, val_loader, test_loader
from image_loop import accuracy, train_epoch
from blocks import conv_block
from cnn import cnn

torch.manual_seed(42)
pooled = nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(32, 10))
print(pooled[2], sum(p.numel() for p in pooled.parameters()))
x, y = next(iter(train_loader))
h = x
for layer in pooled:
    h = layer(h)
    print(type(layer).__name__, tuple(h.shape))

AdaptiveAvgPool2d(output_size=1) 5178
Sequential (64, 16, 14, 14)
Sequential (64, 32, 7, 7)
AdaptiveAvgPool2d (64, 32, 1, 1)
Flatten (64, 32)
Linear (64, 10)

optimizer = optim.Adam(pooled.parameters(), lr=0.001)
best_acc, best_epoch, best_state = 0.0, 0, None
for epoch in range(1, 6):
    start = time.perf_counter()
    train_loss = train_epoch(pooled, optimizer, train_loader)
    seconds = time.perf_counter() - start
    val_acc = accuracy(pooled, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}", f"{seconds:.1f}")
    if val_acc > best_acc:
        best_acc, best_epoch, best_state = val_acc, epoch, copy.deepcopy(pooled.state_dict())
pooled.load_state_dict(best_state)
print(best_epoch, f"{best_acc:.4f}")

1 1.2433 0.7268 5.6
2 0.7767 0.7477 5.1
3 0.6572 0.7882 5.2
4 0.5932 0.8001 5.2
5 0.5507 0.7990 5.5
4 0.8001

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
batches = list(test_loader)
for pixels in (0, 1, 2, 3, 4, 5, 8):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print(pixels, f"{accuracy(pooled, moved):.4f}", f"{accuracy(model, moved):.4f}")
# The pooled head keeps its score as the images move, 0.7982 in place
# and 0.7408 at eight pixels, where the CNN falls from 0.9008 to 0.1385.
# It pays in place: 0.8001 on the validation images against 0.9077,
# and a mean per map keeps how strongly a filter fired, never where.

0 0.7982 0.9008
1 0.7861 0.8752
2 0.7674 0.7795
3 0.7521 0.5196
4 0.7604 0.2861
5 0.7472 0.2303
8 0.7408 0.1385

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want to see what the filters of much deeper networks respond to, or a CNN trained on color images in PyTorch's own tutorial.

• Article - "Feature Visualization" by Chris Olah, Alexander Mordvintsev and Ludwig Schubert, Distill, 2017, under CC BY 4.0. Images optimized to excite one neuron, one channel or one layer of a large image network, from edges and textures in early layers to parts and objects in late ones, and the regularization that keeps the images readable.
• Official tutorial - PyTorch, "Training a Classifier". A small CNN trained on CIFAR-10, 32 by 32 color images in ten classes, with three input channels, its loop, its test score and its per-class accuracies.

Summary

The CNN won, by the margin a measurement gives: 0.9077 against 0.8725 on the validation images, 0.9008 against 0.8677 on the test set, with a fifth of the parameters and longer epochs. The gain was not a list of the old network's mistakes crossed off. It was a different set of mistakes, smaller by 331 images, still crowded among the tops, with shirts still the class it understands least.

The shift test is where the comparison taught the most. Filters do move with the garments, and the measurement says so to the column: a four-pixel move came out of the body as one column of the maps, nearly unchanged. And the network still fell to 0.3505, because the last layer reads those maps the way the fully connected network read pixels, one weight per position. Convolution fixed the first layer's problem and left the same problem in the head. Averaging the position away fixed the head's problem and cost a tenth of the score in place, with every map reduced to how strongly its filter fired and nothing about where.

So a network's tolerance for movement is something to build in, not something an architecture grants by name. The body has the right property, the head has the wrong one, and the third way, changing nothing in the network and showing it moved garments during training, is what Lesson 5.1.4 does with the same CNN.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

def cnn():
    torch.manual_seed(42)
    return nn.Sequential(conv_block(1, 16), conv_block(16, 32), nn.Flatten(), nn.Linear(32 * 7 * 7, 10))

for epoch in range(1, 6):
    train_epoch(model, optimizer, train_loader)
    val_acc = accuracy(model, val_loader)  # eval inside
    if val_acc > best_acc:
        best_acc, best_state = val_acc, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
torch.save(model.state_dict(), "fashion_cnn.pt")

first = model[0](images)  # the first block
cut = first.clone()
cut[:, 7] = 0.0           # filter 7 silenced
acc = (model[1:](cut).argmax(dim=1) == labels).float().mean()