Where MLPs Fail - the Case for Convolution

Context

A fully connected network reads an image as 784 numbers in a fixed order, and its first layer gives every position its own weight in every hidden unit, so a shirt moved two pixels to the right reaches it as other numbers on other weights. Convolution (one small grid of weights slid over every position of an image, a weighted sum taken at each) reads the moved shirt with the same few weights that read it in place.

Objects move in the images a deployed model is handed: a product photo framed a little off center, a garment laid lower on the table, a scan cropped by hand. Scoring a classifier on its own test images moved by a few pixels is one loop over torch.roll, and it belongs before any accuracy is reported as what a model knows. Module 5's Section 5.1 builds its networks from convolutions and trains the filters this lesson only applies.

Without that check, 0.8677 on the test set reads as a network that knows clothing. Moved two pixels to the right, the same 10,000 images score 0.4554, with no error, no warning and an answer for every image. Lesson 4.5.2 trained the network on 50,000 images, scored the test set once and saved both of its models, and this lesson loads them, trains nothing and puts a different question to the same test images: what the network does when they move.

Goal

• Read the first layer's (128, 784) weight as one column per pixel, compute a hidden unit's sum by hand, and draw its weights as a 28 by 28 grid.
• Show from a run that reordering the pixels and the weight's columns by one order leaves every prediction unchanged, and state what that proves about neighbors.
• Tabulate the network's accuracy on test images moved right and down with torch.roll, beside the linear model's, and split the move's cost from the wrap-around's with a black-filled shift.
• Trace one shirt through a two-pixel move, its pixels, its prediction and its confidence, and explain why no error, warning or loss names the cause.
• Apply nn.Conv2d(1, 8, kernel_size=3) to one image, compute one output by hand, and show with torch.allclose that its output on the moved image is its output moved, where both hold.
• Count what a fully connected first layer and a convolution cost in parameters and in multiplications as the image grows.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the argument drawn on a small example first, or the exact contract of the function that rolls the images below.

• YouTube - "Neural Networks Part 8: Image Classification with Convolutional Neural Networks (CNNs)" by StatQuest with Josh Starmer. Fifteen minutes, chaptered on its page: an ordinary neural network classifying an image, a filter slid over the image to make a grid of new values, pooling (shrinking that grid by keeping one number from each small block), the result handed to a neural network, and the letter X classified where it was drawn and then shifted.
• Official docs - PyTorch 2.14, torch.roll. The function that rolls this lesson's images: elements pushed past the last position come back at the first, shifts and dims each take a tuple to roll along several axes at once, and with dims left out the tensor is flattened, rolled and restored to its shape.

Table of Contents

• A Weight per Pixel
• Shifting the Image by a Few Pixels
• Why the Network Cannot Know a Shirt Moved
• What Convolution Does Differently
• The Cost in Parameters

Theory

Every Theory block runs in ~/dev/bootcamp, beside fashion.py, Lesson 4.5.1's, image_loop.py, Lesson 4.5.2's, and the two files Lesson 4.5.2 saved, fashion_mlp.pt and fashion_linear.pt. Every figure the two models print below comes from the files Lesson 4.5.2 saved here at eight threads, and models trained at another thread count can print others: the same five epochs on one thread give a network that scores 0.8667 on the test images, not 0.8677. Nothing here trains: the first subtitle writes trained.py, which loads both models, and the scripts after it import it. The test images are scored throughout as the subject of this lesson's question, what the saved network does when they move, never as the judge of a choice: no setting, epoch or model is chosen on them. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

A Weight per Pixel

What. The network's first layer, network[1], is nn.Linear(784, 128), and its weight is a (128, 784) tensor, one row per hidden unit and one column per pixel: the pixel in row r and column c meets column r * 28 + c, the position Lesson 4.5.2's flattening gives it. Each of the 128 numbers of an image's learned representation, Lesson 4.3.1's, is one row's weighted sum of the 784 pixels in that order, plus a bias, through the ReLU. Why. A weight belongs to a column and a column to a position, so what the first layer knows of a pixel is its index and nothing else. Lesson 4.5.2's flattening forgot which pixel sat beside which before the first weight was applied, and no weight can put it back. The next two subtitles measure what that costs. When. Every time a fully connected layer reads an image, as both of Lesson 4.5.2's models do, and whenever a prediction has to be explained by what the first layer can and cannot see.

Save it as trained.py, the two models built and loaded once for every later script:

import torch
from torch import nn
from fashion import test_loader
from image_loop import mlp, accuracy

network = mlp()
network.load_state_dict(torch.load("fashion_mlp.pt", weights_only=True))
network.eval()

linear = nn.Sequential(nn.Flatten(), nn.Linear(784, 10))
linear.load_state_dict(torch.load("fashion_linear.pt", weights_only=True))
linear.eval()

if __name__ == "__main__":
    # 0.1173
    print(f"{accuracy(mlp(), test_loader):.4f}")
    # 0.8677 0.8372
    print(f"{accuracy(network, test_loader):.4f}", f"{accuracy(linear, test_loader):.4f}")

python3 trained.py runs the two lines under its main guard, and an import trained runs the rest in silence. The file holds the numbers and the script the shape, Lesson 4.4.3's division of the work: mlp(), Lesson 4.5.2's constructor, seeds 42 and builds the network as it stood before its first step, which scores 0.1173, and load_state_dict copies the trained numbers over every parameter, after which the same shape scores 0.8677. The linear model is built bare, since every number it starts with is overwritten. The first layer, from a second script:

import matplotlib.pyplot as plt
import torch
from fashion import test_set, test_loader
from image_loop import mlp, accuracy
from trained import network

first = network[1]
print(first)
print(first.weight.shape, first.bias.shape)
shirt, label = test_set[7]
print(shirt.shape, test_set.classes[label])
flat = shirt.flatten()

Linear(in_features=784, out_features=128, bias=True)
torch.Size([128, 784]) torch.Size([128])
torch.Size([1, 28, 28]) Shirt

Test image 7 is a shirt, and the shirt's pixel in row 10, column 12 meets column 292 of every hidden unit's weights, 10 * 28 + 12. One hidden unit's sum, from the layer and by hand:

with torch.no_grad():
    sums = first(flat)  # all 128 units at once
# row 0 of the weight times the 784 pixels, plus bias 0
unit0 = first.weight[0] @ flat + first.bias[0]
# torch.Size([128]) 0.5781 0.5781
print(sums.shape, round(sums[0].item(), 4), round(unit0.item(), 4))

Every one of the 128 sums is built this way, and none of its 784 terms knows which pixel sits beside its own. Unit 0's 784 weights, put back in the image's layout:

grid = first.weight[0].detach().view(28, 28)
# (0, 16) 0.5521
print(divmod(int(grid.argmax()), 28), round(grid.max().item(), 4))

view(28, 28) lays unit 0's weights out in the image's rows, the only place that layout exists: the layer holds them as one row, and the weight at row 0, column 16 multiplies pixel 16 of every image, whatever surrounds it. Reorder the 784 pixels of every test image by one random order, and the weight's 784 columns by the same order:

order = torch.randperm(784, generator=torch.Generator().manual_seed(42))
print(order[:6])
scrambled = mlp()
scrambled.load_state_dict(network.state_dict())
with torch.no_grad():
    scrambled[1].weight.copy_(first.weight[:, order])
batches = list(test_loader)
mixed = [(x.flatten(1)[:, order].view(-1, 1, 28, 28), y) for x, y in batches]
print(f"{accuracy(scrambled, mixed):.4f}", f"{accuracy(network, mixed):.4f}")
with torch.no_grad():
    before = torch.cat([network(x) for x, y in batches])
    after = torch.cat([scrambled(x) for x, y in mixed])
print(torch.allclose(before, after, atol=1e-4))
print(torch.equal(before.argmax(dim=1), after.argmax(dim=1)))

tensor([470, 186, 370, 171, 190,  57])
0.8677 0.1071
True
True

torch.randperm(784) draws from its own generator seeded 42, the same order on every run of torch 2.14.1 on a processor. Each weight of scrambled meets the pixel it met before, so each hidden sum adds the same 784 products in another order, which float32 rounds a little differently and allclose at 1e-4 accepts, and all 10,000 predictions agree: the scrambled network scores 0.8677 on images no person can read. The saved network scores 0.1071 on the same images, a guess's score, because its weights are tied to positions that now hold other pixels. The three images side by side:

fig, axes = plt.subplots(1, 3, figsize=(12, 4))
axes[0].imshow(shirt.squeeze(0), cmap="gray")
axes[0].set_title("test image 7, Shirt")
axes[1].imshow(mixed[0][0][7].squeeze(0), cmap="gray")
axes[1].set_title("the same 784 values, scrambled")
limit = grid.abs().max().item()
# a color scale centered on zero, so equal sizes look equal
weights = axes[2].imshow(grid, cmap="coolwarm", vmin=-limit, vmax=limit)
fig.colorbar(weights, ax=axes[2])  # the scale, a fourth Axes
axes[2].set_title("unit 0's 784 weights")
fig.savefig("weight_per_pixel.png", dpi=150, bbox_inches="tight")
# 4 [(28, 28), (28, 28), (28, 28)]
print(len(fig.axes), [ax.images[0].get_array().shape for ax in axes])
plt.close(fig)

In weight_per_pixel.png the middle panel is noise to a person and, once the columns are reordered to match, the same input to the network as the left one.

DON'T:

• DON'T read the drawn grid as a layout the layer holds. The network with its columns reordered scored the same 0.8677.
• DON'T reorder an image the network reads without reordering its first weight's columns to match. The same 784 values in another order scored 0.1071.

Shifting the Image by a Few Pixels

What. A shift (every pixel of an image moved the same number of positions along one axis, its value unchanged) is the simplest translation (geometry's word for moving a whole figure without turning or resizing it). torch.roll(x, shifts=k, dims=3) moves a batch shaped (N, 1, 28, 28) by k columns to the right, dims=2 moves it down, and a negative k moves it left or up. What roll pushes past the last column it puts back in the first, the wrap-around. Why. A garment photographed a little to one side is the same garment, so a model that knows clothing calls it the same thing. One loop over a few shifts measures whether this one does. When. Before an image model's accuracy is reported as what it knows, and before it reads images framed less carefully than the ones it was tested on.

import torch
from fashion import test_loader
from image_loop import accuracy
from trained import network, linear

toy = torch.arange(6).reshape(1, 1, 2, 3)
print(toy)
print(torch.roll(toy, shifts=1, dims=3))
print(torch.roll(toy, shifts=1, dims=2))
print(torch.equal(torch.roll(toy, shifts=1, dims=1), toy))

tensor([[[[0, 1, 2],
          [3, 4, 5]]]])
tensor([[[[2, 0, 1],
          [5, 3, 4]]]])
tensor([[[[3, 4, 5],
          [0, 1, 2]]]])
True

On a (1, 1, 2, 3) tensor, dims=3 is the width: each row moves one place right, and its last value wraps to the front. dims=2 is the height, two rows here, so one place down swaps them. dims=1 is the channel axis, one channel long, and rolling it returns the tensor unchanged. The test images, read into memory once and moved right by 0 to 8 pixels:

batches = list(test_loader)
for pixels in (0, 1, 2, 3, 5, 8):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print(pixels, f"{accuracy(network, moved):.4f}")

0 0.8677
1 0.7896
2 0.4554
3 0.2242
5 0.1293
8 0.1086

list(test_loader) reads the ten batches of 1,000 once and keeps them, so every shift rolls tensors already in memory, and accuracy, Lesson 4.5.2's, loops over whatever it is handed and counts len(y), so a list of image and label pairs serves as well as a loader. One pixel costs 0.0781, two pixels take the network to 0.4554, about half, and at eight pixels it scores 0.1086, within a point of the 0.1 that Lesson 4.5.1 gave a model answering one class for every image. Down:

for pixels in (1, 2, 5):
    moved = [(torch.roll(x, shifts=pixels, dims=2), y) for x, y in batches]
    print(pixels, f"{accuracy(network, moved):.4f}")

1 0.8020
2 0.6438
5 0.3988

Down costs less than right at every distance printed, 0.6438 against 0.4554 at two pixels, and the table does not say why. The linear model, at two pixels right:

moved = [(torch.roll(x, shifts=2, dims=3), y) for x, y in batches]
# 0.8372 0.4839
print(f"{accuracy(linear, batches):.4f}", f"{accuracy(linear, moved):.4f}")

The linear model, with no hidden layer, falls 0.3533 to the network's 0.4123, so the fall does not need the hidden layer: a weight per pixel with nothing after it falls too. What roll brings in from the edge:

images = torch.cat([x for x, y in batches])
# per image: any lit pixel in its last two columns
edge = (images[..., -2:] > 0).flatten(1).any(dim=1)
# torch.Size([10000, 1, 28, 28]) 4155 False
print(images.shape, edge.sum().item(), edge[7].item())

4,155 of the 10,000 test images reach into their last two columns, test image 7 not among them, and for those 4,155 roll carries a strip of garment from the right edge to the far left, which no camera does. A shift that brings in black instead, the background these images already have:

def slide(x, pixels):
    out = torch.zeros_like(x)  # a black image, same shape
    # the first 28 - pixels columns, written into the last ones
    out[..., pixels:] = x[..., :-pixels]
    return out

print(torch.equal(slide(images[7:8], 2), torch.roll(images[7:8], shifts=2, dims=3)))
for pixels in (2, 8):
    moved = [(slide(x, pixels), y) for x, y in batches]
    print(pixels, f"{accuracy(network, moved):.4f}")

True
2 0.5456
8 0.1498

On test image 7, whose last two columns are black, slide agrees exactly with roll. With black filled in, two pixels score 0.5456: a garment photographed two pixels further right on the same black background costs the network 0.3221, and roll's wrap-around costs 0.0902 on top of that.

DON'T:

• DON'T forget roll's wrap-around. It moved the right edge of 4,155 test images to their left edge, and at two pixels it costs 0.0902 beyond the move itself.
• DON'T roll dims=1 on a batch of images. It is the channel axis, one long here, and the batch comes back unchanged with no error.

Why the Network Cannot Know a Shirt Moved

What. A shirt moved two pixels reaches the network as 784 numbers in which every one of its values sits at a new index, so each meets other weights, every hidden unit adds a different sum, and the output layer reads a different learned representation. Nothing in the network holds the shirt where it was, to compare. Why. The table's fall has the first subtitle's mechanism: a weight belongs to a position, the moved shirt lights other positions, and the network answers for the 784 numbers in front of it, every time with a class and a probability, so nothing in its output marks a moved image. When. Whenever a test score is about to be read as a property of the model alone, rather than of the model and its test images' framing.

import torch
from fashion import test_loader, test_set
from image_loop import loss_fn
from trained import network

shirt = test_set[7][0].unsqueeze(0)
for pixels in range(4):
    with torch.no_grad():
        logits = network(torch.roll(shirt, shifts=pixels, dims=3))
    probs = torch.softmax(logits, dim=1)
    print(pixels, test_set.classes[probs.argmax().item()], round(probs.max().item(), 3))

0 Shirt 0.957
1 Coat 0.445
2 Pullover 0.673
3 Pullover 0.532

Test image 7 is a shirt to the network where it sits, 0.957 of the probability on Shirt, and from one pixel over it is something else, a pullover with 0.673 at two pixels. No exception and no warning, at any of the four. What the move did to the input:

moved = torch.roll(shirt, shifts=2, dims=3)
print((moved != shirt).sum().item())
lit, lit_moved = shirt > 0, moved > 0
# the indices of the columns and the rows holding a lit pixel
columns = lit[0, 0].any(dim=0).nonzero().flatten()
rows = lit[0, 0].any(dim=1).nonzero().flatten()
print(columns.min().item(), columns.max().item(), rows.min().item(), rows.max().item())
print(lit.sum().item(), lit_moved.sum().item(), (lit & lit_moved).sum().item())
newly, dark = lit_moved & ~lit, lit & ~lit_moved
print(newly.sum().item(), newly[..., 24:].sum().item())
print(dark.sum().item(), dark[..., 4:7].sum().item())
# every value, sorted from the smallest to the largest
values = shirt.flatten().sort().values
print(torch.equal(values, moved.flatten().sort().values))

581
4 24 0 27
504 504 427
77 46
77 47
True

The shirt lights columns 4 to 24 over all 28 rows, 504 pixels before the move and after it, and 581 of the 784 numbers the network reads changed: the 427 positions lit both times hold other pixels of the checked pattern, 77 positions are lit only after the move, 46 of them along the new right edge in columns 24 to 26, and 77 went dark, 47 of them along the old left edge in columns 4 to 6. The whole test set, moved two pixels right:

batches = list(test_loader)
images = torch.cat([x for x, y in batches])
labels = torch.cat([y for x, y in batches])
with torch.no_grad():
    logits = network(images)
    logits_moved = network(torch.roll(images, shifts=2, dims=3))
pred, pred_moved = logits.argmax(dim=1), logits_moved.argmax(dim=1)
print(torch.bincount(pred, minlength=10))
print(torch.bincount(pred_moved, minlength=10))

tensor([ 977,  989, 1178,  951, 1001, 1010,  896,  984, 1003, 1011])
tensor([1334,  116, 1612, 1121,  497,  798,  459,  567, 3014,  482])

In place the predictions spread over the ten classes much as the labels do, 1,000 of each by Lesson 4.5.1's count, and moved two pixels 3,014 of them are Bag and 116 are Trouser. How sure the network is, and its loss:

confidence = torch.softmax(logits, dim=1).max(dim=1).values
confidence_moved = torch.softmax(logits_moved, dim=1).max(dim=1).values
wrong = pred_moved != labels
print(round(confidence.mean().item(), 3), round(confidence_moved.mean().item(), 3))
print(wrong.sum().item(), (confidence_moved[wrong] > 0.9).sum().item())
print(round(loss_fn(logits, labels).item(), 4), round(loss_fn(logits_moved, labels).item(), 4))

0.865 0.696
5446 1065
0.3666 2.1086

max(dim=1).values keeps each row's largest probability, the one the predicted class gets. Its mean falls from 0.865 to 0.696, and 1,065 of the 5,446 moved images the network gets wrong put more than 0.9 of their probability on the wrong class. The cross-entropy rises from 0.3666 to 2.1086, a number that exists only because these images carry labels: an image arriving at a deployed model has none, so nothing computes a loss on it, and even here 2.1086 says how wrong and not why. The figure below draws the mechanism: the same weights under the shirt in both positions, and beside them the one 3 by 3 window the next subtitle slides, reading the same nine values at both.

DON'T:

• DON'T read 0.8677 as how well the network knows clothing. It is how well it reads these 10,000 images framed as they are, and two pixels to the right the same images score 0.4554.
• DON'T read a moved image as a damaged one. Every value of the shirt survived the move, the sorted lists are equal, and the network called it a pullover.
• DON'T take a confident answer for a right one. 1,065 of the moved images the network got wrong carry more than 0.9 of their probability on the wrong class.

What Convolution Does Differently

What. nn.Conv2d(1, 8, kernel_size=3) is a convolution layer with eight filters (a filter is one small grid of weights with a bias, also called a kernel, the word kernel_size uses), each 3 by 3. At every position where a 3 by 3 window fits inside the image, each filter multiplies the nine pixels under the window by its nine weights, adds the products and its bias, and writes the sum into its feature map (the grid of sums one filter writes, one per position). It is applied here to one image with the weights torch.manual_seed(42) draws, and never trained. Why. The same nine weights read every position, so a shirt moved two pixels meets the same weights two positions later, and the filter's sums move with the shirt instead of changing. Where the first layer met the moved shirt with other weights, a filter meets it with its own. When. Whenever the input is an image, or anything else whose values mean what they mean beside their neighbors.

import torch
from torch import nn
from fashion import test_set
from trained import network

shirt = test_set[7][0].unsqueeze(0)
torch.manual_seed(42)
conv = nn.Conv2d(1, 8, kernel_size=3)
print(conv)
for name, p in conv.named_parameters():
    print(name, tuple(p.shape), p.numel())
print(round(conv.weight.min().item(), 4), round(conv.weight.max().item(), 4))

Conv2d(1, 8, kernel_size=(3, 3), stride=(1, 1))
weight (8, 1, 3, 3) 72
bias (8,) 8
-0.33 0.3062

The weight is eight grids of one input channel by 3 by 3, and stride=(1, 1) is the step the window moves by, one pixel along each axis, the layer's default. The 72 weights lie between -0.33 and 0.3062, a seeded draw no fit has touched. What the layer takes and returns:

with torch.no_grad():
    maps = conv(shirt)
print(maps.shape)
try:
    conv(shirt[0, 0])
except RuntimeError as err:
    print(err)

torch.Size([1, 8, 26, 26])
Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [28, 28]

A batch of one shirt, (1, 1, 28, 28), comes back as eight feature maps of 26 by 26, because a 3 by 3 window fits at 28 - 3 + 1 = 26 positions along each axis. A (28, 28) image, its channel dropped, is refused with a message naming the two shapes the layer takes, a batch or one image with its channel. One value of filter 0's map, by hand:

window = shirt[0, 0, 10:13, 12:15]
print(window)
by_hand = (conv.weight[0, 0] * window).sum() + conv.bias[0]
print(round(maps[0, 0, 10, 12].item(), 4), round(by_hand.item(), 4))
flipped = (conv.weight[0, 0].flip(0, 1) * window).sum() + conv.bias[0]
print(round(flipped.item(), 4))

tensor([[0.6706, 0.2196, 0.4431],
        [0.6941, 0.2431, 0.4627],
        [0.4118, 0.0196, 0.2314]])
0.7487 0.7487
0.6917

At row 10, column 12, over nine pixels of the shirt's checked pattern, filter 0 writes its nine weights times those nine pixels, summed, plus its bias, 0.7487 from the layer and by hand, with the grid laid on the window as it is. Mathematics defines convolution with the grid flipped along both axes first, flip(0, 1), and that sum is 0.6917: the layer computes the unflipped sum, which mathematics calls a cross-correlation (the same sliding sum without the flip) and deep learning calls convolution. The shirt, moved two pixels right:

moved = torch.roll(shirt, shifts=2, dims=3)
with torch.no_grad():
    maps_moved = conv(moved)
print(torch.equal(moved[0, 0, 10:13, 14:17], window))
print(round(maps_moved[0, 0, 10, 14].item(), 4))
print(torch.allclose(maps_moved[..., 2:], maps[..., :-2]))
print(torch.allclose(maps_moved, maps))

True
0.7487
True
False

Two columns over, at column 14, the moved shirt puts the same nine pixels under the window, and filter 0 writes the same 0.7487 there. What holds at one position holds at every position the two images share: columns 2 to 25 of all eight moved maps equal columns 0 to 23 of the maps in place, within allclose. The maps themselves are not equal: they moved two columns with the shirt and kept their values. Over the whole width:

print(torch.allclose(maps_moved, torch.roll(maps, shifts=2, dims=3)))
# each filter's bias, to broadcast over the first two columns
bias = conv.bias.view(1, 8, 1, 1)
print(torch.allclose(maps_moved[..., :2], bias))

False
True

Over the whole width the rolled map does not match: the windows at columns 0 and 1 of the moved shirt read nothing but black, so those columns hold each filter's bias alone, broadcast over them by Lesson 4.1.2's rule, and roll puts the far edge of the maps in place there, the wrap-around the shift table carried. The first layer, and other weights:

with torch.no_grad():
    sums = network[1](network[0](shirt))
    sums_moved = network[1](network[0](moved))
print(sums.shape, round((sums_moved - sums).abs().max().item(), 4))
for seed in (0, 1, 7):
    torch.manual_seed(seed)
    other = nn.Conv2d(1, 8, kernel_size=3)
    with torch.no_grad():
        same = torch.allclose(other(moved)[..., 2:], other(shirt)[..., :-2])
    print(seed, same)

torch.Size([1, 128]) 5.1624
0 True
1 True
7 True

The first layer's 128 sums for the two positions are 128 numbers with no positions to move along, and the move changed one of them by 5.1624. Three other seeds' filters agree under the same move, so the agreement comes from sliding one grid over the image, not from what the grid holds.

DON'T:

• DON'T take these filters for trained ones. Their 72 weights are a seeded draw no loss has touched.
• DON'T compare a moved map with the map rolled over its whole width. The full comparison printed False where the columns both hold printed True.

The Cost in Parameters

What. A fully connected first layer holds one weight per pixel for every hidden unit, the pixel count times the width, 784 times 128 at 28 by 28, so its count grows with the image. A convolution holds one grid and one bias per filter, 8 times 9 weights and 8 biases, whatever the size of the image it slides over. Why. Every parameter is a number a fit must set from the training images, and a first layer's count grows with the image while a filter's does not. This is the second argument for convolution, beside the first subtitle's: its parameter count belongs to the filter, not to the image. When. Whenever an image grows past 28 by 28, and whenever two models are compared by how many parameters they hold.

import torch
from torch import nn
from trained import network

counts = [p.numel() for p in network.parameters()]
print(counts, sum(counts))
first_count = sum(p.numel() for p in network[1].parameters())
print(first_count, round(first_count / sum(counts), 3))
torch.manual_seed(42)
conv = nn.Conv2d(1, 8, kernel_size=3)
conv_count = sum(p.numel() for p in conv.parameters())
print(conv_count, first_count / conv_count)

[100352, 128, 1280, 10] 101770
100480 0.987
80 1256.0

The first layer holds 100,480 of the network's 101,770 parameters, 98.7 percent, its 100,352 weights and 128 biases. The convolution, the fourth subtitle's seeded layer again, holds 80, so the first layer costs 1,256 times as many. A larger image:

big = torch.zeros(1, 1, 56, 56)
first_big = nn.Linear(56 * 56, 128)
big_count = sum(p.numel() for p in first_big.parameters())
print(first_big.weight.shape, big_count, big_count / conv_count)
with torch.no_grad():
    print(conv(big).shape)

torch.Size([128, 3136]) 401536 5019.2
torch.Size([1, 8, 54, 54])

A 56 by 56 image, black because only its shape matters here, is 3,136 pixels, and a first layer of 128 units on it holds 401,408 weights and 128 biases, four times the weights for four times the pixels. The convolution slides the same 80 numbers over it and writes 54 by 54 maps. What each layer computes per image is a different count:

for side in (28, 56):
    out = side - 2
    print(side, side * side * 128, out * out * 8 * 9)

28 100352 48672
56 401408 209952

The first layer multiplies each of its weights by its pixel once, 100,352 multiplications per 28 by 28 image, and the convolution multiplies its nine weights at each of 26 times 26 positions for each of 8 filters, 48,672. Both counts grow with the image, to 401,408 and 209,952 at 56 by 56: the convolution's arithmetic grows and its parameters do not.

DON'T:

• DON'T read 80 against 100,480 as the convolution doing 1,256 times less work. It multiplies 48,672 times per 28 by 28 image to the first layer's 100,352.
• DON'T read a parameter count as capacity alone. The convolution stores 80 numbers and writes 5,408 per image, eight maps of 26 by 26, where the first layer stores 100,480 and writes 128.

Lab Exercises

Lab Exercise 1: The Linear Model, Read by Column and Moved

In linear_moved.py, import linear from trained and the test set and loader from fashion. Print the shapes of the linear model's weight and bias. Take test image 2, print its true class and the class the model predicts, then its Trouser logit (class index 1) computed by hand from row 1 of the weight and bias 1, beside the model's own. Print the row and column of the largest of the 784 Trouser weights, with its value.

Reorder the weight's columns and every test image's pixels by one order from torch.randperm(784) with a generator seeded 7, and print the reordered model's accuracy on the reordered images beside the saved model's on them. Then print the saved model's accuracy on the test images moved right by 0, 1, 2, 4 and 8 pixels and down by 1, 2 and 4, then moved right by 2 with black filled in. Finish with the class it predicts for test image 2 moved right by 0 to 3 pixels, and say in a comment which of the two models falls further at two pixels and what they share.

Lab Exercise 2: A Sneaker Moved Down, and Sixteen Filters

In sneaker_filters.py, import network from trained and the test set and loader from fashion. Take test image 9, a sneaker, and print the class and probability the network gives it moved down by 0 to 3 pixels. Over all 10,000 test images moved down by 3, print the accuracy on Sandal, Sneaker and Ankle boot in place and moved, and how many of the moved sneakers land in each predicted class.

Seed 42 and build nn.Conv2d(1, 16, kernel_size=5). Print it, its parameters by name, shape and count, and the shape of its output on the sneaker. Compute filter 3's output at row 12, column 9 by hand and print it beside the layer's. Move the sneaker down by 2 and print whether the moved maps equal the maps in place moved down, first on the rows both hold and then over the whole height, and whether the sneaker lights anything in its first two or its last two rows. Finish with one line per image of 28, 56 and 112 pixels a side: the side, the weights of a 128-unit first layer on it, this convolution's parameters and the convolution's multiplications per image, and say in a comment what the last three columns show.

import torch
from torch import nn
from fashion import test_loader, test_set
from image_loop import accuracy
from trained import linear

weight, bias = linear[1].weight, linear[1].bias
print(weight.shape, bias.shape)
trouser, label = test_set[2]
flat = trouser.flatten()
with torch.no_grad():
    logits = linear(trouser.unsqueeze(0))
    by_hand = weight[1] @ flat + bias[1]
print(test_set.classes[label], test_set.classes[logits.argmax().item()])
print(round(by_hand.item(), 4), round(logits[0, 1].item(), 4))
row = weight[1].detach()
print(divmod(int(row.argmax()), 28), round(row.max().item(), 4))

torch.Size([10, 784]) torch.Size([10])
Trouser Trouser
11.0946 11.0946
(0, 20) 0.6411

order = torch.randperm(784, generator=torch.Generator().manual_seed(7))
scrambled = nn.Sequential(nn.Flatten(), nn.Linear(784, 10))
scrambled.load_state_dict(linear.state_dict())
with torch.no_grad():
    scrambled[1].weight.copy_(weight[:, order])
batches = list(test_loader)
mixed = [(x.flatten(1)[:, order].view(-1, 1, 28, 28), y) for x, y in batches]
print(f"{accuracy(scrambled, mixed):.4f}", f"{accuracy(linear, mixed):.4f}")

for dims, shifts in ((3, (0, 1, 2, 4, 8)), (2, (1, 2, 4))):
    for pixels in shifts:
        moved = [(torch.roll(x, shifts=pixels, dims=dims), y) for x, y in batches]
        print(dims, pixels, f"{accuracy(linear, moved):.4f}")

0.8372 0.1003
3 0 0.8372
3 1 0.7487
3 2 0.4839
3 4 0.1899
3 8 0.1117
2 1 0.7744
2 2 0.6681
2 4 0.4829

def slide(x, pixels):
    out = torch.zeros_like(x)
    out[..., pixels:] = x[..., :-pixels]
    return out

moved = [(slide(x, 2), y) for x, y in batches]
print(f"{accuracy(linear, moved):.4f}")
with torch.no_grad():
    for pixels in range(4):
        out = linear(torch.roll(trouser.unsqueeze(0), shifts=pixels, dims=3))
        print(pixels, test_set.classes[out.argmax().item()])
# The network falls further at two pixels, 0.8677 to 0.4554
# against 0.8372 to 0.4839, and both read one weight per pixel,
# so the fall needs no hidden layer.

0.5202
0 Trouser
1 Trouser
2 Shirt
3 Ankle boot

import torch
from torch import nn
from fashion import test_loader, test_set
from trained import network

sneaker = test_set[9][0].unsqueeze(0)
with torch.no_grad():
    for pixels in range(4):
        logits = network(torch.roll(sneaker, shifts=pixels, dims=2))
        probs = torch.softmax(logits, dim=1)
        print(pixels, test_set.classes[probs.argmax().item()], round(probs.max().item(), 3))

batches = list(test_loader)
images = torch.cat([x for x, y in batches])
labels = torch.cat([y for x, y in batches])
with torch.no_grad():
    pred = network(images).argmax(dim=1)
    pred_moved = network(torch.roll(images, shifts=3, dims=2)).argmax(dim=1)
for c in (5, 7, 9):
    rows = labels == c
    before = (pred[rows] == c).float().mean().item()
    after = (pred_moved[rows] == c).float().mean().item()
    print(test_set.classes[c], f"{before:.3f}", f"{after:.3f}")
print(torch.bincount(pred_moved[labels == 7], minlength=10))

0 Sneaker 0.987
1 Sneaker 0.994
2 Sneaker 0.985
3 Sneaker 0.659
Sandal 0.956 0.795
Sneaker 0.927 0.275
Ankle boot 0.959 0.798
tensor([  0,   0,   0,   0,   0, 504,   0, 275, 192,  29])

torch.manual_seed(42)
conv = nn.Conv2d(1, 16, kernel_size=5)
print(conv)
for name, p in conv.named_parameters():
    print(name, tuple(p.shape), p.numel())
moved = torch.roll(sneaker, shifts=2, dims=2)
with torch.no_grad():
    maps = conv(sneaker)
    maps_moved = conv(moved)
print(maps.shape)
window = sneaker[0, 0, 12:17, 9:14]
by_hand = (conv.weight[3, 0] * window).sum() + conv.bias[3]
print(round(by_hand.item(), 4), round(maps[0, 3, 12, 9].item(), 4))
print(torch.allclose(maps_moved[:, :, 2:], maps[:, :, :-2]))
print(torch.allclose(maps_moved, torch.roll(maps, shifts=2, dims=2)))
print((sneaker[0, 0, :2] > 0).any().item(), (sneaker[0, 0, -2:] > 0).any().item())
for side in (28, 56, 112):
    first = nn.Linear(side * side, 128)
    out = side - 4
    params = sum(p.numel() for p in conv.parameters())
    print(side, first.weight.numel(), params, out * out * 16 * 25)
# A 128-unit first layer's weights, which are also its multiplications
# per image, grow with the pixels, the 416 parameters stay put, and the
# convolution's 230,400 multiplications already pass the 100,352.

Conv2d(1, 16, kernel_size=(5, 5), stride=(1, 1))
weight (16, 1, 5, 5) 400
bias (16,) 16
torch.Size([1, 16, 24, 24])
-0.2641 -0.2641
True
True
False False
28 100352 416 230400
56 401408 416 1081600
112 1605632 416 4665600

Theory Assessment

Q1: The Same Score on Scrambled Pictures

The network with its first weight's columns reordered scored 0.8677 on test images whose pixels were reordered the same way, and the network as saved scored 0.1071 on those images. A teammate concludes that the network never learned anything about the shape of clothing, since pictures no person can read earn its full score. Say what each of the two numbers measures, why the reordered network's 10,000 predictions match the saved network's, and what the two results together say the first layer knows about a pixel.

Hint: Write one hidden unit's sum as 784 products of a weight and a pixel, and ask what reordering both lists by the same order does to each product and to their sum. Then ask, for the saved network on the reordered images, which pixel each weight now meets.

Q2: Two Prices for Two Pixels

Two pixels to the right, torch.roll takes the network from 0.8677 to 0.4554, and the same move with black filled in scores 0.5456. A teammate reports that "a two-pixel shift costs 41 points" and that the test set was spent on finding it out. Say what roll did to the 4,155 test images that light a pixel in their last two columns, which of the two scores belongs to a garment photographed two pixels further right on the same black background, how the 0.4123 that roll cost splits between the move and the wrap-around, and whether the table broke Lesson 3.1.2's rule for the test set.

Hint: Compare what slide and roll put in the two columns that come in on the left. For the rule, ask what act it forbids, and whether this lesson performed it.

Q3: Columns 0 and 1

torch.allclose(maps_moved[..., 2:], maps[..., :-2]) printed True, and torch.allclose(maps_moved, torch.roll(maps, shifts=2, dims=3)) printed False, on the same two outputs of the same eight filters. Say which input columns an output column of a 3 by 3 filter reads, why columns 0 and 1 of maps_moved hold each filter's bias and nothing else, which columns of maps the rolled map puts in their place and what each of them read, and why the network's first layer offers no comparison of either kind.

Hint: Lay the window for output column 0 on the input and count the columns it covers, then do the same for output columns 1, 24 and 25, and look up which columns the shirt lights in place and moved.

Q4: A Filter in Front of a Flattened Layer

A teammate builds nn.Sequential(nn.Conv2d(1, 8, kernel_size=3), nn.Flatten(), nn.Linear(8 * 26 * 26, 10)) and argues two things from this lesson: the convolution's output moved with the shirt, so once trained this model will score on moved images what it scores in place, and it is cheap, because its convolution holds 80 parameters against the network's 101,770. Say what nn.Flatten() hands the last layer for the shirt in place and two pixels over, what the last layer then does with it, how many parameters the model holds and where they sit, and what each of the teammate's two arguments gets wrong.

Hint: Count the values in the flattened row, and ask how many of them change when the maps move two columns. Then count the last layer's weights, one per value in that row for each of ten classes.

import torch
from torch import nn
from fashion import test_set
from trained import network, linear

shirt = test_set[7][0].unsqueeze(0)
moved = torch.roll(shirt, shifts=2, dims=3)
torch.manual_seed(42)
model = nn.Sequential(nn.Conv2d(1, 8, kernel_size=3), nn.Flatten(), nn.Linear(8 * 26 * 26, 10))
counts = [p.numel() for p in model.parameters()]
print(counts, sum(counts))
print(sum(p.numel() for p in network.parameters()), sum(p.numel() for p in linear.parameters()))
with torch.no_grad():
    row = model[1](model[0](shirt))
    row_moved = model[1](model[0](moved))
print(row.shape, ((row_moved - row).abs() > 1e-4).sum().item())
print(round(row[0, 272].item(), 4), round(row_moved[0, 274].item(), 4))

[72, 8, 54080, 10] 54170
101770 7850
torch.Size([1, 5408]) 4827
0.7487 0.7487

Assignment

Assignment: An Edge Filter Written by Hand

The filters of this lesson were a seeded draw. Write edges.py, which applies a filter whose nine weights you choose and asks whether what it reads moves with the garment.

Three things this lesson did not use. torch.nn.functional.conv2d(input, weight, padding=0), with torch.nn.functional imported as F by import torch.nn.functional as F, computes what nn.Conv2d computes with the weight passed in as an argument, a tensor of shape (out_channels, in_channels, kernel height, kernel width), instead of held by a layer. padding=1 surrounds the image with one ring of zeros before the window slides, so a 3 by 3 filter returns a 28 by 28 map. And a filter written as numbers: [[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]] adds the three pixels in the column to the right of the window's middle column and subtracts the three to its left, so its sum is large where brightness rises from left to right, a garment's left edge on a black background, and negative where it falls.

Take test image 2, a pair of trousers, as a (1, 1, 28, 28) batch and print its class and shape. Build that filter as a (1, 1, 3, 3) float32 tensor named vertical, and horizontal as its transpose over the last two dimensions, transpose(2, 3), and print the horizontal grid. Apply vertical without padding and print the map's shape with the row, column and value of its largest and of its smallest entries. Move the trousers 3 pixels right with roll, apply vertical again, print the same six numbers, and print whether the moved map equals the map in place moved, on the columns both hold. Apply vertical with padding=1 and print the shape and whether the padded map's inside, every row and column but its outer ring, equals the unpadded map. Copy vertical into an nn.Conv2d(1, 1, kernel_size=3, bias=False), a layer that holds no bias, under torch.no_grad() and print whether the layer's output equals F.conv2d's. Apply horizontal and print its largest and smallest entries' positions and values, then the sum of the absolute values of each map, to two decimals. Draw the trousers, the vertical map and the horizontal map side by side, the two maps on one color scale centered on zero, save the figure as edges.png, and print how many Axes it holds and each image's shape. Finish with the network's class and probability for the trousers in place and moved 3 pixels right.

Hints:

• divmod(int(m.argmax()), m.shape[-1]) turns the flat index argmax returns into a row and a column, and when several entries share the largest value argmax returns the first of them in reading order.
• Take vmin and vmax from the vertical map's largest absolute value and use them for both maps, so the two are drawn on one scale and the weaker map looks weaker.

import matplotlib.pyplot as plt
import torch
import torch.nn.functional as F
from torch import nn
from fashion import test_set
from trained import network

trouser, label = test_set[2]
x = trouser.unsqueeze(0)
print(test_set.classes[label], x.shape)
rows = [[-1.0, 0.0, 1.0], [-1.0, 0.0, 1.0], [-1.0, 0.0, 1.0]]
vertical = torch.tensor(rows).view(1, 1, 3, 3)
horizontal = vertical.transpose(2, 3)
print(horizontal[0, 0])

def peak(m):
    width = m.shape[-1]
    high, low = int(m.argmax()), int(m.argmin())
    return divmod(high, width), round(m.max().item(), 4), divmod(low, width), round(m.min().item(), 4)

edges = F.conv2d(x, vertical)
print(edges.shape, peak(edges))
moved = torch.roll(x, shifts=3, dims=3)
edges_moved = F.conv2d(moved, vertical)
print(peak(edges_moved))
print(torch.allclose(edges_moved[..., 3:], edges[..., :-3]))

Trouser torch.Size([1, 1, 28, 28])
tensor([[-1., -1., -1.],
        [ 0.,  0.,  0.],
        [ 1.,  1.,  1.]])
torch.Size([1, 1, 26, 26]) ((3, 8), 3.0, (1, 18), -3.0)
((3, 11), 3.0, (1, 21), -3.0)
True

padded = F.conv2d(x, vertical, padding=1)
print(padded.shape, torch.allclose(padded[..., 1:-1, 1:-1], edges))
layer = nn.Conv2d(1, 1, kernel_size=3, bias=False)
with torch.no_grad():
    layer.weight.copy_(vertical)
    print(torch.allclose(layer(x), edges))
flat_edges = F.conv2d(x, horizontal)
print(peak(flat_edges))
print(round(edges.abs().sum().item(), 2), round(flat_edges.abs().sum().item(), 2))
with torch.no_grad():
    for pixels in (0, 3):
        probs = torch.softmax(network(torch.roll(x, shifts=pixels, dims=3)), dim=1)
        print(pixels, test_set.classes[probs.argmax().item()], f"{probs.max().item():.4f}")

torch.Size([1, 1, 28, 28]) True
True
((0, 10), 1.2706, (25, 11), -1.3569)
499.65 74.89
0 Trouser 0.9999
3 Ankle boot 0.8560

fig, axes = plt.subplots(1, 3, figsize=(12, 4))
axes[0].imshow(x[0, 0], cmap="gray")
axes[0].set_title("Trouser, test image 2")
limit = edges.abs().max().item()
for ax, m, title in ((axes[1], edges, "vertical edges"), (axes[2], flat_edges, "horizontal edges")):
    ax.imshow(m[0, 0], cmap="coolwarm", vmin=-limit, vmax=limit)
    ax.set_title(title)
fig.savefig("edges.png", dpi=150, bbox_inches="tight")
# 3 [(28, 28), (26, 26), (26, 26)]
print(len(fig.axes), [ax.images[0].get_array().shape for ax in axes])
plt.close(fig)

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want convolution outside deep learning, or every setting the layer of the fourth subtitle takes.

• YouTube - "But what is a convolution?" by 3Blue1Brown. Twenty-three minutes on discrete convolution, from probability to image processing and FFTs in its own summary, chaptered on its page: adding two random variables, moving averages, image processing at 8:32, measuring runtime, polynomial multiplication and speeding the computation up with FFTs (fast Fourier transforms, a quicker route to the same long convolution).
• Official docs - PyTorch 2.14, torch.nn.Conv2d. The layer's full contract: stride, padding, dilation, groups and padding_mode beside the four arguments this lesson used, the input as (N, C, H, W) or without its batch dimension, the output's height and width as a formula of the kernel size, stride, padding and dilation, and the weight's shape and the range its starting values are drawn from.

Summary

The network did not fail at what it was asked to do. It was asked to map 784 numbers in a fixed order to ten classes, and on images framed as its test images are it does that 8,677 times in 10,000. The move showed what the question left out: nothing ever told the network that pixel 292 sits beside pixel 293, and the reordered images proved it could not know. A garment is a shape, a shape is pixels beside pixels, and the first layer had positions without neighbors.

That is why the failure was quiet. Nothing raised, and the network answered every moved image with a class, more than a thousand of its wrong answers with nine tenths of its probability. The 0.8677 was honest about the images it was computed on and silent about every other framing of them, the gap between a score and a property.

Convolution answers both halves with one decision, the same few weights at every position: the sums a filter writes move with the shirt instead of changing, and its parameter count belongs to the filter instead of the image. What this lesson did not do is train one. Eight seeded grids on one shirt prove a property of sliding, not a classifier, and a network built from filters can still end in a layer that reads positions. Whether such a network keeps its score when the images move is a run Module 5 makes against the 0.8677 this network set.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

network = mlp()              # the shape, seeded 42
network.load_state_dict(torch.load("fashion_mlp.pt", weights_only=True))
batches = list(test_loader)  # read once, rolled many times
for pixels in (0, 1, 2, 3, 5, 8):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print(pixels, f"{accuracy(network, moved):.4f}")

torch.manual_seed(42)
conv = nn.Conv2d(1, 8, kernel_size=3)     # 72 weights, 8 biases
maps = conv(shirt)                        # (1, 8, 26, 26)
moved = torch.roll(shirt, shifts=2, dims=3)
after = conv(moved)[..., 2:]              # columns 2 to 25
torch.allclose(after, maps[..., :-2])     # True, the maps moved
edges = F.conv2d(x, vertical, padding=1)  # 28 by 28 out