The Convolution

Context

A convolution layer is a set of small grids of weights, its filters, each slid over every input channel of an image at once, adding everything under it into one value per position. The size of the grid, the step it moves by and the border of zeros around the image decide how many positions there are, and the loop decides what the weights hold.

Every image model of this section is built from these layers, and so are many pretrained vision models. Sizing the layer that follows a stack of convolutions, reading a pretrained model's printout, choosing how fast a network shrinks a photo, or tracing a shape error raised three layers past the line that caused it all come down to the arithmetic of one layer and the way layers combine.

Lesson 4.5.3 applied one untrained layer of eight filters to one channel of one shirt and showed its maps move with the shirt. A network has several such layers, and each fails in its own way when its arithmetic is guessed: a second layer handed the wrong number of channels is refused, a stride that does not fit the image drops its last row without a word, and a deep filter is read as seeing three pixels when it sees fifteen. And the edge filter that lesson's Assignment wrote by hand is the kind of grid a network is never handed, because the loop finds it.

Goal

• Compute a convolution's output side from its input side, kernel size, stride and padding, and confirm the formula against nn.Conv2d for seven settings.
• Show the row and the column a stride leaves unread, and keep a map at the image's size with padding="same".
• Build a layer that reads eight channels, explain its (16, 8, 3, 3) weight and its 1,168 parameters, and compute one of its outputs by hand as a sum over all eight channels.
• Measure a stack's receptive field from a gradient, and predict it from the kernels and the strides.
• Train a filter with nn.MSELoss and Adam until it holds Lesson 4.5.3's vertical edge filter, and read why a small loss is not yet the right weights.

Prerequisites

Neither is required, and the lesson stands without them. They are here if you want the picture of a filter over several channels drawn first, or the same arithmetic in another course's words.

• YouTube - "C4W1L06 Convolutions Over Volumes" by DeepLearningAI, Andrew Ng's course lecture. A color image of 6 by 6 by 3 convolved with one filter of 3 by 3 by 3 into a single 4 by 4 map, then two filters giving two maps stacked into a volume.
• Official course notes - CS231n, Convolutional Neural Networks, the Convolutional Layer. Stanford's notes on a layer's depth, stride and zero-padding, the output size as (W - F + 2P)/S + 1, parameter sharing, and the receptive field of one neuron.

Table of Contents

• Padding and Stride, the Output Size
• Many Channels In, Many Filters Out
• Stacking Filters, the Receptive Field
• A Filter Learned From Data

Theory

Every Theory block runs in ~/dev/bootcamp, beside fashion.py, Lesson 4.5.1's, which reads FashionMNIST from the data folder that lesson downloaded. Nothing here trains a classifier: the last subtitle trains one filter on 512 training images, in seconds. A seeded draw prints what it prints on torch 2.14.1 on a processor. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Padding and Stride, the Output Size

What. Padding (rings of zeros added around the image before the window slides) and stride (the step, in pixels, between two positions of the window) decide, with the kernel size, how many positions the window takes along each axis. On a side of n pixels, a kernel of k, a stride of s and a padding of p give (n + 2p - k) // s + 1 positions, the // being Python's floor division (division rounded down to a whole number). nn.Conv2d takes the two as stride= and padding=, both 1 and 0 by default, the defaults Lesson 4.5.3 used. Why. The size of a layer's output is the size the next layer reads, and a flatten after the last convolution needs it as a number in its nn.Linear. Padding keeps a map from shrinking by k - 1 at every layer, and a stride of 2 halves the map, which halves the work of every layer after it. When. Every time a convolution is added to a network, and every time a shape error names a size nobody computed.

import torch
from torch import nn
from fashion import train_full

torch.manual_seed(42)
boot, label = train_full[0]
image = boot.unsqueeze(0)  # a batch of one
print(image.shape, train_full.classes[label])

def out_size(n, k, s=1, p=0):
    return (n + 2 * p - k) // s + 1

for k, s, p in ((3, 1, 0), (3, 1, 1), (5, 1, 2), (3, 2, 1), (5, 2, 0), (2, 2, 0), (7, 3, 0)):
    conv = nn.Conv2d(1, 4, kernel_size=k, stride=s, padding=p)
    with torch.no_grad():
        side = conv(image).shape[-1]
    print(k, s, p, out_size(28, k, s, p), side)

torch.Size([1, 1, 28, 28]) Ankle boot
3 1 0 26 26
3 1 1 28 28
5 1 2 28 28
3 2 1 14 14
5 2 0 12 12
2 2 0 14 14
7 3 0 8 8

The first training image, the ankle boot Lesson 4.5.1 printed, goes through seven layers, each column of the table the kernel, the stride, the padding, the formula's side and the side the layer returned. The first row is Lesson 4.5.3's 26. A padding of (k - 1) / 2 keeps the side at 28 for an odd kernel, 1 for a 3 by 3 and 2 for a 5 by 5, because the window then fits at every one of the 28 positions with its middle on a pixel. A stride of 2 with that padding gives 14, half. (28 - 5) // 2 + 1 is 23 // 2 + 1, 12, where 23 divided by 2 is 11.5: the floor drops the half, and with it whatever the window would have read in the last position it no longer reaches. What that costs, on a 7 by 7 grid whose last row and last column are lit:

grid = torch.zeros(1, 1, 7, 7)
grid[..., -1, :] = 100.0      # the last row lit
grid[..., :, -1] = 100.0      # the last column lit
summer = nn.Conv2d(1, 1, kernel_size=2, stride=2, bias=False)
with torch.no_grad():
    summer.weight.fill_(1.0)  # adds the four pixels under it
    out = summer(grid)
print(out.shape, out.abs().sum().item())

torch.Size([1, 1, 3, 3]) 0.0

A 2 by 2 window stepping by 2 starts at columns 0, 2 and 4, covers columns 0 to 5, and stops: a fourth start, column 6, would need a column 7 that does not exist. The filter adds every pixel under it with a weight of 1, and its nine outputs add to 0.0, so the row and the column holding 100 everywhere were never read, with no error and no warning. On a FashionMNIST image the last row and column are usually black, and on a photo they are part of the subject. The padding that keeps a side, computed by the layer:

same3 = nn.Conv2d(1, 4, kernel_size=3, padding="same")
same5 = nn.Conv2d(1, 4, kernel_size=5, padding="same")
with torch.no_grad():
    print(same3(image).shape, same5(image).shape)
    explicit = nn.functional.conv2d(image, same5.weight, same5.bias, padding=2)
    print(torch.allclose(same5(image), explicit))
try:
    nn.Conv2d(1, 4, kernel_size=3, stride=2, padding="same")
except ValueError as err:
    print(err)

torch.Size([1, 4, 28, 28]) torch.Size([1, 4, 28, 28])
True
padding='same' is not supported for strided convolutions

padding="same" asks the layer for the padding that returns the input's side, and on a 5 by 5 kernel it computes what padding=2 computes, through Lesson 4.5.3's F.conv2d with the layer's own weight and bias. With a stride above 1 no padding returns the input's side, so the layer refuses at construction, before it has seen an image.

DON'T:

• DON'T trust a stride to read every pixel. A 2 by 2 window stepping by 2 over 7 by 7 summed the lit last row and column to 0.0, because (7 - 2) // 2 + 1 is 3 positions and the fourth never starts.
• DON'T pass padding="same" beside a stride above 1. The layer is refused when it is built, with ValueError, and the padding a halving layer wants is the number, padding=1 for a 3 by 3.

Many Channels In, Many Filters Out

What. A layer's input is (N, C_in, H, W), and its weight is (C_out, C_in, k, k): each of its C_out filters holds one k by k grid for every input channel. At each position a filter multiplies every channel's window by that channel's grid, adds all C_in * k * k products and its one bias, and writes a single value, so C_out filters write C_out maps whatever C_in is. The layer holds C_out * C_in * k * k + C_out parameters. Why. A second layer reads the first layer's maps, eight channels where the image had one, and a color image arrives as three, its red, green and blue. Every filter reads all of them at once, so it can respond to what several filters below found at the same place, a vertical edge beside a bright patch, which no one channel holds. When. Every layer after the first, and the first layer of every model that reads color.

import torch
from torch import nn
from fashion import train_full

image = train_full[0][0].unsqueeze(0)
torch.manual_seed(42)
first = nn.Conv2d(1, 8, kernel_size=3, padding=1)
second = nn.Conv2d(8, 16, kernel_size=3, padding=1)
with torch.no_grad():
    maps = torch.relu(first(image))  # eight 28 by 28 maps
    deeper = second(maps)
print(maps.shape, deeper.shape)
print(second.weight.shape, second.bias.shape, sum(p.numel() for p in second.parameters()))

torch.Size([1, 8, 28, 28]) torch.Size([1, 16, 28, 28])
torch.Size([16, 8, 3, 3]) torch.Size([16]) 1168

The first layer turns the boot into eight maps, the ReLU of Lesson 4.3.1 between the two layers, and the second turns the eight into sixteen. Its weight is sixteen blocks of 8 by 3 by 3, 72 weights each, so the layer holds 16 * 72 + 16, 1,168 parameters. One value of filter 0's map, at row 10, column 12, by hand:

window = maps[0, :, 9:12, 11:14]  # all 8 channels, rows 9 to 11
with torch.no_grad():
    per_channel = (second.weight[0] * window).sum(dim=(1, 2))
    by_hand = per_channel.sum() + second.bias[0]
print(window.shape)
print(per_channel.round(decimals=4))
print(round(deeper[0, 0, 10, 12].item(), 4), round(by_hand.item(), 4))
print((maps[0, 2:4, 9:12, 11:14] == 0).all().item())

torch.Size([8, 3, 3])
tensor([-0.0886, -0.0610,  0.0000,  0.0000, -0.0826, -0.0012, -0.0228,  0.0130])
-0.2111 -0.2111
True

With padding=1, output row 10, column 12 is the window centered on input row 10, column 12, rows 9 to 11 and columns 11 to 13, through all eight channels. second.weight[0] is filter 0's (8, 3, 3) block, so the product with the (8, 3, 3) window pairs each channel's nine pixels with that channel's nine weights, and sum(dim=(1, 2)) adds each channel's nine products, eight partial sums. They add to the layer's -0.2111 with the bias. Two of the eight are exactly zero, because the ReLU left maps 2 and 3 at zero across this window: those two seeded filters write a negative sum over nearly all of the boot, and the ReLU sets every negative sum to zero. The same layer, handed the boot itself:

try:
    second(image)
except RuntimeError as err:
    print(err)

Given groups=1, weight of size [16, 8, 3, 3], expected input[1, 1, 28, 28] to have 8 channels, but got 1 channels instead

The message names the weight, the input's shape and the two channel counts. groups=1 is an argument every layer of this module leaves at its default of 1, under which every filter reads every input channel. A layer that reads color:

color = nn.Conv2d(3, 8, kernel_size=3)
photo = torch.zeros(1, 3, 32, 32)  # a color image's shape
with torch.no_grad():
    print(color.weight.shape, sum(p.numel() for p in color.parameters()), color(photo).shape)

torch.Size([8, 3, 3, 3]) 224 torch.Size([1, 8, 30, 30])

A black 32 by 32 image stands in for a photo here, because only its shape matters: three channels in, each of the eight filters a 3 by 3 by 3 block of 27 weights, 8 * 27 + 8 parameters, and eight maps out. The figure below draws the sum the second block computed by hand.

DON'T:

• DON'T count a layer's maps as its input channels times its output channels. nn.Conv2d(8, 16, kernel_size=3) writes 16 maps, each filter summing over all 8 channels into one.
• DON'T give a layer the image's channel count when it reads another layer's maps. The 8-channel layer refused the one-channel boot with a message naming both counts.

Stacking Filters, the Receptive Field

What. The receptive field of an output value is the square of input pixels it depends on. One 3 by 3 layer gives a field of 3. A second 3 by 3 layer on top reads three neighboring outputs of the first, which together cover 5 pixels, and every further layer adds k - 1 positions of the layer below. A stride multiplies what every later layer adds: a 3 by 3 layer after a stride of 2 steps over pixels two at a time, so it adds 4. In one line, a layer's field is the field below it plus (k - 1) times the product of every stride before it. Why. A 3 by 3 filter on a 28 by 28 garment sees a corner of a sleeve, never a collar or a whole boot. Stacking grows what the later filters see without growing any filter, and strides grow it faster while shrinking the maps every later layer works on. When. Choosing how many layers and strides a network needs for the size of the things in its images, and reading why a deep filter can respond to a part of a garment that no 3 by 3 window holds.

import torch
from torch import nn
from fashion import train_full

image = train_full[0][0].unsqueeze(0)

def field(stack, row, col):
    x = image.clone().requires_grad_(True)
    stack(x)[0, 0, row, col].backward()  # one output value
    lit = x.grad[0, 0] != 0              # the pixels it depends on
    rows = lit.any(dim=1).nonzero().flatten()
    cols = lit.any(dim=0).nonzero().flatten()
    return rows.min().item(), rows.max().item(), cols.min().item(), cols.max().item()

torch.manual_seed(42)
layers = [nn.Conv2d(1, 1, kernel_size=3, padding=1) for _ in range(3)]
for depth in (1, 2, 3):
    print(depth, field(nn.Sequential(*layers[:depth]), 14, 14))

1 (13, 15, 13, 15)
2 (12, 16, 12, 16)
3 (11, 17, 11, 17)

field runs the stack on a copy of the boot that records its graph, calls backward() on one output value, Lesson 4.1.4's call, and reads which pixels received a gradient that is not zero: a pixel the value never read gets exactly zero. The four numbers are the first and last row, then the first and last column. The value at row 14, column 14 reads rows and columns 13 to 15 through one layer, 3 pixels, then 12 to 16, then 11 to 17, 5 and 7. Nothing sits between these layers, because the field is a property of the kernels and the strides, and a ReLU between them can only zero a gradient where a unit is off. With strides:

torch.manual_seed(42)
strided = nn.Sequential(nn.Conv2d(1, 1, 3, stride=2, padding=1), nn.Conv2d(1, 1, 3, padding=1))
print(strided(image).shape, field(strided, 7, 7))
torch.manual_seed(42)
halving = [nn.Conv2d(1, 1, 3, stride=2, padding=1), nn.Conv2d(1, 1, 3, stride=2, padding=1)]
deep = nn.Sequential(*halving, nn.Conv2d(1, 1, 3, padding=1))
print(deep(image).shape, field(deep, 3, 3))

torch.Size([1, 1, 14, 14]) (11, 17, 11, 17)
torch.Size([1, 1, 7, 7]) (5, 19, 5, 19)

After one halving layer the next 3 by 3 adds 2 * 2 to the first layer's 3, a field of 7 from two layers where stride 1 needed three. After two halvings the third layer adds 2 * 4, so 3 + 2 * 2 + 2 * 4 is 15: one value of the 7 by 7 map depends on rows and columns 5 to 19, more than half of the boot's width. What a ReLU between the layers is for:

torch.manual_seed(42)
plain = nn.Conv2d(1, 8, kernel_size=3, padding=1, bias=False)
other = train_full[1][0].unsqueeze(0)
with torch.no_grad():
    print(torch.allclose(plain(image + other), plain(image) + plain(other), atol=1e-6))
two = nn.Sequential(nn.Conv2d(8, 8, 3), nn.Conv2d(8, 8, 3))
one = nn.Conv2d(8, 8, 5)
print(sum(p.numel() for p in two.parameters()), sum(p.numel() for p in one.parameters()))

True
1168 1608

A convolution without its bias is linear: on the boot plus the second training image it writes the sum of what it writes on each, within float32's rounding. So two convolutions with nothing between them are one convolution, Lesson 4.3.1's collapse of two linear layers in another shape, and a network puts a ReLU after every convolution. The count is the other half of the case for stacking: two 3 by 3 layers on eight channels see the same 5 by 5 as one 5 by 5 layer, for 1,168 parameters against 1,608, and with a ReLU between them they compute what one layer cannot.

DON'T:

• DON'T read a deep layer's 3 by 3 kernel as a 3 by 3 view. After two halving layers, a 3 by 3 layer's value depended on 15 by 15 pixels of the boot.
• DON'T stack convolutions with nothing between them and count the depth. Without a bias a convolution is linear, so two in a row compute one convolution, and the ReLU between them is what the second layer is for.

A Filter Learned From Data

What. A convolution's weight is a parameter like nn.Linear's, and the loop of Section 4.2 sets it from data: predict, measure the loss, backward(), step, clear. Here the target is known, the maps Lesson 4.5.3's hand-written vertical edge filter writes on 512 training images, and a layer of one filter, starting from a seeded draw, is trained with nn.MSELoss and Adam to write the same maps. Why. No one writes the filters of a working network. Lesson 5.1.3's loss sets sixteen and then thirty-two of them for one purpose, calling the garment right, and nobody can say in advance which grids that needs. With a known target the result can be checked against the answer, so a learned filter is shown to be an ordinary fit before it is trusted inside a classifier. When. Every time a convolution layer trains, which is every time a network built from them learns.

import torch
import torch.nn.functional as F
from torch import nn, optim
from fashion import train_set

images = torch.stack([train_set[i][0] for i in range(512)])
rows = [[-1.0, 0.0, 1.0], [-1.0, 0.0, 1.0], [-1.0, 0.0, 1.0]]
vertical = torch.tensor(rows).view(1, 1, 3, 3)
target = F.conv2d(images, vertical)  # what the learner must write
print(images.shape, target.shape)

torch.manual_seed(42)
learner = nn.Conv2d(1, 1, kernel_size=3, bias=False)
print(learner.weight[0, 0].detach().round(decimals=2))

torch.Size([512, 1, 28, 28]) torch.Size([512, 1, 26, 26])
tensor([[ 0.2500,  0.2800, -0.0800],
        [ 0.3100, -0.0700,  0.0700],
        [-0.1600,  0.2000,  0.2900]])

train_set is Lesson 4.5.1's 50,000 fit images, and torch.stack puts its first 512 into one (512, 1, 28, 28) batch. vertical is Lesson 4.5.3's Assignment filter, the column to the right of the middle minus the column to the left, and target is what it writes on the 512 images, 26 by 26 each. The learner holds nine seeded weights and no bias, since the filter it must become has none. The loop, one step per pass over the whole batch:

loss_fn = nn.MSELoss()
optimizer = optim.Adam(learner.parameters(), lr=0.05)
for step in range(1000):
    loss = loss_fn(learner(images), target)
    if step in (0, 25, 50, 100, 200):
        print(step, f"{loss.item():.4f}")
    if step == 200:
        print(learner.weight[0, 0].detach().round(decimals=2))
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
print(learner.weight[0, 0].detach().round(decimals=2))
print(torch.allclose(learner.weight, vertical, atol=0.01))

0 0.8888
25 0.0594
50 0.0045
100 0.0006
200 0.0001
tensor([[-1.0300,  0.0300,  0.9800],
        [-0.9300, -0.0700,  1.0200],
        [-1.0500,  0.0400,  1.0000]])
tensor([[-1.,  0.,  1.],
        [-1., -0.,  1.],
        [-1.,  0.,  1.]])
True

nn.MSELoss is Section 4.2's mean squared error, here over every value of every map, and Adam is Lesson 4.3.4's optimizer, here at 0.05, fifty times the 0.001 Section 4.5's networks trained at, a rate nine weights on one fixed batch take here without trouble. The loss falls from 0.8888 to 0.0001 in 200 steps, and at step 200 the grid is already the edge filter's shape, a negative column, a column near zero and a positive column, while single weights are still off by up to 0.07. After 1,000 steps every weight is within 0.01 of the hand-written filter, and allclose with atol=0.01 accepts the whole grid. -0. is a weight a little below zero that rounds to zero and keeps its sign. The fit never saw the filter, only the maps it writes, and it found the filter from them.

DON'T:

• DON'T read a small loss as the right weights. At step 200 the loss printed 0.0001 while one weight still stood at -0.93 for a -1, and the grid settled 800 steps later.
• DON'T compare a learned grid with ==. A learned weight lands near its target, never on it, and allclose with a stated atol says how near counts.

Lab Exercises

Lab Exercise 1: Five Layers on a Color Shape, and a Strided Sum

In shapes.py, write out_size(n, k, s=1, p=0). On a black (1, 3, 32, 32) image, build nn.Conv2d(3, 16, ...) with each of these kernel, stride and padding settings: 5, 1, 2, then 3, 2, 1, then 7, 2, 3, then 4, 2, 1, then 5, 3, 0. For each, print the three settings, the formula's side, the side the layer returns, the weight's shape and the parameter count.

Then build a three-channel image from the ankle boot, train_full[0][0]: the boot, the boot mirrored left to right with flip(2), and the boot transposed with transpose(1, 2), joined with torch.cat into one (1, 3, 28, 28) batch. Seed 42 and build nn.Conv2d(3, 4, kernel_size=3, stride=2, padding=1). Print the output's shape, then compute filter 2's output at row 5, column 6 by hand: work out which input rows and columns that position reads, print the three per-channel sums, and print their total with the bias beside the layer's value.

Lab Exercise 2: A Field of Eleven, and Two Filters at Once

In fields.py, measure the receptive field of nn.Sequential(nn.Conv2d(1, 1, 5, padding=2), nn.Conv2d(1, 1, 3, stride=2, padding=1), nn.Conv2d(1, 1, 3, padding=1)), built after torch.manual_seed(42), at row 7, column 7 of its output, with the subtitle's gradient method, and print the output's shape and the rows and columns. Say in a comment what the formula predicts and why.

Then train a layer of two filters at once, nn.Conv2d(1, 2, kernel_size=3, bias=False) after torch.manual_seed(42), on the first 512 images of train_set. Its target is the maps the vertical edge filter and the horizontal one, vertical.transpose(2, 3), write on them, the two filters stacked into one (2, 1, 3, 3) weight with torch.cat, so the target has two channels. Train for 1,000 steps with nn.MSELoss and Adam at 0.05, print the loss at steps 0, 100 and the last step, print both learned grids, and print whether the weight matches the stacked filters within 0.01.

import torch
from torch import nn
from fashion import train_full

def out_size(n, k, s=1, p=0):
    return (n + 2 * p - k) // s + 1

torch.manual_seed(42)
photo = torch.zeros(1, 3, 32, 32)
for k, s, p in ((5, 1, 2), (3, 2, 1), (7, 2, 3), (4, 2, 1), (5, 3, 0)):
    conv = nn.Conv2d(3, 16, kernel_size=k, stride=s, padding=p)
    with torch.no_grad():
        side = conv(photo).shape[-1]
    count = sum(q.numel() for q in conv.parameters())
    print(k, s, p, out_size(32, k, s, p), side, tuple(conv.weight.shape), count)

5 1 2 32 32 (16, 3, 5, 5) 1216
3 2 1 16 16 (16, 3, 3, 3) 448
7 2 3 16 16 (16, 3, 7, 7) 2368
4 2 1 16 16 (16, 3, 4, 4) 784
5 3 0 10 10 (16, 3, 5, 5) 1216

boot = train_full[0][0]
x = torch.cat([boot, boot.flip(2), boot.transpose(1, 2)]).unsqueeze(0)
print(x.shape)
torch.manual_seed(42)
conv = nn.Conv2d(3, 4, kernel_size=3, stride=2, padding=1)
with torch.no_grad():
    out = conv(x)
    # output (5, 6) reads rows 2*5-1 to 2*5+1 and columns 2*6-1 to 2*6+1
    parts = (conv.weight[2] * x[0, :, 9:12, 11:14]).sum(dim=(1, 2))
    total = parts.sum() + conv.bias[2]
print(out.shape)
print(parts.round(decimals=4), round(total.item(), 4), round(out[0, 2, 5, 6].item(), 4))

torch.Size([1, 3, 28, 28])
torch.Size([1, 4, 14, 14])
tensor([-0.0150, -0.0091, -0.0071]) 0.1612 0.1612

import torch
import torch.nn.functional as F
from torch import nn, optim
from fashion import train_full, train_set

image = train_full[0][0].unsqueeze(0)

def field(stack, row, col):
    x = image.clone().requires_grad_(True)
    stack(x)[0, 0, row, col].backward()
    lit = x.grad[0, 0] != 0
    rows = lit.any(dim=1).nonzero().flatten()
    cols = lit.any(dim=0).nonzero().flatten()
    return rows.min().item(), rows.max().item(), cols.min().item(), cols.max().item()

torch.manual_seed(42)
stack = nn.Sequential(nn.Conv2d(1, 1, 5, padding=2), nn.Conv2d(1, 1, 3, stride=2, padding=1), nn.Conv2d(1, 1, 3, padding=1))
print(stack(image).shape, field(stack, 7, 7))
# 5 from the first layer, + 2 * 1 from the second, + 2 * 2 from the
# third, which steps over the second's halved map: 11, rows 9 to 19.

torch.Size([1, 1, 14, 14]) (9, 19, 9, 19)

images = torch.stack([train_set[i][0] for i in range(512)])
vertical = torch.tensor([[-1.0, 0.0, 1.0]] * 3).view(1, 1, 3, 3)
# two filters of one channel each, (2, 1, 3, 3)
both = torch.cat([vertical, vertical.transpose(2, 3)])
target = F.conv2d(images, both)  # two channels
torch.manual_seed(42)
learner = nn.Conv2d(1, 2, kernel_size=3, bias=False)
loss_fn = nn.MSELoss()
optimizer = optim.Adam(learner.parameters(), lr=0.05)
for step in range(1000):
    loss = loss_fn(learner(images), target)
    if step in (0, 100, 999):
        print(step, f"{loss.item():.4f}")
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
print(learner.weight.detach().round(decimals=2))
print(torch.allclose(learner.weight, both, atol=0.01))

0 0.6762
100 0.0004
999 0.0000
tensor([[[[-1.,  0.,  1.],
          [-1., -0.,  1.],
          [-1.,  0.,  1.]]],


        [[[-1., -1., -1.],
          [ 0., -0.,  0.],
          [ 1.,  1.,  1.]]]])
True

Theory Assessment

Q1: Thirteen Windows of Three

A teammate adds nn.Conv2d(1, 8, kernel_size=3, stride=2) to a FashionMNIST model and says it reads every pixel, because thirteen windows of three rows cover 39 rows, more than the 28 there are. Say how many positions the layer takes along each axis, which input rows the last row of its output reads, whether row 27 is read, and what padding=1 changes, with the run that shows it.

Hint: Write down the first input row of each window, 0, 2, 4 and on, and ask where the thirteenth starts. Then light row 27 alone and add up what a filter of ones writes.

Q2: Five Hundred and Twelve Maps

A layer is built as nn.Conv2d(16, 32, kernel_size=3). A teammate expects it to write 512 maps, one for each pair of an input channel and a filter, and to hold 32 times 9 weights plus 32 biases. Say how many maps it writes, the shape of its weight, how many parameters it holds, and what one of its output values is the sum of.

Hint: Write the weight's shape in the order PyTorch keeps it, output channels first, and multiply out what one filter holds.

Q3: Two Strides in Four Layers

A stack is four 3 by 3 layers with padding=1, the second and the fourth at stride=2. Say what its output's side is on a 28 by 28 image, the receptive field of one output value as the subtitle's formula gives it, layer by layer, and how to check the number without trusting the formula.

Hint: Keep a running product of the strides before each layer, and remember that a layer's own stride affects only the layers above it.

Q4: A Window of One Pixel

A teammate reads nn.Conv2d(8, 4, kernel_size=1) in a pretrained model's printout and calls it a mistake: a window of one pixel sees no neighbors, so it cannot detect anything a convolution is for. Say what the layer computes at each position from the eight maps beneath it, how many parameters it holds, what familiar layer it is equal to, and what it does to the maps that a ReLU after it cannot.

Hint: Write the weight's shape and drop the two dimensions of size 1. Then ask what an nn.Linear(8, 4) would compute if it were handed the eight values one pixel holds across the channels.

import torch
from torch import nn

grid = torch.zeros(1, 1, 28, 28)
grid[..., 27, :] = 1.0  # only row 27 lit
for p in (0, 1):
    summer = nn.Conv2d(1, 1, kernel_size=3, stride=2, padding=p, bias=False)
    with torch.no_grad():
        summer.weight.fill_(1.0)
        out = summer(grid)
    print(p, out.shape, out.sum().item())

0 torch.Size([1, 1, 13, 13]) 0.0
1 torch.Size([1, 1, 14, 14]) 41.0

import torch
from torch import nn
from fashion import train_full

image = train_full[0][0].unsqueeze(0).requires_grad_(True)
torch.manual_seed(42)
stack = nn.Sequential(nn.Conv2d(1, 1, 3, padding=1), nn.Conv2d(1, 1, 3, stride=2, padding=1), nn.Conv2d(1, 1, 3, padding=1), nn.Conv2d(1, 1, 3, stride=2, padding=1))
out = stack(image)
out[0, 0, 3, 3].backward()
lit = (image.grad[0, 0] != 0).any(dim=1).nonzero().flatten()
print(out.shape, lit.min().item(), lit.max().item())

torch.Size([1, 1, 7, 7]) 6 18

import torch
from torch import nn
from fashion import train_full

image = train_full[0][0].unsqueeze(0)
torch.manual_seed(42)
first = nn.Conv2d(1, 8, kernel_size=3, padding=1)
mix = nn.Conv2d(8, 4, kernel_size=1)
lin = nn.Linear(8, 4)
with torch.no_grad():
    maps = torch.relu(first(image))
    lin.weight.copy_(mix.weight.view(4, 8))
    lin.bias.copy_(mix.bias)
    # channels last, so each pixel's 8 values are a row for the Linear
    per_pixel = lin(maps.permute(0, 2, 3, 1)).permute(0, 3, 1, 2)
    mixed = mix(maps)
print(mix.weight.shape, sum(p.numel() for p in mix.parameters()))
print(mixed.shape, torch.allclose(mixed, per_pixel, atol=1e-6))

torch.Size([4, 8, 1, 1]) 36
torch.Size([1, 4, 28, 28]) True

Assignment

Assignment: A Filter With Gaps

Every filter of this lesson read pixels that touch. Write dilated.py, which builds a filter that reads every second pixel and measures what that buys.

One argument this lesson did not use. nn.Conv2d(..., dilation=d) spreads the kernel's taps d pixels apart instead of 1, so a 3 by 3 kernel at dilation=2 covers a 5 by 5 square with nine weights, reading the pixels at its corners, edge midpoints and middle and skipping the rest. The output size becomes (n + 2p - d * (k - 1) - 1) // s + 1, which at d = 1 is the lesson's formula, and the layer's parameters do not change with d.

Write out_size(n, k, s=1, p=0, d=1) with the dilated formula and check it against nn.Conv2d(1, 1, ...) on the ankle boot for these kernel, stride, padding and dilation settings: 3, 1, 0, 2, then 3, 1, 2, 2, then 3, 1, 4, 4, then 3, 2, 2, 2, then 5, 1, 0, 3, printing the four settings, the formula's side and the layer's. Seed 42 and build nn.Conv2d(1, 1, kernel_size=3, dilation=2), print it and its parameter count, then write its nine weights into a 5 by 5 grid of zeros at every second row and column and print the grid. Show with F.conv2d that an ordinary 5 by 5 filter holding that grid, with the layer's bias, writes what the dilated layer writes. Finally, seed 42 and build a stack of three 3 by 3 layers at dilations 1, 2 and 4, each padded by its own dilation, and, seeded the same, a stack of three ordinary 3 by 3 layers with padding=1. Print the dilated stack's output shape, the receptive field of both stacks at row 14, column 14, and both stacks' parameter counts, and say in a comment what the dilated stack's field is in the formula's terms.

Hints:

• big[..., ::2, ::2] = layer.weight writes a (1, 1, 3, 3) weight into rows and columns 0, 2 and 4 of a (1, 1, 5, 5) tensor, and it must run under torch.no_grad() because the weight records a graph.
• A dilated layer adds d * (k - 1) to the field below it, the span of its taps less one, so the three layers here add 2, 4 and 8.

import torch
import torch.nn.functional as F
from torch import nn
from fashion import train_full

image = train_full[0][0].unsqueeze(0)

def out_size(n, k, s=1, p=0, d=1):
    return (n + 2 * p - d * (k - 1) - 1) // s + 1

torch.manual_seed(42)
for k, s, p, d in ((3, 1, 0, 2), (3, 1, 2, 2), (3, 1, 4, 4), (3, 2, 2, 2), (5, 1, 0, 3)):
    conv = nn.Conv2d(1, 1, kernel_size=k, stride=s, padding=p, dilation=d)
    with torch.no_grad():
        side = conv(image).shape[-1]
    print(k, s, p, d, out_size(28, k, s, p, d), side)

3 1 0 2 24 24
3 1 2 2 28 28
3 1 4 4 28 28
3 2 2 2 14 14
5 1 0 3 16 16

torch.manual_seed(42)
dilated = nn.Conv2d(1, 1, kernel_size=3, dilation=2)
print(dilated, sum(p.numel() for p in dilated.parameters()))
big = torch.zeros(1, 1, 5, 5)
with torch.no_grad():
    big[..., ::2, ::2] = dilated.weight
    same = torch.allclose(dilated(image), F.conv2d(image, big, dilated.bias), atol=1e-6)
print(big[0, 0])
print(same)

Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1), dilation=(2, 2)) 10
tensor([[ 0.2548,  0.0000,  0.2767,  0.0000, -0.0781],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
        [ 0.3062,  0.0000, -0.0730,  0.0000,  0.0673],
        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],
        [-0.1623,  0.0000,  0.1958,  0.0000,  0.2938]])
True

def field(stack, row, col):
    x = image.clone().requires_grad_(True)
    stack(x)[0, 0, row, col].backward()
    lit = x.grad[0, 0] != 0
    rows = lit.any(dim=1).nonzero().flatten()
    cols = lit.any(dim=0).nonzero().flatten()
    return rows.min().item(), rows.max().item(), cols.min().item(), cols.max().item()

torch.manual_seed(42)
spread = nn.Sequential(*[nn.Conv2d(1, 1, 3, padding=d, dilation=d) for d in (1, 2, 4)])
torch.manual_seed(42)
plain = nn.Sequential(*[nn.Conv2d(1, 1, 3, padding=1) for _ in range(3)])
print(spread(image).shape, field(spread, 14, 14), field(plain, 14, 14))
print(sum(p.numel() for p in spread.parameters()), sum(p.numel() for p in plain.parameters()))
# 3 + 2 * 2 + 2 * 4 = 15, rows 7 to 21, against 7 for the plain stack,
# at the same 30 parameters and the same 28 by 28 maps.

torch.Size([1, 1, 28, 28]) (7, 21, 7, 21) (11, 17, 11, 17)
30 30

Additional Resources

Optional, and nothing later depends on either. Both go further than this lesson: the first into computing fields for networks whose paths branch, the second into the shapes a convolution makes when it runs in reverse.

• Article - "Computing Receptive Fields of Convolutional Neural Networks" by Andre Araujo, Wade Norris and Jack Sim, Distill, 2019, under CC BY 4.0. The field and its position for a chain of layers, derived as closed-form expressions, then extended to networks with several paths from the input to the output, as in ResNet or Inception.
• GitHub - vdumoulin/conv_arithmetic, README by Vincent Dumoulin and Francesco Visin, MIT license. Animations of a window sliding with no padding, half padding, full padding, strides and dilation, and of transposed convolution, the layer that grows a map instead of shrinking it, beside the paper the animations come from.

Summary

A convolution layer is small in what it stores and exact in what it computes. Every output it writes is one sum of products, and every shape it returns comes from one floor division. What makes a stack of them hard to reason about is how quietly the arithmetic decides things: the last row a stride never reaches, the eight channels a filter adds into one, the fifteen pixels a 3 by 3 kernel answers to two halvings up. None of it raises an error, so it has to be computed, and the gradient is a way to measure what the formula claims.

The channels change what a filter is. On a grayscale image a filter is a grid, and from the second layer on it is a block that reads every map below it at once, which is why a network can build parts from edges and garments from parts. Depth widens what a filter answers to, a stride widens it faster, and the ReLU between layers is what keeps the depth from being one convolution in disguise.

And the weights are found, never written. The edge filter that took a hand-picked grid in Module 4 came back out of a loop that only saw the maps it writes, to two decimals, after the loss had already said it was done. A classifier's filters are found the same way, against a target nobody can write down, which is where this section goes next.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

def out_size(n, k, s=1, p=0):
    return (n + 2 * p - k) // s + 1      # positions per axis

first = nn.Conv2d(1, 8, 3, padding=1)    # 28 stays 28
second = nn.Conv2d(8, 16, 3, padding=1)  # weight (16, 8, 3, 3)
# 28 to 14
down = nn.Conv2d(16, 16, 3, stride=2, padding=1)
maps = down(torch.relu(second(torch.relu(first(x)))))

x = image.clone().requires_grad_(True)
stack(x)[0, 0, 14, 14].backward()  # one output value
lit = x.grad[0, 0] != 0            # its receptive field
optimizer = optim.Adam(learner.parameters(), lr=0.05)
# a filter, fitted
loss = nn.MSELoss()(learner(images), target)