Data Augmentation

Context

Data augmentation trains a network on altered copies of its training images, drawn fresh every time an image is read: the same garment a few pixels to one side, mirrored, turned a little. The label stays, the pixels change, and the network meets each image in a new form on every epoch without one new image being collected.

It is how image models in real work are made to survive the framing of the images they will be handed, product photos shot a little off center, scans cropped by hand, phone pictures held at an angle. Every training pipeline for a vision model carries a list of these transforms, and choosing them, applying them to the right images and judging what they bought on the right images are three separate decisions.

Lesson 5.1.3's CNN scored 0.9008 on the test set and 0.2861 with the same images four pixels to the right, and its body carried the garments over while its head read them by position. Training on moved images is the fix that changes no layer. Done carelessly it does damage no error reports: a transform set on the dataset the validation images share moves them too and makes every validation score a draw, a transform that keeps the label still makes images the model will never meet, and a network judged on validation images framed in place will lose to one that was never asked to handle a move.

Goal

• Build a transforms.RandomCrop pipeline, show that each read of one image draws a new offset, recover the offset by search, repeat a draw with torch.manual_seed, and count a random flip's draws over 1,000 reads.
• Measure Lesson 5.1.3's CNN on mirrored and upside-down test images, class by class, and distinguish a transform that keeps the label from one that makes images the model will meet.
• Show how a transform set on train_full moves the validation images too, and build an augmented training set that shares train_set's indices and leaves validation untouched.
• Train the CNN on moved images, run the shift table against Lesson 5.1.3's network, and choose between the two on validation images framed as the deployed images will be.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the idea and its common transforms shown first, or the full list of what torchvision can draw.

• YouTube - "C4W2L10 Data Augmentation" by DeepLearningAI, Andrew Ng's course lecture. Mirroring, random cropping, rotation and color shifting as ways to multiply a training set, and how augmentation is run on the processor while the network trains.
• Official docs - torchvision 0.29, "Transforming and augmenting images". Every transform torchvision ships, grouped by kind, geometry, color, composition, the classic transforms and the newer transforms.v2, and how a transform is applied to a dataset.

Table of Contents

• A Transform That Draws
• Label-Preserving, or Not
• Augmenting the Training Images Only
• Training on Moved Images

Theory

Every Theory block runs in ~/dev/bootcamp, beside fashion.py, Lesson 4.5.1's, image_loop.py, Lesson 4.5.2's, and blocks.py, cnn.py and fashion_cnn.pt, Lessons 5.1.2 and 5.1.3's. The third subtitle saves augment.py, and the fourth trains on it and saves fashion_cnn_aug.pt. The trained figures are this machine's on its eight threads, as Lesson 5.1.3's were, and a seeded draw prints what it prints on torch 2.14.1 on a processor. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

A Transform That Draws

What. transforms.RandomCrop(28, padding=2) surrounds a 28 by 28 image with 2 rows and columns of black on every side, making it 32 by 32, and cuts a 28 by 28 window from a random place in it, so the garment lands up to 2 pixels from where it was, in each direction, with black brought in behind it and up to 2 rows or columns cut away at its leading edge, Lesson 4.5.3's slide drawn at random. Inside transforms.Compose before ToTensor, it runs every time the dataset reads an image, and each run draws a new place from torch's own random generator, the one torch.manual_seed sets. Why. A network trained on moved copies meets every training image at up to 25 offsets across its epochs, so a shift becomes something its loss has seen and paid for, where Lesson 5.1.3's network met every image at one. Nothing new is collected and the label is the image's own. When. In the transform of the dataset the training loader reads, and only that one.

import torch
import torch.nn.functional as F
from torchvision import datasets, transforms
from fashion import train_full

crop = transforms.Compose([transforms.RandomCrop(28, padding=2), transforms.ToTensor()])
moving = datasets.FashionMNIST("data", train=True, download=True, transform=crop)
print(moving.transform)
torch.manual_seed(42)
reads = [moving[0][0] for _ in range(4)]
print([torch.equal(reads[0], r) for r in reads], moving[0][1], reads[0].shape)

Compose(
    RandomCrop(size=(28, 28), padding=2)
    ToTensor()
)
[True, False, False, False] 9 torch.Size([1, 28, 28])

moving is a second dataset object over the same files in data that Lesson 4.5.1 downloaded, so download=True finds them and fetches nothing. Image 0, the ankle boot, read four times, gives four different tensors of the same shape, the first equal only to itself, and its label is 9 on every read. Where each read put the boot, found by trying all 25 placements:

image = train_full[0][0]

def place(x, dy, dx):
    padded = F.pad(x, (2, 2, 2, 2))  # 2 black columns, then 2 rows
    return padded[..., 2 - dy:30 - dy, 2 - dx:30 - dx]

def offset(read):
    for dy in range(-2, 3):
        for dx in range(-2, 3):
            if torch.equal(place(image, dy, dx), read):
                return dy, dx

torch.manual_seed(42)
print([offset(moving[0][0]) for _ in range(8)])
torch.manual_seed(42)
print([offset(moving[0][0]) for _ in range(8)])

[(0, 0), (1, -2), (1, 2), (2, -2), (2, -1), (-1, -2), (2, -2), (1, 0)]
[(0, 0), (1, -2), (1, 2), (2, -2), (2, -1), (-1, -2), (2, -2), (1, 0)]

F.pad(x, (2, 2, 2, 2)) adds 2 zeros on the left, right, top and bottom of the last two dimensions, Lesson 4.5.3's padding=1 ring done by hand, and the slice keeps the 28 by 28 window that puts the boot dy rows down and dx columns right. Every read matched exactly one placement: the first left the boot where it was, the second moved it a row down and two columns left, and the eight reads cover seven different placements, with (2, -2) drawn twice. Seeded again, the same eight placements come back in the same order, because the crop draws from torch's generator and torch.manual_seed sets it. A mirror draws too, with a probability:

flip = transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor()])
flipping = datasets.FashionMNIST("data", train=True, download=True, transform=flip)
print(flipping.transform)
torch.manual_seed(42)
reads = [flipping[0][0] for _ in range(1000)]
print(sum(torch.equal(r, image.flip(2)) for r in reads), sum(torch.equal(r, image) for r in reads))

Compose(
    RandomHorizontalFlip(p=0.5)
    ToTensor()
)
524 476

transforms.RandomHorizontalFlip() mirrors an image left to right with a probability of 0.5, its p, and leaves it as it is otherwise. image.flip(2) is the boot mirrored, its columns reversed, since its dimensions are channel, row and column. Of 1,000 seeded reads, 524 came back mirrored and 476 untouched, every read one of the two, a coin drawn per read from the same generator the crop draws from.

DON'T:

• DON'T expect a dataset with a random transform to return the same image twice. The ankle boot came back as four different tensors in four reads, and only the seed brings a sequence of reads back.
• DON'T read a random crop as a move that keeps every pixel. FashionMNIST garments fill their frame, so the window cuts into them: moved 2 columns right, the boot lost its 2 rightmost columns, 29 of its 433 lit pixels, moved 2 left it lost 12, and moved up or down it lost none, because its top and bottom rows are dark. Every read still equaled one of the 25 placements exactly, so a tensor that differs from the stored image is a placement, not corruption.
• DON'T read p=0.5 as every second read mirrored. Each read draws its own coin, and 1,000 seeded reads came back 524 mirrored and 476 untouched.

Label-Preserving, or Not

What. A label-preserving transform leaves an image showing its class: a shirt moved two pixels is a shirt, and so is a shirt mirrored left to right. That is the first test a transform passes. The second is whether its images look like the ones the model will be handed, and a transform can pass the first and fail the second. On digits a half turn fails the first, a 6 becomes a 9. Lesson 5.1.3's CNN, scored on test images mirrored, flip(3), and upside down, flip(2), shows what the second test is about. Why. A transform that keeps the label still spends the network's training on the images it makes. If those images never arrive, the network learned something it will not use, and if they do arrive and the network never saw them, its score on them is what the test set never measured. When. Before a transform is added to a training pipeline, with the images the model will meet in view.

import torch
from fashion import test_set, test_loader, train_full
from image_loop import accuracy
from cnn import cnn

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
batches = list(test_loader)
for name, dim in (("mirror", 3), ("upside down", 2)):
    flipped = [(x.flip(dim), y) for x, y in batches]
    print(name, f"{accuracy(model, flipped):.4f}")

mirror 0.6656
upside down 0.2877

flip(3) reverses each image's columns and flip(2) its rows, and neither changes a label. Mirrored, the test images score 0.6656 where in place they scored 0.9008, and upside down 0.2877. Class by class, mirrored:

model.eval()
labels = test_set.targets
with torch.no_grad():
    pred = torch.cat([model(x) for x, y in batches]).argmax(dim=1)
    mirrored = torch.cat([model(x.flip(3)) for x, y in batches]).argmax(dim=1)
for k in (1, 8, 0, 6, 5, 7, 9):
    mine = labels == k
    print(test_set.classes[k], f"{(pred[mine] == k).float().mean().item():.3f}", f"{(mirrored[mine] == k).float().mean().item():.3f}")

Trouser 0.981 0.975
Bag 0.969 0.954
T-shirt/top 0.865 0.845
Shirt 0.699 0.691
Sandal 0.967 0.453
Sneaker 0.969 0.180
Ankle boot 0.967 0.000

model.eval() is called again because accuracy hands the network back in training mode, Lesson 5.1.3's Lab 2. Trousers, bags, T-shirts and shirts lose a point or two mirrored. The shoes collapse: sandals to 0.453, sneakers to 0.180, and of 1,000 mirrored ankle boots the network calls not one an ankle boot. Why the shoes, from the training images themselves:

for k in (9, 7, 5, 8, 1):
    shown = train_full.data[train_full.targets == k].float() / 255
    left, right = shown[:, :, :14].mean().item(), shown[:, :, 14:].mean().item()
    print(train_full.classes[k], len(shown), round(left, 4), round(right, 4))

Ankle boot 6000 0.1705 0.4319
Sneaker 6000 0.121 0.2144
Sandal 6000 0.1008 0.1726
Bag 6000 0.3453 0.3618
Trouser 6000 0.2166 0.2292

train_full.data holds the raw uint8 pixels of all 60,000 training images, Lesson 4.5.1's, and the mask keeps one class's 6,000. The mean brightness of their left 14 columns and of their right 14 columns is nearly the same for bags and trousers, and for the three shoes the right half is brighter, the ankle boots 0.4319 against 0.1705: every shoe in FashionMNIST faces the same way, its shaft or heel on the right. A mirrored boot is still a boot, and no training image ever showed one. Upside down, class by class, and where the predictions went:

with torch.no_grad():
    upside = torch.cat([model(x.flip(2)) for x, y in batches]).argmax(dim=1)
for k in (6, 1, 8, 3, 9):
    mine = labels == k
    print(test_set.classes[k], f"{(upside[mine] == k).float().mean().item():.3f}")
print(torch.bincount(upside, minlength=10))

Shirt 0.825
Trouser 0.533
Bag 0.525
Dress 0.427
Ankle boot 0.003
tensor([ 860, 1103,   72,  981,  166,  280, 4059,  192, 2280,    7])

Upside down, the shirts look like the class that survives, 0.825, and the count of calls says what that number is: the network calls 4,059 of the 10,000 upside-down images Shirt and 2,280 Bag, torch.bincount counting the calls per class, so a class it names four times in ten catches most of its own images whatever it sees. Ankle boots fall to 0.003. A half turn keeps every label in this dataset and makes images no catalog sends.

DON'T:

• DON'T stop at the label when choosing a transform. A mirrored ankle boot is an ankle boot, and the CNN called 0 of 1,000 of them one, because every boot it trained on faced the other way.
• DON'T add a transform that makes images no deployed model will meet. If every photo a catalog receives has its shoes facing one way, mirroring spends training on a view that never arrives.
• DON'T read a class that holds up under a transform as robust to it. Upside down, Shirt scored 0.825 because the network called 4,059 of the 10,000 upside-down images Shirt.

Augmenting the Training Images Only

What. random_split, Lesson 4.5.1's, hands back two Subsets of one dataset object, train_full, each holding a list of indices into it, so train_set and val_set read their images through train_full's transform. Setting train_full.transform to the random crop moves the validation images too. The fix is a second dataset object over the same files, with the augmenting transform, and a Subset of it built on train_set.indices, so the training images are moved and the validation images keep train_full and its ToTensor alone. Why. The validation images choose the network, so they must be the same images every time they are read. Moved at random, every validation accuracy is a draw, two calls give two numbers, and the choice between two networks rests on luck. When. Every time an augmentation is added to a split built from one dataset, which is every split random_split makes.

import torch
from torchvision import transforms
from fashion import train_full, train_set, val_set, val_loader
from image_loop import accuracy
from cnn import cnn

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
print(f"{accuracy(model, val_loader):.4f}")
print(val_set.dataset is train_full, train_set.dataset is train_full, val_set.indices[:3])
first = val_set[0][0]
train_full.transform = transforms.Compose([transforms.RandomCrop(28, padding=2), transforms.ToTensor()])
torch.manual_seed(42)
print(torch.equal(val_set[0][0], first), torch.equal(val_set[0][0], val_set[0][0]))
print(f"{accuracy(model, val_loader):.4f}", f"{accuracy(model, val_loader):.4f}")

0.9077
True True [59753, 9414, 10988]
True False
0.8195 0.8237

Both halves of the split hold train_full itself, is checks the object, not its contents, and validation image 0 is train_full[59753]. Once train_full.transform crops at random, validation image 0's first read happens to land in place and equals the image read before, its next two reads differ from each other, and the network that scored 0.9077 on the validation images scores 0.8195 and then 0.8237 on two calls of the same accuracy, with no error and nothing in the loop changed. This script set the transform on its own copy of train_full, in its own process, so fashion.py and every other script are untouched. The fix, saved as augment.py in ~/dev/bootcamp:

import torch
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
from fashion import train_set

crop = transforms.Compose([transforms.RandomCrop(28, padding=2), transforms.ToTensor()])
moving = datasets.FashionMNIST("data", train=True, download=True, transform=crop)
aug_set = Subset(moving, train_set.indices)
aug_loader = DataLoader(aug_set, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))

if __name__ == "__main__":
    print(len(aug_set), len(aug_loader), aug_set.indices[:5])

Subset(moving, train_set.indices) is Lesson 4.5.1's Subset built by hand, the same 50,000 indices into a dataset whose transform crops, and aug_loader is train_loader over it, the same batch size and its own generator seeded 42. Checked from a second script:

import time
import torch
from torch.utils.data import random_split
from fashion import train_set, val_set, train_loader
from augment import aug_set, aug_loader, moving

print(len(aug_set), len(aug_loader), aug_set.indices[:5], train_set.indices[:5])
print(aug_set[0][1] == train_set[0][1], moving is not train_set.dataset)
first = val_set[0][0]
print(torch.equal(val_set[0][0], first))
for name, loader in (("plain", train_loader), ("crop", aug_loader)):
    start = time.perf_counter()
    for x, y in loader:
        pass
    print(name, f"{time.perf_counter() - start:.1f}")

50000 782 [37542, 43087, 2992, 25810, 39634] [37542, 43087, 2992, 25810, 39634]
True True
True
plain 1.7
crop 3.1

The augmented set holds the training set's 50,000 images in the same order, [37542, 43087, ...], the first five indices Lesson 4.5.1 printed, with the same labels, through its own dataset object, and validation image 0 reads the same twice. One pass over each loader with no network, in one script: the crop costs reading time, every image cut from a padded copy on every read. The seconds are this machine's in this run. Splitting the cropping dataset again instead, with the same seeded generator:

again_train, again_val = random_split(moving, [50_000, 10_000], generator=torch.Generator().manual_seed(42))
print(again_train.indices == train_set.indices, again_val.indices == val_set.indices, again_val.dataset is moving)
torch.manual_seed(42)
print(torch.equal(again_val[0][0], again_val[0][0]))

True True True
False

The second split draws the same indices, because random_split decides them from the dataset's length and the generator alone, and both are the same. Its validation half reads through moving, so two reads of its first image differ: the split is kept and the trap is rebuilt. Only the training half gets the cropping dataset, through Subset on train_set.indices.

DON'T:

• DON'T set an augmenting transform on the dataset random_split split. The validation images moved with it, and the same network scored 0.8195 and 0.8237 on two calls.
• DON'T split the cropping dataset again to separate the halves. With the same seed it drew the same indices and handed the validation half the crop as well, two reads of one image differing.

Training on Moved Images

What. Lesson 5.1.3's network, cnn(), trained by the same loop with aug_loader in place of train_loader, five epochs of Adam at 0.001, the best epoch kept by validation accuracy and saved as fashion_cnn_aug.pt. Then both networks are run through Lesson 4.5.3's shift table, and the choice between them is made twice, on the validation images framed as they are and on the validation images moved by the same random crop. Why. The network's body already moves with the garments and its head reads by position, Lesson 5.1.3's finding. Training on moved garments shows the head every garment at many positions, so a move inside the trained range is something the head has weights for. What that costs in place, and which network the validation images choose, depend on what the validation images are made to look like. When. When the images a deployed model meets are framed less carefully than the training images, and before any claim that augmentation helped.

import copy
import time
import torch
from torch import optim
from torch.utils.data import DataLoader, Subset
from torchvision import datasets
from fashion import val_set, val_loader, test_set, test_loader
from augment import aug_loader, crop
from image_loop import accuracy, train_epoch
from cnn import cnn

model = cnn()
optimizer = optim.Adam(model.parameters(), lr=0.001)
best_acc, best_epoch, best_state = 0.0, 0, None
for epoch in range(1, 6):
    start = time.perf_counter()
    train_loss = train_epoch(model, optimizer, aug_loader)
    seconds = time.perf_counter() - start
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}", f"{seconds:.1f}")
    if val_acc > best_acc:
        best_acc, best_epoch, best_state = val_acc, epoch, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
print(best_epoch, f"{best_acc:.4f}")
torch.save(model.state_dict(), "fashion_cnn_aug.pt")

1 0.5336 0.8446 7.8
2 0.3928 0.8713 7.8
3 0.3619 0.8797 7.7
4 0.3372 0.8893 7.2
5 0.3249 0.8834 7.4
4 0.8893

cnn() seeds 42 inside, so this network starts from the same weights as Lesson 5.1.3's, and every crop after that is drawn from the generator in the same order on every run. Its training loss ends at 0.3249 and its best validation accuracy is 0.8893 at epoch 4, against Lesson 5.1.3's 0.2155 and 0.9077 at epoch 3. The training losses are taken on different images, moved ones here and images in place there, so they do not compare. The shift table, the augmented network beside Lesson 5.1.3's:

plain = cnn()
plain.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
batches = list(test_loader)
for pixels in (0, 1, 2, 3, 4, 5, 8):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print(pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(plain, moved):.4f}")

def slide(x, pixels):
    out = torch.zeros_like(x)
    out[..., pixels:] = x[..., :-pixels]
    return out

for pixels in (2, 4):
    moved = [(slide(x, pixels), y) for x, y in batches]
    print("slide", pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(plain, moved):.4f}")

0 0.8796 0.9008
1 0.8748 0.8752
2 0.8611 0.7795
3 0.7496 0.5196
4 0.4674 0.2861
5 0.2218 0.2303
8 0.1330 0.1385
slide 2 0.8695 0.8058
slide 4 0.6256 0.3505

The test images answer a question about moved images here, as they did in Lessons 4.5.3 and 5.1.3, and choose nothing. In place the augmented network scores 0.8796, two points under the other's 0.9008. Two pixels right it keeps 0.8611 where the other fell to 0.7795, and slid two pixels with black brought in, the move its crops trained on, 0.8695. Past the trained range the gain fades and then goes: 0.4674 against 0.2861 at four pixels, and at five and eight the two networks are within a point of each other. The crop taught moves of up to two pixels, and the table shows the tolerance it bought ending where the training stopped. Two pixels right, class by class, Lesson 5.1.3's network first:

model.eval()
plain.eval()
labels = test_set.targets
with torch.no_grad():
    before = torch.cat([plain(torch.roll(x, shifts=2, dims=3)) for x, y in batches]).argmax(dim=1)
    after = torch.cat([model(torch.roll(x, shifts=2, dims=3)) for x, y in batches]).argmax(dim=1)
for k, name in enumerate(test_set.classes):
    mine = labels == k
    print(name, f"{(before[mine] == k).float().mean().item():.3f}", f"{(after[mine] == k).float().mean().item():.3f}")

T-shirt/top 0.763 0.836
Trouser 0.965 0.958
Pullover 0.764 0.896
Dress 0.665 0.915
Coat 0.558 0.657
Sandal 0.896 0.891
Shirt 0.583 0.605
Sneaker 0.973 0.970
Bag 0.962 0.982
Ankle boot 0.666 0.901

accuracy handed both networks back in training mode, so model.eval() and plain.eval() come first. The classes the move hurt most gain most: dresses from 0.665 to 0.915, ankle boots from 0.666 to 0.901, pullovers from 0.764 to 0.896. Trousers, sandals and sneakers, which two pixels barely touched, stay within a point. Coats and shirts, the classes the network confuses even in place, gain 0.099 and 0.022 and remain the two worst. Which network to keep:

framed = datasets.FashionMNIST("data", train=True, download=True, transform=crop)
torch.manual_seed(0)
val_moved = [(x, y) for x, y in DataLoader(Subset(framed, val_set.indices), batch_size=1000)]
print(f"{accuracy(plain, val_loader):.4f}", f"{accuracy(model, val_loader):.4f}")
print(f"{accuracy(plain, val_moved):.4f}", f"{accuracy(model, val_moved):.4f}")
test_framed = datasets.FashionMNIST("data", train=False, download=True, transform=crop)
torch.manual_seed(0)
test_moved = [(x, y) for x, y in DataLoader(test_framed, batch_size=1000)]
print(f"{accuracy(model, test_moved):.4f}")

0.9077 0.8893
0.8188 0.8742
0.8615

val_moved is the 10,000 validation images, each cut by the random crop once, under a seed, and kept in a list so that both networks are scored on the same moved images. A validation set made with a random transform is made once and kept, because read twice through the transform it would be two validation sets, and a choice between two networks scored on different images is the shared-dataset trap of the third subtitle in a new form. Framed as they are, the validation images choose Lesson 5.1.3's network, 0.9077 against 0.8893. Moved the way a careless camera moves a garment, they choose the augmented one, 0.8742 against 0.8188. Neither choice is wrong: each answers for the images the validation set was made to look like, and the choice belongs to whoever knows which images the model will be handed. For a catalog whose photos arrive a few pixels off, the moved validation images are the honest ones, the augmented network is the choice, and the test images, moved by the same crop under a seed, score it once: 0.8615.

DON'T:

• DON'T judge an augmentation on validation images framed in place when the deployed images will move. In place the validation images chose the network that scored 0.8188 on moved ones.
• DON'T read the tolerance a crop bought as reaching past its padding. Trained on moves of up to two pixels, the network was within a point of the unaugmented one at five and at eight.
• DON'T compare the two networks' training losses. 0.3249 was taken on moved images and 0.2155 on images in place, two different exams.

Lab Exercises

Lab Exercise 1: A Wider Crop, and Every Class Mirrored

In crop_wide.py, build transforms.Compose([transforms.RandomCrop(28, padding=4), transforms.ToTensor()]) and a FashionMNIST training dataset that reads through it. Write place(x, dy, dx, pad) and offset(read, pad) for a padding of pad, seed 42, and print the placement of ten reads of image 0 and its label. Say in a comment how many placements this crop can draw and how far each read can move the boot.

Then load fashion_cnn.pt into cnn() and print, for all ten classes in label order, its accuracy on the test images in place and mirrored with flip(3), computed in evaluation mode through the ten batches of test_loader. Say in a comment which classes a mirror leaves within two points and which it breaks.

Lab Exercise 2: Four Pixels of Padding, Trained and Tested

In wide_train.py, build a training set that crops with padding=4: a FashionMNIST dataset over the training files with that transform, a Subset of it on train_set.indices, and a loader of 64 shuffled by its own generator seeded 42. Print whether validation image 0 reads the same twice and whether the new set's indices equal train_set's.

Train cnn() on it for three epochs with Adam at 0.001, keeping the best epoch by validation accuracy, and print each epoch's training loss and validation accuracy, then the best. Load fashion_cnn_aug.pt and print both networks' accuracy on the test images rolled right by 2, 4 and 6 pixels. Say in a comment what the wider padding bought and what it cost against the two-pixel crop.

import torch
import torch.nn.functional as F
from torchvision import datasets, transforms
from fashion import test_set, test_loader
from cnn import cnn

wide = transforms.Compose([transforms.RandomCrop(28, padding=4), transforms.ToTensor()])
far = datasets.FashionMNIST("data", train=True, download=True, transform=wide)
image = far.data[0].float().unsqueeze(0) / 255

def place(x, dy, dx, pad):
    padded = F.pad(x, (pad, pad, pad, pad))
    n = 28 + 2 * pad
    return padded[..., pad - dy:n - pad - dy, pad - dx:n - pad - dx]

def offset(read, pad):
    for dy in range(-pad, pad + 1):
        for dx in range(-pad, pad + 1):
            if torch.equal(place(image, dy, dx, pad), read):
                return dy, dx

torch.manual_seed(42)
print([offset(far[0][0], 4) for _ in range(10)], far[0][1])
# 9 rows by 9 columns, 81 placements, each read moving the boot
# up to 4 pixels in each direction, (-4, 4) among the ten here.

[(-2, -1), (-3, 0), (4, 2), (-3, -1), (0, 2), (0, 0), (-4, 4), (4, 0), (2, 0), (1, 0)] 9

model = cnn()
model.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
model.eval()
batches = list(test_loader)
labels = test_set.targets
with torch.no_grad():
    pred = torch.cat([model(x) for x, y in batches]).argmax(dim=1)
    mirrored = torch.cat([model(x.flip(3)) for x, y in batches]).argmax(dim=1)
for k, name in enumerate(test_set.classes):
    mine = labels == k
    print(name, f"{(pred[mine] == k).float().mean().item():.3f}", f"{(mirrored[mine] == k).float().mean().item():.3f}")
# Within two points: every class but the shoes, T-shirts losing most.
# Broken: the three shoes, every one of which faces the same way.

T-shirt/top 0.865 0.845
Trouser 0.981 0.975
Pullover 0.839 0.830
Dress 0.903 0.892
Coat 0.849 0.836
Sandal 0.967 0.453
Shirt 0.699 0.691
Sneaker 0.969 0.180
Bag 0.969 0.954
Ankle boot 0.967 0.000

import copy
import torch
from torch import optim
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
from fashion import train_set, val_set, val_loader, test_loader
from image_loop import accuracy, train_epoch
from cnn import cnn

wide = transforms.Compose([transforms.RandomCrop(28, padding=4), transforms.ToTensor()])
far = datasets.FashionMNIST("data", train=True, download=True, transform=wide)
far_set = Subset(far, train_set.indices)
far_loader = DataLoader(far_set, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
first = val_set[0][0]
print(torch.equal(val_set[0][0], first), far_set.indices == train_set.indices)
model = cnn()
optimizer = optim.Adam(model.parameters(), lr=0.001)
best_acc, best_epoch, best_state = 0.0, 0, None
for epoch in range(1, 4):
    train_loss = train_epoch(model, optimizer, far_loader)
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}")
    if val_acc > best_acc:
        best_acc, best_epoch, best_state = val_acc, epoch, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
print(best_epoch, f"{best_acc:.4f}")

True True
1 0.6650 0.8156
2 0.5013 0.8466
3 0.4551 0.8496
3 0.8496

narrow = cnn()
narrow.load_state_dict(torch.load("fashion_cnn_aug.pt", weights_only=True))
batches = list(test_loader)
for pixels in (2, 4, 6):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print(pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(narrow, moved):.4f}")
# The padding of 4 holds 0.6948 at four pixels and 0.3360 at six,
# against 0.4674 and 0.1749, and pays two points at two pixels and
# about four on the validation images in place, after three epochs.

2 0.8405 0.8611
4 0.6948 0.4674
6 0.3360 0.1749

Theory Assessment

Q1: Four Different Boots

A teammate reads image 0 of the cropping dataset four times, gets four different tensors, and reports the dataset as corrupt. A second teammate says the loader's generator, seeded 42, should have made the reads repeat. Say what draws each crop, why four reads differ, what makes a sequence of reads repeat, and what the loader's own generator does and does not control.

Hint: Ask which generator torch.manual_seed sets and which one DataLoader(..., generator=...) holds, and which of the two a transform inside a dataset reads from.

Q2: A Mirror for the Catalog

A team training a shoe classifier on FashionMNIST adds transforms.RandomHorizontalFlip(), which mirrors each image with a probability of one half, because a mirrored shoe is still a shoe. Say what Lesson 5.1.3's CNN did on mirrored ankle boots and why, what the flip would teach a network trained with it, and what the team must know about its incoming photos before keeping it.

Hint: Compare the left and right halves' brightness the third subtitle printed, and ask whether a catalog's photos ever show a shoe facing the other way.

Q3: 0.8195, Then 0.8237

After train_full.transform was set to the random crop, the same network on the same val_loader scored 0.8195 and then 0.8237. Say why val_loader's images changed, why two calls disagree, what Subset(moving, train_set.indices) does differently, and why rebuilding the split with a fresh random_split on the cropping dataset would be the wrong fix.

Hint: Ask what object a Subset reads its images through, and what a second split of the cropping dataset hands its validation half.

Q4: The Augmentation That Hurt

A teammate reads the fourth subtitle and reports that augmentation made the network worse: its training loss ended at 0.3249 against 0.2155, its validation accuracy at 0.8893 against 0.9077, and its test score in place at 0.8796 against 0.9008. Say what each of the three comparisons measures and which of them compares like with like, what the moved validation images said, and what one sentence of the report should have stated before any number.

Hint: Ask on what images each loss was taken, and which images the validation set was made to look like in each of the two choices.

import copy
import torch
from torch import optim
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
from fashion import train_set, val_loader, test_set, test_loader
from image_loop import accuracy, train_epoch
from cnn import cnn

mirror = transforms.Compose([transforms.RandomHorizontalFlip(), transforms.ToTensor()])
flipping = datasets.FashionMNIST("data", train=True, download=True, transform=mirror)
flip_loader = DataLoader(Subset(flipping, train_set.indices), batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
model = cnn()
optimizer = optim.Adam(model.parameters(), lr=0.001)
best_acc, best_state = 0.0, None
for epoch in range(1, 6):
    train_epoch(model, optimizer, flip_loader)
    val_acc = accuracy(model, val_loader)
    if val_acc > best_acc:
        best_acc, best_state = val_acc, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
model.eval()
labels = test_set.targets
with torch.no_grad():
    pred = torch.cat([model(x) for x, y in test_loader]).argmax(dim=1)
    mirrored = torch.cat([model(x.flip(3)) for x, y in test_loader]).argmax(dim=1)
print(f"{best_acc:.4f}")
for k in (5, 7, 9):
    mine = labels == k
    print(test_set.classes[k], f"{(pred[mine] == k).float().mean().item():.3f}", f"{(mirrored[mine] == k).float().mean().item():.3f}")

0.9018
Sandal 0.967 0.963
Sneaker 0.954 0.959
Ankle boot 0.956 0.951

import torch
from torch.utils.data import DataLoader
from fashion import train_set
from image_loop import loss_fn
from cnn import cnn

in_order = DataLoader(train_set, batch_size=1000)
for name in ("fashion_cnn.pt", "fashion_cnn_aug.pt"):
    net = cnn()
    net.load_state_dict(torch.load(name, weights_only=True))
    net.eval()
    total = 0.0
    with torch.no_grad():
        for x, y in in_order:
            total += loss_fn(net(x), y).item() * len(y)
    print(name, f"{total / len(train_set):.4f}")

fashion_cnn.pt 0.2138
fashion_cnn_aug.pt 0.2946

Assignment

Assignment: A Network That Survives a Turn

The crop taught the network moves. Write turned.py, which trains it on garments turned a little as well as moved, and measures what the turn bought on turned test images.

Two things this lesson did not use. transforms.RandomAffine(degrees=15, translate=(0.1, 0.1)) turns each image by an angle drawn between -15 and 15 degrees and moves it by up to a tenth of its width and of its height, about 2.8 pixels here, filling the corners it uncovers with black, and it draws from the same generator the crop does. torchvision.transforms.functional.rotate(x, angle), imported as TF by import torchvision.transforms.functional as TF, turns a tensor or a batch of images by exactly angle degrees counterclockwise, with the same black fill, so a test batch can be turned by a fixed amount.

Build the affine transform in a Compose with ToTensor, print it, and build its training set and loader the way augment.py builds aug_set and aug_loader, on train_set.indices with a generator seeded 42. Train cnn() on it for five epochs with Adam at 0.001, printing each epoch's training loss and validation accuracy and keeping the best. Load fashion_cnn.pt and print both networks' accuracy on the test images turned by 0, 10, 20 and 30 degrees, then rolled right by 2 and 4 pixels, and say in a comment what the turn bought, what it cost, and how its moves compare with the two-pixel crop's.

Hints:

• RandomAffine prints its range of degrees as degrees=[-15.0, 15.0], because it stores the one number given as a symmetric range.
• TF.rotate takes a whole (1000, 1, 28, 28) batch at once and turns every image by the same angle, so one list of turned batches serves both networks.

import copy
import torch
import torchvision.transforms.functional as TF
from torch import optim
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
from fashion import train_set, val_loader, test_loader
from image_loop import accuracy, train_epoch
from cnn import cnn

turn = transforms.Compose([transforms.RandomAffine(degrees=15, translate=(0.1, 0.1)), transforms.ToTensor()])
print(turn)
turning = datasets.FashionMNIST("data", train=True, download=True, transform=turn)
turn_loader = DataLoader(Subset(turning, train_set.indices), batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
model = cnn()
optimizer = optim.Adam(model.parameters(), lr=0.001)
best_acc, best_epoch, best_state = 0.0, 0, None
for epoch in range(1, 6):
    train_loss = train_epoch(model, optimizer, turn_loader)
    val_acc = accuracy(model, val_loader)
    print(epoch, f"{train_loss:.4f}", f"{val_acc:.4f}")
    if val_acc > best_acc:
        best_acc, best_epoch, best_state = val_acc, epoch, copy.deepcopy(model.state_dict())
model.load_state_dict(best_state)
print(best_epoch, f"{best_acc:.4f}")

Compose(
    RandomAffine(degrees=[-15.0, 15.0], translate=(0.1, 0.1))
    ToTensor()
)
1 0.6426 0.8451
2 0.4919 0.8587
3 0.4489 0.8747
4 0.4265 0.8564
5 0.4103 0.8568
3 0.8747

plain = cnn()
plain.load_state_dict(torch.load("fashion_cnn.pt", weights_only=True))
batches = list(test_loader)
for degrees in (0, 10, 20, 30):
    turned = [(TF.rotate(x, degrees), y) for x, y in batches]
    print(degrees, f"{accuracy(model, turned):.4f}", f"{accuracy(plain, turned):.4f}")
for pixels in (2, 4):
    moved = [(torch.roll(x, shifts=pixels, dims=3), y) for x, y in batches]
    print("right", pixels, f"{accuracy(model, moved):.4f}", f"{accuracy(plain, moved):.4f}")
# Turned 20 degrees, the affine network keeps 0.8105 where the plain
# one falls to 0.4552, and at 30, past its training range, 0.6214
# against 0.2871. It pays in place, 0.8674 against 0.9008. Its moves,
# up to 2.8 pixels, give 0.8488 at two and 0.6596 at four, against
# the crop's 0.8611 and 0.4674.

0 0.8674 0.9008
10 0.8615 0.7977
20 0.8105 0.4552
30 0.6214 0.2871
right 2 0.8488 0.7795
right 4 0.6596 0.2861

Additional Resources

Optional, and nothing later depends on either. Worth opening for how far augmentation goes beyond geometry, or for the transforms that act on several images at once.

• Book chapter - Dive into Deep Learning, "Image Augmentation" by Zhang, Lipton, Li and Smola, under CC BY-SA 4.0. Flipping, cropping and color changes drawn on one photo, combined with Compose, and a network trained on CIFAR-10 with them.
• Official docs - torchvision 0.29, RandomAffine. The transform of the Assignment in full: degrees, translation, scaling, shear, the interpolation used to fill a turned pixel, and the fill for the uncovered corners.

Summary

Augmentation changes nothing in the network and still changes what it can do. The crop showed the head every garment at 25 places, and the moves of up to two pixels that took Lesson 5.1.3's network from 0.9008 to 0.7795 cost the new one under two points. What it was never shown, a move of five pixels, it still fails, so the tolerance a transform buys is the tolerance it trained, measured and bounded, not a property the network acquired in general.

Most of the lesson was about where the transform goes and what judges it. A transform has two tests, whether it keeps the label and whether it makes images the model will meet, and the mirrored boots passed the first and failed the second so completely that the network called none of them a boot, until a run that showed it mirrored shoes brought them back to 0.951. The one class that seemed to survive a half turn was the class the network named for four images in ten, which is a count of calls and not a kind of tolerance. A transform set on a shared dataset moved the validation images and turned every score into a draw. And the choice between the two networks reversed when the validation images were moved, because a validation set answers only for the images it was made to look like.

That last point reaches past images. Every comparison in this section was a measurement on a stated set of images, and the right network depended on which set was the honest stand-in for the world. Section 5.2 starts from networks someone else trained on images of their own choosing, and the first question to put to them is the one this lesson ended on.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

crop = transforms.Compose([transforms.RandomCrop(28, padding=2), transforms.ToTensor()])
moving = datasets.FashionMNIST("data", train=True, download=True, transform=crop)
aug_set = Subset(moving, train_set.indices)  # same split
aug_loader = DataLoader(aug_set, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))

torch.manual_seed(0)  # one draw, kept for both networks
val_moved = [(x, y) for x, y in DataLoader(Subset(framed, val_set.indices), batch_size=1000)]
accuracy(plain, val_moved), accuracy(model, val_moved)