MLP on Images - Flatten and Train

Context

An image reaches a fully connected network (one whose layers feed every input to every unit, as Lesson 4.3.1's MLP does) as one row of numbers. Flattening lays its 28 rows of 28 pixels end to end as 784 values, and that MLP, widened to 128 hidden units, reads them as it read a listing's 18 columns: every pixel feeds every hidden unit, ten output units return one logit per kind of clothing, and Lesson 4.3.3's loop trains it on 50,000 images, 64 at a time.

A network on flattened pixels is a first model for a new image dataset: the score a convolutional network (one that reads an image through a small window slid across it, Module 5's subject) later has to beat, and the run that shows whether the images, the labels and the loaders arrive as the script assumes. A classifier of product photos or of digits on scanned forms is scored the same way, one number on images no choice touched, then the classes behind it.

Without the discipline, a batch handed over unflattened is refused in a message about 1792x28 that never mentions an image, and the rest lets the run finish: a network that ends in a softmax trains on a loss that cannot fall below 1.4612 and reaches 0.8010 where the same network without it reaches 0.8725, and a test accuracy of 0.8677 hides a class called right 623 times in 1,000. Lesson 4.5.1 saved fashion.py, FashionMNIST split and loaded, and this lesson trains on it.

Goal

• Flatten one image with flatten, view(-1) and reshape(784), flatten a batch with flatten(1) and nn.Flatten(), and diagnose the refusal a batch handed over unflattened prints.
• Build the 784-128-10 network, count its 101,770 parameters by layer, and check its seeded start against ten classes: a loss beside ln 10 and a validation accuracy near one in ten.
• Train it for five epochs with optim.Adam at 0.001 through image_loop.py, read each epoch's training loss, validation accuracy and seconds, and save it with torch.save.
• Train a linear model on the same pixels in a script of its own and state what the hidden layer buys as a count of validation images called right.
• Score the test images once, then read the per-class accuracy, the ten-class confusion matrix, one image's logits and softmax, and the images the network got wrong.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see a network read flattened images, or the layer that flattens them, before you build yours.

• YouTube - "PyTorch Tutorial 13 - Feed-Forward Neural Network" by Patrick Loeber. Just over twenty-one minutes on digit classification with the MNIST dataset: a DataLoader with a transform loading the images, a feed-forward network with an input layer, a hidden layer and an output layer and its activation functions, a loss and an optimizer, a training loop over batches, the accuracy computed at the end, and the lines that let the same code run on a GPU.
• Official docs - PyTorch 2.14, torch.nn.Flatten. The layer made for Sequential, its two arguments, start_dim at 1 and end_dim at -1 unless given, the shape rule that multiplies the flattened sizes into one, and two examples on an input of shape (32, 1, 5, 5): (32, 25) with the defaults and (160, 5) with nn.Flatten(0, 2).

Table of Contents

• Flatten: 784 Numbers per Image
• The Network, 101,770 Parameters
• The Loop on 50,000 Images
• A Linear Model on Pixels, for Comparison
• The Test Set, Scored Once, and the Classes It Confuses

Theory

Every block runs in ~/dev/bootcamp, beside the data folder and the fashion.py that Lesson 4.5.1 saved. Its import builds train_full, the 60,000 training images, test_set, the 10,000 test images, train_set and val_set, the 50,000 and 10,000 that random_split carved from the first with a generator seeded 42, and three loaders: train_loader in batches of 64, shuffled by its own generator seeded 42, and val_loader and test_loader in batches of 1,000. The third subtitle saves image_loop.py for the scripts after it to import. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Flatten: 784 Numbers per Image

What. Flattening (laying every value of a tensor out along one dimension, row after row, Lesson 2.2.1's ravel on a tensor) turns an image of shape (1, 28, 28) into 784 numbers. image.flatten(), image.view(-1) and image.reshape(784) flatten one image, and images.flatten(1) or the layer nn.Flatten() flatten a batch from dimension 1 on, so 64 images stay 64 rows. Why. A fully connected layer reads rows of features. nn.Linear(784, 128) multiplies the last dimension of its input by its weight, Lesson 4.3.1's X @ W.T + b, and takes every dimension in front of the last as rows, so an image of 28 rows of 28 pixels is refused until the 28 rows are one row of 784. When. In front of the first fully connected layer that reads an image, as the network's own first layer, so every script hands the model its batches exactly as the loader returns them.

The first training image against a layer built for 784 inputs:

import torch
from torch import nn
from fashion import train_full, train_loader

image, label = train_full[0]
# torch.Size([1, 28, 28]) Ankle boot
print(image.shape, train_full.classes[label])
try:
    nn.Linear(784, 10)(image)
except RuntimeError as err:
    print(err)

mat1 and mat2 shapes cannot be multiplied (28x28 and 784x10)

The layer took the image's 28 rows of 28 pixels as a table of 28 rows with 28 columns, the 28x28 of the message, against its weight transposed, 784 by 10, and the product does not exist. The three flattens:

flat = image.flatten()
print(flat.shape)
print(torch.equal(flat, image.view(-1)), torch.equal(flat, image.reshape(784)))
print(torch.equal(flat.view(1, 28, 28), image))
print(image[0, 10, 10:18])
print(flat[290:298])

torch.Size([784])
True True
True
tensor([0.0000, 0.0000, 0.0000, 0.7569, 0.8941, 0.8549, 0.8353, 0.7765])
tensor([0.0000, 0.0000, 0.0000, 0.7569, 0.8941, 0.8549, 0.8353, 0.7765])

The three calls return the same 784 numbers, view(-1) and reshape as Lesson 4.1.2 used them, and view(1, 28, 28) gives the image back exactly, so no value is lost. The order is the rows' order: pixel (r, c) lands at 28 * r + c, which is why row 10 from column 10 is flat[290:298]. Three pixels of the boot, by position:

spots = [293, 294, 321]
print(flat[spots])
print(image[0, 10, 13], image[0, 10, 14], image[0, 11, 13])

tensor([0.7569, 0.8941, 0.8588])
tensor(0.7569) tensor(0.8941) tensor(0.8588)

Position 293 is row 10, column 13, position 294 its neighbor to the right and position 321 its neighbor below, 28 places on. Flattening keeps every value and forgets which pixel sat beside which: 293 and 321 touch in the image, and nothing in the row of 784 says so. A batch, as the loader hands it over:

images, labels = next(iter(train_loader))
print(images.shape, labels.shape)
print(images.flatten(1).shape, nn.Flatten()(images).shape)
print(torch.equal(images.flatten(1)[5], images[5].flatten()))
print(images.flatten().shape)
print(nn.Flatten())

torch.Size([64, 1, 28, 28]) torch.Size([64])
torch.Size([64, 784]) torch.Size([64, 784])
True
torch.Size([50176])
Flatten(start_dim=1, end_dim=-1)

Row 5 of flatten(1)'s result is image 5 flattened. nn.Flatten() is the same call as a layer, and its print shows the start_dim=1 it uses unless told otherwise. flatten() with no argument starts at dimension 0 and lays all 64 images end to end as one row of 50,176. What a layer makes of each:

layer = nn.Linear(784, 128)
for batch in (images, images.flatten()):
    try:
        layer(batch)
    except RuntimeError as err:
        print(err)
print(layer(images.flatten(1)).shape)

mat1 and mat2 shapes cannot be multiplied (1792x28 and 784x128)
mat1 and mat2 shapes cannot be multiplied (1x50176 and 784x128)
torch.Size([64, 128])

Unflattened, the batch reached the layer as 1,792 rows of 28, 64 images times 28 rows each, and the message counts image rows without naming an image. Flattened from dimension 0 it was one row of 50,176. Flattened from dimension 1 it is 64 rows of 784, and each image comes out as 128 numbers.

DON'T:

• DON'T hand a batch of images to nn.Linear without nn.Flatten() in front of it. The refusal names 1792x28, a count of image rows, and says nothing about flattening.
• DON'T flatten a batch with flatten(). It starts at dimension 0, so 64 images become one row of 50,176, where flatten(1) and nn.Flatten() keep 64 rows of 784.
• DON'T read the row of 784 as the picture. Pixels 293 and 321 touch in the image and sit 28 places apart in the row, and nothing in the row marks them as neighbors.

The Network, 101,770 Parameters

What. Lesson 4.3.1's MLP with nn.Flatten() in front, nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10)): 784 inputs, a hidden layer of 128 units behind a ReLU and ten output units, one logit per class, 101,770 parameters created after torch.manual_seed(42). Its loss is nn.CrossEntropyLoss() on the ten logits and the int64 class index, Lesson 4.3.2's ten-class form, and its optimizer is Lesson 4.3.4's optim.Adam at its default rate, 0.001. Why. The first layer holds one row of 784 weights per hidden unit, so the image's size sets most of the network's size, and the count read before the first epoch says where the parameters sit. The seeded start is read before any step: a network that has learned nothing spreads about 0.1 over each of ten classes, and the loss of that spread is -log(0.1), ln 10. When. Every time a network meets a new input size, the count before the first epoch and the start's loss beside the log of the class count, ln 10 here, before the loop runs.

import math
import torch
from torch import nn
from fashion import train_loader

torch.manual_seed(42)
model = nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))
print(model)
for name, p in model.named_parameters():
    print(name, tuple(p.shape), p.numel())
total = sum(p.numel() for p in model.parameters())
print(total, round(100 * model[1].weight.numel() / total, 1))

Sequential(
  (0): Flatten(start_dim=1, end_dim=-1)
  (1): Linear(in_features=784, out_features=128, bias=True)
  (2): ReLU()
  (3): Linear(in_features=128, out_features=10, bias=True)
)
1.weight (128, 784) 100352
1.bias (128,) 128
3.weight (10, 128) 1280
3.bias (10,) 10
101770 98.6

nn.Flatten() holds nothing and takes position 0, so the first layer's tensors are 1.weight and 1.bias, named by position as Lesson 4.3.1 read them, and the output layer's are 3.weight and 3.bias. The first weight is (128, 784), 100,352 numbers and 98.6 percent of the network, where the output layer's 1,290 turn 128 numbers into ten. Lesson 4.3.1's network on 18 columns held 641. The seeded start, on the first training batch:

loss_fn = nn.CrossEntropyLoss()
images, labels = next(iter(train_loader))
with torch.no_grad():
    logits = model(images)
probs = torch.softmax(logits, dim=1)
print(logits.shape, labels.dtype)
print(probs[0])
print(f"{loss_fn(logits, labels).item():.4f}", f"{math.log(10):.4f}")
print(logits.argmax(dim=1)[:8], labels[:8])

torch.Size([64, 10]) torch.int64
tensor([0.1018, 0.0914, 0.0974, 0.0990, 0.0850, 0.1098, 0.0971, 0.1023, 0.1200,
        0.0961])
2.2971 2.3026
tensor([8, 0, 0, 8, 8, 5, 8, 8]) tensor([3, 2, 0, 4, 4, 2, 8, 7])

The first image's ten probabilities run from 0.0850 to 0.1200, and the batch's loss is 2.2971 beside ln 10's 2.3026. The calls are the seed's draw, the same on every run of torch 2.14.1 on a processor, five of the first eight going to class 8, Bag, and two of the eight right. The same probabilities handed to the loss:

print(f"{loss_fn(probs, labels).item():.4f}")
sure = torch.zeros(64, 10)
sure[torch.arange(64), labels] = 1.0
print(f"{loss_fn(sure, labels).item():.4f}", f"{math.log(9 + math.e) - 1:.4f}")

2.3011
1.4612 1.4612

Handed the probabilities instead of the logits, the loss prints 2.3011, so the check at the start cannot see Lesson 4.3.2's trap. sure holds the probabilities of a network right about all 64 images and certain of each, 1 on every image's own class, written with Lesson 4.3.2's indexing by two arrays, and the loss applies its own softmax to those rows: the right class's e to the 1 against nine e to the 0, log(9 + e) - 1, 1.4612. A network that ends in a softmax trains on a loss that cannot fall below 1.4612, however many images it calls right.

DON'T:

• DON'T end the network in a softmax when its loss is nn.CrossEntropyLoss. The start prints 2.3011 beside the right 2.2971 and passes the check, and the loss it trains on cannot go under 1.4612.
• DON'T size a network on images by its hidden layers. The first layer's 784 by 128 weight is 98.6 percent of the 101,770, and the image's 784 pixels set it.
• DON'T read the seeded start's calls as a preference. Five of the first eight went to Bag by the draw of the seed, and the loss beside ln 10 is what says the network has learned nothing yet.

The Loop on 50,000 Images

What. Lesson 4.3.3's loop over train_loader, one update per batch and 782 an epoch, the 781 batches of 64 and the last of 16 that Lesson 4.5.1 counted, with the validation accuracy read after every epoch. image_loop.py holds the loss, the network, the accuracy and one epoch of the loop. Why. An epoch is 50,000 images, so the loop is where the seconds go, and the validation accuracy, read on images the loop never fits, is the number every choice in this lesson is made on. When. Every time a model trains on the images, with the validation accuracy read after each epoch, and the seconds printed wherever the cost of an epoch is in question.

The four pieces go into one file. Save it as image_loop.py in ~/dev/bootcamp, beside fashion.py:

import torch
from torch import nn

loss_fn = nn.CrossEntropyLoss()

def mlp():
    torch.manual_seed(42)
    return nn.Sequential(nn.Flatten(), nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))

def accuracy(model, loader):
    model.eval()
    correct = total = 0
    with torch.no_grad():
        for x, y in loader:
            correct += (model(x).argmax(dim=1) == y).sum().item()
            total += len(y)
    model.train()
    return correct / total

def train_epoch(model, optimizer, loader):
    total = 0.0
    for x, y in loader:
        loss = loss_fn(model(x), y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        total += loss.item() * len(y)
    return total / len(loader.dataset)

mlp() is the last subtitle's network with the seed inside it, so every call returns the same starting parameters, as Lesson 4.3.1's network() does. accuracy is Lesson 4.2.3's validation pass with a count where the loss was: evaluation mode, no graph, the images whose argmax(dim=1) equals their label counted batch by batch, training mode restored, and the count divided by the images seen. train_epoch is Lesson 4.3.3's inner loop, the five steps in Lesson 4.2.2's order with optimizer.zero_grad() last, and it returns each batch's loss times the batch's size, summed and divided by len(loader.dataset), the 50,000, so the last batch counts for its 16 images. That number is the mean of the losses the epoch took along the way, each before its own update, which Lesson 4.3.3 showed is not the loss of the model that ends the epoch. It comes free with the epoch. Five epochs, each timed with time.perf_counter() as Lesson 4.3.4 timed a fit:

import time
import torch
from torch import optim
from torch.utils.data import DataLoader
from fashion import train_set, train_loader, val_loader
from image_loop import loss_fn, mlp, accuracy, train_epoch

model = mlp()
optimizer = optim.Adam(model.parameters(), lr=0.001)
print(len(train_loader), f"{accuracy(model, val_loader):.4f}")
for epoch in range(1, 6):
    start = time.perf_counter()
    train_loss = train_epoch(model, optimizer, train_loader)
    seconds = time.perf_counter() - start
    print(epoch, f"{train_loss:.4f}", f"{accuracy(model, val_loader):.4f}", f"{seconds:.1f}")
torch.save(model.state_dict(), "fashion_mlp.pt")

782 0.1170
1 0.5727 0.8342 6.2
2 0.4132 0.8627 6.5
3 0.3705 0.8645 6.5
4 0.3439 0.8695 6.6
5 0.3224 0.8725 7.7

Before any step the network calls 1,170 of the 10,000 validation images right, near the one in ten of a guess, and after the first epoch's 782 updates it calls 8,342 right. The validation accuracy rises at every epoch to 0.8725 and the training loss falls at every epoch to 0.3224, and torch.save writes the parameters to fashion_mlp.pt, Lesson 4.4.3's call, for the last subtitle to load. The seconds are this machine's, and another machine, or this one on another run, prints others. Where they go, and what the network that ended epoch 5 scores on its own training images:

start = time.perf_counter()
for x, y in train_loader:
    pass
print(f"{time.perf_counter() - start:.1f}")
in_order = DataLoader(train_set, batch_size=1000)
model.eval()
total = 0.0
with torch.no_grad():
    for x, y in in_order:
        total += loss_fn(model(x), y).item() * len(y)
model.train()
print(f"{total / len(train_set):.4f}", f"{train_loss:.4f}")

4.3
0.3053 0.3224

Reading the 782 batches with no network at all took 4.3 seconds, about two thirds of an epoch here, so much of what an epoch costs is reading the images, whatever network reads them. On all 50,000 training images in evaluation mode the network that ended epoch 5 scores 0.3053, below the 0.3224 train_epoch returned for that epoch, a mean that includes batches scored before most of the epoch's 782 updates. in_order is a second loader over the same images without shuffle, so reading it draws nothing from train_loader's generator.

The losses and accuracies are what torch 2.14.1 printed here on the eight threads it uses by default on this machine, and again on two threads and on six. Run on one thread and on four, the same five epochs printed the same first three lines, then 4 0.3437 0.8690 and 5 0.3226 0.8750: on the very first batch the output layer's gradient computed on one or four threads differs from the eight-thread one by at most 3.73e-09, and by the fourth epoch the difference reaches the printed digits, the thread count moving a run's last digits as it moved the end of Lesson 4.4.1's memorizing run. Every number this lesson's blocks print from a trained model is this machine's on eight threads, the last subtitle's test score included, which the network trained on one thread or on four puts at 0.8667, and your last digits can differ from both.

DON'T:

• DON'T report train_epoch's number as the trained network's loss. It is the mean of 782 batch losses taken along the way, 0.3224, where the network that ended the epoch scores 0.3053 on the same 50,000 images.
• DON'T credit an epoch's seconds to the network alone. Reading the images with no network took 4.3 of them here.
• DON'T read last digits that differ from these as a fault in your loop before you count your threads. The same five epochs printed 0.8750 at epoch 5 on one thread and on four, and 0.8725 on two, six and eight.

A Linear Model on Pixels, for Comparison

What. nn.Sequential(nn.Flatten(), nn.Linear(784, 10)), every pixel weighted straight into the ten logits with no hidden layer, 7,850 parameters created after torch.manual_seed(42), trained by train_epoch for five epochs at the same rate and saved as fashion_linear.pt. Why. The network's 0.8725 says what the network does and nothing about what its hidden layer does, until the model without the hidden layer is measured on the same images in the same loop, the comparison Lesson 4.3.1 made with nn.Linear(18, 1) beside its network. When. Before any sentence about what a hidden layer buys, in a script of its own, so the model reads the same 3,910 batches in the same order the network read.

Save it as linear_pixels.py and run it after the network's script:

import torch
from torch import nn, optim
from fashion import train_loader, val_loader
from image_loop import accuracy, train_epoch

torch.manual_seed(42)
linear = nn.Sequential(nn.Flatten(), nn.Linear(784, 10))
print(linear)
print(sum(p.numel() for p in linear.parameters()))
optimizer = optim.Adam(linear.parameters(), lr=0.001)
for epoch in range(1, 6):
    train_loss = train_epoch(linear, optimizer, train_loader)
    print(epoch, f"{train_loss:.4f}", f"{accuracy(linear, val_loader):.4f}")
torch.save(linear.state_dict(), "fashion_linear.pt")

Sequential(
  (0): Flatten(start_dim=1, end_dim=-1)
  (1): Linear(in_features=784, out_features=10, bias=True)
)
7850
1 0.6700 0.8169
2 0.4895 0.8389
3 0.4551 0.8454
4 0.4375 0.8431
5 0.4243 0.8510

784 weights and a bias per class, and after five epochs a validation accuracy of 0.8510 against the network's 0.8725. The hidden layer called 215 more of the 10,000 validation images right, about two points, with 101,770 parameters against 7,850, about thirteen times as many, and the linear model's training loss ends higher too, 0.4243 against 0.3224. Its validation accuracy fell once, from 0.8454 at epoch 3 to 0.8431 at epoch 4, and passed both at epoch 5. Each script imports fashion afresh, so each train_loader starts from its generator's seed and both models read the same five shuffles. fashion_mlp.pt and fashion_linear.pt stay in ~/dev/bootcamp, and Lesson 4.5.3 loads both.

DON'T:

• DON'T dismiss the linear model before it is measured. With 7,850 parameters it called 8,510 of the 10,000 validation images right, and the hidden layer's whole gain is 215 of them.
• DON'T train the comparison in the network's script. train_loader keeps one generator for the whole script, so a second model trained there reads other shuffles than the first, and the difference between the two then carries the order too.
• DON'T compare the two on the test images. Picking one of them is a choice, and the validation images make it.

The Test Set, Scored Once, and the Classes It Confuses

What. The test score is the share of the 10,000 test images the chosen network calls right, taken once, after the validation images chose the network over the linear model, from fashion_mlp.pt loaded into mlp(). The same 10,000 predictions are then read class by class: each class's accuracy, the ten-class confusion matrix of Lesson 3.4.2, one image's logits and softmax, and the images the network got wrong, drawn. Why. The validation images were read after every epoch and chose between two models, so they are no longer new to the result, and the test images reached no step and no choice. One accuracy over ten classes is also an average of ten that can sit far apart, and the confusion matrix says which classes take each other's images. When. Once, after the last choice, from a script that loads the saved network, and never again to choose anything. Lesson 4.5.3 scores this network on the same images again, in place and moved, to ask a different question.

import matplotlib.pyplot as plt
import torch
from sklearn.metrics import confusion_matrix
from fashion import test_set, test_loader
from image_loop import mlp

model = mlp()
print(model.load_state_dict(torch.load("fashion_mlp.pt", weights_only=True)))
model.eval()
with torch.no_grad():
    logits = torch.cat([model(x) for x, y in test_loader])
labels = test_set.targets
pred = logits.argmax(dim=1)
print(logits.shape, (pred == labels).sum().item())
print(f"{(pred == labels).float().mean().item():.4f}")

<All keys matched successfully>
torch.Size([10000, 10]) 8677
0.8677

test_loader does not shuffle, so its ten batches arrive in the order of test_set.targets, and torch.cat, Lesson 4.1.2's, joins their logits into one (10000, 10) tensor that every number below reads: the test images passed through the network once. 8,677 of 10,000, 0.8677, on images no choice was made on, a little under the validation images' 0.8725. Class by class:

names = test_set.classes
for k, name in enumerate(names):
    mine = labels == k
    print(name, f"{(pred[mine] == k).float().mean().item():.3f}")

T-shirt/top 0.814
Trouser 0.969
Pullover 0.834
Dress 0.849
Coat 0.776
Sandal 0.956
Shirt 0.623
Sneaker 0.927
Bag 0.970
Ankle boot 0.959

Each line is the share of a class's 1,000 test images the network called right, Lesson 3.4.2's recall with that class as the positive one. Bag at 0.970 and Trouser at 0.969 are the best, and Shirt at 0.623 the worst, 377 shirts in 1,000 called something else. Because the ten classes are the same size, 0.8677 is exactly the mean of these ten lines, and it describes none of them. Where the mistakes went:

matrix = confusion_matrix(labels, pred)
print(matrix)
off = [(matrix[i, j], names[i], names[j]) for i in range(10) for j in range(10) if i != j]
for count, truth, called in sorted(off, reverse=True)[:4]:
    print(count, truth, "called", called)
print(matrix[:, 2].sum(), matrix[2, 2])

[[814   1  30  32   3   1 112   0   7   0]
 [  2 969   2  19   3   0   3   0   2   0]
 [ 16   1 834   6  92   0  49   0   2   0]
 [ 19  13  22 849  61   0  31   0   5   0]
 [  0   1 133  14 776   0  74   0   2   0]
 [  0   0   0   1   0 956   0  19   2  22]
 [122   3 151  26  62   0 623   0  13   0]
 [  0   0   0   0   0  43   0 927   0  30]
 [  4   1   6   4   4   4   3   4 970   0]
 [  0   0   0   0   0   6   1  34   0 959]]
151 Shirt called Pullover
133 Coat called Pullover
122 Shirt called T-shirt/top
112 T-shirt/top called Shirt
1178 834

Lesson 3.4.2's layout on ten classes: a row per true class and a column per call, both in label order, the diagonal the 8,677 right. The four largest cells off the diagonal are all among the tops: shirts called pullovers, coats called pullovers, shirts called T-shirts and T-shirts called shirts, each over a hundred of a thousand. The three shoes' 158 mistakes are other shoes in all but four, and trousers and bags are mistaken 31 and 30 times in 1,000. Down the Pullover column the network made 1,178 Pullover calls, 834 of them right, Lesson 3.4.2's precision read off a column, so a Pullover call is right less often than a pullover is found. The first test image, as the network sees it:

print(names[labels[0]])
print(logits[0])
print([round(p, 3) for p in torch.softmax(logits[0], dim=0).tolist()])

Ankle boot
tensor([ -6.9169, -11.1811,  -7.0057,  -7.5771,  -7.8688,   2.2375,  -5.6359,
          2.2349,  -0.7523,   4.3632])
[0.0, 0.0, 0.0, 0.0, 0.0, 0.096, 0.0, 0.096, 0.005, 0.804]

The largest of the ten logits, 4.3632, sits at index 9, the boot's, and the softmax puts 0.804 there, 0.096 each on Sandal and Sneaker, the other two shoes, and 0.005 or less on every other class. The images it got wrong, drawn with Lesson 4.5.1's imshow on Lesson 2.7.2's row of Axes:

wrong = torch.arange(len(labels))[pred != labels]
print(len(wrong), wrong[:4])
fig, axes = plt.subplots(1, 4, figsize=(10, 3))
for ax, i in zip(axes, wrong[:4].tolist()):
    ax.imshow(test_set[i][0].squeeze(0), cmap="gray")
    ax.set_title(f"{names[labels[i]]}\ncalled {names[pred[i]]}")
fig.savefig("mistakes.png", dpi=150, bbox_inches="tight")
print([len(ax.images) for ax in axes])
print([ax.get_title() for ax in axes])
plt.close(fig)

1323 tensor([12, 17, 23, 25])
[1, 1, 1, 1]
['Sneaker\ncalled Sandal', 'Coat\ncalled Pullover', 'Ankle boot\ncalled Sandal', 'Coat\ncalled Pullover']

pred != labels is a mask over the 10,000, and indexing torch.arange with it keeps the positions of the 1,323 mistakes, 10,000 less 8,677. Each Axes holds one image and its title names the truth over the call, so mistakes.png shows two shoes taken for sandals and two coats taken for pullovers, the confusions the matrix counted.

DON'T:

• DON'T score the test images after every epoch. Five readings make them a second validation set, and the epoch picked by the highest reading is a choice the test images made, Lesson 3.1.2's second break.
• DON'T report 0.8677 without the classes behind it. The same network called 969 of 1,000 trousers right and 623 of 1,000 shirts.
• DON'T read a column's total as its class's count. The network made 1,178 Pullover calls for 1,000 pullovers, and 344 of those calls were wrong.

Lab Exercises

Lab Exercise 1: A Wider Network at Its Seeded Start

In wide_start.py, take the first batch of val_loader and print its two shapes. Flatten it with flatten(1), with nn.Flatten() and with view given the batch's length and -1, print the shape once and whether the three results are equal, and prove that row 14 of the first image is positions 392 to 419 of its flat row. Print the message a nn.Linear(784, 256) raises for the batch unflattened.

Then build nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10)) after torch.manual_seed(42), and print each parameter's name, shape and count, then the total and the first layer's weight's share of it in percent to one decimal. Run the seeded network on the batch under torch.no_grad() and print its loss beside ln 10, the smallest and the largest probability its softmax gives, how many of the 1,000 images it calls each class, and the share it calls right.

Lab Exercise 2: Thirty-Two Hidden Units, Read on the Validation Images

In narrow.py, train nn.Sequential(nn.Flatten(), nn.Linear(784, 32), nn.ReLU(), nn.Linear(32, 10)), built after torch.manual_seed(42), with optim.Adam at 0.001 for five epochs through train_epoch, printing its parameter count first and then the epoch, the training loss and the validation accuracy. Then run it once over the 10,000 validation images in evaluation mode under torch.no_grad(), keeping the logits and the labels as two tensors, and print from them the validation accuracy, each class's count of images and accuracy, the four largest cells off the confusion matrix's diagonal, and the class it calls most often with its number of calls. Leave the test images alone, and say in a comment what the 32 units cost and bought against the 128-unit network and the linear model.

import math
import torch
from torch import nn
from fashion import val_loader

images, labels = next(iter(val_loader))
print(images.shape, labels.shape)
flat = images.flatten(1)
same = torch.equal(flat, nn.Flatten()(images)) and torch.equal(flat, images.view(len(images), -1))
print(flat.shape, same)
print(torch.equal(flat[0, 392:420], images[0, 0, 14]))
try:
    nn.Linear(784, 256)(images)
except RuntimeError as err:
    print(err)

torch.manual_seed(42)
wide = nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 10))
for name, p in wide.named_parameters():
    print(name, tuple(p.shape), p.numel())
total = sum(p.numel() for p in wide.parameters())
print(total, round(100 * wide[1].weight.numel() / total, 1))
with torch.no_grad():
    logits = wide(images)
probs = torch.softmax(logits, dim=1)
loss = nn.CrossEntropyLoss()(logits, labels)
print(f"{loss.item():.4f}", f"{math.log(10):.4f}")
print(f"{probs.min().item():.4f}", f"{probs.max().item():.4f}")
calls = logits.argmax(dim=1)
print([(calls == k).sum().item() for k in range(10)])
print(f"{(calls == labels).float().mean().item():.4f}")

torch.Size([1000, 1, 28, 28]) torch.Size([1000])
torch.Size([1000, 784]) True
True
mat1 and mat2 shapes cannot be multiplied (28000x28 and 784x256)
1.weight (256, 784) 200704
1.bias (256,) 256
3.weight (10, 256) 2560
3.bias (10,) 10
203530 98.6
2.3096 2.3026
0.0724 0.1350
[0, 27, 222, 101, 118, 0, 1, 6, 2, 523]
0.0480

import torch
from torch import nn, optim
from sklearn.metrics import confusion_matrix
from fashion import train_full, train_loader, val_loader
from image_loop import accuracy, train_epoch

torch.manual_seed(42)
narrow = nn.Sequential(nn.Flatten(), nn.Linear(784, 32), nn.ReLU(), nn.Linear(32, 10))
print(sum(p.numel() for p in narrow.parameters()))
optimizer = optim.Adam(narrow.parameters(), lr=0.001)
for epoch in range(1, 6):
    train_loss = train_epoch(narrow, optimizer, train_loader)
    print(epoch, f"{train_loss:.4f}", f"{accuracy(narrow, val_loader):.4f}")

narrow.eval()
outputs, truths = [], []
with torch.no_grad():
    for x, y in val_loader:
        outputs.append(narrow(x))
        truths.append(y)
narrow.train()
pred = torch.cat(outputs).argmax(dim=1)
labels = torch.cat(truths)
print(len(labels), f"{(pred == labels).float().mean().item():.4f}")
names = train_full.classes
for k, name in enumerate(names):
    mine = labels == k
    print(name, mine.sum().item(), f"{(pred[mine] == k).float().mean().item():.3f}")
matrix = confusion_matrix(labels, pred)
off = [(matrix[i, j], names[i], names[j]) for i in range(10) for j in range(10) if i != j]
for count, truth, called in sorted(off, reverse=True)[:4]:
    print(count, truth, "called", called)
calls = [(pred == k).sum().item() for k in range(10)]
print(names[calls.index(max(calls))], max(calls))
# 25,450 parameters, a quarter of the 128-unit network's 101,770,
# called 8,605 validation images right: 95 more than the linear
# model's 8,510 and 120 fewer than the 128-unit network's 8,725.

25450
1 0.6675 0.8199
2 0.4582 0.8480
3 0.4198 0.8496
4 0.3979 0.8550
5 0.3808 0.8605
10000 0.8605
T-shirt/top 1074 0.770
Trouser 1027 0.972
Pullover 985 0.836
Dress 945 0.866
Coat 974 0.792
Sandal 999 0.922
Shirt 1033 0.625
Sneaker 952 0.942
Bag 979 0.965
Ankle boot 1032 0.929
155 T-shirt/top called Shirt
139 Shirt called Pullover
121 Shirt called T-shirt/top
106 Coat called Pullover
Pullover 1131

Theory Assessment

Q1: Three Shapes and Two Messages

A batch from train_loader has the shape (64, 1, 28, 28). A teammate hands it to nn.Linear(784, 128) as it is and gets mat1 and mat2 shapes cannot be multiplied (1792x28 and 784x128), flattens it with images.flatten() and gets a message naming 1x50176, then tries images.view(64, 784) and images.squeeze(1).flatten(1), which both work. Say where 1,792 and 50,176 come from, why the last two calls work and return what nn.Flatten() returns, and what all four lose about the picture that no message reports.

Hint: Ask which dimension each call starts flattening at and which dimensions nn.Linear takes as rows. Then follow one pixel and its neighbor below into the flat row.

Q2: The Softmax at the End

A teammate wants the network to return probabilities, wraps it as nn.Sequential(mlp(), nn.Softmax(dim=1)), checks that its seeded start scores 2.3011 on a batch, close to ln 10, and trains it with the lesson's train_epoch and accuracy:

import torch
from torch import nn, optim
from fashion import train_loader, val_loader
from image_loop import mlp, accuracy, train_epoch

trapped = nn.Sequential(mlp(), nn.Softmax(dim=1))
optimizer = optim.Adam(trapped.parameters(), lr=0.001)
for epoch in range(1, 6):
    train_loss = train_epoch(trapped, optimizer, train_loader)
    print(epoch, f"{train_loss:.4f}", f"{accuracy(trapped, val_loader):.4f}")

1 1.7579 0.7568
2 1.6849 0.7847
3 1.6725 0.7894
4 1.6670 0.7956
5 1.6621 0.8010

They report that the probability model trains more slowly to about the same place. Say what loss_fn inside train_epoch receives, why the second column can never print a number below 1.4612, why accuracy still measures the network fairly, what the five epochs cost against the network trained on logits, and how the teammate should have got probabilities.

Hint: Ask what the loss does to whatever it is handed before it takes the log, and what the largest gap between two numbers in a row that sums to 1 can be. For accuracy, ask whether a softmax can change which of ten numbers is the largest.

Q3: The Comparison in One Script

To save a run, a teammate trains the linear model in the network's script, after the network's five epochs, with torch.manual_seed(42) before it as the lesson has it:

import torch
from torch import nn, optim
from fashion import train_loader, val_loader
from image_loop import mlp, accuracy, train_epoch

model = mlp()
optimizer = optim.Adam(model.parameters(), lr=0.001)
for epoch in range(5):
    train_epoch(model, optimizer, train_loader)
torch.manual_seed(42)
linear = nn.Sequential(nn.Flatten(), nn.Linear(784, 10))
optimizer = optim.Adam(linear.parameters(), lr=0.001)
for epoch in range(1, 6):
    train_loss = train_epoch(linear, optimizer, train_loader)
    print(epoch, f"{train_loss:.4f}", f"{accuracy(linear, val_loader):.4f}")

1 0.6707 0.8231
2 0.4893 0.8260
3 0.4555 0.8389
4 0.4356 0.8514
5 0.4239 0.8493

They report the hidden layer's gain as 0.8725 less 0.8493, 232 validation images. Say why the linear model printed other numbers at every epoch from the same seed and the same loop, which of the two linear runs the network should be compared with and why, and what the 17 images between 0.8493 and 0.8510 measure.

Hint: Ask which object in the script draws the shuffles, and how many shuffles it had drawn by the time the linear model started. Then ask what a comparison has to hold fixed for its difference to belong to the hidden layer.

Q4: The File That Scored One in Ten

A teammate serves the saved network from a script of their own. They build it without nn.Flatten(), because they flatten the images themselves, and since the plain load was refused they pass strict=False:

import torch
from torch import nn
from fashion import val_loader

torch.manual_seed(42)
served = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10))
state = torch.load("fashion_mlp.pt", weights_only=True)
print(list(state))
print(served.load_state_dict(state, strict=False))
images, labels = next(iter(val_loader))
with torch.no_grad():
    calls = served(images.flatten(1)).argmax(dim=1)
print(f"{(calls == labels).float().mean().item():.4f}")

['1.weight', '1.bias', '3.weight', '3.bias']
_IncompatibleKeys(missing_keys=['0.weight', '0.bias', '2.weight', '2.bias'], unexpected_keys=['3.weight', '3.bias', '1.weight', '1.bias'])
0.1140

Their report says the file is corrupt, since the saved network calls 114 of 1,000 validation images right. Say whether the file is corrupt, what the second line reports and why both networks hold 101,770 parameters each yet share no name, what network actually called the 1,000 images, and two ways to serve the file correctly, one that keeps the teammate's network.

Hint: nn.Flatten() holds no parameter, and a layer's name is its position. Ask what strict=False loads when no key matches, and what the parameters of served are after a load that loaded nothing.

renamed = {f"{int(key[0]) - 1}{key[1:]}": value for key, value in state.items()}
print(served.load_state_dict(renamed))
with torch.no_grad():
    calls = served(images.flatten(1)).argmax(dim=1)
print(f"{(calls == labels).float().mean().item():.4f}")

<All keys matched successfully>
0.8830

Assignment

Assignment: Two Hidden Layers, Stopped by Their Validation Accuracy

Five epochs was a budget, and nothing measured whether a sixth would help. Write stop_deep.py. At its top level it defines deep(), which seeds 42 and returns nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 128), nn.ReLU(), nn.Linear(128, 10)), and val_loss(model), the mean cross-entropy over the 10,000 validation images in evaluation mode and without a graph. Under its main guard it prints the network's parameter count and trains it with optim.Adam at 0.001 through train_epoch for at most 20 epochs, printing each epoch's training loss, validation loss and validation accuracy, and it stops by Lesson 4.4.2's patience rule read on the validation accuracy instead of the loss, with a patience of 3: a deep copy of the state dict at every new highest accuracy, and a stop once 3 epochs have passed without one. After the loop it prints the epoch it stopped at, the kept epoch with its accuracy and the epoch of the lowest validation loss with that loss, then restores the kept state and saves it as fashion_deep.pt.

Then write check_deep.py, a second script that imports deep from stop_deep without training anything, loads fashion_deep.pt, and prints the load report, the validation accuracy the file gives, how many validation images the network gets wrong, and the five of those it called with the highest probability: each one's position among the 10,000, its class, its call and the probability on the call. Leave the test images alone.

Two calls this lesson has not used. nn.CrossEntropyLoss(reduction="sum") adds the losses of a batch's images instead of averaging them, so its total over a loader divided by the number of images is the mean over every image, whatever the batches' sizes. torch.topk(values, k) returns the k largest entries of a one-dimensional tensor and their positions in it, largest first, as .values and .indices.

Hints:

• The main guard, Lesson 1.7.2's, is what lets the second script import deep without training a network, and the patience test is Lesson 4.4.2's elif with the comparison turned around, a higher accuracy where that lesson wanted a lower loss.
• .max(dim=1) on the softmax's rows returns each image's highest probability and its call as two tensors, Lesson 4.1.2's named pair. torch.topk ranks the wrong ones' probabilities, and its .indices are positions among the wrong ones, not among the 10,000.

import copy
import torch
from torch import nn, optim
from fashion import train_loader, val_loader
from image_loop import accuracy, train_epoch

summed = nn.CrossEntropyLoss(reduction="sum")

def deep():
    torch.manual_seed(42)
    return nn.Sequential(nn.Flatten(), nn.Linear(784, 256), nn.ReLU(), nn.Linear(256, 128), nn.ReLU(), nn.Linear(128, 10))

def val_loss(model):
    model.eval()
    total = 0.0
    with torch.no_grad():
        for x, y in val_loader:
            # the batch's losses added, not averaged
            total += summed(model(x), y).item()
    model.train()
    return total / len(val_loader.dataset)

if __name__ == "__main__":
    model = deep()
    print(sum(p.numel() for p in model.parameters()))
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    best_acc, best_epoch, best_state = 0.0, 0, None
    losses = []
    for epoch in range(1, 21):
        train_loss = train_epoch(model, optimizer, train_loader)
        losses.append(val_loss(model))
        acc = accuracy(model, val_loader)
        print(epoch, f"{train_loss:.4f}", f"{losses[-1]:.4f}", f"{acc:.4f}")
        # a higher accuracy is the new best, kept as a copy
        if acc > best_acc:
            best_acc, best_epoch = acc, epoch
            best_state = copy.deepcopy(model.state_dict())
        elif epoch - best_epoch >= 3:
            break
    lowest = losses.index(min(losses)) + 1
    print(epoch, best_epoch, f"{best_acc:.4f}", lowest, f"{min(losses):.4f}")
    model.load_state_dict(best_state)
    torch.save(model.state_dict(), "fashion_deep.pt")

235146
1 0.5496 0.4520 0.8350
2 0.3832 0.3807 0.8620
3 0.3433 0.3513 0.8729
4 0.3170 0.3617 0.8690
5 0.2999 0.3310 0.8787
6 0.2834 0.3428 0.8776
7 0.2683 0.3530 0.8747
8 0.2565 0.3227 0.8818
9 0.2431 0.3120 0.8915
10 0.2328 0.3163 0.8861
11 0.2261 0.3251 0.8883
12 0.2182 0.3215 0.8879
12 9 0.8915 9 0.3120

import torch
from fashion import train_full, val_loader
from image_loop import accuracy
from stop_deep import deep

model = deep()
print(model.load_state_dict(torch.load("fashion_deep.pt", weights_only=True)))
print(f"{accuracy(model, val_loader):.4f}")
model.eval()
outputs, truths = [], []
with torch.no_grad():
    for x, y in val_loader:
        outputs.append(model(x))
        truths.append(y)
sure, called = torch.softmax(torch.cat(outputs), dim=1).max(dim=1)
labels = torch.cat(truths)
wrong = torch.arange(len(labels))[called != labels]
top = torch.topk(sure[wrong], 5)
names = train_full.classes
print(len(wrong))
for i in wrong[top.indices].tolist():
    print(i, names[labels[i]], "called", names[called[i]], f"{sure[i].item():.4f}")

<All keys matched successfully>
0.8915
1085
9112 Ankle boot called Sandal 1.0000
1975 Pullover called Dress 0.9999
4636 T-shirt/top called Shirt 0.9998
7512 Sandal called Ankle boot 0.9996
3169 Trouser called Dress 0.9996

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the same per-class reading done by a convolutional network, on color images or on these same clothes.

• Official docs - PyTorch Tutorials, "Training a Classifier", the section "5. Test the network on the test data". A small convolutional network trained for two passes over CIFAR-10's color images, 3 by 32 by 32 pixels in ten classes, scored on its 10,000 test images against the 10 percent of picking a class at random, then counted one prediction at a time into an accuracy for each of the ten classes.
• Official docs - PyTorch Tutorials, "Visualizing Models, Data, and Training with TensorBoard", the section "Assessing trained models with TensorBoard". Fashion-MNIST read by a small convolutional network, and in this last section the softmax probabilities of every test image collected and drawn in TensorBoard as one precision-recall curve per class, some classes near the whole area under the curve and others lower, after earlier sections that log the running loss and a figure of four predictions while the model trains.

Summary

The loop did not change. Lesson 4.3.3's loop ran on batches of images as it ran on batches of listings, Lesson 4.3.2's loss took ten logits where it once took two, and the one new layer, nn.Flatten(), holds no parameter at all. What changed was the shape of a row, and the shape decided most of the rest: 784 numbers per image put 98.6 percent of the network into one matrix, a batch in the wrong shape was refused in a message about rows of 28, and the grid in which every pixel has neighbors became a row in which none does.

The numbers that mattered came from asking the same network different questions. The loss beside ln 10 said the start had learned nothing yet, the validation accuracy said each epoch helped, the linear model trained on the same batches in the same order said what the hidden layer was worth, 215 validation images, and the one test score said what all of it was worth on images no choice touched. The per-class view then took that 0.8677 apart into a trouser class near 0.97 and a shirt class near 0.62, and the confusion matrix showed the tops trading their images among themselves.

Quieter facts came with the figures. A softmax at the end of a network passes the check at the start and caps what training can do, and the last digits of a five-epoch run belong partly to the processor's threads. What the network reads is fixed by the flattening: a weight for every position, and no word for two positions that touch. What that costs is the section's last question.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

image.flatten()       # (784,), row after row
images.flatten(1)     # (64, 784), the batch kept
nn.Flatten()(images)  # (64, 784), start_dim=1
images.flatten()      # (50176,), start_dim=0

model = mlp()                              # seeded 42 inside
optimizer = optim.Adam(model.parameters(), lr=0.001)
for epoch in range(1, 6):
    train_loss = train_epoch(model, optimizer, train_loader)
    val_acc = accuracy(model, val_loader)  # eval, no_grad, train
torch.save(model.state_dict(), "fashion_mlp.pt")

model = mlp()
model.load_state_dict(torch.load("fashion_mlp.pt", weights_only=True))
model.eval()
with torch.no_grad():
    logits = torch.cat([model(x) for x, y in test_loader])
pred, labels = logits.argmax(dim=1), test_set.targets
(pred == labels).float().mean()          # once, 0.8677
(pred[labels == 6] == 6).float().mean()  # Shirt, 0.623
confusion_matrix(labels, pred)           # rows truth, columns call