torchvision and Image Tensors

Context

An image tensor is a picture written as numbers, one per pixel (one square of a digital picture's grid) and channel (one full grid of values, a single brightness grid for a grayscale picture and three, red, green and blue, for a color one), held as channels by height by width. torchvision (PyTorch's library of image datasets, transforms and models) downloads collections of pictures and turns each into such a tensor as a script reads it.

Every model that reads pictures is fed this way: a listings site sorting its sellers' photos into kitchens, bathrooms and floor plans, a bank reading handwritten amounts on scanned checks. The folder the files land in, the validation images carved from the training images and the batches a loader stacks are the moves Module 3 and Section 4.3 made on a table, made here on 70,000 pictures of clothing.

Most of what breaks here prints nothing. A script that names a folder of its own downloads the 30.9 MB again, the raw grid read in place of the converted image hands a model values up to 255 where the image holds 1, and numbers taken from one picture turn a dark boot and a brighter copy of it into the same values. A transform handed a tensor instead of a picture at least stops the run. Lesson 4.3.3's loader sliced a tensor that held every listing, where this lesson's builds each image when a batch asks for it.

Goal

• Install torchvision 0.29.1 into (bootcamp) from PyTorch's processor index, and explain why uv adds one package where it resolved thirteen.
• Download FashionMNIST once into data through a module later scripts import, and predict what download=True does when the files are there and when the folder differs.
• Read an image as a (1, 28, 28) float32 tensor with a class index beside it, and distinguish it from the raw uint8 grid in .data that ToTensor converts.
• Draw images with imshow, one alone and one per class on a grid, and check each chart by printing what it holds.
• Compute the fit images' pixel mean and standard deviation, apply them with Normalize inside Compose, and show what numbers from one image destroy.
• Carve 10,000 validation images out of the 60,000 with a seeded random_split, and read the shapes, dtypes and batch counts of the loaders built on the carve.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see an image dataset loaded and turned into tensors before you do it yourself.

• YouTube - "PyTorch for Deep Learning & Machine Learning - Full Course" by freeCodeCamp.org, developed by Daniel Bourke. The link opens chapter 93, "Computer vision input and outputs", at 14:12:36, and the 49 minutes from there to chapter 98, "Mini-batches", hold the chapters the page lists as 94 to 96, "What is a convolutional neural network?", "TorchVision" and "Getting a computer vision dataset".
• Official docs - Torchvision 0.29, ToTensor. The conversion fashion.py's datasets put every image through: a PIL image or a NumPy array, height by width by channels in the range 0 to 255, becomes a float tensor, channels by height by width in the range 0.0 to 1.0, for the image modes the page lists and for uint8 arrays, and any other input comes back unscaled.

Table of Contents

• Installing torchvision
• FashionMNIST, Downloaded Once
• An Image Is a Tensor of Shape (1, 28, 28)
• Looking at an Image
• Normalizing Pixels Like Any Feature
• Batches of Images, and the Validation Split

Theory

Every Python block in this lesson runs in ~/dev/bootcamp with (bootcamp) active, once the first subtitle has installed torchvision. The second subtitle writes fashion.py, whose datasets, carve and loaders later scripts import, as Lesson 4.3.3's scripts imported the two files Lesson 4.3.1 saved. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Installing torchvision

What. torchvision is installed beside torch and imported as torchvision. torchvision==0.29.1 is the release this lesson ran on, and its package lists torch>=2.14.0 among what it needs, so it pairs with Lesson 4.1.1's torch 2.14.1. Why. torch holds tensors, autograd, layers and loaders, and nothing that knows a picture file or a dataset of pictures. torchvision holds the image datasets, the conversion from a picture to a tensor and the transforms that change pictures, written for torch's tensors. When. Once, now, with (bootcamp) active, from the index Lesson 4.1.1's torch came from. It is the second of the two libraries Module 4's setup page names.

Before the install, (bootcamp) has no torchvision to import:

(bootcamp) goku@machine:~/dev/bootcamp$ python3 -c "import torchvision"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'torchvision'

In ~/dev/bootcamp, with (bootcamp) active:

uv pip install torchvision==0.29.1 --index-url https://download.pytorch.org/whl/cpu

Among the lines uv prints, the first reads Resolved 13 packages with the seconds it took, and one starts with +:

 + torchvision==0.29.1+cpu

One line, where Lesson 4.1.1's install printed ten. torchvision's package names three libraries it needs: numpy, torch and pillow (Python's imaging library, which opens, converts and saves pictures and is imported as PIL). numpy and pillow came with Module 2's libraries, pillow as one that matplotlib needs, and torch came with its nine in Lesson 4.1.1, so uv resolved thirteen, found twelve installed and added torchvision alone, a 1.7 MB download on WSL against torch's 196 MB processor wheel.

Confirm the pair from a script, check_torchvision.py:

import torch
import torchvision

print(torchvision.__version__)  # 0.29.1+cpu
print(torch.__version__)        # 2.14.1+cpu

On WSL and Linux both carry +cpu, the processor build. On a Mac with Apple's chip the index's torchvision wheel for Python 3.12 is named torchvision-0.29.1-cp312-cp312-macosx_14_0_arm64.whl, with no +cpu, so by that name a Mac prints 0.29.1, as it prints 2.14.1 for torch.

DON'T:

• DON'T read the single + line as half an install. Twelve of the thirteen packages uv resolved, numpy, pillow, torch and torch's nine, were in (bootcamp) before the line ran.
• DON'T uninstall pillow because no script of yours imports it. torchvision's package needs it, and this lesson's datasets hand every image to its transform as a pillow picture, as the third subtitle prints.

FashionMNIST, Downloaded Once

What. datasets.FashionMNIST(root, train=..., download=..., transform=...) is torchvision's class for FashionMNIST, 70,000 grayscale pictures of clothing, 28 by 28 pixels, in ten classes, 60,000 for training and 10,000 for testing. root is the folder the files live under, train picks the set, and download=True fetches the files into root when they are missing and nothing when they are there. Why. The pictures are files outside Python, 30.9 MB over the network, and a script that downloads into a folder of its own pays that again. One folder, named in one file that later scripts import, is the dataset downloaded once. When. On the first run of fashion.py on a machine, with download=True left in the file for every run after it.

Its own repository calls it Zalando's article images, released by Zalando Research under the MIT license as a drop-in replacement for MNIST, an older set of handwritten digits of the same size and split. Clothing is not what a listings site sells, and that is the point: every picture is 28 by 28, so any 64 stack into one tensor, and all 70,000 fit in memory, Lesson 2.5.2's condition for a table before anything trains on it.

fashion.py builds both datasets in a folder named data beside your scripts, each image passed through a transform (a function applied to every image as it is read), transforms.ToTensor(), the next subtitle's subject. Its next lines carve 10,000 validation images out of the training images with a seeded generator and build three loaders, the last subtitle's subject, so every script that imports it shares one carve. Save it in ~/dev/bootcamp and run python3 fashion.py, whose main guard, Lesson 1.7.2's, prints the three set sizes and the batches each loader holds:

import torch
from torch.utils.data import DataLoader, random_split
from torchvision import datasets, transforms

train_full = datasets.FashionMNIST("data", train=True, download=True, transform=transforms.ToTensor())
test_set = datasets.FashionMNIST("data", train=False, download=True, transform=transforms.ToTensor())
# 50,000 images to fit, 10,000 to choose on, the same draw every run
train_set, val_set = random_split(train_full, [50_000, 10_000], generator=torch.Generator().manual_seed(42))

train_loader = DataLoader(train_set, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
val_loader = DataLoader(val_set, batch_size=1000)
test_loader = DataLoader(test_set, batch_size=1000)

if __name__ == "__main__":
    print(len(train_set), len(val_set), len(test_set))
    print(len(train_loader), len(val_loader), len(test_loader))

50000 10000 10000
782 10 10

That is every run's output once the files are on disk. On the first run there is no data folder, and the first datasets.FashionMNIST line fetches all four of the dataset's files, the test set's two included, so the second line finds them. Before the two lines, the terminal shows one line per file, a percentage that climbs in place to 100.0:

(bootcamp) goku@machine:~/dev/bootcamp$ python3 fashion.py
100.0%
100.0%
100.0%
100.0%
50000 10000 10000
782 10 10

The first run's seconds are your connection's: extracting and reading the files took under 8 of them here. Where the files went:

import os

folder = "data/FashionMNIST/raw"
for name in sorted(os.listdir(folder)):
    print(name, os.path.getsize(f"{folder}/{name}"))

t10k-images-idx3-ubyte 7840016
t10k-images-idx3-ubyte.gz 4422102
t10k-labels-idx1-ubyte 10008
t10k-labels-idx1-ubyte.gz 5148
train-images-idx3-ubyte 47040016
train-images-idx3-ubyte.gz 26421880
train-labels-idx1-ubyte 60008
train-labels-idx1-ubyte.gz 29515

os.listdir returns a folder's names in no fixed order, which sorted fixes, and os.path.getsize a file's size in bytes. Each file fetched is a .gz archive (a compressed file), 30,878,645 bytes for the four, kept beside the file extracted from it, which is what the class reads, 54,950,048 bytes, 85.8 MB in all, and t10k names the 10,000 test images. The datasets:

import torch
from fashion import train_full, test_set

print(train_full)
print(len(train_full), len(test_set))
print(train_full.classes)
print(torch.bincount(train_full.targets))
print(torch.bincount(test_set.targets))

Dataset FashionMNIST
    Number of datapoints: 60000
    Root location: data
    Split: Train
    StandardTransform
Transform: ToTensor()
60000 10000
['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']
tensor([6000, 6000, 6000, 6000, 6000, 6000, 6000, 6000, 6000, 6000])
tensor([1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000])

A dataset prints its size, folder, set and transforms, StandardTransform being torchvision's holder for an image transform and a label transform, of which only the first is set. classes lists the ten names in the order of Lesson 4.3.2's class index, so 0 is a T-shirt or top and 9 an ankle boot. targets holds the 60,000 labels in one int64 tensor, and torch.bincount, Lesson 3.5.2's np.bincount for a tensor, finds 6,000 training images and 1,000 test images of every class, so always answering one class scores 0.1. The pixels are in memory too:

print(train_full.data.shape, train_full.data.dtype)
print(train_full.data.numel() * train_full.data.element_size())
print(test_set.data.numel() * test_set.data.element_size())

torch.Size([60000, 28, 28]) torch.uint8
47040000
7840000

train_full.data is every training picture in one uint8 tensor, read from the extracted file when the dataset was built, one byte a pixel, 47,040,000 bytes by Lesson 4.3.3's numel() times element_size(), and with the test set's 7,840,000 the 70,000 pictures take 54.9 MB.

DON'T:

• DON'T point a second script at a folder of its own. datasets.FashionMNIST("fashion_data", train=True, download=True) fetches all four files again into a second folder, and so does "data" in a script run from another folder, because a bare folder name is read from the folder the script runs in.
• DON'T take download=True out once the files are there. It fetches nothing on a later run, and without it a machine with no data folder stops at Dataset not found. You can use download=True to download it.

An Image Is a Tensor of Shape (1, 28, 28)

What. train_full[0] is a tuple of the first image and its label. The image is a float32 tensor of shape (1, 28, 28), channels by height by width, one channel and then 28 rows of 28 pixels, each from 0, black, to 1, white, and the label is a Python int, the class index. ToTensor (the transform that turns a picture into a float tensor, channels first, every value divided by 255) made it from the raw grid, which train_full.data keeps as uint8 from 0 to 255. Why. A model computes in float32, and one fitted by small steps cares about the scale of what it reads, Lesson 3.2.1's finding, where a raw pixel is a whole number up to 255. The channel dimension is where a color picture keeps its three grids, so a grayscale image carries it at size one, in front, where torchvision's transforms look for it. When. Every time a script reads dataset[i], a loader included.

import numpy as np
import torch
from torchvision import datasets, transforms
from fashion import train_full

item = train_full[0]
img, label = item
print(type(item).__name__, len(item))      # tuple 2
print(img.shape)                           # torch.Size([1, 28, 28])
print(img.dtype)                           # torch.float32
print(img.min().item(), img.max().item())  # 0.0 1.0
print(type(label).__name__, label)         # int 9
print(train_full.classes[label])           # Ankle boot
print(train_full.targets[:8])              # tensor([9, 0, 0, 3, 0, 2, 7, 2])

The raw grid beside the image:

raw = train_full.data[0]
print(raw.shape, raw.dtype, raw.max().item())
print(raw[10, 10:18])
print(img[0, 10, 10:18])
print(193 / 255)
print(torch.equal(img[0], raw / 255))

torch.Size([28, 28]) torch.uint8 255
tensor([  0,   0,   0, 193, 228, 218, 213, 198], dtype=torch.uint8)
tensor([0.0000, 0.0000, 0.0000, 0.7569, 0.8941, 0.8549, 0.8353, 0.7765])
0.7568627450980392
True

raw has no channel and holds whole numbers up to 255. Row 10 runs from black into the boot, 193 where the image reads 0.7569, 193 over 255 to four decimals, and torch.equal says all 784 values are their raw values over 255, exactly: ToTensor divided, and put a channel in front. What it was handed is not the tensor in .data, and a dataset built with no transform shows what it is:

plain = datasets.FashionMNIST("data", train=True)
picture, label = plain[0]
print(type(picture))               # <class 'PIL.Image.Image'>
print(picture.mode, picture.size)  # L (28, 28)
to_tensor = transforms.ToTensor()
# True
print(torch.equal(to_tensor(picture), img))

The dataset turns the stored grid into a pillow picture, pillow's Image in mode L, its one-channel mode, each time an item is read, and hands that to the transform, which is one thing torchvision needs pillow for. A color picture shows the other half of ToTensor's work:

color = np.stack([raw.numpy()] * 3, axis=2)
print(color.shape, to_tensor(color).shape)  # (28, 28, 3) torch.Size([3, 28, 28])

Three copies of the grid along a last dimension make an array laid out height by width by channels, and it comes out channels first. Two inputs it handles differently:

try:
    to_tensor(img)
except TypeError as err:
    print(err)
as_float = to_tensor(raw.numpy().astype(np.float32))
print(as_float.dtype, as_float.max().item())

pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>
torch.float32 255.0

ToTensor refuses a tensor, which every image read from train_full already is. It divides a uint8 array or an 8-bit picture by 255, and a float32 array comes back unscaled, 255.0 at its brightest, with no warning. torchvision also ships a second set of transforms, torchvision.transforms.v2, in which this step is two, v2.ToImage() and then v2.ToDtype(torch.float32, scale=True), whose result on the boot agrees with ToTensor's within float32 rounding. This lesson's transforms.ToTensor is the first set's, the one fashion.py uses, and runs on torchvision 0.29.1 without a warning.

DON'T:

• DON'T call ToTensor on an image the dataset already converted. to_tensor(img) stops with pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>.
• DON'T read .data as the image a model sees. train_full.data[0] is a (28, 28) uint8 grid with no channel, and its 193 is the image's 0.7569.
• DON'T turn the raw grid into floats before ToTensor. The float32 array came back unscaled, 255.0 at its brightest, and nothing said so.

Looking at an Image

What. ax.imshow(grid, cmap="gray") (matplotlib's function that draws a 2-D array as a picture, one square per value) draws on Lesson 2.7.2's fig, ax figure, the gray colormap running from black at the array's smallest value to white at its largest. It draws a (height, width) grid, or (height, width, 3) for color, so the image goes in as img.squeeze(0), Lesson 4.1.2's squeeze at the channel dimension. Why. A picture checks what 784 printed numbers cannot: that the label belongs to the image, the garment is the right way up and the data is what its name says. The chart itself is checked by printing what it holds, Lesson 2.7.2's habit. When. On every new dataset before anything trains on it, and on any image a model gets wrong.

import matplotlib.pyplot as plt
from fashion import train_full

img, label = train_full[0]
fig, ax = plt.subplots(figsize=(4, 4))
ax.imshow(img.squeeze(0), cmap="gray")
ax.set_title(train_full.classes[label])
ax.axis("off")
fig.savefig("first_image.png", dpi=150, bbox_inches="tight")
print(len(ax.images))                  # 1
print(ax.images[0].get_array().shape)  # (28, 28)
print(ax.images[0].get_cmap().name)    # gray
print(ax.images[0].get_clim())         # (0.0, 1.0)
print(ax.get_title())                  # Ankle boot
plt.close(fig)

first_image.png is the boot, light gray on black with its toe to the left, every pixel a visible square, and ax.axis("off") (hides the frame, the ticks and their labels) dropped the pixel counts that would line its edges. ax.images holds what imshow drew, as ax.lines held 2.7.2's lines, and get_clim(), 2.7.2's read-back of a color scale, gives the values the colormap's ends were set to, the image's own darkest and brightest, because imshow stretches the colormap over whatever it is handed. Two ways to draw it wrong:

fig, ax = plt.subplots()
try:
    ax.imshow(img, cmap="gray")
except TypeError as err:
    print(err)
ax.imshow(img.squeeze(0))
print(ax.images[0].get_cmap().name)
plt.close(fig)

Invalid shape (1, 28, 28) for image data
viridis

The channel-first image is refused, and the refusal adds nothing to the Axes. Without cmap, matplotlib uses its default colormap, viridis, dark purple at the smallest value to yellow at the largest, so the background turns purple and the boot yellow-green, the right shape in colors the data never held. One image of each class:

labels = train_full.targets.tolist()
first = [labels.index(c) for c in range(10)]
print(first)  # [1, 16, 5, 3, 19, 8, 18, 6, 23, 0]

fig, axes = plt.subplots(2, 5, figsize=(10, 4.5))
for ax, i in zip(axes.ravel(), first):
    image, label = train_full[i]
    ax.imshow(image.squeeze(0), cmap="gray")
    ax.set_title(train_full.classes[label])
    ax.axis("off")
fig.savefig("one_per_class.png", dpi=150, bbox_inches="tight")
# 10
print(sum(len(ax.images) for ax in axes.ravel()))
# ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat']
print([ax.get_title() for ax in axes.ravel()][:5])
plt.close(fig)

labels.index(c), Lesson 1.4.1's list method, finds the first position of class c. plt.subplots(2, 5) is 2.7.2's grid of Axes, and axes.ravel(), Lesson 2.2.1's ravel, lays the ten out in row order for zip. In one_per_class.png the T-shirt carries a dark band across its chest, the sandal is a thin outline of straps, the bag a patterned rectangle, and the sneaker and the boot both point left.

DON'T:

• DON'T hand imshow the (1, 28, 28) tensor. It is refused with Invalid shape (1, 28, 28) for image data, and img.squeeze(0) is the drawing's shape, never the image's.
• DON'T squeeze at the wrong dimension. img.squeeze(1) returns the (1, 28, 28) image unchanged, because squeeze removes a dimension only when its size is one and dimension 1 holds 28 rows.
• DON'T leave cmap="gray" off a grayscale image. viridis draws the black background purple and the boot yellow-green, and the picture looks finished.

Normalizing Pixels Like Any Feature

What. transforms.Normalize(mean, std) subtracts a mean from every pixel and divides by a standard deviation, one pair per channel, Lesson 3.2.1's standardization on pixels, and transforms.Compose (a list of transforms run in order, each handed the previous one's output) puts it after ToTensor. Unlike StandardScaler it learns nothing: the two numbers are computed beforehand and passed in. Why. ToTensor's 0 to 1 is centered nowhere near zero, and standardized pixels sit around 0 with a spread of 1, the scale Section 4.3 gave the listings' four numeric columns. Module 3's rule decides where the numbers come from: the 50,000 images the model is fitted on, never the validation or test images and never the image being transformed. When. When pixels feed a model fitted by gradient steps and are not already on a small shared scale. fashion.py's loaders use ToTensor alone, because 0 to 1 is already small and the same for every pixel, and Lesson 4.5.2 trains on them as they are, so Normalize is shown here as the standardization pixels can take.

import matplotlib.pyplot as plt
import torch
from torchvision import datasets, transforms
from fashion import train_full, train_set, val_set

fit = train_full.data[train_set.indices].double() / 255
mean, std = fit.mean().item(), fit.std().item()
print(fit.shape, fit.numel())
print(f"{mean:.4f} {std:.4f}")
everything = train_full.data.double() / 255
print(f"{everything.mean().item():.4f} {everything.std().item():.4f}")

torch.Size([50000, 28, 28]) 39200000
0.2857 0.3529
0.2860 0.3530

train_set.indices lists the positions, among the 60,000, of the 50,000 images fashion.py keeps for fitting, so train_full.data[train_set.indices] is their raw grids. .double() makes the sums float64, as Lesson 4.1.1 summed the areas, and dividing by 255 puts them on ToTensor's scale. Over all 60,000 training images the pair would be 0.2860 and 0.3530, and the difference is the 10,000 validation images, Lesson 3.1.2's third break however little it moves. As a transform:

img, label = train_full[0]
normalize = transforms.Normalize((0.2857,), (0.3529,))
z = normalize(img)
print(normalize)
print(z.shape, z.dtype)
print(round(z.min().item(), 4), round(z.max().item(), 4))
print(z[0, 10, 10:18])
print(torch.equal(z, (img - 0.2857) / 0.3529))

Normalize(mean=(0.2857,), std=(0.3529,))
torch.Size([1, 28, 28]) torch.float32
-0.8096 2.0241
tensor([-0.8096, -0.8096, -0.8096,  1.3351,  1.7241,  1.6129,  1.5574,  1.3907])
True

A black pixel lands at -0.8096 and a white one at 2.0241, (0 - 0.2857) / 0.3529 and (1 - 0.2857) / 0.3529, row 10's 0.7569 at 1.3351, and torch.equal says the transform is exactly that arithmetic. Inside a dataset it runs after ToTensor:

pipeline = transforms.Compose([transforms.ToTensor(), normalize])
print(pipeline)
normalized = datasets.FashionMNIST("data", train=True, transform=pipeline)
print(torch.equal(normalized[0][0], z))

Compose(
    ToTensor()
    Normalize(mean=(0.2857,), std=(0.3529,))
)
True

What the two numbers do to each set as a whole:

val = train_full.data[val_set.indices].double() / 255
for name, pixels in (("fit", fit), ("validation", val)):
    scaled = (pixels - 0.2857) / 0.3529
    print(name, f"{scaled.mean().item():.4f} {scaled.std().item():.4f}")

fit 0.0001 1.0000
validation 0.0051 1.0023

The fit pixels' mean lands a hair off 0 because the two numbers were rounded to four decimals, and the validation pixels near 0 and 1 and at neither, the mark Lesson 4.2.3's validation rows carried of a set transformed with numbers that know nothing about it. Numbers from one image do something else:

print(round(img.mean().item(), 4), round(img.std().item(), 4))

def own_numbers(x):
    return transforms.Normalize((x.mean().item(),), (x.std().item(),))(x)

dim = img * 0.5
print(torch.allclose(own_numbers(img), own_numbers(dim), atol=1e-5))
print(round(own_numbers(img)[0, 10, 13].item(), 4), round(own_numbers(dim)[0, 10, 13].item(), 4))
print(round(normalize(img)[0, 10, 13].item(), 4), round(normalize(dim)[0, 10, 13].item(), 4))

0.3814 0.3994
True
0.94 0.94
1.3351 0.2628

dim is the same boot at half the brightness, a darker photo of one garment. Standardized by its own numbers, each picture lands at mean 0 and spread 1 whatever its brightness, so the two come out identical, 0.94 at the same pixel, where the fit images' numbers keep them apart. Under its own numbers a pixel's value depends on the rest of its picture, and an image scored alone is standardized by nothing the model was fitted on, Lesson 3.2.1's scaler refitted on one listing. A picture of the normalized boot shows none of this:

fig, axes = plt.subplots(1, 2, figsize=(6, 3.4))
axes[0].imshow(img.squeeze(0), cmap="gray")
axes[1].imshow(z.squeeze(0), cmap="gray")
fig.savefig("normalized.png", dpi=150, bbox_inches="tight")
print(axes[0].images[0].get_clim())
print([round(v, 4) for v in axes[1].images[0].get_clim()])
plt.close(fig)

(0.0, 1.0)
[-0.8096, 2.0241]

The two halves of normalized.png match color for color, because imshow stretched the colormap over each array's own range, and get_clim() tells them apart. Three ways to get the transform wrong:

backwards = transforms.Compose([normalize, transforms.ToTensor()])
wrong_order = datasets.FashionMNIST("data", train=True, transform=backwards)
try:
    wrong_order[0]
except TypeError as err:
    print(err)
try:
    normalize(img.squeeze(0))
except ValueError as err:
    print(err)
try:
    normalize(train_full.data[0].unsqueeze(0))
except TypeError as err:
    print(err)

img should be Tensor Image. Got <class 'PIL.Image.Image'>
Expected tensor to be a tensor image of size (..., C, H, W). Got tensor.size() = torch.Size([28, 28])
Input tensor should be a float tensor. Got torch.uint8.

Normalize takes a float tensor with a channel dimension. Placed before ToTensor it is handed the pillow picture, and the dataset fails on its first read, not when it is built. Handed the squeezed grid it finds no channel, and handed the raw grid it refuses the uint8.

DON'T:

• DON'T take the two numbers from the image being transformed, or from the batch in hand. The boot and its half-bright copy came out identical under their own numbers, where the fit images' numbers gave 1.3351 and 0.2628.
• DON'T compute them over all 60,000 training images, or over any image you score. 0.2860 and 0.3530 hold the validation images, whether the fourth decimal moves or not.
• DON'T put Normalize before ToTensor in a Compose. The dataset builds without a word and fails on its first read.
• DON'T normalize the image you squeezed for drawing. The (28, 28) grid has no channel, and Normalize refuses it.

Batches of Images, and the Validation Split

What. random_split(dataset, [50_000, 10_000], generator=g) (PyTorch's split of one dataset into random parts of the sizes given, sharing no item) returns two Subsets (a dataset that keeps another dataset and a list of positions in it, and reads only those). fashion.py's loaders are Lesson 4.3.3's DataLoader over the 50,000 fit images, shuffled by a generator of their own, and over the 10,000 validation and 10,000 test images in a fixed order. Why. FashionMNIST ships two sets, and Module 3's rule needs three: images to fit on, images to choose settings on, and images scored once after every choice. The validation images come out of the training images, never the test images, and a seeded generator makes the carve the same in every script that imports fashion.py. When. Once, before anything is fitted, in the one module that later scripts import.

import math
import torch
from torch.utils.data import DataLoader
from fashion import train_full, train_set, val_set, train_loader, val_loader, test_loader

print(type(train_set).__name__, len(train_set), len(val_set))
print(train_set.indices[:5])
print(val_set.indices[:5])
print(train_set.dataset is train_full, val_set.dataset is train_full)
image, label = train_set[0]
print(torch.equal(image, train_full[37542][0]), label)
fit_ids, val_ids = set(train_set.indices), set(val_set.indices)
print(len(fit_ids & val_ids), len(fit_ids | val_ids))
print(torch.bincount(train_full.targets[val_set.indices]))

Subset 50000 10000
[37542, 43087, 2992, 25810, 39634]
[59753, 9414, 10988, 15501, 49861]
True True
True 9
0 60000
tensor([1074, 1027,  985,  945,  974,  999, 1033,  952,  979, 1032])

A Subset holds no images of its own: train_set.dataset is train_full itself, and train_set[0] is train_full[37542], another ankle boot. The two lists share no position and together cover all 60,000. The generator is Lesson 4.1.1's random stream of its own, so every script gets the same carve, and the draw ignores the labels, unlike Lesson 3.1.2's stratify=, so the validation images hold from 945 to 1,074 of each class. The loaders:

print(len(train_loader), len(val_loader), len(test_loader))
print(math.ceil(50_000 / 64), 50_000 - 781 * 64)
images, labels = next(iter(train_loader))
print(images.shape, images.dtype)
print(labels.shape, labels.dtype)
print(labels[:10])

782 10 10
782 16
torch.Size([64, 1, 28, 28]) torch.float32
torch.Size([64]) torch.int64
tensor([3, 2, 0, 4, 4, 2, 8, 7, 8, 9])

len of a loader counts batches: 781 of 64 and a last of 16 for the fit images, 10 of 1,000 for each of the others. A batch is (64, 1, 28, 28), 64 images stacked on a new first dimension, and its labels a (64,) int64 tensor, the target of class indices Lesson 4.3.2's CrossEntropyLoss takes, although the dataset handed out each label as an int. The loader's collation, the stacking Lesson 4.3.3's Assignment named, made both, as an unshuffled loader shows against Lesson 4.1.2's torch.stack:

in_order = DataLoader(train_set, batch_size=64)
first_images, first_labels = next(iter(in_order))
print(torch.equal(first_images, torch.stack([train_set[i][0] for i in range(64)])))
print(torch.equal(first_labels, torch.tensor([train_set[i][1] for i in range(64)])))

True
True

The loader asked for its 64 items one at a time, as Lesson 4.3.3's Assignment counted, and this dataset builds each when asked: a pillow picture from the stored grid, then ToTensor. Removing that conversion measures its cost:

import time
import torch
from torch.utils.data import DataLoader, TensorDataset
from fashion import train_full, train_set, train_loader

start = time.perf_counter()
for images, labels in train_loader:
    pass
print(f"{time.perf_counter() - start:.1f} seconds")

held = (train_full.data[train_set.indices].float() / 255).unsqueeze(1)
tensors = TensorDataset(held, train_full.targets[train_set.indices])
loader = DataLoader(tensors, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
start = time.perf_counter()
for images, labels in loader:
    pass
print(f"{time.perf_counter() - start:.1f} seconds")

4.3 seconds
0.4 seconds

The seconds are this machine's and only their order is read: the per-image path took many times as long as slicing one tensor of the same images, the cost Lesson 4.3.3's Assignment timed on a Dataset that built each row when asked and said Section 4.5 would pay.

DON'T:

• DON'T carve the validation images out of the test images, or choose anything by a score on test_loader. The test images then stop standing in for new ones, Lesson 3.1.2's second break.
• DON'T split without the generator. Every run of an unseeded random_split opens its fit images on another position and puts over 8,000 of fashion.py's validation images among them, eight runs here between 8,295 and 8,361.
• DON'T read len(train_loader) as a count of images. 782 is batches, and the last holds 16.

Lab Exercises

Lab Exercise 1: The Test Images, Read as Tensors

In test_tensors.py, import test_set from fashion.py. Print torchvision's version beside the number of test images, then the number of test images in each class with torch.bincount, and the first eight test labels as a tensor and as a list of class names. Read the first test image and print what the read returns and its length, the image's shape, dtype, smallest and largest value, and the label's type, value and class name. Take the raw grid of the same image and print its shape, dtype and largest value, then row 14 from column 10 to column 15 of the raw grid and of the image.

Then prove ToTensor's arithmetic for the whole test set at once: stack all 10,000 images into one tensor with torch.stack, print its shape and dtype, and print whether it equals the raw grids divided by 255 with a channel dimension added. Finish with the size in bytes of the four test files in data/FashionMNIST/raw, and the bytes the test set's raw grids take in memory.

Lab Exercise 2: The Validation Images, Drawn, Normalized and Counted

In val_check.py, take the first batch of the validation loader from fashion.py, and print the shapes and dtypes of its images and labels and its first ten labels. Draw those ten images on a 2 by 5 grid of Axes, each titled with its class name and with its frame hidden, save the figure as val_first10.png, and print how many images the ten Axes hold and the first five titles.

Compute the mean and standard deviation of the fit images' pixels from their raw grids in float64, and print both to four decimals. Normalize the validation batch with them, rounded to four decimals, through transforms.Normalize, and print the batch's mean and standard deviation before and after, to four decimals, then the normalized batch's smallest and largest value. Finish by walking the validation loader once, counting the batches and the images and adding up each batch's class counts, and print the number of batches, the size of the last one, the images seen, the ten counts, and whether they equal the validation labels counted straight from train_full.targets at val_set.indices.

import os
import torch
import torchvision
from fashion import test_set

print(torchvision.__version__, len(test_set))
print(torch.bincount(test_set.targets))
first8 = test_set.targets[:8]
print(first8)
print([test_set.classes[i] for i in first8.tolist()])

item = test_set[0]
image, label = item
print(type(item).__name__, len(item))
print(image.shape, image.dtype, image.min().item(), image.max().item())
print(type(label).__name__, label, test_set.classes[label])
raw = test_set.data[0]
print(raw.shape, raw.dtype, raw.max().item())
print(raw[14, 10:16])
print(image[0, 14, 10:16])

0.29.1+cpu 10000
tensor([1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000, 1000])
tensor([9, 2, 1, 1, 6, 1, 4, 6])
['Ankle boot', 'Pullover', 'Trouser', 'Trouser', 'Shirt', 'Trouser', 'Coat', 'Shirt']
tuple 2
torch.Size([1, 28, 28]) torch.float32 0.0 1.0
int 9 Ankle boot
torch.Size([28, 28]) torch.uint8 255
tensor([  0,   0,  98, 136, 110, 109], dtype=torch.uint8)
tensor([0.0000, 0.0000, 0.3843, 0.5333, 0.4314, 0.4275])

everything = torch.stack([test_set[i][0] for i in range(len(test_set))])
print(everything.shape, everything.dtype)
print(torch.equal(everything, test_set.data.unsqueeze(1) / 255))

folder = "data/FashionMNIST/raw"
for name in sorted(os.listdir(folder)):
    if name.startswith("t10k"):
        print(name, os.path.getsize(f"{folder}/{name}"))
print(test_set.data.numel() * test_set.data.element_size())

torch.Size([10000, 1, 28, 28]) torch.float32
True
t10k-images-idx3-ubyte 7840016
t10k-images-idx3-ubyte.gz 4422102
t10k-labels-idx1-ubyte 10008
t10k-labels-idx1-ubyte.gz 5148
7840000

import matplotlib.pyplot as plt
import torch
from torchvision import transforms
from fashion import train_full, train_set, val_set, val_loader

images, labels = next(iter(val_loader))
print(images.shape, images.dtype, labels.shape, labels.dtype)
print(labels[:10])

fig, axes = plt.subplots(2, 5, figsize=(10, 4.5))
for ax, image, label in zip(axes.ravel(), images[:10], labels[:10]):
    ax.imshow(image.squeeze(0), cmap="gray")
    ax.set_title(train_full.classes[label])
    ax.axis("off")
fig.savefig("val_first10.png", dpi=150, bbox_inches="tight")
print(sum(len(ax.images) for ax in axes.ravel()))
print([ax.get_title() for ax in axes.ravel()][:5])
plt.close(fig)

torch.Size([1000, 1, 28, 28]) torch.float32 torch.Size([1000]) torch.int64
tensor([6, 1, 8, 7, 5, 1, 7, 1, 1, 0])
10
['Shirt', 'Trouser', 'Bag', 'Sneaker', 'Sandal']

fit = train_full.data[train_set.indices].double() / 255
mean, std = fit.mean().item(), fit.std().item()
print(f"{mean:.4f} {std:.4f}")
normalize = transforms.Normalize((round(mean, 4),), (round(std, 4),))
batch = normalize(images)
print(round(images.mean().item(), 4), round(images.std().item(), 4))
print(round(batch.mean().item(), 4), round(batch.std().item(), 4))
print(round(batch.min().item(), 4), round(batch.max().item(), 4))

counts = torch.zeros(10, dtype=torch.int64)
batches = seen = 0
for xb, yb in val_loader:
    counts += torch.bincount(yb, minlength=10)
    batches += 1
    seen += len(yb)
print(batches, len(yb), seen)
print(counts)
print(torch.equal(counts, torch.bincount(train_full.targets[val_set.indices])))

0.2857 0.3529
0.2817 0.3515
-0.0113 0.9961
-0.8096 2.0241
10 1000 10000
tensor([1074, 1027,  985,  945,  974,  999, 1033,  952,  979, 1032])
True

Theory Assessment

Q1: Thirteen Resolved, One Added, Four Fetched

A teammate installs torchvision, reads Resolved 13 packages above a single + line, and concludes that uv skipped twelve packages it should have installed. Later the same teammate runs python3 ../fashion.py from ~/dev/bootcamp/experiments, a folder inside the project, and watches four progress lines appear although ~/dev/bootcamp/data holds every file. Say what the thirteen packages are and why one was added, where that run put its files and why, why it showed four progress lines and not two, and what download=True costs a run that finds the files.

Hint: Count what torchvision's package names as its own needs and what torch brought in Lesson 4.1.1, and ask which a Module 4 (bootcamp) already holds. For the folder, ask what a bare name like data is measured from, and how many files one datasets.FashionMNIST line fetches.

Q2: Three Views of One Boot

train_full[0][0], train_full.data[0] and plain[0][0], read from a FashionMNIST dataset built with no transform, are three views of the first training image. Say what each is, with its shape or size and its dtype or mode, which one a model reads, what ToTensor did to get from the third to the first, why the first has three dimensions where the second has two, and what train_full[0][1] is and how it becomes the int64 tensor a loss takes.

Hint: One is the store the dataset keeps, one is what the dataset hands its transform, and one is what the transform hands back. Then ask what a loader's collation does with 64 Python integers.

Q3: Whose Numbers Normalize Uses

The fit images' pixels average 0.2857 with a standard deviation of 0.3529, all 60,000 training images' pixels 0.2860 and 0.3530, and the first image's own 0.3814 and 0.3994. Say which pair a Normalize in front of a model takes and why, which of Lesson 3.1.2's breaks the second pair commits, what each image's own pair, like the third, does to an image and its half-bright copy, what the validation images' 0.0051 and 1.0023 after normalization say, and why none of this holds unless random_split draws from a seeded generator.

Hint: Ask which images may shape a number the model is fitted through, and what a validation set is for. Then ask what a second script that carves again without a seed does to the positions in train_set.indices.

Q4: Exactly Zero, Exactly One

Read this script without running it, then say what the teammate who wrote it got right and wrong about its two printed lines.

import torch
from fashion import val_loader, test_loader

for name, loader in (("validation", val_loader), ("test", test_loader)):
    images, labels = next(iter(loader))
    batch = (images - images.mean()) / images.std()
    print(name, round(abs(batch.mean().item()), 4), round(batch.std().item(), 4))

validation 0.0 1.0
test 0.0 1.0

The teammate reports that standardizing each batch by its own mean and standard deviation beats Normalize with the fit images' numbers, which left the validation images at 0.0051 and 1.0023, because both sets now come out at exactly 0 and 1, and that the test batch confirms it. Say what the two printed lines measure, where each batch's two numbers came from and what that does to one pixel's value, what the test line cost, and what this transform does to an image the model is asked about on its own.

Hint: Standardize any tensor by its own mean and spread, and ask what its mean and spread must then be. For the pixel, ask whether its new value depends on itself and on numbers fixed before the run, or on the images it happened to be batched with.

images, labels = next(iter(val_loader))
alone = (images[:1] - images[:1].mean()) / images[:1].std()
in_batch = (images - images.mean()) / images.std()
print(round(images[0, 0, 10, 13].item(), 4))
print(round(in_batch[0, 0, 10, 13].item(), 4), round(alone[0, 0, 10, 13].item(), 4))
print(round(((images[0, 0, 10, 13] - 0.2857) / 0.3529).item(), 4))

0.6745
1.1175 0.6731
1.1018

Assignment

Assignment: One Carve, Two Transforms

Augmentation (a random change made to a training image every time it is read, so a model meets a slightly different picture in every epoch) is Lesson 5.1.4's subject, and its simplest form is a mirror image: a shirt flipped left to right is still a shirt. The validation images must never be augmented, because a score that changes with every read cannot compare two settings. Three things this lesson did not use. transforms.RandomHorizontalFlip(p) mirrors the picture it is handed with probability p, drawing from PyTorch's global stream, the one torch.manual_seed resets, every time an image is read. torch.flip(x, dims=[2]) reverses a tensor along dimension 2, the width of a (1, 28, 28) image, which is what a mirror does to it. Subset(dataset, indices), from torch.utils.data, builds the same kind of object random_split returned, over any dataset and any list of positions.

Write augment.py so that the fit images are flipped half the time and the validation images never. Start with the line that looks like the fix and is not: read val_set[0] and print its position and class name, set train_set.dataset.transform to a Compose of RandomHorizontalFlip(p=1.0) and ToTensor, read val_set[0] again, and print whether it now equals the first read mirrored, whether it equals the first read as it was, and whether val_set, train_set and train_full share one dataset. Put transforms.ToTensor() back and print whether val_set[0] is restored.

Then build the fix: a second FashionMNIST dataset over data with a Compose of RandomHorizontalFlip(p=0.5) and ToTensor, and a Subset of it at train_set.indices. Print the Subset's type, its length and whether its positions are train_set's, then whether val_set[0] is still the first read. Take the first fit image as train_set hands it out and its mirror, and print whether the two are equal. Seed with 0, read the first augmented fit image 1,000 times, and print how many reads are the mirror and how many the image as stored. Seed with 0 again, read it ten times, and print whether those ten match the first ten of the thousand, and whether the label survived. Build a loader over the augmented fit images and one over train_set, both with batches of 64 and generators seeded 42, seed the flips with 0, take the first batch of each, and print whether the labels agree, how many of the 64 images differ and how many of those are mirrors. End with the stored image and its mirror side by side in flip.png, titled as stored and flipped, and print how many images the two Axes hold, their titles, and whether the right one drew the mirror.

Hints:

• Compare val_set.dataset and train_set.dataset with is before you set anything through either, and ask which object a transform set through .dataset belongs to.
• A read is the mirror when it equals torch.flip(stored, dims=[2]), which counts honestly only if the stored image is not its own mirror, and the flips draw from the global stream torch.manual_seed resets, while each loader's order comes from its own generator.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch.utils.data import DataLoader, Subset
from torchvision import datasets, transforms
from fashion import train_full, train_set, val_set

stored, label = val_set[0]
print(val_set.indices[0], train_full.classes[label])
train_set.dataset.transform = transforms.Compose([transforms.RandomHorizontalFlip(p=1.0), transforms.ToTensor()])
after, _ = val_set[0]
print(torch.equal(after, torch.flip(stored, dims=[2])), torch.equal(after, stored))
print(val_set.dataset is train_set.dataset is train_full)
train_full.transform = transforms.ToTensor()
print(torch.equal(val_set[0][0], stored))

59753 Shirt
True False
True
True

augment = transforms.Compose([transforms.RandomHorizontalFlip(p=0.5), transforms.ToTensor()])
fit_images = Subset(datasets.FashionMNIST("data", train=True, transform=augment), train_set.indices)
print(type(fit_images).__name__, len(fit_images), fit_images.indices == train_set.indices)
print(torch.equal(val_set[0][0], stored))

plain, plain_label = train_set[0]
mirror = torch.flip(plain, dims=[2])
print(torch.equal(plain, mirror))
torch.manual_seed(0)
reads = [fit_images[0][0] for _ in range(1000)]
print(sum(torch.equal(r, mirror) for r in reads), sum(torch.equal(r, plain) for r in reads))
torch.manual_seed(0)
again = [torch.equal(fit_images[0][0], mirror) for _ in range(10)]
print(again == [torch.equal(r, mirror) for r in reads[:10]], fit_images[0][1] == plain_label)

Subset 50000 True
True
False
492 508
True True

flipped = DataLoader(fit_images, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
as_stored = DataLoader(train_set, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
torch.manual_seed(0)
(xa, ya), (xb, yb) = next(iter(flipped)), next(iter(as_stored))
differ = sum(not torch.equal(a, b) for a, b in zip(xa, xb))
mirrored = sum(torch.equal(a, torch.flip(b, dims=[2])) and not torch.equal(a, b) for a, b in zip(xa, xb))
print(torch.equal(ya, yb), differ, mirrored)

fig, axes = plt.subplots(1, 2, figsize=(6, 3.4))
for ax, image, title in zip(axes, (plain, mirror), ("as stored", "flipped")):
    ax.imshow(image.squeeze(0), cmap="gray")
    ax.set_title(title)
    ax.axis("off")
fig.savefig("flip.png", dpi=150, bbox_inches="tight")
print(sum(len(ax.images) for ax in axes), [ax.get_title() for ax in axes])
print(np.array_equal(axes[1].images[0].get_array(), mirror.squeeze(0).numpy()))
plt.close(fig)

True 35 35
2 ['as stored', 'flipped']
True

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want random transforms in more depth, or the reasons torchvision's documentation gives for its second set of transforms.

• YouTube - "Pytorch Data Augmentation using Torchvision" by Aladdin Persson. Under ten minutes on an example of transformations performed on images in PyTorch, and the idea its description names data augmentation, expanding a dataset with transformed images.
• Official docs - Torchvision 0.29, "Transforming images, videos, boxes and more", the section "V1 or V2? Which one should I use?". Why the documentation recommends the torchvision.transforms.v2 set over the torchvision.transforms one this lesson uses: it transforms bounding boxes, masks, videos and keypoints beside images, adds transforms such as CutMix and MixUp, runs faster, and is backward compatible, so moving to it is a change of import.

Summary

A picture turned out to be the least mysterious input of the module. Once torchvision had read it, the boot was 784 numbers with a channel in front, a tensor like every tensor since Lesson 4.1.1, and every question asked of it had been asked of a table: what dtype, what shape, what scale, which rows. The new parts were all on the way in, and so were the failures: a transform handed the wrong kind of object, a store read as the image, a picture that looked right in colors the data never held.

Normalization was an old rule in new clothes. Normalize computes Lesson 3.2.1's standardization and learns nothing, so the only decision left was where its two numbers come from, and Module 3 had already made it. The numbers that obeyed the rule and the numbers that broke it differed in the fourth decimal, while the image's own numbers made a dark boot and a brighter one the same, which is why the rule is about what a number has seen and not how far it moves.

What the lesson leaves behind is plumbing: one file that downloads once, carves once and builds three loaders, and the habit of printing what a chart, a batch or a split holds before believing it. The images are now batches of (64, 1, 28, 28) with labels a loss can read, and nothing has been fitted to them yet.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

from torchvision import datasets, transforms

to_tensor = transforms.ToTensor()
train_full = datasets.FashionMNIST("data", train=True, download=True, transform=to_tensor)
img, label = train_full[0]               # (1, 28, 28) float32, int 9
raw = train_full.data[0]                 # (28, 28) uint8, 0 to 255
ax.imshow(img.squeeze(0), cmap="gray")   # drop the channel to draw
len(ax.images), ax.images[0].get_clim()  # 1, (0.0, 1.0)

fit = train_full.data[train_set.indices].double() / 255
# 0.2857 and 0.3529, the fit images only
mean, std = fit.mean().item(), fit.std().item()
normalize = transforms.Normalize((0.2857,), (0.3529,))
# ToTensor first, then Normalize
pipeline = transforms.Compose([to_tensor, normalize])
images, labels = next(iter(train_loader))  # (64, 1, 28, 28), (64,)
len(train_loader), len(val_loader)         # 782, 10