Dataset and DataLoader - Mini-Batches

Context

A mini-batch is a slice of the fit rows, 64 here, that one update of the parameters is computed from, so a pass over 1,725 rows becomes 27 updates instead of one. TensorDataset holds the table as rows a loader can index, DataLoader cuts those rows into the slices of an epoch, in a fresh order every epoch when it shuffles, and the five steps of Lesson 4.2.2 move inside a second loop unchanged.

The image network of Section 4.5 trains this way, its 50,000 training pictures reaching the model a slice at a time, as do most PyTorch training scripts you will open in someone else's repository, and every number such a loop prints is read by the counts this lesson draws apart, epochs, updates and seconds.

Without the discipline the loop still runs and prints a falling number. A loop that reports its last batch's loss as the epoch's reports 0.27 for an epoch at 0.73, rows sorted by price and never shuffled send the network to nan by the second epoch, and a loader rebuilt every epoch with one seed replays one order thirty times under a shuffle=True that reshuffles nothing. Lesson 4.3.1 fitted the network on the full batch, 500 epochs to a validation MAE of 45,585, and this lesson gets within 500 of that in 30 epochs, counts their cost in updates and seconds, and says what a mini-batch buys and what it does not.

Goal

• Count the updates an epoch makes at a given batch size, by arithmetic and by a counter in the loop, and write one mini-batch epoch by hand with torch.randperm and slices.
• Build a TensorDataset and a seeded DataLoader, read len, the batch shapes and the last short batch, and prove from a run that every batch is a slice of the table and every row reaches exactly one batch per epoch.
• Restructure the loop around for xb, yb in loader:, with the validation pass after the inner loop and the epoch's training loss computed over all fit rows in evaluation mode.
• Distinguish a batch's loss from an epoch's loss on a chart of 810 batch losses, and show what a reshuffled order, a fixed order and a sorted order each do to the fit.
• Measure the full batch against batches of 16, 64 and 256 in updates, validation MAE and seconds, and state what a mini-batch buys on 1,725 rows and what it does not.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want PyTorch's own picture of a dataset and a loader before you write the loop around one.

• YouTube - "Introduction to PyTorch" by PyTorch. The first of the PyTorch channel's beginner series, twenty-three minutes surveying tensors, autograd, a model, datasets and a training loop, which the link opens at the three minutes on datasets: a Dataset that answers len() and [i], a DataLoader built over it with a batch size and shuffle=True, and one batch pulled from the loader and printed.
• Official docs - PyTorch Tutorials, "Datasets & DataLoaders". The two classes on FashionMNIST, Section 4.5's images: a dataset loaded through torchvision, a custom Dataset with __init__, __len__ and __getitem__ reading images from a folder, a DataLoader(training_data, batch_size=64, shuffle=True) over the FashionMNIST set, and one batch of shape (64, 1, 28, 28) pulled from the loader.

Table of Contents

• Epochs Against Updates
• TensorDataset and DataLoader
• The Loop, Restructured
• The Noise in a Batch's Loss
• What Mini-Batches Buy, Measured

Theory

Every block in this lesson runs in ~/dev/bootcamp on the 2,875 listings with an area, split as Lesson 4.2.3 split them into 1,725 fit, 575 validation and 575 test rows, through Lesson 4.3.1's table and network: eighteen columns, the price in hundreds of thousands, nn.Sequential(nn.Linear(18, 32), nn.ReLU(), nn.Linear(32, 1)) created after torch.manual_seed(42), nn.MSELoss() and optim.SGD at 0.1, the section's rate. Every script imports what it needs from prep_net.py and train_loop.py, the two files Lesson 4.3.1 saved there, as Lesson 4.2.3's scripts imported prep.py. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Epochs Against Updates

What. An update (one call of optimizer.step(), the parameters moved once along their gradients) and an epoch, Lesson 4.2.1's word, have been one count since that lesson, because every loop so far computed each gradient on all the rows it fits at once, the full batch, 1,725 fit rows here. A mini-batch (a slice of the fit rows, 64 here, that one update is computed from) separates the two: one epoch becomes 27 updates, math.ceil(1725 / 64), 26 slices of 64 and a last of 61. Why. One pass over the rows costs about the same arithmetic either way, every row goes through the network once, and the mini-batch loop spends it on 27 moves of the parameters where the full batch spends it on one. The memory a step holds is bounded by the slice and not by the table. What a mini-batch does not buy is seconds on a table this size, and the last subtitle measures that rather than asserting it. When. Whenever a loop takes its steps on slices of the rows, as Section 4.5's will on its images, and every count such a loop prints says whether it is epochs or updates.

prep_net.py holds the split, prep and the six tensors, and train_loop.py holds loss_fn, network(), validate, run, the full-batch loop with its optimizer handed in, and score, which returns a model's MAE and R² on a frame's own prices as two floats for the caller to format. What this lesson leans on, read back through an import:

from prep_net import X_fit, y_fit, X_val, y_val, val_rows
from train_loop import network, validate, score

print(X_fit.shape, y_fit.shape, X_val.shape)
print(y_fit[:3].squeeze(1), round(y_fit.mean().item(), 3))
model = network()
print(sum(p.numel() for p in model.parameters()))
mae, r2 = score(model, X_val, val_rows)
print(f"{validate(model, X_val, y_val):.4f}", f"{mae:,.0f}")

torch.Size([1725, 18]) torch.Size([1725, 1]) torch.Size([575, 18])
tensor([2.7200, 2.2600, 3.1500]) 3.848
641
16.9906 373,276

The fit tensors' shapes, the first three targets and their mean are the numbers Lesson 4.3.1 printed, and the seeded network's 641 parameters price the validation listings 373,276 off before any step, score's first float in whole units. The loop inside run, written out with one line counting what it does:

import math
import torch
from torch import optim
from prep_net import X_fit, y_fit, X_val, y_val, val_rows
from train_loop import loss_fn, network, validate, score

model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
updates = 0
for epoch in range(1, 11):
    loss = loss_fn(model(X_fit), y_fit)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    updates += 1
    if epoch in (1, 10):
        print(epoch, updates, f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}")

1 1 7.9833 228,189
10 10 0.9882 74,174

Ten epochs, ten updates, each from the gradient over all 1,725 rows: one update brings the validation loss from 16.9906 to 7.9833, and ten bring it to 0.9882, an MAE of 74,174. The arithmetic of a mini-batch epoch:

print(len(X_fit), math.ceil(len(X_fit) / 64))
print(len(X_fit) // 64, len(X_fit) % 64)

1725 27
26 61

One such epoch by hand, before any class does it: a permutation of the row numbers from a seeded generator, Lesson 4.1.1's torch.randperm, cut into slices of 64, each slice indexing the two tensors as Lesson 4.1.1 indexed with a list.

model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
g = torch.Generator().manual_seed(42)
perm = torch.randperm(len(X_fit), generator=g)
print(perm[:5], perm.shape)
updates = 0
for start in range(0, len(X_fit), 64):
    idx = perm[start:start + 64]
    xb, yb = X_fit[idx], y_fit[idx]
    loss = loss_fn(model(xb), yb)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    updates += 1
print(updates, xb.shape, yb.shape)
print(torch.equal(perm.sort().values, torch.arange(len(X_fit))))
print(f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}")

tensor([ 867, 1504,  868,  421, 1650]) torch.Size([1725])
27 torch.Size([61, 18]) torch.Size([61, 1])
True
1.6171 88,999

perm is every row number from 0 to 1,724 once, in the generator's order, which its sorted values prove, so row 867 leads the first slice, as it does on every run of this release on a processor, and no listing is seen twice or skipped. The five steps inside the loop are Lesson 4.2.2's on xb and yb. After one pass over the rows, 27 updates, the validation loss is 1.6171 where one full-batch update left it at 7.9833: the same rows went through the network once in both loops, and the mini-batch loop moved the parameters 27 times on the way. Per update the full batch did better, ten of them reached 0.9882, because each of its gradients is every row's opinion where a slice's is 64 rows' opinion, the noise the fourth subtitle measures. The fair unit between the two loops is the pass over the rows.

DON'T:

• DON'T count epochs as updates once the batch is a slice. Thirty epochs at a batch size of 64 are 810 updates, 500 epochs of the full batch are 500, and a sentence that compares the two by epochs compares nothing.
• DON'T compare the two loops by updates either. Ten full-batch updates reached 0.9882 and 27 slice updates 1.6171, and the unit on which the mini-batch loop wins is the pass over the rows, measured in the last subtitle.

TensorDataset and DataLoader

What. A Dataset (PyTorch's base class for anything a loader reads: an object that answers len() with its row count and [i] with one row and its target) is the table seen one row at a time, and TensorDataset(X, y) is the Dataset that pairs row i of tensors that already hold the table. A DataLoader (the object that hands out the batches of an epoch, built over a dataset with a batch size, a shuffle flag and a generator) is the hand loop above as an iterable: for xb, yb in loader: yields 27 pairs of tensors, and with shuffle=True each epoch draws a fresh random order of the rows when it starts. Why. The scripts you open from here on hide the hand loop's permutation and slices behind these two names, so you read them as the slicing you wrote. The loader also handles the last short batch, keeps its generator from one epoch to the next, and takes a dataset that builds each item when asked, Section 4.5's images, with no change to the loop around it. When. Every loop that slices its rows. The table itself stays in memory as one tensor, Lesson 2.5.2's sentence that a table has to fit in memory before anything trains on it, and the loader slices it.

import torch
from torch.utils.data import TensorDataset, DataLoader
from prep_net import X_fit, y_fit

dataset = TensorDataset(X_fit, y_fit)
print(len(dataset))
x0, y0 = dataset[0]
print(x0.shape, y0)
print(torch.equal(x0, X_fit[0]), torch.equal(y0, y_fit[0]))
print(dataset[0:2][1])
try:
    TensorDataset(X_fit, y_fit[:100])
except AssertionError as err:
    print(err)

1725
torch.Size([18]) tensor([2.7200])
True True
tensor([[2.7200],
        [2.2600]])
Size mismatch between tensors

len(dataset) is the row count, dataset[0] is a tuple of the first row and its target, the same tensors X_fit[0] and y_fit[0] are, and a slice indexes both tensors at once. The two tensors must agree on their first dimension: a target cut to 100 rows is refused at construction with Size mismatch between tensors, an AssertionError. The loader over it:

g = torch.Generator().manual_seed(42)
loader = DataLoader(dataset, batch_size=64, shuffle=True, generator=g)
print(len(loader))
sizes = [len(xb) for xb, yb in loader]
print(sizes[:3], sizes[-1], sum(sizes))
xb, yb = next(iter(loader))
print(xb.shape, yb.shape)

27
[64, 64, 64] 61 1725
torch.Size([64, 18]) torch.Size([64, 1])

len(loader) is the batches per epoch, 27. Iterating the loader once is one epoch, and next(iter(loader)) pulls the first batch of the next epoch, a (64, 18) input and a (64, 1) target, the shapes nn.Linear and nn.MSELoss want. The generator is Lesson 4.1.1's torch.Generator().manual_seed(42) passed as generator=, so the loader's orders come from a stream of their own and the network's initialization keeps the global one. What the loader does is slicing, checked rather than believed. With no shuffle it hands out the table's own slices:

plain = DataLoader(dataset, batch_size=64)
starts = range(0, len(dataset), 64)
same = [torch.equal(xb, X_fit[s:s + 64]) for (xb, yb), s in zip(plain, starts)]
print(sum(same), len(same))

27 27

All 27 batches are X_fit[0:64], X_fit[64:128] and so on to X_fit[1664:1725]. With a shuffle the slices are taken through a permutation, as the hand loop took them, and a third tensor of row numbers in the dataset lets the loader say which rows each batch holds:

indexed = TensorDataset(X_fit, y_fit, torch.arange(len(X_fit)))
print(indexed[0][2], len(indexed[0]))
g = torch.Generator().manual_seed(42)
loader = DataLoader(indexed, batch_size=64, shuffle=True, generator=g)
seen, same = [], []
for xb, yb, idx in loader:
    same.append(torch.equal(xb, X_fit[idx]) and torch.equal(yb, y_fit[idx]))
    seen.append(idx)
seen = torch.cat(seen)
print(sum(same), seen.shape)
print(seen[:5])
print(torch.equal(seen.sort().values, torch.arange(len(X_fit))))

tensor(0) 3
27 torch.Size([1725])
tensor([1426,  827,  848, 1490, 1599])
True

A TensorDataset takes any number of tensors of one length and each item is a tuple of that many, so this loader yields three per batch, the third being the row numbers it indexed. Every batch equals X_fit[idx] and y_fit[idx], the hand loop's X_fit[perm[start:start + 64]] with the loader's own permutation, and the 1,725 row numbers of the epoch sorted are arange(1725): every listing reached exactly one batch. The loader is randperm and slicing, the code it replaced, read back. Row 1426 leads this epoch where 867 led the hand loop's, because the loader makes a draw of its own from the generator before the permutation, so the two do not share an order. What they share is the rule, that the order repeats when the seed does and changes when the epoch does:

a = DataLoader(indexed, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
b = DataLoader(indexed, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
print(next(iter(a))[2][:3], next(iter(b))[2][:3])
print(next(iter(a))[2][:3])
print(next(iter(DataLoader(indexed, batch_size=64)))[2][:3])
dropped = DataLoader(dataset, batch_size=64, shuffle=True, drop_last=True, generator=torch.Generator().manual_seed(42))
print(len(dropped), sum(len(xb) for xb, yb in dropped))

tensor([1426,  827,  848]) tensor([1426,  827,  848])
tensor([591, 174, 213])
tensor([0, 1, 2])
26 1664

Two loaders built from generators seeded 42 open on the same three rows, and the first loader's second epoch opens on three others: the generator moves on with every epoch the loader runs, which is what makes 30 epochs 30 different orders and two runs with one seed the same 30. A next(iter(loader)) is an epoch started, so a loader iterated for a check and then handed to a training run begins that run on its second order, which is why every run in this lesson builds its loader fresh. With no shuffle the first batch opens on rows 0, 1 and 2, the table's order. drop_last=True discards the short batch: 26 batches of 64, 1,664 rows, and 61 listings sit out each epoch, different ones each time under a shuffle.

DON'T:

• DON'T hand TensorDataset tensors of different lengths. TensorDataset(X_fit, y_fit[:100]) is refused with Size mismatch between tensors, and a target that was sliced or filtered on its own is the usual way to get there.
• DON'T set drop_last=True where every row matters. 26 batches cover 1,664 of 1,725 rows, and the 61 that sit out an epoch are 3.5 percent of the fit rows taking no part in that epoch's updates.
• DON'T leave the generator out. shuffle=True alone draws its orders from the global stream, the one torch.manual_seed seeds and the network's initialization draws from, and a generator of its own gives the order a seed that nothing else in the script moves.

The Loop, Restructured

What. The five steps move inside for xb, yb in loader:, one update per batch, 27 per epoch. The validation pass, Lesson 4.2.3's validate, runs once per epoch after the inner loop under both switches, and the epoch's training loss is validate pointed at the fit rows: the loss over all 1,725 rows in evaluation mode after the epoch's 27 updates, never the last batch's loss and never the mean of the 27. Why. A batch's loss is one slice's number, taken before that slice's update, and the next subtitle shows how far the 27 scatter. An epoch's training loss has to be one number about one model on all its rows, as it was for the full batch, or the two curves of Lesson 4.2.3 stop meaning the same thing. A validation pass inside the inner loop would score 27 models per epoch, 26 of them half-finished, at 27 times the cost. When. Every mini-batch loop, and an epoch count quoted from such a loop is an epoch of this loop.

import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
from prep_net import fit_rows, prep, X_fit, y_fit, X_val, y_val, val_rows
from train_loop import loss_fn, network, validate, score

dataset = TensorDataset(X_fit, y_fit)
loader = DataLoader(dataset, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses, maes = [], [], []
for epoch in range(1, 31):
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    train_losses.append(validate(model, X_fit, y_fit))
    val_losses.append(validate(model, X_val, y_val))
    maes.append(score(model, X_val, val_rows)[0])
    if epoch in (1, 10, 22, 30):
        print(epoch, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{maes[-1]:,.0f}")
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4f}")
print(f"{loss.item():.4f}")

1 1.0316 0.7464 62,236
10 0.7662 0.4721 48,617
22 0.7394 0.4459 46,584
30 0.7260 0.4492 46,020
22 0.4459
0.2665

Epoch 1, 27 updates, ends at a validation loss of 0.7464 and an MAE of 62,236, where ten full-batch epochs reached 0.9882. Epoch 10, 270 updates, is at 0.4721 and 48,617, epoch 22 is the lowest validation loss of the thirty, 0.4459, and epoch 30 ends at 0.4492 and 46,020. The last line is the trap: loss after the loop is the loss of the 61-row batch that closed epoch 30, 0.2665, and the epoch's training loss is 0.7260, 2.7 times it. Nothing in the loop says which one a print(loss.item()) reports. The loop as a function for the runs ahead, on a loader built fresh for each, named train_batches because run, the full-batch loop, is train_loop.py's:

def loader_of(batch_size, seed=42, shuffle=True):
    g = torch.Generator().manual_seed(seed)
    return DataLoader(dataset, batch_size=batch_size, shuffle=shuffle, generator=g)

def train_batches(loader, epochs):
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    train_losses, val_losses, maes = [], [], []
    for epoch in range(epochs):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        train_losses.append(validate(model, X_fit, y_fit))
        val_losses.append(validate(model, X_val, y_val))
        maes.append(score(model, X_val, val_rows)[0])
    return train_losses, val_losses, maes

t, v, m = train_batches(loader_of(64), 30)
print(f"{v[-1]:.4f}", f"{m[-1]:,.0f}")

0.4492 46,020

A fresh seeded network, a fresh seeded loader, the same 30 epochs and the same two numbers: the function is the loop above, and from here on a setting is one call.

DON'T:

• DON'T report loss after the inner loop as the epoch's training loss. It is the last batch's, 0.2665 here against 0.7260 on all the fit rows, and the loop raises nothing when you do.
• DON'T put the validation pass inside the batch loop. It then runs 27 times an epoch over all 575 validation rows, and 26 of every 27 readings score a model partway through its epoch, which Q4 shows reported as a "lowest validation loss" no epoch's model ever had.

The Noise in a Batch's Loss

What. A batch's loss is the loss on one slice's rows, computed from the parameters as they stood before that batch's update, so the 27 numbers of an epoch scatter around the epoch's loss and the curve they draw jumps. The order the loader draws is a second source of noise: a different seed is a different sequence of 810 slices, and the number at epoch 30 moves with it. Why. The epoch curve of Lesson 4.2.3 is the one to read, and a curve of batch losses printed every step is not it, however much finer it looks. And a difference between two mini-batch runs is read beside the spread the order alone produces, or it is read as a difference between settings when it is a difference between draws, Lesson 3.1.2's one split as one draw, now one order as one draw. When. Whenever a loop prints a loss per batch, and whenever two runs that differ in a setting also differ in their order.

The loop above with one list added, every batch's loss as it was computed, and the epoch's loss beside it:

loader = loader_of(64)
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
batch_losses, epoch_losses = [], []
for epoch in range(1, 31):
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        batch_losses.append(loss.item())
    epoch_losses.append(validate(model, X_fit, y_fit))
print(len(batch_losses), len(epoch_losses))
first, last = batch_losses[:27], batch_losses[-27:]
print(" ".join(f"{b:.3f}" for b in first[:5]), f"{first[-1]:.3f}")
print(" ".join(f"{b:.3f}" for b in last[:5]), f"{last[-1]:.3f}")
print(f"{min(last):.3f} {max(last):.3f}")
print(f"{np.mean(last):.4f} {np.std(last):.4f} {epoch_losses[-1]:.4f}")

810 30
21.360 10.682 3.099 2.125 2.092 1.386
0.417 0.278 0.608 0.387 0.854 0.267
0.229 2.964
0.7677 0.6757 0.7260

Epoch 1's batch losses open at 21.360 on the first slice, the seeded start, and close at 1.386, the loss falling within the epoch as 27 updates land. Epoch 30's close at 0.267 and run from 0.229 to 2.964 across the epoch, a spread, np.std of the 27, of 0.6757 around a mean of 0.7677, against 0.7260 on all the fit rows in evaluation mode. The last batch's 0.267 is a slice of 61 listings that happened to be priced well, the batch at 2.964 holds some of the dear ones, and the mean of the 27 is not the epoch's loss either, because each was computed on a different set of parameters from the one that ended the epoch. The chart shows the two curves at once:

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 811), batch_losses, label="one batch, before its update")
ax.plot(range(27, 811, 27), epoch_losses, label="all fit rows, after the epoch")
ax.set_yscale("log")
ax.set_xlabel("Update")
ax.set_ylabel("Mean squared error, price in hundreds of thousands")
ax.set_title("Batches of 64 on the listings network, 30 epochs")
ax.legend()
fig.savefig("batch_curve.png", dpi=150, bbox_inches="tight")
print(len(ax.lines), len(ax.lines[0].get_ydata()), len(ax.lines[1].get_ydata()))
print(ax.lines[1].get_xdata()[0], ax.lines[1].get_xdata()[-1])
plt.close(fig)

2 810 30
27 810

Both lines sit on an axis of updates, the batch line with 810 points and the epoch line with 30, placed at updates 27, 54 and so on to 810, where each epoch ends, checked as Lesson 2.7.2 checked a chart. In batch_curve.png the batch line is a band that jumps between 0.2 and 4.4 for the whole second half, and the epoch line runs through it from 1.03 to 0.73. A loss curve from a mini-batch loop is the epoch line. The order is the second noise:

for seed in (0, 1, 2, 3):
    t, v, m = train_batches(loader_of(64, seed=seed), 30)
    print(seed, f"{v[-1]:.4f}", f"{m[-1]:,.0f}")
t, v, m = train_batches(loader_of(64, shuffle=False), 30)
print(f"{v[-1]:.4f}", f"{m[-1]:,.0f}")

0 0.4498 46,615
1 0.4840 51,462
2 0.4452 45,574
3 0.5442 51,546
0.4536 45,596

Five runs that differ in nothing but the order of their 810 slices end at epoch 30 between 0.4452 and 0.5442 in validation loss and between 45,574 and 51,546 in MAE, the seeded run above, 0.4492, among them. That spread is the noise any single mini-batch number carries at this epoch, and a difference smaller than it between two settings is not a difference the runs show. The last line is no shuffle at all, the fit rows in the order train_test_split left them, a random order drawn once and replayed 30 times, and it lands at 0.4536, inside the spread. A fixed order is not what breaks a loop. A sorted one is:

by_price = fit_rows.sort_values("price")
X_sorted, y_sorted = prep(by_price)
print(f"{y_sorted[0].item():.2f} {y_sorted[-1].item():.2f}")
in_order = DataLoader(TensorDataset(X_sorted, y_sorted), batch_size=64)
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
for epoch in range(1, 3):
    batch = []
    for xb, yb in in_order:
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        batch.append(loss.item())
    print(epoch, f"{batch[0]:.3f}", f"{batch[-1]:.3f}", f"{validate(model, X_val, y_val):.4f}")
g = torch.Generator().manual_seed(42)
reshuffled = DataLoader(TensorDataset(X_sorted, y_sorted), batch_size=64, shuffle=True, generator=g)
t, v, m = train_batches(reshuffled, 30)
print(f"{v[-1]:.4f}", f"{m[-1]:,.0f}")

0.60 19.30
1 1.413 92.584 35.2091
2 147.708 nan nan
0.4541 47,541

Sorted by price, the first slice holds the 64 cheapest listings, from 60,000, and the last the 61 dearest, up to 1,930,000. Each batch's gradient pulls the network toward its own price band: the last batch of epoch 1 scores 92.584 on the dearest listings, the step taken on it sends the first batch of epoch 2, the cheapest again, to 147.708, and by the end of epoch 2 the loss is nan, Lesson 4.2.1's explosion from an order rather than a rate. The same sorted tensors through a shuffled loader end at 0.4541, inside the spread, so the shuffle is what removes it, and a table often arrives sorted, by date, by neighborhood or by the price a query ordered it by. One more order, the one a careless loop makes by accident:

indexed = TensorDataset(X_fit, y_fit, torch.arange(len(X_fit)))
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
firsts = []
for epoch in range(30):
    rebuilt = DataLoader(indexed, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
    for i, (xb, yb, idx) in enumerate(rebuilt):
        if i == 0:
            firsts.append(idx[0].item())
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
print(len(set(firsts)), firsts[:3])
print(f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}")
kept = DataLoader(indexed, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
starts = [next(iter(kept))[2][0].item() for epoch in range(30)]
print(len(set(starts)), starts[:3])

1 [1426, 1426, 1426]
0.5730 53,330
30 [1426, 591, 1415]

A loader built inside the epoch loop with a generator seeded 42 each time is a new generator at its start every epoch, so every epoch opens on row 1426 and replays the one order: 30 epochs, one distinct first row, and the loop ends at 0.5730 and 53,330 on that one order. The loader built once, kept, opens its 30 epochs on 30 different rows. A shuffle=True that reshuffles nothing is what the trap leaves you with, and nothing prints to say so.

DON'T:

• DON'T leave shuffle off on rows that arrive sorted. The fit rows sorted by price, batched in that order at 0.1, printed nan by the end of epoch 2, and the same rows through a shuffled loader ended at 0.4541.
• DON'T build the loader inside the epoch loop with a fresh seeded generator. Every epoch then opens on row 1426 and replays one order, where the loader built once opened its 30 epochs on 30 different rows.
• DON'T read 0.03 between two mini-batch runs as a difference between settings. Seeds 0 to 3 spread 0.4452 to 0.5442 at epoch 30 with nothing changed but the order, and a comparison between settings is read beside that spread.

What Mini-Batches Buy, Measured

What. The full batch, through the imported run, against batches of 64 on the same network, the same seed and the same rate, each setting one call, with its updates counted, its validation loss and MAE read, and its seconds taken by time.perf_counter(), Lesson 3.6.3's clock. Batches of 16 and 256 for 30 epochs beside them. Why. Three claims are made for mini-batches, more updates per pass, less memory per step and more speed, and on 1,725 rows the first two hold and the third does not. A number from a run settles which, where a sentence would not. When. Once here, and again on any table where the batch size is a choice: the comparison is updates, validation score and seconds, in one table.

import time
import numpy as np
import torch
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
from prep_net import X_fit, y_fit, X_val, y_val, val_rows
from train_loop import loss_fn, network, run, validate, score

dataset = TensorDataset(X_fit, y_fit)

def full_batch(epochs):
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    run(model, optimizer, epochs)
    return model, epochs

def mini_batch(batch_size, epochs):
    g = torch.Generator().manual_seed(42)
    loader = DataLoader(dataset, batch_size=batch_size, shuffle=True, generator=g)
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    for epoch in range(epochs):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return model, epochs * len(loader)

def report(name, fitted, seconds):
    model, updates = fitted
    val_loss = validate(model, X_val, y_val)
    print(f"{name:<12} {updates:>5} {val_loss:.4f} {score(model, X_val, val_rows)[0]:>7,.0f} {seconds:5.2f}")

# the first optimizer a process builds pays a one-time setup, so one is built before the clock starts
optim.SGD(network().parameters(), lr=0.1)
print("setting      steps   loss     MAE  secs")
for epochs in (10, 100, 500):
    start = time.perf_counter()
    fitted = full_batch(epochs)
    report(f"full x {epochs}", fitted, time.perf_counter() - start)
for epochs in (10, 30, 100):
    start = time.perf_counter()
    fitted = mini_batch(64, epochs)
    report(f"64 x {epochs}", fitted, time.perf_counter() - start)
for batch_size in (16, 256):
    start = time.perf_counter()
    fitted = mini_batch(batch_size, 30)
    report(f"{batch_size} x 30", fitted, time.perf_counter() - start)

setting      steps   loss     MAE  secs
full x 10       10 0.9882  74,174  0.03
full x 100     100 0.4683  47,098  0.11
full x 500     500 0.4442  45,585  0.41
64 x 10        270 0.4721  48,617  0.37
64 x 30        810 0.4492  46,020  0.73
64 x 100      2700 0.4692  47,158  2.31
16 x 30       3240 0.4825  46,917  1.91
256 x 30       210 0.4817  46,857  0.37

The seconds are this machine's and differ on yours, and only their order is read. The one-time setup is real: the first optim.SGD a process builds took 1.5 seconds here and the second a tenth of a millisecond, so it is paid before the clock starts, or the first row of the table carries it. The loss and MAE columns are the same on every run, because every setting starts from the same seeded network and the same seeded loader.

Read by passes over the rows, mini-batches win. Ten epochs of 64, 270 updates, reach 0.4721 and 48,617 where ten full-batch epochs reach 0.9882 and 74,174, and the full batch needs 100 epochs to get to 0.4683. Thirty epochs of 64, 810 updates, reach 0.4492 and 46,020, within 500 of the 45,585 that 500 full-batch epochs reach. That is the first thing a mini-batch buys: 27 updates for the arithmetic of one pass.

Read by seconds, the full batch wins on this table at every matched score: 0.11 seconds to 0.4683 against 0.37 to 0.4721, and 0.41 for 500 epochs against 0.73 for 30 epochs of 64 and 1.91 for 30 of 16, whose 3,240 updates cost the most and reach 0.4825. A pass over the rows is one step for the full batch and one step per slice for a loader: 500 full-batch passes took 0.41 seconds, run's validation pass included, where 30 passes took 0.37 seconds at 256, 0.73 at 64 and 1.91 at 16, so on a table that fits one step the seconds grow with the slices a pass is cut into. What a mini-batch buys in seconds arrives on data that does not fit one step, and this table fits:

model = network()
with torch.no_grad():
    hidden = model[0](X_fit)
    print(hidden.shape, hidden.numel() * hidden.element_size())
    xb, yb = next(iter(DataLoader(dataset, batch_size=64)))
    hidden = model[0](xb)
    print(hidden.shape, hidden.numel() * hidden.element_size())
print(60_000 * 28 * 28 * 4)

torch.Size([1725, 32]) 220800
torch.Size([64, 32]) 8192
188160000

model[0] is the first layer of the Sequential, and its output for every fit row is a (1725, 32) tensor of 220,800 bytes, numel() times element_size(), four bytes a float32, the size of the ReLU's output, which a forward pass holds until backward() has used it. On a batch of 64 the same tensor is 8,192 bytes. Section 4.5's 60,000 images of 28 by 28 pixels are 188,160,000 bytes of input alone in float32, before any layer's output, all of it held by one full-batch step and a slice's share of it by a mini-batch step. That is the second thing a mini-batch buys. The last thing to read is where the 64 run was lowest:

g = torch.Generator().manual_seed(42)
loader = DataLoader(dataset, batch_size=64, shuffle=True, generator=g)
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses = [], []
for epoch in range(100):
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    train_losses.append(validate(model, X_fit, y_fit))
    val_losses.append(validate(model, X_val, y_val))
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{train_losses[lowest]:.4f}", f"{val_losses[lowest]:.4f}")
print(100, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}")
print(sum(v > val_losses[lowest] for v in val_losses[lowest + 1:]))

22 0.7394 0.4459
100 0.6749 0.4692
78

The validation loss is lowest at epoch 22, 0.4459, and every one of the 78 epochs after it scores above that. By epoch 100, 2,700 updates, the training loss has fallen from 0.7394 to 0.6749 and the validation loss has risen to 0.4692: the training curve falling while the validation curve rises, Lesson 4.2.3's gap, on a real network of 641 parameters rather than on noise columns. The table's 64 x 100 row is that model, worse than 64 x 30 after more than three times the updates. Past the validation minimum the further updates, taken together, fitted the fit rows better and the validation rows no better, which Section 4.4 takes up.

DON'T:

• DON'T choose a batch size for speed on a table this size. 500 full-batch epochs took 0.41 seconds against 0.73 for 30 epochs of 64 and 1.91 for 30 of 16, and the seconds grew with the slices each pass was cut into.
• DON'T run mini-batches on past the validation minimum because the training loss still falls. It fell from 0.7394 at epoch 22 to 0.6749 at 100 while the validation loss rose from 0.4459 to 0.4692, and the epoch-100 model is the worse one.
• DON'T rank 16, 64 and 256 by their epoch-30 numbers. 0.4825, 0.4492 and 0.4817 sit inside the 0.4452 to 0.5442 the order alone produced, so these runs rank none of them.

Lab Exercises

Lab Exercise 1: Batches of 32, Counted and Read Back

In batches32.py, on the table prep_net.py builds, print the batches an epoch holds at a batch size of 32 and the size of the last one, by arithmetic. Build the TensorDataset and a loader over it at 32 with shuffle=True and a generator seeded 42, and print len(loader), the first three batch sizes, the last one and their sum. Build the indexed dataset with a third tensor of row numbers and a loader like the first, run one epoch, and print how many batches equal their slices of X_fit and y_fit, how many batches there were, and whether the row numbers of the epoch sorted are every row once. Build two loaders from generators seeded 7, print the first three row numbers each opens on, then the first three the first loader's second epoch opens on. Print how many batches a drop_last=True loader at 32 holds and how many rows they cover.

Then write one epoch by hand at 32 with torch.randperm from a generator seeded 42, counting its updates, and print the count, the last slice's size, and the validation loss and MAE after it. Run 54 full-batch epochs from a fresh seeded network and print the same two numbers. Say in a comment which comparison each pair of numbers answers.

Lab Exercise 2: The Loop at 32, Read Right

In loop32.py, build one optimizer before any clock starts, then run the restructured loop at a batch size of 32 for 30 epochs from a seeded network and a loader seeded 42, timed with time.perf_counter(), keeping the epoch's training loss in evaluation mode, the validation loss, the MAE and every batch's loss. Print the three per-epoch numbers at epochs 1, 10 and 30, the epoch of the lowest validation loss with that loss and its MAE, the number of batch losses, the last batch's loss beside the last epoch's training loss, the first five and the last batch losses of epoch 30 with their smallest, largest, mean and np.std, and the updates and seconds of the run.

Draw the batch losses against the update number with the epoch's training and validation losses placed at the end of each epoch, three lines on one log-scaled Axes, labelled, saved as batch32_curve.png, and print how many lines the Axes holds, how many points each has, and the epoch at which the validation line is lowest. Then time 500 full-batch epochs from a fresh seeded network and print its validation loss, MAE, updates and seconds. Finish with the 30-epoch loop at 32 on generators seeded 0 and 1, printing each run's validation loss and MAE, and say in a comment what the three runs at 32 and the full batch agree on.

import math
import torch
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
from prep_net import X_fit, y_fit, X_val, y_val, val_rows
from train_loop import loss_fn, network, run, validate, score

print(math.ceil(len(X_fit) / 32), len(X_fit) % 32)
dataset = TensorDataset(X_fit, y_fit)
loader = DataLoader(dataset, batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(42))
print(len(loader))
sizes = [len(xb) for xb, yb in loader]
print(sizes[:3], sizes[-1], sum(sizes))

indexed = TensorDataset(X_fit, y_fit, torch.arange(len(X_fit)))
loader = DataLoader(indexed, batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(42))
seen, same = [], []
for xb, yb, idx in loader:
    same.append(torch.equal(xb, X_fit[idx]) and torch.equal(yb, y_fit[idx]))
    seen.append(idx)
seen = torch.cat(seen)
print(sum(same), len(same), torch.equal(seen.sort().values, torch.arange(len(X_fit))))
a = DataLoader(indexed, batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(7))
b = DataLoader(indexed, batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(7))
print(next(iter(a))[2][:3], next(iter(b))[2][:3])
print(next(iter(a))[2][:3])
dropped = DataLoader(dataset, batch_size=32, shuffle=True, drop_last=True, generator=torch.Generator().manual_seed(42))
print(len(dropped), sum(len(xb) for xb, yb in dropped))

54 29
54
[32, 32, 32] 29 1725
54 54 True
tensor([1696,  411,  303]) tensor([1696,  411,  303])
tensor([ 702, 1370, 1318])
53 1696

model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
g = torch.Generator().manual_seed(42)
perm = torch.randperm(len(X_fit), generator=g)
updates = 0
for start in range(0, len(X_fit), 32):
    idx = perm[start:start + 32]
    loss = loss_fn(model(X_fit[idx]), y_fit[idx])
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    updates += 1
print(updates, len(idx))
print(f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}")

model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
run(model, optimizer, 54)
print(f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}")
# 54 updates from slices of 32 after one pass over the rows: 1.3056 and 80,440.
# 54 updates from the full table after 54 passes: 0.5143 and 49,380.
# Per update the full batch wins, per pass over the rows the slices do: one
# pass left the loss at 1.3056 where one full-batch pass left it at 7.9833.

54 29
1.3056 80,440
0.5143 49,380

import time
import numpy as np
import matplotlib.pyplot as plt
import torch
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
from prep_net import X_fit, y_fit, X_val, y_val, val_rows
from train_loop import loss_fn, network, run, validate, score

dataset = TensorDataset(X_fit, y_fit)

def loader_of(batch_size, seed=42):
    g = torch.Generator().manual_seed(seed)
    return DataLoader(dataset, batch_size=batch_size, shuffle=True, generator=g)

# the first optimizer a process builds pays a one-time setup, so one is built before the clock starts
optim.SGD(network().parameters(), lr=0.1)
loader = loader_of(32)
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses, maes, batch_losses = [], [], [], []
start = time.perf_counter()
for epoch in range(1, 31):
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        batch_losses.append(loss.item())
    train_losses.append(validate(model, X_fit, y_fit))
    val_losses.append(validate(model, X_val, y_val))
    maes.append(score(model, X_val, val_rows)[0])
    if epoch in (1, 10, 30):
        print(epoch, f"{train_losses[-1]:.4f}", f"{val_losses[-1]:.4f}", f"{maes[-1]:,.0f}")
seconds = time.perf_counter() - start
lowest = int(np.argmin(val_losses))
print(lowest + 1, f"{val_losses[lowest]:.4f}", f"{maes[lowest]:,.0f}")
print(len(batch_losses), f"{batch_losses[-1]:.4f}", f"{train_losses[-1]:.4f}")
last = batch_losses[-54:]
print(" ".join(f"{b:.3f}" for b in last[:5]), f"{last[-1]:.3f}")
print(f"{min(last):.3f} {max(last):.3f} {np.mean(last):.4f} {np.std(last):.4f}")
print(f"{len(batch_losses)} updates in {seconds:.2f} s")

1 3.5206 2.5136 116,506
10 0.8285 0.4734 47,181
30 0.7183 0.4608 46,662
9 0.4433 45,986
1620 0.2202 0.7183
0.273 0.498 0.329 0.199 0.297 0.220
0.151 5.015 0.7490 0.9176
1620 updates in 1.21 s

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(range(1, 1621), batch_losses, label="one batch, before its update")
ax.plot(range(54, 1621, 54), train_losses, label="all fit rows, after the epoch")
ax.plot(range(54, 1621, 54), val_losses, label="validation rows, after the epoch")
ax.set_yscale("log")
ax.set_xlabel("Update")
ax.set_ylabel("Mean squared error, price in hundreds of thousands")
ax.set_title("Batches of 32 on the listings network, 30 epochs")
ax.legend()
fig.savefig("batch32_curve.png", dpi=150, bbox_inches="tight")
print(len(ax.lines), [len(line.get_ydata()) for line in ax.lines])
print(int(np.argmin(ax.lines[2].get_ydata())) + 1)
plt.close(fig)

start = time.perf_counter()
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
run(model, optimizer, 500)
seconds = time.perf_counter() - start
print(f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}", f"500 updates in {seconds:.2f} s")

for seed in (0, 1):
    loader = loader_of(32, seed=seed)
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    for epoch in range(30):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    print(seed, f"{validate(model, X_val, y_val):.4f}", f"{score(model, X_val, val_rows)[0]:,.0f}")
# Three orders at 32 end between 0.4608 and 0.6010 at epoch 30, the full
# batch's 500 updates end at 0.4442 in under half the seconds, and every
# run's validation loss sits far under the 16.9906 of the seeded start.

3 [1620, 30, 30]
9
0.4442 45,585 500 updates in 0.49 s
0 0.5799 52,449
1 0.6010 60,909

Theory Assessment

Q1: Thirty Epochs, Two Counts

A teammate's loop runs the full batch for 30 epochs and yours runs batches of 64 for 30, both on the 1,725 fit rows. Say how many updates each made and how many times each saw every fit row, what len(loader) returns at 64 and why it is 27 and not 26, what drop_last=True changes about both counts, and what the check torch.equal(perm.sort().values, torch.arange(1725)) proved about the hand epoch that a batch count alone could not.

Hint: Count the slices a permutation of 1,725 rows cuts into at 64, and remember the one that is shorter. Then ask what a sorted permutation looks like when nothing is missing and nothing is repeated.

Q2: One Seed, Thirty Orders

Two loaders built from generators seeded 42 opened on rows 1426, 827 and 848 alike, the first loader's second epoch opened on 591, 174 and 213, and a loader rebuilt inside the epoch loop with a generator seeded 42 each time opened every epoch on 1426. Say what the generator holds and why the second epoch differs from the first, why the rebuilt loader replays one order, and what a next(iter(loader)) taken for a check does to the training run that follows on the same loader.

Hint: A generator is a stream with a position. Ask what moves the position, what resets it, and which of the three loaders had its position reset thirty times.

Q3: Three Numbers for One Epoch

Epoch 30 of the loop at 64 produced a last batch loss of 0.267, a mean of its 27 batch losses of 0.7677 and a loss of 0.7260 on all the fit rows in evaluation mode. Say which of the three is the epoch's training loss and what each of the other two measures, why the mean of the batch losses is not the epoch's loss even though every row is in exactly one batch, and which of the two lines on batch_curve.png Lesson 4.2.3's reading applies to.

Hint: Place each of the three numbers on a timeline of the epoch's 27 updates, and ask which parameters each one was computed with.

Q4: The Finer Curve

Read this script without running it, then say what the teammate who wrote it got right and wrong about its two printed lines.

import torch
from torch import optim
from torch.utils.data import TensorDataset, DataLoader
from prep_net import X_fit, y_fit, X_val, y_val
from train_loop import loss_fn, network, validate

dataset = TensorDataset(X_fit, y_fit)
model = network()
optimizer = optim.SGD(model.parameters(), lr=0.1)
train_losses, val_losses = [], []
for epoch in range(1, 31):
    loader = DataLoader(dataset, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
    for xb, yb in loader:
        loss = loss_fn(model(xb), yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        val_losses.append(validate(model, X_val, y_val))
    train_losses.append(loss.item())
print(len(train_losses), len(val_losses))
print(f"{train_losses[-1]:.4f}", f"{min(val_losses):.4f}", f"{val_losses[-1]:.4f}")

30 810
0.8128 0.4478 0.5730

The teammate reports a training loss of 0.8128 at epoch 30, a lowest validation loss of 0.4478, close to the Theory's 0.4459, and a validation curve of 810 points, finer than the Theory's 30. Say what each of the three lines that differ from the Theory's loop changed, which of the three printed numbers is a number about a model that existed at the end of an epoch, and what the report should have said.

Hint: Count the loaders the script builds and the validation passes it makes. Then ask which model the lowest of 810 readings belongs to, and what the loss at the end of epoch 30 is on all the fit rows.

# 0.8106 402
print(f"{validate(model, X_fit, y_fit):.4f}", val_losses.index(min(val_losses)) + 1)

Assignment

Assignment: A Dataset That Builds Each Row

A TensorDataset holds the whole table as tensors and hands out slices of them, and Section 4.5's loaders cannot, because each of its 60,000 images is converted to the float tensor a network reads only when a batch asks for it. The shape they read from is a Dataset class written for its data, which this lesson did not write: a class that subclasses torch.utils.data.Dataset and defines __len__, returning the row count, and __getitem__(self, i), returning row i as the tuple a TensorDataset item is, built when asked. A DataLoader over it calls __getitem__ once for every row number in a batch and stacks the 64 tuples it gets back into the (64, 18) and (64, 1) tensors the loop sees, which is the loader's collation, and nothing in the loop changes.

Write own_dataset.py with ListingsDataset(frame), a dataset over the fit frame that builds Lesson 4.3.1's eighteen columns for one listing at a time in __getitem__: the four numbers standardized with mu and sd from prep_net.py, and the three words turned into fourteen zeros and ones by hand from enc.categories_, which this lesson did not use: the fitted encoder's list of three arrays, one per text column, each holding that column's levels in the order the encoder's columns follow, seven neighborhoods, three property types and four conditions. The target is the price in hundreds of thousands as a one-element float32 tensor. Keep a counter of the calls __getitem__ receives.

Print the lengths of the three level lists and the levels of the second, the dataset's length and the counter after it, then item 0's feature shape and target and whether both equal X_fit[0] and y_fit[0]. Pull the whole dataset through a loader with batch_size=len(own) and no shuffle, and print the shape of what comes back, whether it equals X_fit and y_fit to the bit, and the counter. Build a TensorDataset loader and a ListingsDataset loader from generators seeded 42 and print whether all 27 batches of an epoch are equal, then the counter. Build one optimizer before any clock starts, then run the restructured loop for 30 epochs on a fresh loader of each kind, timed, and print whether the two lists of validation losses are equal, the final validation loss and both MAEs, and the two timings. Time one pass over each loader with no model in it and print both. End by building a Dataset subclass with __getitem__ and no __len__, and print what len() of a loader over it raises.

Hints:

• enc.categories_[1] holds 'apartment', 'house' and 'townhouse', and the one-hot column for a word sits at that column's offset plus the word's position in its list, so levels.index(word) and a running offset of 0, then 7, then 10 place the three ones among fourteen zeros.
• A loader with no shuffle and batch_size=len(own) returns the whole dataset as one batch, in order, so every row is compared at once, and len(own) costs no __getitem__ call, so the counter after it is still what the earlier lines left.

import time
import numpy as np
import torch
from torch import optim
from torch.utils.data import Dataset, TensorDataset, DataLoader
from prep_net import fit_rows, numeric, text, mu, sd, enc, X_fit, y_fit, X_val, y_val, val_rows
from train_loop import loss_fn, network, validate, score


class ListingsDataset(Dataset):
    def __init__(self, frame):
        self.numbers = frame[numeric].to_numpy()
        self.words = frame[text].to_numpy()
        self.price = frame["price"].to_numpy()
        self.mu, self.sd = mu.to_numpy(), sd.to_numpy()
        self.levels = [list(column) for column in enc.categories_]
        self.calls = 0

    def __len__(self):
        return len(self.price)

    def __getitem__(self, i):
        self.calls += 1
        scaled = (self.numbers[i] - self.mu) / self.sd
        coded = np.zeros(14)
        offset = 0
        for word, levels in zip(self.words[i], self.levels):
            coded[offset + levels.index(word)] = 1.0
            offset += len(levels)
        x = torch.tensor(np.concatenate([scaled, coded]), dtype=torch.float32)
        y = torch.tensor([self.price[i] / 100_000], dtype=torch.float32)
        return x, y


own = ListingsDataset(fit_rows)
print([len(levels) for levels in own.levels], own.levels[1])
print(len(own), own.calls)
x0, y0 = own[0]
print(x0.shape, y0, torch.equal(x0, X_fit[0]), torch.equal(y0, y_fit[0]))
whole = DataLoader(own, batch_size=len(own))
X_own, y_own = next(iter(whole))
print(X_own.shape, torch.equal(X_own, X_fit), torch.equal(y_own, y_fit), own.calls)

[7, 3, 4] ['apartment', 'house', 'townhouse']
1725 0
torch.Size([18]) tensor([2.7200]) True True
torch.Size([1725, 18]) True True 1726

held = TensorDataset(X_fit, y_fit)
a = DataLoader(held, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
b = DataLoader(own, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
print(all(torch.equal(xa, xb) and torch.equal(ya, yb) for (xa, ya), (xb, yb) in zip(a, b)))
print(own.calls)

True
3451

def train_batches(loader, epochs):
    model = network()
    optimizer = optim.SGD(model.parameters(), lr=0.1)
    val_losses = []
    for epoch in range(epochs):
        for xb, yb in loader:
            loss = loss_fn(model(xb), yb)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
        val_losses.append(validate(model, X_val, y_val))
    return val_losses, model

# the first optimizer a process builds pays a one-time setup, so one is built before the clock starts
optim.SGD(network().parameters(), lr=0.1)
a = DataLoader(held, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
b = DataLoader(own, batch_size=64, shuffle=True, generator=torch.Generator().manual_seed(42))
start = time.perf_counter()
val_a, model_a = train_batches(a, 30)
secs_a = time.perf_counter() - start
start = time.perf_counter()
val_b, model_b = train_batches(b, 30)
secs_b = time.perf_counter() - start
print(val_a == val_b, f"{val_a[-1]:.4f}", f"{score(model_a, X_val, val_rows)[0]:,.0f}", f"{score(model_b, X_val, val_rows)[0]:,.0f}")
print(f"{secs_a:.2f} {secs_b:.2f}")
start = time.perf_counter()
for xb, yb in DataLoader(held, batch_size=64):
    pass
secs_held = time.perf_counter() - start
start = time.perf_counter()
for xb, yb in DataLoader(own, batch_size=64):
    pass
secs_own = time.perf_counter() - start
print(f"{secs_held:.3f} {secs_own:.3f}")

True 0.4492 46,020 46,020
0.72 1.47
0.009 0.025

class NoLen(Dataset):
    def __getitem__(self, i):
        return X_fit[i], y_fit[i]


try:
    len(DataLoader(NoLen(), batch_size=64))
except TypeError as err:
    print(err)

object of type 'NoLen' has no len()

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want every argument a loader takes, or a dataset that builds its rows from files on disk.

• Official docs - PyTorch 2.14, torch.utils.data, the DataLoader entry. The class's full signature and one paragraph per argument: batch_size, shuffle, drop_last and generator as this lesson used them, sampler and batch_sampler for an order of your own, collate_fn for the stacking, and num_workers and pin_memory, which load batches in helper processes and into memory a GPU copies from faster, two settings no lesson of this module turns on.
• Official docs - PyTorch Tutorials, "Writing Custom Datasets, DataLoaders and Transforms", the section "Dataset class". A Dataset subclass over a CSV of face landmarks whose __getitem__ reads one image from disk when asked, the memory reason the page gives for doing so, then transforms applied per sample and a loader over it, the Assignment's shape on data that is not one tensor.

Summary

The loop did not change again. The five steps of Lesson 4.2.2 moved one level in, under a for xb, yb in loader:, and the two classes around them do what a seeded randperm and a slice did by hand in the first subtitle, which the row numbers of every batch proved. What changed is the count: an epoch stopped being an update, 27 updates stand where one stood, and every number a loop prints from here on is read by which count it belongs to and which rows it was computed on. A batch's loss belongs to one slice and one moment, an epoch's loss to all the rows and the model the epoch left, and the curve worth reading is the second.

What a mini-batch buys turned out to be two things and not three. It buys updates per pass over the rows, which is why 30 epochs of 64 came within 500 of the MAE the full batch reached in 500 epochs, and it buys a step whose memory is the slice's and not the table's. It does not buy seconds on 1,725 rows, where the full batch finished first at every matched score, and the measured table is what says so. The noise came with the bargain: 27 slice losses scatter around the epoch's, five orders of the same slices land 0.1 apart at epoch 30, and a sorted order with no shuffle reaches nan, so a result from a mini-batch loop carries its seed and its order the way a split carried its draw in Lesson 3.1.2.

The validation loss of the 64 run was lowest at epoch 22 and stayed above that for the 78 epochs after, while the training loss fell from 0.7394 to 0.6749. That is Lesson 4.2.3's gap on a real network, and the next lesson sweeps the optimizer and the rate before Section 4.4 takes up what to do about it.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

from torch.utils.data import TensorDataset, DataLoader

dataset = TensorDataset(X_fit, y_fit)  # len 1725, [i] a tuple
g = torch.Generator().manual_seed(42)  # built once, outside the loop
loader = DataLoader(dataset, batch_size=64, shuffle=True, generator=g)
for epoch in range(1, 31):
    for xb, yb in loader:              # 27 slices, a fresh order
        loss = loss_fn(model(xb), yb)  # one slice, before its update
        loss.backward()
        optimizer.step()               # one update, 810 in 30 epochs
        optimizer.zero_grad()
    # the epoch's loss: all fit rows after its updates, never loss.item()
    train_losses.append(validate(model, X_fit, y_fit))
    val_losses.append(validate(model, X_val, y_val))

indexed = TensorDataset(X_fit, y_fit, torch.arange(len(X_fit)))
for xb, yb, idx in DataLoader(indexed, batch_size=64, shuffle=True, generator=g):
    torch.equal(xb, X_fit[idx])            # True: the loader is slicing
optim.SGD(network().parameters(), lr=0.1)  # the first one built pays the setup
start = time.perf_counter()
model, updates = mini_batch(64, 30)        # 810 updates, 0.73 s here
seconds = time.perf_counter() - start      # full x 500 took 0.41 s