The Architecture Map and the Boundary

Context

Every architecture is an assumption about its data written into the shape of a network. A convolution assumes that what matters in an image is local and the same everywhere, a bag of embeddings assumes word order does not matter, and attention with positions assumes it does. The map pairs each kind of problem with the architecture whose assumptions fit it, and says what each one costs.

The choice comes first on every new project: a team handed a table of listings, a folder of scans, a queue of support tickets or a stream of sensor readings picks a model family before it writes a training loop. It comes back whenever a model fails on inputs that differ from its training inputs in one systematic way, a garment framed off center, a word order reversed, a document longer than the model was built to read.

Choose without the map and the failures are quiet. A network with a weight per position scores well on readings recorded where its training readings were and falls to near chance when the same pattern arrives later in the window. A pretrained encoder refuses a long document outright, and the same encoder used frozen on a few labels loses to a small model trained from scratch. Sections 5.1 to 5.4 built each of these shapes, and this lesson sets them side by side.

Goal

• Measure a convolution's output moving with a shifted image where a linear layer's does not, and count the 10 parameters that carry the assumption against 615,440.
• Trace one pair of reordered queries through a bag, through attention without positions and with them, and through BERT-Mini, and say which of them can tell the two apart.
• Count the parameters of a flat layer, a convolution, attention and a transformer layer at one width, and time a convolution against attention as a sequence grows to 8,192 in one script.
• Show that an encoder reads both ways and tells reordered words apart, and place decoders and encoder-decoders beside it.
• Decide between a pretrained model and training from scratch by a short run on ten labels per intent.
• Choose an architecture for tabular rows, images, short text, long documents and sequences in time, test the choice with a short run, and state where this bootcamp stops.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the transformer family placed in one picture before the runs, or the one-dimensional convolution read from its own page.

• YouTube - "Transformers, explained: Understand the model behind GPT, BERT, and T5" by Google Cloud Tech. Nine minutes on positional encodings, attention and self-attention, and the three model families this lesson places on its map, BERT, GPT and T5.
• Official docs - PyTorch 2.14, torch.nn.Conv1d. The convolution slid along one axis, its (N, C, L) input, its kernel_size, stride, padding and dilation, and the formula for its output length.

Table of Contents

• Inductive Bias
• Order, Sets and Positions
• What Each Shape Costs
• Encoders and Decoders
• Pretrained or From Scratch
• The Map
• The Boundary

Theory

Every block runs in ~/dev/bootcamp, beside text.py, Lesson 5.3.1's, attention.py, Lesson 5.3.2's, encoder.py and bert_features.pt, Lesson 5.4.1's, and image_loop.py, Lesson 4.5.2's. The sixth subtitle and Lab Exercise 2 read Module 3's listings.csv through prep_net.py, Lesson 4.3.1's, both there since Module 4: if you started at Module 5, Module 4's setup page says where the course's download of the CSV goes, and prep_net.py is the first block of Lesson 4.3.1. Nothing here installs or downloads. The first three subtitles run seeded, untrained layers, so their numbers show what a shape can tell apart, never what a trained model does. Trained figures are this machine's on its default eight threads, and seconds are compared only inside the script that timed them. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Inductive Bias

What. An inductive bias is an assumption a model makes about its data before it has seen any, built into its shape rather than learned: which inputs it treats as related, and which changes to an input it treats as no change at all. A fully connected layer assumes nothing about where a value sits, so every pixel has its own weight to every output. A convolution assumes that what matters is local, a 3 by 3 neighborhood, and the same at every position, one filter slid over the whole image, Lesson 5.1.1's weight sharing. Why. An assumption that matches the data is knowledge the network does not have to learn from examples, so it learns from fewer of them. An assumption that does not match is a limit no amount of training removes. Choosing an architecture is choosing which assumptions to pay for. When. Before the first layer of a new model is written, and whenever a model fails on inputs that differ from its training inputs in one systematic way.

One bright patch on a 28 by 28 image, the patch moved 8 pixels right, and both versions through a seeded convolution and a seeded linear layer that maps the 784 pixels to 784 outputs:

import torch
from torch import nn

image = torch.zeros(1, 1, 28, 28)
image[0, 0, 10:18, 4:10] = 1.0
moved = torch.roll(image, shifts=8, dims=3)
torch.manual_seed(42)
conv = nn.Conv2d(1, 1, 3, padding=1)
linear = nn.Linear(784, 784)
with torch.no_grad():
    c, c_moved = conv(image), conv(moved)
    l = linear(image.flatten(1)).view(1, 1, 28, 28)
    l_moved = linear(moved.flatten(1)).view(1, 1, 28, 28)
print(torch.equal(torch.roll(c, 8, dims=3), c_moved))
print(round((torch.roll(l, 8, dims=3) - l_moved).abs().max().item(), 4))
print(sum(p.numel() for p in conv.parameters()), sum(p.numel() for p in linear.parameters()))

True
0.6476
10 615440

The convolution's map of the moved patch is its map of the patch in place rolled 8 columns, bit for bit, so torch.equal holds. That is translation equivariance (move the input, and the output moves the same way and is otherwise unchanged). The linear layer's outputs for the moved patch differ from its own outputs rolled by up to 0.6476, because the moved patch meets other weights, Lesson 4.5.3's finding on FashionMNIST. The convolution's whole assumption costs 10 numbers, nine weights and a bias, against 615,440. Locality, by touching one pixel in the far corner:

poke = image.clone()
poke[0, 0, 0, 27] = 1.0
with torch.no_grad():
    print((conv(poke) != c).sum().item(), (linear(poke.flatten(1)).view(1, 1, 28, 28) != l).sum().item())
print(round(c.mean().item(), 4), round(c_moved.mean().item(), 4), round(c.amax().item(), 4), round(c_moved.amax().item(), 4))
print(round(l.mean().item(), 4), round(l_moved.mean().item(), 4))

4 784
-0.1783 -0.1783 0.8366 0.8366
0.0049 -0.0048

The corner pixel changed 4 of the convolution's 784 outputs, the ones whose 3 by 3 window holds it, and every one of the linear layer's. The convolution's map has the same mean and the same largest value before and after the move, since both read every position and the move only reorders them. Equivariance followed by a pooling over every position is translation invariance (the output does not change at all), and the linear layer has neither.

Section 5.1 measured where that leaves a network. Lesson 5.1.3's CNN kept 0.7795 two pixels right where the fully connected network fell to 0.4554, its body's maps moved one column for every four pixels, and its linear head, a weight per position, scored them by position and took four pixels to 0.2861. Lesson 5.1.3's Assignment pooled the maps with nn.AdaptiveAvgPool2d(1) and kept 0.7408 eight pixels over where the CNN fell to 0.1385, and paid in place, 0.8001 on the validation images against 0.9077. nn.AdaptiveMaxPool2d(1), and nn.AdaptiveMaxPool1d(1) for a sequence, keep each map's largest value instead of its mean, the invariance for a feature that only has to appear somewhere. Lesson 5.1.4 bought tolerance a different way, by data: the random crop trained on moved garments and took two pixels to 0.8611, a tolerance that ended where its training moves ended.

Section 5.2's shapes each add one assumption. A residual block assumes its best default is to pass its input on and add a correction, relu(path(x) + x), the path alone differing from the block's output by up to 4.6291 in Lesson 5.2.1. A network pretrained on ImageNet assumes that the features it learned there fit your images too, and on EuroSAT they did: Lesson 5.2.2's frozen ResNet-18 under a trained head reached 0.9226, and fine-tuned, 0.9567.

DON'T:

• DON'T call a convolution translation invariant. Its map moved with the patch, and the invariance comes from a pooling over every position after it, or not at all, as Lesson 5.1.3's linear head showed at four pixels.
• DON'T measure an assumption by its parameter count. The convolution's 10 numbers carry locality and sharing that the linear layer's 615,440 would have to learn from examples, one position at a time.

Order, Sets and Positions

What. A bag of embeddings treats a query as a multiset (a set that counts repeats) of its words, Lesson 5.3.1's mean over the real tokens. Attention without positions treats it as a set whose members read each other, permutation-equivariant by Lesson 5.3.2's measurement, so a mean over its outputs ignores their order, as the bag's mean does. A position table, Lesson 5.3.3's, adds a learned vector per position to each token's vector, so the same word at two positions is two different inputs. Why. Whether order matters is a property of the task, never of the data format. "From my card to my account" and "from my account to my card" move money in opposite directions, and Lesson 5.3.1's bag gave both transfer_into_account at 0.403. A shape that cannot tell two inputs apart gives them one answer whatever it is trained on. When. Whenever a text or sequence model is chosen, and whenever two inputs that differ only in order come back with the same answer.

The two queries through one seeded embedding, as a bag, through one seeded attention layer with the padding mask, and through the same layer with a seeded position table added:

import torch
from torch import nn
from text import vocab, encode, MAX_LEN
from attention import attention

a = "can i move money from my card to my account"
b = "can i move money from my account to my card"
ids = torch.tensor([encode(a), encode(b)])
pad = (ids == 0).unsqueeze(1)
real = (ids != 0).unsqueeze(-1).float()
torch.manual_seed(42)
embed = nn.Embedding(len(vocab), 64, padding_idx=0)
positions = nn.Embedding(MAX_LEN, 64)
q, k, v = nn.Linear(64, 64), nn.Linear(64, 64), nn.Linear(64, 64)

def pooled(x):
    out, _ = attention(q(x), k(x), v(x), pad)
    return (out * real).sum(dim=1) / real.sum(dim=1)

with torch.no_grad():
    x = embed(ids)
    bag = (x * real).sum(dim=1) / real.sum(dim=1)
    plain = pooled(x)
    placed = pooled(x + positions(torch.arange(MAX_LEN)))
for name, pair in (("bag", bag), ("attention", plain), ("positions", placed)):
    print(name, torch.allclose(pair[0], pair[1], atol=1e-6), round((pair[0] - pair[1]).abs().max().item(), 4))

bag True 0.0
attention True 0.0
positions False 0.0706

The bag's two vectors agree, and so do attention's: the queries hold the same ten tokens, so attention's outputs for the second are its outputs for the first reordered, and the mean over real tokens erases the reorder. With the position table added, the pooled vectors differ by up to 0.0706 before any training. The run shows what each shape is able to distinguish, and whether training uses that ability is up to the data.

The causal mask breaks the symmetry too, Lesson 5.3.2's other finding, since it is built from positions: each token sees only the tokens before it. It reads order by hiding the rest of the query from every position, which suits a model that writes text and costs a classifier every token's later context, the fourth subtitle's subject.

DON'T:

• DON'T add attention to a bag to make it read word order. Without positions its pooled vectors for the two queries agreed within 1e-6, the bag's failure one layer later.
• DON'T read a position table's difference as a trained one. The 0.0706 came from seeded rows, and a table only gives training the means to tell orders apart.

What Each Shape Costs

What. Two costs, read at one width, 64 values per token. The parameters a layer holds, which are fixed when it is built, and the work and memory one input costs, which grow with the input. The layers are a flat nn.Linear reading a 48 by 64 sequence as one vector, nn.Conv1d(64, 64, 3, padding=1) (Lesson 5.1.1's convolution slid along one axis instead of two, taking (N, C, L), channels before length), attention's three projections, and nn.TransformerEncoderLayer, the layer Lesson 5.3.3 checked its block against. Why. An architecture that fits the data still has to fit the machine. A convolution's work grows with the length of its input, attention's with the length squared, Lesson 5.3.2's square matrix, and the gap between the two decides what a long input costs long before accuracy is asked about. When. Before a model is pointed at longer inputs than it was tried on, and whenever two architectures are compared on one budget.

import time
import torch
from torch import nn
from attention import attention

torch.manual_seed(42)
layers = {
    "linear": nn.Linear(48 * 64, 48 * 64),
    "conv": nn.Conv1d(64, 64, 3, padding=1),
    "attention": nn.ModuleList([nn.Linear(64, 64) for _ in range(3)]),
    "block": nn.TransformerEncoderLayer(64, 4, 256, batch_first=True),
}
for name, layer in layers.items():
    print(name, sum(p.numel() for p in layer.parameters()))
conv, (q, k, v) = layers["conv"], layers["attention"]
gen = torch.Generator().manual_seed(42)
for n in (512, 1024, 2048, 4096, 8192):
    x = torch.randn(1, n, 64, generator=gen)
    times = {"conv": [], "attention": []}
    with torch.no_grad():
        for _ in range(5):
            start = time.perf_counter()
            c = conv(x.transpose(1, 2))
            times["conv"].append(time.perf_counter() - start)
            start = time.perf_counter()
            out, w = attention(q(x), k(x), v(x))
            times["attention"].append(time.perf_counter() - start)
    print(n, c.numel(), w.numel(), f"{min(times['conv']) * 1000:.1f}", f"{min(times['attention']) * 1000:.1f}")

linear 9440256
conv 12352
attention 12480
block 49984
512 32768 262144 1.1 2.8
1024 65536 1048576 1.7 6.5
2048 131072 4194304 0.7 9.2
4096 262144 16777216 1.2 69.0
8192 524288 67108864 1.8 255.9

The flat layer holds 9,440,256 weights and takes exactly 3,072 values, 48 tokens and no other length. The convolution, 64 * 64 * 3 + 64, and the three projections, 3 * (64 * 64 + 64), hold about twelve thousand each at any length, and the transformer layer, adding an output projection, two layer normalizations and a feed-forward network of 256 units, 49,984. .transpose(1, 2) hands the convolution its channels-first layout. The columns after the length are the convolution's output entries, attention's weight entries and the fastest of five calls of each in milliseconds. Every doubling doubled the first and quadrupled the second, 128 times the convolution's output at 8,192 tokens, and the milliseconds pulled apart too, attention's rising far faster than the convolution's, by factors that move with the machine and its load.

Whole networks, from the files that counted them: Lesson 5.1.3's fully connected network held 101,770 parameters and its CNN 20,538, ResNet-18 11,689,512, Lesson 5.3.1's bag 145,933, and BERT-Mini, by Lesson 5.4.1's count, 11,170,560. The two pretrained ones are the largest by far, and their training was paid for once, by whoever pretrained them.

DON'T:

• DON'T compare two architectures by parameter count alone. Attention's projections hold 12,480 numbers at every length, while its weights for one sequence grew to 67,108,864 entries at 8,192 tokens.
• DON'T read a flat layer over a sequence as a sequence model. Its 9,440,256 weights belong to 48 fixed positions, so the same word at another position meets other weights and a 49th token has none.

Encoders and Decoders

What. An encoder reads its whole input at once, every position attending to every other, and returns one vector per token. BERT-Mini, Lesson 5.4.1's, is one: four transformer blocks, post-norm as Lesson 5.4.1 rebuilt them, over a position table of 512 rows, pretrained by predicting hidden words from both sides. A decoder, the GPT family (generative pretrained transformers) that Lesson 5.4.1 set beside it, predicts each next token from the ones before it under Lesson 5.3.2's causal mask, and so writes text one token at a time. An encoder-decoder joins the two, the shape of machine translation: the encoder reads the whole source sentence, and the decoder writes the translation token by token, reading its own earlier tokens and, through a second attention whose keys and values come from the encoder, the source. Why. The three differ in what each position may read, so they differ in what they are for: an encoder classifies or tags text that already exists, a decoder generates it, an encoder-decoder turns one sequence into another. Using one for another's job either hides context a task needs or reads tokens a task must not see. When. Whenever a pretrained model is chosen for a task, and whenever its configuration is read.

from huggingface_hub.utils import logging as hub_logging
from transformers import logging
logging.set_verbosity_error()
logging.disable_progress_bar()
hub_logging.set_verbosity_error()

import torch
from encoder import tokenizer, load, pooled

model = load()
model.eval()
pair = tokenizer(["where is my card?", "where is my card."], return_tensors="pt")
print(pair["input_ids"])
with torch.no_grad():
    hidden = model(**pair).last_hidden_state
print((hidden[0] - hidden[1]).abs().max(dim=-1).values.round(decimals=4))
a = "can i move money from my card to my account"
b = "can i move money from my account to my card"
with torch.no_grad():
    p = pooled(model, tokenizer([a, b], return_tensors="pt"))
print(round((p[0] - p[1]).abs().max().item(), 4), model.config.is_decoder)

tensor([[ 101, 2073, 2003, 2026, 4003, 1029,  102],
        [ 101, 2073, 2003, 2026, 4003, 1012,  102]])
tensor([0.9678, 1.3479, 1.6571, 1.0212, 0.9658, 1.8576, 2.0180])
0.1275 False

The first five lines are Lesson 5.4.1's, which keep transformers and the Hub to their errors before encoder loads anything. The two queries differ in their last character alone, ? against ., ids 1029 and 1012, and every one of the seven positions' outputs moved, the [CLS] token at the front by 0.9678: the first position read the last. Lesson 5.3.2's causal mask, given the same pair of endings, left every position but the last unchanged. The reordered pair's pooled vectors differ by up to 0.1275, the pretrained position table at work, where the bag's agreed exactly, and the configuration's is_decoder is False.

The encoder's other limit is its length. Lesson 5.4.1 handed it 602 ids and the 512 rows of its position table refused them, so a document longer than the table is either cut to its first 512 tokens, with everything after them gone, or split into pieces the encoder reads one at a time and combined afterward, and either way the model never sees it whole.

DON'T:

• DON'T read a pretrained encoder's limit as a setting. Its 512 rows were trained, and a 513th row, if one were added, would hold numbers no training ever touched.
• DON'T classify with a causal model's view when the whole text is in hand. BERT-Mini's first position read the query's last character, and under a causal mask it could not.

Pretrained or From Scratch

What. The choice between a model whose weights were trained on far more data than yours and one started from a seeded draw on yours alone, with three ways of using the first: frozen features under a trained head, every weight fine-tuned, or something between, Lesson 5.2.3's ladder. Here it is read on Lesson 5.4.2's Assignment draw, train.groupby("category").sample(n=10, random_state=42), ten queries for each of the 77 intents. Why. Labels are the expensive part of most projects, and few labels is the case transfer learning exists for. The pretrained weights help only as far as what they learned fits your task, and how far that is decides which of the three ways wins, which a short run measures and no rule settles. When. At the start of every project with a pretrained model available for its kind of input, and again whenever the number of labels changes by a large factor.

Lesson 5.4.2's Assignment measured two ends on those 770 queries: the bag from scratch, its vocabulary rebuilt from the 770, best at 0.6555 in epoch 29, and BERT-Mini fine-tuned for 10 epochs at a peak of 3e-4 with warmup and decay, 0.7120. The third way is the frozen end, a head on the mean-pooled features Lesson 5.4.1 saved in bert_features.pt, trained by the bag's own recipe, so the two differ only in what the head reads: Adam at 0.01, batches of 32 shuffled by a generator seeded 42, the weights seeded 42 right before they are built, 30 epochs and the best one kept. Then the same head at 0.001, Lesson 5.4.1's rate:

import time
import torch
from torch import nn, optim
from torch.utils.data import DataLoader, TensorDataset
from text import train, intents
from image_loop import train_epoch, accuracy

cached = torch.load("bert_features.pt")
F, Y, G, Z = cached["F"], cached["Y"], cached["G"], cached["Z"]
few = train.groupby("category").sample(n=10, random_state=42)
rows = torch.tensor(train.index.get_indexer(few.index))
print(F[rows].shape, torch.equal(Y[rows], torch.tensor([intents.index(c) for c in few["category"]])))
val_features = DataLoader(TensorDataset(G, Z), batch_size=250)
for lr in (0.01, 0.001):
    start = time.perf_counter()
    loader = DataLoader(TensorDataset(F[rows], Y[rows]), batch_size=32, shuffle=True, generator=torch.Generator().manual_seed(42))
    torch.manual_seed(42)
    head = nn.Linear(256, 77)
    optimizer = optim.Adam(head.parameters(), lr=lr)
    best = (0.0, 0)
    for epoch in range(1, 31):
        train_epoch(head, optimizer, loader)
        acc = accuracy(head, val_features)
        if acc > best[0]:
            best = (acc, epoch)
    print(lr, f"{best[0]:.4f}", best[1], f"{time.perf_counter() - start:.1f}")

torch.Size([770, 256]) True
0.01 0.5895 21 2.2
0.001 0.5440 29 1.1

bert_features.pt holds a row per training query in train's frame order, so train.index.get_indexer(few.index) turns the 770 drawn row labels into their positions there, and the labels read at those positions equal the drawn queries' own intents. Each line is the rate, the best validation accuracy and its epoch, and the seconds. Frozen, BERT-Mini's features reach 0.5895 at 0.01 and 0.5440 at 0.001, under the bag's 0.6555 by the bag's own recipe: a model pretrained to fill in hidden words, averaged over a query's tokens, gives features that separate 77 banking intents less well than a table trained on the 770 themselves. Fine-tuned, the same weights reach 0.7120, five and a half points over the bag and twelve over themselves frozen, at the price of a pass forward and backward through eleven million parameters on every batch, where the frozen head reads features computed once.

So the frozen end of the ladder, which reached 0.9226 on EuroSAT in Lesson 5.2.2, lost here at both rates, and the fine-tuned end won. ImageNet's features separated land uses before any training, Lesson 5.2.1's nearest-mean probe, and BERT-Mini's mean-pooled features did not separate intents, so the pretrained weights paid only once training moved them. The same order held on all 8,003 labels, frozen 0.7955 in Lesson 5.4.1, the bag 0.8760 and the fine-tune 0.8955, and the fine-tune's lead over the bag grew from 1.95 points there to 5.65 on the 770, Lesson 5.4.2's measure: the fewer the labels, the more the pretrained weights are worth, once they are trained.

DON'T:

• DON'T judge a pretrained model by its frozen features alone. Frozen, BERT-Mini reached 0.5895 under the bag's 0.6555, and fine-tuned on the same 770 queries 0.7120.
• DON'T read a frozen head's score at one rate as the features' ceiling. The same head on the same features reached 0.5895 at 0.01 and 0.5440 at 0.001.

The Map

What. The map pairs a kind of problem with the architecture to try first, the assumption that makes it fit, and the section that built it. Its first row is the one this module did not build: on tabular rows, columns of different meanings side by side, the first model to try is Module 3's gradient boosting, HistGradientBoostingRegressor or XGBoost, whose trees ask one column at a time whether a value is above a threshold. Why. A tree's question compares a value with a threshold, so a column's scale, its units and the order of the columns mean nothing to it, and gaps go to one side of a split, Lesson 3.5.3's HistGradientBoosting fitting a table with gaps as it came. A network needs every column scaled, Lesson 4.3.1's table, and a loop chosen and stopped. On columns with no neighbors and no order between them, the convolution's and attention's assumptions have nothing to fit. When. At the start of every project, before any training loop is written, and again when a model's failure points at an assumption.

The tree's indifference to scale, measured on Lesson 4.3.1's 18 columns, once standardized as the network read them and once with the four numeric columns in their own units:

import numpy as np
from sklearn.ensemble import HistGradientBoostingRegressor
from prep_net import fit_rows, val_rows, numeric, text, enc, X_fit, X_val

def unscaled(frame):
    return np.hstack([frame[numeric].to_numpy(), enc.transform(frame[text])])

print([round(v, 4) for v in X_fit[0, :4].tolist()], unscaled(fit_rows)[0, :4].tolist())
scaled = HistGradientBoostingRegressor(random_state=42).fit(X_fit.numpy(), fit_rows["price"])
raw = HistGradientBoostingRegressor(random_state=42).fit(unscaled(fit_rows), fit_rows["price"])
print(np.array_equal(scaled.predict(X_val.numpy()), raw.predict(unscaled(val_rows))))
print(round(scaled.score(X_val.numpy(), val_rows["price"]), 3), scaled.n_iter_)

[-0.8938, -0.3373, -0.8639, 1.0243] [59.5, 3.0, 1.0, 8.6]
True
0.819 100

The first fit row reads -0.8938 standard deviations of area in one table and 59.5 square meters in the other, and the two boosters predict exactly the same prices for all 575 validation listings, np.array_equal: every split found the same rows on each side of its threshold, since standardizing keeps every value's order. The price goes in as it is, in currency, since a tree's leaves hold prices. At its defaults, 100 rounds, the booster reaches an R² of 0.819 on the validation rows, with no loop written. Lab Exercise 2 sets it beside Module 4's network on the same rows.

Every row is where to start, and a short run against the cheapest baseline that fits decides where to finish, before any long training job.

DON'T:

• DON'T reach for a network first on tabular rows. The booster needed no scaling and no loop, and predicted identically on standardized and raw columns.
• DON'T read the map's first choice as the winner. Module 3's own Lesson 3.5.3 found its booster under the logistic pipeline on sold_in_30_days, 0.612 against 0.642 in area under the curve, and Lab Exercise 2 finds it under a network on the price.

The Boundary

What. This bootcamp's line, stated as Module 5's introduction states it: architecture is in, application is out. You have built the convolution, the residual block, attention, the transformer block and a fine-tuned encoder from operations you wrote. Prompting (writing instructions a large pretrained language model follows), retrieval (fetching documents to hand such a model with a question) and agent systems (a model that calls tools in a loop to finish a task) belong to a separate AI Engineering product and are absent here. Why. Application work stands on a model someone else trained and served, and it is a different craft with its own failures. This bootcamp's craft is the model and what keeps it working: Module 6 takes a model from a script to a monitored service, with tests, tracked experiments, an endpoint, a container, a database and a check on drift that says when to retrain. When. Every time a task in front of you is about getting a model to answer well rather than building, training and running one.

The map ends where the boundary does. A decoder and an encoder-decoder stand on it named and not built, and a team that needs one to write text begins from a pretrained one, where the other product begins.

DON'T:

• DON'T read the boundary as a ranking. It says what this bootcamp covers, and Module 6 is the half of production work that every model, a fine-tuned encoder included, still needs.
• DON'T call a model done when its validation score is. Every model in this module stopped at a score printed by a script, and none of them yet answers a request, logs it or notices that its inputs have changed, Module 6's work.

Lab Exercises

Lab Exercise 1: A Fault Anywhere in the Window

A pump's vibration sensor sends windows of 64 readings, and each window holds one of three faults: a knock, five readings rising to 2.0 and falling back, a judder, rising to 2.0 and swinging to -2.0, or a sag, falling to -2.0 and rising back. The recorded windows all caught the fault in their first 32 readings, and in service it can arrive anywhere.

In faults.py, build the data from torch.Generator().manual_seed(42): noise of 0.3 * torch.randn(n, 1, 64), a label from torch.randint(0, 3, ...) and a start position per window, drawn in that order, the fault's five values added from its start. Draw 3,000 fit windows and 1,000 check windows with starts from 0 to 27, then 1,000 moved windows with starts from 32 to 59. Print the fit windows' shape and their count per fault.

Write run(name, model), which trains a model for 30 full-batch epochs of Adam at 0.01 with nn.functional.cross_entropy, switches it to evaluation mode, and prints its name, its accuracy on the check windows and on the moved windows to four decimals, and its parameter count. Seed 42 before each of four models: a linear layer on the flattened window, an nn.Conv1d(1, 8, 5) with a ReLU and a linear layer on its flattened maps, the same convolution with nn.AdaptiveMaxPool1d(1) before its linear layer, and the same with nn.AdaptiveAvgPool1d(1). Say which assumption each model makes and which one the problem needs.

Lab Exercise 2: Booster or Network on the Listings

A pricing team has Lesson 4.3.1's 18 columns and Lesson 4.4.3's network, network(128) early-stopped, which misses a validation listing by 44,825 on average with an R² of 0.859. Test the map's first row against it on the same fit and validation rows.

In listing_choice.py, import fit_rows, val_rows, X_fit and X_val from prep_net. Fit HistGradientBoostingRegressor(random_state=42) on the 18 columns against the price in currency, then the same with max_depth=2 and max_iter=500, and print each one's validation MAE and R². Then give the booster three columns the network never read: year_built, the year of listing_date, and energy_rating as a number from 0 for A to 6 for G, each as a float with its gaps left as nan. Print the gaps in year_built and energy_rating among the fit rows, then the third booster's MAE and R², and say which model the validation rows choose and what the choice cost.

import torch
from torch import nn, optim

gen = torch.Generator().manual_seed(42)
faults = torch.tensor([[0.0, 1.0, 2.0, 1.0, 0.0], [0.0, 2.0, 0.0, -2.0, 0.0], [0.0, -1.0, -2.0, -1.0, 0.0]])

def windows(n, low, high):
    x = 0.3 * torch.randn(n, 1, 64, generator=gen)
    y = torch.randint(0, 3, (n,), generator=gen)
    starts = torch.randint(low, high, (n,), generator=gen)
    for i in range(n):
        x[i, 0, starts[i]:starts[i] + 5] += faults[y[i]]
    return x, y

X_fit, y_fit = windows(3000, 0, 28)
X_in, y_in = windows(1000, 0, 28)
X_out, y_out = windows(1000, 32, 60)
print(X_fit.shape, y_fit.bincount().tolist())

def run(name, model):
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    for epoch in range(30):
        loss = nn.functional.cross_entropy(model(X_fit), y_fit)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    model.eval()
    with torch.no_grad():
        seen = (model(X_in).argmax(dim=1) == y_in).float().mean().item()
        moved = (model(X_out).argmax(dim=1) == y_out).float().mean().item()
    print(name, f"{seen:.4f}", f"{moved:.4f}", sum(p.numel() for p in model.parameters()))

torch.manual_seed(42)
run("linear", nn.Sequential(nn.Flatten(), nn.Linear(64, 3)))
torch.manual_seed(42)
run("conv flatten", nn.Sequential(nn.Conv1d(1, 8, 5), nn.ReLU(), nn.Flatten(), nn.Linear(8 * 60, 3)))
torch.manual_seed(42)
run("conv max", nn.Sequential(nn.Conv1d(1, 8, 5), nn.ReLU(), nn.AdaptiveMaxPool1d(1), nn.Flatten(), nn.Linear(8, 3)))
torch.manual_seed(42)
run("conv mean", nn.Sequential(nn.Conv1d(1, 8, 5), nn.ReLU(), nn.AdaptiveAvgPool1d(1), nn.Flatten(), nn.Linear(8, 3)))

torch.Size([3000, 1, 64]) [954, 1016, 1030]
linear 0.7470 0.3510 195
conv flatten 0.9460 0.3220 1491
conv max 0.9940 0.9980 75
conv mean 0.5840 0.5850 75

import numpy as np
import pandas as pd
from sklearn.ensemble import HistGradientBoostingRegressor
from sklearn.metrics import mean_absolute_error, r2_score
from prep_net import fit_rows, val_rows, X_fit, X_val

ratings = sorted(fit_rows["energy_rating"].dropna().unique())
print(ratings, fit_rows[["year_built", "energy_rating"]].isna().sum().tolist())

def wider(frame, X):
    year = frame["year_built"].to_numpy(dtype=float)
    listed = pd.to_datetime(frame["listing_date"]).dt.year.to_numpy(dtype=float)
    rating = frame["energy_rating"].map({r: i for i, r in enumerate(ratings)}).to_numpy(dtype=float)
    return np.column_stack([X.numpy(), year, listed, rating])

runs = [
    ("defaults, 18", {}, X_fit.numpy(), X_val.numpy()),
    ("depth 2, 18", {"max_depth": 2, "max_iter": 500}, X_fit.numpy(), X_val.numpy()),
    ("depth 2, 21", {"max_depth": 2, "max_iter": 500}, wider(fit_rows, X_fit), wider(val_rows, X_val)),
]
for name, settings, A, B in runs:
    booster = HistGradientBoostingRegressor(random_state=42, **settings).fit(A, fit_rows["price"])
    pred = booster.predict(B)
    print(name, A.shape[1], round(mean_absolute_error(val_rows["price"], pred)), round(r2_score(val_rows["price"], pred), 3))

['A', 'B', 'C', 'D', 'E', 'F', 'G'] [89, 155]
defaults, 18 18 50669 0.819
depth 2, 18 18 48578 0.839
depth 2, 21 21 46427 0.845

Theory Assessment

Q1: Equivariant Is Not Invariant

A teammate says a CNN is translation invariant by construction, so testing it on moved images is wasted time. Using the first subtitle's run and Lesson 5.1.3's numbers, say what the convolution guarantees, what it does not, and what part of the network decides the difference.

Hint: Compare what torch.equal checked with what the means checked. Then think about what Lesson 5.1.3's head did with maps that had moved by a column.

Q2: A Second Attention Layer

A text model averages its tokens' vectors after one attention layer with no positions, and it gives "from my card to my account" and "from my account to my card" the same answer. A teammate proposes a second attention layer on top of the first. Will the two queries get different answers, and what change would let them?

Hint: Ask what the first layer's outputs are for the reordered query, relative to its outputs for the original. Then ask what a second layer of the same kind does with a reordered input.

Q3: A Contract of 3,000 Tokens

The bank wants BERT-Mini to sort whole loan contracts, about 3,000 tokens each, into five kinds. Say what happens when one is handed to the model as it is, why raising max_position_embeddings to 4,096 is not a fix, and what the attention inside the model would cost at that length against a convolution over the same tokens.

Hint: The position table's rows were trained. The cost subtitle measured how attention's weights and a convolution's output grow per doubling of the length.

Q4: "Pretraining Does Not Help on Banking77"

After the fifth subtitle's run, a teammate writes in the team's notes: "BERT-Mini's pretrained weights scored 0.5895 against the bag's 0.6555 on ten labels an intent, so pretraining does not help on our intents, and we train from scratch." Say what the run actually showed, what Lesson 5.4.2's fine-tune on the same 770 queries changes in that sentence, and what the team should check before it trusts the three numbers as a ranking.

Hint: The pretrained weights were used two ways on these 770 queries. Only one pair of the three runs shares its whole recipe.

Assignment

Assignment: Which Event Came First

The pump's maintenance team now logs two events per window of 64 readings, a knock, five readings rising to 2.0 and back, and its mirror, a sag to -2.0, and the order matters: a knock then a sag means a worn bearing, and a sag then a knock a failing seal. The first event starts anywhere from reading 0 to 24, and the second 10 to 30 readings after the first, so the two never touch.

In event_order.py, build 3,000 fit windows and 1,000 validation windows from torch.Generator().manual_seed(42): noise of 0.3 * torch.randn, then the label, the first start and the gap, drawn in that order. Train Lab Exercise 1's winner, the convolution of 8 filters with a global max, now with 2 outputs, and say why it fails. Then build a model that can read the order: an nn.Conv1d(1, 16, 5, stride=4, padding=2) front end with a ReLU, which turns the 64 readings into 16 positions of 16 features, a learned position table of 16 rows added to them, one transformer layer, a mean over the 16 positions and a linear layer of 2 outputs. Train it twice, with the position table and without it, and print each model's validation accuracy, parameter count and seconds. Train each model for 10 epochs of Adam at 0.003 in mini-batches of 64 from a loader shuffled by a generator seeded 42, seeding 42 before each model.

The transformer layer comes from a container this lesson did not use. nn.TransformerEncoder(layer, num_layers) stacks num_layers copies of one nn.TransformerEncoderLayer and runs them in order, and here nn.TransformerEncoderLayer(16, 2, 32, dropout=0.0, batch_first=True), 16 features, 2 heads and a feed-forward network of 32 units, goes in with num_layers=1. The front end's stride also answers the cost subtitle: say how many weight entries one window's attention holds at 16 positions against 64.

Hints:

• The front end's filters are five readings wide and the events at least ten apart, so no single filter ever sees both events. Ask what a global max keeps of where each filter fired.
• A convolution returns (N, C, L) and the transformer layer, with batch_first=True, takes (N, L, C), so the front end's output is transposed before the positions are added.

import time
import torch
from torch import nn, optim
from torch.utils.data import DataLoader, TensorDataset

gen = torch.Generator().manual_seed(42)
up = torch.tensor([0.0, 1.0, 2.0, 1.0, 0.0])

def windows(n):
    x = 0.3 * torch.randn(n, 1, 64, generator=gen)
    y = torch.randint(0, 2, (n,), generator=gen)
    first = torch.randint(0, 25, (n,), generator=gen)
    gap = torch.randint(10, 31, (n,), generator=gen)
    for i in range(n):
        a, b = (up, -up) if y[i] == 0 else (-up, up)
        x[i, 0, first[i]:first[i] + 5] += a
        x[i, 0, first[i] + gap[i]:first[i] + gap[i] + 5] += b
    return x, y

X_fit, y_fit = windows(3000)
X_val, y_val = windows(1000)

class Ordered(nn.Module):
    def __init__(self, positions=True):
        super().__init__()
        self.front = nn.Conv1d(1, 16, 5, stride=4, padding=2)
        self.positions = nn.Embedding(16, 16) if positions else None
        layer = nn.TransformerEncoderLayer(16, 2, 32, dropout=0.0, batch_first=True)
        self.encoder = nn.TransformerEncoder(layer, num_layers=1)
        self.head = nn.Linear(16, 2)

    def forward(self, x):
        h = torch.relu(self.front(x)).transpose(1, 2)
        if self.positions is not None:
            h = h + self.positions(torch.arange(h.shape[1]))
        return self.head(self.encoder(h).mean(dim=1))

def fit(name, model):
    order = torch.Generator().manual_seed(42)
    loader = DataLoader(TensorDataset(X_fit, y_fit), batch_size=64, shuffle=True, generator=order)
    optimizer = optim.Adam(model.parameters(), lr=0.003)
    start = time.perf_counter()
    for epoch in range(10):
        for x, y in loader:
            loss = nn.functional.cross_entropy(model(x), y)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    model.eval()
    with torch.no_grad():
        acc = (model(X_val).argmax(dim=1) == y_val).float().mean().item()
    print(name, f"{acc:.4f}", sum(p.numel() for p in model.parameters()), f"{time.perf_counter() - start:.1f}")

torch.manual_seed(42)
fit("conv max", nn.Sequential(nn.Conv1d(1, 8, 5), nn.ReLU(), nn.AdaptiveMaxPool1d(1), nn.Flatten(), nn.Linear(8, 2)))
torch.manual_seed(42)
fit("no positions", Ordered(False))
torch.manual_seed(42)
fit("positions", Ordered(True))
print(16 * 16 * 2, 64 * 64 * 2)

conv max 0.5180 66 0.8
no positions 0.4990 2354 1.9
positions 0.9900 2610 2.0
512 8192

Additional Resources

Optional, and nothing later depends on either. Worth opening for the three transformer families laid side by side, or for the container the Assignment used read from its own page.

• Book chapter - Dive into Deep Learning, "Large-Scale Pretraining with Transformers" by Zhang, Lipton, Li and Smola, under CC BY-SA 4.0. Encoder-only models with BERT's pretraining and fine-tuning, encoder-decoders with T5, decoder-only models with GPT, GPT-2 and GPT-3, and how they scale.
• Official docs - PyTorch 2.14, torch.nn.TransformerEncoder. The stack of num_layers encoder layers, its norm, enable_nested_tensor and mask_check arguments, the masks its forward pass takes, and the warning that every layer starts with the same parameters.

Summary

The module's architectures turned out to be one idea repeated: each is a decision about what to treat as the same. The convolution treats a patch in one place and the same patch in another as the same pattern, and Section 5.1 found that a network built from it still has to be told, by its head or by its data, that a moved garment is the same garment. The bag treats a query as its words, attention without positions as a set of words that read each other, and positions put order back in. None of those decisions is right or wrong in general, and the fault windows showed one assumption carrying a problem and failing the next: the global max that ignored where a fault sat was exactly what lost the order of two events.

Cost was the other half of every choice, a convolution's work growing with its input and attention's with its input squared. Pretraining was neither a gift nor a trap: ImageNet's features separated land uses frozen, and BERT-Mini's separated intents only once fine-tuning moved them, so the same strategy that won on EuroSAT lost on Banking77 until the weights were trained.

And the map has edges. On tabular rows the first thing to try is not a network at all, and when a network did win there it won by a margin a team has to price. Past the decoder and the encoder-decoder lies application, a separate product's work. What stays with every model in this bootcamp, whatever its shape, is the question Module 6 opens: a validation score printed by a script is where a model's life starts, not where it ends.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

conv = nn.Conv1d(1, 8, 5)     # local, shared
head = nn.Sequential(
    nn.ReLU(),
    nn.AdaptiveMaxPool1d(1),  # invariant: anywhere
    nn.Flatten(),
    nn.Linear(8, 3),
)
x = embed(ids) + positions(torch.arange(MAX_LEN))
few = train.groupby("category").sample(n=10, random_state=42)
rows = torch.tensor(train.index.get_indexer(few.index))
head_input = cached["F"][rows]
cut = tokenizer(text, truncation=True, max_length=512)

layer = nn.TransformerEncoderLayer(16, 2, 32, dropout=0.0, batch_first=True)
encoder = nn.TransformerEncoder(layer, num_layers=1)
h = torch.relu(front(x)).transpose(1, 2)  # (N, L, C)
out = encoder(h + positions(torch.arange(16))).mean(dim=1)