Text as Tensors - Tokens and Embeddings

Context

A tokenizer cuts a sentence into pieces called tokens, a vocabulary gives each token a number, and an embedding table turns each number into a vector of learned weights the network trains like any other layer. Text arrives as strings of every length, and those three steps make it a fixed-shape tensor of integers a network can read.

Routing a customer's message to the team that handles it, tagging support tickets by topic, flagging a fraud report among thousands of ordinary ones and searching a help center all start from these steps. Every pretrained text model ships its own tokenizer and its own table, and Section 5.4 loads one.

Section 5.2 ended on satellite tiles, whose pixels were numbers before any code touched them. Text becomes numbers only through choices, and the wrong ones fail without an error: pads averaged into a query make its answer depend on how much filler follows it, a word outside the vocabulary becomes a shared row that training never touched, and a model that adds words up gives "from my card to my account" and "from my account to my card" the same answer.

Goal

• Read Banking77 through a cached frame, carve 2,000 validation queries from its 10,003 training queries, and build a vocabulary of 2,202 entries from the 8,003 that remain.
• Tokenize with \w+|[^\w\s], encode every query to 48 ids with <pad> and <unk>, and count the 99 training queries truncation cuts.
• Prove that nn.Embedding returns a one-hot row times its weight matrix, and that padding_idx=0 keeps the pad row at zero with no gradient.
• Average only a query's real tokens through a padding mask, train a bag-of-embeddings classifier to 0.8760 validation accuracy, and show that averaging the pads in changes 40 answers once queries are served without padding.
• Read the trained table through cosine similarity, and trace an unknown word and a reordered query to the answers they produce.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the idea of a word as a vector of numbers first, or the class this lesson trains read from its own page.

• YouTube - "Word Embedding and Word2Vec, Clearly Explained!!!" by StatQuest with Josh Starmer. Why words become numbers before a network can use them, and how word2vec trains a small network on the words around each word so that words used alike end up with similar numbers.
• Official docs - PyTorch 2.14, torch.nn.Embedding. The class as "a simple lookup table", its weight's shape and initialization, and padding_idx, whose row starts at zeros and is not updated during training.

Table of Contents

• Banking77 as a Frame
• Tokens and the Vocabulary
• Encoding, Truncation and Padding
• The Embedding Table
• Averaging Only the Real Tokens
• The Bag-of-Embeddings Classifier
• What the Embeddings Learned
• What the Bag Cannot See

Theory

Every block runs in ~/dev/bootcamp, beside image_loop.py, Lesson 4.5.2's. The data comes through pandas, Module 2's library. The first run of frame downloads Banking77's two CSV files from PolyAI's GitHub, about a megabyte together, into data/banking77, and every later run reads them from there. The third subtitle saves text.py, which every script after it imports, as do the later lessons of Section 5.3. Every training run is seeded, and its figures printed the same on one thread and on eight on torch 2.14.1 on a processor. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Banking77 as a Frame

What. Banking77 is 13,083 short customer queries to a bank, each labeled with one of 77 intents (the request a message makes, such as card_arrival or lost_or_stolen_phone), published by PolyAI under CC BY 4.0 as two CSV files of 10,003 training and 3,080 test queries. frame(name) returns one of them as a DataFrame, downloading it on the first call and reading the saved copy after that. Why. A classifier of text needs text with labels, and a table with a text column and a category column is the form Module 2 already reads, filters and counts. Caching the files makes every run after the first independent of GitHub answering. When. Whenever a dataset lives at a URL: read it once, keep the file, and read the file.

import re
from collections import Counter
from pathlib import Path
import pandas as pd

URL = "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/master/banking_data/"
FOLDER = Path("data/banking77")

def frame(name):
    path = FOLDER / f"{name}.csv"
    if not path.exists():
        FOLDER.mkdir(parents=True, exist_ok=True)
        pd.read_csv(URL + f"{name}.csv").to_csv(path, index=False)
    return pd.read_csv(path)

pool, test = frame("train"), frame("test")
print(pool.shape, test.shape)
print(pool.iloc[1].tolist())
sizes = pool["category"].value_counts()
print(len(sizes), sizes.iloc[0], sizes.iloc[-1])

(10003, 2) (3080, 2)
["What can I do if my card still hasn't arrived after 2 weeks?", 'card_arrival']
77 187 35

Path is the standard library's object for a file path, joined to a name with /, and exists() asks whether the file is there. mkdir(parents=True, exist_ok=True) creates the folder and any missing parent, and does nothing when the folder exists. pd.read_csv takes a URL wherever it takes a path, and Lesson 2.4.3's to_csv with index=False writes the copy that every later call reads. re and Counter serve the second subtitle. The 10,003 training queries are named pool, since the training and the validation queries both come out of it, and the 77 intents are not the same size, from 187 queries down to 35. The validation queries, carved from the pool:

val = pool.sample(n=2_000, random_state=42)
train = pool.drop(val.index)
print(len(train), len(val), len(set(train.index) & set(val.index)))
print(train["category"].nunique(), val["category"].nunique())
intents = sorted(pool["category"].unique())
print(intents[:3], intents.index("card_arrival"))

8003 2000 0
77 77
['Refund_not_showing_up', 'activate_my_card', 'age_limit'] 12

sample(n=2_000, random_state=42) is Lesson 3.1.2's draw, the same 2,000 rows on every run, and drop given row labels returns the table without those rows, so the 8,003 training queries share no row label with the 2,000 validation ones, the 0. All 77 intents appear on both sides. intents is the sorted list of the 77 names, and an intent's class number is its position in it, 12 for card_arrival. Refund_not_showing_up sorts first because Python compares strings character by character by code, and every capital letter's code is below every lowercase letter's. The 3,080 test queries stay untouched by every script in this lesson.

DON'T:

• DON'T build intents from a sample of the rows. A sorted list of the names present gives every intent after a missing one a class number one lower, and a model trained on one numbering and scored on another is wrong on every one of those intents without an error.
• DON'T call pd.read_csv(URL) in every script. frame downloads once, and a script that reads the URL each time runs only while GitHub answers.

Tokens and the Vocabulary

What. A tokenizer cuts text into tokens. re.findall(pattern, text) returns, left to right, every piece of text that matches a regular expression, Lesson 2.5.3's small pattern language. In \w+|[^\w\s], \w+ matches a run of word characters (letters, digits and the underscore), | means or, and [^\w\s] matches one character that is neither a word character nor whitespace, a punctuation mark or a symbol. The text is lowercased first. A vocabulary is a dict from each token to an integer id, built from the training queries' token counts, most frequent first, after two reserved entries: <pad> at 0, the filler that makes queries equally long, and <unk> at 1, the stand-in for any token the vocabulary does not hold. Why. A network reads numbers, and an id is the number of a row in the table the fourth subtitle builds. Lowercasing makes Card and card one token, and a punctuation mark of its own keeps card? and card. from becoming two more words. When. Before any text reaches a model, and once, from the training text alone, for every split that model will ever read.

def tokenize(text):
    return re.findall(r"\w+|[^\w\s]", text.lower())

print(tokenize(pool["text"].iloc[1]))
print(tokenize("Top-up of £20 failed!! Can't I use Apple Pay?"))
print("What can I do?".split())

['what', 'can', 'i', 'do', 'if', 'my', 'card', 'still', 'hasn', "'", 't', 'arrived', 'after', '2', 'weeks', '?']
['top', '-', 'up', 'of', '£', '20', 'failed', '!', '!', 'can', "'", 't', 'i', 'use', 'apple', 'pay', '?']
['What', 'can', 'I', 'do?']

The apostrophe is not a word character, so hasn't becomes three tokens, hasn, ' and t, and can't becomes can, ' and t: the tokenizer knows nothing about English, only about character classes. Every punctuation mark is a token of its own, !! is two, and £ is one, since it is neither a word character nor whitespace. str.split() cuts at whitespace alone and lowercases nothing, so do? keeps its question mark and is a different word from do. The vocabulary, from the training queries:

counts = Counter(t for text in train["text"] for t in tokenize(text))
print(len(counts), sum(counts.values()))
print(counts.most_common(8))
vocab = {"<pad>": 0, "<unk>": 1}
for token, n in counts.most_common():
    vocab[token] = len(vocab)
print(len(vocab), vocab["i"], vocab["card"], vocab["hasn"], vocab["t"])
print(sum(1 for n in counts.values() if n == 1), list(vocab)[-3:])

2200 111243
[('i', 7049), ('?', 5176), ('my', 4579), ('.', 3514), ('to', 3252), ('a', 2855), ('the', 2810), ('card', 2140)]
2202 2 9 130 16
810 ['union', 'greece', 'represented']

Counter.most_common(), Lesson 5.2.1's, lists every token by its count, and each new entry's id is the vocabulary's length when it is added, so the 2,200 distinct tokens among the 111,243 take ids 2 to 2,201, i first. 810 tokens appear exactly once, and most_common orders tokens of equal count by first appearance, so the last ids belong to words seen once, late. What the validation queries meet, against a vocabulary built from every query in the pool:

val_tokens = [t for text in val["text"] for t in tokenize(text)]
print(len(val_tokens), sum(t not in vocab for t in val_tokens))
print(sum(any(t not in vocab for t in tokenize(text)) for text in val["text"]))
pool_counts = Counter(t for text in pool["text"] for t in tokenize(text))
print(len(pool_counts) + 2, sum(t not in pool_counts for t in val_tokens))
test_tokens = [t for text in test["text"] for t in tokenize(text)]
print(len(test_tokens), sum(t not in vocab for t in test_tokens), sum(t not in pool_counts for t in test_tokens))

27532 171
145
2363 0
39156 253 235

Of the validation queries' 27,532 tokens, 171 are not in the training vocabulary, spread over 145 of the 2,000 queries. Built from the whole pool instead, the vocabulary grows to 2,363 entries and validation meets no unknown token at all, while the test queries still hold 235. The 161 entries the pool adds are words only validation queries hold, so their rows never meet a training query and never train, and what validation loses is the count: it stops showing the word nobody stored, which test queries and every new query bring. That is Lesson 3.1.2's rule in a new place: a held-out row must be as new to the model as the rows it will meet in use, and the vocabulary is a decision about the model made before the fit.

DON'T:

• DON'T build the vocabulary from every query you hold. Validation then met 0 unknown tokens where it meets 171, and the test queries still held 235.
• DON'T split text on spaces and call it tokenized. do? stayed glued to its question mark, a separate word from do, and Lab Exercise 1 counts what that does to the vocabulary.

Encoding, Truncation and Padding

What. encode(text) turns a query into exactly MAX_LEN ids: each token's id from vocab.get(t, 1), Lesson 1.4.3's .get with a default, so a token the vocabulary lacks becomes <unk>, then the list cut at 48 ids, which is truncation, then filled with 0s up to 48, which is padding. Every query becomes one row of 48 integers. Why. A batch is one tensor, and every row of a tensor has the same length, while the queries run from a few tokens to 96. Truncation caps the width, and padding fills each shorter row with an id that is reserved for filler and never names a word. When. For every query a model reads, in training and in use, with the same MAX_LEN and the same vocabulary.

MAX_LEN = 48

def encode(text):
    ids = [vocab.get(t, 1) for t in tokenize(text)][:MAX_LEN]
    return ids + [0] * (MAX_LEN - len(ids))

print(encode("My card hasn't materialised")[:8], len(encode("My card hasn't materialised")))
lengths = pd.Series([len(tokenize(text)) for text in train["text"]])
print(lengths.max(), lengths.median(), (lengths > MAX_LEN).sum(), (lengths <= 24).mean().round(4))
longest = train["text"].iloc[lengths.idxmax()]
print(len(tokenize(longest)), tokenize(longest)[MAX_LEN:MAX_LEN + 8])

[4, 9, 130, 10, 16, 1, 0, 0] 48
96 11.0 99 0.8958
96 ['years', 'and', 'i', 'haven', "'", 't', 'had', 'any']

materialised is not in the vocabulary and becomes 1, and 42 zeros fill the row to 48. The training queries' median length is 11 tokens and the longest is 96, and 99 of the 8,003 run past 48, so truncation cuts the tail off one query in about eighty. idxmax() returns the label of a Series' largest value, here the longest query's position, and that query loses its second half, from years and i haven't had any on. 48 keeps every token of the other 7,904 queries in rows half as wide as the longest query would need, and 89.58 percent of the queries fit in 24. The pieces go into one file. Save it as text.py in ~/dev/bootcamp:

import re
from collections import Counter
from pathlib import Path
import pandas as pd
import torch
from torch.utils.data import DataLoader, TensorDataset

URL = "https://raw.githubusercontent.com/PolyAI-LDN/task-specific-datasets/master/banking_data/"
FOLDER = Path("data/banking77")
MAX_LEN = 48

def frame(name):
    path = FOLDER / f"{name}.csv"
    if not path.exists():
        FOLDER.mkdir(parents=True, exist_ok=True)
        pd.read_csv(URL + f"{name}.csv").to_csv(path, index=False)
    return pd.read_csv(path)

def tokenize(text):
    return re.findall(r"\w+|[^\w\s]", text.lower())

pool, test = frame("train"), frame("test")
val = pool.sample(n=2_000, random_state=42)
train = pool.drop(val.index)
intents = sorted(pool["category"].unique())
counts = Counter(t for text in train["text"] for t in tokenize(text))
vocab = {"<pad>": 0, "<unk>": 1}
for token, n in counts.most_common():
    vocab[token] = len(vocab)

def encode(text):
    ids = [vocab.get(t, 1) for t in tokenize(text)][:MAX_LEN]
    return ids + [0] * (MAX_LEN - len(ids))

def tensors(df):
    x = torch.tensor([encode(text) for text in df["text"]])
    y = torch.tensor([intents.index(c) for c in df["category"]])
    return x, y

train_set = TensorDataset(*tensors(train))
val_set = TensorDataset(*tensors(val))
test_set = TensorDataset(*tensors(test))
order = torch.Generator().manual_seed(42)
train_loader = DataLoader(train_set, batch_size=32, shuffle=True, generator=order)
val_loader = DataLoader(val_set, batch_size=250)
test_loader = DataLoader(test_set, batch_size=250)

The file is the blocks above with one addition at the end: tensors(df) encodes every query of a frame into one tensor of ids and turns every intent into its class number, and three TensorDatasets, Lesson 4.3.3's, hold the three splits as rows a loader can index. train_loader shuffles batches of 32 with its own generator, order, seeded 42, and the validation and test loaders read 250 queries at a time in a fixed order. What the file builds:

from text import train, train_set, val_set, test_set, train_loader, val_loader, test_loader

x, y = train_set[0]
print(x.shape, x.dtype, y, train["category"].iloc[0])
print(x[:16])
print(train_set.tensors[0].shape, val_set.tensors[0].shape, test_set.tensors[0].shape)
print(len(train_loader), len(val_loader), len(test_loader))
xb, yb = next(iter(train_loader))
print(xb.shape, yb.shape, (xb != 0).sum(dim=1)[:8])

torch.Size([48]) torch.int64 tensor(12) card_arrival
tensor([ 18,  13,   2,  14,  63,   4,   9,  67, 130,  10,  16, 274, 167, 480,
        333,   3])
torch.Size([8003, 48]) torch.Size([2000, 48]) torch.Size([3080, 48])
251 8 13
torch.Size([32, 48]) torch.Size([32]) tensor([20, 12,  5,  7,  5,  9, 10, 13])

The first training query is the pool's second row, the hasn't arrived after 2 weeks query, since the first went to validation: 16 ids, what can i do if my card still hasn ' t arrived after 2 weeks ?, then 32 zeros, and its label is 12, card_arrival. The three splits are [n, 48] tensors of int64, the dtype torch.tensor gives a list of Python integers. 8,003 queries in batches of 32 make 251 batches, the last of 3, and (xb != 0).sum(dim=1) counts each row's real tokens, a different number per query in one [32, 48] batch.

DON'T:

• DON'T pad with an id that names a word. 0 belongs to <pad> alone, and the fifth subtitle tells real tokens from filler by ids != 0 and nothing else.
• DON'T truncate without counting what is cut. 99 training queries lose their tails at 48, and the longest loses 48 of its 96 tokens.

The Embedding Table

What. nn.Embedding(len(vocab), 64, padding_idx=0) holds a weight matrix of shape [2202, 64], one row of 64 numbers per id, drawn from a standard normal when the layer is created and trained like any weight. Called on a tensor of ids it returns each id's row, so [2, 48] ids become a [2, 48, 64] tensor of vectors. padding_idx=0 starts row 0 at zeros and gives it no gradient, so the pad's vector stays zero through all of training. Why. An id is a name, not a quantity: card at 9 is not more of anything than my at 4, the trap Lesson 3.2.2's category codes set, and one-hot columns are that lesson's fix. A one-hot vector times a weight matrix picks one row of the matrix, so an embedding is a linear layer on one-hot inputs, computed by looking the row up instead of multiplying 2,202 numbers that are almost all zero. When. As the first layer of every network that reads tokens.

import torch
from torch import nn
from text import vocab, train_set

torch.manual_seed(42)
embed = nn.Embedding(len(vocab), 64, padding_idx=0)
print(embed.weight.shape, embed.weight.requires_grad, sum(p.numel() for p in embed.parameters()))
ids = train_set.tensors[0][:2]
vectors = embed(ids)
print(ids.shape, vectors.shape)
print(ids[0, :3], torch.equal(vectors[0, 0], embed.weight[ids[0, 0]]))

torch.Size([2202, 64]) True 140928
torch.Size([2, 48]) torch.Size([2, 48, 64])
tensor([18, 13,  2]) True

The vector at the first query's first position is exactly row 18 of the weight, the row of what. The same rows, through one-hot vectors:

onehot = nn.functional.one_hot(ids, num_classes=len(vocab)).float()
print(onehot.shape, onehot[0, 0].sum().item(), onehot[0, 0].argmax().item())
print(torch.equal(onehot @ embed.weight, vectors))

torch.Size([2, 48, 2202]) 1.0 18
True

nn.functional.one_hot(ids, num_classes=n) returns, for every id, a vector of n zeros with a 1 at the id's position, and .float() turns its integers into floats, the dtype a product with the weight needs. Each position is now 2,202 numbers that sum to 1, with the 1 at 18 for what, and the [2, 48, 2202] one-hot tensor times the [2202, 64] weight is bit for bit the lookup's output: every output number is one weight times 1 plus 2,201 weights times 0. The multiplication does 2,202 multiply-adds for every number the lookup copies. What padding_idx does, and where a gradient goes:

print(embed.weight[0].abs().max().item(), vectors[0, 47].abs().max().item())
vectors.sum().backward()
grad = embed.weight.grad
print(grad[0].abs().max().item(), grad[ids[0, 0]][:3], (ids == ids[0, 0]).sum().item())
print(grad[2][:3], (ids == 2).sum().item())
print((grad.abs().sum(dim=1) > 0).sum().item(), len(set(ids[ids != 0].tolist())))

0.0 0.0
0.0 tensor([1., 1., 1.]) 1
tensor([2., 2., 2.]) 2
26 26

Row 0 is all zeros, so every padded position's vector is zero. With the sum of every vector as the loss, the pad row's gradient is zero although 66 of the 96 positions hold a pad, which is padding_idx at work. The row of what, used once, receives a gradient of 1 in every entry, and the row of i, used twice, a gradient of 2: a row's gradient adds up over every position that looked it up. Only 26 of the 2,202 rows received any gradient, the 26 distinct real tokens of the two queries. A row learns from the queries that hold its token and from nothing else.

DON'T:

• DON'T build the table without padding_idx=0. Row 0 is then a random vector like any other, trained by every pad of every query, and Lab Exercise 2 measures what that does.
• DON'T compute embeddings as one-hot vectors times the weight. The result is the same, through a [2, 48, 2202] tensor for two queries and 2,202 multiply-adds per number the lookup copies.

Averaging Only the Real Tokens

What. A query's 48 vectors become one by averaging the vectors of its real tokens. mask = ids != 0 is True at every real token and False at every pad, mask.sum(dim=1, keepdim=True) counts each query's real tokens, Lesson 4.1.2's keepdim, and since the pad row is zero, the sum over all 48 positions is already the sum over the real ones, so the sum divided by the count is the mean of the real tokens alone. Why. A classifier's last layer takes one vector per query. The mean has the same scale for a query of 5 tokens and one of 40, and the pads are filler whose count says nothing about the query. Averaging all 48 positions divides every query by 48 instead of by its length, which shrinks each vector by its length over 48 and ties the result to how much padding there is. When. Every time a padded sequence of vectors becomes one vector, here and in every model that pools its tokens.

vectors = embed(ids).detach()
mask = ids != 0
real = mask.sum(dim=1, keepdim=True)
print(mask[0, :18].int(), real.flatten())
mean = vectors.sum(dim=1) / real
n = real[0].item()
print(torch.allclose(mean[0], vectors[0, :n].mean(dim=0)))
pads_in = vectors.mean(dim=1)
print(torch.allclose(pads_in, mean * real / 48))
print(mean.norm(dim=1), pads_in.norm(dim=1))
cut = ids[:, :24]
print(torch.allclose(embed(cut).detach().sum(dim=1) / real, mean), torch.allclose(embed(cut).detach().mean(dim=1), pads_in))

tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0],
       dtype=torch.int32) tensor([16, 14])
True
True
tensor([1.9721, 2.2177]) tensor([0.6574, 0.6468])
True False

detach() takes the vectors out of the gradient's record, since nothing here trains. The mask is 1 at the first query's 16 real tokens and 0 from there, and the two queries hold 16 and 14. The masked mean equals the plain mean of the first 16 vectors, and the mean over all 48 positions equals the masked mean times 16 over 48 and 14 over 48: the pads-in vectors are about a third the length of the masked ones, 0.6574 against 1.9721. Cut to 24 columns, the same two queries with 24 pads fewer give the same masked mean and a different pads-in mean, so averaging the pads in makes a query's vector depend on the width it was padded to, a number that has nothing to do with the query.

DON'T:

• DON'T average over every position of a padded batch. The vector of the same query changed when its row was cut from 48 columns to 24, and only the masked mean held.
• DON'T count real tokens from anything but the ids. The mask comes from ids != 0, the one test the encoding guarantees, and a count taken from the text's own length disagrees with it on every truncated query.

The Bag-of-Embeddings Classifier

What. A bag-of-embeddings classifier looks every token up, averages the real tokens' vectors, and sends the mean through one nn.Linear(64, 77) that returns a logit per intent. It is called a bag because the mean keeps which words a query holds and how often, and nothing about where. Bag is an nn.Module subclass, Lesson 4.2.2's, and image_loop.py's train_epoch and accuracy train and score it unchanged, since both take any model that returns a row of logits per input. Why. It is the smallest network that reads text end to end, from ids to an intent, and its score is the baseline any model that reads word order has to beat. When. As the first model on any text classification task, before anything larger is trained.

import torch
from torch import nn, optim
from text import vocab, intents, order, train_loader, val_loader, val_set, encode
from image_loop import train_epoch, accuracy

class Bag(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.embed = nn.Embedding(len(vocab), dim, padding_idx=0)
        self.head = nn.Linear(dim, len(intents))

    def forward(self, ids):
        real = (ids != 0).sum(dim=1, keepdim=True)
        return self.head(self.embed(ids).sum(dim=1) / real)

def fit(model_class):
    order.manual_seed(42)
    torch.manual_seed(42)
    model = model_class(64)
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    for epoch in range(1, 11):
        loss = train_epoch(model, optimizer, train_loader)
        if epoch in (1, 5, 10):
            print(epoch, f"{loss:.4f}", f"{accuracy(model, val_loader):.4f}")
    return model

bag = fit(Bag)
print(sum(p.numel() for p in bag.parameters()), sum(p.numel() for p in bag.embed.parameters()))

1 2.3178 0.7565
5 0.1572 0.8665
10 0.0353 0.8760
145933 140928

fit reseeds order, the training loader's generator from text.py, before every model, because a generator's stream continues from wherever its last draw left it, and a second model trained in the same script would otherwise see the batches in another order than the first. Then it seeds the weights, builds the model at dimension 64 and trains it for ten epochs with Lesson 4.3.4's Adam at 0.01. The training loss falls to 0.0353, so the bag fits its training queries almost perfectly, while validation gains under a point over the last five epochs and ends at 0.8760. Of the 145,933 parameters, 140,928 are the table and 5,005 the head. The same bag with the pads averaged in, and both served the way a deployed model often meets queries, one at a time with no padding:

class PadsIn(Bag):
    def forward(self, ids):
        return self.head(self.embed(ids).mean(dim=1))

pads_in = fit(PadsIn)
X, Y = val_set.tensors
lengths = (X != 0).sum(dim=1)

def served_alone(model):
    model.eval()
    with torch.no_grad():
        alone = torch.cat([model(x[:n].unsqueeze(0)).argmax(dim=1) for x, n in zip(X, lengths)])
        padded = model(X).argmax(dim=1)
    return f"{(alone == Y).float().mean().item():.4f}", (alone != padded).sum().item()

print(served_alone(bag), served_alone(pads_in))

1 3.1951 0.6320
5 0.3026 0.8655
10 0.0978 0.8820
('0.8760', 0) ('0.8790', 40)

PadsIn inherits everything from Bag and overrides forward alone, Lesson 1.8.3's override, with the mean over all 48 positions. On the validation queries, padded to 48 exactly as the training queries were, it scores 0.8820, above the masked bag's 0.8760: the validation score cannot see the defect, because validation was padded the way training was. served_alone runs each validation query at its own length, its real ids only, made a batch of one by Lesson 4.1.2's unsqueeze(0). The masked bag gives all 2,000 queries the answer it gave them padded. The pads-in bag changes 40 answers and falls to 0.8790, because every query served without pads arrives as a vector 48 over its length times longer than the same query's vector in training. Only 40 change because the head is linear: a vector scaled by a factor scales every logit's weighted part by the same factor, and only the bias, unscaled, can move which intent wins.

DON'T:

• DON'T trust a validation score to catch a padding defect. The pads-in bag scored 0.8820 on queries padded as training was, and changed 40 answers the moment the padding changed.
• DON'T train two models from one shuffled loader without reseeding its generator. The second continues the stream the first left, and its comparison with the first then mixes a change of model with a change of batches.

What the Embeddings Learned

What. After training, each row of the table is a token's vector, and Lesson 2.3.1's cosine similarity between two rows compares their directions. nn.functional.normalize(W, dim=1) divides every row of W by its length, leaving the all-zero pad row at zero instead of dividing by zero, so the unit rows times one unit row give every token's cosine with that token at once, and Lesson 4.5.2's topk returns the largest. Why. The table was trained for one job, telling 77 intents apart, and its geometry shows what that job needed: tokens whose queries point to the same intent were pushed to similar vectors, and nothing else pushed them. When. Whenever a trained table needs checking, before its vectors are trusted for anything but the task they were trained on.

inv = {i: t for t, i in vocab.items()}
unit = nn.functional.normalize(bag.embed.weight.detach(), dim=1)

def nearest(unit, word, k=5):
    sims = unit @ unit[vocab[word]]
    top = sims.topk(k + 1).indices[1:]
    return [(inv[i], round(sims[i].item(), 2)) for i in top.tolist()]

for word in ("atm", "fee", "declined", "stolen", "cancel", "pin"):
    print(word, nearest(unit, word))
torch.manual_seed(42)
start = nn.Embedding(len(vocab), 64, padding_idx=0)
print(nearest(nn.functional.normalize(start.weight.detach(), dim=1), "atm"))

atm [('withdrawal', 0.64), ('withdrawals', 0.63), ('withdraw', 0.62), ('atms', 0.61), ('ive', 0.59)]
fee [('fees', 0.77), ('extra', 0.76), ('lesser', 0.7), ('charged', 0.66), ('free', 0.66)]
declined [('declining', 0.67), ('rejected', 0.67), ('trying', 0.63), ('decline', 0.63), ('unsupported', 0.54)]
stolen [('lost', 0.66), ('guide', 0.61), ('tube', 0.59), ('misplaced', 0.58), ('fraudulently', 0.52)]
cancel [('typo', 0.63), ('cancelable', 0.58), ('tomorrow', 0.57), ('accidently', 0.54), ('arranged', 0.48)]
pin [('started', 0.51), ('travelling', 0.49), ('blocked', 0.49), ('ages', 0.48), ('spain', 0.45)]
[('credited', 0.4), ('date', 0.39), ('ruble', 0.36), ('duplicated', 0.35), ('cancelled', 0.34)]

inv reverses the vocabulary, id to token. The top entry of every ranking is the word itself at a cosine of 1, so [1:] drops it. atm sits beside withdrawal, withdraw and atms, fee beside fees, extra and charged, and declined beside declining, rejected and decline: plural, tense and synonym, learned from nothing but intents. cancel sits beside typo, tomorrow, cancelable and accidently, which are no synonyms of it: 6, 10, 1 and 3 training queries hold those four words, and every one of them is a cancel_transfer query, so the table put them where that intent's words are. The same seeded table before training puts credited, date and ruble nearest atm, at cosines under 0.41, so every grouping above is training's. Near means "points to the same intent", which is often a synonym and sometimes a word that happened to share the queries.

DON'T:

• DON'T read a near neighbor as a synonym. typo and tomorrow sit nearest cancel because every training query that holds them is a cancel_transfer query.
• DON'T take this table to another task. Its directions encode what separates 77 banking intents, and words that never mattered to those intents, ages beside pin, sit wherever they were pushed.

What the Bag Cannot See

What. Two things the bag cannot represent. A token outside the vocabulary becomes <unk>, id 1, and every unknown word in every query shares that one row. And the mean of a query's vectors is the same in whatever order the tokens come, so two queries with the same words in another order are the same input. Why. No training query holds an unknown token, since the vocabulary was built from those very queries, so by the fourth subtitle's rule the <unk> row never receives a gradient and stays the seeded draw it started as. A sum does not depend on the order of what it adds, so the order of words is gone before the head sees anything. When. Every time a model of this kind meets a word written after its vocabulary was built, a typo, or a query whose meaning lives in which word comes before which.

print(torch.equal(start.weight[1], bag.embed.weight[1]), torch.equal(start.weight[2], bag.embed.weight[2]))

def predict(model, query):
    model.eval()
    with torch.no_grad():
        probs = model(torch.tensor([encode(query)])).softmax(dim=1)[0]
    return intents[probs.argmax()], round(probs.max().item(), 3)

for query in ("how do i cancel a transfer", "how do i revoke a transfer", "how do i rescind a transfer", "fiver dosh quid"):
    print([t for t in encode(query) if t], predict(bag, query))
has_unk = (X == 1).any(dim=1)
bag.eval()
with torch.no_grad():
    right = bag(X).argmax(dim=1) == Y
print(has_unk.sum().item(), f"{right[has_unk].float().mean().item():.4f}", f"{right[~has_unk].float().mean().item():.4f}")

True False
[19, 14, 2, 144, 7, 27] ('cancel_transfer', 1.0)
[19, 14, 2, 1, 7, 27] ('transfer_into_account', 0.578)
[19, 14, 2, 1, 7, 27] ('transfer_into_account', 0.578)
[1, 1, 1] ('card_acceptance', 0.62)
145 0.7655 0.8846

start is the fresh table the last subtitle drew from seed 42, the draw fit gave the bag's table before its first epoch, and the trained bag's <unk> row is still bit for bit that draw, while the row of i moved. predict returns the top intent and its probability through Lesson 4.3.2's softmax. With cancel the query is cancel_transfer at 1.0. revoke and rescind are not in the vocabulary, so both queries encode to the same six ids, ending 1, 7, 27, and get the same answer, transfer_into_account at 0.578: the model never saw a difference between them, or between them and any other unknown word. Three unknown words alone are three copies of one untrained row, and the bag answers card_acceptance at 0.62, a confident answer about nothing. The 145 validation queries that hold an unknown token score 0.7655, against 0.8846 for the other 1,855. Lesson 5.4.1's subword tokenizer fixes this failure by cutting a word it has not stored into smaller pieces it has. The order, on two queries that move money in opposite directions:

a = "can i move money from my card to my account"
b = "can i move money from my account to my card"
print(encode(a)[:11])
print(encode(b)[:11])
print(sorted(encode(a)) == sorted(encode(b)), predict(bag, a), predict(bag, b))
with torch.no_grad():
    la, lb = bag(torch.tensor([encode(a)])), bag(torch.tensor([encode(b)]))
print(torch.allclose(la, lb, atol=1e-5))

[13, 2, 652, 25, 44, 4, 9, 6, 4, 22, 0]
[13, 2, 652, 25, 44, 4, 22, 6, 4, 9, 0]
True ('transfer_into_account', 0.403) ('transfer_into_account', 0.403)
True

The two queries hold the same ten ids with card, 9, and account, 22, swapped, so their sorted ids are equal, and the bag gives both transfer_into_account at 0.403, though the first moves money off the card and the second onto it. Their logits agree within 1e-5: the same vectors summed in another order, where float32 rounding is the only difference left. The bag knows which words a query holds and never which came first. Reading each word in the light of the words around it is where Lesson 5.3.2 starts, with attention.

DON'T:

• DON'T read a confident answer on a query of unknown words as evidence. fiver dosh quid is three copies of a row training never touched, and it came back card_acceptance at 0.62.
• DON'T expect more epochs to teach <unk> anything. Its row receives a gradient only from training queries that hold it, and the vocabulary was built so that none does.

Lab Exercises

Lab Exercise 1: Splitting on Spaces

In spaces.py, build a second vocabulary from the training queries with text.lower().split() as the tokenizer, kept as a Counter. Print its size counting the two reserved entries beside the size of text.py's vocabulary, the first eight of its entries that start with card, and how many start with card.

Then, for each tokenizer with its own vocabulary, print the number of tokens in the validation queries, how many of them the vocabulary lacks, how many validation queries hold at least one, and how many training queries run past MAX_LEN. Say in a comment what splitting on spaces costs, and why its truncation count is lower.

Lab Exercise 2: A Pad That Learns

In learned_pad.py, write LearnedPad, the bag with its table built as nn.Embedding(len(vocab), dim), no padding_idx, and the same masked mean, dividing by the count of real tokens. Train it the way the sixth subtitle's fit trains, reseeding order and the weights with 42, at dimension 64 with Adam at 0.01 for ten epochs, keeping a copy of the pad row before the first epoch.

Print the validation accuracy and the pad row's length before and after training. Then serve every validation query alone at its own length, as served_alone did, and print that accuracy and how many answers differ from the padded ones. Say in a comment why answers change although the mean divides by the real count.

from collections import Counter
from text import train, val, vocab, tokenize, MAX_LEN

def split(text):
    return text.lower().split()

spaced = Counter(w for text in train["text"] for w in split(text))
print(len(spaced) + 2, len(vocab))
print([w for w in spaced if w.startswith("card")][:8], sum(1 for w in spaced if w.startswith("card")))
for name, cut, held in (("regex", tokenize, vocab), ("spaces", split, spaced)):
    tokens = [w for text in val["text"] for w in cut(text)]
    queries = sum(any(w not in held for w in cut(text)) for text in val["text"])
    print(name, len(tokens), sum(w not in held for w in tokens), queries, sum(len(cut(text)) > MAX_LEN for text in train["text"]))
# Spaces glue punctuation to words, so card, card? and card. are three
# entries: the vocabulary grows by 1,600 and validation meets twice the
# unknowns. Fewer, longer tokens per query also truncate fewer queries.

3802 2202
['card', 'card,', 'card?', "card's", 'card.', 'cards,', 'cards', 'card!'] 19
regex 27532 171 145 99
spaces 23732 356 292 43

import torch
from torch import nn, optim
from text import vocab, intents, order, train_loader, val_loader, val_set
from image_loop import train_epoch, accuracy

class LearnedPad(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.embed = nn.Embedding(len(vocab), dim)
        self.head = nn.Linear(dim, len(intents))

    def forward(self, ids):
        real = (ids != 0).sum(dim=1, keepdim=True)
        return self.head(self.embed(ids).sum(dim=1) / real)

order.manual_seed(42)
torch.manual_seed(42)
model = LearnedPad(64)
start = model.embed.weight[0].detach().clone()
optimizer = optim.Adam(model.parameters(), lr=0.01)
for epoch in range(10):
    train_epoch(model, optimizer, train_loader)
print(f"{accuracy(model, val_loader):.4f}", round(start.norm().item(), 4), round(model.embed.weight[0].norm().item(), 4))
X, Y = val_set.tensors
lengths = (X != 0).sum(dim=1)
model.eval()
with torch.no_grad():
    alone = torch.cat([model(x[:n].unsqueeze(0)).argmax(dim=1) for x, n in zip(X, lengths)])
    padded = model(X).argmax(dim=1)
print(f"{(alone == Y).float().mean().item():.4f}", (alone != padded).sum().item())
# The sum still runs over all 48 positions, and without padding_idx
# every pad adds the pad row, so a padded query's sum carries 48 minus
# its length copies of it, and the same query served alone carries none.

0.8795 8.3975 0.8621
0.8675 131

Theory Assessment

Q1: A Vocabulary That Saw Validation

The vocabulary built from the 8,003 training queries leaves 171 of the validation queries' tokens unknown, and one built from all 10,003 leaves none, while the test queries still hold 235 unknown tokens. Say what the second vocabulary hides from validation, why the 161 entries it adds teach the model nothing, and which rule of Module 3 it breaks.

Hint: Ask what kind of query the 145 validation queries holding an unknown token stand for, and which training queries could ever look up the rows of words only validation holds.

Q2: A Lookup That Is a Layer

nn.Embedding returned bit for bit what a one-hot tensor times its weight returns, and after vectors.sum().backward() the row of what held a gradient of 1, the row of i a gradient of 2 and the pad row 0. Say why the lookup and the product agree, why the two gradients are 1 and 2, what padding_idx=0 changed, and what the rule behind the 26 rows that received a gradient means for the <unk> row of the trained bag.

Hint: A one-hot row has one 1, and a row of the table enters the loss once for every position that looked it up.

Q3: Neighbors That Are Not Synonyms

In the trained table, withdrawal sits nearest atm and declining nearest declined, while typo and tomorrow sit nearest cancel. Say what one training signal put each of these words where it is, why that signal produces synonyms in some cases and not in others, and what the seeded table's neighbors of atm add to the argument.

Hint: The only labels the table ever saw were the intents, and six training queries hold typo.

Q4: The Better Score That Was Worse

A teammate compares the masked bag, 0.8760 on validation, with the pads-in bag, 0.8820, ships the pads-in one, and serves it one query at a time without padding. Say what the served model does that the validation score never showed, why the validation set could not show it, why the change is 40 answers and not hundreds, and what the change would have been with a table built without padding_idx.

Hint: Write a pads-in vector as the masked mean times a factor, ask what a linear layer does with a factor, and take the last part from Lab Exercise 2.

Assignment

Assignment: Teaching <unk> a Meaning

The trained bag's <unk> row is the seeded draw it started as, so fiver dosh quid, three unknown words, came back card_acceptance at 0.62. Write unk_dropout.py, which trains the bag so that <unk> means something.

One technique this lesson did not use. Word dropout hides a random share of the real tokens in every training batch by replacing their ids with the <unk> id, so the <unk> row receives gradients from real queries whose words it stood in for. torch.rand(x.shape, generator=g) draws one uniform number between 0 and 1 for every id of a batch from a seeded generator of its own, so torch.rand(x.shape, generator=g) < rate is True at about rate of the positions, and Lesson 4.1.2's torch.where swaps those positions' ids.

Train the sixth subtitle's bag with its own loop over train_loader, image_loop.py's loss_fn and the five steps, reseeding order and the weights with 42 as fit does, at word dropout rates of 0.0, 0.1 and 0.2, never hiding a pad. For each rate print the rate, the validation accuracy, the accuracy on the 145 validation queries that hold an unknown token, and the top intent of fiver dosh quid with its probability. Say what word dropout taught the <unk> row and what it did not recover.

Hints:

• With the rate at 0.0 nothing is hidden and the run must reproduce the sixth subtitle's 0.8760 and the card_acceptance at 0.62, which proves the loop is the same training.
• Hide a position only where the batch holds a real token, & (x != 0), or the pads become <unk> and the mean divides by a count that no longer matches its sum.

import torch
from torch import nn, optim
from text import vocab, intents, order, train_loader, val_loader, val_set, encode
from image_loop import loss_fn, accuracy

class Bag(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.embed = nn.Embedding(len(vocab), dim, padding_idx=0)
        self.head = nn.Linear(dim, len(intents))

    def forward(self, ids):
        real = (ids != 0).sum(dim=1, keepdim=True)
        return self.head(self.embed(ids).sum(dim=1) / real)

def fit(rate):
    order.manual_seed(42)
    torch.manual_seed(42)
    model = Bag(64)
    optimizer = optim.Adam(model.parameters(), lr=0.01)
    drops = torch.Generator().manual_seed(42)
    for epoch in range(10):
        for x, y in train_loader:
            hide = (torch.rand(x.shape, generator=drops) < rate) & (x != 0)
            loss = loss_fn(model(torch.where(hide, 1, x)), y)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return model

X, Y = val_set.tensors
has_unk = (X == 1).any(dim=1)
nothing = torch.tensor([encode("fiver dosh quid")])
for rate in (0.0, 0.1, 0.2):
    model = fit(rate)
    score = accuracy(model, val_loader)
    model.eval()
    with torch.no_grad():
        right = model(X).argmax(dim=1) == Y
        probs = model(nothing).softmax(dim=1)[0]
    print(rate, f"{score:.4f}", f"{right[has_unk].float().mean().item():.4f}", intents[probs.argmax()], round(probs.max().item(), 3))

0.0 0.8760 0.7655 card_acceptance 0.62
0.1 0.8830 0.7724 getting_virtual_card 0.078
0.2 0.8780 0.7517 transfer_timing 0.114

Additional Resources

Optional, and nothing later depends on either. Worth opening for where word vectors came from, or for how other tokenizers cut text.

• YouTube - "Stanford CS224N: NLP with Deep Learning, Winter 2021, Lecture 1 - Intro and Word Vectors" by Stanford Online, Christopher Manning's opening lecture. Why words as separate symbols share no similarity, the idea that a word is known by the words around it, and word2vec's vectors trained from that context.
• Hugging Face LLM Course - Chapter 2, "Tokenizers". Word-based, character-based and subword tokenization side by side, the unknown token a word-based vocabulary needs, and how a pretrained tokenizer is loaded, saved, and used to encode and decode text.

Summary

Text had to become numbers, and every step that made it so was a choice the network could not see. The tokenizer chose that hasn't is three tokens, the vocabulary chose which 2,200 tokens exist, the encoding chose 48 positions and what fills the empty ones. The network only ever saw blocks of integers, and every defect in this lesson lived in a choice made before the first layer.

The embedding table looked like a new kind of layer and turned out to be an old one: a linear layer on one-hot inputs, computed by lookup, its rows learning only from the queries that use them. That one rule explained both ends of the table. The pad row, kept at zero, could be averaged around safely, and the <unk> row, never looked up in training, arrived at the end of ten epochs exactly as it was drawn, which is why a query of unknown words got a confident answer about nothing.

The bag was good, 0.8760 from the average of a query's words, and its failures were all in what the average cannot carry. A pad averaged in tied the answer to the width of the row, invisibly to a validation set padded the same way. An unknown word became the same row as every other unknown word. And two queries that move money in opposite directions were one input. The first is fixed by a mask, the second by cutting words into pieces a vocabulary can hold, and the third needs a model that reads each word together with the words around it and where it sits, which the rest of this section builds.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

tokens = re.findall(r"\w+|[^\w\s]", text.lower())
ids = [vocab.get(t, 1) for t in tokens][:MAX_LEN]
ids = ids + [0] * (MAX_LEN - len(ids))  # pad to 48

class Bag(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.embed = nn.Embedding(len(vocab), dim, padding_idx=0)
        self.head = nn.Linear(dim, len(intents))

    def forward(self, ids):
        # the count of real tokens per query
        real = (ids != 0).sum(dim=1, keepdim=True)
        return self.head(self.embed(ids).sum(dim=1) / real)