Hugging Face - Loading a Pretrained Encoder

Context

A pretrained encoder is a transformer someone else trained on far more text than you hold, published with its weights and with the tokenizer that cut its training text into ids. The transformers library downloads both from the Hugging Face Hub by a model id, builds the network, fills it with the trained weights, and turns your strings into exactly the ids those weights were trained on.

A team that routes customer messages, tags support tickets or searches a help center starts from such a model instead of an empty embedding table, because it has already met far more English than the team's labeled queries hold. Loading it, reading what it is built from and checking what its frozen features are worth on the team's own labels come before any decision to train it further.

Lesson 5.3.1's bag scored 0.8760 on Banking77 with a vocabulary of 2,202 entries, every word outside it one untrained row, and Lesson 5.3.3 built the encoder block from its parts. A pretrained encoder fails differently, and silently: ids from any tokenizer but its own run through all four layers without an error and come back as confident answers about nothing, and a download that names no commit can change every figure the day someone re-uploads the weights.

Goal

• Install transformers 5.20.2 into (bootcamp), load BERT-Mini's tokenizer and weights at a pinned commit, and find both in the Hub's cache.
• Tokenize Banking77 into WordPiece pieces, read input_ids, attention_mask and the special tokens, and show that the 161 validation words Lesson 5.3.1 could not store become pieces, with no [UNK] in all 13,083 queries.
• Read the load report's nine dropped tensors as the pretraining heads, and prove that a BertLayer is a post-norm block by rebuilding its output from its weights with attention.py.
• Mean-pool last_hidden_state over real tokens, compute frozen features once, and train a linear head to 0.7955 on validation against the bag's 0.8760.
• Feed the encoder Lesson 5.3.1's word ids, measure 0.0090 accuracy at nearly the confidence of the right ids, and write the check that catches it.

Prerequisites

None of this is required, and the lesson stands on its own. The first is the model itself explained before you load it, and the second is the Hub's own page on downloading a file at a fixed version.

• YouTube - "BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token" by Umar Jamil. How BERT is trained, how it is used once trained, how it differs from decoder models such as GPT and LLaMA, fine-tuning, and what the [CLS] token is for.
• Official docs - Hugging Face Hub, "Download files from the Hub", From specific version. The revision parameter, which takes a branch, a tag, a pull request or a commit hash, and the cache every downloaded file lands in.

Table of Contents

• Installing transformers
• The Hub, a Model Id and a Revision
• WordPiece Tokens
• Padding, Truncation and the Attention Mask
• The Pretrained Encoder and Its Load Report
• A BertLayer Is a Post-Norm Block
• Mean Pooling Over Real Tokens
• Frozen Features and a Linear Head
• The Wrong Tokenizer

Theory

Every block runs in ~/dev/bootcamp, beside text.py, Lesson 5.3.1's, which reads Banking77 from data/banking77, attention.py, Lesson 5.3.2's, and image_loop.py, Lesson 4.5.2's. The first subtitle installs transformers, and the first load of the tokenizer and of the model downloads their files from the Hub once, 87 MB in the cache. The seventh subtitle saves encoder.py, which every later script imports, and the eighth saves bert_features.pt, which Lab Exercise 2 and the Assignment load. Every head is seeded, and seconds are compared only inside the script that timed them. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Installing transformers

What. transformers is Hugging Face's library of pretrained models: one class per architecture, BertModel among them, and the Auto classes that read a checkpoint's configuration and build the right one. It is installed under the name transformers, and transformers==5.20.2 is the release every result in this section was printed by. Why. (bootcamp) holds PyTorch, which runs a transformer, and nothing that downloads trained weights, builds the matching architecture or turns text into the ids those weights expect. transformers does all three, on top of the PyTorch Module 4 installed. When. Once, now, with (bootcamp) active, the way Module 4 installed torch.

Before the install, (bootcamp) has no transformers to import:

(bootcamp) goku@machine:~/dev/bootcamp$ python3 -c "import transformers"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'transformers'

In ~/dev/bootcamp, with (bootcamp) active:

uv pip install transformers==5.20.2 tokenizers==0.23.3

uv prints Resolved 27 packages and Installed 22 packages, then one + line for each library it added:

 + annotated-doc==0.0.5
 + anyio==4.16.0
 + click==8.5.0
 + h11==0.16.0
 + hf-xet==1.8.0
 + httpcore2==2.16.0
 + httpx2==2.16.0
 + huggingface-hub==2.4.0
 + idna==3.20
 + markdown-it-py==4.2.0
 + mdurl==0.1.2
 + pygments==2.21.0
 + pyyaml==6.0.3
 + regex==2026.9.29
 + rich==15.0.0
 + safetensors==0.8.0
 + shellingham==1.5.4
 + tokenizers==0.23.3
 + tqdm==4.70.1
 + transformers==5.20.2
 + truststore==0.10.5
 + typer==0.27.3

Three of the 22 do the work this section sees: huggingface-hub downloads a repository's files and caches them, tokenizers is Hugging Face's tokenizer library, and safetensors reads a weights format that holds tensors and no code. Most of the rest are the HTTP client the downloads go through and the libraries behind the hf and transformers command-line tools, which no lesson uses. The other five packages uv resolved, NumPy, packaging, filelock, fsspec and typing-extensions, were already in (bootcamp). transformers and tokenizers are pinned, so only the other lines can read differently on your machine. Confirm the install from a script, check_transformers.py:

import huggingface_hub
import tokenizers
import transformers

print(transformers.__version__)     # 5.20.2
print(tokenizers.__version__)       # 0.23.3
print(huggingface_hub.__version__)  # 2.4.0

DON'T:

• DON'T install transformers without ==5.20.2. Every id, shape and score in this section came from that release, and the line without a version takes whatever release is newest on the day you run it.

The Hub, a Model Id and a Revision

What. The Hugging Face Hub is a website that hosts models as git repositories of files. A model id is organization/name, here google/bert_uncased_L-4_H-256_A-4: Google's uncased BERT with 4 layers (L-4), a hidden size of 256 (H-256) and 4 attention heads (A-4), the model its authors call BERT-Mini. revision= names which version of the repository to read, a branch, a tag or a commit hash, and AutoTokenizer.from_pretrained(NAME, revision=REVISION) downloads the tokenizer's files at that version into ~/.cache/huggingface/hub and builds the tokenizer from them. Why. A repository's default branch, main, moves whenever its owner pushes, and a load without revision= reads whatever main holds that day. A commit hash names one set of files forever, the way Lesson 5.2.1's ResNet18_Weights.IMAGENET1K_V1 names one weights file where DEFAULT names whichever torchvision ships as best. When. In every from_pretrained call whose results anyone will compare with a later run.

from pathlib import Path
from huggingface_hub import try_to_load_from_cache
from transformers import AutoTokenizer
from text import train, val, test, tokenize, vocab

NAME = "google/bert_uncased_L-4_H-256_A-4"
REVISION = "387825ce42dbb39b87911cdf8e383ee3b25184f8"
tokenizer = AutoTokenizer.from_pretrained(NAME, revision=REVISION)
print(type(tokenizer).__name__, len(tokenizer))
for file in ("config.json", "vocab.txt"):
    path = Path(try_to_load_from_cache(NAME, file, revision=REVISION))
    print(path.relative_to(Path.home()), path.stat().st_size)

BertTokenizer 30522
.cache/huggingface/hub/models--google--bert_uncased_L-4_H-256_A-4/snapshots/387825ce42dbb39b87911cdf8e383ee3b25184f8/config.json 383
.cache/huggingface/hub/models--google--bert_uncased_L-4_H-256_A-4/snapshots/387825ce42dbb39b87911cdf8e383ee3b25184f8/vocab.txt 231508

AutoTokenizer read the repository's config.json, found a BERT model, and built a BertTokenizer from vocab.txt, 30,522 lines of one token each, a line's number, counted from 0, being the token's id. try_to_load_from_cache returns where a repository's file sits in the cache at a revision, downloading nothing. Each repository gets a folder and each revision a folder under snapshots named by its commit, so two revisions of one model sit side by side and a pinned load reads its own. relative_to(Path.home()) prints the path from your home folder on.

DON'T:

• DON'T leave revision= out of a load whose figures you will quote. It then reads main, and the day the owner pushes new weights, every number downstream moves with no change to your code.
• DON'T edit or delete files under snapshots. They are links into the cache's shared store of downloaded files, and the library trusts them as downloaded.

WordPiece Tokens

What. BERT's tokenizer is WordPiece. It lowercases the text, splits it at whitespace and punctuation as Lesson 5.3.1's tokenize did, then cuts every word that is not in its vocabulary into the longest pieces that are, left to right. A piece that continues a word starts with ##. tokenizer.tokenize(text) returns the pieces, tokenizer(text) returns their ids with two special tokens added, [CLS] (id 101) at the start and [SEP] (id 102) at the end, and tokenizer.decode(ids) turns ids back into text. Why. A vocabulary of whole words meets a word it never stored, and Lesson 5.3.1's bag made every such word the same untrained <unk> row. A vocabulary that holds frequent words whole and the pieces of rarer ones can spell any word from its pieces, so a new word arrives as pieces the model trained on instead of as nothing. When. Every time text goes into this model, through this tokenizer and no other, the last subtitle's subject.

for query in ("My card hasn't materialised", "how do i rescind a transfer", "how do i revoke a transfer", "fiver dosh quid"):
    print(tokenizer.tokenize(query))
ids = tokenizer("fiver dosh quid")["input_ids"]
print(ids, tokenizer.convert_ids_to_tokens(ids))
print(tokenizer.decode(ids), "|", tokenizer.decode(ids, skip_special_tokens=True))
print(tokenizer.convert_ids_to_tokens([0, 1, 100, 101, 102, 103, 999, 1996, 4003]))
pieces = tokenizer.convert_ids_to_tokens(range(len(tokenizer)))
print(sum(p.startswith("##") for p in pieces), sum(p.startswith("[unused") for p in pieces))

['my', 'card', 'hasn', "'", 't', 'material', '##ised']
['how', 'do', 'i', 'res', '##cin', '##d', 'a', 'transfer']
['how', 'do', 'i', 'rev', '##oke', 'a', 'transfer']
['five', '##r', 'dos', '##h', 'qui', '##d']
[101, 2274, 2099, 9998, 2232, 21864, 2094, 102] ['[CLS]', 'five', '##r', 'dos', '##h', 'qui', '##d', '[SEP]']
[CLS] fiver dosh quid [SEP] | fiver dosh quid
['[PAD]', '[unused0]', '[UNK]', '[CLS]', '[SEP]', '[MASK]', '!', 'the', 'card']
5828 994

materialised, which Lesson 5.3.1 encoded as <unk>, is material and ##ised, and rescind and revoke, which both became the same id 1 there, are three pieces and two, different from each other. fiver dosh quid, three unknown words there, is six pieces here. hasn't still splits at the apostrophe, punctuation coming first. decode joins the pieces back into the lowercased text, skip_special_tokens=True drops [CLS] and [SEP], and convert_ids_to_tokens reads ids as entries: 0 is [PAD], 100 [UNK], 103 [MASK], the token pretraining hid words behind, 994 entries are [unused...] slots ordinary text never produces, and 5,828 are ## pieces. The words Lesson 5.3.1 lost:

unknown = sorted({t for text in val["text"] for t in tokenize(text) if t not in vocab})
split = {word: tokenizer.tokenize(word) for word in unknown}
print(len(unknown), sum(len(p) == 1 for p in split.values()), split["discrepancy"], split["atrocious"])
print([sum(i.count(tokenizer.unk_token_id) for i in tokenizer(list(df["text"]))["input_ids"]) for df in (train, val, test)])

161 128 ['disc', '##re', '##pan', '##cy'] ['at', '##ro', '##cious']
[0, 0, 0]

The validation queries hold 161 distinct words outside Lesson 5.3.1's vocabulary. BERT's vocabulary holds 128 of them whole, words such as above and authorization that 8,003 banking queries never happened to use, and cuts the other 33 into pieces, discrepancy into four. Across all 13,083 queries, training, validation and test, the tokenizer produced [UNK] zero times.

DON'T:

• DON'T read a piece as a word with a meaning of its own. rescind became res, ##cin and ##d, and what the model makes of those pieces is learned from how they occurred in its training text, which the eighth subtitle measures.
• DON'T expect this tokenizer to keep capitals. The checkpoint is uncased, and My came out as my.

Padding, Truncation and the Attention Mask

What. Called on a list of queries with padding=True and return_tensors="pt", the tokenizer returns a batch: input_ids, each row padded with [PAD], id 0, to the batch's longest query, attention_mask, 1 at every real token and 0 at every pad, and token_type_ids, which marks the first and second sentence when a model reads two at once, all 0 for a single query. truncation=True, max_length=64 cuts every query to 64 ids, keeping [SEP] last. Why. Lesson 5.3.1 padded every query to 48 and told pads by ids != 0. A pretrained model reads its pads from attention_mask, built beside the ids, and padding to each batch's longest query spends no compute on columns no query in the batch fills. When. Every batch of more than one query.

queries = ["card arrival?", "my card hasn't arrived", "fiver dosh quid"]
batch = tokenizer(queries, padding=True, return_tensors="pt")
print(list(batch), batch["input_ids"].shape)
print(batch["input_ids"])
print(batch["attention_mask"])
print(batch["token_type_ids"].unique())

['input_ids', 'token_type_ids', 'attention_mask'] torch.Size([3, 8])
tensor([[  101,  4003,  5508,  1029,   102,     0,     0,     0],
        [  101,  2026,  4003,  8440,  1005,  1056,  3369,   102],
        [  101,  2274,  2099,  9998,  2232, 21864,  2094,   102]])
tensor([[1, 1, 1, 1, 1, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1]])
tensor([0])

The batch is 8 wide, the length of the two longest queries with [CLS] and [SEP], and the 5 ids of card arrival? are followed by three pads that its mask marks 0. The batch is a dictionary of three tensors, which is why the model is called with model(**batch), each key becoming a keyword argument. How long the queries run in pieces, and what truncation does:

lengths = [len(i) for i in tokenizer(list(train["text"]))["input_ids"]]
print(max(lengths), sum(n > 64 for n in lengths), sum(n > 48 for n in lengths))
cut = tokenizer(list(train["text"]), truncation=True, max_length=64)["input_ids"]
print(max(len(i) for i in cut), sum(i[-1] == tokenizer.sep_token_id for i in cut))
print(tokenizer.model_max_length, len(tokenizer("card " * 600, truncation=True)["input_ids"]))

98 27 123
64 8003
1000000000000000019884624838656 602

The longest training query is 98 ids, and 27 of the 8,003 run past 64, where 48 ids would cut 123, against the 99 that Lesson 5.3.1's 48 words cut. Truncated at 64, every row still ends on [SEP]. truncation=True alone cuts at model_max_length, which this repository never sets, so the tokenizer carries a placeholder of about 10 to the 30th and 600 words keep all 602 ids.

DON'T:

• DON'T pad a batch without returning attention_mask to the model. The model reads its pads from the mask, and the seventh subtitle measures what it does with pads it is not told about.
• DON'T write truncation=True without max_length. Here it cut nothing, and the 602 ids went through as one row.

The Pretrained Encoder and Its Load Report

What. AutoModel.from_pretrained(NAME, revision=REVISION) reads the repository's config.json, builds a BertModel of that shape, downloads the weights file once, pytorch_model.bin, 45,088,961 bytes, and loads every tensor whose name matches a parameter of the model. It reports the tensors it did not use in a LOAD REPORT, and returns the model in evaluation mode. Why. A checkpoint is saved from the network that trained it, and BERT trained with heads on top of the encoder that a model meant for your task does not carry. The report says what was dropped, which is how you know the encoder arrived whole and only the heads stayed behind. When. Every first load of a checkpoint into a class other than the one that saved it, which is every load of an encoder for a new task.

Save this as load_bert.py:

from transformers import AutoModel

NAME = "google/bert_uncased_L-4_H-256_A-4"
REVISION = "387825ce42dbb39b87911cdf8e383ee3b25184f8"
model = AutoModel.from_pretrained(NAME, revision=REVISION)
print(type(model).__name__, model.training, sum(p.numel() for p in model.parameters()))

It prints this, with a progress bar, Loading weights, counting 71 tensors, after the first line, and the nine rows of the table can come in another order on your run:

Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
[transformers] BertModel LOAD REPORT from: google/bert_uncased_L-4_H-256_A-4
Key                                        | Status     |  |
-------------------------------------------+------------+--+-
cls.seq_relationship.weight                | UNEXPECTED |  |
cls.seq_relationship.bias                  | UNEXPECTED |  |
cls.predictions.transform.dense.weight     | UNEXPECTED |  |
cls.predictions.decoder.bias               | UNEXPECTED |  |
cls.predictions.transform.LayerNorm.bias   | UNEXPECTED |  |
cls.predictions.decoder.weight             | UNEXPECTED |  |
cls.predictions.bias                       | UNEXPECTED |  |
cls.predictions.transform.dense.bias       | UNEXPECTED |  |
cls.predictions.transform.LayerNorm.weight | UNEXPECTED |  |

Notes:
- UNEXPECTED:	can be ignored when loading from different task/architecture; not ok if you expect identical arch.
BertModel False 11170560

The first line is the Hub's own message, sent back with its answer to a request transformers makes on every load, and no lesson needs the token it asks for. The table lists the nine tensors of the file that BertModel has no place for, all under cls., UNEXPECTED meaning in the file and not in the model. model.training is False: from_pretrained hands the model back in evaluation mode, its dropout off. The first load also fetches in the background the Hub's converted copy of the weights, model.safetensors, 45,080,576 bytes in another commit's snapshot, which the pinned load never reads. The three print on every load, so every later script that loads the model starts with these five lines, which set both libraries' messages to errors only:

from huggingface_hub.utils import logging as hub_logging
from transformers import logging
logging.set_verbosity_error()
logging.disable_progress_bar()
hub_logging.set_verbosity_error()

import torch
from torch import nn
from huggingface_hub import try_to_load_from_cache
from transformers import AutoModel, AutoTokenizer
from attention import attention

NAME = "google/bert_uncased_L-4_H-256_A-4"
REVISION = "387825ce42dbb39b87911cdf8e383ee3b25184f8"
tokenizer = AutoTokenizer.from_pretrained(NAME, revision=REVISION)
model = AutoModel.from_pretrained(NAME, revision=REVISION)
stored = torch.load(try_to_load_from_cache(NAME, "pytorch_model.bin", revision=REVISION), weights_only=True)
dropped = sorted(k for k in stored if k.startswith("cls."))
print(len(stored), len(model.state_dict()), len(dropped))
for k in dropped:
    print(k, tuple(stored[k].shape))
print(torch.equal(stored["cls.predictions.decoder.weight"], model.embeddings.word_embeddings.weight))
print(all(torch.equal(stored[f"bert.{k}"], v) for k, v in model.state_dict().items()))

80 71 9
cls.predictions.bias (30522,)
cls.predictions.decoder.bias (30522,)
cls.predictions.decoder.weight (30522, 256)
cls.predictions.transform.LayerNorm.bias (256,)
cls.predictions.transform.LayerNorm.weight (256,)
cls.predictions.transform.dense.bias (256,)
cls.predictions.transform.dense.weight (256, 256)
cls.seq_relationship.bias (2,)
cls.seq_relationship.weight (2, 256)
True
True

torch.load(path, weights_only=True) reads the file as a dictionary of tensors, and refuses any object in it but tensors and plain Python values such as dictionaries, lists, numbers and strings. It holds 80 tensors: the model's 71, saved under bert. and each equal to the one the model holds, and the 9 heads. cls.predictions maps each token's final 256 numbers through a dense layer and a LayerNorm to 30,522 scores, and its decoder weight is the word embedding table itself, so a word's score is how well the final vector matches that word's own row. cls.seq_relationship maps one vector to 2 scores.

Those heads are how BERT learned. It was trained by self-supervision, where the labels come from the text itself: a share of each sentence's tokens was hidden behind [MASK], and the network was trained to predict the hidden words from the words on both sides of them, through cls.predictions. cls.seq_relationship is BERT's second pretraining head, which reads two sentences at once, the token_type_ids of the previous subtitle telling them apart, and scores whether the second followed the first. A decoder such as GPT is trained by the same idea in one direction: it predicts every next token from the tokens before it alone, so its attention carries Lesson 5.3.2's causal mask, and a model trained that way writes text one token at a time. Neither objective names an intent, so the encoder arrives knowing how words fill sentences and nothing about Banking77's 77 intents. What it is built from:

c = model.config
print(c.num_hidden_layers, c.hidden_size, c.num_attention_heads, c.intermediate_size, c.hidden_act, c.max_position_embeddings, c.vocab_size)
for name, part in model.named_children():
    print(name, sum(p.numel() for p in part.parameters()))
try:
    with torch.no_grad():
        model(**tokenizer("card " * 600, truncation=True, return_tensors="pt"))
except RuntimeError as err:
    print(err)

4 256 4 1024 gelu 512 30522
embeddings 7945728
encoder 3159040
pooler 65792
The size of tensor a (602) must match the size of tensor b (512) at non-singleton dimension 1

model.config is the parsed config.json: 4 layers, 256 features per token, 4 heads of 64, a feed-forward layer 1,024 wide, the GELU activation (a smooth version of ReLU, x times the probability that a standard normal value falls below x) and a position table of 512 rows, Lesson 5.3.3's learned positions at BERT's size. The embeddings hold 7,945,728 of the 11,170,560 parameters, mostly the [30522, 256] word table, the four layers hold 3,159,040, and the pooler, the seventh subtitle's, 65,792. The 602 ids that truncation=True let through meet the 512 rows of the position table and are refused with a shape error that never mentions positions.

DON'T:

• DON'T read UNEXPECTED as a failed load. All 71 tensors the model has came from the file, and the nine left out are the pretraining heads, which a model for a new task replaces with its own.
• DON'T send this model more than 512 ids in a row. The 602 were refused with a shape error that names neither the position table nor truncation.

A BertLayer Is a Post-Norm Block

What. model.encoder.layer holds four BertLayers, each Lesson 5.3.3's block at BERT's size, 4 heads of attention and a feed-forward network of 256 to 1,024 to 256 with GELU. BERT normalizes after each sum, LayerNorm(h + attention(h)) and then LayerNorm(a + ffn(a)), the post-norm order of Lesson 5.3.3's norm_first=False, where that lesson's Block normalized before each part. Why. A printout names a model's parts and not the order they run in. Rebuilding one layer from its own weights with attention.py and matching its output proves what it computes, and the order matters, since the same weights in the other order compute another function. When. Before any layer of a pretrained model is changed, frozen or replaced.

print(model.encoder.layer[0])

BertLayer(
  (attention): BertAttention(
    (self): BertSelfAttention(
      (query): Linear(in_features=256, out_features=256, bias=True)
      (key): Linear(in_features=256, out_features=256, bias=True)
      (value): Linear(in_features=256, out_features=256, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (output): BertSelfOutput(
      (dense): Linear(in_features=256, out_features=256, bias=True)
      (LayerNorm): LayerNorm((256,), eps=1e-12, elementwise_affine=True, bias=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
  )
  (intermediate): BertIntermediate(
    (dense): Linear(in_features=256, out_features=1024, bias=True)
    (intermediate_act_fn): GELUActivation()
  )
  (output): BertOutput(
    (dense): Linear(in_features=1024, out_features=256, bias=True)
    (LayerNorm): LayerNorm((256,), eps=1e-12, elementwise_affine=True, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
  )
)

Each LayerNorm sits inside an output part, after its dense layer, which suggests post-norm and does not prove it. The layer rebuilt by hand, on one query with no padding so no mask is needed:

batch = tokenizer(["my card hasn't arrived"], return_tensors="pt")
layer = model.encoder.layer[0]
sa, so = layer.attention.self, layer.attention.output

def heads(x):
    return x.view(1, -1, 4, 64).transpose(1, 2)

def mix(x):
    out, weights = attention(heads(sa.query(x)), heads(sa.key(x)), heads(sa.value(x)))
    return so.dense(out.transpose(1, 2).reshape(1, -1, 256))

def ffn(x):
    return layer.output.dense(nn.functional.gelu(layer.intermediate.dense(x)))

with torch.no_grad():
    h = model.embeddings(input_ids=batch["input_ids"])
    a = so.LayerNorm(h + mix(h))
    post = layer.output.LayerNorm(a + ffn(a))
    b = h + mix(so.LayerNorm(h))
    pre = b + ffn(layer.output.LayerNorm(b))
    print(h.shape, torch.allclose(post, layer(h), atol=1e-5), torch.allclose(pre, layer(h), atol=1e-5))
    x = h
    for each in model.encoder.layer:
        x = each(x)
    print(torch.allclose(x, model(**batch).last_hidden_state, atol=1e-5))

torch.Size([1, 8, 256]) True False
True

model.embeddings adds the word, position and token type rows and normalizes them, which gives the [1, 8, 256] input of the first layer. heads is Lesson 5.3.3's split with view and transpose, 4 heads of 64, and mix runs Lesson 5.3.2's attention on them and merges the heads back. The post-norm rebuild matches the layer within 1e-5, and the same weights in the pre-norm order do not. The four layers called one after another give last_hidden_state, so the encoder is the embeddings and four of these blocks, with no LayerNorm after the last: in the post-norm order every layer's output is already normalized, where Lesson 5.3.3's pre-norm encoder needed one more.

DON'T:

• DON'T read a module's printout as its order of operations. The printout listed each LayerNorm inside an output part, and only the rebuild matching layer(h) showed where it sits.
• DON'T assume a pretrained block is pre-norm because Lesson 5.3.3's was. The same weights in the pre-norm order did not match layer(h).

Mean Pooling Over Real Tokens

What. model(**batch) returns an output object whose last_hidden_state is [batch, tokens, 256], one vector per token after the fourth layer, [CLS], [SEP] and pads included. Mean pooling turns it into one vector per query by Lesson 5.3.1's masked mean, with attention_mask as the mask: multiply each token's vector by its mask entry, sum over tokens, divide by the count of real tokens. The output also carries pooler_output, [CLS]'s final vector through one more dense layer and a tanh, the input of the next-sentence head. Why. A classifier's head takes one vector per query. Every real token's vector has been mixed with every other's through four layers of attention, so their mean summarizes the whole query, and leaving the pads out keeps a query's vector the same whatever it is padded to, exactly as in Lesson 5.3.1. When. Whenever a query's token vectors become the one vector a head reads.

queries = ["card arrival?", "my card hasn't arrived", "how do i revoke a transfer"]
batch = tokenizer(queries, padding=True, return_tensors="pt")
with torch.no_grad():
    out = model(**batch)
    alone = model(**tokenizer(queries[0], return_tensors="pt")).last_hidden_state
    blind = model(input_ids=batch["input_ids"]).last_hidden_state
hidden = out.last_hidden_state
print(type(out).__name__, hidden.shape, out.pooler_output.shape)
real = batch["attention_mask"].unsqueeze(-1).float()
mean = (hidden * real).sum(dim=1) / real.sum(dim=1)
print(real.shape, real.sum(dim=1).flatten(), mean.shape)
print(alone.shape, torch.allclose(hidden[0, :5], alone[0], atol=1e-5), torch.allclose(mean[0], alone[0].mean(dim=0), atol=1e-5))
print(torch.allclose(blind[0, :5], alone[0], atol=1e-5), round((blind[0, :5] - alone[0]).abs().max().item(), 4))

BaseModelOutputWithPoolingAndCrossAttentions torch.Size([3, 9, 256]) torch.Size([3, 256])
torch.Size([3, 9, 1]) tensor([5., 8., 9.]) torch.Size([3, 256])
torch.Size([1, 5, 256]) True True
False 1.8865

unsqueeze(-1) makes the [3, 9] mask [3, 9, 1], so it broadcasts across the 256 features, and the three queries hold 5, 8 and 9 real tokens. card arrival? padded to 9 in the batch gives at its 5 real positions what it gives alone, and its masked mean equals the plain mean of its 5 tokens alone. Called with input_ids and no mask, the model attends to the pads as if they were words, and the same 5 tokens move by up to 1.8865. The pieces go into one file. Save it as encoder.py in ~/dev/bootcamp:

import torch
from transformers import AutoModel, AutoTokenizer
from text import train, val, test, intents

NAME = "google/bert_uncased_L-4_H-256_A-4"
REVISION = "387825ce42dbb39b87911cdf8e383ee3b25184f8"
tokenizer = AutoTokenizer.from_pretrained(NAME, revision=REVISION)

def load():
    return AutoModel.from_pretrained(NAME, revision=REVISION)

def labels(df):
    return torch.tensor([intents.index(c) for c in df["category"]])

def batches(df, size, generator=None):
    texts, y = list(df["text"]), labels(df)
    if generator is None:
        order = torch.arange(len(texts))
    else:
        order = torch.randperm(len(texts), generator=generator)
    for i in range(0, len(texts), size):
        idx = order[i:i + size]
        batch = tokenizer([texts[j] for j in idx.tolist()], padding=True,
                          truncation=True, max_length=64, return_tensors="pt")
        yield batch, y[idx]

def pooled(model, batch):
    hidden = model(**batch).last_hidden_state
    real = batch["attention_mask"].unsqueeze(-1).float()
    return (hidden * real).sum(dim=1) / real.sum(dim=1)

load() returns a fresh pretrained model on every call, so a script that trains one copy can load another untouched. labels(df) turns a frame's intents into class numbers by text.py's intents. batches(df, size) is a generator, a function whose yield hands back one result each time a loop asks for the next and then waits, so it produces (batch, y) pairs: it tokenizes size queries at a time, padded to that batch's longest and cut at 64, in the frame's order, or in a shuffled order drawn from generator when one is given. pooled is the masked mean above.

DON'T:

• DON'T call the model without attention_mask on a padded batch. The real tokens of card arrival? moved by up to 1.8865 the moment the pads were attended to.
• DON'T pool with pooler_output because it is already one vector. It is [CLS] read through a layer trained for the next-sentence head, and the Assignment measures it against the mean.

Frozen Features and a Linear Head

What. In evaluation mode under torch.no_grad(), the pretrained encoder is a fixed function, so its pooled vector for every training and validation query is computed once, 256 numbers each, and a linear head, nn.Linear(256, 77), trains on those vectors alone. It is Lesson 5.2.2's cached-features method, with BERT-Mini where that lesson had ResNet-18. Why. The frozen head is Lesson 5.2.2's baseline, the first rung of every adaptation: it measures what the pretrained features already separate before any of the eleven million weights move, and computing the features once puts the encoder's whole cost in one pass. When. First, on any new task for a pretrained encoder, beside the best model trained from scratch on the same data.

from huggingface_hub.utils import logging as hub_logging
from transformers import logging
logging.set_verbosity_error()
logging.disable_progress_bar()
hub_logging.set_verbosity_error()

import time
import torch
from torch import nn
from encoder import tokenizer, load, labels, batches, pooled
from text import train, val, intents, val_set
from image_loop import loss_fn

model = load()
batch, y = next(batches(val, 4))
model.train()
with torch.no_grad():
    print(torch.equal(pooled(model, batch), pooled(model, batch)))
model.eval()
with torch.no_grad():
    print(torch.equal(pooled(model, batch), pooled(model, batch)))

def features(df):
    with torch.no_grad():
        return torch.cat([pooled(model, batch) for batch, y in batches(df, 64)])

start = time.perf_counter()
F, G = features(train), features(val)
Y, Z = labels(train), labels(val)
print(F.shape, G.shape, f"{time.perf_counter() - start:.1f}")
torch.save({"F": F, "Y": Y, "G": G, "Z": Z}, "bert_features.pt")

False
True
torch.Size([8003, 256]) torch.Size([2000, 256]) 12.0

In training mode the same four queries gave two different vectors, because dropout zeroes a fresh random tenth of the activations on each pass, and in evaluation mode the same vector twice. The 10,003 queries went through the encoder once, and bert_features.pt keeps the four tensors. The head, trained on them by Lesson 5.2.2's loop:

def score(head, X, y):
    with torch.no_grad():
        return f"{(head(X).argmax(dim=1) == y).float().mean().item():.4f}"

torch.manual_seed(42)
head = nn.Linear(256, len(intents))
optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
order = torch.Generator().manual_seed(42)
start = time.perf_counter()
for epoch in range(1, 31):
    perm = torch.randperm(len(F), generator=order)
    for i in range(0, len(F), 32):
        idx = perm[i:i + 32]
        loss = loss_fn(head(F[idx]), Y[idx])
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    if epoch in (1, 5, 10, 20, 30):
        print(epoch, score(head, F, Y), score(head, G, Z))
print(f"{time.perf_counter() - start:.1f}")

1 0.4931 0.4315
5 0.7606 0.6865
10 0.8201 0.7450
20 0.8784 0.7830
30 0.9045 0.7955
3.0

Each line is the epoch, then the accuracy on the training and on the validation features. After thirty epochs the head scores 0.7955 on validation, under Lesson 5.3.1's bag at 0.8760, and 0.9045 on its own training queries, where Lesson 5.3.3 measured the bag at 0.9961, and in the same script its thirty epochs cost less than the one feature pass. The two models differ in more than freezing, the tokenizer, the width, the rate and the epochs among them, so the gap is no one setting's. What the runs do show is where the frozen vectors sit:

def vector(query):
    with torch.no_grad():
        return pooled(model, tokenizer([query], return_tensors="pt"))[0]

def predict(query):
    with torch.no_grad():
        probs = head(vector(query)).softmax(dim=0)
    return intents[probs.argmax()], round(probs.max().item(), 3)

cos = nn.functional.cosine_similarity
cancel = vector("how do i cancel a transfer")
print(round(cos(cancel, vector("how do i make a transfer"), dim=0).item(), 3), round(cos(cancel, vector("please stop the payment i just sent"), dim=0).item(), 3))
unit = nn.functional.normalize(F, dim=1)
sims = unit @ unit.T
match = Y[:, None] == Y[None, :]
same = match & ~torch.eye(len(Y), dtype=torch.bool)
print(round(sims[same].mean().item(), 3), round(sims[~match].mean().item(), 3))
for query in ("how do i cancel a transfer", "how do i revoke a transfer", "how do i rescind a transfer", "fiver dosh quid"):
    print(query, predict(query))

0.961 0.794
0.797 0.737
how do i cancel a transfer ('cancel_transfer', 0.964)
how do i revoke a transfer ('cancel_transfer', 0.772)
how do i rescind a transfer ('transfer_fee_charged', 0.293)
fiver dosh quid ('direct_debit_payment_not_recognised', 0.956)

Cancelling a transfer and making one, opposite requests in nearly the same words, sit at a cosine of 0.961, and a request to cancel in other words at 0.794. Over the training features, two distinct queries of one intent average a cosine of 0.797 and two of different intents 0.737: every query points in nearly one direction, and the intent is a small part of it. Pretraining shaped these vectors to fill in hidden words, and nothing in it asked which of 77 requests a sentence makes, while the bag's table was trained on those requests alone. revoke, which the bag sent to transfer_into_account, reaches cancel_transfer at 0.772 through its pieces, rescind does not, and fiver dosh quid comes back at 0.956, more confident than the bag's 0.62: pieces end the shared <unk> row, and they do not make a head say it does not know.

DON'T:

• DON'T compute features with the model in training mode. The same four queries gave two different vectors in training mode and the same vector twice in evaluation mode.
• DON'T read 0.7955 against 0.8760 as pretraining failing. The frozen vectors were never asked about intents, and the two runs differ in their tokenizer, width, rate and epochs as well.
• DON'T read the head's 0.956 on fiver dosh quid as knowledge. A query with no request in it still lands somewhere, and the head answers for wherever it lands.

The Wrong Tokenizer

What. The tokenizer belongs to the model: id 4003 means card only because BERT's embedding table learned row 4003 from every card in its training text. Ids from any other tokenizer are still integers below 30,522, so the model looks up their rows, runs four layers and returns vectors of the right shape, and nothing anywhere raises. Why. Nothing in a tensor of ids records which vocabulary produced it. The only failure the model can detect is an id past its table, so every other mismatch is silent, and the head reads its features as confidently as it reads right ones. When. Whenever ids reach a pretrained model from anywhere but its own tokenizer: an earlier pipeline's encoded files, a vocabulary built by hand, the tokenizer of another checkpoint.

X, _ = val_set.tensors
print(X[0, :6].tolist(), tokenizer.convert_ids_to_tokens(X[0, :6]))
wrong = []
with torch.no_grad():
    for i in range(0, len(X), 64):
        x = X[i:i + 64]
        wrong.append(pooled(model, {"input_ids": x, "attention_mask": (x != 0).long()}))
W = torch.cat(wrong)
with torch.no_grad():
    right_p, wrong_p = head(G).softmax(dim=1), head(W).softmax(dim=1)
for p in (right_p, wrong_p):
    print(f"{(p.argmax(dim=1) == Z).float().mean().item():.4f}", f"{p.max(dim=1).values.mean().item():.3f}", p.argmax(dim=1).unique().numel())
top = wrong_p.argmax(dim=1).mode().values.item()
print(intents[top], (wrong_p.argmax(dim=1) == top).sum().item())

[11, 12, 141, 17, 29, 6] ['[unused10]', '[unused11]', '[unused136]', '[unused16]', '[unused28]', '[unused5]']
0.7955 0.718 77
0.0090 0.705 33
exchange_via_app 1289

val_set holds the validation queries as Lesson 5.3.1 encoded them, word ids padded with 0 to 48, in the order of G and Z. Read in BERT's vocabulary, the first query's ids are [unused...] slots, and the mask from ids != 0 is a correct mask, so every part of the call is valid except what the ids mean. Each line is the head's accuracy, its average top probability and how many intents it answered. On BERT's own ids: 0.7955, 0.718, all 77. On the word ids: 0.0090, under the 0.0130 of guessing one intent in 77, at 0.705, and mode(), the most frequent value, shows 1,289 of the 2,000 queries sent to exchange_via_app. The fix is the model's own tokenizer, which batches always uses, and the check is what that tokenizer always writes:

def from_tokenizer(ids, mask):
    rows = torch.arange(len(ids))
    first = ids[:, 0] == tokenizer.cls_token_id
    last = ids[rows, mask.sum(dim=1) - 1] == tokenizer.sep_token_id
    return bool((first & last).all())

batch, y = next(batches(val, 64))
print(from_tokenizer(batch["input_ids"], batch["attention_mask"]), from_tokenizer(X[:64], (X[:64] != 0).long()))
try:
    with torch.no_grad():
        model(input_ids=torch.tensor([[101, 30522, 102]]))
except IndexError as err:
    print(err)

True False
index out of range in self

from_tokenizer tests that every row starts with [CLS] and that its last real id, at the position the mask's count of real tokens gives, is [SEP]. rows pairs each row with its own position, so ids[rows, ...] picks one id per row. A batch from batches passes, and the first 64 queries of word ids fail. An id of 30,522, one past the table's last row, is the only kind of mistake the model itself refuses, with an IndexError from the embedding lookup that names neither the tokenizer nor the vocabulary.

DON'T:

• DON'T check a pipeline by whether it raises. 2,000 queries of the wrong ids ran through all four layers without an error.
• DON'T read the head's confidence as a check on its input. It averaged 0.705 on the wrong ids against 0.718 on the right ones, at 0.0090 accuracy.

Lab Exercises

Lab Exercise 1: Pieces for Every Word

In pieces.py, import tokenizer from encoder.py and train, tokenize and vocab from text.py. Tokenize each of the 2,200 words of Lesson 5.3.1's vocabulary, the two reserved entries left out, with BERT's tokenizer, and print how many it keeps as one piece and how many it cuts, then the five words cut into the most pieces with their pieces.

Then, over the 8,003 training queries, print the total count of Lesson 5.3.1's word tokens, the total count of WordPiece pieces with no special tokens, and their ratio to three decimals. Say in a comment why pieces outnumber words by so little.

Lab Exercise 2: Pretrained Against Random

In random_bert.py, load the pretrained encoder with load(), then build a second BertModel of the same shape with random weights: AutoModel.from_config(pretrained.config) after torch.manual_seed(42), put in evaluation mode. Print its class, its parameter count and whether its word embedding table equals the pretrained one.

Compute the random model's mean-pooled features for the training and validation queries through batches and pooled, and load the pretrained ones from bert_features.pt. Train one linear head on each set by the eighth subtitle's recipe and print each head's training and validation accuracy, then each set's mean cosine between every pair of training queries. Say in a comment what pretraining bought, and why a network that learned nothing still scores far above chance.

from encoder import tokenizer
from text import train, tokenize, vocab

words = [w for w in vocab if w not in ("<pad>", "<unk>")]
pieces = {w: tokenizer.tokenize(w) for w in words}
whole = sum(len(p) == 1 for p in pieces.values())
print(len(words), whole, len(words) - whole)
print(sorted(pieces.items(), key=lambda kv: -len(kv[1]))[:5])
n_words = sum(len(tokenize(t)) for t in train["text"])
n_pieces = sum(len(tokenizer.tokenize(t)) for t in train["text"])
print(n_words, n_pieces, round(n_pieces / n_words, 3))
# Most words of a banking query are common English, whole in a vocabulary
# of 30,522, so pieces outnumber words by under 3 percent.

2200 1918 282
[('cryptocurrency', ['crypt', '##oc', '##ur', '##ren', '##cy']), ('unexplained', ['une', '##x', '##pl', '##aine', '##d']), ('beneficiairy', ['ben', '##ef', '##icia', '##ir', '##y']), ('witdhrawals', ['wit', '##dh', '##ra', '##wal', '##s']), ('disposable', ['di', '##sp', '##osa', '##ble'])]
111243 113998 1.025

from huggingface_hub.utils import logging as hub_logging
from transformers import logging
logging.set_verbosity_error()
logging.disable_progress_bar()
hub_logging.set_verbosity_error()

import torch
from torch import nn
from transformers import AutoModel
from encoder import load, labels, batches, pooled
from text import train, val, intents
from image_loop import loss_fn

pretrained = load()
torch.manual_seed(42)
random = AutoModel.from_config(pretrained.config)
random.eval()
same = torch.equal(random.embeddings.word_embeddings.weight, pretrained.embeddings.word_embeddings.weight)
print(type(random).__name__, sum(p.numel() for p in random.parameters()), same)

def features(model, df):
    with torch.no_grad():
        return torch.cat([pooled(model, batch) for batch, y in batches(df, 64)])

def fit(F, Y, G, Z):
    torch.manual_seed(42)
    head = nn.Linear(256, len(intents))
    optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
    order = torch.Generator().manual_seed(42)
    for epoch in range(30):
        perm = torch.randperm(len(F), generator=order)
        for i in range(0, len(F), 32):
            idx = perm[i:i + 32]
            loss = loss_fn(head(F[idx]), Y[idx])
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    with torch.no_grad():
        return [f"{(head(X).argmax(dim=1) == y).float().mean().item():.4f}" for X, y in ((F, Y), (G, Z))]

cached = torch.load("bert_features.pt")
F, G = features(random, train), features(random, val)
sets = {"pretrained": (cached["F"], cached["G"]), "random": (F, G)}
for name, (train_f, val_f) in sets.items():
    unit = nn.functional.normalize(train_f, dim=1)
    print(name, fit(train_f, labels(train), val_f, labels(val)), round((unit @ unit.T).mean().item(), 3))
# Pretraining bought 7.6 points and spread the queries apart. The random
# network still gives every piece its own fixed row, so its pooled vector
# still depends on which pieces a query holds, which a head can read.

BertModel 11170560 False
pretrained ['0.9045', '0.7955'] 0.738
random ['0.7831', '0.7195'] 0.935

Theory Assessment

Q1: A Commit, Not a Name

A teammate loads google/bert_uncased_L-4_H-256_A-4 with no revision= and reports the head's 0.7955 in a design document. Say what that load reads, what a commit hash fixes that the model id does not, where two revisions of one model sit on disk, and what Lesson 5.2.1's ResNet18_Weights.IMAGENET1K_V1 has in common with REVISION.

Hint: Ask what main is in a git repository, and look at the folder name try_to_load_from_cache printed under snapshots.

Q2: Pieces, Not Unknowns

Lesson 5.3.1's vocabulary left 171 validation tokens unknown, and BERT's tokenizer produced [UNK] zero times in all 13,083 queries. Say how WordPiece manages that, what ## marks, what the tokenizer adds around every query and in every padded batch, and why revoke reached cancel_transfer while fiver dosh quid still came back at 0.956.

Hint: A vocabulary that holds single characters can spell anything, and a head answers for wherever a vector lands.

Q3: Where the LayerNorm Sits

Say what the rebuild of the first BertLayer proved and why the printout alone could not, why BERT's encoder ends with no LayerNorm after its fourth layer where Lesson 5.3.3's encoder needed one, and what the nine UNEXPECTED tensors of the load report were trained to do.

Hint: Compare LayerNorm(h + mix(h)) with h + mix(LayerNorm(h)), and ask what comes out of the last sum in each order.

Q4: The Pipeline That Ran

A pipeline feeds BERT-Mini Lesson 5.3.1's word ids with the mask ids != 0, and the frozen head answers every query. Say why nothing raised, why the head's average top probability stayed near its value on the right ids while its accuracy fell to 0.0090, which check catches the mistake and what it tests, and which ids would have made the model raise, and why.

Hint: The model reads an id as a row number of its table, and a softmax's top probability says how far one logit stands above the rest, not whether the input meant anything.

Assignment

Assignment: Which Layer Holds the Intents

The eighth subtitle pooled the encoder's last layer and its head reached 0.7955. Write layers.py, which asks whether that was the best place to read the encoder from.

One option this lesson did not use. model(**batch, output_hidden_states=True) returns, beside last_hidden_state, hidden_states: a tuple of five [batch, tokens, 256] tensors, the embeddings' output first and then each of the four layers' outputs in order.

For every training, validation and test query, compute seven features in one pass per split: the masked mean of each of the five hidden states, named mean 0 to mean 4, the [CLS] vector of the last layer, cls 4, and pooler_output. Train one linear head per feature by the eighth subtitle's recipe, print each one's training and validation accuracy, and confirm that mean 4 is the feature bert_features.pt holds. Let the validation accuracies choose one feature, score the test queries once with its head, Lesson 5.2.3's rule, and say what the five means show about what masked-word pretraining did layer by layer.

Hints:

• The tuple holds one more tensor than the model has layers, and its last is last_hidden_state, so mean 4 must equal bert_features.pt's F within 1e-5.
• [CLS] is position 0 of every row, never a pad, so last_hidden_state[:, 0] needs no mask.

from huggingface_hub.utils import logging as hub_logging
from transformers import logging
logging.set_verbosity_error()
logging.disable_progress_bar()
hub_logging.set_verbosity_error()

import torch
from torch import nn
from encoder import load, labels, batches
from text import train, val, test, intents
from image_loop import loss_fn

model = load()

def views(df):
    found = {}
    with torch.no_grad():
        for batch, y in batches(df, 64):
            out = model(**batch, output_hidden_states=True)
            real = batch["attention_mask"].unsqueeze(-1).float()
            parts = {f"mean {k}": (h * real).sum(dim=1) / real.sum(dim=1) for k, h in enumerate(out.hidden_states)}
            parts["cls 4"] = out.last_hidden_state[:, 0]
            parts["pooler"] = out.pooler_output
            for name, f in parts.items():
                found.setdefault(name, []).append(f)
    return {name: torch.cat(fs) for name, fs in found.items()}

def fit(F, Y):
    torch.manual_seed(42)
    head = nn.Linear(256, len(intents))
    optimizer = torch.optim.Adam(head.parameters(), lr=0.001)
    order = torch.Generator().manual_seed(42)
    for epoch in range(30):
        perm = torch.randperm(len(F), generator=order)
        for i in range(0, len(F), 32):
            idx = perm[i:i + 32]
            loss = loss_fn(head(F[idx]), Y[idx])
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
    return head

def score(head, X, y):
    with torch.no_grad():
        return (head(X).argmax(dim=1) == y).float().mean().item()

T, V = views(train), views(val)
Y, Z = labels(train), labels(val)
print(list(T), T["mean 0"].shape)
heads, scores = {}, {}
for name in T:
    heads[name] = fit(T[name], Y)
    scores[name] = score(heads[name], V[name], Z)
    print(name, f"{score(heads[name], T[name], Y):.4f}", f"{scores[name]:.4f}")
print(torch.allclose(T["mean 4"], torch.load("bert_features.pt")["F"], atol=1e-5))
chosen = max(scores, key=scores.get)
print(chosen, f"{score(heads[chosen], views(test)[chosen], labels(test)):.4f}")

['mean 0', 'mean 1', 'mean 2', 'mean 3', 'mean 4', 'cls 4', 'pooler'] torch.Size([8003, 256])
mean 0 0.9194 0.8435
mean 1 0.9183 0.8300
mean 2 0.8988 0.8105
mean 3 0.9035 0.8035
mean 4 0.9045 0.7955
cls 4 0.8948 0.7735
pooler 0.6364 0.5605
True
mean 0 0.8474

Additional Resources

Optional, and nothing later depends on either. Worth opening for how another family of models cuts its text, or for the paper and the sibling models behind this checkpoint.

• YouTube - "Let's build the GPT Tokenizer" by Andrej Karpathy. The byte-pair encoding tokenizer that GPT models use, another subword scheme beside WordPiece, built in code from raw bytes up.
• Model card - Hugging Face, google/bert_uncased_L-4_H-256_A-4. The 24 compact BERT models this checkpoint belongs to, by layers from 2 to 12 and hidden size from 128 to 768, trained with the standard BERT recipe, the GLUE scores (a public benchmark of sentence tasks) of four of them and the fine-tuning settings behind those scores, and the paper that released them.

Summary

Every model before this lesson started from a seeded draw and knew only what its training loop taught it. This one arrived trained, and loading it was a matter of checking what was trusted: the weights at a named commit, the tokenizer whose ids those weights learned, the heads dropped and accounted for, the mode it came back in.

The tokenizer solved the problem Lesson 5.3.1 ended on, and only that one. No word of Banking77 fell into a shared unknown row, and a query of slang still came back as confidently as a real request. Nothing in an integer says which vocabulary wrote it, so the right tokenizer is a contract the code checks, never one the model enforces.

Frozen, the encoder lost to the bag, and a network with random weights came within eight points of it. That is pretraining describing, not failing: layers shaped to fill in missing words put every query in nearly one direction and leave the intent a small part of it, while the bag's table was shaped by the intents alone. What a frozen head reads depends on which layer it reads, and the next lesson stops reading and trains the encoder's weights on the 77 intents with the loop Module 4 built.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

tokenizer = AutoTokenizer.from_pretrained(NAME, revision=REVISION)
model = AutoModel.from_pretrained(NAME, revision=REVISION)
batch = tokenizer(texts, padding=True, truncation=True,
                  max_length=64, return_tensors="pt")
with torch.no_grad():
    hidden = model(**batch).last_hidden_state
real = batch["attention_mask"].unsqueeze(-1).float()
features = (hidden * real).sum(dim=1) / real.sum(dim=1)

def from_tokenizer(ids, mask):
    rows = torch.arange(len(ids))
    first = ids[:, 0] == tokenizer.cls_token_id
    # the last real id of every row
    last = ids[rows, mask.sum(dim=1) - 1] == tokenizer.sep_token_id
    return bool((first & last).all())