Section 5.3: Attention and the Transformer

A query to a bank has no fixed size, and the words that decide its intent can sit anywhere in it, so text reaches a network as token ids first and as learned vectors second. This section reads Banking77, 13,083 short customer queries in 77 intents, cuts each into words with a vocabulary built from the training queries alone, and trains a bag of embeddings, a network that averages a query's word vectors, to 0.8760 on the validation queries. It then builds attention from matrix multiplications you can already read, every token weighting every other by a softmax over dot products, and assembles the transformer block from it, several heads, position embeddings, layer normalization, a feed-forward network and the residual connection, checked against PyTorch's own layer at every position.

Section 5.2 adapted a network ImageNet had trained. Here every network trains from a seeded draw on 8,003 queries, and the measurements are honest about what that buys. The failures are quiet again. A bag that averages its padding in scores 0.8820 on padded validation queries, above the correct bag, and changes 40 answers the moment the padding changes. Attention without the square-root scale puts on average 0.9321 of a row's weight on a single key at 512 dimensions, and without its padding mask about nine tenths of every real token's weight lands on pads. Without positions, attention cannot tell a query from its own words reordered, and on these short queries the two-block encoder with positions reaches 0.8295, above the same encoder without them at 0.7880 and under the bag, so the validation queries choose the bag, which scores 0.8808 on the test queries once.

Lessons

  1. Text as Tensors - Tokens and Embeddings - Banking77 as a frame, tokens and the vocabulary, encoding, truncation and padding, the embedding table, averaging only the real tokens, and the bag of embeddings with what it cannot see.
  2. Self-Attention - Built From Matmuls - attention as a weighted average, queries, keys and values, the square-root scale, the padding and causal masks, why attention alone ignores order, the check against PyTorch, and the cost of a square matrix.
  3. The Transformer Block - multi-head attention, position embeddings, layer normalization, the feed-forward network and the residual connection, the pre-norm block, and a transformer encoder against the bag on Banking77.

By the end you will be able to:

  • Turn raw queries into padded token tensors with a vocabulary built from the training split, and say what truncation, padding and an unknown word cost.
  • Compute attention by hand and through one function, state every intermediate shape, and mask padding and the future so neither leaks into an output.
  • Assemble a transformer block from attention and the components Modules 4 and 5 supplied, and prove it matches nn.TransformerEncoderLayer.
  • Compare a transformer with a bag of embeddings on the same data and recipe, and choose between them on validation queries.