Section 4.4: Regularization and the Playbook

A network with enough capacity drives its training loss toward zero by memorizing its rows, and its validation loss turns upward while it does. Regularization, Lesson 3.4.1's cost on large weights widened here to every tool that trades a little training loss for a lower validation loss, is the answer, and there are three: weight decay (a shrink of every parameter at each step), dropout (a layer that zeros a random share of its inputs in training mode and passes everything through in evaluation mode) and early stopping (stopping once the validation loss has gone a set number of epochs without a new low, and keeping the parameters of its lowest epoch). The point of this section is not the three tools but the measuring: each is applied alone against one overfit baseline, a 512-unit network that memorizes 200 listings, read by the same three numbers, and only then combined. Then the parameters are saved to a file, loaded in another process and scored again.

Section 4.3 got a network training well, which means it can now overfit properly, and overfitting is a capability before it is an accident. Module 3 met this gap with cross-validation and simpler models, Lessons 3.6.1 and 3.6.2, and this section meets it with three tools measured one at a time. The failures here are quiet. A validation pass run in training mode prints a dropout network's loss higher than evaluation mode does, a quarter higher on average on the full rows, and the gap looks exactly like overfitting, so a day goes into regularizing a bug. Two tools applied at once are credited together. A checkpoint saved every epoch without asking whether the model improved keeps the last model and loses the best, the act Module 0's first trainer called "latest wins". None of them stops the program.

Lessons

  1. Overfitting on Purpose - Capacity and Weight Decay - memorizing a small slice on purpose, the capacity ladder read on it, the gap on the full rows, weight decay as the shrink in the step, and measuring one tool alone.
  2. Dropout and Early Stopping - dropout's two modes, dropout measured on the slice, validation in training mode that looks like overfitting, early stopping at the epoch of the minimum, keeping the best parameters, and the tools together.
  3. Saving and Loading Models - state_dict as the model's memory, torch.save and torch.load, loading in a second process, a checkpoint with its epoch and optimizer, the checkpoint that overwrites the best, and what not to save.

By the end you will be able to:

  • Make a network memorize a small slice of the listings on purpose, read the capacity ladder, and state the gap in the three numbers every tool is measured by.
  • Apply weight decay, dropout and early stopping one at a time against the same baseline, attribute each effect from the validation loss, and combine them only against the single measurements.
  • Explain what dropout does in training mode and in evaluation mode, and name the loss a validation pass in training mode produces as a bug and not a phenomenon.
  • Save a model's parameters, restore them in a separate process to the same weights and validation score, write a checkpoint that resumes a run, and say why a checkpoint saved every epoch loses the best model.