One Neuron - Fit by Hand

Context

Gradient descent is Section 4.1's step repeated: predict, measure the loss, run backward(), move every parameter a small share of its gradient the other way inside torch.no_grad(), clear the gradients, and go again until the loss stops falling. Run on one neuron, a weight times the standardized area plus a bias, it reaches the two-parameter price model least squares computes in one solve, with no grid anywhere: the search Lesson 3.1.3 said Module 4 would run, because a scan is hopeless for thousands of parameters.

Every model from here to the end of Module 6 is fitted by this loop: the classifier of Section 4.3, the model Section 4.4 overfits on purpose, the image network of Section 4.5 with its thousands of parameters. Lesson 4.2.2 replaces each hand-written piece with nn.Linear, nn.MSELoss and optim.SGD, so from the next lesson on you read any PyTorch training script as these five steps, and Lesson 3.2.1's SGDRegressor was this loop inside a library.

Lesson 4.1.4 took one step from w = 3000 and showed .grad doubling when nothing cleared it between two backward() calls. This lesson repeats that step until least squares' answer comes out, checks the answer to rtol=1e-4, and breaks the loop on purpose along the way, on the raw area, with learning rates that crawl, stand still and explode, and with its fifth step missing, so that each failure is known by what the loss prints, because nothing in the loop reports one.

Goal

• Write the five-step cycle on two scalar leaf tensors and run it as a loop over epochs, printing the loss every tenth epoch and the parameters in whole units.
• Standardize the input with its mean and standard deviation kept as tensors, and explain from the runs why the raw area overflows at one learning rate and crawls at another.
• Choose a learning rate by what the loss prints, telling a crawl, a fit, a landing in one step, a mirror, a standstill and an explosion apart.
• Diagnose a loop whose gradients are never cleared from the numbers it prints, with no error and no warning to go on.
• Read a fitted weight back from per standard deviation to per square meter, check the fit against np.linalg.lstsq with np.allclose, and price a new listing through the kept statistics.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want the loop drawn step by step on a toy problem, or PyTorch's own first version of it, before you write yours.

• YouTube - "Gradient Descent, Step-by-Step" by StatQuest with Josh Starmer. A line fitted to a handful of points by the loop this lesson writes: the sum of squared residuals as the loss, its derivative with respect to the intercept and then the slope, a step of the derivative times a learning rate, and the steps repeated until they shrink to nothing, with the learning rate's effect on the step size drawn out.
• Official docs - PyTorch Tutorials, "Learning PyTorch with Examples", the sections "PyTorch: Tensors" and "PyTorch: Tensors and autograd". A third-order polynomial fitted to the sine function by a loop written out in full, first with the gradient computed by hand on tensors and then by backward(), with the update inside torch.no_grad() and each .grad set to None after it, which is this lesson's five steps on a different model.

Table of Contents

• The Neuron and the Five Steps
• Standardize the Input First
• The Loop, Epoch by Epoch
• The Learning Rate Decides Everything
• The Loop Without the Zeroing
• Reading the Fit Back in Square Meters

Theory

Every Theory block runs in ~/dev/bootcamp on the 2,875 listings with an area, Lesson 3.1.3's rows and Section 4.1's, held as float32 tensors, the dtype the module trains in. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

The Neuron and the Five Steps

What. A neuron (one unit that computes a weighted sum of its inputs plus a bias) is Lesson 2.3.1's w . x + b with one input, the area, and two parameters, w and b, held as scalar leaf tensors. Gradient descent (fitting by repeating a step against the gradient) runs Section 4.1's lines as one cycle: predict, measure the loss, backward(), step inside torch.no_grad(), clear the gradients. Why. Lesson 3.1.3 found the bottom of mse(w) by trying 6,001 candidates and said a scan is hopeless for thousands of parameters. The cycle finds the bottom by following the gradient, and costs the same five lines whether the model has two parameters or two million. Lesson 3.2.1's SGDRegressor was this cycle inside a library, and from here on you write it. When. Every time a model is fitted in Modules 4 to 6. A network (many neurons arranged in layers) is Lesson 4.3.1's, and its loop is this one with more parameters inside the step.

The five steps, written once, from Lesson 4.1.4's starting point and at its rate, so the numbers can be checked against it:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
w = torch.tensor(3000.0, requires_grad=True)
b = torch.tensor(50000.0, requires_grad=True)

pred = w * area + b                  # 1 predict: the neuron
loss = ((price - pred) ** 2).mean()  # 2 measure the loss
loss.backward()                      # 3 the gradient into .grad
with torch.no_grad():                # 4 the step, off the record
    w -= 1e-5 * w.grad
    b -= 1e-5 * b.grad
w.grad, b.grad = None, None          # 5 clear before the next pass

Step 1 is Lesson 4.1.2's X @ w + b in one dimension, step 2 is Lesson 3.1.3's mean squared error as Lesson 4.1.4 wrote it, and steps 3, 4 and 5 are 4.1.4's backward(), its in-place step under no_grad and its None. Nothing here is new except that the five lines now sit together, in this order, as one unit of work:

print(f"{loss.item():.4e}")                # 1.8494e+10
print(f"{w.item():,.2f} {b.item():,.2f}")  # 3,077.07 50,000.66
print(w.is_leaf, w.grad)                   # True None
with torch.no_grad():
    after = ((price - (w * area + b)) ** 2).mean()
print(f"{after.item():.4e}")               # 1.7969e+10

Every number is Lesson 4.1.4's, the loss its 18,493,503,179 at the digits .4e shows, and the None in .grad is the state the next cycle starts from. The cycle has no notion of done: it runs once, and the fit is the cycle run until the numbers stop moving.

DON'T:

• DON'T move the clearing line before the step. With w.grad, b.grad = None, None between backward() and the step, the step multiplies 1e-5 by None and raises TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'. The order is the cycle: the gradient is used, then cleared.
• DON'T read one cycle as a fit. The step from 3,000 covered 77 of the 288 units to least squares' 3,288.31 per square meter and left the loss at 1.7969e+10 against a floor of 1.7226e+10.

Standardize the Input First

What. Standardizing the area, Lesson 3.2.1's StandardScaler by hand, is z = (area - mean) / sd with both statistics kept as tensors, sd computed with correction=0 so it is the scaler's number, and the same two tensors applied to every listing the model ever prices. Why. Lesson 3.2.1 said a model fitted by steps cares about scale, and this subtitle shows what the sentence means. When. Before the first epoch (one pass of the loop over every row it fits) of every loop in this module, on every input, with the statistics computed on the rows the loop fits and reused on every other row.

The cycle repeated on the raw area from w = 0, b = 0 at a learning rate of 0.1, one epoch per cycle, with the loss and w printed every epoch:

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(1, 7):
    loss = ((price - (w * area + b)) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad
        b -= 0.1 * b.grad
    w.grad, b.grad = None, None
    print(epoch, f"{loss.item():.4e}", f"{w.item():.2e}")
print(torch.finfo(torch.float32).max)

1 1.8286e+11 8.81e+06
2 9.1076e+17 -2.07e+10
3 5.0228e+24 4.86e+13
4 2.7701e+31 -1.14e+17
5 inf 2.68e+20
6 inf -6.29e+23
3.4028234663852886e+38

w changes sign every epoch and grows about 2,350 times each time, Lesson 4.1.3's 1e-4 step repeated: each step crosses the bottom and lands farther from it than it started, and the next starts from there. By epoch 5 the squared residuals pass 3.4028e+38, the largest number float32 holds, which is the inf. No error, no warning, and nothing stops the loop.

A rate the weight survives, 4e-5, just under the rate Lesson 4.1.3 computed for one step on w alone, with the loop allowed 100,000 epochs and printed at seven of them:

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(1, 100_001):
    loss = ((price - (w * area + b)) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 4e-5 * w.grad
        b -= 4e-5 * b.grad
    w.grad, b.grad = None, None
    if epoch in (1, 2, 10, 200, 1_000, 10_000, 100_000):
        print(epoch, f"{loss.item():.4e}", f"{w.item():,.2f}", f"{b.item():,.2f}")

1 1.8286e+11 3,523.45 30.43
2 1.8319e+10 3,735.70 32.94
10 1.7720e+10 3,749.26 38.84
200 1.7717e+10 3,748.10 176.01
1000 1.7707e+10 3,743.26 749.79
10000 1.7605e+10 3,692.23 6,803.56
100000 1.7261e+10 3,411.22 40,137.78

w is fitted in ten epochs: 3,749.26 at epoch 10 is within a unit of Lesson 3.1.3's bottom for the model with no intercept, 3,749.58, and the loss 1.7720e+10 is that model's floor, Lesson 4.1.3's 17,720,310,796, at the five digits the print shows. b is not. It moved 30 in the first epoch, 2.5 in the second and under one unit an epoch from then on, while w drifts down from its no-intercept bottom toward the one that goes with a fitted intercept. Least squares says where both are heading:

area_np = has_area["area_sqm"].to_numpy()
price_np = has_area["price"].to_numpy()
X = np.column_stack([np.ones(len(area_np)), area_np])
coef, ssr, rank, sv = np.linalg.lstsq(X, price_np)
print(np.round(coef, 2))  # [54716.9   3288.31]
# 1.7226e+10
print(f"{np.mean((price_np - X @ coef) ** 2):.4e}")

After 100,000 epochs the loop is 14,579 short on b, 123 over on w, and its loss 1.7261e+10 differs from the floor in its fourth digit. The two parameters do not take the same rate on the raw area: the one w survives moves b by a fraction of a unit an epoch, and the block below removes the cause by centering and scaling the input, after which the next subtitle's loop settles both in 70 epochs. The standardized input:

mean = area.mean()
sd = area.std(correction=0)
z = (area - mean) / sd
print(mean, sd)  # tensor(99.0200) tensor(44.0589)
print(z[:4])     # tensor([ 1.7404, -1.5552,  1.0118, -1.2942])
# 0.0 1.0
print(round(z.mean().item(), 4), round(z.std(correction=0).item(), 4))

The loss on z is Lesson 3.1.3's loss with its axis renamed and shifted, so every pair w, b on z is one pair on the raw area and the bottom has the same height, 1.7226e+10, which the next subtitle reaches.

DON'T:

• DON'T answer the crawl with a larger rate. The same raw loop at 1e-4 prints 3.1843e+11 at epoch 2 and 5.8343e+18 at epoch 30, growing every epoch, and at 0.1 it was inf by epoch 5.
• DON'T compute sd with area.std(). Its default correction=1 gives tensor(44.0666), 4.1.2's difference from NumPy, so a z built with it is not StandardScaler's, and one listing gets two different z values from a pipeline's scaler and from this loop. correction=0 keeps the two in step.

The Loop, Epoch by Epoch

What. The loop is the five steps inside for epoch in range(...), one epoch per cycle, because the loss is computed on the full batch (every row the loop fits, used in one step). It prints the loss as f"{loss.item():.4e}" every tenth epoch with the parameters rounded to whole units, and the fit is checked once, against least squares. Why. The printed loss is the loop's only instrument. One that falls and settles is a fit, one still falling at the end is a rate too small, one growing is a rate too large, and one bouncing is the fifth subtitle's failure. Least squares is the one model with an exact answer to compare against, so the loop is checked against it once here, and trusted on the models of Section 4.3 that have none. When. Every model in this module, with the loop's body unchanged from here to Section 4.5. Mini-batches, Lesson 4.3.3's, change how many steps an epoch holds and which rows each step's gradient comes from, and nothing in the five lines.

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
mean = area.mean()
sd = area.std(correction=0)
z = (area - mean) / sd

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
lr = 0.1
for epoch in range(1, 101):
    pred = w * z + b
    loss = ((price - pred) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= lr * w.grad
        b -= lr * b.grad
    w.grad, b.grad = None, None
    if epoch <= 3 or epoch % 10 == 0:
        print(epoch, f"{loss.item():.4e}", round(w.item()), round(b.item()))

1 1.8286e+11 28976 76065
2 1.2323e+11 52157 136917
3 8.5071e+10 70701 185599
10 2.0209e+10 129323 339489
20 1.7260e+10 143209 375941
30 1.7226e+10 144700 379855
40 1.7226e+10 144860 380275
50 1.7226e+10 144878 380320
60 1.7226e+10 144879 380325
70 1.7226e+10 144880 380326
80 1.7226e+10 144880 380326
90 1.7226e+10 144880 380326
100 1.7226e+10 144880 380326

The loss printed beside an epoch is the one that epoch's step was taken from, and w and b are where the step landed. The loss stops changing to four digits at epoch 30, 1.7226e+10, the floor least squares reached on the raw area, and the parameters keep moving until epoch 70, where they stay: a loop is settled when its printed parameters stop changing, and the loss reaches its printed floor forty epochs before that. Least squares on z says whether that is the answer:

price_np = has_area["price"].to_numpy()
Z = np.column_stack([np.ones(len(z)), z.numpy().astype(np.float64)])
coef, ssr, rank, sv = np.linalg.lstsq(Z, price_np)
print(np.round(coef, 2))                   # [380325.91 144879.63]
print(f"{b.item():,.2f} {w.item():,.2f}")  # 380,325.84 144,879.59
# True
print(np.allclose([b.item(), w.item()], coef, rtol=1e-4))
# 1.7226e+10, least squares' mean squared error
print(f"{np.mean((price_np - Z @ coef) ** 2):.4e}")
with torch.no_grad():
    final = ((price - (w * z + b)) ** 2).mean()
print(f"{final.item():.4e}")               # 1.7226e+10

The comparison is never ==, because the loop stops where float32's resolution lets it, five or six digits in, and rtol=1e-4 is the module's margin for a float32 number against its float64 value. Seventy steps reached the answer lstsq computes in one solve, by the search without a grid Lesson 3.1.3 promised, and the same five lines fit a network, where no solve exists and a grid is hopeless.

DON'T:

• DON'T read a small loss as a settled loop. At epoch 30 w was still 180 short of where it stops, and the parameters say when the loop is done.
• DON'T check the fit with ==. The loop's 144,879.59 against least squares' 144,879.63 is False under == and True under np.allclose at rtol=1e-4, and the gap between them is float32's, not the loop's.

The Learning Rate Decides Everything

What. The learning rate multiplies the gradient into the step, Lesson 4.1.3's share, and on z six values of it give six different loops from the same zero start: a crawl, a fit, a landing in one step, a mirror of the fit, a standstill and an explosion. Why. Nothing in the loop checks the rate, and every one of the six prints a loss at every epoch without a complaint. When. At the start of every loop, by watching what the first twenty epochs print, and again whenever the input, the loss or the model changes, because the rate that fits one does not carry to the next. Lesson 4.3.4 turns the choice into a sweep.

The loop as a function, so six rates run from one start, each returning its losses and its w after every epoch:

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
z = (area - area.mean()) / area.std(correction=0)

def descend(lr, epochs):
    w = torch.tensor(0.0, requires_grad=True)
    b = torch.tensor(0.0, requires_grad=True)
    losses, ws = [], []
    for epoch in range(epochs):
        loss = ((price - (w * z + b)) ** 2).mean()
        loss.backward()
        with torch.no_grad():
            w -= lr * w.grad
            b -= lr * b.grad
        w.grad, b.grad = None, None
        losses.append(loss.item())
        ws.append(w.item())
    return losses, ws

print("lr    epoch 1    epoch 2    epoch 10   epoch 20   w after 20")
for lr in [0.01, 0.1, 0.5, 0.9, 1.0, 1.1]:
    losses, ws = descend(lr, 20)
    print(f"{lr:<5} {losses[0]:.4e} {losses[1]:.4e} {losses[9]:.4e} {losses[19]:.4e} {ws[-1]:,.0f}")

lr    epoch 1    epoch 2    epoch 10   epoch 20   w after 20
0.01  1.8286e+11 1.7630e+11 1.3237e+11 9.4094e+10 48,157
0.1   1.8286e+11 1.2323e+11 2.0209e+10 1.7260e+10 143,209
0.5   1.8286e+11 1.7226e+10 1.7226e+10 1.7226e+10 144,880
0.9   1.8286e+11 1.2323e+11 2.0209e+10 1.7260e+10 143,209
1.0   1.8286e+11 1.8286e+11 1.8286e+11 1.8286e+11 0
1.1   1.8286e+11 2.5574e+11 4.4271e+12 1.6908e+14 -5,409,459

The road each took, as w after the first three epochs beside the ratio of the second distance to the bottom over the first:

Z = np.column_stack([np.ones(len(z)), z.numpy().astype(np.float64)])
bottom = np.linalg.lstsq(Z, has_area["price"].to_numpy())[0][1]
for lr in [0.01, 0.1, 0.9, 1.0, 1.1]:
    losses, ws = descend(lr, 3)
    gaps = [bottom - w for w in ws]
    print(lr, [f"{w:,.0f}" for w in ws], round(gaps[1] / gaps[0], 2))

0.01 ['2,898', '5,737', '8,520'] 0.98
0.1 ['28,976', '52,157', '70,701'] 0.8
0.9 ['260,783', '52,157', '219,058'] -0.8
1.0 ['289,759', '0', '289,759'] -1.0
1.1 ['318,735', '-63,747', '395,232'] -1.2

One step on z multiplies the distance between w and the bottom, 144,879.63, by 1 - 2 * lr, the ratio in the last column, so 0.1 keeps four fifths of the distance each epoch and settles, 0.01 keeps 98 percent and crawls, and 0.5 keeps nothing, the one-step landing. Above 0.5 the number is negative and the step crosses the bottom. At 0.9 it lands 115,904 past the bottom, at 260,783, exactly as far as 0.1's first step stopped short of it, at 28,976, so the two loops are mirror images with the same losses and w on opposite sides, and 0.9's second step lands on 0.1's second landing, 52,157, a mirror of a mirror. At 1.0 the step crosses to the same distance on the other side every time, twice the answer and then the start, and the loss, which depends only on the distance, prints the same number forever. At 1.1 each step lands farther out than it started, and the loss grows without end. Why the landing lies at 0.5, Lesson 4.1.3's one-step rate computed on z:

print((z ** 2).mean())  # tensor(1.0000)
# 0.5000
print(f"{1 / (2 * (z ** 2).mean().item()):.4f}")

That rate is exact for this one feature on these 2,875 rows, where mean(z ** 2) is 1 by construction, and nowhere else: the Assignment's two features, run at 0.5, reach their own floor 1.5725e+10 in five steps rather than one, and a handful of rows has whatever mean(z ** 2) those rows make. So 0.1 is the rate every loop in this section uses, settled by epoch 70 with a factor of ten in hand below the rate that stands still, and 0.5 is shown once.

DON'T:

• DON'T take identical losses for identical loops. 0.9 printed 0.1's losses while its w crossed the bottom at every step, and a tenth more rate stands still forever.
• DON'T set a rate and look away. 0.01 prints a loss still falling at epoch 20, 1.1 prints one still growing, and the loop raises nothing for either.
• DON'T carry 0.5 anywhere. It lands in one step only on this feature and these rows, and this section's rate is 0.1.

The Loop Without the Zeroing

What. The fifth step left out: w.grad, b.grad = None, None gone, and nothing else changed. .grad then holds the sum of every gradient since the first epoch, Lesson 4.1.4's accumulation, and every step moves by that sum. Why. This is the failure that is hardest to see, because nothing fails. The loss prints every epoch, the program exits with code 0, no warning appears, and at every rate where the zeroed loop settles, the numbers bounce and never settle. Knowing what that looks like is the only way to recognize it, in your own loop or in anyone else's. When. Whenever a loss bounces up and down by factors of ten and never settles whatever the rate, check the fifth step before anything else.

The loop of the third subtitle with one line removed, in no_zero.py, printing the loss, w and what w.grad holds after the step:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
z = (area - area.mean()) / area.std(correction=0)

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(1, 31):
    loss = ((price - (w * z + b)) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad
        b -= 0.1 * b.grad
    if epoch <= 2 or epoch % 5 == 0:
        print(epoch, f"{loss.item():.4e}", round(w.item()), f"{w.grad.item():.4e}")

(bootcamp) goku@machine:~/dev/bootcamp$ python3 no_zero.py
1 1.8286e+11 28976 -2.8976e+05
2 1.2323e+11 81133 -5.2157e+05
5 5.1426e+10 262221 -5.1508e+05
10 4.7213e+10 141403 6.5122e+05
15 1.8034e+11 31935 -3.0828e+05
20 1.3110e+11 291155 -2.6146e+05
25 1.7744e+10 72886 6.3885e+05
30 1.1250e+11 89628 -5.4625e+05
(bootcamp) goku@machine:~/dev/bootcamp$ echo $?
0

Epoch 1 is the zeroed loop's epoch 1: .grad was None when the loop began, the first backward() created it, and the first step lands at 28,976. At epoch 2, .grad holds -5.2157e+05, the first epoch's -2.8976e+05 plus the second epoch's own gradient, so the step moves w by 52,157 instead of 23,181 and lands at 81,133 where the zeroed loop landed at 52,157. From there the sum carries w back and forth across the bottom, and the loss comes three percent above the floor at epoch 25 and leaves at once, because the sum that brought w there is nowhere near zero when w is. A rate a hundredth of the size changes the speed and nothing else: with 0.001 in both steps and range(1, 301), the same script prints 2.8363e+10 at epoch 100, 1.4456e+11 at 200 and 9.2903e+10 at 300, still bouncing.

The evidence is in the numbers: a loss that falls, rises past its own start and falls again, at the rate the zeroed loop settled at, beside a .grad whose size does not shrink as the loss does. The fix is the one line.

DON'T:

• DON'T look for this failure in the error output. There is none, at any rate.
• DON'T lower the rate when a loss bounces. A loss that bounces at every rate where the zeroed loop settles is the fifth step missing, and a rate too large looks different: it grows every epoch, as 1.1 did.

Reading the Fit Back in Square Meters

What. On z, w is a price per standard deviation of area and b is the price at z = 0, the average area. w / sd is the price per square meter, Lesson 4.1.3's chain rule, b - w * mean / sd is the intercept on the raw area, and a new listing is priced through the same mean and sd the loop fitted on. Why. A price per standard deviation answers no one's question, and a listing site asks the price of 85 square meters, not of -0.3182 standard deviations. The two kept tensors make the fitted numbers readable in the units the data came in, and they make a prediction on a new row the model the loop fitted rather than a different one. When. After every fit on a standardized input, when a weight is reported or compared with a model fitted on raw features, and on every prediction, where the statistics are the fit rows' and never the new rows' own.

import numpy as np
import pandas as pd
import torch
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
mean = area.mean()
sd = area.std(correction=0)
z = (area - mean) / sd

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(100):
    loss = ((price - (w * z + b)) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad
        b -= 0.1 * b.grad
    w.grad, b.grad = None, None

with torch.no_grad():
    per_sqm = w / sd
    intercept = b - w * mean / sd
print(f"{per_sqm.item():,.2f}")  # 3,288.31
print(round(intercept.item()))   # 54717
# 380326 380326
print(round(b.item()), round(price.mean().item()))

The conversions run under no_grad because w / sd on a leaf that requires a gradient would otherwise carry a grad_fn, a graph nothing will walk. b is the mean price because z has mean zero: at the average area the model predicts the average price. A new listing goes through the kept statistics:

new_area = torch.tensor([85.0])
with torch.no_grad():
    new_z = (new_area - mean) / sd
    print(new_z)                       # tensor([-0.3182])
    print(torch.round(w * new_z + b))  # tensor([334224.])
model = LinearRegression().fit(has_area[["area_sqm"]], has_area["price"])
# 3288.31 54717
print(round(model.coef_[0], 2), round(model.intercept_))
# [334224.]
print(np.round(model.predict(pd.DataFrame({"area_sqm": [85.0]}))))

334,224 is what Lesson 3.1.1's LinearRegression predicted for the same listing. The statistics are not recomputed on the rows being priced:

batch = torch.tensor([60.0, 85.0, 140.0])
with torch.no_grad():
    kept = w * (batch - mean) / sd + b
    own = w * (batch - batch.mean()) / batch.std(correction=0) + b
print(torch.round(kept))  # tensor([252016., 334224., 515081.])
print(torch.round(own))   # tensor([228581., 336970., 575426.])
one = torch.tensor([85.0])
with torch.no_grad():
    # tensor([nan])
    print(w * (one - one.mean()) / one.std(correction=0) + b)

Three new listings standardized by their own mean and standard deviation land on z values that mean nothing to this w and b, and their prices come out 23,435 low, 2,746 high and 60,345 high, while one listing standardized by itself divides zero by zero.

DON'T:

• DON'T report w as a price per square meter, and don't divide it by a printed sd either: 44.06 instead of the tensor gives 3,288.23, off by eight hundredths, as Lesson 4.1.3 showed.
• DON'T standardize the rows you predict by their own statistics. mean and sd are computed once, on the rows the loop fitted, and travel with the model.

Lab Exercises

Lab Exercise 1: Rooms, Standardized and Fitted by Hand

In rooms_fit.py, load listings.csv and build float32 tensors of every listing's rooms and price, all 3,000 rows, since no listing lacks a room count, and print the rooms tensor's shape and dtype. Run the five-step loop on the raw room count from w = 0, b = 0 at a learning rate of 0.1 for 10 epochs, printing every epoch's number, its loss as .4e and w in whole units with a thousands separator. Print the one-step rate for this input, 1 / (2 * mean(rooms ** 2)), to four decimals, and say in a comment what the ten lines show and how the rate explains it.

Standardize the rooms with their mean and standard deviation kept as tensors, correction=0, and print both rounded. Run the same loop on the standardized rooms for 100 epochs, printing the loss and both parameters in whole units every tenth epoch. Check the two parameters against np.linalg.lstsq on a column of ones beside the standardized rooms with np.allclose at rtol=1e-4, print least squares' mean squared error as .4e to compare with the loop's last loss, read the weight back per room to two decimals, and print b beside the mean price, both in whole units.

Lab Exercise 2: Four Rates, a Missing Line, and a Three-Room Listing

In rooms_rates.py, on the standardized rooms of Lab 1, write descend(lr, epochs, clear=True) that runs the five-step loop from zero and returns w, b and the list of every epoch's loss as Python floats, with the fifth step skipped when clear is False. Run it for 20 epochs at 0.01, 0.1, 0.5 and 1.0 and print one line per rate under a header, holding the loss at epochs 1, 2, 10 and 20 and w after the last epoch in whole units, and say in a comment which rate crawls, which fits, which lands in one step and which stands still.

Run it at 0.1 for 20 epochs with clear=False, print the loss at epochs 1, 2, 5, 10, 15 and 20, then w and what w.grad holds at the end, and say in a comment why the losses bounce. Finally run the clearing loop at 0.1 for 100 epochs and read the fit back: the weight per room, the intercept on the raw room count, both in whole units, and the price of a three-room listing through the kept statistics. Fit LinearRegression on rooms and print its weight, its intercept and its prediction for the same listing, to confirm all three.

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
rooms = torch.tensor(listings["rooms"].to_numpy(), dtype=torch.float32)
price = torch.tensor(listings["price"].to_numpy(), dtype=torch.float32)
print(rooms.shape, rooms.dtype)  # torch.Size([3000]) torch.float32

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(1, 11):
    loss = ((price - (w * rooms + b)) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad
        b -= 0.1 * b.grad
    w.grad, b.grad = None, None
    print(epoch, f"{loss.item():.4e}", f"{w.item():,.0f}")
print(f"{1 / (2 * (rooms ** 2).mean().item()):.4f}")
# The loss grows about five times an epoch and w changes sign every
# epoch: 0.1 is three times the one-step rate for the raw room count,
# past twice it, so every step lands farther out than it started.

1 1.8311e+11 317,147
2 8.7174e+11 -412,830
3 4.4988e+12 1,260,316
4 2.3601e+13 -2,581,392
5 1.2421e+14 6,233,013
6 6.5405e+14 -13,997,039
7 3.4445e+15 32,427,120
8 1.8141e+16 -74,113,448
9 9.5539e+16 170,384,896
10 5.0316e+17 -390,716,096
0.0319

mean = rooms.mean()
sd = rooms.std(correction=0)
zr = (rooms - mean) / sd
# 3.57 1.7084
print(round(mean.item(), 2), round(sd.item(), 4))

w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(1, 101):
    loss = ((price - (w * zr + b)) ** 2).mean()
    loss.backward()
    with torch.no_grad():
        w -= 0.1 * w.grad
        b -= 0.1 * b.grad
    w.grad, b.grad = None, None
    if epoch % 10 == 0:
        print(epoch, f"{loss.item():.4e}", round(w.item()), round(b.item()))

10 2.3397e+10 118629 339879
20 2.0501e+10 131367 376373
30 2.0468e+10 132735 380292
40 2.0467e+10 132882 380712
50 2.0467e+10 132897 380758
60 2.0467e+10 132899 380762
70 2.0467e+10 132899 380763
80 2.0467e+10 132899 380763
90 2.0467e+10 132899 380763
100 2.0467e+10 132899 380763

price_np = listings["price"].to_numpy()
Z = np.column_stack([np.ones(len(zr)), zr.numpy().astype(np.float64)])
coef, ssr, rank, sv = np.linalg.lstsq(Z, price_np)
print(np.round(coef, 2))              # [380763.   132899.24]
# True
print(np.allclose([b.item(), w.item()], coef, rtol=1e-4))
# 2.0467e+10, the loop's last loss
print(f"{np.mean((price_np - Z @ coef) ** 2):.4e}")
with torch.no_grad():
    print(f"{(w / sd).item():,.2f}")  # 77,791.18
# 380763 380763
print(round(b.item()), round(price.mean().item()))

import numpy as np
import pandas as pd
import torch
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
rooms = torch.tensor(listings["rooms"].to_numpy(), dtype=torch.float32)
price = torch.tensor(listings["price"].to_numpy(), dtype=torch.float32)
mean = rooms.mean()
sd = rooms.std(correction=0)
zr = (rooms - mean) / sd

def descend(lr, epochs, clear=True):
    w = torch.tensor(0.0, requires_grad=True)
    b = torch.tensor(0.0, requires_grad=True)
    losses = []
    for epoch in range(epochs):
        loss = ((price - (w * zr + b)) ** 2).mean()
        loss.backward()
        with torch.no_grad():
            w -= lr * w.grad
            b -= lr * b.grad
        if clear:
            w.grad, b.grad = None, None
        losses.append(loss.item())
    return w, b, losses

print("lr    epoch 1    epoch 2    epoch 10   epoch 20   w after 20")
for lr in [0.01, 0.1, 0.5, 1.0]:
    w, b, losses = descend(lr, 20)
    print(f"{lr:<5} {losses[0]:.4e} {losses[1]:.4e} {losses[9]:.4e} {losses[19]:.4e} {w.item():,.0f}")
# 0.01 crawls, 0.1 fits, 0.5 lands in one step, since mean(zr ** 2)
# is 1 here too, and 1.0 stands still, jumping across the bottom.

lr    epoch 1    epoch 2    epoch 10   epoch 20   w after 20
0.01  1.8311e+11 1.7667e+11 1.3353e+11 9.5946e+10 44,175
0.1   1.8311e+11 1.2456e+11 2.3397e+10 2.0501e+10 131,367
0.5   1.8311e+11 2.0467e+10 2.0467e+10 2.0467e+10 132,899
1.0   1.8311e+11 1.8311e+11 1.8311e+11 1.8311e+11 1

w, b, losses = descend(0.1, 20, clear=False)
for epoch in [1, 2, 5, 10, 15, 20]:
    print(epoch, f"{losses[epoch - 1]:.4e}")
print(round(w.item()), f"{w.grad.item():.4e}")
# Without the clearing, .grad holds the sum of every gradient so far
# and each step moves by that sum, so w overshoots the bottom at
# 132,899, swings back past it, and the loss bounces instead of settling.

1 1.8311e+11
2 1.2456e+11
5 5.4050e+10
10 4.9913e+10
15 1.8063e+11
20 1.3229e+11
267079 -2.3984e+05

w, b, losses = descend(0.1, 100)
with torch.no_grad():
    per_room = w / sd
    intercept = b - w * mean / sd
    three = (torch.tensor([3.0]) - mean) / sd
    print(round(per_room.item()))      # 77791
    print(round(intercept.item()))     # 103178
    print(torch.round(w * three + b))  # tensor([336552.])
model = LinearRegression().fit(listings[["rooms"]], listings["price"])
# 77791 103178
print(round(model.coef_[0]), round(model.intercept_))
# [336552.]
print(np.round(model.predict(pd.DataFrame({"rooms": [3.0]}))))

Theory Assessment

Q1: The Raw Input and the Standardized One

Explain, from the two raw-area runs, why no single learning rate serves both w and b on the raw area, and what standardizing the area changes about the loss and what it leaves exactly where it was. Then say why epoch 1 prints 1.8286e+11 at every learning rate, on the raw area and on z alike, and what the w and b on that same line are.

Hint: For the first part, ask what each parameter's step was at the rate w survived, and write w * z + b with z expanded to see what a pair on z is on the raw area. For the last part, ask which loss an epoch's line prints and which parameters.

Q2: What the Printout Says

A loop on z at 0.1 prints 1.7226e+10 at epochs 30, 40, 50 and 60 while w prints 144,700, 144,860, 144,878 and 144,879. Say when the loop is settled, how the answer is checked, and why the check is not ==. Then explain why 0.5 reaches the same answer in one step, why that rate is exact here and nowhere else, and why the section's loops use 0.1 anyway.

Hint: Near the bottom the loss changes by far less than the parameters do. For 0.5, ask what mean(z ** 2) is on a standardized column and what the one-step rate of Lesson 4.1.3 becomes.

Q3: One Line Missing, One Unit Wrong

A loop prints 5.1426e+10 at epoch 5, 4.7213e+10 at 10, 1.8034e+11 at 15 and 1.7744e+10 at 25, exits with code 0 and warns about nothing. Say what w.grad holds after the second epoch's backward() in that loop, why the loss bounces rather than grows, and why lowering the rate does not help. Then say what a fitted w of 144,880 on z is a price per, how it becomes 3,288.31 per square meter, and why b equals the mean price.

Hint: After the second backward(), .grad is two gradients, not one. For the units, write w * z + b with z expanded and read off the coefficient of area.

Q4: Twenty Identical Losses

At a learning rate of 1.0 the loop on z prints 1.8286e+11 at every one of twenty epochs and w prints 0 at the end. A teammate concludes that the gradient was zero from the start, so the loop never moved, and proposes a larger rate. Using this lesson's numbers, say where w stood after epochs 1, 2 and 3, why the loss is the same at every epoch, what a larger rate does from the same start, and which rates reach the answer.

Hint: The printed loss at an epoch is the loss its step started from, and one step on z multiplies the distance to the bottom by 1 - 2 * lr. Ask what that number is at 1.0, and what the loss is at two points the same distance from the bottom.

Assignment

Assignment: Two Features, and a Loop That Stops Itself

Your team's pricing model takes the area and the distance to the center, Lesson 3.1.1's two features, and nobody wants to guess how many epochs it needs: the loop should run until the loss stops moving and say when that was. Write two_features.py on the 2,875 listings with an area.

Build X as a (2875, 2) float32 tensor of area_sqm and distance_km, standardize each column with the column means and standard deviations from X.mean(dim=0) and X.std(dim=0, correction=0), kept as tensors, and make w a leaf of two zeros with torch.zeros(2, requires_grad=True) and b a scalar zero. The prediction is Z @ w + b, and with w one-dimensional the product is one-dimensional, the shape price has, which you confirm by printing both shapes.

The loop gets a stopping rule this lesson did not use: instead of a fixed epoch count, it keeps the previous epoch's loss as a Python float and stops when the loss has fallen by less than one part in a million of itself, previous - loss.item() < 1e-6 * loss.item(). Start previous at float("inf") so the first epoch never stops. Write the loop as descend(epochs=None), which applies the rule when epochs is None and runs a fixed count otherwise, returning w, b, the epoch it ended at and the last loss. Fit LinearRegression on the same two columns and print coef_ rounded to whole units and intercept_ rounded, then run descend() under the rule, print the epoch it stopped at with the loss as .4e, the weights read back per square meter and per kilometer, w / sds, rounded to whole units, and np.allclose of those against coef_ at rtol=1e-4.

Whatever that printed, run descend(100) and print the same three lines again, plus the intercept on the raw features, b - (w * means / sds).sum(), rounded, and say in a comment what the stopping rule measured and what it missed. Finally price the listing of 85 square meters, 3 kilometers out, through the kept statistics, and through LinearRegression, both rounded to whole units.

Hints:

• Compare previous with loss.item(), never with the tensor, and update previous after the check, or the rule compares a loss with itself and stops at epoch 1.
• Judge the two runs on np.allclose at rtol=1e-4, and read the two losses to four digits before you decide what the rule measured.

import numpy as np
import pandas as pd
import torch
from sklearn.linear_model import LinearRegression

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
features = ["area_sqm", "distance_km"]
X = torch.tensor(has_area[features].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
means = X.mean(dim=0)
sds = X.std(dim=0, correction=0)
Z = (X - means) / sds
# torch.Size([2875, 2]) torch.Size([2875])
print(Z.shape, (Z @ torch.zeros(2)).shape)

def descend(epochs=None):
    w = torch.zeros(2, requires_grad=True)
    b = torch.tensor(0.0, requires_grad=True)
    previous = float("inf")
    epoch = 0
    while epochs is None or epoch < epochs:
        epoch += 1
        loss = ((price - (Z @ w + b)) ** 2).mean()
        loss.backward()
        with torch.no_grad():
            w -= 0.1 * w.grad
            b -= 0.1 * b.grad
        w.grad, b.grad = None, None
        # the stopping rule: a fall of less than one part in a million
        if epochs is None and previous - loss.item() < 1e-6 * loss.item():
            break
        previous = loss.item()
    return w, b, epoch, loss.item()

model = LinearRegression().fit(has_area[features], has_area["price"])
# [  3549. -13267.] 101807
print(np.round(model.coef_), round(model.intercept_))

w, b, stopped, loss_stop = descend()
with torch.no_grad():
    per_unit = w / sds
print(stopped, f"{loss_stop:.4e}")  # 43 1.5725e+10
print(np.round(per_unit.numpy()))   # [  3546. -13225.]
# False
print(np.allclose(per_unit.numpy(), model.coef_, rtol=1e-4))

w, b, epochs, loss_100 = descend(100)
with torch.no_grad():
    per_unit = w / sds
    intercept = b - (w * means / sds).sum()
print(epochs, f"{loss_100:.4e}")  # 100 1.5725e+10
# [  3549. -13267.] 101807
print(np.round(per_unit.numpy()), round(intercept.item()))
# True
print(np.allclose(per_unit.numpy(), model.coef_, rtol=1e-4))
# The rule measured how much the loss still moved, and the loss had
# stopped moving to six digits by epoch 43 while the weights were still
# 3 per square meter and 42 per kilometer short of least squares.

new = torch.tensor([[85.0, 3.0]])
with torch.no_grad():
    # tensor([363672.])
    print(torch.round(((new - means) / sds) @ w + b))
# [363672.]
print(np.round(model.predict(pd.DataFrame({"area_sqm": [85.0], "distance_km": [3.0]}))))

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want to see the step and the clearing as PyTorch packages them, or the whole of autograd walked through by the people who wrote it.

• Official docs - PyTorch 2.14, torch.optim, "Taking an optimization step". optimizer.step() after backward(), and optimizer.zero_grad() at the top of its example loop: the fourth and fifth steps of this lesson as two calls on an object, which Lesson 4.2.2 puts in place of the lines you wrote here.
• YouTube - "The Fundamentals of Autograd" by PyTorch. The official walk through autograd, from why training needs it, through a small example and autograd inside a training loop with an optimizer, to turning it off with no_grad and detach, in-place operations and the profiler, reaching past this lesson into Lesson 4.2.2's optimizer.

Summary

The whole of training turned out to be five lines in a loop. Nothing in them was new: the prediction was Module 2's dot product, the loss was Lesson 3.1.3's, and the walk, the step and the clearing were Lesson 4.1.4's. What was new is what happens when they repeat: a model with two parameters finds in seventy steps the answer least squares computes in one solve, by the same five lines a network with a million parameters will use, where neither the solve nor the grid exists.

Every failure the lesson met was a failure of something the loop does not check. It does not check the scale of its input, so the raw area overflowed at one rate and crawled at another, and standardizing was the fix. It does not check the learning rate, so six rates printed six different stories from one start, and the loss every tenth epoch was the only way to tell them apart. It does not check whether .grad was cleared, so the loop without its fifth step bounced forever with an exit code of 0. And it does not know what units its weight is in, so reading the fit back through the kept statistics was the last step of the fit rather than an afterthought.

Lesson 4.2.2 hands each of the five lines to a class, nn.Linear for the neuron, nn.MSELoss for the loss, optim.SGD for the step and the clearing, and proves the numbers unchanged. The classes will hide the lines, and they hide none of the failures, which is why each was met here, on one neuron, with least squares' answer to compare against.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import torch

mean, sd = area.mean(), area.std(correction=0)
z = (area - mean) / sd           # kept: mean, sd
w = torch.tensor(0.0, requires_grad=True)
b = torch.tensor(0.0, requires_grad=True)
for epoch in range(1, 101):
    loss = ((price - (w * z + b)) ** 2).mean()
    loss.backward()              # gradients into .grad
    with torch.no_grad():        # the step, off the record
        w -= 0.1 * w.grad
        b -= 0.1 * b.grad
    w.grad, b.grad = None, None  # the fifth step, never skipped
    if epoch % 10 == 0:
        print(epoch, f"{loss.item():.4e}", round(w.item()))

with torch.no_grad():
    per_sqm = w / sd               # per square meter
    intercept = b - w * mean / sd  # on the raw area
    new_z = (torch.tensor([85.0]) - mean) / sd
    price_85 = w * new_z + b       # through the kept statistics