Derivatives and the Chain Rule - Computed by Hand

FREE~1.5 h2 labs4 questions1 assignment

Context

The derivative of the loss with respect to a parameter is the slope of the loss at the parameter's current value: which way the loss falls from there, and how steeply. A few rules for simple functions, joined by the chain rule, turn that slope into a formula computed from the data in one pass, and a centered difference checks the formula from the loss alone.

You use it every time a model is trained by steps rather than by a scan or a one-shot solve: the price model Lesson 4.2.1 fits on these listings, the network Section 4.3 builds for sold_in_30_days and every image model in Section 4.5 move each parameter by a share of this slope, thousands at a time. Lesson 4.1.4 hands the computation to autograd (PyTorch's own derivative machinery) and checks its number against the one you compute here.

A slope read without its sign moves a parameter uphill. A step that overshoots the bottom and raises the loss gets blamed on the slope when its size was the fault. A numerical check with a step so small that float64 eats the digits reports a derivative wrong in its sixth digit and gets believed. Lesson 3.1.3 found the bottom of mse(w) by trying 6,001 candidates, and this lesson finds which way it lies from one number at one w, the search that scales to thousands of parameters where a grid cannot.

Goal

Compute the slope of 3.1.3's mse(w) at one w with a centered difference, choose its width h by watching the digits, and distinguish it from a forward difference.
Apply the rules for simple functions and the chain rule to write dL_dw for a squared residual, and check each against a centered difference.
Build the derivative of the whole squared-error loss, show that it is zero at lstsq's answer, and extend it to the gradient of a two-parameter model as a vector of partial derivatives.
Take one step against the gradient with a learning rate, trace what four rates did to the loss, and explain precisely why one of them lands on this loss's bottom in one step and would not on a network's.
Explain precisely why the derivative replaces the grid once the parameters number in the thousands.

Prerequisites

None of this is required, and the lesson stands on its own. Both are here if you want the slope drawn as a picture before you measure it in NumPy.

YouTube - "The paradox of the derivative | Chapter 2, Essence of calculus" by 3Blue1Brown. The derivative built as the slope between two points on a curve pulled closer and closer together, with a car's distance against time and its speed as the running example.
YouTube - "Visualizing the chain rule and product rule | Chapter 4, Essence of calculus" by 3Blue1Brown. The sum, product and chain rules as pictures, the chain rule drawn as a small nudge to the input passed through one function into the next, each link multiplying the change.

The Derivative Is the Slope of the Loss
Measuring a Slope With a Centered Difference
The Rules for Simple Functions
The Chain Rule
The Gradient of the Squared-Error Loss
One Step Against the Gradient

Theory

Every Theory block below adds to one script, slope.py in ~/dev/bootcamp, run with python3 slope.py beside listings.csv, and the lesson computes with NumPy alone, no PyTorch: it computes by hand the number Lesson 4.1.4 checks autograd against.

The Derivative Is the Slope of the Loss

What. The derivative of the loss with respect to w (the slope of the loss at one value of the parameter, how much the loss changes per unit change of w) is one number with a sign and a size. Negative means the loss falls as w grows, positive means it rises, and zero means the curve is flat there, which is what the bottom looks like from up close. Why. Lesson 3.1.3 found the bottom of mse(w) by computing the loss at 6,001 candidates and keeping the smallest. The slope at one w says which way the bottom lies and how steep the loss is there, from that one place, and that is the search that survives thousands of parameters. When. At every step of every training loop in this module, for every parameter, and here first, on the one parameter of price = w * area.

The first block rebuilds 3.1.3's mse(w) and its grid on the 2,875 listings that have an area, and keeps the exact bottom lstsq returned there as bottom:

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

listings = pd.read_csv("listings.csv")
# the 2,875 listings with an area, as in 3.1.3
has_area = listings.dropna(subset=["area_sqm"])
area = has_area["area_sqm"].to_numpy()
price = has_area["price"].to_numpy()


def mse(w):
    predicted = w * area        # 3.1.3's model, price = w * area
    return np.mean((price - predicted) ** 2)


grid = np.arange(0, 6_001, 1.0)
losses = np.array([mse(w) for w in grid])
w_ls, ssr, rank, sv = np.linalg.lstsq(area[:, np.newaxis], price)
bottom = w_ls[0]                # 3.1.3's exact bottom
print(grid[np.argmin(losses)])  # 3750.0
print(round(bottom, 2))         # 3749.58
print(f"{mse(3_000):,.0f}")     # 24,320,084,049
print(f"{mse(4_000):,.0f}")     # 18,456,928,403

Moving w by one unit, one currency unit per square meter, measures the curve's direction at each of the three points:

for w in [3_000.0, bottom, 4_000.0]:
    up = mse(w + 1) - mse(w)    # one unit added to w
    down = mse(w - 1) - mse(w)  # one unit taken from w
    print(round(w, 2), f"{up:,.0f}", f"{down:,.0f}")

3000.0 -17,597,566 17,621,059
3749.58 11,746 11,746
4000.0 5,894,747 -5,871,255

The slope is negative at 3,000, where the loss runs downhill to the right, positive at 4,000, and zero at the bottom, where both neighbors sit higher. The two changes at 3,000 differ in size because the curve bends across each one-unit step, so the slope AT 3,000 is neither of them, which the next subtitle settles. The curve with the three points marked, drawn as 3.1.3 drew it and checked by printing what it holds:

fig, ax = plt.subplots(figsize=(8, 5))
ax.plot(grid, losses)
marks = np.array([3_000.0, bottom, 4_000.0])
ax.plot(marks, [mse(w) for w in marks], "o", color="black")
ax.set_xlabel("w, price per square meter")
ax.set_ylabel("mean squared error")
fig.savefig("slope_points.png", dpi=150, bbox_inches="tight")

points = ax.lines[1]                    # the three dots, drawn second
print(len(ax.lines))                    # 2
print(np.round(points.get_xdata(), 2))  # [3000.   3749.58 4000.  ]

Why the slope replaces the grid: 3.1.3's grid spent 6,001 evaluations of the loss to place one bottom, a second parameter makes it 6,001 by 6,001, 36 million candidates, and a third 216 billion. The slope at one w costs two evaluations of the loss, or one pass over the data once its formula is written, and a second parameter adds one more slope. The grid's cost multiplies where the slope's adds, which is what "hopeless for thousands" meant in 3.1.3, and Lesson 4.2.1 turns the slope into the search.

DON'T:

DON'T read a slope without its sign. -17,609,313 at 3,000 says the loss falls as w grows, so w has to grow, and a teammate who drops the sign moves w the wrong way by exactly the right amount.

Measuring a Slope With a Centered Difference

What. The centered difference (the loss a small step above w minus the loss a small step below, divided by the width of the two steps) is the slope of the straight line through the two points that straddle w, and it asks for nothing but the loss itself, so it works at any w and on any function you can call. Why. It is the one measurement that needs no rule and no formula. Every derivative this lesson writes by hand is set beside it, so a mistake in a formula has nowhere to hide. When. To check a derivative, at a few values of the parameter, never to train with: it costs two evaluations of the loss per parameter, and its digits depend on the width you choose.

def centered(f, w, h=1e-3):
    # the rise from w - h to w + h, over the run of 2 * h
    return (f(w + h) - f(w - h)) / (2 * h)


print(f"{centered(mse, 3_000):,.3f}")   # -17,609,312.574
print(f"{centered(mse, 4_000):,.3f}")   # 5,883,001.282
print(f"{centered(mse, bottom):,.3f}")  # 0.000

The width h decides which digits of these three numbers are right, and a ladder of widths at 3,000 shows how:

for h in [1.0, 0.1, 0.01, 1e-3, 1e-4, 1e-6, 1e-8]:
    print(h, f"{centered(mse, 3_000, h):,.3f}")

1.0 -17,609,312.570
0.1 -17,609,312.570
0.01 -17,609,312.571
0.001 -17,609,312.574
0.0001 -17,609,312.630
1e-06 -17,609,310.150
1e-08 -17,609,405.518

At 1e-8 the number is off by 93, in its sixth digit. The drift is float64's arithmetic, not the slope's, and the two losses the subtraction takes apart show it:

up, down = mse(3_000 + 1e-8), mse(3_000 - 1e-8)
print(up)              # 24320084048.519558
print(down)            # 24320084048.871746
print(up - down)       # -0.3521881103515625
print(np.spacing(up))  # 3.814697265625e-06

Float64 holds about sixteen significant digits. Each loss is about 24 billion, so the two agree in their first eleven digits, and np.spacing(up), the distance from up to the next number float64 can hold at that size, is 3.8 millionths: neither loss is known more finely than that. Their true difference is 2 times 1e-8 times the slope, -0.35218625, and the subtraction returned -0.35218811, wrong by about 2 millionths, the rounding the two losses carried in, and dividing by 2e-8 multiplies those 2 millionths by 50 million, into the 93. A wider h makes the true difference larger while the rounding stays at a few millionths, which is why the widths from 1.0 to 0.01 agree.

The forward difference is the measurement to avoid:

def forward(f, w, h=1e-3):
    return (f(w + h) - f(w)) / h            # the rise from w to w + h


print(f"{forward(mse, 3_000):,.3f}")        # -17,609,300.827
print(round(np.mean(area ** 2) * 1e-3, 3))  # 11.746

It is off by 11.74 in its seventh digit, where the centered difference at the same h is off by 0.004 in its eleventh, and the second print says why. The chord from w to w + h on a curve that bends upward is flatter than the tangent at w by the bend across the step, which on this loss is exactly np.mean(area ** 2) * h. The centered chord runs from w - h to w + h, so the bend on the left of w cancels the bend on the right, and on this loss the cancellation is exact at every width, because the loss is a parabola in w, a sum of terms in w ** 2, w and constants, which leans away from its tangent by the same amount on either side of any point: that is why 1.0 and 0.1 returned the same number, and why the only error the centered difference carried was float64's. The module's check is the centered difference with h = 1e-3, wide enough to leave the rounding behind and narrow enough that the two bends still cancel on a curve that is not a parabola.

The three slopes drawn as straight lines through the three points, on the same Axes:

for w in marks:
    slope = centered(mse, w)
    span = np.array([w - 400, w + 400])
    # a straight line through the point at the measured slope
    ax.plot(span, mse(w) + slope * (span - w), color="black")
fig.savefig("slope_tangents.png", dpi=150, bbox_inches="tight")

print(len(ax.lines))
for line in ax.lines[2:]:
    x, y = line.get_xdata(), line.get_ydata()
    print(round((y[1] - y[0]) / (x[1] - x[0])))
plt.close(fig)

The Axes holds five lines, and the slope of each drawn segment, its rise over its run, is the centered difference to the unit: falling at 3,000, flat at the bottom, and climbing at 4,000 a third as steeply.

DON'T:

DON'T shrink h to make the check more exact, and DON'T use the forward difference as the check. At 1e-3 the centered difference is off by 0.004, where at 1e-8 it is off by 93 and the forward difference at 1e-3 is off by 11.74, for the same two evaluations of the loss.
DON'T carry a wide h to a function that is not a parabola. centered(lambda w: w ** 3, 2.0, 1.0) returns 13.0 and centered(lambda w: w ** 3, 2.0) returns 12.000000999998317, 12.000001 to six decimals, and the slope of w ** 3 at 2 is 12: that curve bends more on the right of 2 than on the left, so the two halves of a wide chord stop cancelling.

The Rules for Simple Functions

What. Five rules give the derivative of every piece the squared-error loss is built from, with nothing measured. A constant has derivative 0. w has derivative 1. c * w has derivative c, for any constant c. w ** 2 has derivative 2 * w. A sum or difference of terms has the sum or difference of their derivatives, so a mean of terms, a sum divided by a constant count, has the mean of their derivatives. Why. A rule gives the slope exactly, at every w, in one pass over the data, where the centered difference measures it at one w, to the digits its width allows, from two passes. A training step is written from the rules and checked with the measurement. When. Every time you differentiate a loss by hand, and every time you read a derivative someone else wrote, because every one of them is these rules joined by the next subtitle's.

In mse(w) the data are constants: area and price do not move while the search runs, only w does, so price has derivative 0, w * area is a constant times w with derivative area, and every slope is per unit of w. Each rule, as a one-line function beside its derivative, checked at w = 3000:

def check(name, f, rule, w):
    # the rule's answer beside the measured slope
    print(name, round(rule(w), 3), round(centered(f, w), 3))


check("constant", lambda w: 7.0, lambda w: 0.0, 3_000.0)
check("w", lambda w: w, lambda w: 1.0, 3_000.0)
check("-175.7 * w", lambda w: -175.7 * w, lambda w: -175.7, 3_000.0)
check("w ** 2", lambda w: w ** 2, lambda w: 2 * w, 3_000.0)
check("sum", lambda w: w ** 2 - 175.7 * w + 7.0, lambda w: 2 * w - 175.7, 3_000.0)
check("mean", lambda w: np.mean(w * area), lambda w: np.mean(area), 3_000.0)

constant 0.0 0.0
w 1.0 1.0
-175.7 * w -175.7 -175.7
w ** 2 6000.0 6000.0
sum 5824.3 5824.3
mean 99.02 99.02

The last rule is the one a loss leans on hardest: a mean over listings is differentiated listing by listing and averaged, and the division by 2,875 stays in the derivative.

DON'T:

DON'T differentiate with respect to the data. A slope per square meter of area would answer a question about a listing that changed size, not about the fit, and in mse(w) only w moves.
DON'T drop the mean's count. np.sum in its place returns 2,875 times the slope, -50,626,773,640 at 3,000, so every step taken from it is 2,875 times too long.

The Chain Rule

What. The chain rule (the derivative of a function of a function is the product of the two derivatives, the outer one taken at the inner function's value) joins the rules across the steps of a computation: when L is computed from r and r from w, the slope of L per unit of w is the slope of L per unit of r times the slope of r per unit of w. Why. The squared-error loss is such a chain: w makes a prediction, the prediction makes a residual, the residual is squared, the squares are averaged. No single rule covers (price - w * area) ** 2, because the thing squared is not w, and every network in this module is a longer chain of the same kind. When. Whenever the parameter reaches the loss through intermediate values, which is every loss from here on, each layer of a network one more link.

The first listing at w = 3000, link by link:

w = 3_000.0
a0, p0 = area[0], price[0]    # the first listing, 175.7 and 509000
p = w * a0                    # multiply: the prediction, 527100.0
r = p0 - p                    # subtract: the residual, -18100.0
L = r ** 2                    # square: this listing's loss
dL_dr = 2 * r                 # the rule for a square, 2 times r
dr_dp = -1.0                  # p enters r with a minus sign
dp_dw = a0                    # a constant times w, the constant
print(dL_dr * dr_dp * dp_dw)  # 6360340.0

The product is the slope of this listing's loss per unit of w, and the centered difference on the listing's loss alone agrees:

def one_listing(w):
    return (p0 - w * a0) ** 2              # the first listing's loss alone


print(round(centered(one_listing, w), 3))  # 6360340.001
print(dL_dr)                               # -36200.0

The sign matters. The model overprices this listing, 527,100 for 509,000, so r is negative, raising w pushes the prediction further above the price, and this listing's loss climbs as w grows. 2 * r alone, -36,200, says the opposite, and is not a derivative of anything with respect to w. NumPy applies the chain to all 2,875 listings at once:

r = price - w * area             # 2,875 residuals at w = 3000
dL_dw = 2 * r * (-area)          # three links, the -1 folded into -area
print(dL_dw[:3])                 # [  6360340.  -4727500. -43424640.]
each = centered(lambda w: (price - w * area) ** 2, w)
print(np.allclose(dL_dw, each))  # True

The same rule is why a coefficient changes units when its feature is standardized. Lesson 3.2.1 standardized area to z = (area - mean) / std, and a line fitted on z reports its weight per standard deviation of area, not per square meter:

z = (area - area.mean()) / area.std()  # standardized area, 3.2.1's z on all 2,875 rows
ones = np.ones(len(area))
coef_z, ssr, rank, sv = np.linalg.lstsq(np.column_stack([ones, z]), price)
coef, ssr, rank, sv = np.linalg.lstsq(np.column_stack([ones, area]), price)
print(np.round(coef_z, 2))             # [380325.91 144879.63]
print(np.round(coef, 2))               # [54716.9   3288.31]
print(round(area.std(), 4))            # 44.0589
dz_darea = 1 / area.std()              # standardize: a constant times area
dprice_dz = coef_z[1]                  # multiply: the weight per unit of z
print(round(dprice_dz * dz_darea, 2))  # 3288.31

The two fits are one line in two units: 144,879.63 per standard deviation, divided by the square meters a standard deviation holds, area.std() at 44.0589, is the raw line's 3,288.31 per square meter, and the division runs on the computed deviation, because the printed 44.0589 gives 3,288.32 and 44.06 gives 3,288.23. Lesson 3.2.1's sentence that rescaling a feature "changes the units of its weight" is this rule.

DON'T:

DON'T stop at the outer link. 2 * (p0 - w * a0) is the slope of a square per unit of what was squared, and the residual is not w: without the inner slope, -175.7, the number has the wrong size and the wrong sign.
DON'T compare a weight per standard deviation with a weight per square meter, 144,879.63 and 3,288.31 describe one line.

The Gradient of the Squared-Error Loss

What. The derivative of mse with respect to w is the mean of the chain rule's 2,875 slopes, np.mean(-2 * area * (price - w * area)), exact at every w from one pass over the data. With two parameters, a partial derivative (the derivative with respect to one parameter, with every other parameter held still) exists for each, and the gradient (the vector of all the partial derivatives, one per parameter) is the slope of the loss in every direction at once. Why. Training is built on this number, and least squares is this number set to zero: 3.1.3's exact bottom is where the slope is zero, and lstsq found it in one step by solving for that zero. A network has no such solve, and the same gradient taken step by step is its only way down. When. At every step of every loop from Lesson 4.2.1 on, once per parameter.

def dmse_dw(w):
    residual = price - w * area
    # the mean of the chain rule's 2,875 slopes
    return np.mean(-2 * area * residual)


print(f"{dmse_dw(3_000):,.3f}")     # -17,609,312.570
print(f"{dmse_dw(4_000):,.3f}")     # 5,883,001.280
w_zero = np.sum(area * price) / np.sum(area ** 2)
print(round(w_zero, 2))             # 3749.58
print(np.isclose(w_zero, bottom))   # True
print(dmse_dw(w_zero))              # 1.127823539402174e-08
print(round(dmse_dw(3_749.58), 3))  # 56.046

The formula and the measurement agree at 3,000 to ten digits, the formula's digits all exact and the measurement's last one float64's. Setting the formula to zero and solving for w gives np.sum(area * price) / np.sum(area ** 2), 3,749.58, the exact bottom lstsq returned in 3.1.3, so least squares is this derivative set to zero and solved, and the slope there is 1e-8, float64's rounding over 2,875 terms. At 3,749.58, the bottom rounded to two decimals, the slope is already 56, so a derivative is checked for zero at the number the computation returned, never at its rounded print.

With an intercept, price = w * area + b, the loss has two parameters and the gradient two entries:

def mse2(w, b):
    return np.mean((price - (w * area + b)) ** 2)


def gradient(w, b):
    residual = price - (w * area + b)
    dL_dw = np.mean(-2 * area * residual)  # b held still
    dL_db = np.mean(-2 * residual)         # w held still
    return np.array([dL_dw, dL_db])


for g in gradient(3_000, 50_000):
    print(f"{g:,.3f}")
print(f"{centered(lambda w: mse2(w, 50_000), 3_000):,.3f}")
print(f"{centered(lambda b: mse2(3_000, b), 50_000):,.3f}")

-7,707,309.092
-66,531.617
-7,707,309.092
-66,531.616

dL_db is the same chain as dL_dw with the prediction's slope per unit of b equal to 1, so the area drops out, and the two centered differences agree with the formula to the hundredth. Both are negative, so the loss falls as either parameter grows. Where lstsq put the line with an intercept in Lesson 3.1.1, 54,716.9 and 3,288.31:

b_fit, w_fit = coef                      # lstsq's line with an intercept
print(round(b_fit, 1), round(w_fit, 2))  # 54716.9 3288.31
print(gradient(w_fit, b_fit))            # [4.97569209e-09 7.25621762e-11]
# True: both partial derivatives are zero to a millionth
print(np.allclose(gradient(w_fit, b_fit), 0, atol=1e-6))

On a loss of 17 billion whose slope in w a few hundred units away runs into the millions, an atol of a millionth is float64's rounding and nothing else. The bottom of a two-parameter loss is where every partial derivative is zero at once, the pair of equations lstsq solved, and this gradient is the one Lesson 3.5.3 named when boosting took its name from it.

DON'T:

DON'T rank parameters by the size of their partial derivatives. -7,707,309 per unit of w against -66,532 per unit of b compares one currency unit per square meter, applied to every listing's area, with one currency unit, and says nothing about which parameter matters.
DON'T declare a derivative zero because it prints small. 56 at 3,749.58 is not zero, and the test is np.allclose(..., 0, atol=...) at the unrounded number, with a tolerance chosen for the loss's scale.

One Step Against the Gradient

What. A step moves w by a learning rate (the share of the gradient a step moves by) times the slope, in the direction that lowers the loss: w - lr * dL_dw. The minus sign is the whole direction, a negative slope moves w up and a positive slope moves it down, and either way the loss falls for a small enough step. Why. The slope says which way the bottom lies and how steeply the loss falls, not how far away the bottom is, so the size of the step is a choice, and the choice decides whether the step lands short of the bottom, on it, or past it where the loss is higher than before. When. Once here, from w = 3000. The loop that repeats the step until the slope is zero is Lesson 4.2.1, and the sweep that chooses the rate is Lesson 4.3.4.

w = 3_000.0
dL_dw = dmse_dw(w)
for lr in [1e-5, 5e-5, 1e-4, 2e-4]:
    w_new = w - lr * dL_dw  # minus: against the slope
    print(lr, round(w_new, 2), f"{mse(w_new):,.0f}")

1e-05 3176.09 21,583,439,257
5e-05 3880.47 17,921,542,107
0.0001 4760.93 29,734,705,200
0.0002 6521.86 107,996,146,496

Four rates, one slope. 1e-5 covers 176 of the 750 units to the bottom. 5e-5 lands 131 past the bottom and the loss still falls, because the far side at that distance sits lower than the start did. 1e-4 lands 1,011 past the bottom, farther from it than 3,000 was, so the loss rises above its start. One rate lands exactly:

rate = 1 / (2 * np.mean(area ** 2))
print(f"{rate:.4e}")                    # 4.2567e-05
print(round(w - rate * dL_dw, 2))       # 3749.58
print(f"{mse(w - rate * dL_dw):,.0f}")  # 17,720,310,796
# 1.17 2.35, the second and third rates as shares of this one
print(round(5e-5 / rate, 2), round(1e-4 / rate, 2))
# True: the slope is one constant times the distance to the bottom
print(np.isclose(dL_dw, 2 * np.mean(area ** 2) * (w - w_zero)))

The last print says why: on this loss the slope at every w is 2 * np.mean(area ** 2) * (w - w_zero), one constant times the distance to the bottom, because the loss is a parabola in w, so dividing the slope by that constant gives the distance exactly, the step lstsq takes in one solve. On a parabola any rate above twice the one-step rate lands farther out than it started, which is where the two shares put 1e-4 and not 5e-5. No network's loss is a parabola in its parameters, so no such constant exists for one, and the one-step rate is an insight into this loss and not a method: on a network the rate is chosen, and the step is repeated.

DON'T:

DON'T step along the slope. w + 1e-5 * dL_dw from 3,000 lands at 2,823.91 and the loss rises from 24,320,084,049 to 27,785,197,041, uphill by exactly the step that would have gone down. The minus sign in w - lr * dL_dw is the direction.
DON'T carry 4.2567e-05 anywhere. It is 1 / (2 * np.mean(area ** 2)) for this loss on these 2,875 listings, and another set of listings has another.

Lab Exercises

Lab Exercise 1: The Slope of the Houses' Loss

Work in house_slope.py in ~/dev/bootcamp, on the houses in listings.csv that have an area, 770 of them, under 3.1.3's price = w * area and its mean squared error.

Find the exact bottom with np.linalg.lstsq through the origin and print it to two decimals, with the loss at w = 3000 and at the bottom. Print the change in the loss from one unit added to w and from one unit taken away, at 3,000, at the bottom and at 4,000, and say in a comment what the signs tell you at each.

Write centered(f, w, h) and print the centered difference of the loss at 3,000 for h at 1.0, 1e-3 and 1e-8, then the forward difference at 1e-3, and print np.mean(area ** 2) * 1e-3 beside it. Say in a comment which two agree, why the 1e-8 line drifts and what the forward difference is off by.

Finally write the derivative of 3.1.3's raw residual score, np.mean(price - w * area), by the rules, print it beside the centered difference at 3,000 and at 6,000, print the score itself at both, and say in a comment why 3.1.3's scan of that score returned its last candidate.

Lab Exercise 2: Eastgate's Gradient, and One Step

Work in eastgate_gradient.py in ~/dev/bootcamp, on the Eastgate listings that have an area, 331 of them.

At w = 4000, take the first Eastgate listing through the chain rule link by link, prediction, residual and square, and print the product of the three slopes with the centered difference of that listing's squared residual beside it. Then write dmse_dw(w) for all 331 listings, print it beside the centered difference at 4,000, compute the w where it is zero from np.sum(area * price) / np.sum(area ** 2), and print it beside lstsq's answer through the origin.

Write mse2(w, b) and gradient(w, b) for price = w * area + b, print the gradient at w = 4000, b = 50000 beside the two centered differences, each holding the other parameter still, then print the gradient at the line lstsq fits with an intercept and the result of np.allclose against zero with atol=1e-6.

Finally take one step from w = 4000 under price = w * area with learning rates 1e-5 and 1e-4, print each landing and the loss before and after, compute the one-step rate 1 / (2 * np.mean(area ** 2)) and show where it lands, and say in a comment which rate overshot and how you know without looking at the landing.

Solution 1: The Slope of the Houses' Loss

import numpy as np
import pandas as pd

listings = pd.read_csv("listings.csv")
houses = listings[listings["property_type"] == "house"]
houses = houses.dropna(subset=["area_sqm"])
area = houses["area_sqm"].to_numpy()
price = houses["price"].to_numpy()
print(len(price))             # 770


def mse(w):
    return np.mean((price - w * area) ** 2)


w_ls, ssr, rank, sv = np.linalg.lstsq(area[:, np.newaxis], price)
bottom = w_ls[0]
print(round(bottom, 2))       # 3514.56
print(f"{mse(3_000):,.0f}")   # 34,362,707,753
print(f"{mse(bottom):,.0f}")  # 27,999,080,100
for w in [3_000.0, bottom, 4_000.0]:
    up = mse(w + 1) - mse(w)
    down = mse(w - 1) - mse(w)
    print(round(w, 2), f"{up:,.0f}", f"{down:,.0f}")
# at 3000 one unit up lowers the loss and one unit down raises it,
# so the slope is negative, at 4000 the reverse, so it is positive,
# and at the bottom both sides rise by the same 24,034, so it is zero

3000.0 -24,710,099 24,758,167
3514.56 24,034 24,034
4000.0 23,358,187 -23,310,119

def centered(f, w, h=1e-3):
    return (f(w + h) - f(w - h)) / (2 * h)


for h in [1.0, 1e-3, 1e-8]:
    print(h, f"{centered(mse, 3_000, h):,.3f}")
forward = (mse(3_000 + 1e-3) - mse(3_000)) / 1e-3
print(f"{forward:,.3f}")                    # -24,734,109.238
print(round(np.mean(area ** 2) * 1e-3, 3))  # 24.034
# 1.0 and 1e-3 agree to the hundredth. 1e-8 drifts by 781, because
# the two losses of 34 billion agree in ten digits and dividing
# their difference by 2e-8 magnifies the rounding they carry. The
# forward difference is off by 24.03, the bend of the curve across
# one step of 1e-3, which is mean(area ** 2) times h.

1.0 -24,734,133.273
0.001 -24,734,133.278
1e-08 -24,733,352.661

def raw(w):
    return np.mean(price - w * area)   # 3.1.3's raw residual score


print(round(-np.mean(area), 3))        # -150.108
print(round(centered(raw, 3_000), 3))  # -150.108
print(round(centered(raw, 6_000), 3))  # -150.108
print(f"{raw(3_000):,.0f}")            # 77,207
print(f"{raw(6_000):,.0f}")            # -373,116
# the slope is -150.108, the mean area with a minus sign, at every
# w: the score falls at the same rate forever and never flattens,
# so it has no bottom and a scan returns the last candidate it tried

DON'T:

DON'T read the houses' -24,734,133 against the Theory's -17,609,313 as the same loss, steeper. One is on 770 houses whose bottom sits at 3,514.56, the other on 2,875 listings whose bottom sits at 3,749.58, so 3,000 is a different distance from each.
DON'T widen a grid to find the raw score's bottom. Its slope is the same -150.108 at 3,000 and at 6,000, a slope that never reaches zero has no bottom, and 3.1.3's scan returned its last candidate for that reason, not because the grid stopped early.

Solution 2: Eastgate's Gradient, and One Step

import numpy as np
import pandas as pd

listings = pd.read_csv("listings.csv")
eastgate = listings[listings["neighborhood"] == "Eastgate"]
eastgate = eastgate.dropna(subset=["area_sqm"])
area = eastgate["area_sqm"].to_numpy()
price = eastgate["price"].to_numpy()
print(len(price))                    # 331


def mse(w):
    return np.mean((price - w * area) ** 2)


def centered(f, w, h=1e-3):
    return (f(w + h) - f(w - h)) / (2 * h)


w = 4_000.0
a0, p0 = area[0], price[0]           # 30.5 and 169000
r = p0 - w * a0                      # 47000.0, the line underprices it
dL_dw = 2 * r * (-1.0) * a0          # square, subtract, multiply
print(dL_dw)                         # -2867000.0


def first(w):
    return (p0 - w * a0) ** 2


print(round(centered(first, w), 3))  # -2867000.001

def dmse_dw(w):
    return np.mean(-2 * area * (price - w * area))


print(f"{dmse_dw(w):,.3f}")                 # -34,472,449.426
print(f"{centered(mse, w):,.3f}")           # -34,472,449.432
w_zero = np.sum(area * price) / np.sum(area ** 2)
w_ls, ssr, rank, sv = np.linalg.lstsq(area[:, np.newaxis], price)
print(round(w_zero, 2), np.round(w_ls, 2))  # 5372.75 [5372.75]

def mse2(w, b):
    return np.mean((price - (w * area + b)) ** 2)


def gradient(w, b):
    residual = price - (w * area + b)
    dL_dw = np.mean(-2 * area * residual)
    dL_db = np.mean(-2 * residual)
    return np.array([dL_dw, dL_db])


for g in gradient(4_000, 50_000):
    print(f"{g:,.3f}")
print(f"{centered(lambda w: mse2(w, 50_000), 4_000):,.3f}")
print(f"{centered(lambda b: mse2(4_000, b), 50_000):,.3f}")
ones = np.ones(len(area))
coef, ssr, rank, sv = np.linalg.lstsq(np.column_stack([ones, area]), price)
b_fit, w_fit = coef
print(round(b_fit), round(w_fit, 1))  # 78278 4728.0
print(gradient(w_fit, b_fit))         # [-5.22215921e-08 -5.14900396e-10]
# True: zero to a millionth, on a loss of 21 billion
print(np.allclose(gradient(w_fit, b_fit), 0, atol=1e-6))

-24,130,818.006
-207,134.743
-24,130,818.008
-207,134.741

dL_dw = dmse_dw(w)
print(f"{mse(w):,.0f}")                 # 45,117,999,033
for lr in [1e-5, 1e-4]:
    w_new = w - lr * dL_dw
    print(lr, round(w_new, 2), f"{mse(w_new):,.0f}")
rate = 1 / (2 * np.mean(area ** 2))
print(f"{rate:.4e}")                    # 3.9822e-05
print(round(w - rate * dL_dw, 2))       # 5372.75
print(f"{mse(w - rate * dL_dw):,.0f}")  # 21,456,888,643
# 1e-4 overshot: it is 2.5 times the one-step rate, so the step
# covers 2.5 times the distance to the bottom and lands 1.5 of that
# distance past it, farther out than 4,000 was, where the loss is
# higher. On this loss any rate above twice the one-step rate does.

1e-05 4344.72 34,726,585,978
0.0001 7447.24 75,491,486,025

DON'T:

DON'T reuse the Theory's one-step rate on Eastgate. Here 1 / (2 * np.mean(area ** 2)) is 3.9822e-05 against the city's 4.2567e-05, because Eastgate's areas differ, and the city's rate is 1.069 times Eastgate's, so it lands Eastgate's step past its bottom, at 5,467.39.
DON'T read atol=1e-8 failing as a gradient that is not zero. At lstsq's line Eastgate's partial derivative in w is -5.2e-08, so np.allclose(gradient(w_fit, b_fit), 0, atol=1e-8) prints False on a loss of 21 billion whose slope in w a few hundred units away runs into the millions: that residue is float64's rounding over 331 terms, and atol=1e-6 is what zero means at this scale.

Theory Assessment

Q1: Three Slopes at 3,000

A teammate measures the slope of mse at 3,000 as mse(3_001) - mse(3_000) and reports -17,597,566. A second uses the centered difference with h = 1e-8 and reports -17,609,405.518. The formula gives -17,609,312.570. Say what each of the first two numbers measures, by how much and why each misses, and whether either would mislead a step, a check, or both.

Hint: For the first, ask what the chord from 3,000 to 3,001 does on a curve that bends upward, and what np.mean(area ** 2) has to do with it. For the second, ask how many digits two losses of 24 billion share and what dividing their difference by 2e-8 does to the digits they do not.

Q2: The Outer Link Alone, and the Weight That Changed Units

For the first listing, 175.7 square meters at 509,000, a teammate differentiates (p0 - w * a0) ** 2 with the rule for a square and writes 2 * (p0 - w * a0), -36,200 at w = 3000, where the lesson's product of three slopes gave 6,360,340. Name what is missing and the rule that supplies it, and explain why the correct slope is positive while the teammate's is negative. Then explain, with the same rule, why a line fitted on standardized area reports 144,879.63 while the same line on raw area reports 3,288.31.

Hint: Ask what 2 * r is the slope per unit of, and what one more unit of w does to this listing's residual, which is negative. For the weights, ask how many square meters one standard deviation of area is.

Q3: Two Partial Derivatives, and a Step That Went Up

At w = 3000, b = 50000 the gradient is [-7,707,309.092, -66,531.617], and a teammate concludes that b hardly matters because its partial derivative is a hundred times smaller. Another teammate takes the one-parameter step with 1e-4, watches the loss rise from 24,320,084,049 to 29,734,705,200 and concludes the slope at 3,000 was wrong. Answer both: say what each partial derivative is per unit of and why their sizes do not compare, why both are zero at lstsq's line, and what a step that raises the loss says about the rate and about the slope.

Hint: Ask what one unit of w does to every listing's prediction and what one unit of b does. For the step, ask how far 3,000 is from the bottom, how far 1e-4 times the slope moved w, and whether the slope changed between the four rates the lesson tried.

Q4: A Zero Check That Passed

Read this script without running it.

import numpy as np
import pandas as pd

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = has_area["area_sqm"].to_numpy()
price = has_area["price"].to_numpy()


def mse(w):
    return np.mean((price - w * area) ** 2)


def slope(w):
    return np.sum(-2 * area * (price - w * area))


w_zero = np.sum(area * price) / np.sum(area ** 2)
print(abs(slope(w_zero)) < 1e-3)
w = 3_000.0 - 1e-5 * slope(3_000.0)
print(round(w, 2), f"{mse(w):,.0f}")

A teammate argues that the first line proves slope is right, since the derivative is zero at lstsq's bottom and so is slope, and that the second line therefore shows the lesson's learning rate of 1e-5 is far too large for this data. Say what the two lines print, using the lesson's numbers, and answer both arguments.

Hint: Compare slope with the lesson's dmse_dw term by term. Then ask what a zero check can see when the only error is a constant factor, and which check in the lesson would have caught it.

Answer 1:

The first number is the forward difference across one unit, the slope of the chord from 3,000 to 3,001, not the slope at 3,000: on a curve that bends upward that chord is flatter than the tangent at its left end by np.mean(area ** 2) * h, 11,746.16 at h = 1, and -17,609,312.570 plus 11,746.16 is -17,597,566. The second is the centered difference at a width float64 cannot support: it misses by 93, the rounding the two losses carry divided by 2e-8. The formula has neither error, and the centered difference at h = 1e-3 sits within 0.004 of it.

Neither misleads a step: both are negative and within a tenth of a percent of the slope, so a step from either lands within a unit of the right landing. Both mislead a check: a formula set beside the first would look wrong in its seventh digit and beside the second in its sixth, and the formula would be "fixed" while it was right.

DON'T judge a slope's error by what a step forgives, a step forgives a tenth of a percent and a check does not, so the measurement that checks a formula has to hold more digits than the step ever needs.

Answer 2:

2 * (p0 - w * a0) is 2 * r, the slope of the square per unit of r, the outer link only. The residual is computed from w, so the chain rule multiplies by the inner slopes, the residual's -1 per unit of the prediction and the prediction's 175.7 per unit of w: -36,200 times -175.7 is 6,360,340. The sign flips because the residual is negative, the model overprices the listing by 18,100, so one more unit of w pushes the prediction further above the price and the squared residual grows. The teammate's number says the loss falls as w grows, and a step taken from it would move w up and make this listing's loss worse.

The weights are the same rule on a different chain. The standardized line reaches the price through z, so its weight is the slope of the price per unit of z, one standard deviation of area, 44.0589 square meters, and the slope of z per square meter is 1 / 44.0589: the product, coef_z[1] / area.std(), 144,879.63 over the computed 44.0589, is 3,288.31 per square meter, the raw line's weight, one line read in two units, with the division run on the computed deviation and never on the printed one.

DON'T read the sign of 2 * r as the sign of the slope with respect to w, the inner slope, -area, is negative for every listing and flips it.

Answer 3:

dL_dw is the change in the loss per unit of w, one more currency unit per square meter, which raises every listing's prediction by its area, 99 units on average, while dL_db is the change per unit of b, one currency unit added to every prediction alike. A hundredfold difference between rates per unit of different things says only that a unit of w moves the predictions about a hundred times as far as a unit of b, nothing about which parameter the fit needs. At lstsq's line both are zero to within 1e-8, because that line is the solution of exactly those two equations, each partial derivative set to zero.

The second teammate's slope was right, -17,609,312.570 under all four rates and within 0.004 of the centered difference. The rate was wrong for the distance: 1e-4 times the slope carried w past the bottom to a point farther from it than 3,000 was, so the loss rose, where the same slope with 1e-5 stopped short of the bottom and lowered it. A rising loss says the step was too long, and only a measurement beside the formula says anything about the slope.

DON'T blame the gradient for what the learning rate did, the slope is checked with a centered difference and the rate with a shorter step.

Answer 4:

True
509267.74 3,001,731,782,025,817

slope is dmse_dw with np.sum in place of np.mean, so it returns 2,875 times the derivative at every w: -50,626,773,640 at 3,000, where the derivative is -17,609,312.570. At the bottom the derivative is 1e-8 and 2,875 times that is 3e-5, still under 1e-3, so the zero check passes. It passes for any constant factor, because a constant times zero is zero, and that is why it proves nothing about the size of slope: the check that would have failed is the one against the centered difference, -17,609,312.574 beside -50,626,773,640, off by a factor of 2,875 at first sight.

The step then moves w by 2,875 times the 176 units the lesson's step moved it, to 509,267.74, where the loss is 3 quadrillion. The learning rate did nothing wrong: 1e-5 with the derivative lands at 3,176.09 and lowers the loss to 21,583,439,257. A zero check tests where the slope vanishes, and a step needs its size.

DON'T accept a gradient on a zero check alone, it cannot see a factor, and the factor sets the size of every step.

Assignment

Assignment: The Slopes of the Absolute and the Huber Loss

The pricing team wants Lesson 3.1.3's two other losses trained by gradient steps in Section 4.2, the mean absolute error and the Huber loss its Assignment built, and asks for their derivatives with respect to w under price = w * area, each checked, on the 2,875 listings with an area. Neither loss squares its residual, so the chain's first link changes. The derivative of abs(r) with respect to r is np.sign(r), 1 where the residual is positive and -1 where it is negative, and it does not exist at r = 0, where the V of the absolute value has a corner with a slope of -1 on one side and 1 on the other. The Huber charge with threshold delta is r ** 2 / 2 inside the threshold and delta * (abs(r) - delta / 2) past it, so its derivative with respect to r is r inside and delta * np.sign(r) past it, and the two meet at the threshold, both delta in size, so this derivative exists everywhere.

In abs_huber_slope.py in ~/dev/bootcamp, write mae(w) and its derivative dmae_dw(w) by the chain rule, and print the derivative, the centered difference and the count of residuals exactly zero at 3,000, 3,634 and 4,000. Explain from np.sign(0) why the two agree at 4,000 although the derivative does not exist there. Then print dmae_dw at 3,630, 3,633, 3,634, 3,635 and 3,640, count the listings whose residual changes sign between 3,633 and 3,635, and explain from the formula why this slope moves in jumps and why its bottom, 3.1.3's 3,634, is where the areas of the listings above the line balance the areas below it.

Write the Huber loss at delta = 150_000 and its derivative, print the derivative beside the centered difference at 3,000 and 4,000, then print the derivative and the loss at 3,665 through 3,668 and say where the bottom is and how that agrees with 3.1.3's scan. Finally take one step from w = 3000 with the learning rate 1e-5 on the squared error, the absolute error and the Huber loss, printing each slope, where it lands and the loss before and after, then move the absolute error with the rates 1.0, 5.0 and 10.0, and explain why the rate that moved the squared error 176 units moved the absolute error by less than a thousandth.

Hints:

np.sign(r) and np.where(np.abs(r) <= delta, r, delta * np.sign(r)) are the two inner slopes, one per listing, and the chain's last link, -area, is the same as the squared error's.
The slope of mae is np.mean(-area * np.sign(r)), so write it as a sum of areas on one side of the line minus a sum of areas on the other, and ask what can change it as w moves by one unit.

Solution:

import numpy as np
import pandas as pd

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = has_area["area_sqm"].to_numpy()
price = has_area["price"].to_numpy()


def centered(f, w, h=1e-3):
    return (f(w + h) - f(w - h)) / (2 * h)


def mae(w):
    return np.mean(np.abs(price - w * area))


def dmae_dw(w):
    residual = price - w * area
    # the sign of r, times the residual's slope per unit of w
    return np.mean(-area * np.sign(residual))


for w in [3_000.0, 3_634.0, 4_000.0]:
    zeros = np.sum(price - w * area == 0)
    print(w, round(dmae_dw(w), 3), round(centered(mae, w), 3), zeros)

3000.0 -43.967 -43.967 0
3634.0 -0.002 -0.002 0
4000.0 22.279 22.279 1

for w in [3_630.0, 3_633.0, 3_634.0, 3_635.0, 3_640.0]:
    print(w, round(dmae_dw(w), 3))
before, after = price - 3_633.0 * area, price - 3_635.0 * area
# 3, the listings the line crosses between 3,633 and 3,635
print(np.sum(np.sign(before) != np.sign(after)))
r = price - 3_634.0 * area
print(np.sum(r > 0), np.sum(r < 0))  # 1536 1339
# 142343.5 142339.1, the areas above the line and below it
print(round(area[r > 0].sum(), 1), round(area[r < 0].sum(), 1))

3630.0 -0.45
3633.0 -0.076
3634.0 -0.002
3635.0 0.128
3640.0 0.328

def huber(residuals, delta):
    size = np.abs(residuals)
    return np.where(size <= delta, residuals ** 2 / 2, delta * (size - delta / 2))


def huber_loss(w, delta=150_000):
    return np.mean(huber(price - w * area, delta))


def dhuber_dw(w, delta=150_000):
    residual = price - w * area
    # r inside the threshold, delta with r's sign past it
    inner = np.where(np.abs(residual) <= delta, residual, delta * np.sign(residual))
    return np.mean(-area * inner)


for w in [3_000.0, 4_000.0]:
    print(w, f"{dhuber_dw(w):,.3f}", f"{centered(huber_loss, w):,.3f}")
for w in [3_665.0, 3_666.0, 3_667.0, 3_668.0]:
    print(w, f"{dhuber_dw(w):,.3f}", f"{huber_loss(w):,.0f}")

3000.0 -5,421,851.363 -5,421,851.364
4000.0 2,497,880.139 2,497,880.141
3665.0 -17,161.010 6,719,765,851
3666.0 -9,375.988 6,719,752,582
3667.0 -1,586.751 6,719,747,100
3668.0 6,211.615 6,719,749,411

def mse(w):
    return np.mean((price - w * area) ** 2)


def dmse_dw(w):
    return np.mean(-2 * area * (price - w * area))


w = 3_000.0
losses = [("mse", mse, dmse_dw), ("mae", mae, dmae_dw), ("huber", huber_loss, dhuber_dw)]
for name, loss, slope in losses:
    d = slope(w)
    w_new = w - 1e-5 * d
    print(name, f"{d:,.3f}", round(w_new, 4), f"{loss(w):,.0f}", f"{loss(w_new):,.0f}")
for lr in [1.0, 5.0, 10.0]:
    w_new = w - lr * dmae_dw(w)
    print(lr, round(w_new, 2), f"{mae(w_new):,.0f}")

mse -17,609,312.570 3176.0931 24,320,084,049 21,583,439,257
mae -43.967 3000.0004 105,881 105,881
huber -5,421,851.363 3054.2185 8,517,429,981 8,235,359,706
1.0 3043.97 104,028
5.0 3219.84 98,101
10.0 3439.67 93,638

At 4,000 one listing's price is exactly 4,000 times its area, so its charge sits on the corner of the V. np.sign(0) is 0, so the formula gives that listing no slope, and the centered difference, straddling the corner, averages the slopes on its two sides, -area and area, to the same 0: the two agree while the derivative does not exist there, so the count of zero residuals is part of the check.

The slope of the absolute error is the sum of the areas below the line minus the sum of the areas above it, divided by 2,875, because each listing's charge changes by its area per unit of w, in the direction its sign gives. Only the side of each residual enters, so the slope changes only when the line crosses a listing: between 3,633 and 3,635 it crosses three and the slope jumps from -0.076 to 0.128, and between 3,000 and 3,001 it crosses none. The bottom is where the two sums balance, 142,343.5 above the line against 142,339.1 below it at 3,634, with 1,536 listings above and 1,339 below: 3.1.3 balanced counts for one price, and a slope through the origin balances areas.

The Huber slope crosses zero between 3,667 and 3,668, from -1,586.751 to 6,211.615, so the bottom lies between them, nearer 3,667, and 3.1.3's scan over whole numbers returned 3,667 because it holds the lowest loss of the four. The three slopes at 3,000 are -17,609,312.570, -43.967 and -5,421,851.363: the squared error's is in squared currency per unit of w and the absolute error's in currency per unit of w, 400,000 times smaller, so the same 1e-5 moves w by 176 units on one loss and by 0.0004 on the other. The absolute error's slope is never larger in size than the mean area, 99.02, so moving w by hundreds of units asks for a rate in the ones: 10.0 moved it 440 units, to 3,439.67, and lowered the loss from 105,881 to 93,638.

DON'T carry a learning rate from one loss to another, a rate is chosen against the slope it will multiply. Don't trust a check at a w where a residual is exactly zero, the formula and the measurement agree there by the convention np.sign(0) == 0, and the derivative does not exist.

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the step repeated into the loop before Lesson 4.2.1 writes it, or the numerical and the analytic gradient set side by side in a course's own notes.

YouTube - "Gradient descent, how neural networks learn | Deep Learning Chapter 2" by 3Blue1Brown. The loss of a digit-recognizing network drawn as a landscape over its 13,002 weights and biases, the gradient as the direction of steepest climb, and the step against it repeated until the loss settles, which is the loop this lesson stopped one step short of.
Stanford CS231n - "Optimization: Stochastic Gradient Descent", the section "Computing the gradient". A numerical gradient written as a forward difference with h = 0.00001 over every parameter, with the note that the centered difference works better in practice, a printed run of step sizes from 1e-10 to 1e-1 on an image classifier's loss, where the loss falls and then overshoots, and the analytic gradient as the exact and fast formula whose correctness a gradient check against the numerical one proves.

Summary

The grid Lesson 3.1.3 scanned was a way of asking the loss 6,001 questions to learn one thing, where its bottom was. The derivative asks one question at one place, which way is down and how steeply, and that turned out to be enough, because a search that can read the slope where it stands never needs to see the whole curve. Everything else in this lesson was about trusting the answer: a measurement anyone can make from the loss alone, a handful of rules that give the slope exactly, and the chain rule that joins the rules across every step a computation takes between a parameter and its loss.

Two kinds of error ran through the lesson, and they are worth keeping apart. One belongs to arithmetic, a width too small, a chord taken for a tangent, a sum where a mean belonged, an outer link taken for the whole chain, and each is caught by setting the formula beside the measurement, which is why the measurement exists. The other belongs to choice: a step that raises the loss with a correct slope is a learning rate's fault, and no check of the slope will find it, because nothing was wrong with the slope.

What the lesson did not do is take a second step. One step from 3,000 showed what the slope buys and what the rate decides, and showed that this loss, a parabola, hands over its bottom to a rate that no network's loss will offer. The loop that repeats the step belongs to Lesson 4.2.1, and the first thing Lesson 4.1.4 does is ask PyTorch for the number this lesson computed by hand, so that the machinery every later lesson leans on is checked once against a derivative you can read line by line.

Cheat Sheet

Part A - Reference Table

Concept	Code	Notes
Derivative at `w`	`dL_dw`	slope of the loss per unit of `w`, sign is the direction, size is the steepness
Centered difference	`(f(w + h) - f(w - h)) / (2 * h)`	the check, `h = 1e-3`, two evaluations of the loss
Forward difference	`(f(w + h) - f(w)) / h`	off by the bend times `h`, never the check
The rules	constant 0, `w` 1, `c * w` is `c`, `w ** 2` is `2 * w`, sum and mean term by term	the data are constants, only `w` moves
Chain rule	`dL_dw = dL_dr * dr_dp * dp_dw`	multiply the links, the outer slope taken at the inner value
Squared-error slope	`np.mean(-2 * area * (price - w * area))`	zero at `np.sum(area * price) / np.sum(area ** 2)`, `lstsq`'s answer
Gradient	`np.array([dL_dw, dL_db])`	one partial derivative per parameter, each with the others held still
Zero check	`np.allclose(gradient(w, b), 0, atol=1e-6)`	at the unrounded number, and blind to a constant factor
One step	`w - lr * dL_dw`	minus is against the slope, a rising loss means the rate is too large
One-step rate	`1 / (2 * np.mean(area ** 2))`	lands on the bottom of this parabola only, a network's loss has none
Absolute-error slope	`np.mean(-area * np.sign(r))`	jumps when the line crosses a listing, undefined at `r == 0`
Huber slope	`np.mean(-area * np.where(np.abs(r) <= delta, r, delta * np.sign(r)))`	continuous at the threshold

Part B - Compact Code Block

import numpy as np


def centered(f, w, h=1e-3):
    return (f(w + h) - f(w - h)) / (2 * h)  # the check, two loss calls


def dmse_dw(w):
    # the chain rule's slopes, averaged over the listings
    return np.mean(-2 * area * (price - w * area))


centered(mse, 3_000)                        # -17,609,312.574, measured
dmse_dw(3_000)                              # -17,609,312.570, exact
np.sum(area * price) / np.sum(area ** 2)    # where the slope is zero
w = w - 1e-5 * dmse_dw(w)                   # one step, against the slope
np.allclose(gradient(w, b), 0, atol=1e-6)   # zero check, blind to a factor
np.mean(-area * np.sign(price - w * area))  # the absolute error's slope

Previous: Lesson 2

Mark complete

Next: Lesson 4

Questions about this lesson: #module-4 on Discord

Found an error in this lesson? Report it

Derivatives and the Chain Rule - Computed by Hand

Context

Goal

Prerequisites

Table of Contents

Theory

The Derivative Is the Slope of the Loss

Measuring a Slope With a Centered Difference

The Rules for Simple Functions

The Chain Rule

The Gradient of the Squared-Error Loss

One Step Against the Gradient

Lab Exercises

Lab Exercise 1: The Slope of the Houses' Loss

Lab Exercise 2: Eastgate's Gradient, and One Step

Theory Assessment

Q1: Three Slopes at 3,000

Q2: The Outer Link Alone, and the Weight That Changed Units

Q3: Two Partial Derivatives, and a Step That Went Up

Q4: A Zero Check That Passed

Assignment

Assignment: The Slopes of the Absolute and the Huber Loss

Additional Resources

Summary

Cheat Sheet