Autograd - Checked Against Your Own Derivative

Context

Autograd is PyTorch's record of every operation performed on a tensor that asked for one, and backward() plays that record in reverse to compute the derivative of a loss with respect to every parameter in it. The derivative Lesson 4.1.3 built by hand, one rule per operation and the chain rule between them, comes out of one call, for that loss and for any computation you can write in PyTorch.

Every training loop in Modules 4 to 6 is four lines around it: a forward pass that builds the loss, backward(), a step on each parameter inside torch.no_grad(), and a gradient cleared before the next pass. The same four lines fit the one-neuron price model of Lesson 4.2.1, classify which listings sell within a month in Section 4.3, and train the image network of Section 4.5, whose thousands of parameters nobody differentiates by hand. A loss read for a curve or a prediction handed to matplotlib leaves autograd through .item() or .detach() first.

The failures are quiet. A gradient that was never cleared doubles with every backward() while the loss keeps moving, a parameter reassigned instead of updated in place stops receiving gradients without a word, and a numerical check run in float32 reports a mismatch against a gradient that is right. Lesson 4.1.3 computed the derivative of mse(w) at w = 3000 as -17,609,312.570 and took one step from it. This lesson asks autograd for the same number, checks it to the digit, takes the same step, and meets each of those failures on the way.

Goal

• Build a leaf tensor with requires_grad=True, trace the graph a forward pass records through each tensor's grad_fn, and tell a leaf from a computed tensor.
• Call backward(), read .grad, and check it against Lesson 4.1.3's hand derivative with torch.allclose, for one parameter and for two.
• Diagnose the doubled gradient that two backward() calls leave in .grad, and clear it with zero_() or None.
• Take one gradient step inside torch.no_grad() that keeps the parameter a leaf, and leave autograd through .item() and .detach() without a warning or a refusal.
• Verify a gradient numerically with torch.autograd.gradcheck in float64, and explain why the same check fails in float32 while float32 autograd still agrees with the hand derivative to rtol=1e-4.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see an autograd engine built from nothing, or PyTorch's own first walk through the calls, before you start.

• YouTube - "The spelled-out intro to neural networks and backpropagation: building micrograd" by Andrej Karpathy. A small autograd engine written from scratch in Python: a value that remembers the two inputs and the operation that made it, a backward that applies the chain rule node by node from the output back to the inputs, and the same calls then made in PyTorch. Everything before 1:43:55, where a neural net library begins, covers what this lesson does with requires_grad and backward().
• Official docs - PyTorch Tutorials, "Automatic Differentiation with torch.autograd". The sections "Tensors, Functions and Computational graph", "Computing Gradients" and "Disabling Gradient Tracking" on a one-layer example, with the same grad_fn, backward(), .grad and torch.no_grad() this lesson uses on the listings.

Table of Contents

• requires_grad and the Recorded Graph
• backward and .grad, Checked by Hand
• Gradients Accumulate
• The Step: no_grad, detach and item
• Checking Autograd Numerically

Theory

Every Theory block in this lesson runs in ~/dev/bootcamp on the 2,875 listings with an area, Lesson 3.1.3's rows and Lesson 4.1.3's, held as float64 tensors so that every gradient matches 4.1.3's hand derivative to the digit. The last subtitle switches the same model to float32, the dtype the module trains in from Lesson 4.2.1 on. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

requires_grad and the Recorded Graph

What. Autograd (PyTorch's recording of every operation performed on a tensor that asks for it, played backwards to compute derivatives) is switched on per tensor with requires_grad=True. A tensor created that way is a leaf tensor (a tensor you made yourself rather than computed from others, where the record starts and where gradients land), and every tensor computed from it carries a grad_fn, the record of the operation that made it. Why. Lesson 4.1.3 differentiated mse(w) by hand, one rule per operation and the chain rule between them. A network has thousands of parameters and dozens of operations between them and its loss, and the record lets PyTorch apply the same rules to any computation you write, in the order you wrote it, without you writing a derivative. When. On every parameter a model will learn, and on nothing else. The data tensors are read, never learned, and stay out of the record.

Lesson 4.1.1's torch.tensor on a NumPy column keeps NumPy's float64, and the price column takes dtype=torch.float64 because the module keeps every tensor in a regression loss in one float dtype: its integers would otherwise arrive as int64, a dtype a parameter can never have, as the fourth subtitle shows. The one parameter, w, is created with the request:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)
w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)

print(area.dtype, price.dtype)           # torch.float64 torch.float64
print(w)                                 # tensor(3000., dtype=torch.float64, requires_grad=True)
print(w.is_leaf, w.grad)                 # True None
print(area.requires_grad, area.grad_fn)  # False None
print((area * 2).grad_fn)                # None

The data tensors asked for nothing: their requires_grad is False, their grad_fn is None, and an operation between tensors that ask for nothing records nothing, so (area * 2).grad_fn is None too. The record begins with the first operation that touches w:

pred = w * area
residual = price - pred
squared = residual ** 2
loss = squared.mean()

print(pred.grad_fn.name())               # MulBackward0
print(residual.grad_fn.name())           # SubBackward0
print(squared.grad_fn.name())            # PowBackward0
print(loss.grad_fn.name())               # MeanBackward0
print(pred.is_leaf, pred.requires_grad)  # False True
print(loss)                              # tensor(2.4320e+10, dtype=torch.float64, grad_fn=<MeanBackward0>)
print(f"{loss.item():,.0f}")             # 24,320,084,049

Four operations ran, and four nodes were recorded. Together they are the computational graph (the record of operations from the leaves to the loss, one node per operation, each holding what it needs to compute that operation's derivative). Every tensor the forward pass computed carries the node that made it in grad_fn, named after the operation with Backward0 on the end, and a tensor's own print shows the same name at its end, grad_fn=<MeanBackward0>. pred requires a gradient because w does, and it is not a leaf, because it was computed rather than created. The loss itself is Lesson 3.1.3's mse(3000), 24,320,084,049, now with a record attached.

The model Lesson 4.1.3 took its two-parameter gradient on adds a bias and one more node:

b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)
scaled = w * area
pred = scaled + b
residual = price - pred
squared = residual ** 2
loss = squared.mean()

for t in [scaled, pred, residual, squared, loss]:
    print(t.grad_fn.name(), t.is_leaf)
print(f"{loss.item():,.0f}")

MulBackward0 False
AddBackward0 False
SubBackward0 False
PowBackward0 False
MeanBackward0 False
18,493,503,179

The next subtitle walks this record backwards, and the figure below is that walk drawn out: the forward pass left to right, each node's own derivative multiplied into the next from right to left, and what .grad holds after one walk and after two.

DON'T:

• DON'T set requires_grad=True on the data. area.requires_grad_(True) records every operation on the areas too, and backward() then fills area.grad with 2,875 derivatives of the loss with respect to the areas, one per listing, numbers nothing will ever step.
• DON'T look for a grad_fn on a leaf. w.grad_fn is None, and that None is what marks a leaf, the one kind of tensor whose .grad backward() fills.

backward and .grad, Checked by Hand

What. loss.backward() walks the recorded graph from the loss back to every leaf, multiplying each node's own derivative into the product so far, Lesson 4.1.3's chain rule, and leaves each leaf's result in its .grad. torch.autograd.grad(loss, [w, b]) makes the same walk and returns the results instead of writing them. Why. The number in w.grad is a claim about your loss, and the first time you use autograd on a loss you can differentiate by hand, you check the claim. Once it holds to the digit on mse(w), you can trust it on Section 4.3's losses, which nobody differentiates by hand. When. Once per forward pass, on a scalar loss. The walk spends the graph, and the next forward pass builds a new one.

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)
w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)

loss = ((price - w * area) ** 2).mean()
loss.backward()
print(w.grad)                   # tensor(-17609312.5704, dtype=torch.float64)
print(f"{w.grad.item():,.3f}")  # -17,609,312.570

# 4.1.3's hand derivative, mean(-2 * area * (price - w * area)), in NumPy
area_np, price_np = area.numpy(), price.numpy()
hand = np.mean(-2 * area_np * (price_np - 3000.0 * area_np))
print(f"{hand:,.3f}")           # -17,609,312.570
check = torch.allclose(w.grad, torch.tensor(hand))
print(check)                    # True

The comparison is torch.allclose, never ==, because the two routes sum 2,875 terms in different orders and may differ in their last digits. .numpy() on the data tensors works because neither asked for a gradient, a distinction the fourth subtitle returns to.

Two parameters, two leaves, and a gradient with two entries:

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)
loss = ((price - (w * area + b)) ** 2).mean()

grads = torch.autograd.grad(loss, [w, b])
print(grads)                    # (tensor(-7707309.0922, dtype=torch.float64), tensor(-66531.6174, dtype=torch.float64))
print(w.grad, b.grad)           # None None

loss = ((price - (w * area + b)) ** 2).mean()
loss.backward()
print(f"{w.grad.item():,.3f}")  # -7,707,309.092
print(f"{b.grad.item():,.3f}")  # -66,531.617

hand_w = np.mean(-2 * area_np * (price_np - (3000.0 * area_np + 50000.0)))
hand_b = np.mean(-2 * (price_np - (3000.0 * area_np + 50000.0)))
# True True
print(torch.allclose(w.grad, torch.tensor(hand_w)), torch.allclose(b.grad, torch.tensor(hand_b)))

torch.autograd.grad returns a tuple with one tensor per input, in the order the inputs were given, and writes nothing. It spends the graph exactly as backward() does, which is why the loss is built a second time before backward() runs. backward() leaves 4.1.3's gradient split across the two leaves, each .grad equal to its partial derivative computed by hand.

backward() has two refusals worth reading once:

pred = w * area + b
try:
    pred.backward()
except RuntimeError as err:
    print(err)
try:
    loss.backward()
except RuntimeError as err:
    print(err)

grad can be implicitly created only for scalar outputs
Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.

The second message is the graph being spent: loss was already walked, and the values the walk saved, the residuals PowBackward0 needs for 2 * residual, were freed at its end. The fix is a new forward pass. retain_graph=True, which the message offers, keeps a graph for a second walk, and the next subtitle shows what a second walk does to .grad.

A computed tensor has no .grad of its own, and reading it warns, in nonleaf.py:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)
w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)

pred = w * area
loss = ((price - pred) ** 2).mean()
loss.backward()
print(pred.grad)

/home/goku/dev/bootcamp/nonleaf.py:13: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the .grad field to be populated for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more information. (Triggered internally at /__w/pytorch/pytorch/build/aten/src/ATen/core/TensorBody.h:488.)
  print(pred.grad)
None

The walk passed through pred on its way to w and kept nothing there, because only leaves collect gradients. The warning names the way to ask for it, retain_grad(), called on the computed tensor before backward():

pred = w * area
pred.retain_grad()
loss = ((price - pred) ** 2).mean()
loss.backward()
print(pred.grad[:2])    # tensor([ 12.5913, -53.9130], dtype=torch.float64)
print(pred.grad.shape)  # torch.Size([2875])

pred.grad is the derivative of the loss with respect to each prediction, -2 * (price - pred) / 2875, one value per listing. The first listing, 175.7 square meters listed at 509,000, is predicted at 527,100, a residual of -18,100, so 2 * 18,100 / 2,875 is 12.5913: raising that prediction raises the loss. The second, 30.5 square meters at 169,000, is predicted at 91,500, a residual of 77,500, and its -53.9130 says raising that prediction lowers it. These are the numbers that reach MulBackward0, with SubBackward0's -1 already multiplied in, and it multiplies each by its listing's area and sums the 2,875 products into w.grad.

DON'T:

• DON'T call backward() on a tensor with more than one value. pred.backward() is refused with the scalar-outputs message, and the loss, a scalar, is what has one derivative per parameter.
• DON'T answer the second-backward() message with retain_graph=True. It is for a graph you need to walk twice, and a new step wants a new forward pass. The next subtitle shows the two walks adding up.
• DON'T read a computed tensor's .grad. It is None under a warning however many times backward() runs, and retain_grad() before backward() is how to ask for it when a per-prediction derivative is what you want.

Gradients Accumulate

What. Gradient accumulation (.grad adding each backward()'s result to what it already holds, instead of replacing it) is autograd's default. w.grad.zero_() or w.grad = None clears it. Why. A loop calls backward() once per step, and a .grad that was never cleared holds the sum of every gradient since the start, so each step moves by that sum. Nothing raises, the loss keeps changing, and the only sign is a gradient twice the size it should be after the second step. Seeing it once here is why Lesson 4.2.1's loop clears its gradients on purpose. When. Before every backward() whose gradient is meant to stand alone, which in this module is every one.

Same w, same loss, same derivative, three calls:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)

def mse(w):
    return ((price - w * area) ** 2).mean()

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
mse(w).backward()
print(f"{w.grad.item():,.3f}")  # -17,609,312.570
mse(w).backward()
print(f"{w.grad.item():,.3f}")  # -35,218,625.141
mse(w).backward()
print(f"{w.grad.item():,.3f}")  # -52,827,937.711

A step taken from the third value would move w three times as far as the loss asked, and nothing would have warned. Two lines clear it, and both work:

print(w.grad.zero_())           # tensor(0., dtype=torch.float64)
mse(w).backward()
print(f"{w.grad.item():,.3f}")  # -17,609,312.570

w.grad = None
print(w.grad)                   # None
mse(w).backward()
print(f"{w.grad.item():,.3f}")  # -17,609,312.570

zero_() is Lesson 4.1.2's in-place method: it writes zeros into the gradient tensor and returns it. Setting .grad to None throws the tensor away, and the next backward() creates a new one, which is what happened to the first backward() of this script. retain_graph=True, the option the second-backward() message offered, keeps one graph alive for a second walk, and the two walks still add:

w.grad = None
loss = mse(w)
loss.backward(retain_graph=True)
loss.backward()
print(f"{w.grad.item():,.3f}")  # -35,218,625.141

Two parameters accumulate the same way, each in its own .grad, which is the right half of the figure:

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)

def mse_wb(w, b):
    return ((price - (w * area + b)) ** 2).mean()

mse_wb(w, b).backward()
# -7,707,309.092 -66,531.617
print(f"{w.grad.item():,.3f} {b.grad.item():,.3f}")
mse_wb(w, b).backward()
# -15,414,618.184 -133,063.235
print(f"{w.grad.item():,.3f} {b.grad.item():,.3f}")

PyTorch adds rather than replaces because some loops want the sum: Lesson 4.3.3 splits the listings into mini-batches (small groups of rows the loop handles one at a time), and a loop that wants the gradient of several batches at once gets it by not clearing between them. Everywhere else in this module the sum is the trap.

DON'T:

• DON'T read .grad as the gradient at the current w. After three backward() calls it held three times the derivative, while w sat at 3,000 the whole time.
• DON'T take retain_graph=True as a way to get the same gradient twice for free. Two walks of one graph left the doubled value, exactly as two forward passes did.
• DON'T zero a gradient that does not exist yet. On a fresh leaf w.grad is None, and w.grad.zero_() raises AttributeError: 'NoneType' object has no attribute 'zero_', so the first clearing comes after the first backward(), or sets w.grad = None, which is always allowed.

The Step: no_grad, detach and item

What. torch.no_grad() (a context inside which no operation is recorded) is where a parameter is updated: w -= lr * w.grad inside it moves the leaf in place and keeps it a leaf. .item() turns a one-value tensor into a Python number, .detach() returns a tensor that shares the numbers and carries no graph, and .detach().numpy() is the way back to NumPy and matplotlib. Why. The update is arithmetic on w, and autograd records arithmetic on w. Recorded, the step becomes one more node, and w, now a computed tensor, receives no gradient at the next backward(). The step has to happen off the record, which is all no_grad does. When. Every parameter update, every forward pass that needs no gradient, Lesson 4.2.3's validation pass among them, and every number that leaves autograd for a print, a curve or a NumPy array.

Lesson 4.1.3's step, w - 1e-5 * dL_dw, written on the leaf as it stands:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)

def mse(w):
    return ((price - w * area) ** 2).mean()

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
mse(w).backward()
try:
    w -= 1e-5 * w.grad
except RuntimeError as err:
    print(err)  # a leaf Variable that requires grad is being used in an in-place operation.
print(w)        # tensor(3000., dtype=torch.float64, requires_grad=True)

The in-place step is refused, and w is unchanged. The refusal is the loud failure, and the obvious rewrite is the quiet one. old keeps a second name on the leaf here only so the script can show where the gradient goes:

old = w
w = w - 1e-5 * w.grad
print(w)                          # tensor(3176.0931, dtype=torch.float64, grad_fn=<SubBackward0>)
print(w.is_leaf)                  # False
mse(w).backward()
print(f"{old.grad.item():,.3f}")  # -31,081,790.165

The next backward() walks through the computed w to the leaf it was computed from, the tensor old still names, and adds the derivative at 3,176.09 to the -17,609,312.570 that leaf already held, the -31,081,790.165 printed. w.grad is None under the warning the second subtitle showed, after this backward() and every later one. In a loop with no old, that leaf has no name, every gradient lands on it, and the parameter the loop steps never receives one. The step belongs inside no_grad:

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
mse(w).backward()
with torch.no_grad():
    w -= 1e-5 * w.grad
print(w)                                   # tensor(3176.0931, dtype=torch.float64, requires_grad=True)
print(f"{w.item():,.2f}")                  # 3,176.09
print(w.is_leaf)                           # True
print(f"{w.grad.item():,.3f}")             # -17,609,312.570

with torch.no_grad():
    after = mse(w)
print(after.requires_grad, after.grad_fn)  # False None
print(f"{after.item():,.0f}")              # 21,583,439,257

Inside the context the in-place step runs, nothing is recorded, and w comes out a leaf standing where Lesson 4.1.3's step left it, with 4.1.3's loss after the step. A tensor computed under no_grad carries no grad_fn and does not require a gradient, which is what a loss computed only to be printed should look like. w.grad keeps its value after the step: the step reads .grad and never clears it, so the clearing line of the previous subtitle is a separate line, and the loop that repeats this step is Lesson 4.2.1's, not written here.

A loss leaves autograd through .item(), and a tensor with more than one value through .detach(). The other routes warn or refuse, in leave.py:

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)
w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
pred = w * area
loss = ((price - pred) ** 2).mean()

print(f"{loss.item():,.0f}")  # 24,320,084,049
print(type(loss.item()))      # <class 'float'>
print(f"{float(loss):,.0f}")

24,320,084,049
<class 'float'>
/home/goku/dev/bootcamp/leave.py:14: UserWarning: Converting a tensor with requires_grad=True to a scalar may lead to unexpected behavior.
Consider using tensor.detach() first. (Triggered internally at /__w/pytorch/pytorch/torch/csrc/autograd/generated/python_variable_methods.cpp:820.)
  print(f"{float(loss):,.0f}")
24,320,084,049

float(loss) reaches the same number under a warning, once per program, that names the fix. For pred, 2,875 values, .numpy() refuses outright, and .detach() is the door:

try:
    pred.numpy()
except RuntimeError as err:
    print(err)
plain = pred.detach()
print(plain.requires_grad, plain.grad_fn)
print(plain.numpy()[:2])

Can't call numpy() on Tensor that requires grad. Use tensor.detach().numpy() instead.
False None
[527100.  91500.]

plain shares pred's numbers and carries no graph: no grad_fn, no requires_grad, so .numpy() takes it, and pred.detach().numpy() is the one line this module uses to hand a prediction to NumPy. Detaching has a place, and that place is after the loss is built. Detached inside the forward pass, it cuts the chain:

broken = ((price - pred.detach()) ** 2).mean()
print(broken.requires_grad, broken.grad_fn)  # False None
try:
    broken.backward()
except RuntimeError as err:
    print(err)                               # element 0 of tensors does not require grad and does not have a grad_fn

Nothing between broken and w is recorded, because pred.detach() has no record and price never had one, so broken has no grad_fn and backward() has nowhere to walk. The same message meets a model whose parameters never asked for a gradient. One more gate sits at creation, on what a parameter may be:

try:
    rooms = torch.tensor(has_area["rooms"].to_numpy(), requires_grad=True)
except RuntimeError as err:
    print(err)  # Only Tensors of floating point and complex dtype can require gradients

An integer column arrives as int64, and an integer tensor cannot require a gradient, because a derivative is a rate of change and integers do not change by small amounts. A parameter is always a float tensor, and this module converts a target or a feature to float before it enters a regression loss, so the loss holds one dtype.

DON'T:

• DON'T update a leaf outside torch.no_grad(). In place it raises, and reassigned it runs, turns w into a computed tensor that no backward() will ever fill, and sends every later gradient to a leaf nothing names.
• DON'T take a number out with float(loss). It warns, and the warning names the fix: .item() on a one-value tensor, .detach() on anything bigger.
• DON'T detach() inside the forward pass. The loss built from pred.detach() has no grad_fn, and its backward() raises the does-not-require-grad message. Detach what leaves autograd, after the loss is built.

Checking Autograd Numerically

What. torch.autograd.gradcheck(f, inputs) computes Lesson 4.1.3's centered difference for every input of f, with a step of 1e-6 each way, and compares it with autograd's gradient under allclose at rtol=1e-3 and atol=1e-5, its defaults. It returns True, or raises a GradcheckError whose message prints both values. Why. This lesson checked autograd against a derivative you had computed by hand. A loss you cannot differentiate by hand still has a centered difference, so the numerical check is the one that stays available for every loss in this module, and it has one precondition this subtitle shows: float64 inputs. When. Whenever a loss or an operation is new to you, on float64 copies of the inputs, and never in float32.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)

def mse_wb(w, b):
    return ((price - (w * area + b)) ** 2).mean()

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)
# True
print(torch.autograd.gradcheck(mse_wb, (w, b)))

gradcheck calls mse_wb with w nudged up and down by 1e-6, then b, forms the two centered differences, computes autograd's gradient on a forward pass of its own, and compares. True says the numerical gradient and autograd's agree within its tolerances at this point, for both inputs, without a hand formula anywhere. Section 4.2 trains in float32, PyTorch's default dtype, so the next question is what autograd gives there:

area32, price32 = area.float(), price.float()
w32 = torch.tensor(3000.0, requires_grad=True)
loss32 = ((price32 - w32 * area32) ** 2).mean()
loss32.backward()
print(loss32)    # tensor(2.4320e+10, grad_fn=<MeanBackward0>)
print(w32.grad)  # tensor(-17609312.)

exact = (-2 * area * (price - 3000.0 * area)).mean()
# tensor(True)
print(torch.isclose(w32.grad.double(), exact, rtol=1e-4))
try:
    torch.isclose(w32.grad, exact, rtol=1e-4)
except RuntimeError as err:
    print(err)   # Float did not match Double

isclose wants both tensors in one dtype, so the float32 gradient is converted up with .double() first, and the rtol=1e-4 is the margin this module allows a float32 gradient against its float64 value. That is exact enough to train with, and it is the dtype every model from Lesson 4.2.1 on trains in. It is not exact enough for gradcheck:

def mse32(w):
    return ((price32 - w * area32) ** 2).mean()

w32 = torch.tensor(3000.0, requires_grad=True)
try:
    torch.autograd.gradcheck(mse32, (w32,))
except RuntimeError as err:
    print(err)

/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/torch/autograd/gradcheck.py:2124: UserWarning: Input #0 requires gradient and is not a double precision floating point or complex. This check will likely fail if all the inputs are not of double precision floating point or complex. 
  _check_inputs(tupled_inputs)
Jacobian mismatch for output 0 with respect to input 0,
numerical:tensor([[0.]])
analytical:tensor([[-17609312.]])

The warning comes from PyTorch's own gradcheck.py, not from your script, and says the check will likely fail. It does, and the message ends with a line break of its own, so print leaves an empty line after it. GradcheckError is a RuntimeError, which is why the except catches it. The analytical gradient is autograd's, the one that matched the hand derivative a block ago, so nothing is wrong with autograd. The numerical one, the centered difference, is what failed, and the reason is float32's resolution:

# tensor(True)
print(torch.tensor(3000.000001) == torch.tensor(3000.0))
w64 = torch.tensor(3000.0, dtype=torch.float64)
print(w64 + 1e-6 == w64)                   # tensor(False)
print(torch.finfo(torch.float32).eps)      # 1.1920928955078125e-07
print(torch.tensor(24320084049.0))         # tensor(2.4320e+10)
print(torch.tensor(24320084049.0).item())  # 24320083968.0

gradcheck nudges w by 1e-6, and in float32 the number 3,000.000001 is the number 3,000: eps, the gap between 1 and the next float32, is about 1.2e-7, and the gap grows with the number, so near 3,000 it is far wider than the nudge. The nudged w equals the original, the two losses are identical, and their difference is 0. In float64 the same nudge registers, which is why the first gradcheck passed. The loss has the same problem at its own size: 24,320,084,049 stored in float32 is 24,320,083,968, because near 24 billion float32's neighbors are thousands apart, so even a nudge that registered would move the loss by less than one gap. No float32 centered difference is printed in this module, because its last digits depend on the order the machine sums 2,875 terms in. The check runs in float64, and training runs in float32, where autograd's gradient is within 1e-4 of it.

DON'T:

• DON'T run gradcheck on float32 tensors. The warning says it will likely fail and it does, with the Jacobian mismatch above and nothing wrong in autograd.
• DON'T compare a float32 gradient with a float64 tensor directly. torch.isclose(w32.grad, exact) raises the dtype mismatch, so convert with .double() first and pass rtol=1e-4.
• DON'T quote a float32 centered difference to the digit. Its last digits move with the machine's summation order, so this module prints a float32 gradient as PyTorch prints it or rounded through .item(), and checks it with isclose.

Lab Exercises

Lab Exercise 1: The Rooms Model's Graph, Checked by Hand

In rooms_graph.py, load listings.csv and build float64 tensors of every listing's rooms and price, all 3,000 rows, since no listing lacks a room count. Create two leaves, w at 80000.0 per room and b at 100000.0, both float64 with requires_grad=True, and compute the squared-error loss in five named steps, scaled, pred, residual, squared and loss. Print each step's grad_fn.name() beside its is_leaf, then the loss in whole units with a thousands separator.

Get the two gradients first with torch.autograd.grad, printing the tuple it returns and then what w.grad and b.grad hold afterwards. Then build the loss again, call backward(), print both gradients to three decimals, and check each against Lesson 4.1.3's hand formulas computed in NumPy, mean(-2 * rooms * (price - (w * rooms + b))) and mean(-2 * (price - (w * rooms + b))), with torch.allclose. End with the two refusals, a second backward() on the same loss and a backward() on pred, each caught and its message printed.

Lab Exercise 2: One Step for Two Parameters

In two_step.py, on the 2,875 listings with an area as float64 tensors, start the two-parameter model at w = 3000.0 and b = 50000.0 and write mse_wb(w, b) for its loss. Print the loss in whole units, call backward(), and print both gradients to three decimals. Call backward() on a fresh loss without clearing anything and print both gradients again, with a comment saying what they hold now. Clear both with zero_(), printing what zero_() returns, and compute the gradients once more.

Take one step for both parameters with a learning rate of 1e-5 inside torch.no_grad(), then print the new w and b to three decimals, whether each is still a leaf, and the loss after the step in whole units, computed under no_grad and read with .item(). Print the first three predictions after the step as a NumPy array rounded to whole units, through detach(). Finally run torch.autograd.gradcheck on mse_wb at the stepped parameters and print its result, then print both gradients once more to show what the step and the check left in .grad.

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
rooms = torch.tensor(listings["rooms"].to_numpy(), dtype=torch.float64)
price = torch.tensor(listings["price"].to_numpy(), dtype=torch.float64)
w = torch.tensor(80000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(100000.0, dtype=torch.float64, requires_grad=True)

scaled = w * rooms
pred = scaled + b
residual = price - pred
squared = residual ** 2
loss = squared.mean()
for t in [scaled, pred, residual, squared, loss]:
    print(t.grad_fn.name(), t.is_leaf)
print(f"{loss.item():,.0f}")

grads = torch.autograd.grad(loss, [w, b])
print(grads)
print(w.grad, b.grad)

MulBackward0 False
AddBackward0 False
SubBackward0 False
PowBackward0 False
MeanBackward0 False
20,503,643,000
(tensor(46462., dtype=torch.float64), tensor(9407.3333, dtype=torch.float64))
None None

loss = ((price - (w * rooms + b)) ** 2).mean()
loss.backward()
print(f"{w.grad.item():,.3f}")  # 46,462.000
print(f"{b.grad.item():,.3f}")  # 9,407.333

rooms_np, price_np = rooms.numpy(), price.numpy()
pred_np = 80000.0 * rooms_np + 100000.0
hand_w = np.mean(-2 * rooms_np * (price_np - pred_np))
hand_b = np.mean(-2 * (price_np - pred_np))
# True True
print(torch.allclose(w.grad, torch.tensor(hand_w)), torch.allclose(b.grad, torch.tensor(hand_b)))

try:
    loss.backward()
except RuntimeError as err:
    print(err)
try:
    pred.backward()
except RuntimeError as err:
    print(err)

Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
grad can be implicitly created only for scalar outputs

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)
w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)

def mse_wb(w, b):
    return ((price - (w * area + b)) ** 2).mean()

before = mse_wb(w, b)
print(f"{before.item():,.0f}")  # 18,493,503,179
before.backward()
# -7,707,309.092 -66,531.617
print(f"{w.grad.item():,.3f} {b.grad.item():,.3f}")
mse_wb(w, b).backward()
# -15,414,618.184 -133,063.235, each the sum of two walks
print(f"{w.grad.item():,.3f} {b.grad.item():,.3f}")
# tensor(0., dtype=torch.float64) tensor(0., dtype=torch.float64)
print(w.grad.zero_(), b.grad.zero_())
mse_wb(w, b).backward()
# -7,707,309.092 -66,531.617
print(f"{w.grad.item():,.3f} {b.grad.item():,.3f}")

with torch.no_grad():
    w -= 1e-5 * w.grad
    b -= 1e-5 * b.grad
print(f"{w.item():,.3f} {b.item():,.3f}")  # 3,077.073 50,000.665
print(w.is_leaf, b.is_leaf)                # True True

with torch.no_grad():
    after = mse_wb(w, b)
print(f"{after.item():,.0f}")              # 17,969,218,178
print(np.round((w * area + b).detach().numpy()[:3]))
print(torch.autograd.gradcheck(mse_wb, (w, b)))
print(f"{w.grad.item():,.3f} {b.grad.item():,.3f}")

[590642. 143851. 491868.]
True
-7,707,309.092 -66,531.617

Theory Assessment

Q1: Leaves, Records and Where the Gradient Lands

In loss = ((price - w * area) ** 2).mean(), with w a float64 tensor created with requires_grad=True and area and price plain float64 tensors, say which tensors are leaves, which carry a grad_fn and what each one is named, and why area.grad_fn is None. Then say which tensors have a .grad filled after loss.backward(), and what torch.autograd.grad(loss, [w]) would have changed about that.

Hint: Ask of each tensor whether you created it or computed it, and whether anything in its history asked for a gradient. For the last part, ask what each of the two calls does with the result of the walk.

Q2: Three Calls, One Running Total

A script builds mse(w) at w = 3000 and calls backward() on three fresh forward passes with nothing between them. Say what w.grad holds after each call and why that is the default rather than a defect. Name the two lines that clear it, and explain why one graph walked twice under retain_graph=True leaves the same -35,218,625.141 as two forward passes.

Hint: .grad is a running total, and the question is what a walk adds to it. For the last part, ask whether the second walk computes anything different from the first.

Q3: Three Ways to Move w, and Four Ways Out

Explain what each of these does to w, a leaf with a gradient in .grad: w -= 1e-5 * w.grad, then w = w - 1e-5 * w.grad, then the first line inside with torch.no_grad():. Say which one a loop must use and what goes wrong with each of the other two. Then say what loss.item(), float(loss), pred.numpy() and pred.detach().numpy() each produce on tensors that require a gradient.

Hint: One raises, one runs and breaks something quietly, one is right. For the second part, ask which calls hand back a Python number, which a NumPy array, and which refuse or warn because a graph is attached.

Q4: What the Second backward() Prints

Read this script without running it, then say what the two print lines show, using this lesson's numbers.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)

def mse(w):
    return ((price - w * area) ** 2).mean()

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
mse(w).backward()
with torch.no_grad():
    w -= 1e-5 * w.grad
loss = mse(w)
loss.backward()
print(f"{w.grad.item():,.3f}")
print(w.is_leaf, loss.requires_grad)

A teammate says the first line prints the derivative of the loss at 3,176.09, and the second prints False True, because the in-place step made w a computed tensor. Explain where each conclusion goes wrong.

Hint: Count the backward() calls and the clearing lines. For the second line, ask what no_grad did to the step, and whether mse(w) was computed inside or outside it.

-31,081,790.165
True True

w.grad.zero_()
mse(w).backward()
print(f"{w.grad.item():,.3f}")  # -13,472,477.594

Assignment

Assignment: Backward in Two Stages

Your team's loss code is about to grow past anything you can differentiate by hand, and before it does you want to see, once, that backward() is nothing but Lesson 4.1.3's chain rule carried out along the recorded graph. Write two_stage.py on the 2,875 listings with an area, as float64 tensors, with the two-parameter model at w = 3000.0 and b = 50000.0.

backward() on a tensor with more than one value takes an argument this lesson did not use: gradient=, a tensor of the same shape as the tensor you call it on, holding the derivative of the final scalar with respect to each of that tensor's values. loss.backward() with no argument is the same call with that argument filled in by the implicit 1, the derivative of the loss with respect to itself. Autograd multiplies whatever you hand it through the rest of the graph, exactly as it multiplies its own intermediate results.

Stage one, by hand: the derivative of the mean squared error with respect to each prediction is -2 * (price - pred) / n, the outer factor of 4.1.3's chain rule. Compute it as a tensor that carries no graph, print its first two values, and say in a comment why the first is positive and the second negative. Stage two, by autograd: call pred.backward(gradient=...) with it, print w.grad and b.grad to three decimals, and check both with torch.allclose against what loss.backward() leaves on a fresh graph after the gradients are cleared.

Then clear the gradients, build pred again and call pred.backward(gradient=torch.ones_like(pred)). Print both gradients as PyTorch prints them and explain, in a comment, which two quantities of the listings they are and of which scalar they are the derivatives. Show what happens when gradient has the wrong shape, three ones, caught and printed.

Last, repeat the two-parameter backward() in float32, the dtype the module trains in, print both gradients through .item() in scientific notation with four decimals, and check each against the float64 gradient with torch.isclose at rtol=1e-4, converting dtypes so the comparison runs.

Hints:

• (-2 * (price - pred) / n).detach() is stage one, with n as len(area). pred has to be rebuilt before each backward(), because every walk spends its graph.
• With gradient=torch.ones_like(pred) you are differentiating pred.sum(): ask what the derivative of w * area + b summed over the listings is with respect to w and to b. isclose refuses a float32 against a float64 tensor, so call .double() on the float32 gradient first.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float64)
n = len(area)
w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)

def forward():
    return w * area + b

# stage one, by hand: the derivative of the loss with respect to each prediction
pred = forward()
dL_dpred = (-2 * (price - pred) / n).detach()
print(dL_dpred[:2])             # tensor([ 47.3739, -19.1304], dtype=torch.float64)
# The first listing is predicted at 577,100 against 509,000, so raising
# its prediction raises the loss. The second is predicted at 141,500
# against 169,000, so raising its prediction lowers the loss.

# stage two, by autograd: hand that derivative to the graph to multiply through
pred.backward(gradient=dL_dpred)
print(f"{w.grad.item():,.3f}")  # -7,707,309.092
print(f"{b.grad.item():,.3f}")  # -66,531.617

two_stage = torch.stack([w.grad, b.grad])
w.grad, b.grad = None, None
loss = ((price - forward()) ** 2).mean()
loss.backward()
# True
print(torch.allclose(two_stage, torch.stack([w.grad, b.grad])))

w.grad, b.grad = None, None
pred = forward()
pred.backward(gradient=torch.ones_like(pred))
print(w.grad)  # tensor(284682.6000, dtype=torch.float64)
print(b.grad)  # tensor(2875., dtype=torch.float64)
# A gradient of ones differentiates pred.sum(). Its derivative with
# respect to w is the sum of the areas, 284,682.6 square meters, and
# with respect to b it is the number of listings, 2,875.

pred = forward()
try:
    pred.backward(gradient=torch.ones(3, dtype=torch.float64))
except RuntimeError as err:
    print(err)

exact_w, exact_b = torch.autograd.grad(((price - forward()) ** 2).mean(), [w, b])
area32, price32 = area.float(), price.float()
w32 = torch.tensor(3000.0, requires_grad=True)
b32 = torch.tensor(50000.0, requires_grad=True)
loss32 = ((price32 - (w32 * area32 + b32)) ** 2).mean()
loss32.backward()
print(f"{w32.grad.item():.4e} {b32.grad.item():.4e}")
print(torch.isclose(w32.grad.double(), exact_w, rtol=1e-4))
print(torch.isclose(b32.grad.double(), exact_b, rtol=1e-4))

Mismatch in shape: grad_output[0] has a shape of torch.Size([3]) and output[0] has a shape of torch.Size([2875]).
-7.7073e+06 -6.6532e+04
tensor(True)
tensor(True)

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the practical rules of a numerical gradient check beyond the one gradcheck ran here, or the full list of what that function can be told.

• Course notes - Stanford CS231n, "Neural Networks Part 3: Learning and Evaluation", the section "Gradient checks". Why the centered formula beats the forward one, why the comparison is a relative error and not a difference, why the check runs in double precision, and what a kink such as max(0, x) does to a numerical derivative.
• Official docs - PyTorch 2.14, torch.autograd.gradcheck.gradcheck. The full signature with eps, atol, rtol and raise_exception, and the note that the defaults are designed for double precision inputs and will likely fail on a float tensor.

Summary

Autograd turned out to be bookkeeping. A tensor that asked for a gradient started a record, every operation on it added a node, and backward() read the record from the end, multiplying each node's own derivative into the running product until the leaves. Nothing in that walk was new: the derivative of a square, of a mean, of a product by the area, and the chain rule between them were Lesson 4.1.3's, and the number that came out was 4.1.3's number to the digit. What changed is who does the arithmetic.

The lesson's four failures were all about what the record does not do for you. It does not clear .grad, so the sum stands in for the gradient until a line clears it. It does not know a step from a computation, so a step on the record turns the parameter into a computed tensor that gradients pass through. It does not let the numbers out, so a loss is read through .item() and a prediction through .detach(). And it does not change what float32 can resolve, so the numerical check runs in float64 while training runs in float32, each in the dtype that suits it.

Everything the next section builds is this lesson in a loop: forward pass, backward(), step under no_grad, clear, with the leaves and the step wrapped in the objects Lesson 4.2.2 introduces. The wrapping hides the four lines, and it hides none of the four failures, which is why they were met here, on one parameter, with a derivative you had already computed by hand to compare against.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import torch

w = torch.tensor(3000.0, dtype=torch.float64, requires_grad=True)
b = torch.tensor(50000.0, dtype=torch.float64, requires_grad=True)
loss = ((price - (w * area + b)) ** 2).mean()
loss.grad_fn.name()              # MeanBackward0, the record's last node
loss.backward()                  # fills w.grad and b.grad, spends the graph
w.grad, b.grad                   # the gradient, one tensor per leaf

with torch.no_grad():            # the step, off the record
    w -= 1e-5 * w.grad
    b -= 1e-5 * b.grad
w.grad, b.grad = None, None      # clear before the next backward()

loss.item()                      # a Python float, no warning
(w * area + b).detach().numpy()  # the array for NumPy or matplotlib

def mse_wb(w, b):
    return ((price - (w * area + b)) ** 2).mean()

torch.autograd.gradcheck(mse_wb, (w, b))   # True, float64 inputs only
torch.autograd.grad(mse_wb(w, b), [w, b])  # the pair, nothing written
g32 = w32.grad.double()                    # float32 up to float64
torch.isclose(g32, w.grad, rtol=1e-4)      # the module's float32 margin