Tensor Operations and the Device Pattern

Context

A tensor computes the way a NumPy array computes: one result per element, a smaller shape stretched to meet a larger one, a whole dimension collapsed into one number, and a table multiplied into a vector of weights. Each operation returns a tensor whose dtype and shape follow rules, and a device is the one property of a result that an array never had.

Every layer Section 4.3 builds is X @ W + b on a batch of listings, every loss Section 4.2 writes is a table of residuals reduced to one number, and every standardized feature is a column broadcast against its mean and its standard deviation. The device line sits at the top of every training script in this module and in Section 4.5's image work, and .to(device) follows every tensor a script creates.

The failures are quiet. A float64 column that slips into a float32 model promotes everything it touches, a (2875, 1) column of predictions against a (2875,) vector of prices is a (2875, 2875) table whose mean looks right, the standard deviation PyTorch prints disagrees with the one scikit-learn's rule gives for the same column, and view is refused on the very table the module builds from a frame. Lesson 4.1.1 built tensors from the listings, read their dtype and shape, and shared memory with NumPy. This lesson computes with them, and names the device they compute on.

Goal

• Predict the dtype of a result between tensors of different dtypes, or a tensor and a Python number, and keep the module's data in float32 with .float().
• Predict a broadcast result's shape before trusting a loss, and diagnose the (2875, 1) against (2875,) table by its shape rather than its mean.
• Reduce a tensor along a dim, unpack the pair max(dim=0) returns, and state what std() divides by against NumPy's np.std.
• Price every listing at once with X @ w + b, and read the three refusals a matrix product raises.
• Distinguish a view from a copy and an in-place method from a new tensor, and explain why view refuses the table built from a frame.
• Write the device line once, move every tensor through .to(device), and bring a result back to NumPy through .cpu().numpy().

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see tensors broadcast and move between devices before you type any of it.

• YouTube - "Introduction to PyTorch Tensors" by PyTorch, the chapter "Broadcasting rules" from 9:38, then "Mathematical functions", "Linear algebra" and "In-place operations", and from 26:13 "Moving to GPU". Each is run in a notebook on tensors a few elements long, which this lesson repeats on the 2,875 listings.
• Official docs - PyTorch 2.14, "Broadcasting semantics", the section "General semantics". The rule as PyTorch states it, trailing dimensions first, with the RuntimeError a failed pair raises quoted, and below it the section "In-place semantics", where an in-place operation may not grow the tensor it writes into.

Table of Contents

• Elementwise Operations and Type Promotion
• Broadcasting, and the Shape That Grows
• Reductions and the dim Argument
• Matrix Multiplication, X @ w + b
• Views, Reshapes and In-Place Operations
• The Device Pattern

Theory

Every block below works on the 2,875 listings with an area, has_area, with area, price and the (2875, 2) table X of area and rooms built as Lesson 4.1.1 built them, and nothing installs here. A Python block that opens with its imports is a script of its own, and a block that does not continues the script above it.

Elementwise Operations and Type Promotion

What. An elementwise operation applies one function to every element and returns a tensor of the same shape: +, -, *, /, ** and the comparisons, between a tensor and a number or between two tensors of one shape, and functions such as torch.sqrt, torch.log, torch.exp, torch.abs and torch.clamp. Type promotion (the rule that decides the dtype of a result whose inputs have different dtypes) settles the result's dtype before any value is computed. Why. Lesson 2.2.3's rule carries over whole, and so does its silence: nothing warns when a float64 column meets a float32 model, the result is float64 and so is every tensor it touches afterwards. A model in this module computes in float32, and the one dtype rule that bites is the one that quietly leaves it. When. Every prediction, every residual and every loss in this module is elementwise work before it is anything else, and the dtype is checked whenever a tensor arrives from a frame, a NumPy array or a Python number.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)

pred = area * 3000.0 + 50000.0  # 3.1.1's w * area + b on every listing at once
resid = price - pred
print(pred[:4])
print(resid[:4])
print((resid ** 2)[:4])
print(torch.abs(resid)[:4])
print(torch.sqrt(area)[:4])
print(torch.log(price)[:4])
print(torch.clamp(area, 50.0, 100.0)[:4])
print((pred > price).sum())

tensor([577100.0000, 141500.0000, 480800.0312, 176000.0000])
tensor([-68100.0000,  27500.0000, 101199.9688, -83000.0000])
tensor([4.6376e+09, 7.5625e+08, 1.0241e+10, 6.8890e+09])
tensor([ 68100.0000,  27500.0000, 101199.9688,  83000.0000])
tensor([13.2552,  5.5227, 11.9833,  6.4807])
tensor([13.1402, 12.0377, 13.2742, 11.4404])
tensor([100.,  50., 100.,  50.])
tensor(1330)

Every line returned 2,875 values, four shown, and no loop ran. The third prediction prints 480800.0312 where Lesson 3.1.1's float64 printed 480800., float32's seven digits from Lesson 4.1.1 carried through the multiplication, and the squared residuals, in the billions, print in scientific notation. torch.clamp pins every area into the range 50 to 100, and pred > price is a bool tensor as Lesson 4.1.1's masks were, so its sum counts the listings the line overprices, 1,330 of 2,875.

A result's dtype follows from its inputs, and three rules cover every case this module meets:

rooms = torch.tensor(has_area["rooms"].to_numpy())
area64 = torch.tensor(has_area["area_sqm"].to_numpy())
print(rooms.dtype, area64.dtype)  # torch.int64 torch.float64
print((area + 0.5).dtype)         # torch.float32
print((rooms * 1.5).dtype)        # torch.float32
print((rooms + 1).dtype)          # torch.int64
print((rooms * area).dtype)       # torch.float32
print((area + area64).dtype)      # torch.float64
print((rooms / 2).dtype)          # torch.float32
print((rooms // 2).dtype)         # torch.int64
print((rooms / rooms).dtype)      # torch.float32

A tensor meeting a Python number keeps its own dtype when the number is of its kind, and takes the default float, float32, when an integer tensor meets a float, so rooms * 1.5 is float32 where NumPy gives float64. Two tensors of different kinds take the dtype of the higher kind, rooms * area is float32. Two tensors of one kind take the wider one, so float32 with float64 is float64, the rule that matters: area64, built without dtype=, carries NumPy's float64 into every result it touches. / is true division and never returns an integer: two integer tensors, or an integer tensor and a Python integer, give the default float, float32, and a float tensor divided by an integer tensor or a Python number keeps its own dtype, so area64 / 2 is float64, while // keeps the integer.

A float64 column needs no error to take over:

pred64 = area64 * 3000.0 + 50000.0
print(pred64.dtype)                    # torch.float64
print((price - pred64).dtype)          # torch.float64
print((price - pred64.float()).dtype)  # torch.float32
print((area - area64.mean()).dtype)    # torch.float32

pred64 is float64 because area64 was, the residual against the float32 prices is float64, and .float() on the column, or dtype=torch.float32 when it is built, is how the module keeps its data in one dtype. The last line is the one exception worth knowing: area64.mean() is a float64 scalar tensor, and a scalar tensor does not promote a tensor that has dimensions when both are floats, so the result stays float32.

DON'T:

• DON'T let a column built without dtype= into a float32 computation. area64 * 3000.0 + 50000.0 is float64, so is every residual and loss computed from it, and nothing says so. .float() at the door, or dtype=torch.float32 when the tensor is built.
• DON'T expect an integer from rooms / 2. True division never returns an integer, rooms / 2 is float32, and rooms // 2 is the division that keeps torch.int64.

Broadcasting, and the Shape That Grows

What. Broadcasting stretches a smaller shape to meet a larger one, by Lesson 2.2.4's rules and no others: shapes are lined up on the right, two sizes pair when they are equal or one of them is 1, a missing dimension counts as 1, and the result takes the larger size at every position. torch.broadcast_shapes applies the rule to shape tuples without building a tensor, as np.broadcast_shapes did. Why. The rule is what lets a (2,) vector of means meet a (2875, 2) table, and it is also what turns a (2875, 1) column against a (2875,) vector into a (2875, 2875) table without a word. The second case is the one that reaches Lesson 4.2.2's loss, so a shape is printed before a loss is trusted. When. Every time two tensors of different shapes meet in an arithmetic line, which in this module is every standardization, every bias added to a product and every residual.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)

mu = X.mean(dim=0)        # one mean per column, dim is axis
print(X.shape, mu.shape)  # torch.Size([2875, 2]) torch.Size([2])
print((X - mu).shape)     # torch.Size([2875, 2])
# torch.Size([2875, 2]), from the two shapes alone
print(torch.broadcast_shapes(X.shape, mu.shape))

dim is PyTorch's name for Lesson 2.2.5's axis, the dimension that disappears, so X.mean(dim=0) is the (2,) vector of column means. Lined up on the right, (2875, 2) meets (2,) as (2875, 2) against (1, 2): the 2s pair, the missing dimension counts as 1 and is reused down 2,875 rows, and the result is the table's own shape with each column's mean subtracted.

The shape that grows comes from Lesson 2.4.1's two ways of selecting a column:

pred = area * 3000.0 + 50000.0
price_col = torch.tensor(has_area[["price"]].to_numpy(), dtype=torch.float32)
print(price_col.shape, pred.shape)  # torch.Size([2875, 1]) torch.Size([2875])

table = price_col - pred
print(table.shape)                  # torch.Size([2875, 2875])
print(table[:3, :3])

tensor([[ -68100.0000,  367500.0000,   28199.9688],
        [-408100.0000,   27500.0000, -311800.0312],
        [   4900.0000,  440500.0000,  101199.9688]])

has_area[["price"]] is the one-column DataFrame, so price_col is (2875, 1) where price is (2875,). Lined up on the right, (2875, 1) meets (2875,) as (2875, 1) against (1, 2875): 1 pairs with 2875 in both positions, both tensors are stretched, and the result is a (2875, 2875) table of every price minus every prediction, over eight million numbers where 2,875 were meant. The diagonal holds the residuals, -68100, 27500 and 101199.9688 from the first subtitle, and everything off it pairs one listing's price with another listing's prediction.

print(round(table.mean().item()))           # 33266
print(round((price - pred).mean().item()))  # 33266
print(f"{(table ** 2).mean().item():.4e}")  # 5.6793e+10
# 1.8494e+10, the mean squared residual, which the table's is not
print(f"{((price - pred) ** 2).mean().item():.4e}")

The table's mean is 33,266, and so is the mean residual, which is what makes the trap quiet: the mean of every price minus every prediction is the mean price minus the mean prediction, the diagonal's own number, as Lesson 2.2.4 found on its (4, 4) table. Square the table before the mean, as a mean squared error does, and the agreement is gone, 5.6793e+10 against the true 1.8494e+10, three times the loss, with no error anywhere. The fix is the target's shape, not a call: has_area["price"], the Series, gives the (2875,) tensor that pairs with a (2875,) prediction, and when a prediction arrives as a column, as Lesson 4.3.1's layer returns it, the fifth subtitle's squeeze and unsqueeze move between the two forms.

Two sizes that are neither equal nor 1 cannot pair, and the refusal is the good case:

try:
    torch.ones(3) + torch.ones(4)
except RuntimeError as err:
    print(err)

The size of tensor a (3) must match the size of tensor b (4) at non-singleton dimension 0

PyTorch refuses with a RuntimeError naming both sizes and the dimension they sit in, where NumPy's ValueError named both whole shapes. It stops the script at the line with the wrong shape, where the (2875, 2875) table runs on.

DON'T:

• DON'T trust a loss whose shape you never printed. has_area[["price"]] builds the column and has_area["price"] the vector, and torch.broadcast_shapes on the two shapes predicts the table before any arithmetic runs.
• DON'T read the broadcasting refusal as a fault in the data. It names the two sizes that could not pair and the dimension they sit in, and the fix is a shape, never a row.

Reductions and the dim Argument

What. A reduction collapses a tensor to fewer numbers: sum, mean, min, max, argmin, argmax, std and var, over the whole tensor when called bare and along one dim otherwise. keepdim=True keeps the collapsed dimension as a size of 1, so the result still broadcasts against the table it came from. Why. Every loss in this module is a reduction, the mean of a table of residuals, and so is every statistic a feature is standardized with. Two of PyTorch's defaults differ from NumPy's, max along a dim returns two tensors and std divides by one less than the count, and both have to be known before a number is set beside one Module 3 printed. When. Whenever a table becomes a number per column, a number per listing or one number, and whenever a tensor's statistic is compared with a NumPy or scikit-learn one.

import numpy as np
import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)

print(X.sum(dim=0).round())               # tensor([284683.,  10241.])
print(X.mean(dim=0))                      # tensor([99.0200,  3.5621])
print(X.sum(dim=1).shape)                 # torch.Size([2875])
print(X.mean(dim=0, keepdim=True).shape)  # torch.Size([1, 2])

print(X.max())     # tensor(278.5000)
print(X.argmax())  # tensor(4648)
print(X.max(dim=0))

torch.return_types.max(
values=tensor([278.5000,  10.0000]),
indices=tensor([2324, 1805]))

values, rows = X.max(dim=0)
print(values)               # tensor([278.5000,  10.0000])
print(rows)                 # tensor([2324, 1805])
print(X.argmax(dim=0))      # tensor([2324, 1805])
print(X[2324])              # tensor([278.5000,  10.0000])
print(4648 // 2, 4648 % 2)  # 2324 0

argmax(dim=0) is the indices half by itself.

print(area.std())                  # tensor(44.0666)
np_std = np.std(has_area["area_sqm"].to_numpy())
print(round(float(np_std), 4))     # 44.0589
print(area.std(correction=0))      # tensor(44.0589)
print(X.std(dim=0, correction=0))  # tensor([44.0589,  1.7025])
# 1941.19, the square of 44.0589 under the same correction
print(round(area.var(correction=0).item(), 2))

area.std() prints 44.0666 and NumPy's np.std of the same column prints 44.0589, because PyTorch's correction defaults to 1, dividing by 2,874, and NumPy's ddof to 0, dividing by 2,875, Lesson 2.2.5's two spreads under new names. area.std(correction=0) is NumPy's number and follows the rule Lesson 3.2.1's StandardScaler follows, dividing by the count of the rows it was fitted on: on its 2,300 training rows its scale_ for area was 44.54, and on all 2,875 listings the same rule gives 44.0589, so a tensor standardized to match a scikit-learn pipeline passes correction=0 and uses the rows the scaler saw. var is the square of std under the same correction.

DON'T:

• DON'T unpack X.max(dim=0) as one tensor. It is a pair, values, rows = X.max(dim=0) takes both and .values the values alone, and X.max(dim=0) * 2 raises nothing, because a pair times 2 is the pair repeated, four tensors where two numbers were wanted.
• DON'T compare area.std() with NumPy's np.std and call one of them wrong. correction=1 against ddof=0 is the whole difference, and correction=0 divides by the count, as StandardScaler does.
• DON'T read X.argmax() as a row number. 4648 is a position in the flattened (2875, 2) table, and X.argmax(dim=0) is the row of the largest value in each column.

Matrix Multiplication, X @ w + b

What. X @ w is Lesson 2.3.2's matrix product on tensors: (n, k) @ (k,) gives (n,), the shared k disappears, and (n, k) @ (k, 1) gives (n, 1), one column of predictions. torch.matmul(X, w) is the same operation as a function, torch.mm is its two-matrix form, and torch.dot, or @ between two vectors, is the dot product. Why. A linear layer in Section 4.3 computes X @ W + b on a batch of listings and nothing else, Module 2's promise, so every shape rule of this product is a shape rule of every network in the module. The product also refuses in three ways, each with a message worth knowing on sight. When. Every prediction of a linear model on a table of listings, every layer of every network, and every check that two routes to one product agree.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)

w_vec = torch.tensor([3000.0, 10000.0])  # one weight per column, (2,)
b = 50000.0
print((X @ w_vec).shape)                 # torch.Size([2875])
print((X @ w_vec + b)[:3])               # tensor([637100., 151500., 530800.])

w = torch.tensor([[3000.0], [10000.0]])  # the same weights as a (2, 1) column
pred = X @ w + b
print(pred.shape)                        # torch.Size([2875, 1])
print(pred[:3, 0])                       # tensor([637100., 151500., 530800.])

Both products price the first listing at 637,100: 175.7 square meters at 3,000 each, six rooms at 10,000 each, and the bias. The vector form returns (2875,) as it did in Lesson 2.3.2, and the column form returns (2875, 1), the shape Lesson 4.3.1's layer returns and the one that met the (2875,) prices in the second subtitle, which is why pred[:3, 0] indexes the column out before printing.

try:
    X @ torch.ones(3, 1)
except RuntimeError as err:
    print(err)
try:
    X @ w.double()
except RuntimeError as err:
    print(err)
try:
    torch.mm(area, price)
except RuntimeError as err:
    print(err)
try:
    X * w
except RuntimeError as err:
    print(err)

mat1 and mat2 shapes cannot be multiplied (2875x2 and 3x1)
expected m1 and m2 to have the same dtype, but got: float != double
self must be a matrix
The size of tensor a (2875) must match the size of tensor b (2) at non-singleton dimension 0

Four refusals, three of them the product's own. float and double in the second are PyTorch's names for float32 and float64, and no promotion rescues the pair: elementwise arithmetic promotes, a matrix product refuses, so the float64 column of the first subtitle stops a model here where it slipped through there. The fourth refusal is broadcasting's: X * w is elementwise, (2875, 2) against (2, 1), and the pair fails where @ would have run.

by_matmul = torch.matmul(X, w)
by_mm = torch.mm(X, w)
print(torch.allclose(X @ w, by_matmul))  # True
print(torch.allclose(X @ w, by_mm))      # True
print(area @ price)                      # tensor(1.2662e+11)
print(torch.dot(area, price))            # tensor(1.2662e+11)
print((area * price).sum())              # tensor(1.2662e+11)

@, torch.matmul and torch.mm agree on two matrices within torch.allclose (True when every pair of values agrees within a small tolerance), the comparison for floats, as np.allclose was in Module 2. Between two vectors @ is Lesson 2.3.1's dot product, 1.2662e+11 for the areas against the prices, the same number torch.dot and the elementwise product's sum return, printed as PyTorch prints a float32 reduction of that size.

DON'T:

• DON'T multiply a float32 table by float64 weights. X @ w.double() is refused with float != double, and w.float(), or dtype=torch.float32 when the weights are built, is the fix.
• DON'T write * for @. Against the (2,) vector w_vec, X * w_vec runs and returns the (2875, 2) table of contributions, whose .sum(dim=1) is the product, Lesson 2.3.2's rebuild, never the product itself.

Views, Reshapes and In-Place Operations

What. unsqueeze, squeeze, view, reshape and .T return a tensor of another shape over the same elements, and a view (a tensor that reads and writes the memory of the tensor it came from) is what each of them returns when the memory allows it. An in-place operation writes its result into the tensor it was called on: the methods that end in an underscore, add_, sub_, mul_ and zero_, and the augmented assignments +=, -= and *=. Why. A (2875,) vector and a (2875, 1) column are the two forms every prediction and target in this module take, and moving between them is a shape call. A view that writes through to its parent and an in-place method that changes a tensor another name still points at are Lesson 2.2.2's slice and Lesson 2.2.3's += again, and zero_() is the method Lesson 4.1.4 clears a gradient with. When. Whenever a shape has to change without copying 2,875 listings, and whenever a tensor is updated where it stands, which every gradient step in Section 4.2 does.

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
area = torch.tensor(has_area["area_sqm"].to_numpy(), dtype=torch.float32)
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)

col = area.unsqueeze(1)           # a dimension of size 1 added at position 1
print(col.shape)                  # torch.Size([2875, 1])
print(area[:, None].shape)        # torch.Size([2875, 1])
print(col.squeeze(1).shape)       # torch.Size([2875])
print(area.reshape(-1, 1).shape)  # torch.Size([2875, 1])
print(area.view(5, -1).shape)     # torch.Size([5, 575])
print(X.T.shape)                  # torch.Size([2, 2875])

col[0, 0] = 999.0
print(area[:3])  # tensor([999.0000,  30.5000, 143.6000])
col[0, 0] = 175.7

col is a view. Writing into its first element changed area, the way Lesson 2.2.2's slice wrote through to its parent, and nothing was copied.

print(X.is_contiguous())  # False
print(X.stride())         # (1, 2875)
try:
    X.view(-1)
except RuntimeError as err:
    print(err)

view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

view is refused on the table the module builds from a frame, and the reason is where its numbers sit in memory. A stride (how many elements memory skips to move one step along a dimension) of (1, 2875) says that one step down a column moves one element and one step across a row moves 2,875: the table is stored column after column, because pandas keeps a DataFrame column by column, to_numpy() hands that layout out and torch.tensor copies the numbers in the layout it was given. A contiguous tensor (one stored row after row, so that its memory read in order is its rows in order) has strides (2, 1) at this shape. view only reads the same memory under a new shape, so a shape that would read the elements out of order is refused, and reshape returns a view when it can and a copy when it cannot:

flat = X.reshape(-1)
print(flat[:4])             # tensor([175.7000,   6.0000,  30.5000,   1.0000])
flat[0] = 999.0
print(X[0, 0])              # tensor(175.7000)
print(X.T.is_contiguous())  # True
Xc = X.contiguous()
print(Xc.stride())          # (2, 1)
print(Xc.view(-1)[:4])      # tensor([175.7000,   6.0000,  30.5000,   1.0000])

X.T of this table is contiguous, its strides (2875, 1) read the memory in order, and .contiguous() returns a row-ordered copy whose view(-1) is allowed.

w = torch.tensor([3000.0, 10000.0])
same = w.add_(500.0)
print(same is w)  # True
print(w)          # tensor([ 3500., 10500.])
new = w + 500.0
print(new is w)   # False
w -= 1000.0
print(w)          # tensor([2500., 9500.])
print(w.zero_())  # tensor([0., 0.])

An in-place result has to fit the dtype the tensor already has, Lesson 2.2.3's casting rule with PyTorch's message:

rooms = torch.tensor(has_area["rooms"].to_numpy())
try:
    rooms += 0.5
except RuntimeError as err:
    print(err)

result type Float can't be cast to the desired output type Long

rooms += 0.5 would need a float result in an int64 tensor, Long in the message, and is refused, where rooms + 0.5 builds a float32 tensor. Two names on one tensor make an in-place update reach both, which is the trap += sets:

price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
pred = price      # a second name, not a copy
pred -= 1000.0
print(price[:3])  # tensor([508000., 168000., 581000.])

DON'T:

• DON'T call view on a tensor whose memory is not in row order and read the refusal as a bug. X.contiguous().view(-1) or X.reshape(-1) is the fix, both copies here, because this table is column-major.
• DON'T update in place where a copy was wanted. pred = price then pred -= 1000.0 lowers every price, because the two names hold one tensor, and pred = price - 1000.0 is the copy.
• DON'T add a float into an integer tensor in place. rooms += 0.5 is refused, result type Float can't be cast to the desired output type Long, and rooms.float() first, or rooms + 0.5 as a new tensor, is the fix.

The Device Pattern

What. Every tensor carries its device in .device: cpu, the processor, or an accelerator, cuda for an NVIDIA GPU. The device pattern names the device once, device = torch.device("cuda" if torch.cuda.is_available() else "cpu"), and moves every tensor the script creates through .to(device), so the one line that names it is the only line that changes between machines. Why. The processor build this module installed has no CUDA in it, torch.cuda.is_available() is False, and every lab in this module runs on the processor in minutes. The same scripts run on a machine with a GPU without an edit, PyTorch's documentation says, because nothing in them names "cuda" except the device line. When. At the top of every script that builds tensors, in this module and in every one after it, and every time a tensor is created from data, with .to(device) or device=device.

import pandas as pd
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)                             # cpu

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)
X = X.to(device)
w = torch.tensor([[3000.0], [10000.0]], device=device)
pred = X @ w + 50000.0
print(X.device, w.device, pred.device)    # cpu cpu cpu
print(X.to(device) is X)                  # True
print(torch.cuda.is_available())          # False
print(torch.cuda.device_count())          # 0
print(torch.backends.mps.is_available())  # False

torch.cuda.device_count() counts the GPUs PyTorch can see, none here. The documentation of Tensor.to says that a tensor already on the requested device is returned as it is and that otherwise a copy on that device is returned, so on a machine with a GPU and the CUDA build the same .to(device) returns a copy of X in the GPU's memory, a sentence this lesson takes from the documentation because no machine here can run it. A Mac with Apple's chip offers a second device, mps, which PyTorch's documentation says torch.backends.mps.is_available() reports there, False here, and nothing in this module requires it.

try:
    X.cuda()
except AssertionError as err:
    print(err)
try:
    torch.zeros(3, device="cuda")
except AssertionError as err:
    print(err)
print(torch.device("cuda"))

Torch not compiled with CUDA enabled
Torch not compiled with CUDA enabled
cuda

A direct request for a GPU is refused with an AssertionError, PyTorch's message for a build that has no CUDA in it, whether the request is .cuda(), .to("cuda") or a tensor created with device="cuda". torch.device("cuda") itself constructs, because a device object is a label and nothing is placed until a tensor is, so a script that names "cuda" in a second place fails at that place and not at the top.

back = pred.cpu()
print(back is pred)               # True
print(pred.cpu().numpy()[:3, 0])  # [637100. 151500. 530800.]
print(X.device == "cpu")          # False
print(X.device.type == "cpu")     # True

The documentation of Tensor.numpy says the conversion is performed only if the tensor is on the CPU, so a tensor on a GPU comes back through .cpu() first, another sentence this lesson states from the documentation and the reason the pattern ends with both calls even where the first does nothing. A device compared with a string is False even when the two name the same thing, and .type is the string to compare.

DON'T:

• DON'T name the device in two places. A .cuda() or a device="cuda" anywhere below the device line is the line that fails on the processor build, Torch not compiled with CUDA enabled, and the line that stops the same script from running on the next machine.
• DON'T compare a device with a string. X.device == "cpu" is False on a processor tensor, and X.device.type == "cpu" or X.device == torch.device("cpu") is the test.

Lab Exercises

Lab Exercise 1: Standardize Area and Rooms by Hand

In standardize.py, build the (2875, 2) float32 table X of area_sqm and rooms from the listings with an area, and print its shape and dtype. Compute one mean per column and print it, then print the standard deviation per column twice, with PyTorch's default and with correction=0, and say in a comment which of the two follows Lesson 3.2.1's StandardScaler and why.

Standardize the table, (X - mean) / std with the correction=0 spread, print its shape and its first two rows, prove with torch.allclose and atol=1e-4, the absolute difference it allows, that every column's mean is now zero, print every column's correction=0 spread, and count the listings more than 3 standard deviations from the mean in each column.

Then make two mistakes on purpose. Subtract the per-listing mean, X.mean(dim=1), inside a try and print the message. Repeat it with keepdim=True on both the mean and the spread, which runs, print the result's shape and its first two rows, and say in a comment what the two numbers in every row are. Finally compute the means in float64 with X.double().mean(dim=0), standardize with them, print the dtype of the result, bring it back to float32 and prove with torch.allclose that it matches your float32 result.

Lab Exercise 2: Price Every Listing on the Device

In price_table.py, write the device line first and print the device. Build the (2875, 2) table of area and rooms and the vector of prices as float32 tensors, move both through .to(device), and print whether .to(device) on the table returns the same object.

Build the weights 3000.0 for the area and 10000.0 for the rooms as a (2, 1) column created on the device, set the bias to 50000.0, predict every price with one matrix product, and print the prediction's shape and its first three values as a flat vector. Print the shape of the prediction minus the price vector and say in a comment why it is wrong, then compute the residual with the right shape, print that shape, and print the mean squared residual in scientific notation with four decimals. Multiply the table by the weights converted to float64 inside a try and print the message.

Take a flat view of the weights, set its first element to 3500.0 and print the weights as a flat vector, then subtract 500.0 from the weights in place and print the flat view. Print whether the table is contiguous, call view(-1) on it inside a try, print the message, and print the first four values of reshape(-1) instead. Finally ask the prediction for .cuda() inside a try, print the message, and print the first three predictions as a NumPy array through .cpu().numpy().

import pandas as pd
import torch

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)
print(X.shape, X.dtype)  # torch.Size([2875, 2]) torch.float32

mu = X.mean(dim=0)
print(mu)                # tensor([99.0200,  3.5621])
print(X.std(dim=0))      # tensor([44.0666,  1.7028])
sigma = X.std(dim=0, correction=0)
print(sigma)             # tensor([44.0589,  1.7025])
# correction=0 divides by the count, 2,875, the rule StandardScaler followed in 3.2.1,
# where PyTorch's default correction=1 divides by 2,874

Z = (X - mu) / sigma     # (2875, 2) against (2,), twice
print(Z.shape)           # torch.Size([2875, 2])
print(Z[:2])

tensor([[ 1.7404,  1.4319],
        [-1.5552, -1.5049]])

centered = torch.allclose(Z.mean(dim=0), torch.zeros(2), atol=1e-4)
print(centered)
print(Z.std(dim=0, correction=0))
print((Z.abs() > 3).sum(dim=0))

try:
    X - X.mean(dim=1)
except RuntimeError as err:
    print(err)
# keepdim=True keeps a (2875, 1) shape, which broadcasts across the two columns
wrong_mu = X.mean(dim=1, keepdim=True)
wrong_sigma = X.std(dim=1, keepdim=True, correction=0)
wrong = (X - wrong_mu) / wrong_sigma
print(wrong.shape)
print(wrong[:2])

# every row is 1 and -1: each listing standardized against its own two
# numbers, and its area is always the larger of the two

True
tensor([1., 1.])
tensor([16, 15])
The size of tensor a (2) must match the size of tensor b (2875) at non-singleton dimension 1
torch.Size([2875, 2])
tensor([[ 1., -1.],
        [ 1., -1.]])

mu64 = X.double().mean(dim=0)
Z64 = (X - mu64) / sigma
print(Z64.dtype)                       # torch.float64
print(torch.allclose(Z64.float(), Z))  # True

import pandas as pd
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)                               # cpu

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
X = torch.tensor(has_area[["area_sqm", "rooms"]].to_numpy(), dtype=torch.float32)
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)
X = X.to(device)
price = price.to(device)
print(X.to(device) is X)                    # True

w = torch.tensor([[3000.0], [10000.0]], device=device)
b = 50000.0
pred = X @ w + b
print(pred.shape)                           # torch.Size([2875, 1])
print(pred[:3, 0])                          # tensor([637100., 151500., 530800.])
print((pred - price).shape)                 # torch.Size([2875, 2875])
# (2875, 1) against (2875,) pairs every prediction with every price
resid = pred.squeeze(1) - price
print(resid.shape)                          # torch.Size([2875])
print(f"{(resid ** 2).mean().item():.4e}")  # 1.7331e+10

try:
    X @ w.double()
except RuntimeError as err:
    print(err)

expected m1 and m2 to have the same dtype, but got: float != double

flat = w.view(-1)    # a (2,) view of the (2, 1) column
flat[0] = 3500.0
print(w.squeeze(1))  # tensor([ 3500., 10000.])
w.sub_(500.0)        # in place, so flat sees it too
print(flat)          # tensor([3000., 9500.])

print(X.is_contiguous())
try:
    X.view(-1)
except RuntimeError as err:
    print(err)
print(X.reshape(-1)[:4])

False
view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
tensor([175.7000,   6.0000,  30.5000,   1.0000])

try:
    pred.cuda()
except AssertionError as err:
    print(err)
print(pred.cpu().numpy()[:3, 0])

Torch not compiled with CUDA enabled
[637100. 151500. 530800.]

Theory Assessment

Q1: Dtypes and Shapes Before Running

Without running anything, give the dtype of each result: area * 2 and area + 0.5 for a float32 area, rooms / 2 and rooms // 2 for an int64 rooms, area + area64 for a float64 area64 of the same length, and area - area64.mean(). Then give the shape, or the refusal, of X - X.mean(dim=0) for the (2875, 2) table, of price_col - pred for a (2875, 1) column and a (2875,) vector, and of torch.ones(3) + torch.ones(4), and say which of the three a printed mean would catch.

Hint: For the dtypes, sort each pair into a tensor with a Python number, two tensors of different kinds, and two tensors of one kind, and remember what / always returns. For the shapes, line the two shapes up on the right and pair them position by position.

Q2: One Table, Three Maximums

Explain what X.max(), X.argmax() and X.max(dim=0) each return for the (2875, 2) table of area and rooms, how 4648 becomes a row and a column, and why values, rows = X.max(dim=0) works where the same unpacking of NumPy's np.max(X, axis=0) would not. Then explain why area.std() prints 44.0666 where np.std of the same column prints 44.0589, and which call divides by the count, as StandardScaler does.

Hint: Count how many tensors each call returns, and ask whether a position is counted through a flattened table. For the spread, ask what each library divides by.

Q3: Views, Copies and the Table From a Frame

A teammate runs col = area.unsqueeze(1), then col[0, 0] = 0.0, and is surprised that area[0] is now 0.. They then run flat = X.reshape(-1) on the table built from has_area[["area_sqm", "rooms"]], set flat[0] = 0.0, and are surprised that X[0, 0] is still 175.7. Explain both surprises, say what X.view(-1) would have done in place of reshape, and say what pred = price followed by pred -= 1000.0 does to price.

Hint: Ask whether each call shares memory with its parent, and what decides whether reshape can. For the last part, ask whether pred = price built anything.

Q4: The Script That Passed Its Own Check

Read this script without running it, then say what each print shows.

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
X = torch.tensor([[175.7, 6.0], [30.5, 1.0], [143.6, 5.0]]).to(device)
price = torch.tensor([509000.0, 169000.0, 582000.0], device=device)
w = torch.tensor([[3000.0], [10000.0]], dtype=torch.float64)

print(X.to(device) is X)
try:
    pred = X @ w + 50000.0
except RuntimeError as err:
    print(err)
pred = X @ w.float() + 50000.0
resid = pred - price
print(resid.shape, round(resid.mean().item()))
print(round((pred.squeeze(1) - price).mean().item()))
print(f"{(resid ** 2).mean().item():.4e}")
print(f"{((pred.squeeze(1) - price) ** 2).mean().item():.4e}")

A teammate concludes from the first line that .to(device) copied X, from the try that the float64 weights were promoted and the product ran, and from the two equal means that resid is the residual and the line after them is the loss. Explain where each conclusion goes wrong.

Hint: Ask what .to(device) returns when nothing changes, whether a matrix product promotes, and what shape (3, 1) minus (3,) has. Then ask why the mean of that shape can equal the true mean while its square cannot.

True
expected m1 and m2 to have the same dtype, but got: float != double
torch.Size([3, 3]) 19800
19800
7.6222e+10
6.4458e+09

Assignment

Assignment: Batched Pricing for the Machine the Script Runs On Next

Your team's scoring script prices listings in batches, because the table it will meet in production holds millions of rows and a batch is what fits in an accelerator's memory at once, and it has to run today on your laptop and next month on a rented machine with a GPU without anyone editing it. Write batch_pricing.py in ~/dev/bootcamp.

Name the device once and print it. Build the three feature columns area_sqm, rooms and bathrooms of the listings with an area as float32 tensors and combine them into one (2875, 3) table with torch.stack(tensors, dim=1), which joins tensors of one shape along a new dimension, position 1 here, so that each tensor becomes a column. Prove with torch.equal that the stacked table holds the same numbers as the one built from the three-column frame, and print which of the two is contiguous.

Write predict(X, w, b, device), which moves X and w to the device, raises a ValueError naming both dtypes when they differ, raises a ValueError naming both shapes when w is not a (k, 1) column for an (n, k) table, and returns X @ w + b as an (n,) vector on the device. Show both refusals with weights of 3000.0, 10000.0 and 20000.0 and a bias of 50000.0, once with the weights converted to float64 and once with them flattened to (3,). Then price the whole table and print the result's shape, its device and its first three values.

Cut the table into batches of 1,000 rows with torch.split(X, 1000), which returns a tuple of pieces along dimension 0 with the last one shorter, print the shape of each piece, price every batch through predict, join the pieces back with torch.cat(pieces), which concatenates tensors along an existing dimension, dimension 0 unless told otherwise, and prove with torch.allclose that the batched prices match the whole-table prices.

Finally compute the residual against the real prices, with its shape printed before any loss, and charge it as Lesson 3.1.3's Assignment did under Huber loss with delta at 150,000: r ** 2 / 2 for a residual no larger than delta in size and delta * (abs(r) - delta / 2) for a larger one. torch.where(cond, a, b) builds the tensor that takes a where cond is True and b elsewhere, Lesson 2.2.2's np.where on tensors. Print the mean Huber charge and the mean half-squared residual, both in scientific notation with four decimals, the count of listings past the threshold, the largest residual rounded to a whole number, and the first three prices as a NumPy array.

Hints:

• w.shape != (X.shape[1], 1) is the shape test, because a torch.Size compares equal to a plain tuple of the same numbers, and .squeeze(1) turns the (n, 1) product into the (n,) the function promises.
• A list comprehension over torch.split(X, 1000) prices each batch, and torch.cat takes that list. Both Huber branches are computed for every residual, so torch.where only chooses between two finished tensors.

import pandas as pd
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)                               # cpu

listings = pd.read_csv("listings.csv")
has_area = listings.dropna(subset=["area_sqm"])
columns = ["area_sqm", "rooms", "bathrooms"]
parts = [torch.tensor(has_area[c].to_numpy(), dtype=torch.float32) for c in columns]
X = torch.stack(parts, dim=1)               # three (2875,) tensors as the columns of one table
from_frame = torch.tensor(has_area[columns].to_numpy(), dtype=torch.float32)
print(X.shape, torch.equal(X, from_frame))  # torch.Size([2875, 3]) True
# True False: the stacked table is stored row by row, the frame's is not
print(X.is_contiguous(), from_frame.is_contiguous())
price = torch.tensor(has_area["price"].to_numpy(), dtype=torch.float32)

def predict(X, w, b, device):
    X = X.to(device)
    w = w.to(device)
    if X.dtype != w.dtype:
        raise ValueError(f"X is {X.dtype}, w is {w.dtype}")
    if w.shape != (X.shape[1], 1):
        raise ValueError(f"w is {tuple(w.shape)}, expected {(X.shape[1], 1)}")
    return (X @ w + b).squeeze(1)  # (n, 1) to (n,), on the device

w = torch.tensor([[3000.0], [10000.0], [20000.0]])
b = 50000.0
try:
    predict(X, w.double(), b, device)
except ValueError as err:
    print(err)                    # X is torch.float32, w is torch.float64
try:
    predict(X, w.view(-1), b, device)
except ValueError as err:
    print(err)                    # w is (3,), expected (3, 1)

whole = predict(X, w, b, device)
print(whole.shape, whole.device)  # torch.Size([2875]) cpu
print(whole[:3])                  # tensor([697100., 171500., 590800.])

batches = torch.split(X, 1000)
# [(1000, 3), (1000, 3), (875, 3)]
print([tuple(batch.shape) for batch in batches])
batched = torch.cat([predict(batch, w, b, device) for batch in batches])
print(batched.shape)                   # torch.Size([2875])
print(torch.allclose(batched, whole))  # True

resid = whole - price.to(device)
print(resid.shape)                         # torch.Size([2875])
delta = 150000.0
small = resid ** 2 / 2                     # the charge inside the threshold
large = delta * (resid.abs() - delta / 2)  # the straight line past it
huber = torch.where(resid.abs() <= delta, small, large)
print(f"{huber.mean().item():.4e}")        # 7.5729e+09
print(f"{small.mean().item():.4e}")        # 9.5754e+09
print((resid.abs() > delta).sum())         # tensor(645)
print(round(resid.abs().max().item()))     # 1448700
print(whole.cpu().numpy()[:3])             # [697100. 171500. 590800.]

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the exact condition view checks, or the GPU half of the device pattern as PyTorch documents it.

• Official docs - PyTorch 2.14, torch.Tensor.view. The condition a new shape must meet against the tensor's size and stride for a view to be possible, and the advice to use reshape when it is unclear, which returns a view when the shapes are compatible and copies otherwise.
• Official docs - PyTorch 2.14, "CUDA semantics", the section "Device-agnostic code". The device pattern as PyTorch writes it, with a command-line flag that disables CUDA beside is_available(), and above it the note's account of how tensors are placed on a GPU and moved between several.

Summary

Nothing in this lesson was new arithmetic. Elementwise work, broadcasting, reductions and the matrix product arrived from Module 2 with their rules intact, and what changed was the bookkeeping around them: a dtype that follows a promotion table instead of a cast, a dim where there was an axis, a standard deviation that divides by one less, a maximum that returns a pair. Each is small, and each is the kind of difference that produces a plausible number instead of an error, which is why each was printed beside Module 2's or Module 3's number for the same column.

The shape was the recurring witness. A column of 2,875 prices and a vector of 2,875 prices are the same numbers under two shapes, and the arithmetic between them is a table no one asked for, whose mean agrees with the truth and whose square does not. A table from a frame holds the same numbers as a stacked one and is laid out differently in memory, which a view notices and a reshape quietly works around. Where Module 2 learned to check a dtype and a shape, this lesson added a layout and a device to the list, and none of the four announces itself.

The device pattern is the one piece written for a machine you do not have. Every line of it did nothing here, the move returned the tensor it was given and the move back did the same, and that is the point: a script whose only mention of the hardware is its first line is a script that moves. What the tensors still lack is the second addition Lesson 4.1.1 named, the record of the operations that made them, and the next lesson sets it aside to compute, with nothing but NumPy and a small step, the number autograd will later produce for free.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
X = X.to(device)                    # (2875, 2) float32 table
price = price.to(device)            # (2875,), from a Series

mu = X.mean(dim=0)                  # (2,), dim is axis
sigma = X.std(dim=0, correction=0)  # divides by the count, as StandardScaler does
Z = (X - mu) / sigma                # broadcast, (2875, 2)

w = torch.tensor([[3000.0], [10000.0]], device=device)
pred = X @ w + 50000.0              # (2875, 1), dtypes must match
resid = pred.squeeze(1) - price     # (2875,), never (2875, 2875)
loss = (resid ** 2).mean()          # one number
print(f"{loss.item():.4e}")

values, rows = X.max(dim=0)         # a pair, not a tensor
flat = X.reshape(-1)                # view or copy, by the memory
w.sub_(500.0)                       # in place, same object
pred.cpu().numpy()                  # back to NumPy, both calls