Matrices and Matrix Multiplication

Context

A matrix is a table of numbers in rows and columns, and here it holds one listing per row and one feature per column. Multiplying two matrices takes the dot product of every row of the left one with every column of the right one and lays the results out as a new table, and multiplying a matrix by a vector does the same with a single column.

A fitted model prices a whole batch of listings with one product of the table and its weights, which is how a validation set (listings held back from fitting to check the model on) of thousands of rows gets scored. Several models score the same listings at once when their weight vectors sit side by side as the columns of one matrix, and the linear layers of Module 4's neural networks (models built from layers of such products with a simple nonlinear function between them) compute this same product plus a bias.

The element-wise product, written where the matrix product was meant, broadcasts the weights across every row and returns a table of scaled features without an error. Weight vectors stacked as rows instead of columns raise an error that names two sizes unless there are as many models as features, and when the table is square, with as many listings as features, a transposed table (its rows and columns swapped) raises nothing at all. Lesson 2.3.1 computed one dot product at a time, and this lesson computes all of them in one operation.

Goal

• Build a design matrix with one row per listing and one column per feature, and transpose it with .T.
• Predict every listing's price with X @ w + b, stating the shape of the result before running it.
• Score every listing under several models with one weight matrix, and read the error that mismatched inner sizes raise.
• Distinguish X * w from X @ w by shape, and rebuild the matrix product from the element-wise one with .sum(axis=1).
• Explain why A @ B and B @ A usually differ, and what np.eye(k) does in a product.

Prerequisites

Neither is required, the lesson works out every product in front of you. They are here if you want the row-by-column picture before you start.

• YouTube - "Essential Matrix Algebra for Neural Networks, Clearly Explained!!!" by StatQuest with Josh Starmer. The chapters from "Matrix multiplication" at 7:34 to "Transposing a matrix" at 15:18 cover this lesson, and the one at 18:51 writes a neural network as a matrix equation.
• Official docs - NumPy, numpy.matmul. The function behind @, with its shape rule (n,k),(k,m)->(n,m) and the ValueError it raises when the inner sizes disagree.

Table of Contents

• Matrices as Arrays
• Matrix-Vector Multiplication
• Matrix-Matrix Multiplication
• Element-wise Versus Matrix Product

Theory

Matrices as Arrays

What. A matrix is a 2-D array. The table a model is fitted on is called a design matrix: n rows, one per listing (one sample), by k columns, one per feature, so its shape is (n, k). X.T, the transpose, swaps rows and columns and has the shape (k, n). Why. Stacking the listings as rows turns many vectors into one object that a single operation can process, and the libraries of Modules 3 and 4 expect a table's features in exactly this layout. When. Whenever more than one listing goes through the same calculation, which in practice is always.

Each row is one of Lesson 2.3.1's listing vectors in the same feature order. A design matrix is conventionally named with a capital X, and a single vector keeps a lower-case name. Indexing is 2.2.2's: X[0] is the first listing and X[:, 0] is the area of every listing.

import numpy as np

# one row per listing, columns: area_sqm, rooms, age
X = np.array([
    [80, 3, 20],
    [120, 4, 5],
    [55, 2, 40],
    [95, 3, 12],
])

print(X.shape)  # (4, 3)
print(X[0])     # [80  3 20]
print(X[:, 0])  # [ 80 120  55  95]

The transpose turns every feature into a row: row j of X.T is feature j across every listing. It is a view (2.2.2), so it copies nothing, and a write through it lands in X.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])

print(X.T.shape)                 # (3, 4)
print(np.shares_memory(X, X.T))  # True
print(X.T)

[[ 80 120  55  95]
 [  3   4   2   3]
 [ 20   5  40  12]]

DON'T:

• DON'T write into X.T expecting a separate table, it shares X's memory, and X.T.copy() is the independent version.
• DON'T build the matrix from feature columns as its rows, np.array([areas, rooms, ages]) has the shape (3, n), the transpose of the layout every product in this section expects.

Matrix-Vector Multiplication

What. X @ w dots every row of X with w and returns one number per row: the shapes (n, k) @ (k,) give (n,), and the shared k disappears. Why. One line prices every listing in the table, and the same line scores a table of a hundred thousand rows. When. Every time one model predicts for a table: a validation set, a day's new listings, a whole site.

import numpy as np

# one row per listing, columns: area_sqm, rooms, age
X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
w = np.array([2500, 10000, -1500])
b = 50000

print(X @ w)          # [200000 332500  97500 249500]
print((X @ w).shape)  # (4,)
print(X @ w + b)      # [250000 382500 147500 299500]

The k on the two sides must match, as it did in 2.3.1, and it is the size that vanishes, leaving one entry per row of X. Adding the bias is a scalar applied to every entry (2.2.3), so X @ w + b is the whole linear model applied to the table at once, the line a fitted linear model runs to predict, and Module 3 fits w and b in 3.1.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
w = np.array([2500, 10000, -1500])

# a for loop over a 2-D array visits its rows
print(np.array([row @ w for row in X]))  # [200000 332500  97500 249500]

The loop and the product agree entry for entry, because a matrix-vector product is exactly that loop, run inside NumPy's compiled code instead of one Python dot product at a time. When the features or the weights are floats they can differ in the last digits, the same numbers reached by different routes, and np.allclose (2.2.3) is how they are compared.

DON'T:

• DON'T put the vector first, w @ X pairs the 3 entries of w with the 4 rows of X and raises ValueError ending (size 4 is different from 3).
• DON'T loop over the rows to predict, the loop gives the same numbers, up to float rounding, with one Python call per listing, where X @ w is one call for all of them.

Matrix-Matrix Multiplication

What. For A of shape (n, k) and B of shape (k, m), A @ B is the (n, m) table whose entry [i, j] is row i of A dotted with column j of B. The inner sizes, the two ks, must match, and the outer sizes give the result's shape. Why. A weight matrix with one column per model scores every listing under every model in one product, and the networks of Module 4 push whole batches of rows through products of exactly this shape. When. When several weight vectors meet the same table, comparing candidate models on one table or, in Module 4, computing a layer of a network.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
w_a = np.array([2500, 10000, -1500])  # model A, from Lesson 2.3.1
w_b = np.array([3000, 5000, -1000])   # model B, more per sqm
W = np.array([w_a, w_b]).T            # one column per model
b = np.array([50000, 40000])          # one bias per model

print(W.shape)                        # (3, 2)
print((X @ W).shape)                  # (4, 2)
print(X @ W + b)

[[250000 275000]
 [382500 415000]
 [147500 175000]
 [299500 328000]]

np.array([w_a, w_b]) stacks the two weight vectors as rows, (2, 3), and .T turns each into a column, (3, 2). The (4, 3) @ (3, 2) product gives (4, 2), one row per listing and one column per model, and the (2,) biases pair with the last size, 2, so each column gets its own model's bias (2.2.4). Column 0 repeats Lesson 2.3.1's prices, and column 1 is model B's view of the same four listings.

Forgetting the .T leaves the weight vectors as rows, and with two models on three features the inner sizes stop matching.

# two_models.py
import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
w_a = np.array([2500, 10000, -1500])
w_b = np.array([3000, 5000, -1000])
W_rows = np.array([w_a, w_b])  # (2, 3), the .T forgotten
print(X @ W_rows)

Traceback (most recent call last):
  File "/home/goku/dev/bootcamp/two_models.py", line 8, in <module>
    print(X @ W_rows)
          ~~^~~~~~~~
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3)

This is the signature Lesson 2.3.1 left for later, read in full. (n?,k),(k,m?)->(n?,m?) says the left operand is (n, k), the right one is (k, m) and the result is (n, m), and a ? marks a size that may be missing, which is how a 1-D vector takes part on either side. Operand 1, W_rows, has 2 as its core dimension 0, its first size, where the k brought by X is 3.

Order matters. X @ W works and W @ X raises, because (3, 2) @ (4, 3) pairs an inner 2 with a 4, (size 4 is different from 2). Where both orders run, they are usually different products, a square matrix and the identity below being one of the exceptions.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])

print((X.T @ X).shape)  # (3, 3)
print((X @ X.T).shape)  # (4, 4)

X.T @ X is a feature-by-feature table, each entry two feature columns dotted together, and Lesson 2.3.4 meets it again. X @ X.T is a listing-by-listing table holding the dot product of every pair of listings. Two different objects of two different shapes came from the same two matrices in the two orders.

np.eye(k) is the identity matrix, ones on the diagonal and zeros everywhere else, the 1 of matrix multiplication: X @ np.eye(3) returns X.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])

print(np.eye(3))
print(X @ np.eye(3))

[[1. 0. 0.]
 [0. 1. 0.]
 [0. 0. 1.]]
[[ 80.   3.  20.]
 [120.   4.   5.]
 [ 55.   2.  40.]
 [ 95.   3.  12.]]

The values come back unchanged and as floats, because np.eye builds float64 by default and an integer table times a float table is a float table. np.eye(3, dtype=int) keeps the product in integers.

DON'T:

• DON'T stack weight vectors and forget the .T, np.array([w_a, w_b]) holds one model per row, and X times it raises because the inner sizes are 3 and 2, while three models on three features raise nothing and return a wrong (4, 3) table.
• DON'T treat A @ B and B @ A as interchangeable, the shapes may allow only one of them, and when both run they are usually different tables.
• DON'T expect X @ np.eye(3) to hand back integers, the default identity is float64 and so is the product.

Element-wise Versus Matrix Product

What. X * w multiplies element by element, broadcasting w across every row into an (n, k) table of scaled features, while X @ w takes one dot product per row and returns (n,). Why. X * w raises nothing, so writing it where @ was meant passes an object of the wrong shape into the next step without a word. When. Every time a table meets a vector in a product: decide which of the two is meant, then check the result's shape.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
w = np.array([2500, 10000, -1500])

print((X * w).shape)  # (4, 3)
print(X * w)

[[200000  30000 -30000]
 [300000  40000  -7500]
 [137500  20000 -60000]
 [237500  30000 -18000]]

Each row holds one listing's three products, the terms of its dot product before the adding. The (4, 3) table and the (3,) weights pair from the right, 2.2.4's rule, so w runs along every row, and nothing raised because nothing was wrong with the broadcast. Adding up each row, 2.2.5's axis=1, finishes every dot product.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
w = np.array([2500, 10000, -1500])

print((X * w).sum(axis=1))  # [200000 332500  97500 249500]
print(X @ w)                # [200000 332500  97500 249500]

That is everything @ does between a table and a vector: multiply matching entries along the feature axis, then add along it. The shape is what tells the two apart, (4, 3) where (4,) was wanted.

DON'T:

• DON'T write X * w for predictions, it returns the (4, 3) table of products with no error, and the next step receives a table where it expected one price per listing.
• DON'T rely on the shape alone when the table is square, with as many listings as features the wrong reduction, .sum(axis=0), also comes out (3,).

Lab Exercises

Lab Exercise 1: Build the Table, Then Price Every Row

Five listing records arrive with their keys in mixed orders.

records = [
    {"rooms": 4, "area_sqm": 120, "age": 5},
    {"age": 40, "rooms": 2, "area_sqm": 55},
    {"area_sqm": 95, "age": 12, "rooms": 3},
    {"rooms": 3, "age": 20, "area_sqm": 80},
    {"area_sqm": 70, "rooms": 2, "age": 8},
]

Build them into a design matrix X in the order FEATURES = ["area_sqm", "rooms", "age"], one row per record, and print it, its shape, the rooms column and the shape of X.T. Print whether X.T shares memory with X.

With w = np.array([2500, 10000, -1500]) and b = 50000, write the shape you expect for X @ w + b in a comment, then print the predictions and their shape. Compute the same predictions with a loop of dot products over the rows and print them too.

Lab Exercise 2: Two Models at Once, and the Product That Was Not Meant

Using the X from Lab Exercise 1, score every listing under two new models, w_c = [2000, 15000, -500] with bias 60000 and w_d = [3500, 0, -2000] with bias 30000. Build a weight matrix with one column per model, print the table of predictions and its shape, and print which model prices each listing higher, as a column index per listing.

Then compute X @ np.array([w_c, w_d]) inside a try, catch the ValueError and print its message. Finally, print the shape of X * w_c, and turn that element-wise product into model C's dot products, without the bias, by reducing along the right axis, printing the result next to X @ w_c.

import numpy as np

FEATURES = ["area_sqm", "rooms", "age"]
records = [
    {"rooms": 4, "area_sqm": 120, "age": 5},
    {"age": 40, "rooms": 2, "area_sqm": 55},
    {"area_sqm": 95, "age": 12, "rooms": 3},
    {"rooms": 3, "age": 20, "area_sqm": 80},
    {"area_sqm": 70, "rooms": 2, "age": 8},
]

# one inner list per record, in FEATURES order
X = np.array([[r[n] for n in FEATURES] for r in records])
print(X)
print(X.shape)                   # (5, 3)
print(X[:, 1])                   # [4 2 3 3 2]
print(X.T.shape)                 # (3, 5)
print(np.shares_memory(X, X.T))  # True

[[120   4   5]
 [ 55   2  40]
 [ 95   3  12]
 [ 80   3  20]
 [ 70   2   8]]

# continuing the script above
w = np.array([2500, 10000, -1500])
b = 50000

# expected: (5, 3) @ (3,) gives (5,), one price per listing
preds = X @ w + b
print(preds)        # [382500 147500 299500 250000 233000]
print(preds.shape)  # (5,)

loop = np.array([row @ w + b for row in X])
print(loop)         # [382500 147500 299500 250000 233000]

# continuing Solution 1's script
w_c = np.array([2000, 15000, -500])
w_d = np.array([3500, 0, -2000])
W = np.array([w_c, w_d]).T       # (3, 2), one column per model
biases = np.array([60000, 30000])

table = X @ W + biases
print(table)
print(table.shape)               # (5, 2)
print(np.argmax(table, axis=1))  # [1 0 1 1 1]

[[357500 440000]
 [180000 142500]
 [289000 338500]
 [255000 270000]
 [226000 259000]]

# continuing the script above
try:
    print(X @ np.array([w_c, w_d]))  # (5, 3) @ (2, 3)
except ValueError as err:
    print(err)

matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 2 is different from 3)

# continuing the script above
print((X * w_c).shape)        # (5, 3)
print((X * w_c).sum(axis=1))  # [297500 120000 229000 195000 166000]
print(X @ w_c)                # [297500 120000 229000 195000 166000]

Theory Assessment

Q1: Rows, Columns and the Transpose

A design matrix X holds 200 listings described by 6 features. State its shape, what X[:, 2] holds and its shape, and the shape of X.T. Then explain what happens to X when you write a value into X.T, and how to get a transpose you can change freely.

Hint: Rows are listings. For the second half, ask whether a transpose has to move any numbers at all.

Q2: One Prediction per Row

For the same X and a weight vector w of shape (6,), explain what X @ w computes, the shape of the result and where the 6 went. Then explain why a loop of dot products over the rows gives the same numbers, and why X @ w + b needs no loop for the bias.

Hint: Each output entry belongs to one row. For the bias, recall what NumPy does with a scalar on one side of an operator.

Q3: Which Products Exist

With X of shape (200, 6), a weight matrix W of shape (6, 3) and w of shape (6,), give the shape of each product or say that it raises, and for any that raises, name the two sizes in its message: X @ W, W @ X, X.T @ X, X @ X.T, X @ W.T, w @ X.T and X @ np.eye(6). Then say what the last one returns and in what dtype.

Hint: Write each shape pair down and compare the inner sizes only. A 1-D vector may sit on either side, and the signature's ? says how.

Q4: The Square Test Table

A teammate checks a pricing script on three listings with three features and is satisfied because every result has the shape of three predictions. Read the following without running it, then state what each line prints.

import numpy as np

# three listings, columns: area_sqm, rooms, age
X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40]])
w = np.array([2500, 10000, -1500])

a = X @ w
b = (X * w).sum(axis=1)
c = (X * w).sum(axis=0)
d = X.T @ w
print(a.shape, b.shape, c.shape, d.shape)
print(a)
print(c)
print(d)

Which of a, b, c and d are the predictions, what do the other two compute, and why did the shape check pass all four? Then say what one change to the test data would have exposed both wrong lines, and how each would have failed.

Hint: Write out the first entry of c and of d as a sum of products and ask which listing each term belongs to. For the last part, ask which sizes stop being equal when the table is not square.

(3,) (3,) (3,) (3,)
[200000 332500  97500]
[637500  90000 -97500]
[1317500   44500   40000]

Assignment

Assignment: Another Team's Units

The rentals team fitted two price models, A and B, on listings described by area_sqm, rooms and age in months. Their weights, one column per model, are W_months = np.array([[2500, 3000], [10000, 5000], [-125, -100]]), with the biases [50000, 40000]. Your design matrix holds this lesson's four listings, with the age in years. Build the following in one script.

First, score your listings under both models without editing X by hand. Build a conversion matrix with np.diag, which takes a 1-D array and returns the square matrix with that array on its diagonal and zeros everywhere else, choosing its entries so that X @ D carries the age in months and leaves the other two columns alone. Print X @ D and the (4, 2) table of predictions. Show that X * np.array([1, 1, 12]) gives the same table as X @ D, and say in a comment why a diagonal matrix on the right and an element-wise product agree.

Second, the team would rather ship weights per year. Compute D @ W_months and show that X @ (D @ W_months) plus the biases gives the same predictions as (X @ D) @ W_months plus the biases. Say in a comment which of the two you would ship, and why.

Third, a teammate tries the conversion on a batch of three listings, X[:3], with D on the left, D @ X[:3]. Run it, print what it returns and explain in a comment what it scaled and why nothing raised. Then show what the same line does on the full four-listing X.

Hints:

• When D sits on the right, D[j, j] multiplies column j of X, and when it sits on the left, it multiplies row j.
• (X @ D) @ W_months and X @ (D @ W_months) are the same three matrices grouped differently. A chain of matrix products may be regrouped as long as the matrices stay in the same order, and the result is the same table, exactly with whole numbers like these and up to float rounding otherwise.

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5], [55, 2, 40], [95, 3, 12]])
W_months = np.array([[2500, 3000], [10000, 5000], [-125, -100]])
b = np.array([50000, 40000])

D = np.diag([1, 1, 12])  # age column times 12, years to months
print(X @ D)
print((X @ D) @ W_months + b)
print(X * np.array([1, 1, 12]))
# D is zero off its diagonal, so column j of X @ D is column j of X
# times D[j, j], which is what broadcasting [1, 1, 12] along every
# row does too.

[[ 80   3 240]
 [120   4  60]
 [ 55   2 480]
 [ 95   3 144]]
[[250000 271000]
 [382500 414000]
 [147500 167000]
 [299500 325600]]
[[ 80   3 240]
 [120   4  60]
 [ 55   2 480]
 [ 95   3 144]]

# continuing the script above
W_years = D @ W_months  # the age weights times 12, per year
print(W_years)
print(X @ W_years + b)
# Ship W_years: the conversion is done once, inside the weights, and
# no caller can forget to convert the listings before predicting.

[[ 2500  3000]
 [10000  5000]
 [-1500 -1200]]
[[250000 271000]
 [382500 414000]
 [147500 167000]
 [299500 325600]]

# continuing the script above
print(D @ X[:3])
# D on the left scales ROWS: the third listing's area, rooms and age
# were all multiplied by 12. (3, 3) @ (3, 3) matches, so nothing raised.

try:
    print(D @ X)  # (3, 3) @ (4, 3)
except ValueError as err:
    print(err)

[[ 80   3  20]
 [120   4   5]
 [660  24 480]]
matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 4 is different from 3)

Additional Resources

Optional, and nothing later depends on either. Worth a look if you want the geometry of a product or the history of @.

• YouTube - "Matrix multiplication as composition | Chapter 4, Essence of linear algebra" by 3Blue1Brown. Multiplying two matrices as applying one transformation after another, the geometric reason the order of a product matters.
• Python PEP - PEP 465, "A dedicated infix operator for matrix multiplication", the Motivation section. Why Python added @: the element-wise product and the matrix product competed for *, and numerical code split between the two conventions.

Summary

A matrix is a stack of the vectors from the previous lesson, and that is nearly all it is. Putting the listings one per row did not add a new kind of arithmetic. It let one operation stand in for a loop, and the loop it stands in for is the dot product run once per row, or once per row and column. Every number in a matrix product is a dot product you already know how to read, and the product's only extra job is laying those numbers out in the right place.

The shape rule is the real content of the lesson. The inner sizes must agree and vanish, the outer sizes survive and become the result, and almost every mistake here is a shape mistake: a vector on the wrong side, weights stacked as rows instead of columns, a conversion on the wrong side of the table. NumPy is strict about the inner sizes and reports them by number, which turns a confusing failure into an arithmetic check you can do on paper before running anything.

The quiet failures are the ones the rule cannot see. The element-wise product raises nothing whenever the shapes broadcast and returns a table where a column of prices belonged, and a square table lets a wrong axis or a transposed table through because every size matches every other. Predicting a result's shape before running it, and testing on a table whose rows and columns differ in number, is how those failures surface. The next lesson leaves matrix products behind and measures how far apart two listings are, using the same rows and the same broadcasting.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np

X = np.array([[80, 3, 20], [120, 4, 5]])  # (n, k)
w = np.array([2500, 10000, -1500])        # (k,)
X.T                                       # (k, n), a view
X @ w + 50000                             # (n,), one price per row

W = np.array([w, [3000, 5000, -1000]]).T  # (k, m)
X @ W + np.array([50000, 40000])          # (n, m)
np.eye(3, dtype=int)                      # X @ it returns X
X * w                                     # (n, k), not predictions
(X * w).sum(axis=1)                       # same as X @ w
X @ np.diag([1, 1, 12])                   # scales the age column