Rank, Linear Independence and Collinear Columns

FREE~1.5 h2 labs4 questions1 assignment

Context

Columns are linearly independent when none of them can be rebuilt as a weighted sum of the others, and the rank of a table counts how many independent columns it really holds. A table whose columns repeat each other, exactly or nearly, gives a linear model more weights than the data can pin down.

Feature tables are assembled from several sources, and that is where the repeats come from: area in square meters beside the same area in square feet from a second listing site, a room total stored beside its parts, an age computed from the construction year kept next to the year itself. The rank check before a fit and the condition number (how far a small change in the data can swing the weights) after it decide whether a model's individual weights can be read at all, which matters the moment anyone asks what a feature is worth.

Without the check the failures stay quiet. np.linalg.solve can raise on one collinear table and return ordinary-looking weights on another. lstsq answers either way, with predictions that hold and weights split arbitrarily between the twin columns. A column that nearly repeats another passes the rank check and lets one re-priced listing flip the sign of a weight. The previous lesson fitted weights by least squares, this lesson finds out when those weights mean anything.

Goal

  • Identify a column that is a combination of others, and prove it with a weight vector v for which X @ v is all zeros.
  • Compute a table's rank with np.linalg.matrix_rank and compare it with the column count before fitting.
  • Run np.linalg.solve and np.linalg.lstsq on a collinear table and state exactly what each one returned.
  • Distinguish the predictions of a collinear fit, which hold, from its individual weights, which do not.
  • Measure near-collinearity with np.linalg.cond, and repair it by dropping a column or centering one.

Prerequisites

Neither is required, the lesson builds everything it uses. Both are short pictures of the ideas, there if you want independence and rank drawn before you compute them.

Table of Contents

  • Linear Independence
  • Rank with np.linalg.matrix_rank
  • Collinear Columns in Least Squares
  • Near-Collinearity and the Condition Number

Theory

Linear Independence

What. Columns are linearly independent when none of them is a combination of the others, a weighted sum of other columns. area_sqft = 10.7639 * area_sqm is a combination of one column, and total_rooms = bedrooms + other_rooms is a combination of two. Why. A combination adds no direction: every prediction a linear model can make with it, it could already make without it. The column brings no information, and the model becomes free to share credit between it and its sources in any proportion. When. Whenever a feature table gains a derived column: a unit conversion, a total beside its parts, an age computed from a year, or the one-hot columns of Module 3's encoding lesson, 3.2.2 (one column of 0s and 1s per category, with a 1 in the listing's own category), which then add up to the column of ones.

In plain text, the columns of X are dependent when some weights v, not all zero, give X v = 0, and in NumPy X @ v then comes back all zeros, or within float rounding of them. Splitting the five listings' rooms into bedrooms and other rooms shows it in whole numbers.

import numpy as np

bedrooms = np.array([1.0, 2.0, 2.0, 3.0, 3.0])
other_rooms = np.array([1.0, 1.0, 1.0, 1.0, 2.0])
total_rooms = bedrooms + other_rooms
X = np.column_stack([bedrooms, other_rooms, total_rooms])

v = np.array([1.0, 1.0, -1.0])  # bedrooms + other_rooms - total_rooms
print(X @ v)                    # [0. 0. 0. 0. 0.]

For a model, no information means that two different weight vectors make the same predictions. On a table holding area in square meters and in square feet, crediting the whole price to one unit or the whole price to the other prices every listing identically.

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
X = np.column_stack([np.ones(5), area, 10.7639 * area])

w_sqm = np.array([40.0, 3.0, 0.0])           # all on square meters
w_sqft = np.array([40.0, 0.0, 3 / 10.7639])  # all on square feet
print(np.round(X @ w_sqm, 2))                # [190. 220. 280. 340. 400.]
print(np.round(X @ w_sqft, 2))               # [190. 220. 280. 340. 400.]

Any blend of the two works as well, so the data cannot say how the 3 thousand per square meter splits between them. A ratio such as area / rooms or a square such as rooms ** 2 is derived from other columns without being a weighted sum of them, so deriving a column does not on its own make it dependent.

DON'T:

  • DON'T add a unit-converted or summed column to give a model more to learn from. It hands over a second name for a direction already present, and nothing else.
  • DON'T forget the column of ones when hunting for combinations. age = 2026 - year_built is 2026 times the ones column minus year_built, a combination even though only one feature is in sight.

Rank with np.linalg.matrix_rank

What. The rank of a table is the number of independent columns it holds, np.linalg.matrix_rank(X), and it is never more than the smaller of the table's row count and column count. Why. It turns the hunt for combinations into one number to compare with X.shape[1]. Equal is full column rank, anything less is rank-deficient, and a rank-deficient table cannot give each weight one meaning. When. Before fitting any table you assembled yourself, and again after every join, derived column or encoding step.

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
rooms = np.array([2.0, 3.0, 3.0, 4.0, 5.0])
bedrooms = np.array([1.0, 2.0, 2.0, 3.0, 3.0])
ones = np.ones(5)

X_good = np.column_stack([ones, area, rooms])
X_sqft = np.column_stack([ones, area, 10.7639 * area])
X_rooms = np.column_stack([ones, bedrooms, rooms - bedrooms, rooms])
for X in [X_good, X_sqft, X_rooms]:
    print(X.shape[1], np.linalg.matrix_rank(X))
3 3
3 2
4 3

matrix_rank counts the singular values of X, the fourth value 2.3.4's lstsq returned, that stand above a tolerance scaled to the largest one. Rounding leaves the square-feet table's third singular value tiny rather than exactly zero, the tolerance counts it as zero, and the table comes out at 2. The rank lstsq returns is decided the same way.

import numpy as np

area = np.array([50.0, 60.0, 80.0])
rooms = np.array([2.0, 3.0, 3.0])
bedrooms = np.array([1.0, 2.0, 2.0])
year = np.array([1962.0, 1974.0, 1987.0])
# three listings, five columns
X = np.column_stack([np.ones(3), area, rooms, bedrooms, year])
print(X.shape)                   # (3, 5)
print(np.linalg.matrix_rank(X))  # 3
# False
print(np.linalg.matrix_rank(X) == X.shape[1])

Three listings can hold at most three independent directions, so with five columns at least two are combinations of the others on those rows, whatever the features mean. A table with more columns than listings is rank-deficient before anyone looks at its contents.

DON'T:

  • DON'T compare the rank with the number of rows. Full column rank means a rank equal to X.shape[1], so the three-listing table fails at 3 of 5 although 3 is all its rows allow.
  • DON'T read the rank as a measure of how independent the columns are. It is a count with a tolerance, and a column that nearly repeats another counts in full, as the last subtitle measures.

Collinear Columns in Least Squares

What. With an exactly collinear column, X.T @ X has no inverse, the normal equations have infinitely many solutions, and every one of them gives the same best predictions. Why. Neither tool can be counted on to say so plainly. np.linalg.solve may raise or may return weights, and lstsq returns one answer out of the infinitely many, so what each one did decides which numbers can be read. When. Whenever matrix_rank falls short of the column count, and whenever a fit on a table with derived columns produces weights that look odd.

The square-feet table goes first, with solve inside Module 1's try and except and np.linalg.LinAlgError as the error to catch.

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
X = np.column_stack([np.ones(5), area, 10.7639 * area])

try:
    w = np.linalg.solve(X.T @ X, X.T @ price)
    print("solve returned", w)
except np.linalg.LinAlgError as err:
    print("solve raised:", err)  # solve raised: Singular matrix

w, ssr, rank, sv = np.linalg.lstsq(X, price)
print(rank, ssr)                 # 2 []
# 40.0000 0.0257 0.2763
print(f"{w[0]:.4f} {w[1]:.4f} {w[2]:.4f}")
print(np.round(X @ w, 2))        # [190. 220. 280. 340. 400.]

Here solve raised Singular matrix, and on some machines the same line returns weights instead, for the reason the age table shows next. lstsq reported rank 2 for three columns and an empty sum of squared residuals, since it reports one only at full rank with more rows than columns, and returned predictions identical to 2.3.4's line, up to rounding. Of the infinitely many weight vectors that make those predictions it returns the one with the smallest length, which puts a weight on each of the two columns in proportion to its size.

The age table is built from exact whole numbers, and its third column is exactly 2026 times the first minus the second.

import numpy as np

year = np.array([1962.0, 1974.0, 1987.0, 1999.0, 2011.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
# ones, year_built, age = 2026 - year_built
X = np.column_stack([np.ones(5), year, 2026 - year])

w_solve = np.linalg.solve(X.T @ X, X.T @ price)
# 220.4623 0.1171 -4.2419
print(f"{w_solve[0]:.4f} {w_solve[1]:.4f} {w_solve[2]:.4f}")
w, ssr, rank, sv = np.linalg.lstsq(X, price)
# -0.0019 0.2259 -4.1330
print(f"{w[0]:.4f} {w[1]:.4f} {w[2]:.4f}")
print(rank, ssr)                           # 2 []
print(np.allclose(X @ w_solve, X @ w))     # True
# folded back onto ones and year: -8373.5385 4.3590
print(f"{w[0] + 2026 * w[2]:.4f} {w[1] - w[2]:.4f}")

w_year, ssr_y, rank_y, sv_y = np.linalg.lstsq(X[:, :2], price)
print(f"{w_year[0]:.4f} {w_year[1]:.4f}")  # -8373.5385 4.3590

On the age table solve raised nothing, though its columns are exactly dependent. It reports a singular matrix only when its elimination meets an exact zero, and rounding left a sliver instead, so the weights it returned are an accident of that rounding. Whether the elimination ends on an exact zero decides whether solve raises, so the same line can raise on one machine and return weights on another, and the weights it returns differ from machine to machine. Both sets of weights predict the same prices, and lstsq's three fold back onto the year-only model's two: its -4.1330 per year of age cannot be read as an age effect, because 2026 times it is almost the whole year-only intercept, -8373.5385, moved into the age column.

DON'T:

  • DON'T count on solve to raise on collinear columns. It raised on the square-feet table and returned ordinary-looking weights on the age table, and which happens is decided by rounding.
  • DON'T read the individual weights of collinear columns. The -4.1330 per year of age carries the year-only intercept into another column, and 0.0257 per square meter is one split out of infinitely many that predict the same prices.

Near-Collinearity and the Condition Number

What. A column that repeats another up to a little noise keeps the table at full rank, and np.linalg.cond(X), the largest singular value divided by the smallest, measures how close the table came to losing a direction. Why. Near-collinear columns pass the rank check and fit without a warning, yet their weights swing when the data moves a little. A large condition number is the warning the rank cannot give. When. After matrix_rank reports full rank and before any individual weight is read, reported or compared.

A square-feet column copied from a listing site that shows whole feet carries up to half a square foot of rounding against 10.7639 times the square meters, too little to see in the table and enough to restore full rank. The block prints the condition number with :.4e (scientific notation with four digits after the point).

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
sqft = np.round(10.7639 * area)    # whole square feet
print(sqft)                        # [ 538.  646.  861. 1076. 1292.]

X = np.column_stack([np.ones(5), area, sqft])
print(np.linalg.matrix_rank(X))    # 3
print(f"{np.linalg.cond(X):.4e}")  # 4.0275e+04
w, ssr, rank, sv = np.linalg.lstsq(X, price)
print(np.round(w, 4))              # [ 40.8039 -31.6216   3.2157]
print(f"{sv[0] / sv[-1]:.4e}")     # 4.0275e+04

Full rank, no error, and a weight of -31.6216 thousand per square meter, cancelled by 3.2157 per square foot. Between them the pair still charges about 3 per square meter, and neither number means what it says. sv[0] / sv[-1] is the condition number rebuilt from the singular values lstsq returned, the same 4.0275e+04.

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
X = np.column_stack([np.ones(5), area, np.round(10.7639 * area)])
X_area = np.column_stack([np.ones(5), area])
price_new = price.copy()
price_new[3] = 337.0                    # the 100 sqm listing, 5 thousand higher

w, ssr, rank, sv = np.linalg.lstsq(X, price_new)
print(np.round(w, 4))                   # [37.9608 37.0157 -3.1569]
w, ssr, rank, sv = np.linalg.lstsq(X_area, price_new)
print(np.round(w, 4))                   # [38.75    3.0274]
print(f"{np.linalg.cond(X_area):.4e}")  # 2.8818e+02

One listing re-priced by 5 thousand, under 2 percent of its price, and both weights changed sign: square meters went from -31.6216 to 37.0157 and square feet from 3.2157 to -3.1569. The area-only model, with a condition number about 140 times smaller, moved from 3 to 3.0274 per square meter. The cure is to drop one of the twins or combine them into one column, which Module 3's feature selection lesson, 3.3.2, does on purpose.

The column of ones can be one of the twins. A raw construction year sits near 2000 in every row, so it is nearly 2000 times the column of ones, the table behind 2.3.4's inverse.

import numpy as np

year = np.array([1962.0, 1974.0, 1987.0, 1999.0, 2011.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
X = np.column_stack([np.ones(5), year])
print(f"{np.linalg.cond(X):.4e}")        # 2.2688e+05
print(f"{np.linalg.cond(X.T @ X):.4e}")  # 5.1473e+10

# 2.2.5's centering: subtract the mean year
X_c = np.column_stack([np.ones(5), year - year.mean()])
print(f"{np.linalg.cond(X_c):.4e}")      # 1.7397e+01
w, ssr, rank, sv = np.linalg.lstsq(X, price)
w_c, ssr_c, rank_c, sv_c = np.linalg.lstsq(X_c, price)
print(np.round(w_c, 4))                  # [286.      4.359]
print(np.allclose(X @ w, X_c @ w_c))     # True

Forming X.T @ X squares the condition number, 2.2688e+05 into 5.1473e+10, and that squared figure belongs to the matrix 2.3.4 inverted. Working on X itself, as lstsq does, avoids the squaring. Centering the year cut the condition number to 17.397 without changing a prediction, and turned the intercept into 286, the average price, where the raw table's -8373.5385 was the price of a house built in year zero.

DON'T:

  • DON'T read full rank as a healthy fit. The table with square feet rounded to whole feet passed matrix_rank at 3 of 3 and lstsq raised nothing, and one re-priced listing still flipped the sign of both weights.
  • DON'T keep both of two near-twin columns to be safe. They give the model almost nothing new and turn every weight on them unstable, so drop one, combine them, or center the column that shadows the ones.

Lab Exercises

Lab Exercise 1: Audit the Rooms Table

The five listings' rooms split into bedrooms, 1, 2, 2, 3 and 3, and other rooms, 1, 1, 1, 1 and 2, and a teammate's table keeps both parts beside the total. Build X from a column of ones, the areas, the bedrooms, the other rooms and the room totals, then print its shape, its rank and whether it has full column rank.

Find a weight vector v that proves the dependence and print X @ v. Drop the room totals, then print the new rank and the full-rank check.

Finally, keep only the first three listings of the reduced table, print its shape and rank, and add a comment explaining the rank it reports.

Lab Exercise 2: Fit the Rooms Table Three Ways

Fit price on the five-column table from Lab Exercise 1. Run np.linalg.solve on its normal equations inside try and except, and print whether it raised. Fit the same table with np.linalg.lstsq and print the rank and the sum of squared residuals it reports, then fit the reduced four-column table.

Show that the weights from solve, from lstsq and from the reduced fit all give the same predictions, while the weights themselves disagree. Print both sets of lstsq weights to four decimals and the reduced table's condition number, and add a comment saying which weights you would report.

Solution 1: Audit the Rooms Table 
import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
bedrooms = np.array([1.0, 2.0, 2.0, 3.0, 3.0])
other_rooms = np.array([1.0, 1.0, 1.0, 1.0, 2.0])
rooms = bedrooms + other_rooms            # [2. 3. 3. 4. 5.], 2.3.4's rooms
X = np.column_stack([np.ones(5), area, bedrooms, other_rooms, rooms])
print(X.shape, np.linalg.matrix_rank(X))  # (5, 5) 4
# False
print(np.linalg.matrix_rank(X) == X.shape[1])

# bedrooms + other_rooms - rooms, nothing from ones or area
v = np.array([0.0, 0.0, 1.0, 1.0, -1.0])
print(X @ v)                              # [0. 0. 0. 0. 0.]

X_red = X[:, :4]                          # ones, area, bedrooms, other_rooms
# (5, 4) 4
print(X_red.shape, np.linalg.matrix_rank(X_red))
# True
print(np.linalg.matrix_rank(X_red) == X_red.shape[1])

top3 = X_red[:3]
# (3, 4) 3
print(top3.shape, np.linalg.matrix_rank(top3))
# three rows hold at most three independent directions, so one of the
# four columns is a combination of the others on these rows

DON'T:

  • DON'T drop a column at random to repair the rank. The column to drop is one with a nonzero weight in v, and dropping the areas instead leaves the table at rank 3 of 4, the X_rooms table of the second subtitle.
  • DON'T blame the features for the three-row shortfall. Three listings cannot support four independent columns, and the same four columns reached rank 4 on all five listings.
Solution 2: Fit the Rooms Table Three Ways 
import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
bedrooms = np.array([1.0, 2.0, 2.0, 3.0, 3.0])
other_rooms = np.array([1.0, 1.0, 1.0, 1.0, 2.0])
rooms = bedrooms + other_rooms
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
X = np.column_stack([np.ones(5), area, bedrooms, other_rooms, rooms])
X_red = X[:, :4]

w, ssr, rank, sv = np.linalg.lstsq(X, price)
print(rank, ssr)                          # 4 []
w_red, ssr_red, rank_red, sv_red = np.linalg.lstsq(X_red, price)
print(np.round(w, 4))                     # [ 40.5      3.7222 -16.7037   7.3519  -9.3519]
print(np.round(w_red, 4))                 # [ 40.5      3.7222 -26.0556  -2.    ]
print(np.allclose(X @ w, X_red @ w_red))  # True

try:
    w_solve = np.linalg.solve(X.T @ X, X.T @ price)
    print("solve returned weights")       # solve returned weights
    # True
    print(np.allclose(X @ w_solve, X_red @ w_red))
    print(np.allclose(w_solve, w))        # False
except np.linalg.LinAlgError as err:
    print("solve raised:", err)

print(f"{np.linalg.cond(X_red):.4e}")     # 4.7130e+02
# report w_red: one meaning per column, and no rounding accident
# or choice of shortest vector behind any of its four numbers

DON'T:

  • DON'T take "solve returned weights" as a sign that the table was sound. The rank, 4 of 5, said otherwise before any fit ran, and the weights solve returned are the rounding's choice.
  • DON'T compare the bedroom weights across the fits as if one were the true one. -16.7037 from the full table and -26.0556 from the reduced one belong to fits with the same predictions, and on the full table no single bedroom weight exists.

Theory Assessment

Q1: Four Candidate Columns

A table holds a column of ones, area_sqm and rooms for the lesson's five listings. For each candidate below, say whether adding it would leave the table at full column rank, and why: area_sqft = 10.7639 * area_sqm, area_per_room = area_sqm / rooms, rooms_plus_one = rooms + 1 and rooms_squared = rooms ** 2.

Hint: A combination is a weighted sum of columns already in the table, and one of those columns is easy to forget.

Q2: More Columns Than Listings

Take the second subtitle's table of three listings and five columns, a column of ones, area, rooms, bedrooms and year built, and fit their three prices, 198, 210 and 285, with np.linalg.lstsq. Say what rank and sum of squared residuals it reports, how well its predictions match the three prices, and why the weights it returns cannot be read.

Hint: Count the equations and the unknowns. When the unknowns win, ask how many weight vectors fit perfectly.

Q3: The Split lstsq Chose

On the square-feet table, lstsq returned the weights 40.0000, 0.0257 and 0.2763, while [40, 3, 0] predicts the same prices. Explain why lstsq picked its split rather than that one, what its rank and its empty sum of squared residuals tell you, and which of its outputs you can use.

Hint: Both weight vectors predict the same prices, so the predictions cannot be what lstsq chose by. Compare their lengths with np.linalg.norm.

Q4: Three Conclusions

A teammate fits price on three of this lesson's tables and reports three conclusions. On the square-feet table, solve raised LinAlgError, so "the data is broken and nothing can be predicted from it". On the age table, solve returned weights without an error, so "the fit worked, and each year of age takes 4.2419 thousand off the price". On the table with square feet rounded to whole feet, matrix_rank said 3 of 3 and lstsq raised nothing, so "the table is healthy, and the weights can go in the report". Say what is wrong with each conclusion, using what the lesson ran.

Hint: Separate what each run said about the predictions from what it said about the individual weights, and remember which check each table never got.

Answer 1:

area_sqft is 10.7639 times area_sqm, a combination of one column, so the rank stays at 3 for 4 columns. area_per_room divides one column by another, which is no weighted sum, and the rank rises to 4. rooms_plus_one is rooms plus 1 times the column of ones, a combination of two columns, so the rank stays at 3. rooms_squared squares a column, which is no weighted sum either, and the rank rises to 4.

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
rooms = np.array([2.0, 3.0, 3.0, 4.0, 5.0])
base = [np.ones(5), area, rooms]
for col in [10.7639 * area, area / rooms, rooms + 1, rooms ** 2]:
    X = np.column_stack(base + [col])
    print(np.linalg.matrix_rank(X))
3
4
3
4

The trap is rooms + 1, whose combination runs through the column of ones, which no feature name mentions.

DON'T check a derived column only against the features it was built from, the column of ones takes part in combinations too.

Answer 2:

Three equations in five unknowns leave the rank at most 3, and lstsq reports exactly 3, with an empty sum of squared residuals. Its predictions reproduce the three prices, up to rounding.

import numpy as np

area = np.array([50.0, 60.0, 80.0])
rooms = np.array([2.0, 3.0, 3.0])
bedrooms = np.array([1.0, 2.0, 2.0])
year = np.array([1962.0, 1974.0, 1987.0])
price = np.array([198.0, 210.0, 285.0])
X = np.column_stack([np.ones(3), area, rooms, bedrooms, year])

w, ssr, rank, sv = np.linalg.lstsq(X, price)
print(rank, ssr)                  # 3 []
print(np.allclose(X @ w, price))  # True
print(np.round(w, 4))             # [  0.0354   3.7335 -12.802  -12.8374   0.0253]

A perfect fit here is no achievement. Five weights against three prices leave two free directions, so infinitely many weight vectors fit the three rows exactly, and lstsq handed back the shortest of them, with about minus 12.8 thousand per room and as much again per bedroom. Those numbers are one pick among endless equally perfect ones, and fitting the rows a model was fitted on says nothing about the next listing.

DON'T read a perfect fit on a table with more columns than rows as a good model, the rank guaranteed it before the fit ran.

Answer 3:

lstsq returns, of all the weight vectors with the best predictions, the one with the smallest length. All three candidates below predict the same prices, and its own is the shortest.

import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
X = np.column_stack([np.ones(5), area, 10.7639 * area])
w, ssr, rank, sv = np.linalg.lstsq(X, price)

for cand in [w, np.array([40.0, 3.0, 0.0]), np.array([40.0, 0.0, 3 / 10.7639])]:
    print(np.round(np.linalg.norm(cand), 6), np.allclose(X @ cand, X @ w))
# 0.0929 0.0929
print(np.round(w[1] / w[2], 4), np.round(1 / 10.7639, 4))
40.000963 True
40.112342 True
40.000971 True

The shortest way to deliver 3 thousand per square meter through two parallel columns puts weight on each in proportion to its size, so the square-feet column, 10.7639 times larger, carries 10.7639 times the weight, and the ratio of the two weights comes out at 1 to 10.7639. That is a property of the algorithm, not of the listings. Rank 2 of 3 says one direction is missing, and the empty sum of squared residuals is lstsq declining to report one for a rank-deficient table. The predictions are usable, they are the least squares predictions of the two-column model, and the weights for area and square feet are not.

DON'T read the shortest split as the natural one, it is shortest by construction and would change if either column changed units.

Answer 4:

The first conclusion confuses a failed method with failed data. solve raised because it hit an exact zero, but lstsq on the same table returned rank 2 and predictions identical to the two-column line, up to rounding, so everything a model can predict from this table is still there. Dropping one of the two area columns gives the same predictions and weights with one meaning each.

The second treats silence from solve as success. Its weights came out of rounding residue, differ from machine to machine, and differ from lstsq's, which fold back onto the year-only model, so the 4.2419 per year of age is an accident with a unit attached. The predictions are right, and they are the year-only model's. A rank check, 2 of 3, would have flagged the table before either fit ran.

The third reads full rank as health. The table passed the rank check by half a square foot of rounding, its condition number was 4.0275e+04, and re-pricing one listing by 5 thousand flipped the sign of both area weights. Its predictions are usable and its weights are not, which np.linalg.cond would have said before the report was written.

DON'T let any single check stand for the others, the rank says whether a direction is missing, the condition number says whether one nearly is, and neither says anything about the predictions, which survived all three tables.

Assignment

Assignment: Telling lstsq Where Rank Ends

np.linalg.lstsq takes an argument rcond: any singular value of X smaller than rcond times the largest one is treated as zero, so rcond decides where nearly dependent turns into dependent. Its default is tiny, about machine precision times the larger of the two sizes of X. np.linalg.matrix_rank takes the same kind of cutoff as rtol.

Work on the near-collinear table of the last subtitle, ones, area and square feet rounded to whole feet. Print its singular values, choose an rcond that discards only the smallest, and refit. Report the rank lstsq now gives and its weights before and after the 100 square meter listing is re-priced from 332 to 337, beside the same weights for the table with the square-feet column dropped, then print the new predictions. Confirm that matrix_rank with the matching rtol agrees with the rank lstsq reports. Show what a cutoff one step too large does to the fit.

Then explain why discarding one singular value steadied the weights, and why the weights you got still divide the price per square meter between the two area columns.

Hints:

  • Divide every singular value by the largest one, then pick rcond between the two smallest ratios.
  • The smallest singular value belongs to the direction along which the weights were free to swing. Ask what a fit can still do along a direction it has been told does not exist.
Solution: 
import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
price_new = price.copy()
price_new[3] = 337.0
X = np.column_stack([np.ones(5), area, np.round(10.7639 * area)])
X_area = np.column_stack([np.ones(5), area])

w, ssr, rank, sv = np.linalg.lstsq(X, price)
print(sv)          # [2.07654173e+03 6.67017441e-01 5.15593264e-02]
print(sv / sv[0])  # [1.00000000e+00 3.21215525e-04 2.48294198e-05]

for p in [price, price_new]:
    w, ssr, rank, sv = np.linalg.lstsq(X, p, rcond=1e-4)
    w_a, ssr_a, rank_a, sv_a = np.linalg.lstsq(X_area, p)
    print(rank, np.round(w, 4), np.round(w_a, 4))
2 [40.0407  0.9557  0.1899] [40.  3.]
2 [38.8062  0.9272  0.1951] [38.75    3.0274]
import numpy as np

area = np.array([50.0, 60.0, 80.0, 100.0, 120.0])
price = np.array([198.0, 210.0, 285.0, 332.0, 405.0])
X = np.column_stack([np.ones(5), area, np.round(10.7639 * area)])

w, ssr, rank, sv = np.linalg.lstsq(X, price, rcond=1e-4)
print(np.round(X @ w, 2))                   # [189.98 220.05 279.99 339.93 400.05]
print(np.linalg.matrix_rank(X, rtol=1e-4))  # 2
print(np.round(w[1] + 10.7639 * w[2], 4))   # 2.9996

w, ssr, rank, sv = np.linalg.lstsq(X, price, rcond=1e-3)
# 1 991.32
print(rank, np.round(((price - X @ w) ** 2).sum(), 2))

The ratios are 1, 3.2e-04 and 2.5e-05, so any rcond between the last two discards only the smallest singular value, and 1e-4 does. lstsq then reports rank 2, and matrix_rank with rtol=1e-4 agrees. The weights became [40.0407 0.9557 0.1899] and moved only to [38.8062 0.9272 0.1951] under the same 5 thousand re-pricing that flipped the default fit's signs. Their predictions sit within a tenth of the area-only line, and the area-only weights moved just as little.

The smallest singular value belonged to the direction in which the square-meter and square-feet weights trade against each other, more of one and less of the other, with almost no effect on the predictions. Rounding to whole feet left that direction a sliver of influence, so the default fit read real meaning into noise and swung along it. Declaring the direction absent removes it from the fit altogether, and of the weight vectors that remain, lstsq returns the shortest, as it did for the exact twins. That is why the weights still split the price between the two columns, 0.9557 per square meter and 0.1899 per square foot, which together make 2.9996 per square meter: the split is the shortest-length choice, steady because it no longer depends on the noise, and no more meaningful than before.

One step too far, rcond=1e-3, also discards the middle singular value, the direction that let the intercept move apart from the slope, and the fit collapses to rank 1 with a sum of squared residuals of 991.32, close to the 989.11 of 2.3.4's line through the origin. A cutoff decides how much of the table the model may use, and it is set from the singular values, not tuned until the weights look agreeable.

DON'T raise rcond until the weights look plausible, set it from the gap in the singular values, and don't mistake the steady split it produces for a meaningful one, dropping one of the twin columns gives nearly the same predictions with weights that each mean something.

Additional Resources

Optional, and nothing later leans on either. Worth opening if you want the statistician's view of collinear columns, or the exact tolerance behind a rank.

  • Wikipedia - "Multicollinearity", section "Remedies". Regularized fits such as ridge regression and LASSO, which cope with redundant columns instead of requiring one to be dropped.
  • Official docs - numpy.linalg.matrix_rank, the NumPy 2.5 reference page. Its Notes explain the default tolerance, and why a column only numerically close to a combination can still be counted as dependent.

Summary

A feature table can hold fewer directions than columns, and the rank is the honest count. Every column beyond it is a second name for something already there, and the trouble a twin column brings is not in the predictions on the listings the model was fitted to, which survived every table in this lesson, but in the weights, which stop meaning one thing each. That is the line this lesson keeps drawing: what a fit predicts depends on the directions the columns span, and what each weight says depends on there being exactly one way to reach them.

The tools did not agree on how to report the problem, and that disagreement is the lesson's most practical result. solve raised on one exactly dependent table and quietly returned weights on another, lstsq always answered and chose the shortest weights without comment, and a table that missed exact dependence by half a square foot passed every check except the one that measures closeness. No single tool is a test of soundness. The rank, the condition number and a look at whether the weights hold still when the data moves are three different questions.

The fixes are modest and structural: drop a twin, combine the parts into one column, or move a column away from the one it shadows, as centering moved the year away from the column of ones. Each leaves the solver only directions the data can pin down, instead of asking it to guess. Module 3 meets the same problem with real feature tables, where the twins are rarely labeled as such and the condition number is often the first thing to notice them.

Cheat Sheet

Part A - Reference Table

Task NumPy Note
Prove a dependence X @ v all zeros up to rounding, v not all zero v names the combination
Rank np.linalg.matrix_rank(X) at most the smaller of rows and columns
Full column rank np.linalg.matrix_rank(X) == X.shape[1] check before every fit
Singular solve except np.linalg.LinAlgError raising is never guaranteed
Collinear fit w, ssr, rank, sv = np.linalg.lstsq(X, y) rank short, ssr empty, shortest w
Condition number np.linalg.cond(X) equals sv[0] / sv[-1] up to rounding
X.T @ X np.linalg.cond(X.T @ X) the square of cond(X) up to rounding
Center a column year - year.mean() cures a column that shadows the ones
Rank cutoff np.linalg.lstsq(X, y, rcond=1e-4) singular values below rcond * sv[0] count as zero
Twin columns drop one, or combine the pair predictions hold, weights do not

Part B - Compact Code Block

import numpy as np

r = np.linalg.matrix_rank(X)              # independent columns
full = r == X.shape[1]                    # full column rank?
w, ssr, rank, sv = np.linalg.lstsq(X, y)  # ssr empty if rank short
c = np.linalg.cond(X)                     # sv[0] / sv[-1]
year_c = year - year.mean()               # center a year column
w, ssr, rank, sv = np.linalg.lstsq(X, y, rcond=1e-4)

Questions about this lesson: #module-2 on Discord

Found an error in this lesson? Report it