FREE~1.5 h2 labs4 questions1 assignment

Context

Indexing picks elements out of an array by position: one value, a run of values, a row or a column of a table, or any list of positions in any order, while indexing with a boolean mask, an array of True and False answers to a condition, one per element, keeps the elements where the answer is True. Some of these selections share memory with the array they came from, and some are copies.

Selecting is most of the daily work on a feature table. Pulling the target column away from the features, holding out a few rows for testing, keeping the listings inside a price band, counting how many rows fail a range check and repairing an impossible value are each one line of indexing. A mask is how a filter or a validity check is written once for every row instead of in a loop.

The damage here is silent. A slice of an array is a view onto the same memory, so a what-if edit made on a slice rewrites the original table, and every number computed afterwards comes from data nobody meant to change. Combining two conditions with the word "and" stops the script with an error about ambiguity, and leaving out the parentheses around the comparisons either stops it with an error too or returns the wrong mask without a word. The previous lesson built arrays, and this one takes them apart.

Goal

  • Read single elements, runs, rows, columns and sub-blocks of 1-D and 2-D arrays with indexes and slices.
  • Select and reorder rows with a list of positions, applied to features and prices alike.
  • Predict whether a selection is a view or a copy, and prove it with np.shares_memory.
  • Filter, count and repair values with boolean masks, telling a read through a mask from an assignment through one.
  • Combine conditions with &, | and ~, and choose values element by element with np.where.

Prerequisites

Neither is required, and the lesson stands without them. They are here if you want to see selection done once before you do it yourself.

Table of Contents

  • Indexing and Slicing
  • Fancy Indexing
  • Views, Copies and Copy-on-Write
  • Boolean Masks
  • Combining Conditions and np.where

Theory

The lesson's table is six housing listings, one row each, with three features per row: area_sqm, rooms and age. X is the conventional name for a feature table in ML code, and the prices sit beside it in a 1-D array of their own, one per row.

Indexing and Slicing

What. An index picks one element by position, counting from 0 at the front or from -1 at the back. A slice, start:stop:step, picks a regular run of positions and stops before stop. On a 2-D array, X[row, col] takes one index or slice per dimension, rows first. Why. Every column pull, row split and spot check is a selection by position: the target column, the first rows for training, one listing's features. When. Use an index for one value, a slice for a regular run, and the [row, col] form whenever the array has rows and columns.

A 1-D array indexes and slices exactly as a list did in Lesson 1.4.1: zero-based, negative from the end, start included and stop excluded.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

print(prices[0])     # 310000
print(prices[-1])    # 265000
print(prices[1:4])   # [455000 289000 520000]
print(prices[::2])   # [310000 289000 398000]
print(prices[::-1])  # [265000 398000 520000 289000 455000 310000]

On a 2-D array the comma separates one selection per dimension. One index alone takes a whole row, and : in the row position means every row, so X[:, j] is column j, returned as a 1-D array.

import numpy as np

# one row per listing: area_sqm, rooms, age
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

print(X[1, 0])        # row 1, column 0
print(X[2])           # row 2, every column
print(X[:, 0])        # every row, column 0
print(X[:, 0].shape)  # a column comes back 1-D
print(X[0:2, 1:3])    # rows 0-1, columns 1-2
110.0
[64.  2. 49.]
[ 72.5 110.   64.  135.5  98.   58.5]
(6,)
[[ 3. 26.]
 [ 4. 19.]]

A slice that runs past the end returns whatever exists, and an index past the end raises. Saved as index_error.py:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
print(prices[2:10])
print(prices[6])
[289000 520000 398000 265000]
Traceback (most recent call last):
  File "/home/goku/dev/bootcamp/index_error.py", line 5, in <module>
    print(prices[6])
          ~~~~~~^^^
IndexError: index 6 is out of bounds for axis 0 with size 6

axis 0 is the first dimension, the only one a 1-D array has. On X, X[6, 0] names axis 0 the same way, and X[0, 3] reports index 3 is out of bounds for axis 1 with size 3, the columns.

DON'T:

  • DON'T chain brackets to reach a column. X[:][0] is X[:], every row, and then its row 0, so it returns the first listing, [72.5 3. 26. ], where X[:, 0] returns the first feature of every listing.
  • DON'T read X[0] as the first feature. One index selects along the first dimension, so it is the first listing's row, and the first feature is X[:, 0], with the : in the row position.
  • DON'T trust a slice to tell you it ran short. prices[2:10] quietly returns the four values that exist, and only an index past the end raises.

Fancy Indexing

What. Fancy indexing passes a list, or an integer array, of positions as the index. It returns the elements at those positions in the order given, repeats allowed, and on a 2-D array a list in the row position selects whole rows. Why. A slice can only take a regular run. Holding out listings 1 and 4 for testing, reordering rows by a ranking, or taking the rows an earlier step chose needs arbitrary positions. When. Use it whenever the positions you want are not a regular run, and apply the same list to every array that describes the same rows.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

test_rows = [1, 4]               # two listings held out for testing
test_X = X[test_rows]            # their feature rows
test_prices = prices[test_rows]  # the same positions keep each price with its row
print(test_X)
print(test_prices)
print(prices[[5, 0, 2]])         # any order
print(prices[[2, 2]])            # repeats allowed
[[110.   4.  19.]
 [ 98.   4.  33.]]
[455000 398000]
[265000 310000 289000]
[289000 289000]

The double brackets are two different things: the outer pair is indexing, the inner pair is a Python list of positions. The result is always a new array of its own, which the next subtitle proves.

DON'T:

  • DON'T drop the inner brackets. prices[5, 0, 2] asks for three dimensions of a 1-D array and raises IndexError: too many indices for array: array is 1-dimensional, but 3 were indexed.
  • DON'T select features and prices with different position lists. The same list keeps each price beside its own row, and nothing raises when the two lists disagree.

Views, Copies and Copy-on-Write

What. A basic slice returns a view, a second array that reads and writes the same memory as its parent, so a write through it lands in the parent. .copy() returns an array with memory of its own, and fancy and boolean selections are copies too. Why. A view lets you work on part of a large table without duplicating it, and the price is that an edit you meant to keep local reaches the original without a warning. Lesson 1.4.1's list slice built a new list, and an array slice does not. When. Ask view or copy every time you write into something you selected, and take .copy() whenever the original must stay as it was.

np.shares_memory(a, b) returns True when two arrays use the same memory, which settles the question without changing anything. Here a slice is used to try a what-if price on listing 1:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

window = prices[1:4]                     # a VIEW of positions 1 to 3
window[0] = 450000                       # a what-if price for listing 1
print(window)                            # [450000 289000 520000]
print(prices)                            # [310000 450000 289000 520000 398000 265000]
print(np.shares_memory(prices, window))  # True

prices[1] is now 450000, and nothing said so. .copy() breaks the link:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

safe = prices[1:4].copy()              # memory of its own
safe[0] = 450000
print(safe)                            # [450000 289000 520000]
print(prices)                          # [310000 455000 289000 520000 398000 265000]
print(np.shares_memory(prices, safe))  # False

A column is a slice too, so writing into a column you pulled out of the table writes into the table. A fancy selection is a copy, so a write into its result never reaches the parent. An index on the left of = is different again: it names positions of the original to write into, so prices[[1, 3]] = 0 changes prices itself.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

area = X[:, 0]                           # a column: a VIEW
area[2] = 0
print(X[2])                              # [ 0.  2. 49.]

picked = prices[[1, 2, 3]]               # fancy: a COPY
picked[0] = 450000
print(prices)                            # [310000 455000 289000 520000 398000 265000]
print(np.shares_memory(prices, picked))  # False

prices[[1, 3]] = 0                       # index on the LEFT: writes
print(prices)                            # [310000      0 289000      0 398000 265000]

A reshape from Lesson 2.2.1 is a view as well when it can be, and np.shares_memory(prices, prices.reshape(2, 3)) returns True.

Copy-on-Write is the name of a different rule: a selection behaves as its own copy the moment anyone writes to it, so a write through a selection never reaches the parent. NumPy does not follow it, as window just showed. pandas, the table library Section 2.4 installs, does, so the same-looking line, select part of a column and write into it, changes the original in NumPy and leaves it alone in pandas. Lesson 2.4.2 demonstrates it on a table.

DON'T:

  • DON'T try a what-if on a slice. window = prices[1:4] followed by window[0] = 450000 rewrote prices[1], and a .copy() first is what keeps the original.
  • DON'T pull a column out to clean it and assume the table is safe. X[:, 0] is a view, so zeroing one of its values zeroed X[2, 0].
  • DON'T expect a write into a fancy or boolean selection to reach the parent. picked = prices[[1, 2, 3]] is a copy, and picked[0] = 450000 left prices as it was.
  • DON'T assume a line that selects and writes behaves the same in Section 2.4's table library. NumPy writes through a slice, and pandas, which follows Copy-on-Write, does not.

Boolean Masks

What. A comparison between a numeric array and a number returns a boolean array, dtype bool, of the same shape, True where the condition holds. Indexing with that mask keeps the True positions. Why. A filter, a count and a repair are each written once for every element, with no loop over rows, and the mask that drives them can be read and checked on its own. When. Whenever the elements you want are defined by a condition rather than by positions.

True counts as 1 and False as 0, so sum counts the matches and mean gives their share. Both run here over the whole mask, and Lesson 2.2.5 gives them an axis.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

expensive = prices > 400000  # one True or False per listing
print(expensive)             # [False  True False  True False False]
print(expensive.dtype)       # bool
print(prices[expensive])     # [455000 520000]
print(expensive.sum())       # 2
print(expensive.mean())      # 0.3333333333333333

A mask built from one column has one answer per row, so indexing the whole table with it keeps entire rows, every feature of each listing that passes.

import numpy as np

X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

recent = X[:, 2] < 30  # age under 30, one answer per row
print(recent)
print(X[recent])
[ True  True False  True False False]
[[ 72.5   3.   26. ]
 [110.    4.   19. ]
 [135.5   5.    8. ]]

Reading through a mask builds a copy, like any fancy selection. Assigning through one, the mask on the left of =, writes into the original, which is how an impossible value is repaired in place:

import numpy as np

# one listing is still being built, so its age came out as -1
age = np.array([26, 19, 49, -1, 33, 61])

young = age[age < 30]  # READ through a mask: a copy
young[0] = 99
print(age)             # [26 19 49 -1 33 61]

age[age < 0] = 0       # ASSIGN through a mask: writes into 'age'
print(age)             # [26 19 49  0 33 61]

DON'T:

  • DON'T expect a write into age[age < 30] to reach age. Through a mask, only age[mask] = value, with the mask on the left of =, writes into the original.
  • DON'T build a mask on an array whose dtype you have not checked. On Lesson 2.2.1's text array np.array([1998, 2005, "unknown", 2016]), the comparison == 2005 returns [False False False False], with no error and no match.
  • DON'T index with a mask built from a different-length array. A 5-element mask on the 6 prices raises IndexError: boolean index did not match indexed array along axis 0; size of axis is 6 but size of corresponding boolean axis is 5.

Combining Conditions and np.where

What. &, | and ~ combine masks element by element as and, or and not, with each comparison in its own parentheses. np.where(cond, a, b) builds a new array taking a where cond is True and b elsewhere, and np.where(cond) returns the positions where cond is True. Why. Real filters have several conditions, a price band and enough rooms, and Python's and and or need a single True or False on each side, which an array of six answers cannot give. When. Whenever a selection depends on more than one condition, or every element needs its own if-else.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
rooms = np.array([3, 4, 2, 5, 4, 2])

both = (prices > 300000) & (rooms >= 4)    # and, element by element
print(prices[both])                        # [455000 520000 398000]
either = (prices < 280000) | (rooms >= 5)  # or
print(prices[either])                      # [520000 265000]
print(prices[~both])                       # [310000 289000 265000]

& binds more tightly than >, so without the parentheses Python reads prices > 300000 & rooms >= 4 as prices > (300000 & rooms) >= 4, a chained comparison like Lesson 1.1.4's 0 < x < 1, and a chained comparison needs one True or False from its first half. Both mistakes end in the same error:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
rooms = np.array([3, 4, 2, 5, 4, 2])

try:
    both = (prices > 300000) and (rooms >= 4)
except ValueError as err:
    print("and:", err)

try:
    both = prices > 300000 & rooms >= 4
except ValueError as err:
    print("no parentheses:", err)
and: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
no parentheses: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

np.where with three arguments is an if-else applied to every element, and it returns a new array, leaving its inputs alone. With only the condition, it returns a tuple holding one array of positions per dimension, a single array for a 1-D input.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

tier = np.where(prices > 400000, "premium", "standard")
print(tier)                       # ['standard' 'premium' 'standard' 'premium' 'standard' 'standard']

capped = np.where(prices > 500000, 500000, prices)
print(capped)                     # [310000 455000 289000 500000 398000 265000]
print(prices)                     # [310000 455000 289000 520000 398000 265000]

print(np.where(prices > 400000))  # (array([1, 3]),)

DON'T:

  • DON'T write and, or or not between arrays. Each asks for one True or False and raises ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all(), where &, | and ~ work element by element.
  • DON'T leave out the parentheses around a comparison you combine. & binds more tightly than >, so prices > 300000 & rooms >= 4 raises the same ValueError, and (prices > 300000) & rooms >= 4, one pair short, raises nothing and returns False for every listing.
  • DON'T expect np.where(cond, a, b) to change its inputs. capped holds 500000 where prices still holds 520000.

Lab Exercises

Lab Exercise 1: Slice a Listing Table, and Catch a Slice Writing Back

Start from these two arrays:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

Print the last listing's price with a negative index, the first three prices with a slice, every listing's room count, and the area_sqm and rooms of listings 2 and 3 as one 2 by 2 block.

Hold out listings 0 and 5 as a test set with one list of positions applied to both arrays, and print the test rows and their prices.

Then take rooms_col = X[:, 1], set its first element to 10, print X[0], and print whether rooms_col and X share memory. Rebuild the table, repeat the edit on X[:, 1].copy() instead, and print X[0] and the same check again.

Lab Exercise 2: Filter, Count and Repair With Masks

Start from these three arrays, one value per listing:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
rooms = np.array([3, 4, 2, 5, 4, 2])
age = np.array([26, 19, 49, -1, 33, 61])

Print how many listings cost more than 400000 and what share of all listings that is. Print the prices of listings that have at least 4 rooms and are under 30 years old, then the prices of listings that cost less than 280000 or are older than 50.

Repair the age of -1, a listing still being built, to 0 with one assignment through a mask, and print age.

Label each listing "new" if it is under 20 years old and "established" otherwise with np.where, and print the labels. Finally, print the positions of the listings that cost more than 400000.

Solution 1: Slice a Listing Table, and Catch a Slice Writing Back 
import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

print(prices[-1])
print(prices[:3])
print(X[:, 1])
print(X[2:4, 0:2])  # rows 2-3, columns 0-1

test_rows = [0, 5]
print(X[test_rows])
print(prices[test_rows])
265000
[310000 455000 289000]
[3. 4. 2. 5. 4. 2.]
[[ 64.    2. ]
 [135.5   5. ]]
[[72.5  3.  26. ]
 [58.5  2.  61. ]]
[310000 265000]
import numpy as np

rows = [[72.5, 3, 26], [110.0, 4, 19], [64.0, 2, 49],
        [135.5, 5, 8], [98.0, 4, 33], [58.5, 2, 61]]

X = np.array(rows)
rooms_col = X[:, 1]                    # a slice: a view of X
rooms_col[0] = 10
print(X[0])                            # [72.5 10.  26. ]
print(np.shares_memory(X, rooms_col))  # True

X = np.array(rows)                     # rebuilt from the list
rooms_col = X[:, 1].copy()             # memory of its own
rooms_col[0] = 10
print(X[0])                            # [72.5  3.  26. ]
print(np.shares_memory(X, rooms_col))  # False

DON'T:

  • DON'T let a separate-looking name fool you. rooms_col reads like a variable of its own, and writing into it rewrote listing 0 inside X.
  • DON'T reach for X[2:4][0:2] for a block. Chained, the second slice takes rows again, so the block needs one bracket with a comma, X[2:4, 0:2].
Solution 2: Filter, Count and Repair With Masks 
import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
rooms = np.array([3, 4, 2, 5, 4, 2])
age = np.array([26, 19, 49, -1, 33, 61])

expensive = prices > 400000
print(expensive.sum())   # the count
print(expensive.mean())  # the share

print(prices[(rooms >= 4) & (age < 30)])
print(prices[(prices < 280000) | (age > 50)])

age[age < 0] = 0         # writes into 'age' itself
print(age)

# the repaired 0 counts as new
print(np.where(age < 20, "new", "established"))
print(np.where(prices > 400000))
2
0.3333333333333333
[455000 520000]
[265000]
[26 19 49  0 33 61]
['established' 'new' 'established' 'new' 'established' 'established']
(array([1, 3]),)

DON'T:

  • DON'T build the masks before the repair and reuse them after it. A mask is a snapshot of the values when it was built, and age < 0 computed before the repair still marks listing 3 after its age became 0.
  • DON'T read np.where(prices > 400000) as an array of positions. It is a tuple holding one, and the positions themselves are its first item, [1 3].

Theory Assessment

Q1: Four Selections on One Table

For the six-listing X of this lesson, state the shape of each of X[2], X[:, 2] and X[1:3, :2], what X[-1, 0] returns, and what each one holds in listing terms. Then explain why X[:][0] does not return the first column.

Hint: A single index selects along the first dimension. For the last part, evaluate the expression left to right and ask what X[:] is before the [0] is applied.

Q2: Which Writes Reach the Parent

Starting from prices each time, three selections are taken and each has 0 written into its first element: a = prices[2:5], b = prices[[2, 3, 4]] and c = prices[prices > 300000]. Say which of the three writes changes prices and why the other two do not. Then give two ways to write 0 into position 2 of prices, one with a list of positions and one with a mask, that do change it.

Hint: Sort the three selections into views and copies first. For the second part, the index has to be on the left of =.

Q3: Two Conditions, Three Spellings

Explain why (prices > 300000) and (rooms >= 4) raises an error, why prices > 300000 & rooms >= 4 raises the same error, and what ((prices > 300000) & (rooms >= 4)).mean() returns in plain terms.

Hint: Ask how many True or False values each side of and produces, and which operator Python applies first in the version without parentheses. For the last part, remember what True and False count as.

Q4: Three Ways to Zero an Area

A teammate wants to set area_sqm to 0 for every listing older than 40 years, as a test of a later step, and tries three ways, each on a fresh copy of the lesson's X:

import numpy as np

X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, 33],
    [58.5, 2, 61],
])

Xa = X.copy()
Xa[Xa[:, 2] > 40][:, 0] = 0  # way (a)

Xb = X.copy()
old = Xb[:, 2] > 40
Xb[old, 0] = 0               # way (b)

Xc = X.copy()
area = Xc[:, 0]
area[Xc[:, 2] > 40] = 0      # way (c)

print(Xa[:, 0])
print(Xb[:, 0])
print(Xc[:, 0])

None of them raises. State what each print shows, and explain each result, naming every view and copy involved.

Hint: Classify every selection in each way: a slice, a read through a mask, or an index on the left of =. One of the three writes into something that is thrown away as soon as its line ends.

Answer 1:

X[2] has shape (3,), the third listing's row, its area, rooms and age. X[:, 2] has shape (6,), the age of every listing. X[-1, 0] is a single value, 58.5, the last listing's area. X[1:3, :2] has shape (2, 2), the area and rooms of listings 1 and 2.

X[:][0] is evaluated left to right. X[:] selects every row, which is the whole table again, and [0] then selects row 0 of that, so the result is the first listing, [72.5 3. 26. ], not the first feature. The column needs both selections inside one pair of brackets, X[:, 0].

DON'T split a 2-D selection across two pairs of brackets, the second pair always starts again at the first dimension.

Answer 2:

Only the write into a changes prices. a = prices[2:5] is a basic slice, a view over the same memory, so a[0] = 0 writes 0 into prices[2]. b = prices[[2, 3, 4]] is fancy indexing and c = prices[prices > 300000] is a boolean selection, and both build new arrays with memory of their own, so their writes land in the copies and prices never sees them.

Two ways that do change prices put the index on the left of =, where it names positions of the original: prices[[2]] = 0, a list of positions, and prices[prices == 289000] = 0, a mask. np.shares_memory returns True for a and False for b and c, which settles the question without writing anything.

DON'T decide view or copy by what the selection contains, prices[2:5] and prices[[2, 3, 4]] hold the same three values and only one of them is a view.

Answer 3:

and asks each side for one True or False. prices > 300000 is six answers, and NumPy refuses to boil six answers down to one by guessing, so it raises ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

Without the parentheses, & binds first, so the line is read as prices > (300000 & rooms) >= 4, a chained comparison, which Python evaluates as one comparison and the other. That hidden and meets the same six answers and raises the same error. With every comparison in parentheses, & combines two masks element by element.

((prices > 300000) & (rooms >= 4)).mean() is the share of listings that cost more than 300000 and have at least 4 rooms, because each True counts as 1 and each False as 0. For the lesson's six listings it is 0.5, three of six.

DON'T read the ambiguity error as a problem with your data, it is a problem with the operator, and the fix is &, | or ~ with parentheses.

Answer 4: 
[ 72.5 110.   64.  135.5  98.   58.5]
[ 72.5 110.    0.  135.5  98.    0. ]
[ 72.5 110.    0.  135.5  98.    0. ]

Only way (a) leaves its table unchanged. Xa[Xa[:, 2] > 40] reads through a mask and builds a copy of the two old listings' rows. [:, 0] = 0 then writes into that copy, which is never stored in a name and is thrown away when the line ends.

In way (b), the mask and the column index sit together on the left of =, Xb[old, 0] = 0, which names positions of Xb itself, so listings 2 and 5 get an area of 0.

In way (c), area = Xc[:, 0] is a basic slice, a view of the area column. area[mask] = 0 assigns through a mask, which writes into area, and since area shares memory with Xc, the zeros land in Xc.

DON'T chain a mask read and a write in one line and trust it, write through a single index on the original, X[mask, col] = value, so there is no copy in between.

Assignment

Assignment: Validate, Repair and Rank a Listing Table Without Losing Track of a Row

Eight listings arrive for a review before any model sees them. The rules are: an area above 20 square meters, between 1 and 8 rooms, an age of 0 or more, and a price above 0. One listing's age is -1 because it is still being built, which counts as a repairable 0, and any other row that breaks a rule is dropped.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000, 0, 610000])
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, -1],
    [58.5, 2, 61],
    [0.0, 3, 12],
    [142.0, 12, 4],
])

Repair the age in X itself, with one line that writes through a mask and a column index, and print the repaired row. Build one mask that is True for every row breaking any rule, print how many rows it marks and their positions, then drop those rows from both arrays at once. The reviewer wants the clean table ordered from most to least expensive with every row still next to its price. np.argsort(a) returns the positions that would put a in ascending order, for example [1 2 0] for [3, 1, 2], and those positions work as a fancy index. Print the ordered prices and table. Finally, hand the two most expensive rows to a second reviewer as an array they can edit freely, and prove that an edit there cannot reach your clean table.

Hints:

  • Joining five conditions with | builds the one "breaks a rule" mask, and ~ turns it into the rows to keep. Apply the same mask to X and to prices.
  • Reversing a position array with [::-1] turns ascending into descending, and a fancy selection is a copy.
Solution: 
import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000, 0, 610000])
X = np.array([
    [72.5, 3, 26],
    [110.0, 4, 19],
    [64.0, 2, 49],
    [135.5, 5, 8],
    [98.0, 4, -1],
    [58.5, 2, 61],
    [0.0, 3, 12],
    [142.0, 12, 4],
])

# mask and column on the LEFT of '=': writes into X itself
X[X[:, 2] < 0, 2] = 0
print(X[4])

area, rooms, age = X[:, 0], X[:, 1], X[:, 2]
bad = (area <= 20) | (rooms < 1) | (rooms > 8) | (age < 0) | (prices <= 0)
print(bad.sum())
print(np.where(bad))

keep = ~bad
X_clean = X[keep]                       # one mask for both arrays keeps rows paired
prices_clean = prices[keep]

order = np.argsort(prices_clean)[::-1]  # most expensive first
print(prices_clean[order])
print(X_clean[order])

review = X_clean[order[:2]]             # fancy selection: a copy
review[0, 0] = 0
print(np.shares_memory(X_clean, review))
print(X_clean[3])                       # the top listing, unchanged
[98.  4.  0.]
2
(array([6, 7]),)
[520000 455000 398000 310000 289000 265000]
[[135.5   5.    8. ]
 [110.    4.   19. ]
 [ 98.    4.    0. ]
 [ 72.5   3.   26. ]
 [ 64.    2.   49. ]
 [ 58.5   2.   61. ]]
False
[135.5   5.    8. ]

Row 6 breaks two rules at once, an area of 0.0 and a price of 0, and the mask still counts it once, because | joins the conditions per row before anything is counted. Row 7 breaks the room rule with 12. area, rooms and age are views of X's columns, which is harmless here because the line that builds bad only reads them.

The order of operations matters twice. The repair runs before the mask is built, so the listing still being built passes the age rule and stays, while the same mask built first would have dropped it. The ordering is computed once, as a position array, and applied to both arrays, so every price stays beside its row. np.sort(prices_clean), which returns the values themselves in order, would have sorted the prices alone and left each one beside another listing's features.

DON'T sort a target and its features independently, a sort reorders each array by its own values and silently breaks the row pairing, and don't repair a value by writing into a mask read such as X[X[:, 2] < 0][:, 2] = 0, which writes into a copy and repairs nothing.

Additional Resources

Optional, and nothing later in the bootcamp depends on either. Worth opening if you want the other test for a view, or the one assignment that surprises even people who know this lesson.

Summary

The syntax of this lesson is small, a few brackets, a colon, a comma, and the weight sits in a question the syntax never shows: does the thing you selected own its memory? A slice does not. It is a window onto the array it came from, cheap because nothing is copied, and dangerous for exactly the same reason. Fancy indexing and masks sit on the other side of the line and always hand back something new. Writing through an index on the left of = is a third case again, neither a view nor a copy, a direct instruction to the original. Nearly every surprising result in this lesson came from mistaking one of these for another.

Masks are where selection turns into reasoning about data. A comparison on a numeric array asks every element the same question at once, and the answer is itself an array you can count, average, combine and inspect before you trust it. The two failures that come with it are loud and quiet in turn: and between arrays refuses with an error, while a mask built on the wrong dtype or before a repair answers confidently and wrongly.

The Copy-on-Write rule named here is the other library's answer to the same tension. NumPy chose speed and makes you ask view or copy every time, and the table library of Section 2.4 chose safety and copies the moment you write. Neither is wrong, and knowing which one you are holding is what keeps a what-if edit from becoming a silent change to the data every later number depends on. Lesson 2.2.3 stops selecting and starts computing across whole arrays at once.

Cheat Sheet

Part A - Reference Table

Task Code Gives Watch for
One element a[i], a[-1] a value past the end raises IndexError
A run a[start:stop:step] a VIEW stop excluded, past the end is clipped
Cell, row, column X[r, c], X[r], X[:, c] value, VIEW, VIEW X[:][0] is a row, not a column
Block X[r0:r1, c0:c1] a VIEW one bracket, one comma
Positions a[[i, j, k]], X[[i, j]] a COPY inner brackets, any order
Independent copy a[...].copy() a COPY
Same memory? np.shares_memory(a, b) True or False checks without writing
Mask a > v bool array check the dtype first
Filter a[mask], X[mask] a COPY rows kept whole on a 2-D table
Count, share mask.sum(), mask.mean() count, fraction True counts as 1
Repair in place a[mask] = v, X[mask, c] = v writes the original index on the LEFT of =
Combine (m1) & (m2), (m1) | (m2), ~m bool array parentheses, never and/or
If-else per element np.where(cond, a, b) new array inputs unchanged
Positions of True np.where(cond) tuple of arrays positions are item [0] on 1-D

Part B - Compact Code Block

import numpy as np

prices = np.array([310000, 455000, 289000, 520000])
X = np.array([[72.5, 3, 26], [110.0, 4, 19]])

X[1, 0], X[0], X[:, 0]                 # cell, row, column
w = prices[1:3]                        # VIEW: writes reach prices
s = prices[1:3].copy()                 # COPY: writes stay in s
np.shares_memory(prices, w)            # True

m = (prices > 300000) & (prices < 500000)
prices[m]                              # filter, a COPY
m.sum(), m.mean()                      # count, share
prices[prices > 500000] = 500000       # assign through a mask
np.where(prices > 400000, "hi", "lo")  # new array
np.where(prices > 400000)              # (array([1, 3]),)

Questions about this lesson: #module-2 on Discord

Found an error in this lesson? Report it