Context

Selection takes the columns and rows you need out of a DataFrame, by name, by position or by a condition, and assignment writes values back into it. In pandas 3 an assignment to a selection lands in the selection alone, never in the table it came from.

This is most of what data work is made of. You take price out as the target and a list of columns as the features, keep the listings that pass a validity rule, mark the suspicious ones for review, add a column such as square meters per room, and set aside the most recent listings to test a model on.

Each of those steps has a way to run cleanly and do the wrong thing. A filter such as price > 300000 drops every listing whose price is missing, and so does its opposite, price <= 300000, which is Lesson 2.1.1's NULL trap met again in pandas. A slice by label on a filtered table can return different rows from a slice by position, with no error. A write made through two brackets in a row prints a warning and changes nothing, so a cleaning step looks as though it ran and the data stays as it was. Lesson 2.4.1 built the table and looked at it, and this lesson takes it apart and writes into it.

Goal

• Select a target column and a list of feature columns, and name the column names attribute access gets wrong.
• Select rows and columns by label with .loc and by position with .iloc, and predict how many rows each slice returns, before and after a filter.
• Filter rows with combined conditions, isin and between, and account for every row a missing value drops.
• Write into a table with one .loc[mask, column] = value, and explain why chained assignment and an assignment to a subset never reach the original in pandas 3.

Prerequisites

Optional, both of them. The lesson does not depend on either, they are here for the picture before the detail.

• YouTube - "Python Pandas Tutorial (Part 2): DataFrame and Series Basics - Selecting Rows and Columns" by Corey Schafer. The loc and iloc part, rows and columns picked by label and by position on a real table.
• Official docs - pandas 3.0 User Guide, "Indexing and selecting data", the "Different choices for indexing" section. The ways pandas offers to select, and what each one accepts.

Table of Contents

• Selecting Columns
• loc - Selection by Label
• iloc - Selection by Position
• Boolean Filtering
• Assignment Under Copy-on-Write

Theory

The blocks run in ~/dev/bootcamp on housing.csv, Lesson 2.4.1's download. A block that does not load the file continues the script above it, and a # raises comment marks a line that stops the script, so delete that line once you have seen its error.

Selecting Columns

What. Selecting columns pulls named columns out of a DataFrame. One name in brackets returns that column as a Series, a list of names returns a DataFrame with those columns in the list's order, and df.price is a shortcut for df["price"] that only some names allow. Why. A model that predicts a target takes its features and its target separately, so splitting a table into the two is the first selection of its training script. A wrong name in brackets fails loudly, and the shortcut can fail without a sound. When. Every time you take the target out, choose a feature set, or narrow a wide table to the columns a step needs.

The brackets are Lesson 2.4.1's, df["price"] a Series and df[["price"]] a one-column DataFrame. A list of several names returns them in the order the list gives, whatever their order in the file:

import pandas as pd

housing = pd.read_csv("housing.csv")
target = housing["price"]
features = housing[["rooms", "area_sqm", "year_built"]]
print(list(features.columns))  # ['rooms', 'area_sqm', 'year_built']
print(features.shape)          # (209, 3)

A list holding a name the frame does not have raises, and the message names every missing name, so a typo in a feature list of twenty columns is found at once:

print(housing[["price", "prices"]])  # raises KeyError: "['prices'] not in index"

Attribute access, housing.price, returns the same Series as housing["price"], and it is short enough that tutorials use it often. It works only when the column name is a valid Python name and is not already the name of something a DataFrame has. A name with a space, such as sale price, cannot be written after a dot at all. A name that a DataFrame already uses is worse, because the dot finds the DataFrame's own attribute and the column is never read:

listings = pd.DataFrame({
    "price": [310000, 245000, 189000],
    "size": [82.5, 64.0, 120.3],
})
print(listings["size"].iloc[0])  # 82.5
print(listings.size)             # 6

listings.size is the number of cells in the table, 3 rows times 2 columns, and nothing in the output says that it is not the size column. count, rank and shape are column names that fail the same way, and brackets work for every name.

DON'T:

• DON'T read a column with a dot when its name could belong to the DataFrame. listings.size returned 6, the cell count, where the column held floor areas, brackets read the column whatever its name.
• DON'T create a column with a dot. housing.sqm_per_room = housing["area_sqm"] / housing["rooms"] creates no column, and pandas only warns, UserWarning: Pandas doesn't allow columns to be created via a new attribute name, so later code that reads housing["sqm_per_room"] raises KeyError.

loc - Selection by Label

What. df.loc[rows, columns] selects by label: row labels from the index in the first position, column names in the second. Each position takes one label, a list of labels, a slice of labels, or a boolean mask for the rows. Why. A label names what you want, the listing labeled 3 or the price column, and it keeps naming the same thing after the table is filtered or reordered. One .loc call also selects rows and columns at the same time, which is the form a write into chosen rows takes, later in this lesson. When. Whenever you know which rows or columns you want by name, and whenever rows are chosen by a condition and columns by name in one step.

A row label and a column name give one value. Row 3 is the listing whose price is missing, and .loc returns nan for it rather than raising:

print(housing.loc[0, "price"])  # 437000.0
print(housing.loc[3, "price"])  # nan

A slice of labels includes its end label. That is the opposite of every slice met so far, lists in Module 1 and arrays in Section 2.2, and it holds for column names too:

print(housing.loc[0:4, ["price", "neighborhood"]])
print(housing.loc[0:2, "price":"rooms"])

      price neighborhood
0  437000.0     Lakeview
1  268000.0     Downtown
2  268000.0    Southgate
3       NaN    Hillcrest
4  150000.0    Hillcrest
      price  area_sqm  rooms
0  437000.0     131.8      2
1  268000.0     112.1      5
2  268000.0     102.2      6

0:4 returned the five rows labeled 0 to 4, and "price":"rooms" the three columns from price to rooms. pandas includes the end because a label often has no obvious next label to stop before. A single row label alone, housing.loc[0], returns that row as a Series labeled by column name.

A boolean mask in the row position keeps the rows where it is True, and the column position narrows the result in the same call:

expensive = housing.loc[housing["price"] > 400000, ["price", "neighborhood"]]
print(expensive.shape)
print(expensive.head())

(32, 2)
       price neighborhood
0   437000.0     Lakeview
18  437000.0     Eastgate
29  413000.0     Lakeview
45  463000.0     Eastgate
49  473000.0     Lakeview

The rows keep their labels from housing, 0, 18, 29, so every row of expensive can still be traced to its listing. That is what the next subtitle turns into a trap.

DON'T:

• DON'T expect a .loc slice to stop before its end. housing.loc[0:4] returns five rows, and a loop or a count written for four is off by one without an error.
• DON'T use .loc with a number to mean "the nth row". .loc[3] is the row labeled 3, which on a filtered table may be anywhere or nowhere.

iloc - Selection by Position

What. df.iloc[rows, columns] selects by position, 0 for the first row or column, negative numbers counting from the end, with slices that stop before their end the way list and array slices do. Why. Some questions are about position and nothing else: the first rows of a file, the last listing appended, the first 80 percent of a shuffled table. .iloc answers them whatever the labels are. When. When the position is what you mean. On a table fresh from read_csv without Lesson 2.4.3's index_col, the labels are 0 to n minus 1, so label 3 and position 3 are the same row, and once a filter drops a row, every label after it stops matching its position.

print(housing.iloc[0:4, [0, 4]])
print(housing.iloc[-1, 4])

      price neighborhood
0  437000.0     Lakeview
1  268000.0     Downtown
2  268000.0    Southgate
3       NaN    Hillcrest
Hillcrest

0:4 gave four rows, positions 0 to 3, and [0, 4] the first and fifth columns. .iloc takes positions only, and a column name after its comma raises ValueError: Location based indexing can only have [integer, integer slice (START point is INCLUDED, END point is EXCLUDED), listlike of integers, boolean array] types.

A filter keeps the labels of the rows it keeps, so once it drops a row, labels and positions part ways from that row on. Here are the listings priced under 200,000:

cheap = housing[housing["price"] < 200000]
print(cheap.index[:6])
print(cheap.loc[0:4, ["price", "neighborhood"]])
print(cheap.iloc[0:4, [0, 4]])

Index([4, 7, 12, 20, 25, 26], dtype='int64')
      price neighborhood
4  150000.0    Hillcrest
       price neighborhood
4   150000.0    Hillcrest
7   173000.0    Hillcrest
12  189000.0     Northend
20  194000.0     Downtown

cheap.loc[0:4] asked for the labels 0 to 4, and only label 4 survived the filter, so it returned one row. cheap.iloc[0:4] asked for the first four positions and returned four rows, labeled 4, 7, 12 and 20. Neither raised, and both answers look like plausible tables. cheap.loc[3] does raise, KeyError: 3, because listing 3 has no price and fell out of the filter. Module 3 cuts tables into a training part and a test part, in Lesson 3.1.2, and a cut written with .loc on a filtered or shuffled table takes rows by label, and can take a different set from the one it meant without an error.

DON'T:

• DON'T take the first n rows of a filtered table with .loc[0:n]. The labels of a filtered table are whatever survived, and .iloc[0:n] is the first n rows.
• DON'T give .iloc column names. After the comma, a single name raises the ValueError above, a list of names an IndexError and a slice of names a TypeError, and .loc is the accessor for names.

Boolean Filtering

What. A comparison on a column returns a boolean Series, True where the row passes, and a DataFrame indexed with that Series keeps the True rows. Conditions combine with &, | and ~, and isin and between build the common ones. Why. Filtering is how you ask a table a question: the listings over a price, the ones in two neighborhoods, the ones with a plausible number of rooms. It is also where a missing value disappears from the answer without a word. When. Every time the rows you need are defined by a rule rather than by label or position.

mask = housing["price"] > 300000
print(mask.dtype)           # bool
print(mask.sum())           # 69
print(housing[mask].shape)  # (69, 7)

mask.sum() counts the True values, the same count Section 2.2's array masks gave. Conditions combine with Lesson 2.2.2's &, | and ~, each comparison in its own parentheses, and the rules are the arrays' rules unchanged. isin tests membership in a list, and between tests a range with both ends included:

both = housing[(housing["rooms"] >= 4) & (housing["price"] > 300000)]
central = housing[housing["neighborhood"].isin(["Riverside", "Downtown"])]
mid = housing[housing["rooms"].between(2, 4)]
print(both.shape)
print(central.shape)
print(mid.shape, mid["rooms"].min(), mid["rooms"].max())

(28, 7)
(48, 7)
(100, 7) 2 4

between(2, 4) kept listings with 2, 3 and 4 rooms, both ends in, like SQL's BETWEEN in Lesson 2.1.1. The missing values are where filtering and the first look meet. Twenty prices are missing, and a missing price is False in every comparison but !=, where it is True:

above = (housing["price"] > 300000).sum()
at_or_below = (housing["price"] <= 300000).sum()
# 69 120 189
print(above, at_or_below, above + at_or_below)

The two conditions look like opposites and together cover 189 listings of 209. The 20 with no price fell out of both, the same way a NULL price fell out of both sides of a WHERE in Lesson 2.1.1. To keep them out on purpose, or to find them, filter on notna(), True where a value is present:

priced = housing[housing["price"].notna()]
unpriced = housing[~housing["price"].notna()]
print(len(priced), len(unpriced))  # 189 20

Lesson 2.5.1 gives the second form a name of its own and decides what to do with those rows. Here the point is narrower: a filter on a column with gaps answers a question about the rows that have a value, and the count tells you how many that is.

DON'T:

• DON'T read df[price > x] and df[price <= x] as a split of the table. Every row with a missing price is in neither, 20 rows of housing.csv.
• DON'T join two masks with and or or. Python asks each Series for a single True or False, and pandas refuses with ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
• DON'T drop the parentheses. With one pair gone, (housing["rooms"] >= 4) & housing["price"] > 300000 raises nothing and keeps 0 listings where the full form keeps 28, because & runs before > and no True or False is greater than 300000. With both gone, housing["rooms"] >= 4 & housing["price"] > 300000 evaluates 4 & housing["price"] first and raises TypeError: Cannot perform 'rand_' with a dtyped [float64] array and scalar of type [bool].

Assignment Under Copy-on-Write

What. Assignment writes values into a DataFrame. df["new"] = ... creates or replaces a whole column, and df.loc[mask, "col"] = value writes into the rows the mask selects. Copy-on-Write, the rule pandas 3 follows, makes an assignment to any object you select from a DataFrame land on that object alone, as if it were an independent copy. Why. A cleaning step is a write. A write that lands somewhere other than the table you meant leaves the data as it was, and the script still runs to the end, so you need to know which forms reach the table and which never do. When. Every time you add a feature column, mark rows, or correct values, and every time you take a subset you intend to change.

A new column is an assignment to a name the frame does not have yet, computed from whole columns at once. Division with a missing value on either side gives a missing value, NaN here, so the new column has a gap wherever area_sqm had one:

housing["sqm_per_room"] = housing["area_sqm"] / housing["rooms"]
print(housing["sqm_per_room"].loc[20:23])

20    24.180
21    29.775
22       NaN
23    22.800
Name: sqm_per_room, dtype: float64

A write into chosen rows takes one .loc with the mask and the column together. review_flags.py marks the listings with more than ten rooms for review, then tries to mark the ones over 1,000 square meters the way tutorials often write it, with two brackets:

import pandas as pd

housing = pd.read_csv("housing.csv")
housing["needs_review"] = False
housing.loc[housing["rooms"] > 10, "needs_review"] = True
print(housing["needs_review"].sum())

# two brackets: the second one writes into a temporary
housing[housing["area_sqm"] > 1000]["needs_review"] = True
print(housing["needs_review"].sum())

2
/home/goku/dev/bootcamp/review_flags.py:9: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  housing[housing["area_sqm"] > 1000]["needs_review"] = True
2

The first .loc marked two listings, the 99-room one and a 12-room one. The chained line, two bracket operations in a row, built a temporary table of the large listings with its first bracket and wrote into that temporary with its second. The temporary was thrown away, housing kept 2 flags, and pandas said so in a ChainedAssignmentError, which is a warning: the script went on and printed the unchanged count. The same holds for the other order, housing["needs_review"][mask] = True, which in pandas 2.2 did change the table, and which tutorials written before pandas 3 still show working. One .loc with both parts, one operation on housing itself, writes:

housing.loc[housing["area_sqm"] > 1000, "needs_review"] = True
print(housing["needs_review"].sum())  # 3

The 12-room listing is also one of the two large ones, so one new flag landed. What Copy-on-Write changes most is a subset you keep. A filtered table is its own table, and a write into it stays there:

riverside = housing[housing["neighborhood"] == "Riverside"]
riverside["needs_review"] = True
print(riverside["needs_review"].sum())  # 28
print(housing["needs_review"].sum())    # 3

That is not how Section 2.2's arrays behaved, and the difference is the Copy-on-Write rule Lesson 2.2.2 named. A NumPy slice is a view, so a write through it reaches the array it came from, while a pandas slice taken with .iloc is a separate table as far as assignments are concerned:

top = housing.iloc[0:3]
top.loc[0, "price"] = 1.0
print(top.loc[0, "price"])      # 1.0
print(housing.loc[0, "price"])  # 437000.0

prices = housing["price"].to_numpy(copy=True)
window = prices[0:3]
window[0] = 1.0
print(prices[0])                # 1.0

In pandas 2.2 the write into top reached housing too, the way the NumPy window reached prices, which is why older code often adds .copy() to a selection it means to change. In pandas 3 that .copy() is not needed to protect the original, because an assignment to a selection never writes back. To change housing, write into housing with .loc.

DON'T:

• DON'T write through two brackets in a row. housing[mask]["col"] = value and housing["col"][mask] = value both leave housing unchanged in pandas 3 and print a ChainedAssignmentError, and housing.loc[mask, "col"] = value, one operation on housing itself, writes.
• DON'T expect an assignment to a subset to reach the table it came from. riverside["needs_review"] = True marked 28 rows of riverside and none of housing.
• DON'T make a new column with a masked .loc alone. housing.loc[mask, "needs_review"] = True on a column that does not exist yet creates it with NaN, not False, in every row outside the mask, which is why the column got its default first.

Lab Exercises

Lab Exercise 1: Features, Labels and Positions

In select_lab.py, load housing.csv. Take price as the target and area_sqm, rooms and year_built, in that order, as the features, and print the features' shape and column list.

Select the price and neighborhood of the listings labeled 10 to 14 with .loc, and the same two columns at positions 10 to 14 with .iloc, written with the same numbers. Print how many rows each returned, and say in a comment why the counts differ.

Then keep only the Downtown listings in downtown. Print its first four labels, then the price and room count of downtown.loc[0:30] and of downtown.iloc[0:3], and say which of the two is "the first three Downtown listings" and what the other one is.

Lab Exercise 2: Filter, Count, Then Write a Band

In filter_lab.py, load housing.csv. Count the listings in Southgate or Northend that have 2 to 4 rooms, using isin and between.

Count the listings priced under 250,000 and the ones priced at 250,000 or more, print both and their sum, and print how many listings have a price at all. Say in a comment where the missing listings went.

Then add a column price_band that is "low" under 250,000, "high" at 250,000 or more, and "unknown" where the price is missing, using .loc for every write into chosen rows, and print its counts. Finally, try to set the band of the listings priced over 1,000,000 to "extreme" with two brackets in a row, and print how many listings are now "extreme".

import pandas as pd

housing = pd.read_csv("housing.csv")
target = housing["price"]
features = housing[["area_sqm", "rooms", "year_built"]]
print(features.shape)                   # (209, 3)
print(list(features.columns))           # ['area_sqm', 'rooms', 'year_built']

by_label = housing.loc[10:14, ["price", "neighborhood"]]
by_position = housing.iloc[10:14, [0, 4]]
print(len(by_label), len(by_position))  # 5 4
# .loc includes the end label 14, .iloc stops before position 14.

downtown = housing[housing["neighborhood"] == "Downtown"]
print(downtown.index[:4])
print(downtown.loc[0:30, ["price", "rooms"]])
print(downtown.iloc[0:3, [0, 2]])

Index([1, 19, 20, 26], dtype='int64')
       price  rooms
1   268000.0      5
19  295000.0      1
20  194000.0      5
26  146000.0      1
28  265000.0      1
       price  rooms
1   268000.0      5
19  295000.0      1
20  194000.0      5

import pandas as pd

housing = pd.read_csv("housing.csv")
in_area = housing["neighborhood"].isin(["Southgate", "Northend"])
mid_size = housing["rooms"].between(2, 4)
print(len(housing[in_area & mid_size]))

low = (housing["price"] < 250000).sum()
high = (housing["price"] >= 250000).sum()
print(low, high, low + high)
print(housing["price"].notna().sum())
# 20 listings have no price, and a missing price is False in both tests.

housing["price_band"] = "unknown"
housing.loc[housing["price"] < 250000, "price_band"] = "low"
housing.loc[housing["price"] >= 250000, "price_band"] = "high"
print(housing["price_band"].value_counts())

housing[housing["price"] > 1000000]["price_band"] = "extreme"
print((housing["price_band"] == "extreme").sum())

28
95 94 189
189
price_band
low        95
high       94
unknown    20
Name: count, dtype: int64
/home/goku/dev/bootcamp/filter_lab.py:19: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  housing[housing["price"] > 1000000]["price_band"] = "extreme"
0

Theory Assessment

Q1: Five Rows, Four Rows, One Row

On housing straight from read_csv, say how many rows housing.loc[0:4] and housing.iloc[0:4] return and why they differ. Then say what each returns on cheap = housing[housing["price"] < 200000], and why one of them shrinks to a single row.

Hint: One accessor reads the numbers in the slice as labels and includes the end, the other reads them as positions and stops before the end. Then check which labels survived the filter.

Q2: The Listings That Fell Out

Explain why (housing["price"] > 300000).sum() and (housing["price"] <= 300000).sum() add up to 189 on a table of 209 rows. Name the SQL lesson where the same thing happened, and give the line that selects exactly the rows both filters missed.

Hint: Ask what a comparison returns for a row whose price is NaN, and which method returns True exactly where a value is present.

Q3: Two Brackets, a Subset and a Window

Explain what housing[housing["area_sqm"] > 1000]["needs_review"] = True does in pandas 3 and why housing does not change, then give the line that does change it. Then explain why riverside["needs_review"] = True on a filtered copy leaves housing alone, while a write into a NumPy slice from Section 2.2 changes the array it came from.

Hint: Count the bracket operations, and ask which object each one acts on. For the second half, recall what Lesson 2.2.2 called a view, and what pandas 3 makes every selection behave as.

Q4: A Script That Runs Clean

Read this script without running it, then say everything it prints, the four values and anything else.

import pandas as pd

housing = pd.read_csv("housing.csv")
cheap = housing[housing["price"] < 200000]

first = cheap.loc[0:10, "price"]
cheap.loc[cheap["rooms"] >= 5, "price"] = 0
housing[housing["price"] < 200000]["price"] = 0

print(len(first))
print(first.loc[7])
print((housing["price"] == 0).sum())
print(len(housing[housing["price"] < 200000]) + len(housing[housing["price"] >= 200000]))

Listing 7 is priced at 173,000 and has 5 rooms. Explain each printed line, and say which line of the script changed data and which data it changed.

Hint: Work out which labels cheap holds between 0 and 10. Then follow each write to the object it lands on, and ask whether first, taken before the write, can see it. The last line is a question about missing prices, not about writes.

/home/goku/dev/bootcamp/q4.py:8: ChainedAssignmentError: A value is being set on a copy of a DataFrame or Series through chained assignment.
Such chained assignment never works to update the original DataFrame or Series, because the intermediate object on which we are setting values always behaves as a copy (due to Copy-on-Write).

Try using '.loc[row_indexer, col_indexer] = value' instead, to perform the assignment in a single step.

See the documentation for a more detailed explanation: https://pandas.pydata.org/pandas-docs/stable/user_guide/copy_on_write.html#chained-assignment
  housing[housing["price"] < 200000]["price"] = 0
2
173000.0
0
189

Assignment

Assignment: A Time-Based Holdout That Loses No Listing

A model is tested on listings it never saw while training, and the most honest test set is the newest listings, because that is what the model will meet in use. Split housing.csv by listing_date: listings dated before 1 July 2024 train the model, listings dated on or after it test it, and listings with no date cannot be placed and are marked as such.

The dates in listing_date are text, and the next section converts them to real dates. You do not need to here: dates written as YYYY-MM-DD, with the year first and every part padded to a fixed width, sort and compare in calendar order as plain strings, so housing["listing_date"] < "2024-07-01" compares dates correctly on the text column. A missing date is False in that comparison, as a missing price was.

Write holdout.py that adds a split column holding "train", "test" or "unknown", using .loc for every write into chosen rows, and proves in code that the three counts add up to the number of rows in the file. Build train and test as separate tables. Print the price, neighborhood and date of the first three test listings, and print what test.loc[0:2] returns, with a comment saying why it is not the same. Add a sqm_per_room column to train alone and prove housing did not get it. Finally, print how many test listings have no price, because a test set with gaps in its target is a problem to know about before any model is tested on it.

Hints:

• Decide what the rows with no date get before you write "train" and "test", and write the two only where a date is present. A row whose date is missing is False in both comparisons and keeps whatever it held before them.
• test keeps the labels its rows had in housing. Ask which accessor reads the numbers you give it as positions.

import pandas as pd

CUTOFF = "2024-07-01"

housing = pd.read_csv("housing.csv")
dated = housing["listing_date"].notna()

housing["split"] = "unknown"
before = dated & (housing["listing_date"] < CUTOFF)
after = dated & (housing["listing_date"] >= CUTOFF)
housing.loc[before, "split"] = "train"
housing.loc[after, "split"] = "test"

counts = housing["split"].value_counts()
print(counts)
print(counts.sum() == len(housing))

split
train      146
test        60
unknown      3
Name: count, dtype: int64
True

train = housing[housing["split"] == "train"]
test = housing[housing["split"] == "test"]
columns = ["price", "neighborhood", "listing_date"]

print(test.iloc[0:3][columns])
# loc reads 0:2 as labels, and labels 0 and 2 are training listings
print(test.loc[0:2, columns])

       price neighborhood listing_date
1   268000.0     Downtown   2024-10-27
5   203000.0    Southgate   2024-11-30
10  222000.0    Hillcrest   2024-07-03
      price neighborhood listing_date
1  268000.0     Downtown   2024-10-27

train["sqm_per_room"] = train["area_sqm"] / train["rooms"]
print("sqm_per_room" in train.columns)    # True
print("sqm_per_room" in housing.columns)  # False

print((~test["price"].notna()).sum())     # 3

Additional Resources

Not needed for anything that follows. Here if you want to go further.

• Official docs - pandas 3.0 API reference, DataFrame.query. Filtering with a condition written as a string, "price > 300000 and rooms >= 4", a second way to write the masks of this lesson.
• Official docs - pandas 3.0 release notes, "Consistent copy/view behaviour with Copy-on-Write". What changed from pandas 2 to pandas 3 and why, including the copies pandas avoids making until a write needs one.

Summary

Selection in pandas comes in three kinds that are easy to blur: by name, by position, and by a rule. On a table whose labels still run 0 to n minus 1, a label and a position name the same row, which is exactly what makes them dangerous, because the habit forms on a table where the difference cannot show and is spent on a filtered one where it shows from the first dropped row on. The useful question is never which accessor is shorter to type, it is whether you mean this listing or the listing in this place.

Filtering carries the oldest trap in the module into its newest tool. The NULL that fell out of both sides of a WHERE in the first SQL lesson is the NaN that falls out of both sides of a mask here, for the same reason, a missing value is not greater or smaller than anything. The defense is the same too: count what came out against what went in, and know how many rows had a value before trusting a split that looks complete.

Copy-on-Write is the part of pandas 3 that most separates it from the pandas in older tutorials, and it simplifies more than it complicates. An assignment to a selection lands in the selection and nowhere else. A chained assignment can never work, an assignment to a subset can never quietly change its parent, and the one rule left to remember is to write into the table you mean to change, with one .loc.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import pandas as pd

housing = pd.read_csv("housing.csv")
y = housing["price"]                         # target, a Series
X = housing[["area_sqm", "rooms"]]           # features, a DataFrame

housing.loc[0:4, ["price", "neighborhood"]]  # labels 0-4, five rows
housing.iloc[0:4, [0, 4]]                    # positions 0-3, four rows

cheap = housing[housing["price"] < 200000]
cheap.iloc[0:3]                              # first three cheap rows
mask = (housing["rooms"] >= 4) & housing["price"].notna()
housing[housing["rooms"].between(2, 4)]      # 2, 3 and 4 rooms

housing["needs_review"] = False              # default first
housing.loc[housing["rooms"] > 10, "needs_review"] = True
# chained, never writes: housing[mask]["needs_review"] = True