Installing pandas, the Series and the DataFrame

Context

pandas is a library that puts labels on data. Its Series is one column of values with a label attached to every value, and its DataFrame is a table of such columns sharing one set of row labels, so a value can be reached by the name of its row and its column instead of by a position.

Every dataset in the rest of this module, and every table Module 3 trains on, is held this way. You load the housing listings into a DataFrame, pull out price as the column a model will learn to predict, count the listings in each neighborhood before deciding whether a category is too rare to keep, and in Module 3 hand the feature columns to a model, which takes a DataFrame of numbers with no gaps as it is.

Labels change what arithmetic does. Two Series are combined by matching labels, so a feature built from two sources that do not list the same neighborhoods comes out with missing values instead of an error, and a total over it quietly leaves those rows out. The other failure is the first look nobody took: twenty missing prices, a price in the millions and a listing with no condition sit in the course's file from the start, and none of them raises anything. Section 2.2's arrays reached every value by position, and this lesson gives each value a name.

Goal

• Install pandas 3.0.5 into (bootcamp) with uv pip install and confirm the version your scripts import.
• Construct a Series from a list and from a dict, and select from it by label with .loc and by position with .iloc.
• Predict which labels come out as NaN when two Series meet in arithmetic, and add counts with .add(..., fill_value=0) where a missing label means zero.
• Build a DataFrame from a dict of lists and distinguish df["price"], a Series, from df[["price"]], a one-column DataFrame.
• Run a first look at housing.csv in a fixed order, size, rows, types and gaps, ranges, categories, and state what each step found.

Prerequisites

None of this is required, the lesson stands on its own. Both are here if you want to see a table loaded and inspected before you do it yourself.

• YouTube - "Python Pandas Tutorial (Part 1): Getting Started with Data Analysis - Installation and Loading Data" by Corey Schafer. The loading and first-look part, for the picture of a table on screen. It works in Jupyter, a notebook that runs Python in the browser, where (bootcamp) runs scripts, and its install step is for its own setup, not for (bootcamp).
• Official docs - pandas 3.0 User Guide, "Intro to data structures", the "Series" section. What a Series holds, and how it behaves both like an array and like a dict.

Table of Contents

• Installing pandas
• The Series
• Index Alignment
• The DataFrame
• A First Look at a Dataset

Theory

This lesson's download is housing.csv, the course's housing listings, one listing per row. Save it into ~/dev/bootcamp now, as Module 2's setup page describes, and A First Look at a Dataset opens it.

Installing pandas

What. pandas is a third-party library for labeled tables, built on top of NumPy. It does not ship with Python, so it is installed into (bootcamp) before the first import pandas can work. Why. A NumPy array from Section 2.2 holds one dtype for every value and reaches each value by position, with no name for any row or column. A dataset has a name for every column, a different type in different columns and gaps in some of them, and pandas is the layer that carries all three. When. Once, now, with (bootcamp) active. Every later section of this module works on what this line installs.

In ~/dev/bootcamp, install the version every result in this module was produced with:

# installs into the active (bootcamp)
uv pip install pandas==3.0.5 python-dateutil==2.9.0.post0

Among the progress lines uv prints, the ones that matter start with +, one for each library it added:

 + pandas==3.0.5
 + python-dateutil==2.9.0.post0
 + six==1.17.0

python-dateutil is a library pandas needs, pinned beside it because every result in this module was produced with this version of it, and six is one python-dateutil needs in turn, not pinned, so its line can read differently on your machine. numpy has no + line, because Lesson 2.2.1 installed numpy 2.5.2 and pandas 3.0.5 runs on it as it is. Before the install, import pandas raises ModuleNotFoundError: No module named 'pandas', and after it:

# 3.0.5
python3 -c "import pandas as pd; print(pd.__version__)"

pd is the short name nearly all pandas code gives the module, so every script from here on that uses pandas imports it with import pandas as pd. pandas 3.0.0 was released in January 2026, so most tutorials you will find were written for an earlier pandas, and three of the behaviors this section relies on changed with version 3: the dtype of a text column and what a single integer in plain brackets means, both below, and whether an assignment through a selection reaches the table, in Lesson 2.4.2.

DON'T:

• DON'T save a script of your own as pandas.py in ~/dev/bootcamp. Python searches the script's folder for pandas before the installed libraries, so import pandas finds your file, and the first block of the next subtitle saved under that name fails on its own import with AttributeError: partially initialized module 'pandas' has no attribute 'Series' (most likely due to a circular import).
• DON'T take a tutorial's output as your pandas' behavior. dtype: object on a text column, or s[0] returning the first value of a Series labeled by name, is an older pandas, and pd.__version__ says which one you run.

The Series

What. A Series is one column of values with a label attached to each value. The labels together are its index, and the values share one dtype. Why. A label says what a value belongs to. median_price["Downtown"] names the neighborhood it reads, and it keeps finding Downtown after the values are sorted or filtered, which a position stops doing the moment the order changes. When. One feature of a dataset, one column of a table, or a per-category result such as a count or a median per neighborhood.

Built from a list, a Series labels its values 0 to n minus 1, a RangeIndex:

import pandas as pd

# one price per listing
prices = pd.Series([310000, 245000, 189000, 402000])
print(prices)

0    310000
1    245000
2    189000
3    402000
dtype: int64

The left column is the index, the right column holds the values, and the last line is the dtype. Built from a dict, the Series takes the keys as its labels and the values as its values:

median_price = pd.Series(
    {"Downtown": 310000, "Riverside": 245000, "Hillcrest": 189000}
)
print(median_price)
print(median_price.index)

Downtown     310000
Riverside    245000
Hillcrest    189000
dtype: int64
Index(['Downtown', 'Riverside', 'Hillcrest'], dtype='str')

An int64 or float64 Series keeps its values in a NumPy ndarray, and .to_numpy() returns them as one, so every calculation Section 2.2 ran on an array runs on the values, and only a write into that array is refused, as a DON'T below shows. The dtype follows NumPy's rules for numbers, whole numbers give int64 and a decimal anywhere gives float64. Text is where pandas 3 differs: it gives text a dtype of its own, printed as str, where pandas 2 printed object for the same Series. A str Series keeps its text in pandas' own string array, and .to_numpy() returns a NumPy array of dtype object, NumPy's dtype for arbitrary Python objects.

areas = pd.Series([82.5, 64.0, 120.3, 95.1])
condition = pd.Series(["good", "fair", "excellent", "good"])

print(prices.dtype)       # int64
print(areas.dtype)        # float64
print(condition.dtype)    # str
print(prices.to_numpy())  # [310000 245000 189000 402000]

A value comes out by label or by position. .loc[...] takes a label, .iloc[...] takes a position, counted from 0 and from the end with a negative number exactly as a list is, and plain brackets with a label do what .loc does:

print(median_price.loc["Riverside"])  # 245000
print(median_price["Downtown"])       # 310000
print(median_price.iloc[0])           # 310000
print(median_price.iloc[-1])          # 189000

A single integer in plain brackets is read as a label, never as a position. This index has no label 0, so the line raises:

print(median_price[0])  # raises KeyError: 0

Older pandas fell back to the position here, pandas 2.2 printing the first value with a FutureWarning, and pandas 3 removed the fallback. On a Series whose labels are themselves integers, the same brackets look up the label 0, returning what is labeled 0 wherever in the Series it sits, or raising KeyError: 0 when no label is 0, and Lesson 2.4.2 shows a filtered table handing you exactly such an index.

DON'T:

• DON'T write a single integer in plain brackets to mean a position. On a text index it raises KeyError: 0, and on an integer index it returns what is labeled 0 without a word, wherever it sits, or raises KeyError: 0 when no label is 0, .iloc[0] is the position.
• DON'T write into the array .to_numpy() returns. From an int64 or float64 Series it is a read-only view of the Series' own values, and arr[0] = 1 raises ValueError: assignment destination is read-only. From a str Series it is the Series' own storage, so a write into it changes the Series without a word, and when the Series is a column of a table, the table as well. .to_numpy(copy=True) gives an array you may change.
• DON'T find text columns by testing dtype == object in pandas 3. condition.dtype == object is False for a text Series, and condition.dtype == "str" is True.

Index Alignment

What. Arithmetic between two Series pairs each value with the value that carries the same label in the other Series, not with the value in the same position. Why. Two per-neighborhood results computed from two sources rarely list the neighborhoods in the same order, and pairing by label is what keeps Downtown's price divided by Downtown's area. A label present on only one side has nothing to pair with, and pandas returns a missing value for it, NaN here, NumPy's missing value from Section 2.2, with no error and no warning. When. Every time two Series meet in +, -, * or /: a ratio from two summaries, a total from two monthly counts, the change between two periods.

Here the median price and the median area per neighborhood come from two different summaries, in different orders, and each one lists a neighborhood the other lacks:

import pandas as pd

median_price = pd.Series(
    {"Downtown": 310000, "Riverside": 245000, "Hillcrest": 189000}
)
median_area = pd.Series(
    {"Riverside": 95.0, "Downtown": 70.0, "Eastgate": 110.0}
)
price_per_sqm = median_price / median_area
print(price_per_sqm)

Downtown     4428.571429
Eastgate             NaN
Hillcrest            NaN
Riverside    2578.947368
dtype: float64

Downtown's 310000 was divided by Downtown's 70.0 although Downtown sits first in one Series and second in the other. The result carries every label from either side, Hillcrest has no area and Eastgate has no price, so both are NaN. The same division on the bare arrays pairs by position instead, and returns three numbers that each divide one neighborhood's price by another neighborhood's area, with nothing to show it:

# position pairs Downtown with Riverside's area
print(median_price.to_numpy() / median_area.to_numpy())

[3263.15789474 3500.         1718.18181818]

NaN in a result is the visible half of the trap. The hidden half is what the next step does with it. Two months of new listings per neighborhood, each month missing a neighborhood the other has:

march = pd.Series({"Downtown": 12, "Riverside": 8, "Hillcrest": 5})
april = pd.Series({"Riverside": 6, "Hillcrest": 4, "Eastgate": 9})

both = march + april
print(both)
print(both.sum())

Downtown      NaN
Eastgate      NaN
Hillcrest     9.0
Riverside    14.0
dtype: float64
23.0

44 listings went in and the total says 23, because .sum() skips NaN the way np.nansum does in Section 2.2, and Downtown's 12 and Eastgate's 9 went with them. .add(other, fill_value=0) is the method form of + with a stand-in: a label missing on one side counts as 0 there before the two are added. .sub, .mul and .div take the same argument.

total = march.add(april, fill_value=0)
print(total)
print(total.sum())

Downtown     12.0
Eastgate      9.0
Hillcrest     9.0
Riverside    14.0
dtype: float64
44.0

The totals print as 12.0 because the result is float64 even though every count is whole, and Lesson 2.5.2 converts a column like this back to integers. The stand-in is right here because a neighborhood with no new listings in a month had zero of them.

DON'T:

• DON'T trust a total over two aligned Series without looking for NaN first. .sum() skips them, so (march + april).sum() printed 23.0 where 44 listings existed.
• DON'T use fill_value where the stand-in is not the truth. median_price.div(median_area, fill_value=1) divides Hillcrest's price by an invented area of 1 and reports 189000.0 as its price per square meter.
• DON'T switch to .to_numpy() to get rid of the NaN. Arrays pair by position, and the wrong neighborhoods' values combine with no NaN left to warn you.

The DataFrame

What. A DataFrame is a table of named columns that share one row index. Each column is a Series with a dtype of its own, so one table holds prices as integers, areas as floats and neighborhoods as text. Why. A listing is a row of features. Keeping every feature of a listing under one row label means that sorting or filtering moves its price, area and neighborhood together, and no column drifts out of step with the others. When. As soon as you hold more than one feature per listing, which is every dataset in the rest of this module.

A dict of lists builds one, each key a column name and each list that column's values from top to bottom:

import pandas as pd

listings = pd.DataFrame({
    "price": [310000, 245000, 189000, 402000],
    "area_sqm": [82.5, 64.0, 120.3, 95.1],
    "rooms": [3, 2, 4, 3],
    "neighborhood": ["Downtown", "Riverside", "Hillcrest", "Eastgate"],
})
print(listings)

    price  area_sqm  rooms neighborhood
0  310000      82.5      3     Downtown
1  245000      64.0      2    Riverside
2  189000     120.3      4    Hillcrest
3  402000      95.1      3     Eastgate

Lesson 1.5.3's list of dicts, one dict per row, builds the same table with pd.DataFrame(records). Four attributes describe the frame without printing it:

print(listings.columns)
print(listings.index)
print(listings.shape)
print(listings.dtypes)

Index(['price', 'area_sqm', 'rooms', 'neighborhood'], dtype='str')
RangeIndex(start=0, stop=4, step=1)
(4, 4)
price             int64
area_sqm        float64
rooms             int64
neighborhood        str
dtype: object

.columns and .index are index objects like a Series' index, holding the column names and the row labels. .shape is (rows, columns), as for a 2-D array. .dtypes is a Series of one dtype per column, and its last line, dtype: object, describes that Series itself rather than any column.

One name in brackets returns that column as a Series. A list of names in brackets returns a DataFrame of those columns, and a list holding one name returns a DataFrame with one column:

target = listings["price"]
one_column = listings[["price"]]
features = listings[["area_sqm", "rooms"]]

print(type(target))      # <class 'pandas.Series'>
print(type(one_column))  # <class 'pandas.DataFrame'>
print(target.shape)      # (4,)
print(one_column.shape)  # (4, 1)
print(features.shape)    # (4, 2)

The column keeps the frame's row labels, so target is labeled 0 to 3 exactly as listings is. A name the frame does not have raises:

print(listings["prices"])  # raises KeyError: 'prices'

DON'T:

• DON'T write df[["price"]] when you mean the column. It is a (4, 1) table, not the (4,) Series df["price"] returns, and code written for one does not behave the same on the other.
• DON'T build a DataFrame from lists of different lengths. pd.DataFrame({"price": [310000, 245000], "rooms": [3, 2, 4]}) raises ValueError: All arrays must be of the same length rather than padding the short column.

A First Look at a Dataset

What. A first look is a fixed sequence of read-only questions put to a table you did not build: how big it is, what its rows look like, what type each column has and where the gaps are, what range the numbers cover, and which categories the text columns hold. Why. A file arrives with problems that raise nothing: gaps, impossible values, numbers stored as text. Each one changes the results computed from its column, and each shows in the first minute to someone who asks. Every step is a short read of the table, and none of them changes the data. When. Right after loading any table, before computing anything from it.

Start first_look.py in ~/dev/bootcamp, beside the housing.csv you saved at the start of Theory:

import pandas as pd

housing = pd.read_csv("housing.csv")
print(housing.shape)  # (209, 7)

pd.read_csv reads a comma-separated file into a DataFrame, takes its first line as the column names and turns every empty field into NaN. Its options are Lesson 2.4.3's. The shape answers the first question, 209 listings described by 7 columns. The second question is what a row looks like, from both ends of the file:

print(housing.head())
print(housing.tail())

      price  area_sqm  rooms  year_built neighborhood condition listing_date
0  437000.0     131.8      2        1978     Lakeview      fair   2023-04-15
1  268000.0     112.1      5        1953     Downtown      good   2024-10-27
2  268000.0     102.2      6        2004    Southgate      fair   2023-06-09
3       NaN      66.6      1        2008    Hillcrest      good   2024-01-23
4  150000.0      78.1      1        1979    Hillcrest      good   2023-03-23
        price  area_sqm  rooms  year_built neighborhood condition listing_date
204  437000.0     131.8      2        1978     Lakeview      fair   2023-04-15
205  268000.0     112.1      5        1953     Downtown      good   2024-10-27
206  268000.0     102.2      6        2004    Southgate      fair   2023-06-09
207       NaN      66.6      1        2008    Hillcrest      good   2024-01-23
208  150000.0      78.1      1        1979    Hillcrest      good   2023-03-23

head() and tail() return the first and last five rows. Two things already show. Row 3 has no price, and the last five rows repeat the first five value for value, which Lesson 2.5.4's duplicate check exists for. The third question, types and gaps, is info():

housing.info()

<class 'pandas.DataFrame'>
RangeIndex: 209 entries, 0 to 208
Data columns (total 7 columns):
 #   Column        Non-Null Count  Dtype  
---  ------        --------------  -----  
 0   price         189 non-null    float64
 1   area_sqm      194 non-null    float64
 2   rooms         209 non-null    int64  
 3   year_built    209 non-null    int64  
 4   neighborhood  209 non-null    str    
 5   condition     208 non-null    str    
 6   listing_date  206 non-null    str    
dtypes: float64(2), int64(2), str(3)
memory usage: 11.6 KB

info() prints its report itself, so it is called without print. Every column short of 209 non-null values has gaps: 20 prices are missing, 15 areas, one condition and three listing dates. price is float64 although every price in the file is a whole number, because read_csv stores a column of whole numbers with a gap as floats, Lesson 2.5.2 says why. listing_date is str, dates read as text, which Lesson 2.5.3 converts. The fourth question, the range of every number, is describe():

print(housing.describe())

              price     area_sqm       rooms   year_built
count  1.890000e+02   194.000000  209.000000   209.000000
mean   3.550582e+05   123.048454    3.985646  1988.038278
std    7.906103e+05   208.729092    6.851080    21.754939
min    1.400000e+05    26.400000    1.000000  1911.000000
25%    1.970000e+05    84.225000    2.000000  1970.000000
50%    2.490000e+05   102.350000    3.000000  1990.000000
75%    3.770000e+05   124.550000    5.000000  2005.000000
max    9.900000e+06  2400.000000   99.000000  2023.000000

describe() reports, for every numeric column, the count of values present, the mean, the standard deviation, the minimum, the 25th, 50th and 75th percentiles and the maximum. The price column prints in scientific notation, where e+06 means times ten to the sixth, so its maximum is 9,900,000. Three maximums sit far from the rest of their columns, a price of 9.9 million, an area of 2400 square meters and 99 rooms, and the extreme prices pull the mean to 355,058 while the median, the 50% row, is 249,000. Handling such values is Lesson 2.5.4's. The three text columns are absent, because describe() summarizes the numeric columns only unless told otherwise. The last question, which categories a text column holds and how often, is value_counts():

print(housing["condition"].value_counts())
print(housing["condition"].value_counts(dropna=False))

condition
good         94
fair         56
excellent    34
poor         24
Name: count, dtype: int64
condition
good         94
fair         56
excellent    34
poor         24
NaN           1
Name: count, dtype: int64

The result is a Series labeled by category and sorted from the most frequent down. Its first four counts add up to 208, not 209, because value_counts() leaves missing values out unless dropna=False asks for them as a row of their own. nunique() counts the distinct values present, for one column or for every column at once:

print(housing["neighborhood"].nunique())
print(housing.nunique())

7
price           133
area_sqm        178
rooms             8
year_built       71
neighborhood      7
condition         4
listing_date    165
dtype: int64

Seven neighborhoods and four conditions hold no surprise. Eight distinct room counts is a question worth asking, and housing["rooms"].value_counts() answers it with six counts from 1 to 6 plus one listing each with 12 and 99 rooms. That is the whole first look, in this order every time: size, rows, types and gaps, ranges, categories.

DON'T:

• DON'T read describe() as a summary of the whole table. By default it reports the numeric columns only, and neighborhood, condition and listing_date do not appear in it at all.
• DON'T add up value_counts() and expect the row count. The default leaves missing values out, 208 of 209 here, and dropna=False is what shows them.
• DON'T wrap info() in print. It prints its report and returns None, and print(housing.info()) adds that None as a last line.

Lab Exercises

Lab Exercise 1: A Labeled Series and Two Weeks of Counts

With (bootcamp) active, confirm with a one-line check that pandas imports at version 3.0.5.

Then, in series_lab.py, build rooms from the dict {"Downtown": 3, "Riverside": 2, "Hillcrest": 4, "Eastgate": 5}, the median room count per neighborhood. Print its dtype, the value for Hillcrest by label, the value in the last position, and the array .to_numpy() returns. Then run rooms[1] inside a try that catches the error and prints its type and message.

Finally, two colleagues counted this week's and last week's new listings: week1 = pd.Series({"Downtown": 4, "Riverside": 3, "Eastgate": 2}) and week2 = pd.Series({"Riverside": 5, "Hillcrest": 1, "Downtown": 2}). Print week1 + week2 and its sum, then the sum of the two added with a stand-in of 0. Say in a comment which total is right, and why the other one is smaller.

Lab Exercise 2: Build a Table, Then Look at a Real One

In table_lab.py, build sample from a dict of lists holding five listings with the columns price, area_sqm, rooms and condition, values of your choice. Print its shape, its column names and its dtypes. Print the type and the shape of sample["rooms"] and of sample[["rooms"]].

Then load housing.csv and run the first look in its order, ending with the categories of condition, missing ones included, and the number of distinct values in year_built. Under each print, record in a comment what it showed: the size, which columns have gaps and how many, the largest area and room count and why neither is believable, how many listings are in poor condition, and how many construction years the file holds.

# 3.0.5
python3 -c "import pandas as pd; print(pd.__version__)"

import pandas as pd

rooms = pd.Series({"Downtown": 3, "Riverside": 2, "Hillcrest": 4, "Eastgate": 5})
print(rooms.dtype)                  # int64
print(rooms.loc["Hillcrest"])       # 4
print(rooms.iloc[-1])               # 5
print(rooms.to_numpy())             # [3 2 4 5]

try:
    print(rooms[1])                 # 1 is read as a LABEL
except KeyError as err:
    print(type(err).__name__, err)  # KeyError 1

week1 = pd.Series({"Downtown": 4, "Riverside": 3, "Eastgate": 2})
week2 = pd.Series({"Riverside": 5, "Hillcrest": 1, "Downtown": 2})

print(week1 + week2)
print((week1 + week2).sum())
print(week1.add(week2, fill_value=0).sum())

Downtown     6.0
Eastgate     NaN
Hillcrest    NaN
Riverside    8.0
dtype: float64
14.0
17.0

# 17.0 is right: 9 listings in week1 plus 8 in week2.
# 14.0 lost Eastgate's 2 and Hillcrest's 1. Each appears in one week only,
# so plain + gave them NaN and .sum() skipped them without a word.

import pandas as pd

sample = pd.DataFrame({
    "price": [310000, 245000, 189000, 402000, 268000],
    "area_sqm": [82.5, 64.0, 120.3, 95.1, 77.0],
    "rooms": [3, 2, 4, 3, 2],
    "condition": ["good", "fair", "good", "excellent", "poor"],
})
print(sample.shape)
print(list(sample.columns))
print(sample.dtypes)
print(type(sample["rooms"]), sample["rooms"].shape)
print(type(sample[["rooms"]]), sample[["rooms"]].shape)

(5, 4)
['price', 'area_sqm', 'rooms', 'condition']
price          int64
area_sqm     float64
rooms          int64
condition        str
dtype: object
<class 'pandas.Series'> (5,)
<class 'pandas.DataFrame'> (5, 1)

housing = pd.read_csv("housing.csv")
print(housing.shape)                    # (209, 7)

print(housing.head())
print(housing.tail())
# Row 3 has no price, and the last five rows repeat the first five.

housing.info()
# Gaps: price 20 (189 of 209), area_sqm 15, condition 1, listing_date 3.

print(housing.describe())
# Largest area 2400.0 square meters, largest room count 99. Neither fits
# the rest of its column, whose 75% rows are 124.55 and 5.

print(housing["condition"].value_counts(dropna=False))
# 24 listings in poor condition, and 1 with no condition at all.

print(housing["year_built"].nunique())  # 71

Theory Assessment

Q1: Two Halves of a Series, and the Bracket That Changed

Name the two parts a Series is made of, and say what .loc and .iloc each take. Then explain why median_price[0] raises KeyError: 0 in pandas 3 on the Series labeled Downtown, Riverside and Hillcrest, and what the same line does on a Series labeled [2, 1, 0].

Hint: Ask what plain brackets do with a single integer in pandas 3, and then look at whether that integer exists among the labels in each case.

Q2: The Sum That Came Up Short

march holds new listings for Downtown, Riverside and Hillcrest, and april for Riverside, Hillcrest and Eastgate. Explain which labels march + april holds, which are NaN and why, and what its .sum() leaves out. Then say when fill_value=0 is the right fix, and give a case from the lesson where a stand-in is wrong.

Hint: Pairing happens by label before any addition. For the second half, ask whether a missing label means "zero of these" or "we do not know".

Q3: One Column, Two Shapes, and What the First Look Hides

Explain what listings["price"] and listings[["price"]] each return, with their shapes for the four-listing table. Then name two calls of the first look that leave something out by default on housing.csv, what each leaves out, and which call shows it.

Hint: One of them skips whole columns, the other skips one kind of value inside a column.

Q4: An Average That Looks Fine

Read this script without running it, then say what each print shows.

import pandas as pd

housing = pd.read_csv("housing.csv")
counts = housing["condition"].value_counts()
points = pd.Series({"excellent": 4, "Good": 3, "fair": 2, "poor": 1})

weighted = counts * points
print(len(weighted))
print(weighted.sum() / counts.sum())
print(counts.sum() == len(housing))

The script was meant to compute the average condition score of the listings, 4 for excellent down to 1 for poor. Nothing raises. Say what went wrong, why nothing warned, and what the average should have been.

Hint: Compare the labels of points with the labels value_counts() produced, character by character. Then recall what .sum() does with NaN, and what value_counts() does with a missing condition.

5
1.3076923076923077
False

Assignment

Assignment: A Neighborhood Count Feature That Adds Up

A model will take one count per neighborhood, always in this order:

MODEL_NEIGHBORHOODS = ["Downtown", "Eastgate", "Hillcrest", "Lakeview",
                       "Northend", "Riverside", "Southgate", "Westbrook"]

Westbrook is a neighborhood the model expects and housing.csv has no listings for yet. The count comes from two sources: the number of listings per neighborhood in housing.csv, and this week's new listings, which a colleague typed by hand as {"riverside": 4, "Downtown": 2, "Westbrook": 3, "Eastgate": 1}.

Two more tools do the shaping. pd.DataFrame accepts a dict of Series as well as a dict of lists, and lines the Series up by label: the table gets one row for every label found in any of them, with NaN where a Series has no value for that row. series.reindex(labels, fill_value=0) returns a new Series holding exactly labels, in that order, each value looked up by label in series, 0 for a label series lacks, and nothing at all for a label of series that is not in labels.

Write count_feature.py to build feature, the total count per neighborhood in exactly the model's order, with 0 for a neighborhood that has no listings. Before trusting it, prove two things in code and print the result of each check: its length is 8, and its sum equals the number of rows in housing.csv plus the sum of the new listings. Print a side-by-side table of the existing, new and total counts, so a reader can see where every number came from. When a check fails, find the cause in the printed output, fix the input, and leave a comment saying what the failed check caught.

Hints:

• Combine the two counts with .add(..., fill_value=0) before you reindex. reindex fills a label it adds, and leaves alone a NaN that is already there.
• If the sum check fails, look in the side-by-side table for a row that has a new count and no existing one, and compare its label letter by letter with the model's list.

import pandas as pd

MODEL_NEIGHBORHOODS = ["Downtown", "Eastgate", "Hillcrest", "Lakeview",
                       "Northend", "Riverside", "Southgate", "Westbrook"]

housing = pd.read_csv("housing.csv")
existing = housing["neighborhood"].value_counts()
new = pd.Series({"riverside": 4, "Downtown": 2, "Westbrook": 3, "Eastgate": 1})

total = existing.add(new, fill_value=0)  # no NaN from alignment
feature = total.reindex(MODEL_NEIGHBORHOODS, fill_value=0)

expected = len(housing) + new.sum()      # 209 + 10
print(len(feature) == 8)
print(feature.sum(), expected)
print(pd.DataFrame({"existing": existing, "new": new, "total": total}))

True
215.0 219
           existing  new  total
Downtown       20.0  2.0   22.0
Eastgate       33.0  1.0   34.0
Hillcrest      33.0  NaN   33.0
Lakeview       30.0  NaN   30.0
Northend       28.0  NaN   28.0
Riverside      28.0  NaN   28.0
Southgate      37.0  NaN   37.0
Westbrook       NaN  3.0    3.0
riverside       NaN  4.0    4.0

# The sum check caught 4 listings that reindex dropped without a word,
# typed as "riverside" where the file says "Riverside".
new = pd.Series({"Riverside": 4, "Downtown": 2, "Westbrook": 3, "Eastgate": 1})

total = existing.add(new, fill_value=0)
feature = total.reindex(MODEL_NEIGHBORHOODS, fill_value=0)

print(len(feature) == 8)
print(feature.sum(), expected)
print(feature)

True
219.0 219
neighborhood
Downtown     22.0
Eastgate     34.0
Hillcrest    33.0
Lakeview     30.0
Northend     28.0
Riverside    32.0
Southgate    37.0
Westbrook     3.0
dtype: float64

Additional Resources

Optional, and nothing later depends on either one. Worth a look if you want to go past this lesson.

• Official docs - pandas 3.0 User Guide, "Working with text data", the "Text data types" section. Why pandas stores text as str rather than object, and what changed in version 3.0.
• YouTube - "Python Pandas Tutorial (Part 3): Indexes - How to Set, Reset, and Use Indexes" by Corey Schafer. Turning a column into the row index and back.

Summary

A Series looks like a small thing, a column with names down its side, and the names carry most of the weight. They decide what a lookup returns, what arithmetic pairs, and what survives when two sources disagree about which neighborhoods exist. NumPy taught you to trust an array's order, and pandas asks you to trust its labels instead, because a row's position can change every time you sort or filter the table while its label stays with it. That trade is why an analysis can reshuffle a table a hundred times and still divide Downtown's price by Downtown's area.

The same design produces the silent failures this lesson kept running into, and none of them raised an error. A label on one side only becomes a missing value, a sum skips it, a misspelled category becomes a label of its own, and a reindex drops whatever is not on its list. Each step did exactly what it was built to do. What protects you is not a cleverer call but a habit: compute the number you expect some independent way, and compare.

The first look is that habit applied to a whole file. Five questions that change nothing, and on the course's own file they found twenty missing prices, a listing priced at 9.9 million, a house with 99 rooms, five repeated rows and a missing condition the counts left out. The next three sections are the work those findings ask for, and you can now see that work before you start it.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import pandas as pd

s = pd.Series({"Downtown": 310000, "Riverside": 245000})
s.loc["Riverside"]              # by label
s.iloc[0]                       # by position
# s[0] raises KeyError: 0 on a text index

a = pd.Series({"Downtown": 12, "Riverside": 8})
b = pd.Series({"Riverside": 6, "Eastgate": 9})
a + b                           # Downtown and Eastgate NaN
a.add(b, fill_value=0)          # missing label counts as 0

df = pd.DataFrame({"price": [310000, 245000], "rooms": [3, 2]})
df["price"]                     # Series, shape (2,)
df[["price"]]                   # DataFrame, shape (2, 1)

housing = pd.read_csv("housing.csv")
housing.shape                   # size
housing.head(), housing.tail()  # rows
housing.info()                  # types and gaps, prints itself
housing.describe()              # numeric ranges only
housing["condition"].value_counts(dropna=False)
housing.nunique()               # distinct values per column