Missing Data

Context

A missing value is a cell the source never filled. pandas writes a marker into that cell, and the marker is what every method recognizes when it counts the gaps, drops the rows that hold them, or fills them with a value you choose.

Most real tables you will train a model on have them. A listings export arrives with no floor area for some homes and no price for others, a survey arrives with skipped answers, a sensor log arrives with readings that never came. Before a model sees housing.csv, each of its gaps has to be found and decided, and the decision is a modeling one: dropping a row costs training data, and filling a cell invents a measurement the model will treat as real.

Without the right tools that decision gets made by accident. A check written with == reports no missing prices in a column that has 20, a reflexive drop throws away almost a fifth of the listings, many of them over gaps in columns the model may never use, and a fill with the average writes a number dragged upward by two multi-million listings into every hole. Lesson 2.4.1's info() showed which columns have gaps, and this lesson decides what happens to each one.

Goal

• Identify the marker pandas 3.0.5 writes into each kind of column, NaN, <NA> and NaT, and explain why == np.nan finds none of them.
• Count gaps per column, as a share of each column and per row, with isna, notna and any(axis=1).
• Drop rows or columns with dropna, scoped by subset, how, thresh and axis, and report the rows every drop costs.
• Fill gaps with fillna from one value, a per-column dict or ffill, choosing the median over the mean when outliers pull the mean.
• Record every filled cell in a was-missing indicator column before the fill erases the evidence.

Prerequisites

Optional, and the lesson does not depend on them. They give you the picture first, if you want it.

• YouTube - "How do I handle missing values in pandas?" by Data School. How pandas represents a missing value, how to find one, and the options for dropping or filling it, on a different dataset.
• Official docs - pandas, "Working with missing data", "Values considered missing". Which marker pandas uses for a gap in each kind of column.

Table of Contents

• How pandas Marks Missing Values
• Detecting Missingness
• Dropping Missing Data
• Filling Missing Values

Theory

Every block below is a script saved in ~/dev/bootcamp and run there with python3. Each one builds or loads everything it uses, reading housing.csv from that folder, where Lesson 2.4.1 had you save it, so any block runs on its own.

How pandas Marks Missing Values

What. A missing value is a cell the source left empty, and pandas stores a marker in it, the value its missing-data methods look for: NaN in a numeric column and in pandas 3's default text column, <NA> in the nullable dtypes, and NaT in a date column. Why. One marker per kind of column is what lets mean skip a gap and a detector find it, and the float marker NaN follows a rule that catches everyone once: it is equal to nothing, not even itself. When. Whenever a file with an empty field is read, because read_csv writes the marker into every empty cell, which is how the blank prices in housing.csv arrived as gaps.

NaN is the float nan of Lesson 2.2.3, the value np.sqrt(-4.0) returned, and pandas uses it as the gap in every float64 column. Rows 3 and 6 of housing.csv each hold one.

import pandas as pd

df = pd.read_csv("housing.csv")
print(df.head(7))
print(df.loc[6, "listing_date"])  # the text gap, pulled out
print(type(df.loc[6, "listing_date"]))

      price  area_sqm  rooms  year_built neighborhood condition listing_date
0  437000.0     131.8      2        1978     Lakeview      fair   2023-04-15
1  268000.0     112.1      5        1953     Downtown      good   2024-10-27
2  268000.0     102.2      6        2004    Southgate      fair   2023-06-09
3       NaN      66.6      1        2008    Hillcrest      good   2024-01-23
4  150000.0      78.1      1        1979    Hillcrest      good   2023-03-23
5  203000.0     121.7      2        1984    Southgate      good   2024-11-30
6  281000.0     105.6      6        2021    Riverside      fair          NaN
nan
<class 'float'>

Row 3's missing price and row 6's missing date print the same NaN, though price is float64 and listing_date is the str text dtype Lesson 2.4.1 printed. pandas 3.0.5 stores a gap in its default text column as that same float nan, which is why the cell prints nan and its type is float. Every price carries a .0 because the column has gaps, as Lesson 2.4.1 noted, and Lesson 2.5.2 explains why.

The rule that makes NaN slippery is Lesson 2.2.3's: == with NaN on one side is False against any number, another NaN included. So a test written with == cannot find a gap, and it fails without a word.

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv")
print(np.nan == np.nan)               # False
print((df["price"] == np.nan).sum())  # 0
print((df["price"] == None).sum())    # 0

Both comparisons are False on every one of the 209 rows, so both count zero matches in a column whose info() line in Lesson 2.4.1 read 189 non-null. The column is not clean, the test is wrong.

A Python None placed in a list becomes the column's marker as well, once pandas builds a numeric or a text column from that list.

import pandas as pd

rooms = pd.Series([3, None, 5])  # a count with one gap
condition = pd.Series(["good", None, "fair"])
print(rooms)
print(condition)

0    3.0
1    NaN
2    5.0
dtype: float64
0    good
1     NaN
2    fair
dtype: str

The nullable dtypes, which Lesson 2.5.2 covers as types, use a second marker, pd.NA, printed <NA>, and a comparison with it returns <NA> instead of True or False. Dates use a third, NaT, short for "not a time".

import pandas as pd

# 'Int64' with a capital I is a nullable dtype
rooms = pd.Series([3, None, 5], dtype="Int64")
print(rooms)
print(pd.NA == pd.NA)
# 'pd.to_datetime' turns text into dates, Lesson 2.5.3
dates = pd.to_datetime(pd.Series(["2024-01-23", ""]))
print(dates[1])

0       3
1    <NA>
2       5
dtype: Int64
<NA>
NaT

Int64 kept its other values whole, 3 and 5 rather than 3.0 and 5.0, because <NA> needs no float to live in. Lesson 2.5.3 meets NaT again when it parses listing_date into real dates.

DON'T:

• DON'T test for a gap with == np.nan or == None. Both are False on every row of price, so on housing.csv they report 0 missing prices where there are 20.
• DON'T put a comparison with pd.NA in an if. It returns <NA>, and if pd.NA == pd.NA: raises TypeError: boolean value of NA is ambiguous.

Detecting Missingness

What. isna() returns a frame of the same shape holding True wherever a value is missing, whichever marker it carries, and notna() returns the opposite. Why. True counts as 1 and False as 0, so .sum() over the mask counts the gaps in each column and .mean() gives each column's share, the two numbers that decide between dropping and filling. When. Straight after loading, and again after every drop or fill, to prove the step did what you meant.

import pandas as pd

df = pd.read_csv("housing.csv")
print(df.isna().sum())            # gaps per column
print(df.isna().mean().round(3))  # share of each column

price           20
area_sqm        15
rooms            0
year_built       0
neighborhood     0
condition        1
listing_date     3
dtype: int64
price           0.096
area_sqm        0.072
rooms           0.000
year_built      0.000
neighborhood    0.000
condition       0.005
listing_date    0.014
dtype: float64

The counts are Lesson 2.4.1's info() read from the other side: price printed 189 non-null out of 209 entries, and 209 minus 189 is the 20 here. The shares put every column on one scale, nearly a tenth of the prices against half a percent of the conditions.

A count per column says nothing about how the gaps fall across rows, so the row view comes next. Lesson 2.2.5's rule holds for a frame too: the axis you name is the one that disappears, so any(axis=1) collapses the columns and leaves one True or False per listing.

import pandas as pd

df = pd.read_csv("housing.csv")
row_has_gap = df.isna().any(axis=1)  # one value per listing
print(row_has_gap.sum())             # 39
print(df.notna().all(axis=1).sum())  # 170
print(df.isna().sum(axis=1).max())   # 1

39 listings hold a gap and 170 are complete, which together make all 209. The last line counts the gaps in each row and takes the largest, and 1 says that no listing in the file is missing more than one value, a fact the next subtitle leans on. notna() is also the mask Lesson 2.4.2 filtered with, to keep only the rows that have a value.

DON'T:

• DON'T call df.isna().any() to find rows. Without axis=1 it collapses the rows and returns one True or False per column.
• DON'T round a share to two places. The single gap in condition is 0.0048 of the column and prints 0.00 under round(2), which reads as no gaps at all.

Dropping Missing Data

What. dropna() returns a new frame without the rows that hold a gap anywhere. subset= checks only the columns you name, how="all" drops a row only when every value in it is missing, thresh=n keeps a row that still has at least n present values, and axis=1 drops columns instead of rows. Why. Some gaps cannot be filled honestly. price is the target, the column a model learns to predict, and a listing without one has nothing to teach a price model, so the row goes. When. When the gap sits in a column the work cannot do without, the target above all, or when the rows lost are few, with the row count printed before and after every drop.

import pandas as pd

df = pd.read_csv("housing.csv")
print(len(df))                           # 209
print(len(df.dropna()))                  # 170
print(len(df.dropna(subset=["price"])))  # 189

The bare dropna() cost 39 listings, almost a fifth of the file, and 19 of them had a perfectly good price, lost to a missing area, condition or date that a price model could fill or never use. subset=["price"] removed exactly the 20 listings without a target and kept every other row.

import pandas as pd

df = pd.read_csv("housing.csv")
print(len(df.dropna(how="all")))  # 209
print(len(df.dropna(thresh=6)))   # 209
print(len(df.dropna(thresh=7)))   # 170
# ['rooms', 'year_built', 'neighborhood']
print(df.dropna(axis=1).columns.tolist())

how="all" dropped nothing, because no listing is empty from end to end. thresh=6 kept all 209, because it counts the values a row must still have, and the previous subtitle showed that no row is missing more than one of its seven. thresh=7, all seven present, is the bare dropna() written out. axis=1 kept the three columns without a single gap and threw away price, the target itself, to clear 20 cells.

dropna returns a new frame and leaves df as it was, so its result is assigned, and the listings it removes take their index labels with them.

import pandas as pd

df = pd.read_csv("housing.csv")
before = len(df)
df = df.dropna(subset=["price"])  # a new frame, assigned
print(before, len(df))            # 209 189
print(df.index[:5].tolist())      # [0, 1, 2, 4, 5]

Label 3 is gone, so after a drop the labels and the positions part ways, the split Lesson 2.4.2 showed between loc and iloc, and Lesson 2.5.4 renumbers the rows once the cleaning is done.

DON'T:

• DON'T run a bare dropna() as a reflex. On housing.csv it costs 39 listings, 19 of them with a price, to clear gaps in columns the work may never touch.
• DON'T read thresh=6 as "six may be missing". It is the number of present values a row must keep, so here it keeps every listing.
• DON'T drop columns with axis=1 to tidy a frame. On housing.csv it removes price itself to get rid of 20 gaps.

Filling Missing Values

What. fillna(value) returns a copy with every gap replaced by value, a dict fills each column it names with that column's own value, and ffill() copies the last present value down into the gaps below it. Why. A fill keeps the row and invents the value, so the value chosen decides how far the column bends, and the mean bends toward outliers where the median does not. When. For a feature, an input column a model learns from, that is worth keeping and possible to estimate, never for the target, and ffill only where the row order means something, such as months in sequence.

housing.csv carries two injected listings priced at 5,200,000 and 9,900,000, and they show what an outlier does to each candidate fill. The underscores in 1_000_000 only group the digits, and Python reads it as 1000000.

import pandas as pd

df = pd.read_csv("housing.csv")
print(round(df["price"].mean()))             # 355058
print(df["price"].median())                  # 249000.0
huge = df["price"] > 1_000_000
print(df.loc[huge, "price"].tolist())        # [5200000.0, 9900000.0]
print(round(df.loc[~huge, "price"].mean()))  # 278107
print(df.loc[~huge, "price"].median())       # 247000.0

Two listings out of 189 move the mean by 76,951 and the median by 2,000. The median depends only on which value sits in the middle once the prices are sorted, so the two extremes count as two values above it whatever their size, while the mean adds their full size into the total. A mean fill would write 355,058 into every gap, a price above 135 of the 189 known ones.

price is the target, so its gaps are dropped, and the fill goes to a feature. area_sqm carries the same pull from two injected areas of 1,850 and 2,400 square meters, so its gaps take the median, and an indicator column records which cells were invented before the fill erases them.

import pandas as pd

df = pd.read_csv("housing.csv")
df = df.dropna(subset=["price"])
area_median = df["area_sqm"].median()
print(round(df["area_sqm"].mean(), 2), area_median)
# the indicator first, while isna can still see the gaps
df["area_was_missing"] = df["area_sqm"].isna()
# a dict fills each column it names with its own value
df = df.fillna({"area_sqm": area_median, "condition": "unknown"})
filled = df.loc[df["area_was_missing"], ["area_sqm", "area_was_missing"]]
print(filled.head(3))
print(df.isna().sum())

126.53 102.8
    area_sqm  area_was_missing
22     102.8              True
29     102.8              True
37     102.8              True
price               0
area_sqm            0
rooms               0
year_built          0
neighborhood        0
condition           0
listing_date        3
area_was_missing    0
dtype: int64

All 15 missing areas now read 102.8, each flagged True in area_was_missing, so a model trained later can learn whether an invented area behaves differently from a measured one, a fact the filled column alone has lost. The gap in condition became a label of its own, "unknown", and listing_date keeps its 3 gaps for Lesson 2.5.3, which parses that column into dates. Which rows that median may be computed from, once a model's data is split into training and test rows, is the subject of Lesson 3.2.3.

ffill() belongs to records whose order carries meaning, where the last known value is the best guess for the next one.

import pandas as pd

# one neighborhood's median sale price per month, three months unreported
monthly = pd.Series([None, 248000.0, None, 252000.0, None],
                    index=["2024-01", "2024-02", "2024-03", "2024-04", "2024-05"])
print(monthly.ffill())

2024-01         NaN
2024-02    248000.0
2024-03    248000.0
2024-04    252000.0
2024-05    252000.0
dtype: float64

March took February's value and May took April's, while January stayed empty, because no value sits above it to copy. bfill() fills the other way, from the next present value upward. On the rows of housing.csv, whose order means nothing, either one would copy an unrelated listing's value into the gap.

DON'T:

• DON'T fill with the mean when outliers pull it. On the prices of housing.csv the mean, 355,058, sits above 135 of the 189 known prices.
• DON'T call fillna and drop the result. df["area_sqm"].fillna(area_median) on its own leaves all 15 gaps, and inplace=True on a single column changes nothing either, it only prints a ChainedAssignmentError warning, Lesson 2.4.2's chained assignment in another form.
• DON'T build the indicator after the fill. By then isna() finds nothing and the column holds no True at all.
• DON'T copy fillna(method="ffill") from an older tutorial. pandas 3.0.5 raises TypeError: NDFrame.fillna() got an unexpected keyword argument 'method', and ffill() is the method that does it now.

Lab Exercises

Lab Exercise 1: Count the Gaps Two Ways

Build a DataFrame of five listings in code with three columns: price holding 310000, None, 275000, 248000 and None, rooms holding 3, 4, 2, None and 5, and condition holding "good", "fair", None, "good" and "poor". Print the frame and its dtypes. Then print how many prices == np.nan finds and how many isna() finds, and add a comment saying why the two numbers differ.

Then load housing.csv and print one small report holding, for every column, the number of missing values and the share missing rounded to three places. Finish by printing how many listings hold at least one gap and how many hold none.

Lab Exercise 2: Drop the Rows Without a Target, Fill the Rest

Load housing.csv and print how many rows each of three drops keeps, and how many it costs: dropna(), dropna(subset=["price"]) and dropna(subset=["price", "listing_date"]).

Then build a frame a price model could train on. Drop the listings without a price, printing the row count before and after. On what remains, add the indicator columns area_was_missing and condition_was_missing, then fill area_sqm with its median and condition with the label "unknown" in one fillna call. Print the gaps left in every column and how many rows each indicator marks.

Finally, a lender publishes its mortgage rate once a month. Build rate = pd.Series([6.9, None, 7.1, None], index=["2024-01", "2024-02", "2024-03", "2024-04"]), fill it forward and print the result.

import numpy as np
import pandas as pd

listings = pd.DataFrame({
    "price": [310000, None, 275000, 248000, None],
    "rooms": [3, 4, 2, None, 5],
    "condition": ["good", "fair", None, "good", "poor"],
})
print(listings)
print(listings.dtypes)
print((listings["price"] == np.nan).sum())
print(listings["price"].isna().sum())
# '== np.nan' is False on every row, gaps included, because NaN
# equals nothing, so it counts 0 where isna counts the 2 real gaps.

      price  rooms condition
0  310000.0    3.0      good
1       NaN    4.0      fair
2  275000.0    2.0       NaN
3  248000.0    NaN      good
4       NaN    5.0      poor
price        float64
rooms        float64
condition        str
dtype: object
0
2

import pandas as pd

df = pd.read_csv("housing.csv")
report = pd.DataFrame({
    "missing": df.isna().sum(),
    "share": df.isna().mean().round(3),
})
print(report)
print(df.isna().any(axis=1).sum())
print(df.notna().all(axis=1).sum())

              missing  share
price              20  0.096
area_sqm           15  0.072
rooms               0  0.000
year_built          0  0.000
neighborhood        0  0.000
condition           1  0.005
listing_date        3  0.014
39
170

import pandas as pd

df = pd.read_csv("housing.csv")
drops = {
    "any gap": df.dropna(),
    "price": df.dropna(subset=["price"]),
    "price or date": df.dropna(subset=["price", "listing_date"]),
}
for name, kept in drops.items():
    print(name, len(kept), len(df) - len(kept))

any gap 170 39
price 189 20
price or date 186 23

import pandas as pd

df = pd.read_csv("housing.csv")
before = len(df)
df = df.dropna(subset=["price"])
print(before, len(df))
df["area_was_missing"] = df["area_sqm"].isna()
df["condition_was_missing"] = df["condition"].isna()
fills = {"area_sqm": df["area_sqm"].median(), "condition": "unknown"}
df = df.fillna(fills)
print(df.isna().sum())
print(df["area_was_missing"].sum(), df["condition_was_missing"].sum())

209 189
price                    0
area_sqm                 0
rooms                    0
year_built               0
neighborhood             0
condition                0
listing_date             3
area_was_missing         0
condition_was_missing    0
dtype: int64
15 1

import pandas as pd

rate = pd.Series([6.9, None, 7.1, None],
                 index=["2024-01", "2024-02", "2024-03", "2024-04"])
print(rate.ffill())

2024-01    6.9
2024-02    6.9
2024-03    7.1
2024-04    7.1
dtype: float64

Theory Assessment

Q1: The Check That Found No Gaps

A teammate audits housing.csv with (df["price"] == np.nan).sum(), gets 0, and reports that every listing has a price. Say what that line compared on each row, why the answer is 0, and what to write instead. Then name the marker a gap carries in a float64 column, in pandas 3's default str column, and in an Int64 column.

Hint: Start from what np.nan == np.nan returns. For the second half, print one gap from each kind of column and look at what comes out.

Q2: Twenty Gaps, Thirty-Nine Rows

On housing.csv, df.isna().sum() reports 20 missing prices, yet df.dropna() removes 39 listings. Explain where the other 19 come from, what df.isna().any(axis=1) computes and how its sum relates to the 39, and which dropna call removes exactly the listings a price model cannot use, and why it cannot use them.

Hint: A bare dropna looks at every column of a row. For the last part, ask which column holds the value the model learns to predict.

Q3: What Goes Into the Gap

You are filling the 15 gaps in area_sqm. Explain why the median is a safer fill than the mean for this column, what an area_was_missing column records that the filled column cannot, and why it has to be built before the fill.

Hint: Look for the largest areas in the file. For the second part, ask how a filled 102.8 differs from a measured 102.8 once both sit in the same column.

Q4: Three Lines That Did Not Do What They Say

Read this script without running it and state the three numbers it prints.

import pandas as pd

df = pd.read_csv("housing.csv")
df["area_sqm"].fillna(df["area_sqm"].median())
print(df["area_sqm"].isna().sum())
print(len(df.dropna(thresh=6)))
df = df.fillna({"area_sqm": df["area_sqm"].median()})
df["area_was_missing"] = df["area_sqm"].isna()
print(df["area_was_missing"].sum())

Its author expected 0, then a number well below 209, then 15. Explain each of the three gaps between what was expected and what prints.

Hint: For each printed line, ask one question. Did the result of the line before it land anywhere? What does the number in thresh count? What was in the column at the moment isna ran?

Assignment

Assignment: The Export That Hid Its Gaps

An upstream script that exports listings for your team wrote the number 0 into every floor area it did not know, and nobody wrote that down. Recreate its output with this block first, which reads housing.csv and saves the export beside it with Lesson 2.4.3's to_csv:

import pandas as pd

listings = pd.read_csv("housing.csv")
# the upstream export wrote 0 wherever an area was unknown
export = listings.fillna({"area_sqm": 0})
export.to_csv("housing_export.csv", index=False)

Working from housing_export.csv, do the following. Print the missing count of every column and the number of listings holding any gap, and add a comment saying what that report gets wrong about area_sqm. Count the areas equal to 0 and print the median and the mean of area_sqm as the file stands. Turn every 0 in area_sqm back into a real gap with Series.replace, which returns a copy of a column with every value equal to its first argument swapped for its second, so s.replace(0, np.nan) turns each 0 into NaN. Print the missing count of area_sqm and its median and mean again.

Then finish the job the lesson did on a clean file: drop the listings without a price with the row count printed before and after, add area_was_missing, fill area_sqm with its median and condition with "unknown", and print the gaps left in every column. End with a comment saying what a model trained on the export as delivered would have been told about those listings.

Hints:

• A placeholder such as 0 is a present value as far as pandas can tell, so no isna call will ever see it, and the only test that finds it is one that asks about the value itself.
• housing.csv has no real listing with an area anywhere near 0, so every 0 in the export is a former gap, and you can confirm that by printing the smallest area in housing.csv.

import pandas as pd

df = pd.read_csv("housing_export.csv")
print(df.isna().sum())
print(df.isna().any(axis=1).sum())
# The report shows no gap in 'area_sqm', yet 15 areas are unknown.
# A 0 is a present value to pandas, so isna cannot see it.

price           20
area_sqm         0
rooms            0
year_built       0
neighborhood     0
condition        1
listing_date     3
dtype: int64
24

import numpy as np
import pandas as pd

df = pd.read_csv("housing_export.csv")
# 26.4, the smallest real area
print(pd.read_csv("housing.csv")["area_sqm"].min())
print((df["area_sqm"] == 0).sum())      # 15
print(df["area_sqm"].median())          # 100.0
print(round(df["area_sqm"].mean(), 2))  # 114.22
df["area_sqm"] = df["area_sqm"].replace(0, np.nan)
print(df["area_sqm"].isna().sum())      # 15
print(df["area_sqm"].median())          # 102.35
print(round(df["area_sqm"].mean(), 2))  # 123.05

import numpy as np
import pandas as pd

df = pd.read_csv("housing_export.csv")
df["area_sqm"] = df["area_sqm"].replace(0, np.nan)
before = len(df)
df = df.dropna(subset=["price"])
print(before, len(df))
df["area_was_missing"] = df["area_sqm"].isna()
fills = {"area_sqm": df["area_sqm"].median(), "condition": "unknown"}
df = df.fillna(fills)
print(df.isna().sum())
print(df["area_was_missing"].sum())
# Trained on the export as delivered, a model would have been told that
# 15 listings have a floor area of 0 square meters, a value no real
# listing has, and it would have learned prices against those zeros.

209 189
price               0
area_sqm            0
rooms               0
year_built          0
neighborhood        0
condition           0
listing_date        3
area_was_missing    0
dtype: int64
15

Additional Resources

Optional, and nothing later depends on them. Worth opening if you want to see where filling goes once a model is involved.

• Official docs - scikit-learn, "Imputation of missing values", "Marking imputed values". The was-missing indicator as the library Module 3 installs builds it, including a placeholder other than NaN marked as missing.
• Official docs - pandas, Series.interpolate. Filling an ordered gap along the line between its neighbors instead of copying one of them, with the limit options that stop a fill from spanning a long run of gaps.

Summary

A gap in a table is not one problem but a small chain of decisions, and the order of that chain is what this lesson was really about. You look before you act: the marker first, because a test that cannot see NaN reports a clean column with complete confidence, then the counts and the shares, then the rows. Only after that does anything get removed or invented, and each step leaves a number behind, the rows a drop cost and the cells a fill touched, so the next person can see what was done to the data and why.

Dropping and filling looked like two tools and turned out to be two answers to one question: what is this value worth to the model? A missing price is the model's answer itself, and an answer cannot be invented without teaching the model your guess, so those rows go. A missing area is one input among several, worth repairing, and the repair is a claim you make on the data's behalf. The median makes a modest claim, the middle of what was measured, while the mean lets the two most extreme listings in the file speak for every gap. The indicator column is the honest footnote to that claim, the one place the data still remembers which values were real.

The sharpest lesson came from the cases that raised nothing. A comparison that found zero gaps, a fill whose result went nowhere, a threshold that removed no row, an export whose zeros passed for floor areas: each ran cleanly and was wrong. Cleaning code fails quietly far more often than it fails loudly, which is why the steps here printed a count before and after. The next lesson turns to values that are present but of the wrong kind, where the same habit of counting what a conversion changed is what separates a clean column from a damaged one.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv")
# first, a placeholder becomes a gap
df["area_sqm"] = df["area_sqm"].replace(0, np.nan)
df.isna().sum()                   # gaps per column
df.isna().mean().round(3)         # share per column
df.isna().any(axis=1).sum()       # rows holding a gap

df = df.dropna(subset=["price"])  # the target is never filled
med = df["area_sqm"].median()     # median, not mean
df["area_was_missing"] = df["area_sqm"].isna()
df = df.fillna({"area_sqm": med, "condition": "unknown"})

monthly = pd.Series([None, 248000.0, None])
monthly.ffill()                   # ordered records only