Data Types and Conversion

Context

Every column has a dtype, the one storage type all of its values share, and pandas picks it when it reads the file. Conversion changes it in three ways: a cast to a type you name, a parse that turns text into numbers, and a downcast that moves a numeric column into a smaller type so it takes less memory.

A price scraped from a listings site arrives as text with a dollar sign and commas in it, a room count with one gap arrives as decimals, and a table of millions of rows has to fit in memory before anything can train on it. Every feature a Module 3 model receives is stored as a number, and the dtype is where that gets checked first.

Skip the check and a price column stored as text adds up by gluing its digits together and finds its maximum by comparing characters, and neither step raises. Force the conversion with the wrong option and every value it cannot parse becomes a gap without a word, so a scraped $310,000 drops out of the training data. Narrow a numeric column by hand into a type that is too small and 130 becomes -126. Lesson 2.5.1 dealt with values that are absent, and this lesson deals with values that are present but stored as the wrong kind.

Goal

• Read a frame's dtypes and name what each pandas 3.0.5 dtype holds, int64, float64, bool, str, object and the nullable Int64, Float64 and boolean.
• Cast a column with astype, predict when the cast raises, and carry a gap through the cast with "Int64".
• Parse a text column into numbers with pd.to_numeric, read the value and the position its ValueError names, and count what errors="coerce" turned into gaps.
• Measure a frame's real memory with memory_usage(deep=True) and downcast a numeric column to the smallest type that holds every value.

Prerequisites

None of this is required, and the lesson works without it. It is here if you would like a first look before you start.

• YouTube - "How do I change the data type of a pandas Series?" by Data School. Two ways to change a column's dtype, and what goes wrong when numbers are stored as text, on a different dataset.
• Official docs - pandas, "Essential basic functionality", "dtypes". The table of the dtypes pandas adds to NumPy's, with the name each one takes in astype.

Table of Contents

• pandas dtypes
• astype
• to_numeric
• Memory and Downcasting

Theory

Every block below is a script saved in ~/dev/bootcamp and run there with python3, and each one builds or loads everything it uses, so any block runs on its own, except a block that says it continues the one above.

pandas dtypes

What. A dtype is the one storage type that every value in a column shares. pandas 3.0.5 reads whole numbers as int64, decimals as float64, True and False as bool, and text as str, even in a CSV column whose other values are numbers, while a column built from Python values that mix text with numbers, or that hold other Python objects such as lists, is object, one Python object per value. It also offers nullable dtypes, Int64, Float64 and boolean, that hold a gap as pd.NA. Why. The dtype decides what every operation does to a column. The same sum() adds up a float64 column and glues the values of a str column end to end, so a number stored as text gives a wrong answer without raising anything. When. First thing after loading, with df.dtypes, before any statistic is computed, and again after every conversion to confirm it took.

import pandas as pd

listings = pd.DataFrame({
    "rooms": [3, 4, 2],
    "area_sqm": [131.8, 112.1, 102.2],
    "has_garage": [True, False, True],
    "neighborhood": ["Lakeview", "Downtown", "Southgate"],
    "notes": [1, "two", 3.5],  # a number, a word, a decimal
})
print(listings.dtypes)

rooms             int64
area_sqm        float64
has_garage         bool
neighborhood        str
notes            object
dtype: object

str is the pandas 3 text dtype Lesson 2.4.1 printed for housing.csv. object stores every value as a separate Python object of any type, which is why pandas falls back on it for a column that mixes an integer, a word and a decimal. pandas 2 read text columns as object, so tutorials written for it show them that way, while in pandas 3.0.5 an object column holds values that are not all text, such as a mix of kinds or lists. The closing dtype: object is the listing's own, as Lesson 2.4.1 showed.

A number stored as text answers arithmetic with text rules, and sum, min and max answer without refusing.

import pandas as pd

# three prices stored as text
prices = pd.Series(["99000", "310000", "275000"])
print(prices.dtype)  # str
print(prices.sum())  # 99000310000275000
print(prices.max())  # 99000

sum() glued the three strings together, and max() compared them character by character, where "9" beats "3" and "2", so the smallest price came out as the largest. prices.mean() raises the TypeError Lesson 2.4.3 showed, and median() and std() raise the same way, which makes them the lucky cases.

A column of whole numbers is int64, and the same column with one gap is not.

import pandas as pd

print(pd.Series([3, 4, 5]).dtype)     # int64
print(pd.Series([3, None, 5]).dtype)  # float64

int64 spends all 64 bits of a value on the number, and every one of its 2**64 bit patterns is some whole number, so none is left over to mean "missing". float64 sets patterns aside for nan, so pandas stores the whole column as floats to hold the one gap. That is why price in housing.csv is float64 although every price in it is a whole number of dollars.

The nullable dtypes are pandas' answer to that. Int64, Float64 and boolean hold a gap as pd.NA, printed <NA> as Lesson 2.5.1 showed, so whole numbers stay whole and True and False stay what they are.

import pandas as pd

rooms = pd.Series([3, None, 5], dtype="Int64")
area = pd.Series([131.8, None], dtype="Float64")
garage = pd.Series([True, None, False], dtype="boolean")
# Int64 Float64 boolean
print(rooms.dtype, area.dtype, garage.dtype)
print(pd.Series([True, None, False]).dtype)  # object

Without "boolean", the same True and False list with a gap falls back to object and keeps Python's None as it is. The category dtype and the date dtypes belong to Lesson 2.5.3.

DON'T:

• DON'T compute a statistic on a column whose dtype you have not read. On a str column sum() glues the text and max() compares it, and neither raises.
• DON'T look for text columns under object, as tutorials written for pandas 2 do. pandas 3.0.5 reads text as str, and an object column holds values that are not all text.
• DON'T fill a whole frame with one number. df.fillna(0) on housing.csv writes the integer 0 into condition and listing_date as well, and both turn from str into object.

astype

What. s.astype(dtype) returns a new column with every value converted to the dtype you name, such as float, int, str or "Int64", and leaves s as it was. Why. When you know what a column should be and every value can make the trip, astype says so in one explicit line, and it raises on a value it has no way to convert, such as a gap headed for int, though a number outside a narrow integer type's range wraps instead, as Memory and Downcasting shows, and a label left out of a category dtype's fixed list of labels becomes a missing value, as Lesson 2.5.3 shows. When. For a clean column with a known target type: a count to float for arithmetic, a column of whole-number floats to "Int64", a number to text for an identifier.

import pandas as pd

df = pd.read_csv("housing.csv")
rooms_f = df["rooms"].astype(float)
print(rooms_f.dtype)             # float64
print(rooms_f.head(3).tolist())  # [2.0, 5.0, 6.0]
cut = pd.Series([1.9, 2.5, -1.7]).astype(int)
print(cut.tolist())              # [1, 2, -1]

A float cast to int loses its fraction toward zero, the same cut Lesson 2.2.1's arrays made, so astype(int) is never a way to round.

price holds 20 gaps, and a cast to int has nowhere to put them.

import pandas as pd

df = pd.read_csv("housing.csv")
try:
    df["price"].astype(int)
except ValueError as err:
    print(type(err).__name__)
    print(err)

IntCastingNaNError
Cannot convert non-finite values (NA or inf) to integer.Replace or remove non-finite values or cast to an integer typethat supports these values (e.g. 'Int64')

IntCastingNaNError is pandas' own kind of ValueError, which is why except ValueError caught it, and the missing spaces in "integer.Replace" and "typethat" are in pandas' message as printed. Without the try, the script stops on a traceback that runs through pandas' own files and ends on the same class and message. The message names both ways out: remove the gaps first, or cast to an integer type that holds them, "Int64".

import pandas as pd

df = pd.read_csv("housing.csv")
df["price"].astype("Int64")  # converted, then thrown away
print(df["price"].dtype)
df["price"] = df["price"].astype("Int64")
print(df["price"].head(4))

float64
0    437000
1    268000
2    268000
3      <NA>
Name: price, dtype: Int64

The first cast built a new column and nothing kept it, so df["price"] was still float64. The assigned cast stored whole dollars with every gap kept as <NA>.

What a cast to text does with a gap is worth a run of its own.

import pandas as pd

df = pd.read_csv("housing.csv")
price_text = df["price"].astype(str)
print(price_text.head(4))
print(price_text.isna().sum())

0    437000.0
1    268000.0
2    268000.0
3         NaN
Name: price, dtype: str
20

In pandas 3.0.5 the gap stays a gap through astype(str), still counted by isna, and does not turn into the three letters nan. Every number became its float spelling, 437000.0 with its .0, which is the text an identifier or a join key would carry if it were cast from a float column.

DON'T:

• DON'T astype(int) a column that has gaps. On a float column such as price it raises IntCastingNaNError, and on a text column a plain ValueError, so drop or fill the gaps first, or cast to "Int64".
• DON'T cast without assigning the result back. df["price"].astype("Int64") on its own leaves the column float64.
• DON'T turn text into True and False with astype(bool). Every non-empty string becomes True, the string "False" included, a gap becomes True as well, and only the empty string becomes False.

to_numeric

What. pd.to_numeric(s) parses every value of a text column into a number and returns a numeric column, raising ValueError on the first value it cannot parse, and errors="coerce" turns each such value into NaN instead. Why. A numeric column exported with a currency sign, thousands separators or a note reads as str, and to_numeric is the conversion built for it: its error names the value and its position, where astype(float) names the value alone. When. Whenever a column that should hold numbers arrives as str, which dtypes shows right after loading.

Save this as parse_prices.py and run it with python3 parse_prices.py.

import pandas as pd

# prices as a listings site exported them
raw = pd.Series(["310000", "275000", "$310,000", "call for price", "248000"])
prices = pd.to_numeric(raw)

Traceback (most recent call last):
  File "pandas/_libs/lib.pyx", line 2478, in pandas._libs.lib.maybe_convert_numeric
ValueError: Unable to parse string "$310,000"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/goku/dev/bootcamp/parse_prices.py", line 5, in <module>
    prices = pd.to_numeric(raw)
             ^^^^^^^^^^^^^^^^^^
  File "/home/goku/dev/bootcamp/.venv/lib/python3.12/site-packages/pandas/core/tools/numeric.py", line 235, in to_numeric
    values, new_mask = lib.maybe_convert_numeric(  # type: ignore[call-overload]
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/lib.pyx", line 2521, in pandas._libs.lib.maybe_convert_numeric
ValueError: Unable to parse string "$310,000" at position 2

Read it the way Lesson 1.6.2 taught a chained traceback, bottom first. The last line names the value that could not be parsed, "$310,000", and its position, 2, the third value counting from 0. The block above it is pandas' own parser failing on the same value, and "call for price" at position 3 was never reached, because to_numeric stops at the first failure. astype(float) on the same column stops at the same value with ValueError: could not convert string to float: '$310,000', which leaves you to find it.

The position counts along the column from 0, whatever its index labels are. On a frame that was filtered or had rows dropped the two part ways, so the failing value is s.iloc[2], Lesson 2.4.2's access by position.

errors="coerce" makes the error go away by turning every value it cannot parse into NaN, and nothing announces the loss, so a missing count taken before and after is the check.

import pandas as pd

raw = pd.Series(["310000", "275000", "$310,000", "call for price", "248000"])
print(raw.isna().sum())     # 0
prices = pd.to_numeric(raw, errors="coerce")
print(prices.isna().sum())  # 2
print(prices.tolist())      # [310000.0, 275000.0, nan, nan, 248000.0]

Two values became NaN without a word, and one of them, "$310,000", was a real price. The fix is to clean the text first and coerce only what is left.

import pandas as pd

raw = pd.Series(["310000", "275000", "$310,000", "call for price", "248000"])
# '.str.replace' swaps text inside every value, Lesson 2.5.3
cleaned = raw.str.replace("$", "").str.replace(",", "")
prices = pd.to_numeric(cleaned, errors="coerce")
print(prices.isna().sum())  # 1
print(prices.tolist())      # [310000.0, 275000.0, 310000.0, nan, 248000.0]

The one value lost now is "call for price", which holds no number at all and is a genuine gap for Lesson 2.5.1's tools. The column is float64 because of that gap, and text made only of digits, with no gap, such as pd.Series(["310000", "248000"]), parses to int64.

DON'T:

• DON'T reach for errors="coerce" before reading what the raise names. Here it turned the real price "$310,000" into NaN with no message, so print the missing count before and after every coerce.
• DON'T copy errors="ignore" from an older tutorial. pandas 3.0.5 raises ValueError: invalid error value specified.
• DON'T look up the reported position with loc. After a filter or a drop, the label 2 is a different row from position 2, or no row at all.

Memory and Downcasting

What. memory_usage(deep=True) reports the bytes each column really occupies, and downcasting, pd.to_numeric(s, downcast="integer") or downcast="float", moves a numeric column into the smallest type that still holds every one of its values, a float to within 0.0005. Why. int64 spends 8 bytes on every value while a room count fits in 1, so on millions of rows the dtype decides whether a table fits in memory at all. When. On a table large enough for memory to matter, after its values are clean, and always measured before and after, so the saving is a number rather than a hope.

import pandas as pd

df = pd.read_csv("housing.csv")
shallow = df.memory_usage()
deep = df.memory_usage(deep=True)
print(pd.DataFrame({"shallow": shallow, "deep": deep}))
print(shallow.sum(), deep.sum())

              shallow   deep
Index             132    132
price            1672   1672
area_sqm         1672   1672
rooms            1672   1672
year_built       1672   1672
neighborhood     1672  12011
condition        1672  11226
listing_date     1672  12250
11836 42307

Without deep=True every column costs 8 bytes per value, 1672 bytes for 209 rows, because for a text column pandas counts only the 8-byte reference to each string. deep=True follows the reference and counts the string itself, the three text columns grow about seven times, and the frame goes from 11,836 bytes to 42,307. The memory usage: 11.6 KB line that ends Lesson 2.4.1's info() is the shallow figure, and df.info(memory_usage="deep") prints memory usage: 41.3 KB instead.

A smaller type holds a narrower range. np.iinfo is NumPy's table of an integer dtype's limits, and np.finfo is the same for a float dtype.

import numpy as np

i8, i16 = np.iinfo("int8"), np.iinfo("int16")
print(i8.min, i8.max)                 # -128 127
print(i16.min, i16.max)               # -32768 32767
print(np.finfo("float32").max)        # 3.4028235e+38
print(float(np.float32(16_777_217)))  # 16777216.0

int8 holds -128 to 127 in 1 byte and int16 holds -32,768 to 32,767 in 2. float32 reaches far past any price in 4 bytes, but it keeps only about seven significant digits, so every whole number up to 16,777,216 is exact and 16,777,217 is already stored as its neighbor.

import pandas as pd

df = pd.read_csv("housing.csv")
rooms = pd.to_numeric(df["rooms"], downcast="integer")
year = pd.to_numeric(df["year_built"], downcast="integer")
print(df["rooms"].max())        # 99
print(df["year_built"].max())   # 2023
print(rooms.dtype, year.dtype)  # int8 int16

rooms tops out at 99, the injected outlier, inside the 127 of int8, and the years reach 2023, past int8 and inside int16. downcast reads each column's own minimum and maximum and picks the smallest type that holds both, so it never wraps a value.

import pandas as pd

df = pd.read_csv("housing.csv")
area = pd.to_numeric(df["area_sqm"], downcast="float")
print(area.dtype)
print(area.head(3))

float32
0    131.800003
1    112.099998
2    102.199997
Name: area_sqm, dtype: float32

131.8 became 131.800003, because the nearest float32 to 131.8 sits a hair above it. On a floor area that is noise. On a column that needs every digit it is damage: a latitude of 40.7128123 comes back as 40.712811. A value float32 would move by more than 0.0005, such as the price 16,777,217, makes downcast keep the whole column float64.

price has gaps, and none of the types downcast="integer" chooses from, int8 to int64, has room for them.

import pandas as pd

df = pd.read_csv("housing.csv")
price = pd.to_numeric(df["price"], downcast="integer")
print(price.dtype)  # float64
whole = pd.to_numeric(df["price"].dropna(), downcast="integer")
print(whole.dtype)  # int32

No error and no warning: the column came back float64, exactly as it went in. With the gaps dropped, the same call finds whole numbers up to 9,900,000, past the 32,767 of int16, and returns int32.

Measured on the whole frame, the downcasts show where the bytes really are.

import pandas as pd

df = pd.read_csv("housing.csv")
before = df.memory_usage(deep=True).sum()
df["rooms"] = pd.to_numeric(df["rooms"], downcast="integer")
df["year_built"] = pd.to_numeric(df["year_built"], downcast="integer")
df["area_sqm"] = pd.to_numeric(df["area_sqm"], downcast="float")
after = df.memory_usage(deep=True).sum()
print(before, after)  # 42307 38754

Three downcasts saved 3,553 bytes, about 8 percent, because the three text columns hold 35,487 of the 42,307 bytes and no numeric type touches them. Measuring first showed where the memory sits, and the dtype built for a column of repeated text labels is category, Lesson 2.5.3's.

DON'T:

• DON'T compare your memory figures with a tutorial's or a notebook's. When pyarrow, a separate library for columnar data, is installed, pandas stores str text with it instead, and this same frame then measures 16,722 bytes deep under pyarrow 26.0.0, against the 42,307 printed here in (bootcamp), which has no pyarrow.
• DON'T measure a frame with text columns without deep=True. The shallow figure put this one at 11,836 bytes, under a third of its real size.
• DON'T expect downcast="integer" to fix a float64 column that has gaps. It returns the column unchanged and says nothing.
• DON'T narrow a column by hand with astype("int8"). pd.Series([120, 130, 300]).astype("int8") returns 120, -126 and 44 without a word, where downcast="integer" would have picked a type that holds all three.

Lab Exercises

Lab Exercise 1: Read the Types, Then Cast Them

Load housing.csv and print its dtypes. Then, for rooms and for price in turn, try astype(int) inside a try, printing the column name with the resulting dtype when the cast works and with the error's class name when it fails.

Convert price to the dtype that holds whole numbers and keeps its 20 gaps, assign it back, and print the column's dtype and its gap count. Finally, make a text version of year_built with astype(str) and print its first three values and its dtype.

Lab Exercise 2: Parse a Scraped Column, Then Shrink a Table

A scraper delivered room counts as text: raw_rooms = pd.Series(["3", "4", "two", "5", "", "6"]). Run pd.to_numeric on it in its default mode inside a try and print the error's message. Then convert it with errors="coerce", printing the missing count before and after, and add a comment naming each value that was lost and saying why only one of them raised.

Downcast the coerced column with downcast="integer" and print its dtype, then drop its gaps, downcast again and print the dtype.

Finally, load housing.csv, downcast rooms and year_built with the result assigned back, and print the frame's deep memory before and after, the bytes saved, and the two new dtypes.

import pandas as pd

df = pd.read_csv("housing.csv")
print(df.dtypes)
for col in ["rooms", "price"]:
    try:
        print(col, df[col].astype(int).dtype)
    except ValueError as err:
        print(col, type(err).__name__)
df["price"] = df["price"].astype("Int64")
print(df["price"].dtype, df["price"].isna().sum())
years = df["year_built"].astype(str)
print(years.head(3).tolist(), years.dtype)

price           float64
area_sqm        float64
rooms             int64
year_built        int64
neighborhood        str
condition           str
listing_date        str
dtype: object
rooms int64
price IntCastingNaNError
Int64 20
['1978', '1953', '2004'] str

import pandas as pd

raw_rooms = pd.Series(["3", "4", "two", "5", "", "6"])
try:
    pd.to_numeric(raw_rooms)
except ValueError as err:
    print(err)
print(raw_rooms.isna().sum())
rooms = pd.to_numeric(raw_rooms, errors="coerce")
print(rooms.isna().sum())
# Lost: "two", which holds no digits, and "", which to_numeric reads
# as a gap without raising, so the raise named only "two".

print(pd.to_numeric(rooms, downcast="integer").dtype)
print(pd.to_numeric(rooms.dropna(), downcast="integer").dtype)

Unable to parse string "two" at position 2
0
2
float64
int8

import pandas as pd

df = pd.read_csv("housing.csv")
before = df.memory_usage(deep=True).sum()
for col in ["rooms", "year_built"]:
    df[col] = pd.to_numeric(df[col], downcast="integer")
after = df.memory_usage(deep=True).sum()
print(before, after, before - after)
print(df["rooms"].dtype, df["year_built"].dtype)

42307 39590 2717
int8 int16

Theory Assessment

Q1: The float64 That Holds Only Whole Numbers

The price column of housing.csv is float64, yet every price in the file is a whole number of dollars. Explain why pandas chose float64, what astype(int) does to this column and why, and what astype("Int64") changes.

Hint: Ask what an int64 would have to store in the 20 cells that hold nothing. The message of the failing cast names the way out.

Q2: Two Errors for One Bad Value

Both astype(float) and pd.to_numeric refuse the scraped price "$310,000". Compare what the two error messages tell you, then explain what errors="coerce" does with the same column and why that is more dangerous than either error. Say which two numbers you print to see what a coerce cost.

Hint: Put the two messages side by side and look for a number in one of them. For the last part, a gap made by a coerce looks exactly like a gap that was always there.

Q3: Where the Bytes Are

Explain why memory_usage() without deep=True reports 11,836 bytes for housing.csv when the real figure is 42,307, and why downcasting rooms, year_built and area_sqm together saves only about 8 percent. Then say what pd.to_numeric(df["price"], downcast="integer") returns, and why.

Hint: For a text column, ask what the 8 bytes per value actually point to. For price, count its gaps.

Q4: Three Conversions That Raised Nothing

Read this script without running it and state what each print shows.

import pandas as pd

counts = pd.Series(["3", "4", "120", "130"])
a = pd.to_numeric(counts, downcast="integer")
b = pd.to_numeric(counts).astype("int8")
c = pd.to_numeric(pd.Series(["3", "", "5"]), downcast="integer")
print(a.dtype, a.tolist())
print(b.dtype, b.tolist())
print(c.dtype, c.tolist())

None of the three raises, and two of them hold a surprise. Explain the dtype of each result, and every value that differs from the text it came from.

Hint: Check 130 against the range of int8. Then ask which of the two narrowing steps looked at the values before it chose a type, and what an empty string turns into.

int16 [3, 4, 120, 130]
int8 [3, 4, 120, -126]
float64 [3.0, nan, 5.0]

Assignment

Assignment: Type-Clean a Raw Listings Export

A partner agency sends its listings as a CSV in which prices are written the way people read them and one agent typed a room count as a word. Create the file with this block first, which writes it with Module 1's open:

rows = [
    "listing_id,price,area_sqm,rooms,year_built",
    'L-101,"$437,000",131.8,2,1978',
    'L-102,"$268,000",112.1,5,1953',
    "L-103,call for price,102.2,6,2004",
    'L-104,"$150,000",,1,1979',
    'L-105,"$203,000",121.7,two,1984',
    'L-106,"$281,000",105.6,6,2021',
    'L-107,"$1,250,000",88.0,3,2019',
    'L-108,"$248,000",97.5,4,1999',
]
with open("listings_raw.csv", "w", encoding="utf-8") as f:
    f.write("\n".join(rows) + "\n")

Read listings_raw.csv and print its dtypes, with a comment saying which columns should hold numbers but arrived as text, and what in the file made each one text. Write a function to_number(df, column) that tries pd.to_numeric on the column in its default mode and, if that raises, prints the column name and the error's message, then strips $ and , with .str.replace, converts with errors="coerce", prints the missing count before and after, and returns the converted column. Run it on price and on rooms and assign both results back.

Then store rooms in the dtype that keeps whole numbers and its gap, and shrink every numeric column with a downcast that matches its kind. df.select_dtypes(include="integer") returns only the columns of an integer dtype, the nullable Int64 included, and include="float" only the float ones, so the .columns of each tells you which downcast a column takes. Print the dtypes and the deep memory of the frame before and after the downcasts.

Hints:

• The function only has to act when the default mode raises, so the try returns the plain conversion, and the except prints the message and lets the function fall through to the cleaning, the coerce and the two counts.
• downcast="float" on an integer column converts it to float32, which is why the integer and float columns go through separate loops.

import pandas as pd

df = pd.read_csv("listings_raw.csv")
print(df.dtypes)
# 'price' is text because of the '$' and ',' in every value and the
# note "call for price". 'rooms' is text because of the word "two".

listing_id        str
price             str
area_sqm      float64
rooms             str
year_built      int64
dtype: object

import pandas as pd

def to_number(df, column):
    try:
        return pd.to_numeric(df[column])
    except ValueError as err:
        print(column, "|", err)
    text = df[column].str.replace("$", "").str.replace(",", "")
    out = pd.to_numeric(text, errors="coerce")
    print(column, "missing before:", df[column].isna().sum())
    print(column, "missing after:", out.isna().sum())
    return out

df = pd.read_csv("listings_raw.csv")
df["price"] = to_number(df, "price")
df["rooms"] = to_number(df, "rooms").astype("Int64")
print(df.dtypes)

price | Unable to parse string "$437,000" at position 0
price missing before: 0
price missing after: 1
rooms | Unable to parse string "two" at position 4
rooms missing before: 0
rooms missing after: 1
listing_id        str
price         float64
area_sqm      float64
rooms           Int64
year_built      int64
dtype: object

before = df.memory_usage(deep=True).sum()
for col in df.select_dtypes(include="integer").columns:
    df[col] = pd.to_numeric(df[col], downcast="integer")
for col in df.select_dtypes(include="float").columns:
    df[col] = pd.to_numeric(df[col], downcast="float")
after = df.memory_usage(deep=True).sum()
print(df.dtypes)
print(before, after)

listing_id        str
price         float32
area_sqm      float32
rooms            Int8
year_built      int16
dtype: object
828 660

Additional Resources

Optional, and nothing later depends on them. Worth a look if you want to see how far a text column can shrink, or how the nullable integers behave in arithmetic and comparisons.

• YouTube - "How do I make my pandas DataFrame smaller and faster?" by Data School. Measuring a frame's memory and shrinking its text columns with the category dtype, recorded on an older pandas whose figures differ from yours.
• Official docs - pandas, "Nullable integer data type". How Int64 behaves in arithmetic, comparisons and reductions once pd.NA is in the column.

Summary

Data types are where a table states what it believes its values are, and the first lesson here is that this belief is set by the file, not by the data's meaning. A single gap turned every price into a float, a single word turned a column of room counts into text, and pandas did nothing wrong in either case: it chose a type that could hold everything it was given. Reading dtypes is reading those choices, and every conversion after it is a decision to override one of them, which is only safe when you know what the column is supposed to hold.

The three conversion tools differ in how they treat a value that does not fit, and that difference is the whole of choosing between them. astype trusts you and stops loudly when it cannot comply, except when you narrow a number, where it bends the value into the type without a word, and when a label is missing from a category list, where Lesson 2.5.3 shows it becoming a gap. to_numeric names the value and the place it failed, which is the loud failure worth wanting, until errors="coerce" turns that failure into silent loss. A downcast checks the range before it chooses. Each time, the safe version is the one that either refuses or checks, and the unsafe version is the one that finishes quietly with a different column than it started with.

Memory turned out to be a lesson in measuring before acting. The obvious move, shrinking the integers, saved a few percent, because the bytes were in the text all along, and only deep=True could show that. The figures themselves belong to this machine and this setup, which is why each one was printed rather than remembered. With absent values handled and present ones stored as the right kind, the columns are ready for the next lesson's harder kinds of text: labels that repeat, strings that need cleaning, and dates.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import pandas as pd

df = pd.read_csv("housing.csv")
df.dtypes                                   # check before any math
df["price"] = df["price"].astype("Int64")   # whole, gaps kept

raw = pd.Series(["310000", "$310,000"])
# pd.to_numeric(raw) raises: "$310,000" at position 1
lost = pd.to_numeric(raw, errors="coerce")  # [310000.0, nan]
txt = raw.str.replace("$", "").str.replace(",", "")
nums = pd.to_numeric(txt)                   # parses once cleaned

df.memory_usage(deep=True).sum()            # real bytes, text included
df["rooms"] = pd.to_numeric(df["rooms"], downcast="integer")