Center, Spread, Shape and Correlation

Context

A handful of numbers describe a column of values: where its middle sits, how widely the values scatter around that middle, whether one side stretches much further than the other, and, for a pair of columns, how closely one rises and falls with the other. pandas returns each of them from one method call on a Series or a DataFrame.

They are the first numbers anyone asks about a training table. The typical price is the figure a client reads, the spread of a feature says whether it needs scaling before a distance or a model uses it, a long right tail on the target is the usual reason to model the log of the price, and a correlation with the target is the first evidence that a feature carries signal, while a strong one between two features warns that they carry the same signal twice.

Each of these numbers can be steered by a few rows, and nothing warns you when it is. Two extreme listings in housing.csv pull the mean price more than a hundred thousand above the median, one listing manufactures a correlation between floor area and price that the rest of the table does not have, and NumPy and pandas hand back two different standard deviations for one column. Section 2.6 turned the cleaned table into features and grouped summaries, and this lesson decides which numbers about that table can be trusted.

Goal

• Compute the center of a column with mean(), median() and mode(), and choose the median for a skewed price.
• Measure spread with var(), std(), the range and the IQR, and make NumPy and pandas agree on a standard deviation through ddof.
• Detect a long right tail with skew() and the gap between mean and median, and shrink it with np.log1p.
• Compute Pearson and Spearman correlations and a correlation matrix, and count the rows each coefficient rests on.
• Identify the outlier, the third variable or the curve behind a misleading coefficient, and state precisely what a correlation does and does not establish.

Prerequisites

Optional, and the lesson stands without them. They are here if you want the arithmetic drawn out by hand before pandas does it for you.

• YouTube - "Calculating the Mean, Variance and Standard Deviation, Clearly Explained!!!" by StatQuest with Josh Starmer. The three calculations worked on a small sample, and the pitfall of dividing by n that the sample formula avoids.
• YouTube - "Pearson's Correlation, Clearly Explained!!!" by StatQuest with Josh Starmer. What the coefficient means, how to read it, and where it stops working.

Table of Contents

• Center - Mean, Median and Mode
• Spread - Variance, Standard Deviation and IQR
• Shape - Skew and Tails
• Correlation - Pearson and Spearman
• Correlation Is Not Causation

Theory

Every Theory block adds to one script, describe.py in ~/dev/bootcamp, run with python3 describe.py after each addition. The first block loads housing.csv, the download Lesson 2.4.1 saved there, and drops its repeated rows as Lesson 2.5.4 did. Nothing else is cleaned: the extreme listings 2.5.4 flagged are what this lesson measures.

Center - Mean, Median and Mode

What. The center of a column is one value that stands for a typical row. The mean adds every value and divides by the count, the median is the middle value once the column is sorted, the average of the middle two when the count is even, and the mode is the value that occurs most often. Why. "What does a listing cost" is answered with one of these, and on this table's skewed prices they disagree by more than a hundred thousand, so the one a report quotes decides what its reader believes about the market. When. Compute the mean and the median together for every numeric column, and quote the median when they disagree. Use the mode for a column of categories or a short list of whole numbers, such as condition or rooms.

mean() and median() are Series methods, and like every pandas statistic in this lesson they skip missing values the way 2.6.2's count() does. Both centers below describe the listings that have a price, not every row.

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
price = df["price"]
priced = price.dropna()

# rows, then the prices that are present
print(len(df), price.count())    # 204 185
print(f"{price.mean():,.0f}")    # 356,665
print(f"{price.median():,.0f}")  # 247,000

The gap has the cause Lesson 2.5.1 found when it chose the median to fill area_sqm, so the two centers answer different questions:

extreme = priced[priced > 1_000_000]
total = priced.sum()
n = len(priced)
print(extreme.tolist())                 # [5200000.0, 9900000.0]
print(round(extreme.sum() / total, 2))  # 0.23
print(f"{total:,.0f}")                  # 65,983,000
print(f"{priced.mean() * n:,.0f}")      # 65,983,000
print(f"{priced.median() * n:,.0f}")    # 45,695,000

The mean is the total spread evenly: 185 listings at the mean add back up to the 65,983,000 the listings are worth, and two of them hold 23 percent of that total. That makes the mean the center for a question about the total, such as a portfolio's value. "What does a listing cost" is answered by the median, the middle price once the listings are sorted, which no listing at the top can drag.

mode() answers a different question, which value occurs most often, and it returns a Series rather than a single value, because several values can tie for most frequent.

print(df["condition"].mode())
print(df["rooms"].mode())
print(df["rooms"].value_counts().head(3))

0    good
Name: condition, dtype: str
0    2
1    5
Name: rooms, dtype: int64
rooms
2    36
5    36
6    34
Name: count, dtype: int64

condition has one mode, good. rooms has two, 2 and 5, held by 36 listings each, and the Series carries both. On a continuous column the mode only finds a value that happens to repeat:

print(priced.mode().tolist())     # [203000.0]
print((priced == 203_000).sum())  # 5

Five listings share a price of 203,000, and that coincidence is all the mode of price reports, which is why the mode belongs to categories and small whole numbers.

DON'T:

• DON'T quote the mean as the typical price of a skewed column. It is the total spread evenly, and two listings hold 23 percent of this column's total.
• DON'T take mode()[0] and move on. rooms ties at 2 and 5, and [0] keeps the 2 and drops the 5 without a word.

Spread - Variance, Standard Deviation and IQR

What. Spread measures how far a column's values sit from its center: Lesson 2.2.5's variance and standard deviation, the range, which is the largest value minus the smallest, and Lesson 2.5.4's IQR. Why. A center without a spread hides whether listings cluster near it or scatter over millions, and every scaling step, 2.2.5's standardization first, divides by a spread. Three of these four measures are moved by one extreme listing, and the IQR is not. When. Report the standard deviation beside the mean for a column without extreme values, and the IQR beside the median for a column with them. Set ddof yourself whenever a NumPy number and a pandas number for the same column have to agree.

pandas' std() and var() divide by n minus 1, ddof=1, the sample version Lesson 2.2.5 named, and NumPy's divide by n unless told otherwise. One column therefore has two standard deviations, depending on which library you ask:

arr = priced.to_numpy()
print(f"{priced.std():,.0f}")             # 798,939
print(f"{np.std(arr):,.0f}")              # 796,776
print(f"{priced.std(ddof=0):,.0f}")       # 796,776
print(f"{np.std(arr, ddof=1):,.0f}")      # 798,939
print(f"{price.describe()['std']:,.0f}")  # 798,939

ddof=0 makes pandas print NumPy's number, ddof=1 makes NumPy print pandas', and the std row of 2.4.1's describe() is the ddof=1 value. With 185 values the two differ by little, which is how the mismatch goes unnoticed until two scripts that should agree do not.

print(f"{priced.var():,.0f}")  # 638,303,039,248

The variance is in squared currency units, which is why a report quotes its square root, the standard deviation, instead.

The range and the IQR come from positions in the sorted column, the range from its two ends and the IQR from 2.5.4's Q1 and Q3:

value_range = priced.max() - priced.min()
q1, q3 = priced.quantile(0.25), priced.quantile(0.75)
print(f"{value_range:,.0f}")  # 9,760,000
print(f"{q3 - q1:,.0f}")      # 180,000

The same measures on the prices below a million show which ones the two extreme listings were writing:

# the prices without the two listings 2.5.4 flagged
typical = priced[priced < 1_000_000]
t_range = typical.max() - typical.min()
t_iqr = typical.quantile(0.75) - typical.quantile(0.25)
print(len(typical))             # 183
print(f"{typical.std():,.0f}")  # 98,995
print(f"{t_range:,.0f}")        # 371,000
print(f"{t_iqr:,.0f}")          # 178,500

Two listings out of 185 take the standard deviation from 98,995 to 798,939 and the range from 371,000 to 9,760,000, and the IQR only from 178,500 to 180,000. Squaring each distance hands the farthest values most of the sum, so the full column's standard deviation reports how far two listings sit, the inflation that made 2.5.4's z-score wait for a price past 2,753,481, and the range is set by the two ends alone. The IQR reads two positions inside the sorted column and, like the median, ignores how far the values beyond them reach, which is why 2.5.4 built its outlier fences from it.

DON'T:

• DON'T compare a pandas standard deviation with a NumPy one without setting ddof. 798,939 against 796,776 is not a bug, it is two divisors.
• DON'T report the standard deviation of a column with extreme values as its typical spread. Here two listings take it from 98,995 to 798,939.

Shape - Skew and Tails

What. Shape is whether a column's values pile up on one side of the center and trail off on the other. skew() returns a number that is positive when the right tail, the side of large values, is the longer one, negative when the left tail is, and near 0 for a symmetric column. Why. Prices and areas trail off to the right, and on a target with a long right tail a few expensive listings dominate any error measured in currency, the usual reason a price model is fitted on the log of the price instead. When. Check the skew of every numeric column on a first look, and of the target before choosing what a model predicts. A mean above the median is the quick sign of a right tail before you compute anything.

print(round(priced.skew(), 2))             # 10.58
print(round(typical.skew(), 2))            # 0.64
print(f"{typical.mean():,.0f}")            # 278,049
print(f"{typical.median():,.0f}")          # 245,000
print(round(np.log1p(typical).skew(), 2))  # 0.26
print(round(np.log1p(priced).skew(), 2))   # 3.42
print(type(np.log1p(typical)).__name__)    # Series

The full column scores 10.58, and without the two listings above a million it scores 0.64, a mild right tail that the typical prices' mean of 278,049 above their median of 245,000 announces with no skew computed. Lesson 2.2.3's np.log1p works on a Series and returns one, and it pulls the typical prices in to 0.26, because a log turns a ratio into a distance and so compresses the expensive end far more than the cheap end. On the full column it gets 10.58 down to 3.42 and no further: the logs of 5,200,000 and 9,900,000 still lie far beyond the logs of the other 183 prices, so a log transform shrinks a tail and does not make an extreme listing ordinary.

rng = np.random.default_rng(0)
bell = pd.Series(rng.normal(size=100_000))
print(round(priced.kurt(), 1))   # 118.8
print(round(typical.kurt(), 2))  # -0.85
print(round(bell.kurt(), 2))     # 0.02

kurt() scores how heavy a column's tails are against a bell curve, which scores about 0, so the full column's 118.8 says its tails hold far more extreme values than a bell curve would, and the typical prices' -0.85 says theirs hold fewer.

DON'T:

• DON'T read a skew that fell after a log as a column without extreme values. The log took 10.58 to 3.42, and the two listings behind it are still in the column.
• DON'T read skew() on a column with a couple of extreme values as the shape of the rest. Two listings took price from 0.64 to 10.58.

Correlation - Pearson and Spearman

What. A correlation is one number from -1 to 1 saying how closely two columns rise and fall together, positive when they rise together and negative when one falls as the other rises. Pearson's r, the default of corr(), measures how well one straight line fits the pairs, and Spearman's rank correlation, method="spearman", first replaces every value with its position in sorted order, so it measures how consistently one column rises as the other does. Why. It is the quickest test of whether a feature carries any signal about the target, and of whether two features carry the same signal twice, the collinear columns Lesson 2.3.5 found. When. Compute both coefficients for every feature against the target on a first look, and read the gap between them before trusting either. Count the rows behind every coefficient you quote.

corr() between two Series returns Pearson's r, using only the rows where both have a value:

r = df["price"].corr(df["area_sqm"])
print(round(r, 3))  # 0.695

Pearson rewards a straight line, not merely a rising one. In the table below every price rises with its area, and the rise bends:

sizes = np.arange(40, 201, 10)
curve = pd.DataFrame({"area_sqm": sizes, "price": 0.05 * sizes ** 3})
pearson = curve["area_sqm"].corr(curve["price"])
ranks = curve.corr(method="spearman")
print(round(pearson, 3))               # 0.949
print(ranks.loc["area_sqm", "price"])  # 1.0

Pearson scores the curve 0.949 because the points bend away from any one line, and Spearman scores it 1.0 because ranks only ask whether the two orders agree, and here they agree perfectly. Spearman is asked for on a DataFrame, which returns a matrix of coefficients read with .loc. Passed to a Series, df["price"].corr(df["area_sqm"], method="spearman") hands the work to SciPy (a library of scientific routines) and stops with ModuleNotFoundError: No module named 'scipy', until SciPy arrives with scikit-learn in Lesson 3.1.1.

pair = df[["price", "area_sqm"]]
ranked = pair.corr(method="spearman")
print(ranked)
rho = ranked.loc["price", "area_sqm"]
print(round(rho, 3))  # -0.048

            price  area_sqm
price     1.00000  -0.04788
area_sqm -0.04788   1.00000

Pearson says 0.695, a relationship most people would report, and Spearman says -0.048. One listing is responsible:

big = df["price"] == df["price"].max()
print(df.loc[big, ["price", "area_sqm", "neighborhood"]])

rest = df[~big]
r_rest = rest["price"].corr(rest["area_sqm"])
rest_ranks = rest[["price", "area_sqm"]].corr(method="spearman")
rho_rest = rest_ranks.loc["price", "area_sqm"]
print(round(r_rest, 3))    # -0.006
print(round(rho_rest, 3))  # -0.067

         price  area_sqm neighborhood
203  9900000.0    2400.0     Eastgate

Without the 9,900,000 listing of 2,400 square meters, Pearson falls from 0.695 to -0.006 while Spearman moves from -0.048 to -0.067. Pearson's r is Lesson 2.3.1's dot product taken on two standardized columns, 2.2.5's line with pandas' ddof=1, divided by n minus 1, so every listing adds the product of how far it sits from both means:

both = df[["price", "area_sqm"]].dropna()
z = (both - both.mean()) / both.std()
dot = z["price"] @ z["area_sqm"]
products = z["price"] * z["area_sqm"]
print(len(both) - 1, round(dot, 1))     # 169 117.4
print(round(dot / (len(both) - 1), 3))  # 0.695
print(round(products.max(), 1))         # 117.0

That one listing sits far above both means, and its product is 117.0 of the 117.4 the 170 products add up to, nearly the whole coefficient. A rank cannot exceed the number of rows, so in Spearman the same listing counts as the highest price and the largest area and nothing more.

corr() on a whole DataFrame returns the matrix of every pair. df.corr() on this table stops at the first text column with ValueError: could not convert string to float: 'Lakeview', and numeric_only=True keeps the numeric columns:

matrix = df.corr(numeric_only=True)
print(matrix.round(3))

            price  area_sqm  rooms  year_built
price       1.000     0.695  0.056      -0.183
area_sqm    0.695     1.000  0.059      -0.156
rooms       0.056     0.059  1.000       0.013
year_built -0.183    -0.156  0.013       1.000

Each cell uses only the rows where its own two columns both have a value, so the cells of one matrix rest on different rows:

pairs = [("price", "area_sqm"), ("price", "rooms"), ("rooms", "year_built")]
for a, b in pairs:
    print(a, b, len(df[[a, b]].dropna()))

price area_sqm 170
price rooms 185
rooms year_built 204

The price and area_sqm cell rests on 170 listings and the rooms and year_built cell on all 204, and the matrix prints both to three decimals with nothing to tell them apart.

DON'T:

• DON'T trust a Pearson coefficient on columns with extreme values until Spearman agrees with it. One listing out of 170 turns -0.006 into 0.695.
• DON'T compare two cells of a correlation matrix as if they described the same rows. The cells of this one rest on anything from 170 rows to 204.

Correlation Is Not Causation

What. A correlation says two columns moved together in the rows you measured. It does not say that either one moves the other, which way an influence would run, or that the relationship has the shape a straight line or a ranking assumes. Why. The code computes a coefficient correctly every time. The mistake is made in the sentence written about it, which decides what features a model gets and what a report promises. When. Every time you write a sentence about a coefficient, and every time a coefficient near 0 is about to drop a feature from consideration.

Three things in this lesson can produce a coefficient that misleads without any error: an extreme listing, which the previous subtitle measured, a third variable that moves both columns, and a relationship that is strong but neither a straight line nor a steady rise. The third variable first, in a table built so the answer is known:

rng = np.random.default_rng(7)
# inside each neighborhood, area and price are drawn
# independently, and a single string fills its whole column
east = pd.DataFrame({
    "neighborhood": "Eastgate",
    "area_sqm": rng.normal(140, 10, 40),
    "price": rng.normal(430_000, 20_000, 40),
})
north = pd.DataFrame({
    "neighborhood": "Northend",
    "area_sqm": rng.normal(80, 10, 40),
    "price": rng.normal(200_000, 20_000, 40),
})
both_hoods = pd.concat([east, north], ignore_index=True)


def r_of(frame):
    return round(frame["area_sqm"].corr(frame["price"]), 2)


print(r_of(both_hoods))  # 0.95
print(r_of(east))        # -0.13
print(r_of(north))       # 0.04

Inside each neighborhood, area has no effect on price, by construction, and the two neighborhoods pooled correlate at 0.95. The neighborhood moves both columns: Eastgate listings are larger and dearer, Northend listings smaller and cheaper, so the coefficient measures where a listing is and not what a square meter is worth. A model that learned price from area alone in this table would be reading the neighborhood through the area.

Then the curve, where every price is fixed exactly by the age of the building:

age = np.arange(0, 101, 10)
u = pd.DataFrame({"age": age, "price": 150_000 + 30 * (age - 50) ** 2})
u_ranks = u.corr(method="spearman")

print(u["price"].iloc[[0, 5, 10]].tolist())  # [225000, 150000, 225000]
print(u["age"].corr(u["price"]))             # 0.0
print(u_ranks.loc["age", "price"])           # 0.0

New builds and century-old homes price high, fifty-year-old ones low, and both coefficients are 0.0. The prices fall for fifty years and climb back by the same amounts, so no straight line and no single direction of rise fits better than a flat one. A Pearson coefficient near 0 rules out a straight line, a Spearman coefficient near 0 rules out a steady rise, and neither rules out a relationship.

What a correlation lets you say is narrow: in these rows, these two columns rose together this closely, along a straight line for Pearson or in rank order for Spearman. It does not let you say that larger homes cost more because they are larger, that age does not matter to price, or that the pattern holds for listings outside the table. Each of those sentences needs evidence the coefficient does not contain.

DON'T:

• DON'T read a coefficient near 0 as "unrelated". The age table is an exact function of age and scores 0.0 on both methods.
• DON'T write "area drives price" from a coefficient. Write that the two moved together in these rows, and say how many rows.

Lab Exercises

Lab Exercise 1: Profile the Floor Area

Work in describe_area.py in ~/dev/bootcamp. Load housing.csv without its repeated rows and keep the area_sqm values that are present.

Print how many there are, their mean and median to one decimal place, and the share of them below the mean to two. Print the mode of year_built as a list, and say in a comment how many values it holds and why.

Print the standard deviation of the areas from pandas, then from NumPy, then make NumPy print pandas' number. Print the range and the IQR to one decimal place, then the skew before and after np.log1p, to two.

Finally, repeat the count, mean, median, standard deviation, IQR and skew on the areas under 1,000 square meters, and end with a comment naming the center and the spread you would report for the column, and why.

Lab Exercise 2: Year Built Against Price

Work in year_price.py in ~/dev/bootcamp, loading the table without its repeated rows.

Print Pearson's and Spearman's coefficients between year_built and price, to three decimal places. Print the listings built before 1950, then both coefficients again without them.

Print how many rows the year_built and price coefficient rests on, then how many rows have a value in all four numeric columns, and Pearson's coefficient computed on those complete rows only.

Print the Pearson matrix minus the Spearman matrix for the numeric columns, rounded to three places, and name in a comment the pair on which the two methods disagree most. End with a comment stating what the year_built coefficient lets you claim and what it does not.

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
area = df["area_sqm"].dropna()
below = (area < area.mean()).mean()

print(len(area))                         # 189
print(f"{area.mean():.1f}")              # 123.7
print(f"{area.median():.1f}")            # 102.4
print(round(below, 2))                   # 0.75
print(df["year_built"].mode().tolist())  # [1988, 1992]
# Two values, because 1988 and 1992 tie for the most
# frequent year, and mode() returns every tied value.

arr = area.to_numpy()
span = area.max() - area.min()
iqr = area.quantile(0.75) - area.quantile(0.25)

print(f"{area.std():.1f}")              # 211.4
print(f"{np.std(arr):.1f}")             # 210.9
print(f"{np.std(arr, ddof=1):.1f}")     # 211.4
print(f"{span:.1f}")                    # 2373.6
print(f"{iqr:.1f}")                     # 40.3
print(round(area.skew(), 2))            # 9.63
print(round(np.log1p(area).skew(), 2))  # 3.05

small = area[area < 1000]
small_iqr = small.quantile(0.75) - small.quantile(0.25)

print(len(small))               # 187
print(f"{small.mean():.1f}")    # 102.3
print(f"{small.median():.1f}")  # 102.3
print(f"{small.std():.1f}")     # 28.9
print(f"{small_iqr:.1f}")       # 38.8
print(round(small.skew(), 2))   # 0.09
# Report the median, 102.4, and the IQR, 40.3: two listings above
# 1,000 square meters pull the mean to 123.7 and the standard
# deviation to 211.4, and the median and the IQR barely notice.

import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
num = ["price", "area_sqm", "rooms", "year_built"]


def report(frame, label):
    pearson = frame["year_built"].corr(frame["price"])
    ranks = frame[["year_built", "price"]].corr(method="spearman")
    spearman = ranks.loc["year_built", "price"]
    print(f"{label}: pearson {pearson:.3f}, spearman {spearman:.3f}")


report(df, "all")
print(df.loc[df["year_built"] < 1950, ["price", "year_built"]])
report(df[df["year_built"] >= 1950], "from 1950")

all: pearson -0.183, spearman -0.032
         price  year_built
203  9900000.0        1911
from 1950: pearson 0.101, spearman -0.016

pair_rows = len(df[["year_built", "price"]].dropna())
complete = df[num].dropna()
listwise = complete["year_built"].corr(complete["price"])

print(pair_rows, len(complete))  # 185 170
print(f"{listwise:.3f}")         # -0.190

pearson_m = df.corr(numeric_only=True)
spearman_m = df.corr(method="spearman", numeric_only=True)
print((pearson_m - spearman_m).round(3))

            price  area_sqm  rooms  year_built
price       0.000     0.742  0.102      -0.151
area_sqm    0.742     0.000  0.040      -0.256
rooms       0.102     0.040  0.000       0.018
year_built -0.151    -0.256  0.018       0.000

# price and area_sqm disagree most, 0.742 apart, the one listing
# from Theory. The year_built coefficient lets you claim that in
# these 185 listings, year built and price show no consistent
# relationship once the 1911 listing is set aside. It does not let
# you claim that age has no effect on price, or any effect at all.

Theory Assessment

Q1: The Center a Report Should Quote

The prices in the deduplicated table have a mean of 356,665 and a median of 247,000. Say which one you would quote as the price of a typical listing, and name what in the column makes the two so far apart. Then say what df["rooms"].mode() returns on this table, and what df["rooms"].mode()[0] throws away.

Hint: Ask which of the two centers would change if the most expensive listing cost twice as much again. For the second half, count how many values tie for most frequent.

Q2: Two Listings and Four Measures of Spread

Without the two listings above 1,000,000, the standard deviation of price falls from 798,939 to 98,995 and the range from 9,760,000 to 371,000, while the IQR moves only from 180,000 to 178,500. Explain why. Then explain why priced.std() and np.std(priced.to_numpy()) disagree on the same 185 prices, and what makes them agree. Finally, say why np.log1p takes the skew of price from 10.58 only as far as 3.42.

Hint: Squares, positions and divisors. The standard deviation squares every distance, the IQR reads two positions, and the two libraries divide one sum by two different numbers.

Q3: A Coefficient Someone Wants to Publish

A teammate reads 0.695 for price against area_sqm from df.corr(numeric_only=True) and writes in a report: "bigger homes cost more, floor area explains price". Using only what this lesson ran, give three separate reasons the sentence does not follow from the number. Then write the sentence the number does support.

Hint: One reason is a single row, one is a second method run on the same two columns, and one concerns what any coefficient can say, however it was computed.

Q4: Five Lines, One Column, Three Answers

Read this script without running it. It relies on one more fact: np.std, handed an object that is not a NumPy array but has a std method of its own, calls that method and passes it ddof=0.

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
values = df["price"].to_numpy()

print(np.median(values))
print(df["price"].median())
print(np.std(values))
print(np.std(df["price"]))
print(df["price"].std())

Say what each line prints, in terms of numbers this lesson already printed, and explain why the five lines produce three different kinds of answer for one column.

Hint: Two questions decide every line: does the calculation skip the 19 missing prices or meet them, and does it divide by n or by n minus 1. A NumPy array answers the first question differently from a Series.

nan
247000.0
nan
796776.475901056
798938.6955505057

Assignment

Assignment: A Feature Screen That Shows Its Working

A teammate wants a screen of the numeric features against price, with every number in it checkable. Write screen.py in ~/dev/bootcamp with a function screen(frame, feature, target="price") that returns a dictionary for one feature: how many rows have both the feature and the target, Pearson's coefficient on those rows, Spearman's coefficient as pandas computes it on those rows, Spearman's coefficient again recomputed by hand, and the feature's median, IQR and skew.

The hand-made Spearman uses rank(). On a Series, rank() replaces each value with its position in sorted order, counting from 1, and gives tied values the average of the positions they share, so pd.Series([10, 20, 20, 30]).rank() holds 1.0, 2.5, 2.5 and 4.0. On a DataFrame it ranks each column on its own. Spearman's coefficient is Pearson's coefficient computed on ranks, so if your hand-made value disagrees with pandas' you have ranked the wrong rows.

Run the screen for area_sqm, rooms and year_built on the table without its repeated rows and print one line per feature, marking the line CHECK whenever Pearson and Spearman differ by more than 0.1. Then rank the whole price column and the whole area_sqm column before any row is dropped, correlate the two rankings with Pearson, and print the result to four places beside pandas' Spearman coefficient. Explain in a comment why the two differ and which one is Spearman's coefficient for this pair.

Hints:

• Drop the rows missing either column first, and do every calculation for the pair, the ranks included, on what is left.
• A rank depends on every other value in the column it was computed in. Ask which rows the whole-column ranks were computed among, and which rows the correlation then uses.

import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()


def screen(frame, feature, target="price"):
    pair = frame[[feature, target]].dropna()
    # rank each column of the complete pairs on its own
    ranks = pair.rank()
    col = frame[feature].dropna()
    spearman = pair.corr(method="spearman")
    return {
        "n": len(pair),
        "pearson": pair[feature].corr(pair[target]),
        "spearman": spearman.loc[feature, target],
        "by_hand": ranks[feature].corr(ranks[target]),
        "median": col.median(),
        "iqr": col.quantile(0.75) - col.quantile(0.25),
        "skew": col.skew(),
    }


for feature in ["area_sqm", "rooms", "year_built"]:
    s = screen(df, feature)
    gap = abs(s["pearson"] - s["spearman"])
    flag = "CHECK" if gap > 0.1 else ""
    # :<10 pads to 10 characters, :+ always prints the sign,
    # :g keeps at most six significant digits
    print(
        f"{feature:<10} n={s['n']} pearson={s['pearson']:+.3f} "
        f"spearman={s['spearman']:+.3f} by_hand={s['by_hand']:+.3f} "
        f"median={s['median']:g} iqr={s['iqr']:g} "
        f"skew={s['skew']:.2f} {flag}"
    )

area_sqm   n=170 pearson=+0.695 spearman=-0.048 by_hand=-0.048 median=102.4 iqr=40.3 skew=9.63 CHECK
rooms      n=185 pearson=+0.056 spearman=-0.046 by_hand=-0.046 median=3 iqr=3 skew=12.83 CHECK
year_built n=185 pearson=-0.183 spearman=-0.032 by_hand=-0.032 median=1990.5 iqr=35.5 skew=-0.32 CHECK

whole = df["price"].rank().corr(df["area_sqm"].rank())
rho = df[["price", "area_sqm"]].corr(method="spearman")
print(round(whole, 4), round(rho.loc["price", "area_sqm"], 4))
# -0.0463 -0.0479
# The whole-column ranks were computed among 185 prices and 189
# areas, so each listing's rank counts listings whose other value
# is missing. The correlation then uses only the 170 complete rows,
# with ranks that no longer run 1 to 170. pandas ranks the 170
# complete pairs themselves, so -0.0479 is Spearman's coefficient.

Additional Resources

Optional, and nothing later depends on either. Worth opening if you want the quantity a correlation is built from, or a famous case for drawing a table before trusting its numbers.

• YouTube - "Covariance, Clearly Explained!!!" by StatQuest with Josh Starmer. Covariance, what it does and does not tell you, and why it is a stepping stone to correlation rather than an answer in itself.
• Wikipedia - "Anscombe's quartet". Four small datasets with nearly identical means, variances and correlations that look nothing alike when drawn.

Summary

Every number in this lesson compresses a column into one value, and every compression throws something away. The mean keeps the total and loses the shape, the median keeps the middle and loses the tails, the standard deviation lets the largest distances speak for everyone, and a correlation keeps one direction and loses the rows behind it. The skill is knowing which part of the column each one has chosen to forget.

The housing table made the point with two listings. They moved the mean, the standard deviation, the range, the skew and Pearson's coefficient, and left the median, the IQR and Spearman's coefficient nearly where the other rows put them. That split is worth carrying forward: a statistic built on positions and ranks asks how many values sit where, one built on sums of distances asks how far, and the second kind belongs to the extreme rows whenever a table has any. Reading a column means computing one of each and looking at the gap.

Correlation adds a second discipline, because its mistakes are made in English rather than in code. A coefficient is a fact about a set of rows, and it says nothing about why the columns moved together, whether something else moved them, or what happens outside the sample. The numbers can be checked by re-running a script, the sentences only by the person writing them. The next lesson adds the other check, drawing the table so that the rows a number forgot are in front of you.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np
import pandas as pd

df = pd.read_csv("housing.csv").drop_duplicates()
p = df["price"].dropna()

p.mean(), p.median()                 # median for a skewed column
df["rooms"].mode()                   # a Series, ties kept
p.std()                              # ddof=1, pandas
np.std(p.to_numpy())                 # ddof=0, NumPy
p.quantile(0.75) - p.quantile(0.25)  # IQR, ignores tails
p.skew(), np.log1p(p).skew()         # tail, then shrunk

df["price"].corr(df["area_sqm"])     # Pearson
pair = df[["price", "area_sqm"]]
pair.corr(method="spearman")         # Spearman, on a DataFrame
df.corr(numeric_only=True)           # matrix, pairwise rows
len(pair.dropna())                   # rows behind the cell