Section 3.3: Feature Engineering

Section 3.2's pipeline feeds the model the table's columns as they come. A model can use only what its columns carry, in the form the model takes: a linear model adds its columns up with one weight each, so a price that is an area times a rate per square meter fits better once both sides are logarithms, because the logarithm of a product is the sum of the logarithms. Feature engineering builds new columns from the ones the table already has, a listing's age from its build year, square meters per room from two columns, a listing's area against its neighborhood's median, and then decides which columns to keep. The section covers what makes a feature good, building features from dates, pairs of columns and groups, transforms that match how the data was made, and selecting features without letting a test listing's price into the choice.

A feature can carry the answer. A column computed from the price, or one that only exists after the prediction is made, lifts a held-out score that no new listing will ever reach, and nothing raises an error, so the section's rule is that a feature must exist at the moment of prediction and is never computed from the target. More columns are not better either: columns of noise raise a least squares fit's training score and lower its score on held-out rows, more so the fewer rows there are, twin columns can split a weight in a way the data cannot decide, and a selector that scores features against the price learns from the target exactly as a model does, so it is fitted inside the pipeline on the training rows only.

Lessons

  1. What Makes a Feature Good, and Building Features From the Columns You Have - what a feature must carry, the moment of prediction and two leaks shown as leaks, features from dates, pairs of columns and neighborhood medians learned from the training rows, logarithms that match how a price is built, and features made inside the pipeline.
  2. Selection, and Why More Features Is Not Better - columns of noise and the rows they cost, twin columns and the condition number that finds them, SelectKBest as a step fitted on the target, and choosing how many features to keep on a validation set.

By the end you will be able to:

  • Ask of any column when it becomes known and what it is computed from, and refuse one that fails either question.
  • Build features from dates, pairs of columns and group statistics, with every group statistic learned from the training rows only.
  • Choose a transform that matches how the data was made, and fit a model on a logarithm that still predicts a price in currency.
  • Drop or combine twin columns, and fit a feature selector inside the pipeline so it never sees a held-out row.