Section 3.5: Why One Model Is Not Enough

Every price model so far, Lesson 3.1.2's memorizer aside, has been a line. This section fits models that are not. kNN stores its training rows and does its work when it is asked for a price, and a decision tree left to grow learns its training rows nearly by heart, so one fully grown tree is one draw from its rows: two trees grown on two halves of the same listings disagree by about 96,927 on an average listing. A random forest averages many trees grown to differ, and gradient boosting adds small trees one after another, each fitted to what the sum before it still gets wrong. The section covers kNN and decision trees, random forests and why averaging works, and gradient boosting with XGBoost, the module's second install.

Five traps here print their numbers without a warning. A training R² of 1.0 only says a tree learned its rows, kNN on unscaled columns listens to area and build year and barely hears the neighborhoods, a forest's importances say what its splits used and never what moves a price, boosting's rounds keep raising the training score past the point where the validation score falls, and early stopping (ending the rounds once the score on held-out rows stops improving) on the test rows is choosing by the test score. With each model's settings fixed on validation rows first and the test rows scoring it once, the fully grown tree scores 0.587, the chosen tree 0.673, kNN 0.702, the linear pipeline 0.766, the forest 0.775 and HistGradientBoosting 0.777, so on this market the ensembles barely pass the line, and one split cannot rank them.

Lessons

  1. Decision Trees, and Why They Overfit, carrying the kNN exposure - kNN, the model with no training step, and what scaling does to its distance, how a tree splits and what a leaf predicts, a tree grown until its leaves are pure, depth and leaf size chosen on validation rows, and what a tree cannot do.
  2. Random Forests, and Why Averaging Works - two trees averaged, bootstrap samples, a random subset of features at each split, how many trees, out-of-bag rows and feature importances, and a forest that classifies.
  3. Gradient Boosting, and Why It Is Not Averaging - installing XGBoost, boosting built by hand from residuals, the learning rate against the number of rounds, early stopping on validation rows, and XGBoost at the end of the pipeline.

By the end you will be able to:

  • Fit kNN and a decision tree in a pipeline, and choose k, a depth or a leaf size on validation rows.
  • Explain why averaging trees that differ scores above their average R², by exactly their spread, and fit and read a random forest, its out-of-bag score and its importances.
  • Build boosting by hand from residuals, and trade the learning rate against the number of rounds with early stopping on validation rows.
  • Install XGBoost, fit it at the end of the pipeline, and say what a tree cannot do that a line can.