Section 3.7: Is the Result Real?

A model's test score is a mean over the 600 listings that happened to be held out, so it carries the luck of that draw, and so does a gain between two models scored on them. This section measures that luck on the three price models of Sections 3.2 and 3.5, the linear pipeline, the forest and HistGradientBoosting, each fitted once and never refitted here: an interval around a score, an interval around a gain, and a test of whether a gain could be luck. Then it sets a gain on stored rows against what a model does once its prices are used. Two lessons, no install.

Boosting's 3,788 gain over the linear pipeline passes the test alone, its paired interval (built from the two models' errors on the same listings) running from 365 to 7,303 and staying above 0, and counted among the three comparisons three models allow, its p-value of 0.033 (the chance of a gap at least that large if the two models were equally good) becomes 0.099. The forest's 2,760 does not pass even alone, its interval reaching below 0, and both models' separate intervals overlap the linear pipeline's, which is not the test. A gain can also vanish on the way into use. Measured as the share of listings priced within 10 percent of their price, boosting prices 18 more of the 600 test listings within that 10 percent while the forest prices one fewer than the linear pipeline, an online test that shows each listing one model needs thousands of listings, in a simulation drawn from the test rows, to see a gain this size, and a market with more houses moves every model's score with no change to any model.

Lessons

  1. Sampling, Confidence Intervals and Hypothesis Testing - a test score as a sample and its standard error, the bootstrap and a confidence interval, comparing two models on the same rows, what a p-value says and what it does not, and the luck many comparisons buy.
  2. Offline Gains and Online Reality - a gain measured on stored rows, the metric the business feels, an online test in which each listing sees one model, the mix of listings that moves a score, and a model that changes what it is later scored on.

By the end you will be able to:

  • Put a bootstrap interval around a test score and around the gain between two models scored on the same rows.
  • Run a sign-flip test by hand beside scipy.stats.ttest_rel, and say what its p-value does and does not mean.
  • Explain why many comparisons find a winner by luck, and why separate intervals that overlap are not a test.
  • Size an online test for the gain it looks for, and say why an offline gain can shrink or vanish in use.