Module 2
Module 2 - Data: Queries, Arrays, and the Clean Table
You will be able to query a database down to a table that fits in memory, compute over it, and find and fix everything wrong with it.
Every model Module 3 trains learns from a table, and no table arrives ready. This module covers the work between the data's source and that table: querying a database with SQL (Structured Query Language), computing over whole arrays with NumPy, the linear algebra a model is built from, labeled tables with pandas, cleaning, transforming and combining them, and describing what is left in numbers and in charts.
This is the first half of every ML project, and in real work it is the larger half. You pull the rows you need out of a database instead of loading all of it, turn features into arrays a model can multiply, find the listings with no price, the duplicated records and the impossible values, join a lookup table without inflating a single count, and check that the cleaned table still says what the source said. Module 3 trains models on exactly such a table, and the questions it asks of a score, whether the split was honest and whether a feature leaked, start from the table this module teaches you to build.
Without it, a model is trained on whatever the file happened to contain. A join that doubled some rows, a conversion that turned bad values into missing ones without a word, a filter that dropped every row with a gap: none of these raises an error, all of them change the numbers, and a model trained on them reports a confident score for the wrong data. Module 1 gave you a language. This module gives you the data to point it at.
What you will build
One project, the capstone, at the end of the module.
The Dataset Autopsy (PAID, $29, or $14 at the half price), a table pulled out of a database by query, cleaned, and then made to prove it still says what the database said. It starts from an empty directory and its bootstrap writes everything it starts from, so nothing you did or skipped earlier changes where you begin.
The sections
- SQL, Reading and Shaping Data - 3 lessons. SQLite and a first query with filtering and ordering, joins, then aggregation with GROUP BY and CTEs (common table expressions).
- NumPy, The Array - 5 lessons. Installing NumPy and the ndarray, indexing and boolean masking, vectorized operations, broadcasting, then aggregation and the
axisargument. - Linear Algebra, Computed - 5 lessons. Vectors and the dot product, matrices and matrix multiplication, distance and norms, projection and least squares, then rank and collinear columns.
- pandas, Series and DataFrame - 3 lessons. Installing pandas and its two objects, selection and assignment, then reading and writing data, including from a query.
- Data Cleaning - 4 lessons. Missing data, types and conversion, categorical, string and datetime data, then duplicates, tidy structure, outliers and range validation.
- Transformation and Aggregation - 3 lessons. Element-wise transformation, GroupBy with the MultiIndex and reshaping, then combining datasets.
- Describing and Visualizing Data - 2 lessons. Center, spread, shape and correlation, then matplotlib and seaborn.
Prerequisites
Module 1. You are assumed to be able to write and run a Python script, build lists, dictionaries and comprehensions, write functions, read a traceback and handle an exception, and parse a CSV file by hand and a JSON file with the json module. Module 0's terminal, filesystem and git are assumed as well.
What the exam proves
PAID, $29, or $14 at the half price. That you can predict what a query, an array operation or a DataFrame operation returns, catch the step that runs clean and changes the numbers, a join that multiplies rows, a filter that drops missing values, a conversion that turns bad values into missing ones without a word, and decide which of two disagreeing counts to trust. The pass covers the capstone and the exam with every attempt, for the life of the product.
Set up your machine first
Lesson 2.1.1 sends SQL from Python scripts, and four later lessons install libraries into (bootcamp), so before it starts you need the machine Module 0's setup page builds: a Unix terminal (WSL on Windows), uv, Python 3.12.8 inside a virtual environment at ~/dev/bootcamp, and VS Code opened on that folder. Module 2's setup page checks that machine, explains how a library gets into (bootcamp) without pip, says where this module's scripts and data files go, and ends with a checklist that must pass line for line, uv pip list aside. Read it before Lesson 2.1.1 even if your prompt already shows (bootcamp).