MLflow - Tracking Experiments

Context

Experiment tracking writes down every training run as it happens: the settings it used, the scores it reached, the model file it produced and the commit of the code that ran, all in one store a script or a browser can query afterwards. MLflow, an open-source tracking library, keeps that store as one SQLite file and a folder beside the project.

Every model a team ships goes through dozens of runs that differ in one setting, an imputer's strategy, an indicator switched off, a new feature, and the question that follows is always the same: which run produced this file, and what did the others score? A tracked run answers it from the record, the same way for a teammate, for a pipeline that retrains overnight and for you a month later.

Lesson 6.1.1's train.py printed a score and wrote models/price_pipeline.joblib, and every run wrote over the last one, so the file on disk never said which settings made it and the score was gone the moment the terminal scrolled. Tracking fixes that, and brings two failures of its own that raise nothing: a query run from the wrong folder opens a fresh, empty store and returns no runs, and a run made before a commit is tagged with a commit that does not hold the code that ran.

Goal

• Install MLflow 3.17.1 into the project's own environment with uv add, and measure what it brings with it.
• Turn train.py into a tracked run that logs its parameters, its two test scores and its fitted pipeline, and make three runs that differ in one setting each.
• Read every line MLflow prints on a run, and separate the lines that come from the store, from the experiment and from the model's environment.
• Trace where each part of a run lands, mlflow.db or mlruns/, keep both out of Git, and reproduce the empty store a query opens from the wrong folder.
• Compare the three runs with mlflow.search_runs, by sort, by filter and by pandas, and identify why a run's commit tag can name code that never ran.
• Open the three runs in the MLflow UI, compare them side by side, and stop the server.

Prerequisites

Neither is required, and the lesson stands on its own without them. They are here for anyone who wants to watch tracking done once, or read MLflow's own vocabulary, before the first run.

• YouTube - "MLFlow Tutorial | ML Ops Tutorial" by codebasics, from its chapter "Experiment Tracking in MLFlow" at 08:57. A training script wrapped in a run, its settings and scores logged, and the runs compared in the MLflow UI. The chapters after it go on to the model registry and a shared tracking server.
• MLflow documentation - "MLflow Tracking", the section "Concepts". What MLflow calls a run, a model and an experiment, and what each one records.

Table of Contents

• MLflow in the Project
• A Run, Its Parameters and Its Metrics
• What MLflow Prints on Every Run
• The Tracking Store
• Comparing Runs With search_runs
• The MLflow UI

Theory

Every command runs in ~/dev/listings-service, the project as Lesson 6.1.3 left it, with its 17 tests passing, and (bootcamp) deactivated as Module 6's setup page says. Every run gets a run id, 32 random hexadecimal characters, and a random two-word name, every logged model an id of its own, so yours differ from the ones quoted here, and so do the timestamps and the seconds. Everything else, the scores, the counts and the file names that are not ids, comes out the same.

MLflow in the Project

What. MLflow is an open-source library for the machine learning lifecycle, and this lesson uses its tracking part: a Python API that records runs, a store that keeps them, and a web UI that shows them. It is installed into the project's own environment with uv add, pinned to 3.17.1, so pyproject.toml and uv.lock carry it like every other dependency. Why. A tracking library that lives in the project's lock is the same version on every machine that runs uv sync, and it records runs made with the project's own interpreter and libraries, the ones the model was actually fitted with. When. Once per project, before the first run you want recorded.

Install MLflow into the project:

cd ~/dev/listings-service
uv add mlflow==3.17.1

The first six lines, and the three that name MLflow itself, of the 83 it prints:

Resolved 96 packages in 110ms
   Building listings @ file:///home/goku/dev/listings-service
      Built listings @ file:///home/goku/dev/listings-service
Prepared 1 package in 20ms
Uninstalled 1 package in 0.65ms
Installed 77 packages in 93ms
 + mlflow==3.17.1
 + mlflow-skinny==3.17.1
 + mlflow-tracing==3.17.1

The lock grew from 18 packages to 96. Of the 77 installed, 76 are new and one is the project itself, listings, rebuilt and reinstalled, which is the one package uninstalled. The Prepared count and the timings depend on what uv's cache already holds, so a first install on your machine downloads more and takes longer. MLflow pulls in a web stack, fastapi, uvicorn, flask and gunicorn, a database layer, sqlalchemy and alembic, pyarrow, matplotlib, and skops, which the second subtitle meets. uv add also wrote "mlflow==3.17.1", into the dependencies list of pyproject.toml, so the pin travels with the project.

uv run mlflow --version
du -sh .venv
du -sh .venv/lib/python3.12/site-packages/pyarrow

mlflow, version 3.17.1
677M	.venv
152M	.venv/lib/python3.12/site-packages/pyarrow

uv run mlflow runs the command-line program MLflow installed into .venv/bin, the same way uv run pytest runs pytest. The environment is now 677 MB, and pyarrow, a columnar data library MLflow reads tables with, is 152 MB of it on its own.

DON'T:

• DON'T install MLflow into (bootcamp) or with a bare pip install. The project's lock is what makes the next machine's MLflow the same one, and a library installed beside it is in no lock at all.
• DON'T uv add mlflow without the version. It resolves whatever is newest the day it runs, and exclude-newer in pyproject.toml only caps that date, it does not choose 3.17.1.

A Run, Its Parameters and Its Metrics

What. A run is one execution of training code that MLflow records, and an experiment is the named group of runs it belongs to. Inside a run, mlflow.log_params stores the settings the run used, as text, mlflow.log_metrics stores its scores, as numbers, and mlflow.sklearn.log_model saves the fitted pipeline as the run's model, together with the files that describe the environment it was fitted in. Why. A score is only worth something beside the settings that produced it and the file it belongs to. Logged from inside the training code, the three are written together by the run itself, and none of them depends on someone copying a number into a notebook. When. In every training script whose runs you will ever compare, and from the first run, because a run that was not tracked cannot be tracked afterwards.

build_pipeline gains two parameters, so a run can change the imputer's strategy and switch the indicator off without editing code. src/listings/features.py:

"""The preprocessing and the model, as one unfitted pipeline."""

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler

from listings.data import LEVELS, NOMINAL, NUMERIC


def build_pipeline(strategy="median", indicator=True):
    numeric_steps = Pipeline([
        ("impute", SimpleImputer(strategy=strategy, add_indicator=indicator)),
        ("scale", StandardScaler()),
    ])
    prep = ColumnTransformer([
        ("num", numeric_steps, NUMERIC),
        ("cat", OneHotEncoder(sparse_output=False), NOMINAL),
        ("cond", OrdinalEncoder(categories=[LEVELS]), ["condition"]),
    ])
    return Pipeline([("prep", prep), ("model", LinearRegression())])

The defaults are Lesson 3.2.3's "median" and add_indicator=True, so build_pipeline() with no arguments builds exactly the pipeline Lesson 6.1.1 restated, and every test that calls it still builds that one. src/listings/train.py keeps evaluate and replaces main:

"""Fit the price pipeline, score it, and record the run in MLflow."""

import argparse

import mlflow
from sklearn.metrics import mean_absolute_error, r2_score

from listings.data import load, split
from listings.features import build_pipeline


def evaluate(pipe, X, y):
    predicted = pipe.predict(X)
    return {
        "mae": mean_absolute_error(y, predicted),
        "r2": r2_score(y, predicted),
    }


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--strategy", default="median")
    parser.add_argument("--no-indicator", action="store_true")
    args = parser.parse_args()
    params = {"strategy": args.strategy, "indicator": not args.no_indicator}

    X_train, X_test, y_train, y_test = split(load())
    mlflow.set_experiment("listings-price")
    with mlflow.start_run() as run:
        mlflow.log_params(params)
        pipe = build_pipeline(**params).fit(X_train, y_train)
        scores = evaluate(pipe, X_test, y_test)
        mlflow.log_metrics(scores)
        mlflow.sklearn.log_model(
            pipe, name="model", skops_trusted_types=["numpy.dtype"]
        )
    print(run.info.run_id, f"{scores['mae']:,.0f}", round(scores["r2"], 3))


if __name__ == "__main__":
    main()

This file is the project's own, and it carries no comments, so it reads the same in the lesson as in the repository, and the walkthrough is here instead. argparse is the standard library's command-line parser: add_argument("--strategy", default="median") accepts --strategy mean and falls back to "median", action="store_true" makes --no-indicator a switch that is False until it is typed, and parse_args() reads what the command line carried into args, with the dash in no-indicator turned into an underscore. params is the one dictionary both the pipeline and the log are built from, so the settings recorded are the settings used.

mlflow.set_experiment("listings-price") makes that experiment the active one, creating it in the store the first time. mlflow.start_run() opens a run inside it, and the with block ends the run as it leaves: FINISHED when the block completes, FAILED when an exception escapes it. log_params and log_metrics each take a dictionary, and evaluate already returns one, so its keys mae and r2 become the metric names. run.info.run_id is the run's id, still readable after the block has closed the run.

log_model saves the fitted pipeline under the name model. MLflow 3.17.1 writes a scikit-learn model with skops, a file format for scikit-learn models that records every type a file holds and refuses to load a type it was not told to trust, where Lesson 3.2.3's joblib.load runs whatever its file's maker put in it. MLflow loads the file back right after writing it, as a check, and this pipeline's file holds numpy.dtype objects, the dtypes its SimpleImputer records about the columns it fitted on, so skops_trusted_types=["numpy.dtype"] names that type as trusted. Without it the run stops at log_model, and the last line of the traceback says why:

mlflow.exceptions.MlflowException: The saved sklearn model references untrusted types. If you are sure loading these types is safe, set the 'skops_trusted_types' parameter when calling 'log_model' or 'save_model' to the list of trusted types. Root error: Untrusted types found in the file: ['numpy.dtype'].

The run it leaves behind is FAILED, with its parameters and its MAE already in the store, because both were logged before log_model raised. The first tracked run, with every default:

uv run python -m listings.train

2026/11/03 13:26:27 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/11/03 13:26:27 INFO mlflow.store.db.utils: Updating database tables
2026/11/03 13:26:28 INFO mlflow.tracking.fluent: Experiment with name 'listings-price' does not exist. Creating a new experiment.
2026/11/03 13:26:29 INFO mlflow.utils.uv_utils: Detected uv project: found uv.lock and pyproject.toml in /home/goku/dev/listings-service
2026/11/03 13:26:30 INFO mlflow.utils.uv_utils: Detected uv project: found uv.lock and pyproject.toml in /home/goku/dev/listings-service
2026/11/03 13:26:30 INFO mlflow.utils.environment: Detected uv project at /home/goku/dev/listings-service. Attempting to export requirements via 'uv export'.
2026/11/03 13:26:30 INFO mlflow.utils.uv_utils: Exported 90 dependencies via uv
2026/11/03 13:26:30 INFO mlflow.utils.environment: Successfully exported 90 requirements from uv project. Skipping package capture based inference.
2026/11/03 13:26:30 WARNING mlflow.utils.environment: Failed to resolve installed pip version. ``pip`` will be added to conda.yaml environment spec without a version specifier.
15a2ce771186453791de0a7de8df9af3 56,562 0.766

The last line is train.py's own: the run id, then Lesson 3.2.3's 56,562 0.766, the MAE and R² of the same pipeline on the same 600 test listings. The nine lines above it are MLflow's, and the next subtitle reads them. The project's 17 tests still pass after this change, because none of them calls main.

DON'T:

• DON'T log a setting from a different variable than the one the model was built from. params feeds both log_params and build_pipeline(**params), so the record cannot drift from the run.
• DON'T drop skops_trusted_types=["numpy.dtype"] to quiet the call. The run stops at log_model with Untrusted types found in the file: ['numpy.dtype'] and stays in the store as FAILED.
• DON'T read FAILED as a run that left nothing. Everything logged before the exception, the parameters and the MAE here, is in the store beside the finished runs.

What MLflow Prints on Every Run

What. Every line MLflow prints opens with a date and time, a level, INFO or WARNING, and the name of the MLflow module that printed it, then the message. The first run in a new store prints three lines about the store and the experiment, and every run that logs a model prints six about the model's environment: two that find the uv project, three about exporting its requirements, and one warning about pip. Why. A line that changes between runs is information, and a line that never changes is noise only once you know what it says. Read once, the six model lines tell you that the model carries a frozen copy of the project's dependencies, and the warning tells you something about your environment rather than about your model. When. On the first run of every new project, and whenever a run prints a line you have not seen before.

The three lines only the first run printed come from the store and the experiment. Creating initial MLflow database tables... and Updating database tables are MLflow building mlflow.db, its database, from nothing and bringing its layout to the current version, and Experiment with name 'listings-price' does not exist. Creating a new experiment. is set_experiment making the experiment it was asked for. A second run finds both already there:

uv run python -m listings.train --no-indicator

2026/11/03 13:26:36 INFO mlflow.utils.uv_utils: Detected uv project: found uv.lock and pyproject.toml in /home/goku/dev/listings-service
2026/11/03 13:26:36 INFO mlflow.utils.uv_utils: Detected uv project: found uv.lock and pyproject.toml in /home/goku/dev/listings-service
2026/11/03 13:26:36 INFO mlflow.utils.environment: Detected uv project at /home/goku/dev/listings-service. Attempting to export requirements via 'uv export'.
2026/11/03 13:26:36 INFO mlflow.utils.uv_utils: Exported 90 dependencies via uv
2026/11/03 13:26:36 INFO mlflow.utils.environment: Successfully exported 90 requirements from uv project. Skipping package capture based inference.
2026/11/03 13:26:37 WARNING mlflow.utils.environment: Failed to resolve installed pip version. ``pip`` will be added to conda.yaml environment spec without a version specifier.
d2c87fb04cd4462c82fbc956a4d9504c 56,336 0.766

All six come from log_model. MLflow saves with every model a list of the packages needed to load it, and when it finds uv.lock and pyproject.toml it builds that list by running uv export on the project, without the dev group, so pytest is not among the 90 pins: Skipping package capture based inference means it took the lock's exact versions instead of guessing them from what is imported. The warning is about pip: uv built .venv without pip, so MLflow cannot read its version and lists pip unversioned in one of the environment files it writes, conda.yaml. The run still ends FINISHED with its model saved. Without the indicator the MAE is 56,336, 226 lower, and R² rounds to the same 0.766.

Every one of those lines goes to standard error and only train.py's print goes to standard output, so Lesson 0.2.2's 2>/dev/null keeps the result and drops the rest. The third run, with the mean as the fill:

uv run python -m listings.train --strategy mean 2>/dev/null

287fa7dbcc224046beeb92c68cf08d4e 56,562 0.766

The mean gives the median's 56,562 and 0.766, and the fifth subtitle shows how close the two runs are. Each command took 6 to 12 seconds here, the first run of a fresh environment the slowest, about 3 of them spent importing MLflow, scikit-learn and pandas before the run starts, which is why the UI later reports each run's own duration at under two seconds.

DON'T:

• DON'T silence a run with 2>/dev/null while it is still new to you. A traceback goes to standard error too, and the FAILED run of the second subtitle printed its reason there and nowhere else.
• DON'T read the pip warning as a failed run. The run ended FINISHED with its model saved, and the warning says only that .venv holds no pip.
• DON'T expect the three store lines on every run. They print once per new mlflow.db, so a run that prints them again has opened a store that did not exist a moment ago.

The Tracking Store

What. The tracking store is where MLflow keeps everything a run records, in two places. The metadata, experiments, runs, parameters, metrics, tags and the list of logged models, goes into mlflow.db, a SQLite database, Lesson 2.1.1's kind of single-file database. The files, each logged model's folder, go into mlruns/. MLflow 3.17.1 uses sqlite:///mlflow.db when nothing says otherwise, and the path in that address is relative, read against the folder the command runs in. Why. Knowing where a run lands is how you keep it out of Git, back it up, find a model's files by hand and notice when a script has written to a different store. A store is a file, and a file opened from the wrong folder is a different file. When. After the first run of a project, when deciding what Git should ignore, and whenever a query returns fewer runs than you made.

git status --short
du -sh mlflow.db mlruns
find mlruns -type f | sort

 M pyproject.toml
 M src/listings/features.py
 M src/listings/train.py
 M uv.lock
?? mlflow.db
932K	mlflow.db
600K	mlruns
mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts/MLmodel
mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts/conda.yaml
mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts/model.skops
mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts/python_env.yaml
mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts/requirements.txt
mlruns/1/models/m-b9b98e0ba1434e6f9ddc4f3d72507c67/artifacts/MLmodel
mlruns/1/models/m-b9b98e0ba1434e6f9ddc4f3d72507c67/artifacts/conda.yaml
mlruns/1/models/m-b9b98e0ba1434e6f9ddc4f3d72507c67/artifacts/model.skops
mlruns/1/models/m-b9b98e0ba1434e6f9ddc4f3d72507c67/artifacts/python_env.yaml
mlruns/1/models/m-b9b98e0ba1434e6f9ddc4f3d72507c67/artifacts/requirements.txt
mlruns/1/models/m-ef0729028fb54385ba99d1bc0d2bd610/artifacts/MLmodel
mlruns/1/models/m-ef0729028fb54385ba99d1bc0d2bd610/artifacts/conda.yaml
mlruns/1/models/m-ef0729028fb54385ba99d1bc0d2bd610/artifacts/model.skops
mlruns/1/models/m-ef0729028fb54385ba99d1bc0d2bd610/artifacts/python_env.yaml
mlruns/1/models/m-ef0729028fb54385ba99d1bc0d2bd610/artifacts/requirements.txt

1 is the experiment id of listings-price, and each m- folder is one logged model with its own id, one per run. model.skops is the pipeline itself, 188,680 bytes, MLmodel describes it, and requirements.txt holds the 90 pins uv export produced, with python_env.yaml and conda.yaml describing the same environment for two other tools. Git lists mlflow.db as untracked and does not list mlruns/ at all, because the .gitignore from Lesson 6.1.1 ignores models/, and a pattern ending in a slash matches a folder of that name at any depth, mlruns/1/models/ included, which leaves mlruns/ with nothing Git would show. Each MLmodel names the run that made it:

grep "^run_id:" mlruns/1/models/*/artifacts/MLmodel

mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts/MLmodel:run_id: 15a2ce771186453791de0a7de8df9af3
mlruns/1/models/m-b9b98e0ba1434e6f9ddc4f3d72507c67/artifacts/MLmodel:run_id: d2c87fb04cd4462c82fbc956a4d9504c
mlruns/1/models/m-ef0729028fb54385ba99d1bc0d2bd610/artifacts/MLmodel:run_id: 287fa7dbcc224046beeb92c68cf08d4e

Both the database and the model files are produced by running the code, and both are large or binary, so neither belongs in Git. The last block of .gitignore becomes:

# What the code produces, rebuilt by running it
mlflow.db
mlruns/

train.py no longer writes models/, and the old pipeline file is still there from Lesson 6.1.1, no longer ignored:

git status --short
rm -r models
git status --short

 M .gitignore
 M pyproject.toml
 M src/listings/features.py
 M src/listings/train.py
 M uv.lock
?? models/
 M .gitignore
 M pyproject.toml
 M src/listings/features.py
 M src/listings/train.py
 M uv.lock

Removing it leaves only the five files this lesson changed. Now the failure the relative path makes. A query run from tests/:

cd tests
uv run python -c 'import mlflow; print(mlflow.search_runs(experiment_names=["listings-price"]))'
ls mlflow.db

2026/11/03 13:26:47 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/11/03 13:26:47 INFO mlflow.store.db.utils: Updating database tables
2026/11/03 13:26:48 WARNING mlflow.tracking.fluent: Cannot retrieve experiment by name listings-price
Empty DataFrame
Columns: [run_id, experiment_id, status, artifact_uri, start_time, end_time]
Index: []
mlflow.db

Nothing raised. MLflow read sqlite:///mlflow.db against tests/, found no file there, created one with empty tables, the two store lines the first run printed, and searched it, so the three runs exist and the query cannot see them: the failure Lesson 2.1.1 met when sqlite3.connect created an empty database for a misspelled file name. The MLFLOW_TRACKING_URI environment variable overrides the default store, and with four slashes after sqlite: the path is absolute, so the same query finds the runs from any folder:

rm mlflow.db
MLFLOW_TRACKING_URI=sqlite:////home/goku/dev/listings-service/mlflow.db uv run python -c 'import mlflow; print(len(mlflow.search_runs(experiment_names=["listings-price"])))'
cd ..

3

rm mlflow.db deletes the empty store from tests/ before anything else uses it. Three slashes after sqlite: are the address's own, and a fourth starts the absolute path /home/goku/.... The store records absolute paths of its own too: the experiment row holds /home/goku/dev/listings-service/mlruns/1 as the place its models go.

DON'T:

• DON'T run an MLflow command from a folder other than the project's and trust an empty answer. From tests/ the query created a second mlflow.db and returned no runs, with one WARNING and no error.
• DON'T commit mlflow.db or mlruns/. Both are rebuilt by running the code, and the model files are binary.
• DON'T assume a .gitignore pattern names only the folder you had in mind. models/ hid mlruns/1/models/ from git status until the patterns were changed.

Comparing Runs With search_runs

What. mlflow.search_runs(experiment_names=[...]) returns the runs of the named experiments as a pandas DataFrame, one row per run, with a column for each parameter under params., each metric under metrics. and each tag under tags.. order_by sorts the rows in the store and filter_string selects them there, in MLflow's own filter language. Among the tags MLflow sets by itself is mlflow.source.git.commit, the commit Git's HEAD pointed at when the run started. Why. Three runs printed in a terminal are three lines that scroll away. A DataFrame of them can be sorted, filtered and compared with everything Module 2 taught, and it is the same answer for anyone who opens the same store. When. Whenever a question is about more than one run: which setting scored best, which runs used a given setting, which commit a run came from.

Open Python inside the project's environment with uv run python, from the project folder, and type:

import mlflow

runs = mlflow.search_runs(experiment_names=["listings-price"], order_by=["metrics.mae ASC"])
print(runs.shape)
print(runs[["params.strategy", "params.indicator", "metrics.mae", "metrics.r2"]])
print(runs["params.indicator"].tolist())
print(runs["metrics.mae"].tolist())

(3, 16)
  params.strategy params.indicator   metrics.mae  metrics.r2
0          median            False  56335.628539    0.765626
1          median             True  56561.580011    0.766383
2            mean             True  56561.580011    0.766383
['False', 'True', 'True']
[56335.62853941725, 56561.58001063279, 56561.580010632795]

The frame has 16 columns: the run's id, experiment, status, artifact address, start and end times, the two metrics, the two parameters and six tags. Metrics come back as floats, sorted by order_by from the lowest MAE. Parameters come back as strings, 'True' and 'False', because log_params stores every value as text. The mean and the median runs agree to the eleventh decimal place, 56561.58001063279 against 56561.580010632795, so with the indicator on, the strategy changed nothing a metric can see, as Lesson 6.1.3 found on the predictions themselves. The string type matters the moment a parameter is compared:

print(len(runs[runs["params.indicator"] == True]), len(runs[runs["params.indicator"] == "True"]))
cheap = mlflow.search_runs(experiment_names=["listings-price"], filter_string="params.indicator = 'False' and metrics.mae < 56500")
print(cheap[["params.strategy", "metrics.mae"]])

0 2
  params.strategy   metrics.mae
0          median  56335.628539

Compared with the boolean True the column matches nothing and pandas raises nothing, and compared with the string 'True' it matches the two runs that kept the indicator. filter_string follows the same rule in MLflow's language: a parameter is compared with a quoted string, a metric with a number, and conditions join with and. The no-indicator run is 226 lower on MAE, and that MAE is measured on the test rows, so taking it as the reason to switch off the indicator would be choosing by the test score, the break Lesson 3.1.2 named. The record states what each run scored, and the project keeps its default. The last tag worth reading now:

print(runs["tags.mlflow.source.git.commit"].unique().tolist())

['261ff9ea0db5722eff5cfb0ba2b6f6e6c09c9661']

Type exit() to leave Python, and set that beside Git:

git log -1 --format=%H
git status --short

261ff9ea0db5722eff5cfb0ba2b6f6e6c09c9661
 M .gitignore
 M pyproject.toml
 M src/listings/features.py
 M src/listings/train.py
 M uv.lock

Your hash is your own Lesson 6.1.3 commit, and it matches between the two the same way. All three runs carry the commit HEAD points at, Lesson 6.1.3's, and that commit's train.py has no argparse, no MLflow and no --no-indicator: the code that made the runs was uncommitted, and the tag cannot say so. Commit the lesson's work, so every run from here on names a commit that holds the code it ran:

git add -A
git commit -m "feat: track training runs in MLflow"

[main cf46890] feat: track training runs in MLflow
 5 files changed, 2422 insertions(+), 13 deletions(-)

The five files are .gitignore, pyproject.toml, uv.lock, features.py and train.py, and mlflow.db and mlruns/ stay out under the new patterns. A run made after this commit is tagged with its full hash, cf46890... here and your own commit's on your machine, while the three runs above keep the old one, because a tag is written once, when its run starts.

DON'T:

• DON'T compare a parameter column with a Python boolean or number. runs["params.indicator"] == True matched none of the three runs, without an error.
• DON'T choose a setting by the logged MAE. It is the test score, and acting on the 226 the no-indicator run gained is a choice made on the test rows.
• DON'T trust a run's commit tag for a run made with uncommitted changes. All three runs here name Lesson 6.1.3's commit, whose train.py cannot make any of them.

The MLflow UI

What. mlflow ui starts a local web server that shows the store in a browser: experiments, their runs in a table, each run's parameters, metrics, tags and model, and a page that compares selected runs side by side. It reads the same sqlite:///mlflow.db as the scripts, relative to the folder it starts in, and listens on http://127.0.0.1:5000, the loopback address Lesson 0.2.3 taught, unless told otherwise. Why. A table and a plot of many runs at once is faster to read than a DataFrame printed in a terminal, and the compare page puts the parameters that differ next to the metrics they moved. When. When reading runs rather than querying them, and when showing them to someone at the same machine.

uv run mlflow ui

Backend store URI not provided. Using sqlite:///mlflow.db
Registry store URI not provided. Using backend store URI.
[MLflow] Security middleware enabled with default settings (localhost-only). To allow connections from other hosts, use --host 0.0.0.0 and configure --allowed-hosts and --cors-allowed-origins.
2026/11/03 13:28:49 INFO mlflow.server.jobs.executor_registry: Available backends after discovery: local
2026/11/03 13:28:49 INFO mlflow.server.jobs.executor_registry: Job executor backend 'local' passed requirements check
2026/11/03 13:28:49 INFO:     Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)
2026/11/03 13:28:49 INFO:     Started parent process [132648]
/home/goku/dev/listings-service/.venv/lib/python3.12/site-packages/mlflow/server/fastapi_app.py:23: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
  from starlette.middleware.wsgi import WSGIResponder, build_environ
/home/goku/dev/listings-service/.venv/lib/python3.12/site-packages/mlflow/server/fastapi_app.py:23: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
  from starlette.middleware.wsgi import WSGIResponder, build_environ
/home/goku/dev/listings-service/.venv/lib/python3.12/site-packages/mlflow/server/fastapi_app.py:23: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
  from starlette.middleware.wsgi import WSGIResponder, build_environ
/home/goku/dev/listings-service/.venv/lib/python3.12/site-packages/mlflow/server/fastapi_app.py:23: StarletteDeprecationWarning: starlette.middleware.wsgi is deprecated and will be removed in a future release. Please refer to https://github.com/abersheeran/a2wsgi as a replacement.
  from starlette.middleware.wsgi import WSGIResponder, build_environ
2026/11/03 13:28:53 INFO:     Started server process [132668]
2026/11/03 13:28:53 INFO:     Waiting for application startup.
2026/11/03 13:28:53 INFO:     Started server process [132670]
2026/11/03 13:28:53 INFO:     Waiting for application startup.
2026/11/03 13:28:53 INFO:     Started server process [132671]
2026/11/03 13:28:53 INFO:     Waiting for application startup.
2026/11/03 13:28:53 INFO:     Started server process [132669]
2026/11/03 13:28:53 INFO:     Waiting for application startup.
2026/11/03 13:28:53 INFO:     Application startup complete.
2026/11/03 13:28:53 INFO:     Application startup complete.
2026/11/03 13:28:53 INFO:     Application startup complete.
2026/11/03 13:28:53 INFO:     Application startup complete.

The first two lines say it found no store named on its command line and opened sqlite:///mlflow.db, the scripts' default. The security line says it answers only this machine. Uvicorn, the server program MLflow runs on, starts a parent process and four worker processes, and each worker prints the same StarletteDeprecationWarning, raised inside MLflow's own fastapi_app.py about a part of Starlette, the web toolkit under FastAPI, that it still uses: a message for MLflow's developers, not for this project. The server takes requests once the four Application startup complete. lines are in, about 7 seconds after the command here, and a request sent before that is refused. The terminal stays busy while it runs, so any other command goes in a second terminal, or waits until the server is stopped.

Open http://127.0.0.1:5000 in a browser. The home page lists listings-price and Default, the experiment every store starts with, under Recent Experiments, and a switch at the top left reads GenAI and Model training. Choose Model training, then listings-price, and the Runs page lists the three runs by their names, with a Duration of a second or two each, a Source of train.py and a link to each run's model. Clicking a run's name opens its page: two metrics, two parameters, one logged model, and under About this run the user goku, the source train.py, the branch main and the commit 261ff9e, the same stale tag the DataFrame showed. Tick the box above the table's first column to select all three and press Compare: the page lists the parameters and metrics of the three runs side by side, indicator True, False, True and mae 56561.6, 56335.6, 56561.6, under a parallel coordinates plot, a chart with one vertical axis per parameter and metric and one line per run through them. The search box above the Runs table takes filter_string's language: params.indicator = 'False' leaves 1 matching run.

Press Ctrl+C in the terminal that runs the server. Uvicorn prints Shutting down for each worker, an Exception ignored in block from Python's threading module ending in KeyboardInterrupt, and Stopping parent process last, and port 5000 is free again. Nothing in the store changes: the UI only reads it here.

DON'T:

• DON'T start mlflow ui from another folder. It opens sqlite:///mlflow.db relative to where it starts, so from tests/ it creates an empty store there and shows no runs, the failure the fourth subtitle's query met.
• DON'T pass --host 0.0.0.0 to let another machine in. The store's runs and model files would answer anyone who can reach the port, and the security line names the extra settings that address needs.
• DON'T read a run's Duration as the command's time. The UI timed 1.4 to 1.8 seconds inside each run, and each command took 6 to 12 seconds with the imports.

Lab Exercises

Both labs and the Assignment write their script in Lesson 6.1.3's probe folder, ~/dev/listings-probes, and run it from the project folder as uv run python ../listings-probes/<name>.py, so it gets the project's environment and the project's mlflow.db, and the project leaves this lesson exactly as the Theory left it, with its three runs.

Lab Exercise 1: The Store Without MLflow

mlflow.db is an ordinary SQLite file. In store.py, open it with Lesson 2.1.1's sqlite3, not with MLflow. Print how many tables MLflow created, then each experiment's id, name and artifact_location from the experiments table. For each run in the runs table, oldest first by start_time, print its name and status, its parameters as a dictionary built from the params table and its MAE rounded to the cent from the metrics table, each read through a ? placeholder on the run's id, run_uuid. Finally, join the logged_models table to runs on logged_models.source_run_id = runs.run_uuid, Lesson 2.1.2's join, and print how many models it finds and the first model id with its run's name.

Lab Exercise 2: From a Model Folder to Its Run

A teammate hands you a model folder, mlruns/1/models/m-.../artifacts, and asks which run made it. Write which_run.py, which takes that folder as its one command-line argument through sys.argv, Lesson 1.7.2's, and reads two files in it with open: from MLmodel the lines that start with run_id and model_size_bytes, split at ": " into a dictionary, and from requirements.txt the line pinning scikit-learn. Then find the run with mlflow.search_runs and pandas, and print its name, its strategy and indicator, its MAE to the cent with its commit's first seven characters, and how many runs in the experiment share that MAE to the cent, with their strategies. Run it on the model of your default run, and once more on the folder one level up, without artifacts.

import sqlite3

con = sqlite3.connect("mlflow.db")
tables = con.execute("SELECT name FROM sqlite_master WHERE type = 'table'").fetchall()
print(len(tables))
for row in con.execute("SELECT experiment_id, name, artifact_location FROM experiments"):
    print(row)
runs = con.execute("SELECT run_uuid, name, status FROM runs ORDER BY start_time").fetchall()
for run_id, name, status in runs:
    params = dict(con.execute("SELECT key, value FROM params WHERE run_uuid = ?", (run_id,)).fetchall())
    metrics = dict(con.execute("SELECT key, value FROM metrics WHERE run_uuid = ?", (run_id,)).fetchall())
    print(name, status, params, round(metrics["mae"], 2))
models = con.execute(
    "SELECT logged_models.model_id, runs.name FROM logged_models JOIN runs ON logged_models.source_run_id = runs.run_uuid"
).fetchall()
print(len(models), models[0])
con.close()

uv run python ../listings-probes/store.py

65
(0, 'Default', '/home/goku/dev/listings-service/mlruns/0')
(1, 'listings-price', '/home/goku/dev/listings-service/mlruns/1')
awesome-mouse-211 FINISHED {'strategy': 'median', 'indicator': 'True'} 56561.58
debonair-crab-622 FINISHED {'strategy': 'median', 'indicator': 'False'} 56335.63
salty-loon-837 FINISHED {'strategy': 'mean', 'indicator': 'True'} 56561.58
3 ('m-56dbdc0d5261430385d6a3ad272a98ef', 'awesome-mouse-211')

import sys

import mlflow

folder = sys.argv[1]
recorded = {}
with open(f"{folder}/MLmodel") as f:
    for line in f.read().splitlines():
        if line.startswith(("run_id", "model_size_bytes")):
            key, value = line.split(": ")
            recorded[key] = value
with open(f"{folder}/requirements.txt") as f:
    pins = [line for line in f.read().splitlines() if line.startswith("scikit-learn")]
print(recorded, pins)

runs = mlflow.search_runs(experiment_names=["listings-price"])
run = runs[runs["run_id"] == recorded["run_id"]].iloc[0]
print(run["tags.mlflow.runName"], run["params.strategy"], run["params.indicator"])
print(f"{run['metrics.mae']:,.2f}", run["tags.mlflow.source.git.commit"][:7])
twins = runs[runs["metrics.mae"].round(2) == round(run["metrics.mae"], 2)]
print(len(twins), sorted(twins["params.strategy"]))

uv run python ../listings-probes/which_run.py mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/artifacts
uv run python ../listings-probes/which_run.py mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef

{'model_size_bytes': '188680', 'run_id': '15a2ce771186453791de0a7de8df9af3'} ['scikit-learn==1.9.1']
awesome-mouse-211 median True
56,561.58 261ff9e
2 ['mean', 'median']

Traceback (most recent call last):
  File "/home/goku/dev/listings-service/../listings-probes/which_run.py", line 7, in <module>
    with open(f"{folder}/MLmodel") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: 'mlruns/1/models/m-56dbdc0d5261430385d6a3ad272a98ef/MLmodel'

Theory Assessment

Q1: Where One Run Went

Your default run printed one line of its own and recorded much more than that line. Say where its parameters, its metrics and its fitted pipeline each landed, what the 90 pins in its model folder are and where they came from, and which file connects that folder to the run.

Hint: Split what the run recorded into what goes into a database and what goes into a folder of files, then open the file in the model folder that describes the model.

Q2: Nine Lines, Then Six

The first run printed nine lines from MLflow and the second printed six. Say which three only the first printed and what made each of them, which call printed the six every run prints and what they did, why the WARNING among them does not mean the run failed, and why 2>/dev/null kept the run's result line.

Hint: Ask what existed before the first run and what existed before the second, and which streams the two kinds of line were written to.

Q3: A Query That Finds Nothing

A teammate opens Python in tests/ to check the runs and gets an empty DataFrame. Back in the project folder, they keep only the runs that used the indicator with runs[runs["params.indicator"] == True] and get zero rows again. Say what each step did instead of what they expected, why neither raised an error, and one fix for each.

Hint: One problem is about which file the address sqlite:///mlflow.db names, and the other is about the type log_params stores.

Q4: Rebuilding the Best Run

A teammate sorts the three runs by metrics.mae, decides the no-indicator run is the better model because its MAE is 226 lower, opens its page in the UI, reads its commit, 261ff9e, and to rebuild it clones the repository, checks out that commit and runs uv run python -m listings.train --no-indicator. That commit's train.py is Lesson 6.1.1's, with no argparse and no MLflow. Say what the command does and prints, why the commit tag sent them there, what is wrong with the decision itself, and what would have let the tag be trusted.

Hint: Read main as Lesson 6.1.1 wrote it and ask what it does with arguments it never reads. Then ask which rows the 226 was measured on.

Assignment

Assignment: An Imputer Grid in Its Own Store

Lesson 6.1.3 found that with the indicator on, no imputer strategy moves a prediction, and measured one strategy with it off, the most common value, at an MAE of 61,504. The project's three runs hold two corners of that grid. Record all six, three strategies against the indicator on and off, as tracked runs a query can read back, and without adding a single run to the project's store.

Two things this lesson did not use. mlflow.set_tracking_uri("sqlite:///../listings-probes/sweep.db"), called before any other MLflow call, points the script at a store of its own, the same override as MLFLOW_TRACKING_URI set from inside Python, its path read against the folder the command runs in, so the store lands beside the script. And mlflow.start_run(nested=True), opened inside another run, makes a child run of it, and MLflow tags each child with mlflow.parentRunId, the parent's run id.

Write ../listings-probes/sweep.py. Open one parent run named impute-grid in an experiment listings-sweep, and inside it one child run per combination of "mean", "median" and "most_frequent" with the indicator on and off, each logging its two parameters and its two metrics, without a model, by the project's own build_pipeline and evaluate. Print how many runs the experiment holds and how many are children of the parent, found by a filter_string on the parent tag, then a table of MAE with the strategies as rows and the indicator as columns, rounded to the cent. Then, with the indicator off, print each strategy's fill values for area_sqm and year_built, read from the fitted imputer. End with a comment saying which setting moves the score, which strategy is best without the indicator and why the most common value is the worst, and what the grid may be used for. Confirm that the project's store still holds three runs.

Hints:

• filter_string reads a tag as tags. and its name, compared with a quoted string, and the parent's id is parent.info.run_id from with mlflow.start_run(run_name="impute-grid") as parent:.
• DataFrame.pivot_table(index=..., columns=..., values=...), Lesson 2.6.2's, turns the six children into a three-by-two table, and Lesson 3.2.3's path pipe["prep"].named_transformers_["num"]["impute"].statistics_ holds the fill values, the five numeric columns in order, area and year first.

import mlflow

from listings.data import load, split
from listings.features import build_pipeline
from listings.train import evaluate

mlflow.set_tracking_uri("sqlite:///../listings-probes/sweep.db")
mlflow.set_experiment("listings-sweep")
X_train, X_test, y_train, y_test = split(load())

with mlflow.start_run(run_name="impute-grid") as parent:
    for strategy in ["mean", "median", "most_frequent"]:
        for indicator in [True, False]:
            with mlflow.start_run(nested=True):
                mlflow.log_params({"strategy": strategy, "indicator": indicator})
                pipe = build_pipeline(strategy, indicator).fit(X_train, y_train)
                mlflow.log_metrics(evaluate(pipe, X_test, y_test))

every = mlflow.search_runs(experiment_names=["listings-sweep"])
children = mlflow.search_runs(
    experiment_names=["listings-sweep"],
    filter_string=f"tags.mlflow.parentRunId = '{parent.info.run_id}'",
)
print(len(every), len(children))
grid = children.pivot_table(
    index="params.strategy", columns="params.indicator", values="metrics.mae"
)
print(grid.round(2))

for strategy in ["mean", "median", "most_frequent"]:
    pipe = build_pipeline(strategy, False).fit(X_train, y_train)
    fills = pipe["prep"].named_transformers_["num"]["impute"].statistics_
    print(strategy, fills[:2].round(1))
# Only the indicator moves the score: with it on, all three strategies
# score 56,561.58, Lesson 6.1.3's finding. With it off the fill is all
# the model has for a blank, and the median is best of the three. The
# most common area is 24.0 m2, a small flat's, so every blank area is
# priced as one. Every MAE is a test score: a record, not a choice.

uv run python ../listings-probes/sweep.py
uv run python -c "import mlflow; print(len(mlflow.search_runs(experiment_names=['listings-price'])))"

2026/11/03 13:45:23 INFO mlflow.store.db.utils: Creating initial MLflow database tables...
2026/11/03 13:45:23 INFO mlflow.store.db.utils: Updating database tables
2026/11/03 13:45:24 INFO mlflow.tracking.fluent: Experiment with name 'listings-sweep' does not exist. Creating a new experiment.
7 6
params.indicator     False      True
params.strategy                     
mean              56431.89  56561.58
median            56335.63  56561.58
most_frequent     61504.27  56561.58
mean [  99.6 1990.2]
median [  89.7 1992. ]
most_frequent [  24. 2001.]
3

Additional Resources

Optional, and nothing later depends on either. Worth reading for what else a tracking store can sit on, or for every query the search language accepts.

• MLflow documentation - "Backend Stores", the section "Relational Database (Default)". Why SQLite is the default store, with sqlite:///mlflow.db created in the current directory, what the backend store holds against the artifact store, and the other databases MLflow can store runs in, PostgreSQL among them.
• MLflow documentation - "Search Runs", the section "Search Query Syntax Deep Dive". The full grammar of filter_string and the UI's search box: which comparators work on parameters, tags and metrics, and the attributes a filter can name.

Summary

The training code barely changed. The pipeline, the split and the scores are Lesson 3.2.3's, run for run, and what changed is that each run now leaves something behind that outlives the terminal: a row in a database with the settings beside the scores, and a folder holding the fitted pipeline with a frozen copy of the environment it was fitted in. The question a joblib file could never answer, which settings made it, is now one line in its MLmodel.

What the record holds is exactly what the code told it, and no more. It stored the settings as text because that is what parameters are to it, it tagged the runs with the commit Git pointed at because it cannot see an edit that was never committed, and it opened an empty store in the wrong folder because a relative address is all it was given. Each of those raised nothing and printed something ordinary, and each was caught only by setting the record beside something else: a comparison with the right type, git status, the folder the store lives in.

And the record is not a decision. Three runs side by side made it tempting to read the lowest MAE as the better model, but that MAE is a test score, and acting on it would turn the test rows into training data. The store makes runs comparable. It does not make them chosen, and the rows a choice is made on are still the one thing Module 3 said they had to be.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

mlflow.set_experiment("listings-price")
with mlflow.start_run() as run:
    mlflow.log_params(params)             # text
    pipe = build_pipeline(**params).fit(X_train, y_train)
    mlflow.log_metrics(evaluate(pipe, X_test, y_test))
    mlflow.sklearn.log_model(
        pipe, name="model", skops_trusted_types=["numpy.dtype"]
    )

runs = mlflow.search_runs(experiment_names=["listings-price"])
runs[runs["params.indicator"] == "True"]  # a string, not True

uv run python -m listings.train --no-indicator 2>/dev/null
uv run mlflow ui  # 127.0.0.1:5000