Defensive Code and Systematic Debugging
Context
Defensive code checks its assumptions instead of trusting them. Inputs are validated at the point they arrive, conditions that should be impossible are asserted so a violation stops immediately, and a bug is located by narrowing where it can be rather than by rereading the whole script.
The value is in when the failure lands. A loading function that verifies the expected columns are present fails at load, naming the problem, instead of letting an absent column travel three stages downstream and surface as an unrelated failure inside a metric calculation. Isolating a bug works the same way, checking the data halfway through the pipeline halves the search space every time.
Without this the errors you get are the ones furthest from their cause, which is the expensive kind. A wrong value created at load and used at the end takes the whole pipeline to trace, and the silent version, where the run finishes and the number is merely wrong, costs more because nothing announces it at all. The previous lesson was mostly about catching failures, this lesson raises the ones only your own knowledge of the data can detect.
Goal
- Validate a function's arguments at its boundary with guard clauses before trusting them.
- Use
assertto enforce invariants that must hold for the code to be correct. - Adopt a fail-fast stance, stopping at the first sign of an unrecoverable problem instead of limping on.
- Isolate a bug by narrowing it to its smallest reproducing case.
- Find a bug by inspecting intermediate state with deliberate, targeted printing.
Prerequisites
Optional, neither is required. The lesson stands without them, they are here if you want the background first.
- YouTube - "Python Tutorial: Logging Basics - Logging to Files, Setting Levels, and Formatting" by Corey Schafer, the first four chapters, through "Logging statements", on switching
printcalls for log calls and choosing a level. - Official docs - Python Language Reference, "The assert statement", for exactly what
assertdoes and when it is disabled.
Table of Contents
- Input Validation
- assert
- Fail-Fast
- Bug Isolation
- Inspection Debugging
Theory
When a model later produces a nonsense result, the bug is almost always upstream in the data, and the isolation discipline you build here is what you will use to find it. The previous lessons were reactive, read the crash, catch the exception. This lesson is proactive and methodical: stop bad input at the door, enforce the assumptions your code depends on, and when something does go wrong, find it by narrowing and inspecting rather than guessing. The individual tools are simple, the real skill is the discipline of using them.
Input Validation
What. Input validation is checking a function's arguments before trusting them, confirming a list is non-empty, a number is in range, a value is the expected type, and rejecting bad input immediately, usually by raising. It is built from the familiar conditional, used as a guard clause. Why. A function that trusts its inputs blindly fails deep inside its own body, far from the real cause, with a confusing traceback. Validating at the boundary moves the failure to the door, where the message can name exactly what was wrong with the argument. It also documents the function's contract in executable form, anyone reading it sees precisely what it requires. When. Validate at the boundary of any function that can receive bad input: public functions, anything taking data parsed from a file, anything where a wrong argument would otherwise cause a misleading downstream error. Pure internal helpers with controlled inputs need less of it.
A guard clause is a conditional at the top of a function that checks an argument and raises if it is unacceptable, returning control to the caller before the main logic runs. This is the conditional you already know, used as a gate. The type check is isinstance(values, list) (a built-in that returns True when values is a list), and a tuple of types in the second place, as in isinstance(x, (int, float)), returns True for any of them.
def normalize(values):
# Guard clauses: reject bad input at the boundary, before any real work.
if not isinstance(values, list):
raise TypeError(f"values must be a list, got {type(values).__name__}")
if len(values) == 0:
raise ValueError("values must not be empty")
# Past the guards, what they checked is trusted and the logic is clean.
largest = max(values)
return [v / largest for v in values]
print(normalize([2, 4, 8])) # [0.25, 0.5, 1.0]
print(normalize([])) # ValueError: values must not be emptyPast the guards the body can trust what they checked, a non-empty list, which is what keeps the real logic uncluttered. Without them the empty list would fail inside max with a message about an empty iterable, naming neither normalize nor the argument.
Validation should check what the function genuinely depends on, and the message should state both the requirement and what was actually received. A vague check or message defeats the purpose.
def split_dataset(rows, fraction):
# Validate the RANGE, not just the type, because 1.5 would corrupt the split.
if not (0.0 < fraction < 1.0):
raise ValueError(f"fraction must be between 0 and 1, got {fraction}")
cut = int(len(rows) * fraction)
return rows[:cut], rows[cut:]
train, test = split_dataset([1, 2, 3, 4], 0.5)
print(train, test) # [1, 2] [3, 4]
split_dataset([1, 2], 1.5) # ValueError: fraction must be between 0 and 1, got 1.5Validating the meaningful range rather than merely the type is what matters here. A fraction of 1.5 is a perfectly good float, so a type check passes it, and the resulting split would put every row in train and none in test without raising anything at all.
DON'T:
- DON'T validate so heavily that the checks dwarf the logic. Validate what genuinely breaks the function, type and range that matter, not every conceivable input.
- DON'T raise a vague message like
"bad input". State the requirement and what was received, so the caller can fix it without reading your source.
assert
What. assert condition, message is a statement that enforces a condition you believe must always be true at that point. If the condition is true, nothing happens. If it is false, it raises AssertionError with the message. Crucially, assertions can be globally disabled, so they are for catching bugs, not for handling expected runtime conditions.
Why. Assertions document and check the invariants your code relies on, facts that should never be false if the program is correct, like "after this split, the parts sum to the whole". They turn a silent logic error into a loud, located crash the instant an assumption breaks, which is exactly what makes logic bugs findable. They are a self-check the code performs on itself.
When. Use assert for internal invariants and sanity checks that should hold if your code is correct, never for validating external input or data, because assertions can be turned off. Validate untrusted input with a guard clause that raises a real exception, assert internal assumptions.
An assertion states a fact that must hold. If it holds, execution continues silently, if it does not, AssertionError fires immediately with your message, pinpointing the broken assumption.
def split_dataset(rows, fraction):
cut = int(len(rows) * fraction)
train, test = rows[:cut], rows[cut:]
# Invariant: the two parts together hold as many rows as the whole.
assert len(train) + len(test) == len(rows), "split lost or duplicated rows"
return train, test
# passes silently
train, test = split_dataset([1, 2, 3, 4], 0.5)An invariant is a fact that must hold if the code is correct, and this one says the two parts together hold exactly as many rows as the input. When the code is right the assertion is invisible, and when a future edit leaves the two parts short of rows, it fires at the exact point rather than producing a quietly short dataset.
The critical caveat: assertions are removed when Python is run with the -O (optimize) flag, so any check that must always run cannot be an assertion. This is the line between assert and a real validation that raises.
# WRONG: validating external data with assert, disabled under 'python3 -O'.
def load_age(text):
age = int(text)
# vanishes with -O !
assert age >= 0, "age cannot be negative"
return age
# RIGHT: external/untrusted input gets a real raise that always runs.
def load_age_safe(text):
age = int(text)
if age < 0:
# always enforced
raise ValueError("age cannot be negative")
return ageUnder python3 -O the assertion is not merely skipped, it is removed at compile time, so the first version silently accepts a negative age and the program carries on with it. The second always runs. The dividing line is where the value came from: reserve assert for internal invariants you control, and raise for anything that depends on the outside world.
DON'T:
- DON'T use
assertto validate user input, file data, or arguments from outside your code. Assertions can be disabled with-O, so the check may not run, raise an exception instead. - DON'T put side effects in an assertion.
assert process(x)would stop running under-O, silently removing whateverprocessdid, assertions must only test, never act.
Fail-Fast
What. Fail-fast is the stance of stopping at the very first sign of an unrecoverable problem, rather than continuing in a broken state. You detect the problem early, report it clearly, and halt, instead of limping forward and failing later in a confusing way. Why. A program that continues after something has gone irreparably wrong produces failures far from their cause, a bad value set in one place crashes ten functions later, and the traceback points at the symptom, not the disease. Failing fast keeps the distance between cause and crash as short as possible, which is the single biggest factor in how quickly a bug is found. A loud early failure is a gift, a silent corruption that surfaces later is a nightmare. When. Fail fast whenever a problem is unrecoverable, invalid configuration, missing required data, a violated invariant. Note the contrast with the previous lesson: skip-and-continue is right for recoverable per-row failures, fail-fast is right for unrecoverable whole-program ones. Choosing between them is a judgment about whether continuing can still produce a correct result.
Fail-fast means validating prerequisites up front and stopping immediately when one is missing, rather than discovering the problem deep into the work. The earlier the check, the shorter the path from cause to report.
def train_model(config, data):
# Fail fast: check EVERY prerequisite before doing any expensive work.
if "learning_rate" not in config:
raise ValueError("config missing 'learning_rate'")
if len(data) == 0:
raise ValueError("no training data provided")
if not (0.0 < config["learning_rate"] < 1.0):
raise ValueError(f"learning_rate out of range: {config['learning_rate']}")
# Only now, with everything verified, begin the real work.
print("training...") # stand-in for expensive computationThe alternative is not merely a later crash, it is a run that completes and returns a garbage model.
The opposite, fail-slow, lets a bad value travel until it causes a distant, misleading crash. Comparing the two shows why early detection matters so much.
# FAIL-SLOW: the bad config value is not noticed until much later.
def fail_slow(config):
# silently None if missing
rate = config.get("learning_rate", None)
# ... many lines later ...
adjusted = rate * 0.1 # TypeError here, far from cause
# FAIL-FAST: the same problem is caught immediately, at the source.
def fail_fast(config):
if "learning_rate" not in config:
# caught at the door
raise ValueError("config missing 'learning_rate'")
rate = config["learning_rate"]In fail_slow, a missing learning rate becomes None, which causes no error until rate * 0.1 raises TypeError many lines later, the traceback points at the multiplication, not at the missing config key that is the real cause. In fail_fast, the missing key is caught at the boundary with a message that names it directly. Same bug, but the fail-fast version reports it where it can be fixed.
DON'T:
- DON'T confuse fail-fast with fail-always. Recoverable per-row problems should be skipped and logged (previous lesson), only unrecoverable problems should halt the program.
- DON'T let a missing required value default to
Noneand continue. That converts a clear early failure into a confusing late one, check and raise at the source.
Bug Isolation
What. Bug isolation is narrowing a failure down to its smallest reproducing case, the least amount of code and data that still triggers the bug. You shrink the problem until what remains is small enough to understand at a glance. Why. A bug in a 500-line pipeline running on a 10,000-row file is nearly impossible to reason about. The same bug in a 5-line function on a 2-row input is often obvious. Isolation transforms an intractable problem into a tractable one by removing everything that is not part of the cause. It is the most important debugging skill, and it is a process, not a guess. When. Isolate whenever a bug is not immediately obvious. Rather than staring at the whole system, systematically cut it down: shrink the input, disable parts of the code, and check intermediate results until the failure is cornered in one small place.
The first move is to shrink the input until the bug still reproduces on the smallest possible data. A bug that appears on a huge file usually still appears on two or three carefully chosen rows.
# A bug shows up loading a 10,000-row file. Don't debug on 10,000 rows.
# Shrink to the SMALLEST input that still triggers it.
def parse_row(line):
id_str, score_str = line.split(",")
return int(id_str), float(score_str)
# Halve the input until one failing row is left, that row is your reproducer.
suspect = "5,n/a" # found by halving the file repeatedly
print(parse_row(suspect)) # reproduces the ValueError on ONE rowHalving the file repeatedly is the fastest way there, because each test eliminates half of what remains, so ten thousand rows narrows to one in about fourteen tries. Once the reproducer is a single short string, the bug is no longer about a file at all.
The second move is to narrow the code: isolate the failing operation into the smallest function or snippet that still fails, removing surrounding code that is not involved. Pure functions are dramatically easier to isolate, because their output depends only on their inputs.
# A pure function isolates trivially: same input -> same output, no hidden state.
def scale(values, factor):
return [v * factor for v in values] # depends ONLY on its arguments
# To reproduce any bug, you need just the inputs, nothing else.
print(scale([1, 2, 3], 0.5)) # [0.5, 1.0, 1.5], fully reproducible
# An IMPURE function is harder: its result depends on outside state too.
running_total = 0
def add_impure(x):
global running_total
running_total += x # result depends on hidden history
return running_totalThis is the concrete payoff of the purity idea from earlier in the module. To reproduce anything scale did you need only its arguments, while to reproduce anything add_impure did you must also recreate every call that came before it, in order.
DON'T:
- DON'T try to debug on the full system and full dataset. Shrink the input and narrow the code first, a small reproducer is worth more than any amount of staring.
- DON'T change several things at once while isolating. Change one variable at a time, or you will not know which change mattered.
Inspection Debugging
What. Inspection debugging is finding a bug by examining intermediate state, most simply, by inserting deliberate print statements to see the actual values flowing through the code at each step. Here print is reframed as a diagnostic instrument, not merely a way to produce output.
Why. Most bugs come from a mismatch between what you think a value is and what it actually is at some point. You cannot reason your way past that mismatch by reading the code, because the code was written from the same assumption, you have to look at the real values. Inspection makes the invisible visible, replacing the guess "it should be 5 here" with the fact "it is actually '5\n' here", which is usually the moment the bug becomes obvious.
When. Inspect whenever isolation has cornered the bug but the cause is still unclear. Print the inputs and outputs of the suspect function, the value just before the failing line, and the type when a type mismatch is possible. Inspect methodically, check your assumption about one value at a time.
The technique is to print the actual value (and often its type) at the points where your assumption might be wrong. Seeing the real value next to your expectation exposes the mismatch.
def parse_row(line):
parts = line.split(",")
# Inspect: is 'parts' what I think it is? Print the value AND its length.
print("DEBUG parts:", parts, "len:", len(parts))
id_str, score_str = parts # this is where it failed
return int(id_str), float(score_str)
parse_row("5,0.9,extra")
# DEBUG parts: ['5', '0.9', 'extra'] len: 3
# -> the unpacking expected 2 but got 3, the row has an extra fieldlen: 3 is the smoking gun, and it is the sort of fact that no amount of re-reading the code would have produced, because the code is not what is wrong. The print turned a guess into an observation.
Printing the type is essential when the bug might be a type mismatch, because two values can look identical when printed yet behave differently. A string "5" and an integer 5 print the same but are not equal.
def compute(value):
# A value that LOOKS right can be the wrong type. Print the type to be sure.
print(f"DEBUG value={value!r} type={type(value).__name__}")
return value * 2
compute("5")
# DEBUG value='5' type=str
# -> '5' is a STRING, value * 2 gives "55", not 10, the bug is a missing castThe !r is doing the work here. A plain print(value) shows 5 for both the string and the integer, so it would have hidden the problem entirely, while !r shows '5' with its quotes and the type name settles it. This is the key move for the string-versus-number bugs that plague data parsing, because * is defined for both and quietly means something different for each.
After inspection finds and you fix the bug, remove the debug prints, they were instruments, not permanent output. For larger programs the same idea scales up to the logging module, which lets you leave diagnostics in place and switch them on and off by level.
DON'T:
- DON'T sprinkle prints randomly and hope. Inspect the specific value your assumption depends on, methodically, one at a time, random prints add noise, not insight.
- DON'T leave debug prints in finished code. Remove them once the bug is fixed, or graduate to the
loggingmodule so diagnostics can be turned off cleanly.
Additional
The whole section is a workflow, and it has a natural order: validate inputs so bad data is rejected early, fail fast so problems surface near their cause, and when a bug still gets through, isolate it to a small reproducer and inspect the real values to find it. Reactive handling (last lesson) and proactive defense (this lesson) are two halves of one craft, catch what you cannot prevent, prevent what you can, and have a method for the rest.
# The combined discipline in one small loader.
def load(lines):
if not lines: # validate + fail fast
raise ValueError("load() received no lines")
out = []
skipped = 0
for i, line in enumerate(lines, start=1):
try:
a, b = line.strip().split(",") # may raise (recoverable)
out.append({"id": int(a), "score": float(b)})
except ValueError as err:
skipped += 1
# inspect
print(f"DEBUG row {i} skipped: {line!r} ({err})")
continue # skip recoverable failures
# invariant: every input row was kept or skipped
assert len(out) + skipped == len(lines), "rows lost between input and output"
return outThis loader combines four of the five ideas: a guard clause validates the input and fails fast on empty data, a try/except skips recoverable per-row failures while a debug print inspects each skipped row, and an assert enforces an internal invariant on the result. The reactive and proactive tools work together, validation and fail-fast at the boundary, exception handling for recoverable rows, assertion for the invariant, and inspection for diagnosis.
Lab Exercises
Lab Exercise 1: Guard the Door, Assert the Invariant, Fail Early
Write percent(part, whole) returning part / whole * 100, guarded so that a whole of zero raises ValueError and a non-numeric argument raises TypeError, each with a message naming the requirement and what was received. Test it on (3, 4), (1, 0) and ("a", 4), catching the two failures so the script survives and printing each message.
Then write merge_halves(a, b) returning a + b with an assertion that the merged length equals len(a) + len(b), and buggy_merge(a, b) returning a + b[1:] with the identical assertion. Run both, catching the second, and add a comment saying why an assertion is right here and a raise would not be.
Finally write start_run(config) that fails fast, checking that "epochs" is present and positive and that "lr" is present and strictly between 0 and 1, before printing run started. Demonstrate one valid config and two distinct invalid ones.
Lab Exercise 2: Isolate a Bug, Then Look at the Actual Values
You are told that load(lines) below produces records whose score is wrong, without ever raising.
def load(lines):
out = []
for line in lines:
parts = line.split(",")
out.append({"id": int(parts[0]), "score": parts[1]})
return outShrink the input first. Find the smallest list of lines that still shows the problem, and print the result so the wrongness is visible rather than described.
Then inspect. Add a print inside the loop showing parts, its length, and the repr and type of the value going into "score". State in a comment what the type actually is and what you had assumed.
Then fix load so score is a float, and prove the fix by printing the repaired records and the type of one score. Add a comment on why the original never raised.
Solution 1: Guard the Door, Assert the Invariant, Fail Early
def percent(part, whole):
# Guards run before any arithmetic, so the body can trust what they checked.
if not isinstance(part, (int, float)) or not isinstance(whole, (int, float)):
raise TypeError(f"percent() needs numbers, got {type(part).__name__} and {type(whole).__name__}")
if whole == 0:
raise ValueError("percent() requires a non-zero 'whole', got 0")
return part / whole * 100
print(percent(3, 4)) # 75.0
for part, whole in ((1, 0), ("a", 4)):
try:
percent(part, whole)
except (ValueError, TypeError) as err:
print(type(err).__name__, "|", err)def merge_halves(a, b):
merged = a + b
assert len(merged) == len(a) + len(b), "merge lost or duplicated items"
return merged
def buggy_merge(a, b):
merged = a + b[1:] # the deliberate bug
assert len(merged) == len(a) + len(b), "merge lost or duplicated items"
return merged
print(merge_halves([1, 2], [3, 4])) # [1, 2, 3, 4]
try:
buggy_merge([1, 2], [3, 4])
except AssertionError as err:
print("AssertionError |", err)
# An assertion is right here because the claim is about OUR OWN code being
# correct, not about the caller's data. Both lists were perfectly valid, so
# there is nothing for the caller to fix and nothing to raise a ValueError about.def start_run(config):
if "epochs" not in config:
raise ValueError("config missing 'epochs'")
if config["epochs"] <= 0:
raise ValueError(f"epochs must be positive, got {config['epochs']}")
if "lr" not in config:
raise ValueError("config missing 'lr'")
if not (0.0 < config["lr"] < 1.0):
raise ValueError(f"lr must be between 0 and 1, got {config['lr']}")
print("run started")
start_run({"epochs": 10, "lr": 0.01}) # run started
for bad in ({"epochs": 0, "lr": 0.01}, {"epochs": 10, "lr": 1.5}):
try:
start_run(bad)
except ValueError as err:
print("ValueError |", err)Every check in start_run happens before the line that would do the work, which is the whole of failing fast. Checking lr after training began would report the same error minutes later, attached to a run that has already consumed the time.
DON'T:
- DON'T use
assertfor thepercentguards, those are statements about the caller's arguments and they must run under every interpreter flag. - DON'T check only the type of
lr,1.5is a perfectly good float and it is still wrong.
Solution 2: Isolate a Bug, Then Look at the Actual Values
def load(lines):
out = []
for line in lines:
parts = line.split(",")
out.append({"id": int(parts[0]), "score": parts[1]})
return out
# STEP 1, shrink. One line is enough to show it, so debug on one line.
print(load(["1, 0.9"])) # [{'id': 1, 'score': ' 0.9'}]# STEP 2, inspect. Print the value, its repr and its type, not a description.
def load_debug(lines):
out = []
for line in lines:
parts = line.split(",")
print("DEBUG parts:", parts, "len:", len(parts))
print("DEBUG score:", repr(parts[1]), type(parts[1]).__name__)
out.append({"id": int(parts[0]), "score": parts[1]})
return out
load_debug(["1, 0.9"])
# DEBUG parts: ['1', ' 0.9'] len: 2
# DEBUG score: ' 0.9' str
# The score is a STRING with a leading space. I had assumed it was a float.# STEP 3, fix, then prove the fix rather than asserting it.
def load_fixed(lines):
out = []
for line in lines:
parts = line.split(",")
# cast it
out.append({"id": int(parts[0]), "score": float(parts[1])})
return out
records = load_fixed(["1, 0.9", "2,0.5"])
print(records) # [{'id': 1, 'score': 0.9}, {'id': 2, 'score': 0.5}]
print(type(records[0]["score"]).__name__) # float
# The original never raised because nothing it did was illegal. Building a dict
# with a string value is fine, and 'float()' was simply never called, so nothing
# fails here, and using the score as a number later can raise or mislead.Note that float(" 0.9") handles the leading space on its own, so the cast fixes both problems at once. That is worth knowing and worth not relying on, since int(" 1 ") also tolerates surrounding whitespace while a comparison against " 0.9" never will.
DON'T:
- DON'T debug this on the full file, one line reproduced it completely and one line is what you can hold in your head.
- DON'T print the value alone,
print(parts[1])shows0.9, hiding the type and leaving the leading space easy to miss, and those two are the entire bug.
Theory Assessment
Q1: Guarding at the Door, and Checking the Right Thing
Explain why a guard clause at the top of a function produces a better failure than letting bad input reach the body, using an empty list handed to a function that calls max as the example. Then explain why validating the type of a fraction argument is not enough, and what a value of 1.5 would do to a train/test split that only checked the type.
Hint: Compare the two messages a caller would actually see. For the second half, ask whether 1.5 is a valid float and whether it is a valid fraction.
Q2: Which Check Survives, and How Early to Stop
State the difference between assert cond, msg and if not cond: raise ValueError(msg), including what happens to each under python3 -O, and give the rule for choosing between them. Then explain what fail-fast means and why a failure reported late is harder to fix than the same failure reported early.
Hint: One of the two can be removed from the program entirely before it runs. For the second half, think about the distance between the line that caused a problem and the line that finally complained.
Q3: When to Stop and When to Carry On, and How to Corner a Bug
The previous lesson said to skip bad rows and carry on, and this lesson says to fail fast. Explain when each is right and what distinguishes the two situations. Then describe how you would shrink a bug that appears when loading a ten-thousand-row file down to something you can reason about.
Hint: Ask whether the program can still produce a correct result after the problem. For the second half, each test should eliminate as much of the remaining input as possible.
Q4: The Same File, Run Two Ways
Read the following, then state what it prints when run normally, and what it prints when run as python3 -O.
def load_age(text):
age = int(text)
assert age >= 0, "age cannot be negative"
return age
print(load_age("30"))
print(load_age("-5"))The two runs differ, and one of them is the reason this lesson draws a hard line about where assert may be used. Say which run is which, what the second one means for a program that validates its data this way, and what the code should have been.
Hint: The -O flag does not make assertions fail silently, it does something more thorough than that. Ask what is left in the compiled program.
Answer 1:
A guard clause fails at the door, where the message can name the function, the requirement and the offending value. Without it, an empty list reaches the body and max([]) raises ValueError: max() iterable argument is empty, which is accurate and useless: the message names neither the function the caller invoked nor the argument they passed, and the traceback ends on a line inside code the caller never wrote. The guard turns that into a message naming the actual contract, and it does so before any work has been done.
Validating the type of fraction is not enough because 1.5 is a perfectly good float. It passes any type check and is still meaningless as a fraction. In a split computed as cut = int(len(rows) * fraction), a value of 1.5 puts every row in train and leaves test empty, and nothing raises, so the caller gets a split that is silently wrong rather than an error. The requirement is a range, so the range is what must be checked.
DON'T validate only what is easy to check, validate what the function actually depends on.
Answer 2:
assert cond, msg raises AssertionError when the condition is false, and under python3 -O the whole statement is removed at compile time, so it does not run at all. if not cond: raise ValueError(msg) is ordinary code that always runs and raises a type the caller can catch precisely.
The rule for choosing: use assert for internal invariants, facts that must be true if your own code is correct, where a failure means you have a bug. Use a raise for anything that depends on the outside world, arguments, file contents, config, user input, where a failure means the input is wrong. The first can be disabled safely because a correct program never trips it. The second must never be disabled, because it is the only thing standing between bad data and the rest of the program.
Fail-fast means stopping at the first sign of an unrecoverable problem rather than continuing in a broken state. A late failure is harder to fix because the distance between cause and complaint is where debugging time goes: a bad value set in one function crashes somewhere else entirely, and the traceback faithfully reports the symptom while saying nothing about the disease. The worst case is not a late crash at all but a completed run with a wrong answer.
DON'T use assert to check anything that came from outside your program, -O removes it, so it is not guaranteed to be there when you need it.
Answer 3:
Skip-and-continue is right when the failure is recoverable and local: one malformed row among thousands, where discarding it still leaves a correct result computed from the rest. Fail-fast is right when the failure is unrecoverable and global: a missing config key, an empty dataset, a violated invariant, where nothing you produce afterwards can be trusted.
The distinguishing question is whether the program can still produce a correct result after the problem. If it can, absorb the failure and record it. If it cannot, stop, because everything computed from that point is a wrong answer wearing the shape of a right one.
To shrink a ten-thousand-row bug that one bad row causes, halve the input repeatedly. Run the first half: if the bug appears, discard the second half and halve again, otherwise the bug is in the second half, so keep that instead. Each test eliminates half of what remains, so ten thousand rows narrows to a single row in about fourteen runs. Then narrow the code the same way, cutting the pipeline down to the one function that still reproduces it, and change one thing at a time so you always know which change mattered.
DON'T read the two rules as contradictory, they are answers to different questions about the same failure.
Answer 4:
Run normally, it prints 30, then stops with AssertionError: age cannot be negative, raised in load_age on the assert line, and exits with status 1.
Run as python3 -O, it prints:
30
-5No error, exit status 0, and a negative age returned to the caller as though it were valid.
The difference is that -O does not make assertions fail quietly or skip them at runtime. The assert statement is removed at compile time, so the second run is executing a program in which the check does not exist. There is nothing to skip.
What that means for a program validating its data this way is that the validation is not validation. It works in development, where nobody passes -O, and it silently disappears in any environment that runs with -O, which can be exactly the one where it was most needed. The bad value then travels into whatever consumes it, and the eventual failure, if there is one at all, happens far away with no mention of an age.
The code should have used a raise, because text comes from outside:
def load_age(text):
age = int(text)
if age < 0:
raise ValueError(f"age cannot be negative, got {age}")
return ageDON'T treat an assertion as a cheaper way of writing a validation, they are different tools and only one of them is guaranteed to run.
Assignment
Assignment: Debug an Aggregator, Then Defend the Fixed Version
Build the following in one file, in this order. You are given summarize(rows), which should return the average score per label and instead crashes.
def summarize(rows):
totals = {}
counts = {}
for r in rows:
label = r["label"]
totals[label] = totals.get(label, 0) + r["score"]
counts[label] = counts.get(label, 0)
return {label: totals[label] / counts[label] for label in totals}Isolate it first. Find the smallest input that still crashes, run it, and print the exception type and message rather than describing them.
Then inspect. Print totals and counts just before the return, and add a comment stating which of the two is wrong and what its values reveal about the line that built it.
Then fix it, and defend the fixed version: add a guard clause rejecting an empty rows with a clear ValueError, and one assertion stating an invariant that must hold if your fix is right. Choose an invariant that the broken version would have violated. Demonstrate the fixed function on three rows, two of them sharing a label, and on an empty list, catching the second.
Finally, show why the assertion is not a substitute for the guard. State in a comment what would happen to each of your two checks if the file were run with python3 -O, and which of the two failures that would let through.
Hints:
- One row already crashes it. Print both dictionaries before touching the return line, on an input where one label has two rows, which is what makes the stuck count unmistakable.
- A good invariant compares something you computed against something independently known, such as the number of counted rows against
len(rows).
Solution:
rows = [
{"label": "a", "score": 0.8},
{"label": "a", "score": 0.4},
{"label": "b", "score": 0.6},
]
def summarize(rows): # the broken version, unchanged
totals = {}
counts = {}
for r in rows:
label = r["label"]
totals[label] = totals.get(label, 0) + r["score"]
counts[label] = counts.get(label, 0)
return {label: totals[label] / counts[label] for label in totals}
# STEP 1, shrink. A single row is enough to crash it.
try:
summarize([{"label": "a", "score": 0.8}])
except ZeroDivisionError as err:
print(type(err).__name__, "|", err) # ZeroDivisionError | float division by zero# STEP 2, inspect. Print both dictionaries before the return.
def summarize_debug(rows):
totals = {}
counts = {}
for r in rows:
label = r["label"]
totals[label] = totals.get(label, 0) + r["score"]
counts[label] = counts.get(label, 0)
print("DEBUG totals:", totals)
print("DEBUG counts:", counts)
return totals, counts
summarize_debug(rows)
# DEBUG totals: {'a': 1.2000000000000002, 'b': 0.6}
# DEBUG counts: {'a': 0, 'b': 0}
# 'counts' is wrong: every value is 0, so the line that builds it reads the
# current count and stores it back unchanged. The '+ 1' is missing.# STEP 3, fix and defend.
def summarize_fixed(rows):
if not rows: # a guard: 'rows' comes from outside, so this must always run
raise ValueError("summarize() requires at least one row")
totals = {}
counts = {}
for r in rows:
label = r["label"]
totals[label] = totals.get(label, 0) + r["score"]
# the fix
counts[label] = counts.get(label, 0) + 1
# An invariant: the label counts add up to the number of input rows.
assert sum(counts.values()) == len(rows), "counted rows do not match input rows"
return {label: totals[label] / counts[label] for label in totals}
print(summarize_fixed(rows))
# {'a': 0.6000000000000001, 'b': 0.6}
try:
summarize_fixed([])
except ValueError as err:
print(type(err).__name__, "|", err)
# Under 'python3 -O' the assertion is REMOVED and the guard still runs. So the
# empty-list failure is still caught, and a future re-broken counter is not: a
# missing '+ 1' is back to a ZeroDivisionError on the return line, and a
# miscount that avoids zero passes silently. That is the right way round: the
# guard protects against bad input, the assertion against our own mistake.The invariant is the part worth dwelling on, because a weak one would have been useless here. sum(counts.values()) == len(rows) compares something computed by the loop against something known independently of it, so the broken version fails it immediately: three rows in, sum(counts.values()) is 0, and the assertion fires with a message naming the mismatch instead of a ZeroDivisionError from a line that is not at fault.
An invariant like assert len(counts) == len(totals) would have passed on the broken code, since both dictionaries had the same two keys and only the values were wrong. Asserting a fact that the bug does not disturb is worse than asserting nothing, because it reads like protection and is not.
Note also the float in the output. 0.8 + 0.4 is 1.2000000000000002 rather than 1.2, so the mean prints as 0.6000000000000001. That is ordinary binary floating point, not a symptom of the bug, and it is exactly why the invariant counts rows rather than comparing sums of scores for equality.
DON'T assert a condition the bug leaves true, and don't reach for an assertion where a guard belongs, -O removes one of them and the input does not stop being untrusted.
Additional Resources
Optional, and nothing later depends on either. Worth opening if you want to graduate your debug prints into something switchable.
- Official docs - Logging HOWTO, "Basic Logging Tutorial", for diagnostics you can leave in and turn off by level.
- Official docs -
pdb, "Debugger Commands". Stepping through a running program instead of printing from it, which is where inspection goes once the prints stop scaling.
Summary
Defensive code and systematic debugging are the habits that turn a script into software, and they share a single theme: do not trust, verify. Validation verifies the inputs before the body runs, rejecting the impossible at the door with a message that names the problem, so failures appear where they can be understood instead of deep inside where they cannot. Assertions verify the invariants the code itself depends on, turning a silent logic error into a loud, located crash the instant an assumption breaks, with the firm rule that they guard internal assumptions only, never external data, because they can be switched off. And the fail-fast stance verifies that continuing is even sensible, halting at the first unrecoverable problem rather than limping forward to crash somewhere confusing and far away. The judgment underneath all three is the same one from the previous lesson, seen from the other side: recoverable failures are skipped and logged, unrecoverable ones stop everything now.
When prevention is not enough and a bug gets through anyway, the answer is method, not luck. Isolation shrinks the problem until it is small enough to see, the smallest input and the least code that still fails, because a bug that is invisible in a thousand lines and ten thousand rows is often obvious in five lines and two rows. Inspection then makes the invisible visible, replacing what you assume a value is with what it actually is, and printing the type alongside the value is the move that cracks the silent string-versus-number bugs that data work is full of. Pure functions make both steps easier, because their behavior depends only on their inputs, one of the reasons they are worth preferring. Guessing is the slow way, and it scales with luck. Narrowing and inspecting is the fast way, and it scales with discipline.
This closes the section, and with it a shift in what you are capable of. You began the section having met try and except only in passing, and you end it able to read why your programs crash, catch the failures you anticipate, prevent the ones you can, and methodically hunt the ones that slip through. That is not a collection of tricks, it is a stance toward your own code: skeptical, deliberate, unwilling to confuse "it ran" with "it is correct". Carry it forward, because everything ahead, the data analysis, the statistics, the models, runs on data full of surprises, and when a model eventually produces a result that makes no sense, the bug will almost always be upstream in that data. The isolation and inspection discipline you built here is exactly what you will use to find it.
Cheat Sheet
Part A - Reference Table
| Tool | Use for | Mechanism | Always runs? |
|---|---|---|---|
| Guard clause | external input | if bad: raise ... |
yes |
assert |
internal invariant | assert cond, "msg" |
no (off under -O) |
| Fail-fast | unrecoverable problem | check + raise up front | yes |
| Skip-and-continue | recoverable problem | try/except: continue |
yes |
| Bug isolation | finding a bug | shrink input + narrow code | n/a |
| Inspection | finding a bug | print(value, type) |
n/a |
logging module |
toggleable diagnostics | leave in, switch off by level | configurable |
Part B - Compact Code Block
def process(values, fraction):
# Validate external input -> raise (always enforced).
if not isinstance(values, list):
raise TypeError("values must be a list")
if not values:
# fail fast
raise ValueError("values must not be empty")
if not (0.0 < fraction < 1.0):
raise ValueError(f"fraction out of range: {fraction}")
cut = int(len(values) * fraction)
a, b = values[:cut], values[cut:]
# invariant
assert len(a) + len(b) == len(values), "split lost rows"
return a, b
# Debug: tiny failing case, inspect value AND type
parts = "5,n/a".split(",")
print("DEBUG:", parts, [type(p).__name__ for p in parts])
# remove prints when fixed, or move to logging