Vectorized Operations

Context

A vectorized operation is one line of arithmetic, or one function call, written against a whole array, and NumPy applies it to every element in its own compiled code. Two arrays of the same shape combine position by position, and a single number combines with every element.

Derived columns and measures of a model's error are written this way. Price per square meter comes from dividing one column by another, age comes from subtracting a column of build years from the current year, a price target stretched by a few very expensive listings is log-transformed in one call, and the gap between predictions and prices is one subtraction, squared. Each is one expression covering every row, and it reads like the formula it implements.

The failures here split into loud and silent. The functions of Module 1's math module refuse an array outright. By default, the log of a zero returns negative infinity with nothing but a warning, the script carries on, and every sum computed from that column afterwards is ruined. A float multiplied into an integer array in place refuses, while a float with a fraction assigned into one is quietly cut down, and an array of a small integer type wraps past its limit without a word. The previous lesson selected parts of an array, and this one computes across all of it.

Goal

• Replace a Python loop over an array with one vectorized expression and confirm both give the same values.
• Combine arrays element by element with +, -, *, /, //, % and **, and predict the dtype of the result.
• Transform columns with the universal functions np.sqrt, np.log, np.log1p, np.exp, np.abs and np.maximum, and with np.round and np.clip.
• Diagnose the nan and -inf an invalid input produces, read the RuntimeWarning that announces them, and find them with np.isnan and np.isfinite.
• Update an array in place with += and *= without a casting error, a cut-off fraction or a silent overflow.

Prerequisites

Both are optional, and the lesson stands on its own. They are here if you want the case for leaving loops behind made by someone else first.

• YouTube - "Losing your Loops Fast Numerical Computing with NumPy" by Jake VanderPlas, on the PyCon 2015 channel. From "Python overhead" at 4:52 to the end of "Strategy 1 Using UFUNC" at 11:13: why a loop over numbers is slow in Python, and how a universal function replaces it.
• Official docs - NumPy, "Universal functions (ufunc) basics". The opening section: what a universal function is, an element-by-element wrapper around a function, many of the built-in ones written in compiled C code.

Table of Contents

• Vectorization
• Element-wise Arithmetic
• Universal Functions
• Invalid Values and Warnings
• In-place Operations and dtype

Theory

The lesson works on six listings again: their asking prices, their area_sqm, their year_built, and a model's predicted prices for the same six, one value per listing in every array.

Vectorization

What. Vectorization is writing one operation against a whole array, prices / 1000, instead of a Python loop over its elements. NumPy runs the loop over the elements itself, inside its compiled code. Why. A Python loop runs the interpreter once for every element. A vectorized line runs it once for the whole array and leaves the per-element work to compiled code, so far less Python runs, the result is the same, and the line reads like the formula. When. Whenever the same arithmetic applies to every element of an array.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

# the loop: the interpreter runs once per listing
in_thousands = []
for price in prices:
    in_thousands.append(price / 1000)
in_thousands = np.array(in_thousands)

# vectorized: one line, the loop runs inside NumPy
vectorized = prices / 1000

print(in_thousands)  # [310. 455. 289. 520. 398. 265.]
print(vectorized)    # [310. 455. 289. 520. 398. 265.]
# True
print(np.array_equal(in_thousands, vectorized))

The two results are identical, so the only difference is where the loop runs. On six listings neither is slow. On the millions of values in a real training set, the vectorized line finishes in a small fraction of the loop's time, because the interpreter, the expensive part, runs once instead of once per element. That is a property of how the work is split, so it holds on any machine, while the exact speed-up depends on the machine and the operation.

DON'T:

• DON'T loop over an array to apply the same arithmetic to every element. The vectorized line returns the same values and moves the per-element work into compiled code.
• DON'T build a list in a loop and convert it at the end when one expression builds the array directly, the conversion is a second pass over everything the loop already touched.

Element-wise Arithmetic

What. +, -, *, /, //, % and ** between two arrays of the same shape pair them position by position, and between an array and a single number they apply that number to every element. Why. Most derived columns are arithmetic on other columns: price per square meter, age from a build year, the error between a prediction and a price. When. Whenever two arrays describe the same rows in the same order, or one constant applies to every row.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
area_sqm = np.array([72.5, 110.0, 64.0, 135.5, 98.0, 58.5])
year_built = np.array([2000, 2007, 1977, 2018, 1993, 1965])

price_per_sqm = prices / area_sqm  # position by position
age = 2026 - year_built            # one number against every element
print(price_per_sqm)
print(age)
print(age // 10)                   # whole decades of age
print(age % 10)                    # the years left over

[4275.86206897 4136.36363636 4515.625      3837.63837638 4061.2244898
 4529.91452991]
[26 19 49  8 33 61]
[2 1 4 0 3 6]
[6 9 9 8 3 1]

A single number against an array is the simplest case of broadcasting, NumPy's rule for combining arrays of different shapes, which Lesson 2.2.4 states in full. Comparisons such as prices > 300000 are element-wise in the same way, which is what built every mask in Lesson 2.2.2.

The result's dtype follows the operator. / returns float64 even on two integer arrays, because a quotient can have a fraction, while +, -, *, // and % on integers stay integers.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
predicted = np.array([300000, 470000, 280000, 500000, 410000, 270000])

print((prices / 2).dtype)   # float64
print((prices // 2).dtype)  # int64

errors = predicted - prices
print(errors)               # [-10000  15000  -9000 -20000  12000   5000]
print(errors ** 2)          # [100000000 225000000  81000000 400000000 144000000  25000000]

Squaring makes a miss of 20000 above or below count the same, and makes a large miss count far more than a small one. Two arrays whose shapes cannot pair are refused rather than lined up by guesswork. Saved as shapes.py, with one area missing:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
area_sqm = np.array([72.5, 110.0, 64.0, 135.5, 98.0])
print(prices / area_sqm)

Traceback (most recent call last):
  File "/home/goku/dev/bootcamp/shapes.py", line 5, in <module>
    print(prices / area_sqm)
          ~~~~~~~^~~~~~~~~~
ValueError: operands could not be broadcast together with shapes (6,) (5,) 

DON'T:

• DON'T expect / to keep integers. prices / 2 is float64 even where every result is whole, and // is the operator that stays int64.
• DON'T combine two columns because their shapes match. NumPy pairs whatever sits at each position, so a column sorted differently, or with a row missing and another added, divides the wrong listings into each other without a word.
• DON'T read shapes (6,) (5,) as a NumPy fault. It is the first sign that two columns no longer describe the same rows.

Universal Functions

What. A universal function, a ufunc, is a NumPy function that applies one mathematical operation element by element and returns a new array of the results: np.sqrt, np.exp, np.log, np.log1p, np.abs and np.maximum among them. np.round and np.clip are plain NumPy functions that work element by element the same way. Why. Transforms of tabular data are functions of whole columns, a log to compress a skewed price, an absolute value for the size of an error, a cap on outliers. The functions of Module 1's math module take one number at a time and refuse an array. When. Whenever every element passes through the same function, and in place of any math function you would otherwise call inside a loop.

Prices are skewed, a few listings cost far more than most, and a log compresses that spread. np.log and np.exp undo each other, up to float rounding in the last digits. np.log1p(x) computes the log of 1 + x, which is 0 at zero where np.log has no answer, so it is the usual transform for a price target and for any column that can hold a zero, such as days on the market.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
days_on_market = np.array([0, 12, 45])

print(np.log(prices))
print(np.exp(np.log(prices)))
print(np.log1p(prices))
print(np.log1p(days_on_market))

[12.64432758 13.0280527  12.57418197 13.16158409 12.89420728 12.4874851 ]
[310000. 455000. 289000. 520000. 398000. 265000.]
[12.6443308  13.0280549  12.57418543 13.16158601 12.8942098  12.48748888]
[0.         2.56494936 3.8286414 ]

The rest shape values without transforming their scale. np.abs gives each error's size, np.round(x, d) rounds to d decimals, np.clip(x, low, high) pulls every value into a range, and np.maximum(a, b) keeps the larger of two values at each position.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
area_sqm = np.array([72.5, 110.0, 64.0, 135.5, 98.0, 58.5])
predicted = np.array([300000, 470000, 280000, 500000, 410000, 270000])

print(np.abs(predicted - prices))       # [10000 15000  9000 20000 12000  5000]
print(np.round(prices / area_sqm, 1))   # [4275.9 4136.4 4515.6 3837.6 4061.2 4529.9]
print(np.clip(prices, 280000, 500000))  # [310000 455000 289000 500000 398000 280000]
print(np.maximum(prices - 300000, 0))   # [ 10000 155000      0 220000  98000      0]

np.maximum(prices - 300000, 0) is how far each price sits above 300000, with 0 where it does not. A math function has no notion of an array, so it fails where the ufunc works. Saved as mathsqrt.py:

import math
import numpy as np

area_sqm = np.array([72.5, 110.0, 64.0, 135.5, 98.0, 58.5])
print(math.sqrt(area_sqm))

Traceback (most recent call last):
  File "/home/goku/dev/bootcamp/mathsqrt.py", line 5, in <module>
    print(math.sqrt(area_sqm))
          ^^^^^^^^^^^^^^^^^^^
TypeError: only 0-dimensional arrays can be converted to Python scalars

DON'T:

• DON'T call a math function on an array. math.sqrt(area_sqm) raises TypeError, while np.sqrt(area_sqm) takes an array of any shape.
• DON'T expect np.round to round every half up. np.round(2.5) is 2.0 and np.round(3.5) is 4.0, because a half goes to the even neighbor.
• DON'T read np.maximum(a, b) as the largest value of an array. It compares two inputs position by position and returns an array, and the single largest value is max, an aggregation Lesson 2.2.5 teaches.

Invalid Values and Warnings

What. With NumPy's default settings, a ufunc that returns floats and meets an input outside its domain returns a special value instead of raising. np.log(0.0) is -inf, negative infinity, np.sqrt(-4.0) is nan, not a number, and NumPy prints a RuntimeWarning and carries on. Integers have no special values, so // and % by zero on integer arrays return 0, with the same kind of warning. Why. The warning scrolls past and the script exits normally, while the -inf or nan stays in the array and spreads into every sum() or mean() computed from it. When. Treat every RuntimeWarning as a report that a result now holds a special value, or a wrong number when the result is an integer, and check a transformed column before you compute anything from it.

A column of days on the market holds a 0 for a listing that sold the day it went up, and an area column holds a corrupted negative value. Saved as transform.py:

import numpy as np

days_on_market = np.array([12.0, 0.0, 45.0])
print(np.log(days_on_market))
print(np.sqrt(np.array([64.0, -4.0])))
print("still running")

/home/goku/dev/bootcamp/transform.py:4: RuntimeWarning: divide by zero encountered in log
  print(np.log(days_on_market))
[2.48490665       -inf 3.80666249]
/home/goku/dev/bootcamp/transform.py:5: RuntimeWarning: invalid value encountered in sqrt
  print(np.sqrt(np.array([64.0, -4.0])))
[ 8. nan]
still running

Each warning names the file and the line, then shows the line. It is printed while the line runs, before the print on it can show the result, and it goes to standard error, the stream Module 0 kept apart from standard output, so python3 transform.py > out.txt still shows both warnings on screen. The script finishes, still running prints, and the exit status is 0.

np.nan and np.inf are NumPy's names for these special values. A nan spreads through arithmetic, so a sum or a mean that includes one is nan, and a nan is not equal to anything, itself included, so == cannot find it. np.isnan finds every nan, and np.isfinite is True only for ordinary numbers, neither nan nor an infinity.

import numpy as np

values = np.array([2.5, np.nan, 3.8])

print(values + 1)        # [3.5 nan 4.8]
print(values.sum())      # nan
print(values == np.nan)  # [False False False]
print(np.isnan(values))  # [False  True False]
# [ True False False]
print(np.isfinite(np.array([2.5, -np.inf, np.nan])))

Lesson 2.2.5 meets the aggregations that skip a nan instead of returning it.

DON'T:

• DON'T treat a RuntimeWarning as noise. After a log or a square root the array it describes holds -inf or nan, and its sum() or mean() is -inf or nan too.
• DON'T look for nan with ==. values == np.nan is False everywhere, even at the nan, and np.isnan(values) is the test.
• DON'T wrap a ufunc call in try to catch a bad input. A warning is not an exception, so with NumPy's default settings the except branch never runs and the -inf comes out of the try untouched.
• DON'T count the warnings to count the bad values. Python prints a given warning once per line of code, so a loop that took the log of zero three times printed one warning and produced three -inf.

In-place Operations and dtype

What. +=, -=, *= and /= write the result into the existing array instead of building a new one, so the array keeps its dtype and every view of it sees the change. Why. An in-place update saves a full-size copy on a large array, and it is how running totals and repeated updates are usually written. The price is that the result has to fit the dtype the array already has. When. When you mean to change an array, not make a new one, and its dtype can hold the result.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
recent = prices[3:]  # a view, Lesson 2.2.2

prices += 5000       # written into the existing array
print(prices)        # [315000 460000 294000 525000 403000 270000]
print(recent)        # [525000 403000 270000]

A float result does not fit an integer array, and the in-place form refuses rather than drop the fractions. Saved as inplace.py:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
prices *= 1.05

Traceback (most recent call last):
  File "/home/goku/dev/bootcamp/inplace.py", line 4, in <module>
    prices *= 1.05
numpy._core._exceptions._UFuncOutputCastingError: Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'

The long type name is a NumPy-internal kind of TypeError, so except TypeError catches it. There are two fixes, and both leave the integer prices as it was:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

raised = prices * 1.05  # a NEW float64 array
print(raised.dtype)     # float64
print(prices.dtype)     # int64

as_float = prices.astype(float)
as_float *= 1.05        # now the result fits
print(as_float)         # [325500. 477750. 303450. 546000. 417900. 278250.]

Two neighbors of that error are silent. Assigning a float into an integer array cuts off the fraction, as astype(int) did in Lesson 2.2.1, and an integer array too small for a result wraps around. A single nan cannot be cut down and raises ValueError: cannot convert float NaN to integer instead, while a nan arriving inside a float array, through a slice, a mask or a list of positions, becomes a meaningless integer with only RuntimeWarning: invalid value encountered in cast to say so. uint8, an unsigned 8-bit integer, holds only 0 to 255, the range image files commonly use for a pixel's brightness.

import numpy as np

prices = np.array([310000, 455000, 289000])
prices[0] = 310000.75           # a float into an int64 array
print(prices[0])                # 310000

pixels = np.array([250, 180, 30], dtype=np.uint8)
print(pixels + 10)              # [  4 190  40]
print(pixels.astype(int) + 10)  # [260 190  40]

250 + 10 is 260, which uint8 cannot hold, so it wrapped around to 4, and nothing printed a warning. Converting to a wider type first keeps the true sum. A Python integer the type cannot hold is refused outright, pixels + 300 raises OverflowError: Python integer 300 out of bounds for uint8.

DON'T:

• DON'T multiply an integer array in place by a float. prices *= 1.05 raises the casting error, and prices.astype(float) first, or a new array from prices * 1.05, is the fix.
• DON'T assign a float into an integer array and expect it kept. prices[0] = 310000.75 stored 310000, with no error.
• DON'T do arithmetic on a small integer type without checking its range. On uint8, 250 + 10 is 4.
• DON'T forget that an in-place update reaches every view. recent changed when prices += 5000 ran, though nothing was written to recent by name.

Lab Exercises

Lab Exercise 1: Derive New Columns From Raw Ones

Start from these arrays, one value per listing:

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
area_sqm = np.array([72.5, 110.0, 64.0, 135.5, 98.0, 58.5])
year_built = np.array([2000, 2007, 1977, 2018, 1993, 1965])
predicted = np.array([300000, 470000, 280000, 500000, 410000, 270000])

Compute the price per square meter twice, once with a Python loop over zip(prices, area_sqm) that builds a list and once as one vectorized expression, and print whether the two are equal. Print the price per square meter rounded to whole numbers.

Print each listing's age in 2026, and the decade it was built in, 2000 for 2007 and 1970 for 1977, computed with // and * rather than typed. Print the absolute error of each prediction and the squared error, and print the prices capped to the range 280000 to 500000.

Lab Exercise 2: Transform a Column Safely, Then Update Prices in Place

Start from these arrays:

import numpy as np

days_on_market = np.array([12, 0, 45, 3, 0, 30])
prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])

Take np.log of days_on_market, print the result, and print how many of its values are not finite, without counting by eye. Then apply np.log1p instead, and print the result and the same count.

Next, a price index says every price rose by 3 percent. Keep a view of the last three prices called recent. Try prices *= 1.03 inside a try, catch the error as a TypeError and print its message. Then make it work in place on a float copy of the prices called updated, keep a view of its last three values, and print updated, the view, and recent, each after the update.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
area_sqm = np.array([72.5, 110.0, 64.0, 135.5, 98.0, 58.5])
year_built = np.array([2000, 2007, 1977, 2018, 1993, 1965])
predicted = np.array([300000, 470000, 280000, 500000, 410000, 270000])

looped = []
for price, area in zip(prices, area_sqm):
    looped.append(price / area)
vectorized = prices / area_sqm
print(np.array_equal(np.array(looped), vectorized))
print(np.round(vectorized))

age = 2026 - year_built
decade = year_built // 10 * 10  # drop the last digit, put a 0 back
print(age)
print(decade)

errors = predicted - prices
print(np.abs(errors))
print(errors ** 2)
print(np.clip(prices, 280000, 500000))

True
[4276. 4136. 4516. 3838. 4061. 4530.]
[26 19 49  8 33 61]
[2000 2000 1970 2010 1990 1960]
[10000 15000  9000 20000 12000  5000]
[100000000 225000000  81000000 400000000 144000000  25000000]
[310000 455000 289000 500000 398000 280000]

import numpy as np

days_on_market = np.array([12, 0, 45, 3, 0, 30])

logged = np.log(days_on_market)
print(logged)
print((~np.isfinite(logged)).sum())  # how many are NOT finite

safe = np.log1p(days_on_market)
print(safe)
print((~np.isfinite(safe)).sum())

/home/goku/dev/bootcamp/days.py:5: RuntimeWarning: divide by zero encountered in log
  logged = np.log(days_on_market)
[2.48490665       -inf 3.80666249 1.09861229       -inf 3.40119738]
2
[2.56494936 0.         3.8286414  1.38629436 0.         3.4339872 ]
0

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
recent = prices[3:]

try:
    prices *= 1.03
except TypeError as err:
    print("TypeError:", err)

updated = prices.astype(float)  # a float COPY, prices is untouched
updated_recent = updated[3:]
updated *= 1.03
print(updated)
print(updated_recent)
print(recent)

TypeError: Cannot cast ufunc 'multiply' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'
[319300. 468650. 297670. 535600. 409940. 272950.]
[535600. 409940. 272950.]
[520000 398000 265000]

Theory Assessment

Q1: Same Values, Different Work

Explain what changes and what stays the same when a loop over prices that divides each value by 1000 is replaced by prices / 1000. Then state the dtype of prices / 1000, prices // 1000 and prices * 2 when prices holds integers, and why / is the odd one out.

Hint: Ask where the loop over the elements runs in each version, and how many times the interpreter does work. For the dtype, ask which of the three can produce a fraction.

Q2: Three Calls That Look Alike

Say what each of these does, or why it fails: math.sqrt(area_sqm), np.sqrt(area_sqm) and np.round(np.array([2.5, 3.5])). Then explain the difference between np.maximum(prices, 300000) and the single largest price.

Hint: One of the three was written for one number at a time. For the rounding, look at which neighbor each half goes to, and for the last part ask whether the result is one value or one per listing.

Q3: The Warning and the Value It Leaves Behind

Running print(np.log(np.array([5.0, 0.0]))) in a script prints a line mentioning RuntimeWarning and then an array. Say what the array holds, where the warning goes, and whether a try around the line would catch anything. Then explain why x == np.nan cannot find the nan values in x, and what does.

Hint: A warning and an exception are different things in Python. For the nan, ask what np.nan == np.nan itself returns.

Q4: Four Prints, No Exceptions

Read this script, saved as q4.py, without running it. State exactly what it prints, warnings included. Then say which printed results are wrong for their purpose, and which of those arrived with any message at all.

import numpy as np

prices = np.array([310000, 455000, 289000])
recent = prices[1:]
prices += 1000
scaled = prices / 1000
prices[0] = 299999.9
pixels = np.array([200, 100], dtype=np.uint8)

print(recent)
print(scaled.dtype, prices[0])
print(pixels * 2)
print(np.log(np.array([1.0, 0.0])).sum())

Hint: Track prices line by line and remember what recent is. Then ask what an integer array does with .9, what fits in uint8, and what a sum does with -inf.

import numpy as np

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
print(np.round(np.array([2.5, 3.5])))  # [2. 4.]
print(np.maximum(prices, 300000))      # [310000 455000 300000 520000 398000 300000]

[456000 290000]
float64 299999
[144 200]
/home/goku/dev/bootcamp/q4.py:13: RuntimeWarning: divide by zero encountered in log
  print(np.log(np.array([1.0, 0.0])).sum())
-inf

Assignment

Assignment: A Log Transform That Fails Loudly, and Its Way Back

A model will be trained on log-transformed prices, and its outputs will have to be turned back into prices. Write to_log(prices) that returns np.log1p(prices) but refuses, with an exception instead of a warning, any price of -1 or less, which np.log1p turns into -inf or nan. np.errstate(divide="raise", invalid="raise"), used as with np.errstate(divide="raise", invalid="raise"):, makes the code inside the block raise FloatingPointError for exactly those two warnings. Write from_log(values) that undoes the transform with np.expm1, which computes exp(x) - 1, the inverse of np.log1p.

Show that the six lesson prices survive the round trip, compared once with np.array_equal and once with np.allclose(a, b), which is True when every pair of values differs by no more than 1e-08 plus a hundred-thousandth of the second value, and explain why the two comparisons disagree. Then a data refresh writes a missing price as -1 and another as -5. Run to_log on that batch, catch the error and print its message, and print the positions of the prices that caused it. Finally, show what the same batch turns into without np.errstate, and count its values that are not finite.

Hints:

• np.log1p(-1.0) is the log of zero and np.log1p(-5.0) the log of a negative number, the same two failures as np.log(0.0) and np.sqrt(-4.0).
• Float arithmetic rounds at every step, so a value can come back from a transform and its inverse a tiny fraction away from where it started.

import numpy as np

def to_log(prices):
    with np.errstate(divide="raise", invalid="raise"):
        return np.log1p(prices)

def from_log(values):
    return np.expm1(values)

prices = np.array([310000, 455000, 289000, 520000, 398000, 265000])
back = from_log(to_log(prices))
print(back)
print(np.array_equal(back, prices))
print(np.allclose(back, prices))
print(back - prices)

refresh = np.array([310000, -1, 455000, -5])
try:
    to_log(refresh)
except FloatingPointError as err:
    print("FloatingPointError:", err)
    print(np.where(refresh <= -1))

unguarded = np.log1p(refresh)
print(unguarded)
print((~np.isfinite(unguarded)).sum())

[310000. 455000. 289000. 520000. 398000. 265000.]
False
True
[ 1.74622983e-10  0.00000000e+00 -1.16415322e-10  0.00000000e+00
 -2.32830644e-10  5.82076609e-11]
FloatingPointError: divide by zero encountered in log1p
(array([1, 3]),)
/home/goku/dev/bootcamp/log_target.py:24: RuntimeWarning: divide by zero encountered in log1p
  unguarded = np.log1p(refresh)
/home/goku/dev/bootcamp/log_target.py:24: RuntimeWarning: invalid value encountered in log1p
  unguarded = np.log1p(refresh)
[12.6443308       -inf 13.0280549        nan]
2

Additional Resources

Optional, and nothing later in the bootcamp depends on either. Worth opening if you want more control over warnings, or the limits of every integer type.

• Official docs - NumPy, "Floating point error handling", "How numpy handles numerical exceptions". The six behaviors, from ignoring a floating-point problem to raising on it, and the four kinds of problem each can be set for.
• Official docs - NumPy, "Data types", "Overflow errors". How a fixed-size integer overflows, and np.iinfo, which reports the smallest and largest value every integer type can hold.

Summary

Vectorization looks like a speed trick, and its real effect is on how you think about a column. A loop treats a column as a sequence of separate values to visit, and a vectorized expression treats it as one object that a formula applies to. Price per square meter becomes prices / area_sqm, an error becomes predicted - prices, and the code reads like the definitions you would write on paper. The speed follows from the same idea, because the interpreter that runs your code once, rather than once per element, is the expensive part.

The cost of handing the loop to NumPy is that NumPy decides what happens at the edges, and its decisions are not the ones Python made in Module 1. By default, a log of zero is not an error but a value, -inf, announced by a warning that a script can scroll past. A nan is not equal to itself, so the obvious test misses it. A float with a fraction poured into an integer array loses it, and an array of a small integer type wraps around at its limit, both in silence. None of these are bugs in NumPy, they are the rules of fixed-type arithmetic made visible.

What carries forward is the habit of checking the output of every transform as deliberately as its input: the dtype after the arithmetic, np.isfinite after a log, the range of the type before adding or multiplying. Lesson 2.2.4 turns the single number against an array into a general rule, broadcasting, which lets a whole row of values meet a whole table in the same one-line way, along with a new and quieter way for the shapes to pair wrongly.

Cheat Sheet

Part A - Reference Table

Part B - Compact Code Block

import numpy as np

prices = np.array([310000, 455000, 289000])
area = np.array([72.5, 110.0, 64.0])

prices / area                    # element by element
prices // 1000                   # int64, fraction dropped
np.log1p(prices)                 # log(1 + x), safe at 0
np.round(prices / area, 1)       # halves go to even
np.clip(prices, 280000, 450000)  # cap a range

np.isnan(x), np.isfinite(x)      # never x == np.nan
(~np.isfinite(x)).sum()          # count -inf and nan

f = prices.astype(float)
f *= 1.05                        # in place, now it fits

b = np.expm1(np.log1p(prices))   # undo log1p
np.allclose(b, prices)           # True, array_equal is exact