Introduction to numpy:



Package for scientific computing with Python


Numerical Python, or "Numpy" for short, is a foundational package on which many of the most common data science packages are built. Numpy provides us with high performance multi-dimensional arrays which we can use as vectors or matrices.

The key features of numpy are:

  • ndarrays: n-dimensional arrays of the same data type which are fast and space-efficient. There are a number of built-in methods for ndarrays which allow for rapid processing of data without using loops (e.g., compute the mean).
  • Broadcasting: a useful tool which defines implicit behavior between multi-dimensional arrays of different sizes.
  • Vectorization: enables numeric operations on ndarrays.
  • Input/Output: simplifies reading and writing of data from/to file.

Additional Recommended Resources:
Numpy Documentation
Python for Data Analysis by Wes McKinney
Python Data science Handbook by Jake VanderPlas


Getting started with ndarray

ndarrays are time and space-efficient multidimensional arrays at the core of numpy. Like the data structures in Week 2, let's get started by creating ndarrays using the numpy package.


How to create Rank 1 numpy arrays:

In [1]:
import numpy as np

an_array = np.array([3, 33, 333])  # Create a rank 1 array

print(type(an_array))              # The type of an ndarray is: "<class 'numpy.ndarray'>"

<class 'numpy.ndarray'>
In [2]:
# test the shape of the array we just created, it should have just one dimension (Rank 1)
print(an_array.shape)

(3,)
In [3]:
# because this is a 1-rank array, we need only one index to accesss each element
print(an_array[0], an_array[1], an_array[2]) 

3 33 333
In [4]:
an_array[0] =888            # ndarrays are mutable, here we change an element of the array

print(an_array)

[888  33 333]


How to create a Rank 2 numpy array:

A rank 2 ndarray is one with two dimensions. Notice the format below of [ [row] , [row] ]. 2 dimensional arrays are great for representing matrices which are often useful in data science.

In [5]:
another = np.array([[11,12,13],[21,22,23]])   # Create a rank 2 array

print(another)  # print the array

print("The shape is 2 rows, 3 columns: ", another.shape)  # rows x columns                   

print("Accessing elements [0,0], [0,1], and [1,0] of the ndarray: ", another[0, 0], ", ",another[0, 1],", ", another[1, 0])

[[11 12 13]
 [21 22 23]]
The shape is 2 rows, 3 columns:  (2, 3)
Accessing elements [0,0], [0,1], and [1,0] of the ndarray:  11 ,  12 ,  21


There are many way to create numpy arrays:

Here we create a number of different size arrays with different shapes and different pre-filled values. numpy has a number of built in methods which help us quickly and easily create multidimensional arrays.

In [6]:
import numpy as np

# create a 2x2 array of zeros
ex1 = np.zeros((2,2))      
print(ex1)                              

[[0. 0.]
 [0. 0.]]
In [7]:
# create a 2x2 array filled with 9.0
ex2 = np.full((2,2), 9.0)  
print(ex2)   

[[9. 9.]
 [9. 9.]]
In [8]:
# create a 2x2 matrix with the diagonal 1s and the others 0
ex3 = np.eye(2,2)
print(ex3)  

[[1. 0.]
 [0. 1.]]
In [9]:
# create an array of ones
ex4 = np.ones((1,2))
print(ex4)    

[[1. 1.]]
In [10]:
# notice that the above ndarray (ex4) is actually rank 2, it is a 2x1 array
print(ex4.shape)

# which means we need to use two indexes to access an element
print()
print(ex4[0,1])

(1, 2)

1.0
In [11]:
# create an array of random floats between 0 and 1
ex5 = np.random.random((2,2))
print(ex5)    

[[0.32146023 0.69134539]
 [0.49021038 0.97616247]]


Array Indexing


Slice indexing:

Similar to the use of slice indexing with lists and strings, we can use slice indexing to pull out sub-regions of ndarrays.

In [12]:
import numpy as np

# Rank 2 array of shape (3, 4)
an_array = np.array([[11,12,13,14], [21,22,23,24], [31,32,33,34]])
print(an_array)

[[11 12 13 14]
 [21 22 23 24]
 [31 32 33 34]]

Use array slicing to get a subarray consisting of the first 2 rows x 2 columns.

In [13]:
a_slice = an_array[:2, 1:3]
print(a_slice)

[[12 13]
 [22 23]]

When you modify a slice, you actually modify the underlying array.

In [14]:
print("Before:", an_array[0, 1])   #inspect the element at 0, 1  
a_slice[0, 0] = 1000    # a_slice[0, 0] is the same piece of data as an_array[0, 1]
print("After:", an_array[0, 1])    

Before: 12
After: 1000


Use both integer indexing & slice indexing

We can use combinations of integer indexing and slice indexing to create different shaped matrices.

In [15]:
# Create a Rank 2 array of shape (3, 4)
an_array = np.array([[11,12,13,14], [21,22,23,24], [31,32,33,34]])
print(an_array)

[[11 12 13 14]
 [21 22 23 24]
 [31 32 33 34]]
In [16]:
# Using both integer indexing & slicing generates an array of lower rank
row_rank1 = an_array[1, :]    # Rank 1 view 

print(row_rank1, row_rank1.shape)  # notice only a single []

[21 22 23 24] (4,)
In [17]:
# Slicing alone: generates an array of the same rank as the an_array
row_rank2 = an_array[1:2, :]  # Rank 2 view 

print(row_rank2, row_rank2.shape)   # Notice the [[ ]]

[[21 22 23 24]] (1, 4)
In [18]:
#We can do the same thing for columns of an array:

print()
col_rank1 = an_array[:, 1]
col_rank2 = an_array[:, 1:2]

print(col_rank1, col_rank1.shape)  # Rank 1
print()
print(col_rank2, col_rank2.shape)  # Rank 2

[12 22 32] (3,)

[[12]
 [22]
 [32]] (3, 1)


Array Indexing for changing elements:

Sometimes it's useful to use an array of indexes to access or change elements.

In [19]:
# Create a new array
an_array = np.array([[11,12,13], [21,22,23], [31,32,33], [41,42,43]])

print('Original Array:')
print(an_array)

Original Array:
[[11 12 13]
 [21 22 23]
 [31 32 33]
 [41 42 43]]
In [20]:
# Create an array of indices
col_indices = np.array([0, 1, 2, 0])
print('\nCol indices picked : ', col_indices)

row_indices = np.arange(4)
print('\nRows indices picked : ', row_indices)

Col indices picked :  [0 1 2 0]

Rows indices picked :  [0 1 2 3]
In [21]:
# Examine the pairings of row_indices and col_indices.  These are the elements we'll change next.
for row,col in zip(row_indices,col_indices):
    print(row, ", ",col)

0 ,  0
1 ,  1
2 ,  2
3 ,  0
In [22]:
# Select one element from each row
print('Values in the array at those indices: ',an_array[row_indices, col_indices])

Values in the array at those indices:  [11 22 33 41]
In [23]:
# Change one element from each row using the indices selected
an_array[row_indices, col_indices] += 100000

print('\nChanged Array:')
print(an_array)

Changed Array:
[[100011     12     13]
 [    21 100022     23]
 [    31     32 100033]
 [100041     42     43]]


Boolean Indexing


Array Indexing for changing elements:

In [24]:
# create a 3x2 array
an_array = np.array([[11,12], [21, 22], [31, 32]])
print(an_array)

[[11 12]
 [21 22]
 [31 32]]
In [25]:
# create a filter which will be boolean values for whether each element meets this condition
filter = (an_array > 15)
filter
Out[25]:
array([[False, False],
       [ True,  True],
       [ True,  True]])

Notice that the filter is a same size ndarray as an_array which is filled with True for each element whose corresponding element in an_array which is greater than 15 and False for those elements whose value is less than 15.

In [26]:
# we can now select just those elements which meet that criteria
print(an_array[filter])

[21 22 31 32]
In [27]:
# For short, we could have just used the approach below without the need for the separate filter array.

an_array[(an_array % 2 == 0)]
Out[27]:
array([12, 22, 32])

What is particularly useful is that we can actually change elements in the array applying a similar logical filter. Let's add 100 to all the even values.

In [28]:
an_array[an_array % 2 == 0] +=100
print(an_array)

[[ 11 112]
 [ 21 122]
 [ 31 132]]


Datatypes and Array Operations


Datatypes:

In [29]:
ex1 = np.array([11, 12]) # Python assigns the  data type
print(ex1.dtype)

int64
In [30]:
ex2 = np.array([11.0, 12.0]) # Python assigns the  data type
print(ex2.dtype)

float64
In [31]:
ex3 = np.array([11, 21], dtype=np.int64) #You can also tell Python the  data type
print(ex3.dtype)

int64
In [32]:
# you can use this to force floats into integers (using floor function)
ex4 = np.array([11.1,12.7], dtype=np.int64)
print(ex4.dtype)
print()
print(ex4)

int64

[11 12]
In [33]:
# you can use this to force integers into floats if you anticipate
# the values may change to floats later
ex5 = np.array([11, 21], dtype=np.float64)
print(ex5.dtype)
print()
print(ex5)

float64

[11. 21.]


Arithmetic Array Operations:

In [34]:
x = np.array([[111,112],[121,122]], dtype=np.int)
y = np.array([[211.1,212.1],[221.1,222.1]], dtype=np.float64)

print(x)
print()
print(y)

[[111 112]
 [121 122]]

[[211.1 212.1]
 [221.1 222.1]]
In [35]:
# add
print(x + y)         # The plus sign works
print()
print(np.add(x, y))  # so does the numpy function "add"

[[322.1 324.1]
 [342.1 344.1]]

[[322.1 324.1]
 [342.1 344.1]]
In [36]:
# subtract
print(x - y)
print()
print(np.subtract(x, y))

[[-100.1 -100.1]
 [-100.1 -100.1]]

[[-100.1 -100.1]
 [-100.1 -100.1]]
In [37]:
# multiply
print(x * y)
print()
print(np.multiply(x, y))

[[23432.1 23755.2]
 [26753.1 27096.2]]

[[23432.1 23755.2]
 [26753.1 27096.2]]
In [38]:
# divide
print(x / y)
print()
print(np.divide(x, y))

[[0.52581715 0.52805281]
 [0.54726368 0.54930212]]

[[0.52581715 0.52805281]
 [0.54726368 0.54930212]]
In [39]:
# square root
print(np.sqrt(x))

[[10.53565375 10.58300524]
 [11.         11.04536102]]
In [40]:
# exponent (e ** x)
print(np.exp(x))

[[1.60948707e+48 4.37503945e+48]
 [3.54513118e+52 9.63666567e+52]]


Statistical Methods, Sorting, and

Set Operations:


Basic Statistical Operations:

In [41]:
# setup a random 2 x 4 matrix
arr = 10 * np.random.randn(2,5)
print(arr)

[[ -3.03923239 -13.92622025  10.32790286   6.09696545   9.6381842 ]
 [-17.04049434  -6.62802851  -7.51259611   1.3968755   -9.54392327]]
In [42]:
# compute the mean for all elements
print(arr.mean())

-3.0230566865534216
In [43]:
# compute the means by row
print(arr.mean(axis = 1))

[ 1.81951997 -7.86563335]
In [44]:
# compute the means by column
print(arr.mean(axis = 0))

[-10.03986336 -10.27712438   1.40765337   3.74692047   0.04713046]
In [45]:
# sum all the elements
print(arr.sum())

-30.230566865534215
In [46]:
# compute the medians
print(np.median(arr, axis = 1))

[ 6.09696545 -7.51259611]


Sorting:

In [47]:
# create a 10 element array of randoms
unsorted = np.random.randn(10)

print(unsorted)

[-0.17506183  0.39747066 -1.02875593  0.48305786 -0.51714677 -0.90935873
  0.71749501 -0.04579488  1.06977992  0.45646233]
In [48]:
# create copy and sort
sorted = np.array(unsorted)
sorted.sort()

print(sorted)
print()
print(unsorted)

[-1.02875593 -0.90935873 -0.51714677 -0.17506183 -0.04579488  0.39747066
  0.45646233  0.48305786  0.71749501  1.06977992]

[-0.17506183  0.39747066 -1.02875593  0.48305786 -0.51714677 -0.90935873
  0.71749501 -0.04579488  1.06977992  0.45646233]
In [49]:
# inplace sorting
unsorted.sort() 

print(unsorted)

[-1.02875593 -0.90935873 -0.51714677 -0.17506183 -0.04579488  0.39747066
  0.45646233  0.48305786  0.71749501  1.06977992]


Finding Unique elements:

In [50]:
array = np.array([1,2,1,4,2,1,4,2])

print(np.unique(array))

[1 2 4]


Set Operations with np.array data type:

In [51]:
s1 = np.array(['desk','chair','bulb'])
s2 = np.array(['lamp','bulb','chair'])
print(s1, s2)

['desk' 'chair' 'bulb'] ['lamp' 'bulb' 'chair']
In [52]:
print( np.intersect1d(s1, s2) ) 

['bulb' 'chair']
In [53]:
print( np.union1d(s1, s2) )

['bulb' 'chair' 'desk' 'lamp']
In [54]:
print( np.setdiff1d(s1, s2) )# elements in s1 that are not in s2

['desk']
In [55]:
print( np.in1d(s1, s2) )#which element of s1 is also in s2

[False  True  True]


Broadcasting:

Introduction to broadcasting.
For more details, please see:
https://docs.scipy.org/doc/numpy-1.10.1/user/basics.broadcasting.html

In [56]:
import numpy as np

start = np.zeros((4,3))
print(start)

[[0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]
 [0. 0. 0.]]
In [57]:
# create a rank 1 ndarray with 3 values
add_rows = np.array([1, 0, 2])
print(add_rows)

[1 0 2]
In [58]:
y = start + add_rows  # add to each row of 'start' using broadcasting
print(y)

[[1. 0. 2.]
 [1. 0. 2.]
 [1. 0. 2.]
 [1. 0. 2.]]
In [59]:
# create an ndarray which is 4 x 1 to broadcast across columns
add_cols = np.array([[0,1,2,3]])
add_cols = add_cols.T

print(add_cols)

[[0]
 [1]
 [2]
 [3]]
In [60]:
# add to each column of 'start' using broadcasting
y = start + add_cols 
print(y)

[[0. 0. 0.]
 [1. 1. 1.]
 [2. 2. 2.]
 [3. 3. 3.]]
In [61]:
# this will just broadcast in both dimensions
add_scalar = np.array([1])  
print(start+add_scalar)

[[1. 1. 1.]
 [1. 1. 1.]
 [1. 1. 1.]
 [1. 1. 1.]]

Example from the slides:

In [62]:
# create our 3x4 matrix
arrA = np.array([[1,2,3,4],[5,6,7,8],[9,10,11,12]])
print(arrA)

[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]]
In [63]:
# create our 4x1 array
arrB = [0,1,0,2]
print(arrB)

[0, 1, 0, 2]
In [64]:
# add the two together using broadcasting
print(arrA + arrB)

[[ 1  3  3  6]
 [ 5  7  7 10]
 [ 9 11 11 14]]


Speedtest: ndarrays vs lists

First setup paramaters for the speed test. We'll be testing time to sum elements in an ndarray versus a list.

In [65]:
from numpy import arange
from timeit import Timer

size    = 1000000
timeits = 1000
In [66]:
# create the ndarray with values 0,1,2...,size-1
nd_array = arange(size)
print( type(nd_array) )

<class 'numpy.ndarray'>
In [67]:
# timer expects the operation as a parameter, 
# here we pass nd_array.sum()
timer_numpy = Timer("nd_array.sum()", "from __main__ import nd_array")

print("Time taken by numpy ndarray: %f seconds" % 
      (timer_numpy.timeit(timeits)/timeits))

Time taken by numpy ndarray: 0.001013 seconds
In [68]:
# create the list with values 0,1,2...,size-1
a_list = list(range(size))
print (type(a_list) )

<class 'list'>
In [69]:
# timer expects the operation as a parameter, here we pass sum(a_list)
timer_list = Timer("sum(a_list)", "from __main__ import a_list")

print("Time taken by list:  %f seconds" % 
      (timer_list.timeit(timeits)/timeits))

Time taken by list:  0.008342 seconds


Read or Write to Disk:


Binary Format:

In [70]:
x = np.array([ 23.23, 24.24] )
In [71]:
np.save('an_array', x)
In [72]:
np.load('an_array.npy')
Out[72]:
array([23.23, 24.24])


Text Format:

In [73]:
np.savetxt('array.txt', X=x, delimiter=',')
In [74]:
!cat array.txt

2.323000000000000043e+01
2.423999999999999844e+01
In [75]:
np.loadtxt('array.txt', delimiter=',')
Out[75]:
array([23.23, 24.24])


Additional Common ndarray Operations


Dot Product on Matrices and Inner Product on Vectors:

In [76]:
# determine the dot product of two matrices
x2d = np.array([[1,1],[1,1]])
y2d = np.array([[2,2],[2,2]])

print(x2d.dot(y2d))
print()
print(np.dot(x2d, y2d))

[[4 4]
 [4 4]]

[[4 4]
 [4 4]]
In [77]:
# determine the inner product of two vectors
a1d = np.array([9 , 9 ])
b1d = np.array([10, 10])

print(a1d.dot(b1d))
print()
print(np.dot(a1d, b1d))

180

180
In [78]:
# dot produce on an array and vector
print(x2d.dot(a1d))
print()
print(np.dot(x2d, a1d))

[18 18]

[18 18]


Sum:

In [79]:
# sum elements in the array
ex1 = np.array([[11,12],[21,22]])

print(np.sum(ex1))          # add all members

66
In [80]:
print(np.sum(ex1, axis=0))  # columnwise sum

[32 34]
In [81]:
print(np.sum(ex1, axis=1))  # rowwise sum

[23 43]


Element-wise Functions:

For example, let's compare two arrays values to get the maximum of each.

In [82]:
# random array
x = np.random.randn(8)
x
Out[82]:
array([ 0.01670389,  0.83009752,  0.36429119, -0.02537186,  0.67936302,
       -1.22053292,  0.98126483, -1.63370793])
In [83]:
# another random array
y = np.random.randn(8)
y
Out[83]:
array([ 0.89911683, -0.05790972,  0.94732681,  1.2332476 ,  0.47534952,
        0.37222385, -0.3721468 ,  0.97277289])
In [84]:
# returns element wise maximum between two arrays

np.maximum(x, y)
Out[84]:
array([0.89911683, 0.83009752, 0.94732681, 1.2332476 , 0.67936302,
       0.37222385, 0.98126483, 0.97277289])


Reshaping array:

In [85]:
# grab values from 0 through 19 in an array
arr = np.arange(20)
print(arr)

[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19]
In [86]:
# reshape to be a 4 x 5 matrix
arr.reshape(4,5)
Out[86]:
array([[ 0,  1,  2,  3,  4],
       [ 5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14],
       [15, 16, 17, 18, 19]])


Transpose:

In [87]:
# transpose
ex1 = np.array([[11,12],[21,22]])

ex1.T
Out[87]:
array([[11, 21],
       [12, 22]])


Indexing using where():

In [88]:
x_1 = np.array([1,2,3,4,5])

y_1 = np.array([11,22,33,44,55])

filter = np.array([True, False, True, False, True])
In [89]:
out = np.where(filter, x_1, y_1)
print(out)

[ 1 22  3 44  5]
In [90]:
mat = np.random.rand(5,5)
mat
Out[90]:
array([[0.28284189, 0.30527349, 0.68693739, 0.49736572, 0.74071059],
       [0.61684323, 0.48519612, 0.14083083, 0.84979016, 0.28013682],
       [0.52596907, 0.97263873, 0.74012983, 0.78436145, 0.59322462],
       [0.51077996, 0.63824162, 0.74076584, 0.58948876, 0.42548216],
       [0.73488872, 0.6589017 , 0.81100903, 0.90581186, 0.82099517]])
In [91]:
np.where( mat > 0.5, 1000, -1)
Out[91]:
array([[  -1,   -1, 1000,   -1, 1000],
       [1000,   -1,   -1, 1000,   -1],
       [1000, 1000, 1000, 1000, 1000],
       [1000, 1000, 1000, 1000,   -1],
       [1000, 1000, 1000, 1000, 1000]])


"any" or "all" conditionals:

In [92]:
arr_bools = np.array([ True, False, True, True, False ])
In [93]:
arr_bools.any()
Out[93]:
True
In [94]:
arr_bools.all()
Out[94]:
False


Random Number Generation:

In [95]:
Y = np.random.normal(size = (1,5))[0]
print(Y)

[-1.92560045  1.60358532  0.49160621  0.30087783  0.04338372]
In [96]:
Z = np.random.randint(low=2,high=50,size=4)
print(Z)

[ 4 25 16 34]
In [97]:
np.random.permutation(Z) #return a new ordering of elements in Z
Out[97]:
array([ 4, 16, 25, 34])
In [98]:
np.random.uniform(size=4) #uniform distribution
Out[98]:
array([0.45725956, 0.71088892, 0.14536692, 0.34618903])
In [99]:
np.random.normal(size=4) #normal distribution
Out[99]:
array([ 1.45373053, -0.3538088 , -0.11962385, -0.71401568])


Merging data sets:

In [100]:
K = np.random.randint(low=2,high=50,size=(2,2))
print(K)

print()
M = np.random.randint(low=2,high=50,size=(2,2))
print(M)

[[26 14]
 [ 2 31]]

[[13 46]
 [ 9 13]]
In [101]:
np.vstack((K,M))
Out[101]:
array([[26, 14],
       [ 2, 31],
       [13, 46],
       [ 9, 13]])
In [102]:
np.hstack((K,M))
Out[102]:
array([[26, 14, 13, 46],
       [ 2, 31,  9, 13]])
In [103]:
np.concatenate([K, M], axis = 0)
Out[103]:
array([[26, 14],
       [ 2, 31],
       [13, 46],
       [ 9, 13]])
In [104]:
np.concatenate([K, M.T], axis = 1)
Out[104]:
array([[26, 14, 13,  9],
       [ 2, 31, 46, 13]])