Introduction to Python Programming and Numpy

This tutorial was inspired by Introduction to data science in python

The Python Programming Language: Functions

x = 1
y = 2
x + y

add_numbers is a function that takes two numbers and adds them together.

def add_numbers(x, y):
    return x + y

add_numbers(1, 2)

add_numbers updated to take an optional 3rd parameter. Using print allows printing of multiple expressions within a single cell. If you give None as parameter value then it becomes optional.

def add_numbers(x,y,z=None):
    if (z==None):
        return x+y
    else:
        return x+y+z

print(add_numbers(1, 2))
print(add_numbers(1, 2, 3))

3
6

add_numbers updated to take an optional flag parameter.

def add_numbers(x, y, z=None, flag=False):
    if (flag):
        print('Flag is true!')
    if (z==None):
        return x + y
    else:
        return x + y + z

print(add_numbers(1, 2, flag=True))

Flag is true!
3

Assign function add_numbers to variable a.

def add_numbers(x,y):
    return x+y

a = add_numbers
a(1,2)

The Python Programming Language: Types and Sequences

Use type to return the object's type.

type('This is a string')

str

type(None)

NoneType

type(1)

int

type(1.0)

float

type(add_numbers)

function

Tuples are an immutable data structure (cannot be altered).

x = (1, 'a', 2, 'b')
type(x)

tuple

Lists are a mutable data structure.

x = [1, 'a', 2, 'b']
type(x)

list

Use append to append an object to a list.

x.append(3.3)
print(x)

[1, 'a', 2, 'b', 3.3]

This is an example of how to loop through each item in the list.

for item in x:
    print(item)

1
a
2
b
3.3

Or using the indexing operator:

i=0
while( i != len(x) ):
    print(x[i])
    i = i + 1

1
a
2
b
3.3

Use + to concatenate lists.

[1,2] + [3,4]

[1, 2, 3, 4]

Use * to repeat lists.

[1]*3

[1, 1, 1]

Use the in operator to check if something is inside a list. in operator returns boolean value.

1 in [1, 2, 3]

True

Now let's look at strings. Use bracket notation to slice a string. In slicing x[0:3] so it shows from first start index which in 0 till the last index 3. It's upto index 3 and does not include index 3 in results .!

x = 'This is a string'
print(x[0]) #first character
print(x[0:1]) #first character, but we have explicitly set the end character
print(x[0:2]) #first two characters

T
T
Th

This will return the last element of the string.

x[-1]

'g'

This will return the slice starting from the 4th element from the end and stopping before the 2nd element from the end.

x[-4:-2]

'ri'

This is a slice from the beginning of the string and stopping before the 3rd element.

x[:3]

'Thi'

And this is a slice starting from the 3rd element of the string and going all the way to the end.

x[3:]

's is a string'

firstname = 'Prashant'
lastname = 'Gonarkar'

print(firstname + ' ' + lastname)
print(firstname*3)
print('Chris' in firstname)

Prashant Gonarkar
PrashantPrashantPrashant
False

split returns a list of all the words in a string, or a list split on a specific character.

firstname = 'Prashant Gonarkar Hansen Brooks'.split(' ')[0] # [0] selects the first element of the list
lastname = 'Christopher Arthur Prashant Gonarkar'.split(' ')[-1] # [-1] selects the last element of the list
print(firstname)
print(lastname)

Prashant
Gonarkar

Make sure you convert objects to strings before concatenating.

'Prashant' + 2

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-28-030617db6c26> in <module>()
----> 1 'Prashant' + 2


TypeError: Can't convert 'int' object to str implicitly

'Prashant' + str(2)

'Prashant2'

Dictionaries associate keys with values.

x = {'Prashant Gonarkar': 'prashantgonarkar@gmail.com', 'Bill Gates': 'billg@microsoft.com'}
x['Prashant Gonarkar'] # Retrieve a value by using the indexing operator

'prashantgonarkar@gmail.com'

x['Kevyn Collins-Thompson'] = None
x['Kevyn Collins-Thompson']

Iterate over all of the keys:

for name in x:
    print(x[name])

billg@microsoft.com
prashantgonarkar@gmail.com
None

Iterate over all of the values:

for email in x.values():
    print(email)

billg@microsoft.com
prashantgonarkar@gmail.com
None

Iterate over all of the items in the list:

for name, email in x.items():
    print(name)
    print(email)

Bill Gates
billg@microsoft.com
Prashant Gonarkar
prashantgonarkar@gmail.com
Kevyn Collins-Thompson
None

You can unpack a sequence into different variables:

x = ('Christopher', 'Brooks', 'brooksch@umich.edu')
fname, lname, email = x

fname

'Christopher'

lname

'Brooks'

Make sure the number of values you are unpacking matches the number of variables being assigned.

x = ('Christopher', 'Brooks', 'brooksch@umich.edu', 'Ann Arbor')
fname, lname, email = x

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-40-9ce70064f53e> in <module>()
      1 x = ('Christopher', 'Brooks', 'brooksch@umich.edu', 'Ann Arbor')
----> 2 fname, lname, email = x


ValueError: too many values to unpack (expected 3)

The Python Programming Language: More on Strings

print('Chris' + 2)

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-41-82ccfdd3d5d3> in <module>()
----> 1 print('Chris' + 2)


TypeError: Can't convert 'int' object to str implicitly

print('Chris' + str(2))

Chris2

Python has a built in method for convenient string formatting.

sales_record = {
'price': 3.24,
'num_items': 4,
'person': 'Prashant'}

sales_statement = '{} bought {} item(s) at a price of {} each for a total of {}'

print(sales_statement.format(sales_record['person'],
                             sales_record['num_items'],
                             sales_record['price'],
                             sales_record['num_items']*sales_record['price']))

Prashant bought 4 item(s) at a price of 3.24 each for a total of 12.96

Reading and Writing CSV files

Let's import our datafile mpg.csv, which contains fuel economy data for 234 cars.

mpg : miles per gallon
class : car classification
cty : city mpg
cyl : # of cylinders
displ : engine displacement in liters
drv : f = front-wheel drive, r = rear wheel drive, 4 = 4wd
fl : fuel (e = ethanol E85, d = diesel, r = regular, p = premium, c = CNG)
hwy : highway mpg
manufacturer : automobile manufacturer
model : model of car
trans : type of transmission
year : model year

import csv

%precision 2

with open('mpg.csv') as csvfile:
    mpg = list(csv.DictReader(csvfile))

mpg[:3] # The first three dictionaries in our list.

[{'': '1',
  'class': 'compact',
  'cty': '18',
  'cyl': '4',
  'displ': '1.8',
  'drv': 'f',
  'fl': 'p',
  'hwy': '29',
  'manufacturer': 'audi',
  'model': 'a4',
  'trans': 'auto(l5)',
  'year': '1999'},
 {'': '2',
  'class': 'compact',
  'cty': '21',
  'cyl': '4',
  'displ': '1.8',
  'drv': 'f',
  'fl': 'p',
  'hwy': '29',
  'manufacturer': 'audi',
  'model': 'a4',
  'trans': 'manual(m5)',
  'year': '1999'},
 {'': '3',
  'class': 'compact',
  'cty': '20',
  'cyl': '4',
  'displ': '2',
  'drv': 'f',
  'fl': 'p',
  'hwy': '31',
  'manufacturer': 'audi',
  'model': 'a4',
  'trans': 'manual(m6)',
  'year': '2008'}]

csv.Dictreader has read in each row of our csv file as a dictionary. len shows that our list is comprised of 234 dictionaries.

len(mpg)

keys gives us the column names of our csv.

mpg[0].keys()

dict_keys(['', 'class', 'cty', 'drv', 'displ', 'model', 'manufacturer', 'fl', 'year', 'trans', 'hwy', 'cyl'])

This is how to find the average cty fuel economy across all cars. All values in the dictionaries are strings, so we need to convert to float.

sum(float(d['cty']) for d in mpg) / len(mpg)

16.86

Similarly this is how to find the average hwy fuel economy across all cars.

sum(float(d['hwy']) for d in mpg) / len(mpg)

23.44

Use set to return the unique values for the number of cylinders the cars in our dataset have.

cylinders = set(d['cyl'] for d in mpg)
cylinders

{'4', '5', '6', '8'}

Here's a more complex example where we are grouping the cars by number of cylinder, and finding the average cty mpg for each group.

CtyMpgByCyl = []

for c in cylinders: # iterate over all the cylinder levels
    summpg = 0
    cyltypecount = 0
    for d in mpg: # iterate over all dictionaries
        if d['cyl'] == c: # if the cylinder level type matches,
            summpg += float(d['cty']) # add the cty mpg
            cyltypecount += 1 # increment the count
    CtyMpgByCyl.append((c, summpg / cyltypecount)) # append the tuple ('cylinder', 'avg mpg')

CtyMpgByCyl.sort(key=lambda x: x[0])
CtyMpgByCyl

[('4', 21.01), ('5', 20.50), ('6', 16.22), ('8', 12.57)]

Use set to return the unique values for the class types in our dataset.

vehicleclass = set(d['class'] for d in mpg) # what are the class types
vehicleclass

{'2seater', 'compact', 'midsize', 'minivan', 'pickup', 'subcompact', 'suv'}

And here's an example of how to find the average hwy mpg for each class of vehicle in our dataset.

HwyMpgByClass = []

for t in vehicleclass: # iterate over all the vehicle classes
    summpg = 0
    vclasscount = 0
    for d in mpg: # iterate over all dictionaries
        if d['class'] == t: # if the cylinder amount type matches,
            summpg += float(d['hwy']) # add the hwy mpg
            vclasscount += 1 # increment the count
    HwyMpgByClass.append((t, summpg / vclasscount)) # append the tuple ('class', 'avg mpg')

HwyMpgByClass.sort(key=lambda x: x[1])
HwyMpgByClass

[('pickup', 16.88),
 ('suv', 18.13),
 ('minivan', 22.36),
 ('2seater', 24.80),
 ('midsize', 27.29),
 ('subcompact', 28.14),
 ('compact', 28.30)]

The Python Programming Language: Dates and Times

import datetime as dt
import time as tm

time returns the current time in seconds since the Epoch. (January 1st, 1970)

tm.time()

1480778574.73

Convert the timestamp to datetime.

dtnow = dt.datetime.fromtimestamp(tm.time())
dtnow

datetime.datetime(2016, 12, 3, 15, 23, 14, 372250)

Handy datetime attributes:

dtnow.year, dtnow.month, dtnow.day, dtnow.hour, dtnow.minute, dtnow.second # get year, month, day, etc.from a datetime

(2016, 12, 3, 15, 23, 14)

timedelta is a duration expressing the difference between two dates.

delta = dt.timedelta(days = 100) # create a timedelta of 100 days
delta

datetime.timedelta(100)

date.today returns the current local date.

today = dt.date.today()

today - delta # the date 100 days ago

datetime.date(2016, 8, 25)

today > today-delta # compare dates

True

The Python Programming Language: Objects and map()

An example of a class in python: There are no private and protected members in python. Everything can be accessed through object instance. Also there is no need of explicit constructor.

class Person:
    department = 'School of Information' #a class variable

    def set_name(self, new_name): #a method
        self.name = new_name
    def set_location(self, new_location):
        self.location = new_location

person = Person()
person.set_name('Christopher Brooks')
person.set_location('Ann Arbor, MI, USA')
print('{} live in {} and works in the department {}'.format(person.name, person.location, person.department))

Christopher Brooks live in Ann Arbor, MI, USA and works in the department School of Information

Here's an example of mapping the min function between two lists. For map(function, interatable1,..) first argument is function and other subsquent arguments are interatables such as list or set. Map returns an object with lazy evaluation. It doesn't run until you accessed that item

store1 = [10.00, 11.00, 12.34, 2.34]
store2 = [9.00, 11.10, 12.34, 2.01]
cheapest = map(min, store1, store2)
cheapest

<map at 0x7fe11c10b898>

Now let's iterate through the map object to see the values.

for item in cheapest:
    print(item)

9.0
11.0
12.34
2.01

The Python Programming Language: Lambda and List Comprehensions

Here's an example of lambda that takes in three parameters and adds the first two.

my_function = lambda a, b, c : a + b

my_function(1, 2, 3) 
# video lecture assignment
people = ['Dr. Christopher Brooks', 'Dr. Kevyn Collins-Thompson', 'Dr. VG Vinod Vydiswaran', 'Dr. Daniel Romero']

def split_title_and_name(person):
    return person.split()[0] + ' ' + person.split()[-1]

#option 1
for person in people:
    print(split_title_and_name(person) == (lambda x: x.split()[0] + ' ' + x.split()[-1])(person))

#option 2
list(map(split_title_and_name, people)) == list(map(lambda person: person.split()[0] + ' ' + person.split()[-1], people))

True
True
True
True





True

Let's iterate from 0 to 999 and return the even numbers.

my_list = []
for number in range(0, 1000):
    if number % 2 == 0:
        my_list.append(number)
my_list[:10]

[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

Now the same thing but with list comprehension.

my_list = [number for number in range(0,1000) if number % 2 == 0]
my_list[:10]
# video lecture assignment-1
def times_tables():
    lst = []
    for i in range(10):
        for j in range (10):
            lst.append(i*j)
    return lst

times_tables() == [j*i for i in range(10) for j in range(10)]
# video lecture assignment-2
lowercase = 'abcdefghijklmnopqrstuvwxyz'
digits = '0123456789'

correct_answer = [a+b+c+d for a in lowercase for b in lowercase for c in digits for d in digits]

correct_answer[:10] # Display first 10 ids

['aa00',
 'aa01',
 'aa02',
 'aa03',
 'aa04',
 'aa05',
 'aa06',
 'aa07',
 'aa08',
 'aa09']

The Python Programming Language: Numerical Python (NumPy)

import numpy as np

Creating Arrays

Create a list and convert it to a numpy array

mylist = [1, 2, 3]
x = np.array(mylist)
x

array([1, 2, 3])

Or just pass in a list directly

y = np.array([4, 5, 6])
y

array([4, 5, 6])

Pass in a list of lists to create a multidimensional array.

m = np.array([[7, 8, 9], [10, 11, 12]])
m

array([[ 7,  8,  9],
       [10, 11, 12]])

Use the shape method to find the dimensions of the array. (rows, columns)

m.shape

(2, 3)

arange returns evenly spaced values within a given interval.

n = np.arange(0, 30, 2) # start at 0 count up by 2, stop before 30
n

array([ 0,  2,  4,  6,  8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28])

reshape returns an array with the same data with a new shape.

n = n.reshape(3, 5) # reshape array to be 3x5
n

array([[ 0,  2,  4,  6,  8],
       [10, 12, 14, 16, 18],
       [20, 22, 24, 26, 28]])

linspace returns evenly spaced numbers over a specified interval.

o = np.linspace(0, 4, 9) # return 9 evenly spaced values from 0 to 4
o

array([ 0. ,  0.5,  1. ,  1.5,  2. ,  2.5,  3. ,  3.5,  4. ])

resize changes the shape and size of array in-place.

o.resize(3, 3)
o

array([[ 0. ,  0.5,  1. ],
       [ 1.5,  2. ,  2.5],
       [ 3. ,  3.5,  4. ]])

ones returns a new array of given shape and type, filled with ones.

np.ones((3, 2))

array([[ 1.,  1.],
       [ 1.,  1.],
       [ 1.,  1.]])

zeros returns a new array of given shape and type, filled with zeros.

np.zeros((2, 3))

array([[ 0.,  0.,  0.],
       [ 0.,  0.,  0.]])

eye returns a 2-D array with ones on the diagonal and zeros elsewhere.

np.eye(3)

array([[ 1.,  0.,  0.],
       [ 0.,  1.,  0.],
       [ 0.,  0.,  1.]])

diag extracts a diagonal or constructs a diagonal array.

np.diag(y)

array([[4, 0, 0],
       [0, 5, 0],
       [0, 0, 6]])

Create an array using repeating list (or see np.tile)

np.array([1, 2, 3] * 3)

array([1, 2, 3, 1, 2, 3, 1, 2, 3])

Repeat elements of an array using repeat.

np.repeat([1, 2, 3], 3)

array([1, 1, 1, 2, 2, 2, 3, 3, 3])

Combining Arrays

p = np.ones([2, 3], int)
p

array([[1, 1, 1],
       [1, 1, 1]])

Use vstack to stack arrays in sequence vertically (row wise).

np.vstack([p, 2*p])

array([[1, 1, 1],
       [1, 1, 1],
       [2, 2, 2],
       [2, 2, 2]])

Use hstack to stack arrays in sequence horizontally (column wise).

np.hstack([p, 2*p])

array([[1, 1, 1, 2, 2, 2],
       [1, 1, 1, 2, 2, 2]])

Operations

Use +, -, *, / and ** to perform element wise addition, subtraction, multiplication, division and power.

print(x + y) # elementwise addition     [1 2 3] + [4 5 6] = [5  7  9]
print(x - y) # elementwise subtraction  [1 2 3] - [4 5 6] = [-3 -3 -3]

[5 7 9]
[-3 -3 -3]

print(x * y) # elementwise multiplication  [1 2 3] * [4 5 6] = [4  10  18]
print(x / y) # elementwise divison         [1 2 3] / [4 5 6] = [0.25  0.4  0.5]

[ 4 10 18]
[ 0.25  0.4   0.5 ]

print(x**2) # elementwise power  [1 2 3] ^2 =  [1 4 9]

[1 4 9]

Dot Product:

$ \begin{bmatrix}x_1 \ x_2 \ x_3\end{bmatrix} \cdot \begin{bmatrix}y_1 \ y_2 \ y_3\end{bmatrix} = x_1 y_1 + x_2 y_2 + x_3 y_3$

x.dot(y) # dot product  1*4 + 2*5 + 3*6

z = np.array([y, y**2])
print(len(z)) # number of rows of array

Let's look at transposing arrays. Transposing permutes the dimensions of the array.

z = np.array([y, y**2])
z

array([[ 4,  5,  6],
       [16, 25, 36]])

The shape of array z is (2,3) before transposing.

z.shape

(2, 3)

Use .T to get the transpose.

z.T

array([[ 4, 16],
       [ 5, 25],
       [ 6, 36]])

The number of rows has swapped with the number of columns.

z.T.shape

(3, 2)

Use .dtype to see the data type of the elements in the array.

z.dtype

dtype('int64')

Use .astype to cast to a specific type.

z = z.astype('f')
z.dtype

dtype('float32')

Math Functions

Numpy has many built in math functions that can be performed on arrays.

a = np.array([-4, -2, 1, 3, 5])

a.sum()

a.max()

a.min()

-4

a.mean()

0.60

a.std()

3.26

argmax and argmin return the index of the maximum and minimum values in the array.

a.argmax()

a.argmin()

Indexing / Slicing

s = np.arange(13)**2
s

array([  0,   1,   4,   9,  16,  25,  36,  49,  64,  81, 100, 121, 144])

Use bracket notation to get the value at a specific index. Remember that indexing starts at 0.

s[0], s[4], s[-1]

(0, 16, 144)

Use : to indicate a range. array[start:stop]

Leaving start or stop empty will default to the beginning/end of the array.

s[1:5]

array([ 1,  4,  9, 16])

Use negatives to count from the back.

s[-4:]

array([ 81, 100, 121, 144])

A second : can be used to indicate step-size. array[start:stop:stepsize]

Here we are starting 5th element from the end, and counting backwards by 2 until the beginning of the array is reached.

s[-5::-2]

array([64, 36, 16,  4,  0])

Let's look at a multidimensional array.

r = np.arange(36)
r.resize((6, 6))
r

array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11],
       [12, 13, 14, 15, 16, 17],
       [18, 19, 20, 21, 22, 23],
       [24, 25, 26, 27, 28, 29],
       [30, 31, 32, 33, 34, 35]])

Use bracket notation to slice: array[row, column]

r[2, 2]

And use : to select a range of rows or columns

r[3, 3:6]

array([21, 22, 23])

Here we are selecting all the rows up to (and not including) row 2, and all the columns up to (and not including) the last column.

r[:2, :-1]

array([[ 0,  1,  2,  3,  4],
       [ 6,  7,  8,  9, 10]])

This is a slice of the last row, and only every other element.

r[-1, ::2]

array([30, 32, 34])

We can also perform conditional indexing. Here we are selecting values from the array that are greater than 30. (Also see np.where)

r[r > 30]

array([31, 32, 33, 34, 35])

Here we are assigning all values in the array that are greater than 30 to the value of 30.

r[r > 30] = 30
r

array([[ 0,  1,  2,  3,  4,  5],
       [ 6,  7,  8,  9, 10, 11],
       [12, 13, 14, 15, 16, 17],
       [18, 19, 20, 21, 22, 23],
       [24, 25, 26, 27, 28, 29],
       [30, 30, 30, 30, 30, 30]])

Copying Data

Be careful with copying and modifying arrays in NumPy!

r2 is a slice of r

r2 = r[:3,:3]
r2

array([[ 0,  1,  2],
       [ 6,  7,  8],
       [12, 13, 14]])

Set this slice's values to zero ([:] selects the entire array) key point- [:] => entire array

r2[:] = 0
r2

array([[0, 0, 0],
       [0, 0, 0],
       [0, 0, 0]])

r has also been changed!

array([[ 0,  0,  0,  3,  4,  5],
       [ 0,  0,  0,  9, 10, 11],
       [ 0,  0,  0, 15, 16, 17],
       [18, 19, 20, 21, 22, 23],
       [24, 25, 26, 27, 28, 29],
       [30, 30, 30, 30, 30, 30]])

To avoid this, use r.copy to create a copy that will not affect the original array

r_copy = r.copy()
r_copy

array([[ 0,  0,  0,  3,  4,  5],
       [ 0,  0,  0,  9, 10, 11],
       [ 0,  0,  0, 15, 16, 17],
       [18, 19, 20, 21, 22, 23],
       [24, 25, 26, 27, 28, 29],
       [30, 30, 30, 30, 30, 30]])

Now when r_copy is modified, r will not be changed.

r_copy[:] = 10
print(r_copy, '\n')
print(r)

[[10 10 10 10 10 10]
 [10 10 10 10 10 10]
 [10 10 10 10 10 10]
 [10 10 10 10 10 10]
 [10 10 10 10 10 10]
 [10 10 10 10 10 10]]

[[ 0  0  0  3  4  5]
 [ 0  0  0  9 10 11]
 [ 0  0  0 15 16 17]
 [18 19 20 21 22 23]
 [24 25 26 27 28 29]
 [30 30 30 30 30 30]]

Iterating Over Arrays

Let's create a new 4 by 3 array of random numbers 0-9.

test = np.random.randint(0, 10, (4,3))
test

array([[2, 4, 8],
       [5, 7, 2],
       [9, 5, 8],
       [7, 4, 8]])

Iterate by row:

for row in test:
    print(row)

[2 4 8]
[5 7 2]
[9 5 8]
[7 4 8]

Iterate by index:

for i in range(len(test)):
    print(test[i])

[2 4 8]
[5 7 2]
[9 5 8]
[7 4 8]

Iterate by row and index:

for i, row in enumerate(test):
    print('row', i, 'is', row)

row 0 is [2 4 8]
row 1 is [5 7 2]
row 2 is [9 5 8]
row 3 is [7 4 8]

Use zip to iterate over multiple iterables.

test2 = test**2
test2

array([[ 4, 16, 64],
       [25, 49,  4],
       [81, 25, 64],
       [49, 16, 64]])

for i, j in zip(test, test2):
    print(i,'+',j,'=',i+j)

[2 4 8] + [ 4 16 64] = [ 6 20 72]
[5 7 2] + [25 49  4] = [30 56  6]
[9 5 8] + [81 25 64] = [90 30 72]
[7 4 8] + [49 16 64] = [56 20 72]