Container data-types

In addition to the basic data types covered in the last section, Python has several built-in “containers”.

These come in two main types;

• Lists and Tuples : store a sequence of values
• Dictionaries : map keys to values

Lists

A list, as the name implies, is a sequence of items. For example, the numbers 1 – 10 could be arranged in a list: 1,2,3,4,5,6,7,8,9, and 10.

In Python, lists are created using square brackets, and items separated by commas, e.g.:

[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

List elements can be anything (including lists - which creates “nested lists”), strings, etc.

For now though, lets get to grips with lists by considering flat lists of simple data.

Manipulating lists

Lists can be manipulated using member-functions:

• grown using
  • insert (anywhere in the list)
  • append (always at the end),
  • extend to add multiple items at the end
• shortened, using
  • pop (a specific index - without any number the last element will be removed)
  • remove (a specific value)
• sorted using sort
• searched using index

Lists can also be combined using the plus operator, +, e.g.

[1,2,3] + [4,5] 

results in the list [1,2,3,4,5]

The length of a list can be determined using the built-in function len.

Note

len is not a member function of a list, i.e. you cannot call AListVariable.len()

Instead, use

len(AListVariable) 

to return the length of a list

Anything that can be “iterated” over or cycled through, can be converted into a list using the list function.

Python also contains a built-in function called range which is a convenient way of generating a range of numbers (as an iterable that can be converted into a list):

list(range(10)) 

is the same as

[0,1,2,3,4,5,6,7,8,9]

Exercise : List manipulation

Write a script (name the file exercise_lists.py) to perform the following operations (this is just to test them all out!):

1. Create a variable that contains a list that holds the numbers 10 to 100 (i.e. 10, 11, 12, …, 99)
2. Add the value 100 to the end of the list
3. Remove the 20th element of the list (index 19!)
4. Remove the value 55
5. Add the elements 5,6,7 to the end of the list (hint use extend with [5,6,7]!)
6. Print the length of the list.

    range(stop) -> range object
    range(start, stop[, step]) -> range object

# 1. Create a variable that contains a list that holds the numbers 10 to 100 (i.e. 10, 11, 12, ..., 99)
l = list(range(10, 100))
# 2. Add the value 100 to the end of the list
l.append(100)
# 3. remove the 20th element of the  list (index 19!)
l.pop(19)
# 4. remove the value 55
l.remove(55)
# 5. Add the list `[5,6,7]` to the end of the list (use `extend`!)
l.extend([5,6,7])
# 6. Print the length of the list.
print(len(l))

92

Tuples

A tuple is similar to a list in that it is a sequence of items.

The main difference between a list and a tuple, is that a tuple is immutable which means that elements cannot be changed, added, or removed once created (i.e. it is ~”read-only”!).

This makes tuples computationally faster than lists, but also less versatile.

As a general rule of thumb, when in doubt, use a list.

Dictionaries

Lists are great when we have a sequence of data, but the caveat is that in order to get a specific element we need to keep track of its index or repeatedly call the member-function index.

This can get tricky if the list grows and shrinks.

Motivation: why dictionaries?

For example, if we want to keep track of the colours (and names) of fruit, our first attempt might be to keep two lists - one for the names, and one for the colours:

names   = [ "banana", "orange", "strawberry"]
colours = [ "yellow", "orange", "red"]

Then, to find the colour of a strawberry, we first have to find the index of strawberry, and then use that to find the colour:

ind = names.index("strawberry")
print("The colour is : " + colours[ind])

Not only is this clumsy, it’s also likely to lead to issues as the lists might accidentally become unsynchronized (meaning that the order of the name and the order of the colours doesn’t match).

Attempt number two: as briefly mentioned, lists can hold pretty much anything, including other lists, so we could use a list of lists, e.g.

fruit_colours = [ ["banana", "yellow"], ["orange", "orange"], ["strawberry", "red"]]

Now the name will always be paired with the right colour!

The downside to this approach, is that if we want to find out what colour a fruit is, we need to do a non-trivial search operation (as we’re trying to find lists in lists!) and can no longer use the index function that we used before.

The bottom line is that we could hack together a solution using lists, but luckily for these kinds of situations, Python offers us a much better solution in the form of a dictionary.

Dictionary creation

A dictionary is created using curly-bracket notation { }, for example continuing our fruit names and colours example, dictionary items are provided as a list of key:value pairs:

fruit_colours = { "banana" : "yellow", "orange" : "orange", "strawberry" : "red"}

Now we can access values using keys:

print(fruit_colours["banana"])

will print yellow to the terminal.

Note about keys

Here we’ve used strings as both the keys and values, but you can also use numbers as keys and/or values.

In fact pretty much anything can be a value (much as with lists), and anything that is hashable can be a key – hashable roughly translates as non-changing. A number or string is hashable, as is a tuple (as explained above). Lists and dictionaries are NOT hashable as they can change.

Converting a sequence to a dictionary

Dictionaries can be created from a sequence where each item in the sequence has two elements, by using the dict function e.g.

fruit_colours = dict( [ ["banana", "yellow"], ["orange", "orange"], ["strawberry", "red"]] )

fruit_colours = dict( ( ("banana", "yellow"), ("orange", "orange"), ("strawberry", "red")) )

both produce the same dictionary as in the previous section.

Note that while for this example this way of creating a dictionary might seem superfluous, there are scenarios where it is very useful. For example, if we have a function that generates a list of 2-element lists, but we want the result as a dictionary, we can simply convert the list to a dictionary as per above, without having to write a new function!

Manipulating Dictionaries

Once a dictionary has been created, it can be grown or shrunk slightly differently to lists:

• Adding items : d[NEW_KEY] = NEW_VALUE - creates a new key-value pair, or updates one if it already exists
• Removing items : d.pop(KEY, DEFAULT) - removes KEY and returns its value, or DEFAULT if KEY doesn’t exist in the dictionary.

As you might have spotted, you can’t assign multiple values with the same key - instead if you write

d[ALREADY_EXISTING_KEY] = NEW_VALUE

the old value that ALREADY_EXISTING_KEY pointed to will be overwritten.

Exercise : Dictionaries - Wherefore art thou Romeo!

Write a script (name the file exercise_dicts.py) to count the occurrences of the the words

• "sword"
• "love"
• "wench"
• "fool"

in Shakespeare’s Romeo & Juliet.

Use the following initial two lines (feel free to copy and paste!) which will pull the text from an online source and assign it to a variable called text

import urllib.request
text = urllib.request.urlopen("http://www.textfiles.com/etext/AUTHORS/SHAKESPEARE/shakespeare-romeo-48.txt").read().decode('utf8')

import urllib.request
#text = urllib.request.urlretrieve("http://shakespeare.mit.edu/romeo_juliet/full.html")
text = urllib.request.urlopen("http://www.textfiles.com/etext/AUTHORS/SHAKESPEARE/shakespeare-romeo-48.txt").read().decode('utf8')
counts = {
    'sword': text.count('sword'),
    'love':text.count('love'),
    'wench':text.count('wench'),
    'fool':text.count('fool'),
}

print(counts)

{'sword': 13, 'wench': 5, 'love': 171, 'fool': 11}

Controlling program flow

Up to now, the short scripts we’ve written have been simple linear sequences of commands we want Python to execute.

While some simple tasks won’t go beyond this format, most scripts and programs use conditional execution and looping to achieve their tasks.

Conditional execution: if...else

Many computational tasks require us to conditionally perform some operations, or where the computation forks depending on the value of some variable.

For example, we might have a calculation that at some point generates a distance.

We then want to print the distance in a nice human readable format; if the distance is 0.0001 m we might prefer it to read “0.1 mm”, while if the distance is 32000 m printing “20 miles” would be more useful.

As with most programming languages, this conditional execution is achieved through use of an if-else statement.

For example,

# Somewhere in a chunk of code, we have calculated a distance in meters
# which we store in variable "dist" 
print(dist)                          # This would print 0.0001 or 32000 in the two cases above
if (dist < 0.1):
    print("Which is " + str( dist * 1000 ) + " mm")
elif( dist > 1609 ):
    print("Which is " + str( dist / 1609 ) + " miles")
else:
    print("Which is " + str(dist) + " meters")

Here we’ve shown the three possible parts of an if block; the if part will always be present, while the elif and else are optional.

The syntax is:

if <STATEMENT THAT EVALUATES TO TRUE OR FALSE> : 
    # Stuff to do if it's true

elif <ANOTHER STATEMENT THAT EVALUATES TO TRUE OR FALSE>:     # OPTIONAL 
    # Stuff to do if the `if` part was False but this part is True
 
else: # OPTIONAL 
    # Stuff to do if all previous if/elifs were False. 

Reminder: Indentation!

As mentioned at the end of the first section, the indentation above is not accidental!

Python uses indentation to demarcate logical blocks. In this case, everything after the if statement that is indented is inside the if block.

Returning to the previous level of indentation signals the end of the if block.

The elif and else are the only exceptions in this case, as they are logical extensions to the if block.

To further illustrate this consider the following code;

if a < 5:
    print("Ya")
print("Hoo")

This section of code will always print Hoo (irrespective of the value contained in a) and only print Ya if a is less than 5.

On the other hand

if a < 5:
    print("Ya") 
    print("Hoo")

will only print Ya followed by Hoo (if a is less than 5) or nothing (if a is more than or equal to 5).

Changing just the indentation level of the second print statement, has determined whether it is part of the if block or not.

Other languages

If you’ve learnt C,C++, or Java, then indentation is roughly the same as curly brackets in those languages. For example the last snippet of code in Java would be written as

if( a<5 ){
    System.out.println("Ya");
    System.out.println("Hoo");
}

(where in Java the indentation is optional!)

Looping with for-in

To really start to benefit from having a computer do things for us, we often need it to do almost the same thing repeatedly, with only small changes between each time it repeats.

For example, we might want to batch process a directory full of data files, in which case we change only the filename between each run and the program treats each file in the same way.

We might be iterating over a set of time series and applying the same filters and analyses to each one.

Or on an even finer-scale level, most image processing algorithms work by, at some point, looping over all the pixels in an image and performing the same operation at each location.

Whatever the level of processing, these scenarios all have one thing in common; they’re achieved by using loops. In most cases, we would choose a for loop, which is a way of iterating over a set list of elements (files, line-series, pixels). In most languages, we have to explicitly change the base object of each iteration, usually by indexing into a list.

This can also be done with Python. For example

l = ["a","b","c","d","e"]
for i in range(len(l)):
    print(l[i])

outputs:

a
b
c
d
e

Here,

• we start with a list of 5 letters
• Then we start a for loop over the list of index values generated by range, 0-4
• Inside the loop, we print the value of the list at the index value

i is usually referred to as the loop variable as it’s the thing that changes between each invocation of the code inside the loop.

So in each iteration of the loop, i takes on a value (from 0 to 4) and we use that value to index the original list l, and print the item in the list (letter) at index i.

However, in Python, instead of having to loop over indices, we can loop over the items we’re interested in themselves directly;

l = ["a","b","c","d","e"]
for c in l:
    print(c)

outputs:

a
b
c
d
e

While we haven’t saved any lines of code (yet!), the syntax has become much more readable.

Loosely translated:

for c in l: is the same as for each item in l that we’ll call c.

As shown above, we can still use indecies if we prefer.

Auto-generating the index variable with enumerate

Python even provides a convenience function, called enumerate, that returns both an index and a value for each item in a sequence:

l = ["a","b","c","d","e"]
for i,c in enumerate(l):
    print("At index ", i, "the item is ", c)

outputs:

At index  0 the item is  a
At index  1 the item is  b
At index  2 the item is  c
At index  3 the item is  d
At index  4 the item is  e

Exercise : Conditional Flow & Loops - number sorting

Write a script (name the file exercise_loops.py) that creates a list of the numbers 0 to 49 inclusive (0, 1, .., 49), and then prints out for each one whether it is odd or even.

x%2 

# generate a list of numbers from 0 to 49 inclusive
numbers = list(range(50))
for n in numbers:
    # Check if the number is odd or even by checking modulo 2...
    if n%2:
        print(n, "is odd")
    else:
        print(n, "is even")

0 is even
1 is odd
2 is even
3 is odd
4 is even
5 is odd
6 is even
7 is odd
8 is even
9 is odd
10 is even
11 is odd
12 is even
13 is odd
14 is even
15 is odd
16 is even
17 is odd
18 is even
19 is odd
20 is even
21 is odd
22 is even
23 is odd
24 is even
25 is odd
26 is even
27 is odd
28 is even
29 is odd
30 is even
31 is odd
32 is even
33 is odd
34 is even
35 is odd
36 is even
37 is odd
38 is even
39 is odd
40 is even
41 is odd
42 is even
43 is odd
44 is even
45 is odd
46 is even
47 is odd
48 is even
49 is odd

Another loop construct: while

There is another looping mechanism we can use instead of the for loop, called a while loop. while loops are mainly used when the exact number of times we want to repeat something is not known before starting the loop. For things like pixels and files in a directory, we would usually use for loops as we know before starting the loop how many files there are or how many pixels there are. But if we had an optimization algorithm, such algorithms usually continue until a desired closeness to an ideal solution has been found (or a maximum number of tries has been reached).

As writing optimization algorithms is difficult, we’re going to simulate one using the random number module instead.

For example

import random # This imports the random number module 
n = 1
tries = 0
while n > 0.1:
    n = random.random()     # This generates a random number between 0 and 1
    tries += 1
print("It took", tries, "tries to roll a random number less than 0.1, =", n)

A typical output from this would be

It took 3 tries to roll a random number less than 0.1, = 0.07209438256934753

So what’s happening here?

After importing the random module (more on this in the next section!), we start with two variables, n for the random number, and tries to hold the number of times we’ve tried.

We set n to 1 so that the while loop will be executed at least once (otherwise if e.g. n starts at 0, then the while condition will immediately be False and the loop would never be entered!), and tries to 0.

Then, inside the while loop, we generate a new value for the random number, and increase tries by 1.

As we then come to the end of the body of the while block, the condition gets tested again - is n still more than 0.1 or not? If so, go again. And so on until n is less than (or equal to) 0.1. Once n is less than or equal to 0.1, the comparison n > 0.1 returns False, and the while loop stops.

Then we proceed to the next line and print the text and values.

Can you guess what would happen if we removed the line that generates a new random value for n?

If you’re not sure, work through the snippet line by line and see if you can figure it out.

If you’re still not sure, ask a demonstrator.

Exercise : Conditional while loops

Write a script (name the file exercise_while.py) that determines how many squares (startng with the sequence 1² = 1, 2² = 4, 3² = 9, etc) are needed, such that their sum exceeds 1,000,000.

Have the script print out the number of squares needed and their sum (which should exceed 1,000,000!).

i = 1
summ = 1
while summ <= 1E6:
    i += 1
    summ += i**2
print(i, "squares needed to exceed 1E6 - up to", summ)

144 squares needed to exceed 1E6 - up to 1005720

Modules

Modules (aka libraries) are how Python developers share functionality they have created with the rest of the Python community.

Python is bundled with a large range of modules in what is called the Python Standard Library.

These encompass a very wide range of functionalities including

• os : “Miscellaneous operating system interfaces”
• os.path : “Common pathname manipulations”
• sys : “System-specific parameters and functions”
• math : “Mathematical functions”
• random : “Generate pseudo-random numbers”
• csv : “CSV File Reading and Writing”

which are modules that I use regularly. The full list is much, much more extensive and includes things like

• tkinter : “Python interface to Tcl/Tk” (For creating simple Graphical User Interfaces)
• wave : “Read and write WAV files” (Audio)
• ftplib : “FTP protocol client”
• http.server : “HTTP servers” (i.e. Web server)
• email : “An email and MIME handling package”
• multiprocessing : “Process-based parallelism”
• zipfile : “Work with ZIP archives”

to name an assortement of Standard Library modules I’ve used less frequently.

The total list contains hundreds of modules, and covers a wide spectrum of things you can do with a computer!

In addition to the Standard Library, there are many thousands more (69544(from January this year) 93151 (now!) at last count) modules listed in the Python Package Index, that range from modules created by lone developers to projects involving hundreds to thousands of developers over many years, such as Scipy. You can browse the list at of modules at the Python Package Index or use the search interface to find modules by keywords.

Using modules

To use a Python module we must first import it,

import MODULENAME

Once imported, we access a modules functions using the dot notation, <MODULENAME>.<FUNCTIONNAME>(...).

For example, above we used the random module to generate random numbers, so after the line

import random

we were able to call a module function using, e.g.

random.random()

Python also allows us to provide an alias for a module, which is a different way of refering to it. For example if I wrote

import random as rnd

I would then be able to later use the random.random function by writing

rnd.random()

This can be handy, especially when using modules with long names, or a submodule of a submodule of a submodule!

Lastly, submodules or functions can also be imported individually if desired, by using the from syntax:

from MODULE import FUNCTION

or

from MODULE.SUBMODULE import FUNCTION

to import a function from a module or submodule, which will then be accessible as FUNCTION (i.e. without needing to specify the module name!).

Similarly

from MODULE import SUBMODULE 

imports a submodule without needing to prefix it with the module name.

Creating modules

Creating a module is extremely simple in Python. In fact we’ve created several modules already!

Any file ending in .py is a simple module!

If for example we create a script file called mymod.py and in that script we define a function (which we’ll learn about in the last section of this workshop) called super_function, then we can create another file and import that module using

import mymod

This then gives us access to the function we defined, e.g.

mymod.super_function

Using if __name__ == "__main__": to hide script code

When writing a script to be used as a module, it is useful to have a mechanism for running part of the script only if the script is being run as a script, and not being imported!

This is because when a script is imported, it is essentially run!

To “hide” code from being run when imported Python introduced the following pattern:

def function1():
    ...

def function2():
    ...


if __name__ == "__main__":
    print("I'm being run as a script lets do script stuff!")
    function1(10,20)

We’ll cover the function definitions for the functions function1 and function2 in the next section. Of interest here is the last part of the snippet; by using the special built-in variable __name__ and checking whether it is equal to "__main__", we can check if the file is being run as a script or imported.

This is because the __name__ variable is only equal to "__main__" when being run as a script; if the file is being imported, then __name__ becomes the name of the module.

No exercise here!

While it’s important to understand that it’s actually very easy to create a Python module, doing so isn’t necessary for the context of this course, and so we won’t be practicing this here.

Advanced Users: module folders

More complex modules are created by using a folder or folders that contain a file named in a special way (__init__.py). For example if we have a folder tree:

joesModule / 
    __init__.py
    submodule1.py
    submodule2.py

Then we can use

import joesModule.submodule1 

to import the functionality contained in submodule1.py.

Reading and Writing Data

Python contains a single function for opening files, open.

By passing in a flag for how the file should be opened, we can either read, overwrite, or apend a file.

A file that is opened for reading won’t be modified, so this is always the default mode to prevent accidentally modifying files.

The result of the open function is a file object, which has a number of member functions which we can use to read from or write to the file.

Note: if we open the file using the default read mode, then attempting to call the write or writelines member functions would result in an error.

Aside on relative vs absolute paths

As soon as we want to interact with a file-system, which usually means reading or writing (saving) files, we need to be aware of the difference between relative and absolute paths.

This concept may be a little foreign if you’re used to graphical operating system environments like Windows or MacOS, though you will probably be familiar with the processes of navigating through a file system using e.g. the Windows File Explorer.

For example, when you are viewing the contents of your “My Documents” folder, the full path of that folder is something like

C:\Users\Joe\My Documents

so that when referring to a file in that folder, you would specify the full path to the file as e.g.

C:\Users\Joe\My Documents\File1.txt

However, from within File Explorer, when you are viewing the contents of My Documents, you can double click on File1.txt to open it. In that case, the File Explorers working directory is C:\Users\Joe\My Documents, so that the relative reference to “File1.txt” makes sense as it means “File1.txt” in the current working directory.

Referring to files in Python code is similar!

If you refer to "File1.txt" in Python code, it will look for a file called "File1.txt" in the working directory of the script (i.e. the same folder that the script is in).

If you want to refer to a file that is not in the working directory of the script, you would use its absolute path, e.g. C:\Users\Joe\My Documents\File1.txt

The os module has a submodule called path, (i.e. os.path) which is useful for working with file paths.

Reading files

For the time being, we’re only going to be concerned with the functions readline, and readlines with respect to reading files (though read is also useful!).

These are convenient methods to read either a line at a time (readline) into a string, or the entire file (readlines) into a list of strings.

For example if we have a very simple text file that contains

My 
Name 
is 
Sam

and the file is named sample.txt (which has the full path /path/to/file/sample.txt), then we could use

fd = open("/path/to/file/sample.txt")
print(fd.readline())
fd.close()

and the output would be

My

Similarly, we could use

fd = open("/path/to/file/sample.txt")
print(fd.readlines())
fd.close()

and the output would be

["My", "Name", "is", "Sam"]

If we want to read the file line-by-line (for example, if it is a particularly large file!), we can iterate over the file object itself:

fd = open("/path/to/file/sample.txt")
for line in fd:
    print(line)
fd.close()
    

Why the close?

After opening and accessing the file, we also need to finally close the file, using the close member function.

This is similar to when you work on a word document or any other file; if the file is still open when you try and access it with another program, you will sometimes receive errors, as your computer is warning you that you might still be modifying the file elsewhere and so manipulating it in the meantime is dangerous!

Writing files

The process for writing to a file is very similar to reading from a file; first we get a file object using open, except this time we add the additional mode flag as either "w" or "a", corresponding to write which overwrites the original file, or append which only appends to the original file. Note: if the file doesn’t exist before calling open, these two are the same!

Now we use either write or writelines to write either a single string, or a list of strings to the file.

** Important note **

A major difference however between the read and write operations, is that neither write nor writelines insert newline characters. So slightly deceptively,

fd = open('writetest.txt', 'w')
fd.writelines(['a', 'b', 'c'])
fd.close()

would produce a file called writetest.txt that contains

abc

i.e. not three separate lines!

Instead, to actually write three separate lines, we need to add the newline character, \n to each line:

fd = open('writetest.txt', 'w')
fd.writelines(['a\n', 'b\n', 'c\n'])
fd.close()

A common alternative to “manually” adding a newline character to each line, is to use the string member function join and write a single string:

fd = open('writetest.txt', 'w')
fd.write('\n'.join(['a', 'b', 'c']))
fd.close()

Here, join is a string object member function (in this case the string '\n') that takes a list as an input, and joins all of the items in the list using the string object it was called with.

Exercise : Reading a data file

We have seen above how to read text from a file and display it in the console.

As researchers, we usually need to do more with data and the data is often in numerical format.

Download the data file from here: data_exercise_reading.csv, saving the file to your python exercises scripts directory.

The file contains a table of comma separated values. The values start

Time,Signal
0,100
5,101
10,98
:
9995,102

The first line contains the column headers, and the subsequent lines contain the time and signal values.

Write a new python script (exercise_reading.py) that

• reads in the headers (but you don’t need to keep these!)
• and the data values
  • converts the data values into numbers
• On the signal data, calculate the
  • sum
  • mean
  • and population standard deviation
• output those statistics to the console

# Load data from a text (csv) file, and calculate some stats
from math import sqrt

# OPTIONALLY use the current script's location as the data folder, if that's where the data
import os
ROOT = os.path.realpath(os.path.dirname(__file__))

with open(os.path.join(ROOT, "data_exercise_reading.csv")) as fid:
    # Skip the first line (ie read it but don't keep the data)
    fid.readline()
    # Now read the data
    time = []
    sig = []
    for line in fid:
        t,s = line.split(',')
        time.append(int(t))
        sig.append(int(s))

# Now calculate the stats
sum_s = sum(sig)
mean_s = sum_s/len(sig)

# Population standard deviation = sqrt ( (1/ (N-1)) * sum (x_i - <x>)^2 )
devsq = []
for s in sig:
    devsq.append( (s - mean_s)**2 )
stddev = sqrt(sum(devsq)/(len(sig)-1))

print("Signal and Time data loaded:")
print("N        = ", len(sig))
print("Sum      = ", sum_s)
print("Mean     = ", mean_s)
print("StdDev   = ", stddev)

Signal and Time data loaded:
N        =  2000
Sum      =  199152
Mean     =  99.576
StdDev   =  10.048517818821722

Functions

Now that we’ve learnt about the basics of Python syntax, as well as how to use modules, it’s time to think about starting to modularize our own code!

A function is a way of modularizing code, such that given a set of inputs (or none), the same set of commands are executed each time a function is executed.

Similarity to Mathematical Functions

Functions are related to the idea of mathematical functions e.g. f(x)

Example mathematical function:

f(x) = x + 2

f(5) = ?

Answer:

If f(x) = x + 2

Then f(5) = 5 + 2

Therefore f(5) = 7

Calling functions

We’ve already been using, or calling, functions that were defined by others since we started this workshop. The first function we called was the print function.

All functions are called in the same way:

    <FUNCTION NAME>( VALUE FOR ARGUMENT1, VALUE FOR ARGUMENT2, ...)

where ARGUMENT1, ARGUMENT2 refers to possible input arguments to the function.

In words, functions are called using the name of the function, then open parentheses, then the argument list (comma separated), and then close parentheses. The number of arguments that must be passed into the function depends on how the function was defined.

Defining functions

To define a function in Python, we use the def statement:

def <FUNCTION NAME>( ARGUMENT1, ARGUMENT2, ...):

so very similar to calling a function, except that we start the line with def, and we end the line with the colon, :.

The number of arguments is up to us and is dependent on what inputs the function needs.

The function body follows and is indented relative to the def ... line.

If we want to return values from our function we use the return statement;

def <FUNCTION NAME>(ARGUMENT1, ARGUMENT2,...):
    # Function body
    :
    :
    return value1, value2

# Code no longer in function definition

The function body ends when the indentation level returns to that before the def statement.

So to use our maths function example of f(x) = x + 2 we could write this as a python function like this:

def f(x):
    answer = x + 2 
    return answer

or more compact:

def f(x):
    return x + 2

A function must be defined before it can be called. Every built-in function and standard library module function we’ve been using is defined somewhere.

Exercise : Our first function definition

To get used to defining functions, lets start by defining a trivial function that replaces functionality that we already know.

In a new script file (exercise_function.py)

• define a function called “add_numbers”
  • that takes two inputs,
  • and returns their sum
• Print to the console the result of calling the function on 40 and 2.

# Our first function definition!

def add_numbers(a,b):
    return a+b

print(add_numbers(40,2))

42

Exercise : Upgrading our numerical analysis script

We have already written code that could be modularized in previous exercises. Lets upgrade the code from the exercise on “reading a data file” to a function.

In a new script file (exercise_modularization.py)

• Copy and paste the code from the “reading a data file” exercise in the previous section
• Modularize that code by
  • Creating a function definition called “analyze_file” that
    • Takes a file path as an input
    • Opens the file and reads in the data
    • Returns the statistics
  • Then call this function with the data file name as an input, and print the result to the terminal
  • Verify that the result is the same as the original script

# Load data from a text (csv) file, and calculate some stats
from math import sqrt


def analyze_file(filename):
    with open(filename) as fid:
        # Skip the first line (ie read it but don't keep the data)
        fid.readline()
        # Now read the data
        time = []
        sig = []
        for line in fid:
            t,s = line.split(',')
            time.append(int(t))
            sig.append(int(s))

    # Now calculate the stats
    sum_s = sum(sig)
    mean_s = sum_s/len(sig)

    # Population standard deviation = sqrt ( (1/ (N-1)) * sum (x_i - <x>)^2 )
    devsq = []
    for s in sig:
        devsq.append( (s - mean_s)**2 )
    stddev = sqrt(sum(devsq)/(len(sig)-1))

    return {
        "N"      : len(sig),
        "Sum"    : sum_s,
        "Mean"   : mean_s,
        "StdDev" : stddev,
    }

# OPTIONALLY use the current script's location as the data folder, if that's where the data
import os
ROOT = os.path.realpath(os.path.dirname(__file__))

result = analyze_file( os.path.join(ROOT,"data_exercise_reading.csv"))

print(result)

{'Mean': 99.576, 'StdDev': 10.048517818821722, 'Sum': 199152, 'N': 2000}

Why we would do this?

The answer is that now we can call the same functionality on any file, or more importantly, on many files.

For example we could have 1000 files that all contain such data; the benefits of having a single function that is called on each one instead of a script with hard-coded input file name:

• We don’t need to copy the script file 1000 times and change the input file name in each…
• If we want to add an additional statistic… we don’t need to then update 1000 script files! We only update the one function! The same goes for if we find a bug in the code.

Positional and keyword arguments

“Traditionally” function arguments are positional, meaning that the value that is passed into a function call at the first position, is assigned to the first variable in the function definition, the second to the second, and so on.

As well as these positional arguments, Python functions often accept keyword arguments. Keyword arguments are provided as <KEYWORD>=<VALUE> pairs.

Keyword arguments are always optional, as they are given default values when the function is defined, while positional arguments are non-optional and do not have default values.

For example even the print function takes several keyword arguments.

If you try the following in a script

print("a", end="\n\n\n\n")
print("b")

You should see an output like

a



b

because the print function used four new line characters instead of the default of one new line character.

Similarly

print("a", end="--NEXT--")
print("b")

would output

a--NEXT--b

Exercise : Adding a keyword argument

In a new script file (exercise_keyword.py)

• Copy and paste the function definition for “add_numbers”
• Rename the function to “add_numbers2”
  • Add a keyword argument in the function definition
    • called absolute
    • that defaults to False
  • Add a couple of lines in the code before the sum that
    • if absolute is True, converts all inputs to their absolute values
• Print to the console the result of calling the function on 40 and 2.
• Print to the console the result of calling the function on 2 and -2
• Print to the console the result of calling the function on 2 and -2 if you also pass in the keyword argument absolute set to True

# Updated add function to allow for absolute value addition

def add_numbers2(a,b, absolute=False):
    if absolute:
        return abs(a) + abs(b)
    else:
        return a+b

print(add_numbers2(40,2))
print(add_numbers2(-2,2))
print(add_numbers2(-2,2, absolute=True))

42
0
4

Documenting code

Another feature we need to start adding as our scripts grow, is documentation.

Comments using the hash (aka pound symbol if you come from the USA) symbol typically appear every few lines in well written code.

Pulling a random section of code from a standard python module:

# From : pathlib.py, line 1000
.
.
.

    def absolute(self):
        """Return an absolute version of this path.  This function works
        even if the path doesn't point to anything.

        No normalization is done, i.e. all '.' and '..' will be kept along.
        Use resolve() to get the canonical path to a file.
        """
        # XXX untested yet!
        if self._closed:
            self._raise_closed()
        if self.is_absolute():
            return self
        # FIXME this must defer to the specific flavour (and, under Windows,
        # use nt._getfullpathname())
        obj = self._from_parts([os.getcwd()] + self._parts, init=False)
        obj._init(template=self)
        return obj

    def resolve(self):
        """
        Make the path absolute, resolving all symlinks on the way and also
        normalizing it (for example turning slashes into backslashes under
        Windows).
        """
        if self._closed:

.
.
.

Ignoring the majority of what’s actually written, this section illustrates a few things regarding commenting:

• Comments don’t have to appear every line, just every now-and-again to help people reading the code
• Comments are also used to keep track of when things need attention, e.g. the use of FIXME above

Special comments in the form of multi-line strings (using three single/double quote symbols, """ ... """) are used immediately after function definitions to document functions. These are called docstrings and the Python help system scans source code for these when you call e.g. help <FUNCTION NAME>.

Exercise : Adding documentation

Copy and paste your previous exercise (add_numbers2) into a new script file (exercise_docstring.py) and add a docstring that explains what the function does and how to use it.

Then run python -m pydoc exercise_docstring in the terminal, from your exerice folder.

# Adding a docstring

def add_numbers2(a,b, absolute=False):
    """
    A modified addition function, that optionally performs absolute value addition
    """
    if absolute:
        return abs(a) + abs(b)
    else:
        return a+b

Help on module answer_docstring:

NAME
    answer_docstring - # Adding a docstring

FUNCTIONS
    add_numbers2(a, b, absolute=False)
        A modified addition function, that optionally performs absolute value addition

FILE
    .../answer_docstring.py

Final Exercise

Note

Don’t forget to use the good coding suggestions, including frequent commenting / documentation of your code in this exercise.

In light of revelations regarding government agency snooping, you have decided to encrypt your personal communications!

Being a budding Pythonista, and having heard of the Caesar cipher, you will now write a script (exercise_encryption.py) that

Part 1

• Has an encryption function which takes
  • text(string) and an offset(integer) as a input
  • converts the text to cipher-text using the given offset to generate the cipher
  • outputs the cipher-text
• Test your function in the if __main__ == ... section of your code (check back to the module section for what this means!)
  • Using any string you like, and offset 0 - you should get back the original text
  • The same text but 1 offset – e.g. the cat should become uifadbu
• Add a decryption function that
  • takes text and an offset as an input
  • Generates a cipher with the given offset
  • Decrypts the input text using the cipher
  • Returns the decrypted text
  • HINT: think about how the encryption & decryption functions work (i.e. how similar they are…) - maybe you can avoid creating a whole new decryption function ?!?

Note:

With offset 1, the cipher should become: bcde...xyz a, i.e. include space as a character, in contrast to the example on the wiki page.

I.e. the plain-text too cipher translation table for offset 1 should be:

PLAIN  : "abcd...xyz " 
CIPHER : "bcde...yz a"

Ask a demonstrator if you are unclear about this.

Part 2

• Add a file-reading and writing function that
  • Takes a filename and offset as inputs
  • Opens and reads text in the specified file
  • Calls your encryption function on the text, using the offset provided as an input
  • Writes the output to a file with the same base name and extension as the input text, but is named <INPUT NAME>-encrypted.<INPUT EXTENSION>
  • Returns the encrypted file name
• Download the plain text file from here: data_exercise_encryption.txt,
• Call your script with this file as an input to the previously defined encryption function (and non-zero offset!), and verify that you get a new encrypted file that contains encrypted text.
• Add a file-reading function that
  • Takes a filename and offset as inputs
  • Opens and reads text in the specified file
  • Calls your decryption function on the text, using the offset provided as an input
  • Returns the resulting decrypted text
• Print the return value of this function to the terminal (passing the previously encrypted file)

Part 3: Bonus section

Now that you can encrypt text, and decrypt text that was encrypted with a known offset cipher, let’s test our script using an encrypted file where we don’t know the offset.

To do this we’re going to use a “brute-force” algorithm,

• download the encrypted text from here: data_exercise_encryption_secret.txt
• Write a loop over all possible offsets
  • For each offset, decrpyt the text and print the final message and offset to the terminal
• Skim through the decyrpted texts; hopefully one and only one of the decryped texts should make sense!