5. Iterators & Generators — Python Practice Book

by ⋅ Link

5. Iterators & Generators — Python Practice Book

original source : http://anandology.com/python-practice-book/iterators.html

5. Iterators & Generators

5.1. Iterators

We use for statement for looping over a list.

>>> for i in [1, 2, 3, 4]:
...     print i,
...
1
2
3
4

If we use it with a string, it loops over its characters.

>>> for c in "python":
...     print c
...
p
y
t
h
o
n

If we use it with a dictionary, it loops over its keys.

>>> for k in {"x": 1, "y": 2}:
...     print k
...
y
x

If we use it with a file, it loops over lines of the file.

>>> for line in open("a.txt"):
...     print line,
...
first line
second line

So there are many types of objects which can be used with a for loop. These are called iterable objects.

There are many functions which consume these iterables.

>>> ",".join(["a", "b", "c"])
'a,b,c'
>>> ",".join({"x": 1, "y": 2})
'y,x'
>>> list("python")
['p', 'y', 't', 'h', 'o', 'n']
>>> list({"x": 1, "y": 2})
['y', 'x']

5.1.1. The Iteraton Protocol

The built-in function iter takes an iterable object and returns an iterator.

>>> x = iter([1, 2, 3])
>>> x
<listiterator object at 0x1004ca850>
>>> x.next()
1
>>> x.next()
2
>>> x.next()
3
>>> x.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Each time we call the next method on the iterator gives us the next element. If there are no more elements, it raises a StopIteration.

Iterators are implemented as classes. Here is an iterator that works like built-in xrange function.

class yrange:
    def __init__(self, n):
        self.i = 0
        self.n = n

    def __iter__(self):
        return self

    def next(self):
        if self.i < self.n:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()

The __iter__ method is what makes an object iterable. Behind the scenes, the iter function calls__iter__ method on the given object.

The return value of __iter__ is an iterator. It should have a next method and raise StopIterationwhen there are no more elements.

Lets try it out:

>>> y = yrange(3)
>>> y.next()
0
>>> y.next()
1
>>> y.next()
2
>>> y.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 14, in next
StopIteration

Many built-in functions accept iterators as arguments.

>>> list(yrange(5))
[0, 1, 2, 3, 4]
>>> sum(yrange(5))
10

In the above case, both the iterable and iterator are the same object. Notice that the __iter__method returned self. It need not be the case always.

class zrange:
    def __init__(self, n):
        self.n = n

    def __iter__(self):
        return zrange_iter(self.n)

class zrange_iter:
    def __init__(self, n):
        self.i = 0
        self.n = n

    def __iter__(self):
        # Iterators are iterables too.
        # Adding this functions to make them so.
        return self

    def next(self):
        if self.i < self.n:
            i = self.i
            self.i += 1
            return i
        else:
            raise StopIteration()

If both iteratable and iterator are the same object, it is consumed in a single iteration.

>>> y = yrange(5)
>>> list(y)
[0, 1, 2, 3, 4]
>>> list(y)
[]
>>> z = zrange(5)
>>> list(z)
[0, 1, 2, 3, 4]
>>> list(z)
[0, 1, 2, 3, 4]

Problem 1: Write an iterator class reverse_iter, that takes a list and iterates it from the reverse direction. ::

>>> it = reverse_iter([1, 2, 3, 4])
>>> it.next()
4
>>> it.next()
3
>>> it.next()
2
>>> it.next()
1
>>> it.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

5.2. Generators

Generators simplifies creation of iterators. A generator is a function that produces a sequence of results instead of a single value.

def yrange(n):
    i = 0
    while i < n:
        yield i
        i += 1

Each time the yield statement is executed the function generates a new value.

>>> y = yrange(3)
>>> y
<generator object yrange at 0x401f30>
>>> y.next()
0
>>> y.next()
1
>>> y.next()
2
>>> y.next()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

So a generator is also an iterator. You don’t have to worry about the iterator protocol.

The word “generator” is confusingly used to mean both the function that generates and what it generates. In this chapter, I’ll use the word “generator” to mean the genearted object and “generator function” to mean the function that generates it.

Can you think about how it is working internally?

When a generator function is called, it returns an generator object without even beginning execution of the function. When next` method is called for the first time, the function starts executing until it reaches yield statement. The yielded value is returned by the next call.

The following example demonstrates the interplay between yield and call to next method on generator object.

>>> def foo():
...     print "begin"
...     for i in range(3):
...         print "before yield", i
...         yield i
...         print "after yield", i
...     print "end"
...
>>> f = foo()
>>> f.next()
begin
before yield 0
0
>>> f.next()
after yield 0
before yield 1
1
>>> f.next()
after yield 1
before yield 2
2
>>> f.next()
after yield 2
end
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>>

Lets see an example:

def integers():
    """Infinite sequence of integers."""
    i = 1
    while True:
        yield i
        i = i + 1

def squares():
    for i in integers():
        yield i * i

def take(n, seq):
    """Returns first n values from the given sequence."""
    seq = iter(seq)
    result = []
    try:
        for i in range(n):
            result.append(seq.next())
    except StopIteration:
        pass
    return result

print take(5, squares()) # prints [1, 4, 9, 16, 25]

5.3. Generator Expressions

Generator Expressions are generator version of list comprehensions. They look like list comprehensions, but returns a generator back instead of a list.

>>> a = (x*x for x in range(10))
>>> a
<generator object <genexpr> at 0x401f08>
>>> sum(a)
285

We can use the generator expressions as arguments to various functions that consume iterators.

>>> sum(((x*x for x in range(10)))
285

When there is only one argument to the calling function, the parenthesis around generator expression can be omitted.

>>> sum(x*x for x in range(10))
285

Another fun example:

Lets say we want to find first 10 (or any n) pythogorian triplets. A triplet (x, y, z) is called pythogorian triplet if x*x + y*y == z*z.

It is easy to solve this problem if we know till what value of z to test for. But we want to find first n pythogorian triplets.

>>> pyt = ((x, y, z) for z in integers() for y in xrange(1, z) for x in range(1, y) if x*x + y*y == z*z)
>>> take(10, pyt)
[(3, 4, 5), (6, 8, 10), (5, 12, 13), (9, 12, 15), (8, 15, 17), (12, 16, 20), (15, 20, 25), (7, 24, 25), (10, 24, 26), (20, 21, 29)]

5.3.1. Example: Reading multiple files

Lets say we want to write a program that takes a list of filenames as arguments and prints contents of all those files, like cat command in unix.

The traditional way to implement it is:

def cat(filenames):
    for f in filenames:
        for line in open(f):
            print line,

Now, lets say we want to print only the line which has a particular substring, like grep command in unix.

def grep(pattern, filenames):
    for f in filenames:
        for line in open(f):
            if pattern in line:
                print line,

Both these programs have lot of code in common. It is hard to move the common part to a function. But with generators makes it possible to do it.

def readfiles(filenames):
    for f in filenames:
        for line in open(f):
            yield line

def grep(pattern, lines):
    return (line for line in lines if pattern in lines)

def printlines(lines):
    for line in lines:
        print line,

def main(pattern, filenames):
    lines = readfiles(filenames)
    lines = grep(pattern, lines)
    printlines(lines)

The code is much simpler now with each function doing one small thing. We can move all these functions into a separate module and reuse it in other programs.

Problem 2: Write a program that takes one or more filenames as arguments and prints all the lines which are longer than 40 characters.

Problem 3: Write a function findfiles that recursively descends the directory tree for the specified directory and generates paths of all the files in the tree.

Problem 4: Write a function to compute the number of python files (.py extension) in a specified directory recursively.

Problem 5: Write a function to compute the total number of lines of code in all python files in the specified directory recursively.

Problem 6: Write a function to compute the total number of lines of code, ignoring empty and comment lines, in all python files in the specified directory recursively.

Problem 7: Write a program split.py, that takes an integer n and a filename as command line arguments and splits the file into multiple small files with each having n lines.

5.4. Itertools

The itertools module in the standard library provides lot of intersting tools to work with iterators.

Lets look at some of the interesting functions.

chain – chains multiple iterators together.

>>> it1 = iter([1, 2, 3])
>>> it2 = iter([4, 5, 6])
>>> itertools.chain(it1, it2)
[1, 2, 3, 4, 5, 6]

izip – iterable version of zip

>>> for x, y in itertools.izip(["a", "b", "c"], [1, 2, 3]):
...     print x, y
...
a 1
b 2
c 3

Problem 8: Write a function peep, that takes an iterator as argument and returns the first element and an equivalant iterator.

>>> it = iter(range(5))
>>> x, it1 = peep(it)
>>> print x, list(it1)
0 [0, 1, 2, 3, 4]

Problem 9: The built-in function enumerate takes an iteratable and returns an iterator over pairs (index, value) for each value in the source.

>>> list(enumerate(["a", "b", "c"])
[(0, "a"), (1, "b"), (2, "c")]
>>> for i, c in enumerate(["a", "b", "c"]):
...     print i, c
...
0 a
1 b
2 c

Write a function my_enumerate that works like enumerate.

Problem 10: Implement a function izip that works like itertools.izip.

Further Reading

Generators – Learn Python – Free Interactive Python Tutorial

by ⋅ Link

Generators – Learn Python – Free Interactive Python Tutorial

original source : http://www.learnpython.org/en/Generators

Generators

Generators are very easy to implement, but a bit difficult to understand.

Generators are used to create iterators, but with a different approach. Generators are simple functions which return an iterable set of items, one at a time, in a special way.

When an iteration over a set of item starts using the for statement, the generator is run. Once the generator’s function code reaches a “yield” statement, the generator yields its execution back to the for loop, returning a new value from the set. The generator function can generate as many values (possibly infinite) as it wants, yielding each one in its turn.

Here is a simple example of a generator function which returns 7 random integers:

import random

def lottery():
    # returns 6 numbers between 1 and 40
    for i in xrange(6):
        yield random.randint(1, 40)

    # returns a 7th number between 1 and 15
    yield random.randint(1,15)

for random_number in lottery():
    print "And the next number is... %d!" % random_number

Execute Code

This function decides how to generate the random numbers on its own, and executes the yield statements one at a time, pausing in between to yield execution back to the main for loop.

in my words:

generator는  loop over 가 가능한 object를 return하는 함수 이다. 즉 object가  for구문과 같은 loop구문을 만났을때 이 작업이 가능하게끔 loop over가 가능한 object를 return해주는 것이다. 함수의 return 구문이 들어갈 위치에 yield를 사용 값을 바로 리턴하지 않고 loop over가 가능한 형태로 쌓아나간다.

Attaching custom exceptions to functions and classes

by ⋅ Link

Attaching custom exceptions to functions and classes

original source : http://www.pydanny.com/attaching-custom-exceptions-to-functions-and-classes.html

Having too many custom exceptions on a project can be a pain, but a few choices ones are really nice. The problem is that in complex libraries having to import both functions and exceptions becomes a drag. To mitigate having to remember to import custom exceptions, this is a handy pattern you can use in a project and can be done on both functions and classes.

Attaching a custom exception to a function

This works because Python functions are first-class objects. They can be passed around as things, and in this case, have things assigned to them.

# logic.py
class DoesNotCompute(Exception):
    """ Easy to understand naming conventions work best! """
    pass

def this_function(x):
    """ This function only works on numbers."""
    try:
        return x ** x
    except TypeError:
        raise DoesNotCompute

# Assign DoesNotCompute exception to this_function
this_function.DoesNotCompute = DoesNotCompute

Now I can import the function, and it won’t just through DoesNotCompute exceptions, it will also carry the function along with the import:

>>> from logic import this_function
>>> this_function(5)
3125
>>> this_function(4.5)
869.8739233809259
>>> this_function('will throw an error.')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "logic.py", line 10, in this_function
    raise DoesNotCompute
DoesNotCompute

Alright, that doesn’t seem like much, but let’s add in some exception handling:

>>> try:
...     this_function('is an example')
... except this_function.DoesNotCompute:
...     print('See what attaching custom exceptions to functions can do?')
...
...
See what attaching custom exceptions to functions can do?

Attaching the custom exception to a class

All we have to do is enhance our existing logic.py file by adding ThisClass:

# logic.py
class DoesNotCompute(Exception):
    """ Easy to understand naming conventions work best! """
    pass

# removed the function example for clarity

class ThisClass(object):
    # Since the DoesNotCompute exception exists, let's just assign it
    # as an attribute of ThisClass
    DoesNotCompute = DoesNotCompute

    def this_method(self, x):
        """ This method only works on numbers."""
        try:
            return x ** x
        except TypeError:
            raise DoesNotCompute

Now to demonstrate in the shell (Python REPL for the semantic purists):

>>> from t import ThisClass
>>> this_class = ThisClass()
>>> this_class.this_method(3.3)
51.415729444066585
>>> this_class.this_method('Jack Diederich warned against custom exceptions')
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "logic.py", line 24, in this_method
    raise DoesNotCompute
DoesNotCompute
>>> try:
...     this_class.this_method('I need to write a follow-up on my OAuth post')
... except ThisClass.DoesNotCompute:
...     print('Waiting to see how the OAuth stuff pans out')
...
...
Waiting to see how the OAuth stuff pans out

Admonition: Don’t go crazy

Rather than use this trick all over the place, considering using it in a few places to powerful effect. For example, Django uses it only in a few places, and publicly only on MyModelClass.DoesNotExist andMyModelClass.MultipleObjectsReturned. By limiting Django’s use of this technique, Django libraries are that much easier to comprehend. In this case, less complexity means more.

I say this because this pattern lends itself to creating custom exceptions to the point of effectively replacing Python’s stock exceptions with your own. This makes for harder-to-maintain and harder-to-learn projects.

Not that I’ve ever done that. Ahem.

5.9. Private Functions

by ⋅ Link

5.9. Private Functions

original source :http://www.diveintopython.net/object_oriented_framework/private_functions.html

5.9. Private Functions

Like most languages, Python has the concept of private elements:

  • Private functions, which can’t be called from outside their module
  • Private class methods, which can’t be called from outside their class
  • Private attributes, which can’t be accessed from outside their class.

Unlike in most languages, whether a Python function, method, or attribute is private or public is determined entirely by its name.

If the name of a Python function, class method, or attribute starts with (but doesn’t end with) two underscores, it’s private; everything else is public. Python has no concept of protected class methods (accessible only in their own class and descendant classes). Class methods are either private (accessible only in their own class) or public (accessible from anywhere).

In MP3FileInfo, there are two methods: __parse and __setitem__. As you have already discussed, __setitem__ is a special method; normally, you would call it indirectly by using the dictionary syntax on a class instance, but it is public, and you could call it directly (even from outside the fileinfo module) if you had a really good reason. However, __parse is private, because it has two underscores at the beginning of its name.

In Python, all special methods (like

__setitem__

) and built-in attributes (like

__doc__

) follow a standard naming convention: they both start with and end with two underscores. Don’t name your own methods and attributes this way, because it will only confuse you (and others) later.

Example 5.19. Trying to Call a Private Method

>>> import fileinfo
>>> m = fileinfo.MP3FileInfo()
>>> m.__parse("/music/_singles/kairo.mp3") 
Traceback (innermost last):
  File "<interactive input>", line 1, in ?
AttributeError: 'MP3FileInfo' instance has no attribute '__parse'

If you try to call a private method, Python will raise a slightly misleading exception, saying that the method does not exist. Of course it does exist, but it’s private, so it’s not accessible outside the class.Strictly speaking, private methods are accessible outside their class, just not easily accessible. Nothing in Python is truly private; internally, the names of private methods and attributes are mangled and unmangled on the fly to make them seem inaccessible by their given names. You can access the __parse method of the MP3FileInfo class by the name _MP3FileInfo__parse. Acknowledge that this is interesting, but promise to never, ever do it in real code. Private methods are private for a reason, but like many other things in Python, their privateness is ultimately a matter of convention, not force.

Things to Know About Python Super [1 of 3]

by ⋅ Link

Things to Know About Python Super [1 of 3]

original source : http://www.artima.com/weblogs/viewpost.jsp?thread=236275

Introduction

super is a Python built-in, first introduced in Python 2.2 and slightly improved and fixed in later versions, which is often misunderstood by the average Python programmer. One of the reasons for that is the poor documentation of super: at the time of this writing (August 2008) the documentation is incomplete and in some parts misleading and even wrong. For instance, the standard documentation (even for the new 2.6 version http://docs.python.org/dev/library/functions.html#super) still says:

super(type[, object-or-type])
  Return the superclass of type. If the second argument is omitted the
  super object returned is unbound. If the second argument is an object,
  isinstance(obj, type) must be true. If the second argument is a type,
  issubclass(type2, type) must be true. super() only works for new-style
  classes.

[UPDATE: the final version of Python 2.6 has a better documentation for super, as a direct consequence of this post ;)]. The first sentence is just plain wrong:super does not return the superclass. There is no such a thing as the superclass in a Multiple Inheritance (MI) world. Also, the sentence about unbound is misleading, since it may easily lead the programmer to think about bound and unbound methods, whereas it has nothing to do with that concept. IMNSHO super is one of the most tricky and surprising Python constructs, and we absolutely need a document to shed light on its secrets. The present paper is a first step in this direction: it aims to tell you the truth about super. At least the amount of truth I have discovered with my experimentations, which is certainly not the whole truth 😉

A fair warning is in order here: this document is aimed at expert Pythonistas. It assumes you are familiar with new style classes and the Method Resolution Order(MRO); moreover a good understanding of descriptors would be extremely useful. Some parts also require good familiarity with metaclasses. All in all, this paper is not for the faint of heart 😉

There is no superclass in a MI world

Readers familiar will single inheritance languages, such as Java or Smalltalk, will have a clear concept of superclass in mind. This concept, however, has no useful meaning in Python or in other multiple inheritance languages. I became convinced of this fact after a discussion with Bjorn Pettersen and Alex Martelli oncomp.lang.python in May 2003 (at that time I was mistakenly thinking that one could define a superclass concept in Python). Consider this example from that discussion:

       +-----+
       |  T  |
       |a = 0|
       +-----+
     /         
    /           
+-------+    +-------+
|   A   |    |   B   |
|       |    | a = 2 |
+-------+    +-------+
               /
              /
       +-----+
       |  C  |
       +-----+
          :
          :    instantiation
          c

>>> class T(object):
...     a = 0

>>> class A(T):
...     pass

>>> class B(T):
...     a = 2

>>> class C(A,B):
...     pass

>>> c = C()

What is the superclass of C? There are two direct superclasses (i.e. bases) of C: A and B. A comes before B, so one would naturally think that the superclass of C isA. However, A inherits its attribute a from T with value a=0: if super(C,c) was returning the superclass of C, then super(C,c).a would return 0. This is NOT what happens. Instead, super(C,c).a walks trought the method resolution order of the class of c (i.e. C) and retrieves the attribute from the first class above C which defines it. In this example the MRO of C is [C, A, B, T, object], so B is the first class above C which defines a and super(C,c).a correctly returns the value 2, not 0:

>>> super(C,c).a
2

You may call A the superclass of C, but this is not a useful concept since the methods are resolved by looking at the classes in the MRO of C, and not by looking at the classes in the MRO of A (which in this case is [A,T, object] and does not contain B). The whole MRO is needed, not just the first superclass.

So, using the word superclass in the standard docs is misleading and should be avoided altogether.

Bound and unbound (super) methods

Having established that super cannot return the mythical superclass, we may ask ourselves what the hell it is returning 😉 The truth is that super returns proxy objects.

Informally speaking, a proxy is an object with the ability to dispatch to methods of other objects via delegation. Technically, super is a class overriding the__getattribute__ method. Instances of super are proxy objects providing access to the methods in the MRO. The dispatch is done in such a way that

super(cls, instance-or-subclass).method(*args, **kw)

corresponds more or less to

right-method-in-the-MRO-applied-to(instance-or-subclass, *args, **kw)

There is a caveat at this point: the second argument can be an instance of the first argument, or a subclass of it. In the first case we expect a bound method to be returned and in the second case and unbound method to be returned. This is true in recent versions of Python: for instance, in this example

>>> class B(object):
...     def __repr__(self):
...         return "<instance of %s>" % self.__class__.__name__

>>> class C(B):
...     pass

>>> class D(C):
...     pass

>>> d = D()

you get

>>> print super(C, d).__repr__
<bound method D.__repr__ of <instance of D>>

and

>>> print super(C, D).__repr__
<unbound method D.__repr__>

However, if you are still using Python 2.2 (there are unlucky people forced to use old versions) your should be aware that super had a bug and super(<class>,<subclass>).method returned a bound method, not an unbound one:

>> print super(C, D).__repr__ # in Python 2.2
<bound method D.__repr__ of <class '__main__.D'>>

That means that in Python 2.2 you get:

>> print super(C, D).__repr__() # in Python 2.2
<instance of type>

D, seen as an instance of the (meta)class type, is being passed as first argument to __repr__. This has been fixed in Python 2.3+, where you correctly get aTypeError:

>>> print super(C, D).__repr__() # the same as B.__repr__()
Traceback (most recent call last):
 ...
TypeError: unbound method __repr__() must be called with D instance as first
argument (got nothing instead)

The point is subtle, but usually one does not see problems since typically super is invoked on instances, not on subclasses, and in this case it works correctly in all Python versions:

>>> print super(C, d).__repr__()
<instance of D>

When I was using Python 2.2, due to the bug just discussed, and due to the super docstring

>>> print super.__doc__
super(type) -> unbound super object
super(type, obj) -> bound super object; requires isinstance(obj, type)
super(type, type2) -> bound super object; requires issubclass(type2, type)
Typical use to call a cooperative superclass method:
class C(B):
    def meth(self, arg):
        super(C, self).meth(arg)

I got the impression that in order to get unbound methods I needed to use the unbound super object. This is actually untrue. To understand how bound/unbound methods work we need to talk about descriptors.

super and descriptors

Descriptors (more properly I should speak of the descriptor protocol) were introduced in Python 2.2 by Guido van Rossum. Their primary motivation was technical, since they were needed to implement the new-style object system. Descriptors were also used to introduce new standard concepts in Python, such as classmethods, staticmethods and properties. Moreover, according to the traditional transparency policy of Python, descriptors were exposed to the application programmer, giving him/her the freedom to write custom descriptors. Any serious Python programmer should have a look at descriptors: luckily they are now very well documented (which was not the case when I first studied them :-/) thanks to the beautiful essay of Raimond Hettinger. You should read it before continuing this article, since it explains all the details. However, for the sake of our discussion of super, it is enough to say that a descriptor class is just a regular new-style class which implements a .__get__ method with signature __get__(self, obj, objtyp=None). A descriptor object is just an instance of a descriptor class.

Descriptor objects are intended to be used as attributes (hence their complete name attribute descriptors). Suppose that descr is a given descriptor object used as attribute of a given class C. Then the syntax C.descr is actually interpreted by Python as a call to descr.__get__(None, C), whereas the same syntax for an instance of C corresponds to a call to descr.__get__(c, type©).

Since the combination of descriptors and super is so tricky, the core developers got it wrong in different versions of Python. For instance, in Python 2.2 the only way to get the unboud method __repr__ is via the descriptor API:

>> super(C, d).__repr__.__get__(None, D) # Python 2.2
<unbound method D.__repr__>

You may check that it works correctly:

>> print _(d)
<instance of D>

In Python 2.3 one can get the unbond method by using the super(cls, subcls) syntax, but the syntax super(C, d).__repr__.__get__(None, D) also works; in Python 2.4+ instead the same syntax returns a bound method, not an unbound one:

>>> super(C, d).__repr__.__get__(None, D) # in Python 2.4+
<bound method D.__repr__ of <instance of D>>

The core developers changed the behavior again, making my life difficult while I was writing this paper :-/ I cannot trace the history of the bugs of super here, but if you are using an old version of Python and you find something weird with super, I advice you to have a look at the Python bug tracker before thinking you are doing something wrong. In this case, to be correct, the change is not in super, but in the descriptor implementation. In Python 2.2-2.3 you could get an unbound method from a bound one as follows:

>> d.__repr__.__get__(None, D) # in Python 2.2-2.3
<unbound method D.__repr__>

In Python 2.4 that does not work anymore:

>>> d.__repr__.__get__(None, D) # in Python 2.4+
<bound method D.__repr__ of <instance of D>>

Still, you can get the unbound method by passing for the underlying function first:

>>> d.__repr__.im_func.__get__(None, D) # in Python 2.4+
<unbound method D.__repr__>

Python3 Tutorial: Multiple Inheritance

by ⋅ Link

Python3 Tutorial: Multiple Inheritance

original source : http://www.python-course.eu/python3_multiple_inheritance.php

Multiple Inheritance

Introduction

In the previous chapter of our tutorial, we have covered inheritance, or more specific “single inheritance”. As we have seen, a class inherits in this case from one class. Multiple inheritance on the other hand is a feature in which a class can inherit attributes and methods from more than one parent parent class. The critics point out that multiple inheritance comes along with a high level of complexity and ambiguity in situations such as the diamond problem. We will address this problem later in this chapter.

The widespread prejudice that multiple inheritance is something “dangerous” or “bad” is mostly nourished by programming languages with poorly implemented multiple inheritance mechanisms and above all by improper usage of it. Java doesn’t even support multiple inheritance, while C++ supports it. Python has a sophisticated and well-designed approach to multiple inheritance.

A class definition, where a child class SubClassName inherits from the parent classes BaseClass1, BaseClass2, BaseClass3, and so on, looks like this: 

class SubclassName(BaseClass1, BaseClass2, BaseClass3, ...):
    pass

It’s clear that all the superclasses BaseClass1, BaseClass2, BaseClass3, … can inherit from other superclasses as well. What we get is an inheritance tree.

Example: CalendarClockWe want to introduce the principles of multiple inheritance with an example. For this purpose, we will implement to independent classes: a “Clock” and a “Calendar” class. After this, we will introduce a class “CalendarClock”, which is, as the name implies, a combination of “Clock” and “Calendar”. CalendarClock inherits both from “Clock” and “Calendar”.

The class Clock simulates the tick-tack of a clock. An instance of this class contains the time, which is stored in the attributes self.hours, self.minutes and self.seconds. Principally, we could have written the __init__ method and the set method like this: 

    def __init__(self,hours=0, minutes=0, seconds=0):
        self._hours = hours
        self.__minutes = minutes
        self.__seconds = seconds

    def set(self,hours, minutes, seconds=0):
        self._hours = hours
        self.__minutes = minutes
        self.__seconds = seconds

We decided against this implementation, because we added additional code for checking the plausibility of the time data into the set method. We call the set method from the __init__ method as well, because we want to circumvent redundant code.

The complete Clock class: 

""" 
The class Clock is used to simulate a clock.
"""

class Clock(object):

    def __init__(self, hours, minutes, seconds):
        """
        The paramaters hours, minutes and seconds have to be 
        integers and must satisfy the following equations:
        0 <= h < 24
        0 <= m < 60
        0 <= s < 60
        """

        self.set_Clock(hours, minutes, seconds)

    def set_Clock(self, hours, minutes, seconds):
        """
        The parameters hours, minutes and seconds have to be 
        integers and must satisfy the following equations:
        0 <= h < 24
        0 <= m < 60
        0 <= s < 60
        """

        if type(hours) == int and 0 <= hours and hours < 24:
            self._hours = hours
        else:
            raise TypeError("Hours have to be integers between 0 and 23!")
        if type(minutes) == int and 0 <= minutes and minutes < 60:
            self.__minutes = minutes 
        else:
            raise TypeError("Minutes have to be integers between 0 and 59!")
        if type(seconds) == int and 0 <= seconds and seconds < 60:
            self.__seconds = seconds
        else:
            raise TypeError("Seconds have to be integers between 0 and 59!")

    def __str__(self):
        return "{0:02d}:{1:02d}:{2:02d}".format(self._hours,
                                                self.__minutes,
                                                self.__seconds)

    def tick(self):
        """
        This method lets the clock "tick", this means that the 
        internal time will be advanced by one second.

        Examples:
        >>> x = Clock(12,59,59)
        >>> print(x)
        12:59:59
        >>> x.tick()
        >>> print(x)
        13:00:00
        >>> x.tick()
        >>> print(x)
        13:00:01
        """

        if self.__seconds == 59:
            self.__seconds = 0
            if self.__minutes == 59:
                self.__minutes = 0
                if self._hours == 23:
                    self._hours = 0
                else:
                    self._hours += 1
            else:
                self.__minutes += 1
        else:
            self.__seconds += 1


if __name__ == "__main__":
    x = Clock(23,59,59)
    print(x)
    x.tick()
    print(x)
    y = str(x)
    print(type(y))

If you call this module standalone, you get the following output: 

$ python3 clock.py 
23:59:59
00:00:00
<class 'str'>

Let’s check our exception handling by inputting floats and strings as input. We also check, what happens, if we exceed the limits of the expected values: 

>>> from clock import Clock
>>> x = Clock(7.7,45,17)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "clock.py", line 16, in __init__
    self.set_Clock(hours, minutes, seconds)
  File "clock.py", line 30, in set_Clock
    raise TypeError("Hours have to be integers between 0 and 23!")
TypeError: Hours have to be integers between 0 and 23!
>>> x = Clock(24,45,17)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "clock.py", line 16, in __init__
    self.set_Clock(hours, minutes, seconds)
  File "clock.py", line 30, in set_Clock
    raise TypeError("Hours have to be integers between 0 and 23!")
TypeError: Hours have to be integers between 0 and 23!
>>> x = Clock(23,60,17)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "clock.py", line 16, in __init__
    self.set_Clock(hours, minutes, seconds)
  File "clock.py", line 34, in set_Clock
    raise TypeError("Minutes have to be integers between 0 and 59!")
TypeError: Minutes have to be integers between 0 and 59!
>>> x = Clock("23","60","17")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "clock.py", line 16, in __init__
    self.set_Clock(hours, minutes, seconds)
  File "clock.py", line 30, in set_Clock
    raise TypeError("Hours have to be integers between 0 and 23!")
TypeError: Hours have to be integers between 0 and 23!
>>> 
>>> x = Clock(23,17)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: __init__() takes exactly 4 arguments (3 given)
>>>

We will now create a class “Calendar”, which has lots of similarities to the previously defined Clock class. Instead of “tick” we have an “advance” method, which advances the date by one day, whenever it is called. Adding a day to a date is quite tricky. We have to check, if the date is the last day in a month and the number of days in the months vary. As if this isn’t bad enough, we have February and the leap year problem.

The rules for calculating a leap year are the following:

  • If a year is divisible by 400, it is a leap year.
  • If a year is not divisible by 400 but by 100, it is not a leap year.
  • A year number which is divisible by 4 but not by 100, it is a leap year.
  • All other year numbers are common years, i.e. no leap years.

As a little useful gimmick, we added a possibility to output a date either in British or in American (Canadian) style. 

""" 
The class Calendar implements a calendar.   
"""

class Calendar(object):

    months = (31,28,31,30,31,30,31,31,30,31,30,31)
    date_style = "British"

    @staticmethod
    def leapyear(year):
        """ 
        The method leapyear returns True if the parameter year
        is a leap year, False otherwise
        """

        if year % 4 == 0:
            if year % 100 == 0:
                # divisible by 4 and by 100
                if year % 400 == 0:
                    leapyear = True
                else:
                    leapyear = False
            else:
                # divisible by 4 but not by 100
                leapyear = True
        else:
            # not divisible by 4
            leapyear = False

        return leapyear


    def __init__(self, d, m, y):
        """
        d, m, y have to be integer values and year has to be 
        a four digit year number
        """

        self.set_Calendar(d,m,y)


    def set_Calendar(self, d, m, y):
        """
        d, m, y have to be integer values and year has to be 
        a four digit year number
        """

        if type(d) == int and type(m) == int and type(y) == int:
            self.__days = d
            self.__months = m
            self.__years = y
        else:
            raise TypeError("d, m, y have to be integers!")


    def __str__(self):
        if Calendar.date_style == "British":
            return "{0:02d}/{1:02d}/{2:4d}".format(self.__days,
                                                   self.__months,
                                                   self.__years)
        else: 
            # assuming American style
            return "{0:02d}/{1:02d}/{2:4d}".format(self.__months,
                                                   self.__days,
                                                   self.__years)



    def advance(self):
        """
        This method advances to the next date.
        """

        max_days = Calendar.months[self.__months-1]
        if self.__months == 2 and Calendar.leapyear(self.__years):
            max_days += 1
        if self.__days == max_days:
            self.__days= 1
            if self.__months == 12:
                self.__months = 1
                self.__years += 1
            else:
                self.__months += 1
        else:
            self.__days += 1


if __name__ == "__main__":
    x = Calendar(31,12,2012)
    print(x, end=" ")
    x.advance()
    print("after applying advance: ", x)
    print("2012 was a leapyear:")
    x = Calendar(28,2,2012)
    print(x, end=" ")
    x.advance()
    print("after applying advance: ", x)
    x = Calendar(28,2,2013)
    print(x, end=" ")
    x.advance()
    print("after applying advance: ", x)
    print("1900 no leapyear: number divisible by 100 but not by 400: ")
    x = Calendar(28,2,1900)
    print(x, end=" ")
    x.advance()
    print("after applying advance: ", x)
    print("2000 was a leapyear, because number divisibe by 400: ")
    x = Calendar(28,2,2000)
    print(x, end=" ")
    x.advance()
    print("after applying advance: ", x)
    print("Switching to American date style: ")
    Calendar.date_style = "American"
    print("after applying advance: ", x)  

Starting this script provides us with the following results: 

$ python3 calendar.py
31.12.2012 after applying advance:  01.01.2013
2012 was a leapyear:
28.02.2012 after applying advance:  29.02.2012
28.02.2013 after applying advance:  01.03.2013
1900 no leapyear: number divisible by 100 but not by 400: 
28.02.1900 after applying advance:  01.03.1900
2000 was a leapyear, because number divisibe by 400: 
28.02.2000 after applying advance:  29.02.2000
Switching to American date style: 
after applying advance:  02/29/2000

At last, we will come to our multiple inheritance example. We are now capable of implementing the originally intended class CalendarClock, which will inherit from both Clock and Calendar. The method “tick” of Clock will have to be overloaded. However, the new tick method of CalendarClock has to call the tick method of Clock: Clock.tick(self) 

""" 
Modul, which implements the class CalendarClock.
"""

from clock import Clock
from calendar import Calendar


class CalendarClock(Clock, Calendar):
    """ 
        The class CalendarClock implements a clock with integrated 
        calendar. It's a case of multiple inheritance, as it inherits 
        both from Clock and Calendar      
    """

    def __init__(self,day, month, year, hour, minute, second):
        Clock.__init__(self,hour, minute, second)
        Calendar.__init__(self,day, month, year)


    def tick(self):
        """
        advance the clock by one second
        """
        previous_hour = self._hours
        Clock.tick(self)
        if (self._hours < previous_hour): 
            self.advance()

    def __str__(self):
        return Calendar.__str__(self) + ", " + Clock.__str__(self)


if __name__ == "__main__":
    x = CalendarClock(31,12,2013,23,59,59)
    print("One tick from ",x, end=" ")
    x.tick()
    print("to ", x)

    x = CalendarClock(28,2,1900,23,59,59)
    print("One tick from ",x, end=" ")
    x.tick()
    print("to ", x)

    x = CalendarClock(28,2,2000,23,59,59)
    print("One tick from ",x, end=" ")
    x.tick()
    print("to ", x)

    x = CalendarClock(7,2,2013,13,55,40)
    print("One tick from ",x, end=" ")
    x.tick()
    print("to ", x)

The output of the program hopefully further clarifies what’s going on in this class: 

$ python3 calendar_clock.py 
One tick from  31/12/2013, 23:59:59 to  01/01/2014, 00:00:00
One tick from  28/02/1900, 23:59:59 to  01/03/1900, 00:00:00
One tick from  28/02/2000, 23:59:59 to  29/02/2000, 00:00:00
One tick from  07/02/2013, 13:55:40 to  07/02/2013, 13:55:41

The Diamand Problem or the ,,deadly diamond of death’’

The “diamond problem” (sometimes referred to as the “deadly diamond of death”) is the generally used term for an ambiguity that arises when two classes B and C inherit from a superclass A, and another class D inherits from both B and C. If there is a method “m” in A that B or C (or even both of them) )has overridden, and furthermore, if does not override this method, then the question is which version of the method does D inherit? It could be the one from A, B or C

Let’s look at Python. The first Diamond Problem configuration is like this: Both B and C override the method m of A: 

class A:
    def m(self):
        print("m of A called")

class B(A):
    def m(self):
        print("m of B called")
    
class C(A):
    def m(self):
        print("m of C called")

class D(B,C):
    pass

If you call the method m on an instance x of D, i.e. x.m(), we will get the output “m of B called”. If we transpose the order of the classes in the class header of D in “class D(C,B):”, we will get the output “m of C called”.

The case in which m will be overridden only in one of the classes B or C, e.g. in C: 

class A:
    def m(self):
        print("m of A called")

class B(A):
    pass
    
class C(A):
    def m(self):
        print("m of C called")

class D(B,C):
    pass

x = D()
x.m()

Principially, two possibilities are imaginable: “m of C” or “m of A” could be used

We call this script with Python2.7 (python) and with Python3 (python3) to see what’s happening: 

$ python diamond1.py 
m of A called
$ python3 diamond1.py 
m of C called

Only for those who are interested in Python version2:

To have the same inheritance behaviour in Python2 as in Python3, every class has to inherit from the class “object”. Our class A doesn’t inherit from object, so we get a so-called old-style class, if we call the script with python2. Multiple inheritance with old-style classes is governed by two rules: depth-first and then left-to-right. If you change the header line of A into “class A(object):”, we will have the same behaviour in both Python versions.

super and MROWe have seen in our previous implementation of the diamond problem, how Python “solves” the problem, i.e. in which order the base classes are browsed through. The order is defined by the so-called “Method Resolution Order” or in short MRO.

1

We will extend our previous example, so that every class defines its own method m: 

class A:
    def m(self):
        print("m of A called")

class B(A):
    def m(self):
        print("m of B called")
    
class C(A):
    def m(self):
        print("m of C called")

class D(B,C):
    def m(self):
        print("m of D called")

Let’s apply the method m on an instance of D. We can see, that only the code of the method m of D will be executed. We can also explicitly call the methods m of the other classes via the class name, as we demonstrate in the following interactive Python session: 

>>> from super1 import A,B,C,D
>>> x = D()
>>> B.m(x)
m of B called
>>> C.m(x)
m of C called
>>> A.m(x)
m of A called

Now let’s assume, that the method m of D should execute the code of m of B, C and A as well, when it is called. We could implement it like this: 

class D(B,C):
    def m(self):
        print("m of D called")
        B.m(self)
        C.m(self)
        A.m(self)

The output is what we have been looking for: 

>>> from mro import D
>>> x = D()
>>> x.m()
m of D called
m of B called
m of C called
m of A called

But it turns out once more, that things are more complicated than it seems. How can we cope with the situation, if both m of B and m of C will have to call m of A as well. In this case, we have to take away the call A.m(self) from m in D. The code might look like this, but there is still a bug lurking in it: 

class A:
    def m(self):
        print("m of A called")

class B(A):
    def m(self):
        print("m of B called")
        A.m(self)
    
class C(A):
    def m(self):
        print("m of C called")
        A.m(self)

class D(B,C):
    def m(self):
        print("m of D called")
        B.m(self)
        C.m(self)

The bug is, that the method m of A will be called twice: 

>>> from super3 import D
>>> x = D()
>>> x.m()
m of D called
m of B called
m of A called
m of C called
m of A called

One way to solve this problem – admittedly not a Pythonic one – consists in splitting the methods m of B and C in two methods. The first method, called _m consists of the specific code for B and C and the other method is still called m, but consists now of a call “self._m()” and a call “A.m(self)”. The code of the method m of D consists now of the specific code of D ‘print(“m of D called”)’, and the calls B._m(self), C._m(self) and A.m(self): 

class A:
    def m(self):
        print("m of A called")

class B(A):
    def _m(self):
        print("m of B called")
    def m(self):
        self._m()
        A.m(self)
    
class C(A):
    def _m(self):
        print("m of C called")
    def m(self):
        self._m()
        A.m(self)

class D(B,C):
    def m(self):
        print("m of D called")
        B._m(self)
        C._m(self)
        A.m(self)

Our problem is solved, but – as we have already mentioned – not in a pythonic way: 

>>> from super4 import D
>>> x = D()
>>> x.m()
m of D called
m of B called
m of C called
m of A called

The optimal way to solve the problem, which is the “super” pythonic way, consists in calling the super function: 

class A:
    def m(self):
        print("m of A called")

class B(A):
    def m(self):
        print("m of B called")
        super().m()
    
class C(A):
    def m(self):
        print("m of C called")
        super().m()

class D(B,C):
    def m(self):
        print("m of D called")
        super().m()

It also solves our problem, but in a beautiful design as well: 

>>> from super5 import D
>>> x = D()
>>> x.m()
m of D called
m of B called
m of C called
m of A called

The super function is often used when instances are initialized with the __init__ method: 

class A:
    def __init__(self):
        print("A.__init__")

class B(A):
    def __init__(self):
        print("B.__init__")
        super().__init__()
    
class C(A):
    def __init__(self):
        print("C.__init__")
        super().__init__()


class D(B,C):
    def __init__(self):
        print("D.__init__")
        super().__init__()

We demonstrate the way of working in the following interactive session: 

>>> from super_init import A,B,C,D
>>> d = D()
D.__init__
B.__init__
C.__init__
A.__init__
>>> c = C()
C.__init__
A.__init__
>>> b = B()
B.__init__
A.__init__
>>> a = A()
A.__init__

The question arises how the super functions makes its decision. How does it decide which class has to be used? As we have already mentioned, it uses the so-called method resolution order(MRO). It is based on the “C3 superclass linearisation” algorithm. This is called a linearisation, because the tree structure is broken down into a linear order. The mro method can be used to create this list: 

>>> from super_init import A,B,C,D
>>> D.mro()
[<class 'super_init.D'>, <class 'super_init.B'>, <class 'super_init.C'>, <class 'super_init.A'>, <class 'object'>]
>>> B.mro()
[<class 'super_init.B'>, <class 'super_init.A'>, <class 'object'>]
>>> A.mro()
[<class 'super_init.A'>, <class 'object'>]

Polymorphism

Polymorphism is construed from two Greek words. “Poly” stands for “much” or “many” and “morph” means shape or form. Polymorphism is the state or condition of being polymorphous, or if we use the translations of the components “the ability to be in many shapes or forms. Polymorphism is a term used in many scientific areas. In Crystallography it defines the state, if something crystallizes into two or more chemically identical but crystallographically distinct forms. Biologists know polymorphism as the existence of an organism in several form or colour varieties. The Romans even had a god, called Morpheus, who is able to take any human form: Morheus appears in Ovid’s metamorphoses and is the son of Somnus, the god of sleep. You can admire Morpheus and Iris in the picture on the right side.

So, before we fall to sleep, we get back to Python and to what polymorphism means in the programming language context. Polymorphism in Computer Science is the ability to present the same interface for differing underlying forms. We can have in some programming languages polymorphic functions or methods for example. Polymorphic functions or methods can be applied to arguments of different types, and they can behave differently depending on the type of the arguments to which they are applied. We can also define the same function name with a varying number of parameter.

Let’s have a look at the following Python function: 

def f(x, y):
    print("values: ", x, y)

f(42,43)
f(42, 43.7) 
f(42.3,43)
f(42.0, 43.9)

We can call this function with various types, as demonstrated in the example. In typed programming languages like Java or C++, we would have to overload f for the various type combinations.

Our example could be implemented like this in C++: 

#include <iostream>
using namespace std;

void f(int x, int y ) {
    cout << "values: " << x << ", " << x << endl;
}

void f(int x, double y ) {
    cout << "values: " << x << ", " << x << endl;
}

void f(double x, int y ) {
    cout << "values: " << x << ", " << x << endl;
}

void f(double x, double y ) {
    cout << "values: " << x << ", " << x << endl;
}


int main()
{
    f(42, 43); 
    f(42, 43.7); 
    f(42.3,43);
    f(42.0, 43.9); 
}

Python is implicitly polymorphic. We can apply our previously defined function f even to lists, strings or other types, which can be printed:

>>> def f(x,y):
...     print("values: ", x, y)
... 
>>> f([3,5,6],(3,5))
values:  [3, 5, 6] (3, 5)
>>> f("A String", ("A tuple", "with Strings"))
values:  A String ('A tuple', 'with Strings')
>>> f({2,3,9}, {"a":3.4,"b":7.8, "c":9.04})
values:  {9, 2, 3} {'a': 3.4, 'c': 9.04, 'b': 7.8}
>>>

by ⋅ Link

original source : http://nbviewer.ipython.org/github/rasbt/python_reference/blob/master/tutorials/scope_resolution_legb_rule.ipynb

Introduction to namespaces and scopes

Namespaces

Roughly speaking, namespaces are just containers for mapping names to objects. As you might have already heard, everything in Python – literals, lists, dictionaries, functions, classes, etc. – is an object.
Such a “name-to-object” mapping allows us to access an object by a name that we’ve assigned to it. E.g., if we make a simple string assignment via a_string = "Hello string", we created a reference to the "Hello string" object, and henceforth we can access via its variable name a_string.

We can picture a namespace as a Python dictionary structure, where the dictionary keys represent the names and the dictionary values the object itself (and this is also how namespaces are currently implemented in Python), e.g.,

a_namespace = {'name_a':object_1, 'name_b':object_2, ...}

Now, the tricky part is that we have multiple independent namespaces in Python, and names can be reused for different namespaces (only the objects are unique, for example:

a_namespace = {'name_a':object_1, 'name_b':object_2, ...}
b_namespace = {'name_a':object_3, 'name_b':object_4, ...}

For example, everytime we call a for-loop or define a function, it will create its own namespace. Namespaces also have different levels of hierarchy (the so-called “scope”), which we will discuss in more detail in the next section.

Scope

In the section above, we have learned that namespaces can exist independently from each other and that they are structured in a certain hierarchy, which brings us to the concept of “scope”. The “scope” in Python defines the “hierarchy level” in which we search namespaces for certain “name-to-object” mappings.
For example, let us consider the following code:

In [1]:

i = 1

def foo():
    i = 5
    print(i, 'in foo()')

print(i, 'global')

foo()

1 global
5 in foo()

Here, we just defined the variable name i twice, once on the foo function.

  • foo_namespace = {'i':object_3, ...}
  • global_namespace = {'i':object_1, 'name_b':object_2, ...}

So, how does Python know which namespace it has to search if we want to print the value of the variable i? This is where Python’s LEGB-rule comes into play, which we will discuss in the next section.

Tip:

If we want to print out the dictionary mapping of the global and local variables, we can use the the functions global() and local()

In [11]:

#print(globals()) # prints global namespace
#print(locals()) # prints local namespace

glob = 1

def foo():
    loc = 5
    print('loc in foo():', 'loc' in locals())

foo()
print('loc in global:', 'loc' in globals())    
print('glob in global:', 'foo' in globals())

loc in foo(): True
loc in global: False
glob in global: True

Scope resolution for variable names via the LEGB rule.

We have seen that multiple namespaces can exist independently from each other and that they can contain the same variable names on different hierachy levels. The “scope” defines on which hierarchy level Python searches for a particular “variable name” for its associated object. Now, the next question is: “In which order does Python search the different levels of namespaces before it finds the name-to-object’ mapping?”
To answer is: It uses the LEGB-rule, which stands for

Local -> Enclosed -> Global -> Built-in,

where the arrows should denote the direction of the namespace-hierarchy search order.

  • Local can be inside a function or class method, for example.
  • Enclosed can be its enclosing function, e.g., if a function is wrapped inside another function.
  • Global refers to the uppermost level of the executing script itself, and
  • Built-in are special names that Python reserves for itself.

So, if a particular name:object mapping cannot be found in the local namespaces, the namespaces of the enclosed scope are being searched next. If the search in the enclosed scope is unsuccessful, too, Python moves on to the global namespace, and eventually, it will search the built-in namespace (side note: if a name cannot found in any of the namespaces, a NameError will is raised).

Note:
Namespaces can also be further nested, for example if we import modules, or if we are defining new classes. In those cases we have to use prefixes to access those nested namespaces. Let me illustrate this concept in the following code block:

In [8]:

import numpy
import math
import scipy

print(math.pi, 'from the math module')
print(numpy.pi, 'from the numpy package')
print(scipy.pi, 'from the scipy package')

3.141592653589793 from the math module
3.141592653589793 from the numpy package
3.141592653589793 from the scipy package

(This is also why we have to be careful if we import modules via “from a_module import *”, since it loads the variable names into the global namespace and could potentially overwrite already existing variable names)

1. LG – Local and Global scopes

Example 1.1
As a warm-up exercise, let us first forget about the enclosed (E) and built-in (B) scopes in the LEGB rule and only take a look at LG – the local and global scopes.
What does the following code print?

In [1]:

a_var = 'global variable'

def a_func():
    print(a_var, '[ a_var inside a_func() ]')

a_func()
print(a_var, '[ a_var outside a_func() ]')

c)

global value [ a_var inside a_func() ]  
global value [ a_var outside a_func() ]

[go to solution]

Here is why:

We call a_func() first, which is supposed to print the value of a_var. According to the LEGB rule, the function will first look in its own local scope (L) if a_var is defined there. Since a_func() does not define its own a_var, it will look one-level above in the global scope (G) in which a_var has been defined previously.

Example 1.2
Now, let us define the variable a_var in the global and the local scope.
Can you guess what the following code will produce?

In [2]:

a_var = 'global value'

def a_func():
    a_var = 'local value'
    print(a_var, '[ a_var inside a_func() ]')

a_func()
print(a_var, '[ a_var outside a_func() ]')

b)

local value [ a_var inside a_func() ]
global value [ a_var outside a_func() ]

[go to solution]

Here is why:

When we call a_func(), it will first look in its local scope (L) for a_var, since a_var is defined in the local scope of a_func, its assigned value local variable is printed. Note that this doesn’t affect the global variable, which is in a different scope.

However, it is also possible to modify the global by, e.g., re-assigning a new value to it if we use the global keyword as the following example will illustrate:

In [3]:

a_var = 'global value'

def a_func():
    global a_var
    a_var = 'local value'
    print(a_var, '[ a_var inside a_func() ]')

print(a_var, '[ a_var outside a_func() ]')
a_func()
print(a_var, '[ a_var outside a_func() ]')

global value [ a_var outside a_func() ]
local value [ a_var inside a_func() ]
local value [ a_var outside a_func() ]

But we have to be careful about the order: it is easy to raise an UnboundLocalError if we don’t explicitly tell Python that we want to use the global scope and try to modify a variable’s value (remember, the right side of an assignment operation is executed first):

In [4]:

a_var = 1

def a_func():
    a_var = a_var + 1
    print(a_var, '[ a_var inside a_func() ]')

print(a_var, '[ a_var outside a_func() ]')
a_func()

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-4-a6cdd0ee9a55> in <module>()
      6 
      7 print(a_var, '[ a_var outside a_func() ]')
----> 8 a_func()

<ipython-input-4-a6cdd0ee9a55> in a_func()
      2 
      3 def a_func():
----> 4     a_var = a_var + 1
      5     print(a_var, '[ a_var inside a_func() ]')
      6 

UnboundLocalError: local variable 'a_var' referenced before assignment
1 [ a_var outside a_func() ]

2. LEG – Local, Enclosed, and Global scope

Now, let us introduce the concept of the enclosed (E) scope. Following the order “Local -> Enclosed -> Global”, can you guess what the following code will print?

Example 2.1

In [4]:

a_var = 'global value'

def outer():
    a_var = 'enclosed value'
    
    def inner():
        a_var = 'local value'
        print(a_var)
    
    inner()

outer()

c)

local value

[go to solution]

Here is why:

Let us quickly recapitulate what we just did: We called outer(), which defined the variable a_var locally (next to an existing a_var in the global scope). Next, theouter() function called inner(), which in turn defined a variable with of name a_var as well. The print() function inside inner() searched in the local scope first (L->E) before it went up in the scope hierarchy, and therefore it printed the value that was assigned in the local scope.

Similar to the concept of the global keyword, which we have seen in the section above, we can use the keyword nonlocal inside the inner function to explicitly access a variable from the outer (enclosed) scope in order to modify its value.
Note that the nonlocal keyword was added in Python 3.x and is not implemented in Python 2.x (yet).

In [5]:

a_var = 'global value'

def outer():
       a_var = 'local value'
       print('outer before:', a_var)
       def inner():
           nonlocal a_var
           a_var = 'inner value'
           print('in inner():', a_var)
       inner()
       print("outer after:", a_var)
outer()

outer before: local value
in inner(): inner value
outer after: inner value

3. LEGB – Local, Enclosed, Global, Built-in

To wrap up the LEGB rule, let us come to the built-in scope. Here, we will define our “own” length-funcion, which happens to bear the same name as the in-builtlen() function. What outcome do you excpect if we’d execute the following code?

Example 3

In [6]:

a_var = 'global variable'

def len(in_var):
    print('called my len() function')
    l = 0
    for i in in_var:
        l += 1
    return l

def a_func(in_var):
    len_in_var = len(in_var)
    print('Input variable is of length', len_in_var)

a_func('Hello, World!')

a)

raises an error (conflict with in-built `len()` function)

b)

called my len() function
Input variable is of length 13

c)

Input variable is of length 13

[go to solution]

Here is why:

Since the exact same names can be used to map names to different objects – as long as the names are in different name spaces – there is no problem of reusing the name len to define our own length function (this is just for demonstration pruposes, it is NOT recommended). As we go up in Python’s L -> E -> G -> B hierarchy, the function a_func() finds len() already in the global scope first before it attempts

Self-assessment exercise

Now, after we went through a couple of exercises, let us quickly check where we are. So, one more time: What would the following code print out?

In [59]:

a = 'global'

def outer():
    
    def len(in_var):
        print('called my len() function: ', end="")
        l = 0
        for i in in_var:
            l += 1
        return l
    
    a = 'local'
    
    def inner():
        global len
        nonlocal a
        a += ' variable'
    inner()
    print('a is', a)
    print(len(a))


outer()

print(len(a))
print('a is', a)

Conclusion

I hope this short tutorial was helpful to understand the basic concept of Python’s scope resolution order using the LEGB rule. I want to encourage you (as a little self-assessment exercise) to look at the code snippets again tomorrow and check if you can correctly predict all their outcomes.

A rule of thumb

In practice, it is usually a bad idea to modify global variables inside the function scope, since it often be the cause of confusion and weird errors that are hard to debug.
If you want to modify a global variable via a function, it is recommended to pass it as an argument and reassign the return-value.
For example:

In [42]:

a_var = 2

def a_func(some_var):
    return 2**3

a_var = a_func(a_var)
print(a_var)

8