Codelog

This problem has haunted me many times when I want to work on two frameworks (Turbogears and Pylons) and there are some packages which interfere with each other. Having both installed in /usr/lib/python-2.5/site-packages is a nightmare. You won’t know what causes your nosetests to fail, you will have to comment out the other dirty package and try again. Also some packages may be upgraded which will cause some problem in some place in your application.

All this can be easily solved by using the virtualenv package in python. What it does is - create isolated python working environments. So that packages you install in one environment doesn’t interfere with the other. Also if you can’t install packages in the global site-packages, this will be really helpful.

You just have to easy_install virtualenv and it gets installed as a executable. If you can’t easy_install, there is a single python script (virtualenv.py) which can be used instead.

You can create your sandbox with the command $ virtualenv --no-site-packages env. The –no-site-packages option make sure that the global site-packages doesn’t inherit any packages from the global site-packages.

It will create a directory ‘env’ in your current directory. The bin folder inside this has its own python executable and a seperate site-packages in the lib directory. It also has easy_install installed in the bin.

Now you can install whatever package you want by typing $ env/bin/easy_install packagename

All your commands need to be prefixed with the env/bin/ path else it will use the wrong python version. But you can avoid typing the long commands by using the activate script bundled with the bin folder.

On *nix systems, you have to $ source bin/activate and on a windows machine > \path\to\env\bin\activate.bat. After you are finished with it, you just have to type deactivate and you get back to the old shell.

Most of the times, it would work perfectly, but sometimes the shell will cache the path of the commands. So, it is always advisable to check whether the path is set correctly by typing which python.

Now this solves my problem of mismatched and wrong packages and can start working in Pylons.

Here is a linkdump of resources related to python which I had in my bookmarks. These are some advanced topics which would really help you to better understand the internals and write better code.

• Python’s dot operator and the magic self object - explains about the way object orientedness works in python.
• Want to do operator overloading? Want to know the list of all special __underscore__ methods? It would really help you to have this as a quick reference.
• Metaclasses are classes whose instances are classes. This is a bit confusing to understand and would require you to read through the Part 1, Part 2 and Part 3 of the IBM’s developer works Metaclass programming in python series.

This post is due for a long time since my post on Singular form of a word program and here it is - Doctests in Python. From the python library reference,

The doctest module searches for pieces of text that look like interactive Python sessions, and then executes those sessions to verify that they work exactly as shown.

Doctests are an easy way to test small functions by specifying the output of the function in docstrings. In python you can (should) write docstrings for classes and methods which can be used to generate the documentation for the module. In those docstrings, you just specify calls to the function with the return string specified. The doctest module automatically asserts the output and verifies whether the test passes or fails.

Lets see an example of doctest and how to run them. Lets create a function which adds two numbers.

def add(x, y):
    """Adds two numbers/strings together

    >>> add(1,2)
    3
    >>> add('foo','bar')
    'foobar'
    """
    return x+y

Here we have written the docstrings and have four lines which look like the python shell. There we call the function with various arguments and also specify how the function will return.

So, how do we run the test? Just include these lines in the module and run the file directly. This is the usual way we run doctest (in the __main__ part of the file).

def _test():
    import doctest
    doctest.testmod()

if __name__ == "__main__":
    _test()

You just have to call the doctest.testmod() function and it will search for all such tests in the docstrings and outputs nothing if the tests pass. If you want to get more verbose output, run the file as python filename.py -v

There are many advantages of doctests (wikipedia has a lot more):

• Easy to understand what a function is going to do if called
• Tests and code together - easy to update the test if code changes
• Most of the times I execute my functions on the shell - so I just have to copy-paste the output from the shell to the code

Once you start using doctests, you would find yourself trying to use it in all possible places. For more elaborate testing, you should probably use pyunit.

Is there any other programming language which has tests written directly in the documentations? If you were to design such a testing module for you favourite language, how would you do it (especially if it doesn’t have a interactive shell)? Do explain your views on it below.

In turbogears you have three important config files. One is the dev.cfg file used during development, another is prod.cfg used during deploying your code. The last one is app.cfg file which has parameters which are common to both dev and prod.

When you store the parameters in the app.cfg file, it is good to group related parameters together under a section. A new section can be created by enclosing a word with [ and ], eg: [API]

All this is fine till we come to the point of reading back the values from the config file. We faced this same problem and whenever we tried to read it back, it used to give some error about no such variable present.

After looking through the code of the configparser.py for cherrypy, we saw that there was one more additional parameter called path. The path parameter takes in the section name and the variable inside the section is taken in by the parameter key.

So instead of doing a tg.config.get('url'), you would need to do a tg.config.get('url', path='API').

Though this was slightly different than the usual config parser module, it solved the problem of properly categorizing the config file.

On reading my last post singular form of a word, a friend of mine (Sathya) didn’t understand what I had done in the lambda functions. It was of the form lambda w: w[-3:] == ‘ies’ and w[:-3] + ‘y’. I had forgotten to explain about that in that post.

There is one small restriction in lambdas - you can’t use if-else constructs in them. If you want to use a condition check in your lambda, then it has to be of this form

lambda : <condition> and <if block> or <else block>

This is easy to understand when you compare it with conditional or ternary operators in C (?:) - condition?if block:else block.

Last post I talked about using list comprehensions and promised to post a simple method to find the singular form of a word. When I wanted to write such a function, I initially thought of using a chain of if-elif-else constructs. But I didn’t like it that way. So I used lambda functions and list comprehensions to do the same job.

def singularize(word):
    """Return the singular form of a word

    >>> singularize('rabbits')
    'rabbit'
    >>> singularize('potatoes')
    'potato'
    >>> singularize('leaves')
    'leaf'
    >>> singularize('knives')
    'knife'
    >>> singularize('spies')
    'spy'
    """
    sing_rules = [lambda w: w[-3:] == 'ies' and w[:-3] + 'y',
                  lambda w: w[-4:] == 'ives' and w[:-4] + 'ife',
                  lambda w: w[-3:] == 'ves' and w[:-3] + 'f',
                  lambda w: w[-2:] == 'es' and w[:-2],
                  lambda w: w[-1:] == 's' and w[:-1],
                  lambda w: w,
                  ]
    word = word.strip()
    singleword = [f(word) for f in sing_rules if f(word) is not False][0]
    return singleword

def _test():
    import doctest
    doctest.testmod()

if __name__ == '__main__':
    _test()

This method is simple if you know the rules of plurals in the english language. I have converted each rule into a lambda function which returns the corresponding singular word depending on the word ending. The order of the rules are important, for eg., i should first check for -es before checking for -s. I haven’t taken care of the special plurals like men-man, people-person, children-child.

I then use list comprehensions to apply each function on the input word and pick the 0th element. Now you get the singular for of the word.

If you notice in the function, I have used something called doctests which is a very easy way to test your functions. You can download the source code and run it to run the doctest. Wow, now I got another topic to write about in the next post - doctests in python.

In the last post about Map-Filter-Reduce before finishing the post, I told about slight changes to those functions and how you can do the same job using list comprehensions. I would be talking about it here today.

Three years back Guido van Rossum, the BFDL made a blog post about the fate of reduce() in Python 3000. He stated that map and filter confusing to many people and that list comprehensions were simple to read and understand. Guido wanted reduce() to be totally removed from the language as there weren’t any good uses of it except + and *.

List Comprehensions
So, what is this list comprehensions that Guido was talking about. It is just a concise way to create lists from another list (just like map() and filter()). It is easy on the eyes and clearer to understand.

It consists of an expression followed by a for clause and more for/if clauses. The result of this is a list gotten by evaluating the expression in the context of the for and if clauses.

>>> words = ['FOO', 'BAR', 'BAZ']
>>> [word.lower() for word in words]
['foo', 'bar', 'baz']

>>> nums = [1,2,3,4]
>>> [(x, x*x) for x in nums]
[(1, 1), (2, 4), (3, 9), (4, 16)]

This looks just like the map function. Now just add an if clause after the for, and you get the filter equivalent.

>>> nums = [1,2,3,4,5,6]
>>> [i for i in nums if i%2 == 0]
[2, 4, 6]

List comprehensions are easier to understand by just reading it as words - create a list of is for every i in nums if i%2 is equal to 0. Also list comprehensions are faster than map and filter as it doesn’t have to make function calls.

So, even though Guido wanted all the four functions to be removed from the standard library, he has announced that lambda, map and filter would stay - with the latter two having small changes - returning iterators instead of a list. Unfortunately reduce() is moved over to functools module.

So my next post is going to be a small example of using list comprehensions to find the singular form of a word.

This is my first post here at codelog and all my code snippets for this post would be written in python even though the topic is mostly language agnostic. This is a fairly common feature found popularly in functional programming languages like lisp.

Before we see examples of map-reduce-filter, I would have to explain about lambda functions(which I would be using a lot) - borrowed from lisp.

Lambda Functions

Lambda functions are nameless functions which can be used primarily to define one line functions which need to be passed around. Lambda functions are written using the lambda keyword.

>>> lambda x: x*2
<function <lambda> at 0x81d617c>

This lambda function takes in one argument x and returns twice the input number. This may seems too simple at first, but believe me you can easily write complex stuff using these. You can store these lambda functions and/or pass them around as parameters to other functions.

Map

Map is a function which takes in two arguments - first is a function and second is an array (or list). What it does is simple. It applies the function over all elements of the list and returns another list.

>>> map(lambda x: x*2, [1,2,3,4])
[2, 4, 6, 8]

This doubles all the elements in the list. Here is another example which takes in a list of tuples and adds the two elements in each tuple and creates another list.

>>> map(lambda x: x[0]+x[1], [(1,2), (3,4), (5,6)])
[3, 7, 11]

You can do real cool stuff using these.

Reduce

Reduce also takes in two parameters like map - one is a function and another is a list. But it instead applies the function on two successive parameters from the list. This is useful for adding all the elements in a list.

>>> reduce(lambda x, y: x+y, [1,2,3,4,5,6,7,8,9,10])
55

First it adds 1 and 2, then it adds the result to the next element 3, whose result is added to 4 and so on. This is pretty simple, but you have to keep in mind that the function that you pass in to reduce function takes in 2 parameters.

Filter

Filter is a function which also takes in a method and a list - but returns back another list with all elements which return True when passed to the method. Might seem confusing first - but as the word says it filters out the elements of the list which matches the condition.

>>> filter(lambda x: x%2 == 0, [1,2,3,4,5,6,7,8])
[2, 4, 6, 8]

The lambda function there is very simple. It just checks whether the number is odd or even. It returns True if it is even and False if it is odd. Passing it to a filter function with a list of numbers generates a list with only elements which pass the filter - even numbers.

In Python 3 (which is to be released very soon almost 991 years in advance) there will be slight changes to these four functions. Maybe I should make it as another post in which we will see how we can do the same stuff using list comprehensions - which is the suggested method.