Welcome


... to my place on the web - a personal website where I keep articles, freeware programs and code snippets I've written, and a weblog where I share my ideas and insights on programming, technology, books, and life in general.

To browse the website, use the site map on the right-hand side of the screen. Below, you will find the most recent posts.

Dynamically generating Python test cases

April 2nd, 2014 at 6:44 am

Testing is crucial. While many different kinds and levels of testing exist, there’s good library support only for unit tests (the Python unittest package and its moral equivalents in other languages). However, unit testing does not cover all kinds of testing we may want to do – for example, all kinds of whole program tests and integration tests. This is where we usually end up with a custom "test runner" script.

Having written my share of such custom test runners, I’ve recently gravitated towards a very convenient approach which I want to share here. In short, I’m actually using Python’s unittest, combined with the dynamic nature of the language, to run all kinds of tests.

Let’s assume my tests are some sort of data files which have to be fed to a program. The output of the program is compared to some "expected results" file, or maybe is encoded in the data file itself in some way. The details of this are immaterial, but seasoned programmers usually encounter such testing rigs very frequently. It commonly comes up when the program under test is a data-transformation mechanism of some sort (compiler, encryptor, encoder, compressor, translator etc.)

So you write a "test runner". A script that looks at some directory tree, finds all the "test files" there, runs each through the transformation, compares, reports, etc. I’m sure all these test runners share a lot of common infrastructure – I know that mine do.

Why not employ Python’s existing "test runner" capabilities to do the same?

Here’s a very short code snippet that can serve as a template to achieve this:

import unittest

class TestsContainer(unittest.TestCase):
    longMessage = True

def make_test_function(description, a, b):
    def test(self):
        self.assertEqual(a, b, description)
    return test

if __name__ == '__main__':
    testsmap = {
        'foo': [1, 1],
        'bar': [1, 2],
        'baz': [5, 5]}

    for name, params in testsmap.iteritems():
        test_func = make_test_function(name, params[0], params[1])
        setattr(TestsContainer, 'test_{0}'.format(name), test_func)

    unittest.main()

What happens here:

  1. The test class TestsContainer will contain dynamically generated test methods.
  2. make_test_function creates a test function (a method, to be precise) that compares its inputs. This is just a trivial template – it could do anything, or there can be multiple such "makers" fur multiple purposes.
  3. The loop creates test functions from the data description in testmap and attaches them to the test class.

Keep in mind that this is a very basic example. I hope it’s obvious that testmap could really be test files found on disk, or whatever else. The main idea here is the dynamic test method creation.

So what do we gain from this, you may ask? Quite a lot. unittest is powerful – armed to its teeth with useful tools for testing. You can now invoke tests from the command line, control verbosity, control "fast fail" behavior, easily filter which tests to run and which not to run, use all kinds of assertion methods for readability and reporting (why write your own smart list comparison assertions?). Moreover, you can build on top of any number of third-party tools for working with unittest results – HTML/XML reporting, logging, automatic CI integration, and so on. The possibilities are endless.

One interesting variation on this theme is aiming the dynamic generation at a different testing "layer". unittest defines any number of "test cases" (classes), each with any number of "tests" (methods). In the code above, we generate a bunch of tests into a single test case. Here’s a sample invocation to see this in action:

$ python dynamic_test_methods.py -v
test_bar (__main__.TestsContainer) ... FAIL
test_baz (__main__.TestsContainer) ... ok
test_foo (__main__.TestsContainer) ... ok

======================================================================
FAIL: test_bar (__main__.TestsContainer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "dynamic_test_methods.py", line 8, in test
    self.assertEqual(a, b, description)
AssertionError: 1 != 2 : bar

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1)

As you can see, all data pairs in testmap are translated into distinctly named test methods within the single test case TestsContainer.

Very easily, we can cut this a different way, by generating a whole test case for each data item:

import unittest

class DynamicClassBase(unittest.TestCase):
    longMessage = True

def make_test_function(description, a, b):
    def test(self):
        self.assertEqual(a, b, description)
    return test

if __name__ == '__main__':
    testsmap = {
        'foo': [1, 1],
        'bar': [1, 2],
        'baz': [5, 5]}

    for name, params in testsmap.iteritems():
        test_func = make_test_function(name, params[0], params[1])
        klassname = 'Test_{0}'.format(name)
        globals()[klassname] = type(klassname,
                                   (DynamicClassBase,),
                                   {'test_gen_{0}'.format(name): test_func})

    unittest.main()

Most of the code here remains the same. The difference is in the lines within the loop: now instead of dynamically creating test methods and attaching them to the test case, we create whole test cases – one per data item, with a single test method. All test cases derive from DynamicClassBase and hence from unittest.TestCase, so they will be auto-discovered by the unittest machinery. Now an execution will look like this:

$ python dynamic_test_classes.py -v
test_gen_bar (__main__.Test_bar) ... FAIL
test_gen_baz (__main__.Test_baz) ... ok
test_gen_foo (__main__.Test_foo) ... ok

======================================================================
FAIL: test_gen_bar (__main__.Test_bar)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "dynamic_test_classes.py", line 8, in test
    self.assertEqual(a, b, description)
AssertionError: 1 != 2 : bar

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (failures=1)

Why would you want to generate whole test cases dynamically rather than just single tests? It all depends on your specific needs, really. In general, test cases are better isolated and share less, than tests within one test case. Moreover, you may have a huge amount of tests and want to use tools that shard your tests for parallel execution – in this case you almost certainly need separate test cases.

I’ve used this technique in a number of projects over the past couple of years and found it very useful; more than once, I replaced a whole complex test runner program with about 20-30 lines of code using this technique, and gained access to many more capabilities for free.

Python’s built-in test discovery, reporting and running facilities are very powerful. Coupled with third-party tools they can be even more powerful. Leveraging all this power for any kind of testing, and not just unit testing, is possible with very little code, due to Python’s dynamism. I hope you find it useful too.

Summary of reading: January – March 2014

March 30th, 2014 at 4:03 pm
  • "Why We Get Fat, And What to Do About it" by Gary Taubes – Since I’ve been recently interested in nutrition again, Taubes naturally came up on the radar due to the huge popularity and high ratings of his books (at least on Amazon). I decided to start with this one as an introduction, and read the much longer and more comprehensive "Good Calories, Bad Calories" later if I this one went well. Yeah, it did. The book starts kind-of slow and repetitive and by the end of the first third I wished he would get to the point. But the middle third or so of the book is awesome. What’s most awesome about it is the no-nonsense science. At some point after reading a particularly interesting explanation (of how insulin affects fat intake and processing), I flipped open my copy of "Life: The Science of Biology" and read the relevant few pages. It matched exactly, and from that moment I was hooked. I’m not very easily convinced, but this book definitely made me reconsider some notions I wasn’t questioning before. The author’s points in favor of low-carb diets are definitely convincing, and his attempts to actually unravel and understand relevant research and trials is very commendable. I definitely plan to read "Good Calories, Bad Calories" now to learn more on the hypothesis Taubes presents.
  • "Juvenilia" by Miguel CanĂ© (read in Spanish) – an autobiographic account of the author’s years in the prestigious Colegio Nacional de Buenos Aires. Occasionally amusing, I presume this account is much more interesting to persons familiar with the school in question, or at least the situation in Argentina in the 1860s.
  • "Everything I Know" by Paul Jarvis – your typical motivational book, on how one should stop being afraid and JFDI. Can’t say I related to it too much. Maybe I should try reading it again some time.
  • "Good Calories, Bad Calories: Fats, Carbs, and the Controversial Science of Diet and Health" by Gary Taubes – The earlier, much more comprehensive and encyclopedic precursor to "Why We Get Fat" by the same author. This book provides the same conclusions, but spends significantly more ink on dissecting the published nutritional research of the past century. The author methodically addresses all the common "best practices" of high-carb, low-fat diet for healthy living. This deeper definitely look makes Taubes’s position appear even stronger. I really wish someone would be running the experiments suggested by the author in the epilogue – this definitely seems like an area that needs some proper, peer-reviewed scientific method-backed research.
  • "The Power of Habit" by Charles Duhigg – while the core ideas presented in the first third of the book are interesting, the rest of it felt like an empty filler full with anecdotes that, while well written, have little to do with the main theme of the book. Still, it’s an intriguing and enjoyable book to read; just don’t expect too much from it.

Re-reads:

  • "Masters of Doom" by David Kushner

pyelftools 0.22 released

March 30th, 2014 at 6:57 am

Version 0.22 of pyelftools was released today (on PyPI too). Some nice new features in this release – most notably initial support for DWARF v4, as well as better support for ARM objects (including 64-bit ARM).

By the way, I recently noticed that pyelftools appears in the open source credits page of Chrome (chrome://credits). I knew the Chrome and Chrome OS teams are using it for some of their tools & testing, but hadn’t bothered to check the credits page until recently. This is probably the most high-profile (at least publicly visible) user of pyelftools.

Rewriting the lexer benchmark in Go

March 27th, 2014 at 5:58 am

Last year I was toying with a simple lexer (for the TableGen language, because why not), implementing it using multiple approaches in both Python and Javascript. Redoing the same task using multiple approaches and using more than one language is a very interesting code kata and a great way to learn.

Since I’ve been recently looking at Go, I continued the exercise by reimplementing the lexer (the hand-written one, not a regex-based) in Go. The full code is available here (along with a large input file used for benchmarking).

Naturally, since my previous posts did performance comparisons between Python and Javascript, I wanted to add Go to the graph. I also had to rerun all the benchmarks because from the time of writing those posts I got a new, much faster, machine.

Anyway, here it is:

http://eli.thegreenplace.net/wp-content/uploads/2014/03/comparison-py-js-go.png

Since Python is so slow here, it’s hard to see the difference between the fastest versions, but the handwritten Go lexer is roughly on par with the Javascript one (33 vs. 31 msec). The benchmarks were run on my i7-4771 machine (amd64); go1.2.1, Node.js v0.10.26.

Now, this is quite literally the first non-trivial Go program I’ve written and I’m a neophyte by all measures, so any tips on the code would be very welcome. I tried to stay faithful to the Javascript implementation in terms of the algorithm, so the comparison would be fair.

That said, shortly after completing the code I started wondering if it could be made faster. There’s something about Go that makes you think about performance on a low level, not unlike when programming in C. Maybe it’s because so many things are explicit – pointers, slices, etc.

Anyhow, the code that uses a lexer to fill in a slice of tokens caught my eyes:

toks := []Token{}

startTime := time.Now()
for {
        nt := nl.NextToken()
        toks = append(toks, nt)
        if nt.Name == EOF {
                break
        }
}

That toks = append(toks, nt) in particular. As the size grows, toks will have to be reallocated and all its contents copied over. Since the input in my case had close to 200000 tokens and reallocation doubles the slice size, this means that in the order of 16 reallocations have to happen here, each time copying all the elements over. If that sounds like a lot of wasted work to you, that’s because it is.

So I tried replacing the first line with:

toks := make([]Token, 0, 200000)

And wow, the runtime dropped from 33 to 20 ms, making it 33% faster than the JS version. To be fair to JS I tried to perform the same optimization there (instead of pushing onto an array, create a large one in advance), but this has actually made things slower. Some sources online claim that V8 (which is what I’m running underneath, since my local code runs on Node.js) doesn’t like preallocating large arrays.

So as is often the case with benchmarks, it’s difficult to do an apples-to-apples comparison here. A hunch tells me that in a fully optimized (by a programmer skilled in the language) version of this benchmark, Go would still win, because its nature (typed, compiled to native code, and exposing a lot of low-level details) make it easier to reason about in terms of performance. But performance was not really the point here – I just wanted to see how easy it is to reimplement the same lexer in Go.

Hopefully the code would be useful/interesting to someone; please let me know what I could’ve done better.

Clearing the database with Django commands

February 20th, 2014 at 6:01 am

In a previous post, I presented a method of loading initial data into a Django database by using a custom management command. An accompanying task is cleaning the database up. Here I want to discuss a few options of doing that.

First, some general design notes on Django management commands. If you run manage.py help you’ll see a whole bunch of commands starting with sql. These all share a common idiom – print SQL statements to the standard output. Almost all DB engines have means to pipe commands from the standard input, so this plays great with the Unix philosophy of building pipes of single-task programs.

Django even provides a convenient shortcut for us to access the actual DB that’s being used with a given project – the dbshell command.

As an example, we have the sqlflush command, which returns a list of the SQL statements required to return all tables in the database to the state they were in just after they were installed. In a simple blog-like application with "post" and "tag" models, it may return something like:

$ python manage.py sqlflush
BEGIN;
DELETE FROM "auth_permission";
DELETE FROM "auth_group";
DELETE FROM "django_content_type";
DELETE FROM "django_session";
DELETE FROM "blogapp_tag";
DELETE FROM "auth_user_groups";
DELETE FROM "auth_group_permissions";
DELETE FROM "auth_user_user_permissions";
DELETE FROM "blogapp_post";
DELETE FROM "blogapp_post_tags";
DELETE FROM "auth_user";
DELETE FROM "django_admin_log";

COMMIT;

Note there’s a lot of tables here, because the project also installed the admin and auth applications from django.contrib.

We can actually execute these SQL statements, and thus wipe out all the DB tables in our database, by running:

$ python manage.py sqlflush | python manage.py dbshell

For this particular sequence, since it’s so useful, Django has a special built-in command named flush.

But there’s a problem with running flush that may or may not bother you, depending on what your goals are. It wipes out all tables, and this means authentication data as well. So if you’ve created a default admin user when jump-starting the application, you’ll have to re-create it now.

Perhaps there’s a more gentle way to delete just your app’s data, without messing with the other apps? Yes. In fact, I’m going to show a number of ways.

First, let’s see what other existing management commands have to offer. sqlclear will emit the commands needed to drop all tables in a given app. For example:

$ python manage.py sqlclear blogapp
BEGIN;
DROP TABLE "blogapp_tag";
DROP TABLE "blogapp_post";
DROP TABLE "blogapp_post_tags";

COMMIT;

So we can use it to target a specific app, rather than using the kill-all approach of flush. There’s a catch, though. While flush runs delete to wipe all data from the tables, sqlclear removes the actual tables. So in order to be able to work with the database, these tables have to be re-created. Worry not, there’s a command for that:

$ python manage.py sql blogapp
BEGIN;
CREATE TABLE "blogapp_post_tags" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "post_id" integer NOT NULL REFERENCES "blogapp_post" ("id"),
    "tag_id" varchar(50) NOT NULL REFERENCES "blogapp_tag" ("name"),
    UNIQUE ("post_id", "tag_id")
)
;
CREATE TABLE "blogapp_post" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
 <.......>
)
;
CREATE TABLE "blogapp_tag" (
 <.......>
)
;

COMMIT;

So here’s a first way to do a DB cleanup: pipe sqlclear appname into dbshell. Then pipe sql appname to dbshell.

An alternative way, which I like less, is to take the subset of DELETE statements generated by sqlflush, save them in a text file, and pipe it through to dbshell when needed. For example, for the blog app discussed above, these statements should do it:

BEGIN;
DELETE FROM "blogapp_tag";
DELETE FROM "blogapp_post";
DELETE FROM "blogapp_post_tags";
DELETE
COMMIT;

The reason I don’t like it is that it forces you to have explicit table names stored somewhere, which is a duplication of the existing models. If you happen to change some of your foreign keys, for example, tables will need changing so this file will have to be regenerated.

The approach I like best is more programmatic. Django’s model API is flexible and convenient, and we can just use it in a custom management command:

from django.core.management.base import BaseCommand
from blogapp.models import Post, Tag

class Command(BaseCommand):
    def handle(self, *args, **options):
        Tag.objects.all().delete()
        Post.objects.all().delete()

Save this code as blogapp/management/commands/clear_models.py, and now it can be invoked with:

$ python manage.py clear_models