Automating boring testing activities with tox

May 7th, 2012 at 4:27 am

My pyelftools project has a comprehensive set of tests, which I make sure to run before every release. This turned out to be a somewhat tiresome manual task, because of the following factors:

  1. There are three sets of tests in pyelftools: unit tests, full system tests (comparing the output of my pyelftools-based readelf clone with the real readelf), and example tests (I hate out-of-date examples, so mine are self-testable).
  2. I currently promise to keep pyelftools working on Python versions 2.6, 2.7 and 3.2; the tests should be run on all three.
  3. I want to test that my setup.py script is alright too, so it would be preferable to build a source distribution, install it in a virtualenv and run the tests there. This also helps me test that the package is correctly pip-installable, and that no artifact of my local setup makes the tests pass by chance.

To a programmer, this list just screams "repetition", and hence some form of automation has to be conjured. Before you rush to roll your own, be sure the check out tox.

tox was designed to solve exactly the problem I described above. Its description is "virtualenv-based automation of test activities" – spot on.

All I have to do in order to automate the testing routine described above is install tox, create a configuration file for it, and make sure to execute tox -rv in the root of my project before every release. Done.

Here’s my tox.ini for pyelftools, in all its glory:

[tox]
envlist = py27,py26,py32

[testenv]
commands =
    python test/run_all_unittests.py
    python test/run_examples_test.py
    python test/run_readelf_tests.py

[testenv:py26]
deps =
    unittest2

Basically, tox runs all the commands listed in [testenv] for each "environment" listed in envlist. It creates a virtualenv for each such environment, which is exactly what I wanted.

Moreover, as you can see, tox helps to handle another problem I would have to deal with manually: I’m using the excellent unittest2 package for the pyelftools unit tests. While this package is in the standard library of Python 2.7 and 3.2+, it has to be installed separately in 2.6; tox handles this by providing a deps feature – I can specify the dependencies that need to be installed into the virtualenv for the given environment.

Related posts:

  1. Python unit testing: parametrized test cases
  2. pyelftools ported to Python 3
  3. pyelftools – Python library for parsing ELF and DWARF
  4. New year’s Python meme 2011
  5. unit testing framework – cxxtext

3 Responses to “Automating boring testing activities with tox”

  1. DidierNo Gravatar Says:

    Why not a continuous integration [1] tool such as buildbot [2] or Jenkins [3] ? It should achieve the same goal with the extra bonus of being distributed on several slaves.

    [1] http://en.wikipedia.org/wiki/Continuous_integration
    [2] http://buildbot.net/
    [3] http://jenkins-ci.org/

  2. elibenNo Gravatar Says:

    Didier,

    CI servers are great tools that any serious large project should adopt, but they’re just a bit of overkill in my case. Neither buildbot, nor Jenkins are trivial to configure. Even though someone with experience can probably set them up swiftly, it still doesn’t compare to the negligible time it takes to setup tox. Moreover, I don’t have multiple computers to act as build slaves, and setting VMs is too much work :-)

    I think that the solution can actually be layered. tox is probably a good candidate to run on each slave, handling the test execution itself, while the CI server handles the work distribution to slaves and reporting.

  3. Mahdi YusufNo Gravatar Says:

    I really enjoy your posts extremely informative with clear and concise thoughts! Keep up the great work!

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)