Bootstrapping virtualenv

The packaging situation in Python is "imperfect" for a good reason - packaging is simply a very difficult problem to solve (see the amount of effort poured into Linux distribution package management for reference). One of the core issues is that project X may require version V of library L, and when you come to install project Y it may refuse to work with that version and require a newer one, with which project X can't work. So you're in an impasse.

The solution many Python programmers and projects have adopted is to use virtualenv. If you haven't heard about virtualenv, you're missing out - go read about it now.

I'm not going to write a tutorial about virtualenv or extoll its virtues here - enough bits have been spilled about this on the net already. What I plan to do is share an interesting problem I ran into and the solution I settled on.

I had to install some packages (Sphinx and related tools) on a new machine into a virtualenv. But the machine only had a basic Python installation, without setuptools or distribute, and without virtualenv. These aren't hard to install, but I wondered if there's an easy way to avoid installing anything. Turns out there is.

The idea is to create a "bootstrap" virtual environment that would have all the required tools to create additional virtual environments. It turns out to be quite easy with the following script (inspired by the answer in this SO discussion):

import sys
import subprocess

VENV_VERSION = '1.9.1'
PYPI_VENV_BASE = 'http://pypi.python.org/packages/source/v/virtualenv'
PYTHON = 'python2'
INITIAL_ENV = 'py-env0'

def shellcmd(cmd, echo=True):
    """ Run 'cmd' in the shell and return its standard out.
    """
    if echo: print '[cmd] {0}'.format(cmd)
    out = subprocess.check_output(cmd, stderr=sys.stderr, shell=True)
    if echo: print out
    return out

dirname = 'virtualenv-' + VENV_VERSION
tgz_file = dirname + '.tar.gz'

# Fetch virtualenv from PyPI
venv_url = PYPI_VENV_BASE + '/' + tgz_file
shellcmd('curl -O {0}'.format(venv_url))

# Untar
shellcmd('tar xzf {0}'.format(tgz_file))

# Create the initial env
shellcmd('{0} {1}/virtualenv.py {2}'.format(PYTHON, dirname, INITIAL_ENV))

# Install the virtualenv package itself into the initial env
shellcmd('{0}/bin/pip install {1}'.format(INITIAL_ENV, tgz_file))

# Cleanup
shellcmd('rm -rf {0} {1}'.format(dirname, tgz_file))

The script downloads and unpacks a recent virtualenv (substitute your desired version in VENV_VERSION) from PyPI and uses it directly (without installing) to create a new virtual env. By default, virtualenv will install setuptools and pip into this environment. Then, the script also installs virtualenv into the same environment. This is the bootstrap part.

Voila! py-env0 (or whatever you substituted in INITIAL_ENV) is now a self-contained virtual environment with all the tools you need to create new environments and install stuff into them.

This script is for Python 2 but can be trivially adapted for Python 3. In Python 3, the situation is actually more interesting. Python 3.3 (which is really the one you ought to be using if you've switched to 3 already) comes with virtualenv in the standard library (venv package), so downloading and installing it is not required.

That said, its virtualenv will not install setuptools and pip into the environments it creates. So YMMV here: if you need setuptools and pip there, go with a variation of the script above. If not, you don't need anything special really, just use the python3.3 -m venv.

P.S. The packaging situation is getting better though. There was a lot of focus during the recent PyCon on this. One of the interesting announcements was that distribute is merging back into setuptools.