Making code compatible with Python 2 and 3

May 19th, 2010 at 8:51 am

Update: Thanks for the great comments! To new readers of this post – make sure to skim the comments after you finish reading. There is some great advice there for making the change simpler – especially when you need to be compatible only with 2.6 and not the earlier versions (2.6 was especially designed to make future transition to 3K simpler).

Python 3 has been available for a long time already, but the migration of modules to it is going slower than many Python afficionados would have hoped. Once code is ported to Py3k, it cannot run on 2.x. This is the reason many library authors are afraid to make the step and port their code – they rightfully refuse to maintain two code bases. So we have a "lack of critical mass" problem.

In my opinion, to make the migration easier, it makes sense to write code that can run on both Python 2 and 3, at least for some time. Yes, this can make some parts of the code a bit ugly (although most of it can be hidden) but it will allow porting without actually having to maintain two code-bases. Once the critical mass assembles, the compatibility to 2.x can be dropped.

To contribute my share to the effort, I’ve successfully transformed two of my major code-bases to run on both Python 2.6 and 3.1:

  • pycparser – the ANSI C parser in pure Python: the new version (1.07) can run on both versions of Python (other than that, it isn’t different from 1.06)
  • Luz – the assembler/linker/CPU simulator suite has also been ported.

This porting was easier than I hoped. Since this is the first time I’ve touched Python 3, I had to use a few resources for help in the transition. Some of the best ones: Dive into Python, Mark Lutz’s site and Ned’s post

Here’s a list of some tricks I had to use, in no particular order. First and foremost, I created a file too encapsulate the differences as much as possible. Sometimes I had to use the following check:

if sys.hexversion > 0x03000000

To differentiate between Python versions. Luckily, all such checks could be confined to

Here’s an example of a couple of functions from

def printme(s):

def get_input(prompt):
    if sys.hexversion > 0x03000000:
        return input(prompt)
        return raw_input(prompt)

Python 3 made print into a function, so as a statement it doesn’t even parse. printme is a function which can be called by both versions of Python. It’s not as versatile as print itself, but it’s a small trouble since I mostly used print for debugging, testing and some trivial output.

get_input encapsulates the lack of raw_input in Python 3.

Another problem I commonly had to tackle is catching exceptions. Since the syntax was changed in Python 3, I had to resort to this for portability:

except TypeError:
    err = sys.exc_info()[1]

This code runs in both versions and places the exception message in err.

Some differences were very easy to handle. For example Python 3 removed xrange, so I’ve just used list(range. Had performance really mattered, I would have had to use something more complex. Also, itertools.imap was removed so I replaced it with iter(map. Dictionaries lost their has_key member, but key in dict works well on both versions of Python, so this is another easy change.

Luz is a relatively large project, sub-divided into packages and many modules, so relative vs. absolute imports gave me some trouble. Luckily, the 2.x version I wanted to be compatible with is 2.6, so I could just use relative imports everywhere and it works well on both versions.

The full-test running capabilities in Luz gave me some trouble because I’m using dynamic Python code loading there. The new module disappeared in Python 3, but happily imp.new_module replaces it and works in 2.6 as well. Also, I had to use a trick borrowed from Ned to replace exec with this monstrosity:

# Borrowed from Ned Batchelder
if sys.hexversion > 0x03000000:
    def exec_function(source, filename, global_map):
        exec(compile(source, filename, "exec"), global_map)
def exec_function(source, filename, global_map):
    exec compile(source, filename, "exec") in global_map
    "<exec_function>", "exec"))

Just like catching exceptions, since exec is syntax, you just can’t nicely hide it behind a version check. The parser chokes on it even if that code section doesn’t get executed eventually. Therefore, a brute-force approach using eval(compile is called for, since this one runs at runtime, when only the relevant interpreter sees it.

That’s about it. From now on I plan to keep both pycparser and Luz functional on both versions of Python – it shouldn’t be too hard. In the future when I feel the time is right to make the switch to Py3k, it will be trivial – I’ll just clean-up all the ugly portability code.

P.S.: To complete such a task you really need good unit tests. I can’t imagine making it and staying sane without the extensive tests both code-bases have.

Related posts:

  1. Creating Python extension modules in C
  2. Using GPL-ed code for in-house software
  3. Python internals: Working with Python ASTs
  4. pyelftools ported to Python 3
  5. Browsing Python source code with Vim

12 Responses to “Making code compatible with Python 2 and 3”

  1. ulrikNo Gravatar Says:

    Is the exec monstrosity really needed? I seem to be able to use exec like a function in Python 2.6 just fine –> exec("...", {})

  2. Roger PateNo Gravatar Says:

    Python 2.6 also supports the except Type as err: syntax.

    For functions which are a straight pass-through, why not something more like get_input = input if sys.version_info[0] >= 3 else raw_input? (I don’t like counting the zeros for hexversion, but that’s a small issue.)

  3. Henrik RavnNo Gravatar Says:

    A quick question. As far as I know

    from __future__ import print_function

    works in 2.6, so why did you go with having a printme function?

  4. AnonymousNo Gravatar Says:

    Instead of using your own print function you could have just imported it from __future__.

    from __future__ import print_function
  5. xtianNo Gravatar Says:

    Sounds great!

    However, I don’t understand what you’re saying about xrange and itertools.imap.

    xrange (which produced an iterator rather than a list) went away because Python 3′s range produces an iterator. So Py3 range should be a drop-in replacement for Py2 xrange – you don’t need to do list(range(...)) unless you were also doing list(xrange(...)).

    Similarly, Py3 map and Py2 itertools.imap do exactly the same thing – in Py3 iter(map(...)) is redundant.

    Anyway, hopefully posts like this will encourage more people to port to Python 3 – it’s definitely a cleaner version of the language.

  6. JohnNo Gravatar Says:
    from __future__ import print_function

    works for both 2.6 and 3.1.

    And, just in case, your printme() will probably want a newline output too.

  7. JohnNo Gravatar Says:

    Actually, except syntax between 2.6 is compatible too.

    I would have to presume you mean 2.5.

    Or I’ve misunderstood what you mean when you say:
    “… I’ve successfully transformed two of my major code-bases to run on both Python 2.6 and 3.1:”

  8. Casey DuncanNo Gravatar Says:

    Note that using 2to3 in can alleviate the need to version check and manually adjust syntax in many cases, with the slight disadvantages that your code is written against python 2.x and you must run your tests with python 3 from the build directory (which is easly done using the -w option of nosetests). Just add the following code to the top of your your to automagically run 2to3 against your code when building with python 3:

        from distutils.command.build_py import build_py_2to3 as build_py
    except ImportError:
        if sys.version_info >= (3, 0):
            raise ImportError("build_py_2to3 not found in distutils - it is required for Python 3.x")
        from distutils.command.build_py import build_py
        suffix = ""
        suffix = "-py3k"

    Then add the following argument to the setup() function there:

        # ... all your setup arguments ...
        cmdclass = {'build_py': build_py},
  9. voyagerNo Gravatar Says:

    On your Update: You can target as far as 2.5, using “from future import X”, on 2.6 and 3 you don’t need this. Using the future import, you bypass the syntax problems with exceptions, exec, eval and print.

  10. Martin VilcansNo Gravatar Says:

    You could just let get_input be an alias for the corresponding input function:

    if sys.hexversion > 0x03000000:
        get_input = input
        get_input = raw_input

    This would avoid the call-time version check.

  11. Sanjeev KumarNo Gravatar Says:

    very useful thread and the comments too.


    the above code outputs result in python 2.6
    with parenthesis
    like ('Hello', 3)

    but the same code above works in python 3 as well but prints without parenthesis

    like Hello 3

    how can i get the result in python 2 printed out to be the way i get in python 3

  12. KeishikoNo Gravatar Says:

    Hi! Kudos for this great post. Your link for Dive into Python is broken. Here’s the updated link:

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)