Compiling Python extensions with distutils and MinGW

June 28th, 2008 at 8:32 am

Yesterday I wrote about compiling Python 2.5 extensions using MS Visual Studio. However, I also wanted to make it work using distutils (as described here), which appeared to be the simplest method.

Trying distutils with Visual Studio 2005 Express

I have the free Visual Studio 2005 Express installed. I used it to successfully compile Python from sources, and wanted to tie it to distutils to compile the extension. Unfortunately, distutils is very strict about the MSVC version you compile extensions with, and complained:

running build
running build_ext
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

I consulted the Python IRC channel, and got a warm recommendation to use MinGW anyway, because many people has tested its compatibility with the Python distribution.

distutils with MinGW, first attempt

I’ve downloaded the latest version of MinGW, and tried to call distutils with the appropriate flag:

python setup build -c mingw32

But got this error1:

running build
running build_ext
building 'hello' extension
creating build
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
d:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python25\include -IC:\Python25\PC -c hellomodule.c -o build\temp.win32-2.5\Release\hellomodule.o
writing build\temp.win32-2.5\Release\hello.def
creating build\lib.win32-2.5
d:\mingw\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.5\Release\hellomodule.o build\temp.win32-2.5\Release\hello.def -LC:\Python25\libs -LC:\Python25\PCbuild -lpython25 -lmsvcr71 -o build\lib.win32-2.5\hello.pyd
build\temp.win32-2.5\Release\hellomodule.o:hellomodule.c:(.text+0x3e): undefined reference to _imp___Py_NoneStruct'
build\temp.win32-2.5\Release\hellomodule.o:hellomodule.c:(.text+0x46): undefined reference to
_imp___Py_NoneStruct' collect2: ld returned 1 exit status error: command 'gcc' failed with exit status 1

This got me stumped, because I could find nothing useful about this error in Google. However, I did get to this page, where the solution was found in the comments.

The solution

Apparently, the Python export lib (libs/python25.lib in the Python distribution) is not good enough for MinGW, and it wants a Unix-y .a file to list the exports of Python’s DLL (the one where the interpreter itself lives).

So you have to create libpython25.a file for the Python library. If you want, download the one I’ve created and skip all these steps. It should work with Python 2.5. (Update 25.11.2009: here is the lib for Python 2.6)

To create libpython25.a on your own (taken from here with minor modifications):

  1. Find python25.dll (it will probably be in windows/system32)
  2. Now you have to list the exports from this DLL. Several tools can do this (including dumpbin), but pexports is the most convenient for this task. Get it from here or here (I’ll keep this link alive for a long time, promise).
  3. Run pexports.exe python25.dll > python25.def
  4. Now, if you’ve installed MinGW and added its bin directory to your path, you should be able to just call dlltool (a utility that comes with MinGW): dlltool --dllname python25.dll --def python 25.def --output-lib libpython25.a
  5. This has created libpython25.a

Now just place libpython25.a in the libs directory of your Python distribution, and run distutils again2:

running build
running build_ext
building 'hello' extension
creating build
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
d:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python25\include -IC:\Python25\PC -c hellomodule.c -o build\temp.win32-2.5\Release\hellomodule.o
writing build\temp.win32-2.5\Release\hello.def
creating build\lib.win32-2.5
d:\mingw\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.5\Release\hellomodule.o build\temp.win32-2.5\Release\hello.def -LC:\Python25\libs -LC:\Python25\PCbuild -lpython25 -lmsvcr71 -o build\lib.win32-2.5\hello.pyd

Big success ! You can now use the extension from your Python code.

Updates

  • It has been mentioned in a couple of places that with the latest MinGW the .a file is no longer needed and it’s enough to place the Python DLL in libs. I’m using the latest MinGW, ActiveState Python and distutils, and it didn’t work for me that way.

1 I’m using a trivial sample extension called ‘hello’. Here’s its C file:

#include 
 
static PyObject* say_hello(PyObject* self, PyObject* args)
{
    const char* name;
 
    if (!PyArg_ParseTuple(args, "s",&name))
        return NULL;
 
    printf("Hello %s!\n", name);
 
    Py_RETURN_NONE;
}
 
static PyMethodDef HelloMethods[] =
{
     {"say_hello", say_hello, METH_VARARGS, "Greet somebody."},
     {NULL, NULL, 0, NULL}
};
 
PyMODINIT_FUNC
 
inithello(void)
{
     (void) Py_InitModule("hello", HelloMethods);
}

And here is the distutils setup file:

from distutils.core import setup, Extension
 
module1 = Extension('hello', sources = ['hellomodule.c'])
 
setup (name = 'PackageName',
        version = '1.0',
        description = 'This is a demo package',
        ext_modules = [module1])

2 Note that you can run either setup.py build -c mingw32 or setup.py build --compiler=mingw32 – they are suggested in different places, but actually mean the same.

Related posts:

  1. Creating Python extension modules in C
  2. Book review: “Win32 Perl Programming: The Standard Extensions” by Dave Roth
  3. Compiling C DLLs and using them from Perl
  4. Compiling SQLite on Windows
  5. On spaces in the paths of programs and files on Windows

15 Responses to “Compiling Python extensions with distutils and MinGW”

  1. BasilNo Gravatar Says:

    This is a very cool and useful example, thanks for posting it!

    I followed your tutorial and it worked perfectly (only had to clean up the ampersand HTML from your C code).

    I will just mention that the binary win32 distribution of Python 2.5 ships with the ‘libpython25.a’ file, so users such as myself can skip the pexports step.

    The reason I am here is that I am looking for a solution to building the mcrypt library (libmcrypt) on win32 so that I can then build its extension, python-mcrypt. I am stuck in several ways:

    1. Problem compiling libmcrypt in VS 2003 because the __declspec(dllexport) issue causes a bunch of syntax errors during compile.

    2. Using distutils with “-c mingw32″, I get the error “initializer element is not constant”

    3. Using Cygwin to build libmcrypt works perfectly in Cygwin; unfortunately, I cannot for the life of me figure out how to carry over whatever is built there to the Windows side so I can built python-mcrypt.

    Do you have any clues?

    Thanks,
    Basil

  2. elibenNo Gravatar Says:

    Hi Basil,

    Could it be a problem with VS’s C/C++ standard compatibility ? Although that probably doesn’t explain the problem with mingw32. However, ‘initializer element is not constant’ should be simple enough to diagnose.

    I’m not really an expert on these matters, so you might consider asking in the distutils mailing list.

    Eli

  3. BasilNo Gravatar Says:

    Hi Eli,

    It is possibly one of the few million compiler/linker/project options in VS, but I am not an expert in that either.. or quite possibly my best guess at the project’s composition of files is wrong.

    Both the VS problem and the mingw32 problem have answers floating around the net, though nothing definitive; but I’ll keep looking. After all, I only stumbled on your page after two days of shooting in the Google dark :)

    The other excellent resource is frank’s “Building Python extensions for Windows with only free tools” that you link to. I also have Cygwin. The libmcrypt library compiles successfully only with default (i.e., Cygwin-dependent) settings during ./configure. I had no success in using frank’s recipe with the “-mno-cygwin” switches; but then again, I know very very little about the *POSIX environment.

    I’ll keep looking.
    Keep up the good work!
    Basil

  4. BasilNo Gravatar Says:

    I figured it out! If interested, please see this page.

  5. elibenNo Gravatar Says:

    Basil,
    I’m glad you’ve found the solution. I’ve actually linked to this page in the post itself.

  6. AnishNo Gravatar Says:

    Hi,,thank’s for lot’s of information
    but i’m having one problem again..i have created .pyd file using gcc from mingw for one sample example but when called from python as
    >>import sample
    >>print sample.cal(2,3)
    it’s not showing anyvalue or error, I used SWIG for creating wrap and .o file.

  7. JacoboNo Gravatar Says:

    Thanks for the guide! Helped alot.

  8. VishwajeetNo Gravatar Says:

    you saved my day thank you so much.

  9. TejovathiNo Gravatar Says:

    HI All,

    I am trying to do the same thing i.e I have a C++ DLL and C++ header file. Now I am trying to build a SWIG extension so that I can use it in Python. Now, I create a .i file and passed it to swig. I successfully got the .cxx file. I passes this file as source in distutils. But when I run setup.py, I am getting a undefined refence error. How to solve this? Please reply asap

  10. JanaNo Gravatar Says:

    Hey,

    How do you configure the compiler to turn off warnings in the described setup? I have Windows XP, Python 2.6 and Mingw. The problem is that the compiler stops when it encounters a warning… I know I have to put a -w argument but can’t figure out where…

    running build_ext
    error: don’t know how to compile C/C++ code on platform ‘nt’ with ‘mingw32
    ompiler

    C:\Stuff\PhD\Westpac_work_package\py-editdist-0.3>python setup.py install
    running install
    running build
    running build_ext
    building ‘editdist’ extension
    creating build
    creating build\temp.win32-2.6
    creating build\temp.win32-2.6\Release
    C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python26\include -IC:\
    n26\PC -c editdist.c -o build\temp.win32-2.6\Release\editdist.o
    editdist.c:37:21: warning: type defaults to ‘int’ in declaration of ‘u_int8
    editdist.c:37:30: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token
    editdist.c: In function ‘editdist_distance’:
    editdist.c:102:2: warning: implicit declaration of function ‘edit_distance’
    error: command ‘gcc’ failed with exit status 1

  11. elibenNo Gravatar Says:

    Jana,

    Seems to me you have an error there, which is why the compiler fails. Look at the notice on editdist.c:37:30

  12. ddwNo Gravatar Says:

    Just a short but sincere THANK YOU – I too spent 2 days in a fog trying to include openSSL into ActiveState python25 with minGW so I could deploy an app on Google App Engine with an https logon.

    It took some finding, but your excellent pre-built libpython25.a (and this blog) finally did the trick.

    In my view, this entry should be included and credited on the Google App Engine site itself… many thanks again.

  13. AaronNo Gravatar Says:

    I thought you might find this link handy:
    http://www.develer.com/oss/GccWinBinaries

    This fellow went and created a version of MinGW GCC specifically taylored for building Python extension modules.

  14. JoshuaNo Gravatar Says:

    Dude, you rock!

  15. ojdoNo Gravatar Says:

    Thank you very much for sharing this. Thanks to pexports.exe I managed to build pyglpk and pymprog under ActiveState Python 2.7.