Eli Bendersky's website » Compiling Python extensions with distutils and MinGW

Compiling Python extensions with distutils and MinGW

June 28th, 2008 at 8:32 am

Yesterday I wrote about compiling Python 2.5 extensions using MS Visual Studio. However, I also wanted to make it work using distutils (as described here), which appeared to be the simplest method.

Trying distutils with Visual Studio 2005 Express

I have the free Visual Studio 2005 Express installed. I used it to successfully compile Python from sources, and wanted to tie it to distutils to compile the extension. Unfortunately, distutils is very strict about the MSVC version you compile extensions with, and complained:

running build
running build_ext
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

I consulted the Python IRC channel, and got a warm recommendation to use MinGW anyway, because many people has tested its compatibility with the Python distribution.

distutils with MinGW, first attempt

I’ve downloaded the latest version of MinGW, and tried to call distutils with the appropriate flag:

python setup build -c mingw32

But got this error¹:

running build
running build_ext
building 'hello' extension
creating build
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
d:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python25\include -IC:\Python25\PC -c hellomodule.c -o build\temp.win32-2.5\Release\hellomodule.o
writing build\temp.win32-2.5\Release\hello.def
creating build\lib.win32-2.5
d:\mingw\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.5\Release\hellomodule.o build\temp.win32-2.5\Release\hello.def -LC:\Python25\libs -LC:\Python25\PCbuild -lpython25 -lmsvcr71 -o build\lib.win32-2.5\hello.pyd
build\temp.win32-2.5\Release\hellomodule.o:hellomodule.c:(.text+0x3e): undefined reference to _imp___Py_NoneStruct'

build\temp.win32-2.5\Release\hellomodule.o:hellomodule.c:(.text+0x46): undefined reference to_imp___Py_NoneStruct'
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

This got me stumped, because I could find nothing useful about this error in Google. However, I did get to this page, where the solution was found in the comments.

The solution

Apparently, the Python export lib (libs/python25.lib in the Python distribution) is not good enough for MinGW, and it wants a Unix-y .a file to list the exports of Python’s DLL (the one where the interpreter itself lives).

So you have to create libpython25.a file for the Python library. If you want, download the one I’ve created and skip all these steps. It should work with Python 2.5. (Update 25.11.2009: here is the lib for Python 2.6)

To create libpython25.a on your own (taken from here with minor modifications):

Find python25.dll (it will probably be in windows/system32)
Now you have to list the exports from this DLL. Several tools can do this (including dumpbin), but pexports is the most convenient for this task. Get it from here or here (I’ll keep this link alive for a long time, promise).
Run pexports.exe python25.dll > python25.def
Now, if you’ve installed MinGW and added its bin directory to your path, you should be able to just call dlltool (a utility that comes with MinGW): dlltool --dllname python25.dll --def python 25.def --output-lib libpython25.a
This has created libpython25.a

Now just place libpython25.a in the libs directory of your Python distribution, and run distutils again²:

running build
running build_ext
building 'hello' extension
creating build
creating build\temp.win32-2.5
creating build\temp.win32-2.5\Release
d:\mingw\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python25\include -IC:\Python25\PC -c hellomodule.c -o build\temp.win32-2.5\Release\hellomodule.o
writing build\temp.win32-2.5\Release\hello.def
creating build\lib.win32-2.5
d:\mingw\bin\gcc.exe -mno-cygwin -shared -s build\temp.win32-2.5\Release\hellomodule.o build\temp.win32-2.5\Release\hello.def -LC:\Python25\libs -LC:\Python25\PCbuild -lpython25 -lmsvcr71 -o build\lib.win32-2.5\hello.pyd

Big success ! You can now use the extension from your Python code.

Updates

It has been mentioned in a couple of places that with the latest MinGW the .a file is no longer needed and it’s enough to place the Python DLL in libs. I’m using the latest MinGW, ActiveState Python and distutils, and it didn’t work for me that way.

¹ I’m using a trivial sample extension called ‘hello’. Here’s its C file:

#include 
 
static PyObject* say_hello(PyObject* self, PyObject* args)
{
    const char* name;
 
    if (!PyArg_ParseTuple(args, "s",&name))
        return NULL;
 
    printf("Hello %s!\n", name);
 
    Py_RETURN_NONE;
}
 
static PyMethodDef HelloMethods[] =
{
     {"say_hello", say_hello, METH_VARARGS, "Greet somebody."},
     {NULL, NULL, 0, NULL}
};
 
PyMODINIT_FUNC
 
inithello(void)
{
     (void) Py_InitModule("hello", HelloMethods);
}

And here is the distutils setup file:

from distutils.core import setup, Extension
 
module1 = Extension('hello', sources = ['hellomodule.c'])
 
setup (name = 'PackageName',
        version = '1.0',
        description = 'This is a demo package',
        ext_modules = [module1])

² Note that you can run either setup.py build -c mingw32 or setup.py build --compiler=mingw32 – they are suggested in different places, but actually mean the same.

This entry was posted on Saturday, June 28th, 2008 at 08:32 and is filed under Python. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

15 Responses to “Compiling Python extensions with distutils and MinGW”

Basil Says:
July 25th, 2008 at 07:31
This is a very cool and useful example, thanks for posting it!

I followed your tutorial and it worked perfectly (only had to clean up the ampersand HTML from your C code).

I will just mention that the binary win32 distribution of Python 2.5 ships with the ‘libpython25.a’ file, so users such as myself can skip the pexports step.

The reason I am here is that I am looking for a solution to building the mcrypt library (libmcrypt) on win32 so that I can then build its extension, python-mcrypt. I am stuck in several ways:

1. Problem compiling libmcrypt in VS 2003 because the __declspec(dllexport) issue causes a bunch of syntax errors during compile.

2. Using distutils with “-c mingw32″, I get the error “initializer element is not constant”

3. Using Cygwin to build libmcrypt works perfectly in Cygwin; unfortunately, I cannot for the life of me figure out how to carry over whatever is built there to the Windows side so I can built python-mcrypt.

Do you have any clues?

Thanks,
Basil
eliben Says:
July 25th, 2008 at 07:53
Hi Basil,

Could it be a problem with VS’s C/C++ standard compatibility ? Although that probably doesn’t explain the problem with mingw32. However, ‘initializer element is not constant’ should be simple enough to diagnose.

I’m not really an expert on these matters, so you might consider asking in the distutils mailing list.

Eli
Basil Says:
July 25th, 2008 at 15:15
Hi Eli,

It is possibly one of the few million compiler/linker/project options in VS, but I am not an expert in that either.. or quite possibly my best guess at the project’s composition of files is wrong.

Both the VS problem and the mingw32 problem have answers floating around the net, though nothing definitive; but I’ll keep looking. After all, I only stumbled on your page after two days of shooting in the Google dark

The other excellent resource is frank’s “Building Python extensions for Windows with only free tools” that you link to. I also have Cygwin. The libmcrypt library compiles successfully only with default (i.e., Cygwin-dependent) settings during ./configure. I had no success in using frank’s recipe with the “-mno-cygwin” switches; but then again, I know very very little about the *POSIX environment.

I’ll keep looking.
Keep up the good work!
Basil
Basil Says:
July 26th, 2008 at 19:44
I figured it out! If interested, please see this page.
eliben Says:
July 27th, 2008 at 21:52
Basil,
I’m glad you’ve found the solution. I’ve actually linked to this page in the post itself.
Anish Says:
August 5th, 2008 at 09:29
Hi,,thank’s for lot’s of information
but i’m having one problem again..i have created .pyd file using gcc from mingw for one sample example but when called from python as
>>import sample
>>print sample.cal(2,3)
it’s not showing anyvalue or error, I used SWIG for creating wrap and .o file.
Jacobo Says:
March 26th, 2009 at 17:03
Thanks for the guide! Helped alot.
Vishwajeet Says:
November 13th, 2009 at 11:41
you saved my day thank you so much.
Tejovathi Says:
January 19th, 2010 at 19:28
HI All,

I am trying to do the same thing i.e I have a C++ DLL and C++ header file. Now I am trying to build a SWIG extension so that I can use it in Python. Now, I create a .i file and passed it to swig. I successfully got the .cxx file. I passes this file as source in distutils. But when I run setup.py, I am getting a undefined refence error. How to solve this? Please reply asap
Jana Says:
February 22nd, 2011 at 07:49
Hey,

How do you configure the compiler to turn off warnings in the described setup? I have Windows XP, Python 2.6 and Mingw. The problem is that the compiler stops when it encounters a warning… I know I have to put a -w argument but can’t figure out where…

running build_ext
error: don’t know how to compile C/C++ code on platform ‘nt’ with ‘mingw32
ompiler

C:\Stuff\PhD\Westpac_work_package\py-editdist-0.3>python setup.py install
running install
running build
running build_ext
building ‘editdist’ extension
creating build
creating build\temp.win32-2.6
creating build\temp.win32-2.6\Release
C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python26\include -IC:\
n26\PC -c editdist.c -o build\temp.win32-2.6\Release\editdist.o
editdist.c:37:21: warning: type defaults to ‘int’ in declaration of ‘u_int8
editdist.c:37:30: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token
editdist.c: In function ‘editdist_distance’:
editdist.c:102:2: warning: implicit declaration of function ‘edit_distance’
error: command ‘gcc’ failed with exit status 1
eliben Says:
February 22nd, 2011 at 09:35
Jana,

Seems to me you have an error there, which is why the compiler fails. Look at the notice on editdist.c:37:30
ddw Says:
October 30th, 2011 at 12:49
Just a short but sincere THANK YOU – I too spent 2 days in a fog trying to include openSSL into ActiveState python25 with minGW so I could deploy an app on Google App Engine with an https logon.

It took some finding, but your excellent pre-built libpython25.a (and this blog) finally did the trick.

In my view, this entry should be included and credited on the Google App Engine site itself… many thanks again.
Aaron Says:
January 26th, 2012 at 22:33
I thought you might find this link handy:
http://www.develer.com/oss/GccWinBinaries

This fellow went and created a version of MinGW GCC specifically taylored for building Python extension modules.
Joshua Says:
April 11th, 2012 at 19:49
Dude, you rock!
ojdo Says:
July 23rd, 2012 at 08:16
Thank you very much for sharing this. Thanks to pexports.exe I managed to build pyglpk and pymprog under ActiveState Python 2.7.