Motivation:

Imagine that you’ve written a script that uses several libraries, some of which you’ve written and some you’ve downloaded and installed (for example PyYAML). You want to distribute the script to your friends and co-workers, who already have Python installed with all the standard library. But your script won’t run on their machines, because they have neither your personal libraries, nor PyYAML installed. So what can you do ?

* You can ask them to install PyYAML and other libraries your script uses, and send them your own libraries. This is a lengthy and inconvenient process.
* You can use a tool like py2exe to package your delivery. This has a downside, however. py2exe produces large files (several MBs) and you may not want that.
* You can painstakingly collect the libraries into a directory where your script can find them, and package the directory together with the script.

LibCollect makes the third option trivial, by doing all the dirty work for you.

libcollect really fills a need for me, and I’m much happier about easy distribution of my scripts after I’ve written it. I post it online in the hope that other people will find it useful as well.

I hope others will find the page helpful as well. If you have comments, leave them here.

Trying distutils with Visual Studio 2005 Express

I have the free Visual Studio 2005 Express installed. I used it to successfully compile Python from sources, and wanted to tie it to distutils to compile the extension. Unfortunately, distutils is very strict about the MSVC version you compile extensions with, and complained:

running build
running build_ext
error: Python was built with Visual Studio 2003;
extensions must be built with a compiler than can generate compatible binaries.
Visual Studio 2003 was not found on this system. If you have Cygwin installed,
you can try compiling with MingW32, by passing "-c mingw32" to setup.py.

I consulted the Python IRC channel, and got a warm recommendation to use MinGW anyway, because many people has tested its compatibility with the Python distribution.

distutils with MinGW, first attempt

I’ve downloaded the latest version of MinGW, and tried to call distutils with the appropriate flag:

python setup build -c mingw32

But got this error¹:

running build
running build_ext
building 'hello' extension
creating build
creating buildtemp.win32-2.5
creating buildtemp.win32-2.5Release
d:mingwbingcc.exe -mno-cygwin -mdll -O -Wall -IC:Python25include -IC:Python25PC -c hellomodule.c -o buildtemp.win32-2.5Releasehellomodule.o
writing buildtemp.win32-2.5Releasehello.def
creating buildlib.win32-2.5
d:mingwbingcc.exe -mno-cygwin -shared -s buildtemp.win32-2.5Releasehellomodule.o buildtemp.win32-2.5Releasehello.def -LC:Python25libs -LC:Python25PCbuild -lpython25 -lmsvcr71 -o buildlib.win32-2.5hello.pyd
buildtemp.win32-2.5Releasehellomodule.o:hellomodule.c:(.text+0x3e): undefined reference to `_imp___Py_NoneStruct'
buildtemp.win32-2.5Releasehellomodule.o:hellomodule.c:(.text+0x46): undefined reference to `_imp___Py_NoneStruct'
collect2: ld returned 1 exit status
error: command 'gcc' failed with exit status 1

This got me stumped, because I could find nothing useful about this error in Google. However, I did get to this page, where the solution was found in the comments.

The solution

Apparently, the Python export lib (libs/python25.lib in the Python distribution) is not good enough for MinGW, and it wants a Unix-y .a file to list the exports of Python’s DLL (the one where the interpreter itself lives).

So you have to create libpython25.a file for the Python library. If you want, download the one I’ve created and skip all these steps. It should work with Python 2.5

To create libpython25.a on your own (taken from here with minor modifications):

1. Find python25.dll (it will probably be in windows/system32)
2. Now you have to list the exports from this DLL. Several tools can do this (including dumpbin), but pexports is the most convenient for this task. Get it from here or here (I’ll keep this link alive for a long time, promise).
3. Run pexports.exe python25.dll > python25.def
4. Now, if you’ve installed MinGW and added its bin directory to your path, you should be able to just call dlltool (a utility that comes with MinGW): dlltool --dllname python25.dll --def python 25.def --output-lib libpython25.a
5. This has created libpython25.a

Now just place libpython25.a in the libs directory of your Python distribution, and run distutils again²:

running build
running build_ext
building 'hello' extension
creating build
creating buildtemp.win32-2.5
creating buildtemp.win32-2.5Release
d:mingwbingcc.exe -mno-cygwin -mdll -O -Wall -IC:Python25include -IC:Python25PC -c hellomodule.c -o buildtemp.win32-2.5Releasehellomodule.o
writing buildtemp.win32-2.5Releasehello.def
creating buildlib.win32-2.5
d:mingwbingcc.exe -mno-cygwin -shared -s buildtemp.win32-2.5Releasehellomodule.o buildtemp.win32-2.5Releasehello.def -LC:Python25libs -LC:Python25PCbuild -lpython25 -lmsvcr71 -o buildlib.win32-2.5hello.pyd

Big success ! You can now use the extension from your Python code.

Updates

• It has been mentioned in a couple of places that with the latest MinGW the .a file is no longer needed and it’s enough to place the Python DLL in libs. I’m using the latest MinGW, ActiveState Python and distutils, and it didn’t work for me that way.

¹ I’m using a trivial sample extension called ‘hello’. Here’s its C file:

#include <Python.h>
 
static PyObject* say_hello(PyObject* self, PyObject* args)
{
    const char* name;
 
    if (!PyArg_ParseTuple(args, "s", &amp;name))
        return NULL;
 
    printf("Hello %s!\n", name);
 
    Py_RETURN_NONE;
}
 
static PyMethodDef HelloMethods[] =
{
     {"say_hello", say_hello, METH_VARARGS, "Greet somebody."},
     {NULL, NULL, 0, NULL}
};
 
PyMODINIT_FUNC
 
inithello(void)
{
     (void) Py_InitModule("hello", HelloMethods);
}

And here is the distutils setup file:

from distutils.core import setup, Extension
 
module1 = Extension('hello', sources = ['hellomodule.c'])
 
setup (name = 'PackageName',
        version = '1.0',
        description = 'This is a demo package',
        ext_modules = [module1])

² Note that you can run either setup.py build -c mingw32 or setup.py build --compiler=mingw32 - they are suggested in different places, but actually mean the same.

The exact steps I followed are:

1. Download the Python (2.5) source distribution from http://www.python.org/
2. Build Python from sources with MSVC 2005 Express, in both Debug and Release configurations. For this, go to the directory PCBuild in the source distribution and double click on the .sln file (Visual Studio will ask to convert it, since Python 2.5 was built on Windows with MSVC 2003). Select Debug and build solution. Then the same for Release.
3. Some libraries fail to build - this is normal. These are usually bindings to extra thing you have to install (Tk addons, bz compression, Berkley DB libs, etc). Python works fine without them.
4. Now, following the documentation instructions, I copied the \PC\example_nt directory, opened the solution contained in it, modified the Include and Linker Library directories path back to the source distribution, and built the solution.
5. example.pyd is created, and can be loaded and used by the Python interpreter.

Phew, this was much easier than I expected. It’s another chance to appreciate the work that has been into Python to make it more friendly for Windows users.

Update: You don’t even have to install the source distribution of Python to compile extensions. The binary distribution on Windows has the Include and libs/ directories that contain everything necessary to compile the extensions in Release mode. It is still recommended to have the source distribution handy, though, because it comes with a lot of examples and code from standard extensions to learn from, and you can use it to compile the extensions in Debug mode and run them with a Debug version of Python itself.

Every author has a unique signature, which you begin to recognize after reading a couple of his books. Uris is no different. In fact, he has several very strong signatures that come up time after time in every book. The three I’ve found to be most conspicuous are:

1. A strong affection for Jews and dislike of Arabs. This shows most firmly in the characters. Usually, the Jew characters are moral, gallant, fair and generally very positive.
2. A tendency to digress into long side-stories, usually telling about the backgrounds of the main characters.
3. American machoism, especially in relation to the prototypical “ex-Marine” (whom Uris, unsurprisingly, is by himself)

Well, while reading “Mitla Pass” I found that signature (1) is suppressed to the point of reversion, signature (2) is absurdly intensified, and signature (3) stayed the same. I’ll elaborate:

Somehow, in most of the background stories in this book, Uris chooses to present a lot of extremely negative Jewish characters. It appears that he did a good amount of research on all the negative stereotypes Jews have (such as weakness of character, promiscuity, parsimoniousness, nagging and complaining, the impossible nature of older women of east-European origin and so on), and did quite an effort to amplify it. I wonder at the reason for doing so. Is it that he was criticized for highlighting the positive stereotypes too much in earlier books ?

I have the most nits to pick with (anti-)signature (2). Uris just took it way too far with the amount of digressions in this book. Of the 500 pages, I don’t think it would be an exaggeration to say that about 10% are dedicated to the main plot, and the other 90% are spent in side-stories, mostly about the family of Gideon. Really, it appears like an un-edited version of the book. I was hoping to find interesting historical information about the life in Israel on the verge of the 1956 war in Sinai, and on the war itself, but the meager amount of text the author dedicated to the main plot made it almost trivial and completely devoid of substance.

I don’t want to conclude this review with a negative tone though, since I’ve liked the book, all things considered. Leon Uris definitely knows how to write, and his ability to intermix a page-turning plot with interesting historical background is awesome. While the vast amounts of digressions made the main plot useless, the side-stories themselves are very enjoyable (if you ignore the anti-Jewish stereotypes) and deal with several interesting periods of Jewish history in the late 19th and early 20th centuries.

So all in all, this book is recommended, although you’ll be disappointed if you look for a masterpiece like “Exodus”.

Unfortunately, I’ve miscalculated and misunderstood the topic of the book. I was hoping for a book on the history of the various tribes in the region, their unification into the Inca empire, Spanish conquistadors and their subjugation of the Incas. Instead, the book focuses on the years after the initial conquest - the complex politics and relations between the Spaniards and the local population and surviving members of the Inca royal family. Not much is told on the time before the landing of Pizzaro in Peru.

However, the book is well written and tells its intended story in a detailed and at the same time interesting manner. So if post-conquest Peru in its first few decades is your fancy, you’ll find this book excellent.

This is not to say that I didn’t like the book. It had enough interesting history of both post and pre-conquest to make me satisfied, even if I flipped through some parts of it when it got too detailed with the descriptions of each new successor on the Inca puppet-throne the Spaniards instated. It’s a real history book, textbook-class, so no matter how well it’s written, it’s difficult to digest in large chunks. So I took my time with it - nearly two months of on and off reading.

So I skimmed through the book for a couple of hours. I think that I’m better at organization than I was in the past, so most of the tips in the book were quite obvious. However, I still gleaned a few interesting insights I want to file here for future review:

• Tasks stay on your mind and make you worry mainly because they’re unclear. Replace vague topics by concrete goals that can be acted upon. For example, if a part of your todo list now looks like:
  • Foo
  • Bar

  Replace it by:

  • Talk to Joe about implementing Foo
  • Bar: read about super Widgets
  • Bar: implement a mega Widget for patch 12
  • Bar: do a code review
• Some relevant quotes: “This constant, unproductive preoccupation with all the things we have to do is the single largest consumer of time and energy”. and “It is easier to act yourself into a better way of feeling than to feel yourself into a better way of action”
• You should periodically sort and empty your master to-do list. Emptying it doesn’t necessarily mean doing everything on it. Rather, all the items on the list have to be assorted to more specific locations, like a calendar, per-project to-do lists and so on. Keeping a lot of stuff on the master list for long periods of time is counterproductive.
• Weekly reviews of your tasks and goals are very important. No matter how well we try to organize ourselves, work often gets too hectic to follow a clearly defined plan all the time. A weekly review is a method to keep it all under control in the long run. Once a week, take some time to review all your pending tasks and to-do lists and set up more concrete goals for the week to come.

Anyway, as the book is quite short, I’ve finished it very quickly, and then a weird thing happened - I felt that I have to read it again. This has only happened to me once before, with Eugene Onegin. I did not read it again immediately this time though, but rather 2-3 books later, although I was certain that I should reread it and left it on the “unread” shelf.

“After the quake” is a set of 6 short stories that are very loosely related by the fact that they all mention the catastrophic earthquake in the city of Kobe in 1995 (well, if you leave out the obvious connection of all the plots happening in Japan, of course). I would say that the main theme of the stories is that all the characters contemplate the basic questions of human existence and worth with this traumatic event in the background. Although the characters are very different (from a salesman who travels to the north of Japan on a weird mission after his wife has left him to a successful physician who spends a relaxing vacation in Thailand), there is definitely a binding line between their contemplations.

I’ve only read one book of Murakami before (Norwegian Wood) and found it to be charming and thoughtful in a way I can’t express precisely. I have the same feeling about this one. One thing I’m sure of is that I definitely liked it.

Yes, I’ve finished reading “One hundred years of solitude” in Spanish. I began this endeavor months ago (somewhere in mid 2007 I reckon), and today it was completed. Why did it take so long ? For a couple of reasons, I guess. First of all, I read most of it with a dictionary in hand, which made the pace excruciatingly slow (10 pages per hour or so). Second, the book is far from having a fluid plot that’s easy to follow, so consuming it in large quantities can be very difficult.

Anyway, now that it’s done, I’m proud of having finished it. I read the book before (6-7 years ago), in English. Even though my English was pretty good back then, the book was very difficult, as it used a lot of high language. I recall wondering how reading such a book in its native Spanish would feel - perhaps this is what triggered my desire to learn Spanish, though I’m not certain about this.

CAS (acronym of the Spanish name), as it is frequently referred to, is considered one of the most notable works of Latin literature, widely acclaimed as second only to Don Quixote. Márquez received the Nobel prize in literature for his works, of which CAS is the principal. It tells the story of the Buendía family, set in the 19th and 20th centuries in the small village Macondo (which was founded by the head of the family) in Colombia. The story deals with seven generations of the Buendías, most of which have the same name, which is a frequent criticism of the novel.

Naming all the family males either Aureliano or José Arcadio (or some variation thereof) is a deliberate trick by Márquez, to drill in the main theme of the book - which is the repetitiveness of time and history. Needless to say, it gets quite confusing at times, and if your book doesn’t have a helpful family tree depicted in the beginning, consult an online resource. Otherwise, it’s too easy to get lost (unless you’re reading the whole book in a single sitting - but I wonder who could do that…)

Another recurring theme in the book is surrealism. Insomnia that hit a whole town for weeks, rain that lasts years, yellow butterflies, increased fecundity of farm animals, colored ants and small fish made of gold are only some of the “wacky” fantasies CAS is full of. These too, are part of the theme of eternal recurrence of patterns in the lives of the Buendías.

And it’s impossible to write a review of this book without mentioning solitude and loneliness, that are perhaps the main feeling and state the author tries to transmit to his readers. How you’ll understand it depends on you, and I believe that everyone can see something for himself in this theme.

Finally, I want to provide two quotes from the book I liked and that describe the book well. They’re in Spanish, and I don’t have enough literary pizzazz to translate them to English in a convincing way.

From the conversation of Aureliano Babilonia with Pillar Ternera close to the end:

No había ningún misterio en el corazón de un Buendía que fuera impenetrable para ella, porque un siglo de naipes y de experiencias le había enseñado que la historia de la familia era un engranaje de repeticiones irreparables, una rueda giratoria que hubiera seguido dando vueltas hasta la eternidad, de no haber sido por el desgaste progresivo e irremediable del eje.

And the last sentence in the book:

Sin embargo, antes de llegar al verso final, ya había comprendido que no saldría jamás de ese cuarto, pues estaba previsto que la ciudad de los espejos (o espejismos) sería arrasada por el viento y desterrada de la memoria de los hombres, en el instante en que Aureliano Babilonia acabara de decifrar los pergaminos, y que todo lo escrito en ellos era irrepetible desde siempre y para siempre, porque los estirpes condenados a cien años de soledad no tenían una segunda oportunidad sobre la tierra.

I’m asking myself if I like the book. I don’t have the answer. There are things about it I really dislike, and some things I like beyond my own understanding. After reading it once, I took it up again, and can’t be sure that I won’t read it once again sometime in the future. It’s still a mystery for me.

Update: A few words on my programming background. I’m most proficient in C and Perl, with C++ coming closely behind. Additionally, I’ve programmed in Ruby, Lisp, Scheme, Matlab and Ada. More on this here and here.

About 3 weeks ago, I’ve decided to give Python a try. During this time I have been intensively learning and using Python, so now I’m ready to give a list of my initial impressions.

First, a word on my learning process. Python is a mature and useful language, so from the start I decided not only to play around, but to take it quite seriously. So, apart from reading the official Python tutorial and most of the free online book Dive into Python I spent a lot of time actually using Python, both for writing test scripts that display various features I might need and useful programs, both at work and home. Here’s a rough list of the code I’ve written to get familiar with Python, in no particular order:

• Accessing Windows DLLs from Python code
• GUIs with wxPython
• Tools for working with files, paths and directories
• Code involving threads and sockets
• Various scripts to test how Python finds its modules and packages
• Used py2exe to package scripts as self-contained .exe files

I’ve also written a couple of non-toy applications that actually do something useful:

• wxPyTris – a Tetris clone
• A simple GUI that can work as a XML-RPC client or (threaded) server
• A netlist / pinlist comparison utility useful for my work. I had it previously written in Perl, but now I’ve written a more feature-complete version in Python with a wxPython GUI

In total, I think I’ve written at least 3K lines of Python code and read a lot of documentation, newsgroup posts and tutorials. I’m still far from being proficient in Python, but I can now use it quite effectively.

My Python impressions

The impressions are divided to positive and negative, and inside each section there’s no particular order.

What I like about Python

• Significant indentation is actually a pretty good thing. Since I always pedantically indent my code anyway, I don’t find it to be an impediment. On the contrary – it has some very desirable features: much less punctuation to type (braces, begin/end, etc.), and code written by others is usually easier to understand, because Python forces it to be structurally cleaner.
• In a way, Python is openly supported by a large company that uses it extensively – Google. This means that there are a few smart people who get paid to develop and improve Python. For example Guido Von Rossum, the Python creator, works on Python in Google. I heard that the “release manager” for Python 2.6 also works for Google. This is important: while enthusiasts can create great tools in their free time, my experience tells me that to really perfect a tool, consistently, it is important for some people to earn their living from working on it.
• The Python community doesn’t treat Windows as a Nth-priority platform. All modules I’ve seen so far come with easy installers for Windows that just work. This is very important for me, as 95% of my work is being done on Windows.
• The documentation is very good
• PyCrust is a very helpful tool
• wxPython – very mature and easy to use. The wxPython demo is amazing. Everything works very quickly, with beautiful native GUIs. wxPython is so good that most open-source Python IDEs seem to be written in pure Python + wxPython
• Convenient built-in named arguments
• Docstrings
• The in operator
• List comprehensions
• Chained comparisons in expressions: (1 < x < 4) instead of (x > 1 and x < 4). Remember that this feature really shines when you replace x by self.server.count_connections()
• Batteries included – the Python standard library has a lot in it, and this is a very good thing, especially when you want to quickly distribute your scripts to coworkers who have no idea about Python.
• The way the print function works is quite convenient – it adds spaces between its arguments and a newline at the end, and thus saves boilerplate typing when debugging. And let’s face it, print in Python is used mainly for debugging anyway (files are written with the write() method of file objects).
• Python has native threads !! Although some restrictions on multi-CPU concurrency are imposed by the GIL, I understand that this can be solved in most cases. A nice feature of native threads is that waiting for a system call in one thread doesn’t block the other threads (which happens with Ruby’s green threads).
• Excellent support for loading C/C++ DLLs. The standard ctypes module is very convenient and easy to use.

What I don’t like about Python

• Python doesn’t have a large index of modules like Perl’s CPAN. There’s PyPi, but it appears to be both less complete and less universally used than CPAN.
• len() is a function and not a method of string. In general, Python is less consistent in this respect than, say, Ruby. Some operations are functions (len, dir), some built-in statements (del) and some object methods (reverse, append, etc.)
• string’s join method is OK, but why not also add a join method to a list. It is more elegant to write [1, 2, 3].join(',') than to write ', '.join[1, 2, 3]
• The syntax for a single-item tuple is a parsing-imposed ugliness, akin to the need to separate the closing ‘>’s in C++ nested templates.
• Lack of the ability to use ? and ! in identifiers. I came to like the Ruby way of naming predicates and destructive functions, and with Python I’m back to the C++/Perl dilemmas of isEmpty vs. empty vs. emptyp, etc. instead of the elegant empty?
• The unless keyword is sorely missing
• “Batteries included” also has a downside – there are a lot of clutter and deprecated libraries. While it’s good for keeping backwards compatibility, for new users this is a bit inconvenient, because you’re not always sure what to use.
• I’m not particularly fond of the way private methods are called in Python. while I don’t have a particular problem with methods receiving self and having to use it explicitly (it allows for some surprising flexibility, especially when passing around methods as callable objects), the syntax for private methods is too much on the fingers. Suppose your class has a private method called Foo. In C++, you’d call Foo() from another method in this class. In Python, you have to bang self.__Foo(). A bit too much. And while we’re at it, I don’t like those double underscor

Conclusion

All considered, I really like Python. The cons are mainly nits that are easy to overcome, while the pros are significant. I’ll continue practicing Python, and will write a more organized review later on.