A year with Python

May 15th, 2009 at 11:25 am

Almost exactly a year ago I wrote here about giving Python a try as my main programming language. Before that I was mainly writing in Perl, and eying Ruby from time to time as the replacement.

Well, it will come as no surprise to the readers of this blog that Python won big-time. In the past year I’ve posted more than 30 posts in the Python category of this blog, and it’s clear than Python has indeed become my main programming language. In this post I want to summarize what I’ve done with Python over the last year, and my general thoughts about the language.

General impression

My thoughts on Python are not much different from the short summary I posted just a few weeks after starting with the language. If anything, the small deficiencies "grew on me" – I got used to the couple little quirks in the syntax, and found out that the other things are not so bad.

For example, although Python doesn’t have CPAN, I haven’t yet encountered a need for which no Python library has been written. In this aspect the last year, at least in my view, has shown a trend towards Python from other languages. Google and other companies are releasing Python-only APIs, for instance. And I think it’s hard to find another language with two such great bindings to complete GUI frameworks like PyQt and wxPython.

I also lamented the clutter in Python’s standard library. Well, that is being fixed with Python 2.6 and especially 3.0, which will become mainstream in a few months.

Regarding Python’s pluses, the significant whitespace turned out to be a much bigger one than I’ve ever expected. Over the last year I’ve read the code of many libraries, and almost always found it quite readable and simple to follow. Python’s PEP-8 is a good set of rules that’s followed to some extent by virtually everyone in the community, which adds a lot. Company-wide coding standards exist for a reason, and PEP-8 is a community-wide coding standard.

Moreover, I’ve taught Python to a few junior engineers and interns at work – people with little programming experience. I was amazed at how easy their code was to understand. I’ve had the "pleasure" to read Perl code written by programming newbies, and it’s a painful experience I would never want to go through again. Python is different.

Python-mania

Python is taking over practically all my programming. Over the past year, it has completely replaced Perl and Ruby. It has also somewhat shoved C and C++ out of the picture, because Python has such great bindings to existing C code (ctypes), and also because it has great libraries for practically everything.

But the most surprising effect of Python on me is that it’s slowly but surely pushing Lisp out of my mind. Back when I was programming in Perl, I kept coming back to Lisp from time to time, trying to make the shift, not succeeding and going back. I guess I just felt unsatisfied with what I had.

With Python, this is just gone. I’m fully satisfied with this language. I’ve very glad that I learned the functional programming paradigms from Lisp, and I use them daily in Python. But overall, Python is simply a better language. Moreover, it has a much larger ecosystem – libraries, development community, bindings, and support from large companies.

Getting things done

I did a rough count of the amount of Python code I’ve written this year. I count commented, tested lines of code, meaning that comments and unit tests are included in the count. The result is roughly 40 KLOC, which isn’t bad at all.

Here’s a partial list of stuff I’ve implemented with Python so far. Some of it at work, some at home:

  • A couple of simple web-applications for my own use. One using plain CGI, and the other using the TurboGears web framework.
  • pycparser – a complete parser for ANSI C90.
  • Parts of a VHDL parser. This is still work in progress.
  • A converter between binary data description formats, such as Intel HEX.
  • Several versions of a recursive-descent parser for a simple language
  • Many scripts for Scite
  • At work I’ve developed a framework for using Python for verification of VHDL models. Python is used to write the test scenarios and generate VHDL testbenches from them.
  • A few applications for controlling embedded hardware via the serial port (with elaborate wxPython & PyQt GUIs)
  • A few applications for analyzing test reports and building summaries in a user-friendly format
  • A complex multi-threaded web-page download and analysis program for my own use
  • A couple of games: a Tetris clone with wxPython and a small game with Pygame as part of a tutorial
  • Analysis of numeric data with matplotlib
  • Parts of an assembler and linker for a synthetic language
  • Solved dozens of Project Euler problems, all in Python.

Most of these projects were covered at least to some extent in this blog, and can be found by searching. If something interests you and you can’t find it in the blog search box, drop me an email.

Libraries

Finally, I want mention a few Python libraries I’ve found especially good and/or helpful in my work with Python this past year. In no particular order:

  • docutils & reST (reStructuredText): a great collection of libraries and utilities for formatting text
  • matplotlib: Matlab-like plotting
  • PLY: a Python implementation of Lex & Yacc. It’s very intuitive to use, powerful and useful for complex parsing tasks.
  • py2exe: a wonderful .exe packer that works with all major Python libraries and produces fast-loading executables
  • PyQt: a great binding to the powerful Qt framework – my current choice for GUI programming in Python
  • wxPython: another good GUI-binding
  • PyYAML: a simple-to-use YAML lib. I wish it were part of the standard Python library.
  • pyserial: the best scripting-language binding to the serial (COM) port I’ve seen anywhere
  • BeautifulSoup: a bit cranky but overall an excellent way to retrieve data from HTML pages
  • ctypes: calling C DLLs has never been so easy!

To the creators of these libraries – well done! May others learn from you on creating useful, high-quality and well documented code.

Related posts:

  1. New year’s Python meme 2011
  2. Book review: “Rapid GUI Programming with Python and Qt” by Mark Summerfield
  3. Python
  4. A “live” data monitor with Python, PyQt and PySerial
  5. matplotlib with PyQt GUIs

15 Responses to “A year with Python”

  1. Alexandr CiorniiNo Gravatar Says:

    Did you used such things in Perl as Perl::Critic, perltidy, Moose, autobox?

  2. rgzNo Gravatar Says:

    Python has the Python Package Index (pypi) instead of CPAN. Packages in the PYPI can be downloaded and installed in a single step with easy_install.

  3. Harald Armin MassaNo Gravatar Says:

    Nice summary! with all your parsing needs maybe you should have a look at pyparsing.

    http://pyparsing.wikispaces.com/

    It may or may not give you new insights; and even if you do not like it you will learn some ways to do things (or why you should not take them)

    best wishes Harald

  4. Chris GreggNo Gravatar Says:

    Hi there — thanks for the post! I was literally just today starting a search for a Python-based C parser, and you’ve got it all worked out! Cheers! Now I just have to figure out how to extend the grammar to parse CUDA…

  5. Vagif VerdiNo Gravatar Says:

    I’m glad to see someone ditching overhyped Ruby in favor of Python.
    Could you please share your experience in that regard. Why do you not miss Ruby ?

  6. DavidNo Gravatar Says:

    I’m in the process of starting my Python journey.

    I’ve tried out Ruby/Rails. I like the language – lots of syntactic sugar. But, there seems to be a lot of places that could use some maturing. (I’m not trying to dump on Ruby here)

    I’m hoping that python is more dependable and as fun to programming in. A lot of people that I respect love python, so I’ll give it a serious try.

  7. Grant RettkeNo Gravatar Says:

    Thanks for sharing that. You could have summed up your post very quickly if you had simply titled it “I love Python because it has tons of great libraries!” :) .

  8. John MairNo Gravatar Says:

    @vagif: Python and Ruby are both great languages, you don’t need to bash one to like the other, it’s a matter of personal preference. Get over it. :)

  9. Imran KhanNo Gravatar Says:

    When I started reading about Moose I was glad that I didn’t spend much time on Python. I know Python neat and easy to read and manage but I hope Perl 6 will change this.
    Neatness of a language will serve me no good If I can’t find a decent module to write reports in Excel. My manager will not look at the neat code, he need the report and there are deadlines.

  10. LeonidasNo Gravatar Says:

    @Imran Khan- Codes are not written for the managers. Someone has to read your code somehow for maintaining. If you just believe in making things working is your job then that’s the wrong approach towards programming.
    I personally always believe in this famous quote by Rick Osborne.

    “Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.”

  11. elibenNo Gravatar Says:

    @Vagif Verdi
    Ruby isn’t a bad language. It has some definite pluses over Python, but overall, IMHO, it loses. The strongest points in favor of Python over Ruby are: performance (Python’s much faster), less magic (Ruby’s monkey-patching), and much larger ecosystem (at least English speaking… Ruby is a hit in Japan) – more libraries, more forums/discussions, easier to get help and find code examples.

    Imran Khan
    There are a few Excel-writing Python libs online. I’m pretty sure you can easily find something no less powerful than your Perl library.
    And Moose is a hack, duct-taped on top of Perl. Few use it, few are familiar with it. Compare it to Python’s built-in object system, which is used universally and compatibly, including in all of Python’s standard library.

  12. Brian SmithNo Gravatar Says:

    I had to write a simple script to parse and evaluate a c enum, and your pycparser was super useful and easy to use. If you’re ever in that code again, you should consider making the parse function take a file-like object in addition to the filename (I needed to do my own preprocessing on the file before I sent it to the parse function). Either way though, great work.

  13. ryanNo Gravatar Says:

    I have been doing Python 2 years and feel the same way. It is almost the perfect ecosystem and culture for a programming language. I believe it is the next large market standard language.

    Ruby is great as well but I find Python has broader uses at least right now (plus I like Python syntax better what little differences their are mainly not having to put ‘end’). Python works great for web, gamedev, desktop, services, servers, parsers, pipeline enhangements, system admin and wraps just about every difficult library (take PyRakNet for view for reliable UDP as one). Another example is Jython, Boo and IronPython, all really productive and keeping CPython honest. If Google finishes the port to LLVM we could see Python being as fast as C, it is already three times faster than Ruby, PHP even without psyco. With psyco it is about 2-3 times slower than gcc.

    I think it will be huge and having the backing of MIT, Google, and almost all really smart engineers, it won’t be long now.

  14. elibenNo Gravatar Says:

    Brian
    You can always use the CParser object. Its parse function just takes a string and returns the AST. The parse_file function in __init__ is just an auxiliary for common tasks.

  15. markusNo Gravatar Says:

    > I’m glad to see someone ditching overhyped Ruby in favor of Python.

    The thing which makes me mad – and I am using ruby since 5 years – is when people talk that Ruby is overhyped, but they mean Rails.

    Seriously – the ruby community never hyped ruby. Ruby as a language is great.

    But once rails emerged, that whole field changed. It would be nice if people could FUCKING separate between the two. No, the communities DO NOT largely overlap. I know a great deal more ruby developers who DO NOT USE rails. At the same time, many DO USE rails.

    Oh well…

    Anyway, back to the point. Moving from perl away is always a good thing.

    It is called evolution. Whether this be python or ruby is not so important, but it would be interesting if you one day would move from python to ruby too AND blog about it (no matter the end result).

    PS: But please, core ruby. Use rails only _after_ you spent enough time with ruby. So many people use rails but have no appreciation or understanding for ruby itself… and seriously, if I look at code like Activerecord, it makes me sick….

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)