Rant about Common Lisp and implementations

August 20th, 2007 at 5:51 am

In time I became very frustrated with my educational path for industrial strength Common Lisp. Things should not be as hard as this to install a few libraries that are probably in very common use (relatively). Maybe I’m dense, maybe I’m impatient, or maybe I’m just using the wrong implementation (CLISP on Windows), but I found this experience very painful.

Common Lisp has a comprehensive standard, true. But a standard is not enough. You also want a free implementation that conforms to the standard. Most modern programming languages like Perl and Ruby have one “base” implementation which everyone uses. Is there really a need for more ? I don’t think so (think about Java, C/C++ (gcc), C#, Python, it’s the same story…). The base Perl implementation is quite enough to serve millions of happy Perlists writing useful applications. Sure, Ruby now has forking implementations for the JVM and CLR, but they all try to fully port the base implementation in its entirety, and not just the core language.

Having a “full standard” to which many implementations conform is nice, but do everyday programmers really need it ? I’m not sure at all. Take Common Lisp for instance. That it has many popular implementations makes things worse, in my very humble opinion. Take a typical newbie post to comp.lang.lisp – they’re all very much alike. Newbies are always asking which implementation to use – which is better, what are the differences, etc. Although it isn’t easy for old-timers to relate, a multitude of implementations is confusing. Especially when you pick not the most popular and supported one (which seems to be CmuCL / SBCL on Linux for Common Lisp) – you’re up for a lot of uphill struggle.

I have a theory that what drives programming languages into the spotlight is the mass of newbies, not the few elite hackers who can recite selected parts of the implementation’s source code in their sleep. A language can be average, but once it has a lot of support, it breaks into public awareness and gradually improves to become pretty good.

Take the small and modest Newlisp, for example. It is a dialect of Lisp which is, admittedly, quite inferior to CL and Scheme. However, it has a single well-supported implementation, a lot of useful libraries right out of the box and a vibrant and supportive user community which makes it much more fun for newbies than Common Lisp. Sure, CL hackers walk with their nose pulled high in pride of how CL is vastly superior to Newlisp as a language for “serious programs”. But most of these newbies will take a couple of years to reach the level of those “serious programs” anyway, and some may never reach it. And in the meantime, they will use Newlisp, have fun and brag about it to their friends, while CL hackers will look with envy. Another recent example is the AutoHotKey language I wrote about earlier. As I said in that post, the programming language driving AHK can make any programming purist nauseous. It is that bad, true. However, it is usable, and the language installs easily, has tons of libraries and users who submit a lot of useful scripts and libraries. In the meantime, its responsive author is slowly but surely improving the language. All you need is the initial push, the dynamic resistance is much lower than the static resistance, as we all know from physics.

This is not to say that languages should be weak and all that matters is the implementation, not at all. It is to say that Common Lisp is damn unattractive for new programmers. Probably the least attractive of the popular languages. To make it more attractive, some changes need to be done, especially making it more newbie-friendly. I think that the litmus test for newbie friendliness is something like this:

  1. You can easily download and install a free implementation of the language, preferably for Windows, where the vast majority of computer users is (and hence the newbies and the clients).
  2. With a simple copy-paste of code, you can pop-up some rudimentary GUI, right out of the box.
  3. If you want a library that does *whatever*, you can easily download it from some central repository and install it, without the need to compile anything with your own compiler.
  4. Have any question answered in a well-written documentation and set of Wikis, FAQs and forums with helpful users.

At the moment, Common Lisp fails to pass this test. I think that while this is the situation, CL will fail to become more popular and will stay a curiosity. Some CL users say it’s for their own good, but I doubt that. Even inferior languages like Newlisp pass this test, not to mention Ruby, Python and even Scheme (for example PLT Scheme).

The most successful products in this world are not necessarily the best ones. It is the best marketed ones. I think this is one of those cases.

Related posts:

  1. newlisp – an intriguing dialect of Lisp
  2. Common Lisp vs. Scheme macros
  3. Book review: “Successful Lisp” by David Lamkins
  4. SEO for Lisp’s sake (Lisp tutorial)
  5. The sad state of the Lisp user community

18 Responses to “Rant about Common Lisp and implementations”

  1. LievenNo Gravatar Says:

    I think you’re fallen victim of the simple fact that few open source programmers use Windows. On my debian system I have 164 Common Lisp libraries prepackaged and a build/install system that compiles them automagically for each Common Lisp implementation I have installed. If I add an implementation later, all libraries on the system will be compiled for it at installation time.

  2. fooNo Gravatar Says:

    Windows is dead. Some might not know it. But it is now mostly uninteresting.

    Ruby has one popular implementation. Fine.

    * that one popular implementation is a hundred times slower than some CL implementations.

    * Common Lisp has not been designed to compete with ‘scripting languages’. It has been designed for developing complex applications. Common Lisp has commercial implementations which can run very demanding applications (CAD systems, Computer Algebra Systems, completely different programming languages, large databases, …). These implementations also make it possible to develop and debug those applications with small teams.

    * Common Lisp is much too complicated for many programmers. Let them use simpler languages like Ruby and they are happy. There is no need to dumb down Common Lisp. Ruby/Python is perfectly fine for many apps. Not everybody needs the power and flexibility of Common Lisp. If they want a simpler Lisp, one can also look at one of the nicer Scheme implementations. Common Lisp is a very useful tool for advanced programmers or for programmers which grow with the language they learn.

    * The different implementations are being developed for different scenarios. CLISP is an open source Lisp. It tries to be easily portable (due to an implementation in C). Corman Lisp is specifically designed to be an Lisp for Windows. LispWorks wants to be a cross-platform workstation-grade Lisp.
    There is some competition between those implementations. You can see it also that way: Common Lisp has, say, ten very different implementations which share a base language. Python, Ruby, Rebol are very similar languages with very similar implementations (relatively simple interpreter) which share nothing (no code, no infrastructure, no libs, …). I really prefer to have the choice on the Common Lisp side. At least there is reuse.

  3. elibenNo Gravatar Says:

    Windows isn’t dead, far from it. This is what many people fail to understand and remain stranded. Windows is gaining better and better development tools, and much more serious work is being done on it than before. For each home user switching to Ubuntu, there are two offices dumping their Solaris workstations for Windows.

    Besides, Windows is where most of the new users, the kids, and the curious would-be programmers are. This makes it a very strong target for programming languages and applications that want to increase their share of the market.

  4. m i c h a e lNo Gravatar Says:

    I don’t find newLISP to be at all inferior. I’ve used newlisp-wiki to make a website. I used newLISP-GS to write a font comparison program. I do all of my scripting work using newLISP. In fact, I’m writing this very response in newLISP edit!

    I spent over a decade working through various programming languages, trying to find one that felt just right. I stayed with newLISP.

    m i c h a e l

  5. cormullionNo Gravatar Says:

    Hey Eli,is this ‘inferior’ ‘Newlisp’ of which you speak any relation with the cool and useful little Lisp-like scripting language called ‘newLISP’ which makes my text editors obey my every command like so many muggles in a Harry Potter story? Or are you just careless with words generally… :-)

    That said (hopefully in a gently sarcastic vein which you won’t grumble about too much), I enjoyed this article and there’s some interesting arguments. I for one found it very hard to get a simple basic CL running easily on my machine.

  6. LievenNo Gravatar Says:

    I didn’t want to drag this into a Windows versus the rest of the world discussion, which I find quite boring. It’s just that not much work is being done on free Common Lisp implementations for Windows. Have you tried Lispworks or Allegro Common Lisp for Windows?

  7. Richard CookNo Gravatar Says:

    SBCL on Linux is “where the action is”. There are adequate versions for Windows and Mac OS X, but I have trouble with asdf-install on Windows and neither the Windows or OS X has multithreading.

    I’d say the most painless way to SBCL on Linux is get Knoppix on a CD-ROM and a small external USB hard disk. Boot from Knoppix and set up a “permanent home partition” on the external hard disk, then from now on boot with the “knoppix home=…” options. You can do more than just a home directory, you can install software into /bin, change files in /etc and the knoppix “union” file system merges your changes with the Knoppix on the CD-ROM. If you have the RAM you can also run some Linuxes in partitions with QEmu or the like.

    The above is theoretical for me – I have a Mac, and what I do is have a Parallels VM set up with a small “hard disk”, 5 gig, and boot from a Knoppix .ISO file in Parallels. This keeps the overall file sizes small, and upgrades to the next Knoppix will just be a matter of pointing to a different .ISO file.

  8. wkempfNo Gravatar Says:

    C/C++ don’t have a “base” implementation. GCC is just one of many implementations, and not necessarily even the most popular/used, especially if you’re talking about for any given platform. For example, it’s most definately not the most popular/used implementation on Windows.

  9. elibenNo Gravatar Says:

    I did not mean to bash Newlisp, not at all. I wrote quite favorably about it in this post and before. But its core language *is* inferior to Common Lisp and Scheme, there’s no doubt in that. Especially some key things like lexical binding and by-reference parameter passing which aren’t available “natively” but only through concepts.

    See also my post and the comments on it here: http://eli.thegreenplace.net/2006/04/20/newlisp-an-intriguing-dialect-of-lisp/

  10. m i c h a e lNo Gravatar Says:

    Hi Eli,

    Sorry if it seemed like you were being accused of bashing newLISP. Based on this and your earlier post, I can see you have been quite complimentary of the language. Unfortunately, the pejorative connotations of the word ‘inferior’ conjure up images of an ‘elite’ that degrades others by referring to them as ‘inferior.’ Not, I think, what you intended. Who knows, maybe we newLISPers just have an inferiority complex ;-)

    m i c h a e l

  11. MarkNo Gravatar Says:

    You made some excellent points here.

    It drives me crazy when people say things like “Windows is dead.” Get your head out of your ass. I’m no Windows-lover, but this is simply ignoring reality. LOTS of people use Windows and you can bet that the popularity of a tool/framework/language can only be helped by making it available on Windows.

    I agree that there should be an easy installation/setup for Windows that allows newbies to do something productive with CL. As some people have pointed out, several of the Linux distributions have used some form of apt-get to automate the installation of add-on libraries to CL installations under Linux. That’s great, but there has got to be a way that will work on other systems. I’ve tried using asdf to install libraries on CL installations and it has failed enough times (with some obscure error message) to make the process less-than-pleasant.

    It’s almost like there is a “rite of passage” for people who want to use CL. I can only imagine the number of people who give up quickly because it can been too hard to get started with CL and produce something useful.

  12. Jon HarropNo Gravatar Says:

    If you’re using Windows then I would strongly recommend using native Windows software and, in particular, .NET. Microsoft’s F# programming language is a modern functional programming language neatly integrated into their flagship platform, for example.

  13. JeffNo Gravatar Says:

    newLisp has reference passing. You can pass by symbol and evaluate the symbol or you can wrap the data in a context functor. Rather than:

    (set 'foo '(large list))
    (apply bar foo)

    …you instead use the context and functor (a context with an internal symbol of the same name, such as context foo, internal symbol foo):

    (set 'foo:foo '(large list))
    (apply bar foo)

    I agree that a language should have a base implementation from which fork other, more use-specific implementations. While I personally wouldn’t use Windows, a lot of people do. Any language that does not aim to be truly cross-platform, without a lot of work on the part of the user, is not going to gain a lot of adherents.

    CL suffers from academia in this respect; there are too many affluent common lispers with the attitude that if it were easy then anyone, even undesirable people, could use the language (including, god forbid, people who aren’t computer scientists).

    Scheme is slightly better in that respect in that it has a series of add-on standards that each distribution can implement. Each implementation typically lists which parts of the language it has so that it is much easier to find a language to use.

    Scheme’s big win over CL is it’s documentation. That is the real key for new users of a language. Better online documentation is make-or-break for many users. Starting with CL is very difficult because of very sparse documentation, which then doesn’t apply uniformally. Scheme has very nice documentation, and each language tends to have useful docs which highlight what it does and does not implement as well as extensions to the standard.

    newLisp has some of the best documentation of any Lisp I’ve found. It has a listing of every function with example code. Contrast that with CL docs, which typically spend more time explaining changes to the function throughout the years than how to actually use a function.

  14. Robert GoldmanNo Gravatar Says:

    First, I must agree with wkempf — you have extrapolated from a few, relatively new languages to a generalization about what is normal. It is not normal to have a single implementation for a language. Yes, that’s true of the modern scripting languages — perl. ruby, and python — and of the proprietary languages — C#, Visual Basic, and (somewhat) Java. But it most certainly is not true of the most popular and well-established programming languages: C++, C, Fortran, COBOL, Pascal, Common Lisp, Scheme, etc.

    A second point that seems very odd is your proposed requirement that “If you want a library that does *whatever*, you can easily download it from some central repository and install it, without the need to compile anything with your own compiler.[my emphasis]” With all due respect, this seems to be totally missing the point of dynamic languages like Common Lisp, Scheme, etc. A huge advantage of these languages is that you always have the compiler there to help you. Why on earth would you want a library distributed as object code, if you could get the source? You are never going to use CL without the compiler. That’s the beauty of it.

    For that matter, perl has been exquisitely successful with a huge volume of source-form libraries through CPAN. Yes, you can ignore the fact that you are getting source (but you can ignore that with ASDF-INSTALL, as well), but you are nevertheless….

    I think what you really want is an environment that makes it easy for you to not worry about the fact that you’re getting source libraries. That environment, as others have pointed out, is Linux with SBCL, because that’s the place where the overwhelming preponderance of the CL libraries are developed and tested. There just aren’t enough users out there to test them on some of the others.

    I understand that you would rather be able to use Windows, but that’s the price you pay for demanding open source, free software. You get the software that people want to write, for the platforms those people enjoy. The people who write Lisp libraries overwhelmingly prefer to code on Unix variants like Linux and MacOS.

    If you want a well-supported Common Lisp for Windows, feel free to pay some money to LispWorks or Franz. Both of them offer excellent products for that platform. They each offer free trial versions of their software for you to test out, or for you to use as tools for learning the language. These software packages will install smoothly, come with substantial support libraries (not portable, though), have IDEs, and will Just Work. The more I think about it, the more I think you should just download one of these and take it for a spin, instead of raising your blood pressure.

    If you want free as in freedom, you are going to have to either bend to the preferences of the people who write the free code, or you will have to port the stuff yourself.

  15. Daniel WeinrebNo Gravatar Says:

    I’ve written extensively elsewhere, so I’ll keep this short. Check out survey to learn about the implementations and get plenty of links to textbooks, Wiki’s, libraries, and so on. Check out this blog posting for my feelings about Common Lisp’s general drawbacks. Yes, the situation with free implementations for Windows is not as good as the situation for non-free for Windows, or free for non-Windows. Yes, the library situation needs a lot of improvement, which may actually happen; there are people thinking about it and trying to pull together resources to fix it.

  16. Daniel WeinrebNo Gravatar Says:

    There are a lot of good points above. As one of the Common Lisp designers, I try hard not to be a partisan. I’ve learned over the years that being a language bigot is a very bad idea. Based on what I’ve read above, I want to learn about Newlisp.

    Documentation: The open source implementors don’t seem to put a lot of attention into documentation. I’d like to see what LispWorks and Allegro provide, though. As for the language itself (as opposed to any one implementation), “Practical Common Lisp” is a great textbook. However, the ANSI standard, even the very improved HyperSpec from Kent Pitman, is pretty bad. (Personally, I liked “The Lisp Machine Manual” a lot, but I wrote half of it so I’m not totally unbiased.) A new version of “Common Lisp: The Langauge” that was updated to the X3J13 standard would be great. More online documentation would be nice, too. There are some good Emacs commands around that access the HyperSpec (I think this is all in Slime) for those of you who like Emacs, so that’s better than nothing.

    Many implementations: As was pointed out, this is the norm for older languages. Even Python has several implementations (Jython, IronPython), for good reasons. The Lisp implementations vary in many ways; see my survey paper at

    http://common-lisp.net/~dlw/LispSurvey.html

    And I didn’t do any benchmarking there. For example, some implementations have highly optimized bignums (arbitrary precision integer arithmetic) and others don’t; only some people care about that.

    Windows: More than half (six of eleven) of the current implementations run on Windows. Some of the others are working on it. Corman Common Lisp is quite interesting in that it ONLY runs on Windows, and has many useful features to fit into the Microsoft ecosystem, for those who are interested in that. I think all the others were originally for Linux/Unix and then were ported to Windows. Again, LispWorks and Allegro may do a great job; I’ve never used them, but these guys are real pro’s in a for-money business, so they have serious motivation as well as high-quality hackers.

    Libraries: Yes, there should be a centralized source a la CPAN. Marco Baringer and the Clozure guys have finally announced CLornucopia, which is just getting started. I think it will be the answer to this very important requirement. It’s not ready to be used yet (it’s in alpha and lots of projected cool features haven’t been done yet), but it’s on its way.

  17. tesloidNo Gravatar Says:

    I know I’m late to the party here, but I’d just like to say I feel your pain and am interested if you feel any differently about CL now, a couple years later?

    Generally, I try to learn a new language every year or two, and find that the best way (the only way for me) is to dive in and actually use it for general utility apps (scripts if you will) that come up on a weekly basis.

    I have to say, Lisp was not as hard as I imagined when I first laid eyes on some code 20 years ago. I really like it… love it in fact. I cannot give it up now. And the fact that its used as a tool extension language in my day job makes it even more useful for me to continue. But it sucks that I can’t write some code, put a shebang at the top, and give it to a co-worker and say “try this”. In my opinion, Guile/Scheme demonstrates the desired modern implementation of lisp, but sadly isn’t as deep.

    Any nobody (old timers and lisp wizards) that’s knowledgeable enough to help bring CL to the masses cares to. They’ll just tell me to embrace the REPL, and install asdf…

  18. elibenNo Gravatar Says:

    @tesloid,

    I’ve found Python a year ago and have been happy with it ever since, as you can read in the relevant blog posts. I haven’t really missed Lisp since then, and didn’t look at new implementations. I don’t think I’ll want to use Lisp for my hobbies and work anyway, as I prefer Python. But I’ve learned a lot from Lisp and its philosophy, and still feel it was a worthy experience to try and master it.