Rant about Common Lisp and implementations
August 20th, 2007 at 5:51 amIn time I became very frustrated with my educational path for industrial strength Common Lisp. Things should not be as hard as this to install a few libraries that are probably in very common use (relatively). Maybe I’m dense, maybe I’m impatient, or maybe I’m just using the wrong implementation (CLISP on Windows), but I found this experience very painful.
Common Lisp has a comprehensive standard, true. But a standard is not enough. You also want a free implementation that conforms to the standard. Most modern programming languages like Perl and Ruby have one “base” implementation which everyone uses. Is there really a need for more ? I don’t think so (think about Java, C/C++ (gcc), C#, Python, it’s the same story…). The base Perl implementation is quite enough to serve millions of happy Perlists writing useful applications. Sure, Ruby now has forking implementations for the JVM and CLR, but they all try to fully port the base implementation in its entirety, and not just the core language.
Having a “full standard” to which many implementations conform is nice, but do everyday programmers really need it ? I’m not sure at all. Take Common Lisp for instance. That it has many popular implementations makes things worse, in my very humble opinion. Take a typical newbie post to comp.lang.lisp - they’re all very much alike. Newbies are always asking which implementation to use - which is better, what are the differences, etc. Although it isn’t easy for old-timers to relate, a multitude of implementations is confusing. Especially when you pick not the most popular and supported one (which seems to be CmuCL / SBCL on Linux for Common Lisp) - you’re up for a lot of uphill struggle.
I have a theory that what drives programming languages into the spotlight is the mass of newbies, not the few elite hackers who can recite selected parts of the implementation’s source code in their sleep. A language can be average, but once it has a lot of support, it breaks into public awareness and gradually improves to become pretty good.
Take the small and modest Newlisp, for example. It is a dialect of Lisp which is, admittedly, quite inferior to CL and Scheme. However, it has a single well-supported implementation, a lot of useful libraries right out of the box and a vibrant and supportive user community which makes it much more fun for newbies than Common Lisp. Sure, CL hackers walk with their nose pulled high in pride of how CL is vastly superior to Newlisp as a language for “serious programs”. But most of these newbies will take a couple of years to reach the level of those “serious programs” anyway, and some may never reach it. And in the meantime, they will use Newlisp, have fun and brag about it to their friends, while CL hackers will look with envy. Another recent example is the AutoHotKey language I wrote about earlier. As I said in that post, the programming language driving AHK can make any programming purist nauseous. It is that bad, true. However, it is usable, and the language installs easily, has tons of libraries and users who submit a lot of useful scripts and libraries. In the meantime, its responsive author is slowly but surely improving the language. All you need is the initial push, the dynamic resistance is much lower than the static resistance, as we all know from physics.
This is not to say that languages should be weak and all that matters is the implementation, not at all. It is to say that Common Lisp is damn unattractive for new programmers. Probably the least attractive of the popular languages. To make it more attractive, some changes need to be done, especially making it more newbie-friendly. I think that the litmus test for newbie friendliness is something like this:
- You can easily download and install a free implementation of the language, preferably for Windows, where the vast majority of computer users is (and hence the newbies and the clients).
- With a simple copy-paste of code, you can pop-up some rudimentary GUI, right out of the box.
- If you want a library that does *whatever*, you can easily download it from some central repository and install it, without the need to compile anything with your own compiler.
- Have any question answered in a well-written documentation and set of Wikis, FAQs and forums with helpful users.
At the moment, Common Lisp fails to pass this test. I think that while this is the situation, CL will fail to become more popular and will stay a curiosity. Some CL users say it’s for their own good, but I doubt that. Even inferior languages like Newlisp pass this test, not to mention Ruby, Python and even Scheme (for example PLT Scheme).
The most successful products in this world are not necessarily the best ones. It is the best marketed ones. I think this is one of those cases.
August 20th, 2007 at 9:15 pm
I think you’re fallen victim of the simple fact that few open source programmers use Windows. On my debian system I have 164 Common Lisp libraries prepackaged and a build/install system that compiles them automagically for each Common Lisp implementation I have installed. If I add an implementation later, all libraries on the system will be compiled for it at installation time.
August 20th, 2007 at 11:33 pm
Windows is dead. Some might not know it. But it is now mostly uninteresting.
Ruby has one popular implementation. Fine.
* that one popular implementation is a hundred times slower than some CL implementations.
* Common Lisp has not been designed to compete with ’scripting languages’. It has been designed for developing complex applications. Common Lisp has commercial implementations which can run very demanding applications (CAD systems, Computer Algebra Systems, completely different programming languages, large databases, …). These implementations also make it possible to develop and debug those applications with small teams.
* Common Lisp is much too complicated for many programmers. Let them use simpler languages like Ruby and they are happy. There is no need to dumb down Common Lisp. Ruby/Python is perfectly fine for many apps. Not everybody needs the power and flexibility of Common Lisp. If they want a simpler Lisp, one can also look at one of the nicer Scheme implementations. Common Lisp is a very useful tool for advanced programmers or for programmers which grow with the language they learn.
* The different implementations are being developed for different scenarios. CLISP is an open source Lisp. It tries to be easily portable (due to an implementation in C). Corman Lisp is specifically designed to be an Lisp for Windows. LispWorks wants to be a cross-platform workstation-grade Lisp.
There is some competition between those implementations. You can see it also that way: Common Lisp has, say, ten very different implementations which share a base language. Python, Ruby, Rebol are very similar languages with very similar implementations (relatively simple interpreter) which share nothing (no code, no infrastructure, no libs, …). I really prefer to have the choice on the Common Lisp side. At least there is reuse.
August 21st, 2007 at 6:17 am
Windows isn’t dead, far from it. This is what many people fail to understand and remain stranded. Windows is gaining better and better development tools, and much more serious work is being done on it than before. For each home user switching to Ubuntu, there are two offices dumping their Solaris workstations for Windows.
Besides, Windows is where most of the new users, the kids, and the curious would-be programmers are. This makes it a very strong target for programming languages and applications that want to increase their share of the market.
August 21st, 2007 at 6:21 am
I don’t find newLISP to be at all inferior. I’ve used newlisp-wiki to make a website. I used newLISP-GS to write a font comparison program. I do all of my scripting work using newLISP. In fact, I’m writing this very response in newLISP edit!
I spent over a decade working through various programming languages, trying to find one that felt just right. I stayed with newLISP.
m i c h a e l
August 21st, 2007 at 7:28 pm
Hey Eli,is this ‘inferior’ ‘Newlisp’ of which you speak any relation with the cool and useful little Lisp-like scripting language called ‘newLISP’ which makes my text editors obey my every command like so many muggles in a Harry Potter story? Or are you just careless with words generally…
That said (hopefully in a gently sarcastic vein which you won’t grumble about too much), I enjoyed this article and there’s some interesting arguments. I for one found it very hard to get a simple basic CL running easily on my machine.
August 21st, 2007 at 9:01 pm
I didn’t want to drag this into a Windows versus the rest of the world discussion, which I find quite boring. It’s just that not much work is being done on free Common Lisp implementations for Windows. Have you tried Lispworks or Allegro Common Lisp for Windows?
August 21st, 2007 at 10:47 pm
SBCL on Linux is “where the action is”. There are adequate versions for Windows and Mac OS X, but I have trouble with asdf-install on Windows and neither the Windows or OS X has multithreading.
I’d say the most painless way to SBCL on Linux is get Knoppix on a CD-ROM and a small external USB hard disk. Boot from Knoppix and set up a “permanent home partition” on the external hard disk, then from now on boot with the “knoppix home=…” options. You can do more than just a home directory, you can install software into /bin, change files in /etc and the knoppix “union” file system merges your changes with the Knoppix on the CD-ROM. If you have the RAM you can also run some Linuxes in partitions with QEmu or the like.
The above is theoretical for me - I have a Mac, and what I do is have a Parallels VM set up with a small “hard disk”, 5 gig, and boot from a Knoppix .ISO file in Parallels. This keeps the overall file sizes small, and upgrades to the next Knoppix will just be a matter of pointing to a different .ISO file.
August 21st, 2007 at 10:48 pm
C/C++ don’t have a “base” implementation. GCC is just one of many implementations, and not necessarily even the most popular/used, especially if you’re talking about for any given platform. For example, it’s most definately not the most popular/used implementation on Windows.
August 22nd, 2007 at 5:05 am
I did not mean to bash Newlisp, not at all. I wrote quite favorably about it in this post and before. But its core language *is* inferior to Common Lisp and Scheme, there’s no doubt in that. Especially some key things like lexical binding and by-reference parameter passing which aren’t available “natively” but only through concepts.
See also my post and the comments on it here: http://eli.thegreenplace.net/2006/04/20/newlisp-an-intriguing-dialect-of-lisp/
August 22nd, 2007 at 12:50 pm
Hi Eli,
Sorry if it seemed like you were being accused of bashing newLISP. Based on this and your earlier post, I can see you have been quite complimentary of the language. Unfortunately, the pejorative connotations of the word ‘inferior’ conjure up images of an ‘elite’ that degrades others by referring to them as ‘inferior.’ Not, I think, what you intended. Who knows, maybe we newLISPers just have an inferiority complex
m i c h a e l
August 22nd, 2007 at 9:27 pm
You made some excellent points here.
It drives me crazy when people say things like “Windows is dead.” Get your head out of your ass. I’m no Windows-lover, but this is simply ignoring reality. LOTS of people use Windows and you can bet that the popularity of a tool/framework/language can only be helped by making it available on Windows.
I agree that there should be an easy installation/setup for Windows that allows newbies to do something productive with CL. As some people have pointed out, several of the Linux distributions have used some form of apt-get to automate the installation of add-on libraries to CL installations under Linux. That’s great, but there has got to be a way that will work on other systems. I’ve tried using asdf to install libraries on CL installations and it has failed enough times (with some obscure error message) to make the process less-than-pleasant.
It’s almost like there is a “rite of passage” for people who want to use CL. I can only imagine the number of people who give up quickly because it can been too hard to get started with CL and produce something useful.
August 24th, 2007 at 10:18 am
If you’re using Windows then I would strongly recommend using native Windows software and, in particular, .NET. Microsoft’s F# programming language is a modern functional programming language neatly integrated into their flagship platform, for example.
August 27th, 2007 at 2:58 pm
newLisp has reference passing. You can pass by symbol and evaluate the symbol or you can wrap the data in a context functor. Rather than:
(set 'foo '(large list))(apply bar foo)
…you instead use the context and functor (a context with an internal symbol of the same name, such as context foo, internal symbol foo):
(set 'foo:foo '(large list))(apply bar foo)
I agree that a language should have a base implementation from which fork other, more use-specific implementations. While I personally wouldn’t use Windows, a lot of people do. Any language that does not aim to be truly cross-platform, without a lot of work on the part of the user, is not going to gain a lot of adherents.
CL suffers from academia in this respect; there are too many affluent common lispers with the attitude that if it were easy then anyone, even undesirable people, could use the language (including, god forbid, people who aren’t computer scientists).
Scheme is slightly better in that respect in that it has a series of add-on standards that each distribution can implement. Each implementation typically lists which parts of the language it has so that it is much easier to find a language to use.
Scheme’s big win over CL is it’s documentation. That is the real key for new users of a language. Better online documentation is make-or-break for many users. Starting with CL is very difficult because of very sparse documentation, which then doesn’t apply uniformally. Scheme has very nice documentation, and each language tends to have useful docs which highlight what it does and does not implement as well as extensions to the standard.
newLisp has some of the best documentation of any Lisp I’ve found. It has a listing of every function with example code. Contrast that with CL docs, which typically spend more time explaining changes to the function throughout the years than how to actually use a function.
October 26th, 2007 at 6:23 pm
First, I must agree with wkempf — you have extrapolated from a few, relatively new languages to a generalization about what is normal. It is not normal to have a single implementation for a language. Yes, that’s true of the modern scripting languages — perl. ruby, and python — and of the proprietary languages — C#, Visual Basic, and (somewhat) Java. But it most certainly is not true of the most popular and well-established programming languages: C++, C, Fortran, COBOL, Pascal, Common Lisp, Scheme, etc.
A second point that seems very odd is your proposed requirement that “If you want a library that does *whatever*, you can easily download it from some central repository and install it, without the need to compile anything with your own compiler.[my emphasis]” With all due respect, this seems to be totally missing the point of dynamic languages like Common Lisp, Scheme, etc. A huge advantage of these languages is that you always have the compiler there to help you. Why on earth would you want a library distributed as object code, if you could get the source? You are never going to use CL without the compiler. That’s the beauty of it.
For that matter, perl has been exquisitely successful with a huge volume of source-form libraries through CPAN. Yes, you can ignore the fact that you are getting source (but you can ignore that with ASDF-INSTALL, as well), but you are nevertheless….
I think what you really want is an environment that makes it easy for you to not worry about the fact that you’re getting source libraries. That environment, as others have pointed out, is Linux with SBCL, because that’s the place where the overwhelming preponderance of the CL libraries are developed and tested. There just aren’t enough users out there to test them on some of the others.
I understand that you would rather be able to use Windows, but that’s the price you pay for demanding open source, free software. You get the software that people want to write, for the platforms those people enjoy. The people who write Lisp libraries overwhelmingly prefer to code on Unix variants like Linux and MacOS.
If you want a well-supported Common Lisp for Windows, feel free to pay some money to LispWorks or Franz. Both of them offer excellent products for that platform. They each offer free trial versions of their software for you to test out, or for you to use as tools for learning the language. These software packages will install smoothly, come with substantial support libraries (not portable, though), have IDEs, and will Just Work. The more I think about it, the more I think you should just download one of these and take it for a spin, instead of raising your blood pressure.
If you want free as in freedom, you are going to have to either bend to the preferences of the people who write the free code, or you will have to port the stuff yourself.
January 16th, 2008 at 1:23 pm
I’ve written extensively elsewhere, so I’ll keep this short. Check out survey to learn about the implementations and get plenty of links to textbooks, Wiki’s, libraries, and so on. Check out this blog posting for my feelings about Common Lisp’s general drawbacks. Yes, the situation with free implementations for Windows is not as good as the situation for non-free for Windows, or free for non-Windows. Yes, the library situation needs a lot of improvement, which may actually happen; there are people thinking about it and trying to pull together resources to fix it.
May 10th, 2008 at 2:58 pm
There are a lot of good points above. As one of the Common Lisp designers, I try hard not to be a partisan. I’ve learned over the years that being a language bigot is a very bad idea. Based on what I’ve read above, I want to learn about Newlisp.
Documentation: The open source implementors don’t seem to put a lot of attention into documentation. I’d like to see what LispWorks and Allegro provide, though. As for the language itself (as opposed to any one implementation), “Practical Common Lisp” is a great textbook. However, the ANSI standard, even the very improved HyperSpec from Kent Pitman, is pretty bad. (Personally, I liked “The Lisp Machine Manual” a lot, but I wrote half of it so I’m not totally unbiased.) A new version of “Common Lisp: The Langauge” that was updated to the X3J13 standard would be great. More online documentation would be nice, too. There are some good Emacs commands around that access the HyperSpec (I think this is all in Slime) for those of you who like Emacs, so that’s better than nothing.
Many implementations: As was pointed out, this is the norm for older languages. Even Python has several implementations (Jython, IronPython), for good reasons. The Lisp implementations vary in many ways; see my survey paper at
http://common-lisp.net/~dlw/LispSurvey.html
And I didn’t do any benchmarking there. For example, some implementations have highly optimized bignums (arbitrary precision integer arithmetic) and others don’t; only some people care about that.
Windows: More than half (six of eleven) of the current implementations run on Windows. Some of the others are working on it. Corman Common Lisp is quite interesting in that it ONLY runs on Windows, and has many useful features to fit into the Microsoft ecosystem, for those who are interested in that. I think all the others were originally for Linux/Unix and then were ported to Windows. Again, LispWorks and Allegro may do a great job; I’ve never used them, but these guys are real pro’s in a for-money business, so they have serious motivation as well as high-quality hackers.
Libraries: Yes, there should be a centralized source a la CPAN. Marco Baringer and the Clozure guys have finally announced CLornucopia, which is just getting started. I think it will be the answer to this very important requirement. It’s not ready to be used yet (it’s in alpha and lots of projected cool features haven’t been done yet), but it’s on its way.