newlisp - an intriguing dialect of Lisp
April 20th, 2006 at 5:23 pmThere has recently been a lot of buzz in the hacker blogosphere about the deficiencies of Common Lisp, peaked in Steve Yegge’s post Lisp is not an acceptable Lisp and this discussion on comp.lang.lisp initiated by the Common Lisp veteran Ron Garret.
In one of the numerous comments, I ran into a mention of newlisp - a curious little dialect of Lisp that’s aimed at the “scripting” market - to compete with Perl, Python, Ruby and friends. I took some time to browse the website, downloaded an played with the implementation, and asked some things on their forum to learn more about the language. My first impression is quite positive.
However, the dwellers of comp.lang.lisp treat newlisp with dismission at best, and with disgust and mockery at worst. They point to the following as the deficiencies of newlisp (this probably is a partial list):
- The name. Oh, how much the name newlisp infuriates them. The claim is that newlisp is not in any way more advanced than Common Lisp, so the prefix ‘new’ is wrong and misleading. There is, in fact, truth to this claim, and the newlisp fans agree that the name causes them problems.
- The memory management model is very unusual, without a garbage collector per-se. This is achieved by passing arguments to functions by value, allocating large chunks of memory and managing it internally. Mark-n-sweep garbage collection is used only during errors. Generally, I haven’t yet found any empirical proofs of the pros or cons of this approach. Despite the criticism, newlisp is quite fast for an interpreted language.
- The default scoping in newlisp is dynamic. Indeed, this is a serious deficiency. Lexical scoping is supported in a specialized way, however, through the concept of “contexts”.
- There is no auto-transition from fixnums to bignums. Only 32-bit integers and floats are supported (on 32-bit systems). Generally, I don’t see this as a show stopper - Perl, for instance, also lacks default bignums, and newlisp has a ready-to-use binding to GMP.
- newlisp’s macros can’t be used for compile-time optimization, as they are expanded at runtime exactly like functions.
- Support for OO is very rudimentary and “patchy” - akin to Perl 5’s OO.
On the other hand, there are quite a few advantages:
- Probably the most important - a tiny footprint. All you need to deploy a newlisp script is its 170Kb interpreter. To embed it, use the 180Kb DLL.
- The language is portable and cross-platform
- Despite its unusual memory management, newlisp is fast. It measures very favorably against the mainstream scripting languages - Perl, Pyhton and Ruby.
- PCRE (Perl Compatible Regular Expressions) that works out of the box
- Graphics with Tk out of the box. The tiny footprint plays another role here - the Tk-aware newlisp interpreter is only 1.4Mb in size, and it’s all you need to run swift Tk applications on all platforms.
- A lot of useful and functional libraries, out of the box. TCP/IP and UDP networking, MYSQL and SQLite access, CGI, FTP, HTTP, XML, XML-RPC, Matrices, statistics, Unicode and more.
- Bindings to C / C++ code trivial - very simple to load and use a C++ DLL. This helps bind popular libraries to newlisp (like GMP for arbitrary-precision arithmetic) and optimize tight pieces of code in pure C.
- Macros are functional and can be used to extend the language and write DSLs.
- Very active development. newlisp quickly closes the gap from the other Lisp dialects. Its code is open (GPL) and looks well managed, the documentation is detailed and tidy, and the implementor is open minded for new ideas and improvements.
(There’s a page on newlisp’s website listing the differences from other Lisps)
Curiously, the slogan of quite a few newlisp enthusiasts are “newlisp - because programming is fun again”. Recalling the claims of Steve Yegge and Ron Garret, I can’t help but think - what if they are right ? newlisp certainly is a dialect of Lisp, with everything we love in Lisp - functional programming, support for lists, automatic storage management, dynamic typing, first class functions, uniform syntax, interactive environment, extensibility through macros. Add to this loads of useful libraries that come out of the box, the simplicity to bind to other libraries just-in-case, the small memory footprint, and you have what looks like a fun language. It also looks like a language with a future, that is being actively developed and improved, here and now. Most of the criticism of newlisp is about things that are simple to amend, and I have no doubt they will be fixed in time due. It doesn’t make newlisp any less usable until then, though.
Sure, it is inferior to Common Lisp and Scheme from a theoretical viewpoint. But people are writing real applications in newlisp and are enjoying it. Besides, I’m sure it can only do good to the Lisp community, since it may introduce more newbies to the language. I don’t think anyone will disagree that newlisp is much more newbie-friendly than either CL or Scheme.
Related posts:
April 20th, 2006 at 5:51 pm
From a user’s perspective, newLISP’s memory management works just like garbage collection in other scripting languages: memory gets allocated automatically and unused memory gets recycled. newLISP also does references passing by packaging variables into contexts.
Dynamic scoping style seems not to be a disadvantage when using the language, as long as you divide your code into context modules.
Thanks for this balanced post, which seems to understand the philosophy behind newLISP.
April 20th, 2006 at 6:10 pm
A few notes:
1. c.l.l is not a homogeneous group of people. The fact that some people there react strongly about certain issues doesn’t mean that everyone over there feels the same. It’s not helpful that you characterize them like that.
2. The notion of contexts sounds interesting, I will check it out.
3. Macros aren’t/shouldn’t be used for optimization anyway. They should be used for syntactic abstractions. For efficiency, function inlining (or in Common Lisp also compiler macros) is the more appropriate tool.
4. You also get tiny footprint with some Common Lisp implementations.
5. More importantly, each single Common Lisp implementation provides quite a number of features out of the box as well. You don’t need to switch to a completely different Lisp dialect just to get things out of the box.
You might want to switch to a different Lisp dialect for other reasons, but it escapes me why lack of portability of out-of-the-box libraries in some Common Lisp implementations is held against Common Lisp, but switching to a completely different Lisp dialect is considered a good solution. Something doesn’t compute here. I mean, you don’t get portability to other Common Lisp implementations by switching to newlisp either, do you? What am I missing?
April 20th, 2006 at 6:53 pm
Hi Eli. A good summary - although I should perhaps clarify my earlier comment over at newLISP HQ: the only ‘problem’ with the name ‘newLISP’ is that people misunderstand it. (Personally I think it’s unwise to judge anything solely by its name…!)
But why not use both languages - or more? I have no problem with using three or four languages in the space of a week, and I’m not even a professional programmer. newLISP is certainly one of the best ways I’ve found to date of writing useful and quick scripts to run in - say - BBEdit or at the Mac Finder level, and it’s a good language for thinking in: expressive and responsive, while remaining simple enough for my taste. If I could find a version of Common Lisp that has a proper Mac installation DMG and can be interact with BBEdit I’d use that as well.
On the other hand, ff you want to choose just one language to implement a particular project, then choose the best tool for the job, rather than one pre-determined by your experience to date.
April 20th, 2006 at 7:01 pm
Pascal:
1. True. I’m a lurker on c.l.l for years and years, and there indeed is a big variance in people’s attitude. In my characterization I didn’t intend to insult or even take sides, and I tried to be objective.
3. Common Lisp indeed has the huge advantage of having real *compilers*, which makes it stand-out over other high-level languages in terms of performance. newlisp is no match for CL in speed.
5. I’m not aware of any CL implementation that provides such a set of features out-of-the-box, in a single installation file. Can you point me to one please ?
I see the problem you point out with switching to a different dialect. But something feels wrong about starting with a CL implementation, I can’t put a finger on it, but it just does. Maybe it’s the segmentation in the community, maybe it’s the incompatibilities. But something is wrong, and we can’t just ignore it.
Somehow, newlisp manages to pull people in by simply being a “handy Lisp, ready to use”.
April 20th, 2006 at 7:08 pm
cormullion,
A very good point. I also frequently find myself using several (quite different) languages for different purposes during a single week (my record is 3 in one day
). By comparing the languages we use and learning more about them, we can make better decisions when deciding on that best tool for the job we have at hand.
April 20th, 2006 at 7:32 pm
Eli,
5. OpenLisp is technically not a Common Lisp, but an implementation of ISLISP which is mostly a subset of Common Lisp, with only few exceptions. It provides quite a lot of features, although I am not sure whether it matches everything you need. I think that LispWorks and Allegro Common Lisp are pretty good wrt feature completeness, but many people seem to have the “must-be-open-source” filter that I also don’t understand. So I can’t comment here. My impression was that clisp is also quite complete, but maybe I am wrong here.
Wrt “something feels wrong about starting with a [single] CL implementation”: maybe there’s nothing wrong about it, maybe it’s just your imagination?
April 20th, 2006 at 7:44 pm
Pascal,
I’ll take a look at OpenLisp, thanks.
If that was only my feeling, I could dismiss it as my imagination. But it clearly isn’t - and you as a c.l.l regular surely know it.
April 20th, 2006 at 7:46 pm
Found this and thought it was quite funny and sums up the problems with lisps lack of (easy to find, install and use on windows) libraries.
Answer to “What are the problems with newLisp” on comp.lang.lisp:
“> No garbage collection, no sharing of values: all values are copied
> when passed as function arguments or put in data structures (except
> for some builtin functions), no lexical scoping, numbers are only
> fixnums and floats, silly restrictions with the only justification
> being interpreter simplicity (e.g. length of quoted string constants
> is limited to 2048, strings cannot contain NULs), ugly FFI which e.g.
> represents pointers as integers on the newLisp side (doesn’t work when
> a pointer doesn’t fit in C int), pointless rules like cons behaving
> like list when the second argument is not a list (no practical and
> safe code would make use of this).
But it can move the smile.bmp around the screen.
stefan “
April 20th, 2006 at 7:50 pm
Uh oh, OpenLisp is a commercial product. This just won’t cut, I think. Remember, we’re talking here about an easy newbie entry into the Lisp world.
April 20th, 2006 at 7:55 pm
Ben,
Most of these points are summarized in my post. Some of them I didn’t address:
- no lexical scoping: no longer true, lexical scoping is possible with contexts
- silly restrictions…: I dismiss these as trivially fixable
- ugly FFI: a simple binding to C libraries that just works - is this ugly ?
- pointless rules…: these are just semantics that are a little different, I see no real harm in it as long as the differences are well understood
In fact, you quoted one Marcin Kowalczyk, who as far as I see from his posts has some personal vendetta against newlisp. I don’t understand such an approach, and I would really prefer an objective discussion with real, hard examples.
April 20th, 2006 at 8:48 pm
So you indeed have the “must-be-non-commercial” filter. It doesn’t even allow you to read two sentences further where you could have read that OpenLisp can be downloaded and used for free for non-commercial uses.
You’re missing quite a few options that way.
April 20th, 2006 at 10:22 pm
most of these “silly restrictions”, recycled again and again on some forums, are simply wrong or misleading:
>> No garbage collection
misleading, because has automatic memory management
>> all values are copied when passed as function arguments
not true, can pass by reference, packaged in contexts
>> length of quoted string constants is limited to 2048
not true, misleading! limiting with [text], [/text] tags and can have unlimited length (32 bit max int)
>> strings cannot contain NULL
not true! strings can contain NULLs, many string functions can work on binary content with NULLs or other non-printing characters, newLISP stores the string length in a separate field.
>> ugly FFI:
opinion not a fact, importing functions in newLISP requires ‘C’ language knowledge, but creates very tight and fast couplings to ‘C’ libraries, see modules in the distribution
April 20th, 2006 at 10:59 pm
Pascal,
Actually I did read further. And call me finicky, but I want a free license for commercial uses - just like I have with Perl, with Ruby, with C/C++ for that matter (MS free VC++ command line, Borland C++ 5.5, gcc…)
April 20th, 2006 at 11:14 pm
Nice article!
Skipping the Holy War topics, I’ll trying to say about my newLISP expirience.
Unfortunately, I’m a profane in CL. So, I’ll say from my procedural and object-oriented base:
Like many of us, for me the fast writing of reasonable good programs is more preferable, than their fast executing. My golas are the system management, report generation and a bit of financial computations.
One can try to solve this tasks with C++/Java/etc., but the “normal” guys will prefer Perl or something similar
So, let’s compare:
With newLISP here we can have a speed much like Perl.
We have OS integration features and portability like Perl.
We have GPL’ed engine like Perl.
We have the code size comparable with Perl.
We have more code simplicity than in Perl.
We have less engine size than Perl has.
And, finally, we have much more fun when writing the code. Because LISP is rules forever anyway
The question is: So why not newLISP instead of Perl?
I using newLISP about a half of a year and I have fun.
Another question is: Why not CL for that?
My _personal_ answer is:
- Give me a GPL’ed CL
- That working under _any_ regular OS by installing only executable
- That will work fastly also under P600/64M/W98
- That will be able to import from dll’s
- That will have an IP and PCRE features.
- That give me a way to write programs _fastly_ than I do in newLISP.
…probably there is one.
While I has looking for such thing, I found newLisp.
April 23rd, 2006 at 7:27 am
I use perl, bash, and XSLT. After browsing this discussion, I’d like to say that it taught me something about how people use different languages.
The difference between newLisp and the common lisp implementations might be that newLisp makes a good CGI language, (and its contexts beg you to work with xml, which no other CGI language can claim), while these other lisp versions are for the professional programmer looking for industrial-strength applications.
That’s a newbie perspective, of course. Scripting languages are tools for me to accomplish other non-programming tasks, particularly writing tasks. I’d never stake anything important on my opinion of any of them.
-Noah
May 1st, 2006 at 4:51 am
I was going to use newlisp for a project but the process management sucked bad and I found that there was no advantage over using CLISP. CLISP has everything touted by newlisp plus a lot more, especially standalone executables.