Eli Bendersky's website - Multiple dispatch

The Expression Problem in Go

2018-02-20T05:20:00-08:00

I've written about the expression problem here and later also here. Between them, these posts presented various solutions and non-solutions in C++, Haskell and Clojure.

Today I want to add another language into the mix - Go. It turns out that Go interfaces allow us to solve the expression problem, or at least a limited variant thereof.

The Go way

Go's main vehicle of abstraction and polymorphism is interfaces. Go encourages the use of small interfaces, such that types could be easily implementing multiple interfaces if needed.

It turns out this tool and philosophy is exactly suitable for tackling the expression problem in Go.

The expression problem in Go

For a recap of the expression problem, please refer to the past posts linked at the top. Without further ado, here's a Go solution.

type Expr interface {
}

// Types
type Constant struct {
  value float64
}

type BinaryPlus struct {
  left  Expr
  right Expr
}

Note that the Expr interface is empty - any type implements it. We need something like this so that we can specify the types for the left and right members of BinaryPlus; it really means left and right can be of any type. We'll get back to this discussion shortly.

Now an interface for expressions we can evaluate:

type Eval interface {
  Eval() float64
}

And implementations for our existing types:

func (c *Constant) Eval() float64 {
  return c.value
}

func (bp *BinaryPlus) Eval() float64 {
  return bp.left.(Eval).Eval() + bp.right.(Eval).Eval()
}

The instance for Constant is straightforward, but what's going on with BinaryPlus? Recall that left and right can have arbitrary types at compile time. But at run-time, we need these to be Eval, hence the type casts. This code will fail with a cast error at run-time if the objects occupying b's left or right slots don't, in fact, imlement Eval.

Note that we could force Expr to have at least an Eval method - all expressions have to be evaluable, after all. This would make the Expr interface a bit less empty, and also would remove the need for casts in implementations of Eval. We'd still need casts for other interfaces though, as we'll see shortly.

OK, now we have the basics. Let's add another operation, without modifying existing code:

type Stringify interface {
  ToString() string
}

func (c *Constant) ToString() string {
  return strconv.FormatFloat(c.value, 'f', -1, 64)
}

func (bp *BinaryPlus) ToString() string {
  ls := bp.left.(Stringify)
  rs := bp.right.(Stringify)
  return fmt.Sprintf("(%s + %s)", ls.ToString(), rs.ToString())
}

No surprises here. How about adding a new type:

type BinaryMul struct {
  left  Expr
  right Expr
}

func (bm *BinaryMul) Eval() float64 {
  return bm.left.(Eval).Eval() * bm.right.(Eval).Eval()
}

func (bm *BinaryMul) ToString() string {
  ls := bm.left.(Stringify)
  rs := bm.right.(Stringify)
  return fmt.Sprintf("(%s * %s)", ls.ToString(), rs.ToString())
}

Again, very similar code to what we've written for other types. Finally, let's write some client code that uses these types and operations:

func CreateNewExpr() Expr {
  c11 := Constant{value: 1.1}
  c22 := Constant{value: 2.2}
  c33 := Constant{value: 3.3}
  bp := BinaryPlus{left: &BinaryPlus{left: &c11, right: &c22}, right: &c33}
  return &bp
}

func main() {
  ne := CreateNewExpr()
  fmt.Printf("ne Eval = %g\n", ne.(Eval).Eval())
  fmt.Printf("ne ToString = %s\n", ne.(Stringify).ToString())
}

This snippet demonstrates how we can create new expressions at runtime (the return type of CreateNewExpr is simply Expr) and observe their implementation of the interfaces we've defined. When run, it prints:

ne Eval = 6.6
ne ToString = ((1.1 + 2.2) + 3.3)

Discussion

So, what works well for Go here, and what doesn't?

On one hand, this seems too easy. In my initial post, I've shown an attempt to implement the same thing in C++, and encountered a fundamental problem very early - C++ has to know, at compile-time, which interfaces a class implements. Go doesn't, and herein lies the difference. Go is more like Clojure than like C++ in this respect - methods are defined outside of types, which provides two strong advantages:

Given a new type, we can make it implement a bunch of interfaces without modifying these interfaces.
Given a new interface, we can make existing types implement it without modifying their code, just by adding new code. One could say that Go satisfies the open/closed principle very well.

On the other hand, it's important to understand that the casts involved in this solution lose the static type safety guarantees of the language. Again, here Go is more like Clojure than like C++. The compiler can not verify at compile-time that we assign the right sub-expressions to a BinaryPlus, for example. We can write non-sensical code like this:

bp1 := BinaryPlus{left: "john", right: Constant{value: 2.2}}
fmt.Printf("bp1 Eval = %g\n", bp1.Eval())

And it will compile just fine! It will fail at run-time, though:

panic: interface conversion: string is not main.Eval: missing method Eval

goroutine 1 [running]:
main.(*BinaryPlus).Eval(0xc420047f30, 0xc420047df0)
  goexprsolution.go:47 +0x47
[...]

In fact, one could reasonably claim this is not a valid solution to the expression problem as originally defined by Philip Wadler:

The goal is to define a datatype by cases, where one can add new cases to the datatype and new functions over the datatype, without recompiling existing code, and while retaining static type safety (e.g., no casts).

Note the explicit ban on casts.

The Expression Problem and its solutions

2016-05-12T05:44:00-07:00

The craft of programming is almost universally concerned with different types of data and operations/algorithms that act on this data [1]. Therefore, it's hardly surprising that designing abstractions for data types and operations has been on the mind of software engineers and programming-language designers since... forever.

Yet I've only recently encountered a name for a software design problem which I ran into multiple times in my career. It's a problem so fundamental that I was quite surprised that I haven't seen it named before. Here is a quick problem statement.

Imagine that we have a set of data types and a set of operations that act on these types. Sometimes we need to add more operations and make sure they work properly on all types; sometimes we need to add more types and make sure all operations work properly on them. Sometimes, however, we need to add both - and herein lies the problem. Most of the mainstream programming languages don't provide good tools to add both new types and new operations to an existing system without having to change existing code. This is called the "expression problem". Studying the problem and its possible solutions gives great insight into the fundamental differences between object-oriented and functional programming and well as concepts like interfaces and multiple dispatch.

A motivating example

As is my wont, my example comes from the world of compilers and interpreters. To my defense, this is also the example used in some of the seminal historic sources on the expression problem, as the historical perspective section below details.

Imagine we're designing a simple expression evaluator. Following the standard interpreter design pattern, we have a tree structure consisting of expressions, with some operations we can do on such trees. In C++ we'd have an interface every node in the expression tree would have to implement:

class Expr {
public:
  virtual std::string ToString() const = 0;
  virtual double Eval() const = 0;
};

This interface shows that we currently have two operations we can do on expression trees - evaluate them and query for their string representations. A typical leaf node expression:

class Constant : public Expr {
public:
  Constant(double value) : value_(value) {}

  std::string ToString() const {
    std::ostringstream ss;
    ss << value_;
    return ss.str();
  }

  double Eval() const {
    return value_;
  }

private:
  double value_;
};

And a typical composite expression:

class BinaryPlus : public Expr {
public:
  BinaryPlus(const Expr& lhs, const Expr& rhs) : lhs_(lhs), rhs_(rhs) {}

  std::string ToString() const {
    return lhs_.ToString() + " + " + rhs_.ToString();
  }

  double Eval() const {
    return lhs_.Eval() + rhs_.Eval();
  }

private:
  const Expr& lhs_;
  const Expr& rhs_;
};

Until now, it's all fairly basic stuff. How extensible is this design? Let's see... if we want to add new expression types ("variable reference", "function call" etc.), that's pretty easy. We just define additional classes inheriting from Expr and implement the Expr interface (ToString and Eval).

However, what happens if we want to add new operations that can be applied to expression trees? Right now we have Eval and ToString, but we may want additional operations like "type check" or "serialize" or "compile to machine code" or whatever.

It turns out that adding new operations isn't as easy as adding new types. We'd have to change the Expr interface, and consequently change every existing expression type to support the new method(s). If we don't control the original code or it's hard to change it for other reasons, we're in trouble.

In other words, we'd have to violate the venerable open-closed principle, one of the main principles of object-oriented design, defined as:

software entities (classes, modules, functions, etc.) should be open for extension, but closed for modification

The problem we're hitting here is called the expression problem, and the example above shows how it applies to object-oriented programming.

Interestingly, the expression problem bites functional programming languages as well. Let's see how.

The expression problem in functional programming

Update 2018-02-05: a new post discusses the problem and its solutions in Haskell in more depth.

Object-oriented approaches tend to collect functionality in objects (types). Functional languages cut the cake from a different angle, usually preferring types as thin data containers, collecting most functionality in functions (operations) that act upon them. Functional languages don't escape the expression problem - it just manifests there in a different way.

To demonstrate this, let's see how the expression evaluator / stringifier looks in Haskell. Haskell is a good poster child for functional programming since its pattern matching on types makes such code especially succinct:

module Expressions where

data Expr = Constant Double
          | BinaryPlus Expr Expr

stringify :: Expr -> String
stringify (Constant c) = show c
stringify (BinaryPlus lhs rhs) = stringify lhs
                                ++ " + "
                                ++ stringify rhs

evaluate :: Expr -> Double
evaluate (Constant c) = c
evaluate (BinaryPlus lhs rhs) = evaluate lhs + evaluate rhs

Now let's say we want to add a new operation - type checking. We simply have to add a new function typecheck and define how it behaves for all known kinds of expressions. No need to modify existing code.

On the other hand, if we want to add a new type (like "function call"), we get into trouble. We now have to modify all existing functions to handle this new type. So we hit exactly the same problem, albeit from a different angle.

The expression problem matrix

A visual representation of the expression problem can be helpful to appreciate how it applies to OOP and FP in different ways, and how a potential solution would look.

The following 2-D table (a "matrix") has types in its rows and operations in its columns. A matrix cell row, col is checked when the operation col is implemented for type row:

In object-oriented languages, it's easy to add new types but difficult to add new operations:

Whereas in functional languages, it's easy to add new operations but difficult to add new types:

A historical perspective

The expression problem isn't new, and has likely been with us since the early days; it pops its head as soon as programs reach some not-too-high level of complexity.

It's fairly certain that the name expression problem comes from an email sent by Philip Wadler to a mailing list deailing with adding generics to Java (this was back in the 1990s).

In that email, Wadler points to the paper "Synthesizing Object-Oriented and Functional Design to Promote Re-Use" by Krishnamurthi, Felleisen and Friedman as an earlier work describing the problem and proposed solutions. This is a great paper and I highly recommend reading it. Krishnamurthi et.al., in their references, point to papers from as early as 1975 describing variations of the problem in Algol.

Flipping the matrix with the visitor pattern

So far the article has focused on the expression problem, and I hope it's clear by now. However, the title also has the word solution in it, so let's turn to that.

It's possible to kinda solve (read on to understand why I say "kinda") the expression problem in object-oriented languages; first, we have to look at how we can flip the problem on its side using the visitor pattern. The visitor pattern is very common for this kind of problems, and for a good reason. It lets us reformulate our code in a way that makes it easier to change in some dimensions (though harder in others).

For the C++ sample shown above, rewriting it using the visitor pattern means adding a new "visitor" interface:

class ExprVisitor {
public:
  virtual void VisitConstant(const Constant& c) = 0;
  virtual void VisitBinaryPlus(const BinaryPlus& bp) = 0;
};

And changing the Expr interface to be:

class Expr {
public:
  virtual void Accept(ExprVisitor* visitor) const = 0;
};

Now expression types defer the actual computation to the visitor, as follows:

class Constant : public Expr {
public:
  Constant(double value) : value_(value) {}

  void Accept(ExprVisitor* visitor) const {
    visitor->VisitConstant(*this);
  }

  double GetValue() const {
    return value_;
  }

private:
  double value_;
};

// ... similarly, BinaryPlus would have
//
//    void Accept(ExprVisitor* visitor) const {
//      visitor->VisitBinaryPlus(*this);
//    }
//
// ... etc.

A sample visitor for evaluation would be [2]:

class Evaluator : public ExprVisitor {
public:
  double GetValueForExpr(const Expr& e) {
    return value_map_[&e];
  }

  void VisitConstant(const Constant& c) {
    value_map_[&c] = c.GetValue();
  }

  void VisitBinaryPlus(const BinaryPlus& bp) {
    bp.GetLhs().Accept(this);
    bp.GetRhs().Accept(this);
    value_map_[&bp] = value_map_[&(bp.GetLhs())] + value_map_[&(bp.GetRhs())];
  }

private:
  std::map<const Expr*, double> value_map_;
};

It should be obvious that for a given set of data types, adding new visitors is easy and doesn't require modifying any other code. On the other hand, adding new types is problematic since it means we have to update the ExprVisitor interface with a new abstract method, and consequently update all the visitors to implement it.

So it seems that we've just turned the expression problem on its side: we're using an OOP language, but now it's hard to add types and easy to add ops, just like in the functional approach. I find it extremely interesting that we can do this. In my eyes this highlights the power of different abstractions and paradigms, and how they enable us to rethink a problem in a completely different light.

So we haven't solved anything yet; we've just changed the nature of the problem we're facing. Worry not - this is just a stepping stone to an actual solution.

Extending the visitor pattern

The following is code excerpts from a C++ solution that follows the extended visitor pattern proposed by Krishnamurthi et. al. in their paper; I strongly suggest reading the paper (particularly section 3) if you want to understand this code on a deep level. A complete code sample in C++ that compiles and runs is available here.

Adding new visitors (ops) with the visitor pattern is easy. Our challenge is to add a new type without upheaving too much existing code. Let's see how it's done.

One small design change that we should make to the original visitor pattern is use virtual inheritance for Evaluator, for reasons that will soon become obvious:

class Evaluator : virtual public ExprVisitor {
  // .. the rest is the same
};

Now we're going to add a new type - FunctionCall:

// This is the new ("extended") expression we're adding.
class FunctionCall : public Expr {
public:
  FunctionCall(const std::string& name, const Expr& argument)
      : name_(name), argument_(argument) {}

  void Accept(ExprVisitor* visitor) const {
    ExprVisitorWithFunctionCall* v =
        dynamic_cast<ExprVisitorWithFunctionCall*>(visitor);
    if (v == nullptr) {
      std::cerr << "Fatal: visitor is not ExprVisitorWithFunctionCall\n";
      exit(1);
    }
    v->VisitFunctionCall(*this);
  }

private:
  std::string name_;
  const Expr& argument_;
};

Since we don't want to modify the existing visitors, we create a new one, extending Evaluator for function calls. But first, we need to extend the ExprVisitor interface to support the new type:

class ExprVisitorWithFunctionCall : virtual public ExprVisitor {
public:
  virtual void VisitFunctionCall(const FunctionCall& fc) = 0;
};

Finally, we write the new evaluator, which extends Evaluator and supports the new type:

class EvaluatorWithFunctionCall : public ExprVisitorWithFunctionCall,
                                  public Evaluator {
public:
  void VisitFunctionCall(const FunctionCall& fc) {
    std::cout << "Visiting FunctionCall!!\n";
  }
};

Multiple inheritance, virtual inheritance, dynamic type checking... that's pretty hard-core C++ we have to use here, but there's no choice. Unfortunately, multiple inheritance is the only way C++ lets us express the idea that a class implements some interface while at the same time deriving functionality from another class. What we want to have here is an evaluator (EvaluatorWithFunctionCall) that inherits all functionality from Evaluator, and also implements the ExprVisitorWithFunctionCall interface. In Java, we could say something like:

class EvaluatorWithFunctionCall extends Evaluator implements ExprVisitor {
  // ...
}

But in C++ virtual multiple inheritance is the tool we have. The virtual part of the inheritance is essential here for the compiler to figure out that the ExprVisitor base underlying both Evaluator and ExprVisitorWithFunctionCall is the same and should only appear once in EvaluatorWithFunctionCall. Without virtual, the compiler would complain that EvaluatorWithFunctionCall doesn't implement the ExprVisitor interface.

This is a solution, alright. We kinda added a new type FunctionCall and can now visit it without changing existing code (assuming the virtual inheritance was built into the design from the start to anticipate this approach). Here I am using this "kinda" word again... it's time to explain why.

This approach has multiple flaws, in my opinion:

Note the dynamic_cast in FunctionCall::Accept. It's fairly ugly that we're forced to mix in dynamic checks into this code, which should supposedly rely on static typing and the compiler. But it's just a sign of a larger problem.
If we have an instance of an Evaluator, it will no longer work on the whole extended expression tree since it has no understanding of FunctionCall. It's easy to say that all new evaluators should rather be EvaluatorWithFunctionCall, but we don't always control this. What about code that was already written? What about Evaluators created in third-party or library code which we have no control of?
The virtual inheritance is not the only provision we have to build into the design to support this pattern. Some visitors would need to create new, recursive visitors to process complex expressions. But we can't anticipate in advance which dynamic type of visitor needs to be created. Therefore, the visitor interface should also accept a "visitor factory" which extended visitors will supply. I know this sounds complicated, and I don't want to spend more time on this here - but the Krishnamurthi paper addresses this issue extensively in section 3.4
Finally, the solution is unwieldy for realistic applications. Adding one new type looks manageable; what about adding 15 new types, gradually over time? Imagine the horrible zoo of ExprVisitor extensions and dynamic checks this would lead to.

Yeah, programming is hard. I could go on and on about the limitations of classical OOP and how they surface in this example [3]. Instead, I'll just present how the expression problem can be solved in a language that supports multiple dispatch and separates the defintion of methods from the bodies of types they act upon.

Solving the expression problem in Clojure

There are a number of ways the expression problem as displayed in this article can be solved in Clojure using the language's built-in features. Let's start with the simplest one - multi-methods.

First we'll define the types as records:

(defrecord Constant [value])
(defrecord BinaryPlus [lhs rhs])

Then, we'll define evaluate as a multimethod that dispatches upon the type of its argument, and add method implementations for Constant and BinaryPlus:

(defmulti evaluate class)

(defmethod evaluate Constant
  [c] (:value c))

(defmethod evaluate BinaryPlus
  [bp] (+ (evaluate (:lhs bp)) (evaluate (:rhs bp))))

Now we can already evaluate expressions:

user=> (use 'expression.multimethod)
nil
user=> (evaluate (->BinaryPlus (->Constant 1.1) (->Constant 2.2)))
3.3000000000000003

Adding a new operation is easy. Let's add stringify:

(defmulti stringify class)

(defmethod stringify Constant
  [c] (str (:value c)))

(defmethod stringify BinaryPlus
  [bp]
  (clojure.string/join " + " [(stringify (:lhs bp))
                              (stringify (:rhs bp))]))

Testing it:

user=> (stringify (->BinaryPlus (->Constant 1.1) (->Constant 2.2)))
"1.1 + 2.2"

How about adding new types? Suppose we want to add FunctionCall. First, we'll define the new type. For simplicity, the func field of FunctionCall is just a Clojure function. In real code it could be some sort of function object in the language we're interpreting:

(defrecord FunctionCall [func argument])

And define how evaluate and stringify work for FunctionCall:

(defmethod evaluate FunctionCall
  [fc] ((:func fc) (evaluate (:argument fc))))

(defmethod stringify FunctionCall
  [fc] (str (clojure.repl/demunge (str (:func fc)))
            "("
            (stringify (:argument fc))
            ")"))

Let's take it for a spin (the full code is here):

user=> (def callexpr (->FunctionCall twice (->BinaryPlus (->Constant 1.1)
                                                         (->Constant 2.2))))
#'user/callexpr
user=> (evaluate callexpr)
6.6000000000000005
user=> (stringify callexpr)
"expression.multimethod/twice@52e29c38(1.1 + 2.2)"

It should be evident that the expression problem matrix for Clojure is:

We can add new ops without touching any existing code. We can also add new types without touching any existing code. The code we're adding is only the new code to handle the ops/types in question. The existing ops and types could come from a third-party library to which we don't have source access. We could still extend them for our new ops and types, without ever having to touch (or even see) the original source code [4].

Is multiple dispatch necessary to cleanly solve the expression problem?

I've written about multiple dispatch in Clojure before, and in the previous section we see another example of how to use the language's defmulti/defmethod constructs. But is it multiple dispatch at all? No! It's just single dispatch, really. Our ops (evaluate and stringify) dispatch on a single argument - the expression type) [5].

If we're not really using multiple dispatch, what is the secret sauce that lets us solve the expression problem so elegantly in Clojure? The answer is - open methods. Note a crucial difference between how methods are defined in C++/Java and in Clojure. In C++/Java, methods have to be part of a class and defined (or at least declared) in its body. You cannot add a method to a class without changing the class's source code.

In Clojure, you can. In fact, since data types and multimethods are orthogonal entities, this is by design. Methods simply live outside types - they are first class citizens, rather than properties of types. We don't add methods to a type, we add new methods that act upon the type. This doesn't require modifying the type's code in any way (or even having access to its code).

Some of the other popular programming languages take a middle way. In languages like Python, Ruby and JavaScript methods belong to types, but we can dynamically add, remove and replace methods in a class even after it was created. This technique is lovingly called monkey patching. While initially enticing, it can lead to big maintainability headaches in code unless we're very careful. Therefore, if I had to face the expression problem in Python I'd prefer to roll out some sort of multiple dispatch mechanism for my program rather than rely on monkey patching.

Another Clojure solution - using protocols

Clojure's multimethods are very general and powerful. So general, in fact, that their performance may not be optimal for the most common case - which is single dispatch based on the type of the sole method argument; note that this is exactly the kind of dispatch I'm using in this article. Therefore, starting with Clojure 1.2, user code gained the ability to define and use protocols - a language feature that was previously restricted only to built-in types.

Protocols leverage the host platform's (which in Clojure's case is mostly Java) ability to provide quick virtual dispatch, so using them is a very efficient way to implement runtime polymorphism. In addition, protocols retain enough of the flexibility of multimethods to elegantly solve the expression problem. Curiously, this was on the mind of Clojure's designers right from the start. The Clojure documentation page about protocols lists this as one of their capabilities:

[...] Avoid the 'expression problem' by allowing independent extension of the set of types, protocols, and implementations of protocols on types, by different parties. [...] do so without wrappers/adapters

Clojure protocols are an interesting topic, and while I'd like to spend some more time on them, this article is becoming too long as it is. So I'll leave a more thorough treatment for some later time and for now will just show how protocols can also be used to solve the expression problem we're discussing.

The type definitions remain the same:

(defrecord Constant [value])
(defrecord BinaryPlus [lhs rhs])

However, instead of defining a multimethod for each operation, we now define a protocol. A protocol can be thought of as an interface in a language like Java, C++ or Go - a type implements an interface when it defines the set of methods declared by the interface. In this respect, Clojure's protocols are more like Go's interfaces than Java's, as we don't have to say a-priori which interfaces a type implements when we define it.

Let's start with the Evaluatable protocol, that consists of a single method - evaluate:

(defprotocol Evaluatable
  (evaluate [this]))

Another protocol we'll define is Stringable:

(defprotocol Stringable
  (stringify [this]))

Now we can make sure our types implement these protocols:

(extend-type Constant
  Evaluatable
    (evaluate [this] (:value this))
  Stringable
    (stringify [this] (str (:value this))))

(extend-type BinaryPlus
  Evaluatable
    (evaluate [this] (+ (evaluate (:lhs this)) (evaluate (:rhs this))))
  Stringable
    (stringify [this]
      (clojure.string/join " + " [(stringify (:lhs this))
                                  (stringify (:rhs this))])))

The extend-type macro is a convenience wrapper around the more general extend - it lets us implement multiple protocols for a given type. A sibling macro named extend-protocol lets us implement the same protocol for multiple types in the same invocation [6].

It's fairly obvious that adding new data types is easy - just as we did above, we simply use extend-type for each new data type to implement our current protocols. But how do we add a new protocol and make sure all existing data types implement it? Once again, it's easy because we don't have to modify any existing code. Here's a new protocol:

(defprotocol Serializable
  (serialize [this]))

And this is its implementation for the currently supported data types:

(extend-protocol Serializable
  Constant
    (serialize [this] [(type this) (:value this)])
  BinaryPlus
    (serialize [this] [(type this)
                       (serialize (:lhs this))
                       (serialize (:rhs this))]))

This time, extending a single protocol for multiple data types - extend-protocol is the more convenient macro to use.

Small interfaces are extensibility-friendly

You may have noted that the protocols (interfaces) defined in the Clojure solution are very small - consisting of a single method. Since adding methods to an existing protocol is much more problematic (I'm not aware of a way to do this in Clojure), keeping protocols small is a good idea. This guideline comes up in other contexts as well; for example, it's good practice to keep interfaces in Go very minimal.

In our C++ solution, splitting the Expr interface could also be a good idea, but it wouldn't help us with the expression problem, since we can't modify which interfaces a class implements after we've defined it; in Clojure we can.

[1]

"Types of data" and "operations" are two terms that should be fairly obvious to modern-day programmers. Philip Wadler, in his discussion of the expression problem (see the "historical perspective" section of the article) calls them "datatypes" and "functions". A famous quote from Fred Brooks's The Mythical Man Month (1975) is "Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious."

[2]

Note the peculiar way in which data is passed between Visit* methods in a Expr* -> Value map kept in the visitor. This is due to our inability to make Visit* methods return different types in different visitors. For example, in Evaluator we'd want them to return double, but in Stringifier they'd probably return std::string. Unfortunately C++ won't let us easily mix templates and virtual functions, so we have to resort to either returning void* the C way or the method I'm using here.

Curiously, in their paper Krishnamurthi et.al. run into the same issue in the dialect of Java they're using, and propose some language extensions to solve it. Philip Wadler uses proposed Java generics in his approach.

[3]

I can't resist, so just in brief: IMHO inheritance is only good for a very narrow spectrum of uses, but languages like C++ hail it as the main extension mechanism of types. But inheritance is deeply flawed for many other use cases, such as implementations of interfaces. Java is a bit better in this regard, but in the end the primacy of classes and their "closed-ness" make a lot of tasks - like the expression problem - very difficult to express in a clean way.

[4] In fact, there are plenty of examples in which the Clojure implementation and the standard library provide protocols that can be extended by the user for user-defined types. Extending user-written protocols and multimethods for built-in types is trivial. As an exercise, add an evaluate implementation for java.lang.Long, so that built-in integers could participate in our expression trees without requiring wrapping in a Constant.

[5]

FWIW, we can formulate a multiple dispatch solution to the expression problem in Clojure. The key idea is to dispatch on two things: type and operation. Just for fun, I coded a prototype that does this which you can see here. I think the approach presented in the article - each operation being its own multimethod - is preferable, though.

[6]	The sharp-eyed reader will notice a cool connection to the expression problem matrix. `extend-type` can add a whole new row to the matrix, while `extend-protocol` adds a column. `extend` adds just a single cell.

A polyglot's guide to multiple dispatch - part 4

2016-04-28T06:05:00-07:00

This is part 4 in the series of articles on multiple dispatch. In previous parts we've seen what multiple dispatch is and how it can be emulated in C++ and Python. We've also seen a programming language with first-class support for multiple dispatch - Common Lisp. In this part, I'd like to discuss another language where multiple dispatch is in the standard toolbox - Clojure.

Clojure is a relatively new language but it has ancient roots. It was designed in 2007 by Rich Hickey, and has been rapidly gaining popularity in the last few years as a modern, concurrent and all-around exciting Lisp running on top of the JVM. Clojure has a large, friendly and vibrant user community and many talented developers hacking on it. I certainly harbor the hope that Clojure is what's going to finally bring Lisp mainstream... but I digress.

This article shows how multiple dispatch is done in Clojure; to prepare the ground, I'll start with single dispatch.

Single class-based dispatch

Dispatching based on some property of a value in Clojure is done using the defmulti / defmethod macro duo. These macros are sort-of, but not exactly, the equivalents of defgeneric / defmethod in Common Lisp, and the difference will be apparent very soon.

defmulti declares a "multi-method" and sets its dispatch function, which is an arbitrary Clojure function. At runtime, whenever the multi-method is called, the dispatch function is first invoked on its arguments, and the value it returns is used to choose which method to call. A simple example should help clarify this:

; Single-dispatch multimethod, dispatching on the class of the argument.
; Here 'class' refers to the built-in function clojure.core/class
(defmulti describe-thing class)

; Define dispatcher methods for built-in Long and String values.
(defmethod describe-thing java.lang.Long
  [thing] (println "a Long" (str thing)))

(defmethod describe-thing java.lang.String
  [thing] (println "a String" (str thing)))

We define the multi-method describe-thing and then two implementations, one for the built-in Long type, another for the built-in String type. Here it is in action:

multi.core=> (describe-thing 12)
a Long 12
nil
multi.core=> (describe-thing "tim")
a String tim
nil

We can also define methods for custom types:

; Define a custom class and add a dispatcher for it.
(defrecord Person [name phone])

(defmethod describe-thing Person
  [thing] (println "a Person with name" (:name thing)))

There are several ways to define new types in Clojure. Here I'm using defrecord, which is somewhat similar to a C++ struct (and a Common Lisp class). Now the right describe-thing is called when invoked with an instance of Person:

multi.core=> (describe-thing (Person. "joe" 32))
a Person with name joe

Single value-based dispatch and duck typing

So far so good. Note, however, that I said the dispatch function can be arbitrary. Indeed, in the example above we've used class as the dispatch function, but we could use something else. In fact, we could use any custom function. As another example, here's dispatch based on a field value:

(defmulti promotion-due :position)

(defmethod promotion-due :engineer
  [emp] (> (:lines-of-code emp) 100000))

(defmethod promotion-due :manager
  [emp] (> (:num-reports emp) 10))

; Works with records
(defrecord Employee [name position num-reports lines-of-code])

Forgive my rather simplistic exposition of how merit-based promotion in the tech industry works. Rather, note how cleanly this code avoids a position-based switch that would be the ordinary solution to this problem, essentially delegating the switch to Clojure itself. Let's try with a couple different employees:

=> (promotion-due (Employee. "jim" :manager 12 0))
true
=> (promotion-due (Employee. "sue" :engineer 0 98000))
false

What's the dispatch function here? It's :position. In Clojure, keywords in the first position in a form are taken as functions that access the key named by the keyword from values that support it. It works fine for records. It also works for maps, though. Can promotion-due work on maps?

=> (def joe {:name "joe", :position :manager, :num-reports 9})
=> (promotion-due joe)
false
=> (def tim {:name "tim", :position :engineer, :lines-of-code 124000})
=> (promotion-due tim)
true

Wowza! So Clojure multi-methods don't actually care what the class of the value they dispatch upon is. Indeed, they don't even care whether it's a custom class at all. As long as the dispatch function can be successfully called on the value and returns something that can be dispatched upon, multi-methods work. This is a pretty cool form of duck typing - a programming technique recently popularized by Python but also available in C++ at compile-time with templates.

In fact, the "Clojure way" of programming is to do as much duck typing as possible, on built-in types. If entities can be represented as combinations of vectors and maps, that's great because it means that in addition to custom functions we write to operate on these "types", we can also use all the other functions in our arsenal that work on built-in types. The philosophy, according to the designers of Clojure is - rather than use many different classes with a few methods on each, use a small number of data structures with many different functions that act on them [1].

In any case, as we've just seen, Clojure's records have some map-like behaviors that make it easy to intermix methods operating on them with methods operating on regular maps.

The sample code shown here uses :position as the dispatch function. An alternative method of achieving the same would be more explicit:

(defmulti promotion-due
  (fn [emp]
    (:position emp)))

The previous form is preferred because it's more concise. However, this variant is highlighting the possibility of using any custom function as the dispatcher. For example, it could return a pair of values (position and department, say) and then we'd write methods to dispatch on both. Try it as an exercise.

Multiple dispatch

The last paragraph serves as a segue to multiple dispatch. In the general case, Clojure methods accept multiple arguments (just as regular functions do). All the arguments are passed into the dispatch function, which can then return a tuple. Here's our shape intersection example in Clojure [2]:

(deftype Shape [])
(deftype Rectangle [])
(deftype Ellipse [])
(deftype Triangle [])

(defmulti intersect
  (fn [a b]
    [(class a) (class b)]))

(defmethod intersect [Rectangle Ellipse]
  [r e] (printf "Rectangle x Ellipse [names r=%s, e=%s]\n"
                (class r) (class e)))

(defmethod intersect [Rectangle Rectangle]
  [r1 r2] (printf "Rectangle x Rectangle [names r1=%s, r2=%s]\n"
                  (class r1) (class r2)))

(defmethod intersect [Rectangle Shape]
  [r s] (printf "Rectangle x Shape [names r=%s, s=%s]\n"
                (class r) (class s)))

The dispatch function here takes both arguments and returns a vector of their classes. We can then dispatch upon this vector to define methods for handling different pairs of shapes. Let's try it:

=> (intersect (Rectangle.) (Ellipse.))
Rectangle x Ellipse [names r=class multi.shapes_double_dispatch.Rectangle, e=class multi.shapes_double_dispatch.Ellipse]
nil
=> (intersect (Rectangle.) (Rectangle.))
Rectangle x Rectangle [names r1=class multi.shapes_double_dispatch.Rectangle, r2=class multi.shapes_double_dispatch.Rectangle]
nil
=> (intersect (Rectangle.) (Shape.))
Rectangle x Shape [names r=class multi.shapes_double_dispatch.Rectangle, s=class multi.shapes_double_dispatch.Shape]

So far so good. But now let's try to invoke a base-class default. We didn't define a method to intersect rectangles with triangles, so we'd like the [Rectangle Shape] variant to be invoked here:

=> (intersect (Rectangle.) (Triangle.))

IllegalArgumentException No method in multimethod 'intersect' for dispatch value: [...]

Oops... What did we expect, though? Nothing in our code even tells Clojure that Triangle and Shape are in any way related! How do we do that? deftype won't let us specify base classes. To understand how inheritance hierarchies work in Clojure, we have to take a short detour.

Clojure's ad-hoc inheritance with derive and isa?

Clojure's approach to inheritance is very unusual. Some call it abstract, and some call it "ad-hoc". You'll soon understand why.

Typically, in most languages, inheritance relations are defined between classes at the point where these classes are defined. In Clojure, the concept of an inheritance hierarchy is completely detached from any particular class mechanism - it lives on its own. We define inheritance relationships between abstract "tags", which are customarily either (namespace-qualified) symbols or keywords.

Let's define a simple animal hierarchy to demonstrate [3]:

multi.core=> (derive ::dog ::mammal)
nil
multi.core=> (derive ::cat ::mammal)
nil
multi.core=> (derive ::husky ::dog)
nil

Now we can do some queries:

multi.core=> (parents ::husky)
#{:multi.core/dog}
multi.core=> (ancestors ::husky)
#{:multi.core/mammal :multi.core/dog}
multi.core=> (descendants ::mammal)
#{:multi.core/cat :multi.core/husky :multi.core/dog}

As well as use isa? to answer the customary is-a test of object relationships:

multi.core=> (isa? ::dog ::mammal)
true
multi.core=> (isa? ::husky ::dog)
true
multi.core=> (isa? ::husky ::cat)
false

Another nice feature isa? has is accepting sequences of types and combining a per-element answer with a kind-of every?:

; isa? ::husky ::dog AND isa? ::husky ::cat
multi.core=> (isa? [::husky ::husky] [::dog ::cat])
false
; isa? ::husky ::dog AND isa? ::husky ::mammal
multi.core=> (isa? [::husky ::husky] [::dog ::mammal])
true

A final piece of the puzzle we need to understand to get back to our shape intersections is that multi-methods use isa? rather than = to dispatch. In other words, when the dispatch function is called and returns a result, to find a method that should be invoked Clojure calls (isa? actual-value expected-value) for each method's expected value to find a match. (isa? x y) always returns true if (= x y), so exact matches work as expected.

Mutliple dispatch with base-class defaults

Now we have all we need to make that intersection between a rectangle and a triangle dispatch as expected. Just for kicks, in this sample I'll forego creating new types with deftype and defrecord and will use maps instead:

(derive ::rectangle ::shape)
(derive ::ellipse ::shape)
(derive ::triangle ::shape)

; Helper constructors to create new "instances" of shapes. In a real program
; they could take parameters that would be assigned to fields.
(defn make-shape [] {:kind ::shape})
(defn make-rectangle [] {:kind ::rectangle})
(defn make-ellipse [] {:kind ::ellipse})
(defn make-triangle [] {:kind ::triangle})

Shapes are represented by maps. A special key named :kind holds the kind of the shape, which is a symbol participating in an inheritance hierarchy we've defined. The intersect dispatch is now slightly different, as are the methods:

(defmulti intersect
  (fn [a b]
    [(:kind a) (:kind b)]))

(defmethod intersect [::rectangle ::ellipse]
  [r e] (printf "Rectangle x Ellipse"))

(defmethod intersect [::rectangle ::rectangle]
  [r1 r2] (printf "Rectangle x Rectangle"))

(defmethod intersect [::rectangle ::shape]
  [r s] (printf "Rectangle x Shape"))

Let's try some intersections:

=> (intersect (make-rectangle) (make-ellipse))
Rectangle x Ellipse
=> (intersect (make-rectangle) (make-rectangle))
Rectangle x Rectangle
=> (intersect (make-rectangle) (make-shape))
Rectangle x Shape
=> (intersect (make-rectangle) (make-triangle))
Rectangle x Shape

Now the last intersect call works as expected - picking up the ::rectangle ::shape method variant, since ::triange derives from ::shape. When intersect is called on a rectangle and a triangle, the return value of the dispatch function is ::rectangle ::triangle, which passes the isa? test on ::rectangle ::shape.

The flexibility of Clojure's dispatch mechanism

Clojure's dispatch mechanism is very general, since the dispatch function is arbitrary. In part 3 we've seen how multi-methods in CLOS work: they either directly take a class to dispatch on or have an eql comparison to dispatch by-value. Clojure generalizes that - the dispatch function can do whatever it wants, as far as Clojure is concerned - it's completely unconstrained.

With added flexibility come costs, though. First and foremost, this lets us write obscure code where dispatches are made on runtime-computed values, making the code harder to reason about. I'd strongly advise against these techniques, leaving them for the inevitable 1% of the cases. For 99%, simple isa-based dispatch should be what we're looking for.

The second cost is performance. Since every call to a method has to go through the dispatch mechanism first, these pathways better be well optimized. Deferring the decision so far into the run-time makes it more difficult for the compiler to optimize. The Clojure designers had this concern in mind when they exposed protocols - a sort-of Java-like virtual dispatch mechanism that is much faster, but has some limitations compared to general multi-methods [4]. I'll cover Clojure protocols in a future post.

Thoughts on the ad-hoc inheritance hierarchy

On one hand, Clojure's way of defining inheritance hierarchies is markedly weird. Why can't we just tag the base classes on type definitions, the way it works in other languages (even in CLOS)? Why do we have to provide another set of definitions in parallel with the actual type definitions? Isn't this error prone and overly verbose?

Well, yes. It's a bit error prone (though I find it hard to imagine it can lead to deep bugs in the presence of even basic testing). It's also more verbose than strictly needed - just a list of bases in deftype would be shorter, keeping related things together.

But thinking about it another way, Clojure's choice actually makes sense. Unlike other language, Clojure really isn't centered around objects. In fact, objects and custom types are probably the last tool you should be reaching for, after all other possibilities were exhausted. As a couple of code samples in this article show, Clojure much prefers working with simple maps and compositions of built-in data types, since then the whole capabilities of the Clojure standard library are applicable to values, so we have to write less code all in all. In this light, the detached hierarchy-definition tools appear to be a very reasonable approach. In Clojure you can define new "types" with maps, with deftype, with defrecord - as well as a couple of other ways I don't cover here. Why implement the same base-class capabilities for each, when an orthogonal way of defining inheritance could be created which would then be applicable to all these approaches?

In summary, though I admit the Clojure way here is unusual, I think that overall it aligns itself well with the philosophy of the language and guides developers to think about more Clojure-y solutions to problems rather than reaching for the tools they're familiar with from traditional OO languages like Java.

Conclusion

This concludes the article, as well as the "Polyglot's guide to multiple dispatch" series. It was a fun ride, and I hope it was interesting for readers to see a powerful programming techique explained and demonstrated in four different programming languages. At first - when looking at it from the point-of-view of C++ and Python - multiple dispatch appears to be an advanced feature of OOP. I think that the Common Lisp and Clojure samples demonstrate that it goes well beyond that. In fact, multiple dispatch is best categorized as a generalization of OOP, being more in the domain of generic programming. Kinda like the generic programming of C++, just at runtime instead of being limited to compile-time constructs and metaprogramming.

[1]	When "plain" maps are used for data structures they should be still encapsulated behind functions to decouple their implementation from the interface exposed to clients.

[2]	Note that `deftype` is used here, instead of `defrecord`. Choosing between the two is a matter of style; IMHO when the type has no fields, a `deftype` conveys the meaning in a clearer way.

[3]	The Clojure syntax `::foo` stands for namespace-qualified keyword. Also, `derive` and friends (as well as multi-methods) accept an optional hierarchy map to which to add relationships. By default a global map is used, and it's good enough for our purposes in this article.

[4]	Another optimization of multi-methods in Clojure is that they're not implemented in pure Clojure as a set of macros (the way CLOS is done in Common Lisp). Rather, they dip into the Java innards of Clojure's implementation.

A polyglot's guide to multiple dispatch - part 3

2016-04-26T05:36:00-07:00

This is part 3 in the series of articles on multiple dispatch. Part 1 introduced the problem and discussed the issues surrounding it, along with a couple of possible solutions in C++. Part 2 revisited multiple dispatch in Python, implementing a couple of variations. In this part, I'm going back to the roots of multiple dispatch - Common Lisp - one of the first mainstream programming languages to introduce multi-methods and an OOP system based on multiple dispatch at its core.

Here is a list of posts in the series:

It's strange to leave the language where multiple dispatch is actually built-in to the third article in the series; however, as Common Lisp is not exactly the most widely used language these days, I wanted to start with languages folks are more familiar with. Besides, my hope is that by now readers of the series have a much better understanding and appreciation of multiple dispatch, and it should be interesting to see a very powerful and feature-full version of it directly supported by a language.

The lessons from Common Lisp weren't lost on some modern language designers, and you can find variations of its multi-methods in 21st-century languages. In the next part of the series I'll show how multiple dispatch works in Clojure - a modern Lisp dialect [1].

A bit of history

Young programmers getting started in the 2010s can be forgiven to think that object-oriented programming (OOP) the way it's done in C++, Java and Ruby is how it was always done. But in fact, while C++ appeared in the early 80s and Java in 1995, OOP first appeared in the 1960s in specialized languages like Simula, and then Smalltalk in 1972. Smalltalk itself was influenced by Lisp in many aspects, and Lisp hackers didn't wait long to borrow back OOP ideas from Smalltalk; the first beginnings of OOP libraries in Lisp [2] started popping up in the mid 70s, years before a PhD student from Denmark started tinkering with bolting classes on top of C. That was Bjarne Stroustrup, of course, and C++ saw first light in 1982. Meanwhile, the OOP features in Lisp already started to fragment into multiple competing approaches (everything in the Lisp world fragments, it's an unwritten law). It was then (before 1980) when the initial ideas of what we now know as CLOS - the Common Lisp Object System - started appearing, with multi-methods, custom method combinations, the meta-object protocol [3] and so on.

There's a lesson in this long historical preamble. OOP as it manifests itself in Common Lisp may seem foreign if all you're familiar with is languages like C++ and Java. However, don't let that fool you. OOP is not a single recipe on how to do things - it's a concept, which was realized in many different ways in the past, in different languages. In fact, if you spend a bit of time learning it - you'll quickly realize that CLOS's OOP is a superset of the single-dispatch OOP in C++ and Java, and is much more powerful and general overall.

Common Lisp and CLOS

Common Lisp is the result of a Babelian consolidation of multiple Lisp implementations that happened in the 1980s, with an ANSI standard that appeared in the early 1990s. From the start, CLOS (the Common Lisp Object System) was part of the Common Lisp standard, itself consolidating two earlier attempts at OOP in Lisp - Flavors and LOOPS.

All in all, CLOS is a very powerful OOP system with many advanced features that don't exist in today's mainstream languages. It's well worth learning about - in this humble article I only provide a small taste. I listed some links and references to materials about CLOS I personally liked at the end of the post.

Polymorphic dispatch in CLOS

From the very start CLOS's idea of runtime polymorphism was based on multiple, rather than single dispatch. Let's see why this made sense.

Here's a sample method in C++:

class Person {
public:
    void Frobnicate(Record* r, Spreadsheet* s) {
        // 'this' points to a Person.
        // 'r' points to a Record, 's' to a Spreadsheet.
    }
};

And this is how we may call it:

Person person;
Record record;
Spreadsheet spreadsheet;
// ...
person.Frobnicate(&record, &spreadsheet)     // CALL

The call line marked with CALL is the most interesting here. Note how there are two distinct roles in this call: a special position is reserved to person, which is the object the method is invoked on. It goes before the dot. Method arguments go inside the function call syntax. Within the implementation of Frobnicate, the Person instance is known as this; the arguments are declared with the usual function parameter bindings. This is frequently called "message passing" - we're sending a Frobnicate message to a Person object with Record and Spreadsheet arguments.

In Python this would be very similar, except that the method implementation has a self parameter as a more explicit reference to the object the method is invoked on. Thus most of the widespread OOP languages these days have this special slot for one of the arguments - the object we invoke the method on. Note that this is distinct syntax from regular function calls.

In Lisp, such an approach goes against the philosophy of the language, where uniform syntax is paramount. In Lisp, all the main abstraction techniques have the same syntax, called "a form":

(foo bar baz)

The first position in the form defines the meaning of the form. It may be a function call, a macro invocation, or a special form (such as if); whatever it is, the syntax is uniform. This is one of the greatest strengths of Lisp.

Back to dispatch, though. Since Lisp wants to preserve its uniform invocation syntax, a special syntax for calling methods on symbols won't do. Therefore, what CLOS has is:

(defclass Person () ())

(defmethod frobnicate ((p Person) record spreadsheet)
  (format t "~a ~a ~a~&" (type-of p) (type-of record) (type-of spreadsheet)))

The syntax is different from what we're used to. The method frobnicate is defined outside the class Person. However, the special parameter form (p Person) in the definition of frobnicate makes sure that this method is only called when the first argument is indeed a Person. If we had another type with a frobnicate method, we'd define that method as:

(defclass Asteroid () ())

(defmethod frobnicate ((a Asteroid) velocity size)
  ; do stuff
  )

Then, at runtime:

(frobnicate a-person his-record big-spreadsheet)

Would be routed to the former method, while:

(frobnicate an-asteroid very-fast pretty-small)

Would be routed to the latter. As you can see, this is your standard single-dispatch runtime polymoprhism (the dispatch indeed happens at runtime - Common Lisp is dynamically typed, like Python), but based on a uniform Lisp-y syntax where the first parameter (the actual class instance) doesn't have a special position; besides being the first, of course.

It shouldn't be a huge leap of imagination to go from this to multi-methods. Because, you see, if the class instance has no special syntax dedicated to it, who says it's special at all. Can't we dispatch upon the second argument instead? Something like:

(defmethod interplex (solution (p Person))
  (format t "~a ~a~&" (type-of solution) (type-of p)))

Here, the dispatch happens upon the runtime type of the second argument, so even the position in the call is not special. What is special about Person as far as these methods are concerned? Well, nothing really.

This is where CLOS turns OOP on its head, at least to some degree. In C++, Java and kin, methods are properties of classes; each class has its set of methods. In CLOS, this works differently. Classes and methods are orthogonal. Methods do not belong to classes (as you may have guessed by now, if only from the fact that methods are defined outside of classes entirely, rather than being nested in them). Classes do not belong to methods. Methods are simply functions that are invoked when one or more of their argument matches some runtime criteria - most often type.

So what prevents methods to dispatch based on the types of more than one parameter? Nothing indeed.

CLOS and multi-methods

Without further ado, here is a Common Lisp implementation of the shape intersection problem we've been using throughout the series:

(defclass Shape () ())
(defclass Rectangle (Shape) ())
(defclass Ellipse (Shape) ())
(defclass Triangle (Shape) ())

; Having a defgeneric is not strictly necessary in CLOS. The code would
; work without this definition. However, this is good practice as it gives
; us a natural place to document the generic interface methods are expected
; to implement. It also lets us add a default handler with a meaningful
; error message, if no suitable method was found in some case.
(defgeneric intersect (x y)
  (:documentation "Shape intersection")
  (:method (x y)
    (error "Cannot interesect these shapes")))

(defmethod intersect ((r Rectangle) (e Ellipse))
  (format t "Rectangle x Ellipse [names r=~a, e=~a]~&"
          (type-of r) (type-of e)))

(defmethod intersect ((r1 Rectangle) (r2 Rectangle))
  (format t "Rectangle x Rectangle [names r1=~a, r2=~a]~&"
          (type-of r1) (type-of r2)))

(defmethod intersect ((r Rectangle) (s Shape))
  (format t "Rectangle x Shape [names r=~a, s=~a]~&"
          (type-of r) (type-of s)))

A few things to note here:

Most importantly: intersect is specialized by the types of both its parameters, and a different method is invoked based on the actual combination of runtime types passed in. We'll see this in action soon.
Inheritance is used. defclass lets us list the superclasses of any class after its name, and here Rectangle, Ellipse, and Triangle are all set to inherit from Shape.

Let's create some objects and do some intersect calls:

;; Create some objects
(setf r1 (make-instance 'Rectangle))
(setf r2 (make-instance 'Rectangle))
(setf e1 (make-instance 'Ellipse))
(setf t1 (make-instance 'Triangle))

;; Do intersects
(intersect r1 e1)
(intersect r1 r2)
(intersect r1 t1)

We get:

Rectangle x Ellipse [names r=RECTANGLE, e=ELLIPSE]
Rectangle x Rectangle [names r1=RECTANGLE, r2=RECTANGLE]
Rectangle x Shape [names r=RECTANGLE, s=TRIANGLE]

The dispatch works as expected. Moreover, base-class defaults work out of the box - we didn't define a method for Rectangle x Triangle, so the method for Rectangle x Shape was invoked in the last call to intersect [4].

Multiple dispatch is supported natively in CLOS. Moreover, as the earlier discussion hopefully highlights, multiple dispatch is not just a variation bolted onto the system (like our @multi decorators in part 2), but it is the way polymorphic dispatch works in the language. Single dispatch is just a special case of multiple dispatch.

CLOS methods are just functions

As I've mentioned before, CLOS methods are just functions. They are not special in the same way C++ methods are special (bound to objects). This means we can use them just as we use any other function.

For example, we can map them onto objects, and it will work as expected:

(mapcar #'intersect (list r1 r2 e1) (list r2 e1 t1))

Had intersect been special as it is in C++, it wouldn't be as easy. We'd need something like std::bind to "bind" it to an object instance first. In CLOS, we don't care. We can have a higher-order function that takes a function and applies it to pairs of objects:

(defun domap (func lst1 lst2)
  (mapcar func lst1 lst2))

We can now invoke domap with either a "regular" function or a method, using the same syntax:

;; invoke cons on pairs of objects
(domap #'cons (list r1 r2 e1) (list r2 e1 t1))

;; invoke intersect on pairs of objects
(domap #'intersect (list r1 r2 e1) (list r2 e1 t1))

I'm showing this example to stress that CLOS methods are just functions. Decoupling class declarations from methods that act upon these classes lets us be much more flexible in the usage of these methods. Since Lisp is all about uniform syntax and functional programming, the result is an object-oriented system elegantly blended into the whole.

Variations on a theme

The sample above demonstrates how CLOS gives us powerful multiple-dispatch capabilities right out of the box. As it turns out, this is just the tip of the iceberg, as CLOS supports many more features. I want to give a short taste of a few here - for more information I recommend doing some reading on CLOS.

Let's start with value-based dispatch. So far we've seen how a method can speficy which type of argument it specializes on. We can also specialize methods by the value of an argument. Here's a silly example based on the same world of shapes:

(defgeneric scale (shape factor)
  (:documentation "Scale a shape by a factor"))

(defmethod scale ((shape Shape) factor)
  (format t "Scale shape by ~a~&" factor))

(defmethod scale ((shape Shape) (factor (eql 0)))
  (format t "Scale shape by a zero~&"))

Now given some s - an instance of Shape, the call (scale s 42) will dispatch to the first method, while the call (scale s 0) will dispatch to the second method. Sure, we could have achieved the same by having an if condition in the beginning of scale, and sometimes that's what we need. But eql-based dispatch lets us decouple such code and express our intent more directly. An ad-hoc if-based solution would bury the special case somewhere in the method body, while the approach shown above makes it explicit and visible that a 0 gets special treatment.

CLOS also has the concept of method combinations, which manifects in several ways. For example, with each method we can have a variation marked with :before that can serve as a pre-processor:

(defclass Shape () ())
(defclass Rectangle (Shape) ())
(defclass Ellipse (Shape) ())
(defclass Triangle (Shape) ())

(defgeneric scale (shape factor)
  (:documentation "Scale a shape by a factor"))

(defmethod scale ((r Rectangle) factor)
  (format t "Scale Rectangle by ~a~&" factor))

(defmethod scale ((e Ellipse) factor)
  (format t "Scale Ellipse by ~a~&" factor))

(defmethod scale :before ((s Shape) factor)
  (format t ":before preprocessor on ~a scaled by ~a~&" (type-of s) factor))

Now if we create a couple of shapes and scale them:

(setf r (make-instance 'Rectangle))
(setf e (make-instance 'Ellipse))

(scale r 20)
(scale e 30)

We get:

:before preprocessor on RECTANGLE scaled by 20
Scale Rectangle by 20
:before preprocessor on ELLIPSE scaled by 30
Scale Ellipse by 30

The dispatches of scale happened as we expect by now; additionally, the scale method marked with :before was invoked in both cases before the actual method.

CLOS has :before, :after and :around markings; moreover, we can specify an arbitrary number of preprocessors, postprocessors or "around"-processors and even control their exact invocation order if we're so inclined. There's a huge amount of flexibility here.

But there's more. As we've seen so far, the most specific dispatch method is used it it exists; if it doesn't, the base class's dispatch is tried, and so on. This plays well with our intuition from other OOP languages. But in CLOS, there are powerful ways to customize this flow. What really happens when a method is called is that a list of functions to call is built, including all the :before (and others) methods, as well as the actual primary methods [5], from most to least specific. We can actually control how these invocations happen.

Here's an example:

(defgeneric scale (shape factor)
  (:documentation "Scale a shape by a factor"))

(defmethod scale ((s Shape) factor)
  (format t "Scale Shape by ~a~&" factor))

(defmethod scale ((r Rectangle) factor)
  (format t "Scale Rectangle by ~a~&" factor)
  (call-next-method))

(setf r (make-instance 'Rectangle))

(scale r 20)

Now (scale r 20) prints:

Scale Rectangle by 20
Scale Shape by 20

This is not unlike super calls in Python, or base-class method calls in C++. But in CLOS, the very nature of how multiple viable methods are combined can be customized.

The default combination of primary methods is, as we've seen above, the familiar approach of OOP languages with inheritance support: dispatch to the most specific method and if it doesn't exist try to dispatch with the base class, and so on. Method combinations is a CLOS concept that lets us do it differently. There are a number of standard method combinations supported by CLOS, and custom ones can be defined with define-method-combination. Let's see a standard one for an example.

We'll start with a simple demonstration of the regular dispatch again:

(defgeneric getstuff (shape))

(defmethod getstuff ((s Shape))
  '(shape stuff))

(defmethod getstuff ((r Rectangle))
  '(rectangle stuff))

If we call (getstuff (make-instance 'Rectangle)) we get (rectangle stuff). If we call (getstuff (make-instance 'Shape)) we get (shape stuff); so far so good. But now, let's use the list method combination instead:

(defgeneric getstuff (shape)
  (:method-combination list))

(defmethod getstuff list ((s Shape))
  '(shape stuff))

(defmethod getstuff list ((r Rectangle))
  '(rectangle stuff))

What this means is that when getstuff is invoked and CLOS finds all the viable methods, instead of calling the most specific one (and letting it call the others with call-next-method if it desires), it calls all of them, combining their results with the list function.

Now, calling (getstuff (make-instance 'Rectangle)) produces the list ((rectangle stuff) (shape stuff)). Bizarre? Maybe. Useful? Maybe.

If you're now thinking "this is crazy, those Lisp people have completely jumped the shark", I can't say I entirely disagree. If you're thinking "wow, this is immensely cool" I can't say I disagree either :-) These features are as power as you can get - all implemented as a library - as a set of macros and functions in standard Common Lisp. This is the power of Lisp, friends, and herein lies its beauty... and its biggest pitfall.

IMHO these features are too advanced to be useful in 99% of the cases. It's way too easy to produce write-only code with them. Besides, even though Lisp is a multi-paradigm language and can do OOP when OOP is due, it's not idiomatic to write OOP code in Lisp since the other paradigms it provides are often better suited for the task. For the few applications where OOP really shines, the capabilities are there - but even then, I'd stick to the basics - at least in realistic multi-person code bases, leaving the power-sauce to pet projects and exploratory blog posts.

This is a good place for a quote from Paul Graham's "On Lisp":

With the addition of CLOS, Common Lisp has become the most powerful object-oriented language in widespread use. Ironically, it is also the language in which object-oriented programming is least necessary.

CLOS - yay or nay?

Let's not forget the goal of this series - explore multiple dispatch and its use and implementation in different languages. In this aspect, Common Lisp and CLOS truly shine, I think. Leaving all the crazy features aside, the basic form of OOP in CLOS relies on multiple dispatch at its very core, and lets us write object-oriented code while still remaining fairly Lisp-y.

It's not impossible to reproduce CLOS-like features in languages like Python, but it takes effort. Moreover, Python and other non-Lisp languages miss the biggest trait that makes CLOS possible - uniform syntax.

As for the power-features of CLOS, while I think they provide a sobering perspective to judge the OOP capabilities of C++, Java and Python by, I personally would be wary about using them. The Lisp ideal is growing your language towards the problem you're solving; in other words, construct an embedded DSL on top of Lisp for solving the problem. CLOS is just another example of that, where extremely powerful - to the point of obscurity - features let us really custom-tailor the language to a specific problem when necessary.

Looking forward

Common Lisp is an interesting and, in many ways, inspiring language. However, I'll admit it's not very popular or well known nowadays, and it doesn't feel right ending the series with a language "from the past". Therefore, the upcoming part 4 in the series will focus on Clojure - a modern dialect of Lisp that's become fairly popular in the last few years.

Links

There are plenty of resources to learn about CLOS. Some I found most useful are:

Richard Gabriel's overview of CLOS. He was instrumental in the definition of CLOS in the 1980s - highly recommended.
The excellent and freely available Practical Common Lisp book has a couple of nice chapters on CLOS, starting with Object Reorientation: Generic Functions.
Paul Graham's On Lisp is another classic Common Lisp book that's freely available online. It has an interesting chapter about CLOS.
Finally, Norvig's PAIP has a chapter about CLOS - I find it very well written (like the rest of this extraordinary book).

[1]	Julia and Perl 6 are some of the other modern languages that support multiple dispatch natively.

[2]

This is the place to mention that the name "Lisp" as I use it applies to a big family of programming languages. There were multiple implementations of Lisp almost from the very start, and later on the rifts only deepened with Scheme and Common Lisp. These days we also have Racket, Clojure and others, which add their own idiosyncracies but remain "a Lisp" at their core.

[3]

The meta-object protocol (MOP) is a fascinating subject I won't say more about in this article. I'll just mention that it's the idea wherein classes themselves are instances of other classes - metaclasses. If this reminds you of Python, that's because Python took these ideas from Smalltalk, just as CLOS did.

[4]	What about symmetry? CLOS doesn't support symmetric dispatch - the order of methods matters. As we've seen in part 2, implementing symmetry generically is not trivial and not without certain runtime costs.

[5]	"Primary" methods in CLOS are methods defined with `defmethod` without special tags like `:before`, `:after` or `:around`. In other words, this is what we just call "methods" in other languages. CLOS has the concept of "primary" to distinguish them from specially tagged methods.