Robust exception handling

Executive introduction

  1. Exceptions are important
  2. This article is mainly intended for my own consumption, so take it with a grain of salt
  3. The examples are in Python, but the ideas apply to most languages

Exceptions vs. error status codes

Exceptions are better than returning error status codes. Some languages (like Python) leave you with no choice but to handle exceptions, as the whole language core and standard libraries throw them. But even in languages where this is not so, you should prefer exceptions to error codes.

In the Python world this is sometimes called the EAFP (Easier to Ask for Forgiveness than Permission) versus LBYL (Look Before You Leap) debate. To quote Alex Martelli1 (because I couldn't have said it better myself):

A common idiom in other languages, sometimes known as "look before you leap" (LBYL), is to check in advance, before attempting an operation, for all circumstances that might make the operation invalid. This approach is not ideal for several reasons:

1. The checks may diminish the readability and clarity of the common, mainstream cases where everything is okay.
2. The work needed for checking may duplicate a substantial part of the work done in the operation itself.
3. The programmer might easily err by omitting some needed check.
4. The situation might change between the moment the checks are performed and the moment the operation is attempted

Here's an example of LBYL:

def do_something(filename):
  if not can_open_file(filename):
    return err(...)
  f = open(filename)
  ...

Suppose that can_open_file indicates whether the subsequent open call will succeed. Let's see how Alex's points apply here:

  1. In this case the diminished readability is less obvious, but when more error checks are added, it is. Anyone with enough experience in C is familiar with the bulks of error checking typical at the top of some functions2.
  2. can_open_file definitely duplicates some of the checking done in open. Just because open is a built-in, you feel less DRY-violation.
  3. How can you be sure you've covered everythin in can_open_file ? What if you have not ?
  4. Suppose you work in a multi-processing environment (no need to think far, a web application running on a server is a good example). Between your test with can_open_file and the actual call to open, some other process might have modified the file's properties, opened it, or deleted it. This is a hard-to-debug race condition.

The same code with exception handling (EAFP):

def do_something(filename):
  try:
    f = open(filename)
  except IOError, e:
    raise MyApplicationsExceptionType(e.msg)
    # could even pass whole traceback etc
    # etc...

Suffers of none of these problems and provides greater flexibility. For example, in some cases you can skip the exception checking in do_something completely, because catching it at a higher level is more appropriate. This is difficult to do with error codes.

Never use exceptions for flow-control

Exceptions exist for exceptional situations: events that are not a part of normal execution. When a programmer calls str.find('substring') he doesn't expect an exception to be thrown if the substring isn't found. This is what he called find for. A better approach is to return a special value like None or -1. But if he called str[idx], and str is shorter than idx+1 characters, an exception is appropriate, because the user expected index idx to be valid, but it isn't - this is an unanticipated event.

When used for flow-control, exceptions are like goto. There might be a few esoteric cases in which they're appropriate, but 99.99% of the time they are not.

Handle exceptions at the level that knows how to handle them

By their very nature, exceptions can propagate up a hierarchy and be caught at multiple levels. A questions rises - where is it appropriate to catch and handle an exception ?

The best place is that piece of code that can handle the exception. For some exceptions, like programming errors (e.g. IndexError, TypeError, NameError etc.) exceptions are best left to the programmer / user, because "handling" them will just hide real bugs.

If you have a complete application, it is not appropriate to just fail with an exception, of course. It's better to demonstrate a polite error message (or dialog) and thoroughly log the exception itself, which can later help in the investigation of the bug (kind of like the dialog Microsoft replaced BSODs with - you get a polite message and may send a detailed report to Microsoft).

So before writing try/except, always ask yourself - "is this the right place to handle this exception ? do I have enough information to handle it here ?".

This is also the reason why you should be extremely careful with except: clauses that catch everything. These will not only catch the exceptions you intended, but all of them.

Do not expose implementation details with exceptions

As I noted above, exceptions ride the express train up your implementation hierarchy. As such, they are prone to break encapsulation and expose implementation details. An example:

Suppose you implement an object that manages an internal cache, using a file. An error may occur that that the object can't open its cache file for some reason. The underlying open function will throw IOError. Where should you catch it ?

Do not leave it to the user of the object. This breaks encapsulation, since the user should not be aware that you're using a file at all. Maybe in the next version you'll use a database, or a web connection ?

Rather, you should catch the exception in the object, wrap it with your own custom exception (say, CachingFailedError), and trying to preserve as much information as possible, propagate it up. The user of the object will then only have to catch your exception, and won't have to modify his code when you modify the underlying caching implementation. However, if during testing the user will want to investigate the source of the error, he will have all the information he needs to do so.

Re-raising exceptions correctly is difficult3. Unfortunately there's no one-fit-all recipe - can you recover from the error ? can you provide extra informatin ? do you need the original exception ? do you need its traceback ?

If all you need is to re-raise an exception with a more appropriate type, retaining the information stored in the old exception, do:

class StuffCachingError(Exception): pass

def do_stuff():
    try:
        cache = open(filename)
        # do stuff with cache
    except IOError, e:
        raise StuffCachingError('Caching error: %s' % e)

Benefits:

  1. You didn't expose the internal implementation to outside code. Now users of do_stuff can install handlers for StuffCachingError, without knowing you use files under the hood.
  2. If the user wants to investigate the error in depth, you've also provided the original exception in the error message, and he can do so.

If you think you need to retain the original traceback as well (IMHO is most cases you don't have to), write:

class StuffCachingError(Exception): pass

def do_stuff():
    try:
        cache = open('z.pyg')
        # do stuff with cache
    except IOError:
        exc_class, exc, traceback = sys.exc_info()
        my_exc = StuffCachingError('Caching error: %s' % exc)
        raise my_exc.__class__, my_exc, traceback

Now StuffCachingError will be raised as before, but with the original traceback leading to the call to open.

Document the exceptions thrown by your code

A useful idiom when thinking about programs is in terms of contracts4. Put simply, each function/method has a contract with the outside world:

  1. What it expects from its argument or the state of the world when invoked
  2. The result(s) it returns
  3. Its side effects on the world
  4. The exceptions it throws

For some reason, it is much more common to explicitly document the first three terms, but not the last one. I think this is wrong, and exceptions should be documented as well.

Don't overdo it, of course. Writing "this code can throw NameError because I might have made a coding mistake" is superfluous at best. But thinking about which exceptions your code may throw will help you write better, safer and more encapsulated code. Maybe some exceptions must be hidden from the outside world (just like in the earlier tip on encapsulation). The user of your code must know which handlers to prepare, and which exceptions to propagate forward.

1 Chapter 6 of "Python in a Nutshell" by Alex Martelli

2 Ned Batchelder proposes this example (C/C++):

Using error codes:

STATUS DoSomething(int a, int b)
{
    STATUS st;
    st = DoThing1(a);
    if (st != SGOOD) return st;
    st = DoThing2(b);
    if (st != SGOOD) return st;
    return SGOOD;
}

Using exceptions:

void DoSomething(int a, int b)
{
    DoThing1(a);
    DoThing2(b);
}

3 Ian Bicking treats the topic with some depth, but I'll be glad to get more references.

4 See Dan Weinreib's insightful treatment of the subject of contracts and exceptions here.

For comments, please send me an email.