Robust exception handling

August 21st, 2008 at 10:27 am

Executive introduction

  1. Exceptions are important
  2. This article is mainly intended for my own consumption, so take it with a grain of salt
  3. The examples are in Python, but the ideas apply to most languages

Exceptions vs. error status codes

Exceptions are better than returning error status codes. Some languages (like Python) leave you with no choice but to handle exceptions, as the whole language core and standard libraries throw them. But even in languages where this is not so, you should prefer exceptions to error codes.

In the Python world this is sometimes called the EAFP (Easier to Ask for Forgiveness than Permission) versus LBYL (Look Before You Leap) debate. To quote Alex Martelli1 (because I couldn’t have said it better myself):

A common idiom in other languages, sometimes known as “look before you leap” (LBYL), is to check in advance, before attempting an operation, for all circumstances that might make the operation invalid. This approach is not ideal for several reasons:

1. The checks may diminish the readability and clarity of the common, mainstream cases where everything is okay.
2. The work needed for checking may duplicate a substantial part of the work done in the operation itself.
3. The programmer might easily err by omitting some needed check.
4. The situation might change between the moment the checks are performed and the moment the operation is attempted

Here’s an example of LBYL:

def do_something(filename):
  if not can_open_file(filename):
    return err(...)
  f = open(filename)

Suppose that can_open_file indicates whether the subsequent open call will succeed. Let’s see how Alex’s points apply here:

  1. In this case the diminished readability is less obvious, but when more error checks are added, it is. Anyone with enough experience in C is familiar with the bulks of error checking typical at the top of some functions2.
  2. can_open_file definitely duplicates some of the checking done in open. Just because open is a built-in, you feel less DRY-violation.
  3. How can you be sure you’ve covered everythin in can_open_file ? What if you have not ?
  4. Suppose you work in a multi-processing environment (no need to think far, a web application running on a server is a good example). Between your test with can_open_file and the actual call to open, some other process might have modified the file’s properties, opened it, or deleted it. This is a hard-to-debug race condition.

The same code with exception handling (EAFP):

def do_something(filename):
    f = open(filename)
  except IOError, e:
    raise MyApplicationsExceptionType(e.msg)
    # could even pass whole traceback etc
    # etc...

Suffers of none of these problems and provides greater flexibility. For example, in some cases you can skip the exception checking in do_something completely, because catching it at a higher level is more appropriate. This is difficult to do with error codes.

Never use exceptions for flow-control

Exceptions exist for exceptional situations: events that are not a part of normal execution. When a programmer calls str.find('substring') he doesn’t expect an exception to be thrown if the substring isn’t found. This is what he called find for. A better approach is to return a special value like None or -1. But if he called str[idx], and str is shorter than idx+1 characters, an exception is appropriate, because the user expected index idx to be valid, but it isn’t – this is an unanticipated event.

When used for flow-control, exceptions are like goto. There might be a few esoteric cases in which they’re appropriate, but 99.99% of the time they are not.

Handle exceptions at the level that knows how to handle them

By their very nature, exceptions can propagate up a hierarchy and be caught at multiple levels. A questions rises – where is it appropriate to catch and handle an exception ?

The best place is that piece of code that can handle the exception. For some exceptions, like programming errors (e.g. IndexError, TypeError, NameError etc.) exceptions are best left to the programmer / user, because “handling” them will just hide real bugs.

If you have a complete application, it is not appropriate to just fail with an exception, of course. It’s better to demonstrate a polite error message (or dialog) and thoroughly log the exception itself, which can later help in the investigation of the bug (kind of like the dialog Microsoft replaced BSODs with – you get a polite message and may send a detailed report to Microsoft).

So before writing try/except, always ask yourself – “is this the right place to handle this exception ? do I have enough information to handle it here ?”.

This is also the reason why you should be extremely careful with except: clauses that catch everything. These will not only catch the exceptions you intended, but all of them.

Do not expose implementation details with exceptions

As I noted above, exceptions ride the express train up your implementation hierarchy. As such, they are prone to break encapsulation and expose implementation details. An example:

Suppose you implement an object that manages an internal cache, using a file. An error may occur that that the object can’t open its cache file for some reason. The underlying open function will throw IOError. Where should you catch it ?

Do not leave it to the user of the object. This breaks encapsulation, since the user should not be aware that you’re using a file at all. Maybe in the next version you’ll use a database, or a web connection ?

Rather, you should catch the exception in the object, wrap it with your own custom exception (say, CachingFailedError), and trying to preserve as much information as possible, propagate it up. The user of the object will then only have to catch your exception, and won’t have to modify his code when you modify the underlying caching implementation. However, if during testing the user will want to investigate the source of the error, he will have all the information he needs to do so.

Re-raising exceptions correctly is difficult3. Unfortunately there’s no one-fit-all recipe – can you recover from the error ? can you provide extra informatin ? do you need the original exception ? do you need its traceback ?

If all you need is to re-raise an exception with a more appropriate type, retaining the information stored in the old exception, do:

class StuffCachingError(Exception): pass

def do_stuff():
        cache = open(filename)
        # do stuff with cache
    except IOError, e:
        raise StuffCachingError('Caching error: %s' % e)


  1. You didn’t expose the internal implementation to outside code. Now users of do_stuff can install handlers for StuffCachingError, without knowing you use files under the hood.
  2. If the user wants to investigate the error in depth, you’ve also provided the original exception in the error message, and he can do so.

If you think you need to retain the original traceback as well (IMHO is most cases you don’t have to), write:

class StuffCachingError(Exception): pass

def do_stuff():
        cache = open('z.pyg')
        # do stuff with cache
    except IOError:
        exc_class, exc, traceback = sys.exc_info()
        my_exc = StuffCachingError('Caching error: %s' % exc)
        raise my_exc.__class__, my_exc, traceback

Now StuffCachingError will be raised as before, but with the original traceback leading to the call to open.

Document the exceptions thrown by your code

A useful idiom when thinking about programs is in terms of contracts4. Put simply, each function/method has a contract with the outside world:

  1. What it expects from its argument or the state of the world when invoked
  2. The result(s) it returns
  3. Its side effects on the world
  4. The exceptions it throws

For some reason, it is much more common to explicitly document the first three terms, but not the last one. I think this is wrong, and exceptions should be documented as well.

Don’t overdo it, of course. Writing “this code can throw NameError because I might have made a coding mistake” is superfluous at best. But thinking about which exceptions your code may throw will help you write better, safer and more encapsulated code. Maybe some exceptions must be hidden from the outside world (just like in the earlier tip on encapsulation). The user of your code must know which handlers to prepare, and which exceptions to propagate forward.

1 Chapter 6 of “Python in a Nutshell” by Alex Martelli

2 Ned Batchelder proposes this example (C/C++):

Using error codes:

STATUS DoSomething(int a, int b)
    STATUS st;
    st = DoThing1(a);
    if (st != SGOOD) return st;
    st = DoThing2(b);
    if (st != SGOOD) return st;
    return SGOOD;

Using exceptions:

void DoSomething(int a, int b)

3 Ian Bicking treats the topic with some depth, but I’ll be glad to get more references.

4 See Dan Weinreib’s insightful treatment of the subject of contracts and exceptions here.

Related posts:

  1. Using goto for error handling in C
  2. Handling out-of-memory conditions in C
  3. handling spam
  4. exceptions vs. error codes
  5. Safely using destructors in Python

10 Responses to “Robust exception handling”

  1. ripperNo Gravatar Says:

    How do you stand on enforced exception specification, like Java’s for example? Every method that throws an exception has to declare this fact, and the types of exceptions thrown as part of its signature.

    I haven’t used java intensively, but I find myself quite comfortable without this forced declaration policy. The downside is that if you add some exception time at a low level of the call stack, you either have to propagate the exception deceleration “all the way up”, or mask the exception, and I don’t like both options.

  2. elibenNo Gravatar Says:

    Not familiar with Java, but you can specify which exceptions are thrown by a function/method in C++. On one hand, it’s nice C++ allows this. On the other, I don’t like the masking that happens when an unspecified exception is thrown either.
    I don’t think this has place in dynamic languages like Python, where exceptions might represent runtime-”compile” errors, such as accessing undeclared variables.

  3. ripper234No Gravatar Says:

    In Java, you cannot throw an exception you did not declare. C++’s exception declaration is no more than documentation, because it is not enforced at compile time and leads to dangerous behavior if some other exception is thrown (If I remember correctly something really bad happens like abort(), but I could be wrong).

    I hate runtime-compile errors; I seriously do not understand why people tolerate languages that permit them.

  4. ripper234No Gravatar Says:

    (Correction – “no more” –> “worse” – I wrote that it does change code behavior, just not for the better).

  5. risomtNo Gravatar Says:

    This isn’t directly on topic but related: I’ve found the following snippet incredibly useful in python:

    a = ”
    except Exception, e:
    print ‘Error: ‘, e
    print ‘Error Reason: ‘, e.__doc__
    print ‘Exception: ‘, e.__class__

    Yes – it’s catching all of the errors, but it’s not blindly doing so. It’s great for running shaky test code or when dealing with a very large number of unpredictable variables (a website, for example). All it needs is a way to capture the line # :)

  6. Brisbane Website DesignerNo Gravatar Says:

    Consider implementing a site-wide error handler. Its amazing what you start learning when you start dumping your errors to disk and spend the time to read them and fix your code up. Chances are you will whittle down your exception count significantly in very short order.

  7. DarrenNo Gravatar Says:

    I’m a newbie when it comes to programming in python but the concept of site wide handling mentioned above is quite interesting. Any chances of giving an example of how the coding is done?

    Btw, thanks Eli for such a comprehensive article on exception handling. Really helpful!

  8. DanNo Gravatar Says:

    Your second code section has a syntax error
    it is missing the ‘:’ at the end of this line
    except IOError, e

  9. elibenNo Gravatar Says:

    @Dan: fixed, thanks

  10. SameeNo Gravatar Says:

    I’m new to Python. Can anyone shear the sample Exception handling project?