Python insight: beware of mutable default values for arguments

January 16th, 2009 at 10:10 am

I’ve just added a new gotcha to my Python insights page, and I want to reproduce it here because IMHO it’s very important:

This may surprise you:

>>> class Foo(object):
...     def __init__(self, name='', stuff=[]):
...         self.name = name
...         self.stuff = stuff
...
...     def add_stuff(self, gadget):
...         self.stuff.append(gadget)
...
>>> f = Foo()
>>> f.add_stuff('tree')
>>> f.stuff
['tree']
>>> g = Foo()
>>> g.stuff
['tree']

Where did this tree come from in g??

This is something that confuses a lot of Python programmers, sometimes even the experienced ones. Almost every newbie is hit by this gotcha at one stage or another (unless he diligently read about it in someone else’s blog, book or Python recipe). What happens here is that default values for arguments are created by Python only once per each function/method, at the time of its definition.

We can easily verify it by checking the addresses:

>>> object.__repr__(f.stuff)
'<list object at 0x01B35828>'
>>> object.__repr__(g.stuff)
'<list object at 0x01B35828>'

So how can one do it correctly? One solution is avoid using mutable default values for arguments. But this is hardly satisfactory, as from time to time a new list is a useful default. There are some complex solutions like defining a decorator for functions that deep-copies all arguments. This is an overkill, and the problem can be solved easily as follows:

>>> class Foo(object):
...     def __init__(self, name='', stuff=[]):
...         self.name = name
...         self.stuff = stuff or []
...
...     def add_stuff(self, gadget):
...         self.stuff.append(gadget)
...
>>> f = Foo()
>>> f.add_stuff('tree')
>>> g = Foo()
>>> g.stuff
[]
>>> f.stuff
['tree']

The stuff or [] code does the trick, as the [] in it always creates a fresh new list when the empty list (that is, the default argument) is passed in.

Note that the or operator is quite nondiscriminatory when it comes to booleans, so a lot of values you may consider valid (empty strings, 0, etc.) will be thought of as False. But for most cases this will work just fine (why would anyone pass an empty string instead of a list?)

However, if you’re still concerned of the tricky corner cases, this solution may be more robust (though less simple, which is a disadvantage):

>>> class Foo(object):
...     def __init__(self, name='', stuff=None):
...         self.name = name
...         if stuff is None: stuff = []
...         self.stuff = stuff
...
...     def add_stuff(self, gadget):
...         self.stuff.append(gadget)

Related posts:

  1. Problem passing arguments to Python scripts on Windows
  2. Passing extra arguments to PyQt slots
  3. Passing extra arguments to Qt slots
  4. Weighted random generation in Python

10 Responses to “Python insight: beware of mutable default values for arguments”

  1. tshirtmanNo Gravatar Says:

    Hu ho… I have a whole project to check for that! Thanks for the insight and explications!

  2. Hans LNo Gravatar Says:

    Hey, I don’t understand what the point of the default list value is in this example:

    >>> class Foo(object):
    … def __init__(self, name=”, stuff=[]):
    … self.name = name
    … self.stuff = stuff or []

    Why not have the default value be None there? The code would otherwise be the same, right?

    Personally, I think the last solution is much cleaner, because Python programmers *should* know that default mutable objects are a bad idea. … So any code that has stuff=[] in the function signature *should* look wrong (or at least dangerous).

    Hans

  3. NickNo Gravatar Says:

    FWIW, this particular problem bedeviled me for 2 hours before I figured it out by trying a large number of test cases. It’s nice to know I am not the only one! Thanks for the interesting writings….

  4. elibenNo Gravatar Says:

    @Hans L

    The point is, if the user supplied [], he will get it, and a fresh [] object at that.

  5. LucianNo Gravatar Says:

    if stuff is None: stuff = []
    … self.stuff = stuff

    I think you could also write that on one line as

    self.stuff = stuff if stuff is not None else []

  6. Paul HildebrandtNo Gravatar Says:

    Be aware that pylint will warn you about this gotcha. In fact it was pylint that informed me that this was a problem when it found it in my code.

  7. nesNo Gravatar Says:

    You have to remember that mutable variables are not copied so even in your current version you can do this:

    x=['car']
    >>> f=Foo(stuff=x)
    f.add_stuff(‘tree’)
    >>> x
    ['car', 'tree']

    you might want to use this, depending on what you are doing:

    self.stuff=stuff[:]

  8. AliNo Gravatar Says:

    The problem with “stuff = stuff or []” is the following inconsistent behavior:

    x = []
    f = Foo(stuff=x)
    f.add_stuff(‘bar’)
    print x

    y = ['baz']
    f = Foo(stuff=y)
    f.add_stuff(‘quux’)
    print y

    The “if stuff is None…” fixes this. Of course, nesSays’ recommendation may be preferable since you may not want to modify the caller’s variables at all.

  9. elibenNo Gravatar Says:

    @nes and @Ali,

    Thanks for these clarifications. Indeed, one has to decide if it’s important to receive the user’s own object or a copy of it, and this depends on the actual semantics of your code. Naturally, modifying the user’s data is not very robust, though from time to time it is actually valid.

  10. Bill HayesNo Gravatar Says:

    For me, the key understanding is that the code in the __init__() is re-run for each new instance,
    but the first line of the the class definition is NOT re-run, so stuff=[] is not re-run and hence [] is not newly created.