Eli Bendersky's website - Python internals

The scope of index variables in Python's for loops

2015-01-17T06:24:00-08:00

I'll start with a quiz. What does this function do?

def foo(lst):
    a = 0
    for i in lst:
        a += i
    b = 1
    for t in lst:
        b *= i
    return a, b

If you think "computes the sum and product of the items in lst", don't feel too bad about yourself. The bug here is often tricky to spot. If you did see it, well done - but buried in mountains of real code, and when you don't know it's a quiz, discovering the bug is significantly more difficult.

The bug here is due to using i instead of t in the body of the second for loop. But wait, how does this even work? Shouldn't i be invisible outside of the first loop? [1] Well, no. In fact, Python formally acknowledges that the names defined as for loop targets (a more formally rigorous name for "index variables") leak into the enclosing function scope. So this:

for i in [1, 2, 3]:
    pass
print(i)

Is valid and prints 3, by design. In this writeup I want to explore why this is so, why it's unlikely to change, and also use it as a tracer bullet to dig into some interesting parts of the CPython compiler.

And by the way, if you're not convinced this behavior can cause real problems, consider this snippet:

def foo():
    lst = []
    for i in range(4):
        lst.append(lambda: i)
    print([f() for f in lst])

If you'd expect this to print [0, 1, 2, 3], no such luck. This code will, instead, emit [3, 3, 3, 3], because there's just a single i in the scope of foo, and this is what all the lambdas capture.

The official word

The Python reference documentation explicitly documents this behavior in the section on for loops:

The for-loop makes assignments to the variables(s) in the target list. [...] Names in the target list are not deleted when the loop is finished, but if the sequence is empty, they will not have been assigned to at all by the loop.

Note the last sentence - let's try it:

for i in []:
    pass
print(i)

Indeed, a NameError is raised. Later on, we'll see that this is a natural outcome of the way the Python VM executes its bytecode.

Why this is so

I actually asked Guido van Rossum about this behavior and he was gracious enough to reply with some historical background (thanks Guido!). The motivation is keeping Python's simple approach to names and scopes without resorting to hacks (such as deleting all the values defined in the loop after it's done - think about the complications with exceptions, etc.) or more complex scoping rules.

In Python, the scoping rules are fairly simple and elegant: a block is either a module, a function body or a class body. Within a function body, names are visible from the point of their definition to the end of the block (including nested blocks such as nested functions). That's for local names, of course; global names (and other nonlocal names) have slightly different rules, but that's not pertinent to our discussion.

The important point here is: the innermost possible scope is a function body. Not a for loop body. Not a with block body. Python does not have nested lexical scopes below the level of a function, unlike some other languages (C and its progeny, for example).

So if you just go about implementing Python, this behavior is what you'll likely to end with. Here's another enlightening snippet:

for i in range(4):
    d = i * 2
print(d)

Would it surprise you to find out that d is visible and accessible after the for loop is finished? No, this is just the way Python works. So why would the index variable be treated any differently?

By the way, the index variables of list comprehensions are also leaked to the enclosing scope. Or, to be precise, were leaked, before Python 3 came along.

Python 3 fixed the leakage from list comprehensions, along with other breaking changes. Make no mistake, changing such behavior is a major breakage in backwards compatibility. This is why I think the current behavior stuck and won't be changed.

Moreover, many folks still find this a useful feature of Python. Consider:

for i, item in enumerate(somegenerator()):
    dostuffwith(i, item)
print('The loop executed {0} times!'.format(i+1))

If you have no idea how many items somegenerator actually returned, this is a pretty succinct way to know. Otherwise you'd have to keep a separate counter.

Here's another example:

for i in somegenerator():
    if isinteresing(i):
        break
dostuffwith(i)

Which is a useful pattern for finding things in a loop and using them afterwards [2].

There are other uses people came up with over the years that justify keeping this behavior in place. It's hard enough to instill breaking changes for features the core developers deem detrimental and harmful. When the feature is argued by many to be useful, and moreover is used in a huge bunch of code in the real world, the chances of removing it are zero.

Under the hood

Now the fun part. Let's see how the Python compiler and VM conspire to make this behavior possible. In this particular case, I think the most lucid way to present things is going backwards from the bytecode. I hope this may also serve as an interesting example on how to go about digging in Python's internals [3] in order to find stuff out (it's so much fun, seriously!)

Let's take a part of the function presented at the start of this article and disassemble it:

def foo(lst):
    a = 0
    for i in lst:
        a += i
    return a

The resulting bytecode is:

 0 LOAD_CONST               1 (0)
 3 STORE_FAST               1 (a)

 6 SETUP_LOOP              24 (to 33)
 9 LOAD_FAST                0 (lst)
12 GET_ITER
13 FOR_ITER                16 (to 32)
16 STORE_FAST               2 (i)

19 LOAD_FAST                1 (a)
22 LOAD_FAST                2 (i)
25 INPLACE_ADD
26 STORE_FAST               1 (a)
29 JUMP_ABSOLUTE           13
32 POP_BLOCK

33 LOAD_FAST                1 (a)
36 RETURN_VALUE

As a reminder, LOAD_FAST and STORE_FAST are the opcodes Python uses to access names that are only used within a function. Since the Python compiler knows statically (at compile-time) how many such names exist in each function, they can be accessed with static array offsets as opposed to a hash table, which makes access significanly faster (hence the _FAST suffix). But I digress. What's really important here is that a and i are treated identically. They are both fetched with LOAD_FAST and modified with STORE_FAST. There is absolutely no reason to assume that their visibility is in any way different [4].

So how did this come to be? Somehow, the compiler figured that i is just another local name within foo. This logic lives in the symbol table code, when the compiler walks over the AST to create a control-flow graph from which bytecode is later emitted; there are more details about this process in my article about symbol tables - so I'll just stick to the essentials here.

The symtable code doesn't treat for statements very specially. In symtable_visit_stmt we have:

case For_kind:
    VISIT(st, expr, s->v.For.target);
    VISIT(st, expr, s->v.For.iter);
    VISIT_SEQ(st, stmt, s->v.For.body);
    if (s->v.For.orelse)
        VISIT_SEQ(st, stmt, s->v.For.orelse);
    break;

The loop target is visited as any other expression. Since this code visits the AST, it's worthwhile to dump it to see how the node for the for statement looks:

For(target=Name(id='i', ctx=Store()),
    iter=Name(id='lst', ctx=Load()),
    body=[AugAssign(target=Name(id='a', ctx=Store()),
                    op=Add(),
                    value=Name(id='i', ctx=Load()))],
    orelse=[])

So i lives in a Name node. These are handled in the symbol table code by the following clause in symtable_visit_expr:

case Name_kind:
    if (!symtable_add_def(st, e->v.Name.id,
                          e->v.Name.ctx == Load ? USE : DEF_LOCAL))
        VISIT_QUIT(st, 0);
    /* ... */

Since the name i is clearly tagged with DEF_LOCAL (because of the *_FAST opcodes emitted to access it, but this is also easy to observe if the symbol table is dumped using the symtable module), the code above evidently calls symtable_add_def with DEF_LOCAL as the third argument. This is the right time to glance at the AST above and notice the ctx=Store part of the Name node of i. So it's the AST that already comes in carrying the information that i is stored to in the target part of the For node. Let's see how that comes to be.

The AST-building part of the compiler goes over the parse tree (which is a fairly low-level hierarchical representation of the source code - some background is available here) and, among other things, sets the expr_context attributes on some nodes, most notably Name nodes. Think about it this way, in the following statement:

foo = bar + 1

Both foo and bar are going to end up in Name nodes. But while bar is only being loaded from, foo is actually being stored into in this code. The expr_context attribute is used to distinguish between uses for later consumption by the symbol table code [5].

Back to our for loop targets, though. These are handled in the function that creates an AST for for statements - ast_for_for_stmt. Here are the relevant parts of this function:

static stmt_ty
ast_for_for_stmt(struct compiling *c, const node *n)
{
    asdl_seq *_target, *seq = NULL, *suite_seq;
    expr_ty expression;
    expr_ty target, first;

    /* ... */

    node_target = CHILD(n, 1);
    _target = ast_for_exprlist(c, node_target, Store);
    if (!_target)
        return NULL;
    /* Check the # of children rather than the length of _target, since
       for x, in ... has 1 element in _target, but still requires a Tuple. */
    first = (expr_ty)asdl_seq_GET(_target, 0);
    if (NCH(node_target) == 1)
        target = first;
    else
        target = Tuple(_target, Store, first->lineno, first->col_offset, c->c_arena);

    /* ... */

    return For(target, expression, suite_seq, seq, LINENO(n), n->n_col_offset,
               c->c_arena);
}

The Store context is created in the call to ast_for_exprlist, which creates the node for the target (recall the the for loop target may be a sequence of names for tuple unpacking, not just a single name).

This function is probably the most important part in the process of explaining why for loop targets are treated similarly to other names within the loop. After this tagging happens in the AST, the code for handling such names in the symbol table and VM is no different from other names.

Wrapping up

This article discusses a particular behavior of Python that may be considered a "gotcha" by some. I hope the article does a decent job of explaining how this behavior flows naturally from the naming and scoping semantics of Python, why it can be useful and hence is unlikely to ever change, and how the internals of the Python compiler make it work under the hood. Thanks for reading!

[1]	Here I'm tempted to make a Microsoft Visual C++ 6 joke, but the fact that most readers of this blog in 2015 won't get it is somewhat disturbing (because it reflects my age, not the abilities of my readers).

[2] You could argue that dowithstuff(i) could go into the if right before the break here. But this isn't always convenient. Besides, according to Guido there's a nice separation of concerns here - the loop is used for searching, and only that. What happens with the value after the search is done is not the loop's concern. I think this is a very good point.

[3]	As usual for my articles on Python's internals, this is about Python 3. Specifically, I'm looking at the `default` branch of the Python repository, where work on the next release (3.5) is being done. But for this particular topic, the source code of any release in the 3.x series should do.

[4] Another thing clear from the disassembly is why i remains invisible if the loop doesn't execute. The GET_ITER and FOR_ITER pair of opcodes treat the thing we loop over as an iterator and then call its __next__ method. If that call ends up raising StopIteration, the VM catches it and exits the loop. Only if an actual value is returned does the VM proceed to execute STORE_FAST to i, thus bringing it into existence for subsequent code to refer to.

[5]	It's a curious design, which I suspect stems from the desire for relatively clean recursive visitation code in AST consumers such as the symbol table code and CFG generation.

Using ASDL to describe ASTs in compilers

2014-06-04T06:25:55-07:00

ASTs (Abstract Syntax Trees) are an important data structure in compiler front-ends. If you've written a few parsers, you almost definitely ran into the need to describe the result of the parsing in terms of an AST. While the kinds of nodes such ASTs have and their structure is very specific to the source language, many commonalities come up. In other words, coding "yet another AST" gets really old after you've done it a few times.

Worry not, as you'd expect from the programmer crowd, this problem was "solved" by adding another level of abstraction. Yes, an abstraction over Abstract Syntax Trees, oh my! The abstraction here is some textual format (let's call it a DSL to sound smart) that describes what the AST looks like, along with machinery to auto-generate the code that implements this AST.

Most solutions in this domain are ad-hoc, but one that I've seen used more than once is ASDL - Abstract Syntax Definition Language. The self-description from the website sounds about right:

The Zephyr Abstract Syntax Description Lanuguage (ASDL) is a language designed to describe the tree-like data structures in compilers. Its main goal is to provide a method for compiler components written in different languages to interoperate. ASDL makes it easier for applications written in a variety of programming languages to communicate complex recursive data structures.

To give an example, here's a short snippet from an ASDL definition of a simple programming language:

program = Program(class* classes)

class = Class(identifier name, identifier? parent, feature* features)

[...]

expression = Assign(identifier name, expression expr)
           | StaticDispatch(expression expr, identifier type_name,
                            identifier name, expression* actual)
           | Dispatch(expression expr, identifier name, expression* actual)

[...]

The way to read this is: a program node consists of one or more classes. Each class has these children: a name which is an identifier, an optional parent which is also an identifier, and a (potentially empty) list of features, each of which is a feature node. And so on.

The full details are available in the paper "The Zephyr Abstract Syntax Definition Language" by Wang et.al. Unfortunately, a link to this paper isn't always trivial to find, so I have a PDF copy in the docs directory of my asdl_parser project, which I'm going to discuss soon.

Type safety in ASDL

In addition to providing a concise description of nodes from which code (in many languages) can be generated automatically, I like ASDL for another reason. It provides some type safety when constructing the AST in the parser.

Take the snippet above, for example. A program has the classes attribute, which is a (potentially empty) sequence of class nodes. Each such class has to be a Class, which is precisely defined. It can be nothing else. The expression below that shows it differently - an expression can be either a Assign, StaticDispatch, etc.

The set of possibilities is statically defined. This makes it possible to insert some degree of static checking into the (auto-generated) AST construction code. So the constructed AST can't be completely bogus even before semantic analysis is applied. Even though I love Python, I do appreciate a bit of static type checking in the right places. Key data structures like ASTs are, I believe, one of the places when such type checking makes sense.

ASDL in upstream CPython

Starting with Python 2.5, the CPython compiler (the part responsible for emitting bytecode from Python source) uses an ASDL description to create an AST for Python source. The AST is created by the parser (from the parse tree - more details in PEP 339), and is then used to create the control-flow graph, from which bytecode is emitted.

The ASDL description lives in Parser/Python.asdl in the CPython source tree. Parser/asdl_c.py is a script that runs whenever someone modifies this ASDL description. It uses the Parser/asdl.py module to parse the ASDL file into an internal form and then emits C code that describes the ASTs. This C code lives in Include/Python-ast.h and Python/Python-ast.c [1].

This may be more details than you wanted to hear :-) The gist of it, however, is - CPython's ASTs are described in ASDL, and if you want a quick glance of how these ASTs look, Parser/Python.asdl is the file to look at.

My rewrite of the ASDL parser

Up until very recently, the ASDL description of the CPython AST was parsed by a tool that relied on the SPARK parsing toolkit. In fact, Parser/spark.py was carried around in the distribution just for this purpose.

A few months ago I was looking for something to conveniently implement the AST for a toy compiler I was hacking on. Being a CPython developer, ASDL immediately sprang to mind, but I was reluctant to carry the SPARK dependency and/or learn how to use it. The ASDL language seemed simple enough to not require such machinery. Surely a simple recursive-descent parser would do. So I implemented my own stand-alone parser for ASDL, using modern Python 3.x - and it's available in a public Github repository right here. Feel free to use it, and let me know how it goes!

Since my parser turned out to be much simpler and easier to grok than upstream CPython's SPARK-based parser, I proposed to replace it in Issue 19655. After some delays (caused mainly by waiting for 3.4 release and then getting distracted by other stuff), the change landed in the default branch (on its way to 3.5) about a month ago. The result is pleasing - the new parser is shorter, doesn't require the SPARK dependency (which was now dropped), has tests and is much more maintainable.

In the interest of not changing too much at once, I left the interface to the C code generator (Parser/asdl_c.py) the same, so there is absolutely no difference in the produced C code. Some time in the future it may make sense to revise this decision. The C generator is also fairly old code that could use some modernization and tests.

Historical note - AST description in pycparser

I first ran into this problem (high-level description of ASTs) when I was working on pycparser (which is quite an old project by now).

Back at the time, I looked at the compiler module of Python 2.x and liked its approach of simple textual description of the AST which is then parsed and from which the code for AST nodes is emitted. The compiler module was a maintenance headache (because it duplicated a lot of the AST logic from the actual compiler) and is gone in Python 3.x, replaced by the ast module which provides access to the same C-based AST generated by Parser/asdl_c.py as is used by the CPython compiler.

pycparser's AST description is a simple textual file that's very similar in spirit to ASDL. If I were to do this today, I'd probably also pick ASDL since it's more "standard", as well as for the extra type safety guarantees it provides.

[1]

Even though these files are auto-generated, they are also checked into the CPython Mercurial repository. This is because we don't want people building Python from source to depend on the tools required to generate such files. Only core CPython developers who want to play with the internals need them.

Faster XML iteration with ElementTree

2012-06-17T05:28:44-07:00

As I've mentioned previously, starting with Python 3.3 the C accelerator of the xml.etree.ElementTree module is going to be imported by default. This should make quite a bit of code faster for those who were not aware of the existence of the accelerator, and reduce the amount of boilerplate importing for everyone.

As Python 3.3 is nearing its first beta, more work was done in the past few weeks; mostly fixing all kinds of problems that arose from the aforementioned transition. But in this post I want to focus on one feature that was added this weekend - much faster iteration over the parsed XML tree.

ElementTree offers a few tools for iterating over the tree and for finding interesting elements in it, but the basis for them all is the iter method:

Creates a tree iterator with the current element as the root. The iterator iterates over this element and all elements below it, in document (depth first) order. If tag is not None or '*', only elements whose tag equals tag are returned from the iterator.

And until very recently, this iter was implemented in Python, even when the C accelerator was loaded. This was achieved by calling PyRun_String on a "bootstrap" string defining the method (as well as a bunch of other Python code), when the C extension module was being initialized. In the past few months I've been slowly and surely decimating this bootstrap code, trying to move as much functionality as possible into the C code and replacing stuff with actual C API calls. The last bastion was iter (and its cousin itertext) because its implementation in C is not trivial.

Well, that last bastion has now fallen and the C accelerator of ElementTree no longer has any Python bootstrap code - iter is actually implemented in C. And the great "side effect" of this is that the iter method (and all the other methods that rely on it, like find, iterfind and others) is now much faster. On a relatively large XML document I timed a 10x speed boost for simple iteration looking for a specific tag. I hope that this will make a lot of XML processing code in Python much faster out-of-the-box.

This change is already in Python trunk and will be part of the 3.3 release. I must admit that I didn't spend much time optimizing the C code implementing iter, so there may still be an area for improvement. I have a hunch that it can be made a few 10s of percents faster with a bit of effort. If you're interested to help, drop me a line and I will be happy to discuss it.

Under the hood of Python class definitions

2012-06-15T05:51:41-07:00

This is a fast-paced walk-through of the internals of defining new classes in Python. It shows what actually happens inside the Python interpreter when a new class definition is encountered and processed. Beware, this is advanced material. If the prospect of pondering the metaclass of the metaclass of your class makes you feel nauseated, you better stop now.

The focus is on the official (CPython) implementation of Python 3. For modern releases of Python 2 the concepts are similar, although there will be some slight differences in the details.

On the bytecode level

I'll start right with the bytecode, ignoring all the good work done by the Python compiler [1]. For simplicity, this function will be used to demonstrate the bytecode generated by a class definition, since it's easy to disassemble functions:

def myfunc():
    class Joe:
        attr = 100.02
        def foo(self):
            return 2

Disassembling myfunc will show us the steps needed to define a new class:

>>> dis.disassemble(myfunc.__code__)
 14           0 LOAD_BUILD_CLASS
              1 LOAD_CONST               1 (<code object Joe at 0x7fe226335b80, file "disassemble.py", line 14>)
              4 LOAD_CONST               2 ('Joe')
              7 MAKE_FUNCTION            0
             10 LOAD_CONST               2 ('Joe')
             13 CALL_FUNCTION            2
             16 STORE_FAST               0 (Joe)
             19 LOAD_CONST               0 (None)
             22 RETURN_VALUE

The number immediately preceding the instruction name is its offset in the binary representation of the code object. All the instructions until and including the one at offset 16 are for defining the class. The last two instructions are for myfunc to return None.

Let's go through them, step by step. Documentation of the Python bytecode instructions is available in the dis module.

LOAD_BUILD_CLASS is a special instruction used for creating classes. It pushes the function builtins.__build_class__ onto the stack. We'll examine this function in much detail later.

Next, a code object, followed by a name (Joe) are pushed onto the stack as well. The code object is interesting, let's peek inside:

>>> dis.disassemble(myfunc.__code__.co_consts[1])
 14           0 LOAD_FAST                0 (__locals__)
              3 STORE_LOCALS
              4 LOAD_NAME                0 (__name__)
              7 STORE_NAME               1 (__module__)
             10 LOAD_CONST               0 ('myfunc.<locals>.Joe')
             13 STORE_NAME               2 (__qualname__)

 15          16 LOAD_CONST               1 (100.02)
             19 STORE_NAME               3 (attr)

 16          22 LOAD_CONST               2 (<code object foo at 0x7fe226335c40, file "disassemble.py", line 16>)
             25 LOAD_CONST               3 ('myfunc.<locals>.Joe.foo')
             28 MAKE_FUNCTION            0
             31 STORE_NAME               4 (foo)
             34 LOAD_CONST               4 (None)
             37 RETURN_VALUE

This code defines the innards of the class. Some generic bookkeeping, followed by definitions for the attr attribute and foo method.

Now let's get back to the first disassembly. The next instruction (at offset 7) is MAKE_FUNCTION [2]. This instruction pulls two things from the stack - a name and a code object. So in our case, it gets the name Joe and the code object we saw disassembled above. It creates a function with the given name and the code object as its code and pushes it back to the stack.

This is followed by once again pushing the name Joe onto the stack. Here's what the stack looks like now (TOS means "top of stack"):

TOS> name "Joe"
     function "Joe" with code for defining the class
     function builtins.__build_class__
     -----------------------------------------------

At this point (offset 13), CALL_FUNCTION 2 is executed. The 2 simply means that the function was passed two positional arguments (and no keyword arguments). CALL_FUNCTION first takes the arguments from the stack (the rightmost on top), and then the function itself. So the call is equivalent to:

builtins.__build_class__(function defining "Joe", "Joe")

Build me a class, please

A quick peek into the builtins module in Python/bltinmodule.c reveals that __build_class__ is implemented by the function builtin___build_class__ (I'll call it BBC for simplicity) in the same file.

As any Python function, BBC accepts both positional and keyword arguments. The positional arguments are:

func, name, base1, base2, ... baseN

So we see only the function and name were passed for Joe, since it has no base classes. The only keyword argument BBC understands is metaclass [3], allowing the Python 3 way of defining metaclasses:

class SomeOtherJoe(metaclass=JoeMeta):
  [...]

So back to BBC, here's what it does [4]:

The first chunk of code deals with extracting the arguments and setting defaults.
Next, if no metaclass is supplied, BBC looks at the base classes and takes the metaclass of the first base class. If there are no base classes, the default metaclass type is used.
If the metaclass is really a class (note that in Python any callable can be given as a metaclass), look at the bases again to determine "the most derived" metaclass.

The last point deserves a bit of elaboration. If our class has bases, then some rules apply for the metaclasses that are allowed. The metaclasses of its bases must be either subclasses or superclasses of our class's metaclass. Any other arrangement will result in this TypeError:

metaclass conflict: the metaclass of a derived class must be a (non-strict)
                    subclass of the metaclasses of all its bases

Eventually, given that there are no conflicts, the most derived metaclass will be chosen. The most derived metaclass is the one which is a subtype of the explicitly specified metaclass and the metaclasses of all the base classes. In other words, if our class's metaclass is Meta1, only one of the bases has a metaclass and that's Meta2, and Meta2 is a subclass of Meta1, it is Meta2 that will be picked to serve as the eventual metaclass of our class.

At this point BBC has a metaclass [5], so it starts by calling its __prepare__ method to create a namespace dictionary for the class. If there's no such method, an empty dict is used.

As documented in the data model reference:

If the metaclass has a __prepare__() attribute (usually implemented as a class or static method), it is called before the class body is evaluated with the name of the class and a tuple of its bases for arguments. It should return an object that supports the mapping interface that will be used to store the namespace of the class. The default is a plain dictionary. This could be used, for example, to keep track of the order that class attributes are declared in by returning an ordered dictionary.

The function argument is invoked, passing the namespace dict as the only argument. If we look back at the disassembly of this function (the second one), we see that the first argument is placed into the f_locals attribute of the frame (with the STORE_LOCALS instruction). In other words, this dictionary is then used to populate the class attributes. The function itself returns None - its outcome is modifying the namespace dictionary.
Finally, the metaclass is called with the name, list of bases and namespace dictionary as arguments.

The last step defers to the metaclass to actually create a new class with the given definition. Recall that when some class MyKlass has a metaclass MyMeta, then the class definition of MyKlass is equivalent to [6]:

MyKlass = MyMeta(name, bases, namespace_dict)

The flow of BBC outlined above directly embodies this equivalence.

So what happens next? Well, the metaclass MyMeta is a class, right? And what happens when a class is "called"? It's instantiated. How is a class's instantiation done? By invoking its metaclass's __call__. So wait, this is the metaclass's metaclass we're talking about here, right? Yes! A metaclass is just a class, after all [7], and has a metaclass of its own - so Python has to keep the meta-flow going.

Realistically, what probably happens is this:

Most chances are that your class has no metaclass specified explicitly. Then, its default metaclass is type, so the call above is actually:

MyKlass = type(name, bases, namespace_dict)

The metaclass of type happens to be type itself, so here type.__call__ is called.

In the more complex case that your class does have a metaclass, most chances are that the metaclass itself has no metaclass [8], so type is used for it. Therefore, the MyMeta(...) call is also served by type.__call__.

type_call

In Objects/typeobject.c, the type.__call__ slot is getting mapped to the function type_call. I've already spent some time explaining how it works, so it's important to review that article at this point.

Things are a bit different here, however. The object creation sequence article explained how instances are created, so the tp_new slot called from type_call went to object. Here, since type_call will actually call tp_new on a metaclass, and the metaclass's base is type (see this diagram), we'll have to study how the type_new function (also from Objects/typeobject.c) works.

A brief recap

I feel that the flow here is relatively convoluted, so lest we lose focus, let's have a brief recap of how we got thus far. The following is a much simplified version of the flow described so far in this article:

When a new class Joe is defined...
The Python interpreter arranges the builtin function builtin__build_class__ (BBC) to be called, giving it the class name and its innards compiled into a code object.
BBC finds the metaclass of Joe and calls it to create the new class.
When any class in Python is called, it means that its metaclass's tp_call slot is invoked. So to create Joe, this is the tp_call of its metaclass's metaclass. In most cases this is the type_call function (since the metaclass's metaclass is almost always type, or something that eventually delegates to it).
type_call creates a new instance of the type it's bound to by calling its tp_new slot.
In our case, that is served by the type_new function.

The next section picks up from step 6.

type_new

The type_new function is a complex beast - it's over 400 lines long. There's a good reason for this, however, since it plays a very fundamental role in the Python object system. It's literally responsible for creating all Python types. I'll go over its functionality in major blocks, pasting short snippets of code where relevant.

Let's start at the beginning. The signature of type_new is:

static PyObject *
type_new(PyTypeObject *metatype, PyObject *args, PyObject *kwds)

When called to create our class Joe, the arguments will be:

metatype - the metaclass, so it's type itself.
args - we saw in the description of BBC above that this is the class name, list of base classes and a namespace dict.
kwds - since Joe has no metaclass, this will be empty.

At this point, it may be useful to recall that:

class Joe:
  ... contents

Is equivalent to:

Joe = type('joe', (), dict of contents)

type_new serves both approaches, of course.

It starts by handling the special 1-argument call of the type function, which returns the type. Then, it tries to see if the requested type has a metaclass that's more suitable than the one passed in. This is necessary to handle a direct call to type as shown above - if one of the bases has a metaclass, that metaclass should be used for the creation [9].

Next, type_new handles some special class methods (for example __slots__).

Finally, the type object itself is allocated and initialized. Since the unification of types and classes in Python, user-defined classes are represented similarly to built-in types inside the CPython VM. However, there's still a difference. Unlike built-in types (and new types exported by C extension) which are statically allocated and are essentially "singletons", user-defined classes have to be implemented by dynamically allocated type objects on the heap [10]. For this purpose, Include/object.h defines an "extended type object", PyHeapTypeObject. This struct starts with a PyTypeObject member, so it can be passed around to Python C code expecting any normal type. The extra information it carries is used mainly for book-keeping in the type-handling code (Objects/typeobject.c). PyHeapTypeObject is an interesting type to discuss but would deserve an article of its own, so I'll stop right here.

Just as an example of one of the special cases handled by type_new for members of new classes, let's look at __new__. The data model reference says about it:

Called to create a new instance of class cls. __new__() is a static method (special-cased so you need not declare it as such) that takes the class of which an instance was requested as its first argument.

It's interesting to see how this statement is embodied in the code of type_new:

/* Special-case __new__: if it's a plain function,
   make it a static function */
tmp = _PyDict_GetItemId(dict, &PyId___new__);
if (tmp != NULL && PyFunction_Check(tmp)) {
    tmp = PyStaticMethod_New(tmp);
    if (tmp == NULL)
        goto error;
    if (_PyDict_SetItemId(dict, &PyId___new__, tmp) < 0)
        goto error;
    Py_DECREF(tmp);
}

So when the dict of the new class has a __new__ method, it's automatically replaced with a corresponding static method.

After some more handling of special cases, type_new returns the object representing the newly created type.

Conclusion

This has been a relatively dense article. If you got lost, don't despair. The important part to remember is the flow described in "A brief recap" - the rest of the article just explains the items in that list in more detail.

The Python type system is very powerful, dynamic and flexible. Since this all has to be implemented in the low-level and type-rigid C, and at the same time be relatively efficient, the implementation is almost inevitably complex. If you're just writing Python code, you almost definitely don't have to be aware of all these details. However, if you're writing non-trivial C extensions, and/or hacking on CPython itself, understanding the contents of this article (at least on an approximate level) can be useful and educational.

Many thanks to Nick Coghlan for reviewing this article.

[1]	If you're interested in the compilation part, this article provides a good overview.

[2] In the distant past, MAKE_FUNCTION was used both for creating functions and classes. However, when lexical scoping was added to Python, a new instruction for creating functions was added - MAKE_CLOSURE. So nowadays, as strange as it sounds, MAKE_FUNCTION is only used for creating classes, not functions.

[3]	The other keyword arguments, if they exist, are passed to the metaclass when it's getting called.

[4]	You may find it educational to open the file `Python/bltinmodule.c` from the Python source distribution and follow along.

[5]	There always is some metaclass, because all classes eventually derive from `object` whose metaclass is `type`.

[6]	With the caveat that BBC also calls `__prepare__`. For a more equivalent sequence, take a look at types.new_class.

[7]	As I mentioned earlier, any callable can be specified as a metaclass. If the callable is a function and not a class, it's simply called as the last step of BBC - the rest of the discussion doesn't apply.

[8]	I've never encountered real-world Python code where a metaclass has a metaclass of its own. If you have, please let me know - I'm genuinely curious about the use cases for such a construct.

[9]	If you've noticed that this is a duplication of effort, you're right. BBC also computes the metaclass, but to handle the `type(...)` call, `type_new` has to do this again. I think that creating new classes is a rare enough occurrence that the extra work done here doesn't count for much.

[10]	Since they have to be garbage collected and fully deleted when no longer needed.

Python object creation sequence

2012-04-16T07:03:41-07:00

[The Python version described in this article is 3.x]

This article aims to explore the process of creating new objects in Python. As I explained in a previous article, object creation is just a special case of calling a callable. Consider this Python code:

class Joe:
    pass

j = Joe()

What happens when j = Joe() is executed? Python sees it as a call to the callable Joe, and routes it to the internal function PyObject_Call, with Joe passed as the first argument. PyObject_Call looks at the type of its first argument to extract its tp_call attribute.

Now, what is the type of Joe? Whenever we define a new Python class, unless we explicitly specify a metaclass for it, its type is type. Therefore, when PyObject_Call attempts to look at the type of Joe, it finds type and picks its tp_call attribute. In other words, the function type_call in Objects/typeobject.c is invoked [1].

This is an interesting function, and it's short, so I'll paste it wholly here:

static PyObject *
type_call(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    PyObject *obj;

    if (type->tp_new == NULL) {
        PyErr_Format(PyExc_TypeError,
                     "cannot create '%.100s' instances",
                     type->tp_name);
        return NULL;
    }

    obj = type->tp_new(type, args, kwds);
    if (obj != NULL) {
        /* Ugly exception: when the call was type(something),
           don't call tp_init on the result. */
        if (type == &PyType_Type &&
            PyTuple_Check(args) && PyTuple_GET_SIZE(args) == 1 &&
            (kwds == NULL ||
             (PyDict_Check(kwds) && PyDict_Size(kwds) == 0)))
            return obj;
        /* If the returned object is not an instance of type,
           it won't be initialized. */
        if (!PyType_IsSubtype(Py_TYPE(obj), type))
            return obj;
        type = Py_TYPE(obj);
        if (type->tp_init != NULL &&
            type->tp_init(obj, args, kwds) < 0) {
            Py_DECREF(obj);
            obj = NULL;
        }
    }
    return obj;
}

So what arguments is type_call being passed in our case? The first one is Joe itself - but how is it represented? Well, Joe is a class, so it's a type (all classes are types in Python 3). Types are represented inside the CPython VM by PyTypeObject objects [2].

What type_call does is first call the tp_new attribute of the given type. Then, it checks for a special case we can ignore for simplicity, makes sure tp_new returned an object of the expected type, and then calls tp_init. If an object of a different type was returned, it is not being initialized.

Translated to Python, what happens is this: if your class defines the __new__ special method, it gets called first when a new instance of the class is created. This method has to return some object. Usually, this will be of the required type, but this doesn't have to be the case. Objects of the required type get __init__ invoked on them. Here's an example:

class Joe:
    def __new__(cls, *args, **kwargs):
        obj = super(Joe, cls).__new__(cls)
        print('__new__ called. got new obj id=0x%x' % id(obj))
        return obj

    def __init__(self, arg):
        print('__init__ called (self=0x%x) with arg=%s' % (id(self), arg))
        self.arg = arg

j = Joe(12)
print(type(j))

This prints:

__new__ called. got new obj id=0x7f88e7218290
__init__ called (self=0x7f88e7218290) with arg=12
<class '__main__.Joe'>

Customizing the sequence

As we saw above, since the type of Joe is type, the type_call function is invoked to define the creation sequence for Joe instances. This sequence can be changed by specifying a custom type for Joe - in other words, a metaclass. Let's modify the previous example to specify a custom metaclass for Joe:

class MetaJoe(type):
    def __call__(cls, *args, **kwargs):
        print('MetaJoe.__call__')
        return None

class Joe(metaclass=MetaJoe):
    def __new__(cls, *args, **kwargs):
        obj = super(Joe, cls).__new__(cls)
        print('__new__ called. got new obj id=0x%x' % id(obj))
        return obj

    def __init__(self, arg):
        print('__init__ called (self=0x%x) with arg=%s' % (id(self), arg))
        self.arg = arg

j = Joe(12)
print(type(j))

So now the type of Joe is not type, but MetaJoe. Consequently, when PyObject_Call picks the call function to execute for j = Joe(12), it takes MetaJoe.__call__. The latter prints a notice about itself and returns None, so we don't expect the __new__ and __init__ methods of Joe to be called at all. Indeed, this is the outcome:

MetaJoe.__call__
<class 'NoneType'>

Digging deeper - tp_new

Alright, so now we have a better understanding of the object creation sequence. One crucial piece of the puzzle is still missing, though. While we almost always define __init__ for our classes, defining __new__ is rather rare [3]. Moreover, from a quick look at the code it's obvious that __new__ is more fundamental in a way. This method is used to create a new object. It is called once and only once per instantiation. __init__, on the other hand, already gets a constructed object and may not be called at all; it can also be called multiple times.

Since the type parameter passed to type_call in our case is Joe, and Joe does not define a custom __new__ method, then type->tp_new defers to the tp_new slot of the base type. The base type of Joe (and all other Python objects, except object itself) is object. The object.tp_new slot is implemented in CPython by the object_new function in Objects/typeobject.c.

object_new is actually very simple. It does some argument checking, verifies that the type we're trying to instantiate is not abstract, and then does this:

return type->tp_alloc(type, 0);

tp_alloc is a low-level slot of the type object in CPython. It's not directly accessible from Python code, but should be familiar to C extension developers. A custom type defined in a C extension may override this slot to supply a custom memory allocation scheme for instances of itself. Most C extension types will, however, defer this allocation to the function PyType_GenericAlloc.

This function is part of the public C API of CPython, and it also happens to be assigned to the tp_alloc slot of object (defined in Objects/typeobject.c). It figures out how much memory the new object needs [4], allocates a memory chunk from CPython's memory allocator and initializes it all to zeros. It then initializes the bare essential PyObject fields (type and reference count), does some GC bookkeeping and returns. The result is a freshly allocated instance.

Conclusion

Lest we lose the forest for the trees, let's revisit the question this article began with. What happens when CPython executes j = Joe()?

Since Joe has no explicit metaclass, type is its type. So the tp_call slot of type, which is type_call, is called.
type_call starts by calling the tp_new slot of Joe:
- Since Joe has no explicit base clase, its base is object. Therefore, object_new is called.
- Since Joe is a Python-defined class, it has no custom tp_alloc slot. Therefore, object_new calls PyType_GenericAlloc.
- PyType_GenericAlloc allocates and initializes a chunk of memory big enough to contain Joe.
type_call then goes on and calls Joe.__init__ on the newly created object.
- Since Joe does not define __init__, its base's __init__ is called, which is object_init.
- object_init does nothing.
The new object is returned from type_call and is bound to the name j.

This is the vanilla flow for an object of a class that doesn't have a custom metaclass, doesn't have an explicit base class, and doesn't define its own __new__ and __init__ methods. However, this article should have made it quite clear where these custom capabilities plug in to modify the object creation sequence. As you can see, Python is amazingly flexible. Practically every single step of the process described above can be customized, even for user-defined types implemented in Python. Types implemented in a C extension can customize even more, such as the exact memory allocation strategy used to create instances of the type.

[1]	The `PyTypeObject` structure definition for `type` is `PyType_Type` in `Objects/typeobject.c`. You can see that `type_call` is being assigned to its `tp_call` slot.

[2]	A future article will show how this comes to be when a new class is created.

[3]	Even when we do explicitly override `__new__` in our classes, we almost certainly defer the actual object creation to the base's `__new__`.

[4]	This information is available in the `PyObject` header of any type.