Eli Bendersky's website - Django

Clearing the database with Django commands

2014-02-20T06:01:07-08:00

In a previous post, I presented a method of loading initial data into a Django database by using a custom management command. An accompanying task is cleaning the database up. Here I want to discuss a few options of doing that.

First, some general design notes on Django management commands. If you run manage.py help you'll see a whole bunch of commands starting with sql. These all share a common idiom - print SQL statements to the standard output. Almost all DB engines have means to pipe commands from the standard input, so this plays great with the Unix philosophy of building pipes of single-task programs.

Django even provides a convenient shortcut for us to access the actual DB that's being used with a given project - the dbshell command.

As an example, we have the sqlflush command, which returns a list of the SQL statements required to return all tables in the database to the state they were in just after they were installed. In a simple blog-like application with "post" and "tag" models, it may return something like:

$ python manage.py sqlflush
BEGIN;
DELETE FROM "auth_permission";
DELETE FROM "auth_group";
DELETE FROM "django_content_type";
DELETE FROM "django_session";
DELETE FROM "blogapp_tag";
DELETE FROM "auth_user_groups";
DELETE FROM "auth_group_permissions";
DELETE FROM "auth_user_user_permissions";
DELETE FROM "blogapp_post";
DELETE FROM "blogapp_post_tags";
DELETE FROM "auth_user";
DELETE FROM "django_admin_log";

COMMIT;

Note there's a lot of tables here, because the project also installed the admin and auth applications from django.contrib.

We can actually execute these SQL statements, and thus wipe out all the DB tables in our database, by running:

$ python manage.py sqlflush | python manage.py dbshell

For this particular sequence, since it's so useful, Django has a special built-in command named flush.

But there's a problem with running flush that may or may not bother you, depending on what your goals are. It wipes out all tables, and this means authentication data as well. So if you've created a default admin user when jump-starting the application, you'll have to re-create it now.

Perhaps there's a more gentle way to delete just your app's data, without messing with the other apps? Yes. In fact, I'm going to show a number of ways.

First, let's see what other existing management commands have to offer. sqlclear will emit the commands needed to drop all tables in a given app. For example:

$ python manage.py sqlclear blogapp
BEGIN;
DROP TABLE "blogapp_tag";
DROP TABLE "blogapp_post";
DROP TABLE "blogapp_post_tags";

COMMIT;

So we can use it to target a specific app, rather than using the kill-all approach of flush. There's a catch, though. While flush runs delete to wipe all data from the tables, sqlclear removes the actual tables. So in order to be able to work with the database, these tables have to be re-created. Worry not, there's a command for that:

$ python manage.py sql blogapp
BEGIN;
CREATE TABLE "blogapp_post_tags" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
    "post_id" integer NOT NULL REFERENCES "blogapp_post" ("id"),
    "tag_id" varchar(50) NOT NULL REFERENCES "blogapp_tag" ("name"),
    UNIQUE ("post_id", "tag_id")
)
;
CREATE TABLE "blogapp_post" (
    "id" integer NOT NULL PRIMARY KEY AUTOINCREMENT,
 <.......>
)
;
CREATE TABLE "blogapp_tag" (
 <.......>
)
;

COMMIT;

So here's a first way to do a DB cleanup: pipe sqlclear appname into dbshell. Then pipe sql appname to dbshell.

An alternative way, which I like less, is to take the subset of DELETE statements generated by sqlflush, save them in a text file, and pipe it through to dbshell when needed. For example, for the blog app discussed above, these statements should do it:

BEGIN;
DELETE FROM "blogapp_tag";
DELETE FROM "blogapp_post";
DELETE FROM "blogapp_post_tags";
DELETE
COMMIT;

The reason I don't like it is that it forces you to have explicit table names stored somewhere, which is a duplication of the existing models. If you happen to change some of your foreign keys, for example, tables will need changing so this file will have to be regenerated.

The approach I like best is more programmatic. Django's model API is flexible and convenient, and we can just use it in a custom management command:

from django.core.management.base import BaseCommand
from blogapp.models import Post, Tag

class Command(BaseCommand):
    def handle(self, *args, **options):
        Tag.objects.all().delete()
        Post.objects.all().delete()

Save this code as blogapp/management/commands/clear_models.py, and now it can be invoked with:

$ python manage.py clear_models

Programmatically populating a Django database

2014-02-15T05:43:52-08:00

This is a quick post to demonstrate a very useful way of programmatically populating the models (i.e. database) of a Django application.

The canonical way to accomplish this is fixtures - the loaddata and dumpdata commands, but these seem to be more useful when you already have some data in the DB. Alternatively, you could generate the JSON information loadable by loaddata programmatically, but this would require following its format exactly (which means observing how real dumps are structured). One could, for the very first entries, just laboriously hammer them in through the admin interface. As programmers, however, we have a natural resentment for such methods.

Since Django apps are just Python modules, there's a much easier way. The very first chapter of the Django tutorial hints at the approach by test-driving the shell management command, which opens a Python shell in which the application is accessible, so the model classes can be imported and through them data can be both examined and created.

The same tutorial also mentions that you can bypass manage.py by pointing DJANGO_SETTINGS_MODULE to your project's settings and then calling django.setup(). This provides a clue on how the same steps can be done from a script, but in fact there's an even easier way.

There's no need to bypass manage.py, since it's a wonderful convenience wrapper around the Django project administration tools. It can be used to create custom management commands - e.g. your own commands parallel to shell, dumpdata, and so on. Not only that creating such commands gives you a very succinct, boilterplate-free way of writing custom management scripts, it also gives you a natural location to house them, per application.

Here's some simple code that adds a couple of tags into a blog-like model. Let's say the application is named blogapp:

from django.core.management.base import BaseCommand
from blogapp.models import Post, Tag

class Command(BaseCommand):
    args = '<foo bar ...>'
    help = 'our help string comes here'

    def _create_tags(self):
        tlisp = Tag(name='Lisp')
        tlisp.save()

        tjava = Tag(name='Java')
        tjava.save()

    def handle(self, *args, **options):
        self._create_tags()

This code has to be placed in a file within the blogapp/management/commands directory in your project. If that directory doesn't exist, create it. The name of the script is the name of the custom command, so let's call it populate_db.py. Another thing that has to be done is creating __init__.py files in both the management and commands directories, because these have to be Python packages. The directory tree will look like this:

blogapp
├── admin.py
├── __init__.py
├── management
│   ├── commands
│   │   ├── __init__.py
│   │   └── populate_db.py
│   └── __init__.py
├── models.py
... other files

That's it. Now you should be able to invoke this command with:

$ python manage.py populate_db

All the facilities of manage.py are available, such as help:

$ python manage.py help populate_db
Usage: manage.py populate_db [options] <foo bar ...>

our help string comes here

Options:
...

Note how help and args are taken from the Command class we defined. manage.py will also pass custom positional arguments and keyword options to our command, if needed. More details on writing custom management commands are available in this Django howto.

Once you start playing with such a custom data entry script, some of the existing Django management commands may come in very useful. You can see the full list by running manage.py help, but here's a list of those I found handy in the context of this post.

For dumping, dumpdata is great. Once your data grows a bit, you may find it useful only to dump specific models, or even specific rows by specifying primary keys with --pks. I also find the --indent=2 option to be essential when doing the default JSON dumps.

The flush command will clear the DB for you. A handy "undo" for those very first forays into entering data. Be careful with this command once you have real data in the DB.

Finally, the sqlall command is very useful when you're trying to figure out the structure of your models and the connections between them. IMHO model problems are important to detect early in the development of an application.

To conclude, I just want to mention that while custom management commands live within applications, nothing ties them to a specific app. It is customary for Django management commands to accept app and model names as arguments. While a data entry command is naturally tied to some application and model, this doesn't necessarily have to be the case in general. You can even envision an "app" named my_custom_commands which you can add to projects and reuse its functionality between them.

Django sessions - part III: User authentication

2011-07-17T06:13:11-07:00

In the previous two articles of this series we learned how Django implements sessions, thus allowing the abstraction of persistent state in a web application. The session framework can be employed by developers to implement all kinds of interesting features for their application, but Django also uses it for its own needs. Specifically, Django's user authentication system relies on the session framework to do its job.

The user authentication system allows users to log in and out of the application, and act based on a set of permissions. Borrowing from the Django Book:

This system is often referred to as an auth/auth (authentication and authorization) system. That name recognizes that dealing with users is often a two-step process. We need to

Verify (authenticate) that a user is who he or she claims to be (usually by checking a username and password against a database of users)
Verify that the user is authorized to perform some given operation (usually by checking against a table of permissions)

In this, the final part of the series, I want to explain how Django's user authentication is implemented. I will focus on item 1 in the list above - authentication, which makes actual use of sessions [1].

Snooping on HTTP traffic

Before diving into the source code of Django, let's see how authentication looks when viewed from the level of HTTP traffic. I'll be using this view to test things:

def test_user(request):
    user_str = str(request.user)
    if request.user.is_authenticated():
        return HttpResponse('%s is logged in' % user_str)
    else:
        return HttpResponse('%s is not logged in' % user_str)

Before I logged in, I get the message "AnonymousUser is not logged in". The server doesn't return any cookie.

When I log in with Django's default login template, more interesting things can be observed. The login form sends a POST request to the server, with my login information in the form data:

username:eliben
password:password

Assuming this is a valid username/password pair, the server sends back a session ID in a cookie:

Set-Cookie:sessionid=4980ec04546e434c1ea13c675fafbc98;

What is this session? As we saw in the previous article, it's a key into the django_session DB table. Decoding the session data from the table, I get:

{'_auth_user_id': 1, '_auth_user_backend': 'django.contrib.auth.backends.ModelBackend'}
[26/Jun/2011 11:28:48] "GET /user/ HTTP/1.1" 200 19

By looking into the auth_user table, I can indeed see that eliben is the user with ID = 1. Also, in subsequent requests to the server, my browser sends the aforementioned session ID in a cookie, as expected.

When I log out, the server sends a different session ID, which now contains an empty dictionary - no user is logged in.

Auth middleware

Similarly to the path we've taken with sessions, it's instrumental to first see how authentication middleware is implemented. Or in other words, how did the "user" attribute get into my HTTP request?

The answer is in contrib/auth/middleware.py [2]:

class LazyUser(object):
    def __get__(self, request, obj_type=None):
        if not hasattr(request, '_cached_user'):
            from django.contrib.auth import get_user
            request._cached_user = get_user(request)
        return request._cached_user


class AuthenticationMiddleware(object):
    def process_request(self, request):
        request.__class__.user = LazyUser()
        return None

Behold, we've encountered a rare sighting of one of Python's more obscure, and yet powerful features - descriptors. Explaining descriptors fully will take an article of its own, so I kindly direct you to google it [3]. Here I'll just briefly explain how this specific code works.

AuthenticationMiddleware is a middleware class, implementing the process_request hook. What it does is attach the LazyUser descriptor to the user attribute of the request class. The LazyUser descriptor implements __get__, which will get called when we access request.user in our views. This __get__ simply caches the user object in another attribute of the request class - _cached_user, making sure the possibly costly get_user operation doesn't get fully executed for each access to the user attribute.

Finding the active user

Recall that request.user gets us the currently logged-in user, if there is one. Let's see how this gets done. In the code sample above, the user is accessed with get_user(request). Here's get_user:

def get_user(request):
    from django.contrib.auth.models import AnonymousUser
    try:
        user_id = request.session[SESSION_KEY]
        backend_path = request.session[BACKEND_SESSION_KEY]
        backend = load_backend(backend_path)
        user = backend.get_user(user_id) or AnonymousUser()
    except KeyError:
        user = AnonymousUser()
    return user

Looking at the beginning of the file this function is defined in (auth/__init__.py), we see:

SESSION_KEY = '_auth_user_id'
BACKEND_SESSION_KEY = '_auth_user_backend'

So what get_user does is just try to extract the user from the current session. Recall from the HTTP snooping section that when a user is actually logged in, the _auth_user_id and _auth_user_backend entries are set in the session dictionary. get_user reads them, and turns to the auth backend to fetch the user object, with the backend.get_user method.

The default auth backend is auth.backends.ModelBackend - the DB backed user model (contained in all those auth_* tables that get added to your DB when the auth framework is enabled). Its get_user method simply does this:

def get_user(self, user_id):
    try:
        return User.objects.get(pk=user_id)
    except User.DoesNotExist:
        return None

Standard Django code for fetching data from a DB, where User is a model defined in contrib/auth/models.py.

The User model

User is a fairly standard Django model with a bunch of fields and helper methods that help decoding them. The most interesting part, IMHO, and the one I'll focus here on is setting and verifying the user's password. Here's the set_password method:

def set_password(self, raw_password):
    if raw_password is None:
        self.set_unusable_password()
    else:
        import random
        algo = 'sha1'
        salt = get_hexdigest(algo, str(random.random()), str(random.random()))[:5]
        hsh = get_hexdigest(algo, salt, raw_password)
        self.password = '%s$%s$%s' % (algo, salt, hsh)

We see this uses the accepted modern approach - instead of storing the password itself (in plaintext), a cryptographic hash is computed and stored. Further, the password is salted, to defeat a potential rainbow table cracking attack. The password's hash, together with the salt value and the algorithm used for the hashing (which is SHA1 by default) are then stored in the database, separated by dollar signs. For example, here's my user's password field:

sha1$f0670$2a0781bd8f2361042ebdf0cd1b3ce1e8be3f8dcc

To verify the password, the check_password function is invoked [4]:

def check_password(raw_password, enc_password):
    """
    Returns a boolean of whether the raw_password was correct. Handles
    encryption formats behind the scenes.
    """
    algo, salt, hsh = enc_password.split('$')
    return constant_time_compare(hsh, get_hexdigest(algo, salt, raw_password))

It computes the hash on the password provided and compares it to the one stored in the DB. What is this constant_time_compare call about though? This is a function that compares two strings in time that depends on the length of the strings, but not the amount of matching characters in the beginning. Such comparison is important cryptographically, to thwart timing attacks.

Logging in

Django provides a powerful and versatile login view (in django.contrib.auth.views) to allow an application implement logging-in functionality. What this view does is explained quite well in Django's auth's docs. Here, I will focus on how it works.

login is your typical form-handling Django view. If it gets a GET request, it displays the login form. On the other hand, for a POST request, it tries to log the user in. This is the interesting part.

On first sight, it's hard to see where exactly the login authentication is handled. The POST request handling part of the login view is:

if request.method == "POST":
    form = authentication_form(data=request.POST)
    if form.is_valid():
        netloc = urlparse.urlparse(redirect_to)[1]

        # Use default setting if redirect_to is empty
        if not redirect_to:
            redirect_to = settings.LOGIN_REDIRECT_URL

        # Security check -- don't allow redirection to a different
        # host.
        elif netloc and netloc != request.get_host():
            redirect_to = settings.LOGIN_REDIRECT_URL

        # Okay, security checks complete. Log the user in.
        auth_login(request, form.get_user())

        if request.session.test_cookie_worked():
            request.session.delete_test_cookie()

        return HttpResponseRedirect(redirect_to)

After some head scratching and stunning feats of reverse engineering [5], it became clear that the authentication happens in the call form.is_valid. This call invokes (after some steps [6]) AuthenticationForm.clean, which itself calls authenticate. A bit down the road, the flow of control reaches ModelBackend.authenticate:

def authenticate(self, username=None, password=None):
    try:
        user = User.objects.get(username=username)
        if user.check_password(password):
            return user
    except User.DoesNotExist:
        return None

And we've already seen the definition of check_password in section "The User model". Alright, so now we know how the login view authenticates the user. What does it do next? After some more checks, the auth_login function is called, which is an alias for django.contrib.auth.login. This function uses the session framework to save the cookie we've seen in the first section of this article.

Summary

This concludes the article on how user authentication works, and also the 3-article series on Django sessions. Sessions in general and user logins in particular are one aspect of web applications we take for granted. It just works, automagically. And yet, as I hope these articles have demonstrated, the magic that happens under the hood isn't really difficult to understand. Sure, there are a lot of concepts to grasp and code to read, but with some determination enlightenment is just around the corner.

[1]	Authorization is a simple matter of keeping tables of users, groups and permissions.

[2]	Just a reminder: at the time of writing this series of articles, the latest released version of Django is 1.3, which is what I'm looking at here. In this particular code sample I've removed a longish assertion line from `process_request`.

[3]	But start with this excellent article by Raymond Hettinger.

[4]	It's invoked by the `check_password` method which also handles a backwards-compatibility issue, which I don't cover here.

[5]	I'm kidding :-) All it took was 30 seconds to strategically insert `traceback.print_stack()` into `User.check_password` and see where it's being called from when I log in.

[6]	According to the Django docs, the `clean` method of a subclass of `Form` is responsible for whole-form validation. Not the most intuitive nomenclature, I'd say.

Django sessions - part II: How sessions work

2011-06-29T05:03:14-07:00

This is the second in a series of three articles.

Sessions are Django's high-level tool for keeping a persistent state for users on the server. Sessions allow to store arbitrary data per visitor, and have this data available the next time the visitor visits the site. As we'll learn in this article, sessions are still based on cookies, but cookie management is abstracted away, handling a lot of issues on the way - as sessions provide a more convenient, robust and safe way to store the data.

Example - using sessions

First, it's useful to see an easy example of using Django sessions. Here's a simple view that uses sessions to count the amount of times a user has triggered it [1].

def test_count_session(request):
    if 'count' in request.session:
        request.session['count'] += 1
        return HttpResponse('new count=%s' % request.session['count'])
    else:
        request.session['count'] = 1
        return HttpResponse('No count in session. Setting to 1')

If we compare this to the cookies usage example from part I, a couple of differences are apparent:

Sessions are more uniform - a single session attribute of the request is used for both querying and modifying the session.
While in this example we're only using an integer as the value, the session attribute acts as a dictionary, allowing string keys and almost arbitrary Python objects as values [2].

However, looking at the actual HTTP traffic for this view, we notice yet another, very important difference. Here's the cookie the view returns to the user for this particular instance of the application on my machine:

Set-Cookie:sessionid=a92d67e44a9b92d7dafca67e507985c0;
           expires=Thu, 07-Jul-2011 04:16:28 GMT;
           Max-Age=1209600;
           Path=/

There's no count=1 (or other numeric value) here - the cookie just sets some unique sessionid. We'll see what this means shortly, but I'll just note that this is a very important feature of session management. Think about the security implications, for instance. Suppose the user gets a prize for the 10th time he triggers the view. With a simple cookie passing the count into the user's browser this would be something very easy to spoof. With a session, however, the user has no idea what the correct sessionid - in fact, no such sessionid exists yet, so the user has no real way spoofing his 10th visit [3]

Deciphering the session ID

By default, Django's session module stores sessions in the app's main DB, in table django_session with this schema:

CREATE TABLE "django_session" (
    "session_key" varchar(40) NOT NULL PRIMARY KEY,
    "session_data" text NOT NULL,
    "expire_date" datetime NOT NULL
);

session_key is the ID placed in the cookie, and session_data contains the actual session data in encoded format. Here's how to decipher the session ID we've seen above:

from django.contrib.sessions.models import Session
#...
sess = Session.objects.get(pk='a92d67e44a9b92d7dafca67e507985c0')
print(sess.session_data)
print(sess.get_decoded())

This prints:

ZmEyNDVhNTBhMTk2ZmRjNzVlYzQ4NTFjZDk2Y2UwODc3YmVjNWVjZjqAAn1xAVUFY291bnRxAksG
cy4=

{'count': 6}

As you can see, Django stores the request.session dictionary in the DB, in an encoded manner. Django can recover it from the DB by using the session ID the user's browser returns in a cookie. All of this is done transparently by Django's session module - the application's view just has a simple access to the request.session dictionary. Let's dive into the guts of Django to understand how it manages to make this work.

Session middleware

The first layer of magic I'd like to unwrap has to deal with the session attribute of django.http.HttpRequest. How does the session information even get there, and how can the view change the session by simply modifying the attribute?

The answer is Django's middleware. To borrow a quote from the Django Book:

[...] Django’s middleware framework, which is a set of hooks into Django’s request/response processing. It’s a light, low-level “plug-in” system capable of globally altering both Django’s input and output.

We can think of middleware in the following way. The normal flow of data around the view we're coding in Django looks like this:

The view accepts a HTTP request object, does some application-specific work based on its contents and eventually returns a HTTP response object. Middleware makes this process a bit more complicated:

This is done by allowing the programmer to write "hook classes" with special methods that the middleware framework knows about [4]. These hooks can be registered in the MIDDLEWARE_CLASSES setting in settings.py. Note that the django.contrib.sessions.middleware.SessionMiddleware class is there by default. Looking at its source code, it has two middleware hooks - process_request and process_response [5].

process_request pulls the session key (ID) from a cookie. We can see that sessionid is actually a configurable name - SESSION_COOKIE_NAME, set by default in Django's global settings to sessionid. The request.session attribute is then populated to contain a "session store" object. More on this object a bit later.

process_response saves the session store object (thus making the changes the view did persistent) and attaches a cookie to the response sent to the client. To save on traffic, it does that only if the view actually modified the session, or if the SESSION_SAVE_EVERY_REQUEST setting is set.

This explains how the sessions are translated to cookies. But clearly, a lot of the logic is still hidden, implemented in the store object of sessions. Let's see how that works.

Session store

Sessions can use one of several storage "engines" (backends). This is configurable via the SESSION_ENGINE setting, which points to django.contrib.sessions.backends.db by default - the application's main database (as mentioned above in "Deciphering the session ID"). If you look at the sessions/backends directory in Django's source you'll see other available engines, but unless your needs are very special, you're probably OK with the default one.

Each storage engine exports a StorageSession class which derives from StorageBase. This common base implements most of the functionality of session stores, relying on methods from its specializations to abstract away the actual method of storing the data - whether in DB, file, in-memory cache or some other way. The DB-backed store uses Django's standard ORM, defined in module session.models.

To understand how all these classes play together, let's follow through what happens when the user tries to access request.session in a view, assuming the default DB store:

Session middleware's process_request sets request.session to be an instance of db.SessionStore with session_key passed into the constructor.
The constructor of SessionStore defers to the constructor of SessionBase, which stores the session key for later use.
- Note that the session isn't loaded right away from the DB. This is lazy loading - the actual data is loaded when it's actually being accessed.
process_request is done at this point, so the HTTP request is passed into the view. Suppose we now read its count key, as in the example above.
SessionBase implements a dict-like interface [6], and in particular __getitem__, which takes the key from a _session attribute, which in itself is a property, deferring reads to the _get_session method.
_get_session does the actual lazy loading, using the load method.
load is one of the methods related to the actual storage, so SessionBase doesn't implement it. Instead db.SessionStore implements it and uses the session DB model to load the value from the DB based on the key, decoding it first.

This is about it, except one small detail. How does encoding and decoding work? Let's look at encode:

def encode(self, session_dict):
    "Returns the given session dictionary pickled and encoded as a string."
    pickled = pickle.dumps(session_dict, pickle.HIGHEST_PROTOCOL)
    hash = self._hash(pickled)
    return base64.encodestring(hash + ":" + pickled)

The session dictionary is pickled. Then, a hash is computed and prepended to the pickle string. Finally the whole string is encoded in base 64, which is stored in the session_data field of the DB table.

Conclusion

In this article, we've seen how to use Django sessions, what happens on the low-level of HTTP requests and responses when sessions are being used, and how sessions are actually implemented by Django. While I didn't cover every little detail, I hope there's enough information to understand the big picture. If there's any important information you think I may have missed, please let me know.

[1]	For the sake of this article, I'm ignoring cookie/session expiration issues. Assume they never expire.

[2]	By "almost arbitrary" I refer to Python objects that are pickle-able.

[3]	I say no "real" way because this scheme is, of course, not entirely secure - so don't bet real money on it. Depending on the exact configuration and usage of sessions by the application, by having access to the traffic from other users, the attacker can possibly spoof a session ID.

[4]

These middleware classes are a prime example of Python's duck typing. No need to adhere to any specified interface; no need to derive from some common base or explicitly set the hook methods. Just implement the methods you need in a class, and register that class with the framework. Python's reflection and duck typing capabilities are then used to automatically discover and use these hooks.

[5]	Take a moment to review the Django middleware docs to understand how to use these hooks.

[6]	More formally, a Python mapping type.

Django sessions - part I: Cookies

2011-06-24T12:54:49-07:00

HTTP is a stateless protocol - the server is not required to retain information or status about each user for the duration of multiple requests.

For smart web applications, however, this isn't good enough. You want to login into an application and have it remember you across requests. A good example is maintaining a "shopping cart" at some merchandise website, which you gradually fill as you browse through the products that interest you.

To solve this problem, HTTP cookies were invented by Netscape back in the 1990s. Cookies are formally defined in RFC2965, but to spare you all that jabber, cookies can be described very simply.

A cookie is just an arbitrary string sent by the server to the client as part of the HTTP response. The client will then return this cookie back to the server in subsequent requests. The information stored in the cookie is opaque to the client - it's only for the server's own use. This scheme allows the client to identify itself back to the server with some state the server has assigned it. Here's a more detailed flow of events:

The client connects to the server for the first time, and sends a normal HTTP request (say, a simple GET for the main page).
The server wants to track the client's state and in its HTTP response (which contains the page contents) attaches a Set-Cookie header. This header's information is a set of key, value pairs, where both keys and values are strings that make sense for the server, but for the client are a black box.
In subsequent requests the client makes to the server, it adds a Cookie header in the HTTP requests it sends, with the cookie information the server specified in previous responses.

Implementation-wise, the client stores the latest cookie received from various servers (which are easily identifiable by their URLs). Even if the next time the client accesses the server is a few days after the previous request, it will still send this information (assuming the cookie hasn't expired), and the server will be able to identify it. This is why I can point my browser to Amazon today, not having visited it for some weeks, and the website will greet me with "Hello, Eli".

The above is a necessarily simplified explanation of cookies - I have no intention of repeating the contents of the RFC here. There are a lot of details I've left out like expiration time, filtering of cookies by paths, various size and amount limits the user agents (web browsers, etc.) are forced to abide, and so on. However, it's a sufficient amount of details for the needs of this article, so let's see some code.

Setting cookies in Python, without Django

The following demonstrates how to set cookies in from a Python server-side application without using Django. For simplicity, I'll just use the web server built-in into the Python standard library:

from BaseHTTPServer import HTTPServer
from SimpleHTTPServer import SimpleHTTPRequestHandler
import Cookie

class MyRequestHandler(SimpleHTTPRequestHandler):
    def do_GET(self):
        content = "<html><body>Path is: %s</body></html>" % self.path
        self.send_response(200)
        self.send_header('Content-type', 'text/html')
        self.send_header('Content-length', str(len(content)))

        cookie = Cookie.SimpleCookie()
        cookie['id'] = 'some_value_42'

        self.wfile.write(cookie.output())
        self.wfile.write('\r\n')

        self.end_headers()
        self.wfile.write(content)

server = HTTPServer(('', 59900), MyRequestHandler)
server.serve_forever()

This is a very simple application that just shows the path that the client requested. The more interesting thing happens below the covers - the application also sets a cookie. If we examine the HTTP response sent by this application to a client that connected to it, we'll see this among the headers:

Set-Cookie: id=some_value_42

In a similar manner, the Cookie module allows the server to parse cookies returned by the client in Cookie headers, using the load method.

Setting and reading cookies with Django

Django makes setting and reading cookies almost trivial. Here's a simple view that checks whether the client set the id cookie in its request, and if it hadn't, sends the cookie to the client (so that the client will have it for the next request):

def test_cookie(request):
    if 'id' in request.COOKIES:
        cookie_id = request.COOKIES['id']
        return HttpResponse('Got cookie with id=%s' % cookie_id)
    else:
        resp = HttpResponse('No id cookie! Sending cookie to client')
        resp.set_cookie('id', 'some_value_99')
        return resp

As you can see, cookies are taken from the COOKIES dict-like attribute of Django's HttpRequest, and set by calling the set_cookie method of HttpResponse. Couldn't be any simpler. What we're really here for is to understand how these things work under the hood of Django, so let's dive in.

How cookies are implemented in Django

The recommended way to deploy Django applications is with WSGI, so I'll focus on the WSGI backend implemented in Django. This is a good place to mention that at the time of this writing, I'm looking into the source code of Django 1.3, which is installed in site-packages/django in the usual installation structure of Python.

Looking at Django's WSGIRequest class (which inherits from http.Request) we can see that COOKIES is a property that hides a dict attribute named self._cookies behind a getter/setter pair. The dict is initialized in _get_cookies:

def _get_cookies(self):
    if not hasattr(self, '_cookies'):
        self._cookies = http.parse_cookie(self.environ.get('HTTP_COOKIE', ''))
    return self._cookies

This appears to be a lazy initialization that should aid performance - if the view doesn't want to look into the cookies of a request, there's no need to parse them. Cookies are taken from the HTTP_COOKIE entry of the request's environment object, per the WSGI specification. What about http.parse_cookie? This is a utility method in Django's HTTP module:

def parse_cookie(cookie):
    if cookie == '':
        return {}
    if not isinstance(cookie, Cookie.BaseCookie):
        try:
            c = SimpleCookie()
            c.load(cookie, ignore_parse_errors=True)
        except Cookie.CookieError:
            # Invalid cookie
            return {}
    else:
        c = cookie
    cookiedict = {}
    for key in c.keys():
        cookiedict[key] = c.get(key).value
    return cookiedict

As you can see, it uses the Cookie module from the standard library to parse the cookie with the load method, similarly to what I mentioned above for the non-Django code.

Setting cookies on a response is done with the set_cookie method of HttpResponse. This method simply writes down the new cookie in its self.cookies attribute. WSGIHandler then adds the cookies to its response headers when sending the response.

Wrapping up

As you can see, cookies are relatively easy to handle in Python, and in particular with Django. That said, when writing a Django application it's rare to be needing cookies directly, because cookies are a fairly low-level building block. Django's higher level session framework is much easier to use and is the recommended way to implement persistent state in applications. The next part of the article will examine how to use Django sessions and how they work under the hood.