There are many approaches to improve the situation; here I want to discuss one I’ve been using very successfully in the past few months – a simple "persistent history" that keeps track of history across terminal instances, saving it into a dot-file in my home directory (~/.persistent_history). All commands, from all terminal instances, are saved there, forever. I found this tremendously useful in my work – it saves me time almost every day.

Why does it go into a separate history and not the main one which is accessible by all the existing history manipulation tools? Because IMHO the latter is still worthwhile to be kept separate for the simple need of bringing up recent commands in a single terminal, without mixing up commands from other terminals. While the terminal is open, I want the press "Up" and get the previous command, even if I’ve executed a 1000 other commands in other terminal instances in the meantime.

Persistent history is very easy to set up. Here’s the relevant portion of my ~/.bashrc:

log_bash_persistent_history()
{
  [[
    $(history 1) =~ ^\ *[0-9]+\ +([^\ ]+\ [^\ ]+)\ +(.*)$
  ]]
  local date_part="${BASH_REMATCH[1]}"
  local command_part="${BASH_REMATCH[2]}"
  if [ "$command_part" != "$PERSISTENT_HISTORY_LAST" ]
  then
    echo $date_part "|" "$command_part" >> ~/.persistent_history
    export PERSISTENT_HISTORY_LAST="$command_part"
  fi
}

# Stuff to do on PROMPT_COMMAND
run_on_prompt_command()
{
    log_bash_persistent_history
}

PROMPT_COMMAND="run_on_prompt_command"

The format of the history file created by this is:

2013-06-09 17:48:11 | cat ~/.persistent_history
2013-06-09 17:49:17 | vi /home/eliben/.bashrc
2013-06-09 17:49:23 | ls

Note that an environment variable is used to avoid useless duplication (i.e. if I run ls twenty times in a row, it will only be recorded once).

OK, so we have ~/.persistent_history, how do we use it? First, I should say that it’s not used very often, which kind of connects to the point I made earlier about separating it from the much higher-use regular command history. Sometimes I just look into the file with vi or tail, but mostly this alias does the trick for me:

alias phgrep='cat ~/.persistent_history|grep --color'

The alias name mirrors another alias I’ve been using for ages:

alias hgrep='history|grep --color'

Another tool for managing persistent history is a trimmer. I said earlier this file keeps the history "forever", which is a scary word – what if it grows too large? Well, first of all – worry not. At work my history file grew to about 2 MB after 3 months of heavy usage, and 2 MB is pretty small these days. Appending to the end of a file is very, very quick (I’m pretty sure it’s a constant-time operation) so the size doesn’t matter much. But trimming is easy:

tail -20000 ~/.persistent_history | tee ~/.persistent_history

Trims to the last 20000 lines. This should be sufficient for at least a couple of months of history, and your workflow should not really rely on more than that

Finally, what’s the use of having a tool like this without employing it to collect some useless statistics. Here’s a histogram of the 15 most common commands I’ve used on my home machine’s terminal over the past 3 months:

ls        : 865
vi        : 863
hg        : 741
cd        : 512
ll        : 289
pss       : 245
hst       : 200
python    : 168
make      : 167
git       : 148
time      : 94
python3   : 88
./python  : 88
hpu       : 82
cat       : 80

Some explanation: hst is an alias for hg st. hpu is an alias for hg pull -u. pss is my awesome pss tool, and is the reason why you don’t see any calls to grep and find in the list. The proportion of Mercurial vs. git commands is likely to change in the very near future due to this.

I’m switching my public open-source projects from Bitbucket to Github, and in the process also from Mercurial to git. But the switch has little to do with git vs. Mercurial; it’s mostly about Github winning the platform war vs. Bitbucket. I like Mercurial, and it was a natural choice when switching from SVN a few years back. Back then I wasn’t familiar with git, and I am quite a bit more familiar with it now, but still the actual SCM played little role in the decision.

Well, to be precise there’s one thing I find slightly more convenient about git. I really like its throwaway-branch mode of development. I want to be able to create quick local branches for hacking, throw some away, merge others and keep my history clean. I want to commit every single comma if I feel like it, without worrying about polluting the "official" history. So git merge --squash or squashing with interactive rebasing are workflows I appreciate. It’s not that these aren’t possible with Mercurial, which recently gave up a bit on the pedanticism and allows similar workflows via extensions. But it’s not the natural or the familiar way of working with Mercurial. As a proof of that, I kept receiving pull requests with dozens of useless small commits just to implement some feature, and kept asking the contributors to find a way to send me a single commit, or just leave the pull request machinery behind and send an old’n good patch file.

But I’m getting sidetracked. Github is just way, way more popular these days, especially for open source projects. I was very disappointed seeing many contributors say they’re reluctant to contribute because that would require creating a Bitbucket account and fork my projects there. Would I please just switch to Github? sigh… I can understand that – Github managed to give coding a nice social aspect – your Github profile is part of your online "rep". You want your contributions to other projects to be seen through it, so people were feeling that having a fork on Bitbucket is like this hidden place no one will ever see and attribute to them.

A curious anecdote of the relative popularity is something I noticed when I started doing the switch. I had more followers on Github than on Bitbucket! Oh my, even though I had a number of moderately popular open source projects I was furiously hacking on Bitbucket, vs. a bunch of half-neglected forks and hacks on Github.

So here it is; not an overly coherent set of thoughts, I’ll admit, but I hope it makes sense. I don’t have anything against Mercurial or Bitbucket – I’m still a user of both. But the higher-profile open-source projects are now on Github. Happy hacking.

While the documentation does a good job describing how require finds the module to import, it doesn’t say much about how the importing itself happens, and how the exports and module objects are magically visible and usable in the module’s code. Here I want to provide a lower-level view of this missing link, gleaned from the source code of Node.js v0.10.8 (lib/module.js).

As the documentation linked above explains, there are a few places modules can be found in by require. There’s also a number of different imports require can perform – from folders, from JSON files, from compiled Node modules (C++), and so on. Here I’ll focus on importing from a regular JavaScript source file (.js).

Code from .js files is simply read into a string. Next, the code string is wrapped with a function:

(function (exports, require, module, __filename, __dirname) {
   // <-- MODULE CONTENTS HERE -->
});

These wrapped contents are evaluated by the JavaScript VM, and the result is a function object; let’s call it module_func. This function is invoked as follows:

module_func.apply(module.exports,
                  [module.exports, require, module, filename, dirname]);

And the return value of require is then module.exports.

What’s module? It’s the special module object the require mechanism has built for loading our module. It’s actually described quite well in the modules documentation I mentioned before. That page says:

In each module, the module free variable is a reference to the object representing the current module. In particular module.exports is accessible via the exports module-global. module isn’t actually a global but rather local to each module.

Now it should be obvious how this comes to be. The wrapper function created for our code has the arguments module and exports, which become visible in the code. The apply invocation above shows what they get bound to. What it also does is set this in the global scope of our code to the exports property of the module. So the following are all equivalent ways to add stuff to a module’s exports:

exports.say_hi = function () {console.log('hi');}
module.exports.say_bye = function () {console.log('bye');}
this.say_farewell = function () {console.log('farewell');}

The last way look suspicious. We know that by default, variables in the code don’t get exported. In other words, in:

var foo = 1;
exports.bar = 2;

While bar is exported to require, foo is not. But how can this be, if this is bound to exports? Doesn’t var foo = 1 add foo to this, being the global object?

This is only puzzling if you think of your code as stand-alone JavaScript, in which the global scope is truly global. But recall, from a few paragraphs above, that our code is wrapped in a function. So the "global" scope in the module is actually function scope. In function scope, variables don’t get auto-bound to this. Mystery solved.

One final note: Node.js has a few useful flags you can set as environment variables for debugging. In particular, I’ve found setting:

$ NODE_DEBUG=module node <file.js>

Very handy for following through the module loading process require does.

The discussion and decision process has been long and arduous, but eventually very positive. A collective brain is certainly better than any single one; the final proposal is better in a number of ways than the initial one, and the vast majority of Python core developers now feel good about it (give or take a couple of very minor issues).

I’ve been told enums have been debated on Python development lists for years. There’s at least one earlier PEP (354) that’s been discussed and rejected in 2005.

I think part of the success of the current attempt can be attributed to the advances in metaclasses that has been made in the past few releases (3.x). These advances allow very nice syntax of enum definitions that provides a lot of convenient features for free. I tried to find interesting examples of metaclasses in the standard library in 2011; Once the enum gets pushed I’ll have a much better example

In these 10 years the blog went from an obscure journal on use.perl.org that only a few people knew about to 66,000 unique visitors with half a million page views a month on average (stats for end-of 2012).

While it looks very different from the initial "weblog", it actually went through relatively few incarnations for its age. Starting somewhere in 2005 I became unhappy with use.perl.org and eventually moved the blog to Blogger. But Blogger itself was quite limited for serious programming-related blogging at the time, so it took me only two weeks to figure out it’s not quite what I wanted and move to a WordPress-based blog on my own domain and shared hosting (Bluehost). In the seven years that has passed since, the blog changed very little visually. I stretched the main viewing area bit by bit as high-resolution monitors became more common, installed a couple of minor plugins, but that’s it.

What did change was the quality of content. As I wrote here, the blog became much more technical with time, leaving the shorter and more personal posts to outlets like G+. The amount of technical content is also the reason the blog’s popularity grew rapidly in the past few years. Apparently more people want to know how debuggers work and about the internals of Python and Linux loaders than about some random cool library I found or how I spent the last week at work being bored. Go figure.

So what’s next?

Dedicated readers will notice that the amount of in-depth posts has slowed down in the past few months. This was to be expected, having moved to a different continent and started a much more demanding job. But I still hope to continue posting articles from time to time. This blog is far from being done! As for infrastructure, I was actually pondering to make a large change after yet another break-your-site WordPress upgrade. But I still haven’t found time to do this. Maybe at some point I will.

So thanks for reading, and wishing myself at least another 10 years of blogging. It has been an incredibly positive experience for me in many respects, and I have full intention of keeping it up as long as I can.

The solution many Python programmers and projects have adopted is to use virtualenv. If you haven’t heard about virtualenv, you’re missing out – go read about it now.

I’m not going to write a tutorial about virtualenv or extoll its virtues here – enough bits have been spilled about this on the net already. What I plan to do is share an interesting problem I ran into and the solution I settled on.

I had to install some packages (Sphinx and related tools) on a new machine into a virtualenv. But the machine only had a basic Python installation, without setuptools or distribute, and without virtualenv. These aren’t hard to install, but I wondered if there’s an easy way to avoid installing anything. Turns out there is.

The idea is to create a "bootstrap" virtual environment that would have all the required tools to create additional virtual environments. It turns out to be quite easy with the following script (inspired by the answer in this SO discussion):

import sys
import subprocess

VENV_VERSION = '1.9.1'
PYPI_VENV_BASE = 'http://pypi.python.org/packages/source/v/virtualenv'
PYTHON = 'python2'
INITIAL_ENV = 'py-env0'

def shellcmd(cmd, echo=True):
    """ Run 'cmd' in the shell and return its standard out.
    """
    if echo: print '[cmd] {0}'.format(cmd)
    out = subprocess.check_output(cmd, stderr=sys.stderr, shell=True)
    if echo: print out
    return out

dirname = 'virtualenv-' + VENV_VERSION
tgz_file = dirname + '.tar.gz'

# Fetch virtualenv from PyPI
venv_url = PYPI_VENV_BASE + '/' + tgz_file
shellcmd('curl -O {0}'.format(venv_url))

# Untar
shellcmd('tar xzf {0}'.format(tgz_file))

# Create the initial env
shellcmd('{0} {1}/virtualenv.py {2}'.format(PYTHON, dirname, INITIAL_ENV))

# Install the virtualenv package itself into the initial env
shellcmd('{0}/bin/pip install {1}'.format(INITIAL_ENV, tgz_file))

# Cleanup
shellcmd('rm -rf {0} {1}'.format(dirname, tgz_file))

The script downloads and unpacks a recent virtualenv (substitute your desired version in VENV_VERSION) from PyPI and uses it directly (without installing) to create a new virtual env. By default, virtualenv will install setuptools and pip into this environment. Then, the script also installs virtualenv into the same environment. This is the bootstrap part.

Voila! py-env0 (or whatever you substituted in INITIAL_ENV) is now a self-contained virtual environment with all the tools you need to create new environments and install stuff into them.

This script is for Python 2 but can be trivially adapted for Python 3. In Python 3, the situation is actually more interesting. Python 3.3 (which is really the one you ought to be using if you’ve switched to 3 already) comes with virtualenv in the standard library (venv package), so downloading and installing it is not required.

That said, its virtualenv will not install setuptools and pip into the environments it creates. So YMMV here: if you need setuptools and pip there, go with a variation of the script above. If not, you don’t need anything special really, just use the python3.3 -m venv.

P.S. The packaging situation is getting better though. There was a lot of focus during the recent PyCon on this. One of the interesting announcements was that distribute is merging back into setuptools.

Re-reads:

• "A pale blue dot" by Carl Sagan

Update 23.03.2013: version 1.38 was released with an important bug-fix. If you’ve installed 1.37, please upgrade. If not, still please upgrade

Since pss was first released a year and a half ago, it has become an indispensable tool in my, and many other developers’ toolbox.

pss has matured and underwent multiple feature and bug-fix releases. Today I made a new release with a slightly larger amount of changes than usual (as well as some nice optimizations that make pss run faster) and figured it’s time to move the major version dial from 0, signaling that I consider pss to be a mature and stable project. Here’s the list of changes:

+ Version 1.37 (21.03.2013)

  - Fixed the output of --help-types to describe file patterns more faithfully.
  - Issue #35: Fix the -w option to work properly on Python 3
  - Added '.tox' to the set of directories ignored by default.
  - Issue #34: Support for --cython
  - Added support for the .el extension for Emacs Lisp
  - Issue #36: Fixed the --match option.
  - Issue #37: Allow comma-separated lists in --ignore-dirs
  - Optimize the files-with-matches option by looking for a single match within
    a file instead of all matches.
  - Issue #38: Optimize matching of non-regex patterns, which is very common.
    Measured speedup of 10-20%.

While one of the original goals of pss was to work well on Windows without having to install additional packages, I’m now in a situation where I don’t have easy access to any Windows machine. I’m 99% sure pss keeps working on Windows but haven’t tested it. Therefore, if you’re using pss on Windows – please download the new release from PyPI and let me know if you run into any problems. Thanks in advance.

Here I want to discuss how the same task is done from Python, both with the existing stdlib ctypes package and the new cffi library, developed by the PyPy team and a candidate for inclusion into the Python stdlib in the future.

from ctypes import cdll, Structure, c_int, c_double, c_uint

lib = cdll.LoadLibrary('./libsomelib.so')
print('Loaded lib {0}'.format(lib))

# Describe the DataPoint structure to ctypes.
class DataPoint(Structure):
    _fields_ = [('num', c_int),
                ('dnum', c_double)]

# Initialize the DataPoint[4] argument. Since we just use the DataPoint[4]
# type once, an anonymous instance will do.
dps = (DataPoint * 4)((2, 2.2), (3, 3.3), (4, 4.4), (5, 5.5))

# Grab add_data from the library and specify its return type.
# Note: .argtypes can also be specified
add_data_fn = lib.add_data
add_data_fn.restype = DataPoint

print('Calling add_data via ctypes')
dout = add_data_fn(dps, 4)
print('dout = {0}, {1}'.format(dout.num, dout.dnum))

from cffi import FFI

ffi = FFI()

lib = ffi.dlopen('./libsomelib.so')
print('Loaded lib {0}'.format(lib))

# Describe the data type and function prototype to cffi.
ffi.cdef('''
typedef struct {
    int num;
    double dnum;
} DataPoint;

DataPoint add_data(const DataPoint* dps, unsigned n);
''')

# Create an array of DataPoint structs and initialize it.
dps = ffi.new('DataPoint[]', [(2, 2.2), (3, 3.3), (4, 4.4), (5, 5.5)])

print('Calling add_data via cffi')
# Interesting variation: passing invalid arguments to add_data will trigger
# a cffi type-checking exception.
dout = lib.add_data(dps, 4)
print('dout = {0}, {1}'.format(dout.num, dout.dnum))

from ctypes import (CDLL, byref, Structure, POINTER, c_int,
                    c_void_p, c_long, c_ushort, c_ubyte,
                    c_char, c_char_p, c_void_p)

# CDLL(None) invokes dlopen(NULL), which loads the currently running
# process - in our case Python itself. Since Python is linked with
# libc, readdir_r will be found there.
# Alternatively, we can just explicitly load 'libc.so.6'.
lib = CDLL(None)
print('Loaded lib {0}'.format(lib))

# Describe the types needed for readdir_r.
class DIRENT(Structure):
    _fields_ = [('d_ino', c_long),
                ('d_off', c_long),
                ('d_reclen', c_ushort),
                ('d_type', c_ubyte),
                ('d_name', c_char * 256)]

DIR_p = c_void_p
DIRENT_p = POINTER(DIRENT)
DIRENT_pp = POINTER(DIRENT_p)

# Load the functions we need from the C library. Specify their
# argument and return types.
readdir_r = lib.readdir_r
readdir_r.argtypes = [DIR_p, DIRENT_p, DIRENT_pp]
readdir_r.restype = c_int

opendir = lib.opendir
opendir.argtypes = [c_char_p]
opendir.restype = DIR_p

closedir = lib.closedir
closedir.argtypes = [DIR_p]
closedir.restype = c_int

# opendir's path argument is char*, hence bytes.
path = b'/tmp'
dir_fd = opendir(path)
if not dir_fd:
    raise RuntimeError('opendir failed')

dirent = DIRENT()
result = DIRENT_p()

while True:
    # Note that byref() here is optional since ctypes can do it on its
    # own by observing the argtypes declared for readdir_r. I keep byref
    # for explicitness.
    if readdir_r(dir_fd, byref(dirent), byref(result)):
        raise RuntimeError('readdir_r failed')
    if not result:
        # If (*result == NULL), we're done.
        break
    # dirent.d_name is char[], hence we decode it to get a unicode
    # string.
    print('Found: ' + dirent.d_name.decode('utf-8'))

closedir(dir_fd)

from cffi import FFI

ffi = FFI()
ffi.cdef("""
    typedef void DIR;
    typedef long ino_t;
    typedef long off_t;

    struct dirent {
        ino_t          d_ino;       /* inode number */
        off_t          d_off;       /* offset to the next dirent */
        unsigned short d_reclen;    /* length of this record */
        unsigned char  d_type;      /* type of file; not supported
                                       by all file system types */
        char           d_name[256]; /* filename */
    };

    DIR *opendir(const char *name);
    int readdir_r(DIR *dirp, struct dirent *entry, struct dirent **result);
    int closedir(DIR *dirp);
""")

# Load symbols from the current process (Python).
lib = ffi.dlopen(None)
print('Loaded lib {0}'.format(lib))

path = b'/tmp'
dir_fd = lib.opendir(path)
if not dir_fd:
    raise RuntimeError('opendir failed')

# Allocate the pointers passed to readdir_r.
dirent = ffi.new('struct dirent*')
result = ffi.new('struct dirent**')

while True:
    if lib.readdir_r(dir_fd, dirent, result):
        raise RuntimeError('readdir_r failed')
    if result[0] == ffi.NULL:
        # If (*result == NULL), we're done.
        break
    print('Found: ' + ffi.string(dirent.d_name).decode('utf-8'))

lib.closedir(dir_fd)

Here’s a sample C library compiled into libsomelib.so. First, the header file somelib.h:

#ifndef SOMELIB_H
#define SOMELIB_H

typedef struct {
    int num;
    double dnum;
} DataPoint;

DataPoint add_data(const DataPoint* dps, unsigned n);

#endif /* SOMELIB_H */

And the implementation, somelib.c:

#include "somelib.h"

DataPoint add_data(const DataPoint* dps, unsigned n) {
    DataPoint out = {.num = 0, .dnum = 0.0};

    for (unsigned i = 0; i < n; ++i) {
        out.num += dps[i].num;
        out.dnum += dps[i].dnum;
    }

    return out;
}

Dynamically loading libsomelib.so at runtime and calling add_data from C code is straightforward:

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>

#include "somelib.h"

// Prototype for a function pointer for add_data
typedef DataPoint (*add_data_fn_t)(const DataPoint* dps, unsigned n);

int main(int argc, const char* argv[])
{
    void* libhandle = dlopen("./libsomelib.so", RTLD_LAZY);
    if (!libhandle) {
        fprintf(stderr, "dlopen error: %s\n", dlerror());
        exit(1);
    }

    printf("dlopen success: handle %p\n", libhandle);

    // We know the prototype of add_data so we can directly assign it to a
    // function pointer of the correct type.
    add_data_fn_t add_data_fn = dlsym(libhandle, "add_data");
    char* err = dlerror();
    if (err) {
        fprintf(stderr, "dlsym failed: %s\n", err);
        exit(1);
    }

    DataPoint dp[4] = {{2, 2.2}, {3, 3.3}, {4, 4.4}, {5, 5.5}};

    printf("Calling add_data\n");
    DataPoint dout = add_data_fn(dp, sizeof(dp) / sizeof(DataPoint));

    printf("dout = {%d, %lf}\n", dout.num, dout.dnum);
    return 0;
}

This works great. However, note a certain lack of flexibility. While the shared library can be discovered and loaded at runtime, the interface of the function we call from it has to be defined statically, at compile time – this is the function pointer prototype in the snippet above.

But what if we want the interface to be dynamic as well? In other words, what if we don’t know until runtime what arguments the called function accepts? Alas, if standard C is all we have, we’re stuck. The problem is that to call a function properly, the compiler has to know what arguments it accepts to translate the call to the proper machine code sequence according to the system’s calling convention. When I disassemble both add_data and the call in main, I see this call sequence, which is in accordance with the System V AMD64 ABI [1]:

• dps is passed in %rdi
• n is passed in %esi
• return value is in %xmm0

So to call a function whose signature is determined at runtime, we’d have to implement the calling convention ourselves, packing the arguments into registers and stack as appropriate and unpacking the return value. Moreover, this has to be implemented for each platform the code runs on. And it goes beyond saying that such code is not portable since standard C does not provide direct access to the stack or to the registers.

Luckily, a library exists that implements all of this for us.

#include <dlfcn.h>
#include <ffi.h>
#include <stdio.h>
#include <stdlib.h>

#include "somelib.h"  // For the DataPoint type.

int main(int argc, const char* argv[])
{
    void* libhandle = dlopen("./libsomelib.so", RTLD_LAZY);
    if (!libhandle) {
        fprintf(stderr, "dlopen error: %s\n", dlerror());
        exit(1);
    }

    printf("dlopen success: handle %p\n", libhandle);

    // Assuming we don't know the prototype of add_data at compile-time, we
    // have to save the output of dlsym in a void* and then prepare the
    // calling sequence using libffi.
    void* add_data_fn = dlsym(libhandle, "add_data");
    char* err = dlerror();
    if (err) {
        fprintf(stderr, "dlsym failed: %s\n", err);
        exit(1);
    }

    // Describe the function arguments. Note that ffi_type_pointer is used
    // for any C pointer (the pointee type does not matter in the ABI).
    ffi_type* args[] = {&ffi_type_pointer, &ffi_type_uint};

    // Describe the DataPoint struct to libffi. Elements are described by a
    // NULL-terminated array of pointers to ffi_type.
    ffi_type* dp_elements[] = {&ffi_type_sint, &ffi_type_double, NULL};
    ffi_type dp_type = {.size = 0, .alignment = 0,
                        .type = FFI_TYPE_STRUCT, .elements = dp_elements};

    // Describe the interface of add_data to libffi.
    ffi_cif cif;
    ffi_status status = ffi_prep_cif(&cif, FFI_DEFAULT_ABI, 2, &dp_type,
                                     args);
    if (status != FFI_OK) {
        fprintf(stderr, "ffi_prep_cif failed: %d\n", status);
        exit(1);
    }

    // The avalues argument of ffi_call holds the addresses of arguments.
    // Since our first argument is a pointer itself, we can't just pass
    // &dp (since in C &array == array). So we create a pointer to dp and
    // pass its address.
    DataPoint dp[4] = {{2, 2.2}, {3, 3.3}, {4, 4.4}, {5, 5.5}};
    DataPoint* pdp = dp;
    unsigned nelems = sizeof(dp) / sizeof(DataPoint);
    void* values[] = {&pdp, &nelems};

    printf("Calling add_data via libffi\n");
    DataPoint dout;
    ffi_call(&cif, FFI_FN(add_data_fn), &dout, values);

    printf("dout = {%d, %lf}\n", dout.num, dout.dnum);
    return 0;
}