Eli Bendersky's website - Linux

Building static binaries with Go on Linux

2024-07-30T14:35:00-07:00

One of Go's advantages is being able to produce statically-linked binaries [1]. This doesn't mean that Go always produces such binaries by default, however; in some scenarios it requires extra work to make this happen. Specifics here are OS-dependent; here we focus on Unix systems.

Basics - hello world

This post goes over a series of experiments: we take simple programs and use go build to produce binaries on a Linux machine. We then examine whether the produced binary is statically or dynamically linked. The first example is a simple "hello, world":

package main

import "fmt"

func main() {
  fmt.Println("hello world")
}

After building it with go build, we get a binary. There are a few ways on Linux to determine whether a binary is statically or dynamically linked. One is the file tool:

$ file ./helloworld
helloworld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Flm7stIXKLPfvBhTgXmR/PPwdjFUEkc9NCSPRC7io/PofU_qoulSqJ0Ktvgx5g/eQXbAL15zCEIXOBSPZgY, with debug_info, not stripped

You can see it says "statically linked". Another way is to use ldd, which prints the shared object dependencies of a given binary:

$ ldd ./helloworld
  not a dynamic executable

Alternatively, we can also use the ubiquitous nm tool, asking it to list the undefined symbols in a binary (these are symbols the binary expects the dynamic linker to provide at run-time from shared objects):

$ nm -u ./helloworld
<empty output>

All of these tell us that a simple helloworld is a statically-linked binary. Throughout the post I'll mostly be using ldd (out of habit), but you can use any approach you like.

DNS and user groups

There are two pieces of functionality the Go standard library defers to the system's libc on Unix machines, when some conditions are met. When cgo is enabled (as it often - but not always - is on Unix machines), Go will call the C library for DNS lookups in the net package and for user and group ID lookups in the os/user package.

Let's observe this with an experiment:

package main

import (
  "fmt"
  "net"
)

func main() {
  fmt.Println(net.LookupHost("go.dev"))
}

If we build this program, we notice it's dynamically linked, expecting to load a libc shared object at run-time:

$ go build lookuphost.go
$ ldd ./lookuphost
  linux-vdso.so.1 (0x00007b50cb22a000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007b50cae00000)
  /lib64/ld-linux-x86-64.so.2 (0x00007b50cb22c000)

This is explained in the net package documentation in some detail. The Go standard library does have a pure Go implementation of this functionality (although it may lack some advanced features). We can ask the toolchain to use it in a couple of ways. First, we can set the netgo build tag:

$ go build -tags netgo lookuphost.go
$ ldd ./lookuphost
  not a dynamic executable

Second, we can disable cgo entirely with the CGO_ENABLED env var. This env var is usually on by default on Unix systems:

$ go env CGO_ENABLED
1

If we disable it explicitly for our build, we'll get a static binary again:

$ CGO_ENABLED=0 go build lookuphost.go
$ ldd ./lookuphost
  not a dynamic executable

Similarly, some of the functionality of the os/user package uses libc by default. Here's an example:

package main

import (
  "encoding/json"
  "log"
  "os"
  "os/user"
)

func main() {
  user, err := user.Lookup("bob")
  if err != nil {
    log.Fatal(err)
  }

  je := json.NewEncoder(os.Stdout)
  je.Encode(user)
}

This produces a dynamically-linked binary:

$ go build userlookup.go
$ ldd ./userlookup
  linux-vdso.so.1 (0x0000708301084000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000708300e00000)
  /lib64/ld-linux-x86-64.so.2 (0x0000708301086000)

As with net, we can ask the Go toolchain to use the pure Go implementation of this user lookup functionality. The build tag for this is osusergo:

$ go build -tags osusergo userlookup.go
$ ldd ./userlookup
  not a dynamic executable

Or, we can disable cgo:

$ CGO_ENABLED=0 go build userlookup.go
$ ldd ./userlookup
  not a dynamic executable

Linking C into our go binary

We've seen that the standard library has some functionality that may require dynamic linking by default, but this is relatively easy to override. What happens when we actually have C code as part of our Go program, though?

Go supports C extensions and FFI using cgo. For example:

package main

// #include <stdio.h>
// void helloworld() {
//   printf("hello, world from C\n");
// }
import "C"

func main() {
  C.helloworld()
}

A program built from this source will be dynamically linked, due to cgo:

$ go build cstdio.go
$ ldd ./cstdio
  linux-vdso.so.1 (0x00007bc6d68e3000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007bc6d6600000)
  /lib64/ld-linux-x86-64.so.2 (0x00007bc6d68e5000)

In our C code, printf is a call to libc; even if we don't explicitly call into the C runtime in our C code, cgo may do it in the scaffolding code it generates.

Note that cgo may be involved even if your project has no C code of its own; several dependencies may bring in cgo. Some popular packages - like the go-sqlite3 driver - depend on cgo, and importing them will impose a cgo requirement on a program.

Obviously, building with CGO_ENABLED=0 is no longer an option. So what's the recourse?

Linking a `libc` statically

To recap, once we have C code as part of our Go binary, it's going to be dynamically linked on Unix, because:

The C code calls into libc (the C runtime)
The libc typically used on Unix systems is glibc
The recommended way to link to glibc is dynamically (for various technical and license-related reasons that are outside the scope of this post)
Therefore, go build produces dynamically-linked Go binaries

To change this flow of events, we can interpose at step (2) - use a different libc implementation, one that's statically linked. Luckily, such an implementation exists and is well used and tested - musl.

To follow along, start by installing musl. The standard instructions using ./configure --prefix=<MUSLDIR> and make / make install work well. We'll use $MUSLDIR to refer to the directory where musl is installed. musl comes with a gcc wrapper that makes it easy to pass all the right flags. To re-build our cstdio example using musl, run:

$ CC=$MUSLDIR/bin/musl-gcc go build --ldflags '-linkmode external -extldflags "-static"' cstdio.go
$ ldd ./cstdio
  not a dynamic executable

The CC env var tells go build which C compiler to use for cgo; the linker flags instruct it to use an external linker for the final build (read this for the gory details) and then to perform a static link.

This approach works for more complex use cases as well! I won't paste the code here, but the sample repository accompanying this post has a file called use-sqlite.go; it uses the go-sqlite3 package. Try go build-ing it normally and observe the dynamically linked binary produced; next, try to build it with the flags shown above to use musl, and observe that the produced binary will be statically linked.

Another curious tidbit is that we now have another way to build a statically-linked lookuphost program - by linking it with musl:

$ CC=$MUSLDIR/bin/musl-gcc go build --ldflags '-linkmode external -extldflags "-static"' lookuphost.go
$ ldd ./lookuphost
  not a dynamic executable

Since we didn't provide -tags netgo and didn't disable cgo, the Go toolchain uses calls into libc to implement DNS lookup; however, since these calls end up in the statically-linked musl, the final binary is statically linked!

Using Zig as our C compiler

Another alternative emerged recently to achieve what we want: using the Zig toolchain. Zig is a new systems programming language, which uses a bundled toolchain approach similar to Go. Its toolchain bundles together a Zig compiler, C/C++ compiler, linker and libc for static linking. Therefore, Zig can actually be used to link Go binaries statically with C code!

Instead of installing musl, we could instead install Zig and use its x86_64-linux-musl target (adjust the architecture if needed). This is done by pointing to the zig binary as our CC= env var; assuming Zig is installed in $ZIGDIR:

$ CC="$ZIGDIR/zig cc -target x86_64-linux-musl" go build cstdio.go
$ CC="$ZIGDIR/zig cc -target x86_64-linux-musl" go build use-sqlite.go

These will produce statically-linked Go binaries; the zig driver takes care of setting the right linker flags automatically, so the command-line ends up being slightly simpler than invoking musl-gcc. Another advantage of Zig here is that enables cross-compilation of Go programs that include C code [2].

I did find some issues with this approach, however; for example, attempting to link the lookuphost.go sample fails with a slew of linker errors.

Summary

Making sure Go produces a statically-linked binary on Linux takes a little bit of effort, but works well overall.

There's a long standing accepted proposal about adding a -static flag to go build that would take care of setting up all the flags required for a static build. AFAICT, the proposal is just waiting for someone with enough grit and dedication to implement and test it in all the interesting scenarios.

Code

The code for all the experiments described in this post is available on GitHub.

[1]	A statically-linked binary doesn't have run-time dependencies on other libraries (typically in the form of shared objects), not even the C runtime library (`libc`). I wrote much more about this topic in the past.

[2]

Go is well-known for its cross-compilation capabilities, but it depends on the C toolchain to compile C code. Therefore, when cgo is involved, cross-compilation is challenging. Zig can help with this because its toolchain supports cross compilation for Zig and C! It does so by bundling LLVM with a bunch of targets linked in.

Replacing my home desktop computer

2022-11-02T20:31:00-07:00

After 9 years, it's time to retire my old and faithful home Linux machine. It has served me extremely well, by far the longest time I've used a home computer without any replacements. The fans were starting to show their age, to the extent that the CPU fan sometimes needs a nudge to start after the machine has been off. While this can be fixed, I also wanted a somewhat faster machine with more memory.

I've long been keeping a curious eye on system76; they build and sell HW that was designed for Linux from the start. While my old machine - which was just a custom build - handled Linux fine over the years, it did exhibit some snags here and there (Bluetooth, for example). It would be nice not to deal with this and have HW that just works.

So I went ahead and ordered a "Meerkat" model, which is a really tiny 4.5"x4.5" box (with a comically large power adapter) that manages to pack 32 GiB of memory, a sizable SSD and a 4-core (8-thread) i5-1135G7 CPU. It came with Ubuntu 22.04 preinstalled and was very quick and easy to set up. I moved all my daily operations to this machine a couple of hours after unpacking it. Just the computer itself changed - it's still the same monitor, keyboard and mouse. I could finally use my Bluetooth earphones with it, without any issues.

It's about 33% faster than my older machine (at least for compiling the Go toolchain and medium-sized Rust projects), and it's nice to have twice as much memory and a larger SSD. I'm quite happy with it so far. It's not the cheapest option out there, but it is a "pay for high quality with minimum fuss" option.

Update 2024.11.02: I replaced this computer after two years. It's still working great, but was becoming a bit underpowered for my needs, especially because it lacks a GPU. Being pretty happy with System76, I went for a Thelio Mira with an i7-13700K CPI (16 Cores - 24 Threads 8P+8E), 64 GiB of memory and an inexpensive gaming-class Nvidia GPU.

Unix domain sockets in Go

2019-02-12T05:27:00-08:00

When it comes to inter-process communication (IPC) between processes on the same Linux host, there are multiple options: FIFOs, pipes, shared memory, sockets and so on. One of the most interesting options is Unix Domain Sockets that combine the convenient API of sockets with the higher performance of the other single-host methods.

This post demonstrates some basic examples of using Unix domain sockets with Go and explores some benchmarks comparing them to TCP loop-back sockets.

Unix domain sockets (UDS)

Unix domain sockets (UDS) have a long history, going back to the original BSD socket specification in the 1980s. The Wikipedia definition is:

A Unix domain socket or IPC socket (inter-process communication socket) is a data communications endpoint for exchanging data between processes executing on the same host operating system.

UDS support streams (TCP equivalent) and datagrams (UDP equivalent); this post focuses on the stream APIs.

IPC with UDS looks very similar to IPC with regular TCP sockets using the loop-back interface (localhost or 127.0.0.1), but there is a key difference: performance. While the TCP loop-back interface can skip some of the complexities of the full TCP/IP network stack, it retains many others (ACKs, TCP flow control, and so on). These complexities are designed for reliable cross-machine communication, but on a single host they're an unnecessary burden. This post will explore some of the performance advantages of UDS.

There are some additional differences. For example, since UDS use paths in the filesystem as their addresses, we can use directory and file permissions to control access to sockets, simplifying authentication. I won't list all the differences here; for more information feel free to check out the Wikipedia link and additional resources like Beej's UNIX IPC guide

The big disadvantage of UDS compared to TCP sockets is the single-host restriction, of course. For code written to use TCP sockets we only have to change the address from local to remote and everything keeps working. That said, the performance advantages of UDS are significant enough, and the API is similar enough to TCP sockets that it's quite possible to write code that supports both (UDS on a single host, TCP for remote IPC) with very little difficulty.

Using Unix domain sockets in Go

Let's start with a basic example of a server in Go that listens on a UNIX domain socket:

const SockAddr = "/tmp/echo.sock"

func echoServer(c net.Conn) {
    log.Printf("Client connected [%s]", c.RemoteAddr().Network())
    io.Copy(c, c)
    c.Close()
}

func main() {
    if err := os.RemoveAll(SockAddr); err != nil {
        log.Fatal(err)
    }

    l, err := net.Listen("unix", SockAddr)
    if err != nil {
        log.Fatal("listen error:", err)
    }
    defer l.Close()

    for {
        // Accept new connections, dispatching them to echoServer
        // in a goroutine.
        conn, err := l.Accept()
        if err != nil {
            log.Fatal("accept error:", err)
        }

        go echoServer(conn)
    }
}

UDS are identified with paths in the file system; for our server here we use /tmp/echo.sock. The server begins by removing this file if it exists, what is that about?

When servers shut down, the file representing the socket can remain in the file system unless the server did orderly cleanup after itself. If we re-run another server with the same socket path, we may get the error:

$ go run simple-echo-server.go
2019/02/08 05:41:33 listen error:listen unix /tmp/echo.sock: bind: address already in use

To prevent that, the server begins by removing the socket file, if it exists [1].

Now that the server is running, we can interact with it using Netcat, which can be asked to connect to UDS with the -U flag:

$ nc -U /tmp/echo.sock

Whatever you type in, the server will echo back. Press ^D to terminate the session. Alternatively, we can write a simple client in Go that connects to the server, sends it a message, waits for a response and exits. The full code for the client is here, but the important part is the connection:

c, err := net.Dial("unix", "/tmp/echo.sock")

We can see that writing UDS servers and clients is very similar to writing regular socket servers and clients. The only difference is having to pass "unix" as the network parameter of net.Listen and net.Dial; the rest of the code remains the same. Obviously, this makes it very easy to write generic server and client code that's independent of the actual kind of socket it's using.

HTTP and RPC protocols over UDS

Network protocols compose by design. High-level protocols, such as HTTP and various forms of RPC, don't particularly care about how the lower levels of the stack are implemented as long as certain guarantees are maintained.

Go's standard library comes with a small and useful rpc package that makes it trivial to throw together quick RPC servers and clients. Here's a simple server that has a single procedure defined:

const SockAddr = "/tmp/rpc.sock"

type Greeter struct {
}

func (g Greeter) Greet(name *string, reply *string) error {
    *reply = "Hello, " + *name
    return nil
}

func main() {
    if err := os.RemoveAll(SockAddr); err != nil {
        log.Fatal(err)
    }

    greeter := new(Greeter)
    rpc.Register(greeter)
    rpc.HandleHTTP()
    l, e := net.Listen("unix", SockAddr)
    if e != nil {
        log.Fatal("listen error:", e)
    }
    fmt.Println("Serving...")
    http.Serve(l, nil)
}

Note that we use the HTTP version of the server. It registers a HTTP handler with the http package, and the actual serving is done with the standard http.Serve. The network stack here looks something like this:

An RPC client that can connect to the server shown above is available here. It uses the standard rpc.Client.Call method to connect to the server.

Benchmarking UDS compared to loop-back TCP sockets

Note: benchmarking is hard, so please take these results with a grain of salt. There's some more information on benchmarking different socket types on the Redis benchmarks page and in this paper, as well as many other resources online. I also found this set of benchmarks (written in C) instructive.

I'm running two kinds of benchmarks: one for latency, and one for throughput.

For latency, the full code of the benchmark is here. Run it with -help to see what the flags are, and the code should be very straightforward to grok. The idea is to ping-pong a small packet of data (128 bytes by default) between a server and a client. The client measures how long it takes to send one such message and receive one back, and takes that combined time as "twice the latency", averaging it over many messages.

On my machine, I see average latency of ~3.6 microseconds for TCP loop-back sockets, and ~2.3 microseconds for UDS.

The throughput/bandwidth benchmark is conceptually simpler than the latency benchmark. The server listens on a socket and grabs all the data it can get (and discards it). The client sends large packets (hundreds of KB or more) and measures how long each packet takes to send; the send is done synchronously and the client expects the whole message to be sent in a single call, so it's a good approximation of bandwidth if the packet size is large enough.

Obviously, the throughput measurement is more representative with larger messages. I tried increasing them until the throughput improvements tapered off.

For smaller packet sizes, I see UDS winning over TCP: 10 GB/sec compared to 9.4 GB/sec for 512K. For much larger packet sizes 16-32 MB, the difference becomes negligible (both taper off at about 13 GB/sec). Interestingly, for some packet sizes (like 64K), TCP sockets are winning on my machine.

For very small message sizes we're getting back to latency-dominated performance, so UDS is considerably faster (more than 2x the number of packets per second compared to TCP). In most cases I'd say that the latency measurements are more important - they're more applicable to things like RPC servers and databases. In some cases like streaming video or other "big data" over sockets, you may want to pick the packet sizes carefully to optimize the performance for the specific machine you're using.

This discussion has some really insightful information about why we should expect UDS to be faster. However, beware - it's from 2005 and much in Linux has changed since then.

Unix domain sockets in the real-world Go projects

I was curious to see if UDS are actually used in real-world Go projects. They sure are! A few minutes of browsing/searching GitHub quickly uncovered UDS servers in many components of the new Go-dominated cloud infrastructure: runc, moby (Docker), k8s, lstio - pretty much every project I looked at.

That makes sense - as the benchmarks demonstrate, there are significant performance advantages to using a UDS when the client and server are both on the same host. And the API of UDS and TCP sockets is so similar that the cost of supporting both interchangeably is quite small.

[1]	For internet-domain socket, the same issue exists with ports that are marked taken by processes that die without cleanup. The `SO_REUSEADDR` socket option exists to address this problem.

Measuring context switching and memory overheads for Linux threads

2018-09-04T05:35:00-07:00

In this post I want to explore the costs of threads on modern Linux machines, both in terms of time and space. The background context is designing high-load concurrent servers, where using threads is one of the common schemes.

Important disclaimer: it's not my goal here to provide an opinion in the threads vs. event-driven models debate. Ultimately, both are tools that work well in some scenarios and less well in others. That said, one of the major criticisms of a thread-based model is the cost - comments like "but context switches are expensive!" or "but a thousand threads will eat up all your RAM!", and I do intend to study the data underlying such claims in more detail here. I'll do this by presenting multiple code samples and programs that make it easy to explore and experiment with these measurements.

Linux threads and NPTL

In the dark, old ages before version 2.6, the Linux kernel didn't have much specific support for threads, and they were more-or-less hacked on top of process support. Before futexes there was no dedicated low-latency synchronization solution (it was done using signals); neither was there much good use of the capabilities of multi-core systems [1].

The Native POSIX Thread Library (NPTL) was proposed by Ulrich Drepper and Ingo Molnar from Red Hat, and integrated into the kernel in version 2.6, circa 2005. I warmly recommend reading its design paper. With NPTL, thread creation time became about 7x faster, and synchronization became much faster as well due to the use of futexes. Threads and processes became more lightweight, with strong emphasis on making good use of multi-core processors. This roughly coincided with a much more efficient scheduler, which made juggling many threads in the Linux kernel even more efficient.

Even though all of this happened 13 years ago, the spirit of NPTL is still easily observable in some system programming code. For example, many thread and synchronization-related paths in glibc have nptl in their name.

Threads, processes and the clone system call

This was originally meant to be a part of this larger article, but it was getting too long so I split off a separate post on launching Linux processes and threads with clone, where you can learn about the clone system call and some measurements of how expensive it is to launch new processes and threads.

The rest of this post will assume this is familiar information and will focus on context switching and memory usage.

What happens in a context switch?

In the Linux kernel, this question has two important parts:

When does a kernel switch happen
How it happens

The following deals mostly with (2), assuming the kernel has already decided to switch to a different user thread (for example because the currently running thread went to sleep waiting for I/O).

The first thing that happens during a context switch is a switch to kernel mode, either through an explicit system call (such as write to some file or pipe) or a timer interrupt (when the kernel preempts a user thread whose time slice has expired). This requires saving the user space thread's registers and jumping into kernel code.

Next, the scheduler kicks in to figure out which thread should run next. When we know which thread runs next, there's the important bookkeeping of virtual memory to take care of; the page tables of the new thread have to be loaded into memory, etc.

Finally, the kernel restores the new thread's registers and cedes control back to user space.

All of this takes time, but how much time, exactly? I encourage you to read some additional online resources that deal with this question, and try to run benchmarks like lm_bench; what follows is my attempt to quantify thread switching time.

How expensive are context switches?

To measure how long it takes to switch between two threads, we need a benchmark that deliberatly triggers a context switch and avoids doing too much work in addition to that. This would be measuring just the direct cost of the switch, when in reality there is another cost - the indirect one, which could even be larger. Every thread has some working set of memory, all or some of which is in the cache; when we switch to another thread, all this cache data becomes unneeded and is slowly flushed out, replaced by the new thread's data. Frequent switches back and forth between the two threads will cause a lot of such thrashing.

In my benchmarks I am not measuring this indirect cost, because it's pretty difficult to avoid in any form of multi-tasking. Even if we "switch" between different asynchronous event handlers within the same thread, they will likely have different memory working sets and will interfere with each other's cache usage if those sets are large enough. I strongly recommend watching this talk on fibers where a Google engineer explains their measurement methodology and also how to avoid too much indirect switch costs by making sure closely related tasks run with temporal locality.

These code samples measure context switching overheads using two different techniques:

A pipe which is used by two threads to ping-pong a tiny amount of data. Every read on the pipe blocks the reading thread, and the kernel switches to the writing thread, and so on.
A condition variable used by two threads to signal an event to each other.

There are additional factors context switching time depends on; for example, on a multi-core CPU, the kernel can occasionally migrate a thread between cores because the core a thread has been previously using is occupied. While this helps utilize more cores, such switches cost more than staying on the same core (again, due to cache effects). Benchmarks can try to restrict this by running with taskset pinning affinity to one core, but it's important to keep in mind this only models a lower bound.

Using the two techniques I'm getting fairly similar results: somewhere between 1.2 and 1.5 microseconds per context switch, accounting only for the direct cost, and pinning to a single core to avoid migration costs. Without pinning, the switch time goes up to ~2.2 microseconds [2]. These numbers are largely consistent with the reports in the fibers talk mentioned above, and also with other benchmarks found online (like lat_ctx from lmbench).

What does this mean in practice?

So we have the numbers now, but what do they mean? Is 1-2 us a long time? As I have mentioned in the post on launch overheads, a good comparison is memcpy, which takes 3 us for 64 KiB on the same machine. In other words, a context switch is a bit quicker than copying 64 KiB of memory from one location to another.

1-2 us is not a long time by any measure, except when you're really trying to optimize for extremely low latencies or high loads.

As an example of an artificially high load, here is another benchmark which writes a short message into a pipe and expects to read it from another pipe. On the other end of the two pipes is a thread that echoes one into the other.

Running the benchmark on the same machine I used to measure the context switch times, I get ~400,000 iterations per second (this is with taskset to pin to a single core). This makes perfect sense given the earlier measurements, because each iteration of this test performs two context switches, and at 1.2 us per switch this is 2.4 us per iteration.

You could claim that the two threads compete for the same CPU, but if I don't pin the benchmark to a single core, the number of iterations per second halves. This is because the vast majority of time in this benchmark is spent in the kernel switching from one thread to the other, and the core migrations that occur when it's not pinned greatly ouweigh the loss of (the very minimal) parallelism.

Just for fun, I rewrote the same benchmark in Go; two goroutines ping-ponging short message between themselves over a channel. The throughput this achieves is dramatically higher - around 2.8 million iterations per second, which leads to an estimate of ~170 ns switching between goroutines [3]. Since switching between goroutines doesn't require an actual kernel context switch (or even a system call), this isn't too surprising. For comparison, Google's fibers use a new Linux system call that can switch between two tasks in about the same time, including the kernel time.

A word of caution: benchmarks tend to be taken too seriously. Please take this one only for what it demonstrates - a largely synthetical workload used to poke into the cost of some fundamental concurrency primitives.

Remember - it's quite unlikely that the actual workload of your task will be negligible compared to the 1-2 us context switch; as we've seen, even a modest memcpy takes longer. Any sort of server logic such as parsing headers, updating state, etc. is likely to take orders of magnitude longer. If there's one takeaway to remember from these sections is that context switching on modern Linux systems is super fast.

Memory usage of threads

Now it's time to discuss the other overhead of a large number of threads - memory. Even though all threads in a process share their, there are still areas of memory that aren't shared. In the post about clone we've mentioned page tables in the kernel, but these are comparatively small. A much larger memory area that it private to each thread is the stack.

The default per-thread stack size on Linux is usually 8 MiB, and we can check what it is by invoking ulimit:

$ ulimit -s
8192

To see this in action, let's start a large number of threads and observe the process's memory usage. This sample launches 10,000 threads and sleeps for a bit to let us observe its memory usage with external tools. Using tools like top (or preferably htop) we see that the process uses ~80 GiB of virtual memory, with about 80 MiB of resident memory. What is the difference, and how can it use 80 GiB of memory on a machine that only has 16 available?

Virtual vs. Resident memory

A short interlude on what virtual memory means. When a Linux program allocates memory (with malloc) or otherwise, this memory initially doesn't really exist - it's just an entry in a table the OS keeps. Only when the program actually accesses the memory is the backing RAM for it found; this is what virtual memory is all about.

Therefore, the "memory usage" of a process can mean two things - how much virtual memory it uses overall, and how much actual memory it uses. While the former can grow almost without bounds - the latter is obviously limited to the system's RAM capacity (with swapping to disk being the other mechanism of virtual memory to assist here if usage grows above the side of physical memory). The actual physical memory on Linux is called "resident" memory, because it's actually resident in RAM.

There's a good StackOverflow discussion of this topic; here I'll just limit myself to a simple example:

int main(int argc, char** argv) {
  report_memory("started");

  int N = 100 * 1024 * 1024;
  int* m = malloc(N * sizeof(int));
  escape(m);
  report_memory("after malloc");

  for (int i = 0; i < N; ++i) {
    m[i] = i;
  }
  report_memory("after touch");

  printf("press ENTER\n");
  (void)fgetc(stdin);
  return 0;
}

This program starts by allocating 400 MiB of memory (assuming an int size of 4) with malloc, and later "touches" this memory by writing a number into every element of the allocated array. It reports its own memory usage at every step - see the full code sample for the reporting code [4]. Here's the output from a sample run:

$ ./malloc-memusage
started: max RSS = 4780 kB; vm size = 6524 kB
after malloc: max RSS = 4780 kB; vm size = 416128 kB
after touch: max RSS = 410916 kB; vm size = 416128 kB

The most interesting thing to note is how vm size remains the same between the second and third steps, while max RSS grows from the initial value to 400 MiB. This is precisely because until we touch the memory, it's fully "virtual" and isn't actually counted for the process's RAM usage.

Therefore, distinguishing between virtual memory and RSS in realistic usage is very important - this is why the thread launching sample from the previous section could "allocate" 80 GiB of virtual memory while having only 80 MiB of resident memory.

Back to memory overhead for threads

As we've seen, a new thread on Linux is created with 8 MiB of stack space, but this is virtual memory until the thread actually uses it. If the thread actually uses its stack, resident memory usage goes up dramatically for a large number of threads. I've added a configuration option to the sample program that launches a large number of threads; with it enabled, the thread function actually uses stack memory and from the RSS report it is easy to observe the effects. Curiously, if I make each of 10,000 threads use 400 KiB of memory, the total RSS is not 4 GiB but around 2.6 GiB [5].

How to we control the stack size of threads? One option is using the ulimit command, but a better option is with the pthread_attr_setstacksize API. The latter is invoked programatically and populates a pthread_attr_t structure that's passed to thread creation. The more interesting question is - what should the stack size be set to?

As we have seen above, just creating a large stack for a thread doesn't automatically eat up all the machine's memory - not before the stack is being used. If our threads actually use large amounts of stack memory, this is a problem, because this severely limits the number of threads we can run concurrently. Note that this is not really a problem with threads - but with concurrency; if our program uses some event-driven approach to concurrency and each handler uses a large amount of memory, we'd still have the same problem.

If the task doesn't actually use a lot of memory, what should we set the stack size to? Small stacks keep the OS safe - a deviant program may get into an infinite recursion and a small stack will make sure it's killed early. Moreover, virtual memory is large but not unlimited; especially on 32-bit OSes, we might not have 80 GiB of virtual address space for the process, so a 8 MiB stack for 10,000 threads makes no sense. There's a tradeoff here, and the default chosen by 32-bit Linux is 2 MiB; the maximal virtual address space available is 3 GiB, so this imposes a limit of ~1500 threads with the default settings. On 64-bit Linux the virtual address space is vastly larger, so this limitation is less serious (though other limits kick in - on my machine the maximal number of threads the OS lets one process start is about 32K).

Therefore I think it's more important to focus on how much actual memory each concurrent task is using than on the OS stack size limit, as the latter is simply a safety measure.

Conclusion

The numbers reported here paint an interesting picture on the state of Linux multi-threaded performance in 2018. I would say that the limits still exist - running a million threads is probably not going to make sense; however, the limits have definitely shifted since the past, and a lot of folklore from the early 2000s doesn't apply today. On a beefy multi-core machine with lots of RAM we can easily run 10,000 threads in a single process today, in production. As I've mentioned above, it's highly recommended to watch Google's talk on fibers; through careful tuning of the kernel (and setting smaller default stacks) Google is able to run an order of magnitude more threads in parallel.

Whether this is sufficient concurrency for your application is very obviously project-specific, but I'd say that for higher concurrencies you'd probably want to mix in some asynchronous processing. If 10,000 threads can provide sufficient concurrency - you're in luck, as this is a much simpler model - all the code within the threads is serial, there are no issues with blocking, etc.

[1]	For example, in order to implement POSIX semantics properly, a single thread was designated as a "manager" and managed operations like "create a new thread". This created an unfortunate serialization point and a bottleneck.

[2]	These numbers also vary greatly between CPUs. The numbers reported herein are on my Haswell i7-4771. On a different contemporary machine (a low-end Xeon) I measured switch times that were about 50-75% longer.

[3] Curiously, pinning the Go program to a single core (by means of setting GOMAXPROCS=1 and running with taskset) increases the throughput by only by 10% or so. The Go scheduler is not optimized for this strange use case of endless hammering between two goroutine, but it performs very well regardless.

[4]	Note that while for resident memory there's a convenient `getrusage` API, to report virtual memory size we have to parse `/proc/PID/status`.

[5]	According to Tom Dryer, recent Linux version only approximate this usage, which could explain the discrepancy - see this explanation.

Launching Linux threads and processes with clone

2018-08-01T06:14:00-07:00

Due to variation between operating systems and the way OS courses are taught, some programmers may have an outdated mental model about the difference between processes and threads in Linux. Even the name "thread" suggests something extremely lightweight compared to a heavy "process" - a mostly wrong intuition.

In fact, for the Linux kernel itself there's absolutely no difference between what userspace sees as processes (the result of fork) and as threads (the result of pthread_create). Both are represented by the same data structures and scheduled similarly. In kernel nomenclature this is called tasks (the main structure representing a task in the kernel is task_struct), and I'll be using this term from now on.

In Linux, threads are just tasks that share some resources, most notably their memory space; processes, on the other hand, are tasks that don't share resources. For application programmers, proceses and threads are created and managed in very different ways. For processes there's a slew of process-management APIs like fork, wait and so on. For threads there's the pthread library. However, deep in the guts of these APIs and libraries, both processes and threads come into existence through a single Linux system call - clone.

The `clone` system call

We can think of clone as the unifying implementation shared between processes and threads. Whatever perceived difference there is between processes and threads on Linux is achieved through passing different flags to clone. Therefore, it's most useful to think of processes and threads not as two completely different concepts, but rather as two variants of the same concept - starting a concurrent task. The differences are mostly about what is shared between this new task and the task that started it.

Here is a code sample demonstrating the most important sharing aspect of threads - memory. It uses clone in two ways, once with the CLONE_VM flag and once without. CLONE_VM tells clone to share the virtual memory between the calling task and the new task clone is about to create [1]. As we'll see later on, this is the flag used by pthread_create:

static int child_func(void* arg) {
  char* buf = (char*)arg;
  printf("Child sees buf = \"%s\"\n", buf);
  strcpy(buf, "hello from child");
  return 0;
}

int main(int argc, char** argv) {
  // Allocate stack for child task.
  const int STACK_SIZE = 65536;
  char* stack = malloc(STACK_SIZE);
  if (!stack) {
    perror("malloc");
    exit(1);
  }

  // When called with the command-line argument "vm", set the CLONE_VM flag on.
  unsigned long flags = 0;
  if (argc > 1 && !strcmp(argv[1], "vm")) {
    flags |= CLONE_VM;
  }

  char buf[100];
  strcpy(buf, "hello from parent");
  if (clone(child_func, stack + STACK_SIZE, flags | SIGCHLD, buf) == -1) {
    perror("clone");
    exit(1);
  }

  int status;
  if (wait(&status) == -1) {
    perror("wait");
    exit(1);
  }

  printf("Child exited with status %d. buf = \"%s\"\n", status, buf);
  return 0;
}

Some things to note when clone is invoked:

It takes a function pointer to the code the new task will run, similarly to threading APIs, and unlike the fork API. This is the glibc wrapper for clone. There's also a raw system call which is discussed below.
The stack for the new task has to be allocated by the parent and passed into clone.
The SIGCHLD flag tells the kernel to send the SIGCHLD to the parent when the child terminates, which lets the parent use the plain wait call to wait for the child to exit. This is the only flag the sample passes into clone by default.

This code sample passes a buffer into the child, and the child writes a string into it. When called without the vm command-line argument, the CLONE_VM flag is off, and the parent's virtual memory is copied into the child. The child sees the message the parent placed in buf, but whatever it writes into buf goes into its own copy and the parent can't see it. Here's the output:

$ ./clone-vm-sample
Child sees buf = "hello from parent"
Child exited with status 0. buf = "hello from parent"

But when the vm argument is passed, CLONE_VM is set and the child task shares the parent's memory. Its writing into buf will now be observable from the parent:

$ ./clone-vm-sample vm
Child sees buf = "hello from parent"
Child exited with status 0. buf = "hello from child"

A bunch of other CLONE_* flags can specify other things that will be shared with the parent: CLONE_FILES will share the open file descriptors, CLONE_SIGHAND will share the signal dispositions, and so on.

Other flags are there to implement the semantics required by POSIX threads. For example, CLONE_THREAD asks the kernel to assign the same thread group id to the child as to the parent, in order to comply with POSIX's requirement of all threads in a process sharing a single process ID [2].

Calling `clone` in process and thread creation

Let's dig through some code in glibc to see how clone is invoked, starting with fork, which is routed to __libc_fork in sysdeps/nptl/fork.c. The actual implementation is specific to the threading library, hence the location in the nptl folder. The first thing __libc_fork does is invoke the fork handlers potentially registered beforehead with pthread_atfork.

The actual cloning happens with:

pid = ARCH_FORK ();

Where ARCH_FORK is a macro defined per architecture (exact syscall ABIs are architecture-specific). For x86_64 it maps to:

#define ARCH_FORK() \
  INLINE_SYSCALL (clone, 4,                                                   \
                  CLONE_CHILD_SETTID | CLONE_CHILD_CLEARTID | SIGCHLD, 0,     \
                  NULL, &THREAD_SELF->tid)

The CLONE_CHILD_* flags are useful for some threading libraries (though not the default on Linux today - NPTL). Otherwise, the invocation is very similar to the clone code sample shown in the previous section.

You may wonder where is the function pointer in this call. Nice catch! This is the raw call version of clone, where execution continues from the point of the call in both parent and child - close to the usual semantics of fork.

Now let's turn to pthread_create. Through a dizzying chain of macros it reaches a function named create_thread (defined in sysdeps/unix/sysv/linux/createthread.c) that calls clone with:

const int clone_flags = (CLONE_VM | CLONE_FS | CLONE_FILES | CLONE_SYSVSEM
                       | CLONE_SIGHAND | CLONE_THREAD
                       | CLONE_SETTLS | CLONE_PARENT_SETTID
                       | CLONE_CHILD_CLEARTID
                       | 0);

ARCH_CLONE (&start_thread, STACK_VARIABLES_ARGS,
            clone_flags, pd, &pd->tid, tp, &pd->tid)

Browse through man 2 clone to understand the flags passed into the call. Briefly, it is asked to share the virtual memory, file system, open files, shared memory and signal handlers with the parent thread/process. Additional flags are passed to implement proper identification - all threads launched from a single process have to share its process ID to be POSIX compliant.

Reading the glibc source code is quite an exercise in mental resilience, but it's really interesting to see how everything fits together "in the real world".

Benchmarking process vs. thread creation

Given the information presented earlier in the post, I would expect process creation to be somewhat more expensive than thread creation, but not dramatically so. Since fork and pthread_create route to the same system call in Linux, the difference would come from the different flags they pass in. When pthread_create passes all these CLONE_* flags, it tells the kernel there's no need to copy the virtual memory image, the open files, the signal handlers, and so on. Obviously, this saves time.

For processes, there's a bit of copying to be done when fork is invoked, which costs time. The biggest chunk of time probably goes to copying the memory image due to the lack of CLONE_VM. Note, however, that it's not just copying the whole memory; Linux has an important optimization by using COW (Copy On Write) pages. The child's memory pages are initially mapped to the same pages shared by the parent, and only when we modify them the copy happens. This is very important because processes will often use a lot of shared read-only memory (think of the global structures used by the standard library, for example).

That said, the page tables still have to be copied. The size of a process's page tables can be observed by looking in /proc/<pid>/status - the VmPTE indicator. These can be around tens of kilobytes for small processes, and higher for larger processes. Not a lot of data to copy, but definitely some extra work for the CPU.

I wrote a benchmark that times process and threads launches, as a function of the virtual memory allocated before fork or pthread_create. The launch is averaged over 10,000 instances to remove warm-up effects and jitter:

Several things to note:

Indeed, launching processes is slower than threads, 35 vs. 5 microseconds for a 2-MB heap. But it's still very fast! 35 micro-seconds is not a lot of time at all. If your latency budget is willing to tolerate a 5 us overhead, it will almost certainly be fine with a 35 us overhead, unless you're working on some super-tight hard realtime system (in which case you shouldn't be using Linux!)
As expected, the time to launch a process when the heap is larger grows. The time delta is the time needed to copy the extra page table entries. For threads, on the other hand, there is absolutely no difference since the memory is completely shared.

Interestingly, it's easy to observe from these numbers that not the whole memory image is being copied. On the same machine this benchmark was run on, just a simple memcpy of 2 MB takes over 60 us, so it couldn't have copied 2 MB of heap to the child in the 30 us difference. Copying 64K (a reasonable size for a page table) takes 3 us, which makes sense because the cloning involves more than a simple memcpy. To me this is another sign of how fast these launches are, since we're in the same ballpark of performance with modestly sized memory copies.

Creation time is not the only performance benchmark of importance. It's also interesting to measure how long it takes to switch context between tasks when using threads or processes. This is covered in another post.

[1]	It may be just me, but I find this terminology a bit confusing. In my mind the word clone is synonymous to copy, so when we turn on a flag named "clone the VM" I'd expect the VM to be copied rather than shared. IMHO it would be clearer if this flag was named `SHARE_VM`.

[2]

It's certainly interesting to see this evolution of concepts over time. Thread APIs were defined in times where there was a real difference between processes and threads and their design reflects that. In modern Linux the kernel has to bend over backwards to provide the illusion of the difference although very little of it exists.