Eli Bendersky's website - Linkers and Loaders

Building static binaries with Go on Linux

2024-07-30T14:35:00-07:00

One of Go's advantages is being able to produce statically-linked binaries [1]. This doesn't mean that Go always produces such binaries by default, however; in some scenarios it requires extra work to make this happen. Specifics here are OS-dependent; here we focus on Unix systems.

Basics - hello world

This post goes over a series of experiments: we take simple programs and use go build to produce binaries on a Linux machine. We then examine whether the produced binary is statically or dynamically linked. The first example is a simple "hello, world":

package main

import "fmt"

func main() {
  fmt.Println("hello world")
}

After building it with go build, we get a binary. There are a few ways on Linux to determine whether a binary is statically or dynamically linked. One is the file tool:

$ file ./helloworld
helloworld: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=Flm7stIXKLPfvBhTgXmR/PPwdjFUEkc9NCSPRC7io/PofU_qoulSqJ0Ktvgx5g/eQXbAL15zCEIXOBSPZgY, with debug_info, not stripped

You can see it says "statically linked". Another way is to use ldd, which prints the shared object dependencies of a given binary:

$ ldd ./helloworld
  not a dynamic executable

Alternatively, we can also use the ubiquitous nm tool, asking it to list the undefined symbols in a binary (these are symbols the binary expects the dynamic linker to provide at run-time from shared objects):

$ nm -u ./helloworld
<empty output>

All of these tell us that a simple helloworld is a statically-linked binary. Throughout the post I'll mostly be using ldd (out of habit), but you can use any approach you like.

DNS and user groups

There are two pieces of functionality the Go standard library defers to the system's libc on Unix machines, when some conditions are met. When cgo is enabled (as it often - but not always - is on Unix machines), Go will call the C library for DNS lookups in the net package and for user and group ID lookups in the os/user package.

Let's observe this with an experiment:

package main

import (
  "fmt"
  "net"
)

func main() {
  fmt.Println(net.LookupHost("go.dev"))
}

If we build this program, we notice it's dynamically linked, expecting to load a libc shared object at run-time:

$ go build lookuphost.go
$ ldd ./lookuphost
  linux-vdso.so.1 (0x00007b50cb22a000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007b50cae00000)
  /lib64/ld-linux-x86-64.so.2 (0x00007b50cb22c000)

This is explained in the net package documentation in some detail. The Go standard library does have a pure Go implementation of this functionality (although it may lack some advanced features). We can ask the toolchain to use it in a couple of ways. First, we can set the netgo build tag:

$ go build -tags netgo lookuphost.go
$ ldd ./lookuphost
  not a dynamic executable

Second, we can disable cgo entirely with the CGO_ENABLED env var. This env var is usually on by default on Unix systems:

$ go env CGO_ENABLED
1

If we disable it explicitly for our build, we'll get a static binary again:

$ CGO_ENABLED=0 go build lookuphost.go
$ ldd ./lookuphost
  not a dynamic executable

Similarly, some of the functionality of the os/user package uses libc by default. Here's an example:

package main

import (
  "encoding/json"
  "log"
  "os"
  "os/user"
)

func main() {
  user, err := user.Lookup("bob")
  if err != nil {
    log.Fatal(err)
  }

  je := json.NewEncoder(os.Stdout)
  je.Encode(user)
}

This produces a dynamically-linked binary:

$ go build userlookup.go
$ ldd ./userlookup
  linux-vdso.so.1 (0x0000708301084000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x0000708300e00000)
  /lib64/ld-linux-x86-64.so.2 (0x0000708301086000)

As with net, we can ask the Go toolchain to use the pure Go implementation of this user lookup functionality. The build tag for this is osusergo:

$ go build -tags osusergo userlookup.go
$ ldd ./userlookup
  not a dynamic executable

Or, we can disable cgo:

$ CGO_ENABLED=0 go build userlookup.go
$ ldd ./userlookup
  not a dynamic executable

Linking C into our go binary

We've seen that the standard library has some functionality that may require dynamic linking by default, but this is relatively easy to override. What happens when we actually have C code as part of our Go program, though?

Go supports C extensions and FFI using cgo. For example:

package main

// #include <stdio.h>
// void helloworld() {
//   printf("hello, world from C\n");
// }
import "C"

func main() {
  C.helloworld()
}

A program built from this source will be dynamically linked, due to cgo:

$ go build cstdio.go
$ ldd ./cstdio
  linux-vdso.so.1 (0x00007bc6d68e3000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007bc6d6600000)
  /lib64/ld-linux-x86-64.so.2 (0x00007bc6d68e5000)

In our C code, printf is a call to libc; even if we don't explicitly call into the C runtime in our C code, cgo may do it in the scaffolding code it generates.

Note that cgo may be involved even if your project has no C code of its own; several dependencies may bring in cgo. Some popular packages - like the go-sqlite3 driver - depend on cgo, and importing them will impose a cgo requirement on a program.

Obviously, building with CGO_ENABLED=0 is no longer an option. So what's the recourse?

Linking a `libc` statically

To recap, once we have C code as part of our Go binary, it's going to be dynamically linked on Unix, because:

The C code calls into libc (the C runtime)
The libc typically used on Unix systems is glibc
The recommended way to link to glibc is dynamically (for various technical and license-related reasons that are outside the scope of this post)
Therefore, go build produces dynamically-linked Go binaries

To change this flow of events, we can interpose at step (2) - use a different libc implementation, one that's statically linked. Luckily, such an implementation exists and is well used and tested - musl.

To follow along, start by installing musl. The standard instructions using ./configure --prefix=<MUSLDIR> and make / make install work well. We'll use $MUSLDIR to refer to the directory where musl is installed. musl comes with a gcc wrapper that makes it easy to pass all the right flags. To re-build our cstdio example using musl, run:

$ CC=$MUSLDIR/bin/musl-gcc go build --ldflags '-linkmode external -extldflags "-static"' cstdio.go
$ ldd ./cstdio
  not a dynamic executable

The CC env var tells go build which C compiler to use for cgo; the linker flags instruct it to use an external linker for the final build (read this for the gory details) and then to perform a static link.

This approach works for more complex use cases as well! I won't paste the code here, but the sample repository accompanying this post has a file called use-sqlite.go; it uses the go-sqlite3 package. Try go build-ing it normally and observe the dynamically linked binary produced; next, try to build it with the flags shown above to use musl, and observe that the produced binary will be statically linked.

Another curious tidbit is that we now have another way to build a statically-linked lookuphost program - by linking it with musl:

$ CC=$MUSLDIR/bin/musl-gcc go build --ldflags '-linkmode external -extldflags "-static"' lookuphost.go
$ ldd ./lookuphost
  not a dynamic executable

Since we didn't provide -tags netgo and didn't disable cgo, the Go toolchain uses calls into libc to implement DNS lookup; however, since these calls end up in the statically-linked musl, the final binary is statically linked!

Using Zig as our C compiler

Another alternative emerged recently to achieve what we want: using the Zig toolchain. Zig is a new systems programming language, which uses a bundled toolchain approach similar to Go. Its toolchain bundles together a Zig compiler, C/C++ compiler, linker and libc for static linking. Therefore, Zig can actually be used to link Go binaries statically with C code!

Instead of installing musl, we could instead install Zig and use its x86_64-linux-musl target (adjust the architecture if needed). This is done by pointing to the zig binary as our CC= env var; assuming Zig is installed in $ZIGDIR:

$ CC="$ZIGDIR/zig cc -target x86_64-linux-musl" go build cstdio.go
$ CC="$ZIGDIR/zig cc -target x86_64-linux-musl" go build use-sqlite.go

These will produce statically-linked Go binaries; the zig driver takes care of setting the right linker flags automatically, so the command-line ends up being slightly simpler than invoking musl-gcc. Another advantage of Zig here is that enables cross-compilation of Go programs that include C code [2].

I did find some issues with this approach, however; for example, attempting to link the lookuphost.go sample fails with a slew of linker errors.

Summary

Making sure Go produces a statically-linked binary on Linux takes a little bit of effort, but works well overall.

There's a long standing accepted proposal about adding a -static flag to go build that would take care of setting up all the flags required for a static build. AFAICT, the proposal is just waiting for someone with enough grit and dedication to implement and test it in all the interesting scenarios.

Code

The code for all the experiments described in this post is available on GitHub.

[1]	A statically-linked binary doesn't have run-time dependencies on other libraries (typically in the form of shared objects), not even the C runtime library (`libc`). I wrote much more about this topic in the past.

[2]

Go is well-known for its cross-compilation capabilities, but it depends on the C toolchain to compile C code. Therefore, when cgo is involved, cross-compilation is challenging. Zig can help with this because its toolchain supports cross compilation for Zig and C! It does so by bundling LLVM with a bunch of targets linked in.

Some notes on Luz - an assembler, linker and CPU simulator

2017-01-05T06:27:00-08:00

A few years ago I wrote about Luz - a self-educational project to implement a CPU simulator and a toolchain for it, consisting of an assembler and a linker. Since then, I received some questions by email that made me realize I could do a better job explaining what the project is and what one can learn from it.

So I went back to the Luz repository and fixed it up to be more modern, in-line with current documentation standards on GitHub. The landing README page should now provide a good overview, but I also wanted to write up some less formal documentation I could point to - a place to show-off some of the more interesting features in Luz; a blog post seemed like the perfect medium for this.

As before, it makes sense to start with the Luz toplevel diagram:

Luz is a collection of related libraries and programs written in Python, implementing all the stages shown in the diagram above.

The CPU simulator

The Luz CPU is inspired by MIPS (for the instruction set), by Altera Nios II (for the way "peripherals" are attached to the CPU), and by MPC 555 (for the memory controller) and is aimed at embedded uses, like Nios II. The Luz user manual lists the complete instruction set explaining what each instructions means.

The simulator itself is functional only - it performs the instructions one after the other, without trying to simulate how long their execution takes. It's not very remarkable and is designed to be simple and readable. The most interesting feature it has, IMHO, is how it maps "peripherals" and even CPU control registers into memory. Rather than providing special instructions or traps for OS system calls, Luz facilitates "bare-metal" programming (by which I mean, without an OS) by mapping "peripherals" into memory, allowing the programmer to access them by reading and writing special memory locations.

My inspiration here was soft-core embeddable CPUs like Nios II, which let you configure what peripherals to connect and how to map them. The CPU can be configured before it's loaded onto real HW, for example to attach as many SPI interfaces as needed. For Luz, to create a new peripheral and attach it to the simulator one implements the Peripheral interface:

class Peripheral(object):
    """ An abstract memory-mapped perhipheral interface.
        Memory-mapped peripherals are accessed through memory
        reads and writes.

        The address given to reads and writes is relative to the
        peripheral's memory map.
        Width is 1, 2, 4 for byte, halfword and word accesses.
    """
    def read_mem(self, addr, width):
        raise NotImplementedError()

    def write_mem(self, addr, width, data):
        raise NotImplementedError()

Luz implements some built-in features as peripherals as well; for example, the core registers (interrupt control, exception control, etc). The idea here is that embedded CPUs can have multiple custom "registers" to control various features, and creating dedicated names for them bloats instruction encoding (you need 5 bits to encode one of 32 registers, etc.); it's better to just map them to memory.

Another example is the debug queue - a peripheral useful for testing and debugging. It's a single word mapped to address 0xF0000 in the simulator. When the peripheral gets a write, it stores it in a special queue and optionally emits the value to stdout. The queue can later be examined. Here is a simple Luz assembly program that makes use of it:

# Counts from 0 to 9 [inclusive], pushing these numbers into the debug queue

    .segment code
    .global asm_main

    .define ADDR_DEBUG_QUEUE, 0xF0000

asm_main:
    li $k0, ADDR_DEBUG_QUEUE

    li $r9, 10                          # r9 is the loop limit
    li $r5, 0                           # r5 is the loop counter

loop:
    sw $r5, 0($k0)                      # store loop counter to debug queue
    addi $r5, $r5, 1                    # increment loop counter
    bltu $r5, $r9, loop                 # loop back if not reached limit

    halt

Using the interactive runner to run this program we get:

$ python run_test_interactive.py loop_simple_debugqueue
DebugQueue: 0x0
DebugQueue: 0x1
DebugQueue: 0x2
DebugQueue: 0x3
DebugQueue: 0x4
DebugQueue: 0x5
DebugQueue: 0x6
DebugQueue: 0x7
DebugQueue: 0x8
DebugQueue: 0x9
Finished successfully...
Debug queue contents:
['0x0', '0x1', '0x2', '0x3', '0x4', '0x5', '0x6', '0x7', '0x8', '0x9']

Assembler

There's a small snippet of Luz assembly shown above. It's your run-of-the-mill RISC assembly, with the familiar set of instructions, fairly simple addressing modes and almost every instruction requiring registers (note how we can't store into the debug queue directly, for example, without dereferencing a register that holds its address).

The Luz user manual contains a complete reference for the instructions, including their encodings. Every instruction is a 32-bit word, with the 6 high bits for the opcode (meaning up to 64 distinct instructions are supported).

The code snippet also shows off some special features of the full Luz toolchain, like the special label asm_main. I'll discuss these later on in the section about linking.

Assembly languages are usually fairly simple to parse, and Luz is no exception. When I started working on Luz, I decided to use the PLY library for the lexer and parser mainly because I wanted to play with it. These days I'd probably just hand-roll a parser.

Luz takes another cool idea from MIPS - register aliases. While the assembler doesn't enforce any specific ABI on the coder, some conventions are very important when writing large assembly programs, and especially when interfacing with routines written by other programmers. To facilitate this, Luz designates register aliases for callee-saved registers and temporary registers.

For example, the general-purpose register number 19 can be referred to in Luz assembly as $r19 but also as $s1 - the callee-saved register 1. When writing standalone Luz programs, one is free to ignore these conventions. To get a taste of how ABI-conformant Luz assembly would look, take a look at this example.

To be honest, ABI was on my mind because I was initially envisioning a full programming environment for Luz, including a C compiler. When you have a compiler, you must have some set of conventions for generated code like procedure parameter passing, saved registers and so on; in other words, the platform ABI.

Linker

In my view, one of the distinguishing features of Luz from other assembler projects out there is the linker. Luz features a full linker that supports creating single "binaries" from multiple assembly files, handling all the dirty work necessary to make that happen. Each assembly file is first "assembled" into a position-independent object file; these are glued together by the linker which applies the necessary relocations to resolve symbols across object files. The prime sieve example shows this in action - the program is divided into three .lasm files: two for subroutines and one for "main".

As we've seen above, the main subroutine in Luz is called asm_main. This is a special name for the linker (not unlike the _start symbol for modern Linux assemblers). The linker collects a set of object files produced by assembly, and makes sure to invoke asm_main from the special location 0x100000. This is where the simulator starts execution.

Luz also has the concept of object files. They are not unlike ELF images in nature: there's a segment table, an export table and a relocation table for each object, serving the expected roles. It is the job of the linker to make sense in this list of objects and correctly connect all call sites to final subroutine addresses.

Luz's standalone assembler can write an assembled image into a file in Intel HEX format, a popular format used in embedded systems to encode binary images or data in ASCII.

The linker was quite a bit of effort to develop. Since all real Luz programs are small I didn't really need to break them up into multiple assembly files; but I really wanted to learn how to write a real linker :) Moreover, as already mentioned my original plans for Luz included a C compiler, and that would make a linker very helpful, since I'd need to link some "system" code into the user's program. Even today, Luz has some "startup code" it links into every image:

# The special segments added by the linker.
# __startup: 3 words
# __heap: 1 word
#
LINKER_STARTUP_CODE = string.Template(r'''
        .segment __startup

    LI      $$sp, ${SP_POINTER}
    CALL    asm_main

        .segment __heap
        .global __heap
    __heap:
        .word 0
''')

This code sets up the stack pointer to the initial address allocated for the stack, and calls the user's asm_main.

Debugger and disassembler

Luz comes with a simple program runner that will execute a Luz program (consisting of multiple assembly files); it also has an interactive mode - a debugger. Here's a sample session with the simple loop example shown above:

$ python run_test_interactive.py -i loop_simple_debugqueue

LUZ simulator started at 0x00100000

[0x00100000] [lui $sp, 0x13] >> set alias 0
[0x00100000] [lui $r29, 0x13] >> s
[0x00100004] [ori $r29, $r29, 0xFFFC] >> s
[0x00100008] [call 0x40003 [0x10000C]] >> s
[0x0010000C] [lui $r26, 0xF] >> s
[0x00100010] [ori $r26, $r26, 0x0] >> s
[0x00100014] [lui $r9, 0x0] >> s
[0x00100018] [ori $r9, $r9, 0xA] >> s
[0x0010001C] [lui $r5, 0x0] >> s
[0x00100020] [ori $r5, $r5, 0x0] >> s
[0x00100024] [sw $r5, 0($r26)] >> s
[0x00100028] [addi $r5, $r5, 0x1] >> s
[0x0010002C] [bltu $r5, $r9, -2] >> s
[0x00100024] [sw $r5, 0($r26)] >> s
[0x00100028] [addi $r5, $r5, 0x1] >> s
[0x0010002C] [bltu $r5, $r9, -2] >> s
[0x00100024] [sw $r5, 0($r26)] >> s
[0x00100028] [addi $r5, $r5, 0x1] >> r
$r0   = 0x00000000   $r1   = 0x00000000   $r2   = 0x00000000   $r3   = 0x00000000
$r4   = 0x00000000   $r5   = 0x00000002   $r6   = 0x00000000   $r7   = 0x00000000
$r8   = 0x00000000   $r9   = 0x0000000A   $r10  = 0x00000000   $r11  = 0x00000000
$r12  = 0x00000000   $r13  = 0x00000000   $r14  = 0x00000000   $r15  = 0x00000000
$r16  = 0x00000000   $r17  = 0x00000000   $r18  = 0x00000000   $r19  = 0x00000000
$r20  = 0x00000000   $r21  = 0x00000000   $r22  = 0x00000000   $r23  = 0x00000000
$r24  = 0x00000000   $r25  = 0x00000000   $r26  = 0x000F0000   $r27  = 0x00000000
$r28  = 0x00000000   $r29  = 0x0013FFFC   $r30  = 0x00000000   $r31  = 0x0010000C

[0x00100028] [addi $r5, $r5, 0x1] >> s 100
[0x00100030] [halt] >> q

There are many interesting things here demonstrating how Luz works:

Note the start up at 0x1000000 - this is where Luz places the start-up segment - three instructions that set up the stack pointer and then call the user's code (asm_main). The user's asm_main starts running at the fourth instruction executed by the simulator.
li is a pseudo-instruction, broken into two real instructions: lui for the upper half of the register, followed by ori for the lower half of the register. The reason for this is li having a 32-bit immediate, which can't fit in a Luz instruction. Therefore, it's broken into two parts which only need 16-bit immediates. This trick is common in RISC ISAs.
Jump labels are resolved to be relative by the assembler: the jump to loop is replaced by -2.
Disassembly! The debugger shows the instruction decoded from every word where execution stops. Note how this exposes pseudo-instructions.

The in-progress RTL implementation

Luz was a hobby project, but an ambitious one :-) Even before I wrote the first line of the assembler or simulator, I started working on an actual CPU implementation in synthesizable VHDL, meaning to get a complete RTL image to run on FPGAs. Unfortunately, I didn't finish this part of the project and what you find in Luz's experimental/luz_uc directory is only 75% complete. The ALU is there, the registers, the hookups to peripherals, even parts of the control path - dealing with instruction fetching, decoding, etc. My original plan was to implement a pipelined CPU (a RISC ISA makes this relatively simple), which perhaps was a bit too much. I should have started simpler.

Conclusion

Luz was an extremely educational project for me. When I started working on it, I mostly had embedded programming experience and was just starting to get interested in systems programming. Luz flung me into the world of assemblers, linkers, binary images, calling conventions, and so on. Besides, Python was a new language for me at the time - Luz started just months after I first got into Python.

Its ~8000 lines of Python code are thus likely not my best Python code, but they should be readable and well commented. I did modernize it a bit over the years, for example to make it run on both Python 2 and 3.

I still hope to get back to the RTL implementation project one day. It's really very close to being able to run realistic assembly programs on real hardware (FPGAs). My dream back then was to fully close the loop by adding a Luz code generation backend to pycparser. Maybe I'll still fulfill it one day :-)

Library order in static linking

2013-07-09T05:56:27-07:00

I'll start with a slightly sneaky but educational example. Suppose we have this code:

volatile char src[] = {1, 2, 3, 4, 5};
volatile char dst[50] = {0};

void* memcpy(void* dst, void* src, int len);

int main(int argc, const char* argv[])
{
    memcpy(dst, src, sizeof(src));
    return dst[4];
}

It runs just fine and the return value is 5. Now, suppose this is part of a larger project that consists of many object files and libraries, and somewhere within the project there's a library that contains this code:

void memcpy(char* aa, char* bb, char* cc) {
    int i;
    for (i = 0; i < 100; ++i) {
        cc[i] = aa[i] + bb[i];
    }
}

If the previous snippet gets linked with this library, what happens? Would you expect it to still return 5? Return something else? Crash? The answer is: it depends - the result can be either correct or a segmentation fault. It depends on the order in which the objects and libraries in the project were fed to the linker.

If you fully understand why this depends on linking order, as well as how to avoid the problem (and more serious problems, like circular dependencies) then congratulate yourself and move on - this article is probably not for you. Otherwise, read on.

The basics

Let's start by defining the scope of this article: first, my examples are demonstrating the use of the gcc and binutils toolchain on Linux. Compatible toolchains (like clang instead of gcc) apply too. Second, the discussion here resolves around static linking that's done at compile/link time.

To understand why linking order matters, it's first instructional to understand how the linker works with respect to linking libraries and objects together. Just as a quick reminder - an object file both provides (exports) external symbols to other objects and libraries, and expects (imports) symbols from other objects and libraries. For example, in this C code:

int imported(int);

static int internal(int x) {
    return x * 2;
}

int exported(int x) {
    return imported(x) * internal(x);
}

The names of the functions speak for themselves. Let's compile it and look at the symbol table:

$ gcc -c x.c
$ nm x.o
000000000000000e T exported
                 U imported
0000000000000000 t internal

This means: exported is an external symbol - defined in the object file and visible from the outside. imported is an undefined symbol; in other words, the linker is expected to find it elsewhere. When we talk about linking later, the term undefined can become confusing - so it helps to remember that this is where it comes from originally. internal is defined within the object but invisible from the outside.

Now, a library is simply a collection of object files. Just a bunch of object files glued together. Creating a library is a very trivial operation that doesn't do anything special besides placing many object files into the same file. This in itself is important, because a horde of object files is not convenient to deal with. For example, on my system libc.a (the static version of the C library) consists of almost 1500 object files. It's way nicer to just carry libc.a around.

The linking process

This section defines the linking process in a somewhat dry, algorithmic manner. This process is the key to understanding why linking order matters.

Consider a linker invocation:

$ gcc main.o -L/some/lib/dir -lfoo -lbar -lbaz

The linker is almost always invoked through the compiler driver gcc when compiling C or C++ code. This is because the driver knows how to provide the correct command-line arguments to the linker itself (ld) with all the support libraries, etc. We'll see more of this later.

Anyhow, as you can see the object files and libraries are provided in a certain order on the command-line, from left to right. This is the linking order. Here's what the linker does:

The linker maintains a symbol table. This symbol table does a bunch of things, but among them is keeping two lists:
- A list of symbols exported by all the objects and libraries encountered so far.
- A list of undefined symbols that the encountered objects and libraries requested to import and were not found yet.
When the linker encounters a new object file, it looks at:
- The symbols it exports: these are added to the list of exported symbols mentioned above. If any symbol is in the undefined list, it's removed from there because it has now been found. If any symbol has already been in the exported list, we get a "multiple definition" error: two different objects export the same symbol and the linker is confused.
- The symbols it imports: these are added to the list of undefined symbols, unless they can be found in the list of exported symbols.
When the linker encounters a new library, things are a bit more interesting. The linker goes over all the objects in the library. For each one, it first looks at the symbols it exports.
- If any of the symbols it exports are on the undefined list, the object is added to the link and the next step is executed. Otherwise, the next step is skipped.
- If the object has been added to the link, it's treated as described above - its undefined and exported symbols get added to the symbol table.
- Finally, if any of the objects in the library has been included in the link, the library is rescanned again - it's possible that symbols imported by the included object can be found in other objects within the same library.

When the linker finishes, it looks at the symbol table. If any symbols remain in the undefined list, the linker will throw an "undefined reference" error. For example, when you create an executable and forget to include the file with the main function, you'll get something like:

/usr/lib/x86_64-linux-gnu/crt1.o: In function '_start':
(.text+0x20): undefined reference to 'main'
collect2: ld returned 1 exit status

Note that after the linker has looked at a library, it won't look at it again. Even if it exports symbols that may be needed by some later library. The only time where a linker goes back to rescan objects it has already seen happens within a single library - as mentioned above, once an object from some library is taken into the link, all other objects in the same library will be rescanned. Flags passed to the linker can tweak this process - again, we'll see some examples later.

Also note that when a library is examined, an object file within it can be left out of the link if it does not provide symbols that the symbol table needs. This is a very important feature of static linking. The C library I mentioned before makes a heavy use of this feature, by mostly splitting itself to an-object-per-function. So, for example if the only C standard library function your code uses is strlen, only strlen.o will be taken into the link from libc.a - and your executable will be very small.

Simple examples

The previous section can be hard to digest, so here are some simple examples that show the process in action.

Let's start with the most basic case, of linking two objects together:

$ cat simplefunc.c
int func(int i) {
    return i + 21;
}

$ cat simplemain.c
int func(int);

int main(int argc, const char* argv[])
{
    return func(argc);
}

$ gcc -c simplefunc.c
$ gcc -c simplemain.c
$ gcc simplefunc.o simplemain.o
$ ./a.out ; echo $?
22

Everything links and works as expected. Note that since these are object files, the linking order does not matter. Object files are always taken into the link. We can pass them to the linker in reversed order and it still works:

$ gcc simplemain.o simplefunc.o
$ ./a.out ; echo $?
22

Now let's do something different. Let's put simplefunc.c into a library:

$ ar r libsimplefunc.a simplefunc.o
$ ranlib libsimplefunc.a
$ gcc  simplemain.o -L. -lsimplefunc
$ ./a.out ; echo $?
22

Works like a charm. But note what happens if the linking order is reversed now:

$ gcc  -L. -lsimplefunc  simplemain.o
simplemain.o: In function 'main':
simplemain.c:(.text+0x15): undefined reference to 'func'
collect2: ld returned 1 exit status

Understanding the linking algorithm outlined above makes this case simple to explain. When the linker encounters libsimplefunc.a, it still hasn't seen simplemain.o, which means that func is not yet on the undefined list. When the linker looks into the library it sees simplefunc.o that exports func. But since it doesn't need func, this object file is not included in the link. When the linker does reach simplemain.o and sees that func is, indeed required, it's added to the undefined list (because it's not on the exported list). The linker then reaches the end of the link and func is still undefined.

Note how this doesn't happen in the previous linking order - since simplemain.o comes first, func is on the undefined list before the linker sees the library, so the object file exporting it does get included.

This brings us to the most important corollary of the linking process outlined above:

If object or library AA needs a symbol from library BB, then AA should come before library BB in the command-line invocation of the linker.

Circular dependency

The corollary above is an important summary of the linking process - it's certainly much more practical to keep in mind because it's so short. But it makes one wonder - what happens if AA needs a symbol from BB, but BB also needs a symbol from AA? While officially this isn't a good programming practice, in reality it happens quite a lot. But AA can't come both before and after BB on the command-line, right? That's just silly. Wait, is it, really?

Let's see an example and start simple. Imagine that instead of simplefunc.c, the func symbol is provided thus:

$ cat func_dep.c
int bar(int);

int func(int i) {
    return bar(i + 1);
}
$ cat bar_dep.c
int func(int);

int bar(int i) {
    if (i > 3)
        return i;
    else
        return func(i);
}

These two files depend on each other and get placed into different libraries. If we link them in one order, we fail:

$ gcc  simplemain.o -L.  -lbar_dep -lfunc_dep
./libfunc_dep.a(func_dep.o): In function 'func':
func_dep.c:(.text+0x14): undefined reference to 'bar'
collect2: ld returned 1 exit status

However, the other order does work:

$ gcc  simplemain.o -L. -lfunc_dep -lbar_dep
$ ./a.out ; echo $?
4

Quiz: can you figure out why? Hint: just go over the linking process algorithm with this command-line. What undefined symbols does the symbol table contain when the linker first sees -lfunc_dep?

But this is a very simple case. Let's look at a trickier one. We'll add a dependency to bar on another function from libfunc_dep.a, but one that lives in a different object:

$ cat bar_dep.c
int func(int);
int frodo(int);

int bar(int i) {
    if (i > 3)
        return frodo(i);
    else
        return func(i);
}

$ cat frodo_dep.c
int frodo(int i) {
    return 6 * i;
}

We'll recompile all these files into separate objects, and the libfunc_dep.a library will now be:

$ ar r libfunc_dep.a func_dep.o frodo_dep.o
$ ranlib libfunc_dep.a

Here's a drawing of the libraries, with arrows showing the dependencies:

Now linking fails no matter what order we list the libraries in:

$ gcc  -L. simplemain.o -lfunc_dep -lbar_dep
./libbar_dep.a(bar_dep.o): In function 'bar':
bar_dep.c:(.text+0x17): undefined reference to 'frodo'
collect2: ld returned 1 exit status
$ gcc  -L. simplemain.o -lbar_dep -lfunc_dep
./libfunc_dep.a(func_dep.o): In function 'func':
func_dep.c:(.text+0x14): undefined reference to 'bar'
collect2: ld returned 1 exit status

To solve this, consider that it's perfectly valid to list a library more than once on the link line; so in fact, we can provide libfunc_dep.a both before and after libbar_dep.a:

$ gcc  -L. simplemain.o -lfunc_dep -lbar_dep -lfunc_dep
$ ./a.out ; echo $?
24

Another quiz: will the same trick work providing -lbar_dep twice? Why not?

Using linker flags to control the process

As I've mentioned above, the linker has a number of interesting flags that can be used to control the process in a fine-grained manner. For example, circular dependency problems can be easily resolved with --start-group and --end-group. Here's an instructive portion from man ld:

--start-group archives --end-group

The specified archives are searched repeatedly until no new undefined references are created. Normally, an archive is searched only once in the order that it is specified on the command line. If a symbol in that archive is needed to resolve an undefined symbol referred to by an object in an archive that appears later on the command line, the linker would not be able to resolve that reference. By grouping the archives, they all be searched repeatedly until all possible references are resolved.

Using this option has a significant performance cost. It is best to use it only when there are unavoidable circular references between two or more archives.

Here's how this helps in our case:

$ gcc simplemain.o -L. -Wl,--start-group -lbar_dep -lfunc_dep -Wl,--end-group
$ ./a.out ; echo $?
24

It's interesting to note the "significant performance cost" warning in the excerpt above. This explains why the linking process is the way it is. Presumably, linkers could just re-scan the whole library list until no new symbols got resolved. This would eliminate most circular-dependency and linking order problems in the world, but it would also be slow. Linking is already a critical part of the compilation time of large systems, since it looks at the whole program and requires quite a bit of memory. It's better to make it as fast as possible for well-behaved programs (that got their linking order right), and provide special options like groups for the difficult circular dependency cases.

There's at least one another linker flag that can help us resolve the circular dependency here. We can use the --undefined flag to tell the linker - "buddy, here's a symbol I want you to add to the undefined list". In our case this makes the link error go away even though the libraries are specified only once:

$ gcc simplemain.o -L. -Wl,--undefined=bar -lbar_dep -lfunc_dep
$ ./a.out ; echo $?
24

Figuring out why this works is left as an exercise to the reader.

Back to the original example

Let's go back to the example this article started with. main assumes it gets the correct memcpy from the C library, but the memcpy it gets linked with does something else. Assuming the memcpy here was packed into the libstray_memcpy.a library:

$ gcc  -L. main_using_memcpy.o -lstray_memcpy
$ ./a.out
Segmentation fault (core dumped)

This is the expected behavior. Since -lstray_memcpy was provided after main_using_memcpy.o on the command-line, it gets linked in. But what happens if the order is reversed:

$ gcc  -L. -lstray_memcpy main_using_memcpy.o
$ ./a.out ; echo $?
5

The program links and works correctly. The reason for this is simple: even without us explicitly asking for it, gcc asks the linker to link the C library as well. The full linker invocation command of gcc is pretty complex, and can be examined by passing the -### flag to gcc. But in our case this amounts to:

$ gcc  -L. -lstray_memcpy main_using_memcpy.o -lc

When the linker sees -lstray_memcpy, the symbol table does not yet have an undefined entry for memcpy, so the object file with the wrong function does not get linked. The linker adds this undefined entry only after it sees main_using_memcpy.o. Then, when it reaches -lc, the object file holding memcpy from the C library does get linked in because by now memcpy is on the undefined list.

Conclusion

The algorithm used by the linker to resolve symbols between objects and libraries is pretty simple. As long as you keep it in mind, linker errors and related problems should be easy to understand. If you still run into problematic situations you're not sure how to resolve, this article mentioned two tools that can be very useful in debugging such problems: one is nm, which shows the symbol table of an object or a whole library. The other is the -### flag that gcc accepts and as a result shows the full commands it passes to the underlying tools.

How statically linked programs run on Linux

2012-08-13T04:44:58-07:00

In this article I want to explore what happens when a statically linked program gets executed on Linux. By statically linked I mean a program that does not require any shared objects to run, even the ubiquitous libc. In reality, most programs one encounters on Linux aren't statically linked, and do require one or more shared objects to run. However, the running sequence of such programs is more involved, which is why I want to present statically linked programs first. It will serve as a good basis for understanding, allowing me to explore most of the mechanisms involved with less details getting in the way. In a future article I will cover the dynamic linking process in detail.

The Linux kernel

Program execution begins in the Linux kernel. To run a program, a process will call a function from the exec family. The functions in this family are all very similar, differing only in small details regarding the manner of passing arguments and environment variables to the invoked program. What they all end up doing is issuing the sys_execve system call to the Linux kernel.

sys_execve does a lot of work to prepare the new program for execution. Explaining it all is far beyond the scope of this article - a good book on kernel internals can be helpful to understand the details [1]. I'll just focus on the stuff useful for our current discussion.

As part of its job, the kernel must read the program's executable file from disk into memory and prepare it for execution. The kernel knows how to handle a lot of binary file formats, and tries to open the file with different handlers until it succeeds (this happens in the function search_binary_handler in fs/exec.c). We're only interested in ELF here, however; for this format the action happens in function load_elf_binary (in fs/binfmt_elf.c).

The kernel reads the ELF header of the program, and looks for a PT_INTERP segment to see if an interpreter was specified. Here the statically linked vs. dynamically linked distinction kicks in. For statically linked programs, there is no PT_INTERP segment. This is the scenario this article covers.

The kernel then goes on mapping the program's segments into memory, according to the information contained in the ELF program headers. Finally, it passes the execution, by directly modifying the IP register, to the entry address read from the ELF header of the program (e_entry). Arguments are passed to the program on the stack (the code responsible for this is in create_elf_tables). Here's the stack layout when the program is called, for x64:

At the top of the stack is argc, the amount of command-line arguments. It is followed by all the arguments themselves (each a char*), terminated by a zero pointer. Then, the environment variables are listed (also a char* each), terminated by a zero pointer. The observant reader will notice that this argument layout is not what one usually expects in main. This is because main is not really the entry point of the program, as the rest of the article shows.

Program entry point

So, the Linux kernel reads the program's entry address from the ELF header. Let's now explore how this address gets there.

Unless you're doing something very funky, the final program binary image is probably being created by the system linker - ld. By default, ld looks for a special symbol called _start in one of the object files linked into the program, and sets the entry point to the address of that symbol. This will be simplest to demonstrate with an example written in assembly (the following is NASM syntax):

section    .text
    ; The _start symbol must be declared for the linker (ld)
    global _start

_start:
    ; Execute sys_exit call. Argument: status -> ebx
    mov     eax, 1
    mov     ebx, 42
    int     0x80

This is a very basic program that simply returns 42. Note that it has the _start symbol defined. Let's build it, examine the ELF header and its disassembly:

$ nasm -f elf64 nasm_rc.asm -o nasm_rc.o
$ ld -o nasm_rc64 nasm_rc.o
$ readelf -h nasm_rc64
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 00 00 00 00 00 00 00 00 00
  Class:                             ELF64
  ...
  Entry point address:               0x400080
  ...
$ objdump -d nasm_rc64

nasm_rc64:     file format elf64-x86-64


Disassembly of section .text:

0000000000400080 <_start>:
  400080:     b8 01 00 00 00          mov    $0x1,%eax
  400085:     bb 2a 00 00 00          mov    $0x2a,%ebx
  40008a:     cd 80                   int    $0x80

As you can see, the entry point address in the ELF header was set to 0x400080, which also happens to be the address of _start.

ld looks for _start by default, but this behavior can be modified by either the --entry command-line flag, or by providing an ENTRY command in a custom linker script.

The entry point in C code

We're usually not writing our code in assembly, however. For C/C++ the situation is different, because the entry point familiar to users is the main function and not the _start symbol. Now it's time to explain how these two are related.

Let's start with this simple C program which is functionally equivalent to the assembly shown above:

int main() {
    return 42;
}

I will compile this code into an object file and then attempt to link it with ld, like I did with the assembly:

$ gcc -c c_rc.c
$ ld -o c_rc c_rc.o
ld: warning: cannot find entry symbol _start; defaulting to 00000000004000b0

Whoops, ld can't find the entry point. It tries to guess using a default, but it won't work - the program will segfault when run. ld obviously needs some additional object files where it will find the entry point. But which object files are these? Luckily, we can use gcc to find out. gcc can act as a full compilation driver, invoking ld as needed. Let's now use gcc to link our object file into a program. Note that the -static flag is passed to force static linking of the C library and the gcc runtime library:

$ gcc -o c_rc -static c_rc.o
$ c_rc; echo $?
42

It works. So how does gcc manage to do the linking correctly? We can pass the -Wl,-verbose flag to gcc which will spill the list of objects and libraries it passed to the linker. Doing this, we'll see additional object files like crt1.o and the whole libc.a static library (which has objects with telling names like libc-start.o). C code does not live in a vacuum. To run, it requires some support libraries such as the gcc runtime and libc.

Since it obviously linked and ran correctly, the program we built with gcc should have a _start symbol at the right place. Let's check [2]:

$ readelf -h c_rc
ELF Header:
  Magic:   7f 45 4c 46 02 01 01 03 00 00 00 00 00 00 00 00
  Class:                             ELF64
  ...
  Entry point address:               0x4003c0
  ...

$ objdump -d c_rc | grep -A15 "<_start"
00000000004003c0 <_start>:
  4003c0:     31 ed                   xor    %ebp,%ebp
  4003c2:     49 89 d1                mov    %rdx,%r9
  4003c5:     5e                      pop    %rsi
  4003c6:     48 89 e2                mov    %rsp,%rdx
  4003c9:     48 83 e4 f0             and    $0xfffffffffffffff0,%rsp
  4003cd:     50                      push   %rax
  4003ce:     54                      push   %rsp
  4003cf:     49 c7 c0 20 0f 40 00    mov    $0x400f20,%r8
  4003d6:     48 c7 c1 90 0e 40 00    mov    $0x400e90,%rcx
  4003dd:     48 c7 c7 d4 04 40 00    mov    $0x4004d4,%rdi
  4003e4:     e8 f7 00 00 00          callq  4004e0 <__libc_start_main>
  4003e9:     f4                      hlt
  4003ea:     90                      nop
  4003eb:     90                      nop

Indeed, 0x4003c0 is the address of _start and it's the program entry point. However, what is all that code at _start? Where does it come from, and what does it mean?

Decoding the start sequence of C code

The startup code shown above comes from glibc - the GNU C library, where for x64 ELF it lives in the file sysdeps/x86_64/start.S [3]. Its goal is to prepare the arguments for a function named __libc_start_main and call it. This function is also part of glibc and lives in csu/libc-start.c. Here is its signature, formatted for clarity, with added comments to explain what each argument means:

int __libc_start_main(
         /* Pointer to the program's main function */
         (int (*main) (int, char**, char**),
         /* argc and argv */
         int argc, char **argv,
         /* Pointers to initialization and finalization functions */
         __typeof (main) init, void (*fini) (void),
         /* Finalization function for the dynamic linker */
         void (*rtld_fini) (void),
         /* End of stack */
         void* stack_end)

Anyway, with this signature and the AMD64 ABI in hand, we can map the arguments passed to __libc_start_main from _start:

main:      rdi <-- $0x4004d4
argc:      rsi <-- [RSP]
argv:      rdx <-- [RSP + 0x8]
init:      rcx <-- $0x400e90
fini:      r8  <-- $0x400f20
rdld_fini: r9  <-- rdx on entry
stack_end: on stack <-- RSP

You'll also notice that the stack is aligned to 16 bytes and some garbage is pushed on top of it (rax) before pushing rsp itself. This is to conform to the AMD64 ABI. Also note the hlt instruction at address 0x4003e9. It's a safeguard in case __libc_start_main did not exit (as we'll see, it should). hlt can't be executed in user mode, so this will raise an exception and crash the process.

Examining the disassembly, it's easy to verify that 0x4004d4 is indeed main, 0x400e90 is __libc_csu_init and 0x400f20 is __libc_csu_fini. There's another argument the kernel passes _start - a finish function for shared libraries to use (in rdx). We'll ignore it in this article.

The C library start function

Now that we understood how it's being called, what does __libc_start_main actually do? Ignoring some details that are probably too specialized to be interesting in the scope of this article, here's a list of things that it does for a statically linked program:

Figure out where the environment variables are on the stack.
Prepare the auxiliary vector, if required.
Initialize thread-specific functionality (pthreads, TLS, etc.)
Perform some security-related bookkeeping (this is not really a separate step, but is trickled all through the function).
Initialize libc itself.
Call the program initialization function through the passed pointer (init).
Register the program finalization function (fini) for execution on exit.
Call main(argc, argv, envp)
Call exit with the result of main as the exit code.

Digression: init and fini

Some programming environments (most notably C++, to construct and destruct static and global objects) require running custom code before and after main. This is implemented by means of cooperation between the compiler/linker and the C library. For example, the __libc_csu_init (which, as you can see above, is called before the user's main) calls into special code that's inserted by the linker. The same goes for __libc_csu_fini and finalization.

You can also ask the compiler to register your function to be executed as one of the constructors or destructors. For example [4]:

#include <stdio.h>

int main() {
    return 43;
}

__attribute__((constructor))
void myconstructor() {
    printf("myconstructor\n");
}

myconstructor will run before main. The linker places its address in a special array of constructors located in the .ctors section. __libc_csu_init goes over this array and calls all functions listed in it.

Conclusion

This article demonstrates how a statically linked program is set up to actually run on Linux. In my opinion, this is a very interesting topic to study because it demonstrates how several large components of the Linux eco-system cooperate to enable the program execution process. In this case, the Linux kernel, the compiler and linker, and the C library are involved. In a future article I will present the more complex case of a dynamically linked program, where another agent joins the game - the dynamic linker. Stay tuned.

[1]	Or just read the source, if you're brave.

[2]	Note that due to our static linking of the C runtime into `c_rc`, it's quite large (800 KB on my 64-bit Ubuntu system). Therefore we can't just easily look at the disassembly and have to use some `grep`-fu.

[3]	This is true for glibc 2.16.

[4]	Note that the constructor executes `printf`. Is this safe? If you look at the initialization sequence of `__libc_start_main`, you'll see that the C library is initialized before the user's constructors are called, so yes, it's safe.

Position Independent Code (PIC) in shared libraries on x64

2011-11-11T15:10:11-08:00

The previous article explained how position independent code (PIC) works, with code compiled for the x86 architecture as an example. I promised to cover PIC on x64 [1] in a separate article, so here we are. This article will go into much less detail, since it assumes an understanding of how PIC works in theory. In general, the idea is similar for both platforms, but some details differ because of unique features of each architecture.

RIP-relative addressing

On x86, while function references (with the call instruction) use relative offsets from the instruction pointer, data references (with the mov instruction) only support absolute addresses. As we've seen in the previous article, this makes PIC code somewhat less efficient, since PIC by its nature requires making all offsets IP-relative; absolute addresses and position independence don't go well together.

x64 fixes that, with a new "RIP-relative addressing mode", which is the default for all 64-bit mov instructions that reference memory (it's used for other instructions as well, such as lea). A quote from the "Intel Architecture Manual vol 2a":

A new addressing form, RIP-relative (relative instruction-pointer) addressing, is implemented in 64-bit mode. An effective address is formed by adding displacement to the 64-bit RIP of the next instruction.

The displacement used in RIP-relative mode is 32 bits in size. Since it should be useful for both positive and negative offsets, roughly +/- 2GB is the maximal offset from RIP supported by this addressing mode.

x64 PIC with data references - an example

For easier comparison, I will use the same C source as in the data reference example of the previous article:

int myglob = 42;

int ml_func(int a, int b)
{
    return myglob + a + b;
}

Let's look at the disassembly of ml_func:

00000000000005ec <ml_func>:
 5ec:   55                      push   rbp
 5ed:   48 89 e5                mov    rbp,rsp
 5f0:   89 7d fc                mov    DWORD PTR [rbp-0x4],edi
 5f3:   89 75 f8                mov    DWORD PTR [rbp-0x8],esi
 5f6:   48 8b 05 db 09 20 00    mov    rax,QWORD PTR [rip+0x2009db]
 5fd:   8b 00                   mov    eax,DWORD PTR [rax]
 5ff:   03 45 fc                add    eax,DWORD PTR [rbp-0x4]
 602:   03 45 f8                add    eax,DWORD PTR [rbp-0x8]
 605:   c9                      leave
 606:   c3                      ret

The most interesting instruction here is at 0x5f6: it places the address of myglobal into rax, by referencing an entry in the GOT. As we can see, it uses RIP relative addressing. Since it's relative to the address of the next instruction, what we actually get is 0x5fd + 0x2009db = 0x200fd8. So the GOT entry holding the address of myglob is at 0x200fd8. Let's check if it makes sense:

$ readelf -S libmlpic_dataonly.so
There are 35 section headers, starting at offset 0x13a8:

Section Headers:
  [Nr] Name              Type             Address           Offset
       Size              EntSize          Flags  Link  Info  Align

[...]
  [20] .got              PROGBITS         0000000000200fc8  00000fc8
       0000000000000020  0000000000000008  WA       0     0     8
[...]

GOT starts at 0x200fc8, so myglob is in its third entry. We can also see the relocation inserted for the GOT reference to myglob:

$ readelf -r libmlpic_dataonly.so

Relocation section '.rela.dyn' at offset 0x450 contains 5 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
[...]
000000200fd8  000500000006 R_X86_64_GLOB_DAT 0000000000201010 myglob + 0
[...]

Indeed, a relocation entry for 0x200fd8 telling the dynamic linker to place the address of myglob into it once the final address of this symbol is known.

So it should be quite clear how the address of myglob is obtained in the code. The next instruction in the disassembly (at 0x5fd) then dereferences the address to get the value of myglob into eax [2].

x64 PIC with function calls - an example

Now let's see how function calls work with PIC code on x64. Once again, we'll use the same example from the previous article:

int myglob = 42;

int ml_util_func(int a)
{
    return a + 1;
}

int ml_func(int a, int b)
{
    int c = b + ml_util_func(a);
    myglob += c;
    return b + myglob;
}

Disassembling ml_func, we get:

000000000000064b <ml_func>:
 64b:   55                      push   rbp
 64c:   48 89 e5                mov    rbp,rsp
 64f:   48 83 ec 20             sub    rsp,0x20
 653:   89 7d ec                mov    DWORD PTR [rbp-0x14],edi
 656:   89 75 e8                mov    DWORD PTR [rbp-0x18],esi
 659:   8b 45 ec                mov    eax,DWORD PTR [rbp-0x14]
 65c:   89 c7                   mov    edi,eax
 65e:   e8 fd fe ff ff          call   560 <ml_util_func@plt>
 [... snip more code ...]

The call is, as before, to ml_util_func@plt. Let's see what's there:

0000000000000560 <ml_util_func@plt>:
 560:   ff 25 a2 0a 20 00       jmp    QWORD PTR [rip+0x200aa2]
 566:   68 01 00 00 00          push   0x1
 56b:   e9 d0 ff ff ff          jmp    540 <_init+0x18>

So, the GOT entry holding the actual address of ml_util_func is at 0x200aa2 + 0x566 = 0x201008.

And there's a relocation for it, as expected:

$ readelf -r libmlpic.so

Relocation section '.rela.dyn' at offset 0x480 contains 5 entries:
[...]

Relocation section '.rela.plt' at offset 0x4f8 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
[...]
000000201008  000600000007 R_X86_64_JUMP_SLO 000000000000063c ml_util_func + 0

Performance implications

In both examples, it can be seen that PIC on x64 requires less instructions than on x86. On x86, the GOT address is loaded into some base register (ebx by convention) in two steps - first the address of the instruction is obtained with a special function call, and then the offset to GOT is added. Both steps aren't required on x64, since the relative offset to GOT is known to the linker and can simply be encoded in the instruction itself with RIP relative addressing.

When calling a function, there's also no need to prepare the GOT address in ebx for the trampoline, as the x86 code does, since the trampoline just accesses its GOT entry directly through RIP-relative addressing.

So PIC on x64 still requires extra instructions when compared to non-PIC code, but the additional cost is smaller. The indirect cost of tying down a register to use as the GOT pointer (which is painful on x86) is also gone, since no such register is needed with RIP-relative addressing [3]. All in all, x64 PIC results in a much smaller performance hit than on x86, making it much more attractive. So attractive, in fact, that it's the default method for writing shared libraries for this architecture.

Extra credit: Non-PIC code on x64

Not only does gcc encourage you to use PIC for shared libraries on x64, it requires it by default. For instance, if we compile the first example without -fpic [4] and then try to link it into a shared library (with -shared), we'll get an error from the linker, something like this:

/usr/bin/ld: ml_nopic_dataonly.o: relocation R_X86_64_PC32 against symbol `myglob' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Bad value
collect2: ld returned 1 exit status

What's going on? Let's look at the disassembly of ml_nopic_dataonly.o [5]:

0000000000000000 <ml_func>:
   0:   55                      push   rbp
   1:   48 89 e5                mov    rbp,rsp
   4:   89 7d fc                mov    DWORD PTR [rbp-0x4],edi
   7:   89 75 f8                mov    DWORD PTR [rbp-0x8],esi
   a:   8b 05 00 00 00 00       mov    eax,DWORD PTR [rip+0x0]
  10:   03 45 fc                add    eax,DWORD PTR [rbp-0x4]
  13:   03 45 f8                add    eax,DWORD PTR [rbp-0x8]
  16:   c9                      leave
  17:   c3                      ret

Note how myglob is accessed here, in instruction at address 0xa. It expects the linker to patch in a relocation to the actual location of myglob into the operand of the instruction (so no GOT redirection is required):

$ readelf -r ml_nopic_dataonly.o

Relocation section '.rela.text' at offset 0xb38 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000000c  000f00000002 R_X86_64_PC32     0000000000000000 myglob - 4
[...]

Here is the R_X86_64_PC32 relocation the linker was complaining about. It just can't link an object with such relocation into a shared library. Why? Because the displacement of the mov (the part that's added to rip) must fit in 32 bits, and when a code gets into a shared library, we just can't know in advance that 32 bits will be enough. After all, this is a full 64-bit architecture, with a vast address space. The symbol may eventually be found in some shared library that's farther away from the reference than 32 bits will allow to reference. This makes R_X86_64_PC32 an invalid relocation for shared libraries on x64.

But can we still somehow create non-PIC code on x64? Yes! We should be instructing the compiler to use the "large code model", by adding the -mcmodel=large flag. The topic of code models is interesting, but explaining it would just take us too far from the real goal of this article [6]. So I'll just say briefly that a code model is a kind of agreement between the programmer and the compiler, where the programmer makes a certain promise to the compiler about the size of offsets the program will be using. In exchange, the compiler can generate better code.

It turns out that to make the compiler generate non-PIC code on x64 that actually pleases the linker, only the large code model is suitable, because it's the least restrictive. Remember how I explained why the simple relocation isn't good enough on x64, for fear of an offset which will get farther than 32 bits away during linking? Well, the large code model basically gives up on all offset assumptions and uses the largest 64-bit offsets for all its data references. This makes load-time relocations always safe, and enables non-PIC code generation on x64. Let's see the disassembly of the first example compiled without -fpic and with -mcmodel=large:

0000000000000000 <ml_func>:
   0:   55                      push   rbp
   1:   48 89 e5                mov    rbp,rsp
   4:   89 7d fc                mov    DWORD PTR [rbp-0x4],edi
   7:   89 75 f8                mov    DWORD PTR [rbp-0x8],esi
   a:   48 b8 00 00 00 00 00    mov    rax,0x0
  11:   00 00 00
  14:   8b 00                   mov    eax,DWORD PTR [rax]
  16:   03 45 fc                add    eax,DWORD PTR [rbp-0x4]
  19:   03 45 f8                add    eax,DWORD PTR [rbp-0x8]
  1c:   c9                      leave
  1d:   c3                      ret

The instruction at address 0xa places the address of myglob into rax. Note that its argument is currently 0, which tells us to expect a relocation. Note also that it has a full 64-bit address argument. Moreover, the argument is absolute and not RIP-relative [7]. Note also that two instructions are actually required here to get the value of myglob into eax. This is one reason why the large code model is less efficient than the alternatives.

Now let's see the relocations:

$ readelf -r ml_nopic_dataonly.o

Relocation section '.rela.text' at offset 0xb40 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
00000000000c  000f00000001 R_X86_64_64       0000000000000000 myglob + 0
[...]

Note the relocation type has changed to R_X86_64_64, which is an absolute relocation that can have a 64-bit value. It's acceptable by the linker, which will now gladly agree to link this object file into a shared library.

Some judgmental thinking may bring you to ponder why the compiler generated code that isn't suitable for load-time relocation by default. The answer to this is simple. Don't forget that code also tends to get directly linked into executables, which don't require load-time relocations at all. Therefore, by default the compiler assumes the small code model to generate the most efficient code. If you know your code is going to get into a shared library, and you don't want PIC, then just tell it to use the large code model explicitly. I think gcc's behavior makes sense here.

Another thing to think about is why there are no problems with PIC code using the small code model. The reason is that the GOT is always located in the same shared library as the code that references it, and unless a single shared library is big enough for a 32-bit address space, there should be no problems addressing the PIC with 32-bit RIP-relative offsets. Such huge shared libraries are unlikely, but in case you're working on one, the AMD64 ABI has a "large PIC code model" for this purpose.

Conclusion

This article complements its predecessor by showing how PIC works on the x64 architecture. This architecture has a new addressing mode that helps PIC code be faster, and thus makes it more desirable for shared libraries than on x86, where the cost is higher. Since x64 is currently the most popular architecture used in servers, desktops and laptops, this is important to know. Therefore, I tried to focus on additional aspects of compiling code into shared libraries, such as non-PIC code. If you have any questions and/or suggestions on future directions to explore, please let me know in the comments or by email.

[1]	As always, I'm using x64 as a convenient short name for the architecture known as x86-64, AMD64 or Intel 64.

[2]	Into `eax` and not `rax` because the type of `myglob` is `int`, which is still 32-bit on x64.

[3]	By the way, it would be much less "painful" to tie down a register on x64, since it has twice as many GPRs as x86.

[4]	It also happens if we explicitly specify we don't want PIC by passing `-fno-pic` to `gcc`.

[5]	Note that unlike other disassembly listings we've been looking at in this and the previous article, this is an object file, not a shared library or executable. Therefore it will contain some relocations for the linker.

[6]	For some good information on this subject, take a look at the AMD64 ABI, and `man gcc`.

[7]	Some assemblers call this instruction `movabs` to distinguish it from the other `mov` instructions that accept a relative argument. The Intel architecture manual, however, keeps naming it just `mov`. Its opcode format is `REX.W + B8 + rd`.

Eli Bendersky's website - Linkers and Loaders

Building static binaries with Go on Linux

Basics - hello world

DNS and user groups

Linking C into our go binary

Linking a libc statically

Using Zig as our C compiler

Summary

Code

Some notes on Luz - an assembler, linker and CPU simulator

The CPU simulator

Assembler

Linker

Debugger and disassembler

The in-progress RTL implementation

Conclusion

Library order in static linking

The basics

The linking process

Simple examples

Circular dependency

Using linker flags to control the process

Back to the original example

Conclusion

How statically linked programs run on Linux

The Linux kernel

Program entry point

The entry point in C code

Decoding the start sequence of C code

The C library start function

Digression: init and fini

Conclusion

Position Independent Code (PIC) in shared libraries on x64

RIP-relative addressing

x64 PIC with data references - an example

x64 PIC with function calls - an example

Performance implications

Extra credit: Non-PIC code on x64

Conclusion

Linking a `libc` statically