Eli Bendersky's website - Databases

Linearizability in distributed systems

2024-10-07T19:16:00-07:00

Linearizability is a strong consistency model in concurrent and distributed systems. From the paper introducing it [1]:

Linearizability provides the illusion that each operation applied by concurrent processes takes effect instantaneously at some point between its invocation and its response.

On first reading (and probably on the second and third...) this sounds a bit abstract, but it really is all there is to it. A slightly different way to think about it is - a linearizable system appears as if there's only one copy of data in existence, and all client operations apply to this data atomically. This post dives deeper into what this means in practice.

Registers

Linearizability is a single-object consistency model (see the "Linearizability vs. Serializability" section below for more on this). It's common in distributed systems literature to talk about a register - a single key-value pair, for example, stored in some distributed database. When clients write and read this register concurrently, we can analyze the history of operations and their results and determine if the system maintains linearizability.

Basic example

The following diagram describes a sequence of register reads and writes by three different clients; some of these operations are done concurrently. Time flows from left to right, and a colored rectangle denotes an operation; its left edge is the operation's start, and its right edge the operation's completion [2].

Here are the events, each with its own number in the yellow bubble:

Client A reads the register and gets the value of 0. The read itself happened at some point in time in the database, denoted on the timeline in the very bottom of the diagram.
Client B reads the value 0. Note that this read operation is partially concurrent with the write operation (3); concurrent operations can execute in any order, but here (2) happened to be executed before (3) (we know this because the value 0 was read, not 1).
Client C writes 1 into the register.
Client A reads 1 from the register. This read is also concurrent with the write, and thus could end up with any result, but since the result in this timeline is 1, we know it happened after (3).
Client B reads 1 from the register.

This sequence of events is valid in a linearizable system, because we can construct a serial history of events (the bottom timeline) that's consistent with our results. Each event occurs instantaneously at some point between the start and finish of the client request.

Compare this to the following sequence, which is not valid:

This sequence is similar to the first one, with one difference: B's read in (5) results in 0. Since (5) is concurrent with (3), when seen in isolation this isn't unreasonable. However, since in (4), client A already observed the value 1 in the register and (4) happens before (5), this sequence is invalid in a linearizable system. We can imagine systems with weaker consistency guarantees producing this history, but such systems are not linearizable.

Another way to look at it is examine the timeline in the bottom of the diagram. Notice that (5) reads 0, after (3) happened. We just can't find a way to arrange this history so it looks sequential - therefore, it's inconsistent with linearizability.

A more subtle example

Here's a more subtle example, taken from the linearizability paper:

This sequence of events is invalid for a linearizable system! To understand why, let's follow the timeline at the bottom of the diagram.

(3) Client B's write of 1 executes before (2) client A's read, because A reads 1 from the register. If the read at (2) happened before the write at (3), it (the read) would result in 0, not 1.

Event (4) has to happen after event (2), since it starts after (2) ends. But we've just reasoned that (2) happens after (3); therefore, (4) happens after (3) - even though these two writes are concurrent, their order is imposed by observing other events.

Finally, since we've just proven that (4) happens after (3), the value in the register at the conclusion of (3) is 0, not 1; therefore, the read of 1 in (5) is invalid. This system cannot be linearizable. As before, you can try to arrange the events in the bottom of the diagram into some sequential order - this attempt will fail, because no consistent sequential order can account for the observed events.

A formal definition

I personally found the formal definition of linearizability in the Herlihy & Wang paper somewhat obscured by attention given to potentially unfinished operations. If we assume that every operation has a start and an end, it's easier to restate the formal definition as follows.

An operation e has the timestamps start(e) and end(e); these are the left and right boundaries of the rectangles in the diagrams shown above.

A history H exists with a strong partial order <_H on operations [3]: e_0 <_H e_1 if end(e_0) precedes start(e_1) in H. Operations unrelated by <_H are said to be concurrent in H. In our diagrams, H represents the observed history (the part of the diagram with the overlapping rectangles). The formal definition captures what it means for us to know that some operations precede others, while other operations are concurrent.

For example, in our last diagram above if H is the history shown, then e_3 <_H e_5, but the pair e_3,e_4 is not in the relation <_H, since these operations are concurrent.

If H is a sequential history, then <_H is a total order. It means there are no concurrent operations.

Now it's time for the definition of linearizability. H is linearizable if:

H is equivalent to some sequential history S
<_H \subseteq <_S

The second item requires a bit of elaboration: recall that <_H and <_S are relations. <_H being a subset of <_S means that the partial order of operations in the real-time history H is preserved in the linearization.

We then call S the linearization of H. In our diagrams, S is the bottom line where operations are shown on the server happening immediately; they are still represented by start(e) and end(e) in the history (we can just assume start(e) and end(e) are infinitesimally close in time, since the DB applies operations atomically).

Linearizability vs. Serializability

Linearizability is often confused with serializability - another consistency model. The two are fundamentally different, though:

Serializability is a multi-object property useful to describe transactions that consist of multiple operations that may potentially touch multiple objects; informally, it means that transactions happen atomically, and their sub-operations cannot be observed in isolation or intermix.
Linearizability is a single-object property, talking about the observed effects on a single register, as this post demonstrates.

For a great taxonomy of consistency models, see this page from Jepsen.

Additional resources

Kyle Kingsbury - on his blog and through his company Jepsen - has a wealth of great resources on the subject of linearizability and other consistency models. Some examples:

The taxonomy, as mentioned above, and the related blog post
Knossos, a linearizability checker: blog post and project page
Jepsen's analysis of etcd has an interesting practical discussion of linearizability in a real-world system

[1]	Herlihy, Maurice P.; Wing, Jeannette M. (1990). "Linearizability: A Correctness Condition for Concurrent Objects". ACM Transactions on Programming Languages and Systems.

[2]	The operation itself happens instantaneously on the server at some moment within the rectangle's boundaries, but we don't know exactly when due to network delays.

[3]	For a refresher on the math used here (relations, orders) see this post.

Accessing PostgreSQL databases in Go

2021-07-17T06:20:00-07:00

This post discusses some options for accessing PostgreSQL databases from Go. I'll only be covering low-level packages that provide access to the underlying database; this post is not about ORMs, which were covered earlier in this blog. The full source code accompanying this post is on GitHub.

We're going to be using a simple data model that could serve as a basis for an online course system (like Coursera):

There is a many-to-many relationship between courses and users (a user can take any number of courses, and each course has multiple users signed up), and a one-to-many relationship between courses and projects (a course has multiple projects, but a project belongs to a single course).

The SQL to create these tables is:

create table if not exists courses (
    id bigserial primary key,
    created_at timestamp(0) with time zone not null default now(),
    title text not null,
    hashtags text[]
);

create table if not exists projects (
    id bigserial primary key,
    name text not null,
    content text not null,
    course_id bigint not null references courses (id) on delete cascade
);

create table if not exists users (
    id bigserial primary key,
    name text not null
);

create table if not exists course_user (
    course_id bigint not null references courses (id) on delete cascade,
    user_id bigint not null references users (id) on delete cascade,
    constraint course_user_key primary key (course_id, user_id)
);

Note that the hashtags column is of the PostgreSQL array type: hashtags text[]; this is on purpose, to demonstrate how custom PostgreSQL types are modeled in the various Go approaches presented here.

database/sql with the pq driver

Probably the most common way to access PostgreSQL databases in Go is using the standard library database/sql, along with pq as the database driver. The full code for this approach, applied to our sample database is available here; I'll present some relevant bits and pieces below:

import (
  "database/sql"
  "fmt"
  "log"
  "os"

  _ "github.com/lib/pq"
)

// Check is a helper that terminates the program with err.Error() logged in
// case err is not nil.
func Check(err error) {
  if err != nil {
    log.Fatal(err)
  }
}

func main() {
  db, err := sql.Open("postgres", os.Getenv("MOOCDSN"))
  Check(err)
  defer db.Close()

  // ... use db here
}

There's the usual blank import of the driver package, which registers itself with database/sql; thereafter, the "postgres" name can be used as a driver name to pass to sql.Open. The path to the database is passed in an env var; for example, it could be something like:

MOOCDSN=postgres://testuser:testpassword@localhost/testmooc

If the database was created with the name testmooc, with the user testuser having access to it.

Following this initialization, we can issue queries to the database via db. Before we look at sample queries, here's the data model translated to Go types:

type course struct {
  Id        int64
  CreatedAt time.Time
  Title     string
  Hashtags  []string
}

type user struct {
  Id   int64
  Name string
}

type project struct {
  Id      int64
  Name    string
  Content string
}

Note that, unlike with ORMs, relationships between tables are not captured here. A course does not have a collection of projects; this is something we need to set up manually when querying the DB. Another thing to note is that Hashtags has the type []string which will be mapped to PostgreSQL's text[].

Here's a sample function wrapping an SQL query:

func dbAllCoursesForUser(db *sql.DB, userId int64) ([]course, error) {
  rows, err := db.Query(`
    select courses.id, courses.created_at, courses.title, courses.hashtags
    from courses
    inner join course_user on courses.id = course_user.course_id
    where course_user.user_id = $1`, userId)
  if err != nil {
    return nil, err
  }
  defer rows.Close()
  var courses []course
  for rows.Next() {
    var c course
    err = rows.Scan(&c.Id, &c.CreatedAt, &c.Title, pq.Array(&c.Hashtags))
    if err != nil {
      return nil, err
    }
    courses = append(courses, c)
  }
  if err := rows.Err(); err != nil {
    return nil, err
  }
  return courses, nil
}

Given a user ID, this function obtains all the courses the user is signed up for, by join-ing the courses table with the course_user linking table. database/sql requires reading the result of the query in a scanning loop, and manually placing the results into structs; it's not aware of any mapping between Go structs and SQL tables. PostgreSQL arrays are read by wrapping with a pq.Array type.

Here's a slightly more involved query, which joins three tables to obtain all the projects the user has to finish (there could be multiple projects per course, and a user could be signed up for multiple courses):

func dbAllProjectsForUser(db *sql.DB, userId int64) ([]project, error) {
  rows, err := db.Query(`
    select projects.id, projects.name, projects.content
    from courses
    inner join course_user on courses.id = course_user.course_id
    inner join projects on courses.id = projects.course_id
    where course_user.user_id = $1`, userId)
  if err != nil {
    return nil, err
  }
  defer rows.Close()
  var projects []project
  for rows.Next() {
    var p project
    err = rows.Scan(&p.Id, &p.Name, &p.Content)
    if err != nil {
      return nil, err
    }
    projects = append(projects, p)
  }
  if err := rows.Err(); err != nil {
    return nil, err
  }
  return projects, nil
}

While the SQL is more complicated, the rest of the code is almost identical to the earlier function.

pgx

While pq has been around for a long time and has served the Go community well, it hasn't been very actively maintained recently. In fact, if you read all the way to the end of its README, you'll find this in the Status section:

This package is effectively in maintenance mode and is not actively developed. Small patches and features are only rarely reviewed and merged. We recommend using pgx which is actively maintained.

So what is pgx? It's a driver and toolkit for PostgreSQL:

pgx aims to be low-level, fast, and performant, while also enabling PostgreSQL-specific features that the standard database/sql package does not allow for.

The driver component of pgx can be used alongside the standard database/sql package.

The pgx package has two distinct modes of operation:

It can serve as a standard driver for database/sql.
It can serve as a direct interface to PostgreSQL, which isn't beholden to the standard API of database/sql, and thus can employ PostgreSQL-specific features and code paths.

To use option (1), we can reuse 99% of the previous sample (the database/sql interface is really very well standardized!). All we have to do is replace the driver import with:

_ "github.com/jackc/pgx/v4/stdlib"

And then change the sql.Open call to invoke the pgx driver:

db, err := sql.Open("pgx", os.Getenv("MOOCDSN"))

We don't have to update the rest of the code [1].

What about the direct interface? For this, we'll have to rejigger our code a bit, since the types are slightly different. The full code for this is available here; here are the salient changes:

ctx := context.Background()
conn, err := pgx.Connect(ctx, os.Getenv("MOOCDSN"))
Check(err)
defer conn.Close(ctx)

Instead of using sql.Open, we call pgx.Connect instead. When it's time to query the DB, our function for grabbing all the courses a user is signed up for would be:

func dbAllCoursesForUser(ctx context.Context, conn *pgx.Conn, userId int64) ([]course, error) {
  rows, err := conn.Query(ctx, `
    select courses.id, courses.created_at, courses.title, courses.hashtags
    from courses
    inner join course_user on courses.id = course_user.course_id
    where course_user.user_id = $1`, userId)
  if err != nil {
    return nil, err
  }
  defer rows.Close()
  var courses []course
  for rows.Next() {
    var c course
    err = rows.Scan(&c.Id, &c.CreatedAt, &c.Title, &c.Hashtags)
    if err != nil {
      return nil, err
    }
    courses = append(courses, c)
  }
  if err := rows.Err(); err != nil {
    return nil, err
  }
  return courses, nil
}

Note that the Go struct types representing table entries remain exactly the same. Reading query results with pgx is very similar to database/sql, but array types no longer need to be wrapped in pq.Array, since pgx supports natively reading PostgreSQL arrays into Go slices.

So, what do we get by using pgx instead of database/sql? According to the feature list on its README, quite a lot, including native support for custom PostgreSQL types, JSON, an advanced connection pool and a whole slew of performance-oriented features. Most notably, pgx uses the PostgreSQL binary protocol directly for faster marshaling and unmarshaling of types. According to pgx's benchmarks, there are considerable performance differences in some cases [2].

sqlx

We've seen a few examples of non-trivial SQL queries being scanned into Go objects so far; all of them involve the same pattern:

The query is submitted
The result is iterated row by row
Each row gets manually unmarshaled into struct fields

One of the biggest complaints about database/sql in Go is the verbosity of this process; particularly the second and third steps above. Why can't we just say:

var courses []course
db.FillInQueryResults(&courses, ....)

After all, many packages in the Go standard library already work this way; for example encoding/json, etc. The reason is the variety of types SQL supports. While JSON has relatively few supported types, SQL has many; moreover, SQL types differ by database. Therefore, it was fairly tricky for the Go project to offer such advanced scanning capabilities in the standard library, and we have to rely on third-party packages instead.

Luckily, an abundance of third-party packages exists just for this purpose. One of the most prominent is sqlx. Let's revisit our sample database querying code, this time using sqlx. The full code for this is available here.

The database setup code is very similar to the vanilla database/sql version:

import (
  "fmt"
  "log"
  "os"

  "github.com/jmoiron/sqlx"
  _ "github.com/lib/pq"
)

func Check(err error) {
  if err != nil {
    log.Fatal(err)
  }
}

func main() {
  db, err := sqlx.Open("postgres", os.Getenv("MOOCDSN"))
  Check(err)
  defer db.Close()

  // ... use db here
}

sqlx.Open wraps sql.Open and uses the same database driver registration mechanism. The type it returns is sqlx.DB, which extends sql.DB with some convenience methods. Here's our function to query all courses a user is signed up for, this time using sqlx:

func dbAllCoursesForUser(db *sqlx.DB, userId int64) ([]course, error) {
  var courses []course
  err := db.Select(&courses, `
    select courses.id, courses.created_at, courses.title, courses.hashtags
    from courses
    inner join course_user on courses.id = course_user.course_id
    where course_user.user_id = $1`, userId)
  if err != nil {
    return nil, err
  }
  return courses, nil
}

This is just what we wanted! The code scans the result into a slice of course objects directly, without needing the row-by-row loop. sqlx accomplishes this feat by using reflection - it examines the underlying type of the struct in the slice and maps DB columns to struct fields automatically. It sometimes needs help, though; for example, our course struct has to be modified as follows:

type course struct {
  Id        int64
  CreatedAt time.Time `db:"created_at"`
  Title     string
  Hashtags  pq.StringArray
}

Since sqlx won't map the database created_at column to the CreatedAt field automatically, we have to provide an instruction to do so explicitly in a field tag.

sqlx requires an underlying database/sql driver for the actual DB interactions. In the example above, we've been using pq, but the stdlib driver of pgx can be used as well. Unfortunately, sqlx does not support the native pgx driver. However, a different package called scany does support both the native and the stdlib drivers of pgx. I wrote another version of this sample, using scany; I won't show this code here, since it's very similar to the sqlx example, but you can find it on GitHub.

Is `sqlx` worth it?

Looking at our dbAllCoursesForUser function, the version using sqlx saves about 14 lines of code compared to the vanilla scan with database/sql. I'm on record saying that ORMs are unlikely to be worthwhile in Go, but what about sqlx? Is saving 14 LOC per DB query function worth the trouble of an additional dependency, with its potential quirks, bugs and leaky abstractions?

This question is hard to answer globally, so I'll just say "it depends".

On one hand, 14 LOC per DB query is really not much. Say you have 50 possible SQL queries in your application, this saves 700 LOC of trivial and repetitive code. Is that a lot? In most cases, almost certainly not. In the end, it all boils down to the central thesis of the benefits of extra dependencies as a function of effort.

On the other hand, as opposed to ORMs, packages like sqlx and scany provide a fairly focused utility with not very much magic involved. After all, the standard library already has similar tools built in for unmarshaling JSON, so this is a tried-and-true method that can work for data in relational databases as well. Since the utility of these packages is focused, they are not terribly hard to tear out of a codebase and replace, in case things don't go as expected, so they also present a considerably smaller risk than going all-in on ORMs.

To conclude, packages like sqlx and scany provide a middle ground between raw SQL access and full-blown ORMs; this means mid-of-the-way advantages as well as disadvantages.

[1] There's a small nuance to be aware of if you're following along with the code samples, trying to run them. To be able to read PostgreSQL arrays in Go using the database/sql driver component of pgx, we still need to import pq in order to use its pq.Array type. This type provides custom readers that are required to read custom DB types via the standard interface. When using the pgx direct interface, this is not necessary since pgx supports reading PostgreSQL arrays directly into slices. See this pgx issue for additional information.

[2]	As usual with benchmarks, YMMV. Every case is different, and I imagine that in many scenarios the network overhead of a PostgreSQL connection will subsume any difference observable between different drivers.

To ORM or not to ORM

2019-05-07T06:47:00-07:00

I've been enjoying using Go's database/sql package for working with databases. Recently, some mentions of gorm piqued my curiosity about using ORMs in Go vs. using database/sql directly. Having had some mixed experiences with ORMs in the past, I decided to start with a practical experiment by writing the same simple application with and without gorm, and comparing the results in terms of effort spent.

This led me to write down some general thoughts on the benefits and drawbacks of ORMs. If that kind of thing interests you, read on!

My no-ORM vs. ORM experiment

My experiment involves defining a simple database that could be a subset of a blogging engine, as well as write some Go code that populates and queries this database and compare how it looks using plain SQL vs. using an ORM.

This is the database schema:

While simple, this schema demonstrates an idiomatic normalized database that most likely contains all the elements one needs to build simple wiki or blog apps - it has both one-to-many relationships (between posts and comments) and many-to-many relationships (between posts and tags). If you prefer to read DB schemas as SQL, here's the definition taken from the code sample:

create table Post (
    postID integer primary key,
    published date,
    title text,
    content text
);

create table Comment (
    commentID integer primary key,
    postID integer,
    author text,
    published date,
    content text,

    -- One-to-many relationship between Post and Comment; each Comment
    -- references a Post it's logically attached to.
    foreign key(postID) references Post(postID)
);

create table Tag (
    tagID integer primary key,
    name text unique
);

-- Linking table for the many-to-many relationship between Tag and Post
create table PostTag (
    postID integer,
    tagID integer,

    foreign key(postID) references Post(postID),
    foreign key(tagID) references Tag(tagID)
);

This SQL was tested with SQLIte; other RDBMSs may need minor adjustments. When using gorm, there is no need to write this SQL. Instead, we define "objects" (really structs) with some magic field tags for gorm:

type Post struct {
  gorm.Model
  Published time.Time
  Title     string
  Content   string
  Comments  []Comment `gorm:"foreignkey:PostID"`
  Tags      []*Tag    `gorm:"many2many:post_tags;"`
}

type Tag struct {
  gorm.Model
  Name  string
  Posts []*Post `gorm:"many2many:post_tags;"`
}

type Comment struct {
  gorm.Model
  Author    string
  Published time.Time
  Content   string
  PostID    int64
}

The code working with this database comes in two variants:

No-ORM; using plain SQL queries through the database/sql package.
ORM; using the gorm library for database access.

The sample is doing several things:

Add some data (posts, comments, tags) to the DB.
Query all posts in a given tag.
Query all post details (all comments attached to it, all tags it's marked with).

Just as an example, here are the two variants for task (2) - finding all posts in a given tag (this could be to populate some sort of archives listing page on the blog). First, no-ORM:

func dbAllPostsInTag(db *sql.DB, tagID int64) ([]post, error) {
  rows, err := db.Query(`
    select Post.postID, Post.published, Post.title, Post.content
    from Post
    inner join PostTag on Post.postID = PostTag.postID
    where PostTag.tagID = ?`, tagID)
  if err != nil {
    return nil, err
  }
  var posts []post
  for rows.Next() {
    var p post
    err = rows.Scan(&p.Id, &p.Published, &p.Title, &p.Content)
    if err != nil {
      return nil, err
    }
    posts = append(posts, p)
  }
  return posts, nil
}

This is fairly straightforward if you know SQL. We have to perform an inner join between Post and PostTag and filter it by the tag ID. The rest of the code is just iterating over the results.

Next, the ORM:

func allPostsInTag(db *gorm.DB, t *Tag) ([]Post, error) {
  var posts []Post
  r := db.Model(t).Related(&posts, "Posts")
  if r.Error != nil {
    return nil, r.Error
  }
  return posts, nil
}

In the ORM code, we tend to use objects directly (Tag here) rather than their IDs, for the same effect. The SQL query generated by gorm here will be pretty much the same as the one I wrote manually in the no-ORM variant.

Apart from generating the SQL for us, gorm also provides an easier way to populate a slice of results. In the code using database/sql we explicitly loop over the results, scanning each row separately into individual struct fields. gorm's Related method (and other similar querying methods) will populate structs automatically and will also scan the whole result set in one go.

Feel free to play with the code! I was pleasantly surprised at the amount of code gorm saves here (about 50% savings for the DB-intensive part of the code), and for these simple queries using gorm wasn't hard - the invocations are taken from API docs in a straightforward manner. The only complaint I have about my specific example is that setting up the many-to-many relationship between Post and Tag was a bit finicky, and the gorm struct field tags look ugly and magical.

Layered complexity rears its ugly head

The problem with simple experiments like that above is that it's often difficult to tickle the system's boundaries. It obviously works well for simple cases, but I was interested to find out what happens when it's pushed to the limit - how does it handle complicated queries and DB schemas? So I turned to browsing Stack Overflow. There are many gorm-related questions, and sure enough, the usual layered complexity problem is immediately apparent (example 1, example 2). Let me explain what I mean by that.

Any situation where complex functionality is wrapped in another layer runs the risk of increasing the overall complexity when the wrapping layer is itself complicated. This often comes along with leaky abstractions - wherein the wrapping layer can't do a perfect job wrapping the underlying functionality, and forces programmers to fight with both layers simultaneously.

Unfortunately, gorm is very susceptible to this problem. Stack Overflow has an endless supply of problems where users end up fighting complexities imposed by gorm itself, working around its limitations, and so on. Few things are as aggravating as knowing exactly what you want (i.e. which SQL query you want it to issue) but not being able to concoct the right sequence of gorm calls to end up with that query.

Pros and Cons of using an ORM

One key advantage of using an ORM is apparent from my experiment: it saves quite a bit of tedious coding. About 50% savings in DB-centered code is nontrivial and can make a real difference for some applications.

Another advantage that wasn't obvious here is abstraction from different database backends. This may be less of an issue in Go, however, since database/sql already provides a great portable layer. In languages that lack a standardized SQL access layer, this advantage is much stronger.

As for the disadvantages:

Another layer to learn, with all the idiosyncracies, special syntax, magical tags, and so on. This is mainly a disadvantage if you're already experienced with SQL itself.
Even if you're not experienced with SQL, there is a vast bank of knowledge out there and many folks who can help with answers. Any single ORM is much more obscure knowledge not shared by many, and you will spend considerable amounts of time figuring out how to force-feed it things.
Debugging query performance is challenging, because we're abstracted one level further from "the metal". Sometimes quite a bit of tweaking is required to get the ORM to generate the right queries for you, and this is frustrating when you already know which queries you need.

Finally, a disadvantage that only becomes apparent in the long term: while SQL stays pretty much constant over the years, ORMs are language-specific and also tend to appear and disappear all the time. Every popular language has a large variety of ORMs to choose from; as you move from one team/company/project to another, you may be expected to switch, and that's additional mental burden. Or you may switch languages altogether. SQL is a much more stable layer that stays with you across teams/languages/projects.

Conclusion

Having implemented a simple application skeleton using raw SQL and compared it to an implementation using gorm, I can see the appeal of ORMs in reducing boilerplate. I can also remember myself from many years ago being a DB newbie and using Django with its ORM to implement an application - it was nice! I didn't have to think about SQL or the underlying DB much, it just worked. But that use case was really simple.

With my "experienced and salty" hat on, I can also see many disadvantages in using an ORM. Specifically, I don't think an ORM is useful for me in a language like Go which already has a good SQL interface that's mostly portable across DB backends. I'd much rather spend an extra bit of time typing, but this will save me time reading ORM's documentation, optimizing my queries, and most importantly debugging.

I could see an ORM still being useful in Go if your job is to write large numbers of simple CRUD-like applications, where the savings in typing overcome the disadvantages. In the end, it all boils down to the central thesis of the benefits of extra dependencies as a function of effort: where there is significant effort to spend on a project outside the DB-interfacing code - which should be the case for programs that aren't simple CRUDs - the ORM dependency is not worth it, in my opinion.

SQL inner and outer joins

2019-04-09T05:28:00-07:00

If you store data in a relational database, it's good practice to have the data normalized. This typically requires splitting data to multiple tables that are logically connected through keys. As a result, most non-trivial queries require joins on multiple tables to gather all the interesting columns. This post is a brief tour of SQL joins, focusing on the differences between inner and outer joins.

Cross join

To understand SQL joins, it's best to start with cross joins, because they are the simplest combination of tables supported by SQL. A cross join occurs when we write:

select * from t1, t2;

Throughout this post, we'll be working with two sample tables called t1 and t2:

    t1                         t2

 id |   name              code | id
----+----------          ------+----
  1 | Joanne              x    |  2
  2 | Sam                 z    |  3
  3 | Emmanuel            a    |  7
  4 | Brayden

The SQL code to create these tables and run all the examples in this post is available here. All the code was tested on PostgreSQL 9.5.

Running the cross join on these tables results in:

 id |   name   | code | id
----+----------+------+----
  1 | Joanne   | x    |  2
  2 | Sam      | x    |  2
  3 | Emmanuel | x    |  2
  4 | Brayden  | x    |  2
  1 | Joanne   | z    |  3
  2 | Sam      | z    |  3
  3 | Emmanuel | z    |  3
  4 | Brayden  | z    |  3
  1 | Joanne   | a    |  7
  2 | Sam      | a    |  7
  3 | Emmanuel | a    |  7
  4 | Brayden  | a    |  7

The cross join performs a cross product (or Cartesian product) between the two tables. For each row in t1, it adds all possible rows from t2. The resulting table has all the columns of t1 and of t2, and its number of rows is the product of numbers of rows in t1 and t2.

I find cross joins to be a good starting point because they make inner joins much easier to understand. They are also the basis of joins in relational algebra.

SQL also supports a more explicit way to invoke a cross join:

select * from t1 cross join t2;

This is equivalent to the first statement.

Inner join

An important component of SQL queries is filtering results with a where clause. For example, we can create the following (slightly nonsensical) filter on the cross join shown earlier:

select * from t1, t2 where t2.code = 'x' and t1.name like '%d%'

Resulting in:

 id |   name   | code | id
----+----------+------+----
  4 | Brayden  | x    |  2

One filter that's particularly useful when crossing two tables is checking whether there's a match on some column value. Both t1 and t2 have an id column; let's assume these IDs refer to the same thing, and that we want to find all combinations of rows from the two tables where the IDs match. We can do:

select * from t1, t2 where t1.id = t2.id;

Resulting in:

 id |   name   | code | id
----+----------+------+----
  2 | Sam      | x    |  2
  3 | Emmanuel | z    |  3

This kind of filtering is so useful that it has its own concept: the inner join [1]:

select * from t1 inner join t2 on t1.id = t2.id;

It produces the exact same result table. When the names of the columns we compare are the same in the two tables, there's an even shorter syntax that can be used:

select * from t1 inner join t2 using (id);

The result of this will only have a single id column, since we're making it explicit that ids match between the tables:

 id |   name   | code
----+----------+------
  2 | Sam      | x
  3 | Emmanuel | z

I find the filtering equivalence very useful to understand inner joins. Just remember that it's a cross product of the two tables where only rows that satisfy a certain condition are returned. You may be wondering what's the difference between using where filtering and inner join ... on. While the two are logically equivalent, some things to keep in mind:

At least theoretically, inner join ... on is more efficient because in multi-table joins (which is common) we get to apply the filtering per join and not at the end on one huge table. With modern SQL query optimizers it's not clear whether this is a real advantage, however. It's quite likely that the optimizer will generate exactly the same sequence of low-level operations for the two.
In terms of readability, it's much nicer to be able to see what the join is on close to the join itself, rather than in the end of the query in one large where filter. This can be significant for multi-table joins.

As an example, consider customers making orders, with order details in a separate table (since customers could have multiple orders). We could have a complex join done as:

select *
  from customers, orders, order_details
  where customers.id = order_details.customerid and
        orders.id = order_details.orderid

Compared with:

select *
  from customers
    inner join order_details on customer.id = order_details.customerid
    inner join orders on orders.id = order_details.orderid

In the latter it's much clearer what the criteria for each join is.

Finally, I'll mention that some databases support the natural join, which is a shortcut for "inner join tables on the columns that have the same name". The following query is equivalent to the variant with using shown above:

select * from t1 natural join t2;

Natural join is a term from relational algebra, and it's not commonly used in SQL queries.

Outer join

While the inner join is simple to understand as a special case of the cross product, outer join is a bit trickier. Luckily, it's not hard to grok outer joins once you undererstand inner joins, so we can build this knowledge step by step.

Let's get back to our tables t1 and t2. We could assign a logical meaning to the inner join using (id) as "show me all the codes (from t2) matching names (from t1)". The result is two rows where a match on id was found in the two tables. However, sometimes we want something slightly different; we want to ask "show me all the names (from t1) and all the codes (from t2) that match them, if any". In other words, we want all the names to be in the results, perhaps with null values for code where no match was found in the t2 table [2].

Let's break this request to pieces. We want:

All names from t1 that have a match in t2, with the code from t2
All names from t1 that have no match in t2, with null for the code

In SQL we can express this as follows:

select id, name, code
    from t1 inner join t2 using (id)
  union
select id, name, null
    from t1 where id not in (select id from t2);

Some things to note:

The first query is precisely our inner join from the previous section, and it's answering the first piece.
The second query lists all the names that don't have a match in t2 using a subquery.
We're listing the column names explicitly here because column names must match exactly for the two tables being union-ed.

What we just wrote is called a left outer join in SQL [3], and can be more easily written as:

select * from t1 left outer join t2 using (id);

The result is:

 id |   name   | code
----+----------+------
  2 | Sam      | x
  3 | Emmanuel | z
  4 | Brayden  |
  1 | Joanne   |

This is the left outer join because we want all the rows from the left-hand side table to appear in the result. As you may have guessed, there's also a right outer join:

select * from t1 right outer join t2 using (id);

 id |   name   | code
----+----------+------
  2 | Sam      | x
  3 | Emmanuel | z
  7 |          | a

Here all the rows from the right-hand side table appear in the result, with a matching column (name) from the left-hand side if found, null otherwise.

Finally, we may want rows from both sides of the join to always appear in the result table. That's called a full outer join:

select * from t1 full outer join t2 using (id);

Resulting in:

 id |   name   | code
----+----------+------
  2 | Sam      | x
  3 | Emmanuel | z
  7 |          | a
  4 | Brayden  |
  1 | Joanne   |

A full outer join is straightforward to express using a union of left and right joins:

select * from t1 left join t2 using (id)
  union
select * from t1 right join t2 using (id);

There's a slight caveat, though. While union removes duplicates, full outer join does not; therefore, the results can be different in some special cases. In the event that you care about seeing duplicates in the output and the database doesn't support a full outer join, this is a more accurate (though less efficient) translation:

select * from t1 left join t2 using (id)
  union all
select * from t1 right join t2 using (id) where t1.id is null;

Joins on multiple columns

The examples so far showed joins on a single shared column - id. While this is the most common case, sometimes more complex matching criteria are used. SQL doesn't restrict the syntax of join to a single condition, so we can join on multiple columns and arbitrary conditions. Let's add another column to our two tables:

        t1                             t2

 id |   name   | ranking        code | id | ranking
----+----------+---------      ------+----+--------
  1 | Joanne   |       7        x    |  2 |       8
  2 | Sam      |       7        z    |  3 |       6
  3 | Emmanuel |       6
  4 | Brayden  |       2

We can run joins on both id and ranking:

select * from t1 inner join t2 on t1.id = t2.id and t1.ranking = t2.ranking;

Resulting in:

 id |   name   | ranking | code | id | ranking
----+----------+---------+------+----+---------
  3 | Emmanuel |       6 | z    |  3 |       6

And with using:

select * from t1 inner join t2 using (id, ranking);

Resulting in:

 id | ranking |   name   | code
----+---------+----------+------
  3 |       6 | Emmanuel | z

Similarly, we can run outer joins:

select * from t1 left outer join t2 using (id, ranking);

Resulting in:

 id | ranking |   name   | code
----+---------+----------+------
  3 |       6 | Emmanuel | z
  2 |       7 | Sam      |
  4 |       2 | Brayden  |
  1 |       7 | Joanne   |

And so on.

Joins on multiple tables

In real-life databases, data is often split to multiple tables; it's not uncommon for queries to probe 4-5 or more tables to gather all the interesting information. Let's use three table for an example. We'll have a table of customers and a table of items:

         customers                       items

 customerid |   name          itemid | description | price
------------+----------      --------+-------------+-------
          1 | Robert               1 | Napkins     |   1.5
          2 | Jennifer             2 | Granola     |  4.25
          3 | Yoshi                3 | Cheese      |     3
          4 | Xi

In addition we'll have a linking table to record orders made by customers:

 customerid | itemid | orderdate
------------+--------+------------
          1 |      2 | 2019-03-02
          1 |      3 | 2019-03-02
          1 |      1 | 2019-03-03
          2 |      1 | 2019-02-22
          3 |      3 | 2019-01-15
          3 |      2 | 2019-02-20
          4 |      3 | 2019-02-21
          4 |      3 | 2019-02-22

We may be interested in all the customers who ordered cheese, and the date of the order. This requires joining all three tables:

select name, orderdate, description
    from (customers
    inner join orders using (customerid))
    inner join items using (itemid)
    where items.description = 'Cheese';

Resulting in:

  name  | orderdate  | description
--------+------------+-------------
 Robert | 2019-03-02 | Cheese
 Yoshi  | 2019-01-15 | Cheese
 Xi     | 2019-02-21 | Cheese
 Xi     | 2019-02-22 | Cheese

Note the parens around the first join. This is not strictly necessary for this query, but I find it useful to control the order of joining explicitly. We can join as many tables as we want, but the order has to make sense. Each join produces a new logical table that participates in other joins, and for some queries the order of joins is important.

While it will be more common to see sequences of inner joins in such queries, it's also possible to mix and match with outer joins; whatever makes sense.

[1]	When we use the `join` keyword in SQL, inner join is the default, so the keyword `inner` is optional. That said, to distinguish inner joins from outer joins IMHO it's preferable to be explicit.

[2] This sounds contrived with our simplistic tables, but in reality it's an extremely common database query. Imagine our t1 is customers with unique IDs and names, and our t2 is some code assigned to each customer. Suppose we want to display all our customers, regardless of who already has a code assigned. For customers that do have a code we want to show it.

[3]	With the keyword `left` before a `join`, the keyword `outer` is optional, so we could just say `left join` instead of `left outer join`. I like the explicitness of having `outer` there. The same applies for `right` and `full` joins.

Design patterns in Go's database/sql package

2019-03-27T06:25:00-07:00

Using SQL databases from Go is easy, in three steps:

// Step 1: import the main SQL package
import "database/sql"

// Step 2: import a driver package to use a specific SQL database
import _ "github.com/mattn/go-sqlite3"

// Step 3: open a database using a registered driver name
func main() {
  // ...
  db, err := sql.Open("sqlite3", "database.db")
  // ...
}

From this point on, the db object can be used to query and modify the database, with the same code suitable for all the supported SQL databases. If we want to change our database from SQLite to PostgreSQL, it's very likely that we only need to import a different driver and provide a different name in the call to sql.Open [1].

In this post I want to briefly examine some of the design patterns and architecture behind database/sql that makes this all possible.

The main design pattern

The architecture of database/sql is governed by one overarching design pattern. I was trying to figure out which of the classical design patterns it resembles most, and the Strategy Pattern seems the closest, though it's not quite that. Let me know if you can think of a closer correspondence [2].

It goes like this: we have a common interface we want to present to users, with an implementation that's specific to every DB backend. Obviously, this sounds like a classic interface + implementation, which Go is particularly good at with its robust support for interfaces.

So the first idea would be: create some DB interface which the user interacts with, and each backend implements this interface. Sounds simple, right?

Sure, but there are some issues with this approach. Remember that Go recommends interfaces to be small, with just a handful of methods to implement. Here we'd need a much larger DB interface, and this leads to problems:

Adding user-facing capabilities is difficult because they may require adding methods to the interface. This breaks all the interface implementations and requires multiple standalone projects to update their code.
Encapsulating functionality that is common to all database backends is difficult, because there is no natural place to add it if the user interacts directly with the DB interface. It has to be implemented separately for each backend, which is wasteful and logistically complicated.
If backends want to add optional capabilities, this is challenging with a single interface without resorting to type-casts for specific backends.

Therefore, a better idea seems to be: split up the user-facing type and functionality from the common backend interface. Graphically, it looks like this:

DB is a user-facing type. It's not an interface, but a concrete type (a struct) implemented in database/sql itself. It is backend-independent and encapsulates a lot of functionality that is common to all backends, like connection pooling.

To do backend-specific work (such as issue SQL queries to the actual database), DB uses an interface called database/sql/driver.Driver (and several other interfaces that define connections, transactions, etc). This interface is lower-level, and it's implemented by each database backend. In the diagram above we can see implementations from the pq package (for PostreSQL) and from the sqlite3 package.

This approach helps database/sql elegantly address the problems mentioned earlier:

Adding user-facing capabilities doesn't necessarily require an interface change now, as long as the capability can be implemented in the backend-independent layer (DB and its sister types).
Functionality that's common to all database backends now has a natural place to be in. I've mentioned connection pooling, but there is a lot of other stuff the backend-independent types in database/sql add on top of the backend-specific implementations. Another example: handling retries for bad connection to the database server.
If backends add optional capabilities, these can be selectively utilized in the backend-independent layer without exposing them directly to the user.

Registering drivers

Another interesting aspect of the design of database/sql is how database drivers register themselves with the main package. It's a nice example of implementing compile-time plugins in Go.

As the code sample at the top of this post shows, database/sql knows about the imported drivers' names, and can open them by name with sql.Open. How does that work?

The trick is in the blank import:

import _ "github.com/mattn/go-sqlite3"

While it doesn't actually import any names from the package, it does invoke its init function, which in case of sqlite3 is:

func init() {
    sql.Register("sqlite3", &SQLiteDriver{})
}

In sql.go, Register adds a mapping from a string name to an implementation of the driver.Driver interface; the mapping is in a global map:

var (
    driversMu sync.RWMutex
    drivers   = make(map[string]driver.Driver)
)

// Register makes a database driver available by the provided name.
// If Register is called twice with the same name or if driver is nil,
// it panics.
func Register(name string, driver driver.Driver) {
    driversMu.Lock()
    defer driversMu.Unlock()
    if driver == nil {
        panic("sql: Register driver is nil")
    }
    if _, dup := drivers[name]; dup {
        panic("sql: Register called twice for driver " + name)
    }
    drivers[name] = driver
}

When sql.Open is called, it looks up the name in the drivers map and can then instantiate a DB object with the proper driver implementation attached. You can also call the sql.Drivers function at any time to get the names of all the registered drivers.

This approach implements a compile-time plugin, because the imports for the included backends happen when the Go code is compiled. The binary has a fixed set of database drivers built into it. Go also has support for run-time plugins, but that is a topic for a separate post.

Custom types with the `Scanner` interface

Another interesting architectural feature of the database/sql package is supporting storage and retrieval of custom types in the database. The Rows.Scan method is typically used to read columns from a row. It takes a sequence of interface{} to be generic, using a type switch underneath to select the right reader depending on the type of an argument.

For customization, Rows.Scan supports types that implement the sql.Scanner interface, and then invokes their Scan method to perform the actual data read.

One built-in example is sql.NullString. If we try to Scan a column into a string variable:

var id int
var username string
err = rows.Scan(&id, &username)

and that column has a NULL value, we'll get an error:

sql: Scan error on column index 1, name "username":
    unsupported Scan, storing driver.Value type <nil> into type *string

We can avoid this by using a sql.NullString instead:

var id int
var username sql.NullString
err = rows.Scan(&id, &username)

Here username will have its Valid field set to false for a NULL column. This works because NullString implements the Scanner interface.

A more interesting example involves types that are specific to certain database backends. For example, while PostgrSQL supports array types, some other databases (like SQLite) do not. So database/sql cannot support array types natively, but features like the Scanner interface make it possible for user code to interact with such data fairly easily anyway.

To extend the previous example, suppose our rows also have a list of activities (as strings) for each user [3]. Then the Scan would go like this:

var id int
var username sql.NullString
var activities []string
err = rows.Scan(&id, &username, pq.Array(&activities))

The pq.Array function is provided by the pq PostgreSQL binding. It takes a slice and converts it to an anonymous type that implements the sql.Scanner interface.

This is a nice way to escape the abstraction when necessary. Even though it's great to have a uniform interface to access many kinds of databases, sometimes we really do want to use a specific DB with its specific features. It would be a shame to give up database/sql in this case, and we don't have to - because of these features that let specific database backends provide custom behavior.

[1]	Assuming we only use standard SQL syntax in our queries that both databases support, of course.

[2]	I first encountered an explicit discussion of this pattern in the Go CDK project, which I recently joined. The Go CDK uses a similar approach for its portable types, and its design documentation calls it the portable type and driver pattern.

[3]	I realize that multi-valued fields are not good relational design. This is just an example.

Eli Bendersky's website - Databases

Linearizability in distributed systems

Registers

Basic example

A more subtle example

A formal definition

Linearizability vs. Serializability

Additional resources

Accessing PostgreSQL databases in Go

database/sql with the pq driver

pgx

sqlx

Is sqlx worth it?

To ORM or not to ORM

My no-ORM vs. ORM experiment

Layered complexity rears its ugly head

Pros and Cons of using an ORM

Conclusion

SQL inner and outer joins

Cross join

Inner join

Outer join

Joins on multiple columns

Joins on multiple tables

Design patterns in Go's database/sql package

The main design pattern

Registering drivers

Custom types with the Scanner interface

Is `sqlx` worth it?

Custom types with the `Scanner` interface