Eli Bendersky's website - Network Programming

Implementing Raft: Part 5 - Exactly-once delivery

2024-12-18T06:01:00-08:00

This is Part 5 in a series of posts describing the Raft distributed consensus algorithm and its complete implementation in Go. Here is a list of posts in the series:

Part 0: Introduction
Part 1: Elections
Part 2: Commands and log replication
Part 3: Persistence and optimizations
Part 4: Key/Value database
Part 5: Exactly-once delivery (this post)

In this part, we're completing the implementation of a replicated key / value database based on Raft consensus. At the end of Part 4 we discussed a consistency issue that may arise due to client retry logic; now is the time to address it.

All the code for this part is located in this directory.

Adding an `APPEND` operation to our database

As a quick reminder, these are the basic operations our KV DB from part 4 supports:

PUT(k,v): assign value v to key k
GET(k): retrieve the value associated with key k
CAS(k, cmp, v): atomic compare-and-swap. First, it reads curV - the current value associated with key k. If curV==cmp, assigns value v to k instead; otherwise, it's a no-op. In any case, curV is returned.

Let's add another operation to this set; this is APPEND(k,v), which appends v to the value of key k (in our implementation, keys and values are both arbitrary Go strings); if there was no k in the DB before this operation, it behaves like PUT(k,v).

For example, consider this sequence of commands (in order from left to right):

PUT("x","foo")  APPEND("x", "bar")  APPEND("y","hello")

Applied to an empty DB, these commands will result in these keys / values:

x=foobar
y=hello

The problem with client retries - demonstrated with `APPEND`

The way our KV client works is described in detail in Part 4. As a reminder, the client tries the KV services one by one, submitting a command to them until it gets a success response from a leader. The client also remembers which service was the leader the last time it tried, to avoid wasting time on the search next time.

Suppose we've already submitted PUT("x","foo") successfully to the database, and now we want to send the APPEND("x","bar") command. Suppose also that our client remembers that service B was the leader (in a cluster of three services: A, B and C). It sends the APPEND("x","bar") request to service B. What happens if the client doesn't get a response from B? It assumes something happened to B (maybe it has crashed or was partitioned from the network), and retries the same request - sending it to C.

But now suppose that B actually received the APPEND command and committed it to the Raft cluster, but crashed before sending the HTTP response back to the client (or maybe the HTTP response got delayed beyond the client's timeout, due to a network glitch). Due to the same error, B then loses cluster leadership. The client will keep retrying this request, until it finds a leader that answers with success; therefore, the APPEND may be applied twice (or even more times, if the failure mode recurs) and the value of x in the DB will end up being "foobarbar". This is bad!

You may be tempted to blame the client's retry behavior here; but let's think this through. Suppose we didn't have the client layer doing retries; we send an APPEND command to a service, and don't hear anything back. What do we do next? Is there any way to know that the request was actually committed? Well, we can send a GET request to check, but this quickly gets complicated in a real distributed system, because our operation is no longer atomic (some other client may have changed the key's value since then, so what is our GET supposed to check?).

The problem isn't the retry itself; it's retrying with insufficient safety guarantees in the core algorithm.

Solving the retry problem with command de-duplication

The problem described above isn't just an issue in our implementation. It's explicitly called out in the original Raft paper, in section 8:

However, as described so far Raft can execute a command multiple times: for example, if the leader crashes after committing the log entry but before responding to the client, the client will retry the command with a new leader, causing it to be executed a second time. The solution is for clients to assign unique serial numbers to every command. Then, the state machine tracks the latest serial number processed for each client, along with the associated response. If it receives a command whose serial number has already been executed, it responds immediately without re-executing the request.

The paper also suggests a solution to the problem, and this is what we're going to implement. If we can uniquely identify commands committed to the Raft log, the KV service can avoid applying the same commands twice.

The idea is to identify commands uniquely as follows:

Each client has a globally unique ID
Each command sent by a client has a unique ID, distinct from all other IDs sent by the client. Moreover, to make our algorithm efficient this ID is monotonically increasing; assuming each client has its own monotonic clock, a command sent at time T_1 will have an ID larger than a command sent at T_0 iff T_1>T_0.

In our APPEND example, let's say the client's ID is 42, and let's say the ID of the APPEND("x","bar") command it sends is 1. The command is sent to B, which commits it successfully - with the ID tuple (42,1) - to the Raft log; B crashes before responding to the client, so the client retries the command with C. Since it's the same command, it has the same ID tuple (42,1). The KV service in C will notice that such an ID was already applied, and will not apply it again [1]!

Implementing de-duplication

We'll start with the client. Two fields are added to the KVClient struct:

// clientID is a unique identifier for a client; it's managed internally
// in this file by incrementing the clientCount global.
clientID int64

// requestID is a unique identifier for a request a specific client makes;
// each client manages its own requestID, and increments it monotonically and
// atomically each time the user asks to send a new request.
requestID atomic.Int64

The clientID field is assigned when a client is created, using a global atomic that auto-increments:

func New(serviceAddrs []string) *KVClient {
  return &KVClient{
    // ... other fields
    clientID:      clientCount.Add(1),
  }
}

// clientCount is used to assign unique identifiers to distinct clients.
var clientCount atomic.Int64

This provides uniqueness for our tests, but in realistic applications you'll probably want something stronger. A simple and pragmatic approach could be using an UUID here - I'll leave this as an exercise for motivated readers.

The requestID field tracks the ID of the last request this client has sent. Each time the client sends a new request, this is incremented [2]. For example, here's the new Append method:

// Append the value to the key in the store. Returns an error, or
// (prevValue, keyFound, false), where keyFound specifies whether the key was
// found in the store prior to this command, and prevValue is its previous
// value if it was found.
func (c *KVClient) Append(ctx context.Context, key string, value string) (string, bool, error) {
  appendReq := api.AppendRequest{
    Key:       key,
    Value:     value,
    ClientID:  c.clientID,
    RequestID: c.requestID.Add(1),
  }
  var appendResp api.AppendResponse
  err := c.send(ctx, "append", appendReq, &appendResp)
  return appendResp.PrevValue, appendResp.KeyFound, err
}

Both IDs are part of the HTTP request sent to service. Here's the request struct for appends:

type AppendRequest struct {
  Key   string
  Value string

  ClientID  int64
  RequestID int64
}

The other commands are modified similarly; all the retry logic (in the send method) remains the same - it just keeps retrying with the same client+request IDs.

The changes in the service are slightly deeper, but not too difficult overall. First, we add a field to the KVService struct:

// lastRequestIDPerClient helps de-duplicate client requests. It stores the
// last request ID that was applied by the updater per client; the assumption
// is that client IDs are unique (keys in this map), and for each client the
// requests IDs (values in this map) are unique and monotonically increasing.
lastRequestIDPerClient map[int64]int64

We also add some fields to the Command struct:

// ClientID and RequestID uniquely identify the request+client.
ClientID, RequestID int64

// IsDuplicate is used to mark the command as a duplicate by the updater. When
// the updater notices a command that has a client+request ID that has already
// been executed, the command is not applied to the datastore; instead,
// IsDuplicate is set to true.
IsDuplicate bool

As a reminder, Command is the "payload" we submit to the Raft log [3].

The bulk of the logic is in the goroutine running runUpdater:

func (kvs *KVService) runUpdater() {
  go func() {
    for entry := range kvs.commitChan {
      cmd := entry.Command.(Command)

      // Duplicate command detection.
      // Only accept this request if its ID is higher than the last request from
      // this client.
      lastReqID, ok := kvs.lastRequestIDPerClient[cmd.ClientID]
      if ok && lastReqID >= cmd.RequestID {
        kvs.kvlog("duplicate request id=%v, from client id=%v", cmd.RequestID, cmd.ClientID)
        // Duplicate: this request ID was already applied in the past!
        cmd = Command{
          Kind:        cmd.Kind,
          IsDuplicate: true,
        }
      } else {
        kvs.lastRequestIDPerClient[cmd.ClientID] = cmd.RequestID

        switch cmd.Kind {
        case CommandGet:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.Get(cmd.Key)
        case CommandPut:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.Put(cmd.Key, cmd.Value)
        case CommandAppend:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.Append(cmd.Key, cmd.Value)
        case CommandCAS:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.CAS(cmd.Key, cmd.CompareValue, cmd.Value)
        default:
          panic(fmt.Errorf("unexpected command %v", cmd))
        }
      }

      // Forward this command to the subscriber interested in its index, and
      // close the subscription - it's single-use.
      if sub := kvs.popCommitSubscription(entry.Index); sub != nil {
        sub <- cmd
        close(sub)
      }
    }
  }()
}

This relies on the IDs from each client being monotonically increasing, and thus we only have to maintain O(1) of state per client [4]. When we apply a client request to the state machine, we remember the ID of this request. If the code is ever asked to apply the same ID (or a lower ID), it refuses, marking the command as IsDuplicate=true instead. Then, the HTTP handler that tried to submit the command has to deal with this situation; for example, in handleAppend:

sub := kvs.createCommitSubscription(logIndex)

select {
case commitCmd := <-sub:
  if commitCmd.ServiceID == kvs.id {
    if commitCmd.IsDuplicate {
      kvs.sendHTTPResponse(w, api.AppendResponse{
        RespStatus: api.StatusDuplicateRequest,
      })
    } else {
      kvs.sendHTTPResponse(w, api.AppendResponse{
        RespStatus: api.StatusOK,
        KeyFound:   commitCmd.ResultFound,
        PrevValue:  commitCmd.ResultValue,
      })
    }
  } else {
    kvs.sendHTTPResponse(w, api.AppendResponse{RespStatus: api.StatusFailedCommit})
  }
case <-req.Context().Done():
  return
}

For duplicates the service returns a special API status: api.StatusDuplicateRequest. Our client treats this as an error and surfaces it to the user. As an exercise, try changing it so the return value from duplicates is normal (success). The challenge here is to record - for each request - what the result is for returning to the client (e.g. the previous value of a key in case of PUT or APPEND).

Revisiting our consistency guarantees

In Part 4, we've discussed the consistency guarantees of our KV service in detail, and concluded that it's strict serializable, which is the strongest consistency guarantee for distributed systems.

However, when adding the client module we've also noted that - due to the client retry problem - the whole system is no longer linearizable (and hence no longer strict serializable). Linearizability extended to the client is known to be tricky; this isn't surprising - after all, the client is yet another network-connected component in the system, with inherently unreliable communication to the service.

With de-duplication, our entire system is strict-serializable again. Even if a client re-sends a command that was already committed to the Raft log, this retried command won't be committed a second time due to the de-duplication logic. Users will not observe non-linearizable behavior.

Exactly-once delivery

The discussion around delivery semantics can get quite heated - people have strong feelings about a topic in which academic rigor is often insufficiently applied. Still, I'll take the chance of discussing these semantics in the context of the Raft-based KV DB we've built through this series of posts.

Consider a case in which we have no client retries. A client submits a user's request just once to the underlying Raft service. Raft guarantees that if the system is sufficiently connected (e.g. there are enough live and connected peer to achieve consensus), the command will be applied to the Raft log once. The Raft protocol is both theoretically and practically proven at this point, so let's take this as an axiom. So we can say that without client retries, a command is delivered at most once. It can fail to be delivered (if there's a persistent network partition preventing the Raft cluster from having enough live peers for consensus), but it will not be delivered more than once.

For some applications, at most once delivery is a sufficient guarantee. Think about some sort of distributed logging or telemetry, for example. This isn't the case for the kind of KV DB we're trying to build, though, because it's intended to serve as a rock-solid basis for other distributed applications.

Next we've added client retries; this is essential in light of the imperfect physical world in which our code operates. With client retries, assuming the HW and network is working "in the long term" (e.g. network partitions get fixed within some reasonable time, and crashing servers get restarted or replaced), we get at least once semantics. The client will just keep retrying until it gets notified that the command was applied. However, as we've seen at the beginning of this post, this also means duplicate delivery when some failure scenarios occur.

The goal of adding de-duplication is to move our system to exactly-once delivery. Exactly-once delivery is a highly debated topic, but with some reasonable real-world assumptions, it can be achieved. Consider our implementation, for example (with de-duplication of retries). As long as the network and HW are not permanently broken, a command will either be applied exactly once to the DB, or the client will be notified of an error.

Here's a post by Ron Garret where he provides a useful framework to think about this. The post is long but worth reading; here's a quote I liked:

If you can get at-least-once delivery, you can build exactly-once on top of that

I believe this Raft series has demonstrated how this is done. Another interesting read is this post by the Kafka developers where they discuss how exactly-once semantics were added to Kafka a few years ago. If you read it you'll see it's pretty much the same technique - client retries with de-duplication of commands in the distributed log.

Finally, in Designing Data-Intensive Applications, Martin Kleppman touches upon this topic in several places; for example, in chapter 12 he discusses exactly-once execution of operations and the engineering required to make it possible.

Conclusion

This concludes our series of posts on implementing a full Raft consensus module in Go, and building a strictly serializable KV DB on top of it.

For any questions or comments about these posts or the code, please send me an email or open an issue on GitHub.

Appendix 1: Retries with PUT and linearizability

In this post we've added an APPEND operation to demonstrate the issues that arise with retries and duplication. But APPEND differs significantly from the original operations we've discussed in part 4: it's not idempotent. Applying APPEND more than once creates an invalid value, which isn't true for PUT - if we had retried PUT multiple times, the only outcome would be writing the result twice.

While this seems logical on the surface, it turns out that even without operations like APPEND, the system isn't linearizable if retries without de-duplication are allowed. Here's a diagram (following the style of my post on linarizability) that demonstrates the issue:

This diagram describes a single "register", let's assume the key "foo". Let's also assume that before any writes, the default value of all keys is 0. Here's what happens:

Client A issues a PUT('foo',1) request. The request gets committed by the leader, which crashes before a reply is made to the client. The client will continue periodically retrying.
In the meantime, a new leader is elected. Client C reads the value of the register and gets a successful response (since the PUT of 1 was committed).
Client B uses the new leader to commit a new PUT, with value 2.
Client C reads the value of 2 from the register.
Client A's retry reaches the new leader, which commits another instance of PUT('foo',1), overwriting the value 2. Now when client C reads the register again, it gets 1 again.

This sequence isn't linearizable; as far as the clients are concerned, since operation (2) retrieved the value 1, it means (1) finished before (2); the result of (4) also means that (3) happened after (1). This makes (5) impossible in a linearizable system.

This is a known failure scenario in distributed systems; it's called lost update. In our example, the PUT(2) operation is essentially lost outside a brief window just following it. The retry of PUT(1) overwrites it.

Appendix 2: lost updates in etcd with client retries

To demonstrate that the issue discussed in this post isn't purely academic, here's a real world example.

etcd is an industrial-strength key value DB based on Raft. It's used extensively inside k8s and other projects. While the etcd service itself is strict serializable, some of its client libraries turned out to have the exact retry problem described in this post.

Here's a Jepsen analysis of jetcd, a Java-based client that automatically retries on failures. The analysis concludes that this mechanism results in loss of linearizability, and recommends to disable it. There's also a lengthy discussion in a GitHub issue with the etcd developers about this. And another issue has interesting information as well.

The Jepsen analysis has a great quote which I want to repost here, because it's so relevant to our discussion (and Appendix 1 in particular):

It is easy to assume that set(x, 5) is idempotent because applying it twice in a row still produces the state x = 5. However, this operation is not longer idempotent if its executions are interleaved with other writes—then, it leads to lost update.

Interestingly, etcd doesn't support operations like APPEND at all. It can be emulated with transactions, however, since etcd's data store is versioned. These features also allow one to be more careful around failures when non-linearizable behavior isn't acceptable. For example, we can perform writes like this:

1. Read revision --> $rev
2. TXN
      if mod_revision(k) == $rev
      PUT(k, v)

This will only assign store(k)=v if there were no changes to the DB in the meantime. The failure scenario shown in Appendix 1 cannot happen, because each successful write increments the revision of the store. If unsuccessful, this sequence of operations can be retried safely (its only problem is with liveness, not safety).

[1]	For this reason it's important to store the ID of the command in the Raft log along with the command itself - we need this de-duplication to work across peers.

[2]	If you're concerned that a client may send more than 9.2 quintillion requests in its lifetime, this is easy to change to an arbitrarily large number using `math/big.Int`.

[3]	And we also reuse it to communicate results back from the updater goroutine to request handlers - this is why `IsDuplicate` is there. A simple refactoring exercise could be to use a separate data structure for this purpose.

[4]

This assumes the total number of distinct clients this service is dealing with is not overly large; in realistic systems, this is a reasonable assumption. In extreme cases, if we foresee having to deal with an unbounded number of clients, some sort of "garbage collection" scheme should be maintained (we can "forget" a client after some timeout of not hearing from it).

Implementing Raft: Part 4 - Key/Value Database

2024-10-10T19:50:00-07:00

This is Part 4 in a series of posts describing the Raft distributed consensus algorithm and its complete implementation in Go. Here is a list of posts in the series:

In this part, we're going to use our Raft module to implement a simple but realistic application - a replicated key / value database with strong consistency semantics. All the code for this part is located in this directory.

Key / value database as a state machine

First of all, what's a key / value database (KV DB)? Think of it as a Go map, or as an extremely simple version of NoSQL databases like Redis or CouchDB. The basic operations our KV DB supports are:

PUT(k,v): assign value v to key k
GET(k): retrieve the value associated with key k
CAS(k, cmp, v): atomic compare-and-swap. First, it reads curV - the current value associated with key k. If curV==cmp, assigns value v to k instead; otherwise, it's a no-op. In any case, curV is returned.

For example, suppose the commands in some Raft log are (in order from left to right):

PUT("x","2")  PUT("y","3")  PUT("x","4")  PUT("z","5")  CAS("x","4","8")  CAS("z","4","9")

Applied to an empty DB, this log will result in these keys / values:

x=8
y=3
z=5

System diagram

In this part we're going to build a complete KV DB system - including the service and a client library:

The diagram presents a cluster with 3 replicas [1]. Each replica is a KV DB service.
A KV service contains a Raft Consensus Module (the diagram doesn't show the log, assuming it's just part of the CM), and a data store module that implements the actual database.
The Raft CM of each replica is connected to the others via RPCs - these are the Raft protocol RPCs discussed extensively in previous parts.
The KV service presents a REST API to the external world; clients can send HTTP commands to the service and get results.
"KV Client" is a client library with a convenient API that encapsulates the HTTP interactions with KV services. This is also part of our demo, and we'll discuss it later in the post.

KV service architecture

The KV service consists of several key components:

An instance of a Raft server; as described back in Part 1, a Raft Server wraps a consensus module with some RPC scaffolding. In this part we reuse our final Raft server code from Part 3, without any modifications.
An underlying "data store". For our demonstration, a simple Go map will do; this is implemented in kvservice/datastore.go. This data store implements the Get, Put and CAS commands described earlier. All keys and values are Go strings (naturally, anything can be encoded in a string value).
An HTTP server for the REST API of the service exposed to the external world.

Commands

If you recall from Part 2, we submit new commands to the Raft cluster with the ConsensusModule.Submit method. A Command is an arbitrary any value; whenever the Raft cluster reaches consensus on a log entry, it sends a "commit entry" with this command on the commit channel. Commands are application-specific, and since we're working on a concrete application now, it's time to define our command for the KV service:

// Command is the concrete command type KVService submits to the Raft log to
// manage its state machine. It's also used to carry the results of the command
// after it's applied to the state machine. These are the supported commands:
//
// CommandGet: queries a key's value
//
// * Key is the key to get, Value is ignored
// * CompareValue is ignored
// * ResultFound is true iff Key was found in the store
// * ResultValue is the value, if Key was found in the store
//
// CommandPut: assigns value to the key
//
// * Key,Value are the pair to assign (store[key]=value)
// * CompareValue is ignored
// * ResultFound is true iff Key was previously found in the store
// * ResultValue is the old value of Key, if it was previously found
//
// CommandCAS: atomic compare-and-swap, performs:
//
//    if Store[Key] == CompareValue {
//      Store[Key] = Value
//    } else {
//      nop
//    }
//
// * Key is the key this command acts on
// * CompareValue is the previous value the command compares to
// * Value is the new value the command assigns
// * ResultFound is true iff Key was previously found in the store
// * ResultValue is the old value of Key, if it was previously found
type Command struct {
  Kind CommandKind

  Key, Value string

  CompareValue string

  ResultValue string
  ResultFound bool

  // id is the Raft ID of the server submitting this command.
  Id int
}

type CommandKind int

const (
  CommandInvalid CommandKind = iota
  CommandGet
  CommandPut
  CommandCAS
)

For simplicity, I chose to include fields for several commands in the same struct instead of using an algebraic data type here.

One important thing to note is that the service's Raft cluster ID is part of the command; it will soon become clear why this is needed.

Life of a PUT request to the service

Before we dive deep into the code, let's examine the journey a successful PUT request makes through the system:

A client sends a PUT("k", "v") request to a service, via HTTP. Let's assume it reaches the service which is currently the Raft cluster leader (we'll discuss what happens if it reaches a follower later on).
The service's HTTP handler receives the request, constructs a Command of kind CommandPut representing it and submits it to its Raft CM.
1. At this point, the HTTP handler waits; it can't reply to the client until it knows that the command was properly replicated to the Raft cluster and committed by the CM.
2. Once the command it submitted appears on the commit channel, the HTTP handler can return a success status to the client.
Meanwhile, a process in the service watches its commit channel for new commands that reached consensus by the cluster, and updates the underlying data store.
At the same time, the other services in the cluster - the followers - are also watching their commit channels and update their own replicas of the data store with the new PUT command.

Note that steps 2.2 and 3 happen concurrently. One process (in the sense of CSP) handles a client request, while another process takes care to execute commands arriving on the commit channel. In fact, there's more concurrency here than meets the eye. Our service can handle multiple concurrent requests, each with its own command - and it should all just work. This kind of concurrency is natural in Go - and now it's time to see how it works.

KV service code walk-through

All the code described in this section is located in kvservice/kvservice.go. Here's the struct defining the service:

type KVService struct {
  sync.Mutex

  // id is the service ID in a Raft cluster.
  id int

  // rs is the Raft server that contains a CM
  rs *raft.Server

  // commitChan is the commit channel passed to the Raft server; when commands
  // are committed, they're sent on this channel.
  commitChan chan raft.CommitEntry

  // commitSubs are the commit subscriptions currently active in this service.
  // See the createCommitSubscription method for more details.
  commitSubs map[int]chan Command

  // ds is the underlying data store implementing the KV DB.
  ds *DataStore

  // srv is the HTTP server exposed by the service to the external world.
  srv *http.Server
}

Don't worry about understanding exactly what each field means right now; note the correlation to the descriptions in "KV service architecture", though. A service holds a Raft server, a datastore, and an HTTP server. Other entities, like the commit channel, should be familiar by now.

A new service is created with this constructor:

// New creates a new KVService
//
//   - id: this service's ID within its Raft cluster
//   - peerIds: the IDs of the other Raft peers in the cluster
//   - storage: a raft.Storage implementation the service can use for
//     durable storage to persist its state.
//   - readyChan: notification channel that has to be closed when the Raft
//     cluster is ready (all peers are up and connected to each other).
func New(id int, peerIds []int, storage raft.Storage, readyChan <-chan any) *KVService {
  gob.Register(Command{})
  commitChan := make(chan Command)

  // raft.Server handles the Raft RPCs in the cluster; after Serve is called,
  // it's ready to accept RPC connections from peers.
  rs := raft.NewServer(id, peerIds, storage, readyChan, commitChan)
  rs.Serve()
  kvs := &KVService{
    id:         id,
    rs:         rs,
    commitChan: commitChan,
    ds:         NewDataStore(),
    commitSubs: make(map[int]chan Command),
  }

  kvs.runUpdater()
  return kvs
}

We'll get back to what runUpdater is a little later; for now, let's look at how the HTTP server is launched:

// ServeHTTP starts serving the KV REST API on the given TCP port. This
// function does not block; it fires up the HTTP server and returns. To properly
// shut down the server, call the Shutdown method.
func (kvs *KVService) ServeHTTP(port int) {
  if kvs.srv != nil {
    panic("ServeHTTP called with existing server")
  }
  mux := http.NewServeMux()
  mux.HandleFunc("POST /get/", kvs.handleGet)
  mux.HandleFunc("POST /put/", kvs.handlePut)
  mux.HandleFunc("POST /cas/", kvs.handleCAS)

  kvs.srv = &http.Server{
    Addr:    fmt.Sprintf(":%d", port),
    Handler: mux,
  }

  go func() {
    kvs.kvlog("serving HTTP on %s", kvs.srv.Addr)
    if err := kvs.srv.ListenAndServe(); err != http.ErrServerClosed {
      log.Fatal(err)
    }
    kvs.srv = nil
  }()
}

This should be familiar if you've written Go HTTP servers before. Listening is done in a goroutine to enable clean shutdown of the HTTP server specifically and the whole service in general; check out the Shutdown method for more details.

In the previous section, I mentioned that multiple HTTP requests can be handled concurrently; this is just the nature of the standard Go HTTP server. Here we see the handleXXX handlers registered with the server; each handler is invoked in a separate goroutine, and our code has to account for this. To understand what this means in practice, let's look at the updater goroutine.

// runUpdater runs the "updater" goroutine that reads the commit channel
// from Raft and updates the data store; this is the Replicated State Machine
// part of distributed consensus!
// It also notifies subscribers (registered with createCommitSubscription).
func (kvs *KVService) runUpdater() {
  go func() {
    for entry := range kvs.commitChan {
      cmd := entry.Command.(Command)

      switch cmd.Kind {
      case CommandGet:
        cmd.ResultValue, cmd.ResultFound = kvs.ds.Get(cmd.Key)
      case CommandPut:
        cmd.ResultValue, cmd.ResultFound = kvs.ds.Put(cmd.Key, cmd.Value)
      case CommandCAS:
        cmd.ResultValue, cmd.ResultFound = kvs.ds.CAS(cmd.Key, cmd.CompareValue, cmd.Value)
      default:
        panic(fmt.Errorf("unexpected command %v", cmd))
      }

      // Forward this entry to the subscriber interested in its index, and
      // close the subscription - it's single-use.
      if sub := kvs.popCommitSubscription(entry.Index); sub != nil {
        sub <- cmd
        close(sub)
      }
    }
  }()
}

The updater goroutine is responsible for implementing step (3) described in the "Life of..." section. It watches the commit channel for new committed commands, applies these commands to the datastore and then notifies "subscribers" about it. The first two tasks is what we'd expect from an implementation of a Raft-based replicated state machine; the last task needs some elaboration.

Recall step 2.1 from the "Life of..." section; once an HTTP handler submits a command to the Raft cluster, it has to wait and see if this command was properly committed. The way we implement it is:

The handler submits a command to the Raft CM, and keeps note of the log index the command is placed in.
The handler than registers a "subscription" with the updater, telling it: "hey, if you see a command submitted for this index, let me know". The subscription is implemented with a channel.
The handler can then wait on the channel.

Here's the code of handlePut, demonstrating this in action:

func (kvs *KVService) handlePut(w http.ResponseWriter, req *http.Request) {
  pr := &api.PutRequest{}
  if err := readRequestJSON(req, pr); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }
  kvs.kvlog("HTTP PUT %v", pr)

  // Submit a command into the Raft server; this is the state change in the
  // replicated state machine built on top of the Raft log.
  cmd := Command{
    Kind:  CommandPut,
    Key:   pr.Key,
    Value: pr.Value,
    Id:    kvs.id,
  }
  logIndex := kvs.rs.Submit(cmd)
  // If we're not the Raft leader, send an appropriate status
  if logIndex < 0 {
    renderJSON(w, api.PutResponse{RespStatus: api.StatusNotLeader})
    return
  }

  // Subscribe for a commit update for our log index. Then wait for it to
  // be delivered.
  sub := kvs.createCommitSubscription(logIndex)

  // Wait on the sub channel: the updater will deliver a value when the Raft
  // log has a commit at logIndex. To ensure clean shutdown of the service,
  // also select on the request context - if the request is canceled, this
  // handler aborts without sending data back to the client.
  select {
  case commitCmd := <-sub:
    // If this is our command, all is good! If it's some other server's command,
    // this means we lost leadership at some point and should return an error
    // to the client.
    if commitCmd.Id == kvs.id {
      renderJSON(w, api.PutResponse{
        RespStatus: api.StatusOK,
        KeyFound:   commitCmd.ResultFound,
        PrevValue:  commitCmd.ResultValue,
      })
    } else {
      renderJSON(w, api.PutResponse{RespStatus: api.StatusFailedCommit})
    }
  case <-req.Context().Done():
    return
  }
}

The code is well-commented, but I want to specifically call out a few important points:

When kvs.rs.Submit is called with the command, it returns -1 if the current Raft CM is not the leader. In this case, we return a special status to the client - "I'm not the leader" - and abort the handler. We'll see what the client does about this further down in the post.

For a leader, Submit returns the log index at which the command was submitted. This is the index used to subscribe to notifications from the commit channel.
The handler waits on a receive on this channel. This can be canceled if the HTTP request is canceled by the client (e.g. timeout); otherwise, we just wait. In practice, with the optimizations in Part 3, it takes just a handful of milliseconds to fully commit new commands in a functioning Raft cluster. In case of problems (disconnections, crashes etc.) this may take longer, but our application prioritizes consistency over availability (see Part 0 on fault tolerance in Raft and the CAP theorem).
When notified that a commit was made for this log index, there's still an important safety check to make! Is it actually our command that was committed there? This is what the id field on the command is for.

Consider the following case: peer A is the leader, and a client submits a command. A places it in log index 42, but gets disconnected before it manages to tell followers about it. After a while, C becomes the new leader; C is unaware that A placed something in its log at index 42. Therefore, when C receives a new command from another client, it commits it at index 42 (since this is still the "next index for entries" for all connected cluster members). At some point later, A gets reconnected to the cluster, becomes a follower (since its term is out of date), and sees the commit from C at index 42. At this point it realizes that it failed to commit its own command (because the ID doesn't match), and replies with a "failed commit" status to the client.

I'll leave figuring out the mechanics of channel subscriptions to you as an exercise. Just read the createCommitSubscription and popCommitSubscription methods - they're fairly straightforward.

Consistency guarantees

I wrote in detail about linearizable semantics recently. Our KV service is linearizable based on that definition, due to the nature of Raft consensus. An operation only becomes visible to clients after it's committed; and it's committed by cluster consensus, at a "moment in time" relative to other operations in the Raft log.

Moreover, it's also serializable for transactions like CAS: these are performed by a single service (the leader) atomically, so clients can never observe the results of sub-operations in isolation.

By being both linearizable and serializable, our service is strict serializable, which is the strongest consistency guarantee for distributed systems.

As discussed before, this strong consistency comes at the expense of availability in the face of network partitions (as it must, due to the CAP theorem limits). It's a "CP" system; the following diagram is from Wikipedia:

What are such services good for? Though it can serve as a NoSQL database, it won't be very performant - every operation has to reach consensus among multiple peers before being considered "done". Instead, such strict serializable services are used as the very bottom layer of large distributed systems. For example, it can be used to coordinate distributed locks, elect leaders (these are fairly easy to build on top of our CAS primitive) or store some critical low-volume configuration data for a complex system.

Plumbing read-only operations through the Raft log

You'll note that all the commands our KV service supports - PUT, GET and CAS - are implemented fairly consistently and follow the sequence described in the "Life of..." section. This raises an important question: is this really necessary for the read-only GET operations? After all, they don't really change the state machine, so why add them as Raft log commands?

While it's true that a stray GET command won't harm the integrity of the internal data store, it may result in stale reads or other events inconsistent with the linearizable semantics of our service.

To see why, let's work by contradiction; assume we don't plumb GET through the Raft log, but instead let leaders immediately reply to GET requests based on their local datastore. Here's what can happen:

The KV DB has the key-value pair k=v.
A used to be a leader, but got disconnected from its peers; after a suitable election timeout, C was elected as the new leader. A still thinks it's the leader, however.
At some point, a client contacts C and submits PUT(k,v2). C successfully replicates this command to the remaining connected peers.
A bit later, another client sends GET(K) to C and gets the correct response v2.
Then, a different client sends GET(k) to A (perhaps the client remembered that the previous time it contacted the service, A was the leader [2]). Since A still thinks it's the leader, it will happily reply with the value v to the client's request.

This sequence of events breaks the linearizability guarantees of our service! The read GET(K) --> v is stale, since another client already read the value as v2. There is no single-threaded history in which this sequence of events is possible.

This problem is explicitly called out in Section 8 of the Raft paper. The canonical solution is what our service is doing: plumb all commands - even the read-only ones - through the Raft log [3]. A service won't respond to a client's request unless it was able to successfully commit this command to the Raft log.

Since we plumb GET commands through the Raft log, in our example the problem in the last step couldn't happen, because A would not respond to its client while disconnected from the cluster. Instead, it would have to wait to be reconnected, and at that point would discover that it's no longer the leader. The client would then ask the real leader and get the right response. However, even if due to additional disconnections or crashes A resumed leadership, it would have to process the PUT(k,v2) before processing the client's GET(k), since the state machine is updated in log order.

KV client

Now it's time to discuss the final piece of our system - the KV client library. Since the KV service API is just REST, we don't necessarily need a client library - we could just use curl calls or any other way to generate HTTP requests to interact with it. However, a convenient, idiomatic client library goes a long way in improving the quality of life of users - and it will be particularly useful in this case because it encodes some essential logic - finding and keeping track of the cluster leader.

So far, everything in our system has been replicated by N, which is the Raft cluster size (typically 3 or 5). The client is a single entity - just user code that wants to use the KV service. All the client code is in kvclient/kvclient.go; let's walk through how a single request works, starting with the type and constructor:

type KVClient struct {
  addrs []string

  // assumedLeader is the index (in addrs) of the service we assume is the
  // current leader. It is zero-initialized by default, without loss of
  // generality.
  assumedLeader int

  clientID int32
}

// New creates a new KVClient. serviceAddrs is the addresses (each a string
// with the format "host:port") of the services in the KVService cluster the
// client will contact.
func New(serviceAddrs []string) *KVClient {
  return &KVClient{
    addrs:         serviceAddrs,
    assumedLeader: 0,
    clientID:      clientCount.Add(1),
  }
}

// clientCount is used internally for debugging
var clientCount atomic.Int32

To create a client, we have to provide it with a list of addresses for the KV services that constitute a cluster; before the client sends its first request, the services should be launched and listening on these addresses.

All client requests follow the same steps; let's use Put as an example:

// Put the key=value pair into the store. Returns an error, or
// (prevValue, keyFound, false), where keyFound specifies whether the key was
// found in the store prior to this command, and prevValue is its previous
// value if it was found.
func (c *KVClient) Put(ctx context.Context, key string, value string) (string, bool, error) {
  putReq := api.PutRequest{
    Key:   key,
    Value: value,
  }
  var putResp api.PutResponse
  err := c.send(ctx, "put", putReq, &putResp)
  return putResp.PrevValue, putResp.KeyFound, err
}

Types like PutRequest and PutResponse are defined in api/api.go (you may have noticed them in the service code as well); they're trivial, so I won't spend more time on them.

All the client logic is encapsulated in the send method:

func (c *KVClient) send(ctx context.Context, route string, req any, resp api.Response) error {
  // This loop rotates through the list of service addresses until we get
  // a response that indicates we've found the leader of the cluster. It
  // starts at c.assumedLeader
FindLeader:
  for {
    // There's a two-level context tree here: we have the user context - ctx,
    // and we create our own context to impose a timeout on each request to
    // the service. If our timeout expires, we move on to try the next service.
    // In the meantime, we have to keep an eye on the user context - if that's
    // canceled at any time (due to timeout, explicit cancellation, etc), we
    // bail out.
    retryCtx, retryCtxCancel := context.WithTimeout(ctx, 50*time.Millisecond)
    path := fmt.Sprintf("http://%s/%s/", c.addrs[c.assumedLeader], route)

    c.clientlog("sending %#v to %v", req, path)
    if err := sendJSONRequest(retryCtx, path, req, resp); err != nil {
      // Since the contexts are nested, the order of testing here matters.
      // We have to check the parent context first - if it's done, it means
      // we have to return.
      if contextDone(ctx) {
        c.clientlog("parent context done; bailing out")
        retryCtxCancel()
        return err
      } else if contextDeadlineExceeded(retryCtx) {
        // If the parent context is not done, but our retry context is done,
        // it's time to retry a different service.
        c.clientlog("timed out: will try next address")
        c.assumedLeader = (c.assumedLeader + 1) % len(c.addrs)
        retryCtxCancel()
        continue FindLeader
      }
      retryCtxCancel()
      return err
    }
    c.clientlog("received response %#v", resp)

    // No context/timeout on this request - we've actually received a response.
    switch resp.Status() {
    case api.StatusNotLeader:
      c.clientlog("not leader: will try next address")
      c.assumedLeader = (c.assumedLeader + 1) % len(c.addrs)
      retryCtxCancel()
      continue FindLeader
    case api.StatusOK:
      retryCtxCancel()
      return nil
    case api.StatusFailedCommit:
      retryCtxCancel()
      return fmt.Errorf("commit failed; please retry")
    default:
      panic("unreachable")
    }
  }
}

There's some context subtlety going on here - hopefully the comments make that clear enough.

The client keeps track of the last service it saw that accepted a command as a leader. When asked to send a new command to the service, this is the service it starts from. If its request to the assumed leader times out, or that service says it's no longer the leader, the client retries to the next service in the cluster.

During normal operation, the leader will typically be stable, each client will quickly discover who it is and from that point on will address the leader directly. When there's a cluster disruption, the client will spend a bit of time looking for the leader - but this can be optimized if needed [4].

If a client can't find a leader, it will just keep trying; since we use the Go context idiom, this can always be controlled by the user - by imposing a timeout on client operations, or canceling them for other reasons.

Future work

The KV service presented in this post provides strong consistency guarantees, as discussed. However, keeping systems linearizable all the way through the client is notoriously tricky, and the simple client we presented in this post is not immune to issues.

The problem is with its retry logic; when a client sends a PUT command to a leader and the request times out, what is the right thing to do? Our client just retries, looking for a different leader. Is this the right approach?

Not necessarily! Consider what happens if the leader committed the command, but crashed before responding to the client. If the client now retries, the command may end up duplicated in the log. While it may seem like this shouldn't be a problem because PUT is idempotent [5], it can in fact cause non-linearizable behavior to be observed, if some other client managed to PUT another value for the same key in-between the replies.

This isn't a trivial problem; in fact, it's also mentioned in section 8 of the Raft paper. We'll spend the next part in the series discussing this problem in detail, presenting one potential solution and talking about how real-world distributed KV services deal with it.

[1]	For the terms used in this description, refer to Part 0.

[2]	This is exactly how our client implementation works, as we'll see soon.

[3]	The paper also discusses some ideas for optimizations of this process. Since this optimizes the uncommon path (when crashes and disconnections disrupt the normal operation of the Raft cluster), I leave this out of my implementation.

[4] Here's an exercise: the AppendEntries RPC sent by leaders to followers contains a "leader ID" field; so followers know who the current leader is. We already have it in our Raft implementation; try to plumb this information all the way through to the client. When a follower sends a "I'm not a leader" response to the client, it can include the ID of the service it thinks is the current leader; this can reduce the search time somewhat.

[5]	Applying `PUT(k1, v1)` right after another `PUT(k1,v1)` doesn't affect the correctness of the DB.

Linearizability in distributed systems

2024-10-07T19:16:00-07:00

Linearizability is a strong consistency model in concurrent and distributed systems. From the paper introducing it [1]:

Linearizability provides the illusion that each operation applied by concurrent processes takes effect instantaneously at some point between its invocation and its response.

On first reading (and probably on the second and third...) this sounds a bit abstract, but it really is all there is to it. A slightly different way to think about it is - a linearizable system appears as if there's only one copy of data in existence, and all client operations apply to this data atomically. This post dives deeper into what this means in practice.

Registers

Linearizability is a single-object consistency model (see the "Linearizability vs. Serializability" section below for more on this). It's common in distributed systems literature to talk about a register - a single key-value pair, for example, stored in some distributed database. When clients write and read this register concurrently, we can analyze the history of operations and their results and determine if the system maintains linearizability.

Basic example

The following diagram describes a sequence of register reads and writes by three different clients; some of these operations are done concurrently. Time flows from left to right, and a colored rectangle denotes an operation; its left edge is the operation's start, and its right edge the operation's completion [2].

Here are the events, each with its own number in the yellow bubble:

Client A reads the register and gets the value of 0. The read itself happened at some point in time in the database, denoted on the timeline in the very bottom of the diagram.
Client B reads the value 0. Note that this read operation is partially concurrent with the write operation (3); concurrent operations can execute in any order, but here (2) happened to be executed before (3) (we know this because the value 0 was read, not 1).
Client C writes 1 into the register.
Client A reads 1 from the register. This read is also concurrent with the write, and thus could end up with any result, but since the result in this timeline is 1, we know it happened after (3).
Client B reads 1 from the register.

This sequence of events is valid in a linearizable system, because we can construct a serial history of events (the bottom timeline) that's consistent with our results. Each event occurs instantaneously at some point between the start and finish of the client request.

Compare this to the following sequence, which is not valid:

This sequence is similar to the first one, with one difference: B's read in (5) results in 0. Since (5) is concurrent with (3), when seen in isolation this isn't unreasonable. However, since in (4), client A already observed the value 1 in the register and (4) happens before (5), this sequence is invalid in a linearizable system. We can imagine systems with weaker consistency guarantees producing this history, but such systems are not linearizable.

Another way to look at it is examine the timeline in the bottom of the diagram. Notice that (5) reads 0, after (3) happened. We just can't find a way to arrange this history so it looks sequential - therefore, it's inconsistent with linearizability.

A more subtle example

Here's a more subtle example, taken from the linearizability paper:

This sequence of events is invalid for a linearizable system! To understand why, let's follow the timeline at the bottom of the diagram.

(3) Client B's write of 1 executes before (2) client A's read, because A reads 1 from the register. If the read at (2) happened before the write at (3), it (the read) would result in 0, not 1.

Event (4) has to happen after event (2), since it starts after (2) ends. But we've just reasoned that (2) happens after (3); therefore, (4) happens after (3) - even though these two writes are concurrent, their order is imposed by observing other events.

Finally, since we've just proven that (4) happens after (3), the value in the register at the conclusion of (3) is 0, not 1; therefore, the read of 1 in (5) is invalid. This system cannot be linearizable. As before, you can try to arrange the events in the bottom of the diagram into some sequential order - this attempt will fail, because no consistent sequential order can account for the observed events.

A formal definition

I personally found the formal definition of linearizability in the Herlihy & Wang paper somewhat obscured by attention given to potentially unfinished operations. If we assume that every operation has a start and an end, it's easier to restate the formal definition as follows.

An operation e has the timestamps start(e) and end(e); these are the left and right boundaries of the rectangles in the diagrams shown above.

A history H exists with a strong partial order <_H on operations [3]: e_0 <_H e_1 if end(e_0) precedes start(e_1) in H. Operations unrelated by <_H are said to be concurrent in H. In our diagrams, H represents the observed history (the part of the diagram with the overlapping rectangles). The formal definition captures what it means for us to know that some operations precede others, while other operations are concurrent.

For example, in our last diagram above if H is the history shown, then e_3 <_H e_5, but the pair e_3,e_4 is not in the relation <_H, since these operations are concurrent.

If H is a sequential history, then <_H is a total order. It means there are no concurrent operations.

Now it's time for the definition of linearizability. H is linearizable if:

H is equivalent to some sequential history S
<_H \subseteq <_S

The second item requires a bit of elaboration: recall that <_H and <_S are relations. <_H being a subset of <_S means that the partial order of operations in the real-time history H is preserved in the linearization.

We then call S the linearization of H. In our diagrams, S is the bottom line where operations are shown on the server happening immediately; they are still represented by start(e) and end(e) in the history (we can just assume start(e) and end(e) are infinitesimally close in time, since the DB applies operations atomically).

Linearizability vs. Serializability

Linearizability is often confused with serializability - another consistency model. The two are fundamentally different, though:

Serializability is a multi-object property useful to describe transactions that consist of multiple operations that may potentially touch multiple objects; informally, it means that transactions happen atomically, and their sub-operations cannot be observed in isolation or intermix.
Linearizability is a single-object property, talking about the observed effects on a single register, as this post demonstrates.

For a great taxonomy of consistency models, see this page from Jepsen.

Additional resources

Kyle Kingsbury - on his blog and through his company Jepsen - has a wealth of great resources on the subject of linearizability and other consistency models. Some examples:

The taxonomy, as mentioned above, and the related blog post
Knossos, a linearizability checker: blog post and project page
Jepsen's analysis of etcd has an interesting practical discussion of linearizability in a real-world system

[1]	Herlihy, Maurice P.; Wing, Jeannette M. (1990). "Linearizability: A Correctness Condition for Concurrent Objects". ACM Transactions on Programming Languages and Systems.

[2]	The operation itself happens instantaneously on the server at some moment within the rectangle's boundaries, but we don't know exactly when due to network delays.

[3]	For a refresher on the math used here (relations, orders) see this post.

Reading Google Sheets from a Go program

2024-05-31T18:07:00-07:00

I recently needed to process some data from a Google Sheet in a Go program, and was looking for the most straightforward way to do so on my local machine. This post lists some approaches that I found to work, with full source code.

To access the Sheets API, you'll need a GCP project, and would typically have the gcloud command-line tool installed. To enable the sheets API for your project, run:

$ gcloud services enable sheets.googleapis.com --project=<PROJECT-NAME>

If you want to list which APIs are already enabled, you can do:

$ gcloud services list --enabled --project=<PROJECT-NAME>

The simplest approach I found to work was using a service account. This post demonstrates this approach, as well as a (slightly) more involved approach that uses Oauth 2.0

Service account

A service account on GCP can be thought of as a virtual account, along with its own email address, attached to a project. These accounts have their own auth, permissions, etc. This is very useful for running on a VM - you typically don't want the VM to be logged in with your primary Google account, and this service account can be specific to a given VM (or a group thereof).

Start by creating a new service account on this page. Once created, select Manage Keys in the Actions menu, and add a new key. This will download a private key to your machine; keep it safe! The following program expects this key file to be provided with the -keyfile flag:

package main

import (
  "context"
  "flag"
  "fmt"
  "io/ioutil"
  "log"

  "golang.org/x/oauth2/google"
  "google.golang.org/api/option"
  "google.golang.org/api/sheets/v4"
)

func main() {
  keyFilePath := flag.String("keyfile", "", "path to the credentials file")
  flag.Parse()

  ctx := context.Background()
  credentials, err := ioutil.ReadFile(*keyFilePath)
  if err != nil {
    log.Fatal("unable to read key file:", err)
  }

  scopes := []string{
    "https://www.googleapis.com/auth/spreadsheets.readonly",
  }
  config, err := google.JWTConfigFromJSON(credentials, scopes...)
  if err != nil {
    log.Fatal("unable to create JWT configuration:", err)
  }

  srv, err := sheets.NewService(ctx, option.WithHTTPClient(config.Client(ctx)))
  if err != nil {
    log.Fatalf("unable to retrieve sheets service: %v", err)
  }

  // ...

We can specify the requested scopes (permissions) when creating an auth config. Here we're asking for read-only access to the Google Sheets.

Once auth succeeds (sheets.NewService returns w/o an error), we can use the sheets package to read and analyze the sheet; the code below simply prints the document's title and emits all the values from columns A and B in Sheet1.

  docId := "1qsNWsZuw98r9HEl01vwxCO5O1sIsI-fr0bJ4KGVvWsU"
  doc, err := srv.Spreadsheets.Get(docId).Do()
  if err != nil {
    log.Fatalf("unable to retrieve data from document: %v", err)
  }
  fmt.Printf("The title of the doc is: %s\n", doc.Properties.Title)

  val, err := srv.Spreadsheets.Values.Get(docId, "Sheet1!A:B").Do()
  if err != nil {
    log.Fatalf("unable to retrieve range from document: %v", err)
  }

  fmt.Printf("Selected major dimension=%v, range=%v\n", val.MajorDimension, val.Range)
  for _, row := range val.Values {
    fmt.Println(row)
  }
}

Note the docId passed to the sheets package; this is the path segment in your spreadsheet's URL following the /d/. In this example, I'm using a test sheet I've created.

Important: unless your sheet is world-readable, your service account won't be able to access it. Here the account's email comes in handy; you can take it from the service account's GCP IAM page (Details tab), and give this email permissions to the sheet. This way you can have the program processing a private sheet that only you have access to.

OAuth

Another way to achieve what we want is with OAuth. This also requires a bit of setup in your project's GCP console. Follow the Go quickstart docs for that. Our sample assumes you've saved the credentials.json file somewhere locally and will pass it through the -credfile flag. Unlike the quickstart, it handles all the token exchange process automatically without having to ask you to copy a code from a web page. You still have to authenticate the first time you run it, of course.

The full code of the sample is available on GitHub; while the auth part is different, the actual sheets processing code is identical to the service account sample.

For an overview of the OAuth protocol, see my earlier post.

P.S. ADC

Initially, I had trouble accessing the sheet using ADC (Application Default Credentials), but following a HN comment on this post, I was motivated to try again and it worked. I may have mixed up my auth JSON files previously, because the code is identical to what I've originally tried. In any case, the code is available on GitHub along with the other options. Depending on the exact use case, ADC may be simpler than using a service account (though IMHO the service account is a more "reliable" method across machines because its configuration is more explicit - less is happening under the hood).

Sign in with Google in Go

2024-01-13T06:33:00-08:00

This post provides some code samples for implementing a "Sign-in with Google" flow for your web application in Go. For an overview of auth/authz and the OAuth protocol, please refer to my earlier post about Sign-in with GitHub.

Sign-in with Google has existed in one form or another for many years, and the technical approach to it evolved over time. I will start by presenting the currently recommended way - and the one that will feel most familiar to users - and will then mention a slightly more complicated and flexible alternative.

Using Google Identity Service (GIS)

The currently recommended way to implement sign-in with Google is using the Google Identity Service client library. There's a lot of documentation on that page, and it's worth reading through.

Using this approach, we'll get a standard looking button:

When clicked, a popup opens with a standard-looking Google account selection:

There's also an option for "one tap" which surfaces the currently logged in Google user in an in-page popup to click (this also works well on mobile!)

To get started, you'll need to register an application at https://console.cloud.google.com/apis/credentials and obtain a client ID and secret (the secret isn't used for this approach, but is needed for the next one).

We'll use a Google-provided JS client library called GSI which we include in our web page; it implements the actual button and all the client-side handling; here's a snippet from our Go server for this sample:

// This should be taken from https://console.cloud.google.com/apis/credentials
var GoogleClientID = os.Getenv("GOOGLE_CLIENT_ID")

const servingSchema = "http://"
const servingAddress = "localhost:8080"
const callbackPath = "/google/callback"

var rootHtmlTemplate = template.Must(template.New("root").Parse(`
<!DOCTYPE html>
<html>
<body>
    <script src="https://accounts.google.com/gsi/client" async></script>

    <h1>Welcome to this web app!</h1>
    <p>Let's sign in with Google:</p>
    <div
        id="g_id_onload"
        data-client_id="{{.ClientID}}"
        data-login_uri="{{.CallbackUrl}}">
    </div>
    <div
        class="g_id_signin"
        data-type="standard"
        data-theme="filled_blue"
        data-text="sign_in_with"
        data-shape="rectangular"
        data-width="200"
        data-logo_alignment="left">
    </div>
</body>
</html>
`))

func main() {
  if len(GoogleClientID) == 0 {
    log.Fatal("Set GOOGLE_CLIENT_ID env var")
  }

  mux := http.NewServeMux()
  mux.HandleFunc("/", rootHandler)
  mux.HandleFunc(callbackPath, callbackHandler)

  log.Printf("Listening on: %s%s\n", servingSchema, servingAddress)
  log.Panic(http.ListenAndServe(servingAddress, mux))
}

func rootHandler(w http.ResponseWriter, req *http.Request) {
  err := rootHtmlTemplate.Execute(w, map[string]string{
    "CallbackUrl": servingSchema + servingAddress + callbackPath,
    "ClientID":    GoogleClientID,
  })
  if err != nil {
    panic(err)
  }
}

Note the configuration on the <div> element for the button; there's a ton of potential customization there - the docs explain these in detail.

We serve this page on the root ("/") handler of our local web server, and also register a callback URL which the Google auth server will redirect to once it confirms the user's identity [1]. When it does, it sends the logged-in user's information as a JSON Web Token (JWT); our server-side application should verify the validity of the token and then it's ready to accept the signed-in user.

We'll be using the google.golang.org/api/idtoken package for token verification (this comes from the official GCP client library for Go); here's our callback:

func callbackHandler(w http.ResponseWriter, req *http.Request) {
  defer req.Body.Close()

  if err := req.ParseForm(); err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
    return
  }

  // The following steps follow
  // https://developers.google.com/identity/gsi/web/guides/verify-google-id-token
  //
  // 1. Verify the CSRF token, which uses the double-submit-cookie pattern and
  //    is added both as a cookie value and post body.
  token, err := req.Cookie("g_csrf_token")
  if err != nil {
    http.Error(w, "token not found", http.StatusBadRequest)
    return
  }

  bodyToken := req.FormValue("g_csrf_token")
  if token.Value != bodyToken {
    http.Error(w, "token mismatch", http.StatusBadRequest)
  }

  // 2. Verify the ID token, which is returned in the `credential` field.
  //    We use the idtoken package for this. `audience` is our client ID.
  ctx := context.Background()
  validator, err := idtoken.NewValidator(ctx)
  if err != nil {
    panic(err)
  }
  credential := req.FormValue("credential")
  payload, err := validator.Validate(ctx, credential, GoogleClientID)
  if err != nil {
    http.Error(w, err.Error(), http.StatusBadRequest)
  }

  // 3. Once the token's validity is confirmed, we can use the user identifying
  //    information in the Google ID token.
  for k, v := range payload.Claims {
    fmt.Printf("%v: %v\n", k, v)
  }
}

The numbered comments follow the documentation step by step, so it should be easy to decipher. The end result is dumping some information about the user from the token; specifically, the Google email and registered name will be provided. Once you have that, you know who the user is and that they have an actual Google account.

This is it! I like how all the complicated front-end details are handled by Google's own JS library; there's a lot of potential nuance there with one-tap and automatic sign-in, different UI requirements for desktop browsers and mobile, etc. If you're developing a standard web-app, this definitely seems like the way to go.

Using OpenID Connect

OpenID Connect is an authentication protocol built on top of OAuth 2. The Google documentation for integrating it into your applications is here.

This is a more generic approach than using GIS - you're not beholden to use specific JS or HTML elements, but can integrate it into your flow in whatever way you wish - similarly to the GitHub approach outlined in my last post. I won't paste the code here - but it's well documented and follows the official docs. I have two separate samples:

Using the go-oidc package: the full code is here.
Using the gologin package: the full code is here. You may recognize the gologin package from my previous post on GitHub sign-in; indeed, the code delta between the two options is very small!

[1]	Please refer to my description of OAuth to understand the redirection flow. This Google sign-in flow still relies on OAuth underneath.