Eli Bendersky's website - Go

watgo - a WebAssembly Toolkit for Go

2026-04-09T19:28:00-07:00

I'm happy to announce the general availability of watgo - the WebAssembly Toolkit for Go. This project is similar to wabt (C++) or wasm-tools (Rust), but in pure, zero-dependency Go.

watgo comes with a CLI and a Go API to parse WAT (WebAssembly Text), validate it, and encode it into WASM binaries; it also supports decoding WASM from its binary format.

At the center of it all is wasmir - a semantic representation of a WebAssembly module that users can examine (and manipulate). This diagram shows the functionalities provided by watgo:

Parse: a parser from WAT to wasmir
Validate: uses the official WebAssembly validation semantics to check that the module is well formed and safe
Encode: emits wasmir into WASM binary representation
Decode: read WASM binary representation into wasmir

CLI use case

watgo comes with a CLI, which you can install by issuing this command:

go install github.com/eliben/watgo/cmd/watgo@latest

The CLI aims to be compatible with wasm-tools [1], and I've already switched my wasm-wat-samples projects to use it; e.g. a command to parse a WAT file, validate it and encode it into binary format:

watgo parse stack.wat -o stack.wasm

API use case

wasmir semantically represents a WASM module with an API that's easy to work with. Here's an example of using watgo to parse a simple WAT program and do some analysis:

package main

import (
  "fmt"

  "github.com/eliben/watgo"
  "github.com/eliben/watgo/wasmir"
)

const wasmText = `
(module
  (func (export "add") (param i32 i32) (result i32)
    local.get 0
    local.get 1
    i32.add
  )
  (func (param f32 i32) (result i32)
    local.get 1
    i32.const 1
    i32.add
    drop
    i32.const 0
  )
)`

func main() {
  m, err := watgo.ParseWAT([]byte(wasmText))
  if err != nil {
    panic(err)
  }

  i32Params := 0
  localGets := 0
  i32Adds := 0

  // Module-defined functions carry a type index into m.Types. The function
  // body itself is a flat sequence of wasmir.Instruction values.
  for _, fn := range m.Funcs {
    sig := m.Types[fn.TypeIdx]
    for _, param := range sig.Params {
      if param.Kind == wasmir.ValueKindI32 {
        i32Params++
      }
    }

    for _, instr := range fn.Body {
      switch instr.Kind {
      case wasmir.InstrLocalGet:
        localGets++
      case wasmir.InstrI32Add:
        i32Adds++
      }
    }
  }

  fmt.Printf("module-defined funcs: %d\n", len(m.Funcs))
  fmt.Printf("i32 params: %d\n", i32Params)
  fmt.Printf("local.get instructions: %d\n", localGets)
  fmt.Printf("i32.add instructions: %d\n", i32Adds)
}

One important note: the WAT format supports several syntactic niceties that are flattened / canonicalized when lowered to wasmir. For example, all folded instructions are lowered to unfolded ones (linear form), function & type names are resolved to numeric indices, etc. This matches the validation and execution semantics of WASM and its binary representation.

These syntactic details are present in watgo in the textformat package (which parses WAT into an AST) and are removed when this is lowered to wasmir. The textformat package is kept internal at this time, but in the future I may consider exposing it publicly - if there's interest.

Testing strategy

Even though it's still early days for watgo, I'm reasonably confident in its correctness due to a strategy of very heavy testing right from the start.

WebAssembly comes with a large official test suite, which is perfect for end-to-end testing of new implementations. The core test suite includes almost 200K lines of WAT files that carry several modules with expected execution semantics and a variety of error scenarios exercised. These live in specially designed .wast files and leverage a custom spec interpreter.

watgo hijacks this approach by using the official test suite for its own testing. A custom harness parses .wast files and uses watgo to convert the WAT in them to binary WASM, which is then executed by Node.js [2]; this harness is a significant effort in itself, but it's very much worth it - the result is excellent testing coverage. watgo passes the entire WASM spec core test suite.

Similarly, we leverage wabt's interp test suite which also includes end-to-end tests, using a simpler Node-based harness to test them against watgo.

Finally, I maintain a collection of realistic program samples written in WAT in the wasm-wat-samples repository; these are also used by watgo to test itself.

[1]	Though not all of wasm-tools's functionality is supported yet.

[2]	To stick to a pure-Go approach also for testing, I originally tried using wazero for this, but had to give up because wazero doesn't support some of the recent WASM proposals that have already made it into the standard (most notably Garbage Collection).

Consistent hashing

2025-09-27T05:54:00-07:00

This post is an introduction to consistent hashing, an algorithm for designing a hash table such that only a small portion of keys has to be recomputed when the table's size changes.

Motivating use case

Suppose we're designing a caching web proxy, but the expected storage demands are higher than what a single machine can handle. So we distribute the cache across multiple machines. How do we do that? Given a URL, how do we make sure that we can easily find out which server we should approach for a potentially cached version [1]?

An approach that immediately comes to mind is hashing. Let's calculate a numeric hash of the URL and distribute it evenly between N nodes (that's what we'll call the servers in this post):

hash := calculateHashFunction(url)
nodeId := hash % N

This process works but turns out to have serious downsides in real-world applications.

The problem with the naive hashing approach

Consider our caching use case again; in a realistic application at "internet scale", one of the assumptions we made implicitly doesn't hold - the cache nodes are not static. New nodes are added to the system if the load is high (or if new machines come into service); existing nodes can crash or be taken offline for maintenance. In other words, the number N in our application is not a constant.

The problem may be apparent now; to demonstrate it directly, let's take an actual implementation of hashItem using Go's md5 package [2]:

// hashItem computes the slot an item hashes to, given a total number of slots.
func hashItem(item string, nslots uint64) uint64 {
  digest := md5.Sum([]byte(item))
  digestHigh := binary.BigEndian.Uint64(digest[8:16])
  digestLow := binary.BigEndian.Uint64(digest[:8])
  return (digestHigh ^ digestLow) % nslots
}

The terminology is slightly adjusted:

Instead of url, we'll just refer to a generic item
"Slot" is a common concept in hash tables: our hashItem computes a slot number for an item, given the total number of available slots

Let's say we started with 32 slots, and we hashed the strings "hello", "consistent" and "marmot". We get these slots:

hello       (n=32): 4
consistent  (n=32): 14
marmot      (n=32): 5

Now suppose that another node is added, and the total nslots grows to 33. Hashing our items again:

hello       (n=33): 23
consistent  (n=33): 18
marmot      (n=33): 31

All the slots changed!

This is a significant problem with the naive hashing approach. Whenever nslots changes, we get completely different slots for pretty much any item. In a realistic application it means that whenever a new node joins or leaves our caching cluster, there will be a flood of cache misses on every query until the new cluster settles. And node changes sometimes occur at the most incovenient times; imagine that the load is spiking (maybe a site was mentioned in a high-profile news outlet, or there's a live event streaming) and new nodes are added to handle it. This isn't a great time to temporarily lose all caching!

Consistent hashing

The consistent hashing algorithm solves the problem in an elegant way. The key idea is to map both nodes and items onto an interval, and then an item belongs to a node closest to it. Concretely, we take the unit circle [3], and map nodes and items to angles on this circle. Here's an example that explains how this method works in more detail:

This shows five nodes: N1 through N5, and three items: Ix, Iy, Iz. Initially, we add the nodes: using a hashing operation we map them onto the circle (details later). Then, as items come in, we determine which node they belong to, as follows:

Use the same hashing operation to find the item's location on the circle
The node this item belongs to is the closest one, in the clockwise direction

In our diagram, Ix is mapped to N1, Iy to N2, and Iz to N3. So far so good, but the benefit of this approach becomes apparent when the nodes change. In our diagram, suppose N3 is removed. Then Iz will map to N5. The mapping of the other items doesn't change!

Adding nodes has a similar outcome. If a new node N6 is added and it hashes to a position between Iy and N2 on the circle, from that moment Iy will be mapped to N6, but the other items keep their mapping.

Suppose we have a total of M items that we need to distribute across N nodes. Using the naive hashing approach, whenever we add or remove a node, all M items change their mapping. On the other hand, with consistent hashing only about \frac{M}{N} need to change. This is a huge difference.

The original consistent hashing paper (see [1]) calls this the monotonicity property of the algorithm:

If items are initially assigned to a set of buckets V_1, and then some new buckets are added to form V_2, then an item may move from an old bucket to a new bucket, but not from one old bucket to another.

Implementing consistent hashing

Implementing the consistent hashing algorithm as described above is fairly easy. The most critical part of the implementation is finding which node an item maps to - this involves some kind of search. The original consistent hashing paper suggests using a balanced binary tree for the search; the implementation I'm demonstrating here uses a slightly different but equivalent approach: binary search in a linear array of node positions (slots) [4].

First, some practical considerations:

Theoretically, the unit circle can be seen as the continuous range [0, 1). In programming we much prefer the discrete domain, however, so we're going to "quantize" this range to [0, ringSize), where ringSize is some suitably large number that avoids collisions.
Looking at the circle diagram above, imagine that 0 degrees is the "north" direction (12 o'clock), and angles increase clockwise. In our discrete domain, 12 o'clock is 0, 3 o'clock is ringSize/4, and so on.

When a node is added to the consistent hash, its location is found by applying a hash function like hashItem as described above, with nslots=ringSize. The nodes are stored using a pair of data structures, as follows; this example uses the approximate locations of the nodes N1 through N5 in the circle diagram above (assume ringSize=1024 here):

The positions of the nodes on the circle are stored in slots, which is sorted. nodes holds the corresponding node names. For each i, nodes[i] is at position slots[i] on the circle.

Here's the ConsistentHasher data structure in Go:

type ConsistentHasher struct {
  // nodes is a list of nodes in the hash ring; it's sorted in the same order
  // as slots: for each i, the node at index slots[i] is nodes[i].
  nodes []string

  // slots is a sorted slice of node indices.
  slots []uint64

  ringSize uint64
}

// NewConsistentHasher creates a new consistent hasher with a given maximal
// ring size.
func NewConsistentHasher(ringSize uint64) *ConsistentHasher {
  return &ConsistentHasher{
    ringSize: ringSize,
  }
}

And this is how finding which node a given item maps to is implemented:

// FindNodeFor finds the node an item hashes to. It's an error to call this
// method if the hasher doesn't have any nodes.
func (ch *ConsistentHasher) FindNodeFor(item string) string {
  if len(ch.nodes) == 0 {
    panic("FindNodeFor called when ConsistentHasher has no nodes")
  }
  ih := hashItem(item, ch.ringSize)

  // Since ch.slots is a sorted list of all the node indices for our nodes, a
  // binary search is what we need here. ih is mapped to the node that has the
  // same or the next larger node index. slices.BinarySearch does exactly this,
  // by returning the index where the value would be inserted.
  slotIndex, _ := slices.BinarySearch(ch.slots, ih)

  // When the returned index is len(slots), it means the search wrapped
  // around.
  if slotIndex == len(ch.slots) {
    slotIndex = 0
  }

  return ch.nodes[slotIndex]
}

The key here is the binary search invocation. Adding and removing nodes is done similarly using binary search - see the full code.

Better item distribution with virtual nodes

A common issue that comes up in the implementation of consistent hashing is unbalanced distribution of items across the different nodes. With M items and a total of N nodes, the average distribution will be about \frac{M}{N} per node, but in practice it won't be very balanced - some nodes will have many more items assigned to them than others (see the Appendix for more details).

In a real application, this may mean that some cache servers will be much busier than others, which is a bad thing as far as capacity planning and efficient use of HW. Luckily, there's an elegant tweak to the consistent hashing algorithm that significantly mitigates the problem: virtual nodes.

Instead of mapping each node to a single location on the circle, we'll map it to V locations instead. There are several ways to do this - the simplest is just to tweak the node name in some way. For example, when AddNode is called to add node, it will run:

for i := range V {
  vnodeName = fmt.Sprintf("%v@%v", node, i)

  // ... now add vnodeName to the nodes/slots slices
}

Then, when looking up an item we'll run into one of the virtual nodes, decode the node's name from it (in our example just strip the @<number> suffix) and return that. Implementing node removal is similarly simple.

The idea is that given a node named foo, the virtual node names foo@0, foo@1, foo@2 etc. will be spread all around the circle and not cluster in a single place. See the Appendix for a calculation of how this affects the final distribution.

The source code for this post includes a ConsistentHasherV type that is very similar to ConsistentHasher, except that it implements the virtual node strategy. The user interface remains exactly the same - it's only the internal implementation that changes slightly.

Code

The full source code for this post is on GitHub.

Appendix

The quality of the hash function is very important for good shuffling of nodes on the circle, but even if we take a perfect hash function that produces uniformly distributed values, the outcome is likely to be sub-optimal for our needs.

Let's say we select N points on the unit circle, uniformly in the range [0, 1). If we sort the points by angle, the gaps between neighboring angles are the order statistics. These follow the Beta distribution with parameters (1, N-1), which has a mean of \frac{1}{N} and a variance of \frac{N-1}{N^2(N+1)}.

This is quite significant. Consider a circle with 20 nodes. The standard deviation of the distribution is the square root of the variance; substituting N=20, we get:

\[\sigma=\sqrt{\frac{19}{20^2\cdot 21}}=0.048\]

With 20 nodes uniformly distributed around a circle, we can expect an average of 18 degrees distance between two nodes. A standard deviation of 0.048 means 17 degrees, which is comparable to the average!

We can also do a realistic example to demonstrate this. Let's generate 20 random angles on a circle, and show how the node distribution looks:

In this particular sample, the average angle between two adjacent nodes is 18 degrees (as expected). The smallest angle is just 1.04 degrees, while the largest one is 42 degrees. This means that some nodes will get 40x as many items assigned to them as others!

It's easy to see how virtual nodes help; imagine that each server maps to some number of randomly distributed nodes on the circle; some of these will be farther than others from their closest neighbor, but the average will have much less variety. Mathematically, given a set of n uniformly distributed random variables with variance v, the variance of their average is \frac{v}{n}.

As a concrete experiment, I ran a similar simulation to the one above, but with 10 virtual nodes per node. We'll consider the total portion of the circle mapped to a node when it maps to either of its virtual nodes. While the average remains 18 degrees, the variance is reduced drastically - the smallest one is 11 degrees and the largest 26.

You can find the code for these experiments in the demo.go file of the source code repository.

[1]

This is close to the original motivation for the development of consistent caching by researchers at MIT. Their work, described in the paper "Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web" became the foundation of Akamai Technologies.

There are other use cases of consistent hashing in distributed systems. AWS popularized one common application in their paper "Dynamo: Amazon’s Highly Available Key-value Store", where the algorithm is used to distribute storage keys across servers.

[2] In a real application, we'd probably want to find a different hash function. md5 is relatively slow, and we don't need any of its cryptographic guarantees in this use case. That said, the chosen hash function should be suitable and produce a good mixing of bits even for items that are only slightly different (e.g. "node-1" vs. "node-2"); I found that Go's built-in hash/fnv package isn't great for this purpose, for example.

[3]	A circle is used instead of a linear range to conveniently handle boundary conditions. It's similar to using modular arithmetic for the naive hashing approach.

[4]

I didn't bother to compare performance, but I suspect the array-based approach will be faster for lookup because of being relatively cache-friendly. For insertion/removal, the array approach is O(n) whereas the tree is O(\log\ n), but it's safe to assume that lookup performance is significantly more important. Nodes don't change very often, and lookups happen orders of magnitude more frequently.

Implementing Forth in Go and C

2025-08-26T20:38:00-07:00

I first ran into Forth about 20 years ago when reading a book about designing embedded hardware. The reason I got the book back then was to actually learn more about the HW aspects, so having skimmed the Forth chapter I just registered an "oh, this is neat" mental note and moved on with my life. Over the last two decades I heard about Forth a few more times here and there, such as that time when Factor was talked about for a brief period, maybe 10-12 years ago or so.

It always occupied a slot in the "weird language" category inside my brain, and I never paid it much attention. Until June this year, when a couple of factors combined fortuitously:

After spending much of the earlier part of 2025 exploring the inner workings of LLMs and digging in random mathy and algorithmic topics, I had an itch to just write some code.
I somehow found Dave Gauer's page about Forth and also the one on Implementing a Forth.

And something clicked. I'm going to implement a Forth, because... why not?

So I spent much of my free hacking time over the past two months learning about Forth and implementing two of them.

Forth: the user level and the hacker level

It's useful to think of Forth (at least standard Forth, not offshoots like Factor) as having two different "levels":

User level: you just want to use the language to write programs. Maybe you're indeed bringing up new hardware, and find Forth a useful calculator + REPL + script language. You don't care about Forth's implementation or its soul, you just want to complete your task.
Hacker level: you're interested in the deeper soul of Forth. Isn't it amazing that even control flow constructs like IF...THEN or loops like BEGIN...UNTIL are just Forth words, and if you wanted, you could implement your own control flow constructs and have them be first-class citizens, as seamless and efficient as the standard ones?

Another way to look at it (useful if you belong to a certain crowd) is that user-level Forth is like Lisp without macros, and hacker-level Forth has macros enabled. Lisp can still be great and useful without macros, but macros take it to an entire new level and also unlock the deeper soul of the language.

This distinction will be important when discussing my Forth implementations below.

goforth and ctil

There's a certain way Forth is supposed to be implemented; this is how it was originally designed, and if you get closer to the hacker level, it becomes apparent that you're pretty much required to implement it this way - otherwise supporting all of the language's standard words will be very difficult. I'm talking about the classical approach of a linked dictionary, where a word is represented as a "threaded" list [1], and this dictionary is available for user code to augment and modify. Thus, much of the Forth implementation can be written in Forth itself.

The first implementation I tried is stubbornly different. Can we just make a pure interpreter? This is what goforth is trying to explore (the Go implementation located in the root directory of that repository). Many built-in words are supported - definitely enough to write useful programs - and compilation (the definition of new Forth words using : word ... ;) is implemented by storing the actual string following the word name in the dictionary, so it can be interpreted when the word is invoked.

This was an interesting approach and in some sense, it "works". For the user level of Forth, this is perfectly usable (albeit slow). However, it's insufficient for the hacker level, because the host language interpreter (the one in Go) has all the control, so it's impossible to implement IF...THEN in Forth, for example (it has to be implemented in the host language).

That was a fun way to get a deeper sense of what Forth is about, but I did want to implement the hacker level as well, so the second implementation - ctil - does just that. It's inspired by the jonesforth assembly implementation, but done in C instead [2].

ctil actually lets us implement major parts of Forth in Forth itself. For example, variable:

: variable create 1 cells allot ;

Conditionals:

\ IF, ELSE, THEN work together to compile to lower-level branches.
\
\ IF ... THEN compiles to:
\   0BRANCH OFFSET true-part rest
\ where OFFSET is the offset of rest
\
\ IF ... ELSE ... THEN compiles to :
\   0BRANCH OFFSET true-part BRANCH OFFSET2 false-part rest
\ where OFFSET is the offset of false-part and OFFSET2 is the offset of rest
: if immediate
  ' 0branch ,
  here
  0 ,
  ;

: then immediate
  dup
  here swap -
  swap
  ! ;

: else immediate
  ' branch ,
  here
  0 ,
  swap
  dup
  here swap -
  swap
  ! ;

These are actual examples of ctil's "prelude" - a Forth file loaded before any user code. If you understand Forth, this code is actually rather mind-blowing. We compile IF and the other words by directly laying our their low-level representation in memory, and different words communicate with each other using the data stack during compilation.

Thoughts on Forth itself

Forth made perfect sense in the historic context in which it was created in the early 1970s. Imagine having some HW connected to your computer (a telescope in the case of Forth's creator), and you have to interact with it. In terms of languages at your disposal - you don't have much, even BASIC wasn't invented yet. Perhaps your machine still didn't have a C compiler ported to it; C compilers aren't simple, and C isn't very great for exploratory scripting anyway. So you mostly just have your assembly language and whatever you build on top.

Forth is easy to implement in assembly and it gives you a much higher-level language; you can use it as a calculator, as a REPL, and as a DSL for pretty much anything due to its composable nature.

Forth certainly has interesting aspects; it's a concatenative language, and thus inherently point-free. A classical example is that instead of writing the following in a more traditional syntax:

eat(bake(prove(mix(ingredients))))

You just write this:

ingredients mix prove bake eat

There is no need to explicitly pass parameters, or to explicitly return results. Everything happens implicitly on the stack.

This is useful for REPL-style programming where you use your language not necessarily for writing large programs, but more for interactive instructions to various HW devices. This dearth of syntax is also what makes Forth simple to implement.

All that said, in my mind Forth is firmly in the "weird language" category; it's instructive to learn and to implement, but I wouldn't actually use it for anything real these days. The stack-based programming model is cool for very terse point-free programs, but it's not particularly readable and hard to reason about without extensive comments, in my experience.

Consider the implementation of a pretty standard Forth word: +!. It expects and address at the top of stack, and an addend below it. It adds the addend to the value stored at that address. Here's a Forth implementation from ctil's prelude:

: +!        ( addend addr -- )
  tuck      ( addr addend addr )
  @         ( addr addend value-at-addr )
  +         ( addr updated-value )
  swap      ( updated-value addr )
  ! ;

Look at that stack wrangling! It's really hard to follow what goes where without the detailed comments showing the stack layout on the right of each instruction (a common practice for Forth programs). Sure, we can create additional words that would make this simpler, but that just increases the lexicon of words to know.

My point is, there's fundamental difficulty here. When you see this C code:

int func(int a, int b) {
  return foo(a, bar(b));
}

Even without any documentation, you can immediately know several important things:

bar has one parameter and one return value
foo has two parameters and one return value
func also has two parameters and one return value
It's immediately obvious how the various values flow from one function call to the next.

Written in Forth [3]:

: func bar foo ;

How can you know the arity of the functions without adding explicit comments? Sure, if you have a handful of words like bar and foo you know like the back of your hand, this is easy. But imagine reading a large, unfamiliar code base full of code like this and trying to comprehend it.

Summary and links

The source code of my goforth project is on GitHub; both implementations are there, with a comprehensive test harness that tests both.

The learn Forth itself, I found these resources very useful:

Dave Gauer's Forth page
Starting Forth - a free online book / tutorial on Forth for beginners

To learn how to implement Forth:

Dave Gauer's page on Implementing a Forth
jonesforth implementation
Threaded Interpretive Languages - an old but nice book that explains how Forth implementations typically work

Implementing Forth is a great self-improvement project for a coder; there's a pleasantly challenging hump of understanding to overcome, and you gain valuable insights into stack machines, interpretation vs. compilation and mixing these levels of abstraction in cool ways.

Also, implementing programming languages from scratch is fun! It's hard to beat the feeling of getting to interact with your implementation for the first time, and then iterating on improving it and making it more featureful. Just one more word!

[1]	This has nothing to do with threads in the sense of concurrency. Rather, it's thread like in sewing, where the elements of the list are all connected to each other as if with a thread. See this page for more details.

[2]

Which is another deviation from the norm. Forth is really supposed to be implemented in assembly - this is what it was designed for, and it's very clear from its structure that it must be so in order to achieve peak performance.

But where's the fun in doing things the way they were supposed to be done? Besides, jonesforth is already a perfectly fine Forth implementation in assembly, so I wouldn't have learned much by just copying it.

I had a lot of fun coding in C for this one; it's been a while since I last wrote non-trivial amounts of C, and I found it very enjoyable.

[3]	Assuming the convention that multi-parameter functions have their parameters pushed to the stack from left to right.

Bloom filters

2025-05-01T18:35:00-07:00

The original motivation for the creation of Bloom filters is efficient set membership, using a probabilistic approach to significantly reduce the time and space required to reject items that are not members in a certain set.

The data structure was proposed by Burton Bloom in a 1970 paper titled "Space/Time Trade-offs in Hash Coding with Allowable Errors". It's a good paper that's worth reading.

Why Bloom filters?

Suppose that we store some information on disk and want to check if a certain file contains a certain entry. Reading from disk is time consuming, so we want to minimize it as much as possible. A Bloom filter is a data structure that implements a cache with probabilistic properties:

If the cache says the key is not present in a specific file, then it's 100% certain we should not be reading the file.
If the cache says the key is present in the file, there's a small chance this is a false positive (and in fact the key isn't there). In this case we just read the file as usual.

In a scenario where the majority of queries "is this key in that file?" have a negative answer, a Bloom filter can significantly speed up the system [1]. Moreover, the probabilistic nature (the existence of false positives) allows Bloom filters to be extremely fast and occupy very little space. Here's a quote from the Bloom paper:

The new hash-coding methods to be introduced are suggested for applications in which the great majority of messages to be tested will not belong to the given set. For these applications, it is appropriate to consider as a unit of time (called reject time) the average time required to classify a test message as a nonmember of the given set.

How a Bloom filter works

A Bloom filter is a special kind of a hash table with open addressing. It's an array of bits (the length is typically denoted m), and some fixed number (k) of hash functions. We'll assume each hash function can take an arbitrary sequence of bytes and hash it into an integer in the inclusive range [0, m-1]. A Bloom filter supports two operations:

Insert an item: the item is hashed using each of the k hash functions, and the appropriate bits in the underlying array are set to 1.

Test if an item is a member: the item is hashed using each of the k hash functions. If any of the bits indicated by their results is 0, we return "false" with certainty. If all the bits are 1, we return "true" - and there's a small chance of false positives.

Here's how the Bloom paper describes it:

The hash area is considered as N individual addressable bits, with addresses 0 through N - 1. It is assumed that all bits in the hash area are first set to 0.

Next, each message in the set to be stored is hash coded into a number of distinct bit addresses, say a1, a2, . . . , ad. Finally, all d bits addressed by a1 through ad are set to 1.

To test a new message a sequence of d bit addresses, say a'1, a'2, ... a'd, is generated in the same manner as for storing a message. If all d bits are 1, the new message is accepted. If any of these bits is zero, the message is rejected.

Hopefully it's clear why this data structure is probabilistic in nature: it's possible that different items hash to the same number, and therefore when we test some X, all its hashes point to bits turned on by the hashing of other data. Read the Math appendix for the math behind Bloom filters and how to calculate (and design for a specific) the false positive rate.

Here's an example:

We start with an empty bloom filter with m=16 and k=3. All bits are initialized to 0.
Insertion of "x". The three hashes return indices 1, 6, 15, so these bits in the array are set to 1.
Insertion of "y". Hashing returns indices 6, 9 and 13, so these bits in the array are set to 1. Note that bit 6 is set for both "x" and "y", and that's fine.

Next, let's look at some membership tests:

Test "x". Hashing returns 1, 6, 15; all these bits are 1 in the array, so the answer is "true". This is a true positive.
Test "w". Hashing returns 3, 9, 13. Since the bit at position 3 is 0, the answer is "false".
Test "v". Hashing returns 9, 13, 15; all these bits are 1 in the array, so the answer is "true". This is a false positive.

Note that it's trivial to prove (by the law of contraposition) that all "false" answers from a Bloom filter's test operation are true negatives.

Implementation

Here's a simple implementation of a Bloom filter in Go:

// New creates a new BloomFilter with capacity m, using k hash functions.
// You can calculate m and k from the number of elements you expect the
// filter to hold and the desired error rate using CalculateParams.
func New(m uint64, k uint64) *BloomFilter {
  return &BloomFilter{
    m:      m,
    k:      k,
    bitset: newBitset(m),
    seed1:  maphash.MakeSeed(),
    seed2:  maphash.MakeSeed(),
  }
}

type BloomFilter struct {
  m      uint64
  k      uint64
  bitset []uint64

  // seeds for the double hashing scheme.
  seed1, seed2 maphash.Seed
}

// Insert a data item into the bloom filter.
func (bf *BloomFilter) Insert(data []byte) {
  h1 := maphash.Bytes(bf.seed1, data)
  h2 := maphash.Bytes(bf.seed2, data)
  for i := range bf.k {
    loc := (h1 + i*h2) % bf.m
    bitsetSet(bf.bitset, loc)
  }
}

// Test if the given data item is in the bloom filter. If Test returns false,
// it's guaranteed that data was never added to the filter. If it returns true,
// there's an eps probability of this being a false positive. eps depends on
// the parameters the filter was created with (see CalculateParams).
func (bf *BloomFilter) Test(data []byte) bool {
  h1 := maphash.Bytes(bf.seed1, data)
  h2 := maphash.Bytes(bf.seed2, data)
  for i := range bf.k {
    loc := (h1 + i*h2) % bf.m
    if !bitsetTest(bf.bitset, loc) {
      return false
    }
  }
  return true
}

The bitsetSet and bitsetTest functions can be seen in the full code repository.

This implementation uses double hashing to generate k different hash functions from just two hashes.

The code also mentions the CalculateParams function:

// CalculateParams calculates optimal parameters for a Bloom filter that's
// intended to contain n elements with error (false positive) rate eps.
func CalculateParams(n uint64, eps float64) (m uint64, k uint64) {
  // The formulae we derived are:
  // (m/n) = -ln(eps)/(ln(2)*ln(2))
  // k = (m/n)ln(2)

  ln2 := math.Log(2)
  mdivn := -math.Log(eps) / (ln2 * ln2)
  m = uint64(math.Ceil(float64(n) * mdivn))
  k = uint64(math.Ceil(mdivn * ln2))
  return
}

You'll have to read the Math appendix to understand how it works.

Practicalities

Let's look at a practical example of a realistic Bloom filter and how it performs. Suppose we want to store about 1 billion items, and have a false positive rate of 1% (meaning that if the filter returns "true" for a test, there's a 99% chance that the item was previously added to the filter). Using these requirements, we can invoke CalculateParams to get the Bloom filter parameters:

CalculateParams(1000000000, 0.01) ===> (9585058378 7)

This means m is about 9.6 billion (bits) and k is 7. In other words, our Bloom filter requires about 1.2 GB of space to cache the membership test of a billion items (that could be of arbitrary size). Moreover, the lookup is very fast - it's just 7 applications of the hash function. The constant lookup cost is particularly attractive, as it doesn't depend on the number of items actually inserted, or on any particular pattern in the data (i.e. there are no worst case scenarios with asymptotically higher cost).

On my machine, benchmarking with the Go implementation shown above I get ~80 nanoseconds per lookup. Mind you, this is the simplest Go implementation I could think of - nothing here is optimized; I'm sure this can be improved at least 2x by using a more speed-optimized hash implementation, for example.

Now imagine how long it would take to ascertain if data is present in a file with 1 billion entries, even if the file contains proper indexing for fast lookups. Just asking the OS to read the file's first few KiBs to get at the index would take orders of magnitude longer than 80 ns.

Recall that Bloom filters are best suited for cases "in which the great majority of messages to be tested will not belong to the given set". Moreover, even if the data exists in the file, false positives only happen 1% of the time. Therefore, the number of times we'll have to go to the disk just to find the data is not there is a very small fraction of total accesses.

Code

The full code for this post, with tests, is available on GitHub.

Appendix: the Math behind Bloom filters

A reminder on notation:

m: size (in bits) of the set
n: how many keys were inserted into the filter
k: number of hash functions used to insert/test each key

For a specific bit in the set, assuming our hash functions distribute the keys randomly, the probability of it not being set by a specific hash function is:

\[p_0=1-\frac{1}{m}\]

And the probability it’s not set by either of our k hash functions is:

\[p_0=\left ( 1-\frac{1}{m}\right )^k=\left( \left ( 1-\frac{1}{m}\right )^m\right)^\frac{k}{m}\]

The last formula is constructed to use an approximation of e^x for a large enough m (see the appendix in this post) to write:

\[p_0\approx e^{-\frac{k}{m}}\]

After inserting n elements, the probability that it’s 0 is:

\[p_0\approx e^{-\frac{kn}{m}}\]

Meaning that the probability of it being 1 is:

\[p_1\approx 1-e^{-\frac{kn}{m}}\]

Recap: this is the probability of any given bit being 1 after n bits were inserted into a set of size m with k different hash functions.

Assuming independence between our hash functions (this is not super rigorous, but a reasonable assumption in practice), let’s calculate the false positive rate. Suppose we have a new key that’s not in the set, and we’re trying to check its membership by hashing it with our k hash functions. The false positive rate is the probability that all hashes land on a bit that’s already set to 1:

\[p_{fp}\approx\left ( 1-e^{-\frac{kn}{m}} \right )^k\approx \varepsilon\]

This is also called the error rate of our filter, or \varepsilon. To get an optimal (minimal) false positive rate, let’s minimize \varepsilon. Since the logarithm function is monotonically increasing, it will be more convenient to minimize \ln(\varepsilon):

\[ln(\varepsilon)=\ln\left ( 1-e^{-\frac{kn}{m}} \right )^k=k\cdot \ln\left ( 1-e^{-\frac{kn}{m}} \right )\]

We’ll calculate the derivative w.r.t. k and set it to 0:

\[\begin{aligned} \frac{d}{dk}ln(\varepsilon)&=\frac{d}{dk}k\cdot \ln\left ( 1-e^{-\frac{kn}{m}} \right )\\ &=\ln\left ( 1-e^{-\frac{kn}{m}} \right ) + k\frac{e^{-\frac{kn}{m}}\cdot \frac{n}{m}}{1-e^{-\frac{kn}{m}}} \end{aligned}\]

Substituting a variable t=e^{-\frac{kn}{m}} and using some more calculus and algebra, we can find that:

\[k= \frac{m}{n}\cdot \ln(2)\]

A numerical example: if we have a set with 1 million bits, and we expect to insert about 100,000 elements, the optimal number of hash functions is:

\[k= 10\cdot \ln(2)= 6.93 \approx 7\]

However, it’s more useful to aim for a certain error rate, and set the filter parameters accordingly. Let’s assume we’ll be using this optimal value of k. Substituting k= \frac{m}{n}\cdot \ln(2) into the equation for \varepsilon from above:

\[\begin{aligned} \varepsilon&\approx \left ( 1-e^{-\frac{n}{m}\cdot{\frac{m}{n}\cdot \ln 2}} \right )^{\frac{m}{n}\cdot \ln 2}\\ &\approx \left ( 1-e^{-\ln 2} \right )^{\frac{m}{n}\cdot \ln 2}\\ &\approx \left ( \frac{1}{2} \right )^{\frac{m}{n}\cdot \ln 2}\\ \end{aligned}\]

If we use the numerical example from before with \frac{m}{n}=10, the error rate with an optimal k will be 0.5^{6.93}\approx 0.8\%.

What often happens is that we have an error rate in mind and we want to calculate how many bits per element we want to dedicate in our set. Let’s take the previous equation and try to isolate \frac{m}{n} from it using a logarithm:

\[\ln \varepsilon\approx \frac{m}{n}\cdot \ln 2 \cdot \ln 2^{-1}=-\frac{m}{n}\ln^2(2)\]

Then:

\[\frac{m}{n}\approx -\frac{\ln \varepsilon}{\ln^2(2)}\]

Final numerical example: suppose we want an error (false positive) rate of 1%. This means our set should have:

\[\frac{m}{n}\approx -\frac{\ln 0.01}{\ln^2 (2)}=9.58\]

... bits per element. So if we expect about 100,000 elements, the bit set used for our filter should have at least 958,000 bits. And, as calculated earlier, we should be using k=7 hash functions to achieve this optimal error rate.

[1]	For this reason, Bloom filters are very common in data storage systems. Here's a discussion about Cassandra, but there are many others.

Implementing Raft: Part 5 - Exactly-once delivery

2024-12-18T06:01:00-08:00

This is Part 5 in a series of posts describing the Raft distributed consensus algorithm and its complete implementation in Go. Here is a list of posts in the series:

Part 0: Introduction
Part 1: Elections
Part 2: Commands and log replication
Part 3: Persistence and optimizations
Part 4: Key/Value database
Part 5: Exactly-once delivery (this post)

In this part, we're completing the implementation of a replicated key / value database based on Raft consensus. At the end of Part 4 we discussed a consistency issue that may arise due to client retry logic; now is the time to address it.

All the code for this part is located in this directory.

Adding an `APPEND` operation to our database

As a quick reminder, these are the basic operations our KV DB from part 4 supports:

PUT(k,v): assign value v to key k
GET(k): retrieve the value associated with key k
CAS(k, cmp, v): atomic compare-and-swap. First, it reads curV - the current value associated with key k. If curV==cmp, assigns value v to k instead; otherwise, it's a no-op. In any case, curV is returned.

Let's add another operation to this set; this is APPEND(k,v), which appends v to the value of key k (in our implementation, keys and values are both arbitrary Go strings); if there was no k in the DB before this operation, it behaves like PUT(k,v).

For example, consider this sequence of commands (in order from left to right):

PUT("x","foo")  APPEND("x", "bar")  APPEND("y","hello")

Applied to an empty DB, these commands will result in these keys / values:

x=foobar
y=hello

The problem with client retries - demonstrated with `APPEND`

The way our KV client works is described in detail in Part 4. As a reminder, the client tries the KV services one by one, submitting a command to them until it gets a success response from a leader. The client also remembers which service was the leader the last time it tried, to avoid wasting time on the search next time.

Suppose we've already submitted PUT("x","foo") successfully to the database, and now we want to send the APPEND("x","bar") command. Suppose also that our client remembers that service B was the leader (in a cluster of three services: A, B and C). It sends the APPEND("x","bar") request to service B. What happens if the client doesn't get a response from B? It assumes something happened to B (maybe it has crashed or was partitioned from the network), and retries the same request - sending it to C.

But now suppose that B actually received the APPEND command and committed it to the Raft cluster, but crashed before sending the HTTP response back to the client (or maybe the HTTP response got delayed beyond the client's timeout, due to a network glitch). Due to the same error, B then loses cluster leadership. The client will keep retrying this request, until it finds a leader that answers with success; therefore, the APPEND may be applied twice (or even more times, if the failure mode recurs) and the value of x in the DB will end up being "foobarbar". This is bad!

You may be tempted to blame the client's retry behavior here; but let's think this through. Suppose we didn't have the client layer doing retries; we send an APPEND command to a service, and don't hear anything back. What do we do next? Is there any way to know that the request was actually committed? Well, we can send a GET request to check, but this quickly gets complicated in a real distributed system, because our operation is no longer atomic (some other client may have changed the key's value since then, so what is our GET supposed to check?).

The problem isn't the retry itself; it's retrying with insufficient safety guarantees in the core algorithm.

Solving the retry problem with command de-duplication

The problem described above isn't just an issue in our implementation. It's explicitly called out in the original Raft paper, in section 8:

However, as described so far Raft can execute a command multiple times: for example, if the leader crashes after committing the log entry but before responding to the client, the client will retry the command with a new leader, causing it to be executed a second time. The solution is for clients to assign unique serial numbers to every command. Then, the state machine tracks the latest serial number processed for each client, along with the associated response. If it receives a command whose serial number has already been executed, it responds immediately without re-executing the request.

The paper also suggests a solution to the problem, and this is what we're going to implement. If we can uniquely identify commands committed to the Raft log, the KV service can avoid applying the same commands twice.

The idea is to identify commands uniquely as follows:

Each client has a globally unique ID
Each command sent by a client has a unique ID, distinct from all other IDs sent by the client. Moreover, to make our algorithm efficient this ID is monotonically increasing; assuming each client has its own monotonic clock, a command sent at time T_1 will have an ID larger than a command sent at T_0 iff T_1>T_0.

In our APPEND example, let's say the client's ID is 42, and let's say the ID of the APPEND("x","bar") command it sends is 1. The command is sent to B, which commits it successfully - with the ID tuple (42,1) - to the Raft log; B crashes before responding to the client, so the client retries the command with C. Since it's the same command, it has the same ID tuple (42,1). The KV service in C will notice that such an ID was already applied, and will not apply it again [1]!

Implementing de-duplication

We'll start with the client. Two fields are added to the KVClient struct:

// clientID is a unique identifier for a client; it's managed internally
// in this file by incrementing the clientCount global.
clientID int64

// requestID is a unique identifier for a request a specific client makes;
// each client manages its own requestID, and increments it monotonically and
// atomically each time the user asks to send a new request.
requestID atomic.Int64

The clientID field is assigned when a client is created, using a global atomic that auto-increments:

func New(serviceAddrs []string) *KVClient {
  return &KVClient{
    // ... other fields
    clientID:      clientCount.Add(1),
  }
}

// clientCount is used to assign unique identifiers to distinct clients.
var clientCount atomic.Int64

This provides uniqueness for our tests, but in realistic applications you'll probably want something stronger. A simple and pragmatic approach could be using an UUID here - I'll leave this as an exercise for motivated readers.

The requestID field tracks the ID of the last request this client has sent. Each time the client sends a new request, this is incremented [2]. For example, here's the new Append method:

// Append the value to the key in the store. Returns an error, or
// (prevValue, keyFound, false), where keyFound specifies whether the key was
// found in the store prior to this command, and prevValue is its previous
// value if it was found.
func (c *KVClient) Append(ctx context.Context, key string, value string) (string, bool, error) {
  appendReq := api.AppendRequest{
    Key:       key,
    Value:     value,
    ClientID:  c.clientID,
    RequestID: c.requestID.Add(1),
  }
  var appendResp api.AppendResponse
  err := c.send(ctx, "append", appendReq, &appendResp)
  return appendResp.PrevValue, appendResp.KeyFound, err
}

Both IDs are part of the HTTP request sent to service. Here's the request struct for appends:

type AppendRequest struct {
  Key   string
  Value string

  ClientID  int64
  RequestID int64
}

The other commands are modified similarly; all the retry logic (in the send method) remains the same - it just keeps retrying with the same client+request IDs.

The changes in the service are slightly deeper, but not too difficult overall. First, we add a field to the KVService struct:

// lastRequestIDPerClient helps de-duplicate client requests. It stores the
// last request ID that was applied by the updater per client; the assumption
// is that client IDs are unique (keys in this map), and for each client the
// requests IDs (values in this map) are unique and monotonically increasing.
lastRequestIDPerClient map[int64]int64

We also add some fields to the Command struct:

// ClientID and RequestID uniquely identify the request+client.
ClientID, RequestID int64

// IsDuplicate is used to mark the command as a duplicate by the updater. When
// the updater notices a command that has a client+request ID that has already
// been executed, the command is not applied to the datastore; instead,
// IsDuplicate is set to true.
IsDuplicate bool

As a reminder, Command is the "payload" we submit to the Raft log [3].

The bulk of the logic is in the goroutine running runUpdater:

func (kvs *KVService) runUpdater() {
  go func() {
    for entry := range kvs.commitChan {
      cmd := entry.Command.(Command)

      // Duplicate command detection.
      // Only accept this request if its ID is higher than the last request from
      // this client.
      lastReqID, ok := kvs.lastRequestIDPerClient[cmd.ClientID]
      if ok && lastReqID >= cmd.RequestID {
        kvs.kvlog("duplicate request id=%v, from client id=%v", cmd.RequestID, cmd.ClientID)
        // Duplicate: this request ID was already applied in the past!
        cmd = Command{
          Kind:        cmd.Kind,
          IsDuplicate: true,
        }
      } else {
        kvs.lastRequestIDPerClient[cmd.ClientID] = cmd.RequestID

        switch cmd.Kind {
        case CommandGet:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.Get(cmd.Key)
        case CommandPut:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.Put(cmd.Key, cmd.Value)
        case CommandAppend:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.Append(cmd.Key, cmd.Value)
        case CommandCAS:
          cmd.ResultValue, cmd.ResultFound = kvs.ds.CAS(cmd.Key, cmd.CompareValue, cmd.Value)
        default:
          panic(fmt.Errorf("unexpected command %v", cmd))
        }
      }

      // Forward this command to the subscriber interested in its index, and
      // close the subscription - it's single-use.
      if sub := kvs.popCommitSubscription(entry.Index); sub != nil {
        sub <- cmd
        close(sub)
      }
    }
  }()
}

This relies on the IDs from each client being monotonically increasing, and thus we only have to maintain O(1) of state per client [4]. When we apply a client request to the state machine, we remember the ID of this request. If the code is ever asked to apply the same ID (or a lower ID), it refuses, marking the command as IsDuplicate=true instead. Then, the HTTP handler that tried to submit the command has to deal with this situation; for example, in handleAppend:

sub := kvs.createCommitSubscription(logIndex)

select {
case commitCmd := <-sub:
  if commitCmd.ServiceID == kvs.id {
    if commitCmd.IsDuplicate {
      kvs.sendHTTPResponse(w, api.AppendResponse{
        RespStatus: api.StatusDuplicateRequest,
      })
    } else {
      kvs.sendHTTPResponse(w, api.AppendResponse{
        RespStatus: api.StatusOK,
        KeyFound:   commitCmd.ResultFound,
        PrevValue:  commitCmd.ResultValue,
      })
    }
  } else {
    kvs.sendHTTPResponse(w, api.AppendResponse{RespStatus: api.StatusFailedCommit})
  }
case <-req.Context().Done():
  return
}

For duplicates the service returns a special API status: api.StatusDuplicateRequest. Our client treats this as an error and surfaces it to the user. As an exercise, try changing it so the return value from duplicates is normal (success). The challenge here is to record - for each request - what the result is for returning to the client (e.g. the previous value of a key in case of PUT or APPEND).

Revisiting our consistency guarantees

In Part 4, we've discussed the consistency guarantees of our KV service in detail, and concluded that it's strict serializable, which is the strongest consistency guarantee for distributed systems.

However, when adding the client module we've also noted that - due to the client retry problem - the whole system is no longer linearizable (and hence no longer strict serializable). Linearizability extended to the client is known to be tricky; this isn't surprising - after all, the client is yet another network-connected component in the system, with inherently unreliable communication to the service.

With de-duplication, our entire system is strict-serializable again. Even if a client re-sends a command that was already committed to the Raft log, this retried command won't be committed a second time due to the de-duplication logic. Users will not observe non-linearizable behavior.

Exactly-once delivery

The discussion around delivery semantics can get quite heated - people have strong feelings about a topic in which academic rigor is often insufficiently applied. Still, I'll take the chance of discussing these semantics in the context of the Raft-based KV DB we've built through this series of posts.

Consider a case in which we have no client retries. A client submits a user's request just once to the underlying Raft service. Raft guarantees that if the system is sufficiently connected (e.g. there are enough live and connected peer to achieve consensus), the command will be applied to the Raft log once. The Raft protocol is both theoretically and practically proven at this point, so let's take this as an axiom. So we can say that without client retries, a command is delivered at most once. It can fail to be delivered (if there's a persistent network partition preventing the Raft cluster from having enough live peers for consensus), but it will not be delivered more than once.

For some applications, at most once delivery is a sufficient guarantee. Think about some sort of distributed logging or telemetry, for example. This isn't the case for the kind of KV DB we're trying to build, though, because it's intended to serve as a rock-solid basis for other distributed applications.

Next we've added client retries; this is essential in light of the imperfect physical world in which our code operates. With client retries, assuming the HW and network is working "in the long term" (e.g. network partitions get fixed within some reasonable time, and crashing servers get restarted or replaced), we get at least once semantics. The client will just keep retrying until it gets notified that the command was applied. However, as we've seen at the beginning of this post, this also means duplicate delivery when some failure scenarios occur.

The goal of adding de-duplication is to move our system to exactly-once delivery. Exactly-once delivery is a highly debated topic, but with some reasonable real-world assumptions, it can be achieved. Consider our implementation, for example (with de-duplication of retries). As long as the network and HW are not permanently broken, a command will either be applied exactly once to the DB, or the client will be notified of an error.

Here's a post by Ron Garret where he provides a useful framework to think about this. The post is long but worth reading; here's a quote I liked:

If you can get at-least-once delivery, you can build exactly-once on top of that

I believe this Raft series has demonstrated how this is done. Another interesting read is this post by the Kafka developers where they discuss how exactly-once semantics were added to Kafka a few years ago. If you read it you'll see it's pretty much the same technique - client retries with de-duplication of commands in the distributed log.

Finally, in Designing Data-Intensive Applications, Martin Kleppman touches upon this topic in several places; for example, in chapter 12 he discusses exactly-once execution of operations and the engineering required to make it possible.

Conclusion

This concludes our series of posts on implementing a full Raft consensus module in Go, and building a strictly serializable KV DB on top of it.

For any questions or comments about these posts or the code, please send me an email or open an issue on GitHub.

Appendix 1: Retries with PUT and linearizability

In this post we've added an APPEND operation to demonstrate the issues that arise with retries and duplication. But APPEND differs significantly from the original operations we've discussed in part 4: it's not idempotent. Applying APPEND more than once creates an invalid value, which isn't true for PUT - if we had retried PUT multiple times, the only outcome would be writing the result twice.

While this seems logical on the surface, it turns out that even without operations like APPEND, the system isn't linearizable if retries without de-duplication are allowed. Here's a diagram (following the style of my post on linarizability) that demonstrates the issue:

This diagram describes a single "register", let's assume the key "foo". Let's also assume that before any writes, the default value of all keys is 0. Here's what happens:

Client A issues a PUT('foo',1) request. The request gets committed by the leader, which crashes before a reply is made to the client. The client will continue periodically retrying.
In the meantime, a new leader is elected. Client C reads the value of the register and gets a successful response (since the PUT of 1 was committed).
Client B uses the new leader to commit a new PUT, with value 2.
Client C reads the value of 2 from the register.
Client A's retry reaches the new leader, which commits another instance of PUT('foo',1), overwriting the value 2. Now when client C reads the register again, it gets 1 again.

This sequence isn't linearizable; as far as the clients are concerned, since operation (2) retrieved the value 1, it means (1) finished before (2); the result of (4) also means that (3) happened after (1). This makes (5) impossible in a linearizable system.

This is a known failure scenario in distributed systems; it's called lost update. In our example, the PUT(2) operation is essentially lost outside a brief window just following it. The retry of PUT(1) overwrites it.

Appendix 2: lost updates in etcd with client retries

To demonstrate that the issue discussed in this post isn't purely academic, here's a real world example.

etcd is an industrial-strength key value DB based on Raft. It's used extensively inside k8s and other projects. While the etcd service itself is strict serializable, some of its client libraries turned out to have the exact retry problem described in this post.

Here's a Jepsen analysis of jetcd, a Java-based client that automatically retries on failures. The analysis concludes that this mechanism results in loss of linearizability, and recommends to disable it. There's also a lengthy discussion in a GitHub issue with the etcd developers about this. And another issue has interesting information as well.

The Jepsen analysis has a great quote which I want to repost here, because it's so relevant to our discussion (and Appendix 1 in particular):

It is easy to assume that set(x, 5) is idempotent because applying it twice in a row still produces the state x = 5. However, this operation is not longer idempotent if its executions are interleaved with other writes—then, it leads to lost update.

Interestingly, etcd doesn't support operations like APPEND at all. It can be emulated with transactions, however, since etcd's data store is versioned. These features also allow one to be more careful around failures when non-linearizable behavior isn't acceptable. For example, we can perform writes like this:

1. Read revision --> $rev
2. TXN
      if mod_revision(k) == $rev
      PUT(k, v)

This will only assign store(k)=v if there were no changes to the DB in the meantime. The failure scenario shown in Appendix 1 cannot happen, because each successful write increments the revision of the store. If unsuccessful, this sequence of operations can be retried safely (its only problem is with liveness, not safety).

[1]	For this reason it's important to store the ID of the command in the Raft log along with the command itself - we need this de-duplication to work across peers.

[2]	If you're concerned that a client may send more than 9.2 quintillion requests in its lifetime, this is easy to change to an arbitrarily large number using `math/big.Int`.

[3]	And we also reuse it to communicate results back from the updater goroutine to request handlers - this is why `IsDuplicate` is there. A simple refactoring exercise could be to use a separate data structure for this purpose.

[4]

This assumes the total number of distinct clients this service is dealing with is not overly large; in realistic systems, this is a reasonable assumption. In extreme cases, if we foresee having to deal with an unbounded number of clients, some sort of "garbage collection" scheme should be maintained (we can "forget" a client after some timeout of not hearing from it).

Eli Bendersky's website - Go

watgo - a WebAssembly Toolkit for Go

CLI use case

API use case

Testing strategy

Consistent hashing

Motivating use case

The problem with the naive hashing approach

Consistent hashing

Implementing consistent hashing

Better item distribution with virtual nodes

Code

Appendix

Implementing Forth in Go and C

Forth: the user level and the hacker level

goforth and ctil

Thoughts on Forth itself

Summary and links

Bloom filters

Why Bloom filters?

How a Bloom filter works

Implementation

Practicalities

Code

Appendix: the Math behind Bloom filters

Implementing Raft: Part 5 - Exactly-once delivery

Adding an APPEND operation to our database

The problem with client retries - demonstrated with APPEND

Solving the retry problem with command de-duplication

Implementing de-duplication

Revisiting our consistency guarantees

Exactly-once delivery

Conclusion

Appendix 1: Retries with PUT and linearizability

Appendix 2: lost updates in etcd with client retries

Adding an `APPEND` operation to our database

The problem with client retries - demonstrated with `APPEND`