On concurrency in Go HTTP servers



Go's built-in net/http package is convenient, solid and performant, making it easy to write production-grade web servers. To be performant, net/http automatically employs concurrency; while this is great for high loads, it can also lead to some gotchas. In this post I want to explore this topic a bit.

Ensuring safe concurrent access from handlers to data

Let's start with a very simple example of a HTTP server that implements a table of counters accessible to the user. We can create counters (or set values of existing counters) with the set?name=N&val=V query, get their values with the get?name=N query and increment them with the inc?name=N query. Here's a simple interaction recorded with a server running in the background on port 8000:

$ curl "localhost:8000/set?name=x&val=0"
ok
$ curl "localhost:8000/get?name=x"
x: 0
$ curl "localhost:8000/inc?name=x"
ok
$ curl "localhost:8000/get?name=x"
x: 1

And a basic server implementing this functionality:

package main

import (
  "fmt"
  "log"
  "net/http"
  "os"
  "strconv"
)

type CounterStore struct {
  counters map[string]int
}

func (cs CounterStore) get(w http.ResponseWriter, req *http.Request) {
  log.Printf("get %v", req)
  name := req.URL.Query().Get("name")
  if val, ok := cs.counters[name]; ok {
    fmt.Fprintf(w, "%s: %d\n", name, val)
  } else {
    fmt.Fprintf(w, "%s not found\n", name)
  }
}

func (cs CounterStore) set(w http.ResponseWriter, req *http.Request) {
  log.Printf("set %v", req)
  name := req.URL.Query().Get("name")
  val := req.URL.Query().Get("val")
  intval, err := strconv.Atoi(val)
  if err != nil {
    fmt.Fprintf(w, "%s\n", err)
  } else {
    cs.counters[name] = intval
    fmt.Fprintf(w, "ok\n")
  }
}

func (cs CounterStore) inc(w http.ResponseWriter, req *http.Request) {
  log.Printf("inc %v", req)
  name := req.URL.Query().Get("name")
  if _, ok := cs.counters[name]; ok {
    cs.counters[name]++
    fmt.Fprintf(w, "ok\n")
  } else {
    fmt.Fprintf(w, "%s not found\n", name)
  }
}

func main() {
  store := CounterStore{counters: map[string]int{"i": 0, "j": 0}}
  http.HandleFunc("/get", store.get)
  http.HandleFunc("/set", store.set)
  http.HandleFunc("/inc", store.inc)

  portnum := 8000
  if len(os.Args) > 1 {
    portnum, _ = strconv.Atoi(os.Args[1])
  }
  log.Printf("Going to listen on port %d\n", portnum)
  log.Fatal(http.ListenAndServe("localhost:"+strconv.Itoa(portnum), nil))
}

This code is simple; too simple, in fact, to the degree that it's wrong. Our sample curl session sends all its requests in a serial manner. The problems appear when concurrent connections are present, however. A good way to simulate many concurrent connections is with ApacheBench:

$ ab -n 20000 -c 200 "127.0.0.1:8000/inc?name=i"

Benchmarking 127.0.0.1 (be patient)
Completed 2000 requests
Completed 4000 requests

Test aborted after 10 failures

apr_socket_connect(): Connection reset by peer (104)
Total of 4622 requests completed

Oops... what happened? Checking the logs of the Go server, we'll see something like:

<normal server logs>
fatal error: concurrent map writes

goroutine 6118 [running]:
runtime.throw(0x6b0a5c, 0x15)
  /usr/local/go/src/runtime/panic.go:608 +0x72 fp=0xc00060dba8 sp=0xc00060db78 pc=0x42ba12

Reviewing our code, the problem is apparent. The request handlers can run concurrently but they all manipulate a shared CounterStore. For example, the inc handler is being called concurrently for multiple requests and attempts to mutate the map. This leads to a race condition since in Go, map operations are not atomic. Luckily, the Go runtime detects this and dies with a helpful message; it would be worse if data were silently corrupted.

The simplest solution is to serialize all map accesses using a mutex. Here's an excerpt from a complete code sample that implements this:

type CounterStore struct {
  sync.Mutex
  counters map[string]int
}

func (cs *CounterStore) inc(w http.ResponseWriter, req *http.Request) {
  log.Printf("inc %v", req)
  cs.Lock()
  defer cs.Unlock()
  name := req.URL.Query().Get("name")
  if _, ok := cs.counters[name]; ok {
    cs.counters[name]++
    fmt.Fprintf(w, "ok\n")
  } else {
    fmt.Fprintf(w, "%s not found\n", name)
  }
}

Note two changes:

  1. We embed a sync.Mutex in CounterStore, and each handler starts by locking the mutex (and deferring an unlock).
  2. We change the receiver inc is defined on to a pointer *CounterStore. In fact, the previous version of the code was wrong in this respect - methods that modify data should always be defined with pointer receivers. We got lucky that the data was shared at all with value receivers because maps are reference types. Pointer receivers are particularly critical when mutexes are involved.

If we rerun the ab benchmark with this fixed server, it passes; the race condition is gone.

Synchronizing with channels vs. mutexes

To programmers experienced in most other languages, adding a mutex to synchronize accesses to a CounterStore is a natural solution. One of the mottos of Go is, however, "Share memory by communicating, don't communicate by sharing memory". Does this apply here?

Instead of mutexes, we could use channels to synchronize access to shared data. This code sample reimplements the mutex-using server to use channels instead. We start by defining a "counter manager" which is a background goroutine with access to a closure that stores the actual data:

type CommandType int

const (
  GetCommand = iota
  SetCommand
  IncCommand
)

type Command struct {
  ty        CommandType
  name      string
  val       int
  replyChan chan int
}

func startCounterManager(initvals map[string]int) chan<- Command {
  counters := make(map[string]int)
  for k, v := range initvals {
    counters[k] = v
  }

  cmds := make(chan Command)

  go func() {
    for cmd := range cmds {
      switch cmd.ty {
      case GetCommand:
        if val, ok := counters[cmd.name]; ok {
          cmd.replyChan <- val
        } else {
          cmd.replyChan <- -1
        }
      case SetCommand:
        counters[cmd.name] = cmd.val
        cmd.replyChan <- cmd.val
      case IncCommand:
        if _, ok := counters[cmd.name]; ok {
          counters[cmd.name]++
          cmd.replyChan <- counters[cmd.name]
        } else {
          cmd.replyChan <- -1
        }
      default:
        log.Fatal("unknown command type", cmd.ty)
      }
    }
  }()
  return cmds
}

Instead of accessing the map of counters directly, handlers will send Commands on a channel and will receive replies on a reply channel they provide.

The shared object for the handlers will now be a Server:

type Server struct {
  cmds chan<- Command
}

And here's the inc handler:

func (s *Server) inc(w http.ResponseWriter, req *http.Request) {
  log.Printf("inc %v", req)
  name := req.URL.Query().Get("name")
  replyChan := make(chan int)
  s.cmds <- Command{ty: IncCommand, name: name, replyChan: replyChan}

  reply := <-replyChan
  if reply >= 0 {
    fmt.Fprintf(w, "ok\n")
  } else {
    fmt.Fprintf(w, "%s not found\n", name)
  }
}

Each handler deals with the manager synchronously; the Command send is blocking, and so is the read from the reply channel. But note - not a mutex in sight! Mutual exclusion is accomplished by a single goroutine having access to the actual data.

While it certainly looks like an interesting technique, for our particular use case this approach seems like an overkill. In fact, overuse of channels is one of the common gotchas for Go beginners. Quoting from the Go Wiki entry:

Most locking issues can be solved using either channels or traditional locks.

So which should you use?

Use whichever is most expressive and/or most simple.

It then goes on to suggest that mutexes are preferable for protecting shared state. I tend to agree, since a mutex feels more natural here.

Limiting the degree of concurrency for our server

In addition to synchronization, another aspect of concurrency we have to worry about is overloading of the server. Imagine exposing a server to the internet - without any safeguards it would be fairly easy to bring it down with a denial of service (DoS) attack. That could happend even unintentionally, if we didn't provision the proper computing power behind the service. In these cases failure is unavoidable but should be graceful.

It's very easy to do these things in Go, and there are many strategies. One of the simplest is rate limiting, which can mean either restricting the number of simultaneous connections, or restricting the number of connections per unit of time. For the former, we can add some middleware (full code here):

// limitNumClients is HTTP handling middleware that ensures no more than
// maxClients requests are passed concurrently to the given handler f.
func limitNumClients(f http.HandlerFunc, maxClients int) http.HandlerFunc {
  // Counting semaphore using a buffered channel
  sema := make(chan struct{}, maxClients)

  return func(w http.ResponseWriter, req *http.Request) {
    sema <- struct{}{}
    defer func() { <-sema }()
    f(w, req)
  }
}

And wrap a handler using it:

// Limit to max 10 connections for this handler.
http.HandleFunc("/inc", limitNumClients(store.inc, 10))

This ensures that no more than 10 simultaneous clients will be allowed to invoke the inc handler. It should be simple to extend this approach to share a single limiting channel between multiple handlers. Or we could use a more heavy-handed solution and limit the number of simulatenous connections at the listener level, using something like netutil.LimitListener.

An alternative approach to rate limiting is time-based. Instead of saying "no more than 10 requests at a time", we can say "no more than one request in a 50 millisecond period". Go provides the convenient time.Tick channel that makes it easy to implement, as this Go example demonstrates. Adjusting this approach to our sample server is left as an exercise to the reader.

Appendix: where net.http goes concurrent

This is a brief technical appendix, for folks interested following through the code of Go's net/http package to see where the concurrency is actually implemented.

The relevant source code for the serving part of net/http is in https://golang.org/src/net/http/server.go

ListenAndServe invokes Serve, which calls Accept in a loop. Accept is similar to the socket accept syscall, creating a new connection whenever a new client is accepted; the connection is then used for interacting with the client. This connection is type conn in the server.go file, a private type with server state for each client connection. Its serve method actually serves a connection, pretty much as you would expect; it reads some data from the client and invokes the user-suppied handler depending on the path.

http.Server.Serve calls conn.serve as:

go c.serve(ctx)

Wherein lies the concurrency. From this point on, a separate goroutine handles this connection. This is why multiple goroutines could be executing user-specified handlers (including a single handler) at any given time.


Summary of reading: October - December 2018



  • "Wonder" by R.J. Palacio - school-age story about a boy with severe facial deformities who started going to 5th grade after being home-schooled earlier in life. A bit of bullying, lots of kindness, and interesting insights into the minds of 10-year-olds.
  • "The Story of Human Language" by John McWhorter (audio course) - comprehensive introduction to linguistics and human languages. I've read Prof. McWhorter's books before (e.g. "Power of Babel"), but listening to it is a different experience. There's a lot of nuance in pronunciation that's hard to convey on a printed page. Very enjoyable course overall, highly recommended for folks interested in this topic.
  • "Space - The whole whizz-bang story" by Glenn Murphy - a gentle introduction to astronomy and the planets of the solar system, mostly aimed at kids and teens, though I think adults can enjoy the book too. Funny cartoons make the book light-hearted, but it actually provides a balanced account of the topic and doesn't dumb things down too much. Great book to read together with kids.
  • "Bad Blood: Secrets and Lies in a Silicon Valley Startup" by John Carreyrou - by the journalist who exposed the fraud and lies conducted by Elizabeth Holmes and her associates in Theranos. This is an expanded account of Carreyrou's series of WSJ articles in 2015. Excellent book, a truly mesmerizing read. To anyone who've ever observed vaporware being spun, Theranos is the culmination. It's also a great example of where cult of personality can lead people. You would expect that experienced statesmen like George Shultz and Henry Kissinger would be able to see through the reality distortion field; that they didn't is a poignant reminder of the power of charisma and salesmanship.
  • "Lilac Girls" by Martha Hall Kelly - intertwined stories of three women during and after WWII - a Polish prisoner in Ravensbruck (a women's concentration camp), a German doctor who performed surgery experiments on her, and an American activist. Pretty good book overall, with some chilling accounts of what went on in Ravensbruck. The author tries to do a bit too much, though; IMHO the biographic accounts of Caroline in the first part of the book, and Paul specifically, don't mix well with the rest of the narrative.
  • "Code Girls: The Untold Story of the American Women Code Breakers of World War II" by Liza Mundy - an interesting account of the US WWII code-breaking efforts vs. the Japanese and Germans, told from the vantage point of women who participated in the effort. It's an interesting parable of why universal education is so important to tap the full resources of a population. America is really lucky that at least some mathematically and linguistically inclined women managed to get to universities before the war, even though it was largely discouraged.
  • "It doesn't have to be crazy at work" by Jason Fried and David Heinemeier Hansson - the founders of Basecamp (née 37 signals) lay out their management philosophy. Definitely interesting short read, even though it's quite likely a marketing/hiring pitch. I agree with much of what the authors say in this book, and especially enjoyed the ruminations about what seeking continual revenue growth turns companies into. I've been thinking about this too, for a long time now. It seems like a unique privilege of privately-owned firms to do "the right thing" in this case. It's also interesting to ponder a futuristic utopia where most jobs are high-skill, working fewer hours (to better distribute the jobs across the population) and mostly remotely (removing a lot of "rush hour economics"); companies like Basecamp appear to be the early beacons of this phenomenon.
  • "Darwin" by Adrian Desmond and James Moore - an extremely thorough biography by Charles Darwin. Very long and dense book, but the writing is good so it's not too taxing to plow through.
  • "Types and Programming Languages" by Benjamin Pierce - the bible of static typing, focusing on ML-like languages and developing typing theory all the way from trivially typed lambda calculus to higher-order types / kinds and varieties of polymorphism. The book is heavy on theory, trying to prove many properties on such type systems. It comes with a comprehensive set of implementations in OCaml, though it takes quite a bit of work to relate these implementations to the text.
  • "Naked Money" by Charles Wheelan - a really good explanation of how money works, with in-depth discussions of inflation/deflation, the 2008 crisis, Japan, Euro and the U.S.-China trade situation.
  • "The Magic of Reality" by Richard Dawkins - a delightful little book that's aimed at exciting younger people about science. It covers a bunch of topics ranging from evolution, discovering life on other planets, earthquakes to rainbows and doesn't dumb anything down. It's probably more suitable for teens at the earliest, since it does discuss a few topics that would be uncomfortable for younger kids. That said, this was a fun read even for me.
  • "The Kubernetes Book" by Nigel Poulton - a short and dense introduction to Kubernetes; easy to go through the book in a couple of hours and get a good sense of what Kubernetes is and how to use it. The book explains the basics well, but suffers from a very common issue with books of this kind - too much time spent on technical minutia, too little spent on motivation; why would I want to do this, what are the alternatives, etc. I don't think the book has a single real-life example - all apps are synthetic and imaginary; it's not even clear how much real-life production system experience the author has (as opposed to teaching courses and writing books). In many ways, such books are just dressed-up documentation. Not bad per se, just something to keep in mind for to tune your expectations if you plan to read it.
  • "Hawaii" by James Michener - another epic by Michener, perhaps one of his most famous ones. Tells the story of Hawaii through the peoples that settled it, starting with the Polynesians a thousand years ago, and through shortly after WWII. Well written book, and the author's love and appreciation of Hawaii and its people shows all through.
  • "Little House on the Prairie" by Laura Ingalls Wilder - the third, and probably the most famous book in the series. Here Laura and her family move to settle in a new territory in Kansas. As usual, I'm mostly impressed by the feats of independence and self-reliance these folks demonstrated.
  • "The Phoenix Project" by Gene Kim et. al. - a novel-formatted story of modern IT and DevOps in an old-style company transitioning to the new know-how. As a developer I found the book very interesting, getting a rare glimpse "from the other side of the field". The novel plot is very forced and unnatural, but I guess that's necessary to bring the points across.

Re-reads:

  • "Crypto" by Steven Levy

Beware of copying mutexes in Go



Suppose we have a struct that contains a map, and we want to modify the map in a method. Here's a simple example:

package main

import "fmt"

type Container struct {
  counters map[string]int
}

func (c Container) inc(name string) {
  c.counters[name]++
}

func main() {
  c := Container{counters: map[string]int{"a": 0, "b": 0}}

  doIncrement := func(name string, n int) {
    for i := 0; i < n; i++ {
      c.inc(name)
    }
  }

  doIncrement("a", 100000)

  fmt.Println(c.counters)
}

A Container holds a map of counters, keyed by name. Its inc method increments the specified counter (let's assume that the counter already exists). main calls inc many times in a loop.

If we run this snippet, it will print out:

map[a:100000 b:0]

Now say that we want two goroutines to call inc concurrently. Since we are wary of race conditions, we'll use a Mutex to lock around the critical region:

package main

import (
  "fmt"
  "sync"
  "time"
)

type Container struct {
  sync.Mutex                       // <-- Added a mutex
  counters map[string]int
}

func (c Container) inc(name string) {
  c.Lock()                         // <-- Added locking of the mutex
  defer c.Unlock()
  c.counters[name]++
}

func main() {
  c := Container{counters: map[string]int{"a": 0, "b": 0}}

  doIncrement := func(name string, n int) {
    for i := 0; i < n; i++ {
      c.inc(name)
    }
  }

  go doIncrement("a", 100000)
  go doIncrement("a", 100000)

  // Wait a bit for the goroutines to finish
  time.Sleep(300 * time.Millisecond)
  fmt.Println(c.counters)
}

What would you expect the output to be? I get something like this:

fatal error: concurrent map writes

goroutine 5 [running]:
runtime.throw(0x4b765b, 0x15)

<...> more goroutine stacks
exit status 2

We were careful to use a mutex, so what went wrong? Can you see how to fix it? Hint: it's a single-character code change!

The problem with the code is that whenever inc is called, our container c is copied into it, because inc is defined on Container, not *Container; in other words, it's a value receiver, not a pointer receiver. Therefore, inc can't really modify the contents of c per se.

But wait, how did the original sample work then? In the single-goroutine sample, we passed c by value too, but it worked - main observed the changes to the map done by inc. This is because maps are special - they are reference types, not value types. What's stored in Container is not the actual map data, but a pointer to it. So even when we create a copy of the Container, its counters member still contains the address of the same data.

So the original code sample is wrong too. Even though it works, it goes against the guidelines; methods that modify the object should be defined on pointers, not values. Using a map here leads us to a false sense of security. As an exercise, try to replace the map with just a single int counter in the original example, and notice how inc increments a copy of it, so that in main its effects will not be seen.

The Mutex is a value type (see definition in Go's source, including the comment that explicitly asks not to copy mutexes), so copying it is wrong. We're just creating a different mutex, so obviously the exclusion no longer works.

The one-character fix is, therefore, to add a * in front of Container in the definition of inc:

func (c *Container) inc(name string) {
  c.Lock()
  defer c.Unlock()
  c.counters[name]++
}

Then c is passed by pointer into the method, and actually refers to the same instance of Container in memory as the one the caller has.

This is not an uncommon problem! In fact, go vet will warn about it:

$ go tool vet method-mutex-value-receiver.go
method-mutex-value-receiver.go:19: inc passes lock by value: main.Container

It often comes up in scenarios like HTTP handlers, which are invoked concurrently without the programmer's explicitly writing any go statements. I'll write more about this in a future post.

This issue really helps clarify the difference between value and pointer receivers in Go, in my opinion. To drive the point home, here's another code sample, unrelated to the last two. It leverages Go's ability to create pointers to objects using & and examine their addresses with the %p formatting directive:

package main

import "fmt"

type Container struct {
  i int
  s string
}

func (c Container) byValMethod() {
  fmt.Printf("byValMethod got &c=%p, &(c.s)=%p\n", &c, &(c.s))
}

func (c *Container) byPtrMethod() {
  fmt.Printf("byPtrMethod got &c=%p, &(c.s)=%p\n", c, &(c.s))
}

func main() {
  var c Container
  fmt.Printf("in main &c=%p, &(c.s)=%p\n", &c, &(c.s))

  c.byValMethod()
  c.byPtrMethod()
}

Its output is (in one particular run on my machine - for you the addresses may be different, though the relations between them should be the same):

in main &c=0xc00000a060, &(c.s)=0xc00000a068
byValMethod got &c=0xc00000a080, &(c.s)=0xc00000a088
byPtrMethod got &c=0xc00000a060, &(c.s)=0xc00000a068

The main function creates a Container and prints out its address and the address of field s. It then invokes two Container methods.

byValMethod has a value receiver, and it prints out different addresses because it gets a copy of c. On the other hand, byPtrMethod has a pointer receiver and the addresses it observes are identical to the ones in main, because it takes the address of the actual c when invoked, not a copy.