Does a concrete type implement an interface in Go?



A very common question that comes up in Go forums is "how to I check that some type implements a certain interface?". A common immediate reaction is that the question makes no sense, because Go is statically typed, so the compiler already knows whether a type implements an interface or not. But it turns out that the question in the general sense is more nuanced, and it's worth spending some time understanding the variations folks are usually interested in.

Let's start with a basic example. Assume this Munger interface that has a single Munge method:

type Munger interface {
    Munge(int)
}

And say that we have a type that doesn't have a Munge method; for example, int. If we try to do this:

var i int
var mm Munger = i

The compiler will rightfully complain:

cannot use i (type int) as type Munger in assignment:
    int does not implement Munger (missing Munge method)

The same happens if you attempt to pass i into a function that expects a Munger, and so on.

So the gist of the common response is: you don't need to perform this check, the compiler will do it for you on first use.

A more deliberate compile-time check

It's sometimes useful to have a more conscious compile-time check that a certain type implements a certain interface. Think something similar to a static_assert in C++. A nice trick to accomplish this is:

var _ Munger = (*Foo)(nil)

This statement checks if Foo implements Munger at compile time [1]. It can be placed on the top-level (outside functions) in a .go file, and won't generate any executable code in case the check passes. There are plenty of examples of this pattern in the Go ecosystem; here's one from the standard library (io/multi.go):

var _ StringWriter = (*multiWriter)(nil)

After defining the multiWriter struct, the code checks if it implements the StringWriter interface. This can be useful to ensure that we get a clear compile error in an expected place if the interface changes in some way.

The trick here is the usage of nil to create a typed value for the compiler to see without declaring any explicit vars.

Run-time check

A more common request is checking whether a type implements an interface at run-time. Note that this is semantically different from what the compiler does for us - we actually want to make a run-time decision based on what interfaces a given type implements. This can be useful in many scenarios, such as testing, plugins etc.

At first it seems easy - Go has good support for type assertions, after all. But there's a problem. Suppose we have the Munger interface again, and some type Foo; we want to check whether Foo implements Munger. A type assertion would go like this:

var f Foo
_, ok := f.(Munger)

But the compiler complains:

invalid type assertion: f.(Munger) (non-interface type Foo on left)

This is because only values of interface types are allowed on the left-hand-side of a type assertion. So what can we do? Well, if the compiler wants an interface, we can give it an interface:

var f Foo
_, ok := interface{}(f).(Munger)

We start by converting f to the empty interface type. This conversion is always successful because all types implement the empty interface. Now the check will work and will return true if Foo implements Munger, and false otherwise.

Run-time check using reflection

Another way to accomplish the run-time check is using reflection:

var f Foo

iMunger := reflect.TypeOf((*Munger)(nil)).Elem()
// ... now iMunger is a reflect.Type representing Munger

ok := reflect.TypeOf(&f).Implements(iMunger)

The reflect machinery has access to the underlying implementation of Go objects and types at run-time, and it uses this information to answer queries such as "does this type implement that interface".

Note that this technique also uses the trick shown in the "deliberate compile-time check section", wherein a nil pointer is given a type in order to avoid a temporary object.


[1]More precisely, it checks if *Foo implements Munger. But due to the way Go automatically dereferences if needed, this must be true if Foo implements Munger.

GitHub webhook payload as a cloud function



Back in 2014, I wrote a post describing a simple payload server for GitHub webhooks, using Python 3. That server could be deployed to any VPS listening on a custom port.

Now it's 2019, and deploying servers to VPSs doesn't make me feel hip enough. All the cool kids are into serverless now, so I decided to rewrite the same payload server in Go and deploy it as a Google Cloud Function. This brief post can serve as a basic tutorial on how to do it.

I assume you already have a GCP account (there's a free tier), and the gcloud command-line tool is configured to authenticate with your account and project name.

You can see the full code here, but this is the important part:

func Payload(w http.ResponseWriter, r *http.Request) {
  body, err := ioutil.ReadAll(r.Body)
  if err != nil {
    log.Println(err)
    return
  }
  log.Println("Header:\n---------")
  fmt.Println(r.Header)

  if !validateSignature(body, r) {
    m := "Signature validation failed"
    log.Println(m)
    w.Write([]byte(m))
    return
  }

  fmt.Println("Body:\n---------")
  log.Println(string(body))
}

Payload is a standard http.HandlerFunc, and its signature should be familiar to anyone who has written HTTP servers in Go. GitHub sends its payload as a POST request (hence our reading from r.Body) with some special headers for validation. The validation code runs a SHA1 HMAC to ensure that GitHub knows a secret key shared with the application (this helps keep intruders away from your payload server).

The full code sample has this validation code, as well as a simple unit test for Payload. It doesn't actually attempt to create a properly signed message, but checks that Payload is alive and returns a valid HTTP response. In general, it is highly recommended to unit-test these handlers locally, because cloud function deployment takes many seconds and isn't very convenient for short edit-test cycles.

To deploy this function, we'll go to the directory where payloadserver.go lives, and run:

$ gcloud functions deploy payloadserver \
      --entry-point Payload \
      --runtime go111 \
      --trigger-http \
      --set-env-vars HOOK_SECRET_KEY=<your secret key>

This prints out a URL for your function; it looks something like:

httpsTrigger:
  url: https://<region>-<project-name>.cloudfunctions.net/payloadserver

Which is what you'll point the GitHub webhook to. Configure the webhook to send all events, and then test it by creating or modifying some issue in your repository. The webhook management page on GitHub should now show the event in "Recent deliveries". You can also check the logs of your cloud function, either from the GCP control panel, or from the command-line:

$ gcloud functions logs read payloadserver

If everything ran successfully, you'll see the headers and the body of the payload emitted to the log. That's it!

Jokes about hipness aside, once you get the hang of it cloud functions seem like a particularly easy way to deploy simple web servers and apps for specific needs. There's a lot of powerful functionality behind the simple facade - for example, resources will automatically scale with the load. That said, be aware of the costs if you're doing it for anything high-volume.


Unix domain sockets in Go



When it comes to inter-process communication (IPC) between processes on the same Linux host, there are multiple options: FIFOs, pipes, shared memory, sockets and so on. One of the most interesting options is Unix Domain Sockets that combine the convenient API of sockets with the higher performance of the other single-host methods.

This post demonstrates some basic examples of using Unix domain sockets with Go and explores some benchmarks comparing them to TCP loop-back sockets.

Unix domain sockets (UDS)

Unix domain sockets (UDS) have a long history, going back to the original BSD socket specification in the 1980s. The Wikipedia definition is:

A Unix domain socket or IPC socket (inter-process communication socket) is a data communications endpoint for exchanging data between processes executing on the same host operating system.

UDS support streams (TCP equivalent) and datagrams (UDP equivalent); this post focuses on the stream APIs.

IPC with UDS looks very similar to IPC with regular TCP sockets using the loop-back interface (localhost or 127.0.0.1), but there is a key difference: performance. While the TCP loop-back interface can skip some of the complexities of the full TCP/IP network stack, it retains many others (ACKs, TCP flow control, and so on). These complexities are designed for reliable cross-machine communication, but on a single host they're an unnecessary burden. This post will explore some of the performance advantages of UDS.

There are some additional differences. For example, since UDS use paths in the filesystem as their addresses, we can use directory and file permissions to control access to sockets, simplifying authentication. I won't list all the differences here; for more information feel free to check out the Wikipedia link and additional resources like Beej's UNIX IPC guide.

The big disadvantage of UDS compared to TCP sockets is the single-host restriction, of course. For code written to use TCP sockets we only have to change the address from local to remote and everything keeps working. That said, the performance advantages of UDS are significant enough, and the API is similar enough to TCP sockets that it's quite possible to write code that supports both (UDS on a single host, TCP for remote IPC) with very little difficulty.

Using Unix domain sockets in Go

Let's start with a basic example of a server in Go that listens on a UNIX domain socket:

const SockAddr = "/tmp/echo.sock"

func echoServer(c net.Conn) {
    log.Printf("Client connected [%s]", c.RemoteAddr().Network())
    io.Copy(c, c)
    c.Close()
}

func main() {
    if err := os.RemoveAll(SockAddr); err != nil {
        log.Fatal(err)
    }

    l, err := net.Listen("unix", SockAddr)
    if err != nil {
        log.Fatal("listen error:", err)
    }
    defer l.Close()

    for {
        // Accept new connections, dispatching them to echoServer
        // in a goroutine.
        conn, err := l.Accept()
        if err != nil {
            log.Fatal("accept error:", err)
        }

        go echoServer(conn)
    }
}

UDS are identified with paths in the file system; for our server here we use /tmp/echo.sock. The server begins by removing this file if it exists, what is that about?

When servers shut down, the file representing the socket can remain in the file system unless the server did orderly cleanup after itself. If we re-run another server with the same socket path, we may get the error:

$ go run simple-echo-server.go
2019/02/08 05:41:33 listen error:listen unix /tmp/echo.sock: bind: address already in use

To prevent that, the server begins by removing the socket file, if it exists [1].

Now that the server is running, we can interact with it using Netcat, which can be asked to connect to UDS with the -U flag:

$ nc -U /tmp/echo.sock

Whatever you type in, the server will echo back. Press ^D to terminate the session. Alternatively, we can write a simple client in Go that connects to the server, sends it a message, waits for a response and exits. The full code for the client is here, but the important part is the connection:

c, err := net.Dial("unix", "/tmp/echo.sock")

We can see that writing UDS servers and clients is very similar to writing regular socket servers and clients. The only difference is having to pass "unix" as the network parameter of net.Listen and net.Dial; the rest of the code remains the same. Obviously, this makes it very easy to write generic server and client code that's independent of the actual kind of socket it's using.

HTTP and RPC protocols over UDS

Network protocols compose by design. High-level protocols, such as HTTP and various forms of RPC, don't particularly care about how the lower levels of the stack are implemented as long as certain guarantees are maintained.

Go's standard library comes with a small and useful rpc package that makes it trivial to throw together quick RPC servers and clients. Here's a simple server that has a single procedure defined:

const SockAddr = "/tmp/rpc.sock"

type Greeter struct {
}

func (g Greeter) Greet(name *string, reply *string) error {
    *reply = "Hello, " + *name
    return nil
}

func main() {
    if err := os.RemoveAll(SockAddr); err != nil {
        log.Fatal(err)
    }

    greeter := new(Greeter)
    rpc.Register(greeter)
    rpc.HandleHTTP()
    l, e := net.Listen("unix", SockAddr)
    if e != nil {
        log.Fatal("listen error:", e)
    }
    fmt.Println("Serving...")
    http.Serve(l, nil)
}

Note that we use the HTTP version of the server. It registers a HTTP handler with the http package, and the actual serving is done with the standard http.Serve. The network stack here looks something like this:

RPC / HTTP / Unix domain socket stack

An RPC client that can connect to the server shown above is available here. It uses the standard rpc.Client.Call method to connect to the server.

Benchmarking UDS compared to loop-back TCP sockets

Note: benchmarking is hard, so please take these results with a grain of salt. There's some more information on benchmarking different socket types on the Redis benchmarks page and in this paper, as well as many other resources online. I also found this set of benchmarks (written in C) instructive.

I'm running two kinds of benchmarks: one for latency, and one for throughput.

For latency, the full code of the benchmark is here. Run it with -help to see what the flags are, and the code should be very straightforward to grok. The idea is to ping-pong a small packet of data (128 bytes by default) between a server and a client. The client measures how long it takes to send one such message and receive one back, and takes that combined time as "twice the latency", averaging it over many messages.

On my machine, I see average latency of ~3.6 microseconds for TCP loop-back sockets, and ~2.3 microseconds for UDS.

The throughput/bandwidth benchmark is conceptually simpler than the latency benchmark. The server listens on a socket and grabs all the data it can get (and discards it). The client sends large packets (hundreds of KB or more) and measures how long each packet takes to send; the send is done synchronously and the client expects the whole message to be sent in a single call, so it's a good approximation of bandwidth if the packet size is large enough.

Obviously, the throughput measurement is more representative with larger messages. I tried increasing them until the throughput improvements tapered off.

For smaller packet sizes, I see UDS winning over TCP: 10 GB/sec compared to 9.4 GB/sec for 512K. For much larger packet sizes 16-32 MB, the difference becomes negligible (both taper off at about 13 GB/sec). Interestingly, for some packet sizes (like 64K), TCP sockets are winning on my machine.

For very small message sizes we're getting back to latency-dominated performance, so UDS is considerably faster (more than 2x the number of packets per second compared to TCP). In most cases I'd say that the latency measurements are more important - they're more applicable to things like RPC servers and databases. In some cases like streaming video or other "big data" over sockets, you may want to pick the packet sizes carefully to optimize the performance for the specific machine you're using.

This discussion has some really insightful information about why we should expect UDS to be faster. However, beware - it's from 2005 and much in Linux has changed since then.

Unix domain sockets in the real-world Go projects

I was curious to see if UDS are actually used in real-world Go projects. They sure are! A few minutes of browsing/searching Github quickly uncovered UDS servers in many components of the new Go-dominated cloud infrastructure: runc, moby (Docker), k8s, lstio - pretty much every project I looked at.

That makes sense - as the benchmarks demonstrate, there are significant performance advantages to using a UDS when the client and server are both on the same host. And the API of UDS and TCP sockets is so similar that the cost of supporting both interchangeably is quite small.


[1]For internet-domain socket, the same issue exists with ports that are marked taken by processes that die without cleanup. The SO_REUSEADDR socket option exists to address this problem.