Eli Bendersky's website - Lexer

Performance of coroutine-style lexers in Go

2022-06-06T20:32:00-07:00

Back in 2011, Rob Pike gave a talk on the topic of Lexical Scanning in Go, where he presented an interesting approach to lexical analysis with coroutines (implemented via goroutines communicating over a channel). Since I've been pondering the connection between coroutines and state machines in the past, Rob's talk inspired me to try approximating this approach in Python using sub-generators.

Since 2011, I've seen this talk and the technique presented in it mentioned many times, both in Go forums and in general programming communities. There's something in this approach that feels elegant - it's a problem very well suited for coroutines. However, there was always a small nagging voice in the back of my head doubting the efficiency of the approach.

Since I've recently found myself playing with lexers again, this seemed like a good opportunity to revisit Rob Pike's lexer and compare its performance to other approaches, using the same problem and benchmark for fairness.

Rob Pike's original lexer

In the talk, Rob is describing a lexer he designed for Go's templating package. The lexer presented in the talk and slides is relatively simple; a much more featureful version of it still lives in the text/template package - lex.go. As such, this lexer is heavily used in production every day.

I've transcribed the original lexer from the talk into my GitHub repository; it's available here, with tests.

The main aesthetic appeal of this lexer is avoiding an explicit state machine by using a separate goroutine to perform the lexing. This goroutine switches states by returning a new "state function", and emits tokens onto a channel which can be read by the lexer's clients.

This approach is particularly attractive when parsing templates because it oscillates between two major states - lexing free-standing text and lexing inside actions delimited by {{ }}. Using the concurrent approach avoids the need to have an explicit "am I inside an action" state flag that has to be checked every time a new token is requested [1].

Lexing TableGen

To be able to have a meaningful performance comparison, I've rewritten my TableGen lexer once again, this time using the coroutine approach. The full code for this lexer with tests is available here.

The API is very similar to my previous TableGen lexers - all the implementation details (like having a channel to read tokens from) are hidden from the user. The token type is the same:

type Token struct {
  Name TokenName
  Val  string
  Pos  int
}

The NextToken method also has the same signature, though the implementation uses a channel now:

func (l *Lexer) NextToken() Token {
  return <-l.tokens
}

The constructor creates a new Lexer, creates a channel for the emitted tokens to go into and launches the goroutine that does the actual lexing:

// Lex creates a new Lexer
func Lex(input string) *Lexer {
  l := &Lexer{
    input:  input,
    tokens: make(chan Token),
  }
  go l.run()
  return l
}

The run method serves as a trampoline to advance the lexer from state to state (while the state functions emit tokens into the channel):

type stateFn func(*Lexer) stateFn

func (l *Lexer) run() {
  for state := lexText; state != nil; {
    state = state(l)
  }
  close(l.tokens) // no more tokens will be delivered
}

And so on. The implementation follows Rob Pike's lexer very closely, with the same primitives. For the TableGen language, which does not have the "two states" feature of templates, I found this approach to be less of a stylistic win, though it still makes state management simpler.

Performance

In the previous post, the fastest Go lexer achieved with Go 1.18 runs the benchmark in about 6 ms (with a GOGC=1600 setting).

For a level playing field, I ran the new coroutine-style lexer on the same input file, with the same benchmarking invocation and the same GOGC setting:

$ GOGC=1600 TDINPUT=<path to input file> go test -bench=Preall -benchtime=5s
goos: linux
goarch: amd64
pkg: example.com
cpu: Intel(R) Core(TM) i7-4771 CPU @ 3.50GHz
BenchmarkLexerPrealloc-8            80          70507009 ns/op
PASS
ok    example.com     5.885s

It takes... 70 ms, more than 10x slower!

While I'm not surprised that this approach is slower, I am somewhat surprised at the magnitude. Let's think this through. What does each lexer do in its hot inner loop?

In my original lexer, each call to the NextLexer function:

Skips whitespace: iterates over the input string rune by rune until a non-whitespace rune is encountered.
Examines the current rune and decides which kind of token it belongs to.
Finishes lexing the token and returns it as a string slice.

Whereas in the coroutine-style lexer, each call to NextLexer:

Invokes a channel receive on the token channel.

In the meantime, the lexing goroutine:

Skips whitespace and examines the current rune, just like in the regular lexer.
Invokes a channel send on the token channel.

The channel send/receive is the main culprit for the large performance difference. Channels in Go are fully synchronized, which implies locking inside the inner loop. Moreover, since there are two goroutines involved that perform channel operations, the Go runtime has much more work to do to handle suspending goroutines when the channel blocks and waking them up when it unblocks. All these operations are highly optimized in Go, but when they appear in the inner loop of a fast scanning process, the relative cost is high.

If we profile the new scanner with pprof, this cost is easily observed:

In the previous lexer, the "fetch next rune" method is very dominant. Here it accounts for only 5.8% of the execution time! Much more time goes on chansend1 and chanrecv1 which are runtime functions with names that should be self-describing. There is also goroutine management code in the runtime that accounts for a large % there.

Using a buffered channel

Go's make primitive creates an unbuffeered channel by default, meaning that every send into it blocks until a receive takes the item out. What would happen if we created a buffered channel instead? Theoretically, this should improve the lexer's execution time as there will be less suspension and waking up of goroutines.

Let's see what different values of the buffer give us; I re-ran the benchmark with buffer sizes starting from 1 to 16384 in jumps of 4x:

As expected, using a buffered channel makes lexing significantly faster, leveling out at 1024 where it takes about 24 ms for our benchmark. This is a large improvement, though still much slower than the 6 ms we had with our non-concurrent lexer.

Channels have many uses in Go; sometimes they are used as synchronization mechanisms, so having a large buffer is not always feasible. In cases like our lexer, a buffer actually makes sense because there should be no problem for the lexing goroutine to run ahead a little bit. Note that this doesn't work for any input kind, though; had we been lexing C, for instance, we'd might want to have a feedback mechanism back into the lexer (for handling the grammar's context sensitivity).

FWIW, the template lexer Rob Pike added to the standard library uses an unbuffered channel. Maybe it would make sense to add a modest buffer there to speed it up somewhat :-) See also go issue #53261.

Does performance matter here?

For the task at hand, the coroutine-style lexer is still plenty fast. Note that it's much faster than some of the Python and JS-based lexers I wrote for the same task a while ago.

This lexer is used by the standard library for parsing templates, but these are (1) rarely very big and (2) almost always OK to parse just once during the lifetime of a program, so the time it takes to parse them is not too important; in other words, it's very unlikely to dominate your application's benchmarks.

That said, I can envision scenarios where this could matter. Suppose you're writing a frontend for a nontrivial programming language (or configuration language etc.) and a fast interpreter for this language [2]. This could lead to the frontend's performance being a bottleneck. In these scenarios, it just may be important to have the fastest lexer you can reasonably implement.

[1] I do wonder if this would work in more complicated cases though. In templates, the kinds of tokens that appear inside actions can never appear outside them. But suppose they could; suppose there would be some token TOK which could legally appear both in "text mode" and in "action mode". What would the state function representing TOK return when it's done parsing it? How would it know which mode it has to go back to? It's possible that some sort of explicit state variable would be unavoidable in this scenario.

[2]	I'm saying interpreter on purpose, because interpreter backends tend to be very simple and quick. In full compilers, backends typically run many expensive optimizations that dominate the compile time.

Rewriting the lexer benchmark in Rust

2022-05-30T05:53:00-07:00

I've reimplemented my lexical analyzer (lexer) for the the TableGen language in Python, JS and Go so far. The latest post in the series discussed how several years of new Go versions improved my lexer's performance by roughly 2x, and how several additional optimizations won another 37%; the final result is a lexer that churns through 1 MiB of source code in just 5.6 milliseconds.

Since I've also been playing with Rust recently, I thought it would be interesting to re-implement this lexer once again, this time in Rust. Rust is certainly the lowest-level language among those I've used so far for this task, so I expect to see some top-notch performance.

The full code for this post is available on GitHub.

Designing an API

I find that Rust's strict ownership rules makes one think carefully about API design from very early on. If we want to create a lexer and pass it a string as input - who owns the string? In Rust, the answer to this question is not an implicit contract (like it always is in C and sometimes in C++), and it cannot be deferred to the runtime either (like one would do in Python, JS or Go). The answer has to be explicitly encoded into the types of the program.

Since one of the goals of this series of posts is performance, I decided to go with a zero-copy API at first, where the user of the lexer owns the input string; the lexer, in turn, returns tokens that contain references into this input string - so the user ends up owning these too. Rust's lifetime specifiers make this fairly natural; here's the type:

pub struct Lexer<'source> {
  input: &'source str,
  iter: Peekable<CharIndices<'source>>,

  // c is the last char taken from iter, and ci is its offset in the input.
  c: char,
  ci: usize,

  // error is true iff the lexer encountered and error.
  error: bool,
}

Ignore the fields for now, focusing just on the struct definition. This is the constructor:

impl<'source> Lexer<'source> {
    pub fn new(input: &'source str) -> Self {
      // ...
    }
}

The 'source lifetime specifier is used to explicitly annotate the lifetime of the input string slice, since we want to store it in our Lexer and refer to it later. Once a Lexer is created, the user obtains new tokens by calling next_token:

pub fn next_token(&mut self) -> Token<'source> {
  // ...
}

Note that the returned Token also has the same lifetime annotation. Here's how the type is defined:

#[derive(Debug, PartialEq, Clone, Copy)]
pub struct Token<'source> {
    pub value: TokenValue<'source>,
    pub pos: usize,
}

#[derive(Debug, PartialEq, Clone, Copy)]
pub enum TokenValue<'source> {
    EOF,
    Error,

    Plus,
    Minus,
    Multiply,
    Divide,
    Period,
    Backslash,
    Colon,
    Percent,
    Pipe,
    Exclamation,
    Question,
    Pound,
    Ampersand,
    Semi,
    Comma,
    LeftParen,
    RightParen,
    LeftAng,
    RightAng,
    LeftBrace,
    RightBrace,
    LeftBracket,
    RightBracket,
    Equals,
    Comment(&'source str),
    Identifier(&'source str),
    Number(&'source str),
    Quote(&'source str),
}

Now it should be clear how these lifetimes are tied together. Some tokens hold slices into the user's input, and this is encoded in the explicit lifetimes. The signature of the next_token method says "we return tokens with a lifetime that's tied to the lifetime of the input passed into the constructor".

We can also provide a more natural iteration API for the Lexer by implementing the Iterator trait:

// Lexer is an Iterator; it returns tokens until EOF is encountered, when it
// returns None (the EOF token itself is not returned). Note that errors are
// still returned as tokens with TokenValue::Error.
impl<'source> Iterator for Lexer<'source> {
    type Item = Token<'source>;
    fn next(&mut self) -> Option<Self::Item> {
        if self.error {
            // If an error has already been set before we invoke next_token,
            // it means we've already returned TokenValue::Error once and now
            // we should terminate the iteration.
            return None;
        }

        let tok = self.next_token();
        if tok.value == TokenValue::EOF {
            None
        } else {
            Some(tok)
        }
    }
}

The iterator implementation makes it possible to integrate the lexer with the rest of Rust very elegantly; for example, to obtain all the tokens in a given input as a vector:

pub fn tokenize_all_collect<'source>(data: &'source str) -> Vec<Token<'source>> {
    let lex = Lexer::new(&data);
    lex.collect()
}

Implementation

Generally, the implementation of the lexer in Rust follows the same approach used by my previous hand-written lexers in this series. I'd like to highlight a couple of Rust-specific aspects here.

Our lexer fully supports Unicode, so I decided to use Rust's string iterator support to obtain chars from &str. Rust provides a helpful iterator called CharIndices, which yields chars along with their position in the input - we need the position to report token locations. Furthermore, since our lexer requires a bit of look-ahead [1], the iterator is wrapped in Peekable, which provides the peek method. As we've seen above already, the final definition is:

iter: Peekable<CharIndices<'source>>,

With this in mind, the code of the lexer should be very readable, even for someone not too familiar with Rust.

Another note is how sub-string extraction happens when tokens are returned. As an example, let's look at the scan_number method which is invoked when the lexer's current character is a digit:

fn scan_number(&mut self) -> Token<'source> {
    let startpos = self.ci;
    while self.c.is_digit(10) {
        self.scan_char();
    }
    Token {
        value: TokenValue::Number(&self.input[startpos..self.ci]),
        pos: startpos,
    }
}

Recall that the Number variant of the TokenValue enum is defined as Number(&'source str) - it contains a reference to the input source string. In the code for scan_number, we see how this is actually implemented, by sub-slicing the input slice. This creates a slice with the same lifetime as the input slice (which is encoded in the lifetimes of the types). As in Go, sub-slicing is a very cheap operation in Rust (no heap allocation).

Performance

The performance of this Rust-implemented lexer is very good! I ran it on the same large TableGen file I've used for all the benchmarking, and it finishes tokenizing it in just 3.7 ms; this is about 33% faster than my fastest Go version from the previous post.

Profiling Rust is quite a bit trickier than Go, but I managed to cobble together enough magic flags and perf invocations to ascertain that next_token indeed takes the bulk of the time; this is good - the time is being spent where it's supposed to be spent.

Trying a variant with allocations

As described above, the API for this Rust lexer was designed to be zero-copy. Since my Go lexers experimented with different approaches, I've decided to see how a different API in Rust would look.

There are two aspects to ownership we can change: taking ownership of the input string, and/or returning owned Strings in the tokens.

For taking ownership of the input string, our constructor would have to look like:

pub fn new(input: String) -> Self

Does it make sense for a lexer to own its input? This isn't clear, and in reality it turned out to be tricky to implement due to lifetime issues. Rust is very unhappy when a struct field is a reference to another field of the same struct, because there is no safe way to move instances of such structs. In our case, the iterator (a struct field) needs a reference to the string (another field in the same struct), and this is doesn't pass the Rust compiler's scrutiny. I wanted to be able to implement this without actively fooling the compiler by using opaque indices or unsafe.

When the language fights you this hard, it may be a good sign that the design is wrong - it's better to leave the ownership to the code that creates a Lexer.

Ownership of returned tokens was easier to set up. The code for this is available in the owning.rs file of the repository. In this variant, the constructor doesn't change, but tokens are defined differently:

#[derive(Debug, PartialEq, Clone)]
pub struct Token {
    pub value: TokenValue,
    pub pos: usize,
}

#[derive(Debug, PartialEq, Clone)]
pub enum TokenValue {
    // .. types
    Comment(String),
    Identifier(String),
    Number(String),
    Quote(String),
}

Note that variants like Identifier now hold an owning String instead of a string reference. Therefore, lifetime annotations are no longer necessary on Token.

The scanning code now allocates a new String and reads characters into it from the iterator. We've seen previously how scan_number looks; here it is again, for the owning token variant (with some helper methods):

fn scan_number(&mut self) -> Token {
    let startpos = self.ci;
    Token {
        value: TokenValue::Number(self.scan_while_true(|c| c.is_digit(10))),
        pos: startpos,
    }
}

// Helper to scan chars while `pred(c)` returns true, into the given `s`.
fn scan_while_true_into<F>(&mut self, s: &mut String, pred: F)
where
    F: Fn(char) -> bool,
{
    while pred(self.c) {
        s.push(self.c);
        self.scan_char();
    }
}

// Helper to scan chars while `pred(c)` returns true and return all scanned
// chars in a new String.
fn scan_while_true<F>(&mut self, pred: F) -> String
where
    F: Fn(char) -> bool,
{
    let mut s = String::with_capacity(8);
    self.scan_while_true_into(&mut s, pred);
    s
}

Performance of this variant

When I first tried this variant, its performance wasn't great at all! It was about 30% slower than the string-copying version in Go. I think this has to do with the slight difference in approach - in Go, my lexer figures out the token boundaries using numerical indices and then converts a []byte into string in one fell swoop. The Rust version fetches chars one by one from an iterator and writes them into a String.

In particular, since Rust's String is dynamically allocated, this may incur reallocations (depending on how much is allocated initially).

So my solution was to create these strings with_capacity - as you can see in the previous code sample. This cut down the execution time to be roughly equal to Go's version where strings are copied.

Which API is better?

IMHO there's little doubt that the original "slice" API is better, for multiple reasons:

Performance: the results speak for themselves - the slice API is zero-copy and incurs no additional heap allocations. Like in Go, it deals in slice headers, which are just pointers to parts of a string.
The ownership story is very clear, symmetrical and explicitly documented with lifetime annotations. The 'source lifetime controls everything: it's the lifetime of the &str passed into the lexer's constructor, and it's the lifetime of the tokens. The symmetry feels important - the code that creates a lexer controls the lifetime of the input, as well as the output.
In general, there's a known good practice in API design which is not to force allocations on users, if possible. This is well articulated in this blog post for Go, but it applies equally well in Rust. In our slice API, the user can always call to_owned on the returned slice, if they so wish. But why do it for them? Why should we assume the users want us to return them an owned string? Returning a slice provides more flexibility in using the API.

Performance improvement: avoiding `Peekable`

After this post was published, the reader Utkarsh Kukreti mentioned that using Peekable is not the most optimal approach, since its next is slower than the underlying iterator's next, and calling it is on the hot path. Instead of using Peekable, we could just clone the iterator when we see a / and invoke next on the clone; iterators are small so cloning them is cheap.

Here's a patch with this change. Applying it to my lexer makes it ~11% faster on my machine, finishing the benchmark in about 3.4 ms!

To be honest, this is a little bit surprising and disappointing to me, since I was hoping that Peekable would fall into the zero abstraction promise of Rust. Perhaps this is something that can fixed in the future.

[1]	For properly tokenizing comments; when the lexer sees `/`, it needs to peek forward to see if this is the beginning of a `//`, or else the division operator. This is a common use case in lexers, especially when tokenizing multi-character operators (like `>=`).

A faster lexer in Go

2022-05-03T19:53:00-07:00

It's been a while since I've last rewritten my favorite lexical analyzer :-) That post is the last in a series implementing a lexer for the TableGen language in a variety of programming languages, using multiple techniques. The last lexer written, in Go, was very fast indeed - processing 1 MiB of source in about 20 milliseconds.

The other day I started wondering whether Go compiler improvements from the last few years made this code run any faster. Back in 2014 I measured it with Go 1.2, and now Go 1.18 is out. So I tried (on the same machine) with some newer Go versions; the full code for this is still here, and the benchmark is run as follows [1]:

$ TDINPUT=input.td go test -bench=Preall -benchtime=5s

These are the results:

Go 1.5 is comparable to 1.2, but by 1.10 there was a significant improvement in performance, and a further improvement in later versions. The code produced by Go 1.18 is more than twice as fast as the original lexer. Now it takes only ~9.6 ms to process the same 1 MiB of TableGen source!

Looking deeper into the performance of this lexer

This got me curious - what does the lexer spend its time on? Since Go has fantastic tooling for performance profiling, time to whip up some flags... This will be for the most recent version of Go (1.18):

$ TDINPUT=input.td go test -cpuprofile cpu.out -bench=Preall -benchtime=5s
...

# --nodefraction tells pprof to ignore nodes that take less than 5% of the
# total time - this significantly reduces the clutter in the produced graph

$ go tool pprof --nodefraction=0.05 ./example.com.test cpu.out
...
(pprof) web

Here's the important part of the profile:

As expected, the next function is very heavy in the profile, as it should be, since this is the main code taking characters from the input stream and making them ready for the lexer to process:

func (lex *Lexer) next() {
  if lex.nextpos < len(lex.buf) {
    lex.rpos = lex.nextpos

    // r is the current rune, w is its width. We start by assuming the
    // common case - that the current rune is ASCII (and thus has width=1).
    r, w := rune(lex.buf[lex.nextpos]), 1

    if r >= utf8.RuneSelf {
      // The current rune is not actually ASCII, so we have to decode it
      // properly.
      r, w = utf8.DecodeRune(lex.buf[lex.nextpos:])
    }

    lex.nextpos += w
    lex.r = r
  } else {
    lex.rpos = len(lex.buf)
    lex.r = -1 // EOF
  }
}

Assuming that most characters in the stream are within ASCII bounds, what this function does on each call is very little. FWIW, it's very similar to how the inner loop of Go's own text/scanner package works.

But notice the 16.5% spent on slicebytetostring - what is that all about? Note that it's invoked from several scan* methods. This has to be the string conversion of buffer slices, e.g. from scanIdentifier:

return Token{IDENTIFIER, string(lex.buf[startpos:lex.rpos]), startpos}

In Go, strings are immutable while []byte is mutable; the two cannot safely alias. Therefore, when we have some b []byte and we do string(b), Go allocates a copy of the slice to a new location and creates a string header to point to it.

We can verify this by running memory profiling and looking at the allocations:

$ TDINPUT=input.td go test -memprofile=mem.mprof -bench=Preall -benchtime=5s
...

$ go tool pprof --alloc_objects example.com.test mem.mprof
...
(pprof) list scanIdentifier
Total: 27388406
ROUTINE ======================== example%2ecom.(*Lexer).scanIdentifier
  21501647   21501647 (flat, cum) 78.51% of Total
         .          .    238:func (lex *Lexer) scanIdentifier() Token {
         .          .    239: startpos := lex.rpos
         .          .    240: for isAlpha(lex.r) || isDigit(lex.r) {
         .          .    241:         lex.next()
         .          .    242: }
  21501647   21501647    243: return Token{IDENTIFIER, string(lex.buf[startpos:lex.rpos]), startpos}
         .          .    244:}
         .          .    245:
         .          .    246:func (lex *Lexer) scanNumber() Token {
         .          .    247: startpos := lex.rpos
         .          .    248: for isDigit(lex.r) {

As expected, many allocations come from the suspected line. Looking back at the CPU profiling graph, we also see that much of the run-time of slicebytetostirng is spent on the allocation (mallocgc). This is actually a good clue to the stark performance improvement in Go since 1.2; while the compiler certainly became better since then (especially with the register-based ABI introduced in 1.17), it didn't improve that much. What did improve a whole lot is the garbage collector; many allocations mean lots of heap to scan and track, and lots of garbage to clean up.

Avoiding allocations with sub-strings

Before we move on, let's get one thing out of the way. There are all kinds of tricks you can play in Go using unsafe to explicitly alias byte slices and strings and thus avoid allocations; this is even done in the standard library. I'll leave this topic out of this post and may cover it separately in the future.

I did wonder about the cost of these allocations, though; what if we avoid all the copies? At the moment, the lexer's API is such that it takes a []byte as input:

func NewLexer(buf []byte) *Lexer

And the tokens we return have string values:

type Token struct {
  Name TokenName
  Val  string
  Pos  int
}

What if we created a lexer with a string input instead? Since Go strings are immutable, subslices alias each other and are trivial to create and pass around; consider this code sample:

s := "hello there"
s2 := s[6:]

When s is created, it allocates an 11-byte sequence of bytes and creates a 2-word string header in s: a pointer to the sequence of bytes and a len field with the value 11. When the second line is run, Go just creates a new string header that points into the same byte buffer (with offset 6) and length 5. There are no allocations (string headers are typically created on the stack just like integers).

So I went ahead and rewrote the lexer using this different API; the code changes are pretty minor and the full code is available here. The scanIdentifier method now looks like this:

func (lex *Lexer) scanIdentifier() Token {
  startpos := lex.rpos
  for isAlpha(lex.r) || isDigit(lex.r) {
    lex.next()
  }
  return Token{IDENTIFIER, lex.buf[startpos:lex.rpos], startpos}
}

Note there's no string(...) cast for the token value; since lex.buf is already a string in this version, that wouldn't be necessary. Instead, we just return a string slice, which creates a new 2-word header (instead of allocating a new string and copying the data into it).

If we benchmark this version, it runs in 7.7 ms for our input, about 20% faster.

Is this a good API though?

OK, so we found a way to further improve the performance of the lexer by taking a string input; is this API useful though? Is it common for users of a lexer to have a fully formed string? To be fair, the question applies equally to the existing []byte API. IMHO, in most cases the answer is it depends. If the input to parse is already in memory (say, it was entered into some sort of text box in your GUI), then yes, string is fine in an API. More typically, though, this data is read from somewhere.

Go has useful I/O interfaces like io.Reader; this interface encapsulates "a place we read data from", and this data is typically read incrementally; e.g. you don't slurp the whole input file in a single go, but read it in chunks as needed. My old-ish SSD has read speeds in the order of 500 MiB / sec, meaning that a 1 MiB file will take about 2 ms to read from disk. If we really care about performance, overlapping this read with lexing makes sense. But this brings us to a io.Reader based API, where our substring optimization won't really work.

Let's see how Go itself does it; the text/scanner package is initialized thus:

func (s *Scanner) Init(src io.Reader) *Scanner

And to obtain the text of a token, you call TokenText, which returns a string. If we look under the hood, TokenText does the equivalent of string(some []byte buffer), which incurs an allocation and copy.

When the API is io.Reader, there's really no choice for this, though. It's really hard to safely return a string that aliases part of a []byte buffer accumulated from reading some external source. This is what other standard library package do as well - io.Reader input is very idiomatic.

The Go compiler itself has a lexical scanner in src/cmd/compile/internal/scanner.go, and it also takes an io.Reader (in its init method); naturally, it also has to allocate and copy literal values with a string(...) conversion (these are stored in its lit field).

Garbage Collector

We're not done yet. Let's look at the profile of the string version, using the same profiling invocation as before:

We see that our Lexer methods no longer allocate heap data - that's good! We can prove this further by looking at memory profiling or the compiler's escape analysis with -gcflags="-m" - I'll leave this as an exercise for diligent readers.

That said, the garbage collector (GC) still clearly runs, taking non-trivial %s of CPU, as we can see from the right side of the graph. Why does this happen?

This is a good example of why benchmarking is tricky, particularly in GC'd languages. The Go GC is concurrent, running "silently" alongside our application whenever it sees fit. In benchmarking situations, this often depends on the nature of the specific benchmark; specifically, in our benchmark the top-level function tokenizeAllPrealloc is invoked in a loop. This function pre-allocates a slice of 200k tokens with make. Each Token in our lexer occupies 4 words, which is 32 bytes on my 64-bit machine. Overall, the pre-allocation of the token slice takes somewhere on the order of 6.4 MiB. Running in a benchmarking loop for 5 seconds results in over 700 cycles, for a total of 4.5 GiB of heap data (disregarding any additional heap the benchmarking machinery uses) - which is a lot!

Go lets us control GC behavior with the GOGC env var, which sets the percentage of heap growth that triggers a collection cycle. The default is 100, which means a cycle is triggered when the heap doubles. If we use GODEBUG=gctrace=1 to trace GC invocations, we'll see that the GC runs over 700 cycles during our benchmark; clearly, this affects performance.

What happens if we try to tweak GOGC? Here's a chart for different values:

The fastest run of ~6 ms is achieved with GOGC=1600 and stays stable thereafter. Using tracing again, we see that with this setting the GC only runs ~50 times during the 5 sec benchmark, compared to 700+ previously. FWIW, disabling GC entirely with GOGC=off produces a slightly slower run-time at 6.5 ms (there's quite a lot of data here, so an overly large heap may occasionally lead to swapping).

Does any of this matter? This depends on your application! Admittedly, we're deep in the weeds here, trying to shave off sub-milliseconds from fully lexing a large input file. If you care about performance at this level of granularity, it's very likely that you'll consider the full lifecycle of your application and will have the GC tuned already.

As we've seen with the stark performance improvements from older versions of Go, the GC has improved a lot. And it keeps improving! Right now an accepted proposal aims to add more control for users with a new GOMEMLIMIT var. This is likely to land in the next Go version (or the one after it).

A longer term proposal to add arenas to Go is also on the table. This permits much finer-grained memory control, and is particularly suited for programs like compilers or other users of lexical analysis. Such programs have a lifecycle that's very suitable for arenas - you allocate a bunch of data per phase, and then at the very end you release everything in one step.

Conclusion

This post discussed some potential optimizations to a lexical scanner written in Go. It touched upon the relative efficiency of converting byte slices to strings vs. taking substrings, discussed some strategies w.r.t. API design in Go, and even got into optimizing GC behavior.

I'm pleasantly surprised at how fast lexical scanning in Go is. Even the copying version without GC tuning takes 9.6 ms to process 1 MiB of input source - that's over 100 MiB /sec! Depending on where your source is stored, this could be almost as fast as, or faster than your CPU can read the source into memory. With API changes and GC tuning we were able to improve this by a further 37% to 6 ms, though the specifics here depend on your actual application. Compared to our original version compiled with Go 1.2, the new lexer is more than 3 times as fast!

Update (2022-05-24): The in-development version of Go (1.19) runs this benchmark even faster at 5.6 ms (with GOGC=1600)! Go 1.19 should be released in August.

Update (2023-06-02): Today's in-development version of Go (1.21) runs this benchmark in 4.5 ms (with GOGC=1600)! Go 1.21 should be released in August 2023, but you can try it today with gotip.

[1]

To run this with Go 1.5 I had to comment out the t.Run wrapper in the tests since this method did not exist in 1.5 yet; the tests still pass just fine, but in case of failure the reporting would be less helpful.

Also note the TDINPUT environment variable. This is just to reuse the large-ish input file in multiple places without having to commit it to Git many times.

Rewriting the lexer benchmark in Go

2014-03-27T05:58:29-07:00

Last year I was toying with a simple lexer (for the TableGen language, because why not), implementing it using multiple approaches in both Python and Javascript. Redoing the same task using multiple approaches and using more than one language is a very interesting code kata and a great way to learn.

Since I've been recently looking at Go, I continued the exercise by reimplementing the lexer (the hand-written one, not a regex-based) in Go. The full code is available here (along with a large input file used for benchmarking).

Naturally, since my previous posts did performance comparisons between Python and Javascript, I wanted to add Go to the graph. I also had to rerun all the benchmarks because from the time of writing those posts I got a new, much faster, machine.

Anyway, here it is:

Since Python is so slow here, it's hard to see the difference between the fastest versions, but the handwritten Go lexer is roughly on par with the Javascript one (33 vs. 31 msec). The benchmarks were run on my i7-4771 machine (amd64); go1.2.1, Node.js v0.10.26.

Now, this is quite literally the first non-trivial Go program I've written and I'm a neophyte by all measures, so any tips on the code would be very welcome. I tried to stay faithful to the Javascript implementation in terms of the algorithm, so the comparison would be fair.

That said, shortly after completing the code I started wondering if it could be made faster. There's something about Go that makes you think about performance on a low level, not unlike when programming in C. Maybe it's because so many things are explicit - pointers, slices, etc.

Anyhow, the code that uses a lexer to fill in a slice of tokens caught my eye:

toks := []Token{}

startTime := time.Now()
for {
        nt := nl.NextToken()
        toks = append(toks, nt)
        if nt.Name == EOF {
                break
        }
}

That toks = append(toks, nt) in particular. As the size grows, toks will have to be reallocated and all its contents copied over. Since the input in my case had close to 200000 tokens and reallocation doubles the slice size, this means that in the order of 16 reallocations have to happen here, each time copying all the elements over. If that sounds like a lot of wasted work to you, that's because it is.

So I tried replacing the first line with:

toks := make([]Token, 0, 200000)

And wow, the runtime dropped from 33 to 20 ms, making it 33% faster than the JS version. To be fair to JS I tried to perform the same optimization there (instead of pushing onto an array, create a large one in advance), but this has actually made things slower. Some sources online claim that V8 (which is what I'm running underneath, since my local code runs on Node.js) doesn't like preallocating large arrays.

So as is often the case with benchmarks, it's difficult to do an apples-to-apples comparison here. A hunch tells me that in a fully optimized (by a programmer skilled in the language) version of this benchmark, Go would still win, because its nature (typed, compiled to native code, and exposing a lot of low-level details) make it easier to reason about in terms of performance. But performance was not really the point here - I just wanted to see how easy it is to reimplement the same lexer in Go.

Hopefully the code would be useful/interesting to someone; please let me know what I could've done better.

Update (2022-05-03): A newer version of Go and some additional optimizations make this lexer more than 3x faster. See details in this post.

Hand-written lexer in JavaScript compared to the regex-based ones

2013-07-16T06:25:10-07:00

A few weeks ago I wrote about the comparison between regex-based lexers in Python and Javascript. Javascript running on Node.js (V8) ended up being much faster than Python, and in both languages a speed improvement could be gained by switching to a single regex and letting the regex engine do the hard work.

However, in the real world you'll find that most lexers (particularly lexers for real programming languages) are not written that way. They are carefully hand-written to go over the input and dispatch to the appropriate token handling code depending on the first character encountered.

This technique makes more sense for complex languages because it's much more flexible than regexes (for instance, representing nested comments with regexes is a big challenge). But I was curious also about its performance implications.

So I hacked together a hand-written lexer for exactly the same language used in the previous benchmark and also exercised it on the same large file to make sure the results are correct. Its runtime, however, surprised me. Here's the runtime of lexing a large-ish document (smaller is better):

I was expecting the runtime to be much closer to the single-regex version; in fact I was expecting it to be a bit slower (because most of the regex engine work is done at a lower level). But it turned out to be much faster, more than 2.5x. Another case of the real world beating intuition.

I was lazy to look, but if V8's regex implementation is anything like Python's, then alternation of many options (foo|bar) is not as efficient as one might expect because the regex engine does not use real DFAs, but rather backtracking. So alternation essentially means iteration deep in the regex engine. On the other hand, the hand-written code seems like something quite optimizable by a JIT compiler like V8 (the types are simple and consistent) so there's a good chance it got converted into tight machine code. Throw some inlining in and the speed is not very unlikely.

Anyway, here is the hand-written lexer. It's significantly more code than the regex-based ones, but I can't say it was particularly difficult to write - the main part took a bit more than an hour, or so. If you have any additional ideas about the speed difference, I'll be happy to hear about them.

'use strict';

var Lexer = exports.Lexer = function() {
  this.pos = 0;
  this.buf = null;
  this.buflen = 0;

  // Operator table, mapping operator -> token name
  this.optable = {
    '+':  'PLUS',
    '-':  'MINUS',
    '*':  'MULTIPLY',
    '.':  'PERIOD',
    '\\': 'BACKSLASH',
    ':':  'COLON',
    '%':  'PERCENT',
    '|':  'PIPE',
    '!':  'EXCLAMATION',
    '?':  'QUESTION',
    '#':  'POUND',
    '&':  'AMPERSAND',
    ';':  'SEMI',
    ',':  'COMMA',
    '(':  'L_PAREN',
    ')':  'R_PAREN',
    '<':  'L_ANG',
    '>':  'R_ANG',
    '{':  'L_BRACE',
    '}':  'R_BRACE',
    '[':  'L_BRACKET',
    ']':  'R_BRACKET',
    '=':  'EQUALS'
  };
}

// Initialize the Lexer's buffer. This resets the lexer's internal
// state and subsequent tokens will be returned starting with the
// beginning of the new buffer.
Lexer.prototype.input = function(buf) {
  this.pos = 0;
  this.buf = buf;
  this.buflen = buf.length;
}

// Get the next token from the current buffer. A token is an object with
// the following properties:
// - name: name of the pattern that this token matched (taken from rules).
// - value: actual string value of the token.
// - pos: offset in the current buffer where the token starts.
//
// If there are no more tokens in the buffer, returns null. In case of
// an error throws Error.
Lexer.prototype.token = function() {
  this._skipnontokens();
  if (this.pos >= this.buflen) {
    return null;
  }

  // The char at this.pos is part of a real token. Figure out which.
  var c = this.buf.charAt(this.pos);

  // '/' is treated specially, because it starts a comment if followed by
  // another '/'. If not followed by another '/', it's the DIVIDE
  // operator.
  if (c === '/') {
    var next_c = this.buf.charAt(this.pos + 1);
    if (next_c === '/') {
      return this._process_comment();
    } else {
      return {name: 'DIVIDE', value: '/', pos: this.pos++};
    }
  } else {
    // Look it up in the table of operators
    var op = this.optable[c];
    if (op !== undefined) {
      return {name: op, value: c, pos: this.pos++};
    } else {
      // Not an operator - so it's the beginning of another token.
      if (Lexer._isalpha(c)) {
        return this._process_identifier();
      } else if (Lexer._isdigit(c)) {
        return this._process_number();
      } else if (c === '"') {
        return this._process_quote();
      } else {
        throw Error('Token error at ' + this.pos);
      }
    }
  }
}

Lexer._isnewline = function(c) {
  return c === '\r' || c === '\n';
}

Lexer._isdigit = function(c) {
  return c >= '0' && c <= '9';
}

Lexer._isalpha = function(c) {
  return (c >= 'a' && c <= 'z') ||
         (c >= 'A' && c <= 'Z') ||
         c === '_' || c === '$';
}

Lexer._isalphanum = function(c) {
  return (c >= 'a' && c <= 'z') ||
         (c >= 'A' && c <= 'Z') ||
         (c >= '0' && c <= '9') ||
         c === '_' || c === '$';
}

Lexer.prototype._process_number = function() {
  var endpos = this.pos + 1;
  while (endpos < this.buflen &&
         Lexer._isdigit(this.buf.charAt(endpos))) {
    endpos++;
  }

  var tok = {
    name: 'NUMBER',
    value: this.buf.substring(this.pos, endpos),
    pos: this.pos
  };
  this.pos = endpos;
  return tok;
}

Lexer.prototype._process_comment = function() {
  var endpos = this.pos + 2;
  // Skip until the end of the line
  var c = this.buf.charAt(this.pos + 2);
  while (endpos < this.buflen &&
         !Lexer._isnewline(this.buf.charAt(endpos))) {
    endpos++;
  }

  var tok = {
    name: 'COMMENT',
    value: this.buf.substring(this.pos, endpos),
    pos: this.pos
  };
  this.pos = endpos + 1;
  return tok;
}

Lexer.prototype._process_identifier = function() {
  var endpos = this.pos + 1;
  while (endpos < this.buflen &&
         Lexer._isalphanum(this.buf.charAt(endpos))) {
    endpos++;
  }

  var tok = {
    name: 'IDENTIFIER',
    value: this.buf.substring(this.pos, endpos),
    pos: this.pos
  };
  this.pos = endpos;
  return tok;
}

Lexer.prototype._process_quote = function() {
  // this.pos points at the opening quote. Find the ending quote.
  var end_index = this.buf.indexOf('"', this.pos + 1);

  if (end_index === -1) {
    throw Error('Unterminated quote at ' + this.pos);
  } else {
    var tok = {
      name: 'QUOTE',
      value: this.buf.substring(this.pos, end_index + 1),
      pos: this.pos
    };
    this.pos = end_index + 1;
    return tok;
  }
}

Lexer.prototype._skipnontokens = function() {
  while (this.pos < this.buflen) {
    var c = this.buf.charAt(this.pos);
    if (c == ' ' || c == '\t' || c == '\r' || c == '\n') {
      this.pos++;
    } else {
      break;
    }
  }
}

Eli Bendersky's website - Lexer

Performance of coroutine-style lexers in Go

Rob Pike's original lexer

Lexing TableGen

Performance

Using a buffered channel

Does performance matter here?

Rewriting the lexer benchmark in Rust

Designing an API

Implementation

Performance

Trying a variant with allocations

Performance of this variant

Which API is better?

Performance improvement: avoiding Peekable

A faster lexer in Go

Looking deeper into the performance of this lexer

Avoiding allocations with sub-strings

Is this a good API though?

Garbage Collector

Conclusion

Rewriting the lexer benchmark in Go

Hand-written lexer in JavaScript compared to the regex-based ones

Performance improvement: avoiding `Peekable`