Eli Bendersky's website - Testing

File-driven testing in Go

2022-02-12T20:20:00-08:00

This is a quick post about a testing pattern I've found useful in multiple programming languages; Go's testing package makes it particularly easy to use and implement.

Table-driven tests are a well-known and recommended technique for structuring unit tests in Go. I assume the reader is familiar with table-driven tests; if not, check out this basic example or google for the many tutorials available online.

In some cases, the input to a test can be more data than we're willing to paste into a _test.go file, or the input can be non-text. For these cases, file-driven tests are a useful extension to the table-driven technique.

Suppose - for the sake of demonstration - that we want to test Go's go/format package. This package provides programmatic access to gofmt's capabilities with the Source function that takes unformatted Go text as input and produces formatted text. We could use table-driven tests for this, but some inputs may be large - whole Go files, or significant chunks thereof. It can be more convenient to have these inputs in external files the test can read and invoke format.Source on.

This toy GitHub repository shows how it's done. Here is the directory structure:

$ tree
.
├── go.mod
└── somepackage
    ├── somepackage_test.go
    └── testdata
        ├── funcs.golden
        ├── funcs.input
        ├── simple-expr.golden
        └── simple-expr.input

Our test code is in somepackage_test.go - we'll get to it shortly. Alongside it is the testdata directory with pairs of files named <name>.input and <name>.golden. Each pair serves as a test: the input file is fed into the formatter, and the formatter's output is compared to the golden file.

This is the entirety of our test code:

func TestFormatFiles(t *testing.T) {
  // Find the paths of all input files in the data directory.
  paths, err := filepath.Glob(filepath.Join("testdata", "*.input"))
  if err != nil {
    t.Fatal(err)
  }

  for _, path := range paths {
    _, filename := filepath.Split(path)
    testname := filename[:len(filename)-len(filepath.Ext(path))]

    // Each path turns into a test: the test name is the filename without the
    // extension.
    t.Run(testname, func(t *testing.T) {
      source, err := os.ReadFile(path)
      if err != nil {
        t.Fatal("error reading source file:", err)
      }

      // >>> This is the actual code under test.
      output, err := format.Source(source)
      if err != nil {
        t.Fatal("error formatting:", err)
      }
      // <<<

      // Each input file is expected to have a "golden output" file, with the
      // same path except the .input extension is replaced by .golden
      goldenfile := filepath.Join("testdata", testname+".golden")
      want, err := os.ReadFile(goldenfile)
      if err != nil {
        t.Fatal("error reading golden file:", err)
      }

      if !bytes.Equal(output, want) {
        t.Errorf("\n==== got:\n%s\n==== want:\n%s\n", output, want)
      }
    })
  }
}

The code is well commented, but here are a few highlights:

The test file pairs are auto-discovered using filepath.Glob. Placing additional file pairs in the testdata directory will automatically ensure they are used by subsequent test executions. When we run go test, the current working directory will be set to the package that's being tested, so finding the testdata directory is easy.
The testdata name is special for the Go toolchain, which will ignore files in it (so you can place files named *.go there, for example, and they won't be built or analyzed).
For each file pair we create a subtest with T.Run. This means it's a separate test as far as the test runner is concerned - reported on its own in verbose output, can be run in parallel with other tests, etc.

If we run the tests, we get:

$ go test -v ./...
=== RUN   TestFormatFiles
=== RUN   TestFormatFiles/funcs
=== RUN   TestFormatFiles/simple-expr
--- PASS: TestFormatFiles (0.00s)
    --- PASS: TestFormatFiles/funcs (0.00s)
    --- PASS: TestFormatFiles/simple-expr (0.00s)
PASS
ok    example.com/somepackage 0.002s

Note how each input file in testdata generated its own test with a distinct name.

The "golden file" approach shown here is just one of the possible patterns for using file-driven tests. Often there is no separate file for the expected output, and instead the input file itself contains some special markers that drive the test's expectations. It's really dependent on the specific testing scenario; the Go project and subprojects (such as x/tools) use several variations of this testing pattern.

Testing multiple implementations of a trait in Rust

2021-11-06T05:32:00-07:00

I've been hacking on a small practice project in Rust where I implement the same data structure in several different ways. When testing this project, I want to run exactly the same set of tests on several types that implement the same trait.

As a demonstrative example, let's take the following trait:

pub trait Calculator {
    fn new() -> Self;
    fn add(&self, a: u32, b: u32) -> u32;
}

A straightforward implementation could be Foo:

pub struct Foo {}

impl Calculator for Foo {
    fn new() -> Self {
        Self {}
    }

    fn add(&self, a: u32, b: u32) -> u32 {
        a + b
    }
}

Or, if you enjoy the Peano axioms, a somewhat more involved implementation could be Bar:

pub struct Bar {}

impl Calculator for Bar {
    fn new() -> Self {
        Self {}
    }

    fn add(&self, a: u32, b: u32) -> u32 {
        if b == 0 {
            a
        } else {
            self.add(a, b - 1) + 1
        }
    }
}

Our task is to write the same set of tests once, and invoke it on both Foo and Bar with as little boilerplate as possible. Let's examine several approaches for doing this [1].

Straightforward trait-based testing

The most basic approach to testing our types would be something like:

#[cfg(test)]
mod tests {
    use crate::calculator::{Bar, Calculator, Foo};

    fn trait_tester<C: Calculator>() {
        let c = C::new();
        assert_eq!(c.add(2, 3), 5);
        assert_eq!(c.add(10, 43), 53);
    }

    #[test]
    fn test_foo() {
        trait_tester::<Foo>();
    }

    #[test]
    fn test_bar() {
        trait_tester::<Bar>();
    }
}

The trait_tester function can be invoked on any type that implements the Calculator trait and can host a collection of tests. "Concrete" test functions like test_foo then call trait_tester; the concrete test functions are what the Rust testing framework sees because they're marked with the #[test] attribute.

On the surface, this approach seems workable; looking deeper, however, there is a serious issue.

Suppose we want to write multiple test functions that test different features and usages of our Calculator. We could add trait_tester_feature1, trait_tester_feature2, etc. Then, the concrete test functions would look something like:

#[test]
fn test_foo() {
    trait_tester::<Foo>();
    trait_tester_feature1::<Foo>();
    trait_tester_feature2::<Foo>();
}

#[test]
fn test_bar() {
    trait_tester::<Bar>();
    trait_tester_feature1::<Bar>();
    trait_tester_feature2::<Bar>();
}

Taken to the limit, there's quite a bit of repetition here. In a realistic project the number of tests can easily run into the dozens.

The problem doesn't end here, though; in Rust, the unit of testing is test_foo, not the trait_tester* functions. This means that only test_foo will show up in the testing report, there's no easy way to select to run only trait_tester_feature1, etc. Moreover, test parallelization can only happen between #[test] functions.

The fundamental issue here is: what we really want is to mark each of the trait_tester* functions with #[test], but this isn't trivial because #[test] is a compile-time feature, and the compiler is supposed to know what concrete types partake in each #[test] function definition.

Thankfully, Rust has just the tool for generating code at compile time.

First attempt with macros

Macros can help us generate functions tagged with #[test] at compile time. Let's try this:

macro_rules! calculator_tests {
    ($($name:ident: $type:ty,)*) => {
    $(
        #[test]
        fn $name() {
            let c = <$type>::new();
            assert_eq!(c.add(2, 3), 5);
            assert_eq!(c.add(10, 43), 53);
        }
    )*
    }
}

#[cfg(test)]
mod tests {
    use crate::calculator::{Bar, Calculator, Foo};

    calculator_tests! {
        foo: Foo,
        bar: Bar,
    }
}

The calculator_tests macro generates multiple #[test]-tagged functions, one per type. If we run cargo test, we'll see that the Rust testing framework recognizes and runs them:

[...]
test typetest::tests::bar ... ok
test typetest::tests::foo ... ok
[...]

However, there's an issue; how to we add more testing functions per type, as discussed previously? If only we could do something like fn ${name}_feature1 to name a function. Due to macro hygiene rules this is tricky, though procedural macros like the paste crate can help; see this code for an example.

In any case, I believe there's a better solution.

Second attempt with macros

Instead of encoding the type variant in the function name, we can use a Rust sub-module:

macro_rules! calculator_tests {
    ($($name:ident: $type:ty,)*) => {
    $(
        mod $name {
            use super::*;

            #[test]
            fn test() {
                let c = <$type>::new();
                assert_eq!(c.add(2, 3), 5);
                assert_eq!(c.add(10, 43), 53);
            }
        }
    )*
    }
}

#[cfg(test)]
mod tests {
    use crate::calculator::{Bar, Calculator, Foo};

    calculator_tests! {
        foo: Foo,
        bar: Bar,
    }
}

Now all functions are named test, but they're namespaced inside a module with a configurable name. And yes, now we can easily add more testing functions:

macro_rules! calculator_tests {
    ($($name:ident: $type:ty,)*) => {
    $(
        mod $name {
            use super::*;

            #[test]
            fn test() {
                let c = <$type>::new();
                assert_eq!(c.add(2, 3), 5);
                assert_eq!(c.add(10, 43), 53);
            }

            #[test]
            fn test_feature1() {
                let c = <$type>::new();
                assert_eq!(c.add(6, 9), 15);
            }
        }
    )*
    }
}

If we run cargo test, it works as expected:

test typetestmod::tests::bar::test ... ok
test typetestmod::tests::bar::test_feature1 ... ok
test typetestmod::tests::foo::test_feature1 ... ok
test typetestmod::tests::foo::test ... ok

Each test has its own full path, and is invoked separately. We can select which tests to run from the command line - running only the tests for Bar, say, or run all the feature1 tests for all types. Also notice that the test names are reported "out of order"; this is because they are all run concurrently!

To conclude, with some macro hackery the goal is achieved. We can now write any number of tests in a generic way, and invoke all these tests on multiple types with minimal duplication - just one extra line per type [2].

It's not all perfect, though. Macros add a layer of indirection and it leaks in the error messages. If one of the assert_eq! invocations fails, the reported line is at the point of macro instantiation, which is the same line for all tests for any given type. Re-running the test with RUST_BACKTRACE=1 helps find which of the asserts failed, since it appears in the trace.

[1]	The full source code for this post can be found on GitHub.

[2] Sharp-eyed readers will note that using this approach the common trait isn't actually needed at all! Macros work by textual substitution (AST substitution, to be precise), so the generated code creates a concrete type and invokes its methods. The macro-based tests would work even if Foo and Bar didn't declare themselves as implementing the Calculator trait.

Testing flag parsing in Go programs

2020-04-04T07:24:00-07:00

This is a short post about writing tests for Go programs that have command-line flags, and in particular programs where flags are a significant chunk of their functionality (think command-line tools like grep). I'll cover a technique that works well for code that uses the standard flag package for parsing flags. If you're using a framework like Cobra, refer to its documentation for best practices.

Here's a typical flow of a program that parses flags and other command-line arguments [1]:

For some programs the "parse flags" part is trivial enough that you wouldn't typically even consider writing unit tests for it (whether this is a good choice or not is outside the scope of this post). However, in many other programs flag parsing is a critical part of the functionality. In some common command-line tools, like grep or find, it would be fair to say that flags are a major part of the user-visible functionality.

Therefore, when we talk about testing such programs there are two aspects we'll need help with:

Writing unit tests only for the "parse flags" part of the program.
Writing integration tests where the input is the command line given to the program and the output is the full effect of the program (observable output and/or environment changes).

The technique described in this post is helpful for both. It's a very simple concept that should be familiar to most programmers. The main goal of this post is to show how to do it with the tools provided by Go's flag package.

Let's start by showing how the program flow described in the previous is split to enable testing:

Instead of parsing a flag and invoking its related functionality directly, the values of parsed flags will be saved in a Config value that can be used in tests. This lets us write unit-tests for the "parse flags" part of the program.

The following is sample parseFlags for a toy program. The full code with tests is available here.

type Config struct {
  verbose  bool
  greeting string
  level    int

  // args are the positional (non-flag) command-line arguments.
  args []string
}

// parseFlags parses the command-line arguments provided to the program.
// Typically os.Args[0] is provided as 'progname' and os.args[1:] as 'args'.
// Returns the Config in case parsing succeeded, or an error. In any case, the
// output of the flag.Parse is returned in output.
// A special case is usage requests with -h or -help: then the error
// flag.ErrHelp is returned and output will contain the usage message.
func parseFlags(progname string, args []string) (config *Config, output string, err error) {
  flags := flag.NewFlagSet(progname, flag.ContinueOnError)
  var buf bytes.Buffer
  flags.SetOutput(&buf)

  var conf Config
  flags.BoolVar(&conf.verbose, "verbose", false, "set verbosity")
  flags.StringVar(&conf.greeting, "greeting", "", "set greeting")
  flags.IntVar(&conf.level, "level", 0, "set level")

  err = flags.Parse(args)
  if err != nil {
    return nil, buf.String(), err
  }
  conf.args = flags.Args()
  return &conf, buf.String(), nil
}

This code uses two features of the flag package to make the approach work:

Creating a custom flag.FlagSet instead of using the default global one.
Using the XxxVar variant of flag definition methods (e.g. BoolVar) to tie parsed flag values to pre-defined variables.

There's a bit of care around handling usage flags like -h, but otherwise this code is trivial. Here's the rest of the program, simulating some arbitrary "work" given the parsed configuration:

func doWork(config *Config) {
  fmt.Printf("config = %+v\n", *config)
}

func main() {
  conf, output, err := parseFlags(os.Args[0], os.Args[1:])
  if err == flag.ErrHelp {
    fmt.Println(output)
    os.Exit(2)
  } else if err != nil {
    fmt.Println("got error:", err)
    fmt.Println("output:\n", output)
    os.Exit(1)
  }

  doWork(conf)
}

With our parseFlags, unit-testing of flags is straightforward:

func TestParseFlagsCorrect(t *testing.T) {
  var tests = []struct {
    args []string
    conf Config
  }{
    {[]string{"-verbose"},
      Config{verbose: true, greeting: "", level: 0, args: []string{}}},

    {[]string{"-level", "8", "-greeting", "joe", "-verbose", "foo"},
      Config{verbose: true, greeting: "joe", level: 8, args: []string{"foo"}}},

    // ... many more test entries here
  }

  for _, tt := range tests {
    t.Run(strings.Join(tt.args, " "), func(t *testing.T) {
      conf, output, err := parseFlags("prog", tt.args)
      if err != nil {
        t.Errorf("err got %v, want nil", err)
      }
      if output != "" {
        t.Errorf("output got %q, want empty", output)
      }
      if !reflect.DeepEqual(*conf, tt.conf) {
        t.Errorf("conf got %+v, want %+v", *conf, tt.conf)
      }
    })
  }
}

If we want to write integration tests for the whole program driven by a command line, it would look very similar to the test above. However, instead of comparing a Config with an expected value, we'd compare the output and/or side effects of doWork with expected values. Naturally, the test would likely have to be split into multiple variants with different table-driven outcomes, depending on the nature of doWork. If doWork's only observable side effect is text output, things are simple. But doWork could also modify the file system and/or run network servers/clients, in which case it's a bit trickier but no different from testing any such program, regardless of the flag parsing part.

The code accompanying this post has a more complete set of tests and exercises some error scenarios.

[1]	Given the invocation `grep -i joe file.go`, we'll refer to `-i` as a flag and to `joe` and `file.go` as positional arguments. The distinction is not terribly important for the sake of this post, however.

Dynamically generating Python test cases

2014-04-02T06:44:15-07:00

Testing is crucial. While many different kinds and levels of testing exist, there's good library support only for unit tests (the Python unittest package and its moral equivalents in other languages). However, unit testing does not cover all kinds of testing we may want to do - for example, all kinds of whole program tests and integration tests. This is where we usually end up with a custom "test runner" script.

Having written my share of such custom test runners, I've recently gravitated towards a very convenient approach which I want to share here. In short, I'm actually using Python's unittest, combined with the dynamic nature of the language, to run all kinds of tests.

Let's assume my tests are some sort of data files which have to be fed to a program. The output of the program is compared to some "expected results" file, or maybe is encoded in the data file itself in some way. The details of this are immaterial, but seasoned programmers usually encounter such testing rigs very frequently. It commonly comes up when the program under test is a data-transformation mechanism of some sort (compiler, encryptor, encoder, compressor, translator etc.)

So you write a "test runner". A script that looks at some directory tree, finds all the "test files" there, runs each through the transformation, compares, reports, etc. I'm sure all these test runners share a lot of common infrastructure - I know that mine do.

Why not employ Python's existing "test runner" capabilities to do the same?

Here's a very short code snippet that can serve as a template to achieve this:

import unittest

class TestsContainer(unittest.TestCase):
    longMessage = True

def make_test_function(description, a, b):
    def test(self):
        self.assertEqual(a, b, description)
    return test

if __name__ == '__main__':
    testsmap = {
        'foo': [1, 1],
        'bar': [1, 2],
        'baz': [5, 5]}

    for name, params in testsmap.iteritems():
        test_func = make_test_function(name, params[0], params[1])
        setattr(TestsContainer, 'test_{0}'.format(name), test_func)

    unittest.main()

What happens here:

The test class TestsContainer will contain dynamically generated test methods.
make_test_function creates a test function (a method, to be precise) that compares its inputs. This is just a trivial template - it could do anything, or there can be multiple such "makers" fur multiple purposes.
The loop creates test functions from the data description in testmap and attaches them to the test class.

Keep in mind that this is a very basic example. I hope it's obvious that testmap could really be test files found on disk, or whatever else. The main idea here is the dynamic test method creation.

So what do we gain from this, you may ask? Quite a lot. unittest is powerful - armed to its teeth with useful tools for testing. You can now invoke tests from the command line, control verbosity, control "fast fail" behavior, easily filter which tests to run and which not to run, use all kinds of assertion methods for readability and reporting (why write your own smart list comparison assertions?). Moreover, you can build on top of any number of third-party tools for working with unittest results - HTML/XML reporting, logging, automatic CI integration, and so on. The possibilities are endless.

One interesting variation on this theme is aiming the dynamic generation at a different testing "layer". unittest defines any number of "test cases" (classes), each with any number of "tests" (methods). In the code above, we generate a bunch of tests into a single test case. Here's a sample invocation to see this in action:

$ python dynamic_test_methods.py -v
test_bar (__main__.TestsContainer) ... FAIL
test_baz (__main__.TestsContainer) ... ok
test_foo (__main__.TestsContainer) ... ok

======================================================================
FAIL: test_bar (__main__.TestsContainer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "dynamic_test_methods.py", line 8, in test
    self.assertEqual(a, b, description)
AssertionError: 1 != 2 : bar

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=1)

As you can see, all data pairs in testmap are translated into distinctly named test methods within the single test case TestsContainer.

Very easily, we can cut this a different way, by generating a whole test case for each data item:

import unittest

class DynamicClassBase(unittest.TestCase):
    longMessage = True

def make_test_function(description, a, b):
    def test(self):
        self.assertEqual(a, b, description)
    return test

if __name__ == '__main__':
    testsmap = {
        'foo': [1, 1],
        'bar': [1, 2],
        'baz': [5, 5]}

    for name, params in testsmap.iteritems():
        test_func = make_test_function(name, params[0], params[1])
        klassname = 'Test_{0}'.format(name)
        globals()[klassname] = type(klassname,
                                   (DynamicClassBase,),
                                   {'test_gen_{0}'.format(name): test_func})

    unittest.main()

Most of the code here remains the same. The difference is in the lines within the loop: now instead of dynamically creating test methods and attaching them to the test case, we create whole test cases - one per data item, with a single test method. All test cases derive from DynamicClassBase and hence from unittest.TestCase, so they will be auto-discovered by the unittest machinery. Now an execution will look like this:

$ python dynamic_test_classes.py -v
test_gen_bar (__main__.Test_bar) ... FAIL
test_gen_baz (__main__.Test_baz) ... ok
test_gen_foo (__main__.Test_foo) ... ok

======================================================================
FAIL: test_gen_bar (__main__.Test_bar)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "dynamic_test_classes.py", line 8, in test
    self.assertEqual(a, b, description)
AssertionError: 1 != 2 : bar

----------------------------------------------------------------------
Ran 3 tests in 0.000s

FAILED (failures=1)

Why would you want to generate whole test cases dynamically rather than just single tests? It all depends on your specific needs, really. In general, test cases are better isolated and share less, than tests within one test case. Moreover, you may have a huge amount of tests and want to use tools that shard your tests for parallel execution - in this case you almost certainly need separate test cases.

I've used this technique in a number of projects over the past couple of years and found it very useful; more than once, I replaced a whole complex test runner program with about 20-30 lines of code using this technique, and gained access to many more capabilities for free.

Python's built-in test discovery, reporting and running facilities are very powerful. Coupled with third-party tools they can be even more powerful. Leveraging all this power for any kind of testing, and not just unit testing, is possible with very little code, due to Python's dynamism. I hope you find it useful too.

Fixing tests by writing more tests

2013-08-14T06:01:31-07:00

When a test fails, the first instinct is usually to dive right in - go through the code, sprinkle printouts and logging statements all around, fire up the debugger, and so on. But often, the correct approach is to write additional tests. Does this sound crazy? Well, it isn't. It's one of those true and tried techniques used by many experienced programmers.

Allow me to elaborate. Tests for software fall broadly into the following categories. This taxonomy is a blunt generalization and varies considerably with software size and other characteristics. But it's not important in itself - I just use it to illustrate my point. So starting with the most specific and ending with the most general, these are:

Unit tests
Module-level tests
Whole program tests
Integration tests (with other software)

From the first item (highly specific, targeted tests) on the list to the last (very general tests), coverage grows, as well as the complexity of debugging a failure. Debugging a unit test failure is usually quite easy. Integration tests can be devilishly difficult to debug, or even isolate and analyze. An unrelated component of the system may start failing intermittently when some other component (which passed all of its own tests) is changed. A compiler may start generating wrong code, but only when bootstrapped by compiling itself. It's those cases that send programmers crying to their therapist's couch.

So back to my original point. I posit that a great way to debug general tests is to write additional specific tests. Your module-level tests are failing? Write more unit tests. Your integration tests are failing? Write additional whole program tests, module tests or unit tests. The more specific the new tests are, the better. Remember that the most specific tests are the easiest to debug.

Debugging is all about the scientific method - you make assumptions and test them. Based on the results you make additional assumptions. Rinse. Repeat. While pondering about why something doesn't work - fortify your assumptions by writing tests for them. Facing a failing high-level test, try to write a more specific test that also fails. The benefits of this are two-fold. First, this will help you find the bug. Bugs are usually results of bad assumptions. This is especially true on the high generality level when pieces of software get integrated with others. More specific tests help verify those assumptions and are easier to debug. Second, you'll have more tests in the system when this bug-hunt ends, and that's always a good thing.