<<< Back to blog Archives

Python unit testing: parametrized test cases

August 2nd, 2011 at 9:29 am

Python’s standard unittest library is great and I use it all the time. One thing missing from it, however, is a simple way of running parametrized test cases. In other words, you can’t easily pass arguments into a unittest.TestCase from outside.

Consider the use case: I have some TestCase I want to invoke several times, each time passing it a different argument.

One approach often mentioned is create a base TestCase for the bulk functionality and derive sub-classes from it for variations. But this isn’t flexible enough – what if you want to add parameters from the outside (command-line) or test with a large amount of parameters?

Fortunately, Python is dynamic enough (and unittest flexible enough) to allow a relatively straightforward solution.

Here’s a class that makes it possible:

import unittest

class ParametrizedTestCase(unittest.TestCase):
    """ TestCase classes that want to be parametrized should
        inherit from this class.
    """
    def __init__(self, methodName='runTest', param=None):
        super(ParametrizedTestCase, self).__init__(methodName)
        self.param = param

    @staticmethod
    def parametrize(testcase_klass, param=None):
        """ Create a suite containing all tests taken from the given
            subclass, passing them the parameter 'param'.
        """
        testloader = unittest.TestLoader()
        testnames = testloader.getTestCaseNames(testcase_klass)
        suite = unittest.TestSuite()
        for name in testnames:
            suite.addTest(testcase_klass(name, param=param))
        return suite

Before I explain how this works, here’s a sample usage. Let’s define some test case that can be parametrized with an extra param argument:

class TestOne(ParametrizedTestCase):
    def test_something(self):
        print 'param =', self.param
        self.assertEqual(1, 1)

    def test_something_else(self):
        self.assertEqual(2, 2)

Note how nothing except inheriting ParametrizedTestCase is required. self.param automagically becomes available in all test methods (as well as in setUp, tearDown, etc.)

And here is how to create and run parametrized instances of this test case:

suite = unittest.TestSuite()
suite.addTest(ParametrizedTestCase.parametrize(TestOne, param=42))
suite.addTest(ParametrizedTestCase.parametrize(TestOne, param=13))
unittest.TextTestRunner(verbosity=2).run(suite)

As expected, we get:

test_something (__main__.TestOne) ... param = 42
ok
test_something_else (__main__.TestOne) ... ok
test_something (__main__.TestOne) ... param = 13
ok
test_something_else (__main__.TestOne) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Now, a word on how ParametrizedTestCase works. It’s a subclass of unittest.TestCase, and the parametrization is done by defining its own constructor, which is similar to TestCase‘s constructor but adds an extra param argument. This param is then saved as the instance attribute self.param. Test cases interested in being parametrized should then derive from ParametrizedTestCase.

To actually create the parametrized test, ParametrizedTestCase.parametrize should be invoked. It accepts two arguments:

  1. A subclass of ParametrizedTestCase – essentially our custom test case class
  2. The parameter we want to pass to this instance of the test case

And then uses the test name discovery facilities available in unittest.TestLoader to create the tests and parametrize them.

As you can see in the usage example, the approach is easy to use and works quite well. I have a couple of qualms with it, however:

  • It directly calls TestCase.__init__, which isn’t an officially documented feature.
  • When different parametrized instances of our test case run, we can’t know which parameter was passed. I suppose some hack can be crafted that attaches the parameter value to the test name, but this is very much application-specific.

I’m really interested in feedback on this post. Could this be done better? Any alternative approaches to achieve the same effect?

Related posts:

  1. unit testing framework – cxxtext
  2. testing and test-benches
  3. Automating boring testing activities with tox
  4. Book review: “Test driven development by example”, Kent Beck
  5. smoke tests

This entry was posted on Tuesday, August 2nd, 2011 at 09:29 and is filed under Python. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

10 Responses to “Python unit testing: parametrized test cases”

  1. Austin BinghamNo Gravatar Says:
    August 2nd, 2011 at 12:03

    There was some discussion of this topic on the issue tracker recently:

    http://bugs.python.org/issue12600

    I use a form of testcase parameterization which is similar to what Michael Foord describes toward the end, and I’m generally quite happy with it. Essentially, you define your “testcase” without actually subclassing unittest.TestCase. Then a testcase factory takes in your class along with a set of parameterizations, and it creates new TestCase subclasses with your parameters added as class-level attributes.

    Ultimately it achieves much the same results as what you’ve got. It does avoid the need to create a specialized TestCase subclass (your ParameterizedTestCase), but on the other hand it requires the factory method.

    One benefit I like with my approach is that the TestCase subclasses are actually named differently based on their parameterization. So, if my base testcase class is named MyTests, the factory will generate classes named MyTests_param1, MyTests_param2 (where “param1″ and “param2″ are just standins, and you can call them whatever you like in your param specification.)

    One upside to this is that the output generated when running the tests can be cut-and-pasted very easily to run a specific parameterization of a specific test, or all tests in a specific parameterization. With your approach, I’m not sure how I would do those kinds of things without modifying the code.

  2. elibenNo Gravatar Says:
    August 2nd, 2011 at 12:31

    Austin,

    This is an interesting approach, thanks for the link. Thinking about it, it appears to offer a different angle of parametrization – the one I mentioned early in my post, made easier to implement with the factory Michael proposed. You create a family of related TestCase classes this way.

    The parametrization presented in my post allows you to run a given TestCase several times, each time passing it different arguments. So IIUC the two approaches aren’t exactly alternatives.

    And to continue this thought, it’s also a point in favor of not adding this directly to unittest, since parametrization means different things for different people, and adding all kinds of parametrization to unittest would probably over-complicate it.

  3. Austin BinghamNo Gravatar Says:
    August 2nd, 2011 at 13:00

    True, the implementations of the two approaches can result in significantly different behaviors. One in particular (at least for me) is the “many subclasses” approach ensures that setUpClass() and tearDownClass() are run for each parameterization/subclass. In your approach, I *believe* that the unittest machinery will only run those once over all parameterizations; unittest just tracks the last TestCase used and only runs setUp/tearDownClass() when that value changes.

    And yes, based on the issue discussion and such, I also agree that testcase parameterization is best left to external implementation. Time and energy permitting, I’d like to package up the testcase-factory stuff and put it in the cheeseshop.

  4. Ken DyckNo Gravatar Says:
    August 2nd, 2011 at 17:28

    Whenever I’ve run into the parameterization problem in unittest, the superclass-with-subclass-variants approach has sufficed. Thanks for presenting this more general alternative.

    One more generalization you could make would be to define the param parameter of the ParameterizedTestCase.__init__() with the ** form so that it can take any number of named parameters, storing each as an instance variable in the test case.

  5. elibenNo Gravatar Says:
    August 3rd, 2011 at 04:50

    Ken,

    Thanks for the feedback. Yes, a more general parameter passing scheme is a natural extension to this technique.

  6. Pere MartirNo Gravatar Says:
    August 8th, 2011 at 12:20

    It’s important that the names of “each” parameterized test methods are different if the test results will be parsed by automated tools. For example, we use a Jenkins plugin to parse the generated JUnit XML and display the test result nicely. We use py.test which generates the method name automatically from the names and values of the parameters.

    http://pytest.org/latest/funcargs.html#parametrized-test-functions

  7. elibenNo Gravatar Says:
    August 9th, 2011 at 05:46

    Pere Martir,

    Thanks for the feedback. I’ve intentionally avoided any unit-testing libraries, to stay only with stdlib/unittest, but I will take a look at py.test for inspiration.

  8. Ben FinneyNo Gravatar Says:
    August 14th, 2011 at 02:44

    How does this approach compare with the existing and widely-used ‘testscenarios’ library?

  9. elibenNo Gravatar Says:
    August 14th, 2011 at 05:45

    Ben,

    I wasn’t aware of this library’s existence. Will take a look, thanks.

  10. Mohammad SadiqNo Gravatar Says:
    December 13th, 2012 at 05:48

    Here is another way, though the reporting might not be very clear:

    class TestSuite(unittest.TestCase):
    def use_parameters(self, param1, param2, param3):

    def test_something(self):
    self.use_parameters(a, b, c)
    self.use_parameters(a1, b1, c1)
    self.use_parameters(a2, b2, c2)

    This will show up as one test case in the report, but will run all the tests.

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)