smoke tests

April 26th, 2005 at 9:22 pm

It’s funny that I first learned what smoke tests are just a few minutes ago. autrijus is mentioning them in his pugs reports all the time, but I just thought it’s some slang for tests ;-/

Anyway, the scenario is: daily builds are good, so is frequent testing.

However, sometime testing takes time. Testbases grow big (which is good) and sometimes take a long time to run – up to days for large projects that use tests to exercise boundary cases. This is a problem – how do you build daily with such long tests ?

Solution: smoke tests – a small and quick set of tests that take a hour or two to run. These assure that the system “doesn’t smoke” after a build – i.e. doesn’t crash the machine, throw a segfault, declare that 2+2=5. Just a few trivial tests can go a long way cleaning up major and basic bugs.

So, you build daily + run a smoke-test suite. If it fails – you found a bug and you know it’s new – was entered today -> easy to fix. If it passes, fine. Comprehensive tests run all the time and are ready the test the program more thoroughly.

Note: smoke tests are not too applicable to small programs that don’t have huge testbases.

Related posts:

  1. Writing tests first
  2. Joel’s 12 steps for better code
  3. unit testing framework – cxxtext
  4. What TDD means to me
  5. pycparser v1.06 released

2 Responses to “smoke tests”

  1. ripper234No Gravatar Says:

    What I like to do is separate test cases by runtime. I don’t like to pre-designate tests as “basic” or “advanced” (“smoke” or “other”), I just measure the runtime of tests. It depends on your project, but you should have one group of tests that runs in 1-10 minutes total. Any new test that is short (under 2 seconds – adjustable-rule-of-thumb) is added to this group.

    Another set of tests is “medium weight” – these might take up to an hour to run (under ideal conditions. Unfortunately schedule doesn’t always permit you to optimize tests so. Our current “medium” test set takes about 2 hours to run – not good). A good test duration for this group is 2-120 seconds.

    Tests longer than 2 minutes should probably go into a different category. This way, on every check-in, you have several seperate test suits gnawing at your code. Regardless of this division, you should make sure your test runner informs of errors immediately, so you won’t have to wait for the entire test run to find problems

  2. elibenNo Gravatar Says:

    Ron, this is an interesting idea, and probably a better definition of what smoke tests are. I guess that the shortest tests you run all the time are the smoke tests. It indeed is better to make the classification by run-time and not functionality, because the two are certainly not correlated linearly.