From nose to testr: Rationale

When I started at SwiftStack, my first priority was to write as many automated tests as I could. I wanted to use a framework/runner already familiar to the developers, has good online support / community, and unittest-compatible because it was almost certain I'd have to change frameworks eventually. I chose nose.

Over the course of the next 6 months, I was a little bit too successful at writing automated tests, and I arrived at the point that a full test run in the default environment took 13 hours, there was not enough time in the day to run the tests against all of the different environments, and I was unable to effectively test the effect of changes to the automation infrastructure before merging.

I needed to speed up the tests.

While there were some tests where the test itself was slow, the bigger problem was setup for the tests. The fixtures I needed for many of the tests were very expensive, and even if nose let me set them up on a per-class basis rather than a per-test basis, it was still adding time. Enter testresources, whose OptimisedTestSuite analyses the resources required by each test, orders them according to those resources, and shares resources across tests.

I tried for quite some time to get testresources to work with nose...I wrote a nose plugin that allowed me to get by, but there were some distinct problems, and the primary maintainer for nose was not available (he has recently requested hand-off to a new maintainer). I eventually decided to go with what the OpenStack community had been telling me all along - that I should use the testrepository (testr) / subunit / testtools / testresources collection of tools.

There have been challenges. I'm still not done. But I've got insights to share:

• More Flexible Test Discovery
• Output Capture
• Capturing and Recovery from OptimisedTestSuite errors

And more to come.