From nose to testr: Output Capture

The thing that drove me most crazy when I was trying to write nose-testresources (a plugin for nose that would put ResourcedTestCases inside OptimisedTestSuites) was that I lost nose's default behaviour of capturing log output and stdout. I put a lot of work into logging information that would help me troubleshoot failures, and was flailing without it.

I talked to Robert Collins, the primary maintainer of this chain of tools, and he suggested I use testtools.TestCase, which supports addFixture() and python-fixtures, which includes FakeLogger. Now, this transition turned out to be a little bit sticky - both testtools.TestCase and testresources.ResouredTestCase inherit from unittest.TestCase, so dual inheritance should have worked, but I was running into this strange problem that resources were never being cleaned up. It turned out that testtools.TestCase calls setUp() but doesn't call tearDown() - it uses addCleanup() for that purpose. As part of the debugging process for this issue, I axed the multiple inheritance an just created a ResourcedTestCase that follows testtools.TestCase norms as follows:


from testresources import setUpResources, tearDownResources, _get_result
import testtools
import fixtures

class ResourcedTestCase(testtools.TestCase):
    """Inherit from testtools.TestCase instead of unittest.TestCase in order
    to have self.useFixture()."""

    def setUp(self):
        # capture output
        FORMAT = '%(asctime)s [%(levelname)s] %(name)s %(lineno)d: %(message)s'
        self.useFixture(fixtures.FakeLogger(format=FORMAT))
        # capture stdout
        stdout = self.useFixture(fixtures.StringStream('stdout')).stream
        self.useFixture(fixtures.MonkeyPatch('sys.stdout', stdout))
        # capture stderr
        stderr = self.useFixture(fixtures.StringStream('stderr')).stream
        self.useFixture(fixtures.MonkeyPatch('sys.stderr', stderr))
        super(ResourcedTestCase, self).setUp()
        setUpResources(self, self.resources, _get_result)
        self.addCleanup(tearDownResources, self, self.resources, _get_result)

With this new base class, my tests now capture logging, stdout, and stderr under both testr and nose, with and without nose-testresources.

From nose to testr: More Flexible Test Discovery

testrepository requires a results stream in subunit format, and the python implementation of that is testtools. testtools uses the standard unittest.TestLoader. This loader will let you discover tests from one specific directory, or run tests from multiple fully specified test modules. I am accustomed to nose, which lets you specify an arbitrary number of directories and modules, so I modified the TestLoader to bend it to my will.

import os
import unittest

class TestLoader(unittest.TestLoader):
    """Test loader that extends unittest.TestLoader to:

    * support names that can be a combination of modules and directories
    """

    def loadTestsFromNames(self, names, module=None):
        """Return a suite of all tests cases found using the given sequence
        of string specifiers. See 'loadTestsFromName()'.
        """
        suites = []
        for name in names:
            if os.path.isdir(name):
                top_level = os.path.split(name)[0]
                suites.extend(self.discover(name, top_level_dir=top_level))
            else:
                suites.extend(self.loadTestsFromName(name, module))
        return self.suiteClass(suites)

Then it just became a matter of borrowing from the subunit runner script and instantiating SubunitTestProgram with the new loader.

SubunitTestProgram(module=None, argv=sys.argv, testRunner=SubunitTestRunner,

        stdout=sys.stdout, testLoader=TestLoader())

From nose to testr: Rationale

When I started at SwiftStack, my first priority was to write as many automated tests as I could. I wanted to use a framework/runner already familiar to the developers, has good online support / community, and unittest-compatible because it was almost certain I'd have to change frameworks eventually. I chose nose.

Over the course of the next 6 months, I was a little bit too successful at writing automated tests, and I arrived at the point that a full test run in the default environment took 13 hours, there was not enough time in the day to run the tests against all of the different environments, and I was unable to effectively test the effect of changes to the automation infrastructure before merging.

I needed to speed up the tests.

While there were some tests where the test itself was slow, the bigger problem was setup for the tests. The fixtures I needed for many of the tests were very expensive, and even if nose let me set them up on a per-class basis rather than a per-test basis, it was still adding time. Enter testresources, whose OptimisedTestSuite analyses the resources required by each test, orders them according to those resources, and shares resources across tests.

I tried for quite some time to get testresources to work with nose...I wrote a nose plugin that allowed me to get by, but there were some distinct problems, and the primary maintainer for nose was not available (he has recently requested hand-off to a new maintainer). I eventually decided to go with what the OpenStack community had been telling me all along - that I should use the testrepository (testr) / subunit / testtools / testresources collection of tools.

There have been challenges. I'm still not done. But I've got insights to share:

• More Flexible Test Discovery
• Output Capture
• Capturing and Recovery from OptimisedTestSuite errors

And more to come.