How do you generate dynamic (parameterized) unit tests in python?

Question

I have some kind of test data and want to create a unit test for each item. My first idea was to do it like this:

import unittest

l = [[\"foo\", \"a\", \"a\         


        

Answer 1

As of Python 3.4 subtests have been introduced to unittest for this purpose. See the documentation for details. TestCase.subTest is a context manager which allows one to isolate asserts in a test so that a failure will be reported with parameter information but does not stop the test execution. Here's the example from the documentation:

class NumbersTest(unittest.TestCase):

def test_even(self):
    """
    Test that numbers between 0 and 5 are all even.
    """
    for i in range(0, 6):
        with self.subTest(i=i):
            self.assertEqual(i % 2, 0)

The output of a test run would be:

======================================================================
FAIL: test_even (__main__.NumbersTest) (i=1)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_even (__main__.NumbersTest) (i=3)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0

======================================================================
FAIL: test_even (__main__.NumbersTest) (i=5)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtests.py", line 32, in test_even
    self.assertEqual(i % 2, 0)
AssertionError: 1 != 0

This is also part of unittest2, so it is available for earlier versions of Python.

Answer 2

I use metaclasses and decorators for generate tests. You can check my implementation python_wrap_cases. This library doesn't require any test frameworks.

Your example:

import unittest
from python_wrap_cases import wrap_case


@wrap_case
class TestSequence(unittest.TestCase):

    @wrap_case("foo", "a", "a")
    @wrap_case("bar", "a", "b")
    @wrap_case("lee", "b", "b")
    def testsample(self, name, a, b):
        print "test", name
        self.assertEqual(a, b)

Console output:

testsample_u'bar'_u'a'_u'b' (tests.example.test_stackoverflow.TestSequence) ... test bar
FAIL
testsample_u'foo'_u'a'_u'a' (tests.example.test_stackoverflow.TestSequence) ... test foo
ok
testsample_u'lee'_u'b'_u'b' (tests.example.test_stackoverflow.TestSequence) ... test lee
ok

Also you may use generators. For example this code generate all possible combinations of tests with arguments a__list and b__list

import unittest
from python_wrap_cases import wrap_case


@wrap_case
class TestSequence(unittest.TestCase):

    @wrap_case(a__list=["a", "b"], b__list=["a", "b"])
    def testsample(self, a, b):
        self.assertEqual(a, b)

Console output:

testsample_a(u'a')_b(u'a') (tests.example.test_stackoverflow.TestSequence) ... ok
testsample_a(u'a')_b(u'b') (tests.example.test_stackoverflow.TestSequence) ... FAIL
testsample_a(u'b')_b(u'a') (tests.example.test_stackoverflow.TestSequence) ... FAIL
testsample_a(u'b')_b(u'b') (tests.example.test_stackoverflow.TestSequence) ... ok

Answer 3

Super late to the party, but I had trouble making these work for setUpClass.

Here's a version of @Javier's answer that gives setUpClass access to dynamically allocated attributes.

import unittest


class GeneralTestCase(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        print ''
        print cls.p1
        print cls.p2

    def runTest1(self):
        self.assertTrue((self.p2 - self.p1) == 1)

    def runTest2(self):
        self.assertFalse((self.p2 - self.p1) == 2)


def load_tests(loader, tests, pattern):
    test_cases = unittest.TestSuite()
    for p1, p2 in [(1, 2), (3, 4)]:
        clsname = 'TestCase_{}_{}'.format(p1, p2)
        dct = {
            'p1': p1,
            'p2': p2,
        }
        cls = type(clsname, (GeneralTestCase,), dct)
        test_cases.addTest(cls('runTest1'))
        test_cases.addTest(cls('runTest2'))
    return test_cases

Outputs

1
2
..
3
4
..
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK

Answer 4

You can use nose-ittr plugin (pip install nose-ittr).

It's very easy to integrate with existing tests, minimal changes (if any) are required. It also supports nose multiprocessing plugin.

Not that you can also have a customize setup function per test.

@ittr(number=[1, 2, 3, 4])   
def test_even(self):   
    assert_equal(self.number % 2, 0)

It is also possible to pass nosetest parameters like with their build-in plugin attrib, this way you can run only a specific test with specific parameter:

nosetest -a number=2

Answer 5

The metaclass-based answers still work in Python3, but instead of the __metaclass__ attribute one has to use the metaclass parameter, as in:

class ExampleTestCase(TestCase,metaclass=DocTestMeta):
    pass

Answer 6

load_tests is a little known mechanism introduced in 2.7 to dynamically create a TestSuite. With it, you can easily create parametrized tests.

For example:

import unittest

class GeneralTestCase(unittest.TestCase):
    def __init__(self, methodName, param1=None, param2=None):
        super(GeneralTestCase, self).__init__(methodName)

        self.param1 = param1
        self.param2 = param2

    def runTest(self):
        pass  # Test that depends on param 1 and 2.


def load_tests(loader, tests, pattern):
    test_cases = unittest.TestSuite()
    for p1, p2 in [(1, 2), (3, 4)]:
        test_cases.addTest(GeneralTestCase('runTest', p1, p2))
    return test_cases

That code will run all the TestCases in the TestSuite returned by load_tests. No other tests are automatically run by the discovery mechanism.

Alternatively, you can also use inheritance as shown in this ticket: http://bugs.python.org/msg151444