Is it possible to make a sub-test “expected failure” in Python unittest?

Question

In Python's unit test framework it is possible to declare that a test is expected to fail using the expectedFailure decorator. I use 'expected failures' to denote bugs in the implementation that need to be fixed at some later time, but are not critical. However, the expcetedFailure decorator applies to the whole test. I was wondering if there is a way to declare that a sub-test is expected to fail?

For example, consider the following test for some is_prime routine that does not declare 2 as a prime number (perhaps because it first filters out all even numbers):

class NumbersTest(unittest.TestCase):
    def test_primes(self):
        for i in range(2, 13):
            with self.subTest(i=i):
                if i in [2, 3, 5, 7, 11]:
                    self.assertTrue(is_prime(i))
                else:
                    self.assertFalse(is_prime(i))

The test obviously fails for i=2 because of the bug in is_prime , but as I am sure that for the now I will never call is_prime on 2 , I would not make a big fuss out of it for the moment. I would like to have a means to declare that only the sub test for i=2 is expected to fail. If I decorate the entire test_primes with @expectedFailure then the routine does not get tested at all, so if someone changes the implementation and breaks its functionality it will not be noticed.

One possiblity is to use skipTest instead:

class NumbersTest(unittest.TestCase):
    def test_primes(self):
        for i in range(2, 13):
            with self.subTest(i=i):
                if i == 2:
                    self.skipTest("`is_prime` does not handle 2 correctly")
                if i in [2, 3, 5, 7, 11]:
                    self.assertTrue(is_prime(i))
                else:
                    self.assertFalse(is_prime(i))

But I am not too happy with skipping the test, as a test should normally be skipped if it cannot be performed (eg, because of unavailable resources, etc.). In this case nothing prohibits us from running the test for i==2 , it only fails and the failure is a known bug.

Answer 1

I realize this is kind of old, but if you can refactor the test to a method that accepts inputs, you can achieve this with minimal duplication

class NumbersTest(unittest.TestCase):
    def test_primes(self):
        for i in range(2, 13):
            with self.subTest(i=i):
                if i in [2, 3, 5, 7, 11]:
                    self.assertTrue(is_prime(i))
                else:
                    self.assertFalse(is_prime(i))

becomes

class NumbersTest(unittest.TestCase):
    def _test_primes(self, i, expectation):
       with self.subTest(i=i, expectation=expectation):
             self.assertEqual(is_prime(i), expectation)

    @unittest.expectedFailure
    def test_failing_primes(self):
         values = [2]
         for i in values:
             self._test_primes(i, False)

    def test_primes(self):
         for i in range(3, 13):
            if i in [3, 5, 7, 11]:
                self._test_primes(i, True)
            else:
                self._test_primes(i, False)

Answer 2

I had the same wish for marking subtests as expected failures. This is especially relevant to me because my setUp/tearDown sets up a new virtual machine (qemu or dosemu2) with the application for each test which usually takes longer than a second, so bundling many tests as subtests helps keep the time down.

I went with the following code :

        for (msg, inst, code, success) in asmtests:
            with self.subTest(msg = msg + ": " + inst):
                actuallysucceeded = 0
                try:
                    # do the test
                    actuallysucceeded = 1
                except AssertionError as e:
                    if success:
                        raise e
                    else:
                        pass
                if actuallysucceeded and not success:
                    raise AssertionError("unexpected success")

If the test raises an AssertionError (which seems to be what unittest's asserts do on failures) we check the success flag: If it is set (expected success) we simply re-raise the exception for unittest to handle. Otherwise we ignore this exception. If the actuallysucceeded flag is set afterwards and the success flag is not set then we manually raise an AssertionError .

This program logic negates the sense of an expected failure subtest and allows other subtests to work as expected using the same handling.