Python unittest: run same test over a list of values, but treat them as separate tests

Question

I'm writing a test to validate that my program is able to solve the problem at hand for different levels of complexity. The expected outcome is always the same (solution is complete), so a single test definition works for all problems.

How do I run the same test for a list of values read from a file, but tell unittest to treat each of these problems as a separate test so I can pinpoint all failing/passing cases? (Preferably no external libraries)

To avoid explicitly having test_solution_1, test_solution_2... test_solution_n , my initial thought was to have a for loop run through each item of the list, and run the assertions one at a time:

class TestProblem(unittest.TestCase):
    def test_all(self):
        results = Counter()
        rng = 50
        for i in range(rng):
            p = Problem(matrix_index=i)  # generate instance of problem.
            p.solve()
            results[p.is_complete()] += 1  # log result of test.
            self.assertTrue(p.is_complete(), msg="Failed at %s" % str(i))
        # Now ensure all instances passed (useful when above assertion not included).
        self.assertEqual(results[True], rng, msg="SCORE: %s / %s" % (results[True], rng))

The problem with this approach is that the first failure stops the rest from running, so its more difficult to get a complete picture of what is wrong.

Answer 1

Use the subTest() context manager to distinguish tests inside a single test body. (Python 3.4+)

    def test_all(self):
        results = Counter()
        rng = 50
        for i in range(rng):
            with self.subTest(i=i):  # <----------------------
                p = Problem(matrix_index=i)  # generate instance of problem.
                p.solve()
                results[p.is_complete()] += 1  # log result of test.
                self.assertTrue(p.is_complete())
        self.assertEqual(results[True], rng, msg="SCORE: %s / %s" % (results[True], rng))

Result from PyCharm:

You can see individual results for each case and see all failed cases at once. Notice that its still treated as a single logical test (" Ran 1 test "), which makes sense since its testing the same functionality. Each case is treated as a sub test.

---

Other things to consider:

• If using an older version of Python or you want to use another library, you can look at ddt to add test data via decorator @data(3, 4, 12, 23) .