How to run python unittests repeatedly from a script and collect results

Question

I cannot figure out how to run single unit tests from within a python script and collect the results.

Scenario: I have a battery of tests that check various methods producing various statistical distributions of different objects. The tests sometimes fail, as they should given that I am basically checking for particular kinds of randomness. I would like to run the tests repeatedly from a script or even from the interpreter and collect results for further analysis.

Suppose I have a module myTest.py with:

class myTest(unittest.TestCase):
    def setup(self):
       ...building objects, etc....
    def testTest1(self):
           ..........
    def testTest2(self):
           ..........

Basically I need to:

1. run the setup method
2. run testTest1 (say), 100 times
3. collect the failures
4. return the failures

The closest I got to was (using code from a similar question):

from unittest import TextTestRunner, TestSuite
runner = TextTestRunner(verbosity = 2)
tests = ['testTest1']
suite = unittest.TestSuite(map(myTest, tests))
runner.run(suite)

But this does not work, because:

runner.run(suite) does not run the setup method

and

I cannot catch the exception it throws when testTest1 fails

Answer 1

you simply need to add the test that you want to run multiple times to the suite.

Here is a complete code example. You can also see this code running in an interactive Python console to prove that it does actually work .

import unittest
import random

class NullWriter(object):
    def write(*_, **__):
        pass

    def flush(*_, **__):
        pass

SETUP_COUNTER = 0

class MyTestCase(unittest.TestCase):

    def setUp(self):
        global SETUP_COUNTER
        SETUP_COUNTER += 1

    def test_one(self):
        self.assertTrue(random.random() > 0.3)

    def test_two(self):
        # We just want to make sure this isn't run
        self.assertTrue(False, "This should not have been run")


def suite():
    tests = []
    for _ in range(100):
        tests.append('test_one')

    return unittest.TestSuite(map(MyTestCase, tests))

results = unittest.TextTestRunner(stream=NullWriter()).run(suite())
print dir(results)
print 'setUp was run', SETUP_COUNTER, 'times'