How to run the python unittest N number of times

Question

I have a python unittest like below , I want to run this whole Test N Number of times

class Test(TestCase)

    def test_0(self):
            .........
            .........
            .........

Test.Run(name=__name__)

Any Suggestions?

Answer 1

You can use parameterized tests. There are different modules to do that. I use nose to run my unittests (more powerful than the default unittest module) and there's a package called nose-parameterized that allows you to write a factory test and run it a number of times with different values for variables you want. If you don't want to use nose, there are several other options for running parameterized tests.

Alternatively you can execute any number of test conditions in a single test (as soon as one fails the test will report error). In your particular case, maybe this makes more sense than parameterized tests, because in reality it's only one test, only that it needs a large number of runs of the function to get to some level of confidence that it's working properly. So you can do:

import random
class Test(TestCase)

    def test_myfunc(self):
        for _ in range(100):
            input = random.random() 
            self.assertEquals(input, input + 2)


Test.Run(name=__name__) 

Answer 2

Why because... the test_0 method contains a random option.. so each time it runs it selects random number of configuration and tests against those configurations. so I am not testing the same thing multiple times..

Randomness in a test makes it non-reproducible. One day you might get 1 failure out of 100, and when you run it again, it's already gone.

Use a modern testing tool to parametrize your test with a sequential number, then userandom.seed to have a random but reproducible test case for each number in a sequence.

portusato suggests nose, but pytest is a more modern and popular tool:

import random, pytest

@pytest.mark.parametrize('i', range(100))
def test_random(i):
    orig_state = random.getstate()
    try:
        random.seed(i)
        data = generate_random_data()
        assert my_algorithm(data) == works
    finally:
        random.setstate(orig_state)

pytest.mark.parametrize “explodes” your single test_random into 100 individual tests — test_random[0] through test_random[99] :

$ pytest -q test.py
....................................................................................................
100 passed in 0.14 seconds

Each of these tests generates different, random, but reproducible input data to your algorithm. If test_random[56] fails, it will fail every time , so you will then be able to debug it.

Answer 3

If you don't want your test to stop after the first failure, you can use subTest .

class Test(TestCase):
    def test_0(self):
        for i in [1, 2, 3]:
            with self.subTest(i=i):
                self.assertEqual(squared(i), i**2)

Docs