Generating tests for all-pairs testing

Question

I have a short (Java) method with several parameters that performs some non-intuitive computation. Writing unit tests for this method in order to achieve code coverage is trivial, but the tests don't cover the subtleties of the method. I am looking for a testing framework or some other tool that will allow me to do all-pairs (all-tuples) testing for this method, based on partitions that I define for the parameters.

For example, suppose I have the following method:

int foo(int begin, int end, int first, int last) {
    ...
}

I already know some constraints, which may be enforced externally:

begin <= end
first <= last

I want to define explicit equivalence relations, based on my own knowledge of the parameters. For example:

begin == MIN
begin > MIN && begin < 0
begin == 0
begin == 1
begin > 1 && begin < MAX
begin == MAX

I would also like to define equivalence relations involving more than one parameter. For example:

begin + 1 < end
begin + 1 == end
begin == end

A combination to test is the selection of an equivalence class from each of the equivalence relations (eg, begin == 0 from the first relation and begin + 1 < end from the second relation), such that the constraints are satisfiable. [Incidentally, I realize satisfiability is NP-complete. I'm willing to live with it, given combinations that comprise a relatively small set of constraints.] With several parameters, the number of combinations becomes unwieldy. Manually writing a test for each combination and finding arguments that satisfy the resultant set of constraints is tedious and error prone.

I'm looking for a testing framework that can automatically test each combination, or a test generator that can generate a test for each logically consistent combination (invariant checks to be inserted by yours truly). This is different from test-generation tools like CodePro , which generate test cases from the actual implementation. Effectively, I want to generate tests for an interface of the method, rather than its implementation. Not only would this allow me to write the tests before the method is implemented (TDD), but it would ensure the method is comprehensively tested even after its later modification.

Does such a tool exist, or can something be kludged together? Or perhaps I'm approaching this the wrong way, and you have another suggestion...

Answer 1

Wouldn't simple nested loops, where a test method is invoked at each innermost loop iteration, fit the bill (or at least mostly fit the bill)?

Something like:

int[] ends = new int[] {-50, -10, -1, 0, 1, 50, MIN, MAX};
for (int end : ends) {
    int[] begins = new int[] {end - 10, end - 1, end, MIN, MAX};
    for (int begin : begins) {
        if (begin <= end && begin >= MIN && begin <= MAX) {
             testFoo(begin, end);
        }
    }
}

Answer 2

有关此方面的一些想法，请参阅David Saff在理论测试方面 的工作。

Answer 3

Can you use a tool outside your test framework? If so, I've used James Bach's AllPairs with good results. There's also Hexawise .

If you weren't aware of it, http://pairwise.org is a great place to read on pairwise and combinatorial testing and tools!

Answer 4

If you are still interested in libraries that perform combinatorial tests under JUnit framework. I recently implemented a library 'JCUnit', which does exactly it.

It generates test cases under JUnit and run the test method with them.

http://dakusui.github.io/jcunit/

Answer 5

I may suggest that if you TDDed the algorithm, then you already likely have sufficient test coverage. If anyone subsequently comes in and changes the algorithm, you should see at least one of the original tests break. If you have some edge conditions to test, you could write a couple, but if these edges aren't interesting enough to turn red once you've added them, then they probably aren't interesting enough to keep around and duplicate the existing tests. Redundant tests are less valuable and I tend to delete them.