如何针对未定义的行为进行单元测试？

Question

I have a set of related classes which take a variety of inputs and produce expected outputs. These are ideal low-level candidates for unit testing, and all works well for valid inputs.

The difficulty comes with invalid inputs, particularly when attempting to remove items from collections which were not added, for which we currently have undefined behaviour: some of the classes will simply produce a rubbish result (GIGO ¹ wins) but some will throw an exception (perhaps a KeyNotFoundException ).

Given that there is no valid, consistent behaviour for these invalid inputs (it means that something has been mis-configured elsewhere and no sensible results can be produced) and that our API explicitly states that the caller must only remove something which they added, how can this be reflected in our unit tests?

It clearly cannot be a "test", as there is no defined behaviour (simply recording our current behaviour will be fragile if our implementation of any of them should change in the future), but I want to have a way to preclude the possibility of some zealous team member adding one in the future without being aware of the potential problems.

The unit test method for one of them currently looks something like this:

[TestCase("1", "2", "1", ExpectedResult = "|2|")]
[TestCase("1", "2", "2", ExpectedResult = "|1|")]
public object InsertTwoDeleteOne(string insertedValue1,
                                 string insertedValue2,
                                 string deletedValue1)
{
    // Apply tests here
}

The two ways I can see to deal with this are either to add explicit code in the test method along the lines of:

    if (deletedValue1 != insertedValue1 &&
        deletedValue1 != insertedValue2)
    {
        Assert.Fail("Invalid inputs");
    }

but that is "out of line" and less easy to see among the other test cases or else by adding a TestCase which is purely for documentation saying "don't run this", like this:

[TestCase("1", "2", "3", Ignore = true, Reason = "Invalid inputs")]

but that yields a "Skipped test" result which is untidy.

Is there anything better?

---

[Edit] The API in question is a public interface, and we have a number of implementations of it in our product: it is these implementations which I am in the process of updating the tests for. However, installations are free to write their own implementations as plugins (by creating their own assembly, implementing their own objects, and instantiating them through configuration), so our framework will ensure that the data is valid before calling them.

In our current model, it is unlikely that installations would re-use the objects and call them from their own code.

The reason why we have chosen not to concern ourselves with validating the data in each object is twofold:

1. It will, in our default product configuration, always receive data which has already been validated by the caller.
2. Performance: we are storing a lot of data here - currently limited to 100,000 rows of data (one of our objects per field in the row, so perhaps between 20 and 50 objects in total) but our customers are already asking about increasing that limit to 1,000,000 - so where we already store a dictionary of the data in our calling code so we can validate it there we would have to store a duplicate of it within these objects. That's between 20MB and 50MB if they are simply double s on current limits, or 200MB - 500MB on the projected future needs.

That's a massive overhead for something we don't currently need to do!

---

¹ Warning: Some people might prefer not to google for that in an office!

Answer 1

It might depend on the project's criticality and quality standards, but my gut feeling is that you should normally not let "undefined behaviour" creep into your system, especially if "rubbish results" are produced.

You say you fear that a zealous team member adds an inconsistent test to the suite. You might be assuming that team members will always add tests before writing production code and thus come across your "parapet" test, but what if they don't ? Wouldn't the primary safety measure be to prevent them from using the API the wrong way in the first place (ie handling edge cases properly) ?