Measure input coverage in Java test cases

Question

Is it possible to measure how many distinct inputs were passed on to the methods of a class under test from existing test cases.

I'd like to measure something like code coverage, but for inputs instead.

Answer 1

I don't know of any COTS tools that compute input coverage, so I'd expect you to have to build a tool that did what you wanted.

My technical paper Branch Coverage for Arbitrary Languages Made Easy describes an approach for building test coverage tools for arbitrary languages using a Program transformation system to insert arbitrary probes into source code.

The paper is naturally focused on building code coverage, but the probe insertion technique is general and you can decide where to place probes and what they do. In your case, you want to place probes only at method entry, and you want the probes to track the input argument instances. The paper shows how to place probes anywhere by using a source code pattern to indicate the point of insertion; method entry is easy to describe as a pattern.

Capturing the input instances is more awkward but doable. You'll have to decide what an "input" is; is it just the argument values, or some kind of deep copy of the arguments? Likely what you need to do is create (per-method instrumented) an object type whose data members corresponds to the parameters, instantiate such an object with a copy (to appropriate depth) of the arguments, and store that object in a per-method hash table. (The transforamtion rules can insert all this once you know what you want to do as a code idiom). With all that, at execution, your hash table builds up the argument set, which is the key to what you want.

You can (continuously) count unique argument-set instances by controlling what happens when you insert duplicates into the hash table; that count (per method) can be managed in a global array that is exported at program completion. The paper discusses such a global array, and the various ways to export/display it in general.

Our line of test coverage and profilers are built using the techniques in the paper. The profilers keep counts/times in such global arrays (essentially what you need) and export them to a display engine that draws heat histograms, showing where the hot spots are. Those display engines are language and probe-data-source agnostic off-the-shelf in that they come in any of our (profiler) tools, including the Java profiler, so you could press one of them into service for the display task.