何时为状态测试选择基于示例的测试和基于属性的测试

Question

I'm doing unit testing mostly these days for Android SDK in Android Studio and using Jqwik which is a Property-Based testing tool on the JUnit Platform.

While exploring the different test techniques approach with my seniors, I learned about example-based testing and property-based testing. I want to know when to choose which one.

Answer 1

Purpose of sharing this : To double-check with the community if I'm thinking in the right direction.

Notations: EBT = Example-Based Testing; PBT = Property-Based Testing;

What I understood : EBT is done to check if features are working as they are supposed to be, while PBT takes to one level above, by falsifying the given preconditions(inputs) against the invariants(domain behavior)

What I feel : I feel like the Jqwik tool is like a parameterized test of JUnit but on steroids

Example bases testing

Pros:

• Easy visibility of which test passes with their specified names by the programmer in contrast to PBT just shows the total count of tries tho we can log the generated test cases but the JUnit way to is more visually appealing and easy to traverse and read through

• Can easily read and understand the inputs as mentioned in the test case itself in contrast to PBT where we have to look into the Provider method from where Jqwik get all the Arbitraries.

• Good for testing state machine where there are not any combination of dependencies.

• Might be easy for the Non-Tech person to understand the working of state machine if he knows how that domain component works

Cons :

• Usually don't cover all the edges cases, as author of Jqwik also said in presentation about EBT that “ It cant keep its promise”
• Not the best option when when there are tons of combinations to cover
• In example based testing we really on the gut, because of this every programmer cover the test with different test cases, in contrast PBT is kind a exhaustive.

Properties based testing

Pros:

• Good for random and specific input data generation.
• Pretty good when there are a lot of combinations of the unique test cases, for that it works just dandy.
• It is not all about exhaustive set input combinations but it's one subset of PBT

Cons:

For PBT, mainly I've observed cons are around the readability of test cases:

• Hard to read and traverse through the generated test cases as its just logs
• What are the current inputs for the test cases, we have to look into the Provides annotation functions.

Answer 2

For me it's usually not an either-or question, but examples and properties often complement each other. Whereas examples are good starting points for the intended behaviour of the code under test, properties serve to give me sufficient trust in the breadth of functionality and expected behaviour in edge and corner cases.

There are situations where examples are better replaced by properties, eg when many examples can be translated into a single or just a few properties. Even then I tend to keep some examples, because they are easier to understand.