Why to group related tests

Question

What are the use cases for grouping tests using TestNG groups or JUnit categories?

I usually groups by tests by function using JUnit categories: unit-tests, integration-tests, etc. But at my current team, we just had a conversation and the team decided we want to run all the tests all the time because they don't see any benefits for grouping tests.

So I'm curious to know if people here group their tests and why.

Answer 1

There are multiple kinds of tests and JUnit and TestNG supports them all. In the case of Unit Tests, you can run them all and get the feedback within seconds or minutes. When it comes down to integration or end-to-end tests you might want to group your tests because of the time factor.

Let's say, you have unit tests, API tests, and even GUI tests.

While unit tests can run with each build, the other tests might take too long.

Example:

In one project I had over 300 GUI tests and it took 2 hours to run them in parallel. When we introduced a hotfix for a specific component and we needed to deploy it as fast as we could - we would run regression tests just for the component. That's when the grouping might come in handle.

Another example: In my current project, I have data-driven API tests. To test a component, I have to perform 5000 requests with automated test and it takes up to 30 minutes. It's just for 1 component where we have around 14 for now. Imagine, running a full test suite.

The solution to run all kind of tests for the full regression with each build would be tough for Continous Integration/Continous Development.

The other approach is to run just the smoke tests, but you still have to either group your tests or create specific Runner class (Just like JUnit runner or Cucumber runner) to run just the portion of the tests.

The whole purpose of automated tests is to provide quick feedback if the version developed does not contain bugs due to regression of quality. If we have to wait few hours for the feedback. If we can't have it, we would have to delay each version/build (depends when we run those tests) which even might collide with SLA we agreed on with the customer.

To be even more specific: Let's suppose we have a critical bug in payment systems and per SLA we have to fix the critical bugs like this within 8 hours. The developer fixes the bug, creates a build of the application to deploy to the testing environment and we want to make sure we did not introduce any new bugs. Full regression of automated tests, including unit tests, API tests, and GUI tests might take up to few hours but we have only those few hours to introduce the change to our clients (production environment). Instead of running the whole test suite, we can run a group of tests regarding the payments.

Hope it helps