How to pass a failed testsuite run (junit.xml parsing)?

Question

We are working in a automated continues integration/deployment pipeline. we got hundreds of testcases and 4 stages (Red, Orange, Yellow, Green).

The issue I'm facing is that a test can fail (bug, timings, stuck process etc.) and it will fail the entire regression run.

I think that we need some sort of weight to determine amount of pass/fail tests to be considered as 'fail' build.

any ideas? something you created on your pipeline?

Thanks, -M

Answer 1

Having a failed build does not always reflect the quality of the product, mainly if the fails are related to testing infrastructure issues.

Reduce the risk of unwanted fails which are not related to application bugs(timings, stuck process), by building a strong and stable framework, that can be easily maintained and extended.

When speaking of fails related to application bugs, the type of tests that failed are more important than the amount of fails. This is defect severity. You can have 3 trivial fails that don't have big impact, and you can also have only 1 fail that is critical. You need to flag your tests accordingly.

Additionally to this, there is a Jenkins plugin that creates an easy to follow test run history, where you can see the number of tests that failed the most times in the last runs.