Android CTS - Random test fails

Question

When I run android CTS full test using below command

run cts --plan CTS

Every time it shows different result for some of the packages, I mean some packages some tests passes/fails randomly every time I re-run full test. But when I run package individually (The package in which some tests failed), all the tests passes in it.

Why I am seeing this behavior?

Environment: OS: Android L CTS version: 5.1_r7

Answer 1

有时某个测试会随机失败，这是因为一段时间未满足测试条件，有时会因为超时测试而失败。

Answer 2

Some cts tests involve specific timeouts set for some event to occur. For example if you are running cts test related to data calls like turning mobile data off/on and timeout to get mobile data connected is set to 10 seconds, then some time this test will pass and sometime it will fail. In this case, increasing that timeout will resolve this issue.

Regarding issue of test case failing when running multiple packages, there could be possibility that test case before failed one has not set device in a neutral/original state for next test. It is a good practice to revert all changes made during a test while exiting a test case.