How to run unit test + Espresso repeatedly?

Question

I need to collect some data on my current app in order to analyse performance speed by checking the average ellapsed time during Activity start up. I would like to run a test battery where the activity is started 10, 100, 1000 and 5000 times. For each test, it should remain open for at least 10 seconds (time needed to collect all data that happens asynchronously). What I want is exactly this behaviour without having to write these many methods:

@RunWith(AndroidJUnit4.class)
@LargeTest
public class TestStreamLoadingPerformance {

    private static final long TIME_OUT = 2;
    private static final long WAITING_TIME = 10000;

    @Rule
    public ActivityTestRule mActivityRule = new ActivityTestRule(HomepageActivity.class);

    private ElapsedTimeIdlingResource mIdleRes = new ElapsedTimeIdlingResource(WAITING_TIME);


    @Before
    public void setUp() {
        IdlingPolicies.setMasterPolicyTimeout(TIME_OUT, TimeUnit.HOURS);
        IdlingPolicies.setIdlingResourceTimeout(TIME_OUT, TimeUnit.HOURS);
        Espresso.registerIdlingResources(mIdleRes);
    }

    @After
    public void tearDown() {
        Espresso.unregisterIdlingResources(mIdleRes);
    }

    @Test
    public void test01() {
    }

    @Test
    public void test02() {
    }

    @Test
    public void test03() {
    }

    @Test
    public void test04() {
    }

    @Test
    public void test05() {
    }

    @Test
    public void test06() {
    }

    @Test
    public void test07() {
    }

    @Test
    public void test08() {
    }

    @Test
    public void test09() {
    }
}

Answer 1

With the help of @Be_negative comments, this blog post and this answer, I was able to solve the problem with the code below:

@RunWith(AndroidJUnit4.class)
@LargeTest
public class TestStreamLoadingPerformance {

    @Rule
    public ActivityTestRule mActivityRule = new ActivityTestRule(HomepageActivity.class, false, false);
    @Rule
    public RepeatRule mRepeatRule = new RepeatRule();


    @After
    public void tearDown() {
        closeActivity();
    }

    private void closeActivity() {
        final int N = 10;
        try {
            for (int i = 0; i < N; i++) {
                Espresso.pressBack();
            }
        } catch (NoActivityResumedException e) {
            Log.e(TestStreamLoadingPerformance.class.getSimpleName(), "Unable to close activities", e);
        }
    }

    @Test
    @RepeatRule.Repeat(times = 10)
    public void collectData() {
        mActivityRule.launchActivity(null);
    }
}

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

public class RepeatRule implements TestRule {
    @Retention(RetentionPolicy.RUNTIME)
    @Target({
            java.lang.annotation.ElementType.METHOD
    })
    public @interface Repeat {
        public abstract int times();
    }

    private static class RepeatStatement extends Statement {

        private final int times;
        private final Statement statement;

        private RepeatStatement(int times, Statement statement) {
            this.times = times;
            this.statement = statement;
        }

        @Override
        public void evaluate() throws Throwable {
            for (int i = 0; i < times; i++) {
                statement.evaluate();
            }
        }
    }

    @Override
    public Statement apply(Statement statement, Description description) {
        Statement result = statement;
        Repeat repeat = description.getAnnotation(Repeat.class);
        if (repeat != null) {
            int times = repeat.times();
            result = new RepeatStatement(times, statement);
        }
        return result;
    }
}

Answer 2

The easiest (as in least amount of new code required) way to do this is to run the test as a parametrized test (annotate with an @RunWith(Parameterized.class) and add a method to provide 10 empty parameters). That way the framework will run the test 10 times.

This test would need to be the only test in the class, or better put all test methods should need to be run 10 times in the class.

Here is an example:

@RunWith(Parameterized.class)
public class RunTenTimes {
    @Parameterized.Parameters
    public static List<Object[]> data() {
        return Arrays.asList(new Object[10][0]);
    }

    public RunTenTimes() {
    }

    @Test
    public void runsTenTimes() {
        System.out.println("run");
    }
}

With the above, it is possible to even do it with a parameter-less constructor, but I'm not sure if the framework authors intended that, or if that will break in the future.

If you are implementing your own runner, then you could have the runner run the test 10 times. If you are using a third party runner, then with 4.7, you can use the new @Rule annotation and implement the MethodRule interface so that it takes the statement and executes it 10 times in a for loop. The current disadvantage of this approach is that @Before and @After get run only once. This will likely change in the next version of JUnit (the @Before will run after the @Rule), but regardless you will be acting on the same instance of the object (something that isn't true of the Parameterized runner). This assumes that whatever runner you are running the class with correctly recognizes the @Rule annotations. That is only the case if it is delegating to the JUnit runners.

If you are running with a custom runner that does not recognize the @Rule annotation, then you are really stuck with having to write your own runner that delegates appropriately to that Runner and runs it 10 times.

Note that there are other ways to potentially solve this (such as the Theories runner) but they all require a runner. Unfortunately JUnit does not currently support layers of runners. That is a runner that chains other runners.

Answer 3

I had a very similar issue and as a result I've created a library to run Android UI tests multiple times. Might be useful in your case: https://github.com/stepstone-tech/AndroidTestXRunner