How can I run a specific set of Android instrumentation tests with Gradle?

Question

I have multiple test packages:

com.mypackage.blackbox    - Robotium UI tests
com.mypackage.integration - REST integration tests
com.mypackage.unit        - low level unit tests

Our server team needs to be able to run just the integration tests on every push (they take a couple of minutes), but then run all tests every night (the black box UI tests take more than 10 minutes).

This great answer provides a slightly hacky (but effective) way to do it by overloading an existing JUnit annotation like @SmallTest or @LargeTest .

The Gradle documentation suggests that test filters are the way to do this, eg

./gradlew connectedAndroidTestDevDebug --tests com.mypackage.integration.*

However, that fails with an > Unknown command-line option '--tests'. error (presumably because the Android Gradle plugin doesn't support everything that vanilla Gradle does?).

The same documentation says in future they plan to support these alternatives:

• Filtering based on custom annotations (future)
• Filtering based on test hierarchy; executing all tests that extend ceratain base class (future)
• Filtering based on some custom runtime rule, eg particular value of a system property or some static state (future)

Does anybody know a clean way to get this to work right now? For now I'm planning to use the @MediumTest annotation on the base class that all my integration tests extend, but I'd love to be able to specify particular package(s) instead. Using @MediumTest or @LargeTest abuses those annotations, as both my integration and black box tests are large tests according to the guidelines .

Answer 1

This is now possible with the addition of Android's recent Testing Support Library , you can now use AndroidJUnitRunner and filter the tests you run by your own custom annotations.

Filter test run to tests with a custom annotation (com.myapp.MyAnnotation in this example):

adb shell am \
  instrument -w -e annotation com.myapp.MyAnnotation \
  com.myapp/android.support.test.runner.AndroidJUnitRunner

Complete AndroidJUnitRunner Documentation

You'll need to annotate your test cases with your custom annotation to get this to work. Example test case:

import android.support.test.runner.AndroidJUnit4;
import com.myapp.MyAnnotation;

@RunWith(AndroidJUnit4.class)
public class CalculatorTest {

    @MyAnnotation
    @Test
    public void testAddition() {
        //Do testing here
    }

}

Here is what your "MyAnnotation" would look like:

package com.myapp;

import java.lang.annotation.ElementType;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

/**
 * My custom Annotation to specify a type of tests to run.
 */
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.METHOD, ElementType.TYPE})
public @interface MyAnnotation {
}

Answer 2

To run specific tests using gradle you have to create custom instrumentation test runner class and run tests using that class. Ie create class

package com.my.package;

public class MyInstrumentationTestRunner extends InstrumentationTestRunner {
    @Override
    public void onCreate(Bundle instrumentationArguments) {
        instrumentationArguments.putString("size", "medium"); // Run medium tests
        // Add more ...
        super.onCreate(instrumentationArguments);
    }
}

Then in your build.gradle set testInstrumentationRunner (it's under android -> defaultConfig , ie

// ...

android {
    // ...

    defaultConfig {
        // ...
        testInstrumentationRunner "com.my.package.MyInstrumentationTestRunner"
    }
}

// ...

Hope it helps!

Note. build.gradle is from :lib , the tests are located under src/androidTest/java where the MyInstrumentationTestRunner is created.

Answer 3

Sam's answer is the most versatile answer. However, the simplest solution is probably to use the -e package option on the InstrumentationTestRunner :

Running all tests in a java package: adb shell am instrument -w -e package com.android.foo.subpkg com.android.foo/android.test.InstrumentationTestRunner

You can combine this option with using Square's Spoon library, as it allows you to specify either individual classes, or use -e to pass options through to the test runner (eg the package option):

--class-name        Test class name to run (fully-qualified)

--method-name       Test method name to run (must also use --class-name)

--e                 Arguments to pass to the Instrumentation Runner. This can be used
                    multiple times for multiple entries. Usage: --e <NAME>=<VALUE>.
                    The supported arguments varies depending on which test runner 
                    you are using, e.g. see the API docs for AndroidJUnitRunner.

For the record, Shazam's Fork has a more powerful regex option:

android.test.classes=REGEX - comma separated regexes that specify a pattern for the classes/packages to run