Testing Android app release build variant

Question

I have written many unit and instrumented tests for my Android app. So far, I only run these against the debug build variant. Is it necessary to run tests against the release build variant? What difference can there be that might give different results from testing? The main one that I can think of is when ProGuard is enabled, which I haven't done. What will ProGuard do that makes it necessary to run my test suite? What other issues should I be aware of that require testing the release build variant?

Answer 1

Is it necessary to run tests against the release build variant?

I think you should.

What difference can there be that might give different results from testing?

A couple of examples:

• You might have code that uses fields of the BuildConfig class to enable/disable certain workflows. Some libraries might also use that, especially BuildConfig.BUILD_TYPE . It's common to do things like:

   if (BuildConfig.BUILD_TYPE.equals("debug") { ACRA.init(...); Stetho.init(...); ... } 

  but have code fail in release builds due to trying to use components/libraries that were not initialized correctly.

• As you mentioned, ProGuard might throw away some of your classes unless it's properly configured (eg imagine you forgot to add rules for some 3rd party library). Running your tests against the release variant ensures that the ProGuard configuration is correct.

What will ProGuard do that makes it necessary to run my test suite?

ProGuard might remove classes/methods/fields that are, for example, loaded via reflection unless you add the @Keep annotation to them. It might also rename classes used by libraries like Realm, Retrofit, Gson or Volley resulting in all unit and integration tests passing on debug builds (where ProGuard isn't enabled) but failing on release builds. You definitely want to test these before shipping our a new APK.

What other issues should I be aware of that require testing the release build variant?

The release build might also apply PNG crunching, specify different parameters via the buildConfigField method in Gradle, apply splits by ABI or density or enable/disable multidex and others. All of these can have implications in the way that your app works, so why not be on the safe side and test them too.

Another frequent problem that you can catch using these is ensure that you've not accidentally put code in the wrong location (eg /src/debug/java/ ) that happens to be loaded in debug builds but not in other variants.