20

Nov

2014

Avoiding Robolectric Shock While Testing Android with Gradle and Espresso

By: November 20, 2014

The Android development community’s opinion of the available automated testing capabilities on Android is mediocre at best. Google’s provided framework mixes unit and integration testing capabilities in a single poorly documented package. Unit tests provide a purpose separate from integration tests and should be treated as such. The Android testing framework requires a device or emulator in order to run and this slowness introduced into the feedback cycle tends to remove the agility from following a Test Driven Development (TDD) based methodology. Dealing with things like the Android Lifecycle and the moving parts created by the asynchronous, message based nature of the platform tends to add unnecessary complexity for unit tests that should otherwise be extremely simplistic and fast to execute.

Do Android (Developer)s Dream of Electric Sheep (or better testing tools)?

Since many Android developers come from a Java background where JUnit is the norm, there is high demand for a better approach to testing. Google eventually provided a framework called Espresso which made writing tests for Android more declarative and eliminated the need to write custom polling code or resort to using sleep() in order to wait for the asynchronous pieces of the framework to align so that you can assert the expected behavior. Unfortunately, Espresso still required a running instance of Android on a device or an emulator. JUnit runs on the JVM but Android runs on the Dalvik VM. The Dalvik VM requires Dalvik bytecode which translates to additional steps that must be done before your tests can actually be run in this environment, increasing the time between writing the test and getting feedback from it.

Espresso may not have left Android developers dreaming of electric sheep, but it still left them longing for a more human way to unit test their application code. Enter Robolectric, which essentially stubs out the entire Android platform and provides a test runner that allows you to run your tests in the JVM without being connected to a device or emulator. This dramatically decreases the code/run/feedback cycle that is so critical when following TDD.

Robolectric seemed to be the solution that Android developers were hoping for. Even so, all electrical systems have the potential to cause harm, so it should come as no surprise that you may get a few shocks when you have Robolectric coursing through your Android project. Especially if you don't take the proper precautions.

Edison (Espresso) vs. Westinghouse (Robolectric)

When I started at AWeber, our Android projects were already happily using both Robolectric and Espresso in an environment based on using Maven as our build tool. At this same time, the complexities of building Android applications to support older versions of the platform as well as the proliferation of new form factors including TV, auto, and wearables, started to show the limitations of the traditional Ant/Maven based build system. Google was obviously committed to replacing Ant/Maven with Gradle and we felt like it was time to make the switch.

On the surface, our goal was simple: take the contents of our git repository and produce the same artifacts as before (binaries, test and reports, etc.) using Gradle. The fact that the whole Android project structure had changed in addition to the tooling, made the process of moving everything Android at AWeber from Maven to Gradle a bit more complicated. Our decision to continue using both Robolectric and Espresso in our Android projects kept manifesting as the root cause of many of the issues that we ran into. Just as moving electricity from point A to point B resulted in the War of Currents, it felt as if we had our own battle between testing frameworks and would need to choose one of the two libraries as our standard for testing on Android.

A Bounty Hunter to the Rescue (Competing Test Runners)

A quick Google search made us realize that we weren't the only ones having difficulty using both Robolectric and Espresso with Gradle within the new Android project structure. It seemed that the two test runners just could not get along. This led us to discover Deckard which was the Blade Runner to provide the policing for the simplest possible way of running both Robolectric and Espresso tests together in Gradle. We had to tweak some of the Gradle script in our environment due to conflicting transitive dependencies between the testing and other third party Android libraries that we used, but the sample code in Deckard provided the initial insight into what we had to do in order to get everything working.

Deckard seemed to rescue us from our problems, but after digging deeper into the solution, it started to look like it could become problematic. Even though we were no longer forced to choose between Robolectric or Espresso, the way everything was integrated required too much secret knowledge of the solution in order to guarantee that new problems weren't going to manifest in the future. This solution, required us to run the plugin-generated, gradle ‘test<VariantName>’ task to run unit tests and the standard ‘connectedAndroidTest’ task to run integration tests. It also forced us to exclude mockito-all dependencies in favor of mockito-core because it lacked the hamcrest packages that conflicted with Espresso. Some other immediately apparent issues included:

  • Espresso was not provided in a Gradle-consumable format. We had to download the java archives (JAR files) (which often included transitive dependencies) and consume them as local JAR files. This didn’t really give us the flexibility needed to make Espresso work with our other third party libraries without potentially modifying the downloaded JAR files or allowing Espresso to dictate the versions of these dependencies that we used.
  • The Robolectric (unit) and Espresso (integration) tests were commingled in the project hierarchy. This meant that at first glance, there was no easy way to distinguish them short of a package or file naming convention. This would require us to build this knowledge into our Gradle files so that we could easily run unit tests and integration tests separately so that we could get the quick feedback we were looking for as part of our TDD.
  • Deckard uses the androidTestCompile closure for all tests. This is bothersome not only because this is the default ‘android’ closure used for ‘connected’ tests (requiring you to be ‘connected’ to an Android device or emulator, which Robolectric tests are not), but also because doing this exacerbates the transitive dependency problems since all libraries needed for the Espresso tests are also on the Robolectric tests classpath.

Clearly, Deckard’s approach wasn’t going to be the solution for making order out of our testing chaos, but it did point us in the right direction for further iteration.

Coffee Break (Problems Dealing with Espresso in Gradle)

Jake Wharton provides the solution to the problem of Espresso not being in a Gradle-consumable format by repackaging Espresso as Double Espresso. This solved one of the aforementioned problems,  only to cause another related to commingling unit and integration tests to manifest:

After incorporating Double Espresso, any attempt at running our unit tests (Robolectric) resulted in a compilation failure for all of our integration tests (Espresso). Our integration tests were unable to find Espresso and its dependencies. A quick look at the Robolectric Gradle plugin source code revealed why… Double Espresso is packaged as a group of Gradle-friendly Android Archive files (aar) (rather than the JAR files that we initially had working). The Robolectric Gradle plugin creates a Gradle task for compiling the Robolectric tests (which in our case due to the Deckard-inspired solution were commingled with our Espresso tests). This task (compileTestDebugJava) assumes that many of the additional Android specific build steps (like dexing) are unnecessary because Robolectric tests are supposed to be able to run on a pure JVM without the need for doing anything to make the bytecode Android-friendly. Unfortunately, the dexing task dependency is what unpackages .aar files so that the JAR files that they contain can be used by the ‘compile’ closures. At the time, there was no way to exclude the Espresso tests from compilation even though we knew when running gradle testDebug they should not be compiled because the Robolectric Gradle plugin allows us to exclude them from being run based on our package naming convention. The writers of the Robolectric Gradle Plugin have since provided a mechanism for excluding files from compilation, but at the time that we ran into this problem, I came up with a temporary workaround for this issue.

This new issue pointed back to an old nagging problem… we had all of these Robolectric (unit) tests running around pretending they were Android integration tests. They pulled their dependencies from the androidTestCompile closure, but violated other contracts (like the need for dexing) that were assumptions made by running the gradle connectedAndroidTest task.

What we needed here, was something akin to a Voight-Kampff device that allowed us to easily distinguish unit tests from integration tests so that we could make sure that assumptions made by one group wasn't going to cause problems for the other.

Our not-so-Voight-Kampff solution (Distinguishing Unit Tests from Integration Tests)

By this point, we were pretty much convinced that Robolectric had no business being anywhere near our connected Android tests and decided to split them out into a Gradle submodule. A Gradle submodule is an individual component in a multi project build that can create its own output and define an isolated classpath. We heard about others trying this solution with varying degrees of success but, at the time, no one had really documented exactly how they had gone about doing this.

We decided to put the notion that Robolectric runs on a straight JVM to the test and created a pure Java Gradle submodule (use the ‘java’ Gradle plugin for the submodule instead of the ‘android’ Gradle plugin) for our Robolectric tests. Since this submodule was no longer an Android project, there was no need to continue using the Robolectric Gradle plugin. This plugin was the source of many of our issues because of the way it integrated Robolectric with the Android Gradle DSL. On the other hand, using that plugin took care of some things which we had to do manually now that our submodule created for our Robolectric tests adhered to the pure Java Gradle DSL. The most notable issues included:

  • Our Robolectric submodule had no knowledge of the local Android Maven repositories installed using the Android SDK Manager.
  • Our Robolectric submodule had no knowledge of the Android submodule or its compile time dependencies.
  • Our Robolectric submodule had no knowledge of the Android platform in general. This was necessary since Robolectric creates ‘Shadow’s for many pieces of Android.

We were able to get around each of these issues with some minor additions to the build.gradle file of our Robolectric submodule.

We exposed the Android Maven repositories by adding:

The other issues were resolved by adding additional entries to our dependencies closure (code comments provided to indicate the particular issue from above that the code is meant to solve):

All of a sudden, we could run unit tests with gradle test (the default ‘test’ task for java Gradle projects) and run integration tests with gradle connectedAndroidTest and we no longer had to worry about dependency collisions between Espresso and Robolectric. Even better, we didn't need to rely on writing Gradle tasks that listened for the addition of other Gradle tasks added by third party plugins in order to accomplish this.

We eventually ran into an issue with this approach when we started to componentize our applications and had Robolectric tests in our Android projects that needed to access resources from our Android Library projects, but the ultimate solution was based on a tweak to our Robolectric configuration and not related to our decision to break up our unit testing into a submodule.

Deckard was really a replicant (Spoiler Alert!)

Even though Deckard fooled us into prematurely thinking that we had our testing problems solved, we’re ultimately pretty satisfied with how the approach to testing our Android applications spawned by it has worked out for us here at AWeber. The approach is fairly intuitive and avoids the transitive dependency nightmare that would likely get even worse as the complexity of our applications grow. This approach also lays the groundwork for some things we would like to do as our componentization strategy evolves. Look for a future blog post where we discuss our componentization strategy in detail.