In this article I’ll explain what Robolectric framework for Android is, why automated tests that use this framework aren’t unit tests and highlight the issues caused by over-reliance on Robolectric in your tests suite.

Unit Testing

Unit testing is a practice of writing code that verifies the correctness of another part of code. Unit tests fall within a broader category of automated tests and are characterized by the following properties:

• Unit tests exercise the smallest logically cohesive parts of the code in the codebase, known as units.
• Unit tests don’t depend on external resources.
• Unit tests can be executed locally, in isolation.
• Unit tests are fast.

Unfortunately, there is no canonical definition of what constitutes a unit. Therefore, the scope of the code that a unit test can exercise is up for a debate (and, boy, there are debates about that). However, the isolated nature and the speed are generally accepted and uncontroversial characteristics of unit tests.

Android SDK Stubs

To understand what Robolectric is, we shall start with the Android SDK. That’s a software bundle that we install on our computers to develop Android apps.

If you browse the directory where you installed a specific Android SDK, you’ll notice a file named android-stubs-src.jar. Unzip this file, and you’ll find a hierarchy of directories inside it, containing .java files that correspond to various Android components.

For example, that’s the content of android/app/Activity.java from this unzipped jar:

package android.app;

@SuppressWarnings({"unchecked", "deprecation", "all"})
public class Activity extends android.view.ContextThemeWrapper implements android.view.LayoutInflater.Factory2, android.view.Window.Callback, android.view.KeyEvent.Callback, android.view.View.OnCreateContextMenuListener, android.content.ComponentCallbacks2 {

public Activity() { throw new RuntimeException("Stub!"); }

public android.content.Intent getIntent() { throw new RuntimeException("Stub!"); }

public void setIntent(android.content.Intent newIntent) { throw new RuntimeException("Stub!"); }

... more code ...

}

What we see here is a class that has all the methods of Android Activity object, but, for some reason, its methods have weird implementations and just throw RuntimeException. What’s going on here?

To answer this question, we shall recall that:

• To verify that your Android code uses the Android framework correctly (i.e. perform static type checks), the compiler needs to know about Android framework’s classes.
• The compiler is only interested in the high-level public API because that’s the only part that your code uses directly.
• The full Android framework is very complex and heavyweight, and depends on external components that it expects to find on real devices (e.g. SQLite database).

In summary, the compiler needs to “know” the public APIs of Android classes, but these classes contain lots of implementing code and depend on various components not present in your local environment.

A clever hack was employed to optimize the performance, limit the amount of code that the compiler and other tools need to deal with and remove external dependencies: create a representation of the Android SDK that has the same public APIs, but strip all implementing code. This eliminates all the aforementioned issues and enables the tools to do their part. It is this alternative representation that’s packaged into android-stubs-src.jar.

[Technically speaking, the tools use android.jar file from the same directory, but the distinction isn’t important for our discussion here]

Testing with Calls to Android APIs

Even though our source code is compiled against empty stubs of the Android SDK, when you deploy your app to a real device, it’ll use the full Android SDK installed on it. But what happens if you write a test for a method that calls to an Android API and execute this test on your machine?

When you run tests locally, the stubs that we saw earlier will be used to represent the Android SDK. Since all the methods in these stubs throw exceptions, the test will fail the moment it encounters an Android API call.

There are several ways to work around this issue:

• Remove calls to Android APIs from the tested code.
• Replace Android APIs with your custom test-doubles (i.e. “mock them”).
• Use Robolectric framework.

The last option is the topic of this article, so let’s understand what it does.

Robolectric

At a high level, Robolectric framework is a test-double of type fake of the entire Android SDK. It re-implements the Android APIs to simulate Android’s feature set even when executed on your own machine. When you use Robolectric, all the calls to the Android APIs, which would normally reach the stubs, are redirected to the respective Robolectric’s implementations.

Robolectric is a simple way to work around the inability to use the Android APIs in local tests. It is also an amazing tool for testing code that depends on location, filesystem, SQLite, and many other “external” dependencies. For example, I leveraged Robolectric to test my SettingsHelper library, which is a wrapper around SharedPreferences. Furthermore, Robolectric allows you to run your tests against the test-doubles of multiple versions of the Android SDK, so you can even capture “fragmentation” bugs with it.

All in all, Robolectric is a very powerful framework that opens new amazing opportunities for local automated testing. The only caveat is that tests that use Robolectric aren’t unit tests.

Integration Testing with Robolectric

Here is an interesting question: when you test code that uses Android APIs, what do you test fundamentally?

Well, since calls to Android APIs are the points where your application integrates with Android, tests that exercise this code are integration tests, by definition.

Another way to arrive at the same conclusion is to realize that, unlike other third-party libraries and frameworks that your app might use, Android SDK doesn’t become part of the distributable artifacts during a build process. It is an external dependency from the point of view of your application.

Therefore, even if you replace the stub implementation of the Android SDK with Robolectric, fundamentally, these tests will still be integration tests, not unit tests.

Issues with Robolectric Tests

In my experience, there are two main issues with Robolectric tests: speed of execution and reliability of the results.

Let’s start with the smaller of the problems: reliability of the results.

Since Robolectric is a complex test-double of the entire Android SDK, there can be small deviations in its behavior as compared to the original. These situations are rare, but if this happens, your passing tests can give you a false positive indication and a bug will slip into production. Furthermore, after the bug is discovered, a green test covering the feature will make it more challenging to identify the issue because, naturally, you assume that you can trust your tests.

The bigger problem with Robolectric tests is their speed.

It’s only natural for integration tests, which are larger in scope than unit tests, to take more time. Robolectric tests are actually relatively quick in the context of the general integration testing, but they are much slower than unit tests. The exact numbers will vary between setups, of course, but just to give you a perspective, consider the following example.

In this configuration, tutorialTest executes in less than 1 millisecond on my machine:

@FixMethodOrder(value = MethodSorters.NAME_ASCENDING)
class TutorialTest {
    @Test
    fun aWarmupTest() {
        1.shouldBe(1)
    }

    @Test
    fun tutorialTest() {
        1.shouldBe(1)
    }
}

If I run the same tests with Robolectric, tutorialTest takes 6 milliseconds to complete:

@FixMethodOrder(value = MethodSorters.NAME_ASCENDING)
@RunWith(RobolectricTestRunner::class)
class TutorialTest {
    @Test
    fun aWarmupTest() {
        1.shouldBe(1)
    }

    @Test
    fun tutorialTest() {
        1.shouldBe(1)
    }
}

Please note how the execution time skyrocketed, even though the test case doesn’t actually call any Android APIs. If it would, the overhead could be even higher.

In the context of a small tests suite, an average overhead of tens of milliseconds per test case doesn’t sound like a big issue. However, once your project reaches hundreds and then thousands of test cases, milliseconds add up to seconds, then to minutes. At some point, this overhead becomes a considerable drag on developers’ productivity.

I was involved in one project where developers did a great job with test coverage, but most of the tests used Robolectric. Even though the project was of moderate size (less than 40k lines of code), it took more than 4 minutes to run the tests locally and even longer on the CI machine. For comparison, executing a comparable number of unit tests would probably take less than 20 seconds. This overhead caused by Robolectric tests became a major productivity issue for me during that engagement, and I’m sure that it similarly affected the other team members as well.

Conclusion

Robolectric is a powerful integration testing framework for Android apps and libraries that opens new and exciting possibilities for automated testing.

Unfortunately, Robolectric is often seen as a unit testing framework and, consequently, gets overused. This leads to a major increase in the execution time of larger test suites, which, in turn, translates into productivity loss.

Therefore, I recommend avoiding Robolectric as much as possible and use it only when the benefits clearly justify its cost, and there is no simple alternative that you can use instead. In my experience, you can eliminate most of the use cases for this framework by writing decoupled, testable code, and then using simpler test-doubles to isolate the tests from individual Android API calls.

As usual, thanks for reading and please leave your comments and questions below.