There is nothing more painful for maintainers of Android applications, and more expensive for companies, than bugs that affect end users. You can spend hours or even days investigating the problem, only to discover that it boils down to something as “trivial” as off-by-one error in a single statement in your codebase.

For simple consumer apps, these bugs can lead to frustrated users and one-star reviews on Google Play. For more serious types of applications, the consequences can be much more severe than that, including huge public relation crises and losing strategic customers.

Evidently, not allowing bugs to slip into production should be a top priority for any Android project. In this post, I’ll describe the main approaches and techniques that you can use to achieve this noble goal.

Manual Testing

The most straightforward way to test your Android app is to install it on a device or an emulator, and then interact with it as a user. If you read this article, chances are that you know how to perform this kind of testing.

Manual testing has many benefits. First and foremost, it’s simple conceptually because you don’t need special tools and you don’t need to learn any techniques to play with your app. All you need is time and attention to details. Another benefit is that, despite its simplicity, this kind of testing covers many aspects of app’s operation. Therefore, by just playing with your app, you can detect many different types of issues: UX problems, UI glitches, functional bugs, poor performance, problems with external integrations, etc. Lastly, if you perform manual testing on a wide range of Android devices, you can also detect fragmentation issues (i.e. inconsistent behavior on different devices). This is a hell of a problem in Android world.

The main drawback of manual testing is the fact that it takes a lot of time. Depending on the complexity of the application, a full regression suite of manual tests can take anywhere from several minutes to several weeks to execute. This time overhead grows very quickly. To some degree, you can mitigate this issue by hiring more QA engineers. However, for complex projects, just adding QA staff won’t solve this problem because, even if the budget is unlimited, there is only so much you can parallelize. In addition, the more people you bring, the higher the communication overhead will grow.

All in all, manual testing is the most widespread technique for verifying the functionality of Android applications. For small projects, or projects with very slow pace of change, doing just manual testing can be sufficient. However, many companies waste enormous amounts of their resources when they keep relying on manual testing exclusively, long after the situation had already called for more sophisticated strategy.

Automated Testing

Automated testing is an umbrella that covers many different tools and techniques which, after they’re set up, can be invoked with just a “button click”. In other words, as opposed to manual testing, the marginal cost (i.e. the additional effort) of executing an automated test suite is close to zero (or, at least, exceptionally low compared to manual testing).

Now, that “close to zero marginal cost” property can seem magical. However, in reality, there are no free lunches in software development. Therefore, it’s important to understand the main trade-off behind the automated testing: you need to invest effort upfront (learn techniques, get familiar with tools, set up processes, etc.) to spare effort down the road. In many cases, that upfront investment into automated testing can amount to many man-months or even man-years for even average-sized Android projects.

Therefore, whether to adopt automated testing or not shouldn’t be a hasty decision. Instead, you need to have a clear understanding of the associated costs and also make sure that there is a positive ROI for the project in the longer term.

In the following sections, I’ll discuss the most widespread approaches to automated testing.

Unit Testing

Unit testing is an automated functional verification technique which, generally speaking, amounts to applying predefined inputs to specific parts of your code and then verifying that the code produces the expected outputs. The parts of the code that you exercise in unit tests are called “units”, thus the name.

The main characteristic of unit tests is that units must not have any non-trivial external dependencies. This means that when you write unit tests, you exercise your code exclusively. There is no user interface, database, network, framework’s code, etc. involved. You just verify the functional correctness of the code you wrote, in complete isolation.

Unit testing has many benefits. The most important one, probably, is that these tests are quick. For example, you can have a test suite with thousands of unit tests that executes in several seconds. Just imagine what level of confidence you get when you know that you’re just one click and several seconds away from verifying the functional correctness of your code. Honestly, I can’t explain this feeling in words, but if you feel it once, you wouldn’t like to ever go back.

Another major benefit of unit testing is “high resolution”. Since unit can be as small as a single class, you can write unit tests for even the smallest features in your app. This makes it relatively simple to test the unit with all possible inputs and also allows you to pinpoint the problems if you accidentally break existing functionality in the future.

The main drawback of unit testing is its complexity, and I’m not talking about the tools. The tools are pretty straightforward and can be mastered in a day or two. However, the mental complexity of unit testing is surprisingly high. On the first sight, this argument can seem odd. After all, you can write your first unit test in a matter of minutes, so what’s so complex about it? However, if you look at the bigger picture, you’ll realize that everybody talks about unit testing, but very few developers actually use this technique. Furthermore, there is no shortage of teams that attempted to incorporate unit testing into their development flow, and failed. All these are evidence that there is more to unit testing than meets the eye. I won’t go into details here, but I do want to give you one example to illustrate this point.

You might’ve heard arguments concerning “mocks”. Some developers claim that mocking practice is useless and even harmful, while others use it in their unit tests for years without issues. So, what’s the right answer here? Well, I wrote a very long article in which I reviewed “anti-mock” arguments from several sources. The conclusion of that post is that not only these arguments stand on a shaky ground, but also that in many cases these are different and even contradictory arguments. This shows you that even developers who employ unit testing (and give talks about it), can still lack understanding of the basic terminology they use. And that’s just one, relatively minor aspect of unit testing practice.

So, unit testing is very complex technique. That’s the reason why relatively few individuals and companies are willing to invest the required effort to learn, adopt and use unit testing in practice. On the other hand, if you peek into the source code of the biggest and the most valuable projects out there, you’re almost guaranteed to find thousands of unit tests. In my opinion, that’s an indication of the utility, and even necessity of unit testing practice.

There is one corollary from the above discussion which I want to emphasize here. Since most of the complexity of unit testing is mental and fundamental (i.e. not related to your choice of a language, framework, etc.), if you learn unit testing in one software stack, you’ll know how to unit test in any other stack as well. In other words, you need to learn this technique properly only once, and then you’ll be able to reuse this knowledge with very little effort, even in different environments.

As you might’ve already guessed, I’m a big proponent of unit testing. So much, that I even created a course about unit testing in Android. If you want to understand what the fuss around unit testing is all about and learn this technique, check this course out.

Integration Testing

Integration testing is a close relative of unit testing. However, even though these two practices might seem almost identical on the first sight, there are major conceptual differences between them.

The biggest difference between integration testing and unit testing is their high-level objective. While unit testing aims to test relatively small units in isolation and exhaust their functionality, the goal of integration testing is to verify integrations. I guess this doesn’t explain much, so let’s talk about the meaning of the term “integration”.

Integration is interconnection between different units in a system. These units can be source code constructs (classes, features or even modules developed by different teams), third-party libraries (e.g. image processing library), frameworks (e.g. Android itself) or even remote systems (e.g. backend servers). Integration testing is concerned with verifying the correctness of the resulting entity.

Fundamentally, every time there is an interconnection between different units, the boundary that emerges can become a source of bugs. For example, the infamous Mars Climate Orbiter space vehicle was lost on Mars due to a mismatch in measurement units used by its subsystems. Theoretically, this defect could have been detected on Earth by testing the interface between these units and save $300+ millions mission.

Integration tests are more involved than unit tests, take more time to execute and are harder to maintain. Therefore, integration testing shouldn’t be used to cover scenarios that can be tested with unit tests. For example, testing any algorithm for a boundary condition using integration testing is probably a bad idea.

Conceptually, the domain of integration testing starts exactly where the domain of unit testing ends. In practice, however, there is always a gray area between these techniques. Having gray area is fine in general, but in some cases it leads to very unfortunate confusions. For example, many Android developers believe that you can write unit tests using Robolectric framework. Since tests that use Robolectric are very similar to regular unit tests in form, developers get an impression that these are the same tests. However, when you use Robolectric, you basically test the integration between your code and Android framework. This is clearly a domain of integration testing. Therefore, while theoretically there is nothing that prevents you from replacing all unit tests with Robolectric tests, this approach has severe long-term consequences.

On the flip side, if you understand the scope of integration testing and use it appropriately, it’s much simpler conceptually than unit testing. For example, consider a desktop computer. Obviously, it’s much simpler to test that it works in general as a “computer” (e.g. by connecting peripherals and booting it), rather than verifying that CPU unit performs floating-point operations without mistakes.

The example of a desktop computer can also illustrate another point. Notice that from the point of view of a guy who sells computers, integration testing involves the body of the desktop as a unit and e.g. keyboard as another unit. He might just assume that each of these components has already been unit tested in isolation, and they were proved to work as standalone units. Therefore, once he can prove that he interconnected them properly, he is ready to ship the computer. However, from the point of view of the manufacturer of the keyboard, integration testing means verifying that keyboard works as a component because, for them, keyboard is a complex system consisting of many different units. This example illustrates the fact that the exact domains of unit and integration testing aren’t absolute, but depend on the specifics of your domain of responsibility.

UI Testing

User interface testing, or UI testing in short, is a special form of integration testing. Basically, an integration test becomes a UI test when you feed test inputs and observe test outputs through app’s user interface. In other words, an integration test becomes a UI test when the text fixture interacts with the app like a regular user.

Please note that while the term “user interface” often refers to so-called Graphical User Interfaces, it’s actually wider in scope than just GUIs. For example, for a backend server, user interface can refer to the API exposed over HTTP to the external world. However, for most Android applications, user interface is what users see on the screen. Therefore, for simplicity, in the rest of this post I’ll treat the term UI as a synonym of GUI.

The benefit of UI testing is that it allows you to test the user interface of your application. However, what exactly “test the user interface” means is a nuanced question. For example, you might want to verify that if you enter a specific number into a field and click on a button, then another screen will be shown and that number will be displayed there. This is a simple UI test. However, if you want to verify that the button has a specific shape, color and position relative to other elements, that’s a much harder task. Therefore, before you write UI tests, you should consider what kind of guarantees you’d like this activity to provide and also understand its limitations.

Since UI tests should interact with the app like users, you’ll need to simulate user actions during UI testing (click, drag, enter text, etc.). This is a non-trivial task all by itself. To make your life a bit easier in this context, you can use a wide range of complex tools developed to take care of that task. However, no tool can hide all the additional complexity, so UI tests are usually much more complex in structure and more tools-oriented than other types of tests.

In my opinion, the biggest drawback of UI tests is that they require much more maintenance than either unit or integration tests. That’s not the fault of the technique itself, though, but just the reflection of the fact that user interface is the most volatile part in many GUI applications. Therefore, even something as simple as moving a button from one place on the screen to another can break UI tests, which, in turn, might require much more effort to fix than the original change.

End-to-End Testing

End-to-End testing, or E2E testing in short, is a special form of integration testing that involves the entire system, including, ideally, all of its external integrations. The main goal of E2E testing is to verify that the system (potentially distributed) is operational as a whole.

Note that since E2E testing involves the entire system, it automatically includes the user interface as well. Therefore, in Android applications, E2E tests are also UI tests.

In special cases, some integrations can be omitted during E2E testing. For example, if you’d test nuclear missiles launch system, you wouldn’t like to actually launch the missiles, right? Or, if you’d be working on spaceship’s navigation system, you wouldn’t be able to test zero gravity conditions using the real gravity sensor. So, there are exceptions. However, when you write E2E tests, you shouldn’t exclude parts of the overall solution just to make testing easier, or because you believe that they aren’t relevant to a specific flow.

Given the scope of E2E testing, these tests tend to be big, complex, slow, difficult to maintain and flaky. This means that it’s very challenging and costly to test the system using this approach. Therefore, in most cases, E2E testing is used to ensure proper integration of the overall solution by executing several core flows.

Please note that I described automated E2E testing here. Many resources on the internet will also describe manual E2E testing, but, in my opinion, manual testing is pretty much E2E by nature. Therefore, I don’t see much difference between “manual testing” described at the beginning of this post and “manual E2E testing” concept.

Static Code Analysis

Static code analysis tools are special checkers that can detect issues in the source code. “Static” here means that these tools inspect the bare source code and identify suspicious patterns, without actually exercising it “dynamically”. In other words, you don’t need to compile the code and run it to perform static code analysis.

Unlike “dynamic” testing strategies (either manual or automated), static code analysis doesn’t provide any guarantees with respect to the functional correctness of the code. Even if these checks pass, your code can still be completely broken. Therefore, while these are handy tools that can find many types of issues in your app, they aren’t an alternative to any of the functional testing strategies discussed above.

Android Studio has a built-in static code analysis tool called Lint, which works out-of-the-box and performs many useful checks. Given how simple it is to run Lint in Android Studio, there is really no reason not to use it. At the very least, run Lint checks before releases and make sure you see no major issues in Lint report.

Security Testing

Security of Android applications is a concern which can also be tested. However, security testing requires a special expertise, which absolute majority of Android developers don’t possess. Therefore, in my experience, even when projects did engage in security testing, this activity was outsourced to companies specializing in this area.

I don’t know much about security testing, so I won’t discuss this practice any further. I just wanted to mention it by name for the completeness of our discussion here.

Conclusion

In this post I described the most popular approaches to testing and quality assurance of Android applications. In addition, I tried to provide you with a bit of insight into the benefits and the drawbacks of each of these techniques.

Given so many alternatives, coming up with a high-level testing and QA strategy for your Android project can be very challenging. For example, you might wonder whether you should invest into automated testing, or manual testing will do in your case? Furthermore, if you decide that automation is required, then what type of tests should you concentrate on? These are important questions and answering them incorrectly can lead to major problems down the road.

Unfortunately, I can’t give any general recommendation in this context because, in my opinion, a general testing and QA strategy should take into account the specifics of your situation and the existing constraints. For example, as I already told you, I’m a big proponent of unit testing. However, I also know that many teams failed to adopt this practice and, in some cases, such attempts made their situations worse. Therefore, even though I see unit testing as one of the best practices in software development, I still can’t tell you that you should start writing unit tests in your project. Therefore, you’ll need to choose your high-level testing strategy by yourself, or in collaboration with your teammates. Hopefully, after reading this post, you have better understanding of the available options and their associated trade-offs.

As usual, thank you for reading this article. If you think that I missed an important tool or technique, or you disagree with any of my claims, feel free to leave a comment below.