When it comes to test automation or test coverage in general only a few technical organizations can boast by saying that they develop their software TDD (Test Driven Development) style and have the test process in place at the start of the project. Well, we (here, I mean Android team for C More) are not one of them either.
Approximately one year ago when we wrote this blog post about the app launch we had approximately 0.0001 tests written for our wonderful brand new Android project. Shame on us. As it often happens, one has to compromise on something to be able to secure the delivery before the deadline. No surprise really that it happened to be automated tests.
Fast forward to June 2017: we have delivered several important user features and the app is up and running for a good half year. Everyone is happy. Well, not everyone: the more features we offer in the app, the more time we have to spend testing them. And when it is manual testing only, the time spent testing grows exponentially. Development team together with the testers is forced to test the same things over and over again week in, week out to ensure that small changes somewhere won’t break the core functionality. It does not contribute much to having fun at work, that is coding for developers and discovering really quirky bugs for the testers.
Being inspired by Soundcloud’s example of test ownership by the the development team: Android team at Soundcloud does not have a dedicated manual tester and the testing of the app is done by the development team itself by the means of automated tests to the greater extent. We decided to break our development-test pattern and start with test automation too.
After considering several alternatives available on the market, we have ended up with the one being free and the closest to the core Android platform development: Espresso
Google has first released Espresso test framework in October 2013 and since Espresso 2.0 it became a part of the Android official support library. Espresso is primarily targeted at the developers that are familiar with the main project codebase. At the same time it may be used as black-box testing tool taking advantage of Espresso Test Recorder available in the Android Studio (usual IDE for Android development) allowing a broader audience of users to be in engaged in the process as e.g. testers.
Espresso Test Recorder can be a helping hand to the developers who just start their journey in writing automated tests: the tool will translate the UI interactions into test steps in the form of code. I would recommend it in learning purposes – to get acquainted with the Espresso syntaxis and in-built functionality. The tests that are recorded with Espresso Test Recorder require some optimizations like removing excessive sleep() commands that make test run way longer and revising the test context as the Recorder uses the UI without understaing the app functionality.
Code syntax in Espresso is based on the classical test approach ”Given – When – Then” that is presented as
where ViewMatcher aka ”Given” – provides a way to find a view instance on view hierarchy that matches a given criteria e.g. view with certain id, view with a certain content description, view with a certain text;
ViewAction aka ”When” – allows to check if some view object state matches with a given criteria (this can make the test fail) e.g. view content matches certain query;
ViewAssertion aka ”Then” – provides the instructions to perform an Action on view object e.g perform a click on a view or any other interaction.
Espresso maybe used both for UI testing as well as data testing (that is, the data that is backing the view you’d like to match). The latter is verified with the following construction:
where DataOptions are method calls like onAdapterView, atPosition, onChildView etc
Not as many others, we had a luxury of having dedicated time to get the test automation in place. We ended up spending 10 wonderful sunny days this summer by only writing automation tests in a two-man (two-women, actually) team. As the result of this activity we got approximately 90 tests that would take around 22 minutes to run covering the main use cases such as free text search, browsing different type of content, authentication with accounts having different subscriptions, content of the webviews, saving content to the list of favorites adding reminders for live sport event and many others. In turn, this allowed us to cut the test verification process for each app release from 3 days to a couple of hours and eliminated the necessity of the dedicated test personnel participation in the release process. The tests have revealed several bugs that we have missed out after months of manual testing.
One may choose to run the tests on a emulator, a physical Android device or use cloud testing platforms like Firebase Test Lab. We tried all three of them! We have been using Firebase service for quite a while now for performance tracking and crash reporting, so it was exciting to try their Test Lab as well. Here is a short video how a part of the test run performed on Firebase Test Lab with Nexus 6 physical phone looks like:
Video 1: Espresso tests run covering different states of the movie content detail page and interaction for introduction slides that are shown on the very first start of the application run on Firebase Test Lab
Currently, Android development team at C More has two weeks delivery cycles with customer release every other Thursday. On Monday of the release week the team lead creates the release-candidate-app that is submitted to the test automation tool for verification. After all the automation tests are passed, the release-candidate-app is tested by the development team according to the new features/bug-fixes that are introduces in the this app version and do some exploratory testing. If the development team discovers any bugs or inconsistencies at this stage or at the stage of automation tests, the fixes are performed and verified immediately. Only when the development team is satisfied with the result of the internal testing, the release-candidate-app is delivered to the manual tester. This approach helped us to make our deliveries more secure, robust and be more human-resource independent in our releases, while liberating lots of time to be spent on more exciting things.
Four months after our initial test automation initiative, we have 110+ automation tests that take approximately 28 minutes to be executed and writing tests became our DoD (Definition of Done). It transformed from being painful experience in the beginning – ”How cool it would be to spend days and weeks writing the tests for the project that is already out live” said no developer never – to something we now enjoy doing and being a little bit more proud of our work we do and code we write.
Test automation is the future of the development, and the future is already here.
Anna Babaryka – C More Android Tech Lead