BlogQA/Test Automation

Parallel mobile testing with Appium – Part 1: Android

By March 13, 2017 No Comments

Why mobile parallelism?

There are many factors to take into account when developing and testing apps for mobile devices. One of the most important things to consider is fragmentation. Fragmentation occurs when there are multiple OS versions currently on the market or multiple device OEMs with independent versions of the same OS, as is the case for Android. In these instances, developers and QA engineers need to make sure that the developed apps support the multitude of devices and OS versions available—which can be a struggle.

Automated tests could be executed on multiple devices but in agile development testing, execution time matters and testing on multiple devices sequentially would take countless hours. The solution is to execute tests on multiple devices at the same time. It’s not necessary to test every single combination of device and OS on the market, simply a representative sample of the most popular ones. Android is aware of the fragmentation problem with their OS and is providing developers with percentages of active OS versions, which can help narrow down the sample. Additionally, the sample should include devices from different OEMs and screen sizes, as every OEM customizes the Android OS.

AWS Device Farm

If there is no budget for dedicated Android devices, using a hosted device farm is a logical solution. Included in the wide range of AWS services is the AWS Device Farm, which supports Android, iOS, FireOS and multiple testing frameworks. It also supports manual testing on hosted devices in real-time through a web interface. During these manual sessions, logs with actions and a video of the sessions are recorded.

There’s a free trial period that includes 250 device minutes, after which users are charged $0.17 per device minute. There’s also an unmetered plan for $250 per month, per device. The per-minute plan offers great flexibility, as users only pay for the minutes they use and adding additional devices to the starting device pool is simple and easy to do.

The process of running tests is user-friendly and well documented. New test runs are created through a wizard in the device farm console. Tests must be packed in a zip file and the app must be built to an APK file. Once the files are uploaded through the wizard, a device pool is chosen and the tests are executed in parallel.

For example, a regression suite lasting 30 minutes and running once a day on five devices would total $25.50 per day, or around $500 per month. This estimation is a best-case scenario, since in real-life development tests are typically run more often and can get pricey. Other Amazon web services get cheaper with time but the device farm is specific, as new devices need to be added all the time. This could lead to the service not getting cheaper any time soon.

What if you already have the devices to run tests on?

If you already have Android devices on hand for testing, there’s probably no need to pay for additional devices on a hosted device farm. Of course, it’s unlikely that one would have as many devices as there are on a hosted device farm but there may be just enough for testing on multiple Android versions with multiple screen resolutions.

If you’re automating both Android and iOS, the test framework of choice is Appium, in combination with any language you’re familiar with. Running parallel tests with Appium simply means running multiple Appium server instances with different server flags. The idea is to run Appium instances with a unique port, bootstrap port and device ID attached to it. Then, tests are run in parallel against launched Appium instances, where every instance communicates with one of the connected devices.

The shell script

The process of launching these instances and running the tests can be automated with a shell script. Each shell script does the following:

– fetches the IDs of connected Android devices
– launches Appium instances for those devices
– runs a set of tests on them
– cleans up the launched processes

It is then possible to launch a test suite against multiple devices with a single script call.

Fetching the device IDs

The device IDs can be fetched using the Android Debug Bridge (ADB). ADB is a command line tool that can be used to communicate with an Android emulator or a connected Android device. ADB can also get the IDs of connected devices by running:

The script stores these in an array and uses them to launch the Appium instances.

If ADB daemon is not running, it is started with the first execution of ‘adb devices.’ There is a wait period after the first launch and then after the daemon is running, a list with connected devices and emulators is retrieved. Then the IDs of the devices which are authorized and are not an emulator are placed into an array for further use.

Running the Appium instances

The number of Appium instances to be launched is equal to the number of device IDs fetched from ADB. For each device ID, a shell function is called which launches an Appium instance.

The function is called while looping through the collection of fetched IDs. Also, while looping through the IDs, Android version information is fetched with ADB and stored in a new collection together with the Appium port and the Android ID. This collection is to be used for running the tests:

Running the tests

Similar to the Appium instances, a shell function is called for every connected device that is going to run the tests.

The current implementation supports running Appium tests implemented with Ruby and RSpec or Java and TestNG, but it can be easily extended for running Appium tests implemented with other languages. Every test run is associated with an Appium port, a device ID and the device platform version – the data that was fetched while starting the Appium instances. The data needed for the test runs is passed to the tests with environment variables. This means that the tests need to be configured to expect these arguments in order for the script to start them. Every concurrent test run has its process ID, which are stored in PID_DATA and that collection is to be used for the cleanup procedures.

The function is then called for every item in the Appium data collection (for every connected device).

Cleanup

Before cleaning up the Appium instances, all test runs need to finish. For that reason, the process IDs for the test runs were collected. The following wait ensures that all test runs finish before proceeding with the script execution.

After that, a cleanup method is called. The method uses ‘pkill’ to terminate all of the Appium instances, which are Node.js processes.

Next steps

This is the first part of this topic, covering parallel test execution with Android devices. The second part will be cover the challenges faced while setting up the infrastructure for running iOS tests in parallel. The complete script, regarding this part of the topic, is hosted on our public GitHub repository (https://github.com/ATLANTBH/testing-research/tree/master/parallel_mobile_execution/android). There is also a README file with steps on how to run the script and how to structure the test code written in Java or Ruby to comply with the parameters of the script. Example tests for both Java and Ruby are also provided. A great way to contribute would be to fork the repo and add support for other languages/test frameworks, or to try running the script with different Android devices and suggest improvements or report issues. Be sure to check out part two of Parallel mobile testing with Appium!