BlogQA/Test Automation

Parallel mobile testing with Appium – Part 2: iOS

Preface

This article assumes that you’ve already read ‘Parallel mobile test execution with Appium: Part 1’, which talks about motivations, approaches and challenges of establishing parallelism in mobile test execution with Appium and Android. The next step forward was to do the same thing with Appium and iOS. In general, the iOS platform in combination with XCode and the specifics of MacOSX usually lead to more headaches in mobile automation. We assumed this would be the case, but we didn’t expect the challenges we would face. This article will describe the problems we were trying to solve, our approaches as well as the simple solution we eventually found. Stay tuned for more details…

Initial problem

As explained in the previous blog post related to parallel mobile execution with Appium and Android, the general idea is to connect as many devices as possible to a machine (the number of connected devices is restricted by the number of USB ports), deploy the app on all connected devices and run Appium tests that will be executed against the target app on all devices in parallel.

We initially used the same approach for Appium and iOS, but we encountered one major limitation: only one instance of Apple Instruments can be run on MacOSX. Since Appium uses Apple Instruments underneath to run tests against an iOS app, that meant we are unable to spin up multiple Appium server instances on a single machine. We could spawn multiple Appium server instances (just like in the case of Appium/Android), but all of them would connect to the same Instruments instance (i.e. one iOS device per machine)

Our approach (first try)

When we encountered the limitation above, our immediate reaction was ‘What if we can spin up multiple MacOSX VM machines that are all hosted on one physical Mac machine?’ The way we envisioned this was:

  • Launch a MacOSX virtual machine for each iOS device connected to the host.
  • Connect the iOS devices to guest machines instead of the host (i.e. map physical USB ports to virtual USB ports on VMs).
  • Provision test resources and Appium on each guest machine (i.e. start Appium, run tests, stop Appium).

In theory this made sense, but in practice, we found out that it was not a legal approach since Apple only allows users to run one MacOSX VM per one physical MacOSX machine, which did not help us in our scenario.

Our approach (second try)

When searching for various solutions, we found out that the limitation existed on XCode itself. XCode version < 6.3.2 didn’t support running multiple Instruments instances on one machine. By upgrading to a newer version of XCode and using Appium 1.5, which added support for spawning multiple Instruments instances, we could make pretty similar solution, like in the case of an Appium/Android combination, which can be found here:

https://github.com/ATLANTBH/testing-research/tree/master/parallel_mobile_execution/ios.

This overall diagram describes how the flow works:

In short, the mobile devices are connected to a USB port. When the ios_runner_sh script is executed, it invokes the number of Appium server instances that is equal to the number of UDIDs and then executes the test script for each UDID (each real device). Each Appium server instance invokes one Instruments instance to communicate with a real device, which has its own tmp file for intermediate operations.

The full script can be found at https://github.com/ATLANTBH/testing-research/blob/master/parallel_mobile_execution/ios/ios_runner.sh, but here we will just talk about the most important parts of the script and how they differ from the Appium/Android script described in Part 1 of this blog:

  1. Discover all UDIDs using the system_profiler tool (in other words, discover the IDs of the connected iOS devices):

2. Deploy the app using the ios-deploy tool on all the connected devices (in general, the ios-deploy tool is a helper tool for deploying iOS apps using the command line without using Xcode’s cumbersome cli tools):

3. Based on the number of UDID devices, spawn an equal number of Appium server instances with one subtle but important difference: specifying the instrument’s output tmp file (to spawn multiple instrument instances and have each Appium server work with a different instrument instance, you need to specify different tmp files for the instrument’s output. This is done using the –tmp flag):

4. Start TestNG or RSpec tests for each combination of UDID and Appium server instance ports. This works in pretty much the same way as in the Appium/Android combination.

5. Clean up by shutting down the Appium server instances once the tests have been completed. This works in pretty much the same way as in the Appium/Android combination.

Conclusion

In general, mobile parallelization is becoming a real need these days, and more and more cloud mobile providers are offering solutions based on the fact that you can choose a farm of devices and execute any number of tests against them. Since this solution is currently very costly (as explained on the sample AWS Device Farm in part 1 of this blog), it is good to know that an in-house solution can be used for your own infrastructure. After all, why not try out parallelization (and see how tests work in general) at no charge and then decide to shift towards cloud mobile providers if you need more infrastructure. As explained in the first article, the diversity of mobile phones makes everyday testing of mobile apps a big problem so a good solution is very often somewhere in the middle (between using your own infrastructure and purchasing cloud mobile solutions). After careful planning and fragmenting devices that you want your app to be tested against, you can make a smart decision to use your own infrastructure for devices that are available to you and purchase cloud mobile solutions only for devices you are missing. Therefore, the solution we presented in this blog can be a good starting point to make a parallelization effort at no cost and utilize the infrastructure you already have.

  • Alexander Poulikakos

    Thanks for a great article! One question:
    What is the reason for using ios-deploy to install the app, instead of letting appium do it?

    /Alex

  • Bakir Jusufbegovic

    Thanks Alex,

    We used ios-deploy because we were unable to install app via Appium on the real device (on emulator, it works out of the box when you run your tests). We needed cli tool for app installation on device since we planned to run this in CI environment.

    Hope it helps,
    Bakir