BlogQA/Test Automation

Parallel Test Execution

By March 21, 2016 March 27th, 2017 6 Comments

Writing automated tests is “must have” practice in everyday testing work if you want to have prompt response on any changes that are made on your software.

Continuous integration process is crucial in everyday software development practice. In order to have quick feedback on your application quality, you need to have rich test suite that will cover “application ground” as much as possible. But what if your automated test suite is so big that it requires couple of hours of execution before you get feedback on your application quality? Even worse, what if it is a release day?

If that is the case, parallel test execution is your answer. This article gives an overview of things to consider when doing parallelization of your tests in any testing framework but also gives practical solution on how to do it using RSpec test framework.

Things to consider?

Having parallel test execution is not something that can be magically turned on and off with a switch. You need to consider how to approach this challenge so you have real benefit from it and not become “slave of test scripts refactoring”. These are the things to consider:

Do you need parallelization at all? – First, analyze duration of your test suite execution. Is it too slow for your team needs? Do you have duplicated checks that are testing same things in different tests and that are increasing overall duration of your test suite? Refactoring is often an answer before taking further steps with tests parallelization

Do you have infrastructure for parallelization? – If your CI environment does not have multi-core processor, you will not see real benefit of parallelization. Tests will still be executed in one process. If this is the case, can you utilize cloud resources for your tests? If answer is yes, you can try solutions like running tests inside AWS EC2 instance where you could choose which instance size provides enough processing power to see real benefit of parallelization. Of course, when considering cloud solutions, that comes with a price of resource usage that must be calculated

Do you have tests capable of being executed in parallel? – This is often a problem that emerges first time we actually try to execute tests in parallel. Some tests from your suite share same resources which makes them unreliable for running in parallel. Whenever you consider parallelization in computer science, there is a potential for “race condition” problem. In programming, this is handled with variety of techniques like locks, semaphores, monitors..etc. When creating tests, analogy is the same. This time, you need to take care about resources that your tests are using. For example, if two tests read same file, do something and then check that something is written in those files, that is a potential problem when running these tests in parallel. Problem that can emerge from this situation is that suddenly some of your tests are failing and you are getting false positives (tests are failing even though application/feature works). As any multithreaded problem, it is hard to reproduce and figure out what creates a problem. Therefore, first make sure that all tests are isolated (have setup and teardown process and use unique resources). This will probably require some test refactoring or even redesigning of test scripts but will eventually save you from lot of headaches that you will certainly have when troubleshooting why some of the tests are failing intermittently. There are some approaches (which I will present later in the article) which give ability to execute tests that use same resources after all but I would recommend to use this only as last solution if refactoring of your tests is too expensive for the time being. Advise: Tests need to be isolated!

How to parallelize tests with RSpec?

Now the you have good idea about the problems that can emerge when parallelizing your test suite, next thing to look is how to do it using RSpec test framework (This article focuses on RSpec but other popular frameworks also provide parallelization solutions). First thing to look at is parallel_tests gem (https://github.com/grosser/parallel_tests). Readme provides a lot of information on how to setup your tests to be executed in parallel (primarily unit tests in Rails), but in this article, I will focus on how to execute parallel functional RSpec tests which does not necessarily need to be connected with Rails framework.

First thing to do is install parallel_tests gem (put it inside your project Gemfile and run bundle install). To run your test suite in parallel, just simply run this command:

parallel_rspec spec

This will run all tests inside your spec folder in parallel and will use all available processor cores on your CI environment. Of course, parallel_rspec command offers couple of useful options:

-n – number of parallel processes that will be used

-p – pattern of tests that will be run in parallel (This enables running only specific tests in parallel while executing other tests sequentially like before)

-s – this enables running certain tests in single process (Good candidates are all tests that use same resources. Of course, with this approach we will not gain parallelization benefit from these tests but it will save us the time needed for refactoring)

–serialize-output – tests output will be serialized and will be written after execution of tests once process is completed. This will give us nice view of tests execution logs in contrast to the approach when we don’t use this option and output is written in parallel too. This makes troubleshooting of test execution much harder

This is more complex command that uses some of the options that parallel_rspec offers:

parallel_rspec -n 4 –single _np –serialize-stdout spec

This means that tests will be executed in 4 processes in parallel, where one process will run only tests which match pattern – _np and output will be serialized and written in friendly way

Good vs. bad design of rspec test from parallelization perspective

These are two examples of test specs that have bad and good design from parallelization perspective. Both examples have two specs (first_spec and seconds_spec)

Bad Example:

Both tests are doing the similar thing (writing data with different content into the same file and then trying to read those data). Since both tests are using same resource (same file), when running RSpec tests regularly (in sequence), no issues will happen. However, if we run these tests in parallel potential problem can happen because both tests are reading data from same file expecting different output. Therefore, problem will happen intermittently and tests will fail but not always and not all of them which makes this kind of problem very hard to debug and reproduce (this example is simplified version and problem is obvious, however in real tests which are more complex, this problem is very hard to debug). Output of execution when one of the tests fail looks like this:

Good Example: On the other hand, we have good example which contains same tests but with two key differences:

1. Both tests have setup and teardown part (before and after blocks)

2. In these blocks, they are creating and deleting file with different name

This will make sure that parallel execution of tests does not make any difference in terms of test results. Therefore, key thing is to have unique resources that your tests are using. In this example, we used very obvious shared resource (file). However, when testing real application, that can be any part of application (to be more precise, any type of create/update/delete action on your application, like create user..etc.)

How did we benefit from parallelization of tests?

Everything that was said up until now is only theory. In real world scenario, we used parallelization to resolve problem with slow execution of regression tests. It is usually practice to run regression tests against our application couple of times in a week. However, as part of regular release process, we need to run regression tests. Because of that, long lasting execution significantly delays feedback that we have on our application quality on release day. If some issue is discovered and requires hot fix, we need to run regression tests once again. Therefore, release process lasts couple of hours more than it should which can lead to breaking application release deadlines.

Effort to enable parallelization was a bit complicated to handle because large part of tests were tightly coupled and they were not designed to be executed in parallel (from the start). That required some refactoring and redesigning of tests and some of them were too expensive to be redesigned for this purpose. Therefore, we focused mostly on tests that are long lasting and tried to parallelize them.

This is the stats we had before parallelization (CI server):

EC2 instance m1.small: 1xCPU, 1.7GB RAM, 160GB storage (old generation instance)

Number of tests: 120

Execution time: ~ 3 hr 45 min

This is the stats we have after parallelization (CI server):

EC2 instance c3.xlarge (on-demand instance): 4xCPU, 7.5GB RAM, 2x40GB SSD storage

Number of tests run in parallel: 77

Number of tests run in one thread sequentially: 43

Execution time: ~ 1.5 hr

We used expensive AWS instance only because we use it in “on-demand” way. This means that instance is running only when tests are executed. Therefore, running this instance couple of hours (cca. 10 hours) per month didn’t cost much. That enabled us to have much powerful multi-core instance which can be fully utilized for parallel test execution. How to run AWS instances in “on-demand” way is explained in another article on our blog.

Lessons learned?

Design tests in a way that they can be parallelized later on if needed (the sooner the better)

Think about hardware costs if running tests in cloud (consider using instances in on-demand fashion)

Identify tests that are long lasting and parallelize them first, then move on to ones that don’t take too much time to be executed

Consider if you need parallelization at all (don’t do it just to be fancy)