QA/Test Automation

Managing test execution time with RSpec

By February 16, 2017 No Comments

RSpec is equipped with a option when running tests, so that when one or more steps fail, the whole group of tests that is started stops. This is handy when you have critical steps that must succeed, and when they fail, running the rest of the steps has no value. This also saves time, because you get the results and test reports earlier. However, RSpec does not support fast failure on the script level. This scenario is useful when you have, e.g., 50 scripts in a regression suite and you want a script to stop when one or more steps in it have failed, and the rest of the scripts to continue executing, which is the main difference when compared with the option. With a few small tweaks to the test scripts, this can be done.

Test scripts

Below is an example of a dummy script with an explanation underneath:

The main thing to examine is the hook. This hook is executed on each example (a block of code). This hook takes the as a parameter, executes the code before it, and executes the example itself and some code after it.

We use this to check if . If this is , we run the example. After that, we check if the example resulted in an error, and if it did, we increase the counter. When reaches , every following example will be skipped.

This is the output when we run the test above:

As we see, the text from the skipped step is not printed to the output, since it was skipped, and we have a indicator in brackets next to the skipped example.

Jenkins integration

If you want to use this in combination with Jenkins, and turn fast fail on or off when you run your test suite, you need to pass a parameter from the Jenkins job to your test scripts.

First, add a Boolean parameter to the Jenkins job:

This adds a checkbox to the Run with Parameters screen on the Jenkins job menu.

We then fetch this parameter in the Jenkins step and place it in the environment variable:

In the same way, we also add string parameter to configure how many failed steps we allow, directly from Jenkins.

After that, we need to fetch these environment variables in the file, so that they are available in every script.

The Jenkins shell parses the parameter as a string. We need to check it and, based on the result, assign or to our Boolean. We also parse and assign its value to as an integer.

The only thing left to do is to use and in our dummy test script from the beginning:

This is now a complete flow that allows us to choose from Jenkins when we want to run our test suite with fast fail, and when not to.

Bash script for easier adoption

Assume that you have a complex test suite with numerous test scripts. It is handy to insert the hook, and the variables used in it, with help from a Bash script, rather than doing everything manually.

Below is an example of a Bash script that does exactly that. First, we place an hook in a .txt file, to use it as a template.

After that, in the script, shown below, the variable is renamed for every file, based on the file name, so that it has a different name in each test script. This is necessary when running tests in parallel, because in Ruby with RSpec, variables are shared between scripts when you run them as a group. This way, every test script has its own counter and it functions properly.