Managing test execution time with RSpec

RSpec is equipped with a fail-fast option when running tests, so that when one or more steps fail, the whole group of tests that is started stops. This is handy when you have critical steps that must succeed, and when they fail, running the rest of the steps has no value. This also saves time, because you get the results and test reports earlier. However, RSpec does not support fast failure on the script level. This scenario is useful when you have, e.g., 50 scripts in a regression suite and you want a script to stop when one or more steps in it have failed, and the rest of the scripts to continue executing, which is the main difference when compared with the –fail-fast option. With a few small tweaks to the test scripts, this can be done.

Test scripts

Below is an example of a dummy script with an explanation underneath:

failed_steps_allowed = 1
steps_failed = 0

describe "Example test" do
 around(:example) do |example|
     if steps_failed < failed_steps_allowed
     example.run
      if example.exception
        steps_failed  += 1
      end
    end
 end

 after(:all) do
     #clean up everything that might have been created in the test
 end

 context "Context 1" do
     it "Valid step" do
       puts "This should get printed to output"
     end

   it "Invalid step" do
     expect(3).to eq(2)
   end

   it "Step that should be skipped" do
     puts "This is not supposed to get printed to output"
   end
 end
end

The main thing to examine is the around hook. This hook is executed on each example (a block of code). This hook takes the example as a parameter, executes the code before it, and executes the example itself and some code after it.

We use this to check if steps_failed < failed_steps_allowed . If this is true, we run the example. After that, we check if the example resulted in an error, and if it did, we increase the counter. When steps_failed reaches failed_steps_allowed , every following example will be skipped.

This is the output when we run the test above:

Context 1

This should get printed to output

Valid step

Invalid step (FAILED - 1)

Step that should be skipped (PENDING: around hook at ./spec/smoke/example.rb:6

did not execute the example)

As we see, the text from the skipped step is not printed to the output, since it was skipped, and we have a PENDING indicator in brackets next to the skipped example.

Jenkins integration

If you want to use this in combination with Jenkins and turn fast fail on or off when you run your test suite, you need to pass a parameter from the Jenkins job to your test scripts.

First, add a Boolean parameter to the Jenkins job:

This adds a checkbox to the Run with Parameters screen on the Jenkins job menu.

We then fetch this parameter in the Jenkins Execute shell step and place it in the FAST_FAIL environment variable:

#code before
FAST_FAIL = ${FAST_FAIL}
#code after

In the same way, we also add string parameter STEPS_ALLOWED to configure how many failed steps we allow, directly from Jenkins.

After that, we need to fetch these environment variables in the init_rspec.rb file, so that they are available in every script.

RSpec.configure do |rspec_config|
 #code before
 rspec_config.before(:all) do
   #code before
   @is_fast_fail = false
   @is_fast_fail_string = ENV['FAST_FAIL'] #this gets the FAST_FAIL env variable and
   #passes it on to every test
   @is_fast_fail = true if @is_fast_fail_string == "true"
   @failed_steps_allowed = Integer(ENV['STEPS_ALLOWED'])

   puts "RUNNING TESTS WITH FAST FAIL OPTION ON" if @is_fast_fail
   #code after
 end
 #code after
end

The Jenkins shell parses the FAST_FAIL parameter as a string. We need to check it and, based on the result, assign true or false to our @is_fast_fail Boolean. We also parse STEPS_ALLOWED and assign its value to @failed_steps_allowed as an integer.

The only thing left to do is to use @is_fast_fail and @failed_steps_allowed in our dummy test script from the beginning:

steps_failed = 0

describe "Example test" do

 around(:example) do |example|
     if steps_failed < @failed_steps_allowed
     example.run
      if example.exception && @is_fast_fail
        steps_failed  += 1
      end
    end
 end

 after(:all) do
     #clean up everything that might have been created in the test
 end

 context "Context 1" do
     it "Valid step" do
       puts "This should get printed to output"
     end

   it "Invalid step" do
     expect(3).to eq(2)
   end

   it "Step that should be skipped" do
     puts "This is not supposed to get printed to output"
   end
 end
end

This is now a complete flow that allows us to choose from Jenkins when we want to run our test suite with fast fail, and when not to.

Bash script for easier adoption

Assume that you have a complex test suite with numerous test scripts. It is handy to insert the around hook, and the variables used in it, with help from a Bash script, rather than doing everything manually.

Below is an example of a Bash script that does exactly that. First, we place an around hook in a .txt file, to use it as a template.

around(:example) do |example|
   if template_steps_failed < @failed_steps_allowed
      example.run
   if example.exception && @is_fast_fail
       template_steps_failed += 1
   end
 end
end

After that, in the script, shown below, the template_steps_failed variable is renamed for every file, based on the file name, so that it has a different name in each test script. This is necessary when running tests in parallel, because in Ruby with RSpec, variables are shared between scripts when you run them as a group. This way, every test script has its own counter and it functions properly.

#!/bin/bash
FILES=regression/*

variable_end="_steps_failed"
variable_value='=0\n'

for f in $FILES
do
    filename=$(basename "$f")
    filename="${filename%.*}"
    variable_name=$filename$variable_end

    template_text=""

    #reading out the template file
    while IFS='' read -r line || [[ -n "$line" ]]; do
       template_text=$template_text$line"\n"
    done < template.txt

    #replace template variable name
    text_to_insert="${template_text//template_steps_failed/$variable_name}"

    variable_text=$variable_name$variable_value

    echo "Processing $f file..."

    #insert variable for counting steps
    sed -i "1s/^/${variable_text} /" ${f}
    #insert around hook
     sed -i "/.*describe.*/a\ \n${text_to_insert}" ${f}
done