RSpec is equipped with a fail-fast option when running tests, so that when one or more steps fail, the whole group of tests that is started stops. This is handy when you have critical steps that must succeed, and when they fail, running the rest of the steps has no value. This also saves time, because you get the results and test reports earlier. However, RSpec does not support fast failure on the script level. This scenario is useful when you have, e.g., 50 scripts in a regression suite and you want a script to stop when one or more steps in it have failed, and the rest of the scripts to continue executing, which is the main difference when compared with the –fail-fast option. With a few small tweaks to the test scripts, this can be done.
Test scripts
Below is an example of a dummy script with an explanation underneath:
failed_steps_allowed = 1 steps_failed = 0 describe "Example test" do around(:example) do |example| if steps_failed < failed_steps_allowed example.run if example.exception steps_failed += 1 end end end after(:all) do #clean up everything that might have been created in the test end context "Context 1" do it "Valid step" do puts "This should get printed to output" end it "Invalid step" do expect(3).to eq(2) end it "Step that should be skipped" do puts "This is not supposed to get printed to output" end end end
The main thing to examine is the around hook. This hook is executed on each example (a block of code). This hook takes the example as a parameter, executes the code before it, and executes the example itself and some code after it.
We use this to check if steps_failed < failed_steps_allowed . If this is true, we run the example. After that, we check if the example resulted in an error, and if it did, we increase the counter. When steps_failed reaches failed_steps_allowed , every following example will be skipped.
This is the output when we run the test above:
Context 1 This should get printed to output Valid step Invalid step (FAILED - 1) Step that should be skipped (PENDING: around hook at ./spec/smoke/example.rb:6 did not execute the example)
As we see, the text from the skipped step is not printed to the output, since it was skipped, and we have a PENDING indicator in brackets next to the skipped example.
Jenkins integration
If you want to use this in combination with Jenkins and turn fast fail on or off when you run your test suite, you need to pass a parameter from the Jenkins job to your test scripts.
First, add a Boolean parameter to the Jenkins job:
This adds a checkbox to the Run with Parameters screen on the Jenkins job menu.
We then fetch this parameter in the Jenkins Execute shell step and place it in the FAST_FAIL environment variable:
#code before FAST_FAIL = ${FAST_FAIL} #code after
In the same way, we also add string parameter STEPS_ALLOWED to configure how many failed steps we allow, directly from Jenkins.
After that, we need to fetch these environment variables in the init_rspec.rb file, so that they are available in every script.
RSpec.configure do |rspec_config| #code before rspec_config.before(:all) do #code before @is_fast_fail = false @is_fast_fail_string = ENV['FAST_FAIL'] #this gets the FAST_FAIL env variable and #passes it on to every test @is_fast_fail = true if @is_fast_fail_string == "true" @failed_steps_allowed = Integer(ENV['STEPS_ALLOWED']) puts "RUNNING TESTS WITH FAST FAIL OPTION ON" if @is_fast_fail #code after end #code after end
The Jenkins shell parses the FAST_FAIL parameter as a string. We need to check it and, based on the result, assign true or false to our @is_fast_fail Boolean. We also parse STEPS_ALLOWED and assign its value to @failed_steps_allowed as an integer.
The only thing left to do is to use @is_fast_fail and @failed_steps_allowed in our dummy test script from the beginning:
steps_failed = 0 describe "Example test" do around(:example) do |example| if steps_failed < @failed_steps_allowed example.run if example.exception && @is_fast_fail steps_failed += 1 end end end after(:all) do #clean up everything that might have been created in the test end context "Context 1" do it "Valid step" do puts "This should get printed to output" end it "Invalid step" do expect(3).to eq(2) end it "Step that should be skipped" do puts "This is not supposed to get printed to output" end end end
This is now a complete flow that allows us to choose from Jenkins when we want to run our test suite with fast fail, and when not to.
Bash script for easier adoption
Assume that you have a complex test suite with numerous test scripts. It is handy to insert the around hook, and the variables used in it, with help from a Bash script, rather than doing everything manually.
Below is an example of a Bash script that does exactly that. First, we place an around hook in a .txt file, to use it as a template.
around(:example) do |example| if template_steps_failed < @failed_steps_allowed example.run if example.exception && @is_fast_fail template_steps_failed += 1 end end end
After that, in the script, shown below, the template_steps_failed variable is renamed for every file, based on the file name, so that it has a different name in each test script. This is necessary when running tests in parallel, because in Ruby with RSpec, variables are shared between scripts when you run them as a group. This way, every test script has its own counter and it functions properly.
#!/bin/bash FILES=regression/* variable_end="_steps_failed" variable_value='=0\n' for f in $FILES do filename=$(basename "$f") filename="${filename%.*}" variable_name=$filename$variable_end template_text="" #reading out the template file while IFS='' read -r line || [[ -n "$line" ]]; do template_text=$template_text$line"\n" done < template.txt #replace template variable name text_to_insert="${template_text//template_steps_failed/$variable_name}" variable_text=$variable_name$variable_value echo "Processing $f file..." #insert variable for counting steps sed -i "1s/^/${variable_text} /" ${f} #insert around hook sed -i "/.*describe.*/a\ \n${text_to_insert}" ${f} done