Using image comparison in Automated Visual Testing

Overview

When it comes to image elements testing, using the functional testing paradigm, is not enough to verify if the image appears and shows content as expected. Therefore, QA automation testers leave such elements for manual testing only. After some time, the manual image check can become a burden for testers, as this is highly repetitive work. The chance for bug leakage is higher, as testers tend to check such features less often and with less detailed test cases. Hence, it is very important to try to include visual tests into daily automation regressions, as much as possible.

The easiest way to include image validation into testing practices is by using image comparison tools. It is pretty straight-forward to verify static image content because the output should always be the same. The basic idea is to have a default image (mockup) which will be used for comparison with images present in the application, while tests are executed.

However, usage of image comparison tools is not the best solution, as it requires further manual work. So, the best way to avoid any manual testing, and human-induced errors, is to automate image comparison using libraries for image processing. As our testing framework uses Ruby, Rspec, and Watir, we found Chunky PNG gem, as the most appropriate for this very purpose.

Chunky PNG is a Ruby library that provides a wide spectrum of image related functionalities. One of them is image comparison with the ability to save actual state and pinpoint image differences.

Chunky PNG divides the current image (image taken while regression tests are running) and mockup (image reference) into a pixel matrix, which are then compared by color. Such pixel representation can be manipulated, by using read/write access to the image’s pixels and metadata.

Most importantly, the used library has no dependencies on external image libraries. Based on this, it is easier to integrate its functionalities inside the test automation solution.

How to use chunky PNG gem for comparing two images?

Sample scenario

The challenge is to compare two images that are locally saved.

In this example the first_image.png is the mockup one and the second_image.png is the one which will be compared with the mockup.

first_image.png

second_image.png

The following flow is used to automate the static images check:

• Include the ChunkyPNG library inside script
• Create an array of ChunkyPNG: Image that will read images from their location
• Create an image array that would later be used for the creation of a comparison result image object.
• During the comparison process, each pixel from both images will be compared and saved inside the second “differences” array. After the whole process is finished, the markup is added for spotted changes.
• At the end of the process array, it will be converted and saved as an actual image on the desired location

The resulting image has a bounding box around the redpoint, we added to the image. The resulting output tells us that almost 0.03% of the pixels in the image changed.

differences.png

require 'chunky_png'

 images = [
   ChunkyPNG::Image.from_file("././lib/data/first_image.png"),
   ChunkyPNG::Image.from_file("././lib/data/second_image.png")
 ]

 differences = []

 images.first.height.times do |y|
   images.first.row(y).each_with_index do |pixel, x|
     diff << [x,y] unless pixel == images.last[x,y]
   end
 end

 puts "pixels (total):     #{images.first.pixels.length}"
 puts "pixels changed:     #{differences.length}"
 puts "pixels changed (%): #{(differences.length.to_f / images.first.pixels.length) * 100}%"

 x, y = differences.map{ |xy| xy[0] }, differences.map{ |xy| xy[1] }

 images.last.rect(x.min, y.min, x.max, y.max, ChunkyPNG::Color.rgb(0, 169, 45))
 images.last.save('differences.png')

Output:

pixels (total):     1076700
pixels changed:     285
pixels changed (%): 0.026469768737809974%

Web application scenario

When such automated tests are integrated into the web application regression test, a similar flow as the one above is used. The actual difference, is that the second image needs to be mapped as a page element. In our example, the canvas element is used for locating map images.

The canvas element should be saved as an image using ..toDataURL() method.

The output will be the base64 encoded string, so it should be decoded using the Base64.decode64() method. To be able to use this, you have to have Base64 gem installed.

After that, the decoded string should be read as a PNG data stream and saved as a PNG file.

image_data = @homepage.get_main.canvas_element

script = <<-JS
            return arguments[0].toDataURL('/png',1.0)
          JS

image_data1 = @browser.execute_script(script,image_data)

image_data = Base64.decode64(image_data1['data:image/png;base64,'.length .. -1])

data_stream = ChunkyPNG::Datastream.from_blob(image_data)
data_stream.save("././lib/data/app_image.png")

After the image is saved locally, the approach is the same as for a sample scenario.

Conclusions

Visual testing proved to be of great help in our everyday automated testing. Therefore, we are still investigating the ways to avoid manual image comparison, wherever it is possible. In our next blog, we will deliver some ideas on how to overcome issues with more complex UI elements.