BlogQA/Test Automation

How to use color algorithm in Automated Visual Testing?

By October 20, 2017 No Comments

More complex UI issues

After static image processing proved to be useful in our everyday automated regression testing (read about it here), we decided to cover more complex UI issues using visual testing concepts. Many applications in our company contain dynamic (interactive) UI elements, which have to be checked manually each time new features are added.

The best way to picture such example is to expand our previously explained case where application contains map functionalities. Instead of using static image as map and map layers preview, we can use interactive maps, which are not static by any means. Such example can become tricky to automate, as interactive maps are not static images (they can take time to load, and their components can have some offset in preview depending on service you are using). Even though you can play around scaling images or setting some specific time for taking screenshots, these tests will become flaky over time, and you will end up having too many special cases and complex code to maintain. Also, real nightmares start when you have multiple browsers, devices or headless mode to run your visual tests, as image comparison depends tremendously on screen resolution. Therefore we contemplated new way to handle such situations. The optimal solution in our case was to combine functional and visual testing paradigms to avoid manual testing. The ultimate goal was to be sure at any time that maps, markers and map layers are loaded. Other UI components linked to maps (modal titles, map description or legends) could be automated using standard regression testing (via html elements check).

How to automate interactive visual elements?

We found using dominant color algorithm, in combination with processing screenshots, as the most effective way to overcome issues with interactive elements.

Following flow proved to be successful for our test scope:

  • Identify image area (canvas or other html element)
  • Take screenshot of such area
  • Use ImageMagick options to centre and standardize image (to avoid frames and to minimize offset)
  • Use quantize method to extract N most frequent colors (number depends on which feature is tested)
  • Use histogram to show most frequent colors in decreasing order
  • Save histogram and convert its colors to hexadecimal values
  • Check if expected most dominant colors exist on screenshot.
  • If map has map layers (pins, markers, heat-map or other data preview), base-map can be turned off, and layers existence checked. This can be done by checking if white is not the only dominant color for selected UI element (which would mean that element is empty – no layers added)

This method does not give information if maps are always loaded identically – which is the main criteria for static images. The valuable information we get from this method is if map is loaded at all, and if its layers appear every time they are supposed to. This method saved tremendous amount of time we spent manually checking if our map services work as expected (as some of our applications rely very much on maps and similar geo-services). Also, we found this method useful if application contain some kind of ads (especially video ads), which helped us getting away from the situations where our links were broken and we did not notice it on time.

Test implementation

Various implementations of dominant color algorithm can be found over the Internet, depending on programming language and testing framework used for visual tests. As we mentioned before, our tests are written using RSpec, Watir, and Ruby, so we found RMagick gem (a binding from Ruby to the ImageMagick image manipulation library) to be most suitable for test implementation. Hence, we found dominant color algorithm implementation that uses RMagick.

Original map

Before we dive into algorithm details, we will show you how it is used for testing purposes, on dummy map example. Image below shows base-map with red circles showing the intensity of earthquakes in certain geographic area. 

Those circles represent map layer. For purpose of showing dominant color algorithm usage, we set up following scenario:

  • Test should check if map layers are loaded correctly. In such case we do not need to know if maps are generated below such layers, so we configured test environment to show only layers in selected HTML element.
  • This way we can focus on layers of interest only. Image can be found below.

            Image used for layer testing

All steps regarding test implementation will be explained in following sections.

Identify area of interest and save it as image

Similar to the still image identification, we used Watir to identify where our maps are shown (we are fetching element location via Watir xpath wrapper). RMagick method from_blob() in combination with resize_to_fill() is used to manipulate saved image, so that it is resized to fill certain resolution. Such normalized image is saved in desired folder.

 

Use image quantize method to extract N most dominant colours

After image is saved, that image should be quantized. Quantize method is used to reduce image colors to N most frequent ones, by using RMagick quantize() method. This is the first method used to implement dominant color algorithm.

Create histogram from quantized image

Another method from mentioned algorithm is used to manipulate quantized image. Color histogram is created from such image, and that histogram is sorted by decreasing frequency. color_histogram() method from RMagick is used.

Created histogram for our example can be seen here:

Convert histogram values to hexadecimal color notation

To be able to create assertion for our tests, histogram colors are converted into hexadecimal values. Those values are created using to_color() method which converts any pixel color to adequate colors using All Compliances.

 

Create assertions for checking selected UI elements

Finally we can check if our color histogram shows any other colors then white (#000000 in hexadecimal notation). If it is not case, that means that our layers are not loaded, and test will fail.

Output of the test:

Created histogram (saved as image):

Algorithm implementation

In previous steps, we described how dominant color algorithm is used in our scripts. Now we will show algorithm methods we used for testing (complete algorithm can be found here https://gist.github.com/Olefine/5336515):

 

When to use algorithms in visual testing?

Diving deep into image processing algorithms, to be able to use them in visual testing, can be seen as to much work and to little value for QA team. This is true in case you have few images to check, or simple application to test. But when you have complex business logic or complex application, where there are intense set of rules for image presentation and appearance, these kind of test can be of great help and also time-saving. In our case it was worth the effort as we:

  • do not spend our time comparing images manually anymore
  • do not have to analyze complex business rules to see if images are generated correctly

Having all this in mind, it is up to QA team to decide if they are willing to dive into such adventure, and if the output is good enough to replace manual testing.

Conclusions

In our blogs regarding UI testing, we explained two methods that proved the most efficient for our automated visual testing. Those methods can be further expanded to test whole pages. Also they can be combined to get more precise and more stable tests. Our future goal is to create whole visual test suite which will be reliable as much as our standard regression tests are. Previously explained algorithms set us on a good path to achieve these goals.